A Deep Dive into Same‑Machine Named Pipes (FIFOs)

by

Introduction – Why “Pipe Dreams” Matter on a Single Host

Imagine two applications on the same server that need to exchange data instantly, without the overhead of network sockets or the complexity of shared memory. That’s the sweet spot where same‑machine named pipes, also known as FIFOs, shine.

These lightweight, file‑system objects have been part of Unix‑like operating systems since the early 1970s, yet many developers still overlook them in favor of more “modern” IPC mechanisms. In reality, a FIFO can be the most straightforward, secure, and performant way to set up interprocess communication (IPC) between scripts, services, or daemons running on the same host.

In this post we’ll explore what same‑machine named pipes are, how they differ from anonymous pipes, when to use them, and step‑by‑step how to create, manage, and troubleshoot them on Linux and macOS. By the end, you’ll have a ready‑to‑run toolkit for adding FIFO‑based communication to any local application stack.

 

1. What Exactly Is a Same‑Machine Named Pipe (FIFO)?

1.1 Definition and Core Characteristics

A named pipe—technically a FIFO (First‑In‑First‑Out) special file—acts like a conduit that lives in the file system. Unlike an anonymous pipe (the `|` operator in a shell), a FIFO has a persistent name, usually under `/tmp` or another directory you control.

| Feature | Anonymous Pipe | Named Pipe (FIFO) |
|———|—————-|——————-|
| Lifetime | Exists only while the creating process runs | Persists until explicitly removed (`rm`) |
| Visibility | Invisible to other processes | Visible as a file node (type `p`) |
| Scope | Typically parent‑child processes | Any processes on the same host can open it |
| Creation | `pipe()` system call | `mkfifo` command or `mkfifo()` API |

Because the data flow follows a strict first‑in‑first‑out order, a FIFO guarantees that the first byte written is the first byte read—ideal for streaming logs, command output, or simple request/response patterns.

1.2 How FIFOs Fit Into the IPC Landscape

When you think about interprocess communication, you usually picture sockets, shared memory, message queues, or DBus. FIFOs sit in a niche that blends the simplicity of files with the speed of kernel‑mediated data transfer:

    • Low latency – Data never hits the network stack; the kernel copies it directly between the writer’s buffer and the reader’s buffer.
    • Security by permissions – Since a FIFO is a file, you can set Unix permissions (`chmod`, `chown`) to restrict who can read or write.
    • No need for a broker – Unlike a message queue that may require a daemon, a FIFO works out‑of‑the‑box.

These traits make FIFOs perfect for:

    • Log aggregation – Multiple processes write logs to a single FIFO, and a consumer daemon parses them in real time.
    • Task pipelines – A producer script generates work items; a worker reads them, processes, and optionally writes results to another FIFO.
    • Configuration hot‑reload – A service watches a FIFO for new configuration snippets, applying them without a restart.

 

2. Creating and Using FIFOs – The Hands‑On Guide

2.1 Making a FIFO with `mkfifo`

The classic command‑line tool is `mkfifo`. The syntax is straightforward:

“`bash

mkfifo /tmp/mypipe

Or set explicit permissions at creation time

mkfifo -m 0640 /tmp/secure_pipe
“`

Behind the scenes, `mkfifo` invokes the `mkfifo()` system call, which registers a special file node of type p (for pipe) in the directory you specify.

2.2 Opening a FIFO in a Shell Script

A FIFO behaves like a regular file for `cat`, `echo`, or redirection operators. Here’s a minimal producer‑consumer pair:

“`bash

consumer.sh – reads line by line

#!/bin/bash
while IFS= read -r line; do
echo “Received: $line”
done < /tmp/mypipe
“`

“`bash

producer.sh – writes data

#!/bin/bash
for i in {1..5}; do
echo “Message $i” > /tmp/mypipe
sleep 1
done
“`

Run `consumer.sh` in one terminal, then `producer.sh` in another. The consumer blocks until the producer writes, demonstrating the blocking semantics of FIFOs:

    • Writer blocks when the pipe’s buffer (usually 64 KB) is full.
    • Reader blocks when the pipe is empty.

2.3 Using FIFOs in C / Python

#### C Example

“`c
#include
#include
#include

int main(void) {
const char *path = “/tmp/c_fifo”;
mkfifo(path, 0666); // Create if not exists

int fd = open(path, O_WRONLY); // Open for writing (blocks until reader)
const char *msg = “Hello from C!n”;
write(fd, msg, strlen(msg));
close(fd);
return 0;
}
“`

#### Python Example

“`python
import os, time

fifopath = “/tmp/pyfifo”
if not os.path.exists(fifo_path):
os.mkfifo(fifo_path)

Producer

with open(fifo_path, “w”) as fifo:
for i in range(3):
fifo.write(f”Python says {i}n”)
fifo.flush()
time.sleep(1)
“`

Both snippets illustrate that the same API (`open`, `read`, `write`) works across languages, making FIFOs a language‑agnostic IPC choice.

2.4 Managing FIFO Lifetime

Because a FIFO is a file, you must clean it up when it’s no longer needed:

“`bash
rm -f /tmp/mypipe
“`

If you forget to remove a FIFO, subsequent runs of a script that calls `mkfifo` may fail with `File exists`. A common pattern is to test for existence first:

“`bash
[ -p /tmp/mypipe ] && rm /tmp/mypipe
mkfifo /tmp/mypipe
“`

 

3. Best Practices – Making FIFOs Work for You

3.1 Choose the Right Directory

    • Temporary data – `/tmp` is fine, but remember it may be cleared on reboot.
    • Persistent pipelines – Use a dedicated directory like `/var/run/myapp/` and ensure the owning user has write permission.

3.2 Set Tight Permissions

Never leave a FIFO world‑writable unless you truly need it. Example for a service that only a specific user should access:

“`bash
mkfifo -m 0600 /var/run/myapp/command_fifo
chown myservice:myservice /var/run/myapp/command_fifo
“`

3.3 Handle Blocking Gracefully

Blocking reads are useful, but they can also deadlock your program if a writer never appears. Mitigation strategies:

    • Open the FIFO in non‑blocking mode (`O_NONBLOCK`) and poll with `select()` or `poll()`.
    • Use a timeout loop in shell scripts:

“`bash
timeout 5 cat /tmp/mypipe || echo “No data within 5 seconds”
“`

    • Provide a “heartbeat” writer that periodically writes a keep‑alive line, ensuring the reader never hangs indefinitely.

3.4 Monitor Buffer Limits

The kernel pipe buffer size varies (often 64 KB, configurable via `/proc/sys/fs/pipe-max-size`). If your producer writes large bursts, you may encounter EAGAIN errors in non‑blocking mode. To avoid this:

    • Chunk data into smaller pieces.
    • Increase the buffer (Linux only) with `fcntl(fd, FSETPIPESZ, newsize)`.

3.5 Combine FIFOs with Systemd or Supervisor

When running services under systemd, you can declare a FIFO as a `StandardInput` or `StandardOutput` target:

“`ini
[Service]
ExecStart=/usr/local/bin/worker
StandardInput=pipe:/run/myapp/input_fifo
StandardOutput=pipe:/run/myapp/output_fifo
“`

Systemd will create the FIFO for you if it doesn’t exist, and it will handle clean‑up on service stop. This integration reduces boilerplate and ensures proper startup ordering.

 

4. Real‑World Use Cases – When FIFOs Outperform Other IPC

4.1 Log Forwarding for Containerized Apps

In a Docker container, writing directly to `stdout` may be noisy. Instead, an internal process can write structured logs to a FIFO (`/var/log/app.fifo`). A side‑car container mounts the same volume and reads the FIFO, forwarding logs to a central ELK stack. This decouples log generation from transport, keeping the container lightweight.

4.2 Lightweight Job Queues

For small‑scale background processing, a FIFO can replace a heavyweight message broker. Example:

    • Producer – A web server script pushes URLs to `/tmp/url_queue`.
    • Consumer – A pool of worker processes reads from the FIFO, fetches the URLs, and stores results elsewhere.

Because the FIFO guarantees order, you get a simple first‑come‑first‑served queue without installing RabbitMQ or Redis.

4.3 Real‑Time Monitoring Dashboards

A system monitoring daemon writes metric snapshots to a FIFO (`/tmp/metrics.fifo`). A web UI built with Node.js reads the pipe, parses JSON lines, and pushes updates via WebSockets to browsers. The result is a low‑latency, push‑based dashboard without any external database.

 

5. Troubleshooting Common FIFO Issues

| Symptom | Likely Cause | Quick Fix |
|———|————–|———–|
| Reader hangs forever | No writer opened, or writer opened after reader blocked | Start writer first, or open FIFO in non‑blocking mode and retry |
| `EPIPE` error on write | Reader closed the pipe (e.g., consumer exited) | Handle `SIGPIPE` or check return value; restart consumer |
| “File exists” on `mkfifo` | Stale FIFO left from previous run | `rm -f /path/to/fifo` before creating, or use `mkfifo -m` with `-f` logic |
| Data loss under heavy load | Pipe buffer overflow, writer blocked, and writer timed out | Increase buffer size (`fcntl`), or throttle producer |
| Permission denied | FIFO owned by another user or group | Adjust ownership (`chown`) and mode (`chmod`) to match your processes |

A handy diagnostic command is `ls -l /path/to/fifo` to verify type (`p`) and permissions, and `lsof | grep fifo` to see which processes currently have the pipe open.

—

Conclusion – Key Takeaways

  • Same‑machine named pipes (FIFOs) are simple, fast, and secure IPC primitives that live as special files in the filesystem.
  • They persist beyond process lifetimes, can be permission‑controlled, and work across any language that can open a file descriptor.
  • Use `mkfifo` (or the `mkfifo()` API) to create them, and remember to clean up with `rm` when done.
  • Follow best practices: place FIFOs in appropriate directories, set tight permissions, handle blocking behavior, and monitor buffer limits.
  • Real‑world scenarios—log forwarding, lightweight job queues, and real‑time dashboards—show that FIFOs often outperform sockets or message brokers for local, ordered data streams.

By mastering same‑machine named pipes, you add a versatile tool to your developer’s toolbox that can dramatically simplify local communication patterns, reduce dependency overhead, and keep your applications snappy. Give FIFOs a try in your next micro‑service or script, and you’ll see why they’ve endured for half a century.

—

Keywords: same machine named pipes, FIFO, interprocess communication, IPC, mkfifo, Linux pipe, Unix named pipe, pipe buffer, systemd FIFO, log aggregation, job queue, real‑time monitoring

Leave a Comment