Pipes and more pipes on Linux

Most people who spend time on the Linux command line move quickly into using pipes. In fact, pipes were one of the things that really got me excited when I first used the command line on a Unix system. My appreciation of their power and convenience continues even after decades of using Linux. Using pipes, I discovered how much I could get done by sending the output of one command to another command, and sometimes a command after that, to further tailor the output that I was looking for. Commands incorporating pipes – like the one shown below – allowed me to extract just the information that I needed without having to compile a program or prepare a script.

$ cat myposts | grep Linux | wc -l
1456

The command above would send the content of a file called “myposts” to a grep command looking for the word “Linux” and then send that output to a wc command to count the number of lines in the output.

As you probably suspect, pipes are called “pipes” largely because of how they resemble the function of pipelines. You might see them referred to as “unnamed pipes” or “anonymous pipes” because they are not the only kind of pipes that Linux provides. We’ll get to that shortly.

In fact, pipes are so useful that I often turn some of the commands that incorporate them into aliases to make running them even that much easier. For example, the command shown below that lists all processes associated with the current login account could be added to the user’s .bashrc file with a line like this:

$ echo 'alias myprocs="ps -ef | grep `whoami`"' >> ~/.bashrc
$ tail -1 ~/.bashrc
alias myprocs="ps -ef | grep `whoami`"

Once you source your .bashrc file to , the alias will be ready to use.

$ myprocs
root        3219     738  0 12:15 ?        00:00:00 sshd: shs [priv]
shs         3229       1  4 12:15 ?        00:00:00 /usr/lib/systemd/systemd --user
shs         3245    3229  0 12:15 ?        00:00:00 (sd-pam)
shs         3269    3219  0 12:15 ?        00:00:00 sshd: shs@pts/0
shs         3284    3269  0 12:15 pts/0    00:00:00 -bash
shs         3319    3284  0 12:15 pts/0    00:00:00 ps -ef
shs         3320    3284  0 12:15 pts/0    00:00:00 grep --color=auto shs

To see just the process IDs, you could set up an alias like this one:

$ alias myps=”ps aux | grep ^`whoami` | awk '{print $2}'”

Note that it only looks for the username in the first field (indicated by the ^ to mark the beginning of the lines and displays the second field. The ensures that $2 is not interpreted until the alias is use.

The pipes that are used to set up a data stream that allows commands to pass their output to other commands isn’t, however, the only kind of pipe available on Linux systems. On Linux systems, there are actually two quite different forms of pipes – those shown above and another form of pipes that are called “named pipes”.

Named pipes

Unlike unnamed pipes, named pipes are quite different in that they can send data in either direction. Commands can send data to named pipes and command can read that content. An addition, the content of named pipes doesn’t reside in the file system, but only in virtual memory.

They allow processes to communicate with each other. They are set up as special files in the file system (indicated by the first character in a long listing being a “p”. Other permissions indicate who can read or write to the pipe.

Here’s an example of creating a named pipe with the mkfifo command:

$ mkfifo mypipe
$ ls -l mypipe
prw-r--r--. 1 justme justme 0 Aug  8 13:55 mypipe

Note the initial “p” in the listing shown above that indicates the file is a named pipe and the 0 (5th field) that shows it has no content.

Using a -m argument, you can set permissions to allow other users to write to pipes. Note that the default is that the owner can read an write and others can only read. Here’s an example:

$ mkfifo -m 666 sharedpipe
$ ls -l mypipe0
prw-rw-rw-. 1 shs shs 0 Aug  7 12:50 sharedpipe

Even when you send data to a named pipe, it appears to be empty.

$ echo “Here comes some content” > mypipe
$ ls -l mypipe
prw-r--r--. 1 justme justme 0 Aug   13:57 mypipe

Another process might read the content with a command like this:

$ cat mypipe
Here comes some content

If we run the command that sends data into the pipe in the background, we can read it with a cat command. Notice that the file still appears to be empty in the file listing, but we can retrieve the text just once with a cat command. After it’s rea, it’s gone. Nothing, however, keeps us from sending more data into the pipe – as long as it still exists.

$ echo "Here comes some content" > mypipe &
[1] 1593
$ ls -l mypipe
prw-r--r--. 1 fedora fedora 0 Aug  5 13:55 mypipe
$ cat mypipe
Here comes some content
[1]+  Done                    echo "Here comes some content" > mypipe
$ cat mypipe
^C

Wrap-up

Named pipes are more complex than unnamed pipes and are far less frequently used, but they play an interesting role on Linux systems. Here are two earlier posts – one on using unnamed posts and one on using named pipes.

Using pipes on Linux to get more done

Why use named pipes on Linux