Basic Tutorial on Shell Pipe Redirection

Pipeline exists to solve the problem of inter-process communication. It allows data to be transferred between two processes, and the output data of one process is transferred to another process as its input data

1.8.1 Anonymous Pipe"|"

The pipe symbol means as its name suggests. It passes the data from the pipe inlet to the pipe outlet through the pipe, just like a pipe.

Pipeline exists to solve the problem of inter-process communication. It allows data to be transferred between two processes, and the output data of one process is transferred to another process as its input data. The left side of the pipe is the data giver, and the right side of the pipe is the data receiver.

For exampleecho "abcd" | passwd --stdin username means that the output result "abcd" of the process echo is used as the input data of the process passwd.

Basic pipe symbols and their usage are easy to understand. The question now is, for ps aux | grep "ssh" , why does the grep process appear in the results?

[root@xuexi ~]# ps aux | grep ssh
root    1211 0.0 0.1 82544 3600 ?    Ss  Jul26  0:00 /usr/sbin/sshd -D
root   25236 0.0 0.2 145552 5524 ?    Ss  05:28  0:00 sshd: root@pts/0
root   25720 0.1 0.2 145416 5524 ?    Ss  06:15  0:00 sshd: root@pts/1
root   25770 0.0 0.0 112648  948 pts/1  S+  06:15  0:00 grep --color=auto ssh

According to the general idea, ps is executed first, and after getting the output, the output data is passed to grep. At this time, grep has not run but ps has finished running. Why? Can we collect statistics about the grep process? The reason is that the pipeline implements inter-process communication, and there is overlap between the two processes. After running the ps process, the process information starts to be collected. grep has also started and is in the state of waiting to receive data. When ps collects any data, the output will be released. The data is passed into memory and piped to grep for filtering.

The essence of a pipe is data transfer. The output data on the left side of the pipe is put into memory and read by the process on the right side of the pipe. If the memory is not enough to completely store the output data, the process on the left side of the pipe will wait until the right side of the pipe takes out part of the data in the memory to allow the process on the left side of the pipe to continue output, and the process on the right side of the pipe will start immediately after the process on the left side of the pipe starts. Started, but it has been in a waiting state, waiting to receive data passed by the pipe.

In other words, the processes on the left and right sides of the pipe run in almost no order.

So how does ps aux | grep "ssh" prevent grep's own process from appearing in the results? There are two methods:

Method one: ps aux | grep "ssh" | grep -v "grep"

Method two: ps aux | grep "ss[h ]"

[root@xuexi ~]# ps aux | grep ss[h]
root    1211 0.0 0.1 82544 3600 ?    Ss  Jul26  0:00 /usr/sbin/sshd -D
root   25236 0.0 0.2 145552 5524 ?    Ss  05:28  0:00 sshd: root@pts/0
root   25720 0.0 0.2 145416 5524 ?    Ss  06:15  0:00 sshd: root@pts/1

Method one is to use the "-v" feature of grep, and method two is to use the feature of regular expressions.

In the process of using anonymous pipes, you may have discovered that the processes on both sides of the pipe belong to the same process group, which means that the data on the left side of the pipe can only be passed to the process on the right side of the pipe. No process can read this data. But in addition to anonymous pipes, there are also named pipes. A named pipe stores the data of a process in a pipe file (fifo). Other processes can read the pipe file to read the data in it, which means that the data is no longer restricted. Reader. Regarding named pipes, please refer to Linux/unix operating system kernel or programming books, which will generally provide detailed introductions.

1.8.2 Redirection

1.8.2.1 Redirect basics

The most common The file descriptors of standard input (stdin), standard output (stdout) and standard error output (stderr) are 0, 1 and 2 respectively, where 0, 1 and 2 can also be considered as their numerical codes. The output information can be thought of as the information printed on the screen, and the one that does not give an error is the standard output, and the one that gives an error prompt is the standard error output. Of course, this explanation is biased, but it is easy to understand. You can also customize your own descriptors to implement advanced redirection. Their usage may be introduced in future articles.

Standard input = /dev/stdin = code 0 = < or << symbol.

Standard output = /dev/stdout = code 1 = > or >> symbol.

Standard error output = /dev/stderr = code 2 = use 2> or 2>> notation.

<, >, 2> implement the overwriting function, >>, 2>> implement the additional function, but << is not an additional function, but means here Generate a document (here document), which is explained later in the content of the cooperation between cat and redirection. In addition, there is <<<, which means a here string, see also below.

Sometimes, using "-" also means /dev/stdin. For example:

##

[root@xuexi ~]# cat /etc/fstab | cat -

The symbols 2>&1 and &> are common in scripts. They all mean to redirect both stdout and stderr to the same place, that is, redirect all Output content. Such as the most common "&> /dev/null".

Throwing stdout or stderr to /dev/null means discarding the output information. Conversely, redirecting /dev/null to a file means clearing the file.

[root@xuexi ~]# cat /dev/null > ab.sh

In addition, there are several ways to quickly clear files

[root@xuexi ~]# > ab.sh
[root@xuexi ~]# : > ab.sh       # 或"true >ab.sh"，其实它们都等价于">ab.sh"
[root@xuexi ~]# echo '' > ab.sh
[root@xuexi ~]# truncate -s 0 ab.sh  # truncate命令用于收缩和扩展文件大小
[root@xuexi ~]# dd if=/dev/null of=ab.sh

最后最重要的一点：在有重定向符号的语句中，命令执行之前已经将文件截断了。所以如果正在编辑一个文件并将编辑的结果重定向回这个文件将出现异常，因为截断后就没有合适的内容用于编辑。一个简单的示例如下：

[root@xuexi ~]# head a.log > a.log

有些时候直接使用">"覆盖输出是比较危险的。可以使用set -C来设置如果输出重定向文件已经存在则不覆盖。使用set +C来取消set -C的效果。如果在设置了set -C时仍然想强制覆盖，可以使用“>|”代替“>”来重定向输出。同理错误输出也有此特性。

[root@xuexi tmp]# set -C
[root@xuexi tmp]# cat flip >ttt.txt
-bash: ttt.txt: cannot overwrite existing file
[root@xuexi tmp]# cat flip >| ttt.txt
[root@xuexi tmp]# set +C

1.8.2.2 cat和重定向配合

配合cat使用可以分行输入内容到文件中。

[root@xuexi tmp]# cat <<eof>log.txt  # 覆盖的方式输入到log.txt
> this is stdin character
> eof

也可以使用下面的方法。

[root@xuexi tmp]# cat >log1.txt <<eof 
> this is stdin character first!
> eof

一方面，eof部分都必须使用"<

[root@xuexi ~]# cat <<abcx
> 123
> 345
> abcx
123
345

另一方面，>log1.txt表示将document的内容覆盖到log1.txt文件中，如果是要追加，则使用>>log1.txt。所以，追加的方式如下：

[root@xuexi tmp]# cat >>log1.txt <<eof 
> this is stdin character first!
> eof

或

[root@xuexi tmp]# cat <<eof>>log1.txt 
> this is stdin character first!
> eof

1.8.2.3 tee双重定向

可以使用tee双重定向。一般情况下，重定向要么将信息输入到文件中，要么输出到屏幕上，但是既想输出到屏幕又想输出到文件就比较麻烦。使用tee的双重定向功能可以实现该想法。如图。

tee [-a] file

选项说明：

-a：默认是将输出覆盖到文件中，使用该选项将变为追加行为。

file：除了输出到标准输出中，还将输出到file中。如果file为"-"，则表示再输入一次到标准输出中。

例如下面的代码，将a开头的文件内容全部保存到b.log，同时把副本交给后面的的cat，使用这个cat又将内容保存到了x.log。其中"-"代表前面的stdin。

[root@xuexi tmp]# cat a* | tee b.log | cat - >x.log

还可以直接输出到屏幕：

[root@xuexi tmp]# cat a* | tee b.log | cat

tee默认会使用覆盖的方式保存到文件，可以使用-a选项来追加到文件。如：

[root@xuexi tmp]# cat a* | tee -a b.log | cat

现在就可以在使用cat和重定向创建文件或写入内容到文件的同时又可以在屏幕上显示一份。

[root@xuexi tmp]# cat <<eof | tee ttt.txt
> x y
> z 1
> eof
x y
z 1

1.8.2.4 <<和<<<

在bash中，<<和<<<是特殊重定向符号。<<表示的是here document，<<<表示的是here string。

here document在上文已经解释过了，对于here string，表示将<<<后的字符串作为输入数据。

例如：

passwd --stdin user <<< password_value

等价于：

echo password_value | passwd --stdin user

The above is the detailed content of Basic Tutorial on Shell Pipe Redirection. For more information, please follow other related articles on the PHP Chinese website!