How Can I Achieve Non-Blocking Line-by-Line Output Filtering from Subprocesses in Python?

Non-Blocking Output Filtering from Subprocesses: A Line-by-Line Approach

In Python programming, the ability to capture and process output from external commands is crucial. One common scenario involves using the subprocess module to execute a command and retrieve its output. However, challenges arise when the output is extensive, and we need to filter and display it incrementally.

The Problem of Delayed Output

Consider the following Python script that calls a utility that generates大量输出：

import time
i = 0
while True:
    print(hex(i)*512)
    i += 1
    time.sleep(0.5)

In our parent process, we attempt to read and filter the output:

import subprocess
proc = subprocess.Popen(['python', 'fake_utility.py'], stdout=subprocess.PIPE)
for line in proc.stdout:
    # perform filtering logic
    print("test:", line.rstrip())

Initially, the expectation was that the output from the utility would be displayed line by line as it became available. However, this did not occur; instead, the output was only shown after a significant amount had been produced.

Inefficiency of Iterators

The reason for this delay lies in the use of the for loop over proc.stdout. This implicitly reads the entire output into memory before processing it iteratively. To resolve this issue, we can employ a more efficient method: using readline().

import subprocess
proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
while True:
  line = proc.stdout.readline()
  if not line:
    break
  # perform filtering logic
  print "test:", line.rstrip()

This approach enables us to read the output line by line as it becomes available, eliminating the delay and providing a more responsive filter process.

Conclusion

Filtering output from subprocesses can be a common task in Python. By understanding the nuances of I/O buffering and employing efficient techniques like readline(), developers can implement non-blocking filters that process and display data incrementally, enhancing the responsiveness of their applications.

The above is the detailed content of How Can I Achieve Non-Blocking Line-by-Line Output Filtering from Subprocesses in Python?. For more information, please follow other related articles on the PHP Chinese website!