How to Efficiently Read and Process Subprocess Output Line by Line in Python?

How to Read Subprocess stdout Line by Line

In Python, using subprocess to call a Linux utility that generates a large volume of output can pose challenges if you want to capture and process the output incrementally.

In the parent process, the for line in proc.stdout statement reads the entire input before iterating over it. This can lead to the output not appearing in your application until a significant amount has been generated.

To address this issue, use proc.stdout.readline() instead:

import subprocess

proc = subprocess.Popen(['python', 'fake_utility.py'], stdout=subprocess.PIPE)
while True:
  line = proc.stdout.readline()
  if not line:
    break
  #the real code does filtering here
  print "test:", line.rstrip()

This solution allows you to filter and print each line of output as it becomes available from the subprocess. Keep in mind that you may still need to consider the subprocess's buffering behavior.

The above is the detailed content of How to Efficiently Read and Process Subprocess Output Line by Line in Python?. For more information, please follow other related articles on the PHP Chinese website!