Why does `subprocess.Popen` with `readline()` hang when reading from a Ruby script, and how can this be fixed?

Python subprocess readlines() hangs

Problem:

When reading the output of a ruby script using subprocess.Popen and readline() in a streaming fashion, readline() hangs indefinitely and never returns.

Background:

The goal is to stream the output of a ruby file line-by-line, printing it without buffering the entire output.

from subprocess import Popen, PIPE, STDOUT
import pty
import os

file_path = '/Users/luciano/Desktop/ruby_sleep.rb'

command = ' '.join(["ruby", file_path])

master, slave = pty.openpty()
proc = Popen(command, bufsize=0, shell=True, stdout=slave, stderr=slave, close_fds=True)     
stdout = os.fdopen(master, 'r', 0)

while proc.poll() is None:
    data = stdout.readline()
    if data != "":
        print(data)
    else:
        break

print("This is never reached!")

The ruby_sleep.rb script outputs a simple message with a 2-second delay:

puts "hello"

sleep 2

puts "goodbye!"

Root Cause:

readline() remains hung because the ruby script outputs data without terminating lines (i.e., without newlines). This causes readline() to wait indefinitely for a newline to complete the line.

Solutions:

Several solutions exist depending on platform availability:

• For Linux:

  Use pty from the standard library to open a pseudo-terminal (tty) and enable line buffering on the ruby's side, ensuring that each line is terminated with a newline.

  import os
  import pty
  from subprocess import Popen, STDOUT
  
  master_fd, slave_fd = pty.openpty()  # provide tty to enable
                                       # line-buffering on ruby's side
  proc = Popen(['ruby', 'ruby_sleep.rb'],
               stdin=slave_fd, stdout=slave_fd, stderr=STDOUT, close_fds=True)
  os.close(slave_fd)
  try:
      while 1:
          try:
              data = os.read(master_fd, 512)
          except OSError as e:
              if e.errno != errno.EIO:
                  raise
              break # EIO means EOF on some systems
          else:
              if not data: # EOF
                  break
              print('got ' + repr(data))
  finally:
      os.close(master_fd)
      if proc.poll() is None:
          proc.kill()
      proc.wait()
  print("This is reached!")

• For Linux-based platforms:

  Use pty from the standard library and select to monitor the master file descriptor for activity, ensuring that data is read in a non-blocking manner.

  import os
  import pty
  import select
  from subprocess import Popen, STDOUT
  
  master_fd, slave_fd = pty.openpty()  # provide tty to enable
                                       # line-buffering on ruby's side
  proc = Popen(['ruby', 'ruby_sleep.rb'],
               stdout=slave_fd, stderr=STDOUT, close_fds=True)
  
  timeout = .04 # seconds
  while 1:
      ready, _, _ = select.select([master_fd], [], [], timeout)
      if ready:
          data = os.read(master_fd, 512)
          if not data:
              break
          print("got " + repr(data))
      elif proc.poll() is not None: # select timeout
          assert not select.select([master_fd], [], [], 0)[0] # detect race condition
          break # proc exited
  os.close(slave_fd) # can't do it sooner: it leads to errno.EIO error
  os.close(master_fd)
  proc.wait()
  
  print("This is reached!")

• Cross-platform option:

  Use stdbuf to enable line buffering in non-interactive mode.

  from subprocess import Popen, PIPE, STDOUT
  
  proc = Popen(['stdbuf', '-oL', 'ruby', 'ruby_sleep.rb'],
               bufsize=1, stdout=PIPE, stderr=STDOUT, close_fds=True)
  for line in iter(proc.stdout.readline, b''):
      print line,
  proc.stdout.close()
  proc.wait()

These solutions all enable line buffering on the ruby's side, ensuring that each line is terminated with a newline, allowing readline() to function correctly.

The above is the detailed content of Why does `subprocess.Popen` with `readline()` hang when reading from a Ruby script, and how can this be fixed?. For more information, please follow other related articles on the PHP Chinese website!