How can I execute multiple Bash commands concurrently in Python?

Parallel Processing of Bash Subprocesses in Python

Running subprocesses sequentially can hinder the performance of your application. To execute multiple bash commands concurrently, you can leverage the threading and subprocess modules in Python.

Using Subprocesses Directly

While threading may seem necessary for parallel processing, it's not required. You can use the subprocess.Popen function to launch processes in parallel:

<code class="python">from subprocess import Popen

commands = [
    'date; ls -l; sleep 1; date',
    'date; sleep 5; date',
    'date; df -h; sleep 3; date',
    'date; hostname; sleep 2; date',
    'date; uname -a; date',
]

# Run commands in parallel
processes = [Popen(cmd, shell=True) for cmd in commands]

# Perform other tasks while processes run

# Wait for completion
for p in processes:
    p.wait()</code>

Limiting Concurrent Subprocesses

To limit the number of concurrent processes, consider using the multiprocessing.dummy.Pool module, which simulates multiprocessing using threads:

<code class="python">from functools import partial
from multiprocessing.dummy import Pool
from subprocess import call

pool = Pool(2)  # Limit to two concurrent commands

# Iterate over commands and return codes
for i, returncode in enumerate(pool.imap(partial(call, shell=True), commands)):
    if returncode != 0:
        print("%d command failed: %d" % (i, returncode))</code>

Limiting Without Pools

You can also limit concurrency without thread pools:

<code class="python">from subprocess import Popen
from itertools import islice

max_workers = 2

processes = (Popen(cmd, shell=True) for cmd in commands)
running_processes = list(islice(processes, max_workers))  # Start initial processes

while running_processes:
    for i, process in enumerate(running_processes):
        if process.poll() is not None:  # Process has finished
            running_processes[i] = next(processes, None)  # Start new process
            if running_processes[i] is None:  # No new processes
                del running_processes[i]
                break</code>

The above is the detailed content of How can I execute multiple Bash commands concurrently in Python?. For more information, please follow other related articles on the PHP Chinese website!