The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.
In [1]:
import subprocess
In [23]:
completed = subprocess.run(['ls', '-l'])
completed
Out[23]:
The standard input and output channels for the process started by run() are bound to the parent’s input and output. That means the calling program cannot capture the output of the command. Pass PIPE for the stdout and stderr arguments to capture the output for later processing.
In [24]:
completed = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE)
completed
Out[24]:
For cases where the output should not be shown or captured, use DEVNULL to suppress an output stream. This example suppresses both the standard output and error streams.
In [32]:
import subprocess
try:
completed = subprocess.run(
'echo to stdout; echo to stderr 1>&2; exit 1',
shell=True,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
except subprocess.CalledProcessError as err:
print('ERROR:', err)
else:
print('returncode:', completed.returncode)
print('stdout is {!r}'.format(completed.stdout))
print('stderr is {!r}'.format(completed.stderr))
setting the shell argument to a true value causes subprocess to spwan an intermediate shell process which runs the command. the default is to run the command directly
In [29]:
import subprocess
completed = subprocess.run('echo $HOME', shell=True, stdout=subprocess.PIPE)
completed
Out[29]:
if you don't run this command on a shell, this is a error, because HOME is not defined
In [31]:
import subprocess
try:
completed = subprocess.run('echo $HOME', stdout=subprocess.PIPE)
except:
print("Get Error if don't execute on shell")
The functions run(), call(), check_call(), and check_output() are wrappers around the Popen class. Using Popen directly gives more control over how the command is run, and how its input and output streams are processed. For example, by passing different arguments for stdin, stdout, and stderr it is possible to mimic the variations of os.popen().
In [33]:
import subprocess
print("read:")
proc = subprocess.Popen(['echo', '"to stdout"'],
stdout = subprocess.PIPE)
stdout_value = proc.communicate()[0].decode("utf-8")
print('stdout', repr(stdout_value))
In [34]:
import subprocess
cat = subprocess.Popen(
['cat', 'index.rst'],
stdout=subprocess.PIPE,
)
grep = subprocess.Popen(
['grep', '.. literalinclude::'],
stdin=cat.stdout,
stdout=subprocess.PIPE,
)
cut = subprocess.Popen(
['cut', '-f', '3', '-d:'],
stdin=grep.stdout,
stdout=subprocess.PIPE,
)
end_of_pipe = cut.stdout
print('Included files:')
for line in end_of_pipe:
print(line.decode('utf-8').strip())
In [ ]:
The process management examples for the os module include a demonstration of signaling between processes using os.fork() and os.kill(). Since each Popen instance provides a pid attribute with the process id of the child process, it is possible to do something similar with subprocess. The next example combines two scripts. This child process sets up a signal handler for the USR signal.
In [ ]:
# %load signal_child.py
import os
import signal
import time
import sys
pid = os.getpid()
received = False
def signal_usr1(signum, frame):
"Callback invoked when a signal is received"
global received
received = True
print('CHILD {:>6}: Received USR1'.format(pid))
sys.stdout.flush()
print('CHILD {:>6}: Setting up signal handler'.format(pid))
sys.stdout.flush()
signal.signal(signal.SIGUSR1, signal_usr1)
print('CHILD {:>6}: Pausing to wait for signal'.format(pid))
sys.stdout.flush()
time.sleep(3)
if not received:
print('CHILD {:>6}: Never received signal'.format(pid))
In [ ]:
# %load signal_parent.py
import os
import signal
import subprocess
import time
import sys
proc = subprocess.Popen(['python3', 'signal_child.py'])
print('PARENT : Pausing before sending signal...')
sys.stdout.flush()
time.sleep(1)
print('PARENT : Signaling child')
sys.stdout.flush()
os.kill(proc.pid, signal.SIGUSR1)
In [41]:
!python signal_parent.py
If the process created by Popen spawns sub-processes, those children will not receive any signals sent to the parent. That means when using the shell argument to Popen it will be difficult to cause the command started in the shell to terminate by sending SIGINT or SIGTERM.
In [43]:
import os
import signal
import subprocess
import tempfile
import time
import sys
script = '''#!/bin/sh
echo "Shell script in process $$"
set -x
python3 signal_child.py
'''
script_file = tempfile.NamedTemporaryFile('wt')
script_file.write(script)
script_file.flush()
proc = subprocess.Popen(['sh', script_file.name])
print('PARENT : Pausing before signaling {}...'.format(
proc.pid))
sys.stdout.flush()
time.sleep(1)
print('PARENT : Signaling child {}'.format(proc.pid))
sys.stdout.flush()
os.kill(proc.pid, signal.SIGUSR1)
time.sleep(3)
The pid used to send the signal does not match the pid of the child of the shell script waiting for the signal, because in this example there are three separate processes interacting:
To send signals to descendants without knowing their process id, use a process group to associate the children so they can be signaled together. The process group is created with os.setpgrp(), which sets process group id to the process id of the current process. All child processes inherit their process group from their parent, and since it should only be set in the shell created by Popen and its descendants, os.setpgrp() should not be called in the same process where the Popen is created. Instead, the function is passed to Popen as the preexec_fn argument so it is run after the fork() inside the new process, before it uses exec() to run the shell. To signal the entire process group, use os.killpg() with the pid value from the Popen instance.
In [44]:
import os
import signal
import subprocess
import tempfile
import time
import sys
def show_setting_prgrp():
print('Calling os.setpgrp() from {}'.format(os.getpid()))
os.setpgrp()
print('Process group is now {}'.format(
os.getpid(), os.getpgrp()))
sys.stdout.flush()
script = '''#!/bin/sh
echo "Shell script in process $$"
set -x
python3 signal_child.py
'''
script_file = tempfile.NamedTemporaryFile('wt')
script_file.write(script)
script_file.flush()
proc = subprocess.Popen(
['sh', script_file.name],
preexec_fn=show_setting_prgrp,
)
print('PARENT : Pausing before signaling {}...'.format(
proc.pid))
sys.stdout.flush()
time.sleep(1)
print('PARENT : Signaling process group {}'.format(
proc.pid))
sys.stdout.flush()
os.killpg(proc.pid, signal.SIGUSR1)
time.sleep(3)
In [ ]: