How to run shell commands in Python

Python can be used to automate various tasks, and sometimes these tasks involve interacting with the underlying operating system. Running shell commands allows you to execute system commands as part of your automation scripts.

In some cases, it might be quicker to use existing shell commands to perform certain tasks rather than writing the equivalent functionality in Python, especially for one-off or quick prototyping situations.

For example: suppose you want to create a Python script that backs up a directory. You could leverage the existing functionality of the rsync command (a popular file synchronization tool) to achieve this.

There are different ways to run shell commands in Python, but the recommended way is to use the subprocess module, which provides more flexibility and control over the execution of external commands.

Using os.system()

The os.system() function is a simple way to run a shell command in Python. It takes a string as an argument and passes it to the system’s shell.

import os

# Run the 'ls' command to list files in the current directory
os.system('ls')

# Find all Python files in the current directory
os.system("find . -name '*.py'")

However, this function has some limitations and drawbacks. It does not return the output of the command, only the exit code. It also does not allow you to manipulate the input or output streams of the command. Moreover, it is unsafe if the command contains user input or special characters that need to be escaped.

Therefore, it is not recommended to use this function for complex or sensitive tasks.

Using subprocess.run()

A better alternative is to use the subprocess module, which provides finer control over shell commands, including capturing output, error handling, and non-blocking execution.

import subprocess

# Run a simple command and capture the output
result = subprocess.run(["ls", "-l"], capture_output=True)
print(result.stdout.decode())  # Print the command output

# Get the return code of the command
return_code = result.returncode  # 0 for success

# Run a command with arguments
subprocess.run(["grep", "-i", "error", "log.txt"])

The subprocess module also allows you to specify the arguments as a list, which avoids the need for shell escaping. It’s generally safer to use a list of arguments instead of a single string when providing the command to run.

You can also use the shell=True argument to enable the shell features, such as pipes and redirections, but this should be used with caution as it can pose a security risk, and make sure that any user input is properly sanitized to prevent command injection vulnerabilities.

Using subprocess.Popen()

The subprocess.Popen() function is a low-level interface to create and manage subprocesses. It takes a list or a string as the first argument, which specifies the command and its arguments. It also takes several optional arguments to control the input/output/error streams, the working directory, the environment variables, and other aspects of the subprocess.

import subprocess
process = subprocess.Popen(["echo", "Hello world"], stdout=subprocess.PIPE)
output, error = process.communicate()
print(output.decode())

This will print Hello world to the standard output. The communicate() method waits for the subprocess to terminate and returns a tuple of its output and error streams. The output is a byte string, so you need to decode it to get a normal string.

If you want to run a shell command that contains special characters or features, such as pipes, redirections, or wildcards, you need to use the shell=True argument. This will invoke the system’s shell to execute the command. However, this can be a security risk if the command contains user input or untrusted data. You also need to pass the command as a single string, not a list. For example, you can use it to run a command that lists the files in the current directory and counts them:

import subprocess
command = "ls -l | wc -l"
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
output, error = process.communicate()
print(output.decode())

This will print the number of files in the current directory to the standard output.

The subprocess.run() is a higher-level, simplified approach for running external commands and is often preferred for its ease of use and improved readability, especially in Python 3.5 and later versions. subprocess.Popen() provides more control and flexibility, but also requires more code and error handling.