ubelt.util_cmd module

This module exposes the ubelt.cmd() command, which provides a simple means for interacting with the command line. This uses subprocess.Popen under the hood, but improves upon existing subprocess functionality by:

(1) Adding the option to “tee” the output, i.e. simultaneously capture and write to stdout and stderr.

(2) Always specify the command as a string. The subprocess module expects the command as either a List[str] if shell=False and str if shell=True. If necessary, ubelt.util_cmd.cmd() will automatically convert from one format to the other, so passing in either case will work.

(3) Specify if the process blocks or not by setting detach. Note: when detach is True it is not possible to tee the output.

Example

>>> import ubelt as ub
>>> # Running with verbose=1 will write to stdout in real time
>>> info = ub.cmd('echo "write your command naturally"', verbose=1)
write your command naturally
>>> # The return type is a dictionary of information depending
>>> # on how `ub.cmd` was invoked.
>>> print('info = ' + ub.repr2(info))
info = {
    'command': 'echo "write your command naturally"',
    'cwd': None,
    'err': '',
    'out': 'write your command naturally\n',
    'proc': <...Popen...>,
    'ret': 0,
}

The cmd is able to handle common uses cases of the subprocess module with a simpler interface.

import subprocess
import ubelt as ub

Run without capturing output and without printing to the screen

# stdlib
subprocess.run(['ls', '-l'], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, universal_newlines=True)

# ubelt equivalent
ub.cmd(['ls', '-l'], capture=False)

Print output to the screen, but no programmatic access to the data

# stdlib
subprocess.check_call(['ls', '-l'])

# ubelt equivalent
ub.cmd(['ls', '-l'], verbose=1, capture=False)

Get programmatic access to the data but don’t show it on screen

# stdlib
subprocess.check_output(['ls', '-l'], universal_newlines=True)

# ubelt equivalent
ub.cmd(['ls', '-l'])['out']

Get programatic access AND show it on screen

# stdlib has no easy way to to this

# ubelt has "tee" functionality
ub.cmd(['ls', '-l'], verbose=1)
ubelt.util_cmd.cmd(command, shell=False, detach=False, verbose=0, tee=None, cwd=None, env=None, tee_backend='auto', check=False, system=False, timeout=None, capture=True)[source]

Executes a command in a subprocess.

The advantage of this wrapper around subprocess is that (1) you control if the subprocess prints to stdout, (2) the text written to stdout and stderr is returned for parsing, (3) cross platform behavior that lets you specify the command as a string or tuple regardless of whether or not shell=True. (4) ability to detach, return the process object and allow the process to run in the background (eventually we may return a Future object instead).

Parameters:

  • command (str | List[str]) – command string, tuple of executable and args, or shell command.

  • shell (bool) – if True, process is run in shell. Defaults to False.

  • detach (bool) – if True, process is detached and run in background. Defaults to False.

  • verbose (int) – verbosity mode. Can be 0, 1, 2, or 3. Defaults to 0.

  • tee (bool | None) – if True, simultaneously writes to stdout while capturing output from the command. If not specified, defaults to True if verbose > 0. If detach is True, then this argument is ignored.

  • cwd (str | PathLike | None) – Path to run command. Defaults to current working directory if unspecified.

  • env (Dict[str, str] | None) – environment passed to Popen

  • tee_backend (str) – backend for tee output. Valid choices are: “auto”, “select” (POSIX only), and “thread”. Defaults to “auto”.

  • check (bool) – if True, check that the return code was zero before returning, otherwise raise a subprocess.CalledProcessError. Does nothing if detach is True. Defaults to False.

  • system (bool) – if True, most other considerations are dropped, and os.system() is used to execute the command in a platform dependant way. Other arguments such as env, tee, timeout, and shell are all ignored. Defaults to False. (New in version 1.1.0)

  • timeout (float | None) – If the process does not complete in timeout seconds, raise a subprocess.TimeoutExpired. (New in version 1.1.0).

  • capture (bool) – if True, the stdout/stderr are captured and returned in the information dictionary. Ignored if detatch or system is True.

Returns:

info - information about command status. if detach is False info contains captured standard out, standard error, and the return code if detach is True info contains a reference to the process.

Return type:

dict | CmdOutput

Raises:

  • ValueError - on an invalid configuration –

  • subprocess.TimeoutExpired - if the timeout limit is exceeded –

  • subprocess.CalledProcessError - if check and the return value is non zero –

Note

When using the tee output, the stdout and stderr may be shuffled from what they would be on the command line.

Related Work:

Similar to other libraries: [SubprocTee], [ShellJob], [CmdRunner], [PyInvoke].

References

CommandLine

xdoctest -m ubelt.util_cmd cmd:6
python -c "import ubelt as ub; ub.cmd('ping localhost -c 2', verbose=2)"
pytest "$(python -c 'import ubelt; print(ubelt.util_cmd.__file__)')" -sv --xdoctest-verbose 2

Example

>>> import ubelt as ub
>>> info = ub.cmd(('echo', 'simple cmdline interface'), verbose=1)
simple cmdline interface
>>> assert info['ret'] == 0
>>> assert info['out'].strip() == 'simple cmdline interface'
>>> assert info['err'].strip() == ''

Example

>>> import ubelt as ub
>>> info = ub.cmd('echo str noshell', verbose=0)
>>> assert info['out'].strip() == 'str noshell'

Example

>>> # windows echo will output extra single quotes
>>> import ubelt as ub
>>> info = ub.cmd(('echo', 'tuple noshell'), verbose=0)
>>> assert info['out'].strip().strip("'") == 'tuple noshell'

Example

>>> # Note this command is formatted to work on win32 and unix
>>> import ubelt as ub
>>> info = ub.cmd('echo str&&echo shell', verbose=0, shell=True)
>>> assert info['out'].strip() == 'str' + chr(10) + 'shell'

Example

>>> import ubelt as ub
>>> info = ub.cmd(('echo', 'tuple shell'), verbose=0, shell=True)
>>> assert info['out'].strip().strip("'") == 'tuple shell'

Example

>>> import pytest
>>> import ubelt as ub
>>> info = ub.cmd('echo hi', check=True)
>>> import subprocess
>>> with pytest.raises(subprocess.CalledProcessError):
>>>     ub.cmd('exit 1', check=True, shell=True)

Example

>>> import ubelt as ub
>>> from os.path import join, exists
>>> dpath = ub.Path.appdir('ubelt', 'test').ensuredir()
>>> fpath1 = (dpath / 'cmdout1.txt').delete()
>>> fpath2 = (dpath / 'cmdout2.txt').delete()
>>> # Start up two processes that run simultaneously in the background
>>> info1 = ub.cmd(('touch', str(fpath1)), detach=True)
>>> info2 = ub.cmd('echo writing2 > ' + str(fpath2), shell=True, detach=True)
>>> # Detached processes are running in the background
>>> # We can run other code while we wait for them.
>>> while not exists(fpath1):
...     pass
>>> while not exists(fpath2):
...     pass
>>> # communicate with the process before you finish
>>> # (otherwise you may leak a text wrapper)
>>> info1['proc'].communicate()
>>> info2['proc'].communicate()
>>> # Check that the process actually did finish
>>> assert (info1['proc'].wait()) == 0
>>> assert (info2['proc'].wait()) == 0
>>> # Check that the process did what we expect
>>> assert fpath1.read_text() == ''
>>> assert fpath2.read_text().strip() == 'writing2'

Example

>>> # Can also use ub.cmd to call os.system
>>> import pytest
>>> import ubelt as ub
>>> import subprocess
>>> info = ub.cmd('echo hi', check=True, system=True)
>>> with pytest.raises(subprocess.CalledProcessError):
>>>     ub.cmd('exit 1', check=True, shell=True)