ubelt.util_cmd module

This module exposes the ubelt.cmd() command, which provides a simple means for interacting with the commandline. This uses subprocess.Popen under the hood, but improves upon existing subprocess functionality by:

(1) Adding the option to “tee” the output, i.e. simultaniously capture and write to stdout and stderr.

(2) Always specify the command as a string. The subprocess module expects the command as either a List[str] if shell=False and str if shell=True. If necessary, ubelt.util_cmd.cmd() will automatically convert from one format to the other, so passing in either case will work.

(3) Specificy if the process blocks or not by setting detach. Note: when detach is True it is not possible to tee the output.

Example

>>> import ubelt as ub
>>> # Running with verbose=1 will write to stdout in real time
>>> info = ub.cmd('echo "write your command naturally"', verbose=1)
write your command naturally
>>> # Unless `detach=True`, `cmd` always returns an info dict.
>>> print('info = ' + ub.repr2(info))
info = {
    'command': 'echo "write your command naturally"',
    'cwd': None,
    'err': '',
    'out': 'write your command naturally\n',
    'proc': <...Popen...>,
    'ret': 0,
}
ubelt.util_cmd.cmd(command, shell=False, detach=False, verbose=0, tee=None, cwd=None, env=None, tee_backend='auto', check=False, system=False, timeout=None)[source]

Executes a command in a subprocess.

The advantage of this wrapper around subprocess is that (1) you control if the subprocess prints to stdout, (2) the text written to stdout and stderr is returned for parsing, (3) cross platform behavior that lets you specify the command as a string or tuple regardless of whether or not shell=True. (4) ability to detach, return the process object and allow the process to run in the background (eventually we may return a Future object instead).

Parameters
  • command (str | List[str]) – command string, tuple of executable and args, or shell command.

  • shell (bool, default=False) – if True, process is run in shell.

  • detach (bool, default=False) – if True, process is detached and run in background.

  • verbose (int, default=0) – verbosity mode. Can be 0, 1, 2, or 3.

  • tee (bool | None) – if True, simultaneously writes to stdout while capturing output from the command. If not specified, defaults to True if verbose > 0. If detach is True, then this argument is ignored.

  • cwd (str | PathLike | None) – Path to run command. Defaults to current working directory if unspecified.

  • env (Dict[str, str] | None) – environment passed to Popen

  • tee_backend (str, default=’auto’) – backend for tee output. Valid choices are: “auto”, “select” (POSIX only), and “thread”.

  • check (bool, default=False) – if True, check that the return code was zero before returning, otherwise raise a subprocess.CalledProcessError. Does nothing if detach is True.

  • system (bool, default=False) – if True, most other considerations are dropped, and os.system() is used to execute the command in a platform dependant way. Other arguments such as env, tee, timeout, and shell are all ignored. (new in version 1.1.0)

  • timeout (float) – If the process does not complete in timeout seconds, raises a subprocess.TimeoutExpired. (new in version 1.1.0) Currently unhandled when tee is True.

  • log (Callable | None) – If specified, verbose output is written using this function, otherwise the builtin print function is used.

Returns

info - information about command status. if detach is False info contains captured standard out, standard error, and the return code if detach is True info contains a reference to the process.

Return type

dict

Raises
  • ValueError - on an invalid configuration

  • subprocess.TimeoutExpired - if the timeout limit is exceeded

  • subprocess.CalledProcessError - if check and the return value is non zero

Note

Inputs can either be text or tuple based. On UNIX we ensure conversion to text if shell=True, and to tuple if shell=False. On windows, the input is always text based. See [SO_33560364] for a potential cross-platform shlex solution for windows.

When using the tee output, the stdout and stderr may be shuffled from what they would be on the command line.

Related Work:

https://github.com/pycontribs/subprocess-tee https://github.com/mortoray/shelljob https://github.com/netinvent/command_runner https://www.pyinvoke.org/prior-art.html

References

SO_11495783

https://stackoverflow.com/questions/11495783/redirect-subprocess-stderr-to-stdout

SO_7729336

https://stackoverflow.com/questions/7729336/how-can-i-print-and-display-subprocess-stdout-and-stderr-output-without-distorti

SO_33560364

https://stackoverflow.com/questions/33560364/python-windows-parsing-command-lines-with-shlex

CommandLine

xdoctest -m ubelt.util_cmd cmd:6
python -c "import ubelt as ub; ub.cmd('ping localhost -c 2', verbose=2)"
pytest "$(python -c 'import ubelt; print(ubelt.util_cmd.__file__)')" -sv --xdoctest-verbose 2

Example

>>> import ubelt as ub
>>> info = ub.cmd(('echo', 'simple cmdline interface'), verbose=1)
simple cmdline interface
>>> assert info['ret'] == 0
>>> assert info['out'].strip() == 'simple cmdline interface'
>>> assert info['err'].strip() == ''

Example

>>> import ubelt as ub
>>> info = ub.cmd('echo str noshell', verbose=0)
>>> assert info['out'].strip() == 'str noshell'

Example

>>> # windows echo will output extra single quotes
>>> import ubelt as ub
>>> info = ub.cmd(('echo', 'tuple noshell'), verbose=0)
>>> assert info['out'].strip().strip("'") == 'tuple noshell'

Example

>>> # Note this command is formatted to work on win32 and unix
>>> import ubelt as ub
>>> info = ub.cmd('echo str&&echo shell', verbose=0, shell=True)
>>> assert info['out'].strip() == 'str' + chr(10) + 'shell'

Example

>>> import ubelt as ub
>>> info = ub.cmd(('echo', 'tuple shell'), verbose=0, shell=True)
>>> assert info['out'].strip().strip("'") == 'tuple shell'

Example

>>> import pytest
>>> import ubelt as ub
>>> info = ub.cmd('echo hi', check=True)
>>> import subprocess
>>> with pytest.raises(subprocess.CalledProcessError):
>>>     ub.cmd('exit 1', check=True, shell=True)

Example

>>> import ubelt as ub
>>> from os.path import join, exists
>>> fpath1 = join(ub.get_app_cache_dir('ubelt'), 'cmdout1.txt')
>>> fpath2 = join(ub.get_app_cache_dir('ubelt'), 'cmdout2.txt')
>>> ub.delete(fpath1)
>>> ub.delete(fpath2)
>>> # Start up two processes that run simultaneously in the background
>>> info1 = ub.cmd(('touch', fpath1), detach=True)
>>> info2 = ub.cmd('echo writing2 > ' + fpath2, shell=True, detach=True)
>>> # Detached processes are running in the background
>>> # We can run other code while we wait for them.
>>> while not exists(fpath1):
...     pass
>>> while not exists(fpath2):
...     pass
>>> # communicate with the process before you finish
>>> # (otherwise you may leak a text wrapper)
>>> info1['proc'].communicate()
>>> info2['proc'].communicate()
>>> # Check that the process actually did finish
>>> assert (info1['proc'].wait()) == 0
>>> assert (info2['proc'].wait()) == 0
>>> # Check that the process did what we expect
>>> assert ub.readfrom(fpath1) == ''
>>> assert ub.readfrom(fpath2).strip() == 'writing2'

Example

>>> # Can also use ub.cmd to call os.system
>>> import pytest
>>> import ubelt as ub
>>> import subprocess
>>> info = ub.cmd('echo hi', check=True, system=True)
>>> with pytest.raises(subprocess.CalledProcessError):
>>>     ub.cmd('exit 1', check=True, shell=True)