Python module to shellquote/unshellquote? [duplicate]

Question

Is there anything in the Python standard library that will properly parse/unparse strings for using in shell commands? I'm looking for the python analog to perl's String::ShellQuote::shell_quote:

$ print String::ShellQuote::shell_quote("hello", "stack", "overflow's", "quite", "cool")
hello stack 'overflow'\''s' quite cool

And, even more importantly, something which will work in the reverse direction (take a string and decompose it into a list).

Freehearted

Answer 1

pipes.quote is now shlex.quote in python 3. It is easy enough to use that piece of code.

https://github.com/python/cpython/blob/master/Lib/shlex.py#L281

That version handles zero-length argument correctly.

Changteh

Answer 2

Looks like

try:  # py3
    from shlex import quote
except ImportError:  # py2
    from pipes import quote

quote("hello stack overflow's quite cool")
>>> '"hello stack overflow\'s quite cool"'

gets me far enough.

Freehearted

Sure, subprocess is swell for actually starting processes in python, but for code generation, this was just what I needed! +1 – Polecat

In Python 3, this is shlex.quote. – Cod

On windows, subprocess.list2cmdline is more accurate. pipes.quote always uses single quotes, which is not acceptable in Windows command line environment. – Wandy

@Cod but pipes isn't deprecated in python3 – Liguria

"Deprecated since version 2.7" "It is finally exposed publicly in Python 3.3 as the quote function in the shlex module." – Strutting

Answer 3

To unquote, try shlex.split()

Sport

Answer 4

I'm pretty sure that pipes.quote is broken, and should not be used, because it does not handle zero-length arguments correctly:

>>> from pipes import quote
>>> args = ['arg1', '', 'arg3']
>>> print 'mycommand %s' % (' '.join(quote(arg) for arg in args))
mycommand arg1  arg3

I believe the result should be something like

mycommand arg1 '' arg3

Shulman

fair enough. but then we need a better solution :-) – Freehearted

print 'mycommand %s' % (' '.join(quote(arg) or "''" for arg in args)) ? – Calcutta

At John's initiative, this was fixed in Python 2.6. – Asco

... but the use of join defeats most of the purpose of using quote here! – Stentorian

Why do you say that? – Shulman

Answer 5

For shell quoting, this works: I've rigorously tested it on Posix. [I'm assuming that the list2cmdline function supplied by Python works as advertised on Windows]

# shell.py
import os
if os.name == 'nt':
    from subprocess import list2cmdline

    def quote(arg):
        return list2cmdline([arg])[0]
else:
    import re
    _quote_pos = re.compile('(?=[^-0-9a-zA-Z_./\n])')

    def quote(arg):
        r"""
        >>> quote('\t')
        '\\\t'
        >>> quote('foo bar')
        'foo\\ bar'
        """
        # This is the logic emacs uses
        if arg:
            return _quote_pos.sub('\\\\', arg).replace('\n',"'\n'")
        else:
            return "''"

    def list2cmdline(args):
        return ' '.join([ quote(a) for a in args ])

The tests are here, if anyone cares.

Solidarity

Counter example: string to be quoted contains "\xC3\xA9", which is an é in UTF-8, and thus not uncommon in filenames. Code above puts backslashes in front of both characters, which is incorrect. pipes.quote will put it in single quotes. – Hord

Answer 6

The standard library module subprocess has the list2cmdline function which does this, albeit according to Microsoft rules so I am not sure how reliable it works in Unix-like environments for more complicated command lines.

Isar

Answer 7

The quotefunction is available for quite some time (Python 2.7?) -- the major drawback is it moved from pipe module to shlex between 3.2 and 3.3.

You have to be prepared to handle both cases while importing that function:

try:
    from shlex import quote
except ImportError:
    from pipes import quote

Stentorian

Answer 8

You should never have to shell quote. The correct way to do a command is to not do shell quoting and instead use subprocess.call or subprocess.Popen, and pass a list of unquoted arguments. This is immune to shell expansion.

i.e.

subprocess.Popen(['echo', '"', '$foo'], shell=False)

If you want to unquote shell quoted data, you can use shlex.shlex like this:

list(shlex.shlex("hello stack 'overflow'\''s' quite cool"))

Ragucci

What if I need to pass a command (that requires escaping) for ssh to execute once it reaches the other side? – Monoculture

This is not a helpful answer (well it answers one half my question, so it's half helpful...). There are are any number of occasions when you need to shell quote -- Mike Boers gives just one great example (in fact, that's the one I'm running into) – Freehearted

actually even worse, the given example breaks: (Pdb) list(shlex.shlex("hello stack 'overflow'\''s' quite cool")) *** Error in argument: '(shlex.shlex("hello stack \'overflow\'\\\'\'s\' quite cool"))' – Freehearted

From subprocess documentation: If shell is True, it is recommended to pass args as a string rather than as a sequence. ... This includes, for example, quoting or backslash escaping filenames with spaces in them. - That's just another example where you need quoting. – Swaggering

This answer is not good, you may be generating a shell script to run later and not executing the command directly. – Gunfight

My use case for shell quoting is: my Python program is using subprocess.Popen() as suggested, but when it does I'd like it to print something to the console which can be directly cut-and-pasted into a shell, for manual debugging should the command fail. – C