Problems with command using * wildcard in subprocess

Question

I'm trying to copy files from one location to another using subprocess library and Popen method. When runing following script I'm getting the error cp: cannot stat /some/dev_path/*. I was told that the * is not expanded to the file names and that's where the problem is. Also in some other posts people were suggesting to use call instead of Popen, but call will not return stderr as far as I know.

devPath = '/some/dev_path/'
productionPath = '/some/prod_path/'

p = subprocess.Popen(['cp', '-r', devPath + '*', productionPath], stdout = subprocess.PIPE, stderr = subprocess.PIPE)
pout, perr = p.communicate()

if perr != '':
    sys.exit('Error: ' + perr)

Beecham

Expanding '*' is a function of whichever shell you are using, except you aren't using a shell here. See docs.python.org/library/subprocess.html for the several ways of adding shell functionality. – Chafe

Why not use shutil.copytree() instead so you have something portable that doesn't require subprocesses? – Pinxit

Python has built-in supprt for this, using the shutil and glob modules. You don't really need to call a subprocess. – Trichromatic

Answer 1

Expanding the * (globbing) is a function of your shell, bash for example. Therefore you'd have to use the keyword argument shell=True in your subprocess.Popen call.

However, for this case I'd strongly suggest to use shutil.copytree instead.

(First of all, because it's much simpler (see Zen of Python) and less error-prone. Dealing with errors is much cleaner, you get nice exceptions including a list of errors (for multi file operations like yours), and you don't have to deal with spawning a subprocess and communicating with it. Second, it's an unnecessary waste of resources to fork a child process if you don't need to. Other issues include quoting / escaping and possibly introducing security vulnerabilities into your code if you fail to properly sanitize user input.)

For example:

from shutil import copytree
from shutil import Error

try:
   copytree('dir_a', 'dir_b')
except (Error, OSError), e:
    print "Attempt to copy failed: %s" % e

Also, you shouldn't build filesystem paths by concatenating strings together, but instead use os.path.join(). That will use the correct directory separator (os.sep) for the current OS and allow you to easily write portable code.

Example:

>>> import os
>>> os.path.join('/usr/lib', 'python2.7')
'/usr/lib/python2.7'

Note: os.path.join still only does (smart) string manipulation - it doesn't care if that path is accessible or even exists.

Peisch

Thanks you for your response. I wasn't aware of the shutil. Why shutil is prefered here over subprocess? – Beecham

@Beecham Updated the answer some details about why to avoid unnecessary subprocesses. – Peisch

Answer 2

I had exactly the same issue in Python 3.x. I was able to use os.system() and command.command() in python 2.x but now need to use Popen and run of subprocess.

The solution is actually really straightforward but not very well documented in my opinion.

When using shell=True pass a string.

When using shell=False pass a list.

e.g:

p = subprocess.Popen('cp -r ' + devPath + '* ' + productionPath, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell=True)
pout, perr = p.communicate()

I couldn't figure out why i was getting continual stat errors in a build system I'm writing that uses a lot of copy commands. This solves all the issues.

Fullscale