Subprocess changing directory

Question

I want to execute a script inside a subdirectory/superdirectory (I need to be inside this sub/super-directory first). I can't get subprocess to enter my subdirectory:

tducin@localhost:~/Projekty/tests/ve$ python
Python 2.7.4 (default, Sep 26 2013, 03:20:26) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> import os
>>> os.getcwd()
'/home/tducin/Projekty/tests/ve'
>>> subprocess.call(['cd ..'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/subprocess.py", line 524, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1308, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Python throws OSError and I don't know why. It doesn't matter whether I try to go into an existing subdir or go one directory up (as above) - I always end up with the same error.

Passmore

What happens if use os.chdir() instead. – Obolus

Answer 1

What your code tries to do is call a program named cd ... What you want is call a command named cd.

But cd is a shell internal. So you can only call it as

subprocess.call('cd ..', shell=True) # pointless code! See text below.

But it is pointless to do so. As no process can change another process's working directory (again, at least on a UNIX-like OS, but as well on Windows), this call will have the subshell change its dir and exit immediately.

What you want can be achieved with os.chdir() or with the subprocess named parameter cwd which changes the working directory immediately before executing a subprocess.

For example, to execute ls in the root directory, you either can do

wd = os.getcwd()
os.chdir("/")
subprocess.Popen("ls")
os.chdir(wd)

or simply

subprocess.Popen("ls", cwd="/")

Under

cd usually also exists as a binary, not only a shell built-in. The real problem of the OP was that he was calling a binary cd .., yes. (And your third paragraph would have been his next problem, so good answer.) – Polypus

@LeonWeber How should cd be able to work as a binary? It cannot chante its parent's working dir. – Under

@LeonWeber Note that the OP uses Linux. On Windows, things may be different. – Under

I was talking about Linux. Good point though. I was wondering myself, and here’s the answer: /usr/bin/cd consists of builtin cd "$@" — so it just calls the shell built-in cd as well. – Polypus

@LeonWeber I don't have such a file, and I think your version of the file doesn't have a shebang line, so that it is just sourced from the shell and not executed in a subprocess. I'm not sure if it will be callable from another program... I'll test that. – Under

@LeonWeber Called from the shell, it doesn't do anything here, and calling it from Python leads to a OSError: [Errno 8] Exec format error. – Under

it has a shebang line, I left that off to save space in the comment. However, I’m not sure what use the executable has, because calling the executable from my shell will fork off another shell with a different pwd, and when that one exits, nothing has changed in my original shell of course. So yeah, your answer is definitely the right way to do it. – Polypus

Note that, in my answer, os.chdir is done by the Popen. So, it is still achievend (sic) the same way in the background. – Salvatore

@Salvatore Your answer applies if there is another subprocess supposed to be started inside the changed subdir. In this case it is the way to go, but I don't see this is the case. – Under

@glglgl: If no process can change another process's working directory, then why do we accomplish changing directory in CMD(or any shell) without any problem? By the way, what did you mean by subshell here? Thanks – Apure

@Apure That's why cd must be implemented as internal shell command. There's no other way to do it. Internal shell commands are executed within the same process as the shell. What I meant by subshell is the shell executed for shell=True. It gets the command to be executed, executes that and exits. – Under

I think an example or two of your suggested approach would be useful. – Yonah

Answer 2

To run your_command as a subprocess in a different directory, pass cwd parameter, as suggested in @wim's answer:

import subprocess

subprocess.check_call(['your_command', 'arg 1', 'arg 2'], cwd=working_dir)

A child process can't change its parent's working directory (normally). Running cd .. in a child shell process using subprocess won't change your parent Python script's working directory i.e., the code example in @glglgl's answer is wrong. cd is a shell builtin (not a separate executable), it can change the directory only in the same process.

Depreciate

Guess why I wrote "But it is pointless to do so."? – Under

@Under I guess that is why I wrote "code example .. is wrong" and not the "answer" itself ;) – Depreciate

Maybe, it's been a long time since then. But as I already wrote back then, the code isn't wrong, it just doesn't do what might expect of it. – Under

Answer 3

subprocess.call and other methods in the subprocess module have a cwd parameter.

This parameter determines the working directory where you want to execute your process.

So you can do something like this:

subprocess.call('ls', shell=True, cwd='path/to/wanted/dir/')

Check out docs subprocess.popen-constructor

Galantine

Answer 4

You want to use an absolute path to the executable, and use the cwd kwarg of Popen to set the working directory. See the docs.

If cwd is not None, the child’s current directory will be changed to cwd before it is executed. Note that this directory is not considered when searching the executable, so you can’t specify the program’s path relative to cwd.

Salvatore

It depends on if another subprocess is supposed to be executed. If so, your way is the right one. But for only having the own program acting inside a different directory, that won't help. – Under

What do you mean it won't help? This is the one obvious way to do it. – Salvatore

No, as it just changes the cwd of the process I am going to launch, such as subprocess.call(['ls', '-l'], cwd='/'). This changes the cwd to / and then runs ls with -l as argument. But if I want to do os.chdir('/') and then open('etc/fstab', 'r'), I cannot replace os.chdir() with anything about subprocess.XXX(cwd='/') as it won't help, as said. These are two complete different scenarios. – Under

That's why my answer says to use an absolute path to the executable, did you miss that part? – Salvatore

No, I didn't. I think I give up. If I want to change the current working directory and open a file, I have no executable. It is a completely different situation. BTW: There is no need to use an absolute path if I use cwd= as intended. I can as well do subprocess.call(['bin/ls', '-l'], cwd='/'). – Under

OK, I'm not sure why you are talking about opening a file in the first place. We can see the Popen in the traceback of the question. – Salvatore

That is because the OP wanted to use it for doing a cd in order to thange their own working directory, not for doing something completely different. That was what it was all about in the first place. And if they use subprocess only for executing cd, a cwd= won't help here as well. – Under

Oh, I see now. By the way your comment above seems to contradict the Popen docs, so I created a question about that here – Salvatore

There is generally no reason that the full path is required. Sometimes, if you have knowledge of the binary's precise location, but can't control the invoking user's PATH, it makes sense to spell out the path to the executable you want; but in the normal case, simply trust the PATH. – Smattering

@Smattering Yes, fair enough. The main point of my answer in 2014 is that this OP really wanted the cwd keyword to Popen. jfs answer came two years later, and the accepted answer incorrectly stated that what you want can only achievend with os.chdir() until an edit in 2016.. – Salvatore

Answer 5

I guess these days you would do:

import subprocess

subprocess.run(["pwd"], cwd="sub-dir")

Ivoryivorywhite

Would this work with shell=False, no matter what the OS is? – Luffa

Answer 6

Another option based on this answer: https://mcmap.net/q/137884/-running-multiple-bash-commands-with-subprocess

This allows you to execute multiple commands (e.g cd) in the same process.

import subprocess

commands = '''
pwd
cd some-directory
pwd
cd another-directory
pwd
'''

process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = process.communicate(commands.encode('utf-8'))
print(out.decode('utf-8'))

Bilabiate

This is just a roundabout and inefficient way to do shell=True, executable='/bin/bash' – Smattering

Answer 7

just use os.chdir
Example:

>>> import os
>>> import subprocess
>>> # Lets Just Say WE want To List The User Folders
>>> os.chdir("/home/")
>>> subprocess.run("ls")
user1 user2 user3 user4

Banna

Answer 8

If you want to have cd functionality (assuming shell=True) and still want to change the directory in terms of the Python script, this code will allow 'cd' commands to work.

import subprocess
import os

def cd(cmd):
    #cmd is expected to be something like "cd [place]"
    cmd = cmd + " && pwd" # add the pwd command to run after, this will get our directory after running cd
    p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) # run our new command
    out = p.stdout.read()
    err = p.stderr.read()
    # read our output
    if out != "":
        print(out)
        os.chdir(out[0:len(out) - 1]) # if we did get a directory, go to there while ignoring the newline 
    if err != "":
        print(err) # if that directory doesn't exist, bash/sh/whatever env will complain for us, so we can just use that
    return

Riesling

if you are expecting user input shell=True is VERY unsafe – Banna

Answer 9

If you need to change directory, run a command and get the std output as well:

import os
import logging as log
from subprocess import check_output, CalledProcessError, STDOUT
log.basicConfig(level=log.DEBUG)

def cmd_std_output(cd_dir_path, cmd):
    cmd_to_list = cmd.split(" ")
    try:
        if cd_dir_path:
            os.chdir(os.path.abspath(cd_dir_path))
        output = check_output(cmd_to_list, stderr=STDOUT).decode()
        return output
    except CalledProcessError as e:
        log.error('e: {}'.format(e))

def get_last_commit_cc_cluster():
    cd_dir_path = "/repos/cc_manager/cc_cluster"
    cmd = "git log --name-status HEAD^..HEAD --date=iso"
    result = cmd_std_output(cd_dir_path, cmd)
    return result

log.debug("Output: {}".format(get_last_commit_cc_cluster()))

Output: "commit 3b3daaaaaaaa2bb0fc4f1953af149fa3921e\nAuthor: user1<[email protected]>\nDate:   2020-04-23 09:58:49 +0200\n\n

Vitalize

You are reinventing check_call, poorly. – Smattering