Python subprocess/Popen with a modified environment

Question

I believe that running an external command with a slightly modified environment is a very common case. That's how I tend to do it:

import subprocess, os
my_env = os.environ
my_env["PATH"] = "/usr/sbin:/sbin:" + my_env["PATH"]
subprocess.Popen(my_command, env=my_env)

I've got a gut feeling that there's a better way; does it look alright?

Glazier

Also prefer to use os.pathsep instead of ":" for paths that work across platforms. See #1499519 – Semitropical

@phaedrus I'm not sure it's very relevant when he's using paths like /usr/sbin :-) – Slant

Answer 1

I think os.environ.copy() is better if you don't intend to modify the os.environ for the current process:

import subprocess, os
my_env = os.environ.copy()
my_env["PATH"] = f"/usr/sbin:/sbin:{my_env['PATH']}"
subprocess.Popen(my_command, env=my_env)

Masseuse

>>> env = os.environ.copy >>> env['foo'] = 'bar' Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'instancemethod' object does not support item assignment – Excoriation

@Excoriation You are assigning the actual method os.environ.copy to the env variable but you need to assign the result of calling the method os.environ.copy() to env. – Keiko

The environment variable resolution only actually works if you use shell=True in your subprocess.Popen invocation. Note that there are potentially security implications of doing that. – Steffi

Inside subprocess.Popen(my_command, env=my_env) -- what is "my_command" – Paterson

@Paterson - my_command is just command to run. It could be for instance /path/to/your/own/program or any other "executable" statement. – Weisman

This also worked great for me using subprocess.check_output() without shell=True, in case that's helpful for someone. env = os.environ.copy(); env["STDERRLOG"] = my_stderrlog; result = subprocess.check_output(['./my_script.sh', "-my_arg"]); print(result) – Pecker

Answer 2

That depends on what the issue is. If it's to clone and modify the environment one solution could be:

subprocess.Popen(my_command, env=dict(os.environ, PATH="path"))

But that somewhat depends on that the replaced variables are valid python identifiers, which they most often are (how often do you run into environment variable names that are not alphanumeric+underscore or variables that starts with a number?).

Otherwise you'll could write something like:

subprocess.Popen(my_command, env=dict(os.environ, 
                                      **{"Not valid python name":"value"}))

In the very odd case (how often do you use control codes or non-ascii characters in environment variable names?) that the keys of the environment are bytes you can't (on python3) even use that construct.

As you can see the techniques (especially the first) used here benefits on the keys of the environment normally is valid python identifiers, and also known in advance (at coding time), the second approach has issues. In cases where that isn't the case you should probably look for another approach.

Reseat

upvote. I didn't knew that you could write dict(mapping, **kwargs). I thought it was either or. Note: it copies os.environ without modifying it as @Daniel Burke suggested in the currently accepted answer but your answer is more succinct. In Python 3.5+ you could even do dict(**{'x': 1}, y=2, **{'z': 3}). See pep 448. – Resurrectionist

This answer explains some better ways (and why this way is not so great) to merge two dictionaries into one new one: https://mcmap.net/q/36055/-how-do-i-merge-two-dictionaries-in-a-single-expression-in-python – Galvani

@krupan: what disadvantage do you see for this specific use-case? (merging arbitrary dicts and copy/update the environ are different tasks). – Resurrectionist

@Galvani First of all the normal case is that environment variables would be valid python identifiers, which means the first construct. For that case none of your objections holds. For the second case your main objection still fails: the point about non-string keys are not applicable in this case as the keys are basically required to be strings in the environment anyway. – Reseat

@J.F.Sebastian You are correct that for this specific case this technique is fine and I should have explained myself better. My apologies. I just wanted to help those (such as myself) who might have been tempted to take this technique and apply it to the general case of merging two arbitrary dictionaries (which has some gotcha's, as the answer I linked to pointed out). – Galvani

@krupan: if you can read Russian; you might find this question useful: Почему нельзя просто взять и сложить два словаря? (or just look at the code examples and follow the corresponding links) – Resurrectionist

@Galvani A acknowledge your concern and for that reason I've updated my answer to be more clear that this approach assumes that the keys are valid python identifiers. – Reseat

Or {**os.environ, "PATH": "path"} – Fergus

Environment variable names don't have to be valid Python identifier. Environment variables can contain a much wider chargers than what's acceptable for Python identifiers, including spaces. That said, there are many tools that may make it rather harder if you need to work with env variables that contain any characters other than alphanumeric and underscore, so it's a good practice to limit to that, but the standard allows nearly any characters except null and = to be used in Env variable names. – Provolone

Answer 3

With Python 3.5 you could do it this way:

import os
import subprocess

my_env = {**os.environ, 'PATH': '/usr/sbin:/sbin:' + os.environ['PATH']}

subprocess.Popen(my_command, env=my_env)

Here we end up with a copy of os.environ and overridden PATH value.

It was made possible by PEP 448 (Additional Unpacking Generalizations).

Another example. If you have a default environment (i.e. os.environ), and a dict you want to override defaults with, you can express it like this:

my_env = {**os.environ, **dict_with_env_variables}

Hilde

Answer 4

you might use my_env.get("PATH", '') instead of my_env["PATH"] in case PATH somehow not defined in the original environment, but other than that it looks fine.

Lithology

Answer 5

To temporarily set an environment variable without having to copy the os.envrion object etc, I do this:

process = subprocess.Popen(['env', 'RSYNC_PASSWORD=foobar', 'rsync', \
'rsync://[email protected]::'], stdout=subprocess.PIPE)

Embellish

Answer 6

The env parameter accepts a dictionary. You can simply take os.environ, add a key (your desired variable) (to a copy of the dict if you must) to that and use it as a parameter to Popen.

Uranic

This is the simplest answer if you just want to add a new environment variable. os.environ['SOMEVAR'] = 'SOMEVAL' – Redden

You will get into troubles with a normal dictionary, always use same class of os.environ – Audit

Answer 7

I know this has been answered for some time, but there are some points that some may want to know about using PYTHONPATH instead of PATH in their environment variable. I have outlined an explanation of running python scripts with cronjobs that deals with the modified environment in a different way (found here). Thought it would be of some good for those who, like me, needed just a bit more than this answer provided.

Sikang

Answer 8

In newer versions of Python, you can use the env keyword argument of the run function, like this:

import subprocess
import os

subprocess.run(["mycommand"], env=dict(os.environ) | {"FOO": "bar"})

Because I wanted the process to inherit the existing environment variables, I merged dict(os.environ) with my own dictionary of overridden environment variables using the union operator, which works in Python 3.9+.

Shiest

Answer 9

In certain circumstances you may want to only pass down the environment variables your subprocess needs, but I think you've got the right idea in general (that's how I do it too).

Purree

Answer 10

This could be a solution :

new_env = dict([(k,(':'.join([env[k], v]) if k in env else v)) for k,v in os.environ.items()])

Ressieressler