python -c vs python -<< heredoc

Question

I am trying to run some piece of Python code in a Bash script, so i wanted to understand what is the difference between:

#!/bin/bash
#your bash code

python -c "
#your py code
"

vs

python - <<DOC
#your py code
DOC

I checked the web but couldn't compile the bits around the topic. Do you think one is better over the other? If you wanted to return a value from Python code block to your Bash script then is a heredoc the only way?

Upholstery

In your first example, you'll need to escape the double quotes if you want to use strings with double double quotes. – Psychometrics

Ok Evert, this is one aspect. What are the others to prefer one over the other? – Upholstery

This really is a question about here docs and not about Python – Sidonie

Answer 1

The main flaw of using a here document is that the script's standard input will be the here document. So if you have a script which wants to process its standard input, python -c is pretty much your only option.

On the other hand, using python -c '...' ties up the single-quote for the shell's needs, so you can only use double-quoted strings in your Python script; using double-quotes instead to protect the script from the shell introduces additional problems (strings in double-quotes undergo various substitutions, whereas single-quoted strings are literal in the shell).

As an aside, notice that you probably want to single-quote the here-doc delimiter, too, otherwise the Python script is subject to similar substitutions.

python - <<'____HERE'
print("""Look, we can have double quotes!""")
print('And single quotes! And `back ticks`!')
print("$(and what looks to the shell like command substitutions and $variables!)")
____HERE

As an alternative, escaping the delimiter works identically, if you prefer that (python - <<\____HERE)

Talmudist

I don't understand your comment about returning a value to the shell. Both constructs print to standard output and return an exit code, both of which can be examined by the shell if you want to. – Talmudist

Agreed. You need to read shell for the output that python printed to the standard output. e.g. #!/bin/bash function current_datetime { python - <<'__COB' import datetime print datetime.datetime.now() __COB } dtm=$(current_datetime) echo Current date and time: $dtm – Upholstery

Though the date command is probably a better fit for that particular use case. Also, again, quote your strings. – Talmudist

You can use '\''to "embed" a single quote "inside" a single quoted string in a shell. E.g. echo 'Say '\''Hello'\'', Fred'. – Subtilize

Yeah, that's a closing single quote, an unquoted but backslash-escaped literal single quote, and an opening single quote for the remainder of the single-quoted string. This is probably too complex and unreadable for routine use, but if you have just one or two of these, it's workable. Synonymously, you can double-quote the lone literal single quote - '"'"'; I like to call this "seesaw quoting". – Talmudist

Good answer! But one can use inputs while using a here-doc, see https://mcmap.net/q/462037/-python-c-vs-python-lt-lt-heredoc – Recombination

Answer 2

If you are using bash, you can avoid heredoc problems if you apply a little bit more of boilerplate:

python <(cat <<EoF

name = input()
print(f'hello, {name}!')

EoF
)

This will let you run your embedded Python script without you giving up the standard input. The overhead is mostly the same of using cmda | cmdb. This technique is known as Process Substitution.

If want to be able to somehow validate the script, I suggest that you dump it to a temporary file:

#!/bin/bash

temp_file=$(mktemp my_generated_python_script.XXXXXX.py)

cat > $temp_file <<EoF
# embedded python script
EoF

python3 $temp_file && rm $temp_file

This will keep the script if it fails to run.

Eolithic

+1 but you should remove the temporary file when you are done. See also #687514 – Talmudist

In the case above, the file is removed if the script works correctly. Using trap would require a bit of control boilerplate that I see as unnecessary complexity for most of the cases. – Eolithic

Yeah, for a temporary hack that's fine. Just pointing out the not entirely obvious if somebody wants to do this in a production script. – Talmudist

well, I do this in production. The extra thing I use is a find in cron setting some limits on the age of the files. When I apply this technique, the script is an import and a function call. I think that the real issue is that this technique should be a palliative for when you have a simple business logic that you need in a shell script and don't have the time or money to improve the python module. Anything different from that, like processing user input or exposing actual functionality, is very risky from a security perspective. Side-effects is a no-no, IMHO; you should improve the module. – Eolithic

Answer 3

If you prefer to use python -c '...' without having to escape with the double-quotes you can first load the code in a bash variable using here-documents:

read -r -d '' CMD << '--END'
print ("'quoted'")
--END
python -c "$CMD"

The python code is loaded verbatim into the CMD variable and there's no need to escape double quotes.

Ezra

Answer 4

How to use here-docs with input

tripleee's answer has all the details, but there's Unix tricks to work around this limitation:

So if you have a script which wants to process its standard input, python -c is pretty much your only option.

This trick applies to all programs that want to read from a redirected stdin (e.g., ./script.py < myinputs) and also take user input:

python - <<'____HERE'
import os

os.dup2(1, 0)
print(input("--> "))
____HERE

Running this works:

$ bash heredocpy.sh
--> Hello World!
Hello World!

If you want to get the original stdin, run os.dup(0) first. Here is a real-world example.

---

This works because as long as either stdout or stderr are a tty, one can read from them as well as write to them. (Otherwise, you could just open /dev/tty. This is what less does.)

In case you want to process inputs from a file instead, that's possible too -- you just have to use a new fd:

Example with a file

cat <<'____HERE' > file.txt
With software there are only two possibilites:
either the users control the programme
or the programme controls the users.
____HERE

python - <<'____HERE' 4< file.txt
import os

for line in os.fdopen(4):
  print(line.rstrip().upper())
____HERE

Example with a command

Unfortunately, pipelines don't work here -- but process substitution does:

python - <<'____HERE' 4< <(fortune)
import os

for line in os.fdopen(4):
  print(line.rstrip().upper())
____HERE

Recombination