Embed bash in python

Question

I am writting a Python script and I am running out of time. I need to do some things that I know pretty well in bash, so I just wonder how can I embed some bash lines into a Python script.

Thanks

Cruzcruzado

-1: "Some things"? If you had particular things you wanted help with, we could have helped you avoid the mistake of embedding bash in Python. – Comeon

@Comeon I know this was a long time ago, but I just took my first dive into bash scripting. I've determined that I definitely want to use Python instead whenever possible (which is my primary language ATM). Is there a quick way to explain why embedding bash in Python is a mistake? – Raines

Answer 1

If you want to call system commands, use the subprocess module.

Hussar

Answer 2

The ideal way to do it:

def run_script(script, stdin=None):
    """Returns (stdout, stderr), raises error on non-zero return code"""
    import subprocess
    # Note: by using a list here (['bash', ...]) you avoid quoting issues, as the 
    # arguments are passed in exactly this order (spaces, quotes, and newlines won't
    # cause problems):
    proc = subprocess.Popen(['bash', '-c', script],
        stdout=subprocess.PIPE, stderr=subprocess.PIPE,
        stdin=subprocess.PIPE)
    stdout, stderr = proc.communicate()
    if proc.returncode:
        raise ScriptException(proc.returncode, stdout, stderr, script)
    return stdout, stderr

class ScriptException(Exception):
    def __init__(self, returncode, stdout, stderr, script):
        self.returncode = returncode
        self.stdout = stdout
        self.stderr = stderr
        Exception().__init__('Error in script')

You might also add a nice __str__ method to ScriptException (you are sure to need it to debug your scripts) -- but I leave that to the reader.

If you don't use stdout=subprocess.PIPE etc, the script will be attached directly to the console. This is really handy if you have, for instance, a password prompt from ssh. So you might want to add flags to control whether you want to capture stdout, stderr, and stdin.

Vshaped

I'm pretty sure that's 10x cooler than what he was looking for. – Fairfax

Nice! added to my toolbox. – Styrax

what is the usage of parameter stdin for function run_script ? – Waverly

Why do you use -c? – Sunfish

-c tells bash to execute the string. So bash "echo hi" would try to execute the script file named "echo hi" (which won't work!), while bash -c "echo hi" would try to run "echo hi" (and will display "hi") – Vshaped

Answer 3

Assuming the command is supported by the host system:

import os
os.system('command')

If you have a long command, or a set of commands. you can use variables. eg:

# this simple line will capture column five of file.log
# and then removed blanklines, and gives output in filtered_content.txt.

import os

filter = "cat file.log | awk '{print $5}'| sed '/^$/d' > filtered_content.txt"

os.system(filter)

Phane

Answer 4

Is

import os
os.system ("bash -c 'echo $0'")

going to do it for you?

EDIT: regarding readability

Yes, of course, you can have it more readable

import os
script = """
echo $0
ls -l
echo done
"""
os.system("bash -c '%s'" % script)

EDIT2: regarding macros, no python does not go so far as far as i know, but between

import os
def sh(script):
    os.system("bash -c '%s'" % script)

sh("echo $0")
sh("ls -l")
sh("echo done")

and previous example, you basically get what you want (but you have to allow for a bit of dialectical limitations)

Picturize

great! and, is it some shortcut or workaround for making it easier or more eye-appealing? – Cruzcruzado

I mean, can i define some macro in python like;(#define sh os.system(" ....."), so in the code i just write, sh ps -ef ? – Cruzcruzado

Answer 5

subprocess and os.system() works fine when bash commands are simple and does not have brackets, commas and quotes. Simple way to embed complex bash argument is to add bash script at the end of python script with a unique string comments and use simple os.system() commands to tail and convert to bash file.

#!/usr/bin/python
## name this file  "file.py"
import os
def get_xred(xx,yy,zz):
    xred=[]
####gaur###
    xred.append([     zz[9] ,  zz[19] ,  zz[29]     ])
    xred.append([     zz[9] ,  xx[9]  ,  yy[9]      ])
    xred.append([     zz[10],  zz[20] ,  zz[30]     ])
    xred.append([     zz[10],  xx[10] ,  yy[10]     ])
###nitai###
    xred=np.array(xred)
    return xred
## following 3 lines executes last 6 lines of this file.
os.system("tail -n 6 file.py >tmpfile1")
os.system("sed 's/###123//g' tmpfile1>tmpfile2")
os.system("bash tmpfile2")
###### Here ###123 is a unique string to be removed
###123#!/bin/sh
###123awk '/###gaur/{flag=1;next}/###nitai/{flag=0} flag{print}' file.py >tmp1
###123cat tmp1 | awk '{gsub("xred.append\\(\\[","");gsub("\\]\\)","");print}' >tmp2
###123awk 'NF >0' tmp2 > tmp3
###123sed '$d' tmp3 |sed '$d' | sed '$d' >rotation ; rm tmp*

Brande

Answer 6

I created Sultan to address exactly what you're trying to do. It doesn't have any external dependencies, and tries to be as light as possible and provides a Pythonic interface to Bash.

https://github.com/aeroxis/sultan

Reconnaissance

Sultan looks cool and I am going to use it; in a case where the tool looks genuinely useful / applicable and is open-source, I don't think people should down-vote simply on the principle of disliking anything that might seem self-promotional. The tool looks good! The answer is reasonable. – Bright

Answer 7

@Ian Bicking's answer is useful but it if only allow us to run scripts. Instead we can come up with a more flexible code where we can run command as well. I have a different approach from his.

#!/usr/bin/env python3

from subprocess import Popen, PIPE


class BashCommandsException(Exception):
    def __init__(self, returncode, output, error_msg):
        self.returncode = returncode
        self.output = output
        self.error_msg = error_msg
        Exception.__init__('Error in executed command')


def popen_communicate(cmd, stdout_file=None):
    """Acts similir to lib.run(cmd) but also returns the output message captures on
    during the run stdout_file is not None in case of nohup process writes its
    results into a file
    """
    cmd = list(map(str, cmd))  # all items should be string
    if stdout_file is None:
        p = Popen(cmd, stdout=PIPE, stderr=PIPE)
    else:
        with open(stdout_file, "w") as outfile:
            # output written into file, error will be returned
            p = Popen(cmd, stdout=outfile, stderr=PIPE, universal_newlines=True)
            output, error = p.communicate()
            p.wait()
            return p, output, error

    output, error = p.communicate()
    output = output.strip().decode("utf-8")
    error = error.decode("utf-8")
    return p, output, error


def run(cmd):
    log_file = "/tmp/log.txt"
    # if log_file is not provided returned output will be stored in output
    p, output, error_msg = popen_communicate(cmd, log_file)
    if p.returncode != 0:
        raise BashCommandsException(p.returncode, output, error_msg, str(cmd))
    return output


if __name__ == "__main__":
    # This could be any command you want to execute as you were in bash
    cmd = ["bash", "script_to_run.sh"]
    try:
        run(cmd)
    except Exception as e:
        print(e)

Sunfish

Answer 8

There is also the commands module to give more control over the output: https://docs.python.org/2/library/commands.html

Minsk

@gander I think you got down voted because Commands is being deprecated out of Python. – Leund

Oh! Not a good start. Surely its still going to be valid for people stuck on older versions though? – Minsk

Answer 9

As aforementioned, you could use os.system(); it's quick and dirty, bu it's easy to use and works for most cases. It's literally a mapping on to the C system() function.

http://docs.python.org/2/library/os.html#os.system

http://www.cplusplus.com/reference/cstdlib/system/

Jargon