Json parsing Python subprocess

Question

Here's the code:

inputDomain = subprocess.Popen("cat /etc/localdomains", shell=True,  stdout=subprocess.PIPE)
domains = inputDomain.stdout.read().splitlines()

for domain in domains:
   cmd = "whmapi1 domainuserdata domain " + domain
   output = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
   jsonS = json.dumps(output.communicate())
   print json.loads(jsonS)['data']

here's there error

root@server [~/testP]# python copie.py Traceback (most recent call last): File "copie.py", line 18, in print json.loads(jsonS)['data'] TypeError: list indices must be integers, not str

and this is an example of the json i need to parse:

{ 
   "data":{ 
      "userdata":{ 
      "phpopenbasedirprotect":1,
      "options":"ExecCGI Includes",
     "ip":"10.0.0.1",
     "hascgi":"1",
     "group":"user",
     "usecanonicalname":"Off",
     "scriptalias":[ 
        { 
           "url":"/cgi-bin/",
           "path":"/home/user/public_html/cgi-bin"
        },
        { 
           "url":"/cgi-bin/",
           "path":"/home/user/public_html/cgi-bin/"
        }
     ],
     "user":"user",
     "ifmodulemodsuphpc":{ 
        "group":"user"
     },
     "owner":"root",
     "documentroot":"/home/user/public_html",
     "userdirprotect":"",
     "serveralias":"parkeddomain.com www.parkeddomain.com www.example.com",
     "port":"80",
     "homedir":"/home/user",
     "ifmoduleconcurrentphpc":{ 

     },
     "customlog":[ 
        { 
           "target":"/usr/local/apache/domlogs/example.com",
           "format":"combined"
        },
        { 
           "target":"/usr/local/apache/domlogs/example.com-bytes_log",
           "format":"\"%{%s}t %I .\\n%{%s}t %O .\""
        }
     ],
     "servername":"example.com",
     "serveradmin":"[email protected]"
  }
}

So i need the user and the domaine, but python always answer that i need a int. Thanks for the help guys.

Haunt

are you sure jsonS is not a list? in which case you'd have to do json.loads(jsonS)[0]['data'] – Urbanize

and what the hell with inputDomain = subprocess.Popen("cat /etc/localdomains", shell=True, stdout=subprocess.PIPE)? python can read text files without the need of calling cat !! – Urbanize

yes i already try this.. it give me the same error... – Haunt

You'r right jf i can change that, but that is not the problem for now.. – Haunt

just print(type(jsonS)) just to see the type – Urbanize

it print <type 'tuple'> – Haunt

Answer 1

I had problems running the above under Python3, so here is a way to achieve what OP is asking in Python3

import subprocess
import json


def getProcessOutput(cmd):
    process = subprocess.Popen(
        cmd,
        shell=True,
        stdout=subprocess.PIPE)
    process.wait()
    data, err = process.communicate()
    if process.returncode is 0:
        return data.decode('utf-8')
    else:
        print("Error:", err)
    return ""


for domain in getProcessOutput("cat /etc/localdomains").splitlines():
    cmd = "whmapi1 domainuserdata domain " + domain
    print(json.loads(getProcessOutput(cmd))['data'])

It outputs Error: None because my machine doesn't have /etc/localdomains but otherwise it seems to work just fine.

Scission

It should be noted that this is awful code. It seemed to work, but it lacks tests, and couples the success and error into a method, in a way I'm not comfortable with. Something like JavaScript style promises would make me a lot happier for this. – Scission

Works for any code, no need for a "data" key in your output of the run CLI command, any command you run can be mapped to the data, err. The data, err = process.communicate() runs the CLI command, the output then is in data, which you then can check as a json object. For example, to get the message content of the chatbot answer from curl, run print(json.loads(data)['choices'][0]['message']['content']). see How can I open a licensed GPT chat like on https://chat.openai.com/ on my own user if I have an API key that is not from my user?. – Cosper

Answer 2

since your process returns a json string, there's no need to dump it to load it again.

# stdout, stderr
jsonS,_ = output.communicate()

now you have a string, that you can load using json

d = json.loads(jsonS)

now d['data'] yields the info you want

Aside: as I said:

inputDomain = subprocess.Popen("cat /etc/localdomains", shell=True,  stdout=subprocess.PIPE)
domains = inputDomain.stdout.read().splitlines()

could be replaced by native python:

with open("/etc/localdomains") as f: domains = f.read().splitlines()

Chivaree

Hi thanks but i already try tho, this is the error i got when i do this..TypeError: expected string or buffer So its not a string.. its kinda strange – Haunt

can you type jsonS,_ = output.communicate(), then print(jsonS) – Urbanize

just did output = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE) jsonS,_ = output.communicate() print json.loads(jsonS) --> ValueError: No JSON object could be decoded – Haunt

just print(jsonS) don't try to loads. – Urbanize

ican print it i see the value its like a tuple – Haunt

sonS,_ = output.communicate(): note the ,_. Else you get a tuple – Urbanize

is it possible to parse a tuple into json – Haunt

yes, but it will be the tuple for output + error. You're mixing up "tuple in a string" and actual "tuple" object. – Urbanize

But when I use shell=True, the output contains also the shell's prompt on the first line so it breaks the JSON. How to avoid it? In this case, the output would be $ cat /etc/localdomains\n{ "json": ... }. – Ezequieleziechiele

strange. Anyway, try without shell=True as it is often useless when the command is properly written – Urbanize

also in python 3, to get a string from subprocess you need to decode: inputDomain.stdout.decode() – Urbanize

OK, got it. It contains the prompt only when run on a Windows machine, on a Linux one, the output contains only the output of the command being executed. Do you know how to make it work also on Windows systems, please? – Ezequieleziechiele

I need shell=True because I want to run it without an extension, i.e. subprocess.run(['cm'], shell=True). Then I have two shell scripts, cm for Linux and MacOS and cm.cmd on Windows. With shell=True, the Python script works on both systems. – Ezequieleziechiele

cm.cmd would need shell=True as well or cmd /c prefix. oooh I think I'm getting it. Your cm.cmd should start by @echo off or you'll get some traces. – Urbanize

Answer 3

communicate() returns a tuple: (stdoutdata, stderrdata). when you loads() it back you get a list, and then trying to index it with 'data' will fail. Example:

import json
import subprocess

cmd = "/bin/date"
output = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
jsonS = json.dumps(output.communicate())
print jsonS # ["Fri Feb 10 14:33:42 GMT 2017\n", null]
print json.loads(jsonS)['data'] # TypeError: list indices must be integers, not str

It might be enough to do

jsonS = output.communicate()[0]

Donnelldonnelly

it give me back ValueError: No JSON object could be decoded – Haunt

Ok. Stick a print on the jsonS to see whether the data actually is a JSON. – Donnelldonnelly

it says that jsonS is a tuple – Haunt

I meant, after you make the change Jean and I suggested, to take the first element of the return value of communicate(). – Donnelldonnelly

'data' = output.communicate()[0] – Sacculate

True, process = subprocess.Popen( cmd, shell=True, stdout=subprocess.PIPE) process.wait() process.communicate()[0]['data'] returns the needed "data" key, tested. Therefore, the json.loads() is not needed, you can shorten this, as you say at the end. – Cosper