How to use the output of a curl command like a dictionary in Python

Question

When I execute the command:

 curl http://cs-service:5000/swdpconfig/swdp_templateConfigData/robot_framework

the output is:

{
  "adm_ts_path": "/data/ngxp_test_automation/bin/admin",
  "be_ts_path": "/data/ngxp_test_automation/bin/backend",
  "fe_ts_path": "/data/ngxp_test_automation/bin/frontend",
  "ip": "40.124.25.232",
  "password": "Er1c550n@123",
  "port": "22",
  "user": "ngxpcdd"
}

Now I want to use the key-value pairs as parameters in my python script. How can I do that?

Perce

a_dict = json.loads(a_string) ... maybe (probably) – Bergmann

Answer 1

You could pipe it directly to python if you want it executed in cli, and if not as someone mentioned you can save it as a file and load it in python with json.loads

Here is already answered stackoverflow question that might peak your interest

Pipe output from shell command to a python script

Farrier

that the main question how to pipe it directly to python?. – Tantalize

@VishalSingh with a [bash pipe and python script reading stdin ] [#11110359 also edited the main answer for visibility – Farrier

Answer 2

why would you mix python and curl ... that basically doesnt make much sense ... but you easily can

import subprocess,json
data = json.loads(subprocess.check_output('curl ...'))

it would be a much better idea to just use python (either requests or urllib)

import requests
url = "http://cs-service:5000/swdpconfig/swdp_templateConfigData/robot_framework"
data = requests.get(url).json()

if you wanted to use urllib instead i think its just

import urllib.request, json
data = json.loads(urllib.request.urlopen(urlData).read())

Bergmann

Why was this downvoted? With this answer, I found the keywords to search for. I tested the first two code lines and I did not get it to work, but that was for sure my own fault. This answer still helped me clearly to get further: run the comman of the CLI from within Python, that is how it should be done, not by making the output of the CLI and then trying to pass that to Python. – Mackenie

Answer 3

After the hint of the other answer, I got it done with subprocess.Popen(), see the code at GenAI How can I open a licensed GPT chat like on https://chat.openai.com/ on my own user if I have an API key that is not from my user?, search for the heading: "Run the curl string in the CLI and get its output in Python" at the end:

Run the curl string in the CLI and get its output in Python

With this string at hand, you can run curl in the CLI and still get its output in Python, taken from Json parsing Python subprocess:

import subprocess 
import json

def getProcessOutput(cmd):
    process = subprocess.Popen(
        cmd,
        shell=True,
        stdout=subprocess.PIPE)
    process.wait()
    data, err = process.communicate()
    if process.returncode is 0:
        return data.decode('utf-8')
    else:
        print("Error:", err)
    return ""

# for domain in getProcessOutput("cat /etc/localdomains").splitlines():
cmd = str_curl
data = getProcessOutput(cmd) 
json.loads(data) 

Out:

{'id': 'chatcmpl-abc123......................', 
'object': 'chat.completion',  'created': 1706123456,  'model':
'gpt-4-0613',  'choices': [{'index': 0,    'message': {'role':
'assistant',
    'content': '"arn:aws:sagemaker:us-west-2:123456789012:model-package/mymodelpackage"'},
'logprobs': None,    'finish_reason': 'stop'}],  'usage':
{'prompt_tokens': 32, 'completion_tokens': 24, 'total_tokens': 56}, 
'system_fingerprint': None}

With that done and loaded into a json dictionary, you can check the mere answer in the json tree with:

print(json.loads(data)['choices'][0]['message']['content'])

Out:

"arn:aws:sagemaker:us-west-2:123456789012:model-package/mymodelpackage"

The main clue of subprocess.Popen() comes from Stack Overflow Json parsing Python subprocess and another answer there that shows that you can shorten it to process.communicate()[0].

Mackenie