Get single-line json from aws cli

Question

I have this command:

aws ec2 describe-regions --output=json

the output looks like:

{
    "Regions": [
        {
            "Endpoint": "ec2.eu-north-1.amazonaws.com", 
            "RegionName": "eu-north-1"
        }, 
        {
            "Endpoint": "ec2.ap-northeast-2.amazonaws.com", 
            "RegionName": "ap-northeast-2"
        }, 
        {
            "Endpoint": "ec2.us-west-2.amazonaws.com", 
            "RegionName": "us-west-2"
        }
    ]
}

that's not really usable by machine though, I am looking for output on a single-line like so:

{"Regions":[{"Endpoint":"ec2.eu-north-1.amazonaws.com","RegionName":"eu-north-1"},{"Endpoint":"ec2.ap-northeast-2.amazonaws.com","RegionName":"ap-northeast-2"},{"Endpoint":"ec2.us-west-2.amazonaws.com","RegionName":"us-west-2"}]}

is there some command with the aws-cli that gives me machine-readable JSON? Something like:

aws ec2 describe-regions --output='json-for-machines'

?

Dupuis

whys that not readable? – Neighborly

I have a streaming-parser than splits the json by newline character, if there are newlines in the json string my parser won't work ... newline delimited json parsers are common so.. – Dupuis

Answer 1

Use jq to compact it like so.

aws ec2 describe-regions --output=json| jq -c

output becomes

{"Regions":[{"Endpoint":"ec2.eu-north-1.amazonaws.com","RegionName":"eu-north-1"},{"Endpoint":"ec2.ap-northeast-2.amazonaws.com","RegionName":"ap-northeast-2"},{"Endpoint":"ec2.us-west-2.amazonaws.com","RegionName":"us-west-2"}]}

https://stedolan.github.io/jq/

Neighborly

Answer 2

jq answer is a good choice. If you don't have jq but have python:

aws ec2 describe-regions --output=json | python -c 'import json, sys; json.dump(json.load(sys.stdin), sys.stdout)'

If you don't have jq but have node:

aws ec2 describe-regions --output=json | node -e 'console.log(JSON.stringify(JSON.parse(require("fs").readFileSync(0, "utf8"))))'

Arum

Answer 3

if you are using windows cmd:

aws ec2 describe-regions  --output json | ConvertFrom-Json | ConvertTo-Json -Compress

• ConvertFrom-Json: converts it to manageable object
• ConvertTo-Json -Compress: gets rid of newlines

Laurencelaurene

I can't verify this works, but hopefully someone will eventually – Dupuis

$output = -join $output | ConvertFrom-Json This worked for me – Barnett

Answer 4

Native AWS CLI solution

If you'd really like to get compact JSON without an additional program, you can use JMESPath's to_string function in the an AWS command's --query client-side filter arg to serialize into compact JSON, and then --output=text to avoid double-quoting it.

Example

$ aws ec2 describe-regions --query='to_string(@)' --output=text
{"Regions":[{"Endpoint":"ec2.ap-south-1.amazonaws.com","RegionName":"ap-south-1","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.eu-north-1.amazonaws.com","RegionName":"eu-north-1","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.eu-west-3.amazonaws.com","RegionName":"eu-west-3","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.eu-west-2.amazonaws.com","RegionName":"eu-west-2","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.eu-west-1.amazonaws.com","RegionName":"eu-west-1","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.ap-northeast-3.amazonaws.com","RegionName":"ap-northeast-3","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.ap-northeast-2.amazonaws.com","RegionName":"ap-northeast-2","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.ap-northeast-1.amazonaws.com","RegionName":"ap-northeast-1","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.ca-central-1.amazonaws.com","RegionName":"ca-central-1","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.sa-east-1.amazonaws.com","RegionName":"sa-east-1","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.ap-southeast-1.amazonaws.com","RegionName":"ap-southeast-1","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.ap-southeast-2.amazonaws.com","RegionName":"ap-southeast-2","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.eu-central-1.amazonaws.com","RegionName":"eu-central-1","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.us-east-1.amazonaws.com","RegionName":"us-east-1","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.us-east-2.amazonaws.com","RegionName":"us-east-2","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.us-west-1.amazonaws.com","RegionName":"us-west-1","OptInStatus":"opt-in-not-required"},{"Endpoint":"ec2.us-west-2.amazonaws.com","RegionName":"us-west-2","OptInStatus":"opt-in-not-required"}]}

Important caveat from the AWS CLI docs

⚠️ Important

The output type you specify changes how the --query option operates:

• If you specify --output text, the output is paginated before the --query filter is applied, and the AWS CLI runs the query once on each page of the output. Due to this, the query includes the first matching element on each page which can result in unexpected extra output. To additionally filter the output, you can use other command line tools such as head or tail.

• If you specify --output json, the output is completely processed as a single, native structure before the --query filter is applied. The AWS CLI runs the query only once against the entire structure, producing a filtered result that is then output.

This means that technically this approach will yield you one line of JSON per page of results from the server.

If you would only like to fetch one page to begin with, you can pass the --no-paginate option to disable the AWS CLI's automatic fetch-all-pages feature.

Fini

Answer 5

If you're using node.js to parse a json-stream, you can use:

https://github.com/ORESoftware/json-parser

const k = cp.spawn('bash');

k.stdin.end(`
   aws ec2 describe-instances --output=json --region='us-west-2'; echo '∆';
   aws ec2 describe-instances --output=json --region='us-west-1'; echo '∆';
   aws ec2 describe-instances --output=json --region='us-east-1'; echo '∆';
`);

k.stdout.pipe(new JSONParser({delimiter:'∆'}).on('data', d => {
    // { Reservations: [{}] }
});

just use some weird unicode character delimiter and should be no worse than using newlines to delimit, eg:

const delimiter = 'ƒ∆ø';

Dupuis