How to paginate over an AWS CLI response?

Question

I'm trying to paginate over EC2 Reserved Instance offerings, but can't seem to paginate via the CLI (see below).

% aws ec2 describe-reserved-instances-offerings --max-results 20                                                                                 
{
    "NextToken": "someToken", 
    "ReservedInstancesOfferings": [
        {
             ...
        }
    ]
}
% aws ec2 describe-reserved-instances-offerings --max-results 20 --starting-token someToken
Parameter validation failed:
Unknown parameter in input: "PaginationConfig", must be one of: DryRun, ReservedInstancesOfferingIds, InstanceType, AvailabilityZone, ProductDescription, Filters, InstanceTenancy, OfferingType, NextToken, MaxResults, IncludeMarketplace, MinDuration, MaxDuration, MaxInstanceCount

The documentation found in [1] says to use start-token. How am I supposed to do this?

[1] http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-reserved-instances-offerings.html

Thoroughwort

Answer 1

Looks like some busted documentation.

If you run the following, this works:

aws ec2 describe-reserved-instances-offerings --max-results 20 --next-token someToken

Translating the error message, it said it expected NextToken which can be represented as next-token on the CLI.

Thoroughwort

Answer 2

With deference to a 2017 solution by marjamis which must have worked on a prior CLI version, please see a working approach for paginating from AWS in bash from a Mac laptop and aws-cli/2.1.2

# The scope of this example requires that credentials are already available or
# are passed in with the AWS CLI command.  
# The parsing example uses jq, available from https://stedolan.github.io/jq/

# The below command is the one being executed and should be adapted appropriately.
# Note that the max items may need adjusting depending on how many results are returned.
aws_command="aws emr list-instances --max-items 333 --cluster-id $active_cluster"
unset NEXT_TOKEN

function parse_output() {
  if [ ! -z "$cli_output" ]; then
    # The output parsing below also needs to be adapted as needed.
    echo $cli_output | jq -r '.Instances[] | "\(.Ec2InstanceId)"' >> listOfinstances.txt
    NEXT_TOKEN=$(echo $cli_output | jq -r ".NextToken")
  fi
}

# The command is run and output parsed in the below statements.
cli_output=$($aws_command)
parse_output

# The below while loop runs until either the command errors due to throttling or
# comes back with a pagination token.  In the case of being throttled / throwing
# an error, it sleeps for three seconds and then tries again.
while [ "$NEXT_TOKEN" != "null" ]; do
  if [ "$NEXT_TOKEN" == "null" ] || [ -z "$NEXT_TOKEN" ] ; then
    echo "now running: $aws_command "
    sleep 3
    cli_output=$($aws_command)
    parse_output
  else
    echo "now paginating: $aws_command --starting-token $NEXT_TOKEN"
    sleep 3
    cli_output=$($aws_command --starting-token $NEXT_TOKEN)
    parse_output
  fi
done  #pagination loop

Gassman

Answer 3

If you continue to read the reference documentation that you provided, you will learn that:

--starting-token (string)

A token to specify where to start paginating. This is the NextToken from a previously truncated response.

Moreover:

--max-items (integer)

The total number of items to return. If the total number of items available is more than the value specified in max-items then a NextToken will be provided in the output that you can use to resume pagination.

Weissmann

1. I think you posted the wrong link...? 2. But if you use starting-token as a flag, the command won't work. You have to use next-token. – Thoroughwort

@Thoroughwort 1. Sorry, I have updated the post with the correct link. 2. You cannot use the --starting-token argument the first time you execute the command, since you have not yet received any NextToken (it will be provided as part of the response, if the response was truncated due to the pagination). – Weissmann

Answer 4

I found a caveat using --max-results for paginating and noticed it took a very long time and the results had many duplicates. A quick look at the documentation warns that --max-results and --page-size should be set to the same value.

From the docs:

If you specify different values for --page-size and --max-items, you can get unexpected results with missing or duplicated items. To prevent this, use the same number for --page-size and --max-items to sync the AWS CLI pagination with the pagination of the underlying service.

Once I made --page-size the same value as --max-results, not only did it eliminate the duplicates but the results starting coming back quickly.

Chassepot