Filter S3 list-objects results to find a key matching a pattern

Question

I would like to use the AWS CLI to query the contents of a bucket and see if a particular file exists, but the bucket contains thousands of files. How can I filter the results to only show key names that match a pattern? For example:

aws s3api list-objects --bucket myBucketName --query "Contents[?Key==*mySearchPattern*]"

Explore

Answer 1

The --query argument uses JMESPath expressions. JMESPath has an internal function contains that allows you to search for a string pattern.

This should give the desired results:

aws s3api list-objects --bucket myBucketName --query "Contents[?contains(Key, `mySearchPattern`)]"

(With Linux I needed to use single quotes ' rather than back ticks ` around mySearchPattern.)

If you want to search for keys starting with certain characters, you can also use the --prefix argument:

aws s3api list-objects --bucket myBucketName --prefix "myPrefixToSearchFor"

Explore

On Linux using single quotes didn't work. But escaping the backticks worked. (Eg. \`mySearchPattern\` ) – Subaltern

Also, if mySearchPattern is a pure number (eg. 20150101 ) then the CLI complains it should be a string and not an integer. Even through you did quote it. Didn't find a solution except include a non-digit in the searchPattern (eg. 20150101/ ) then it doesn't complain. – Subaltern

Thanks for the single quotes comment ! I was wondering wtf was going on ,,,, – Tinware

--prefix did the trick for me which worked well with real "S3 prefixes" combined with a prefix for an object name, for example --prefix myS3Prefix/myFileNamePrefix. Where the full object key is myS3Prefix/myFileNamePrefix_blahblah. – Incompetent

Answer 2

I tried on Ubuntu 14, awscli 1.2

--query "Contents[?contains(Key,'stati')].Key" 
--query "Contents[?contains(Key,\'stati\')].Key" 
--query "Contents[?contains(Key,`stati`)].Key"

Illegal token value '?contains(Key,'stati')].Key'

After upgraded the aws version to 1.16 , worked with

--query "Contents[?contains(Key,'stati')].Key" 

Corsica