AWS CLI for S3 Select

Question

I have the following code, which is used to run a SQL query on a keyfile, located in a S3 bucket. This runs perfectly. My question is, I do not wish to have the output written over to an output file. Could I see the output on the screen (my preference #1)? If not, what about an ability to append to the output file, rather than over-write it (my preference #2). I am using the AWS-CLI binaries to run this query. If there is another way, I am happy to try (as long as it is within bash)

aws s3api select-object-content \
    --bucket "project2" \
    --key keyfile1 \
    --expression "SELECT * FROM s3object s where Lower(s._1) = '[email protected]'" \
    --expression-type 'SQL' \
    --input-serialization '{"CSV": {"FieldDelimiter": ":"}, "CompressionType": "GZIP"}' \
    --output-serialization '{"CSV": {"FieldDelimiter": ":"}}' "OutputFile"

Kashakashden

I see you put a bounty on the question. Does it mean that the answer with /dev/stdout does not work? Can you provide an info why it does not work? Any errors? – Danyelldanyelle

Hi Marcin. thanks for checking in. i tried it on cygwin but it didnt work. No errors, no output. I could try on my ubuntu machine as well, in case u think this could be OS specific – Kashakashden

No problem. You could comment at @jellycsc so that he gets notified. Maybe he knows how to do it on cygwin? But on ubuntu or any other linux I don't see a reason why it would not work. Thus I asked in the first place. – Danyelldanyelle

Answer 1

Of course, you can use AWS CLI to do this since stdout is just a special file in linux.

aws s3api select-object-content \
--bucket "project2" \
--key keyfile1 \
--expression "SELECT * FROM s3object s where Lower(s._1) = '[email protected]'" \
--expression-type 'SQL' \
--input-serialization '{"CSV": {"FieldDelimiter": ":"}, "CompressionType": "GZIP"}' \
--output-serialization '{"CSV": {"FieldDelimiter": ":"}}' /dev/stdout

Note the /dev/stdout in the end.

Dullish

@ jellycsc - could you pls confirm if this would work on cygwin? – Kashakashden

Answer 2

The AWS CLI does not offer such options.

However, you are welcome to instead call it via an AWS SDK of your choice.

For example, in the boto3 Python SDK, there is a select_object_content() function that returns the data as a stream. You can then read, manipulate, print or save it however you wish.

Seamanship

Thanks John. I am trying to move away from Python (long story why). Is there any other workaround here? Could I append to a single output file maybe? – Kashakashden

You can use any of the other programming languages, too: C++, Go, Java, .Net, Node, PHP, Ruby. FYI, the AWS CLI is written in Python and simply calls the above command. – Seamanship

Answer 3

I think it opens /dev/stdout twice causing kaos.

Pouched