Best way to 'listen' to s3 bucket for new file?

Question

I have connected zapier to a webhook I am listening too, which sends a JSON file into my s3 bucket.

I have some python code that I want to execute when a new file is uploaded into the bucket, in real time over the file.

What is the best way to 'listen' for the upload of this file into the s3 bucket?

Thoroughpaced

Have you considered using a lambda function? docs.aws.amazon.com/lambda/latest/dg/… – Bonilla

@LiamFoley I have not thought about using a lambda function, open to all suggestions. Something that is easy to setup is exactly what I am looking for. – Thoroughpaced

Answer 1

You can setup an Amazon's CloudWatch Event to listen for when a new object is put into a S3 bucket. You can configure the event to work with a specific S3 bucket. When a new object is put into the bucket you can configure the event to trigger a Lambda function that will execute whatever custom Python code you want to run when a new object is added.

Your question is pretty broad so I can't get into specifics, but you can checkout the AWS documentation that provides examples to guide you through the process. AWS CloudWatch Documentation.

Boar

do you think I could read in the file from the s3 bucket into aws lambda for the python bit? – Thoroughpaced

Yes. If you configure a Lambda function to get triggered by a Put event details about the object such as the bucket and key can be passed to the Lambda function. You can leverage that information to read the file that has been uploaded. – Boar

Answer 2

David here, from the Zapier Platform team.

Seems like you've already found your answer, which is great. I just wanted to plug Zapier as an option (since you had mentioned you're already using it). Our S3 integration has a "new file in bucket" trigger, which you can combine with any other step (such as a Python Code step). Additionally, you could skip the middleman and structure your zap as:

1. some trigger
2. Add file to S3
3. Run Python

And not need to worry about webhooks at all.

​Let me know if you've got any other questions!

Contented

thank you for the reply! I would love to keep everything under one roof. Currently I have setup zapier webhook to s3 then my code. My code has libraries such as pandas, requests, and json to name a few. Would these dependencies run on zapier platform? – Thoroughpaced

Not as Python, no. Unfortunately we don't support python libraries in code steps. You can use JS libraries in a CLI app, but that's a pretty substantial re-write that you don't need to do. In this case, it sounds like you could still benefit from the S3 trigger, but the action would send a webhook to wherever your code is. – Contented

Answer 3

There are a few recommended options involving event notifications in S3 buckets outlined here in AWS docs https://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-event-notifications.html

Vaden

Thank you, in your opinion what is the easiest way? – Thoroughpaced