How to create S3 and triggered lambda in 2 different cloudformation templates

Question

Can a S3 bucket and triggered Lambda be created in separate CloudFormation templates. I want to keep long running resources stack separate from the likes of Lambda which get updated quite frequently

When tried to create Lambda separately it says that bucket defined in lambda event should be defined in same template and cannot be referenced.

S3 events must reference an S3 bucket in the same template.

GetFileMetadata:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub '${targetenv}-lambdaname'
      CodeUri: target-file-0.0.1-SNAPSHOT.jar
      Handler: LambdaFunctionHandler::handleRequest
      Runtime: java8
      Timeout: 30
      MemorySize: 512
      Environment:
        Variables:
          STAGE: !Sub '${targetenv}'
          
      Events:
        S3Event:
          Type: S3
          Properties:
            Bucket:
              Ref: MyS3Bucket
            Events:
              - 's3:ObjectCreated:*'
  
  MyS3Bucket:
      Type: 'AWS::S3::Bucket'
      DependsOn: BucketPermission
      Properties:
          BucketName: !Sub 'bucketname-${targetenv}'

Strathspey

Answer 1

On November 21 2021, AWS announced S3 Event Notifications with Amazon EventBridge. Consequently, you can deploy one stack with an S3 bucket with EventBridge integration enabled and then a second stack with a Lambda function that is triggered by EventBridge events for the specific bucket.

Persistence Stack:

AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: 'Stack with S3 bucket with EventBridge event notification enabled'

Parameters:
  BucketName:
    Type: String
    Description: 'Name of the bucket to be created'

Resources:

  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref BucketName
      NotificationConfiguration:
        EventBridgeConfiguration:
          EventBridgeEnabled: true

#       Alternatively shorthand config
#       EventBridgeConfiguration: {}

Application Stack:

AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Stack with Lambda for procesing S3 events via EventBridge

Parameters:
  BucketName:
    Type: String
    Description: Name of the bucket to listen events from

Resources:
  S3EventProcessor:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: S3EventListener
      Architectures:
        - arm64
      Runtime: nodejs14.x
      Handler: index.handler
      InlineCode: |
          exports.handler = (event, context) => {
            console.log('event:', JSON.stringify(event));
          }
      Events:
        S3EventBridgeRule:
          Type: EventBridgeRule
          Properties:
            Pattern:
              source:
                - aws.s3
              detail:
                bucket:
                  name:
                    - !Ref BucketName

By configuring the Pattern, you can filter the events stream for more specific events such as Object Create or Object Deleted, file names, file extension, etc. Please find more info in the EventBridge userguide

Dowdell

Answer 2

This could not be done when this answer was originally written, but there has been progress in this area. Since then, S3 has added support for SNS and SQS Event as AWS::S3::Bucket NotificationConfiguration which could be declared in one stack and then imported to the other stack. More recently, AWS has also added EventBridge as yet another option, please see my other answer.

This is not possible in SAM version 2016-10-31. Copied from the S3 event source type in the SAM documentation:

NOTE: To specify an S3 bucket as an event source for a Lambda function, both resources have to be declared in the same template. AWS SAM does not support specifying an existing bucket as an event source.

Dowdell

i can't believe that 2 years later this is still true and that I'm now facing the same issue despite the fact that the ticket for fixing this has already been closed... – Portend

@KamilJanowski AWS has added support for S3 and EventBridge integration, please read my updated answer – Dowdell

Answer 3

The template is creating a bucket (MyS3Bucket).

Then, the serverless function is referencing it:

        Bucket:
          Ref: MyS3Bucket

If you want to refer to that bucket from another template, you can export the bucket name from the first stack:

Outputs:

  S3Bucket:
    Description: Bucket that was created
    Value: !Ref MyS3Bucket
    Export:
      Name: Stack1-Bucket

Then, import it into the second stack:

        Bucket:
            Fn::ImportValue:
              Stack1-Bucket

See: Exporting Stack Output Values - AWS CloudFormation

Elisavetgrad

Tried, still gives same error -Event with id [S3Event] is invalid. S3 events must reference an S3 bucket in the same template. – Strathspey

Okay. The above works for a "normal" CloudFormation template, but might not work for a SAM template. – Elisavetgrad