How to support more than one trigger in AWS Lambda in Golang?
Asked Answered
B

5

14

I am building an AWS Lambda function in Golang that copy the content from n to m S3 buckets. There is a requirement to support for S3 trigger as well as fetching the data from an SQS where all source S3 bucket change is stored. The code can be found here: https://github.com/maknahar/s3copy

I tried following:

func main() {
    lambda.Start(ProcessIncomingS3Events)
    lambda.Start(ProcessIncomingEvents)
}

func ProcessIncomingS3Events(event events.S3Event) error {
    ...
    log.Println("Got S3 Event")
    return processS3Trigger(config, event)
}

func ProcessIncomingEvents() error {
    ...
    log.Println("Defaulting to SQS")
    return processSQSMessage(config)
}

In this case, the first event ProcessIncomingS3Events is triggered every time.

I tried following as well

func main() {
    lambda.Start(ProcessIncomingEvents)
}

func ProcessIncomingEvents(event interface{}) error {
    ...
    switch request := event.(type) {
    case events.S3Event:
        log.Println("Got S3 Event")
        return processS3Trigger(config, request)

    case types.Nil:
        log.Println("Defaulting to SQS")
        return processSQSMessage(config)

    default:
        log.Println("Could not find the event type")

    }

    return nil
}

In this case, Lambda could not detect the type and Could not find the event type is logged in every trigger.

Is there a way to support multiple triggers via AWS SDK at all for the function?

Burnell answered 27/3, 2018 at 15:9 Comment(3)
Why not separate lambdas for separate event type?Reformatory
Creating different lambda function would require maintaining that many functions. All of which will do the same job. Currently, I do not have CI setup for deployment so changing code or configuration would require manual setup for all as well.Burnell
Another reason might be for a lambda function that's part of a VPC. To avoid incurring a VPC cold start delay it may be nice to have a scheduled ping event to keep the function warm alongside whatever type of event that we actually want the function to handle.Miscellanea
U
6

I achieved to listen to multiple events by implementing the AWS Handler interface, it defines one method:

Invoke(ctx context.Context, payload []byte) ([]byte, error)

I implemented a multievent Handler as follows

type Handler struct {
//add global variables or context information that your handler may need
}

func (h Handler) Invoke(ctx context.Context, data []byte) ([]byte, error) {
  //for demonstration purposes, not the best way to handle
  apiGatewayEvent := new(events.APIGatewayProxyRequest)
  if err := json.Unmarshal(data, apiGatewayEvent); err != nil {
    log.Println("Not a api gateway event")
  }
  snsEvent := new(events.SNSEvent)
  if err := json.Unmarshal(data, snsEvent); err != nil {
    log.Println("Not a sns event")
  }
  return nil, nil
}

func main() {
  lambda.StartHandler(Handler{})
}

As you can see, you could get the raw bytes of any event and handle them as you need giving you the possibility to listen to any AWS event with the same lambda. However, think carefully before using this approach, because as noted above, lambdas are best used handling just one type of event.

Utley answered 29/9, 2018 at 22:8 Comment(1)
doesnt work. we get the error : json: cannot unmarshal object into Go value of type []uint8: UnmarshalTypeError null while the bytes[] represent the right object: S3EventTaxeme
C
2

You can configure multiple event sources to trigger one or more Lambda functions.

However, in Go the lambda.Start call is blocking so it's not very easy to write a single function that handles multiple event types. You are strongly encouraged to make a separate Lambda function for every event source.

The idiomatic Go solution is to have your function logic defined once in the main package, and write multiple programs that take the source event and call your function. So the project layout would be:

  • s3copy/main.go
  • s3copy/handlers/s3/main.go
  • s3copy/handlers/sqs/main.go

See my boilerplate Lambda and Go app for an example project layout and Makefile.

Colbycolbye answered 30/3, 2018 at 14:42 Comment(1)
The blocking lambda.Start only blocks the current invocation. It does not interfere with concurrent invocations (possibly invoked by other triggers).Monogamist
S
1

I don't know if you found the solution yet but i have found solution.

Here it is..

package main

import (
    "errors"
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

func main() {
    lambda.Start(Handler)
}

type Event1 struct {
    //Event Attributes
}

type Event2 struct {
    //Event Attributes
}
type Composite struct {
    *Event1
    *Event2
    *events.S3Event
}

func Handler(c Composite) error {
    if c.Event1 != nil {
        //go along with code based on Event1
    } else if c.Event2 != nil {
        //go along with code based on Event2
    } else if c.S3Event != nil {
        //go along with code based on S3Event
    } else {
        return errors.New("wrong event type")
    }
    return nil
}

This will work fine as you can see whichever event is fired that struct will not be empty and also you can easily add extra events by adding their pointer in Composite struct just like i have added *Event1 and *Event2.

There is one drawback that you can not Put SQSEvent and S3Event togather as their attributes have same name and json tag Record Also if to event have same attributes that will also backfire as it will put zero value in all attributes which are same in different struct

Also there is one more solution which can overcome above problems

package main

import (
    "encoding/json"
    "errors"
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

func main() {
    lambda.Start(Handler)
}

type CustomEvent struct {
    //Give EventSource "aws:manual" value to determine event is manual
    EventSource string `json:"eventSource"`

    //Other CustomEvent Attributes
}

func Handler(event map[string]interface{}) error {
    //marshal map to get json of data
    eventMarshaled, err := json.Marshal(event)
    if err != nil {
        return err
    }

    //Flag for determining if any one of the event is parsed

    var eventParsed bool

    //declare CustomEvent variable to use while Unmarshaling
    var c CustomEvent
    if err := json.Unmarshal(eventMarshaled, &c); err == nil {
        if c.EventSource == "aws:manual" {
            eventParsed = true

            //your code based on CustomEvent
        }
        return nil
    }
    if !eventParsed {
        //declare SQSEvent variable to use while Unmarshaling
        var sqsEvent events.SQSEvent
        if err := json.Unmarshal(eventMarshaled, &sqsEvent); err == nil {
            for _, message := range sqsEvent.Records {
                if message.EventSource == "aws:sqs" {
                    eventParsed = true
                    //Your Code based on sqs event
                }
            }
            return nil
        }
    }

    if !eventParsed {
        //declare S3Event variable to use while Unmarshaling
        var s3Event events.S3Event
        if err := json.Unmarshal(eventMarshaled, &s3Event); err == nil {
            for _, message := range s3Event.Records {
                if message.EventSource == "aws:sqs" {
                    eventParsed = true
                    //Your Code based on s3 event
                }
            }
            return nil
        }
    }

    return errors.New("wrong event type")
}

As you can see the handler gets map[string]interface{} by default from aws sdk you just need to marshal it and then unmarshal it to your desired type.you have to give eventSource attributes to all your custom events and check their eventSource so that you can do your operations based on that. also don't forget to send that eventSource whenever you are sending manual event other then aws specific like SQS,SNS,S3 etc... and you can see in above example how to check eventSource for SQS and S3 you can find eventSource attribute in mostly every Events aws provides but i am not sure but you can easily find some unique attributes in every aws provided Events which can be checked for nil and you can then determine which type of event it is. this method fixes the problem in first solution but it also adds unnecessary marshaling and unmarshaling of json and also there is different methods of checking different events but i think there is no sureshot way to do this

Sadyesaechao answered 3/5, 2023 at 8:8 Comment(0)
R
0

In approach one, you are directly calling ProcessIncomingS3Events in first statement, so every time this is called.

Read this - Lambda Function Handler (Go)

In above link, the author is parsing event's name field. Similarly, you can check for any field which is always present in S3 event e.g. "eventSource":"aws:s3" (S3 event structure see here)

If present then S3 event else other. Or you can also check for SQS event's field.

Reformatory answered 27/3, 2018 at 16:8 Comment(2)
I was actually looking for some way where SDK provides the functionality to add multiple triggers. In my current use case, there is a possibility to add more trigger. So, I cannot hardcode one type of event as it is done in the example given in the link provided by you. However, link given by you helped me understand what can be put as input event and thanks for that.Burnell
Lambda SDK tries to map event(json from source) directly to specified type in lambda handler. So not possible from SDK. You can write your own utility library(wrapper) to do this and improve that as more requirement comes in future. E.g in one of my project there was a need to maintain state for tracing purpose. So I wrote one super lambda with required functionality and now all my other lambdas are extending from that. you can think of having one facing lambda, only taking String as the event and then you do all checks suggested in the answer and forward it to right handler. Hope it helps.Reformatory
P
0

You can use embedding in Go to solve this:

import (
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
    "reflect"
)

type Event struct {
    events.SQSEvent
    events.APIGatewayProxyRequest
    //other event type
}

type Response struct {
    events.SQSEventResponse `json:",omitempty"`
    events.APIGatewayProxyResponse `json:",omitempty"`
   //other response type
}

func main() {
    lambda.Start(eventRouter)
}

func eventRouter(event Event) (Response, error) {
    var response Response
    switch {
    case reflect.DeepEqual(event.APIGatewayProxyRequest, events.APIGatewayProxyRequest{}):
        response.SQSEventResponse = sqsEventHandler(event.SQSEvent)
    case reflect.DeepEqual(event.SQSEvent, events.SQSEvent{}):
        response.APIGatewayProxyResponse = apiGatewayEventHandler(event.APIGatewayProxyRequest)
  //another case for a event handler
    }
    return response, nil
}

func sqsEventHandler(sqsEvent events.SQSEvent) events.SQSEventResponse {
    //do something with the SQS event 
}

func apiGatewayEventHandler(apiEvent events.APIGatewayProxyRequest) events.APIGatewayProxyResponse {
    //do something with the API Gateway event
}

Note: if the basic events have the same field names, you will need to look for another compare method instance of DeepEqual.

Paranoid answered 14/2, 2022 at 20:27 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.