Unmarshal map[string]DynamoDBAttributeValue into a struct

Question

I'm trying to set-up an AWS-lambda using aws-sdk-go that is triggered whenever a new user is added to a certain dynamodb table.

Everything is working just fine but I can't find a way to unmarshal a map map[string]DynamoDBAttributeValue like:

{
    "name": {
        "S" : "John"
    },
    "residence_address": {
        "M": {
            "address": {
                "S": "some place"
            }
        }
    }
}

To a given struct, for instance, a User struct. Here is shown an example of unsmarhaling a map[string]*dynamodb.AttributeValue into a given interface, but I can't find a way to do the same thing with map[string]DynamoDBAttributeValue even though these types seem to fit the same purposes.

map[string]DynamoDBAttributeValue is returned by a events.DynamoDBEvents from package github.com/aws/aws-lambda-go/events. This is my code:

package handler

import (
    "context"
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
    "github.com/aws/aws-sdk-go/service/dynamodb"
)

func HandleDynamoDBRequest(ctx context.Context, e events.DynamoDBEvent) {

    for _, record := range e.Records {

        if record.EventName == "INSERT" {

            // User Struct
            var dynamoUser model.DynamoDBUser

            // Of course this can't be done for incompatible types
            _ := dynamodbattribute.UnmarshalMap(record.Change.NewImage, &dynamoUser)
        }

    }

}

Of course, I can marshal record.Change.NewImage to JSON and unmarshal it back to a given struct, but then, I would have to manually initialize dynamoUser attributes starting from the latter ones.

Or I could even write a function that parses map[string]DynamoDBAttributeValue to map[string]*dynamodb.AttributeValue like:

func getAttributeValueMapFromDynamoDBStreamRecord(e events.DynamoDBStreamRecord) map[string]*dynamodb.AttributeValue {
    image := e.NewImage
    m := make(map[string]*dynamodb.AttributeValue)
    for k, v := range image {
        if v.DataType() == events.DataTypeString {
            s := v.String()
            m[k] = &dynamodb.AttributeValue{
                S : &s,
            }
        }
        if v.DataType() == events.DataTypeBoolean {
            b := v.Boolean()
            m[k] = &dynamodb.AttributeValue{
                BOOL : &b,
            }
        }
        // . . .
        if v.DataType() == events.DataTypeMap {
            // ?
        }
    }
    return m
}

And then simply use dynamodbattribute.UnmarshalMap, but on events.DataTypeMap it would be quite a tricky process.

Is there a way through which I can unmarshal a DynamoDB record coming from a events.DynamoDBEvent into a struct with a similar method shown for map[string]*dynamodb.AttributeValue?

Hunchback

Answer 1

I tried the function you provided, and I met some problems with events.DataTypeList, so I managed to write the following function that does the trick:

// UnmarshalStreamImage converts events.DynamoDBAttributeValue to struct
func UnmarshalStreamImage(attribute map[string]events.DynamoDBAttributeValue, out interface{}) error {

    dbAttrMap := make(map[string]*dynamodb.AttributeValue)

    for k, v := range attribute {

        var dbAttr dynamodb.AttributeValue

        bytes, marshalErr := v.MarshalJSON(); if marshalErr != nil {
            return marshalErr
        }

        json.Unmarshal(bytes, &dbAttr)
        dbAttrMap[k] = &dbAttr
    }

    return dynamodbattribute.UnmarshalMap(dbAttrMap, out)

}

Abercrombie

Thanks for your answer. Tomorrow I'll try it. About lists you are right, in the example I did not cover them but It's possible to extend my function to handle them. Anyway your solution is by far better than mine. – Hunchback

I tried your function and it works perfectly. I made small fixes to it since dbAttrMap was badly declared and you made no error handling for marshalErr. As soon as you accept the edit I'll accept the answer. – Hunchback

It's better now, approve it. – Abercrombie

Answer 2

I was frustrated that the type of NewImage from the record wasn't map[string]*dynamodb.AttributeValue so I could use the dynamodbattribute package.

The JSON representation of events.DynamoDBAttributeValue seems to be the same as the JSON represenation of dynamodb.AttributeValue.

So I tried creating my own DynamoDBEvent type and changed the type of OldImage and NewImage, so it would be marshalled into map[string]*dynamodb.AttributeValue instead of map[string]events.DynamoDBAttributeValue

It is a little bit ugly but it works for me.

package main

import (
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
    "github.com/aws/aws-sdk-go/service/dynamodb"
    "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
    "fmt"
)

func main() {

    lambda.Start(lambdaHandler)
}

// changed type of event from: events.DynamoDBEvent to DynamoDBEvent (see below)
func lambdaHandler(event DynamoDBEvent) error {

    for _, record := range event.Records {

        change := record.Change
        newImage := change.NewImage // now of type: map[string]*dynamodb.AttributeValue

        var item IdOnly
        err := dynamodbattribute.UnmarshalMap(newImage, &item)
        if err != nil {
            return err
        }

        fmt.Println(item.Id)
    }

    return nil
}

type IdOnly struct {
    Id string `json:"id"`
}

type DynamoDBEvent struct {
    Records []DynamoDBEventRecord `json:"Records"`
}

type DynamoDBEventRecord struct {
    AWSRegion      string                       `json:"awsRegion"`
    Change         DynamoDBStreamRecord         `json:"dynamodb"`
    EventID        string                       `json:"eventID"`
    EventName      string                       `json:"eventName"`
    EventSource    string                       `json:"eventSource"`
    EventVersion   string                       `json:"eventVersion"`
    EventSourceArn string                       `json:"eventSourceARN"`
    UserIdentity   *events.DynamoDBUserIdentity `json:"userIdentity,omitempty"`
}

type DynamoDBStreamRecord struct {
    ApproximateCreationDateTime events.SecondsEpochTime             `json:"ApproximateCreationDateTime,omitempty"`
    // changed to map[string]*dynamodb.AttributeValue
    Keys                        map[string]*dynamodb.AttributeValue `json:"Keys,omitempty"`
    // changed to map[string]*dynamodb.AttributeValue
    NewImage                    map[string]*dynamodb.AttributeValue `json:"NewImage,omitempty"`
    // changed to map[string]*dynamodb.AttributeValue
    OldImage                    map[string]*dynamodb.AttributeValue `json:"OldImage,omitempty"`
    SequenceNumber              string                              `json:"SequenceNumber"`
    SizeBytes                   int64                               `json:"SizeBytes"`
    StreamViewType              string                              `json:"StreamViewType"`
}

Dilatation

This is an elegant solution. I like this. – Disadvantage

Answer 3

I have found the same problem and the solution is to perform a simple conversion of types. This is possible because in the end the type received by lambda events events.DynamoDBAttributeValue and the type used by the SDK V2 of AWS DynamoDB types.AttributeValue are the same. Next I show you the conversion code.

package aws_lambda

import (
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
    "github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)

func UnmarshalDynamoEventsMap(
    record map[string]events.DynamoDBAttributeValue, out interface{}) error {
    asTypesMap := DynamoDbEventsMapToTypesMap(record)
    err := attributevalue.UnmarshalMap(asTypesMap, out)
    if err != nil {
        return err
    }
    return nil
}

func DynamoDbEventsMapToTypesMap(
    record map[string]events.DynamoDBAttributeValue) map[string]types.AttributeValue {
    resultMap := make(map[string]types.AttributeValue)
    for key, rec := range record {
        resultMap[key] = DynamoDbEventsToTypes(rec)
    }
    return resultMap
}

// DynamoDbEventsToTypes relates the dynamo event received by AWS Lambda with the data type that is
// used in the Amazon SDK V2 to deal with DynamoDB data.
// This function is necessary because Amazon does not provide any kind of solution to make this
// relationship between types of data.
func DynamoDbEventsToTypes(record events.DynamoDBAttributeValue) types.AttributeValue {
    var val types.AttributeValue

    switch record.DataType() {
    case events.DataTypeBinary:
        val = &types.AttributeValueMemberB{
            Value: record.Binary(),
        }
    case events.DataTypeBinarySet:
        val = &types.AttributeValueMemberBS{
            Value: record.BinarySet(),
        }
    case events.DataTypeBoolean:
        val = &types.AttributeValueMemberBOOL{
            Value: record.Boolean(),
        }
    case events.DataTypeList:
        var items []types.AttributeValue
        for _, value := range record.List() {
            items = append(items, DynamoDbEventsToTypes(value))
        }
        val = &types.AttributeValueMemberL{
            Value: items,
        }
    case events.DataTypeMap:
        items := make(map[string]types.AttributeValue)
        for k, v := range record.Map() {
            items[k] = DynamoDbEventsToTypes(v)
        }
        val = &types.AttributeValueMemberM{
            Value: items,
        }
    case events.DataTypeNull:
        val = nil
    case events.DataTypeNumber:
        val = &types.AttributeValueMemberN{
            Value: record.Number(),
        }
    case events.DataTypeNumberSet:
        val = &types.AttributeValueMemberNS{
            Value: record.NumberSet(),
        }
    case events.DataTypeString:
        val = &types.AttributeValueMemberS{
            Value: record.String(),
        }
    case events.DataTypeStringSet:
        val = &types.AttributeValueMemberSS{
            Value: record.StringSet(),
        }
    }

    return val
}

Cheltenham

Answer 4

There is a package that allows conversion from events.DynamoDBAttributeValue to dynamodb.AttributeValue

https://pkg.go.dev/github.com/aereal/go-dynamodb-attribute-conversions/v2

From there one can unmarshal AttributeValue into struct

func Unmarshal(attribute map[string]events.DynamoDBAttributeValue, out interface{}) error {

    av := ddbconversions.AttributeValueMapFrom(attribute)

    return attributevalue.UnmarshalMap(av, out)
}

Autocracy