Can AWS Glue write to DynamoDB?

Question

I need to do some grouping job from a Source DynamoDB table, then write each resulting Item to another Target DynamoDB table (or a secondary index of the Source one).

Here I see that DynamoDB can be used as a Source (as well as reported in Connection Types).

However, it's not clear to me if a DynamoDB table can be used as Target as well.

Note: each resulting grouping item must be written into a separate DynamoDB Item (i.e., if there are X objects resulting from grouping, X Items must be written to Target DynamoDB table).

Auberge

Answer 1

Glue can now read and write to DynamoDB. The option to write is not available via the console, but can be done by editing the script. Example:

Datasink1 = glueContext.write_dynamic_frame.from_options(
    frame=ApplyMapping_Frame1,
    connection_type="dynamodb",
    connection_options={
        "dynamodb.output.tableName": "myDDBTable",
        "dynamodb.throughput.write.percent": "1.0"
    }
)

As per:

• https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html#etl-connect-dynamodb-as-sink
• https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-dynamo-db-cross-account.html

Bilbe

I wonder what happens when I need to update (UpdateItem) a DDB row instead of inserting (PutItem). I could hardly find any docs on that. – Ghetto

@Ghetto did you find any resources for that? Is it even possible? Facing the same use case. – Freedwoman

I could not find any resources on that. I ended up using boto3 to updated my table. – Ghetto

it's write_dynamic_frame.from_options() not write_dynamic_frame_from_options(). just a typo – Speck

Answer 2

The Glue Job scripts can be customized to write to any datasource. If you are using the auto generated scripts, you can add boto3 library to write to DynamoDb tables.

If you want to test the scripts easily, you can create a Dev endpoint through AWS console & launch a jupyter notebook to write and test your glue job scripts.

Heeled