Run Dynamodb local as part of a Gradle Java project

Question

I am trying to run DynamoDB local for testing purposes. I followed the steps amazon provides for setting it up and running the jar by itself works fine (link to amazon's tutorial Here). However, the tutorial doesn't go over running the jar within your own project. I don't want all the other developers to have to grab a jar and run it locally every time they test their code.

That is where my question comes in. I've had a real hard time finding any examples online of how to configure a Gradle project to run the DynamoDB local server as part of my tests. I found the following maven example https://github.com/awslabs/aws-dynamodb-examples/blob/master/src/test/java/com/amazonaws/services/dynamodbv2/DynamoDBLocalFixture.java#L32 and am trying to convert it to a Gradle, but am getting errors for all of com.amazonaws.services.dynamodbv2.local import statements they are using. The errors are that the resource cannot be found.

I went into their project's pom and put the following into my build.gradle file to emulate it.

//dynamodb local dependencies
testCompile('com.amazonaws:aws-java-sdk-dynamodb:1.10.42')
testCompile('com.amazonaws:aws-java-sdk-cloudwatch:1.10.42')
testCompile('com.amazonaws:aws-java-sdk:1.3.0')
testCompile('com.amazonaws:amazon-kinesis-client:1.6.1')
testCompile('com.amazonaws:amazon-kinesis-connectors:1.1.1')
testCompile('com.amazonaws:dynamodb-streams-kinesis-adapter:1.0.2')
testCompile('com.amazonaws:DynamoDBLocal:1.10.5.1')

The import statements still fail. Here is an example of one that fails.

import com.amazonaws.services.dynamodbv2.local.embedded.DynamoDBEmbedded;

TL;DR

Has anyone managed to get the DynamoDB local JAR to execute as part of a Gradle project or have a link to a good tutorial (it doesn't have to be the tutorial I linked to).

Bullace

I realize that if I can just get gradle to run the following command should solve my problem: java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb – Bullace

Answer 1

We have DynamoDB local working with gradle. Here's what you need to add to your gradle.build file:

For gradle 4.x and below versions

1) Add to the repositories section:

    maven {
        url 'http://dynamodb-local.s3-website-us-west-2.amazonaws.com/release'
    }

2) Add to the dependencies section (assuming you're using this for your tests):

    testCompile group: 'com.amazonaws', name: 'DynamoDBLocal', version: 1.11.0

3) These next two steps are the tricky part. First copy the native files to a directory:

task copyNativeDeps(type: Copy) {
    from (configurations.testCompile) {
        include "*.dylib"
        include "*.so"
        include "*.dll"
    }
    into 'build/libs'
}

4) Then make sure you include this directory (build/libs in our case) in the java library path like so:

test.dependsOn copyNativeDeps
test.doFirst {
    systemProperty "java.library.path", 'build/libs'
}

Now you should be able to run ./gradlew test and have your tests hit your local DynamoDB.

Tetrachloride

Thanks for the answer Serega. I am no longer working on this project, so it will take me awhile to test and verify your answer... – Bullace

No problem. Hope it's useful for anyone else that stumbles on to this thread looking to fix a similar issue. – Tetrachloride

Where Can I get the latest version of DynamoDBLocal ... I have already setup a gradle project and using 1.11.0 version - please reply – Sancho

docs.aws.amazon.com/amazondynamodb/latest/developerguide/… – Tetrachloride

Answer 2

For Gradle 5.x the below solution works

 maven {
        url 'http://dynamodb-local.s3-website-us-west-2.amazonaws.com/release'
    }

configurations {
        dynamodb
    }

dependencies {
    testImplementation 'com.amazonaws:DynamoDBLocal:1.11.477'
    dynamodb fileTree (dir: 'lib', include: ["*.dylib", "*.so", "*.dll"])
    dynamodb 'com.amazonaws:DynamoDBLocal:1.11.477'
}

task copyNativeDeps(type: Copy) {
    from configurations.dynamodb
    into "$project.buildDir/libs/"
}

test.dependsOn copyNativeDeps
test.doFirst {
    systemProperty "java.library.path", 'build/libs'
}

Kale

This also works for Gradle 6.0.1 if you want to upgrade from Gradle 5 – Griceldagrid

If you are having troubles getting Gradle to find the dependency, see: #70894471 – Juryman

Answer 3

I run into the same problem and first I tried to add sqlite4java.library.path to the Gradle script as it has been mentioned in the other comments.

This worked for command line, but were not working when I was running the tests from IDE (IntelliJ IDEA), so finally I come up with a simple init method, that is called at the beginning of each of integration tests:

AwsDynamoDbLocalTestUtils.initSqLite();
AmazonDynamoDBLocal amazonDynamoDBLocal = DynamoDBEmbedded.create();

Implementation can be found here: https://github.com/redskap/aws-dynamodb-java-example-local-testing/blob/master/src/test/java/io/redskap/java/aws/dynamodb/example/local/testing/AwsDynamoDbLocalTestUtils.java

I put a whole example to GitHub, it might be helpful: https://github.com/redskap/aws-dynamodb-java-example-local-testing

Votaw

Answer 4

In August 2018 Amazon announced new Docker image with Amazon DynamoDB Local onboard. It does not require downloading and running any JARs as well as adding using third-party OS-specific binaries like sqlite4java.

It is as simple as starting a Docker container before the tests:

docker run -p 8000:8000 amazon/dynamodb-local

You can do that manually for local development, as described above, or use it in your CI pipeline. Many CI services provide an ability to start additional containers during the pipeline that can provide dependencies for your tests. Here is an example for Gitlab CI/CD:

test:
  stage: test
  image: openjdk:8-alpine
  services:
    - name: amazon/dynamodb-local
      alias: dynamodb-local
  script:
    - ./gradlew clean test

So, during the test task DynamoDB will be available on http://dynamodb-local:8000.

Another, more powerful tool is localstack. It supports two dozen of AWS services, DynamoDB is one of them. The isage is very similar, you have to start it before running the tests and it will expose AWS-compatible APIs on given ports:

test:
  stage: test
  image: openjdk:8-alpine
  services:
    - name: localstack/localstack
    alias: localstack
  script:
    - ./gradlew clean test

The idea is to move all the configuration out of your build tool and tests and provide the dependency externally. Think of it as of dependency injection / IoC but for the whole service, not just a single bean. This way, you code is more clean and maintainable. You can see that even in the examples above: you can switch mock implementation from DynamoDB Local to localstack by simply changing the image part!

Hubbs

Answer 5

The easiest way, in my opinion, is to:

1. Download the JAR from here: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html#DynamoDBLocal.DownloadingAndRunning

2. Then unzip the downloaded folder and add its content to the /libs folder in the project (create the /libs folder before that)

3. Finally, add to the build.gradle:

   dependencies {
       runtime files('libs/DynamoDBLocal.jar')
   }
   

Breault

Links can and usually do go dead. It'd be better to edit your answer to include a path to get the jar or something similar. – Scrofula

Answer 6

I didn't want to create a specific configuration for dynamo for gradle 6+ so I tweaked the original answer instructions. Also, this is in kotlin gradle DSL rather than groovy.

val copyNativeDeps by tasks.creating(Copy::class) {
    from(configurations.testRuntimeClasspath) {
        include("*.dylib")
        include("*.so")
        include("*.dll")
    }
    into("$buildDir/libs")
}

tasks.withType<Test> {
    dependsOn.add(copyNativeDeps)
    doFirst { systemProperty("java.library.path", "$buildDir/libs") }
}

By leveraging the testRuntimeClasspath configuration, gradle is able to locate the relevant files for you without needing to create a custom configuration. Obviously this has the side effect that if your test runtime has many native deps, they will also be copied which would make the custom configuration approach more ideal.

Nuclide

Answer 7

Gradle 8 Solution

Adapting from @togise's solution and pairing with another SO question.

// ... Source set, plugins...

// Define a custom configuration set to allow us to configure DynamoDb local.
configurations {
    dynamoDbLocal
}


repositories {
    mavenCentral()
    // Add another Maven source
    maven {
        name "DynamoDB Local Release Repository"
        url "https://s3-us-west-2.amazonaws.com/dynamodb-local/release"
    }
}

dependencies {
    // Adds a locally running DynamoDB instance
    testImplementation 'com.amazonaws:DynamoDBLocal:2.0.0'
    configurations.dynamoDbLocal { fileTree(dir: 'lib', include: ["*.dylib", "*.so", "*.dll"]) }
    configurations.dynamoDbLocal { 'com.amazonaws:DynamoDBLocal:2.0.0' }

    // Other dependencies...
}

// Register a new Gradle task that may be called by others tasks
tasks.register('copyNativeDeps', Copy) {
    from configurations.dynamoDbLocal
    into "$project.buildDir/libs/"
}

test {
    // These two statements are required to set up DynamoDb local
    dependsOn copyNativeDeps
    doFirst {
        systemProperty "java.library.path", 'build/libs'
    }

    // Other test configuration, like useJUnitPlatform()
}

Juryman