Embedded MongoDB when running integration tests

Question

My question is a variation of this one.

Since my Java Web-app project requires a lot of read filters/queries and interfaces with tools like GridFS, I'm struggling to think of a sensible way to employ MongoDB in the way the above solution suggests.

Therefore, I'm considering running an embedded instance of MongoDB alongside my integration tests. I'd like it to start up automatically (either for each test or the whole suite), flush the database for every test, and shut down at the end. These tests might be run on development machines as well as the CI server, so my solution will also need to be portable.

Can anyone with more knowledge on MongoDB help me get idea of the feasibility of this approach, and/or perhaps suggest any reading material that might help me get started?

I'm also open to other suggestions people might have on how I could approach this problem...

Milestone

If you are using maven, you can use ours mvnrepository.com/artifact/com.wenzani/mongodb-maven-plugin – Invulnerable

You can also check this project which simulate a MongoDB inside JVM memory. github.com/thiloplanz/jmockmongo But it is still in development. – Vacillation

Not [just for] for unit testings, but read this blog post if you like to run MongoDB (even a cluster) as in-memory deployment if you're using Linux. edgystuff.tumblr.com/post/49304254688 Would be great to have it out of the box like RavenDB though. – Zworykin

Similar to the embedmongo-maven-plugin mentioned here, there is also a Gradle Mongo Plugin available. Like the Maven plugin it also wraps the flapdoodle EmbeddedMongoDb api and allows you to run a managed instance of Mongo from your Gradle builds. – Ozoniferous

Check this code example here: github.com/familysyan/embedded-mongo-integ. No installation, no dependency. It's simply a platform independent ant script that do download and setup for you. It also cleans up everything after your tests. – Seligmann

Answer 1

Here's an updated (for 2022) version of the accepted answer from @rozky (a lot has been changed in both the Mongo and Embedded MongoDB libraries).

package com.example.mongo;

import com.mongodb.BasicDBObject;
import com.mongodb.MongoClient;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
import de.flapdoodle.embed.mongo.MongodExecutable;
import de.flapdoodle.embed.mongo.MongodProcess;
import de.flapdoodle.embed.mongo.MongodStarter;
import de.flapdoodle.embed.mongo.config.MongodConfig;
import de.flapdoodle.embed.mongo.config.Net;
import de.flapdoodle.embed.mongo.distribution.Version;
import de.flapdoodle.embed.process.runtime.Network;
import java.util.Date;
import org.junit.After;
import static org.junit.Assert.*;
import org.junit.Before;
import org.junit.Test;

public class EmbeddedMongoTest {

    private static final String DATABASE_NAME = "embedded";

    private MongodExecutable mongodExe;
    private MongodProcess mongod;
    private MongoClient mongo;

    @Before
    public void beforeEach() throws Exception {
        MongodStarter starter = MongodStarter.getDefaultInstance();
        String bindIp = "localhost";
        int port = 12345;
        MongodConfig mongodConfig = MongodConfig.builder()
            .version(Version.Main.PRODUCTION)
            .net(new Net(bindIp, port, Network.localhostIsIPv6()))
            .build();
        this.mongodExe = starter.prepare(mongodConfig);
        this.mongod = mongodExe.start();
        this.mongo = new MongoClient(bindIp, port);
    }

    @After
    public void afterEach() throws Exception {
        if (this.mongod != null) {
            this.mongod.stop();
            this.mongodExe.stop();
        }
    }

    @Test
    public void shouldCreateNewObjectInEmbeddedMongoDb() {
        // given
        MongoDatabase db = mongo.getDatabase(DATABASE_NAME);
        db.createCollection("testCollection");
        MongoCollection<BasicDBObject> col = db.getCollection("testCollection", BasicDBObject.class);

        // when
        col.insertOne(new BasicDBObject("testDoc", new Date()));

        // then
        assertEquals(1L, col.countDocuments());
    }

}

Outdistance

Repeated start and stop of Embedded mongo for each test fails most of the tests. Its better to start before all the tests and shutdown once all have executed – Tremulant

You need to include @DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_CLASS) along with the change above – Tremulant

@Tremulant You could also maybe use a random port so that you can still run your tests concurrently on a fresh embedded mongo instance. See the docs here. – Outdistance

Yup, this works like a charm. Came here from baeldung - this example just kept throwing Caused by: java.lang.IllegalArgumentException: Database name must not be empty! While i did provide it in (MongoClients.create(String.format(CONNECTION_STRING, ip, port)), "cipresale"); – Torsibility

Guys any help for these two imports. I am getting error on these even after adding <artifactId>de.flapdoodle.embed.mongo</artifactId> . import de.flapdoodle.embed.mongo.config.IMongodConfig; import de.flapdoodle.embed.mongo.config.MongodConfigBuilder; – Shotwell

@sashikanta I just updated the code for the latest changes (specifically, I changed IMongodConfig mongodConfig = new MongodConfigBuilder() to MongodConfig mongodConfig = MongodConfig.builder()). Please try it again now. – Outdistance

@CollinKrawll perfectly working now. Thank you so much. – Shotwell

Answer 2

I have found Embedded MongoDB library which looks quite promising and does what you have asked for.

Currently supports MongoDB versions: 1.6.5 to 3.1.6, provided the binaries are still available from the configured mirror.

Here is short example of use, which I have just tried and it works perfectly:

public class EmbeddedMongoTest {
    private static final String DATABASE_NAME = "embedded";

    private MongodExecutable mongodExe;
    private MongodProcess mongod;
    private Mongo mongo;

    @Before
    public void beforeEach() throws Exception {
        MongoDBRuntime runtime = MongoDBRuntime.getDefaultInstance();
        mongodExe = runtime.prepare(new MongodConfig(Version.V2_3_0, 12345, Network.localhostIsIPv6()));
        mongod = mongodExe.start();
        mongo = new Mongo("localhost", 12345);
    }

    @After
    public void afterEach() throws Exception {
        if (this.mongod != null) {
            this.mongod.stop();
            this.mongodExe.stop();
        }
    }

    @Test
    public void shouldCreateNewObjectInEmbeddedMongoDb() {
        // given
        DB db = mongo.getDB(DATABASE_NAME);
        DBCollection col = db.createCollection("testCollection", new BasicDBObject());

        // when
        col.save(new BasicDBObject("testDoc", new Date()));

        // then
        assertThat(col.getCount(), Matchers.is(1L));
    }
}

Sufism

Just used this library and it worked perfectly JUnit testing a Mongo API on a Mac. Recommended. – Swage

+1 excellent find! When I first started using mongodb a year ago, not having a programmatic way to test against a database was one of the downsides. We got around this by having a test instance on every environment, configured through a Java properties file but of course that needed to have mongo installed on every environment. This looks like it will solve all that. – Triaxial

Nice! deleted my answer since it's no longer accurate. Anyone any idea how mature this is? I can imagine it having to simulate MongoDB on a very low level would be quite complicated and judging from the source it looks pretty high level. – Winegrower

Finally got to play with this in my project and can report that is was incredibly easy to set-up and run. The low-level calls are all part of the official com.mongodb Java API so it is no more complicated than using the regular API. – Triaxial

Be careful with this solution. It just gathers information about the current operating system and downloads the appropriate platform-specific MongoDB binaries from the internet, runs the daemon and does some other configuration stuff. As an enterprise solution, this is not. Mocking may be the only real option. – Batson

Answer 3

There is Foursquare product Fongo. Fongo is an in-memory java implementation of mongo. It intercepts calls to the standard mongo-java-driver for finds, updates, inserts, removes and other methods. The primary use is for lightweight unit testing where you don't want to spin up a mongo process.

Dianetics

Does Fongo happen to intercept calls to the network, e.g. to localhost:27017 so that it can serve as a drop-in fake server to enable integration testing without code changes? – Agulhas

mongo-java-server is a drop-in fake server implementation that can be used for integration testing without code changes. – Terrenceterrene

Answer 4

If you're using Maven you may be interested in a plugin I've created that wraps the flapdoodle.de 'embedded mongo' API:

embedmongo-maven-plugin

It provides a start goal that you can use to start any version of MongoDB you want (e.g. during pre-integration-test), and a stop goal that will stop MongoDB (e.g. during post-integration-test).

The real benefit of using this plugin over others is that there is no requirement for MongoDB to be installed beforehand. MongoDB binaries are downloaded and stored in ~/.embedmongo for future builds.

Strapless

And here's the Clojure version for Leiningen: github.com/joelittlejohn/lein-embongo – Strapless

Answer 5

If you are using sbt and specs2, I wrote the same kind of wrapper for embedmongo

https://github.com/athieriot/specs2-embedmongo

Soemba

Answer 6

with spring-boot 1.3 you can use EmbeddedMongoAutoConfiguration

pom.xml

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.3.2.RELEASE</version>
</parent>
 ...
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-mongodb</artifactId>
    </dependency>
    <dependency>
        <groupId>de.flapdoodle.embed</groupId>
        <artifactId>de.flapdoodle.embed.mongo</artifactId>
        <version>${embedded-mongo.version}</version>
    </dependency>

MongoConfig

@Configuration
@EnableAutoConfiguration(exclude = { EmbeddedMongoAutoConfiguration.class })
public class MongoConfig{
}

Trixy

can you explain what is the "@EnableAutoConfiguration(exclude = { EmbeddedMongoAutoConfiguration.class })" annotation actually doing? – Dwelt

The reason is most probably the de.flapdoodle.embed.mongo dependency not marked for test scope. Not to pick it up and run embedded mongo in production application setup the exclusion is needed. – Wallin

Answer 7

You can run MongoDB in memory as of version 3.2.6. From the site:

Starting in MongoDB Enterprise version 3.2.6, the in-memory storage engine is part of general availability (GA) in the 64-bit builds. Other than some metadata and diagnostic data, the in-memory storage engine does not maintain any on-disk data, including configuration data, indexes, user credentials, etc.

Softwood

Answer 8

Not just for unit testing, but also explained how to use inmemory mongodb with rest api.

maven dependency:

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-mongodb</artifactId>
        </dependency>

        <dependency>
            <groupId>de.flapdoodle.embed</groupId>
            <artifactId>de.flapdoodle.embed.mongo</artifactId>
        </dependency>

=============================================================================

application.properties

server.port = 8080
spring.data.mongodb.database=user_db
spring.data.mongodb.port=27017
spring.data.mongodb.host=localhost

=============================================================================

UserRepository.java

public interface UserRepository extends MongoRepository{

}

for reference and all java code use below link:(step by step explanation)

https://www.youtube.com/watch?v=2Tq2Q7EzhSA&t=7s

Halpern

Answer 9

Performances are better when executing mongod with storageEngine='ephemeralForTest'

new MongodConfigBuilder()
    .version(Version.Main.PRODUCTION)
    .cmdOptions(new MongoCmdOptionsBuilder()
         .useStorageEngine("ephemeralForTest")
         .build())
    .net(new Net("localhost", port, Network.localhostIsIPv6()))
    .build()

Aspirin

Answer 10

To run Embedded mongodb for integration test following are the maven dependency needed:

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-mongodb</artifactId>
            <version>2.5.2</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-autoconfigure</artifactId>
            <version>2.5.2</version>
        </dependency>

        <dependency>
            <groupId>de.flapdoodle.embed</groupId>
            <artifactId>de.flapdoodle.embed.mongo</artifactId>
            <version>3.0.0</version>
            <scope>test</scope>
        </dependency>

Try using below code snipped for EmbeddedMongoAutoConfiguration:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.mongo.MongoProperties;
import org.springframework.boot.autoconfigure.mongo.embedded.EmbeddedMongoAutoConfiguration;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;

@SpringBootApplication
public class EmbeddedMongoApplication {

    public static void main(String[] args) {
         System.setProperty("os.arch", "x86_64");
         SpringApplication.run(EmbeddedMongoApplication.class, args);
    }
    
    @Bean
    public EmbeddedMongoAutoConfiguration embeddedMongoAutoConfiguration(MongoProperties mongoProperties) {
        return new EmbeddedMongoAutoConfiguration(mongoProperties);
    }
}

Note:

Embedded mongodb will be downloaded to below Path. So take into consideration that path have proper permission.

Linux : $HOME/.embedmongo/linux/mongodb-linux-x86_64-3.2.2.tgz
Windows : C:\Users\<username>\.embedmongo\win32\mongodb-win32-x86_64-3.x.x.zip

Pantagruel

Answer 11

In production, you will be using a real database.

If you want your tests to reflect how your product behaves in production, use a real instance of Mongo.

A fake implementation may not behave exactly the same as a real one. When testing, you should strive for correctness. Speed of execution comes second.

Ambidextrous

I think you missed my purpose. I wasn't looking for a fake instance of Mongo, I wanted a real instance but embedded in my tests. The reason was to start up MongoDB and put it into a particular state without a polluting an existing database, run a series of operations, and then inspect the outcome without needing to sift through arbitrary data that was unrelated to my test. As real as it can possibly be whilst still maintaining a controlled test environment. – Milestone

Sorry, the word "simulate" and all these "in-memory" suggestions caused me to forget the meaning of "embedded" in Java-land. Glad to hear it. – Ambidextrous