Test Container test cases are failing due to "Could not find a valid Docker environment"

Question

I am very new to using test containers. My test is failing with below exception.

Running com.mastercard.example.testcontainers.testcontainersexampple.DemoControllerTest
2020-04-08 14:27:08.441  INFO   --- [           main] o.s.t.c.support.AbstractContextLoader    
: Could not detect default resource locations for test class 
resource found for suffixes {-context.xml, Context.groovy}.
2020-04-08 14:27:08.449  INFO   --- [           main] t.c.s.AnnotationConfigContextLoaderUtils : Could not detect default configuration classes for test class [com.mastercard.example.testcontainers.testcontainersexampple.DemoControllerTest]: DemoControllerTest does not declare any static, non-private, non-final, nested classes annotated with @Configuration.
2020-04-08 14:27:08.611  INFO   --- [           main] .b.t.c.SpringBootTestContextBootstrapper : Found @SpringBootConfiguration com.mastercard.example.testcontainers.testcontainersexampple.TestContainersExampleApplication for test class com.mastercard.example.testcontainers.testcontainersexampple.DemoControllerTest
2020-04-08 14:27:08.701  INFO   --- [           main] .b.t.c.SpringBootTestContextBootstrapper : Loaded default TestExecutionListener class names from location [META-INF/spring.factories]: [org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener, org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListener, org.springframework.boot.test.autoconfigure.restdocs.RestDocsTestExecutionListener, org.springframework.boot.test.autoconfigure.web.client.MockRestServiceServerResetTestExecutionListener, org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrintOnlyOnFailureTestExecutionListener, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverTestExecutionListener, org.springframework.test.context.web.ServletTestExecutionListener, org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener, org.springframework.test.context.support.DependencyInjectionTestExecutionListener, org.springframework.test.context.support.DirtiesContextTestExecutionListener, org.springframework.test.context.transaction.TransactionalTestExecutionListener, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener, org.springframework.test.context.event.EventPublishingTestExecutionListener]
2020-04-08 14:27:08.725  INFO   --- [           main] .b.t.c.SpringBootTestContextBootstrapper : Using TestExecutionListeners: [org.springframework.test.context.web.ServletTestExecutionListener@117159c0, org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener@3e27ba32, org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener@7ef82753, org.springframework.boot.test.autoconfigure.SpringBootDependencyInjectionTestExecutionListener@3b0fe47a, org.springframework.test.context.support.DirtiesContextTestExecutionListener@202b0582, org.springframework.test.context.transaction.TransactionalTestExecutionListener@235ecd9f, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener@1ca3b418, org.springframework.test.context.event.EventPublishingTestExecutionListener@58cbafc2, org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListener@2034b64c, org.springframework.boot.test.autoconfigure.restdocs.RestDocsTestExecutionListener@75d3a5e0, org.springframework.boot.test.autoconfigure.web.client.MockRestServiceServerResetTestExecutionListener@74d1dc36, org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrintOnlyOnFailureTestExecutionListener@7161d8d1, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverTestExecutionListener@74e28667]
2020-04-08 14:27:08.781 ERROR   --- [           main] o.t.d.DockerClientProviderStrategy       : Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
2020-04-08 14:27:08.782 ERROR   --- [           main] o.t.d.DockerClientProviderStrategy       :     UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (ping failed). Root cause NoSuchFileException (/var/run/docker.sock)
2020-04-08 14:27:08.782 ERROR   --- [           main] o.t.d.DockerClientProviderStrategy       : As no valid configuration was found, execution cannot continue

My question is do I need docker installed locally in order to use testcontainers? If yes, how does this works from pipeline like Jenkins?

Caused by: java.lang.IllegalStateException: Could not find a valid Docker environment. Please see logs and check configuration
at org.testcontainers.dockerclient.DockerClientProviderStrategy.lambda$getFirstValidStrategy$3(DockerClientProviderStrategy.java:158)
at java.util.Optional.orElseThrow(Optional.java:290)
at org.testcontainers.dockerclient.DockerClientProviderStrategy.getFirstValidStrategy(DockerClientProviderStrategy.java:150)
at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:111)
at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:175)
at org.testcontainers.containers.JdbcDatabaseContainer.<init>(JdbcDatabaseContainer.java:36)
at org.testcontainers.containers.PostgreSQLContainer.<init>(PostgreSQLContainer.java:32)
at com.mastercard.example.testcontainers.testcontainersexampple.DemoControllerTest.<clinit>(DemoControllerTest.java:27)
... 25 more

UPDATE I have installed docker in local (to try it out) but still I am not able to run test cases due to not able to download image. My corporate network might be an issue here. Does anyone has good solution for above situations?

Caused by: com.github.dockerjava.api.exception.DockerClientException: Could not pull image: unexpected EOF
    at com.github.dockerjava.core.command.PullImageResultCallback.checkDockerClientPullSuccessful(PullImageResultCallback.java:96)
    at com.github.dockerjava.core.command.PullImageResultCallback.throwFirstError(PullImageResultCallback.java:111)
    at com.github.dockerjava.core.async.ResultCallbackTemplate.awaitCompletion(ResultCallbackTemplate.java:91)
    at com.github.dockerjava.core.command.PullImageResultCallback.awaitSuccess(PullImageResultCallback.java:124)

Lecompte

Yes, Testcontainers require a valid docker installation. When it comes to your second problem (after you've installed Docker): please try to pull the image with CLI, i.e. run a console and do docker pull <image name> and post the result. – Selfliquidating

Thanks @jannis. It seems my corporate VPN network does not allow to download, I tried from CLI as well and got EOF exception in that. Is there any way to get image otherway and instruct testcontainers and/or docker to work offline? – Lecompte

You need to get the image to the local registry somehow. Options are: a] build the image locally b] pull from remote registry (but as you wrote this does not work) c] use docker save to export the image to a file and then docker load to load it to local registry. – Selfliquidating

how do I build the image locally? i presume I need to download somehow to my corporate machine which is stopping me to do that – Lecompte

how do I build the image locally? You need a Dockerfile and context files (any jars/scripts needed by the Dockerfile). What is this image? Is it a DB image? Spring boot application? – Selfliquidating

@Selfliquidating what do you mean by valid docker container ? i have thesmae issue only in my case test container works fine locally using postgress, but when i push to gitlab my pipeline is failing, shows thesame error. – Peper

@Peper I wrote "valid docker installation" not "valid docker container". – Selfliquidating

Answer 1

If you're using MAC, in my case, to solve the problem I had to add a link.

Follow the command below:

sudo ln -s $HOME/.docker/run/docker.sock /var/run/docker.sock

Decoder

Don't forget to restart docker after this (or at least in my case it was needed to make this solution work). – Lapwing

One addition is that mac seems to remove this ln between upgrades (I've had to bookmark this answer for that reason) – Isacco

In my case (Rancher Desktop) the command was: sudo ln -s $HOME/.rd/docker.sock /var/run/docker.sock – Fad

Answer 2

In my case, this error was caused by root permission. If you are using ubuntu, docker wants permission to create a container and test-containers can not ask you for this permission at runtime. So you need to make docker runnable without sudo.

Try these steps to create a docker group and add a connected user:

$ sudo groupadd docker
$ sudo gpasswd -a $USER docker
$ sudo service docker restart

Try this in step 3 if you are using Ubuntu 14.04-15.10:

$ sudo service docker.io restart

After all, steps try some docker commands in the terminal without sudo like:

$ docker ps

If you are taking permission error, restart your computer, and then this must be fixed.

According to @AmyDoxy's suggestion, in addition to these steps, if the test container still can't find docker run the following command:

$ sudo chmod 666 /var/run/docker.sock

Ewing

In addition to these steps, I had to sudo chmod 666 /var/run/docker.sock – Yukyukaghir

i have this problem sudo: groupadd: command not found – Robinrobina

@IlyaY Then you are not running Ubuntu. The group management commands will differ a bit between Linux distros; look for documentation perhaps starting from useradd which should be reasonably standard. Another possibility is that you don't have /sbin on your PATH, which will hide this and many other administrative commands; but adding it is obviously easy, or just use the full path to the binary /sbin/groupadd – Uptown

@AmyDoxy sudo chmod 666 /var/run/docker.sock FTW!!! Thanks a lot! – Architectural

remember must re-login ssh, then command will take effect – Ransome

this answer didn't work for you and you're on a macos? Go down to @Pratham's answer. TL;DR: Docker Preferences -> Advanced -> "Enable default Docker socket (Requires password)" -> Restart Docker -> Happy Days! – Ballot

Answer 3

No other answers across the internet worked for me, likely because my case was somewhat unique. I am hoping this answer allows others in this weird, particular scenario to realize what is the problem.

So If:

1. You are on Linux
2. You are running your tests through an IDE
3. You installed IDE through flatpak, snap, or any other package manager that does sandboxing

The issue is that you are sandboxed by the package manager in the application. If you have a socket at /var/run/docker.sock (default), but the tests can't pick it up, this is why. That file does not exist in the sandbox, only on the host filesystem. Depending on the package manager, /var/run may be mounted inside the sandbox, so check your particular documentation. The most straightforward way to resolve this is to install the IDE outside of a sandbox via a tarball or some other thing.

Orestes

If you're running your IDE (Intellij in my case) via Flatpak, you can add an override, like described here github.com/flathub/com.jetbrains.IntelliJ-IDEA-Ultimate/issues/… via command line or in Flatseal – Combustion

podman is rootless by default and works with intellij if you pretend that you have docker installed: ln -s /run/user/$(id -u)/podman/podman.sock /var/run/user/$(id -u)/docker.sock – Lewd

Answer 4

This issue also happens when your docker demon is not running. Starting it and running the tests again would fix the issue if it used to work before.

Negotiant

Answer 5

I had a similar exception on my M1 Mac, when switching from Docker Desktop to Rancher Desktop. The approach suggested by @Wanderson Xesquevixos de Sique turned out to solve things, except that the path for symbolic linkage turned out to be different:

sudo ln -s ~/.rd/docker.sock /var/run/docker.sock

Nl

Answer 6

It could be one of several issues:

• Unstable network connection as you suggested
• You have not logged in to dockerhub from your terminal

In my case I needed to also add my user to the docker user group as per this link:

• I also pulled the image before running the test, just in case.
• One more thing I had to do was to specify the image name when initialising the TestContainers instance as is shown below:

Inconvertible

Never forget to login to your dockerhub :) – Shinberg

Lol logging in solved my issue :) Thanks. – Streaming

@Shinberg I have the same issue but I couldn't fix it. Can you help me? #77122954 – Uninhibited

Answer 7

Testcontainers communicate to docker via sockets. In the newer version of Docker Desktop sockets are disabled by default.

This can be enabled Preferences -> Advanced -> "Enable default Docker socket (Requires password)"

Jampacked

This is specifically what fixed it for me! Thanks for sharing. Macos developers, be warned that upgrade to Sonoma is likely to trigger docker to upgrade itself and this flag may be set to false, forcing you to go for this solution by @Pratham. – Ballot

this works perfectly for sonoma – Lidstone

Thanks a lot! This solved my issue after a couple of wasted hours. – Woof

Answer 8

If you are using Docker Desktop, simply check the flag "Allow the default Docker socket to be used" in Settings

Tewell

Answer 9

testcontainers tries to auto-detect how to connect to docker by inspecting the environment variable DOCKER_HOST or the file <user home>/.testcontainers.properties

In my case (Win10 + Docker Desktop on WSL2), I had to expose the docker daemon on tcp://localhost:2375 without TLS and then either: 1) set the environment variable and its value (in IDE or system) to DOCKER_HOST=tcp://localhost:2375 or 2) create the file <user home>/.testcontainers.properties and add this lines:

docker.host=tcp\://localhost\:2375
# This second line might be not completely necessary:
docker.client.strategy=org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy

BTW, if you expose the daemon, it would be better if you restrict access to it from outside your machine, using the Windows Firewall (it's not sarcasm ;-) )

Phytopathology

Answer 10

Maybe you forgot start docker on your computer. This solved the problem for me.

Cambric

Answer 11

I am using colima, and only this worked for me(2019 i9 mac).

export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock
export DOCKER_HOST=unix://${HOME}/.colima/docker.sock
sudo ln -s $HOME/.colima/docker.sock /var/run/docker.sock

Eckardt

Before I saw your answer and used sudo ln -s $HOME/.colima/default/docker.sock /var/run/docker.sock (no export statements) and that worked for me. – Weiman

That would not work with TESTCONTAINERS for me. I had to do the whole thing. – Eckardt

Answer 12

The official documentation will help you.

$ dockerd-rootless-setuptool.sh install

Bizet

Answer 13

None of the solutions listed here did not work for me. I installed docker desktop, and docker.sock was not in /var/run/docker.sock.

So I had to find docker.sock using ss -x -a | grep docker.sock:

u_str LISTEN 0      4096                     /home/xxx/.docker/desktop/docker.sock 37055              * 0             
u_str ESTAB  0      0                        /home/xxx/.docker/desktop/docker.sock 5890598            * 5891857       
u_str ESTAB  0      0                        /home/xxx/.docker/desktop/docker.sock 44658              * 49293  

I found that the file was in $HOME/.docker/desktop/docker.sock.

So to make it work, I had to run sudo ln -s $HOME/.docker/desktop/docker.sock /var/run/docker.sock

Hope it helps you :)

Kingkingbird

I think this is due to the latest changes in Docker Desktop. Once you "Enable default Docker socket (requires password)" in Preferences -> Advanced as suggested by @Jampacked the link will be created! – Huambo

Answer 14

If you are using colima, use

sudo ln -s $HOME/.colima/default/docker.sock /var/run/docker.sock

Bullace

Answer 15

I am using Colima with multiple profiles and this worked for me:

sudo ln -s $HOME/.colima/{profilename}/docker.sock /var/run/docker.sock

Hungerford

Answer 16

I had the same problem but only when I try to debug code. In my case the problem was too many breakpoints in Intellij. When I deleted all of them (especially from previous debuggings) and then put only one which I needed, the test container started normally. Maybe my response can be helpful in case when any other options don't work.

Transmigrate

Answer 17

In my case, Docker Desktop wasn't able to pull images because the CPU usage was 100%. Restarting the PC fixed my issue,

Iatric

Answer 18

For people using a rootless Podman, TestContainers currently only officially support Docker. Given Podman's aim to be a stand-in replacement for Docker, it is however possible to use it with TestContainers:

# Enable the Podman socket
systemctl --user enable podman.socket --now

# This environment variable is used by TestContainers
export DOCKER_HOST=unix:///run/user/`id -u`/podman/podman.sock

# Ryuk needs to be disabled to avoid running root containers in Podman
export TESTCONTAINERS_RYUK_DISABLED=true

The systemctl command only needs to be executed once. (will still work after reboots) The two export commands can be added to your bashrc or similar shell configuration.

Demona

Answer 19

Make sure you have Docker Desktop running locally

Wommera