Running tests with Docker and Jenkins - Test Result

Question

I have an application written in Rails. In order to test it I've built a docker image. Everything works fine.

The problem, however, appears when I configure Jenkins to run those tests. Here's what I do, at the building step:

docker-compose up rspec

Where rspec is a service defined at docker-compose.yml and contains the following command:

command: "rspec spec/"

When rspec returns an error the build still succeeds. Here an example of the output:

    ...
21:42:24 [36mrspec_1  |[0m       should save second profile
21:42:24 [36mrspec_1  |[0m 
21:42:24 [36mrspec_1  |[0m Failures:
21:42:24 [36mrspec_1  |[0m 
21:42:24 [36mrspec_1  |[0m   1) New profile Should persist new_profile_pricture
21:42:24 [36mrspec_1  |[0m      Failure/Error: jump_to_four_phase_with(new_profile_picture)
21:42:24 [36mrspec_1  |[0m      RuntimeError:
21:42:24 [36mrspec_1  |[0m        Timeout for '#new_profile' (1) appearance reached!
...
21:42:25 [36mcomposes_rspec_1 exited with code 1
21:42:25 [0m[Profiler] $ /bin/sh -xe /tmp/hudson4606189750126491465.sh
21:42:25 Finished: SUCCESS

36mcomposes_rspec_1 returned 1 and the build still succeeded.

If I check the container by its id with docker ps -a I get "Exited (1) 2 minutes ago"

Do you guys know what is going on?

Is there an easy way to fail the build when the container fails?

Mesomorph

Answer 1

Docker compose has since added the functionality to get an exit code from the specific container/service running the tests:

docker-compose up \
  --abort-on-container-exit \
  --exit-code-from test-runner 

Original Answer

Jenkins uses the exit status of the process to judge success or failure.

docker-compose up is designed to orchestrate many containers. When you are dealing with multiple services/containers there is a bit of a grey area as to what constitutes success and failure. All that docker-compose reports on exit is that the docker-compose command completed successfully, not that all containers it ran are ok.

docker-compose run <service> <command> will run a single command for a service and return that commands exit status.

If you rely on multiple services/containers for the tests, then have docker-compose up only bring up the required services. Then run docker-compose run rubyservice rspec afterwards for your tests.

Compose seperation

If you want to keep the tests seperate from the app containers, create a second docker-compose-test.yml file that contains a service definition just for the tests.

version: "2.1"
tests:
  build:
    context: .
    dockerfile: Dockerfile.tests
  cmd: rspec

After your main app containers have been brought up, run

docker-compose -f docker-compose-test.yml run tests

Audry

Just as a note, you need to include the service name in the docker-compose command. So, it would be docker-compose run web rspec. - docs.docker.com/compose/reference/run – Superclass

awesome, no problem! – Superclass

docker-compose run doesn't seem to have an -f flag, at least not in the latest version (18.03) – Volcano

@Volcano you're right, it's up a level on docker-compose itself – Audry