Separating unit tests and integration tests in Go

Question

Is there an established best practice for separating unit tests and integration tests in GoLang (testify)? I have a mix of unit tests (which do not rely on any external resources and thus run really fast) and integration tests (which do rely on any external resources and thus run slower). So, I want to be able to control whether or not to include the integration tests when I say go test.

The most straight-forward technique would seem to be to define a -integrate flag in main:

var runIntegrationTests = flag.Bool("integration", false
    , "Run the integration tests (in addition to the unit tests)")

And then to add an if-statement to the top of every integration test:

if !*runIntegrationTests {
    this.T().Skip("To run this test, use: go test -integration")
}

Is this the best I can do? I searched the testify documentation to see if there is perhaps a naming convention or something that accomplishes this for me, but didn't find anything. Am I missing something?

Borodin

I think the stdlib uses -short to disable tests which hit the network (and other longrunning stuff too). Other wise your solution looks okay. – Laminated

-short is a good option, as is your custom build flags, but your flags need not be in main. if you define the var as var integration = flag.Bool("integration", true, "Enable integration testing.") outside of a function, the variable will show up in package scope, and the flag will work properly – Quijano

Answer 1

@Ainar-G suggests several great patterns to separate tests.

This set of Go practices from SoundCloud recommends using build tags (described in the "Build Constraints" section of the build package) to select which tests to run:

Write an integration_test.go, and give it a build tag of integration. Define (global) flags for things like service addresses and connect strings, and use them in your tests.

// +build integration

var fooAddr = flag.String(...)

func TestToo(t *testing.T) {
    f, err := foo.Connect(*fooAddr)
    // ...
}

go test takes build tags just like go build, so you can call go test -tags=integration. It also synthesizes a package main which calls flag.Parse, so any flags declared and visible will be processed and available to your tests.

As a similar option, you could also have integration tests run by default by using a build condition // +build !unit, and then disable them on demand by running go test -tags=unit.

@adamc comments:

For anyone else attempting to use build tags, it's important that the // +build test comment is the first line in your file, and that you include a blank line after the comment, otherwise the -tags command will ignore the directive.

Also, the tag used in the build comment cannot have a dash, although underscores are allowed. For example, // +build unit-tests will not work, whereas // +build unit_tests will.

Containment

I have been using this for some time now and it is by far the most logical and simple approach. – Enclasp

if you have unit tests in same package, you need set // + build unit in units tests and use -tag unit for run the tests – Touchhole

It seems the latest version of go have deprecate the tags functionality. – Apologist

@Apologist can you provide a link or more details about deprecation of tags? I didn't find such information. I'm using tags with the go1.8 for the way described in the answer and also for mocking types and functions in tests. It is good alternative to interfaces I think. – Horselaugh

For anyone else attempting to use build tags, it's important that the // +build test comment is the first line in your file, and that you include a blank line after the comment, otherwise the -tags command will ignore the directive. Also, the tag used in the build comment cannot have a dash, although underscores are allowed. For example, // +build unit-tests will not work, whereas // +build unit_tests will – Stocktaking

How to handle wildcards? go test -tags=integration ./... doesnt work, it ignores the tag – Knowle

I'm working with go1.11.4. For separate my unit test and integration test in same package,I have to add different tags for unit_test.go and integration_test.go at the same time,otherwise go test -tag=integration will run both test case. – Mohun

why have you added an exclamation mark in "+build !unit" ? – Albescent

Decided to investigate this approach of using the -short tag + Makefile. I tried build tags approach earlier with VSCode and experienced linting errors from gopls. Checkout github.com/golang/go/issues/29202. It appears to be a know issue and it was taking up too much of my time trying to tweak gopls settings to recognise build flags. Furthermore, these settings are global for all go projects. Managed to get linting errors down to 1 problem with go.mod not recognising a package with build flags name and then gave up. Ended up using the -short tag approach below. – Untutored

You need to be careful with multiple build tags and their syntax. I encountered unexpected issues when I had a build tag like // +build unit, !int, !dark. Firstly, don't use comma followed by spaces which leads to unpredictable behavior. Changing the above to // +build unit,!int,!dark lead to expected behavior. Furthermore, it was simpler to just have // +build unit. – Germainegerman

I m running go 1.15. I followed the instruction above, so I tried to run go test -tags=integration but it's not working. Moreover, the go test help does not mention any parameter -tag. I named my integration test with TestIntegration_what_it_test and run go test -run TestIntegration and it works! – Bicarbonate

How can we work around this when we want to use an IDE, say VSCode, and simply right click the root folder and run all tests, but only want to run the unit tests? – Beslobber

@Beslobber well if your IDE doesn't have the feature, you can't use the feature. Can you set up a custom task that VSCode will run for you from a right-click menu? – Containment

Answer 2

To elaborate on my comment to @Ainar-G's excellent answer, over the past year I have been using the combination of -short with Integration naming convention to achieve the best of both worlds.

Unit and Integration tests harmony, in the same file

Build flags previously forced me to have multiple files (services_test.go, services_integration_test.go, etc).

Instead, take this example below where the first two are unit tests and I have an integration test at the end:

package services

import "testing"

func TestServiceFunc(t *testing.T) {
    t.Parallel()
    ...
}

func TestInvalidServiceFunc3(t *testing.T) {
    t.Parallel()
    ...
}

func TestPostgresVersionIntegration(t *testing.T) {
    if testing.Short() {
        t.Skip("skipping integration test")
    }
    ...
}

Notice the last test has the convention of:

1. using Integration in the test name.
2. checking if running under -short flag directive.

Basically, the spec goes: "write all tests normally. if it is a long-running tests, or an integration test, follow this naming convention and check for -short to be nice to your peers."

Run only Unit tests:

go test -v -short

this provides you with a nice set of messages like:

=== RUN   TestPostgresVersionIntegration
--- SKIP: TestPostgresVersionIntegration (0.00s)
        service_test.go:138: skipping integration test

Run Integration Tests only:

go test -run Integration

This runs only the integration tests. Useful for smoke testing canaries in production.

Obviously the downside to this approach is if anyone runs go test, without the -short flag, it will default to run all tests - unit and integration tests.

In reality, if your project is large enough to have unit and integration tests, then you most likely are using a Makefile where you can have simple directives to use go test -short in it. Or, just put it in your README.md file and call it the day.

Commie

love the simplicity – Sells

Do you create separate package for such test to access only the public parts of package? Or all mixed? – Lightfingered

@Lightfingered Well, that's OT from the answer. But personally, I prefer both: a different package name for the tests so I can import my package and test against it, which ends up showing me what my API looks like to others. I then follow up with any remaining logic that needs to be covered as internal test package names. – Commie

@Commie Thanks for the answer! So as I understand here package services contains an integration test sute, so to test the APIfo the package as a blackbox we should name it another way package services_integration_test it will not give us a chance to work with internal structures. So the package for unit tests (accessing internals) should be named package services. Is it so? – Lightfingered

That's correct, yes. Here's a clean example of how I do it: github.com/eduncan911/podcast (notice 100% code coverage, using Examples) – Commie

@Commie but here in your reply "Unit and Integration tests harmony, in the same file" it looks like you mixing tests for internals with integration tests in the same scope of package services. If it so you can accidentially access internals. – Lightfingered

Correct. That podcast repo is a different concept I have to blog about. But you were asking about different packages, which you can see in the examples.go files that they are different packages. – Commie

Good one. For running only Integration Tests, I had to use go regex go test -v -run ".Integration" ./... here go regex and a good example – Vickievicksburg

Decided to investigate this approach of using the -short tag + Makefile. I tried build tags approach earlier with VSCode and experienced linting errors from gopls. Checkout github.com/golang/go/issues/29202. It appears to be a know issue and it was taking up too much of my time trying to tweak gopls settings to recognise build flags. Furthermore, these settings are global for all go projects. Managed to get linting errors down to 1 problem with go.mod not recognising a package with build flags name and then gave up. So using this approach saving frustration for other devs on proj. – Untutored

@Untutored yep, I strive for zero linting issues on strict and simple Makefiles. That's why I use this approach. :-) – Commie

Answer 3

I see three possible solutions. The first is to use the short mode for unit tests. So you would use go test -short with unit tests and the same but without the -short flag to run your integration tests as well. The standard library uses the short mode to either skip long-running tests, or make them run faster by providing simpler data.

The second is to use a convention and call your tests either TestUnitFoo or TestIntegrationFoo and then use the -run testing flag to denote which tests to run. So you would use go test -run 'Unit' for unit tests and go test -run 'Integration' for integration tests.

The third option is to use an environment variable, and get it in your tests setup with os.Getenv. Then you would use simple go test for unit tests and FOO_TEST_INTEGRATION=true go test for integration tests.

I personally would prefer the -short solution since it's simpler and is used in the standard library, so it seems like it's a de facto way of separating/simplifying long-running tests. But the -run and os.Getenv solutions offer more flexibility (more caution is required as well, since regexps are involved with -run).

Virile

note that community test runners (e.g. Tester-Go) common to IDEs (Atom, Sublime, etc) have the built-in option to run with -short flag, along with -coverage and others. therefore, I use a combination of both Integration in the test name, along with if testing.Short() checks within those tests. it allows me to have the best of both worlds: run with -short within IDEs, and explicitly run only integration tests with go test -run "Integration" – Commie

Answer 4

I was trying to find a solution for the same recently. These were my criteria:

• The solution must be universal
• No separate package for integration tests
• The separation should be complete (I should be able to run integration tests only)
• No special naming convention for integration tests
• It should work well without additional tooling

The aforementioned solutions (custom flag, custom build tag, environment variables) did not really satisfy all the above criteria, so after a little digging and playing I came up with this solution:

package main

import (
    "flag"
    "regexp"
    "testing"
)

func TestIntegration(t *testing.T) {
    if m := flag.Lookup("test.run").Value.String(); m == "" || !regexp.MustCompile(m).MatchString(t.Name()) {
        t.Skip("skipping as execution was not requested explicitly using go test -run")
    }

    t.Parallel()

    t.Run("HelloWorld", testHelloWorld)
    t.Run("SayHello", testSayHello)
}

The implementation is straightforward and minimal. Although it requires a simple convention for tests, but it's less error prone. Further improvement could be exporting the code to a helper function.

Usage

Run integration tests only across all packages in a project:

go test -v ./... -run ^TestIntegration$

Run all tests (regular and integration):

go test -v ./... -run .\*

Run only regular tests:

go test -v ./...

This solution works well without tooling, but a Makefile or some aliases can make it easier to user. It can also be easily integrated into any IDE that supports running go tests.

The full example can be found here: https://github.com/sagikazarmark/modern-go-application

Gainer

Unless I have misunderstood, with this method you can only have one function named TestIntegration in a single file – Normannormand

Actually, you can have one function per package. You can create multiple functions with different names (eg. TestIntegrationFoo). You would also have to remove the dollar sign from the regex. – Gainer

Answer 5

I encourage you to look at Peter Bourgons approach, it is simple and avoids some problems with the advice in the other answers: https://peter.bourgon.org/blog/2021/04/02/dont-use-build-tags-for-integration-tests.html

Precipitate

Answer 6

There are many downsides to using build tags, short mode or flags, see here.

I would recommend using environment variables with a test helper that can be imported into individual packages:

func IntegrationTest(t *testing.T) {
    t.Helper()
    if os.Getenv("INTEGRATION") == "" {
        t.Skip("skipping integration tests, set environment variable INTEGRATION")
    }
}

In your tests you can now easily call this at the start of your test function:

func TestPostgresQuery(t *testing.T) {
    IntegrationTest(t)
    // ...
}

Why I would not recommend using either -short or flags:

Someone who checks out your repository for the first time should be able to run go test ./... and all tests are passing which is often not the case if this relies on external dependencies.

The problem with the flag package is that it will work until you have integration tests across different packages and some will run flag.Parse() and some will not which will lead to an error like this:

go test ./... -integration
flag provided but not defined: -integration
Usage of /tmp/go-build3903398677/b001/foo.test:

Environment variables appear to be the most flexible, robust and require the least amount of code with no visible downsides.

Eustis

I think flag.Parse() is not needed inside the test files and can be omitted. See source. So I don't see the problem using the flag package. – Conditional

@OsmosisD.Jones It will be a problem once some of your packages have integration tests and others do not. – Eustis

Answer 7

I think all three options presented have some merit.

I agree that build tags can get quite messy, but I think both t.Short() and environment variables are valid options.

I think the one big downside to using environment variabels over t.Short(). Is that you can no longer run a single integration test from IDE testing like VSC. Visual Studio Code will render a "run test" button above a test function and if that is gated behind an environment variable that functionality will no longer work.

This could be solved by using a negative environment variable like SKIP_INTEGRATION_TESTS, but at that point you might as well use t.Short() as it seems like a cleaner solution in Go.

I think it largely depends on what kind of integration tests you have and what you want the default behaviour to be of a new developer working on this project. If everything is automated, including dependency setup / seeding (using testcontainers/dockertest or anything like that, you wouldn't have failing tests on go test ./... they would just take longer than go test ./... -short as they run the full suite. If you have manual dependency setup steps than maybe it is better to use environment variables.

TLDR: Standard development answer: "It Depends"

Kickapoo