Testing os.Exit scenarios in Go with coverage information (coveralls.io/Goveralls)

Question

This question: How to test os.exit scenarios in Go (and the highest voted answer therein) sets out how to test os.Exit() scenarios within go. As os.Exit() cannot easily be intercepted, the method used is to reinvoke the binary and check the exit value. This method is described at slide 23 on this presentation by Andrew Gerrand (one of the core members of the Go team); the code is very simple and is reproduced in full below.

The relevant test and main files look like this (note that this pair of files alone is an MVCE):

package foo

import (
    "os"
    "os/exec"
    "testing"
)

func TestCrasher(t *testing.T) {
    if os.Getenv("BE_CRASHER") == "1" {
        Crasher() // This causes os.Exit(1) to be called
        return
    }
    cmd := exec.Command(os.Args[0], "-test.run=TestCrasher")
    cmd.Env = append(os.Environ(), "BE_CRASHER=1")
    err := cmd.Run()
    if e, ok := err.(*exec.ExitError); ok && !e.Success() {
        fmt.Printf("Error is %v\n", e)
    return
    }
    t.Fatalf("process ran with err %v, want exit status 1", err)
}

and

package foo

import (
    "fmt"
    "os"
)

// Coverage testing thinks (incorrectly) that the func below is
// never being called
func Crasher() {
    fmt.Println("Going down in flames!")
    os.Exit(1)
}

However, this method appears to suffer certain limitations:

1. Coverage testing with goveralls / coveralls.io does not work - see for instance the example here (the same code as above but put into github for your convenience) which produces the coverage test here, i.e. it does not record the test functions being run. NOTE that you don't need to those links to answer the question - the above example will work fine - they are just there to show what happens if you put the above into github, and take it all the way through travis to coveralls.io

2. Rerunning the test binary appears fragile.

Specifically, as requested, here is a screenshot (rather than a link) for the coverage failure; the red shading indicates that as far as coveralls.io is concerned, Crasher() is not being called.

Is there a way around this? Particularly the first point.

At a golang level the problem is this:

• The Goveralls framework runs go test -cover ..., which invokes the test above.

• The test above calls exec.Command / .Run without -cover in the OS arguments

• Unconditionally putting -cover etc. in the argument list is unattractive as it would then run a coverage test (as the subprocess) within a non-coverage test, and parsing the argument list for the presence of -cover etc. seems a heavy duty solution.

• Even if I put -cover etc. in the argument list, my understanding is that I'd then have two coverage outputs written to the same file, which isn't going to work - these would need merging somehow. The closest I've got to that is this golang issue.

---

Summary

What I am after is a simple way to run go coverage testing (preferably via travis, goveralls, and coveralls.io), where it is possible to both test cases where the tested routine exits with OS.exit(), and where the coverage of that test is noted. I'd quite like it to use the re-exec method above (if that can be made to work) if that can be made to work.

The solution should show coverage testing of Crasher(). Excluding Crasher() from coverage testing is not an option, as in the real world what I am trying to do is test a more complex function, where somewhere deep within, under certain conditions, it calls e.g. log.Fatalf(); what I am coverage testing is that the tests for those conditions works properly.

Noahnoak

Would excluding the fucntion from the coverage test be ok ? (I assume not, but just in case) – Juno

@Tensibai, no the result that I want to achieve is to ensure I get a full coverage test. In reality in any case what I am checking is coverage for particular lines (and their checks) within existing functions which do (e.g.) log.Fatalf, so excluding the entire function would be substantially worse than the current position. – Noahnoak

My ruby feeling would be to stub the os.Exit function and test a return value instead of an exit... but that's probably out of scope (digging into GO coverage) – Juno

@Juno I don't think you can do that where (e.g.) os.Exit() is called by log.Fatalf() etc., as you'd have to make the return value pass all the way up. That would be substantial changes to the go source code, as well as to the tested application. Making os.Exit() panic and using recover might help slightly, but would be suboptimal if recover was being used internally. – Noahnoak

After a bunch of research, best approach seems to refactor code, using custom functions for exit and log, so you can stub them in test. Sources This answer and for log.Fatalf this one. Sorry but even looking to interfaces from here I can't see a proper solution as cover "count" in the current process if statement was called source here – Juno

FWIW I even tried with a TestMain, this simplify a little the test case for the Crash one, but this doesn't help as relaunching is needed and the "visit" interface is never called within the current test process. – Juno

@Juno I can't see how you can make that (refactoring log.Fatalf work) - you can't call t. functions (pass/fail) within other threads than the main test thread. I think you'd have to do pretty serious code surgery, which rather gets you away from a coverage test. I suspect the answer is somehow to detect the coverage test, then run the test case with -cover and somehow merge the outputs - i.e. run the visit interface twice. goveralls already does some merging. – Noahnoak

The main idea is to replace native log.x calls by customs Mylog.x call, then you can override (stub/mock) Mylog.x in your test cases to return without exiting the test process. I may write an answer in this way if you think it could be of help, but that would be copying a bunch from the other answers. – Juno

The point is rewriting your code to never call log.Fatalf directly but always through a wrapper you can stub to avoid the exit and just return. – Juno

TL;DR: go library isn't quite designed for testability at the moment. The cleanest approach would be to tweak your version of Go library to fix the issue. If you control your test environment, this will not be a problem. You don't need the testability improvement patches on production machines. – Bullbat

Some one had a similar issue check this answer if you think it is oke i will put an answer #39691009 – Meed

Answer 1

With a slight refactoring, you may easily achieve 100% coverage.

foo/bar.go:

package foo

import (
    "fmt"
    "os"
)

var osExit = os.Exit

func Crasher() {
    fmt.Println("Going down in flames!")
    osExit(1)
}

And the testing code: foo/bar_test.go:

package foo

import "testing"

func TestCrasher(t *testing.T) {
    // Save current function and restore at the end:
    oldOsExit := osExit
    defer func() { osExit = oldOsExit }()

    var got int
    myExit := func(code int) {
        got = code
    }

    osExit = myExit
    Crasher()
    if exp := 1; got != exp {
        t.Errorf("Expected exit code: %d, got: %d", exp, got)
    }
}

Running go test -cover:

Going down in flames!
PASS
coverage: 100.0% of statements
ok      foo        0.002s

Yes, you might say this works if os.Exit() is called explicitly, but what if os.Exit() is called by someone else, e.g. log.Fatalf()?

The same technique works there too, you just have to switch log.Fatalf() instead of os.Exit(), e.g.:

Relevant part of foo/bar.go:

var logFatalf = log.Fatalf

func Crasher() {
    fmt.Println("Going down in flames!")
    logFatalf("Exiting with code: %d", 1)
}

And the testing code: TestCrasher() in foo/bar_test.go:

func TestCrasher(t *testing.T) {
    // Save current function and restore at the end:
    oldLogFatalf := logFatalf
    defer func() { logFatalf = oldLogFatalf }()

    var gotFormat string
    var gotV []interface{}
    myFatalf := func(format string, v ...interface{}) {
        gotFormat, gotV = format, v
    }

    logFatalf = myFatalf
    Crasher()
    expFormat, expV := "Exiting with code: %d", []interface{}{1}
    if gotFormat != expFormat || !reflect.DeepEqual(gotV, expV) {
        t.Error("Something went wrong")
    }
}

Running go test -cover:

Going down in flames!
PASS
coverage: 100.0% of statements
ok      foo     0.002s

Objurgate

Interesting. However osExit no longer stops the execution flow (as os.Exit() did), which may give false negatives (i.e. things shown as covered in the coverage test which wouldn't normally be) and other issues. I wonder whether myFatalF could instead the failure into a channel (then block), Crasher() could be in its own goroutine, and the main goroutine could call t.Fail() on reception of an error on the channel. It's a shame this can't be done without rewriting the code though. – Noahnoak

@Noahnoak I agree, if there are statements after an os.Exit() call, normally they wouldn't get executed, but in this test they would be. This can be avoided by not writing such code (it wouldn't make sense anyway). – Objurgate

@izca The problem is code like if foo.IsBroken() {log.Fatalf("The foo is broken")}. If the code continues to executed after that (after the if block), you might will get a panic or similar because, well foo is broken (or whatever). – Noahnoak

@Noahnoak That could be "solved" by adding a return statement after the logFatal() even though normally it wouldn't be required (only for making the code testable). – Objurgate

Answer 2

Interfaces and mocks

Using Go interfaces possible to create mock-able compositions. A type could have interfaces as bound dependencies. These dependencies could be easily substituted with mocks appropriate to the interfaces.

type Exiter interface {
    Exit(int)
}

type osExit struct {}

func (o* osExit) Exit (code int) {
    os.Exit(code)
}

type Crasher struct {
    Exiter
}

func (c *Crasher) Crash() {
    fmt.Println("Going down in flames!")
    c.Exit(1)
}

Testing

type MockOsExit struct {
    ExitCode int
}

func (m *MockOsExit) Exit(code int){
    m.ExitCode = code
}

func TestCrasher(t *testing.T) {
    crasher := &Crasher{&MockOsExit{}}
    crasher.Crash() // This causes os.Exit(1) to be called
    f := crasher.Exiter.(*MockOsExit)
    if f.ExitCode == 1 {
        fmt.Printf("Error code is %d\n", f.ExitCode)
        return
    }
    t.Fatalf("Process ran with err code %d, want exit status 1", f.ExitCode)
}

Disadvantages

Original Exit method still won't be tested so it should be responsible only for exit, nothing more.

Functions are first class citizens

Parameter dependency

Functions are first class citizens in Go. A lot of operations are allowed with functions so we can do some tricks with functions directly.

Using 'pass as parameter' operation we can do a dependency injection:

type osExit func(code int)

func Crasher(os_exit osExit) {
    fmt.Println("Going down in flames!")
    os_exit(1)
}

Testing:

var exit_code int 
func os_exit_mock(code int) {
     exit_code = code
}

func TestCrasher(t *testing.T) {

    Crasher(os_exit_mock) // This causes os.Exit(1) to be called
    if exit_code == 1 {
        fmt.Printf("Error code is %d\n", exit_code)
        return
    }
    t.Fatalf("Process ran with err code %v, want exit status 1", exit_code)
}

Disadvantages

You must pass a dependency as a parameter. If you have many dependencies a length of params list could be huge.

Variable substitution

Actually it is possible to do it using "assign to variable" operation without explicit passing a function as a parameter.

var osExit = os.Exit

func Crasher() {
    fmt.Println("Going down in flames!")
    osExit(1)
}

Testing

var exit_code int
func osExitMock(code int) {
    exit_code = code
}

func TestCrasher(t *testing.T) {
    origOsExit := osExit
    osExit = osExitMock
    // Don't forget to switch functions back!
    defer func() { osExit = origOsExit }()

    Crasher()
    if exit_code != 1 {
        t.Fatalf("Process ran with err code %v, want exit status 1", exit_code)
    }
}

disadvantages

It is implicit and easy to crash.

Design notes

If you plan to declare some logic below Exit an exit logic must be isolated with else block or extra return after exit because mock won't stop execution.

func (c *Crasher) Crash() {
    if SomeCondition == true {
        fmt.Println("Going down in flames!")
        c.Exit(1)  // Exit in real situation, invoke mock when testing
    } else {
        DoSomeOtherStuff()
    }

}

Extinguish

A first-class function could also be used. NewCrasher(exit func(code int)) func() {exit()} – Optical