fixing versions of tools used by go

Question

I am looking to create reproducible builds with go. For individual projects we are using glide.

So for example I use:

glide get github.com/stretchr/testify

to fix the version of the "testify" package. This does not work for tools however. For example:

glide install github.com/tebeka/go2xunit

returns success but does not actually install go2xunit so I have to use:

go get github.com/tebeka/go2xunit

which installs go2xunit to $GOPATH/bin.

Q How can I fix the version of tools like go2xunit?

I also note that glide says use dep instead and dep says golang has diverged from its implementation and will probably end up using something based on vgo. There are a plethora of dependency management tools for go perhaps one of the less well known ones supports this?

In case its relevant I'm using go 1.7.4 as provided by Debian9.

Stonge

I'm not sure about other go dependency tools, but dep will also not install binaries. There's an issue about it already open. What I do, is keep a separate gotools GOPATH where I keep all my go based tools. Especially ones that vscode uses. This keeps my project GOPATH from being "contaminated" with external things. – Hackamore

Its not clear if vgo will solve this either - go.googlesource.com/proposal/+/master/design/… – Stonge

I just played around with the go mod that was added in go1.11. It like the other dependency tools don't install packages, they just download them. So you won't be able to track a build dependency tool like go2xunit. – Hackamore

Doh! Reproducible builds are still a way off then! Thanks for the insight though. – Stonge

I'm sure this must be a known issue but having failed to find it I've raised it here - github.com/golang/go/issues/28512 – Stonge

For go mod see #52428730 – Peadar

Answer 1

The solution for go1.11 using go modules is to create a fake tools package. You create a tools.go file like the following:

// +build tools

package tools

import (
        _ "github.com/tebeka/go2xunit"
)

+build tools is a magic comment which prevents the package being built.

>go mod init tools

Will create a go.mod file for the fake tools package

>go install github.com/tebeka/go2xunit

Will install go2xunit and update go.mod as follows.

module tools

require github.com/tebeka/go2xunit v1.4.8 // indirect

Now if you run go install github.com/tebeka/go2xunit in the future (for a clean build say) its version will be fixed to v1.4 by the go.mod

---

For versions of go before 1.11 the tool to use is retool. It works like this:

bootstrap:

go get github.com/twitchtv/retool

add tool:

retool add github.com/jteeuwen/go-bindata/go-bindata origin/master

use tool:

retool do go-bindata -pkg testdata -o ./testdata/testdata.go ./testdata/data.json

Adding support for this may be on the roadmap to target go 1.12 (https://github.com/golang/go/issues/27653)

Stonge

Note that this answer is essentially a duplicate of the answer to #52428730 except that this also covers older versions of go – Stonge

Answer 2

I did this very similarly, but just different enough that I think it's worth sharing again:

If you get an error

I was not seeing the dependency that I wanted added to the go.mod and I was getting this error:

tools/tools.go:6:5: import "github.com/UnnoTed/fileb0x" is a program, not an importable package

(fileb0x is the thing I'm trying to add)

I'm not 100% clear on the sequence of events that fixed it, but I did all of these things:

Using a "tools" package

I made a tools directory:

mkdir -p tools

I put the tools package inside of it (as mentioned above):

// +build tools

package tools

import (
    _ "github.com/UnnoTed/fileb0x"
)

Note that the tag is mostly not important. You could use foo:

// +build foo

However, you cannot use ignore. That's a special predefined tag.

// +build ignore

// NO NO NO NO NO
// `ignore` is a special keyword which (surprise) will cause
// the file to be ignore, even for dependencies

Updating go.mod

The best way is probably to run go mod tidy:

go mod tidy

However, before I did that I ran a number of commands trying to figure out which one would cause it to go into go.mod:

go install github.com/UnnoTed/fileb0x # didn't seem to do the trick
go get
go generate ./...
go build ./...
go install ./...
go mod vendor

Later I did a git reset and rm -rf ~/go/pkg/mod; mkdir ~/go/pkg/mod and found that go mod tidy did well enough on its own.

vendoring

In order to actually take advantage of the modules cache in a project you need to copy-in the source code

go mod vendor

That will grab all dependencies from go.mod

You also need to change nearly all of your go commands to use -mod=vendor in any Makefiles, Dockerfiles or other scripts.

go fmt -mod=vendor ./... # has a bug which should be fixed in go1.15
go generate -mod=vendor ./...
go build -mod=vendor ./...

That includes go build, go get, go install, and any go run called by go generate (and even the go generate itself)

//go:generate go run -mod=vendor github.com/UnnoTed/fileb0x b0x.toml
package main

// ...

Balaam