Manually fetch dependencies from go.mod?

Question

I'm using go 1.11 with module support. I understand that the go tool now installs dependencies automatically on build/install. I also understand the reasoning.

I'm using docker to build my binaries. In many other ecosystems its common to copy over your dependency manifest (package.json, requirements.txt, etc) and install dependencies as a separate stage from build. This takes advantage of docker's layer caching, and makes rebuilds much faster since generally code changes vastly outnumber dependency changes.

I was wondering if vgo has any way to do this?

Koralie

Answer 1

It was an issue #26610, which is fixed now.

So now you can just use:

go mod download

For this to work you need just the go.mod / go.sum files.

For example, here's how to have a cached multistage Docker build: (source)

FROM golang:1.17-alpine as builder
RUN apk --no-cache add ca-certificates git
WORKDIR /build

# Fetch dependencies
COPY go.mod go.sum ./
RUN go mod download

# Build
COPY . ./
RUN CGO_ENABLED=0 go build

# Create final image
FROM alpine
WORKDIR /
COPY --from=builder /build/myapp .
EXPOSE 8080
CMD ["./myapp"]

---

Also see the article Containerize Your Go Developer Environment – Part 2, which describes how to leverage the Go compiler cache to speed up builds even further.

Stearn

Not completely. Commands like go get -t -x will still download more stuff. Even after a COPY . ./, go get -t -x will download even more stuff, despite a tidy'd go.mod. Where is the command to download everything from go.mod? – Roussel

go mod download downloads exactly what's listed in go.mod, which should be enough to build and test the project. go get primary purpose is to add new dependencies or versions to the project. As a side effect it also populates the local cache. – Stearn

Answer 2

You may use the go mod vendor command which will create a vendor folder in the main module's root folder, and copy all dependencies into it. After this you may pass the -mod=vendor param to the go tool, and then dependencies from the vendor folder will be used to build / compile / test your app.

So what you may do to speed up your builds:

1. Run the go mod vendor command to have an actual version of your dependencies.
2. Save / cache this vendor folder.
3. During builds, restore this vendor folder, and build / install your app by passing the -mod=vendor argument to the go tool, so no dependencies will be downloaded, but the content of the vendor folder will be used.

Quoting from go help mod:

Modules and vendoring

When using modules, the go command completely ignores vendor directories.

By default, the go command satisfies dependencies by downloading modules from their sources and using those downloaded copies (after verification, as described in the previous section). To allow interoperation with older versions of Go, or to ensure that all files used for a build are stored together in a single file tree, 'go mod vendor' creates a directory named vendor in the root directory of the main module and stores there all the packages from dependency modules that are needed to support builds and tests of packages in the main module.

To build using the main module's top-level vendor directory to satisfy dependencies (disabling use of the usual network sources and local caches), use 'go build -mod=vendor'. Note that only the main module's top-level vendor directory is used; vendor directories in other locations are still ignored.

Lax

Answer 3

I wanted to re-download all the dependencies using go mod, this is what I did:

1. Go to your GOROOT
2. sudo rm -rf pkg/mod/
3. Go to the directory where the go.mod file exists
4. go mod download

Wirer

Answer 4

You can use a package manager, There are many of them like dep, glide, and govendor. dep is newer and is going to be integrated into go toolchain as official dependency management tool.

We also make docker images for go applications and we use dind to make those images and we prepared a CI/CD image with all dependencies preinstalled to make the builds faster. Though, it took a little bit of scripting to glue everything together.

Moreover, layering up the dependencies could result in big size of docker images. I suggest try multi-stage builds which could help making images super lite.

Glick

This is more a question of whether the new modules system that landed in 1.11 has a built-in way to do this. If you check the dep repo, you'll see its actually being deprecated for modules :( – Koralie