Pre-compiling Golang project dependencies to cache

Question

In short, my current use-case involves dynamically creating a Golang plugin inside a Docker container. The compilation involves some new input from the user (which is why it is not compiled beforehand), but the dependencies are static, and won't change.

Currently, the full compilation is done from scratch inside the Docker container (though go mod download is used to reduce the time by a bit). I noticed that the go build command ends up compiling a lot of the dependencies, which adds a non-trivial amount of time for the plugin compilation, which affects the usability of my application.

Is there a Go supported method or command to read through the go.mod file and populate the GOCACHE directory? With such a command, I would run it in my Dockerfile itself, causing the Docker image to contain the cache with all the compiled build dependencies.

What I've tried:

1. go mod download: This only downloads the dependencies; it does not compile them.
2. I do have this working with a temporary workaround: I created a barebones main.go that imports all the dependencies, and run go build within my Dockerfile to populate the cache. As mentioned, this does solve my problem, but it feels like a bit of a hack. Additionally, if the dependencies change in the future, it requires someone to change this as well, which isn't ideal.
3. A lot of the answers I saw online for this involve CI/CD. With CI/CD, the container just has a partition mounted to the host, which contains a cache that is persisted after runs. This does not solve my immediate problem, which is for building the container itself.

Villalpando

Another hack you can try is mounting your local cache folder into the container so it can be reused. – Bifocals

Related issues: github.com/golang/go/issues/45474 and github.com/golang/go/issues/27719 – Overijssel

workaround #2 doesn't seem too hacky. I wouldn't lose sleep over it – Maness

since u mentioned plugin dependencies are static (not dependent on the user input). it is likely that plugin (template) code may be copied to docker image followed by go build when building the image. later (template) source may be modified as per user input and recompiled. – Prem

Answer 1

Since the issues (1, 2) in the golang repository are still open, all that we can do is "hacking", I think. So, we can do something like that for the dependencies caching and pre-compilation as a separate docker layer:

FROM golang:1.19-buster as builder

COPY ./go.* /src/

WORKDIR /src

# burn the modules cache
RUN set -x \
    # cache go dependencies
    && go mod download \
    # pre-compile common dependencies
    && mkdir /tmp/gobin \
    && for p in $(go list -m -f '{{if and (not .Indirect) (not .Main)}}{{.Path}}/...@{{.Version}}{{end}}' all); do \
      GOBIN=/tmp/gobin go install $p; \
    done \
    && rm -r /tmp/gobin

COPY . /src

RUN go build ...

Without this trick, the building (docker buildx build --platform linux/amd64,linux/arm64 ...) takes about 9 minutes, with it ~6 minutes (profit 30%). But the pre-compilation step becomes longer by ~40%.

Selfabasement