Manage local/private Golang packages and modules for docker builds

Question

I'm pretty new to golang and docker so after looking around in many threads and internet resources I ended up confused about how I should set things up.

My problem is that I'm facing a docker build error of a golang app because of a not found local dependency.

FYI (local machine): go version go1.13.4 linux/amd64

Here is the context of my problem:

I have a repository that will hold sources for simple micro-services (written in Go) that will be running in Kubernetes. So each folder in the repository represents a service. I also have a folder "tools" that regroups helper functions and ressources that many services share. My repository is not in any Go special paths, it's located in the root of one of my drives. So here is how my repo looks like:

.
├── service1
│   ├── main.go
│   ├── Dockerfile
│   ├── go.mod
│   └── go.sum
├── service2
├── service3
│   ...
├── serviceX
├── tools
│   ├── helpers.go
│   ├── ressources.go
│   ├── go.mod
└── └── go.sum

So each folder/service is a go module to handle its dependencies independently. The folder tools is a module as well. I initialized each module with go mod init FOLDER_NAME

In the service service1 I reference the module tools to use some functions. Here is how I implemented it : (src of ./service1/main.go)

package main

import (
    "fmt"

    st "../tools"

    // other modules imports
)

func main() {
    st.ExecHelperFunc()
    // http server inits
}

When I locally run service1's main.go or if I locally build it and run the binary afterwards the service works fine.

But when I attempt to build the dockerfile of service1 I get golang build error: build _/go/src/app/tools: cannot find module for path _/go/src/app/tools

Here is my dockerfile:

FROM golang:1.13 as builder
ENV GO111MODULE=on
WORKDIR /go/src/app
COPY ./tools ./tools
COPY ./service1 ./service1

WORKDIR /go/src/app/tools
RUN go mod download
WORKDIR /go/src/app/service1
RUN go mod download

WORKDIR /go/src/app

RUN go build -o server /go/src/app/service1/main.go

FROM centos:7
RUN yum -y update && yum clean all
COPY --from=builder /go/src/app/server .
EXPOSE 3000
CMD ["./server"]

So I dont understand why docker fails to build the service? Apparently he is not able to locate/identify the module tools but why? I tried many different setups for the dockerfile but I still couldn't figure it out.

Thank you in advance for your help.

UPDATE:

As suggested in an answer, changing the value of GO111MODULE in the dockerfile from on to auto didn't fix the build error but gets me a new one:

unexpected directory layout:
    import path: _/go/src/app/tools
    root: /go/src
    dir: /go/src/app/tools
    expand root: /go
    expand dir: /go/src/app/tools
    separator: /

Scapular

Answer 1

You should change your environment variable GO111MODULE as auto.

The final Dockerfile is:

FROM golang:1.13 as builder
ENV GO111MODULE=auto
WORKDIR /go/src/app
COPY ./tools ./tools
COPY ./service1 ./service1

WORKDIR /go/src/app/tools
RUN go mod download
WORKDIR /go/src/app/service1
RUN go mod download

WORKDIR /go/src/app

RUN go build -o server /go/src/app/service1/main.go

FROM centos:7
RUN yum -y update && yum clean all
COPY --from=builder /go/src/app/server .
EXPOSE 3000
CMD ["./server"]

UPDATE

In my opinion, if GO111MODULE is equal to 'auto', golang will disable the module features, and find the third-party package in GOPATH. Before compile the code, you should go get -u github.com/op/go-logging. I think it is not you want.

According to the example here, I updated my repo, here. You can try to build the code inside docker container. It can build succeed, but you should change the package name.

Crumpet

Hi ! thank you for your answer. Unfortunately, changing GO111MODULE to auto didn't make the docker build succeed. Now I have a new error still at go build line: unexpected directory layout: import path: _/go/src/app/tools root: /go/src dir: /go/src/app/tools expand root: /go expand dir: /go/src/app/tools separator: / – Scapular

Do you miss something important ? I created a demo here.(github.com/juxuny/go-build-trial). You just step into the root directory and run docker build -t go-build-demo -f service1/Dockerfile . – Crumpet

Thank you for all your effort and help ! I really appreciate it. I cloned your repo and could build the dockerfile with no errors. But the tool module was "naked" and the one I'm trying to build actually uses different public modules. So I added a simple logging module and tried to build again. It failed with the same error as mine. I will push my modifications in a branch if you allow me to – Scapular

Ok, you can push your modifications in a branch .let me see what is going wrong. – Crumpet

I forked your repo then created a PR to make things simple and clean ;) – Scapular

It seams to be no way to compile.Because you set GOPATH environment variable, you can build the code successfully on local machine. – Crumpet

Let us continue this discussion in chat. – Crumpet

Answer 2

For anyone facing a similar issue - I solved a similar problem using the vendor directory. Basically, you download the dependencies on the host by running go mod vendor and they will then get copied to the docker container automatically.

From the Dockerfile, you have to remove RUN go mod download, and you also need to modify the build command with the -mod=vendor flag.

To build the docker image, you need to run these commands

go mod vendor
docker build . -t image-name

I hope this will help someone. I have found this solution here: https://smartystreets.com/blog/2018/09/private-dependencies-in-docker-and-go/

Slung