Dockerfile: Docker build can't download packages: centos->yum, debian/ubuntu->apt-get behind intranet

Question

PROBLEM: Any build, with a Dockerfile depending on centos, ubuntu or debian fails to build.

ENVIRONMENT: I have a Mac OS X, running VMWare with a guest Ubuntu 14.04, running Docker:

mdesales@ubuntu ~ $ sudo docker version
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070

BEHAVIOR: Using "docker build" fails to download packages. Here's an example of such Dockerfile: https://github.com/Krijger/docker-cookbooks/blob/master/jdk8-oracle/Dockerfile, https://github.com/ottenhoff/centos-java/blob/master/Dockerfile

I know that we can run a container with --dns, but this is during the build time.

• CENTOS

  FROM centos RUN yum install a b c

• UBUNTU

  FROM ubuntu RUN apt-get install a b c

Users have reported that it might be problems with DNS configuration, others and the configuration has the Google's DNS servers commented out.

Step 2 : RUN yum install -y curl; yum upgrade -y; yum update -y;  yum clean all
 ---> Running in 5f11b65c87b8
Loaded plugins: fastestmirror
Couldn't resolve host 'mirrorlist.centos.org

Still the problem persisted... So, most users on #docker@Freenode mentioned that it might be a problem with the DNS configuration... So here's my Ubuntu:

$ sudo cat /etc/resolv.conf 
nameserver 127.0.1.1
search localdomain

I tried changing that, same problem...

Cullum

Answer 1

PROBLEM

Talking to some developers at #docker@freenode, the problem was clear to everyone: DNS and the environment. The build works just fine at a regular Internet connection at home.

SOLUTION:

This problem occurs in an environment that has a private DNS server, or the network blocks the Google's DNS servers. Even if the docker container can ping 8.8.8.8, the build still needs to have access to the same private DNS server behind your firewall or Data Center.

Start the Docker daemon with the --dns switch to point to your private DNS server, just like your host OS is configured. That was found by trial and error.

Details

My MAC OS X, host OS, had a different DNS configured on my /etc/resolv.conf:

mdesales@Marcello-Work ~ (mac) $ cat /etc/resolv.conf
search corp.my-private-company.net
nameserver 172.18.20.13
nameserver 172.20.100.29

My host might be dropping the packets to the Google's IP address 8.8.8.8 while building... I just took those 2 IP addresses and placed under the Ubuntu's docker daemon configuration:

mdesales@ubuntu ~ $ cat /etc/default/docker
...
...
# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 172.18.20.13 --dns 172.20.100.29 --dns 8.8.8.8"
...

The build now works as expected!

$ sudo ./build.sh 
Sending build context to Docker daemon 7.168 kB
Sending build context to Docker daemon 
Step 0 : FROM centos
 ---> b157b77b1a65
Step 1 : MAINTAINER [email protected]
 ---> Running in 49bc6e233e4c
 ---> 2a380810ffda
Removing intermediate container 49bc6e233e4c
Step 2 : RUN yum install -y curl; yum upgrade -y; yum update -y;  yum clean all
 ---> Running in 5f11b65c87b8
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: mirror.supremebytes.com
 * extras: centos.mirror.ndchost.com
 * updates: mirrors.tummy.com
Resolving Dependencies
--> Running transaction check
---> Package systemd.x86_64 0:208-11.el7 will be updated
---> Package systemd.x86_64 0:208-11.el7_0.2 will be an update
---> Package systemd-libs.x86_64 0:208-11.el7 will be updated
---> Package systemd-libs.x86_64 0:208-11.el7_0.2 will be an update
--> Finished Dependency Resolution

Thanks to @BrianF and others who helped in the IRC channel!

Permanent VM Solution - UPDATE JULY 2, 2015

We now have GitHub Enterprise and CoreOS Enterprise Docker Registry in the mix... So, it was important for me to add the corporate DNS servers from the HOST machine in order to get the VM also to work.

Replacing the /etc/resolv.conf from the guest OS with the Host's /etc/resolv.conf also resolved the problem! Docker 1.7.0. I just created a new VM using Ubuntu 15.04 on VMWare Fusion and I had this problem again...

/etc/resolv.conf BEFORE

~/dev/github/public/stackedit on ⭠ master ⌚ 20:31:02
$ cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search localdomain

/etc/resolv.conf AFTER*

~/dev/github/public/stackedit on ⭠ master ⌚ 20:56:09
$ cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
search corp.mycompany.net
nameserver 10.180.194.35
nameserver 10.180.194.36
nameserver 192.168.1.1

Cullum

On Ubuntu 14.04, the systemd's file is /lib/systemd/system/docker.service Add the same line to the EXEC instruction. – Cullum

Related answer to the location of the DOCKER_OPTS is at https://mcmap.net/q/429568/-docker_opts-do-not-work-in-config-file-etc-default-docker – Cullum

Another solution I found was to use the same HOST's /etc/resolv.conf in the VM... – Cullum

resolve.conf was pointing to my loopback 127.0.01. Changing it to google DNS nameserver 8.8.8.8 made it work. Thanks ! – Martyr

I also had to remove private DNS as in (askubuntu.com/questions/627899/…) BONUS oneliner: DOCKER_OPTS=$(cat /etc/resolv.conf | grep nameserver | awk '{ printf " --dns " $2 }') && sudo service docker restart – Lorola

Answer 2

I had pretty the same problem. The provided solution didn't help in my case. But it worked as soon I updated my Dockerfile adding environment variables for the proxy in it.

ENV HTTP_PROXY http://<proxy_host>:<port>
ENV HTTPS_PROXY http://<proxy_host>:<port>
ENV http_proxy http://<proxy_host>:<port>
ENV https_proxy http://<proxy_host>:<port>

Clupeid

Answer 3

It's likely due to your local caching name server listening on 127.0.1.1 which is not accessible from within the container.

Try putting the following into your Dockerfile:

CMD "sh" "-c" "echo nameserver 8.8.8.8 > /etc/resolv.conf"

Yellows

That did not work for me :( Behind an intranet, the only solution was to add the DNS servers in the daemon configuration. – Cullum

Answer 4

also, just adding the nameservers from the host (in my case mac osx ) to the docker-machine vm solves the problem.

Mushroom

Answer 5

For me the problem was that my ISP blocked google's DNS (8.8.8.8) which docker uses as a fallback default.

The trick here is to find out your DNS IP and tell docker to use it.

In my case (running Ubuntu 17.04), trying to get this information from /etc/resolv.conf did not work, but I used this command:

nmcli dev show | grep IP4.DNS

Then I took this IP and added in /etc/defaults/docker:

DOCKER_OPTS="--dns 192.168.50.1"

Now restart your docker daemon, and try building again.

Inchoate

Answer 6

In my case, the issue is that our company's DNS is flawed in few ways, which requires tampering the /etc/hosts, and for docker, /etc/docker/daemon.json. That's the file which was hiding the error:

{
    "dns": ["10.5...", "10.5...", "10.5..."]
}

I have backed this up and replaced with

{
    "dns": ["8.8.8.8", "8.8.4.4"]
}

And it started working. I am looking for a solution that would work in all cases - on our VPN which needs the custom DNS servers as well as home on a normal network.

Note that in modern Linux, /etc/hosts is generated and DNS is managed by SystemD. I am not sure how Docker handles this, but perhaps it could be enough to point it to SystemD's fake DNS at 127.0.0.53.

Androsphinx

Answer 7

Create a local repo mirror - this can also be done as a docker-mirror-packages-repo

Then run "docker build --add-host "archive.ubuntu.com:repo-docker-ip" to have the build process download from your local mirror. That is not only faster but it ensures a better reproducibility of your builds.

I am using that for the testsuite of the docker-systemctl-replacement which is testing compatibility with a number of distros each with dozens of docker rebuilds.

Denadenae