Flannel and docker don't start

Question

I'm trying to set up a kubernetes cluster on 2 nodes , centos 7.1 using this guide. However when I attempt to start the services on the minion like so:

for SERVICES in kube-proxy kubelet docker flanneld; do
    systemctl restart $SERVICES
    systemctl enable $SERVICES
    systemctl status $SERVICES 
done

I get the following error:

-- Logs begin at Wed 2015-12-23 13:00:41 UTC, end at Wed 2015-12-23 16:03:54 UTC. --
Dec 23 16:03:47 sc-test2 systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Dec 23 16:03:47 sc-test2 systemd[1]: Failed to start Docker Storage Setup.
-- Subject: Unit docker-storage-setup.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker-storage-setup.service has failed.
-- 
-- The result is failed.
Dec 23 16:03:47 sc-test2 systemd[1]: Unit docker-storage-setup.service entered failed state.
Dec 23 16:03:48 sc-test2 flanneld[36477]: E1223 16:03:48.187350 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:49 sc-test2 flanneld[36477]: E1223 16:03:49.189860 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:50 sc-test2 flanneld[36477]: E1223 16:03:50.192894 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:51 sc-test2 flanneld[36477]: E1223 16:03:51.194940 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:52 sc-test2 flanneld[36477]: E1223 16:03:52.197222 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:53 sc-test2 flanneld[36477]: E1223 16:03:53.199248 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)
Dec 23 16:03:54 sc-test2 flanneld[36477]: E1223 16:03:54.201160 36477 network.go:53] Failed to retrieve network config: 100: Key not found (/atomic.io)

I'm sure I set the key on the master with : etcdctl mk /coreos.com/network/config '{"Network":"172.17.0.0/16"}'

By far installation seems to be the hardest bit on using kubernetes :(

Impression

If you look at the error message, it's not looking under /coreos.com; it appears to be looking under /atomic.io: Key not found (/atomic.io) – Eustoliaeutectic

True..here's an excerpt from the config file on the minion # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_KEY="/atomic.io/network" – Impression

Do I change this here? I'm using centos.. – Impression

You can change the value of FLANNEL_ETCD_KEY in /etc/sysconfig/flanneld, or you can just create the appropriate path in etcd. Are you sure you're using a current CentOS atomic image? In my environment (CentOS 7 atomic; rpm-ostree status says 7.20151118), FLANNEL_ETCD_KEY is already set to "/coreos.com/network". – Eustoliaeutectic

I don't have the rpm-ostree status command..my file now looks like # Flanneld configuration options # etcd url location. Point this to the server where etcd runs FLANNEL_ETCD="http://x.x.x.x:2379" # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_KEY="/coreos.com/network" # Any additional options that you want to pass #FLANNEL_OPTIONS="" However flannel still fails but with a different error : ` Failed to retrieve network config: invalid character 'e' after top-level value` – Impression

Changing the key to /atomic.io/network worked for me. – Impression

Answer 1

Today's christmas but I spent the whole day trying to get this to work :) This is what I did:

#1 FLANNEL

As mentioned I'd set the flannel etcd key on the master with:

etcdctl mk /coreos.com/network/config '{"Network":"172.17.0.0/16"}'

but I got this error when trying to start flannel on the minion:

Failed to retrieve network config: 100: Key not found (/atomic.io)

So I edited the /etc/sysconfig/flanneld file on the minion from:

# Flanneld configuration options  

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD="http://master:2379"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_KEY="/coreos.com/network"

# Any additional options that you want to pass
#FLANNEL_OPTIONS=""

to:

# Flanneld configuration options  

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD="http://master:2379"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_KEY="/atomic.io/network"

# Any additional options that you want to pass
#FLANNEL_OPTIONS=""

i.e. changed the FLANNEL_ETCD key.

After this systemctl start flanneld worked.

#2 DOCKER

I didn't find a way to make the version installed as a dependency by kubernetes work so I uninstalled it and following the docker docs for Centos installed docker-engine and manually created a docker.service file for systemctl.

cd /usr/lib/systemd/system

and the contents of the docker.service:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket
Requires=flanneld.service
After=flanneld.service

[Service]
EnvironmentFile=/etc/sysconfig/flanneld
ExecStart=/usr/bin/docker daemon -H fd:// --bip=${FLANNEL_SUBNET}
Restart=on-failure
RestartSec=5


[Install]
WantedBy=multi-user.target

then start and enable the daemon with systemctl as well as query the status.

systemctl restart docker
systemctl enable docker
systemctl status docker

Impression

I've been working on the same thing. The instructions from severalnines.com seem to depend on more storage; I was trying this out in centos guests in virtual box, and the docker startup failed because docker-storage-setup failed because it was unable to create a logical volume to hold docker images. Since I was working with virtual box, I just created larger (12GB) centos guests, and that worked. Then I was stuck on the flannel problem. It looks like their setup for the flanneld configuration was wrong. Your change here fixed it for me. – Valais

Nope, spoke too soon. 12GB VMs wasn't large enough. Any ideas how big I should make it? – Valais

I don't think storage is an issue, I was able to get kubernetes working by uninstalling the existing docker package (installed via yum install docker) and installing docker-engine as per the docker docs. Then swapping out the service file values with the above. – Impression

I did find that adding the argument --storage-opt="dm.blocksize=64k" to the docker daemon command fixed it. From github.com/docker/docker/issues/6325 – Impression