M

9

86

I have recently installed privacy vpn, and it turns out that enabled openvpn breaks docker.

When I try to run docker-compose up i get following error

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

Disabling vpn fixes the problem (however I'd rather not disable it). Is there any way to make these two co-exist peacefully? I use debian jessie, and my openvpn has following version string

 OpenVPN 2.3.4 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Jun 26 2017

A lot of people "solved" this problem by disabling the openvpn, so I'm asking specifically on how to make these two work at the same time.

References:

If this makes any difference my vpn provider is: https://www.ovpn.com/ and here is (somewhat redacted) config file:

client
dev tun

proto udp

remote host port
remote-random

mute-replay-warnings
replay-window 256

push "dhcp-option DNS 46.227.67.134"    
push "dhcp-option DNS 192.165.9.158"

remote-cert-tls server
cipher aes-256-cbc
pull

nobind
reneg-sec 432000
resolv-retry infinite

comp-lzo
verb 1

persist-key
persist-tun
auth-user-pass /etc/openvpn/credentials
ca ovpn-ca.crt
tls-auth ovpn-tls.key 1

Mick answered 15/8, 2017 at 11:50 Comment(0)

M

95

Solution (TL;DR;)

Create /etc/openvpn/fix-routes.sh script with following contents:

#!/bin/sh

echo "Adding default route to $route_vpn_gateway with /0 mask..."
ip route add default via $route_vpn_gateway

echo "Removing /1 routes..."
ip route del 0.0.0.0/1 via $route_vpn_gateway
ip route del 128.0.0.0/1 via $route_vpn_gateway

Add executable bit to the file: chmod o+x /etc/openvpn/fix-routes.sh. Change owner of this file to root: chown root:root /etc/openvpn/fix-routes.sh.

Add to your config following two lines:

 script-security 2
 route-up  /etc/openvpn/fix-routes.sh

Explanation

Openvpn adds routes that for following networks: 0.0.0.0/1 and 128.0.0.0/1 (these routes cover entire IP range), and docker can't find range of IP addresses to create it's own private network.

You need to add a default route (to route everything through openvpn) and disable these two specific routes. fix-routes script does that.

This script is called after openvpn adds its own routes. To execute scripts you'll need to set script-security to 2 which allows execution of bash scripts from openvpn context.

Thanks

I'd like to thank author of this comment on github, also thanks to ovpn support.

Mick answered 15/8, 2017 at 14:4 Comment(6)

Careful with this suggestion - if you want all traffic to go over the VPN, this script will send your traffic via your previous (unprotected) default route. OpenVPN adds a /1 route which is more specific than default. To really make this work you need to remove the default route in the route-up script and then add it back in in a down script. OpenVPN passes $route_net_gateway, so something like ip route delete default via $route_net_gateway for route-up and ip route add default via $route_net_gateway for down. – Fite 12/1, 2018 at 20:10

Actually it looks like I have VPN through VPN (all traffic is routed there). Looks like my system has more than a single default route, and OpenVPN one is just first (and thus only one used). Some of the traffic goes through other routes (local traffic) but in my setup local traffic is the least of my concerns. However your suggestion would make routing table nicer. I definetely need to read up more about linux networking though. – Mick 13/1, 2018 at 14:17

I didn't want to route everything through my VPN, so I used: ip route add 10.8.0.0/24 via "$route_vpn_gateway" . This answer was what led me to the solution though. Thanks! – Innocence 6/12, 2019 at 0:34

@Fite can you please explain your comments for noobs? As far as I understood from your comment, this script will prevent my traffic from going over my VPN But I still see the this happening when I use traceroute. Am I missing something? What kind of traffic would not go over the VPN if I use this script? thanks – Mcbee 9/3, 2021 at 17:17

Where exactly should I add these two mentioned lines? Which config file? – Misbecome 28/9, 2021 at 14:8

What would be the proper way to route all traffic through VPN that belongs to public IP addresses but leave traffic to private IPs untouched? – Knighthood 24/2, 2022 at 21:2

Z

59

You can also get docker-compose working if you define the subnet CIDR in your docker compose file:

networks:
  your-network:
   ipam:
      config:
      - subnet: 172.16.238.0/24
        gateway: 172.16.238.1

Another option: create first the network with the subnet CIDR and then specify in the docker compose file that you want to use this network:

docker network create your-network --subnet 172.24.24.0/24

In your docker compose file:

networks:
  your-network:
    external: true

Zoba answered 20/6, 2018 at 13:2 Comment(3)

This worked and is a much better answer. Thank you! – Lobation 4/8, 2018 at 16:3

gateway: does not currently work with docker-compose V3.0 Its a known limitation, the only workaround is to stick with V2 or use the network create command that was also suggested. – Fighter 21/10, 2018 at 17:51

Worked in version 1.25.5 of docker compose. Thank you – Throughway 2/7, 2020 at 13:37

H

22

Based on answer from Anas El Barkani, here's a complete step-by-step example using PostgreSQL.

While VPN is not connected, create a permanent docker network:

docker network create my-network --subnet 172.24.24.0/24

In docker-compose file, specify network as external:

version: "2"

services:
  postgres:
    container_name: postgres
    image: postgres
    volumes:
      - ./volumes/postgres/data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=dummy
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=123456
      - POSTGRES_HOST=localhost
    networks:
      - default
    ports:
      - "127.0.0.1:5432:5432"

networks:
  default:
    external:
      name: my-network

That's all. Now you can enable your VPN, and start/stop container as usual:

docker-compose up -d
docker-compose down

No need to turn VPN on/off every time, or to add weird scripts as root.

Highball answered 31/10, 2018 at 14:16 Comment(3)

"not need to turn vpn off" but create network is run only when vpn is not connected... – Sucking 23/2, 2019 at 17:21

@Sucking You disable VPN once when creating network. Network is persistent. You can reuse that network in many compose files. – Highball 25/2, 2019 at 15:32

You're right, but for example we use a tool to create all of containers, networks everything automatically by project. And again for other project. We don't want to these manual steps slows our developers. They switchs projects every week. The openvpn ip range setup by config file is much more acceptable solution for our case. – Sucking 25/2, 2019 at 16:45

P

20

Disclaimer:

This solution originally designed for next configuration:

Ubuntu 18.04
OpenVPN 2.4.4
Docker-CE 19.03.5
Docker-Compose 1.24.0
Bridge IPV4 networks
Docker-Swarm not used

and may differ for other configurations.

Problem

Start your VPN connection.

Case 1

When you try to restart docker daemon you'll get in the logs:

failed to start daemon: Error initializing network controller: list bridge addresses failed: PredefinedLocalScopeDefaultNetworks

Case 2

When you try to create bridge network (implicitly docker and docker-compose try to create this kind of network) within next cases:

docker create network without defining subnet parameter
docker-compose up without defining subnet parameter

you'll get:

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

Solution (TL;BPR)

Select address range for docker network from the private address space that not planned be to use for resources inside your VPN. Imagine that it is 172.26.0.0/16.
Add changes to the Docker's daemon config file daemon.json file. :
```
{
    "bip": "172.26.0.1/17",
    "fixed-cidr": "172.26.0.0/17", 
    "default-address-pools" : [
        {
            "base" : "172.26.128.0/17",
            "size" : 24
        }
    ]
}
```
Where:
- bip - aka «bridge ip»: specific bridge IP address for the docker0 bridge network, which used by default if other was not specified.
- fixed-cidr - CIDR range for docker0 interface and local containers. Only needed if you want to restrict the IP range defined by bip.
- default-address-pools - CIDR range for docker_gwbridge (needed for docker-swarm) interface and bridge networks. size parameter set the default submask for newly created networks inside this range.
We divide our initial 172.26.0.0/16 range by equal 172.26.0.0 - 172.26.127.255 and 172.26.128.0 - 172.26.255.255 pools in this example.

Be careful with daemon.json formating, otherwise you'd get the error like this when restart docker's daemon
```
unable to configure the Docker daemon with file /etc/docker/daemon.json
```
Run your VPN connection
Find your device name by running the comand. Usually it something like tun0
```
ip addr show type tun
```
Show created routes
```
ip route show dev tun0
```
Find the pool that overlaps with our selected address, let it be:
```
172.16.0.0/12 via 10.8.0.1
```
Split that pool on subnets on chunks with our selected Docker pool 172.26.0.0/16. You can use this amazing calculator by David C. We've got:
```
172.16.0.1/13
172.24.0.1/15
172.26.0.0/16
172.27.0.1/16
172.28.0.1/14
```

Create /etc/openvpn/mynetwork-route-up.sh script, for OpenVPN for excluding our subnet from routes, with following contents (note that we excluded our network):

#!/usr/bin/env bash

echo "Remove the route that conflicts with the Docker's subnet"
ip route del 172.16.0.0/12 via $route_vpn_gateway

echo "Bring back routes that don't intersect"
ip route add 172.16.0.0/13 via $route_vpn_gateway dev $dev
ip route add 172.24.0.0/15 via $route_vpn_gateway dev $dev
ip route add 172.27.0.0/16 via $route_vpn_gateway dev $dev
ip route add 172.28.0.0/14 via $route_vpn_gateway dev $dev

Create /etc/openvpn/mynetwork-route-pre-down.sh script with following contents (note that we excluded our network):

#!/usr/bin/env bash

echo "Remove manually created routes"
ip route del 172.16.0.0/13 dev $dev
ip route del 172.24.0.0/15 dev $dev
ip route del 172.27.0.0/16 dev $dev
ip route del 172.28.0.0/14 dev $dev

echo "Creating original route because OpenVPN will try to del that"
ip route add 172.16.0.0/12 via $route_vpn_gateway dev $dev

Make that scripts executable

sudo chmod u+x /etc/openvpn/mynetwork-route-up.sh
sudo chmod u+x /etc/openvpn/mynetwork-route-pre-down.sh

Add this lines to the end of your .ovpn config

script-security 2
route-up /etc/openvpn/mynetwork-route-up.sh
route-pre-down /etc/openvpn/mynetwork-route-pre-down.sh

Restart your OpenVPN
Run (for removing networks that may conflict when daemon will restart)

docker network prune

Restart Docker daemon
```
sudo service docker restart
```

Cause

OpenVPN used frequently to route all traffic through tunnel, or at least, proxy private pools. So why docker fails, when it started?

Case 1

When you starting the Docker daemon, it checks daemon's config bridge network for overlaping with routes (up->down stacktrace):

As you can see here, you can also disable creating the default bridge network in the daemon config for fixing this error.

Case 2

When the Docker's component libnetwork try to create it's network, it check all available addresses for overlaping with routes. If nothing found, it returns an error (up->down stacktrace):

Of course, other scenarios to get this errors exists too. Gotta Catch 'Em All!

Workarounds (not recommended)

Create network with the subnet parameter

Docker allow you to pass subnet address range explicitly and it seems that doesn't perform an overlap check in this case.

https://github.com/docker/libnetwork/blob/922cd533eac14b6e0754756c5cacf9f44af5d699/network.go#L1657

Create a network when OpenVPN stopped, then start it

I don't dive deep in that, but I think, OpenVPN doesn't check overlapping.

P.S.

Thanks (https://stackoverflow.com/users/7918/jb)[jb] for his great answer, it inspire me a lot for wrtiting this answer.

For in-depth network understanding of Docker, you can read this articles:

«Configure Custom CIDR Ranges in Docker EE» by Dave Thompson
«Plan your installation» by Docker

Also, don't forget this!

Perisarc answered 10/2, 2020 at 19:52 Comment(1)

Great answer with thorough explanation and references. – Microhenry 23/7, 2021 at 7:11

P

3

The default routes which cause the problem are pushed to the OpenVPN client by the OpenVPN server.

Rather then create a script to delete the routes you can simply stop the problematic routes from being created in the first place.

There are two ways to do this:

If you can change the settings on the OpenVPN server, edit the config and remove the redirect-gateway option. On my EdgeRouter the relevant line looks like this:

openvpn-option "--push redirect-gateway def1"

One a Linux server I believe it would look like this:

push "redirect-gateway def1"
If you can't change settings on the OpenVPN server you can tell your OpenVPN client to ignore the pushed routes from the server. On my Linux client the relevant line looks like this:

pull-filter ignore redirect-gateway

Once you've made those changes and restarted the OpenVPN service you should be able to start Docker containers without the dreaded could not find an available, non-overlapping IPv4 address error.

Pass answered 23/4, 2021 at 7:47 Comment(0)

C

1

Maybe one way to do it is to add all routes excluding 172.16.0.0/12 to route through VPN so we are sure everything going out is properly handled:

sudo ip route add 192.0.0.0/2 via $route_vpn_gateway
sudo ip route add 128.0.0.0/3 via $route_vpn_gateway
sudo ip route add 176.0.0.0/4 via $route_vpn_gateway
sudo ip route add 160.0.0.0/5 via $route_vpn_gateway
sudo ip route add 168.0.0.0/6 via $route_vpn_gateway
sudo ip route add 174.0.0.0/7 via $route_vpn_gateway
sudo ip route add 173.0.0.0/8 via $route_vpn_gateway
sudo ip route add 172.128.0.0/9 via $route_vpn_gateway
sudo ip route add 172.64.0.0/10 via $route_vpn_gateway
sudo ip route add 172.32.0.0/11 via $route_vpn_gateway
sudo ip route add 172.0.0.0/12 via $route_vpn_gateway

# And finally delete the default route which handle 172.16.0.0/12
sudo ip route del 128.0.0.0/1 via $route_vpn_gateway

Cleary answered 11/10, 2018 at 21:21 Comment(0)

B

1

What worked for me: After seeing

Creating network "airflow_default" with the default driver
ERROR: could not find an available, non-overlapping IPv4 address pool 
among the defaults to assign to the network

just create that network manually:

docker network create airflow_default --subnet 172.24.24.0/24

Boardman answered 2/9, 2021 at 11:31 Comment(0)

W

0

Some additional context here: the 0.0.0.0 and 128.0.0.0 routes are only created if the OpenVPN server (aka Access Server) is configured to push routes to send all the endpoint's Internet traffic via the VPN. By adding these broad routes, the user's Internet traffic can be routed while not interfering with routing on the local LAN, and ensuring that the endpoint remains able to route the OpenVPN traffic itself to the local router.

If sending all Internet traffic via the OpenVPN server isn't a requirement, you may be better off asking your VPN admin to create a profile that only routes traffic to required destinations (such as private IP address ranges) via the VPN instead of everything. That should avoid having to mess with the routes on the endpoint.

Wycoff answered 1/10, 2018 at 20:6 Comment(0)

G

0

Based on the answer from alexanderlukanin13. It's worked for me for the docker-compose file's version: "3.8". I added a usage pre-existing network from the documentation.

Turn off the OpenVPN connection.
Create a custom docker network: sudo docker network create your-network --subnet 172.24.24.0/24
At the bottom of the docker-compose file:

version: "3.8"

services:
...

volumes:
...

networks:
  default:
    name: your-network
    external: true

Turn on the OpenVPN connection

After that: sudo docker compose -f docker-compose.yml up -d --build works without the error:

failed to create network your-network-name_default: Error response from daemon: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

Gualtiero answered 23/12, 2022 at 15:55 Comment(0)