On a Synology NAS, it appears the default setup for docker/iptables is to source NAT traffic going to the container to the gateway IP. This fundamentally causes problems when the container needs to see the correct/real client IP. I do not see this issue when running on an ubuntu system, for example.
Let me walk you though what I'm seeing. On the Synology NAS, I'll run a simple nginx container:
docker run --rm -ti -p 80:80 nginx
(I have disabled the Synology NAS from using port 80)
I'll do a curl from my laptop and I see the following log line:
172.17.0.1 - - [05/May/2020:17:02:44 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.64.1" "-"
Hm... the client IP (172.17.0.1) is the docker0 interface on the NAS.
user@synology:~$ ifconfig docker0
docker0 Link encap:Ethernet HWaddr 02:42:B5:BE:C5:51
inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0
<snip>
When I first saw this, I was quite confused because I did not recall docker networking working this way. So I fired up my ubuntu VM and tried the same test (same docker run command from earlier).
The log line from my curl test off box:
172.16.207.1 - - [05/May/2020:17:12:04 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.64.1" "-"
In this case the docker0 IP was not the source of the traffic as seen by the container.
user@ubuntu-vm:~# ifconfig docker0
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
<snip>
So it appears that the docker network setup for a Synolgy NAS is different vs a more standard ubuntu deployment. Okay, neat. So how does one fix this?
This is where I'm struggling. Clearly something is happening with iptables. Running the same iptables -t nat -L -n command on both systems shows vastly different results.
user@synology:~$ iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DEFAULT_OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
DEFAULT_POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
Chain DEFAULT_OUTPUT (1 references)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain DEFAULT_POSTROUTING (1 references)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:80
Chain DOCKER (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:172.17.0.2:80
user@ubuntu-vm:~# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
MASQUERADE all -- 172.18.0.0/16 0.0.0.0/0
MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:80
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:172.17.0.2:80
I do not really understand how these chains work or why they're different for different systems. I haven't changed any docker level settings on either system, these are default configs.
Has anyone ran into this before? Is there a quick way to toggle this?