Some exposed Docker ports are not accessible from outside (dovecot, postfix)

Question

this is driving me nuts, but for some reason, I cannot access certain ports from outside. I've checked almost every possible setting, but I have no idea why this is not working. I have no idea where to fix this issue (Ubuntu firewall vs. Docker vs. Dovecot vs. Postfix) and I'm really hoping that you guys can help me.

FYI: mycooldomain.com is just a placeholder for my domain which I don't want to share at the moment.

I'll buy a beer for the one who helps me fixing this! :)

Short intro

On my root server, I'm running the following docker mail server container (https://github.com/tomav/docker-mailserver/) which exposes several ports (143, 25, 587, 993). The container runs dovecot (IMAP) and postfix (SMTP).

tl;dr

I have the aforementioned mail server (dovecot, postfix) running in a docker container and can only connect to one port (143 - IMAP). Other ports 25, 587 and 993 are not accessible from outside - only from the host directly. Docker exposes them correctly and they are listed in the iptables correctly. The logs in the container (dovecot, postfix) don't show any reaction when requests come in (no rejected request or something like that), so I suppose they never reach the service in the container. Other docker container ports (e.g. my web server: 80, 443).

Here's the detailed information:

I can connect to every port from the host directly (let's test the IMAP secure port):

mastix@localhost:~$ telnet mail.mycooldomain.com 993
Trying xxx.xxx.xxx.xxx…
Connected to mail.mycooldomain.com.
Escape character is '^]'.
220 mail.mycooldomain.com ESMTP Postfix (Ubuntu)

But not from my local machine:

mastixmc$ telnet mail.mycooldomain.com 993
Trying xxx.xxx.xxx.xxx…
telnet:connect to address xxx.xxx.xxx.xxx: Operation timed out
telnet: Unable to connect to remote host

docker-compose ps shows me that they are exposed and bound correctly:

110/tcp,
0.0.0.0:143-143/tcp,
0.0.0.0:25->25/tcp, 4190/tcp,     
0.0.0.0:587->587/tcp,             
0.0.0.0:993->993/tcp, 995/tcp

netstat -ntlp tells me the same:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 :::587                  :::*                    LISTEN      -
tcp6       0      0 :::143                  :::*                    LISTEN      -
tcp6       0      0 :::80                   :::*                    LISTEN      -
tcp6       0      0 :::25                   :::*                    LISTEN      -
tcp6       0      0 :::443                  :::*                    LISTEN      -
tcp6       0      0 :::2332                 :::*                    LISTEN      -
tcp6       0      0 :::993                  :::*                    LISTEN      -

I can access my web server (80/443) and unsecure IMAP (143) - which I will not allow in the future. But all other ports are not accessible.

I even set up UFW (although that's not needed) to make sure that Ubuntu doesn't block anything:

Added user rules (see 'ufw status' for running firewall):
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 25/tcp
ufw allow 587/tcp
ufw allow 143/tcp
ufw allow 993/tcp

When looking at the logs (Dovecot, Postfix) I don't see any information, so it looks like the request does not reach the services.

I DID NOT touch the following Docker/UFW settings:

/etc/default/ufw

DEFAULT_FORWARD_POLICY="DROP"

And DID NOT change iptables = false in the docker engine.

Here's the iptables -L call, where you can clearly see that the DOCKER chain adds the rules correctly (also my ufw rules are there):

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ufw-before-logging-input  all  --  anywhere             anywhere
ufw-before-input  all  --  anywhere             anywhere
ufw-after-input  all  --  anywhere             anywhere
ufw-after-logging-input  all  --  anywhere             anywhere
ufw-reject-input  all  --  anywhere             anywhere
ufw-track-input  all  --  anywhere             anywhere

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-ISOLATION  all  --  anywhere             anywhere
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ufw-before-logging-forward  all  --  anywhere             anywhere
ufw-before-forward  all  --  anywhere             anywhere
ufw-after-forward  all  --  anywhere             anywhere
ufw-after-logging-forward  all  --  anywhere             anywhere
ufw-reject-forward  all  --  anywhere             anywhere
ufw-track-forward  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ufw-before-logging-output  all  --  anywhere             anywhere
ufw-before-output  all  --  anywhere             anywhere
ufw-after-output  all  --  anywhere             anywhere
ufw-after-logging-output  all  --  anywhere             anywhere
ufw-reject-output  all  --  anywhere             anywhere
ufw-track-output  all  --  anywhere             anywhere

Chain DOCKER (3 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             172.19.0.3           tcp dpt:https
ACCEPT     tcp  --  anywhere             172.19.0.3           tcp dpt:http
ACCEPT     tcp  --  anywhere             172.19.0.7           tcp dpt:imaps
ACCEPT     tcp  --  anywhere             172.19.0.7           tcp dpt:submission
ACCEPT     tcp  --  anywhere             172.19.0.7           tcp dpt:imap2
ACCEPT     tcp  --  anywhere             172.19.0.7           tcp dpt:smtp

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere
[...]
Chain ufw-user-input (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:imaps
ACCEPT     udp  --  anywhere             anywhere             udp dpt:imaps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:2332
ACCEPT     udp  --  anywhere             anywhere             udp dpt:2332
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     udp  --  anywhere             anywhere             udp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     udp  --  anywhere             anywhere             udp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:submission
ACCEPT     udp  --  anywhere             anywhere             udp dpt:submission
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:imap2
ACCEPT     udp  --  anywhere             anywhere             udp dpt:imap2

So everything looks correct to me.

Update #1:

nmap from docker host (against mycooldomain.com):

PORT    STATE    SERVICE
25/tcp  filtered smtp
80/tcp  filtered http
143/tcp filtered imap
443/tcp filtered https
587/tcp filtered submission
993/tcp filtered imaps

nmap from my local machine (against mycooldomain.com):

PORT    STATE SERVICE
80/tcp  open  http
143/tcp open  imap
443/tcp open  https

As you can see the ports are not open, therefore I can't connect from remote to the ports 25, 587 and 993.

UPDATE #2:

So I've stopped all containers and openend a simple python HTTP server:

sudo python -m SimpleHTTPServer 993

Also can't access this port. So I opened the same server with port 8080 => success.

It looks like either Ubuntu or my hoster is blocking port 993! I'm currently in contact with the one who provides the root server.

I already filed an issue in the project itself (https://github.com/tomav/docker-mailserver/issues/602), but they couldn't help me either.

Thank you very much in advance,

Greetz,

Sascha

Montford

Does remote access work without docker? Could you run anything w/o docker on the non-functional ports and try to connect. Maybe your hoster is blocking some ports. Output of an nmap scan would also be interesting. – Redouble

Sure - the SSH daemon is running as is accessible. I also have other docker containers that are accessible from remote (e.g. a reverse proxy listening on port 80 and 443). Will add nmap port scan result right after this reply. – Montford

I'm currently in contact with the hoster to see whether they block the mail server ports. – Montford

You may delete this post now that the problem should be fixed on the root server. – Frame

You can't delete a question when there's a bounty. – Montford

Answer 1

I fixed it... or at least my hoster did. There was another hardware firewall blocking certain ports. They had to enable a certain "Mail server" firewall ruleset, which allows running a mail server.

Thanks for your help and hints!!!

Greetz,

Sascha

Montford