What is the theoretical maximum number of open TCP connections that a modern Linux box can have

Question

Assuming infinite performance from hardware, can a Linux box support >65536 open TCP connections?

I understand that the number of ephemeral ports (<65536) limits the number of connections from one local IP to one port on one remote IP.

The tuple (local ip, local port, remote ip, remote port) is what uniquely defines a TCP connection; does this imply that more than 65K connections can be supported if more than one of these parameters are free. e.g. connections to a single port number on multiple remote hosts from multiple local IPs.

Is there another 16 bit limit in the system? Number of file descriptors perhaps?

Telegraphy

Answer 1

A single listening port can accept more than one connection simultaneously.

There is a '64K' limit that is often cited, but that is per client per server port, and needs clarifying.

Each TCP/IP packet has basically four fields for addressing. These are:

source_ip source_port destination_ip destination_port
<----- client ------> <--------- server ------------>

Inside the TCP stack, these four fields are used as a compound key to match up packets to connections (e.g. file descriptors).

If a client has many connections to the same port on the same destination, then three of those fields will be the same - only source_port varies to differentiate the different connections. Ports are 16-bit numbers, therefore the maximum number of connections any given client can have to any given host port is 64K.

However, multiple clients can each have up to 64K connections to some server's port, and if the server has multiple ports or either is multi-homed then you can multiply that further.

So the real limit is file descriptors. Each individual socket connection is given a file descriptor, so the limit is really the number of file descriptors that the system has been configured to allow and resources to handle. The maximum limit is typically up over 300K, but is configurable e.g. with sysctl.

The realistic limits being boasted about for normal boxes are around 80K for example single threaded Jabber messaging servers.

Lantz

You can theoretically have more than 64K outgoing connections if you (a) use SO_REUSEADDR and (b) target different destination IP addresses. But kernel memory limits will probably stop you first. – Protector

@Protector I thought SO_REUSEADDR was for servers binding when being restarted? – Lantz

Yes, that too. It basically relaxes the early checks for address conflicts for new sockets. – Protector

The sysctl limit is for the whole system, right? There's also a limit configurable with ulimit, which limits the maximum number of file descriptors for a process. That is by default a lot less than 300K, usually 1024. – Culpa

A slight technicality: A client machine can also have multiple IP addresses assigned from a router. These can all be assigned to a single MAC, or that machine can have multiple physical network interfaces for additional IP addresses. The OP specified 1 IP, but it's important for others not to rule out more IP addresses. – Sharpeared

Be aware that the tcp_fin_timeout blocks the same socket (source, target, port combination) for another 60 seconds by default, which greatly reduces the number of actually available tcp connection between two systems, if connections are disconnected and reconnected frequently. This problem can be minimized by allowing to reuse (tcp_tw_reuse=1) sockets in TIME_WAIT state (not always supported) or by breaking the TCP/IP standard in reducing this timeout to a lower value (usually works fine anyways). – Frosty

So if I have two NICs or one NIC with two IP addresses, the max number of TCP connections gets doubled, isn't it? – Coronal

One more thing, TCP server can have > 64k connections, but, with default options, TCP client with a single IP address can NOT have > 64k connections because of port conflict. – Senseless

Answer 2

If you are thinking of running a server and trying to decide how many connections can be served from one machine, you may want to read about the C10k problem and the potential problems involved in serving lots of clients simultaneously.

Tamarau

C10k is 10 years old and no longer fun. [Read this] to see how C1024K can be tackled. – Colby

@Colby - did you mean metabrew.com/article/… ? – Lederer

@MikkoRantalainen - yes. I think there are better benchmarks available now. Phoenix guys have already pushed it to 2 million simultaneous connection. – Colby

@Colby - there is Dell demo with 12M connections: mrotaru.wordpress.com/2013/06/20/… – Lederer

Quite a few years ago: Intel Atom D2700, 2GB RAM, 1.2M concurrent connections. The only issues I had were with the Windows boxes in the test work; these regularly went belly up while attempting to DoS the Intel Atom box... – Ambulate

ieeexplore.ieee.org/abstract/document/8422993 -- C40m -- Not sure if Linux or not – Liston

Answer 3

If you used a raw socket (SOCK_RAW) and re-implemented TCP in userland, I think the answer is limited in this case only by the number of (local address, source port, destination address, destination port) tuples (~2^64 per local address).

It would of course take a lot of memory to keep the state of all those connections, and I think you would have to set up some iptables rules to keep the kernel TCP stack from getting upset &/or responding on your behalf.

Chough