Sockets On Same Machine For Windows and Linux

Question

How efficient is it to use sockets when doing IPC as compared to named pipes and other methods on Windows and Linux?

Right now, I have 4 separate apps on 4 separate boxes that need to communicate. Two are .NET 3.5 applications running on Windows Server 2003 R2. Two are Linux (Suse Linux 10). They're not generally CPU bound. The amount of traffic is not that large but its very important that it be low latency. We're using sockets right now with nagle dis-abled and the sles10 slow start patch installed on the linux machines.

How much of a speed boost do you think we would get by simply running the two windows apps on the same windows box and the two linux apps on the same linux box and making no code changes (ie still using sockets).

Will the OS's realize that the endpoints are on the same machine and know not to go out to the ethernet with the packets? Will they packets still have to go through the whole networking stack? How much faster it be if we took the time to change to named pipes or memory mapped files or something else?

Westing

This is easy to test. Just test it. – Aaren

+1, jeffamaphone: no vague comments or anecdotal evidence matters much here. What matters is how Michael's programs behave. – Barrybarrymore

Agreed, I should test it. That is the only way to know for sure given all the differences in networks, setups, etc. But I was just hoping someone would have a link to someone who has already done a very similar test. I can't imagine that rigorously comparing the performance of these things is a new idea - its probably been done 100 times by people who know a lot more about networking than I do. – Westing

I don't have the data available anymore so you'll have to take my word for it. But Unix Domain sockets on Linux are significantly faster than TCP over loopback. – Accentual

Answer 1

As for TCP performance, I have done this sort of test recently on an HP-UX server (8 Intel Itanium 2 processors 1.5 GHz 6 MB, 400 MT/s bus) and on Red Hat Linux (2 IA-64 1,6 Ghz). I used iperf in order to test TCP performance. I found that speed of TCP exchange is more than ten times faster when I run iperf on the same machine comparing to running iperf on two different machines.

You can also give it a try as there are options that might be of interest to you - length of buffer to read or write, set TCP no delay and so on. Also you can compare speed of TCP exchange on Windows machines as there is a version of iperf for Winddws.

This is a more detailed comparison:

1) Speed of TCP exchange between two iperf applicatons running on different HP-UX server, default TCP window 32K: 387 Mbits/sec

2) Speed of TCP exchange between two iperf applicatons running on different HP-UX server, TCP window 512K: 640 Mbits/sec

3) Speed of TCP exchange between two iperf applicatons running on the same HP-UX server, default TCP window 32K: 5.60 Gbits/sec

4) Speed of TCP exchange between two iperf applicatons running on the same HP-UX server, default TCP window 512K: 5.70 Gbits/sec.

5) Speed of TCP exchange between two iperf applicatons running on the same Linux server, TCP window 512K: 7.06 Gbits/sec

6) Speed of TCP exchange between two iperf applicatons running on HP-UX and Linux, TCP window 512K: 699 Mbits/sec

Crescin

That's perfect. Will run that on my setup as a start. Thanks! – Westing

Answer 2

Local named pipes will be faster since they run in kernel mode.

Celinacelinda

Agreed. But in terms of magnitude, which is the bigger jump as you go from (Sockets On Different Machines) --> (Sockets On Same Machine) --> (Named Pipes)? How much does it buy you to just run the apps on the same machine vs. spending the time to convert to named pipes? – Westing

running the apps on the same machine will buy you a big performance boost, since ethernet latency will be drastically reduced. Moving to named pipes will probably get you more of a boost, if you need it. I would recommend you try with just the TCP/IP stack first, then move to named pipes if you absolutely need that boost. Odds are you won't, but I can't know that for certain, since I don't know enough about your situation. – Celinacelinda

Use UNIX domain sockets! These will be equal to or close to named pipes in performance, yet nearly identical to TCP sockets in usage. – Henry

I've heard that those are good. But do you have any web site where they compare the same data transfer using regular sockets over ethernet vs. same machine vs. unix domain sockets vs. pipes? – Westing