How slow are TCP sockets compared to named pipes on Windows for localhost IPC?

Question

I am developing a TCP Proxy to be put in front of a TCP service that should handle between 500 and 1000 active connections from the wild Internet.

The proxy is running on the same machine as the service, and is mostly-transparent. The service is for the most part unaware of the proxy, the only exception being the notification of the real remote IP address of the clients.

This means that, for every inbound open TCP socket, there are two more sockets on the server: the secondth of the pair in the Proxy, and the one on the real service behind the proxy.

The send and recv window sizes on the two Proxy sockets are set to 1024 bytes.

What are the performance implications on this? How slow is this configuration? Should I put some effort on changing the service to use Named Pipes (or other IPC mechanism), or a localhost TCP socket is for the most part an efficient IPC?

The merge of the two apps is not an option. Right now we are stuck with the two process configuration.

EDIT: The reason for having two separate process on the same hardware is 100% economics. We have one server only, and we are not planning on getting more (no money).

The TCP service is a legacy software in Visual Basic 6 which grew beyond our expectations. The proxy is C++. We don't have the time, money nor manpower to rewrite and migrate the VB6 code to a modern programming environment.

The proxy is our attempt to mitigate a specific performance issue on the service, a DDoS attack we are getting from time to time.

The proxy is open source, and here is the project source code.

Syzran

An in-host TCP connection will be implemented as efficiently as possible (read: as a bi-directional local pipe, or something equivalent to that) on any modern network stack worth its salt, so I'd be surprised (and a bit disappointed in Microsoft) if there was a noticeable performance difference between TCP and Named Pipes for this use case. – Feola

@JeremyFriesner I agree with you, I'd love to assume that's the case on "Windows Server 2008". – Syzran

Answer 1

It will be the same (or at least not measurably different). Winsock is smart enough to know if it's talking to a socket on the same host and, in that case, it will short-circuit pretty much everything below IP and copy data directly buffer-to-buffer. In terms of named pipes vs. sockets, if you need to potentially be able to communicate to different machines ever in the future, choose sockets. If you know for a fact that you'll never need to do that, pick whichever one your developers are most familiar or most comfortable with.

Bellinzona

Sure about that? Do you have any reference? I imagine that loopback connections are expected to flow through a network interface and pass through the networking stack. This way they can be subject of firewall filtering rules. Ie, "if a SYN packet comes from locahost to localhost onto port XYZ, drop the packet". Windows do have some (hidden) firewall. – Syzran

@vz0: in my experience, Windows Firewall never filters packets going to the local host. For example, you can run up a webserver and use it on the local machine without having to change the firewall rules. – Relent

My references to that are that I used to work for Microsoft on windows networking. Even still, the firewall operates on the IP layer and above, and what I'm talking about is that winsock is short-circuiting everything below that. There's no reason that you couldn't apply firewall rules to the loopback (other than the obvious "why would you bother?"). Things on the loopback still pass through the networking stack, just not the entire networking stack and the remaining overhead is so negligible that it's insignificant unless you're doing this tens of thousands of times simultaneously. – Bellinzona

@JeffTucker Do you still feel Sockets is the way to go? I am having some issues and would appreciate your thoughts. Please see #26654124 – Pratincole

@JeffTucker How do you think the socket loopback fast path affects the performance compared to named pipes? – Latishalatitude

@JeffTucker please see this article: blogs.technet.microsoft.com/wincat/2012/12/05/… … according to this, in the "Using the TCP Loopback Fast Path" part, you need to actively ask windows to establish a fastpath connection. it does not happen automatically. – Displode

Any particular reason to opt for sockets if different machines? To my knowledge, Named Pipe supports network communication. – Worship

Answer 2

For anyone that comes to read this later, I want to add some findings that answer the original question.

For a utility we are developing we have a networking class that can use named pipes, or TCP with the same calls.

Here is a typical loop back file transfer on our test system:

TCP/IP Transfer time: 2.5 Seconds
Named Pipes Transfer time: 3.1 Seconds

Now, if you go outside the machine and connect to a remote computer on your network the performance for named pipes is much worse:

TCP/IP Transfer time: 12 Seconds
Named Pipes Transfer time: 2.5 Minutes (Yes Minutes!)

I realize that this is just one system (Windows 7) But I think it is a good indicator of how slow named pipes can be...and it seems like TCP is the way to go.

Alate

Comments on remote named pipes are a red herring for the question, which was about the same machine scenario. Named pipes are really same-machine objects, implemented by the kernel via shared memory. Remote named pipes are really "real named pipes" + SMB + a proprietary protocol for mapping communication to pipe name endpoints via SMB. There's all sorts of things which can happen in these additional parts to affect performance, which have nothing to do with "real" named pipe performance. – Pintail

Where is located do stating "Named pipes are really same-machine objects, implemented by the kernel via shared memory."? – Kiefer

what was the file size? how big where the packets? what kind of pipes/sockets? IOCP? – Tiling

What about highly transactional data exchange, like remote procedure calls? Pipes have a flush, while sockets have not. So i assume NP are a magnitude better on a local machine. – Atheism

Answer 3

I know this topic is very old, but it was still relevant for me, and maybe others will look at this in the future as well.

I implemented IPC between Excel (VBA) and another process on the same machine, both via a TCP connection as well as via Named Pipes.

In a quick performance test, I submitted a message than consisted of 26 bytes from client (Excel) to server (not Excel), and waited for the reply message from the other process (which consisted of 12 bytes in the example). I executed this a ton of times in a loop and measured the average execution time.

With TCP on localhost (Windows 7, no fastpath), one "conversation" (request+reply) took around 300-350 microseconds. Especially sending data was quite slow (sending the 26 bytes took around 200microseconds via TCP). With Named Pipes, one conversation took around 60 microseconds on average - so a LOT faster.

I'm not entirely sure why the difference was so large. The corporate environment I tested this in has a strict firewall, package inspections and what not, so I THINK this may have been caused as even the localhost-based TCP connection went through security measures significantly slowing it down, while named pipe ones likely did not.

TL:DR: In my case, Named Pipes were around 5-6 times faster than TCP for small packages (have not tested with bigger ones yet)

Vaucluse

Interesting, +1, but when you're sending lots of small packets like that the overhead (e.g., kernel transitions) is likely to be dominant - so the outcome will depend a lot on things like which API functions you're using, whether you've chosen asynchronous or synchronous I/O, and perhaps external factors like what anti-virus software is installed and whether Meltdown mitigation is in place. I guess the bottom line is that everybody needs to profile their own scenario. – Relent

... if you really needs the best performance possible you should probably be using shared memory anyway. :-) – Relent

@HarryJohnston Indeed; in my case, synchronous I/O (I think async wouldn't have helped in my particular case, can't make use of that much in VBA). E.g. 200 microseconds was just spent on the call to WinSock's "send" function. As for shared memory - agree, have/am trying that too, but didn't manage to get it right yet, tried with small buffers but had some odd effects, not sure if due to lack of VBA volatile reading capabilities .. but may come back and try this at a later point – Vaucluse

Hello @Vaucluse , thanks for the amazingly detailed answer. can you update your answer to also benchmark the cost of connecting, in addition to the request+reply benchmark. because I wonder how would the results change if you had to connect each time you sent a request? – Aronson

Answer 4

http://msdn.microsoft.com/en-us/library/aa178138(v=sql.80).aspx

Let me sum it up for you. If you are worried about performance then use TCP/IP. But if you have a really fast network and your not worried about performance then Named Pipes would be "neat" in that it might save you some code.

Not to mention, if you stick to TCP then you will have something that can be scaled, and even load balanced when the time comes.

Cheers,

Dettmer

Thanks. However, that article talks about Sockets vs. Named Pipes over a network for IPC on different machines. My programs will never run nor talk each other on separate hardware. – Syzran

A more pertinent summing up of the linked MSDN article would single out this section: "If the server application is running locally on the computer running an instance of Microsoft® SQL Server™ 2000, the local Named Pipes protocol is an option. Local named pipes runs in kernel mode and is extremely fast". Named pipes are faster than TCP/IP for IPC on the same machine. – Pintail

@Dan, Isn't IPC supposed to be faster than TCP? – Fancie

Might be now.. but certainly was not the case with earlier versions of windows. Vista for example, which was my primary source of info. – Dettmer

Answer 5

In the scenario you describe, the local TCP connections are very unlikely to be a bottleneck. It will introduce some overhead, of course, but this should be negligible unless your CPU is already running hot.

At a guess, if your server's CPU usage is normally below 50% or so (with the proxy in place) it isn't worth worrying about minimizing the overhead associated with the local TCP connections.

If CPU usage is regularly above 80% you should probably be doing some profiling. I'd start by comparing the CPU load (or, better still, the performance, if you can measure it meaningfully) when the proxy is in place to when it isn't. Unless the proxy is doing some complicated processing, the overhead associated with the extra TCP connections is probably a significant fraction of the total overhead introduced by the proxy, so that should give you at least an order-of-magnitude estimate of how much you'd gain by using a more efficient form of IPC.

Relent

Answer 6

What is the reason to have a proxy on the SAME machine, just curious?

Anyway:

There are several methods for IPC, TCP/IP, named Pipes are comparable in speed and complexity. If you really want something that scales well and has almost no overhead: use shared memory. Best used in combination with a lock free algorithm for advancing the pointers (or use one buffer for each reader (the proxy/the service) and writer(the service/the proxy)).

Trudeau

We have one server only, we can not run the programs on different machines. – Syzran

@vz0: That figures.. well. What Dan says: if you use TCP/IP for IPC you can also cross machine boundaries later when needed; otherwise shared mem is the fastest. – Trudeau

Named Pipes are implemented using shared memory (MMF) on Windows, so the point is kinda moot :) – Hypogynous

@STATUS_ACCESS_DENIED: No it's not! You still have the overhead of memory buffer management, going to kernel and back for BOTH processes and the syncronization used. When using your own buffer you can be magnitutes faster, especially when using quite small messages (eg 100 bytes and less). – Trudeau

@Ritsaert Hornstra: please be precise: it's not implemented by means of MMF (if that's your claim, please provide credible sources) or the overhead is bigger than raw MMF? Well obviously the overhead exists and is bigger than no overhead. I won't even argue that. – Hypogynous

@s_a_d: Lot's of examples. A bit crippled but: codeproject.com/Articles/14740/…. But for you: YOU claim the piont is kinda moot: prove it. I have working code here that proves otherwise. – Trudeau