what is benefit from socket reuse in C#

Question

I am developing open source socket server library: https://sourceforge.net/projects/socketservers/

And I would to like to add socket reuse feature to this lib. I have implement draft of this feature, but I do not see any benefits in my tests. The client makes 32K connect-disconnects by 8 items to the server and measure the time. But here is no difference between reusing socket and not reusing socket - same time elapsed for this test.

What I am doing wrong in test?

What benefit should server get when reuse sockets, and how to measure this benefit?

Primine

Answer 1

The question is if the OS or the runtime does not perform the reuse automatically when invoking a new socket.
The Socket.Disconnect Method documentation points into this direction:

Closes the socket connection and allows reuse of the socket.

So this seem to be an over-optimization.

Singleaction

The msdn has no any point about automatically socket reusing in winapi and .net Socket class. – Primine

Thank you for reply, I do not understand where is point in Socket.Disconnect? I use Socket.Disconnect(true) for socket reusing and Socket.Close when do not reuse socket. – Primine

For each connection a TCP Control Block (TCB – Data structure using 0.5 KB pagepool and 0.5 KB non-pagepool) is maintained. The TCBs are pre-allocated and stored in a table, to avoid spending time on allocating/deallocating the TCBs every time connections are created/closed. The TCB Table enables reuse/caching of TCBs and improves memory management, but the static size limits how many connections TCP can support simultaneously (Active + TIME_WAIT). – Primine

I can not find what winsock doing when all TCB are used, but it looks like before winsock reuse socket itself. – Primine

Answer 2

I can explain what happens from an unmanaged point of view and how DisconnectEx() is used, perhaps someone can then map this to the managed scenario.

In unmanaged code you would use DisconnectEx() to reuse a socket for an subsequent AcceptEx() or ConnectEx() call, more likely the former. So you'd initially create x sockets and post your overlapped async accept operations using AcceptEx(). When clients connect to these pending connection you would do your server stuff and then at the end call DisconnectEx() on the socket and post a new AcceptEx() using that socket. This avoids the need to create a new socket at this point and it's thus more efficient for the server. The performance difference is probably pretty small but worth having on heavily loaded servers that are accepting lots of short lived connections.

So I suggest you post some code showing how you're reusing your socket after calling Disconnect(true) on it...

Clot

Thank you. I could upload development version to the sourceforge's SVN, but I use several classes for socket reusing and it require a some time understand the logic. – Primine

I am agree with your theory. I have implement it in this way (except I do not create sockets before, but reuse created by Accept). But I do not any performance difference in my test. The test clients create 4096 times 8 short lived connection - no difference. – Primine

Are sure that socket reusing has any performance benefit in real application? – Primine

How do you reuse the sockets created by accept? How many concurrent connections, how quickly are the connections occurring? How loaded is the CPU? etc. As I said, the perf increase is likely pretty small. In unmanaged code you ARE avoiding some work by reusing the socket and so you should get more other work done because of that. – Clot

Yes, I understand that it depends on many factors. Is there simple test for this? I have measure (new Socket() & Bind) actions and Disconnect(reuse) in .net application - no difference almost ( ~5% ). – Primine

What about DisconnectEx, how fast is it vs new socket&bind? – Primine

Of course there's not a simple test for it. All you can do is profile a server that uses both styles under a variety of loads on a variety of platforms. In general, if written correctly, reusing the socket should work out better in terms of performance I would expect. Also it will depend on how you're shutting down the connection, if you're performing the active close then your disconnect will take the time wait period to complete before the socket can be reused... – Clot

>active close then your disconnect will take the time wait period to complete before – Primine

Do you talk about TIME_WAIT? I guess, it is related to {ip:port} tuple not to socket resources? – Primine

"there's not a simple test for it." It is very sadly, so we have only theory and no real confirmation for all this assumptions. – Primine

Of course it is possible that I have any bugs in implementations, but if here is no test at all how to validate it? – Primine

What about that .net Disconnect(for reusing) take same time like new Socket() & Bind() ? – Primine

My comments re time wait are from the DisconnectEx() msdn page... I think the key point about my "not a simple test for it" is that the testing isn't simple not that you cant test it. – Clot

msdn: While in the TIME_WAIT state, a socket pair cannot be re-used. – Primine

Yes, msdn has some confusion. But I believe that the socket pair is IP and port, not resources of socket. – Primine

Thank you for your comments. I really need a opponent for continue experiments. – Primine

I agree it's not clear, but "Note The socket level disconnect is subject to the behavior of the underlying transport. For example, a TCP socket may be subject to the TCP TIME_WAIT state, causing the DisconnectEx call to be delayed." implies that the DisconnectEx could be delayed by the time wait. So I think my question about who's doing the active close in your tests is still valid. I don't think I can help further without seeing some code snippets of how you reuse the socket. – Clot

A client is doing active close. – Primine

I have upload new version w/ socket reusing to sourceforge.net/projects/socketservers , you could find socket reusing in BasTcpServer.cs and SocketRecycle.cs files. Thanks. – Primine

I think you're making things too complex and doing too much work around the socket reuse. It probably defeats the perf advantage of reuse, especially if you reuser locks a collection of reusable sockets as you're adding contention in there which otherwise might not exist. – Clot

I am using lock-free pool. But it was too complex before I add reuse. So I expect to see difference between complex solution w/o reuse & complex solution w/ reuse. What exactly complex things do you mean - I just add lock-free pool for storing sockets? – Primine

Have you seen my comment about TCB reusing by a system? Actually the Windows can reuse socket resources itself. – Primine

Given that socket reuse was added so that when issuing an AcceptEx from an IOCP thread you don't need to do anything 'expensive' (pp 188 Network Programming for MS Windows) and given that DisconnectEx came along after TransmitFile which first had the 'disconnect for reuse' option. I think it's safe to say that the intended usage of disconnected sockets is for subsequent reuse in an AcceptEx that is issued straight after the disconnect completes... So, in summary, I think you're using them wrong, so perhaps that's why you don't get the perf gain you expect. The TCB stuff is not, IMHO relevant. – Clot

Maybe but not in my case, the test client create-destroy connection w/o delay, so it looks like the server use disconnected socket almost after disconnect(). But can you offer algorithm for subsequent reuse? I guess it is impossible in real application. – Primine

You're trading socket creation time for resource usage. If you reuse straight away this is a worthwhile trade, IMHO, if you add complexity with pooling the sockets for reuse and delay their reuse then you probably may as well not bother unless you can get noticeable perf gains, which, it seems, you cant. Reuse with subsequent accepts is pretty straight forward. – Clot

I assume you got the TCB info from here ssmax.net/archives/916.html or the KB article that it links to (useful stuff by the way). This is at a lower level than we're talking here. Yes there will be a TCB data structure related to the socket that you're reusing but the idea behind reuse is to speed up the allocation of the socket structures and to reduce the time involved in going through the winsock layered providers to allocate a new socket, etc. – Clot

What is socket structure can you give some ref? I suppose, the TCB is socket structure. – Primine

I can not imagine straight way (w/o socket pool) for real server application. Ok, connection is closed, what I should to do with disconnected socket? – Primine

Answer 3

In case you mean something like SO_REUSEADDR or SO_REUSEPORT:

Socket reuse is essentially important if e.g. your server crashes but there are still connections lingering.

If you restart your server, you'd normally have to wait till the operating system gracefully closed those connections, before you can rebind your socket to that port.

This could mean, that some processes which heavily rely on your server to come to a halt till it has been restarted.

Due to the socket reuse feature, you circumvent this problem.

There might be other uses for this, but I can only think of this one right now. Hope that helped.

Dhar

Thank you. No, I mean DisconnectEx with TF_REUSE_SOCKET flag instead closesocket function in win api and then use disconnected socket again. Or I use Socket.Disconnect(true) in .net – Primine