What is the optimal size of a UDP packet for maximum throughput?

Question

I need to send packets from one host to another over a potentially lossy network. In order to minimize packet latency, I'm not considering TCP/IP. But, I wish to maximize the throughput uisng UDP. What should be the optimal size of UDP packet to use?

Here are some of my considerations:

• The MTU size of the switches in the network is 1500. If I use a large packet, for example 8192, this will cause fragmentation. Loss of one fragment will result in the loss of the entire packet, right?

• If I use smaller packets, I'll incur the overhead of the UDP and IP header

• If I use a really large packet, what is the largest that I can use? I read that the largest datagram size is 65507. What is the buffer size I should use to allow me to send such sizes? Would that help to bump up my throughput?

• What are the typical maximum datagram size supported by the common OSes (eg. Windows, Linux, etc.)?

Updated:

Some of the receivers of the data are embedded systems for which TCP/IP stack is not implemented.

I know that this place is filled with people who are very adament about using what's available. But I hope to have better answers than just focusing on MTU alone.

Woodard

At the customer site, the network load is unpredictable, and can be very high, resulting many losses. But each time we test, we get different results. We can't repeat the traffic patterns at our lab setup. And, there are limits on how much testing we can do at the customer's. – Woodard

You might want to look at doing some network impairment. We found a cheap and reasonably good product called the mini-maxwell <iwl.com/content/blogcategory/33/123>. – Crandell

are you requiring the far end to acknowledge receipt of the packets? that'll have more effect on latency than MTU issues. – Glennaglennie

See also #1099397 – Uranyl

Answer 1

The best way to find the ideal datagram size is to do exactly what TCP itself does to find the ideal packet size: Path MTU discovery.

TCP also has a widely used option where both sides tell the other what their MSS (basically, MTU minus headers) is.

Halfway

Would discovering the MTU give me the best datagram performance? – Woodard

@sep61: if it did not, there would be no reason for TCP to use PMTUD. Discovering the PMTU has a cost, but the implementors of TCP felt the benefits justfied the costs. – Halfway

Downside to PMTUD is when overzealous and underinformed sysadmins break it by disabling ALL ICMP on their firewalls. – Surrogate

Path MTU Discovery is an IP functionnality, as far as I remember. So it should be also available to UDP. – Everard

Answer 2

Alternative answer: be careful to not reinvent the wheel.

TCP is the product of decades of networking experience. There is a reson for every or almost every thing it does. It has several algorithms most people do not think about often (congestion control, retransmission, buffer management, dealing with reordered packets, and so on).

If you start reimplementing all the TCP algorithms, you risk ending up with an (paraphasing Greenspun's Tenth Rule) "ad hoc, informally-specified, bug-ridden, slow implementation of TCP".

If you have not done so yet, it could be a good idea to look at some recent alternatives to TCP/UDP, like SCTP or DCCP. They were designed for niches where neither TCP nor UDP was a good match, precisely to allow people to use an already "debugged" protocol instead of reinventing the wheel for every new application.

Halfway

Related question with other alternatives in the answers: #108168 – Halfway

This is more a comment on the question than an answer to the question. Sometimes, you're just required to do this, for instance out of historical reason: an application has been using UDP for decades, and the scope of usage is changing, and you need to adapt the UDP communication to something new. You don't want to lose the decades of experience your team put into the application protocol, so it might look like reimplementing the wheel to outsiders, whereas it is just making an already existing wheel smarter. – Everard

Not an answer. "How do I do A?" - "Do B." I despise this behaviour. – Kylynn

@Kenji: see "Pounding A Nail: Old Shoe or Glass Bottle?" weblogs.asp.net/alex_papadimoulis/408925 – Halfway

@Halfway "Some of the receivers of the data are embedded systems for which TCP/IP stack is not implemented." there is no hammer available. Assume that if someone asks questions like that, they have carefully considered the alternatives and only the proposed solutions are available. It is utterly disrespectful to disregard the constraints of the question. – Kylynn

@Kenji: first, that part of the question wasn't there when this answer was written (check the question edit history). Second, I pointed to alternatives (SCTP, DCCP) which most people don't think of, so I can't assume they have been considered. And, as the link I posted explains, sometimes the problem is with the constraints of the question: is the use of UDP a real constraint, or is it just tunnel vision? In this question, after the edit, it's probably a real constraint, but even then the algorithms from SCTP/DCCP could be useful. – Halfway

TCP is, more often than not, the poorest choice and a good amount of technical debt, plus pointless most of the time as there is a lot of data for which you don't want an ack or a retry. Plus, you can write a proper implementation of an UDP protocol with the berkeley sockets in a couple of days, so I wouldn't recommend TCP for other than novices. – Interradial

Answer 3

Well, I've got a non-MTU answer for you. Using a connected UDP socket should speed things up for you. There are two reasons to call connect on your UDP socket. The first is efficiency. When you call sendto on an unconnected UDP socket what happens is that the kernel temporarily connects the socket, sends the data and then disconnects it. I read about a study indicating that this takes up nearly 30% of processing time when sending. The other reason to call connect is so that you can get ICMP error messages. On an unconnected UDP socket the kernel doesn't know what application to deliver ICMP errors to and so they just get discarded.

Gaptoothed

Answer 4

The easiest workaround to find mtu in c# is to send udp packets with dontfragment flag set to true. if it throws an exception, try reduce the packet size. do this until there is no exception thrown. you can start with 1500 packet size.

Mert

No the easiest way is to use ping – Isham

lttlrck. youre incorrect. udp and ping may have different header size. much so with windows vs linux. so, the most foolproof method to find max packet size in udp environment is by sending 'dontfragment packets' in UDP environment. – Mert

Whether it is UDP/TCP/SCTP makes no difference at all to the MTU. Using ping with the 'no fragment' (-f on Windows) is a well known way to discover the MTU. – Isham

Answer 5

IP header is >= 20 bytes but mostly 20 and UDP header is 8 bytes. This leaves you 1500 - 28 = 1472 bytes for you data. PATH MTU discovery finds the smallest possible MTU on the way to destination. But this does not necessarily mean that, when you use the smallest MTU, you will get the best possible performance. I think the best way is to do a benchmark. Or maybe you should not care about the smallest MTU on the way at all. A network device may very well use a small MTU and also transfer packets very fast. And its value may very well change in the future. So you can not discover this and save it somewhere to use later on, you have to do it periodically. If I were you, I would set the MTU to something like 1440 and benchmark the application...

Goldfish

Answer 6

Another thing to consider is that some network devices don't handle fragmentation very well. We've seen many routers that drop fragmented UDP packets or packets that are too big. The suggestion by CesarB to use Path MTU is a good one.

Maximum throughput is not driven only by the packet size (though this contributes of course). Minimizing latency and maximizing throughput are often at odds with one other. In TCP you have the Nagle algorithm which is designed (in part) to increase overall throughput. However, some protocols (e.g., telnet) often disable Nagle (i.e., set the No Delay bit) in order to improve latency.

Do you have some real time constraints for the data? Streaming audio is different than pushing non-realtime data (e.g., logging information) as the former benefits more from low latency while the latter benefits from increased throughput and perhaps reliability. Are there reliability requirements? If you can't miss packets and have to have a protocol to request retransmission, this will reduce overall throughput.

There are a myriad of other factors that go into this and (as was suggested in another response) at some point you get a bad implementation of TCP. That being said, if you want to achieve low latency and can tolerate loss using UDP with an overall packet size set to the PATH MTU (be sure to set the payload size to account for headers) is likely the optimal solution (esp. if you can ensure that UDP can get from one end to the other.

Crandell

Answer 7

Even though the MTU at the switch is 1500, you can have situations (like tunneling through a VPN) that wrap a few extra headers around the packet- you may do better to reduce them slightly, and go at 1450 or so.

Can you simulate the network and test performance with different packet sizes?

Starkey