What % of traffic is network overhead on top of HTTP/S requests

Question

If we:
1) Count bytes/bits at the network adapter level (raw # of bits through the NIC) and,
2) Count bytes in all HTTP/S request/responses.

Assuming only HTTP/S traffic is on the box, and assuming a statistically relevant amount of "typical" web traffic:

I want to know about how much more traffic will be counted at the NIC level than at the HTTP/S level (counting http headers and all) because of the extra network overhead.

Dispossess

For just the bits/bytes of the TCP/IP overhead (so NOT including the additional overhead of HTTP/S, nor computing the overhead as a %), I found answer #8903083 helpful. – Rosaleerosaleen

Answer 1

You have zero knowledge about the layers below HTTP. You can't even assume the HTTP request will be delivered over TCP/IP. Even if it is, you have zero knowledge about the overhead added by the network layer. Or what the reliability of the route will be and what overhead will be due to dropped/resent packets.

Update: Based on your comment, here are some back-of-the-napkin estimates:

The maximum segment size (which does not include the TCP or IP headers) is typically negotiated between the layers to the size of the MTU minus the headers size. For Ethernet MTU is usually configured at 1500 bytes. The TCP header is 160 bits, or 20 bytes. The fixed part of the IPv4 header is 160 bits, or 20 bytes as well. The fixed part of the IPv6 header is 320 bits, or 40 bytes. Thus:

• for HTTP over TCP/IPv4

overhead = TCP + IP = 40 bytes

payload = 1500 - 40 = 1460 bytes

overhead % = 2.7% (40 * 100 / 1460)

• for HTTP over TCP/IPv6

overhead = TCP + IP = 60 bytes

payload = 1500 - 60 = 1440 bytes

overhead % = 4.2% (60 * 100 / 1440)

Here are the assumptions:

• Amazon counts the NIC payload without the Ethernet headers, not the whole NIC packet
• your HTTP responses are fully utilizing the TCP/IP packet - your typical page size + HTTP headers results in one or more full TCP/IP packets and one with more than 50% used payload
• you set explicit expiration date on cached content to minimize 302 response
• you avoid redirects or your URLs are long enough to fill the payload

Fumarole

My question arises because I run a server on amazon's EC2 cloud. They count bytes at the NIC, I get a log of bytes at the HTTP level on my server. I would like to take a guestimate at determining how much more they will count than I will see on the logs. Given all the variables, I just want to plug a reasonable number into some estimates. Assuming a high volume of typical traffic this should be possible. – Dispossess

That's exactly the rundown I was looking for, and the assumptions you listed provided some great food for thought. Really appreciate it! – Dispossess

I have the same question, except that I am using file transfer between two linux systems over Bluetooth (python http server at server side, wget at client side, physical interface is bluetooth). And then there's bluetooth headers as well. Any rough idea? – Nightclub

20 bytes is the minimum TCP header size; in reality, due to TCP options like NOP and Timestamping, the TCP header is usually around 32 bytes long. – Braca

There's quite a bit of other overhead too, handshake to set up and tear down a TCP connection and ACKs for TCP segments. – Eada

Answer 2

With 100Mbit/s Ethernet, a large file transfers at 94.1Mbit/s.

That's 6% overhead. If you also count the TCP ACKs flowing in the opposite direction, it's closer to 9%. For gigabit Ethernet the overhead (in percent) remains the same. Assumptions: TCP/IPv4 and file size >100kB. (At this size we can neglect the initial HTTP and TCP setup.)

When comparing download rates, beware the factor 8 from bits to bytes. I guess nobody will charge you for Ethernet preamble or interframe gap, but "payload" shouldn't be taken literally.

Calculation: payload / overall
payload = 1500 - 20 - 32 (Ethernet_MTU - IPv4 - TCP)
overall = 8 + 14 + 1500 + 4 + 12 (Preamble + Ethernet_header + Ethernet_MTU + CRC + Interframe_gap)

Because Ethernet is always full-duplex these days, the occasional TCP ACK flowing the other way does not change the transfer rate. If you add one ACK for every two data frames to the overhead (that's what I observed in Wireshark), you get 8.5% total overhead. And while the MTU size is usually 1500 bytes, it can be smaller in some networks, or much larger if every piece of equipment in the path is configured for it.

Likely

Answer 3

What extra network overhead? the TLS overhead on top of the HTTP amounts to the key exchange. It's mostly processing overhead that you notice.

http://en.wikipedia.org/wiki/HTTP_Secure#Difference_from_HTTP

Down the line (after the server) wan accelerator or proxies will treat your traffic differen't as it isn't cacheable or compressible.

Lifeless

I mean to ask about the TCP/IP overhead not HTTPS vs HTTP. Assuming a typical amount of HTTP and HTTPS traffic, what is the typical TCP/IP overhead that would be counted at the NIC level monitoring vs. the amount of HTTP bytes that would be counted by a server. – Dispossess