Why tc cannot do ingress shaping? Does ingress shaping make sense?

Question

In my work, I found tc can do egress shaping, and can only do ingress policing. I wonder that why tc doesn't implement ingress shaping?

Code sample:

#ingress
tc qdisc add dev eth0 handle ffff: ingress
tc filter add dev eth0 parent ffff: protocol ip prio 50 \
   u32 match ip src 0.0.0.0/0 police rate 256kbit \
   burst 10k drop flowid :1
#egress
tc qdisc add dev eth0 root tbf \
   rate 256kbit latency 25ms burst 10k

But I can't do this:

#ingress shaping, using tbf
tc qdisc add dev eth0 ingress tbf \
   rate 256kbit latency 25ms burst 10k

I found a solution called IFB(updated IMQ) can redirect the traffic to egress. But it seems not a good solution because it's wasting CPU. So I don't want to use this.

Does ingress shaping make sense? And why tc doesn't support it?

Overlie

Answer 1

Although tc shaping rules for ingress are very limited, you can create a virtual interface and apply egress rules to it, as described here:

https://serverfault.com/questions/350023/tc-ingress-policing-and-ifb-mirroring

(You may not need the virtual interface if your VMs already use virtual interfaces and you can apply tc to them.)

The caveat with ingress shaping is that it may take a long time for an incoming stream to respond to your shaping actions, due to all the buffers in routers between the stream source and your interface. And until the stream does respond to a reduced limit, it will continue to flood your downstream! Meanwhile you will be throwing away good packets, reducing your throughput.

Likewise when a high-priority stream ends or drops off, it will take some time for the low-priority stream to grow back to its full rate. This can be quite disruptive if it happens often!

The result of this is that dynamic shaping may work as desired for groups of steady rate long-lived streams, but will offer little advantage to short-lived or varying rate high-priority streams when your downstream is flooded: the low-priority streams will simply take too long to back off. However classifying and limiting low and medium-priority packets to a static rate somewhere below your maximum downrate could be helpful, to guarantee at least some space for high-priority data.

I don't have any figures on this, and latency has improved a lot since the ADSL days. So I think it may be worth testing, if low latency or high throughput of high-priority packets is something you desire more than overall throughput, and you can live with the limitations above.

As Janoszen and the ADSL HOWTO mention, streams could respond much more quickly if we could adjust the TCP window size as part of the shaping.

Search TLDP for further research.

Galumph

Arguably, limiting incoming traffic is the whole point. If traffic is TCP there classic way to slow the sender is not to return ACK, which is accomplished, albeit hackily, by dropping incoming packets, then recv window is incomplete and ACK is not sent (or sent for subrange only). Nowadays there's ECN, but that indeed requires to send something back. Per @janoszen's answer that's not possible. So what's best then? ifb+ecn? dropping traffic? doing nothing? – Martyrdom

I used this technique when I was stuck on a 16kb/s 2G connection in Malaysia (using a "Yes huddle"). Without shaping, a large download would flood the interface and prevent any new connections from being established. But if I used ingress shaping with a max rate of 12kb/s then even under load, new connections could establish in a timely manner. (I did also delve into setting up classes and deprioritising heavy long-lived streams, but that's story for another time...) – Galumph

See also: networkengineering.stackexchange.com/a/54404/36597 (IFB+ECN+fq_codel) – Elkin

Answer 2

Shaping works on the send buffer. Ingress shaping would require control over the remote send buffer.

Fractious

I didn't find any indication the TCP window size could be changed with TC. Also, this is a TCP-specific option and only useful for a single stream, so if a remote host were to open multiple connections, it still would't work very well. There is a workaround involving a dummy interface and some sick hacks, but I can't seem to find it any more. – Fractious

yes. But in our scenes, we need to limit the VMs' traffic, either ingress and egress. Maybe a good thought is trying to solve this in VMs. But it seems to be a hard job. – Overlie

In that case it's a little different since you are in control of the guest's kernel. I suggest you post a new question with the specifics about your virtualization. – Fractious

Docs say that "ingress" queue can drop, but not delay packets... Seems on par but slightly different than what you say. – Martyrdom