Simulate delayed and dropped packets on Linux

Question

I would like to simulate packet delay and loss for UDP and TCP on Linux to measure the performance of an application. Is there a simple way to do this?

Ardyce

This tutorial on networking physics simulations contains a C++ class in the sample code for simulating latency and packet loss in a UDP connection and may be of guidance. See the public latency and packetLoss variables of the Connection class found in the Connection.h file of the [downloadable source code](web.archive.org/web/20160103025448/https://… – Romanticism

See also/dupe: Network tools that simulate slow network connection – Fiden

For OS X see: superuser.com/questions/173882/… – Evelineevelinn

Answer 1

netem leverages functionality already built into Linux and userspace utilities to simulate networks. This is actually what Mark's answer refers to, by a different name.

The examples on their homepage already show how you can achieve what you've asked for:

Examples

Emulating wide area network delays

This is the simplest example, it just adds a fixed amount of delay to all packets going out of the local Ethernet.

# tc qdisc add dev eth0 root netem delay 100ms

Now a simple ping test to host on the local network should show an increase of 100 milliseconds. The delay is limited by the clock resolution of the kernel (Hz). On most 2.4 systems, the system clock runs at 100 Hz which allows delays in increments of 10 ms. On 2.6, the value is a configuration parameter from 1000 to 100 Hz.

Later examples just change parameters without reloading the qdisc

Real wide area networks show variability so it is possible to add random variation.

# tc qdisc change dev eth0 root netem delay 100ms 10ms

This causes the added delay to be 100 ± 10 ms. Network delay variation isn't purely random, so to emulate that there is a correlation value as well.

# tc qdisc change dev eth0 root netem delay 100ms 10ms 25%

This causes the added delay to be 100 ± 10 ms with the next random element depending 25% on the last one. This isn't true statistical correlation, but an approximation.

Delay distribution

Typically, the delay in a network is not uniform. It is more common to use a something like a normal distribution to describe the variation in delay. The netem discipline can take a table to specify a non-uniform distribution.

# tc qdisc change dev eth0 root netem delay 100ms 20ms distribution normal

The actual tables (normal, pareto, paretonormal) are generated as part of the iproute2 compilation and placed in /usr/lib/tc; so it is possible with some effort to make your own distribution based on experimental data.

Packet loss

Random packet loss is specified in the 'tc' command in percent. The smallest possible non-zero value is:

2⁻³² = 0.0000000232%

# tc qdisc change dev eth0 root netem loss 0.1%

This causes 1/10th of a percent (i.e. 1 out of 1000) packets to be randomly dropped.

An optional correlation may also be added. This causes the random number generator to be less random and can be used to emulate packet burst losses.

# tc qdisc change dev eth0 root netem loss 0.3% 25%

This will cause 0.3% of packets to be lost, and each successive probability depends by a quarter on the last one.

Prob_n = 0.25 × Prob_n-1 + 0.75 × Random

Note that you should use tc qdisc add if you have no rules for that interface or tc qdisc change if you already have rules for that interface. Attempting to use tc qdisc change on an interface with no rules will give the error RTNETLINK answers: No such file or directory.

Aegisthus

The original website has that error, I just copied that text directly. But yes, 2^(-32)=2.33e-10 – Aegisthus

Note that tc -p qdisc ls dev eth0 will list current defined rules, and tc qdisc del dev eth0 root will delete them – Junji

upvoted for pointing out error when trying to change a non existing entry – Mohamed

Do you know why I get these errors? ubuntu@anmol-vm1-new:/home/hadoop/yarnpp/workloads/RESULTS$ sudo tc qdisc add dev eth0 root netem delay 100ms RTNETLINK answers: File exists ubuntu@anmol-vm1-new:/home/hadoop/yarnpp/workloads/RESULTS$ sudo tc qdisc change dev eth0 root netem delay 100ms RTNETLINK answers: Invalid argument serverfault.com/questions/743885/… – Musso

It would probably be a good idea to include details how to limit entire interfaces throughput to simulate low speed links – Dashed

@MonaJalal: try the delete rule first: tc qdisc del dev eth0 root (sometimes, if you ask a new question, the answer is faster) – Anthia

@Aegisthus Can something similar be done on an IP level? E.g. drop and delay packets for a single IP address? stackoverflow.com/questions/46945705 – Postbox

Command works fine. But /proc/net/dev dropped packet counter shows still zero. – Fearful

If tc is too complex for u, try: github.com/CovenantSQL/GNTE – Searby

In case you don't want to barf your existing (and possibly) only connection. You can do this on a VLAN or Tunnel interface level as well. – Complicacy

This only applies on outgoing packets, which isn't suitable to simulate network loss. iptables applies correctly to the incoming traffic. – Monroemonroy

There is a high-level interface written in Python: github.com/thombashi/tcconfig It simplifies the usage of tc. It can also show the currently active filters in JSON style. – Sulphurate

netem cannot be used together with other qdiscs though (this is a well-documented limitation) ☹ – Balbinder

I had to use replace instead of change (bbs.archlinux.org/viewtopic.php?id=238231). If this is the case generally, the answer should be changed. – Toulouselautrec

Answer 2

For dropped packets I would simply use iptables and the statistic module.

iptables -A INPUT -m statistic --mode random --probability 0.01 -j DROP

Above will drop an incoming packet with a 1% probability. Be careful, anything above about 0.14 and most of you tcp connections will most likely stall completely.

Undo with -D:

iptables -D INPUT -m statistic --mode random --probability 0.01 -j DROP 

Take a look at man iptables and search for "statistic" for more information.

Mureil

Why would TCP connections stall at anything above 14%? – Grimona

@DavidWolever: Because of the way the tcp sliding windows size is adjusted. But the 14% is purely from experience, git it a try yourself and you will see that ssh becomes mostly unusable at 14% and above, but actually works quite well at lower levels of packet drop rates. – Mureil

For safety, it is probably best to limit the rule to apply only to the port(s), which you want tested: iptables -A INPUT --dport FOO -m statistics .... This way, your ssh and other connections will remain unmolested and you can bump the drop-rate for the service of interest to be able to reproduce any problems with it faster. – Outland

Note that DROP on outbound connections rather ridiculously causes send() operations to return EPERM, rather then just dropping packets (like it should). – Legged

Is this all that is needed to undo that command? iptables -D INPUT -m statistic --mode random --probability 0.01 -j DROP – Vesiculate

@BjarkeFreund-Hansen But /proc/net/dev dropped packet counter shows still zero! – Fearful

@VijaySB: No packets are dropped by the network device, they never reach it, they are dropped much earlier by iptables. – Mureil

@MikhailT. to use --dport you must also specify the protocol: -p tcp – Fleischer

Answer 3

iptables(8) has a statistic match module that can be used to match every nth packet. To drop this packet, just append -j DROP.

Precentor

Answer 4

One of the most used tool in the scientific community to that purpose is DummyNet. Once you have installed the ipfw kernel module, in order to introduce 50ms propagation delay between 2 machines simply run these commands:

./ipfw pipe 1 config delay 50ms
./ipfw add 1000 pipe 1 ip from $IP_MACHINE_1 to $IP_MACHINE_2

In order to also introduce 50% of packet losses you have to run:

./ipfw pipe 1 config plr 0.5

Here more details.

Brookes

Answer 5

Haven't tried it myself, but this page has a list of plugin modules that run in Linux' built in iptables IP filtering system. One of the modules is called "nth", and allows you to set up a rule that will drop a configurable rate of the packets. Might be a good place to start, at least.

Hohenlohe

Answer 6

An easy to use network fault injection tool is Saboteur. It can simulate:

• Total network partition
• Remote service dead (not listening on the expected port)
• Delays
• Packet loss -TCP connection timeout (as often happens when two systems are separated by a stateful firewall)

Affirmatory

Sadly, the last commit to that project was on Aug 28, 2015, that is, almost 4 years ago. The open issues are now 5 years old. – Dogtooth