How to reproduce a silently dropped TCP/IP connection?

Question

I have a situation where a Java program establishes a long-running TCP/IP connection to a server and said connection is functioning exactly as documented, except for one thing in an environment outside my control.

Every Saturday that server is rebooted, but for some reason this is not communicated properly to my client so the connection just hangs waiting for a response for ever and ever. This is exactly the same behaviour seen with stale JDBC-connections, where some router in between server and client has found the connection to be idle and dropped it without notification.

I need to be able to reproduce this scenario to be able to create a proper fix.

Hence I need to know if there is a good way to simulate a router silently dropping connections under my control?

I can put a Linux box or a Mac between my development environment and the server. I can run the JVM in a virtual machine (vmware player) if I can do the dropping there.

Suggestions are very welcome.

---

EDIT 2012-12-18: The solution that worked for me was to use a Java-based SOCKS5-proxy locally which could be suspended with Ctrl-Z at will, and tell my application to go through it.

Video

Are you using TCP Keepalives on the client and server's socket (SO_KEEPALIVE option)? – Mesoblast

@Mike, likely not. The socket code is in a third-part library. – Turnage

Well, if clients hanging indefinitely is unacceptable, it sounds like keepalives would be useful. – Mesoblast

The proper fix is for the client to set TCP keepalive, if you don't mind it waiting two hours, or else call Socket.setSoTimeout() with a more realistic timeout value, e.g. a few minutes. – Donettedoney

@Mike, most likely, except that I cannot change said third-part library, unfortunately. – Turnage

Were you able to fix the hanging? – Excusatory

@Excusatory I was able to create a software workaround based on the technique described here. – Turnage

Answer 1

Use Socat to forward your connection (it's very easy to set up as a TCP forwarder/proxy) even on the same machine if you want (by introducing a new port for the proxy to operate on and pointing it to the 'official' endpoint)

Check out the simple TCP forwarder example

Then when you want to test outages you can either kill it (and the socket will be closed) or potentially try stopping the process (CTRL Z linux) to see how your destination handles stalled connections.

Any questions? I'll try to answer... Good luck!

Aridatha

I am familiar with socat - I use it for Remote Desktop through a SOCKS proxy. – Turnage

It turned out that the scenario involved multiple network ports, and it was easier to use JSocks (jsocks.sourceforge.net) than socat to handle them correctly as the -DsocksProxyHost=... system property transparently instructs the JVM to use the Socks proxy. I use the Ctrl-Z trick though - works very nicely. – Turnage

Ah cool! Thanks for sharing, I'll check it out next time Im playing with connection logic :) – Aridatha

I tried JSocks on Windows 7 with Java 7. It's easy to use but it triggers immediately a "SocketException: Connection reset" when killed. So in my case it wasn't appropriate to silently drop the connection. – Likable

I got the same result with socat, "Connection reset". – Likable

Answer 2

You should be able to temporarily apply a rule using iptables to drop all packets coming from the remote server (a.b.c.d). Something along the lines of :

iptables -A INPUT -s a.b.c.d -j DROP

When you want to turn the filter off

iptables -D INPUT -s a.b.c.d -j DROP

You can play with those rules to drop other specific packets as well, but I think this will accurately model what your client software would see if the server simply disappeared.

Note though, that when you turn the connection back on, the remote server may still have the old connection alive, so this won't perfectly simulate a reboot.

Prankster

Or block the TCP connection with Windows Firewall sometime after it has been established – Mesoblast

This will not work for connections that are already established. – Waal

Answer 3

A native Linux based solution is to use tc with netem. Netem is a special queue that can be attached to various network interfaces to cause delay, reordering and loss (see the web page for details). Some examples of netem use include:

tc qdisc change dev eth0 root netem loss 0.1%

To make eth0 drop 0.1% of all outgoing packets, and

tc qdisc change dev eth0 root netem loss 0.3% 25%

To create bursts of packet loss:

This will cause 0.3% of packets to be lost, and each successive probability depends by a quarter on the last one.

If you need to drop incoming packets as well, you can use the ifb (Intermediate Functional Block device). You set it up, forward all packets received by eth0 to the ifb, and attach netem to the ifb to create loss or delay. See ifb's documentation for more details.

Zachariah

Answer 4

Unplug the network cable from your computer.

Bring up a simple test of just this module. Connect to the server and get to the waiting part. Then, pull out your network plug.

It's important, of course, that you isolate this module and bring up only this module when you run this test. There may be other aspects of your program that require network access, and these should not be running.

Pronator

Answer 5

on Windows you can run clumsy: https://github.com/jagt/clumsy/releases no need to setup explicit proxies, just configure it to drop packets on whatever interface/port combo you need. it's a very handy tool.

Asphodel

Nice to see a good tool for this has been written since. – Turnage