What causes a TCP/IP reset (RST) flag to be sent?

Question

I'm trying to figure out why my app's TCP/IP connection keeps hiccuping every 10 minutes (exactly, within 1-2 seconds). I ran Wireshark and discovered that after 10 minutes of inactivity the other end is sending a packet with the reset (RST) flag set. A google search tells me "the RESET flag signifies that the receiver has become confused and so wants to abort the connection" but that is a little short of the detail I need. What could be causing this? And is it possible that some router along the way is responsible for it or would this always come from the other endpoint?

Edit: There is a router (specifically a Linksys WRT-54G) sitting between my computer and the other endpoint -- is there anything I should look for in the router settings?

Extant

Here's another: Comcast – Nitramine

Heh luckily I don't have a dependency on Comcast as this is occurring within a LAN. I wish I could shift the blame that easily tho ;) – Extant

Did you ever get this figured out? I can't comment because I don't have enough points, but I have the same exact problem you were having and I am looking for a fix. – Opossum

What service this particular case refers to? It may be possible to set keepalive on the socket (from the app-level) so long idle periods don't result in someone (in the middle or not) trying to force a connection reset for lack of resources. – Thelmathem

"Comcast" you say? :D Check out this related repo: github.com/tylertreat/comcast – Forsooth

Answer 1

A 'router' could be doing anything - particularly NAT, which might involve any amount of bug-ridden messing with traffic...

One reason a device will send a RST is in response to receiving a packet for a closed socket.

It's hard to give a firm but general answer, because every possible perversion has been visited on TCP since its inception, and all sorts of people might be inserting RSTs in an attempt to block traffic. (Some 'national firewalls' work like this, for example.)

Christeenchristel

Either the router has a 10 minute timeout for TCP connections or the router has "gateway smart packet detection" enabled. – Imperforate

It's a bit rich to suggest that a router might be bug-ridden. – Florescence

Answer 2

Run a packet sniffer (e.g., Wireshark) also on the peer to see whether it's the peer who's sending the RST or someone in the middle.

System

Answer 3

One thing to be aware of is that many Linux netfilter firewalls are misconfigured.

If you have something like:

-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT 

-A FORWARD -p tcp -j REJECT --reject-with tcp-reset 

then packet reordering can result in the firewall considering the packets invalid and thus generating resets which will then break otherwise healthy connections.

Reordering is particularly likely with a wireless network.

This should instead be:

-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT 

-A FORWARD -m state --state INVALID -j DROP 

-A FORWARD -p tcp -j REJECT --reject-with tcp-reset 

Basically anytime you have:

... -m state --state RELATED,ESTABLISHED -j ACCEPT 

it should immediately be followed by:

... -m state --state INVALID -j DROP

It's better to drop a packet then to generate a potentially protocol disrupting tcp reset. Resets are better when they're provably the correct thing to send... since this eliminates timeouts. But if there's any chance they're invalid then they can cause this sort of pain.

Materi

Oh my god man, thank you so much for this! You fixed my firewall! I've been looking for a solution for days. In my case I was using NetworkManager with "ipv4.method = shared" and had to apply this fix to my upstream interface which had the restrictive iptables rules on it. How or where exactly did you learn of this? – Football

Answer 4

I've just spent quite some time troubleshooting this very problem. None of the proposed solutions worked. Turned out that our sysadmin by mistake assigned the same static IP to two unrelated servers belonging to different groups, but sitting on the same network. The end results were intermittently dropped vnc connections, browser that had to be refreshed several times to fetch the web page, and other strange things.

Vitriform

How did you troubleshoot that? – Loader

Answer 5

Here are some cases where a TCP reset could be sent.

• Non-Existence TCP endpoint: The client sends SYN to a non-existing TCP port or IP on the server side. The server will send a reset to the client.
• SYN matches the existing TCP endpoint: The client sends SYN to an existing TCP endpoint, which means the same 5-tuple. The server will send a reset to the client.
• Accept Queue Full: When the accept queue is full on the server side, tcp_abort_on_overflow is set. The server will send a reset to the client.
• Half-Open Connections: When the server restarts itself. Then all connections before would receive a reset from the server side.
• Firewall: The firewall could send a reset to the client or server
• Time-Wait Assassination: When the client in the time-wait state, receives a message from the server-side, the client will send a reset to the server.
• Aborting Connection: When the client or server aborts the connection, it could send a reset to the server or client
• Connection Timeout: If a TCP connection is idle for an extended period, one of the parties might send a reset packet to close the connection.

Squabble

Answer 6

RST is sent by the side doing the active close because it is the side which sends the last ACK. So if it receives FIN from the side doing the passive close in a wrong state, it sends a RST packet which indicates other side that an error has occured.

Tellurate

Both sides send and receive a FIN in a normal closure. There is nothing wrong with this situation, and therefore no reason for one side to issue a reset. The first sentence doesn't even make sense. – Florescence

[RST, ACK] can also be sent by the side receiving a SYN on a port not being listened to. In a case I ran across, the RST/ACK came about 60 seconds after the first SYN. FWIW – Consanguineous

@MarquisofLorne, the first sentence itself may be treated as incorrect. But the phrase "in a wrong state" in second sentence makes it somehow valid. – Sterol

Answer 7

If there is a router doing NAT, especially a low end router with few resources, it will age the oldest TCP sessions first. To do this it sets the RST flag in the packet that effectively tells the receiving station to (very ungracefully) close the connection. this is done to save resources.

Suppurate

Answer 8

Some firewalls do that if a connection is idle for x number of minutes. Some ISPs set their routers to do that for various reasons as well.

In this day and age, you'll need to gracefully handle (re-establish as needed) that condition.

Camisole

The connection is re-established just fine, the problem is that the brief period of disconnect causes an alert unnecessarily. – Extant

I've had problems specifically with Cisco PIX/ASA equipment. They have especially short timeouts as defaults. The cheaper equipment is usually "better" in this regard (as in they don't timeout real fast)... – Camisole

I would even add that TCP was never actually completely reliable from persistent connections point of view. It just becomes more noticeable from time to time. Another interesting example: some people may implement logic that marks a TCP client as offline as soon as connection closure or reset is being detected. And then sometimes they don't bother to give a client a chance to reconnect. This is obviously not completely correct. – Sterol

Answer 9

This is because there is another process in the network sending RST to your TCP connection.

Normally RST would be sent in the following case

• A process close the socket when socket using SO_LINGER option is enabled
• OS is doing the resource cleanup when your process exit without closing socket.

In your case, it sounds like a process is connecting your connection(IP + port) and keeps sending RST after establish the connection.

Inmesh

Answer 10

In most applications, the socket connection has a timeout. If there is no communication between the client and the server within the timeout, the connection is reset as you observe. A great example is a FTP server, if you connect to the server and just leave the connection without browsing or downloading files, the server will kick you off the connection, usually to allow other to be able to connect. I guess this is what you are experiencing with your connection. So take a look in the server application, if that is where you get the reset from, and see if it indeed has a timeout set for the connection in the source code.

Urticaceous

Answer 11

Just had a case. Client1 connected to Server. They are sending data via websocket protocol and the TCP connection is kept alived. Server is python flask and listening on Port 5000. Then Client2(same IP address as Client1) send a HTTP request to Server. Now if you interrupt Client1 to make it quit. Then a "connection reset by peer 104" happens in Server side and Client2.

Simply put, the previous connection is not safely closed and a request is sent immediately for a 3 way handshake. The Server side got confused and sent a RST message.

Persuasion

What you described doesn't seem like the normal behavior for a server. Is there a reason why the second connection was RST to client 2, simply because client one quit? (And by quit, I assume you mean the client program shut down the connection in an orderly fashion). – Philippopolis

Answer 12

In general devices (the other end) have a TCP Keep Alive time setting which I think is set to 10 minutes in your case. The application needs to poll the device before that time, if not the other end send a TCP RST flag to close the connection. You may have to extend that time or modify your application to send requests more frequently than 10 minutes.

Curvaceous