How to debug error in kube-proxy: Connection reset by peer

Question

I got errors in my kube-proxy:

E0107 21:48:57.738867       1 proxysocket.go:160] I/O error: read tcp 10.2.11.253:37568: connection reset by peer

How can I trace quickly which pod has IP 10.2.11.253? And how can I know which request that was, from which pod to which pod?

Or can we change the kube-proxy log level to verbose or debug?

I got another errors, same connection reset error, but the IP is a node's IP

E0107 21:52:53.483363       1 proxysocket.go:160] I/O error: read tcp 192.168.166.180:11732: connection reset by peer

192.168.166.x is my kubernetes node subnet, but how can kube-proxy forwards request to a node IP?

I'm using kubelet 1.0.1 and CoreOS v773.1.0 (docker 1.7.1, kernel 4.1.5) as my cluster nodes.

Thanks for any help!

Rugen

Answer 1

In my circumstance, it was because kube-proxy (v1.1.4) was missing the --proxy-mode=iptables flag. Evidently in 1.1.4, the default is something other than iptables, and specifying that flag made the logs immediately stop spewing those messages.

Scheer

I can confirm that adding --proxy-mode=iptables to our proxy configs stopped the "connection reset by peer" messages caused by our haproxy health checks. Hopefully the iptables mode does not remain "experimental" for long. – Quintuplet

Just upgraded to v1.1.7, hopefully that won't happen again – Rugen

it is happening with v1.9.6 also. – Choiseul