go: http server does not work on osx

Question

The example http server doesn't work for me. Source code:

package main

import (
    "fmt"
    "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path[1:])
}

func main() {
    http.HandleFunc("/", handler)
    http.ListenAndServe("127.0.0.1:8080", nil)
}

When I try to connect all I get is:

$ curl 127.0.0.1:8080
curl: (7) Failed to connect to 127.0.0.1 port 8080: Operation timed out
$ nc -v -G 5 127.0.0.1 8080
nc: connectx to 127.0.0.1 port 8080 (tcp) failed: Operation timed out

While nc tries to connect tcpdump shows only SYN packets:

$ tcpdump -i lo0 port 8080
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo0, link-type NULL (BSD loopback), capture size 262144 bytes
18:21:30.906638 IP localhost.52799 > localhost.http-alt: Flags [S], seq 3375761924, win 65535, options [mss 16344,nop,wscale 5,nop,nop,TS val 118569352 ecr 0,sackOK,eol], length 0
18:21:31.006824 IP localhost.52799 > localhost.http-alt: Flags [S], seq 3375761924, win 65535, options [mss 16344,nop,wscale 5,nop,nop,TS val 118569452 ecr 0,sackOK,eol], length 0
18:21:31.106989 IP localhost.52799 > localhost.http-alt: Flags [S], seq 3375761924, win 65535, options [mss 16344,nop,wscale 5,nop,nop,TS val 118569552 ecr 0,sackOK,eol], length 0
18:21:31.208141 IP localhost.52799 > localhost.http-alt: Flags [S], seq 3375761924, win 65535, options [mss 16344,nop,wscale 5,nop,nop,TS val 118569653 ecr 0,sackOK,eol], length 0
18:21:31.308288 IP localhost.52799 > localhost.http-alt: Flags [S], seq 3375761924, win 65535, options [mss 16344,nop,wscale 5,nop,nop,TS val 118569753 ecr 0,sackOK,eol], length 0
18:21:31.408336 IP localhost.52799 > localhost.http-alt: Flags [S], seq 3375761924, win 65535, options [mss 16344,nop,wscale 5,nop,nop,TS val 118569853 ecr 0,sackOK,eol], length 0
18:21:31.609143 IP localhost.52799 > localhost.http-alt: Flags [S], seq 3375761924, win 65535, options [mss 16344,nop,wscale 5,nop,nop,TS val 118570053 ecr 0,sackOK,eol], length 0
18:21:32.011215 IP localhost.52799 > localhost.http-alt: Flags [S], seq 3375761924, win 65535, options [mss 16344,nop,wscale 5,nop,nop,TS val 118570453 ecr 0,sackOK,eol], length 0
18:21:32.812512 IP localhost.52799 > localhost.http-alt: Flags [S], seq 3375761924, win 65535, options [mss 16344,nop,wscale 5,nop,nop,TS val 118571253 ecr 0,sackOK,eol], length 0
18:21:34.414686 IP localhost.52799 > localhost.http-alt: Flags [S], seq 3375761924, win 65535, options [mss 16344,nop,wscale 5,nop,nop,TS val 118572853 ecr 0,sackOK,eol], length 0

And here's the last part of dtruss output of the server-side (starting from socket call):

socket(0x2, 0x1, 0x0)            = 3 0
fcntl(0x3, 0x2, 0x1)             = 0 0
fcntl(0x3, 0x3, 0x0)             = 2 0
fcntl(0x3, 0x4, 0x6)             = 0 0
setsockopt(0x3, 0xFFFF, 0x20)            = 0 0
setsockopt(0x3, 0xFFFF, 0x4)             = 0 0
bind(0x3, 0xC8200BA6AC, 0x10)            = 0 0
listen(0x3, 0xFDE8, 0x10)                = 0 0
kqueue(0x3, 0xFDE8, 0x10)                = 4 0
fcntl(0x4, 0x2, 0x1)             = 0 0
kevent(0x4, 0xC8200558C0, 0x2)           = 0 0
getsockname(0x3, 0xC82005592C, 0xC820055928)             = 0 0
accept(0x3, 0xC820055AA4, 0xC820055A94)          = -1 Err#35
kevent(0x4, 0x0, 0x0)            = 0 0
select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)               = 0 0

Nothing more appears when I run nc, so it does not even notice connection attempts.

I don't have firewall enabled, everything else works, only Go programs have this issue (in particular I can't use docker-machine)

How do I fix it?

Edit: I assume that #Err35 is:

#define EDEADLK     35  /* Resource deadlock would occur */

...wat?

Edit2:

• go: version go1.6 darwin/amd64
• OSX: 10.11.4

The server is not exiting, it listens indefinitely.

Edit3:

I tried both "go build" and "go run - no difference

Using localhost instead of 127.0.0.1 - no difference

Using w.Write([]byte(fmt.Sprintf("Hi there, I love %s!", r.URL.Path[1:])) instead - no difference. Btw - it says nc: connectx to 127.0.0.1 port 8080 (tcp) failed: Operation timed out, so the connection attempt times out, not the read

ifconfig -a and netstat -r output: https://gist.github.com/mabn/ed171f180725b563d32bb86d5ec61988

Uranous

what happens when you listen on :8080 only with no ip address? – Thermotaxis

same thing, also tried on other interfaces – Uranous

did you check if an error is returned from ListenAndServe? – Thermotaxis

tcp4 0 0 127.0.0.1:8080 . LISTEN, tcp4 0 0 127.0.0.1:53123 127.0.0.1:8080 SYN_SENT – Uranous

this is under docker or native on your system? – Thermotaxis

Was the server in the listening loop or did it return right away? – Hagiographer

OS X version? Go version? I can't replicate on 10.11.4 w/ Go 1.6. – Arbitrate

what if you use different port? maybe some process is binding 8080? – Maybellemayberry

Changing the port does not help – Uranous

How do you run your Go server? With go run filename.go? Try go build and launch the executable that is created. Also try localhost:8080 in Go, and connect to localhost:8080. – Percheron

@Uranous I have OSX machine. If you share your compiled binary on github, googledisk or something I can at least check if it works on mine. – Explore

@Uranous could you try spawning a goroutine individually ? If you're new to go and find it confusing I can post a gist for you. Im thinking that it blocks indefinitely and isn't spawning a goroutine to handle the socket accept. – Levee

@Uranous could you try this gist and see if the count increases ? – Levee

@Uranous Curious as to what happens if you don't use Fprintf and just use w.Write([]byte(fmt.Sprintf("Hi there, I love %s!", r.URL.Path[1:])). It looks like a response is not written and so it times out. – Risibility

@Levee I'm very new to go. I've run your gist, it outputs 2 routines, 3 routines, 4 routines... etc. I'm not sure if you want me to do anything more to it. – Uranous

@Explore I've added the code and the binary here: github.com/mabn/golang_issue – Uranous

@Uranous I was trying to figure out if goroutines were getting spawned to handle a request. Since the gist works, there is no problem there. That's all I got , no idea how to help further. – Levee

@Uranous Your binary provided in your gist just works on my machine. So it's OSX issue or your specific environment. Check /etc/hosts and firewall policy on your machine for instance. – Explore

Answer 1

I fixed it! I remembered that a while earlier I tried to increase limit of connections on my machine and applied the changes described here: https://apple.stackexchange.com/a/169606

Reverting them fixed the issue. This is what helped:

sudo rm /etc/sysctl.conf
sudo sysctl kern.ipc.somaxconn=128
sudo rm /Library/LaunchDaemons/limit.maxproc.plist
sudo rm /Library/LaunchDaemons/limit.maxfiles.plist

Btw: after some more investigation I found out that:

• the problem was that accept() syscall was returning EAGAIN every time but not accepting any connections
• it behaved the same for non-blocking connections in C and Java programs

But I still don't know why exactly it was broken.

Uranous

Answer 2

I'd like to see the output of 'ifconfig -a'.

This smells like a local configuration issue; I can't reproduced this on OS X 10.11.4, Go 1.6

My W.A.G is that a tunnel device is actually using 127.0.0.1 not the loop back device.

Vitamin

added to the question – Uranous

is it possible that you're linking against a locally modified net/http package? – Vitamin

I don't think so. The root problem is that docker-machine is not able to connect to it's rpc server and I just grabbed binary release of docker-machine. I tried later building it myself and debugging and reduced the problem to the issue I described in this question. – Uranous

Answer 3

Do you have Cisco Anyconnect installed? If so find websecurity_uninstall.sh and uninstall websecurity

sudo /opt/cisco/anyconnect/bin/websecurity_uninstall.sh

It forwards port 8080 to port 5001 and you cannot use netstat, lsof or any of the normal tools to see why nothing can use port 8080.

Horsecar

Don't have it. I tried other ports too and it's the same – Uranous

Answer 4

I'm sure you checked this, but by any chance do you have packet filtering (pfctl) or some other firewall turned on? See https://superuser.com/questions/505128/deny-access-to-a-port-from-localhost-on-osx

Atrabilious

Don't have it. I uninstalled firewall and it didn't help. – Uranous