Nginx cannot find unix socket file with Unicorn (no such file or directory)

Question

I am deploying a Rails 4 app to a Fedora 19 x64 server using Nginx and Unicorn. The problem is that I get an error when visiting the address: "We're sorry, but something went wrong."

My Nginx error log (/var/log/nginx/error.log) shows:

2014/03/08 03:50:12 [warn] 23934#0: conflicting server name "localhost" on 0.0.0.0:80, ignored
2014/03/08 03:50:12 [warn] 23936#0: conflicting server name "localhost" on 0.0.0.0:80, ignored
2014/03/08 03:50:14 [crit] 23939#0: *1 connect() to unix:/tmp/unicorn.[app name].sock failed (2: No such file or directory) while connecting to upstream, client: [client IP], server: localhost, request: "GET /v1/industries/1.xml HTTP/1.1", upstream: "http://unix:/tmp/unicorn.[app name].sock:/v1/industries.json", host: "api.[app name].ca"

As far as I can see from this, Nginx is not aware that the socket exists. However, looking in /tmp, it does:

[root@localhost tmp]# ls
unicorn.[app name].sock

I keep getting stuck at this point, no matter how I modify my Unicorn config file or my Nginx config file. Both are attacted:

/var/www/[app name]/config/unicorn.rb:

working_directory "/var/www/[app name]"
pid "/var/www/[app name]/pids/unicorn.pid"
stderr_path "/var/www/[app name]/log/unicorn.log"
stdout_path "/var/www/[app name]/log/unicorn.log"
listen "/tmp/unicorn.[app name].sock"
worker_processes 2
timeout 30

/etc/nginx/conf.d/default.conf:

upstream app {
    server unix:/tmp/unicorn.[app name].sock fail_timeout=0;
}
server {
    listen 80;
    server_name localhost;
    root /var/www/[app name]/public;
    try_files $uri/index.html $uri @app;
    location @app {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://app;
    }
    error_page 500 502 503 504 /500.html;
    client_max_body_size 4G;
    keepalive_timeout 10;
}

The way I have been starting these two daemons is as follows:

unicorn_rails -c /var/www/[app name]/config/unicorn.rb -D -E production
service nginx start

The Unicorn logs contain no relevant information, nor do the production logs. This setup seems straight forward, has anyone experienced this before? Thanks for any help you can provide.

By the way, I was initially following this tutorial: https://www.digitalocean.com/community/articles/how-to-deploy-rails-apps-using-unicorn-and-nginx-on-centos-6-5

Naos

Answer 1

After many hours and a grand total of 3 beers, I've managed to figure out the problem. After hours of digging, I finally came across this Server Fault answer

In layman terms, it appears that programs that create files in /tmp (or /var/tmp as I have discovered) are the only programs that are able to see the files in that directory. Unicorn was creating the UNIX socket file, however Nginx could not see it.

The solution I have employed is to have Unicorn create sockets in /var/sockets.

Naos

How did you make unicorn change the dir it users to store sockets? – Rigging

Simply modify the listen variable in your Unicorn config file. Ex: listen "/var/sockets/unicorn.[app name].sock", then configure Nginx to proxy all connections to your server to that socket file such as server unix:/var/sockets/unicorn.[app name].sock fail_timeout=0; – Naos

Just adding a googleable note that I came here by way of nginx reporting a 504 Gateway Timeout error while using an upstream puma server. The only reason I stumbled upon this answer was that the logs indicated the same strange looking http://unix:///tmp… path being explored. So yeah, certainly not a unicorn specific bug in case anybody else using puma, or any other server, finds this. Ubuntu 14.04 is apparently exhibiting the same security protocol as fedora. – Millen

@Millen I am having the same issue can you help ? – Synagogue

@Millen I'm having the same issue where it's looking for the unix socket with an http:// prefix. The socket file definitely exists, is definitely accessible to the nginx user (verified with sudo -u), and is in a user-custom location that isn't /tmp. What's even weirder is that this works on another server with pretty much exactly the same configuration. Did you ever figure out what causes it to interpret the unix socket as http? – Footrace

Answer 2

I suddenly had a similar situation after changing nginx to use a systemd startup service based on their template.

It ends up that the problem was with PrivateTmp=true, which makes it so that nginx was unable to access the socket file created by gunicorn. Once I changed this to PrivateTmp=false the error resolved.

Amritsar