Nginx error: (13: Permission denied) while connecting to upstream

Question

I am getting this error in my nginx-error.log file:

2014/02/17 03:42:20 [crit] 5455#0: *1 connect() to unix:/tmp/uwsgi.sock failed (13: Permission denied) while connecting to upstream, client: xx.xx.x.xxx, server: localhost, request: "GET /users HTTP/1.1", upstream: "uwsgi://unix:/tmp/uwsgi.sock:", host: "EC2.amazonaws.com"

The browser also shows a 502 Bad Gateway Error. The output of a curl is the same, Bad Gateway html

I've tried to fix it by changing permissions for /tmp/uwsgi.sock to 777. That didn't work. I also added myself to the www-data group (a couple questions that looked similar suggested that). Also, no dice.

Here is my nginx.conf file:

nginx.conf

worker_processes 1;
worker_rlimit_nofile 8192;

events {
  worker_connections  3000; 
}

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on; 
    #tcp_nopush     on; 

    keepalive_timeout  65; 

    #gzip  on; 

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

I am running a Flask application with Nginsx and Uwsgi, just to be thorough in my explanation. If anyone has any ideas, I would really appreciate them.

---

EDIT

I have been asked to provide my uwsgi config file. So, I never personally wrote my nginx or my uwsgi file. I followed the guide here which sets everything up using ansible-playbook. The nginx.conf file was generated automatically, but there was nothing in /etc/uwsgi except a README file in both apps-enabled and apps-available folders. Do I need to create my own config file for uwsgi? I was under the impression that ansible took care of all of those things.

I believe that ansible-playbook figured out my uwsgi configuration since when I run this command

uwsgi -s /tmp/uwsgi.sock -w my_app:app

it starts up and outputs this:

*** Starting uWSGI 2.0.1 (64bit) on [Mon Feb 17 20:03:08 2014] ***
compiled with version: 4.7.3 on 10 February 2014 18:26:16
os: Linux-3.11.0-15-generic #25-Ubuntu SMP Thu Jan 30 17:22:01 UTC 2014
nodename: ip-10-9-xxx-xxx
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/username/Project
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 4548
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/uwsgi.sock fd 3
Python version: 2.7.5+ (default, Sep 19 2013, 13:52:09)  [GCC 4.8.1]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x1f60260
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72760 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x1f60260 pid: 26790 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 26790, cores: 1)

Archoplasm

Show the uwsgi configuration and the proxy in nginx. – Coarsen

Answer 1

The permission issue occurs because uwsgi resets the ownership and permissions of /tmp/uwsgi.sock to 755 and the user running uwsgi every time uwsgi starts.

The correct way to solve the problem is to make uwsgi change the ownership and/or permission of /tmp/uwsgi.sock such that nginx can write to this socket. Therefore, there are three possible solutions.

1. Run uwsgi as the www-data user so that this user owns the socket file created by it.

   uwsgi -s /tmp/uwsgi.sock -w my_app:app --uid www-data --gid www-data
   

2. Change the ownership of the socket file so that www-data owns it.

   uwsgi -s /tmp/uwsgi.sock -w my_app:app --chown-socket=www-data:www-data
   

3. Change the permissions of the socket file, so that www-data can write to it.

   uwsgi -s /tmp/uwsgi.sock -w my_app:app --chmod-socket=666
   

I prefer the first approach because it does not leave uwsgi running as root.

The first two commands need to be run as root user. The third command does not need to be run as root user.

The first command leaves uwsgi running as www-data user. The second and third commands leave uwsgi running as the actual user that ran the command.

The first and second command allow only www-data user to write to the socket. The third command allows any user to write to the socket.

I prefer the first approach because it does not leave uwsgi running as root user and it does not make the socket file world-writeable .

Gelatin

I am facing same problem but i am using gunicorn instead of uwsgi.How to solve this..? – Toothache

@Toothache This question and answer is not about gunicorn. If you have a question about gunicorn, you should post a separate question. – Gelatin

I had this problem on centOS, and your commands did not work for me. I used: chown -R www-data:www-data /tmp – Aerobiosis

So this basically means you have to run uwsgi as root, correct? Otherwise your uwsgi can't do any chown'ing. What if you want to run uwsgi under a non-root account? (Which I do want because of other permissions issues that arise when using sorl + uwsgi under root) – Sargasso

Another approach would be to use internet socket instead of unix socket, although unix socket will be more performant for local communication - uwsgi-docs.readthedocs.org/en/latest/WSGIquickstart.html – Dayna

@Aerobiosis you've got to check why uwsgi's were not able to give proper access control. In theory it should, unless your configuration is very special. – Blades

@Sargasso I have updated the answer to show the usage of --uid and --gid options which would run uwsgi as a non-root account. The command using these options still needs to be run as root user but this command launches uwsgi such that it runs as non-root user. – Gelatin

When I try the first command, I get a bind error that the address is already in use. When I remove the --uid and --gid options it seems to work fine. Any idea what would be causing this? – Gheber

@LukeMat You probably need to rm /tmp/uwsgi.sock before running uwsgi with the --uid and --gid options. The issue occurred probably because you first ran uwsgi as root without the --uid and --gid options. That caused /tmp/uwsgi.sock file to be created with root as the owner. Later when you do run uwsgi with the --uid www-data and --gid www-data options, www-data is unable to write to the socket file because the owner of this file is still root. Therefore, removing the old socket file before using --uid and --gid options would resolve the issue. – Gelatin

Yup that turned out to be it! I figured this out a bit after commenting and forgot to come back and give an update. Thank you though! – Gheber

With Ubuntu 18.04 I just restart the server and voilà: The problem is gone! (this after trying a lot of things...) – Digiovanni

Be sure to restart nginx service and UWSGI if you try it – Justice

Answer 2

While the accepted solution is true there might also SELinux be blocking the access. If you did set the permissions correctly and still get permission denied messages try:

sudo setenforce Permissive

If it works then SELinux was at fault - or rather was working as expected! To add the permissions needed to nginx do:

  # to see what permissions are needed.
sudo grep nginx /var/log/audit/audit.log | audit2allow
  # to create a nginx.pp policy file
sudo grep nginx /var/log/audit/audit.log | audit2allow -M nginx
  # to apply the new policy
sudo semodule -i nginx.pp

After that reset the SELinux Policy to Enforcing with:

sudo setenforce Enforcing

Coal

Thanks @enaut! selinux was indeed my problem. I've been getting the same error message in the log and once I checked selinux it was in Enforcing mode. I've been tearing my hair out and double checking my file permissions for the past hour when indeed the problem was selinux policy. Much obliged. – Liberty

Great. but you don't need to change enforce mode. For more information, visit nginx blog: nginx.com/blog/using-nginx-plus-with-selinux – Unship

This is a great solution. After spending hours and rechecking everything, I found this. Just superb!! – Aeschines

Answer 3

Anyone who lands here from the Googles and is trying to run Flask on AWS using the default Ubuntu image after installing nginx and still can't figure out what the problem is:

Nginx runs as user "www-data" by default, but the most common Flask WSGI tutorial from Digital Ocean has you use the logged in user for the systemd service file. Change the user that nginx is running as from "www-data" (which is the default) to "ubuntu" in /etc/nginx/nginx.conf if your Flask/wsgi user is "ubuntu" and everything will start working. You can do this with one line in a script:

sudo sed -i 's/user www-data;/user ubuntu;/' /etc/nginx/nginx.conf

Trying to make Flask and uwsgi run as www-data did not work off the bat, but making nginx run as ubuntu worked just fine since all I'm running with this instance is Flask anyhow.

Gearalt

this saved my life – Scilicet

Thank you! Worked for me on django+gunicorn+nignx setup as well! – Coccidioidomycosis

thanks a lot, I spent hours until found this – Esse

Answer 4

Nginx connect to .sock failed (13:Permission denied) - 502 bad gateway

change the name of the user on the first line in /etc/nginx/nginx.conf file.

the default user is www-data and change it to root or your username

Onlybegotten

It worked on my AWS EC2 Ubuntu 22 – Pooka

Answer 5

You have to set these permissions (chmod/chown) in uWSGI configuration.

It is the chmod-socket and the chown-socket.

http://uwsgi-docs.readthedocs.org/en/latest/Options.html#chmod-socket http://uwsgi-docs.readthedocs.org/en/latest/Options.html#chown-socket

Coarsen

I tried changing the permissions (manually) on the socket to 666, but it still gave me the same error. – Archoplasm

@Coarsen can you please write the syntax ? – Paterfamilias

@MitulShah the accepted answer has the correct syntax already. – Coarsen

Answer 6

I know it's too late, but it might helps to other. I'll suggest to follow Running flask with virtualenv, uwsgi, and nginx very simple and sweet documentation.

Must activate your environment if you run your project in virtualenv.

here is the yolo.py

from config import application

if __name__ == "__main__":
    application.run(host='127.0.0.1')

And create uwsgi.sock file in /tmp/ directory and leave it blank. As @susanpal answer said "The permission issue occurs because uwsgi resets the ownership and permissions of /tmp/uwsgi.sock to 755 and the user running uwsgi every time uwsgi starts." it is correct.

So you have to give permission to sock file whenever uwsgi starts. so now follow the below command

uwsgi -s /tmp/uwsgi.sock -w yolo:application -H /var/www/yolo/env --chmod-socket=666 

A little different command from @susanpal. And for persist connection, simply add "&" end of command

uwsgi -s /tmp/uwsgi.sock -w yolo:app -H /var/www/yolo/env --chmod-socket=666 &

Paterfamilias

Answer 7

In my case changing some php permission do the trick

sudo chown user:group -R /run/php

I hope this helps someone.

Petronel

In some cases you should use sudo chown user:group -R /run/php-fpm – C

Answer 8

You should post both nginx and uwsgi configuration file for your application (the ones in /etc/nginx/sites-enabled/ and /etc/uwsgi/ - or wherever you put them).

Typically check that you have a line similar to the following one in your nginx app configuration:

uwsgi_pass unix:///tmp/uwsgi.sock;

and the same socket name in your uwsgi config file:

socket=/tmp/uwsgi.sock

Tracitracie

I don't have that uwsgi config file, see edit to question. Do I need one even though I'm able to run the uwsgi command and it works? – Archoplasm

@AlexChumbley There is no need to create a configuration file if you don't want one. Configuration file makes the setup a bit neater in my opinion because most of the command line options move to the configuration file which means you have to use less options in the command to work with. – Gelatin