(13: Permission denied) while connecting to upstream:[nginx]

Question

I am working with configuring Django project with Nginx and Gunicorn.

While I am accessing my port gunicorn mysite.wsgi:application --bind=127.0.0.1:8001 in Nginx server, I am getting the following error in my error log file;

2014/05/30 11:59:42 [crit] 4075#0: *6 connect() to 127.0.0.1:8001 failed (13: Permission denied) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8001/", host: "localhost:8080"

Below is the content of my nginx.conf file;

server {
    listen 8080;
    server_name localhost;
    access_log  /var/log/nginx/example.log;
    error_log /var/log/nginx/example.error.log;

    location / {
        proxy_pass http://127.0.0.1:8001;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $http_host;
    }
}

In the HTML page I am getting 502 Bad Gateway.

What mistake am I doing?

Giverin

Answer 1

Disclaimer

Make sure there are no security implications for your use-case before running this.

Answer

I had a similar issue getting Fedora 20, Nginx, Node.js, and Ghost (blog) to work. It turns out my issue was due to SELinux.

This should solve the problem:

setsebool -P httpd_can_network_connect 1

Details

I checked for errors in the SELinux logs:

sudo cat /var/log/audit/audit.log | grep nginx | grep denied

And found that running the following commands fixed my issue:

sudo cat /var/log/audit/audit.log | grep nginx | grep denied | audit2allow -M mynginx
sudo semodule -i mynginx.pp

Option #2 (probably more secure)

setsebool -P httpd_can_network_relay 1

https://security.stackexchange.com/questions/152358/difference-between-selinux-booleans-httpd-can-network-relay-and-httpd-can-net

References

http://blog.frag-gustav.de/2013/07/21/nginx-selinux-me-mad/
https://wiki.gentoo.org/wiki/SELinux/Tutorials/Where_to_find_SELinux_permission_denial_details
http://wiki.gentoo.org/wiki/SELinux/Tutorials/Managing_network_port_labels

Plasmagel

thanks. I needed to yum install policycoreutils-python in order to get audit2allow first. Reference: centos.org/forums/viewtopic.php?t=5012 – Are

See also here. In my case I had to add nginx to the group of the user in whose home directory the wwwroot was stored. – Marenmarena

Sid's answer should be the accepted answer, it's better to use built-in policies rather than create your own, much less overhead, especially when dealing with multiple servers. – April

On Fedora 23 installing the policycoreutils-python did not provide the command audit2allow. After some research I found you should install the devel packageyum install policycoreutils-devel. Reference: danwalsh.livejournal.com/61710.html – Torpor

Can someone please tell me how someone would reverse these policy changes? – Maurey

amazing, and also very criptic (not your answer, but that this is required). I installed two centos instances and one required it the other didn't, confusing. Also are there any security implications to this to be aware of? – Arst

I was on RHEL 7.2 and the "setsebool -P httpd_can_network_connect 1" saved me. Big thank you! – Speechmaking

Just never could have imagined that when the default welcome page is displayed but proxying an upstream server doesn't work, it is still the fault of a third-party security policy. I just focused my debugging effort on the config file... Though should have been able to find the error log earlier indeed. – Monastery

This SHOULD be in the cherrpy and nginx documentation for many unix OSes, because I wasted 8 hours trying to figure it out after following all the docs! – Constantina

Please note that setting httpd_can_network_connect may pose as a serious security risk on your public facing server if turned on. See this and this – Muscle

Thanks a lot, I spent almost whole day troubleshooting, you saved me. Why dont Nginx forum add this in their documentation. – Heretic

The _relay made my day, thank you so much :) And this doc explains good: security.stackexchange.com/questions/152358/… – Irritate

Thank you for pointing out the log file that should be checked! – Guarani

Answer 2

I’ve run into this problem too. Another solution is to toggle the SELinux boolean value for httpd network connect to on (Nginx uses the httpd label).

setsebool httpd_can_network_connect on

To make the change persist use the -P flag.

setsebool httpd_can_network_connect on -P

You can see a list of all available SELinux booleans for httpd using

getsebool -a | grep httpd

Jeep

This worked, thanks. I updated from CentOS 6.5 -> 6.7 and it must have defaulted the value to off during the update, because it was working fine before the update. Simple fix. – April

Please note that setting httpd_can_network_connect may pose as a serious security risk on your public facing server if turned on. See this and this – Muscle

Answer 3

I have solved my problem by running my Nginx as the user I'm currently logged in with, mulagala.

By default the user as nginx is defined at the very top section of the nginx.conf file as seen below;

user nginx; # Default Nginx user

Change nginx to the name of your current user - here, mulagala.

user mulagala; # Custom Nginx user (as username of the current logged in user)

However, this may not address the actual problem and may actually have casual side effect(s).

For an effective solution, please refer to Joseph Barbere's solution.

Giverin

Answer 4

Had a similar problem on Centos 7. When I tried to apply the solution prescribed by Sorin, I started moving in cycles. First I had a permission {write} denied. Then when I solved that I had a permission { connectto } denied. Then back again to permission {write } denied.

Following @Sid answer above of checking the flags using getsebool -a | grep httpd and toggling them I found that in addition to the httpd_can_network_connect being off. http_anon_write was also off resulting in permission denied write and permission denied {connectto}

type=AVC msg=audit(1501830505.174:799183): avc:  
denied  { write } for  pid=12144 comm="nginx" name="myroject.sock" 
dev="dm-2" ino=134718735 scontext=system_u:system_r:httpd_t:s0 
tcontext=system_u:object_r:default_t:s0 tclass=sock_file

Obtained using sudo cat /var/log/audit/audit.log | grep nginx | grep denied as explained above.

So I solved them one at a time, toggling the flags on one at a time.

setsebool httpd_can_network_connect on -P

Then running the commands specified by @sorin and @Joseph above

sudo cat /var/log/audit/audit.log | grep nginx | grep denied | 
audit2allow -M mynginx
sudo semodule -i mynginx.pp

Basically you can check the permissions set on setsebool and correlate that with the error obtained from grepp'ing' audit.log nginx, denied

Vimen

Answer 5

if "502 Bad Gateway" error throws on centos api url for api gateway proxy pass on nginx , run following command to solve the issue

sudo setsebool -P httpd_can_network_connect 1

Beryllium

Answer 6

1. Check the user in /etc/nginx/nginx.conf
2. Change ownership to user.

sudo chown -R nginx:nginx /var/lib/nginx

Holms

Answer 7

1. For first see what is denied:

sudo cat /var/log/audit/audit.log | grep nginx | grep denied

type=AVC msg=audit(1618940614.934:38415): avc:  denied  { connectto } for
pid=18016 comm="nginx" path="/home/deployer/project/tmp/sockets/puma.sock" scontext=system_u:system_r:httpd_t:s0
tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
tclass=unix_stream_socket permissive=1

2. In my case it helps on CentOS7:

sudo setenforce 0

setsebool httpd_can_network_connect on -P
setsebool httpd_can_network_relay on -P

After you can see what is enable:

getsebool -a | grep httpd

httpd_anon_write --> off
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_connect_ftp --> off
httpd_can_connect_ldap --> off
httpd_can_connect_mythtv --> off
httpd_can_connect_zabbix --> off
httpd_can_network_connect --> on
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> on
httpd_can_network_memcache --> off
httpd_can_network_relay --> on
httpd_can_sendmail --> off
httpd_dbus_avahi --> off
httpd_dbus_sssd --> off
httpd_dontaudit_search_dirs --> off
httpd_enable_cgi --> off
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_graceful_shutdown --> on
httpd_manage_ipa --> off
httpd_mod_auth_ntlm_winbind --> off
httpd_mod_auth_pam --> off
httpd_read_user_content --> off
httpd_run_ipa --> off
httpd_run_preupgrade --> off
httpd_run_stickshift --> off
httpd_serve_cobbler_files --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_sys_script_anon_write --> off
httpd_tmp_exec --> off
httpd_tty_comm --> off
httpd_unified --> off
httpd_use_cifs --> off
httpd_use_fusefs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off
httpd_use_openstack --> off
httpd_use_sasl --> off
httpd_verify_dns --> off

Dayan

Thank you! setenforce 0 resolve my issue – Pashto

@Pashto Ok, good! – Rivy

Please don't do this in a secure environment, turning off security policies is a band-aid fix and makes the system insecure. – Kev

Answer 8

I had the same issue. I tried @joebarbere solution of running the below command which fixed the issue but I was concerned about security

setsebool -P httpd_can_network_relay 1

I then came across the following solution

https://serverfault.com/questions/634294/nodejs-nginx-error-13-permission-denied-while-connecting-to-upstream

I was using non standard port and SELinux was blocking the port. Use the following command to check if the port is allowed

sudo semanage port --list | grep 4343

And added the port to allowed list

sudo semanage port --add --type http_port_t --proto tcp 4343

I restarted the nginx service, then the url was accessible. I would assume this is more secure.

Pyrochemical

thank you! what operating system you're on? I'm using CentOS Stream release 9 and this solved my issue – Xanthous

I was using CentOS 7 – Pyrochemical

Answer 9

13-permission-denied-while-connecting-to-upstreamnginx on centos server -

setsebool -P httpd_can_network_connect 1

Snaggy

Answer 10

I’ve run into this problem too. I'm using Nginx with HHVM, below solution fixed my issue:

sudo semanage fcontext -a -t httpd_sys_rw_content_t "/etc/nginx/fastcgi_temp(/.*)?"

sudo restorecon -R -v /etc/nginx/fastcgi_temp

Moser

Answer 11

Another reason could be; you are accessing your application through nginx using proxy but you did not add gunicorn.sock file for proxy with gunicorn.

You need to add a proxy file path in nginx configuration.

location / {
        include proxy_params;
        proxy_pass http://unix:/home/username/myproject/gunicorn.sock;
    }

Here is a nice tutorial with step by step implementation of this

https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-16-04#configure-nginx-to-proxy-pass-to-gunicorn

Note: if you did not created anyname.sock file you have to create if first, either use above or any other method or tutorial to create it.

Anastomose

I do not think you need to use a gunicorn.sock file. That is optional. You should be able to setup Nginx, Gunicorn, and Django without a gunicorn.sock if desired. – Typify

Answer 12

The SELinux Policy management tool has a set of standard ports that are allowed.
Usually 8001, 5000, 4000, etc are not one of them.

Step 1

check if the port you are trying to use is allowed (or assigned to something else)

sudo semanage port --list | grep http_port_t 

Output

http_port_t  tcp  80, 81, 443, 488, 8008, 8009, 8443, 9000

You can also confirm its use (assigned to something else)

sudo semanage port --list | grep 8001

Step 2

If it is not available then you can either add it or use one of the above options

Option 1

To add the port.

sudo semanage port --add --type http_port_t --proto tcp 8001

Option 2 (I recommend)

To use one of the allowed ports (instead of adding a new one)

server {
    ....

    location / {
        proxy_pass http://127.0.0.1:9000;
        ...
    }
}

In our case, we have changed 8001 to 9000.

Note: you should also change the port on your project to 9000
For this case is gunicorn mysite.wsgi:application --bind=127.0.0.1:9000

Security Implication

I consider this more secure than the previous answers

Cavanagh

I am running on perfect fine http_port_t 9000, however, i still have the same error – Passer

Hey @Wang, this solution assumes all other setups on your server are good such as Nginx and project-specific settings and the only thing that is denying your requests is SELinux Policy management for denying ports. You can also visit Using NGINX and NGINX Plus with SELinux for more details. – Cavanagh

Answer 13

In my case, it was fixed (from this blog) by running the command: semanage permissive -a httpd_t and restarting NGINX service.

Discover

this is an insecure fix in the long run. Better approach is to check logs, and correct the labels that are causing SELinux to behave in that way. – Deshabille

Answer 14

I encountered a similar issue while deploying my Django application locally with Nginx as a reverse proxy. The error message (13: Permission denied) while connecting to upstream typically indicates a permission problem with accessing the upstream server, in this case, your Django app.

The solution that worked for me was related to the nginx.conf file. Inside this file, the user directive plays a significant role, as it defines the user and group that Nginx runs as. Sometimes, a comment preceding the user directive can cause permission issues.

To resolve the error, follow these steps:

1. Locate your nginx.conf file (usually in /etc/nginx/).
2. Find the user directive. It should look like: user <user>;.
3. Make sure there are no comments (lines starting with #) before the user directive. Comments can sometimes disrupt the configuration parsing.
4. Save the nginx.conf file.
5. Restart Nginx with: sudo systemctl restart nginx.

After applying these steps, the (13: Permission denied) error should no longer occur, and your Django application should function properly when accessed through the Nginx proxy.

Lurlene

Answer 15

I had the same error. These commands work for me

sudo chmod +x /home/(username)/
sudo chmod +x /home/(username)/(your_project_folder)/

Iconolatry

Answer 16

Details: You can grep in auditd.log and identify the exact reason.

In my case the problem was that nis_enabled option was set to off.

You can check the current config with:

semanage boolean --list | grep nis

And you can enable it with:

setsebool -P nis_enabled 1

Then you can try start nginx service.

Outspread

Answer 17

Do not install SELinux on Ubuntun 22.10. I did that, and I could no longer boot. Ubuntun 22.10 gave me the GRUB menu, I don't know how to recover: I ended up reinstalled Ubuntun 22.10.

Galarza

This does not really answer the question. If you have a different question, you can ask it by clicking Ask Question. To get notified when this question gets new answers, you can follow this question. Once you have enough reputation, you can also add a bounty to draw more attention to this question. - From Review – Pasteurization