Jenkins with reverse proxy to alternative port number

Question

I have a jenkins builder server and I'm trying to set up the reverse proxy with nginx. I followed all the howto's and documentation from the jenkins site but the only thing different is I need the server to be reachable on a different port then the standerd https port.

The server has to be reachable at https://jenkins.example.com:9090 which is working now but I'm still having some issues with it. In Manage Jenkins I keep getting the message

It appears that your reverse proxy set up is broken

also when I login or apply or save some configuration change I keep getting redirected to https://jenkins.example.com with out the port number.

When I check with curl and look in the headers for a some pages it keeps setting the location header to the correct url but without the port number.

I have the following configuration in nginx

server {
  listen 443 ssl spdy;
  server_name jenkins.example.com;
  add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";
  add_header X-Frame-Options "DENY";

  ssl on;
  ssl_certificate /etc/nginx/ssl/server.chain.crt;
  ssl_certificate_key /etc/nginx/ssl/server.key;
  ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:ECDHE-RSA-AES128-GCM-SHA256:AES256+EECDH:DHE-RSA-AES128-GCM-SHA256:AES256+EDH:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-$
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

  # Diffie-Hellman parameter for DHE ciphersuites, recommended 2048 bits
  ssl_dhparam /etc/nginx/ssl/dhparam.pem;

  # enable ocsp stapling (mechanism by which a site can convey certificate revocation information to visitors in a privacy-preserving, scalable manner)
  # http://blog.mozilla.org/security/2013/07/29/ocsp-stapling-in-firefox/
  resolver 8.8.8.8;
  ssl_stapling on;
  ssl_trusted_certificate /etc/nginx/ssl/server.crt;

  access_log            /var/log/nginx/jenkins.access.log;

  location / {
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;

    # Fix the "It appears that your reverse proxy set up is broken" error.
    proxy_pass          http://127.0.0.1:8080/;
    proxy_read_timeout  90;

    proxy_redirect      http://127.0.0.1:8080 https://jenkins.example.com:9090;
  }
}

In the default config for jenkins I added --httpListenAddress=127.0.0.1 and in the Manage Jenkins --> Configure System I have added the correct url with port number https://jenkins.example.com:9090/ to Jenkins Location.

These are the headers when I use curl to check them.

curl -I  https://jenkins.example.com:9090/scriptApproval
HTTP/1.1 302 Found
Server: nginx/1.9.4
Date: Thu, 24 Sep 2015 13:17:56 GMT
Content-Length: 0
Connection: keep-alive
X-Content-Type-Options: nosniff
Location: https://jenkin.example.com/scriptApproval/
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: DENY

Update 1

When adding proxy_set_header X-Forwarded-Port 9090; to the nginx config this seems to fix the error It appears that your reverse proxy set up is broken on the settings page.

Update 2

Maybe it has something to do with having a trailing slash. When I call https://build.example.com:9090/pluginManager/ with curl I get a 403 Forbidden repsonse from jenkins But when a call https://build.example.com:9090/pluginManager without the trailing slash I get a 302 Found response with the location header set to https://build.example.com/pluginManager/

Update 3

This server is connected on a shared internet connect with more server running on which are beyond my control. Its only running Jenkins CI and nginx which should be the reverse proxy. The WAN port on the router is listing to port 9090 which forwards to the server on port 443 which should be Nginx which should proxy everything to Jenkins-CI which is listening to port 8080.

Update 4

This is current config I have tried. Which also doesn't seem to work.

upstream jenkins {
  server 127.0.0.1:8080 fail_timeout=0;
}

server {
  listen 9090 default ssl http2;
  server_name build.pixplicity.com;

  ssl on;
  ssl_certificate /etc/nginx/ssl/server.chain.crt;
  ssl_certificate_key /etc/nginx/ssl/server.key;

  access_log            /var/log/nginx/jenkins.access.log;

  location / {
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto https;
    proxy_set_header        X-Forward-Port 9090;

    # Fix the "It appears that your reverse proxy set up is broken" error.
    proxy_pass          http://127.0.0.1:8080;
    proxy_read_timeout  90;

    proxy_redirect http://127.0.0.1:8080 https://build.pixplicity.com:9090;
    #proxy_redirect default;
  }
}

Frerichs

If I'm reading this correctly, shouldn't you just need to change the listen 443 ssl spdy; line to listen 9090 ssl spdy;? – Aphotic

(plus then changing your proxy_pass to redirect to 8080). – Aphotic

@DevinHoward changing nginx to listen to port 9090 doesn't solve the problem. the Location header still redirects to jenkins.example.com with out the port number. And proxy_pass is already set to port 8080? – Frerichs

I wonder if you have other nginx sites interfering with this one. Can you try disabling all other sites-enabled nginx config files and then restarting nginx and seeing if it makes a difference? – Aphotic

This server is only running jenkins+nginx and nothing else – Frerichs

Answer 1

Update the following lines in your Update 4 configuration:

listen 443 default ssl http2;

proxy_set_header        Host $host:9090;

Conch

This answer was worth the bounty! Thanks @Conch – Bulgar

Can we get a bit more information, why did this solve the problem ? – Bonnybonnyclabber

@Bonnybonnyclabber If only the line proxy_set_header Host $host; is used, the port number in the url will disappear every time you perform an redirecting action in the Jenkins web interface. The necessary information for obtaining the port number can be found in the nginx documentation: proxy_set_header Host $host:$proxy_port; Unfortunately, I cannot provide more detailed information, since reverse proxy configurations are not my area of expertise. – Conch

Answer 2

This is usually due to the proxy_redirect being at set at an incorrect value (most often people try to change it from the default value of default, making stuff not work).

I'm not sure I understand your config — why is your listen directive only mentions 443, without any mentions of 9090, yet in proxy_redirect you're doing a replacement for 9090?

You should probably set proxy_redirect to the default value of default, setup listen correctly (instead of doing redirections through the firewall?), and then the problems will likely go away.

---

If the above doesn't help, then you'd have to find out what Location jenkins provides when it replies to the requests from nginx, and specify such prefix within the proxy_redirect directive. I would recommend something like sudo tcpdump -A -ilo port 8080 whilst executing the curl query as above to find out exactly what it is that Jenkins returns, to make sure to account for all the extra variables when the requests are made from within nginx, like the extra X- headers and such.

---

If all else fails and you don't really care for hairy configs, and you know that you don't plan to redirect to any foreign hosts from within this jenkins server, then perhaps something like the following is worth a try as well:

proxy_redirect ~^https?://[^/]+/(?<u>.*)$ https://jenkins.example.com:9090/$u;

(Or, of course, you could also use some extra regular expressions, depending on what it is that jenkins returns in its Location headers back to nginx.)

However, just to make sure people don't copy-paste the above without reading all the prior disclaimers, let's reiterate that the real advice is still as follows:

proxy_redirect default;

Doloroso

the reason I set the proxy_redirect is because that is described on the jenkins website when you want to use nginx with subdomain and ssl. See wiki.jenkins-ci.org/display/JENKINS/… as for the listing to 443 on nginx see my comment on Devin's answer about router --> nginx --> jenkins. 9090 --> 443 --> 8080 – Frerichs

You keep changing your question; yet still don't address the solutions proposed -- so, if you're not listening to 9090 with nginx, then indeed you cannot use default, but you could still do the other things described in this answer. Did you try any of that? What results did you get? – Doloroso

I have set listen to 9090 directly and tried both proxy_redirect default. the one I had before and both don't seem to work. – Frerichs

@Aegis, what about the hairy proxy_redirect as above? – Doloroso

@Aegis, I've tested your latest posted configuration, and it seems to work perfectly -- curl -k -v https://build.pixplicity.com:9090/pluginManager |&fgrep Location < Location: https://build.pixplicity.com:9090/pluginManager/; should you probably just clear your cache or whatever – Doloroso

This is because i'm using apache now which just works. – Frerichs

Answer 3

Step 1: change listen 443 ssl spdy; to listen 9090 ssl spdy;

Step 2: change proxy_pass http://127.0.0.1:8080/; to proxy_pass http://127.0.0.1:8080;

I might also try getting rid of those weird headers in lines 4 & 5, but that's just a guess.

If this doesn't work, I'd start again from scratch alongside reading the Digital Ocean tutorial, it's really good.

It throws me a lot that you're listening on port 443 with nginx but expecting it to serve on port 9090. In all likelihood, when you curl 9090, you're accessing jenkins directly. Either that or you copy-pasted the wrong block in SO.

Anyways, you probably need to get clearer on which server is serving on which port, and the order requests come through. In my mind, it should be one of these two:

1) https://jenkins.example.com:9090 goes to nginx listening on port 9090, which proxies it to jenkins, which is at http(s)://127.0.0.1:8080 (different app, same server).

2) https://jenkins.example.com goes to nginx listening on port 443, which proxies it to jenkins, which is at http(s)://127.0.0.1:9090.

Aphotic

The server is connected to the internet on a shared internet connection so I can only access it from the outside on port 9090. The router does the 9090 --> 443 transformation. I also tried setting nginx to directly listen directly on port 9090. also both nginx and jenkins are running on the same physical server. jenkins is listing on port 8080 as with the default installation. and NGINX is listening on port 443. – Frerichs