How to change nginx config in amazon elastic beanstalk running a docker instance

Question

After i login and the cookie is set I get error 502. When i read the log i get the error:

014/05/17 01:54:43 [error] 11013#0: *8 upstream sent too big header while reading response
header from upstream, client: 83.248.134.236, server: , request: "GET /administration
HTTP/1.1", upstream:

After some fast googling i found: http://developernote.com/2012/09/how-i-fixed-nginx-502-bad-gateway-error/

and I want to try to set fastcgi_buffers and fastcgi_buffer_size to a different value. But how do i set variable on nginx in amazon elasticbeanstalk?

The nginx server is before my docker instance.

Heartbreak

Answer 1

I also needed to modify the nginx configuration.

1. Create a script that modifies the nginx configuration (probably you want /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker.conf) and restarts the nginx service (service nginx restart).
2. You need to execute that script after this nginx config file is written which is after normal ebextensions are executed. This is undocumented, but Evan shared how to do this here: essentially you use an ebextension to copy the script into a directory with hooks that gets executed at the proper time.

An example ebextension config is .ebextensions/01modify_nginx.config:

container_commands:
  copy:
    command: "cp .ebextensions/01rewrite_nginx_config.py /opt/elasticbeanstalk/hooks/appdeploy/enact/"
  make_exe:
    command: "chmod +x /opt/elasticbeanstalk/hooks/appdeploy/enact/01rewrite_nginx_config.py"

This is working nicely now for my project (here is the source where you can see it in action).

Bonine

I had to rename the editing script 01rewrite_nginx_config.py to 12_rewrite_nginx_config to get it to execute late enough. – Sharpeyed

Answer 2

Amazon actually recommends editing the staging version of the nginx deployment file. There are several located at /tmp/deployment/config/, one for editing the general 'http' context, and then a few for configuring different aspects of the server.

I wanted to attach caching functionality to the default proxy server, so I wrote an .ebextensions config file to replace #etc#nginx#conf.d#00_elastic_beanstalk_proxy.conf, which is then copied over to /etc/nginx/conf.d during deployment. You can inline the file if its simple enough, but I put mine in S3 so that different applications and pull it down and use it. Here's the config file:

commands: 
  01-get-nginx-conf-file:
    command: aws s3 cp s3://<bucket-name>/custom-nginx.conf /home/ec2-user

container_commands:
  01-replace-default-nginx-config:
    command: mv -f /home/ec2-user/custom-nginx.conf /tmp/deployment/config/#etc#nginx#conf.d#00_elastic_beanstalk_proxy.conf

Heliogravure

After all kinds of different attempts this one worked like a charm. No need to restart nginx at exactly the right time or fear for having your script overwritten, just overwrite it at the source. Beautiful. – Cerumen

Does it matter what you name this file? Also, how do you deploy your custom nginx config file to S3? Is there a way have eb deploy copy the file from your machine to the S3 bucket rather than copying it manually? I'm sure you could create another S3 bucket for this but I want to use the same EBS bucket that was created for me. – Nuclei

The only restriction on the filename is that it ends in .config: docs.aws.amazon.com/elasticbeanstalk/latest/dg/…. Not sure I understand the second part though. You only need to copy it to S3 one time and then every time you deploy it copies it FROM S3 to your instance. If you don't want to use S3 at all you can just paste the entire file inline, but this is a little brittle as your applications are now not sharing the same centralized file. – Heliogravure

Do you know if I can use a file in /var/app/current? I'm not sure where in the deployment lifecycle this is happening - ie if my files will be there yet. – Vashtivashtia

^ In response to my question, doing so did not work for me. – Vashtivashtia

Answer 3

Another way to extend Elastic Beanstalk nginx config is to create a file in the .ebextensions directory, named for example nginx.config with the following content :

    files:
      "/etc/nginx/conf.d/000_my_config.conf":
      content: |
        upstream nodejsserver {
          server 127.0.0.1:8081;
          keepalive 256;
        }

        server {
          listen 8080;

          location / {
            proxy_pass  http://nodejsserver;
            proxy_set_header   Connection "";
            proxy_http_version 1.1;
            proxy_set_header        Host            $host;
            proxy_set_header        X-Real-IP       $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
          }

          location /myconfig {
            proxy_pass http://my_proxy_pass_host;
          }
        }

/etc/nginx/conf.d/000_my_config.conf is the filename which will be created on the Elastic Beanstalk EC2 instances. By default this configuration is in the file /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf. So if you prefix with 000, it guarantees you that your configuration will be taken into account first.

The content has been copied from the default nginx configuration (/etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf again), then customized with my own configuration.

Agnail

Dunno about previous versions, but currently if you do this you'll get a nasty error message from Nginx, duplicate upstream "nodejs" in /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf:12. Not only that, you'll have to login to each Beanstalk node and manually remove the 000_my_config.conf file. Be warned. – Creamcups

@TimFulmer : that's exactly the reason why I called my upstream "nodejsserver" and not "nodejs". – Agnail

You can remove files by adding a command section to the same file commands: 01remove_backup: command: rm -f [your file] (updating idententation etc) – Duthie

Yeah, see this AWS article addressing this – Idolism

Answer 4

Update as of 2022 Feb 2nd

Seems like AWS has changed some stuff in newer versions of Elastic Beanstalk that uses Amazon Linux 2 so the approach that was mentioned by @jsebfranck no longer works if you are using EBS with Amazon Linux2.

The newer approach is to create a ".platform" folder in root of your zip bundle and add your nginx.conf file at "./platform/nginx/nginx.conf" the eb engine should swap the file if found in that location.

See here for details: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.migration-al.html

Also see "Reverse proxy configuration" > "Configuring nginx" section here.

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/platforms-linux-extend.html

Captain

Answer 5

A cleaner approach (if you're using the Java or Go platform on Elastic Beanstalk) is to have nginx .conf files with your wished changes in a subfolder in .ebextensions:

You can now place an nginx.conf file in the .ebextensions/nginx folder to override the Nginx configuration. You can also place configuration files in the .ebextensions/nginx/conf.d folder in order to have them included in the Nginx configuration provided by the platform.

Source

Desdee

It is worth to mention that this solution work only with some ELB Platforms - Java and Go. If you use for example Docker Platform with nginx as proxy you have to use solution with configuration file (that use files keyword) placed inside .ebextensions – Broch

@Broch i was about to go crazy. been stucked with this for hours, thank you for pointing it out – Mutilate

By "ELB" I think you mean Elastic Beanstalk and not Elastic Load Balancer. I wish they would support this for all platforms! :/ – Mulciber

Answer 6

Update as of 2023 october

As Amazon Linux 2 (AL2) will reach its end of life in 2025, AWS now recommends to start every new ElasticBeanstalk environment with a platform based on Amazon Linux 2023.

All the configuration related to nginx remains the same as it were with AL2.

You can continue to place custom configurations in

.platform/nginx/nginx.conf (to override the Elastic Beanstalk default nginx configuration completely)

or

.platform/nginx/conf.d/myconf.conf (to extend the Elastic Beanstalk default nginx configuration)

Extending Elastic Beanstalk Linux platforms

Brathwaite

In other words: if you are deploying a Docker-based containerized app (like I am now) - use the .platform directory method described above. Initially I wasn't sure what my platform is using - in my EB config I just put platform: docker – Chengteh