Kubernetes Nginx Ingress file upload returning 502

Question

I am trying to upload files from a client through an nginx ingress. I have set the following annotations on the ingress after receiving a 413 response;

Annotations:   nginx.ingress.kubernetes.io/body-size: 1024m
               nginx.ingress.kubernetes.io/client-body-buffer-size: 50m
               nginx.ingress.kubernetes.io/client-max-body-size: 50m
               nginx.ingress.kubernetes.io/proxy-body-size: 1024m
               nginx.ingress.kubernetes.io/proxy-buffer-size: 32k
               nginx.ingress.kubernetes.io/proxy-buffers-number: 8

The client is an Angular application. It sends a base64 string of the file in the request body. I have tried uploading images of a few KB, so I definitely ain't hitting any of these limits. I'm new to Kubernetes. Do I need to restart the ingress for these annotations to take effect?

I have also tried creating a ConfigMap;

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-ingress-configuration
  namespace: development
  labels:
    app.kubernetes.io/name: [name of ingress]
    app.kubernetes.io/part-of: [name of ingress]
data:
  proxy-connect-timeout: "50"
  proxy-read-timeout: "120"
  proxy-send-timeout: "120"
  body-size: "1024m"
  client-body-buffer-size: "50m"
  client-max-body-size: "50m"
  proxy-body-size: "1024m"
  proxy-buffers: "8 32k"
  proxy-buffer-size: "32k"

Still getting a 502.

Not sure how to access the nginx.conf through kubectl, it seems from the docs if I update this ConfigMap the settings are changed in nginx anyway.

Any help appreciated.

UPDATE

nginx.conf

# Configuration checksum: 1961171210939107273

# setup custom paths that do not require root access
pid /tmp/nginx.pid;

daemon off;

worker_processes 2;

worker_rlimit_nofile 523264;

worker_shutdown_timeout 240s ;

events {
    multi_accept        on;
    worker_connections  16384;
    use                 epoll;
}

http {
    client_max_body_size 100M;

    lua_package_path "/etc/nginx/lua/?.lua;;";
    
    lua_shared_dict balancer_ewma 10M;
    lua_shared_dict balancer_ewma_last_touched_at 10M;
    lua_shared_dict balancer_ewma_locks 1M;
    lua_shared_dict certificate_data 20M;
    lua_shared_dict certificate_servers 5M;
    lua_shared_dict configuration_data 20M;
    
    init_by_lua_block {
        collectgarbage("collect")
        
        -- init modules
        local ok, res
        
        ok, res = pcall(require, "lua_ingress")
        if not ok then
        error("require failed: " .. tostring(res))
        else
        lua_ingress = res
        lua_ingress.set_config({
            use_forwarded_headers = false,
            use_proxy_protocol = false,
            is_ssl_passthrough_enabled = false,
            http_redirect_code = 308,
        listen_ports = { ssl_proxy = "442", https = "443" },
            
            hsts = true,
            hsts_max_age = 15724800,
            hsts_include_subdomains = true,
            hsts_preload = false,
        })
        end
        
        ok, res = pcall(require, "configuration")
        if not ok then
        error("require failed: " .. tostring(res))
        else
        configuration = res
        end
        
        ok, res = pcall(require, "balancer")
        if not ok then
        error("require failed: " .. tostring(res))
        else
        balancer = res
        end
        
        ok, res = pcall(require, "monitor")
        if not ok then
        error("require failed: " .. tostring(res))
        else
        monitor = res
        end
        
        ok, res = pcall(require, "certificate")
        if not ok then
        error("require failed: " .. tostring(res))
        else
        certificate = res
        end
        
        ok, res = pcall(require, "plugins")
        if not ok then
        error("require failed: " .. tostring(res))
        else
        plugins = res
        end
        -- load all plugins that'll be used here
    plugins.init({})
    }
    
    init_worker_by_lua_block {
        lua_ingress.init_worker()
        balancer.init_worker()
        
        monitor.init_worker()
        
        plugins.run()
    }
    
    geoip_country       /etc/nginx/geoip/GeoIP.dat;
    geoip_city          /etc/nginx/geoip/GeoLiteCity.dat;
    geoip_org           /etc/nginx/geoip/GeoIPASNum.dat;
    geoip_proxy_recursive on;
    
    aio                 threads;
    aio_write           on;
    
    tcp_nopush          on;
    tcp_nodelay         on;
    
    log_subrequest      on;
    
    reset_timedout_connection on;
    
    keepalive_timeout  75s;
    keepalive_requests 100;
    
    client_body_temp_path           /tmp/client-body;
    fastcgi_temp_path               /tmp/fastcgi-temp;
    proxy_temp_path                 /tmp/proxy-temp;
    ajp_temp_path                   /tmp/ajp-temp;
    
    client_header_buffer_size       1M;
    client_header_timeout           60s;
    large_client_header_buffers     4 5M;
    client_body_buffer_size         1M;
    client_body_timeout             60s;
    
    http2_max_field_size            1M;
    http2_max_header_size           5M;
    http2_max_requests              1000;
    http2_max_concurrent_streams    128;
    
    types_hash_max_size             2048;
    server_names_hash_max_size      1024;
    server_names_hash_bucket_size   64;
    map_hash_bucket_size            64;
    
    proxy_headers_hash_max_size     512;
    proxy_headers_hash_bucket_size  64;
    
    variables_hash_bucket_size      256;
    variables_hash_max_size         2048;
    
    underscores_in_headers          off;
    ignore_invalid_headers          on;
    
    limit_req_status                503;
    limit_conn_status               503;
    
    include /etc/nginx/mime.types;
    default_type text/html;
    
    gzip on;
    gzip_comp_level 5;
    gzip_http_version 1.1;
    gzip_min_length 256;
    gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/javascript text/plain text/x-component;
    gzip_proxied any;
    gzip_vary on;
    
    # Custom headers for response
    
    server_tokens on;
    
    # disable warnings
    uninitialized_variable_warn off;
    
    # Additional available variables:
    # $namespace
    # $ingress_name
    # $service_name
    # $service_port
    log_format upstreaminfo '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id';
    
    map $request_uri $loggable {
        
        default 1;
    }
    
    access_log /var/log/nginx/access.log upstreaminfo  if=$loggable;
    
    error_log  /var/log/nginx/error.log notice;
    
    resolver 10.245.0.10 valid=30s;
    
    # See https://www.nginx.com/blog/websocket-nginx
    map $http_upgrade $connection_upgrade {
        default          upgrade;
        
        # See http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive
        ''               '';
        
    }
    
    # Reverse proxies can detect if a client provides a X-Request-ID header, and pass it on to the backend server.
    # If no such header is provided, it can provide a random value.
    map $http_x_request_id $req_id {
        default   $http_x_request_id;
        
        ""        $request_id;
        
    }
    
    # Create a variable that contains the literal $ character.
    # This works because the geo module will not resolve variables.
    geo $literal_dollar {
        default "$";
    }
    
    server_name_in_redirect off;
    port_in_redirect        off;
    
    ssl_protocols TLSv1.2;
    
    ssl_early_data off;
    
    # turn on session caching to drastically improve performance
    
    ssl_session_cache builtin:1000 shared:SSL:10m;
    ssl_session_timeout 10m;
    
    # allow configuring ssl session tickets
    ssl_session_tickets on;
    
    # slightly reduce the time-to-first-byte
    ssl_buffer_size 4k;
    
    # allow configuring custom ssl ciphers
    ssl_ciphers '';
    ssl_prefer_server_ciphers on;
    
    ssl_ecdh_curve auto;
    
    # PEM sha: ---
    ssl_certificate     /etc/ingress-controller/ssl/default-fake-certificate.pem;
    ssl_certificate_key /etc/ingress-controller/ssl/default-fake-certificate.pem;
    
    proxy_ssl_session_reuse on;
    
    upstream upstream_balancer {
        ### Attention!!!
        #
        # We no longer create "upstream" section for every backend.
        # Backends are handled dynamically using Lua. If you would like to debug
        # and see what backends ingress-nginx has in its memory you can
        # install our kubectl plugin https://kubernetes.github.io/ingress-nginx/kubectl-plugin.
        # Once you have the plugin you can use "kubectl ingress-nginx backends" command to
        # inspect current backends.
        #
        ###
        
        server 0.0.0.1; # placeholder
        
        balancer_by_lua_block {
            balancer.balance()
        }
        
        keepalive 32;
        
        keepalive_timeout  60s;
        keepalive_requests 100;
        
    }
    
    # Cache for internal auth checks
    proxy_cache_path /tmp/nginx-cache-auth levels=1:2 keys_zone=auth_cache:10m max_size=128m inactive=30m use_temp_path=off;
    
    # Global filters
    
    ## start server _
    server {
        server_name _ ;
        
        listen 80 default_server reuseport backlog=511 ;
        listen [::]:80 default_server reuseport backlog=511 ;
        listen 443 default_server reuseport backlog=511 ssl http2 ;
        listen [::]:443 default_server reuseport backlog=511 ssl http2 ;
        
        set $proxy_upstream_name "-";
        
        ssl_certificate_by_lua_block {
            certificate.call()
        }
        
        location / {
            
            set $namespace      "";
            set $ingress_name   "";
            set $service_name   "";
            set $service_port   "";
            set $location_path  "/";
            
            rewrite_by_lua_block {
                lua_ingress.rewrite({
                    force_ssl_redirect = false,
                    ssl_redirect = false,
                    force_no_ssl_redirect = false,
                    use_port_in_redirects = false,
                })
                balancer.rewrite()
                plugins.run()
            }
            
            # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
            # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
            # other authentication method such as basic auth or external auth useless - all requests will be allowed.
            #access_by_lua_block {
            #}
            
            header_filter_by_lua_block {
                lua_ingress.header()
                plugins.run()
            }
            
            body_filter_by_lua_block {
            }
            
            log_by_lua_block {
                balancer.log()
                
                monitor.call()
                
                plugins.run()
            }
            
            access_log off;
            
            port_in_redirect off;
            
            set $balancer_ewma_score -1;
            set $proxy_upstream_name "upstream-default-backend";
            set $proxy_host          $proxy_upstream_name;
            set $pass_access_scheme  $scheme;
            
            set $pass_server_port    $server_port;
            
            set $best_http_host      $http_host;
            set $pass_port           $pass_server_port;
            
            set $proxy_alternative_upstream_name "";
            
            client_max_body_size                    1m;
            
            proxy_set_header Host                   $best_http_host;
            
            # Pass the extracted client certificate to the backend
            
            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            
            proxy_set_header                        Connection        $connection_upgrade;
            
            proxy_set_header X-Request-ID           $req_id;
            proxy_set_header X-Real-IP              $remote_addr;
            
            proxy_set_header X-Forwarded-For        $remote_addr;
            
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            
            proxy_set_header X-Scheme               $pass_access_scheme;
            
            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
            
            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";
            
            # Custom headers to proxied server
            
            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;
            
            proxy_buffering                         off;
            proxy_buffer_size                       5M;
            proxy_buffers                           4 5M;
            
            proxy_max_temp_file_size                1024M;
            
            proxy_request_buffering                 on;
            proxy_http_version                      1.1;
            
            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;
            
            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout;
            proxy_next_upstream_timeout             0;
            proxy_next_upstream_tries               3;
            
            proxy_pass http://upstream_balancer;
            
            proxy_redirect                          off;
            
        }
        
        # health checks in cloud providers require the use of port 80
        location /healthz {
            
            access_log off;
            return 200;
        }
        
        # this is required to avoid error if nginx is being monitored
        # with an external software (like sysdig)
        location /nginx_status {
            
            allow 127.0.0.1;
            
            allow ::1;
            
            deny all;
            
            access_log off;
            stub_status on;
        }
        
    }
    ## end server _
    
    ## start server dev-api
    server {
        server_name dev-api ;
        
        listen 80  ;
        listen [::]:80  ;
        listen 443  ssl http2 ;
        listen [::]:443  ssl http2 ;
        
        set $proxy_upstream_name "-";
        
        ssl_certificate_by_lua_block {
            certificate.call()
        }
        
        location / {
            
            set $namespace      "development";
            set $ingress_name   "app-ingress";
            set $service_name   "app-api-svc";
            set $service_port   "80";
            set $location_path  "/";
            
            rewrite_by_lua_block {
                lua_ingress.rewrite({
                    force_ssl_redirect = false,
                    ssl_redirect = true,
                    force_no_ssl_redirect = false,
                    use_port_in_redirects = false,
                })
                balancer.rewrite()
                plugins.run()
            }
            
            # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
            # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
            # other authentication method such as basic auth or external auth useless - all requests will be allowed.
            #access_by_lua_block {
            #}
            
            header_filter_by_lua_block {
                lua_ingress.header()
                plugins.run()
            }
            
            body_filter_by_lua_block {
            }
            
            log_by_lua_block {
                balancer.log()
                
                monitor.call()
                
                plugins.run()
            }
            
            port_in_redirect off;
            
            set $balancer_ewma_score -1;
            set $proxy_upstream_name "development-app-api-svc-80";
            set $proxy_host          $proxy_upstream_name;
            set $pass_access_scheme  $scheme;
            
            set $pass_server_port    $server_port;
            
            set $best_http_host      $http_host;
            set $pass_port           $pass_server_port;
            
            set $proxy_alternative_upstream_name "";
            
            client_max_body_size                    1024M;
            
            client_body_buffer_size                 50M;
            
            proxy_set_header Host                   $best_http_host;
            
            # Pass the extracted client certificate to the backend
            
            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            
            proxy_set_header                        Connection        $connection_upgrade;
            
            proxy_set_header X-Request-ID           $req_id;
            proxy_set_header X-Real-IP              $remote_addr;
            
            proxy_set_header X-Forwarded-For        $remote_addr;
            
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            
            proxy_set_header X-Scheme               $pass_access_scheme;
            
            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
            
            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";
            
            # Custom headers to proxied server
            
            proxy_connect_timeout                   50s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;
            
            proxy_buffering                         off;
            proxy_buffer_size                       5M;
            proxy_buffers                           8 5M;
            
            proxy_max_temp_file_size                1024M;
            
            proxy_request_buffering                 on;
            proxy_http_version                      1.1;
            
            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;
            
            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout;
            proxy_next_upstream_timeout             0;
            proxy_next_upstream_tries               3;
            
            proxy_pass http://upstream_balancer;
            
            proxy_redirect                          off;
            
        }
        
    }
    ## end server dev-api
    
.......

UPDATE 2

Log from kubectl logs -n nginx-ingress-controller-XXX command

127.0.0.1 - - [16/Jul/2020:10:11:14 +0000] "POST [Ingress/Service endpoint] HTTP/2.0" 502 4 "https://[client-host-name]/[client-path]" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36 Edg/83.0.478.58" 9351 0.659 [service-name-80] [] 10.244.1.72:80 14 0.652 502 7b7bdf8a9319e88c80ba3444372daf2d

Pamela

Can you please provide more details and minimal reproducible example? Testing it directly to the pods or the service works fine? – Brachy

The error only occurs when trying to send a base64 string representation of a file in the body of the request, other requests work fine, so I know the service is working. I have checked the logs of the ingress and I can see 502, the request never gets to the service. Testing locally works fine, only when it's deployed to K8S uisng nginx ingress does the error appear. – Pamela

@mWatney added the nginx.conf file there and removed and identifying host/pod names etc. – Pamela

As far as I can tell, your nginx.conf looks right. Can you share the nginx and application logs? – Brachy

When i cat var/log/nginx/error.log and var/log/nginx/access.log nothing is returned. Do I need to turn on nginx logging explicitly? – Pamela

What you can see on kubectl logs -n <namespace> nginx-ingress-controller-XXX? – Brachy

Sorry, yes I got them earlier, I see the 502 errors, but there's no error message or further information. I will add log to question. – Pamela

Answer 1

My original issue was nginx, but after I changed the limits it was forwarding the request to the service but I wasn't checking the correct logs. So @mWatney was correct to double check directly to the service/pod.

For anyone else the issue I was seeing was to do with running a .NET core 3.1 application in the Linux alpine container. Within the app I was using a version of System.Drawing.Common which causes exceptions running under Linux, whihc look like this;

System.TypeInitializationException: The type initializer for 'Gdip' threw an exception. ---> System.DllNotFoundException: Unable to load shared library 'libgdiplus'

The solution was to add to the dockerfile;

RUN apk add libgdiplus-dev fontconfig ttf-dejavu --update-cache --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ --allow-untrusted

This allows the use of System.Drawing.Common under Linux by adding the ability to load shared library libgdiplus.

Credit here: https://github.com/dotnet/dotnet-docker/issues/618#issuecomment-467619498

The more permanent solution is to remove the dependency on System.Drawing.Common entirely from the application. Thanks @mWatney for help, you got me on the right track.

Pamela

Answer 2

You need to make sure you have the file size set up on the ingress controller. Nginx will catch up the settings. try this. For more information about the annotations follow this. https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: service-api-tls-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 8m

Nimitz

I have added those annotations already, I added kubernetes.io/ingress.class: nginx also, and still get a 502. I have added them to the Ingress resource and the nginx-ingress-controller service resource – Pamela

Answer 3

In case anyone came across this page and found it weird that Ingress Nginx returns a 502 error, which is usually corresponding with this line in the Nginx logs:

upstream sent too big header while reading response header from upstream

The proxy_buffering in Ingress Nginx has been always disabled by default (since 2018) [1]

That means all proxy buffering settings are NOT active unless you explicitly enable it via the annotation:

nginx.ingress.kubernetes.io/proxy-buffering: 'on'

[1] https://github.com/kubernetes/ingress-nginx/commit/b0206865991e22c6e0cf6f59ad42473d63cfd6dd#diff-51e84dc53d33590d1f9d03afbfd9cbd174399ef97d346d0123cb5f4450bf01bbR414

Wetterhorn