413 error with Kubernetes and Nginx ingress controller

Question

I'm trying to change the client_max_body_size value, so my NGINX ingress will not return the HTTP 413 Content Too Large error (as seen in the logs).

I've tested a few solutions.
Here is my config map:

kind: ConfigMap
apiVersion: v1
data:
  proxy-connect-timeout: "15"
  proxy-read-timeout: "600"
  proxy-send-timeout: "600"
  proxy-body-size: "8m"
  hsts-include-subdomains: "false"
  body-size: "64m"
  server-name-hash-bucket-size: "256"
  client-max-body-size: "50m"
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app: ingress-nginx

These changes had no effect at all: in NGINX controller's log I can see the information about reloading the config map, but the values in nginx.conf are the same:

$ cat /etc/nginx/nginx.conf | grep client_max                                                                                                       
                            client_max_body_size                    "8m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";
                            client_max_body_size                    "1m";

My nginx-controller config uses this image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.13.0

How can I force NGINX to change this setting? I need to change it globally, for all my ingresses.

Haversine

Answer 1

You can use the annotation nginx.ingress.kubernetes.io/proxy-body-size to set the max-body-size option right in your Ingress object instead of changing a base ConfigMap.

Here is the example of usage:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-app
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
...

Modie

I wanted to set it globally, but actually this approach is better as I have my yaml files within the project repository. So it won't be lost. Thanks. – Haversine

So the client-max-body-size config line can be removed entirely from the metadata.annotations ? – Veracity

@Veracity it depends. refer to https://mcmap.net/q/325299/-413-error-with-kubernetes-and-nginx-ingress-controller. If the body is too big, client-max-body-size is required to improve the performance. See Annotations: Client Body Buffer Size – Bucket

kubectl patch ingress my-ingress -p '{"metadata": {"annotations": {"nginx.ingress.kubernetes.io/proxy-body-size": "0"}}}' – Batsheva

For me it was nginx.ingress.kubernetes.io/client-body-buffer-size – Figured

I was experiencing the same issue, your answer helped me. Related question: Do you guys know if there is a default value for nginx.ingress.kubernetes.io/proxy-body-size. If yes what is it? I was getting 413 when trying to upload files >20mb. Could that be the default? I'm not sure, because didn't find anything related in the docs. – Kevel

Answer 2

There is confusion here about whether the correct annotation is nginx.ingress.kubernetes.io/proxy-body-size or nginx.org/client-max-body-size. The answer is that it might be either!

In the unassailable wisdom of everyone involved, there are two different "NGINX ingresses" in the world. It's hard to tell which you are running, and Google Search is useless at differentiating here (but what's new).

If you are using the official NGINX Ingress, the correct annotation is here: https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/

If you are using the official Kubernetes Ingress based on NGINX, the correct annotation is this: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-max-body-size

If the Docker image name of your ingress implementation is k8s.gcr.io/ingress-nginx/controller then you're using the official Kubernetes version. By elimination, you can determine if you're using the NGINX implementation.

I hope everyone who contributed to this confusing state has a very bad day. The rest of you... stay cheery!

Asexual

This comment made my day at the end :D – Palacio

The part about figuring out which version I had running finally helped me! – Wingback

More info and comparison of two Nginx ingresses can be found in Nginx blog. – Harville

Thank you @aaron , this works like a charm. Pretty much everything else I tried didn't work. And its super confusing why the proxy-body-size isn't found on main nginx website documentation. Very well written ! – Mazda

Answer 3

To set it globally, this configmap.md documentation might be helpful. Turns out the variable to set is proxy-body-size, not client-max-body-size.

When you deploy the helm chart, you can set --set-string controller.config.proxy-body-size="4m".

Desiccator

Answer 4

Maybe some people got stuck here recently. We just made out that the annotation nginx.ingress.kubernetes.io/proxy-body-size is not longer being applied and the correct one would be:

  annotations:
    nginx.org/client-max-body-size: "999m"

Hope this helps to others.

Brabazon

By the way, those two annotations for two different nginx ingress controllers. The domain in the prefix is a good reference. – Voluptuous

@Voluptuous indeed. The problem is that google search sends people to this post and I thought (maybe wrong) that, however this is not the correct question, this response can shorten their search for a solution. – Brabazon

this one is the right one! – Seldom

Answer 5

Update:

I have been experiencing the same problem and no solutions were working. After reading through countless blogs and docs that all had the same suggested solution I found that they have changed the naming convention.

It is no longer denoted by "proxy-body-size" or this just never works for me.

link below shows that the correct configmap variable to use is "client-max-body-size"

https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/

Aster

For those reading, this comment applies when using the NGINX-issued ingress controller. If you are using the Kubernetes-issued NGINX controller the property name is still proxy-body-size. You can tell this by the annotation key which will be nginx.ingress.kubernetes.io/xxxxx for Kubernetes-issued rather than nginx.org/xxxxx which is NGINX-issued. – Glebe

I encourage the folks at Nginx and Kubernetes to get together and develop a common spec – Vernon

Answer 6

you can always make a global change and disable it at all

kubectl -n ingress-nginx patch configmap \
nginx-configuration -p '{"data": {"proxy-body-size": "0"}}'

or set to 5tb

kubectl -n ingress-nginx patch configmap \
nginx-configuration -p '{"data": {"proxy-body-size": "5tb"}}'

Danube

Answer 7

I have tried both proxy-body-size and client-max-body-size on the configmap and did a rolling restart of the nginx controller pods and when I grep the nginx.conf file in the pod it returns the default 1m. I am trying to do this within Azure Kubernetes Service (AKS). I'm working with someone from their support. They said its not on them since it appears to be a nginx config issue.

The weird hting is we had other clusters in Azure that this wasnt an issue on until we discovered this with some of the newer deployments. The initial fix they came up with is what is in this thread but it just refuses to change.

Below is my configmap:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  client-max-body-size: 0m
  proxy-connect-timeout: 10s
  proxy-read-timeout: 10s
kind: ConfigMap
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"nginx-nginx-ingress-controller-7b9bff87b8-vxv8q","leaseDurationSeconds":30,"acquireTime":"2020-03-10T20:52:06Z","renewTime":"2020-03-10T20:53:21Z","leaderTransitions":1}'
  creationTimestamp: "2020-03-10T18:34:01Z"
  name: ingress-controller-leader-nginx
  namespace: ingress-nginx
  resourceVersion: "23928"
  selfLink: /api/v1/namespaces/ingress-nginx/configmaps/ingress-controller-leader-nginx
  uid: b68a2143-62fd-11ea-ab45-d67902848a80

After issuing a rolling restart: kubectl rollout restart deployment/nginx-nginx-ingress-controller -n ingress-nginx

Grepping the nginx ingress controller pod to query the value now reveals:

kubectl exec -n ingress-nginx nginx-nginx-ingress-controller-7b9bff87b8-p4ppw cat nginx.conf | grep client_max_body_size
            client_max_body_size                    1m;
            client_max_body_size                    1m;
            client_max_body_size                    1m;
            client_max_body_size                    1m;
            client_max_body_size                    21m;

Doesnt matter where I try to change it. On the configmap for global or the Ingress route specifically.......this value above never changes.

Execute

I can confirm this, we are running in exactly the same problem. AKS version 1.18.4. – Legendary

It seems adding nginx.ingress.kubernetes.io/proxy-body-size: "50m" annotation on ingress updates the "client_max_body_size" value on the nginx controller config file. – Toor

Answer 8

I am able to fix it by redeploying the nginx-ingress controller after modifying the configmap.

kubectl patch deployment your_deployment -p "{\"spec\": {\"template\": {\"metadata\": { \"labels\": {  \"redeploy\": \"$(date +%s)\"}}}}}"

Blocker