Ulimits on AWS ECS Fargate

Question

The default ULIMIT "NOFILE" is set to 1024 for containers launched using Fargate. So if I have a cluster of let's say 10 services with two or three tasks each (all running on Fargate), what are the implications if I set them all to use a huge NOFILE number such as 900000?

More specifically, do we need to care about the host machine? It's my assumption that if I were using the EC2 launch type and set all my tasks to effectively use as many files as they wanted, the hosting EC2 instance(s) could easily get overwhelmed. Or maybe the hosts wouldn't get overwhelmed but the containers registered on the hosts would get a first come first served number of files they can open potentially leading to one service starving another? But as we don't manage the instances on EC2, what's the harm in setting the ULIMIT as high as possible for all services? Do our containers sit side-by-side on a host and would therefore share the hosts resource limits. Or do we get a host per service / per task?

Of course it's possible my assumptions are wrong about how this all works.

Rosenblum

This is probably not the right place for that question. Maybe you should look for the answer to Super User or Server Fault? – Yokum

What were your findings hers, Tom? – Weidman

Answer 1

A slight correction on this answer. Like the linked documentation states, these are the DEFAULT soft and hard limits for ulimit nofile. You can override this by updating your ECS Task Definition. The Ulimit settings go under the ContainerDefinitions section of the Definition.

I've successfully set the soft and hard limits for nofile on some of my AWS Fargate Tasks using this method.

So while you cannot use the Linux "ulimit -n" command to change this on the fly, you can alter it via the ECS Task Definition.

EDIT: I've done some testing and for my setup, running AWS ECS Fargate on a Python Bullseye distro, I was able to max out at NOFILE = 1024 x 1024 = 1048576 files.

{
  "ulimits": [
    {
      "name": "nofile",
      "softLimit": 1048576,
      "hardLimit": 1048576
    }
  ],
}

Any integer multiple added to this (1024 x 1024 x INT) caused ECS to report an error when trying to start up the ECS Fargate Task:

CannotStartContainerError: ResourceInitializationError: failed to create new container runtime task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container

Hope this helps someone.

Please refer to: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions-ulimit.html

Finny

Answer 2

The maximum nofile limit on fargate is 4096

Amazon ECS tasks hosted on Fargate use the default resource limit values set by the operating system with the exception of the nofile resource limit parameter which Fargate overrides. The nofile resource limit sets a restriction on the number of open files that a container can use. The default nofile soft limit is 1024 and hard limit is 4096.

https://docs.aws.amazon.com/AmazonECS/latest/userguide/task_definition_parameters.html

Christiechristin

That's the default hard limit. It can be upped. – Bourgogne