how many uvicorn workers do I have to have in production?

Question

My Environment

• FastAPI
• Gunicorn & Uvicorn Worker
• AWS EC2 c5.2xlarge (8 vCPU)

Document

https://fastapi.tiangolo.com/deployment/server-workers/

Question

Currently I'm using 24 Uvicorn workers in production server. (c5.2xlarge)

gunicorn main:app --workers 24 --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:80

I've learn that one process runs on one core. Therefore If i have 8 processes, I can make use of whole cores (c5.2xlarge's vCpu == 8)

I'm curious that in this situation, Is there any performance benefit if I got more processes than 8?

Poverty

(most EC2s) 1 vCPU = 1 Thread, and each core has 2 threads. so in your case c5.2xlarge = 4 cpu cores docs.aws.amazon.com/AWSEC2/latest/UserGuide/… – Acetum

Answer 1

Number of recommended workers is 2 x number_of_cores +1

You can read more about it at https://docs.gunicorn.org/en/stable/design.html#:~:text=Gunicorn%20should%20only%20need%204,workers%20to%20start%20off%20with.

In your case with 8 CPU cores, you should be using 17 worker threads.

Additional thoughts on async systems:

The two times core is not a scientific figure as says in the article. But the idea is that one thread can do I/O and another CPU processing at the same time. This makes maximum use of simultaneous threads. Even with async systems, conceptually this works and should give you maximum efficiency.

Creeps

Thank you! But I'm curious is this applicable to async workers? Since they can process other thing while waiting for the IO (due to event loop) – Poverty

I have added this in the answer. For everyone's reference. – Creeps

Answer 2

In general the best practice is:

number_of_workers = number_of_cores x 2 + 1

or more precisely:

number_of_workers = number_of_cores x num_of_threads_per_core + 1

--> BUT be careful not to confuse number_of_cores with vCPU.

The reason for it is CPU hyperthreading, which allows each core to run multiple concurrent threads. The number of concurrent threads is decided by the chip designers.

Two concurrent threads per CPU core are common, but some processors can support more than two.

vCPU that is mentioned for AWS ec2 resource is already the hyperthreaded amount of processing units you have on the machine (num_of_cores x num_of_threads_per_core). Not to be confused with number_of_cores available on that machine.

So, in your case, c5.2xlarge has 8 vCPUs, meaning you have 8 available concurrent workers.

Hindi

So he should run --workers 9 ( number_of_workers = number_of_cores x num_of_threads_per_core + 1 = 8 + 1 ) and the accepted answer is in fact misleading ? – Deirdra

For the time of writing there is no accepted answer, but if you mean, should he run 8 vCPUs x 2 + 1 = 17, so probably not as he will choke the workers when high traffic will hit and workers would like to do 17 concurrent CPU operations while having only 8 available. – Hindi