Boto3 Client Creation is Slow

Question

I am running boto3 on AWS Lambda but getting a boto3 client is very slow. I am currently running a python3.9 runtime on AWS Lambda that creates handfuls of presigned url for s3 resources.

A key line of code required to generate the url with boto3 is

print("Getting boto3")
self.s3_client = boto3.client('s3')
print("Got client s3")

shockingly, the above single line of code takes a full 2000ms to execute according to the logs in cloudwatch:

2023-04-18T13:34:18.891-04:00   Getting boto3
2023-04-18T13:34:20.733-04:00   Got client s3

Is this normal and can I make it go faster? I saw an old post indicating that client acquisition speed might be improved by explicitly passing credentials or increasing memory size. It seems strange that the basic s3 interface cannot be acquired in less than 2s.

Bladdernut

Make sure that the client initialization is outside of the lambda handler, this way it will be cached for (close) subsequent executions. See docs.aws.amazon.com/lambda/latest/dg/best-practices.html – Mainmast

Lambda memory allotment also controls CPU allotment. IIRC, 128 MB means 1/24th of a CPU. You can see a comparison of times versus memory in an answer I wrote several years ago, but which remains accurate (and is for Java, but the same issue applies to Python -- there's a lot that happens when creating a client). – Nkvd

Initializing it outside the function handler makes it reusable for warm starts but more importantly it's executed with burst CPU and costs less (potentially nothing). – Seneschal

@Seneschal - Nice! TIL – Nkvd

All this is good information. Simply increasing Memory did force boto3 s3 initialization to a handful of milliseconds. – Bladdernut

Answer 1

Unfortunately I do not have a better solution as increasing the memory size to improve the creation time significantly There is also a blog article according to this.

However I am facing similar issues with ssm and boto3: self.ssm_client = boto3.client('ssm') takes 2800 ms whereas the actual call only takes ~ 200 ms. I am very interested in a better solution as well.

Gamic

This does not really answer the question. If you have a different question, you can ask it by clicking Ask Question. To get notified when this question gets new answers, you can follow this question. Once you have enough reputation, you can also add a bounty to draw more attention to this question. - From Review – Thoroughbred