What is my Lambda doing between startup and first line?

Question

I have some Lambda functions written in C# running in the .NET Core 2.1 runtime in AWS. The cold start time on them is very large (>8s with 256MB, >4s with 512).

However, I'm not sure if it is just cold start time or something else; I have other lambdas that are written in dotnet and they seem to have shorter startup times.

An X-Ray trace shows a big gap between "Initilization" being completed and anything happening. I start an X-Ray subsegment on the first line of my handler (seen in the trace as "Configure").

Is there something I'm missing?

Affluence

I should add that once warm the entire duration is basically spent on the S3 request, so it only seems to happen on the first request. The package is only around 3MB. – Affluence

If the time is linear or nearly so, compared to the memory you specify for the container, then -- whatever it is -- it's CPU-bound. – Corinthians

Yeah agreed, it's definitely CPU-bound. Just not sure if it's my code, or the AWS environment doing something. – Affluence

Answer 1

The "?" in the image you provided is Lambda system code.

The "initialization" segment includes running the constructor and static blocks for your function. But it also includes some reflection on your assembly to validate and find your function class and constructor. It also finds and validates your serializer, if you're using one.

I can't give a definitive answer as to why this function is behaving differently from other functions. Could you show some graphs of the functions you're comparing to with descriptions of the functions (memory, what the functions are doing, what dependencies are in the zip package)?

Juieta

Ahh ok, I didn't know about the reflection part - but that makes sense I guess. Is there any documentation, a blog post or something that explains what happens during cold starts? Never managed to find much detail when I've looked. I'll get some traces of the other functions and add those to the question for comparison. – Affluence

Happy to award this answer the bounty if you can provide a source for the reflection bits? – Affluence

docs.aws.amazon.com/lambda/latest/dg/running-lambda-code.html this might be useful but it doesn't mention the reflection bits – Longshoreman