I am trying to find cause for high CPU usage of my app.
TL;DR
My relatively simple .Net6 API application consumes 100% of 1vCPU doing only ~20 requests (each ~100ms) per second (only 2 concurrent requests). dotnet-trace shows inconclusive bottlenecks.
Bit of context:
A .Net6, ASP API application. App runs in docker container on kubernetes cluster. It is in .Net6, and it has dedicated 1000m CPU (meaning 1 full vCPU).
In order to fulfill a request the app calls DB twice or thrice, and external APIs few times. The app does not have any heavy calculations (no image processing, no huge lists, etc.). Just awaited DB calls and API calls, some JSON serialization, HttpClient calls, etc.
The issue is, that 1 pod with dedicated 1000m CPU (1 vCPU) is handling about 200 requests per second, each request is about 100ms, meaning it handles only about 2 requests concurrently. Such load consumes almost 100% dedicated CPU, and new pods are created.
IMO, with majority of work being "waiting for DB or API to return data" my server (each pod) should handle about 10x more load.
dotnet-trace
I run dotnet-trace in my container, and I can't really understand the data. My code is responsible for less that 5% of total CPU consumption. Following functions consumes most of it:
ad 1. ~21% of CPU System.Reflection.Emit.DynamicMethod.CreateDelegate - what this might mean? With the following hint does it mean NewtonsoftJson compiles code every time it serialize something?
ad 2. ~13% of CPU `System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath' - does it mean my program constantly loads some assemblies? Hint:
ad 3. ~11% of CPU System.Threading.LowLevelLifoSemaphore.WaitForSignal - are semaphores so expensive in use? I use it like this:
private readonly SemaphoreSlim _mySemaphore = new SemaphoreSlim(1);
(...)
await _mySemaphore.WaitAsync();
(...)
finally
{
_mySemaphore.Release();
}