Adding global tags with System.Diagnostics.Metrics & OpenTelemetry

I might have missed it in the documentation, but I just can't find a way to do it.

I'm looking for a way to globally set some tags/labels that will be added to all metrics in my service. Motivation: I have multiple DCs and multiple environments in a single DC. I want all metrics reported with DC and ENV as a dimension. I'd prefer not to have to add those dimensions manually on each measurement, especially since the developers will probably forget to add those.

I've created a small console application to demonstrate what I'm missing (please find code bellow or at https://dotnetfiddle.net/aIRBu0). Checking the prometheus endpoint all I get is:

# TYPE click_c_clicks_total counter
# UNIT click_c_clicks_total clicks_total
# HELP click_c_clicks_total count the number of clicks
click_c_clicks_total{key="a"} 14 1682246108134
click_c_clicks_total{key="s"} 15 1682246108134
click_c_clicks_total{key="d"} 14 1682246108134
click_c_clicks_total{key="e"} 2 1682246108134
click_c_clicks_total{key="r"} 2 1682246108134
click_c_clicks_total{key="w"} 1 1682246108134

# TYPE click_t_ms histogram
# UNIT click_t_ms ms
# HELP click_t_ms time to click after prompted
click_t_ms_bucket{le="0"} 0 1682246108134
click_t_ms_bucket{le="5"} 0 1682246108134
click_t_ms_bucket{le="10"} 0 1682246108134
click_t_ms_bucket{le="25"} 0 1682246108134
click_t_ms_bucket{le="50"} 0 1682246108134
click_t_ms_bucket{le="75"} 17 1682246108134
click_t_ms_bucket{le="100"} 24 1682246108134
click_t_ms_bucket{le="250"} 42 1682246108134
click_t_ms_bucket{le="500"} 44 1682246108134
click_t_ms_bucket{le="750"} 47 1682246108134
click_t_ms_bucket{le="1000"} 47 1682246108134
click_t_ms_bucket{le="2500"} 47 1682246108134
click_t_ms_bucket{le="5000"} 47 1682246108134
click_t_ms_bucket{le="7500"} 47 1682246108134
click_t_ms_bucket{le="10000"} 47 1682246108134
click_t_ms_bucket{le="+Inf"} 48 1682246108134
click_t_ms_sum 29085 1682246108134
click_t_ms_count 48 1682246108134

# EOF

I'm looking for a way to have the DC and the ENV as part of those metrics. I didn't find a way in the documentation of OTEL or microsoft for system diagnostics metrics.

Ideas?

code:

var host = Host
    .CreateDefaultBuilder()
    .ConfigureLogging(logger => logger.AddConsole())
    .ConfigureServices(services =>
    {
        services.AddOpenTelemetry()
            .WithMetrics(metricsBuilder =>
            {
                metricsBuilder.ConfigureResource(resourceBuilder =>
                {
                    resourceBuilder
                        .AddService("ClickService")
                        .AddAttributes(new[]
                        {
                            new KeyValuePair<string, object>("dc", Environment.GetEnvironmentVariable("dc") ?? "us1"),
                            new KeyValuePair<string, object>("env", Environment.GetEnvironmentVariable("env") ?? "local"),
                        })
                        .AddTelemetrySdk();
                });
                var meterProviderBuilder = metricsBuilder.AddMeter("custom-console-meter");

                metricsBuilder.AddConsoleExporter();

                metricsBuilder.AddPrometheusHttpListener(httpListenerOptions =>
                {
                    httpListenerOptions.UriPrefixes = new[] { "http://localhost:9464/" };
                });
            });

        services.AddHostedService<ClickAppHost>();
    });

await host.RunConsoleAsync();

this is the click app host

// a simple application the loops until a CTRL-C combination is not added
// counts the number of clicked buttons
public class ClickAppHost : IHostedService
{
    private readonly CancellationTokenSource _work = new();

    public Task StartAsync(CancellationToken cancellationToken)
    {
        Console.CancelKeyPress += (sender, args) =>
        {
            Console.WriteLine("cancellation requested");
            _work.Cancel();
        };

        while (!_work.Token.IsCancellationRequested)
        {
            Console.WriteLine("Click a key");
            var sw = Stopwatch.GetTimestamp();
            
            var key = Console.ReadKey(intercept: true);
            
            var elapsed = Stopwatch.GetElapsedTime(sw);
            var combination = GetKeyLabel(key);
            Console.WriteLine($"Clicked: {combination}");

            Meters.ClickInstrument.Add(1,
                new KeyValuePair<string, object?>("key", combination));
            Meters.DurationInstrumentation.Record((long)elapsed.TotalMilliseconds);
        }

        return Task.CompletedTask;

        static string GetKeyLabel(ConsoleKeyInfo keyInfo) 
            => keyInfo.Modifiers != ConsoleModifiers.None 
                ? $"{keyInfo.Modifiers}-{keyInfo.KeyChar}" 
                : keyInfo.KeyChar.ToString();
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        _work.Cancel();

        return Task.CompletedTask;
    }
}

public static class Meters
{
    private static readonly Meter _applicationMeter = new("custom-console-meter", "1.0.0");

    public static readonly Counter<int> ClickInstrument 
        = _applicationMeter.CreateCounter<int>("click_c", "clicks_total", "count the number of clicks");

    public static readonly Histogram<long> DurationInstrumentation 
        = _applicationMeter.CreateHistogram<long>("click_t", "ms", "time to click after prompted");
}

required dependencies:

    <PackageReference Include="Microsoft.Extensions.Hosting" Version="8.0.0-preview.3.23174.8" />
    <PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.5.0-alpha.2" />
    <PackageReference Include="OpenTelemetry.Exporter.Prometheus.HttpListener" Version="1.5.0-alpha.1" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.5.0-alpha.2" />

Recommended topics

Hot tags