How to create Context using traceId in Open Telemetry

Question

I try to get all spans created in the following chain associated to the same trace context/traceId by context propagation:

service1 -> aws sqs queue -> service2

Auto. context propagation is not working with aws sqs and aws sdk v2 atm (https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/3684), even though the AwsTraceHeader is actually set in the sqs message, I have to take care for it explicitly by

• service1: Writing traceId in sqs message user attribute traceId=Span.current().getSpanContext().getTraceId()
• service2: Reading traceId from sqs message user attribute traceId and overwriting current span.traceId / essentially creating Context of service1

However, it is now unclear how to actually overwrite span.traceId in the span that service2 created which is confusing because for example with Golang it seems to be straightforward: How to create opentelemetry span from a string traceid

I see only getters e.g. Span.current().getSpanContext().getTraceId() but no setters or builder methods.

Update:

Even by creating a new span and making it current (not sure if this goes in the right direction) the tracer.spanBuilder does no offer setters for traceId AFAIU)

@Inject
io.opentelemetry.api.trace.Tracer tracer;

Span consumeMessageSpan = tracer.spanBuilder("consumeMessage").startSpan();

consumeMessage.makeCurrent();

Update 2

This snippet from otel official docs looks promising

To link spans from remote processes, it is sufficient to set the Remote Context as parent.

Span childRemoteParent = tracer.spanBuilder("Child").setParent(remoteContext).startSpan(); 

However, also no examples or ideas how to create remoteContext and setting traceId to the one extracted from the sqs message

Any hints how to do that?

Megaspore

Answer 1

I've done the following for a child JVM (that is running using the OTel auto-instrumentation agent):

    public static void main(String[] args) {
        Span span = createSpanLinkedToParent();
        try (Scope scope = span.makeCurrent()) {
            // do stuff
        } finally {
            span.end();
        }
    }

    private static Span createSpanLinkedToParent() {
        // Fetch the trace and span IDs from wherever you've stored them
        String traceIdHex = System.getProperty("otel.traceid");
        String spanIdHex = System.getProperty("otel.spanid");

        SpanContext remoteContext = SpanContext.createFromRemoteParent(
                traceIdHex,
                spanIdHex,
                TraceFlags.getSampled(),
                TraceState.getDefault());

        return GlobalOpenTelemetry.getTracer("")
                .spanBuilder("root span name")
                .setParent(Context.current().with(Span.wrap(remoteContext)))
                .startSpan();
    }

The next improvement I plan to make it to serialise the flags and state, perhaps using code here in Context Propagation https://opentelemetry.io/docs/instrumentation/java/manual/#context-propagation but the above works for now.

Trinatte

Hi Graham, I tested it and saw that the link from remote to current span is created. However, after the code in the main method above I call methods which are annotated with @WithSpan("mySpan") which are not associated to the spanFromParent but have new traceID which I find very confusing. Any ideas why there is no link from spanFromParent to mySpan? – Megaspore

1. Are those in the try finally block where //do stuff is? 2. If so, are those methods being called inside a new thread? 3. Are the threadid and span id being picked up correctly? i.e. print/log them and confirm – Trinatte

I've just noticed that this only works with v1.15.0, it breaks after this version of the Java OTel agent. – Trinatte

Turns out I was just missing the point that all spans which should be associated with the restored trace context must really be created inside the spanFromParent (at ` // do stuff`). I just tested it with latest otel agent 1.20.1 and as far as I see it works, see github.com/syr/sqs-otel-quartz/tree/manual-context-propagation Thanks btw. for the helpful sample code. – Megaspore

Answer 2

The OpenTelemetry java instrumentation guide has sample code for some of this

        Context extractedContext = opentelemetry.getPropagators().getTextMapPropagator()
                .extract(Context.current(), headers, getter);
        ⋮
        openTelemetry.getPropagators().getTextMapPropagator().inject(Context.current(), transportLayer, setter);

Basically you pass all the headers from the request to the textMapPropagator and it knows how to pull out the ones required for setting up spans correctly.

Kauai