AWS XRay service map components are disconnected
Asked Answered
B

0

6

I'm using open telemetry to export trace information of the following application:

  1. A nodejs kafka producer sends messages to input-topic. It uses kafkajs instrumented with opentelemetry-instrumentation-kafkajs library. I'm using the example from AWS OTEL for NodeJS example. Here is my tracer.js:
module.exports = () => {
  diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.ERROR);

  // create a provider for activating and tracking with AWS IdGenerator
  const attributes = {
      'service.name': 'nodejs-producer',
      'service.namespace': 'axel'
  }

  let resource = new Resource(attributes)

  const tracerConfig = {
    idGenerator: new AWSXRayIdGenerator(),
    plugins: {
        kafkajs: { enabled: false, path: 'opentelemetry-plugin-kafkajs' }
    },
    resource: resource
  };
  const tracerProvider = new NodeTracerProvider(tracerConfig);

  // add OTLP exporter
  const otlpExporter = new CollectorTraceExporter({
    url: (process.env.OTEL_EXPORTER_OTLP_ENDPOINT) ? process.env.OTEL_EXPORTER_OTLP_ENDPOINT : "localhost:55680"
  });
  tracerProvider.addSpanProcessor(new SimpleSpanProcessor(otlpExporter));
  tracerProvider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter()));

  // Register the tracer with X-Ray propagator
  tracerProvider.register({
    propagator: new AWSXRayPropagator()
  });

  registerInstrumentations({
    tracerProvider,
    instrumentations: [new KafkaJsInstrumentation({})],
  });

  // Return a tracer instance
  return trace.getTracer("awsxray-tests");
}

  1. A Java application that reads from input-topic and produces to final-topic. Also instrumented with AWS OTEL java agent. Java app is launched like below:
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_PROPAGATORS=xray
export OTEL_RESOURCE_ATTRIBUTES="service.name=otlp-consumer-producer,service.namespace=axel"
export OTEL_METRICS_EXPORTER=none

java -javaagent:"${PWD}/aws-opentelemetry-agent.jar" -jar "${PWD}/target/otlp-consumer-producer.jar"
  1. I'm using otel/opentelemetry-collector-contrib that has AWS XRay exporter:
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  awsxray:
    region: 'eu-central-1'
    max_retries: 10

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [awsxray]

I can see from log messages and also from XRay console that traces are being published (with correct parent trace ids). NodeJS log message:

{
  traceId: '60d0c7d4cfc2d86b2df8624cb4bccead',
  parentId: undefined,
  name: 'input-topic',
  id: '3e289f00c4499ae8',
  kind: 3,
  timestamp: 1624295380734468,
  duration: 3787,
  attributes: {
    'messaging.system': 'kafka',
    'messaging.destination': 'input-topic',
    'messaging.destination_kind': 'topic'
  },
  status: { code: 0 },
  events: []
}

and Java consumer with headers:

Headers([x-amzn-trace-id:Root=1-60d0c7d4-cfc2d86b2df8624cb4bccead;Parent=3e289f00c4499ae8;Sampled=1])

As you see parent and root ids match each other. However the service map is constructed in a disconnected way: service map

What other configuration I'm missing here to compile a correct service map?

Bigamy answered 21/6, 2021 at 17:8 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.