How to trace a request through an SQS queue with AWS X-Ray
Asked Answered
C

4

19

I'm trying to get a toy example up and running with an AWS Lambda function, f, that's triggered by a message on one SQS queue, sqs, publishes to another queue sqs', and then a worker, f' reads from sqs' and processes the message where the entire "request" is traced with X-Ray.

sqs -> f -> sqs' -> f'

Currently, I have the queues in place and the functions writing and receiving from the queue. I also have X-Ray tracing the request from the the first function f to the sqs queue.

My current challenge is: how do I propagate the trace to the final worker so I can see the entire process in x-ray.


Here are my current functions:

public class Hello implements RequestHandler<SQSEvent, Void> {
    String OUTPUT_QUEUE_URL = "...";

    private AmazonSQS sqs = AmazonSQSClientBuilder.standard()
        .withRequestHandlers(new TracingHandler(AWSXRay.getGlobalRecorder()))
        .build();

    public Void handleRequest(SQSEvent event, Context context)
    {
        for(SQSMessage msg : event.getRecords()){
            System.out.println(new String(msg.getBody()));
        }

        SendMessageRequest send_msg_request = new SendMessageRequest()
            .withQueueUrl(OUTPUT_QUEUE_URL)
            .withMessageBody("hello world")
            .withDelaySeconds(5);
        sqs.sendMessage(send_msg_request);
        return null;
    }
}

public class World implements RequestHandler<SQSEvent, Void>{
    public Void handleRequest(SQSEvent event, Context context)
    {
        for(SQSMessage msg : event.getRecords()){
            System.out.println(new String(msg.getBody()));
        }
        return null;
    }
}
Cowry answered 21/8, 2018 at 18:29 Comment(1)
have exactly the same problem with a node.js lambda. SQS seems to be treated the same way as API Gateway (shows up as "User" in the X-Ray view)Calli
B
4

As of today, AWS X-Ray now has native support for Amazon SQS: https://aws.amazon.com/about-aws/whats-new/2019/08/aws-x-ray-now-supports-amazon-sqs/

Balneology answered 29/8, 2019 at 0:7 Comment(1)
@justin-bell did this solve your problem? I feel like I have the exact same setup but Lambda is always starting its own trace.Mcgill
Z
6

Even though SQS supports X-Ray tracing now, it doesn't propagate the trace to a lambda function. That's why with SQS->Lambda setup, Lambda always starts a new trace. There is an issue for it in the nodejs xray SDK https://github.com/aws/aws-xray-sdk-node/issues/208


On Nov 21 2022, that issue was resolved, referencing a blog post titled AWS X-Ray adds trace linking for event-driven applications built on Amazon SQS and AWS Lambda.

Zimbabwe answered 30/6, 2020 at 5:44 Comment(0)
B
4

As of today, AWS X-Ray now has native support for Amazon SQS: https://aws.amazon.com/about-aws/whats-new/2019/08/aws-x-ray-now-supports-amazon-sqs/

Balneology answered 29/8, 2019 at 0:7 Comment(1)
@justin-bell did this solve your problem? I feel like I have the exact same setup but Lambda is always starting its own trace.Mcgill
C
2

Although as Matthew Werber rightly says this is not officially supported at the moment, you can work around this by creating a new AWS-Xray segment inside the Lambda Function and using the incoming TraceID from the SQS message. This will result in two Segments for your lambda invocation. One which Lambda itself creates, and one which you create to extend the existing trace. Whether that's acceptable or not for your use case is something you'll have to decide for yourself!

If you're working with Python you can do it with aws-xray-lambda-segment-shim.
If you're working with NodeJS you can follow this guide on dev.to.
If you're working with .NET there are some examples on this GitHub issue.

Candlepin answered 28/8, 2021 at 10:36 Comment(1)
Hey Sam, thanks for pointing to this blog post. Here is a repo I created based on the suggested fix: github.com/renatoargh/lambda-sqs-lambda-xray-poc It uses serverless so just running "npx sls deploy" is enough to deploy the entire POC setup to AWSGalatia
E
0

I'm planing to build the same prototype and I noticed the following announcement (date 2022-11-21): https://aws.amazon.com/about-aws/whats-new/2022/11/aws-x-ray-trace-linking-event-driven-applications-amazon-sqs-lambda/

It seems the propagation is supported now.

Elanorelapid answered 15/6, 2023 at 7:51 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.