AWS X-Ray GoLang Lambda to lambda tracing and displayed in the service map

Question

I have the API Gateway that calls Lamdba function 1 and that invokes lambda function 2 in Go. I want to see these 2 functions joined in the service map.

The only way i have been able to do this so far is to create a custom segment eg called "parent" and the create a subsegment from this context eg called "child". Then using client.InvokeWithContext to invoke the function 2 passing the "child" segment context.

sess := session.Must(session.NewSession())
client := lambda.New(sess, &aws.Config{Region: aws.String(region)})

xray.Configure(xray.Config{LogLevel: "trace"})
xray.AWS(client.Client)

ctx, seg := xray.BeginSegment(context.Background(), "Parent")
ctx, subseg := xray.BeginSubsegment(ctx, "Child")
result, _ := client.InvokeWithContext(ctx, 
    lambda.InvokeInput{FunctionName: aws.String(functionName), Payload: nil})
subseg.Close(nil)   
seg.Close(nil)

Problem is that this creates trace parent -> child in sevice map but also has function 1 too.

What is the best way to join these 2 functions on the service map please ? Note. I have more than 2 that i want to see linked up on the service map to show me my whole flow through lambdas.

Please help.

Thanks Rick

Dismast

To view the whole flow of multiple lambdas, I moved away from XRay and used OpenTracing -> Zipkin – Linguistician

Answer 1

This Go and Lambda boilerplate app demonstrates Lambda to Lambda traces on the X-Ray service map:

https://github.com/nzoschke/gofaas/blob/master/worker.go

Resulting X-Ray Service Map

WorkCreateFunction (function 1) is an API Gateway handler function. It calls WorkerFunction (function 2) via a Lambda.InvokeWithContext call.

The trick is to instrument the Lambda API client with xray before making Lambda API calls:

// Lambda is an xray instrumented Lambda client
func Lambda() *lambda.Lambda {
    c := lambda.New(sess)
    xray.AWS(c.Client)
    return c
}

out, err := Lambda().InvokeWithContext(ctx, &lambda.InvokeInput{
    FunctionName:   aws.String(os.Getenv("WORKER_FUNCTION_NAME")),
    InvocationType: aws.String("Event"), // async
})
if err != nil {
    return responseEmpty, errors.WithStack(err)
}

The aws-xray-sdk-go copies the X-Amzn-Trace-Id header from function 1 into the Lambda API request for function 2:

https://github.com/aws/aws-xray-sdk-go/blob/master/xray/aws.go#L56

If this is not working, try updating to the latest aws-xray-sdk-go.

Microlith

Answer 2

You don't need to add a subsegment for the "child" call unless you want to add annotation/metadata.

The API gateway adds a trace ID called X-Amzn-Trace-Id to the header of incoming requests, which X-ray picks up. If you forward that trace ID in your call from lambda 1 to lambda 2, then X-ray will visually represent the calls with an arrow from lambda 1 to lambda 2 in the overview and include the trace details of lambda 2 when viewing the trace details of lambda 1.

As long as you forward the top trace ID through the call chain, X-ray will correctly visualize the call chain from service to service with nodes and arrows.

From https://aws.amazon.com/xray/faqs/:

Q: What is a trace?

An X-Ray trace is a set of data points that share the same trace ID. For example, when a client makes a request to your application, it is assigned a unique trace ID. As the request makes its way through services in your application, the services relay information regarding the request back to X-Ray using this unique trace ID. The piece of information relayed by each service in your application to X-Ray is a segment, and a trace is a collection of segments.

https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-tracingheader

https://docs.aws.amazon.com/xray/latest/devguide/xray-services-apigateway.html

Pharmacognosy