AWS XRAY SDK issue: Failed to begin subsegment named 'Amazon S3': segment cannot be found

Question

We are adding XRAY to our Spring Boot application and I'm unable to resolve the following error:

Failed to begin subsegment named 'Amazon S3': segment cannot be found.

Here's the relevant parts of our code:

pom.xml:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-xray-recorder-sdk-core</artifactId>
    <version>1.2.1</version>
</dependency>
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-xray-recorder-sdk-aws-sdk</artifactId>
    <version>1.2.1</version>
</dependency>
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-xray-recorder-sdk-aws-sdk-instrumentor</artifactId>
    <version>1.2.1</version>
</dependency>
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-xray-recorder-sdk-apache-http</artifactId>
    <version>1.2.1</version>
</dependency>
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-xray-recorder-sdk-spring</artifactId>
    <version>1.3.1</version>
</dependency>
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-xray-recorder-sdk-sql-postgres</artifactId>
    <version>1.2.1</version>
</dependency> 

SpringApplication.java

@Bean
public Filter TracingFilter() {
    return new AWSXRayServletFilter("myService");
}

Class making call to S3

@PostConstruct
public void runOnStartup(){
    String fileName = "myFileName";
    String bucketName = "myBucketName";

    amazonS3Client = AmazonS3ClientBuilder.standard()
        .withCredentials(new ProfileCredentialsProvider("MyCredentials"))
        .withClientConfiguration(getClientConfiguration())
        .withRegion(region)
        .build();

    Segment segment = AWSXRay.beginSegment("do-startup-operation");
    S3Object s3Object = amazonS3Client.getObject(bucketName, fileName);
    AWSXRay.endSegment();

    //Do stuff with S3Object
}

What I've tried so far:

1) I've tried with and without the sdk-aws-sdk-instrumentor import, adding in the TracingHandler configuration when doing so as outlined in this question and this documentation.

.withRequestHandlers(new TracingHandler(AWSXRay.getGlobalRecorder()))

2) I've found this thread which seems to suggest using AWSXRAY.createSegment(), but I'm not sure what would go in the lambda or if it's relevant to my scenario

Other documentation/code I've read and found relevant: https://docs.aws.amazon.com/xray/latest/devguide/scorekeep-startup.html

https://docs.aws.amazon.com/xray/latest/devguide/scorekeep-sdkclients.html

https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java-multithreading.html

https://docs.aws.amazon.com/xray/latest/devguide/scorekeep-workerthreads.html

https://github.com/aws/aws-xray-sdk-java/blob/master/aws-xray-recorder-sdk-core/src/main/java/com/amazonaws/xray/contexts/ThreadLocalSegmentContext.java#L23

P.S. I simplified my code and left out some error handling to make it easier for those who view this question to read

Mazy

Answer 1

This is now working after replacing

AWSXRay.getGlobalRecorder()

with

Entity mySegment = AWSXRay.beginSegment("do-startup-operation");
AWSXRay.getGlobalRecorder().setTraceEntity(mySegment);
S3Object s3Object = amazonS3Client.getObject(bucketName, fileName);
AWSXRay.endSegment();

Mazy

Answer 2

The X-Ray servlet filter will open a segment when it receives a request and close it before returning a response. The segment it creates represents a full request/response lifecycle. Anything captured as part of serving this request (in this case might be some AWS service calls) is called subsegment. As you can see the subsegments needs the tracing context (which segment is this for).

The issue is that on server startup the instrumentor tries to capture the S3 call but it cannot find the context as there is no request coming in yet. One option is to set an environment variable AWS_XRAY_CONTEXT_MISSING to LOG_ERROR so there is no exception but only a log entry. See more details at https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java-configuration.html#xray-sdk-java-configuration-envvars.

For running on lambda the lambda container will create a segment for each invocation. It sets the tracing context as environment variable. So as long as the code being captured is within the handler class, the context should always be present.

Foolhardy