AmazonsS3Client throws UnknownHostException if attempting to connect to a local service (MinIO)

Question

In Short: Using AmazonS3Client to connect to a local instance of MinIO results in a UnknownHostException thrown because the url is resolved to http://{bucket_name}.localhost:port.

---

Detailed description of the problem:

I'm creating an integration test for a Java service that uses AmazonS3Client lib to retrieve content from S3. I'm using MinIO inside a test container to perform the role of Amazon S3, as follows:

@Container
static final GenericContainer<?> minioContainer = new GenericContainer<>("minio/minio:latest")
    .withCommand("server /data")
    .withEnv(
        Map.of(
            "MINIO_ACCESS_KEY", AWS_ACCESS_KEY.getValue(),
            "MINIO_SECRET_KEY", AWS_SECRET_KEY.getValue()
        )
    )
    .withExposedPorts(MINIO_PORT)
    .waitingFor(new HttpWaitStrategy()
        .forPath("/minio/health/ready")
        .forPort(MINIO_PORT)
        .withStartupTimeout(Duration.ofSeconds(10)));

and then I export its url dynamically (because test containers are deployed at a random port) using something like this:

String.format("http://%s:%s", minioContainer.getHost(), minioContainer.getFirstMappedPort())

which in turn results in a url like this:

http://localhost:54123

The problem I encountered during the runtime of my test lies within the actual implementation of AmazonS3Client.getObject(String,String) - when creating the request it performs the following validation (class S3RequestEndpointResolver, method resolveRequestEndpoint):

...    
if (shouldUseVirtualAddressing(endpoint)) {
        request.setEndpoint(convertToVirtualHostEndpoint(endpoint, bucketName));
        request.setResourcePath(SdkHttpUtils.urlEncode(getHostStyleResourcePath(), true));
    } else {
        request.setEndpoint(endpoint);
        request.setResourcePath(SdkHttpUtils.urlEncode(getPathStyleResourcePath(), true));
    }
}

private boolean shouldUseVirtualAddressing(final URI endpoint) {
    return !isPathStyleAccess && BucketNameUtils.isDNSBucketName(bucketName)
            && !isValidIpV4Address(endpoint.getHost());
}

This in turn returns true for the url http://localhost:54123 and as a result this method

private static URI convertToVirtualHostEndpoint(URI endpoint, String bucketName) {
    try {
        return new URI(String.format("%s://%s.%s", endpoint.getScheme(), bucketName, endpoint.getAuthority()));
    } catch (URISyntaxException e) {
        throw new IllegalArgumentException("Invalid bucket name: " + bucketName, e);
    }
}

concatenates the name of the bucket to the host resulting in: http://mybucket.localhost:54123 and this ultimately results in a UnknownHostException to be thrown. I can work around this by setting the host to 0.0.0.0 instead of localhost, but this is hardly a solution.

Therefore I was wondering if i) this a bug/limitation in AmazonS3Client?; ii) I'm the one who is missing something, e.g. poor configuration ?

Thank you for your time

Gensler

Answer 1

I was able to find a solution. Looking at the method used by the resolver:

private boolean shouldUseVirtualAddressing(final URI endpoint) {
    return !isPathStyleAccess && BucketNameUtils.isDNSBucketName(bucketName)
            && !isValidIpV4Address(endpoint.getHost());
}

which was returning true and leading the flow to the wrong concatenation I found that we can set the first variable isPathStyleAccess when building the client. In my case, I created a bean in my test configuration to override the main one:

@Bean
@Primary
public AmazonS3 amazonS3() {

    return AmazonS3Client.builder()
        .withPathStyleAccessEnabled(true) //HERE
        .withCredentials(new AWSStaticCredentialsProvider(
            new BasicAWSCredentials(AWS_ACCESS_KEY.getValue(), AWS_SECRET_KEY.getValue())
        ))
        .withEndpointConfiguration(
            new AwsClientBuilder.EndpointConfiguration(s3Endpoint, region)
        )
        .build();
    
}

Gensler

Answer 2

For the SDK V2, the solution was pretty similar:

S3AsyncClient s3 = S3AsyncClient.builder()
                .forcePathStyle(true) // adding this one
                .endpointOverride(new URI(s3Endpoint))
                .credentialsProvider(() -> AwsBasicCredentials.create(s3Properties.getAccessKey(), s3Properties.getSecretKey()))
                .build()

Humanitarian