Does Spring Data Elasticsearch support Amazon Elasticsearch?

Question

From the research I have done it seems that the two do not work together because of HTTP only support for Amazon Elasticsearch.

Hoping someone can clarify if in fact it is not something that is possible with Spring Data Elasticsearch.

Danie

Answer 1

From different discussions:
- Spring data ES and searchly
- port for the transport protocol

and the AWS documentation on ES service limitations; go to the bottom, the last line says:

The service supports HTTP on port 80, but does not support TCP transport.

It's not yet possible to use Spring Data ES which uses the Java APIs, with Amazon service which only available through REST.

Gaea

It is possible, please refer #34010373 – Markup

Answer 2

Looks like Spring data elastic search from version 3.2.0 works with http rest client, so it is possible to connect to aws elastic instance through Rest API and port 443. Somehow they integrated spring-data-jest approach into spring data. I use RestHighLevelClient:

    @Bean
    public RestHighLevelClient client() {
        return new RestHighLevelClient(RestClient.builder(HttpHost.create(awsUrl)));
    }

awsUrl format is: https://some_aws_generated_address.us-east-n.es.amazonaws.com:443

NOTE: If you are using spring boot with default bom.xml, you need to upgrade spring boot to 2.2.1.RELEASE or newer

Platus

Answer 3

There is a nice project providing Jest implementation of Spring Data ES that works with AWS managed ES service.

Checkout https://github.com/VanRoy/spring-data-jest

Reparation

Answer 4

It is possible to use Spring Data Elasticsearch with Amazon Elasticsearch

As Excerpt from Spring-data elastic search doc

TransportClient is deprecated as of Elasticsearch 7 and will be removed in Elasticsearch 8. This is in favor of Java High Level REST Client. Spring Data Elasticsearch will support the TransportClient as long as it is available in the Elasticsearch.

The Java High Level REST Client now is the default client of Elasticsearch, it provides a straight forward replacement for the TransportClient as it accepts and returns the very same request/response objects and therefore depends on the Elasticsearch core project

Spring Data ElasticSearch has streamlined with the latest standards of ElasticSearch, hence from spring-data-elasticsearch:3.2.X it provides a flexible way to achieve a custom RestHighLevelClient.(link) Even though it is possible to use HTTP based elastic search API calls with or without authentication, it will not solve the problem associated with the AWS elastic-search API calls.

Because any HTTP requests to AWS services or APIGW backed services have to follow "Signature Version 4 Signing Process(SigV4)" which eventually adds authentication information to AWS requests sent by HTTP. For security, most requests to AWS must be signed with an access key, which consists of an accesskey ID and secret access key. Hence, we have to follow the standards when calling the AWS ElasticSearch service.

Let's get our hands dirty with code and delve deep into the implementation

Please follow the steps:

Step 1: Adding required dependencies

       <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
            <version>2.2.2.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-elasticsearch</artifactId>
            <version>1.11.346</version>
        </dependency>

Step 2: Adding AWS CredentialsProvider

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class AWSCredentialsConfiguration {

    @Value("${aws.es.accessKey}")
    private String esAccessKey = null;

    @Value("${aws.es.secretKey}")
    private String esSecretKey = null;

    @Bean
    public AWSStaticCredentialsProvider awsDynamoCredentialsProviderDevelopment() {
        return new AWSStaticCredentialsProvider(new BasicAWSCredentials(
                esAccessKey, esSecretKey));
    }
}

Or if your Application running on AWS instance and you don't want to use the property-driven/hardcoded AccessKey and SecretKey then you have to assign the IAM role to your Amazon ECS task for more.

import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class AWSCredentialsConfiguration {

    @Bean
    public AWSCredentialsProvider amazonAWSCredentialsProvider() {
        return new EC2ContainerCredentialsProviderWrapper();
    }

}

Step 3: Adding ElasticSearchRestClientConfiguration

• If you observe the below code we are providing a custom implementation of **RestHighLevelClient** to abstract method **AbstractElasticsearchConfiguration#elasticsearchClient()**. In this way, we are injecting the incorporated customRestHighLevelClient of "elasticsearchOperations", "elasticsearchTemplate" beans into the spring container.
• HttpRequestInterceptor is another important thing to notice. A Special thanks to AWSRequestSigningApacheInterceptor.java, a sample implementation provided by AWSLabs helps us to add the interceptor to the RestClient with the AWS4Signer mechanism.
• @EnableElasticsearchRepositories annotation help to enable the elasticsearch data repositories.

import com.amazonaws.auth.AWS4Signer;
import com.amazonaws.auth.AWSCredentialsProvider;
import org.apache.http.HttpHost;
import org.apache.http.HttpRequestInterceptor;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.config.AbstractElasticsearchConfiguration;
import org.springframework.data.elasticsearch.repository.config.EnableElasticsearchRepositories;


@Configuration
@EnableElasticsearchRepositories(basePackages = "com.demo.aws.elasticsearch.data.repository")
public class ElasticSearchRestClientConfiguration extends AbstractElasticsearchConfiguration {

    @Value("${aws.es.endpoint}")
    private String endpoint = null;

    @Value("${aws.es.region}")
    private String region = null;

    @Autowired
    private AWSCredentialsProvider credentialsProvider = null;

    @Override
    @Bean
    public RestHighLevelClient elasticsearchClient() {
        AWS4Signer signer = new AWS4Signer();
        String serviceName = "es";
        signer.setServiceName(serviceName);
        signer.setRegionName(region);
        HttpRequestInterceptor interceptor = new AWSRequestSigningApacheInterceptor(serviceName, signer, credentialsProvider);
        return new RestHighLevelClient(RestClient.builder(HttpHost.create(endpoint)).setHttpClientConfigCallback(e -> e.addInterceptorLast(interceptor)));
    }
}

Bravo Zulu! that's it. We have completed the configuration part. Now with this solution, you can leverage the spring-data elastic benefits along with Amazon elastic search service. The complete solution has been documented in Medium Post

If there are any issues related to permission regarding accessing indices (ex: refresh), you can add the permission using the answer

Markup

Does not work. I'm getting [HTTP/1.1 403 Forbidden] {"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}]. While connecting directly via RestHighLevelClient works – Kakemono

@shark, for the corresponding error, you have to check with your AWSCredentialsProvider. Please validate your credentials accessKey and secretId. I have been using this snippet for more than 2 microservices. I have not seen such 403 issue – Markup

I have DefaultAWSCredentialsProviderChain and I also checked if it takes valid key and it does. What is curious This code creates indexes correctly but has problem with storing entity – Kakemono

Should I do anything specific with permissions, roles etc ? My instance is public – Kakemono

DefaultAWSCredentialsProviderChain takes credentials in the following order. ` Environment Variables or Java System Properties or Web Identity Token credentials from the environment or container or Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI `. Index creation works fine means your API calls authentication works correctly, you have to check the role for your AWS service, whether you have access to perform write operation for the index. – Markup

I have restricted access using the index pattern similar to dev-*, nonprod-* and have assigned role to the ECS instance which using ES. Similarly, you have to check the same with your credentials, Check using Kibana devtools with your credentials, if you are able to do write operations to index then you are on track or else you have to adjust your role permission – Markup

@Vinod, it should work with spring-boot-starter-data-elasticsearch:2.2.2.RELEASE aws-java-sdk-elasticsearch:1.11.346, I have production running applications with the above versions – Markup

@PrasanthRajendran I had a similar issue now this can occur due to multiple reasons: To validate this please insert any records from Spring boot endpoint to AWS Elasticsearch do check are you able to see the record in the Kibana view or not and if you see the record being inserted and still getting the error 403 that is because of AWSRequestSigningApacheInterceptor.java file issue; please do get the file from Github (awslabs). Ideally in this case search by id and find all should work. The second case can be a version issue check Elasticsearch version and spring boot parent version. – Pancratium