opensearch authentication with opensearch-py on aws lambda

Question

I am trying to connect to AWS OpenSearch domain from AWS Lambda using the opensearch python client (development purposes, non production).

I was trying the following:

from opensearchpy import OpenSearch
import boto3
from requests_aws4auth import AWS4Auth
import os
import config
my_region = os.environ['AWS_REGION']
service = 'es' # still es???
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, my_region, service, session_token=credentials.token)
openSearch_endpoint = config.openSearch_endpoint

# sth wrong here:
openSearch_client = OpenSearch(hosts = [openSearch_endpoint], auth = awsauth)

as per the following blogs:

• https://aws.amazon.com/blogs/database/indexing-metadata-in-amazon-elasticsearch-service-using-aws-lambda-and-python/
• https://docs.aws.amazon.com/opensearch-service/latest/developerguide/search-example.html

but it does not work (it does not want to authenticate, "errorMessage":"AuthorizationException(403, '')" . However if I don't use the python client but simply go through requests instead:

import requests
host = config.openSearch_endpoint
url = host + '/' +'_cat/indices?v'
# this one works:
r = requests.get(url, auth=awsauth)

, my lambda function does communicate with the OpenSearch domain.

I consulted the OpenSearch() documentation but it is not clear to me how its parameters map to boto3 session credentials, and/or to AWS4Auth. So what should this line

openSearch_client = OpenSearch(hosts = [openSearch_endpoint], auth = awsauth)

be?

Benzaldehyde

Answer 1

actually managed to find the solution a couple of hours later:

from opensearchpy import OpenSearch, RequestsHttpConnection
my_region = os.environ['AWS_REGION']
service = 'es' # still es?
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, my_region, service, session_token=credentials.token)
host = config.openSearch_endpoint

openSearch_client = OpenSearch(
        hosts=[openSearch_endpoint],
        http_auth = awsauth,
        use_ssl = True,
        verify_certs = True,
        ssl_assert_hostname = False,
        ssl_show_warn = False,
        connection_class=RequestsHttpConnection
        )

Benzaldehyde

This code is still wrong. You are not specifiying the port: 443 – Battled

Could you provide a link to where you found your answer? Thx! – Digestible

Answer 2

OpenSearch Service requires port 443 for incoming requests therefore you need to add a new Inbound Rule under Security Group attached to your OpenSearch Service domain.

Try the rule:

• Type: HTTPS
• Protocol: TCP
• Port range: 443
• Source: 0.0.0.0/0 (Anywhere-IPv4)

Additionally, you should have a Resource-based policy for your Lambda function to perform requests to your OpenSearch Service domain.

Comply

At my case the security group with source 0.0.0.0/0 was the final step to complete the access from lambda to Opensearch, however is not a good practice. Instead you can check in Network Interfaces looking for the eni-xxxxx created by lambda, then copy the IPs and add to the OpenSearch security group to restrict the access just for the lambda – Semang