How to connect to Elasticsearch with python using SSL?

Question

I am trying to connect to an Elasticsearch node from Python with SSL.

I'm using the basic code for that:

from elasticsearch import Elasticsearch
from ssl import create_default_context

context = create_default_context(cafile="path/to/cafile.pem")
es = Elasticsearch("https://elasticsearch.url:port", ssl_context=context, http_auth=('elastic','yourpassword'))

From: https://github.com/elastic/elasticsearch-py

I need to supply cafile.pem, and http_auth parameters. On the server where my Python is running, SSL connection is already set up, so I can do basic queries to Elasticsearch. It was set up using keys in the ~/.ssh directory: id_rsa, id_rsa.pub.

So, now I am wondering whether I should supply id_rsa.pub key in place of path/to/cafile.pem, and if yes, then I would need to change permissions of ~/.ssh folder which seems like not a good idea from security perspective.

Then, I am not sure that .pub is the same as .pem, do I need to convert it first? Then, should http_auth just be omitted since I do not use any password when I do simple queries from the terminal?

How should I go about this issue of setting up access in Python to ES with SSL according to best practices?

I tried both .pub and generated from it pem: https://serverfault.com/questions/706336/how-to-get-a-pem-file-from-ssh-key-pair

But both failed to create_default_context with an unknown error in context.load_verify_locations(cafile, capath, cadata).

Immobility

You are missing the things, the id_rsa, id_rsa.pub are your ssh keys which your user uses to connect to your server without the need to provide a password, to make a secure connection with elasticsearch using SSL you need to configure it in your cluster and use the certificate generated during the configuration process. – Bonnett

You can configure SSL in your cluster using search-guard.com – Piroshki

Answer 1

The answer for my particular case turned out to be very simple. I found it here:

https://elasticsearch-py.readthedocs.io/en/master/

es = Elasticsearch(['https://user:secret@localhost:443'])

Just specified https url inside and it worked out right away.

Immobility

Thanks, it's simple but helped when switching from HTTP to HTTPS in Elasticsearch. – Psychosocial

I had to also verify_certs=False , see https://mcmap.net/q/1119523/-elastic_transport-tlserror-tls-error-caused-by-tlserror-tls-error-caused-by-sslerror-ssl-wrong_version_number-wrong-version-number-_ssl-c-852 – Galegalea

Answer 2

Elasticsearch Docker image & Python2.7. Have Copied ssl certificate file to root of the project. Made sure it's readable, ownership and group ownership will allow read access. Put pass and login to constants.

es = Elasticsearch(
    hosts=[
            "https://localhost:9200"
    ],
    http_auth=(USR_LOGIN, USR_PASS),
    use_ssl=True,
    verify_certs=True,
    ca_certs="./http_ca.crt",
)

Cioban

use_ssl is not a valid parameter. – Province

Answer 3

We are using Elasticsearch 8.12.0 deployed by docker-compose with self-signed certs

from elasticsearch import Elasticsearch

es = Elasticsearch(
    "https://hostname:9200/",
    ca_certs="path/to/ca.crt",
    basic_auth=("username", "password")
)
#print(es.info())
indices = es.indices.get(index="_all")
for index in indices:
    print(index)

Breunig

Answer 4

For self-signed certificates, using:

• The official Python client for Elasticsearch (https://pypi.org/project/elasticsearch/) 8.7.0
• The shared elastic-transport library (https://pypi.org/project/elastic-transport/) 8.4.0

from elastic_transport import NodeConfig
from elasticsearch import AsyncElasticsearch

client = AsyncElasticsearch(
    hosts=[
        NodeConfig(
            scheme= "https",
            host="<host URL>",
            port=443,
            verify_certs=False,
            ca_certs=None,
            ssl_show_warn=False,
        )
    ],
    http_auth=("username", "password"),
    verify_certs=False,
    ca_certs="/path/to/cafile.pem",  # PEM format
    client_cert="/path/to/tls.cert"  # PEM format
    client_key="/path/to/tls.key"    # PEM format
)
client.info()

Explanation:

• The CA file and the TLS cert/key pair were generated based on the guide for Set up basic security for the Elastic Stack plus secured HTTPS traffic: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html#security-basic-setup-https. As mentioned in the comment you would need to properly generate your TLS certs/keys, then pass that to the Elasticsearch client library
• Since this answer is for self-signed certificates, you would need to also pass your CA file (in PEM format).
• The verify_certs=False disables the underlying Python SSL modules from verifying the self-signed certs, but properly sends it upstream to the server. For non-self-signed certificates, you should try enabling verify_certs=True.
• The code above is using AsyncElasticsearch, but if you need the sync Elasticsearch version, it should be directly compatible as all the parameters are the same. See: https://elasticsearch-py.readthedocs.io/en/v8.8.1/async.html#getting-started-with-async)

So, now I am wondering whether I should supply id_rsa.pub key in place of path/to/cafile.pem, and if yes, then I would need to change permissions of ~/.ssh folder which seems like not a good idea from security perspective.

These SSH keys are likely not related to Elasticsearch, but for allowing you to connect (via SSH) to your nodes running your Elasticsearch instance.

Kiddush