Connecting ElasticSearch to Django using "django-elasticsearch-dsl" results in ConnectionError when attempting to create/rebuild index

Question

I am trying to call a local ES instance running on docker. I used the following instructions to setup my ES instance: https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-dev-mode

I am able to play around with my instance on Kibana at http://0.0.0.0:5601/app/dev_tools#/console. Everything works upto here.

Now I am trying to define some sample documents using django models and index them through the library; I am following the instructions here: https://django-elasticsearch-dsl.readthedocs.io/en/latest/quickstart.html#install-and-configure

First, I add pip installed and added django_elasticsearch_dsl to INSTALLED_APPS

Next, I added in settings.py:

ELASTICSEARCH_DSL = {
    'default': {
        'hosts': 'localhost:9200'
    },
}

Then I create a sample model and document that looks like this:

# models.py

from django.db import models

class Car(models.Model):
    name = models.CharField(max_length=30)
    color = models.CharField(max_length=30)
    description = models.TextField()
    type = models.IntegerField(choices=[
        (1, "Sedan"),
        (2, "Truck"),
        (4, "SUV"),
    ])

# documents.py

from django_elasticsearch_dsl import Document
from django_elasticsearch_dsl.registries import registry
from .models import Car

@registry.register_document
class CarDocument(Document):
    class Index:
        # Name of the Elasticsearch index
        name = 'cars'
        # See Elasticsearch Indices API reference for available settings
        settings = {'number_of_shards': 1,
                    'number_of_replicas': 0}

    class Django:
        model = Car # The model associated with this Document

        # The fields of the model you want to be indexed in Elasticsearch
        fields = [
            'name',
            'color',
            'description',
            'type',
        ]

Finally, running python3 manage.py search_index --rebuild results in the following connection error:

    raise ConnectionError("N/A", str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))) caused by: ProtocolError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))

I suspect that there might be an issue with my ELASTICSEARCH_DSL setup as I have not specified any config for https but the documentation does not make this clear.

How do I resolve this issue?

Django version: 
Django==4.0.1
django-elasticsearch-dsl==7.2.2

Python version: Python 3.9.10

Thanks!

Lactose

Also asked this question on github with no response: github.com/django-es/django-elasticsearch-dsl/issues/397 – Lactose

Answer 1

I figured that it was an issue with my certificate.

I needed to add some additional config param to the ELASTICSEARCH_DSL variable. Adding this solves the issue:

from elasticsearch import RequestsHttpConnection

# Elasticsearch configuration in settings.py
ELASTICSEARCH_DSL = {
    'default': {
        'hosts': 'localhost:9200',
        'use_ssl': True,
        'http_auth': ('user', 'password'),
        'ca_certs': '/path/to/cert.crt'
        'connection_class': RequestsHttpConnection
    }
}

See this section of the elastic docs on verifying a certificate.

If you haven't setup a certificate to authenticate the connection with and just need to get something up and running quickly, you can pass 'verify_certs': False and set 'ca_certs' to None.

Lactose

Answer 2

If you're using docker, this reply (mentioned by @cs95) helped.

You set 'hosts': 'localhost:9200' but you have to be sure your port is exposed on localhost, also if your python service is running in a docker as well, it has to be in same network as you elasticsearch, and in this case, instead of localhost, you should have the service name, instead of localhost.

Presa

Answer 3

I'm using django-elasticsearch-dsl, but the accepted answer didn't work for me using docker.

This is what worked:

settings.py

ELASTICSEARCH_DSL = {
    "default": {
        "hosts": "https://localhost:9200",
        "http_auth": ("user (default is `elastic`)", 
                      "password (copy it from `Password for the user`)"),
        "ssl_assert_fingerprint": "fingerprint (copy it from `HTTP CA certificate SHA-256 fingerprint`)",
    }
}

No need to import RequestsHttpConnection from elasticsearch (which, by the way, throws error).

example:

ELASTICSEARCH_DSL = {
    "default": {
        "hosts": "https://localhost:9200",
        "http_auth": ("elastic", "=Al60CnKGKq+AsFLG99A"),
        "ssl_assert_fingerprint": "15c88cc1244225c987a243172d63d25d8737hb3bec3b512bf1216lq34e595775",
    }
}

Richierichlad

Answer 4

This worked for me. I also had to change my elastic user password using https://www.elastic.co/guide/en/elasticsearch/reference/current/reset-password.html

ELASTICSEARCH_DSL = {
    "default": {
        "hosts": "https://localhost:9200",
        "http_auth": ("elastic", "f152vDw9IGReO9SwN5Pq"),
        'verify_certs': False,
        'ca_certs' : None,
    }
}

Flop