How to install ElasticSeach plugins using docker compose

Question

I have a docker-compose.yml file with an elastic search image:

elasticsearch:
  image: elasticsearch
  ports:
    - "9200:9200"
  container_name: custom_elasticsearch_1

If I want to install additional plugins like the HQ interface or the attachment-mapper I have to do a manual installation with the following commands:

$ docker exec custom_elasticsearch_1 plugin install royrusso/elasticsearch-HQ
$ docker exec custom_elasticsearch_1 plugin install mapper-attachments

Is there a way to install them automatically when I run the docker-compose up command?

Tartar

Answer 1

Here is a blog post by Elastic pertaining to exactly that! You need to use a Dockerfile which executes commands to extend an image. Your Dockerfile will look something like this:

FROM custom_elasticsearch_1

RUN elasticsearch-plugin install royrusso/elasticsearch-HQ

Cranium

Mmm so there is no way to achieve that without adding another Dockerfile? I would like to do it inside the docker-compose – Tartar

Docker compose uses the Dockerfile if you add the build command to docker-compose.yml. Here is an example of use of a Dockerfile with Compose. But, here is an example of someone installing plugins without a Dockerfile. – Cranium

I see, apparently an additional file is always required. – Tartar

Thanks, @fylie. Your answer probably saved me hours today. – Aureaaureate

You're welcome @missingfaktor! Glad my answer helped :) – Cranium

Thanks @fylie, I ended using the second link in your comment – Penniepenniless

this does not work with latest Docker version as of 2021. The RUN plugin install <plugin_name> throws error saying plugin in not found! The following works: RUN elasticsearch-plugin install analysis-icu Please update the answer as this has been accepted as answer by the question owner. – Octahedrite

@Asrar, done. (The command has nothing to do with Docker, it's changed by Elasticsearch.) – Ehf

Answer 2

Inspired by @NickPridorozhko's answer, but updated and tested with elasticsearch^7.0.0 (with docker stack / swarm), example with analysis-icu:

elasticsearch:
  image: docker.elastic.co/elasticsearch/elasticsearch:7.3.0
  user: elasticsearch
  command: >
    /bin/sh -c "./bin/elasticsearch-plugin list | grep -q analysis-icu 
    || ./bin/elasticsearch-plugin install analysis-icu; 
    /usr/local/bin/docker-entrypoint.sh"
  ...

The main difference are the updated commands for ^7.0.0, and the use of the docker entrypoint instead of ./bin/elasticsearch (in a stack's context, you'd get an error related to a limit of spawnable processes).

Reassure

this gives me an error "WARNING: plugin requires additional permissions " when i use it to install repoistory-s3 – Ender

@Ender That doesn't seem to be related. See this github issue and this other answer. – Reassure

For Opensearch, this works as well, with /usr/share/opensearch/bin/opensearch-plugin and /usr/share/opensearch/opensearch-docker-entrypoint.sh – Layfield

Answer 3

The ingest-attachment plugin requires additional permissions and prompts the user during the installation. I used the yes command :

elasticsearch:
  image: elasticsearch:6.8.12
  command: >
    /bin/sh -c "./bin/elasticsearch-plugin list | grep -q ingest-attachment 
    || yes | ./bin/elasticsearch-plugin install --silent ingest-attachment; 
    /usr/local/bin/docker-entrypoint.sh eswrapper"

Uranyl

tried on docker compose can confirm this works – Reservation

This should be the accepted answer. Worked great for me on ES 7.x and does not require messing around with a custom Dockerfile. – Vicarage

Should we create a new docker image of Elasticsearch with this we are trying to add the azure repository plugin – Beholden

Thanks, confirm this works fine in 7.17.21 – Lilia

Answer 4

This works for me. Install plugin before and then continue with starting the elasticsearch.

elasticsearch:
  image: elasticsearch
  command:
    - sh
    - -c
    - "plugin list | grep -q plugin_name || plugin install plugin_name;
       /docker-entrypoint.sh elasticsearch"

Equivoque

Answer 5

If you're using the ELK stack from sebp/elk

You need to setup your Dockerfile like

FROM sebp/elk

ENV ES_HOME /opt/elasticsearch
WORKDIR ${ES_HOME}

RUN yes | CONF_DIR=/etc/elasticsearch gosu elasticsearch bin/elasticsearch-plugin \
    install -b mapper-attachments

As seen on https://elk-docker.readthedocs.io/#installing-elasticsearch-plugins.

It should also work just for Elastic Search only as well.

Wooden

Answer 6

An example with Elasticsearch v6.8.15. For simplicity we will use a docker-compose.yml and a Dockerfile.

The content of Dockerfile:

FROM docker.elastic.co/elasticsearch/elasticsearch:6.8.15

RUN elasticsearch-plugin install analysis-icu
RUN elasticsearch-plugin install analysis-phonetic

And the content docker-compose.yml:

version: '2.2'
services:
  elasticsearch:
    #image: docker.elastic.co/elasticsearch/elasticsearch:6.8.15
    build:
      context: ./
      dockerfile: Dockerfile
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - http.cors.allow-headers=X-Requested-With,Content-Type,Content-Length,Authorization
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
      - esplugins1:/usr/share/elasticsearch/plugins
    ports:
      - 9268:9200
    networks:
      - esnet
  elasticsearch2:
    #image: docker.elastic.co/elasticsearch/elasticsearch:6.8.15\
    build:
      context: ./
      dockerfile: Dockerfile
    container_name: elasticsearch2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - http.cors.allow-headers=X-Requested-With,Content-Type,Content-Length,Authorization
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata2:/usr/share/elasticsearch/data
      - esplugins2:/usr/share/elasticsearch/plugins
    networks:
      - esnet

volumes:
  esdata1:
    driver: local
  esdata2:
    driver: local
  esplugins1:
    driver: local
  esplugins2:
    driver: local

networks:
  esnet:

This is the default Elasticsearch 6.8.15 docker-compose.yml file from Elasticsearch website itself https://www.elastic.co/guide/en/elasticsearch/reference/6.8/docker.html#docker-cli-run-prod-mode. And I added two named data volumes, esplugins1 and esplugins2, for two of these nodes. So these plugins can be persisted between docker-compose down.

Note, if you ever run docker-compose down -v then these volumes will be removed!

I commented out the image line and moved that image to Dockerfile. And then using RUN command added the elasticsearch-plugin install command. This elasticsearch-plugin command is natively available in the elasticsearch container. And you can check this once you are in the container shell.

Octahedrite

Answer 7

This is inspired by @Bigood answer to disable the prompt (i.e when installing repository-gcs)

elasticsearch:
  image: docker.elastic.co/elasticsearch/elasticsearch:7.3.0
  user: elasticsearch
  command: >
    /bin/sh -c "./bin/elasticsearch-plugin list | grep -q repository-gcs 
    || ./bin/elasticsearch-plugin install --batch repository-gcs; 
    /usr/local/bin/docker-entrypoint.sh"
  ...

added --batch flag to disable the yes/no prompt during instalation

Astridastride

Answer 8

Just for somebody who is using elasticsearch version starting from 7 and want to install plugin through the dockerfile then use the --batch flag

FROM elasticsearch:7.16.2
RUN bin/elasticsearch-plugin install repository-azure --batch

Sanjay