ElasticsearchSecurityException: invalid configuration for xpack.security.transport.ssl

Question

I'm getting the following error when trying to spin up an elasticsearch cluster in EC2.

org.elasticsearch.ElasticsearchSecurityException: invalid configuration for xpack.security.transport.ssl - [xpack.security.transport.ssl.enabled] is not set, but the following settings have been configured in elasticsearch.yml : [xpack.security.transport.ssl.keystore.secure_password,xpack.security.transport.ssl.truststore.secure_password]

This is the script I'm using to bootstrap the cluster

"#!/bin/bash",
"sudo yum update -y",
"sudo yum install java-1.8.0 -y",
"sudo rpm -i https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-${var.elastic_version}-x86_64.rpm",
"sudo systemctl daemon-reload",
"sudo systemctl enable elasticsearch.service",
"sudo chmod -R 777 /etc/elasticsearch",
"sudo sed -i 's@-Xms1g@-Xms${aws_instance.elastic_datanodes[count.index].root_block_device[0].volume_size / 2}g@g' /etc/elasticsearch/jvm.options",
"sudo sed -i 's@-Xmx1g@-Xmx${aws_instance.elastic_datanodes[count.index].root_block_device[0].volume_size / 2}g@g' /etc/elasticsearch/jvm.options",
# "sudo sed -i 's/#network.host: 192.168.0.1/network.host: 0.0.0.0/g' /etc/elasticsearch/elasticsearch.yml",
"sudo rm /etc/elasticsearch/elasticsearch.yml",
"sudo cp elasticsearch.yml /etc/elasticsearch/elasticsearch.yml",
"sudo systemctl start elasticsearch.service"

This is my elasticsearch yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: ${cluster_name}
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ${node_name}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: ${node}
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#

discovery.seed_hosts:
%{ for seed_host in seed_hosts ~}
  - "${seed_host}"
%{ endfor ~}

#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#

cluster.initial_master_nodes:
%{ for master_node in initial_master_nodes ~}
  - "${master_node}"
%{ endfor ~}

node.roles: [data, master]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

Why is this happening? I can probably get around this by disabling all security settings, but I'd rather not do that for obvious reasons.

Maximilien

can you share the content of Elasticsearch.yml ? – Appulse

@Amit-ESenthusiast added the elasticsearch.yml to the question – Maximilien

Answer 1

That possible you have previous ElasticSearch installation that settings cached, you can try to reset security settings with commands below:

rm /etc/elasticsearch/elasticsearch.keystore
/usr/share/elasticsearch/bin/elasticsearch-keystore create

Imply

this is working in Elasticsearch 8.11 version. Thanks you @Imply – Dedradedric

Answer 2

This happens because elasticsearch seemingly sets these values in a new installation, in their keystore, not the yaml config file.

A bit less invasive than XPS's correct answer would be to

elasticsearch-keystore  list

and then to remove the properties that are mentioned, e.g.

elasticsearch-keystore remove xpack.security.http.ssl.keystore.secure_password

Paly

Answer 3

I believe you need to set

xpack.security.transport.ssl.enabled: true

In your elasticsearch.yml file.

Benny

Answer 4

Additionally to what @Paulo mentioned, you also need to set the following parameters if you enable xpack security to true

xpack.security.transport.ssl.enabled: true

Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

Enable encryption and mutual authentication between cluster nodes

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

Washhouse