Is the information stored inside GKE "etcd" encrypted?

Question

I am using GKE(Google Kubernetes Engine) 1.13.6-gke.6 and I need to provide etcd encryption evidence for PCI purposes.
I have used --data-encryption-key flag and used a KMS key to encrypt secrets following this documentation.

I need to give a set of commands which will prove that the information stored in etcd of the master node is encrypted.

Here is how we verify that the secrets stored inside a normal Kuebrnetes Cluster (not GKE) are encrypted.

As we know GKE is a managed service and master node is managed by GCP. Is there a way to access GKE "etcd" to see the stored secrets and data at rest ?

Opus

You can create a K8s cluster with a master and few worker nodes in (GCP) your account. Master node will install etcd as its part of master component. Then you create some secrets, configMaps which will be stored in etcd. Then you may play with etcd to find out how they are being stored. +1 good question. – Christology

@RobertRanjan but we will not be using GKE this way, right? The master node is independent of Google Cloud and this way I would not be able to verify if the data is encrypted inside GKE. – Opus

I never used GKE. If we don’t have access to GKE’s etcd, we can’t verify using the way I mentioned above. – Christology

That is what I am looking for. A way to access the etcd or to verify the encryption by any other means. – Opus

If GKE is a service, it's etcd holds information about many more cluster just like yours. So, NO ONE should be having access to it except their admins. GKE should be having their own PCI compliance and much more to protect customer/cluster data. "You should not try to get access to GKE's etcd". You should look for other ways like reading GKE's documentation or reaching out to them to find out what you are looking for. – Christology

@RobertRanjan But I am just looking for a confirmation on the encryption part or that there is no way to access GKE etcd directly. That should be enough for this task. However, your points make sense. – Opus

Answer 1

Why do you have to prove that the information is encrypted? GKE is covered by Google Cloud's PCI DSS certification and since the master is a part of the "cluster as a service" that should be out of scope for what you need to show since you don't (and can't) control the way in which the storage is implemented.

One thing you can do is use Application-layer Secrets Encryption to encrypt your secrets with your own key stored in Cloud KMS. For your secrets you would be able to run commands to prove that additional level of encryption.

Wight

Thank you for the PCI link. It mostly does the work for me. However, I am just curious to learn. I followed this (cloud.google.com/kubernetes-engine/docs/how-to/…) documentation and created a cluster. Create a kuberntes secret using kubectl create secret command, is there a way in which I can verify the secret stored inside the k8 cluster is encrypted? – Opus

Yeah, it looks like you are right and that the encryption / decryption happens behind the scenes. The one thing you can do is destroy your key to prove to yourself that it is working, but that will render the encrypted data unusable so you can't do that on your production clusters. – Wight

Even after destroying the key in KMS, I am able to access the secrets kubectl get secrets -o yaml. I am still not able to prove anything. – Opus

The how-to you linked to says "When you destroy a key in Cloud KMS used to encrypt a Secret in GKE, that Secret is no longer available" so if you can show that isn't happening please file a bug with Google's Cloud support team. – Wight

WARNING: Cluster etcd-cluster is DEGRADED with message: "Codes: [None] Messages: [u'Kubernetes Engine Service Agent account unable to use CloudKMS key configured for Application Level encryption. Please, refer to https://cloud.google.com/kubernetes-engine/docs/how-to/encrypting-sec rets to fix this.'].". This is the error message I see when I do gcloud container cluster list but every kubectl command works. I think I will report a bug or a query for sure. – Opus

Answer 2

Ref: https://cloud.google.com/kubernetes-engine/docs/concepts/control-plane-security#etcd_security

In Google Cloud, customer content is encrypted at the filesystem layer by default. So disks that host etcd storage for GKE clusters are encrypted at the filesystem layer. For more information, see Encryption at Rest.

Farlay