Securing Kubernetes Service with TLS

Question

I have an application that is internal and exposed only to other application on the cluster by a service with cluster IP. Other services are accessing this application via it's DNS (serviceName-namespace.svc.cluster.local). This application handles sensitive data, so although all the communication is inside the cluster I would like to use TLS to secure the communications to this application.

My question is - how can I enable TLS on a service? Is there something already exist or should I handle it on the application code? Also, is there already a CA I can use on the cluster that can sign certificates for .svc.cluster.local?

To clarify, I know I can use ingress for this purpose. The only problem is keeping this service internal only - so only services inside the cluster will be able to access it.

Thanks, Omer

Serpent

Did you check out Jetstack's cert-manager project already? – Dato

No, but now after taking a quick look - I'm not sure this will work for internal services, with internal DNS – Serpent

Answer 1

I just found that Kubernetes API can be used to generate a certificate that will be trusted by all the pods running on the cluster. This option might be simpler than the alternatives. You can find the documentation here, including full flow of generating a certificate and using it.

Serpent

Interesting, and easier than my answer. +1 – Bini

Does this work OK for you? The documentation you linked to describes putting the pod IPs as SANs in the certificate. Surely these will change on a regular basis when the pod is recycled and become invalid? – Griefstricken

I did not play with it yet. If I'll have the chance, I'll give it a try and update – Serpent

@OmerLeviHevroni: Is it working with this solution? Also, the document says Now you can use server.crt and server-key.pem as the keypair to start your HTTPS server. What does it mean? example.default.svc.cluster.local is default URL for all services. Where do we need to provide that certificate? – Wandis

You need to provide them to your service to start the TLS connection. It really depend on the stack - different technologies have different configuration options for that. – Serpent

I provide certificate information in nginx ingress for publically facing domains. But, there is no ingress for default.svc.cluster.local domains. Not sure how can I configure it. – Wandis

yes, you will have to handle TKS directly in your code - or use tools like Istio that handle the TLS for you – Serpent

hey hi, I followed this ref (kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster) I am struck with same question Now you can use server.crt and server-key.pem as the keypair to start your HTTPS server. What does it mean? example.default.svc.cluster.local is default URL for all services. Where do we need to provide that certificate? @OmerLeviHevroni – Kapellmeister

You need to provide it you your server on start so it could use it to serve TLS traffic, which language are you using? – Serpent

Answer 2

Following @vonc comments from bellow, I think I have a solution:

• Purchase a public valid domain for this service (e.g. something.mycompany.com).
• Use CoreDNS to add override rule so all requests to something.mycompany.com will go to something-namesapce.svc.cluster.local, as the service is not exposed externally (this can be done also with normal A record for my use case).
• Use Nginx or something else to handle TLS with the certificate for something.mycompany.com.

This sounds pretty complicated but might work. What do you think?

Serpent

the Service works at Layer 4 mate... what you're saying is Layer 7 has nothing to do with this... – Unregenerate

Answer 3

Check if the tutorial "Secure Kubernetes Services with Ingress, TLS and LetsEncrypt" could apply to you:

Ingress can be backed by different implementations through the use of different Ingress Controllers. The most popular of these is the Nginx Ingress Controller, however there are other options available such as Traefik, Rancher, HAProxy, etc. Each controller should support a basic configuration, but can even expose other features (e.g. rewrite rules, auth modes) via annotations.

Give it a domain name and enable TLS. LetsEncrypt is a free TLS certificate authority, and using the kube-lego controller we can automatically request and renew LetsEncrypt certificates for public domain names, simply by adding a few lines to our Ingress definition!

In order for this to work correctly, a public domain name is required and should have an A record pointing to the external IP of the Nginx service.

For limiting to inside the cluster domain though (svc.cluster.local), you might need CoreDNS.

Bini

I know this can be done with ingress, I used ingress a couple of times before. I don't want to use ingress because I want to keep this service internal (e.g, accessible only from inside the cluster). It seems that it might be possible by reading here, but this seems a bit complicated. Also, what certificate shuld I use? – Serpent

@OmerLeviHevroni You can use a self-signed certificate for testing (kubernetes.io/docs/concepts/services-networking/…) – Bini

I know, but that doesn't sound like a something I would like to use in production. This means I should roll the self-signed certificate to each client consuming this service. – Serpent

@OmerLeviHevroni Agreed: at work we are using a wildcard certificate managed by F5 (our proxy: f5.com/glossary/reverse-proxy) – Bini

F5 is running on the cluster and acting as the CA? I am not familiar with it enough so sorry if this is a trivial question. Also, this sounds like expensive solution... – Serpent

Yes, I don't have a generic solution, I was just pointing out what we do. – Bini

Answer 4

On Google Cloud you can make load balancer service internal, like this:

    annotations = {
      "cloud.google.com/load-balancer-type" = "Internal"
    }

Wick