Istio Ingress Gateway with TLS termination returning 503 service unavailable

Question

We want to to route https traffic to an https endpoint using Istio Ingress Gateway.

We terminate the TLS traffic at the Ingress Gateway, but our backend service uses https as well.

I have the following manifests:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: datalake-dsodis-istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "gw-hdfs-spark.dsodis.domain"
    - "spark-history.dsodis.domain"
    port:
      name: https-wildcard
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: gw-spark-history-istio-vs
spec:
  gateways:
  - default/datalake-dsodis-istio-gateway
  hosts:
  - "spark-history.dsodis.domain"
  http:
    - match:
      - uri:
          prefix: /
      route:
        - destination:
            host: gateway-svc-clusterip.our_application_namespace.svc.cluster.local
            port:
              number: 8443

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: originate-tls-for-spark-history
spec:
  host: gateway-svc-clusterip.our_application_namespace.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 8443 
      tls:
        mode: SIMPLE

The problem is most likely, that we are sending TLS terminated traffic, (so to say) HTTP traffic, to the HTTPS backend. Therefore we might get 503 Service Unavailable when accessing the service through Istio.

The command accessing it is:

curl -vvvv -H"Host: spark-history.dsodis.domain" --resolve "spark-history.dsodis.domain:31390:IP" https://spark-history.dsodis.domain:31390/gateway/default/sparkhistory  -k

My question is, how can I tell Istio to route traffic to the backend service using https?

Thanks in advance.

Best regards, rforberger

Ecosphere

Hi, Ronny there are two different approaches for this : 1. Use mTLS migration and change backend service to HTTP. This would result in internal traffic between services to also be secure. OR 2. Replace TLS termination with ingress gateway with SNI passthrough. This would result in HTTPS ingress access to an HTTPS service. I'm not sure if its possible with HTTPS backend and TLS termination. – Safari

If You are ok with my suggestions. Which approach do You prefer? – Safari

Hi @PiotrMalec thanks for your explainations. It actually works with TLS termination and HTTPS backend. The problem was the DestinationRule not being in the istio-system namespace, where the traffic to the backend originates. – Ecosphere

Hi @RonnyForberger, for the benefit of the community could you elaborate in an answer how you solved? – Citrin

@Citrin I solved it by adding the namespace istio-system to my DestinationRule that tells the traffic to the destination service to be TLS. – Ecosphere

Answer 1

As RonnyForberger mentioned in his comment this can be achieved by creating DestinationRule that tells the traffic to the destination service to be TLS connection.

So in this scenario:

1. HTTPS request gets TLS terminated at GateWay to HTTP.
2. Then the HTTP request is translated to TLS with DestinationRule to HTTPS.
3. HTTPS request reaches HTTPS backend.

Safari

Answer 2

You can also use the Istio Ingress Gateway in PASSTHROUGH mode. This will forward the original TLS traffic as it to the destination and you can avoid the TLS termination overhead at the Gateway.

https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-sni-passthrough/#configure-an-ingress-gateway

Rincon