Allow mutating webhooks to work with tls-enabled istio

Question

I have the following MutatingWebhookConfiguration

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: example-webhook
webhooks:
  - name: example-webhook.default.svc.cluster.local
    admissionReviewVersions:
      - "v1beta1"
    sideEffects: "None"
    timeoutSeconds: 30
    objectSelector:
      matchLabels:
        example-webhook-enabled: "true"
    clientConfig:
      service:
        name: example-webhook
        namespace: default
        path: "/mutate"
      caBundle: "LS0tLS1CR..."
    rules:
      - operations: [ "CREATE" ]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]

I want to inject the webhook pod in an istio enabled namespace with istio having strict TLS mode on.

Therefore, (I thought) TLS should not be needed in my example-webhook service so it is crafted as follows:

apiVersion: v1
kind: Service
metadata:
  name: example-webhook
  namespace: default
spec:
  selector:
    app: example-webhook
  ports:
    - port: 80
      targetPort: webhook
      name: webhook

However when creating a Pod (that does indeed trigger the webhook) I get the following error:

▶ k create -f demo-pod.yaml
Error from server (InternalError): error when creating "demo-pod.yaml": Internal error occurred: failed calling webhook "example-webhook.default.svc.cluster.local": Post "https://example-webhook.default.svc:443/mutate?timeout=30s": no service port 443 found for service "example-webhook"

Can't I configure the webhook not to be called on 443 but rather on 80? Either way TLS termination is done by the istio sidecar.

Is there a way around this using VirtualService / DestinationRule?

edit: on top of that, why is it trying to reach the service in the example-webhook.default.svc endpoint? (while it should be doing so in example-webhook.default.svc.cluster.local) ?

Update 1

I have tried to use https as follows:

I have created a certificate and private key, using istio's CA.

I can verify that my DNS names in the cert are valid as follows (from another pod)

echo | openssl s_client -showcerts -servername example-webhook.default.svc -connect example-webhook.default.svc:443 2>/dev/null | openssl x509 -inform pem -noout -text

...
 Subject: C = GR, ST = Attica, L = Athens, O = Engineering, OU = FOO, CN = *.cluster.local, emailAddress = [email protected]
...
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:*.default.svc.cluster.local, DNS:example-webhook, DNS:example-webhook.default.svc
...

but now pod creation fails as follows:

▶ k create -f demo-pod.yaml
Error from server (InternalError): error when creating "demo-pod.yaml": Internal error occurred: failed calling webhook "example-webhook.default.svc.cluster.local": Post "https://example-webhook.default.svc:443/mutate?timeout=30s": x509: certificate is not valid for any names, but wanted to match example-webhook.default.svc

Update 2

The fact that the certs the webhook pod are running with were appropriately created using the istio CA cert, is also validated.

curl --cacert istio_cert https://example-webhook.default.svc
Test

where istio_cert is the file containing istio's CA certificate

What is going on?

Audryaudrye

can you try following steps 1, 3,4 &5 as shown here technekey.com/how-to-setup-a-mutation-webhook-in-kubernetes. I am able to successfully run a custom webhook with istio enabled in the cluster. – Decompress

Unfortunately I am on GKE (managed k8s) therefore I don't have access to the root node – Audryaudrye

Answer 1

Not sure if you can use webhook on port 80...

Perhaps some of this will be useful to you, I used the following script to generate certificates, you can change it to suit your needs:

#!/bin/bash

set -e

service=webhook-svc
namespace=default
secret=webhook-certs
csrName=${service}.${namespace}

cat <<EOF >> csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${service}
DNS.2 = ${service}.${namespace}
DNS.3 = ${service}.${namespace}.svc
EOF

openssl genrsa -out server-key.pem 2048
openssl req -new -key server-key.pem -subj "/CN=${service}.${namespace}.svc" -out server.csr -config csr.conf

kubectl delete csr ${csrName} 2>/dev/null || true

cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${csrName}
spec:
  groups:
  - system:authenticated
  request: $(< server.csr base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

sleep 5

kubectl certificate approve ${csrName}

for i in {1 .. 10}
do
    serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
    if [[ ${serverCert} != '' ]]; then
        break
    fi
    sleep 1
done
if [[ ${serverCert} == '' ]]; then
    echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
    exit 1
fi


echo "${serverCert}" | openssl base64 -d -A -out server-cert.pem


# create the secret with CA cert and server cert/key
kubectl create secret generic ${secret} \
        --from-file=key.pem=server-key.pem \
        --from-file=cert.pem=server-cert.pem \
        --dry-run -o yaml |
    kubectl -n ${namespace} apply -f -

The script creates a secret, which I then mounted into the webhook, deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webhook-deployment
  namespace: default
  labels:
    app: webhook
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webhook
  template:
    metadata:
      labels:
        app: webhook
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      containers:
        - name: webhook
          image: webhook:v1
          imagePullPolicy: IfNotPresent
          volumeMounts:
          - name: webhook-certs
            mountPath: /certs
            readOnly: true
      volumes:
      - name: webhook-certs
        secret:
          secretName: webhook-certs

service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: webhook-svc
  namespace: default
  labels:
    app: webhook
spec:
  ports:
  - port: 443
    targetPort: 8443
  selector:
    app: webhook

Pronominal

Answer 2

Did you try adding the port attribute in your MutatingWebhookConfiguration

clientConfig:
      service:
        name: example-webhook
        namespace: default
        path: "/mutate"
        port: 80

Cineaste

I am getting the same error but in port 80 now: error when creating "demo-pod.yaml": Internal error occurred: failed calling webhook "example-webhook.default.svc.cluster.local": Post "https://example-webhook.default.svc:80/mutate?timeout=30s": x509: certificate is not valid for any names, but wanted to match example-webhook.default.svc – Audryaudrye

This is a different error than the previous one. It mentions a certificate error. – Cineaste

If you check my update on the original question you will see that this is the error I am getting after enabling tls. either way, the call from the api server to the webhook service always seems to use https protocol, therefore it expects some certificates – Audryaudrye

If you inspect the certificate, it is one generated by istio and not one generate by your web hook listener – Biddy

Answer 3

You can try changing value

cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${csrName}
spec:
  groups:
  - system:authenticated
  request: $(< server.csr base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

Nathalie