Vault sidecar injector permission denied only for vault enterprise

Question

I am trying to explore vault enterprise but getting permission denied for sidecar when I use the vault enterprise but seems to work fine when I tried to use local vault server.

Here is the repository that contains a working example with the local vault vault-sidecar-injector-app

Vault config

export VAULT_ADDR="https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200"
export VAULT_NAMESPACE="admin"

#install agent
helm upgrade --install vault hashicorp/vault --set "injector.externalVaultAddr=$VAULT_ADDR"
vault auth enable kubernetes
# get certs & host
VAULT_HELM_SECRET_NAME=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("vault-token-")).name')
TOKEN_REVIEW_JWT=$(kubectl get secret $VAULT_HELM_SECRET_NAME --output='go-template={{ .data.token }}' | base64 --decode)
KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

# set Kubernetes config
vault write auth/kubernetes/config \
        token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
        kubernetes_host="$KUBE_HOST" \
        kubernetes_ca_cert="$KUBE_CA_CERT" \
        issuer="https://kubernetes.default.svc.cluster.local" \
        disable_iss_validation="true" \
        disable_local_ca_jwt="true"


vault auth enable approle
# create admin policy
vault policy write admin admin-policy.hcl
vault write auth/approle/role/admin policies="admin"
vault read auth/approle/role/admin/role-id
# generate secret
vault write -f  auth/approle/role/admin/secret-id
#Enable KV
vault secrets enable -version=2 kv

I can see the role and policy

Admin policy

Here is the admin policy for the enterprise

path "*" {
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

Deploy Script for helm

here is the deploy script, tried hcp-root root policy but no luck

RELEASE_NAME=demo-managed
NAMESPACE=default
ENVIRONMENT=develop
export role_id="f9782a53-823e-2c08-81ae-abc"
export secret_id="1de3b8c5-18c7-60e3-24ca-abc"
export VAULT_ADDR="https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200"
export VAULT_TOKEN=$(vault write -field="token"  auth/approle/login role_id="${role_id}" secret_id="${secret_id}")
vault write auth/kubernetes/role/${NAMESPACE}-${RELEASE_NAME} bound_service_account_names=${RELEASE_NAME} bound_service_account_namespaces=${NAMESPACE} policies=hcp-root ttl=1h
helm upgrade --install $RELEASE_NAME ../helm-chart --set environment=$ENVIRONMENT --set nameOverride=$RELEASE_NAME

also tried with root token

RELEASE_NAME=demo-managed
NAMESPACE=default
ENVIRONMENT=develop
vault write auth/kubernetes/role/${NAMESPACE}-${RELEASE_NAME} bound_service_account_names=${RELEASE_NAME} bound_service_account_namespaces=${NAMESPACE} policies=hcp-root ttl=1h
helm upgrade --install $RELEASE_NAME ../helm-chart --set environment=$ENVIRONMENT --set nameOverride=$RELEASE_NAME

Sidecar config

With namespace annotation, as my understanding namespace is required

vault.hashicorp.com/namespace - configures the Vault Enterprise namespace to be used when requesting secrets from Vault.

https://www.vaultproject.io/docs/platform/k8s/injector/annotations

vault.hashicorp.com/namespace : "admin"

Error

  | Error making API request.
  |
  | URL: PUT https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200/v1/admin/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied

Without namespace annotation getting below error

  | URL: PUT https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200/v1/auth/kubernetes/login
  | Code: 400. Errors:
  |
  | * missing client token

Even enabling debug logs vault.hashicorp.com/log-level : "debug" does not help me with this error, any help or suggestions will be appreciated.

Also tried https://support.hashicorp.com/hc/en-us/articles/4404389946387-Kubernetes-auth-method-Permission-Denied-error

So seems like I am missing something very specific to the vault enterprise

Schistosomiasis

Answer 1

Finally able to resolve the weird issue with vault, posting as an answer might help someone else.

The only thing that I missed to understand the flow between vault server, sidecar, and Kubernetes.

Kubernetes should be reachable to vault enterprise for Token review API calls. As you can see the when sidecar makes a request to the vault, then the vault enterprise server performs a token review API call.

Use the /config endpoint to configure Vault to talk to Kubernetes. Use kubectl cluster-info to validate the Kubernetes host address and TCP port.

https://www.vaultproject.io/docs/auth/kubernetes

  | Error making API request.
  |
  | URL: PUT https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200/v1/admin/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=2.99s

This error does not indicate it has a connectivity issue but this also happens when the vault is not able to communicate with the Kubernetes cluster.

Kube Host

vault write auth/kubernetes/config \
        token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
        kubernetes_host="$KUBE_HOST" \
        kubernetes_ca_cert="$KUBE_CA_CERT" \
        issuer="https://kubernetes.default.svc.cluster.local"

KUBE_HOST should be reachable for vault enterprise for tokenreview process.

So for the vault to communicate with our cluster, we need a few changes.

minikube start --apiserver-ips=14.55.145.30 --vm-driver=none

Now update the vaul-config.sh file

KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

change this to

KUBE_HOST=""https://14.55.145.30:8443/"

No manual steps, for the first time configuration run

./vault-config.sh

and for the rest of the deployment in your CI/CD you can use

./vault.sh

Each release has only been able to access its own secrets.

Furter details can be found start-minikube-in-ec2

TLDR,

Note: Kubernetes cluster should be reachable to vault enterprise for authentication, so vault enterprise would not able to communicate with your local minikube cluster. Better to test it out on EC2

Schistosomiasis

Answer 2

When you have set the Kubernetes auth into the vault you have used the

vault write auth/kubernetes/config \
        token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
        kubernetes_host="$KUBE_HOST" \
        kubernetes_ca_cert="$KUBE_CA_CERT" \
        issuer="https://kubernetes.default.svc.cluster.local" \
        disable_iss_validation="true" \
        disable_local_ca_jwt="true"

you have mentioned

issuer="https://kubernetes.default.svc.cluster.local" \

it's HTTP and looks like running the local K8s server, not sure vault will be able to access this endpoint or not.

Did you try sending auth at this endpoint bypassing the Token & CA cert?

Your vault is able to connect and authenticate over the internet of your local cluster?

also i have faced this weird issue, with vault i used configure using both ways CLI and UI then it was working for me while in a single way it was giving error of permission denied.

Trachyte

I do not think the vault server communicates with the Kubernetes cluster, it's the sidecar the managed inside Kubernetes stuff and the sidecar communicates with vault server. second it should return timeout instead of permission denied ? – Schistosomiasis

Vault agent: Auto-Auth - Automatically authenticate to Vault and manage the token renewal process for locally-retrieved dynamic secrets. learn.hashicorp.com/tutorials/vault/agent-kubernetes?in=vault/… – Schistosomiasis

Thanks man, finally figure it out, its the kube_host that play the role. – Schistosomiasis

Glad to hear that you resolve issue. – Trachyte

yeah it was really weird , I have to reproduce the issue by running vault in docker and then minikube, and then I figure out the vault server logs, its not able to communicate with Kubernetes API. btw I added each everything in the repo with zero configuration – Schistosomiasis

Great...! thanks for sharing details that might be helpful to others, checked that repo updated 2-3 hours back with the handy shell script. i have also documente locally but i think cant attach the document in the answer, it's also the same with minimalist config. – Trachyte