Kubernetes Namespaces stuck in Terminating status

Question

We have one cluster where it seems that namespaces never want to be deleted completely and now can't re-create custom-metrics namespace to be able to collect custom metrics to properly setup HPA. I fully understand that I can create another namespace with all custom-metrics resources, but a little concerned with the overall health of the cluster, given that the namespaces get stuck in "Terminating" state

$ kubectl get ns
NAME             STATUS        AGE
cert-manager     Active        14d
custom-metrics   Terminating   7d
default          Active        222d
nfs-share        Active        15d
ingress-nginx    Active        103d
kube-public      Active        222d
kube-system      Active        222d
lb               Terminating   4d
monitoring       Terminating   6d
production       Active        221d

I already tried to export namespaces to JSON, delete finalizers and re-create using edited JSON files. also tried to kubectl edit ns custom-metrics and delete "- kubernetes" finalizer. all to no avail.

does anyone have any other recommendations on how else I can try to destroy these "stuck" namespaces"

curl to https://master-ip/api/v1/namespace/...../finalize doesn't seem to work on Google Kubernetes Engine for me, I'm assuming these operations are not allowed on GKE cluster

Trying things like doesn't work as well:

$ kubectl delete ns custom-metrics --grace-period=0 --force

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (Conflict): Operation cannot be fulfilled on namespaces "custom-metrics": The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system.

and there're no resources listed in this namespaces at all: kubectl get all -n custom-metrics or looping through all api-resources in this namespace shows no resources exist at all: kubectl api-resources --namespaced=true -o name | xargs -n 1 kubectl get -n custom-metrics

Mills

Could you please check if any resource is still pending deletion in custom-metrics using kubectl get all -n custom-metrics – Shyamal

No resources are pending and only "kubernetes" is listed in finalizers. But removing this finalizer manually doesn't save. Finalizer "kubernetes" comes right back after removing it. and namespace remains in Terminating status – Mills

This has been already answered here: #55853812 – Cadena

Answer 1

I did something similar to rahul.tripathi except the curl did not work for me - I followed https://medium.com/@craignewtondev/how-to-fix-kubernetes-namespace-deleting-stuck-in-terminating-state-5ed75792647e which does the following:

NAMESPACE=
kubectl get namespace $NAMESPACE -o json > $NAMESPACE.json
sed -i -e 's/"kubernetes"//' $NAMESPACE.json
kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f ./$NAMESPACE.json

Voila! Namespace is deleted

Update: One-liner version of this solution (requires jq)

NAMESPACE= ; kubectl get namespace $NAMESPACE -o json | jq 'del(.spec.finalizers[0])' | kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f -

Update #2: Terraform version

resource "kubernetes_namespace" "this" {
  for_each = toset( var.namespaces )
  metadata {
    name = each.key
  }
  provisioner "local-exec" {
    when = destroy
    command    = "nohup ${path.module}/namespace-finalizer.sh ${each.key} 2>&1 &"
  }
}

namespace-finalizer.sh

sleep 30; kubectl get namespace $1 && kubectl get namespace $1 -o json | jq 'del(.spec.finalizers[0])' | kubectl replace --raw "/api/v1/namespaces/$1/finalize" -f -

Colchester

This is the actual fix, you saved my time, thank you. – Overunder

Just to add that sometimes you need to forcefully delete all PVC or some individual objects with 'finalizers' there; And after that, this command can actually delete namespace. – Medallion

handy little command, I'll keep it :) – Gay

Thank you. You can do the same in one line using pipes: ``` kubectl get namespace $NAMESPACE -o json | sed -e 's/"kubernetes"//' | kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f - ``` – Charissa

Looks like this is also the "official" answer for EKS: aws.amazon.com/premiumsupport/knowledge-center/… – Gibby

Perfect Perfect. – Grunter

Answer 2

Looks like this is a known issue with people having mixed results trying a mix of different things:

• Bounce the kube-controller-manager
• Bounce all the kubelets
• Bounce the whole cluster
• kubectl delete ns <name> --grace-period=0 --force
• Patching finalizers "null" everywhere.

Some more background but at the pod level here too.

Backfield

obviously i tried this as well to no avail: kubectl delete ns custom-metrics --grace-period=0 --force warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (Conflict): Operation cannot be fulfilled on namespaces "custom-metrics": The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system. and it still stays in place forever in same Terminating status – Mills

Answer 3

For me, deletion with --grace-period=0 --force has never worked. Rico's answer is good, but probably you can do it without restarting your cluster.

In my case, there are ALWAYS some objects which were recreated after you have "deleted" your namespace.

To see which Kubernetes resources are and aren’t in a namespace:

kubectl api-resources --namespaced=true
kubectl api-resources --namespaced=false

What I am doing is to go through it and find all k8s objects which were in some use of that specific namespace, and delete them manually.

EDIT: Another useful command for finding objects that should be deleted:

kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -l <label>=<value> -n <namespace>

Medallion

"kubectl api-resources" ends up with the following error: "error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" – Dragging

Can you check if only kubectl api-resources works for you. If not, there is something wrong with your Kube API or you don't have enough permissions – Medallion

Answer 4

The only solution that worked for me was:

1. kubectl get namespace annoying-namespace-to-delete -o json > tmp.json

2. edit tmp.json and remove"kubernetes" from "spec": { "finalizers":[]}

3. curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://kubernetes-cluster-ip/api/v1/namespaces/annoying-namespace-to-delete/finalize

and this should delete your namespace,

Entire

Answer 5

• 1. Get state current:

kubectl get namespace <namespace-to-delete> -o json > tmp.json

Obs.: Replace

• 2. edit tmp.json and remove "kubernetes" from "spec": { "finalizers":['kubernetes']}

Ex.: "spec": { "finalizers":[ ]}

• 3. Alter the namespace and delete:

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://<kubernetes-cluster-ip>/api/v1/namespaces/<namespace-to-delete>/finalize -H "Authorization: Bearer <token-Account>"

Obs.: Replace namespace-to-delete, token-Account and token-Account

Arvid

Do you know who adds the kubernetes finalizer? – Clod

Answer 6

No need to modify anything.

No need to use curl to talk to the api-server.

$ export NAMESPACE=cert-manager
$ kubectl get ns ${NAMESPACE} -o json | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/${NAMESPACE}/finalize" -f -

Now, make sure the namespace is gone:

$ kubectl get ns ${NAMESPACE}
Error from server (NotFound): namespaces "cert-manager" not found

Dragging

Answer 7

Had the problem deleting a namespace:

kubectl delete namespaces "localkube-ns" 

Error from server (Conflict):
Operation cannot be fulfilled on namespaces "localkube-ns": 
The system is ensuring all content is removed from this namespace.  
Upon completion, this namespace will automatically be purged by the system.

After several long minutes, the problem disapeard.

The namespaced was probably long to delete after a problem that did generate a lot of Evicted pods.

This command seems faster: kubectl delete all --all --namespace=<NAMESPACE_NAME>

Antagonist

Answer 8

It's very simple. you can simply follow 2 easy ways,

1. Remove/commend finalizers under metadata by using kubectl edit ns <namespace-name>

2. if there is no Finalizers under meta data only under spec then you can follow the below,

   export NAMESPACE=name-space then

   kubectl get namespace $NAMESPACE -o json | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" | kubectl replace --raw /api/v1/namespaces/$NAMESPACE/finalize -f -

Modulator

Answer 9

Was able to reproduce by installing a Prometheus operator from this repo and then just trying to delete a namespace.

First run:

k apply -f manifests/

That command creates monitoring namespace, a bunch of namespaced resources like deployments and configmaps as well as non-namespaced ones like roles etc.

Then imperatively deleting the namespace:

k delete ns monitoring

with an idea the Kubernetes will delete all the corresponding resources. As a result all objects in the namespace were deleted however namespace itself get stuck in the Terminated state

Just to illustrate, here is a list of stray resources left after "deleting" the namespace. Those resources got deleted only after running the kubectl delete on the corresponding folder:

customresourcedefinition.apiextensions.k8s.io "podmonitors.monitoring.coreos.com" deleted
customresourcedefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" deleted
customresourcedefinition.apiextensions.k8s.io "prometheusrules.monitoring.coreos.com" deleted
customresourcedefinition.apiextensions.k8s.io "servicemonitors.monitoring.coreos.com" deleted
clusterrole.rbac.authorization.k8s.io "prometheus-operator" deleted
clusterrolebinding.rbac.authorization.k8s.io "prometheus-operator" deleted
clusterrole.rbac.authorization.k8s.io "kube-state-metrics" deleted
clusterrolebinding.rbac.authorization.k8s.io "kube-state-metrics" deleted
clusterrole.rbac.authorization.k8s.io "node-exporter" deleted
clusterrolebinding.rbac.authorization.k8s.io "node-exporter" deleted
apiservice.apiregistration.k8s.io "v1beta1.metrics.k8s.io" deleted
clusterrole.rbac.authorization.k8s.io "prometheus-adapter" deleted
clusterrole.rbac.authorization.k8s.io "system:aggregated-metrics-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "prometheus-adapter" deleted
clusterrolebinding.rbac.authorization.k8s.io "resource-metrics:system:auth-delegator" deleted
clusterrole.rbac.authorization.k8s.io "resource-metrics-server-resources" deleted
rolebinding.rbac.authorization.k8s.io "resource-metrics-auth-reader" deleted
clusterrole.rbac.authorization.k8s.io "prometheus-k8s" deleted
clusterrolebinding.rbac.authorization.k8s.io "prometheus-k8s" deleted
rolebinding.rbac.authorization.k8s.io "prometheus-k8s" deleted
role.rbac.authorization.k8s.io "prometheus-k8s" deleted

This experiment is likely proving the idea if your namespace is stuck in Terminated state there are always resources left referring it and preventing it to get deleted. The easiest (and correct) way to clean it up is using the same instrumentation as when creating it (kubectl apply, Helm etc).

Punishable

Answer 10

This solution worked for me,

kubectl patch namespace cattle-system -p '{"metadata":{"finalizers":[]}}' --type='merge' -n cattle-system

kubectl delete namespace cattle-system --grace-period=0 --force

"cattle-system" is the namespace to delete

Massachusetts

Didn't work for me. I still see "finalizers":["kubernetes"], hence the namespace not terminated. – Dragging

Answer 11

In my case the solution was:

kubectl delete -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Indeed, when setting up metrics, I tried to use metrics-server, but didn't work. So I used https://bitnami.com/stack/prometheus-operator/helm instead, but didn't remove metrics-server.
By doing

kubectl api-resources --verbs=list --namespaced -o name \                                                                         
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n devops-87

then
kubectl describe APIService v1beta1.metrics.k8s.io

I realized that the problem was this unused metrics-server.

Indican