Namespace “stuck” as Terminating

Question

I am getting issue while terminating the namesapce in the cluster, It's showing many parameters inside the namespace JSON. I followed this link https://medium.com/@craignewtondev/how-to-fix-kubernetes-namespace-deleting-stuck-in-terminating-state-5ed75792647e

 "spec": {},
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2021-01-11T08:41:48Z",
                "message": "All resources successfully discovered",
                "reason": "ResourcesDiscovered",
                "status": "False",
                "type": "NamespaceDeletionDiscoveryFailure"
            },
            {
                "lastTransitionTime": "2021-01-11T08:41:48Z",
                "message": "All legacy kube types successfully parsed",
                "reason": "ParsedGroupVersions",
                "status": "False",
                "type": "NamespaceDeletionGroupVersionParsingFailure"
            },
            {
                "lastTransitionTime": "2021-01-11T08:41:48Z",
                "message": "All content successfully deleted, may be waiting on finalization",
                "reason": "ContentDeleted",
                "status": "False",
                "type": "NamespaceDeletionContentFailure"
            },
            {
                "lastTransitionTime": "2021-01-11T08:42:09Z",
                "message": "All content successfully removed",
                "reason": "ContentRemoved",
                "status": "False",
                "type": "NamespaceContentRemaining"
            },
            {
                "lastTransitionTime": "2021-01-11T08:41:48Z",
                "message": "All content-preserving finalizers finished",
                "reason": "ContentHasNoFinalizers",
                "status": "False",
                "type": "NamespaceFinalizersRemaining"
            }
        ],
        "phase": "Terminating"
    }
}```

Kono

I've already seen this when there is a webhook still active. In this cases it took around 30 mins to delete the namespace. How long did you wait? – Stav

Does this answer your question? Namespace "stuck" as Terminating, How do I remove it? – Belong

Still in showing terminating since 215 min almost – Kono

@SaikatChakrabortty no it's not helping even I tried kubectl delete ns fleet-system --grace-period=0 --force --namespace -n fleet-system warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. warning: deleting cluster-scoped resources, not scoped to the provided namespace namespace "fleet-system" force deleted And stuck here------ – Kono

Answer 1

I have found the answer to terminate the stuck namespace.

for ns in $(kubectl get ns --field-selector status.phase=Terminating -o jsonpath='{.items[*].metadata.name}')
do
  kubectl get ns $ns -ojson | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/$ns/finalize" -f -
done

for ns in $(kubectl get ns --field-selector status.phase=Terminating -o jsonpath='{.items[*].metadata.name}')
do
  kubectl get ns $ns -ojson | jq '.metadata.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/$ns/finalize" -f -
done

Kono

This solution magically works. Thanks @rajendra sharma for posting the solution – Leggett

Thank you, this works. While k8s dashboard's delete button doesn't. – Russom

This actually works for me – Distrustful

Answer 2

Firstly export your namespace name in env which got struck in Terminating state

export NAMESPACE="monitoring"

Then run below command to delete the Terminating namespace

kubectl get namespace $NAMESPACE -o json   | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/"   | kubectl replace --raw /api/v1/namespaces/$NAMESPACE/finalize -f -

Juster

this worked for me thank you! – Clarkson

Answer 3

The tutorial you've used is not proper because deleting the namespace by removing finalizers is not good way to go since it could leave resources registered to a non existing namespace. Please take a look at this post: finalizer-kubernetes-ns.

You can try to find out which resources in the namespace are pending deletion by:

• Finding all resources that still exist using command kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n $yotur-ns-to-delete
• Checking if any apiservice is unavailable and hence doesn't serve its resources by executing command kubectl get apiservice|grep False

Take a look also at this problem: ns-kubernetes-stuck-terminating.

Amphiboly

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n prometheus error: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request No resources found in prometheus namespace. kubectl get apiservice|grep False v1beta1.custom.metrics.k8s.io prometheus/prometheus-adapter False (ServiceNotFound) 60d – Kono

Try to execute command: $ kubectl delete apiservice v1beta1.custom.metrics.k8s.io – Amphiboly

Error from server (NotFound): apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io" not found – Kono

Can you also try $ kubectl api-resources --verbs=list --namespaced -o name \ | xargs -n 1 kubectl get --show-kind --ignore-not-found -l <label>=<value> -n <namespace> I think error occurred due to previously deleted finalizer which is improper, steps I have mention you should have followed from beginning. – Amphiboly

Thanks @Amphiboly I had wanted to clean up a prometheus-adapter PoC and blew away everything with a k delete -f but forgot my NS was in there. ``` kubectl get apiservice|grep False v1beta1.custom.metrics.k8s.io prometheus/g-prometheus-adaptor-prometheus-adapter False (ServiceNotFound) 5h9m ``` Deleting custom metrics api allowed my NS to finally terminate. ``` kubectl delete apiservice v1beta1.custom.metrics.k8s.io ``` – Geehan