Namespace "stucked" as Terminating, How I removed it

Question

I've had a "stuck" namespace that I deleted showing in this eternal "terminating" status.

Buddhism

Was your namespace empty? – Palikar

#52954674 – Marine

Answer 1

Assuming you've already tried to force-delete resources like: Pods stuck at terminating status, and your at your wits' end trying to recover the namespace...

You can force-delete the namespace (perhaps leaving dangling resources):

(
NAMESPACE=your-rogue-namespace
kubectl proxy &
kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize
)

• This is a refinement of the answer here, which is based on the comment here.

• I'm using the jq utility to programmatically delete elements in the finalizers section. You could do that manually instead.

• kubectl proxy creates the listener at 127.0.0.1:8001 by default. If you know the hostname/IP of your cluster master, you may be able to use that instead.

• The funny thing is that this approach seems to work even when using kubectl edit making the same change has no effect.

Smectic

This was the only solution that worked for me. K8s v1.11.5, kubectl v1.8.0. jq installed from stedolan.github.io/jq/download – Glindaglinka

Would be better if you move the kubectl edit solution on top as it's much easier and does the same thing, applicable to all objects. – Plod

This is bad since it leaves leftovers in the cluster. Its more tricking yourself rather than actually fixing the issue. There is a reason the namespace can't be removed and you better fix that rather than going for a cosmetic solution. – Cale

@AntonioGomezAlvarado: Right. This wouldn't normally be the first thing that you try. But, unfortunately, it isn't always easy to discover the underlying problem. In some cases, the need to recover the namespace may be so that you can continue to get your work done -- not just cosmetics. – Smectic

better soln - https://mcmap.net/q/93411/-namespace-quot-stucked-quot-as-terminating-how-i-removed-it please select that as the answer for new clusters - – Speleology

You should state, that you can bypass the API authentication with curl if you use kubectl proxy. – Bobette

https://mcmap.net/q/94461/-kubernetes-namespaces-stuck-in-terminating-status – Marine

Answer 2

As mentioned before in this thread there is another way to terminate a namespace using API not exposed by kubectl by using a modern version of kubectl where kubectl replace --raw is available (not sure from which version). This way you will not have to spawn a kubectl proxy process and avoid dependency with curl (that in some environment like busybox is not available). In the hope that this will help someone else I left this here:

kubectl get namespace "stucked-namespace" -o json \
  | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/" \
  | kubectl replace --raw /api/v1/namespaces/stucked-namespace/finalize -f -

Moneychanger

this one worked for me in kubectl version v1.17, k8 v1.16 – Goldeneye

Great. Only thing that worked for us when there were no nodes up in the cluster. – Anneliese

Mine errors on "tr". The term tr is not recognized. – Verda

export NS=istio-system && kubectl get namespace "$NS" -o json \ | tr -d "\n" | sed "s/\"finalizers\": [[^]]\+]/\"finalizers\": []/" \ | kubectl replace --raw /api/v1/namespaces/$NS/finalize -f - – Vitrescent

doesnt work for me... – Schmeltzer

worked on AWS EKS 1.19 – Airfield

This one worked for me in Azure – Puttier

For those of you struggling with the tr and sed commands, the following will work as well if you have jq installed: shell kubectl get namespace $ns -o=json | jq 'del(.spec.finalizers)' | kubectl replace --raw /api/v1/namespaces/$ns finalize -f - Where $ns is the namespace to be updated – Unshapen

Answer 3

This is caused by resources still existing in the namespace that the namespace controller is unable to remove.

This command (with kubectl 1.11+) will show you what resources remain in the namespace:

kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>

Once you find those and resolve and remove them, the namespace will be cleaned up

Yardarm

This shows no resources on my cluster, I don't think is always the solution. – Kati

But definitely better to try first. Always force deleting the namespace is not good advice. – Cocci

I got here but am using powershell as my scripting environment (annoys me that this isn't mandatory information on answers) so to do the same thing is: kubectl api-resources --verbs=list --namespaced -o name | ForEach-Object { kubectl get $_ --show-kind --ignore-not-found -n <namespace> } – Rondure

Windows Powershell equivalent: Foreach($x in (kubectl api-resources --verbs=list --namespaced -o name)){ kubectl get --show-kind --ignore-not-found -n YOUR_NAMESPACE_HERE $x } – Middlings

This worked for me. I had a dangling CRD associated with that namespace. – Adrastus

What's the actual command for deleting the api resources found? It is not in the document. Thanks – Kerosene

I get error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request – Pavlish

get same error about metrics.k8s.io/v1beta1 – Brenner

In case you get an error about metrics.k8s.io/v1beta1, it probably means that you have deleted a namespace with a custom metrics apiresource (such as monitoring, or prometheus). Some other answers here explain how to fix this, which involve running kubectl delete apiservice/v1beta1.metrics.k8s.io – Ataman

Answer 4

Solution:

Use command below without any changes. it works like a charm.

NS=`kubectl get ns |grep Terminating | awk 'NR==1 {print $1}'` && kubectl get namespace "$NS" -o json   | tr -d "\n" | sed "s/\"finalizers\": \[[^]]\+\]/\"finalizers\": []/"   | kubectl replace --raw /api/v1/namespaces/$NS/finalize -f -

Enjoy

Groome

Not the best idea. This will most likely leave some dangling resources which caused the initial problem with termination. – Hydrolysate

Tried this and namespace still in "Terminating" status. – Hus

@Hus This command remove first terminating namespace in your list. Fore example if you have 3 namespace by terminating status you shold run this commant 3 times. If not work try another way to solve. – Groome

Note: If your kube/config contain token for autentication of the user maybe got error. You should coppy conf file from /etc/kubernetes by this command and replace it old to Config. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config – Groome

Answer 5

Need to remove the finalizer for kubernetes.

Step 1:

kubectl get namespace <YOUR_NAMESPACE> -o json > <YOUR_NAMESPACE>.json

• remove kubernetes from finalizers array which is under spec

Step 2:

kubectl replace --raw "/api/v1/namespaces/<YOUR_NAMESPACE>/finalize" -f ./<YOUR_NAMESPACE>.json

Step 3:

kubectl get namespace

You can see that the annoying namespace is gone.

Stipule

I'm missing this namespaces api. Is there a way to install it? – Verda

This can also be achieved with kubectl edit <stuck_resource> and delete the item in the 'finalizers' list. This has the advantage that it can be used to delete other stuck resources without knowing their api path. I used it to remove a stubborn CRD. – Contradistinction

You can also use a ui-based tool like Lens, which makes this and many other tasks/information very easy to perform/access. – Middlings

doesnt work for me.. – Schmeltzer

If you have jq installed, the following snippet might help you to remove the stuck namespace: namespace=stuck-namespace; kubectl get namespace $namespace -o json | jq '.spec.finalizers= []' | kubectl replace --raw "/api/v1/namespaces/$namespace/finalize" -f - – Protrusile

Answer 6

I loved this answer extracted from here It is just 2 commands.

In one terminal:

kubectl proxy

In another terminal:

kubectl get ns delete-me -o json | \
  jq '.spec.finalizers=[]' | \
  curl -X PUT http://localhost:8001/api/v1/namespaces/delete-me/finalize -H "Content-Type: application/json" --data @-

Thalamus

I tried a few here, this is the first one which worked - thank you! – Lichtenfeld

Removing the finalisers could lead to unregistered hanging resources later on. – Mustache

this worked for me – Kauppi

Interestingly this is the only solution that worked for me. – Noisemaker

This worked for me. If you're Mac, first install jq utility using the command brew install jq – Hube

Thank you. I was have issue removing the terminating NS. This helped me to get rid of my terminating rook-ceph namespace. – Karolekarolina

Only one that worked for me. – Impression

This worked for us. – Paginate

This is the only solution out of all the answers that worked for me. Thanks. – Metchnikoff

Answer 7

Single line command

kubectl patch ns <Namespace_to_delete> -p '{"metadata":{"finalizers":null}}'

Option : 2

If Patching not working you check the error if it's like in status

kubectl get ns <namespace-name>

Discovery failed for some groups, 1 failing: unable to retrieve the 22 complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently 23 unable to handle the request

You get the error details or API details which is failing for you, list available apiservice

kubectl get apiservice

Look for ones the AVAILABLE is False

kubectl delete apiservice <apiservice-name>

Example

kubectl delete apiservice metrics.k8s.io/v1beta1

and run above patch command again or wait for while namespace will get removed.

Read article for more : https://medium.com/@harsh.manvar111/kubernetes-namespace-stuck-on-terminating-state-25d0cda8e3ff

Extra - Simple trick

You can edit namespace on console only kubectl edit <namespace name> remove/delete "Kubernetes" from inside the finalizer section(Should be like "finalizers": [ ]) and press enter or save/apply changes.

in one step also you can do it.

Trick : 1

1. kubectl get namespace annoying-namespace-to-delete -o json > tmp.json

2. then edit tmp.json and remove "kubernetes" from Finalizers

3. Open another terminal Run command kubectl proxy and run below Curl

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://localhost:8001/api/v1/namespaces/<NAMESPACE NAME TO DELETE>/finalize

and it should delete your namespace.

Step by step guide

Start the proxy using command :

1. kubectl proxy

kubectl proxy & Starting to serve on 127.0.0.1:8001

find namespace

2. kubectl get ns

{Your namespace name} Terminating 1d

put it in file

3. kubectl get namespace {Your namespace name} -o json > tmp.json

edit the file tmp.json and remove the finalizers

}, "spec": { "finalizers": [ "kubernetes" ] },

after editing it should look like this

}, "spec": { "finalizers": [ ] },

we are almost there simply now run the curl with updating namespace value in it

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/{Your namespace name}/finalize

and it's gone

**

Dari

Do not remove finalizers github.com/kubernetes/kubernetes/issues/… – Variegation

you are the man. You've saved me in the "TLDR" part. :) – Extramundane

haha, Great glad to know that..! – Dari

Answer 8

For us it was the metrics-server crashing.

So to check if this is relevant to you'r case with the following run: kubectl api-resources

If you get

error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

Then its probably the same issue

Credits goes to @javierprovecho here

Cale

for us it was a service catalogue CRD leftover: servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request – Fortuitous

And what do you do? @PawełPrażak remove it? Or how do I find it? – Lenorelenox

@Lenorelenox AFAIR we removed it CR to solve the issue, but I can't remember how we've figured it out, sorry – Fortuitous

You can deploy the latest (0.4.1) version of metrics-server to resolve this: kubectl apply -f github.com/kubernetes-sigs/metrics-server/releases/download/…; – Kabob

@Lenorelenox at that point you should indeed remove that resource. as mentioned in another answer run kubectl api-resources --verbs=list --namespaced -o name \ | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace> to find it; I then removed it with kubectl delete APIService v1beta1.custom.metrics.k8s.io – Western

Similarly, kubectl get apiservice|grep False led to kubectl delete APIService v1beta1.metrics.k8s.io and resolution for me. – Wootten

Hit this when trying to delete a namespace with an empty node pool so that deploy/metrics-server in kube-system was not running. After scaling up to one node, enough to run this metrics pod, the namespace finally went away as expected. – Intermarry

Answer 9

I write simple script to delete your stucking namespace based on @Shreyangi Saxena 's solution.

cat > delete_stuck_ns.sh << "EOF"
#!/usr/bin/env bash

function delete_namespace () {
    echo "Deleting namespace $1"
    kubectl get namespace $1 -o json > tmp.json
    sed -i 's/"kubernetes"//g' tmp.json
    kubectl replace --raw "/api/v1/namespaces/$1/finalize" -f ./tmp.json
    rm ./tmp.json
}

TERMINATING_NS=$(kubectl get ns | awk '$2=="Terminating" {print $1}')

for ns in $TERMINATING_NS
do
    delete_namespace $ns
done
EOF

chmod +x delete_stuck_ns.sh

This Script can detect all namespaces in Terminating state, and delete it.

---

PS:

• This may not work in MacOS, cause the native sed in macos is not compatible with GNU sed.

  you may need install GNU sed in your MacOS, refer to this answer.

• Please confirm that you can access your kubernetes cluster through command kubectl.

• Has been tested on kubernetes version v1.15.3

---

Update

I found a easier solution:

kubectl patch RESOURCE NAME -p '{"metadata":{"finalizers":[]}}' --type=merge

Doti

A very nice, easy and working solution. Good work and thanks! – Lapful

Do not remove the finalizers to force a deletion as some resources will most likely remain dangling. – Mustache

Answer 10

I've written a one-liner Python3 script based on the common answers here. This script removes the finalizers in the problematic namespace.

python3 -c "namespace='<my-namespace>';import atexit,subprocess,json,requests,sys;proxy_process = subprocess.Popen(['kubectl', 'proxy']);atexit.register(proxy_process.kill);p = subprocess.Popen(['kubectl', 'get', 'namespace', namespace, '-o', 'json'], stdout=subprocess.PIPE);p.wait();data = json.load(p.stdout);data['spec']['finalizers'] = [];requests.put('http://127.0.0.1:8001/api/v1/namespaces/{}/finalize'.format(namespace), json=data).raise_for_status()"

💡 rename namespace='<my-namespace>' with your namespace. e.g. namespace='trust'

---

Full script: https://gist.github.com/jossef/a563f8651ec52ad03a243dec539b333d

Swiftlet

This worked! I finally was able to remove stuck namespaces! – Aguiar

Woah that worked after a few other methods didn't, including deleting the finalizer myself by editing! Thank you! – Thirtythree

This worked! I finally was able to remove stuck namespaces! – Comprise

This is much easier answer! – Orpha

This worked on me too! – Prevalent

Finallyyyyy - this script works!! – Euphuism

Answer 11

here is a (yet another) solution. This uses jq to remove the finalisers block from the json, and does not require kubectl proxy:

namespaceToDelete=blah

kubectl get namespace "$namespaceToDelete" -o json \
  | jq 'del(.spec.finalizers)' \
  | kubectl replace --raw /api/v1/namespaces/$namespaceToDelete/finalize -f -

Polytrophic

It worked on k8 v1.24.6 – Karolekarolina

it works on k8s 1.20.x .. and it's the only one which works – Ague

Answer 12

Run kubectl get apiservice

For the above command you will find an apiservice with Available Flag=Flase.

So, just delete that apiservice using kubectl delete apiservice <apiservice name>

After doing this, the namespace with terminating status will disappear.

Hermaphroditus

In my case, all are Available=True, so this does not help (at least not always) – Bunyan

Was getting a 404 with the other described methods. This solved my issue – Tilley

What is next if all kubectl get apiservice response showing "true" only – Kurman

Answer 13

Forcefully deleting the namespace or removing finalizers is definitely not the way to go since it could leave resources registered to a non existing namespace.

This is often fine but then one day you won't be able to create a resource because it is still dangling somewhere.

The upcoming Kubernetes version 1.16 should give more insights into namespaces finalizers, for now I would rely on identification strategies. A cool script which tries to automate these is: https://github.com/thyarles/knsk

However it works across all namespaces and it could be dangerous. The solution it s based on is: https://github.com/kubernetes/kubernetes/issues/60807#issuecomment-524772920

tl;dr

1. Checking if any apiservice is unavailable and hence doesn't serve its resources: kubectl get apiservice|grep False
2. Finding all resources that still exist via kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n $your-ns-to-delete

(credit: https://github.com/kubernetes/kubernetes/issues/60807#issuecomment-524772920)

Mustache

Agree, definitely not the way to go. Stuck removal is most often due to some resources not being properly deleted thus blocking the namespace termination. Check and remove those and the namespace will finally be removed. It's worth to mention that the namespace stuck in 'terminating' will block another namespace removal. At least this is what has happened to me, looks like removal is sequential in GKE. – Hydrolysate

Answer 14

1. Using Curl Command

Issue Mentioned: https://amalgjose.com/2021/07/28/how-to-manually-delete-a-kubernetes-namespace-stuck-in-terminating-state/

export NAMESPACE=<specifice-namespace>
kubectl get namespace $NAMESPACE -o json > tempfile.json

Edit the JSON file and remove all values from spec.finalizers

Save it and then apply this command on separate tab (Must be open in separate Tab)

kubectl proxy

And run this command on same tab:

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tempfile.json http://127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize

Check namespace if terminating namespace is removed or not

kubectl get namespaces

2. Using Kubectl Command

Issue Mentioned: https://aws.amazon.com/premiumsupport/knowledge-center/eks- terminated-namespaces/

1. Save a JSON file similar to the following:

export NAMESPACE=<specific-namespace>
kubectl get namespace $NAMESPACE -o json > tempfile.json

2. Edit the JSON file and remove all values from spec.finalizers
3. To apply the changes, run a command similar to the following:

kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f ./tempfile.json

4. Verify that the terminating namespace is removed:

kubectl get namespaces

Prosopopoeia

Answer 15

In my case the problem was caused by a custom metrics.

To know what is causing the issue, just run this command:

kubectl api-resources | grep -i false

That should give you which api resources are causing the problem. Once identified just delete it:

kubectl delete apiservice v1beta1.custom.metrics.k8s.io

Once deleted, the namespace should disappear.

Quickel

false was lowercase in my case. so maybe: kubectl api-resources | grep -i false – Arch

Nice. kubectl delete APIService v1beta1.metrics.k8s.io worked. – Noisemaker

Answer 16

Please try with below command:

kubectl patch ns <your_namespace> -p '{"metadata":{"finalizers":null}}'

Huberty

From Review: Command/Code-only answers are discouraged on Stack Overflow because they don't explain how it solves the problem. Please edit your answer to explain what this code does and how it answers the question, so that it is useful to the OP as well as other users with similar issues. See: How do I write a good answer?. Thanks – Pragmatist

Only this one worked for me. No other solutions using POST json to finalize api work. – Venturous

Answer 17

I tried 3-5 options to remove ns, but only this one works for me.

This sh file will remove all namespaces with Terminating status

$ vi force-delete-namespaces.sh

$ chmod +x force-delete-namespaces.sh

$ ./force-delete-namespaces.sh

#!/usr/bin/env bash

set -e
set -o pipefail

kubectl proxy &
proxy_pid="$!"
trap 'kill "$proxy_pid"' EXIT

for ns in $(kubectl get namespace --field-selector=status.phase=Terminating --output=jsonpath="{.items[*].metadata.name}"); do
    echo "Removing finalizers from namespace '$ns'..."
    curl -H "Content-Type: application/json" -X PUT "127.0.0.1:8001/api/v1/namespaces/$ns/finalize" -d @- \
        < <(kubectl get namespace "$ns" --output=json | jq '.spec = { "finalizers": [] }')

    echo
    echo "Force-deleting namespace '$ns'..."
    kubectl delete namespace "$ns" --force --grace-period=0 --ignore-not-found=true
done

Selfjustifying

This is the best script to delete all stuck namespace kubernetes. A great time saver! Thank you! – Icbm

trap "kill $proxy_pid" SIGINT SIGTERM ERR EXIT is working, your single quotes aren't :) – Icbm

Answer 18

Replace ambassador with your namespace

Check if the namespace is stuck

kubectl get ns ambassador

NAME         STATUS        AGE
ambassador   Terminating   110d

This is stuck from a long time

Open a admin terminal/cmd prompt or powershell and run

kubectl proxy

This will start a local web server: Starting to serve on 127.0.0.1:8001

Open another terminal and run

kubectl get ns ambassador -o json >tmp.json

edit the tmp.json using vi or nano

from this

{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
    "annotations": {
        "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"ambassador\"}}\n"
    },
    "creationTimestamp": "2021-01-07T18:23:28Z",
    "deletionTimestamp": "2021-04-28T06:43:41Z",
    "name": "ambassador",
    "resourceVersion": "14572382",
    "selfLink": "/api/v1/namespaces/ambassador",
    "uid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
"spec": {
    "finalizers": [
        "kubernetes"
    ]
},
"status": {
    "conditions": [
        {
            "lastTransitionTime": "2021-04-28T06:43:46Z",
            "message": "Discovery failed for some groups, 3 failing: unable to retrieve the complete list of server APIs: compose.docker.com/v1alpha3: an error on the server (\"Internal Server Error: \\\"/apis/compose.docker.com/v1alpha3?timeout=32s\\\": Post https://0.0.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews: write tcp 0.0.0.0:53284-\u0026gt;0.0.0.0:443: write: broken pipe\") has prevented the request from succeeding, compose.docker.com/v1beta1: an error on the server (\"Internal Server Error: \\\"/apis/compose.docker.com/v1beta1?timeout=32s\\\": Post https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews: write tcp 0.0.0.0:5284-\u0026gt;10.96.0.1:443: write: broken pipe\") has prevented the request from succeeding, compose.docker.com/v1beta2: an error on the server (\"Internal Server Error: \\\"/apis/compose.docker.com/v1beta2?timeout=32s\\\": Post https://0.0.0.0:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews: write tcp 1.1.1.1:2284-\u0026gt;0.0.0.0:443: write: broken pipe\") has prevented the request from succeeding",
            "reason": "DiscoveryFailed",
            "status": "True",
            "type": "NamespaceDeletionDiscoveryFailure"
        },
        {
            "lastTransitionTime": "2021-04-28T06:43:49Z",
            "message": "All legacy kube types successfully parsed",
            "reason": "ParsedGroupVersions",
            "status": "False",
            "type": "NamespaceDeletionGroupVersionParsingFailure"
        },
        {
            "lastTransitionTime": "2021-04-28T06:43:49Z",
            "message": "All content successfully deleted",
            "reason": "ContentDeleted",
            "status": "False",
            "type": "NamespaceDeletionContentFailure"
        }
    ],
    "phase": "Terminating"
}

}

to

    {
  "apiVersion": "v1",
  "kind": "Namespace",
  "metadata": {
    "annotations": {
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"ambassador\"}}\n"
    },
    "creationTimestamp": "2021-01-07T18:23:28Z",
    "deletionTimestamp": "2021-04-28T06:43:41Z",
    "name": "ambassador",
    "resourceVersion": "14572382",
    "selfLink": "/api/v1/namespaces/ambassador",
    "uid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  },
  "spec": {
    "finalizers": []
  }
}

by deleting status and kubernetes inside finalizers

Now use the command and replace ambassador with your namespace

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/ambassador/finalize

you will see another json like before then run

then run the command

 kubectl get ns ambassador
Error from server (NotFound): namespaces "ambassador" not found

If it still says terminating or any other error make sure you format your json in a proper way and try the steps again.

Noctilucent

Answer 19

The only way I found to remove a "terminating" namespace is by deleting the entry inside the "finalizers" section. I've tried to --force delete it and to --grace-period=0 none of them worked, however, this method did:

on a command line display the info from the namespace:

$ kubectl get namespace your-rogue-namespace -o yaml

This will give you yaml output, look for a line that looks similar to this:

deletionTimestamp: 2018-09-17T13:00:10Z
  finalizers:
  - Whatever content it might be here...
  labels:

Then simply edit the namespace configuration and delete the items inside that finalizers container.

$ kubectl edit namespace your-rogue-namespace

This will open an editor (in my case VI), went over the line I wanted to delete and deleted it, I pressed the D key twice to delete the whole line.

Save it, quit your editor, and like magic. The rogue-namespace should be gone.

And to confirm it just:

$ kubectl get namespace your-rogue-namespace -o yaml

Buddhism

this short-circuits cleanup of resources associated with the namespace and leaves orphaned resources associated with that namespace – Yardarm

it might be crashing/missing metrics API server or service catalogue API server, see answers from JordanLiggitt and AntonioGomezAlvarado – Fortuitous

Answer 20

There are a couple of things you can run. But what this usually means, is that the automatic deletion of namespace was not able to finish, and there is a process running that has to be manually deleted. To find this you can do these things:

Get all prossesse attached to the name space. If this does not result in anything move on to next suggestions

$ kubectl get all -n your-namespace

Some namespaces have apiserivces attached to them and it can be troublesome to delete. This can for that matter be whatever resources you want. Then you delete that resource if it finds anything

$ kubectl get apiservice|grep False

But the main takeaway, is that there might be some things that is not completly removed. So you can see what you initially had in that namespace, and then see what things is spun up with your YAMLs to see the processes up. Or you can start to google why wont service X be properly removed, and you will find things.

Lenorelenox

Answer 21

1. Run the following command to view the namespaces that are stuck in the Terminating state:

   kubectl get namespaces

2. Select a terminating namespace and view the contents of the namespace to find out the finalizer. Run the following command:

   kubectl get namespace -o yaml

3. Your YAML contents might resemble the following output:

        apiVersion: v1
        kind: Namespace
        metadata:
           creationTimestamp: 2019-12-25T17:38:32Z
           deletionTimestamp: 2019-12-25T17:51:34Z
           name: <terminating-namespace>
           resourceVersion: "4779875"
           selfLink: /api/v1/namespaces/<terminating-namespace>
           uid: ******-****-****-****-fa1dfgerz5
         spec:
           finalizers:
           - kubernetes
         status:
           phase: Terminating

4. Run the following command to create a temporary JSON file:

   kubectl get namespace -o json >tmp.json

5. Edit your tmp.json file. Remove the kubernetes value from the finalizers field and save the file. Output would be like:

    {
        "apiVersion": "v1",
        "kind": "Namespace",
        "metadata": {
            "creationTimestamp": "2018-11-19T18:48:30Z",
            "deletionTimestamp": "2018-11-19T18:59:36Z",
            "name": "<terminating-namespace>",
            "resourceVersion": "1385077",
            "selfLink": "/api/v1/namespaces/<terminating-namespace>",
            "uid": "b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5"
        },
        "spec": {
        },

        "status": {
            "phase": "Terminating"
        }
    }

6. To set a temporary proxy IP and port, run the following command. Be sure to keep your terminal window open until you delete the stuck namespace:

   kubectl proxy

7. Your proxy IP and port might resemble the following output:

   Starting to serve on 127.0.0.1:8001

8. From a new terminal window, make an API call with your temporary proxy IP and port:

  curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/your_terminating_namespace/finalize

Your output would be like:

    {
       "kind": "Namespace",
       "apiVersion": "v1",
       "metadata": {
         "name": "<terminating-namespace>",
         "selfLink": "/api/v1/namespaces/<terminating-namespace>/finalize",
         "uid": "b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5",
         "resourceVersion": "1602981",
         "creationTimestamp": "2018-11-19T18:48:30Z",
         "deletionTimestamp": "2018-11-19T18:59:36Z"
       },
       "spec": {

       },
       "status": {
         "phase": "Terminating"
       }
    }

9. The finalizer parameter is removed. Now verify that the terminating namespace is removed, run the following command:

   kubectl get namespaces

Egress

Answer 22

Edit: It is not recommended to remove finalizers. Correct approach would be:

• Delete all the resources in the namespace.

Github issue link

My usual workspace is a small k8s cluster which I frequently destroy and rebuild it back, and that's why removing finalizers method works for me.

Original answer: I usually run into same problem.

This is what I do

kubectl get ns your-namespace -o json > ns-without-finalizers.json

Edit ns-without-finalizers.json. replace all finalizers with empty array.

Run kubectl proxy ( usually run it on another terminal )

Then curl this command

curl -X PUT http://localhost:8001/api/v1/namespaces/your-namespace/finalize -H "Content-Type: application/json" --data @ns-without-finalizers.json

Felder

Answer 23

For anyone looking for few commands for later version of Kubernetes, this helped me.

NAMESPACE=mynamespace
kubectl get namespace $NAMESPACE -o json | sed 's/"kubernetes"//' | kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f -

Tested in Kubernetes v1.24.1

Note: the command replaces all kubernetes strings in the json, please use it with care

Mana

This replaces all "kubernetes" strings in the json. Looks scary to me. – Vernita

Answer 24

If the namespace stuck in Terminating while the resources in that namespace have been already deleted, you can patch the finalizers of the namespace before deleting it:

kubectl patch ns ns_to_be_deleted -p '{"metadata":{"finalizers":null}}';

then

kubectl delete ns ns_to_be_deleted;

Edit:

Please check @Antonio Gomez Alvarado's Answer first. The root cause could be the metrics server that mentioned in that answer.

Kabob

Answer 25

Completing the already great answer by nobar. If you deployed your cluster with Rancher there is a caveat.

Rancher deployments change EVERY api call, prepending /k8s/clusters/c-XXXXX/ to the URLs.

The id of the cluster on rancher (c-XXXXX) is something you can easily get from the Rancher UI, as it will be there on the URL.

So after you get that cluster id c-xxxx, just do as nobar says, just changing the api call including that rancher bit.

(
NAMESPACE=your-rogue-namespace
kubectl proxy &
kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
curl -k -H "Content-Type: application/json" \
  -X PUT --data-binary @temp.json \
  127.0.0.1:8001/k8s/clusters/c-XXXXX/api/v1/namespaces/$NAMESPACE/finalize
)

Anglocatholic

Answer 26

Something similar happened to me in my case it was pv & pvc , which I forcefully removed by setting finalizers to null. Check if you could do similar with ns

kubectl patch pvc <pvc-name> -p '{"metadata":{"finalizers":null}}'

For namespaces it'd be

kubectl patch ns <ns-name> -p '{"spec":{"finalizers":null}}'

Uglify

Answer 27

Debugging a similar issue.

Two important things to consider:

1 ) Think twice before deleting finalizers from your namespace because there might be resources that you wouldn't want to automatically delete or at least understand what was deleted for troubleshooting.

2 ) Commands like kubectl api-resources --verbs=list might not give you resources that were created by external crds.

---

In my case:

I viewed my namespace real state (that was stuck on Terminating) with kubectl edit ns <ns-name> and under status -> conditions I saw that some external crds that I installed were failed to be deleted because they add a finalizers defined:

 - lastTransitionTime: "2021-06-14T11:14:47Z"
    message: 'Some content in the namespace has finalizers remaining: finalizer.stackinstall.crossplane.io
      in 1 resource instances, finalizer.stacks.crossplane.io in 1 resource instances'
    reason: SomeFinalizersRemain
    status: "True"
    type: NamespaceFinalizersRemaining

Ruffle

Answer 28

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json 127.0.0.1:8001/k8s/clusters/c-mzplp/api/v1/namespaces/rook-ceph/finalize

This worked for me, the namespace is gone.

Detailed explanation can be found in the link https://github.com/rook/rook/blob/master/Documentation/ceph-teardown.md.

This happened when I interrupted kubernetes installation(Armory Minnaker). Then I proceeded to delete the namespace and reinstall it. I was stuck with pod in terminating status due to finalizers. I got the namespace into tmp.json, removed finalizers from tmp.json file and did the curl command. Once I get past this issue, I used scripts for uninstalling the cluster to remove the residues and did a reinstallation.

Hatti

perhaps you would like to break down your URL and explain what is going on? – Buddhism

Answer 29

kubectl edit namespace ${stucked_namespace}

Then delete finalizers in vi mode and save.

It worked in my case.

Blackberry

Answer 30

Editing NS yaml manually didn't work for me, no error was thrown on editing but changes did not take effect.

This worked for me:

In one session:

kubectl proxy

in another shell:

kubectl get ns <rouge-ns> -o json | jq '.spec.finalizers=[]' | curl -X PUT http://localhost:8001/api/v1/namespaces/<rouge-ns>/finalize -H "Content-Type: application/json" --data @-

source: https://virtual-simon.co.uk/vsphere-kubernetes-force-deleting-stuck-terminating-namespaces-and-contexts/

Volny

Answer 31

I had similar problem with metrics-server, after to execute kubectl get namespace <NAMESPACE-HERE> -o json, I found this message:

"Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1?timeout=32s\": Unauthorized") has prevented the request from succeeding

I had installed an old version of metrics-server on this cluster. I only removed these resources. As it was install with:

kubectl apply -f metrics-server-0.3.7/deploy/1.8+/ --dry-run -o yaml | kubectl apply -f -

This command removed all of them:

kubectl apply -f metrics-server-0.3.7/deploy/1.8+/ --dry-run -o yaml | kubectl delete -f -

Finally, the namespace disappeared after some seconds.

I assume, if you have a more recent versions, you can delete with:

1. YAML installation (here):

kubectl delete -f https://github.com/kubernetes-sigs/metrics-server/releases/download/<VERSION-HERE>/components.yaml

2. Helm chart installation (here)

helm uninstall metrics-server

Don't forget to install this again with the correct version.

Margy

Answer 32

My attempt with a script at kubectl version v1.26.3 and jq:

remove_finalizers.sh

#!/usr/bin/bash

set -x

function rmfin() {
  for e in $(kubectl get $1 -o name); do
    kubectl get $e -o json | jq '.metadata.finalizers = []' | kubectl replace $e -f -
  done
}

rmfin $*

Execution

$ sh remove_finalizers ns

Kirstinkirstyn

Answer 33

This script deletes all Kubernetes resources blocking the deletion of a specified namespace. It first removes the finalizers of each resource, then deletes the resource itself. Finally, it deletes the specified namespace.

#!/bin/bash
#
# This script deletes all Kubernetes resources blocking the deletion of a specified namespace.
# It first removes the finalizers of each resource, then deletes the resource itself.
# Finally, it deletes the specified namespace.
#
# Usage: ./delete_namespace_resources.sh [OPTIONS]
# Options:
#   -n, --namespace <terminating-namespace>   Namespace to be deleted
#   -h, --help                               Display this help and exit
#

set -euo pipefail

# log messages
log() {
    local message=$1
    echo "[INFO] $message"
}

delete_finalizers() {
    local resource_name=$1
    local namespace=$2

    # Get the resource with finalizers removed
    log "Deleting finalizers of $resource_name"
    kubectl patch "$resource_name" -n "$namespace" --type='json' -p='[{"op": "remove", "path": "/metadata/finalizers"}]'
}

delete_resource() {
    local resource_name=$1
    local namespace=$2

    # Delete the resource
    log "Deleting resource: $resource_name in namespace: $namespace"
    kubectl delete "$resource_name" -n "$namespace" --ignore-not-found
}

delete_blocking_resources() {
    local terminating_namespace=$1

    # Get all Kubernetes resources in the namespace
    resources=$(kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n "$terminating_namespace" | tail -n +2)

    # Loop through each resource
    while read -r resource; do
        local resource_name=$(echo "$resource" | awk '{print $1}')

        # Delete finalizers of the resource
        delete_finalizers "$resource_name" "$terminating_namespace"

        # Delete the resource itself
        delete_resource "$resource_name" "$terminating_namespace"
    done <<< "$resources"
}

display_help() {
    cat <<EOF
Usage: $0 [OPTIONS]
Options:
  -n, --namespace <terminating-namespace>   Namespace to be deleted
  -h, --help                               Display this help and exit
EOF
}

# Parse command line options
while [[ $# -gt 0 ]]; do
    key="$1"

    case $key in
        -n|--namespace)
            TERMINATING_NAMESPACE="$2"
            shift
            shift
            ;;
        -h|--help)
            display_help
            exit 0
            ;;
        *)
            echo "Unknown option: $1" >&2
            display_help
            exit 1
            ;;
    esac
done

# Check if terminating namespace is provided
if [ -z "${TERMINATING_NAMESPACE:-}" ]; then
    echo "Error: Terminating namespace not provided." >&2
    display_help
    exit 1
fi

# Delete resources blocking namespace deletion
delete_blocking_resources "$TERMINATING_NAMESPACE"

# Delete the namespace
log "Deleting namespace $TERMINATING_NAMESPACE"
kubectl delete namespace "$TERMINATING_NAMESPACE"
log "Namespace $TERMINATING_NAMESPACE successfully deleted"

Tequila

Answer 34

The simplest and most easiest way of doing this is copying this bash script

#!/bin/bash

###############################################################################
# Copyright (c) 2018 Red Hat Inc
#
# See the NOTICE file(s) distributed with this work for additional
# information regarding copyright ownership.
#
# This program and the accompanying materials are made available under the
# terms of the Eclipse Public License 2.0 which is available at
# http://www.eclipse.org/legal/epl-2.0
#
# SPDX-License-Identifier: EPL-2.0
###############################################################################

set -eo pipefail

die() { echo "$*" 1>&2 ; exit 1; }

need() {
    which "$1" &>/dev/null || die "Binary '$1' is missing but required"
}

# checking pre-reqs

need "jq"
need "curl"
need "kubectl"

PROJECT="$1"
shift

test -n "$PROJECT" || die "Missing arguments: kill-ns <namespace>"

kubectl proxy &>/dev/null &
PROXY_PID=$!
killproxy () {
    kill $PROXY_PID
}
trap killproxy EXIT

sleep 1 # give the proxy a second

kubectl get namespace "$PROJECT" -o json | jq 'del(.spec.finalizers[] | select("kubernetes"))' | curl -s -k -H "Content-Type: application/json" -X PUT -o /dev/null --data-binary @- http://localhost:8001/api/v1/namespaces/$PROJECT/finalize && echo "Killed namespace: $PROJECT"

# proxy will get killed by the trap

Add the above code in the deletenamepsace.sh file.

And then execute it by providing namespace as parameter(linkerd is the namespace i wanted to delete here)

➜ kubectl get namespaces
linkerd           Terminating   11d

➜ sh deletenamepsace.sh linkerd
Killed namespace: linkerd

➜ kubectl get namespaces

The above tip has worked for me.

Honestly i think kubectl delete namespace mynamespace --grace-period=0 --force is not at all worth trying.

Special Thanks to Jens Reimann! I think this script should be incorporated in kubectl commands.

Wirework

sure have copied the solution here – Wirework

Answer 35

Delete all the resources listed by:

kubectl delete -n YOURNAMESPACE —-all

Use kubectl delete -n YOURNAMESPACE <resource> <id> or (if you copy paste from the above output) kubectl delete -n YOURNAMESPACE <resource>/<id>, for each resource that you see listed there.

You can also do it at once kubectl delete -n YOURNAMESPACE <resource>/<id1> <resource>/<id2> <resource2>/<id3> <resource2>/<id4> <resource3>/<id5> etc..

Probably you tried to remove resources but they are getting recreated because of the deployment or replicaset resource, preventing the namespace from freeing up depending resources and from being cleaned up.

Archaimbaud