Project deletion struck in Terminating

Question

Openshift v3.11.0+0cbc58b

I am trying to delete project from openshift as admin user, but project get struck in terminating state.

I have tried running the following command but no use,

oc delete project testing

Error from server (Conflict): Operation cannot be fulfilled on namespaces "testing": The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system.

I have checked the previous posts but I didn't get any solution. Can anyone help me to solve this.

Scincoid

The project likely contains a resource object which has a finalizer. You will need to work out what the resource is, delete the finalizer from the object so it is deleted, and then the project will be deleted. This can occur when using operators and the operator which owned the resource was deleted before the resource was. – Digress

Thanks @GrahamDumpleton I will check on it – Scincoid

I'd refer to github issue for more information regarding this topic. The finalizer is there for a specific reason and you should rather search for the root cause there. – Rambo

Answer 1

When getting such error on oc delete namespace:

Error from server (Conflict): Operation cannot be fulfilled on namespaces

You can try to delete the namespace with a curl command to Openshift API:

1. Get namespace status into json file:

oc get namespace $NS_TO_DELETE -o json > tmp_ns.json

2. Remove the line with "kubernetes" (under "finalizers"):

vi tmp_ns.json

3. Open proxy connection to your cluster (as a background process)

oc proxy &

4. Push the json content to your cluster with curl:

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp_ns.json http://127.0.0.1:8001/api/v1/namespaces/$NS_TO_DELETE/finalize

5. your-namespace should be gone now:

oc get namespace $NS_TO_DELETE # namespaces not found

6. Switch to the background process (fg) of the Proxy and stop it (Ctrl+c), or directly with:

kill -9 %%

Note: If you have jq installed, you can run steps 1 & 2 by: oc get namespace $NS_TO_DELETE -o json | jq '.spec = {"finalizers":[]}' > tmp_ns.json

Tiernan

Thank you so much! this worked like a charm! if only i could give you more than one upvote – Emelinaemeline

@meaningqo, glad it helped you. Note that this is only a workaround, but it is important to find what caused the namespace delete failure originally. – Tiernan

yes. we figured the problem out but after we have already flagged the namespace for deletion. the issue was that there were still custom resources for an operator on the namespace but with the operator already removed there was no way to delete the custom resources and no way to reinstall the operator. so removing finalizer, deleting namespace, recreating namespace and deleting leftover resources was the way to go. thank you again – Emelinaemeline

Many thanks, really appreciated this – Victor

Answer 2

As answered earlier - your namespace contains some resources that are hanging around for a finalizer. For a brute force approach to getting rid of them (which I'm assuming is the plan if you're deleting the namespace), you can try:

oc api-resources --verbs=list --namespaced -o name | xargs -n 1 oc get --show-kind --ignore-not-found -n <your-namespace>

This will list the resources that are still hanging around. Command might take a while to provide you with the resources, depending on what's going on in your cluster.

Then oc edit <resource> -n <namespace> and delete the finalizers section. Save your changes, and repeat for the rest of the listed resources.

Use with caution, though. More knowledgeable people than me will be able to tell you what actually happens to those resources.

Cruet

Great answer, Thank you! – Pastorale

Deleting that finalizer section from the piece that was stuck worked for me too. Nice job! – Imprimis

Answer 3

The namespace contains some objects that need a finalizer. If you do not care about the objects in the namespace, you can do:

oc delete namespace --now=true testing

From the documentation, this is the same as --grace-period=1.

Pigtail

Unfortunately this is not solving the issue, as the command results into following error message: Error from server (Conflict): Operation cannot be fulfilled on namespaces "NAMESPACE_NAME": The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system. – Rambo

Answer 4

oc patch checluster codeready-workspaces -p '{ "metadata": { "finalizers": null }}' --type merge -n=$NAMESAPCE-NAME

che cluster will be deleted and project or namespace will get deleted (will not rest in terminate state)

Canterbury