How to solve nvidia-smi command stuck and not showing anything?

Question

My server do not response to nvidia-smi after I use ctrl+c kill the process running my GPU-training code.
Before today, when I tap ctrl+c, the process first shows keyboard interrupt and then killed by linux.
But today, when I use ctrl+c, it response with keyboard interrupt but is not killed.
After I kill this process withkill -9 <pid>, I can not use nvidia-smi anymore and I cant train my code.
How to solve this, please.

Shrine

Did you find an answer? – Pekin