How do you debug python code with kubernetes and skaffold?

Question

I am currently running a django app under python3 through kubernetes by going through skaffold dev. I have hot reload working with the Python source code. Is it currently possible to do interactive debugging with python on kubernetes?

For example,

def index(request):
    import pdb; pdb.set_trace()
    return render(request, 'index.html', {})

Usually, outside a container, hitting the endpoint will drop me in the (pdb) shell.

In the current setup, I have set stdin and tty to true in the Deployment file. The code does stop at the breakpoint but it doesn't give me access to the (pdb) shell.

Jarmon

Answer 1

There is a kubectl command that allows you to attach to a running container in a pod:

kubectl attach <pod-name> -c <container-name> [-n namespace] -i -t

-i  (default:false) Pass stdin to the container
-t  (default:false) Stdin is a TTY

It should allow you to interact with the debugger in the container. Probably you may need to adjust your pod to use a debugger, so the following article might be helpful:

• How to use PDB inside a docker container.

There is also telepresence tool that helps you to use different approach of application debugging:

Using telepresence allows you to use custom tools, such as a debugger and IDE, for a local service and provides the service full access to ConfigMap, secrets, and the services running on the remote cluster.

Use the --swap-deployment option to swap an existing deployment with the Telepresence proxy. Swapping allows you to run a service locally and connect to the remote Kubernetes cluster. The services in the remote cluster can now access the locally running instance.

Meantime

Answer 2

It might be worth looking into Rookout which allows in-prod live debugging of Python on Kubernetes pods without restarts or redeploys. You lose path-forcing etc but you gain loads of flexibility for effectively simulating breakpoint-type stack traces on the fly.

Crippling

Answer 3

This doesn't use Skaffold, but you can attach the VSCode debugger to any running Python pod with an open source project I wrote.

There is some setup involved to install it on your cluster, but after installation you can debug any pod with one command:

robusta playbooks trigger python_debugger name=myapp namespace=default 

Weeny

Answer 4

You can take a look at okteto/okteto. There's a good tutorial which explains how you can develop and debug directly on Kubernetes.

Lucey