I'd like to have the ability to remote debug a Spark job written in Scala running in a docker container with VsCode. This is what I have so far,
- VsCode with scala-metals v0.8 installed
- This debug launch configuration
- I spin up a spark cluster with docker-compose up
- I submit a spark job to the cluster started above with the following command
docker exec -it -e SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=4000 -w /spark spark-job-starter_master_1 bin/spark-submit --class example.StarterSparkJob /build/example.jar
After doing so I see the output Listening for transport dt_socket at address: 4000
- I attempt to launch the debugger from VsCode, but I get the error.
Debugger failed to attach: handshake failed - received >Content-Length< - expected >JDWP-Handshake<
The full details of the code are here https://github.com/aedenj/spark-job-starter/tree/vscode-debug-setup. Your guidance is appreciated.