How to use Cloud Composer to schedule Jupyter Notebooks on a Compute Engine VM?

Question

I have a series of notebooks I want to execute each weekday, each of which is dependent on the previous one, and I'd like to have this entire process automated with Cloud Composer.

I'm familiar with the Papermill library, so that part isn't an issue. What I am having issues with is how to have the DAG in the bucket communicate with the virtual machine instance on Compute Engine.

Any help is appreciated, thanks!

Goldarned

Answer 1

Composer allows you to interact with its GCP Virtual Machines through the python operatorby using the Compute Engine Pyhton API.

You could have a startup script that makes use of Papermill in order to interact with the notebook(s) every time you start your machine (and use Composer in order to schedule when should these be stopped or started).

Would this work for you? Is there anything about your use-case that would make this not a feasible option?

Revival