Vertex AI pipelines import custom modules

Question

I'm developing a complex pipeline in Vertex AI using Pipelines and components. I would like to import some custom modules and functions I developed for this use case. Unfortunately, I cannot figure out how to import those custom functions in my code without creating ad-hoc Docker images or without publishing my code on public repositories like PyPi.

There are two pain points in pasting those custom functions' code in each component:

1. The code becomes huge and difficult read
2. The function's code completely loses the maintenability because at each small change, I have to replicate it for each component.

Maharajah

There are two ways to use Custom modules, one is using a custom container and the other one is uploading it into a public repository and install the module into vertex. – Orlantha

@JoseGutierrezPaliza both ways are not very fast and comfortable during the development phase. I agree that when the code is ready and tested, the smartest way is to containerize the component. But, during the development phase is very boring and time-wasting. – Maharajah

Answer 1

KFP supports Function-based components which can run a standalone function in a generic python image.

Rhythmical

Yes, I've already implemented by pipeline using function-based components but the main problem is the size of these components. In each component, I have to include a lot of custom support functions, and my question is: is there a method to systematically import that functions in each component without creating a PyPi repository? – Maharajah

I see why function-based components would be unappealing. Wrapping all your code in a single image and uploading it to Artifact Registry would be the most straightforward solution in that case. You would then create component specs with different python commands to execute each function, all pointed at the same image. – Rhythmical