Can you get a static external IP address for Google Cloud Composer / Airflow?

Question

I know how to assign a static external IP address to a Compute Engine, but can this be done with Google Cloud Composer (Airflow)? I'd imagine most companies need that functionality since they'd generally be writing back to a warehouse that is possibly behind a firewall, but I can't find any doc's on how to do this.

Adverbial

AFAIK you can get the endpoint from Kubernetes cluster running Cloud composer. – Abecedary

@Abecedary thanks, does that mean you think it's possible to assign a static IP for Cloud Composer since it runs on Kubernetes? I wasn't completely clear where the Python processes for Composer were running. – Adverbial

Yes, looks like it. – Abecedary

Answer 1

Now is possible. You will need to configure the Cloud Composer in a private VPC and expose the cluster using a Cloud NAT.

The documentation has some information on how to do this: https://cloud.google.com/composer/docs/concepts/private-ip

Inordinate

Answer 2

It's not possible to assign a static IP to the underlying GKE cluster in a Composer environment. The endpoint @kaxil mentioned is the Kubernetes master endpoint but not the GKE nodes.

If the intent is to let all outgoing network connections from Composer tasks use the same external IP, consider setting up a GCE VM with a static IP and adding a proxy service for all workloads running inside your Composer GKE cluster.

More information: https://groups.google.com/forum/#!topic/kubernetes-users/zNytc8GVB5s.

Tournai