How do I select my Airflow or Python version with Cloud Composer?

Question

I am using Cloud Composer and I noticed that it selects the version of Apache Airflow and Python (2.7.x) for me. I want to use a different version of Airflow and/or Python. How can I change this?

Protozoal

Answer 1

Cloud Composer deploys the latest stable build of Airflow. New versions of Airflow are usually deployed by Composer within a a few weeks of their stable release. The Airflow version deployed and the Python version installed cannot be changed at this time. A future release of Cloud Composer may offer the ability to select the Airflow and/or Python version for new environments.

Protozoal

Can you confirm whether Cloud Composer supports Python 3 today? – Manatarms

It does not. Only 2.7 is supported today. – Viviyan

@AbhiSivasailam I'm looking for a source for this information, do you have one ? – Merri

Composer is Python 2.7 at the moment. We're working to add 3.x in the next several weeks. – Protozoal

Is there a status page to see when Python 3.x will be available on Cloud Composer @James? – Mazarin

@bukzor Good question. We're (over time) moving to using the public tracker more so there's better visibility. For this particular item, we're targeting mid-October for Python 3.6 (and Airflow 1.10.) – Protozoal

Beta Python3 support is scheduled to be available starting October 9: cloud.google.com/composer/docs/release-notes – Fisk

@TrevorEdwards could you give an example of how you can deploy an airflow with Python3. What will happen to the logging if you change the Python, Airflow or Cloud Composer version? – Sustentation

@PaulVelthuis Take a look at the answers in #52745340 for how to deploy Python 3 Composer environments. AFAIK, the logging experience should be consistent across Python/Airflow versions. – Fisk

@taylorEdmiston cloud composer support python 3 only if you will enable the beta version present at the top corner of it. – Timotheus

@AshishKumar Thanks. It looks like GA Python 3 support (Python 3.6.6) in Composer was rolled out in the Composer 1.4.0 release on Nov 26, 2018. cloud.google.com/composer/docs/… – Manatarms

Answer 2

If you want to deploy a specific version of Airflow you will need to use the gcloud CLI tool in order to specify this. It is not currently possible to do this from the web front end.

Have a look at the follow page to see the available versions https://cloud.google.com/composer/docs/concepts/versioning/composer-versions

If you would like to deploy say Airflow 1.10 and Python 3 to your environment you would use the

--image-version

--python-version

flags in order to set this. For example if you used the following it would install with Composer 1.4.1, Airflow 1.10 and Python 3

gcloud beta composer environments create ENV_NAME --image-version composer-1.4.1-airflow-1.10.0 --python-version 3

You will need to specify all the other parameters and arguments required for the environment as well. The above only shows the two arguments to set the Airflow and Python versions.

Hoon