How can I setup Cloud Composer to send email?

Question

I want to get email notifications with Cloud Composer but I am unsure how to do that. How can I configure a Composer environment to send email notifications?

Georgeanngeorgeanna

Answer 1

Because sending email or opening SMTP email relays can be a dangerous or prohibited Cloud behavior, we recommend you use Sendgrid to send email from Composer environments. The Composer environment setup directions contain a section specifically on how to setup Sendgrid with Cloud Composer.

Georgeanngeorgeanna

I tried to edit the answer but the queue is full - the link posted here does not have the Sendgrid instructions. Those are now linked from the writing DAGs page EmailOperator section cloud.google.com/composer/docs/how-to/using/… – Barina

Answer 2

You can use your own SMTP (or even gmail) instead of Sendgrid.

Add your SMTP details to Cloud Composer as follows:

1. Configure email and smtp settings by overriding the email backend:
   https://cloud.google.com/composer/docs/composer-2/configure-email#configure_third-party_smtp_services
   Because I was using SSL, I had to set:

    port=465
    smtp_ssl=True
    smtp_starttls=False

2. Add your smtp password to GCP Secret Manager:
   https://cloud.google.com/composer/docs/composer-2/configure-email#smtp_password
   Additional note: don't forget to give your Cloud Composer service account access to the password by give it the role Secret Manager Secret Accessor in IAM.

Or alternatively you can use Gmail.
I didn't try this myself, but I think it's good to know that this is also possible.
Here's an example of someone describing the process: https://helptechcommunity.wordpress.com/2020/04/04/airflow-email-configuration/

Ornithine