configuring Airflow to work with CeleryExecutor

Question

I try to configure Airbnb AirFlow to use the CeleryExecutor like this:

I changed the executer in the airflow.cfg from SequentialExecutor to CeleryExecutor:

# The executor class that airflow should use. Choices include
# SequentialExecutor, LocalExecutor, CeleryExecutor
executor = CeleryExecutor

But I get the following error:

airflow.configuration.AirflowConfigException: error: cannot use sqlite with the CeleryExecutor

Note that the sql_alchemy_conn is configured like this:

sql_alchemy_conn = sqlite:////root/airflow/airflow.db

I looked at Airflow's GIT (https://github.com/airbnb/airflow/blob/master/airflow/configuration.py)

and found that the following code throws this exception:

def _validate(self):
        if (
                self.get("core", "executor") != 'SequentialExecutor' and
                "sqlite" in self.get('core', 'sql_alchemy_conn')):
            raise AirflowConfigException("error: cannot use sqlite with the {}".
                format(self.get('core', 'executor')))

It seems from this validate method that the sql_alchemy_conn cannot contain sqlite.

Do you have any idea how to configure the CeleryExecutor without sqllite? please note that I downloaded rabitMQ for working with the CeleryExecuter as required.

Revanche

Answer 1

It is said by AirFlow that the CeleryExecutor requires other backend than default database SQLite. You have to use MySQL or PostgreSQL, for example.

The sql_alchemy_conn in airflow.cfg must be changed to follow the SqlAlchemy connection string structure (see SqlAlchemy document)

For example,

sql_alchemy_conn = postgresql+psycopg2://airflow:[email protected]:5432/airflow

Catastrophism

Can I know How to add mysql connection instead of postgresql – Cahier

@Cahier msql connection string for sqlalchemy can be something like mysql://username:password@hostname:port/database_name – Psychodiagnosis

Answer 2

To configure Airflow for mysql firstly install mysql this might help or just google it

• goto airflow installation director usually /home//airflow
• edit airflow.cfg

• locate

  sql_alchemy_conn = sqlite:////home/vipul/airflow/airflow.db

and add # in front of it so it looks like

#sql_alchemy_conn = sqlite:////home/vipul/airflow/airflow.db 

if you have default sqlite

• add this line below

  sql_alchemy_conn = mysql://:@localhost:3306/

• save the file

• run command

  airflow initdb

and done !

Dierdredieresis

This is the tutoriall I have followed support.rackspace.com/how-to/installing-mysql-server-on-ubuntu. Also, you need to specify user,password and database in the connection mysql://:user:password@localhost:3306/database – Pericardium

Answer 3

As other answers have stated you need to use a different database besides SQLite. Additionally you need to install rabbitmq, configure it appropriately, and change each of your airflow.cfg's to have the correct rabbitmq information. For an excellent tutorial on this see A Guide On How To Build An Airflow Server/Cluster.

Stouffer

Answer 4

If you run it on a kubernetes cluster. Use the following config:

airflow:
  config:
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://postgres:airflow@airflow-postgresql:5432/airflow

Kaenel

Answer 5

The main problem is your sqlite. As document on Sequential Executor said:

The SequentialExecutor is the default executor when you first install airflow. It is the only executor that can be used with sqlite since sqlite doesn’t support multiple connections. This executor will only run one task instance at a time. For production use case, please use other executors.

In other word: if you are using sqlite as sql_alchemy_conn, then you only choice is Sequential Executor.

Volitive