Refreshing dags without web server restart apache airflow

Question

Is there any way to reload the jobs without having to restart the server?

Anglia

In Airflow 2 the refresh button and endpoints have been removed github.com/apache/airflow/blob/main/… – Stir

Answer 1

In your airflow.cfg, you've these two configurations to control this behavior:

# after how much time a new DAGs should be picked up from the filesystem
min_file_process_interval = 0
dag_dir_list_interval = 60

You might have to reload the web-server, scheduler and workers for your new configuration to take effect.

Samal

Some more details in #51558813 – Hotheaded

Answer 2

I had the same question, and didn't see this answer yet. I was able to do it from the command line with the following:

python -c "from airflow.models import DagBag; d = DagBag();"

When the webserver is running, it refreshes dags every 30 seconds or so by default, but this will refresh them in between if necessary.

Bethlehem

Unfortunately that no longer works for me in Airflow 2.4.0. – Lachman

I'm having the same issue with 2.4.0 – Germander

Works fine in 2.7.3 – Verbose

Answer 3

Dags should be reloaded when you update the associated python file.

If they are not, first try to manually refresh them in UI by clicking the button that looks like a recycle symbol:

If that doesn't work, delete all the .pyc files in the dags folder.

Usually though, when I save the python file the dag gets updated within a few moments.

Jihad

I can't find those icons in the UI. I'm using Google Composer, version 2.2.5+composer. Where are those icons located? – Reeducate

@Reeducate this answer is 5 years old now. I don't 100% know what the UI you are using looks like. Sometimes you can click the airflow spinner icon in the top left of the airflow dags page and that will refresh them. – Jihad

Answer 4

On the webserver, issue a airflow dags reserialize to force Airflow to read from the dag bags configured.

You'll also be able to look at the logs of this to figure out the complete error stack, if some dags can't be interpreted.

Airsickness

Answer 5

I'm pretty new to airflow, but I had initially used sample code, which got picked up right away and then edited it to call my own code.

This was ultimately giving an error, but I only found this out once I had deleted the DAG with the example code, on the airflow webserver's UI (the trash button):

Once deleted, it showed me the error which was preventing it from loading the new dag.

Bier