How to remove default example dags in airflow

Question

I am a new user of Airbnb's open source workflow/datapipeline software airflow. There are dozens of default example dags after the web UI is started. I tried many ways to remove these dags, but I've failed to do so.

• load_examples = False is set in airflow.cfg.
• Folder lib/python2.7/site-packages/airflow/example_dags is removed.

States of those example dags are changed to gray after I removed the dags folder, but the items still occupy the web UI screen. And a new dag folder is specified in airflow.cfg as dags_folder = /mnt/dag/1. I checked this dag folder, nothing is there. It's really weird to me why it is so difficult to remove these examples.

Nipha

Answer 1

When you startup airflow, make sure you set:

load_examples = False

inside your airflow.cfg

If you have already started airflow with this not set to false, you can set it to false and run airflow db reset(if you are using 1.x airflow resetdb) in the cli (!which will destroy all current dag information!).

Alternatively you can go into the airflow_db and manually delete those entries from the dag table.

Phillida

I've just tried this in 1.8.1 and it doesn't seem to be enough. I can confirm that the dag table is empty after running airflow resetdb, and I've made the change to the cfg file, but the example dags are all still there in the webserver. I had to restart the webserver to make this work. That involved killing a lot of processes and running airflow webserver again. Probably best to run with systemd or upstart to manage restarts airflow.incubator.apache.org/… – Nancee

running airflow resetdb command works fine in 1.10.5. should be accepted solution (upvoting for now) – Universe

This answer makes sense. The database is reset and uploaded with new data. I think OP forgot to check. @Bronzels – Winter

Since this is highly voted, people disabling example dags may also want to disable default connections. A similar property exists in 1.10.10 load_default_connections = False #38407814 – Byrann

Please note, this might cause your webserver to throw an error and won't load up. I had to recreate the user for the DAG to be up, and examples still are there. They didn't go anywhere. – Rebatement

Beware that resetdb will remove all the connections/variables that you have defined. – Euphonium

for the airflow 2.0.1 , as I know; after setting load_examples = False inside your airflow.cfg use airlow db reset instead of airflow resetdb – Darnall

Answer 2

For Airflow 2.0, in docker-compose.yaml you can set AIRFLOW__CORE__LOAD_EXAMPLES: 'false' to not load them instead of editing the .cfg file.

Soothsayer

This saved me a lot of banging my head against a wall. Thank you! The default docker-compose file in the documentation has this setting set and it overrides whatever is in the airflow.cfg file. – Semitic

Everyone focused on the cfg file while the problem was with compose file, thanks! – Manufactory

Answer 3

Like others have said, you can change load_examples = False within airflow.cfg. However this requires that the cfg file already existing.

You can init the airflow DB without having to configure the cfg file by using ENV variables.

export AIRFLOW__CORE__LOAD_EXAMPLES=False
airflow initdb

See docs for more information.

Protrusion

Answer 4

Before you start airflow make sure you set load_example variable to False in airflow.cfg file. By default it is set to True.

load_examples = False

If you have already started airflow, you have to manually delete example DAG from the airflow UI. Click on delete icon available on the right side of the DAG to delete it.

Instead of manually deleting example DAG, you can reset your database by using airflow resetdb command but that will delete your connections and variables and other important information. Do not use airflow resetdb option in production.

Headcloth

the command is now airflow db reset – Caracas

Answer 5

Easy Way:

1. put ..

   load_examples = False

into the airflow.cfg file

2. then close and restart the webserver and scheduler

Ballet

Restarting server and scheduler is the important part. Thank you! – Acevedo

Answer 6

TL;DR: check that you have only DAG files in your dags_folder -- Airflow will traverse this directory recursively and try to load all .py files.

I've lost some time debugging similar behaviour of Airflow: even though load_examples = False, airflow was still loading tons of unnecessary stuff, including example_dags. The problem was that I was having a virtualenv directory venv/ in dags_folder, and -- I was not expecting that -- Airflow searches for dags recursively in the dags dir. So it was loading example_dags from the apache-airflow installed in that virtualenv.

UPD: there's a .airflowignore file to ignore directories from dags_folder

Bade

Answer 7

while starting up the airflow make sure to change

load_examples = False

in the airflow.cfg file .Then close and restart the webserver and scheduler.

Aeroembolism

Answer 8

As others pointed out, change the load_examples variable to False in airflow.cfg. Should look something like:

load_examples = False

If you have already started the server you need to reset the database. In the newer versions, the command to reset the DB is:

airflow db reset

Some of the answers above use the old CLI commands

Stoplight

Answer 9

If LDAP based authentication is turned on after airflow resetdb and restarting airflow your login window may not appear and may give an error due to cached login id ( but no password ). If this happens, clear your cache and try. If that still doesn't work, turn off authentication, stop airflow and start. Then turn on authentication stop and start - you will be able to see login window and login with your LDAP authentication -Suresh

Predation