how to kill airflow scheduler and webserver?

Question

I am new to airflow, tried to run a dag by starting airflow webserver and scheduler. After I closed the scheduler and airflow webserver, the airflow processes are still running.

ps aux | grep airflow shows 2 airflow webserver running, and scheduler running for all dags.

I tried running kill $(ps aux | grep airflow | awk '{print $2}') but it did not help.

I don't have sudo permissions and webserver UI access.

Episcopate

What do you mean by "closed the scheduler and airflow webserver"? – Neall

closed meaning terminated by hitting Ctrl + C. – Episcopate

Answer 1

If you run Airflow locally and start it with the two commands airflow scheduler and airflow webserver, then those processes will run in the foreground. So, simply hitting Ctrl-C for each of them should terminate them and all their child processes.

If you don't have those two processes running in the foreground, there is another way. Airflow creates files with process IDs of the scheduler and gunicorn server in its home directory (by default ~/airflow/).

Running

kill $(cat ~/airflow/airflow-scheduler.pid)

should terminate the scheduler.

Unfortunately, airflow-webserver.pid contains the PID of the gunicorn server and not the initial Airflow command that started it (which is the parent of the gunicorn process). So, we will first have to find the parent PID of the gunicorn process and then kill the parent process.

Running

kill $(ps -o ppid= -p $(cat ~/airflow/airflow-webserver.pid))

should terminate the webserver.

If simply running kill (i.e., sending SIGTERM) for these processes does not work you can always try sending SIGKILL: kill -9 <pid>. This should definitely kill them.

NOTE: Starting with Airflow 2.2.0, there is the standalone CLI command that runs all Airflow components, which are needed for local development. The command can be terminated with simple Ctrl-C and it automatically shuts down all running components.

Neall

I tried killing the process by kill -9 <pid of airflow> but the process re-runs with another PID. – Episcopate

I have added an image also, please have a look – Episcopate

@Episcopate have you tried the commands I provided? – Neall

Yes. As you mentioned, kill $(cat ~/airflow/airflow-scheduler.pid) did kill/removed the airflow-webserver.pid but the process still re-spawns. – Episcopate

@Episcopate that command kills the scheduler. To kill the webserver you need to run the last kill command from my answer (eventually with -9). – Neall

Correct me if i am wrong. For some reasons, I see it as airflow-webserver-monitor.pid – Uniflorous

The kill $(ps -o ppid= -p $(cat ~/airflow/airflow-webserver.pid)) command needed permissions, and running it with sudo restarted my entire mac. I'm not sure why exactly but maybe beware of using it. – Transpontine

Answer 2

if you are running Airflow Webserver in the background with (airflow webserver -d -p 8080) command you will be able to use existing terminal/Ubuntu window for further commands. In this case use (airflow webserver) command. You will receive a message that Airflow Server is already running with PID xxx. Take that PID and use (kill xxx) command while replace xxx with PID (also called Process ID) and you will be done.

If you are running Airflow Webserver with (airflow webserver) command, then you are running Airflow Webserver in the foreground, and may not be able to use existing terminal/Ubuntu window. In this case try using (Ctrl + c) command inside the terminal that is running Airflow Server. You won't see your name prompt, but still a blinking cursor, click by the blinking cursor and press (Ctrl + c). If this works, then you are done closing Airflow Webserver. If not then open another terminal/Ubuntu window and use (airflow webserver) command. You will receive a message that Airflow Server is already running with PID xxx. Thake that PID and use (kill xxx) command while replace xxx with PID (also called Process ID) and you will be done.

Romney