Constantly monitor a program/process using Python

Question

I am trying to constantly monitor a process which is basically a Python program. If the program stops, then I have to start the program again. I am using another Python program to do so.

For example, say I have to constantly run a process called run_constantly.py. I initially run this program manually, which writes its process ID to the file "PID" (in the location out/PROCESSID/PID).

Now I run another program which has the following code to monitor the program run_constantly.py from a Linux environment:

def Monitor_Periodic_Process():

    TIMER_RUNIN = 1800
    foo = imp.load_source("Run_Module","run_constantly.py")
    PROGRAM_TO_MONITOR = ['run_constantly.py','out/PROCESSID/PID']
    while(1):
        # call the function checkPID to see if the program is running or not
        res = checkPID(PROGRAM_TO_MONITOR)
        # if res is 0 then program is not running so schedule it
        if (res == 0):
            date_time = datetime.now()
            scheduler.add_cron_job(foo.Run_Module, year=date_time.year, day=date_time.day, month=date_time.month, hour=date_time.hour, minute=date_time.minute+2)
            scheduler.start()
            scheduler.get_jobs()
            time.sleep(TIMER_NOT_RUNIN)
            continue
        else:
            #the process is running sleep and then monitor again
            time.sleep(TIMER_RUNIN)
            continue

I have not included the checkPID() function here. checkPID() basically checks if the process ID still exists (i.e. if the program is still running) and if it does not exist, it returns 0. In the above program, I check if res == 0, and if so, then I use Python's scheduler to schedule the program. However, the major problem that I am currently facing is that the process ID of this program and the run_constantly.py program turns to be same once I schedule the run_constantly.py using the scheduler.add_cron_job() function. So if the program run_constantly.py crashes, the following program still thinks that the run_constantly.py is running (since both process IDs are same), and therefore continues to go into the else loop to sleep and monitor again.

Can someone tell me how to solve this issue? Is there a simple way to constantly monitor a program and reschedule it when it has crashed?

Integrated

Answer 1

There are many programs that can do this.

On Ubuntu there is upstart (installed by default)

Lots of people like http://supervisord.org/

monit as mentioned by @nathan

If you are looking for a python alternative there is a library that has just been released called circus which looks interesting.

And pretty much every linux distro probably has one of these built in.

The choice is really just down to which one you like better, but you would be far better off using one of these than writing it yourself.

Hope that helps

Telophase

+1 on supervisord. I use it and its really easy and cross platform. – Mako

@Mark Lakewood Thanks, Monit seems to be pretty interesting, although it lacks proper documentation. It works pretty neat – Integrated

writing one ourselves allows us to extend it to suit our need (e.g. log the data for visualize them later) – Gondolier

Answer 2

If you are willing to control the monitored program directly from python instead of using cron, have a look at the subprocess module :

The subprocess module allows you to spawn new processes,
connect to their input/output/error pipes, and obtain their return codes.

Check examples like track process status with python on SO for examples and references.

Mastin

Answer 3

You could just use monit http://mmonit.com/monit/

It monitors processes and restarts them (and other things.)

Yeh

monit can do a lot of usefull thinks. – Karmen

But anyway how is your scheduler run the application? It is not clear from your code sample. To resolve problems with PIDs you have run it correctly in the system. – Karmen

Answer 4

I thought I'd add a more versatile solution, which is one that I personally use all the time as well.

It's name is Immortal (source is at https://github.com/immortal/immortal)

To have it monitor and instantly restart a program if it stops, simply run the following command:

immortal <command>

So in your case I would run run_constantly.py like so:

immortal python run_constantly.py

The command ps aux | grep run_constantly.py should return 2 process IDs, one for the Immortal command, and one for the separate command Immortal started (just the regular command. As long as the Immortal process is running, run_constantly.py will stay running.

Petra