How to get accurate process CPU and memory usage with python?

Question

I am trying to create a process monitor but I am unable to get accurate results comparing my results to windows task manager.

I have been using psutil which seems to work fine when looking at the overall cpu and memory usage but doesn't appear to be very accurate for a single process. Memory usage is always higher than task manager and CPU it always random.

I am setting the process once on initialise with self.process = psutil.Process(self.pid) and then calling the below method once a second, the process in task manager is running at a constant 5.4% cpu usage and 130mb ram however the below code produces:

CPU: 12.5375
Memory 156459008
CPU: 0.0
Memory 156459008
CPU: 0.0
Memory 156459008
CPU: 0.0
Memory 156459008
CPU: 12.5375
Memory 156459008
CPU: 0.0
Memory 156459008
CPU: 0.0
Memory 156459008

Example code:

def process_info(self):    
        # I am calling this method twice because I read the first time gets ignored?
        ignore_cpu = self.process.cpu_percent(interval=None) / psutil.cpu_count()
        time.sleep(0.1)
        process_cpu = self.process.cpu_percent(interval=None) / psutil.cpu_count()
        
        # I also tried the below code but it was much worse than above
        # for j in range(10):
        #     if j == 0:
        #         test_list = []
        #     p_cpu = self.process.cpu_percent(interval=0.1) / psutil.cpu_count()
        #     test_list.append(p_cpu)
        # process_cpu = (sum(test_list)) / len(test_list)

        # Memory is about 25mb higher than task manager
        process_memory = self.process.memory_info().rss

        print(f"CPU: {process_cpu}")
        print(f"Memory: {process_memory}")

am I using psutil incorrectly or is there a more accurate way to grab the data?

Burgin

I think that windows task manager could simply be not 100% correct for simple performance reasons since it greps all processes rather than one – Natascha

Thanks for the reply, that may be the case and could possibly adjust my question, I am not concerned with getting accurate CPU and memory usage, I would like to get the values used by windows task manager as this is how it is currently measured therefore making it better or worse changes this benchmark. – Burgin

I believe you would want to change it to display the mean over a timeframe rather then a single moment, it's normal that a process is not using the CPU at a given time, I'd look at the median of a given time or something similar, the memory usage probably differs because the task manager is not really accurate, I might be really wrong tho, I don't really know how the task managers works, just guesses. you could look at similar open source software like TOP/HTOP to get an idea – Natascha

Answer 1

I think you are misinterpreting the output of psutil.cpu_percent(). Returns a float representing the current system-wide CPU utilization as a percentage. When interval is > 0.0 compares process times to system CPU times elapsed before and after the interval (blocking). When interval is 0.0 or None compares process times to system CPU times elapsed since last call, returning immediately. That means that the first time this is called it will return a meaningful 0.0 value on which subsequent calls can be based. In this case is recommended for accuracy that this function be called with at least 0.1 seconds between calls.

So, if you call it with interval=0.1, it will return the percentage of time the process was running in the last 0.1 seconds. If you call it with interval=None, it will return the percentage of time the process was running since the last time you called it. So, if you call it twice in a row with interval=None, the first call will return the percentage of time the process was running since the last time you called it, and the second call will return the percentage of time the process was running since the last time you called it. So, if you want to get the percentage of time the process was running in the last 0.1 seconds, you should call it with interval=0.1.

Overstreet

Thanks for explaining how it works, you are correct I was misinterpreting the output, however I have also tried the way you suggest (see the commented out code in my example) but it gave some very random results. – Burgin

Answer 2

Like @Axeltherabbit mentioned in his comment, Task Manager grabs all processes under a given name, whereas psutils grabs an individual process. psutils is correct, task manager over-scopes and decides to grab all processes.

Lynx

That doesn't explain why the numbers are different. Any docs showing which one is more accurate? – Deflection

It does explain why the numbers are different, it's because it's pulling the numbers from the locations mentioned above. psutils docs should explain that – Lynx

Answer 3

this should be more accurate i think?

    import psutil

for process in [psutil.Process(pid) for pid in psutil.pids()]:
    try:
        process_name = process.name()
        process_mem = process.memory_percent()
        process_cpu = process.cpu_percent(interval=0.5)
    except psutil.NoSuchProcess as e:
        print(e.pid, "killed before analysis")
    else:
        print("Name:", process_name)
        print("CPU%:", process_cpu)
        print("MEM%:", process_mem)

Hillegass

This only works for system info, not for an individual process: 'Process' object has no attribute 'virtual_memory' TypeError: Process.cpu_percent() got an unexpected keyword argument 'percpu' – Burgin

no, thanks for trying but that gives the same results that I was getting in the above example, except memory is now in a percent rather than size – Burgin