memory_limit: How does it work?

Question

I have a php script that runs for about 2 hours. It is a cron job. The cron job runs every 4 hours.

At the end of the script, I display some memory values.

The memory_get_usage() result is 881568 Bytes (0.840766906738M)
The memory_get_peak_usage() result is 1340304 Bytes (1.27821350098M)
The memory_get_usage(true) result is 1572864 Bytes (1.5M)
The memory_get_peak_usage(true) result is 1835008 Bytes (1.75M)

The memory_limit in php.ini was 128M and it didn't work. I raise it to 256M and now it works.

But as the memory peak of the script is less than 2M....

So how does the memory_limit parameter work?

Is it the total memory used by the script? If so, how can I calculate it?

Is it the memory peak of the script? if so, am I calculating it right?

I'm using php 5.3.16.

EDIT

I don't have any error messages. When the limit was 128M The script executes, but never finishes.

Autecology

what else are you running on the same server? – Accrete

If your script crashed with memory limit error, how did you see the log messages? – Putumayo

@Putumayo It didn't crash when the limit was upped to 256MB, it crashed when it was set to 128MB. However now it's set to 256MB Marm see's that the peak memory usage is no where near the 128MB limit, his question is so why did he need to up the limit to make it work? – Acentric

One possibility is that somewhere in the script you have a function (or another local scope), which locally requires a lot of memory (e.g. to retrieve a lot of data from the database). When the control leaves that function, after some time, garbage collector frees that memory - before you display memory usage at the end of the script. Thus, somewhere along the execution, script requires more memory, but later on it's freed; therefore when you display memory usage it shows lower values than the max required to execute the script. – Home

What error did you get when it wouldn't work? – Muscadine

@AleksG Thus the use of memory_get_peak_usage() instead of memory_get_usage(). The latter tells you how much memory is used at the point when it's being called, while the former tells you the peak usage at any point during the script. Is there a case when ..._peak_usage() won't actually return peak usage? – Monteria

@Aleks G: The memory_peak function is supposed to tell me the peak of the script? If not, is there a way to get the real peak? – Autecology

@Rene Pot: There is about 10 websites on the server, so a lot of things are running. – Autecology

There is no error messages, see the edit for details – Autecology

If they are all on the same PHP instance, then they all count. These numbers are just the max for only that script – Accrete

@Rene Pot: Every script runs on its own php instance. Also, memory_limit paramter is for each script. – Autecology

Answer 1

Try using a function that uses your operating system to report actual memory such as this one.

function unix_get_usage() { 
      $pid = getmypid(); 
      exec("ps -o rss -p $pid", $output); 
      return $output[1] *1024; 
 }

function windows_get_usage(){
    $output = array();
    exec('tasklist /FI "PID eq '.getmypid().'" /FO LIST', $output );
    return preg_replace( '/[^0-9]/', '', $output[5] ) * 1024;
}

Your script probably consumes a lot of memory that PHP doesn't consider when returning from memory_get_usage() (which looks at the heap btw). Stack allocated variables would have been cleared up by the time memory_get_usage() was called and returned.

You should also try running this function and others at other points during execution, e.g. after big MySQL calls or file processing. memory_get_peak_usage() may not work on all OS's, especially depending on compile options.

Polo

That makes sense, but should that not be a problem with memory_get_peak_usage()? I wouldn't think so, but it wouldn't hurt to check memory usage a different way as you suggest. – Monteria

Not if both (memory_get_peak_usage and memory_get_usage) only look on the heap and most of your memory is purely automatic/stack allocated variables. – Polo

And I must call this function at every places I thing I use a lot of memory? Is there a system function to get the stack peak? Because I'm not sure where my script uses a lot of memory – Autecology

Actually, after experimenting with stack, heap, recursion I'm having a hard time finding an instance where memory_get_peak_usage() doesn't agree with memory_get_usage() at its peak (using a tracked global variable). This is on Mac OSX and PHP 5.4.4 and 5.3.14, so I guess the question is what OS and PHP version are you using? Also what kind of work are you doing so that I may try to replicate it? – Polo

After doing a little research it appears that Windows may not work/respond to these functions as expected. ibm.com/developerworks/opensource/library/os-php-v521 – Polo

@Jared Kipe: I'm using php 5.3.16 on centos 5.8. Basically, the cron job makes about 20,000 curl requests and receive a 60,000 lines XML response for each one. After that, I'm parsing the XML and insert the data in my innoDB database. – Autecology

Also I'd really like to know what is the meory_limit (stack, heap, both of them, other...) and if it's the peak or the total memory usage – Autecology

Are the curl requests to a script on the same host or somewhere else? memory_limit will include both stack and heap. Is the cron task running PHP directly (CLI) or via a wget or curl request? – Polo

the curl requests are made on another server. My cron task is running directly. – Autecology

Can you post a simple script that reproduces it (say with setting the memory limit to something suitably low like 8MB and causing it to run out of memory via something simple like stack overflowing yet, never saying it actually uses more than a few hundred k of RAM?)? – Polo

let us continue this discussion in chat – Polo