Correct way to cap Mathematica memory use?

Question

Under a 32-bit operating system, where maximum memory allocated to any one program is limited, Mathematica gracefully terminates the kernel and returns a max memory allocation error.

On a 64-bit OS however, Mathematica will freely use all the memory available and grind the system to a halt. Therefore, what is the correct way to cap memory usage? One could use MemoryConstrained combined with $Pre or CellEvaluationFunction but I would rather not tie up either of those for this purpose, or have to modify existing uses to incorporate this function.

Is there another way to globally restrict memory usage, such as a kernel flag, or system $Option?

Colliery

A good operating system should be able to present to the user program a managed view of its resources, including CPU and memory. – Oilcloth

@belisarius speak plainly man, what are you saying? ;-) Is there an option to set maximum RAM under Windows 7? Of course this assumes Windows is good... – Colliery

I was talking about good operating systems ;) ... No, really I don't know about Win7 – Oilcloth

@belisarius I was afraid of that. The Unix crowd always laughs at what Windows doesn't do. I cannot recall, what OS do you use? – Colliery

OS x also allows mma to grind it to a halt, and, from what I recall from when I was using it, Linux does, too (things may have changed since then though). So a good question. – Wikiup

@belisarius I guess I didn't forget; I don't think I knew that. acl, interesting, thank you. – Colliery

@Mr. I guess it could be done by using this, but not without some pain – Oilcloth

@belisarius your suggestion is not unwelcome, but I am looking for a Mathematica method. Otherwise, I would have posted to SuperUser, asking how to limit the memory use of a generic application. – Colliery

@Mr. One should not make a program to behave like that. The operating system provides you with a virtual environment, so you don't really know what reality is. For example, the OS could compress (ziplike) your memory pages without your knowledge nor acknowledge, and your perceived memory usage could be off by zillions. I saw some beasts that implemented things like that, but is not usually a good idea. – Oilcloth

@belisarius I am not sure what you are referring to. Are you suggesting that there should not be a self-limiting option in Mathematica? In that case, it already has one, MemoryConstrained, but I am looking for a different way to apply it. If you are saying that the operating system should not set a hard limit on a program, I don't understand; I would rather the OS said "MathKernel has used too much RAM and has been terminated. Goodbye" than to lock up the whole system. I realize that true RAM use is not going to be the same as calculated RAM use in many cases, but I can accept that. – Colliery

@Mr. What I am saying is that the memory (real or virtual) a program thinks it is using may differ from what the OS is really assigning to it. The limit should be managed by the OS and not the program. As for the MemoryConstrained thing, I believe the Mma team is trying to compensate OS deficiencies. – Oilcloth

This reminds me of the old Mac OS (sans 'X') where you sometimes had to manually increase the amount of memory a program was allowed to use. Personally I'd use the $Pre approach. – Hyades

@Brett Setting memory limits for processes is a serious concern for OS designers publib.boulder.ibm.com/infocenter/zos/v1r12/… - BTW How was the conference? – Oilcloth

The conference was good; got to meet Sjoerd and Leonid in person, unfortunately mostly in passing. And they scheduled my talks (or my teammates') at the same time as other talks I would really have liked to see. – Hyades

@Brett I wish I'd be able to go. Perhaps next time! – Oilcloth

@Mr. Probably one can use belisarius' idea from inside Mathematica by using NETLink. – Ume

@Mr. Wizard I posted my attempt to solve this here (the function I called totalMemoryConstrained) here: #6405804 . I have an impression though that this does not always solve the problem. See if this can work for you. – Tollefson

@Alexey, I have almost no experience with MathLink/NETLink. Would you at least give some tips for implementing this in an answer below? Leonid, thank you, I shall read through that question/answer later, and vote accordingly. :-) – Colliery

@Mr. I have no much experience with .NET, so I cannot say how to call the Job functions. But I can refer you to a basic NETLink Documentation page: "Setting the Kernel Process Priority." You will see that it is really simple to manipulate the kernel process priority. And it is also easy to get information on the kernel current memory usage. In this way, one can write a simple monitoring program which will check how many memory this process currently uses. But probably the idea with Job objects is much better. – Ume

@Mr. You can also be interested in my answer: "How to kill slave kernel securely?" – Ume

@Mr. As an alternative, you can also be interested in my unpublished function which implements FreeMemoryConstrained functionality. But it is really very complex and currently works completely correctly only with version 7. I cannot publish it as an answer because it is huge. But I can give some code snippets which demonstrate basic ideas. Please note that it requires running two MathKernel processes: (1) "master" which monitors the current memory usage of (2) "slave" process in which all the target computations are performed. – Ume

@Alexey Perhaps you could use only one kernel and RunScheduledTask – Oilcloth

Answer 1

In Mathematica 8 you could start a memory watchdog, something along the lines of:

maxMemAllowed        = 15449604;
intervalBetweenTests = 1; (*seconds*)
iAmAliveSignal       = 0;
Dynamic[iAmAliveSignal]
RunScheduledTask[
       If[MemoryInUse[] > maxMemAllowed , Quit[], iAmAliveSignal++],      
       intervalBetweenTests];

Remember to run

RemoveScheduledTask[ScheduledTasks[]];

to disable it.

Edit

You may alert or interactively decide what to do before quitting. As requested, here is a trial with 1.3GB allocated. I can't go much further than that in this machine.

maxMemAllowed = 1.3 1024^3; (*1.3 GB*)
intervalBetweenTests = 1; (*Seconds*)
iAmAliveSignal = 0;
leyendToPrint = "";
Dynamic[leyendToPrint]
RunScheduledTask[
  If[MemoryInUse[] > maxMemAllowed, 
   CreateDialog[CancelButton["Max Mem Reached", DialogReturn[]]]; 
   Quit[],
   Print["Memory in use: ", MemoryInUse[]]; 
   leyendToPrint = 
    "Seconds elapsed = " <> ToString[iAmAliveSignal++]], 
  intervalBetweenTests];
IntegerPartitions[320, {15}];

Oilcloth

Edit acknowledged. If we can get confirmation that this works across other operating systems, I will accept the answer. Thank you again. – Colliery

@belisarius +1 Interesting idea. Can this function be extended for restarting the kernel with the same or new code if the previous session was not a fresh MathKernel session? – Ume

@Alexey I really don't know :( – Oilcloth

@belisarius I have created separate question on it: "Self-restarting MathKernel - is it possible in Mathematica?" – Ume

it's ridiculoous that the program itself doesn't have an option to cap the memory usage. 11 versions and still the same problem. – Nabalas