Subkernel memory control in Mathematica

Question

I have a somewhat similar question as: Mathematica running out of memory

I am interested in something like this:

ParallelTable[F[i], {i, 0, 14.9, 0.001}]

where F[i] is a complicated numerical integral (I haven't yet found an easy way to reproduce the problem without page filling definitions for an integral).

My problem is that the subkernels blow up in memory and I have to stop evaluation if I won't let the machine swapping.

But even if I have stopped evaluation the kernels won't give free their occupied memory.

ClearSystemCache[] 

I even have tried

ParallelEvaluate[ClearSystemCache[]]

but

ParallelEvaluate[MemoryInUse[]]

stays at

{823185944, 833146832, 812429208, 840150336, 850057024, 834441704, 
847068768, 850424224}

it seems that all memory controlling only works for the main kernel? By now the only way is to shut down all the kernels and launch them again.

I really hope there are some solutions out there... Thanks a lot.

Evite

Answer 1

Memory control works for the kernel where control expressions involving such functions as MemoryConstrained, MemoryInUse, Clear, Unset, Remove, $HistoryLength, ClearSystemCache etc. are evaluated. It seems that in your case the source of the memory leaks is not due to Mathematica's internal caching mechanism (thanks for the link, BTW!).

Have you tried to evaluate $HistoryLength=0; in all subkernels before using them for computations? If you have not yet, I strongly recommend to try.

Since you are working with numerical integration functions, I suggest also to try to optimize usage of them. For example, if you make numerical integration using NDSolve and need only a limited set of calculated points (or even the only one point) you should use the form NDSolve[eqns,y,{x,x_needed_min,x_needed_max}] (or even NDSolve[eqns,y,{x,x_max,x_max}]) instead of NDSolve[eqns,y,{x,x_min,x_max}] or NDSolve[eqns,y,{x,0,x_max}]. This can dramatically reduce memory usage in some cases! You can also use EventLocator for memory control.

Nocti

So i submitted now my problem to the wolfram support and it seems NIntegrate indeed can have a memory leak. I could manage this leak on my one with ClearSystemCache but this will not work for parallel evaluation since ClearSystemCache does not affect subkernels started by an parallel evaluation. I will update any further information i get. – Evite

@Michael If so, probably evaluating ClearSystemCache[] in all subkernels can help? – Nocti

Answer 2

I was(am?) having the exact same problem, almost word for word. I just had some good luck with adding the option to the problem integral:

Method-> {"GlobalAdaptive", "SymbolicProcessing"->False}

You can probably choose any other method if you'd like, but I had success with this within the last few minutes. Also, a lot of nasty inconsistencies I used to be getting are gone, and integration proceeds MUCH faster.

Eubanks