When to call SetProcessWorkingSetSize? (Convincing the memory manager to release the memory)

Question

In a previous post ( My program never releases the memory back. Why? ) I show that FastMM can cache (read as hold for itself) pretty large amounts of memory. If your application just loaded a large data set in RAM, after releasing the data, you will see that impressive amounts of RAM are not released back to the memory pool.

I looked around and it seems that calling the SetProcessWorkingSetSize API function will "flush" the cache to disk. However, I cannot decide when to call this function. I wanted to call it at the end of the OnClick event on the button that is performing the RAM intensive operation. However, some people are saying that this may cause AV.

If anybody used this function successfully, please let me (us) know.

Many thanks.

---

Edit:
1. After releasing the data set, the program still takes large amounts of RAM. After calling SetProcessWorkingSetSize the size returns to few MB. Some argue that nothing was released back. I agree. But the memory foot print is now small AND it is NOT increasing back after using the program normally (for example when performing normal operation that does not involves loading large data sets). Unfortunately, there is no way to demonstrate that the memory swapped to disk is ever loaded back into memory, but I think it is not. 2. I have already demonstrated (I hope) this is not a memory leak:
My program never releases the memory back. Why?
How to convince the memory manager to release unused memory

Cesura

why is it that your program doesn't result in memory being released, but other programs do? Is it possible that your program's memory allocation pattern is the cause of the problem? – Woodham

Hi David. I create about 260 MDI child windows. Each contains a graphic control, a tstringgrid and a complex (it has many many fields, properties, methods...) object derived from TObject. Then I release all childs at once. Is this enough for a memory fragmentation? – Cesura

The answer is: never. In your situation you should not call SetProcessWorkingSetSize at all. – Fivefinger

Could you please explain what you check to see how much RAM your process is using? Task manager? Which Windows version? What column? I want to know if you are talking about the working set size or not. It differs a great deal. – Unclose

Answer 1

Problem solved:
I don't need to use SetProcessWorkingSetSize. FastMM will eventually release the RAM.

---

To confirm that this behavior is generated by FastMM (as suggested by Barry Kelly) I crated a second program that allocated A LOT of RAM. As soon as Windows ran out of RAM, my program memory utilization returned to its original value.

Cesura

Can you define "eventually",. like 10 minutes, 60 minutes, 5 hours? Random? I'm running into what appear to be similar issues, where FastMM is causing a huge overhead in the MemoryManagerUsageSummary.OverheadBytes, but the memory generally seems to run out after a while, rather than get freed. And there's no leaks. – Courtneycourtrai

Answer 2

If SetProcessWorkingSetSize would solve your problem, then your problem is not that FastMM is keeping hold of memory. Since this function will just trim the workingset of your application by writing the memory in RAM to the page file. Nothing is released back to Windows.

In fact you only have made accessing the memory again slower, since it now has to be read from disc. This method has the same effect as minimising your application. Then Windows presumes you are not going to use the application again soon and also writes the workingset in RAM to the pagefile. Windows does a good job of deciding when to write RAM to the pagefile and tries to keep the most used memory in RAM as long as it can. It will make the workinset size smaller (write to pagefile) when there is little RAM left. I would not mess with it just to give the illusion that you program is using less memory while in fact it is using just as much as before, only now it is slower to access. Memory that is accessed again will be loaded into RAM again and make the workinset size grow again. Touching less memory keeps the workingset size smaller.

So no, this will not help you forcing FastMM to release the memory. If your goal is for your application to use less memory you should look elsewhere. Look for leaks, look for heap fragmentations look for optimisations and if you think FastMM is keeping you from doing so you should try to find facts to support it. If your goal is to keep your workinset size small you could try to keep your memory access local. Maybe FastMM or another memory manager could help you with it, but it is a very different problem compared to using to much memory. And maybe this function does help you solve the problem you are having, but I would use it with care and certainly not use it just to keep the illusion that your program has a low memory usage.

Unclose

After disabling FastMM this behavior disappear. If it would be a memory leak, shouldn't it persist? Also, minimizing the application does not decrease the memory footprint. It works for other applications but not for this one. But calling SetProcessWorkingSetSize manually works. – Cesura

Here is a Delphi article recommending using it: delphi.about.com/od/windowsshellapi/ss/… – Cesura

"Nothing is released back to Windows" - Yes. Sorry, I made my OP to sound a bit confusing. I did my homework and read about this. Indeed the cached memory is not released BUT it also does not occupy RAM pages. And the trick REALLY works for me. Please see my edited question. – Cesura

Altar: the trick DOES NOT work for you, you just don't understand. There's no point at all to use SetProcessWorkingSize in your condition. You're NOT freeing memory. You're offloading it to disk. There's no need to do that: OS will offload your pages to disc automatically when some other app needs memory. You're just making things worse because if you try to access this memory (for instance, by creating an object) and it's on the disk, you'll have to read it back now. – Fivefinger

@Altar: How do you 'disable' FastMM? You can only do that by choosing another memory manager. Which one did you choose? – Unclose

Answer 3

I agree with Lars Truijens 100%, if you don't than you can check the FasttMM memory usage via FasttMM calls GetMemoryManagerState and GetMemoryManagerUsageSummary before and after calling API SetProcessWorkingSetSize.

Nassir

Answer 4

Are you sure there is a problem? Working sets might only decrease when there really is a memory shortage.

Fluency

So you say that FastMM will release the cached memory when needed? – Cesura

FastMM will not, but Windows will automatically do an equivalent of SetProcessWorkingSetSize, except better. – Fivefinger

Answer 5

I used this function just once, when I implemented TWebBrowser. This component took me so much memory even if I destroyed the instance.

Kapp

Hi daemon_X. Did you also used FastMM? – Cesura

Hi Altar, I thought it's already part of my Delphi distribution. I have D2009, but even though when I browsed for a while with that TWebBrowser (especially Google Maps), the memory increases a lot (about 300 MB higher) and I can't release this memory even after component instance release. But this was probably a cache problem of that browser and I had no idea how to flush it. – Kapp

But anyway, in my case I worked with interfaced COM object, what can be different from your standard MDI forms. I never had this problem with controls, which I could handle. Some memory always remains me reserved after freeing those objects, but it's always just a fraction. And FastMM (in the debug mode) doesn't tell me, there's something, what I should care of. – Kapp

TWebBrowser is a wrapper around IE and does not use the Delphi memory manager (FastMM or otherwise) much. IE is not written in Delphi and uses another memory manager. – Unclose

Yes, you're right, but if you measure it using standard task manager, then you'll see, that just your process increases its memory. I haven't tested it with running IE itself, but I woul'd say I'll get the same result - my process memory leak. – Kapp