OutOfMemoryException when creating multiple byte arrays

Question

I'm constantly hitting an OutOfMemoryException inside a method that creates and processes some byte arrays. The code looks like this:

1. Create MemoryStream to get some data (about 60MB).
2. Create byte array (same size as MemoryStream, about 60MB)
3. Fill the array with bytes from memory stream
4. Close MemoryStream
5. Process data from byte array
6. Leave method

When this method is called like 20-30 times I get OutOfMemoryException right where the byte array is allocated. But I don't think it's the system memory issue. Application memory usage is around 500MB (private working set) and the test machine is 64-bit with 4GB of RAM.

Is it possible that memory used by the byte array or MemoryStream is not released after method finishes? But then, it does not look like this memory is allocated for the process as private working set is only 500MB or so.

What may cause the OutOfMemoryException when creating large byte array (60MB) beside physical memory shortage?

[Edited to add code sample] Source comes from PdfSharp lib

Exception is thrown at line byte[] imageBits = new byte[streamLength]; It indeed looks like LOH fragmentation issue.

/// <summary>
/// Reads images that are returned from GDI+ without color palette.
/// </summary>
/// <param name="components">4 (32bpp RGB), 3 (24bpp RGB, 32bpp ARGB)</param>
/// <param name="bits">8</param>
/// <param name="hasAlpha">true (ARGB), false (RGB)</param>
private void ReadTrueColorMemoryBitmap(int components, int bits, bool hasAlpha)
{
  int pdfVersion = Owner.Version;
  MemoryStream memory = new MemoryStream();
  image.gdiImage.Save(memory, ImageFormat.Bmp);
  int streamLength = (int)memory.Length;

  if (streamLength > 0)
  {
    byte[] imageBits = new byte[streamLength];
    memory.Seek(0, SeekOrigin.Begin);
    memory.Read(imageBits, 0, streamLength);
    memory.Close();

    int height = image.PixelHeight;
    int width = image.PixelWidth;

    if (ReadWord(imageBits, 0) != 0x4d42 || // "BM"
        ReadDWord(imageBits, 2) != streamLength ||
        ReadDWord(imageBits, 14) != 40 || // sizeof BITMAPINFOHEADER
        ReadDWord(imageBits, 18) != width ||
        ReadDWord(imageBits, 22) != height)
    {
      throw new NotImplementedException("ReadTrueColorMemoryBitmap: unsupported format");
    }
    if (ReadWord(imageBits, 26) != 1 ||
      (!hasAlpha && ReadWord(imageBits, 28) != components * bits ||
       hasAlpha && ReadWord(imageBits, 28) != (components + 1) * bits) ||
      ReadDWord(imageBits, 30) != 0)
    {
      throw new NotImplementedException("ReadTrueColorMemoryBitmap: unsupported format #2");
    }

    int nFileOffset = ReadDWord(imageBits, 10);
    int logicalComponents = components;
    if (components == 4)
      logicalComponents = 3;

    byte[] imageData = new byte[components * width * height];

    bool hasMask = false;
    bool hasAlphaMask = false;
    byte[] alphaMask = hasAlpha ? new byte[width * height] : null;
    MonochromeMask mask = hasAlpha ?
      new MonochromeMask(width, height) : null;

    int nOffsetRead = 0;
    if (logicalComponents == 3)
    {
      for (int y = 0; y < height; ++y)
      {
        int nOffsetWrite = 3 * (height - 1 - y) * width;
        int nOffsetWriteAlpha = 0;
        if (hasAlpha)
        {
          mask.StartLine(y);
          nOffsetWriteAlpha = (height - 1 - y) * width;
        }

        for (int x = 0; x < width; ++x)
        {
          imageData[nOffsetWrite] = imageBits[nFileOffset + nOffsetRead + 2];
          imageData[nOffsetWrite + 1] = imageBits[nFileOffset + nOffsetRead + 1];
          imageData[nOffsetWrite + 2] = imageBits[nFileOffset + nOffsetRead];
          if (hasAlpha)
          {
            mask.AddPel(imageBits[nFileOffset + nOffsetRead + 3]);
            alphaMask[nOffsetWriteAlpha] = imageBits[nFileOffset + nOffsetRead + 3];
            if (!hasMask || !hasAlphaMask)
            {
              if (imageBits[nFileOffset + nOffsetRead + 3] != 255)
              {
                hasMask = true;
                if (imageBits[nFileOffset + nOffsetRead + 3] != 0)
                  hasAlphaMask = true;
              }
            }
            ++nOffsetWriteAlpha;
          }
          nOffsetRead += hasAlpha ? 4 : components;
          nOffsetWrite += 3;
        }
        nOffsetRead = 4 * ((nOffsetRead + 3) / 4); // Align to 32 bit boundary
      }
    }
    else if (components == 1)
    {
      // Grayscale
      throw new NotImplementedException("Image format not supported (grayscales).");
    }

    FlateDecode fd = new FlateDecode();
    if (hasMask)
    {
      // monochrome mask is either sufficient or
      // provided for compatibility with older reader versions
      byte[] maskDataCompressed = fd.Encode(mask.MaskData);
      PdfDictionary pdfMask = new PdfDictionary(document);
      pdfMask.Elements.SetName(Keys.Type, "/XObject");
      pdfMask.Elements.SetName(Keys.Subtype, "/Image");

      Owner.irefTable.Add(pdfMask);
      pdfMask.Stream = new PdfStream(maskDataCompressed, pdfMask);
      pdfMask.Elements[Keys.Length] = new PdfInteger(maskDataCompressed.Length);
      pdfMask.Elements[Keys.Filter] = new PdfName("/FlateDecode");
      pdfMask.Elements[Keys.Width] = new PdfInteger(width);
      pdfMask.Elements[Keys.Height] = new PdfInteger(height);
      pdfMask.Elements[Keys.BitsPerComponent] = new PdfInteger(1);
      pdfMask.Elements[Keys.ImageMask] = new PdfBoolean(true);
      Elements[Keys.Mask] = pdfMask.Reference;
    }
    if (hasMask && hasAlphaMask && pdfVersion >= 14)
    {
      // The image provides an alpha mask (requires Arcrobat 5.0 or higher)
      byte[] alphaMaskCompressed = fd.Encode(alphaMask);
      PdfDictionary smask = new PdfDictionary(document);
      smask.Elements.SetName(Keys.Type, "/XObject");
      smask.Elements.SetName(Keys.Subtype, "/Image");

      Owner.irefTable.Add(smask);
      smask.Stream = new PdfStream(alphaMaskCompressed, smask);
      smask.Elements[Keys.Length] = new PdfInteger(alphaMaskCompressed.Length);
      smask.Elements[Keys.Filter] = new PdfName("/FlateDecode");
      smask.Elements[Keys.Width] = new PdfInteger(width);
      smask.Elements[Keys.Height] = new PdfInteger(height);
      smask.Elements[Keys.BitsPerComponent] = new PdfInteger(8);
      smask.Elements[Keys.ColorSpace] = new PdfName("/DeviceGray");
      Elements[Keys.SMask] = smask.Reference;
    }

    byte[] imageDataCompressed = fd.Encode(imageData);

    Stream = new PdfStream(imageDataCompressed, this);
    Elements[Keys.Length] = new PdfInteger(imageDataCompressed.Length);
    Elements[Keys.Filter] = new PdfName("/FlateDecode");
    Elements[Keys.Width] = new PdfInteger(width);
    Elements[Keys.Height] = new PdfInteger(height);
    Elements[Keys.BitsPerComponent] = new PdfInteger(8);
    // TODO: CMYK
    Elements[Keys.ColorSpace] = new PdfName("/DeviceRGB");
    if (image.Interpolate)
      Elements[Keys.Interpolate] = PdfBoolean.True;
  }
}

Aerodyne

#1 question, are you properly disposing your streams? Also, you might want to provide some of your code. – Sero

@Sero there is nothing to Dispose besides MemoryStream. Which is being closed. And as a matter of fact MemoryStream.Close() is just a call to MemoryStream.Dispose(true) followed by GC.SuppressFinalize(this). Byte arrays are not disposable, so there is nothing more I can do. As for the code... it's quite complex and comes from PdfSharp lib. It's not my code but I'm trying to understand and fix the issues it renders. – Aerodyne

I was reading up on some of the fragmentation issues. It seems that on 64 bit windows, the underlying memory manager should not expose this behavior. Is it possible that your process runs in WOW64, i.e., that you compiled for x86 only? I updated my answer with more info on the fragmentation issue. – Furst

Quite a few copies and helper arrays flying around here. This might be a rare occasion where calling GC.Collect() might help, preferably after or at the end of ReadTrueColorMemoryBitmap(). – Smutch

We all agree that the problem is with the GC failing to recover, or fragmenting, the managed memory heap. Then, a good alternative could be using Unmanaged memory: System.IO.UnmanagedMemoryStream The problem is you have to know beforehand the space required. Or, at least, have an upper limit. The documentation clearly says that this stream doesn't allocate memory on the heap. Another problem being your program needs security settings that allows doing this. – Maisiemaison

Answer 1

I hope you are using MemoryStream.GetBuffer() and that you are not copying to a new array.

Your main problem is not a direct lack of memory but fragmentation of the LOH. That can be a tricky problem, the main issue is allocating large buffers of varying size. Items on the LOH are collected but not compacted.

Solutions might be:

• first make sure you are not blocking anything from being collected. use a profiler.
• try to reuse your buffer(s).
• round allocations up to a set of fixed numbers.

The last 2 both require you to work with oversized arrays, may take some work.

Smutch

Unfortunately code author indeed copies MemoryStream into byte array. – Aerodyne

@SiliconMind: one thing that comes to mind is to use a small memorystream and to write only small chunks to the target byte array, This prevents at least one of the two large arrays to exist and saves a lot in LOH fragmentation. – Furst

I've already tried MemoryStream.GetBuffer() instead and that seems to help a bit but only to some extent. Reusing buffers is virtually impossible, as data may be as small as few bytes and as large as... well very large. The code processes bitmaps which may be 1x1 pixel or 10000x10000 pixels large. Anyway it seems like this is indeed the LOH issue. What worries me is that calling GC.Collect() does not solve the problem and OutOfMemoryException is thrown sooner or later. – Aerodyne

Can you change the code at all? And you really need to post (bits of) it for better answers. – Smutch

@HenkHolterman: I've edited question to include code sample. Yes I can change the code to fix the issue. As a matter of fact I would prefer to use Bitmap.LockBits but what I really want from you guys is the explanation why :) I'd like to understand and know the real cause of these OutOfMemoryExceptions. – Aerodyne

Accepting this answer as it seems that it is indeed the LOH fragmentation issue. Using MemoryStream.GetBuffer() helps, but it is possible to use Bitmap.LockBits instead and get even better results. – Aerodyne

Ok, good that you found some solution. Did you give GC.Collect() a try? – Smutch

@HenkHolterman In this particular case GC.Collect() did not help at all. I was using it anyway in my own code after processing each PDF page. Overall it helped, but the code that caused LOH issues still did. However it is possible that if I wouldn't call GC.Collect() I wouldn't get OutOfMemoryException after 20-30 calls to that method... probably I would get that exception after 10th call :) – Aerodyne

@HenkHolterman I've also experimented with static reusable buffer (I had to use static as using instance "global" buffer would require too many code changes). I'd grew the buffer each time it was too small to hold whole data. Unfortunately that also did not help. Probably there was still to many byte arrays flying around when the buffer grew. Maybe if I'd create some kind of wrapper class for chunked small sized byte arrays that would act as a large buffer, I could avoid the LOH fragmentation issues too. – Aerodyne

Answer 2

Disposing has been suggested, but only for the MemoryStream, this will do you nothing. Also, GC.Collect has been suggested. This may not help, as it seems your memory is not substantially growing. Be careful, calling GC.Collect can be an expensive operation.

Fragmentation

It really looks like you're hitting the infamous Large Object Heap fragmentation issue. This may well be caused by often allocating and freeing chunks of 60MB of memory. If the LOH gets fragmented, it stays fragmented. This is a major issue with long-running .NET applications and a reason why ASP.NET is often configured to restart at intervals.

• Large object fragmentation (Stackoverflow)
• Excellent explanation and easy to follow code on Codeproject.
• Reported to Microsoft, but nothing will be done yet.

Prevent OutOfMemoryException

See the above CodeProject article on how to do this. The trick is, use MemoryFailPoint and catch the InsufficientMemoryException. This way, you can gracefully degrade and your application will not become unstable.

Possible general solution

Make sure your large objects live as long as possible. Reuse the buffer. Allocate it once with sufficient size and zero the buffer when you need it again. This way you won't run into any other memory issues. When your objects stay below 85k in size, they generally don't run into the LOH and won't clutter.

64 bit machines should not have this issue

EDIT: According to this post (workaround tab) and this post (see open comment), this issue should not appear on 64 bit machines. Since you say you run your code on 64 bit machines, perhaps you compiled with the configuration set to x86?

Furst

although my test machine is 64-bit, the target platform is unfortunately 32-bit WinXP. It also seems that the issue described on CodeProject has been fixed long time ago with a patch from MS. – Aerodyne

@SiliconMind: no, the issue is not fixed. It's just that in 64 bit mode, it will occur much less frequent because of the larger virtual address space (not the physical address space). Why do you target 32 bit? If you test machine is 64 bit and runs, say, Windows 7, build your project for 64 bit and see if it makes a difference. – Furst

Answer 3

Your Heap memory is throwing this exception, try calling GC.Collect() at the end to free up resources. You can also MemoryProfiler to find out the heap memory usage, it comes with 14 days trial

Hulse

Answer 4

Try enclosing it in a using(MemoryStream x = ...) { } block, which will dispose the object for you.

Although Close is supposed to Dispose the object, according to .NET guidelines, perhaps it's different in MemoryStream.

Maisiemaison

In an earlier, now deleted, answer from nonnb, this was suggested. However, as Jon Skeet commented there, disposing a MemoryStream does not free the buffer (it merely sets the Open and Writable flags to false). The GC needs to do kick in to free the byte arrays. – Furst

Of course. You are supposed to let the GC dispose of your memory, or use GC.Collect() not too frequently. But the using block warranties that the stream is freed even if something goes wrong (i.e. an Exception is thrown, ro you forget to close/dispose the object). – Maisiemaison

I would agree, but in case of MemoryStream it is simply not true. The only resource the MS holds is a byte array, no unmanaged resources. It's a good practice to use using{..} when objects are disposable, but in some cases, an object is only IDisposable because the inheritance chain demands it. In this case, Dispose() is virtually a no-op and doesn't free up resources. Check Reflector if you don't belief my claims ;). Note: it doesn't hurt to use using here, but it doesn't do anything either. – Furst

Thans for your comment. You're completely right. In many cases it's unimportant but I'm sure you agree it helps just a little bit because the reference to the object disappears when leaving the using block scope, so it can be collected by the GC. That would happend in any kind of scope (for body or whatever). This could get useful in the case of many streams being created and not getting out of scope. Please, correct me if I'm wrong. – Maisiemaison

Answer 5

I have same problem in code like below:

ImageData = (byte[])pDataReader.qDataTable.Rows[0][11];
if (ImageData != null)
{
    ms = new MemoryStream(ImageData);
    button1.BackgroundImage = Image.FromStream(ms);
}
ImageData = null;

ImageData = (byte[])pDataReader.qDataTable.Rows[0][12];
if (ImageData != null)
{
    ms = new MemoryStream(ImageData);
    button1.BackgroundImage = Image.FromStream(ms);
}
ImageData = null;
ms.Close();

Removing ms.Close(); solves the problem.
I think the problem rise because you define MemoryStream memory = new MemoryStream(); outside of if block and close it in the if block, same as me!

Middelburg

Do you really get an "out of memory" exception when closing a stream that was not assigned? The code in the question always creates and fills the stream before the "if", so I cannot see any relevance of your "answer" for the original question. – Schaller

@Alessio Cantarella; yes, I get an "out of memory" exception and as I said I don't know , why, maybe you can try the SiliconMind code and answer the question. It assigned, otherwise cannot build. I edit my answer to add two photos please review them – Middelburg

Answer 6

First of all, it is not recommended that you read/write large FILESTREAM data through TSQL. The recommeded approach is by using the Win32/DOTNET APIs provided by SQL server. The code that I posted above shows how to access the FILESTREAM data using the SqlFileStream() .net class. It also shows how to send the data in smaller chunks.

Providing that your total memory is sufficient, you can prevent Out of memory exceptions resulting from LOH fragmentation by creating a bunch of smaller arrays, and wrapping them in a single IList, or some other indexed interface.

Phillips

It's not SQL-related issue. There is no SQL. What are you talking about? – Aerodyne