Reduce the multiple copies of the same object in the Large Object Heap

Question

I am trying to upload the bytes of a large file (around 30MB) with the HTTPWebRequest to some server. The problem is that since the size of bytes is more than 85000, it is getting stored in the LargeObjectHeap(LOH). The problem is that my code is creating at-least 5 instances of the same object in the LOH, which then didn't get removed from the memory even after closing the response stream. Below is the code snippet which is causing this issue. Before this code block there was only one instance of the file in the LOH.

using (IO.Stream requestStream = webReqest.GetRequestStream())
{
    List<byte> uploadData = new List<byte>();
    uploadData.AddRange(Encoding.UTF8.GetBytes(stringContainingHeaderInfo));
    uploadData.AddRange(bytesOfTheLargeFile);

    byte[] fileFullData = uploadData.ToArray();
    requestStream.Write(fileFullData, 0, fileFullData.Length);
    requestStream.Close();

    uploadData.Clear();
    uploadData = null;
    fileFullData = null;
    fileEntityBytes = null;

   using (WebResponse webResponse = webRequest.GetResponse())
   {
      //Do Something with the response
   }
 }

Is there a way to further optimize this code block so that less number of copy gets created in the heap.

Bomke

What kind of server? Is it IIS? – Crum

Yes @RobertHarvey I am trying to upload it to a SharePoint site which will be running on IIS. – Bomke

Since you're already working with streams - why don't you just use streams throughout and small, fixed size buffers, rather than loading all of the data into a byte[]? – Gora

Also, there is no value in doing uploadData.Clear(); uploadData = null; fileFullData = null; fileEntityBytes = null; – Chemosynthesis

@JohnSaunders Actually when the multiple copies were getting created, I tried to set the variables to null which are no more required. But yes, you're right they didn't had any impact. – Bomke

Hey @JohnSaunders just wanted to add that while setting objects to null is irrelevant but, clearing the List collection has its immediate impact. If there were 5 objects in the LOH clearing the List brought the number down to 3. Since getting a response is a long time there's no need to keep the values in the List. – Bomke

Thanks @Gora that did the trick. When I directly wrote the bytes to the requestStream, only 2 instances gets in the LOH instead of 5. One for the original byte[] and other perhaps for the requestStream. – Bomke

@user2223043 the point is to have zero copies in the LOH. Copy the bytes in small chunks. More recent versions of the framework have a Stream.Copy method that does this for you. In other words, you don't need bytesOfTheLargeFile. – Synthesize

@Gora That should be turned into an answer. – Creolized

Answer 1

Microsoft has recently introduced LargeObjectHeapCompactionMode for GC in .NET 4.5.1 Please use the following link which might help you: http://msdn.microsoft.com/en-us/library/system.runtime.gcsettings.largeobjectheapcompactionmode(v=vs.110).aspx

Medium

Thnx for the info. This surely is a useful addition by Microsoft. However, it should be used as a last resort. Even Microsoft has issued the warning that “LOH compaction can be an expensive operation and should only be used after significant performance analysis, both to determine that LOH fragmentation is a problem, but also to decide when to request compaction.” So the better option is to try and avoid it as much as possible. Which is what I was asking here to reduce the duplicate entries in the LOH. If there are less copies in the LOH then fragmentation will not be an issue. – Bomke