EF6 OutOfMemoryException Evaluating Entity with varbinary(max) Property

Question

In my WebAPI web app, I've added a varbinary(max) field to a table, and a byte[] field to the POCO (BatchCharge). This entity has a child entity (Charge).

Visual Studio 2013, Entity Framework 6, SQL Server 2014.

The data model is as follows:

public class BatchCharge
{
    public int ID { get; set; }
    public byte[] FileData { get; set; }
    public virtual ICollection<Charge> Charges { get; set; } 
}

public class Charge
{
    public int ID { get; set; }
    public DateTime CreatedUTC { get; set; }
    public decimal Amount { get; set; }
    public virtual BatchCharge BatchCharge { get; set; }
}

The mapping is done on the Charge entity (child) as follows:

public class ChargeMap : EntityTypeConfiguration<Charge>
{
    public ChargeMap()
    {
        // Primary Key
        HasKey(t => t.ID);

        // Table and Column Mappings
        ToTable("Charge");
        Property(t => t.ID).HasColumnName("ID");
        Property(t => t.ID).IsRequired();

        Property(t => t.CreatedUTC).HasColumnName("CreatedUTC");
        Property(t => t.CreatedUTC).IsRequired();

        Property(t => t.Amount).HasColumnName("Amount");
        Property(t => t.Amount).IsRequired();

        HasRequired(t => t.BatchCharge)
            .WithMany(t => t.Charges)
            .HasForeignKey(d => d.BatchChargeID);
    }
}

When retrieving a list of BatchCharges using the method below:

    [ActionName("GetBatchCharges")]
    [HttpGet]
    [Route("api/charges/batches")]
    [Authorize(Roles = "Administrator")]
    public HttpResponseMessage GetBatchCharges(int skip = 0,
        int take = 25,
        int statusFilter = 0)
    {
        try
        {
            var batchCharges = _centralDb.BatchCharges.AsQueryable();
            if (statusFilter > 0)
            {
                batchCharges = batchCharges.Where(bc => bc.StatusID == statusFilter);
            }

            // Page and list.
            var allBatchCharges = batchCharges.OrderByDescending(c => c.CreatedUTC);
            var totalCount = allBatchCharges.Count();
            var thePage = allBatchCharges.Skip(take * skip).Take(take).ToList();

            // Transform and return.
            var result = new
            {
                TotalCount = totalCount,
                CurrentPage = skip,
                BatchCharges = thePage.Select(c => MapperFactory.Mapper.Map<BatchCharge, BatchChargeDTO>(c)).ToList()
            };
            return Request.CreateResponse(HttpStatusCode.OK, result);
        }
        catch (Exception ex)
        {
            const string message = "Exception getting batch charges.";
            Logger.Error(ex, message);
            return Request.CreateErrorResponse(HttpStatusCode.InternalServerError, new HttpError(message));
        }
    }

I get an OutOfMemoryException. When I break on the line above and try to view the query results, I see instead the error "The function evaluation was disabled because of an out of memory exception.". See screenshot below.

This tells me the out of memory exception is occurring inside EF6.

I've seen several related answers about how to increase process memory. I don't believe this is the problem. The test data includes six rows in the BatchCharge (parent) table, and a max file size of 27 KB!

The file is a CSV file with the child entity (Charges) information. I have no problem uploading and saving BatchCharges with file data using this model. I've also used it successfully on small numbers of Charges (child entities) per BatchCharge.

The problem started when I uploaded a BatchCharge with 600 Charges as children. As I mentioned, it's not a file size issue, because the file size is 27KB.

If there is a circular reference here, it should happen in many other similarly configured relationships I have. Or is the byte array field creating the problem in conjunction with the parent/child relationship? If so, how?

EDIT: When I surround the code with a try/catch block, and run it step by step in a debugger, it doesn't throw an exception anywhere in my code! However, the response that reaches the browser includes a standard IIS unhandled exception page, with a server error as follows (run the snippet to see output):

         body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;} 
         p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}
         b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}
         H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }
         H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }
         pre {font-family:"Consolas","Lucida Console",Monospace;font-size:11pt;margin:0;padding:0.5em;line-height:14pt}
         .marker {font-weight: bold; color: black;text-decoration: none;}
         .version {color: gray;}
         .error {margin-bottom: 10px;}
         .expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }
         @media screen and (max-width: 639px) {
          pre { width: 440px; overflow: auto; white-space: pre-wrap; word-wrap: break-word; }
         }
         @media screen and (max-width: 479px) {
          pre { width: 280px; }
         }

    <body bgcolor="white">

            <span><H1>Server Error in '/' Application.<hr width=100% size=1 color=silver></H1>

            <h2> <i>Exception of type 'System.OutOfMemoryException' was thrown.</i> </h2></span>

            <font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">

            <b> Description: </b>An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

            <br><br>

            <b> Exception Details: </b>System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.<br><br>

            <b>Source Error:</b> <br><br>

            <table width=100% bgcolor="#ffffcc">
               <tr>
                  <td>
                      <code>

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.</code>

                  </td>
               </tr>
            </table>

            <br>

            <b>Stack Trace:</b> <br><br>

            <table width=100% bgcolor="#ffffcc">
               <tr>
                  <td>
                      <code><pre>

[OutOfMemoryException: Exception of type &#39;System.OutOfMemoryException&#39; was thrown.]
   System.IO.MemoryStream.set_Capacity(Int32 value) +89
   System.IO.MemoryStream.EnsureCapacity(Int32 value) +90
   System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) +326
   Microsoft.VisualStudio.Web.PageInspector.Runtime.Tracing.ArteryFilter.Write(Byte[] buffer, Int32 offset, Int32 count) +106
   System.Web.HttpWriter.FilterIntegrated(Boolean finalFiltering, IIS7WorkerRequest wr) +475
   System.Web.HttpResponse.FilterOutput() +154
   System.Web.CallFilterExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +80
   System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step) +247
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean&amp; completedSynchronously) +117
</pre></code>

                  </td>
               </tr>
            </table>

            <br>

            <hr width=100% size=1 color=silver>

            <b>Version Information:</b>&nbsp;Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.7.3056.0

            </font>

    </body>

Magen

What does your code look like? I have the feeling that you're pushing EF entities directly to the client. This serializes a single ChargeMap, including its many Charges (through property Charges) and then back to ChargeMap again (through property BatchCharge). This loops until the memory is full and, bam, OutOfMemoryException. But again, it's impossible to tell if this is the case without actual code (screenshot doesn't count as its incomplete). – Rejoin

I have edited the question to include the complete controller method. As you can see, I am not pushing EF entities directly to the client, but mapping to DTOs. Your description of a potential circular reference between Charge and BatchCharge (note: not ChargeMap - that is the mapping class), is correct, but I have plenty of other entity pairs with parent-child relationships and bi-directional navigation properties. Shouldn't EF handle this? Why don't I get out of memory exceptions in all other cases of this relationship? – Magen

It would be nice if you include the corresponding DTOs and AutoMapper configuration. May be you have circular DTOs here and not in other places, I don't know, but the problem doesn't seem to be EF related. – Orometer

Just to pin it down: can you get the data without mapping? – Mardellmarden

@IvanStoev Thanks, your question helped me find the problem: it was a mix of DTO and EF issue: I referenced EF entities from within the DTOs due to a copy-paste error. This caused the out of memory exception. Bi-directional references only in EF, or only in DTOs (assuming you configure the JSON.NET correctly) are not an issue in themselves. – Magen

Answer 1

Thanks to @IvanStoev, found the problem in the DTOs: they were referencing EF properties (highlighted below), which caused out of memory exceptions with larger data sets.

public class BatchChargeDTO
{
    public int ID { get; set; }
    public byte[] FileData { get; set; }
    // Problem is here: type should be ChargeDTO!!
    public ICollection<Charge> Charges { get; set; }
}

public class ChargeDTO
{
    public int ID { get; set; }
    public DateTime CreatedUTC { get; set; }
    public decimal Amount { get; set; }
    public int? BatchChargeID { get; set; }
    // Problem is here: type should be BatchChargeDTO!!
    public BatchCharge BatchCharge { get; set; }
}

Magen