FastMM4 says "The block footer has been corrupted"

Question

I have this function in our Delphi 7 application, which worked very well until I included FastMM4 v4.99 into the project. Once included, FastMM4 raised the following error message: "FastMM has detected an error during a FreeMem operation. The block footer has been corrupted." The execution halts in the FreeMem line.

function BinaryFieldToArrayOfWord( aBinaryField   : TVarBytesField;
                                   out aArrValues : TArrWord ) : Boolean;
var
  p : Pointer;
begin
  if not aBinaryField.IsBlob then
  begin
    GetMem( p, aBinaryField.DataSize );     
    try
      if aBinaryField.GetData( p ) then         
      begin
        // do something
      end;
    finally
      FreeMem( p, aBinaryField.DataSize );
    end;
  end; // if
end;

First I thought that there must be a bug in the function, but it is practically the same as this TField.GetData method example in the Delphi 7 help:

{ Retrieve the "raw" data from Field1 }
with Field1 do
begin
  if not IsBlob  { this does not work for BLOB fields }
  begin
    { Allocate space }
    GetMem(MyBuffer, DataSize);
    try
      if not GetData(MyBuffer) then
         MessageDlg(DisplayName + ' is NULL', mtInformation, [mbOK], 0)
      else 
         { Do something with the data };
    finally
      { Free the space }
      FreeMem(MyBuffer, DataSize);
    end;
  end;
end;

I found on the internet that the above error message is often because there is not enough space for the data. So I increased the size of the memory block and the error message vanished and the function worked as expected. After some experiment, I figured out that 2 bytes is required and enough:

    GetMem( p, aBinaryField.DataSize + 2 );

    FreeMem( p, aBinaryField.DataSize + 2 );

Although it solved my problem, I am not completely relaxed with this solution as I don't know its background. It would be nice to know the reason of this message. Does FastMM4 needs additional 2 bytes for its own footer?

UPDATE: After spending a day with debugging, I am now believe that there is a bug in the Delphi 7 TField.GetData method. It writes 2 zero bytes beyond the allocated memory block. I tried it with and without FastMM4, and it overwrites in both cases (so it is not a FastMM4 error). I also tried with typecasting the field as TVarBytesField, no difference. Here is the code I used with detailed comments, containing the results. My only remaining question: have they corrected this bug in later Delphis?

  procedure TfrmMain_PBC_TH.btnDEBUGClick(Sender: TObject);
  type
    TArrBytes = array of Byte;  
  var
    p       : Pointer;
    qryTest : TADOQuery;
  begin
    qryTest := TADOQuery.Create( Application );
    try
      // The type of the TQM_BinaryData.BinData column in the MSSQL database is a       varbinary(7900). 
      // Load the #168 binary data record. It contains exactly 7900 bytes, and the value of each byte is 255
      qryTest.Connection := MainConn;
      qryTest.SQL.Add('SELECT [BinData] FROM [TQM_BinaryData] WHERE [Id] = 168');
      qryTest.Open;

      // Allocate the memory block for this.
      GetMem( p, qryTest.FieldByName('BinData').DataSize );
      // DataSize is 7902 because all TVarBytesFields have 2 byte prefix in Delphi, containing the data length.
      // So the size of the allocated memory block is 7902 bytes - we are correct so far.
      try
        // Values of the first four bytes beyond the end of the memory block (memory thrash at this point) before GetData:
        // TArrBytes(p)[7902] = 96
        // TArrBytes(p)[7903] = 197
        // TArrBytes(p)[7904] = 219
        // TArrBytes(p)[7905] = 43

        // Critical point: get the data from the field with the Delphi GetData method
        qryTest.FieldByName('BinData').GetData( p );

        // Values after GetData:
        // TArrBytes(p)[0]    = 220    TArrBytes(p)[0] and TArrBytes(p)[1] contains the length of the binary data
        // TArrBytes(p)[1]    = 30     it is correct as 30 * 256 + 220 = 7900
        // TArrBytes(p)[2]    = 255    actual data starts
        // TArrBytes(p3[2]    = 255    
        // ...
        // TArrBytes(p)[7900] = 255
        // TArrBytes(p)[7901] = 255    actual data ends
        // TArrBytes(p)[7902] = 0      changed from 96!
        // TArrBytes(p)[7903] = 0      changed from 197!
        // TArrBytes(p)[7904] = 219    no change
        // TArrBytes(p)[7905] = 43     no change
      finally
        // Here FastMM4 throws the block footer corrupt error because GetData modified the 2 bytes after the allocated memory block 
        FreeMem( p );
      end;

      qryTest.Close;
    finally
      qryTest.Free;
    end;
  end;

After adding a Data Breakpoint, the call stack is:

    @FillChar(???,???,???)
    TDataSet.DataConvert($7D599770,$12F448,$7D51F7F0,True)
    VarToBuffer
    TCustomADODataSet.GetFieldData($7D599770,$7D51F7F0,True)
    TField.GetData($7D51F7F0,True)
    TfrmMain_PBC_TH.btnDEBUGClick($7FF7A380)
    TControl.Click
    TButton.Click

Flameproof

Answer 1

FastMM allocates some memory at the end of the block that you allocate and writes known values there. Then, when you deallocate, FastMM checks that those values are as expected. If not, then the error that you see is raised because FastMM knows that your code is writing beyond the end of the memory block.

So, something inside the try/finally block is writing beyond the end of the block of memory. And that's why increasing its size removes the FastMM error. Without seeing that code, we cannot tell what exactly is wrong and you will need to do some debugging to solve the problem.

You are quite right to be concerned by your "solution". Trial and error is never a reasonable way to program. You must find out why your program is writing beyond the end of the block.

One way to do that is to set a data breakpoint for the address immediately beyond the end of this block. That would force the debugger to break on the code that writes beyond the end of the block.

As an aside, you don't need to pass the second parameter to FreeMem. Doing so makes your code harder to maintain and serves absolutely no purpose. Pass just the pointer to FreeMem.

Saliferous

During the debugging I commented out everything between the try/finally except the aBinaryField.GetData( p ) row, so my final code looked like exactly as above. Removing the second parameter from FreeMem did not solve this problem. But if FastMM writes some known values at the end of my allocated memory block, and then I fill up with the block completely with another data (GetData does this IMHO) then the known values will be overwritten, even if I used only the allocated block, isnt it? So the +2 byte is for those FastMM known values. – Flameproof

I didn't say that the second parameter to FreeMem would solve the problem, just that it was pointless to pass it. Something is writing beyond the block end. – Saliferous

"But if FastMM writes some known values at the end of my allocated memory block, and then I fill up with the block completely with another data (GetData does this IMHO) then the known values will be overwritten, even if I used only the allocated block, isnt it?" Not really. What happens is, you ask for 10 bytes, say, but FastMM allocates 14, say. It gives you a pointer to the beginning of this block which you think has 10 bytes in. But it then writes known values in the extra 4 bytes at the end. If overwrite them, FastMM has found a bug. Something is writing beyond the end. – Saliferous

Updated my original post with further test results. – Flameproof

What is the value of qryTest.FieldByName('BinData').DataSize? – Saliferous

It is 7902, so it counts the +2 bytes. I have to allocate 7904 bytes to avoid the error message. – Flameproof