Can I modify a constant in the RTL class System.Classes.TStream and rebuild it at runtime in Delphi XE6?

Question

I am trying to work around a known ugly performance limitation in System.Classes.pas, which has a 1980s era constant buffer limit ($F000) that looks like this:

function TStream.CopyFrom(const Source: TStream; Count: Int64): Int64;
const
  MaxBufSize = $F000;
....

This is causing major performance penalties in our Delphi application. In delphi XE2 through XE5, we were able to modify this and use one of the following approaches:

• I could modify the Delphi sources, and then, by invoking dcc32.exe from a batch file, rebuild the System.Classes.dcu file in the Delphi library folder. I realize this is ugly and I didn't like doing this, but I don't like this ugly performance issue in the RTL either, and our users can not live with the performance headaches it causes.

• I could try to put a modified system.classes.pas file somewhere in my project search path.

Neither of the above approaches is working for me in Delphi XE6, now, thanks probably to some internal compiler changes. The error I get in a minimal command line application that includes System.Contnrs in its uses clause, is this:

[dcc32 Fatal Error] System.Classes.pas(19600): F2051 Unit System.Contnrs was compiled with a different version of System.Classes.TComponent

The sample program to reproduce this problem (assuming you have modified System.Classes.pas and changed the MaxBufSize constant), is shown here:

program consoletestproject;

{$APPTYPE CONSOLE}

{$R *.res}

uses
   System.Contnrs,
   System.SysUtils;

var
  List:System.Contnrs.TObjectList;
begin
   WriteLn('Hello world');
end.

Again, this problem reproduces easily in Delphi XE6, but is not a problem in XE5, or earlier.

What is the recommended practice when you absolutely MUST work around a fundamental RTL or VCL limitation using a modified copy of System.Classes.pas or System.SysUtils.pas or some other very low level unit? (Yes, I know you should NOT do this if you don't have to, don't bother with a lecture.)

Are there a magic set of command line parameters you can use via "dcc32.exe" on the command line, to produce a modified DCU that will link properly with the application example above?

As a secondary question, are there .dcu files for which no source exists that will break when one tries to do this, in which case the answer to all of the above is, "you can't fix this, and if there's a bug in the RTL, you're out of luck"?

One possible workaround is to include "$(BDS)\source\rtl\common" in your project search path (or library path), forcing each broken (needing recompile) DCU to rebuild EACH time, but this seems ugly and wrong.

Bitolj

You can avoid this kind of issues patching the method on memory using a detour. – Anomalistic

Any samples or examples of that approach? I could use a code hook and replace the entire method? – Bitolj

Yes replace the entire method. I've posted code to do that many times here. So have others. There are libraries, but that is overkill for a complete swap out. – Tornado

@WarrenP, I just posted a sample – Anomalistic

@Warren FWIW, CopyFrom is probably not the real problem. The problem is probably a lack of buffering in the stream classes. – Tornado

Answer 1

You can overcome this limitation using a detour, try this sample which uses the Delphi Detours Library

First define the signature of the method to hook

var
 Trampoline_TStreamCopyFrom : function (Self : TStream;const Source: TStream; Count: Int64): Int64 = nil;

then implement the detour

function Detour_TStreamCopyFrom(Self : TStream;const Source: TStream; Count: Int64): Int64;
const
  MaxBufSize = 1024*1024; //use 1 mb now :)
var
  BufSize, N: Integer;
  Buffer: TBytes;
begin
  if Count <= 0 then
  begin
    Source.Position := 0;
    Count := Source.Size;
  end;
  Result := Count;
  if Count > MaxBufSize then BufSize := MaxBufSize else BufSize := Count;
  SetLength(Buffer, BufSize);
  try
    while Count <> 0 do
    begin
      if Count > BufSize then N := BufSize else N := Count;
      Source.ReadBuffer(Buffer, N);
      Self.WriteBuffer(Buffer, N);
      Dec(Count, N);
    end;
  finally
    SetLength(Buffer, 0);
  end;
end;

Finally replace the original function by the trampoline (you can use this code in the initialization part of some unit)

  Trampoline_TStreamCopyFrom     := InterceptCreate(@TStream.CopyFrom,   @Detour_TStreamCopyFrom);

And to release the hook you can use

 if Assigned(Trampoline_TStreamCopyFrom) then
   InterceptRemove(@Trampoline_TStreamCopyFrom);

Anomalistic

This works fantastically. It could also be used with MadExcept or other detour libraries, as users require. For MadExcept, for example, the code is: if madCodeHook.HookCode( @TStream.CopyFrom, @Detour_TStreamCopyFrom, @Trampoline_TStreamCopyFrom, 0) then HookActive := true; end; and MadCodeHook.UnhookCode(@Trampoline_TStreamCopyFrom); – Bitolj

You can write a basic devouring library in 20 lines of code. So long as you don't need trampolines. I wouldn't take a dependency on an external library just for this. Although I comment this library highly if you do need a trampoline. – Tornado

We already use MadExcept because we need to do a lot of low level hooks. Your basic devouring library would be a great question, I should ask that one, if you have the answer ready. – Bitolj

@WarrenP, you can found samples of basic implementations of a detour on these answers #16877477 , #17042992 – Anomalistic

Answer 2

Update 1: The suggestion below does not work for the Classes unit in XE6. The basic technique is sound and does solve similar problems. But for XE6, at least the Classes unit, it is not immediately obvious how to re-compile it.

This appears to be a fault introduced in XE6 because this technique is meant to work and is officially endorsed by Embarcadero: http://blog.marcocantu.com/blog/2014_august_buffer_overflow_bitmap.html

Update 2:

In XE7, this problem no longer exists. It would appear that whatever was broken in XE6 has been fixed.

---

You need the compiler options to match those used when the unit was compiled by Embarcadero. That's the reason why your implementation section only change fails when it seems like it ought to succeed.

Start a default project and use CTRL + O + O to generate these options. I get

{$A8,B-,C+,D+,E-,F-,G+,H+,I+,J-,K-,L+,M-,N-,O+,P+,Q-,R-,S-,T-,U-,V+,W-,X+,Y+,Z1}

when I do this in XE6.

Put that at the top of your copy of the unit and you should be good to go. You can probably get away with a cut-down subset of these, depending on your host project options. In my code I find that:

{$R-,T-,H+,X+}

suffices.

Tornado

It is possible that in XE6, the library is not compiled with the same defaults as a new project. I'm trying to figure out which thing is wrong in XE6 but have not identified it yet. That is, the string generated in XE6 on a new project when I do a Control+O+O does not fix the issue. – Bitolj

That's plausible. Classes is quite special. This approach has always worked for me in the past but maybe XE6 is different. – Tornado

This approach is known to work in XE2, XE3, XE4, and XE5 (just tested it with System.classes.pas) and appears NOT to work in the case of System.Classes.pas in XE6, perhaps due to some Very Low Level Compiler Magic. – Bitolj

Could be RTTI settings perhaps. – Tornado

Hmm. So perhaps this approach COULD work if I add some RTTI magic declaration. – Bitolj

I found I could get a different error by using various RTTI declarations, but not make all the problems go away. I believe the problem is fundamental, and due to the Magic status of System.Classes.pas – Bitolj

Very weird. I tried this just now. Behaves quite differently in XE6. So I guess you need a detour. Frankly, that's the way I would tackle this. I only ever recompile units if I cannot get it done with a detour. – Tornado

The vendor simply might have failed to distribute a matching source file, which actually have happened before. Non-matching breakpoints through the end is one non-deterministic indication. – Lenna

That's entirely possible! – Bitolj