Delphi: Copy FileStream to MemoryStream

Question

I want to copy a part of a FileStream to a MemoryStream.

FileStream.Write(Pointer(MemoryStream)^, MemoryStream.Size);
FileStream.Read(Pointer(MemoryStream)^, count);

Is that right? It isn't working for me.

Bloodstream

to format your code, select it in the editor and press Control-K. – Granary

Answer 1

You have to Read() from the FileStream into a separate buffer and then Write() that to the MemoryStream, ie:

var
  Buffer: PByte;

GetMem(Buffer, NumberOfBytes);
try
  FileStream.ReadBuffer(Buffer^, NumberOfBytes);
  MemoryStream.WriteBuffer(Buffer^, NumberOfBytes);
finally
  FreeMem(Buffer);
end;

Since you are dealing with two TStream objects, it would be easier to use the TStream.CopyFrom() method instead, ie:

MemoryStream.CopyFrom(FileStream, NumberOfBytes);

Alumroot

Thanks a lot! I used the CopyFrom, but I think your solutin will give me a better performance. Thanks again. – Bloodstream

the TStream.CopyFrom() method uses a similar read-into-buffer-than-write-it approach internally, but does so with more error handling and buffer management than what I showed. – Alumroot

ReadBuffer should be used when the number of bytes to read is known and fixed, better to use Read - it can actually return less bytes than the buffer size when there's no more. I'd write: BytesRead := FileStream.Read(Buffer^, NumberOfBytes); MemoryStream.Write(Buffer^, BytesRead); – Flores

Since the OP is wanting to copy a portion of a file, presumably the OP does know ahead of time how many bytes to copy. – Alumroot

Just wanted to help anyone who might encounter the same problem as I did. Using solution Remy posted is MUCH, MUCH faster for very large files (> 1.5GB) than using MemoryStream.CopyFrom method. I spent 2 days figuring it out... :( – Ballonet

Better stick to Write, CopyFrom seems about 5 times slower. – Wildebeest

Answer 2

The following solution does not use a separate buffer as the solution that was already posted. Instead it writes directly to the buffer of the destination memory stream. This is faster because the other solution copies twice, first into the temporary buffer and finally into the memory stream.

...
try
  MemoryStream.SetSize(NumberOfBytes); // Allocating buffer
  FileStream.ReadBuffer(MemoryStream.Memory^, NumberOfBytes);
finally
  MemoryStream.Free();
...

This works because SetSize also allocates the buffer of the memory stream. See SetSize documentation.

Use SetSize to set the Size of a memory stream before filling it with data. SetSize allocates the memory buffer to hold NewSize bytes [...].

---

I also tested the solution with CopyFrom, but that solution is very slow working with giant files because it seems to use a very small buffer.

If files are to great to read directly with the method above it can be done with an own function that reads chunks directly to the memory stream. In order to be faster than the CopyFrom method, these chunks should be bigger. The following code uses a flexible buffer e.g. 256 MiB. Please feel free to make a function out of it.

var
  ...
  MemoryStreamPointer: Pointer;
  BlockSize: Integer;
  BytesToRead: Integer;
  BytesRead: Integer;
  RemainingBytes: Integer;

begin
  ...
  BlockSize := 256 * 1024 * 1024; // 256 MiB block size

  MemoryStream.SetSize(NumberOfBytes); // Allocating buffer
  MemoryStreamPointer := MemoryStream.Memory;

  RemainingBytes := NumberOfBytes;
  while RemainingBytes > 0 do
  begin
    BytesToRead := min(RemainingBytes, BlockSize);
    BytesRead := FileStream.Read(MemoryStreamPointer^, BytesToRead);
    RemainingBytes := RemainingBytes - BytesRead;
    MemoryStreamPointer := Pointer(NativeInt(MemoryStreamPointer) + BytesRead);
  end;
  ...
end;

Please take caution that the above code contains no error handling. Further think about setting the file streams position to 0 before reading.

Oppose