How can I in C# stream.Read into unmanaged memory stream?

Question

I can read unmanaged memory in C# using UnmanagedMemoryStream, but how can I do the reverse?

I want to read from a managed stream directly into unmanaged memory, instead of first reading into a byte[] and then copying. I'm doing async stream reading on a large number of requests, so the added memory is significant (not to mention the additional copy).

Disappearance

The memory is already allocated in C++, but if it's really not possible I might have to change the existing code to accept a pointer to fixed managed memory :( – Disappearance

Answer 1

I think that it is not really possible. When you talk about a managed stream, I suppose you are refering to an instance of System.IO.Stream or a subclass hereof.

The members of this class take byte[] as a parameter, and that is a managed class.

I guess that the closest you can come is creating a managed byte[], and then creating a byte* in an unsafe block, and then passing the byte* to whatever needs an unmanaged reference:

unsafe function test()
{
    var buffer = new byte[1024];
    fixed (byte* bufferPtr = &buffer[0])
    {   
        // Read bytes and pass the ptr to a function that needs to
        // operate on data directly 
    }
}

But that is not exactly what you're asking for

edit: Be careful though. You mention something about async reads. As soon as you're outside the fixed boundary, the GC may move the array around in memory. And if you've passed the pointer to some "unsafe" function that continues to operate in a different thread, the pointer will point to an invalid memory location.

Astri

Answer 2

This actually isn't too hard if you have a known destination buffer and you know how big your data is. The key realization, once again, is that an UnmanagedMemoryStream is just a thin wrapper around a sequence of native bytes.

What you want to do is grab a pointer to your native destination buffer in the usual fashion. Then you can construct a writable UnmanagedMemoryStream on top of it. Copy your source stream to the destination stream and voila! A single copy has moved you from the managed memory world to the native memory world. In C++/CLI, it looks something like this:

void CopyStreamToNativePtr(Stream^ src, unsigned char* dst)
{
    // Assuming we want to start the copy at the beginning of src
    src->Seek(0, SeekOrigin::Begin);

    // Create an UnmanagedMemoryStream on top of the destination
    // with an appropriate size and capacity (We assume the buffer is
    // is sized to the source data in this function!)
    UnmanagedMemoryStream target(dst, src->Length, src->Length, FileAccess::Write);

    // Copy to your heart's content!
    src->CopyTo(%target);

    // We made the UnmanagedMemoryStream local so that we wouldn't have
    // to explicitly Dispose() of it.
}

Claudell

how to copy only a certain # of bytes from one stream to another? src->CopyTo will copy the entire stream. I only want to copy n bytes. – Leporid

Answer 3

The managed stream will always need a "managed" object reference to a valid byte[] object. However, you can use a managed byte[] which is pinned en lieu of a unmanaged memory allocation and therefore make it accessible as an unmanaged memory block also:

byte[] data = new byte[];
GCHandle pin = GCHandle.Alloc(data, GCHandleType.Pinned);
try {
  IntPtr dataUnmanagedPtr = Marshal.UnsafeAddrOfPinnedArrayElement(data, 0);
  // managed.Read(data, index, count);
  // use the unmanaged pointer here for unmanaged code
} finally {
  pin.Free(); // but make sure that no unmanaged code uses the pinned data anymore upon release
}

Cashandcarry

I found this to be the best solution for two reasons: 1. It does not require unsafe code 2. It prevents the GC from moving around our array in memory (for as long as we need) – Conchiolin

Answer 4

You can use C# and read file by Windows API directly into unmanaged memory.

If you want to read from FileStream, this could be helpful. Check FileStream implementation and do something similar.

var handle = Win32Native.SafeCreateFile(path, fAccess, share, secAttrs, mode, flagsAndAttributes, IntPtr.Zero);

fixed(byte* p = bytes) 
{   
   r = Win32Native.ReadFile(handle, p + offset, count, out numBytesRead, IntPtr.Zero);
}

Vullo