Using System.IO.Pipelines together with Stream

Question

I consider replacing Stream-based IO in our application with System.IO.Pipelines to avoid unnecessary memory allocation (considered first RecyclableMemoryStream but it seems to be discontinued). But in some places I still have to use Stream because of the interface imposed by an external library. So my PipeWriter will need wrap its data in a Stream.

I didn't find much on this topic, but found a suggestion to use decorator pattern (C# Stream Pipe (Stream Spy)) in an answer to a different question. I am not sure it would be a right to hide Pipelines behind a Stream wrapper but can't find anything else that will let me piped data to a stream. Am I missing something?

UPDATE. Here's an example using SSH.NET open source library to upload a file to an FTP server (https://gist.github.com/DavidDeSloovere/96f3a827b54f20d52bcfda4fe7a16a0b):

using (var fileStream = new FileStream(uploadfile, FileMode.Open))
    {
        Console.WriteLine("Uploading {0} ({1:N0} bytes)", uploadfile, fileStream.Length);
        client.BufferSize = 4 * 1024; // bypass Payload error large files
        client.UploadFile(fileStream, Path.GetFileName(uploadfile));
    }

Note that we open a FileStream to read a file and then pass a Stream reference to an SftpClient. Can I use System.IO.Pipelines here to reduce memory allocation? I will still need to provide a Stream for SftpClient.

Ileneileo

Can you please explain a little more what do you wish to achieve. – Catchall

@Catchall I updated my question with the example. – Ileneileo

Answer 1

Disclaimer: I'm no expert, just putting together the pieces...

---

The answer (as of Janurary 2019) seems to be: there is no official support for this.

System.IO.Pipelines was created primarily for networking use cases. In fact, the pipelines code released in 2.1 had no support for any endpoints:

Here we need a bit of caveat and disclaimer: the pipelines released in .NET Core 2.1 do not include any endpoint implementations.

There is a proposed design for an API for generic stream adapter but that is part of the .NET Core 3.0 Milestone.

There even seems to be some reticence to implementing file-based pipelines access (AKA a FileStream pipelines equivalent). This is particularly disappointing since I too was hoping for pipelines powered file I/O.

I think your best bet at the moment is using the UsePipe() methods in https://github.com/AArnott/Nerdbank.Streams

---

Update: Here's another example I just found https://github.com/tulis/system-io-pipelines-demo/tree/master/src/SystemIoPipelinesDemo/SystemIoPipelinesDemo

---

Update: I had a go at making a Pipeline based file reader. You can read all about it here: https://github.com/atruskie/Pipelines.File.Unofficial

Essentially, from a performance perspective, using a pipeline stream adapter like Nerdbank.Streams is a good way to go!

Haemo

Thank very much for the clarification. It's disappointing that there is no support for file streams but at least I understand now why I wasn't able to find any examples of how to use pipelines with them. – Ileneileo

@Anthony Truskinger I wondered why synchronous I/O is significantly faster than async I/O in your benchmarks, and I think I spotted the reason for it: your file streams are not opened with FileOptions.Asynchronous. See github.com/atruskie/Pipelines.File.Unofficial/issues/1 – Ornithosis

I've updated the benchmarks to test with and without FileOptions.Asynchronous – Haemo

I haven't worked on any of this in a while, but from what I read from the GitHub link above (github.com/dotnet/corefx/issues/27246) it appears some adapters should be present in .NET Core 3.0 – Haemo

Answer 2

It seems to be a way now to leverage AsStream() extension off of PipeReader:

using (var fileStream = new FileStream(uploadfile, FileMode.Open))
    {
        Console.WriteLine("Uploading {0} ({1:N0} bytes)", uploadfile, fileStream.Length);
        client.BufferSize = 4 * 1024; // bypass Payload error large files
        var pipeReader = PipeReader.Create(fileStream);
        client.UploadFile(pipeReader.AsStream(), Path.GetFileName(uploadfile));
    }

T