TLS/SSL with System.IO.Pipelines
Asked Answered
Q

1

13

I have noticed the new System.IO.Pipelines and are trying to port existing, stream based, code over to it. The problems with streams are well understood, but at the same time it features a rich echosystem of related classes.

From the example provided here, there is a small tcp echo server. https://blogs.msdn.microsoft.com/dotnet/2018/07/09/system-io-pipelines-high-performance-io-in-net/

A snippet of the code is attached here:

    private static async Task ProcessLinesAsync(Socket socket)
    {
        Console.WriteLine($"[{socket.RemoteEndPoint}]: connected");

        var pipe = new Pipe();
        Task writing = FillPipeAsync(socket, pipe.Writer);
        Task reading = ReadPipeAsync(socket, pipe.Reader);

        await Task.WhenAll(reading, writing);

        Console.WriteLine($"[{socket.RemoteEndPoint}]: disconnected");
    }

    private static async Task FillPipeAsync(Socket socket, PipeWriter writer)
    {
        const int minimumBufferSize = 512;

        while (true)
        {
            try
            {
                // Request a minimum of 512 bytes from the PipeWriter
                Memory<byte> memory = writer.GetMemory(minimumBufferSize);

                int bytesRead = await socket.ReceiveAsync(memory, SocketFlags.None);
                if (bytesRead == 0)
                {
                    break;
                }

                // Tell the PipeWriter how much was read
                writer.Advance(bytesRead);
            }
            catch
            {
                break;
            }

            // Make the data available to the PipeReader
            FlushResult result = await writer.FlushAsync();

            if (result.IsCompleted)
            {
                break;
            }
        }

        // Signal to the reader that we're done writing
        writer.Complete();
    }

    private static async Task ReadPipeAsync(Socket socket, PipeReader reader)
    {
        while (true)
        {
            ReadResult result = await reader.ReadAsync();

            ReadOnlySequence<byte> buffer = result.Buffer;
            SequencePosition? position = null;

            do
            {
                // Find the EOL
                position = buffer.PositionOf((byte)'\n');

                if (position != null)
                {
                    var line = buffer.Slice(0, position.Value);
                    ProcessLine(socket, line);

                    // This is equivalent to position + 1
                    var next = buffer.GetPosition(1, position.Value);

                    // Skip what we've already processed including \n
                    buffer = buffer.Slice(next);
                }
            }
            while (position != null);

            // We sliced the buffer until no more data could be processed
            // Tell the PipeReader how much we consumed and how much we left to process
            reader.AdvanceTo(buffer.Start, buffer.End);

            if (result.IsCompleted)
            {
                break;
            }
        }

        reader.Complete();
    }

    private static void ProcessLine(Socket socket, in ReadOnlySequence<byte> buffer)
    {
        if (_echo)
        {
            Console.Write($"[{socket.RemoteEndPoint}]: ");
            foreach (var segment in buffer)
            {
                Console.Write(Encoding.UTF8.GetString(segment.Span));
            }
            Console.WriteLine();
        }
    }

When using streams, you could easily add SSL/TLS to your code just by wrapping it in SslStream. How is this intended to be solved with Pipelines?

Quechua answered 12/12, 2018 at 21:13 Comment(4)
At first glance it looks like Kestrel uses Streams, including SslStream, to feed its Pipes. That's roughly covered in Mark Gravell's blog on Pipes - see the 'Pumping the pipe' section, and 'Here's one I made earlier' which explicitly mentions TLS.Faxun
Interesting, but given that pro-Pipelines arguments are somewhats negative towards Stream, doesnt it defeat its purpose?Quechua
There are a few things to be noted. 1. System.IO.Pipelines is just the first step of the overall development, so there are no APIs available to act as endpoints for System.IO.Pipelines (such as dotnetty, Kestrel or .NET sockets which directly accepts the data and pumps it in there). THey are planned for the future though, but the .NET Core team needs first to design a clean api around it and find the time to implemente these,Phan
2. The pipes are there to solve a specific problem of receiving chunked data that needs to be parsed in a most efficient and easy to use way, where the users of this api do not have to worry about how to pool continues buffers or glue together the chunks and the memory management behind it. and 3. to allow the process in the background to process the data as it comes from the wire (i.e. start parsing the heads as soon as we get the first bytes)Phan
R
9

Named pipes are a network protocol, just as HTTP, FTP, and SMTP are. Lets look at the .net Framework for some quick examples:

  • SSL is leveraged by HTTP connections automatically, depending on the base URI. If the URI beings with "HTTPS:", SSL is used.
  • SSL is leveraged by FTP connections manually, by setting the EnableSsl property to true prior to calling GetResponse().
  • SSL is leveraged by SMTP in the same way as FTP.

But what if we are using a different network protocol, such as pipes? Right off the bat we know there is nothing similar to an "HTTPS" prefix. Furthermore, we can read the documentation for System.IO.Piplines and see that there is no "EnableSsl" method. However, in both .NET Framework and .NET Core, the SslStream class is available. This class allows you to build a SslStream out of almost any available Stream.

Also available in both .NET Framework and .NET Core is the System.IO.Pipes Namespace. The classes available in the Pipes namespace are pretty helpful.

  • AnonymousPipeClientStream
  • AnonymousPipeServerStream
  • NamedPipeClientStream
  • NamedPipeServerStream
  • PipeStream

All of these classes return some kind of object which inherits from Stream, and can thus be used in the constructor for a SslStream.

How does this relate back to the System.IO.Piplines Namespace? Well... it doesn't. None of the Classes, Structs, or Interfaces defined in the System.IO.Pipelines namespace inherit from Stream. So we can not use the SslStream class directly.

Instead, we have access to PipeReaders and PipeWriters. Sometimes we only have one of these available to us, but lets consider a bi-directional pipe so that we have access to both at the same time.

The System.IO.Piplines namespace helpfully provides an IDuplexPipe interface. If we want to wrap the PipeReader and PipeWriters in an SSL stream, we will need to define a new type that implements IDuplexPipe.

In this new type:

  • We will define an SslStream.
  • We will use generic pipes as input and output buffers.
  • The PipeReader will use the reader of the input buffer. We will use this input buffer to get data from the SSL stream.
  • The PipeWriter will use the writer of the output buffer. We will use this output buffer to send data to the SSL stream.

Here is an example in pseudocode:

SslStreamDuplexPipe : IDuplexPipe
{ 
    SslStream sslStream;
    Pipe inputBuffer;
    Pipe outputBuffer;

    public PipeReader Input = inputBuffer.Reader;
    public PipeWriter Output = outputBuffer.Writer;

    ReadDataFromSslStream()
    {
        int bytes = sslStream.Read(new byte[2048], 0, 2048);
        inputBuffer.Writer.Advance(bytes)
        inputBuffer.Writer.Flush();
    }

    //and the reverse to write to the SslStream 
 }

As you can see, we are still using the SslStream class from the System.Net.Security namespace, it just took us a few more steps.

Does this mean that you are basically still using streams? Yep! But, once you have fully implemented your SslStreamDuplexPipe class, you get to work with only pipes. No need to wrap an SslStream around everything.

Marc Gravell wrote a much, much, more detailed explanation of this. The first of the 3 parts can be found here: https://blog.marcgravell.com/2018/07/pipe-dreams-part-1.html

Additionally, you can read about the various .NET classes mentioned:

Redoubtable answered 20/12, 2018 at 17:6 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.