SerialPort.BaseStream.ReadAsync drops or scrambles bytes when reading from a USB Serial Port

Question

Edit: I've added the sending code and an example of the received output I'm getting.

---

I am reading data from a USB "virtual" serial port connected to an embedded system. I have written two methods for receiving the data, one synchronous and one asynchronous. The synchronous one works, and the asynchronous one loses or scrambles a little bit of the incoming data. I cannot tell why the second one fails.

The method that works calls SerialPort.Read with a read timeout set to zero, and it requests everything in the receive buffer. I check the return value to see how many bytes were actually read and then put the data into a circular buffer for use elsewhere. This method is called by a timer interrupt, and it works perfectly to receive serial data (typically at rates above 1.6 Mbps with no data loss). However, the polling timer has become a problem for me and I would prefer to receive the data asynchronously wrt the rest of my code.

The method that loses data awaits ReadAsync on the serial port BaseStream and loops until cancelled. This approach sort of works, but it often returns the leading byte of a packet out of order, loses a single byte fairly frequently (approximately once every few thousand data bytes), and occasionally loses hundreds of sequential bytes from a packet.

It is possible that there are two completely different problems here, because the loss of larger chunks of data loss seem to be correlated with higher data rates and heavier system activity. That particular part of the problem could potentially be due to buffer overruns -- perhaps through a failure of USB handshaking when the USB scheduler encounters a delay -- but the example I am showing here has only a very small amount of data being transferred at 50 msec intervals, and the system is idle except for this test routine.

I have observed that ReadAsync frequently returns the first byte of a packet on one read, and the remainder of the packet on the next read. I believe this is expected behavior because MSDN says that if no data is available for some period of time, ReadAsync will return with the first byte it receives. However, I think this behavior is somehow related to my problem because when a single byte is missing or out of order, it is "always" that first byte, with the rest of the packet arriving normally.

When the packets are small, the "missing" byte from the front of the packet often (but not always) appears to be delivered in the next read after the remainder of the packet, and this just makes absolutely no sense to me. With larger packets this still happens occasionally, but more often the first byte is just missing when the packets are large.

I've searched far and wide, and have read every SO question I could find on this topic. I found other people with what appears to be a similar problem (ex: SerialPort.BaseStream.ReadAsync missing the first byte), but nobody with any accepted or even plausible solutions.

Ben Voigt (http://www.sparxeng.com/blog/software/must-use-net-system-io-ports-serialport) and others who really seem to know serial comms have recommended the use of ReadAsync on the basestream, and Microsoft's IOT team has also recommended this approach, so I have to believe the approach should work.

Question 1: Why is my code using ReadAsync on a USB Serial BaseStream dropping /scrambling bytes?

Question 2: If ReadAsync cannot be made to reliably return all the bytes received bytes in the correct order, can I just put an async wrapper around the traditional SerialPort.Read and await / loop it so I don't have to poll from a timer? I've read that this is a bad idea, but I've also read that the SerialPort class is internally asynchronous, so perhaps that makes it OK? Or is my only alternative to put this on a worker thread and just let it spend all its time waiting?

My code is below. I have set serialPort1.ReadTimeout = 0; and serialPort1.BaseStream.ReadTimeout = 0; (and I have tried other durations). I have enabled RTS and DTR, and since this is a USB_serial port it should handle handshake internally, and it certainly appears to do so when I read synchronously -- but perhaps that's not true when I read from the BaseStream?

Here is the first method:

// this method works perfectly when called from a timer.
// SerialPort.ReadTimeout must be set to zero for this to work.
// It handles incoming bytes reliably at rates above 1.6 Mbps.

private void ReadSerialBytes()
{
    if (!serialPort1.IsOpen)
        return;

    if (serialPort1.BytesToRead > 0)
    {
        var receiveBuffer = new byte[serialPort1.ReadBufferSize];

        var numBytesRead = serialPort1.Read(receiveBuffer, 0, serialPort1.ReadBufferSize);
        var bytesReceived = new byte[numBytesRead];
        Array.Copy(receiveBuffer, bytesReceived, numBytesRead);

        // Here is where I audit the received data.
        // the NewSerialData event handler displays the 
        // data received (as hex bytes) and writes it to disk.
        RaiseEventNewSerialData(bytesReceived);

        // serialInBuffer is a "thread-safe" global circular byte buffer 
        // The data in serialInBuffer matches the data audited above.
        serialInBuffer.Enqueue(bytesReceived, 0, numBytesRead);
    }
}

Here is the second method, Edited to remove the tail recursion noted by @Lucero. Now I won't run out of memory :) but the original data loss problem, of course, remains.

// This method is called once after the serial port is opened,
// and it repeats until cancelled. 
// 
// This code "works" but periodically drops the first byte of a packet, 
// or returns that byte in the wrong order.
// It occasionally drops several hundred bytes in a row.
private async Task ReadSerialBytesAsync(CancellationToken ct)
{
    while((!ct.IsCancellationRequested) && (serialPort1.IsOpen))
    {
        try
        {
            serialPort1.BaseStream.ReadTimeout = 0;
            var bytesToRead = 1024;
            var receiveBuffer = new byte[bytesToRead];
            var numBytesRead = await serialPort1.BaseStream.ReadAsync(receiveBuffer, 0, bytesToRead, ct);

            var bytesReceived = new byte[numBytesRead];
            Array.Copy(receiveBuffer, bytesReceived, numBytesRead);

             // Here is where I audit the received data.
             // the NewSerialData event handler displays the 
             // data received (as hex bytes) and writes it to disk.
             RaiseEventNewSerialData(bytesReceived);

            // serialInBuffer is a "thread-safe" global circular byte buffer 
            // The data in serialInBuffer matches the data audited above.
            serialInBuffer.Enqueue(receiveBuffer, 0, numBytesRead);
        }
        catch (Exception ex)
        {
            MessageBox.Show("Error in ReadSerialBytesAsync: " + ex.ToString());
            throw;
        }
    }
}

Here is C++ code from the sending system (teensy 3.2 with an ARM chip). It sends a sequence of bytes from 00 through FF, repeated every 50 msec.

 void SendTestData()
 {
    byte asyncTestBuffer[256] = { 0 };
    for (int i = 0; i < 256; i++)
        asyncTestBuffer[i] = i;

    while(true)
    {
    Serial.write(asyncTestBuffer, sizeof(asyncTestBuffer));
    delay(50);
    }
}

The traditional synchronous SerialPort.Read (called from a timer) receives each block completely exactly as expected, with no data loss. It looks like this, over and over:

=====
32 msec => Received 256 bytes 
000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====

Now here is what SerialPort.BaseStream.ReadAsync receives. In another version I appended a terminal packet sequence number to prove that when I see a zero followed by another zero, there's not really an entire missing packet between them. Packet sequence numbers were all present, so the leading byte really does seem to be missing or delivered out of order.

7 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
5 msec => Received 1 bytes 
00
=====
55 msec => Received 1 bytes 
00
=====
4 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
42 msec => Received 1 bytes 
00
=====
5 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
68 msec => Received 1 bytes 
00
=====
7 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
31 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
9 msec => Received 1 bytes 
00
=====
33 msec => Received 1 bytes 
00
=====
10 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
55 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
12 msec => Received 1 bytes 
00
=====
12 msec => Received 1 bytes 
00
=====
15 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
68 msec => Received 255 bytes 
0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====
16 msec => Received 1 bytes 
00
=====
14 msec => Received 256 bytes 
000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF
=====

I've spent a couple of weeks tracking down this problem, which originally manifested itself in bizarre behavior from a product under development. I am pretty sure I must be doing something wrong, but I just can't see it, and at this point I am quite desperate for any thoughts or suggestions!

Indicate

What is serialInBuffer? Your async method should not use shared state, especially if that is not threadsafe. Also, the tail recursion is a bad idea... – Jolo

@Jolo The comments at the top of the code block explain that serialInBuffer is an instance of a circular byte buffer class. I will edit to note that it is threadsafe, but this is not the problem. I audit the received data by raising an event before it goes into serialInBuffer. I'll annotate the code to indicate that. – Indicate

@Jolo why is the tail recursion a bad idea in this case? I originally had this in a While (true) loop, but changed it so it would look more like what Ben Voigt has published, and to make the logic visible without having to jump between code blocks. – Indicate

Because this will accumulate memory for the task objects until it is done. https://mcmap.net/q/901491/-tail-recursion-with-tasks – Jolo

@Jolo OK, thanks for calling that out. I've changed that code, but as expected the data loss problem remains. I've added additional information and examples of the output showing the problem. – Indicate

No problem, I did not expect this to solve the issue but in long-running code the tail would behave like a memory leak so I thought I'd mention it. Your code is pretty simple and I don't see much which could go wrong, so this raises the question whether this might actually be a driver issue of your USB device? I've heard previously from people using USB-to-Serial dongles that these were often unreliable, so if you have the possibility to try with another device this might give you some additional information whether the code seems to work or not. – Jolo

Can you consider using non-MS libraries, such as github.com/jcurl/serialportstream – Jolo

@Jolo Yes, I can try non-MS libraries -- thx for the reference, and I'll try that next if nobody has any answers to offer for this question. WRT the driver, this is actually a straight USB-USB connection to the embedded ARM on the Teensy board -- no actual UART or USB-Serial conversion cable is used, only a virtual serial port on each end. And of course I'm having no problem with synchronous reads -- only the async ones... – Indicate

Answer 1

I finally came up with an answer after stepping through the decompiled source code for the .Net SerialPort class (with resharper installed just Rclick on SerialPort->Navigate->Decompiled Sources).

Answer #1: The bytes out of order problem was due to an error earlier in my program. I had canceled and restarted the readAsync loop, but I was using the wrong cancellation token so there were two copies of the loop both awaiting readAsync from the serial port. Both were issuing interrupts to return the received data, but of course it was a race condition as to which one got there first.

Answer #2: Note the way I am using the synchronous read method: I don't use the Received event (which doesn't work correctly) or check the number of bytes to read (which is unreliable) or anything like that. I merely set a timeout of zero, attempt to read with a large buffer, and check how many bytes I got back.

When called in this way, the synchronous SerialPort.Read first attempts to fulfill a read request from an internal cache[1024] of data bytes received. If it still doesn't have enough data to meet the request, it then issues a ReadAsync request against the underlying BaseStream using the exact same buffer, (adjusted)offset, and (adjusted)count.

Bottom line: When used the way I am using it, the synchronous SerialPort.Read method behaves exactly like SerialPort.ReadAsync. I conclude that it would probably be fine to put an async wrapper around the synchronous method, and just await it. However, I don't need to do that now that I can read from the basestream reliably.

Update: I now reliably receive more than 3Mbps from my serial port using a Task containing a loop that continuously awaits SerialPort.Basestream.ReadAsync and adds the results to a circular buffer.

Update 2: I continue to receive requests to post my code that works to receive data at high speed without scrambling data. The main thing I can say is this:

DO NOT ATTEMPT TO USE THE DataReceived EVENT

Because of the way it is written, it is impossible for that event to work correctly.

There are other ways to accomplish this, but here is excerpted code that works for me in production in a multi-threaded context. This code has been stable across "many" versions of .net. You can get rid of all the cancellation token stuff if you don't need it.

Task ReadSerialTask;
// this is started when the session starts
protected void BeginLoop_ReadSerialPort()
{
    // New token required for each connection
    // because EndLoop() cancels and disposes it each time.
    CTS_ReadSerial?.Dispose();  // should already be disposed
    CTS_ReadSerial = new CancellationTokenSource();
    ct_ReadSerial = CTS_ReadSerial.Token;

    ReadSerialTask = Task.Run(() => { ReadSerialBytesAsyncLoop(ct_ReadSerial); }, ct_ReadSerial);
}

protected void EndLoop_ReadSerialPort()
{
    try
    {
        CTS_ReadSerial?.Cancel();
        ReadSerialTask?.Wait();
    }
    catch (Exception e)
    {
        var typ = Global.ProgramSettings.DbgExceptions;
        if (e is TaskCanceledException)
        {
            dbg_EventHandler(typ, $"Task Cancelled: {((TaskCanceledException)e).Task.Id}\n");
        }
        else
        {
            dbg_EventHandler(typ, $"Task Exception: {e.GetType().Name}\n");
        }
    }
    finally
    {
        CTS_ReadSerial?.Dispose();
    }
}


private async void ReadSerialBytes_AsyncLoop(CancellationToken ct)
{
    const int bytesToRead = 1024;
    while ((serialPort1.IsOpen) && (!ct.IsCancellationRequested))
    {
        try
        {
            var receiveBuffer = new byte[bytesToRead];
            var numBytesRead = await serialPort1.BaseStream?.ReadAsync(receiveBuffer, 0, bytesToRead, ct);
            var byteArray = new byte[numBytesRead];
            Array.Copy(receiveBuffer, byteArray, numBytesRead);

            InBuffer.Enqueue(byteArray, 0, numBytesRead); // add the new data to a "thread-safe" buffer
        }
        catch (Exception e)
        {
            // Any exception means the connection is gone or the port is gone, so the session must be stopped.
            // Note that an IOException is always thrown by the serial port basestream when exit is requested.
            // In my context, there is no value in passing these exceptions along.
            if (IsHandleCreated)
                BeginInvoke((MethodInvoker)delegate // needed because the serial port is a control on the ui thread
                {
                    if (ConsoleMode != Mode.Stopped)
                        StopSession();
                });
            else
            {
                if (serialPort1?.BaseStream != null)
                {
                    serialPort1?.Dispose();
                }
            }
        }
    }
}

Indicate

Wouldn't that bottom line conclude in an async being wrapped around a sync being wrapped around an async? Wouldn't it be faster/more reliable to just use the SerialPort.ReadASync? – Tempe

@Tempe if you mean SerialPort.Basestream.ReadAsync, then yes! The wrapper question was just because Basestream.ReadAsync was causing me to lose bytes, but that turned out to be the secret extra thread competing for the same basestream. I now reliably receive about 3Mbps using BaseStream.ReadAsync on a serial port. – Indicate

That indeed is what I meant, nice that you have been able to fix it – Tempe

@Indicate You have several users who are interested in your full final code. Would you consider sharing it? – Collectivity

@FrankerZ I will be happy to share a working version of the (finally!) stable & high-performing code that I now use. Since this question was mainly about scrambled bytes, which was a separate issue, I'll post as a new Q & A and will tag you when it's done. – Indicate

@Indicate If you could link to that final code, I'd appreciate it. I've been struggling for years trying to solve this and was never able to figure it out. I may now finally be able to understand how this is done. – Dehypnotize

@Indicate Can you share the final code as I'm facing a similar type of issue when the DataReceived event is called even after overriding the default ReceivedBytesThreshold property to receive fix no. of bytes without blocking the UI using ReadAsync. – Tripe

@Indicate I've been testing this code and I think the ReadSerialTask?.Wait(); part isn't working as expected. The task is completed from the moment you start it. I fixed it by changing async void in ReadSerialBytes_AsyncLoop method descriptor to async task and calling the task as Task.Run(async () => { await ReadSerialBytes_AsyncLoop(CT_ReadSerial); }, CT_ReadSerial); . Cheers – Madras

Answer 2

I know it's quite some time since question was asked/solved, but noticed it while searching. I have had same kind of "problems" earlier. Nowadays I use a Pipereader over the BaseStream of the serial port to handle reading. This allows me to only clear the incoming buffers when I have a complete message (and receive several messages at the same time). And it seems to perform very well.

Code is something like this:

        var reader = PipeReader.Create(serial.BaseStream);
        while (!token.IsCancellationRequested)
        {
            ReadResult result = await reader.ReadAsync(token);

            // find and handle packets
            // Normally wrapped in a handle-method and a while to allow processing of several packets at once 
            // while(HandleIncoming(result))
            // {
                    result.Buffer.Slice(10); // Moves Buffer.Start to position 10, which we use later to advance the reader
            // }

            // Tell the PipeReader how much of the buffer we have consumed. This will "free" that part of the buffer
            reader.AdvanceTo(result.Buffer.Start, result.Buffer.End);

            // Stop reading if there's no more data coming
            if (result.IsCompleted)
            {
                break;
            }
        }

See the documentation for pipelines here: https://learn.microsoft.com/en-us/dotnet/standard/io/pipelines

Frankel

Do you use it without Pipe and PipeWriter? I can't find any method that allows to set stream to Pipe. – Lyndsaylyndsey

Answer 3

I can confirm that the scrambled sequence still persists (or is back?) in NET 6.

I'm writing my first NET 6 desktop application in January 2022 and ran into this problem for the first time. I have been using the SerialPort class for at least 4 or 5 years and never experienced this problem. I use it in almost every app I wirte to communicate with various devices.

I'm just learning that this problem existed for a long time!. The oldest report I a saw was from 2012, ... and it is still around?, seriously?.

Until now the serialport apps that I wrote are based on NET Framework 4.7.2 and older. In this framework SerialPort was part of System.dll. In NET 6 SerialPort is a platform extension moved to System.IO.Ports.dll that has to be installed as a nugget package. Could it be possible that they ported an old, bugged version?

In my test I have the old NET Framework 4.7.2 app sending a string every 20 ms over a phisical port COM3 (no USB adapter). The NET 6 app reading the strings is in the same desktop listening on a second phisical port (COM4). Both ports are linked by a short NULL modem cable, only TX, RX, GND connected. No handshake. This is the scrambled result that, as far as I can tell, is random in nature:

<-- port open with tranmission running already -->
 fox jumps over the lazy Dog - 123456789]
[The quick brown fox jumps over the lazy Dog - 123456789]
[The quick brown[The quick brown fox jumps over the lazy Dog - 123456789]
[The quick brown fox jumps over the lazy Dog - 123456789]
[The quick brown fox jumps over the lazy Dog - 123456789]
<--- many good lines removed for brevity --->
[The quick brown fox jumps over the lazy Dog - 123456789]
[The quick brown fox jumps over he lazy Dog - 1t23456789]
[The quick brown fox jumps over the lazy Dog - 123456789]
<--- many good lines removed for brevity --->
[The quick brown fox jumps over the lazy Dog - 123456789]
[The quick brownfox jumps over  the lazy Dog - 123456789]
[The quick brown fox jumps over the lazy Dog - 123456789]

Notice that the third line has the first three words of the first line!. After that is only one byte out of place. It looks like a slopy double buffer implementation ...

ops! I forgot these bytes!, here you go!

If I open a second good old terminal (Net Framework 4.7.2) and listen to the same stream the output is perfect.

In the NET 6 project I'm using exactly the same class I wrote long time ago to encapsulate the SerialPort functionality to use it from project to the next.

Until now I subscribed to the SerialPort.DataReceived event and then in the event handler the reading goes like this inside a Task (started but not waited by the event handler):

var bytesToRead = _serialPort.BytesToRead;
byte[] Data = new Byte[bytesToRead];
int received = _serialPort.Read(Data, 0, bytesToRead);
... notify the class user.

I'll test the work arround suggested here ...

Siege

Unfortunatelly after intensive testing, it turns out that my good old terminal program implemented with SerialPort in Net Framework 4.7.2 it ALSO SCRAMBLES characters, but not as nearly as grotesque as the samples above. So, I implemented a replacement class using native Win32 system calls in C++/CLR that allows interop with other .NET languages. If someone finds this can be of any use let me know. – Siege

I don't know if that's the issue but you say: Until now I subscribed to the SerialPort.DataReceived event and then in the event handler the reading goes like this inside a Task (started but not waited by the event handler). Keep in mind that if DataReceived is raised multiple times, you will end up having multiple tasks running concurrently, in which case you lose any guarantee about the order in which the bits are processed. Make sure you add proper synchronization so you have only one task reading/processing the data at a given time – Ginni