Pyserial buffer fills faster than I can read

Question

I am reading data from a microcontroller via serial, at a baudrate of 921600. I'm reading a large amount of ASCII csv data, and since it comes in so fast, the buffer get's filled and all the rest of the data gets lost before I can read it. I know I could manually edit the pyserial source code for serialwin32 to increase the buffer size, but I was wondering if there is another way around it?

I can only estimate the amount of data I will receive, but it is somewhere around 200kB of data.

Nonet

Can you slow the baudrate down? – Venery

No, unfortunately I don't have control of the uC baudrate. All I can do is send the transmission byte and receive the data. – Nonet

Answer 1

There's a "Receive Buffer" slider that's accessible from the com port's Properties Page in Device Manager. It is found by following the Advanced button on the "Port Settings" tab.

More info:

http://support.microsoft.com/kb/131016 under heading Receive Buffer

http://tldp.org/HOWTO/Serial-HOWTO-4.html under heading Interrupts

Try knocking it down a notch or two.

Squeegee

I will give this a try as well. However it sounds like it will just help with the hardware buffer, so while I am probably overrunning that buffer too, I know I am overrunning the pySerial buffer, which is the first concern I have to address. – Nonet

On my Windows 7, the setting seems to be: Device Manager | Ports | COMn | Properties | Port Settings | Advanced | Receive(Bytes). This is a drop-down list and I'm already at the max of 4096. My buffer seems to be limited to about 2K. – Overcharge

The reason why your settings don't look like the image supplied by @Squeegee is because that microsoft page applies to windows 95 – Gaol

Answer 2

Have you considered reading from the serial interface in a separate thread that is running prior to sending the command to uC to send the data?

This would remove some of the delay after the write command and starting the read. There are other SO users who have had success with this method, granted they weren't having buffer overruns.

If this isn't clear let me know and I can throw something together to show this.

EDIT

Thinking about it a bit more, if you're trying to read from the buffer and write it out to the file system even the standalone thread might not save you. To minimize the processing time you might consider reading say 100 bytes at a time serial.Read(size=100) and pushing that data into a Queue to process it all after the transfer has completed

Pseudo Code Example

def thread_main_loop(myserialobj, data_queue):
    data_queue.put_no_wait(myserialobj.Read(size=100))

def process_queue_when_done(data_queue):
    while(1):
        if len(data_queue) > 0:
            poped_data = data_queue.get_no_wait()
            # Process the data as needed
        else:
            break;

Colchicum

I will give this a try, though after doing a little math, I still don't know if this will help. currently the buffer will be overrun in 0.03 seconds. And to top that, I'm fairly certain there is no end of stream notification either. The joys of homebrew hardware. – Nonet

@Nonet That's pretty quick.... Check out my edit for additional ways to speed it up. – Colchicum

Answer 3

You do not need to manually change pyserial code.

If you run your code on Windows platform, you simply need to add a line in your code

ser.set_buffer_size(rx_size = 12800, tx_size = 12800)

Where 12800 is an arbitrary number I chose. You can make receiving(rx) and transmitting(tx) buffer as big as 2147483647a

---

See also:

https://docs.python.org/3/library/ctypes.html

https://msdn.microsoft.com/en-us/library/system.io.ports.serialport.readbuffersize(v=vs.110).aspx

You might be able to setup the serial port from the DLL // Setup serial

 mySerialPort.BaudRate = 9600;
 mySerialPort.PortName = comPort;
 mySerialPort.Parity = Parity.None;
 mySerialPort.StopBits = StopBits.One;
 mySerialPort.DataBits = 8;
 mySerialPort.Handshake = Handshake.None;
 mySerialPort.RtsEnable = true;
 mySerialPort.ReadBufferSize = 32768;

Property Value Type: System.Int32 The buffer size, in bytes. The default value is 4096; the maximum value is that of a positive int, or 2147483647

And then open and use it in Python

Oro

Answer 4

I am somewhat surprised that nobody has yet mentioned the correct solution to such problems (when available), which is effective flow control through either software (XON/XOFF) or hardware flow control between the microcontroller and its sink. The issue is well described by this web article.

It may be that the source device doesn't honour such protocols, in which case you are stuck with a series of solutions that delegate the problem upwards to where more resources are available (move it from the UART buffer to the driver and upwards towards your application code). If you are losing data, it would certainly seem sensible to try and implement a lower data rate if that's a possibility.

Isolecithal

Answer 5

For me the problem was it was overloading the buffer when receiving data from the Arduino. All I had to do was mySerialPort.flushInput() and it worked.

I don't know why mySerialPort.flush() didn't work. flush() must only flush the outgoing data?

All I know is mySerialPort.flushInput() solved my problems.

Callous