Reading serial data in realtime in Python

Question

I am using a script in Python to collect data from a PIC microcontroller via serial port at 2Mbps.

The PIC works with perfect timing at 2Mbps, also the FTDI usb-serial port works great at 2Mbps (both verified with oscilloscope)

Im sending messages (size of about 15 chars) about 100-150x times a second and the number there increments (to check if i have messages being lost and so on)

On my laptop I have Xubuntu running as virtual machine, I can read the serial port via Putty and via my script (python 2.7 and pySerial)

The problem:

• When opening the serial port via Putty I see all messages (the counter in the message increments 1 by 1). Perfect!
• When opening the serial port via pySerial I see all messages but instead of receiving 100-150x per second i receive them at about 5 per second (still the message increments 1 by 1) but they are probably stored in some buffer as when I power off the PIC, i can go to the kitchen and come back and im still receiving messages.

Here is the code (I omitted most part of the code, but the loop is the same):

ser = serial.Serial('/dev/ttyUSB0', 2000000, timeout=2, xonxoff=False, rtscts=False, dsrdtr=False) #Tried with and without the last 3 parameters, and also at 1Mbps, same happens.
ser.flushInput()
ser.flushOutput()
While True:
  data_raw = ser.readline()
  print(data_raw)

Anyone knows why pySerial takes so much time to read from the serial port till the end of the line? Any help?

I want to have this in real time.

Thank you

Underground

Can you try using ser.read() instead of ser.readline()? – Spermatogonium

I used ser.read(), to read while a \n was not received, adding character by character to the string as they were received, still the same speed. – Underground

Don't wait until a \n is received - just print out each character as it arrives. Do you get a sudden flood of characters at once or does each individual character arrive on its own? – Spermatogonium

since i was doing ser.read() and then print() i was getting one character per line – Underground

@VascoBaptista With the accepted solution (to remove timeout and use inWaiting()), it could still be possible that the Python had to wait for all bytes to be received before printing out the entire message (multi-bytes). This would logically cause a delay. Was your system still real-time Can you confirm it ? – Monet

Answer 1

You can use inWaiting() to get the amount of bytes available at the input queue.

Then you can use read() to read the bytes, something like that:

While True:
    bytesToRead = ser.inWaiting()
    ser.read(bytesToRead)

Why not to use readline() at this case from Docs:

Read a line which is terminated with end-of-line (eol) character (\n by default) or until timeout.

You are waiting for the timeout at each reading since it waits for eol. the serial input Q remains the same it just a lot of time to get to the "end" of the buffer, To understand it better: you are writing to the input Q like a race car, and reading like an old car :)

Asparagus

It worked, now I will try to separate all strings and run my code for each one of them. I will give feedback soon – Underground

Yes I understand, but since my messages are something like: 213531\n 616516\n 516861\n I would think it would read quite fast... anyway my timeout was 2 seconds and the messages were arriving on my script faster than that (printed on screen after readline). Meaning that it was never reaching the timeout. Even adding \r\n did not solve anything. Setting timeout=0 did not change anything, same speed, same everything (except that it was locking when I was turning OFF the PIC) – Underground

@Vasco Baptista one more thing, print() is influenced by the terminal you work with... I'll give you a reference to this post try to read it it can help. – Asparagus

Thanks for the tip @Kobi K, I know about that, but for the purpose of checking in enough because I am not not printing the messages in my final script. Also the script is now working, i added some extra code to separate the messages i receive and put them in a queue to be processed in other thread. Thanks all for the help – Underground

So with serial data does it queue up until it is read off the buffer? For instance if you did serial.write(b'100\n') serial.write('b'200\n') serial.write(b'300\n'). Then a serial.readline() would you get >100 and if you continued to do readline it would get the next one in line? – Lathing

@Lathing Yes, readline() in ended by CR meaning read a '\n' terminated line – Asparagus

Answer 2

A very good solution to this can be found here:

Here's a class that serves as a wrapper to a pyserial object. It allows you to read lines without 100% CPU. It does not contain any timeout logic. If a timeout occurs, self.s.read(i) returns an empty string and you might want to throw an exception to indicate the timeout.

It is also supposed to be fast according to the author:

The code below gives me 790 kB/sec while replacing the code with pyserial's readline method gives me just 170kB/sec.

class ReadLine:
    def __init__(self, s):
        self.buf = bytearray()
        self.s = s

    def readline(self):
        i = self.buf.find(b"\n")
        if i >= 0:
            r = self.buf[:i+1]
            self.buf = self.buf[i+1:]
            return r
        while True:
            i = max(1, min(2048, self.s.in_waiting))
            data = self.s.read(i)
            i = data.find(b"\n")
            if i >= 0:
                r = self.buf + data[:i+1]
                self.buf[0:] = data[i+1:]
                return r
            else:
                self.buf.extend(data)

ser = serial.Serial('COM7', 9600)
rl = ReadLine(ser)

while True:

    print(rl.readline())

Bathtub

Answer 3

You need to set the timeout to "None" when you open the serial port:

ser = serial.Serial(**bco_port**, timeout=None, baudrate=115000, xonxoff=False, rtscts=False, dsrdtr=False) 

This is a blocking command, so you are waiting until you receive data that has newline (\n or \r\n) at the end: line = ser.readline()

Once you have the data, it will return ASAP.

Attwood

Answer 4

From the manual:

Possible values for the parameter timeout: … x set timeout to x seconds

and

readlines(sizehint=None, eol='\n') Read a list of lines, until timeout. sizehint is ignored and only present for API compatibility with built-in File objects.

Note that this function only returns on a timeout.

So your readlines will return at most every 2 seconds. Use read() as Tim suggested.

Agha

I had problems using readlines. So I tried Kobi K solution. It works, now i just have to parse the received data and separate the strings – Underground