Identification of packets in a byte stream

Question

I'm having a bit of a problem with the communication to an accelerometer sensor. The sensor puts out about 8000 readings/second continuously. The sensor is plugged in to a usb port with an adaper and shows up as com4. My problem is that I can't seem to pick out the sensor reading packets from the byte stream. The packets have the size of five bytes and have the following format:

            High nibble                     Low nibble

Byte 1      checksum, id for packet start   X high
Byte 2      X mid                           X low
Byte 3      Y high                          Y mid
Byte 4      Y low                           Z high
Byte 5      Y mid                           Y low

X, y, z is the acceleration.

In the documentation for the sensor it states that the high nibble in the first byte is the checksum (calculated Xhigh+Xlow+Yhigh+Ylow+Zhigh+Zlow) but also the identification of the packet start. I'm pretty new to programming against external devices and can't really grasp how the checksum can be used as an identifier for the start of the package (wouldn't the checksum change all the time?). Is this a common way for identifying the start of a packet? Does anyone have any idea how to solve this problem?

Any help would be greatly appreciated.

Yellowstone

What kind of packets are these? UDP? TCP? or which one? – Heraldic

@J.Davidson does it have to be one of them. Can it be byte-blocks read from external device? – Puma

5 bytes/packet, 8000 per second = 40Kb/sec, you'd need baud rate of 400000 for that, not a standard issue COM setting – Dorison

It's 8192 samples/second and the buad rate is set to 921600bps (according to the manual). I'm not sure about UDP,TCP, i'm reading via the com port class in C# .net. – Yellowstone

Have you observed the low nibble in the first byte? Can it be the id? – Puma

I have observed the nibbles (all of them) but I can't find anyone that remains constant. – Yellowstone

@Yellowstone But you don't help us much. No sample input, no code, no links to documents. What can we do blindly? – Puma

I'd like to see the exact wording for byte 1 high nibble. Can you post a link for the device? Have you tried reading five bytes and just ignoring byte 1 high nibble to see if the data looks good? 921600 is a strange value. If you divide it by 8 you get a normal com port speed. – Locust

@Locust Here is a link to the device: sequoia.it/en/fast_tracer.htm I have tried to ignore the high nibble but the results seem strange. I will try to lower the bps and give it a go again tomorrow – Yellowstone

@Puma Sorry, I'm writing this home from work. I will gather sample input etc. when I'm at work tomorrow. :) – Yellowstone

I have got it to work pretty good now. I have to check each nibble until I find one that is the checksum for the following nibbles according to the algorithm in my question. Thank you for the help all! – Yellowstone

Answer 1

... can't really grasp how the checksum can be used as an identifier for the start of the package (wouldn't the checksum change all the time?).

Yes, the checksum would change since it is derived from the data.
But even a fixed-value start-of-packet nibble would (by itself) not be sufficient to (initially) identify (or verify) data packets. Since this is binary data (rather than text), the data can take on the same value as any fixed-value start-of-packet. If you had a trivial scan for this start-nibble, that algorithm could easily misidentify a data nibble as the start-nibble.

Is this a common way for identifying the start of a packet?

No, but given the high data rate, it seems to be a scheme to minimize the packet size.

Does anyone have any idea how to solve this problem?

You probably have to initially scan every sequence of bytes five at a time (i.e. the length of a packet).
Calculate the checksum of this "packet", and compare it to the first nibble.
A match indicates that you (may) have packet alignment.
A mismatch means that you should toss the first byte, and test the next possible packet that would start with what was the second byte (i.e. shift the 4 remaining bytes and append a new 5th byte).

Once packet alignment has been achieved (or assumed), you need to continually verify the checksum of every packet in order to confirm data integrity and ensure packet data alignment. Any checksum error should force another hunt for correct packet data alignment (starting at the 2nd byte of the current "packet").

Tetrahedral

Answer 2

What you need to do is get some free SerialPortTerminal in c# import in your project and first check all the data and packets you are getting, unless you have already done that. Than just to read you will need to do something like...

   using System;
   using System.IO.Ports;
   using System.Windows.Forms;

   namespace SPE
   {
     class SerialPortProgram
     {
       // Create the serial port with basic settings
       private SerialPort port = new SerialPort("COM4",      9600, Parity.None, 8, StopBits.One);

       [STAThread]
       static void Main(string[] args)
       { 
         // Instatiate this class
         new SerialPortProgram();
       }

       private SerialPortProgram()
       {
         Console.WriteLine("Incoming Data:");

         // Attach a method to be called when there      // is data waiting in the port's buffer
         port.DataReceived += new         SerialDataReceivedEventHandler(port_DataReceived);

         // Begin communications
         port.Open();

         // Enter an application loop to keep this thread alive
         Application.Run();
       }

       private void port_DataReceived(object sender,      SerialDataReceivedEventArgs e)
       {
         // Show all the incoming data in the port's buffer
         Console.WriteLine(port.ReadExisting());
       }
     }
   }

Heraldic

Do you think OP has problem with reading the data or interpreting it. I see in the comments I have observed the nibbles (all of them) but I can't find anyone that remains constant – Puma

Yes, I seem to be able to read the data, but I can't interpret it properly. I will try your code tomorrow and compare it to the results from mine. – Yellowstone