Avro deserialization from Kafka using fastavro

Question

I am building an application which receives data from Kafka. When using standard avro library provided by Apache ( https://pypi.org/project/avro-python3/ ) the results are correct, however, the deserialization process is terribly slow.

class KafkaReceiver:
    data = {}

    def __init__(self, bootstrap='192.168.1.111:9092'):
        self.client = KafkaConsumer(
            'topic',
            bootstrap_servers=bootstrap,
            client_id='app',
            api_version=(0, 10, 1)
        )
        self.schema = avro.schema.parse(open("Schema.avsc", "rb").read())
        self.reader = avro.io.DatumReader(self.schema)

    def do(self):
        for msg in self.client:
            bytes_reader = io.BytesIO(msg.value)
            decoder = BinaryDecoder(bytes_reader)

            self.data = self.reader.read(decoder)

While reading why this is so slow I've found fastavro which should be much faster. I am using this this way:

    def do(self):

        schema = fastavro.schema.load_schema('Schema.avsc')
        for msg in self.client:
            bytes_reader = io.BytesIO(msg.value)
            bytes_reader.seek(0)
            for record in reader(bytes_reader, schema):
                self.data = record

And, since everything is working when using Apache's librabry, I would expect that everything will be working the same way with fastavro. However, when running this, I am getting

  File "fastavro/_read.pyx", line 389, in fastavro._read.read_map
  File "fastavro/_read.pyx", line 290, in fastavro._read.read_utf8
  File "fastavro/_six.pyx", line 22, in fastavro._six.py3_btou
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 3: invalid start byte

I don't usually program in Python, so I don't exactly know how to approach this. Any ideas?

Longspur

I assume you're on Python 3? How does your schema look like? Can the fastavro command line utility read it? – Armand

Answer 1

The fastavro.reader expects the avro file format that includes the header. It looks like what you have is a serialized record without the header. I think you might be able to read this using the fastavro.schemaless_reader.

So instead of:

for record in reader(bytes_reader, schema):
    self.data = record

You would do:

self.data = schemaless_reader(bytes_reader, schema)

Gasometer