I have this exception in the consumer when trying to cast the record.value() into java object : ClassCastException: class org.apache.avro.generic.GenericData$Record cannot be cast to class [...].Pu...

For Maven there is an official Avro plugin to generate Java classes from Avro schemas. However, for Gradle there exists no official plugin. There is davidmc24/gradle-avro-plugin, but it is no longe...

I have been reading a lot about Apache Avro these days and I am more inclined towards using it instead of using JSON. Currently, what we are doing is, we are serializing the JSON document using Jac...

I need to use the Confluent kafka-avro-serializer Maven artifact. From the official guide I should add this repository to my Maven pom <repository> <id>confluent</id> <url&g...

Scenario I create an eventhub on Jan 1st 2018. My eventhub retention period is set to be 1 day. I enable the "capture", leaving the default capture parameters of every 5 mins or 300 mb, feature ...

I have these two avsc files, the first makes a reference to the second but when trying to compile using avrogen an error is returned and the schema is not generated. Error: Exception occurred. Und...

I am building a Kafka Listener / Consumer with Spring Boot that consumes Avro data from a topic. Here's some of the properties spring.kafka.properties.specific.avro.reader=true spring.kafka.consume...

As per the definition of "default" attribute in Avro docs: "A default value for this field, used when reading instances that lack this field (optional)." This means that if the corresponding fiel...

I am taking samples from a Bayesian statistical model, serializing them with Avro, uploading them to S3, and querying them with Athena. I need help writing a query that unnests an array in the tab...

Apache Avro provides a compact, fast, binary data format, rich data structure for serialization. However, it requires user to define a schema (in JSON) for object which need to be serialized. In ...

I get the following error when I want to send an AVRO message which contains a field that has the type long: Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing A...

According to this question on nesting Avro schemas, the right way to nest a record schema is as follows: { "name": "person", "type": "record", "fields": [ {"name": "firstname", "type": "string...

Is it possible to have an optional field in an Avro schema (i.e. the field does not appear at all in the .JSON file)? In my Avro schema, I have two fields: {"name": "author", "type": ["null", "st...

I'm trying to get my head around on extracting attributes from Avro and JSON. I'm able to extract attributes from JSON by using EvaluateJsonPath processor. I'm trying to do the same on Avro, but i'...

I recently had a requirement where I needed to generate Parquet files that could be read by Apache Spark using only Java (Using no additional software installations such as: Apache Drill, Hive, Spa...

Avro schemas are defined using JSON. Schemas are composed of primitive types (null, boolean, int, long, float, double, bytes, and string) and complex types (record, enum, array, map, union, and fix...

I'm using Kafka schema registry for producing/consuming Kafka messages, for example I have two fields they are both string type, the pseudo schema as following： {"name": "test1", "type": "string"}...

I am receiving from a remote server Kafka Avro messages in Python (using the consumer of Confluent Kafka Python library), that represent clickstream data with json dictionaries with fields like use...

I need to refer to this Student schema from inside another schema: { "type": "record", "namespace": "data.add", "name": "Student", &q...

A brief explanation of what I want to achieve: I want to do functional tests for a kafka stream topology (using TopologyTestDriver) for avro records. Issues: Can't "mock" schemaRegistry to automa...

I'm using a Kafka Source in Spark Structured Streaming to receive Confluent encoded Avro records. I intend to use Confluent Schema Registry, but the integration with spark structured streaming seem...

I used avro-tools to generate java classes from avsc files, using: java.exe -jar avro-tools-1.7.7.jar compile -string schema myfile.avsc Then I tried to serialize such objects to json by Object...

We have a glue crawler that read avro files in S3 and create a table in glue catalog accordingly. The thing is that we have a column named 'foo' that came from the avro schema and we also have some...

There are at least two different ways of creating a hive table backed with Avro data: Creating a table based on an Avro schema (in this example, stored in hdfs): CREATE TABLE users_from_avro_schem...

Some characteristics of Apache Parquet are: Self-describing Columnar format Language-independent In comparison to Apache Avro, Sequence Files, RC File etc. I want an overview of the formats. I ha...