I have windows environment and my own set of kafka and zookeeper running. To use custom objects, I started to use Avro. But I needed to get the registry started. Downloaded Confluent platform and r...

I have a no-sql db where I am storing serialized avro objects. I use sbt-avro plugin to generate java classes specifically addSbtPlugin("com.cavorite" % "sbt-avro-1-8" % "1.1.5"). Now I want to add...

Is there a way to convert a JSON string to an Avro without a schema definition in Python? Or is this something only Java can handle?

I like to use the same record type in an Avro schema multiple times. Consider this schema definition { "type": "record", "name": "OrderBook", "namespace": "my.types", "doc": "Test order updat...

Currently we are using Avro data format in production. Out of several good points using Avro, we know that it is good in schema evolution. Now we are evaluating Parquet format because of its effic...

I am fairly new to Avro and going through documentation for nested types. I have the example below working nicely but many different types within the model will have addresses. Is it possible to de...

I am seeing some conflicting information about this in different places online, so would appreciate and authoritative answer from someone, who actually knows. Suppose, I am serializing some stuff ...

I am trying to read avro messages from Kafka, using PySpark 2.4.0. The spark-avro external module can provide this solution for reading avro files: df = spark.read.format("avro").load("examples/s...

There is somethigng I'm trying to understand about how Avro-serialized messages are treated by Kafka and Schema Registry - from this post I've understood the schema ID is stored in an predictable p...

I have a spring application that is my kafka producer and I was wondering why avro is the best way to go. I read about it and all it has to offer, but why can't I just serialize my POJO that I cre...

What is the correct way to create avro schema for object with array of strings? I am trying to create avro schema to object that have array of strings according to official documenation? but I ge...

I am trying to create a Kafka Streams Application which processes Avro records, but I am getting the following error: Exception in thread "streams-application-c8031218-8de9-4d55-a5d0-81c30051a829-...

I am curious to understand the best practice for encoding one very specific type of data within Avro: UUIDs.

Now that Spark 2.4 has built-in support for Avro format, I'm considering changing the format of some of the data sets in my data lake - those that are usually queried/joined for entire rows rather ...

I'm using following tool <dependency> <groupId>tech.allegro.schema.json2avro</groupId> <artifactId>converter</artifactId> <version>0.2.7</version> <...

I have an event sourced application built on top of Kafka. Currently I have one Topic that has multiple message types in it. All serialized/deserialized with JSON. The schema registry from conflue...

I get tweets from kafka topic with Avro (serializer and deserializer). Then i create a spark consumer which extracts tweets in Dstream of RDD[GenericRecord]. Now i want to convert each rdd to a dat...

I have been trying to connect with kafka-avro-console-consumer from Confluent to our legacy Kafka cluster, which was deployed without Confluent Schema Registry. I provided schema explicitly using ...

We have vanilla apache Kafka setup in current infrastructure and we started logging some data that we want to process using Kafka Connect. Currently we use Avro for our message format, but there's ...

How to use Spring-Kafka to read AVRO message with Confluent Schema registry? Is there any sample? I can't find it in official reference document.

How to convert Avro GenericRecord to Json using while coverting timestamp fields from milliseconds to datetime? Currently using Avro 1.8.2 Timestamp tsp = new Timestamp(1530228588182l); Schema...

I can do, { "type": "record", "name": "Foo", "fields": [ {"name": "bar", "type": { "type": "record", "name": "Bar", "fields": [ ] }} ] } and that works fine, but supposing I want to spl...

I am new to AWS Glue and am having difficulty fully understanding the AWS docs, but am struggling through the following use case: We have an s3 bucket with a number of Avro files. We have decided ...

I do have a number of queries about AVRO schema. I have read that, we need to pass a schema id and the message in the Kafka event.The body of my Kafka event is like - { "componentName": "ABC"...

I have a JSON file and an avro schema file, which correctly describes it's structure. I then convert the JSON file with the Avro tools into an avro file, without getting an error, like this: java ...