This is effectively the same as my previous question, but using Avro rather than JSON as the data format. I'm working with a Spark dataframe which could be loading data from one of a few different...

My question is similar to Unable to compile and create .avro file from .avsc using Maven I have tried all possible things, checked the maven project 100 times, still i am not able to run the avro-...

Whenever I am trying to read the message from kafka queue, I am getting following exception : [error] (run-main-0) java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot ...

Is it possible to write an Avro schema/IDL that will generate a Java class that either extends a base class or implements an interface? It seems like the generated Java class extends the org.apach...

I have the following csv : field1;field2;field3;field4;field5;field6;field7;field8;field9;field10;field11;field12; eu;4523;35353;01/09/1999; 741 ; 386 ; 412 ; 86 ; 1.624 ; 1.038 ; 469 ; 117 ; an...

I am writing a spark job using python. However, I need to read in a whole bunch of avro files. This is the closest solution that I have found in Spark's example folder. However, you need to submi...

I have a JSON response from an API and I would like to validate against an existing Avro schema(strict validation with names and types). The response is of type {"name":"alex","age":23,"sex":"M"...

For the dataframe below, which was generated from an avro file, I'm trying to get the column names as a list or other format so that I can use it in a select statement. node1 and node2 have the sam...

I'm trying to create hive/impala tables base on avro files in HDFS. The tool for doing the transformations is Spark. I can't use spark.read.format("avro") to load the data into a datafram...

I'm actually trying to serialize objects containing dates with Avro, and the deserialized date doesn't match the expected value (tested with avro 1.7.2 and 1.7.1). Here's the class I'm serializing ...

We're using avro for (de)serialization of messages that flow through a message broker. For the purpose of storing the avro files a schema registry (apicurio) is used. This provides two benefits - s...

I'm new to Maven and have been looking at tutorials and web for documentation on how to build a .avro from a schema file .avsc. Based on the documentation that on the apache.maven.org site. I have ...

What I have in my Schema.avdl: array<string> names = null; Now instead of null, I want to have the default value as an empty array, so i tried and failed: array<string> names = [];...

I am using apache avro maven plugin with the aim of generating java from avsc. I am running this on an m1 mac(though i didn't see anything online about that being an issue). The issue appears to be...

I am trying to use the avro-gradle-plugin on github, but have not gotten any luck getting it to work. Does anyone have any sample code on how they get it to work?

I'm looking for a way to convert a POJO to an avro object in a generic way. The implementation should be robust to any changes of the POJO-class. I have achieved it but filling the avro record expl...

For unions in avro Idl something like below what would be the default values? 1. union {null, string} var = null; 2. union {string, null} = "xyz"; 3. union {null, string} = "xyz"; 4. union {nul...

We need to serialize some data for putting into solr as well as hadoop. I am evaluating serialization tools for the same. The top two in my list are Gson and Avro. As far as I understand, Avro =...

I am currently developing a Flink 1.4 application that reads an Avro file from a Hadoop cluster. However, running it in local mode on my IDE works perfectly fine. But when I submit it to the Jobman...

Kafka schema-registry provides a nice way to serialize and deserialize the data from Kafka using common data contract. However, the data contract (.avsc file) is the glue between the producer and c...

I have a sink connector with the following configuration { "name": "sink-test-mariadb-MY_TOPIC", "config": { "connector.class":"io.confluent.connect.jdbc.JdbcSinkConnector", "tasks.max":"10",...

I have a specific issue with Avro serialization of Java Objects. I have POJO's generated from xsd schemas which I am then trying to serialize using avro to place on a kafka topic. Some of the xmlEl...

I have the following configuration for my Kafka Stream application Properties config = new Properties(); config.put(StreamsConfig.APPLICATION_ID_CONFIG,this.applicaionId); config.put(StreamsCo...

It is common to convert avro avdl files (idl files) to avsc files (schema files). I want to convert in the other direction, from avsc to avdl, because I have some avsc files created manually and wa...

I'm trying to validate a JSON file using an Avro schema and write the corresponding Avro file. First, I've defined the following Avro schema named user.avsc: {"namespace": "example.avro", "type":...