I am writing an Apache Flink streaming application that deserializes data (Avro format) read off a Kafka bus (more details on here). The data is being deserialized into a Scala case class. I am get...

I am building an Apache Flink application in Scala which reads streaming data from a Kafka bus and then performs summarizing operations on it. The data from Kafka is in Avro format and needs a spec...

I have two similar schemas where only one nested field changes (it is called onefield in schema1 and anotherfield in schema2). schema1 { "type": "record", "name": "event", "namespace": "foo", ...

Say you have this AVDL as a simplified example: @namespace("example.avro") protocol User { record Man { int age; } record Woman { int age; } record User { union { Man, Woman } user_in...

I am looking to build a Spark Streaming application using the DataFrames API on Spark 1.6. Before I get too far down the rabbit hole, I was hoping someone could help me understand how DataFrames de...

A simple example illustrates my problem. In essence, I am working on a large project that has code split across multiple repositories. In repo 1 there is an Avro schema "S1" defined in an .avdl fi...

I am trying to create Union field in Avro schema and send corresponding JSON message with it but to have one of the fields - null. https://avro.apache.org/docs/1.8.2/spec.html#Unions What is exam...

how to create a ".avsc" file from avro header ? Does the first line of content is a avsc file for that avro? Or does the avsc content should starts from : {"type":"record" upto "}avro? I tried ...

The current Apache Avro (1.8.2) documentation mentions a "duration" logical type: A duration logical type annotates Avro fixed type of size 12, which stores three little-endian unsigned integers...

I've got my Kafka Streams processing configuration for AUTO_REGISTER_SCHEMAS set to true. I noticed in this auto generated schema it creates the following 2 types { "name": "id", "type": { "ty...

I am trying to use Confluent kafka-avro-console-consumer, but how to pass parameters for Schema Registry to it?

Working on a pet project (cassandra, spark, hadoop, kafka) I need a data serialization framework. Checking out the common three frameworks - namely Thrift, Avro and Protocolbuffers - I noticed most...

I am in Spark, I have an RDD from an Avro file. I now want to do some transformations on that RDD and save it back as an Avro file: val job = new Job(new Configuration()) AvroJob.setOutputKeySchem...

I have a JSON document that I would like to convert to Avro and need a schema to be specified for that purpose. Here is the JSON document for which I would like to define the avro schema: { "uid"...

I don't have exact idea of key schema, that what it is, and why it must be used as key is auto-generated and we just pass a value(message). For value, we pass a schema to the AVRO Serialiser and t...

I have a simple JSON String jsonPayload = "{\"empid\": \"6\",\"empname\": \"Saurabh\",\"address\": \"home\"}"; jsonPayload.getBytes(); I created avro schema {"namespace": "sample.namespace"...

I need to be able to mark some fields in the AVRO schema so that they will be encrypted at serialization time. A logicalType allows to mark the fields, and together with a custom conversion should...

I have a code to convert my avro record to Row using function avroToRowConverter() directKafkaStream.foreachRDD(rdd -> { JavaRDD<Row> newRDD= rdd.map(x->{ Injection<GenericRecord...

I'm planning to use one of the hadoop file format for my hadoop related project. I understand parquet is efficient for column based query and avro for full scan or when we need all the columns data...

I use Apache avro schema with Kafka 0.0.8V. I Use same schema at producer/consumer ends. There is NO ANY Changes in the schema. But i get some exception at the consumer, when i try to consume the m...

I am testing a new schema registry which loads and retrieves different kinds of avro schemas. In the process of testing, I need to create a bunch of different types of avro schemas. As it involves ...

I would like to know what the proper avro schema would be for some json to avro conversion that is in this format: {"entryDate": "2018-01-26T12:00:40.930"} My schema: { "type" : "record", "na...

I try to submit a job on Flink 1.4 and getting the following exception. Any idea how to solve the problem? Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed. ...

HI Looking for APIs to write parquest with Pojos that I have. I was able to generate avro schema using reflection and then create parquet schema using AvroSchemaConverter. Also i am not able to fi...

I have the following json data object: { "name": "John", "favorite_number": 5, "favorite_color" : "green" } The JSON schema for this object looks like this: { "$schema": "http://json-schema...