avro Questions
1
I was able to publish my java bean class as avro record to kafka. but when i try to consume i get class cast exception. Why this occurs?
producer
Schema schema = new Schema.Parser().parse(new Fil...
Godly asked 14/4, 2016 at 7:41
6
Solved
Latest Avro compiler (1.8.2) generates java sources for dates logical types with Joda-Time based implementations. How can I configure Avro compiler to produce sources that used Java 8 date-time API...
3
Solved
Does anyone knows how to create Avro schema which contains list of objects of some class?
I want my generated classes to look like below :
class Child {
String name;
}
class Parent {
list<C...
3
Is there any tool able to create an AVRO schema from a 'typical' JSON document.
For example:
{
"records":[{"name":"X1","age":2},{"name":"X2","age":4}]
}
I found http://jsonschema.net/reboot/#/ ...
2
Is it possible to use date-time fields such as "2019-08-24T14:15:22.000Z" in Avro?
The docs says that one needs to use type int/long with logical type for dates/timestamps. But in this ca...
2
I have run the below code for 522 gzip files of size 100 GB and after decompressing, it will be around 320 GB data and data in protobuf format and write the output to GCS. I have used n1 standard m...
Colbert asked 9/1, 2021 at 12:39
2
Solved
I'm using this Avro schema:
prices-state.avsc
{
"namespace": "com.company.model",
"name": "Product",
"type": "record",
"fields": [
{
"name": "product_id",
"type": "string"
},
{
"name": ...
Acquainted asked 19/4, 2018 at 16:39
6
I have an RDD of type Row i.e, RDD[Row] and avro schema object .I need to create a dataframe with this info.
I need toconvert avro schema object into StructType for creating DataFrame.
Can you ...
Xebec asked 24/11, 2016 at 14:40
1
Getting exception while reading from kafka topic: Caused by:
org.apache.kafka.common.errors.SerializationException: Error
deserializing Avro message for id 1 Caused by:
org.apache.kafka.common...
Spellman asked 16/6, 2019 at 21:18
3
This question concerns Avro version 1.8.1.
We have the following field in our AVRO schema:
"name" : "sale_price",
"type" : ["bytes", "null"],
"logicalType": "decimal",
"precision": 18,
"scale...
1
I'm working on developing an .NET Standard pub/sub package using Kafka and Confluent's .NET client. My producer has the following interface.
IEventPublisher.cs
public interface IEventPublisher&l...
Dendrology asked 13/5, 2019 at 19:46
3
Solved
I get an error when running kafka-mongodb-source-connect
I was trying to run connect-standalone with connect-avro-standalone.properties and MongoSourceConnector.properties so that Connect write dat...
Abscind asked 3/1, 2020 at 0:56
2
Solved
Is the Avro SpecificRecord (i.e. the generated java classes) compatible with schema evolution? I.e. if I have a source of Avro messages (in my case, kafka) and I want to deserialize those messages ...
2
Using Spring Boot, I am trying to set up my Kafka consumers in batch receiving mode:
@Bean
public ConcurrentKafkaListenerContainerFactory<String, GenericData.Record> kafkaListenerContainerFa...
Belia asked 7/2, 2019 at 23:45
3
I have an enum field for status in my Avro schema in which the possible statuses currently are
PENDING
APPROVED
REJECTED
I want to add one more value in this enum “RESUBMIT”. Is this change bac...
Polypody asked 6/3, 2020 at 16:3
4
I am curious of is it a way to deal with Avro Python in the same way as in Java or C++ implementations.
According to the official Avro Python documentation, I have to provide an Avro schema in run...
3
I'm evaluating kinesis as replacement for kafka. One of the things I'm missing is Schema registry equivalent solution. In particular I need:
schema upgrade - validate compatibility with the previ...
Swim asked 15/11, 2018 at 9:50
1
I am trying to learn avro and have a question in schema.
Some documents say
{
"name": "userid",
"type" : "string",
"logicalType" : "uuid&qu...
Poriferous asked 9/3, 2021 at 4:1
4
Solved
I am running into some issues setting up default values for Avro fields. I have a simple schema as given below:
data.avsc:
{
"namespace":"test",
"type":"record",
"name":"Data",
"fields":[
{ ...
3
Solved
I have found some code here https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-dotnet-avro-serialization#Scenario2 that does the reverse of what I need:
//Define the schema in JSON
const ...
Lampley asked 26/7, 2017 at 23:6
3
Scenario -
Client serializes a POJO using Avro Reflect Datum Writer and writes GenericRecord to a file.
The schema obtained through reflection is something like this (Note the ordering A, B, D, C)...
2
I am receiving Avro records from Kafka. I want to convert these records into Parquet files. I am following this blog post: http://blog.cloudera.com/blog/2014/05/how-to-convert-existing-data-into-pa...
1
I was looking for method to mark avro field deprecated in a way that generated Java code (getters, and setters for the field) are marked with @Deprecated annotation.
Puting @Deprecated into "doc" ...
3
Solved
I use the Java API for Avro from Scala and wonder if there is an easy programmatical way to add a field to an existing record schema using the Avro GenericRecord / SchemaBuilder API?
Collar asked 7/11, 2017 at 15:19
6
Solved
I am trying to convert a Json string into a generic Java Object, with an Avro Schema.
Below is my code.
String json = "{\"foo\": 30.1, \"bar\": 60.2}";
String schemaLines = "{\"type\":\"record\",...
© 2022 - 2024 — McMap. All rights reserved.