BigQuery generally does a good job of loading Avro data, but "bq load" is having a lot of trouble with timestamps and other date/time fields that use the Avro logicalType attribute. My data with ...

We’re trying to decide between providing generic vs specific record formats for consumption by our clients with an eye to providing an online schema registry clients can access when the schemas are...

We are trying to write back the values from a topic to a postgres database using the Confluent JDBC Sink Connector. connector.class=io.confluent.connect.jdbc.JdbcSinkConnector connection.password=...

I am trying to read avro files using https://github.com/databricks/spark-avro and the avro schema evolved over time. I read like this with mergeSchema option set to true hoping that it would merge ...

I would like to use the kafka-avro-console-producer with the schema registry. I have big schemas (over 10k chars) and I can't really past them as a command line argument. Besides that I'd like to u...

I want to convert json schema to avro schema using python because I'm building my microservice in Python Fastapi. json-schema { "type":"object", "properties":{ &quo...

I have a use case where I want to convert a struct field to an Avro record. The struct field originally maps to an Avro type. The input data is avro files and the struct field corresponds to a fiel...

Is there a python library that converts Avro schemas to BigQuery schemas? I noticed that the Java SDK for Apache Beam has a utility that converts from Avro to BigQuery. However, the python SDK for ...

I am using Confluent's kafka-connect-jdbc to read data from different RDBMS into kafka. Here is my test table: CREATE TABLE DFOCUSVW.T4( COL1 VARCHAR(100) NOT null, COL2 DECIMAL(6, 3) NOT null, ...

I'm a noob to Kafka and Avro. So i have been trying to get the Producer/Consumer running. So far i have been able to produce and consume simple Bytes and Strings, using the following : Configuratio...

I have a Spark application that writes output files in Avro format. Now I would like that data to be available in Hive, because an application which would utilise that data can only do so through a...

I am building an application which receives data from Kafka. When using standard avro library provided by Apache ( https://pypi.org/project/avro-python3/ ) the results are correct, however, the des...

I am currently using the code below to write parquet via Avro. This code writes it to a file system but I want to write to S3. try { StopWatch sw = StopWatch.createStarted(); Schema avroSchema ...

I'm writing my first Avro schema, which uses JSON as the schema language. I know you cannot put comments into plain JSON, but I'm wondering if the Avro tool allows comments. E.g. Perhaps it strips ...

I have this avro schema in which I have capture declared as boolean and I wanted to give default value as true. { "namespace": "abc", "type": "record", "name": "Request", "fields": [ { "name...

This question is about architecture and kafka topics migrating. Original problem: schema evolution without backward compatibility. https://docs.confluent.io/current/schema-registry/avro.html I a...

I'm trying to use the spark-avro package as described in Apache Avro Data Source Guide. When I submit the following command: val df = spark.read.format("avro").load("~/foo.avro") I get an error...

How do you extract first the schema and then the data from an avro file in Java? Identical to this question except in java. I've seen examples of how to get the schema from an avsc file but not an ...

I am evaluating using Apache AVRO for my Jersey REST services. I am using Springboot with Jersey REST. Currently I am accepting JSON as input which are converted to Java Pojos using the Jackson o...

I'm trying to send a message to my broker, using Avro schema, but "im always getting error: 2020-02-01 11:24:37.189 [nioEventLoopGroup-4-1] ERROR Application - Unhandled: POST - /api/orchestr...

I'm not being able to read spark files using the spark-avro library. Here are the steps I took: Got the jar from: http://mvnrepository.com/artifact/com.databricks/spark-avro_2.10/0.1 Invoked spar...

We have been using Kafka streams for a while but never got to write tests to cover our topologies. We decided to give it a go and use the Topology Test driver provided by the streams library. Unfor...

I need to create AVRO file but for that I need 2 things: 1) JSON 2) Avro Schema From these 2 requirements - I have JSON: {"web-app": { "servlet": [ { "servlet-name": "cofaxCDS", "servlet-...

Got a fairly complicated Avro schema (which I can not modify). Trying to mock JSON example in java: GenericRecord genericRecord = AvroUtil.jsonToGenericRecord(jsonData, avroSchema); It keeps fa...

We are trying to find a way to serialize Thrift schema into Avro format. The flow is as follows: We have a project 'A' which uses Thrift. The output result object is a thrift based object. We ...