How to create a valid JSON example from Avro Schema?

Question

Got a fairly complicated Avro schema (which I can not modify).

Trying to mock JSON example in java:

GenericRecord genericRecord = AvroUtil.jsonToGenericRecord(jsonData, avroSchema);

It keeps failing:

Exception in thread "main" org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_STRING

Is there e.g. online tool that will provide example of JSON data for any given Avro schema? (so that it can match correctly)

Tried mocking JSON data for hours and still no success..

Elihu

Answer 1

You can create random data using trevni dependency and test scope. Here you have a sample code

import org.apache.avro.Schema;
import org.apache.trevni.avro.RandomData;

import java.util.Iterator;

public class JSONExample {
    public static void main(String [] args){
        Schema schema = new Schema.Parser().parse("{\n" +
                "     \"type\": \"record\",\n" +
                "     \"namespace\": \"com.acme\",\n" +
                "     \"name\": \"Test\",\n" +
                "     \"fields\": [\n" +
                "       { \"name\": \"name\", \"type\": \"string\" },\n" +
                "       { \"name\": \"age\", \"type\": \"int\" },\n" +
                "       { \"name\": \"sex\", \"type\": \"string\" },\n" +
                "       { \"name\": \"active\", \"type\": \"boolean\" }\n" +
                "     ]\n" +
                "}");

        Iterator<Object> it = new RandomData(schema, 1).iterator();
        System.out.println(it.next());
    }
}

output

{"name": "cjnyvbmetf", "age": -1757126879, "sex": "", "active": false}

maven dependencies

<dependencies>
    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
        <version>1.8.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>trevni-core</artifactId>
        <classifier>tests</classifier>
        <version>1.8.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>trevni-avro</artifactId>
        <classifier>tests</classifier>
        <version>1.8.2</version>
    </dependency>
</dependencies>

Slushy

Where were you all my life? :) I think it is great.. not perfect (one part still need to fix manually), but it is very very near perfect.. Thanks! – Elihu

your welcome! glad it worked, what do you mean it need to be fix manually? just in case i can improve the answer. thanks! – Slushy

I had one issue with enum types.. It looks like it generated nicely based on input avro schema but when tried creating genericRecord - it complained so after putting null for it - all worked out good.. – Elihu

Is the Parser the recommended way to get a schema? I would think using the Avro Maven Plugin generate a class from a schema be a better option – Methane

it should produce the same result as long as you use the same schema to compile the classes from the avro maven plugin. The only difference could be the string type if you dont specify it in the mvn plugin – Slushy

For nullable fields, I still need to make manual changes like setting null or {<type>:<value>}, otherwise I'll get errors. – Unlatch

@Slushy In the random data generated, sometimes the arrays are left empty. How do I make sure that the arrays are not empty? – Chaconne

It does not work correctly with Union types. For example, given this avro schema: {"type":"record","name":"myRecord","namespace":"com.myApps.myRecordData","fields":[{"name":"eventId","type":["null","string"],"default":null}]}. The expected output JSON should include the data type string like this: {"eventId":{"string":"ID1254632"}}. But, the actual result is this one: {"eventId":"ID1254632"} – Bade

Answer 2

You need to import below packages

compile "org.apache.avro:avro:1.8.2"
compile "org.apache.avro:avro-tools:1.8.2"
compile "org.apache.avro:trevni-core:1.8.2"
compile "org.apache.avro:trevni-avro:1.8.2"

Backfire