Avro with Java 8 dates as logical type

Question

Latest Avro compiler (1.8.2) generates java sources for dates logical types with Joda-Time based implementations. How can I configure Avro compiler to produce sources that used Java 8 date-time API?

Wuhu

Answer 1

Currently (avro 1.8.2) this is not possible. It's hardcoded to generate Joda date/time classes.

The current master branch has switched to Java 8 and there is an open issue (with Pull Request) to add the ability to generate classes with java.time.* types.

I have no idea on any kind of release schedule for whatever is currently in master unfortunately. If you feel adventurous you can apply the patch to 1.8.2, since in theory it should all be compatible. The underlying base types when serializing / deserializing are still integers and longs.

Offish

Answer 2

You need to create your own Conversions to support java-8 date-time api, below is a conversion for java.time.LocalDate:

class Java8LocalDateConversion extends Conversion<LocalDate> {
    @Override
    public Class<LocalDate> getConvertedType() {
        return LocalDate.class;
    }

    @Override
    public String getLogicalTypeName() {
        //      v--- reuse the logical type `date`
        return "date";
    }

    @Override
    // convert LocalDate to Integer
    public Integer toInt(LocalDate value, Schema schema, LogicalType type) {
        return (int) value.toEpochDay();
    }

    @Override
    // parse LocalDate from Integer
    public LocalDate fromInt(Integer value, Schema schema, LogicalType type) {
        return LocalDate.ofEpochDay(value);
    }
}

The logical types can be reused in avro, so you can using the existing date logical type, for example:

Schema schema = LogicalTypes.date().addToSchema(Schema.create(Type.INT));

For the serializing & deserializing you should set the GenericData which will find your own conversion, for example:

//serializing
DatumWriter<T> out = new SpecificDatumWriter<>(schema, data());

// deserializing
DatumReader<T> in = new SpecificDatumReader<>(schema, schema, data());

private SpecificData data() {
    SpecificData it = new SpecificData();
    it.addLogicalTypeConversion(new Java8LocalDateConversion());
    return it;
}

If you don't want to configure the GenericData every time, you can use the global GenericData instead, for example:

//      register the conversion globally ---v
SpecificData.get().addLogicalTypeConversion(new Java8LocalDateConversion());

Malti

here is the test I wrote on github. – Malti

It doesn't solve a problem: avro compiler still generate joda's date field for such logical type. – Wuhu

Answer 3

• use Avro version >= 1.9.0 < 1.10.0
• add <dateTimeLogicalTypeImplementation>jsr310</dateTimeLogicalTypeImplementation> into configuration section.
• use Avro version >= 1.10.0
• java8/jsr310 native date/time classes used by default

Rogue

Answer 4

Avro v1.11.0 TimeConversions has built-in support for Java 8 time classes such as LocalDate, Instant, etc. The TimeConversions must be registered with the SpecificData class or its sub-classes, such as ReflectData. Here is a complete example of how to use TimeConversions:

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Account
{
    private int id;
    private LocalDate createdOn;

    public static void main(String[] args) throws IOException
    {
        // create schema from POJO Account class
        ReflectData reflectData = ReflectData.get();
        reflectData.addLogicalTypeConversion(new TimeConversions.DateConversion());
        Schema schema = reflectData.getSchema(Account.class);
        System.out.println("Schema in JSON:\n" + schema + "\n");

        // create avro writer
        DatumWriter<Account> datumWriter = new ReflectDatumWriter<>(schema);
        DataFileWriter<Account> dataFileWriter = new DataFileWriter<>(datumWriter);
        // dataFileWriter.setCodec(CodecFactory.snappyCodec()); // for compression
        dataFileWriter.create(schema, new File("accounts.avro"));
        dataFileWriter.append(new Account(123, LocalDate.of(2001, 1, 1)));
        dataFileWriter.append(new Account(234, LocalDate.of(2002, 2, 2)));
        dataFileWriter.close();

        // create avro reader
        DatumReader<Account> datumReader = new ReflectDatumReader<>(schema);
        DataFileReader<Account> dataFileReader = new DataFileReader<>(new File("accounts.avro"), datumReader);
        dataFileReader.forEach(System.out::println);
    }
}

Copaiba

Similarly there are the following TimeConversions available: LocalTimestampMicrosConversion, LocalTimestampMillisConversion, TimestampMicrosConversion, TimestampMillisConversion, TimeMicrosConversion, TimeMillisConversion * DateConversion I had to use LocalTimestampMillisConversion for LocalDateTime field in my POJO. – Polyphone

Answer 5

With avro 1.9.2 you can use e.g. date for LocalDate:

    {
      "name": "Transaction",
      "type": "record",
      "fields": [
        {
          "name": "time",
          "type": {
            "type": "int",
            "logicalType": "date"
          }
        },

See Logical Types for the other types.

Holozoic

Answer 6

AVRO 1.10 added support for LocalDateTime, see Apache Avro™ 1.10.0 Specification

Granulose

There is no such thing as "LocalDateTime" on the page you referenced. – Rogue

I meant "Local timestamp". This gets translated into "LocalDateTime" in Java. – Granulose