Decimal Datatype Support in Avro Schema and Generated Files

Question

This question concerns Avro version 1.8.1.

We have the following field in our AVRO schema:

"name" : "sale_price", 
"type" : ["bytes", "null"], 
"logicalType": "decimal", 
"precision": 18, 
"scale": 17,

As you can see, the logicalType of the field has been defined as decimal.

But when we use the avro-maven-plugin, it does not generate the correct data type within the generated Java source files. Instead it generates, java.nio.ByteBuffer. How would you have the correct data type generated within the Java files?

This is our plugin configuration:

<plugin>
    <groupId>org.apache.avro</groupId>
    <artifactId>avro-maven-plugin</artifactId>
    <version>1.8.1</version>
    <configuration>
        <stringType>String</stringType>
        <enableDecimalLogicalType>true</enableDecimalLogicalType>
    </configuration>
</plugin>

Thickleaf

enableDecimalLogicalType was introduced in Avro 1.8.2 (not yet released). – Warnock

What's the expected correct data type? Are you expecting the logicalType as the correct data type? – Egghead

Answer 1

If you want BigDecimal you need to use version 1.8.2 and add enableDecimalLogicalType parameter with true value to your pom file:

<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<executions>
    <execution>
        <id>generate-avro-sources</id>
        <phase>generate-sources</phase>
        <goals>
            <goal>schema</goal>
        </goals>
        <configuration>
            <sourceDirectory>${basedir}/src/main/avro/</sourceDirectory>
            <outputDirectory>${basedir}/generated-sources/main/java/</outputDirectory>
            <enableDecimalLogicalType>true</enableDecimalLogicalType>
        </configuration>
    </execution>
</executions>

For more info, check this issue: https://issues.apache.org/jira/browse/AVRO-1847

Jezabel

Answer 2

I had to do the Following. (Since i could not find any documentation on how to set decimal value, it might help others)

1. Define decimal field

    {
    "namespace": "com.billionDollarCompany",
    "type": "record",
    "name": "AverageObject",
    "fields": [
      {
        "name": "average",
        "type": {
          "type": "bytes",
          "logicalType": "decimal",
          "precision": 4,
          "scale": 2
        }
      }
    ]
   }
   

2. Convert Decimal number of byteBuffer

    public static ByteBuffer ByteBuffer(Schema classSchema, String fieldName, double value)
    {
        LogicalType type = classSchema.getField(fieldName).schema().getLogicalType();
        BigDecimal bigDecimal = new BigDecimal(value, MathContext.DECIMAL32).setScale(((LogicalTypes.Decimal) type).getScale(), BigDecimal.ROUND_HALF_EVEN);
        return ByteBuffer.wrap(bigDecimal.unscaledValue().toByteArray());
    }
   

   What is Scale and precision?

   Scale is the number of digits to the right of the decimal point in a number. For example, the number 123.45 has a precision of 5 and a scale of 2.

Aesthetic

I could not find anything on the docs regarding what precision refers to, thank you. – Feldt

learn.microsoft.com/en-us/sql/t-sql/data-types/… – Vermiform

Answer 3

As the Avro docs explain,

Language implementations must ignore unknown logical types when reading, and should use the underlying Avro type.

A decimal logical type annotates Avro bytes or fixed types.

Hence in the generated Java files, the decimal logical type is represented in the underlying Avro type bytes, i.e., java.nio.ByteBuffer

Egghead