How to serialize java.math.BigDecimal in Avro?

Question

Avro schemas are defined using JSON. Schemas are composed of primitive types (null, boolean, int, long, float, double, bytes, and string) and complex types (record, enum, array, map, union, and fixed). I want to ask which one is proper for BigDecimal.

Wag

Answer 1

Avro introduced logical types in 1.7.7 (I believe) that should help you serialize decimal.

https://avro.apache.org/docs/1.8.1/spec.html#Decimal

Fishy

This worked for me for adding data to Kafka, but I couldn't use InfluxDB connector with kafka type set to bytes, since it couldn't cast the values to InfluxDB readable format. Changing bytes to string solve the problem. – Retroact

Answer 2

To serialize a Java BigDecimal in Avro, you can define an Avro schema that uses the "decimal" logical type. The "decimal" logical type allows you to specify the precision and scale of the BigDecimal. Here's an example schema definition for a BigDecimal with 2 digits after the decimal point:

{
  "name": "amount",
  "type": {
    "type": "bytes",
    "logicalType": "decimal",
    "precision": 18,   // Adjust precision as needed
    "scale": 2        // Adjust scale as needed
  }
}

• "amount" is the field name.
• "bytes" is the base Avro type used to store the BigDecimal.
• "logicalType" is set to "decimal" to indicate that this field represents a decimal number.
• "precision" is set to 18 to allow up to 18 digits in total (adjust as needed).
• "scale" is set to 2 to specify that 2 of the digits are after the decimal point (adjust as needed).

Yokefellow

Answer 3

Here is the code :

"type": [ {
    "type": "string",
    "java-class": "java.math.BigDecimal"
 } ]

Picaresque

Please format your code properly – Off

Does this really work? what if I want to do this in python or Golang? – When

This seems to work, but instead of value it creates an object, so you have to access the actual value like object.string. Because of this I've decided for @Roberts answer. – Retroact

@When yes for serialization it does work. For deserialization, no it does not work. It fails with classcast exception because of out.writeString((CharSequence) datum); in org.apache.avro.generic.GenericDatumWriter#writeString(java.lang.Object, org.apache.avro.io.Encoder). It's one of things with does not work in avro or arent taken seriously by team (enums serialization, optionality, mime-types ...) – Idioblast