Schema registry on AWS

Question

I'm evaluating kinesis as replacement for kafka. One of the things I'm missing is Schema registry equivalent solution. In particular I need:

• schema upgrade - validate compatibility with the previous version
• version avro schemas in a similar way as schema registry does

What are the options to handle 2 above? The only thing that I found was glue catalogue, but it doesn't seem to
At the end I also want to use firehose (output to redshift), but from what I understand this is not possible and will require writing custom lambda.

Swim

Kinesis only stores bytes. If you want to store Avro schemas somewhere that doesn't necessarily require Kafka, you can try Hortonwork's Registry – Rollicking

Did you ever find a solution to this? This is exactly what I want to do. – Bespectacled

Answer 1

AWS just launched Schema Registry functionality for AWS Kinesis, https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html

Knife

Answer 2

https://docs.aws.amazon.com/kinesisanalytics/latest/dev/sch-dis-ref.html Is where it gets us - still not as complete / elegant as what the kafka ecosystem by confluent bundle, say is. I am doubting how compatible schema work here too.

Might deviate from your ques - but would you consider MSK, or deploying kafka on AWS (although an overhead for managing it) ?

Illuminative

Answer 3

AWS Glue provide option for schema registry which can be used with AVRO data format. at this moment glue only support java as producer and consumer. it is quite easy to use

Lafond