We use protocol buffers
(protobuf) to define our schemas internally and have translators to translate to various other internal representations such as JSON Schema(Protobuf to JSON schema converter) so as to have protobuf schemas to be the source of truth.
We are also planning on creating SQL DDLs from the protobuf schemas and couldn't find a reliable, well supported converter for that. Some of the ones we have found such as jsonutils don't seem to be active.
SQL to protobuf generators are plenty(Example: https://github.com/jackskj/protoc-gen-map/)
We want a simple converter to translate protobuf schemas to CREATE
statements on protobuf schema creation and potentially ALTER
statements when new columns are added to protobuf schema(we don't support changing field types, or deleting fields in our protobuf schemas) in SnowSQL.
Nested messages can be stored as variant
and doesn't have to be flattened out.
Are there any tools out there or should we write a custom tool for this?
message Foo {
string uuid = 1;
MessageType1 message_type1 = 2;
MessageType2 message_type2 = 3;
google.protobuf.Timestamp updated_at = 4;
float some_num = 5;
}
gets translated to
CREATE TABLE FOO (
uuid string,
message_type1 variant,
message_type2 variant,
updated_at TIMESTAMP_LTZ(9),
some_num float
);
Update to a schema
message Foo {
string uuid = 1;
MessageType1 message_type1 = 2;
MessageType2 message_type2 = 3;
google.protobuf.Timestamp updated_at = 4;
float some_num = 5;
string new_field = 6;
}
will translate to
ALTER TABLE FOO ADD COLUMN new_field string;
or
CREATE TABLE FOO (
uuid string,
message_type1 variant,
message_type2 variant,
updated_at TIMESTAMP_LTZ(9),
some_num float,
new_field string
);