Kafka SMT ValueToKey - How use multiple values as key?

Question

I'm using the Confluent JDBCSourceConnector to read from an Oracle table. I am trying to use SMT to generate a key composed of 3 concatenated fields.

transforms=createKey
transforms.createKey.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.createKey.fields=BUS_OFC_ID_CD,SO_TYPE,SO_NO

Using the above transformation, I get something like this:

{"BUS_OFC_ID_CD":"111","SO_TYPE":"I","SO_NO":"55555"}

I would like something like:

111I55555

Any idea on how I could concatenate the values only?

Sharolynsharon

Tracking: github.com/confluentinc/kafka-connect-jdbc/issues/359 – Englut

Answer 1

I could not resolve the issue above in the properties file. So, the work around was:

• Create a view (we already had to do that to get mode=timestamp to work with our Oracle DB)
• Add an extra field with the concatenated values for KEYNAME
• Extract the concatenated value to use as the key.

For example:

CREATE VIEW XX_TEST_V AS 
SELECT BUS_OFC_ID_CD, SO_TYPE, SO_NO, BUS_OFC_ID_CD||SO_TYPE||SO_NO as KEYNAME 
FROM XX_TEST;

This would then give you a JSON key message of

{"KEYNAME ":"111I55555"}

To strip off the JSON to have just the text is then done in the properties file

For example:

transforms=createKey,extractString
transforms.createKey.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.createKey.fields=KEYNAME
transforms.extractString.type=org.apache.kafka.connect.transforms.ExtractField$Key
transforms.extractString.field=KEYNAME

This should then give you the following as the key

"111I55555" 

Regards Peter

Ratter