Put Map data via Firehose to Athena

Question

I have a column in Athena with Map type. I have defined the schema in Glue as Map .

I have defined a firehose stream that refers to the glue schema and converts it to parquet format. However, I am getting the following error upon sending data

The schema is invalid. Error parsing the schema: Error: type expected at the position 0 of 'MAP <STRING, INT>' but 'MAP ' is found.

I am using boto3 firehose client to submit data.

Here is what I have tried

records=[{'Data': '{"id": "aac4da11-d166-42bd-83db-c670f5b5676d", "createdutc": "2019-06-27 10:25:33","tokens": {"black": 1, "spot": 1, "joint": 1, "leg": 1, "due": 1, "itchinghow": 1, "reduce": 1}'}]

firehoseClient.put_record_batch(
    DeliveryStreamName='conversations-analysis',
    Records=records)

Cottingham

running into same issue, were you able to figure this out? – Bodwell

@Bodwell nop Apparently I was not to solve it – Cottingham

Answer 1

Apparently, I was able to solve it, mention everything in your schema in small letter and you should be able to save map data. For e.g map instead of MAP

Cottingham

I ran into the same problem, but this did not seem to help me. – Wernsman

Sorry, scratch that, I just realized that my schema update somehow didn't get saved the first time. I went through again, making sure that all of my schema's nested column types were lowercase, and it worked. Thanks for the answer! BTW, I found this related post on the AWS Forums: forums.aws.amazon.com/thread.jspa?messageID=863428&tstart=0 – Wernsman

Answer 2

@Ibhagawati's response is correct about using lowercase type names, but I also found that one needs to make sure that there are no whitespaces around the type names. This is also indicated by the error message:

... but 'MAP ' is found.
    --------^

in my case it was:

... but ' double' is found
    -----^

An error message when the case is the problem looks like:

... but 'STRUCT' is found
    -----^^^^^^

Easygoing