AWS Athena - GENERIC_INTERNAL_ERROR: Number of partition values does not match number of filters

Question

I'm querying a table in Athena that is giving the error: GENERIC_INTERNAL_ERROR: Number of partition values does not match number of filters

I was able to query it earlier, but added another partition (AWS glue job) to try and optimize joins I will be doing in the query later. I did re-run the crawler to update the table to reflect the partitions.

I'm wondering what this error actually means.

I tried recrawling the data, but no luck. Next up I'm wiping out the underlying S3 data and going to reprocess the entire set from scratch and then re-crawl it.

SELECT *
FROM mydb.mytable
LIMIT 10

I'd like to understand what the error means and if there is a way around it other than reprocessing the entire raw data set and re-crawling it.

Sleeping

Values for partition keys are encoded in Hive metastore in partition name (a comma-separated string). You need to have these partition names updated. If re-crawling the data doesn't help, I don't think there is an other option short of recreating the table definition. Note: when re-crawling make sure your data layout actually matches your new partitioning scheme. – Valuation

Answer 1

I had the same issue today. In my case, it was because some of the underlying data in s3 had been removed, so the number of partitions in the Glue table didn't match the number of files in s3.

To fix it, I just deleted the tables in Glue, then re-ran the crawler.

Organo

Life saver comment. Thanks! – Kelcy

First thing is deleting the table will not delete the data from S3 bucket. It will only delete the athena table. I think for existing data to be loaded again, if the crawler crawls for specific moment please run load partitions from aws console after running the crawler. This will include all the data from S3 bucket. – Arlinearlington

Answer 2

I deleted tables in Glue and re created. Saved me.

Palp