Unload status in Redshift

Question

When you load data to your Amazon Redshift tables, you can check the load status using the table STV_LOAD_STATE.

I would like to know if there's a way to achieve the same, but with the unload operation. In other words, I'd like to know if there's a way to find out the current stage of an unload process.

Jerusalem

As far as I can tell, the "Unload" operation is atomic, which means that the S3 files will be added as soon as the "unload" ends. Can't you check for the files on your S3 Bucket every once in a while ? (Simple HTTP Get for a certain file path? ) – Pyrotechnics

In fact, I'm looking for a way to retrieve the unload progress. The table STV_LOAD_STATE has a field called "pct_complete" which can be used to retrieve the completeness of a load operation, in percentage. I'm looking for a way to get the same information, but with the unload operation. – Jerusalem

Answer 1

Unlike loading data into Redshift, Unloading actually has to run a select statement. Therefore it can't tell us a status like it does when it's loading.

e.g if the select statement has to join multiple tables and scan a lot of tables to generate the output then it might take long even though the actual unload part might not be the long part.

So I usually check the query execution steps in AWS console to have a rough idea about where the unload is.

I also check the S3 folder that I am unloading to see if the files start coming in yet. They usually come in batches so it can give you an idea as well.

Weisshorn

Answer 2

2021, and we have a solution STL_UNLOAD_LOG https://docs.aws.amazon.com/redshift/latest/dg/r_STL_UNLOAD_LOG.html

Theodoratheodore