Schema evolution can be (very) expensive.
In order to figure out schema, you basically have to read all of your parquet files and reconcile/merge their schemas during reading time which can be expensive depending on how many files or/and how many columns in there in the dataset.
Thus, since Spark 1.5, they switched off schema merging by default. You can always switch it back on).
Since schema merging is a relatively expensive operation, and is not a
necessity in most cases, we turned it off by default starting from
1.5.0.
Without schema evolution, you can read schema from one parquet file, and while reading rest of files assume it stays the same.
Parquet schema evolution is implementation-dependent.
Hive for example has a knob parquet.column.index.access=false
that you could set to map schema by column names rather than by column index.
Then you could delete columns too, not just add.
As I said above, it is implementation-dependent, for example, Impala would not read such parquet tables correctly (fixed in recent Impala 2.6 release) [Reference].
Apache Spark, as of version 2.0.2, seems still only support adding columns:
[Reference]
Users can start with a simple schema, and gradually add more columns
to the schema as needed. In this way, users may end up with multiple
Parquet files with different but mutually compatible schemas. The
Parquet data source is now able to automatically detect this case and
merge schemas of all these files.
PS: What I have seen some folks do to have more agility on schema changes, is that they create a view on top of actual parquet tables that map two (or more ) different but compatible schemas to one common schema.
Let's say you have added one new field (registration_date) and dropped another column (last_login_date) in your new release, then this would look like:
CREATE VIEW datamart.unified_fact_vw
AS
SELECT f1..., NULL as registration_date
FROM datamart.unified_fact_schema1 f1
UNION ALL
SELECT f2..., NULL as last_login_date
FROM datamart.unified_fact_schema2 f2
;
You got the idea. Nice thing it would work the same across all sql on Hadoop dialects (like I mentioned above Hive, Impala and Spark), and still have all the benefits of Parquet tables (columnar storage, predicate push-down etc).
P.P.S: adding some information regarding common_metadata summary files that Spark can create to make this answer more complete.
Have a look at SPARK-15719
Parquet summary files are not particular useful nowadays since
- when schema merging is disabled, we assume
schema of all Parquet part-files are identical,
thus we can read the footer from any part-files.
- when schema merging is enabled, we need to read footers
of all files anyway to do the merge.
On the other hand, writing summary files can be expensive,
because footers of all part-files must be read and merged.
This is particularly costly when appending a small dataset
to a large existing Parquet dataset.
So some points are against enabling common_metadata :
When a directory consists of Parquet files with a mixture of different schemas, _common_metadata allows readers to figure out a sensible schema for the whole directory without reading the schema of each individual file. Since Hive and Impala can access an SQL schema for said files from the Hive metastore, they can immediately start processing the individual files and match each of them against the SQL schema upon reading instead of exploring their common schema beforehand. This makes the common metadata feature unnecessary for Hive and Impala.
Even though Spark processes Parquet files without an SQL schema (unless using SparkSQL) and therefore in theory could benefit from _common_metadata, this feature was still deemed not to be useful and consequently got disabled by default in SPARK-15719.
Even if this feature were useful for querying, it is still a burden during writing. The metadata has to be maintained, which is not only slow, but also prone to racing conditions and other concurrency issues, suffers from the lack of atomicity guarantees, and easily leads to data correctness issues due to stale or inconsistent metadata.
The feature is undocumented and seems to be considered as deprecated (only "seems to be" because it never seems to have been supported officially at all in the first place, and a non-supported feature can not be deprecated either).
From one of Cloudera engineers: "I don't know whether the behavior has changed on the read side to avoid looking at each footer if the common_metadata file is present. But regardless, writing that file in the first place is a HUGE bottleneck, and has caused a lot of problems for our customers. I'd really strongly recommend they don't bother with trying to generate that metadata file."
"_common_metadata" and "_metadata" files are Spark specific and are not written by Impala and Hive for example, and perhaps other engines.
Summary metadata files in Spark may still have its use cases though - when there are no concurrency and other issues described above - for example, some streaming use cases - I guess that's why this feature wasn't completely removed from Spark.
avsc,avdletc.) ? – Gstring