Synapse Spark - Deltalake configs for Schema evolution and write optimizations
Asked Answered
M

0

7

I am looking for databricks equivalent properties in Synapse spark. Please let me know if there are any or workaround for the same.

  1. Using MERGE command to Insert/update the data. However, it does not support schema merge. Is there any property to enable auto merge ? spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled ","true") in Databricks

  2. How to control number of part files or optimize writes with delta Merge command ? set spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite = true; set spark.databricks.delta.properties.defaults.autoOptimize.autoCompact = true;

Mouth answered 19/7, 2021 at 12:21 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.