Deduplication and Late-Arriving Data Handling

How we stopped rewriting the entire lake every night and started scoping dedup work to exactly the partitions that changed Domain: Data Engineering Stack: Azure Databricks, Delta Lake (MERGE + Z-Orde