Harshita Chaudharyharshita.hashnode.dev·May 30, 2024Slowly Changing Dimensions with PySpark and Delta LakeSlowly Changing Dimensions (SCDs) are a vital concept in data warehousing, particularly in managing data that changes over time. As the entities evolve over time, it’s crucial to track and manage these changes effectively. This is where Slowly Changi...Discussdata-engineering
Kiran ReddyforDatabricks - PySparkdatabricks-pyspark-blogs.hashnode.dev·Apr 3, 2024Understanding Databricks Managed and External Tables: A Comprehensive GuideIntroduction In the dynamic landscape of data analytics and processing, Databricks has emerged as a cornerstone platform, empowering organizations to extract valuable insights from vast datasets with unparalleled efficiency. Founded by the creators o...Discuss·10 likesManagedTables
Hitekhitek.hashnode.dev·Jan 9, 2024Delta table with change data capture(CDF)What is CDF: The Change Data Feed (CDF) feature allows Delta tables to track row-level changes between versions of a Delta table. When enabled on a Delta table, the runtime records “change events” for all the data written into the table. This include...Discussdelta table
