Sandeep PawarProfabric.guru·Oct 3, 2024Enabling Column Mapping For Spaces In Column Names In Delta TableOne of the annoying limitations of Direct Lake (rather of the SQL endpoint) was that you could not have spaces in table and column names in the delta table. It was supported in the delta table but the table was not query-able in the SQL endpoint whic...Discuss·618 readscolumn mapping
Fritz LarcoforSling Data Blogblog.slingdata.io·Sep 7, 2024Efficient Data Lake Management with Sling and Delta LakeUnlocking Delta Lake Insights with Sling: Efficient Read-Only Access In the ever-evolving landscape of big data, Delta Lake has emerged as a powerful open-source storage layer that brings reliability and performance to data lakes. Today, we're thrill...Discuss·112 readsdata-engineering
Mehul Kansalmehulkansal.hashnode.dev·Sep 2, 2024Week 15: Lakehouse Architecture and Delta Engine 🏠Hey data engineers! 👋 In this blog, we explore the Azure Lakehouse architecture, focusing on how Databricks Lakehouse integrates the benefits of data lakes and data warehouses. We'll dive into its core components, such as the storage layer, Delta La...DiscussData-lake
Mehul Kansalmehulkansal.hashnode.dev·Aug 26, 2024Week 14: Delta Lake on Azure Databricks 🚀Hey there! 👋 In this week's blog, we’ll delve into the specific challenges associated with traditional Data Lakes and demonstrate how Delta Lake, an open-source storage layer, effectively addresses these issues, with hands-on examples using Azure Da...DiscussAzure
Harshita Chaudharyharshita.hashnode.dev·May 30, 2024Slowly Changing Dimensions with PySpark and Delta LakeSlowly Changing Dimensions (SCDs) are a vital concept in data warehousing, particularly in managing data that changes over time. As the entities evolve over time, it’s crucial to track and manage these changes effectively. This is where Slowly Changi...Discussdata-engineering
Aaron Jevil Nazarethaarons-space.hashnode.dev·Mar 4, 2024Delta Lake(.Parquet) vs JSON Formats for storageIntroduction Fast storage and retrieval of data are vital for maintaining a competitive edge, enhancing user experience, and facilitating efficient decision-making, especially in a fast-paced digital environment where responsiveness and scalability a...Discuss·11 likes·38 readsJavaScript
Harshita Chaudharyharshita.hashnode.dev·Jan 7, 2024Delta Lake - Introduction and ArchitectureA Delta Lake is not different from a Parquet file with a robust versioning system. It utilizes transaction logs stored in JSON files to maintain a comprehensive change history for the file. This allows us to not only access previous versions but also...Discuss·35 readsBeyond the Lake: Exploring Delta Lakedeltalake
Anshul Mathuranshulmathur.hashnode.dev·Sep 30, 2023Azure Synapse Analytics cost optimizationsimage generated using Bing Image Creator Recently I undertook a task for cost optimization on Azure Synapse Analytics that was incurring a running cost of $200K per month (notice the K at the end). A glance at the resources and you could easily find ...Discuss·41 readsAzure
