Sandeep Pawarfabric.guru·16 hours agoQuickTest: Switching Between Fabric Python And PySpark NotebooksFile this under a test I have been wanting to do for some time. If I am exploring some data in a Fabric notebook using PySpark, can I switch between Python and PySpark engines with minimal code changes in an interactive session? The goal is to use th...661 readspysaprk
Sandeep Pawarfabric.guru·Dec 14, 2024Unstructured To Structured Data : Which Engine Was Used To Create A Delta Table ?In previous blogs, I discussed using Polars, DuckDB, Daft, Spark, and others to create Delta Lake tables in Fabric. Additionally, you can use other Fabric engines like Dataflow Gen2, Pipeline, and DWH to create Delta tables. You can also create a sho...710 readsstructured output
Sandeep Pawarfabric.guru·Dec 4, 2024Working With Delta Tables In Fabric Python Notebook Using PolarsThe much-anticipated Python notebook in Fabric is finally available and the Fabric users have already developed cool libraries and blogged about the usefulness of these notebooks. Duckdb is everyone’s favorite, but I am a Python guy 😁 so here is qui...1.2K readspython notebook
Sachin Nandanwarwww.azureguru.net·Nov 28, 2024Microsoft Fabric Delta Lake optimizationIn my previous two part article, I explained how Z order works and functions. You can read about it here and here. In this article, I will focus on demonstrating the practical implementation of optimization techniques for a delta table. I will demons...71 readsmicrosoftfabric
Sachin Nandanwarwww.azureguru.net·Nov 22, 2024Z Order in Delta Lake - Part 2In my previous article on Z Order for Data Lake we delved into the underlying algorithm of Z order curve and saw how elegantly the data is organized that suffices the overall optimization of the search process and how instead of vertical efficiency o...#Z Order
Sandeep Pawarfabric.guru·Nov 8, 2024Analyze Delta Tables In FabricI have been sitting on this code for a long time. I shared the first version in one of my blogs on Direct Lake last year. I have been making updates to it since then as needed. I waited for the lakehouse schema to become available and then forgot to ...1.4K readsmicrosoftfabric
Sandeep Pawarfabric.guru·Oct 3, 2024Enabling Column Mapping For Spaces In Column Names In Delta TableOne of the annoying limitations of Direct Lake (rather of the SQL endpoint) was that you could not have spaces in table and column names in the delta table. It was supported in the delta table but the table was not query-able in the SQL endpoint whic...1.1K readscolumn mapping
Fritz Larcoblog.slingdata.io·Sep 7, 2024Efficient Data Lake Management with Sling and Delta LakeUnlocking Delta Lake Insights with Sling: Efficient Read-Only Access In the ever-evolving landscape of big data, Delta Lake has emerged as a powerful open-source storage layer that brings reliability and performance to data lakes. Today, we're thrill...234 readsdata-engineering
Mehul Kansalmehulkansal.hashnode.dev·Sep 2, 2024Week 15: Lakehouse Architecture and Delta Engine 🏠Hey data engineers! 👋 In this blog, we explore the Azure Lakehouse architecture, focusing on how Databricks Lakehouse integrates the benefits of data lakes and data warehouses. We'll dive into its core components, such as the storage layer, Delta La...Data-lake
Mehul Kansalmehulkansal.hashnode.dev·Aug 26, 2024Week 14: Delta Lake on Azure Databricks 🚀Hey there! 👋 In this week's blog, we’ll delve into the specific challenges associated with traditional Data Lakes and demonstrate how Delta Lake, an open-source storage layer, effectively addresses these issues, with hands-on examples using Azure Da...Azure
