Priyansh KhodiyarforDatazipdatazip.io·Jan 7, 2025OLake Architecture, How did we do it?When building OLake, our goal was simple: Fastest DB to Data LakeHouse (Apache Iceberg to start) data pipeline. Checkout GtiHub repository for OLake - https://github.com/datazip-inc/olake Over time, many of us who’ve worked with data pipelines have d...97 readsOLakeolake
Rosette Kastime-travel-in-microsoft-fabric.hashnode.dev·Dec 21, 2024Historical Data StorageQuerying Historical Data using Time Travel 1 Time Travel Syntax Use the @ symbol followed by a timestamp to specify the desired time point in your queries. 2 Example Query Select * from SalesData where OrderDate @ '2024-01-01' to query data from Janu...Data Retantion
Sachin Nandanwarwww.azureguru.net·Dec 18, 2024Demystifying roles and access in Microsoft Fabric lake houseIn Microsoft Fabric, SQL analytics endpoint is automatically generated for a Lakehouse. With SQL analytics endpoint a user can view the Lake house data using TSQL. When it comes to lakehouse roles and access the lakehouse security model is not comple...106 readsmicrosoftfabric
Sandeep Pawarfabric.guru·Dec 7, 2024Delta Lake Tables For Optimal Direct Lake Performance In Fabric Python NotebookIn my last blog, I showed how to use Polars in a Fabric Python notebook to read, write and transform data. What I did not cover was how to write a Delta Lake table in a Python notebook that’s optimal for DirectLake performance. All engines in Fabric ...2 likes·2.3K readsmicrosoftfabric
Priyansh KhodiyarforDatazipdatazip.io·Nov 21, 2024Common Challenges Using Debezium and Kafka Connect for CDChttps://datazip.io/blog/debezium-vs-olake Change Data Capture (CDC) is essential for modern data architectures that require real-time data replication and synchronization across systems. Debezium (a Java utility based on the Qurkus framework), coup...779 readsDebezium and CDCdebezium
Peer Grønneruppeerinsights.hashnode.dev·Oct 28, 2024Automating Fabric: Kickstart your Fabric Data Platform setupSetting up and managing workspaces in Microsoft Fabric can be a time-consuming task, especially when you need multiple workspaces for various stages of the data lifecycle across different environments. This blog post demonstrates how to streamline yo...704 readsmicrosoftfabric
Sandeep Pawarfabric.guru·Sep 19, 2024Programmatically Removing & Updating Default Lakehouse Of A Fabric NotebookI have written about default lakehouse of a Fabric notebook before here and here. However, unless you used the notebook API, there was no easy/quick way of removing all/selective lakehouses or updating the default lakehouse of a notebook. But thanks ...1.9K readsnotebookutils
Sandeep Pawarfabric.guru·Sep 4, 2024Querying SQL Endpoint of Fabric Lakehouse/Warehouse In A Notebook With T-SQLI am not sharing anything new. The spark data warehouse connector has been available for a couple months now. It had some bugs, but it seems to be stable now. This connector allows you to query the lakehouse or warehouse endpoint in the Fabric notebo...2.9K readssql endpoint
Sandeep Pawarfabric.guru·Aug 26, 2024Quick Tip : Using NotebookUtils To Get Fabric Lakehouse PropertiesNotebookutils in Fabric now has getWithProperties() method to get the properties of a lakehouse. Earlier the get() method only returned abfs path and not the SQL endpoint. By default it returns the JSON: //sample output {'id': '<>', 'type': 'Lake...1.8K readsmicrosoftfabric
Sandeep Pawarfabric.guru·Jul 30, 2024Using fsspec To Define OneLake Filesystem in FabricI mentioned on X the other day that, like other filesystem backends such as S3 and GCS, you can use fsspec to define the OneLake filesystem too. In this blog, I will explain how to define it and why it's important to know about it. https://x.com/Pawa...2 likes·910 readsdaft
