Sandeep PawarProfabric.guru·Sep 4, 2024Querying SQL Endpoint of Fabric Lakehouse/Warehouse In A Notebook With T-SQLI am not sharing anything new. The spark data warehouse connector has been available for a couple months now. It had some bugs, but it seems to be stable now. This connector allows you to query the lakehouse or warehouse endpoint in the Fabric notebo...Discuss·874 readssql endpoint
Sandeep PawarProfabric.guru·Aug 26, 2024Quick Tip : Using NotebookUtils To Get Fabric Lakehouse PropertiesNotebookutils in Fabric now has getWithProperties() method to get the properties of a lakehouse. Earlier the get() method only returned abfs path and not the SQL endpoint. By default it returns the JSON: //sample output {'id': '<>', 'type': 'Lake...Discuss·558 readsmicrosoftfabric
Sandeep PawarProfabric.guru·Jul 30, 2024Using fsspec To Define OneLake Filesystem in FabricI mentioned on X the other day that, like other filesystem backends such as S3 and GCS, you can use fsspec to define the OneLake filesystem too. In this blog, I will explain how to define it and why it's important to know about it. https://x.com/Pawa...Sachin Nandanwar and 1 other are discussing this2 people are discussing thisDiscuss·2 likes·499 readsdaft
Sandeep PawarProfabric.guru·Jul 24, 2024How to Attach a Default Lakehouse to a Notebook in FabricI wrote a blog post a while ago on mounting a lakehouse (or generally speaking a storage location) to all nodes in a Fabric spark notebook. This allows you to use the File API file path from the mounted lakehouse. Mounting a lakehouse using mssparkut...Frithjof Vassbø and 1 other are discussing this2 people are discussing thisDiscuss·1 like·1.1K readsmicrosoftfabric
Sandeep PawarProfabric.guru·Jul 19, 2024Calculating The Size of The Dataflow Gen2 Staging LakehousesMy friend Alex Powers (PM, Fabric CAT) wrote a blog post about cleaning the staging lakehouses generated by Dataflow Gen2. Before reading this blog, go ahead and read his blog first on the mechanics of it and the whys. Note that these are system gene...Discuss·512 readsmssparkutils
StarRocks Engineeringstarrocks.hashnode.dev·May 31, 2024How WeChat’s Lakehouse Design Efficiently Handles Trillions of RecordsAbout WeChat WeChat is the world’s largest standalone mobile app, serving over 1.3 billion monthly active users as a platform for instant messaging, social media, and mobile payments. To support its unprecedented and rapidly expanding user base, WeCh...Discusslakehouse
Anuj Syalanujsyal.com·Feb 26, 2024How I Passed Databricks Data Engineer Professional ExamIntroduction You've aced the Databricks Data Engineer Associate Exam – congrats! Or maybe, you’re just curious as you heard someone becoming a Certified Data Engineering Professional. Guess it’s time to set your data engineering career in motion. But...Discuss·2.2K readsDatabricks
Stefan StarkeforCloudflight Engineering Blogengineering.cloudflight.io·Feb 14, 2024Lakehouse: Securing data accessIn this article, we'll delve into best practices for securing data access when using Microsoft Fabric Lakehouses. The goal is ensuring that your valuable data remains protected and enable people from within your organization to see subsets of the dat...Discuss·828 readsfabric
Sandeep PawarProfabric.guru·Nov 19, 2023Programmatically Creating, Managing Lakehouses in FabricAt MS Ignite, Microsoft unveiled a variety of new APIs designed for working with Fabric items, such as workspaces, Spark jobs, lakehouses, warehouses, ML items, and more. You can find detailed information about these APIs here. These APIs will be cri...Discuss·1 like·4.2K readslakehouse
Sandeep PawarProfabric.guru·Sep 23, 2023🚀Lightning Fast Copy In Fabric NotebookIf you are familiar with mssparkutils , you know that it is packed with utilities to perform common notebook tasks such as getting a list of files and folders, mount points, copying files, running notebooks etc. in Fabric. The Fabric notebook team ha...Discuss·3 likes·2.0K readsmicrosoft fabric
