Sachin Nandanwarwww.azureguru.net·Jan 30, 2025Automate conversion of csv file to parquet and upload to a lake house tableLet's say you have one or more CSV files and you want to convert them to Parquet format and also upload them to a Lakehouse table. The available options for this in the Fabric environment are either through a notebook or a data pipeline but there are...microsoftfabric
Sachin Nandanwarwww.azureguru.net·Jan 28, 2025Expose your Microsoft Fabric Lakehouse data through REST APIAs many of you may know, Microsoft Fabric offers a GraphQL API to access data from Warehouse, Fabric Lakehouse through SQL Analytics and Fabric SQL databases. However it lacks a built-in feature to expose data as a REST API. Imagine you have an exist...30 readsmicrosoftfabric
Sachin Nandanwarwww.azureguru.net·Jan 21, 2025Analyzing Delta lake tables in Microsoft Fabric through Parquet.Net and ADLS GEN2 APIThis article is inspired by Sandeep Pawar’s earlier article. Sandeep in that article demonstrated an innovative approach to obtain essential delta lake metadata details across single/multiple lakehouses. This gathered data can play crucial role for a...51 readsmicrosoftfabric
Shreyan Dasshreyandas.hashnode.dev·Jul 19, 2024Need for speed, but where do I store all of this data?Dear Diary, The most dreaded day in the life of any data engineer came to me a while ago — our TL sat us down to tell us that we had been billed an enormous amount for our cloud storage in the last quarter, and we needed to find a way to cut down sto...40 likes·29 readsConfessions of a Data Engineerdata-engineering
Kaustubhworkernode.hashnode.dev·Apr 12, 2024Understanding Parquet FilesApache Parquet is a open source, column-based file format that's great for storing and retrieving data quickly. It has smart compression and encoding methods to handle large amounts of data easily. It's perfect for both regular and interactive tasks....big data
Vaishnave Subbramanianvaishnave.page·Apr 4, 2024Sparks FlyFile Formats In the realm of data storage and processing, file formats play a pivotal role in defining how information is organized, stored, and accessed. These formats, ranging from simple text files to complex structured formats, serve as the blue...1 like·569 readsDabbling with Apache Sparkspark
David Marquisdavidmarquis.hashnode.dev·Mar 23, 2024Fastparquet: A Guide for Python Data EngineersRecently I was on the path to hunt down a way to read and test parquet files to help one of the remote teams out. The Apache Parquet file format, known for its high compression ratio and speedy read/write operations, particularly for complex nested d...764 readsfastparquet
Aaron Jevil Nazarethaarons-space.hashnode.dev·Mar 4, 2024Delta Lake(.Parquet) vs JSON Formats for storageIntroduction Fast storage and retrieval of data are vital for maintaining a competitive edge, enhancing user experience, and facilitating efficient decision-making, especially in a fast-paced digital environment where responsiveness and scalability a...11 likes·41 readsJavaScript
Aditya Tiwaridataml.hashnode.dev·Jan 27, 2024Parquet batch ingestion with Python, Postgres and Docker.Introduction We'll use Python to ingest into a Postgres database the NYC TLC green taxi trip records for September 2019, which are available in a Parquet file at the following URL:(https://d37ci6vzurychx.cloudfront.net/trip-data/green_tripdata_2019-0...10 likes·442 readsDE Zoomcamp 2024 BlogathonPython
Maheshwar Ligadefortechwastitechwasti.com·Jan 14, 2024Parquet File Handling in Go: A Complete Guide!Parquet, a columnar storage file format, is efficient for large-scale data processing. Handling Parquet files in Go allows efficient data storage and retrieval. This guide covers the essentials of working with Parquet files in Go, including reading, ...1.2K readsgo-languagegolang