KARTHICK KESAVANRiot-beginners-guide.hashnode.dev·16 hours agoETL Testing: A Step-by-Step Guide for NewbiesIntroduction In this present data-centric world, organizations mainly resort to data warehousing and business intelligence tools to facilitate factual decision-making. Processes such as extraction, transformation, and loading of data are mainly respo...ETL
Akash Singhskysingh04.hashnode.dev·Feb 15, 2025How I reduced $10000 Monthly AWS Glue Bill to $400 using AirflowDuring my time as a Devops Engineer at Vance, we were running around 80 ETL pipelines on AWS Glue, but as our workloads scaled, so did our costs — hitting a staggering $10,000 per month. This wasn’t sustainable. After analyzing our pipeline, we reali...32 readsAWS
Sachin Nandanwarwww.azureguru.net·Feb 11, 2025Datalake incremental updates using Microsoft Fabric DataFlow Gen2Incremental updates means refreshing the most recent or changed data rather than reloading the entire dataset. This is required to improve performance and reduce resource consumption and also to ensure that the data remains up-to-date without data re...51 readsmicrosoftfabric
Ahamad Tawsif Chowdhuryahamad.hashnode.dev·Feb 8, 2025Power Automate vs Power BI vs Python for Handling Multiple Excel Files - My ExperienceRecently, I had to process multiple Excel files in a folder, each following one of two table layouts. My goal was to extract specific columns based on which table layout the file followed, perform calculations and aggregate the data effectively. I tr...Data Science
Chandrasekar(Chan) Rajaramcr88.hashnode.dev·Feb 4, 2025Building a Parameterized Full & Incremental Load Pipeline in Azure Data FactoryIn today’s dynamic data landscape, building a dynamic reusable ETL pipelines is essential. In this blog post, we will see how to build a parameterized Azure Data Factory (ADF) pipeline that supports both full and incremental loads using a metadata-dr...31 readsfull-load
Umesh Panditumeshpandit.hashnode.dev·Feb 2, 20256 Popular Azure ETL Tools for Data Engineers in 2025In today's world, businesses handle a lot of data. ETL (Extract, Transform, Load) is a process that helps them collect data from different sources, change it into a useful format, and store it for analysis. ETL tools make this process easier and more...Azure
Mikuzmikuz.hashnode.dev·Jan 31, 2025ETL vs ELT: Key Differences, Evolution, and Choosing the Right Data Integration ApproachIn the world of data integration, two main approaches have emerged as dominant architectural patterns: ETL vs ELT. ETL (Extract-Transform-Load) represents the traditional method where data undergoes transformation before reaching its final destinatio...ETL
Mikuzmikuz.hashnode.dev·Jan 23, 2025Choosing the Right ETL Tools: Streamlining Data Integration, Transformation, and Scalability for Modern BusinessesIn today's digital landscape, data exists in multiple formats across various platforms, from cloud services to streaming applications. Traditional data processing methods can no longer handle the complexity and speed of modern data requirements. Orga...ETL
Blessing Aliublessingaliu.hashnode.dev·Jan 13, 2025🌤️ Day 3: NBA Sports Analytics Data Lake SetupThis project showcases how to build a powerful NBA Data Lake on AWS for analyzing basketball data, leveraging key AWS services to store, process, and query the data. 🔹 Amazon S3 – for storing raw and processed data 🔹 AWS Glue – for seamless data ca...1 like30 Day DevOps Challenge | DevOpsAllStarsChallengeathena
kiran sabnedatabasedeveloper.dev·Jan 3, 2025PostgreSQL Concurrency and Locking: A Comprehensive GuideIntroduction to Locking in PostgreSQL Locking in PostgreSQL is essential for ensuring data consistency and isolation across concurrent transactions. PostgreSQL uses multi-version concurrency control (MVCC) to allow multiple transactions to access dat...30 readsPostgreSQL
