Jul 21, 2025 · 1 min read · Over the past 4 years, I’ve had the opportunity to work as a DevOps Engineer specializing in Azure DevOps, managing end-to-end CI/CD pipelines, automating infrastructure, and improving developer- workflows. I’ve helped to design, automate, and mainta...
Join discussionApr 27, 2025 · 1 min read · Are you curious about how Azure Data Factory (ADF) efficiently handles massive data movement across diverse environments—both cloud and on-premises? This visual simplifies the inner workings of ADF Copy Activity Scaling Control & Orchestration ADF...
Join discussionFeb 8, 2025 · 8 min read · Microsoft's Data Factory in Microsoft Fabric represents the evolution of Azure Data Factory (ADF), offering enhanced cloud-scale data movement and transformation services. This advancement aims to provide a more integrated, user-friendly, and enterpr...
Join discussion
Jan 31, 2025 · 3 min read · Secure File Transfer Protocol (SFTP) relies heavily on SSH key pairs for robust authentication and secure data transmission. This post provides a comprehensive guide to generating and managing these crucial keys. Key Pair Creation: A Step-by-Step Wal...
Join discussion
Dec 2, 2024 · 3 min read · Introduction to Slowly Changing Dimensions (SCD) Type 2 Slowly Changing Dimensions (SCD) Type 2 is a data warehousing technique used to track historical changes in dimension data over time. Unlike SCD Type 1, which overwrites old data, Type 2 preserv...
AAruna commented
Dec 2, 2024 · 1 min read · Step 1: Inspecting the CSV Files in Data Lake: Your First Step to Data Optimization Step 2: Configuring the Data Flow Sources: Pointing to the Customer.CSV Files and use Join tool after that. Step 3: Use Join on Customer id as that is the common fi...
Join discussion
Dec 2, 2024 · 1 min read · Step 1: Inspecting the CSV File in Data Lake: Your First Step to Data Optimization Step 2: Configuring the Data Flow Source: Pointing to the Customer.CSV File Step 3: Filtering Even Customer IDs: Streamlining Data with ADF's Filter Data Flow Step ...
Join discussion
Nov 27, 2024 · 3 min read · Caching is a technique of storing intermediate results in memory or disk. Computing the whole data again is not needed if we are using it again in further data processing. In SPARK we do cache the DataFrame so we can use the result in next tranforma...
Join discussionNov 24, 2024 · 5 min read · When designing pipelines in Microsoft Fabric or Azure Data Factory (ADF), understanding the execution flow of activities is critical. One setting that often causes confusion is "Wait on Completion"—especially in how it differs from parallel execution...
Join discussion
Nov 12, 2024 · 1 min read · Step 1: Check the existing data in the employee table: Step 2: Change the data manually in the csv file in data lake: we need to check whether this change will be visible in the table or not (definitely at the end). Step 3: Settings in “Sink” opti...
Join discussion