Jan 26, 2025 · 3 min read · The Parent-Child design pattern in Azure Data Factory (ADF) is a powerful approach to dynamically orchestrating data pipelines. It simplifies handling scenarios where multiple tasks need to process data dynamically based on input. This blog post outl...
Join discussion
Dec 2, 2024 · 1 min read · (SCD Type 1 implementation via ADF) Step 1: Setting Up Your Azure SQL Database for SCD Type 1. Create the emp_scdtype1 table in Azure SQL Database. Step 2: Populating Your Table: Adding Initial Data Entries. Step 3: Visualizing Data: Confirming Tab...
Join discussion
Dec 2, 2024 · 1 min read · Step 1: Exploring the Data Lake: Initial File Inspection Step 2: Dataflow Blueprint: A Snapshot of the Transformation Process Step 3: Connecting the Dots: Linking to Your Data Source Step 4: Filtering the Blues: Excluding Specific Data Entries St...
Join discussion
Dec 2, 2024 · 1 min read · Step 1: Inspecting the CSV File in Data Lake and SQL Table present in Azure SQL DB Step 2: Overview of the Dataflow for the task and then we will dig deeper into each step of this snapshot. Choose both sources i.e. “SQL DB and CSV file in ADLS” Ste...
Join discussion
Nov 12, 2024 · 1 min read · Step 1: Do we have the table employee in SQL? Step 2: Use the dataset which is pointing to the file in Data Lake: Step 3: Set up the Source and Sink in the copy activity: Source - It will point to the dataset which is pointing to the employee file ...
Join discussion
Oct 24, 2024 · 1 min read · “Use Case”: Whenever we have more than 100 records in the customer table, we copy the customer data to another table customer_copy within the SQL DB. However, whenever we do this copy, we first truncate the table 'customer_copy' and then copy the dat...
Join discussion
Oct 23, 2024 · 1 min read · Step 1: Create an instance of Azure Data Factory: Step 2: Set up Linked Services: Step 3: Create a dataset for both (source-SQL Database and destination-Azure Data Lake): Now, this is the most important step for dynamic nomelcature: Step 4: B...
Join discussion
Aug 27, 2024 · 4 min read · When building solutions in Microsoft Azure, choosing the right data storage service is crucial for balancing performance, cost, scalability, and ease of management. Azure offers various data storage options, with Azure Blob Storage, Data Lake Storage...
Join discussion
Jul 7, 2024 · 5 min read · Introduction In the world of big data analytics, securing access to your data storage is paramount. As organizations increasingly adopt cloud-based solutions, the need for robust, scalable, and secure data access mechanisms becomes crucial. This blog...
Join discussion
Jun 15, 2024 · 8 min read · A. Create an Azure Data Storage resource and upload the input CSV files In this task, we will create an Azure Data Storage account to store the input data files (CSVs) received from the hospitals. Name the container "hospitaldata" and upload the CSV ...
Join discussion