Coding Pro Masterscodingpromasters.hashnode.dev·Feb 11, 2024Data Integration with Azure Data FactoryIn the rapidly evolving digital landscape, the ability to efficiently manage and integrate data from various sources has become crucial for businesses seeking to harness the power of their data. Azure Data Factory (ADF) emerges as a powerful cloud-ba...DiscussADF Traning
KUNAL DASkunaldas.hashnode.dev·Jan 21, 2024Deploying Azure Data Factory, Azure Data bricks, Azure Data Lake storage & MySql DB using TerraformReach at : https://heylink.me/kunaldas Here I am going to share some terraform code to deploy ADF, ADLS, ADB, and several other necessary resources. Table of Contents Deploying Azure Data Factory, Azure Data bricks, Azure Data Lake storage & MySql D...DiscussMySql DB
Data Senseisihan.hashnode.dev·Jan 21, 2024Azure Project ETL Workshop: File Structure Validation, Slow Changing Dimension and Data LoadingChallenges: Building Dynamic Data Pipeline: Processing multiple files with different column names; Need a dynamic pipeline that can handle varying file structures. Maintaining file structure integrity: Need to validate file structures against a dat...Discuss·38 readsSlow Changing Dimension
Om Panigrahiompngrh.hashnode.dev·Jan 17, 2024Azure Data NexusAzure Data Nexus Overview Integration of Varied Data Sources Explanation: Azure Data Nexus offers the capability to integrate data from diverse sources such as Azure SQL Database and Blob Storage. This integration is essential for organizations tha...DiscussAzure
Jishu Dayaljishudayal.hashnode.dev·Jan 1, 2024Streamlined Data Processing: Linking Databricks, GitHub, and Azure Data FactoryThis blog dives into building a collaborative data processing pipeline using Databricks, GitHub, and Azure Data Factory (ADF). We'll explore how to set up this architecture with clear steps for three developers (A, B, and C) with varying access level...DiscussDatabricks
Sumit Mondalsumit007.hashnode.dev·Dec 13, 2023Power of Azure Data Factory: A Creative Dive into Data OrchestrationIntroduction In the ever-evolving landscape of data management, organizations are constantly seeking innovative solutions to streamline their data workflows and harness the power of their information. Azure Data Factory, a cloud-based data integratio...DiscussAzure - TheoryAzure
Harshita Chaudharyharshita.hashnode.dev·Nov 7, 2023Incremental Data LoadIncremental data load refers to the process of integrating new or updated data into an existing dataset or database without the need to reload all the data from the beginning. This method is commonly employed in combination with techniques like chang...DiscussPySpark
Karol LuszczekforBricks in the Cloudazure-data.hashnode.dev·Nov 7, 2023Single Node Databricks Job Clusters in Azure Data FactoryIn this short post, we will delve into issues you may encounter when trying to create Unity-compatible, single-node clusters in Azure Databricks using a linked service connector in Azure Data Factory. A workaround is provided as well. Scenario 💡 AD...Discuss·173 readsDatabricks
Abhinandan Shresthaabhinandanshrestha.hashnode.dev·Oct 3, 2023What is Azure Key Vault and how to use it in your Azure Pipeline?To understand Azure Key vaults, think of a scenario like a backend connection to databases. The question is “Do we directly use connection strings in our code?”. Storing connection strings and other sensitive information directly in your code is gene...Discuss·28 readsAzure Key Vault
JTProchatjpt.blog·Aug 27, 2023Part 4: Turning the Page on Data PipelinesTurning the Page on Data Pipelines In the world of data pipelines, there comes a time when we need to address the issue of paging. Paging refers to the process of retrieving data in chunks or pages, rather than all at once. This is particularly relev...Discuss·125 readsA Microsoft Fabric Data Factory exploration using Google Scholar/SERPAPIAPIs
