Niels Humbeckbuilding-data-products.com·Dec 18, 2024Mastering Data Pipelines: The Secret to Fast and Reliable Data OperationsIn today’s data-driven world, data pipelines are the backbone of efficient and scalable DataOps. These pipelines are vital for managing both data and code, automating complex workflows, and minimizing manual data handling. Data pipelines can be descr...Building Data Products with DataOps Methodologydataops
Sharath Kumar Thungathurthisharaththungathurthi.hashnode.dev·Dec 18, 2024Aws GlueHere are some common AWS Glue questions and answers that can help you understand the service better: 1. What is AWS Glue? Answer: AWS Glue is a fully managed ETL (Extract, Transform, Load) service that allows you to prepare and load data for analytic...2Articles1Week
John Ryanarticles.analytics.today·Dec 10, 2024Snowflake Streams and Tasks: Best PracticesAs a Data Engineer, it’s vital to understand the techniques available for Change Data Capture (CDC). Using these methods, you can quickly identify whether incoming data has changed and whether you need to process the new/modified data. Snowflake Stre...384 readssnowflake
Arpit Tyagidataminds.hashnode.dev·Dec 2, 2024Mastering Slowly Changing Dimensions (SCD) "Type 2" with Azure Data Factory: A Step-by-Step GuideIntroduction to Slowly Changing Dimensions (SCD) Type 2 Slowly Changing Dimensions (SCD) Type 2 is a data warehousing technique used to track historical changes in dimension data over time. Unlike SCD Type 1, which overwrites old data, Type 2 preserv...Azure Data FactoryAzure
Arpit Tyagidataminds.hashnode.dev·Dec 2, 2024Mastering Slowly Changing Dimensions (SCD) Type 1 with Azure Data Factory: A Step-by-Step Guide(SCD Type 1 implementation via ADF) Step 1: Setting Up Your Azure SQL Database for SCD Type 1. Create the emp_scdtype1 table in Azure SQL Database. Step 2: Populating Your Table: Adding Initial Data Entries. Step 3: Visualizing Data: Confirming Tab...8 likesAzure Data FactoryAzure
Arpit Tyagidataminds.hashnode.dev·Dec 2, 2024Mastering DataFlow Techniques in Azure Data Factory with a Data Transformation example:Step 1: Exploring the Data Lake: Initial File Inspection Step 2: Dataflow Blueprint: A Snapshot of the Transformation Process Step 3: Connecting the Dots: Linking to Your Data Source Step 4: Filtering the Blues: Excluding Specific Data Entries St...5 likesAzure Data FactoryAzure
Fritz Larcoblog.slingdata.io·Dec 2, 2024Introducing the Sling Data PlatformThe modern data landscape is complex and challenging. Organizations need to move data between various sources and destinations, transform it along the way, and ensure everything runs smoothly in production. Setting up data pipelines traditionally inv...61 readsETL
Arpit Tyagidataminds.hashnode.dev·Dec 2, 2024Simplifying Data Integration // Data Transformations with ADF: Merge Sources and Export to Parquet.Step 1: Inspecting the CSV File in Data Lake and SQL Table present in Azure SQL DB Step 2: Overview of the Dataflow for the task and then we will dig deeper into each step of this snapshot. Choose both sources i.e. “SQL DB and CSV file in ADLS” Ste...10 likesAzure Data FactoryAzure
Arpit Tyagidataminds.hashnode.dev·Dec 2, 2024Azure Data Factory: "Join" 2 or more CSV Files and Convert to JSON FormatStep 1: Inspecting the CSV Files in Data Lake: Your First Step to Data Optimization Step 2: Configuring the Data Flow Sources: Pointing to the Customer.CSV Files and use Join tool after that. Step 3: Use Join on Customer id as that is the common fi...5 likesAzure Data FactoryADF
Arpit Tyagidataminds.hashnode.dev·Dec 2, 2024Optimizing "Data Transfer" and "Data Transformation" in ADF: Filtering Even Customer IDs from CSV to SQLStep 1: Inspecting the CSV File in Data Lake: Your First Step to Data Optimization Step 2: Configuring the Data Flow Source: Pointing to the Customer.CSV File Step 3: Filtering Even Customer IDs: Streamlining Data with ADF's Filter Data Flow Step ...7 likesAzure Data Factory#DataPipelines
