Arpit Tyagidataminds.hashnode.dev·Dec 2, 2024Simplifying Data Integration // Data Transformations with ADF: Merge Sources and Export to Parquet.Step 1: Inspecting the CSV File in Data Lake and SQL Table present in Azure SQL DB Step 2: Overview of the Dataflow for the task and then we will dig deeper into each step of this snapshot. Choose both sources i.e. “SQL DB and CSV file in ADLS” Ste...10 likesAzure Data FactoryAzure
Arpit Tyagidataminds.hashnode.dev·Dec 2, 2024Azure Data Factory: "Join" 2 or more CSV Files and Convert to JSON FormatStep 1: Inspecting the CSV Files in Data Lake: Your First Step to Data Optimization Step 2: Configuring the Data Flow Sources: Pointing to the Customer.CSV Files and use Join tool after that. Step 3: Use Join on Customer id as that is the common fi...5 likesAzure Data FactoryADF
Souhayla EL MEFTAHIfuture-data-pipelines-ai.hashnode.dev·Nov 26, 2024The Future of Data Engineering: AI Driving the ELT to STL ShiftIn the ever-evolving landscape of data engineering, new methodologies are continually reshaping the way we collect, process, and analyze data. One such shift is the transition from traditional ETL (Extract, Transform, Load) processes to STL (Stream, ...data-engineering
Natalia Polomkinaskyvia.hashnode.dev·Sep 19, 2024Key Differences Between Data Integration and ETL ExplainedData has become a valuable asset in today’s business landscape, and managing it is like safeguarding a treasure—ensuring it’s well-organized, easily accessible, and secure. While there are various approaches to data management, this article focuses o...68 readsdataworkflow
Mehul Kansalmehulkansal.hashnode.dev·Sep 16, 2024Week 16: Azure Data Factory Fundamentals 🏗Hey data engineers! 👋 Azure Data Factory plays a key role in automating data workflows, allowing organizations to transfer, transform, and orchestrate large amounts of data with minimal operational overhead. In this blog, we’ll explore ADF's core fe...Azure
Fritz Larcoblog.slingdata.io·Aug 28, 2024Reading Apache Iceberg Data with SlingWe're excited to announce that Sling now supports reading the Apache Iceberg format, bringing enhanced data lake management capabilities to our users. This addition opens up new possibilities for efficient and flexible data handling in large-scale en...266 readsapacheiceberg
Mayra Peñadbtdataqueen.hashnode.dev·May 24, 202410 Best Data Transformation Tools for a Smoother ETL/ELTData teams deciding on data transformation tools need to consider various aspects before deciding on how they will develop and orchestrate data pipelines. They also need to accelerate infrastructure deployment to deliver at the pace the business requ...dbt
Prakhar Srivastavaprakhar1209.hashnode.dev·May 20, 2024Etl Vs Elt🚨 ETL vs ELT which to prefer for data ingestions : 👇 🚨 ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) are both data integration processes, but they differ in their approach and use cases. Here are five key points of comparison...ETL
Kaushik IskaforPeerDB Blogblog.peerdb.io·May 7, 2024PeerDB Cloud is Now in Public Beta!🚀 Today, we're excited to announce that PeerDB Cloud is officially entering public beta. If you're a data engineer or an organization looking for a fast, simple, and cost-effective way to replicate data from Postgres to data warehouses such as Snowf...1 like·215 readsdata-movement
Sai SrirampurforPeerDB Blogblog.peerdb.io·May 6, 2024PeerDB Streams - Simple, Native Postgres Change Data CaptureWe spent the past 7 months building a solid experience to replicate data from Postgres to Data Warehouses such as Snowflake, BigQuery, ClickHouse and Postgres. Now, we want to expand and bring a similar experience for Queues. With that spirit, we are...3 likes·3.8K readsPostgreSQL
