Kaushik IskaforPeerDB Blogblog.peerdb.io·a day agoPeerDB Cloud is Now in Public Beta!🚀 Today, we're excited to announce that PeerDB Cloud is officially entering public beta. If you're a data engineer or an organization looking for a fast, simple, and cost-effective way to replicate data from Postgres to data warehouses such as Snowf...Discuss·1 like·67 readsdata-movement
Priti Biyanipriti-musings.hashnode.dev·May 6, 2024The Data Lineage AdvantageIn a data world as heavy as banking platforms, there is always an origin, data transformation with a set of rules flooded across multiple services and data is stored for using it further by downstream. To be a true data product, it should be addressa...DiscussModernizing Wealth Management Systemlineage
Cloud Tunedcloudtuned.hashnode.dev·Apr 22, 2024Understanding Data Pipelines: Streamlining Data FlowUnderstanding Data Pipelines: Streamlining Data Flow In today's data-driven world, businesses are inundated with vast amounts of data generated from various sources. Managing, processing, and extracting insights from this data efficiently is crucial ...Discussdata pipeline
Kinyanjui Karanjaoverflow.hashnode.dev·Mar 14, 2024Automating Data Ingestion with AWS Lambda and CDKIntroduction Data ingestion, the process of collecting and importing data for storage and analysis, is a crucial step in many data pipelines. In this blog post, we'll explore how to automate data ingestion workflows using the AWS Cloud Development Ki...Discussaws-cdk
Isaac Otengisaacoteng.hashnode.dev·Mar 7, 2024Data ingestion using AWS ServicesData ingestion using AWS Services, Part 2 Querying AWS S3 data from AWS Athena using SQL. AWS Athena is an interactive query service that makes it easy to analyze data on Amazon using standard SQL. In this second part of the tutorial, we are going to...DiscussAWS
Isaac Otengisaacoteng.hashnode.dev·Mar 7, 2024Data Ingestion Using AWS Services, Part 1Data ingestion using AWS Services, Part 1 Data ingestion is the process of collecting, importing, and transferring raw data from various sources to a storage or processing system where it can be further analyzed, transformed, and used for various pur...DiscussAWS
Victor Ndutidatacurious.hashnode.dev·Mar 4, 2024Crafting a Basic Data Pipeline with Airflow.From setup to mastery: A Guide to Crafting Your Inaugural DAG In a previous blog post, we explored the fundamental concepts of Apache Airflow—a versatile workflow management platform that empowers users to orchestrate complex data pipelines with eas...Discuss·1 like·212 readsapache
Daniel Odanai.hashnode.dev·Feb 17, 2024Building Data Pipelines: Ingesting Data on AWSIn today's Data Engineering exploration, lets dive into creating data pipelines on AWS. Our focus will be on the ingestion to buffer (Kinesis) phase. The data used for this project is from UCI machine learning repository. Overview Key Steps Lambda ...Discuss·26 readsdataengineering
Victor Ndutidatacurious.hashnode.dev·Feb 11, 2024Streamlining Airflow Deployment: Leveraging Docker ComposeData pipelines constitute a crucial element in any data project, enabling us to monitor the flow of data from its source to the desired destination. In this context, Apache Airflow emerges as a powerful platform for managing and orchestrating data pi...Discuss·68 readsdata pipeline
Sumit Mondalsumit007.hashnode.dev·Feb 4, 2024Demystifying AWS Data Pipeline: A Step-by-Step GuideIntroduction: In the fast-paced world of cloud computing, managing and orchestrating data workflows efficiently is crucial for businesses to thrive. AWS Data Pipeline offers a seamless solution to automate the movement and transformation of data betw...DiscussAWS - HandsOn #AWSAutomation
