Arpit Tyagidataminds.hashnode.dev·Nov 12, 2024Copy data from the Data Lake to the SQL Database, deleting any existing data each time before loading the new data:Step 1: Check the existing data in the employee table: Step 2: Change the data manually in the csv file in data lake: we need to check whether this change will be visible in the table or not (definitely at the end). Step 3: Settings in “Sink” opti...Discuss·7 likesAzure Data FactoryAzure Data Factory
BuzzGKbuzzgk.hashnode.dev·Nov 7, 2024Data Pipeline Design 101In recent years, data pipeline design has undergone a significant transformation. The traditional approach of moving data from OLTP to OLAP databases has given way to more complex and diverse pipelines. Today's data pipelines integrate components fro...Discussdata pipeline
BuzzGKbuzzgk.hashnode.dev·Nov 4, 2024Enhancing Data Integrity with TraceabilityData pipelines involve multiple sources and technologies, making it challenging to uphold data integrity and compliance. This is where data traceability proves invaluable. By tracking data movement and logging access or modifications, traceability en...Discussdata pipeline
Sujit Nirmalblackshadow.hashnode.dev·Oct 24, 2024Introduction to Chatbots and GenAIChatbots are AI-driven programs designed to simulate conversation with human users. They can be used for customer service, personal assistance, and more. Generative AI, on the other hand, involves using AI models to generate new content, such as text...Discusschatbot
Arpit Tyagidataminds.hashnode.dev·Oct 23, 2024How to Copy SQL Database Tables Post-Join in Azure Data Lake via ADF!!Step 1: Create an instance of Azure Data Factory: Step 2: Set up Linked Services: Step 3: Create a dataset for both (source-SQL Database and destination-Azure Data Lake): Step 4: Build the Data Pipeline with the help of datasets: Step 5: Test ...Discuss·10 likesAzure Data FactoryAzure
KAPUPA HAAMBAYIdatasmithery.hashnode.dev·Oct 22, 2024A Day In The Life of a Super Azure Data EngineerWhen I was starting out my data engineering journey, I often imagined what it would be like to work as a great data engineer, especially in a fast-paced, data-driven environment like manufacturing. Fun fact*: I actually worked for a manufacturing com...Discuss#techinmanufacturing
Gyuhang Shimplto001.hashnode.dev·Oct 14, 2024Lambda vs Kappa Architecture in Data Pipeline (Korean)Lambda Architecture 구성 요소 Batch Layer 정기적으로 대량의 Historical Data 를 처리합니다. (예: Daily 또는 Hourly) 이를 통해 높은 정확도와 데이터 Completeness (완전성) 을 보장하며, 복잡한 Data 변환을 처리합니다. Speed Layer 실시간 Data 를 처리하여 low-latency 시간의 결과를 제공합니다. Batch Layer 가 동일한 Data 를 처리할...Discusskappa architecture
BuzzGKbuzzgk.hashnode.dev·Oct 11, 2024Data Pipeline Automation: Key Strategies for Streamlined Data OperationsAutomated data pipelines are designed to streamline the entire data lifecycle, from ingestion and integration to transformation, analysis, and reporting. By leveraging advanced features like data observability and pipeline traceability, modern data p...Discussdata
Shaileshshaileshpashte.hashnode.dev·Sep 30, 2024Amazon MSK and Big Data Ingestion Pipelines: Overview and Best PracticesIntroduction As the volume of data generated by applications and systems continues to grow, organizations require scalable and efficient solutions to ingest, process, and manage their data. Amazon MSK (Managed Streaming for Apache Kafka) is a fully m...DiscussBigDataIngestion
Rahul Dasschemasensei.hashnode.dev·Sep 25, 2024Streaming data from Kafka to BigQuery using Apache BeamIn this guide, we will walk through the process of reading data from Kafka and storing it in BigQuery using Apache Beam. Apache Beam is a unified programming model for defining both batch and streaming data-parallel processing pipelines. Prerequisite...Discusskafka
