Feb 12 · 10 min read · Why Simple Windowing Approaches Fail at Scale Early streaming systems used processing-time windows—grouping events by when they arrived at the system. This approach breaks down immediately when dealing with distributed event sources. A user action on...
Join discussionMar 21, 2025 · 3 min read · Imagine a world where you can instantly understand what's happening with your data. That's the power of streaming analytics! In this blog post, we'll explore how Google Cloud's Pub/Sub, Dataflow, and Looker work together to make real-time data analys...
Join discussion
Feb 5, 2025 · 4 min read · Introduction Apache Beam is an open-source, unified programming model that enables developers to build efficient and scalable batch and stream data processing pipelines. Originally developed by Google, Apache Beam is now a part of the Apache Software...
Join discussionNov 13, 2024 · 4 min read · hi everyone! as the title says, i will try to explain google cloud dataflow in this article. why? well, let’s just say i struggled quite a bit while figuring it out! so, i decided to document everything i learned to make it easier to understand (and ...
Join discussion
Oct 14, 2024 · 8 min read · Lambda Architecture 구성 요소 Batch Layer 정기적으로 대량의 Historical Data 를 처리합니다. (예: Daily 또는 Hourly) 이를 통해 높은 정확도와 데이터 Completeness (완전성) 을 보장하며, 복잡한 Data 변환을 처리합니다. Speed Layer 실시간 Data 를 처리하여 low-latency 시간의 결과를 제공합니다. Batch Layer 가 동일한 Data 를 처리할...
Join discussionNov 4, 2023 · 3 min read · In today's data-driven world, the ability to process and analyze vast amounts of data efficiently is crucial for businesses and organizations of all sizes. Google Dataflow and Apache Beam are two powerful tools designed to help users manage and proce...
Join discussion
Sep 13, 2023 · 2 min read · What is Streaming Engine "By default, the Dataflow pipeline runner executes the steps of your streaming pipeline entirely on worker virtual machines, consuming worker CPU, memory, and Persistent Disk storage. Dataflow's Streaming Engine moves pipelin...
Join discussionSep 5, 2023 · 2 min read · What are Windows? Windows are a way to group your data by their event times. But, why do you want to group on time? So you can apply aggregations! An example might be if you have a stream of analytic data coming from mobile phones and want to count t...
Join discussionSep 4, 2023 · 2 min read · Overview If you want to split a PCollection into multiple PCollections based on a function, use the Partition transform. When You Should Use the Keys Transform When you want to split the input PCollection into multiple PCollections. You might want to...
Join discussionSep 4, 2023 · 1 min read · Overview If you need to convert a PCollection into a KV pair using dynamically generated keys derived from the input element, you should check out the WithKeys transform. When You Should Use the WithKeys Transform When you only want to convert a PCol...
Join discussion