Ajay Veerabommaajayveerabomma.hashnode.dev·Jun 24, 2024Comparing Python Consumers and Spark Structured Streaming: Reading from Kinesis and KafkaIn the world of real-time data processing, Amazon Kinesis and Apache Kafka are popular tools for handling large streams of data. Processing this data can be done using various frameworks and libraries, among which Python consumers and Apache Spark St...1 like·89 readsRealTimeProcessing
Janga Venkata Phanindra Reddyphanindrareddyjv.hashnode.dev·Jun 5, 2023Streaming Data Pipeline Using Confluent Cloud and Dataproc on GCPIntroduction In This Article, we'll look into How we can build a streaming data pipeline using Confluent Kafka and Spark Structured Streaming on Dataproc(GCP). Objective: Create a Confluent Kafka Topic and Stream events from Kafka to Google cloud sto...1 like·208 readsCloud
Warui Wanjiruwaruithemystery.hashnode.dev·Apr 30, 2023Setting up a Spark Cluster to Read Messages from Kafka(Beat~lytica part 2)Welcome to my second installation of the Beatlytica Series. Welcome back, fellow data enthusiasts! I'm thrilled to share with you the second installation of the Beatlytica blog series, where I dive into the exciting world of real-time data streaming ...10 likesProject Beatlyticadataengineering
