© 2022 Hashnode
Intro Over the years, Apache Spark has become the unified engine for large-scale data analytics, data engineering, data science, and machine learning on single-node machines or clusters. Apache Spark …
What is covered In this blog, I would describe my experiments about setting up a local (standalone) Spark cluster on Mac M1 machine. Would start pyspark with an option to use the master setup in above step. Explore the UI for: Master Work…
Apache Spark : a general-purpose distributed data processing engine that imports large amounts of data, performs calculations on it in a distributed environment, and then stores it for future processi…
Designing Data-Intensive Applications [Book Highlights] [ Part II ] Distributed Data why we distribute a database across multiple machines Scalability - data volume, read/write load grows more than what a single machine can handle Fault To…
Designing Data-Intensive Applications [Book Highlights] [Part I : Chapter IV] Encoding and Evolution Avro To maintain forward and backward compatibility its important to add or remove a field that has some default value. In case of Avro, Y…
Spark Structured Streaming Dynamic Dataset Join I have came across a unique problem, I had a list of valid names in some remote configuration which changes often. Now I have a spark structured streaming job which runs always and it is an al…
Objective: With the orc file created from the Delta table in the post, now we will read that orc file and upsert the Delta table score (which has the same schema as the student table mentioned in that post). Initial setup Create the datab…
Databricks Pyspark to create a ORC file Objective: We will read the data from Delta table and create the IRC file using Spark readStream and writeStream. For this blog I used the Databricks Community edition (free). This approach can…
This is a beginner to pro guide to deal with PySpark clusters. Complete jupyter notebook can be found here: Link To GitHub Apache Spark is an in-memory distributed computing platform built on top of …
Spark Streaming Misc Spark Streaming's issue with s3/hdfs as stream source With reference to https://spoddutur.github.io/spark-notes/s3-filesystem-as-datasource-in-spark.html It is evident that when we use s3/hdfs as source for our streamin…
