---
title: "Discussion on \"Spark Illuminated\" | Hashnode"
description: "Discussion on \"Spark Illuminated\". Unveiling Key Concepts for Effective Data Manipulation"
url: "https://hashnode.com/posts/spark-illuminated/66046ddd2f86edc7acae90f9"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Spark Illuminated](https://vaishnave.page/spark-illuminated?utm_source=hashnode&utm_medium=feed)

Unveiling Key Concepts for Effective Data Manipulation

Mar 27, 2024 · 16 min read · vaishnave.page  
Read the article

-   [#spark](https://hashnode.com/tag/spark)
-   [#sparksql](https://hashnode.com/tag/sparksql)
-   [#apache-spark](https://hashnode.com/tag/apache-spark)
-   [#rdd](https://hashnode.com/tag/rdd)
-   [#rdd-resilient-distributed-dataset](https://hashnode.com/tag/rdd-resilient-distributed-dataset)
-   [#dataframe](https://hashnode.com/tag/dataframe)
-   [#python](https://hashnode.com/tag/python)
-   [#scala](https://hashnode.com/tag/scala)
-   [#joins](https://hashnode.com/tag/joins)
-   [#hadoop](https://hashnode.com/tag/hadoop)

## Written by

[Vaishnave Subbramanian](https://hashnode.com/@justnave)

@justnave

  
[View profile](https://hashnode.com/@justnave)

Software Engineer by day, Musician by night

## Responses 1

Markdown supported · ⌘↵ to post

[Kolisetty Sasiram](https://hashnode.com/@Sasi410)

[Apr 6, 2024](https://hashnode.com/posts/spark-illuminated/66046ddd2f86edc7acae90f9/comment/6610f3a04af55564656eec9a)

I think, sort-merge join also, shuffling happens. In sort merge join, shuffle is the 1st step, then in each executor this sorting and merge operation will be performed I think. Pls correct me If I am wrong.

[Vaishnave Subbramanian](https://hashnode.com/@justnave)

[Apr 7, 2024](https://hashnode.com/posts/spark-illuminated/66046ddd2f86edc7acae90f9/comment/6610f3a04af55564656eec9a#reply-66129a2da90a0f30c35f46ad)

Hi [Kolisetty Sasiram](https://hashnode.com/@Sasi410) The shuffling does happen in the case of sort-merge join as well. I didn't include it because only if the data is not already partitioned by the join key, Spark performs a shuffle operation to redistribute the data across the cluster based on the join key. But, since I'm listing out the process, it makes more sense to add it. Thank you for your input!

[Kolisetty Sasiram](https://hashnode.com/@Sasi410)

[Apr 7, 2024](https://hashnode.com/posts/spark-illuminated/66046ddd2f86edc7acae90f9/comment/6610f3a04af55564656eec9a#reply-6612a4e78181a7284cf2e4f8)

Thanks for the clarification 😊 Great blog and great work from your side. Pls continue it
