---
title: "Discussion on \"Sparks Fly\" | Hashnode"
description: "Discussion on \"Sparks Fly\". Igniting Fast Data Processing with Advanced File Formats and Serialization"
url: "https://hashnode.com/posts/sparks-fly/660ee730f3e8f690989ae4b1"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Sparks Fly](https://vaishnave.page/sparks-fly?utm_source=hashnode&utm_medium=feed)

Igniting Fast Data Processing with Advanced File Formats and Serialization

Apr 4, 2024 · 18 min read · vaishnave.page  
Updated Apr 7, 2024  
Read the article

-   [#spark](https://hashnode.com/tag/spark)
-   [#apache-spark](https://hashnode.com/tag/apache-spark)
-   [#parquet](https://hashnode.com/tag/parquet)
-   [#avro](https://hashnode.com/tag/avro)
-   [#orc](https://hashnode.com/tag/orc)
-   [#sparksql](https://hashnode.com/tag/sparksql)
-   [#json](https://hashnode.com/tag/json)
-   [#csv](https://hashnode.com/tag/csv)
-   [#serialization](https://hashnode.com/tag/serialization)
-   [#deserialization](https://hashnode.com/tag/deserialization)
-   [#encoding](https://hashnode.com/tag/encoding)
-   [#python](https://hashnode.com/tag/python)
-   [#pyspark](https://hashnode.com/tag/pyspark)
-   [#file-formats](https://hashnode.com/tag/file-formats)

## Written by

[Vaishnave Subbramanian](https://hashnode.com/@justnave)

@justnave

  
[View profile](https://hashnode.com/@justnave)

Software Engineer by day, Musician by night

## Responses 1

Markdown supported · ⌘↵ to post

[Kolisetty Sasiram](https://hashnode.com/@Sasi410)

[Apr 6, 2024](https://hashnode.com/posts/sparks-fly/660ee730f3e8f690989ae4b1/comment/6610ec521644e47ea7a489be)

Its great blog. Small doubt, In disk how spark stores the data is it in serialized form (byte format) or de-serialized form (object format). I felt this one is not clearly mentioned in the blog ? Because data spilling involves both serialization and de-serialization

[Vaishnave Subbramanian](https://hashnode.com/@justnave)

[Apr 7, 2024](https://hashnode.com/posts/sparks-fly/660ee730f3e8f690989ae4b1/comment/6610ec521644e47ea7a489be#reply-66129e6b8181a7284cf2e4eb)

Hey, you're right. Thanks for letting me know that I need to bring more clarity to that part. So, memory or RAM (Deserialised) <--> Disk Spill (Serialised into temporary spill files)

I've updated the article on this, so check it out: [vaishnave.page/sparks-fly](https://vaishnave.page/sparks-fly#heading-serialization-between-memory-and-disk)
