371 reads
2 comments
·Apr 6, 2024
Apr 6, 2024
Its great blog. Small doubt, In disk how spark stores the data is it in serialized form (byte format) or de-serialized form (object format). I felt this one is not clearly mentioned in the blog ? Because data spilling involves both serialization and de-serialization
1
·
·1 reply
Author
·Apr 7, 2024
Hey, you're right. Thanks for letting me know that I need to bring more clarity to that part. So, memory or RAM (Deserialised) <--> Disk Spill (Serialised into temporary spill files)
I've updated the article on this, so check it out: vaishnave.page/sparks-fly#heading-serializa..
1
·