Its great blog.
Small doubt, In disk how spark stores the data is it in serialized form (byte format) or de-serialized form (object format). I felt this one is not clearly mentioned in the blog ?
Because data spilling involves both serialization and de-serialization
Hey, you're right. Thanks for letting me know that I need to bring more clarity to that part.
So,
memory or RAM (Deserialised) <--> Disk Spill (Serialised into temporary spill files)
Kolisetty Sasiram
Its great blog. Small doubt, In disk how spark stores the data is it in serialized form (byte format) or de-serialized form (object format). I felt this one is not clearly mentioned in the blog ? Because data spilling involves both serialization and de-serialization