I Got Tired of Guessing Spark Configs. So I Built SparkLens.
Every data engineer has been here.
You tune your spark-submit. You set the memory, the cores, the executors. It looks right. You hit submit.
Three hours later — ExecutorLostFailure.
You open the Spark
bytesofdeepankar.hashnode.dev3 min read