How to choose number of executors required for our cluster ?
Let's consider it is a "20 Node Cluster"
Each Node (30 Cores , 128GB RAM
For good throughput let's assign 5 CORES per EXECUTOR
--executor-cores = 5
Should leave 1 core for Background activity (Hadoop/Yarn daemons) Number of cores available = 30 -...
blog.naveenpn.com1 min read