Machine Learning Interpretability — Shapley Values with PySparkInterpreting Isolation Forest’s predictions — and not only Photo by Joshua Golde on Unsplash The problem: how to interpret Isolation Forest’s predictions More specifically, how to tell which features are contributing more to the predictions. Since Is...Mar 20, 2021 · 23 min readtryexceptfinally.hashnode.dev
A Spark Attack!It is well known — or should be — that spark is not secured by default. It is right there in the docs Security in Spark is OFF by default So you should be well aware that you’ll need to put the effort to secure your cluster. And there are many th...Jan 7, 2021 · 5 min readtryexceptfinally.hashnode.dev
A Spark AttackTime to check your spark clusters Image by Author It is well known — or should be — that spark is not secured by default. It is right there in the docs Security in Spark is OFF by default So you should be well aware that you’ll need to put the effo...Nov 28, 2020 · 3 min readtryexceptfinally.hashnode.dev
Isolation Forest and PysparkLessons learned Debugging PySpark and Isolation Forest — Image by author So, after a few runs with the PySpark ml implementation of Isolation Forest presented here, I stumbled upon a couple of things and I thought I’d write about them so that you don...Mar 24, 2020 · 5 min readtryexceptfinally.hashnode.dev
Pyspark — Parallel read from databaseHow to leverage spark to read in parallel from a database Spark Parallelization A usual way to read from a database, e.g. Postgres, using spark would be something like the following: user = 'postgres' password = 'secret' db_driver = 'org.postgresql.D...Mar 21, 2020 · 3 min readtryexceptfinally.hashnode.dev