Comment by Ahmet Özel on "Machine Learning from Scratch: Supervised vs Unsupervised Learning (and How to Tell If a Model Actually Learned)"

Good structure, and the how to tell if a model actually learned part is the half that people skip. One thing worth adding to the evaluation section while it is still fresh in study notes: a strong test score is only meaningful if the test set could not have influenced any decision you made. Choosing a threshold, picking features or stopping training based on test performance leaks information, and the resulting number is optimistic in a way no amount of confusion-matrix analysis will reveal. That is what the separate validation split is for, and it is the discipline that survives long after the specific algorithms in a course change. The unsupervised side has a quieter version of the same trap: cluster quality metrics tell you the clusters are compact, not that they correspond to anything a person cares about.

Hey Ahmet, thanks for this, genuinely.

you're right about the validation split thing, and it actually calls out something I got wrong in part 2. the mini practice section literally has people sweep test_size and max_depth and pick whatever scores best on the test set. that's the leak you're talking about basically. once you pick anything based on the test number, that number isn't clean anymore. should've had readers tune on a validation split instead and only touch test once at the end. gonna fix that.

on the clustering side, small pushback: ARI actually does use ground truth to compute, so it's not quite the same trap as something like silhouette score which is purely internal. but yeah your point still holds in general, iris is kind of a special case because we happen to have labels to check against. most real clustering you don't get that, so you're stuck with internal metrics and "tight clusters" can absolutely not mean anything useful.

appreciate you actually writing this out instead of just dropping a generic comment. going back to rework that section.

Search Hashnode