Jul 14, 2025 · 13 min read · You’ve probably used Adam as your go to optimizer. But do you know why it works? In this article, we’ll unpack the Adam optimizer introduced in this paper. This post is for anyone who wants to deeply understand what is happening when they use Adam. A...
Join discussion
Mar 10, 2025 · 10 min read · 🚀 Implementation of ADAM In earlier posts, we implemented a momentum-based optimizer, which offered several advantages: Momentum helps avoid getting stuck in local minima, much like in physics 🏃♂️. It smooths updates by acting as a moving averag...
Join discussion
Feb 27, 2025 · 3 min read · Implementing Momentum-based Optimizer 🚀 Context Last time, we implemented a simple learning rate scheduler and identified a major flaw: over time, it becomes unreliable for optimizing the training process. So, we decided to upgrade it. I had always ...
Join discussion
May 19, 2024 · 1 min read · The Adam (Adaptive Moment estimation) algorithm optimizes gradient descent. If it notices that gradient descent is taking small steps in the same direction, it increases the value of alpha so that it can reach the minimum faster. On the other extreme...
Join discussion
Jun 6, 2023 · 10 min read · Introduction istockphoto.com Aside from understanding the regularization techniques used to address the issue of high variance or overfitting in machine learning algorithms, it is essential to explore optimization algorithms that enhance the perform...
Join discussion