learning
As a passionate Machine Learning and Deep Learning enthusiast, I document my learning journey on Hashnode. My experience encompasses various projects, from exploring foundational algorithms to implementing advanced neural networks. I enjoy breaking down complex concepts into digestible insights, making them accessible for all. Join me as I share my thoughts, tutorials, and tips to navigate the exciting world of ML and DL. Connect with me on LinkedIn to explore collaboration opportunities!
Markdown supported · ⌘↵ to post
Vineet Theodore
In table 3, the function for Swish needs sigmoid(beta*x), unless beta is identically 1. From what I understand, beta is itself a trainable parameter, making it key to differentiate it from SiLU, where beta is identically 1.
Thus, the derivative should be: sigmoid(beta*x) plus beta times x times derivative_of_the_sigmoid function, where derivative of the sigmoid equals sigmoid(beta times x) times (1 minus sigmoid(beta times x))
Or if sigmoid is y, then derivative of y is equal to y times (1-y)