---
title: "Comment by Vineet Theodore on \"Comprehensive Guide to the ReLU Activation Function in Neural Networks: Definition, Role, and Type Explained\" | Hashnode"
description: "In table 3, the function for Swish needs sigmoid(beta*x), unless beta is identically 1. From what I understand, beta is itself a trainable parameter, making it"
url: "https://hashnode.com/posts/comprehensive-guide-to-the-relu-activation-function-in-neural-networks-definition-role-and-type-explained/66e18a8cdb99bf3dc707d638/comment/684b43a1a132ee2871d880ea"
---

[Discussion](https://hashnode.com/posts/comprehensive-guide-to-the-relu-activation-function-in-neural-networks-definition-role-and-type-explained/66e18a8cdb99bf3dc707d638)

Comment [Comprehensive Guide to the ReLU Activation Function in Neural Networks: Definition, Role, and Type Explained](https://hashnode.com/posts/comprehensive-guide-to-the-relu-activation-function-in-neural-networks-definition-role-and-type-explained/66e18a8cdb99bf3dc707d638)

[Vineet Theodore](https://hashnode.com/@vf30kf)

[Jun 12, 2025](https://hashnode.com/posts/comprehensive-guide-to-the-relu-activation-function-in-neural-networks-definition-role-and-type-explained/66e18a8cdb99bf3dc707d638/comment/684b43a1a132ee2871d880ea)

In table 3, the function for Swish needs sigmoid(beta\*x), unless beta is identically 1. From what I understand, beta is itself a trainable parameter, making it key to differentiate it from SiLU, where beta is identically 1.

Thus, the derivative should be: sigmoid(beta\*x) plus beta times x times derivative\_of\_the\_sigmoid function, where derivative of the sigmoid equals sigmoid(beta times x) times (1 minus sigmoid(beta times x))

Or if sigmoid is y, then derivative of y is equal to y times (1-y)
