ML Experiment Log: Squaring as activation layer

The purpose of an activation layer is to add some non-linearity to the network. This allows expressing much more complex tasks with our networks. But why do we generally use functions that have a low/zero less than zero component and a nearly 1 comp...