Virtual Study Hub

Using a saturating activation function, such as ReLU, can cause slow convergence in deep neural networks because it produces very small gradients when the input is large. This makes optimization difficult. Switching to a non-saturating function like the log sigmoid or tanh can help, as they produce larger gradients and improve optimization.

Leave a Reply

Your email address will not be published. Required fields are marked *