Using a saturating activation function, such as ReLU, can cause slow convergence in deep neural networks because it produces very small gradients when the input is large. This makes optimization difficult. Switching to a non-saturating function like the log sigmoid or tanh can help, as they produce larger gradients and improve optimization.