Here’s a great list of articles covering optimizers, momentum, adaptive learning rates, batch normalization, and more in regards to optimizing neural networks.
“The goal is to hit the sweet spot of maximum value optimization, where foolish risk is balanced against excessive caution.” ― Steven J. Bowen