Deep Dive into Deep Learning

Optimization Algorithms

If you have read the book in order, you have already used several optimization algorithms to train deep learning models. They are the tools that let us keep updating model parameters to reduce the loss on the training set. Someone who treats optimization as a black box for minimizing objective functions in simple settings may be satisfied knowing that a few such recipes exist, with names like “SGD” and “Adam”.

To do well, however, deeper knowledge is needed. On the one hand, training a complex deep learning model can take hours, days or even weeks, and the performance of the optimization algorithm directly affects training efficiency. On the other hand, understanding the principles of different optimization algorithms and the role of their hyperparameters lets us tune them in a targeted way to improve the performance of deep learning models.

This chapter explores common deep learning optimization algorithms in depth. Almost all optimization problems in deep learning are nonconvex, yet designing and analyzing algorithms for convex problems has proved very instructive. That is why the chapter includes a primer on convex optimization and a proof for a very simple stochastic gradient descent algorithm on a convex objective function.

 

Book link

Post links

Browse the book

Cite this book

Taima, Alaa. (2022). Optimization Algorithms. Deep Learning in Arabic. https://dlarabic.com/en/articles/optimization-algorithms/

Similar books you may like

Leave a comment

Your email address will not be published. Required fields are marked *