What is optimization in deep learning?

Optimization in deep learning is the process of adjusting the model’s weights and parameters to minimize the loss function and make the model learn the most suitable representations of the data. It is done with optimization algorithms such as gradient descent and its variants (SGD, Adam, RMSProp), which rely on computing derivatives (gradients) to move the weights toward their optimal values. Thanks to optimization, the model becomes more accurate in its predictions and can generalize to new data instead of merely memorizing the training data.