Gradient Descent
التدرج الاشتقاقي
Definition
An iterative optimization algorithm that minimizes the loss function by moving the parameters in small steps opposite to the gradient.
Definition
Gradient descent is an iterative optimisation algorithm that minimises a function by repeatedly updating the parameters in the direction of the negative gradient, scaled by a learning rate.
Practical examples
theta, lr = 1.0, 0.1
for step in range(5):
grad = 2 * theta # مشتقة theta**2
theta -= lr * grad
print(step + 1, round(theta, 4))Expected output
1 0.8 2 0.64 3 0.512 4 0.4096 5 0.3277
Mathematical equations
\theta_{t+1} = \theta_t - \eta\, \nabla_{\theta} L(\theta_t)Scientific sources
References and scientific sources
-
BookDeep Learning A comprehensive reference book on deep learning, free to read on the web.
-
Research paperAdam: A Method for Stochastic Optimization The Adam algorithm for gradient-based optimization.
