Gradient Descent

التدرج الاشتقاقي

Abbreviation: GD

Definition

An iterative optimization algorithm that minimizes the loss function by moving the parameters in small steps opposite to the gradient.

Definition

Gradient descent is an iterative optimisation algorithm that minimises a function by repeatedly updating the parameters in the direction of the negative gradient, scaled by a learning rate.

Practical examples

Code example Python
theta, lr = 1.0, 0.1
for step in range(5):
    grad = 2 * theta          # مشتقة theta**2
    theta -= lr * grad
    print(step + 1, round(theta, 4))

Expected output

1 0.8
2 0.64
3 0.512
4 0.4096
5 0.3277

Mathematical equations

\theta_{t+1} = \theta_t - \eta\, \nabla_{\theta} L(\theta_t)

Scientific sources

References and scientific sources

  • Book
    Deep Learning Ian Goodfellow, Yoshua Bengio, Aaron Courville · 2016 · MIT Press A comprehensive reference book on deep learning, free to read on the web.
  • Research paper
    Adam: A Method for Stochastic Optimization Diederik P. Kingma, Jimmy Ba · 2014 · arXiv:1412.6980 (ICLR 2015) The Adam algorithm for gradient-based optimization.

Learn and review

Suggested terms to learn next