Momentum

الزخم

Definition

An addition to gradient descent that accumulates a moving average of past gradients to speed up movement and dampen oscillation.

Definition

Momentum accumulates an exponentially decaying average of past gradients to accelerate descent along consistent directions and damp oscillations.

Academic explanation

\(v_{t+1}=\beta v_t+g_t,\;\theta_{t+1}=\theta_t-\eta v_{t+1}\).

Mathematical equations

v_{t+1}=\beta v_t+g_t,\qquad \theta_{t+1}=\theta_t-\eta\,v_{t+1}

Learn and review

Suggested terms to learn next