Momentum
الزخم
Definition
An addition to gradient descent that accumulates a moving average of past gradients to speed up movement and dampen oscillation.
Definition
Momentum accumulates an exponentially decaying average of past gradients to accelerate descent along consistent directions and damp oscillations.
Academic explanation
\(v_{t+1}=\beta v_t+g_t,\;\theta_{t+1}=\theta_t-\eta v_{t+1}\).
Mathematical equations
v_{t+1}=\beta v_t+g_t,\qquad \theta_{t+1}=\theta_t-\eta\,v_{t+1}