Gradient Clipping

قص التدرج

Definition

Capping the size of the gradient at an upper limit before updating the weights, to prevent it from exploding.

Definition

Gradient clipping caps the norm (or value) of gradients before the update step, preventing exploding gradients and unstable training.

Academic explanation

\(g\leftarrow g\cdot\min(1,\tau/\|g\|)\).

Learn and review

Suggested terms to learn next