Gradient Clipping
قص التدرج
Definition
Capping the size of the gradient at an upper limit before updating the weights, to prevent it from exploding.
Definition
Gradient clipping caps the norm (or value) of gradients before the update step, preventing exploding gradients and unstable training.
Academic explanation
\(g\leftarrow g\cdot\min(1,\tau/\|g\|)\).
