Regularization

التنظيم

Definition

A set of techniques that constrain a model or penalize its complexity during training to improve its generalization to new data.

Definition

Regularization is any modification to a learning algorithm intended to reduce its generalisation error but not its training error, commonly by adding a penalty on model complexity such as the L1 or L2 norm of the weights.

Practical examples

Code example Python
import numpy as np
from sklearn.linear_model import Lasso, Ridge

rng = np.random.default_rng(0)
X = rng.normal(size=(100, 5))
y = 3 * X[:, 0] + 0.1 * rng.normal(size=100)  # خاصية واحدة مفيدة فقط

print(np.round(Ridge(alpha=1.0).fit(X, y).coef_, 2))
print(np.round(Lasso(alpha=0.1).fit(X, y).coef_, 2))

Expected output

[ 2.95  0.01  0.01 -0.    0.  ]
[ 2.87  0.    0.   -0.    0.  ]

Mathematical equations

J(\mathbf{w}) = L(\mathbf{w}) + \lambda\, \Omega(\mathbf{w}), \quad \Omega_{L2} = \lVert \mathbf{w} \rVert_2^2, \quad \Omega_{L1} = \lVert \mathbf{w} \rVert_1

Scientific sources

References and scientific sources

Learn and review

Suggested terms to learn next