Regularization
التنظيم
Definition
A set of techniques that constrain a model or penalize its complexity during training to improve its generalization to new data.
Definition
Regularization is any modification to a learning algorithm intended to reduce its generalisation error but not its training error, commonly by adding a penalty on model complexity such as the L1 or L2 norm of the weights.
Practical examples
import numpy as np
from sklearn.linear_model import Lasso, Ridge
rng = np.random.default_rng(0)
X = rng.normal(size=(100, 5))
y = 3 * X[:, 0] + 0.1 * rng.normal(size=100) # خاصية واحدة مفيدة فقط
print(np.round(Ridge(alpha=1.0).fit(X, y).coef_, 2))
print(np.round(Lasso(alpha=0.1).fit(X, y).coef_, 2))Expected output
[ 2.95 0.01 0.01 -0. 0. ] [ 2.87 0. 0. -0. 0. ]
Mathematical equations
J(\mathbf{w}) = L(\mathbf{w}) + \lambda\, \Omega(\mathbf{w}), \quad \Omega_{L2} = \lVert \mathbf{w} \rVert_2^2, \quad \Omega_{L1} = \lVert \mathbf{w} \rVert_1Scientific sources
References and scientific sources
-
Research paperRegression Shrinkage and Selection via the Lasso Lasso regression and L1 regularization.
-
Research paperRidge Regression: Biased Estimation for Nonorthogonal Problems Ridge regression and L2 regularization.
-
Research paperDropout: A Simple Way to Prevent Neural Networks from Overfitting The dropout technique for reducing overfitting.
-
BookDeep Learning A comprehensive reference book on deep learning, free to read on the web.
