Overfitting
الضبط الزائد (الملائمة الزائدة)
Definition
A situation in which the model memorizes the details and noise of the training data, so it performs well on them and poorly on new data.
Definition
Overfitting occurs when a model fits the training data, including its noise, so closely that it fails to generalise to unseen data; it shows as low training error but high validation or test error.
Practical examples
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
for depth in [None, 3]:
m = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_tr, y_tr)
print(depth, round(m.score(X_tr, y_tr), 3), round(m.score(X_te, y_te), 3))Expected output
None 1.0 0.912 3 0.967 0.947
Mathematical equations
\mathbb{E}\big[(y-\hat f(x))^2\big] = \mathrm{Bias}\big[\hat f(x)\big]^2 + \mathrm{Var}\big[\hat f(x)\big] + \sigma^2Scientific sources
References and scientific sources
-
ArticleA Few Useful Things to Know About Machine Learning An article on overfitting, feature engineering and model generalization.
-
BookThe Elements of Statistical Learning (2nd ed.) A statistical reference on machine learning, freely available from the authors' website.
-
Research paperDropout: A Simple Way to Prevent Neural Networks from Overfitting The dropout technique for reducing overfitting.
