KL Divergence
تباعد كولباك–لايبلر
Definition
A measure of how much a probability distribution differs from a reference distribution; it is not symmetric.
Definition
KL divergence measures how one probability distribution differs from a reference distribution; it is non-negative, asymmetric, and zero only if the distributions are equal.
Academic explanation
\(\mathrm{KL}(p\Vert q)=\sum_x p(x)\log\frac{p(x)}{q(x)}\).
Practical examples
import numpy as np
p = np.array([0.5, 0.5])
q = np.array([0.9, 0.1])
print(round(float((p * np.log(p / q)).sum()), 3))Expected output
0.511
Mathematical equations
\mathrm{KL}(p\,\Vert\,q)=\sum_{x}p(x)\log\frac{p(x)}{q(x)}