KL Divergence

تباعد كولباك–لايبلر

Abbreviation: KL

Definition

A measure of how much a probability distribution differs from a reference distribution; it is not symmetric.

Definition

KL divergence measures how one probability distribution differs from a reference distribution; it is non-negative, asymmetric, and zero only if the distributions are equal.

Academic explanation

\(\mathrm{KL}(p\Vert q)=\sum_x p(x)\log\frac{p(x)}{q(x)}\).

Practical examples

Code example Python
import numpy as np

p = np.array([0.5, 0.5])
q = np.array([0.9, 0.1])
print(round(float((p * np.log(p / q)).sum()), 3))

Expected output

0.511

Mathematical equations

\mathrm{KL}(p\,\Vert\,q)=\sum_{x}p(x)\log\frac{p(x)}{q(x)}

Learn and review

Suggested terms to learn next