KL Divergence

Shipping

array_kl_divergence

Kullback-Leibler divergence D(p‖q) between two probability distributions

Signature

Inputs

pArrayrequired— The 'true' / reference distribution $p$ (non-negative).
qArrayrequired— The 'model' / approximating distribution $q$ (non-negative, same length as $p$).

Outputs

kl_divergenceScalar— $D_{\mathrm{KL}}(p\,\|\,q) = \sum_i p_i \ln(p_i / q_i)$ (nats).

Description

KL Divergence computes the Kullback-Leibler divergence $D_{KL} (p ∥ q) = \sum_{i} p_{i} ln \frac{p _{i}}{q _{i}}$ — the information lost, in nats, when distribution $q$ is used to approximate the reference distribution $p$ . It is $0$ if and only if $p = q$ and strictly positive otherwise, giving a directed 'distance' between distributions.

KL divergence is asymmetric: $D_{KL} (p ∥ q) \neq = D_{KL} (q ∥ p)$ in general, so port order matters — p is the truth, q the approximation. The node is stateless.

Mathematics

D_{KL} (p ∥ q) = i \sum p_{i} ln \frac{p _{i}}{q _{i}} \geq 0, D_{KL} (p ∥ q) = 0 ⟺ p = q

Examples

Identical distributions

With p = q = [0.5, 0.5], every term $p_{i} ln (p_{i} / q_{i}) = 0$ , so kl_divergence $= 0$ — no information is lost when the model matches the truth exactly.

Applications

Measuring how far a fitted/model distribution strays from an empirical one.
Drift detection: divergence of a live histogram from a reference baseline.
Objective terms in variational and information-theoretic methods.

Neat

The divergence is directed: swapping `p` and `q` generally changes the value, which is exactly why the ports are named rather than symmetric.

Feed two Array-Histogram count vectors to get an empirical KL between two datasets straight from raw samples.