Divergence (statistics)

Suppose S is a space of all probability distributions with common support. Then a divergence on S is a function D(· || ·): S×S → R satisfying [1]

1. D(p || q) ≥ 0 for all p, q ∈ S,
2. D(p || q) = 0 if and only if p = q,

The dual divergence D* is defined as

${\displaystyle D^{*}(p\parallel q)=D(q\parallel p).}$

Many properties of divergences can be derived if we restrict S to be a statistical manifold, meaning that it can be parametrized with a finite-dimensional coordinate system θ, so that for a distribution p ∈ S we can write p = p(θ).

For a pair of points p, q ∈ S with coordinates θ_p and θ_q, denote the partial derivatives of D(p || q) as

${\displaystyle {\begin{aligned}D((\partial _{i})_{p}\parallel q)\ \ &{\stackrel {\mathrm {def} }{=}}\ \ {\tfrac {\partial }{\partial \theta _{p}^{i}}}D(p\parallel q),\\D((\partial _{i}\partial _{j})_{p}\parallel (\partial _{k})_{q})\ \ &{\stackrel {\mathrm {def} }{=}}\ \ {\tfrac {\partial }{\partial \theta _{p}^{i}}}{\tfrac {\partial }{\partial \theta _{p}^{j}}}{\tfrac {\partial }{\partial \theta _{q}^{k}}}D(p\parallel q),\ \ \mathrm {etc.} \end{aligned}}}$

Now we restrict these functions to a diagonal p = q, and denote [2]

${\displaystyle {\begin{aligned}D[\partial _{i}\parallel \cdot ]\ &:\ p\mapsto D((\partial _{i})_{p}\parallel p),\\D[\partial _{i}\parallel \partial _{j}]\ &:\ p\mapsto D((\partial _{i})_{p}\parallel (\partial _{j})_{p}),\ \ \mathrm {etc.} \end{aligned}}}$

By definition, the function D(p || q) is minimized at p = q, and therefore

${\displaystyle {\begin{aligned}&D[\partial _{i}\parallel \cdot ]=D[\cdot \parallel \partial _{i}]=0,\\&D[\partial _{i}\partial _{j}\parallel \cdot ]=D[\cdot \parallel \partial _{i}\partial _{j}]=-D[\partial _{i}\parallel \partial _{j}]\ \equiv \ g_{ij}^{(D)},\end{aligned}}}$

where matrix g^(D) is positive semi-definite and defines a unique Riemannian metric on the manifold S.

Divergence D(· || ·) also defines a unique torsion-free affine connection ∇^(D) with coefficients

${\displaystyle \Gamma _{ij,k}^{(D)}=-D[\partial _{i}\partial _{j}\parallel \partial _{k}],}$

and the dual to this connection ∇* is generated by the dual divergence D*.

Thus, a divergence D(· || ·) generates on a statistical manifold a unique dualistic structure (g^(D), ∇^(D), ∇^(D*)). The converse is also true: every torsion-free dualistic structure on a statistical manifold is induced from some globally defined divergence function (which however need not be unique).[3]

For example, when D is an f-divergence for some function ƒ(·), then it generates the metric g^(D_f) = c·g and the connection ∇^(D_f) = ∇^(α), where g is the canonical Fisher information metric, ∇^(α) is the α-connection, c = ƒ′′(1), and α = 3 + 2ƒ′′′(1)/ƒ′′(1).

The two most important divergences are the relative entropy (Kullback–Leibler divergence, KL divergence), which is central to information theory and statistics, and the squared Euclidean distance (SED). Minimizing these two divergences is the main way that linear inverse problem are solved, via the principle of maximum entropy and least squares, notably in logistic regression and linear regression.[4]

The two most important classes of divergences are the f-divergences and Bregman divergences; however, other types of divergence functions are also encountered in the literature. The only divergence that is both an f-divergence and a Bregman divergence is the Kullback–Leibler divergence; the squared Euclidean divergence is a Bregman divergence (corresponding to the function ${\displaystyle x^{2}}$), but not an f-divergence.

f-divergences

Main article: f-divergence

This family of divergences are generated through functions f(u), convex on u > 0 and such that f(1) = 0. Then an f-divergence is defined as

${\displaystyle D_{f}(p\parallel q)=\int p(x)f{\bigg (}{\frac {q(x)}{p(x)}}{\bigg )}dx}$

Kullback–Leibler divergence:	${\displaystyle D_{\mathrm {KL} }(p\parallel q)=\int p(x)\ln \left({\frac {p(x)}{q(x)}}\right)dx}$
squared Hellinger distance:	${\displaystyle H^{2}(p,\,q)=2\int {\Big (}{\sqrt {p(x)}}-{\sqrt {q(x)}}\,{\Big )}^{2}dx}$
Jeffreys divergence:	${\displaystyle D_{J}(p\parallel q)=\int (p(x)-q(x)){\big (}\ln p(x)-\ln q(x){\big )}dx}$
Chernoff's α-divergence:	${\displaystyle D^{(\alpha )}(p\parallel q)={\frac {4}{1-\alpha ^{2}}}{\bigg (}1-\int p(x)^{\frac {1-\alpha }{2}}q(x)^{\frac {1+\alpha }{2}}dx{\bigg )}}$
exponential divergence:	${\displaystyle D_{e}(p\parallel q)=\int p(x){\big (}\ln p(x)-\ln q(x){\big )}^{2}dx}$
Kagan's divergence:	${\displaystyle D_{\chi ^{2}}(p\parallel q)={\frac {1}{2}}\int {\frac {(p(x)-q(x))^{2}}{p(x)}}dx}$
(α,β)-product divergence:	${\displaystyle D_{\alpha ,\beta }(p\parallel q)={\frac {2}{(1-\alpha )(1-\beta )}}\int {\Big (}1-{\Big (}{\tfrac {q(x)}{p(x)}}{\Big )}^{\!\!{\frac {1-\alpha }{2}}}{\Big )}{\Big (}1-{\Big (}{\tfrac {q(x)}{p(x)}}{\Big )}^{\!\!{\frac {1-\beta }{2}}}{\Big )}p(x)dx}$

If a Markov process has a positive equilibrium probability distribution ${\displaystyle p^{*}}$ then ${\displaystyle D_{f}(p(t)\parallel p^{*})}$ is a monotonic (non-increasing) function of time, where the probability distribution ${\displaystyle p(t)}$ is a solution of the Kolmogorov forward equations (or Master equation), used to describe the time evolution of the probability distribution in the Markov process. This means that all f-divergencies ${\displaystyle D_{f}(p(t)\parallel p^{*})}$ are the Lyapunov functions of the Kolmogorov forward equations. Reverse statement is also true: If ${\displaystyle H(p)}$ is a Lyapunov function for all Markov chains with positive equilibrium ${\displaystyle p^{*}}$ and is of the trace-form (${\displaystyle H(p)=\sum _{i}h(p_{i},p_{i}^{*})}$) then ${\displaystyle H(p)=D_{f}(p(t)\parallel p^{*})}$, for some convex function f.[5][6] Bregman divergences in general do not have such property and can increase in Markov processes.

The term "divergence" for a statistical distance was used informally in various contexts from c. 1910 to c. 1940. Its formal use dates at least to Bhattacharyya (1943) harvtxt error: no target: CITEREFBhattacharyya1943 (help), entitled "On a measure of divergence between two statistical populations defined by their probability distributions", which defined the Bhattacharyya distance, and Bhattacharyya (1946) harvtxt error: no target: CITEREFBhattacharyya1946 (help), entitled "On a Measure of Divergence between Two Multinomial Populations", which defined the Bhattacharyya angle. The term was popularized by its use for the Kullback–Leibler divergence in Kullback & Leibler (1951) harvtxt error: no target: CITEREFKullbackLeibler1951 (help), its use in the textbook Kullback (1959) harvtxt error: no target: CITEREFKullback1959 (help), and then by Ali & Silvey (1966) harvtxt error: no target: CITEREFAliSilvey1966 (help) generally, for the class of f-divergences. The term "Bregman distance" is still found, but "Bregman divergence" is now preferred. In information geometry, alternative terms were initially used, including "quasi-distance" Amari (1982, p. 369) harvtxt error: no target: CITEREFAmari1982 (help) and "contrast function" Eguchi (1985), though "divergence" was used in Amari (1985) harvtxt error: no target: CITEREFAmari1985 (help) for the α-divergence, and has become standard (e.g., Amari & Cichocki (2010) harvtxt error: no target: CITEREFAmariCichocki2010 (help)).

• Statistical distance