f-divergence

Let P and Q be two probability distributions over a space Ω such that P is absolutely continuous with respect to Q. Then, for a convex function f such that f(1) = 0, the f-divergence of P from Q is defined as

${\displaystyle D_{f}(P\parallel Q)\equiv \int _{\Omega }f\left({\frac {dP}{dQ}}\right)\,dQ.}$

If P and Q are both absolutely continuous with respect to a reference distribution μ on Ω then their probability densities p and q satisfy dP = p dμ and dQ = q dμ. In this case the f-divergence can be written as

${\displaystyle D_{f}(P\parallel Q)=\int _{\Omega }f\left({\frac {p(x)}{q(x)}}\right)q(x)\,d\mu (x).}$

The f-divergences can be expressed using Taylor series and rewritten using a weighted sum of chi-type distances (Nielsen & Nock (2013)).

Many common divergences, such as KL-divergence, Hellinger distance, and total variation distance, are special cases of f-divergence, coinciding with a particular choice of f. The following table lists many of the common divergences between probability distributions and the f function to which they correspond (cf. Liese & Vajda (2006)).

Divergence	Corresponding f(t)
KL-divergence	${\displaystyle t\log t}$
reverse KL-divergence	${\displaystyle -\log t}$
squared Hellinger distance	${\displaystyle ({\sqrt {t}}-1)^{2},\,2(1-{\sqrt {t}})}$
Total variation distance	${\displaystyle {\frac {1}{2}}|t-1|\,}$
Pearson ${\displaystyle \chi ^{2}}$-divergence	${\displaystyle (t-1)^{2},\,t^{2}-1,\,t^{2}-t}$
Neyman ${\displaystyle \chi ^{2}}$-divergence (reverse Pearson)	${\displaystyle {\frac {1}{t}}-1,\,{\frac {1}{t}}-t}$
α-divergence	${\displaystyle {\begin{cases}{\frac {4}{1-\alpha ^{2}}}{\big (}1-t^{(1+\alpha )/2}{\big )},&{\text{if}}\ \alpha \neq \pm 1,\\t\ln t,&{\text{if}}\ \alpha =1,\\-\ln t,&{\text{if}}\ \alpha =-1\end{cases}}}$
Jensen-Shannon Divergence	${\displaystyle (t+1)\log {\big (}{\frac {2}{t+1}}{\big )}+t\log t}$
α-divergence (other designation)	${\displaystyle {\begin{cases}{\frac {t^{\alpha }-t}{\alpha (\alpha -1)}},&{\text{if}}\ \alpha \neq 0,\,\alpha \neq 1,\\t\ln t,&{\text{if}}\ \alpha =1,\\-\ln t,&{\text{if}}\ \alpha =0\end{cases}}}$

The function ${\displaystyle f(t)}$ is defined up to the summand ${\displaystyle c(t-1)}$, where ${\displaystyle c}$ is any constant.

In particular, the monotonicity implies that if a Markov process has a positive equilibrium probability distribution ${\displaystyle P^{*}}$ then ${\displaystyle D_{f}(P(t)\parallel P^{*})}$ is a monotonic (non-increasing) function of time, where the probability distribution ${\displaystyle P(t)}$ is a solution of the Kolmogorov forward equations (or Master equation), used to describe the time evolution of the probability distribution in the Markov process. This means that all f-divergences ${\displaystyle D_{f}(P(t)\parallel P^{*})}$ are the Lyapunov functions of the Kolmogorov forward equations. Reverse statement is also true: If ${\displaystyle H(P)}$ is a Lyapunov function for all Markov chains with positive equilibrium ${\displaystyle P^{*}}$ and is of the trace-form (${\displaystyle H(P)=\sum _{i}f(P_{i},P_{i}^{*})}$) then ${\displaystyle H(P)=D_{f}(P(t)\parallel P^{*})}$, for some convex function f.[2][3] For example, Bregman divergences in general do not have such property and can increase in Markov processes.[4]

• Kullback–Leibler divergence
• Bregman divergence