Sanov's theorem

In information theory, Sanov's theorem gives a bound on the probability of observing an atypical sequence of samples from a given probability distribution. In the language of large deviations theory, Sanov's theorem identifies the rate function for large deviations of the empirical measure of a sequence of i.i.d. random variables.

Let A be a set of probability distributions over an alphabet X, and let q be an arbitrary distribution over X (where q may or may not be in A). Suppose we draw n i.i.d. samples from q, represented by the vector ${\displaystyle x^{n}=x_{1},x_{2},\ldots ,x_{n}}$. Further, let us ask that the empirical measure, ${\displaystyle {\hat {p}}_{x^{n}}}$, of the samples falls within the set A—formally, we write ${\displaystyle \{x^{n}:{\hat {p}}_{x^{n}}\in A\}}$. Then,

${\displaystyle q^{n}(x^{n})\leq (n+1)^{|X|}2^{-nD_{\mathrm {KL} }(p^{*}||q)}}$,

where

• ${\displaystyle q^{n}(x^{n})}$ is shorthand for ${\displaystyle q(x_{1})q(x_{2})\cdots q(x_{n})}$, and
• ${\displaystyle p^{*}}$ is the information projection of q onto A.

In words, the probability of drawing an atypical distribution is a function of the KL divergence from the true distribution to the atypical one; in the case that we consider a set of possible atypical distributions, there is a dominant atypical distribution, given by the information projection.

Furthermore, if A is the closure of its interior,

${\displaystyle \lim _{n\to \infty }{\frac {1}{n}}\log q^{n}(x^{n})=-D_{\mathrm {KL} }(p^{*}||q).}$