Universal approximation theorem

The classical arbitrary width version of the theorem may be stated as follows.

Universal approximation theorem; arbitrary width.[4] Let ${\displaystyle \varphi$ :\mathbb {R} \to \mathbb {R} } be any continuous function (called the activation function). Let ${\displaystyle K\subseteq \mathbb {R} ^{n}}$ be compact. The space of real-valued continuous functions on ${\displaystyle K}$ is denoted by ${\displaystyle C(K)}$. Let ${\displaystyle {\mathcal {M}}}$ denote the space of functions of the form

${\displaystyle F(x)=\sum _{i=1}^{N}v_{i}\varphi \left(w_{i}^{T}x+b_{i}\right)}$

for all integers ${\displaystyle N\in \mathbb {N} }$, real constants ${\displaystyle v_{i},b_{i}\in \mathbb {R} }$ and real vectors ${\displaystyle w_{i}\in \mathbb {R} ^{m}}$ for ${\displaystyle i=1,\ldots ,N}$.

Then, if and only if ${\displaystyle \varphi }$ is nonpolynomial, the following statement is true: given any ${\displaystyle \varepsilon >0}$ and any ${\displaystyle f\in C(K)}$, there exists ${\displaystyle F\in {\mathcal {M}}}$ such that

${\displaystyle |F(x)-f(x)|<\varepsilon }$

for all ${\displaystyle x\in K}$.

In other words, ${\displaystyle {\mathcal {M}}}$ is dense in ${\displaystyle C(K)}$ with respect to the uniform norm if and only if ${\displaystyle \varphi }$ is nonpolynomial.

This theorem extends straightforwardly to networks with any fixed number of hidden layers: the theorem implies that the first layer can approximate any desired function, and that later layers can approximate the identity function. Thus any fixed-depth network may approximate any continuous function, and this version of the theorem applies to networks with bounded depth and arbitrary width.

The 'dual' version of the theorem considers networks of bounded width and arbitrary depth. Unlike the previous theorem, it gives sufficient but not necessary conditions.

Universal approximation theorem; arbitrary depth.[7] Let ${\displaystyle \varphi$ :\mathbb {R} \to \mathbb {R} } be any nonaffine continuous function which is continuously differentiable at at least one point, with nonzero derivative at that point. Let ${\displaystyle K\subseteq \mathbb {R} ^{n}}$ be compact. The space of real vector-valued continuous functions on ${\displaystyle K}$ is denoted by ${\displaystyle C(K;\mathbb {R} ^{m})}$. Let ${\displaystyle {\mathcal {N}}}$ denote the space of feedforward neural networks with ${\displaystyle n}$ input neurons, ${\displaystyle m}$ output neurons, and an arbitrary number of hidden layers each with ${\displaystyle n+m+2}$ neurons, such that every hidden neuron has activation function ${\displaystyle \varphi }$ and every output neuron has the identity as its activation function. Then given any ${\displaystyle \varepsilon >0}$ and any ${\displaystyle f\in C(K;\mathbb {R} ^{m})}$, there exists ${\displaystyle F\in {\mathcal {N}}}$ such that

${\displaystyle |F(x)-f(x)|<\varepsilon }$

for all ${\displaystyle x\in K}$.

In other words, ${\displaystyle {\mathcal {N}}}$ is dense in ${\displaystyle C(K;\mathbb {R} ^{m})}$ with respect to the uniform norm.

Certain necessary conditions for the bounded width, arbitrary depth case have been established, but there is still a gap between the known sufficient and necessary conditions.[5][6][9]

The arbitrary depth case includes polynomial activation functions, which are specifically excluded from the arbitrary width case. This is an example of a qualitative difference between (particular interpretations of) deep and shallow neural networks.

• Kolmogorov–Arnold representation theorem
• No free lunch theorem
• Stone–Weierstrass theorem
• Fourier series