Smooth maximum

In mathematics, a smooth maximum of an indexed family x₁, ..., x_n of numbers is a differentiable approximation to the maximum function

x_{1},\ldots ,x_{n}\mapsto \max(x_{1},\ldots ,x_{n}),

and the concept of smooth minimum is similarly defined.

Examples

For large positive values of the parameter $\alpha >0$ , the following formulation is one smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.

{\mathcal {S}}_{\alpha }(x_{1},\ldots ,x_{n})={\frac {\sum _{i=1}^{n}x_{i}e^{\alpha x_{i}}}{\sum _{i=1}^{n}e^{\alpha x_{i}}}}

${\mathcal {S}}_{\alpha }$ has the following properties:

${\mathcal {S}}_{\alpha }\to \max$ as $\alpha \to \infty$
${\mathcal {S}}_{0}$ is the arithmetic mean of its inputs
${\mathcal {S}}_{\alpha }\to \min$ as $\alpha \to -\infty$

The gradient of ${\mathcal {S}}_{{\alpha }}$ is closely related to softmax and is given by

\nabla _{x_{i}}{\mathcal {S}}_{\alpha }(x_{1},\ldots ,x_{n})={\frac {e^{\alpha x_{i}}}{\sum _{j=1}^{n}e^{\alpha x_{j}}}}[1+\alpha (x_{i}-{\mathcal {S}}_{\alpha }(x_{1},\ldots ,x_{n}))].

This makes the softmax function useful for optimization techniques that use gradient descent.

LogSumExp

Another smooth maximum is LogSumExp:

\mathrm {LSE} (x_{1},\ldots ,x_{n})=\log(\exp(x_{1})+\ldots +\exp(x_{n}))

This can also be normalized if the $x_{i}$ are all non-negative, yielding a function with domain $[0,\infty )^{n}$ and range $[0,\infty )$ :

g(x_{1},\ldots ,x_{n})=\log(\exp(x_{1})+\ldots +\exp(x_{n})-(n-1))

The $(n-1)$ term corrects for the fact that $\exp(0)=1$ by canceling out all but one zero exponential, and $\log 1=0$ if all $x_{i}$ are zero.

References

M. Lange, D. Zühlke, O. Holz, and T. Villmann, "Applications of lp-norms and their smooth approximations for gradient based learning vector quantization," in Proc. ESANN, Apr. 2014, pp. 271-276. (https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2014-153.pdf)

This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.

Smooth maximum

Examples

LogSumExp

See also

References