Cybenko theorem

The Cybenko theorem is a theorem proved by George Cybenko in 1989 that says that a single hidden layer, feed forward neural network is capable of approximating any continuous, multivariate function to any desired degree of accuracy and that failure to map a function arises from poor choices for $mathbf\{w\}\_1,\; mathbf\{w\}\_2,\; dots\; ,\; mathbf\{w\}\_N,\; mathbf\{alpha\},$ and $mathbf\{\; heta\}$ or an insufficient number of hidden neurons.

Formal statement

Let $varphi$ be any continuous sigmoid-type function, e.g., $varphi(xi)\; =\; 1/(1+e^\{-xi\})$. Then, given any continuous real-valued function $f$ on $[0,1]\; ^n$ (or any other compact subset of $R^n$) and $epsilon\; >\; 0$, there exist vectors $mathbf\{w\_1\},\; mathbf\{w\_2\},\; dots,\; mathbf\{w\_N\},\; mathbf\{alpha\}$ and $mathbf\{\; heta\}$ and a parameterized function $G(mathbf\{cdot\},mathbf\{w\},mathbf\{alpha\},mathbf\{\; heta\}):\; [0,1]\; ^n\; ightarrow\; R$ such that

for all $mathbf\{x\}\; in\; [0,1]\; ^n$

where

$G(mathbf\{x\},mathbf\{w\},mathbf\{alpha\},mathbf\{\; heta\})\; =\; sum\_\{i=1\}^Nalpha\_ivarphi(mathbf\{w\}\_i^Tmathbf\{x\}\; +\; heta\_i)$

and $mathbf\{w\}\_i\; in\; R^n,\; alpha\_i,\; heta\_i\; in\; R,\; mathbf\{w\}\; =\; (mathbf\{w\}\_1,\; mathbf\{w\}\_2,\; dots\; mathbf\{w\}\_N),\; mathbf\{alpha\}\; =\; (alpha\_1,\; alpha\_2,\; dots,\; alpha\_N),$ and $mathbf\{\; heta\}\; =\; (\; heta\_1,\; heta\_2,\; dots\; ,\; heta\_N)$.

References

* Cybenko, G.V. (1989). Approximation by Superpositions of a Sigmoidal function, "Mathematics of Control, Signals and Systems", vol. 2 no. 4 pp. 303-314. [http://actcomm.dartmouth.edu/gvc/papers/approx_by_superposition.pdf electronic version]
* Hassoun, M. (1995) "Fundamentals of Artificial Neural Networks" MIT Press, p.48