Universal approximation theorem

Universal approximation theorem

In the mathematical theory of neural networks, the universal approximation theorem states[1] that the standard multilayer feed-forward network with a single hidden layer that contains finite number of hidden neurons, and with arbitrary activation function are universal approximators on a compact subset of Rn.

The theorem was proved by George Cybenko in 1989 for a sigmoid activation function, thus it is also called the Cybenko theorem.[2]

Kurt Hornik (1991)[3] showed that it is not the specific choice of the activation function, but rather the multilayer feedforward architecture itself which gives neural networks the potential of being universal approximators. The output units are always assumed to be linear. For notational convenience we shall explicitly formulate our results only for the case where there is only one output unit. (The general case can easily be deduced from the simple case.)

The theorem[2][3][4][5] in mathematical terms:

Formal statement

Let φ(·) be a nonconstant, bounded, and monotonically-increasing continuous function. Let Im denote the m-dimensional unit hypercube [0,1]m. The space of continuous functions on Im0 is denoted by C(Im). Then, given any function fC(Im) and є > 0, there exist an integer N and sets of real constants αi, biR, wiRm, where i = 1, ..., N such that we may define:

  F( x ) =
  \sum_{i=1}^{N} \alpha_i \varphi \left( w_i^T x + b_i\right)
as an approximate realization of the function f; that is,
| F(x) − f(x) | < ε
for all xIm.


  1. ^ Balázs Csanád Csáji. Approximation with Artificial Neural Networks; Faculty of Sciences; Eötvös Loránd University, Hungary
  2. ^ a b Cybenko., G. (1989) "Approximations by superpositions of sigmoidal functions", Mathematics of Control, Signals, and Systems, 2 (4), 303-314
  3. ^ a b Kurt Hornik (1991) "Approximation Capabilities of Multilayer Feedforward Networks", Neural Networks, 4(2), 251–257
  4. ^ Haykin, Simon (1998). Neural Networks: A Comprehensive Foundation, Volume 2, Prentice Hall. ISBN 0132733501.
  5. ^ Hassoun, M. (1995) Fundamentals of Artificial Neural Networks MIT Press, p. 48

Wikimedia Foundation. 2010.

Поможем написать курсовую

Look at other dictionaries:

  • Newton's theorem of revolving orbits — Figure 1: An attractive force F(r) causes the blue planet to move on the cyan circle. The green planet moves three times faster and thus requires a stronger centripetal force, which is supplied by adding an attractive inverse cube force. The …   Wikipedia

  • Nyquist–Shannon sampling theorem — Fig.1: Hypothetical spectrum of a bandlimited signal as a function of frequency The Nyquist–Shannon sampling theorem, after Harry Nyquist and Claude Shannon, is a fundamental result in the field of information theory, in particular… …   Wikipedia

  • Central limit theorem — This figure demonstrates the central limit theorem. The sample means are generated using a random number generator, which draws numbers between 1 and 100 from a uniform probability distribution. It illustrates that increasing sample sizes result… …   Wikipedia

  • PCP theorem — In computational complexity theory, the PCP theorem states that every decision problem in the NP complexity class has probabilistically checkable proofs (proofs that can be checked by a randomized algorithm) of constant query complexity and… …   Wikipedia

  • Grushko theorem — In the mathematical subject of group theory, the Grushko theorem or the Grushko Neumann theorem is a theorem stating that the rank (that is, the smallest cardinality of a generating set) of a free product of two groups is equal to the sum of the… …   Wikipedia

  • Newton's law of universal gravitation — Classical mechanics Newton s Second Law History of classical mechanics  …   Wikipedia

  • List of theorems — This is a list of theorems, by Wikipedia page. See also *list of fundamental theorems *list of lemmas *list of conjectures *list of inequalities *list of mathematical proofs *list of misnamed theorems *Existence theorem *Classification of finite… …   Wikipedia

  • List of mathematics articles (U) — NOTOC U U duality U quadratic distribution U statistic UCT Mathematics Competition Ugly duckling theorem Ulam numbers Ulam spiral Ultraconnected space Ultrafilter Ultrafinitism Ultrahyperbolic wave equation Ultralimit Ultrametric space… …   Wikipedia

  • Feedforward neural network — A feedforward neural network is an artificial neural network where connections between the units do not form a directed cycle. This is different from recurrent neural networks.The feedforward neural network was the first and arguably simplest… …   Wikipedia

  • Vector space — This article is about linear (vector) spaces. For the structure in incidence geometry, see Linear space (geometry). Vector addition and scalar multiplication: a vector v (blue) is added to another vector w (red, upper illustration). Below, w is… …   Wikipedia

Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”