- Reproducing kernel Hilbert space
In
functional analysis (a branch ofmathematics ), a reproducing kernel Hilbert space is aHilbert space of functions in which pointwise evaluation is a continuous linear functional. Equivalently, they are spaces that can be defined by reproducing kernels. The subject was originally and simultaneously developed byNachman Aronszajn (1907-1980) andStefan Bergman (1895-1987) in1950 .In this article we assume that
Hilbert space s are complex. The main reason for this is that many of the examples of reproducing kernelHilbert space s are spaces of analytic functions, although some real Hilbert spaces also have reproducing kernels.An important subset of the reproducing kernel Hilbert spaces are the reproducing kernel Hilbert spaces associated to a continuous kernel. These spaces have applications in machine learning.
Definition
Let "X" be an arbitrary set and "H" a
Hilbert space of complex-valued functions on "X". We say that "H" is a reproducing kernel Hilbert space if the linear map:f mapsto f(x)
from "H" to the complex numbers is continuous for any "x" in "X". By the
Riesz representation theorem , this implies that for every "x" in "X" there exists an element "K""x" of "H" with the property that::f(x) = langle f, K_x angle quad forall f in H quad (*)
The function "K""x" is called the point-evaluation functional at the point "x".
Since "H" is a space of functions, the element "K""x" is itself a function and can therefore be evaluated at every point. We define the function K: X imes X o mathbb{C} by
:K(x,y) stackrel{mathrm{def{=} K_x(y).
This function is called the reproducing kernel for the Hilbert space "H" and it is determined entirely by "H" because the Riesz representation theorem guarantees, for every "x" in "X", that the element "K""x" satisfying (*) is unique.
Examples
For example, when "X" is finite and "H" consists of all complex-valued functions on "X", then an element of "H" can be represented as an array of complex numbers. If the usual inner product is used, then "K""x" is the function whose value is 1 at "x" and 0 everywhere else. In this case, "H" is isomorphic to mathbb{C}^n.
A more sophisticated example is the
Hardy space "H"2. This is a space ofholomorphic functions on the unit disc, so here "X" is the unit disc. It can be shown that the reproducing kernel for "H"2 is:K_x(y)=frac{1}{1-overline{x}y}This kernel is known as the Szegő kernel. [cite book |title= Pick Interpolation and Hilbert Function Spaces|last= Agler|first= Jim|coauthors= John E. McCarthy|year= 2002|publisher= American Mathematical Society|isbn= 0-8218-2898-3 ]Properties
The reproducing property
It holds that:langle K(x,cdot), K(y,cdot) angle = K(x,y)
Orthonormal sequences
If extstyle left{ phi_{k} ight} _{k=1}^{infty} is an orthonormal sequence such that the closure of its span is equal to H, then:Kleft( x,y ight) =sum_{k=1}^{infty}phi_{k}left( x ight) phi _{k}left( y ight) .
Moore-Aronszajn theorem
In the previous section, we defined a kernel function in terms of a reproducing kernel Hilbert space. It follows from the definition of an inner product that the kernel we defined is symmetric and positive definite. The Moore-Aronszajn theorem goes in the other direction; it says that every symmetric, positive definite kernel defines a unique reproducing kernel Hilbert space. The theorem first appeared in Aronszajn's "Theory of Reproducing Kernels", although he attributes it to
E. H. Moore .Theorem.Suppose "K" is a symmetric,
positive definite kernel on a set "E". Then there is a unique Hilbert space of functions on "E" for which "K" is a reproducing kernel.Proof.Define, for all "x" in "E", K_x = K(x, cdot).Let "H"0 be the linear span of K_x: x in E } .Define an inner product on "H"0 by:left langle sum_{j=1}^n b_j K_{y_j}, sum_{i=1}^m a_i K_{x_i} ight angle = sum_{i=1}^m sum_{j=1}^n overline{a_i} b_j K(y_j, x_i).The symmetry of this inner product follows from the symmetry of "K" and the non-degeneracy follows from the fact that "K" is positive definite.
Let "H" be the completion of "H"0 with respect to this inner product. Then "H" consists of functions of the form:f(x) = sum_{i=1}^infty a_i K_{x_i} (x)where sum_{i=1}^infty a_i^2 K (x_i, x_i) < infty. The fact that the above sum converges for every "x" follows from the Cauchy-Schwartz inequality.
Now we can check the RKHS property, (*)::langle f, K_x angle = left langle sum_{i=1}^infty a_i K_{x_i}, K_x ight angle= sum_{i=1}^infty a_i K (x_i, x) = f(x).
To prove uniqueness, let "G" be another Hilbert space of functions for which "K" is a reproducing kernel. For any "x" and "y" in "E",(*) implies thatlangle K_x, K_y angle_H = K(x, y) = langle K_x, K_y angle_G.By linearity, langle cdot, cdot angle_H = langle cdot, cdot angle_G on the span of K_x: x in E } . Then "G" = "H" by the uniqueness of the completion.
Bergman kernel
The Bergman kernel is defined for open sets "D" in C"n". Take the Hilbert "H" space of
square-integrable function s, for theLebesgue measure on "D", that areholomorphic function s. The theory is non-trivial in such cases as there are such functions, which are not identically zero. Then "H" is a reproducing kernel space, with kernel function the "Bergman kernel"; this example, with "n" = 1, was introduced by Bergman in 1922.See also
*
Positive definite kernel
*Mercer's theorem
*Kernel trick References
* Nachman Aronszajn, "Theory of Reproducing Kernels", Transactions of the American Mathematical Society, volume 68, number 3, pages 337-404, 1950.
* George Kimeldorf and Grace Wahba, [http://www.stat.wisc.edu/~wahba/ftp1/oldie/kw71.pdf "Some results on Tchebycheffian Spline Functions"] , J. Mathematical Analysis and Applications, 33, 1 (1971) 82-95.
*
Grace Wahba , "Spline Models for Observational Data", [http://www.siam.org/books/ SIAM] , 1990.* Felipe Cucker and Steve Smale, "On the Mathematical Foundations of Learning", Bulletin of the American Mathematical Society, [http://www.ams.org/journals/bull/2002-39-01/home.html volume 39, number 1] , pages 1-49, 2002.
Wikimedia Foundation. 2010.