Reproducing kernel Hilbert space

In functional analysis (a branch of mathematics), a reproducing kernel Hilbert space is a Hilbert space of functions in which pointwise evaluation is a continuous linear functional. Equivalently, they are spaces that can be defined by reproducing kernels. The subject was originally and simultaneously developed by Nachman Aronszajn (1907-1980) and Stefan Bergman (1895-1987) in 1950.

In this article we assume that Hilbert spaces are complex. The main reason for this is that many of the examples of reproducing kernel Hilbert spaces are spaces of analytic functions, although some real Hilbert spaces also have reproducing kernels.

An important subset of the reproducing kernel Hilbert spaces are the reproducing kernel Hilbert spaces associated to a continuous kernel. These spaces have applications in machine learning.

Definition

Let "X" be an arbitrary set and "H" a Hilbert space of complex-valued functions on "X". We say that "H" is a reproducing kernel Hilbert space if the linear map

:$f\; mapsto\; f(x)$

from "H" to the complex numbers is continuous for any "x" in "X". By the Riesz representation theorem, this implies that for every "x" in "X" there exists an element "K"_"x" of "H" with the property that:

:$f(x)\; =\; langle\; f,\; K\_x\; angle\; quad\; forall\; f\; in\; H\; quad\; (*)$

The function "K"_"x" is called the point-evaluation functional at the point "x".

Since "H" is a space of functions, the element "K"_"x" is itself a function and can therefore be evaluated at every point. We define the function $K:\; X\; imes\; X\; o\; mathbb\{C\}$ by

:$K(x,y)\; stackrel\{mathrm\{def\{=\}\; K\_x(y).$

This function is called the reproducing kernel for the Hilbert space "H" and it is determined entirely by "H" because the Riesz representation theorem guarantees, for every "x" in "X", that the element "K"_"x" satisfying (*) is unique.

Examples

For example, when "X" is finite and "H" consists of all complex-valued functions on "X", then an element of "H" can be represented as an array of complex numbers. If the usual inner product is used, then "K"_"x" is the function whose value is 1 at "x" and 0 everywhere else. In this case, "H" is isomorphic to $mathbb\{C\}^n$.

A more sophisticated example is the Hardy space "H"². This is a space of holomorphic functions on the unit disc, so here "X" is the unit disc. It can be shown that the reproducing kernel for "H"² is:$K\_x(y)=frac\{1\}\{1-overline\{x\}y\}$This kernel is known as the Szegő kernel. [cite book |title= Pick Interpolation and Hilbert Function Spaces|last= Agler|first= Jim|coauthors= John E. McCarthy|year= 2002|publisher= American Mathematical Society|isbn= 0-8218-2898-3 ]

Properties

The reproducing property

It holds that:$langle\; K(x,cdot),\; K(y,cdot)\; angle\; =\; K(x,y)$

Orthonormal sequences

If $extstyle\; left\{\; phi\_\{k\}\; ight\}\; \_\{k=1\}^\{infty\}$ is an orthonormal sequence such that the closure of its span is equal to $H$, then:$Kleft(\; x,y\; ight)\; =sum\_\{k=1\}^\{infty\}phi\_\{k\}left(\; x\; ight)\; phi\; \_\{k\}left(\; y\; ight)$.

Moore-Aronszajn theorem

In the previous section, we defined a kernel function in terms of a reproducing kernel Hilbert space. It follows from the definition of an inner product that the kernel we defined is symmetric and positive definite. The Moore-Aronszajn theorem goes in the other direction; it says that every symmetric, positive definite kernel defines a unique reproducing kernel Hilbert space. The theorem first appeared in Aronszajn's "Theory of Reproducing Kernels", although he attributes it to E. H. Moore.

Theorem.Suppose "K" is a symmetric, positive definite kernel on a set "E". Then there is a unique Hilbert space of functions on "E" for which "K" is a reproducing kernel.

Proof.Define, for all "x" in "E", $K\_x\; =\; K(x,\; cdot)$.Let "H"₀ be the linear span of $\{K\_x:\; x\; in\; E\; \}$.Define an inner product on "H"₀ by:$left\; langle\; sum\_\{j=1\}^n\; b\_j\; K\_\{y\_j\},\; sum\_\{i=1\}^m\; a\_i\; K\_\{x\_i\}\; ight\; angle\; =\; sum\_\{i=1\}^m\; sum\_\{j=1\}^n\; overline\{a\_i\}\; b\_j\; K(y\_j,\; x\_i).$The symmetry of this inner product follows from the symmetry of "K" and the non-degeneracy follows from the fact that "K" is positive definite.

Let "H" be the completion of "H"₀ with respect to this inner product. Then "H" consists of functions of the form:$f(x)\; =\; sum\_\{i=1\}^infty\; a\_i\; K\_\{x\_i\}\; (x)$where . The fact that the above sum converges for every "x" follows from the Cauchy-Schwartz inequality.

Now we can check the RKHS property, (*)::$langle\; f,\; K\_x\; angle\; =\; left\; langle\; sum\_\{i=1\}^infty\; a\_i\; K\_\{x\_i\},\; K\_x\; ight\; angle=\; sum\_\{i=1\}^infty\; a\_i\; K\; (x\_i,\; x)\; =\; f(x).$

To prove uniqueness, let "G" be another Hilbert space of functions for which "K" is a reproducing kernel. For any "x" and "y" in "E",(*) implies that$langle\; K\_x,\; K\_y\; angle\_H\; =\; K(x,\; y)\; =\; langle\; K\_x,\; K\_y\; angle\_G$.By linearity, $langle\; cdot,\; cdot\; angle\_H\; =\; langle\; cdot,\; cdot\; angle\_G$ on the span of $\{K\_x:\; x\; in\; E\; \}$. Then "G" = "H" by the uniqueness of the completion.

Bergman kernel

The Bergman kernel is defined for open sets "D" in C^"n". Take the Hilbert "H" space of square-integrable functions, for the Lebesgue measure on "D", that are holomorphic functions. The theory is non-trivial in such cases as there are such functions, which are not identically zero. Then "H" is a reproducing kernel space, with kernel function the "Bergman kernel"; this example, with "n" = 1, was introduced by Bergman in 1922.

See also

*Positive definite kernel
*Mercer's theorem
*Kernel trick

References

* Nachman Aronszajn, "Theory of Reproducing Kernels", Transactions of the American Mathematical Society, volume 68, number 3, pages 337-404, 1950.

* George Kimeldorf and Grace Wahba, [http://www.stat.wisc.edu/~wahba/ftp1/oldie/kw71.pdf "Some results on Tchebycheffian Spline Functions"] , J. Mathematical Analysis and Applications, 33, 1 (1971) 82-95.

* Grace Wahba, "Spline Models for Observational Data", [http://www.siam.org/books/ SIAM] , 1990.

* Felipe Cucker and Steve Smale, "On the Mathematical Foundations of Learning", Bulletin of the American Mathematical Society, [http://www.ams.org/journals/bull/2002-39-01/home.html volume 39, number 1] , pages 1-49, 2002.