- Infomax
Infomax is an optimization principle for
neural networks and other information processing systems. It prescribes that a function that maps a set of input values I to a set of output values O should be chosen or learned so as to maximize the average Shannonmutual information between I and O, subject to a set of specified constraints and/or noise processes. Infomax algorithms are learningalgorithms that perform this optimization process. The principle was described by Linsker in 1987.In the zero-noise limit, infomax is related to the principle of redundancy reduction proposed for biological sensory processing by
Horace Barlow in 1961, and applied quantitatively to retinal processing by Atick and Redlich.One of the applications of infomax has been to an
Independent component analysis algorithm that finds independent signals by maximisingentropy . Infomax-based ICA was described by Bell and Sejnowski in 1995.References
* Atick, J. J. and Redlich, A. N. (1992). What does the retina know about natural scenes? Neural Computation 4:196-210, 1992.
* Barlow, H. (1961). Possible principles underlying the transformations of sensory messages. In: Sensory Communication, W. Rosenblith (ed.), pp. 217-234. MIT Press, Cambridge, MA.
* [http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?CMD=search&DB=pubmed Bell, A. J. and Sejnowski, T. J. (1995). An information-maximization approach to blind separation and blind deconvolution. Neural Computation 7(6):1129-59.]
* [http://www.ncbi.nlm.nih.gov/entrez/utils/fref.fcgi?itool=Citation-def&PrId=3048&uid=9425547&db=pubmed&url=http://linkinghub.elsevier.com/retrieve/pii/S0042-6989(97)00121-1 Bell, A. J. and Sejnowski, T. J. (1997). The "independent components" of natural scenes are edge filters. Vision Res. 37(23):3327-38.]
*Linsker, R. (1988). [http://doi.ieeecomputersociety.org/10.1109/2.36 Self-organization in a perceptual network] . IEEE Computer 21(3):105-17.
*Linsker, R. (1997). [http://neco.mitpress.org/cgi/content/abstract/9/8/1661?maxtoshow=&HITS=10&hits=10&RESULTFORMAT=&author1=linsker&searchid=1&FIRSTINDEX=0&sortspec=relevance&resourcetype=HWCIT A local learning rule that enables information maximization for arbitrary input distributions] . Neural Computation 9:1661-65.
*Stone, J. V. (2004). Independent Component Analysis: A tutorial introduction. Cambridge, MA., MIT Press.
Wikimedia Foundation. 2010.
