# Probabilistic latent semantic analysis

Probabilistic latent semantic analysis

Probabilistic latent semantic analysis (PLSA), also known as probabilistic latent semantic indexing (PLSI, especially in information retrieval circles) is a statistical technique for the analysis of two-mode and co-occurrence data. PLSA evolved from Latent semantic analysis, adding a sounder probabilistic model. PLSA has applications in information retrieval and filtering, natural language processing, machine learning from text, and related areas. It was introduced in 1999 by Jan Puzicha and Thomas Hofmann, [Thomas Hofmann, [http://www.jgaa.info/~th/papers/Hofmann-SIGIR99.pdf "Probabilistic Latent Semantic Indexing"] , Proceedings of the Twenty-Second Annual International SIGIR Conference on Research and Development in Information Retrieval (SIGIR-99), 1999] and it is related to non-negative matrix factorization.

Compared to standard latent semantic analysis which stems from linear algebra and downsizes the occurrence tables (usually via a singular value decomposition), probabilistic latent semantic analysis is based on a mixture decomposition derived from a latent class model. This results in a more principled approach which has a solid foundation in statistics.

Considering observations in the form of co-occurrences $\left(w,d\right)$ of words and documents, PLSA models the probability of each co-occurrence as a mixture of conditionally independent multinomial distributions:

: $P\left(w,d\right) = sum_c P\left(c\right) P\left(d|c\right) P\left(w|c\right) = P\left(d\right) sum_c P\left(c|d\right) P\left(w|c\right)$

The first formulation is the "symmetric" formulation, where $w$ and $d$ are both generated from the latent class $c$ in similar ways (using the conditional probabilities $P\left(d|c\right)$ and $P\left(w|c\right)$), whereas the second formulation is the "asymmetric" formulation, where, for each document $d$, a latent class is chosen conditionally to the document according to $P\left(c|d\right)$, and a word is then generated from that class according to $P\left(w|c\right)$. Although we have used words and documents in this example, the co-occurrence of any couple of discrete variables may be modelled in exactly the same way.

It is reported that the aspect model used in the probabilistic latent semantic analysis has severe overfitting problems [ cite journal|title=Latent Dirichlet Allocation|journal=Journal of Machine Learning Research|date=2003|first=David M.|last=Blei|coauthors=Andrew Y. Ng, Michael I. Jordan|volume=3|pages=993–1022|id= |url=http://jmlr.csail.mit.edu/papers/volume3/blei03a/blei03a.pdf|doi=10.1162/jmlr.2003.3.4-5.993] . The number of parameters grows linearly with the number of documents. In addition, although PLSA is a generative model of the documents in the collection it is estimated on, it is not a generative model of new documents.

PLSA may be used in a discriminative setting, via Fisher kernels. [Thomas Hofmann, [http://www.cs.brown.edu/people/th/papers/Hofmann-NIPS99.ps "Learning the Similarity of Documents : an information-geometric approach to document retrieval and categorization"] , Advances in Neural Information Processing Systems 12, pp-914-920, MIT Press, 2000]

Evolutions of PLSA

* Hierarchical extensions:
** Asymmetric: MASHA ("Multinomial ASymmetric Hierarchical Analysis") [Alexei Vinokourov and Mark Girolami, [http://citeseer.ist.psu.edu/rd/30973750,455249,1,0.25,Download/http://citeseer.ist.psu.edu/cache/papers/cs/22961/http:zSzzSzcis.paisley.ac.ukzSzvino-ci0zSzvinokourov_masha.pdf/vinokourov02probabilistic.pdf A Probabilistic Framework for the Hierarchic Organisation and Classification of Document Collections] , in "Information Processing and Management", 2002 ]
** Symmetric: HPLSA ("Hierarchical Probabilistic Latent Semantic Analysis") [Eric Gaussier, Cyril Goutte, Kris Popat and Francine Chen, [http://www.xrce.xerox.com/Publications/Attachments/2002-004/gaussier02hierarchical.ps.gz A Hierarchical Model for Clustering and Categorising Documents] , in "Advances in Information Retrieval -- Proceedings of the 24th BCS-IRSG European Colloquium on IR Research (ECIR-02)", 2002 ]

* Generative models: The following models have been developed to address an often-criticized shortcoming of PLSA, namely that it is not a proper generative model for new documents.
** Latent Dirichlet allocation - adds a Dirichlet prior on the per-document topic distribution

* Higher-order data: Although this is rarely discussed in the scientific literature, PLSA extends naturally to higher order data (three modes and higher), ie it can model co-occurrences over three or more variables. In the symmetric formulation above, this is done simply by adding conditional probability distributions for these additional variables. This is the probabilistic analogue to non-negative tensor factorisation.

References and notes

* Compound term processing
* Latent Dirichlet allocation
* Latent semantic analysis
* Vector space model

* [http://www.cs.brown.edu/people/th/papers/Hofmann-UAI99.pdf Probabilistic Latent Semantic Analysis]

Wikimedia Foundation. 2010.

### Look at other dictionaries:

• Latent semantic analysis — (LSA) is a technique in natural language processing, in particular in vectorial semantics, of analyzing relationships between a set of documents and the terms they contain by producing a set of concepts related to the documents and terms. LSA was …   Wikipedia

• Latent Semantic Analysis — Analyse sémantique latente L’analyse sémantique latente (LSA, de l anglais : Latent semantic analysis) ou indexation sémantique latente (ou LSI, de l anglais : Latent semantic indexation) est un procédé de traitement des langues… …   Wikipédia en Français

• Latent class model — In statistics, a latent class model (LCM) relates a set of observed discrete multivariate variables to a set of latent variables. It is a type of latent variable model. It is called a latent class model because the latent variable is discrete. A… …   Wikipedia

• Semantic memory — refers to the memory of meanings, understandings, and other concept based knowledge unrelated to specific experiences. The conscious recollection of factual information and general knowledge about the world,cite web… …   Wikipedia

• Analyse Sémantique Latente Probabiliste — L’analyse sémantique latente probabiliste ou PLSA (de l anglais : Probabilistic latent semantic analysis) aussi appelée indexation sémantique latente probabiliste ou PLSI, est une méthode de traitement automatique des langues inspirée de l… …   Wikipédia en Français

• Analyse semantique latente probabiliste — Analyse sémantique latente probabiliste L’analyse sémantique latente probabiliste ou PLSA (de l anglais : Probabilistic latent semantic analysis) aussi appelée indexation sémantique latente probabiliste ou PLSI, est une méthode de traitement …   Wikipédia en Français

• Analyse sémantique latente probabiliste — L’analyse sémantique latente probabiliste (de l anglais, Probabilistic latent semantic analysis : PLSA), aussi appelée indexation sémantique latente probabiliste (PLSI), est une méthode de traitement automatique des langues inspirée de l… …   Wikipédia en Français

• PLSA — Analyse sémantique latente probabiliste L’analyse sémantique latente probabiliste ou PLSA (de l anglais : Probabilistic latent semantic analysis) aussi appelée indexation sémantique latente probabiliste ou PLSI, est une méthode de traitement …   Wikipédia en Français

• Analyse Sémantique Latente — L’analyse sémantique latente (LSA, de l anglais : Latent semantic analysis) ou indexation sémantique latente (ou LSI, de l anglais : Latent semantic indexation) est un procédé de traitement des langues naturelles, dans le cadre de la… …   Wikipédia en Français

• Analyse semantique latente — Analyse sémantique latente L’analyse sémantique latente (LSA, de l anglais : Latent semantic analysis) ou indexation sémantique latente (ou LSI, de l anglais : Latent semantic indexation) est un procédé de traitement des langues… …   Wikipédia en Français