Unstructured data

Unstructured data (or unstructured information) refers to (usually) computerized information that either does not have a data structure or has one that is not easily usable by a a computer program. The term distinguishes such information from data stored in fielded form in databases or annotated (semantically tagged) in documents.

The term is imprecise: software that creates machine-processable structure exploits the linguistic, auditory, and visual structure that is inherent in all forms of human communication.ref|IntelligentEnterprise This inherent structure can be inferred from text, for instance, by examining word morphology, sentence syntax, and other small- and large-scale patterns. Examples of "unstructured data" may include audio, video, and unstructured text such as the body of an e-mail message, Web page, or word processor document.

Merrill Lynch in 1998 cited estimates that as much as 80% of all potentially usable business information originates in unstructured form.ref|ML Such estimates may not be based on primary research, but they are nonetheless widely accepted.ref|Clarabridge

Data with some form of structure may be characterized as unstructured if its structure is not helpful for the desired processing task. For example, an HTML Web page is tagged, but HTML mark-up is typically designed solely for rendering. It does not capture the meaning or function of tagged elements in ways that support automated processing of the information content of the page. XHTML tagging does allow machine processing of elements although it typically does not capture or convey the semantic meaning of tagged terms.

A lot of the unstructured data is noisy text. Spontaneous communication (such as e-mail, SMS, blogs, and web pages) contains noisy text and processing noise for example from automatic speech recognition produce noisy text. Noise in text is defined as any kind of difference between the surface form of a coded representation of the text and the intended, correct, or original text.

Dealing with unstructured data

Data mining and text analytics and noisy text analytics techniques are different methods used to find patterns in, or otherwise interpret, this information. Common techniques for structuring text usually involve manual tagging with metadata or Part-of-speech tagging for further text mining-based structuring. UIMA provides a common framework for processing this information to extract meaning and create structured data about the information.

Notes

# [http://www.intelligententerprise.com/showArticle.jhtml?articleID=59301538 Structure, Models and Meaning: Is "unstructured" data merely unmodeled?] , Intelligent Enterprise, March 1, 2005.
# [http://emarkets.grm.hia.no/gem/Topic7/eip_ind.pdf Christopher C. Shilakes and Julie Tylman, "Enterprise Information Portals"] , Merrill Lynch, 16 November, 1998.
# [http://clarabridge.com/default.aspx?tabid=137&ModuleID=635&ArticleID=551 Unstructured Data and the 80 Percent Rule] , Clarabridge Bridgepoints, 2008 Q3.

ee also

*UIMA
*Data mining
*Metadata
*Noisy text

External links

* [http://blogs.ittoolbox.com/database/soup/archives/005588.asp Database soup: Unstructured data as an oxymoron]
* [http://www.dmreview.com/article_sub.cfm?articleId=1009161 Two Worlds of Data – Unstructured and Structured]