Charset detection

Character encoding detection, charset detection, or code page detection is the process of heuristically guessing the character encoding of a series of bytes that represent text. This algorithm usually involves statistical analysis of byte patterns. This type of analysis can require frequency distribution of trigraphs of various languages encoded in each code page that will be detected. This process is not foolproof because it depends on statistical data; for example, some versions of the Windows operating system would mis-detect the phrase "Bush hid the facts" in ASCII as Chinese UTF-16LE.

One of the few cases where charset detection works reliably is detecting UTF-8. This is due to the large percentage of invalid byte sequences in UTF-8, so that text in any other encoding that uses bytes with the high bit set is extremely unlikely to pass a UTF-8 validity test. Unfortunately badly written charset detection routines do not run the reliable UTF-8 test first, and may decide that UTF-8 is some other encoding.

Due to the unreliability of charset detection, it is usually better to properly label datasets with the correct encoding. For example, HTML documents can declare their encoding in a meta element, thus:

<meta http-equiv="Content-Type" content="text/html;charset=UTF-8">

Alternatively, when documents are conveyed through HTTP, the same metadata can be conveyed out-of-band using the Content-type header.

See also

• International Components for Unicode - A library that can perform charset detection.

External links

• Frequency distributions of English trigraphs
• IMultiLanguage2::DetectInputCodepage
• API reference for ICU charset detection
• Mozilla Charset Detectors
• Java port of Mozilla Charset Detectors
• Delphi/Pascal port of Mozilla Charset Detectors

This character encoding article is a stub. You can help Wikipedia by expanding it.v · Categories: Character encoding Digital typography stubs Wikimedia Foundation. 2010. Игры ⚽ Нужно решить контрольную? Useless ID/Spyhole Collins (Surrey cricketer) Look at other dictionaries: Character encoding — Special characters redirects here. For the Wikipedia editor s handbook page, see Help:Special characters. A character encoding system consists of a code that pairs each character from a given repertoire with something else, such as a sequence of… …   Wikipedia Windows-1252 — Windows 1252, sometimes called incorrectly ANSI . Blue dots indicate unused or control characters Windows 1252 or CP 1252 is a character encoding of the Latin alphabet, used by default in the legacy components of Microsoft Windows in English and… …   Wikipedia ISO/IEC 646 — This article is about a character encoding standard. For the ISO C header file, see iso646.h. ISO/IEC 646:1991, Information technology ISO 7 bit coded character set for information interchange, is an ISO standard that since its first edition in… …   Wikipedia Code page — is another term for character encoding. It consists of a table of values that describes the character set for a particular language. The term code page originated from IBM s EBCDIC based mainframe systems,[1] but many vendors use this term… …   Wikipedia ISO/IEC 8859-11 — ISO/IEC 8859 11:2001, Information technology 8 bit single byte coded graphic character sets Part 11: Latin/Thai alphabet, is part of the ISO/IEC 8859 series of ASCII based standard character encodings, first edition published in 2001. It is… …   Wikipedia ISO/IEC 8859-6 — ISO/IEC 8859 6:1999, Information technology 8 bit single byte coded graphic character sets Part 6: Latin/Arabic alphabet, is part of the ISO/IEC 8859 series of ASCII based standard character encodings, first edition published in 1987. It is… …   Wikipedia Windows-1258 — is a codepage used in Microsoft Windows to represent Vietnamese texts. It makes use of combining diacritical marks. Windows 1258 is not compatible with VISCII. It is very similar to windows 1252 with the differences being that s caron and z caron …   Wikipedia Windows-1251 — (a.k.a. code page CP1251) is a popular 8 bit character encoding, designed to cover languages that use the Cyrillic alphabet such as Russian, Bulgarian, Serbian Cyrillic and other languages. It is the most widely used for encoding the Bulgarian,… …   Wikipedia Mac OS Roman — is a character encoding primarily used by Mac OS to represent text. It encodes 256 characters, the first 128 of which are identical to ASCII, with the remaining characters including mathematical symbols, diacritics, and additional punctuation… …   Wikipedia Windows-1250 — is a code page used under Microsoft Windows to represent texts in Central European and Eastern European languages that use Latin script, such as Polish, Czech, Slovak, Hungarian, Slovene, Bosnian, Croatian, Serbian (Latin script), Romanian and… …   Wikipedia 18+ © Academic, 2000-2024 Contact us: Technical Support, Advertising Dictionaries export, created on PHP, Joomla, Drupal, WordPress, MODx. Mark and share Search through all dictionaries Translate… Search Internet Share the article and excerpts Direct link … Do a right-click on the link above and select “Copy Link”