UTF-8
32188
225927751
2008-07-16T01:38:40Z
Yurivict
150173
/* External links */ improved link to Online Tool
'''UTF-8''' (8-[[bit]] [[Universal Character Set|UCS]]/[[Unicode Transformation Format]]) is a [[Variable-width encoding|variable-length]] [[character encoding]] for [[Unicode]]. It is able to represent any character in the Unicode standard, yet the initial encoding of byte codes and character assignments for UTF-8 is [[backwards compatible]] with [[ASCII]]. For these reasons, it is steadily becoming the preferred encoding for [[e-mail]], [[web pages]]<ref name="GoogleUnicode">{{cite web | url=http://googleblog.blogspot.com/2008/05/moving-to-unicode-51.html | title=Moving to Unicode 5.1 | date=[[May 5]] [[2008]] | publisher=Official Google Blog| accessdate=2008-05-08}}</ref>, and other places where characters are [[Computer data storage|stored]] or [[Stream (computing)|streamed]].
UTF-8 encodes each character in one to four [[octet (computing)|octet]]s (8-bit [[byte]]s):
#One byte is needed to encode the 128 US-ASCII characters (Unicode range {{U+|0000}} to U+007F).
#Two bytes are needed for [[Latin alphabet|Latin]] letters with [[diacritic]]s and for characters from [[Greek alphabet|Greek]], [[Cyrillic alphabet|Cyrillic]], [[Armenian alphabet|Armenian]], [[Hebrew alphabet|Hebrew]], [[Arabic alphabet|Arabic]], [[Syriac alphabet|Syriac]] and [[Thaana]] alphabets (Unicode range U+0080 to U+07FF).
#Three bytes are needed for the rest of the [[Basic Multilingual Plane]] (which contains virtually all characters in common use).
#Four bytes are needed for characters in the [[Mapping of Unicode characters|other planes of Unicode]], which are rarely used in practice.
Four bytes may seem like a lot for one character ([[Universal Character Set|code point]]). However, code points outside the Basic Multilingual Plane are generally very rare. Furthermore, [[UTF-16]] (the main alternative to UTF-8) also needs four bytes for these code points. Whether UTF-8 or UTF-16 is more efficient depends on the range of code points being used. However, the differences between different encoding schemes can become negligible with the use of traditional compression systems like [[DEFLATE]]. For short items of text where traditional algorithms do not perform well and size is important, the [[Standard Compression Scheme for Unicode]] could be considered instead.
The [[Internet Engineering Task Force]] (IETF) requires all [[Internet]] [[protocol (computing)|protocol]]s to identify the [[character encoding|encoding]] used for character data, and the supported character encodings must include UTF-8.<ref name="rfc2277">{{citation | first=H. | last=Alvestrand | title=RFC 2277 | contribution=IETF Policy on Character Sets and Languages | publisher=[[Internet Engineering Task Force]] | year=1998}}</ref> The [[Internet Mail Consortium]] (IMC) recommends that all email programs be able to display and create mail using UTF-8.<ref name="IMC">{{cite web | url=http://www.imc.org/mail-i18n.html | title=Using International Characters in Internet Mail | date=[[August 1]] [[1998]] | publisher=Internet Mail Consortium | accessdate=2007-11-08}}</ref>
{{Table Unicode}}
==History==
By early 1992 a search was on for a good byte-stream encoding of multi-byte character sets. The draft [[ISO 10646]] standard contained a non-required annex called UTF that provided a byte-stream encoding of its 32-bit characters. This encoding was not satisfactory on performance grounds, but did introduce the notion that bytes in the ASCII range of 0–127 represent themselves in UTF, thereby providing backward compatibility.
In July 1992 the [[X/Open]] committee XoJIG was looking for a better encoding. Dave Prosser of [[Unix System Laboratories]] submitted a proposal for one that had faster implementation characteristics and introduced the improvement that 7-bit ASCII characters would ''only'' represent themselves; all multibyte sequences would include only 8-bit characters, ''i.e.'' those where the high bit was set.
In August 1992 this proposal was circulated by an [[IBM]] X/Open representative to interested parties. [[Ken Thompson (computer programmer)|Ken Thompson]] of the [[Plan 9 from Bell Labs|Plan 9]] [[operating system]] group at [[Bell Laboratories]] then made a crucial modification to the encoding, to allow it to be self-synchronizing, meaning that it was not necessary to read from the beginning of the string in order to find character boundaries. Thompson's design was outlined on [[September 2]], [[1992]], on a [[placemat]] in a [[New Jersey]] [[diner]] with [[Rob Pike]]. The following days, Pike and Thompson implemented it and updated [[Plan 9 from Bell Labs|Plan 9]] to use it throughout, and then communicated their success back to X/Open.<ref>{{cite web
| url = http://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt
| first = Rob
| last = Pipe
| title = UTF-8 history
| date = 2003-04-03
}}</ref>
UTF-8 was first officially presented at the [[USENIX]] conference in [[San Diego]], from [[January 25]]–[[January 29|29]] [[1993]].
==Description==
There are several current definitions of UTF-8 in various standards documents:
* RFC 3629 / STD 63 (2003), which establishes UTF-8 as a standard Internet protocol element
* ''The Unicode Standard, Version 5.0'', §3.9 D92, $3.10 D95 (2007)
* ''The Unicode Standard, Version 4.0'', §3.9–§3.10 (2003)
* ISO/IEC 10646:2003 Annex D (2003)
They supersede the definitions given in the following obsolete works:
* ISO/IEC 10646-1:1993 Amendment 2 / Annex R (1996)
* ''The Unicode Standard, Version 2.0'', Appendix A (1996)
* RFC 2044 (1996)
* RFC 2279 (1998)
* ''The Unicode Standard, Version 3.0'', §2.3 (2000) plus Corrigendum #1: UTF-8 Shortest Form (2000)
* ''Unicode Standard Annex #27: Unicode 3.1'' (2001)
They are all the same in their general mechanics, with the main differences being on issues such as allowed range of code point values and safe handling of invalid input.
The bits of a Unicode character are divided into several groups which are then divided among the lower bit positions inside the UTF-8 bytes. A character whose code point is below U+<code>0080</code> is encoded with a single byte that contains its code point: these correspond exactly to the 128 characters of 7-[[bit]] [[ASCII]]. In other cases, up to four bytes are required. The [[most significant bit]] of these bytes is 1, to prevent confusion with 7-bit ASCII characters and therefore keep standard byte-oriented string processing safe.
The original specification allowed for sequences of up to six bytes covering numbers up to 31 bits (the original limit of the [[universal character set]]). However, UTF-8 was restricted by RFC 3629 to use only the area covered by the formal Unicode definition, U+<code>0000</code> to U+<code>10FFFF</code>, in November 2003. With these restrictions, the following byte values never appear in a legal UTF-8 sequence:
{| class="wikitable"
! Codes ([[binary numeral system|binary]])
! Codes ([[hexadecimal]])
! Notes
|-
| <code>1100000x</code>
| <code>C0</code>, <code>C1</code>
| Overlong encoding: lead byte of a 2-byte sequence, but code point <= <code>127</code>
|-
| <code>11110101</code><br><code>1111011x</code>
| <code>F5</code>, <code>F6</code>, <code>F7</code>
| Restricted by RFC 3629: lead byte of 4-byte sequence for codepoint above <code>10FFFF</code>
|-
| <code>111110xx</code><br><code>1111110x</code>
| <code>F8</code>, <code>F9</code>, <code>FA</code>, <code>FB</code>, <code>FC</code>, <code>FD</code>
| Restricted by RFC 3629: lead byte of a sequence 5 or 6 bytes long
|-
| <code>1111111x</code>
| <code>FE</code>, <code>FF</code>
| Invalid: lead byte of a sequence 7 or 8 bytes long
|}
While the two categories labeled "Restricted by RFC" above were technically allowed by earlier UTF-8 specifications, no characters were ever assigned to the code points they represent, so they should never have appeared in UTF-8-encoded text.
Given these restrictions, the following sequences are allowed, ordered width by first byte:
{| class="wikitable"
! [[binary numeral system|Binary]]
! [[Hexadecimal]]
! [[Decimal]]
! [[Width]]
|-
| <code>00000000</code>-<code>01111111</code>
| align="center"| <code>00</code>-<code>7F</code>
| align="right"| <code>0</code>-<code>127</code>
| 1 byte
|-
| <code>11000010</code>-<code>11011111</code>
| align="center"| <code>C2</code>-<code>DF</code>
| align="right"| <code>194</code>-<code>223</code>
| 2 bytes
|-
| <code>11100000</code>-<code>11101111</code>
| align="center"| <code>E0</code>-<code>EF</code>
| align="right"| <code>224</code>-<code>239</code>
| 3 bytes
|-
| <code>11110000</code>-<code>11110100</code>
| align="center"| <code>F0</code>-<code>F4</code>
| align="right"| <code>240</code>-<code>244</code>
| 4 bytes
|}
So the first 128 characters (US-ASCII) need one byte. The next 1920 characters need two bytes to encode. This includes Latin alphabet characters with [[diacritic]]s, [[Greek alphabet|Greek]], [[Cyrillic alphabet|Cyrillic]], [[Coptic alphabet|Coptic]], [[Armenian Alphabet|Armenian]], [[Hebrew alphabet|Hebrew]], and [[Arabic alphabet|Arabic]] characters. The rest of the [[Basic Multilingual Plane|BMP]] characters use three bytes, and additional characters are encoded in four bytes. By continuing the pattern given above it is possible to deal with much larger numbers.
Accordingly, the UTF-8 specification allows for the following allocation of code points.
{| class="wikitable"
! style="width: auto;" | Code range<br>[[hexadecimal]]
! style="width: auto;" | Scalar value<br>[[binary numeral system|binary]]
! style="width: auto;" | UTF-8<br>[[binary numeral system|binary]] / [[hexadecimal]]
! style="width: 25%;" | Notes
|-
| rowspan="2" | <code>000000</code>–<code>00007F</code> <br><small> 128 code points </small>
| <code>00000000</code> <code>00000000</code> <code>0zzzzzzz</code>
| <code>0zzzzzzz</code>
| rowspan="2" | ASCII equivalence range; byte begins with zero
|-
| <small>seven <code>z</code></small>
| <small>seven <code>z</code>; byte value <code>00</code>–<code>7F</code></small>
|-
| rowspan="2" | <code>000080</code>–<code>0007FF</code> <br><small> 2048 total code points (1920 allowed, after removing overlong encoded points) </small>
| <code>00000000</code> <code>00000yyy</code> <code>yyzzzzzz</code>
| style="text-align: left;" | <code>110yyyyy</code> <code>10zzzzzz</code>
| rowspan="2" align="top" | first byte begins with <code>110</code>, the following byte begins with <code>10</code>.
|-
| <small>three <code>y</code>; two <code>y</code>, six <code>z</code></small>
| <small>five <code>y</code>, six <code>z</code>; byte values <code>C2</code>–<code>DF</code> <!-- C2 is correct here because C0 and C1 cannot occur --> and <code>80</code>–<code>BF</code></small>
|-
| rowspan="2" | <code>000800</code>–<code>00D7FF </code> <br> <code>00E000</code>–<code>00FFFF</code> <br> <small> 65536 total code points (61440 allowed, after removing 2048 overlong encoded points and 2048 surrogates for UTF-16) </small> {{ref label|D800|Note 1|Note 1}}
| <code>00000000</code> <code>xxxxyyyy</code> <code>yyzzzzzz</code>
| style="text-align: left;" | <code>1110xxxx</code> <code>10yyyyyy</code> <code>10zzzzzz</code>
| rowspan="2" align="top" | first byte begins with <code>1110</code>, the following 2 bytes begin with <code>10</code>.
|-
| <small>four <code>x</code>, four <code>y</code>; two <code>y</code>, six <code>z</code></small>
| <small>four <code>x</code>, six <code>y</code>, six <code>z</code>; byte values <code>E0</code>–<code>EF</code> and 2x <code>80</code>–<code>BF</code></small>
|-
| rowspan="2" | <code>010000</code>–<code>10FFFF</code> <br><small> 2097152 total code points (1048576 allowed, after removing RFC 3629 restricted and invalid lead byte sequences, and 65536 overlong code points) </small>
| <code>000wwwxx</code> <code>xxxxyyyy</code> <code>yyzzzzzz</code>
| style="text-align: left;" | <code>11110www</code> <code>10xxxxxx</code> <code>10yyyyyy</code> <code>10zzzzzz</code>
| rowspan="2" | First byte begins with <code>11110</code>, the following 3 bytes begin with <code>10</code>
|-
| <small>three <code>w</code>, two <code>x</code>; four <code>x</code>, four <code>y</code>; two <code>y</code>, six <code>z</code></small>
| <small>three <code>w</code>; six <code>x</code>; six <code>y</code>; six <code>z</code>; byte values <code>F0</code>–<code>F4</code><!-- Please note that F4 is correct! --> and 3x <code>80</code>–<code>BF</code></small>
|}
:{{note label|D800|Note 1|Note 1}} The range D800-DFFF (2048 code points) is disallowed by Unicode. The encoding scheme reliably transforms values in that range, but they are not valid scalar values in Unicode. See Table 3-7 in the Unicode 5.0 standard.
For example, the character aleph (א), which is Unicode U+<code>05D0</code>, is encoded into UTF-8 in this way:
* It falls into the range of U+<code>0080</code> to U+<code>07FF</code>. The table shows it will be encoded using two bytes, <code>''110''yyyyy</code> <code>''10''zzzzzz</code>.
* [[Hexadecimal]] 0x<code>05D0</code> is equivalent to [[binary numeral system|binary]] <code>101</code>-<code>1101</code>-<code>0000</code>.
* The eleven bits are put in their order into the positions marked by "<code>y</code>"-s and "<code>z</code>"-s: <code>110'''10111'''</code> <code>10'''010000'''</code>.
* The final result is the two bytes, more conveniently expressed as the two hexadecimal bytes 0x<code>D7</code> 0x<code>90</code>. That is the encoding of the character aleph (א) in UTF-8.
Another example: when the number of bits to be filled is less than the maximum number of free bits available, the high bits are padded with <code>0</code>'s.
For example, the Cent Sign (¢), which is Unicode U+<code>00A2</code>, is encoded into UTF-8 in this way.
* It falls into the range of U+<code>0080</code> to U+<code>07FF</code>. The table shows it will be encoded using two bytes, <code>''110''yyyyy</code> <code>''10''zzzzzz</code>.
* Hexadecimal 0x<code>00A2</code> is equivalent to binary <code>1010</code>-<code>0010</code>.
* The eight bits are put in their order into the positions marked by "<code>y</code>"-s and "<code>z</code>"-s: <code>110000'''10'''</code> <code>10'''100010'''</code>.
* The final result is the two bytes, expressed as the two hexadecimal bytes 0x<code>C2</code> 0x<code>A2</code>. That is the encoding of the character Cent Sign (¢) in UTF-8.
==UTF-8 derivations==
===Windows===
Many [[Microsoft Windows|Windows]] programs (including Windows [[Notepad]]) use the byte sequence <code>EF BB BF</code> at the beginning of a file to indicate that the file is encoded using UTF-8. This is the [[Byte Order Mark]] U+FEFF encoded in UTF-8, which appears as the [[ISO-8859-1]] characters "" in most [[text editor]]s and [[web browser]]s not prepared to handle UTF-8.
===Java===
In normal usage, the [[Java (programming language)|Java programming language]] supports standard UTF-8 when reading and writing strings through {{Javadoc:SE|java/io|InputStreamReader}} and {{Javadoc:SE|java/io|OutputStreamWriter}}.
However, Java also supports a non-standard variant of UTF-8 called {{Javadoc:SE|name=modified UTF-8|java/io|DataInput|modified-utf-8}} for object [[serialization#Java|serialization]], for the [[Java Native Interface]], and for embedding constants in [[Class (file format)|class files]]. There are two differences between modified and standard UTF-8.
The first difference is that the [[null character]] (U+0000) is encoded as 0xc0 0x80 rather than 0x00. (0xc0 0x80 is not legal standard UTF-8 because it is not the shortest possible representation.) This guarantees that if an extra null terminator byte 0x00 is placed at the end of the string, it will be the only 0x00 encountered if a string containing embedded null characters is processed in a language such as [[C (programming language)|C]] using traditional [[ASCIIZ]] string functions. In standard UTF-8 the embedded nulls would be encoded as 0x00, signalling the end of the string and causing premature truncation.
The second difference is in the way characters outside the [[Mapping of Unicode characters#Basic Multilingual Plane|Basic Multilingual Plane]] are encoded. In standard UTF-8 these characters are encoded using the four-byte format above. In modified UTF-8 these characters are first represented as surrogate pairs (as in UTF-16), and then the surrogate pairs are encoded individually in sequence as in [[CESU-8]], taking up 6 bytes in total. Each Java character represents a 16-bit value. This aspect of the language predates the supplementary planes of Unicode; however, it is important for performance as well as backwards compatibility, and is unlikely to change.
Because modified UTF-8 is '''not''' UTF-8, one needs to be very careful to '''avoid mislabelling''' data in modified UTF-8 as UTF-8 when interchanging information over the [[Internet]].
===Tcl===
[[Tcl]] uses the same modified UTF-8 as Java for internal representation of Unicode data.
===Mac OS X===
The [[Mac OS X]] Operating System uses canonically decomposed Unicode, encoded using UTF-8 for file names in the [[filesystem]]. This is sometimes referred to as UTF-8-MAC. In canonically decomposed Unicode, the use of precomposed characters is forbidden and [[Combining character|combining diacritics]] must be used to replace them.
A common argument {{Fact|date=May 2008}} is that this makes sorting far simpler, but this argument is easily refuted {{Fact|date=May 2008}}: for one, sorting is [[Collating sequence|language dependent]] (in German, the '''ä''' character sorts just after the '''a''' character, while in Swedish '''ä''' sorts after '''z'''). Therefore, it can be confusing for software built around the assumption that precomposed characters are the norm and combining diacritics are only used to form unusual combinations. This is an example of the [[Unicode_normalization#Standards|NFD]] (Normalization Form Canonical Decomposition) variant of [[Unicode normalization]]—most other platforms, including [[Microsoft Windows|Windows]] and [[Linux]], use the [[Unicode_normalization#Standards|NFC]] (Normalization Form Canonical Composition) form of Unicode normalization, which is also used by [[W3C]] standards, so [[Unicode_normalization#Standards|NFD]] data must typically be converted to [[Unicode_normalization#Standards|NFC]] for use on other platforms or the Web.
This is discussed in Apple Q&A 1173.<ref name="apple1173>{{cite web | url=http://developer.apple.com/qa/qa2001/qa1173.html | title=Text Encodings in VFS | publisher=[[Apple Inc.]] | date=[[February 10]] [[2003]] | accessdate=2007-11-08}}</ref>
===CESU-8===
:''See main article: [[CESU-8]].
[[Oracle Database|Oracle]] databases use CESU-8. Characters outside the BMP are first encoded as surrogate pairs, which are then each encoded as UTF-8. It is the same as modified UTF-8 from Java, but without the special encoding of the NUL character. It is not valid UTF-8.
==Rationale behind UTF-8's design==
As a consequence of the design of UTF-8, the following properties of multi-byte sequences hold:
* The [[most significant bit]] of a single-byte character is always <code>0</code>.
* The most significant bits of the first byte of a multi-byte sequence determine the length of the sequence. These most significant bits are <code>110</code> for two-byte sequences; <code>1110</code> for three-byte sequences, and so on.
* The remaining bytes in a multi-byte sequence have <code>10</code> as their two most significant bits.
* A UTF-8 stream contains neither the byte FE nor FF. This makes sure that a UTF-8 stream never looks like a UTF-16 stream starting with U+FEFF ([[Byte-order mark]])
UTF-8 was designed to satisfy these properties in order to guarantee that no byte sequence of one character is contained within a longer byte sequence of another character. This ensures that byte-wise sub-string matching can be applied to search for words or phrases within a text; some older variable-length 8-bit encodings (such as [[Shift-JIS]]) did not have this property and thus made string-matching algorithms rather complicated. Although this property adds [[redundancy (information theory)|redundancy]] to UTF-8–encoded text, the advantages outweigh this concern; besides, [[data compression]] is not one of Unicode's aims and must be considered independently. This also means that if one or more complete bytes are lost due to error or corruption, one can resynchronize at the beginning of the next character and thus limit the damage.
Also due to the design of the byte sequences, if a sequence of bytes supposed to represent text validates as UTF-8 then it is fairly safe to assume it is UTF-8. The chance of a random sequence of bytes being valid UTF-8 and not pure ASCII is 3.1% for a 2 byte sequence, 0.39% for a 3 byte sequence and even lower for longer sequences.
While natural languages encoded in traditional encodings are not random byte sequences, they are also unlikely to produce byte sequences that would pass a UTF-8 validity test and then be misinterpreted. For example, for [[ISO-8859-1]] text to be misrecognized as UTF-8, the '''only''' non-ASCII characters in it would have to be in sequences starting with either an accented letter or the multiplication symbol and ending with a symbol. Pure ASCII text would pass a UTF-8 validity test and it would be interpreted correctly because the UTF-8 encoding for the same text is the same as the ASCII encoding.
The bit patterns can be used to identify UTF-8 characters. If the byte's first hex code begins with 0–7, it is an ASCII character. If it begins with C or D, it is an 11-bit character (expressed in two bytes). If it begins with E, it is 16-bit (expressed in 3 bytes), and if it begins with F, it is 21 bits (expressed in 4 bytes). 8 through B cannot be first hex codes, but all ''following'' bytes must begin with a hex code between 8 through B. Thus, at a glance, it can be seen that "0xA9" is not a valid UTF-8 character, but that "0x54" and "0xE3 0xB4 0xB1" ''are'' valid UTF-8 characters.
There is no good validity test for traditional 8-bit encodings like [[ISO-8859-1]]. It must be known otherwise which encoding is used, otherwise bad text will be shown. This is called [[mojibake]] and other names. The fact that there is a working validity test for UTF-8-encoded texts is a big advantage.
==Overlong forms, invalid input, and security considerations==
The exact response required of a UTF-8 decoder on invalid input is not uniformly defined by the standards. In general, there are several ways a UTF-8 decoder might behave in the event of an invalid byte sequence:
# Insert a replacement character (usually '?' or '�' (U+FFFD)).
# Ignore the bytes.
# Interpret each byte according to another encoding (often [[ISO/IEC 8859-1|ISO-8859-1]] or [[CP1252]]).
# Not notice and decode as if the bytes were some similar bit of UTF-8.
# Stop decoding and report an error (possibly giving the caller the option to continue).
It is possible for a decoder to behave in different ways for different types of invalid input.
<nowiki>RFC 3629</nowiki> states that "Implementations of the decoding algorithm MUST protect against decoding invalid sequences."<ref name="rfc3629">{{citation | first=F. | last=Yergeau | title=RFC 3629 | contribution=UTF-8, a transformation format of ISO 10646 | publisher=[[Internet Engineering Task Force]] | year=2003}}</ref> ''The Unicode Standard'' requires a Unicode-compliant decoder to "…treat any ill-formed code unit sequence as an error condition. This guarantees that it will neither interpret nor emit an ill-formed code unit sequence."<!--anyone have a copy of ISO/IEC 10646-1:2000 annex D for comparison?-->
Overlong forms are one of the most troublesome types of UTF-8 data. The current RFC says they must not be decoded, but older specifications for UTF-8 only gave a warning, and many simpler decoders will happily decode them. Overlong forms have been used to bypass security validations in high profile products including Microsoft's [[Internet Information Services|IIS]] web server. Therefore, great care must be taken to avoid security issues if validation is performed before conversion from UTF-8, and it is generally much simpler to handle overlong forms before any input validation is done.
Another common problem is decoders that do not check that the trailing bytes are really trailing bytes. This will cause more characters to be lost than necessary if some bytes are lost or corrupted.
To maintain security in the case of invalid input, there are a few options. The first is to decode the UTF-8 ''before'' doing any input validation checks. The second is to use a decoder that, in the event of invalid input either returns an error or text that the application knows to be harmless. A third possibility is to not decode the UTF-8 at all, this is quite practical if the system only treats some ASCII characters (like slash and NUL) specially, and treats all other bytes as identifiers or other data but requires care to avoid passing invalid UTF-8 to other code (such as third party libraries or an operating system) that cannot safely handle it.
== Advantages and disadvantages ==
=== A note on string length and character indexes ===
{{Unreferencedsection|date=April 2008}}
A common criticism from beginners of variable-length encoding such as UTF-8 is that the algorithms to find the number of characters between two points, or the point that is n characters after another point, are not O(1) (constant time), causing programs using them to be slower. However the use of these algorithms by actual working software is often over-estimated:
* In virtually every case, the value being provided to one of these algorithms has been calculated by calling the other algorithm previously. Therefore changing both of them to use the byte count in place of the character count will get the exact same working program with the same O(1) performance.
* The majority of other uses are just wrong, such as assuming that the width a string will take to display, or the location of a cursor, can be calculated faster by having this information (combining characters, double width characters, proportional fonts, non-printing characters and right-to-left characters all contribute to the impossibility of this).
* The remaining uses are data manipulation, such as [[compression|data compression]], [[encryption]], or a [[string search algorithm]]. These can almost always be rewritten to use bytes as data units. The self-synchronizing nature of UTF-8 can be used to find the glyph boundary that is near a pointer, though often this is unnecessary as the algorithm will produce the correct boundaries as a side effect anyway.
So while the number of octets in a UTF-8 string or substring is related in a more complex way to the number of code points than for UTF-32, it is very rare to encounter a situation where this makes a difference in practice, and this cannot be used as either an advantage or disadvantage of UTF-8.
=== General ===
====Advantages====
* UTF-8 is a [[superset]] of ASCII. Since a plain ASCII string is also a valid UTF-8 string, no conversion needs to be done for existing ASCII text. Software designed for traditional non-extended ASCII character sets can generally be used with UTF-8 with few or no changes.
* Sorting of UTF-8 strings using standard byte-oriented sorting routines will produce the same results as sorting them based on Unicode code points. (This has limited usefulness, though, since it is unlikely to represent the culturally acceptable sort order of any particular language or locale.) For the sorting to work correctly, the bytes must be treated as unsigned values.
* UTF-8 and UTF-16 are the standard encodings for [[Extensible Markup Language|XML]] documents. All other encodings must be specified explicitly either externally or through a text declaration. [http://www.w3.org/TR/REC-xml/#charencoding]
* Any [[byte oriented]] [[string search algorithm]] can be used with UTF-8 data (as long as one ensures that the inputs only consist of complete UTF-8 characters). Care must be taken with regular expressions and other constructs that count characters, however.
* UTF-8 strings can be fairly reliably recognized as such by a simple algorithm. That is, the probability that a string of characters in any other encoding appears as valid UTF-8 is low, diminishing with increasing string length. For instance, the octet values C0, C1, and F5 to FF never appear. For better reliability, regular expressions can be used to take into account illegal overlong and surrogate values (see the [http://www.w3.org/International/questions/qa-forms-utf-8 W3 FAQ: Multilingual Forms] for a Perl regular expression to validate a UTF-8 string).
====Disadvantages====
* A badly-written (and not compliant with current versions of the standard) UTF-8 [[parser]] could accept a number of different pseudo-UTF-8 representations and convert them to the same Unicode output. This provides a way for information to leak past validation routines designed to process data in its eight-bit representation.
=== Compared to single-byte encodings ===
====Advantages====
* UTF-8 can encode any [[Unicode]] character, avoiding the need to figure out and set a "[[code page]]" or otherwise indicate what character set is in use, and allowing output in multiple languages at the same time. It needs only to indicate to the operating system that text is in UTF-8. For Windows, this involves setting code page 65001.<ref>[http://msdn2.microsoft.com/en-us/library/ms776446(VS.85).aspx Code Page Identifiers] from MSDN. Accessed on [[January 5]], [[2008]].</ref>
====Disadvantages====
* UTF-8 encoded text is larger than the appropriate single-byte encoding except for plain ASCII characters. In the case of languages which commonly used 8-bit character sets with non-Latin alphabets encoded in the upper half (such as most [[Cyrillic]] and [[Greek]] alphabet code pages), UTF-8 text will be almost double the size of the same text in a single-byte encoding.
* Single byte per character encodings make string cutting easy even with simple-minded APIs.
=== Compared to other multi-byte encodings ===
====Advantages====
* UTF-8 can encode any [[Unicode]] character. In most cases, multi-byte encodings can be converted to Unicode and back with no loss and — as UTF-8 is an encoding of Unicode — this applies to it too.
* Character boundaries are easily found from anywhere in an octet stream (scanning either forwards or backwards). This implies that if a stream of bytes is scanned starting in the middle of a multi-byte sequence, only the information represented by the partial sequence is lost and decoding can begin correctly on the next character. Similarly, if a number of bytes are corrupted or dropped, then correct decoding can resume on the next character boundary. Many multi-byte encodings are much harder to resynchronise.
* A byte sequence for one character never occurs as part of a longer sequence for another character as it did in older variable-length encodings like [[Shift-JIS]] (see the previous section on this). For instance, US-ASCII octet values do not appear otherwise in a UTF-8 encoded character stream. This provides compatibility with file systems or other software (e.g., the [[printf]]() function in C libraries) that parse based on US-ASCII values but are transparent to other values.
* The first byte of a multi-byte sequence is enough to determine the length of the multi-byte sequence. This makes it extremely simple to extract a sub-string from a given string without elaborate parsing. This was often not the case in multi-byte encodings.
* Efficient to encode using simple bit operations. UTF-8 does not require slower mathematical operations such as multiplication or division (unlike the obsolete [[UTF-1]] encoding).
====Disadvantages====
* UTF-8 often takes more space than an encoding made for one or a few languages. Latin letters with diacritics and characters from other alphabetic scripts typically take one byte per character in the appropriate multi-byte encoding but take two in UTF-8. East Asian scripts generally have two bytes per character in their multi-byte encodings yet take three bytes per character in UTF-8.
=== Compared to UTF-7 ===
====Advantages====
* UTF-8 uses significantly fewer bytes per character for all non-ASCII characters.
* UTF-8 encodes "+" as itself whereas [[UTF-7]] encodes it as "+-".
====Disadvantages====
* UTF-8 requires the transmission system to be [[eight-bit clean]]. In the case of e-mail this means it has to be further encoded using [[quoted printable]] or [[base64]] in some cases. This extra stage of encoding carries a significant size penalty. However, this disadvantage is not so important an issue any more because most [[mail transfer agent]]s in modern use are eight-bit clean and support the [[8BITMIME]] SMTP extension as specified in RFC 1869.
=== Compared to UTF-16 ===
====Advantages====
* Unlike UTF-16, byte values of 0 (The ASCII NUL character) do not appear in the encoding unless U+0000 (the Unicode NUL character) is represented. This means that for UTF-8, the [[standard C library]] string functions (such as strcpy()) that use a [[null terminator]] will '''not''' incorrectly truncate strings, as can happen in UTF-16.
* Characters below U+0080 take only one byte in UTF-8 and take two bytes in UTF-16. Text consisting of mostly diacritic-free Latin letters will be around half the size in UTF-8 than it would be in [[UTF-16]]. Text in many other alphabets will be slightly smaller in UTF-8 than it would be in UTF-16 because of the presence of spaces, newlines, numbers, and punctuation.
* Most existing [[computer program]]s (including [[operating system]]s) were not written with Unicode in mind. Using UTF-16 with them while maintaining compatibility with existing programs (such as was done with Windows) requires every system API, library function, and structure that takes a string to be duplicated (note: it is irrelevant whether one of these API's is implemented by translating it to another, the added complexity is still there). UTF-8 only requires APIs that specially treat bytes with the high bit set to be duplicated.
* In UTF-8, characters outside the basic multilingual plane are not a special case. UTF-16 is often mistaken to be constant-length, leading to code that works for most text but suddenly fails for non-[[Basic Multilingual Plane|BMP]] characters. Retrofitting code tends to be hard, so it's better to implement support for the entire range of Unicode from the start.
* UTF-8 uses a byte as its atomic unit while UTF-16 uses a 16-bit word which is generally represented by a pair of bytes. This representation raises a couple of potential problems of its own.
** When representing a word in UTF-16 as two bytes instead of one 16-bit word, the order of those two bytes becomes an issue. A variety of mechanisms can be used to deal with this issue (for example, the [[Byte Order Mark]]), but they still present an added complication for software and protocol design.
** If a byte is missing from a character in UTF-16, software that then tries to read the UTF-16 string from that point will be mis-indexed. The software will think that the first byte it reads is the start of a new character when in reality it is in the middle of a character that lost its beginning byte(s). The result will be either invalid UTF-16 or completely meaningless text. In UTF-8, if part of a multi-byte character is removed, only that character is affected and not the rest of the text. i.e. UTF-8 was made to be self-synchronizing, whereas UTF-16 was not.
* Conversion of a string of random 16-bit values that is assumed to be UTF-16 to UTF-8 is lossless. But conversion of a string of random bytes that is assumed to be UTF-8 to UTF-16 will lose or mangle invalid byte sequences. This makes UTF-8 a safe way to hold data that ''might'' be text; this is surprisingly important in some software.
====Disadvantages====
* Characters above U+0800 in the [[Basic Multilingual Plane|BMP]] use three bytes in UTF-8, but only two in UTF-16. As a result, text in (for example) Chinese, Japanese or Hindi takes up more space when represented in UTF-8. It could be understood for Chinese with its several thousands of characters that more bytes are needed, but in India there have been complaints, in part since their western neighbors using the Arabic letters have only two bytes per character. This disadvantage can be more than offset by the fact that characters below U+0080 (Latin letters, numbers and punctuation marks, space, carriage return and line feed) that frequently appear in those texts take only one byte in UTF-8 while they take two bytes in UTF-16.
* Although both UTF-8 and UTF-16 suffer from the need to handle overlong or invalid strings as described above under general disadvantages, a miswritten parser for UTF-16 is much less dangerous as it will typically only produce half a surrogate pair, or a code outside the BMP, which most software does not assign any important meaning to.
==See also==
* [[Alt codes]]
* [[ASCII]]
* [[Byte Order Mark]]
* [[Comparison of email clients#Features]]
* [[Comparison of Unicode encodings]]
* [[Character encodings in HTML]]
* [[ISO 8859]]
* [[iconv]]—a standardized [[Application programming interface|API]] used to convert between different [[character encoding]]s
* [[GB18030]]
* [[Percent-encoding#Current standard|UTF-8 in URIs]]
* [[Unicode and e-mail]]
* [[Unicode and HTML]]
* [[Universal Character Set]]
* [[UTF-16]]
* [[UTF-9 and UTF-18]]
== References ==
{{reflist}}
==External links==
* [http://plan9.bell-labs.com/sys/doc/utf.html Original UTF-8 paper] ([http://plan9.bell-labs.com/sys/doc/utf.pdf or pdf]) for [[Plan 9 from Bell Labs]]
* RFC 5198 defines UTF-8 [[Unicode normalization|NFC]] for Network Interchange
* UTF-8 test pages by [http://freenet-homepage.de/prilop/multilingual-1.html Andreas Prilop] and the [http://www.w3.org/2001/06/utf-8-test/UTF-8-demo.html World Wide Web Consortium]
* Unix/Linux: [http://www.cl.cam.ac.uk/~mgk25/unicode.html UTF-8/Unicode FAQ], [http://www.linux.org/docs/ldp/howto/Unicode-HOWTO.html Linux Unicode HOWTO], [http://www.gentoo.org/doc/en/utf-8.xml UTF-8 and Gentoo]
* [http://www.utf8-chartable.de/ The Unicode/UTF-8-character table] displays UTF-8 in a variety of formats (with Unicode and HTML encoding information)
* [http://netzreport.googlepages.com/online_tool_for_url_en_decoding.html Online Tool for URL Encoding/Decoding] according to RFC 3986 / RFC 3629([[JavaScript]], [[GNU General Public License|GPL]])
* [http://www.alanwood.net/unicode/browsers.html Unicode and Multilingual Web Browsers] from Alan Wood’s Unicode Resources describes support and additional configuration of Unicode/UTF-8 in modern browsers
* [http://jspwiki.org/wiki/JSPWikiBrowserCompatibility JSP Wiki Browser Compatibility page] details specific problems with UTF-8 in older browsers
* [http://tlt.psu.edu/suggestions/international/bylanguage/math.html#browsers Mathematical Symbols in Unicode]
* [http://en.unicode.se Unicode.se] shows how to set your homepages and databases to UTF-8
* [http://demo.icu-project.org/icu-bin/convexp?conv=UTF-8 Graphical View of UTF-8 in ICU's Converter Explorer]
{{character encoding}}
[[Category:Unicode|UTF-08]]
[[Category:Character sets]]
[[Category:Encodings]]
[[Category:Character encoding]]
[[Category:Unicode Transformation Formats]]
[[ar:صيغة التحويل الموحد-8]]
[[cs:UTF-8]]
[[da:UTF-8]]
[[de:UTF-8]]
[[el:UTF-8]]
[[es:UTF-8]]
[[eo:UTF-8]]
[[fr:UTF-8]]
[[ko:UTF-8]]
[[hr:UTF-8]]
[[it:UTF-8]]
[[he:UTF-8]]
[[lv:UTF-8]]
[[lt:UTF-8]]
[[hu:UTF-8]]
[[nl:UTF-8]]
[[ja:UTF-8]]
[[no:UTF-8]]
[[nn:UTF-8]]
[[pl:UTF-8]]
[[pt:UTF-8]]
[[ru:UTF-8]]
[[sk:UTF-8]]
[[sl:UTF-8]]
[[fi:Unicode#UTF-8]]
[[sv:UTF-8]]
[[tr:UTF-8]]
[[uk:UTF-8]]
[[zh:UTF-8]]