Cart
[Home] [By Thread] [By Date] [Recent Entries]
On Wed, Oct 03 2007 10:07:55 +0100, mike@s... wrote: > In UTF-8, at the start of a file it is >> just a nonsense character, useless, out-of-place, a sign of >> bad programming, and it messes up encoding detectors > > I'd have said it's a three-byte sequence which tells you pretty reliably > that you're dealing with a UTF-8 encoded file - provided of course that you > are looking for it. Agreed, BOM is a misnomer. > > I'm not defending the decision to add it to the XML spec by means of an > erratum, however. The usefulness or otherwise of U+FEFF in UTF-8 has been subject to reinterpretation over the years. In the Unicode Standard 2.0, there was no mention of U+FEFF with UTF-8, either in the section on the BOM or in the appendix defining UTF-8. In the Unicode Standard 3.0, section 13.6, "Specials", includes: Although there are never any questions of byte-order with UTF-8 text, this sequence can serve as signature for UTF-8 encoded text where the character set is unmarked. In the Unicode Standard 5.0, section 3.10, "Unicode Encoding Schemes", includes: While there is obviously no need for a byte order signature when using UTF-8, there are occasions when processes convert UTF-16 or UTF-32 data containing a byte order mark into UTF-8. When represented in UTF-8, the byte order mark turns into the byte sequence <EF BB BF>. Its usage at the beginning of a UTF-8 data stream is neither required nor recommended by the Unicode Standard, but its presence does not affect conformance to the UTF-8 encoding scheme. Identification of the <EF BB BF> byte sequence at the beginning of a data stream can, however, be taken as a near-certain indication that the data stream is using the UTF-8 encoding scheme. So it's gone from irrelevant to useful to "Oh, if you must". (BTW, in other reinterpretations, "Unicode Encoding Scheme" results from splitting the meaning of "UTF", and the use of U+FEFF to indicate non-breaking is deprecated these days.) The Unicode FAQ both lists its use as a signature [1] and says to avoid its use where "byte oriented protocols expect ASCII characters at the beginning of a file" [2]. However, I don't think that XML necessarily counts as one such byte oriented protocol. Regards, Tony Graham. ====================================================================== Tony.Graham@M... http://www.menteithconsulting.com Menteith Consulting Ltd Registered in Ireland - No. 428599 Registered Office: 13 Kelly's Bay Beach, Skerries, Co. Dublin, Ireland ---------------------------------------------------------------------- Menteith Consulting -- Understanding how markup works ====================================================================== [1] http://www.unicode.org/faq/utf_bom.html#29 [2] http://www.unicode.org/faq/utf_bom.html#28
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] |
