[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

There is a serious amount of character encoding conversionsoccurring ins

  • From: "Costello, Roger L." <costello@mitre.org>
  • To: "xml-dev@lists.xml.org" <xml-dev@lists.xml.org>
  • Date: Fri, 28 Dec 2012 14:01:48 +0000

There is a serious amount of character encoding conversionsoccurring ins
Hi Folks,

All of the characters in this XML file are encoded in iso-8859-1:

<?xml version="1.0" encoding="iso-8859-1"?>
<Name>López</Name>

The following problem occurred to me: 

    Suppose I search that XML document for the string López, 
    where the characters in López are encoded in UTF-8. Will the 
    search application find a match?

I did the search and it found a match.

Amazing!

How did it find a match?

The underlying byte sequence for the iso-8859-1 López is: 4C F3 70 65 7A (one byte -- F3 -- is used to encode ó).

The underlying byte sequence for the UTF-8 López is: 4C C3 B3 70 65 7A (two bytes -- C3 B3 -- are used to encode ó).

The search application cannot be doing a byte-for-byte match, else it would find no match.

The codepoint for the UTF-8 ó character is F3. 

Hey, iso-8859-1 uses F3 to encode ó.

So perhaps the search application is converting the UTF-8 bytes to codepoints and then comparing those codepoints to the iso-8859-1 bytes. That would result in a match.

I am told that each search application may do things differently.

In any case, there's a whole lot of encoding conversions occurring ... transparently ... without our knowing.

A second problem occurred to me: Suppose I copy the iso-8859-1 López, and paste it into Google. Will Google only return those web pages that contain iso-8859-1-encoded López strings? 

Wow!

That is a mind-boggling thought.

If Google only returns iso-8859-1-encoded López strings then that would force users to perform searches using many different encodings -- search for  iso-8859-1-encoded López and then search for UTF-8-encoded López, and so forth.

That would be a nightmare!

Clearly we don't do that. Google must be converting all search strings and all Web pages to ... what? ... Unicode codepoints? 

A third problem occurred to me: Suppose I write an XSLT program and encode all its characters in iso-8859-1:

--------------------------------------------------------
<?xml version="1.0" encoding="iso-8859-1"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">

    <xsl:output method="text"/>

    <xsl:template match="/">
        <xsl:text>López</xsl:text>
    </xsl:template>

</xsl:stylesheet>
--------------------------------------------------------

If I apply that XSLT program to an XML document, also encoded in iso-8859-1, what will be the encoding of the result? 

I did it and the XSLT processor output the result in UTF-8. I wonder why?

This character encoding stuff is fascinating.

Some mighty smart fellows figured this character encoding stuff out long ago and now it is buried so deep in the fabric of our computers and the Web that we are completely oblivious to all the encoding conversions that are happening.

/Roger



[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index]


PURCHASE STYLUS STUDIO ONLINE TODAY!

Purchasing Stylus Studio from our online shop is Easy, Secure and Value Priced!

Buy Stylus Studio Now

Download The World's Best XML IDE!

Accelerate XML development with our award-winning XML IDE - Download a free trial today!

Don't miss another message! Subscribe to this list today.
Email
First Name
Last Name
Company
Subscribe in XML format
RSS 2.0
Atom 0.3
 

Stylus Studio has published XML-DEV in RSS and ATOM formats, enabling users to easily subcribe to the list from their preferred news reader application.


Stylus Studio Sponsored Links are added links designed to provide related and additional information to the visitors of this website. they were not included by the author in the initial post. To view the content without the Sponsor Links please click here.

Site Map | Privacy Policy | Terms of Use | Trademarks
Free Stylus Studio XML Training:
W3C Member
Stylus Studio® and DataDirect XQuery ™are products from DataDirect Technologies, is a registered trademark of Progress Software Corporation, in the U.S. and other countries. © 2004-2013 All Rights Reserved.