[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: Detection of non-Unicode characters

• To: xml-dev@l...
• Subject: Re: Detection of non-Unicode characters
• From: Matt Gushee <mgushee@h...>
• Date: Fri, 23 Aug 2002 16:36:30 -0600
• In-reply-to: <3D66A823.2060109@t...>
• References: <4DBDB4044ABED31183C000508BA0E97F040ABF38@f...> <3D66A823.2060109@t...>
• Reply-to: Matt Gushee <mgushee@h...>
• Sender: <mgushee@h...>
• User-agent: Mutt/1.3.27i

On Fri, Aug 23, 2002 at 02:24:51PM -0700, Tim Bray wrote:
> Mark Feblowitz wrote:
> >We've gotten ourselves in a slight muddle. We've copied Word documentation
> >into (many) xs:annotation blocks in our UTF-8 .xsd files (there are around
> >300 files). In the process, we have apparently brought along some
> >non-Unicode characters. This is not tolerated equally well by all tools.
> 
> Second possible problem is that the UTF-8 is good but it encodes Unicode 
> characters that aren't allowed in XML, like for example  - any 
> decent XML parser should catch this and give you helpful error messages, 
> if you have an expat around your system (and a lot of people do these 
> days) "xmlwf [filename here]" will do the trick.  D'oh, now that I think 
> about it in fact I bet xmlwf (or equivalent) would probably catch the 
> UTF8 breakage too.  -Tim

I would bet it's this. Just this past week I have been debugging a
broken application that is supposed to generate XML from Word documents.
The main problem I found was that the Word documents are full of
characters like 0x07, 0x2012-0x2019, and the like. The latter range
consists of common punctuation symbols like dashes and left and right
quotes (AKA 'smart quotes'). They appear to be using Code Page 1252
mapped directly into Unicode.

P.S. Why are those punctuation symbols illegal in XML, anyway?

-- 
Matt Gushee
Englewood, Colorado, USA
mgushee@h...
http://www.havenrock.com/

• Follow-Ups:
  • Re: Detection of non-Unicode characters
    • From: Ann Navarro <ann@w...>
  • Re: Detection of non-Unicode characters
    • From: "J.Pietschmann" <j3322ptm@y...>

• References:
  • Detection of non-Unicode characters
    • From: Mark Feblowitz <mfeblowitz@f...>
  • Re: Detection of non-Unicode characters
    • From: Tim Bray <tbray@t...>

• Prev by Date: [ANN] XML and Web Services Connections (31 Oct- 2 Nov in Orlando)
• Next by Date: Re: XML indexing/search engine
• Previous by thread: Re: Detection of non-Unicode characters
• Next by thread: Re: Detection of non-Unicode characters
• Index(es):
  • Date
  • Thread