[XSL-LIST Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: Safe-guarding codepoints-to-string() from wrong in

Subject: Re: Safe-guarding codepoints-to-string() from wrong input
From: Abel Braaksma <abel.online@xxxxxxxxx>
Date: Wed, 20 Dec 2006 16:18:31 +0100
Re:  Safe-guarding codepoints-to-string() from wrong in
Andrew Welch wrote:

If you are receiving strings containing literal control characters then they're almost definitely encoded in Windows-1252 - just parse them using that and you'll be ok.

No, that's not it. The codepoints are encoded using literal numeric hexadecimal strings (compare &#x0A; in XML, which would be [0A] in the original example)



If the string contains control characters as character references, then its a bit harder because the references get expanded using unicode codepoints, and not those specified in the Windows-1252 mappings... So you need to parse/serialize the string to expand the references (I personally use JTidy with the CharEncoding set to Configuration.RAW which forces the Tidy to output the bytes instead of a reference)

Its a pain....

Well, that's encouraging ;)


The project contains strings that are "escaped" in several ways (texts are literal):
C-style: \x0ASome text \x22between quotes\x22
Local style: <0A>Some text <22>between quotes<22>
Other style: Text with <22,24,54> multiple special chars
XML-like: &0A;Some text &22;between quotes&22;


In short: the input is rubbish. But we know for a fact how to get the codepoints. However, in the past, users have made mistakes. The original application simply ignored those mistakes, replacing the illegal codepoints with nothingness.

The good news is: all codepoints are Unicode codepoints.

Thanks,
-- Abel

Current Thread

PURCHASE STYLUS STUDIO ONLINE TODAY!

Purchasing Stylus Studio from our online shop is Easy, Secure and Value Priced!

Buy Stylus Studio Now

Download The World's Best XML IDE!

Accelerate XML development with our award-winning XML IDE - Download a free trial today!

Don't miss another message! Subscribe to this list today.
Email
First Name
Last Name
Company
Subscribe in XML format
RSS 2.0
Atom 0.3
Site Map | Privacy Policy | Terms of Use | Trademarks
Free Stylus Studio XML Training:
W3C Member
Stylus Studio® and DataDirect XQuery ™are products from DataDirect Technologies, is a registered trademark of Progress Software Corporation, in the U.S. and other countries. © 2004-2011 All Rights Reserved.