[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: Postel's law, exceptions

• To: xml-dev@l...
• Subject: Re: Postel's law, exceptions
• From: Joe English <jenglish@f...>
• Date: Wed, 14 Jan 2004 18:13:26 -0800
• In-reply-to: <87CE71C5-4646-11D8-81EB-000A95A51C9E@t...>
• References: <87CE71C5-4646-11D8-81EB-000A95A51C9E@t...> <40048CA2.2070609@g...> <r02000200-1028-D1176A3A460611D8AD220003937A08C2@[192.168.124.11]> <5B1A714A-461A-11D8-8ADE-000A95CCC59E@x...> <200401140201.i0E21to01550@d...>

Tim Bray wrote:
> On Jan 13, 2004, at 6:01 PM, Joe English wrote:
> > True, but the aggregator as a whole might still accept it --
> > possibly by noticing that the encoding is mislabelled and
> > munging the data into proper UTF-8 before passing it to the
> > parser.
>
> Wow, is there any software that actually does this?  I hadn't 
> encountered it.

My toy aggregator does, after a fashion.


> > In fact any aggregator that doesn't do something like this
> > is doomed to fail -- *nobody's* feed has the encoding labelled
> > properly.  (Well, maybe not "nobody", but certainly not very many.)
>
> On the contrary; the vast majority of them are correct. -Tim

That's not been my experience.  In a small sample of
the ~50 feeds I'm subscribed to, I find:

    5   with Content-Type: text/xml, no charset parameter [*],
        XML declaration claims to be UTF-8;

    5   with Content-Type: text/xml, no charset parameter,
        XML declaration claims to be ISO-8859-1;

    3   with Content-Type: text/html (!); charset="iso-8859-1",
        XML declaration claims to be "UTF-8";

    1   with Content-Type: text/html; charset="utf-8";
        XML declaration at least agrees about the "utf-8" part

    1   RSS feed with content-type: "text/html", no XML declaration

    1   with Content-Type: text/plain, no charset parameter [*],
        XML declaration says UTF-8;

    1   with Content-type: text/plain, no charset parameter,
        XML declaration says ISO-8859-1

    2   with Content-type: text/plain, no charset parameter,
        XML declaration says nothing (these might actually
        be correct, but probably only by accident since they
        happen to contain only 7-bit characters).

    1   with Content-Type: httpd/unix-directory (?!?)


[*] Which means either "US-ASCII" or "ISO-8859-1", depending
on which RFC you take as authoritative.

In 4 cases, the HTTP header and XML declaration agree
on utf-8, in 2 they agree on ISO-8859-1, and rest are
all "application/*".  Of the ones that agree, I can't
say for sure if they're accurate, since the feed itself
happens to contain only 7-bit data at the moment.

I'm surprised at the large number of feeds that don't
even get the *media type* right; I have serious doubts
about the accuracy of the charset parameter.


--Joe English

  jenglish@f...

• Follow-Ups:
  • Re: Postel's law, exceptions
    • From: Amelia A Lewis <amyzing@t...>

• References:
  • Re: Postel's law, exceptions
    • From: Tim Bray <tbray@t...>
  • Re: Postel's law, exceptions
    • From: Julian Reschke <julian.reschke@g...>
  • Postel's law, exceptions
    • From: "Simon St.Laurent" <simonstl@s...>
  • Re: Postel's law, exceptions
    • From: Michael Champion <mc@x...>
  • Re: Postel's law, exceptions
    • From: Joe English <jenglish@f...>

• Prev by Date: Re: Looking for small Java XML Parser
• Next by Date: Re: Postel's law, exceptions
• Previous by thread: Re: Postel's law, exceptions
• Next by thread: Re: Postel's law, exceptions
• Index(es):
  • Date
  • Thread