[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: combining XMLEvent lists

• From: "Johannes.Lichtenberger" <Johannes.Lichtenberger@uni-konstanz.de>
• To: xml-dev@lists.xml.org
• Date: Tue, 28 Sep 2010 16:19:50 +0200

On 09/28/2010 02:42 PM, Michael Kay wrote:
> Clearly a job for XSLT 2.0, where the processing will be vastly easier
than in Java.

Hi Michael,

it's the reduce functionality in a Hadoop MapReduce task. I'm going to
split up the huge wikipedia articles revision file (several thousands of
articles/pages) into

<page>
  ...
  <revision><timestamp>...</timestamp></revision>
  <revision><timestamp>...</timestamp></revision>
  ...
</page>

chuncks (each page represents one article with several revisions which
are not entirely sorted), then split into key (timestamp) and values
(page-fragments) for every page concurrently:

<page>
  <revision><timestamp>...</timestamp></revision>
</page>

<page>
  <revision><timestamp>...</timestamp></revision>
</page>

and then let Hadoop group and sort according to the key timestamp.

Though I would need to serialize the lists temporarily, process the file
with for example Saxon and then serialize the results once again.

It would be great if it would be possible to build a tree representation
or even streaming variant out of XMLEvents, but I think that's not
possible with Saxon nor planned!? :-/ I agree that it's for sure simpler
with XSLT even though I would need some assistance in writing the
stylesheet :-)

regards,
Johannes

• Follow-Ups:
  • Re: combining XMLEvent lists
    • From: Michael Kay <mike@saxonica.com>

• References:
  • combining XMLEvent lists
    • From: "Johannes.Lichtenberger" <Johannes.Lichtenberger@uni-konstanz.de>
  • Re: combining XMLEvent lists
    • From: Michael Kay <mike@saxonica.com>