[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: combining XMLEvent lists

• From: Michael Kay <mike@saxonica.com>
• To: xml-dev@lists.xml.org
• Date: Tue, 28 Sep 2010 15:33:20 +0100

> it's the reduce functionality in a Hadoop MapReduce task. I'm going to
> split up the huge wikipedia articles revision file (several thousands of
> articles/pages) into
>
> <page>
>    ...
>    <revision><timestamp>...</timestamp></revision>
>    <revision><timestamp>...</timestamp></revision>
>    ...
> </page>
>
> chuncks (each page represents one article with several revisions which
> are not entirely sorted), then split into key (timestamp) and values
> (page-fragments) for every page concurrently:
>
> <page>
>    <revision><timestamp>...</timestamp></revision>
> </page>
>
> <page>
>    <revision><timestamp>...</timestamp></revision>
> </page>
>

Sounds fascinating, and I wish I had time to get involved. It would 
certainly be elegant if you could have both the productivity of writing 
this declaratively in XSLT and the performance of running it on Hadoop 
MapReduce. Intrinsically, the two seem to fit together hand in glove, 
but I suspect some engineering effort is needed to make it work.

Michael Kay
Saxonica

• Follow-Ups:
  • Re: combining XMLEvent lists
    • From: "Johannes.Lichtenberger" <Johannes.Lichtenberger@uni-konstanz.de>

• References:
  • combining XMLEvent lists
    • From: "Johannes.Lichtenberger" <Johannes.Lichtenberger@uni-konstanz.de>
  • Re: combining XMLEvent lists
    • From: Michael Kay <mike@saxonica.com>
  • Re: combining XMLEvent lists
    • From: "Johannes.Lichtenberger" <Johannes.Lichtenberger@uni-konstanz.de>