[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: I processed a 3GB XML file ... using XSLT streaming

• From: Martin Honnen <Martin.Honnen@gmx.de>
• To: "Costello, Roger L." <costello@mitre.org>, "xml-dev@l..." <xml-dev@l...>
• Date: Fri, 13 Sep 2013 17:58:57 +0200

Costello, Roger L. wrote:

> I processed this 3GB Open Street Map (OSM) XML file for the state of Massachusetts:
>
> http://downloads.cloudmade.com/americas/northern_america/united_states/massachusetts/massachusetts.osm.bz2
>
> An OSM file consists of <node> elements (a node represents a point), <way> elements (a way represents a line or area), and <relation> elements (a relationship represents a relationship between other elements). This page describes the elements:
>
> http://wiki.openstreetmap.org/wiki/Tags
>
> Here is a snippet of the XML file; it shows the data for one of the schools in Massachusetts (a school is identified by @k="amenity" @v="school"):



> Problem: What are all the schools in the state of Massachusetts?
>
> I was able to answer that problem using the new streaming capability in XSLT 3.0.
>
> Below are two versions of an XSLT program. The first version (non-streaming version) is how one might try to solve the problem prior to XSLT streaming. I ran it on the XML file and it quickly halted with an "Out of memory" error message. The second version uses XSLT streaming and it solved the problem. There are over 6000 schools in Massachusetts! At the bottom of this message I show some of the schools.

> --------------------------------------------------
>      Streaming approach
> --------------------------------------------------
> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
>                           xmlns:xs="http://www.w3.org/2001/XMLSchema"
>                           exclude-result-prefixes="#all"
>                           version="3.0">
>
>      <xsl:output method="xml" />
>
>      <xsl:template match="/">
>          <xsl:stream href="../huge-file/massachusetts.xml">
>              <count>
>                  <xsl:for-each select="osm">
>                      <xsl:iterate select="node">

If 'a school is identified by @k="amenity" @v="school"', I wonder why 
that condition does not need to appear in your code.

>                          <xsl:param name="count" select="0" as="xs:decimal"/>
>                          <xsl:next-iteration>
>                              <xsl:with-param name="count" select="$count+1"/>
>                          </xsl:next-iteration>
>                          <xsl:on-completion>
>                              <xsl:value-of select="$count"/>
>                          </xsl:on-completion>
>                      </xsl:iterate>
>                  </xsl:for-each>
>              </count>
>          </xsl:stream>
>      </xsl:template>
>
> </xsl:stylesheet>
>
> --------------------------------------------------
>      Schools in Massachusetts
> --------------------------------------------------
> <Schools>
>      <school>1. Wayland Middle School</school>
>      <school>2. Walnut Hill School for the Arts</school>

Where do you generate that list? The streaming version of your program 
seems to just output a 'count' element but not 'Schools' nor 'school' 
elements.

• Follow-Ups:
  • RE: I processed a 3GB XML file ... using XSLT streaming
    • From: "Costello, Roger L." <costello@mitre.org>

• References:
  • I processed a 3GB XML file ... using XSLT streaming
    • From: "Costello, Roger L." <costello@mitre.org>