[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: I processed a 3GB XML file ... using XSLT streaming

• From: Hermann Stamm-Wilbrandt <STAMMW@de.ibm.com>
• To: liam@w3.org
• Date: Sat, 14 Sep 2013 01:57:25 +0200

> The point of streaming is (of course) being able to do such things
> without using much memory, even if it's slower.
>
Correct, therefore I did convert Roger's streaming stylesheet so that
it can be processed by DataPower XSLT processor in streaming mode (C).

Processing time is 42 seconds (A).
Memory used while processing is 805MB (1973615K-1149849K), (B)

What I was not able to do is the increment of the counter as
possible with XSLT 3.0 as Roger did -- I did use position() as a
different alternative -- this seems to be a limit of DataPower streaming
processing.

On the other hand DataPower XSLT processor allows for "partial streaming"
(slide 9 of [1]), not sure whether XSLT 3.0 has a similar concept.

> Not everyone has 96G, or even 16G of memory available... :-)
>
Lucky me ;-)


[1] http://www-01.ibm.com/support/docview.wss?uid=swg27019118

(A)
[stammw@dp0-l3 osm]$ time curl --data-binary @massachusetts.osm
http://firestar:2102 >o2.xml
  % Total    % Received % Xferd  Average Speed   Time    Time     Time
Current
                                 Dload  Upload   Total   Spent    Left
Speed
100 2644M    0  285k  100 2643M   7174  64.9M  0:00:40  0:00:40 --:--:--
80.6M

real	0m42.055s
user	0m1.210s
sys	0m2.866s
[stammw@dp0-l3 osm]$ xpath++ "count(/*/*)" o2.xml
6009
[stammw@dp0-l3 osm]$


(B)
xi52# show mem

    Memory Usage: 2 %
    Total Memory: 82333842 kilobytes
     Used Memory: 1973615 kilobytes
     Free Memory: 80360227 kilobytes
Requested Memory: 2034564 kilobytes
     Hold Memory: 60949 kilobytes
 Reserved Memory: 16863558 kilobytes
Installed Memory: 99197400 kilobytes


xi52# show mem

    Memory Usage: 1 %
    Total Memory: 82333842 kilobytes
     Used Memory: 1149849 kilobytes
     Free Memory: 81183993 kilobytes
Requested Memory: 1271360 kilobytes
     Hold Memory: 121511 kilobytes
 Reserved Memory: 16863558 kilobytes
Installed Memory: 99197400 kilobytes


xi52#


(C)
[stammw@dp0-l3 osm]$ cat stream.xsl
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
>
  <xsl:output method="xml" />

  <xsl:template match="/osm">
    <Schools>
      <xsl:apply-templates select="node"/>
    </Schools>
  </xsl:template>

  <xsl:template match="node">
    <xsl:choose>
      <xsl:when test="tag[(@k = 'amenity') and (@v = 'school')]">
        <school>
          <!-- cannot increment counter, take (different) position() -->
          <xsl:value-of select="position()" />

          <xsl:text>. </xsl:text>
          <xsl:value-of select="tag[@k = 'name']/@v" />
        </school>
      </xsl:when>
    </xsl:choose>
  </xsl:template>

</xsl:stylesheet>
[stammw@dp0-l3 osm]$


Mit besten Gruessen / Best wishes,

Hermann Stamm-Wilbrandt
Level 3 support for XML Compiler team and Fixpack team lead
WebSphere DataPower SOA Appliances
https://www.ibm.com/developerworks/mydeveloperworks/blogs/HermannSW/
https://twitter.com/HermannSW/     http://www.stamm-wilbrandt.de/ce/
----------------------------------------------------------------------
IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


|------------>
| From:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Liam R E Quin <liam@w3.org>                                                                                                                       |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Hermann Stamm-Wilbrandt/Germany/IBM@IBMDE,                                                                                                        |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Cc:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |"Costello, Roger L." <costello@mitre.org>, xml-dev@l...                                                                                  |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |09/13/2013 09:17 PM                                                                                                                               |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Re:  I processed a 3GB XML file ... using XSLT streaming                                                                                 |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|





On Fri, 2013-09-13 at 20:53 +0200, Hermann Stamm-Wilbrandt wrote:
> I did give your non-streaming stylesheet (B) a try (A).
> Slight modifications were necessary to get back to XSLT 1.0
> (eq -> = , doc -> document).
> 16GB of memory were used (17851470K-1067694K), (D).

The point of streaming is (of course) being able to do such things
without using much memory, even if it's slower. Not everyone has 96G, or
even 16G of memory available... :-)

Liam

--
Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/
Pictures from old books: http://fromoldbooks.org/
Ankh: irc.sorcery.net irc.gnome.org freenode/#xml


_______________________________________________________________________

XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
subscribe: xml-dev-subscribe@lists.xml.org
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php

• References:
  • I processed a 3GB XML file ... using XSLT streaming
    • From: "Costello, Roger L." <costello@mitre.org>
  • Re: I processed a 3GB XML file ... using XSLT streaming
    • From: Hermann Stamm-Wilbrandt <STAMMW@de.ibm.com>
  • Re: I processed a 3GB XML file ... using XSLT streaming
    • From: Liam R E Quin <liam@w3.org>