Cart
|
[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message] Re: I processed a 3GB XML file ... using XSLT streaming
Nice sample file, but not reall deep [count(/*/*/*/*)=0].
Even xsltproc was able to process "count(/*)" for that file, see (B) below.
It did need 39GB of resident size though (C), and completed in 1:44 min.
dp0-l3 has 12 cores @3GHz and 96GB memory, but xsltproc did only use one.
In (A) xpath++ tool executes non-streaming and includes network transfer
time from dp0-l3 server to firestar DataPower appliance.
(A)
[stammw@dp0-l3 osm]$ time xpath++ "count(/*)" massachusetts.osm
1
real 0m29.488s
user 0m1.112s
sys 0m2.787s
[stammw@dp0-l3 osm]$ time xpath++ "count(//*)" massachusetts.osm
34840742
real 0m31.547s
user 0m1.144s
sys 0m3.305s
[stammw@dp0-l3 osm]$ time xpath++ "count(/*/*)" massachusetts.osm
10926903
real 0m32.047s
user 0m0.996s
sys 0m2.844s
[stammw@dp0-l3 osm]$ time xpath++ "count(/*/*/*)" massachusetts.osm
23913838
real 0m31.377s
user 0m1.005s
sys 0m2.828s
[stammw@dp0-l3 osm]$ time xpath++ "count(/*/*/*/*)" massachusetts.osm
0
real 0m32.280s
user 0m0.966s
sys 0m2.784s
[stammw@dp0-l3 osm]$
(B)
(xpath++ was configured to use xsltproc here)
[stammw@dp0-l3 osm]$ time xpath++ "count(/*)" massachusetts.osm
1
real 1m44.435s
user 1m28.347s
sys 0m15.707s
[stammw@dp0-l3 osm]$
(C)
top - 19:51:09 up 85 days, 9:56, 6 users, load average: 0.95, 0.68, 0.41
Tasks: 483 total, 3 running, 468 sleeping, 0 stopped, 12 zombie
Cpu(s): 9.1%us, 0.5%sy, 0.0%ni, 90.4%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 99054652k total, 52322948k used, 46731704k free, 2064480k buffers
Swap: 100663288k total, 0k used, 100663288k free, 3192556k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31459 stammw 20 0 39.6g 39g 1204 R 99.6 41.9 1:41.08 xsltproc
Mit besten Gruessen / Best wishes,
Hermann Stamm-Wilbrandt
Level 3 support for XML Compiler team and Fixpack team lead
WebSphere DataPower SOA Appliances
https://www.ibm.com/developerworks/mydeveloperworks/blogs/HermannSW/
https://twitter.com/HermannSW/ http://www.stamm-wilbrandt.de/ce/
----------------------------------------------------------------------
IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
|------------>
| From: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|"Costello, Roger L." <costello@mitre.org> |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|"xml-dev@lists.xml.org" <xml-dev@lists.xml.org>, |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|09/13/2013 05:39 PM |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
| I processed a 3GB XML file ... using XSLT streaming |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
Hi Folks,
I processed this 3GB Open Street Map (OSM) XML file for the state of
Massachusetts:
http://downloads.cloudmade.com/americas/northern_america/united_states/massachusetts/massachusetts.osm.bz2
An OSM file consists of <node> elements (a node represents a point), <way>
elements (a way represents a line or area), and <relation> elements (a
relationship represents a relationship between other elements). This page
describes the elements:
http://wiki.openstreetmap.org/wiki/Tags
Here is a snippet of the XML file; it shows the data for one of the schools
in Massachusetts (a school is identified by @k="amenity" @v="school"):
<osm version="0.6" generator="osm-extract.pl">
...
<node id="358264143" version="1" timestamp="2009-03-10T04:54:34Z"
uid="4732" user="iandees" changeset="774950"
lat="42.2017681"
lon="-70.7561527">
<tag k="gnis:created" v="08/27/2002"/>
<tag k="gnis:county_id" v="023"/>
<tag k="name" v="Scituate Center Central School"/>
<tag k="amenity" v="school"/>
<tag k="gnis:feature_id" v="602607"/>
<tag k="gnis:state_id" v="25"/>
<tag k="ele" v="34"/>
</node>
...
</osm>
Problem: What are all the schools in the state of Massachusetts?
I was able to answer that problem using the new streaming capability in
XSLT 3.0.
Below are two versions of an XSLT program. The first version (non-streaming
version) is how one might try to solve the problem prior to XSLT streaming.
I ran it on the XML file and it quickly halted with an "Out of memory"
error message. The second version uses XSLT streaming and it solved the
problem. There are over 6000 schools in Massachusetts! At the bottom of
this message I show some of the schools.
How long did it take for the streaming XSLT program to process the 3GB XML
file? Answer: 134 seconds.
I ran my streaming XSLT program using oXygen XML, which invoked SAXON. I
ran it on my laptop running Windows 7.
--------------------------------------------------
Non-streaming approach
--------------------------------------------------
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
version="3.0">
<xsl:output method="xml" />
<xsl:variable name="MA" select="doc
('../huge-file/massachusetts.xml')" />
<xsl:template match="/">
<xsl:apply-templates select="$MA/osm" />
</xsl:template>
<xsl:template match="osm">
<Schools>
<xsl:for-each select="node">
<xsl:if test="tag[(@k eq 'amenity') and (@v eq 'school')]">
<school>
<xsl:value-of select="tag[@k eq 'name']/@v" />
</school>
</xsl:if>
</xsl:for-each>
</Schools>
</xsl:template>
</xsl:stylesheet>
--------------------------------------------------
Streaming approach
--------------------------------------------------
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
exclude-result-prefixes="#all"
version="3.0">
<xsl:output method="xml" />
<xsl:template match="/">
<xsl:stream href="../huge-file/massachusetts.xml">
<count>
<xsl:for-each select="osm">
<xsl:iterate select="node">
<xsl:param name="count" select="0"
as="xs:decimal"/>
<xsl:next-iteration>
<xsl:with-param name="count" select="$count
+1"/>
</xsl:next-iteration>
<xsl:on-completion>
<xsl:value-of select="$count"/>
</xsl:on-completion>
</xsl:iterate>
</xsl:for-each>
</count>
</xsl:stream>
</xsl:template>
</xsl:stylesheet>
--------------------------------------------------
Schools in Massachusetts
--------------------------------------------------
<Schools>
<school>1. Wayland Middle School</school>
<school>2. Walnut Hill School for the Arts</school>
<school>3. Library</school>
<school>4. Resource Center</school>
<school>5. Deerfield Academy</school>
<school>6. Arthur W Coolidge Middle</school>
<school>7. Lilliput School</school>
<school>8. L H Coffin</school>
<school>9. St Joseph Central High</school>
<school>10. Stoneham High School</school>
<school>11. St Louis</school>
<school>12. Franklin</school>
<school>13. East Falmouth Elem</school>
<school>14. May Institute (Woburn)</school>
<school>15. Frolio Jr Hs</school>
<school>16. Barbieri Elem</school>
<school>17. Bishop Stang High</school>
<school>18. Jackson Mann</school>
<school>19. Douglas</school>
<school>20. Brookfield Elementary</school>
...
<school>5996. Hawley Grammar School (was here)</school>
<school>5997. Shady Hill School</school>
<school>5998. Cotting School</school>
<school>5999. Morril 2</school>
<school>6000. Morril 4 North</school>
<school>6001. Morril 3</school>
<school>6002. John W Wynn Middle School</school>
<school>6003. John W. Wynn Middle</school>
<school>6004. Boston Architectural College</school>
<school>6005. High School</school>
<school>6006. Sargent College</school>
<school>6007. Stop -n- Go Driving Academy</school>
<school>6008. Ashland High School</school>
<school>6009. Center Stage Dance Academy</school>
</Schools>
_______________________________________________________________________
XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.
[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
subscribe: xml-dev-subscribe@lists.xml.org
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] |
PURCHASE STYLUS STUDIO ONLINE TODAY!Purchasing Stylus Studio from our online shop is Easy, Secure and Value Priced! 
Download The World's Best XML IDE!Accelerate XML development with our award-winning XML IDE - Download a free trial today! Subscribe in XML format
|
|||||||||
