[XSL-LIST Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Transform a million XML documents

Subject: Transform a million XML documents
From: "Costello, Roger L. costello@xxxxxxxxx" <xsl-list-service@xxxxxxxxxxxxxxxxxxxxxx>
Date: Fri, 10 Feb 2017 18:43:09 -0000
 Transform a million XML documents
Hi Folks,

Eliot Kimber raised a neat question on the SAXON mailing list.

Here is a summary of the ensuing discussion.

Scenario: There are a million XML documents that need to be transformed. Each
file is in the 1-4KB range. The files are organized into directories about 4
or 5 deep and some directories have 100s or 1000s of files.

Use XSLT to do the transformations.

Specifically, use the XSLT collection() function along with
saxon:discard-document().

Transforming a million files is easily handled by Saxon-EE, which uses
multiple threads for document parsing (equally xsl:result-document will use
multiple threads for writing the result). A key thing to remember is to use
saxon:discard-document() to ensure that the documents are garbage collected
after processing.

Here is the XSLT code:

<xsl:for-each select="for $x in
collection('file:///c:/path/to/xml?select=*.xml;recurse=yes;on-error=ignore')
return saxon:discard-document(f.doTransform($x))">

Current Thread

PURCHASE STYLUS STUDIO ONLINE TODAY!

Purchasing Stylus Studio from our online shop is Easy, Secure and Value Priced!

Buy Stylus Studio Now

Download The World's Best XML IDE!

Accelerate XML development with our award-winning XML IDE - Download a free trial today!

Don't miss another message! Subscribe to this list today.
Email
First Name
Last Name
Company
Subscribe in XML format
RSS 2.0
Atom 0.3
Site Map | Privacy Policy | Terms of Use | Trademarks
Free Stylus Studio XML Training:
W3C Member
Stylus Studio® and DataDirect XQuery ™are products from DataDirect Technologies, is a registered trademark of Progress Software Corporation, in the U.S. and other countries. © 2004-2013 All Rights Reserved.