[XSL-LIST Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: "Heap" of trouble handling input file of 500 MByte

Subject: Re: "Heap" of trouble handling input file of 500 MByte
From: Liam R E Quin <liam@xxxxxx>
Date: Sun, 20 Feb 2011 02:46:03 -0500
Re:  "Heap" of trouble handling input file of 500 MByte
On Sat, 2011-02-19 at 19:47 +0000, thehulk@xxxxxxxxxxx wrote:
> Hello,
> 
> Thanks mainly to this list, I am successfully processing 6,335 of my
> 6,337 input files. The 6,335 are under 250 MByte each. The two problem
> cases are each just under 500 MByte. 
> 
> Are there any tips or tricks or tools which will make this possible on
> my 32-bit Windows XP SP3 machine? 
> 
> I am using Java code and the Javax.xml.* classes to do the transform.
I would not be surprised to find a DOM implementation using 5 or 10
times as much memory as the document, if it's markup-heavy, and for
500MBytes that could mean 5 gigabytes of memory; if you then do an
in-memory DOM to DOM transform, you'll need memory to store the output.

You could try Saxon 9 with its new experimental streaming mode, but if
you are doing sorting, that my not help.

Or you could try Saxon, msxml or libxml2 and see if they are better.

Or just pick up a 64-bit computer with (say) 10 G of RAM, and install a
64-bit Linux on it :-)

It's hard to say more without seeing the nature of the XML input and the
XSLT you're using.

Liam

-- 
Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/
Pictures from old books: http://fromoldbooks.org/

Current Thread

PURCHASE STYLUS STUDIO ONLINE TODAY!

Purchasing Stylus Studio from our online shop is Easy, Secure and Value Priced!

Buy Stylus Studio Now

Download The World's Best XML IDE!

Accelerate XML development with our award-winning XML IDE - Download a free trial today!

Don't miss another message! Subscribe to this list today.
Email
First Name
Last Name
Company
Subscribe in XML format
RSS 2.0
Atom 0.3
Site Map | Privacy Policy | Terms of Use | Trademarks
Free Stylus Studio XML Training:
W3C Member
Stylus Studio® and DataDirect XQuery ™are products from DataDirect Technologies, is a registered trademark of Progress Software Corporation, in the U.S. and other countries. © 2004-2013 All Rights Reserved.