Re: GByte Transforms

On Thursday 03 June 2004 03:11, Kevin Jones wrote:
> We have been doing some work on the Sarvega processor to extend the size of
> the input documents it can cope with. In the latest revision we have
> managed a transform of a 5.5 GByte input document on standard 32-bit Intel
> PC hardware using a customer sample transform. This is not an upper limit
> but enough to break over some 4 GByte boundary issues that we wanted to
> test.

Small comment only;  Doesn't the these kind of transforms have to be so basic, 
that it makes more sense to do them directly in a SAX application??

I have only worked with 100's MB documents, and never attempted to do 
transforms with XSLT, since memory requirement easily run amok. Instead I 
built a 'single branch node builder', which never holds more data than a 
single branch (i.e. the elements from the root to any leaf), and have most 
work happen on SAX endElement events.

Performance bottle neck is then near storage performance, and memory 
requirements in the 10's of MB in my cases.


Cheers
Niclas
-- 
   +------//-------------------+
  / http://www.bali.ac        /
 / http://niclas.hedhman.org / 
+------//-------------------+