On Sat, 2012-12-01 at 21:08 -0500, Graydon wrote:
> On Sat, Dec 01, 2012 at 06:50:12PM -0500, Liam R E Quin scripsit:
[...]
> > You might want to investigate using an XQuery implementation that
> > builds an index to the documents - even if you just use it to generate
> > temporary documents that you then use to help your XSLT
> > transformations.
> 
> Which is actually what I'm doing -- use BaseX to load the ~4.5
> GB of date I need to search, use XQuery to build a ~100 MB file of
> locations and then use that file to search against when generating the
> report.

100 MBytes is still a fairly large document to use with XSLT.

> The search time seems to be roughly 30 seconds plus one second per
> megabyte of "find stuff in here and check it" content.
> 
> Dealing with "no, that isn't going in memory" volumes of XML has
> certainly been a learning experience!

Yes - people find the same thing with almost any data structure.

On the other hand, adding more physical memory can be an effective way
to make things go faster...

Liam

-- 
Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/
Pictures from old books: http://fromoldbooks.org/
Ankh: irc.sorcery.net irc.gnome.org freenode/#xml