RE: Fwd: HUGE xml input files

> I would like to ask again about the effect of xml-databases 
> on the transformation of huge input files. Why could this 
> enable the transformation of huge files? How can 
> transformations be executed from within xml-db's? Could it be 
> compared to the view-concept of relational databases? Greets, Jan

Databases are generally geared towards handling of large volumes of
data. Most XML databases prefer to handle large numbers of small XML
files, rather than small numbers of huge XML files, but that's a detail
that you sort out when you load the data. The important thing is that
once the data is loaded, it can be indexed for efficient retrieval, and
queried by using the indexes. You do a lot of work at data loading time
to save work at retrieval time, and you optimize the storage for disk
rather than for main memory.

If you really want a transformation that creates 200Mb of output from
200Mb of input, with no sequential relationship between the two, then a
database probably isn't going to help you very much. But if you step
back and ask, why are you doing that transformation? then the chances
are that you are doing it not because someone actually wants all that
data in a different format, but because they want to do further
processing on it. Putting the data in a database and letting people get
the parts of the data that they need, when they need it, in the format
they want it, is an alternative architecture to doing humungous
transformations of the whole data-set. But if you really do need to
build a 200Mb output file, then you can do it without hitting main
memory limitations, because the data is on disk.

Michael Kay
Software AG
home: Michael.H.Kay@xxxxxxxxxxxx
work: Michael.Kay@xxxxxxxxxxxxxx 


 XSL-List info and archive:  http://www.mulberrytech.com/xsl/xsl-list