Re: [xquery-talk] ETL for XML !??? Nobody realizes that XML toolseven EX

I have had to implement  workarounds when trying to deploy XSLT/XQuery on datasets circa 250MB, presumably some hardware jiggery pokery could increase my processing capacity butt there clearly is a limit.

With plain text files this was simply a matter of running the transformation on files that had fragmented courtesy of grep, awk and friends, but the XML input required the  replication of a homespun mapreduce to get the required result, which to be fair that was a bit more than a straightforward ETL.

Point being there was some X(SLT|QUERY) jiggery pokery (Bill McLaren RIP) going on, many would throw in the towel at the first heap error and never get to the stage I got to. Secondly the bash parts of the pipeline which include  joins and sorts that aggregate across about 15 files run much much quicker. Hence  I started with an XSLT transformation that did everything because it looked like a simple upconversion  and ended up with the XSLT transformation reduced to a final step that runs on heavily pre-processed and fragmented files and still takes the bulk of the processing time.

I don't know the extent to which the availability of streaming capabilities (both for text and XML) would change that scenario but the fact is that making streaming only available in the enterprise versions of these products has left a lacuna for other products to step in and fill. That's not a criticism and it is not my place or intention to opine on the release strategy of Saxon or any other offering, but it does seem to be a logical conclusion.

Of course that's not to say that it would make a blind bit of difference to these people. They are still at circa 1980 on the evolutionary cycle (just discovering that you actually do need a query language) so remember that before you remind them it is not 1995.