Tool development: by Perl-wrapped XQuery

[Hans-Juergen Rennau]

Hello David,

yes, I agree that it is desirable to be free in one's choice how to shape the units of functionality, large or small. Still, I would like to make a couple of points.

You wrote:
"... so they have instead built 'monolithic' programs that do all the work within 1 program (xquery, xslt etc) and take a list of filenames input and maybe filenames output to process. ... I suggest that programming model is no longer necessary, and was arrived at due to the poor performance of doing it any other way,"

Sometimes, perhaps even often, this may be the case. But one should not forget that XQuery's aggregative potential teaches us to think in terms of sequences rather than in terms of items. This sweeping approach has nothing to to with "monolithic", just as nobody would call XPath expressions monolithic.

When it comes to the other dimension of scaling - the number of processing steps (rather than the number of units processed in a given step) - it is not at all per se better to have many small programs. As a rule of thumb I suggest "divide the processing if and only if reuseable steps result". So, for example, generic preprocessings (as the one Michael Kay recently recommended in the case of Excel data) are fine examples where dividing is desirable. But if the dividing produces pieces where the second depends on the first but the first serves no other purpose than prepare the second - then the dividing is bad and an unnecessary increase of complexity. Speaking of XQuery - the design of modules and functions is important, not the creation of little programs. XQuery is a powerful language suited for writing highly complex programs. Doing this does not mean following a "monolithic" programming model.

This being said, two final remarks. First, xmlsh seems so important to me because it aims to overcome a sheer anachronism of conventional Unix shells - their inability to deliver structured information. Second, I regard XProc as the sorely needed integration technology which may greatly enhance the value of the various languages (XQuery, ...); even more, it may pave the way to a new paradigm of data processing, in which concepts like "step" and "ports" become so natural to us as "documents" and "objects".

With kind regards,
Hans-Juergen




----- Ursprüngliche Mail ----
Von: David A. Lee <http://x-query.com/mailman/listinfo/talk>
An: Hans-Juergen Rennau <http://x-query.com/mailman/listinfo/talk>
CC: http://x-query.com/mailman/listinfo/talk; http://x-query.com/mailman/listinfo/talk
Gesendet: Freitag, den 4. September 2009, 10:10:37 Uhr
Betreff: Re:  Tool development: by Perl-wrapped XQuery

I agree that if you stick to use case #1 (only call xquery once or few 
times) and all your input and output are files (or file names) then 
pretty much any scripting language that lets you launch a subprocess is 
equivalent and you will have very little performance hit.  
Historically, I suggest many people use that use case primarily because 
its horrendously expensive not to (in most languages) so they have 
instead built 'monolithic' programs that do all the work within 1 
program (xquery, xslt etc) and take a list of filenames input and maybe 
filenames output to process.
This does work.

My point is that I suggest that programming model is no longer 
necessary, and was arrived at due to the poor performance of doing it 
any other way,
not because its a desirable way of coding.
If you use a language that can call xquery *efficiently* you can design 
in a more modular way and use 'itty bitty' xquery programs run thousands 
of times instead of one huge program run once,  just as efficiently, and 
in my opinion much easier to develop, debug, and understand.  It frees 
you to design and develop in modular ways, using languages of your 
choice for different operations instead of forcing you into a single 
monolithic application.
Of course it is a personal *opinion* that that is a 'better' way of 
doing things :)  I certainly enjoy it more.
Others may prefer writing one huge program that does everything.  
This exact same philosophical (and technical) debate is what 
distinguished unix from the mainframe model of programming.  I find it 
fascinating that the same fundamental design issues and discussions 
evolved again 40 years later.



David A. Lee
>