Tool development: by Perl-wrapped XQuery

[Hans-Juergen Rennau]

Hello David,

gradually I start to get a feeling for your perspective. Pulling "an efficient mechanism for interconnecting them" into focus you move across the boundaries of the XQuery spec's field of vision. Your criticism that design is not only function and module design gains power. Vaguely but vividly this reminds me of the SOAP versus REST controversy: with SOAP clinging to an artificial complication ("I am calling something...") and REST arriving at the more fundamental abstractions of resource and address,  enabling a new ease of access and of combining existing resources. Modules are a category emerging only *after* I have called a query, reusable inside queries, whereas the outside is, to put it mildly, bigger and more important.

The great primitives of Unix - the fundamental tools - target the basic entities of an OS environment: (byte) streams, files, directories, users, ... I wonder what new targets emerge with the transition to xmlsh? This is a very important question, relevant as much to xmlsh as to tool development in general - which tools should occupy the top of the community's agenda? ! It occurs to me that there are (at least) two targets of prime importance:

1. (of course) item streams (or 'sequences', 'infosets', 'XDM instances) 
=> XDM info commands (or 'services', 'tools', ...)

2. (but also) XML Schemas (!) 
=> xsd info commands

There were no "schemas" to describe byte streams, but we have them to describe item streams, and I think here is a great demand of standard tools unlocking their wealth of information. For example:

get-simple-type myElementName | simple-type-pattern
=> yielding the pattern constraining an element (e.g.  "\D\D\D\d+"), or

get-paths myFile | invalid-paths myTargetNamespacePrefix
=> yielding any paths not compatible with a certain schema.

Reading
'XQuery is useful and 'important' at the single expression level, all the way up to a full "program"'

and remembering that XQuery is a fully composable language one feels that it is very important to think about the INTEGRATION of XQuery into software and processes in all its aspects. (Especially because XQuery is an Information Processing Language, but not a general purpose programming language!) One could even argue that the still small acceptance of XQuery is related to still insufficient integration.

I used to regard XProc as *the* enabler, *the* integrator. Now I get a feeling that one might look at XProc and xmlsh as sister technologies resembling XSLT and XQuery: both built on the same basis, the XDM and the XPath language; the former a little more powerful and feature-rich, with the cost of rather heavy syntax, the latter very slender and flexible.

With kind regards,
Hans-Juergen




----- Ursprüngliche Mail ----
Von: David A. Lee <http://x-query.com/mailman/listinfo/talk>
An: Hans-Juergen Rennau <http://x-query.com/mailman/listinfo/talk>
CC: http://x-query.com/mailman/listinfo/talk
Gesendet: Samstag, den 5. September 2009, 04:57:12 Uhr
Betreff: Re:  Tool development: by Perl-wrapped XQuery

Hello Hans-Juergen

I do agree with a lot of your comments ...
However to add to this one There's another side to this.
> When it comes to the other dimension of scaling - the number of processing steps (rather than the number of units processed in a given step) - it is not at all per se better to have many small programs. As a rule of thumb I suggest "divide the processing if and only if reuseable steps result". So, for example, generic preprocessings (as the one 
The third aspect you don't mention is ease of development.   I personally find its *vastly easier* to author, debug, and later maintain an xml translation that is decomposed into say 5  steps in individual files with concrete results in each "step".   I can run each xquery /xslt individually and output the results (either command line or in a GUI) and debug each one separately.   Then put them together and debug the results.   Even if *none* of the steps are ever "reusable" either in the past or the future to me this is a very powerful advantage of "divide and conquer".    If the end result can be run as efficiently as if they were all in one big program I see that as a huge win.


And as such I disagree with this comment

"Speaking of XQuery - the design of modules and functions is important, not the creation of little programs. "

I suggest that is inventing/imposing an unnecessary distinction between "program" and "modules" and presupposing what is "important".
It may not have been 'important' in the authors of xquery's mind, but thats the strength of reusable software - to grow beyond the original intent.
XQuery is useful and 'important' at the single expression level, all the way up to a full "program" whatever that is.
Just because xquery supports a module implementation does not mean thats the only valuable way of composing larger programs that use xquery as s component.

This is what Unix shell programming taught us 40 years ago, and what programs like XProc are teaching us today..
Its also what frameworks like .NET that support things like inheritance across languages teaches us.

That the distinction between "program" and "module" is fuzzy.   And that by making 'little programs' that do one thing only, but do it well,
and providing an efficient mechanism for interconnecting them, you can produce exponentially complex *behavior* without exponentially complex *code*.
The distinction between 'program' and 'module' is subjective and has overlapping meanings in different contexts.
Is "ls" a program or a module ?   How about "more" ? then what about  'lsmore' implemented as "ls | more" ?   It is this artificial distinction between "module" and "program" itself that leads to what I called "monolithic" programs.

There's a long ways we could go with this.  Imagine a language that could 'inherit' a single template from xslt in the context of another language ?
( such as .NET allows across disparate languages) .
And of course taking it too far you can produce exponentially inefficient and complex code that does nothing but blow up the cpu  :)

Were beginning to see a more of this (xquery in small pieces), such as using tiny bits of XQuery inside other programs like "XQuery in the browser".   Little bits of XQuery in XML data bases.  XQuery being used as part of the business logic in a CRM system (yes this exists) coexisting with other languages.
Are these "programs" or "modules" or even less just single expressions ?  It doesn't matter.  

What matters is can you easily and efficiently hook them up and use them in ways the original authors may have never dreamed.
But in my opinion they are definitely 'important'

David A. Lee
http://x-query.com/mailman/listinfo/talk  http://www.calldei.com
http://www.xmlsh.org
812-482-5224