[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

RE: Transformative Programming: Flow-based, functional,and mor

• From: David Lee <dlee@calldei.com>
• To: Simon St.Laurent <simonstl@simonstl.com>, "xml-dev@l..."<xml-dev@l...>
• Date: Wed, 16 Oct 2013 18:57:00 +0000

I'm sure it could be made to work, but I'm not sure it should be made to 
work.
=============

My very reason for implement xmlsh pipelines *in process* was that it is highly inefficient in practice to have lots of little tasks
in a pipeline if you have a heavyweight protocol connecting them (say, starting a sub-process).  In my case 100x - 10,000x inefficient.

I simply cannot imagine writing an pipeline where every step involved a REST call ...
It *could* be done, and maybe in cases parts of a pipeline are well suited to an external REST call ... (say fetching an external resource)
but basing the fundamental architecture of a pipeline infrastructure on REST to me makes no sense.  It would lead to the opposite,
optimizing the pipeline so it had as few (but huge) pieces as possible to get over the infrastructure cost of step invocation.

Now to flip sides, some standardized workflow engines are based on either SOAP (BPEL) or REST (AWS Workflow) ... 
but IMHO  for similar reasons, each step tends to be huge and the workflow coordinates as few as possible big steps because the infrastructure
to cross domains is heavy.   This is a mega-scale pipeline, not a micro-scale.  Scale matters.



-David

----------------------------------------
David A. Lee
dlee@calldei.com
http://www.xmlsh.org

• References:
  • Transformative Programming: Flow-based, functional, and more
    • From: "Simon St.Laurent" <simonstl@simonstl.com>
  • Re: Transformative Programming: Flow-based, functional, and more
    • From: Peter Hunsberger <peter.hunsberger@gmail.com>
  • Re: Transformative Programming: Flow-based, functional, and more
    • From: Uche Ogbuji <uche@ogbuji.net>
  • Re: Transformative Programming: Flow-based, functional, and more
    • From: Peter Hunsberger <peter.hunsberger@gmail.com>
  • Re: Transformative Programming: Flow-based, functional,and more
    • From: "Simon St.Laurent" <simonstl@simonstl.com>
  • Re: Transformative Programming: Flow-based, functional, and more
    • From: Uche Ogbuji <uche@ogbuji.net>
  • RE: Transformative Programming: Flow-based, functional,and more
    • From: David Lee <dlee@calldei.com>
  • RE: Transformative Programming: Flow-based, functional,and more
    • From: "Rushforth, Peter" <Peter.Rushforth@NRCan-RNCan.gc.ca>
  • Re: Transformative Programming: Flow-based, functional,and more
    • From: "Simon St.Laurent" <simonstl@simonstl.com>