1. Performing impossible feats of XML processing with pipelining
XML Open 2004
Sean McGrath
Propylon

2. Contents The pipelining philosophy
Major functional elements of pipelines
Some examples
Pipelining and Grids
Pipelining and Web Services/SOAs
Some anticipated objections (and answers)
Some musings
Some technology pointers

3. What is XML pipelining?
It is an architectural framework for developing robust, scaleable, manageable XML processing systems.
based on proven mechanical manufacturing patterns. Specifically:
Assembly Lines (divide and conquer)
Component assembly and component re-use

4. What is XML pipelining and why is it useful?
A way of thinking about systems that focuses on XML dataflows rather than object APIs. (This is critical and non-trivial focus-shift for many programmers!)
Why? Because pipelining provides a mechanical, inspiration-free, genius-free way of handling the mind-boggling complexity of complex XML transformation projects.

5. Pipelining Philosophy
XML is all about complex hierarchical data structures…

6. Pipelining Philosophy
Cars are complex, hierarchical structures
Henry Ford’s Model T Ford Assembly Line – 1914

7. Pipelining Philosophy
Lunch is a complex, hierarchical structure
Lunch Assembly Line. NY, 2004

8. Pipelining Philosophy
We are complex, hierarchical structures

9. Pipelining philosophy
What have these scenes got it common?
Complex construction of cars, tuna melts and tendons made possible and efficient through
assembly line manufacturing pattern of divide and conquer
re-usable component processes and component materials
Why not apply this approach to XML “manufacturing”?

10. Pipeline philosophy Why does the assembly line approach work?
Transformation task decomposition
Re-usable transformation components
Transformation decomposition is the key to complexity management. Just ask:
Henry Ford
Herbert Simon (The Two Watchmakers – “The Architecture of Complexity”)
George Miller (7+/-2)
Adam Smith (An Inquiry into the Nature And Causes of the Wealth of Nations,1776)
Any electrical or chemical engineer.

11. Pipeline philosophy Component re-use is the key to productivity
Ask any form of engineer (electrical, chemical etc.) apart from software engineers…
Component re-use remains a holy grail in software engineering
Pipelining is yet another attempt based on data transformation and data flow rather than algorithms

12. Pipeline philosophy
A lot of data processing for the forseable future will consist of XML to XML transformation
A lot of non-XML data processing can consist of XML to XML transformations with the addition of top and tail transformations to non-XML formats
An XML pipeliners mantra:
Get data into XML as quickly as possible
Keep it in XML until the last possible minute
Bring all your XML tools to bear on solving the data processing problem

13. Pipeline philosophy Input XML Output XML Top Transformation
Tail Transformation
Non-XML
Input
Non-XML
Output

14. Pipeline philosophy
The philosophy hinges on the fact that every complex XML transformation can be broken down into a series of smaller ones than can be chained together

15. Pipeline philosophy
Only so many ways to re-arrange an XML tree structure
A finite number of fundamental transformations, from which all transformations can be derived

16. Pipeline philosophy
Starting point: data at time T conforming to “spec” A. Data at time T2 conforming to “spec.” B.
Transformation Analysis/Decomposition – decompose the problem of getting from A to B into independent XML in, XML out stages
Decide what transformation components you already have.
Implement the ones you don’t – make them re-usable for the next transformation project.

17. Pipeline philosophy Transformation analysis & decomposition leads to
a series of small, manageable, “stand alone” problems with an XML input “spec” and an XML output “spec”. “Spec” = schemas + structure rules + narrative.
Can build, test, use and then re-use these transformation components
Very team development friendly – parallel development of loosely coupled components
Very debugging friendly – log2(n) “chops” to find any given problem.

18. Pipeline debugging … Schema Delta 1 Schema Delta N Schema A Schema B
Input
XML
Output
XML
XML
Delta 1
XML
Delta N
Top Transformation
Tail Transformation
Non-XML
Input
Non-XML
Output

19. Pipeline philosophy
The answer to the SAX/DOM question is “mu”. (More on this later)
No such thing as “the” correct abstraction for processing XML
Pipeline approach means you can mix ‘n’match black-box components that internally use whatever paradigm best suited the problem
Lexical
SAX,STAX,DOM,XOM
COmega,XSLT, XQuery
XDuce, Pyxie, Java, C#, Groovy, Ruby, Haskell, WebIt! Etc. etc.

20. Sample Pipeline Lexical Lexical DOM Schematron/ RelaxNG/ Rhino JythonDB
/CMS
Character
Set Mods
Add
Doctype
+ validate
+ strip doctype
Lexical
Re-arrange
Elements
Validation
Lexical
DOM
Stats + FTP
Schematron/
RelaxNG/ Rhino
SQL
Replace
Jython
XHTML
Generate
Java
XSLT

21. Pipeline philosophy Many XML transformations end up monolithic
Assertion : developers would use a more component based approach to XML processing if they did not have to write the plumbing (orchestration, exception handling) themselves
“Gee, this problem is complex. Maybe I’ll do it in multiple stages! Gee, now I have to orchestrate the stages somehow. Batch files/shell scripts/driver program – all ugly and error prone. Maybe I’ll just write a single program after all. Besides, it will run faster...”

22. Pipeline philosophy
“Professional developers spend 50 percent of their time writing plumbing” – Adam Bosworth
Pipelining promotes the creation of a reusable plumbing “layer” letting developers concentrate on the application in hand.

23. Philosophy Summary
Think flow - data processing == data transformation w.r.t. time – Michael Jackson
XML is the current runaway winner in the self-descriptive data stakes and a very good IDDL (Intermediate Data Description Language) for all types of data that are not natively XML based

24. Philosophy Summary
Inside every complex XML transformation is a sequence of simpler XML transformations trying to get out – a pipeline
Decomposed transformation:
new transformations +
already componentized transformations
-> Component Reuse Nirvana

25. Pipeline Philosophy Level 2 – Rudimentary orchestration
Out
Level 2 – Rudimentary orchestration In
Out
Level 1 - pipeline In
Out
Level 0 – transformation component In
Out

26. Simple pipeline transformation component examples
Fundamental Operation – Rename Element
Rename
Input : <foo>baz</foo>
Output: <bar>baz</bar>
foo
bar
baz
baz

27. Simple pipeline transformation component examples
Fundamental Operation - Peel
Input : <foo><bar>baz</bar></foo>
Output: <foo>baz</foo>
foo
foo
bar
baz
baz

28. Simple pipeline transformation component examples
Compound Operation - Matryoshka
Input:
<foo><bar>baz</bar></foo>
Output:
<foo></foo><bar></bar>baz
foo
bar
foo
bar
baz
baz

29. Simple pipeline transformation component examples
KlingonCloak
Input:
<foo><bar>baz</bar></foo>
Output:
<tag name=“foo”><tag name=“bar”>baz</tag></tag>
foo
tag
type=“foo”
bar
tag type=“bar”
baz
baz

30. Simple pipeline transformation component examples
Reading a file is an XML to XML transformation
<file>lewisscarrol.xml</file>
<poem><line>Twas brillig, and the slithy tomes, did gyre and gimbal in the wave</line>…</poem>

31. Simple pipeline transformation component examples
Arithmetic is an XML to XML transformation
<expr>1 + 2</expr>
<res>3</res>

32. Simple pipeline transformation component examples
Unix pipe utilities e.g. tr
hello world
HELLO WORLD

33. A little orchestration in a transformation component
Conditionals are XML to XML transformation “tee junctions” triggered by XPaths
if XPath TRUE branch In
if XPath
if XPath FALSE branch

34. Validation as a transformation component
XML A
XML A’
RelaxNG
Schematron
Jython/Java/JACL
XComponent
Input
Output
Validation
Log
Error

35. Sample Transformation Component Examples
Once you start thinking in terms of pipes – components appear everywhere:
Regular fragmentations
Doctype changer
Namespace normalizer
Character set transcoder
Hash generator
Architectural form processing
RelaxNG/Schematron etc

36. First objection “It will be dog slow” or (stronger form):
“Re-usable tree transforming components won’t work in my shop – my XML files are too big to schlep around in strings, never mind DOMs!”

37. Document fulcra and the scatter/gather pattern
For any given transformation t to be performed on documents conforming to schema s, there is a fragment expression that can be used to chop each document into n pieces, on which t can be performed.
I call these points fulcra and are a function of (t,s)

38. Identifying Fulcra
For data-oriented XML, the fulcra often coincide with the “record” iteration in the XML schema and may be independent of t.
For document-oriented XML, the fulcra are much more dependent on t.

39. Document fulcra and scatter/gather pattern
Having identified the fulcra:-
Chop the input document into fragments – scatter phase
Perform t
Join all the processed fragments together to constitute the output document – gather phase
Three stage pipeline – scatter & gather either side of the core component

40. Document Fulcra Input Doc Scatter n fragments TIME Invoke t t t t t t
Gather
Output
Doc

41. Document Fulcra
Note the data domain de-composition – meets XML markup.
Trivially parallelizable 

42. Document Fulcra
A good fulcra based scatter/gather will make performance head north faster, cheaper and with a high upper limit than any amount of hand-crafted, genius level XML coding of your transformations in horrid SAX or lexical parse mode.
Massive Parallelism will kill all von Neumann throughput arguments
Documents per second, not seconds per document – throughput is the true measure of XML processing speed
Document fulcra – Locality of reference (Denning) applies to XML processing (more on this later)

43. More objections (with more answers)
It will be slow
No it won’t - Premature optimization is the root of all evil!
Speed is a three headed monster. I’m old enough to have left the X axis and currently heading for Y through Z
The 3 Axes to Speed

44. Some objections (with some answers)
Component based software? Harumph! We have heard that one before…
Pipelines are data flow based not API based (COM, VBX, CORBA)
Two pin interfaces and minimal “verbs”
The XML “payload” is what is important – not the API - RESTian

45. Revisiting the XSLT/DOM -> SAX non-sequiter
XSLT and DOM are memory bound – trade off between ease of use and resource usage – ease of use favoured
SAX is not memory bound – trade off between ease of use and resource usage – low resource usage favoured
On xml-dev users often advised to rewrite their apps using SAX! Ugh!

46. XSLT/DOM -> pipeline
Pipelines and scatter/gather allow you to keep the ease of use of XSLT/DOM with the finite resource utilization of SAX
As long as you can identify a good fulcrum function
They exist more often than not
If they exist, they are very easily found and “drop out” of document analysis – eg: xpath expressions in XSLT stylesheet templates

47. Pipelining and Grids
Grid Technologies – computational power “on tap” (
A match made in heaven (bandwidth permitting)

48. An XML Processing Grid – on demand
Out In
Out
DMZ

49. Grids - caveats
For large data volumes it is simple not feasible to shunt the data over the wire – Jim Gray
Organizations are sensitive about their data going beyond firewalls
Pay-per-use “racks” in your back-office a better bet. – Rent a grid the way you would rent a chainsaw.

50. A Service Oriented Architecture
“service” = XML transformation with side optional effects

51. Pipelines and Service Oriented Architecture
Can usefully blurr the distinction between a message queue and a transformation pipe
Services have the same XML-in, XML-out interface
All components can be services
All pipes can be services
All SOAs can be services…

52. Federated SOA’s
Pipeline transformation

53. Musings #1 - Debugging Pipelines are very debugging friendly
log2(N) time required for fault diagnosis
“Probes” in the form of loggers, RelaxNG validators, easily plug-inable (as transformation components) to a pipe to watch what is going on.
Pre/Post condition on/off switch is a useful “design by contract” debugger
XML-aware browsers as “breakpoints”

54. Musings #2 – Validation – grammers versus rules versus FYI’s
Pipelines make it natural to segregate “business rules” from “grammar rules” and can dramatically simplify both
Some of the most useful business “rules” are non dyadic. “FYIs” are really, really useful monitoring/QA tools.

55. Musings #3 – Inbetween-ing and component development
Transformation analysts spec the transformation
Only need to code new components
Spec == Documentation of what the transform needs to do with pre/post etc. but no code
Provides built in JIT-style acceptance test via the pre/post conditions
Outsource friendly, parallelisability friendly and third-party market friendly

56. Musing #4 - Web Services
First generation will be a total blind alley – RPC
Document Oriented Messaging – not Object Oriented Messaging -> SOAs
The next stage in encapsulation and loose coupling – something like pipelining will be a pre-requisite in a doc/literal world.

57. Musing #5 – naming and parametric typing
Naming components is a really hard problem
Programmers don’t do metadata 
Finding components to re-use is a real problem – the Google lesson
Numerous components that do the same thing but optimized on different axes:
Space
Time
Infoset considerations

58. Musing #6 – Pre-validation Transformation
Killing ourselves seeking one-shot expressivity in schema validation languages
Many complex validations become a lot simpler if you do some transformation(s) first
Co-occurrence constraints
Contextual constraints
Clear analog with formatting (pre-flow transformation(s) + flow = DSSSL/XSL)

59. Musing #7 – grids, scheduling and compilers
Scheduling transformations on a pipeline grid is hard – manufacturing lore needs to be brought to bear (e.g. Flow Shop Scheduling).
Pipe -> Component via compiler is a powerful idea
Both for grids (IO optimisation) and for general program distribution
Pipe compilation can beat the IO problems while retaining the simple, componentised development approach.
Back to the future with Jackson’s Program Inversion

60. Musing #8 – Higher order transformations
What if, instead of transforming an instance, you transformed a grammer?
Auto-generation of instance transformation primitives
Limited to non-PCDATA transforms and side-effect free transforms but useful nonetheless

61. Some pipeline-related open source technologies
| - Unix Pipes
SAX Filters
XBeans
Cocoon
Xpipe (sadly under resourced)
axKit
xvif
DSDL
Ant, W3C Pipeline Note

62. Thank you (question,answer?)*