Getting Some Facts Straight on Performance Claims (Was: : [A

Given the provocative comment:

| XSL processing with EZ/X is usually 2-3
| times faster than Lotus/IBM/Apache or
| Oracle, and even faster than that
| when dealing with complex XML/XSL.

you can imagine it was fun to dig in and try
to see what was behind this claim...

When I discovered that the benchmark program
was running the EZ/X XSLT Engine with a
"NullOutputStream" instead of actually writing
out a file, I started to get suspicious...

When I saw that the Oracle test was not
using our latest 2.0.2.5 release and not
using the proper XSLProcessor.processXSL()
API to write output to a PrintWriter obeying
all the rules of <xsl:output>, I got a little
flustered...

When I saw in the "readme.txt" file for the
benchmarks the comment:

  | NOTE: The Oracle times will be very fast,
  |       deceptively so, since they are not
  |       doing much work.  The stylesheet is
  |       simply passed through because Oracle
  |       does not yet conform with the October
  |       1999 specification.

I knew this was just plain wrong, so decided
I had to roll up my sleeves and debunk
some of these statements.

To compare apples to apples, I modified the benchmark
to create a FileOutputStream in both cases and write
the transformed output to a file instead of to some
"magic" NullOutputStream. I used our latest release
from http://technet.oracle.com/tech/xml (which *is* 
compliant with the XSLT REC) and ran their
benchmarks again...

Here's the results I came up with below. As you can
see, while we don't win every single test, it is
certainly not the case that EZ/X is "2-3 times faster."

Once the test was modified to actually produce output
to a file, it was clear that their output documents
were appreciably larger in 3 of the 4 cases. I'd recommend
the EZ/X folks do their homework a little better before
making such claims in the future...

Overview
~~~~~~~~

Machine: Pentium III, Dell Optiplex, 400Mhz
JDK 1.2.2 that comes with JDeveloper 3.0

XST Engines Compared
--------------------
EZ/X Build 60 
Oracle XSLT Release 2.0.2.5

Benchmark Description
--------- ------------------------------
   1      Ken Holman's showtree-19991008.xsl
          on a 667 byte source document

   2      Mike Brown's Fancy_XML_Tree_Viewer_34.xsl
          on the same 667 byte source document 

   3      formatPlay.xsl 
          on a 2,096,966 byte source document
          produced by taking Jon Bosak's
          209,710 byte all_well.xml (Shakespeare's 
          "All's Well that Ends Well") and repeating
          it ten times inside the file.

   4      dict2.xsl
          on a 197,164 byte source document
          that is file 1 of 4 of Webster's
          Revised Unabridged Dictionary.

Performance Info
~~~~~~~~~~~~~~~~

Benchmark   EzX      OraXSL    Diff%
--------- --------  -------- -----------
   1         63ms      66ms   5%  slower  
   2        220ms     102ms   54% faster
   3      41764ms   28503ms   32% faster 
   4       1574ms    2005ms   27% slower

File Size Information
             
Benchmark   XML Src     EzX Out  OraXSL Out Diff%
--------- ----------  ---------- ---------- ------------
   1            667        2,323     1,658  29% smaller
   2            667       31,367     8,592  73% smaller 
   3      2,096,966    6,076,459 4,090,477  33% smaller   
   4        197,164      134,436   138,053   3% larger

===========================================================
Benchmark 1
===========================================================

E:\xml\ezx\examples\xsl\benchmark> run
Running model.xml with showtree-19991008.xsl 100 times...
xsl.processor property not set, using 'Ezx'
Testing EZ/XSL build 60 100 times...

After 10: 76.2ms
After 20: 62.0ms
After 30: 61.1ms
After 40: 62.1ms
After 50: 61.1ms
After 60: 62.1ms
After 70: 62.1ms
After 80: 61.1ms
After 90: 62.1ms
After 100: 61.1ms

Average transform time is 63ms

E:\xml\ezx\examples\xsl\benchmark> run_oracle
Running model.xml with showtree-19991008.xsl 100 times...
Testing Oracle XML Parser       2.0.2.5.0       Production 100 times...

After 10: 77.1ms
After 20: 65.1ms
After 30: 67.1ms
After 40: 65.1ms
After 50: 65.1ms
After 60: 64.1ms
After 70: 68.1ms
After 80: 65.1ms
After 90: 65.1ms
After 100: 65.1ms

Average transform time is 66ms

===========================================================
Benchmark 2
===========================================================
E:\xml\ezx\examples\xsl\benchmark> run2
Running model.xml with Fancy_XML_Tree_Viewer_34.xsl 100 times...
xsl.processor property not set, using 'Ezx'
Testing EZ/XSL build 60 100 times...

After 10: 233.4ms
After 20: 219.3ms
After 30: 219.3ms
After 40: 219.3ms
After 50: 220.3ms
After 60: 220.3ms
After 70: 218.4ms
After 80: 220.3ms
After 90: 219.3ms
After 100: 219.3ms

Average transform time is 220ms

E:\xml\ezx\examples\xsl\benchmark> run2_oracle
Running model.xml with Fancy_XML_Tree_Viewer_34.xsl 100 times...
Testing Oracle XML Parser       2.0.2.5.0       Production 100 times...

After 10: 118.2ms
After 20: 102.1ms
After 30: 100.2ms
After 40: 99.1ms
After 50: 100.2ms
After 60: 100.1ms
After 70: 102.1ms
After 80: 102.2ms
After 90: 101.1ms
After 100: 101.2ms

Average transform time is 102ms

===========================================================
Benchmark 3
===========================================================
E:\xml\ezx\examples\xsl\benchmark> run3
Running all_well_10x.xml with formatPlay.xsl 5 times...
xsl.processor property not set, using 'Ezx'
Testing EZ/XSL build 60 5 times...


Average transform time is 41764ms

E:\xml\ezx\examples\xsl\benchmark> run3_oracle
Running all_well_10x.xml with formatPlay.xsl 5 times...
Testing Oracle XML Parser       2.0.2.5.0       Production 5 times...


Average transform time is 28503ms

===========================================================
Benchmark 4
===========================================================
E:\xml\ezx\examples\xsl\benchmark> run4
Running sample200k.xml with dict2.xsl 50 times...
xsl.processor property not set, using 'Ezx'
Testing EZ/XSL build 60 50 times...

After 10: 1575.2ms
After 20: 1554.3ms
After 30: 1581.2ms
After 40: 1570.3ms
After 50: 1592.3ms

Average transform time is 1574ms

E:\xml\ezx\examples\xsl\benchmark> run4_oracle
Running sample200k.xml with dict2.xsl 50 times...
Testing Oracle XML Parser       2.0.2.5.0       Production 50 times...

After 10: 2042.9ms
After 20: 1990.9ms
After 30: 2013.9ms
After 40: 1988.8ms
After 50: 1991.9ms

Average transform time is 2005ms

_________________________________________________________
Steve Muench, Consulting Product Manager & XML Evangelist
Business Components for Java Development Team






 XSL-List info and archive:  http://www.mulberrytech.com/xsl/xsl-list