Benchmarking on XQuery Implementation in DBMS

[Michael Kay]

A fair bit of work has been done but more is always needed.
 
I think you need to start out with very clear objectives, especially in the
following areas:
 
* are you going to try and measure how well the products handle something
close to a "real-life" workload (including factors such as updating,
database loading, etc), or are you going to measure a small number of
synthetic queries?
 
* are you going to make strenuous efforts to get the best possible
performance out of each product that you test, or are you simply going to
measure what it achieves "out of the box"?
 
* do you just want to produce some numbers, or do you want to use those
numbers to test some theories or hypotheses?
 
Most of the papers I've seen on XSLT, XPath, or XQuery benchmarks over the
years suffer from one or more of the following drawbacks:
 
* Some of the queries they measure are totally unrepresentative of real
life, for example IIRC one benchmark has a query like
/descendant::*/ancestor::*/descendant::*. Who cares whether that performs
well? (But it can be useful, of course, if your objective is to find out how
path expressions are implemented internally).
 
* Simple flaws in the measurement methodology, for example failing to take
account of Java VM startup time, or to distinguish query compilation,
document parsing, and query evaluation costs. Such flaws can make your
numbers meaningless, but they are surprisingly common in published papers.
 
* Running products in a sub-optimal way. This might say something about the
product's usability or documentation, but if you run with the wrong
configuration options, it tells you very little about the product's true
performance capabilities. This is especially true where your knowledge of
one of the products is much better than your knowledge of others: a factor
that taints all comparative benchmarks produced by vendors.
 
* Making incorrect inferences from the measurements (for example, the
authors of one paper drew inferences about the internal workings of Saxon
which were completely and bizarrely wrong, and they didn't check them with
me before publication.)
 
It's worth remembering that in real life, the performance you achieve
depends much more on your skills as a developer and your knowledge and
experience of the technology you are using than on the absolute capabilities
of the chosen product. Performance problems in real life don't occur because
the technology is intrinsically slow, they occur because it is misused. This
places a severe limitation on the usefulness of benchmark numbers.
 
Regards,

Michael Kay
http://www.saxonica.com/
http://twitter.com/michaelhkay 


  _____  

From: http://x-query.com/mailman/listinfo/talk [mailto:http://x-query.com/mailman/listinfo/talk] On Behalf
Of Wely Live
Sent: 29 October 2009 02:57
To: http://x-query.com/mailman/listinfo/talk
Subject:  Benchmarking on XQuery Implementation in DBMS


Hi,
 
I am planning to research the comparison and benchmarking XQuery
Implementation in DBMS (SQL Server, Oracle, DB2). There are a few question
to discuss as following:
1. I wonder if anybody has ever done the similar research?
    I would be appreciate if anybody could provide my any related source or
document.
2. To do the benchmarking, we of course need a sample dataset. Anybody has
reference where can I find a good dataset sample?
 
Thanks!
 
Regards,
Wely
Post Graduate Student
Nanyang Technological University

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://x-query.com/pipermail/talk/attachments/20091029/babab73b/attachment.htm