Cart
[Home] [By Thread] [By Date] [Recent Entries]
Hi Mike,
On Jul 19, 2006, at 07:21, Michael Champion wrote:
> The conclusion is "At the time of writing of this first draft, it
> is too early to give conclusions drawn from the test results. This
> draft Note is being published to encourage review and comment as
> this work continues." I see the results of a lot of hard work, and
> that work seems to be accomplishing some of the unfinished business
> of the old Binary XML Characterization WG. Still, I don't see
> anything to disconfirm my a priori belief that it will be somewhere
> between difficult and impossible to to develop a Binary XML format
> that covers a wide range of use cases and yields a useful degree of
> compression and speedup (and all the other properties).
Well, it does say that it's too early to give conclusions so why
would you expect it to provide some? 1/2-)
The EXI WG has taken very seriously the feelings expressed by some
that the argument needs to be made with extensive numbers and all,
and that's what we're doing. There have been quite a few previous
studies on the fact that EXI works, but most of them compared just a
small number candidates (often two), looked into only a set of
specific aspects (often only compactness and speed), worked on
specific data sets, or on specific sets of use cases. Taken all
together there's enough to build more confidence than I have in many
specs that have been spawned (and no I won't reopen the XML Schema
discussion here :), but there's been a clear message that folks are
asking for more.
So we're trying our best to provide a much more comprehensive study,
but it's arduous work. Getting all candidate formats to run in the
same framework is trickier business than one would think, not because
it's experimental but because it's software (case in point: we've hit
problems with *gzip* and I don't think that's experimental in
anyone's book). We review test documents, metrics (for both documents
themselves and performance), results, and a bunch of other things.
It's taking time because the complexity of benchmarking on this scale
is a bunch more than the sum of benchmarking individual bits, but I
think it's time well spent.
The first draft of the Note that was just released provides (in our
opinion) enough information that people can start commenting on the
methods, and on very preliminary results. It's not meant to be
conclusive yet.
> So what am I missing? Is there a bunch of result data that I'm
> just not finding a pointer to?
The WG has a lot more data than what is currently being provided
because we're cross-checking it to make sure it's right. One thing
that we're doing is that we're having the Measurements Note in TR
space (where publication is slow and cautious) and a more up to date
analysis that we tend to edit weekly in the public group space at
http://www.w3.org/XML/EXI/report. Read it while taking all caveats
into account, and don't hesitate to send comments.
The latest runs of the results is always pointed to from http://
www.w3.org/XML/EXI/test-report. Currently the reason why the analysis
does not cover processing efficiency (and more importantly, its
relationship to compactness) is because we're still seeing strange
anomalies in the PE runs and are not completely confident in the
quality of the results (a new batch ought to be available soon with
fewer issues).
I don't know if I should just encourage folks to look at those pages
on a regular basis, or if notifying the WG's public list (public-
exi@w...) when there are updates is better. Suggestions welcome.
> I realize this effort isn't complete, but what evidence can be
> gleaned from this to support an argument that the EXI WG is on
> track to discover or produce a spec that will meet its objectives
> spelled out in http://www.w3.org/2005/09/exi-charter-final.html ?
We're on track to meeting those objectives, knowing that said
objectives are about demonstrating viability or the lack thereof. If
there were sufficient evidence at the level which is required of us
at this point we'd have published conclusions.
> Specifically, they are supposed to be 6 months from a Last Call
> working draft of a standard EXI format; does anyone think that is
> likely to happen?
The benchmarks have a dual purpose: proving viability and picking the
best candidate to start creating a spec from. I expect those two to
happen at pretty much the same moment. If the spec for that candidate
is good, getting to Last Call fast should be possible. That being
said the current thinking is that it's best to be late with the LC if
it means the benchmarking is better so some delay on that is not a
huge concern. The benchmarking also cuts out a lot of work out of CR
since we already have a framework for testing, a test methodology,
tests, etc..
--
Robin Berjon
Senior Research Scientist
Expway, http://expway.com/
|
