[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: How many unit tests should I create for my XML application

• From: Michael Kay <mike@saxonica.com>
• To: xml-dev@lists.xml.org
• Date: Sat, 24 Nov 2012 22:21:41 +0000

1. The schema-for-schemas has very little to do with the number of tests 
in the XSD validation suite. Firstly, the schema-for-schemas is only 
about determining whether schemas are valid, whereas most of the tests 
in the test suite are about determining whether XML instances are valid 
against a schema, which is quite outside the scope of the S4S. Secondly, 
there are many things that can make a schema "semantically" invalid even 
though it conforms to the grammar defined in the S4S. The S4S is thus 
only a small part of the specification under test, while the number of 
tests is related to the size of the specification as a whole.

2. The XSD test suite is the union of tests submitted by different 
member companies. There was no attempt to eliminate duplication of tests 
among the test submissions. (In fact, with some of the submissions, 
there is considerable internal duplication. This may have been caused by 
lack of communication between different test authors; in some cases I 
strongly suspect that test authors were measured on the number of tests 
they wrote (with the assumption more=better)).

3. Testing is never complete. Do you need to test obscure corner cases 
in your specification? I think that depends on the requirements of your 
project. The optimum level of test coverage, from a cost-benefit 
viewpoint, is probably less than 100%.

Michael Kay
Saxonica

On 24/11/2012 18:29, Costello, Roger L. wrote:
> Hi Folks,
>
> I am building an XML application. That is, my application consumes XML documents and then performs processing. The XML documents conform to an XML Schema.
>
> How many unit tests should I write for my XML application?
>
> To answer this question I looked at one other XML application. Specifically I looked at XML Schema validators.
>
> An XML Schema validator is an XML application -- an XML Schema validator consumes XML Schema documents (and XML instance documents) and then performs processing. The XML Schemas must conform to a schema-for-schemas.
>
> I downloaded the XML Schema Test Suite to see how many tests it contains and I discovered that it contains 14,285 unit tests.
>
> The schema-for-schemas is 2,374 lines. So there are 6 unit tests for each line of schema code. Is that a reasonable benchmark -- 6 units tests for each line of XML Schema?
>
> Thus, if my XML application consumes XML documents which conform to an XML Schema that is 100 lines long, then I should create 600 unit tests. Does that sound about right?
>
> /Roger
>
> _______________________________________________________________________
>
> XML-DEV is a publicly archived, unmoderated list hosted by OASIS
> to support XML implementation and development. To minimize
> spam in the archives, you must subscribe before posting.
>
> [Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
> Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
> subscribe: xml-dev-subscribe@lists.xml.org
> List archive: http://lists.xml.org/archives/xml-dev/
> List Guidelines: http://www.oasis-open.org/maillists/guidelines.php
>
>

• References:
  • How many unit tests should I create for my XML application?
    • From: "Costello, Roger L." <costello@mitre.org>