The test suite is located in the Web Service Description Working Group area of the W3C CVS repository.
| Host: | dev.w3.org |
|---|---|
| Repository path: | /sources/public |
| Directory: | /2002/ws/desc/test-suite |
The test suite contains test cases and reports. See W3C WSDL 2.0 Test Suite for a description.
The most common kind of test case is a document test case. This consists of a main WSDL 2.0 document plus any other WSDL 2.0 or XSD documents that it references. The document test cases are stored in the documents directory of the test suite.
All WSDL 2.0 documents in the test suite must be schema valid, that is, we are not interested in test cases that violate the schema because this type of error can be easily checked by a validating XML parser. Instead, we are interested in two types of WSDL 2.0 document, which we refer to as good or bad documents. These test cases are stored in the good and bad subdirectories.
A good document satisfies all the rules defined by the WSDL 2.0 specification. We are interested in building up a suite of good documents that cover all aspects of the specification.
In contrast, a bad document violates at least one rule. The rules defined by the specification are referred to as testable assertions and are marked up in the specification and summarized in appendices of Part 1 and Part 2. We are interested in building up a suite of bad documents that in total violate each testable assertion.
W3C WSDL 2.0 Test Suite contains brief instructions for contributing test cases. These are expanded here.
The simplest way to contribute a test case is to send an email describing the test case to the Working Group public mailing list, www-ws-desc@w3.org and attach a WSDL 2.0 document to it. However, the preferred way to contribute a test case is to generate a CVS patch for it. Check out the test suite from CVS, add your test case, generate a CVS patch, and attach that to your email instead of just the WSDL 2.0 document.
We are aiming to have complete coverage of all XML elements, attributes, and enumerated values that can occur in WSDL 2.0 documents. You can see how well the test suite covers these items by looking at the WSDL 2.0 Document Test Case Coverage Report. This report is generated by counting the matches of a list of XPath expressions on each good and bad document. Each XPath expression is assigned a red, yellow, or good status depending on the count. A count of zero corresponds to red status. A positive count of less then four is assigned yellow, and a count of four or more is assigned green. Our first priority is to have no red counts, so start there. Look for XPath expressions that have red status and construct good documents that match. After we get rid of all the reds, we'll focus on the yellows.
Note that this coverage report includes both good and bad documents. However, it is desirable to have good documents that cover all the XPath expressions since these can also serve as examples of correct usage to authors.
We are aiming to have complete coverage of all testable assertions. This means that for each testable assertion, we need a document that violates it. Look at the Assertion Coverage Report to see which assertions are already covered. Red status means no coverage, and green means one or more.
To create a bad document, it is often easiest to start with a good document and then modify it to violate the assertion.
Note that some assertions are optional, in which case the document is good. Optional assertions are suggestions or best practices. They correspond to the keywords SHOULD, SHOULD NOT, and RECOMMENDED in the specification.
The Dashboard.html report combines results of the various testing methodologies and is built using the build.xml Ant task. The Ant tasks that build the report summary have the following dependencies:
At this point you should be able to run build.xml->Dashboard.html to rebuild just the rollup report, or build.xml->main to schemavalidate and build the zip files, or results-message/build.xml->main to rebuild the message test results, or results/build.xml->main to rebuild the component model tests, or results/build.xml->Validation.html to build the validation test results. For a complete build of test results (sans test coverage), do all the above builds (in approximately reverse order.)
The report is produced by XSLT transforms that are invoked by an Ant script. These are described in detail in W3C WSDL 2.0 Test Suite Document Coverage Summary.
The test suite uses the WS-I XML format for message logs. See Logging Messages for the W3C WSDL 2.0 Test Suite for a description of where to get logging tools and how to use them.
The latest stable test suite is identified by a CVS tag. Editors of the test suite are responsible for tagging stable versions. For example, as of 10:32 AM, 2007-01-04, the latest stable test suite is tagged as:
v20070104T1532The recommended format for this tag is:
vYYYYMMDDThhmm
The date in UTC, e.g. 20070104
The time in UTC, e.g. 1532 (10:32 AM EST)
Note that UTC is used since we span many timezones.
The latest stable version tag is stored in the file test-suite.properties, e.g.:
latest.stable.version=v20070104T1532