XProc Test Suite markup

Table of Contents

1 Introduction

The XProc Test Suite consists of a set of test suite documents and assorted ancillary files. The test suite documents are defined by the schema described in this document.

The test suite schema is normatively defined in RELAX NG Compact Syntax, but it isn’t necessary for an implementation to use this schema for validation.

Each individual test is defined by a test element. Tests can be grouped together with div elements and/or collected together in a test-suite. A test suite document must begin with one of these elements. All of the test suite elements are in the test suite namespace, see Namespaces. 

start = test | test-suite | test-div

2 Metadata

All tests, test suites, and divisions begin with a required metadata element named info. Metadata consists of a title, a simple string with inline markup, a revision-history that documents changes to the test, and optional specref elements. 

info =
    element t:info {
        title
        & revision-history
        & specref*
    }

2.1 Revision history

A revision history is a collection of one or more revisions. 

revision-history =
    element t:revision-history {
        revision+
    }

Revisions must be listed in reverse chronological order, so the most recent revision appears first. Each revision consists of a date, an author, and a description of the revision (see Description). 

revision =
    element t:revision {
        attribute initials { text }?,
        date,
        author?,
        description
    }

The date must be an ISO 8601 date or dateTime. An author has a name and an optional email address and uri. The description should describe what has been changed and why. 

author =
    element t:author {
        attribute initials { text }?
      & name?
      & email?
      & uri?
    }

2.1.1 Author initials

Repeating all of the author information in each revision can be tedious. The initials attribute can simplify the markup: it acts as an ID/IDREF link. The first time an author is mentioned, provide as much information as possible and mark that author with a (unique) set of initials. In other revisions, specify the author initials but leave the author element otherwise empty. For example, both of these revisions have the same authorship: 

<t:revision>
  <t:date>2020—01-04</t:date>
  <t:author initials="jd"/>
  <t:description xmlns="http://www.w3.org/1999/xhtml">
    <p>Corrected typo in test.</p>
  </t:description>
</t:revision>

<t:revision>
  <t:date>2019-12—31</t:date>
  <t:author initials="jd">
    <t:name>Jane Doe</t:name>
    <t:email>jane@example.com</t:email>
    <t:uri>https://example.com/jane/</t:uri>
  </t:author>
  <t:description xmlns="http://www.w3.org/1999/xhtml">
    <p>Initial checkin.</p>
  </t:description>
</t:revision>

2.2 Specref

If a test, or set of tests, relates to a particular aspect of a particular specification, a link can be provided. The link consists of a spec attribute that identifies the specification and a linkend attribute that must contain the ID value in the specification with which this test is associated. 

specref =
    element t:specref {
        attribute spec { "xproc" | "steps" }?
      & attribute linkend { xsd:NMTOKEN }
      & empty
    }

(It’s not clear if this turned out to be as useful in practice as it was in theory.)

3 Description

The description can contain any mixture of text and HTML elements. HTML elements must be in the HTML namespace: http://www.w3.org/1999/xhtml.

4 Test suites and divisions

The test-suite and div elements serve to group tests together. 

test-suite =
    element t:test-suite {
        testattr,
        (info, description?, property*, test*, test-div*)
    }

Divisions may be nested. 

test-div =
    element t:div {
        (info, description?, property*, test*, test-div*)
    }

5 Common attributes

Each test (and test-suite) has attributes to identify the features it requires and the circumstances under which it should be run. 

testattr = attribute features {
               list { xsd:token+ }
           } ?
           & attribute when { text }?
           & attribute src { text }?
           & attribute version { text }?

The attributes should be interpreted as follows:

  • features is a set of one or more feature tokens. Feature tokens identify features that the implementation must support in order for this test to pass. The test should be skipped if it identifies any unsupported features.

    The set of feature tokens is a bit ad hoc. Here’s a summary of the feature tokens at the time this document was written:

    • archive-order, ???
    • dtd-id-ref-warning, a RELAX NG validator capable of reporting DTD-based ID/IDREF warnings is required.
    • lazy-eval, lazy evaluation of variable bindings is required.
    • p-validate-with-relax-ng, a RELAX NG validator is required.
    • p-validate-with-schematron, a Schematron validator is required.
    • p-validate-with-xsd, a W3C XML Schema validator is required.
    • webaccess, access to the web is required (this test cannot be run offline).
    • xquery_1_0, an XQuery 1.0 processor is required.
    • xquery_3_0, an XQuery 3.0 processor is required.
    • xquery_3_1, an XQuery 3.1 processor is required.
    • xslt-1, an XSLT 1.0 processor is required.
    • xslt-1-output-base-uri, ???
    • xslt-2, an XSLT 2.0 processor is required.
    • xslt-3, an XSLT 3.0 processor is required.
    • xslt-serialization, ???
  • when specifies an XPath expression that can be evaluated statically by the test processor. If the effective boolean value of the result is false(), the test is ignored.
  • src specifies a URI where the test (or test-suite) is located. If this attribute is specified, the element on which it occurs must be empty.
  • version specifies the version of the XProc processor required for this test.

6 Tests

There are two kinds of tests, passing tests and failing tests. 

test = passingTest | failingTest

Passing and failing tests are mostly the same. Tests have common attributes, metadata, and a description. The body of the test in each case consists of property, input, output, and pipeline elements.

A passing test has an expected attribute (that must have the value be pass) and may have schematron tests to verify the results. 

passingTest =
    element t:test {
        testattr
      & attribute expected { "pass" }
      & attribute platform { ("Linux" | "MacOS" | "Windows") }?
      & (info, description?,
         (property* & input* & option* & pipeline? & schematron?))
    }

A failing test has an expected attribute (that must have the value be fail) and must specify one or more expected error conditions. 

failingTest =
    element t:test {
        testattr
      & attribute expected { "fail" }
      & attribute code { xsd:NMTOKENS }
      & attribute platform { ("Linux" | "MacOS" | "Windows") }?
      & (info, description?,
         (property* & input* & option* & pipeline?))
    }

Broadly speaking, the test harness sets the specified properties and runs the specified pipeline with the specified inputs and options. A passing test must complete without error, or it must be recorded as a failure. If it completes, its output is tested against any Schematron rules provided. No assertions must be raised.

The platform attribute on a test marks this test to be run on the named platform only and to be skipped otherwise. The attribute's value is expected to be one of “Linux”, “MacOS”, or “Windows”.

If a failing test raises one of the specified error codes, it is a “pass” from the perspective of the test suite, otherwise it must be recorded as a failure.

Like tests and test-suites, inputs, pipelines, and Schematron rules can be specified inline or with a src attribute that points to their content.

6.1 Properties

A property is a name/value pair. These must be made available to the implementation, but the precise mechanism is necessarily implementation defined. 

property =
    element t:property {
        attribute name { text }
      & attribute value { text }
    }

On the Java platform, these are system properties.

6.2 Inputs

Each input element identifies the input for a source port on the pipeline. XML and text documents can be placed inline; other media types must be stored externally and identified by URI.

If several inputs identify the same port, then the sequence of documents identified appears on that port, in document order of the input elements in the test. 

input =
    element t:input {
        attribute port { text }
      & attribute src { text }?
      & any*
    }

6.3 Options

Each option element identifies the value for a pipeline option. If a select attribute is provided, it must be an XPath expression. Evaluating that expression gives the value of the option. If the select attribute is not provided, the contents of the element is the value of the option. 

option =
    element t:option {
        attribute name { text }
      & attribute select { text }?
      & any*
    }

6.4 Pipeline

The pipeline element, which can only occur once, provides the pipeline to be tested. The pipeline must be written so that it has only a single, primary output port named result. 

pipeline =
    element t:pipeline {
        attribute src { text }?
      & any*
    }

6.5 Schematron

The schematron element contains Schematron rules. Assertions in the Schematron are tested against the results of the pipeline. 

schematron =
    element t:schematron {
        attribute src { text }?
      & any*
    }

7 Namespaces

The following namespaces are defined in this schema. 

namespace rng  = "http://relaxng.org/ns/structure/1.0"
namespace t = "http://xproc.org/ns/testsuite/3.0"
namespace h = "http://www.w3.org/1999/xhtml"
default namespace = "http://xproc.org/ns/testsuite/3.0"

8 Relax NG Schema

These fragments are stitched together (with a few other fragments) into a complete schema. 

The following namespaces are defined in this schema.

#+NAME: namespaces
#+BEGIN_SRC rnc :noweb yes
namespace rng  = "http://relaxng.org/ns/structure/1.0"
namespace t = "http://xproc.org/ns/testsuite/3.0"
namespace h = "<<html_namespace>>"
default namespace = "http://xproc.org/ns/testsuite/3.0"
#+END_SRC

#+NAME: html_namespace
#+BEGIN_SRC rnc :exports none
http://www.w3.org/1999/xhtml
#+END_SRC


start = test | test-suite | test-div

testattr = attribute features {
               list { xsd:token+ }
           } ?
           & attribute when { text }?
           & attribute src { text }?
           & attribute version { text }?

test-suite =
    element t:test-suite {
        testattr,
        (info, description?, property*, test*, test-div*)
    }

passingTest =
    element t:test {
        testattr
      & attribute expected { "pass" }
      & (info, description?,
         (property* & input* & option* & pipeline? & schematron?))
    }

failingTest =
    element t:test {
        testattr
      & attribute expected { "fail" }
      & attribute code { xsd:NMTOKENS }
      & (info, description?,
         (property* & input* & option* & pipeline?))
    }

test-div =
    element t:div {
        (info, description?, property*, test*, test-div*)
    }

info =
    element t:info {
        title
        & revision-history
        & specref*
    }

author =
    element t:author {
        attribute initials { text }?
      & name?
      & email?
      & uri?
    }

name = element t:name { text }
email = element t:email { text }
uri = element t:uri { xsd:anyURI }


A revision history is a collection of one or more revisions.

#+NAME: revision-history
#+BEGIN_SRC rnc
revision-history =
    element t:revision-history {
        revision+
    }
#+END_SRC

Revisions /must/ be listed in reverse chronological order, so the most
recent revision appears first. Each revision consists of a ~date~, an ~author~,
and a ~description~ of the revision (see [[*Description][Description]]).

#+NAME: revision
#+BEGIN_SRC rnc
revision =
    element t:revision {
        attribute initials { text }?,
        date,
        author?,
        description
    }
#+END_SRC

The ~date~ must be an ISO 8601 /date/ or /dateTime/. An ~author~
has a ~name~ and an optional ~email~ address and ~uri~. The ~description~
should describe what has been changed and why.

#+NAME: author
#+BEGIN_SRC rnc
author =
    element t:author {
        attribute initials { text }?
      & name?
      & email?
      & uri?
    }
#+END_SRC

*** Author initials
:PROPERTIES:
:CUSTOM_ID: author-initials
:END:

Repeating all of the author information in each revision can be
tedious. The ~initials~ attribute can simplify the markup: it acts as
an ID/IDREF link. The first time an author is mentioned, provide as
much information as possible and mark that author with a (unique) set
of initials. In other revisions, specify the author initials but leave
the author element otherwise empty. For example, both of these
revisions have the same authorship:

#+BEGIN_SRC xml
    <t:revision>
      <t:date>2020—01-04</t:date>
      <t:author initials="jd"/>
      <t:description xmlns="http://www.w3.org/1999/xhtml">
        <p>Corrected typo in test.</p>
      </t:description>
    </t:revision>

    <t:revision>
      <t:date>2019-12—31</t:date>
      <t:author initials="jd">
        <t:name>Jane Doe</t:name>
        <t:email>jane@example.com</t:email>
        <t:uri>https://example.com/jane/</t:uri>
      </t:author>
      <t:description xmlns="http://www.w3.org/1999/xhtml">
        <p>Initial checkin.</p>
      </t:description>
    </t:revision>
#+END_SRC


revision =
    element t:revision {
        attribute initials { text }?,
        date,
        author?,
        description
    }

title =
    element t:title { text }

date =
    element t:date { xsd:date|xsd:dateTime }


If a test, or set of tests, relates to a particular aspect of a
particular specification, a link can be provided. The link consists of
a ~spec~ attribute that identifies the specification and a ~linkend~
attribute that must contain the ID value in the specification with
which this test is associated.

#+NAME: specref
#+BEGIN_SRC rnc
specref =
    element t:specref {
        attribute spec { "xproc" | "steps" }?
      & attribute linkend { xsd:NMTOKEN }
      & empty
    }
#+END_SRC

(It’s not clear if this turned out to be as useful in practice as it was in theory.)


test = passingTest | failingTest


The ~pipeline~ element, which can only occur once, provides the pipeline
to be tested. The pipeline must be written so that it has only a single, primary
output port named ~result~.

#+NAME: pipeline
#+BEGIN_SRC rnc
pipeline =
    element t:pipeline {
        attribute src { text }?
      & any*
    }
#+END_SRC



The ~schematron~ element contains Schematron rules. Assertions in the
Schematron are tested against the results of the pipeline.

#+NAME: schematron
#+BEGIN_SRC rnc
schematron =
    element t:schematron {
        attribute src { text }?
      & any*
    }
#+END_SRC


input =
    element t:input {
        attribute port { text }
      & attribute src { text }?
      & any*
    }

option =
    element t:option {
        attribute name { text }
      & attribute select { text }?
      & any*
    }

any =
    element * {
        attribute * { text }*
      & (text | any)*
    }

anyhtml =
    element h:* {
        attribute * { text }*
      & (text | anyhtml)*
    }

description =
    element t:description {
        anyhtml+
    }

property =
    element t:property {
        attribute name { text }
      & attribute value { text }
    }