XProc: An XML Pipeline Language

2.1 Steps

[Definition: A step is the basic computational unit of a pipeline.] All steps have a name; if the pipeline author does not provide a name for a step, a default name is manufactured automatically.

Steps are either atomic or compound. [Definition: An atomic step is a step that performs a unit of XML processing, such as XInclude or transformation, and has no internal subpipeline.] Atomic steps carry out fundamental XML operations and can perform arbitrary amounts of computation, but they are indivisible. An XSLT step, for example, performs XSLT processing; an XML Schema Validation step validates one input with respect to some set of XML Schemas, etc.

There are many types of atomic steps. The standard library of atomic steps is described in Section 7, “Standard Step Library”, but implementations may provide others as well. What additional step types, if any, are provided is implementation-defined. Each use, or instance, of an atomic step invokes the processing defined by that type of step. A pipeline may contain instances of many types of steps and many instances of the same type of step.

Compound steps, on the other hand, control and organize the flow of documents through a pipeline, reconstructing familiar programming language functionality such as conditionals, iterators and exception handling. They contain other steps, whose evaluation they control.

[Definition: A compound step is a step that contains one or more subpipelines. That is, a compound step differs from an atomic step in that its semantics are at least partially determined by the steps that it contains.] Every compound step contains one or more subpipelines. [Definition: The steps that occur directly inside a compound step are called contained steps.] [Definition: A compound step which immediately contains another step is called its container.]

This simple distinction between atomic and compound steps is occasionally stretched. The immediate children of some compound steps, e.g. p:choose and p:try, are special. In the case of p:choose, the p:when and p:otherwise elements serve as wrappers around different pipelines at most one of which will be processed. In the case of p:try, the p:catch element is a wrapper around a subpipeline that will only be processed if the initial p:group fails. Acknowledging this slight irregularity, we nevertheless treat all compound steps as if they directly contained one or more subpipelines.

[Definition: The steps (and the connections between them) within a compound step form a subpipeline.] [Definition: The last step in a subpipeline is the last step in document order within its container. ]

subpipeline = (p:for-each|p:viewport|p:choose|p:group|p:try|pfx:other-step|p:documentation|ipfx:ignored)*

A compound step can contain one or more subpipelines and it determines how and which of its subpipelines are evaluated.

Steps have “ports” into which inputs and outputs are connected or “bound”. Each step has a number of input ports and a number of output ports; a step can have zero input ports and/or zero output ports. (All steps have an implicit output port for reporting errors that must not be declared.) The names of all ports on each step must be unique on that step (you can't have two input ports named “source”, nor can you have an input port named “schema” and an output port named “schema”).

Steps have any number of options, all with unique names. A step can have zero options.

Steps have parameter input ports, on which parameters can be passed. The parameters passed on a particular parameter input port must be uniquely named. If multiple parameters with the same name are used, only one of the values will actually be available to the step. A step can have zero parameter input ports, and each parameter port can have zero parameters passed on it.

2.2 Inputs and Outputs

Although some steps can read and write non-XML resources, what flows between steps through input ports and output ports are exclusively XML documents or sequences of XML documents.

An implementation may make it possible for a step to produce non-XML output (through channels other than a named output port)—for example, writing a PDF document to a URI—but that output cannot flow through the pipeline. Similarly, one can imagine a step that takes no pipeline inputs, reads a non-XML file from a URI, and produces an XML output. But the non-XML data cannot arrive on an input port to a step.

It is a dynamic error (err:XD0001) if a non-XML resource is produced on a step output or arrives on a step input.

The common case is that each step has one or more inputs and one or more outputs. Figure 3, “An atomic step” illustrates symbolically an atomic step with two inputs and one output.

Figure 3. An atomic step

All atomic steps are defined by a p:declare-step. The declaration of an atomic step defines the input ports, output ports, and options of all steps of that type. For example, every p:xslt step has two inputs, named “source” and “stylesheet”, and one output named “result” and the same set of options.

The situation is slightly more complicated for compound steps because they don't have separate declarations; each instance of a compound step serves as its own declaration. Compound steps don't have declared inputs, but they do have declared outputs, and unlike atomic steps, on compound steps, the number and names of the outputs can be different on each instance of the step.

Figure 4, “A compound step” illustrates symbolically a compound step with one output. As you can see from the diagram, the output from the compound step comes from one of the outputs of the subpipeline within the step.

Figure 4. A compound step

[Definition: The input ports declared on a step are its declared inputs.] [Definition: The output ports declared on a step are its declared outputs.] When a step is used in a pipeline, it is connected to other steps through its inputs and outputs.

When a step is used, all of the declared inputs of the step must be connected. Each input can be connected to:

When an input accepts a sequence of documents, the documents can come from any combination of these locations.

The declared outputs of a step may be connected to:

The primary output port of a step must be connected, but other outputs can remain unconnected. Any documents produced on an unconnected output port are discarded.

Output ports on compound steps have a dual nature: from the perspective of the compound step's siblings, its outputs are just ordinary outputs and must be connected as described above. From the perspective of the compound step itself, they are inputs into which something must be connected.

Within a compound step, the declared outputs of the step can be connected to:

Each input and output is declared to accept or produce either a single document or a sequence of documents. It is not an error to connect a port that is declared to produce a sequence of documents to a port that is declared to accept only a single document. It is, however, an error if the former step actually produces more than one document at run time.

[Definition: The signature of a step is the set of inputs, outputs, and options that it is declared to accept.] Each atomic step (e.g. XSLT or XInclude) has a fixed signature, declared globally or built-in, which all its instances share, whereas each compound step has its own implicit signature.

[Definition: A step matches its signature if and only if it specifies an input for each declared input, it specifies no inputs that are not declared, it specifies an option for each option that is declared to be required, and it specifies no options that are not declared.] In other words, every input and required option must be specified and only inputs and options that are declared may be specified. Options that aren't required do not have to be specified.

Steps may also produce error, warning, and informative messages. These messages appear on a special “error output” port. The error output port is only bound to an input in the catch clause of a try/catch. Outside of a try/catch, the disposition of error messages is implementation-dependent.

2.3 Primary Inputs and Outputs

As a convenience for pipeline authors, each step may have one input port designated as the primary input port and one output port designated as the primary output port.

[Definition: If a step has a document input port which is explicitly marked “primary='true'”, or if it has exactly one document input port and that port is not explicitly marked “primary='false'”, then that input port is the primary input port of the step.] If a step has a single input port and that port is explicitly marked “primary='false'”, or if a step has more than one input port and none is explicitly marked as the primary, then the primary input port of that step is undefined.

[Definition: If a step has a document output port which is explicitly marked “primary='true'”, or if it has exactly one document output port and that port is not explicitly marked “primary='false'”, then that output port is the primary output port of the step.] If a step has a single output port and that port is explicitly marked “primary='false'”, or if a step has more than one output port and none is explicitly marked as the primary, then the primary output port of that step is undefined.

The special significance of primary input and output ports is that they are connected automatically by the processor if no explicit binding is given. Generally speaking, if two steps appear sequentially in a subpipeline, then the primary output of the first step will automatically be connected to the primary input of the second.

Additionally, if a p:pipeline has no declared inputs and the first step in its subpipeline has an unbound primary input, then an implicit primary input port (named “source”) will be added to the p:pipeline (and consequently bound to the first step's primary input port). If a compound step has no declared outputs and the last step in its subpipeline has an unbound primary output, then an implicit primary output port (named “result”) will be added to the compound step (and consequently the last step's primary output will be bound to it).

The practical consequence of these rules is that straightforward, linear pipelines are much simpler to read, write, and understand. The following pipeline has a single input which is transformed by the XSLT step; the result of that XSLT step is the result of the pipeline:

<p:pipeline xmlns:p="http://www.w3.org/ns/xproc">
<p:xslt>
  <p:input port="stylesheet">
    <p:document href="docbook.xsl"/>
  </p:input>
</p:xslt>
</p:pipeline>

It is semantically equivalent to this pipeline:

<p:pipeline name="main" xmlns:p="http://www.w3.org/ns/xproc">
<p:input port="source"/>
<p:input port="parameters" kind="parameter"/>
<p:output port="result">
  <p:pipe step="transform" port="result"/>
</p:output>

<p:xslt name="transform">
  <p:input port="source">
    <p:pipe step="main" port="source"/>
  </p:input>
  <p:input port="stylesheet">
    <p:document href="docbook.xsl"/>
  </p:input>
  <p:input port="parameters">
    <p:pipe step="main" port="parameters"/>
  </p:input>
</p:xslt>

</p:pipeline>

(Parameter input ports are a special case discussed in Section 2.5, “Parameters”.)

2.4 Options

Some steps accept options. Options are name/value pairs.

[Definition: An option is a name/value pair where the name is an expanded name and the value must be a string.] If a document, node, or other value is given, its [XPath 1.0] string value is computed and that string is used.

[Definition: The options declared on a step are its declared options.] All of the options specified on an atomic step must have been declared. Option names are always expressed as literal values, pipelines cannot construct option names dynamically.

[Definition: The options on a step which have specified values, either because a p:option element specifies a value or because the declaration included a default value, are its specified options.]

2.5 Parameters

Some steps accept parameters. Parameters are name/value pairs.

[Definition: A parameter is a name/value pair where the name is an expanded name and the value must be a string.] If a document, node, or other value is given, its [XPath 1.0] string value is computed and that string is used.

Unlike options, which have names known in advance to the pipeline, parameters are not declared and their names may be unknown to the pipeline author. Pipelines can dynamically construct sets of parameters. Steps can read dynamically constructed sets on parameter input ports.

[Definition: A parameter input port is a distinguished kind of input port which accepts (only) dynamically constructed parameter name/value pairs.] See Section 5.1.2, “Parameter Inputs”.

Analogous to primary input ports, steps that have parameter inputs may designate at most one parameter input port as a primary parameter input port.

[Definition: If a step has a parameter input port which is explicitly marked “primary='true'”, or if it has exactly one parameter input port and that port is not explicitly marked “primary='false'”, then that parameter input port is the primary parameter input port of the step.] If a step has a single parameter input port and that port is explicitly marked “primary='false'”, or if a step has more than one parameter input port and none is explicitly marked as the primary, then the primary parameter input port of that step is undefined.

Additionally, if a p:pipeline does not declare any parameter input ports, but contains a step which has a primary parameter input port, then an implicit primary parameter input port (named “parameters”) will be added to the pipeline. (If the pipeline declares an ordinary input named “parameters”, the implicit primary parameter input port will be named “parameters1”. If that's not available, then “parameters2”, etc. until an available name is found.)

How an implementation maps parameters specified to the application, or through some API, to parameters accepted by the p:pipeline is implementation-defined.

2.6 Connections

Steps are connected together by their input ports and output ports. It is a static error (err:XS0001) if there are any loops in the connections between steps: no step can be connected to itself nor can there be any sequence of connections through other steps that leads back to itself.

2.7 Environment

[Definition: The environment of a step is the static information available to each instance of a step in a pipeline.]

The environment consists of:

[Definition: The empty environment contains no readable ports, no in-scope options, and an undefined default readable port. ]

Unless otherwise specified, the environment of a contained step is its inherited environment. [Definition: The inherited environment of a contained step is an environment that is the same as the environment of its container with the standard modifications. ]

The standard modifications made to an inherited environment are:

A step with no parent inherits the empty environment.

2.8 XPath Context

XProc uses [XPath 1.0] as an expression language. XPath expressions can occur in several places: on compound steps, in the expressions used to compute option and parameter values, and in values passed to atomic steps.

Broadly, these can be divided into two classes: expressions evaluated by the XProc processor and expressions evaluated by the implementations of individual steps.

This distinction can be seen in the following example:

<p:option name="home" value="http://example.com/docs"/>

<p:load name="read-from-home">
  <p:option name="href" select="concat($home,'/document.xml')"/>
</p:load>

<p:split-sequence name="select-chapters">
  <p:input port="source" select="//section"/>
  <p:option name="test" value="@role='chapter'"/>
</p:split-sequence>

The href option of the p:load step step is evaluated by the XProc processor. The actual href option received by the step is simply the string literal “http://example.com/docs/document.xml”. (The selection on the source input of the select-chapters step is also evaluated by the XProc processor.)

Conversely, the XPath expression “@role='chapter'” is passed literally to the test option on the p:split-sequence step. That's because the nature of the p:split-sequence is that it evaluates the expression. Only some options on some steps expect XPath expressions.

The XProc processor evaluates all of the XPath expressions in select attributes on steps, options, parameters, and inputs and in test attributes on p:when steps. (XPath expressions in value attributes are passed literally to the step for evaluation.)

3.1 XProc Namespaces

The XML syntax for XProc uses three namespaces:

3.2 Scoping of Names

The scope of the names of the step types is the union of all the pipelines and pipeline libraries available directly or via p:import.

Step types are:

All the step types in a pipeline must have unique names: it is a static error (err:XS0036) if any step type name is built-in and/or declared or defined more than once in the same scope.

The scope of the names of the steps themselves is determined by the environment of each step. In general, the name of a step, the names of its sibling steps, the names of any steps that it contains directly, the names of its ancestors, and the names of its ancestor's siblings are all in the same scope. All the named steps in the same scope must have unique names: it is a static error (err:XS0002) if two steps with the same name appear in the same scope.

The scope of an input or output port name is the step on which it is defined. The names of all the ports on any step must be unique.

Taken together, these uniqueness constraints guarantee that the combination of a step name and a port name uniquely identifies exactly one port on exactly one in-scope step.

The scope of option names is the step on which they occur and the descendants of that step. The names of all of the options specified on a step must be unique. If a step specifies a value for an option with the same name as some option specified on one of its ancestors, the new value shadows the previous value on the current step and its descendants.

Parameter names are not scoped; they are distinct on each step.

3.3 Global Attributes

The following attributes may appear on any element in a pipeline:

3.4 Associating Documents with Ports

[Definition: A binding associates an input or output port with some data source.] A document or a sequence of documents can be bound to a port in four ways: by source, by URI, by providing an inline document, or by making it explicitly empty. Each of these mechanisms is allowed on the p:input, p:output, p:xpath-context, p:iteration-source, and p:viewport-source elements.

Note that a p:input or p:output element may contain more than one p:pipe, p:document, or p:inline element. If more than one binding is provided, then the specified sequence of documents is made available on that port in the same order as the bindings.

3.5 Documentation

Pipeline authors may add documentation to their pipeline documents with the p:documentation element. Except when it appears as a descendant of p:inline, the p:documentation element is completely ignored by pipeline processors, it exists simply for documentation purposes. (If a p:documentation is provided as a descendant of p:inline, it has no special semantics, it is treated literally as part of the document to be provided on that port.)

Pipeline processors that inspect the contents of p:documentation elements and behave differently on the basis of what they find are not conformant. Processor extensions must be specified with extension elements.

3.6 Ignored namespaces

In order to facilitate extension elements, the processor can be instructed to ignore elements from selected namespaces. [Definition: Any element in an ignored namespace is an ignorable element.]

If a processor encounters an ignorable element as the child of a p:pipeline or p:pipeline-library then it behaves in an implementation-defined manner if it recognizes the element, otherwise it must behave as if the element (and its content) had not been present.

Syntactically, a pipeline author can specify the set of ignored namespaces with the ignore-prefixes attribute. This attribute can appear on the p:pipeline and p:pipeline-library elements.

The value of the ignore-prefixes attribute is a sequence of tokens, each of which must be the prefix of an in-scope namespace. It is a static error (err:XS0005) if any token specified in the prefix list is not the prefix of an in-scope namespace.

Ignored namespaces specified on a p:pipeline-library are inherited by pipelines that occur within that library.

It is a static error (err:XS0015) to specify as an ignored namespace the XProc namespace, the namespace of any imported p:pipeline, or any namespace in which an atomic step is declared.

3.7 Extension attributes

[Definition: An element from the XProc namespace may have any attribute not from the XProc namespace, provided that the expanded-QName of the attribute has a non-null namespace URI. Such an attribute is called an extension attribute.] Extension attributes are always allowed and do not have to be declared with ignored namespaces.

The presence of an extension attribute must not cause the connections between steps to differ from the connections that any other conformant XProc processor would produce. They must not cause the processor to fail to signal an error that a conformant processor is required to signal. This means that an extension attribute must not change the effect of any XProc element except to the extent that the effect is implementation-defined or implementation-dependent.

A processor which encounters an extension attribute that it does not recognize must behave as if the attribute was not present.

3.8 Extension elements

[Definition: An extension element is any element that is not in the XProc namespace and is not a step.] The presence of an extension element must not cause the connections between steps to differ from the connections that any other conformant XProc processor would produce. They must not cause the processor to fail to signal an error that a conformant processor is required to signal. This means that an extension element must not change the effect of any XProc element except to the extent that the effect is implementation-defined or implementation-dependent.

An element is only an extension element if it is an ignorable element that occurs as a direct child of a p:pipeline or p:pipeline-library.

In other words, elements in a subpipeline are interpreted as follows:

3.9 Syntax Summaries

The description of each element in the pipeline namespace is accompanied by a syntactic summary that provides a quick overview of the element's syntax:

<p:some-element
  some-attribute? = some-type>
    (some |
     elements |
     allowed)*,
    other-elements?
</p:some-element>

For clarity of exposition, some attributes and elements are elided from the summaries:

The types given for attributes should be understood as follows:

A number of errors apply generally:

If an XProc processor can determine statically that a dynamic error will always occur, it may report that error statically.

4.1 p:pipeline

A pipeline is specified by the p:pipeline element. It encapsulates the behavior of a subpipeline. Its children declare the inputs, outputs, and options that the pipeline exposes and identify the steps in its subpipeline.

A pipeline can declare additional steps (e.g., ones that are provided by a particular implementation or in some implementation-defined way) and import other pipelines. If a pipeline has been imported, it may be invoked as a step within the pipeline that imported it.

<p:pipeline
  name? = NCName
  type? = QName
  ignore-prefixes? = prefix list>
    (p:input |
     p:output |
     p:option |
     p:import |
     p:declare-step |
     p:log |
     p:serialization)*,
    subpipeline
</p:pipeline>

Viewed from the outside, a p:pipeline is a black box which performs some calculation on its inputs and produces its outputs. From the pipeline author's perspective, the computation performed by the pipeline is described in terms of contained steps which read the pipeline's inputs and produce the pipeline's outputs.

The environment inherited by the contained steps of a p:pipeline is the empty environment with these modifications:

If the p:pipeline has a primary output port and that port has no binding, then it is bound to the primary output port of the last step in the subpipeline. It is a static error (err:XS0006) if the primary output port has no binding and the last step in the subpipeline does not have a primary output port.

There are two additional constraints on pipelines:

If the pipeline initially invoked by the processor has inputs or outputs, those ports are bound to documents outside of the pipeline in an implementation-defined manner.

If a pipeline has a type then that type may be used as the name of a step to invoke the pipeline. This most often occurs when the it has been imported into another pipeline, but pipelines may also invoke themselves recursively. If it does not have a type, then its name is used to invoke it as a step.

For pipelines that are part of a p:pipeline-library, see Section 5.9, “p:pipeline-library” for more details on how p:pipeline names are used to compute step names.

<p:pipeline name="pipeline" xmlns:p="http://www.w3.org/ns/xproc">
<p:input port="document" primary="true"/>
<p:input port="stylesheet"/>
<p:output port="result" primary="true"/>

<p:xinclude/>

<p:validate-xml-schema>
  <p:input port="schema">
    <p:document href="http://example.com/path/to/schema.xsd"/>
  </p:input>
</p:validate-xml-schema>

<p:xslt>
  <p:input port="stylesheet">
    <p:pipe step="pipeline" port="stylesheet"/>
  </p:input>
</p:xslt>

</p:pipeline>

4.2 p:for-each

A for-each is specified by the p:for-each element. It processes a sequence of documents, applying its subpipeline to each document in turn.

<p:for-each
  name? = NCName>
    (p:iteration-source?,
     (p:output |
      p:option |
      p:log)*,
     subpipeline)
</p:for-each>

When a pipeline needs to process a sequence of documents using a subpipeline that only processes a single document, the p:for-each construct can be used as a wrapper around that subpipeline. The p:for-each will apply that subpipeline to each document in the sequence in turn.

The result of the p:for-each is a sequence of documents produced by processing each individual document in the input sequence. If the p:for-each has one or more output ports, what appears on each of those ports is the sequence of documents that is the concatenation of the sequence produced by each iteration of the loop on the port to which it is connected.

The p:iteration-source is an anonymous input: its binding provides a sequence of documents to the p:for-each step. If no iteration sequence is explicitly provided, then the iteration source is read from the default readable port.

A portion of each input document can be selected using the select attribute. If no selection is specified, the document node of each document is selected.

Each subtree selected by the p:for-each from each of the inputs that appear on the iteration source is wrapped in a document node and provided to the subpipeline.

The processor provides each document, one at a time, to the subpipeline represented by the children of the p:for-each on a port named current.

For each declared output, the processor collects all the documents that are produced for that output from all the iterations, in order, into a sequence. The result of the p:for-each on that output is that sequence of documents.

The environment inherited by the contained steps of a p:for-each is the inherited environment with these modifications: