Recipe Language 2 Architecture¶
Important
Recipe language 2.0.0 is the only production parsing and execution
path. Maintained recipes and setup templates use version 2; external
version 1 files are rejected with migration diagnostics.
Exact version 2 fields, types, defaults, and examples are in the generated
Recipe Language 2.0 Reference. The aggregate schema is also available as
recipe_language.schema.json.
Design and dependency direction¶
The version 2 design has one structural definition. Strict, frozen Pydantic models own field names, types, required status, defaults, descriptions, examples, aliases, discriminators, serialization metadata, and JSON Schema. Consumers inspect either the typed model or its generated schema; they do not maintain another field registry.
The modules and artifacts have deliberately one-way dependencies:
pypts.recipe_language
| Pydantic fields and discriminated unions
+--------------------------+
| |
v v
pypts.recipe_parser JSON Schema generator
| |
| v
| recipe_language.schema.json
| |
| v
| JSON-only RST renderer
| |
v v
typed Recipe generated reference RST
| |
v v
recipe.py runtime Sphinx
construction
recipe_language never imports YAML, runtime classes, concrete steps, YamVIEW, or
Sphinx. recipe_parser depends on the models and PyYAML, but still does not
import runtime or UI code. Documentation generation reads the model only to
create JSON Schema; the RST renderer reads the JSON generated for the current
build and has no Pydantic or Sphinx dependency.
Parsing and information flow¶
Pydantic validates already-constructed Python values; it is not a YAML parser. Safe loading, source positions, structural validation, and application semantics therefore remain separate stages:
recipe YAML text/file
|
v
PyYAML YAML front end
| source information
+--> compose_all(SafeLoader) --> node/path/span index -----+
| | |
| +--> duplicate/recursive-alias diagnostics ------+
|
+--> safe_load_all() --> safe Python documents |
| |
v |
strict Pydantic models |
| |
v |
cross-document semantic pass |
| |
+--> diagnostics <---------+
| code, path, severity,
| source, nearest span
v
frozen aggregate Recipe
|
+--> canonical multi-document YAML
|
+--> recipe.py runtime construction
parse_recipe_text and parse_recipe_file return ParseResult. A
valid result owns an aggregate RecipeHeader plus one or more
Sequence models. require_recipe() raises with the complete
diagnostic tuple when errors exist. recipe_to_yaml returns deterministic
version 2 YAML without file I/O; comments, quoting, and original formatting
are not preserved. Parse/serialize/reparse preserves the aggregate definition.
Here, “composition” is PyYAML terminology, not a PyPTS adapter or an additional
recipe representation. yaml.compose_all(..., Loader=yaml.SafeLoader)
returns PyYAML node objects with source marks, which the parser uses to detect
duplicate keys and recursive aliases and to index diagnostic spans.
yaml.safe_load_all() separately constructs ordinary safe Python values for
Pydantic. Both operations use PyYAML’s safe loader; neither constructs runtime
Recipe or Step objects.
Structural and custom semantic rules¶
Rules local to one model stay beside that model. Pydantic reports them with a precise nested location, which the parser translates to the PyPTS diagnostic envelope and nearest YAML span.
For example, an indexed DirectInput must hold a list:
@model_validator(mode="after")
def indexed_values_are_lists(self) -> DirectInput:
if self.indexed and not isinstance(self.value, list):
raise PydanticCustomError(
"invalid_indexed_input", "Indexed direct input value must be a list."
)
return self
A Python method action requires method_name:
@model_validator(mode="after")
def method_actions_have_names(self) -> PythonModuleStep:
if self.action_type == "method" and not self.method_name:
raise PydanticCustomError(
"missing_method_name", "Method actions require method_name."
)
return self
Likewise, WaitStep requires a named wait_time input:
@model_validator(mode="after")
def has_wait_time(self) -> WaitStep:
if "wait_time" not in self.input_mapping:
raise PydanticCustomError("missing_required_input", "WaitStep requires input 'wait_time'.")
return self
Other rules require context that no individual JSON object or JSON Schema can
see. They remain in one explicit semantic pass. Sequence names must be
unique and main_sequence must resolve:
by_name: dict[str, tuple[int, Sequence]] = {}
for document_index, sequence in sequences:
path = (document_index, "sequence_name")
if sequence.sequence_name in by_name:
diagnostics.append(_diagnostic(
"duplicate-sequence",
f"Duplicate sequence '{sequence.sequence_name}'.",
path,
source_name,
spans,
))
else:
by_name[sequence.sequence_name] = (document_index, sequence)
if complete_sequences and header is not None and header.main_sequence not in by_name:
diagnostics.append(_diagnostic(
"unknown-main-sequence",
f"Main sequence '{header.main_sequence}' does not exist.",
(0, "main_sequence"),
source_name,
spans,
))
Every SequenceStep target is then resolved across all loaded documents:
if isinstance(step, SequenceStep) and step.sequence.name not in by_name:
diagnostics.append(_diagnostic(
"unknown-sequence-reference",
f"Sequence '{sequence.sequence_name}' references unknown sequence "
f"'{step.sequence.name}'.",
step_path + ("sequence", "name"),
source_name,
spans,
))
Indexed lists on one step must have equal lengths, while PassthroughOutput must be the only verdict-producing output:
indexed_lengths = [
len(value.value)
for value in step.input_mapping.values()
if isinstance(value, DirectInput) and value.indexed
]
if len(set(indexed_lengths)) > 1:
diagnostics.append(_diagnostic(
"unequal-indexed-inputs",
"Indexed input lists must have equal lengths.",
step_path + ("input_mapping",),
source_name,
spans,
))
verdicts = [
value for value in step.output_mapping.values() if isinstance(value, verdict_types)
]
if any(isinstance(value, PassthroughOutput) for value in verdicts) and len(verdicts) != 1:
diagnostics.append(_diagnostic(
"mixed-passthrough",
"'passthrough' must be the sole verdict mapping.",
step_path + ("output_mapping",),
source_name,
spans,
))
SSH rules need both recipe globals and execution order. The semantic pass checks required connection globals and credentials, rejects an upload before a connection, and requires an opened connection to be closed:
ssh_steps = [item for item in flattened if item[2].steptype.startswith("SSH")]
if ssh_steps and header is not None:
for required in ("ssh_client", "host", "user", "port"):
if required not in header.globals:
diagnostics.append(_diagnostic(
"missing-ssh-global",
f"SSH step requires global '{required}'.",
(0, "globals", required),
source_name,
spans,
))
if "password" not in header.globals and "private_key" not in header.globals:
diagnostics.append(_diagnostic(
"missing-ssh-credential",
"SSH steps require password or private_key global.",
(0, "globals"),
source_name,
spans,
))
connected = False
unclosed_connect: tuple[str, int] | None = None
for section, index, step in flattened:
if step.steptype == "SSHConnectStep":
connected = True
unclosed_connect = (section, index)
elif step.steptype == "SSHUploadStep" and not connected:
diagnostics.append(_diagnostic(
"missing-ssh-connect",
f"Sequence '{sequence.sequence_name}' uploads before an SSH connection.",
(document_index, section, index),
source_name,
spans,
))
elif step.steptype == "SSHCloseStep":
connected = False
unclosed_connect = None
if unclosed_connect is not None:
diagnostics.append(_diagnostic(
"missing-ssh-close",
f"Sequence '{sequence.sequence_name}' opens SSH without a later close.",
(document_index, "teardown_steps"),
source_name,
spans,
))
These rules are documented manually because JSON Schema describes the aggregate structure, not multi-document YAML safety, source spans, equality between sibling list lengths, reference resolution, or ordered lifecycle state.
How YamVIEW consumes the language¶
YamVIEW treats the aggregate JSON Schema as its form description. The
Step, InputMapping, and OutputMapping discriminator maps enumerate
available variants; referenced definitions provide properties, required
fields, strict types, defaults, descriptions, examples, and allowed literal
values.
The editor flow is:
generated/published JSON Schema
|
+--> discriminator choices --> step/mapping selectors
|
+--> referenced properties --> labels, controls, help, defaults
|
v
edited aggregate document --> canonical YAML text
|
v
parse_recipe_text()
| |
| +--> diagnostics and source spans
v
typed Recipe
Local widget code may choose a suitable control for a JSON type, but it must not own supported step names or field rules. Whole-recipe validation always goes through the parser so semantic rules and YAML diagnostics are identical between YamVIEW, command-line tools, and runtime loading.
YamVIEW retains a locally valid structured edit even when whole-recipe
semantics fail. The parser diagnostics are displayed and persistence is
disabled until the edit is repaired or the last valid text is restored.
Schema-invalid raw YAML remains available in the text editor, while the
structured sequencer is disabled until an aggregate definition can be formed.
Structured edits and persistence call recipe_to_yaml and therefore replace
comments, quoting, and source layout with deterministic canonical formatting.
How the sequencer consumes the model¶
Runtime construction begins only after parsing succeeds. It receives the aggregate typed model, not raw YAML or loosely typed dictionaries:
ParseResult.require_recipe()
|
v
frozen Recipe model
|
v
recipe.py: Recipe
| |
| +-------> sequence table and nested reference binding
v
recipe.py: Sequence
|
v
Step.build_step() for each typed definition
|
v
STEP_TYPE_REGISTRY --> concrete classes in steps.py
|
v
setup_steps -> steps -> teardown_steps
This is implemented in the existing runtime construction path, not a separate
adapter module. Recipe receives the validated aggregate model, Sequence
iterates typed definitions, and Step.build_step() is the sole typed factory
that selects an executable class from steps.py.
The runtime registry remains because a canonical discriminator such as
PythonModuleStep must be associated with the Python class that implements
its behavior. It is a behavior registry, not a second language schema: field
names, types, defaults, and structural rules remain exclusively in the
Pydantic models. A completeness test requires every step-definition model
discriminator to have exactly one executable implementation.
Concrete _step() methods in steps.py continue to own execution. For
example, the executable PythonModuleStep still imports and invokes Python
code; it no longer validates an untrusted recipe dictionary. Common definition
fields are dumped once by Step.build_step() and passed to the existing
constructors. IndexedStep remains a runtime-generated wrapper and is
never added to the StepDefinition model union.
Synthetic runtime operations are also constructed directly. For example,
Recipe.run() must not fabricate a recipe dictionary merely to execute the
main sequence. No runtime construction reparses YAML or repeats Pydantic
structural validation, and invalid recipes never instantiate executable steps.
Execution events, error policy, reports, hardware access, and GUI interaction
remain downstream of the frozen language model.
Canonical documentation recipe¶
This documentation-owned fixture demonstrates the version 2 header, two sequences, nested execution, canonical step names, explicit discriminators, indexed input, every input variant, and representative verdict, storage, image, and passthrough outputs. Tests validate and round-trip it with the production parser. It is not a bundled recipe.
# SPDX-FileCopyrightText: 2026 CERN <home.cern>
# SPDX-License-Identifier: CC-BY-SA-4.0
---
name: Recipe language 2 documentation example
version: "1.0"
recipe_version: 2.0.0
description: Demonstrate canonical version 2 syntax and typed mappings.
main_sequence: Main
continue_on_error: false
report: overwrite
report_name_include_serial: true
test_package: acceptance.tests
globals:
target: 12
saved_result: null
---
sequence_name: Main
description: Run a measurement and then the calibration sequence.
parameters: {}
outputs: {}
locals:
expected: 12
measured: null
setup_steps: []
steps:
- steptype: PythonModuleStep
step_name: Measure channels
description: Exercise every input mapping and representative outputs.
id: measure-channels
skip: false
critical: true
continue_on_error: false
action_type: method
module: measurements.py
method_name: measure
input_mapping:
channels: {type: direct, value: [0, 1], indexed: true}
expected: {type: local, local_name: expected}
target: {type: global, global_name: target}
transform: {type: method, value: normalize}
output_mapping:
passed: {type: passfail}
exact: {type: equals, value: 12}
bounded: {type: range, min: 10, max: 14}
measured: {type: local, local_name: measured}
saved: {type: global, global_name: saved_result}
chart: {type: image}
- steptype: SequenceStep
step_name: Calibrate
description: Run a nested sequence and use its aggregate verdict.
sequence: {type: internal, name: Calibration}
input_mapping: {}
output_mapping:
result: {type: passthrough}
stored: {type: local, local_name: measured}
teardown_steps: []
---
sequence_name: Calibration
description: Wait for the equipment to stabilize.
parameters: {}
outputs: {}
locals: {}
setup_steps: []
steps:
- steptype: WaitStep
step_name: Stabilize
description: Wait before returning to the caller.
input_mapping:
wait_time: {type: direct, value: 1}
output_mapping: {}
teardown_steps: []
Maintaining the documentation¶
See Maintaining the Recipe Language for the complete extension and version
upgrade workflow, including the definition/runtime boundary and the purpose of
STEP_TYPE_REGISTRY.
Every Sphinx build generates both artifacts before reading documentation sources:
Pydantic models
|
v
_generated/recipe_language.schema.json
|
v
JSON-only RST renderer
|
v
_generated/recipe_language_reference.rst
The Sphinx builder-inited hook writes these files to an ignored staging
directory under docs/source. The schema is copied into the HTML output as a
download, and the generated RST is included in the toctree. Neither generated
file is maintained manually or treated as a committed source artifact.
Pydantic is a core production dependency. CI and any documentation build image
must install the doc extra and include the model, schema generator, and
JSON-only renderer sources.
The documentation contract is protected by tests that generate into temporary directories, verify deterministic model-to-JSON and JSON-to-RST output, count all discriminator variants, validate the example, check literal-include markers, and build Sphinx with warnings treated as errors. Generation failure therefore fails the same build that would publish the documentation.