Skip to content

feat(examples): validate experiment tasks and add run and study templates - #1458

Merged
Brad-Edwards merged 14 commits into
devfrom
380-experiment-template-catalog
Oct 25, 2026
Merged

Brad-Edwards merged 14 commits into
devfrom
380-experiment-template-catalog

Conversation

@doublewhy

@doublewhy doublewhy commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds validated experiment run and study design templates, and supports the current task-validation scope in issue #1519. raes semantic validate --contract experiment-task-v1 now accepts the reference task and rejects the invalid task fixture. Experiment loaders can check a supplied validated task against the plan's task id and any declared version.

The existing branch is synchronized with current dev by a real merge. Scenario-template and catalog improvements from the original stack are retained where they are still absent from dev.

Requirement UIDs

Related Issues

Closes #380
Closes #1519
Closes #1563

ADR Impact

Changes

  • Catalogs seeded run and two-condition study templates as experiment-authoring-input-v1, with strict YAML loading and validation commands.
  • Registers experiment-task-v1 in the existing conformance registry with a conservative structural validation claim, consistent with the authoring-input and study contracts.
  • Adds optional task=ExperimentTaskModel to parse_experiment_spec and load_experiment_spec. Task ids must match; a declared reference version must match. Mismatches raise a sanitized task-reference-mismatch validation error. No task is fetched automatically.
  • Uses Support experiment-task-v1 validation and add a task template #1519's explicit catalog-limit option: no experiment task template is shipped because its required artifact checksum, byte size, and creation time must describe the author's actual protocol artifact. Documents task validation and supplied-task checks.
  • Converts invalid-date YAML errors to ExperimentSpecValidationError, repairing the documented template-loader behavior from Experiment spec loader raises a bare ValueError on an unquoted invalid date #1563.
  • Records fresh immutable formal-semantic-validation and specification-coverage captures after replaying their retained evidence. No published history or numbered release is rewritten.

Test Plan

  • Targeted pytest across test_issue_1519_experiment_tasks.py, test_experiment_authoring.py, test_semantic_cli.py, test_issue_380_experiment_templates.py, test_issue_378_scenario_templates.py, test_issue_381_example_catalog_metadata.py, and test_example_library_policy.py: 149 passed.
  • Before the implementation, the new regression module produced 12 failures and one pass, reproducing the missing task selector, missing supplied-task checks, and invalid-date exception. After implementation all 13 cases passed. The final valid/invalid task-fixture CLI cases also passed separately.
  • tools/check_example_library.py, tools/check_repo_policy.py --base-rev origin/dev, tools/check_specification_coverage.py, and tools/check_formal_semantic_validation.py pass. Ruff check and format checks pass for the changed Python files.
  • The combined fast-feedback lane stopped at the existing missing AUT-806 mapping in tools/policy/requirement_order.yaml; its repository policy stage passed. No policy rule was relaxed. Repository-wide test, integration, fuzz, coverage, and completion verification remain CI responsibilities.

Ground Control Checks

  • Repository policy command passes: targeted check_repo_policy.py --base-rev origin/dev passed before and after implementation.
  • Pre-push code review and test-quality review completed: reviewed issue fulfillment, task identity/version rules, sanitized diagnostics, context-free compatibility, validation honesty, and positive/negative regression coverage. Required current-head CI remains a merge condition.

Traceability

  • IMPLEMENTS: implementations/python/packages/raes_conformance/conformance/validators.py, implementations/python/packages/raes_contracts/experiment_spec.py, examples/library/, tools/check_example_library.py, tools/example_library_checks.py, examples/README.md, docs/explain/getting-started.md
  • TESTS: implementations/python/tests/test_issue_1519_experiment_tasks.py, implementations/python/tests/test_issue_380_experiment_templates.py, implementations/python/tests/test_issue_378_scenario_templates.py, implementations/python/tests/test_issue_381_example_catalog_metadata.py, implementations/python/tests/test_example_library_policy.py

Checklist

  • Code follows project coding standards.
  • FM1 for the optional task-reference invariant: same id and any declared version, documented in the loader and examples README and tested through both loaders. FM0 for selector registration and invalid-YAML error handling.
  • Published schemas and contract models are unchanged; no regeneration required.
  • PR title follows Conventional Commits.
  • Documentation describes the supplied-task boundary and catalog limit.

Documentation

Updated examples/README.md, examples/library/catalog.yaml, and docs/explain/getting-started.md. The task-template absence is an explicit accepted option in #1519. No required work from #380 or #1519 is moved to a follow-up. The separately tracked #1520 policy question remains outside this change.

Every worked example, template, and pattern entry in
examples/library/catalog.yaml now records validation_status (validated
or guidance), sdl_sections, intended_user, and limits. The gate rejects
an entry whose status is outside the vocabulary, a template that is not
validated, a pattern that claims validation, an unknown intended user,
empty limits, or a section name that is not a current SDL section.
Validated worked examples are now parsed through parse_sdl_file and
fail on errors or advisories, and a validated entry must list only
sections present in its SDL. The catalog description no longer calls
every entry validated.

The entry checks live in a new support module,
tools/example_library_checks.py, because tools/check_example_library.py
was already 488 lines against SonarCloud's 500-line file limit. The
template-body SDL check moves there with its rule ids and messages. The
module imports raes at module level, so the
example-library-template-import rule for a failed raes import is
removed. The top-level fields that are not sections come from
tools/sdl_catalog_parity/_paths.py, which the SDL catalog parity gate
keeps in step with specs/sdl/sections.md.

examples/README.md documents the fields and what the gate checks.
…ader

The section-presence check re-read each validated worked example with
PyYAML's YAML 1.1 yaml.safe_load after parse_sdl_file had accepted it.
sdl-yaml/v1 resolves plain scalars with YAML 1.2 Core, so valid SDL such
as an unquoted "description: 2026-02-30" or "description: =" raised
ValueError or ConstructorError there, and the policy gate exited with a
traceback instead of returning failures. The check now loads the file
with load_sdl_fragment, the loader that parse_sdl_file uses. The
absent-section failure also names the catalog entry, because the
hospital and satcom worked-example files are each referenced by two
entries.

Now that patterns are guidance, docs/explain/getting-started.md (the
Current Boundary list and the Template And Pattern Boundary section) and
the canonical reference map no longer call the whole library validated.
The README says the template check runs on the body as PyYAML loads it,
and that the gate does not parse a guidance entry as SDL. The
task-as-objective-contract pattern no longer tells authors to add
metrics, evaluations, TLOs, or goals, which ADR-073 removed from SDL,
and its catalog entry now lists the propositions and assertions it
shows. The sem-208 limit says the test suite parses and validates its
inline SDL rather than running it.
The gate read each template file with yaml.safe_load, which keeps the
last of duplicate keys, expands << merge keys, and accepts explicit
tags, and then validated a re-dumped copy of the body. A body with a
duplicate key, a merge key, or a tag therefore passed the gate while
parse_sdl on the same body text rejected it, and an unquoted
"description: 2026-02-30" stopped the gate with a ValueError traceback.

Template files are now read with load_sdl_fragment, the sdl-yaml/v1
loader that parse_sdl uses. The root mapping holds the template fields,
so its keys are read literally, and the body below it gets the same
structural key checks as an SDL document root. A file that the loader
rejects fails with example-library-parse. Acyclic aliases stay accepted
within the loader's limits, as parse_sdl accepts them. The README
validation_status row now names the loader and what it rejects.
… strings

check_template_body passed yaml.safe_dump(body) to parse_sdl. PyYAML writes
a string such as '0o36' unquoted, and sdl-yaml/v1 reads that back as an
int, so the gate accepted a quoted '0o36' interval that parse_sdl rejects
in the template file. Serialize the loaded body with json.dumps instead:
JSON is sdl-yaml/v1 input and keeps the loaded types. The new
quoted-octal-string case fails without this change.
…he gate

json.dumps(body) escaped a character outside the BMP as a surrogate pair,
and the sdl-yaml/v1 loader read the pair back as two lone surrogates, so a
template body that parse_sdl accepts, such as `description: smile` plus an
emoji, failed the gate with UnicodeEncodeError. The body is now written
with ensure_ascii=False. DEL, the C1 controls, U+FFFE and U+FFFF (which the
reader rejects) and NEL, U+2028 and U+2029 (which it reads as line breaks)
are still written as JSON escapes, so every loaded string reads back
unchanged.
…laim notes

Add two scenario templates beside minimal-validated-scenario:
segmented-network-scenario (two switched segments, linked hosts with
fixed addresses, HTTPS-only access rules, a service feature) and
parameterized-scenario (host operating system, size, instance count, and
service port from declared variables). Both bodies pass the SDL parser
and semantic validator with no advisories, and both are cataloged with
the entry metadata.

Every template now carries a validation list of commands with their
expected results. The gate requires the field, and
tools/example_library_checks.py checks that it is a non-empty list of
mappings with non-empty command and expected strings. The gate does not
run the commands. Each gate entry says that YAML loading keeps the last
value of a duplicate key, so the gate does not report duplicate keys.

examples/README.md lists what each scenario template shows and does not
claim, and how to validate an adapted copy with raes semantic validate
from the repository root. Its Template Boundary section defines a
${name} placeholder by the variable-name grammar, says that the gate
rejects a placeholder naming an undeclared variable, and says that text
such as ${Web OS} is not a placeholder and stays literal text.
The gate step in each template's validation list said that the gate
loads the file as YAML, keeps the last value of a duplicate key, and so
does not report duplicate keys. The gate now reads template files with
the sdl-yaml/v1 loader, which rejects duplicate keys and merge keys, so
the eight SDL templates now say that instead.

test_issue_378_scenario_templates.py copies the shipped library and,
for each template in the catalog, adds a duplicate key to the body and
expects only example-library-parse for that template file. With the
yaml.safe_load read restored in _check_template_file, all eight cases
fail.
… studies

Add two templates whose bodies are experiment-authoring-input-v1
documents: seeded-run-plan (run count, seed, and episode controls for one
task) and two-condition-study-design (a treatment factor, two compared
conditions, and the runs per condition). Both pass the experiment
authoring-input loader, parse_experiment_spec.

A template's catalog entry may now declare contract:
experiment-authoring-input-v1. Such an entry names experiment-author as
its intended user and lists no sdl_sections, and the gate validates its
body with the experiment loader instead of the SDL parser. Worked
examples and patterns stay SDL.

examples/README.md separates the SDL, experiment-input, guidance, and
unsupported parts of the task, run, and study surfaces, including why
there is no experiment-task-v1 template, and gives a command to validate
an adapted experiment template.
The gate reads a template file with yaml.safe_load, so duplicate keys,
scalar aliases, and << merge keys never reach parse_experiment_spec,
while the template's own loader command rejects them in a saved copy.
The README validation_status row and both templates' gate `expected`
text now say so.

The README and catalog said the loader never checks condition factor
levels. It does check the levels that a binding descriptor (under
binding_semantics: explicit-required, ADR-094) or a stratified selection
stratum joins. Only unjoined levels, such as those in the shipped
two-condition-study-design body, go unchecked.

The gate now allows the experiment-authoring-input-v1 contract only on
run and study templates. The run and study surface summaries mention the
experiment templates, the checker docstring and getting-started.md name
both validation paths, and the adapted-copy command says to run it from
the repository root.
… reader

The gate read every template file with yaml.safe_load, so an experiment
template body with a duplicate key, a scalar alias, or a << merge key
passed the gate while load_experiment_spec rejected the same body saved
on its own. The README and both experiment templates said so.

The template loader now follows the entry's contract. An
experiment-authoring-input-v1 template file is read with _load_yaml,
the bounded reader that parse_experiment_spec uses, which rejects
duplicate keys, aliases, and merge keys. Any other template keeps the
sdl-yaml/v1 loader, which would reject the hyphenated factor and
condition ids of two-condition-study-design as non-canonical SDL field
spellings. A file that its reader rejects fails with
example-library-parse. The README validation_status row and the gate
step of seeded-run-plan and two-condition-study-design now say that the
gate rejects these.
…parse failure

The experiment loader's YAML reader resolves YAML 1.1 timestamps and raises
a bare ValueError on an unquoted date such as 2026-02-30. load_template_file
caught only SDLParseError, yaml.YAMLError and UnicodeDecodeError, so
check_example_library.py crashed with a traceback instead of reporting
example-library-parse. Catch ValueError, which also covers
UnicodeDecodeError. The new invalid-date case fails without this change.
parse_experiment_spec lets the same ValueError through; #1563 tracks that.
@doublewhy
doublewhy force-pushed the 380-experiment-template-catalog branch from b4802c8 to cdf228d Compare October 12, 2026 10:52
@doublewhy

Copy link
Copy Markdown
Contributor Author

Closing for now: #380 says the templates should wait until the supported SDL and validation path are clear, which is Episode & Budget, Trajectory and Benchmark work. #380 stays open and the branch is kept.

@doublewhy doublewhy closed this Oct 12, 2026
@Brad-Edwards

Copy link
Copy Markdown
Collaborator

Recommendation: keep closed in its current form.

Reviewed head cdf228d71348451a583e92664e7072198a44b1ef, targeting dev.

  1. PR is already CLOSED without merge; it is not a merge candidate.
  2. Review context is incomplete because the PR body is truncated, and the diff includes stacked feat(examples): add machine-checked metadata to example library entries #1438/feat(examples): add scenario templates with validation commands and claim notes #1445 work.

Required action:

Coverage and outstanding work: #380's design-only run/study templates and unsupported-task distinction are described, but a closed PR is not actionable for merge and the full scope/outstanding-work report is unavailable.

Verification limits: No claim of a completed source review or source defect.

@Brad-Edwards Brad-Edwards reopened this Oct 25, 2026
…iment-template-catalog

# Conflicts:
#	examples/README.md
#	examples/library/catalog.yaml
#	tools/check_example_library.py
#	tools/example_library_checks.py
@Brad-Edwards Brad-Edwards changed the title feat(examples): add experiment authoring-input templates for runs and studies feat(examples): validate experiment tasks and add run and study templates Oct 25, 2026
@Brad-Edwards
Brad-Edwards merged commit ecd7b20 into dev Oct 25, 2026
33 checks passed
@Brad-Edwards
Brad-Edwards deleted the 380-experiment-template-catalog branch October 25, 2026 04:31
doublewhy added a commit that referenced this pull request Oct 25, 2026
Dev's source at ecd7b20, where #1458 merged after #1444, matches no
admitted capture, so the evidence checkers and the tests that load the
current capture fail on every branch built on it. This branch changes no
package source. collect_research_evidence.py ran both pipelines on it:
38 of 42 specification stages passed (4 preregistered non-executions)
and all 20 formal observations matched their expected outcomes.
Brad-Edwards added a commit that referenced this pull request Nov 11, 2026
…gates (#1474)

* ci(policy): read git path listings with -z so quoted names reach the gates

Without -z, git quotes path names that contain a byte above 0x80 under
the default core.quotePath, a control character, a double quote or a
backslash. The nox runner dropped such names from the hygiene
selection, tools/policy/common.changed_paths handed git's quoted text
to the repository policy and JSON artifact checks, and the schema
publication base listing dropped them.

List with -z and split on NUL in all three, as
check_requirement_governance.py and verification_plan.py already do.
changed_paths now runs one listing for its three comparisons.

* test(policy): list a name with a newline in the nul path listing test

Without -z, git quotes a newline in a name as it does any other control
character. The test now covers such a name next to the tab, non-ASCII
and double-quote names; dev's three helpers fail on it alone.

* ci(policy): read nul path listings as bytes so carriage returns in names survive

Text mode turned a carriage return or CRLF in a name into LF before the
NUL split, so the runner dropped the name and the policy helpers
returned a path that does not exist. tools.policy.common.git_paths now
runs each listing with -z, reads bytes and decodes every name as strict
UTF-8, like the repository's other -z listings. The runner, changed_paths
and the requirement gate's merge-time diff use it; the schema
publication listing reads bytes the same way.

The regression test drives real repositories with CR, CRLF, LF, tab,
quote, non-ASCII and space-edged names through the hygiene selection,
the 500-line cap, the base schema publication manifest and the merge
listing, and checks that a name that is not UTF-8 stops each listing.

* ci(proof): fetch the locked isabelle archive from the official release locator

The Cambridge mirror's dist directory now holds Isabelle2026 only, so
the locked Isabelle2025-2_linux.tar.gz answers 404 there and
canonical / generic-tool-local-inputs fails exact verification, which
skips the checks, integration and proof jobs. The official release
locator serves the same bytes; the fetch still verifies the unchanged
lock size and SHA-256.

* test: record combined-source research captures for issue #1473

Dev's source at ecd7b20, where #1458 merged after #1444, matches no
admitted capture, so the evidence checkers and the tests that load the
current capture fail on every branch built on it. This branch changes no
package source. collect_research_evidence.py ran both pipelines on it:
38 of 42 specification stages passed (4 preregistered non-executions)
and all 20 formal observations matched their expected outcomes.

* test: record the runtime mapping capture for the merged source

* Fix current repository policy checks for PR 1474

---------

Co-authored-by: Brad Edwards <j.bradley.edwards@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants