Repository navigation
feat(examples): validate experiment tasks and add run and study templates - #1458
Merged
Merged
Conversation
doublewhy
force-pushed
the
380-experiment-template-catalog
branch
3 times, most recently
from
October 9, 2026 13:00
3ad4b31 to
8897527
Compare
doublewhy
marked this pull request as ready for review
October 9, 2026 13:27
This was referenced Oct 9, 2026
Closed
doublewhy
force-pushed
the
380-experiment-template-catalog
branch
from
October 10, 2026 17:20
8897527 to
f33380e
Compare
This was referenced Oct 11, 2026
Every worked example, template, and pattern entry in examples/library/catalog.yaml now records validation_status (validated or guidance), sdl_sections, intended_user, and limits. The gate rejects an entry whose status is outside the vocabulary, a template that is not validated, a pattern that claims validation, an unknown intended user, empty limits, or a section name that is not a current SDL section. Validated worked examples are now parsed through parse_sdl_file and fail on errors or advisories, and a validated entry must list only sections present in its SDL. The catalog description no longer calls every entry validated. The entry checks live in a new support module, tools/example_library_checks.py, because tools/check_example_library.py was already 488 lines against SonarCloud's 500-line file limit. The template-body SDL check moves there with its rule ids and messages. The module imports raes at module level, so the example-library-template-import rule for a failed raes import is removed. The top-level fields that are not sections come from tools/sdl_catalog_parity/_paths.py, which the SDL catalog parity gate keeps in step with specs/sdl/sections.md. examples/README.md documents the fields and what the gate checks.
…ader The section-presence check re-read each validated worked example with PyYAML's YAML 1.1 yaml.safe_load after parse_sdl_file had accepted it. sdl-yaml/v1 resolves plain scalars with YAML 1.2 Core, so valid SDL such as an unquoted "description: 2026-02-30" or "description: =" raised ValueError or ConstructorError there, and the policy gate exited with a traceback instead of returning failures. The check now loads the file with load_sdl_fragment, the loader that parse_sdl_file uses. The absent-section failure also names the catalog entry, because the hospital and satcom worked-example files are each referenced by two entries. Now that patterns are guidance, docs/explain/getting-started.md (the Current Boundary list and the Template And Pattern Boundary section) and the canonical reference map no longer call the whole library validated. The README says the template check runs on the body as PyYAML loads it, and that the gate does not parse a guidance entry as SDL. The task-as-objective-contract pattern no longer tells authors to add metrics, evaluations, TLOs, or goals, which ADR-073 removed from SDL, and its catalog entry now lists the propositions and assertions it shows. The sem-208 limit says the test suite parses and validates its inline SDL rather than running it.
The gate read each template file with yaml.safe_load, which keeps the last of duplicate keys, expands << merge keys, and accepts explicit tags, and then validated a re-dumped copy of the body. A body with a duplicate key, a merge key, or a tag therefore passed the gate while parse_sdl on the same body text rejected it, and an unquoted "description: 2026-02-30" stopped the gate with a ValueError traceback. Template files are now read with load_sdl_fragment, the sdl-yaml/v1 loader that parse_sdl uses. The root mapping holds the template fields, so its keys are read literally, and the body below it gets the same structural key checks as an SDL document root. A file that the loader rejects fails with example-library-parse. Acyclic aliases stay accepted within the loader's limits, as parse_sdl accepts them. The README validation_status row now names the loader and what it rejects.
doublewhy
force-pushed
the
380-experiment-template-catalog
branch
from
October 12, 2026 08:09
f33380e to
930cf9c
Compare
This was referenced Oct 12, 2026
doublewhy
force-pushed
the
380-experiment-template-catalog
branch
from
October 12, 2026 08:38
930cf9c to
236ddc6
Compare
… strings check_template_body passed yaml.safe_dump(body) to parse_sdl. PyYAML writes a string such as '0o36' unquoted, and sdl-yaml/v1 reads that back as an int, so the gate accepted a quoted '0o36' interval that parse_sdl rejects in the template file. Serialize the loaded body with json.dumps instead: JSON is sdl-yaml/v1 input and keeps the loaded types. The new quoted-octal-string case fails without this change.
3 tasks
doublewhy
force-pushed
the
380-experiment-template-catalog
branch
from
October 12, 2026 09:44
236ddc6 to
b4802c8
Compare
…he gate json.dumps(body) escaped a character outside the BMP as a surrogate pair, and the sdl-yaml/v1 loader read the pair back as two lone surrogates, so a template body that parse_sdl accepts, such as `description: smile` plus an emoji, failed the gate with UnicodeEncodeError. The body is now written with ensure_ascii=False. DEL, the C1 controls, U+FFFE and U+FFFF (which the reader rejects) and NEL, U+2028 and U+2029 (which it reads as line breaks) are still written as JSON escapes, so every loaded string reads back unchanged.
…laim notes
Add two scenario templates beside minimal-validated-scenario:
segmented-network-scenario (two switched segments, linked hosts with
fixed addresses, HTTPS-only access rules, a service feature) and
parameterized-scenario (host operating system, size, instance count, and
service port from declared variables). Both bodies pass the SDL parser
and semantic validator with no advisories, and both are cataloged with
the entry metadata.
Every template now carries a validation list of commands with their
expected results. The gate requires the field, and
tools/example_library_checks.py checks that it is a non-empty list of
mappings with non-empty command and expected strings. The gate does not
run the commands. Each gate entry says that YAML loading keeps the last
value of a duplicate key, so the gate does not report duplicate keys.
examples/README.md lists what each scenario template shows and does not
claim, and how to validate an adapted copy with raes semantic validate
from the repository root. Its Template Boundary section defines a
${name} placeholder by the variable-name grammar, says that the gate
rejects a placeholder naming an undeclared variable, and says that text
such as ${Web OS} is not a placeholder and stays literal text.
The gate step in each template's validation list said that the gate loads the file as YAML, keeps the last value of a duplicate key, and so does not report duplicate keys. The gate now reads template files with the sdl-yaml/v1 loader, which rejects duplicate keys and merge keys, so the eight SDL templates now say that instead. test_issue_378_scenario_templates.py copies the shipped library and, for each template in the catalog, adds a duplicate key to the body and expects only example-library-parse for that template file. With the yaml.safe_load read restored in _check_template_file, all eight cases fail.
… studies Add two templates whose bodies are experiment-authoring-input-v1 documents: seeded-run-plan (run count, seed, and episode controls for one task) and two-condition-study-design (a treatment factor, two compared conditions, and the runs per condition). Both pass the experiment authoring-input loader, parse_experiment_spec. A template's catalog entry may now declare contract: experiment-authoring-input-v1. Such an entry names experiment-author as its intended user and lists no sdl_sections, and the gate validates its body with the experiment loader instead of the SDL parser. Worked examples and patterns stay SDL. examples/README.md separates the SDL, experiment-input, guidance, and unsupported parts of the task, run, and study surfaces, including why there is no experiment-task-v1 template, and gives a command to validate an adapted experiment template.
The gate reads a template file with yaml.safe_load, so duplicate keys, scalar aliases, and << merge keys never reach parse_experiment_spec, while the template's own loader command rejects them in a saved copy. The README validation_status row and both templates' gate `expected` text now say so. The README and catalog said the loader never checks condition factor levels. It does check the levels that a binding descriptor (under binding_semantics: explicit-required, ADR-094) or a stratified selection stratum joins. Only unjoined levels, such as those in the shipped two-condition-study-design body, go unchecked. The gate now allows the experiment-authoring-input-v1 contract only on run and study templates. The run and study surface summaries mention the experiment templates, the checker docstring and getting-started.md name both validation paths, and the adapted-copy command says to run it from the repository root.
… reader The gate read every template file with yaml.safe_load, so an experiment template body with a duplicate key, a scalar alias, or a << merge key passed the gate while load_experiment_spec rejected the same body saved on its own. The README and both experiment templates said so. The template loader now follows the entry's contract. An experiment-authoring-input-v1 template file is read with _load_yaml, the bounded reader that parse_experiment_spec uses, which rejects duplicate keys, aliases, and merge keys. Any other template keeps the sdl-yaml/v1 loader, which would reject the hyphenated factor and condition ids of two-condition-study-design as non-canonical SDL field spellings. A file that its reader rejects fails with example-library-parse. The README validation_status row and the gate step of seeded-run-plan and two-condition-study-design now say that the gate rejects these.
…parse failure The experiment loader's YAML reader resolves YAML 1.1 timestamps and raises a bare ValueError on an unquoted date such as 2026-02-30. load_template_file caught only SDLParseError, yaml.YAMLError and UnicodeDecodeError, so check_example_library.py crashed with a traceback instead of reporting example-library-parse. Catch ValueError, which also covers UnicodeDecodeError. The new invalid-date case fails without this change. parse_experiment_spec lets the same ValueError through; #1563 tracks that.
doublewhy
force-pushed
the
380-experiment-template-catalog
branch
from
October 12, 2026 10:52
b4802c8 to
cdf228d
Compare
Contributor
Author
Collaborator
|
Recommendation: keep closed in its current form. Reviewed head
Required action:
Coverage and outstanding work: #380's design-only run/study templates and unsupported-task distinction are described, but a closed PR is not actionable for merge and the full scope/outstanding-work report is unavailable. Verification limits: No claim of a completed source review or source defect. |
…iment-template-catalog # Conflicts: # examples/README.md # examples/library/catalog.yaml # tools/check_example_library.py # tools/example_library_checks.py
doublewhy
added a commit
that referenced
this pull request
Oct 25, 2026
Dev's source at ecd7b20, where #1458 merged after #1444, matches no admitted capture, so the evidence checkers and the tests that load the current capture fail on every branch built on it. This branch changes no package source. collect_research_evidence.py ran both pipelines on it: 38 of 42 specification stages passed (4 preregistered non-executions) and all 20 formal observations matched their expected outcomes.
Brad-Edwards
added a commit
that referenced
this pull request
Nov 11, 2026
…gates (#1474) * ci(policy): read git path listings with -z so quoted names reach the gates Without -z, git quotes path names that contain a byte above 0x80 under the default core.quotePath, a control character, a double quote or a backslash. The nox runner dropped such names from the hygiene selection, tools/policy/common.changed_paths handed git's quoted text to the repository policy and JSON artifact checks, and the schema publication base listing dropped them. List with -z and split on NUL in all three, as check_requirement_governance.py and verification_plan.py already do. changed_paths now runs one listing for its three comparisons. * test(policy): list a name with a newline in the nul path listing test Without -z, git quotes a newline in a name as it does any other control character. The test now covers such a name next to the tab, non-ASCII and double-quote names; dev's three helpers fail on it alone. * ci(policy): read nul path listings as bytes so carriage returns in names survive Text mode turned a carriage return or CRLF in a name into LF before the NUL split, so the runner dropped the name and the policy helpers returned a path that does not exist. tools.policy.common.git_paths now runs each listing with -z, reads bytes and decodes every name as strict UTF-8, like the repository's other -z listings. The runner, changed_paths and the requirement gate's merge-time diff use it; the schema publication listing reads bytes the same way. The regression test drives real repositories with CR, CRLF, LF, tab, quote, non-ASCII and space-edged names through the hygiene selection, the 500-line cap, the base schema publication manifest and the merge listing, and checks that a name that is not UTF-8 stops each listing. * ci(proof): fetch the locked isabelle archive from the official release locator The Cambridge mirror's dist directory now holds Isabelle2026 only, so the locked Isabelle2025-2_linux.tar.gz answers 404 there and canonical / generic-tool-local-inputs fails exact verification, which skips the checks, integration and proof jobs. The official release locator serves the same bytes; the fetch still verifies the unchanged lock size and SHA-256. * test: record combined-source research captures for issue #1473 Dev's source at ecd7b20, where #1458 merged after #1444, matches no admitted capture, so the evidence checkers and the tests that load the current capture fail on every branch built on it. This branch changes no package source. collect_research_evidence.py ran both pipelines on it: 38 of 42 specification stages passed (4 preregistered non-executions) and all 20 formal observations matched their expected outcomes. * test: record the runtime mapping capture for the merged source * Fix current repository policy checks for PR 1474 --------- Co-authored-by: Brad Edwards <j.bradley.edwards@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds validated experiment run and study design templates, and supports the current task-validation scope in issue #1519.
raes semantic validate --contract experiment-task-v1now accepts the reference task and rejects the invalid task fixture. Experiment loaders can check a supplied validated task against the plan's task id and any declared version.The existing branch is synchronized with current
devby a real merge. Scenario-template and catalog improvements from the original stack are retained where they are still absent fromdev.Requirement UIDs
Related Issues
Closes #380
Closes #1519
Closes #1563
ADR Impact
Changes
experiment-authoring-input-v1, with strict YAML loading and validation commands.experiment-task-v1in the existing conformance registry with a conservative structural validation claim, consistent with the authoring-input and study contracts.task=ExperimentTaskModeltoparse_experiment_specandload_experiment_spec. Task ids must match; a declared reference version must match. Mismatches raise a sanitizedtask-reference-mismatchvalidation error. No task is fetched automatically.ExperimentSpecValidationError, repairing the documented template-loader behavior from Experiment spec loader raises a bare ValueError on an unquoted invalid date #1563.Test Plan
test_issue_1519_experiment_tasks.py,test_experiment_authoring.py,test_semantic_cli.py,test_issue_380_experiment_templates.py,test_issue_378_scenario_templates.py,test_issue_381_example_catalog_metadata.py, andtest_example_library_policy.py: 149 passed.tools/check_example_library.py,tools/check_repo_policy.py --base-rev origin/dev,tools/check_specification_coverage.py, andtools/check_formal_semantic_validation.pypass. Ruff check and format checks pass for the changed Python files.tools/policy/requirement_order.yaml; its repository policy stage passed. No policy rule was relaxed. Repository-wide test, integration, fuzz, coverage, and completion verification remain CI responsibilities.Ground Control Checks
check_repo_policy.py --base-rev origin/devpassed before and after implementation.Traceability
implementations/python/packages/raes_conformance/conformance/validators.py,implementations/python/packages/raes_contracts/experiment_spec.py,examples/library/,tools/check_example_library.py,tools/example_library_checks.py,examples/README.md,docs/explain/getting-started.mdimplementations/python/tests/test_issue_1519_experiment_tasks.py,implementations/python/tests/test_issue_380_experiment_templates.py,implementations/python/tests/test_issue_378_scenario_templates.py,implementations/python/tests/test_issue_381_example_catalog_metadata.py,implementations/python/tests/test_example_library_policy.pyChecklist
Documentation
Updated
examples/README.md,examples/library/catalog.yaml, anddocs/explain/getting-started.md. The task-template absence is an explicit accepted option in #1519. No required work from #380 or #1519 is moved to a follow-up. The separately tracked #1520 policy question remains outside this change.