Repository navigation
Expectation flags, shared expectations on scenario sets, and per-rule results - #101
Merged
Merged
Conversation
Expected behaviour entries can be plain strings or objects
{"id", "text", "required"}. A non-required item is graded and reported
but never changes the severity; the judge model sees only the text.
Required items without an id are stored as plain strings, so scenarios
that use no flags keep their content and hash.
- scenarios/expectations.py: validation, stored form, form parsing.
- ScenarioSetVersion.shared_expectations (migration 0006), included in
the set hash only when non-empty. Publishing carries the latest rules
over; set_shared_expectations publishes a new version with every
scenario's frozen revision. Revert restores the old version's rules,
and version diffs show flag and rule changes.
- Worker passes the set's shared rules to infra/engine.py, which merges
them through simpleaudit.expectations. An engine without expectation
objects gets the texts only.
- Scenario editor: row editor with a "counts" checkbox; set page:
shared expectations panel.
- Run page: "Shared checks" panel per rule (violated / met / n/a /
missing, links to violating scenarios), read from list_summary.
- API accepts objects; export uses format_version 3 only when objects or
shared rules are present; import accepts 2 and 3. Generated scripts
apply the shared rules.
- docs/domain-model.md documents the fields.
Refs #100, SimulaMet/SimpleAudit#157
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
- demo audit S03 and agentic D-pack: "May ..." lines become
{"text", "required": false}, since grading no longer reads wording.
- Pack seeding stores expected behaviour in Studio's form.
- Set export takes ?version= (the version being viewed) and records
set_version; dialog texts mention flags and shared expectations.
- Tests: set export/import round trip with flags and shared rules.
The flags and shared expectations take effect only with an engine that has simpleaudit.expectations (released in 0.8.0).
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements most of #100 (Studio side of SimulaMet/SimpleAudit#157, core PR SimulaMet/SimpleAudit#158). cc @kelkalot
What changes
expected_behaviorentries can be plain strings or{"id", "text", "required"}. A non-required item is graded and reported but never changes the severity; the judge model sees only the text. Required items without an id are stored as plain strings, so existing scenarios keep their content and hash. Changing a flag creates a new revision.ScenarioSetVersion.shared_expectations(migration0006), part of the set hash only when non-empty. Publishing carries the latest rules over; saving new rules publishes a version with every scenario's frozen revision; revert restores the old version's rules; version diffs show flag and rule changes.infra/engine.py, which merges them viasimpleaudit.expectations(source: "shared"). With an engine that lacks expectation objects (0.7.0), only texts are passed and every item counts; nothing breaks.ScenarioResult.list_summaryso no transcript is loaded.format_version3 only when objects or shared rules are present; import accepts 2 and 3; generated scripts apply the shared rules withapply_shared_expectations.docs/domain-model.md.Not in this PR
required: false.Checks
uv run pytest -m "not embedded_hatchet": 1865 passed, 4 skipped.infra/tests/test_expectation_flags.pyandtest_codegen.py; also run against the core branch (all pass, including the merge test that is skipped on 0.7.0).uv run ruff check .passes;makemigrations --checkreports no changes.