Repository navigation
Conversation
RepeatedExperimentResults.save() writes every run's meta, but load()
rebuilt each run as AuditResults(results), so the judge, target and
auditor settings were gone after a round trip. AuditResults.load() already
passes meta=data.get("meta"); the repeated loader now does the same.
AuditExperiment expands a pack name to its scenario list before calling
ModelAuditor.run_async, and _run_meta only records scenario_pack when it
gets a string. Runs started with AuditExperiment.run("safety") therefore
had no scenario_pack, while ModelAuditor.run("safety") had one. The
experiment now records the pack name itself, with the same isinstance
check; a list of scenarios still records none.
AuditResults.score is None when nothing was graded, so the example printed "Score: None/100". It now prints the same "n/a (nothing graded)" that summary() prints in that case.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three small follow-ups to the run metadata from #138, one commit each.
Keep each run's meta when loading repeated experiment results (
simpleaudit/repeated_results.py)RepeatedExperimentResults.save()writes every run'smeta, butload()rebuilt each run asAuditResults(results), so after a round tripmetawas{}.AuditResults.load()already passesmeta=data.get("meta"), and the repeated loader now does the same.New test
TestSerialization::test_save_load_preserves_each_runs_metafails onmainand passes here.Record the scenario pack in AuditExperiment runs' meta (
simpleaudit/experiment.py)AuditExperimentexpands a pack name to its scenario list before callingrun_async, and_run_metarecordsscenario_packonly when it gets a string. SoAuditExperiment.run("safety")gave runs withoutscenario_pack, whileModelAuditor.run("safety")gave one. The experiment now records the pack name with the sameisinstancecheck; a list of scenarios still records none.New test
TestAuditExperimentRepetitions::test_pack_name_is_recorded_in_each_runs_metafails onmain([None, None] == ['safety', 'safety']) and passes here.Show a missing score in the README results example (
README.md)AuditResults.scoreisNonewhen nothing was graded, so the example printedScore: None/100. It now printsn/a (nothing graded), the same textsummary()uses.Testing: full suite in a fresh Python 3.13 venv with
.[dev,tracing], run withpytest -p no:randomly -n 4.main(007eaf4) gave 1939 passed and 6 skipped; this branch gives 1941 passed and 6 skipped. The only per-test difference is the two new tests.ruff check --select E9,F821 .is clean.AI disclosure: implemented with Claude Code.