Repository navigation
fix(rubrics_verifier): a context fact naming an env as the task wrote it resolves in a bundle run - #641
Merged
Conversation
… it resolves in a bundle run A bundle run namespaces the ids it writes as <bundle id root>/<id>, so metadata keyed by an env id, such as env_trigger_state, holds the namespaced key, while a context_facts_for_judge path still names the id as the task wrote it and rendered "(not available)". A dict segment that misses is now also tried as <bundle id root>/<segment>, the root being the run's task id up to its last segment, so the judge sees the fact as it would in the store the task came from. A key as written still wins; a task the store holds under its own id gets no fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…context facts resolve through them Replaces the previous commit's inference from the task id with the renames the bundle actually made. Resolving a bundle task now records each id it rewrote, as the task names it, against the id it names now: one of the bundle's entities by its name, and an output a step writes by the name the step gave it, which sits under the task id rather than the bundle root. The bundle stores that map on the task (Task.entity_ids, like requires; absent when nothing was renamed), and Task.run puts it in context.metadata["entity_ids"], so every runner of a bundle-written task has it, not only agent-env run. run_entity_id(metadata, id) resolves through it, and context facts use it for a key that misses. Translating a plan to registry ids carries over, since the map is derived from the references and outputs it rewrites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bundle run reuses the task version its ledger recorded when the task's inputs are unchanged, so a bundle written before tasks carried entity_ids kept a stored task without them, and its runs still resolved renamed ids as written. The ledger now rewrites a task whose store's latest version doesn't record the entity_ids its bundle gives it, saying so. Digests are unchanged, so no other entity is rewritten and no image rebuilt; a task with no renamed ids is reused as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n to, and an earlier orphan is rewritten ResolvedEntry.entity_ids now pairs each reference field of a task's steps as written with the same field as resolved, so it holds exactly the ids the bundle rewrote: its own entities, the outputs its steps write, alias rewrites, and a registration's translation, which all rewrite the config. A reference the plan adds without rewriting a field, as --agent's for a run's solver, no longer enters it, so a run with --agent stores the task as written and the next run reuses it. Outputs no longer need the name their step gave them. A version an interrupted run left unrecorded is adopted only when it records the entity_ids the task gives now, so a task an earlier release wrote and didn't record is written again too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A bundle rewrites the ids a task's steps name in the fields they declare as references (
entity_refs):<bundle id root>/<name>, for example@local/~/runs/pn/world;<task id>/<name>.Run metadata keyed by those ids, such as
env_trigger_state, therefore holds the rewritten keys. But anything else the task author wrote that names an entity is still the id as written. Acontext_facts_for_judgepath likeenv_trigger_state.world.statuses.wf-1is one example. So in a bundle run every such fact rendered "(not available)", and judges graded without facts they get when the same task runs from the store. A judge checking whether scheduled world events fired then passed a run where one never did.This makes the rename explicit, instead of leaving each reader to guess it:
ResolvedEntry.entity_idspairs each reference field of its steps as written with the same field as resolved. So it holds exactly the ids the bundle rewrote: its own entities, the outputs its steps write, alias rewrites, and a registration's translation, which all rewrite the config. A reference the plan adds without rewriting a field, such as--agent's for one run's solver, adds nothing.Task.entity_ids, besiderequires. It's absent when nothing was renamed, so a task stored as written is unchanged.Task.runputs the map incontext.metadata["entity_ids"]. Every runner of a bundle-written task therefore has it, not onlyagent-env run.run_entity_id(metadata, id)inagent_env.task_step.contextreturns the id the run's task gave an entity it names asid, elseiditself. Context facts use it for a dict key that misses, and the key as written still wins.bundle register) rewrites the config, so the map follows.The path syntax can't hold a rewritten id, and rewriting authored strings would mean finding every field that might name one. So resolution goes through the map, and code or tools that read a stored run can use the same map.
How it was tested
Unit:
tst/unit/bundle/resolve_test.py: a task records the env and output ids it rewrote, and nothing for a store id; a reference added without rewriting a field adds nothing.tst/unit/bundle/ledger_test.py: a task written before it recorded its map is written again, saying why, and then reused; so is one an interrupted earlier run left unrecorded.tst/unit/bundle/materialize_test.py: the written task carries the map.tst/unit/task/entity_ids_test.py: the map round-trips and is left out when empty; a run puts it in the metadata, whererun_entity_idresolves through it; a task stored as written adds nothing.tst/unit/task_step/verifiers/test_rubrics_verifier_context_facts_bundle_ids.py: a renamed env and a renamed output are found; the key as written wins; with no rename, keys resolve only as written; the judge prompt shows the resolved fact.tst/unit/bundle,tst/unit/task,tst/unit/task_stepandtst/unit/runnerpass: 4,061 passed, 11 skipped.tst/integration/cli/run_bundle_built_agents_test.pypasses, including the--agentswap.End to end. A benchmark task with a scripted world (timed events and colleague agents) was run as a bundle with
agent-env runon a Linux VM. The build combined this branch with #640, so a validity verifier can fail the task.claude-code→@local/agentenv-framework/agents/claude-code)"armed"and failed the world rowVerifierNotPassingErroron its first attempt, and the scoring step never ran. The quality judges had scored it 0.83."fired", score 1.0The stored task carries the same map. Before this change, the same broken run's judge saw every trigger fact as "(not available)" and passed it.
A dry run of a real bundle folder an earlier build wrote plans the task as
v2 (its stored version doesn't record the ids the bundle gave what it names), and leaves everything else unchanged. A folder this build wrote stays atv1, unchanged.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no new actionable issues were found.Summary
The PR records rewritten entity IDs on tasks and exposes them to context-fact lookup.
Reviews (7) · Last reviewed commit: "fix(bundle): entity_ids are the ids a ta..." · Reviewed by Greptile