Skip to content

fix(rubrics_verifier): a context fact naming an env as the task wrote it resolves in a bundle run - #641

Merged
earakely-scale merged 4 commits into
mainfrom
edgararakelyan/context-facts-bundle-ids
Nov 12, 2026
Merged

earakely-scale merged 4 commits into
mainfrom
edgararakelyan/context-facts-bundle-ids

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

A bundle rewrites the ids a task's steps name in the fields they declare as references (entity_refs):

  • one of the bundle's own entities becomes <bundle id root>/<name>, for example @local/~/runs/pn/world;
  • an output a step writes becomes <task id>/<name>.

Run metadata keyed by those ids, such as env_trigger_state, therefore holds the rewritten keys. But anything else the task author wrote that names an entity is still the id as written. A context_facts_for_judge path like env_trigger_state.world.statuses.wf-1 is one example. So in a bundle run every such fact rendered "(not available)", and judges graded without facts they get when the same task runs from the store. A judge checking whether scheduled world events fired then passed a run where one never did.

This makes the rename explicit, instead of leaving each reader to guess it:

  • Resolving records the map. For a task, ResolvedEntry.entity_ids pairs each reference field of its steps as written with the same field as resolved. So it holds exactly the ids the bundle rewrote: its own entities, the outputs its steps write, alias rewrites, and a registration's translation, which all rewrite the config. A reference the plan adds without rewriting a field, such as --agent's for one run's solver, adds nothing.
  • The task stores it. The bundle stores the map on the task as Task.entity_ids, beside requires. It's absent when nothing was renamed, so a task stored as written is unchanged.
  • A run exposes it. Task.run puts the map in context.metadata["entity_ids"]. Every runner of a bundle-written task therefore has it, not only agent-env run.
  • One function resolves through it. run_entity_id(metadata, id) in agent_env.task_step.context returns the id the run's task gave an entity it names as id, else id itself. Context facts use it for a dict key that misses, and the key as written still wins.
  • Registry translation carries over. Translating a plan to registry ids (bundle register) rewrites the config, so the map follows.
  • Bundles written before this change pick it up. The ledger rewrites a task when the store's latest version doesn't record the map its bundle gives it now. It also no longer adopts such a version that an interrupted run left unrecorded. Digests are unchanged, so no other entity is rewritten and no image is rebuilt.

The path syntax can't hold a rewritten id, and rewriting authored strings would mean finding every field that might name one. So resolution goes through the map, and code or tools that read a stored run can use the same map.

How it was tested

Unit:

  • tst/unit/bundle/resolve_test.py: a task records the env and output ids it rewrote, and nothing for a store id; a reference added without rewriting a field adds nothing.
  • tst/unit/bundle/ledger_test.py: a task written before it recorded its map is written again, saying why, and then reused; so is one an interrupted earlier run left unrecorded.
  • tst/unit/bundle/materialize_test.py: the written task carries the map.
  • tst/unit/task/entity_ids_test.py: the map round-trips and is left out when empty; a run puts it in the metadata, where run_entity_id resolves through it; a task stored as written adds nothing.
  • tst/unit/task_step/verifiers/test_rubrics_verifier_context_facts_bundle_ids.py: a renamed env and a renamed output are found; the key as written wins; with no rename, keys resolve only as written; the judge prompt shows the resolved fact.

tst/unit/bundle, tst/unit/task, tst/unit/task_step and tst/unit/runner pass: 4,061 passed, 11 skipped. tst/integration/cli/run_bundle_built_agents_test.py passes, including the --agent swap.

End to end. A benchmark task with a scripted world (timed events and colleague agents) was run as a bundle with agent-env run on a Linux VM. The build combined this branch with #640, so a validity verifier can fail the task.

World Map on the run Validity verdict Result
broken: one scheduled event delayed past the run, validity rubric unchanged the env, the universe artifact, and the agent from the shipped agents bundle (claude-code → @local/agentenv-framework/agents/claude-code) the judge saw the delayed event as "armed" and failed the world row The run failed at validity with VerifierNotPassingError on its first attempt, and the scoring step never ran. The quality judges had scored it 0.83.
intact the same three every event "fired", score 1.0 The scoring step ran after validity

The stored task carries the same map. Before this change, the same broken run's judge saw every trigger fact as "(not available)" and passed it.

A dry run of a real bundle folder an earlier build wrote plans the task as v2 (its stored version doesn't record the ids the bundle gave what it names), and leaves everything else unchanged. A folder this build wrote stays at v1, unchanged.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new actionable issues were found.

Summary

The PR records rewritten entity IDs on tasks and exposes them to context-fact lookup.

  • Bundle tasks keep the ids they rewrite for each run.
  • Rubric judges can find bundle facts by the task's original entity names.

Reviews (7) · Last reviewed commit: "fix(bundle): entity_ids are the ids a ta..." · Reviewed by Greptile

… it resolves in a bundle run

A bundle run namespaces the ids it writes as <bundle id root>/<id>, so metadata keyed by an env id, such as
env_trigger_state, holds the namespaced key, while a context_facts_for_judge path still names the id as the task
wrote it and rendered "(not available)". A dict segment that misses is now also tried as <bundle id root>/<segment>,
the root being the run's task id up to its last segment, so the judge sees the fact as it would in the store the
task came from. A key as written still wins; a task the store holds under its own id gets no fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 12, 2026 18:42
…context facts resolve through them

Replaces the previous commit's inference from the task id with the renames the bundle actually made. Resolving a
bundle task now records each id it rewrote, as the task names it, against the id it names now: one of the bundle's
entities by its name, and an output a step writes by the name the step gave it, which sits under the task id rather
than the bundle root. The bundle stores that map on the task (Task.entity_ids, like requires; absent when nothing was
renamed), and Task.run puts it in context.metadata["entity_ids"], so every runner of a bundle-written task has it, not
only agent-env run. run_entity_id(metadata, id) resolves through it, and context facts use it for a key that misses.
Translating a plan to registry ids carries over, since the map is derived from the references and outputs it rewrites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread src/agent_env/bundle/materialize.py Resolved
earakely-scale and others added 2 commits November 12, 2026 12:36
A bundle run reuses the task version its ledger recorded when the task's inputs are unchanged, so a bundle written
before tasks carried entity_ids kept a stored task without them, and its runs still resolved renamed ids as written.
The ledger now rewrites a task whose store's latest version doesn't record the entity_ids its bundle gives it,
saying so. Digests are unchanged, so no other entity is rewritten and no image rebuilt; a task with no renamed ids
is reused as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n to, and an earlier orphan is rewritten

ResolvedEntry.entity_ids now pairs each reference field of a task's steps as written with the same field as
resolved, so it holds exactly the ids the bundle rewrote: its own entities, the outputs its steps write, alias
rewrites, and a registration's translation, which all rewrite the config. A reference the plan adds without
rewriting a field, as --agent's for a run's solver, no longer enters it, so a run with --agent stores the task
as written and the next run reuses it. Outputs no longer need the name their step gave them.

A version an interrupted run left unrecorded is adopted only when it records the entity_ids the task gives now,
so a task an earlier release wrote and didn't record is written again too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit ef6b55a into main Nov 12, 2026
87 of 91 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/context-facts-bundle-ids branch November 12, 2026 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant