TRACE

Trajectory-Relative Agent Context comprEssion

A tool for verifying context-compression policies for long-horizon agents. It answers one question about a compression policy: do the agent's next actions get worse when its raw history is replaced by a summary?

"Worse" is measured in two channels, both from real execution:

• blocked — the action failed, per the environment's own error contract;
• refetch — the action re-ran something already executed before.

Measurement is trajectory-relative: for each real compaction boundary the environment is rebuilt to the identical pre-compaction state twice, and the agent is shown either its raw history (PRE, the control) or a summary (POST). Both arms replay the same code prefix, so the agent's own run-to-run variance is differenced out. A boundary's score is the difference between the two arms; it is not a property of a summary on its own, and no reference summary is required.

The repository contains the verifier, a policy optimizer that uses it as an objective, and a frozen 590-boundary AppWorld cohort to run them against.

Install and run

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt          # matplotlib, and nothing else

python tests/test_core.py                # 24 unit tests, no data needed
python tests/test_optimize.py            # 28 tests, base policy is the only input
python tests/test_reproduction.py        # 6 tests against the frozen data (~5 min)

python -m trace_cc.scan  --data data --out results/scan
python -m trace_cc.plots --scan results/scan --out figures

Import name. The project is TRACE and the distribution is trace, but the import package is trace_cc (cc = Context Compression). Python ships a stdlib module named trace, and the stdlib sits earlier on sys.path than site-packages — so a package installed as trace would be permanently shadowed and unimportable, while a local ./trace/ directory would shadow the stdlib in the other direction. trace_cc avoids both.

trace_cc.scan scores every boundary in a collection and writes six plot-ready CSVs. It takes about three minutes on the shipped cohort, most of it bootstrap resampling:

boundaries=590 steps=20133 -> results/scan
  k=1  dU=+0.131[+0.101,+0.160]*  dE=+0.108[+0.086,+0.132]*  dR=+0.031[+0.005,+0.058]*
  ...
  k=5  dU=+0.509[+0.432,+0.588]*  dE=+0.280[+0.234,+0.327]*  dR=+0.277[+0.215,+0.341]*

dU is the union channel (a step wasted for either reason), dE blocked, dR refetch, each as mean(POST) − mean(PRE) per boundary over the first k post-boundary actions; * marks a bootstrap 95% CI excluding zero. The CSVs are byte-identical to the shipped tables in data/burden_scan/, which tests/test_reproduction.py checks by sha256.

Optimizing a policy

trace_cc.optimize searches over compression policies using the verifier as the objective:

base policy P0
    -> pick evidence boundaries (stratified)
    -> generate candidate summaries there            [compressor LLM]
    -> score each by POST rollouts vs frozen PRE     [AppWorld]
    -> build contrastive best/worst pairs
    -> propose improved policies                     [proposer LLM]
    -> screen candidates on HELD-OUT boundaries      [AppWorld]
    -> run the dev split end to end, twice per arm   [AppWorld]
    -> promote on pass^2 only if it beats the baseline

The output is in data/compression_policy_optimized/: five candidate policies and the exact prompt that produced them.

A policy is eleven editable text slots carved out of data/compression_policy_base/update_summary.jinja. The input blocks, the nine section headings and the ## Final Deliverable contract are immutable, and the decomposition roundtrips byte for byte, so "edit only the slots" is enforced rather than promised.

PRE rollouts are never re-run — the control does not depend on the summary — so only POST is executed per candidate.

Three constraints in the loop that are easy to get wrong:

• The proposer sees the current policy and, per boundary, two summaries labelled better and worse. Nothing measured. No objective, no blocked/refetch split, no rate ceiling, no scores, no failure-rate statistics. Given measurements, a proposer back-fits a causal story to them: in one run it concluded that Python variables do not persist across cells, which the agent's own instructions guarantee they do, and three of five candidates wrote that into the policy. build_proposer_prompt takes no argument that could carry a measurement.
• Evidence boundaries are stratified across blocked causes in proportion to their population frequency. Selecting the worst boundaries by score alone skews the failure mix, and the skew ends up in the policy.
• Promotion is decided end to end, on pass^2. The dev split is run twice per arm and a task counts only if it passes in both runs, so a policy cannot win on tasks it holds half the time. Boundary-level scoring is a screen ahead of it: it scores one boundary at a time against a frozen control, and so never sees a policy consume its own earlier summary. A candidate that does not beat the baseline is not promoted, and a leader whose bootstrap CI crosses zero is reported as unproven rather than as a win.

Dependencies

Only open-source packages, and deliberately very few.

Need	Package	License
Verifier, stats, scan, optimizer logic	(standard library only)	—
Figures	matplotlib	matplotlib (BSD-compatible)
Rendering a policy	jinja2	BSD-3-Clause
Collecting new rollouts (optional)	appworld	Apache-2.0
Reference model adapter (optional)	openai, tiktoken	Apache-2.0, MIT

No NumPy, SciPy or pandas: the bootstrap and resampling tests are implemented directly in trace_cc/stats.py, so results do not shift with a BLAS build or a SciPy release. Full license inventory: docs/DEPENDENCIES.md.

Layout

trace_cc/
  core.py              trajectory parsing, action signatures, io   [stdlib]
  verifier.py          the blocked/refetch contract + burden arithmetic
  stats.py             bootstrap and resampling tests              [stdlib]
  scan.py              score a collection -> six plot-ready CSVs
  plots.py             figures                                    [matplotlib]
  collect/
    boundaries.py      boundary + PRE/POST context construction   [stdlib]
    checks.py          admissibility checks on a rollout set      [stdlib]
    runner.py          closed-loop rollout                        [appworld]
  optimize/
    policy.py          the search variable: 11 slots + immutable contract
    select.py          stratified evidence choice, candidate selection [stdlib]
    loop.py            orchestration, LLMs and rollouts injected as adapters
data/                  frozen artifacts, ~73 MB. See docs/DATA.md
docs/                  protocol, data glossary, reproducibility, collection
tools/                 one-off exporters kept for auditability
tests/                 unit tests + reproduction tests

The analysis path never imports collect.runner, which is what keeps its dependency list to one package.

Documentation

• docs/PROTOCOL.md — how a boundary, a rollout and the two arms are defined, and what counts as blocked or refetch.
• docs/DATA.md — what each data/ directory contains.
• docs/REPRODUCIBILITY.md — what is byte-reproducible and what is not.
• docs/COLLECTION.md — collecting new rollouts against a live environment.
• docs/DEPENDENCIES.md — licenses and attribution.

Provenance

Extracted from a larger research codebase. The extraction is verified rather than assumed: the scan regenerates the original tables byte for byte, and all 590 boundaries rebuild with all four identity hashes matching.

The rollout data derives from AppWorld tasks and environments (Apache-2.0); docs/DEPENDENCIES.md records the attribution. This repository does not yet carry a license file — add one before distributing, since the frozen data is a derived work.