Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions docs/PLUGIN.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ event): no nudge is counted, and there is no one left to redirect.
| Still stuck after the redirect | terminal abort, in autonomous runs | a one-line **notice** to the person, ending with how to stop the turn on that agent and in that mode (§3); the plugin never stops a turn, and `loopguard: stop` is read as `nudge` | Not every agent's hooks can stop a turn (Codex and Copilot ignore `continue: false`), and the person decides: the benchmark guard's own design note says "interactive callers should map it to a hand-off" |
| Micro-skills' reading | a domain vocabulary per benchmark (Python/git, Node/web); the proposer's digest counts edited Python files, lists test invocations and flags Python import errors, and its prompt leads with the test command | **generic**, whatever the repository: every ecosystem's interpreters, package tools and test runners at once, builds and checks (`make`, `tsc`, `npm run build`, `cargo check`, ...) as their own intent, every edited file, and a language-neutral digest (failing commands with a normalized error line, paths relative to the repository) and prompt, with tests one topic among several. `vocabulary=python-git` or `node-web` selects an experiment's | A person's repository may be in any language, or several; the experiments keep the paper's reading (its prompt is pinned by a test) |
| Micro-skills | on in the arms that measured it: the proposer every 5 tasks (on Codex, every 10, over the free layer) | **off by default**; `on` runs the free layer under the proposer, every `microskills_propose_every` sessions (3 by default: after the 2nd, 5th, 8th, ...), since a real session usually carries more work than one benchmark task | The proposer calls a model on the person's plan, and what it injects is derived from command output a repository controls: turning it on is the person's call. In the experiments the free layer alone formed almost nothing once the harness set each environment up properly, so `on` keeps the proposer |
| Optimizer model | the agent's own model, so overhead could not look cheaper than its arm | the cheapest capable model on the user's own agent CLI (`paid_model`; when empty, `haiku` on Claude Code, the agent's default model on Codex and Pi, and on Copilot a concrete `COPILOT_MODEL` or a BYOK provider's Haiku): the proposer when micro-skills is on, the loop guard's judge and redirect writer whenever the guard is on | Every call is still charged to the session ledger |
| Optimizer model | the agent's own model, so overhead could not look cheaper than its arm | the cheapest capable model on the user's own agent CLI (`paid_model`; when empty, `haiku` on Claude Code, the agent's default model on Codex and Pi, the session's own model on opencode 2.x, and on Copilot a concrete `COPILOT_MODEL` or a BYOK provider's Haiku): the proposer when micro-skills is on, the loop guard's judge and redirect writer whenever the guard is on | Every call is still charged to the session ledger |

## 3. Verified agent behaviour

Expand Down Expand Up @@ -356,8 +356,10 @@ settings would be):
only after `opencode service restart`.
* Model side calls run `opencode run` with `--pure` (1.x) or `--standalone` (2.x, so the shared
service's other plugins — this one included — cannot fire inside them) and `bash`, `edit`,
`webfetch`, `websearch` denied; a `read`, `task` or `*` denial stalls 1.19.5 at start-up. On
2.x a reply that used no tool carries no usage, so its cost is estimated.
`webfetch`, `websearch` denied; a `read`, `task` or `*` denial stalls 1.19.5 at start-up. With
no `opencode_paid_model` or `paid_model`, a 2.x call runs on the model the session itself is
on, not the CLI's configured default. On 2.x a reply that used no tool carries no usage, so
its cost is estimated.

**End to end, as a new user, 2026-10-14** — on each agent a fresh home followed `README.md` and
this plugin's guide step by step (Claude Code 2.1.300, Codex 0.153.1, Copilot CLI 1.0.83, Pi
Expand Down
11 changes: 7 additions & 4 deletions plugin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -432,10 +432,13 @@ when its turn ends and a deleted session when it is deleted. Quitting the TUI gi
chance to say so: that session is folded by the next session's start, once its export has been
quiet for 10 minutes.

**Model calls.** The proposer and the loop guard's judge run through `opencode run` on your
default opencode model (`paid_model` names another): tools that change anything are denied,
the session is deleted afterwards, and no other plugin runs (`--pure` on 1.x; `--standalone` on
2.x). On 2.x a reply that used no tool carries no usage, so its cost is estimated.
**Model calls.** The proposer and the loop guard's judge run through `opencode run` on
`opencode_paid_model` (`paid_model` names another model for every agent). With neither set they
follow the model the session itself is running — the 2.x plugin reports it from
`session.step.started` — instead of the CLI's configured default, which can be a free model the
session has already moved past. Tools that change anything are denied, the session is deleted
afterwards, and no other plugin runs (`--pure` on 1.x; `--standalone` on 2.x). On 2.x a reply
that used no tool carries no usage, so its cost is estimated.

**Verified live,** on opencode 1.19.5 and 2.2.0 (Python 3.12), with a scripted local model
endpoint and, on 1.19.5, the free `opencode/big-pickle` model; the 2.x entry is also exercised
Expand Down
13 changes: 13 additions & 0 deletions plugin/opencode/tokeneconomizer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -342,11 +342,15 @@ const setupV2 = async (ctx: any) => {
const records = new Map<string, any[]>(); // sessionID -> messages, in the export's shape
const steps = new Map<string, any>(); // assistantMessageID -> assistant message
const names = new Map<string, string>(); // tool call id -> tool name
// The model each session actually runs on, in the CLI's `provider/model#variant` form: the
// plugin's own calls follow it, rather than whatever the CLI's default happens to be.
const models = new Map<string, string>();

const base = (sessionID: string, transcript = "") => ({
session_id: sessionID,
transcript_path: transcript,
cwd: directory,
...(models.get(sessionID) ? { model: models.get(sessionID) } : {}),
});
const notify = (text: unknown) => {
if (typeof text === "string" && text) console.error(text);
Expand Down Expand Up @@ -495,6 +499,15 @@ const setupV2 = async (ctx: any) => {
if (t === "session.step.started") {
const m = assistant(id, p.assistantMessageID);
m.info.modelID = p.model?.id ?? p.model?.modelID ?? m.info.modelID;
// `session.step.started` names the model the turn runs on; the plugin's own calls use
// it when no `paid_model` is configured, so a paid provider is not looked past.
const provider = p.model?.providerID ?? p.model?.provider;
const modelID = p.model?.id ?? p.model?.modelID;
if (provider && modelID)
models.set(
id,
`${provider}/${modelID}${p.model?.variant ? `#${p.model.variant}` : ""}`,
);
} else if (t === "session.step.ended") {
const m = assistant(id, p.assistantMessageID);
m.info.cost = p.cost ?? 0;
Expand Down
102 changes: 102 additions & 0 deletions tests/test_live_hooks.py
Original file line number Diff line number Diff line change
Expand Up @@ -1698,3 +1698,105 @@ def fail(tools, **kw):
monkeypatch.setattr(toolcheck, "measure", fail)
assert cli.main(["--host", "claude", "--cwd", repo, "toolprune", "measure", "Workflow"]) == 1
assert capsys.readouterr().out.strip() == "not measured: `claude -p` exited 1: boom"


# ---- the model a session runs on -----------------------------------------------------------
def test_the_model_the_plugin_sends_is_kept_in_the_session_state(env, repo):
"""`session.step.started` names the model; the 2.x plugin sends it with every hook, and the
session keeps it so a later judge or fold does not fall back to the CLI's default."""
from tokeneconomizer.plugin import hosts

assert hosts.OpenCode().session_model({"model": "opencode-go/m#high"}) == "opencode-go/m#high"
assert hosts.Host().session_model({"model": "opencode-go/m#high"}) == ""
base = {"session_id": "ses_model", "cwd": repo, "model": "opencode-go/m#high"}
call(env, dict(base, hook_event_name="UserPromptSubmit", prompt="hi", prompt_id="t1"),
host="opencode")
call(env, dict(base, hook_event_name="Stop", prompt_id="t1", stop_hook_active=False),
host="opencode")
s = store_for(env, repo, host="opencode").read("sessions/ses_model.json")
assert s["model"] == "opencode-go/m#high"


def test_a_side_call_follows_the_session_model_only_on_opencode(monkeypatch):
"""No `paid_model`: opencode takes the session's own model. An explicit setting still wins,
and another host's default (Claude's haiku) is not replaced by a session model."""
from tokeneconomizer.plugin import sidecall
from tokeneconomizer.plugin.config import LiveConfig

seen = []
monkeypatch.setattr(sidecall, "opencode_call",
lambda prompt, model="", **kw: (seen.append(model), ("", 0))[1])
monkeypatch.setattr(sidecall, "claude_call",
lambda prompt, model="", **kw: (seen.append(model), ("", 0))[1])
cfg = LiveConfig()
sidecall.runner_for("opencode", cfg, session_model="opencode-go/m#high")("x")
assert seen[-1] == "opencode-go/m#high"
cfg.opencode_paid_model = "explicit/model"
sidecall.runner_for("opencode", cfg, session_model="opencode-go/m#high")("x")
assert seen[-1] == "explicit/model"
sidecall.runner_for("claude", cfg, session_model="opencode-go/m#high")("x")
assert seen[-1] == "haiku"


def test_the_judge_follows_the_session_model(env, repo, monkeypatch):
"""The loop guard's judge runs through the model the session is on."""
from tokeneconomizer.plugin import judge, sidecall

seen = {}

def fake_runner_for(host, cfg, role=None, session_model=""):
seen[role] = session_model
return lambda prompt, _m="agent": ("", 0)

monkeypatch.setattr(sidecall, "runner_for", fake_runner_for)
monkeypatch.setattr(sidecall, "paused", lambda *a, **k: None)
ctx = LiveContext.build(hosts.OpenCode(), repo, env)
ctx.store.write("sessions/ses_j.json", {"session_id": "ses_j",
"model": "opencode-go/j#high"})
ctx.store.write("judge/ses_j.request.json", {"id": "job1", "digest": "d", "threshold": 0})
judge.run_job(ctx, "ses_j", "job1")
# The judge follows the session; the nudge writer keeps the host's own default.
assert seen["judge"] == "opencode-go/j#high"
assert seen["nudge"] == ""


def test_a_fold_takes_the_model_from_the_session_state(env, repo, tmp_path, monkeypatch):
"""The fold hands the proposer the model the plugin's hooks recorded for the session. The
log names another model, so a fallback to it would show: `opencode run --model` needs
`provider/model`, and the log keeps the bare id."""
import tokeneconomizer.microskills as ms
from tokeneconomizer.plugin import fold, sidecall

from fixtures import opencode_session_export

seen = {}

def fake_runner_for(host, cfg, role=None, session_model=""):
seen["model"] = session_model
return lambda prompt, _m="agent": ("", 0)

monkeypatch.setattr(sidecall, "runner_for", fake_runner_for)

class FakeStage:
def __init__(self, *a, **kw):
pass

def after_task(self, *a, **kw):
return 0, "fake"

def render(self):
return ""

def injected_ids(self):
return []

monkeypatch.setattr(ms, "PrefixStage", FakeStage)
export = tmp_path / "ses_s.json"
export.write_text(opencode_session_export("ses_s"), encoding="utf-8")
ctx = LiveContext.build(hosts.OpenCode(), repo, env)
ctx.config.microskills = "envfacts+distiller"
ctx.store.write("sessions/ses_s.json", {"session_id": "ses_s",
"model": "opencode-go/s#high"})
rec = fold.fold_session(ctx, "ses_s", transcript_path=str(export))
assert rec is not None
assert seen["model"] == "opencode-go/s#high"
20 changes: 19 additions & 1 deletion tests/test_opencode_plugin.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,12 @@
location: { directory: process.env.TE_WORK },
session: { hook: async (name: string, fn: any) => { hooks["session." + name] = fn; } },
tool: { hook: async (name: string, fn: any) => { hooks["tool." + name] = fn; } },
event: { subscribe: async function* () {} },
event: { subscribe: async function* () {
yield { type: "session.step.started", properties: { sessionID: "ses_t", assistantMessageID: "a1", model: { id: "te-model", providerID: "te", variant: "high" } } };
} },
});
// The event stream is consumed off the setup path; let it land before the hooks that carry it.
await new Promise((r) => setTimeout(r, 200));
await hooks["session.prompt"]({ sessionID: "ses_t", messageID: "m1", prompt: { text: "say hi" } });
await hooks["tool.execute.after"]({ sessionID: "ses_t", status: "completed", tool: "bash",
id: "c1", input: { command: "echo hi" }, result: { content: "hi", metadata: { exit: 0 } } });
Expand Down Expand Up @@ -113,6 +117,20 @@ def test_the_plugins_hooks_open_no_console_window(tmp_path):
assert all(r["visible"] for r in control if r["hook"]), control


def test_the_plugin_sends_the_model_the_session_runs_on(tmp_path):
"""`session.step.started` names the model the turn runs on; the plugin passes it with the
hooks and the session state keeps it, so the judge and the fold need not fall back to the
CLI's configured default."""
_run(
tmp_path,
os.path.join(REPO, "plugin", "opencode", "tokeneconomizer.ts"),
"model",
)
(state,) = (tmp_path / "te-model").glob("repos/*/opencode/sessions/ses_t.json")
doc = json.loads(state.read_text(encoding="utf-8"))
assert doc.get("model") == "te/te-model#high", doc


class _Chat(BaseHTTPRequestHandler):
"""An OpenAI-compatible chat endpoint that answers every request with "done"."""

Expand Down
5 changes: 5 additions & 0 deletions tokeneconomizer/plugin/controller.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,11 @@ def _session(self):
"cwd": self.host.cwd(self.p, self.ctx.env), "turns": [], "nudges": 0,
"aborts": 0, "judge_tokens": 0.0, "writer_tokens": 0.0,
**self.host.session_state(self.ctx.env)}
# The model the session runs on, when its payload names one: the plugin's own calls
# follow it (`sidecall`) instead of the CLI's default.
model = self.host.session_model(self.p)
if model:
s["model"] = model
return s

def _save(self, s):
Expand Down
13 changes: 10 additions & 3 deletions tokeneconomizer/plugin/fold.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,8 @@ def _microskills_config(ctx, profile):
max_tokens=ctx.config.microskills_max_tokens)


def _fold_microskills(ctx, adapter, run_dir, session_id, injected=None, imported=False):
def _fold_microskills(ctx, adapter, run_dir, session_id, injected=None, imported=False,
model=""):
profile = ctx.config.microskills
if profile == "off":
return None
Expand All @@ -83,7 +84,8 @@ def _fold_microskills(ctx, adapter, run_dir, session_id, injected=None, imported
# layer learns from it, and the proposer waits for sessions the plugin sees end.
runner = None
if icfg.paid and not imported:
runner = (sidecall.runner_for(ctx.host.name, ctx.config, role="proposer")
runner = (sidecall.runner_for(ctx.host.name, ctx.config, role="proposer",
session_model=model)
if not sidecall.paused(ctx.host.name, ctx.root)
else sidecall.local_runner("proposer", ctx.config))
stage = PrefixStage(icfg, ctx.repo.label, ctx.mech_dir, False, runner=runner,
Expand Down Expand Up @@ -540,9 +542,14 @@ def fold_locked(ctx, session_id, transcript_path=None, imported=False, span=None
catalog = {} if imported else load_catalog(ctx.root, ctx.host.name)
adapter = LiveAdapter(ctx.host, ctx.repo.root, vocabulary=ctx.config.vocabulary,
catalog=catalog, protect=ctx.config.toolprune_protect)
# The model this session ran on, as its own hooks recorded it: the proposer follows
# it instead of the CLI's configured default. The session log keeps a bare model id
# (no provider), which `opencode run --model` cannot resolve, so it is not a source.
model = str(sess.get("model") or "")
with ctx.store.locked("mechanisms"):
inj = _fold_microskills(ctx, adapter, run_dir, session_id,
injected=sess.get("injected_ids"), imported=imported)
injected=sess.get("injected_ids"), imported=imported,
model=model)
# A session that ran no prompt is no evidence about the tools.
unt = None if prompts == 0 else _fold_toolprune(
ctx, adapter, run_dir, session_id, transcript=transcript,
Expand Down
12 changes: 12 additions & 0 deletions tokeneconomizer/plugin/hosts.py
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,13 @@ def session_state(self, env):
"""What a session's state keeps from the environment of its first hook."""
return {}

def session_model(self, p):
"""The model the session runs on, in the agent CLI's own `--model` form, or "" when the
payload does not say. A host that reports it lets the plugin's own calls follow the
session instead of whatever model the CLI would pick by default -- which is how a
paid provider the person selected can be looked past."""
return ""

def tool_usage(self, trace_path):
return trace.tool_usage(trace_path)

Expand Down Expand Up @@ -486,6 +493,11 @@ class OpenCode(Host):
def turn_key(self, p):
return str(p.get("prompt_id") or "")

def session_model(self, p):
"""`session.step.started` names the model on `provider/model#variant`; the plugin sends
the session's own, so the plugin's calls do not fall back to the CLI default."""
return str(p.get("model") or "")

def headless(self, env, transcript=None):
return str(env.get("TE_OPENCODE_MODE", "")).strip() == "run"

Expand Down
7 changes: 6 additions & 1 deletion tokeneconomizer/plugin/judge.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
for the session: a verdict that lands after its turn -- or its session -- has ended was still paid
for, and must not drop out of the account.
"""
import os
import time

from . import sidecall
Expand Down Expand Up @@ -68,10 +69,14 @@ def run_job(ctx, session_id, job_id):
if req.get("id") != job_id: # superseded by a newer request
return None
paused = sidecall.paused(ctx.host.name, ctx.root)
# The model the session runs on, so the judge follows it rather than the CLI's default.
sess = ctx.store.read(os.path.join("sessions", f"{session_id}.json")) or {}
session_model = str(sess.get("model") or "")
# Each call asks for its own runner: `local_roles` may put the judge on-device and leave
# the redirect on the agent's CLI, or the other way round. A paused host's CLI is skipped,
# but a local runner (which needs no login) still answers.
runner = sidecall.runner_for(ctx.host.name, ctx.config, role="judge") if not paused \
runner = sidecall.runner_for(ctx.host.name, ctx.config, role="judge",
session_model=session_model) if not paused \
else sidecall.local_runner("judge", ctx.config)
writer = sidecall.runner_for(ctx.host.name, ctx.config, role="nudge") if not paused \
else sidecall.local_runner("nudge", ctx.config)
Expand Down
Loading