diff --git a/CLAUDE.md b/CLAUDE.md index 6f39b19a..6e9a9ae2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,7 +1,7 @@ # TokenEconomizer A plugin for Claude Code, Codex, the Copilot CLI and Pi (experimentally opencode, Cursor, Grok Build, -Hermes Agent, the Antigravity CLI and Factory Droid) that reads finished sessions and changes what +Hermes Agent, the Antigravity CLI, Factory Droid and ZCode) that reads finished sessions and changes what later sessions in the same repository pay for: micro-skills, tool pruning, and a loop guard. See `README.md` for the overview and the method, `docs/PLUGIN.md` for the design and the verified agent behavior, and `docs/ARCHITECTURE.md` for how the package fits together. @@ -12,7 +12,7 @@ agent behavior, and `docs/ARCHITECTURE.md` for how the package fits together. - `toolprune/` — unused-tool evidence and the deny-list proposal; Codex capability packages - `loopguard/` — the per-call failure-loop guard and its per-repository envelope - `trace/` — each agent's session-log format (Claude Code, Codex, Copilot CLI, Pi, opencode, - Cursor, Grok Build, Hermes Agent, Antigravity CLI, Factory Droid) + Cursor, Grok Build, Hermes Agent, Antigravity CLI, Factory Droid, ZCode) - `plugin/` — the runtime: hook dispatch, per-repository state, settings, the `te` CLI - `config.py`, `ledger.py`, `pricing.py`, `domain.py`, `stage.py` — shared by all three - `plugin/` — the installable plugin directory: manifests per agent, hook files, launchers @@ -21,7 +21,8 @@ agent behavior, and `docs/ARCHITECTURE.md` for how the package fits together. skills (`codex-skills/`, Codex having no plugin slash commands), the Pi extension, the opencode plugin (`opencode/`, one file whose default export serves 1.x via `server` and 2.x via `setup`), the Hermes plugin, and the Droid manifest (`plugin/.factory-plugin/`, with - `plugin/hooks/droid-hooks.json`); `lib/` links the package + `plugin/hooks/droid-hooks.json`), and the ZCode hook entries (`plugin/hooks/zcode-hooks.json`, added + to a repository's `zcode.json`); `lib/` links the package - `experiments/` — benchmark-only: `te_bench/` (adapters, arms, facade, benchmark-only controllers), `te_harness/` (Docker runners), `scripts/`, `workloads/`, its own `tests/`; its `README.md` is the run book diff --git a/README.md b/README.md index 967ee41f..c0dd4b2f 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,7 @@ mechanisms: TokenEconomizer is a plugin for **Claude Code**, **Codex**, the **GitHub Copilot CLI** and **Pi**, with experimental support for **opencode**, **Cursor Agent**, **Grok Build**, **Hermes Agent**, -the **Antigravity CLI** and **Factory Droid**. Each italic +the **Antigravity CLI**, **Factory Droid** and **ZCode**. Each italic letter above is a setting, shown with its default ([Settings](#settings)), and each mechanism can be turned on or off. The plugin announces each action in one line, and if any part of it fails, your session carries on as if it were not installed. Its model calls (the fact proposer, and the judge with the @@ -185,6 +185,14 @@ micro-skills' paid proposer and the loop guard's judge stay off until `droid_sid (when on, they may use only Droid's read-only `Read` tool). See **[`docs/DROID.md`](docs/DROID.md)** for what works and how to import earlier sessions. +**ZCode** (experimental; micro-skills, the loop guard and session cost, not tool pruning) + +ZCode has no plugin format this repository builds. Add the entries of `plugin/hooks/zcode-hooks.json` +to the `hooks.events` of the repository's `zcode.json`, then trust them (`zcode hooks trust +review`). The fold reads ZCode's SQLite store, not the hook's own transcript. Its `-p` side call +stays off until `zcode_sidecall` is on. See **[`docs/ZCODE.md`](docs/ZCODE.md)**. Kimi Code is +not supported yet; **[`docs/KIMI.md`](docs/KIMI.md)** says what is missing. + **Cursor Agent** (experimental) ```sh @@ -380,6 +388,7 @@ are these settings: | *r* | `loopguard_nudges` | 1 | redirects per turn before the guard only tells you | | *b* | `loopguard.backstop_mult` | 2.5 | how many times its usual length a churning turn may run before it is redirected without the judge | | | `droid_sidecall` | off | on Droid, allow the plugin's own model calls, which run `droid exec` | +| | `zcode_sidecall` | off | on ZCode, allow the plugin's own model calls, which run `zcode -p` | Change them with `config set`, shown here in Claude Code's form (other agents type it as in [Everyday commands](#everyday-commands)): @@ -501,7 +510,7 @@ agent's own limits; afterwards, `te status` counts the redirects and "still stuc | [`tokeneconomizer/plugin`](tokeneconomizer/plugin/) | The runtime the hooks and the `te` command run | | [`plugin`](plugin/) | The installable plugin: manifests, hook files, launchers, slash commands, the Pi extension, the opencode and Hermes plugins, the Droid bundle (`.factory-plugin/`) | | [`experiments`](experiments/) | The harness, launchers and task sets behind the paper's results | -| [`docs`](docs/) | `PLUGIN.md` (design, verified agent behavior), `ARCHITECTURE.md` (how the package fits together) and `DROID.md` (the Factory Droid adapter) | +| [`docs`](docs/) | `PLUGIN.md` (design, verified agent behavior), `ARCHITECTURE.md` (how the package fits together), `ZCODE.md` and `KIMI.md` (the ZCode adapter; why Kimi Code has none) and `DROID.md` (the Factory Droid adapter) | | [`tests`](tests/) | Offline tests for the package and the plugin | The package is standard-library only. Run the tests with `python -m pytest tests` (add diff --git a/docs/KIMI.md b/docs/KIMI.md new file mode 100644 index 00000000..e087e2e8 --- /dev/null +++ b/docs/KIMI.md @@ -0,0 +1,39 @@ +# TokenEconomizer on Kimi Code + +Kimi Code 2.x is not supported. It is observe-only: there is no adapter, no reader and no hook +entry, and `tests/test_kimi.py` pins that. This note says why and what is needed. + +## What is known (Kimi Code 2.1.1) + +- The binary is `~/.kimi-code/bin/kimi`. The Python `kimi_cli` in `~/.kimi/` is the older 1.x. +- A session is `~/.kimi-code/sessions/wd__/session_/agents/main/wire.jsonl`, one + JSON record per line with a `type` key. The types seen are `metadata`, `config.update` and + `tools.set_active_tools`. No record of a prompt, a tool call or a tool result has been seen. +- Hooks are `[[hooks]]` in `~/.kimi-code/config.toml` only. Events include `SessionStart`, + `UserPromptSubmit`, `PreToolUse`, `PostToolUse`, `Stop` and `SessionEnd`. Exit 2 blocks on + `UserPromptSubmit`, `PreToolUse` and `Stop`; other failures let the call go on. There is no + `updatedInput`. +- Context: the docs say stdout of an exit-0 hook "may be appended to context". There is no + `additionalContext` field. This has not been tried. +- Side call: `kimi -p ` runs under the `auto` permission policy and has no flag that + removes tools, so a side call could act on text a repository controls. + +## What is missing, and why no reader was written + +A reader written from the three record types above would have to guess where a prompt and a tool +result are, and would pass its own tests while counting the wrong things. Two pieces of evidence +come first. Each has a skipped test in `tests/test_kimi.py` that becomes a real one. + +1. **A multi-turn `wire.jsonl`.** In one Kimi Code session, send two prompts; make one of them + run a shell command and read a file. Then list the record `type` and the key names of every + line of that session's `agents/main/wire.jsonl` (types and key names only, no text) and save + them as `tests/fixtures/kimi_wire_types.json`. A one-line `jq` over the file is enough: + `jq -c '[.type, (keys)]' wire.jsonl`. +2. **An injection test.** In a scratch directory, add a `UserPromptSubmit` `[[hooks]]` entry to + `~/.kimi-code/config.toml` whose command prints a made-up sentence and exits 0. Ask Kimi Code + to repeat the sentence the hook printed. Record whether it can. + +If the sample has the three records and the injection works, the adapter is the ZCode one with +a `wire.jsonl` reader (`trace/kimi.py`) instead of the SQLite one, and plain-stdout context. If +the injection fails, Kimi stays observe-only with micro-skills' block and the loop guard's judge +off: nothing can be injected, and its side call cannot be restricted. diff --git a/docs/PLUGIN.md b/docs/PLUGIN.md index 9faf3834..373ee45e 100644 --- a/docs/PLUGIN.md +++ b/docs/PLUGIN.md @@ -720,6 +720,38 @@ was not.) call also writes a session log of its own, so side calls stay off until `droid_sidecall` is on (or `TE_DROID_SIDECALL=1`). +**ZCode 0.17.1 -- adapter built 2026-11-12, read from its code, not run live:** + +* **Offline, tested here:** the session reader over a synthetic copy of ZCode's SQLite store (its + JSON keys were read off a real store, keys only); prompt counting; usage; host detection; the + `zcode -p` side call's command; and the hooks end to end through `hook.main`, with a payload + built as ZCode's own code builds it. **Not done:** a live ZCode session with the hook file + (`docs/ZCODE.md`, "Verify it"), and a real `zcode -p` call (what it prints, whether + `--disallowed-tools` keeps out every tool). +* The hook contract is Claude-shaped: `session_id`, `tool_name`, `tool_input`, `tool_use_id`, + `tool_response`, `hook_event_name`, beside ZCode's own camelCase fields (`hookEventName`, + `sessionId`, `toolName`, `turnId`, `agentName`, `mode`). Events: `SessionStart`, + `UserPromptSubmit`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `Stop`, and no + `SessionEnd`. Its output schema takes `hookSpecificOutput.additionalContext` (the event's own + name in `hookEventName`), a top-level `additionalContext` and `systemMessage`; the contexts of + SessionStart, UserPromptSubmit, PostToolUse, PostToolUseFailure and Stop are added to the + conversation as a `hook_context` entry, cut at 24,000 characters. `systemMessage` is used only to + explain a block, and `continue: false` is a block request on every event but Stop. +* `transcript_path` and `transcriptPath` are one temporary file, `zcode-hook-*/transcript.jsonl`, + with the event's prompt or answer, deleted after the hook. That name is how `hosts.detect` tells + ZCode from Claude Code and from Grok (which also sends `hookEventName`). The log is the SQLite + store, `/cli/db/db.sqlite`. +* ZCode's tool names are Claude Code's. `model_usage.input_tokens` counts the cached reads (total + = input + output on every row of a real store, counted by aggregate queries only; + `cache_creation_input_tokens` was 0 on all of them); the ledger's input does not, so the reader subtracts them. +* **Side call.** `zcode -p --mode plan --cwd --disallowed-tools `; `-p` + has no `--model`, and defaults to `yolo`. A deny list cannot be inverted, so the call is off + until `zcode_sidecall` is on (or `TE_ZCODE_SIDECALL=1`). + +**Kimi Code 2.1.1 -- not supported** (`docs/KIMI.md`): the `wire.jsonl` record that carries a +prompt or a tool call has not been seen, and whether an exit-0 hook's stdout reaches the model has +not been tried. + ## 4. State and privacy All state is local, under `$TOKENECONOMIZER_HOME` (default `~/.tokeneconomizer`), one directory diff --git a/docs/ZCODE.md b/docs/ZCODE.md new file mode 100644 index 00000000..f97b48d6 --- /dev/null +++ b/docs/ZCODE.md @@ -0,0 +1,78 @@ +# TokenEconomizer on ZCode + +ZCode (Z.ai) support is experimental. It was written against ZCode `0.17.1` on macOS from its +hook code, its command-line help and the schema of its session store. It has not yet been run +under a live ZCode session with this plugin's hook file: the "Verify it" steps below do that. + +## What works, and what does not + +| Mechanism | ZCode | Why | +|---|---|---| +| Micro-skills (repository facts at session start) | yes, unverified live | `SessionStart` and `UserPromptSubmit` take `additionalContext`; ZCode adds it to the conversation as a `hook_context` entry | +| Session cost / effective tokens | yes | `model_usage` in the store; the cached part is taken out of `input_tokens`, which counts it | +| Loop guard (redirect a failing loop) | partially | `PostToolUse` takes `additionalContext`; the judge needs `zcode -p`, which is off by default | +| History import (earlier sessions) | no | the store holds them, but no importer reads it yet | +| Tool pruning | no | ZCode has no per-tool setting a hook can write | + +## How a session is read + +ZCode keeps every session in one SQLite file, `/cli/db/db.sqlite` +(tables `session`, `message`, `part`, `tool_usage`, `model_usage`). The hook's own +`transcript_path` is no log: ZCode writes the one prompt or answer of the event to a temporary +file and deletes it when the hook returns. The fold therefore opens the store read-only, +copies one session's tool calls, exit codes and token usage into a file of the plugin's +(`/zcode-sessions/.json`), and reads that. The copy has no words: a prompt is +counted and not kept, and what the model said or reasoned is not read. + +ZCode has no `SessionEnd`. A session is folded by the next session start in its repository once +it has been quiet, as on the Antigravity CLI. + +## How ZCode is told from Claude Code + +ZCode's payload has `session_id`, `tool_name`, `tool_input` and the other Claude names, so +nothing in them tells it apart. Grok Build also sends the camelCase `hookEventName`. What only +ZCode sends is the transcript: both `transcript_path` and `transcriptPath` hold +`/zcode-hook-/transcript.jsonl`, and `detect` takes a payload for ZCode only when +both are that, and equal. The environment is not a mark: `ZCODE_SESSION_ID` is set for ZCode's +hooks and shell, and an agent started from that shell inherits it, so it only chooses the host of +a `te` command run in a shell (`default_host`), after every other agent's variable. The hook file +passes `--host zcode`, so detection is the fallback. + +## Install + +ZCode reads workspace hooks from `zcode.json` at the root of a repository (its schema is +`{"hooks": {"enabled": true, "events": {...}}}`) and runs a workspace hook only after the person +trusts it (`zcode hooks trust review`). `plugin/hooks/zcode-hooks.json` holds the five entries; +add them to the `events` of the repository's `zcode.json`, after any entry that guards a tool, +and point the command at the clone if it is not in `~/.tokeneconomizer/src`. + +## Verify it + +1. Start `zcode` in a repository where the entries are trusted. Run + `~/.tokeneconomizer/src/plugin/scripts/te --host zcode status`: a session is listed after its + first hook. +2. After a session that ran a shell command, run `te --host zcode fold` (or start another session + in the same repository after a quiet period). `te --host zcode status` shows the folded + session with its prompts and tokens. +3. To see that a learned fact reaches the model, ask ZCode to repeat what its session start added. + +## Side calls + +`te` asks the model for facts and for a verdict on a stuck turn through `zcode -p`, in a throwaway +directory, in `plan` mode (its `-p` default is `yolo`), with the acting tools removed by +`--disallowed-tools`. That option is a deny list, and a tool an MCP server adds is not on it; +the call also writes a session into the store and reports no usage. So it is off until +`te --host zcode config set zcode_sidecall=on` (or `TE_ZCODE_SIDECALL=1`). `zcode -p` has no +`--model` flag, so there is no `zcode_paid_model`: the call uses the model in ZCode's settings. +What `zcode -p` prints has not been checked against a real model: the plugin takes its stdout. + +## Known limits + +- Not run end to end under ZCode; the reader is tested on a synthetic store whose JSON keys were + read off a real one. +- ZCode's `hook_context` entries after a resumed session were not traced, so `keeps_start_context` + keeps its default (the context is not added again on resume). +- A subagent's own hook calls carry its session id; its rows are counted in the parent's usage, + and a session forked from another is a session of its own. +- `model_usage.cache_creation_input_tokens` was 0 in every row seen, so cache writes are not + priced. diff --git a/plugin/hooks/zcode-hooks.json b/plugin/hooks/zcode-hooks.json new file mode 100644 index 00000000..ca47d7c6 --- /dev/null +++ b/plugin/hooks/zcode-hooks.json @@ -0,0 +1,62 @@ +{ + "hooks": { + "enabled": true, + "events": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "\"$HOME/.tokeneconomizer/src/plugin/scripts/te-hook\" --host zcode", + "timeout": 15 + } + ] + } + ], + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": "\"$HOME/.tokeneconomizer/src/plugin/scripts/te-hook\" --host zcode", + "timeout": 20 + } + ] + } + ], + "PostToolUse": [ + { + "hooks": [ + { + "type": "command", + "command": "\"$HOME/.tokeneconomizer/src/plugin/scripts/te-hook\" --host zcode", + "timeout": 60 + } + ] + } + ], + "PostToolUseFailure": [ + { + "hooks": [ + { + "type": "command", + "command": "\"$HOME/.tokeneconomizer/src/plugin/scripts/te-hook\" --host zcode", + "timeout": 60 + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"$HOME/.tokeneconomizer/src/plugin/scripts/te-hook\" --host zcode", + "timeout": 10 + } + ] + } + ] + } + } +} diff --git a/tests/conftest.py b/tests/conftest.py index 7f7978d0..ddeb40a4 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -58,5 +58,5 @@ def _no_model_calls(request, monkeypatch): if request.node.get_closest_marker("real_sidecalls"): return for name in ("claude_call", "codex_call", "copilot_call", "droid_call", "grok_call", - "hermes_call", "pi_call"): + "hermes_call", "pi_call", "zcode_call"): monkeypatch.setattr(sidecall, name, lambda *a, **k: ("", None)) diff --git a/tests/test_kimi.py b/tests/test_kimi.py new file mode 100644 index 00000000..7d762bfc --- /dev/null +++ b/tests/test_kimi.py @@ -0,0 +1,70 @@ +"""Kimi Code 2.x: observe-only, and what must be learned before a reader is written. + +Kimi Code keeps a session at `~/.kimi-code/sessions/wd__/session_/agents/main/ +wire.jsonl`, one JSON record per line with a `type` key. The only record types seen so far are +`metadata`, `config.update` and `tools.set_active_tools`; the record that carries a prompt, a tool +call or a tool result has not been seen, so no reader exists: a reader written from a guess would +count the wrong things and still pass its own tests. Its hooks are `[[hooks]]` entries in +`~/.kimi-code/config.toml`; the docs say only that stdout of an exit-0 hook "may be appended to +context", with no `additionalContext` field, and that has not been tried. + +The two skipped tests below are the work that is missing. Each names the evidence it needs and +the check it will run once there is one (docs/KIMI.md has the steps for the person who has Kimi +Code installed). The other two tests pin what holds until then. + +Proves TC-980-2 (TASK-980): an unknown payload is not taken for Kimi (nor for ZCode). +""" +import pytest + +from tokeneconomizer import trace +from tokeneconomizer.plugin import hosts + +WIRE = "/home/me/.kimi-code/sessions/wd_repo_1a2b/session_0f0f/agents/main/wire.jsonl" + + +def test_kimi_is_not_a_host_yet(): + assert "kimi" not in hosts.HOSTS + assert not any(name.startswith("kimi") for name in dir(trace)) + + +@pytest.mark.parametrize("payload", [ + {"session_id": "s", "hook_event_name": "UserPromptSubmit", "transcript_path": WIRE}, + {"transcript_path": WIRE.replace("/", "\\")}, + {}, +]) +def test_a_payload_that_names_a_kimi_session_log_is_claimed_by_no_other_agent(payload): + """Until a Kimi adapter exists a Kimi payload is Claude-shaped at best, and must not be + claimed by an adapter for another agent: not ZCode, not Grok, not Codex.""" + assert type(hosts.detect(payload, {}, hint="kimi")).__name__ == "ClaudeCode" + assert type(hosts.detect(payload, {})).__name__ == "ClaudeCode" + + +def test_a_camel_case_payload_with_a_kimi_log_is_not_zcode(): + """The Grok adapter still takes any camelCase `hookEventName`, as it did before ZCode; what + this pins is that ZCode's marker is the throwaway transcript, not the field names.""" + payload = {"sessionId": "s", "hookEventName": "PostToolUse", "transcriptPath": WIRE, + "transcript_path": WIRE} + assert not isinstance(hosts.detect(payload, {}), hosts.ZCode) + + +@pytest.mark.skip(reason="NEEDS EVIDENCE: a multi-turn wire.jsonl. Run two prompts in one Kimi " + "Code session, one of which makes the agent run a shell command and " + "read a file, then save the record `type` of every line of " + "~/.kimi-code/sessions/.../agents/main/wire.jsonl (types and key " + "names only, no text) as tests/fixtures/kimi_wire_types.json.") +def test_the_wire_log_names_a_prompt_a_tool_call_and_a_tool_result(): + """Once the sample exists: assert that the fixture holds a record type for each of the three, + then write `trace/kimi.py` (prompt count, tool calls with exit codes, usage) and replace this + test with the reader's tests, as `test_zcode.py` has.""" + raise AssertionError("no fixture") + + +@pytest.mark.skip(reason="NEEDS EVIDENCE: an injection test. In a scratch directory put a " + "[[hooks]] entry for UserPromptSubmit in ~/.kimi-code/config.toml whose " + "command prints a made-up sentence and exits 0, ask Kimi Code 'repeat " + "the sentence the hook printed', and record whether it can.") +def test_stdout_of_an_exit_zero_hook_reaches_the_model(): + """If it does, the Kimi adapter returns context as plain stdout from SessionStart and + UserPromptSubmit and sets `prompt_context`; if it does not, Kimi stays observe-only with the + loop guard's judge and the micro-skills block off (nothing could be injected).""" + raise AssertionError("no result") diff --git a/tests/test_live_core.py b/tests/test_live_core.py index 7e52ca7f..f04db802 100644 --- a/tests/test_live_core.py +++ b/tests/test_live_core.py @@ -1001,8 +1001,10 @@ def test_claude_side_calls_stay_out_of_the_persons_history_where_the_cli_allows( def test_every_agent_with_side_calls_has_its_own_paid_model_setting(): from tokeneconomizer.plugin import config as live_config, hosts, sidecall cfg = live_config.LiveConfig() - for name in set(hosts.HOSTS) - sidecall.NO_SIDE_CALLS: + for name in set(hosts.HOSTS) - sidecall.NO_SIDE_CALLS - sidecall.NO_MODEL_FLAG: assert cfg.set(f"{name}_paid_model", "m"), name + # ZCode's `-p` takes no model, so it has no such setting (and is not told it has). + assert not cfg.set("zcode_paid_model", "m") @pytest.mark.real_sidecalls diff --git a/tests/test_zcode.py b/tests/test_zcode.py new file mode 100644 index 00000000..7d75e91b --- /dev/null +++ b/tests/test_zcode.py @@ -0,0 +1,528 @@ +"""ZCode (Z.ai): the session reader over its SQLite store, its host adapter and host detection, +side calls, and the hook entry points end to end. + +ZCode keeps sessions in `/cli/db/db.sqlite` (tables `session`, +`message`, `part`, `tool_usage`, `model_usage`). The database here is synthetic: the schema is the +columns the reader names, and the JSON of `message.data` and `part.data` has the keys read off a +real ZCode 0.17.1 store (keys only; no conversation was copied). The words in it are markers that +must never reach an export. + +Proves TC-980-1 (TASK-980): the suite passes on a ZCode session fixture. +Proves TC-980-2 (TASK-980): an unknown payload is not taken for ZCode (see +`test_an_unknown_payload_is_not_taken_for_zcode`); Kimi's half is in `test_kimi.py`. +""" +import io +import json +import os +import sqlite3 +import subprocess +import time + +import pytest + +from tokeneconomizer import trace +from tokeneconomizer.plugin import accounting, denylist, fold, hook, hosts +from tokeneconomizer.plugin.context import LiveContext + +SCHEMA = """ +create table session (id text primary key, project_id text not null default 'p', + parent_id text, slug text not null default 's', directory text not null, + title text not null default 'MARKER-TITLE', version text not null default '0.17.1', + time_created integer not null default 0, time_updated integer not null default 0); +create table message (id text primary key, session_id text not null, + time_created integer not null, time_updated integer not null default 0, + data text not null, sequence integer); +create table part (id text primary key, message_id text not null, session_id text not null, + time_created integer not null, time_updated integer not null default 0, + data text not null, sequence integer); +create table tool_usage (id text primary key, session_id text not null, turn_id text, + tool_call_id text not null, tool_name text not null, status text not null, + started_at integer not null default 0, exit_code integer); +create table model_usage (id text primary key, logical_request_id text not null default '', + session_id text not null, turn_id text, query_source text not null, + provider_id text not null default 'zai', model_id text not null, + status text not null default 'completed', started_at integer not null default 0, + input_tokens integer not null default 0, output_tokens integer not null default 0, + reasoning_tokens integer not null default 0, + cache_creation_input_tokens integer not null default 0, + cache_read_input_tokens integer not null default 0); +""" + +FAIL = "python: can't open file 'tests/runtests.py': [Errno 2] No such file or directory" +OK = "Ran 3 tests in 0.01s\nOK" +# (tool, input, output, status, exit_code) +TEST_CALLS = [("Bash", {"command": "python tests/runtests.py"}, FAIL, "error", 2), + ("Bash", {"command": "python runtests.py"}, OK, "completed", 0)] + + +class Db: + """A synthetic ZCode store being written; every row gets a rising time and id.""" + + def __init__(self, path): + self.path = str(path) + self.con = sqlite3.connect(self.path) + self.con.executescript(SCHEMA) + self.n = 0 + + def _id(self, prefix): + self.n += 1 + return f"{prefix}_{self.n:05d}" + + def session(self, sid, cwd, parent=None): + self.con.execute("insert into session (id, directory, parent_id) values (?, ?, ?)", + (sid, str(cwd), parent)) + + def message(self, sid, data, parts): + mid = self._id("msg") + self.con.execute("insert into message (id, session_id, time_created, data) " + "values (?, ?, ?, ?)", (mid, sid, self.n, json.dumps(data))) + for p in parts: + self.con.execute("insert into part (id, message_id, session_id, time_created, data) " + "values (?, ?, ?, ?, ?)", + (self._id("prt"), mid, sid, self.n, json.dumps(p))) + + def user(self, sid, text, **extra): + self.message(sid, dict({"role": "user", "agent": "main", "time": {"created": self.n}, + "model": {"providerID": "zai", "modelID": "glm-5.1"}}, **extra), + [{"type": "text", "text": text}]) + + def assistant(self, sid, cwd, calls=(), text="MARKER-ANSWER"): + parts = [{"type": "step-start"}, {"type": "reasoning", "text": "MARKER-REASONING"}, + {"type": "text", "text": text}] + for i, (tool, inp, out, status, code) in enumerate(calls): + call_id = f"call_{self.n}_{i}" + state = {"status": status, "input": inp, "metadata": {}, "time": {}} + state["error" if status == "error" else "output"] = out + parts.append({"type": "tool", "callID": call_id, "tool": tool, "state": state}) + self.con.execute("insert into tool_usage (id, session_id, tool_call_id, tool_name, " + "status, exit_code) values (?, ?, ?, ?, ?, ?)", + (self._id("tu"), sid, call_id, tool, status, code)) + parts.append({"type": "step-finish", "reason": "stop"}) + self.message(sid, {"role": "assistant", "modelID": "glm-5.1", "providerID": "zai", + "path": {"cwd": str(cwd), "root": str(cwd)}}, parts) + + def usage(self, sid, source="main_turn", model="glm-5.1", i=1000, o=100, cache_read=0): + self.con.execute( + "insert into model_usage (id, session_id, query_source, model_id, input_tokens, " + "output_tokens, cache_read_input_tokens) values (?, ?, ?, ?, ?, ?, ?)", + (self._id("mu"), sid, source, model, i, o, cache_read)) + + def close(self): + self.con.commit() + self.con.close() + return self.path + + +def make_db(path, cwd, sid="sess_1", calls=TEST_CALLS): + """One session: a real prompt, two tool calls, machine messages around them, a subagent.""" + db = Db(path) + db.session(sid, cwd) + db.user(sid, "MARKER-PROMPT run the tests") + db.assistant(sid, cwd, calls) + # Messages ZCode writes itself: not prompts. + db.user(sid, "MARKER-REMINDER", source="todo_reminder") + db.user(sid, "MARKER-SYNTH", synthetic=True) + db.session("sess_sub", cwd, parent=sid) + db.user("sess_sub", "MARKER-SUBPROMPT") + db.assistant("sess_sub", cwd, [("Bash", {"command": "ls"}, "a", "completed", 0)]) + db.usage(sid, i=5000, o=400, cache_read=4000) + db.usage(sid, source="session_title", i=300, o=20) + db.usage("sess_sub", source="subagent", i=700, o=50, cache_read=500) + return db.close() + + +def export(tmp_path, **kw): + db = make_db(tmp_path / "db.sqlite", tmp_path, **kw) + out = str(tmp_path / "export.json") + assert trace.zcode_export(db, kw.get("sid", "sess_1"), out) is True + return db, out + + +# ---- the reader ----------------------------------------------------------------------------- +def test_export_keeps_the_calls_and_their_exit_codes_and_none_of_the_words(tmp_path): + _db, out = export(tmp_path) + text = open(out, encoding="utf-8").read() + for word in ("MARKER-PROMPT", "MARKER-ANSWER", "MARKER-REASONING", "MARKER-TITLE", + "MARKER-REMINDER", "MARKER-SYNTH", "MARKER-SUBPROMPT"): + assert word not in text + assert trace.zcode_prompt_count(out) == 1 + assert trace.zcode_session_info(out)["directory"] == str(tmp_path) + + +def test_a_session_reads_as_claude_lines_with_exit_codes_deciding_failure(tmp_path): + _db, out = export(tmp_path) + claude = str(tmp_path / "claude.jsonl") + assert trace.zcode_session_to_claude(out, claude) == 2 # the subagent's call is not here + calls = trace.load_trajectory(claude) + assert [c[0] for c in calls] == ["Bash", "Bash"] + assert calls[0][1]["command"] == "python tests/runtests.py" + assert FAIL in calls[0][2] and calls[0][3] is False # (name, args, result, ok) + assert calls[1][3] is True + assert trace.call_cwds(claude) == [str(tmp_path), str(tmp_path)] + + +def test_a_non_zero_exit_is_a_failure_though_the_tool_finished(tmp_path): + calls = [("Bash", {"command": "pytest"}, "1 failed", "completed", 1)] + _db, out = export(tmp_path, calls=calls) + claude = str(tmp_path / "claude.jsonl") + trace.zcode_session_to_claude(out, claude) + assert trace.load_trajectory(claude)[0][3] is False + + +def test_prompts_are_the_typed_ones_only(tmp_path): + """ZCode writes reminders, goal continuations and background-task notices as user messages; + each is marked by `source` or `synthetic`, and none is a prompt.""" + _db, out = export(tmp_path) + assert trace.zcode_prompt_count(out) == 1 + db = Db(tmp_path / "many.sqlite") + db.session("s", tmp_path) + for src in ("todo_reminder", "background_task", "goal-continuation", "goal_state_change", + "agent_listing_delta"): + db.user("s", "x", source=src) + db.user("s", "one") + db.user("s", "two") + db.user("s", " ") # no words, no prompt + path = db.close() + assert trace.zcode_export(path, "s", str(tmp_path / "e.json")) + assert trace.zcode_prompt_count(str(tmp_path / "e.json")) == 2 + + +def test_usage_excludes_the_cache_from_input_and_adds_the_subagent(tmp_path): + """ZCode's `input_tokens` counts the cached reads (its total is input + output), the ledger's + does not.""" + _db, out = export(tmp_path) + u = trace.zcode_session_usage(out) + assert u["input_tokens"] == (5000 - 4000) + 300 + (700 - 500) + assert u["cache_read_input_tokens"] == 4000 + 500 + assert u["output_tokens"] == 400 + 20 + 50 + assert u["requests"] == 3 and u["model"] == "glm-5.1" + eff, priced_by = accounting.effective(u) + assert eff and priced_by in ("model", "fallback") + + +def test_the_subagents_own_session_is_not_folded_into_the_parents_calls(tmp_path): + _db, out = export(tmp_path) + assert "sess_sub" not in open(out, encoding="utf-8").read() + + +def test_the_store_is_opened_read_only(tmp_path): + db = make_db(tmp_path / "db.sqlite", tmp_path) + before = (os.path.getsize(db), os.stat(db).st_mtime_ns) + assert trace.zcode_export(db, "sess_1", str(tmp_path / "e.json")) + assert (os.path.getsize(db), os.stat(db).st_mtime_ns) == before + assert not os.path.exists(db + "-wal") or os.path.getsize(db + "-wal") == 0 + + +def test_the_store_location_follows_zcode_storage_dir(tmp_path): + assert trace.zcode_db({"ZCODE_STORAGE_DIR": str(tmp_path / "z")}) == str( + tmp_path / "z" / "cli" / "db" / "db.sqlite") + assert trace.zcode_db({}) == os.path.join(os.path.expanduser("~"), ".zcode", "cli", "db", + "db.sqlite") + + +def test_what_cannot_be_read_is_no_session_and_creates_nothing(tmp_path): + out = str(tmp_path / "e.json") + assert trace.zcode_export(str(tmp_path / "missing.sqlite"), "sess_1", out) is False + assert not os.path.exists(str(tmp_path / "missing.sqlite")) and not os.path.exists(out) + db = make_db(tmp_path / "db.sqlite", tmp_path) + assert trace.zcode_export(db, "sess_nope", out) is False + junk = tmp_path / "junk.sqlite" + junk.write_bytes(b"this is not a database") + assert trace.zcode_export(str(junk), "sess_1", out) is False + # A store of another shape (a later ZCode that renamed a table) is no session either. + other = tmp_path / "other.sqlite" + con = sqlite3.connect(other) + con.execute("create table session (id text)") + con.execute("insert into session values ('sess_1')") + con.commit() + con.close() + assert trace.zcode_export(str(other), "sess_1", out) is False + assert not os.path.exists(out) + assert trace.zcode_session_to_claude(str(tmp_path / "nope.json"), str(tmp_path / "c")) is None + assert trace.zcode_prompt_count(str(tmp_path / "nope.json")) == 0 + assert trace.zcode_session_usage(str(tmp_path / "nope.json"))["input_tokens"] == 0 + + +@pytest.mark.parametrize("sid", ["", "../x", "a b", "sess_1; drop table session", None]) +def test_a_session_id_that_is_no_id_is_refused(tmp_path, sid): + db = make_db(tmp_path / "db.sqlite", tmp_path) + assert trace.zcode_export(db, sid, str(tmp_path / "e.json")) is False + + +def test_an_apply_patch_call_is_an_edit_of_the_files_it_names(): + patch = "*** Begin Patch\n*** Update File: src/a.py\n@@\n-x\n+y\n*** End Patch" + assert trace.zcode_tool("ApplyPatch", {"input": patch}) == ( + "apply_patch", {"path": "src/a.py", "paths": ["src/a.py"]}) + assert trace.zcode_tool("Bash", {"command": "ls"}) == ("Bash", {"command": "ls"}) + assert trace.zcode_tool("Bash", None) == ("Bash", {}) + + +# ---- detection: what is ZCode, and what is not ------------------------------------------------ +def zcode_payload(event="PostToolUse", sid="sess_1", tmp="/var/folders/ab/T"): + """The payload ZCode 0.17.1 builds (`createCompatibleHookStdin` in its bundle): its own + camelCase fields, the snake_case aliases of them, and a throwaway transcript.""" + tp = f"{tmp}/zcode-hook-Ab12Cd/transcript.jsonl" + p = {"hookEventName": event, "sessionId": sid, "cwd": "/repo", "mode": "yolo", + "agentName": "main", "traceId": "t", "turnId": "u", + "agent_type": "main", "hook_event_name": event, "permission_mode": "yolo", + "session_id": sid, "transcript_path": tp, "transcriptPath": tp} + if event in ("PostToolUse", "PostToolUseFailure"): + p.update(toolName="Bash", toolInput={"command": "ls"}, toolCallId="c1", + tool_name="Bash", tool_input={"command": "ls"}, tool_use_id="c1") + if event == "PostToolUse": + p.update(toolResponse={"output": "a\n", "exit_code": 0}, + tool_response={"output": "a\n", "exit_code": 0}) + return p + + +def test_detect_tells_a_zcode_hook_from_claude_code_and_from_grok(): + """Grok also sends `hookEventName`, so a ZCode payload read as Grok's before this host + existed; the throwaway transcript is what only ZCode sends.""" + for event in ("SessionStart", "UserPromptSubmit", "PostToolUse", "Stop"): + assert isinstance(hosts.detect(zcode_payload(event), {}), hosts.ZCode), event + p = zcode_payload() + p["transcript_path"] = p["transcriptPath"] = p["transcript_path"].replace("/", "\\") + assert isinstance(hosts.detect(p, {}), hosts.ZCode) # a Windows temp directory + + +def test_an_explicit_host_hint_wins_without_a_marker(): + assert isinstance(hosts.detect({}, {}, hint="zcode"), hosts.ZCode) + assert isinstance(hosts.detect({"session_id": "s"}, {}, hint="zcode"), hosts.ZCode) + + +CLAUDE = {"session_id": "s", "hook_event_name": "PostToolUse", "cwd": "/repo", + "transcript_path": "/home/me/.claude/projects/-repo/s.jsonl", "tool_name": "Bash"} + + +@pytest.mark.parametrize("payload, env", [ + # TC-980-2: nothing but a ZCode hook's own throwaway transcript makes a payload ZCode's. + (CLAUDE, {}), + (CLAUDE, {"ZCODE_SESSION_ID": "sess_1", "ZCODE_PROJECT_DIR": "/repo"}), # started in ZCode + ({}, {"ZCODE_SESSION_ID": "sess_1"}), + ({}, {}), + ({"hookEventName": "PostToolUse", "sessionId": "s", "transcriptPath": "/home/me/.grok/" + "sessions/x/s.jsonl"}, {}), # Grok's + ({"hookEventName": "PostToolUse", "sessionId": "s"}, {}), # camelCase only + (dict(zcode_payload(), transcriptPath="/tmp/zcode-hook-Zz9/transcript.jsonl"), {}), # mismatch + (dict(zcode_payload(), transcript_path="/tmp/zcode-hook-Ab12Cd/other.jsonl", + transcriptPath="/tmp/zcode-hook-Ab12Cd/other.jsonl"), {}), # another file + (dict(zcode_payload(), transcript_path="/tmp/my-zcode-hook-1/transcript.jsonl", + transcriptPath="/tmp/my-zcode-hook-1/transcript.jsonl"), {}), # not that folder + (dict(zcode_payload(), transcript_path="/tmp/zcode-hook-/transcript.jsonl", + transcriptPath="/tmp/zcode-hook-/transcript.jsonl"), {}), # empty suffix + (dict(zcode_payload(), transcript_path=None, transcriptPath=None), {}), + (dict(zcode_payload(), transcript_path=7, transcriptPath=7), {}), + ({"transcript_path": "/home/me/.kimi-code/sessions/wd_x/session_1/agents/main/wire.jsonl"}, + {}), +]) +def test_an_unknown_payload_is_not_taken_for_zcode(payload, env): + assert not isinstance(hosts.detect(payload, env), hosts.ZCode) + + +@pytest.mark.parametrize("hint", ["kimi", "zcode2", "ZCODE", "", None, "../zcode"]) +def test_an_unknown_host_hint_is_not_zcode(hint): + assert not isinstance(hosts.detect(CLAUDE, {}, hint=hint), hosts.ZCode) + + +def test_detect_does_not_fail_on_an_empty_payload(): + for bad in (None, [], {}): + assert not isinstance(hosts.detect(bad, {}), hosts.ZCode) + + +def test_a_zcode_shell_is_about_zcode_unless_another_agent_runs_in_it(monkeypatch): + for k in ("TE_HOST", "CLAUDECODE", "COPILOT_CLI", "CURSOR_AGENT", "CURSOR_PROJECT_DIR", + "GROK_SESSION_ID", "FACTORY_SESSION_ID", "DROID_PLUGIN_ROOT"): + monkeypatch.delenv(k, raising=False) + assert hosts.default_host({"ZCODE_SESSION_ID": "sess_1"}) == "zcode" + assert hosts.default_host({"ZCODE_SESSION_ID": "s", "CLAUDECODE": "1"}) == "claude" + assert hosts.default_host({"ZCODE_SESSION_ID": "s", "GROK_SESSION_ID": "g"}) == "grok" + assert hosts.default_host({"ZCODE_SESSION_ID": "s", "TE_HOST": "codex"}) == "codex" + assert hosts.default_host({}) == "claude" + + +# ---- the host --------------------------------------------------------------------------------- +def test_the_host_reads_the_payload_zcode_sends(): + host = hosts.ZCode() + p = zcode_payload() + assert host.name == "zcode" and host.session_id(p) == "sess_1" + assert host.session_id({"sessionId": "sess_2"}) == "sess_2" + assert host.turn_key(p) == "u" + name, args, text, code = host.tool_call(p) + assert (name, args["command"], text.strip(), code) == ("Bash", "ls", "a", 0) + # ZCode's failure hook sends `error` (a message) and no response. + fail = dict(zcode_payload("PostToolUseFailure"), error="Exit code 2\nboom") + name, _args, text, code = host.tool_call(fail) + assert name == "Bash" and "boom" in text and code == 2 + + +def test_the_throwaway_transcript_is_never_the_sessions_log(): + """ZCode writes one line to a temp file and deletes it when the hook returns: recording its + path would send the fold to a file that is gone.""" + host = hosts.ZCode() + assert host.transcript(zcode_payload()) == "" + assert host.mechanism_trace("", "/tmp") is None + assert host.prompt_count("") is None or host.prompt_count("") == 0 + + +def test_the_project_dir_is_the_payload_cwd_then_zcodes_variable(): + host = hosts.ZCode() + assert host.cwd({"cwd": "/a"}, {"ZCODE_PROJECT_DIR": "/b"}) == "/a" + assert host.cwd({}, {"ZCODE_PROJECT_DIR": "/b"}) == "/b" + assert host.env_session_id({"ZCODE_SESSION_ID": "sess_9"}) == "sess_9" + assert host.env_session_id({"TE_SESSION_ID": "t", "ZCODE_SESSION_ID": "z"}) == "t" + + +def test_the_host_answers_in_the_shape_zcode_parses(): + """`hookSpecificOutput.additionalContext` with the event's own name is what ZCode's schema + takes; a name that differs from the event's is an error there. Hook messages are not shown + to the person, so a notice goes through the model.""" + host = hosts.ZCode() + assert host.prompt_context is True and host.relays_notices is True + assert host.context("SessionStart", "facts") == { + "hookSpecificOutput": {"hookEventName": "SessionStart", "additionalContext": "facts"}} + assert host.notice("hello") == {} + assert "continue" not in host.context("Stop", "x") # `continue: false` would block + + +def test_complete_exports_the_session_from_the_store_into_the_state(tmp_path, monkeypatch): + db = make_db(tmp_path / "db.sqlite", tmp_path) + monkeypatch.setenv("TOKENECONOMIZER_HOME", str(tmp_path / "state")) + host = hosts.ZCode() + path = host.complete("", "sess_1", str(tmp_path), {"zcode_db": db}) + assert path and path.startswith(str(tmp_path / "state")) + assert trace.zcode_prompt_count(path) == 1 + out = host.mechanism_trace(path, str(tmp_path)) + assert trace.load_trajectory(out)[0][0] == "Bash" + assert host.prompt_count(path) == 1 + assert host.session_usage(path)["output_tokens"] == 470 + # The fold reads the file it just wrote; the host's own log is the store, and its + # activity is what the plugin saved at each hook. + assert host.last_active("", {"updated": 12.5}) == 12.5 + + +def test_complete_waits_for_a_store_that_is_not_there_yet_then_gives_up(tmp_path, monkeypatch): + monkeypatch.setenv("TOKENECONOMIZER_HOME", str(tmp_path / "state")) + host = hosts.ZCode() + now = time.time() + with pytest.raises(hosts.FoldLater): + host.complete("", "sess_1", str(tmp_path), {"zcode_db": str(tmp_path / "no.sqlite"), + "updated": now}) + # A session nobody has touched for a week is folded as unreadable, not retried for ever. + assert host.complete("", "sess_1", str(tmp_path), { + "zcode_db": str(tmp_path / "no.sqlite"), "updated": now - host.FOLD_RETRY_S - 5}) == "" + # A session id that is no id never reaches a query. + assert host.complete("", "../x", str(tmp_path), {"zcode_db": "x"}) == "" + + +def test_session_state_records_where_the_store_is(): + assert hosts.ZCode().session_state({"ZCODE_STORAGE_DIR": "/z"}) == { + "zcode_db": "/z/cli/db/db.sqlite"} + + +def test_tool_pruning_is_not_supported_for_zcode(): + assert denylist.supports("zcode") is False + + +# ---- side calls ------------------------------------------------------------------------------- +def test_side_calls_are_off_unless_the_person_enables_them(tmp_path, monkeypatch): + from tokeneconomizer.plugin import sidecall + from tokeneconomizer.plugin.config import load_config, write_config + assert sidecall.zcode_enabled({}) is False + assert sidecall.zcode_enabled({"TE_ZCODE_SIDECALL": "1"}) is True + monkeypatch.delenv("TE_ZCODE_SIDECALL", raising=False) + cfg = load_config(str(tmp_path), env={}) + assert cfg.zcode_sidecall is False and sidecall.runner_for("zcode", cfg) is None + monkeypatch.setenv("TE_ZCODE_SIDECALL", "1") + assert callable(sidecall.runner_for("zcode", cfg)) + monkeypatch.delenv("TE_ZCODE_SIDECALL") + write_config(str(tmp_path / "config.json"), {"zcode_sidecall": "on"}) + assert callable(sidecall.runner_for("zcode", load_config(str(tmp_path), env={}))) + + +def test_the_setting_is_shown_by_config_show(): + from tokeneconomizer.plugin.cli import _config_view + from tokeneconomizer.plugin.config import LiveConfig + assert _config_view(LiveConfig())["zcode"] == {"state": "off", "zcode_sidecall": False} + + +@pytest.mark.real_sidecalls +def test_a_zcode_side_call_runs_in_plan_mode_in_an_empty_directory_with_tools_removed( + monkeypatch): + from tokeneconomizer.plugin import sidecall + monkeypatch.setattr(sidecall, "program", lambda cli: [cli]) + seen = {} + + def fake_run(cmd, **kw): + seen.update(cmd=cmd, **kw) + return subprocess.CompletedProcess(cmd, 0, stdout="- use python -m pytest\n", stderr="") + + monkeypatch.setattr(sidecall.subprocess, "run", fake_run) + text, eff = sidecall.zcode_call("propose facts") + assert text == "- use python -m pytest" and eff is None # ZCode reports no usage here + cmd = seen["cmd"] + assert cmd[:3] == ["zcode", "-p", "propose facts"] + assert cmd[cmd.index("--mode") + 1] == "plan" # its `-p` default is `yolo` + removed = cmd[cmd.index("--disallowed-tools") + 1].replace(",", " ").split() + assert {"Bash", "Edit", "Write", "WebFetch", "WebSearch", "Agent"} <= set(removed) + assert "--cwd" in cmd and cmd[cmd.index("--cwd") + 1] != os.getcwd() + assert seen["env"]["TE_SIDECALL"] == "1" and seen["stdin"] == subprocess.DEVNULL + assert seen["cwd"] == cmd[cmd.index("--cwd") + 1] + + +@pytest.mark.real_sidecalls +def test_a_zcode_side_call_that_answers_nothing_is_noted(monkeypatch, tmp_path): + from tokeneconomizer.plugin import sidecall + monkeypatch.setenv("TOKENECONOMIZER_HOME", str(tmp_path / "state")) + monkeypatch.setattr(sidecall, "program", lambda cli: [cli]) + monkeypatch.setattr(sidecall.subprocess, "run", lambda cmd, **kw: subprocess.CompletedProcess( + cmd, 1, stdout="", stderr="Please run /login")) + assert sidecall.zcode_call("p") == ("", None) + assert sidecall.paused("zcode") is not None + + +# ---- the hooks, end to end ---------------------------------------------------------------------- +@pytest.fixture +def state(tmp_path): + return {"TOKENECONOMIZER_HOME": str(tmp_path / "home"), "TE_FOLD_SYNC": "1", + "TE_MICROSKILLS": "envfacts", "TE_TOOLPRUNE": "off", + "GIT_CEILING_DIRECTORIES": str(tmp_path), "PATH": os.environ.get("PATH", "")} + + +def run_hook(env, payload, *args): + out = io.StringIO() + hook.main(list(args), io.StringIO(json.dumps(payload)), out, env) + return json.loads(out.getvalue().splitlines()[-1]) + + +def test_a_zcode_session_is_followed_and_folded_from_the_store(tmp_path, state): + repo = tmp_path / "repo" + (repo / ".git").mkdir(parents=True) + (repo / "pyproject.toml").write_text("[project]\nname='w'\n") + store = make_db(tmp_path / "db.sqlite", repo) + env = dict(state, ZCODE_STORAGE_DIR=str(tmp_path / "zstore")) + os.makedirs(tmp_path / "zstore" / "cli" / "db") + os.replace(store, tmp_path / "zstore" / "cli" / "db" / "db.sqlite") + for event in ("SessionStart", "UserPromptSubmit", "PostToolUse", "Stop"): + p = dict(zcode_payload(event), cwd=str(repo)) + out = run_hook(env, p, "--host", "zcode") + assert isinstance(out, dict) + assert "continue" not in out # a hook never stops ZCode + ctx = LiveContext.build(hosts.ZCode(), str(repo), env) + assert ctx.store.read("sessions/sess_1.json")["host"] == "zcode" + assert not ctx.store.read("sessions/sess_1.json").get("transcript") + rec = fold.fold_session(ctx, "sess_1") + assert rec["host"] == "zcode" and rec["prompts"] == 1 + assert rec["effective_tokens"] and "error" not in rec + # Folded once: a second fold finds nothing to do. + assert fold.fold_session(ctx, "sess_1") is None + + +def test_a_payload_without_a_marker_and_without_a_hint_is_followed_as_claude_not_zcode( + tmp_path, state): + """What a Claude-shaped payload does with this plugin is unchanged by the new host.""" + repo = tmp_path / "repo" + (repo / ".git").mkdir(parents=True) + out = run_hook(state, dict(CLAUDE, cwd=str(repo), hook_event_name="SessionStart")) + assert isinstance(out, dict) + ctx = LiveContext.build(hosts.ClaudeCode(), str(repo), state) + assert ctx.store.read("sessions/s.json")["host"] == "claude" diff --git a/tokeneconomizer/plugin/cli.py b/tokeneconomizer/plugin/cli.py index 5d00c72a..0271f8e3 100644 --- a/tokeneconomizer/plugin/cli.py +++ b/tokeneconomizer/plugin/cli.py @@ -48,7 +48,7 @@ import sys import time -from .. import loopguard +from .. import loopguard, trace from . import (accounting, denylist, fold, history, hosts, paths, repomap, repos, sidecall, toolcheck) from .config import canonical, unset_config, write_config @@ -585,6 +585,8 @@ def _config_view(cfg): "history_dirs": d.pop("history_dirs")}, "droid": {"state": "on" if cfg.droid_sidecall else "off", "droid_sidecall": d.pop("droid_sidecall")}, + "zcode": {"state": "on" if cfg.zcode_sidecall else "off", + "zcode_sidecall": d.pop("zcode_sidecall")}, "other": d, "warnings": cfg.warnings, } @@ -625,7 +627,8 @@ def cmd_config(args): print(f"{path}: {json.dumps(cur, sort_keys=True)}") print("applies from the next hook call; a new fact-size cap applies once the next session " "has been folded") - if {canonical(k) for k in updates} & {"microskills", "loopguard", "droid_sidecall"} and sidecall.resume( + if {canonical(k) for k in updates} & { + "microskills", "loopguard", "droid_sidecall", "zcode_sidecall"} and sidecall.resume( ctx.host.name, ctx.root): print(f"the plugin's model calls through `{sidecall.call_name(ctx.host.name)}` will be " f"tried again") @@ -816,12 +819,12 @@ def cmd_doctor(args): import shutil as _sh # Only the agents installed here: a Codex-only machine has no Claude Code catalog to probe. agents = [h for h in ("antigravity", "claude", "codex", "copilot", "droid", "grok", "hermes", - "opencode", "pi") + "opencode", "pi", "zcode") if _sh.which(sidecall.claude_cli() if h == "claude" else sidecall.hermes_cli() if h == "hermes" else "agy" if h == "antigravity" else h)] print("agents " + (', '.join(agents) or 'none of antigravity (agy), claude, codex, ' - 'copilot, droid, grok, hermes, opencode, pi on PATH')) + 'copilot, droid, grok, hermes, opencode, pi, zcode on PATH')) ctx = _ctx(args) # the per-agent lines below read the settings in effect for h in agents: pause = sidecall.paused(h, root) @@ -867,6 +870,16 @@ def cmd_doctor(args): side = ("off (set `droid_sidecall=on` or TE_DROID_SIDECALL=1 to try them)") print(f"droid sessions {sessions}; side calls {side}; tool pruning is not " f"available in the TUI") + if "zcode" in agents: + if getattr(ctx.config, "zcode_sidecall", False): + side = "on (the `zcode_sidecall` setting)" + elif sidecall.zcode_enabled(): + side = "on (TE_ZCODE_SIDECALL=1)" + else: + side = "off (set `zcode_sidecall=on` or TE_ZCODE_SIDECALL=1 to try them)" + db = trace.zcode_db() + print(f"zcode sessions {db if os.path.exists(db) else 'not found at ' + db}; side " + f"calls {side}; tool pruning is not available") if "hermes" in agents: link = os.path.join(sidecall.hermes_home(), "plugins", "tokeneconomizer") print(f"hermes plugin {'linked at ' + link if os.path.exists(link) else 'not in ' + link}" diff --git a/tokeneconomizer/plugin/config.py b/tokeneconomizer/plugin/config.py index d95eb087..1428f1cc 100644 --- a/tokeneconomizer/plugin/config.py +++ b/tokeneconomizer/plugin/config.py @@ -57,6 +57,8 @@ * droid_sidecall -- on Droid, allow the plugin's own model calls. They run `droid exec`, which writes a session log of its own, so this is off by default: `microskills=envfacts` learns without it; + * zcode_sidecall -- on ZCode, the same switch: `zcode -p` cannot run with no tool + and writes a session of its own into ZCode's store; * history_import -- on Claude Code, fold a repository's sessions from before the plugin (Claude Code's own logs) once, at its first session start, when nothing is folded there yet. Off by default: @@ -196,6 +198,7 @@ class LiveConfig: pi_cli: str = "" hermes_cli: str = "" droid_sidecall: bool = False + zcode_sidecall: bool = False history_import: bool = False history_max_sessions: int = 200 history_days: int = 365 diff --git a/tokeneconomizer/plugin/history.py b/tokeneconomizer/plugin/history.py index 95067b97..df6ac3b4 100644 --- a/tokeneconomizer/plugin/history.py +++ b/tokeneconomizer/plugin/history.py @@ -512,7 +512,7 @@ def summary(ctx): # The agent's name as a person reads it, for the lines that name whose logs were read. HOST_LABEL = {"claude": "Claude Code", "codex": "Codex", "copilot": "Copilot CLI", "pi": "Pi", "opencode": "opencode", "cursor": "Cursor", "hermes": "Hermes Agent", - "droid": "Droid"} + "droid": "Droid", "zcode": "ZCode"} def describe(doc, host="claude"): diff --git a/tokeneconomizer/plugin/hook.py b/tokeneconomizer/plugin/hook.py index 52ab1618..86425fff 100644 --- a/tokeneconomizer/plugin/hook.py +++ b/tokeneconomizer/plugin/hook.py @@ -1,7 +1,7 @@ """The entry point every plugin hook executes: hook JSON on stdin, hook JSON on stdout. python -m tokeneconomizer.plugin.hook [--host antigravity|claude|codex|copilot|cursor|droid|grok| - hermes|opencode|pi] [--event NAME] + hermes|opencode|pi|zcode] [--event NAME] It never fails the agent. Whatever goes wrong -- unreadable input, a full disk, a bug -- the hook prints `{}` (which changes nothing) and records the traceback in `/errors.log`. It diff --git a/tokeneconomizer/plugin/hosts.py b/tokeneconomizer/plugin/hosts.py index 5351a9e7..b7e3ba3c 100644 --- a/tokeneconomizer/plugin/hosts.py +++ b/tokeneconomizer/plugin/hosts.py @@ -35,9 +35,11 @@ still applies (verified on SessionStart, UserPromptSubmit and PostToolUse). So a Copilot notice is carried as `PROGRESS` lines that `hook.main` writes ahead of the output. -`detect` picks the adapter from the payload: Codex's `turn_id`, or a transcript under `.codex/` -named `rollout-*`; Copilot's session-state transcript, or no transcript at all inside a Copilot -hook process; Claude Code otherwise. Each plugin hook file except Claude Code's also passes +ZCode (Z.ai) runs the same Claude-shaped hook contract, with its own camelCase fields beside the +snake_case ones, and keeps its sessions in a SQLite store (`ZCode` below). `detect` picks the +adapter from the payload: ZCode's throwaway transcript (`_zcode_transcript`), Codex's `turn_id`, +or a transcript under `.codex/` named `rollout-*`; Copilot's session-state transcript, or no +transcript at all inside a Copilot hook process; Claude Code otherwise. Each plugin hook file except Claude Code's also passes `--host` explicitly, so detection is the fallback rather than the rule. """ import json @@ -1317,9 +1319,105 @@ def folder_matches(self, root, folder): return folder == os.path.abspath(root).replace("\\", "-").replace("/", "-") +class ZCode(Host): + """ZCode (Z.ai), read from 0.17.1: the hook code in its bundle, its CLI help and the schema + and JSON keys of its store. Not yet run under a live session with this plugin's hook file. + + ZCode runs the Claude-shaped hook contract: `SessionStart`, `UserPromptSubmit`, + `PostToolUse`, `PostToolUseFailure` and `Stop` (there is no `SessionEnd`, so a session is + folded by the next start in its repository once it has been quiet, as on Antigravity). Each + payload is ZCode's own camelCase fields (`hookEventName`, `sessionId`, `toolName`, `turnId`, + ...) with the snake_case aliases of them (`session_id`, `tool_name`, `tool_input`, + `tool_use_id`, `tool_response`), so the controller reads it as it is. What differs: + + * `transcript_path` (and `transcriptPath`) name a temporary `zcode-hook-*/transcript.jsonl` + that holds the one prompt or answer of the event and is deleted when the hook returns. It + is no session log, so it is never recorded (`transcript`); the fold reads the SQLite store + instead (`complete`, `trace.zcode_export`), and `detect` uses the file's name to tell a + ZCode payload from Claude Code's and Grok's, which also sends `hookEventName`. + * Context goes in `hookSpecificOutput.additionalContext` (the event's own name in + `hookEventName`, or ZCode raises an error) and reaches the model from SessionStart, + UserPromptSubmit and PostToolUse as a `hook_context` entry of the conversation. A hook's + `systemMessage` is read only to explain a block, so no notice reaches the person; one + goes through the model (`relay`), as on Droid. + * `continue: false` is a block request on every event but Stop, so no output here sets it. + + Tool pruning is not supported (ZCode has no per-tool setting a hook can write), and a + subagent's calls are in its own session of the store, not the parent's trace. + """ + name = "zcode" + prompt_context = True # UserPromptSubmit's additionalContext is collected + relays_notices = True + keeps_history = False # the store holds old sessions; no import reads them yet + # A session the store cannot give is retried at each start in its repository for this long, + # then folded as unreadable. + FOLD_RETRY_S = 7 * 86400 + + def session_id(self, p): + return str(p.get("session_id") or p.get("sessionId") or "") + + def transcript(self, p): + return "" + + def cwd(self, p, env): + return (p.get("cwd") or env.get("ZCODE_PROJECT_DIR") or env.get("CLAUDE_PROJECT_DIR") + or os.getcwd()) + + def env_session_id(self, env): + return str(env.get("TE_SESSION_ID") or env.get("ZCODE_SESSION_ID") or "") + + def turn_key(self, p): + return str(p.get("turnId") or "") + + def notice(self, text): + return {} + + def tool_call(self, p): + name, args, text, code = super().tool_call(p) + name, args = trace.zcode_tool(name, args) + return name, args, text, code + + def session_state(self, env): + return {"zcode_db": trace.zcode_db(env)} + + def last_active(self, transcript_path, session): + """A session has no log of its own to date, but every hook saves its state.""" + return float(session.get("updated") or session.get("started") or 0) + + def complete(self, transcript_path, session_id, cwd=None, session=None): + """The session as the store holds it now, copied to a file of the plugin's. The copy + stays as it was if the store cannot be read; with no copy either the fold waits, up to + `FOLD_RETRY_S`: the store may be busy, or this process, a `te fold` in a shell, may see + another ZCODE_STORAGE_DIR than the session's.""" + from .paths import state_root + if not re.fullmatch(r"\w[\w.-]*", session_id or ""): + return transcript_path + session = session or {} + path = os.path.join(state_root(), "zcode-sessions", f"{session_id}.json") + if trace.zcode_export(session.get("zcode_db") or trace.zcode_db(), session_id, path) \ + or os.path.exists(path): + return path + if time.time() - self.last_active(path, session) < self.FOLD_RETRY_S: + raise FoldLater(session_id) + return transcript_path + + def mechanism_trace(self, transcript_path, workdir): + if not transcript_path: + return None + out = os.path.join(workdir, "zcode_trace.jsonl") + n = trace.zcode_session_to_claude(transcript_path, out) + return out if n is not None else None + + def prompt_count(self, transcript_path): + return trace.zcode_prompt_count(transcript_path) + + def session_usage(self, transcript_path): + return trace.zcode_session_usage(transcript_path) + + HOSTS = {"claude": ClaudeCode, "codex": Codex, "copilot": Copilot, "droid": Droid, "opencode": OpenCode, "pi": Pi, "cursor": Cursor, "hermes": Hermes, "grok": Grok, - "antigravity": Antigravity} + "antigravity": Antigravity, "zcode": ZCode} def default_host(env=None, recent=None): @@ -1342,13 +1440,32 @@ def default_host(env=None, recent=None): # inside Droid is read as its own host. if env.get("FACTORY_SESSION_ID") or env.get("DROID_PLUGIN_ROOT"): return "droid" + # ZCode exports its session id to its shell and its hooks (with Claude's names beside it), and + # an agent started from a ZCode shell inherits it: last, for the same reason as Droid's. + if env.get("ZCODE_SESSION_ID"): + return "zcode" return recent if recent in HOSTS else "claude" +# ZCode hands every hook a transcript it writes to `/zcode-hook-/transcript.jsonl` +# and deletes afterwards, named in both `transcript_path` and `transcriptPath`. Nothing else sends +# that name; the environment is no mark, since an agent started from a ZCode shell inherits it. +_ZCODE_TRANSCRIPT = re.compile(r"(?:^|/)zcode-hook-[\w-]+/transcript\.jsonl$") + + +def _zcode_transcript(payload): + tp, alias = payload.get("transcript_path"), payload.get("transcriptPath") + return (isinstance(tp, str) and tp == alias + and bool(_ZCODE_TRANSCRIPT.search(tp.replace("\\", "/")))) + + def detect(payload, env=None, hint=None): """The adapter for this hook call: an explicit `hint`, else inferred from the payload.""" if hint in HOSTS: return HOSTS[hint]() + # Before Grok's test: ZCode also sends `hookEventName`. + if isinstance(payload, dict) and _zcode_transcript(payload): + return ZCode() if (payload or {}).get("conversationId") and "/antigravity" in str( payload.get("transcriptPath") or "").replace("\\", "/"): return Antigravity() diff --git a/tokeneconomizer/plugin/sidecall.py b/tokeneconomizer/plugin/sidecall.py index 73a68115..6391286f 100644 --- a/tokeneconomizer/plugin/sidecall.py +++ b/tokeneconomizer/plugin/sidecall.py @@ -51,6 +51,11 @@ left with no tool at all. Its usage is read from the call's result or from the settings file beside that log. + * ZCode: `zcode -p --mode plan --cwd --disallowed-tools ` on the model in ZCode's own settings (`zcode -p` has no `--model`). Its `-p` + default mode is `yolo`, and the tool list is a deny list, which ZCode cannot invert, so the + call is OFF unless `zcode_sidecall` is on (or `TE_ZCODE_SIDECALL=1`), as Droid's is; it + writes a session of its own into ZCode's store, and reports no usage, so it is unpriced. Every side call but Hermes's runs with `TE_SIDECALL=1`, which the plugin's hook entry point treats as "do nothing": without it the nested agent would run this plugin's hooks and recurse. A call that returns no reply -- a CLI that is not signed in, say -- leaves one line in the plugin's @@ -752,6 +757,11 @@ def copilot_call(prompt, model="", cli="copilot", timeout=TIMEOUT_S): NO_SIDE_CALLS = frozenset({"antigravity"}) +# Hosts whose side-call command has no flag that names a model, so they have no +# `_paid_model` setting: a setting that changed nothing would only mislead. +NO_MODEL_FLAG = frozenset({"zcode"}) + + def droid_enabled(env=None): """Whether `TE_DROID_SIDECALL` allows Droid side calls. The `droid_sidecall` setting does the same from the config file, which is the only way from the desktop app. They are OFF by @@ -862,6 +872,41 @@ def droid_call(prompt, model="", cli="droid", timeout=TIMEOUT_S): return text, eff +def zcode_enabled(env=None): + """Whether `TE_ZCODE_SIDECALL` allows ZCode side calls; the `zcode_sidecall` setting does the + same from the config file. OFF by default, as Droid's are: a ZCode call leaves a session in + the person's store, and its tools can only be removed by name.""" + env = os.environ if env is None else env + return str(env.get("TE_ZCODE_SIDECALL") or "").strip().lower() in ("1", "true", "yes", "on") + + +# A ZCode side call only answers, and its prompt quotes commands the session ran, so it must not +# be able to act. `zcode -p` defaults to `yolo`, so the call asks for `plan` mode, and removes by +# name every tool of the store's tool list that acts or reaches out. `--disallowed-tools` is a deny +# list (it cannot be inverted), so a tool a later ZCode adds, or one an MCP server adds, is not +# removed: that is why the call is off by default. Every name here is one ZCode 0.17.1 recorded +# in its own store; a name it does not know is not risked. +ZCODE_REMOVED = ("Bash", "Edit", "Write", "WebFetch", "WebSearch", "Agent", "Skill", "TaskOutput", + "TaskStop", "SendMessage", "CronCreate", "AskUserQuestion") + + +def zcode_call(prompt, model="", cli="zcode", timeout=TIMEOUT_S): + """A text completion through `zcode -p` in an empty throwaway directory, in plan mode, with + the acting tools removed (`ZCODE_REMOVED`). `model` is not passed: ZCode's `-p` takes none, so + the call runs on the model of the person's ZCode settings. Its reply is stdout; ZCode gives no + usage for it, so it is not priced. `TE_SIDECALL=1` makes this plugin's own hooks no-ops in the + nested session, so the call cannot recurse.""" + with tempfile.TemporaryDirectory(prefix="te-side-", ignore_cleanup_errors=True) as cwd: + cmd = [*program(cli), "-p", prompt, "--mode", "plan", "--cwd", cwd, + "--disallowed-tools", ",".join(ZCODE_REMOVED)] + out = subprocess.run(cmd, capture_output=True, encoding="utf-8", errors="replace", + timeout=timeout, env=_env(), cwd=cwd, stdin=subprocess.DEVNULL) + text = (out.stdout or "").strip() if out.returncode == 0 else "" + if not text: + _no_reply("zcode", out) + return text, None + + # The command each agent's side calls run, as the notices name it; the rest run ` -p`. _CALLS = {"codex": "codex exec", "droid": "droid exec", "opencode": "opencode run", "cursor": "cursor-agent -p", "hermes": "hermes chat"} @@ -912,4 +957,10 @@ def _host_call(host_name, config): if not (droid_enabled() or getattr(config, "droid_sidecall", False)): return None return lambda prompt, _m="agent": droid_call(prompt, model=model) + if host_name == "zcode": + # Off unless turned on, like Droid's: see `zcode_enabled`. No runner means the free layer + # still learns; only the paid proposer and the judge wait. + if not (zcode_enabled() or getattr(config, "zcode_sidecall", False)): + return None + return lambda prompt, _m="agent": zcode_call(prompt) return lambda prompt, _m="agent": claude_call(prompt, model=model or "haiku", **kw) diff --git a/tokeneconomizer/trace/README.md b/tokeneconomizer/trace/README.md index adc6f14b..8b4fcabb 100644 --- a/tokeneconomizer/trace/README.md +++ b/tokeneconomizer/trace/README.md @@ -17,6 +17,7 @@ session as an ordered list of `(tool_name, args, result_text, success)` calls th | `grok.py` | Grok Build sessions: `updates.jsonl` (ACP updates) written as a Claude transcript; prompts and usage from its session files | | `hermes.py` | Hermes Agent sessions: a copy of a session's `state.db` rows without the conversation's words, written as a Claude transcript; usage | | `pi.py` | Pi's session tree: the active branch, written as a Claude transcript; usage | +| `zcode.py` | ZCode sessions: a copy of a session's rows from its SQLite store without the conversation's words, written as a Claude transcript; usage with the cache taken out of input | | `jsonl.py` | line and record readers shared by the above | A new agent needs a reader here and nothing else: once its log reads as `(name, args, result, diff --git a/tokeneconomizer/trace/__init__.py b/tokeneconomizer/trace/__init__.py index f9d0de08..e32ea60e 100644 --- a/tokeneconomizer/trace/__init__.py +++ b/tokeneconomizer/trace/__init__.py @@ -7,7 +7,7 @@ `call_cwds`, `tool_usage`, `strip_rtk_prefix`; * `catalog` -- the tool list a session loaded, from whichever source the agent publishes; * `antigravity`, `claude`, `codex`, `copilot`, `cursor`, `droid`, `grok`, `hermes`, - `opencode`, `pi` -- each agent's own session file: usage, metadata, + `opencode`, `pi`, `zcode` -- each agent's own session file: usage, metadata, prompt-turns, and the conversions that let the readers above handle it. Stdlib only. @@ -42,6 +42,8 @@ opencode_session_to_claude, opencode_session_usage) from .pi import (pi_active_branch, pi_json_reply, pi_prompt_count, pi_session_header, pi_session_to_claude, pi_session_usage) +from .zcode import (zcode_db, zcode_export, zcode_prompt_count, zcode_session_info, + zcode_session_to_claude, zcode_session_usage, zcode_tool) __all__ = [ "RTK_PREFIX_RE", "call_cwds", "load_trajectory", "strip_rtk_prefix", "tool_usage", @@ -70,4 +72,6 @@ "opencode_session_to_claude", "opencode_session_usage", "pi_active_branch", "pi_json_reply", "pi_prompt_count", "pi_session_header", "pi_session_to_claude", "pi_session_usage", + "zcode_db", "zcode_export", "zcode_prompt_count", "zcode_session_info", + "zcode_session_to_claude", "zcode_session_usage", "zcode_tool", ] diff --git a/tokeneconomizer/trace/zcode.py b/tokeneconomizer/trace/zcode.py new file mode 100644 index 00000000..08946814 --- /dev/null +++ b/tokeneconomizer/trace/zcode.py @@ -0,0 +1,267 @@ +"""ZCode (Z.ai): a session read from its SQLite store. + +ZCode keeps every session in one database, `/cli/db/db.sqlite` +(read off 0.17.1; the schema is opencode's, with usage tables of ZCode's own): + + session (id, directory, parent_id, ...) one row per conversation + message (id, session_id, time_created, data) `data` JSON: role, modelID, tokens, ... + part (id, message_id, session_id, data) `data` JSON: type text|reasoning|tool|... + tool_usage (session_id, tool_call_id, exit_code, ...) + model_usage (session_id, query_source, model_id, input_tokens, ...) + +A hook's own `transcript_path` is no log: ZCode writes one line to a temporary file and deletes it +when the hook returns. So a fold copies a session's rows into one JSON file beside the other state +(`zcode_export`), and this module reads that file: + + {"info": {"id", "directory"}, + "messages": [{"prompt": true} | {"role": "assistant", "model", "cwd", "tools": [...]}], + "usage": []} + +The copy leaves out the words: a prompt is counted and not kept, and what the model said or +reasoned is not read at all. A tool's input and output stay, since they are what is learned +(a file written by a tool is cut to `CAP` characters, where the guard folds a result too). + +What the store says, checked on a real database (aggregates only, no conversation read): + + * A person's prompt is a `user` message with a non-empty `text` part. ZCode also writes user + messages itself: a reminder, a goal's continuation, a background task's notice. They carry + `source` (`todo_reminder`, `background_task`, `goal-continuation`, `goal_state_change`, + `agent_listing_delta`) or `synthetic`, and are no prompts. + * Tool names are Claude Code's (`Bash`, `Read`, `Write`, `Edit`, `Agent`), so no mapping is + needed; `tool_usage.exit_code` has a shell call's real exit code. + * `model_usage.input_tokens` COUNTS the cached reads (every row's total is input + output, and + the cached part is never larger than the input), where the ledger's input does not, so the + cache is taken out of it. `cache_creation_input_tokens` was 0 in every row. A subagent's + calls are rows of its own session (`parent_id` set) with `query_source` `subagent`; the + session that ran it paid for them, so they are added. A session forked from another has a + parent too, but its rows say `main_turn`: it is its own session and not added. + +Stdlib only. The store is opened read-only: ZCode is the only writer. +""" +import json +import os +import pathlib +import re +import sqlite3 + +CAP = 20000 +# A session's id as ZCode writes it (`sess_...`); it names a file in the state and binds into a +# query, so anything else is refused. +_ID_RE = re.compile(r"\w[\w.-]*") +_PATCH_FILE_RE = re.compile(r"^\*\*\* (?:Update|Add|Delete) File:\s*(.+?)\s*$", re.M) +# The `source` of a user message ZCode wrote itself. A source not listed counts as a prompt: +# losing a prompt costs less than a reminder counted as one. +MACHINE_SOURCES = frozenset({"todo_reminder", "background_task", "goal-continuation", + "goal_state_change", "agent_listing_delta"}) +PROMPT = "[ZCode prompt]" +_USAGE_COLUMNS = ("model_id", "query_source", "input_tokens", "output_tokens", "reasoning_tokens", + "cache_creation_input_tokens", "cache_read_input_tokens") + + +def zcode_db(env=None): + """Where ZCode keeps its sessions: `ZCODE_STORAGE_DIR` (ZCode's own override; its beta build + sets `~/.zcode-beta`) or `~/.zcode`.""" + env = os.environ if env is None else env + root = str(env.get("ZCODE_STORAGE_DIR") or "").strip() or os.path.join( + os.path.expanduser("~"), ".zcode") + return os.path.join(root, "cli", "db", "db.sqlite") + + +def zcode_tool(name, args): + """(name, args) of a call as the mechanisms read it: ZCode's names are Claude Code's, but a + GPT model on ZCode edits with a whole patch, the format of Codex's `apply_patch`.""" + args = args if isinstance(args, dict) else {} + if str(name).lower() in ("applypatch", "apply_patch"): + patch = next((v for v in (args.get("input"), args.get("patch"), args.get("command")) + if isinstance(v, str)), "") + paths = _PATCH_FILE_RE.findall(patch) + return "apply_patch", {"path": paths[0] if paths else "", "paths": paths} + return name, args + + +# -- the live database, for a fold ---------------------------------------------------------- +def _connect(db_path): + con = sqlite3.connect(pathlib.Path(db_path).as_uri() + "?mode=ro", uri=True, timeout=5) + con.row_factory = sqlite3.Row + return con + + +def _json(text): + try: + v = json.loads(text) + except (TypeError, ValueError): + return {} + return v if isinstance(v, dict) else {} + + +def _cut(v): + return v[:CAP] if isinstance(v, str) else v + + +def _is_prompt(message, parts): + """Whether a `user` message is something a person typed.""" + if message.get("synthetic") or message.get("source") in MACHINE_SOURCES: + return False + return any(p.get("type") == "text" and not p.get("synthetic") + and str(p.get("text") or "").strip() for p in parts) + + +def _tool(part, exit_codes): + state = part.get("state") if isinstance(part.get("state"), dict) else {} + call_id = part.get("callID") + code = exit_codes.get(call_id) + return {"id": call_id, "tool": part.get("tool"), + "input": {k: _cut(v) for k, v in (state.get("input") or {}).items()} + if isinstance(state.get("input"), dict) else {}, + "output": _cut(state.get("output")) if isinstance(state.get("output"), str) else "", + "error": _cut(state.get("error")) if isinstance(state.get("error"), str) else "", + "status": state.get("status"), + "exit_code": code if isinstance(code, int) and not isinstance(code, bool) else None} + + +def zcode_export(db_path, session_id, path): + """Write one session's rows as the JSON this module reads; whether it was written. False for + a store that is missing, locked, not ZCode's or lacking the session, and then nothing is + written (nor is a store created: it is opened read-only).""" + if not isinstance(session_id, str) or not _ID_RE.fullmatch(session_id): + return False + try: + con = _connect(db_path) + try: + row = con.execute("SELECT id, directory FROM session WHERE id = ?", + (session_id,)).fetchone() + if not row: + return False + messages = con.execute("SELECT id, data FROM message WHERE session_id = ? " + "ORDER BY time_created, id", (session_id,)).fetchall() + parts = {} + for r in con.execute("SELECT message_id, data FROM part WHERE session_id = ? " + "ORDER BY id", (session_id,)): + parts.setdefault(r["message_id"], []).append(_json(r["data"])) + exit_codes = {r["tool_call_id"]: r["exit_code"] for r in con.execute( + "SELECT tool_call_id, exit_code FROM tool_usage WHERE session_id = ?", + (session_id,))} + usage = con.execute( + "WITH RECURSIVE sub(id) AS (SELECT id FROM session WHERE parent_id = ? " + "UNION SELECT s.id FROM session s JOIN sub ON s.parent_id = sub.id) " + f"SELECT {', '.join(_USAGE_COLUMNS)} FROM model_usage WHERE session_id = ? " + "OR (session_id IN (SELECT id FROM sub) AND query_source = 'subagent')", + (session_id, session_id)).fetchall() + finally: + con.close() + out = [] + for m in messages: + data, mparts = _json(m["data"]), parts.get(m["id"], []) + if data.get("role") == "user": + if _is_prompt(data, mparts): + out.append({"role": "user", "prompt": True}) + elif data.get("role") == "assistant": + tools = [_tool(p, exit_codes) for p in mparts if p.get("type") == "tool"] + if tools: + cwd = (data.get("path") or {}).get("cwd") if isinstance( + data.get("path"), dict) else None + out.append({"role": "assistant", "model": data.get("modelID"), + "cwd": cwd if isinstance(cwd, str) else None, "tools": tools}) + doc = {"info": {"id": row["id"], "directory": row["directory"]}, "messages": out, + "usage": [dict(u) for u in usage]} + os.makedirs(os.path.dirname(path), mode=0o700, exist_ok=True) + tmp = f"{path}.{os.getpid()}.tmp" + with open(os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600), "w", + encoding="utf-8") as fh: + json.dump(doc, fh) + os.replace(tmp, path) + return True + except (OSError, ValueError, TypeError, sqlite3.Error): + return False + + +# -- the export file ------------------------------------------------------------------------ +def _load(path): + """(info, messages, usage) of an export file; ({}, [], []) when it cannot be read.""" + try: + with open(path, encoding="utf-8") as fh: + data = json.load(fh) + except (OSError, ValueError, TypeError): + return {}, [], [] + if not isinstance(data, dict): + return {}, [], [] + rows = [[r for r in data.get(k) or [] if isinstance(r, dict)] if isinstance( + data.get(k) or [], list) else [] for k in ("messages", "usage")] + return (data.get("info") if isinstance(data.get("info"), dict) else {}), rows[0], rows[1] + + +def zcode_session_info(path): + """The session's row ({id, directory}), {} if the export does not carry it.""" + return _load(path)[0] + + +def zcode_prompt_count(path): + """How many prompts the conversation ran.""" + return sum(1 for m in _load(path)[1] if m.get("prompt")) + + +def _failed(tool): + code = tool.get("exit_code") + return tool.get("status") == "error" or (isinstance(code, int) and code != 0) + + +def zcode_session_to_claude(path, out_path): + """Write an export as Claude Code transcript lines. Returns the number of tool calls written, + or None when the file is missing or holds no messages. A call that never finished is written + without a result, as a session that ended mid-call.""" + if not path or not os.path.exists(path): + return None + info, messages, _ = _load(path) + if not messages: + return None + n = 0 + with open(out_path, "w", encoding="utf-8") as out: + for m in messages: + if m.get("prompt"): + out.write(json.dumps({"type": "user", "message": { + "role": "user", "content": PROMPT}}) + "\n") + continue + tools = [t for t in m.get("tools") or [] if isinstance(t, dict)] + cwd = m.get("cwd") or info.get("directory") + blocks = [] + for t in tools: + name, args = zcode_tool(str(t.get("tool") or ""), t.get("input")) + blocks.append({"type": "tool_use", "id": t.get("id"), "name": name, "input": args}) + n += len(blocks) + if blocks: + out.write(json.dumps(dict({"type": "assistant", "message": { + "role": "assistant", "content": blocks, "model": m.get("model")}}, + **({"cwd": cwd} if isinstance(cwd, str) and cwd else {}))) + "\n") + for t in tools: + if t.get("status") in ("running", None): + continue + out.write(json.dumps({"type": "user", "message": {"role": "user", "content": [{ + "type": "tool_result", "tool_use_id": t.get("id"), + "content": t.get("output") or t.get("error") or "", + "is_error": _failed(t)}]}}) + "\n") + return n + + +def _int(row, key): + v = row.get(key) + return v if isinstance(v, int) and not isinstance(v, bool) else 0 + + +def zcode_session_usage(path): + """Token breakdown of a session and its subagents, in the ledger's fields. `model` is the one + that served the most calls; the title call a session makes of its own counts as one.""" + totals = {"input_tokens": 0, "cache_read_input_tokens": 0, + "cache_creation_input_tokens": 0, "output_tokens": 0} + models, requests = {}, 0 + for row in _load(path)[2]: + cached = _int(row, "cache_read_input_tokens") + totals["input_tokens"] += max(0, _int(row, "input_tokens") - cached) + totals["cache_read_input_tokens"] += cached + totals["cache_creation_input_tokens"] += _int(row, "cache_creation_input_tokens") + # Reasoning is billed as output; the store reported 0 of it in every row seen. + totals["output_tokens"] += _int(row, "output_tokens") + _int(row, "reasoning_tokens") + requests += 1 + if row.get("model_id"): + models[row["model_id"]] = models.get(row["model_id"], 0) + 1 + totals.update(requests=requests, model=max(models, key=models.get) if models else None) + return totals