diff --git a/docs/architecture/canonical-session.md b/docs/architecture/canonical-session.md index 8a7c0182..e2ab3892 100644 --- a/docs/architecture/canonical-session.md +++ b/docs/architecture/canonical-session.md @@ -38,7 +38,7 @@ stays excluded. ## Projection version -`session.ProjectionVersion = 12`. The server's push handler compares +`session.ProjectionVersion = 13`. The server's push handler compares `projection_version >= session.ProjectionVersion` before short- circuiting, so bumping this constant forces existing sessions to be re-projected on the next push from any client. Recent versions: @@ -49,7 +49,9 @@ Hermes parent edges from `state.db` and transcript `parent_session_id` fields; v10 projects Hermes `state.db` rows to per-session canonical JSONL; v11 maps `ParentSessionID` onto the push wire and imports Claude Code subagent transcripts under their real on-disk naming; v12 -projects special-session `Kinds` into the `session_kinds` table. +projects special-session `Kinds` into the `session_kinds` table; v13 reads +Hermes usage from the `state.db` session-row token counters and projects +them as a leading `session_usage` line. | Version | Brought | |---|---| @@ -65,6 +67,7 @@ projects special-session `Kinds` into the `session_kinds` table. | 10 | Hermes `state.db` rows project to per-session canonical JSONL — the raw artifact for those sessions is a per-session `.jsonl` instead of the multi-session `.db` container; `raw_hash` / `raw_size` describe the projected JSONL. | | 11 | parent edges reach the server — `sessionToProto` maps `Session.ParentSessionID` onto the wire (previously dropped at push, so the server never stored an edge). Claude Code subagent transcripts are imported under their real layouts: the walker accepts `agent-.jsonl` alongside the older `agent-.jsonl`, both directly under `subagents/` (Agent tool) and nested at `subagents/workflows/wf_/` (Workflow tool). The child session id is the filename stem because every record inside carries the parent's `sessionId`; the parent UUID is the directory above the innermost `subagents` component. | | 12 | special-session classification projected — `Session.Kinds` carries any of `goal` (Codex `` first user turn), `workflow` (Claude Code `Workflow` tool), `ralph-loop` (Claude Code `/ralph-loop:ralph-loop` command), and `orchestrator` (session that spawned subagents). Stored in the `session_kinds` table (migration `0010_session_kinds` local / `0013_session_kinds` server) and pushed on the wire as `Session.kinds`. `goal`/`workflow`/`ralph-loop` are derived per session by `internal/sessionkind`; `orchestrator` is edge-derived — the client reconciles it post-sweep via `store.RefreshOrchestratorKinds` and the server derives it from parent edges at push (`deriveOrchestratorKinds`). The Codex goal `` is also unwrapped into `FirstPrompt` instead of the raw scaffold. | +| 13 | Hermes usage read from `state.db`'s `sessions` token counters instead of `messages.token_count`, which Hermes no longer populates. `InputTokens` is the cache-inclusive sum; `reasoning_tokens` stays out of `OutputTokens`. The projected JSONL gains a leading `{"type":"session_usage","data":{…}}` line, so `raw_hash` changes and Hermes sessions re-push. | ## Import eligibility @@ -310,7 +313,8 @@ and a `sessions.json` index). The full reference is `docs/sources/hermes.md`. | `FirstPrompt` | first `messages.role=="user"` row (SQLite) or first user line/message (JSONL/JSON) with non-empty `content` (text or first text item of an array); whitespace-collapsed + truncated to 200 runes | | `Model` | `sessions.model` (SQLite); `model` (JSON snapshot); first assistant-side `model` encountered (JSONL) | | `ParentSessionID` | `sessions.parent_session_id` (SQLite); top-level `parent_session_id` (JSON snapshot); first per-message `parent_session_id` encountered (JSONL / snapshot messages) | -| `RawPath` / `RawHash` / `RawSize` | per-session JSONL at `$PROSA_HOME/raw/hermes///.jsonl`; `RawHash`/`RawSize` describe the per-session artifact. For `.jsonl` / `session_*.json` shapes the source bytes are preserved verbatim (extension follows the source); for `state.db`, each `sessions` row is projected to its own JSONL (`messages` rows + hidden reasoning/codex/tool-call columns, one per line). The multi-session `.db` is **not** copied — see issue #235 | +| `Usage` | `state.db` rows: the `sessions` token counters. `InputTokens` is `input_tokens + cache_read_tokens + cache_write_tokens` because Hermes stores uncached input alone; `CacheReadTokens`/`CachedTokens` from `cache_read_tokens`, `CacheCreationTokens` from `cache_write_tokens`, `OutputTokens` from `output_tokens`. `reasoning_tokens` is already inside `output_tokens` and is provenance only. All-zero counters classify Unknown, not ExplicitZero. `.jsonl` / `session_*.json` shapes carry no counters, so a session that defers to a fuller sibling transcript has no usage — see the dual-source gap in `docs/sources/hermes.md` | +| `RawPath` / `RawHash` / `RawSize` | per-session JSONL at `$PROSA_HOME/raw/hermes///.jsonl`; `RawHash`/`RawSize` describe the per-session artifact. For `.jsonl` / `session_*.json` shapes the source bytes are preserved verbatim (extension follows the source); for `state.db`, each `sessions` row is projected to its own JSONL (a leading `session_usage` line when the row carries token counters, then `messages` rows + hidden reasoning/codex/tool-call columns, one per line). The multi-session `.db` is **not** copied — see issue #235 | ### `session.Turn` diff --git a/docs/sources/hermes.md b/docs/sources/hermes.md index b26a01aa..e76adad8 100644 --- a/docs/sources/hermes.md +++ b/docs/sources/hermes.md @@ -57,18 +57,24 @@ time — same idiom as Cursor and Gemini. ```sql CREATE TABLE sessions ( - id TEXT PRIMARY KEY, - source TEXT NOT NULL, - model TEXT, - model_config TEXT, - system_prompt TEXT, - parent_session_id TEXT, - started_at REAL NOT NULL, - ended_at REAL, - end_reason TEXT, - message_count INTEGER, - tool_call_count INTEGER, - title TEXT + id TEXT PRIMARY KEY, + source TEXT NOT NULL, + model TEXT, + model_config TEXT, + system_prompt TEXT, + parent_session_id TEXT, + started_at REAL NOT NULL, + ended_at REAL, + end_reason TEXT, + message_count INTEGER, + tool_call_count INTEGER, + input_tokens INTEGER DEFAULT 0, + output_tokens INTEGER DEFAULT 0, + cache_read_tokens INTEGER DEFAULT 0, + cache_write_tokens INTEGER DEFAULT 0, + reasoning_tokens INTEGER DEFAULT 0, + estimated_cost_usd REAL, + title TEXT ); CREATE TABLE messages ( @@ -98,6 +104,37 @@ importer normalizes both to UTC `time.Time` after parse. hold plain text or JSON-encoded values; the importer treats them as opaque strings unless a column is explicitly parsed (`tool_calls`). +Hermes counts tokens **per session**, on the `sessions` row. The importer +reads those five counters and ignores `messages.token_count`, which Hermes +stopped populating when the counters landed. The mapping onto prosa's +canonical aggregate: + +| prosa `TokenUsage` | Hermes column | +| --- | --- | +| `InputTokens` | `input_tokens + cache_read_tokens + cache_write_tokens` | +| `CacheReadTokens`, `CachedTokens` | `cache_read_tokens` | +| `CacheCreationTokens` | `cache_write_tokens` | +| `OutputTokens` | `output_tokens` | + +Two rules the mapping depends on. `input_tokens` holds **uncached** input +only, so prosa's cache-inclusive `InputTokens` is the sum of all three prompt +columns — Hermes derives its own `prompt_tokens` the same way. And +`reasoning_tokens` is already counted inside `output_tokens`, which Hermes's +`total_tokens` confirms by not adding it; prosa keeps it as provenance and +never folds it into `OutputTokens`. + +A row whose counters are all zero classifies Unknown, not ExplicitZero, so it +is still imported. This mirrors Hermes's own `has_usage` flag, which only +records usage once a counter is non-zero. + +`estimated_cost_usd` is read for reference only. Hermes prices +subscription-covered routes at zero, so prosa estimates cost from its own rate +table instead, the same treatment Claude Code and Codex sessions get. + +A Hermes build predating the counters still imports: the sessions query +projects missing columns as `NULL`, and usage falls back to +`messages.token_count`. + ## Transcript files (`.jsonl`) Top-level `.jsonl` is one message object per line. The @@ -323,12 +360,18 @@ What `session.Turn` and `session.ToolUsage` surface for Hermes today: - every per-message hidden column — `messages.reasoning`, `reasoning_content`, `reasoning_details`, `codex_reasoning_items`, `codex_message_items`, `tool_call_id`, `tool_name`, `finish_reason`, - `token_count`; + `token_count` (no longer the usage source — see the session-level + counters above, which the projection carries on its own line); - the full body of every `tool_calls` payload and `tool`-role `content` blob, beyond what the `ToolUsage` aggregate counts; - for sibling `.jsonl` / `session_.json` shapes, the source bytes verbatim (including the snapshot envelope's `system_prompt` / `platform` / `last_updated` when present). +- **Projected usage line**: a `state.db` row carrying token counters leads + its projected JSONL with + `{"type":"session_usage","data":{"input_tokens":…,"output_tokens":…,"cache_read_tokens":…,"cache_write_tokens":…,"reasoning_tokens":…}}`, + so the preserved raw explains the usage prosa derives from it. Rows with no + counters project messages only. - **Not preserved in raw** for `state.db`-sourced sessions: - session-level columns without a per-message equivalent (`sessions.system_prompt`, `model_config`, `end_reason`, `title`, @@ -341,4 +384,7 @@ What `session.Turn` and `session.ToolUsage` surface for Hermes today: other had more messages), only the winning surface lands as raw — there is no separate copy of the dropped side. A future cut that wants to merge them must read both sources at projection time; the - importer at this cut does not attempt the merge. + importer at this cut does not attempt the merge. The visible cost of + that gap is usage: a session that defers to a sibling transcript gets + no token counters, because they live only on the `state.db` row and a + transcript's raw is a verbatim copy that cannot carry them. diff --git a/internal/importers/hermes/importer.go b/internal/importers/hermes/importer.go index ff5f2cbc..66cf3710 100644 --- a/internal/importers/hermes/importer.go +++ b/internal/importers/hermes/importer.go @@ -15,6 +15,7 @@ package hermes import ( "context" + "encoding/json" "fmt" "path/filepath" "time" @@ -184,6 +185,16 @@ func (i *Importer) importStateDB(ctx context.Context, path string, sink importer if err != nil { return importer.ImportResult{}, fmt.Errorf("project session %s: %w", row.id, err) } + // Hermes counts tokens on the session row, not per message, so the + // counters lead the projection: without them the raw could not + // explain the usage prosa derives from it. + if _, ok := row.usage.toTokenUsage(); ok { + usageLine, err := marshalProjectedUsageLine(row.usage) + if err != nil { + return importer.ImportResult{}, fmt.Errorf("project session %s: %w", row.id, err) + } + lines = append([]json.RawMessage{usageLine}, lines...) + } rawPath, rawHash, rawSize, err := importerutil.PreserveProjectedJSONL(Name, row.id, sess.StartedAt, lines) if err != nil { return importer.ImportResult{}, fmt.Errorf("preserve projected raw %s: %w", row.id, err) diff --git a/internal/importers/hermes/importer_test.go b/internal/importers/hermes/importer_test.go index 5da576fa..db81ecc7 100644 --- a/internal/importers/hermes/importer_test.go +++ b/internal/importers/hermes/importer_test.go @@ -60,6 +60,14 @@ type hermesStateRow struct { model string startedAt float64 messages []hermesStateMessage + + // Session-level token counters. Hermes stores uncached input in + // inputTokens; cacheRead/cacheWrite are the rest of the prompt. + inputTokens int64 + outputTokens int64 + cacheRead int64 + cacheWrite int64 + reasoning int64 } type hermesStateMessage struct { @@ -82,10 +90,23 @@ type hermesStateMessage struct { codexMessageItems string } -// buildHermesStateDB writes a state.db with the Hermes schema. Optional -// message-level columns land as NULL when the field is zero, matching -// older Hermes builds that lack those columns. +// buildHermesStateDB writes a state.db with the current Hermes schema, +// session-level token counters included. Optional message-level columns land +// as NULL when the field is zero. func buildHermesStateDB(t *testing.T, dir string, rows []hermesStateRow) string { + t.Helper() + return buildStateDB(t, dir, rows, true) +} + +// buildLegacyHermesStateDB writes a state.db from a Hermes build predating +// the session-level token counters, so the importer's NULL-placeholder +// projection stays covered. +func buildLegacyHermesStateDB(t *testing.T, dir string, rows []hermesStateRow) string { + t.Helper() + return buildStateDB(t, dir, rows, false) +} + +func buildStateDB(t *testing.T, dir string, rows []hermesStateRow, withUsage bool) string { t.Helper() require.NoError(t, os.MkdirAll(dir, 0o755)) dbPath := filepath.Join(dir, "state.db") @@ -94,6 +115,16 @@ func buildHermesStateDB(t *testing.T, dir string, rows []hermesStateRow) string require.NoError(t, err) defer func() { _ = db.Close() }() + usageColumns := "" + if withUsage { + usageColumns = ` + input_tokens INTEGER DEFAULT 0, + output_tokens INTEGER DEFAULT 0, + cache_read_tokens INTEGER DEFAULT 0, + cache_write_tokens INTEGER DEFAULT 0, + reasoning_tokens INTEGER DEFAULT 0,` + } + _, err = db.Exec(` CREATE TABLE sessions ( id TEXT PRIMARY KEY, @@ -106,7 +137,7 @@ CREATE TABLE sessions ( ended_at REAL, end_reason TEXT, message_count INTEGER, - tool_call_count INTEGER, + tool_call_count INTEGER,` + usageColumns + ` title TEXT ); CREATE TABLE messages ( @@ -129,10 +160,21 @@ CREATE TABLE messages ( require.NoError(t, err) for _, r := range rows { - _, err = db.Exec( - `INSERT INTO sessions(id, source, model, parent_session_id, started_at, message_count) VALUES (?, ?, ?, ?, ?, ?)`, - r.id, "cli", r.model, r.parentID, r.startedAt, len(r.messages), - ) + if withUsage { + _, err = db.Exec( + `INSERT INTO sessions( + id, source, model, parent_session_id, started_at, message_count, + input_tokens, output_tokens, cache_read_tokens, cache_write_tokens, reasoning_tokens + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + r.id, "cli", r.model, r.parentID, r.startedAt, len(r.messages), + r.inputTokens, r.outputTokens, r.cacheRead, r.cacheWrite, r.reasoning, + ) + } else { + _, err = db.Exec( + `INSERT INTO sessions(id, source, model, parent_session_id, started_at, message_count) VALUES (?, ?, ?, ?, ?, ?)`, + r.id, "cli", r.model, r.parentID, r.startedAt, len(r.messages), + ) + } require.NoError(t, err) for _, m := range r.messages { _, err = db.Exec( @@ -610,6 +652,150 @@ func TestImportStateDBProjectsJSONL(t *testing.T) { require.NotContains(t, plainAssistant, "codex_reasoning_items") } +// TestImportStateDBUsesSessionTokenCounters pins the mapping from Hermes's +// session-level counters onto the canonical aggregate. Two things are easy to +// get wrong and both are asserted here: InputTokens is the cache-inclusive +// sum, because Hermes stores uncached input only, and reasoning_tokens stays +// out of OutputTokens, because Hermes already counts it there. +func TestImportStateDBUsesSessionTokenCounters(t *testing.T) { + ctx := context.Background() + t.Setenv("PROSA_HOME", filepath.Join(t.TempDir(), "prosa-home")) + + hermesHome := filepath.Join(t.TempDir(), ".hermes") + require.NoError(t, os.MkdirAll(filepath.Join(hermesHome, "sessions"), 0o755)) + + base := time.Date(2026, 8, 1, 9, 0, 0, 0, time.UTC) + dbPath := buildHermesStateDB(t, hermesHome, []hermesStateRow{{ + id: "usage-1", model: "gpt-5.5", startedAt: float64(base.Unix()), + inputTokens: 100, outputTokens: 20, cacheRead: 1000, cacheWrite: 50, reasoning: 7, + messages: []hermesStateMessage{ + {role: "user", content: "prompt", timestamp: float64(base.Unix())}, + { + role: "assistant", content: "answer", + timestamp: float64(base.Add(time.Second).Unix()), + // A stale per-message count must lose to the session row. + tokenCount: ptrInt64(3), + }, + }, + }}) + + sink := newSink() + _, err := New().Import(ctx, dbPath, sink, importer.ImportOptions{}) + require.NoError(t, err) + + usage := sink.Sessions["usage-1"].Usage + require.NotNil(t, usage) + require.Equal(t, int64(1150), usage.InputTokens, "input must include cache read and cache write") + require.Equal(t, int64(20), usage.OutputTokens, "reasoning is already inside output") + require.Equal(t, int64(1000), usage.CacheReadTokens) + require.Equal(t, int64(1000), usage.CachedTokens) + require.Equal(t, int64(50), usage.CacheCreationTokens) + require.Equal(t, int64(1170), usage.TotalTokens) +} + +// TestImportStateDBProjectsUsageProvenance covers the raw side of the same +// change: the counters prosa derives usage from must be visible in the +// preserved artifact, so the projection leads with a session_usage line. +func TestImportStateDBProjectsUsageProvenance(t *testing.T) { + ctx := context.Background() + t.Setenv("PROSA_HOME", filepath.Join(t.TempDir(), "prosa-home")) + + hermesHome := filepath.Join(t.TempDir(), ".hermes") + require.NoError(t, os.MkdirAll(filepath.Join(hermesHome, "sessions"), 0o755)) + + base := time.Date(2026, 8, 2, 9, 0, 0, 0, time.UTC) + dbPath := buildHermesStateDB(t, hermesHome, []hermesStateRow{{ + id: "prov-1", model: "gpt-5.5", startedAt: float64(base.Unix()), + inputTokens: 11, outputTokens: 22, cacheRead: 33, cacheWrite: 44, reasoning: 55, + messages: []hermesStateMessage{ + {role: "user", content: "prompt", timestamp: float64(base.Unix())}, + }, + }}) + + sink := newSink() + _, err := New().Import(ctx, dbPath, sink, importer.ImportOptions{}) + require.NoError(t, err) + + sess := sink.Sessions["prov-1"] + contents, err := os.ReadFile(sess.RawPath) + require.NoError(t, err) + lines := strings.Split(string(contents), "\n") + require.Len(t, lines, 2, "one usage line then one message line") + require.JSONEq(t, + `{"type":"session_usage","data":{"input_tokens":11,"output_tokens":22,`+ + `"cache_read_tokens":33,"cache_write_tokens":44,"reasoning_tokens":55}}`, + lines[0], + ) + require.Contains(t, lines[1], `"role":"user"`) + + // The hash must still describe the bytes actually written. + require.Equal(t, int64(len(contents)), sess.RawSize) + sum := sha256.Sum256(contents) + require.Equal(t, hex.EncodeToString(sum[:]), sess.RawHash) +} + +// TestImportStateDBZeroCountersStayAdmitted guards the sessions that carry no +// usage at all. Claiming a usage event was seen would classify them +// ExplicitZero, and the import policy drops those outright — so a session +// that exists today would silently vanish from the store. +func TestImportStateDBZeroCountersStayAdmitted(t *testing.T) { + ctx := context.Background() + t.Setenv("PROSA_HOME", filepath.Join(t.TempDir(), "prosa-home")) + + hermesHome := filepath.Join(t.TempDir(), ".hermes") + require.NoError(t, os.MkdirAll(filepath.Join(hermesHome, "sessions"), 0o755)) + + base := time.Date(2026, 8, 3, 9, 0, 0, 0, time.UTC) + dbPath := buildHermesStateDB(t, hermesHome, []hermesStateRow{{ + id: "zero-1", model: "gpt-5.5", startedAt: float64(base.Unix()), + messages: []hermesStateMessage{ + {role: "user", content: "prompt", timestamp: float64(base.Unix())}, + }, + }}) + + sink := newSink() + _, err := New().Import(ctx, dbPath, sink, importer.ImportOptions{}) + require.NoError(t, err) + + sess, ok := sink.Sessions["zero-1"] + require.True(t, ok, "a session with no counters must still be imported") + require.Nil(t, sess.Usage) + + contents, err := os.ReadFile(sess.RawPath) + require.NoError(t, err) + require.NotContains(t, string(contents), "session_usage", + "no counters means no provenance line to project") +} + +// TestImportStateDBWithoutUsageColumns exercises the NULL-placeholder +// projection: a Hermes build predating the session counters must still +// import, falling back to the per-message token_count it does have. +func TestImportStateDBWithoutUsageColumns(t *testing.T) { + ctx := context.Background() + t.Setenv("PROSA_HOME", filepath.Join(t.TempDir(), "prosa-home")) + + hermesHome := filepath.Join(t.TempDir(), ".hermes") + require.NoError(t, os.MkdirAll(filepath.Join(hermesHome, "sessions"), 0o755)) + + base := time.Date(2026, 8, 4, 9, 0, 0, 0, time.UTC) + dbPath := buildLegacyHermesStateDB(t, hermesHome, []hermesStateRow{{ + id: "legacy-1", model: "claude-opus-4-7", startedAt: float64(base.Unix()), + messages: []hermesStateMessage{ + {role: "user", content: "prompt", timestamp: float64(base.Unix()), tokenCount: ptrInt64(9)}, + }, + }}) + + sink := newSink() + _, err := New().Import(ctx, dbPath, sink, importer.ImportOptions{}) + require.NoError(t, err) + + sess, ok := sink.Sessions["legacy-1"] + require.True(t, ok) + require.NotNil(t, sess.Usage) + require.Equal(t, int64(9), sess.Usage.TotalTokens) + require.Zero(t, sess.Usage.InputTokens, "the old column carries no input/output split") +} + // TestImportStateDBNoLongerCopiesFullDB is the explicit regression guard // for issue #235: after importing a state.db, no `.db` files may exist // anywhere under the prosa raw tree. @@ -665,6 +851,7 @@ func TestImportStateDBProjectionDeterministic(t *testing.T) { rows := []hermesStateRow{ { id: "det-1", model: "claude-sonnet-4-6", startedAt: float64(base.Unix()), + inputTokens: 12, outputTokens: 3, cacheRead: 400, cacheWrite: 5, reasoning: 1, messages: []hermesStateMessage{ { role: "assistant", content: "answer", diff --git a/internal/importers/hermes/parse.go b/internal/importers/hermes/parse.go index d7d0ebb8..afa2ba57 100644 --- a/internal/importers/hermes/parse.go +++ b/internal/importers/hermes/parse.go @@ -64,6 +64,44 @@ type stateDBRow struct { startedAt sql.NullFloat64 startedAtStr sql.NullString messageCount sql.NullInt64 + usage stateDBUsage +} + +// stateDBUsage mirrors the token counters Hermes keeps on the `sessions` +// row. Field order and JSON keys are the projected `session_usage` line's +// schema, so they must stay stable. +type stateDBUsage struct { + Input int64 `json:"input_tokens"` + Output int64 `json:"output_tokens"` + CacheRead int64 `json:"cache_read_tokens"` + CacheWrite int64 `json:"cache_write_tokens"` + Reasoning int64 `json:"reasoning_tokens"` +} + +// toTokenUsage maps Hermes's counters onto the canonical aggregate. +// +// Hermes stores uncached input in `input_tokens`, so InputTokens is the +// cache-inclusive sum — the same convention the Claude Code importer uses, +// and exactly how Hermes derives its own `prompt_tokens`. `reasoning_tokens` +// is already inside `output_tokens` and is provenance only; adding it would +// double count. +// +// ok is false when every counter is zero, mirroring Hermes's own has_usage +// flag. That keeps a session with no counters classified Unknown rather than +// ExplicitZero, which the import policy would drop. +func (u stateDBUsage) toTokenUsage() (*session.TokenUsage, bool) { + if u.Input == 0 && u.Output == 0 && u.CacheRead == 0 && u.CacheWrite == 0 { + return nil, false + } + input := u.Input + u.CacheRead + u.CacheWrite + return &session.TokenUsage{ + TotalTokens: input + u.Output, + InputTokens: input, + OutputTokens: u.Output, + CachedTokens: u.CacheRead, + CacheReadTokens: u.CacheRead, + CacheCreationTokens: u.CacheWrite, + }, true } // toolCall is one entry of a Hermes `tool_calls` array. @@ -305,15 +343,32 @@ func readStateDBSessions(ctx context.Context, path string) ([]stateDBRow, error) } defer func() { _ = db.Close() }() - hasParentID, err := tableHasColumn(ctx, db, "sessions", "parent_session_id") + cols, err := columnSet(ctx, db, "sessions") if err != nil { return nil, err } - parentExpr := "NULL AS parent_session_id" - if hasParentID { - parentExpr = "parent_session_id" + // Optional columns: `parent_session_id` and the token counters were each + // added by a later Hermes build. A missing one becomes a `NULL AS ` + // placeholder so the scan targets stay fixed. The order here matches the + // Scan target order below — keep them in sync. + optional := []string{ + "parent_session_id", + "input_tokens", + "output_tokens", + "cache_read_tokens", + "cache_write_tokens", + "reasoning_tokens", } - query := fmt.Sprintf(`SELECT id, model, %s, started_at, message_count FROM sessions ORDER BY started_at`, parentExpr) + exprs := make([]string, 0, len(optional)) + for _, name := range optional { + if cols[name] { + exprs = append(exprs, name) + } else { + exprs = append(exprs, "NULL AS "+name) + } + } + query := "SELECT id, model, started_at, message_count, " + strings.Join(exprs, ", ") + + " FROM sessions ORDER BY started_at" rows, err := db.QueryContext(ctx, query) if err != nil { return nil, fmt.Errorf("query sessions: %w", err) @@ -331,11 +386,27 @@ func readStateDBSessions(ctx context.Context, path string) ([]stateDBRow, error) parentID sql.NullString startedAt any messageCount sql.NullInt64 + usage [5]sql.NullInt64 // input, output, cache read, cache write, reasoning ) - if err := rows.Scan(&id, &model, &parentID, &startedAt, &messageCount); err != nil { + if err := rows.Scan( + &id, &model, &startedAt, &messageCount, &parentID, + &usage[0], &usage[1], &usage[2], &usage[3], &usage[4], + ); err != nil { return nil, fmt.Errorf("scan session: %w", err) } - row := stateDBRow{id: id, model: model, parentID: parentID, messageCount: messageCount} + row := stateDBRow{ + id: id, + model: model, + parentID: parentID, + messageCount: messageCount, + usage: stateDBUsage{ + Input: usage[0].Int64, + Output: usage[1].Int64, + CacheRead: usage[2].Int64, + CacheWrite: usage[3].Int64, + Reasoning: usage[4].Int64, + }, + } switch v := startedAt.(type) { case float64: row.startedAt = sql.NullFloat64{Float64: v, Valid: true} @@ -498,6 +569,14 @@ func projectStateDBSession(ctx context.Context, path string, row stateDBRow) (se } sess, turns, tools, state := projectMessagesWithDefaults(msgs, envStart, time.Time{}, envModel) + // The session row's counters are the authoritative usage signal: Hermes + // tracks tokens per session, and `messages.token_count` has been NULL + // since the counters landed. Fall back to the message-derived state when + // the row carries none. + if usage, ok := row.usage.toTokenUsage(); ok { + sess.Usage = usage + state = session.UsageStatePresent + } sess.ID = row.id if row.parentID.Valid { if parentID := strings.TrimSpace(row.parentID.String); parentID != "" { @@ -507,14 +586,6 @@ func projectStateDBSession(ctx context.Context, path string, row stateDBRow) (se return sess, turns, tools, state, msgs, nil } -func tableHasColumn(ctx context.Context, db *sql.DB, table, column string) (bool, error) { - cols, err := columnSet(ctx, db, table) - if err != nil { - return false, err - } - return cols[column], nil -} - // columnSet returns the set of column names declared on `table`. Used to // build dynamic SELECT lists that tolerate Hermes builds whose `messages` // schema lacks the newer reasoning/codex columns. diff --git a/internal/importers/hermes/state_db_project.go b/internal/importers/hermes/state_db_project.go index 74676247..174016c5 100644 --- a/internal/importers/hermes/state_db_project.go +++ b/internal/importers/hermes/state_db_project.go @@ -6,6 +6,33 @@ import ( "fmt" ) +// projectedLine wraps content that is not message-shaped so the projected +// JSONL stays self-describing. Message lines are emitted bare. +type projectedLine struct { + Type string `json:"type"` + Data stateDBUsage `json:"data"` +} + +// marshalProjectedUsageLine renders the session row's token counters as the +// projection's leading line, so the preserved raw carries the provenance for +// the usage prosa derives from it. Encoded with the same settings as +// marshalProjectedJSONL to keep the artifact's sha256 stable. +func marshalProjectedUsageLine(u stateDBUsage) (json.RawMessage, error) { + var buf bytes.Buffer + enc := json.NewEncoder(&buf) + enc.SetEscapeHTML(false) + if err := enc.Encode(projectedLine{Type: "session_usage", Data: u}); err != nil { + return nil, fmt.Errorf("marshal hermes session usage: %w", err) + } + out := buf.Bytes() + if n := len(out); n > 0 && out[n-1] == '\n' { + out = out[:n-1] + } + line := make(json.RawMessage, len(out)) + copy(line, out) + return line, nil +} + // marshalProjectedJSONL serializes msgs as one JSON object per line with no // trailing newline, matching the shape of per-session .jsonl files. // HTML escaping is disabled so `&`/`<`/`>` round-trip literally; the result diff --git a/pkg/session/types.go b/pkg/session/types.go index e4e29bcf..6c5595a6 100644 --- a/pkg/session/types.go +++ b/pkg/session/types.go @@ -142,7 +142,15 @@ type Session struct { // spawned subagents). Stored in the session_kinds table and pushed // to the server. The Codex goal objective is also unwrapped into // FirstPrompt instead of the raw scaffold. -const ProjectionVersion = 12 +// v13: Hermes usage read from state.db's `sessions` token counters +// instead of `messages.token_count`, which Hermes stopped +// populating. InputTokens is the cache-inclusive sum +// (input + cache_read + cache_write) per the canonical convention; +// reasoning_tokens stays out of OutputTokens because Hermes already +// counts it there. The projected JSONL gains a leading +// {"type":"session_usage","data":{…}} line carrying the counters, so +// raw_hash changes and sync_reconcile re-pushes Hermes sessions. +const ProjectionVersion = 13 // DefaultProfile is the profile name every agent has by default. const DefaultProfile = "default"