Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions docs/architecture/canonical-session.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ stays excluded.

## Projection version

`session.ProjectionVersion = 12`. The server's push handler compares
`session.ProjectionVersion = 13`. The server's push handler compares
`projection_version >= session.ProjectionVersion` before short-
circuiting, so bumping this constant forces existing sessions to be
re-projected on the next push from any client. Recent versions:
Expand All @@ -49,7 +49,9 @@ Hermes parent edges from `state.db` and transcript `parent_session_id`
fields; v10 projects Hermes `state.db` rows to per-session canonical
JSONL; v11 maps `ParentSessionID` onto the push wire and imports
Claude Code subagent transcripts under their real on-disk naming; v12
projects special-session `Kinds` into the `session_kinds` table.
projects special-session `Kinds` into the `session_kinds` table; v13 reads
Hermes usage from the `state.db` session-row token counters and projects
them as a leading `session_usage` line.

| Version | Brought |
|---|---|
Expand All @@ -65,6 +67,7 @@ projects special-session `Kinds` into the `session_kinds` table.
| 10 | Hermes `state.db` rows project to per-session canonical JSONL — the raw artifact for those sessions is a per-session `.jsonl` instead of the multi-session `.db` container; `raw_hash` / `raw_size` describe the projected JSONL. |
| 11 | parent edges reach the server — `sessionToProto` maps `Session.ParentSessionID` onto the wire (previously dropped at push, so the server never stored an edge). Claude Code subagent transcripts are imported under their real layouts: the walker accepts `agent-<hex>.jsonl` alongside the older `agent-<uuid>.jsonl`, both directly under `subagents/` (Agent tool) and nested at `subagents/workflows/wf_<id>/` (Workflow tool). The child session id is the filename stem because every record inside carries the parent's `sessionId`; the parent UUID is the directory above the innermost `subagents` component. |
| 12 | special-session classification projected — `Session.Kinds` carries any of `goal` (Codex `<codex_internal_context source="goal">` first user turn), `workflow` (Claude Code `Workflow` tool), `ralph-loop` (Claude Code `/ralph-loop:ralph-loop` command), and `orchestrator` (session that spawned subagents). Stored in the `session_kinds` table (migration `0010_session_kinds` local / `0013_session_kinds` server) and pushed on the wire as `Session.kinds`. `goal`/`workflow`/`ralph-loop` are derived per session by `internal/sessionkind`; `orchestrator` is edge-derived — the client reconciles it post-sweep via `store.RefreshOrchestratorKinds` and the server derives it from parent edges at push (`deriveOrchestratorKinds`). The Codex goal `<objective>` is also unwrapped into `FirstPrompt` instead of the raw scaffold. |
| 13 | Hermes usage read from `state.db`'s `sessions` token counters instead of `messages.token_count`, which Hermes no longer populates. `InputTokens` is the cache-inclusive sum; `reasoning_tokens` stays out of `OutputTokens`. The projected JSONL gains a leading `{"type":"session_usage","data":{…}}` line, so `raw_hash` changes and Hermes sessions re-push. |

## Import eligibility

Expand Down Expand Up @@ -310,7 +313,8 @@ and a `sessions.json` index). The full reference is `docs/sources/hermes.md`.
| `FirstPrompt` | first `messages.role=="user"` row (SQLite) or first user line/message (JSONL/JSON) with non-empty `content` (text or first text item of an array); whitespace-collapsed + truncated to 200 runes |
| `Model` | `sessions.model` (SQLite); `model` (JSON snapshot); first assistant-side `model` encountered (JSONL) |
| `ParentSessionID` | `sessions.parent_session_id` (SQLite); top-level `parent_session_id` (JSON snapshot); first per-message `parent_session_id` encountered (JSONL / snapshot messages) |
| `RawPath` / `RawHash` / `RawSize` | per-session JSONL at `$PROSA_HOME/raw/hermes/<YYYY>/<MM>/<session-id>.jsonl`; `RawHash`/`RawSize` describe the per-session artifact. For `.jsonl` / `session_*.json` shapes the source bytes are preserved verbatim (extension follows the source); for `state.db`, each `sessions` row is projected to its own JSONL (`messages` rows + hidden reasoning/codex/tool-call columns, one per line). The multi-session `.db` is **not** copied — see issue #235 |
| `Usage` | `state.db` rows: the `sessions` token counters. `InputTokens` is `input_tokens + cache_read_tokens + cache_write_tokens` because Hermes stores uncached input alone; `CacheReadTokens`/`CachedTokens` from `cache_read_tokens`, `CacheCreationTokens` from `cache_write_tokens`, `OutputTokens` from `output_tokens`. `reasoning_tokens` is already inside `output_tokens` and is provenance only. All-zero counters classify Unknown, not ExplicitZero. `.jsonl` / `session_*.json` shapes carry no counters, so a session that defers to a fuller sibling transcript has no usage — see the dual-source gap in `docs/sources/hermes.md` |
| `RawPath` / `RawHash` / `RawSize` | per-session JSONL at `$PROSA_HOME/raw/hermes/<YYYY>/<MM>/<session-id>.jsonl`; `RawHash`/`RawSize` describe the per-session artifact. For `.jsonl` / `session_*.json` shapes the source bytes are preserved verbatim (extension follows the source); for `state.db`, each `sessions` row is projected to its own JSONL (a leading `session_usage` line when the row carries token counters, then `messages` rows + hidden reasoning/codex/tool-call columns, one per line). The multi-session `.db` is **not** copied — see issue #235 |

### `session.Turn`

Expand Down
74 changes: 60 additions & 14 deletions docs/sources/hermes.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,18 +57,24 @@ time — same idiom as Cursor and Gemini.

```sql
CREATE TABLE sessions (
id TEXT PRIMARY KEY,
source TEXT NOT NULL,
model TEXT,
model_config TEXT,
system_prompt TEXT,
parent_session_id TEXT,
started_at REAL NOT NULL,
ended_at REAL,
end_reason TEXT,
message_count INTEGER,
tool_call_count INTEGER,
title TEXT
id TEXT PRIMARY KEY,
source TEXT NOT NULL,
model TEXT,
model_config TEXT,
system_prompt TEXT,
parent_session_id TEXT,
started_at REAL NOT NULL,
ended_at REAL,
end_reason TEXT,
message_count INTEGER,
tool_call_count INTEGER,
input_tokens INTEGER DEFAULT 0,
output_tokens INTEGER DEFAULT 0,
cache_read_tokens INTEGER DEFAULT 0,
cache_write_tokens INTEGER DEFAULT 0,
reasoning_tokens INTEGER DEFAULT 0,
estimated_cost_usd REAL,
title TEXT
);

CREATE TABLE messages (
Expand Down Expand Up @@ -98,6 +104,37 @@ importer normalizes both to UTC `time.Time` after parse.
hold plain text or JSON-encoded values; the importer treats them as
opaque strings unless a column is explicitly parsed (`tool_calls`).

Hermes counts tokens **per session**, on the `sessions` row. The importer
reads those five counters and ignores `messages.token_count`, which Hermes
stopped populating when the counters landed. The mapping onto prosa's
canonical aggregate:

| prosa `TokenUsage` | Hermes column |
| --- | --- |
| `InputTokens` | `input_tokens + cache_read_tokens + cache_write_tokens` |
| `CacheReadTokens`, `CachedTokens` | `cache_read_tokens` |
| `CacheCreationTokens` | `cache_write_tokens` |
| `OutputTokens` | `output_tokens` |

Two rules the mapping depends on. `input_tokens` holds **uncached** input
only, so prosa's cache-inclusive `InputTokens` is the sum of all three prompt
columns — Hermes derives its own `prompt_tokens` the same way. And
`reasoning_tokens` is already counted inside `output_tokens`, which Hermes's
`total_tokens` confirms by not adding it; prosa keeps it as provenance and
never folds it into `OutputTokens`.

A row whose counters are all zero classifies Unknown, not ExplicitZero, so it
is still imported. This mirrors Hermes's own `has_usage` flag, which only
records usage once a counter is non-zero.

`estimated_cost_usd` is read for reference only. Hermes prices
subscription-covered routes at zero, so prosa estimates cost from its own rate
table instead, the same treatment Claude Code and Codex sessions get.

A Hermes build predating the counters still imports: the sessions query
projects missing columns as `NULL`, and usage falls back to
`messages.token_count`.

## Transcript files (`.jsonl`)

Top-level `<session-id>.jsonl` is one message object per line. The
Expand Down Expand Up @@ -323,12 +360,18 @@ What `session.Turn` and `session.ToolUsage` surface for Hermes today:
- every per-message hidden column — `messages.reasoning`,
`reasoning_content`, `reasoning_details`, `codex_reasoning_items`,
`codex_message_items`, `tool_call_id`, `tool_name`, `finish_reason`,
`token_count`;
`token_count` (no longer the usage source — see the session-level
counters above, which the projection carries on its own line);
- the full body of every `tool_calls` payload and `tool`-role
`content` blob, beyond what the `ToolUsage` aggregate counts;
- for sibling `<id>.jsonl` / `session_<id>.json` shapes, the source
bytes verbatim (including the snapshot envelope's `system_prompt` /
`platform` / `last_updated` when present).
- **Projected usage line**: a `state.db` row carrying token counters leads
its projected JSONL with
`{"type":"session_usage","data":{"input_tokens":…,"output_tokens":…,"cache_read_tokens":…,"cache_write_tokens":…,"reasoning_tokens":…}}`,
so the preserved raw explains the usage prosa derives from it. Rows with no
counters project messages only.
- **Not preserved in raw** for `state.db`-sourced sessions:
- session-level columns without a per-message equivalent
(`sessions.system_prompt`, `model_config`, `end_reason`, `title`,
Expand All @@ -341,4 +384,7 @@ What `session.Turn` and `session.ToolUsage` surface for Hermes today:
other had more messages), only the winning surface lands as raw —
there is no separate copy of the dropped side. A future cut that
wants to merge them must read both sources at projection time; the
importer at this cut does not attempt the merge.
importer at this cut does not attempt the merge. The visible cost of
that gap is usage: a session that defers to a sibling transcript gets
no token counters, because they live only on the `state.db` row and a
transcript's raw is a verbatim copy that cannot carry them.
11 changes: 11 additions & 0 deletions internal/importers/hermes/importer.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ package hermes

import (
"context"
"encoding/json"
"fmt"
"path/filepath"
"time"
Expand Down Expand Up @@ -184,6 +185,16 @@ func (i *Importer) importStateDB(ctx context.Context, path string, sink importer
if err != nil {
return importer.ImportResult{}, fmt.Errorf("project session %s: %w", row.id, err)
}
// Hermes counts tokens on the session row, not per message, so the
// counters lead the projection: without them the raw could not
// explain the usage prosa derives from it.
if _, ok := row.usage.toTokenUsage(); ok {
usageLine, err := marshalProjectedUsageLine(row.usage)
if err != nil {
return importer.ImportResult{}, fmt.Errorf("project session %s: %w", row.id, err)
}
lines = append([]json.RawMessage{usageLine}, lines...)
}
rawPath, rawHash, rawSize, err := importerutil.PreserveProjectedJSONL(Name, row.id, sess.StartedAt, lines)
if err != nil {
return importer.ImportResult{}, fmt.Errorf("preserve projected raw %s: %w", row.id, err)
Expand Down
Loading
Loading