diff --git a/docs/architecture.md b/docs/architecture.md index 7dd9e4b..fb6f782 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -47,8 +47,11 @@ replaced, or truncated file fails with a retryable error. - Claude reconstructs the active UUID ancestry selected by `last-prompt`, validates compaction back-edges, and excludes inactive branches/meta prompts. -- Codex replays canonical legacy `response_item` history, deduplicates UI - projections, and rejects paginated/history-base lineage. +- Codex replays canonical legacy `response_item` history. For a complete root + paginated rollout it validates contiguous ordinals and takes visible turns + only from `event_msg.item_completed` TurnItems, never from contextual + provider messages. External `history_base` and subagent projections remain + fail-closed. - Pi follows the v3 `id`/`parentId` active tree and rejects unsupported schema versions. - OMP follows its v3 active tree after validating the fixed 256-byte title diff --git a/docs/exploration-log.md b/docs/exploration-log.md index b3b15a8..624590d 100644 --- a/docs/exploration-log.md +++ b/docs/exploration-log.md @@ -533,8 +533,8 @@ in [Mistral Vibe session format](vibe-format.md). - Repeat authenticated semantic recall when a supported target/provider version changes. - Add native fixtures for remote-URL images, branching, and schema drift when sanitized examples can be generated safely. -- Implement Codex paginated/history-base lineage only after ordinal, contextual - user, compaction, rollback, and inter-agent semantics are independently gated. +- Implement Codex `history_base` and paginated subagent projection only after + external-prefix, rollback, and inter-agent semantics are independently gated. - Re-run the pinned integration suite for every supported agent version/schema combination. @@ -717,3 +717,35 @@ indexes bounded native titles/IDs without bodies, and represents selection as validates its install bundle, for 324 ordered routes. Detailed contracts are in [Hermes](hermes-format.md), [MastraCode](mastracode-format.md), and [Devin](devin-format.md). + +## 2026-09-11: Codex 0.153 root paginated history + +PR #4 was reviewed against the official Codex `rust-v0.153.4` source rather +than its original synthetic assumption. `rollout/src/policy.rs` shows that +paginated history persists canonical `TurnItem`s as +`event_msg.item_completed`; legacy user/assistant UI events are deliberately +not persisted in that mode. `rollout/src/ordinal.rs` requires paginated +records to carry a monotonic ordinal, and `protocol/src/protocol.rs` defines +`history_base` as an external exclusive prefix and +`subagent_history_start_ordinal` as the boundary between inherited context and +the subagent's own projection. + +A content-free audit then examined 172 recent native paginated rollouts without +`history_base`: nine roots and 163 subagent projections. It inspected only +record/item types, structural fields, counts, and hashed content equality; no +message, tool, media, ID, or path value was printed or committed. Across the +nine roots, 75 canonical user items matched provider user messages and all 242 +provider assistant messages matched canonical assistant items. The provider +stream also contained 21 additional user-shaped messages and 53 developer +messages that had no canonical completed turn. Replaying every +`response_item.message`, as the initial PR did, would therefore turn internal +context into visible user history. + +The corrected reader makes completed `UserMessage`/`AgentMessage` TurnItems the +only paginated source of conversation turns, retains non-message response items +for portable tool/reasoning data, and records ignored provider messages as +opaque losses. It accepts only self-contained roots with integer ordinals +contiguous from zero. `history_base`, subagent projections, unknown modes, and +damaged ordinal streams fail closed. The sanitized 0.153.4 fixture mirrors the +official nested item shape and includes an unmatched contextual provider +message as a regression sentinel. diff --git a/docs/format-compatibility.md b/docs/format-compatibility.md index 8ee8c0a..25e070d 100644 --- a/docs/format-compatibility.md +++ b/docs/format-compatibility.md @@ -275,13 +275,24 @@ ordered `response_item` envelopes. Text messages use `type: "message"` with `function_call_output`; call arguments are a JSON-encoded string, and both records share `call_id`. -`response_item` is the canonical, model-visible history. `event_msg` records -drive list preview and UI display. The writer emits both for text messages so -the text is visible to the resumed model and to the interface. The reader -deduplicates UI messages against response-item messages. It uses UI events as a -legacy fallback when a rollout has no canonical messages. In a mixed partial -rollout, exact normalized duplicates are removed; unmatched UI projections are -retained as marked messages and reported in the manifest. +For legacy history, `response_item` is the canonical, model-visible history and +`event_msg` records drive list preview and UI display. The writer emits both +for text messages so the text is visible to the resumed model and to the +interface. The reader deduplicates UI messages against response-item messages. +It uses UI events as a legacy fallback when a rollout has no canonical +messages. In a mixed partial legacy rollout, exact normalized duplicates are +removed; unmatched UI projections are retained as marked messages and +reported in the manifest. + +Codex 0.147+ root paginated rollouts use a different authority: completed +`UserMessage` and `AgentMessage` TurnItems inside +`event_msg.item_completed`. The reader requires an integer ordinal on every +record, contiguous from zero. It ignores provider `response_item` messages as +conversation because those records can include synthetic environment and +developer context; non-message response items still supply portable tool and +reasoning data. Direct image/audio user inputs are retained when represented +by the canonical TurnItem. Local media paths and other TurnItem blocks remain +explicit opaque losses. Current legacy rollouts can contain `compacted.replacement_history`. Codex installs that array as the effective history at the checkpoint and replays only @@ -294,21 +305,22 @@ post-compaction context transfer. The paired `event_msg.context_compacted` UI notification is deduplicated against the checkpoint. Other observed envelopes include `compacted`, `turn_context`, `world_state`, -reasoning response items, inter-agent communication, and newer paginated or -fork-related state. Those records are not all portable conversation history. +reasoning response items, inter-agent communication, and fork-related state. +Those records are not all portable conversation history. ## Route support Every ordered pair among the eighteen formats is implemented, for 324 routes: -- full portable adapters: Claude, Codex legacy, Pi, OMP, OpenCode, Copilot, +- full portable adapters: Claude, Codex legacy and root paginated, Pi, OMP, OpenCode, Copilot, Antigravity, Vibe, Muse, Qwen, Kimi, Grok, Kilo, OpenHands, Hermes, MastraCode, and Devin; - experimental text-only adapter: Cursor. Same-format routes are portable rewrites into new sessions, not byte copies. -Codex paginated/history-base sources remain fail-closed. Cursor is experimental, -build-pinned, and deliberately transfers only ordered user/assistant text. The +Codex `history_base` lineage and paginated subagent projections remain +fail-closed. Cursor is experimental, build-pinned, and deliberately transfers +only ordered user/assistant text. The detailed table below explains the original Claude/Codex pair; target-specific behavior is documented in [Additional native formats](additional-target-formats.md) and [Muse/Qwen/Kimi](muse-qwen-kimi-formats.md). Grok, Kilo, and OpenHands @@ -348,7 +360,8 @@ Legend: | Inactive Claude branches | **Unsupported** | N/A | They become opaque events and are counted as dropped; no forks are created. | | Claude sidechains/subagents | **Unsupported** | N/A | The catalog indexes nested sidechains as unsupported, but direct lookup/conversion does not import them; transfer the parent session. | | Codex legacy linear history | N/A | **Supported** | Ordered response items become one linear Claude UUID graph. | -| Codex paginated history/forks | N/A | **Unsupported** | Non-legacy `history_mode` and `history_base` are rejected rather than risking an incomplete import. Replacement-history compaction uses the expanded-transcript policy above. | +| Codex root paginated history | N/A | **Supported** | Contiguous canonical completed TurnItems supply user/assistant turns; provider-context messages are not replayed. Replacement-history compaction uses the expanded-transcript policy above. | +| Codex paginated forks/subagents | N/A | **Unsupported** | `history_base` and `subagent_history_start_ordinal` require external or projected history and are rejected rather than silently truncating or duplicating a conversation. | | Codex UI-only messages | N/A | **Lossy fallback** | Used as the conversation when no response-item messages exist. In a mixed partial file, exact normalized duplicates are removed and unmatched projections are retained with `message:ui_only_projection`; fuzzy matching is never used. | | Turn context, policies, world state, snapshots | **Unsupported** | **Unsupported** | Codex `turn_context` is counted as context; `world_state` and `security_risk_score` become counted opaque events. Shell snapshots, approvals, external credential stores, MCP state, memories, goals, and configuration are outside transcript conversion. | | Unknown source records/blocks | **Unsupported** | **Unsupported** | They become content-free opaque/sentinel events where recognized and are counted at write time, including unknown nested tool-result blocks. | diff --git a/docs/session-catalog.md b/docs/session-catalog.md index 525667b..244bf8b 100644 --- a/docs/session-catalog.md +++ b/docs/session-catalog.md @@ -177,9 +177,11 @@ This keeps refresh work proportional to the small inventory table, not the total transcript corpus. Consequently, archived sessions, duplicate UUIDs, nested sidechains/subagents, -malformed files, and absent Copilot/Cursor native stores remain discoverable. Claude -sidechains and Codex paginated/history-base sessions are listed as -`unsupported`; listing them does not make them convertible. +malformed files, and absent Copilot/Cursor native stores remain discoverable. +Complete root Codex paginated rollouts are candidates. Claude sidechains, +Codex `history_base` lineage, and paginated subagent projections are listed as +`unsupported`; listing them does not make them convertible. A paginated root +with missing or non-contiguous ordinals is `corrupt`. ## Quick start @@ -320,7 +322,7 @@ registered roots. | --- | --- | | `candidate` | Fast structural metadata scan passed; full conversion has not been requested. OpenCode and Kilo rows remain candidates until their one-session official export is parsed. | | `validated` | The exact stat identity was fully parsed, dry-converted, and target-validated during `refresh --validate`. | -| `unsupported` | The file is a recognized session type intentionally rejected by conversion, such as a Claude sidechain or Codex paginated/history-base rollout. | +| `unsupported` | The file is a recognized session type intentionally rejected by conversion, such as a Claude sidechain, Codex `history_base` lineage, or a Codex paginated subagent projection. | | `corrupt` | JSONL, SQLite/protobuf, native structure, or explicit conversion validation failed. | | `oversized` | The source exceeds the migrator's bounded input limits. | | `busy` | The source changed while it was being scanned; retry after the native CLI finishes appending. | diff --git a/docs/specification.md b/docs/specification.md index 5f893df..72303c5 100644 --- a/docs/specification.md +++ b/docs/specification.md @@ -30,7 +30,8 @@ creates a new independent target session; it does not move/delete the source, clone runtime state byte-for-byte, synchronize future turns, or re-execute historical tools. -Codex paginated/history-base lineage and Claude sidechain import remain +Complete root Codex paginated rollouts are transferable. Codex `history_base` +lineage, paginated subagent projections, and Claude sidechain import remain non-transferable. They are discoverable in the catalog and fail closed. ## Functional requirements diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 4c585d8..a3e3bdc 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -261,12 +261,14 @@ separately authenticated Devin installation; session-migrate never copies credentials. See the [Hermes](hermes-format.md), [MastraCode](mastracode-format.md), and [Devin](devin-format.md) format notes. -## Codex paginated or history-base source - -These lineage modes are recognized but unsupported. `--format codex` cannot -bypass the guard. The safe root-paginated subset still needs ordinal, -contextual-user, compaction/rollback/inter-agent, and lineage semantics before -it can be enabled. +## Codex paginated source is rejected + +A complete root paginated rollout is supported when every record has an integer +ordinal contiguous from zero. Missing/gapped ordinals are treated as corrupt. +`history_base` and `subagent_history_start_ordinal` mean the file depends on an +external or projected prefix and remain unsupported; `--format codex` cannot +bypass either guard. Resume or export the root session instead of flattening +the dependent file by hand. ## Claude sidechain/subagent diff --git a/docs/validation-report.md b/docs/validation-report.md index abb4439..9b6d36f 100644 --- a/docs/validation-report.md +++ b/docs/validation-report.md @@ -1292,11 +1292,33 @@ is committed. Default CI performs the frozen parser and 324 conversion cases without network access or model spend; live gates remain explicit release commands. +### 2026-09-11 Codex paginated-source review + +The root paginated reader was checked against the official Codex 0.153.4 +protocol, rollout persistence policy, and ordinal implementation. A +content-free structural audit over nine recent root rollouts found 81 canonical +user items, 244 canonical assistant items, 21 provider-only user-shaped +messages, and 53 provider developer messages. Only counts and content hashes +were compared; private values were never printed or added to fixtures. + +The regression fixture now uses real `event_msg.item_completed` nesting, +canonical `UserMessage`/`AgentMessage` shapes, integer ordinals, and the current +token-usage envelope. Its provider stream deliberately contains an unmatched +context message. Tests prove that canonical text and image history matches the +legacy portable projection, the context sentinel is never replayed, damaged +ordinals fail closed, and external/history-projection metadata remains +unsupported. Catalog tests independently pin candidate, corrupt, and +unsupported classification. All nine audited native roots were parsed and +converted through each of the eighteen target byte validators: 162/162 local, +content-safe conversions passed without writing artifacts or invoking a model. + ## Known boundaries -- Codex paginated history and `history_base` lineage remain fail-closed until - their effective-history and fork semantics can be reproduced and native - tested without relying on derived SQLite state. +- Complete root Codex paginated history is supported from canonical completed + TurnItems after strict ordinal validation. `history_base` lineage and + paginated subagent projections remain fail-closed until their external or + projected prefix semantics can be reproduced without relying on derived + SQLite state. - Provider-encrypted Codex replacement-history state cannot be translated to Claude. The migrator retains visible expanded history and reports the semantic difference. diff --git a/scripts/validate-additional-target-corpus.py b/scripts/validate-additional-target-corpus.py index f3cc006..098951d 100644 --- a/scripts/validate-additional-target-corpus.py +++ b/scripts/validate-additional-target-corpus.py @@ -205,6 +205,12 @@ def expected_source_rejection(source_format: AgentFormat, exc: SessionMigrateErr return "codex_history_mode" if "history_base lineage is not supported" in message: return "codex_history_base" + if "subagent history projection is not supported" in message: + return "codex_subagent_history" + if "conflicting history modes" in message: + return "codex_history_mode_conflict" + if "paginated" in message and "ordinal" in message: + return "codex_paginated_ordinals" return None diff --git a/scripts/validate-antigravity-native.py b/scripts/validate-antigravity-native.py index 51507e9..4fbd20a 100644 --- a/scripts/validate-antigravity-native.py +++ b/scripts/validate-antigravity-native.py @@ -199,6 +199,12 @@ def expected_rejection(source_format: AgentFormat, exc: SessionMigrateError) -> return "codex_history_mode" if "history_base lineage is not supported" in message: return "codex_history_base" + if "subagent history projection is not supported" in message: + return "codex_subagent_history" + if "conflicting history modes" in message: + return "codex_history_mode_conflict" + if "paginated" in message and "ordinal" in message: + return "codex_paginated_ordinals" return None diff --git a/scripts/validate-copilot-native.py b/scripts/validate-copilot-native.py index 3e14477..9b01974 100644 --- a/scripts/validate-copilot-native.py +++ b/scripts/validate-copilot-native.py @@ -226,6 +226,12 @@ def _expected_rejection(source_format: AgentFormat, exc: SessionMigrateError) -> return "codex_history_mode" if "history_base lineage is not supported" in message: return "codex_history_base" + if "subagent history projection is not supported" in message: + return "codex_subagent_history" + if "conflicting history modes" in message: + return "codex_history_mode_conflict" + if "paginated" in message and "ordinal" in message: + return "codex_paginated_ordinals" return None diff --git a/scripts/validate-core-target-native.py b/scripts/validate-core-target-native.py index 83fa29d..4e78711 100755 --- a/scripts/validate-core-target-native.py +++ b/scripts/validate-core-target-native.py @@ -172,6 +172,12 @@ def expected_rejection(source_format: AgentFormat, exc: SessionMigrateError) -> return "codex_history_mode" if "history_base lineage is not supported" in message: return "codex_history_base" + if "subagent history projection is not supported" in message: + return "codex_subagent_history" + if "conflicting history modes" in message: + return "codex_history_mode_conflict" + if "paginated" in message and "ordinal" in message: + return "codex_paginated_ordinals" return None diff --git a/scripts/validate-muse-qwen-kimi-corpus.py b/scripts/validate-muse-qwen-kimi-corpus.py index 2bfc614..4bda86c 100755 --- a/scripts/validate-muse-qwen-kimi-corpus.py +++ b/scripts/validate-muse-qwen-kimi-corpus.py @@ -178,6 +178,12 @@ def expected_rejection(source_format: AgentFormat, exc: SessionMigrateError) -> return "codex_history_mode" if "history_base lineage is not supported" in message: return "codex_history_base" + if "subagent history projection is not supported" in message: + return "codex_subagent_history" + if "conflicting history modes" in message: + return "codex_history_mode_conflict" + if "paginated" in message and "ordinal" in message: + return "codex_paginated_ordinals" return None diff --git a/src/session_migrate/catalog.py b/src/session_migrate/catalog.py index 2de1aae..2c34fe1 100644 --- a/src/session_migrate/catalog.py +++ b/src/session_migrate/catalog.py @@ -25,6 +25,7 @@ from session_migrate.errors import JsonlError, SessionMigrateError from session_migrate.formats import ( antigravity, + codex, devin, hermes, kimi, @@ -2077,6 +2078,10 @@ def _scan_file(path: Path, agent_format: AgentFormat, root: Path) -> _Scan: cli_version = None history_mode = None history_base = False + codex_subagent_history = False + codex_ordinals_complete = True + codex_history_mode_conflict = False + codex_selected_history_mode = None sidechain = False records = 0 has_conversation = False @@ -2114,6 +2119,12 @@ def _scan_file(path: Path, agent_format: AgentFormat, root: Path) -> _Scan: if title: labels.append(_Label("ai_title", title, record.index, 90)) elif agent_format == AgentFormat.CODEX: + ordinal = value.get("ordinal") + codex_ordinals_complete = codex_ordinals_complete and ( + isinstance(ordinal, int) + and not isinstance(ordinal, bool) + and ordinal == records - 1 + ) if record_type in {"user", "assistant"} and isinstance(value.get("message"), dict): wrong_format = True payload = value.get("payload") @@ -2131,8 +2142,21 @@ def _scan_file(path: Path, agent_format: AgentFormat, root: Path) -> _Scan: or _string(value.get("timestamp")) ) cli_version = cli_version or _string(payload.get("cli_version")) - history_mode = history_mode or _string(payload.get("history_mode")) + raw_history_mode = _string(payload.get("history_mode")) + observed_history_mode = raw_history_mode or "legacy" + codex_history_mode_conflict = codex_history_mode_conflict or ( + codex_selected_history_mode is not None + and observed_history_mode != codex_selected_history_mode + ) + codex_selected_history_mode = ( + codex_selected_history_mode or observed_history_mode + ) + history_mode = history_mode or raw_history_mode history_base = history_base or payload.get("history_base") is not None + codex_subagent_history = ( + codex_subagent_history + or payload.get("subagent_history_start_ordinal") is not None + ) elif record_type == "response_item": has_conversation = has_conversation or payload.get("type") in { "message", @@ -2141,13 +2165,20 @@ def _scan_file(path: Path, agent_format: AgentFormat, root: Path) -> _Scan: "function_call_output", "custom_tool_call_output", } - elif record_type == "event_msg" and payload.get("type") == "thread_name_updated": - title = _bounded( - _string(payload.get("name")) or _string(payload.get("thread_name")), - LABEL_LIMIT, - ) - if title: - labels.append(_Label("thread_name", title, record.index, 110)) + elif record_type == "event_msg": + if payload.get("type") == "item_completed": + item = payload.get("item") + has_conversation = has_conversation or ( + isinstance(item, dict) + and item.get("type") in {"UserMessage", "AgentMessage"} + ) + elif payload.get("type") == "thread_name_updated": + title = _bounded( + _string(payload.get("name")) or _string(payload.get("thread_name")), + LABEL_LIMIT, + ) + if title: + labels.append(_Label("thread_name", title, record.index, 110)) elif agent_format in {AgentFormat.PI, AgentFormat.OMP}: if record_type in {"user", "assistant", "session_meta", "response_item"}: wrong_format = True @@ -2264,10 +2295,21 @@ def _scan_file(path: Path, agent_format: AgentFormat, root: Path) -> _Scan: elif agent_format == AgentFormat.CODEX: if not has_session_meta: status, reason = "corrupt", "missing_session_meta" - elif history_mode and history_mode != "legacy": - status, reason = "unsupported", "codex_history_mode" elif history_base: status, reason = "unsupported", "codex_history_base" + elif codex_subagent_history: + status, reason = "unsupported", "codex_subagent_history" + elif codex_history_mode_conflict: + status, reason = "corrupt", "codex_history_mode_conflict" + elif ( + codex_selected_history_mode + and codex_selected_history_mode not in codex.SUPPORTED_HISTORY_MODES + ): + status, reason = "unsupported", "codex_history_mode" + elif codex_selected_history_mode == "paginated" and first_record_type != "session_meta": + status, reason = "corrupt", "missing_session_meta" + elif codex_selected_history_mode == "paginated" and not codex_ordinals_complete: + status, reason = "corrupt", "codex_paginated_ordinals" elif not has_conversation: status, reason = "corrupt", "no_conversation_records" elif agent_format == AgentFormat.PI: diff --git a/src/session_migrate/formats/codex.py b/src/session_migrate/formats/codex.py index b7f61db..e691aae 100644 --- a/src/session_migrate/formats/codex.py +++ b/src/session_migrate/formats/codex.py @@ -17,9 +17,20 @@ PINNED_CODEX_VERSION = "0.144.4" +# Codex 0.147+ paginated rollouts persist canonical TurnItems in +# event_msg/item_completed records. Provider response_item messages are not an +# equivalent transcript: they also contain synthetic environment and developer +# context. Paginated parsing therefore takes user/assistant messages only from +# the canonical completed items while retaining non-message response items for +# tool and reasoning data. +SUPPORTED_HISTORY_MODES = frozenset({"legacy", "paginated"}) + def parse(path: Path) -> Session: records = list(iter_jsonl(path)) + history_mode = _history_mode(records) + if history_mode == "paginated": + _validate_paginated_root(records) events: list[Event] = [] fallback_events: list[Event] = [] context_compacted_events: list[Event] = [] @@ -32,7 +43,6 @@ def parse(path: Path) -> Session: response_message_count = 0 response_messages: Counter[tuple[Role | None, str]] = Counter() title = None - canonical_meta_seen = False for record in records: value = record.value @@ -41,15 +51,6 @@ def parse(path: Path) -> Session: payload = object_value(value.get("payload")) provenance = Provenance(record.index, record_type) if record_type == "session_meta": - if not canonical_meta_seen: - history_mode = string(payload.get("history_mode")) - if history_mode and history_mode != "legacy": - raise SessionMigrateError( - f"Codex history mode {history_mode!r} is not supported; expected legacy" - ) - if payload.get("history_base") is not None: - raise SessionMigrateError("Codex history_base lineage is not supported") - canonical_meta_seen = True session_id = ( session_id or string(payload.get("id")) or string(payload.get("session_id")) ) @@ -60,7 +61,23 @@ def parse(path: Path) -> Session: model_provider = model_provider or string(payload.get("model_provider")) continue if record_type == "response_item": - parsed = _response_item_events(payload, timestamp, provenance) + if history_mode == "paginated" and string(payload.get("type")) in { + "message", + "agent_message", + }: + parsed = [ + Event( + kind=EventKind.OPAQUE, + timestamp=timestamp, + payload={ + "source_item_type": string(payload.get("type")) or "", + "reason": "paginated_provider_message", + }, + provenance=provenance, + ) + ] + else: + parsed = _response_item_events(payload, timestamp, provenance) events.extend(parsed) response_message_count += sum( event.kind == EventKind.MESSAGE and event.role in {Role.USER, Role.ASSISTANT} @@ -75,7 +92,9 @@ def parse(path: Path) -> Session: ) elif record_type == "event_msg": event_type = string(payload.get("type")) - if event_type == "user_message": + if event_type == "item_completed" and history_mode == "paginated": + events.extend(_paginated_completed_item_events(payload, timestamp, provenance)) + elif event_type == "user_message" and history_mode == "legacy": fallback_events.append( Event( kind=EventKind.MESSAGE, @@ -85,7 +104,7 @@ def parse(path: Path) -> Session: provenance=provenance, ) ) - elif event_type == "agent_message": + elif event_type == "agent_message" and history_mode == "legacy": fallback_events.append( Event( kind=EventKind.MESSAGE, @@ -167,7 +186,9 @@ def parse(path: Path) -> Session: ) ) - if response_message_count == 0: + if history_mode == "paginated": + events.sort(key=lambda event: event.provenance.record_index) + elif response_message_count == 0: events.extend(event for event in fallback_events if event.text) events.sort(key=lambda event: event.provenance.record_index) else: @@ -206,6 +227,191 @@ def parse(path: Path) -> Session: ) +def _history_mode(records: list[Any]) -> str: + selected_mode = "legacy" + metadata_seen = False + for record in records: + if string(record.value.get("type")) != "session_meta": + continue + payload = object_value(record.value.get("payload")) + history_mode = string(payload.get("history_mode")) or "legacy" + if history_mode not in SUPPORTED_HISTORY_MODES: + raise SessionMigrateError( + f"Codex history mode {history_mode!r} is not supported; " + f"expected one of {', '.join(sorted(SUPPORTED_HISTORY_MODES))}" + ) + if payload.get("history_base") is not None: + raise SessionMigrateError("Codex history_base lineage is not supported") + if payload.get("subagent_history_start_ordinal") is not None: + raise SessionMigrateError( + "Codex paginated subagent history projection is not supported" + ) + if metadata_seen and history_mode != selected_mode: + raise SessionMigrateError("Codex session metadata has conflicting history modes") + selected_mode = history_mode + metadata_seen = True + if selected_mode == "paginated" and ( + not records or string(records[0].value.get("type")) != "session_meta" + ): + raise SessionMigrateError("Codex paginated history must start with session metadata") + return selected_mode + + +def _validate_paginated_root(records: list[Any]) -> None: + """Fail closed if a root rollout is not a complete canonical ordinal stream.""" + + for expected, record in enumerate(records): + ordinal = record.value.get("ordinal") + if isinstance(ordinal, bool) or not isinstance(ordinal, int): + raise SessionMigrateError( + f"Codex paginated record {record.index} is missing an integer ordinal" + ) + if ordinal != expected: + raise SessionMigrateError( + "Codex paginated ordinals must be contiguous from zero; " + f"record {record.index} has ordinal {ordinal}, expected {expected}" + ) + + +def _paginated_completed_item_events( + payload: dict[str, Any], + timestamp: str | None, + provenance: Provenance, +) -> list[Event]: + item = object_value(payload.get("item")) + item_type = string(item.get("type")) + if item_type == "UserMessage": + return _paginated_user_message_events(item, timestamp, provenance) + if item_type == "AgentMessage": + content = item.get("content") + if not isinstance(content, list): + return [_opaque_completed_item(item_type, timestamp, provenance, "invalid_content")] + result: list[Event] = [] + for block_index, block in enumerate(content): + block_provenance = Provenance( + provenance.record_index, + provenance.record_type, + block_index=block_index, + ) + if not isinstance(block, dict): + result.append( + _opaque_completed_item(item_type, timestamp, block_provenance, "invalid_block") + ) + continue + block_type = string(block.get("type")) + text_value = string(block.get("text")) + if block_type in {"Text", "text"} and text_value: + result.append( + Event( + kind=EventKind.MESSAGE, + role=Role.ASSISTANT, + text=text_value, + timestamp=timestamp, + provenance=block_provenance, + ) + ) + else: + result.append( + _opaque_completed_item( + item_type, + timestamp, + block_provenance, + f"unsupported_block:{block_type or ''}", + ) + ) + return result + return [_opaque_completed_item(item_type, timestamp, provenance)] + + +def _paginated_user_message_events( + item: dict[str, Any], + timestamp: str | None, + provenance: Provenance, +) -> list[Event]: + content = item.get("content") + if not isinstance(content, list): + return [_opaque_completed_item("UserMessage", timestamp, provenance, "invalid_content")] + result: list[Event] = [] + for block_index, block in enumerate(content): + block_provenance = Provenance( + provenance.record_index, + provenance.record_type, + block_index=block_index, + ) + if not isinstance(block, dict): + result.append( + _opaque_completed_item("UserMessage", timestamp, block_provenance, "invalid_block") + ) + continue + block_type = string(block.get("type")) + if block_type == "text": + text_value = string(block.get("text")) + if text_value: + result.append( + Event( + kind=EventKind.MESSAGE, + role=Role.USER, + text=text_value, + timestamp=timestamp, + provenance=block_provenance, + ) + ) + elif block_type == "image": + result.append( + Event( + kind=EventKind.CONTEXT, + role=Role.USER, + timestamp=timestamp, + payload={ + "block_type": "image", + "image_url": string(block.get("image_url")), + }, + provenance=block_provenance, + ) + ) + elif block_type == "audio": + result.append( + Event( + kind=EventKind.CONTEXT, + role=Role.USER, + timestamp=timestamp, + payload={ + "block_type": "audio", + "audio_url": string(block.get("audio_url")), + }, + provenance=block_provenance, + ) + ) + else: + result.append( + _opaque_completed_item( + "UserMessage", + timestamp, + block_provenance, + f"unsupported_block:{block_type or ''}", + ) + ) + return result + + +def _opaque_completed_item( + item_type: str | None, + timestamp: str | None, + provenance: Provenance, + reason: str | None = None, +) -> Event: + return Event( + kind=EventKind.OPAQUE, + timestamp=timestamp, + payload={ + "source_event_type": "item_completed", + "source_item_type": item_type or "", + **({"reason": reason} if reason else {}), + }, + provenance=provenance, + ) + + def serialize( session: Session, *, diff --git a/tests/fixtures/codex-0.153.4/paginated.jsonl b/tests/fixtures/codex-0.153.4/paginated.jsonl new file mode 100644 index 0000000..46c6a3f --- /dev/null +++ b/tests/fixtures/codex-0.153.4/paginated.jsonl @@ -0,0 +1,14 @@ +{"timestamp":"2026-08-17T13:00:00Z","ordinal":0,"type":"session_meta","payload":{"session_id":"30000000-0000-4000-8000-000000000000","id":"30000000-0000-4000-8000-000000000000","timestamp":"2026-08-17T13:00:00Z","cwd":"/work","originator":"fixture","cli_version":"0.153.4","source":"cli","model_provider":"openai","history_mode":"paginated"}} +{"timestamp":"2026-08-17T13:00:00Z","ordinal":1,"type":"response_item","payload":{"type":"message","role":"user","content":[{"type":"input_text","text":"INTERNAL ENVIRONMENT CONTEXT THAT MUST NOT BE REPLAYED"}]}} +{"timestamp":"2026-08-17T13:00:00Z","ordinal":2,"type":"response_item","payload":{"type":"message","role":"user","content":[{"type":"input_text","text":"Remember synthetic migrator nonce BETA-2048."},{"type":"input_image","image_url":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="}]}} +{"timestamp":"2026-08-17T13:00:00Z","ordinal":3,"type":"event_msg","payload":{"type":"item_completed","item":{"type":"UserMessage","id":"10000000-0000-4000-8000-000000000001","client_id":"10000000-0000-4000-8000-000000000002","content":[{"type":"text","text":"Remember synthetic migrator nonce BETA-2048.","text_elements":[]},{"type":"image","image_url":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="}]}}} +{"timestamp":"2026-08-17T13:00:00Z","ordinal":4,"type":"token_usage_record","payload":{"response_id":"resp_fixture_1","root_turn_id":"10000000-0000-4000-8000-000000000010","session_id":"30000000-0000-4000-8000-000000000000","thread_id":"30000000-0000-4000-8000-000000000000","turn_id":"10000000-0000-4000-8000-000000000010","turn_token_usage":{"input_tokens":10,"output_tokens":5},"thread_token_usage":{"input_tokens":10,"output_tokens":5},"usage":{"input_tokens":10,"output_tokens":5}}} +{"timestamp":"2026-08-17T13:00:01Z","ordinal":5,"type":"response_item","payload":{"type":"message","role":"assistant","content":[{"type":"output_text","text":"I will remember the synthetic nonce."}]}} +{"timestamp":"2026-08-17T13:00:01Z","ordinal":6,"type":"event_msg","payload":{"type":"item_completed","item":{"type":"AgentMessage","id":"msg_fixture_1","content":[{"type":"Text","text":"I will remember the synthetic nonce."}],"phase":"final_answer"}}} +{"timestamp":"2026-08-17T13:00:02Z","ordinal":7,"type":"response_item","payload":{"type":"function_call","name":"shell","arguments":"{\"command\":\"pwd\"}","call_id":"call_fixture_1"}} +{"timestamp":"2026-08-17T13:00:03Z","ordinal":8,"type":"response_item","payload":{"type":"function_call_output","call_id":"call_fixture_1","output":[{"type":"input_text","text":"/work"},{"type":"input_image","image_url":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="}]}} +{"timestamp":"2026-08-17T13:00:04Z","ordinal":9,"type":"compacted","payload":{"message":"Synthetic summary: remember nonce BETA-2048 and the completed shell result."}} +{"timestamp":"2026-08-17T13:00:05Z","ordinal":10,"type":"response_item","payload":{"type":"message","role":"user","content":[{"type":"input_text","text":"Continue after the synthetic compaction."}]}} +{"timestamp":"2026-08-17T13:00:05Z","ordinal":11,"type":"event_msg","payload":{"type":"item_completed","item":{"type":"UserMessage","id":"10000000-0000-4000-8000-000000000003","content":[{"type":"text","text":"Continue after the synthetic compaction.","text_elements":[]}]}}} +{"timestamp":"2026-08-17T13:00:06Z","ordinal":12,"type":"response_item","payload":{"type":"message","role":"assistant","content":[{"type":"output_text","text":"The synthetic post-compaction fixture is complete."}]}} +{"timestamp":"2026-08-17T13:00:06Z","ordinal":13,"type":"event_msg","payload":{"type":"item_completed","item":{"type":"AgentMessage","id":"msg_fixture_2","content":[{"type":"Text","text":"The synthetic post-compaction fixture is complete."}],"phase":"final_answer"}}} diff --git a/tests/test_catalog.py b/tests/test_catalog.py index 4763c86..03d019d 100644 --- a/tests/test_catalog.py +++ b/tests/test_catalog.py @@ -106,6 +106,24 @@ def _codex_records( }, } ) + if history_mode == "paginated": + records[0]["payload"]["cli_version"] = "0.153.4" # type: ignore[index] + records.append( + { + "timestamp": "2026-08-18T13:00:03Z", + "type": "event_msg", + "payload": { + "type": "item_completed", + "item": { + "type": "UserMessage", + "id": "10000000-0000-4000-8000-000000000001", + "content": [{"type": "text", "text": "not indexed", "text_elements": []}], + }, + }, + } + ) + for ordinal, record in enumerate(records): + record["ordinal"] = ordinal return records @@ -336,7 +354,7 @@ def test_codex_active_archive_paginated_and_native_titles(tmp_path: Path) -> Non with _catalog(tmp_path) as catalog: result = catalog.refresh(codex_roots=(home,), include_auto=False) - assert result.statuses == {"candidate": 2, "unsupported": 1} + assert result.statuses == {"candidate": 3} event_name = catalog.list_sessions(query="rollout event") assert len(event_name) == 1 assert event_name[0].title_kind == "thread_name" @@ -348,10 +366,10 @@ def test_codex_active_archive_paginated_and_native_titles(tmp_path: Path) -> Non assert catalog.list_sessions(query="forbidden preview") == [] assert catalog.list_sessions(query="forbidden first-message") == [] - unsupported = catalog.list_sessions(statuses=("unsupported",)) - assert len(unsupported) == 1 - assert unsupported[0].history_mode == "paginated" - assert unsupported[0].reason == "codex_history_mode" + paginated_entry = catalog.list_sessions(query=PAGINATED_ID) + assert len(paginated_entry) == 1 + assert paginated_entry[0].history_mode == "paginated" + assert paginated_entry[0].status == "candidate" # A transiently unavailable vendor cache must not erase title metadata # already derived from it when the authoritative JSONL changes. @@ -373,6 +391,58 @@ def test_codex_active_archive_paginated_and_native_titles(tmp_path: Path) -> Non assert retained_name[0].title == "Native saved name" +def test_codex_paginated_catalog_fails_closed_on_incomplete_projections( + tmp_path: Path, +) -> None: + home = tmp_path / "codex" + cases = { + "40000000-0000-4000-8000-000000000001": ("codex_history_base", "unsupported"), + "40000000-0000-4000-8000-000000000002": ( + "codex_subagent_history", + "unsupported", + ), + "40000000-0000-4000-8000-000000000003": ( + "codex_paginated_ordinals", + "corrupt", + ), + "40000000-0000-4000-8000-000000000004": ("codex_history_mode", "unsupported"), + "40000000-0000-4000-8000-000000000005": ( + "codex_history_mode_conflict", + "corrupt", + ), + } + for session_id, (reason, _) in cases.items(): + path = home / "sessions" / "2026" / "08" / "18" / f"rollout-synthetic-{session_id}.jsonl" + records = _codex_records(session_id, history_mode="paginated") + meta = records[0]["payload"] + assert isinstance(meta, dict) + if reason == "codex_history_base": + meta["history_base"] = { + "thread_id": CODEX_ID, + "end_ordinal_exclusive": 2, + "end_byte_offset": 512, + } + elif reason == "codex_subagent_history": + meta["subagent_history_start_ordinal"] = 2 + elif reason == "codex_paginated_ordinals": + records[1]["ordinal"] = 99 + elif reason == "codex_history_mode": + meta["history_mode"] = "future" + else: + duplicate = json.loads(json.dumps(records[0])) + duplicate["payload"]["history_mode"] = "legacy" + duplicate["ordinal"] = len(records) + records.append(duplicate) + _write_jsonl(path, records) + + with _catalog(tmp_path) as catalog: + catalog.refresh(codex_roots=(home,), include_auto=False) + entries = catalog.list_sessions(statuses=("unsupported", "corrupt"), include_paths=True) + + observed = {entry.session_id: (entry.reason, entry.status) for entry in entries} + assert observed == cases + + def test_refresh_is_incremental_and_validation_is_explicit(tmp_path: Path) -> None: home = tmp_path / "claude" session = home / "projects" / "-synthetic" / f"{CLAUDE_ID}.jsonl" diff --git a/tests/test_codex_paginated_history_mode.py b/tests/test_codex_paginated_history_mode.py new file mode 100644 index 0000000..d4399a8 --- /dev/null +++ b/tests/test_codex_paginated_history_mode.py @@ -0,0 +1,151 @@ +"""Codex 0.147+ root paginated-history compatibility and safety tests.""" + +import json +from pathlib import Path + +import pytest + +from session_migrate.conversion import ConversionOptions, convert_session +from session_migrate.errors import SessionMigrateError +from session_migrate.formats import codex +from session_migrate.model import EventKind, Role, TargetFormat + +FIXTURES = Path(__file__).parent / "fixtures" +LEGACY = FIXTURES / "codex-0.144.4" / "basic.jsonl" +PAGINATED = FIXTURES / "codex-0.153.4" / "paginated.jsonl" + + +def _portable(session): + return [ + ( + event.kind, + event.role, + event.text, + event.tool_name, + event.tool_call_id, + event.payload.get("block_type"), + event.payload.get("image_url"), + event.payload.get("content_blocks"), + ) + for event in session.events + if event.kind != EventKind.OPAQUE + ] + + +def _rewrite_fixture(tmp_path: Path, mutate) -> Path: + records = [json.loads(line) for line in PAGINATED.read_text().splitlines()] + mutate(records) + path = tmp_path / "paginated.jsonl" + path.write_text("".join(json.dumps(record) + "\n" for record in records)) + return path + + +def test_paginated_uses_canonical_completed_items_and_matches_legacy() -> None: + session = codex.parse(PAGINATED) + + assert session.cli_version == "0.153.4" + assert _portable(session) == _portable(codex.parse(LEGACY)) + assert "INTERNAL ENVIRONMENT CONTEXT" not in " ".join( + event.text or "" for event in session.events + ) + assert [ + (event.role, event.text) for event in session.events if event.kind == EventKind.MESSAGE + ] == [ + (Role.USER, "Remember synthetic migrator nonce BETA-2048."), + (Role.ASSISTANT, "I will remember the synthetic nonce."), + (Role.USER, "Continue after the synthetic compaction."), + (Role.ASSISTANT, "The synthetic post-compaction fixture is complete."), + ] + + +def test_paginated_provider_messages_and_accounting_stay_opaque() -> None: + session = codex.parse(PAGINATED) + opaque = [event for event in session.events if event.kind == EventKind.OPAQUE] + + assert sum(event.payload.get("reason") == "paginated_provider_message" for event in opaque) == 5 + assert ( + sum(event.payload.get("source_record_type") == "token_usage_record" for event in opaque) + == 1 + ) + + +@pytest.mark.parametrize("target_format", tuple(TargetFormat)) +def test_paginated_context_cannot_leak_to_any_target( + tmp_path: Path, target_format: TargetFormat +) -> None: + artifact = convert_session( + codex.parse(PAGINATED), + ConversionOptions( + target_format=target_format, + session_id="50000000-0000-4000-8000-000000000001", + cwd=tmp_path, + ), + ) + + assert artifact.native_bytes + assert b"INTERNAL ENVIRONMENT CONTEXT" not in artifact.native_bytes + + +@pytest.mark.parametrize("ordinal", [None, True, "3", 99]) +def test_paginated_invalid_or_noncontiguous_ordinals_fail_closed( + tmp_path: Path, ordinal: object +) -> None: + def mutate(records): + if ordinal is None: + records[3].pop("ordinal") + else: + records[3]["ordinal"] = ordinal + + path = _rewrite_fixture(tmp_path, mutate) + with pytest.raises(SessionMigrateError, match="ordinal"): + codex.parse(path) + + +def test_unknown_history_mode_still_refused(tmp_path: Path) -> None: + def mutate(records): + records[0]["payload"]["history_mode"] = "some-future-mode" + + path = _rewrite_fixture(tmp_path, mutate) + with pytest.raises(SessionMigrateError, match="history mode"): + codex.parse(path) + + +def test_history_base_still_refused(tmp_path: Path) -> None: + def mutate(records): + records[0]["payload"]["history_base"] = { + "thread_id": "10000000-0000-4000-8000-000000000000", + "end_ordinal_exclusive": 12, + "end_byte_offset": 1024, + } + + path = _rewrite_fixture(tmp_path, mutate) + with pytest.raises(SessionMigrateError, match="history_base"): + codex.parse(path) + + +def test_paginated_subagent_projection_still_refused(tmp_path: Path) -> None: + def mutate(records): + records[0]["payload"]["subagent_history_start_ordinal"] = 8 + + path = _rewrite_fixture(tmp_path, mutate) + with pytest.raises(SessionMigrateError, match="subagent history projection"): + codex.parse(path) + + +def test_paginated_metadata_must_be_first_and_consistent(tmp_path: Path) -> None: + def move_meta(records): + records[0], records[1] = records[1], records[0] + records[0]["ordinal"], records[1]["ordinal"] = 0, 1 + + with pytest.raises(SessionMigrateError, match="start with session metadata"): + codex.parse(_rewrite_fixture(tmp_path, move_meta)) + + def duplicate_conflicting_meta(records): + duplicate = json.loads(json.dumps(records[0])) + duplicate["payload"]["history_mode"] = "legacy" + records.append(duplicate) + for ordinal, record in enumerate(records): + record["ordinal"] = ordinal + + with pytest.raises(SessionMigrateError, match="conflicting history modes"): + codex.parse(_rewrite_fixture(tmp_path, duplicate_conflicting_meta)) diff --git a/tests/test_conversion.py b/tests/test_conversion.py index a36b017..03c81b6 100644 --- a/tests/test_conversion.py +++ b/tests/test_conversion.py @@ -902,7 +902,7 @@ def parse_then_append(source_path: Path) -> Session: load_session(path, AgentFormat.CLAUDE) -def test_rejects_paginated_and_expands_replacement_history(tmp_path: Path) -> None: +def test_rejects_unknown_history_mode_and_expands_replacement_history(tmp_path: Path) -> None: base_meta = { "timestamp": "2026-08-17T12:00:00Z", "type": "session_meta", @@ -914,11 +914,14 @@ def test_rejects_paginated_and_expands_replacement_history(tmp_path: Path) -> No "model_provider": "openai", }, } - paginated = json.loads(json.dumps(base_meta)) - paginated["payload"]["history_mode"] = "paginated" - paginated_path = write_jsonl(tmp_path / "paginated.jsonl", [paginated]) + # "paginated" is the layout Codex 0.147+ writes and is now accepted; see + # tests/test_codex_paginated_history_mode.py for the equivalence evidence. + # An unrecognized mode must still fail closed. + future = json.loads(json.dumps(base_meta)) + future["payload"]["history_mode"] = "some-future-mode" + future_path = write_jsonl(tmp_path / "future-mode.jsonl", [future]) with pytest.raises(SessionMigrateError, match="history mode"): - codex.parse(paginated_path) + codex.parse(future_path) replacement_path = write_jsonl( tmp_path / "replacement.jsonl",