From 2ce92a7fd649e64e84d292a16e40790264496ee0 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Thu, 10 Sep 2026 13:49:20 -0500 Subject: [PATCH 1/3] feat(codex): accept history_mode "paginated" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex CLI sessions recorded by 0.147 and later cannot be migrated at all: the adapter requires session_meta.history_mode to be "legacy", and every release from 0.147 onward writes "paginated" instead. On a machine running current Codex, that is effectively the whole local corpus — 1330 of 1403 rollouts on the one I checked, and the proportion only grows as older sessions age out. The failure is quiet in the sense that matters: the CLI exits with a clear message, so it reads as "this session is unsupported" rather than "this adapter is pinned to a release from before the format changed", which is probably why it has gone unreported. The paginated layout keeps the records this adapter reads at the same top level. It adds two record types: `item_completed`, which wraps items that are ALSO emitted as their own `response_item`/`event_msg` records, and `token_usage_record` accounting. Neither carries model-visible content, and because the adapter already routes unrecognized record types to OPAQUE events, both are retained for provenance without being rendered into a target transcript. So the existing parse is correct for paginated sessions as written — the guard was refusing input it could already handle. `item_completed` being a duplicate view is the one real hazard here, and it is why this does not simply delete the check: interpreting those wrappers as turns would double every assistant message. A test pins them as OPAQUE so a later change cannot start reading them without the duplication showing up. `history_base` remains a hard error. That field means the session is a fork whose earlier turns live in a different rollout file, and resolving that lineage is not implemented — converting such a session without it would silently drop the parent's history, which is worse than refusing. The two conditions were previously checked together; they are different problems and now fail differently. Verified against real sessions from Codex 0.153.4: an ordinary paginated session converts (exit 0), and a forked one still refuses with the history_base error. The conversion manifest continues to report `unvalidated_source_version` when the source CLI differs from the pinned target, so accepting the newer layout does not suppress the existing version warning. Adds tests/fixtures/codex-0.153.4/paginated.jsonl — synthetic and credential-free, derived from the existing 0.144.4 fixture so the two can be compared turn for turn. Tests assert the visible conversation is identical across layouts, that the extra records stay OPAQUE, that an unknown future history_mode still fails closed, and that history_base still refuses. Not verified here: whether the converted output resumes in the target client. docs/development.md asks for a native-resume oracle for adapter changes; I could not run scripts/verify-native-resume.sh (pinned Docker image) in this environment, so that gate is unmet and worth running before merge. Updates test_rejects_paginated_and_expands_replacement_history rather than removing its paginated half: the case now asserts that an unrecognized mode still fails closed, and is renamed to test_rejects_unknown_history_mode_and_expands_replacement_history to match. Its replacement_history coverage is untouched. Assisted-by: AI --- src/session_migrate/formats/codex.py | 18 +++- tests/fixtures/codex-0.153.4/paginated.jsonl | 20 +++++ tests/test_codex_paginated_history_mode.py | 86 ++++++++++++++++++++ tests/test_conversion.py | 13 +-- 4 files changed, 130 insertions(+), 7 deletions(-) create mode 100644 tests/fixtures/codex-0.153.4/paginated.jsonl create mode 100644 tests/test_codex_paginated_history_mode.py diff --git a/src/session_migrate/formats/codex.py b/src/session_migrate/formats/codex.py index b7f61db..1983d75 100644 --- a/src/session_migrate/formats/codex.py +++ b/src/session_migrate/formats/codex.py @@ -17,6 +17,19 @@ PINNED_CODEX_VERSION = "0.144.4" +# Codex writes one of these in session_meta.history_mode. "legacy" is the +# original single-file layout; "paginated" was introduced during 0.147 and is +# what every later release writes. Both place the conversation items this +# adapter reads at the same top level, so the same parse applies: `paginated` +# additionally emits `item_completed` wrappers around items that are already +# present as their own records, plus `token_usage_record` accounting. Neither is +# a source of model-visible content, so ignoring them loses nothing. +# +# `history_base` is a genuinely different matter and stays a hard error below: +# it means the session is a fork whose earlier turns live in another file, and +# resolving that lineage is not implemented. +SUPPORTED_HISTORY_MODES = frozenset({"legacy", "paginated"}) + def parse(path: Path) -> Session: records = list(iter_jsonl(path)) @@ -43,9 +56,10 @@ def parse(path: Path) -> Session: if record_type == "session_meta": if not canonical_meta_seen: history_mode = string(payload.get("history_mode")) - if history_mode and history_mode != "legacy": + if history_mode and history_mode not in SUPPORTED_HISTORY_MODES: raise SessionMigrateError( - f"Codex history mode {history_mode!r} is not supported; expected legacy" + f"Codex history mode {history_mode!r} is not supported; " + f"expected one of {', '.join(sorted(SUPPORTED_HISTORY_MODES))}" ) if payload.get("history_base") is not None: raise SessionMigrateError("Codex history_base lineage is not supported") diff --git a/tests/fixtures/codex-0.153.4/paginated.jsonl b/tests/fixtures/codex-0.153.4/paginated.jsonl new file mode 100644 index 0000000..34a8e6a --- /dev/null +++ b/tests/fixtures/codex-0.153.4/paginated.jsonl @@ -0,0 +1,20 @@ +{"timestamp": "2026-08-17T13:00:00Z", "type": "session_meta", "payload": {"session_id": "30000000-0000-4000-8000-000000000000", "id": "30000000-0000-4000-8000-000000000000", "timestamp": "2026-08-17T13:00:00Z", "cwd": "/work", "originator": "fixture", "cli_version": "0.153.4", "source": "cli", "model_provider": "openai", "history_mode": "paginated"}} +{"timestamp": "2026-08-17T13:00:00Z", "type": "event_msg", "payload": {"type": "user_message", "message": "Remember synthetic migrator nonce BETA-2048."}} +{"timestamp": "2026-08-17T13:00:00Z", "type": "response_item", "payload": {"type": "message", "role": "user", "content": [{"type": "input_text", "text": "Remember synthetic migrator nonce BETA-2048."}, {"type": "input_image", "image_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="}]}} +{"timestamp": "2026-08-17T13:00:00Z", "type": "item_completed", "payload": {"type": "item_completed", "item": {"type": "UserMessage"}}} +{"timestamp": "2026-08-17T13:00:00Z", "type": "token_usage_record", "payload": {"type": "token_usage_record", "input_tokens": 10, "output_tokens": 5}} +{"timestamp": "2026-08-17T13:00:01Z", "type": "event_msg", "payload": {"type": "agent_message", "message": "I will remember the synthetic nonce."}} +{"timestamp": "2026-08-17T13:00:01Z", "type": "response_item", "payload": {"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "I will remember the synthetic nonce."}]}} +{"timestamp": "2026-08-17T13:00:01Z", "type": "item_completed", "payload": {"type": "item_completed", "item": {"type": "AgentMessage"}}} +{"timestamp": "2026-08-17T13:00:01Z", "type": "token_usage_record", "payload": {"type": "token_usage_record", "input_tokens": 10, "output_tokens": 5}} +{"timestamp": "2026-08-17T13:00:02Z", "type": "response_item", "payload": {"type": "function_call", "name": "shell", "arguments": "{\"command\":\"pwd\"}", "call_id": "call_fixture_1"}} +{"timestamp": "2026-08-17T13:00:03Z", "type": "response_item", "payload": {"type": "function_call_output", "call_id": "call_fixture_1", "output": [{"type": "input_text", "text": "/work"}, {"type": "input_image", "image_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="}]}} +{"timestamp": "2026-08-17T13:00:04Z", "type": "compacted", "payload": {"message": "Synthetic summary: remember nonce BETA-2048 and the completed shell result."}} +{"timestamp": "2026-08-17T13:00:05Z", "type": "event_msg", "payload": {"type": "user_message", "message": "Continue after the synthetic compaction."}} +{"timestamp": "2026-08-17T13:00:05Z", "type": "response_item", "payload": {"type": "message", "role": "user", "content": [{"type": "input_text", "text": "Continue after the synthetic compaction."}]}} +{"timestamp": "2026-08-17T13:00:05Z", "type": "item_completed", "payload": {"type": "item_completed", "item": {"type": "UserMessage"}}} +{"timestamp": "2026-08-17T13:00:05Z", "type": "token_usage_record", "payload": {"type": "token_usage_record", "input_tokens": 10, "output_tokens": 5}} +{"timestamp": "2026-08-17T13:00:06Z", "type": "event_msg", "payload": {"type": "agent_message", "message": "The synthetic post-compaction fixture is complete."}} +{"timestamp": "2026-08-17T13:00:06Z", "type": "response_item", "payload": {"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "The synthetic post-compaction fixture is complete."}]}} +{"timestamp": "2026-08-17T13:00:06Z", "type": "item_completed", "payload": {"type": "item_completed", "item": {"type": "AgentMessage"}}} +{"timestamp": "2026-08-17T13:00:06Z", "type": "token_usage_record", "payload": {"type": "token_usage_record", "input_tokens": 10, "output_tokens": 5}} diff --git a/tests/test_codex_paginated_history_mode.py b/tests/test_codex_paginated_history_mode.py new file mode 100644 index 0000000..68b042b --- /dev/null +++ b/tests/test_codex_paginated_history_mode.py @@ -0,0 +1,86 @@ +"""Codex 0.147+ writes history_mode="paginated"; earlier releases wrote "legacy". + +The paginated layout keeps the model-visible items this adapter reads at the same +top level. It adds `item_completed` wrappers around items that are already +emitted as their own records, plus `token_usage_record` accounting — neither +carries model-visible content. These tests pin that equivalence so the adapter +cannot start depending on the wrappers, and pin that a forked session +(`history_base`) is still refused rather than silently truncated. +""" + +from pathlib import Path + +import pytest + +from session_migrate.errors import SessionMigrateError +from session_migrate.formats import codex + +FIXTURES = Path(__file__).parent / "fixtures" +LEGACY = FIXTURES / "codex-0.144.4" / "basic.jsonl" +PAGINATED = FIXTURES / "codex-0.153.4" / "paginated.jsonl" + + +def _visible(session): + """Role/text pairs a target format receives, excluding inert records. + + Paginated sessions carry `item_completed` and `token_usage_record` entries + that the adapter keeps as OPAQUE events — preserved for provenance, never + rendered into a target transcript. They are excluded here so this compares + the conversation itself. + """ + return [(event.role, event.text) for event in session.events if event.role is not None] + + +def test_paginated_history_mode_parses(): + session = codex.parse(PAGINATED) + assert session.cli_version == "0.153.4" + assert session.events, "paginated session produced no events" + + +def test_paginated_matches_legacy_conversation(): + """The added wrappers must not change, duplicate, or drop visible turns.""" + assert _visible(codex.parse(PAGINATED)) == _visible(codex.parse(LEGACY)) + + +def test_paginated_extra_records_stay_opaque(): + """`item_completed` duplicates items already parsed; it must not become a turn. + + This is the real hazard in accepting the paginated layout: `item_completed` + wraps items that are ALSO emitted as their own `response_item` records, so + interpreting both would double every assistant message. + """ + session = codex.parse(PAGINATED) + extra = [event for event in session.events if event.role is None] + assert extra, "expected the paginated-only records to be retained" + assert {event.payload.get("source_record_type") for event in extra} == { + "item_completed", + "token_usage_record", + } + assert all(event.kind.value == "opaque" for event in extra) + + +def test_unknown_history_mode_still_refused(tmp_path): + """An unrecognized mode must fail closed, not be parsed hopefully.""" + lines = PAGINATED.read_text().splitlines() + lines[0] = lines[0].replace('"paginated"', '"some-future-mode"') + path = tmp_path / "future.jsonl" + path.write_text("\n".join(lines) + "\n") + with pytest.raises(SessionMigrateError, match="history mode"): + codex.parse(path) + + +def test_history_base_still_refused(tmp_path): + """A fork's earlier turns live in another file; refusing beats truncating.""" + import json + + lines = PAGINATED.read_text().splitlines() + meta = json.loads(lines[0]) + meta["payload"]["history_base"] = { + "thread_id": "10000000-0000-4000-8000-000000000000", + "end_ordinal_exclusive": 12, + } + lines[0] = json.dumps(meta) + path = tmp_path / "forked.jsonl" + path.write_text("\n".join(lines) + "\n") + with pytest.raises(SessionMigrateError, match="history_base"): + codex.parse(path) diff --git a/tests/test_conversion.py b/tests/test_conversion.py index a36b017..03c81b6 100644 --- a/tests/test_conversion.py +++ b/tests/test_conversion.py @@ -902,7 +902,7 @@ def parse_then_append(source_path: Path) -> Session: load_session(path, AgentFormat.CLAUDE) -def test_rejects_paginated_and_expands_replacement_history(tmp_path: Path) -> None: +def test_rejects_unknown_history_mode_and_expands_replacement_history(tmp_path: Path) -> None: base_meta = { "timestamp": "2026-08-17T12:00:00Z", "type": "session_meta", @@ -914,11 +914,14 @@ def test_rejects_paginated_and_expands_replacement_history(tmp_path: Path) -> No "model_provider": "openai", }, } - paginated = json.loads(json.dumps(base_meta)) - paginated["payload"]["history_mode"] = "paginated" - paginated_path = write_jsonl(tmp_path / "paginated.jsonl", [paginated]) + # "paginated" is the layout Codex 0.147+ writes and is now accepted; see + # tests/test_codex_paginated_history_mode.py for the equivalence evidence. + # An unrecognized mode must still fail closed. + future = json.loads(json.dumps(base_meta)) + future["payload"]["history_mode"] = "some-future-mode" + future_path = write_jsonl(tmp_path / "future-mode.jsonl", [future]) with pytest.raises(SessionMigrateError, match="history mode"): - codex.parse(paginated_path) + codex.parse(future_path) replacement_path = write_jsonl( tmp_path / "replacement.jsonl", From fdd5cc7a2877d62b954bea0d20bbb4bc4b9e7d57 Mon Sep 17 00:00:00 2001 From: xhluca Date: Fri, 11 Sep 2026 15:41:11 -0400 Subject: [PATCH 2/3] fix(codex): parse paginated roots from canonical items --- scripts/validate-additional-target-corpus.py | 6 + scripts/validate-antigravity-native.py | 6 + scripts/validate-copilot-native.py | 6 + scripts/validate-core-target-native.py | 6 + scripts/validate-muse-qwen-kimi-corpus.py | 6 + src/session_migrate/catalog.py | 62 ++++- src/session_migrate/formats/codex.py | 244 +++++++++++++++++-- tests/fixtures/codex-0.153.4/paginated.jsonl | 34 ++- tests/test_catalog.py | 80 +++++- tests/test_codex_paginated_history_mode.py | 187 +++++++++----- 10 files changed, 515 insertions(+), 122 deletions(-) diff --git a/scripts/validate-additional-target-corpus.py b/scripts/validate-additional-target-corpus.py index f3cc006..098951d 100644 --- a/scripts/validate-additional-target-corpus.py +++ b/scripts/validate-additional-target-corpus.py @@ -205,6 +205,12 @@ def expected_source_rejection(source_format: AgentFormat, exc: SessionMigrateErr return "codex_history_mode" if "history_base lineage is not supported" in message: return "codex_history_base" + if "subagent history projection is not supported" in message: + return "codex_subagent_history" + if "conflicting history modes" in message: + return "codex_history_mode_conflict" + if "paginated" in message and "ordinal" in message: + return "codex_paginated_ordinals" return None diff --git a/scripts/validate-antigravity-native.py b/scripts/validate-antigravity-native.py index 51507e9..4fbd20a 100644 --- a/scripts/validate-antigravity-native.py +++ b/scripts/validate-antigravity-native.py @@ -199,6 +199,12 @@ def expected_rejection(source_format: AgentFormat, exc: SessionMigrateError) -> return "codex_history_mode" if "history_base lineage is not supported" in message: return "codex_history_base" + if "subagent history projection is not supported" in message: + return "codex_subagent_history" + if "conflicting history modes" in message: + return "codex_history_mode_conflict" + if "paginated" in message and "ordinal" in message: + return "codex_paginated_ordinals" return None diff --git a/scripts/validate-copilot-native.py b/scripts/validate-copilot-native.py index 3e14477..9b01974 100644 --- a/scripts/validate-copilot-native.py +++ b/scripts/validate-copilot-native.py @@ -226,6 +226,12 @@ def _expected_rejection(source_format: AgentFormat, exc: SessionMigrateError) -> return "codex_history_mode" if "history_base lineage is not supported" in message: return "codex_history_base" + if "subagent history projection is not supported" in message: + return "codex_subagent_history" + if "conflicting history modes" in message: + return "codex_history_mode_conflict" + if "paginated" in message and "ordinal" in message: + return "codex_paginated_ordinals" return None diff --git a/scripts/validate-core-target-native.py b/scripts/validate-core-target-native.py index 83fa29d..4e78711 100755 --- a/scripts/validate-core-target-native.py +++ b/scripts/validate-core-target-native.py @@ -172,6 +172,12 @@ def expected_rejection(source_format: AgentFormat, exc: SessionMigrateError) -> return "codex_history_mode" if "history_base lineage is not supported" in message: return "codex_history_base" + if "subagent history projection is not supported" in message: + return "codex_subagent_history" + if "conflicting history modes" in message: + return "codex_history_mode_conflict" + if "paginated" in message and "ordinal" in message: + return "codex_paginated_ordinals" return None diff --git a/scripts/validate-muse-qwen-kimi-corpus.py b/scripts/validate-muse-qwen-kimi-corpus.py index 2bfc614..4bda86c 100755 --- a/scripts/validate-muse-qwen-kimi-corpus.py +++ b/scripts/validate-muse-qwen-kimi-corpus.py @@ -178,6 +178,12 @@ def expected_rejection(source_format: AgentFormat, exc: SessionMigrateError) -> return "codex_history_mode" if "history_base lineage is not supported" in message: return "codex_history_base" + if "subagent history projection is not supported" in message: + return "codex_subagent_history" + if "conflicting history modes" in message: + return "codex_history_mode_conflict" + if "paginated" in message and "ordinal" in message: + return "codex_paginated_ordinals" return None diff --git a/src/session_migrate/catalog.py b/src/session_migrate/catalog.py index 2de1aae..2c34fe1 100644 --- a/src/session_migrate/catalog.py +++ b/src/session_migrate/catalog.py @@ -25,6 +25,7 @@ from session_migrate.errors import JsonlError, SessionMigrateError from session_migrate.formats import ( antigravity, + codex, devin, hermes, kimi, @@ -2077,6 +2078,10 @@ def _scan_file(path: Path, agent_format: AgentFormat, root: Path) -> _Scan: cli_version = None history_mode = None history_base = False + codex_subagent_history = False + codex_ordinals_complete = True + codex_history_mode_conflict = False + codex_selected_history_mode = None sidechain = False records = 0 has_conversation = False @@ -2114,6 +2119,12 @@ def _scan_file(path: Path, agent_format: AgentFormat, root: Path) -> _Scan: if title: labels.append(_Label("ai_title", title, record.index, 90)) elif agent_format == AgentFormat.CODEX: + ordinal = value.get("ordinal") + codex_ordinals_complete = codex_ordinals_complete and ( + isinstance(ordinal, int) + and not isinstance(ordinal, bool) + and ordinal == records - 1 + ) if record_type in {"user", "assistant"} and isinstance(value.get("message"), dict): wrong_format = True payload = value.get("payload") @@ -2131,8 +2142,21 @@ def _scan_file(path: Path, agent_format: AgentFormat, root: Path) -> _Scan: or _string(value.get("timestamp")) ) cli_version = cli_version or _string(payload.get("cli_version")) - history_mode = history_mode or _string(payload.get("history_mode")) + raw_history_mode = _string(payload.get("history_mode")) + observed_history_mode = raw_history_mode or "legacy" + codex_history_mode_conflict = codex_history_mode_conflict or ( + codex_selected_history_mode is not None + and observed_history_mode != codex_selected_history_mode + ) + codex_selected_history_mode = ( + codex_selected_history_mode or observed_history_mode + ) + history_mode = history_mode or raw_history_mode history_base = history_base or payload.get("history_base") is not None + codex_subagent_history = ( + codex_subagent_history + or payload.get("subagent_history_start_ordinal") is not None + ) elif record_type == "response_item": has_conversation = has_conversation or payload.get("type") in { "message", @@ -2141,13 +2165,20 @@ def _scan_file(path: Path, agent_format: AgentFormat, root: Path) -> _Scan: "function_call_output", "custom_tool_call_output", } - elif record_type == "event_msg" and payload.get("type") == "thread_name_updated": - title = _bounded( - _string(payload.get("name")) or _string(payload.get("thread_name")), - LABEL_LIMIT, - ) - if title: - labels.append(_Label("thread_name", title, record.index, 110)) + elif record_type == "event_msg": + if payload.get("type") == "item_completed": + item = payload.get("item") + has_conversation = has_conversation or ( + isinstance(item, dict) + and item.get("type") in {"UserMessage", "AgentMessage"} + ) + elif payload.get("type") == "thread_name_updated": + title = _bounded( + _string(payload.get("name")) or _string(payload.get("thread_name")), + LABEL_LIMIT, + ) + if title: + labels.append(_Label("thread_name", title, record.index, 110)) elif agent_format in {AgentFormat.PI, AgentFormat.OMP}: if record_type in {"user", "assistant", "session_meta", "response_item"}: wrong_format = True @@ -2264,10 +2295,21 @@ def _scan_file(path: Path, agent_format: AgentFormat, root: Path) -> _Scan: elif agent_format == AgentFormat.CODEX: if not has_session_meta: status, reason = "corrupt", "missing_session_meta" - elif history_mode and history_mode != "legacy": - status, reason = "unsupported", "codex_history_mode" elif history_base: status, reason = "unsupported", "codex_history_base" + elif codex_subagent_history: + status, reason = "unsupported", "codex_subagent_history" + elif codex_history_mode_conflict: + status, reason = "corrupt", "codex_history_mode_conflict" + elif ( + codex_selected_history_mode + and codex_selected_history_mode not in codex.SUPPORTED_HISTORY_MODES + ): + status, reason = "unsupported", "codex_history_mode" + elif codex_selected_history_mode == "paginated" and first_record_type != "session_meta": + status, reason = "corrupt", "missing_session_meta" + elif codex_selected_history_mode == "paginated" and not codex_ordinals_complete: + status, reason = "corrupt", "codex_paginated_ordinals" elif not has_conversation: status, reason = "corrupt", "no_conversation_records" elif agent_format == AgentFormat.PI: diff --git a/src/session_migrate/formats/codex.py b/src/session_migrate/formats/codex.py index 1983d75..e691aae 100644 --- a/src/session_migrate/formats/codex.py +++ b/src/session_migrate/formats/codex.py @@ -17,22 +17,20 @@ PINNED_CODEX_VERSION = "0.144.4" -# Codex writes one of these in session_meta.history_mode. "legacy" is the -# original single-file layout; "paginated" was introduced during 0.147 and is -# what every later release writes. Both place the conversation items this -# adapter reads at the same top level, so the same parse applies: `paginated` -# additionally emits `item_completed` wrappers around items that are already -# present as their own records, plus `token_usage_record` accounting. Neither is -# a source of model-visible content, so ignoring them loses nothing. -# -# `history_base` is a genuinely different matter and stays a hard error below: -# it means the session is a fork whose earlier turns live in another file, and -# resolving that lineage is not implemented. +# Codex 0.147+ paginated rollouts persist canonical TurnItems in +# event_msg/item_completed records. Provider response_item messages are not an +# equivalent transcript: they also contain synthetic environment and developer +# context. Paginated parsing therefore takes user/assistant messages only from +# the canonical completed items while retaining non-message response items for +# tool and reasoning data. SUPPORTED_HISTORY_MODES = frozenset({"legacy", "paginated"}) def parse(path: Path) -> Session: records = list(iter_jsonl(path)) + history_mode = _history_mode(records) + if history_mode == "paginated": + _validate_paginated_root(records) events: list[Event] = [] fallback_events: list[Event] = [] context_compacted_events: list[Event] = [] @@ -45,7 +43,6 @@ def parse(path: Path) -> Session: response_message_count = 0 response_messages: Counter[tuple[Role | None, str]] = Counter() title = None - canonical_meta_seen = False for record in records: value = record.value @@ -54,16 +51,6 @@ def parse(path: Path) -> Session: payload = object_value(value.get("payload")) provenance = Provenance(record.index, record_type) if record_type == "session_meta": - if not canonical_meta_seen: - history_mode = string(payload.get("history_mode")) - if history_mode and history_mode not in SUPPORTED_HISTORY_MODES: - raise SessionMigrateError( - f"Codex history mode {history_mode!r} is not supported; " - f"expected one of {', '.join(sorted(SUPPORTED_HISTORY_MODES))}" - ) - if payload.get("history_base") is not None: - raise SessionMigrateError("Codex history_base lineage is not supported") - canonical_meta_seen = True session_id = ( session_id or string(payload.get("id")) or string(payload.get("session_id")) ) @@ -74,7 +61,23 @@ def parse(path: Path) -> Session: model_provider = model_provider or string(payload.get("model_provider")) continue if record_type == "response_item": - parsed = _response_item_events(payload, timestamp, provenance) + if history_mode == "paginated" and string(payload.get("type")) in { + "message", + "agent_message", + }: + parsed = [ + Event( + kind=EventKind.OPAQUE, + timestamp=timestamp, + payload={ + "source_item_type": string(payload.get("type")) or "", + "reason": "paginated_provider_message", + }, + provenance=provenance, + ) + ] + else: + parsed = _response_item_events(payload, timestamp, provenance) events.extend(parsed) response_message_count += sum( event.kind == EventKind.MESSAGE and event.role in {Role.USER, Role.ASSISTANT} @@ -89,7 +92,9 @@ def parse(path: Path) -> Session: ) elif record_type == "event_msg": event_type = string(payload.get("type")) - if event_type == "user_message": + if event_type == "item_completed" and history_mode == "paginated": + events.extend(_paginated_completed_item_events(payload, timestamp, provenance)) + elif event_type == "user_message" and history_mode == "legacy": fallback_events.append( Event( kind=EventKind.MESSAGE, @@ -99,7 +104,7 @@ def parse(path: Path) -> Session: provenance=provenance, ) ) - elif event_type == "agent_message": + elif event_type == "agent_message" and history_mode == "legacy": fallback_events.append( Event( kind=EventKind.MESSAGE, @@ -181,7 +186,9 @@ def parse(path: Path) -> Session: ) ) - if response_message_count == 0: + if history_mode == "paginated": + events.sort(key=lambda event: event.provenance.record_index) + elif response_message_count == 0: events.extend(event for event in fallback_events if event.text) events.sort(key=lambda event: event.provenance.record_index) else: @@ -220,6 +227,191 @@ def parse(path: Path) -> Session: ) +def _history_mode(records: list[Any]) -> str: + selected_mode = "legacy" + metadata_seen = False + for record in records: + if string(record.value.get("type")) != "session_meta": + continue + payload = object_value(record.value.get("payload")) + history_mode = string(payload.get("history_mode")) or "legacy" + if history_mode not in SUPPORTED_HISTORY_MODES: + raise SessionMigrateError( + f"Codex history mode {history_mode!r} is not supported; " + f"expected one of {', '.join(sorted(SUPPORTED_HISTORY_MODES))}" + ) + if payload.get("history_base") is not None: + raise SessionMigrateError("Codex history_base lineage is not supported") + if payload.get("subagent_history_start_ordinal") is not None: + raise SessionMigrateError( + "Codex paginated subagent history projection is not supported" + ) + if metadata_seen and history_mode != selected_mode: + raise SessionMigrateError("Codex session metadata has conflicting history modes") + selected_mode = history_mode + metadata_seen = True + if selected_mode == "paginated" and ( + not records or string(records[0].value.get("type")) != "session_meta" + ): + raise SessionMigrateError("Codex paginated history must start with session metadata") + return selected_mode + + +def _validate_paginated_root(records: list[Any]) -> None: + """Fail closed if a root rollout is not a complete canonical ordinal stream.""" + + for expected, record in enumerate(records): + ordinal = record.value.get("ordinal") + if isinstance(ordinal, bool) or not isinstance(ordinal, int): + raise SessionMigrateError( + f"Codex paginated record {record.index} is missing an integer ordinal" + ) + if ordinal != expected: + raise SessionMigrateError( + "Codex paginated ordinals must be contiguous from zero; " + f"record {record.index} has ordinal {ordinal}, expected {expected}" + ) + + +def _paginated_completed_item_events( + payload: dict[str, Any], + timestamp: str | None, + provenance: Provenance, +) -> list[Event]: + item = object_value(payload.get("item")) + item_type = string(item.get("type")) + if item_type == "UserMessage": + return _paginated_user_message_events(item, timestamp, provenance) + if item_type == "AgentMessage": + content = item.get("content") + if not isinstance(content, list): + return [_opaque_completed_item(item_type, timestamp, provenance, "invalid_content")] + result: list[Event] = [] + for block_index, block in enumerate(content): + block_provenance = Provenance( + provenance.record_index, + provenance.record_type, + block_index=block_index, + ) + if not isinstance(block, dict): + result.append( + _opaque_completed_item(item_type, timestamp, block_provenance, "invalid_block") + ) + continue + block_type = string(block.get("type")) + text_value = string(block.get("text")) + if block_type in {"Text", "text"} and text_value: + result.append( + Event( + kind=EventKind.MESSAGE, + role=Role.ASSISTANT, + text=text_value, + timestamp=timestamp, + provenance=block_provenance, + ) + ) + else: + result.append( + _opaque_completed_item( + item_type, + timestamp, + block_provenance, + f"unsupported_block:{block_type or ''}", + ) + ) + return result + return [_opaque_completed_item(item_type, timestamp, provenance)] + + +def _paginated_user_message_events( + item: dict[str, Any], + timestamp: str | None, + provenance: Provenance, +) -> list[Event]: + content = item.get("content") + if not isinstance(content, list): + return [_opaque_completed_item("UserMessage", timestamp, provenance, "invalid_content")] + result: list[Event] = [] + for block_index, block in enumerate(content): + block_provenance = Provenance( + provenance.record_index, + provenance.record_type, + block_index=block_index, + ) + if not isinstance(block, dict): + result.append( + _opaque_completed_item("UserMessage", timestamp, block_provenance, "invalid_block") + ) + continue + block_type = string(block.get("type")) + if block_type == "text": + text_value = string(block.get("text")) + if text_value: + result.append( + Event( + kind=EventKind.MESSAGE, + role=Role.USER, + text=text_value, + timestamp=timestamp, + provenance=block_provenance, + ) + ) + elif block_type == "image": + result.append( + Event( + kind=EventKind.CONTEXT, + role=Role.USER, + timestamp=timestamp, + payload={ + "block_type": "image", + "image_url": string(block.get("image_url")), + }, + provenance=block_provenance, + ) + ) + elif block_type == "audio": + result.append( + Event( + kind=EventKind.CONTEXT, + role=Role.USER, + timestamp=timestamp, + payload={ + "block_type": "audio", + "audio_url": string(block.get("audio_url")), + }, + provenance=block_provenance, + ) + ) + else: + result.append( + _opaque_completed_item( + "UserMessage", + timestamp, + block_provenance, + f"unsupported_block:{block_type or ''}", + ) + ) + return result + + +def _opaque_completed_item( + item_type: str | None, + timestamp: str | None, + provenance: Provenance, + reason: str | None = None, +) -> Event: + return Event( + kind=EventKind.OPAQUE, + timestamp=timestamp, + payload={ + "source_event_type": "item_completed", + "source_item_type": item_type or "", + **({"reason": reason} if reason else {}), + }, + provenance=provenance, + ) + + def serialize( session: Session, *, diff --git a/tests/fixtures/codex-0.153.4/paginated.jsonl b/tests/fixtures/codex-0.153.4/paginated.jsonl index 34a8e6a..46c6a3f 100644 --- a/tests/fixtures/codex-0.153.4/paginated.jsonl +++ b/tests/fixtures/codex-0.153.4/paginated.jsonl @@ -1,20 +1,14 @@ -{"timestamp": "2026-08-17T13:00:00Z", "type": "session_meta", "payload": {"session_id": "30000000-0000-4000-8000-000000000000", "id": "30000000-0000-4000-8000-000000000000", "timestamp": "2026-08-17T13:00:00Z", "cwd": "/work", "originator": "fixture", "cli_version": "0.153.4", "source": "cli", "model_provider": "openai", "history_mode": "paginated"}} -{"timestamp": "2026-08-17T13:00:00Z", "type": "event_msg", "payload": {"type": "user_message", "message": "Remember synthetic migrator nonce BETA-2048."}} -{"timestamp": "2026-08-17T13:00:00Z", "type": "response_item", "payload": {"type": "message", "role": "user", "content": [{"type": "input_text", "text": "Remember synthetic migrator nonce BETA-2048."}, {"type": "input_image", "image_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="}]}} -{"timestamp": "2026-08-17T13:00:00Z", "type": "item_completed", "payload": {"type": "item_completed", "item": {"type": "UserMessage"}}} -{"timestamp": "2026-08-17T13:00:00Z", "type": "token_usage_record", "payload": {"type": "token_usage_record", "input_tokens": 10, "output_tokens": 5}} -{"timestamp": "2026-08-17T13:00:01Z", "type": "event_msg", "payload": {"type": "agent_message", "message": "I will remember the synthetic nonce."}} -{"timestamp": "2026-08-17T13:00:01Z", "type": "response_item", "payload": {"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "I will remember the synthetic nonce."}]}} -{"timestamp": "2026-08-17T13:00:01Z", "type": "item_completed", "payload": {"type": "item_completed", "item": {"type": "AgentMessage"}}} -{"timestamp": "2026-08-17T13:00:01Z", "type": "token_usage_record", "payload": {"type": "token_usage_record", "input_tokens": 10, "output_tokens": 5}} -{"timestamp": "2026-08-17T13:00:02Z", "type": "response_item", "payload": {"type": "function_call", "name": "shell", "arguments": "{\"command\":\"pwd\"}", "call_id": "call_fixture_1"}} -{"timestamp": "2026-08-17T13:00:03Z", "type": "response_item", "payload": {"type": "function_call_output", "call_id": "call_fixture_1", "output": [{"type": "input_text", "text": "/work"}, {"type": "input_image", "image_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="}]}} -{"timestamp": "2026-08-17T13:00:04Z", "type": "compacted", "payload": {"message": "Synthetic summary: remember nonce BETA-2048 and the completed shell result."}} -{"timestamp": "2026-08-17T13:00:05Z", "type": "event_msg", "payload": {"type": "user_message", "message": "Continue after the synthetic compaction."}} -{"timestamp": "2026-08-17T13:00:05Z", "type": "response_item", "payload": {"type": "message", "role": "user", "content": [{"type": "input_text", "text": "Continue after the synthetic compaction."}]}} -{"timestamp": "2026-08-17T13:00:05Z", "type": "item_completed", "payload": {"type": "item_completed", "item": {"type": "UserMessage"}}} -{"timestamp": "2026-08-17T13:00:05Z", "type": "token_usage_record", "payload": {"type": "token_usage_record", "input_tokens": 10, "output_tokens": 5}} -{"timestamp": "2026-08-17T13:00:06Z", "type": "event_msg", "payload": {"type": "agent_message", "message": "The synthetic post-compaction fixture is complete."}} -{"timestamp": "2026-08-17T13:00:06Z", "type": "response_item", "payload": {"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "The synthetic post-compaction fixture is complete."}]}} -{"timestamp": "2026-08-17T13:00:06Z", "type": "item_completed", "payload": {"type": "item_completed", "item": {"type": "AgentMessage"}}} -{"timestamp": "2026-08-17T13:00:06Z", "type": "token_usage_record", "payload": {"type": "token_usage_record", "input_tokens": 10, "output_tokens": 5}} +{"timestamp":"2026-08-17T13:00:00Z","ordinal":0,"type":"session_meta","payload":{"session_id":"30000000-0000-4000-8000-000000000000","id":"30000000-0000-4000-8000-000000000000","timestamp":"2026-08-17T13:00:00Z","cwd":"/work","originator":"fixture","cli_version":"0.153.4","source":"cli","model_provider":"openai","history_mode":"paginated"}} +{"timestamp":"2026-08-17T13:00:00Z","ordinal":1,"type":"response_item","payload":{"type":"message","role":"user","content":[{"type":"input_text","text":"INTERNAL ENVIRONMENT CONTEXT THAT MUST NOT BE REPLAYED"}]}} +{"timestamp":"2026-08-17T13:00:00Z","ordinal":2,"type":"response_item","payload":{"type":"message","role":"user","content":[{"type":"input_text","text":"Remember synthetic migrator nonce BETA-2048."},{"type":"input_image","image_url":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="}]}} +{"timestamp":"2026-08-17T13:00:00Z","ordinal":3,"type":"event_msg","payload":{"type":"item_completed","item":{"type":"UserMessage","id":"10000000-0000-4000-8000-000000000001","client_id":"10000000-0000-4000-8000-000000000002","content":[{"type":"text","text":"Remember synthetic migrator nonce BETA-2048.","text_elements":[]},{"type":"image","image_url":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="}]}}} +{"timestamp":"2026-08-17T13:00:00Z","ordinal":4,"type":"token_usage_record","payload":{"response_id":"resp_fixture_1","root_turn_id":"10000000-0000-4000-8000-000000000010","session_id":"30000000-0000-4000-8000-000000000000","thread_id":"30000000-0000-4000-8000-000000000000","turn_id":"10000000-0000-4000-8000-000000000010","turn_token_usage":{"input_tokens":10,"output_tokens":5},"thread_token_usage":{"input_tokens":10,"output_tokens":5},"usage":{"input_tokens":10,"output_tokens":5}}} +{"timestamp":"2026-08-17T13:00:01Z","ordinal":5,"type":"response_item","payload":{"type":"message","role":"assistant","content":[{"type":"output_text","text":"I will remember the synthetic nonce."}]}} +{"timestamp":"2026-08-17T13:00:01Z","ordinal":6,"type":"event_msg","payload":{"type":"item_completed","item":{"type":"AgentMessage","id":"msg_fixture_1","content":[{"type":"Text","text":"I will remember the synthetic nonce."}],"phase":"final_answer"}}} +{"timestamp":"2026-08-17T13:00:02Z","ordinal":7,"type":"response_item","payload":{"type":"function_call","name":"shell","arguments":"{\"command\":\"pwd\"}","call_id":"call_fixture_1"}} +{"timestamp":"2026-08-17T13:00:03Z","ordinal":8,"type":"response_item","payload":{"type":"function_call_output","call_id":"call_fixture_1","output":[{"type":"input_text","text":"/work"},{"type":"input_image","image_url":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="}]}} +{"timestamp":"2026-08-17T13:00:04Z","ordinal":9,"type":"compacted","payload":{"message":"Synthetic summary: remember nonce BETA-2048 and the completed shell result."}} +{"timestamp":"2026-08-17T13:00:05Z","ordinal":10,"type":"response_item","payload":{"type":"message","role":"user","content":[{"type":"input_text","text":"Continue after the synthetic compaction."}]}} +{"timestamp":"2026-08-17T13:00:05Z","ordinal":11,"type":"event_msg","payload":{"type":"item_completed","item":{"type":"UserMessage","id":"10000000-0000-4000-8000-000000000003","content":[{"type":"text","text":"Continue after the synthetic compaction.","text_elements":[]}]}}} +{"timestamp":"2026-08-17T13:00:06Z","ordinal":12,"type":"response_item","payload":{"type":"message","role":"assistant","content":[{"type":"output_text","text":"The synthetic post-compaction fixture is complete."}]}} +{"timestamp":"2026-08-17T13:00:06Z","ordinal":13,"type":"event_msg","payload":{"type":"item_completed","item":{"type":"AgentMessage","id":"msg_fixture_2","content":[{"type":"Text","text":"The synthetic post-compaction fixture is complete."}],"phase":"final_answer"}}} diff --git a/tests/test_catalog.py b/tests/test_catalog.py index 4763c86..03d019d 100644 --- a/tests/test_catalog.py +++ b/tests/test_catalog.py @@ -106,6 +106,24 @@ def _codex_records( }, } ) + if history_mode == "paginated": + records[0]["payload"]["cli_version"] = "0.153.4" # type: ignore[index] + records.append( + { + "timestamp": "2026-08-18T13:00:03Z", + "type": "event_msg", + "payload": { + "type": "item_completed", + "item": { + "type": "UserMessage", + "id": "10000000-0000-4000-8000-000000000001", + "content": [{"type": "text", "text": "not indexed", "text_elements": []}], + }, + }, + } + ) + for ordinal, record in enumerate(records): + record["ordinal"] = ordinal return records @@ -336,7 +354,7 @@ def test_codex_active_archive_paginated_and_native_titles(tmp_path: Path) -> Non with _catalog(tmp_path) as catalog: result = catalog.refresh(codex_roots=(home,), include_auto=False) - assert result.statuses == {"candidate": 2, "unsupported": 1} + assert result.statuses == {"candidate": 3} event_name = catalog.list_sessions(query="rollout event") assert len(event_name) == 1 assert event_name[0].title_kind == "thread_name" @@ -348,10 +366,10 @@ def test_codex_active_archive_paginated_and_native_titles(tmp_path: Path) -> Non assert catalog.list_sessions(query="forbidden preview") == [] assert catalog.list_sessions(query="forbidden first-message") == [] - unsupported = catalog.list_sessions(statuses=("unsupported",)) - assert len(unsupported) == 1 - assert unsupported[0].history_mode == "paginated" - assert unsupported[0].reason == "codex_history_mode" + paginated_entry = catalog.list_sessions(query=PAGINATED_ID) + assert len(paginated_entry) == 1 + assert paginated_entry[0].history_mode == "paginated" + assert paginated_entry[0].status == "candidate" # A transiently unavailable vendor cache must not erase title metadata # already derived from it when the authoritative JSONL changes. @@ -373,6 +391,58 @@ def test_codex_active_archive_paginated_and_native_titles(tmp_path: Path) -> Non assert retained_name[0].title == "Native saved name" +def test_codex_paginated_catalog_fails_closed_on_incomplete_projections( + tmp_path: Path, +) -> None: + home = tmp_path / "codex" + cases = { + "40000000-0000-4000-8000-000000000001": ("codex_history_base", "unsupported"), + "40000000-0000-4000-8000-000000000002": ( + "codex_subagent_history", + "unsupported", + ), + "40000000-0000-4000-8000-000000000003": ( + "codex_paginated_ordinals", + "corrupt", + ), + "40000000-0000-4000-8000-000000000004": ("codex_history_mode", "unsupported"), + "40000000-0000-4000-8000-000000000005": ( + "codex_history_mode_conflict", + "corrupt", + ), + } + for session_id, (reason, _) in cases.items(): + path = home / "sessions" / "2026" / "08" / "18" / f"rollout-synthetic-{session_id}.jsonl" + records = _codex_records(session_id, history_mode="paginated") + meta = records[0]["payload"] + assert isinstance(meta, dict) + if reason == "codex_history_base": + meta["history_base"] = { + "thread_id": CODEX_ID, + "end_ordinal_exclusive": 2, + "end_byte_offset": 512, + } + elif reason == "codex_subagent_history": + meta["subagent_history_start_ordinal"] = 2 + elif reason == "codex_paginated_ordinals": + records[1]["ordinal"] = 99 + elif reason == "codex_history_mode": + meta["history_mode"] = "future" + else: + duplicate = json.loads(json.dumps(records[0])) + duplicate["payload"]["history_mode"] = "legacy" + duplicate["ordinal"] = len(records) + records.append(duplicate) + _write_jsonl(path, records) + + with _catalog(tmp_path) as catalog: + catalog.refresh(codex_roots=(home,), include_auto=False) + entries = catalog.list_sessions(statuses=("unsupported", "corrupt"), include_paths=True) + + observed = {entry.session_id: (entry.reason, entry.status) for entry in entries} + assert observed == cases + + def test_refresh_is_incremental_and_validation_is_explicit(tmp_path: Path) -> None: home = tmp_path / "claude" session = home / "projects" / "-synthetic" / f"{CLAUDE_ID}.jsonl" diff --git a/tests/test_codex_paginated_history_mode.py b/tests/test_codex_paginated_history_mode.py index 68b042b..d4399a8 100644 --- a/tests/test_codex_paginated_history_mode.py +++ b/tests/test_codex_paginated_history_mode.py @@ -1,86 +1,151 @@ -"""Codex 0.147+ writes history_mode="paginated"; earlier releases wrote "legacy". - -The paginated layout keeps the model-visible items this adapter reads at the same -top level. It adds `item_completed` wrappers around items that are already -emitted as their own records, plus `token_usage_record` accounting — neither -carries model-visible content. These tests pin that equivalence so the adapter -cannot start depending on the wrappers, and pin that a forked session -(`history_base`) is still refused rather than silently truncated. -""" +"""Codex 0.147+ root paginated-history compatibility and safety tests.""" +import json from pathlib import Path import pytest +from session_migrate.conversion import ConversionOptions, convert_session from session_migrate.errors import SessionMigrateError from session_migrate.formats import codex +from session_migrate.model import EventKind, Role, TargetFormat FIXTURES = Path(__file__).parent / "fixtures" LEGACY = FIXTURES / "codex-0.144.4" / "basic.jsonl" PAGINATED = FIXTURES / "codex-0.153.4" / "paginated.jsonl" -def _visible(session): - """Role/text pairs a target format receives, excluding inert records. - - Paginated sessions carry `item_completed` and `token_usage_record` entries - that the adapter keeps as OPAQUE events — preserved for provenance, never - rendered into a target transcript. They are excluded here so this compares - the conversation itself. - """ - return [(event.role, event.text) for event in session.events if event.role is not None] - - -def test_paginated_history_mode_parses(): +def _portable(session): + return [ + ( + event.kind, + event.role, + event.text, + event.tool_name, + event.tool_call_id, + event.payload.get("block_type"), + event.payload.get("image_url"), + event.payload.get("content_blocks"), + ) + for event in session.events + if event.kind != EventKind.OPAQUE + ] + + +def _rewrite_fixture(tmp_path: Path, mutate) -> Path: + records = [json.loads(line) for line in PAGINATED.read_text().splitlines()] + mutate(records) + path = tmp_path / "paginated.jsonl" + path.write_text("".join(json.dumps(record) + "\n" for record in records)) + return path + + +def test_paginated_uses_canonical_completed_items_and_matches_legacy() -> None: session = codex.parse(PAGINATED) - assert session.cli_version == "0.153.4" - assert session.events, "paginated session produced no events" - -def test_paginated_matches_legacy_conversation(): - """The added wrappers must not change, duplicate, or drop visible turns.""" - assert _visible(codex.parse(PAGINATED)) == _visible(codex.parse(LEGACY)) + assert session.cli_version == "0.153.4" + assert _portable(session) == _portable(codex.parse(LEGACY)) + assert "INTERNAL ENVIRONMENT CONTEXT" not in " ".join( + event.text or "" for event in session.events + ) + assert [ + (event.role, event.text) for event in session.events if event.kind == EventKind.MESSAGE + ] == [ + (Role.USER, "Remember synthetic migrator nonce BETA-2048."), + (Role.ASSISTANT, "I will remember the synthetic nonce."), + (Role.USER, "Continue after the synthetic compaction."), + (Role.ASSISTANT, "The synthetic post-compaction fixture is complete."), + ] + + +def test_paginated_provider_messages_and_accounting_stay_opaque() -> None: + session = codex.parse(PAGINATED) + opaque = [event for event in session.events if event.kind == EventKind.OPAQUE] + + assert sum(event.payload.get("reason") == "paginated_provider_message" for event in opaque) == 5 + assert ( + sum(event.payload.get("source_record_type") == "token_usage_record" for event in opaque) + == 1 + ) + + +@pytest.mark.parametrize("target_format", tuple(TargetFormat)) +def test_paginated_context_cannot_leak_to_any_target( + tmp_path: Path, target_format: TargetFormat +) -> None: + artifact = convert_session( + codex.parse(PAGINATED), + ConversionOptions( + target_format=target_format, + session_id="50000000-0000-4000-8000-000000000001", + cwd=tmp_path, + ), + ) + + assert artifact.native_bytes + assert b"INTERNAL ENVIRONMENT CONTEXT" not in artifact.native_bytes + + +@pytest.mark.parametrize("ordinal", [None, True, "3", 99]) +def test_paginated_invalid_or_noncontiguous_ordinals_fail_closed( + tmp_path: Path, ordinal: object +) -> None: + def mutate(records): + if ordinal is None: + records[3].pop("ordinal") + else: + records[3]["ordinal"] = ordinal + + path = _rewrite_fixture(tmp_path, mutate) + with pytest.raises(SessionMigrateError, match="ordinal"): + codex.parse(path) -def test_paginated_extra_records_stay_opaque(): - """`item_completed` duplicates items already parsed; it must not become a turn. +def test_unknown_history_mode_still_refused(tmp_path: Path) -> None: + def mutate(records): + records[0]["payload"]["history_mode"] = "some-future-mode" - This is the real hazard in accepting the paginated layout: `item_completed` - wraps items that are ALSO emitted as their own `response_item` records, so - interpreting both would double every assistant message. - """ - session = codex.parse(PAGINATED) - extra = [event for event in session.events if event.role is None] - assert extra, "expected the paginated-only records to be retained" - assert {event.payload.get("source_record_type") for event in extra} == { - "item_completed", - "token_usage_record", - } - assert all(event.kind.value == "opaque" for event in extra) - - -def test_unknown_history_mode_still_refused(tmp_path): - """An unrecognized mode must fail closed, not be parsed hopefully.""" - lines = PAGINATED.read_text().splitlines() - lines[0] = lines[0].replace('"paginated"', '"some-future-mode"') - path = tmp_path / "future.jsonl" - path.write_text("\n".join(lines) + "\n") + path = _rewrite_fixture(tmp_path, mutate) with pytest.raises(SessionMigrateError, match="history mode"): codex.parse(path) -def test_history_base_still_refused(tmp_path): - """A fork's earlier turns live in another file; refusing beats truncating.""" - import json +def test_history_base_still_refused(tmp_path: Path) -> None: + def mutate(records): + records[0]["payload"]["history_base"] = { + "thread_id": "10000000-0000-4000-8000-000000000000", + "end_ordinal_exclusive": 12, + "end_byte_offset": 1024, + } - lines = PAGINATED.read_text().splitlines() - meta = json.loads(lines[0]) - meta["payload"]["history_base"] = { - "thread_id": "10000000-0000-4000-8000-000000000000", - "end_ordinal_exclusive": 12, - } - lines[0] = json.dumps(meta) - path = tmp_path / "forked.jsonl" - path.write_text("\n".join(lines) + "\n") + path = _rewrite_fixture(tmp_path, mutate) with pytest.raises(SessionMigrateError, match="history_base"): codex.parse(path) + + +def test_paginated_subagent_projection_still_refused(tmp_path: Path) -> None: + def mutate(records): + records[0]["payload"]["subagent_history_start_ordinal"] = 8 + + path = _rewrite_fixture(tmp_path, mutate) + with pytest.raises(SessionMigrateError, match="subagent history projection"): + codex.parse(path) + + +def test_paginated_metadata_must_be_first_and_consistent(tmp_path: Path) -> None: + def move_meta(records): + records[0], records[1] = records[1], records[0] + records[0]["ordinal"], records[1]["ordinal"] = 0, 1 + + with pytest.raises(SessionMigrateError, match="start with session metadata"): + codex.parse(_rewrite_fixture(tmp_path, move_meta)) + + def duplicate_conflicting_meta(records): + duplicate = json.loads(json.dumps(records[0])) + duplicate["payload"]["history_mode"] = "legacy" + records.append(duplicate) + for ordinal, record in enumerate(records): + record["ordinal"] = ordinal + + with pytest.raises(SessionMigrateError, match="conflicting history modes"): + codex.parse(_rewrite_fixture(tmp_path, duplicate_conflicting_meta)) From 0adbca045dfaf413c691b5219c530dedd513f4ad Mon Sep 17 00:00:00 2001 From: xhluca Date: Fri, 11 Sep 2026 15:41:18 -0400 Subject: [PATCH 3/3] docs: define safe Codex paginated boundary --- docs/architecture.md | 7 +++++-- docs/exploration-log.md | 36 +++++++++++++++++++++++++++++++-- docs/format-compatibility.md | 39 ++++++++++++++++++++++++------------ docs/session-catalog.md | 10 +++++---- docs/specification.md | 3 ++- docs/troubleshooting.md | 14 +++++++------ docs/validation-report.md | 28 +++++++++++++++++++++++--- 7 files changed, 106 insertions(+), 31 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 7dd9e4b..fb6f782 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -47,8 +47,11 @@ replaced, or truncated file fails with a retryable error. - Claude reconstructs the active UUID ancestry selected by `last-prompt`, validates compaction back-edges, and excludes inactive branches/meta prompts. -- Codex replays canonical legacy `response_item` history, deduplicates UI - projections, and rejects paginated/history-base lineage. +- Codex replays canonical legacy `response_item` history. For a complete root + paginated rollout it validates contiguous ordinals and takes visible turns + only from `event_msg.item_completed` TurnItems, never from contextual + provider messages. External `history_base` and subagent projections remain + fail-closed. - Pi follows the v3 `id`/`parentId` active tree and rejects unsupported schema versions. - OMP follows its v3 active tree after validating the fixed 256-byte title diff --git a/docs/exploration-log.md b/docs/exploration-log.md index b3b15a8..624590d 100644 --- a/docs/exploration-log.md +++ b/docs/exploration-log.md @@ -533,8 +533,8 @@ in [Mistral Vibe session format](vibe-format.md). - Repeat authenticated semantic recall when a supported target/provider version changes. - Add native fixtures for remote-URL images, branching, and schema drift when sanitized examples can be generated safely. -- Implement Codex paginated/history-base lineage only after ordinal, contextual - user, compaction, rollback, and inter-agent semantics are independently gated. +- Implement Codex `history_base` and paginated subagent projection only after + external-prefix, rollback, and inter-agent semantics are independently gated. - Re-run the pinned integration suite for every supported agent version/schema combination. @@ -717,3 +717,35 @@ indexes bounded native titles/IDs without bodies, and represents selection as validates its install bundle, for 324 ordered routes. Detailed contracts are in [Hermes](hermes-format.md), [MastraCode](mastracode-format.md), and [Devin](devin-format.md). + +## 2026-09-11: Codex 0.153 root paginated history + +PR #4 was reviewed against the official Codex `rust-v0.153.4` source rather +than its original synthetic assumption. `rollout/src/policy.rs` shows that +paginated history persists canonical `TurnItem`s as +`event_msg.item_completed`; legacy user/assistant UI events are deliberately +not persisted in that mode. `rollout/src/ordinal.rs` requires paginated +records to carry a monotonic ordinal, and `protocol/src/protocol.rs` defines +`history_base` as an external exclusive prefix and +`subagent_history_start_ordinal` as the boundary between inherited context and +the subagent's own projection. + +A content-free audit then examined 172 recent native paginated rollouts without +`history_base`: nine roots and 163 subagent projections. It inspected only +record/item types, structural fields, counts, and hashed content equality; no +message, tool, media, ID, or path value was printed or committed. Across the +nine roots, 75 canonical user items matched provider user messages and all 242 +provider assistant messages matched canonical assistant items. The provider +stream also contained 21 additional user-shaped messages and 53 developer +messages that had no canonical completed turn. Replaying every +`response_item.message`, as the initial PR did, would therefore turn internal +context into visible user history. + +The corrected reader makes completed `UserMessage`/`AgentMessage` TurnItems the +only paginated source of conversation turns, retains non-message response items +for portable tool/reasoning data, and records ignored provider messages as +opaque losses. It accepts only self-contained roots with integer ordinals +contiguous from zero. `history_base`, subagent projections, unknown modes, and +damaged ordinal streams fail closed. The sanitized 0.153.4 fixture mirrors the +official nested item shape and includes an unmatched contextual provider +message as a regression sentinel. diff --git a/docs/format-compatibility.md b/docs/format-compatibility.md index 8ee8c0a..25e070d 100644 --- a/docs/format-compatibility.md +++ b/docs/format-compatibility.md @@ -275,13 +275,24 @@ ordered `response_item` envelopes. Text messages use `type: "message"` with `function_call_output`; call arguments are a JSON-encoded string, and both records share `call_id`. -`response_item` is the canonical, model-visible history. `event_msg` records -drive list preview and UI display. The writer emits both for text messages so -the text is visible to the resumed model and to the interface. The reader -deduplicates UI messages against response-item messages. It uses UI events as a -legacy fallback when a rollout has no canonical messages. In a mixed partial -rollout, exact normalized duplicates are removed; unmatched UI projections are -retained as marked messages and reported in the manifest. +For legacy history, `response_item` is the canonical, model-visible history and +`event_msg` records drive list preview and UI display. The writer emits both +for text messages so the text is visible to the resumed model and to the +interface. The reader deduplicates UI messages against response-item messages. +It uses UI events as a legacy fallback when a rollout has no canonical +messages. In a mixed partial legacy rollout, exact normalized duplicates are +removed; unmatched UI projections are retained as marked messages and +reported in the manifest. + +Codex 0.147+ root paginated rollouts use a different authority: completed +`UserMessage` and `AgentMessage` TurnItems inside +`event_msg.item_completed`. The reader requires an integer ordinal on every +record, contiguous from zero. It ignores provider `response_item` messages as +conversation because those records can include synthetic environment and +developer context; non-message response items still supply portable tool and +reasoning data. Direct image/audio user inputs are retained when represented +by the canonical TurnItem. Local media paths and other TurnItem blocks remain +explicit opaque losses. Current legacy rollouts can contain `compacted.replacement_history`. Codex installs that array as the effective history at the checkpoint and replays only @@ -294,21 +305,22 @@ post-compaction context transfer. The paired `event_msg.context_compacted` UI notification is deduplicated against the checkpoint. Other observed envelopes include `compacted`, `turn_context`, `world_state`, -reasoning response items, inter-agent communication, and newer paginated or -fork-related state. Those records are not all portable conversation history. +reasoning response items, inter-agent communication, and fork-related state. +Those records are not all portable conversation history. ## Route support Every ordered pair among the eighteen formats is implemented, for 324 routes: -- full portable adapters: Claude, Codex legacy, Pi, OMP, OpenCode, Copilot, +- full portable adapters: Claude, Codex legacy and root paginated, Pi, OMP, OpenCode, Copilot, Antigravity, Vibe, Muse, Qwen, Kimi, Grok, Kilo, OpenHands, Hermes, MastraCode, and Devin; - experimental text-only adapter: Cursor. Same-format routes are portable rewrites into new sessions, not byte copies. -Codex paginated/history-base sources remain fail-closed. Cursor is experimental, -build-pinned, and deliberately transfers only ordered user/assistant text. The +Codex `history_base` lineage and paginated subagent projections remain +fail-closed. Cursor is experimental, build-pinned, and deliberately transfers +only ordered user/assistant text. The detailed table below explains the original Claude/Codex pair; target-specific behavior is documented in [Additional native formats](additional-target-formats.md) and [Muse/Qwen/Kimi](muse-qwen-kimi-formats.md). Grok, Kilo, and OpenHands @@ -348,7 +360,8 @@ Legend: | Inactive Claude branches | **Unsupported** | N/A | They become opaque events and are counted as dropped; no forks are created. | | Claude sidechains/subagents | **Unsupported** | N/A | The catalog indexes nested sidechains as unsupported, but direct lookup/conversion does not import them; transfer the parent session. | | Codex legacy linear history | N/A | **Supported** | Ordered response items become one linear Claude UUID graph. | -| Codex paginated history/forks | N/A | **Unsupported** | Non-legacy `history_mode` and `history_base` are rejected rather than risking an incomplete import. Replacement-history compaction uses the expanded-transcript policy above. | +| Codex root paginated history | N/A | **Supported** | Contiguous canonical completed TurnItems supply user/assistant turns; provider-context messages are not replayed. Replacement-history compaction uses the expanded-transcript policy above. | +| Codex paginated forks/subagents | N/A | **Unsupported** | `history_base` and `subagent_history_start_ordinal` require external or projected history and are rejected rather than silently truncating or duplicating a conversation. | | Codex UI-only messages | N/A | **Lossy fallback** | Used as the conversation when no response-item messages exist. In a mixed partial file, exact normalized duplicates are removed and unmatched projections are retained with `message:ui_only_projection`; fuzzy matching is never used. | | Turn context, policies, world state, snapshots | **Unsupported** | **Unsupported** | Codex `turn_context` is counted as context; `world_state` and `security_risk_score` become counted opaque events. Shell snapshots, approvals, external credential stores, MCP state, memories, goals, and configuration are outside transcript conversion. | | Unknown source records/blocks | **Unsupported** | **Unsupported** | They become content-free opaque/sentinel events where recognized and are counted at write time, including unknown nested tool-result blocks. | diff --git a/docs/session-catalog.md b/docs/session-catalog.md index 525667b..244bf8b 100644 --- a/docs/session-catalog.md +++ b/docs/session-catalog.md @@ -177,9 +177,11 @@ This keeps refresh work proportional to the small inventory table, not the total transcript corpus. Consequently, archived sessions, duplicate UUIDs, nested sidechains/subagents, -malformed files, and absent Copilot/Cursor native stores remain discoverable. Claude -sidechains and Codex paginated/history-base sessions are listed as -`unsupported`; listing them does not make them convertible. +malformed files, and absent Copilot/Cursor native stores remain discoverable. +Complete root Codex paginated rollouts are candidates. Claude sidechains, +Codex `history_base` lineage, and paginated subagent projections are listed as +`unsupported`; listing them does not make them convertible. A paginated root +with missing or non-contiguous ordinals is `corrupt`. ## Quick start @@ -320,7 +322,7 @@ registered roots. | --- | --- | | `candidate` | Fast structural metadata scan passed; full conversion has not been requested. OpenCode and Kilo rows remain candidates until their one-session official export is parsed. | | `validated` | The exact stat identity was fully parsed, dry-converted, and target-validated during `refresh --validate`. | -| `unsupported` | The file is a recognized session type intentionally rejected by conversion, such as a Claude sidechain or Codex paginated/history-base rollout. | +| `unsupported` | The file is a recognized session type intentionally rejected by conversion, such as a Claude sidechain, Codex `history_base` lineage, or a Codex paginated subagent projection. | | `corrupt` | JSONL, SQLite/protobuf, native structure, or explicit conversion validation failed. | | `oversized` | The source exceeds the migrator's bounded input limits. | | `busy` | The source changed while it was being scanned; retry after the native CLI finishes appending. | diff --git a/docs/specification.md b/docs/specification.md index 5f893df..72303c5 100644 --- a/docs/specification.md +++ b/docs/specification.md @@ -30,7 +30,8 @@ creates a new independent target session; it does not move/delete the source, clone runtime state byte-for-byte, synchronize future turns, or re-execute historical tools. -Codex paginated/history-base lineage and Claude sidechain import remain +Complete root Codex paginated rollouts are transferable. Codex `history_base` +lineage, paginated subagent projections, and Claude sidechain import remain non-transferable. They are discoverable in the catalog and fail closed. ## Functional requirements diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 4c585d8..a3e3bdc 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -261,12 +261,14 @@ separately authenticated Devin installation; session-migrate never copies credentials. See the [Hermes](hermes-format.md), [MastraCode](mastracode-format.md), and [Devin](devin-format.md) format notes. -## Codex paginated or history-base source - -These lineage modes are recognized but unsupported. `--format codex` cannot -bypass the guard. The safe root-paginated subset still needs ordinal, -contextual-user, compaction/rollback/inter-agent, and lineage semantics before -it can be enabled. +## Codex paginated source is rejected + +A complete root paginated rollout is supported when every record has an integer +ordinal contiguous from zero. Missing/gapped ordinals are treated as corrupt. +`history_base` and `subagent_history_start_ordinal` mean the file depends on an +external or projected prefix and remain unsupported; `--format codex` cannot +bypass either guard. Resume or export the root session instead of flattening +the dependent file by hand. ## Claude sidechain/subagent diff --git a/docs/validation-report.md b/docs/validation-report.md index abb4439..9b6d36f 100644 --- a/docs/validation-report.md +++ b/docs/validation-report.md @@ -1292,11 +1292,33 @@ is committed. Default CI performs the frozen parser and 324 conversion cases without network access or model spend; live gates remain explicit release commands. +### 2026-09-11 Codex paginated-source review + +The root paginated reader was checked against the official Codex 0.153.4 +protocol, rollout persistence policy, and ordinal implementation. A +content-free structural audit over nine recent root rollouts found 81 canonical +user items, 244 canonical assistant items, 21 provider-only user-shaped +messages, and 53 provider developer messages. Only counts and content hashes +were compared; private values were never printed or added to fixtures. + +The regression fixture now uses real `event_msg.item_completed` nesting, +canonical `UserMessage`/`AgentMessage` shapes, integer ordinals, and the current +token-usage envelope. Its provider stream deliberately contains an unmatched +context message. Tests prove that canonical text and image history matches the +legacy portable projection, the context sentinel is never replayed, damaged +ordinals fail closed, and external/history-projection metadata remains +unsupported. Catalog tests independently pin candidate, corrupt, and +unsupported classification. All nine audited native roots were parsed and +converted through each of the eighteen target byte validators: 162/162 local, +content-safe conversions passed without writing artifacts or invoking a model. + ## Known boundaries -- Codex paginated history and `history_base` lineage remain fail-closed until - their effective-history and fork semantics can be reproduced and native - tested without relying on derived SQLite state. +- Complete root Codex paginated history is supported from canonical completed + TurnItems after strict ordinal validation. `history_base` lineage and + paginated subagent projections remain fail-closed until their external or + projected prefix semantics can be reproduced without relying on derived + SQLite state. - Provider-encrypted Codex replacement-history state cannot be translated to Claude. The migrator retains visible expanded history and reports the semantic difference.