test(audit): make ChatGPT anchor census reproducible (#3759) - #3759
Closed
Sinity wants to merge 3 commits into
Closed
test(audit): make ChatGPT anchor census reproducible (#3759)#3759Sinity wants to merge 3 commits into
Sinity wants to merge 3 commits into
Conversation
Problem: the polylogue-uqwd blocker required a fresh production-route check against the current quarantined ChatGPT population before any comparison-layer exception could be considered. What changed: add the read-only classifier evidence packet, index it in the generated docs surface, and add a moved-anchor red twin proving different lifecycle content remains a conflict. Compatibility/migration: no production behavior, archive database, Beads state, daemon state, or services changed. The packet records the current source/index identities, cohort coverage, historical cohort outcomes, and residual uncertainty.
📝 WalkthroughWalkthroughAdded a read-only ChatGPT lifecycle-anchor audit command. It parses quarantined revisions through production routes, classifies revision relationships, counts target conflicts, emits deterministic sanitized receipts, registers the command, and documents the audit evidence. ChangesChatGPT lifecycle-anchor audit
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant AuditCLI
participant ArchiveDatabases
participant ChatGPTParser
participant RevisionClassifier
Operator->>AuditCLI: Run lifecycle-anchor audit
AuditCLI->>ArchiveDatabases: Read quarantined memberships and accepted heads
AuditCLI->>ChatGPTParser: Parse selected ChatGPT revisions
ChatGPTParser-->>AuditCLI: Return revision projections
AuditCLI->>RevisionClassifier: Compare and classify revisions
RevisionClassifier-->>AuditCLI: Return relation and cohort outcomes
AuditCLI-->>Operator: Print deterministic sanitized receipt
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Problem: the PR's current-corpus conclusion depended on an untracked report, and its direct ParsedSession test did not exercise the historical parser ordering defect. What changed: add a read-only production-route census command, a real SQLite archive regression, an end-to-end parser-to-classifier ordering test, generated command reference, and a packet that preserves graph gates pending a historical replay fixture. Compatibility/migration: no archive, blob, daemon, service, or Beads state is changed. Co-Authored-By: Claude <noreply@anthropic.com>
Problem: the current-corpus census needed a reviewable artifact tied to the committed production-route audit command. What changed: record the sanitized aggregate receipt with its SQL selection, predicate, denominators, source and index provenance, and explicit no-de-gating limit. Compatibility/migration: the receipt is evidence only. It performs no archive or service mutation. Co-Authored-By: Claude <noreply@anthropic.com>
Sinity
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@devtools/chatgpt_lifecycle_anchor_audit.py`:
- Around line 93-108: Update _matches_target to require exactly one
generation_lifecycle event per session rather than exactly one total event,
while allowing other lifecycle events. Compare event content after removing
source_message_provider_id instead of comparing complete hashed content, so
differing anchors with identical non-anchor content match; add a regression test
covering that positive case and preventing a false zero target_pair_count.
In `@tests/unit/devtools/test_chatgpt_lifecycle_anchor_audit.py`:
- Around line 99-131: Add a target-positive audit fixture with equal non-anchor
lifecycle content and different moved anchors, reusing the existing archive/test
fixture patterns. Update
test_audit_runs_the_parser_to_classifier_route_read_only_and_is_sanitized or add
a focused test to assert outcomes["pair_relation_counts"]["conflict"] equals 1
and outcomes["target_pair_count"] equals 1, while preserving the existing
read-only and sanitization assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 053d6421-0411-4729-95a9-542139e83f40
📒 Files selected for processing (11)
devtools/chatgpt_lifecycle_anchor_audit.pydevtools/command_catalog.pydevtools/docs_surface.pydocs/README.mddocs/audits/2026-08-04-polylogue-uqwd-chatgpt-lifecycle-anchor-receipt.jsondocs/audits/2026-08-04-polylogue-uqwd-chatgpt-lifecycle-anchor.mddocs/audits/README.mddocs/devtools.mdtests/unit/archive/test_session_revision_membership.pytests/unit/devtools/test_chatgpt_lifecycle_anchor_audit.pytests/unit/sources/test_parsers_chatgpt.py
Comment on lines
+93
to
+108
| def _matches_target(left: _ParsedMember, right: _ParsedMember, relation: _Relation) -> bool: | ||
| if len(left.session.session_events) != 1 or len(right.session.session_events) != 1: | ||
| return False | ||
| left_event, right_event = left.session.session_events[0], right.session.session_events[0] | ||
| left_projection = left.revision.projection | ||
| right_projection = right.revision.projection | ||
| return ( | ||
| left_event.event_type == right_event.event_type == "generation_lifecycle" | ||
| and left_event.source_message_provider_id != right_event.source_message_provider_id | ||
| and left_projection.message_contents == right_projection.message_contents | ||
| and left_projection.attachment_contents == right_projection.attachment_contents | ||
| and left_projection.event_contents != right_projection.event_contents | ||
| and {content for _, content in left_projection.event_contents} | ||
| == {content for _, content in right_projection.event_contents} | ||
| and relation == "conflict" | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Implement the declared target predicate.
Line 94 requires exactly one total session event. The predicate requires exactly one generation_lifecycle event and permits other lifecycle events.
Lines 105-106 also compare complete event content hashes. polylogue/pipeline/ids.py Lines 591-686 hash a content_payload that includes source_message_provider_id. Different anchors therefore produce different hashes, so the intended moved-anchor pair cannot pass this condition.
Count generation events explicitly. Compare lifecycle content after excluding the anchor field. Add a positive regression test with different anchors and equal non-anchor content. Otherwise, the audit can report a false zero target_pair_count.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@devtools/chatgpt_lifecycle_anchor_audit.py` around lines 93 - 108, Update
_matches_target to require exactly one generation_lifecycle event per session
rather than exactly one total event, while allowing other lifecycle events.
Compare event content after removing source_message_provider_id instead of
comparing complete hashed content, so differing anchors with identical
non-anchor content match; add a regression test covering that positive case and
preventing a false zero target_pair_count.
Comment on lines
+99
to
+131
| def test_audit_runs_the_parser_to_classifier_route_read_only_and_is_sanitized(tmp_path: Path) -> None: | ||
| root = _archive_with_ordered_exports(tmp_path) | ||
| source_before = (root / "source.db").read_bytes() | ||
| index_before = (root / "index.db").read_bytes() | ||
|
|
||
| receipt = run_audit(root) | ||
|
|
||
| assert receipt["schema"] == SCHEMA | ||
| assert receipt["target_predicate"] == TARGET_PREDICATE | ||
| assert receipt["denominators"] == { | ||
| "selected_quarantined_chatgpt_raw_count": 2, | ||
| "selected_membership_row_count": 2, | ||
| "membershipless_selected_raw_count": 0, | ||
| "logical_source_key_count": 1, | ||
| "singleton_cohort_count": 0, | ||
| "multi_candidate_cohort_count": 1, | ||
| "raws_in_multi_candidate_cohorts": 2, | ||
| "parsed_and_projected_raw_count": 2, | ||
| } | ||
| outcomes = cast(dict[str, object], receipt["outcomes"]) | ||
| assert cast(dict[str, int], outcomes["pair_relation_counts"]) == { | ||
| "equal": 1, | ||
| "a_contains_b": 0, | ||
| "b_contains_a": 0, | ||
| "conflict": 0, | ||
| } | ||
| assert outcomes["target_pair_count"] == 0 | ||
| rendered = json.dumps(receipt, sort_keys=True) | ||
| assert "raw-left" not in rendered | ||
| assert "raw-right" not in rendered | ||
| assert "/redacted/chatgpt-export.json" not in rendered | ||
| assert (root / "source.db").read_bytes() == source_before | ||
| assert (root / "index.db").read_bytes() == index_before |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Add a target-positive census fixture.
This test only verifies a zero result for equal projections. It cannot detect a regression where _matches_target never recognizes a valid moved-anchor conflict. The classifier fixture in tests/unit/archive/test_session_revision_membership.py also changes state, but the target predicate requires equal non-anchor lifecycle content.
Add a fixture with equal lifecycle content and different anchors. Assert that run_audit() reports one conflict and target_pair_count == 1.
🧰 Tools
🪛 ast-grep (0.45.0)
[info] 125-125: use jsonify instead of json.dumps for JSON output
Context: json.dumps(receipt, sort_keys=True)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/unit/devtools/test_chatgpt_lifecycle_anchor_audit.py` around lines 99 -
131, Add a target-positive audit fixture with equal non-anchor lifecycle content
and different moved anchors, reusing the existing archive/test fixture patterns.
Update test_audit_runs_the_parser_to_classifier_route_read_only_and_is_sanitized
or add a focused test to assert outcomes["pair_relation_counts"]["conflict"]
equals 1 and outcomes["target_pair_count"] equals 1, while preserving the
existing read-only and sanitization assertions.
Owner
Author
|
This branch is superseded by PR #3835, which carries the audit onto current master and includes the two substantive review fixes plus a positive target fixture. The published branch is preserved without force-push. |
Owner
Author
|
Closed as superseded by PR #3835; the repaired current-master stack is the merge target. |
Sinity
added a commit
that referenced
this pull request
Aug 6, 2026
## Summary Make the ChatGPT lifecycle-anchor audit reproducible against current master and make its target predicate detect the historical moved-anchor conflict shape. ## Problem The earlier audit branch had diverged from current master and its target predicate could report zero when sessions contained other lifecycle events or when the only difference was the anchor message reference. That left the audit unable to prove the incident shape required by the reindex evidence gate. ## Solution The synchronized branch preserves the read-only audit route and current archive provenance checks. It requires exactly one generation_lifecycle event per session, permits unrelated lifecycle events, compares lifecycle content after removing the anchor reference, and adds a positive parser-to-classifier fixture that asserts one conflict pair and one target match. The branch also contains the current-master merge so the audit uses the same semantics as the reindex campaign. ## Verification - `devtools test tests/unit/devtools/test_chatgpt_lifecycle_anchor_audit.py tests/unit/sources/test_parsers_chatgpt.py tests/unit/archive/test_session_revision_membership.py`: 179 passed. - Pre-push quick verification: all 24 steps passed. - The audit remains read-only and emits sanitized provenance receipts. Ref polylogue-0qfy and Ref #3759. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Codex <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the ChatGPT lifecycle-anchor census reproducible and adds the missing parser-to-classifier regression.
Problem
The prior conclusion came from an untracked report, and its added test constructed ParsedSession values after parsing. It therefore could not detect the historical mapping-order tie-break that moved a generation_lifecycle anchor and caused the real membership classifier to conflict.
Solution
Adds
devtools workspace chatgpt-lifecycle-anchor-audit, which opens source.db and index.db withmode=ro, reads blobs, and routes each selected raw through_parse_one,session_revision_projection,_relation, andclassify_membership_revisions. Its committed receipt contains the exact SQL selection, predicate, aggregate denominators, source/index provenance, and no raw ids, native ids, paths, hashes, titles, or payloads.Adds an end-to-end mapping-order regression. It passes two otherwise identical ChatGPT exports through the parser and classifier, asserting one accepted raw and one equivalent raw. The identical scenario reproduced
node_bversusnode_aanchors, a conflict, and ambiguous raws against the parent of parser fixb1e01d878.The existing direct ParsedSession test remains as a classifier-only different-content guard. No classifier production behavior, archive state, graph state, Beads state, daemon state, or service state changes.
Evidence
The committed receipt records 7,498 selected quarantined ChatGPT raws, 2,570 persisted cohorts, 2,555 multi-candidate cohorts, 71 genuine conflict pairs, and
target_pair_count: 0. The target predicate requires exactly one lifecycle event per side, different anchors, equal transcript and attachment content, equal non-anchor lifecycle content, and a conflict relation.This current-parser snapshot does not establish the required pre-fix historical replay fixture. The
uqwd,xselt, and818fygraph gates remain unchanged.Verification
devtools test tests/unit/devtools/test_chatgpt_lifecycle_anchor_audit.py tests/unit/sources/test_parsers_chatgpt.py tests/unit/archive/test_session_revision_membership.py:172 passed.devtools verify --quick: all checks passed in89.79sduring pre-push.devtools workspace chatgpt-lifecycle-anchor-audit --archive-root /realm/db/polylogue --receipt docs/audits/2026-08-04-polylogue-uqwd-chatgpt-lifecycle-anchor-receipt.json: committed sanitized receipt withtarget_pair_count: 0.Acceptance criteria
Summary by CodeRabbit