Skip to content

Correct 023: the embedding perturbation explanation is refuted - #120

Merged
GiulioDER merged 1 commit into
masterfrom
claude/023-correct-perturbation
Oct 6, 2026
Merged

GiulioDER merged 1 commit into
masterfrom
claude/023-correct-perturbation

Conversation

@GiulioDER

Copy link
Copy Markdown
Owner

Summary

Appends a dated correction to preregistration/023-corpus-lineage-tiers.md, below the frozen marker. Nothing above it changes, and no number in the existing result is edited.

The result section explained the T0 to T1 move (5 harmed to 2) as a retrieval perturbation: writing valid_from into all 4,911 documents was said to change every embedding. That explanation is wrong.

  • Code: recall at 492501c6 (the last master commit before the tiers were built) indexes .md as text/markdown. It chunks the body with the frontmatter removed and moves the keys into metadata. The embedded context prefix reads only a title: key.
  • Run records: the tier generations have been garbage collected, but the session streams survive.
    • Same pipeline fingerprint in all three tiers.
    • Bracketed thresholds of about 0.530 to 0.534 in every tier.
    • The three queries issued verbatim in both T0 and T2 returned the same ranked sources, with scores equal to four decimals in two of them.
    • None of T0's 76 distinct queries recurs in T1.

Consequences recorded in the correction

  • T0 to T1 is most plausibly run variance.
  • The 75% / 25% decomposition and the rescoring of Prediction 4 against T1 are withdrawn.
  • The defensible summary is T0 against T2: 4 cells of 30, p = 0.1945, not attributable.

Test plan

  • tests/test_public_tree.py, tests/test_no_host_inventory.py and tests/test_prereg.py pass (38 passed).
  • scripts/check_public_tree.py on the changed file: clean. The private-term check did not run because RECALL_PUBLIC_TREE_DENYLIST is not set in this shell.
  • This is a documentation-only change; no behaviour test applies.

🤖 Generated with Claude Code

…ords refute

The result section attributed the T0 to T1 move to frontmatter changing every
embedding. recall at 492501c6 strips Markdown frontmatter before chunking, and the
surviving session streams show identical queries scoring identically across tiers
with matching thresholds. The correction is appended below the frozen marker; no
prediction or number above it is edited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GiulioDER
GiulioDER merged commit c1b1572 into master Oct 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant