Skip to content

Use valid repeated-measures inference for conditional IC - #62

Merged
stefan-jansen merged 1 commit into
mainfrom
codex/conditional-ic-significance
Sep 9, 2026
Merged

Use valid repeated-measures inference for conditional IC#62
stefan-jansen merged 1 commit into
mainfrom
codex/conditional-ic-significance

Conversation

@stefan-jansen

Copy link
Copy Markdown
Contributor

Closes #59

Summary

  • stop returning a fabricated significance p-value for ungrouped inputs that provide only one pooled IC per quantile
  • compute matched per-date IC observations for panel inputs and compare conditioning quantiles with a Friedman repeated-measures test
  • preserve descriptive IC range and variation when inference is unavailable

Reproduction and verification

  • before the fix, independent arrays produced 40 p-values below 0.05 across 100 deterministic seeds
  • uv run pytest tests/test_evaluation/test_conditional_ic.py -q -n 0 (29 passed)
  • uv run ty check
  • pre-commit run --all-files
  • uv run pytest tests/ -q -n auto --timeout 120 (5342 passed, 75 skipped)

Copilot AI lite review requested due to automatic review settings September 9, 2026 05:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@stefan-jansen
stefan-jansen merged commit 16553d0 into main Sep 9, 2026
36 checks passed
@stefan-jansen
stefan-jansen deleted the codex/conditional-ic-significance branch September 9, 2026 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace invalid conditional-IC significance calculation

2 participants