Skip to content

test: run the suite from the sdist and in isolation; fix rolling trust docs - #365

Merged
kevincostner17 merged 1 commit into
mainfrom
chore/tests-sdist-docs
Sep 15, 2026
Merged

kevincostner17 merged 1 commit into
mainfrom
chore/tests-sdist-docs

Conversation

@kevincostner17

Copy link
Copy Markdown
Contributor

Summary

  • tests/test_no_network_in_runtime.py fails when run on its own (ssl import vs socket patch) #347 tests/test_no_network_in_runtime.py: the autouse fixture replaced socket.socket with a function. fd.clean(semantic_mode="auto") / fd.learn lazily import the model registry → urllib.request → ssl, and ssl subclasses socket.socket, so the file failed with TypeError whenever it ran before anything imported ssl (alone, -k, sharding, random order). The fixture now monkeypatches socket.socket.connect, socket.socket.connect_ex, socket.create_connection and socket.getaddrinfo, blocking any network attempt while leaving the class intact.
  • sdist ships tests/ without benchmarks/, training/ and scripts/: 37 collection errors #348 tests/conftest.py: the sdist ships tests/ without benchmarks/, training/ and scripts/, so collection from an unpacked sdist stopped with ModuleNotFoundError. A pytest_ignore_collect hook checks each directory independently and, only when it is missing, ignores the suites that depend on it (tests/benchmark, tests/performance, tests/truthbench, tests/training, and test modules importing benchmarks, bench, harness_metrics, public_benchmark, results_schema, training, dataset_loader, fixture_download, fetch_online_fixtures, expectations, golden_util). Nothing is ignored in a checkout; pyproject.toml is unchanged.
  • docs/streaming.md calls rolling_trust_score row-weighted; it is a plain batch mean #349 docs/streaming.md: rolling_trust_score is an unweighted mean of the last rolling_trust_window per-batch trust scores (only cumulative_trust_score is row-weighted); the docs row now says so.

Tests

  • pytest tests/test_no_network_in_runtime.py on its own: 4 passed (was 3 failed, 1 passed).
  • Repo copy without benchmarks/, training/, scripts/: pytest --collect-only -q tests → 2843 collected, 0 errors (was 28 errors), on pytest 9.1.1 and 8.4.2. Without scripts/ only: 3669 collected, 0 errors.

Verification

  • ruff check .: clean
  • pytest -m "not online and not large": py3.12 4239 passed; py3.9 / pandas 1.5 4235 passed

Follow-up (not in this PR): from the sdist a few tests still fail at run time because they read benchmarks/ files inside test bodies (e.g. test_execution/test_benchmarks.py, test_streaming_cli.py benchmark generator tests).

Closes #347
Closes #348
Closes #349

…lling trust docs

tests/test_no_network_in_runtime.py: the autouse fixture replaced
socket.socket with a plain function. The clean/learn tests lazily import
the model registry, which imports urllib.request and therefore ssl, and
ssl subclasses socket.socket, so the import raised TypeError whenever the
file ran before anything else had imported ssl (running the file alone,
-k selections, sharding, random order). The fixture now patches
socket.socket.connect, socket.socket.connect_ex, socket.create_connection
and socket.getaddrinfo through monkeypatch, which still fails any network
attempt but leaves the socket class intact.

tests/conftest.py: the sdist ships tests/ without the repo-only benchmarks/,
training/ and scripts/ directories, so collection from an unpacked sdist
stopped with ModuleNotFoundError errors. A pytest_ignore_collect hook now
checks each directory independently and, only when it is missing, ignores
the suites that depend on it: tests/benchmark, tests/performance and
tests/truthbench for benchmarks/, tests/training for training/, and any
test module importing a module that lives there (benchmarks, bench,
harness_metrics, public_benchmark, results_schema; training;
dataset_loader, fixture_download, fetch_online_fixtures, plus the
expectations and golden_util helpers that import them). In a checkout
nothing is ignored. pyproject.toml is unchanged.

docs/streaming.md: rolling_trust_score was described as row-weighted, but
StreamingState takes an unweighted mean of the last rolling_trust_window
batch scores; only cumulative_trust_score is row-weighted. The docs row
now says so.

Closes #347
Closes #348
Closes #349
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: a26ec8cb-ebfa-49af-a3c5-a1a65ff4ca12


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kevincostner17
kevincostner17 merged commit f4d22bd into main Sep 15, 2026
18 checks passed
kevincostner17 added a commit that referenced this pull request Sep 15, 2026
Add Unreleased Added, Changed and Fixed entries for merged PRs #351,
#352, #353, #354, #356, #357, #358, #360, #361, #362, #363, #364, #365,
#367, #368, #369, #370, #371, #372, #374, #375, #377, #378 and #379.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant