Repository navigation
test: run the suite from the sdist and in isolation; fix rolling trust docs - #365
Merged
Merged
Conversation
…lling trust docs tests/test_no_network_in_runtime.py: the autouse fixture replaced socket.socket with a plain function. The clean/learn tests lazily import the model registry, which imports urllib.request and therefore ssl, and ssl subclasses socket.socket, so the import raised TypeError whenever the file ran before anything else had imported ssl (running the file alone, -k selections, sharding, random order). The fixture now patches socket.socket.connect, socket.socket.connect_ex, socket.create_connection and socket.getaddrinfo through monkeypatch, which still fails any network attempt but leaves the socket class intact. tests/conftest.py: the sdist ships tests/ without the repo-only benchmarks/, training/ and scripts/ directories, so collection from an unpacked sdist stopped with ModuleNotFoundError errors. A pytest_ignore_collect hook now checks each directory independently and, only when it is missing, ignores the suites that depend on it: tests/benchmark, tests/performance and tests/truthbench for benchmarks/, tests/training for training/, and any test module importing a module that lives there (benchmarks, bench, harness_metrics, public_benchmark, results_schema; training; dataset_loader, fixture_download, fetch_online_fixtures, plus the expectations and golden_util helpers that import them). In a checkout nothing is ignored. pyproject.toml is unchanged. docs/streaming.md: rolling_trust_score was described as row-weighted, but StreamingState takes an unweighted mean of the last rolling_trust_window batch scores; only cumulative_trust_score is row-weighted. The docs row now says so. Closes #347 Closes #348 Closes #349
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tests/test_no_network_in_runtime.py: the autouse fixture replacedsocket.socketwith a function.fd.clean(semantic_mode="auto")/fd.learnlazily import the model registry →urllib.request→ssl, andsslsubclassessocket.socket, so the file failed withTypeErrorwhenever it ran before anything importedssl(alone,-k, sharding, random order). The fixture now monkeypatchessocket.socket.connect,socket.socket.connect_ex,socket.create_connectionandsocket.getaddrinfo, blocking any network attempt while leaving the class intact.tests/conftest.py: the sdist shipstests/withoutbenchmarks/,training/andscripts/, so collection from an unpacked sdist stopped withModuleNotFoundError. Apytest_ignore_collecthook checks each directory independently and, only when it is missing, ignores the suites that depend on it (tests/benchmark,tests/performance,tests/truthbench,tests/training, and test modules importingbenchmarks,bench,harness_metrics,public_benchmark,results_schema,training,dataset_loader,fixture_download,fetch_online_fixtures,expectations,golden_util). Nothing is ignored in a checkout;pyproject.tomlis unchanged.docs/streaming.md:rolling_trust_scoreis an unweighted mean of the lastrolling_trust_windowper-batch trust scores (onlycumulative_trust_scoreis row-weighted); the docs row now says so.Tests
pytest tests/test_no_network_in_runtime.pyon its own: 4 passed (was 3 failed, 1 passed).benchmarks/,training/,scripts/:pytest --collect-only -q tests→ 2843 collected, 0 errors (was 28 errors), on pytest 9.1.1 and 8.4.2. Withoutscripts/only: 3669 collected, 0 errors.Verification
ruff check .: cleanpytest -m "not online and not large": py3.12 4239 passed; py3.9 / pandas 1.5 4235 passedFollow-up (not in this PR): from the sdist a few tests still fail at run time because they read
benchmarks/files inside test bodies (e.g.test_execution/test_benchmarks.py,test_streaming_cli.pybenchmark generator tests).Closes #347
Closes #348
Closes #349