Repository navigation
Fix measurement: outages, denominators, cache identity (SLOP-1 to SLOP-3); 0.1.1 - #2
Merged
Merged
Conversation
….1.1 From the 2026-11-13 audit. tests/test_audit_measurement.py fails on the previous code for every reported defect and passes now. SLOP-1: pkgguard reports an unreachable registry as exists=False with REVIEW. slopbench read exists=False alone as NOT_FOUND, so a timeout, a 429 or a 503 became an invented package. Only a BLOCK-confirmed absence is NOT_FOUND now; any other exists=False is UNKNOWN and excluded. A confirmed 404 still counts. SLOP-2: unverifiable references were dropped from the numerator but kept in the denominator, so the audit fixture (one invented, one unverifiable, each twice) reported 50% instead of 100%. occurrence_rate and every tier rate are now over verified references. Result gains unverifiable_occurrences and verified_occurrences; to_dict adds verification_coverage. A tier with nothing verifiable reports rate None, rendered n/a, never a measured zero. Reports show how many references the registry verified, and the "excluded from every count" note is now true. SLOP-3: the completion cache key ignored the endpoint, so changing SLOPBENCH_BASE_URL or OLLAMA_HOST reused another server's completions. The key now includes the normalized endpoint URL (scheme, host, port, path; never userinfo, query or credentials) whenever it differs from the provider's default, so caches written for default setups stay valid. Each cache file records its endpoint. Version 0.1.1, now single-sourced: pyproject reads slopbench.__version__, so a release tag can no longer disagree with the built package. pytest: 76 passed. Build and twine --strict pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011GFY8v4QeVYzawpSLcJCzz
CI installs slopbench without the optional pkgguard extra, so the SLOP-1 tests failed on import. They now drive the adapter with a stand-in shaped like pkgguard's Assessment (REVIEW/exists=False for an outage, BLOCK for a confirmed absence), as the existing pkgguard tests do. A separate test runs against the real pkgguard when it is installed and skips otherwise. Without pkgguard: 75 passed, 2 skipped; the new tests still fail on the previous code for every reported defect (8 of 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011GFY8v4QeVYzawpSLcJCzz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the three slopbench findings from the 13 November portfolio audit.
tests/test_audit_measurement.pyfails on the previous code for every reported defect and passes now.SLOP-1 (P1): a registry outage was counted as an invented package
Problem: pkgguard reports an unreachable registry as
exists=Falsewith REVIEW, but slopbench looked only atexists=Falseand recorded the name asNOT_FOUND. A timeout, a 429 or a 503 therefore became evidence of an invented package.Fix: only a BLOCK-confirmed absence is
NOT_FOUND. Any otherexists=FalseisUNKNOWNand excluded from the counts. A confirmed 404 still counts.SLOP-2 (P2): unverifiable references diluted the rate
Problem: unverifiable references were left out of the numerator but kept in the denominator. On the audit's fixture (one invented name and one unverifiable name, each seen twice) slopbench reported 50%; the correct rate is 100%.
Fix:
occurrence_rateand every per-tier rate are now calculated over verified references only.Resultgainsunverifiable_occurrencesandverified_occurrences, and the JSON output addsverification_coverage.None, shown as n/a in reports, never as a measured zero.SLOP-3 (P2): different endpoints shared cache entries
Problem: the cache key ignored which server answered, so changing
SLOPBENCH_BASE_URLorOLLAMA_HOSTreturned another server's cached completions.Fix: the key includes the endpoint URL whenever it differs from the provider's default. Only the scheme, host, port and path are used, never usernames, passwords, query strings or credentials. Caches written for default setups therefore stay valid, and each cache file records its endpoint.
Version 0.1.1
The 0.1.0 package on PyPI has these bugs.
pyproject.tomlnow reads its version fromslopbench.__version__(0.1.1), so a release tag can't disagree with the package that gets built. After merging, create a GitHub release taggedv0.1.1to publish it.Verification
twine check --strictpass.reports/are unaffected: they were computed by the HALLUX pipeline, not this code path.Generated by Claude Code