Skip to content

Fix measurement: outages, denominators, cache identity (SLOP-1 to SLOP-3); 0.1.1 - #2

Merged
rxslice merged 2 commits into
mainfrom
fix/audit-measurement
Nov 14, 2026
Merged

rxslice merged 2 commits into
mainfrom
fix/audit-measurement

Conversation

@rxslice

@rxslice rxslice commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Fixes the three slopbench findings from the 13 November portfolio audit. tests/test_audit_measurement.py fails on the previous code for every reported defect and passes now.

SLOP-1 (P1): a registry outage was counted as an invented package

Problem: pkgguard reports an unreachable registry as exists=False with REVIEW, but slopbench looked only at exists=False and recorded the name as NOT_FOUND. A timeout, a 429 or a 503 therefore became evidence of an invented package.

Fix: only a BLOCK-confirmed absence is NOT_FOUND. Any other exists=False is UNKNOWN and excluded from the counts. A confirmed 404 still counts.

SLOP-2 (P2): unverifiable references diluted the rate

Problem: unverifiable references were left out of the numerator but kept in the denominator. On the audit's fixture (one invented name and one unverifiable name, each seen twice) slopbench reported 50%; the correct rate is 100%.

Fix:

  • occurrence_rate and every per-tier rate are now calculated over verified references only.
  • Result gains unverifiable_occurrences and verified_occurrences, and the JSON output adds verification_coverage.
  • A tier where nothing could be verified reports its rate as None, shown as n/a in reports, never as a measured zero.
  • The HTML and Markdown reports show how many references the registry verified. The note saying unverifiable names are "excluded from every count" is now actually true.

SLOP-3 (P2): different endpoints shared cache entries

Problem: the cache key ignored which server answered, so changing SLOPBENCH_BASE_URL or OLLAMA_HOST returned another server's cached completions.

Fix: the key includes the endpoint URL whenever it differs from the provider's default. Only the scheme, host, port and path are used, never usernames, passwords, query strings or credentials. Caches written for default setups therefore stay valid, and each cache file records its endpoint.

Version 0.1.1

The 0.1.0 package on PyPI has these bugs. pyproject.toml now reads its version from slopbench.__version__ (0.1.1), so a release tag can't disagree with the package that gets built. After merging, create a GitHub release tagged v0.1.1 to publish it.

Verification

  • pytest: 76 passed.
  • The package build and twine check --strict pass.
  • The October HALLUX panel figures in reports/ are unaffected: they were computed by the HALLUX pipeline, not this code path.

Generated by Claude Code

….1.1

From the 2026-11-13 audit. tests/test_audit_measurement.py fails on the
previous code for every reported defect and passes now.

SLOP-1: pkgguard reports an unreachable registry as exists=False with
REVIEW. slopbench read exists=False alone as NOT_FOUND, so a timeout, a
429 or a 503 became an invented package. Only a BLOCK-confirmed absence is
NOT_FOUND now; any other exists=False is UNKNOWN and excluded. A confirmed
404 still counts.

SLOP-2: unverifiable references were dropped from the numerator but kept
in the denominator, so the audit fixture (one invented, one unverifiable,
each twice) reported 50% instead of 100%. occurrence_rate and every tier
rate are now over verified references. Result gains
unverifiable_occurrences and verified_occurrences; to_dict adds
verification_coverage. A tier with nothing verifiable reports rate None,
rendered n/a, never a measured zero. Reports show how many references the
registry verified, and the "excluded from every count" note is now true.

SLOP-3: the completion cache key ignored the endpoint, so changing
SLOPBENCH_BASE_URL or OLLAMA_HOST reused another server's completions.
The key now includes the normalized endpoint URL (scheme, host, port, path;
never userinfo, query or credentials) whenever it differs from the
provider's default, so caches written for default setups stay valid. Each
cache file records its endpoint.

Version 0.1.1, now single-sourced: pyproject reads slopbench.__version__,
so a release tag can no longer disagree with the built package.

pytest: 76 passed. Build and twine --strict pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011GFY8v4QeVYzawpSLcJCzz
CI installs slopbench without the optional pkgguard extra, so the SLOP-1
tests failed on import. They now drive the adapter with a stand-in shaped
like pkgguard's Assessment (REVIEW/exists=False for an outage, BLOCK for a
confirmed absence), as the existing pkgguard tests do. A separate test runs
against the real pkgguard when it is installed and skips otherwise.

Without pkgguard: 75 passed, 2 skipped; the new tests still fail on the
previous code for every reported defect (8 of 8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011GFY8v4QeVYzawpSLcJCzz
@rxslice
rxslice merged commit a5d1059 into main Nov 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants