Skip to content

risk formulas: replace language tiers with a measured fidelity table + a language-strictness table (child of #2560) #2716

Description

@squid-protocol

Design pass for #2653, approved 2026-09-04. Full write-up (measurements, retractions, the three-input model): the "Language Confidence Tiers" memo, reproduced in summary here so the issue stands alone.

Where we are

keyword-rosetta shows identical §2 raw signals producing different §3 risk scores by language. Most of that is signal_processor._get_tier: two inline hand lists pick fc / irc / ot per language, and 45 of 59 defined languages fall through to the harshest setting (fc 0.60 / irc 5 / ot 1.40).

Measured on language-crucible (2,649 scored files) by scanning once as committed and once with TIER_VARS flattened — baseline reproduces golden_master_zero_dep_audit.json with zero diffs:

  • 78.3% of files change; 18.2% by ≥ 25 points; 11.0% by ≥ 50. Tech Debt +19.5 mean on the 31.7% it moves (max 54.9); Documentation +14.7 on 21.2% (max 81.5); Testing max +77.6.
  • On rosetta with content held identical: risk_documentation +87% and risk_tech_debt +56% tier 1 → tier 3.
  • embedded_python is tier 3 (not the literal "python"): boot.py +77.5 Testing, bldc_driver.py +55.4 Documentation. Mean doc movement +26.6 vs python's +1.0.
  • The heaviest-corrected languages are declarative: makefile +29.3 debt, jcl +29.1, m4 +14.9, html +9.7 doc. They have no runtime.

Why one dial can't be made right

fc / irc / ot is one three-valued lookup doing three separable jobs:

job truth lives in today
our rules catch less in some languages (fidelity) rosetta measures it per language × signal guessed by bucket; 21 of 45 tier-3 languages have full defence-rule coverage and rosetta catches their plants, yet lose 40% of credit
this file does runtime-dynamic things a regex can't follow the file — high_risk_execution, reflection_metaprogramming, eval are already signals flat irc by language; 80% of python files and 84% of groovy files contain zero dynamic execution and pay it anyway
this language lets you leave things unsaid (types, errors, memory, globals) the language spec two undocumented inline sets + a dead FIDELITY_TIERS that disagrees with them for ~20 languages

Rosetta cannot validate any §3 correction — its null hypothesis is "no correction" — so it corrects the measurement layer only. That is the reason this drifted unnoticed.

The plan

Same equations. The constants they read come from different places.

  1. fc becomes a measured, per-signal fidelity coefficient generated from rosetta: fc(lang, signal) = min(1, planted / measured). Over-firing rules get their credit scaled to the planted truth; under-firing is a to-fix list, not a coefficient (a coefficient that compensates for a fixable gap stops the fixing). Ledgered n/a → 1.0. Retires systems_buffer_ratio, whose fc < 1.0 gate is the cause of the safety sign inversion.
  2. irc / ot become a documented language-strictness table in analysis_lens: objective yes/no columns (static types · enforced errors · memory safe · no implicit globals), one row per language, family map so embedded_pythonpython. No-runtime languages get None → no language-level term. Weight small and bounded (UEF small-file normalization isn't achieving its intent — redesign scoring so identical intent scores identically regardless of file size (design pass for Fable) #2655's invariant stays: corrects measured risk, never creates it) until the outcome pilot says otherwise. Deletes FIDELITY_TIERS.
  3. Per-file dynamism replaces the flat irc in the six formulas that read it, one formula at a time, each with its own crucible before/after.
  4. Outcome pilot: the crucible is snapshot copies (PROVENANCE.json, 218 repos pinned by SHA, no .git), so clone ~10 upstream at their pins with history, classify fix commits, and ask whether strictness / dynamism predict later fixes at equal evidence. Decides how much weight (2) may ever carry.
  5. Docs: 08-03 rewritten as a chain — what a regex can't see → the three inputs and their sources → the fixture that scores it → the UEF small-file normalization isn't achieving its intent — redesign scoring so identical intent scores identically regardless of file size (design pass for Fable) #2655 invariant — with per-language tables generated, never typed.

What "done" looks like: identical §2 gives near-identical §3 across languages; the only language-level offset left is the small documented strictness term, which rosetta prints rather than subtracts out with the F.3 within-tier band.

Children

Related: #2653 (narrowed to the membership audit; closed by PR 1), #2669 (rosetta epic), #2655 (evidence-mass floor and the invariant), #2705.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    core-engineModifications to the central physics and parsing engineepicLarge-scale architectural milestone or multi-issue initiativemetricsHeuristics, risk exposures, and topological math updates

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions