Design (D.1): shape-descriptor comparability — the 160 cells decompose into five mechanisms, only 15 of which are a real design question

[squid-protocol]

Batch D.1 of #2669. Design item — no code changed, and none should be until this is settled.

Measured against the post-Batch-C corpus (keyword-rosetta 3d94ca17, engine aacfe5da), not the
pre-A figures #2669's item text was written from.

The premise was wrong in a useful way

#2669 D.1 frames this as one family — "shape-descriptor invariance (avg_func_complexity,
control_flow_ratio, cog_raw, gini, internal density, structural_mass — 166 cells)" — with three
options that would apply to all of it: normalise against planted branch count, exclude as documented
design, or ledger as morphology.

The 160 out-of-band cells in that family today are five different mechanisms, and they want
four different answers. Only the last bucket is the design question the item was asking about.

cells	mechanism
18	A — the language has zero functions, so every per-function descriptor is undefined
30	B — control_flow_ratio, a ratio against a denominator the corpus never controls
92	C — derived from an input metric that is itself out of band
5	D — the slicer segmented a different number of functions than were planted
15	E — residual: the actual open question

A — 18 cells: markdown and html have no functions

Both report functions_found = 0, so all nine per-function descriptors are not "deviant", they are
undefined. docs/GATING.md already has the right precedent one layer over: a cell is n/a
because the rule is None. The same reasoning applies to a per-function average when there are no
functions.

Recommendation: report as n/a, not as a red cell. Corpus tooling only — no engine change, no
rebless. This is the cheapest 18 cells on the board.

B — 30 cells: control_flow_ratio divides by something the corpus never plants

total_control_flow_ratio = branch_hits / max(branch_hits + linear_hits, 1)   # detector.py:1288

linear_hits is structural_boundaries. No manifest in the corpus asserts structural_boundaries
anywhere, and the bias report does not include it among its 57 metrics — so the denominator of the
single largest out-of-band metric is neither planted, gated, nor displayed.

Measured per language (struct_linear, whole-corpus scan):

language	branch	structural_boundaries	control_flow_ratio
solidity	3	45	0.063
ada	4	40	0.091
python	3	31	0.088
javascript	3	19	0.136
makefile	4	4	0.500
assembly	24	17	0.585

The numerator is pinned at 3–4 by design across all 46 languages. The entire spread is the
denominator, which ranges 0–45.

This kills option 1 for this metric: "normalise against planted branch count" cannot help, because
branch is already controlled. The corpus simply cannot validate control_flow_ratio today.

Recommendation: plant and gate structural_boundaries in the corpus, or mark
control_flow_ratio incomparable until that happens. Prefer the former — the signal is real and
every language has the rule; it is the corpus that is silent, not the engine. Either way the bias
report should display the denominator, since a ratio cell is unexplainable without it.

C — 92 cells: composites of already-deviating inputs

cog_raw's formula (signal_processor.py ~L800):

effective_loc = min(loc, (branches + args + 1) * 10)
sum_function_impacts = ((branches + 1) * sqrt(args + 1) + 0.05 * effective_loc) * 10

Its inputs are branch, args and LOC — and args morphology and LOC normalisation are already
validated ledger entries (args-no-parameter-surface-morphology from Batch A.4, and the tech-debt
LOC artifact noted in #2653). 15 of cog_raw's 26 out-of-band languages are also out of band in one
of its own inputs; across the whole family, 92 cells are this shape.

Recommendation: not a new deviation — mark derived. This is what E.1 was specified to do (a cell
counts as "explained" when a validated ledger entry names that metric + language); E.1 just needs to
follow the derivation edge from a composite to its inputs. No engine change and no new ledger
entries — the entries already exist, the tooling does not yet connect them.

D — 5 cells: the slicer found a different number of functions

sqlite 30, dockerfile 17, livecode/lua/matlab/ruby/shell 16, makefile 14, against 13
planted everywhere else. Per-function averages move because the divisor moved.

Recommendation: ledger as morphology, per-language, naming the segmentation cause.

E — 15 cells: the real design question

Everything above is bookkeeping. This is not:

cog_raw          c, dart, go, java, javascript, kotlin, php, python, rust, typescript
structural_mass  c, embedded_python, python, scala, typescript

These are mainstream procedural languages whose every input is in band — branch, args and LOC
all green — yet the composite lands out of band. That points at non-linearity in the formula itself:
the (branches + 1) * sqrt(args + 1) product and the min(loc, (signals + 1) * 10) clamp amplify
differences too small to flag on their own.

Options:

1. Accept and document — the composite is supposed to be non-linear (that is what makes it a
   cognitive-load proxy rather than a count), so a spread on identical intent may be correct
   behaviour to ledger rather than flatten.
2. Re-band composites on a wider tolerance — derived metrics get a band accounting for the
   amplification of their inputs, instead of the flat 25%/50% used for raw counts.
3. Decompose in the report — show cog_raw beside its three inputs, so a reader can see whether
   a deviation entered through an input or through the formula.

Recommendation: 1 + 3. Document the non-linearity as intended and make the inputs visible.
Option 2 is a threshold change that would mask real regressions later.

One hypothesis I tested and rejected

It is natural to suspect the median mixes incomparable populations — 46 languages, ~19 of them
config/DSL shapes — so that "deviation from median" flags the healthy procedural languages. This is
false. Recomputing each metric's median over procedural languages only:

metric	median (all 46)	median (procedural only)	procedural cells out of band: all-median → procedural-median
cog_raw	0.055	0.050	14 → 14
avg_func_complexity	0.188	0.188	5 → 5
func_internal_density	0.034	0.030	5 → 6

The two populations agree on the centre almost exactly. Recorded so nobody re-proposes
population-splitting as a fix.

Adjacent decisions this pass folds in

• risk formulas: audit language-tier membership (embedded_python lands tier3) + decide how rosetta bias reporting treats documented Fc/Irc tier constants #2653 (tier constants) — a separate mechanism: _get_tier assigns fc/irc constants by
  literal string membership, so ~32 tier3 languages carry an automatic risk floor and ~30% less
  documentation credit for byte-identical content. Should be decided on its own; the concrete
  sub-item (embedded_python is not the literal string "python", so it lands tier3 despite being
  python) looks like a plain bug worth fixing regardless of the wider policy question.
• matlab: mandatory 'out = x' return-convention assignments count as state_mutation (+450% vs median) — intended? #2654 (matlab return-convention assignments) — the issue already states its resolution path:
  affirm current behaviour and record a validated return-assignment-counted-as-mutation entry.
  Nothing here changes that.
• state-flux ×3 branch-context weighting: undocumented, and phantom-triggered by branch keywords inside string literals #2546 (×3 flux) is closed; its documented half is the deliberate weighting and the FP half
  went with the literal-shielding fix.

What I am asking for

Approval per bucket, before any edit — D.2 does not start until this is settled:

• A → report n/a for per-function descriptors when functions_found == 0 (corpus tooling)
• B → plant + gate structural_boundaries in the corpus, and show it in the report
• C → E.1 follows derivation edges; no new entries, no engine change
• D → per-language morphology ledger entries
• E → document the non-linearity as intended + decompose composites in the report

A, C and D are corpus/tooling only. B is corpus work. E is the only one that touches
interpretation of engine behaviour, and none of the five requires an engine code change — which is
itself the main conclusion: the shape-descriptor family is largely a reporting problem, not an
engine bias.

Child of #2669 (Batch D.1), epic #2560.

[squid-protocol]

Correction to bucket B — my own recommendation above is wrong, and the fix I proposed would not have worked.

I recommended "plant and gate structural_boundaries in the corpus" to make control_flow_ratio comparable. Then I read the rule. It is not a structural count at all — it is a per-language keyword-vocabulary tally, and the vocabularies differ in kind, not just in content:

language	what structural_boundaries matches
ada	procedure|function|package|begin|end|return — 6 structural words
python	def|class|return|import|from|as|pass|continue|break|await|assert|del|global|nonlocal|type — 16
solidity	the above kind plus every type name: uint\d*, int\d*, bytes, address, mapping …
assembly	every instruction mnemonic: mov, lea, push, pop, add, sub, ldr, str …

Measured on the corpus's own a.* files, where the planted intent is identical:

solidity   10 hits   uint x6, return x3, import x1      <- 6 of 10 are a TYPE NAME
python      9 hits   def x3, return x3, from, import, assert
ada         9 hits   procedure x3, begin x3, end x3

Every typed declaration in solidity scores a hit that its python equivalent cannot; every instruction in assembly does the same. The denominator is a property of the language's vocabulary, not of the planted logic — so no amount of planting equalises it. Gating it would only make an incomparable number asserted rather than unknown, and the cells would stay red for exactly the same reason.

Revised recommendation for B: mark control_flow_ratio cross-language-incomparable in the rosetta report (the same treatment n/a cells get: shown, excluded from bands and medians), with a validated ledger entry recording the vocabulary-tally cause.

Note what this does not say: the metric is not broken, and nothing in the engine should change. control_flow_ratio is used to compare files within one repository (security_auditor.py reads it that way), where the vocabulary is held constant and the ratio means what it claims. It is only the cross-language comparison — which is all this corpus does — that the denominator makes meaningless.

That is 21 of the 224 currently-unexplained cells, resolved by a ledger entry and a report change rather than by corpus work that would not have helped.

[squid-protocol]

Closing: the plan is approved and executed

Every bucket has shipped. Mapping each to where it landed, since the answers were spread across four PRs:

bucket	cells	outcome
A — per-function descriptors for a language with no functions	12	keyword-rosetta#40 — the undefined verdict
B — control_flow_ratio's uncontrolled denominator	21	keyword-rosetta#41 — ledger entry; recommendation reversed, see below
C — composites of already-deviating inputs	88	keyword-rosetta#40 — the derived verdict, table corrected in keyword-rosetta#44
D — slicer segmentation	5	fixed in the engine: gitgalaxy#2691, merged in #2693
E — composite non-linearity	16	keyword-rosetta#41 — affirmed as intended, no engine change

Shape family: 0 unexplained cells. Corpus-wide 444 → 175.

One recommendation in this issue was wrong, and is withdrawn

Bucket B proposed planting and gating structural_boundaries so the ratio would become comparable. Reading the rule showed that cannot work: it is a per-language keyword vocabulary tally, not a structural count. ada matches six structural words, python sixteen, solidity those plus every type name, assembly every instruction mnemonic — and on the corpus's own a.sol, six of solidity's ten hits are the word uint. No planting equalises that. Recorded as incomparable instead, with the metric left untouched in the engine, where within-repo comparison is its real job.

Two engine bugs came out of verifying rather than asserting

Bucket D was filed expecting "morphology". Checking it produced gitgalaxy#2691 (the slicer's __global_context__ bucket counted as a function in five languages) and, from the residual cells, gitgalaxy#2692 (ada counting one typed parameter as three). Both are now fixed and merged in #2693, and the corpus followed in keyword-rosetta#44.

And the follow-through found a defect in the tooling this issue motivated

Retiring #2691's ledger entry resurfaced three cells, which exposed that E.1's DERIVED_INPUTS table omitted state_mutation, concurrency, reflection_metaprogramming and api — inputs _calc_cog_load and file_mass actually read. matlab's composites are inherited from its state_mutation at +1150% (#2654's shape), not new findings. Corrected in keyword-rosetta#44.

What this issue concluded, and it held

The shape-descriptor family was a reporting problem, not engine bias — and no line of scoring code changed. The two engine fixes that did land were extraction defects the reporting work uncovered, not the scoring questions the issue was opened to settle.

The median-mixes-populations hypothesis stays tested and rejected (procedural-only medians move counts by 0–1 cells), recorded so it is not re-proposed.