Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/wiki/03-02-claim-2-explicitness.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ Before the age of global travel, some isolated languages lacked words for concep

To ensure comparative fairness, we must normalize the "Material Properties" of the languages. We do not judge a Shell script for being "worse" than Go; we acknowledge that it is built from a more opaque material.

* **The Steel Bridge (Go - Tier 1):** The structure is explicit. We can clearly see the bolts and the tension cables. If there are no visible cracks, we can be 99% sure the bridge is safe.
* **The Stone Bridge (Shell - Tier 3):** The internal integrity is hidden inside the masonry. A visual inspection might show zero cracks, but the material might still be hiding faults.
* **The Steel Bridge (Go — one strictness gap):** The structure is explicit. We can clearly see the bolts and the tension cables. If there are no visible cracks, we can be 99% sure the bridge is safe.
* **The Stone Bridge (Shell — three strictness gaps):** The internal integrity is hidden inside the masonry. A visual inspection might show zero cracks, but the material might still be hiding faults.

To make the comparison fair, GitGalaxy applies a Fidelity Tax (the Fidelity Coefficient $Fc$) and an Implicit Risk Correction ($Irc$). We assume a baseline level of hidden risk to correct for the opacity of the material. This ensures that a "Safe" rating in Shell requires significantly more defensive effort than in Go, reflecting the reality of the engineering challenge.
To make the comparison fair, GitGalaxy applies two separate corrections (#2716). The Fidelity Coefficient $Fc$ is *measured*: keyword-rosetta plants identical defence in every language, and a rule that over-fires has each of its hits credited at the planted-to-measured ratio, so the same defence earns the same credit everywhere. The Implicit Risk Correction $Irc$ is *tabulated*: one point per thing the language lets you leave unsaid — static types, enforced error paths, memory safety, declared globals — from `analysis_lens.LANGUAGE_STRICTNESS`. A "Safe" rating in Shell still requires more defensive effort than in Go, but the amount is now written down per column rather than picked by bucket, and [08-03](08-03-transforming-regex-counts.md) renders both tables from the data the engine reads.

### 2.4.1.C. The Fidelity Matrix (40 Languages x 51 Signals)

Expand All @@ -33,7 +33,7 @@ This matrix maps the structural "Broadcast Power" of coding languages across his
* 🟧 **I (Implicit):** The concept exists, but must be inferred from context, conventions, or secondary logic.
* ⚪ **- (None):** The concept is not structurally applicable to the language.

| Language (Year) | Tier | CF | Phys | Risk | Domain | Thermo |
| Language (Year) | Tier (2025 buckets, retired by #2718 — see [08-03](08-03-transforming-regex-counts.md) for the live strictness rows) | CF | Phys | Risk | Domain | Thermo |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| MLIR (2019) | 1 | 🟦 **E** | 🟦 **E** | 🟦 **E** | ⚪ **-** | 🟦 **E** |
| Zig (2016) | 1 | 🟦 **E** | 🟦 **E** | 🟦 **E** | 🟦 **E** | 🟦 **E** |
Expand Down Expand Up @@ -100,7 +100,7 @@ By trading the microscope of an AST for the telescope of blAST, we accept a 5% m

### 2.4.1.D. Conclusion: Design for Scannability

The progression from Tier 3 (Implicit) to Tier 1 (Explicit) suggests that Scannability, the trending towards explicitness is increase. The younger the language, the louder it "screams" its intent. The blAST engine attempts normalize this variation in explicitiness by applying the Fidelity Coefficient ($Fc$) to account for the silence of the ancients/the fog of war/the opaqueness of the language.
The progression from implicit to explicit suggests that scannability — the trend towards explicitness is increasing. The younger the language, the louder it "screams" its intent. The blAST engine normalises what it can *measure* of that variation with the per-signal Fidelity Coefficient ($Fc$), and writes down what it cannot measure — the silence of the ancients — as the strictness table's $Irc$, one point per thing the language leaves unsaid.

<br><br>

Expand Down
103 changes: 99 additions & 4 deletions docs/wiki/08-03-transforming-regex-counts.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,104 @@ Applies four stabilizing principles:
5. **Evidence-Mass Floor:** Every per-file density divides by $\max(LOC, 50)$, never by raw LOC. Below 50 coding lines a file is scored on its *counts* (a two-hit file is a two-hit file whether it is 5 or 49 lines long), so identical intent scores identically regardless of file length; at or above the floor the density regime is untouched. This is the per-file analog of the mass-weighted averaging used at directory scope, and it is the *only* small-file mechanism -- it replaced six independent guards (a `<15 LOC` cognitive-load cliff, two `+20` paddings, a `loc/15` dampener, an unbounded $Irc/LOC$ floor, a `max(total_loc, 10)` API guard) that each fired on a different LOC range and fought each other (#2655). Files below the floor carry `mass_floored: true` and their `evidence_mass` in telemetry. Two consistency rules follow from it: $Irc$ *corrects* measured risk and never creates it (zero measured evidence scores zero in every tier), and a file with no branches carries no cognitive load at any length.
The per-function descriptor has the same floor, derived the same way (#2705): `func_internal_density` $= \text{avg\_func\_complexity} / \max(\text{avg\_func\_loc},\ 12)$, where 12 is the golden-master median lines-per-function exactly as 50 is the median coding LOC per file. Below the floor the column is a pure rescale of `avg_func_complexity` -- it reports branch structure, and two files with the same branches per function read the same density however tersely one of them is written; at or above it the column is unchanged. That means for roughly half of real files the two columns say the same thing, by design.

Language Confidence Tiers (1 to 3) apply Fidelity Coefficients ($Fc$) and Implicit Risk Corrections ($Irc$) based on language strictness.
General Risk Equation:
$$RiskExposure = \left( \frac{((RiskHits + Irc) \times Weight) - (DefenseHits \times Fc)}{\max(LOC,\ 50)} \right) \times Mp$$
$$RiskExposure = \left( \frac{((RiskHits + Irc) \times Weight) - \sum_{s}(DefenseHits_s \times Fc_s)}{\max(LOC,\ 50)} \right) \times Mp$$

### Language corrections: three inputs, not one dial (#2716)

A regex sees different amounts in different languages, so a raw count is not comparable across them and something has to correct for it. Until #2718 that "something" was one lookup on the language's name: two inline hand lists picked $Fc$ / $Irc$ / $Ot$ per language, with 45 of 59 defined languages falling through to the harshest setting. Measured on the language-crucible corpus that single term moved 78% of files and 18% of them by 25 points or more; it inverted the safety ordering above 6 attack hits (#2717); and `embedded_python` paid a tier-3 penalty for not being the literal string `"python"` (#2653). The lookup was doing three separable jobs, and each now has its own input sourced from where the truth about it lives:

| what it corrects for | where the truth lives | the input | replaces |
|---|---|---|---|
| **Our rules catch less in some languages.** A `safety` rule that fires 3 times on 2 planted constructs over-credits every hit. | Measured. keyword-rosetta plants identical defence in every language and records what each rule found. | $Fc_s = \min(1,\ planted_s / measured_s)$ per language **and per signal**, generated into `gitgalaxy/standards/fidelity_table.py`. Under-firing stays at 1.0 — that is a rule to fix, and a coefficient that compensates for a fixable gap is a way to stop fixing it. | the single per-language $Fc$, and `_calc_safety`'s `systems_buffer_ratio` |
| **This language lets you leave things unsaid.** Unenforced error paths, implicit globals, no memory safety, no static types. | The language specification — objective yes/no columns, checkable against documentation. | `analysis_lens.LANGUAGE_STRICTNESS`: one row per language; $Irc$ = number of gaps (0–4), $Ot = 1 + 0.1 \cdot Irc$. Data, markup and configuration formats have no runtime and carry **no** term. Dialects resolve through `LANGUAGE_FAMILY`. | the tier lists |
| **This file does runtime-dynamic things a regex cannot follow.** `eval`, reflection, dynamic dispatch. | The file — these are already registry signals, counted per file. | Per-file dynamism replaces the flat $Irc$ term in the six equations that read it (#2719, in progress). | the flat $Irc$ |

Two invariants bound the mechanism. $Irc$ *corrects* measured risk and never creates it: zero measured evidence scores zero in every language (#2655). And the weight on the strictness term is provisional — nothing in the system can validate a language-level shift yet (keyword-rosetta's null hypothesis *is* "no correction", the crucible has no risk labels), so #2720 pilots an outcome fixture before that weight is trusted with more than it carries now.

The tables the engine actually reads are rendered below from the data files, never typed by hand — a hand-written copy is how the repo once carried two tier tables that disagreed for twenty languages.

#### Language strictness

<!-- generated:strictness -->
| language | static types | enforced errors | memory safe | no implicit globals | Irc | Ot |
|---|---|---|---|---|---|---|
| `abap` | yes | **no** | yes | yes | 1 | 1.10 |
| `ada` | yes | **no** | yes | yes | 1 | 1.10 |
| `agc_assembly` | **no** | **no** | **no** | **no** | 4 | 1.40 |
| `apex` | yes | **no** | yes | yes | 1 | 1.10 |
| `assembly` | **no** | **no** | **no** | **no** | 4 | 1.40 |
| `batch` | **no** | **no** | yes | **no** | 3 | 1.30 |
| `blp` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `c` | yes | **no** | **no** | yes | 2 | 1.20 |
| `cobol` | yes | **no** | yes | **no** | 2 | 1.20 |
| `cpp` | yes | **no** | **no** | yes | 2 | 1.20 |
| `csharp` | yes | **no** | yes | yes | 1 | 1.10 |
| `css` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `csv` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `dart` | yes | **no** | yes | yes | 1 | 1.10 |
| `dockerfile` | **no** | **no** | yes | **no** | 3 | 1.30 |
| `fortran` | yes | **no** | **no** | **no** | 3 | 1.30 |
| `glsl` | yes | **no** | yes | yes | 1 | 1.10 |
| `go` | yes | **no** | yes | yes | 1 | 1.10 |
| `groovy` | **no** | **no** | yes | **no** | 3 | 1.30 |
| `haskell` | yes | yes | yes | yes | 0 | 1.00 |
| `hlo` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `html` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `java` | yes | yes | yes | yes | 0 | 1.00 |
| `javascript` | **no** | **no** | yes | **no** | 3 | 1.30 |
| `jcl` | **no** | **no** | yes | **no** | 3 | 1.30 |
| `json` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `kotlin` | yes | **no** | yes | yes | 1 | 1.10 |
| `livecode` | **no** | **no** | yes | **no** | 3 | 1.30 |
| `lua` | **no** | **no** | yes | **no** | 3 | 1.30 |
| `m4` | **no** | **no** | yes | **no** | 3 | 1.30 |
| `makefile` | **no** | **no** | yes | **no** | 3 | 1.30 |
| `markdown` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `matlab` | **no** | **no** | yes | yes | 2 | 1.20 |
| `mlir` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `nix` | **no** | **no** | yes | yes | 2 | 1.20 |
| `objective-c` | yes | **no** | **no** | yes | 2 | 1.20 |
| `pbtxt` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `perl` | **no** | **no** | yes | **no** | 3 | 1.30 |
| `php` | **no** | **no** | yes | yes | 2 | 1.20 |
| `plaintext` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `powershell` | **no** | **no** | yes | yes | 2 | 1.20 |
| `proto` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `python` | **no** | **no** | yes | yes | 2 | 1.20 |
| `ruby` | **no** | **no** | yes | yes | 2 | 1.20 |
| `rust` | yes | yes | yes | yes | 0 | 1.00 |
| `scala` | yes | **no** | yes | yes | 1 | 1.10 |
| `scheme` | **no** | **no** | yes | yes | 2 | 1.20 |
| `shell` | **no** | **no** | yes | **no** | 3 | 1.30 |
| `solidity` | yes | **no** | yes | yes | 1 | 1.10 |
| `sqlite` | **no** | **no** | yes | yes | 2 | 1.20 |
| `swift` | yes | yes | yes | yes | 0 | 1.00 |
| `tcl` | **no** | **no** | yes | yes | 2 | 1.20 |
| `td` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `typescript` | yes | **no** | yes | yes | 1 | 1.10 |
| `xml` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `yacc` | **no** | **no** | **no** | **no** | 4 | 1.40 |
| `yaml` *(no runtime)* | — | — | — | — | 0 | 1.00 |
| `zig` | yes | yes | **no** | yes | 1 | 1.10 |

Dialects read their family's row: `embedded_python` → `python`, `micropython` → `python`.
<!-- /generated:strictness -->

#### Fidelity coefficients the engine reads (signals below 1.00 only)

<!-- generated:fidelity -->
| language | `safety` | `test` | `doc` | `ownership` |
|---|---|---|---|---|
| `c` | 1.00 | **0.67** | 1.00 | 1.00 |
| `embedded_python` | **0.50** | **0.67** | 1.00 | 1.00 |
| `haskell` | **0.67** | 1.00 | 1.00 | 1.00 |
| `perl` | 1.00 | **0.67** | 1.00 | 1.00 |
| `solidity` | 1.00 | 1.00 | 1.00 | **0.25** |
| `typescript` | **0.67** | 1.00 | 1.00 | 1.00 |

Every language and signal not listed reads 1.00. Source: keyword-rosetta `6377eb59` (46 languages), regenerated by `tests/tools/fidelity_table.py`.
<!-- /generated:fidelity -->

## Pipeline Integration
- **Inputs:** Raw regex counts, LOC, language metadata.
Expand All @@ -32,7 +127,7 @@ $$RiskExposure = \left( \frac{((RiskHits + Irc) \times Weight) - (DefenseHits \t
Scanner Extraction -> Universal Exposure Framework -> Quantized Tier Output

## Tradeoffs
The sigmoid gating principle aggressively suppresses minor risks in large files, intentionally sacrificing micro-level visibility to prevent "alert fatigue" on the macro level. Language confidence tiers generalize thousands of languages into three buckets, reducing precision for niche languages.
The sigmoid gating principle aggressively suppresses minor risks in large files, intentionally sacrificing micro-level visibility to prevent "alert fatigue" on the macro level. The strictness table is four yes/no columns per language: enough to be reviewable, too coarse to rank two languages that share a row. That coarseness is deliberate until #2720 can say what finer resolution would be measuring.

## Limitations
- The 2.5x defensive multiplier is empirically derived and may not perfectly align with specific internal security postures.
Expand All @@ -42,7 +137,7 @@ The sigmoid gating principle aggressively suppresses minor risks in large files,
Processing utilizes constant-time floating-point math per file component, resulting in $O(1)$ metric transformation time per unit post-extraction.

## Future Work
- Machine learning parameter tuning for Fidelity Coefficients based on historical vulnerability tracking.
- Outcome calibration of the strictness weight against fix-commit history (#2720); the fidelity coefficients are already measured, not tuned.
- Configurable Breach Cap thresholds per repository.

## Related Components
Expand Down
2 changes: 1 addition & 1 deletion docs/wiki/08-08-technical-debt.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,4 +66,4 @@ Currently, the system is strictly lexical and static. Future iterations plan to
## Related Components
- Static Analysis Engine
- Path Modifier ($Mp$)
- Implicit Risk Correction ($Irc$)
- Implicit Risk Correction ($Irc$) — one point per strictness gap of the language ([08-03](08-03-transforming-regex-counts.md)), 0 for data and markup formats
6 changes: 3 additions & 3 deletions docs/wiki/08-09-documentation-risk.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,14 @@ Evaluates four contextual dimensions:
**Mathematical Formulation**
1. **Knowledge Shield Defense:**
$$\text{UmbrellaDefense} = \text{doc\_umbrella} \times 50.0$$
$$\text{DefenseHits} = \left( \text{InlineDocs} + (\text{Ownership} \times 0.5) + (\text{DocLOC} \times 0.33) + \text{UmbrellaDefense} \right) \times Fc$$
$$\text{DefenseHits} = (\text{InlineDocs} \times Fc_{doc}) + (\text{Ownership} \times 0.5 \times Fc_{ownership}) + (\text{DocLOC} \times 0.33) + \text{UmbrellaDefense}$$
2. **Undocumented Risk Calculation:**
$$\text{UndocumentedRisk} = \sum_{\text{undocumented}} \left( 5.0 + \ln(\text{Impact}) \right)$$
$$\text{RiskHits} = \text{UndocumentedRisk} + (\text{API\_Exposure} \times 2.0) + Irc$$
3. **Net Exposure & Line Density:**
$$\text{NetExposure} = \max\left(0, \text{RiskHits} - \frac{\text{DefenseHits}}{2.0}\right)$$
$$\text{Density} = \left( \frac{\text{NetExposure}}{\max(\text{LOC}, 50) + 20} \right) \times 100.0$$
The denominator is the UEF evidence-mass floor plus the equation's smoothing pad ([08-03](08-03-transforming-regex-counts.md)). $Irc$ corrects measured risk, it never creates it: when $\text{UndocumentedRisk} + \text{API\_Exposure} = 0$ the score is $0$ in every language tier (#2655).
The denominator is the UEF evidence-mass floor plus the equation's smoothing pad ([08-03](08-03-transforming-regex-counts.md)). $Irc$ corrects measured risk, it never creates it: when $\text{UndocumentedRisk} + \text{API\_Exposure} = 0$ the score is $0$ in every language (#2655).
4. **Systemic Multipliers & Mapping:**
$$\text{FinalMultiplier} = \left(1.0 + \frac{\text{Pop}}{10}\right) \times \left(1.0 + \frac{\text{Silo}}{200}\right) \times Mp$$
$$\text{RawRisk} = \frac{100.0}{1 + e^{-0.2 \times (\text{Density} - 10.0)}}$$
Expand All @@ -62,7 +62,7 @@ flowchart LR
## Limitations
- Cannot semantically read comments to confirm they explain the code (a comment saying "stuff happens here" provides defense weight).
- Network multipliers (popularity) only track internal repository imports and cannot measure external library consumers.
- Implicit languages with less rigid docstring structures might receive lower fidelity coefficients ($Fc$), unfairly elevating risk in Python or JavaScript projects lacking explicit type tags.
- $Fc_{doc}$ and $Fc_{ownership}$ are measured per language on the keyword-rosetta corpus ([08-03](08-03-transforming-regex-counts.md)); a language whose doc rule reads exactly on plant carries 1.0, so the old flat 0.60 for "implicit" languages no longer exists. $Irc$ comes from the strictness table.

## Performance Notes
Calculating the undocumented risk loop requires iterating over all functions within the file ($O(F)$ where $F$ is function count). Since $F$ is typically small, execution remains exceptionally fast and bounded.
Expand Down
2 changes: 1 addition & 1 deletion docs/wiki/08-15-concurrency-exposure.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The static analysis engine extracts concurrency keywords and synchronization pri
| `raw_concurrency` | Keywords | **1.0x** | Asynchronous and threading constructs: `async`, `await`, `Promise`, `thread`, `spawn`, `go`, `chan`, `synchronized`. |
| `sync_locks` | Mitigations | **-1.5x** | Synchronization primitives (mutexes, locks, semaphores). Each lock mitigates 1.5 thread spawns. |
| `loc` | Denominator | **Base Density** | Meaningful lines of code, padded by `loc_padding` (default 150). |
| `irc` | Language Modifier | **0.1x** | Implicit Risk Correction for dynamically typed or implicit concurrency models. |
| `irc` | Language Modifier | **0.1x** | Implicit Risk Correction: the language's strictness-gap count (0–4) from `analysis_lens.LANGUAGE_STRICTNESS` ([08-03](08-03-transforming-regex-counts.md)); 0 for data and markup formats. |
| `mp` | Path Modifier | **Threshold Modifier** | Context-specific modifier (e.g., `0.5` for UI components where race conditions trigger UI defects). |

The calculation balances raw concurrency against synchronization locks and applies a sigmoid transformation.
Expand Down
Loading
Loading