Skip to content

Commit c664eb9

Browse files
committed
docs(observability-map): correct the corpus counts and drop the stale timings
Updating the workflow's entry count left three prose figures behind: the README said 44 rewrites applied and 43 defended, and both the README and INTERNALS said four and a half minutes. The corpus has 53 entries. The worse one was not a count. The README claimed one hole is open when KNOWN_GAPS has two, so the document under-reported what the corpus cannot defend, which is the one thing it has to be honest about. The second gap, dead-conjunction-instanceof-if, is now written out beside the first with its reason. Timings measured rather than scaled: the corpus step is about 1.7 minutes on CI's runner and about 8 on a laptop, so the earlier five-minute figure in the workflow was wrong too. Replaced the single numbers with the range, since an exact figure is machine-dependent and nothing asserts it.
1 parent 0df50af commit c664eb9

3 files changed

Lines changed: 18 additions & 13 deletions

File tree

.github/workflows/observability-map.yml

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -118,16 +118,17 @@ jobs:
118118
} >> "$GITHUB_OUTPUT"
119119
120120
# The tree-scale mutation corpus: every known laundering shape applied to the whole route tree,
121-
# asserting the score does not rise. Roughly five minutes for 53 entries, which is why it is
122-
# gated out of the package's default `pnpm test` and run here instead. Unlike the report job
121+
# asserting the score does not rise. 53 entries, a couple of minutes of a runner and a good deal
122+
# longer on a laptop, which is why it is gated out of the package's default `pnpm test` and run
123+
# here instead. Unlike the report job
123124
# below it has no token to lose, so it runs for fork PRs too, and unlike the report job it is
124125
# allowed to fail the build.
125126
#
126127
# Gated to this package's own paths rather than running on every route pull request. What the
127128
# corpus measures is the TOOL's resistance to laundering, and only an edit to the tool can weaken
128-
# that, so a routes-only change was paying five minutes of a 4x runner for a result that could
129-
# not differ from the last one. It was also the worst kind of job to spend that on: a red x that
130-
# fires on a large share of webapp pull requests, is allowed to fail, and gates nothing, which is
129+
# that, so a routes-only change was paying a couple of minutes of a 4x runner for a result that
130+
# could not differ from the last one. It was also the worst kind of job to spend that on: a red x
131+
# that fires on a large share of webapp pull requests, is allowed to fail, and gates nothing, which is
131132
# the shape people learn to scroll past.
132133
#
133134
# What this gives up is real and small. A route landing a shape no corpus entry has seen can make

internal-packages/observability-map/INTERNALS.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -497,4 +497,5 @@ steps degrade to a stale report, which at least stays one comment.
497497

498498
The corpus runs on the package's own paths and on a schedule rather than on every route pull request,
499499
because it measures the tool's resistance to laundering, which only an edit to the tool can weaken,
500-
and it costs four and a half minutes. The nightly covers tree drift late rather than not at all.
500+
and it costs a couple of minutes of a runner. The nightly covers tree drift late rather than not at
501+
all.

internal-packages/observability-map/README.md

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ find there.
9191

9292
### What stops it being gamed
9393

94-
`src/mutationCorpus.test.ts` applies 44 semantics-preserving or handling-deleting rewrites to the
94+
`src/mutationCorpus.test.ts` applies 53 semantics-preserving or handling-deleting rewrites to the
9595
whole route tree in a temp copy and asserts three things for each: the published global does not
9696
rise, the mean over the routes measured in both runs does not rise, and for a semantics-preserving
9797
rewrite no individual route's score rises or drops out of the measured set. Every laundering shape a
@@ -102,14 +102,17 @@ score from 19 to 8, so the metric does not pay you for removing error handling.
102102
in `try { ... } catch (e) { throw e }` leaves the global at 19 and raises no route, so it does not
103103
pay you for adding error handling that does nothing either.
104104

105-
One hole is open and the corpus says so. A catch over `try { 0; }` is refused, but `canRaise`
105+
Two holes are open and the corpus says so. A catch over `try { 0; }` is refused, but `canRaise`
106106
accepts any call, so `try { String(0); }` reads as real error handling: it takes the tree from 19 to
107107
44 and raises 224 routes. Telling an inert call from one that can throw needs types the scanner does
108-
not have. That entry, `dead-classifying-try-with-call`, runs as an expected failure with the
109-
residual written out beside it, so the claim is "43 rewrites are defended and here is the one that
110-
is not", never "unpaddable".
111-
112-
The corpus takes about four and a half minutes, so it is gated behind `OBS_MAP_MUTATION_CORPUS=1`
108+
not have. The second, `dead-conjunction-instanceof-if`, is a dead condition rather than a dead arm:
109+
`selectsADistinctPath` folds the arm, and `literalTruth` treats `&&` as always null on purpose so a
110+
live guard is never read as dead, so widening that fold is a different rule needing its own
111+
measurement. Both run as expected failures with the residual written out beside them, so the claim
112+
is "51 rewrites are defended and here are the two that are not", never "unpaddable".
113+
114+
The corpus takes minutes rather than seconds, under two on CI's runner and closer to eight on a
115+
laptop, so it is gated behind `OBS_MAP_MUTATION_CORPUS=1`
113116
and runs as its own CI job rather than in `pnpm test`. Run it if you change this package:
114117

115118
```bash

0 commit comments

Comments
 (0)