Skip to content

Say what a create run's failure actually was, per the first consumer - #12

Merged
wormeyman merged 2 commits into
mainfrom
probe-lessons-from-the-first-consumer
Aug 18, 2026
Merged

Say what a create run's failure actually was, per the first consumer#12
wormeyman merged 2 commits into
mainfrom
probe-lessons-from-the-first-consumer

Conversation

@wormeyman

Copy link
Copy Markdown
Contributor

The first probe a consumer wrote against this tool found a real defect in it, and
cost four lessons the three documents exist to carry. This is both.

The defect

A create run that wrote its dump under a different name reported:

no dump was written. The most common cause is a factorio_version mismatch,
which makes Factorio skip the mod in silence.

That names a cause the run had already ruled out. The probe raised DUMPED-OK,
and a mod skipped over a version mismatch never runs, so it cannot raise the
sentinel. The 270 KB it had just written was listed in the report's own files
array the whole time. The message sent a reader to info.json for a mod that
demonstrably loaded.

So the message now splits on sentinel_seen, which RunFacts already carried
and the Create arm ignored:

  • No sentinel. Nothing ran, so the silent skip really is the first thing to
    check, and the message still says so.
  • Sentinel seen. The message says the mismatch is excluded and points at the
    filename instead.

Both arms now name oracle-dump.json rather than saying "no dump". That name is
this tool's contract rather than the game's, and it was documented nowhere -
not in the README, not in docs/, only in a constant in run.rs.

Both new tests were mutation-checked: collapsing the two arms back to the one old
message fails both. One existing test in run.rs asserted contains("no dump")
and now asserts the two properties that distinguish the arms.

The docs

  • docs/gotchas.md - the filename contract, and that basis_noise's
    gradient table is a constant of the engine rather than of a version. It
    recovers byte-identically from 2.0.77 and 2.1.14, and scores 512 of 512 exact
    against a fixture first captured on 2.1.11 and re-verified on 2.1.12.
  • docs/method.md - a wrong inversion returns a full set of plausible
    numbers. The one-corner form the consumer's own notes describe recovers 2 of
    256 slots, raises nothing, and returns 512 values of plausible size. Plus
    scoring against what the method never read, pricing a limit the method cannot
    see, and a concrete case of a control that was malformed rather than failing.
  • README.md - the output filename, where a probe author will look for it.
  • CLAUDE.md - the test count was 247 and is 255, and the measured facts
    now record what that probe settled.

Verification

cargo fmt --all -- --check && cargo clippy --all-targets -- -D warnings && cargo test --all-targets

Clean. 255 unit tests and 13 integration tests. All four gates were satisfied
on the machine that ran them, including FACTORIO_ORACLE_PROVENANCE_DIR and
FACTORIO_ORACLE_OLD_FACTORIO, so none of the 13 skipped.

🤖 Generated with Claude Code

https://claude.ai/code/session_016Qg3gYjaG2uEtvv5U6AjY6

wormeyman and others added 2 commits August 18, 2026 12:13
The first probe written against this tool by a consumer wrote its result
as `basis-gradient-probe.json`. The run reported "no dump was written.
The most common cause is a factorio_version mismatch, which makes
Factorio skip the mod in silence." That named a cause the run had already
ruled out: the probe raised `DUMPED-OK`, and a mod skipped over a version
mismatch never runs, so it cannot raise the sentinel. The 270 KB it had
just written was listed in the report's own `files` array the whole time.

So `create`'s failure message now splits on the sentinel. Without one,
the silent skip really is the first thing to check and the message says
so. With one, the message says the mismatch is excluded and points at the
filename instead. Both arms now name `oracle-dump.json` rather than
saying "no dump", because that name is this tool's contract rather than
the game's and it was documented nowhere.

Both tests were mutation-checked: collapsing the two arms back to the one
old message fails both.

The three documents get what else that probe cost, since they exist so a
fifth repo does not pay it again:

- gotchas: the filename contract, and that `basis_noise`'s gradient table
  is a constant of the engine rather than of a version - it recovers
  byte-identically from 2.0.77 and 2.1.14, and scores 512 of 512 against
  a fixture captured on 2.1.11.
- method: a wrong inversion returns a full set of plausible numbers. The
  one-corner form the consumer's notes describe recovers 2 of 256 slots
  and raises nothing. Plus scoring against what the method never read,
  pricing what it cannot see, and a concrete case of a control that was
  malformed rather than failing.
- README: the output filename, where a probe author will look for it.

CLAUDE.md's test count was 247 and is 255.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Qg3gYjaG2uEtvv5U6AjY6
Two corrections and one addition, all from checking the claim against
FactorioMapWebUI's own PROVENANCE.json rather than from memory.

The fixture the recovered table scores 512/512 against was first captured
on 2.1.11 and re-verified bit-for-bit on 2.1.12, so the agreement spans
four versions rather than three: 2.0.77, 2.1.11, 2.1.12 and 2.1.14.

And the outcome is worth stating plainly next to the tool's own measured
facts. The first real use of this tool by a consumer took three of that
repo's fixtures from 473/512, 36/38 and 231/266 bit-exact to 512/512,
38/38 and 266/266, all at worst error 0, with no change to its kernel.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Qg3gYjaG2uEtvv5U6AjY6
@wormeyman
wormeyman merged commit 93a4936 into main Aug 18, 2026
1 check passed
@wormeyman
wormeyman deleted the probe-lessons-from-the-first-consumer branch August 18, 2026 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant