From 95db796dfcca7d4b04d1fb1bc41564b274ab5f63 Mon Sep 17 00:00:00 2001 From: Eric J Date: Tue, 18 Aug 2026 12:13:35 -0700 Subject: [PATCH 1/2] Say what a create run's failure actually was, per the first consumer The first probe written against this tool by a consumer wrote its result as `basis-gradient-probe.json`. The run reported "no dump was written. The most common cause is a factorio_version mismatch, which makes Factorio skip the mod in silence." That named a cause the run had already ruled out: the probe raised `DUMPED-OK`, and a mod skipped over a version mismatch never runs, so it cannot raise the sentinel. The 270 KB it had just written was listed in the report's own `files` array the whole time. So `create`'s failure message now splits on the sentinel. Without one, the silent skip really is the first thing to check and the message says so. With one, the message says the mismatch is excluded and points at the filename instead. Both arms now name `oracle-dump.json` rather than saying "no dump", because that name is this tool's contract rather than the game's and it was documented nowhere. Both tests were mutation-checked: collapsing the two arms back to the one old message fails both. The three documents get what else that probe cost, since they exist so a fifth repo does not pay it again: - gotchas: the filename contract, and that `basis_noise`'s gradient table is a constant of the engine rather than of a version - it recovers byte-identically from 2.0.77 and 2.1.14, and scores 512 of 512 against a fixture captured on 2.1.11. - method: a wrong inversion returns a full set of plausible numbers. The one-corner form the consumer's notes describe recovers 2 of 256 slots and raises nothing. Plus scoring against what the method never read, pricing what it cannot see, and a concrete case of a control that was malformed rather than failing. - README: the output filename, where a probe author will look for it. CLAUDE.md's test count was 247 and is 255. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_016Qg3gYjaG2uEtvv5U6AjY6 --- CLAUDE.md | 10 ++++++++- README.md | 5 +++++ docs/gotchas.md | 19 ++++++++++++++++ docs/method.md | 39 +++++++++++++++++++++++++++++++++ src/outcome.rs | 58 +++++++++++++++++++++++++++++++++++++++++++++---- src/run.rs | 9 +++++--- 6 files changed, 132 insertions(+), 8 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 03d22ba..b720b89 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -50,7 +50,7 @@ cargo run -- refs worktree 2.0.77 # a real tree, under the cache cargo run -- refs docs 2.1.14 runtime-api.json --which ``` -Test counts to expect: **247 unit tests**, plus **13 integration tests** split +Test counts to expect: **255 unit tests**, plus **13 integration tests** split across four files. `tests/acceptance.rs` has 3: two run offline against a committed fixture, and one (`the_real_install_reproduces_it_too`) is install-gated. `tests/provenance.rs` has 3: one always-on; one gated on the @@ -107,6 +107,14 @@ code did. **A fake can only be wrong in the ways its author already considered.* `error("DUMPED-OK")` sentinel, goes to stdout. Measured three ways on 2.1.14, and again on Windows: stderr was zero bytes every time. The original code checked stderr, so the sentinel was never seen on any real run. +- **A `create` run's failure message splits on the sentinel, because the sentinel + rules out the usual cause.** A mod skipped over a `factorio_version` mismatch + never runs, so it cannot raise `DUMPED-OK`. Measured 2026-08-18 on the first + probe a consumer wrote: it called + `helpers.write_file("basis-gradient-probe.json")`, raised the sentinel, and got + back "no dump was written ... factorio_version mismatch" while its 270 KB sat + in the report's own `files` array. `create` keys success off the one filename + `oracle-dump.json`, which is this tool's contract rather than the game's. - **Omission in `mod-list.json` means enabled.** Factorio rewrites the file at startup and adds back every bundled mod the file does not mention, with `enabled: true`. A file naming only `base` comes back naming five, all loaded. diff --git a/README.md b/README.md index ad6f744..18f4af9 100644 --- a/README.md +++ b/README.md @@ -201,6 +201,11 @@ They are prose for a person, not input to anything. - [`docs/gotchas.md`](docs/gotchas.md) - specific facts about Factorio and its API, each of which cost a failed run. +One thing to know before reading them: a probe's `control.lua` must write its +result as `oracle-dump.json`. That name is this tool's contract, not the game's - +`helpers.write_file` accepts any name, and a `create` run reports failure unless +that one file appears. + Lifted with attribution from `factorio-blueprint-editor` and `FactorioMapWebUI`, which learned most of it the expensive way. That borrowing had been done by hand, repo to repo, and drifted; this gives it one home. diff --git a/docs/gotchas.md b/docs/gotchas.md index ba793b6..0c0a78f 100644 --- a/docs/gotchas.md +++ b/docs/gotchas.md @@ -37,6 +37,16 @@ re-measured here. **Measured 2026-08-16 on 2.1.14.** `script.active_mods` is populated there too. That is where metadata belongs: no collision surface, and it costs no ticks. +- **The dump must be written as `oracle-dump.json`.** **Measured 2026-08-18**, + on the first probe a consumer wrote against this tool. The name is this tool's + contract rather than the game's: `helpers.write_file` accepts any name, the + file lands in `script-output` next to where the tool looks, and `create` keys + its success off that one filename. A probe that wrote + `basis-gradient-probe.json` got `ok: false` with all 270 KB of its data sitting + in the report's own `files` array. The failure message now says so when the + sentinel was seen, because a mod that raised one cannot have been skipped over + a version mismatch. + - **Toplevel is for metadata, not for sampling.** `game.surfaces[1]` does not exist at control-stage toplevel, so anything calling `calculate_tile_properties` or `get_tile` still needs `on_init`. The two rules compose rather than compete. @@ -238,6 +248,15 @@ re-measured here. `tree-plant` and the three demolisher corpses. None was among the 155 that repo knows, so nothing broke - but nothing would have told them either. (#142) +- **Some tables are a constant of the engine rather than of a version, and which + one it is has to be measured.** **Measured 2026-08-18** on 2.0.77 (build 84539) + and 2.1.14 (build 87180): `basis_noise`'s 256-slot gradient table recovers + byte-identically from both, across 4,352 samples per axis, and the recovered + table also scores 512 of 512 exact against a FactorioMapWebUI fixture captured + on 2.1.11. Three versions, one table. The second run cost one command, and a + disagreement would have been a finding larger than the probe that found it. + (FactorioMapWebUI #234) + - **Run a probe against two versions when you can.** The older one reproducing a known-good answer is what earns trust in the new one. `examples/pumpjack-terminals` does this, and it is how FactorioTools#81 was diff --git a/docs/method.md b/docs/method.md index aa4f864..803e7df 100644 --- a/docs/method.md +++ b/docs/method.md @@ -165,6 +165,14 @@ into an 8-versus-0 finding. **If a control fails, suspect the question first.** +A control comparing a table recovered from the game against a repo's own seeding +reimplementation failed outright on 2026-08-18, and read like a defect in the +recovery. The control was malformed. The repo's fixture stores a canonical gauge, +its `a` table starting `0,1,2,4,8,16,32,15`, while the reimplementation returns +the game's own tables. Both produce identical noise, so equality was never the +right test. The real control is gauge *equivalence*, checked across all 65,536 +lattice cells, and it passes. (FactorioMapWebUI #234) + ## Ask what a limit is before comparing it to a number `zoom_limits` has three fields, not two, and they are not all the same kind of @@ -216,6 +224,37 @@ that it should be an expected outcome rather than a surprise. A probe that only ever confirms the plan is not being pointed at anything. +## An inversion is a model, and a wrong one still returns a full set of numbers + +FactorioMapWebUI's notes said a sample at `(I + 1/256, J)` "isolates the `(I,J)` +corner" of `basis_noise`. It does not. `d` in the game's kernel is the *squared* +distance, so the corner at `(1,0)` sits at `d = 0.9922`, inside the falloff's +support, and contributes about 1.2e-4 of the near term - 1,014 times f32 epsilon. +Isolation is not merely awkward, it is impossible: leaving one corner live needs +`fy^2 > fx` and `fx^2 > fy` at once, which forces `fy > 1`. + +**Measured 2026-08-18.** The one-corner inversion recovers 2 of 256 slots. It +raises nothing and looks entirely healthy: 512 numbers, every one of plausible +size. Handling both corners and iterating twice recovers 248, and a 1-ULP search +scored against the probe's own samples reaches 254 with a worst error of 3.6e-12. + +Two rules come out of that: + +- **A `~=` in someone else's notes is a claim about a model, not a rounding.** + Check the arithmetic of the kernel before building on the approximation. Here + it cost an afternoon of simulation and no game runs at all, which is the + cheapest question that settles it. +- **Score against something the method never read.** The recovery never opens + the fixture it is later scored on, and samples different coordinates entirely, + so 512 of 512 exact is a test rather than a fit. That is stronger than holding + points back, because there is nothing left to leak. + +**Know what the method cannot see, and price it.** Two of the 256 slots are +near zero, `-1.8e-7` and `5.0e-8`, and no probe geometry can pin them: their ULP +is 3.6e-15 and the game returns f32. Perturbing both by 1.0e-9, 280 times the +residual, changes 0 of the 512 fixture values. A limit that has been measured to +cost nothing is a documented limit, not an open question. + ## Fixture policy Capture once, commit the JSON with its provenance, assert offline. Shared by both diff --git a/src/outcome.rs b/src/outcome.rs index 5fddc49..c61c06e 100644 --- a/src/outcome.rs +++ b/src/outcome.rs @@ -1,6 +1,7 @@ //! Deciding whether a run succeeded. The rule is per mode, not global. use crate::probe::Mode; +use crate::run::{PROBE_DUMP_FILE, SENTINEL}; /// What was observed after the process ended. #[derive(Debug, Clone)] @@ -38,12 +39,30 @@ pub fn evaluate(mode: Mode, facts: &RunFacts) -> Outcome { Mode::Create => { if facts.dump_exists { Outcome::Ok + } else if facts.sentinel_seen { + // The sentinel rules the usual cause out. A mod skipped over a + // factorio_version mismatch never runs, so it cannot raise one; + // seeing it means the probe ran and finished on purpose. Naming + // the mismatch here would send a reader to look at `info.json` + // when the mod demonstrably loaded. + // + // Measured 2026-08-18, the first probe written by a consumer: + // it called helpers.write_file("basis-gradient-probe.json"), + // raised the sentinel, and got back "no dump was written ... + // factorio_version mismatch". The 270 KB it had just written was + // listed in the report's own `files` array the whole time. + Outcome::Failed(format!( + "the probe raised {SENTINEL} but no {PROBE_DUMP_FILE} exists. \ + It ran and finished, so this is not a factorio_version mismatch. \ + Check the name given to helpers.write_file against this report's \ + `files` list: the dump must be written as {PROBE_DUMP_FILE}." + )) } else { - Outcome::Failed( - "no dump was written. The most common cause is a factorio_version \ + Outcome::Failed(format!( + "no {PROBE_DUMP_FILE} was written, and the probe never raised \ + {SENTINEL}. The most common cause is a factorio_version \ mismatch, which makes Factorio skip the mod in silence." - .to_string(), - ) + )) } } @@ -91,6 +110,37 @@ mod tests { assert!(matches!(out, Outcome::Failed(_))); } + #[test] + fn a_create_failure_with_no_sentinel_names_the_version_mismatch() { + // Nothing ran, so the silent skip really is the first thing to check. + let Outcome::Failed(why) = evaluate(Mode::Create, &facts(Some(1), false, false)) else { + panic!("expected a failure"); + }; + assert!(why.contains("factorio_version"), "{why}"); + assert!(why.contains(PROBE_DUMP_FILE), "{why}"); + } + + #[test] + fn a_create_failure_after_the_sentinel_rules_the_version_mismatch_out() { + // The mod ran to its own last line, so info.json is not the place to + // look. The first consumer probe landed exactly here by writing its + // dump under another name. + let Outcome::Failed(why) = evaluate(Mode::Create, &facts(Some(1), false, true)) else { + panic!("expected a failure"); + }; + assert!( + why.contains("not a factorio_version mismatch"), + "the sentinel excludes that cause: {why}" + ); + assert!( + !why.contains("most common cause"), + "that is the other arm's claim: {why}" + ); + assert!(why.contains(SENTINEL), "{why}"); + assert!(why.contains("helpers.write_file"), "{why}"); + assert!(why.contains(PROBE_DUMP_FILE), "{why}"); + } + #[test] fn dump_data_fails_on_a_non_zero_exit_even_if_a_dump_is_present() { // A non-zero exit is real information here, and a stale dump from an diff --git a/src/run.rs b/src/run.rs index 7140860..73e032d 100644 --- a/src/run.rs +++ b/src/run.rs @@ -27,12 +27,12 @@ pub struct RunRequest { } /// What a probe raises to say it finished on purpose rather than crashed. -const SENTINEL: &str = "DUMPED-OK"; +pub(crate) const SENTINEL: &str = "DUMPED-OK"; /// The dump file a `--dump-data` run writes, named by the game. const DUMP_DATA_FILE: &str = "data-raw-dump.json"; /// The default dump name for a probe mod. -const PROBE_DUMP_FILE: &str = "oracle-dump.json"; +pub(crate) const PROBE_DUMP_FILE: &str = "oracle-dump.json"; /// The preview image name. const PREVIEW_FILE: &str = "preview.png"; @@ -661,7 +661,10 @@ mod tests { let result = run_probe(&request, &NoDump).unwrap(); assert_eq!(result["ok"], false); - assert!(result["error"].as_str().unwrap().contains("no dump")); + // This spawner raises no sentinel, so the report names the silent skip. + let why = result["error"].as_str().unwrap(); + assert!(why.contains(PROBE_DUMP_FILE), "{why}"); + assert!(why.contains("factorio_version"), "{why}"); assert!(result["stderrTail"] .as_str() .unwrap() From 811fe60fe29c3c49b0db752626f750c818450379 Mon Sep 17 00:00:00 2001 From: Eric J Date: Tue, 18 Aug 2026 12:36:07 -0700 Subject: [PATCH 2/2] Record what the first consumer probe actually settled Two corrections and one addition, all from checking the claim against FactorioMapWebUI's own PROVENANCE.json rather than from memory. The fixture the recovered table scores 512/512 against was first captured on 2.1.11 and re-verified bit-for-bit on 2.1.12, so the agreement spans four versions rather than three: 2.0.77, 2.1.11, 2.1.12 and 2.1.14. And the outcome is worth stating plainly next to the tool's own measured facts. The first real use of this tool by a consumer took three of that repo's fixtures from 473/512, 36/38 and 231/266 bit-exact to 512/512, 38/38 and 266/266, all at worst error 0, with no change to its kernel. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_016Qg3gYjaG2uEtvv5U6AjY6 --- CLAUDE.md | 8 ++++++++ docs/gotchas.md | 8 ++++---- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index b720b89..fa4f4ee 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -115,6 +115,14 @@ code did. **A fake can only be wrong in the ways its author already considered.* back "no dump was written ... factorio_version mismatch" while its 270 KB sat in the report's own `files` array. `create` keys success off the one filename `oracle-dump.json`, which is this tool's contract rather than the game's. +- **That first consumer probe worked, and its answer held on both versions.** + FactorioMapWebUI #234 recovered `basis_noise`'s 256-slot gradient table from + the running game rather than deriving it from a formula, and 2.0.77 and 2.1.14 + returned byte-identical captures. It took three of that repo's fixtures from + 473/512, 36/38 and 231/266 bit-exact to 512/512, 38/38 and 266/266, all at + worst error 0, with no change to its kernel. So the first real use of this + tool by a consumer settled something its own test suite had carried as a + known gap for a year. - **Omission in `mod-list.json` means enabled.** Factorio rewrites the file at startup and adds back every bundled mod the file does not mention, with `enabled: true`. A file naming only `base` comes back naming five, all loaded. diff --git a/docs/gotchas.md b/docs/gotchas.md index 0c0a78f..9ce77bf 100644 --- a/docs/gotchas.md +++ b/docs/gotchas.md @@ -252,10 +252,10 @@ re-measured here. one it is has to be measured.** **Measured 2026-08-18** on 2.0.77 (build 84539) and 2.1.14 (build 87180): `basis_noise`'s 256-slot gradient table recovers byte-identically from both, across 4,352 samples per axis, and the recovered - table also scores 512 of 512 exact against a FactorioMapWebUI fixture captured - on 2.1.11. Three versions, one table. The second run cost one command, and a - disagreement would have been a finding larger than the probe that found it. - (FactorioMapWebUI #234) + table also scores 512 of 512 exact against a FactorioMapWebUI fixture first + captured on 2.1.11 and re-verified bit-for-bit on 2.1.12. Four versions, one + table. The second run cost one command, and a disagreement would have been a + finding larger than the probe that found it. (FactorioMapWebUI #234) - **Run a probe against two versions when you can.** The older one reproducing a known-good answer is what earns trust in the new one.