diff --git a/CLAUDE.md b/CLAUDE.md index 03d22ba..fa4f4ee 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -50,7 +50,7 @@ cargo run -- refs worktree 2.0.77 # a real tree, under the cache cargo run -- refs docs 2.1.14 runtime-api.json --which ``` -Test counts to expect: **247 unit tests**, plus **13 integration tests** split +Test counts to expect: **255 unit tests**, plus **13 integration tests** split across four files. `tests/acceptance.rs` has 3: two run offline against a committed fixture, and one (`the_real_install_reproduces_it_too`) is install-gated. `tests/provenance.rs` has 3: one always-on; one gated on the @@ -107,6 +107,22 @@ code did. **A fake can only be wrong in the ways its author already considered.* `error("DUMPED-OK")` sentinel, goes to stdout. Measured three ways on 2.1.14, and again on Windows: stderr was zero bytes every time. The original code checked stderr, so the sentinel was never seen on any real run. +- **A `create` run's failure message splits on the sentinel, because the sentinel + rules out the usual cause.** A mod skipped over a `factorio_version` mismatch + never runs, so it cannot raise `DUMPED-OK`. Measured 2026-08-18 on the first + probe a consumer wrote: it called + `helpers.write_file("basis-gradient-probe.json")`, raised the sentinel, and got + back "no dump was written ... factorio_version mismatch" while its 270 KB sat + in the report's own `files` array. `create` keys success off the one filename + `oracle-dump.json`, which is this tool's contract rather than the game's. +- **That first consumer probe worked, and its answer held on both versions.** + FactorioMapWebUI #234 recovered `basis_noise`'s 256-slot gradient table from + the running game rather than deriving it from a formula, and 2.0.77 and 2.1.14 + returned byte-identical captures. It took three of that repo's fixtures from + 473/512, 36/38 and 231/266 bit-exact to 512/512, 38/38 and 266/266, all at + worst error 0, with no change to its kernel. So the first real use of this + tool by a consumer settled something its own test suite had carried as a + known gap for a year. - **Omission in `mod-list.json` means enabled.** Factorio rewrites the file at startup and adds back every bundled mod the file does not mention, with `enabled: true`. A file naming only `base` comes back naming five, all loaded. diff --git a/README.md b/README.md index ad6f744..18f4af9 100644 --- a/README.md +++ b/README.md @@ -201,6 +201,11 @@ They are prose for a person, not input to anything. - [`docs/gotchas.md`](docs/gotchas.md) - specific facts about Factorio and its API, each of which cost a failed run. +One thing to know before reading them: a probe's `control.lua` must write its +result as `oracle-dump.json`. That name is this tool's contract, not the game's - +`helpers.write_file` accepts any name, and a `create` run reports failure unless +that one file appears. + Lifted with attribution from `factorio-blueprint-editor` and `FactorioMapWebUI`, which learned most of it the expensive way. That borrowing had been done by hand, repo to repo, and drifted; this gives it one home. diff --git a/docs/gotchas.md b/docs/gotchas.md index ba793b6..9ce77bf 100644 --- a/docs/gotchas.md +++ b/docs/gotchas.md @@ -37,6 +37,16 @@ re-measured here. **Measured 2026-08-16 on 2.1.14.** `script.active_mods` is populated there too. That is where metadata belongs: no collision surface, and it costs no ticks. +- **The dump must be written as `oracle-dump.json`.** **Measured 2026-08-18**, + on the first probe a consumer wrote against this tool. The name is this tool's + contract rather than the game's: `helpers.write_file` accepts any name, the + file lands in `script-output` next to where the tool looks, and `create` keys + its success off that one filename. A probe that wrote + `basis-gradient-probe.json` got `ok: false` with all 270 KB of its data sitting + in the report's own `files` array. The failure message now says so when the + sentinel was seen, because a mod that raised one cannot have been skipped over + a version mismatch. + - **Toplevel is for metadata, not for sampling.** `game.surfaces[1]` does not exist at control-stage toplevel, so anything calling `calculate_tile_properties` or `get_tile` still needs `on_init`. The two rules compose rather than compete. @@ -238,6 +248,15 @@ re-measured here. `tree-plant` and the three demolisher corpses. None was among the 155 that repo knows, so nothing broke - but nothing would have told them either. (#142) +- **Some tables are a constant of the engine rather than of a version, and which + one it is has to be measured.** **Measured 2026-08-18** on 2.0.77 (build 84539) + and 2.1.14 (build 87180): `basis_noise`'s 256-slot gradient table recovers + byte-identically from both, across 4,352 samples per axis, and the recovered + table also scores 512 of 512 exact against a FactorioMapWebUI fixture first + captured on 2.1.11 and re-verified bit-for-bit on 2.1.12. Four versions, one + table. The second run cost one command, and a disagreement would have been a + finding larger than the probe that found it. (FactorioMapWebUI #234) + - **Run a probe against two versions when you can.** The older one reproducing a known-good answer is what earns trust in the new one. `examples/pumpjack-terminals` does this, and it is how FactorioTools#81 was diff --git a/docs/method.md b/docs/method.md index aa4f864..803e7df 100644 --- a/docs/method.md +++ b/docs/method.md @@ -165,6 +165,14 @@ into an 8-versus-0 finding. **If a control fails, suspect the question first.** +A control comparing a table recovered from the game against a repo's own seeding +reimplementation failed outright on 2026-08-18, and read like a defect in the +recovery. The control was malformed. The repo's fixture stores a canonical gauge, +its `a` table starting `0,1,2,4,8,16,32,15`, while the reimplementation returns +the game's own tables. Both produce identical noise, so equality was never the +right test. The real control is gauge *equivalence*, checked across all 65,536 +lattice cells, and it passes. (FactorioMapWebUI #234) + ## Ask what a limit is before comparing it to a number `zoom_limits` has three fields, not two, and they are not all the same kind of @@ -216,6 +224,37 @@ that it should be an expected outcome rather than a surprise. A probe that only ever confirms the plan is not being pointed at anything. +## An inversion is a model, and a wrong one still returns a full set of numbers + +FactorioMapWebUI's notes said a sample at `(I + 1/256, J)` "isolates the `(I,J)` +corner" of `basis_noise`. It does not. `d` in the game's kernel is the *squared* +distance, so the corner at `(1,0)` sits at `d = 0.9922`, inside the falloff's +support, and contributes about 1.2e-4 of the near term - 1,014 times f32 epsilon. +Isolation is not merely awkward, it is impossible: leaving one corner live needs +`fy^2 > fx` and `fx^2 > fy` at once, which forces `fy > 1`. + +**Measured 2026-08-18.** The one-corner inversion recovers 2 of 256 slots. It +raises nothing and looks entirely healthy: 512 numbers, every one of plausible +size. Handling both corners and iterating twice recovers 248, and a 1-ULP search +scored against the probe's own samples reaches 254 with a worst error of 3.6e-12. + +Two rules come out of that: + +- **A `~=` in someone else's notes is a claim about a model, not a rounding.** + Check the arithmetic of the kernel before building on the approximation. Here + it cost an afternoon of simulation and no game runs at all, which is the + cheapest question that settles it. +- **Score against something the method never read.** The recovery never opens + the fixture it is later scored on, and samples different coordinates entirely, + so 512 of 512 exact is a test rather than a fit. That is stronger than holding + points back, because there is nothing left to leak. + +**Know what the method cannot see, and price it.** Two of the 256 slots are +near zero, `-1.8e-7` and `5.0e-8`, and no probe geometry can pin them: their ULP +is 3.6e-15 and the game returns f32. Perturbing both by 1.0e-9, 280 times the +residual, changes 0 of the 512 fixture values. A limit that has been measured to +cost nothing is a documented limit, not an open question. + ## Fixture policy Capture once, commit the JSON with its provenance, assert offline. Shared by both diff --git a/src/outcome.rs b/src/outcome.rs index 5fddc49..c61c06e 100644 --- a/src/outcome.rs +++ b/src/outcome.rs @@ -1,6 +1,7 @@ //! Deciding whether a run succeeded. The rule is per mode, not global. use crate::probe::Mode; +use crate::run::{PROBE_DUMP_FILE, SENTINEL}; /// What was observed after the process ended. #[derive(Debug, Clone)] @@ -38,12 +39,30 @@ pub fn evaluate(mode: Mode, facts: &RunFacts) -> Outcome { Mode::Create => { if facts.dump_exists { Outcome::Ok + } else if facts.sentinel_seen { + // The sentinel rules the usual cause out. A mod skipped over a + // factorio_version mismatch never runs, so it cannot raise one; + // seeing it means the probe ran and finished on purpose. Naming + // the mismatch here would send a reader to look at `info.json` + // when the mod demonstrably loaded. + // + // Measured 2026-08-18, the first probe written by a consumer: + // it called helpers.write_file("basis-gradient-probe.json"), + // raised the sentinel, and got back "no dump was written ... + // factorio_version mismatch". The 270 KB it had just written was + // listed in the report's own `files` array the whole time. + Outcome::Failed(format!( + "the probe raised {SENTINEL} but no {PROBE_DUMP_FILE} exists. \ + It ran and finished, so this is not a factorio_version mismatch. \ + Check the name given to helpers.write_file against this report's \ + `files` list: the dump must be written as {PROBE_DUMP_FILE}." + )) } else { - Outcome::Failed( - "no dump was written. The most common cause is a factorio_version \ + Outcome::Failed(format!( + "no {PROBE_DUMP_FILE} was written, and the probe never raised \ + {SENTINEL}. The most common cause is a factorio_version \ mismatch, which makes Factorio skip the mod in silence." - .to_string(), - ) + )) } } @@ -91,6 +110,37 @@ mod tests { assert!(matches!(out, Outcome::Failed(_))); } + #[test] + fn a_create_failure_with_no_sentinel_names_the_version_mismatch() { + // Nothing ran, so the silent skip really is the first thing to check. + let Outcome::Failed(why) = evaluate(Mode::Create, &facts(Some(1), false, false)) else { + panic!("expected a failure"); + }; + assert!(why.contains("factorio_version"), "{why}"); + assert!(why.contains(PROBE_DUMP_FILE), "{why}"); + } + + #[test] + fn a_create_failure_after_the_sentinel_rules_the_version_mismatch_out() { + // The mod ran to its own last line, so info.json is not the place to + // look. The first consumer probe landed exactly here by writing its + // dump under another name. + let Outcome::Failed(why) = evaluate(Mode::Create, &facts(Some(1), false, true)) else { + panic!("expected a failure"); + }; + assert!( + why.contains("not a factorio_version mismatch"), + "the sentinel excludes that cause: {why}" + ); + assert!( + !why.contains("most common cause"), + "that is the other arm's claim: {why}" + ); + assert!(why.contains(SENTINEL), "{why}"); + assert!(why.contains("helpers.write_file"), "{why}"); + assert!(why.contains(PROBE_DUMP_FILE), "{why}"); + } + #[test] fn dump_data_fails_on_a_non_zero_exit_even_if_a_dump_is_present() { // A non-zero exit is real information here, and a stale dump from an diff --git a/src/run.rs b/src/run.rs index 7140860..73e032d 100644 --- a/src/run.rs +++ b/src/run.rs @@ -27,12 +27,12 @@ pub struct RunRequest { } /// What a probe raises to say it finished on purpose rather than crashed. -const SENTINEL: &str = "DUMPED-OK"; +pub(crate) const SENTINEL: &str = "DUMPED-OK"; /// The dump file a `--dump-data` run writes, named by the game. const DUMP_DATA_FILE: &str = "data-raw-dump.json"; /// The default dump name for a probe mod. -const PROBE_DUMP_FILE: &str = "oracle-dump.json"; +pub(crate) const PROBE_DUMP_FILE: &str = "oracle-dump.json"; /// The preview image name. const PREVIEW_FILE: &str = "preview.png"; @@ -661,7 +661,10 @@ mod tests { let result = run_probe(&request, &NoDump).unwrap(); assert_eq!(result["ok"], false); - assert!(result["error"].as_str().unwrap().contains("no dump")); + // This spawner raises no sentinel, so the report names the silent skip. + let why = result["error"].as_str().unwrap(); + assert!(why.contains(PROBE_DUMP_FILE), "{why}"); + assert!(why.contains("factorio_version"), "{why}"); assert!(result["stderrTail"] .as_str() .unwrap()