From 87f7a85019725664ed46381bf47431f5e283b720 Mon Sep 17 00:00:00 2001 From: Eric J Date: Mon, 17 Aug 2026 19:11:31 -0700 Subject: [PATCH 1/2] Add the three knowledge documents Build-order step 8. The probe-writing knowledge that four repos have been accumulating separately now has one home instead of copies that drift. docs/order-of-attack.md - factorio-data first, then the oracle, then the binary. Most behaviour is Lua shipped in the clear, and reaching for a disassembler usually means steps 1 and 2 were not exhausted. docs/method.md - the epistemics, and the part that transfers to any repo. A control must be able to fail while the hypothesis holds. Last man standing is not a measurement. A probe entity is part of the question, not a neutral instrument. Transcribe a proposed rule into the probe before writing code. Every rule here was paid for by a probe that produced a clean, confident, wrong answer, and each names the issue where it happened. docs/gotchas.md - the facts, roughly 40 of them, each of which cost a run. Marked measured where this repo measured it, with the version and date, and attributed to the sibling repo otherwise. factorio-blueprint-editor's README already said its method was borrowed from FactorioMapWebUI by hand. That borrowing had drifted, which is the argument for doing it once here rather than a fourth time. Docs only. 167 tests unchanged. --- CLAUDE.md | 13 ++ README.md | 18 +++ docs/gotchas.md | 268 ++++++++++++++++++++++++++++++++++++++++ docs/method.md | 240 +++++++++++++++++++++++++++++++++++ docs/order-of-attack.md | 82 ++++++++++++ 5 files changed, 621 insertions(+) create mode 100644 docs/gotchas.md create mode 100644 docs/method.md create mode 100644 docs/order-of-attack.md diff --git a/CLAUDE.md b/CLAUDE.md index 85e8ac5..f58ba70 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -176,6 +176,19 @@ code did. **A fake can only be wrong in the ways its author already considered.* afternoon ruling it out. Provenance entries stay a `Value` anyway, because only two keys are required and the rest must round-trip untouched. +### Writing a probe + +Three documents in `docs/` carry the accumulated probe-writing knowledge, lifted +with attribution from the sibling repos. Read them before writing a new probe; +they are the reason a fifth repo does not have to learn the same lessons again. + +- `docs/order-of-attack.md` - factorio-data first, then the oracle, then the + binary. +- `docs/method.md` - the epistemics. A control must be able to fail while the + hypothesis holds; last man standing is not a measurement. +- `docs/gotchas.md` - the facts, each of which cost a run. Overlaps this section + on purpose: the facts above are the ones this tool's own code depends on. + ### Writing Lua for a probe - **Factorio 2.1 deleted `LuaEntity.fluidbox` and the whole `LuaFluidBox` diff --git a/README.md b/README.md index ad2b8f6..07c9909 100644 --- a/README.md +++ b/README.md @@ -124,6 +124,24 @@ pass. A mismatch is a finding. goes. Run against 2.0.77 and 2.1.14, it settled FactorioTools issue #81 and caught a breaking runtime API change on the way. +## Writing a probe + +Three documents carry what probe-writing has cost across the four repos so far. +They are prose for a person, not input to anything. + +- [`docs/order-of-attack.md`](docs/order-of-attack.md) - which source to ask + first. factorio-data, then the oracle, then the binary, and why that order + saves the most time. +- [`docs/method.md`](docs/method.md) - how to design a probe whose answer you can + trust. Controls that can fail, refuting the rival rule, and why a probe entity + is part of the question. +- [`docs/gotchas.md`](docs/gotchas.md) - specific facts about Factorio and its + API, each of which cost a failed run. + +Lifted with attribution from `factorio-blueprint-editor` and `FactorioMapWebUI`, +which learned most of it the expensive way. That borrowing had been done by hand, +repo to repo, and drifted; this gives it one home. + ## Scope Behavioural reverse engineering for interoperability - understanding what the diff --git a/docs/gotchas.md b/docs/gotchas.md new file mode 100644 index 0000000..fc7c8f1 --- /dev/null +++ b/docs/gotchas.md @@ -0,0 +1,268 @@ +# Gotchas + +Facts about Factorio and its API that each cost a failed run. Lifted with +attribution from `factorio-blueprint-editor`'s `tools/oracle/README.md` and +`FactorioMapWebUI`'s `test/oracle/README.md`, plus what this tool measured while +being built. + +For transferable reasoning rules rather than facts, see [method.md](method.md). +For which source to ask first, see [order-of-attack.md](order-of-attack.md). + +Anything here marked **measured** names the version and date. Anything not marked +came across from a sibling repo with its own attribution, and has not been +re-measured here. + +## Writing the mod + +- **`helpers.write_file` and `helpers.table_to_json`, not `game.*`.** They moved + in 2.1 and the old names are gone. (factorio-blueprint-editor) + +- **`factorio_version` in `info.json` must match the binary's major.minor**, or + the mod is silently skipped, no dump appears, and nothing in Factorio's output + names the cause. Derive it from `factorio --version`; never hardcode it. Every + factorio-blueprint-editor probe up to #141 hardcodes `"2.1"`, which was correct + only because the only install on that machine was 2.1.12. This tool derives it + in `version.rs`. + +- **Embed a blueprint string or a noise expression in a Lua long bracket** + (`[==[ ... ]==]`) so base64, braces, quotes and `var('...')` survive verbatim. + (both repos) + +- **`script.on_init` takes exactly one handler.** A second registration silently + replaces the first, with no error. **Measured 2026-08-16 on 2.1.14.** 17 of 18 + factorio-blueprint-editor probes register an `on_init`, so anything a runner + injects must not use one. + +- **`helpers.write_file` works at `control.lua` toplevel**, with no event at all. + **Measured 2026-08-16 on 2.1.14.** `script.active_mods` is populated there too. + That is where metadata belongs: no collision surface, and it costs no ticks. + +- **Toplevel is for metadata, not for sampling.** `game.surfaces[1]` does not + exist at control-stage toplevel, so anything calling `calculate_tile_properties` + or `get_tile` still needs `on_init`. The two rules compose rather than compete. + Do not read "toplevel works" as "probes should stop using `on_init`". + +- **`--instrument-mod` is worse than useless for a probe runner.** **Measured + 2026-08-16 on 2.1.14.** It does give earlier hooks: `instrument-data.lua` ran + at 0.045s against `data.lua` at 0.129s. But `instrument-control.lua`'s + `script.on_init` handler **never fired**, because `control.lua` registered one + too and the later registration replaced it. So it hands a probe an earlier hook + whose event registration the consumer then destroys, silently. It would appear + to work on every probe that does not register `on_init`, and fail on every + probe that does. + +## Launching the game + +- **`error("DUMPED-OK")` makes Factorio exit non-zero. That is success.** Key off + the dump file existing, never the exit code. (both repos) + +- **Factorio writes nothing to stderr.** **Measured three ways on 2.1.14, and + again on Windows: stderr was zero bytes every time.** A control-stage + `error()`, a data-stage error, and an unknown command line flag all print to + stdout. This tool's first version checked stderr for the sentinel, so + `sentinelSeen` was false on every real run while sixty unit tests passed. + +- **Omission in `mod-list.json` means enabled.** **Measured 2026-08-17 on + 2.1.14.** Factorio rewrites the file at startup and adds back every bundled mod + the file does not mention, with `enabled: true`. A file naming only `base` came + back naming five, all loaded. An explicit `enabled: false` **is** honoured, so + naming a mod is the only way to get a smaller game than the install ships with. + An empty mod directory keeps out *user* mods and nothing else. + +- **`read-data` is layout-dependent, so never write it as a relative token.** + `__PATH__executable__/../data` is right for a macOS `.app` bundle and wrong + everywhere else. Windows and Linux put the binary at `bin/x64/`, so it resolves + to `bin/data` and the game exits 1 with "There is no package core in". Use an + absolute path. Windows accepts either slash style. + +- **`--write-data` is not a CLI flag on any platform.** It is a `config.ini` + `[path]` key reached with `-c`. `--dump-data` honours it, and the dump lands in + that directory's own `script-output`, which is what makes a stale dump from an + earlier capture impossible to pick up. **Verified 2026-08-17 on 2.1.14.** + +- **`--create` needs no map-gen settings file.** **Measured 2026-08-16 on + 2.1.14.** It generates a map, loads the mod and dumps with no settings file at + all. `--map-gen-seed` overrides a seed inside the settings file, so writing one + `seed` through both channels makes the precedence irrelevant. + +- **A blueprint import needs no `--map-gen-settings`** either, since nothing is + being generated. A noise probe does need one, to route + `property_expression_names.elevation` at the probe expression. (both repos) + +- **`factorio.exe` is a GUI-subsystem binary on Windows.** Launched from + PowerShell it returns immediately, does not wait, and captures no stdout. The + run looks like a 0.03 second success that produced nothing. Use + `Start-Process -Wait -NoNewWindow -PassThru -RedirectStandardOutput`. Rust's + `Command::spawn` plus `wait_with_output` waits correctly, so this bites shell + invocations rather than this tool. + +- **Nothing in the consumer repos has a timeout**, which is one of the things + this tool exists to fix. A hung game hangs the capture forever. + +## Reading the answer back + +- **`--dump-data` is byte-identical across CPU architectures.** **Measured + 2026-08-17.** macOS arm64 Steam and Windows x64 standalone, same build, both + produced 29,506,804 bytes with sha256 `acc944e6...` from byte-identical inputs. + All 3,506 `.lua`/`.json`/`.cfg` files under `data/` matched too. So a capture + is not platform-specific. + +- **`runtime-api.json` publishes no `defines` values.** **Measured on 2.1.14:** + across all 1,554 entries the only keys are `name`, `order` and `description`. + Reading `order` as the value happens to be right for directions today, only + because Factorio declares them clockwise from `north = 0` with no gaps. `order` + is a dense `0..n-1` index, so it cannot express a gap, a duplicate, or a + non-zero start. Read real values with a probe. + +- **`calculate_tile_properties(property_names, positions)` takes property names + first.** The HTML docs and `runtime-api.json` list `positions` first and are + wrong; the `order` field is authoritative. (FactorioMapWebUI) + +- **`help()` is gone in 2.1.** `LuaEntity` and `LuaTrain` have no such key, so + the usual way of asking an object what it can do does not work. The install + ships `doc-html/runtime-api.json`, the whole runtime API as data, which is a + better source than either `help()` or `strings` on the binary. + (factorio-blueprint-editor #115) + +- **`MapPosition` is 1/256 fixed-point.** A world offset below `1/256` rounds to + zero. Choose probe offsets accordingly. (FactorioMapWebUI) + +- **Entities snap, so a requested position is not a measured one.** A rail asked + for at (-4,-4) lands at (-3,-3) on the 2-tile rail grid, and several accepted + (position, direction) triples collapse to one real placement. Read offsets off + the created entity: 16 raw acceptances around a straight rail are 4 actual + signal positions. (factorio-blueprint-editor #133) + +- **`loadedMods` cannot come from `script.active_mods`**, because it never + reports `core` and the fixtures list it. Grep the game's stdout for + `Loading mod `. + +- **One case per blueprint string.** factorio-blueprint-editor's first probe put + four cases in one string and got a single `import_stack` code of -1 for the + lot, which could not be attributed. Worse, entities *did* come back, so -1 + looked like it might mean something other than what it does. + +## Fields that do not mean what their name says + +- **`tile_width` and `tile_height` are a centring parity, not a footprint.** The + runtime docs say so outright: the field decides "if the center should be in the + center of the tile (odd tile size dimension) or on the tile border (even tile + size dimension)". For 146 of the 155 entities factorio-blueprint-editor knows, + that coincides with the enclosing rectangle, which is exactly why reading it as + a footprint looks right. For the other 9, every curved rail and `rail-ramp`, + the published rectangle does **not contain the entity's own collision box**: + `curved-rail-b` is 2x2 against a box 4.88 tiles tall, and `rail-ramp` is 2x16 + against a box 3.6 tiles wide. (#142) + + **A field whose name reads like a size is not therefore a size. Check what the + docs say it decides.** + +- **`support_range` is a load path, not a radius.** `RailSupportPrototype` + declares 11 and `RailRampPrototype` 9, unchanged between the 2.0.73 and 2.1.12 + tags, and the docs give no units and no shape. One spot five tiles from a + support, asked four times with only the rails between them changing, answers + refused / accepted / refused / accepted. A lone support permits exactly the + rails resting on it, and everything beyond has to be reached through rails that + already exist. (factorio-blueprint-editor #141) + +- **`can_place_entity` answers a different question per `build_check_type`.** + `blueprint_ghost` skips rail adjacency entirely: a rail signal ghost is legal + on bare grass hundreds of tiles from any rail, 2704 of 2704 in the #95 probe's + empty-ground control, against 0 for `manual`. Use `manual` for "may this be + built here". A probe that accepts either will read as "anything goes anywhere". + +- **`create_entity` and `can_place_entity` disagree, and the gap is + buildability, not collision.** `create_entity` will build an elevated rail on + bare ground with no support beneath it; `can_place_entity` refuses one + everywhere for exactly that reason. (factorio-blueprint-editor #133) + +- **Ghosts are not a placement test.** Stamping a blueprint and reviving what it + leaves looks like the most faithful way to ask whether the game accepts a + layout. It is not: reviving one of two overlapping ghosts destroys the other + whether or not the layout is legal, so a legal arrangement loses an entity + exactly like an illegal one. #95 built and discarded that whole measurement. + +- **"How far apart" has two answers when the thing is built incrementally.** A + hand walk outward from one rail support stops at **12** tiles, because each + rail has to be legal at the moment it is placed and the walk only approaches + from one side. A **finished** line, built whole then knocked out and rebuilt, + is legal to **20**, because the far half is reachable from the other support. A + blueprint is a finished configuration, so 20 is the number that describes real + exports, and every export in the corpus spaces supports exactly 20 apart. + Measuring only the walk would have produced a rule that refuses every real + elevated bridge. (#141) + +## Version and seed traps + +- **A version stamp is not enough on its own.** See method.md's fixture policy: + `elevated-rail-collision.json` was captured on 2.1.12 for an editor targeting + 2.0.73, and one of twenty-one entries genuinely differs between those tags. + +- **2.0.77 ships five bundled mods, not six.** No `recycler`. So a 2.0-versus-2.1 + comparison can never be perfectly single-variable on mod set. That is a + property of the game, not a setup error. + +- **The Space-Age paths force a planet's surface seed to the map seed, which no + real save does.** A planet generates at `mapSeed + crc32(planet.name)`, zero + only for Nauvis. A forced-seed sample validates an expression port but is blind + to seed plumbing, and the two disagree completely rather than subtly: 9.7% tile + agreement. It cost a session to spot. (FactorioMapWebUI) + +- **Factorio 2.1 deleted `LuaEntity.fluidbox` and the whole `LuaFluidBox` + class**, flattening it onto `LuaEntity` as `get_fluid_box_*`. So + `entity.fluidbox.get_pipe_connections(i)` became + `entity.get_fluid_box_pipe_connections(i)`. + `entity.prototype.fluidbox_prototypes[i].pipe_connections` still works in both, + so prototype-level reads need no branching. + +- **Feature detection needs a `pcall`.** Reading an unknown key on `LuaEntity` + *raises* in 2.0 rather than returning nil, so the obvious guard throws on the + older version: + + ```lua + -- throws on 2.0 + if entity.get_fluid_box_pipe_connections then ... end + -- works on both + local has_new = pcall(function() return entity.get_fluid_box_pipe_connections end) + ``` + +- **Run a probe against two versions when you can.** The older one reproducing a + known-good answer is what earns trust in the new one. `examples/pumpjack-terminals` + does this, and it is how FactorioTools#81 was settled: all four rotations + matched the planner on 2.0.77, which ruled out a broken probe, and two of them + disagreed on 2.1.14. + +## Numbers and determinism + +These are specific to this tool, and each is load-bearing on `trim --check` +being a usable diff. + +- **`serde_json` mis-parses long decimal literals by one ULP**, in both + directions, on values like + `0.394500000000000028421709430404007434844970703125`. **Measured 2026-08-17.** + Round-tripping the 25 MB dump found 9,744 differing lines. The crate's + *printer* is fine and its *parser* is not; Rust's own `f64::from_str` is + correctly rounded and agrees with CPython bit for bit. The fix is + `arbitrary_precision` plus re-parsing through `std`. Do not remove it. + +- **`preserve_order` must stay off** so `serde_json::Map` is a `BTreeMap` and + output sorts to match Python's `sort_keys=True`. + +- **Sampled f32 values must round-trip exactly.** Shortest round-trip + representation, never a fixed precision, never widened to f64. Scoring a port + by exact-match count beats any error bound: two kernels shared an identical + worst error of 2.682e-7 and differed by 42 exact matches of 512. + (FactorioMapWebUI) + +## The lesson underneath all of these + +**A fake can only be wrong in the ways its author already considered.** + +Three real defects in this tool survived sixty unit tests and were caught by the +first run against the actual game: the sentinel read off the wrong stream, a +`mod-list.json` believed to load only `base`, and a seed hardcoded in `main` so +a spec's value went nowhere. Two of the three were invisible to the unit tests by +construction, because the fake encoded the same wrong belief the code did. + +So make the fake wrong in the same way the real thing is, and settle any claim +about the game by running it. diff --git a/docs/method.md b/docs/method.md new file mode 100644 index 0000000..f019a49 --- /dev/null +++ b/docs/method.md @@ -0,0 +1,240 @@ +# Method + +How to ask the game a question and get an answer you can trust. These are +transferable rules, not Factorio facts. For the facts, see +[gotchas.md](gotchas.md). + +Every rule here was paid for by a probe that produced a clean, confident, wrong +answer. The examples name the repo and issue so you can go read the wreckage. + +## A control must be able to fail while the hypothesis holds + +This is the one that catches the most, and it is the easiest to get wrong, +because a control that cannot fail still passes and still feels like rigour. + +factorio-blueprint-editor's rail-on-rail probe (#133) used "a rail on an +identical rail overlaps nowhere" as its second control. It failed on every +curved orientation: 4 overlapping spots per `curved-rail-a` orientation, 8 per +`legacy-curved-rail`. Nothing was actually wrong. A curved rail's tile rectangle +is mostly empty, so a second identical curved rail sits legally beside it with +the rectangles overlapping and the curves not. + +The problem was that the control restated the hypothesis. It could only agree or +announce the finding, never invalidate the apparatus. It was replaced with "the +anchor's own position is never accepted", which is about the rig. + +**The null-result form is worse**, because nothing looks wrong. In PR #222, four +of seven cases would have reported "the coordinates did not change" whether the +probe was right or reading the same field twice. What rescued it was a +`shifted-entities` case that moves two chests three tiles left and four up +without touching any snapping property. Its coordinates *have* to differ. Without +a control that can fail, "nothing moved" is not evidence of anything. + +## Include a control that can invalidate the probe itself + +Not a control for the behaviour. One for the measurement. + +factorio-blueprint-editor's `probe-copy-settings-schedule.mjs` (#115) runs a case +where `copy_settings` is never called, because `create_blueprint` groups a +train's locomotives into one `schedules` entry on its own. Without that case, a +merged entry would have looked like a successful copy. + +The filter-count probe (#93) learned the same lesson the expensive way. It read +50 as a cap when it was the game deduplicating 50 cycled item names. + +**Report a section its control voids as unmeasured.** When +factorio-blueprint-editor's elevated-rail sweep came back 0 accepted against a +0-of-8 empty-ground control, the zeros were not a finding. The section had voided +itself. + +## Last man standing is not a measurement + +Evaluate every probe against every rule, and report the rules that survive **all** +of them, not the one that fits the last result. + +The zoom probe (#206) had three candidate rules for what a notch does from an +off-ladder start. All three agree on every sample taken from a rung, so only an +off-rung start says anything, and *which pair* it separates depends on where in +the gap it sits. A start at 0.83 sits below its gap's midpoint, so a notch **in** +rules out "multiply where you are" and cannot tell "snap to the nearest rung then +step" from "move to the next rung in the direction of travel". Both say +0.905724, exactly. A notch **out** from the same value separates them, 0.742997 +against 0.820335, and settled it. + +Reading the first probe as a verdict would have adopted the wrong rule with a +number matching to nine digits. + +**Which hypotheses a probe value can separate is part of the question.** Decide +that before you run it, not after. + +## Refute the rival + +A rule that explains your data is not the same as a rule your data picked out. +When two rules both fit, the useful probe is the one designed to kill one of +them, not the one that confirms both again. + +This is the same discipline as the section above, applied before the probe is +written rather than after it returns. + +## A probe entity is part of the question, not a neutral instrument + +"Which tiles does this rail occupy" sounds like a property of the rail. It is +not. Collision is continuous, so the answer depends on how big the thing you ask +with is. + +factorio-blueprint-editor measured this with four 1x1 references (#133). A +`small-electric-pole` (0.3 x 0.3) fits on cells a `wooden-chest` (0.7 x 0.7) is +refused on, and a `transport-belt` (0.8 x 0.8) is refused on cells the chest is +not: 28 and 24 of 38 rail orientations respectively. + +**Sweep more than one reference before concluding anything of the form "X +occupies Y". If they disagree, that is the finding.** + +**A probe entity's own lattice counts too.** The #141 probe put a `rail-support` +at the coordinates of the rail it was meant to hold, because both are +`build_grid_size = 2` and it looked like one lattice. It is not. A north-south +line of `elevated-straight-rail` sits on odd y and its supports on **even** y, +between two rails, and which parity is legal depends on the support's +orientation. A support on the wrong parity is created happily, stands there, and +holds nothing. Every profile came back byte-identical to having no support at +all, across all 16 orientations and 16 distances. That reads as "supports do not +participate in buildability", which is clean, confident and wrong. + +Sweeping both parities of both axes cost four lines. What actually caught it was +not the probe: it was decoding a real export from the corpus and looking at where +the game had put the supports. + +**When a probe says an entity does nothing, check a real example of it doing +something before believing the probe.** + +## Ask the cheapest question that settles it + +Before building a rig, check whether something simpler answers the same question. +A prototype field beats a placement test. A read beats a sweep. A sweep of a +table beats a list written in advance. + +`probe-rail-placement.mjs` skipped elevated rails because placing one needs rail +supports. The question underneath was collision, and +`LuaEntityPrototype.collision_mask` needs nothing placed at all. + +## Transcribe the rule into the probe before writing any code + +Put the rule you are about to implement into the probe, and report what it would +get wrong across every measured row. + +factorio-blueprint-editor's rail-on-rail probe carries two transcriptions of the +editor's arms, before and after, and scores both. The first draft of a fix, +"allow whenever the prototypes differ", produced four corruption-class rows. The +re-run caught it in eight seconds, and no test would have suggested it. + +**A transcribed rule needs a coordinate control, not only a logic one.** +`probe-entity-tile-size.mjs` compares two candidate footprints against a fixture +whose blocked cells are stored relative to `floor(position)` rather than in world +coordinates. The first transcription keyed its rectangles absolutely. Both arms +inflated together, 440/356 against 435/399, which preserved the verdict and +looked entirely plausible. Nothing about the shape of the output said it was +wrong. + +What says so is a row whose answer is known independently. A cardinal +`straight-rail` is exact on both arms, so it must come back all zeros, and that +is asserted. + +## Sweep two window sizes, and make the wider one an explicit control + +`probe-rail-placement.mjs` swept plus or minus 3 tiles. That is ample for a 2x2 +rail and not for a 4x8 one. `legacy-curved-rail`'s legal signal positions reach +an offset of 3.5, so the fixture lost one or two of the four at every orientation +and recorded 2 or 3. + +Nothing in the output looked wrong. The spots it did find were real. + +The fix is to sweep at two window sizes and make "the wider one finds nothing +new" an explicit control, which is what `probe-rail-signal-spots.mjs` does. It +caught 16 clipped sweeps of 76. + +The rule built on the clipped number survived re-checking, so the cost was a +wrong count rather than wrong behaviour. That is luck, and it was only knowable +by re-measuring. + +## Ask for the whole answer, not the one value you brought + +The #95 probe asked whether a gate was placeable at one direction, the one its +control had picked while standing in open ground, and got a false negative +because that direction was parallel to the rail. Sweeping all sixteen turned it +into an 8-versus-0 finding. + +**If a control fails, suspect the question first.** + +## Ask what a limit is before comparing it to a number + +`zoom_limits` has three fields, not two, and they are not all the same kind of +thing. `closest` is a zoom. `furthest` is a **rule**: at most 200 tiles across +the window, capped at 500. Comparing that to a readback answers "DISAGREE" for a +limit that is perfectly correct. + +Transcribing the documented formula turned an incomparable field into a third +instrument that agrees with the readback to 1e-9. The third field, +`furthest_game_view`, was not known to the design at all, and is the one that +matters to a viewer with no map view. + +## A script-set value is not a measured one + +The command that sets an off-ladder zoom trips the live sampler on the same tick, +so 0.83 arrives looking exactly like a rung the wheel stopped on. It would join +the distinct set, sit between two real rungs, and break the constant-ratio check +for a reason that has nothing to do with the game. + +**Tag what the probe itself wrote, and exclude it by tag rather than by value.** + +## "No headless probe can reach it" is not "not worth measuring" + +The zoom design (#206) wrote the per-notch step off as "an input-handling +constant no headless probe can reach", and proposed shipping a guessed 1.33. + +The real answer is `2^(1/7)`, three notches of the real ladder away from the +guess, and getting it took one sitting with a human at the keyboard. + +Some questions need a person. That is a cost, not a wall. The first claim was +allowed to stand in for the second, and it should not have been. + +## Be ready for the measurement to refuse the change it was capturing evidence for + +This has happened four times in factorio-blueprint-editor, which is often enough +that it should be an expected outcome rather than a surprise. + +- **#142**, "make the editor's footprints agree with the game's `tile_width`". + They already agreed for 146 of 155 entities, and adopting the numbers made + agreement with measured occupancy worse in both directions. +- **#206**, "use the game's zoom range". 30 of 367 corpus blueprints are wider + than the game's 200-tile floor allows, so adopting it makes 8% of real + blueprints impossible to view whole. The game's limit exists to stop a player + seeing ungenerated chunks, which an editor does not have to care about. Its + map editor floor, 0.1, is the number that transfers. +- **#141**, where the measured half turned out not to be a geometry at all. +- **PR #222**, where the premise was that a blueprint's grid position moves its + entities. It does not. + +A probe that only ever confirms the plan is not being pointed at anything. + +## Fixture policy + +Capture once, commit the JSON with its provenance, assert offline. Shared by both +sibling repos, and enforced here by `factorio-oracle provenance check`. + +- **Never hand-edit a fixture to make a test pass.** A mismatch is a finding. +- **Version-stamp every capture.** Steam updates the binary without asking. +- **A fixture's provenance is a record of the moment it was captured, not a live + claim.** Never edit one to make it current. factorio-blueprint-editor has six + fixtures carrying a `versionCaveat` that was true when written and is not any + more; the right fix is to correct the probe so a future recapture states it + properly, not to rewrite history. +- **A probe that owns a fixture recaptures it behind a flag, never on every run.** + A probe that rewrote its own fixture would turn "the game changed" into "the + fixture changed". +- **Record a cross-check, not only the binary.** `elevated-rail-collision.json` + was captured on 2.1.12 for an editor targeting 2.0.73. Cross-checking against + the Lua at the 2.0.73 tag found one real difference in twenty-one entries: + `core/lualib/collision-mask-defaults.lua` has `["cargo-bay"] = building_tall()` + at 2.0.73, which carries the elevated rail layer, and `building()` at 2.1.12, + which does not. A fixture saying only "2.1.12" would have been correct and + still produced the wrong rule. diff --git a/docs/order-of-attack.md b/docs/order-of-attack.md new file mode 100644 index 0000000..4a7f6a3 --- /dev/null +++ b/docs/order-of-attack.md @@ -0,0 +1,82 @@ +# Order of attack + +Which source to ask first when you need to know what Factorio does. This is the +part that saves the most time, and it is not "open a disassembler". + +Lifted from `factorio-blueprint-editor`'s `tools/oracle/README.md`, where it was +written after the fact, once the expensive ordering had already been tried. + +## 1. factorio-data first + +Most behaviour is Lua, shipped in the clear at +[github.com/wube/factorio-data](https://github.com/wube/factorio-data), with one +git tag per release. It is free to read, it needs no install, and it answers +more questions than people expect. + +**Grep for the definition site, not a bare name.** `name *= *""` finds +where a prototype is declared. A bare name matches every caller, which on a big +prototype can be hundreds of lines of noise. + +**Check out the tag that matches what you target, not the newest one.** The four +consumer repos target different versions on purpose, and a rule grounded against +the wrong tag can be right about a game nobody is running. factorio-blueprint-editor +learned this the hard way: it targets 2.0.45 to 2.0.73, but its blueprint corpus +spans 2.0.32 to 2.1.12, and two of its twelve files declare 2.1.12. A rule +checked only against the 2.0.73 tag predates those two files. + +**A clone is not a checkout.** `~/GitHub/factorio-data` is one working tree that +three repos want at three different tags. Read at a tag without moving `HEAD` +(`git show :`, `git grep `), or take a worktree. Moving +`HEAD` in a shared clone breaks whatever else is reading it, silently. + +**The migration files are the same bytes as an install's.** Verified 2026-08-17 +across all 16 `.json` files under `data/base/migrations` and +`data/space-age/migrations` on 2.1.14: `diff -rq` against a real install reports +no differing file. So a rename can be checked with no Factorio installed at all. + +## 2. Then the oracle + +This tool. Use it for anything the Lua does not answer because it is engine +behaviour rather than data: import rules, validation, placement, numeric +primitives, and every `defines` value. + +The dividing line is whether the answer is written down anywhere. If it is in +`data.raw`, the migrations, or `runtime-api.json`, read it. If the only way to +learn it is to make the game do the thing and watch, that is a probe. + +**Check whether a prototype field answers it before building a placement rig.** +factorio-blueprint-editor's `probe-rail-placement.mjs` skipped elevated rails +because placing one needs rail supports, and it was asking a placement question. +The question underneath was collision, and `LuaEntityPrototype.collision_mask` +needs nothing placed at all. Sweeping the prototype table gave the complete set +rather than a list written in advance. + +## 3. Only then the binary + +Factorio ships unstripped, so a short list of genuinely compiled things can be +recovered from it. FactorioMapWebUI has done this for the noise generator's +gradient table, which no formula reproduces. + +Nothing in factorio-blueprint-editor has needed it yet, after eighteen probes. +Treat reaching for it as a signal that steps 1 and 2 were not exhausted. + +## Two sources that answer a different question + +`README.md` splits sources into two kinds, and the split decides how each may be +used. + +**Authoritative** sources say what the game accepts. They ship with the install, +read offline, and are byte-identical run to run: `--dump-data`, +`data/*/migrations/*.json`, and `doc-html/runtime-api.json`. Only these may gate +a capture. + +**Reference** sources say *why* something changed, which is what tells you which +captured value now needs review: `data/changelog.txt`, and the Friday Facts blog +at . Nothing automated reads them and nothing should. +The blog is Wube's own writing, so it outranks forum and wiki advice, but it +describes intent, and the shipped game can differ. Check it against a capture. + +Search the blog at `https://factorio.com/blog/search/`, which covers the +whole archive. Word choice matters: `mirroring` does not return FFF #442, the +post that introduced entity mirroring, while `flip` and `fluid` both do. Try two +or three wordings before concluding a topic was never written about. From a9c145b688b17a2ef34f1d6531bf8fe5119045e9 Mon Sep 17 00:00:00 2001 From: Eric J Date: Mon, 17 Aug 2026 19:21:37 -0700 Subject: [PATCH 2/2] Give every borrowed fact a source you can check An accuracy review against both sibling READMEs confirmed every number transcribed correctly - the counts, ratios, tile offsets, issue numbers and hashes all match. What it found instead was four bullets carrying neither a "measured" tag nor an attribution, which is the exact ambiguity the file's own opening paragraph promises will not happen. The worst of the four claimed 2.0.77 ships five bundled mods rather than six, with no trail anywhere in this repo or either source. It is true, and it now says how it is known: measured on a standalone 2.0.77 install, and confirmed independently by the fixture diff, where recycler appears in loadedMods only on 2.1.14. A durable reference that asks to be trusted is worse than one that shows its working. Also: the f32 round-trip claim is attributed to the notes file it actually comes from rather than to an oracle README that does not contain it; the zoom probe regains its second issue number; the cross-check bullet regains the versionDifferences field the source names; and a dropped lesson is restored - re-capturing footprints on 2.0.x found four entities that move between versions, which nothing was looking for. Docs only. 167 tests unchanged. --- docs/gotchas.md | 51 ++++++++++++++++++++++++++--------------- docs/method.md | 19 +++++++-------- docs/order-of-attack.md | 9 ++++---- 3 files changed, 47 insertions(+), 32 deletions(-) diff --git a/docs/gotchas.md b/docs/gotchas.md index fc7c8f1..ba793b6 100644 --- a/docs/gotchas.md +++ b/docs/gotchas.md @@ -56,8 +56,8 @@ re-measured here. - **`error("DUMPED-OK")` makes Factorio exit non-zero. That is success.** Key off the dump file existing, never the exit code. (both repos) -- **Factorio writes nothing to stderr.** **Measured three ways on 2.1.14, and - again on Windows: stderr was zero bytes every time.** A control-stage +- **Factorio writes nothing to stderr.** **Measured 2026-08-17, three ways on + 2.1.14 and again on Windows: stderr was zero bytes every time.** A control-stage `error()`, a data-stage error, and an unknown command line flag all print to stdout. This tool's first version checked stderr for the sentinel, so `sentinelSeen` was false on every real run while sixty unit tests passed. @@ -73,12 +73,12 @@ re-measured here. `__PATH__executable__/../data` is right for a macOS `.app` bundle and wrong everywhere else. Windows and Linux put the binary at `bin/x64/`, so it resolves to `bin/data` and the game exits 1 with "There is no package core in". Use an - absolute path. Windows accepts either slash style. + absolute path. Windows accepts either slash style. (this repo, PR #2) - **`--write-data` is not a CLI flag on any platform.** It is a `config.ini` `[path]` key reached with `-c`. `--dump-data` honours it, and the dump lands in that directory's own `script-output`, which is what makes a stale dump from an - earlier capture impossible to pick up. **Verified 2026-08-17 on 2.1.14.** + earlier capture impossible to pick up. **Measured 2026-08-17 on 2.1.14.** - **`--create` needs no map-gen settings file.** **Measured 2026-08-16 on 2.1.14.** It generates a map, loads the mod and dumps with no settings file at @@ -89,9 +89,10 @@ re-measured here. being generated. A noise probe does need one, to route `property_expression_names.elevation` at the probe expression. (both repos) -- **`factorio.exe` is a GUI-subsystem binary on Windows.** Launched from - PowerShell it returns immediately, does not wait, and captures no stdout. The - run looks like a 0.03 second success that produced nothing. Use +- **`factorio.exe` is a GUI-subsystem binary on Windows.** **Measured 2026-08-17** + on a standalone win64 install. Launched from PowerShell it returns immediately, + does not wait, and captures no stdout. The run looks like a 0.03 second success + that produced nothing. Use `Start-Process -Wait -NoNewWindow -PassThru -RedirectStandardOutput`. Rust's `Command::spawn` plus `wait_with_output` waits correctly, so this bites shell invocations rather than this tool. @@ -102,13 +103,15 @@ re-measured here. ## Reading the answer back - **`--dump-data` is byte-identical across CPU architectures.** **Measured - 2026-08-17.** macOS arm64 Steam and Windows x64 standalone, same build, both - produced 29,506,804 bytes with sha256 `acc944e6...` from byte-identical inputs. + 2026-08-17 on 2.1.14, build 87180.** macOS arm64 Steam and Windows x64 + standalone, same build, both produced 29,506,804 bytes with sha256 + `acc944e6...` from byte-identical inputs. All 3,506 `.lua`/`.json`/`.cfg` files under `data/` matched too. So a capture is not platform-specific. -- **`runtime-api.json` publishes no `defines` values.** **Measured on 2.1.14:** - across all 1,554 entries the only keys are `name`, `order` and `description`. +- **`runtime-api.json` publishes no `defines` values.** **Measured 2026-08-17 on + 2.1.14:** across all 1,554 entries the only keys are `name`, `order` and + `description`. Reading `order` as the value happens to be right for directions today, only because Factorio declares them clockwise from `north = 0` with no gaps. `order` is a dense `0..n-1` index, so it cannot express a gap, a duplicate, or a @@ -133,8 +136,8 @@ re-measured here. the created entity: 16 raw acceptances around a straight rail are 4 actual signal positions. (factorio-blueprint-editor #133) -- **`loadedMods` cannot come from `script.active_mods`**, because it never - reports `core` and the fixtures list it. Grep the game's stdout for +- **`loadedMods` cannot come from `script.active_mods`** (this repo), because it + never reports `core` and the fixtures list it. Grep the game's stdout for `Loading mod `. - **One case per blueprint string.** factorio-blueprint-editor's first probe put @@ -198,9 +201,12 @@ re-measured here. `elevated-rail-collision.json` was captured on 2.1.12 for an editor targeting 2.0.73, and one of twenty-one entries genuinely differs between those tags. -- **2.0.77 ships five bundled mods, not six.** No `recycler`. So a 2.0-versus-2.1 - comparison can never be perfectly single-variable on mod set. That is a - property of the game, not a setup error. +- **2.0.77 ships five bundled mods, not six.** No `recycler`. **Measured + 2026-08-17** on a standalone 2.0.77 install, and confirmed independently by the + fixture-level diff between 2.0.77 and 2.1.14, where `recycler` appears in + `loadedMods` only on the newer one. So a 2.0-versus-2.1 comparison can never be + perfectly single-variable on mod set. That is a property of the game, not a + setup error. - **The Space-Age paths force a planet's surface seed to the map seed, which no real save does.** A planet generates at `mapSeed + crc32(planet.name)`, zero @@ -226,9 +232,16 @@ re-measured here. local has_new = pcall(function() return entity.get_fluid_box_pipe_connections end) ``` +- **Re-capturing on a second version finds drift you were not looking for.** + factorio-blueprint-editor re-took its entity footprints on a 2.0.x binary + rather than 2.1.12, and four entities move footprint between the two: + `tree-plant` and the three demolisher corpses. None was among the 155 that + repo knows, so nothing broke - but nothing would have told them either. (#142) + - **Run a probe against two versions when you can.** The older one reproducing a - known-good answer is what earns trust in the new one. `examples/pumpjack-terminals` - does this, and it is how FactorioTools#81 was settled: all four rotations + known-good answer is what earns trust in the new one. + `examples/pumpjack-terminals` does this, and it is how FactorioTools#81 was + settled: all four rotations matched the planner on 2.0.77, which ruled out a broken probe, and two of them disagreed on 2.1.14. @@ -252,7 +265,7 @@ being a usable diff. representation, never a fixed precision, never widened to f64. Scoring a port by exact-match count beats any error bound: two kernels shared an identical worst error of 2.682e-7 and differed by 42 exact matches of 512. - (FactorioMapWebUI) + (FactorioMapWebUI, `docs/noise/basis-noise-NOTES.md`) ## The lesson underneath all of these diff --git a/docs/method.md b/docs/method.md index f019a49..aa4f864 100644 --- a/docs/method.md +++ b/docs/method.md @@ -52,14 +52,14 @@ itself. Evaluate every probe against every rule, and report the rules that survive **all** of them, not the one that fits the last result. -The zoom probe (#206) had three candidate rules for what a notch does from an -off-ladder start. All three agree on every sample taken from a rung, so only an -off-rung start says anything, and *which pair* it separates depends on where in -the gap it sits. A start at 0.83 sits below its gap's midpoint, so a notch **in** -rules out "multiply where you are" and cannot tell "snap to the nearest rung then -step" from "move to the next rung in the direction of travel". Both say -0.905724, exactly. A notch **out** from the same value separates them, 0.742997 -against 0.820335, and settled it. +The zoom probe (#206, #211) had three candidate rules for what a notch does +from an off-ladder start. All three agree on every sample taken from a rung, so +only an off-rung start says anything, and *which pair* it separates depends on +where in the gap it sits. A start at 0.83 sits below its gap's midpoint, so a +notch **in** rules out "multiply where you are" and cannot tell "snap to the +nearest rung then step" from "move to the next rung in the direction of +travel". Both say 0.905724, exactly. A notch **out** from the same value +separates them, 0.742997 against 0.820335, and settled it. Reading the first probe as a verdict would have adopted the wrong rule with a number matching to nine digits. @@ -237,4 +237,5 @@ sibling repos, and enforced here by `factorio-oracle provenance check`. `core/lualib/collision-mask-defaults.lua` has `["cargo-bay"] = building_tall()` at 2.0.73, which carries the elevated rail layer, and `building()` at 2.1.12, which does not. A fixture saying only "2.1.12" would have been correct and - still produced the wrong rule. + still produced the wrong rule. Record the cross-check itself in a + `versionDifferences` field, not only the binary the capture came from. diff --git a/docs/order-of-attack.md b/docs/order-of-attack.md index 4a7f6a3..2a6aa06 100644 --- a/docs/order-of-attack.md +++ b/docs/order-of-attack.md @@ -19,10 +19,11 @@ prototype can be hundreds of lines of noise. **Check out the tag that matches what you target, not the newest one.** The four consumer repos target different versions on purpose, and a rule grounded against -the wrong tag can be right about a game nobody is running. factorio-blueprint-editor -learned this the hard way: it targets 2.0.45 to 2.0.73, but its blueprint corpus -spans 2.0.32 to 2.1.12, and two of its twelve files declare 2.1.12. A rule -checked only against the 2.0.73 tag predates those two files. +the wrong tag can be right about a game nobody is running. +factorio-blueprint-editor learned this the hard way: it targets 2.0.45 to +2.0.73, but its blueprint corpus spans 2.0.32 to 2.1.12, and two of its twelve +files declare 2.1.12. A rule checked only against the 2.0.73 tag predates those +two files. **A clone is not a checkout.** `~/GitHub/factorio-data` is one working tree that three repos want at three different tags. Read at a tag without moving `HEAD`