Record where each fixture came from, and check that the record stays honest

CLAUDE.md

@@ -45,8 +45,16 @@ cargo run -- run  --probe dump-data.json --work-dir /tmp/w > /tmp/run.json
 cargo run -- trim --run /tmp/run.json --spec trim-spec.json --out fixture.json [--check]
 ```
 
-Test counts to expect: **113 unit tests**, plus **6 install-gated integration
-tests** (3 in `tests/acceptance.rs`, 3 in `tests/real_game.rs`).
+Test counts to expect: **159 unit tests**, plus **8 integration tests** split
+across three files. `tests/acceptance.rs` has 3: two run offline against a
+committed fixture, and one (`the_real_install_reproduces_it_too`) is
+install-gated. `tests/provenance.rs` has 2: one always-on, and one gated on
+the `FACTORIO_ORACLE_PROVENANCE_DIR` environment variable naming another
+repo's fixture directory - not an install gate, so it skips even when
+Factorio is present. `tests/real_game.rs` has 3, all install-gated. That is
+**4 install-gated tests** in total. Without a real Factorio install they skip
+rather than fail, so a green run on a machine with no game proves less than it
+looks. Check which happened before trusting it.
 
 ## Layout
 
@@ -63,6 +71,7 @@ tests** (3 in `tests/acceptance.rs`, 3 in `tests/real_game.rs`).
 | `run.rs` | Wiring the pure builders to disk and a spawner |
 | `numbers.rs` | Preserving the bits the game produced |
 | `trim/` | Cutting a full `data.raw` dump down to a consumer's slice |
+| `provenance/` | Which Factorio each fixture came from, and whether that record is still honest |
 
 Five run modes, and **the success predicate differs per mode**: `dump-data`
 (no mod at all, the mod dir exists only to be empty), `create`, `interactive`
@@ -124,6 +133,48 @@ code did. **A fake can only be wrong in the ways its author already considered.*
 - **`loadedMods` cannot come from the active-mods prelude**, because
   `script.active_mods` never reports `core` and the fixtures list it. Grep the
   game's stdout for `Loading mod <name>`.
+- **A real provenance manifest has two keys per entry, not eight.** Measured
+  2026-08-17 across FactorioMapWebUI's 100 entries: every one carries
+  `factorioVersion` and `evidence`, and `factorioBuild`, `branch`,
+  `loadedMods`, `capturedOn`, `capturedBy` and `targetVersionRange` appear zero
+  times. The design sketched all eight. A checker requiring the sketch would
+  reject the only real manifest there is, so the other six are optional and
+  carried through untouched.
+- **`evidence` is free text, and enforcing a grade would reject 48 of 100.**
+  First word across the same 100 entries: `stated` 48, `captured` 34,
+  `RE-CAPTURED` 8, `inferred` 4, `re-captured` 3, `UNDOCUMENTED` 1, and twice
+  it is just `the`. The design's enum was `stated | inferred | unknown`, which
+  accepts 52 of the 100 first words and rejects the other 48. The grade that
+  is real is `factorioVersion: "unknown"`, which is a field, and that is what
+  the ratchet counts.
+- **An extension allowlist is how ground truth goes unrecorded.** MapWebUI's
+  provenance test globs `.json` and `.png`. Its fixture directory also holds 10
+  `.txt` map exchange strings, and **8 of the 10 are read as ground truth** by
+  `decode.spec.ts`, `encode.spec.ts` and `jsonExport.spec.ts`. None has an
+  entry and none can get one while the glob decides. So this tool names every
+  file, and a deliberate non-fixture goes in `notFixtures` with a reason.
+- **Manifest keys are relative paths, not bare filenames.** The design says
+  bare filenames, which is true of MapWebUI's flat directory and was never
+  tested against a tree. This crate's own `tests/fixtures/` is two levels deep.
+  The walk joins components with `/` by hand rather than using
+  `Path::display()`, so a Windows run and a macOS run produce the same
+  committed key.
+- **The walk skips dotfiles and dot-directories, plus two Windows names, and
+  that is load-bearing on both platforms.** `.DS_Store` appears in any
+  directory a Finder window has opened; demanding an entry for it would make
+  the check fail on the machine that can fix it and pass in CI. The skip
+  `continue`s before the directory branch, so a dot-prefixed name is skipped
+  whole even when it is a directory - a consumer keeping fixtures under
+  `.golden/` gets nothing recorded for that subtree, not an error. `Thumbs.db`
+  and `desktop.ini` (matched case-insensitively) are skipped for the same
+  reason on Windows: Explorer writes `Thumbs.db` into any folder of images it
+  has thumbnailed, and MapWebUI's fixture directory holds `.png` files.
+- **`#[serde(flatten)]` works under `arbitrary_precision`.** Measured
+  2026-08-17: a struct with two named string fields plus a flattened
+  `BTreeMap<String, Value>` parsed a document holding both an integer and a
+  decimal, with no error. Recorded as a negative result so nobody spends an
+  afternoon ruling it out. Provenance entries stay a `Value` anyway, because
+  only two keys are required and the rest must round-trip untouched.
 
 ### Writing Lua for a probe
 

README.md

@@ -78,6 +78,45 @@ Practical notes, measured 2026-08-17:
   never gate a capture: captures must stay reproducible offline and byte for
   byte.
 
+## Provenance
+
+A fixture that does not say which Factorio produced it is a number with no
+claim attached. Provenance records that, in a `PROVENANCE.json` beside the
+fixtures rather than inside them: most fixtures are verbatim copies of the
+game's own JSON, asserted byte for byte, so a metadata key added inside one
+would be data pollution.
+
+```bash
+# Structural check. No Factorio needed. Exits 1 on a finding.
+factorio-oracle provenance check tests/fixtures
+
+# Version comparison. Needs an install. Always exits 0.
+factorio-oracle provenance report tests/fixtures
+```
+
+Enforcement splits in two on purpose. `check` answers questions a machine can
+settle - is every file recorded, does every entry name a file that exists, is
+every entry well formed, and has the `unknown` count grown - so it runs in CI
+with no game. `report` answers a question a machine cannot: a fixture captured
+on 2.1.11 is not wrong because the binary moved on, it just has not been
+re-validated since, and whether that matters depends on whether the subsystem
+changed. So it never fails a build.
+
+Two required keys per entry, `factorioVersion` and `evidence`. Any other key is
+carried through and never validated. `evidence` is free text; the grade that is
+enforced is `factorioVersion: "unknown"`, and `maxUnknown` caps how many entries
+may say it. That number is a ratchet, not a cap: the check fails if the count
+rises above it and also if the count drops below it and the number is not
+lowered.
+
+Every file in the tree needs an entry. A file that is deliberately not ground
+truth goes in `notFixtures` with a reason. There is no extension filter,
+because an ignore rule that costs nothing gets used without thinking.
+
+**A fixture's provenance is a record of the moment it was captured, not a live
+claim.** Never edit one to make it current, and never edit one to make a test
+pass. A mismatch is a finding.
+
 ## Examples
 
 - [`examples/pumpjack-terminals`](examples/pumpjack-terminals) - a `create` probe

src/install.rs

@@ -152,6 +152,39 @@ pub fn discover(home: &Path, env_bin: Option<&Path>) -> Vec<DiscoveredInstall> {
         .collect()
 }
 
+/// Whether a discovered install answers to `version`.
+///
+/// An install whose binary would not run has no version, and is never picked:
+/// every command here needs the version, either to stamp it or to build a mod
+/// that declares it.
+pub fn matches_version(found: &DiscoveredInstall, version: Option<&str>) -> bool {
+    match (version, &found.version) {
+        (Some(want), Some(got)) => got.triple() == want,
+        (None, Some(_)) => true,
+        _ => false,
+    }
+}
+
+/// Picks one install.
+///
+/// `factorio` wins over `FACTORIO_BIN`, and either is offered as an extra
+/// candidate root rather than as the only one, which is what `run` has always
+/// done. With no version given, the first install that reported one wins.
+///
+/// Parameters are ordered `factorio` before `env_bin` deliberately, matching
+/// the precedence documented above - so the signature itself shows which one
+/// wins, rather than relying on a reader to check the body.
+pub fn select(
+    home: &Path,
+    factorio: Option<&Path>,
+    env_bin: Option<&Path>,
+    version: Option<&str>,
+) -> Option<DiscoveredInstall> {
+    discover(home, factorio.or(env_bin))
+        .into_iter()
+        .find(|d| matches_version(d, version))
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -322,4 +355,39 @@ mod tests {
         seen.dedup();
         assert_eq!(seen.len(), roots.len(), "duplicate candidate in {roots:?}");
     }
+
+    fn discovered(version_line: Option<&str>) -> DiscoveredInstall {
+        DiscoveredInstall {
+            layout: InstallLayout {
+                root: PathBuf::from("/somewhere"),
+                binary: PathBuf::from("/somewhere/bin/x64/factorio"),
+                data_dir: PathBuf::from("/somewhere/data"),
+                doc_dir: PathBuf::from("/somewhere/doc-html"),
+            },
+            version: version_line.and_then(crate::version::parse_version_line),
+        }
+    }
+
+    #[test]
+    fn an_exact_version_is_what_matches() {
+        let found = discovered(Some("Version: 2.1.14 (build 87180, mac-arm64, steam)"));
+        assert!(matches_version(&found, Some("2.1.14")));
+        assert!(!matches_version(&found, Some("2.1.13")));
+        // major.minor is what a mod declares, not what selects an install.
+        assert!(!matches_version(&found, Some("2.1")));
+    }
+
+    #[test]
+    fn no_version_asked_for_takes_any_install_that_has_one() {
+        assert!(matches_version(
+            &discovered(Some("Version: 2.0.77 (build 84539, mac-arm64, full)")),
+            None
+        ));
+    }
+
+    #[test]
+    fn an_install_whose_binary_will_not_run_is_never_picked() {
+        assert!(!matches_version(&discovered(None), None));
+        assert!(!matches_version(&discovered(None), Some("2.1.14")));
+    }
 }

src/lib.rs

@@ -12,6 +12,7 @@ pub mod lua;
 pub mod numbers;
 pub mod outcome;
 pub mod probe;
+pub mod provenance;
 pub mod run;
 pub mod scaffold;
 pub mod spawn;

src/main.rs

@@ -57,6 +57,11 @@ enum Command {
         #[arg(long)]
         check: bool,
     },
+    /// Check and report on fixture provenance
+    Provenance {
+        #[command(subcommand)]
+        action: ProvenanceAction,
+    },
 }
 
 #[derive(Subcommand)]
@@ -65,6 +70,28 @@ enum InstallsAction {
     List,
 }
 
+#[derive(Subcommand)]
+enum ProvenanceAction {
+    /// Check a fixture directory against its PROVENANCE.json. Needs no
+    /// Factorio, and exits 1 on any finding.
+    Check {
+        /// The fixture directory. Its manifest is the PROVENANCE.json inside it.
+        dir: PathBuf,
+    },
+    /// Compare each fixture's recorded version against an install. Always
+    /// exits 0, because deciding whether a version gap matters needs a human.
+    Report {
+        /// The fixture directory
+        dir: PathBuf,
+        /// Select an install by version, for example 2.0.77
+        #[arg(long)]
+        version: Option<String>,
+        /// Select an install by path
+        #[arg(long)]
+        factorio: Option<PathBuf>,
+    },
+}
+
 fn main() -> anyhow::Result<()> {
     let cli = Cli::parse();
     match cli.command {
@@ -83,7 +110,7 @@ fn main() -> anyhow::Result<()> {
                         "binary": d.layout.binary,
                         "dataDir": d.layout.data_dir,
                         "docDir": d.layout.doc_dir,
-                        "version": d.version.as_ref().map(|v| format!("{}.{}.{}", v.major, v.minor, v.patch)),
+                        "version": d.version.as_ref().map(|v| v.triple()),
                         "modFactorioVersion": d.version.as_ref().map(|v| v.major_minor()),
                         "buildLine": d.version.as_ref().map(|v| v.line.clone()),
                     })
@@ -107,17 +134,13 @@ fn main() -> anyhow::Result<()> {
             let spec: factorio_oracle::probe::ProbeSpec =
                 serde_json::from_str(&std::fs::read_to_string(&probe)?)?;
 
-            let installs = install::discover(&home, factorio.as_deref().or(env_bin.as_deref()));
-            let chosen = installs
-                .into_iter()
-                .find(|d| match (&version, &d.version) {
-                    (Some(want), Some(got)) => {
-                        format!("{}.{}.{}", got.major, got.minor, got.patch) == *want
-                    }
-                    (None, Some(_)) => true,
-                    _ => false,
-                })
-                .ok_or_else(|| anyhow::anyhow!("no Factorio install matched"))?;
+            let chosen = install::select(
+                &home,
+                factorio.as_deref(),
+                env_bin.as_deref(),
+                version.as_deref(),
+            )
+            .ok_or_else(|| anyhow::anyhow!("no Factorio install matched"))?;
 
             let work = match work_dir {
                 Some(dir) => {
@@ -222,6 +245,65 @@ fn main() -> anyhow::Result<()> {
                 println!("Wrote {}", out.display());
             }
         }
+        Command::Provenance {
+            action: ProvenanceAction::Check { dir },
+        } => {
+            let manifest = factorio_oracle::provenance::manifest::load(&dir)?;
+            let on_disk = factorio_oracle::provenance::walk_fixtures(&dir)?;
+            let report = factorio_oracle::provenance::check::check(&manifest, &on_disk);
+
+            println!("{}", serde_json::to_string_pretty(&report.to_json(&dir))?);
+
+            if !report.ok() {
+                // The JSON is the interface and the summary is the error
+                // message. A consumer's CI prints stderr on a failure and
+                // nothing else, so a bare exit code would say only that
+                // something is wrong.
+                eprintln!("{} provenance findings:", dir.display());
+                for line in report.summary() {
+                    eprintln!("  {line}");
+                }
+                std::process::exit(1);
+            }
+        }
+        Command::Provenance {
+            action:
+                ProvenanceAction::Report {
+                    dir,
+                    version,
+                    factorio,
+                },
+        } => {
+            let manifest = factorio_oracle::provenance::manifest::load(&dir)?;
+            let home = PathBuf::from(std::env::var("HOME").unwrap_or_default());
+            let env_bin = std::env::var_os("FACTORIO_BIN").map(PathBuf::from);
+
+            // No install is an error, because there is nothing to compare
+            // against. Every comparison result is not: the whole point of this
+            // half is that a version gap is a finding for a human, not a
+            // failing build.
+            let chosen = install::select(
+                &home,
+                factorio.as_deref(),
+                env_bin.as_deref(),
+                version.as_deref(),
+            )
+            .ok_or_else(|| {
+                anyhow::anyhow!(
+                    "no Factorio install matched, so there is nothing to compare against"
+                )
+            })?;
+            let found = chosen
+                .version
+                .expect("select filters to installs with a version");
+
+            print!(
+                "{}",
+                factorio_oracle::provenance::report::render(
+                    &factorio_oracle::provenance::report::compare(&manifest, &found.triple())
+                )
+            );
+        }
     }
     Ok(())
 }