test(top-module): reproduce over-rebuild on unreferenced lib's .mli change#14476
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new blackbox cram test that documents and pins the current dune ocaml top-module rebuild behavior when a library dependency’s interface changes, intended as an observational baseline for future dependency-precision improvements.
Changes:
- Added
cctx-wide-cmi-deps.tto reproduce/pin over-invalidation of the top-module compile rule when an “unreferenced” library’s.cmichanges. - Encoded expected rebuild outcomes (
REBUILT) for both a real dependency change and the over-invalidation probe case.
Match the four sibling tests in this directory
([load-from-{exe,lib}.t], [load-with-{pp,ppx}.t]) which all use
[make_dune_project] for [dune-project] generation. Pinned to
[3.24] to track current dev behaviour.
Addresses Copilot review at
ocaml#14476 (comment).
Signed-off-by: Robin Bate Boerop <me@robinbb.com>
The pipeline's exit status was [head]'s (always zero with input), laundering away [dune build]'s exit status. Failure modes: - [dune build @check] fails noisily: error text appears in pipe output, cram diff catches the unexpected output. Not masked. - [dune build @check] fails silently: pipe output empty, matches expected empty output. Masked. The cap added no value on success (output already empty); on failure it acts as a masking surface. Drop the pipe — the bare [$ dune build @check] catches both modes via the diff. Addresses Copilot review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Previously [m.ml] contained [let tag = Dep_for_intf.Tag], a real textual reference to [Dep_for_intf]. The topmod rule's rebuild on [dep_for_intf.cmi] change was therefore *legitimate*, not the cctx-wide-glob over-invalidation under test. The prose claimed [m.ml] never mentioned [Dep_for_intf]; the code contradicted it. Fix: change [Dep_for_intf.t] from a constructor variant ([type t = Tag]) to a primitive alias ([type t = int]). [m.ml] now constructs [let tag = 42] with no textual reference to [Dep_for_intf]: - ocamldep on [m.ml] reports no [Dep_for_intf] entry. - The topmod compile of [m.ml] (with [.mli] dropped) infers [tag : int] without needing [dep_for_intf.cmi] for type-checking. - The cctx-wide glob still pulls [dep_for_intf.cmi] into the topmod rule's deps — that is the over-invalidation. [Dep_for_intf]'s [.mli]-only Perturbation 2 edit also moves to a primitive shape (adds [val zero : t]), still mutating the cmi. Addresses Copilot review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
…ertions [stat -c '%y'] is GNU coreutils — not portable to macOS/BSD [stat] (which uses [-f]). Replace the mtime-comparison harness with trace-based assertions that test the property directly: "did the topmod compile rule run during the second invocation?" Each [dune ocaml top-module] invocation now writes to a phase-specific trace file ([_build/trace-impl.csexp], [_build/trace-intf.csexp]). After each perturbation, the test queries the trace for any action whose target matches [\\.topmod/mylib/m\\.ml/mylib__M\\.cmo$]. A non-empty array means the rule re-evaluated; an empty array would mean it stayed cached. Drops [stat -c] entirely and asserts the property under observation rather than a file-system side-effect. Addresses Copilot review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
|
The title, PR text, test title, and test paragraphs sound way too verbose to me, it's hard to follow. I've left a few comments, sorry if they sound harsh but AI-generated PRs like this need adversarial reviews |
Replace "Observational baseline: [dune ocaml top-module] over-invalidates the top-module compile when an unreferenced library's interface changes." with "Reproduction: `dune ocaml top-module` over-rebuilds when an unreferenced library's interface changes." — adopting the term, code formatting, and verb from ElectreAAS's suggestion, while keeping the original phrase about which library is involved. Addresses review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Per ElectreAAS's review of ocaml#14476: - r3221893397: drop the "This test pins..." paragraph and fold its substance into the `Reproduction:` opener ("The test passes while the bug is present; promote when fixed."). Preserves the polarity signal and maintainer hint in fewer words. - r3221942293: reword Control and Probe prose to lead with the `.mli` change. Both sections edit both `.ml` and `.mli`; the `.ml` change is scaffolding so the package builds, while the `.mli` change is what alters the `.cmi` and triggers the rebuild under test. - r3221908837: simplify `mylib/m.ml` from two values to one (`let tag = Dep_for_impl.value`). Same cmi-deps; `tag` is now meaningful instead of throwaway. - r3221888985: trim dune-internals from the cctx paragraph (`Compilation_context.set_obj_dir`, `Module_compilation.build_module`, `Module.set_source`, "cctx"). Keep the dropped-`.mli` precondition and the cmi-deps-from-full-closure fact. Add a labelled "Code under test: `src/dune_rules/top_module.ml`" pointer. Also: convert remaining `[identifier]` code-spans to backticks (follow-up from the r3221878493 thread; backticks are the ~82% majority convention in this repo). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
It is a "baseline" meaning that it what happens currently, and we expect a change in future (as it is a bug). It is "observed" to emphasise that it is the current behaviour but not the desired behaviour. It's a term I've been using for a whole bunch of tests recently. A better industry standard term is "characterisation tests" (a term coined by Michael Feathers). Really, I want a term that says "this is a regression test that passes, but it is not documenting desired behaviour". |
None sound harsh. Good comments. I have made improvements. See what you think, now, @ElectreAAS. |
This is what I've been calling a reproduction case 🤷♀️ Bar my comment about modifying the .ml file (and maybe the title of the test |
Reviewers asked why the edit steps rewrote both the `.ml` and the `.mli` when the test prose framed the scenario as an interface change (PR ocaml#14476, r3221942293, r3226815617). Switch to the "pre-implement, then expose" pattern: the `.ml` files carry the full set of definitions from the start; the initial `.mli` exposes a subset; the edit step adds a `val …` to the `.mli` alone. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Fair enough. I would expect a "reproduction case" to reproduce the bug (fail the test), that being the difference.
Excellent! Thanks for your work on this. I've made your suggested improvement. Please have another look, @ElectreAAS. |
Match the four sibling tests in this directory
([load-from-{exe,lib}.t], [load-with-{pp,ppx}.t]) which all use
[make_dune_project] for [dune-project] generation. Pinned to
[3.24] to track current dev behaviour.
Addresses Copilot review at
ocaml#14476 (comment).
Signed-off-by: Robin Bate Boerop <me@robinbb.com>
The pipeline's exit status was [head]'s (always zero with input), laundering away [dune build]'s exit status. Failure modes: - [dune build @check] fails noisily: error text appears in pipe output, cram diff catches the unexpected output. Not masked. - [dune build @check] fails silently: pipe output empty, matches expected empty output. Masked. The cap added no value on success (output already empty); on failure it acts as a masking surface. Drop the pipe — the bare [$ dune build @check] catches both modes via the diff. Addresses Copilot review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Previously [m.ml] contained [let tag = Dep_for_intf.Tag], a real textual reference to [Dep_for_intf]. The topmod rule's rebuild on [dep_for_intf.cmi] change was therefore *legitimate*, not the cctx-wide-glob over-invalidation under test. The prose claimed [m.ml] never mentioned [Dep_for_intf]; the code contradicted it. Fix: change [Dep_for_intf.t] from a constructor variant ([type t = Tag]) to a primitive alias ([type t = int]). [m.ml] now constructs [let tag = 42] with no textual reference to [Dep_for_intf]: - ocamldep on [m.ml] reports no [Dep_for_intf] entry. - The topmod compile of [m.ml] (with [.mli] dropped) infers [tag : int] without needing [dep_for_intf.cmi] for type-checking. - The cctx-wide glob still pulls [dep_for_intf.cmi] into the topmod rule's deps — that is the over-invalidation. [Dep_for_intf]'s [.mli]-only Perturbation 2 edit also moves to a primitive shape (adds [val zero : t]), still mutating the cmi. Addresses Copilot review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
…ertions [stat -c '%y'] is GNU coreutils — not portable to macOS/BSD [stat] (which uses [-f]). Replace the mtime-comparison harness with trace-based assertions that test the property directly: "did the topmod compile rule run during the second invocation?" Each [dune ocaml top-module] invocation now writes to a phase-specific trace file ([_build/trace-impl.csexp], [_build/trace-intf.csexp]). After each perturbation, the test queries the trace for any action whose target matches [\\.topmod/mylib/m\\.ml/mylib__M\\.cmo$]. A non-empty array means the rule re-evaluated; an empty array would mean it stayed cached. Drops [stat -c] entirely and asserts the property under observation rather than a file-system side-effect. Addresses Copilot review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Replace "Observational baseline: [dune ocaml top-module] over-invalidates the top-module compile when an unreferenced library's interface changes." with "Reproduction: `dune ocaml top-module` over-rebuilds when an unreferenced library's interface changes." — adopting the term, code formatting, and verb from ElectreAAS's suggestion, while keeping the original phrase about which library is involved. Addresses review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Per ElectreAAS's review of ocaml#14476: - r3221893397: drop the "This test pins..." paragraph and fold its substance into the `Reproduction:` opener ("The test passes while the bug is present; promote when fixed."). Preserves the polarity signal and maintainer hint in fewer words. - r3221942293: reword Control and Probe prose to lead with the `.mli` change. Both sections edit both `.ml` and `.mli`; the `.ml` change is scaffolding so the package builds, while the `.mli` change is what alters the `.cmi` and triggers the rebuild under test. - r3221908837: simplify `mylib/m.ml` from two values to one (`let tag = Dep_for_impl.value`). Same cmi-deps; `tag` is now meaningful instead of throwaway. - r3221888985: trim dune-internals from the cctx paragraph (`Compilation_context.set_obj_dir`, `Module_compilation.build_module`, `Module.set_source`, "cctx"). Keep the dropped-`.mli` precondition and the cmi-deps-from-full-closure fact. Add a labelled "Code under test: `src/dune_rules/top_module.ml`" pointer. Also: convert remaining `[identifier]` code-spans to backticks (follow-up from the r3221878493 thread; backticks are the ~82% majority convention in this repo). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Reviewers asked why the edit steps rewrote both the `.ml` and the `.mli` when the test prose framed the scenario as an interface change (PR ocaml#14476, r3221942293, r3226815617). Switch to the "pre-implement, then expose" pattern: the `.ml` files carry the full set of definitions from the start; the initial `.mli` exposes a subset; the edit step adds a `val …` to the `.mli` alone. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
|
All good, I'd just change the name of the test: |
033f154 to
7db6ba7
Compare
Match the four sibling tests in this directory
([load-from-{exe,lib}.t], [load-with-{pp,ppx}.t]) which all use
[make_dune_project] for [dune-project] generation. Pinned to
[3.24] to track current dev behaviour.
Addresses Copilot review at
ocaml#14476 (comment).
Signed-off-by: Robin Bate Boerop <me@robinbb.com>
The pipeline's exit status was [head]'s (always zero with input), laundering away [dune build]'s exit status. Failure modes: - [dune build @check] fails noisily: error text appears in pipe output, cram diff catches the unexpected output. Not masked. - [dune build @check] fails silently: pipe output empty, matches expected empty output. Masked. The cap added no value on success (output already empty); on failure it acts as a masking surface. Drop the pipe — the bare [$ dune build @check] catches both modes via the diff. Addresses Copilot review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Previously [m.ml] contained [let tag = Dep_for_intf.Tag], a real textual reference to [Dep_for_intf]. The topmod rule's rebuild on [dep_for_intf.cmi] change was therefore *legitimate*, not the cctx-wide-glob over-invalidation under test. The prose claimed [m.ml] never mentioned [Dep_for_intf]; the code contradicted it. Fix: change [Dep_for_intf.t] from a constructor variant ([type t = Tag]) to a primitive alias ([type t = int]). [m.ml] now constructs [let tag = 42] with no textual reference to [Dep_for_intf]: - ocamldep on [m.ml] reports no [Dep_for_intf] entry. - The topmod compile of [m.ml] (with [.mli] dropped) infers [tag : int] without needing [dep_for_intf.cmi] for type-checking. - The cctx-wide glob still pulls [dep_for_intf.cmi] into the topmod rule's deps — that is the over-invalidation. [Dep_for_intf]'s [.mli]-only Perturbation 2 edit also moves to a primitive shape (adds [val zero : t]), still mutating the cmi. Addresses Copilot review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
…ertions [stat -c '%y'] is GNU coreutils — not portable to macOS/BSD [stat] (which uses [-f]). Replace the mtime-comparison harness with trace-based assertions that test the property directly: "did the topmod compile rule run during the second invocation?" Each [dune ocaml top-module] invocation now writes to a phase-specific trace file ([_build/trace-impl.csexp], [_build/trace-intf.csexp]). After each perturbation, the test queries the trace for any action whose target matches [\\.topmod/mylib/m\\.ml/mylib__M\\.cmo$]. A non-empty array means the rule re-evaluated; an empty array would mean it stayed cached. Drops [stat -c] entirely and asserts the property under observation rather than a file-system side-effect. Addresses Copilot review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Replace "Observational baseline: [dune ocaml top-module] over-invalidates the top-module compile when an unreferenced library's interface changes." with "Reproduction: `dune ocaml top-module` over-rebuilds when an unreferenced library's interface changes." — adopting the term, code formatting, and verb from ElectreAAS's suggestion, while keeping the original phrase about which library is involved. Addresses review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Per ElectreAAS's review of ocaml#14476: - r3221893397: drop the "This test pins..." paragraph and fold its substance into the `Reproduction:` opener ("The test passes while the bug is present; promote when fixed."). Preserves the polarity signal and maintainer hint in fewer words. - r3221942293: reword Control and Probe prose to lead with the `.mli` change. Both sections edit both `.ml` and `.mli`; the `.ml` change is scaffolding so the package builds, while the `.mli` change is what alters the `.cmi` and triggers the rebuild under test. - r3221908837: simplify `mylib/m.ml` from two values to one (`let tag = Dep_for_impl.value`). Same cmi-deps; `tag` is now meaningful instead of throwaway. - r3221888985: trim dune-internals from the cctx paragraph (`Compilation_context.set_obj_dir`, `Module_compilation.build_module`, `Module.set_source`, "cctx"). Keep the dropped-`.mli` precondition and the cmi-deps-from-full-closure fact. Add a labelled "Code under test: `src/dune_rules/top_module.ml`" pointer. Also: convert remaining `[identifier]` code-spans to backticks (follow-up from the r3221878493 thread; backticks are the ~82% majority convention in this repo). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Reviewers asked why the edit steps rewrote both the `.ml` and the `.mli` when the test prose framed the scenario as an interface change (PR ocaml#14476, r3221942293, r3226815617). Switch to the "pre-implement, then expose" pattern: the `.ml` files carry the full set of definitions from the start; the initial `.mli` exposes a subset; the edit step adds a `val …` to the `.mli` alone. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
|
Thanks. Agreed. Renamed to Back to you, @ElectreAAS. |
…validation [dune ocaml top-module] derives a cctx via [Compilation_context.set_obj_dir] to a private obj_dir and drops [.mli] via [Module.set_source ~ml_kind:Intf None]. The derived cctx's compile rule pulls cmi-deps from the parent library's full [(libraries ...)] closure, so editing any [(libraries ...)] dep's interface invalidates the top-module artefact — even libraries referenced ONLY from the (dropped) [.mli]. This test pins the current REBUILT behaviour. A future change that filters [top_module]'s compile-rule cmi-deps to match the [.ml]'s actual references would flip the assertion to NOT REBUILT and require promotion. Reported and reproduced during review of ocaml#14474. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Match the four sibling tests in this directory
([load-from-{exe,lib}.t], [load-with-{pp,ppx}.t]) which all use
[make_dune_project] for [dune-project] generation. Pinned to
[3.24] to track current dev behaviour.
Addresses Copilot review at
ocaml#14476 (comment).
Signed-off-by: Robin Bate Boerop <me@robinbb.com>
The pipeline's exit status was [head]'s (always zero with input), laundering away [dune build]'s exit status. Failure modes: - [dune build @check] fails noisily: error text appears in pipe output, cram diff catches the unexpected output. Not masked. - [dune build @check] fails silently: pipe output empty, matches expected empty output. Masked. The cap added no value on success (output already empty); on failure it acts as a masking surface. Drop the pipe — the bare [$ dune build @check] catches both modes via the diff. Addresses Copilot review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Previously [m.ml] contained [let tag = Dep_for_intf.Tag], a real textual reference to [Dep_for_intf]. The topmod rule's rebuild on [dep_for_intf.cmi] change was therefore *legitimate*, not the cctx-wide-glob over-invalidation under test. The prose claimed [m.ml] never mentioned [Dep_for_intf]; the code contradicted it. Fix: change [Dep_for_intf.t] from a constructor variant ([type t = Tag]) to a primitive alias ([type t = int]). [m.ml] now constructs [let tag = 42] with no textual reference to [Dep_for_intf]: - ocamldep on [m.ml] reports no [Dep_for_intf] entry. - The topmod compile of [m.ml] (with [.mli] dropped) infers [tag : int] without needing [dep_for_intf.cmi] for type-checking. - The cctx-wide glob still pulls [dep_for_intf.cmi] into the topmod rule's deps — that is the over-invalidation. [Dep_for_intf]'s [.mli]-only Perturbation 2 edit also moves to a primitive shape (adds [val zero : t]), still mutating the cmi. Addresses Copilot review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
…ertions [stat -c '%y'] is GNU coreutils — not portable to macOS/BSD [stat] (which uses [-f]). Replace the mtime-comparison harness with trace-based assertions that test the property directly: "did the topmod compile rule run during the second invocation?" Each [dune ocaml top-module] invocation now writes to a phase-specific trace file ([_build/trace-impl.csexp], [_build/trace-intf.csexp]). After each perturbation, the test queries the trace for any action whose target matches [\\.topmod/mylib/m\\.ml/mylib__M\\.cmo$]. A non-empty array means the rule re-evaluated; an empty array would mean it stayed cached. Drops [stat -c] entirely and asserts the property under observation rather than a file-system side-effect. Addresses Copilot review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Replace "Observational baseline: [dune ocaml top-module] over-invalidates the top-module compile when an unreferenced library's interface changes." with "Reproduction: `dune ocaml top-module` over-rebuilds when an unreferenced library's interface changes." — adopting the term, code formatting, and verb from ElectreAAS's suggestion, while keeping the original phrase about which library is involved. Addresses review at ocaml#14476 (comment). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Per ElectreAAS's review of ocaml#14476: - r3221893397: drop the "This test pins..." paragraph and fold its substance into the `Reproduction:` opener ("The test passes while the bug is present; promote when fixed."). Preserves the polarity signal and maintainer hint in fewer words. - r3221942293: reword Control and Probe prose to lead with the `.mli` change. Both sections edit both `.ml` and `.mli`; the `.ml` change is scaffolding so the package builds, while the `.mli` change is what alters the `.cmi` and triggers the rebuild under test. - r3221908837: simplify `mylib/m.ml` from two values to one (`let tag = Dep_for_impl.value`). Same cmi-deps; `tag` is now meaningful instead of throwaway. - r3221888985: trim dune-internals from the cctx paragraph (`Compilation_context.set_obj_dir`, `Module_compilation.build_module`, `Module.set_source`, "cctx"). Keep the dropped-`.mli` precondition and the cmi-deps-from-full-closure fact. Add a labelled "Code under test: `src/dune_rules/top_module.ml`" pointer. Also: convert remaining `[identifier]` code-spans to backticks (follow-up from the r3221878493 thread; backticks are the ~82% majority convention in this repo). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Reviewers asked why the edit steps rewrote both the `.ml` and the `.mli` when the test prose framed the scenario as an interface change (PR ocaml#14476, r3221942293, r3226815617). Switch to the "pre-implement, then expose" pattern: the `.ml` files carry the full set of definitions from the start; the initial `.mli` exposes a subset; the edit step adds a `val …` to the `.mli` alone. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
7db6ba7 to
a90d023
Compare
Summary
Adds a regression test under
test/blackbox-tests/test-cases/top-module/that reproduces adune ocaml top-moduleover-rebuild: editing the interface of a library referenced only from.mli(never from.ml) rebuilds the top-module artefact, even thoughtop-moduledrops the.mli. The test passes today while the bug is present.Related: #14477.