fix(hooks): stop unit tests reading the developer's config.toml - #3461
Open
David-Martel wants to merge 2 commits into
Open
fix(hooks): stop unit tests reading the developer's config.toml#3461David-Martel wants to merge 2 commits into
David-Martel wants to merge 2 commits into
Conversation
|
|
`get_rewritten` called `Config::load()`, so the hook decision path read `$CONFIG_DIR/rtk/config.toml` from inside unit tests. On a machine with `[hooks] exclude_commands` set, commands the tests expect to be rewritten were excluded, and the decision came back `Defer` instead of a rewrite. Measured against a pristine checkout: 23 of the 108 `hooks::hook_cmd` tests fail with such a config present and all 108 pass with the file moved aside. CI stays green only because CI has no user config, so `cargo test` is red for configured contributors and green for the project. Split the file I/O out of the decision path, mirroring the existing `check_command_with_rules` convention: - `get_rewritten_with` / `decide_from_verdict_with` take exclusions as parameters and do no I/O. - `hook_exclusions()` is the single place that loads config, and is `#[cfg(test)]`-stubbed to empty so no test path can reach the filesystem. - `decide_with_rules` (test helper) now passes exclusions explicitly. Production behavior is unchanged: `decide_from_verdict` still loads the user's exclusions before delegating. Added `test_get_rewritten_with_honors_exclusions` so the exclusion behavior the other tests now bypass stays pinned. Verified with the user config present: 2579 passed, 0 failed; clippy `-D warnings` clean; `fmt --check` clean. Agent: claude Co-authored-by: Claude <noreply@anthropic.com>
`evaluate` called `check_command`, which loads `~/.claude/settings.json` and `$PROJECT_ROOT/.claude/settings.json`. The `unattestable_passthrough` tests pass explicit empty exclusions but still inherited that ambient verdict, so on a machine with a matching deny rule the outcome came back `Deny` instead of `Ask` and `test_plain_command_still_rewrites` and `test_fd_dup_redirect_still_rewrites` failed. CI has no such settings, so it stayed green. Same treatment as the preceding commit: `evaluate_with_verdict` takes the verdict as a parameter and does no I/O, `evaluate` loads it and delegates, and the test module supplies `PermissionVerdict::Default` explicitly. Production behavior is unchanged. With both commits applied and a populated user config in place, the unit test binary goes from 25 failures to 0 (2571 passed). Agent: claude Co-authored-by: Claude <noreply@anthropic.com>
David-Martel
force-pushed
the
fix/hermetic-hook-tests-20260806
branch
from
August 6, 2026 21:21
0b0501f to
7c511c7
Compare
Collaborator
|
Hi @David-Martel, Thanks for your PR. Could you extend it with copilot fix? The flaky DB tests are already fixes in #3206 with commit 2aef8e8 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
cargo testis red for contributors who use rtk and green in CI, which is why this hasn'tsurfaced upstream. Two places in the hook decision path do file I/O, so unit tests inherit the
contributor's machine configuration.
1.
get_rewrittencallsConfig::load()— reads$CONFIG_DIR/rtk/config.toml. With aconfigured
commands the tests expect to be rewritten are excluded,
get_rewrittenreturnsNone, and thedecision comes back
Defer.test_claude_rewrite_git_status— which assertsgit status→rtk git status— fails, along with 22 others spanning every host handler (claude, cursor,gemini, droid, vibe, copilot), since they all funnel through
decide_from_verdict.2.
evaluatecallscheck_command— reads~/.claude/settings.jsonand$PROJECT_ROOT/.claude/settings.json. Theunattestable_passthroughtests pass explicit emptyexclusions but still inherit that ambient verdict, so with a matching deny rule the outcome is
Denyinstead ofAskand 2 more tests fail.Excluding
gitis a reasonable real-world config — it's what you'd set to keep exactgitoutput for a coding agent — so this is not an exotic setup.
Measured
Pristine checkout of
develop, no source changes, only the config file moved aside:hooks::hook_cmdconfig.tomlpresent with[hooks] exclude_commandsFix
Split file I/O out of the decision path, mirroring the convention
permissionsalreadyestablished with
check_command_with_rules— whose doc comment reads "Internal implementationallowing tests to inject rules without file I/O." The seam existed; these paths just didn't
use it.
stop unit tests reading the developer's config.tomlget_rewritten_with/decide_from_verdict_withtake exclusions as parameters;hook_exclusions()becomes the single loader and is#[cfg(test)]-stubbed to empty so no test path reaches the filesystem.stop rewrite tests reading the host's permission settingsevaluate_with_verdicttakes the verdict as a parameter;evaluateloads it and delegates; the test module suppliesPermissionVerdict::Default.Production behavior is unchanged in both cases — the loading wrappers still load. The only
behavioral difference is inside
#[cfg(test)].Added
test_get_rewritten_with_honors_exclusionsso the exclusion behavior the other tests nowbypass stays pinned rather than silently untested.
Verification
Run with a populated
config.tomlandsettings.jsonin place — the condition that used tofail:
cargo clippy --release --all-targets -- -D warnings— cleancargo fmt --all -- --check— cleanNote for maintainers
Two related issues found while investigating, not addressed here:
tests/copilot_selfheal_test.rs(added in fix(hooks): copilot self heal dual hooks (drop camelCase entry) #3449) fails 10 of 12 on a pristinedevelopcheckout on a machine with existing Copilot hook config — same class of ambient-state
dependency, in brand-new code. Happy to extend this PR to cover it if you'd like.
core::trackingtests are flaky under full-suite parallelism — they share one real SQLitetracking DB. They pass 14/14 in isolation, both parallel and single-threaded.