Skip to content

fix(hooks): stop unit tests reading the developer's config.toml - #3461

Open
David-Martel wants to merge 2 commits into
rtk-ai:developfrom
David-Martel:fix/hermetic-hook-tests-20260806
Open

fix(hooks): stop unit tests reading the developer's config.toml#3461
David-Martel wants to merge 2 commits into
rtk-ai:developfrom
David-Martel:fix/hermetic-hook-tests-20260806

Conversation

@David-Martel

@David-Martel David-Martel commented Aug 6, 2026

Copy link
Copy Markdown

Problem

cargo test is red for contributors who use rtk and green in CI, which is why this hasn't
surfaced upstream. Two places in the hook decision path do file I/O, so unit tests inherit the
contributor's machine configuration.

1. get_rewritten calls Config::load() — reads $CONFIG_DIR/rtk/config.toml. With a
configured

[hooks]
exclude_commands = ["git", "ls", "grep", ...]

commands the tests expect to be rewritten are excluded, get_rewritten returns None, and the
decision comes back Defer. test_claude_rewrite_git_status — which asserts git status
rtk git status — fails, along with 22 others spanning every host handler (claude, cursor,
gemini, droid, vibe, copilot), since they all funnel through decide_from_verdict.

2. evaluate calls check_command — reads ~/.claude/settings.json and
$PROJECT_ROOT/.claude/settings.json. The unattestable_passthrough tests pass explicit empty
exclusions but still inherit that ambient verdict, so with a matching deny rule the outcome is
Deny instead of Ask and 2 more tests fail.

Excluding git is a reasonable real-world config — it's what you'd set to keep exact git
output for a coding agent — so this is not an exotic setup.

Measured

Pristine checkout of develop, no source changes, only the config file moved aside:

condition hooks::hook_cmd
config.toml present with [hooks] exclude_commands 23 of 108 fail
same file moved aside 108 pass, 0 fail

Fix

Split file I/O out of the decision path, mirroring the convention permissions already
established with check_command_with_rules — whose doc comment reads "Internal implementation
allowing tests to inject rules without file I/O."
The seam existed; these paths just didn't
use it.

commit change
stop unit tests reading the developer's config.toml get_rewritten_with / decide_from_verdict_with take exclusions as parameters; hook_exclusions() becomes the single loader and is #[cfg(test)]-stubbed to empty so no test path reaches the filesystem.
stop rewrite tests reading the host's permission settings evaluate_with_verdict takes the verdict as a parameter; evaluate loads it and delegates; the test module supplies PermissionVerdict::Default.

Production behavior is unchanged in both cases — the loading wrappers still load. The only
behavioral difference is inside #[cfg(test)].

Added test_get_rewritten_with_honors_exclusions so the exclusion behavior the other tests now
bypass stays pinned rather than silently untested.

Verification

Run with a populated config.toml and settings.json in place — the condition that used to
fail:

  • unit test binary: 25 failures → 0 (2571 passed, 8 ignored)
  • cargo clippy --release --all-targets -- -D warnings — clean
  • cargo fmt --all -- --check — clean

Note for maintainers

Two related issues found while investigating, not addressed here:

  1. tests/copilot_selfheal_test.rs (added in fix(hooks): copilot self heal dual hooks (drop camelCase entry) #3449) fails 10 of 12 on a pristine develop
    checkout
    on a machine with existing Copilot hook config — same class of ambient-state
    dependency, in brand-new code. Happy to extend this PR to cover it if you'd like.
  2. core::tracking tests are flaky under full-suite parallelism — they share one real SQLite
    tracking DB. They pass 14/14 in isolation, both parallel and single-threaded.

Copilot AI lite review requested due to automatic review settings August 6, 2026 21:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

David-Martel and others added 2 commits August 6, 2026 17:04
`get_rewritten` called `Config::load()`, so the hook decision path read
`$CONFIG_DIR/rtk/config.toml` from inside unit tests. On a machine with
`[hooks] exclude_commands` set, commands the tests expect to be rewritten
were excluded, and the decision came back `Defer` instead of a rewrite.

Measured against a pristine checkout: 23 of the 108 `hooks::hook_cmd` tests
fail with such a config present and all 108 pass with the file moved aside.
CI stays green only because CI has no user config, so `cargo test` is red for
configured contributors and green for the project.

Split the file I/O out of the decision path, mirroring the existing
`check_command_with_rules` convention:

- `get_rewritten_with` / `decide_from_verdict_with` take exclusions as
  parameters and do no I/O.
- `hook_exclusions()` is the single place that loads config, and is
  `#[cfg(test)]`-stubbed to empty so no test path can reach the filesystem.
- `decide_with_rules` (test helper) now passes exclusions explicitly.

Production behavior is unchanged: `decide_from_verdict` still loads the user's
exclusions before delegating. Added `test_get_rewritten_with_honors_exclusions`
so the exclusion behavior the other tests now bypass stays pinned.

Verified with the user config present: 2579 passed, 0 failed; clippy
`-D warnings` clean; `fmt --check` clean.

Agent: claude
Co-authored-by: Claude <noreply@anthropic.com>
`evaluate` called `check_command`, which loads `~/.claude/settings.json` and
`$PROJECT_ROOT/.claude/settings.json`. The `unattestable_passthrough` tests
pass explicit empty exclusions but still inherited that ambient verdict, so on
a machine with a matching deny rule the outcome came back `Deny` instead of
`Ask` and `test_plain_command_still_rewrites` and
`test_fd_dup_redirect_still_rewrites` failed. CI has no such settings, so it
stayed green.

Same treatment as the preceding commit: `evaluate_with_verdict` takes the
verdict as a parameter and does no I/O, `evaluate` loads it and delegates, and
the test module supplies `PermissionVerdict::Default` explicitly.

Production behavior is unchanged.

With both commits applied and a populated user config in place, the unit test
binary goes from 25 failures to 0 (2571 passed).

Agent: claude
Co-authored-by: Claude <noreply@anthropic.com>
@David-Martel
David-Martel force-pushed the fix/hermetic-hook-tests-20260806 branch from 0b0501f to 7c511c7 Compare August 6, 2026 21:21
@KuSh

KuSh commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Hi @David-Martel,

Thanks for your PR. Could you extend it with copilot fix? The flaky DB tests are already fixes in #3206 with commit 2aef8e8

@KuSh KuSh self-assigned this Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants