Watcher keeps indexing newly-ignored files: `.ignore` changes don't trigger an ignore-rules refresh

[shengyfu]

Summary

While tgrep serve is running, the file watcher does not notice changes to .ignore files. As a result the watcher keeps applying the ignore rules it loaded at startup and happily indexes files and folders that a newly added/edited .ignore says to exclude — so the live overlay ends up containing content that a fresh tgrep index over the same tree would never index.

Root cause

tgrep-cli/src/serve.rs:

fn is_ignore_rules_file(root: &Path, path: &Path) -> bool {
    path.file_name().and_then(|name| name.to_str()) == Some(".gitignore")
        || path == root.join(tgrep_core::gitignore::P4IGNORE_FILENAME)
}

handle_fs_event uses this predicate to decide whether an event should set ignore_rules_dirty and schedule schedule_ignore_rules_refresh, which is what rebuilds and republishes ServerState::gitignore.

The predicate covers .gitignore (at any depth) and root-level p4ignore.ini, but it omits .ignore, even though .ignore is a first-class ignore source everywhere else in the codebase:

• walker::walk_dir collects ignore_files separately from gitignore_files.
• gitignore::matcher_from_ignore_paths_with_options feeds both into the matcher, and .ignore even outranks .gitignore.
• Unlike .gitignore, .ignore is not git-gated — it applies outside a repo too, which is exactly the case covered by tgrep-cli/tests/watcher_dot_ignore.rs.

Because the event never matches, the .ignore write also falls through to the normal reindex path where should_skip_watcher_path drops it (any dot-prefixed segment is skipped), so nothing at all happens.

Impact

• Create or edit .ignore while the server is up → files under the newly ignored directory keep getting indexed and remain searchable.
• Delete or relax .ignore → the stale matcher keeps excluding paths that should now be indexed, until the next full stale check or restart.
• In-memory overlay and on-disk index diverge, which is the exact class of drift should_skip_watcher_path exists to prevent.

.gitignore and root p4ignore.ini do not have this problem; only .ignore is missed.

Reproduction

1. tgrep index <root> --index-path <idx> on a tree with no .git directory.
2. tgrep serve --index-path <idx> <root>, wait for the watcher to come up.
3. echo "secret/" > <root>/.ignore
4. mkdir <root>/secret && echo "leak_marker" > <root>/secret/creds.txt
5. Search leak_marker → it matches. Expected: no matches, because secret/ is excluded by .ignore.

The existing tgrep-cli/tests/watcher_dot_ignore.rs passes only because its .ignore exists before the initial index/walk, so the startup matcher already contains the rule.

Expected behavior

A create/modify/remove event for a .ignore file should be treated exactly like a .gitignore event: mark ignore_rules_dirty and schedule the background stale refresh so the matcher is rebuilt and republished.

Proposed fix

Extend is_ignore_rules_file to recognize .ignore alongside .gitignore (matching by file name at any depth, since nested .ignore files are honored by the matcher), keeping p4ignore.ini root-scoped as it is today. Add unit coverage in serve.rs plus an end-to-end test that writes .ignore after the server is live.

[shengyfu]

Field confirmation from the Office monorepo team, which matches this issue exactly:

In a controlled Git repo with 40,212 physical directories but only 200 searchable files after .gitignore, the real Copilot CLI registered exactly 40,212 Linux inotify watches. [...] it even watched .git despite passing --exclude .git. With --no-watch, startup was immediate and it registered zero watches. The bundled tgrep 1.0.2 [...] still starts a recursive watch on the root before applying ignore and exclude filters to delivered events.

The 1:1 ratio between directories and watches is the signature of the root cause: inotify has no recursive mode, so notify's RecursiveMode::Recursive walks the tree and spends one watch descriptor per directory, then our code filters the resulting events on delivery. Filtering after the OS has already paid for the subscription cannot reduce the count.

Two consequences worth spelling out, because they are worse than wasted descriptors:

• The watcher can fail outright. notify's registration loop propagates the first failure with ?. At 40,212 watches against a common fs.inotify.max_user_watches default of 65,536 — shared across every process on the machine, including editors and language servers — a slightly larger repo or a second concurrent tool pushes registration over the limit, watch() returns an error, and the server silently falls back to the hourly reconcile with no watching at all.
• It explains the startup delay. Registration walks the tree and issues a syscall per directory before the server is usable, which is exactly why --no-watch "started immediately".

#105 fixes this by subscribing per directory on inotify backends and pruning ignored, hidden and --excluded subtrees before descending into them. Measured against a reproduction of the reported layout (40,210 directories, ~200 searchable files, .git both hidden and --excluded):

	Directories watched	Time to compute
Before	40,212	walk + one syscall each
After	202	13.7 ms

Pruning happens before descent, so the cost tracks the surviving tree rather than the physical one — the same measurement over 8,458 directories takes 12.6 ms. The reported .git symptom is covered twice, as a hidden directory and as an --exclude match.

Note for the reporting team: the repo is at version 1.0.2, so the fix is unreleased — #105 needs review, merge and a release before it reaches the bundled CLI. Until then --no-watch is a sound workaround; it trades live index updates for the periodic reconcile.

One honest caveat: the reduction is proportional to how much of the tree is pruned by directory-level ignore rules. If a repo's directories are mostly tracked but its files are excluded by file patterns, those directories are still watched — correctly, since a new source file can appear in any of them. The new [trace] watcher subscriptions: N directories line reports the real number, so this is verifiable on the actual monorepo rather than assumed.