Skip to content

fix: add dbt_packages and dbt_modules to file watcher ignore list#501

Open
anandgupta42 wants to merge 1 commit intomainfrom
fix/dbt-packages-emfile-watcher
Open

fix: add dbt_packages and dbt_modules to file watcher ignore list#501
anandgupta42 wants to merge 1 commit intomainfrom
fix/dbt-packages-emfile-watcher

Conversation

@anandgupta42
Copy link
Contributor

@anandgupta42 anandgupta42 commented Mar 27, 2026

Summary

Fixes #500

  • Add dbt_packages and dbt_modules to the FOLDERS ignore set in the file watcher, preventing EMFILE: too many open files errors on macOS when working in dbt projects with many dependencies
  • The @parcel/watcher was recursively watching all files inside dbt_packages/ (often 443+ macro files), exhausting file descriptor limits via fs-events
  • Follows the same pattern as node_modules, vendor, __pycache__, etc.

Test plan

  • Unit tests: dbt_packages/dbt_modules matched by FileIgnore.match at any nesting depth
  • Unit tests: FileIgnore.PATTERNS array contains both directories (used by watcher)
  • E2E test: simulated dbt project with realistic directory tree — user files NOT ignored, dependency files ARE ignored
  • E2E test: 450 simulated files across 6 packages all correctly ignored (fd exhaustion prevention)
  • False-positive tests: filenames containing "dbt_packages" as substring are NOT matched
  • All 178 existing file tests still pass

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

  • New Features

    • DBT dependency directories (dbt_packages and dbt_modules) are now automatically excluded from file watching and ignore operations.
  • Tests

    • Added comprehensive test coverage validating proper exclusion of DBT dependency directories across various project configurations and path patterns.

…#500)

The `@parcel/watcher` file watcher was watching all files inside
`dbt_packages/` recursively, exhausting macOS file descriptor limits
(EMFILE) in projects with many dbt dependencies.

- Add `dbt_packages` and `dbt_modules` to the `FOLDERS` ignore set
- Add unit tests for dbt directory matching in `FileIgnore.match`
- Add e2e tests simulating realistic dbt project structures
- Verify false-positive prevention for filenames containing `dbt_packages`

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link

@claude claude bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

@coderabbitai
Copy link

coderabbitai bot commented Mar 27, 2026

📝 Walkthrough

Walkthrough

These changes extend the file ignore system to exclude DBT dependency directories (dbt_packages and dbt_modules) from file watching, addressing file descriptor exhaustion issues. Test coverage validates proper handling of these directories and their nested contents while preventing false positives.

Changes

Cohort / File(s) Summary
DBT Directory Ignore Configuration
packages/opencode/src/file/ignore.ts
Added dbt_packages and dbt_modules to FileIgnore.FOLDERS constant to exclude these directories from file watching patterns.
File Ignore Test Coverage
packages/opencode/test/file/ignore.test.ts
Added test cases verifying FileIgnore.match() correctly ignores dbt directories, handles Windows path separators, avoids false positives for filenames containing these strings, and validates exported pattern presence.
Watcher Ignore E2E Tests
packages/opencode/test/file/watcher-ignore.test.ts
New comprehensive test suite validating file watcher ignore behavior for dbt dependencies, including realistic directory layouts, whitelist override behavior, nested file scenarios, and edge cases.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Poem

🐰 wiggles nose with satisfaction
Too many dbt files to keep in sight,
Our file watcher watched day and night!
Now dbt_packages we gently dismiss,
No more EMFILE—just watcher bliss! 📁✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding dbt_packages and dbt_modules to the file watcher ignore list to fix EMFILE errors.
Description check ✅ Passed The PR description follows the template with Summary, Test Plan sections completed, clearly explaining the changes and comprehensive test coverage.
Linked Issues check ✅ Passed The code changes fully address the linked issue #500 by adding dbt_packages and dbt_modules to the ignore list to prevent EMFILE errors caused by watching too many dependency files.
Out of Scope Changes check ✅ Passed All changes are directly scoped to addressing the linked issue: extending FileIgnore to include dbt directories and adding corresponding test coverage with no unrelated modifications.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/dbt-packages-emfile-watcher

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@anandgupta42 anandgupta42 requested a review from mdesmet March 27, 2026 03:40
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
packages/opencode/test/file/watcher-ignore.test.ts (1)

192-245: Consider extracting shared dbt path fixtures to reduce drift.

Several hardcoded dbt path lists are repeated across tests; centralizing them would improve maintainability as ignore semantics evolve.

♻️ Optional refactor sketch
+const DBT_SAMPLE_IGNORED_PATHS = [
+  "dbt_packages/dbt_utils/macros/sql/generate_series.sql",
+  "dbt_packages/dbt_utils/macros/sql/get_column_values.sql",
+  "dbt_packages/dbt_expectations/macros/schema_tests/expect_column_values_to_be_unique.sql",
+  "dbt_packages/codegen/macros/generate_source.sql",
+  "dbt_modules/legacy_pkg/macros/old_macro.sql",
+]
+
 describe("watcher ignore: dbt directories", () => {
   test("FileIgnore.match ignores files inside dbt_packages at any depth", () => {
-    expect(FileIgnore.match("dbt_packages/dbt_utils/macros/sql/generate_series.sql")).toBe(true)
-    expect(FileIgnore.match("dbt_packages/dbt_utils/macros/sql/get_column_values.sql")).toBe(true)
+    for (const p of DBT_SAMPLE_IGNORED_PATHS) {
+      expect(FileIgnore.match(p)).toBe(true)
+    }
   })
 })
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/opencode/test/file/watcher-ignore.test.ts` around lines 192 - 245,
Extract the repeated dbt path lists into a shared test fixture and import it
into watcher-ignore.test.ts: create a new test helper (e.g., export const
dbtPackages and dbtRealWorldPaths) containing the arrays currently defined as
packages, subdirs and realWorldPaths, then replace the in-file packages/subdirs
and realWorldPaths references with imports and use them with FileIgnore.match in
the tests; update any other tests that duplicate these lists to import the same
fixture so the ignore-paths are centralized and easier to maintain (look for
symbols packages, subdirs, realWorldPaths, and FileIgnore.match in this file to
locate usages).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@packages/opencode/test/file/watcher-ignore.test.ts`:
- Around line 192-245: Extract the repeated dbt path lists into a shared test
fixture and import it into watcher-ignore.test.ts: create a new test helper
(e.g., export const dbtPackages and dbtRealWorldPaths) containing the arrays
currently defined as packages, subdirs and realWorldPaths, then replace the
in-file packages/subdirs and realWorldPaths references with imports and use them
with FileIgnore.match in the tests; update any other tests that duplicate these
lists to import the same fixture so the ignore-paths are centralized and easier
to maintain (look for symbols packages, subdirs, realWorldPaths, and
FileIgnore.match in this file to locate usages).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: c6e35513-a35a-4e40-885f-d0b3123a99fe

📥 Commits

Reviewing files that changed from the base of the PR and between abcaa1d and c06c007.

📒 Files selected for processing (3)
  • packages/opencode/src/file/ignore.ts
  • packages/opencode/test/file/ignore.test.ts
  • packages/opencode/test/file/watcher-ignore.test.ts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

altimate-dbt execute fails with EMFILE: too many open files, watch on macOS

1 participant