feat(file): search workspace files by regular expression (#7370)

* feat(file): search workspace files by regular expression

Search read its query as literal text. It now reads it as a line-oriented
regular expression by default, with a Match setting on the block to go back
to verbatim text.

The segment store and its `gin_trgm_ops` index already support this: pg_trgm
extracts trigrams from a regex source too, so `~` / `~*` plan as a bitmap
index scan exactly like `LIKE` / `ILIKE`. No migration, no new index.

One compiled pattern owns every mode-specific decision — how PostgreSQL
matches a segment, whether the segment must hold a whole line, and where the
match sits inside it — so the repository builds one query shape and the
preview renderer one preview shape. Compilation happens in the application
use case, not the route adapter, so every surface gets the same semantics.

The supported syntax is the intersection of PostgreSQL ARE and JavaScript
RegExp, because the same source drives both the indexed predicate and the
client-side match location a preview centres on. Anything the two engines
read differently is rejected by name rather than silently reinterpreted, and
`\b` is rewritten to `\y` on the way to PostgreSQL.

Safety, in four independent layers:

- A pattern must contain 3 consecutive literal characters every match will
  include. pg_trgm indexes nothing shorter, and an unextractable pattern
  plans as a sequential scan across every workspace's segments.
- `new RegExp` proves it compiles in JavaScript.
- PostgreSQL proves it compiles in ARE; 2201B becomes a 400, not a 500.
- `statement_timeout` bounds the read. This one covers exact matching too,
  which has always been able to reach the same scan through a
  punctuation-only or non-ASCII query.

`mode` is a builder setting, withheld from the model like `maxResults`. The
model cannot see it and the two readings disagree on every metacharacter, so
`toolEnrichment` replaces the declared syntax with the active mode's — a
regex sent to a block set to exact matching would otherwise be searched for
verbatim and silently find nothing.

Verified against PostgreSQL 17: 14 behavioural checks end to end, a live
search over 150,012 segments in 14ms, 6/6 representative patterns reaching
the trigram index, and the guard cutting a 12s pattern at 10.08s into an
actionable message.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(file): locate regex matches in PostgreSQL, never in JavaScript

Preview rendering ran the user's compiled pattern with `RegExp.exec` to centre
the excerpt on the match. `RegExp` matches by backtracking, and the literal-run
gate admits nested quantifiers, so `(a+)+bcd` against a long segment cost 768ms
at 40 leading `a`s and doubles with each one — synchronously, on the event loop,
once per returned row, and entirely outside the statement timeout that bounds
the query which found the row.

PostgreSQL runs that same pattern in 0.49ms: its engine does not backtrack, and
`regexp_instr` runs inside the read's transaction, so locating a match can never
cost more than having found it. Regex mode now selects the match offsets
alongside the row and `findMatchRange` returns null for it, which is the
interface's contract rather than an omission. Exact mode is unchanged — scanning
for a known string is linear.

PostgreSQL counts characters where JavaScript slices by UTF-16 unit, so the
offsets are converted by walking the segment rather than assuming either width.

Also fixes two audit failures: `getErrorMessage` in place of a hand-written
`instanceof Error` ternary, and regenerated tool metadata and integration docs
for the search params.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(file): accept open-ended repeats, count characters, classify lock waits

Three defects from review, none of which the tests caught:

`{n,}` was rejected. `readQuantifierAt` reports an unbounded maximum as
Infinity, and the repeat cap compared it directly, so every open-ended repeat
failed as "exceeds 1000" — a form the tool's own documentation offers. Only a
stated maximum is measured now, and the minimum always is, since that is what
an expansion unrolls.

Query bounds and literal runs were measured in UTF-16 units while claiming
characters, so two astral characters read as four and slipped a gate written
for three. Both now count characters, which is also what pg_trgm indexes.

`lock_timeout` was set without classifying what it raises. A wait on
conflicting DDL surfaced as an unclassified server error, and folding it in
with the timeout arm would have told the caller to fix a pattern that is
already correct. It now maps to a distinct error the caller is told to retry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(file): keep long previews honest, credit forced repeats, drop bad hints

Four review findings, each reproduced before it was changed.

A regex match has no length limit, so `abc.*` on a long line produces a match
larger than the whole preview budget. The layout passed it through whole and
let the final byte cap cut it, which removed the closing marker along with the
text — 2048 bytes of output ending mid-line with nothing to say so. The match
is now clipped against a budget that reserves that marker, and a clipped match
always carries one.

A variable repeat was scored at one occurrence when its minimum forces more:
`(?:ab){2,5}` cannot match without `abab` in it, but the run was counted as 2
and the pattern rejected against a gate of 3. It now contributes the copies its
minimum forces.

`\Y`, `\m` and `\M` were rejected with a suggestion to write `\b`, `^` or `$`.
Those are different assertions — a non-boundary, and two word edges rather than
the line's — so the hint handed back different semantics as a fix. They now say
no supported escape means the same thing. `\y`, `\A` and `\Z` keep theirs,
which are genuine.

Smart case was documented as reacting to any uppercase letter, but it reads
literals only, so `\D` and `[A-Z]` do not make a search case-sensitive. The
tool, block and generated docs now say what the code does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(file): credit runs across a repeat, count matching lines not matches

The tool promised "each match" while the query is distinct on file and line,
so several matches on one line return one row. An agent reading the contract
would have expected otherwise; it now says each matching line once.

A repetition of a non-fixed atom was scored at what one copy guarantees, but
from two copies on its own tail and head meet: every match of
`(?:a(?:x|y)bc){2}` contains `bca`, which neither copy contains alone. That
run is now credited, so patterns the index can serve are no longer rejected.

Scores are capped alongside the strings they measure. Joining two capped
strings yields twice the cap, which `concatenate` could already exceed — the
gate never noticed, since it only compares against three, but the bound is
documented and now holds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(file): reject overflowed repeat bounds, describe what search covers

An upper bound too large for `Number` arrives as Infinity, and the exception
that lets `{n,}` skip the repeat cap could not tell the two apart — so
`needle{1,<400 digits>}` passed the cap that `needle{1,5000}` fails. The
quantifier now records whether a bound was written at all, and a written one
must be at or under the cap however large it is.

The tool promised every active workspace file. It searches what the index
currently holds: a file still pending, failed, or skipped as unsupported is
not searched, and an agent reading "every file" would take an empty result as
proof of absence. Both descriptions now say so and point at `complete` and
`indexStatus`, which already carry the detail.

The declared query description spoke only for regex mode, which is what the
catalog and the generated docs render — so a builder using exact matching was
told to write a regular expression and to obey a rule that does not apply to
them. It now names both readings; the runtime schema is still enriched with
whichever is in force. The docs overview said literal text, which stopped
being true when regex became the default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(file): close the allowlist around PostgreSQL bracket expressions

`[:class:]` was rejected while `[=equivalence=]` and `[.collating.]` were
forwarded unchanged. All three are PostgreSQL bracket expressions with no
JavaScript counterpart, and the parser exists to admit only what both engines
spell the same way — so two of them passed an allowlist whose whole point is
to close, and were accepted by documentation that says POSIX classes are not
supported.

They are now rejected by the construct they open, each named in its own error.
An ordinary class holding a literal dot, `[.]` or `[a.b]`, is untouched: the
form only matches on a bracket nested inside a class.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: icecrasher321

apps/docs/content/docs/integrations/file.mdx

@@ -16,7 +16,7 @@ The File block is a built-in Sim block for working with files stored in the work
 With the File block, you can:
 
 - **Read and extract content**: Load workspace file objects and extract their text content
-- **Search workspace content**: Find literal text across indexed active workspace files with bounded line-level results
+- **Search workspace content**: Match a regular expression, or an exact piece of text, against the indexed lines of active workspace files with bounded line-level results
 - **Fetch from URLs**: Retrieve and parse files from external URLs with custom headers
 - **Write and append**: Create new workspace files or append content to existing ones
 - **Compress and decompress**: Bundle files into a .zip archive or extract an archive into the workspace
@@ -70,13 +70,14 @@ Extract the text content of one or more workspace files from selected file objec
 
 ### File Search
 
-Search indexed text across active workspace files using literal smart-case substring matching.
+Search the indexed text of active workspace files for lines matching a regular expression, and return each matching line once with its file ID and line number. Coverage is what the index currently holds, so check "complete" and "indexStatus" before concluding that something is absent.
 
 #### Input
 
 | Parameter | Type | Required | Description |
 | --------- | ---- | -------- | ----------- |
-| `query` | string | Yes | Literal text to find \(3-512 characters\). Uppercase Unicode letters make matching case-sensitive. |
+| `query` | string | Yes | A regular expression matched against each line, 3-512 characters. Supports "." "*" "+" "?" "\{n,m\}" and their lazy forms, character classes such as "\[a-z\]" and "\[^0-9\]", the classes \d \w \s and \D \W \S, alternation "\|", groups "\(...\)" and "\(?:...\)", the anchors "^" and "$", and the word boundary \b. Lookahead, lookbehind, backreferences, named groups, inline flags such as "\(?i\)", \p\{...\} and POSIX "\[\[:alpha:\]\]" classes are not supported, and a pattern cannot span a line break. The pattern must contain at least 3 consecutive literal characters that every match will include — write "error \d+" rather than "\w+ \d+". Escape any metacharacter you mean literally. Matching is case-insensitive until the pattern contains an uppercase letter you are searching for; uppercase inside an escape or a character class, such as \D or \[A-Z\], does not make it case-sensitive. When the workflow builder sets Match to exact instead, the query is matched verbatim and no metacharacter needs escaping. |
+| `mode` | string | No | How the query is read, chosen by the workflow builder: "regex" \(default\) as a regular expression, or "exact" as verbatim text. |
 | `maxResults` | number | No | Hard result cap configured by the workflow builder \(1-200, default 50\). |
 
 #### Output

apps/sim/blocks/blocks/file.test.ts

@@ -72,15 +72,29 @@ describe('FileV5Block', () => {
         query: '',
         maxResults: '25',
       })
-    ).toEqual({ query: '', maxResults: 25 })
+    ).toEqual({ query: '', mode: 'regex', maxResults: 25 })
 
     const query = FileV5Block.subBlocks.find((subBlock) => subBlock.id === 'query')
+    const mode = FileV5Block.subBlocks.find((subBlock) => subBlock.id === 'mode')
     const maxResults = FileV5Block.subBlocks.find((subBlock) => subBlock.id === 'maxResults')
     expect(query?.paramVisibility).toBe('user-or-llm')
+    expect(mode?.paramVisibility).toBe('user-only')
     expect(maxResults?.paramVisibility).toBe('user-only')
     expect(query?.canonicalParamId).toBeUndefined()
     expect(maxResults?.canonicalParamId).toBeUndefined()
     expect(maxResults?.value?.()).toBe('50')
+    expect(mode?.value?.()).toBe('regex')
+  })
+
+  it.each([
+    [undefined, 'regex'],
+    ['exact', 'exact'],
+    ['regex', 'regex'],
+    ['glob', 'regex'],
+  ])('resolves the builder-configured match mode %s to %s', (mode, expected) => {
+    expect(buildParams({ operation: 'file_search', query: 'needle', mode })).toMatchObject({
+      mode: expected,
+    })
   })
 
   it('uses the default search cap when the builder field is cleared', () => {
@@ -90,7 +104,7 @@ describe('FileV5Block', () => {
         query: 'needle',
         maxResults: '',
       })
-    ).toEqual({ query: 'needle', maxResults: 50 })
+    ).toEqual({ query: 'needle', mode: 'regex', maxResults: 50 })
   })
 
   it.each(['10.5', '10results', '0', '201'])(

apps/sim/blocks/blocks/file.ts

@@ -908,7 +908,9 @@ export const FileV5Block: BlockConfig<FileParserV3Output> = {
   - Get Content is how you read file text. It accepts file objects or canonical file IDs and returns a "contents" array with one extracted text string per file (PDF, DOCX, CSV, etc. are parsed automatically).
   - To read the text of files produced by another block, chain into Get Content: set its file input to the upstream file output, e.g. <file.files>, <agent.files>, or <start.files>. Never assume Read (or any file-object output) already contains the text.
   - Get Content's "contents" can be large; it is persisted through the execution large-value system automatically, so prefer it over inlining file text any other way.
-  - Search finds literal text across all active workspace files and returns structured results with fileId, lineNumber, and text. Lowercase queries are case-insensitive; adding any uppercase letter makes the search case-sensitive.
+  - Search finds text across all active workspace files and returns one result per matching line — not per match — with fileId, lineNumber, and text. Queries are case-insensitive until they contain an uppercase letter being searched for; in a regular expression, uppercase inside an escape or character class such as \\D or [A-Z] does not affect this.
+  - Search reads the query as a line-oriented regular expression: quantifiers, character classes, \\d \\w \\s, alternation, groups, "^" and "$" anchors, and \\b word boundaries. Lookaround, backreferences and patterns spanning a line break are not supported, and a pattern needs at least 3 consecutive literal characters that every match will contain. Set Match to "Exact match" to search for the query text verbatim instead.
+  - Match is a builder setting, not an agent one: the agent writes the query, and Match decides how every query from that block is read.
   - Search is eventually consistent. Check "complete" and "indexStatus" when pending, failed, skipped, or partially indexed files matter to the task.
   - Use Fetch for external file URLs. Add headers for authenticated downloads, for example Slack private file URLs require an Authorization Bearer token.
   - Use Write to create a new workspace file and Append to add content to an existing one. Write adds a numeric suffix when the name is taken; turn on "Overwrite Existing File" to replace the contents of the file at that exact path (folder and name) instead — a same-named file in another folder is left alone.
@@ -1007,12 +1009,26 @@ export const FileV5Block: BlockConfig<FileParserV3Output> = {
       condition: { field: 'operation', value: 'file_get_content' },
       required: { field: 'operation', value: 'file_get_content' },
     },
+    {
+      id: 'mode',
+      title: 'Match',
+      type: 'dropdown' as SubBlockType,
+      options: [
+        { label: 'Regular expression', id: 'regex' },
+        { label: 'Exact match', id: 'exact' },
+      ],
+      description:
+        'How the query is read. Regular expressions match one line at a time and need at least 3 consecutive literal characters.',
+      value: () => 'regex',
+      condition: { field: 'operation', value: 'file_search' },
+      paramVisibility: 'user-only',
+    },
     {
       id: 'query',
       title: 'Query',
       type: 'short-input' as SubBlockType,
-      placeholder: 'Text to find across workspace files',
-      description: 'Literal search text, 3-512 characters. Leave blank for the agent to supply.',
+      placeholder: 'Pattern to find across workspace files',
+      description: 'Search pattern, 3-512 characters. Leave blank for the agent to supply.',
       condition: { field: 'operation', value: 'file_search' },
       required: { field: 'operation', value: 'file_search' },
       paramVisibility: 'user-or-llm',
@@ -1268,6 +1284,7 @@ export const FileV5Block: BlockConfig<FileParserV3Output> = {
           }
           return {
             query: params.query,
+            mode: params.mode === 'exact' ? 'exact' : 'regex',
             maxResults,
           }
         }
@@ -1500,7 +1517,11 @@ export const FileV5Block: BlockConfig<FileParserV3Output> = {
       type: 'string',
       description: 'Operation to perform (read, search, get content, fetch, write, or append)',
     },
-    query: { type: 'string', description: 'Literal workspace file search query' },
+    query: { type: 'string', description: 'Workspace file search query' },
+    mode: {
+      type: 'string',
+      description: 'How the search query is read: a regular expression (default) or an exact match',
+    },
     maxResults: { type: 'number', description: 'Hard maximum search results (1-200)' },
     readFileInput: {
       type: 'json',

apps/sim/lib/internal/file/execute-tool.test.ts

@@ -148,7 +148,13 @@ describe('executeFileTool', () => {
     })
     expect(mocks.searchContent).toHaveBeenCalledWith({
       principal: expect.objectContaining({ serviceId: 'executor' }),
-      input: { workspaceId: 'workspace-1', query: 'needle', maxResults: 25, signal: undefined },
+      input: {
+        workspaceId: 'workspace-1',
+        query: 'needle',
+        mode: 'regex',
+        maxResults: 25,
+        signal: undefined,
+      },
     })
     expect(mocks.executeManage).not.toHaveBeenCalled()
   })
@@ -168,7 +174,7 @@ describe('executeFileTool', () => {
 
     expect(mocks.searchContent).toHaveBeenCalledWith(
       expect.objectContaining({
-        input: { workspaceId: 'workspace-1', query: 'needle', maxResults: 50 },
+        input: { workspaceId: 'workspace-1', query: 'needle', mode: 'regex', maxResults: 50 },
       })
     )
     expect(mocks.getProvenance).toHaveBeenCalledWith(
@@ -191,6 +197,7 @@ describe('executeFileTool', () => {
     [{ query: 'abc\0def', maxResults: 50 }, 400],
     [{ query: 'needle', maxResults: 201 }, 400],
     [{ query: 'needle', maxResults: 0 }, 400],
+    [{ query: 'needle', mode: 'glob' }, 400],
   ])('rejects invalid search input before authorization', async (input, status) => {
     const response = await executeFileTool(request('file_search', input))
 
@@ -199,6 +206,16 @@ describe('executeFileTool', () => {
     expect(mocks.searchContent).not.toHaveBeenCalled()
   })
 
+  it('forwards an explicitly configured exact-match mode', async () => {
+    await executeFileTool(request('file_search', { query: 'needle', mode: 'exact' }))
+
+    expect(mocks.searchContent).toHaveBeenCalledWith(
+      expect.objectContaining({
+        input: expect.objectContaining({ query: 'needle', mode: 'exact' }),
+      })
+    )
+  })
+
   it('does not expose unexpected search infrastructure errors', async () => {
     mocks.searchContent.mockRejectedValueOnce(new Error('database host and query details'))
 

apps/sim/lib/internal/file/execute-tool.ts

@@ -31,6 +31,7 @@ import {
   FILE_SEARCH_MAX_RESULTS,
   FILE_SEARCH_MIN_QUERY_LENGTH,
 } from '@/lib/workspace-files/search/constants'
+import { FILE_SEARCH_MODES } from '@/lib/workspace-files/search/pattern'
 
 const logger = createLogger('FileToolExecution')
 
@@ -57,6 +58,7 @@ const fileSearchInputSchema = z
       .min(FILE_SEARCH_MIN_QUERY_LENGTH)
       .max(FILE_SEARCH_MAX_QUERY_LENGTH)
       .refine((query) => !query.includes('\0'), 'Search query cannot contain NUL characters'),
+    mode: z.enum(FILE_SEARCH_MODES).default('regex'),
     maxResults: z
       .number()
       .int()
@@ -110,6 +112,7 @@ export const executeFileTool: InternalToolOperationHandler = async (request) =>
         input: {
           workspaceId,
           query: searchInput.data.query,
+          mode: searchInput.data.mode,
           maxResults: searchInput.data.maxResults,
           signal: request.signal,
         },

apps/sim/lib/workspace-files/application/search-workspace-file-content.ts

@@ -2,12 +2,20 @@ import { OrchestrationError } from '@/lib/core/orchestration/types'
 import { loadActiveWorkspaceContext } from '@/lib/uploads/contexts/workspace'
 import { defineAuthorizedWorkspaceFileUseCase } from '@/lib/workspace-files/application/authorized-workspace-file-use-case'
 import { fileOperations } from '@/lib/workspace-files/application/operations'
-import { searchWorkspaceFileIndex } from '@/lib/workspace-files/search/repository'
-import { isFileSearchCaseSensitive } from '@/lib/workspace-files/search/text'
+import {
+  compileFileSearchPattern,
+  type FileSearchMode,
+  FileSearchPatternError,
+} from '@/lib/workspace-files/search/pattern'
+import {
+  searchWorkspaceFileIndex,
+  WorkspaceFileSearchUnavailableError,
+} from '@/lib/workspace-files/search/repository'
 
 export interface SearchWorkspaceFileContentInput {
   workspaceId: string
   query: string
+  mode: FileSearchMode
   maxResults: number
   signal?: AbortSignal
 }
@@ -24,12 +32,28 @@ export const searchWorkspaceFileContent = defineAuthorizedWorkspaceFileUseCase({
   operation: fileOperations.searchContent,
   resolveContext: ({ input }: { input: SearchWorkspaceFileContentInput }) =>
     resolveSearchWorkspaceFileContext(input),
-  execute: ({ input, context }) =>
-    searchWorkspaceFileIndex({
-      workspaceId: context.workspaceId,
-      query: input.query,
-      maxResults: input.maxResults,
-      caseSensitive: isFileSearchCaseSensitive(input.query),
-      signal: input.signal,
-    }),
+  execute: async ({ input, context }) => {
+    try {
+      return await searchWorkspaceFileIndex({
+        workspaceId: context.workspaceId,
+        pattern: compileFileSearchPattern(input.query, input.mode),
+        maxResults: input.maxResults,
+        signal: input.signal,
+      })
+    } catch (error) {
+      /**
+       * A rejected or too-expensive pattern is the caller's to fix, and the
+       * message names the construct and the supported alternative — so it is
+       * classified rather than left to become the surface's generic failure text.
+       */
+      if (error instanceof FileSearchPatternError) {
+        throw new OrchestrationError('validation', error.message)
+      }
+      /** Nothing is wrong with the query, so the caller is told to retry, not to rewrite it. */
+      if (error instanceof WorkspaceFileSearchUnavailableError) {
+        throw new OrchestrationError('locked', error.message)
+      }
+      throw error
+    }
+  },
 })

apps/sim/lib/workspace-files/search/constants.ts

@@ -1,5 +1,29 @@
+/**
+ * `pg_trgm` can only extract a trigram from three consecutive characters, so a
+ * shorter query has nothing for the segment GIN index to probe and degrades to a
+ * scan of every tenant's segments. It bounds the literal query length and, in
+ * regex mode, the shortest literal run every match is guaranteed to contain.
+ */
 export const FILE_SEARCH_MIN_QUERY_LENGTH = 3
 export const FILE_SEARCH_MAX_QUERY_LENGTH = 512
+
+/**
+ * Caps the analyzer's bookkeeping strings so a bounded repeat cannot expand a
+ * short pattern into a large intermediate. Only {@link FILE_SEARCH_MIN_QUERY_LENGTH}
+ * characters are ever needed, so truncating past this loses no decision.
+ */
+export const FILE_SEARCH_PATTERN_LITERAL_CAP = 512
+export const FILE_SEARCH_PATTERN_MAX_REPEAT = 1000
+export const FILE_SEARCH_PATTERN_MAX_DEPTH = 20
+
+/**
+ * Backstop for a pattern whose trigrams the planner cannot use — a punctuation-only
+ * or non-ASCII literal, or a regex whose guaranteed run yields no trigram. Those
+ * plan as a sequential scan across every workspace's segments, so the search must
+ * not be able to hold a pooled connection open indefinitely.
+ */
+export const FILE_SEARCH_STATEMENT_TIMEOUT_MS = 10 * 1000
+export const FILE_SEARCH_LOCK_TIMEOUT_MS = 5 * 1000
 export const FILE_SEARCH_DEFAULT_MAX_RESULTS = 50
 export const FILE_SEARCH_MAX_RESULTS = 200