Skip to content

Warn before saving an agent rule that RuleBot can't carry out - #104807

Open
yuwenmemon wants to merge 4 commits into
mainfrom
yuwen-683438-agentRuleSaveCheck
Open

yuwenmemon wants to merge 4 commits into
mainfrom
yuwen-683438-agentRuleSaveCheck

Conversation

@yuwenmemon

@yuwenmemon yuwenmemon commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Explanation of Change

Admins can type an agent rule straight into the agent rule page, and the AI rules builder never sees it. A rule that RuleBot can't carry out still saves, adds RuleBot as an admin and never runs. This checks a typed or edited rule with GenerateRule when the admin taps Create rule or Save. An impossible answer shows the describe box's refusal, and the button becomes Create anyway or Save anyway, because this only warns. Any other answer, a failed check, or no answer within 20 seconds saves as before. A suggestion or a describe flow draft that the admin leaves unchanged skips the check, because Concierge wrote it.

The check runs for every admin, because App#104861 removed the describeRule beta. Web-Expensify#56886, which lets GenerateRule answer impossible, is in production.

Fixed Issues

$ https://github.com/Expensify/Expensify/issues/683438
PROPOSAL: N/A

Tests

The backend answers impossible only after Web-Expensify#56886, so the snippet below swaps the backend's answer for answer in the browser and keeps its generationID.

  1. As a workspace admin on a Control workspace, open Rules > Agents > Add agent rule (or Add AI rule), and paste the snippet into the dev console.
  2. Run answer = {state: 'impossible', reason: 'noCapability', explanation: "Agents can't send text messages.", suggestedPrompt: 'When a report is submitted, send the approver a direct message.'}.
  3. Open the Edit tab, type "Text the approver's phone whenever a report is submitted", and tap Create rule.
  4. Verify the button loads and "Checking your rule..." replaces the disclaimer.
Checking a typed agent rule 5. Verify the rule does not save, the refusal shows under the text box, and the button says Create anyway. Refusal with Create anyway 6. Run `answer = {state: 'agentRule'}`, tap Try this instead, and verify the suggestion replaces the text, is checked, and saves. 7. Open the rule you just saved, change its text, tap Save, and verify the refusal shows with Save anyway after `answer` is impossible again. Edit page refusal with Save anyway 8. Add a character to the text and verify the refusal hides and the button says Save again. 9. Tap Save anyway on a refused rule and verify it saves. 10. Pick a suggestion on the Suggested tab and tap Create rule without changing it. Verify it saves at once with no check.
Console snippet that swaps the GenerateRule answer locally
window.answer = null;
const swap = (value) => (value?.generationID && value.state && window.answer ? {generationID: value.generationID, ...window.answer} : value);
const [set, merge, update, multiSet] = [Onyx.set, Onyx.merge, Onyx.update, Onyx.multiSet].map((method) => method.bind(Onyx));
Onyx.set = (key, value) => set(key, key === 'generatedRule' ? swap(value) : value);
Onyx.merge = (key, value) => merge(key, key === 'generatedRule' ? swap(value) : value);
Onyx.update = (updates) => update(updates.map((u) => (u.key === 'generatedRule' ? {...u, value: swap(u.value)} : u)));
Onyx.multiSet = (data) => multiSet(Object.fromEntries(Object.entries(data).map(([key, value]) => [key, key === 'generatedRule' ? swap(value) : value])));
  • Verify that no errors appear in the JS console

Offline tests

  1. Go offline, type an agent rule on the Edit tab, and tap Create rule.
  2. Verify it saves at once with no check, as it does today.

QA Steps

Staging answers impossible only after Web-Expensify#56886. Until then, check that saving still works:

  1. Open a Control workspace > Rules > Agents > Add agent rule, type a rule on the Edit tab, and tap Create rule.
  2. Verify "Checking your rule..." shows, and the rule saves after a few seconds.
  3. Pick a suggestion, tap Create rule without changing it, and verify it saves at once.
  4. Open a saved agent rule, change its text, tap Save, and verify it saves after the check.
  • Verify that no errors appear in the JS console

PR Author Checklist

  • I linked the correct issue in the ### Fixed Issues section above
  • I wrote clear testing steps that cover the changes made in this PR
    • I added steps for local testing in the Tests section
    • I added steps for the expected offline behavior in the Offline steps section
    • I added steps for Staging and/or Production testing in the QA steps section
    • I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
    • I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
    • I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).
  • I included screenshots or videos for tests on all platforms
  • I ran the tests on all platforms & verified they passed on:
    • Android: Native
    • Android: mWeb Chrome
    • iOS: Native
    • iOS: mWeb Safari
    • MacOS: Chrome / Safari
  • I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
  • I verified any copy / text that was added to the app is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
  • If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
  • I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
  • If new assets were added or existing ones were modified, I verified that:
    • The assets are optimized and compressed (for SVG files, run npm run compress-svg)
    • The assets load correctly across all supported platforms.
  • If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
  • If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
  • If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
  • If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
  • If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:
    • I verified that all the inputs inside a form are aligned with each other.
    • I added Design label and/or tagged @Expensify/design so the design team can review the changes.
  • If the PR adds or modifies the UI:
    • I asked an AI agent to review the changes for accessibility issues and addressed its findings.
    • I tested with a screen reader (VoiceOver on macOS) and verified all new/changed elements are reachable with a logical focus order.
    • I verified all new/changed elements have meaningful accessible names and roles.
    • I verified state changes are announced (e.g. checked/unchecked, expanded/collapsed, selected).
  • I added unit tests for any new feature or bug fix in this PR to help automatically prevent regressions in this user flow.
  • If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native
Android: mWeb Chrome
iOS: Native
iOS: mWeb Safari
MacOS: Chrome / Safari Refusal with Create anyway

@OSBotify

Copy link
Copy Markdown
Contributor

🦜 Polyglot Parrot! 🦜

Squawk! Looks like you added some shiny new English strings. Allow me to parrot them back to you in other tongues:

View the translation diff
diff --git a/src/languages/de.ts b/src/languages/de.ts
index c68d5bff..0b2355d4 100644
--- a/src/languages/de.ts
+++ b/src/languages/de.ts
@@ -8823,7 +8823,7 @@ Fügen Sie weitere Ausgabelimits hinzu, um den Cashflow Ihres Unternehmens zu sc
                 createRule: 'Regel erstellen',
                 createAnyway: 'Trotzdem erstellen',
                 saveAnyway: 'Trotzdem speichern',
-                checkingRule: 'Ihre Regel wird geprüft...',
+                checkingRule: 'Ihre Regel wird überprüft ...',
                 fallbackBanner:
                     'Dies ist keine Standardregel, daher haben wir sie als Agentenregel erstellt. Agentenregeln greifen, nachdem sich eine Ausgabe oder ein Bericht geändert hat, zum Beispiel indem sie die Ausgabe zurückhalten oder den Bericht ablehnen.',
                 fallbackBannerWithExplanation: (explanation: string) => `Dies ist keine Standardregel, daher haben wir sie als Agentenregel formuliert. ${explanation}`,
diff --git a/src/languages/fr.ts b/src/languages/fr.ts
index ca9efc6e..a107fdf5 100644
--- a/src/languages/fr.ts
+++ b/src/languages/fr.ts
@@ -8846,7 +8846,7 @@ Ajoutez davantage de règles de dépenses pour protéger la trésorerie de l’e
                 createRule: 'Créer une règle',
                 createAnyway: 'Créer quand même',
                 saveAnyway: 'Enregistrer quand même',
-                checkingRule: 'Vérification de votre règle...',
+                checkingRule: 'Vérification de votre règle…',
                 fallbackBanner:
                     'Ce n’est pas une règle standard, nous l’avons donc rédigée comme une règle d’agent. Les règles d’agent agissent après la modification d’une dépense ou d’une note de frais, par exemple en bloquant la dépense ou en rejetant la note de frais.',
                 fallbackBannerWithExplanation: (explanation: string) => `Ce n’est pas une règle standard, nous l’avons donc rédigée comme une règle d’agent. ${explanation}`,
diff --git a/src/languages/ja.ts b/src/languages/ja.ts
index 87b560c1..56b1861f 100644
--- a/src/languages/ja.ts
+++ b/src/languages/ja.ts
@@ -8661,9 +8661,9 @@ ${reportName}`,
                 describeRuleForConcierge: 'ルールの内容を入力すると、Concierge が自動作成します',
                 gotIt: '了解しました',
                 createRule: 'ルールを作成',
-                createAnyway: 'このまま作成',
+                createAnyway: 'それでも作成',
                 saveAnyway: 'このまま保存',
-                checkingRule: 'ルールを確認しています...',
+                checkingRule: 'ルールを確認しています…',
                 fallbackBanner:
                     'これは標準ルールではないため、エージェントルールとして作成しました。エージェントルールは、経費やレポートに変更があった後に動作し、例えば経費を保留にしたり、レポートを却下したりします。',
                 fallbackBannerWithExplanation: (explanation: string) => `これは標準ルールではないため、エージェントルールとして作成しました。${explanation}`,
diff --git a/src/languages/pl.ts b/src/languages/pl.ts
index 1e9357e7..a5d6be34 100644
--- a/src/languages/pl.ts
+++ b/src/languages/pl.ts
@@ -8778,7 +8778,7 @@ Dodaj więcej zasad wydatków, żeby chronić płynność finansową firmy.`,
                 createRule: 'Utwórz regułę',
                 createAnyway: 'Utwórz mimo to',
                 saveAnyway: 'Zapisz mimo to',
-                checkingRule: 'Sprawdzanie reguły...',
+                checkingRule: 'Sprawdzanie twojej reguły...',
                 fallbackBanner:
                     'To nie jest standardowa reguła, więc przygotowaliśmy ją jako regułę agenta. Reguły agenta działają po zmianie wydatku lub raportu, na przykład wstrzymując wydatek albo odrzucając raport.',
                 fallbackBannerWithExplanation: (explanation: string) => `To nie jest standardowa reguła, więc przygotowaliśmy ją jako regułę agenta. ${explanation}`,
diff --git a/src/languages/zh-hans.ts b/src/languages/zh-hans.ts
index 50b347c7..2ad2a549 100644
--- a/src/languages/zh-hans.ts
+++ b/src/languages/zh-hans.ts
@@ -8452,7 +8452,7 @@ ${reportName}`,
                 createRule: '创建规则',
                 createAnyway: '仍然创建',
                 saveAnyway: '仍然保存',
-                checkingRule: '正在检查你的规则...',
+                checkingRule: '正在检查您的规则…',
                 fallbackBanner: '这不是一条标准规则,所以我们将其制定为代理规则。代理规则会在报销或报表发生变更后生效,例如暂扣该报销或驳回该报表。',
                 fallbackBannerWithExplanation: (explanation: string) => `这不是一条标准规则,因此我们将其起草为代理规则。${explanation}`,
             },

Note

You can apply these changes to your branch by copying the patch to your clipboard, then running pbpaste | git apply 😉

View workflow run

@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Looks like you've decreased code coverage for some files. Please write tests to increase, or at least maintain, the existing level of code coverage. See our documentation here for how to interpret this table.

Files with missing lines Coverage Δ
src/CONST/index.ts 95.90% <ø> (ø)
src/libs/actions/Policy/Rules.ts 79.72% <100.00%> (+1.36%) ⬆️
...es/workspace/rules/AgentRules/AddAgentRulePage.tsx 87.30% <100.00%> (+0.63%) ⬆️
...orkspace/rules/AgentRules/AddAgentRuleWriteTab.tsx 80.00% <100.00%> (+80.00%) ⬆️
...s/workspace/rules/AgentRules/EditAgentRulePage.tsx 83.82% <100.00%> (+83.82%) ⬆️
src/pages/workspace/rules/DescribeRuleForm.tsx 94.02% <100.00%> (-0.42%) ⬇️
src/pages/workspace/rules/ImpossibleRuleNotice.tsx 100.00% <100.00%> (ø)
...kspace/rules/AgentRules/useAgentRulePromptCheck.ts 96.96% <96.96%> (ø)
... and 10 files with indirect coverage changes

@yuwenmemon
yuwenmemon marked this pull request as ready for review October 28, 2026 06:08
@yuwenmemon
yuwenmemon requested review from a team as code owners October 28, 2026 06:08
@melvin-bot
melvin-bot Bot requested a review from eh2077 October 28, 2026 06:08
@yuwenmemon
yuwenmemon requested review from JS00001 and removed request for eh2077 October 28, 2026 06:08
@melvin-bot
melvin-bot Bot removed the request for review from a team October 28, 2026 06:08
@melvin-bot

melvin-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown

@eh2077 Please copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button]

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T06:33:32.401180Z eada752 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@melvin-bot
melvin-bot Bot requested review from joekaufmanexpensify and removed request for a team October 28, 2026 06:08
@yuwenmemon
yuwenmemon requested review from a team and rushatgabhane October 28, 2026 06:08
@melvin-bot
melvin-bot Bot requested review from marufsharifi and removed request for a team October 28, 2026 06:08
@melvin-bot

melvin-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown

@marufsharifi Please copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button]

@melvin-bot
melvin-bot Bot requested review from garrettmknight and removed request for a team October 28, 2026 06:08
@yuwenmemon

Copy link
Copy Markdown
Contributor Author

Okay @JS00001 @rushatgabhane this is ready to go

@marufsharifi

marufsharifi commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Reviewer Checklist

  • I have verified the author checklist is complete (all boxes are checked off).
  • I verified the correct issue is linked in the ### Fixed Issues section above
  • I verified testing steps are clear and they cover the changes made in this PR
    • I verified the steps for local testing are in the Tests section
    • I verified the steps for Staging and/or Production testing are in the QA steps section
    • I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
    • I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
  • I checked that screenshots or videos are included for tests on all platforms
  • I included screenshots or videos for tests on all platforms
  • I verified that the composer does not automatically focus or open the keyboard on mobile unless explicitly intended. This includes checking that returning the app from the background does not unexpectedly open the keyboard.
  • I verified tests pass on all platforms & I tested again on:
    • Android: HybridApp
    • Android: mWeb Chrome
    • iOS: HybridApp
    • iOS: mWeb Safari
    • MacOS: Chrome / Safari
  • If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
  • I verified any copy / text that was added to the app is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
  • If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
  • I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
  • If a new component is created I verified that:
    • A similar component doesn't exist in the codebase
    • All props are defined accurately
    • The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
    • The only data being stored in the state is data necessary for rendering and nothing else
    • The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions
  • If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
  • If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
  • If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
  • If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
  • If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:
    • I verified that all the inputs inside a form are aligned with each other.
    • I added Design label and/or tagged @Expensify/design so the design team can review the changes.
  • If the PR adds or modifies the UI:
    • I asked an AI agent to review the changes for accessibility issues and addressed its findings.
    • I tested with a screen reader (VoiceOver on macOS) and verified all new/changed elements are reachable with a logical focus order.
    • I verified all new/changed elements have meaningful accessible names and roles.
    • I verified state changes are announced (e.g. checked/unchecked, expanded/collapsed, selected).
  • For any bug fix or new feature in this PR, I verified that sufficient unit tests are included to prevent regressions in this flow.
  • If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
  • I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: HybridApp
Android: mWeb Chrome
iOS: HybridApp
iOS: mWeb Safari
MacOS: Chrome / Safari

savePrompt: saveRule,
});

const trySuggestedPrompt = (suggestedPrompt: string) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ CONSISTENCY-3 (docs)

The agent-rule-check wiring is duplicated between EditAgentRulePage.tsx and AddAgentRuleWriteTab.tsx: both declare the same isChecking subscription, call useAgentRulePromptCheck, define an identical trySuggestedPrompt helper (formRef.current?.resetForm(...) + checkSuggestedPrompt(...)), and render the identical ImpossibleRuleNotice + disclaimer Text block (with the same checkingRule/disclaimer ternary). When this glue changes, both files have to be kept in sync by hand.

Consider extracting the shared footer and helper into one small shared component so both pages render it identically:

// Shared by AddAgentRuleWriteTab and EditAgentRulePage
function AgentRulePromptFooter({refusal, isChecking, onTrySuggestion}: AgentRulePromptFooterProps) {
    const {translate} = useLocalize();
    const styles = useThemeStyles();
    return (
        <>
            {!!refusal && <ImpossibleRuleNotice refusal={refusal} onTrySuggestion={onTrySuggestion} />}
            <Text style={[styles.textMicroSupporting, styles.textAlignCenter, styles.mt2]}>
                {translate(isChecking ? 'workspace.rules.agentRules.checkingRule' : 'workspace.rules.agentRules.disclaimer')}
            </Text>
        </>
    );
}

Reviewed at: eada752 | Please rate this suggestion with 👍 or 👎 to help us improve! Reactions are used to monitor reviewer efficiency.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Yuwen's Agent) Fixed in ad0ad01. The hook now owns the isChecking subscription and resets the form for Try this instead, so both pages only call it and render ImpossibleRuleNotice.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eada7520e1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

};

/** Hides the refusal when the admin changes the text, so that the next save checks the new text */
const dismissRefusal = () => setCheckGenerationID(undefined);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Cancel the pending check when the prompt changes

If the admin edits the still-enabled input while a check is running, this callback only hides the refusal; it leaves pendingCheckRef and its timeout active. A non-impossible response or the timeout then saves pendingCheck.prompt—the text submitted before the edit—and navigates away, silently discarding the newer input in both the add and edit flows. Cancel the pending check when the value changes, or prevent editing until it finishes.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Yuwen's Agent) Fixed in ad0ad01. Editing the text now stops the running check and its timeout, and ends the loading state, so the answer for the old text can't save it. The admin taps Create rule or Save again to check the new text. Added a test for it.

@joekaufmanexpensify joekaufmanexpensify left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change makes sense to me from a product perspective.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants