feat(slack): Reply to @-mentions when Seer runs out of credits by alexsohn1126 · Pull Request #130440 · getsentry/sentry

alexsohn1126 · 2026-11-09T15:28:03Z

This PR replies to the thread when @sentry is mentioned and the organization does not have enough seer credits for the operation.

There are multiple ways that a Seer Agent will fail.

When mentioned, the org does not have any seer credits
When mentioned, the org has seer credits, but during the seer call, seer returns with out-of-credit error
When mentioned, the org has seer credits, but during the PR generation, the PR generation fails due to no seer credit

In all cases, we send a message like this:

And for PR creation, this would be shown (along with the standard message we send)

Fixes CW-2325

linear-code · 2026-11-09T15:28:14Z

CW-2325

sentry · 2026-11-09T15:56:39Z

Sentry Snapshot Testing

Name	Added	Removed	Changed	Renamed	Unchanged	Skipped	Status
sentry-frontend `sentry-frontend`	0	0	0	0	824	0	✅ Unchanged

⚙️ sentry-frontend Snapshot Settings

alexsohn1126 · 2026-11-10T21:36:12Z

bugbot review

cursor

Stale Bugbot comment from a previous run.

alexsohn1126 · 2026-11-12T15:29:25Z

bugbot review

cursor

Stale Bugbot comment from a previous run.

alexsohn1126 · 2026-11-12T17:40:55Z

    title: str | None = None
    description: str | None = None
    integration_id: str | None = None
+    idempotency_key: str | None = Field(default=None, exclude=True)


this allows us to give each PR push attempt its own unique key, so we can send "out of credits" message again if the PR push fails, and user @ mentions again to do a PR push.

alexsohn1126 · 2026-11-12T17:41:36Z

bugbot review

cursor

Stale Bugbot comment from a previous run.

billyvg · 2026-11-12T20:46:03Z

+            key
+            for pr in _get_repo_pr_states(state)
+            if _is_pr_out_of_credits(pr)
+            and (key := f"pr_out_of_credits:{pr.repo_name}:{pr.idempotency_key}") not in replies


🤖 Reply-cache docstring doesn't list the new out-of-credits keys

The docstring on SlackSeerAgentReplyCache in cache.py (lines 68–72, outside this diff) lists every reply key: the answer's block id, approval:<input id>, approval_dm:<input id>, and error. It describes itself as the only record of what was posted. This PR adds out_of_credits:<turn's first block id> and pr_out_of_credits:<repo>:<push key>. Add them to that list so the next person changing dedupe logic sees every key kind in one place.

_{Written by an AI agent and included in this review by its author.}

billyvg · 2026-11-12T20:46:03Z

+            _send_out_of_credits_message(
+                entrypoint=entrypoint,
+                organization=organization,
+                thread_ts=thread_ts or ts,


I'm not sure if this is true or not

🤖 Top-level @-mentions likely never see the ephemeral out-of-credits reply

For a top-level @-mention, thread_ts is None. The ephemeral therefore goes out with thread_ts=ts, into a thread on the user's own message that has no replies yet. Slack's chat.postEphemeral docs say ephemeral messages in threads only show if the thread is already active. This path returns before anything public is posted, so that thread never becomes active. The person who mentioned @sentry would see the "is thinking..." status clear and nothing else. That's case 1 in the description, and probably the most common one.

The halt paths a few lines up already handle this with thread_ts if thread_ts else "" and the comment "In a thread, show the prompt in the thread; top-level, show in the channel". Doing the same here posts the notice in the channel for top-level mentions:

Suggested change

thread_ts=thread_ts or ts,

thread_ts=thread_ts if thread_ts else "",

(_send_out_of_credits_message's docstring says "in the thread", so it would need a small tweak too.)

TASK_KWARGS always sets thread_ts, so the new tests only cover mentions inside a thread. Add a case with thread_ts=None that asserts the ephemeral is sent with thread_ts="". If you've already seen this render for a top-level mention, ignore this comment.

_{Written by an AI agent and included in this review by its author.}

billyvg · 2026-11-12T20:46:03Z

        assert second_data.missing_scope_settings_url is None

+    @patch("sentry.seer.entrypoints.slack.entrypoint.send_thread_update", return_value="1.1")
+    def test_on_agent_update_out_of_credits_replies_once(self, mock_send: MagicMock) -> None:


🤖 Two new on_agent_update tests only repeat what later tests already catch

Two of the seven new tests don't catch any bug the others miss:

test_on_agent_update_out_of_credits_replies_once (this one). It catches two bugs: the run-level OUT_OF_CREDITS branch being removed, or its key in replies check being dropped. Both bugs also fail test_on_agent_update_run_and_pr_out_of_credits_share_one_notice and test_on_agent_update_out_of_credits_notice_once_per_turn. Both of those refresh twice and assert a single SeerAgentOutOfCredits.

test_on_agent_update_pr_out_of_credits_posts_notice_once (line 1349). test_on_agent_update_pr_out_of_credits_notice_per_push_attempt already asserts [SeerAgentResponse, SeerAgentOutOfCredits] on the first refresh, and asserts no repeat when the same state is refreshed again. The extra summary and repo_pr_states checks on the answer are covered by the existing PR-status tests.

Deleting both keeps the same coverage with less to maintain.

_{Written by an AI agent and included in this review by its author.}

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit a5f33c0. Configure here.}

Slack posts the agent out-of-credits notification when a mention is stopped by the credit check, when an agent chat turn stops for credits, and when a PR push fails for credits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

When a push runs out of credits Seer marks every remaining repo with the same error, so each repo posted its own copy of the org-level notice. The run-level notice used a different key, so a turn where a PR failed and the run then stopped got it from both paths. Share one notice across all PRs that newly failed, and key the run and PR paths on the turn so a turn gets at most one. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

The out-of-credits notice for a failed PR was deduplicated on the repo, PR number and commit SHA. A push that fails leaves the number and SHA as they were (both unset when the PR never landed), so a retry in a later turn of the same thread looked like the failure already reported and got no notice. Key the notice on the push's idempotency key instead, which Seer sets per attempt. A failure left in the run state from an earlier turn still stays silent. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

The out-of-credits notification now has a copy per reader. These replies don't pass who is reading yet, so they use the member copy. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

…essage The pre-check reply is private to the person who mentioned Seer, so use their billing access directly. The replies posted when a run or a PR push fails use the Slack user stored with the run. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

alexsohn1126 mentioned this pull request Nov 9, 2026

feat(slack): Reply in the thread when an autofix run runs out of Seer credits #129975

Open

alexsohn1126 added this pull request to stack #130441 November 9, 2026 15:28

github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Nov 9, 2026

alexsohn1126 force-pushed the alexsohn/cw-2325-slack-out-of-credits-mention-replies branch from 40899d5 to fa1ae81 Compare November 9, 2026 15:54

alexsohn1126 force-pushed the alexsohn/cw-2325-slack-out-of-credits-mention-replies branch 6 times, most recently from c62ef58 to 93db726 Compare November 9, 2026 20:23

alexsohn1126 removed this pull request from stack #130441 November 9, 2026 20:24

alexsohn1126 added this pull request to stack #130501 November 9, 2026 20:24

alexsohn1126 force-pushed the alexsohn/cw-2325-slack-out-of-credits-mention-replies branch from 93db726 to c90d978 Compare November 9, 2026 20:29

alexsohn1126 mentioned this pull request Nov 9, 2026

feat(notifications): Add out-of-credits notifications for Seer Autofix and Agent #130500

Open

alexsohn1126 force-pushed the alexsohn/cw-2325-slack-out-of-credits-mention-replies branch 7 times, most recently from e18a83b to 7bc8ef8 Compare November 10, 2026 21:04

cursor Bot reviewed Nov 10, 2026

View reviewed changes

Comment thread src/sentry/seer/entrypoints/slack/entrypoint.py Resolved

alexsohn1126 force-pushed the alexsohn/cw-2325-slack-out-of-credits-mention-replies branch 2 times, most recently from 3d425d4 to 27944c5 Compare November 12, 2026 15:26

cursor Bot reviewed Nov 12, 2026

View reviewed changes

Comment thread src/sentry/seer/entrypoints/slack/entrypoint.py Resolved

alexsohn1126 commented Nov 12, 2026

View reviewed changes

cursor Bot reviewed Nov 12, 2026

View reviewed changes

alexsohn1126 marked this pull request as ready for review November 12, 2026 17:51

alexsohn1126 requested a review from a team as a code owner November 12, 2026 17:51

alexsohn1126 force-pushed the alexsohn/cw-2325-slack-out-of-credits-mention-replies branch from 540ee18 to 6360888 Compare November 12, 2026 17:59

srest2021 approved these changes Nov 12, 2026

View reviewed changes

billyvg reviewed Nov 12, 2026

View reviewed changes

alexsohn1126 force-pushed the alexsohn/cw-2325-slack-out-of-credits-mention-replies branch from 6360888 to a5f33c0 Compare November 12, 2026 21:45

cursor Bot reviewed Nov 12, 2026

View reviewed changes

Comment thread src/sentry/seer/entrypoints/slack/tasks.py Resolved

alexsohn1126 and others added 7 commits November 13, 2026 11:29

Update client_models.py

38ea1f0

ref(slack): Shorten out-of-credits comments to one line

3172d5d

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

test(slack): Expect the member copy in the agent out-of-credits replies

46400a4

The out-of-credits notification now has a copy per reader. These replies don't pass who is reading yet, so they use the member copy. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

alexsohn1126 force-pushed the alexsohn/cw-2325-slack-out-of-credits-mention-replies branch from a5f33c0 to 13ec451 Compare November 13, 2026 16:33

	thread_ts=thread_ts or ts,
	thread_ts=thread_ts if thread_ts else "",

Conversation

alexsohn1126 commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linear-code Bot commented Nov 9, 2026

Uh oh!

sentry Bot commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sentry Snapshot Testing

Uh oh!

alexsohn1126 commented Nov 10, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexsohn1126 commented Nov 12, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexsohn1126 Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

alexsohn1126 commented Nov 12, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

billyvg Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

billyvg Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

billyvg Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alexsohn1126 commented Nov 9, 2026 •

edited

Loading

sentry Bot commented Nov 9, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading