Skip to content

Commit a4aac27

Browse files
committed
fix(seeder): example chats read correctly under today's chat rules
Wakes are wake-shaped (banner renders), no prose points at tool rows that no longer render, footer-era phrasing removed, the four-wake variation matrix became one coherent story, and tests now enforce all of it (wake ids parse via the real WakeBanner, removed-UI phrases are banned).
1 parent fe81d57 commit a4aac27

2 files changed

Lines changed: 100 additions & 32 deletions

File tree

apps/webapp/seed-agent-examples-chats.mts

Lines changed: 54 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -7,15 +7,21 @@
77
* created. The panel renders them through the production renderer with no demo
88
* code involved, so what a reviewer sees here is what a real chat looks like.
99
*
10-
* Two translation rules, because the store is narrower than the mockup:
10+
* Three translation rules, because the store is narrower than the mockup:
1111
*
1212
* 1. Only what the production renderer handles survives: `text`, `reasoning`,
13-
* `tool-*` rows, `tool-render_view` (real cards), `tool-get_report` (the
14-
* report card) and `source-url`. Investigation cards, watch chips, intent
15-
* bubbles and prompt rows have no stored representation yet, so those beats
16-
* become assistant text — and where a card carried the diagnosis, a real
17-
* `diagnosis` view block carries it instead.
18-
* 2. Chats that exist only to show a transient UI state (streaming text, a tool
13+
* `tool-render_view` (real cards), `tool-get_report` (the report card) and
14+
* `source-url`. Investigation cards, watch chips, intent bubbles and prompt
15+
* rows have no stored representation yet, so those beats become assistant
16+
* text — and where a card carried the diagnosis, a real `diagnosis` view
17+
* block carries it instead.
18+
* 2. A landed tool call renders NOTHING. Completed `tool-*` parts are still
19+
* stored — they are what the answer was grounded on, and cards read them —
20+
* but no prose may point at one, because there is no row on screen to point
21+
* at. Only a failed call keeps a row, so a failure can still be seen.
22+
* Likewise: a wake narration is an assistant message whose id is
23+
* `wake:watch:{watchId}:{fired|expired}` — the id is what draws the banner.
24+
* 3. Chats that exist only to show a transient UI state (streaming text, a tool
1925
* mid-call, an unsent draft) are not ported: a stored transcript can't be
2026
* mid-flight, and storing one would render as a turn that never finishes.
2127
* `SKIPPED_DEMO_CHATS` records them.
@@ -100,7 +106,7 @@ export const SKIPPED_DEMO_CHATS: ReadonlyArray<{ id: string; reason: string }> =
100106
},
101107
{
102108
id: "demo:base-tool-in-flight",
103-
reason: "A tool row mid-call. Stored, it would render as 'Running…' forever.",
109+
reason: "A tool mid-call. Stored, it would render as a pending pill that never resolves.",
104110
},
105111
{
106112
id: "demo:base-composer-draft",
@@ -132,6 +138,19 @@ function assistant(slug: string, parts: unknown[]): SeedMessage {
132138
return { id: messageId(slug), role: "assistant", parts };
133139
}
134140

141+
/**
142+
* A wake narration: an assistant turn the watch started, not the user.
143+
*
144+
* The panel spots one by its message id — `wake:watch:{watchId}:{fired|expired}`,
145+
* the same id the agent's `narrateWatchWake` writes — and draws the banner from
146+
* that, so the parts are ordinary prose. The seeder creates no watch rows, so the
147+
* banner takes its kind-agnostic wording ("Watch update — condition met") rather
148+
* than colouring by watch kind.
149+
*/
150+
function wake(watchId: string, outcome: "fired" | "expired", parts: unknown[]): SeedMessage {
151+
return { id: `wake:watch:${watchId}:${outcome}`, role: "assistant", parts };
152+
}
153+
135154
function text(value: string) {
136155
return { type: "text", text: value, state: "done" };
137156
}
@@ -581,7 +600,7 @@ Deployment: \`${deploymentUri}\`.`
581600

582601
chats.push({
583602
slug: "watch-created-and-wake",
584-
title: "Tell me when the backlog drains",
603+
title: "Tell me when the retry finishes",
585604
minutesAgo: 7,
586605
messages: [
587606
user("watch", "Tell me when the retry finishes."),
@@ -590,47 +609,49 @@ Deployment: \`${deploymentUri}\`.`
590609
`Watching \`${w.failedRunId}\` — I'll check every minute for up to 2 hours and tell you the moment it settles, whichever way it goes. I only speak once per watch, so it won't repeat itself.`
591610
),
592611
]),
593-
assistant("watch", [
612+
wake(`watch_seed_${w.failedRunId}_finished`, "fired", [
594613
text(
595614
`**The retry finished.** \`${w.failedRunId}\` completed successfully 4 minutes ago, on attempt 2 — the provider accepted the request once the delay pushed it out of the rate-limit window.
596615
597-
I've stopped watching it. The other runs from the same burst are still queued behind the concurrency limit; ask me if you want them watched too.`
616+
That's this watch done. The rest of the ${w.firstFailureClock} burst is still queued behind \`${w.queue}\`'s concurrency limit, and the rate-limit error is worth watching for the next 12 hours in case the fix didn't take — say the word for either.`
598617
),
599618
sourceUrl(runLink(w.failedRunId), `Run ${w.failedRunId}`),
600619
]),
601-
assistant("watch", [
602-
text(
603-
`Two things you might want next: the remaining runs from the ${w.firstFailureClock} burst are still queued behind \`${w.queue}\`'s concurrency limit, and the rate-limit error itself is worth a watch for the next 12 hours in case the fix didn't take. Say the word for either.`
604-
),
605-
]),
606620
],
607621
});
608622

609623
chats.push({
610-
slug: "watch-expiry-and-cancel",
611-
title: "Watch for that error recurring",
624+
slug: "watch-fired-and-expired",
625+
title: "Watch the error and the backlog",
612626
minutesAgo: 40,
613627
messages: [
628+
user(
629+
"watch-end",
630+
`Watch two things for me: that 429 coming back, and \`${w.backlogQueue}\` draining.`
631+
),
614632
assistant("watch-end", [
615633
text(
616-
`Four watches from this conversation have finished, and each one said so exactly once — below, in the order they spoke.`
634+
`Both armed. The error watch covers fingerprint \`${w.errorFingerprint}\` on \`${w.taskId}\` for the next 12 hours; the backlog watch covers \`${w.backlogQueue}\` for 6. Each one speaks once — when its condition is met, or when its window runs out with nothing to report.`
617635
),
618636
]),
619-
assistant("watch-end", [
637+
wake(`watch_seed_${w.errorFingerprint}_recurrence`, "fired", [
620638
text(
621-
`**I've stopped watching \`${w.backlogQueue}\`.** The 6-hour window is up and the backlog never fully drained — it's down from ${w.pending.toLocaleString("en-US")} to 610 pending, so it's clearing, just slower than the window I was given.
639+
`**The 429 is back.** \`${w.taskId}\` failed with the same fingerprint 9 minutes ago, so the retry change hasn't taken — the attempts are still landing inside one rate-limit window.
622640
623-
Ask again if you want another 6 hours.`
641+
That's the error watch spent. Cap \`${w.queue}\` at 20 concurrency if you want the burst to stop before the next deploy.`
624642
),
643+
sourceUrl(failedRunsLink, `Failed ${w.taskId} runs`),
644+
sourceUrl(queueLink, `Queue ${w.queue}`),
625645
]),
626-
assistant("watch-end", [
646+
wake("watch_seed_backlog_drain", "expired", [
627647
text(
628-
`**I've stopped watching prod's health, but I couldn't verify the condition at expiry.** The health data was unavailable on my last few checks, so I can't tell you whether prod recovered — only that I never saw it recover.
648+
`**Six hours are up and \`${w.backlogQueue}\` never fully drained.** It's down from ${w.pending.toLocaleString(
649+
"en-US"
650+
)} to 610 pending, so it is clearing — just slower than the window I was given, which is why I have no answer rather than bad news.
629651
630-
Re-run the health report to get a current answer.`
652+
Ask again if you want another 6 hours.`
631653
),
632654
]),
633-
assistant("watch-end", [text(`Stopped watching \`${w.waitingRunId}\`.`)]),
634655
],
635656
});
636657

@@ -650,7 +671,7 @@ Re-run the health report to get a current answer.`
650671
]),
651672
assistant("rep-h", [
652673
text(
653-
"Nothing needs you. Every reading is inside the band you've run at for the past week, and telemetry is seconds behind, so this is a current picture rather than a stale one."
674+
"Every reading is inside the band you've run at for the past week, and telemetry is seconds behind, so this is a current picture rather than a stale one. The one thing worth knowing: this window is quiet because the arrival rate is quiet — it is not proof the concurrency limit is high enough for a busy hour."
654675
),
655676
]),
656677
],
@@ -673,8 +694,9 @@ Re-run the health report to get a current answer.`
673694
text(
674695
`Your code is fine — ${w.failureRatePct}% of runs failed and p95 duration hasn't moved. You've been pinned at the environment's concurrency ceiling of ${w.envConcurrencyLimit} for ${w.pinnedMinutes} of the last 60 minutes, ${w.pending.toLocaleString("en-US")} runs are pending, and \`${w.queue}\` accounts for ${Math.round(
675696
w.worstQueueShare * 100
676-
)}% of them. Raising the limit clears it now; doing nothing clears it in about ${w.drainMinutes} minutes, once the ${w.triggeredPerMin.toLocaleString("en-US")}-a-minute arrival spike drops back under the ${w.donePerMin.toLocaleString("en-US")} a minute you're completing.`
697+
)}% of them — so this is one queue's spike, not the whole environment misbehaving. Arrivals are ${w.triggeredPerMin.toLocaleString("en-US")} a minute against the ${w.donePerMin.toLocaleString("en-US")} you're completing, and that gap is already narrowing. The oldest thing still waiting is \`${w.waitingRunId}\`, if you want to see what a queued run looks like from the inside.`
677698
),
699+
sourceUrl(runLink(w.waitingRunId), `Run ${w.waitingRunId}`),
678700
sourceUrl(queueLink, `Queue ${w.queue}`),
679701
]),
680702
],
@@ -693,7 +715,7 @@ Re-run the health report to get a current answer.`
693715
694716
- \`maxAttempts\` counts the *first* attempt, so \`3\` means one try plus two retries.
695717
- The delay is \`minTimeoutInMs * factor^(attempt - 1)\`, capped at \`maxTimeoutInMs\`.
696-
- \`randomize: true\` adds jitter, which is what stops a whole batch retrying in lockstep — the thing that bit \`${w.taskId}\` above.`
718+
- \`randomize: true\` adds jitter, which is what stops a whole batch retrying in lockstep — the thing that bit \`${w.taskId}\` on \`${w.queue}\` this morning.`
697719
),
698720
sourceUrl("https://trigger.dev/docs/errors-retrying", "Errors & retrying"),
699721
sourceUrl("https://trigger.dev/docs/tasks/overview", "Task options"),
@@ -705,13 +727,13 @@ Re-run the health report to get a current answer.`
705727

706728
chats.push({
707729
slug: "base-resumed",
708-
title: "Queue health over time",
730+
title: "Did this happen earlier today too?",
709731
minutesAgo: 21,
710732
messages: [
711733
user("res", "Did this happen earlier today too?"),
712734
assistant("res", [
713735
text(
714-
`Yes — same error, same task, earlier in the same burst. \`${w.taskId}\` hit the same rate limit at ${w.firstFailureClock} and it was diagnosed then too; the card below is that diagnosis, replayed from this conversation rather than re-run. The retry config hasn't changed since, which is why it came back.`
736+
`Yes — same error, same task, earlier in the same burst. \`${w.taskId}\` hit the same rate limit at ${w.firstFailureClock} and I diagnosed it then; both diagnoses are here — this run's, and \`${w.priorRunId}\` from the start of the burst. The retry config hasn't changed since, which is why it came back.`
715737
),
716738
// Revisions 0 and 1 of one diagnosis (the renderer collapses to the
717739
// highest revision) plus one envelope-less block from an older transcript.
@@ -720,7 +742,7 @@ Re-run the health report to get a current answer.`
720742
assistant("res", [
721743
renderView([failuresChart]),
722744
text(
723-
"The chart block runs live against your current environment every time this transcript is opened."
745+
`The chart is live, so it answers the same question for whatever has happened since: two clusters today, both on \`${w.taskId}\`, nothing on the other senders.`
724746
),
725747
]),
726748
],

apps/webapp/test/seedAgentExamplesChats.test.ts

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import { describe, expect, it } from "vitest";
2+
import { wakeRefFromMessageId } from "~/components/dashboard-agent/WakeBanner";
23
import { resolveTriggerUri, type TriggerUriScope } from "~/services/resolveTriggerUri.server";
34
import { buildAgentExampleChats, type AgentExamplesWorld } from "../seed-agent-examples-chats.mjs";
45

@@ -122,6 +123,51 @@ describe("agent example transcripts", () => {
122123
}
123124
});
124125

126+
it("stores every wake narration under a wake message id", () => {
127+
// A wake is spotted by its id, not by its parts, so an ordinary `msg_` id
128+
// would render as an answer to a question nobody asked — no banner.
129+
const wakes = chats.flatMap((chat) =>
130+
chat.messages
131+
.map((message) => ({ chat: chat.slug, ref: wakeRefFromMessageId(message.id) }))
132+
.filter((entry) => entry.ref !== null)
133+
);
134+
expect(wakes.map((entry) => entry.ref!.outcome)).toEqual(
135+
expect.arrayContaining(["fired", "expired"])
136+
);
137+
138+
// Only an assistant turn can be a wake, and every conversation is started by
139+
// the user — an assistant message in first position has no turn to belong to.
140+
for (const entry of wakes) {
141+
const chat = chats.find((candidate) => candidate.slug === entry.chat)!;
142+
const message = chat.messages.find(
143+
(candidate) => wakeRefFromMessageId(candidate.id)?.watchId === entry.ref!.watchId
144+
)!;
145+
expect(message.role, `${entry.chat}/${message.id}`).toBe("assistant");
146+
}
147+
for (const chat of chats) {
148+
expect(chat.messages[0]!.role, `${chat.slug} opens`).toBe("user");
149+
}
150+
});
151+
152+
it("never points prose at UI the panel no longer renders", () => {
153+
// A landed tool call leaves no row, and text has no raw toggle — so prose
154+
// that says "see the call above" or "show raw" describes a panel that is gone.
155+
const forbidden = [
156+
/show raw/i,
157+
/raw (json|output|input)/i,
158+
/expand(ed)? (the )?(tool|output)/i,
159+
/\btool (call|row)s? (above|below)\b/i,
160+
/you can see (it|the) call/i,
161+
];
162+
for (const { chat, part } of parts()) {
163+
if (part.type !== "text" && part.type !== "reasoning") continue;
164+
const value = part.text as string;
165+
for (const pattern of forbidden) {
166+
expect(pattern.test(value), `${chat}: ${pattern} in ${value.slice(0, 80)}`).toBe(false);
167+
}
168+
}
169+
});
170+
125171
it("resolves every trigger:// citation inside the seeded scope", () => {
126172
const text = JSON.stringify(chats);
127173
const uris = [...text.matchAll(/trigger:\/\/[^\s"`\\]+/g)].map((match) => match[0]);

0 commit comments

Comments
 (0)