feat: Kiln assistant — auto mode, background eval jobs, and sub-agents (runtime, tabs UI, steer/stop)

[leonardmq]

Consolidated PR — this PR was previously stacked on #1517 (leonard/kil-686-eval-job). The stack has been collapsed: this PR now targets main and carries all of #1517's work; #1517 is closed in favor of this one and its full description is preserved below under Carried from #1517. kiln_server counterpart: Kiln-AI/kiln_server#255.

Desktop + web UI for assistant sub-agents:

• Local sub-agent runtime: registry, runner, orchestration tools, and the /api/conversations sub-agent API surface.
• Sub-agent UI: tab strip with overflow/status dots, live child transcripts with full main-transcript parity, steer (message a running child) and stop, consent flow, and report chips/panels.
• Reports delivered exclusively via the injection channel (<subagent_report> messages); kickoff message echoed onto the child stream at run start.
• Lifecycle fixes: children lookup for auto-mode parents (leaf trace id aliasing), working indicator for a child awaiting its first response, errorText surfaced in chat stream errors.
• Regenerated OpenAPI web types; desktop version bumped to 1.0.4 (the release that ships sub-agents).

Follow-up work on this stack lands in PR #1558 (into this branch).

Note: the judge_feedback_batch subsystem this stack previously carried has been removed entirely — it was an MVP prompt-optimizer attempt we decided to drop.

Latest: mid-turn message injection, event-loop stalls, forensic debug logging

• Main-agent queued messages now inject mid-turn. Sends on a live interactive run POST straight to the server inbox (the transport sub-agents/cross-tab already used); the engine folds them into the next upstream round instead of holding them until the whole turn settles. The client-side queue remains only for non-injectable moments (approval box open, consent resolving, no session id, failed POST) and flushes into the live run as soon as it becomes injectable. Server complement: idle sends drain any stranded inbox (error-settle leftovers) into the fresh turn, in send order.
• Remaining event-loop stalls fixed (the /api/conversations?parent= hang/pile-ups): agent_overview and get_tags (plus a third endpoint since removed with the judge-feedback-batch subsystem) still ran full dataset disk scans on the loop; now offloaded via asyncio.to_thread (same treatment as a02cfeb).
• Forensic debug logging (KILN_CHAT_DEBUG_LOG, defaulted on in the dev server): JSONL timeline of run/round/tool-batch/inbox events (ids + timings, never content). Every upstream request now carries X-Kiln-Conversation-Id, joining the backend's debug log (kiln_server PR #255) per conversation; the chat footer shows a copyable conversation id while the flag is on.
• Sub-agent tab strip resilience (from the first debug-log post-mortem, cv_dvs5hq23cn23): the children-list fetch is now bounded (AbortSignal.timeout(10s)) so a request stuck behind the browser's per-origin connection cap can no longer permanently blind the reconcile loop; same-parent blips keep the current tabs (no flicker) while failed cross-parent switches still clear; a 15s watchdog reconnects a firehose EventSource wedged before headers; failed reconciles are logged instead of swallowed. Forensics: sub-agent settles emit run_settled to the debug log, and the empty-model-response error carries finish_reason.

---

Carried from #1517: assistant auto-mode + the eval/judge job backend (context gauge, chat resilience, queued messages)

This is the app branch behind KIL-686 (background jobs) that has grown to carry the whole assistant-driven eval/optimize workflow. It began as "run an eval as a background job" and consolidated several stacked PRs — auto mode (#1518), the context gauge / compaction indicator (#1519), the jobs-API cleanups (#1527), auto-mode resilience (#1529), worker properties (#1530), chat retry + Stop (#1532/#1533), and queued messages (#1534) — into a single branch so one desktop_server serves everything the merged kiln-chat skill needs.

No existing eval UI is changed; the eval job work is purely additive (new background-job paths for agents).

Note: the judge_feedback_batch subsystem (#1536) previously carried by this branch has been removed entirely — it was an MVP prompt-optimizer attempt we decided to drop.

Main features

• Eval as a background job — EvalJobWorker + a typed, non-streaming POST /api/jobs/evals/run kickoff for agents, plus a bulk POST /api/jobs/wait to await many jobs at once. Agents use these instead of the SSE eval endpoints.
• Assistant auto mode — the assistant can run steps autonomously (reflective optimization) with explicit consent and a hard Stop, backed by a self-contained chat/auto/ app-server subsystem and matching web UI.
• Context-usage gauge + "Summarizing…" indicator (KIL-727) — a glanceable context-window gauge in the assistant footer and a compaction indicator during the server's pre-inference summarization window.
• Chat resilience — transient-error retry with ramping backoff in both interactive and auto modes, plus a real Stop control (confirmed by a dialog).
• Mid-turn message queueing — messages typed during a turn are queued (not dropped or fired immediately) with Send-now / Edit / Discard controls, and flushed at the next round boundary.
• Jobs-table run summaries — workers publish static descriptive properties; the jobs table renders an eval run-config-style summary.
• Structured add-example dialog (Support structured data entry in the add example dialog #1540, via main) — the add/edit example dialog is now a reusable component with schema-aware, field-by-field entry.

Agent permissions

Agent-allowed job endpoints: POST /api/jobs/evals/run (allow + approval — evals cost AI credits) and POST /api/jobs/wait (allow). The UI-only /api/chat/auto/* set and the SSE run_comparison / run_calibration eval endpoints are agent-forbidden (deny). Regenerated agent-check annotations and the OpenAPI TS schema throughout.

Testing

New worker, endpoint, registry, auto-mode, retry, queued-message, context-gauge, and compaction-indicator tests across Python and web. uv run ./checks.sh --agent-mode → green (Python lint/format/type/tests, web lint/format/check/test/build, OpenAPI schema in sync).

Changelog

Linear with the commit history — each entry is one holistic increment (roughly one merged sub-PR / ticket). Append new entries at the bottom.

1. Run evals as background jobs

• EvalJobWorker (jobs/workers/eval.py) wraps the existing EvalRunner so an eval runs in the background. Idempotent, supports_pause=True (EvalRunner excludes already-run (eval_config, run_config, dataset) triples), progress against full eval-set size.
• POST /api/jobs/evals/run — typed, non-streaming kickoff (body is the 5 eval ids; returns {job_id, status}). Allow + requires-approval. Two-segment path so it can't be shadowed by the generic job route.
• SSE eval-run endpoints (run_comparison, run_calibration) flipped to agent-forbidden.
• Reconcile fix: JobDerivedState.error is now int | None; a worker that can't derive its error count from entities returns None, and _apply_derived keeps the live reported count instead of clobbering it to 0.

2. Assistant auto mode (rescoped, #1518)

• Self-contained chat/auto/ subsystem: auto-run registry, runner, event bus, SSE, and API (/api/chat/auto/*: enable, decline, resolve, sessions, per-run message/stop/events).
• Core: enable_auto_mode / disable_auto_mode built-in tools + tool-registry wiring.
• Web UI: auto_run_store, consent dialog, chat-history grouping, reworked chat.svelte (in-transcript live working/idle, inject-on-send, reconnect handling), preserving the base's UI refactors (scrollbar-to-side, DaisyUI send/stop, centering wrapper).
• Salvaged from the dropped eval/finetune/RAG-via-job refactor stack (feat: run eval, finetune via background jobs #1436/feat: run RAG via background jobs #1450); the one real coupling (a generic SSE keepalive helper) was ported into jobs/events.py (KeepalivePing / iter_with_keepalive, generic over a TypeVar).
• Review fixes: Literal["user"] role, optimistic-flag reset on inject failure, consent-event trace fallback, keyboard-focus reveal, consentPending held through requestEnable, and a guard so resyncOnLoad can't overwrite a newly-selected session on a mid-flight conversation switch.

3. Multi-job wait + eval per-item error logging

• Multi-job wait backed by JobRegistry.wait_many() — blocks until all given jobs are terminal and returns their records. Lets a caller await several eval jobs in one call.
• EvalRunner.run() now accepts observers; EvalJobWorker forwards each failed dataset item's exception to ctx.report_error (with dataset_id + run_config_id). Previously only the error count was reported, so GET /api/jobs/{id}/errors showed "no errors recorded" even when every item failed.

4. Context-usage gauge + compaction indicator — KIL-727 (#1519)

• studio_server proxy — ChatSessionSnapshot gains a context_usage field (context_tokens, context_limit, context_percent, compacted) so a session reopened from history renders the gauge. Relies on Pydantic's default extra="ignore" (made explicit) to drop unmodeled upstream keys at the client boundary (extra="forbid" deliberately avoided). SSE stays a raw passthrough; passthrough tests added.
• Context gauge (context_usage_gauge.svelte) — a small grey bar in the input footer (opposite Auto mode), percentage above and a tooltip showing approximate used / total tokens; hidden before the first turn. Token counting is intentionally approximate (overestimate-biased; copy uses "≈"). Tooltip fixed to measure after visible and made keyboard-accessible.
• "Summarizing…" indicator — on the backend's compaction signal, the Thinking activity indicator shows "Summarizing earlier messages to free up context…" during the pre-inference window. Not cleared on finished (a fast started→finished pair would collapse the window); cleared by the first real assistant content, on error, and on reset.
• Wired through streaming_chat.ts / chat_session_store.ts / auto_run_store.ts so the gauge updates during auto-mode bursts too.

5. Auto/manual congruence + gauge polish

• The auto runner and the manual chat path are kept congruent so auto-mode bursts behave identically to a user driving each step; tool-availability reconciled between manual and server-driven runs, and tool-congruence handling hardened (chat/auto/runner.py).

6. Jobs-API consolidation — bulk wait only (#1527)

• Dropped the single-job GET /{id}/wait (bulk wait covers it with one id) and the generic POST /api/jobs/{type} create endpoint (production only runs evals, via the typed route — the generic path had no real caller).
• GET /api/jobs/wait converted to POST /api/jobs/wait with a WaitForJobsRequest body (ids as JSON, not repeated query params); with the generic POST gone, the route-ordering workaround is dropped.
• The temporary /jobs test page becomes a real jobs panel; NoopJobWorker stays as a registry-level test fixture but is no longer registered in production.

7. Auto-mode resilience — KIL-692 (#1529)

• Side-note framing — mid-burst user messages are wrapped in a <system-reminder> telling the model to weave its reply into ongoing work and keep going, so a quick aside no longer settles the burst IDLE.
• Transient-error retry — bounded full-jitter exponential backoff on retryable upstream failures (429/5xx). Only failures that streamed no content are retried (no duplicate output). New auto-mode-retry SSE event drives a "retrying N/M…" transcript affordance.
• Stop pressed while retrying settles USER_STOPPED (publishes auto-mode-off) instead of leaving the flag on.

8. Worker properties + eval summary in the jobs table (#1530)

• Generic mechanism for workers to publish static, descriptive properties derived once from params at create time (mirrors the compute_state pure-read pattern; stored dict-on-the-wire like progress_detail, computed behind a guard so a failing describe() never breaks job creation).
• The eval worker publishes eval name, run config (name, model, resolved prompt name, tool/skill counts), and judge (name, algorithm, model), resolving prompt ids to names the way the frontend does. The jobs-table Details cell renders this as a run-config-style summary.
• Hardening: full describe() → type-check → model_dump() path wrapped so a bad payload can never break create(); undeclared-model case made explicit; test coverage for the contract guard and the previously-dead prompt-resolution branches.

9. Surface the original eval error (#1531)

• The eval error-log observer unwraps KilnRunError.original so GET /api/jobs/{id}/errors shows the real failure instead of the generic "An unexpected error occurred." wrapper text.
• Hardened against a buggy __str__ on the original exception (falls back to the class name), and covered the common RetryableError(str(original)) path.

10. Chat retry in both modes + hard Stop (#1532, #1533)

• Shared iter_round_with_retries (stream_session.py) owns the retry loop and is used by both the auto runner and interactive ChatStreamSession.stream() — previously only auto mode retried. Backoff is an explicit ramp capped at 60s (1, 2, 5, 10, 20, 30, 60, 60, 60, 60, ±15% jitter, ~5 min total); MAX_CHAT_RETRIES derives from its length.
• SSE event generalized auto-mode-retry → kiln-chat-retry; the frontend renders "Temporary issue — retrying N/M…" from either source. Stop is honored during the backoff (no re-POST).
• Auto mode's "waiting for you" affordance replaced with a real Stop button, confirmed by auto_mode_stop_dialog.svelte (notes that already-kicked-off background jobs keep running).

11. Mid-turn message queueing (#1534)

• A message typed while a turn is in flight is held in a client-side queue instead of being dropped (interactive) or fired immediately (auto). The composer surfaces the queued message with Send now, Discard, and Edit; a second send coalesces.
• Auto-sends when the turn yields (auto: injected at the next round boundary) and appears in the trace at the point it was actually sent; the textarea is no longer disabled during a turn.
• Re-attach/refresh correctness: dedupe injected-message echo, render an in-flight round into a fresh turn, preserve queued text on rejected dispatch, and strip <system-reminder> side-note framing from hydrated transcripts.

12. Eval job model docs

• Added Field descriptions to the eval job models.

13. Judge feedback batch subsystem (#1536) — since removed

• This branch previously merged the judge_feedback_batch capability (datamodel + runner + synchronous REST API + job-backed worker) as an MVP prompt-optimizer gate. See entry 15 — it has been removed from the branch entirely. The eval job's validated concurrency param (feat(judge-feedback-batch): expose concurrency in job params #1544) remains.

14. Structured add-example dialog (#1540, via main)

• The data-guide add/edit example dialog is promoted to a reusable $lib/components/add_example_dialog.svelte with configurable include_input / include_output and schema-aware, field-by-field entry (RunInputFormElement) for structured tasks; plaintext tasks keep the textarea. The route-local copy is deleted and callers consume the shared component + exported GuideSample type.

15. Remove the judge feedback batch subsystem

• Deleted the whole judge_feedback_batch subsystem added in entry 13: core datamodel + runner, the synchronous REST API, the job-backed worker, their tests, the jobs-table UI branch, the OpenAPI tag, and the (never-shipped) agent-check annotations. Regenerated the OpenAPI TS schema. It was an MVP prompt-optimizer attempt, dropped in favor of not shipping it at all.

🤖 Generated with Claude Code