Skip to content

Worker that exits before start_session leaves no session row, so the failure is never persisted or surfaced #124

Description

@Dor-bl

Summary

When a task worker exits before DataEngine.start_session() inserts its sessions row, the queue worker's terminal-status write can never succeed. session_repo.update_session_status() is a bare UPDATE ... WHERE session_id = ? that returns False when no row matches, and there is no INSERT/upsert fallback. The run therefore ends with no row in the DB and nothing in /api/sessions, so no API consumer (Showcase UI, MCP pollers) ever observes a terminal state for the task. The only artifact left behind is an orphaned traces/<session_id>/ directory.

This is provider-agnostic: any pre-start_session failure reaches it — a missing or invalid credential, a config resolution error, a device acquisition failure, an import error.

Observed

Submitting a Flash task through the Showcase UI with only ANTHROPIC_API_KEY set (no Google key). The worker aborts during provider validation:

╭──────────────────────── Missing API Key ─────────────────────────╮
│ ✖ Authentication Error: Planner requires GOOGLE_API_KEY in .env  │
╰──────────────────────────────────────────────────────────────────╯

[QueueWorker] Task [ab5b3685-…] exited with returncode 1
ERROR  [QueueWorker] Could not persist terminal DB status 'failed'   task_queue_service.py:662
       for session ab5b3685-…

State immediately after the run:

sqlite> select count(*) from sessions;   ->  0
GET /api/sessions                        ->  []
ls traces/ab5b3685-…/                    ->  exists, orphaned

The Could not persist terminal DB status line goes to the server console only. From every API surface, the task never existed — it does not appear as failed, and it does not appear at all.

The authentication error itself is a separate concern (see #95 / #47); this report is about the status-persistence path, which behaves the same way for any early worker exit.

Mechanism (at 371aa6d)

File Line What happens
artemis/config/llm.py 91 Provider validation raises "{name} requires GOOGLE_API_KEY in .env".
artemis/interfaces/cli/commands/run.py 498-510 The worker catches it, prints the panel to its own stdout, and raise SystemExit(1).
artemis/sdk/agent.py 1161 context.data_engine.start_session(...) — never reached, so no sessions row is inserted.
apps/admin_console/database/repositories/session_repository.py 513-527 update_session_status() runs UPDATE sessions SET status = ? ... WHERE session_id = ? and returns cursor.rowcount > 0; with no row this is permanently False, and there is no INSERT/upsert path.
apps/admin_console/services/task_queue_service.py 655-665 _persist_terminal_session_status() logs the failed write and moves on.
apps/admin_console/services/task_queue_service.py 669-692 The 3-attempt retry loop wraps only the trace_store write (catching OSError); the DB write above it gets no retry.

The comment at task_queue_service.py:657-659 notes that "a failed DB write must not pass silently" because the DB row is what MCP pollers reconcile against — but in this case the write cannot succeed at all, since the row does not exist.

Steps to reproduce

  1. Configure .env with a non-Google credential only (e.g. ANTHROPIC_API_KEY=...), leaving GEMINI_API_KEY / GOOGLE_API_KEY unset.
  2. Start the server (./start.sh) and submit any task from the Showcase UI.
  3. Observe the worker exit with returncode 1 and the Could not persist terminal DB status error in the server console.
  4. GET /api/sessions returns [], and select count(*) from sessions in traces/data_engine.db is 0, while traces/<session_id>/ exists.

Observed: the failed task leaves no record on any API surface; the submitting client has no terminal state to transition to.

Expected: a task that fails before its session row exists is still recorded as failed and exposed through /api/sessions, with the worker's error message retained.

Suggested fix

  • Make the terminal-status write an upsert (INSERT ... ON CONFLICT(session_id) DO UPDATE), or have _persist_terminal_session_status() fall back to inserting a minimal failed row when update_session_status() reports no row matched. The queue worker already knows the session id, the goal and the returncode.
  • Alternatively, insert the sessions row at enqueue time rather than inside the worker, so a row always exists to update.
  • Consider capturing the worker's stderr/stdout tail into the persisted record, so the reason for an early exit is visible without reading the server console.
  • Extend the retry loop to cover the DB write, or distinguish "transient write failure" (retry) from "no such row" (upsert) — the current code treats a permanent condition as a loggable transient one.

Environment

  • Commit 371aa6d, macOS 15 (arm64), Python 3.14.5
  • Physical Android device connected and detected (adb devices OK)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions