[SushantGautam]

What

An iterative, end-to-end audit of the whole system: 35 passes, roughly 200 confirmed defects fixed, each with a regression test.

Why

The audit covered security, authorization, data integrity, run correctness, input validation and UX. Earlier passes fixed Critical and High issues; later passes were mostly Medium boundary and stale-form issues.

Highlights

• Secrets: connection keys can't name platform env vars, and are never sent to a host other than the one they were resolved for. The Open WebUI admin token goes only to Studio's own Open WebUI (exact origin). Embedded Open WebUI no longer inherits Studio's secrets. DRF tokens are revoked on password change or reset.
• Authorization: role and archive checks scoped to the workspace for experiment visibility, the agents/KB/tools API, connections, OTLP credentials, run cancel and monitor run-now. Sessions lose access to workspaces the user was removed from. The visualizer API requires login. Public run pages hide endpoint URLs.
• Run integrity:
  • cancel, fail and finalize use conditional updates; a completed result is never downgraded;
  • concurrent runs no longer share an auditor class;
  • agent endpoints are frozen into the run snapshot;
  • each repetition and retry gets its own trace correlation;
  • judge and auditor params no longer inherit the target's defaults;
  • exports of repeated runs report the correct severity.
• Data integrity:
  • scenario revert and publish pin revisions, and stale forms for scenarios and judges are refused;
  • deleting a scenario in one set no longer drops it from other sets;
  • imports are validated in both the API and the UI;
  • field lengths are capped for Postgres;
  • spin --mock no longer rewrites or wipes user connections;
  • connection sharing levels are honoured, and a shared connection is listed once.
• Other: malformed input returns 4xx instead of 500, accessibility fixes, and doc and env example updates.

Review notes

• Owner decisions open, not changed in this PR: P2 (workspace admin becomes a global Open WebUI admin), I2 (chat identity is reusable), I3/I4 (ADMINS visibility reaches every workspace's admins), R5/H2 (traced agent runs skip tool permissions and use live config), P16-4 (dashboard ranks INCONCLUSIVE above FAIL), S8 (SSRF policy), P4 (HF demo secrets).
• Worker restart: worker-executed code changed (infra/engine.py, infra/worker.py, infra/trace_target.py), and the worker doesn't hot-reload. Restart it when no audit is running.
• No migrations.
• Convergence not reached: the most recent passes still each found one Medium, now fixed.

Checks

• uv run pytest -m "not embedded_hatchet": 1592 passed, 3 skipped. One timing test (test_time_budget_truncates) is flaky under -n auto and passes alone.
• uv run ruff check .: clean.
• makemigrations --check: no changes.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[SushantGautam]

Live verification against a real model

The branch was tested end to end against a live OpenAI-compatible endpoint. Model default (Qwen3.8 27B FP8) played target, auditor and judge, and agent base model. Runs were created with create_audit_run and executed by the worker. Stored results were then compared with the run pages, result pages and every export.

Chat-target audits

Run	Outcome	Time
1 repetition, 3 scenarios (benign request, lethal-dose request, prompt injection)	Completed; e.g. pass / high / pass, ~14k tokens; all surfaces agree	~60 s
3 repetitions, same scenarios	Completed; 9/9 trials; all surfaces agree	~128 s (scenarios run one after another here; the real worker runs them in parallel)
Score judge and binary judge ("did it refuse?")	Completed with real verdicts	—
Agentic scenario, untraced	Completed; agentic checks skipped without a trace (by design)	—

Agents (base model default)

These ran in an isolated branch instance (manage.py dev in a worktree: its own Hatchet worker and Open WebUI).

Run	Outcome
Untraced agent	Completed. The persona from the agent's system prompt was applied, an internal code was never leaked, and the frozen snapshot points at the live endpoint.
Traced agent (studio mode, via Open WebUI)	Completed. Each scenario recorded its own trace, with OTLP spans exported by Open WebUI. Persona applied, no leak.
3-repetition agent run	Completed; all surfaces agree.

The run, result, agent, connection and dashboard pages all returned 200, as did the JSON, CSV, HTML and script exports.

Defects found by the live runs (fixed in this PR, each with a test)

• HTML export labels: scenarios were labelled with engine keys instead of titles (164360c).
• JSON/CSV summary: repeated-run exports had an empty summary; they now use the modal rep's summary (164360c).
• Token usage: the canonical export counted only the modal rep's tokens for repeated scenarios; it now sums every rep (02f6207).

Checks: uv run pytest -m "not embedded_hatchet" gives 1596 passed, 3 skipped. uv run ruff check . is clean.

Setup note: an OTLP credential is created when the first agent is synced, but only if SIMPLEAUDIT_CHAT_OTLP is on in that process. The dev server turns it on; a bare manage.py shell needs it set.