cherry-pick: viewer middle-click support (upstream #3442)#24
Merged
Conversation
QuantumLove
force-pushed
the
cherry-pick/viewer-middle-click
branch
from
5311c43 to
6102eee
Compare
Author
|
Small good deed, UX request |
rasmusfaber
approved these changes
Mar 18, 2026
rasmusfaber
left a comment
rasmusfaber
left a comment
There was a problem hiding this comment.
Merge conflict (just needs a merge/rebase and a new frontend build).
Note that you will also need a PR for inspect-action with a new NPM build.
Cherry-pick of UKGovernmentBEIS#3442. Adds onCellMouseDown handler to LogListGrid and SamplesGrid for middle-click (button === 1) to open items in new browser tabs. SampleList already supports this natively via <a> tags on the hotfix branch. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
QuantumLove
force-pushed
the
cherry-pick/viewer-middle-click
branch
from
6102eee to
5de22b7
Compare
github-merge-queue Bot
pushed a commit
to METR/inspect-action
that referenced
this pull request
Mar 18, 2026
## Summary Bumps `inspect-ai` git pin from `b47eb00c` to `9e879d16` (hotfix HEAD). This brings in middle-click support to open tasks and samples in a new browser tab in the log viewer, cherry-picked from upstream [UKGovernmentBEIS/inspect_ai#3442](UKGovernmentBEIS/inspect_ai#3442) and merged into the `hotfix` branch via [METR/inspect_ai#24](METR/inspect_ai#24). ## Changes - `pyproject.toml`: bump `inspect-ai` rev - `uv.lock` + all `terraform/modules/*/uv.lock`: regenerated
revmischa
pushed a commit
to METR/hawk
that referenced
this pull request
Mar 24, 2026
Bumps `inspect-ai` git pin from `b47eb00c` to `9e879d16` (hotfix HEAD). This brings in middle-click support to open tasks and samples in a new browser tab in the log viewer, cherry-picked from upstream [UKGovernmentBEIS/inspect_ai#3442](UKGovernmentBEIS/inspect_ai#3442) and merged into the `hotfix` branch via [METR/inspect_ai#24](METR/inspect_ai#24). - `pyproject.toml`: bump `inspect-ai` rev - `uv.lock` + all `terraform/modules/*/uv.lock`: regenerated
revmischa
pushed a commit
to METR/hawk
that referenced
this pull request
Mar 24, 2026
## Summary Bumps `inspect-ai` git pin from `b47eb00c` to `9e879d16` (hotfix HEAD). This brings in middle-click support to open tasks and samples in a new browser tab in the log viewer, cherry-picked from upstream [UKGovernmentBEIS/inspect_ai#3442](UKGovernmentBEIS/inspect_ai#3442) and merged into the `hotfix` branch via [METR/inspect_ai#24](METR/inspect_ai#24). ## Changes - `pyproject.toml`: bump `inspect-ai` rev - `uv.lock` + all `terraform/modules/*/uv.lock`: regenerated
revmischa
pushed a commit
to METR/hawk
that referenced
this pull request
Mar 25, 2026
Bumps `inspect-ai` git pin from `b47eb00c` to `9e879d16` (hotfix HEAD). This brings in middle-click support to open tasks and samples in a new browser tab in the log viewer, cherry-picked from upstream [UKGovernmentBEIS/inspect_ai#3442](UKGovernmentBEIS/inspect_ai#3442) and merged into the `hotfix` branch via [METR/inspect_ai#24](METR/inspect_ai#24). - `pyproject.toml`: bump `inspect-ai` rev - `uv.lock` + all `terraform/modules/*/uv.lock`: regenerated
revmischa
pushed a commit
to METR/hawk
that referenced
this pull request
Mar 25, 2026
Bumps `inspect-ai` git pin from `b47eb00c` to `9e879d16` (hotfix HEAD). This brings in middle-click support to open tasks and samples in a new browser tab in the log viewer, cherry-picked from upstream [UKGovernmentBEIS/inspect_ai#3442](UKGovernmentBEIS/inspect_ai#3442) and merged into the `hotfix` branch via [METR/inspect_ai#24](METR/inspect_ai#24). - `pyproject.toml`: bump `inspect-ai` rev - `uv.lock` + all `terraform/modules/*/uv.lock`: regenerated
revmischa
added a commit
to METR/hawk
that referenced
this pull request
Mar 25, 2026
* fix: rename hawk auth auth-login to hawk auth login (#974) ## Summary This PR fixes an awkward command naming that was introduced in commit b2efdc4 (PR #684). The auth login command was accidentally named `auth-login` within the auth group, creating the redundant command path `hawk auth auth-login`. This change: - Renames `hawk auth auth-login` to `hawk auth login` for consistency - Maintains the root-level `hawk login` command for backward compatibility - Both `hawk login` and `hawk auth login` now work as expected ## Context The issue was introduced on December 24, 2025 when the auth command group was added. The login command within the group was mistakenly given the name "auth-login" instead of just "login", resulting in the awkward `hawk auth auth-login` command. ## Testing & Validation - [x] All CLI tests pass (`uv run pytest tests/cli/ -n auto`) - [x] Both `hawk login` and `hawk auth login` commands work correctly - [x] Verified with `hawk --help` and `hawk auth --help` - [x] No errors or warnings from basedpyright ## Checklist - [x] Code follows the project's style guidelines (ruff check and format pass) - [x] Self-review completed - [x] Tests pass - [x] Documentation references to `hawk login` remain accurate * fix: add diagnostic logging for Okta token refresh failures (#981) ## Summary - Log the full HTTP response body from Okta when token refresh fails (both `refresh_token.py` and `credential_helper.py`) - Truncate error bodies to 500 chars to avoid huge log entries - Use specific exception types `(OSError, ValueError)` instead of bare `Exception` ## Context Investigating recurring `invalid_refresh_token` errors from Okta affecting eval sets. Previously, only the HTTP status code was logged on refresh failures — the Okta error body (which contains the specific error reason like `invalid_refresh_token`) was discarded. This made it difficult to diagnose the root cause. Related investigation: IAM PR #152 was closed — `refresh_token` is not a valid value for Okta policy rule `grantTypes.include`. ## Test plan - [ ] Deploy to staging - [ ] Trigger a token refresh failure and verify the Okta error body appears in logs - [ ] Verify normal refresh flow still works (no regressions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix(eval_log_reader): add s3:GetObject IAM permission for .models.json (#976) ## Summary - PR #972 added `.models.json` fallback logic to the eval_log_reader Lambda, but the Lambda's IAM role lacks `s3:GetObject` permission on the supporting S3 access point - The access point policy grants the permission on the resource side, but the Lambda role also needs an explicit IAM Allow - Production logs confirm: `"Could not read evals/.../.models.json: AccessDenied"` ## Root cause The S3 access point policy added in #972 grants `s3:GetObject` to the Lambda role for `evals/*/.models.json`, but S3 authorization requires the IAM principal to also have the permission in its own policy. The access point resource policy alone is not sufficient. ## Fix Add `s3:GetObject` to the Lambda role's `policy_statements` in `lambda.tf`, scoped to `evals/*/.models.json` on the supporting access point. ## Test plan - [ ] `tofu plan` shows only the IAM policy change (no other resource modifications) - [ ] Apply to production and verify artifact access works via `aws s3 cp` from the Object Lambda endpoint - [ ] Confirm Lambda logs show `"using .models.json from evals/..."` instead of `AccessDenied` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Remove fargate spot for API (#984) Causing some annoyance on staging --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Cherry pick fixes (#987) Cherry-pick METR/inspect_ai@cbdaa28 and METR/inspect_ai@4b7f380 * Pre-release option for publish version script * Lock all after running --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Hotfix for unclosed connector error silencing (#988) METR/inspect_ai@b47eb00 * security: bump joserfc >=1.6.3 (CVE-2026-27932) (#980) ## Summary Bumps joserfc minimum version from 1.0.4 to 1.6.3 to fix **CVE-2026-27932**. ## Why this is urgent **token-broker-lambda** is the only internet-facing Lambda — it has a public Function URL with `authorization_type = "NONE"`. External attackers can send crafted JWTs to the endpoint, which are passed directly to `joserfc.jwt.decode()` for validation. ## Changes | File | Change | |------|--------| | `terraform/modules/token_broker/pyproject.toml` | `joserfc>=1.0.4` → `joserfc>=1.6.3` | | `pyproject.toml` (api extra) | `joserfc>=1.0.4` → `joserfc>=1.6.3` | | `pyproject.toml` (cli extra) | `joserfc>=1.0.4` → `joserfc>=1.6.3` | ## Context - PR #968 in this repo attempted the same fix but was closed on March 9 - Companion PR: [METR/platform#136](METR/platform#136) - Identified during weekly security triage (2026-03-16) * chore: bump inspect-ai to 9e879d16 (viewer middle-click support) (#989) Bumps `inspect-ai` git pin from `b47eb00c` to `9e879d16` (hotfix HEAD). This brings in middle-click support to open tasks and samples in a new browser tab in the log viewer, cherry-picked from upstream [UKGovernmentBEIS/inspect_ai#3442](UKGovernmentBEIS/inspect_ai#3442) and merged into the `hotfix` branch via [METR/inspect_ai#24](METR/inspect_ai#24). - `pyproject.toml`: bump `inspect-ai` rev - `uv.lock` + all `terraform/modules/*/uv.lock`: regenerated * Fix e2e flakiness from minikube memory exhaustion (#992) ## Summary - Lower runner memory in `.env.local` from 16Gi to 512Mi (sufficient for e2e dummy/simple tasks) - Set explicit `--memory=4096` on `minikube start` for predictable behavior ## Problem Runner pods request 16Gi memory limits but minikube starts with default memory (~2-4GB). When multiple eval sets run concurrently in e2e tests, the second pod fails to schedule with `Insufficient memory`. ## Test plan - [ ] E2e tests pass consistently without "Insufficient memory" errors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Add Datadog metrics hook for rate limit visibility (#978) - Adds `hawk/runner/datadog_metrics.py` — a new inspect_ai hook that emits model usage metrics to DogStatsD - Wired into all three runner entry points (`run_eval_set`, `run_scan`, `run_scan_resume`) - Gated by `INSPECT_DATADOG_METRICS_ENABLED` env var - Updates CiliumNetworkPolicy to allow DogStatsD UDP egress to host:8125 - Includes a Datadog dashboard JSON for visualizing metrics | Metric | Type | Tags | |--------|------|------| | `inspect.model.tokens.input` | count | model, eval_set_id, task_name, run_id | | `inspect.model.tokens.output` | count | model, eval_set_id, task_name, run_id | | `inspect.model.tokens.total` | count | model, eval_set_id, task_name, run_id | | `inspect.model.call_duration` | histogram | model, eval_set_id, task_name, run_id | | `inspect.model.retries` | count | model, eval_set_id, task_name, run_id | | `inspect.eval_set.active` | gauge | eval_set_id | Runner pods send DogStatsD metrics via UDP to the Datadog agent's hostPort (8125) on the node. The CiliumNetworkPolicy needed a `toEntities: host` rule (not `world`) since the traffic goes to the node itself, not outside the cluster. Model names include provider prefixes (e.g. `openai/gpt-4`). The hook strips the prefix so only the model name appears in tags (`model:gpt-4`), avoiding exposure of provider-model associations in Datadog. - Requires [inspect_ai PR `ModelUsageData` eval context fields (`eval_set_id`, `run_id`, `eval_id`, `task_name`, `retries`) - No new Python dependencies — uses a minimal built-in UDP client for DogStatsD protocol | Var | Default | Description | |-----|---------|-------------| | `INSPECT_DATADOG_METRICS_ENABLED` | (unset) | Set to `1` or `true` to enable | | `DOGSTATSD_HOST` | `localhost` | DogStatsD agent host | | `DOGSTATSD_PORT` | `8125` | DogStatsD agent port | - [x] Run with `INSPECT_DATADOG_METRICS_ENABLED=true` and verify metrics appear in Datadog Metrics Explorer - [x] Run without the env var and verify no metrics are sent - [x] Verify provider prefix is stripped from model tags - [ ] Verified end-to-end on staging: `inspect.model.tokens.input/output/total` all appearing with `model:gpt-4o-mini` tag - [ ] Enable DogStatsD hostPort on production DatadogAgent CRD 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat: add RLS functions, roles, and policies for model group access control (#962) Adds the database infrastructure for row-level security (RLS) so read-only warehouse users will only see data for models they have access to. Builds on #951 (model group mapping). **RLS is not yet enabled** — SECURITY`. - Creates RLS policies on all 8 public tables with cascading logic: `eval` and `scan` check model access via `user_has_model_access()`; child tables (`sample`, `score`, `message`, `sample_model`, `scanner_result`) cascade via `EXISTS` against their parent; `model_role` checks model access directly to avoid circular recursion - `user_has_model_access(text[])` SQL function checks `middleman.model → model_group → pg_has_role(current_user, group_name)` - `get_eval_models()` / `get_scan_models()` — SECURITY DEFINER helpers that read `model_role` bypassing RLS (prevents circular recursion between eval/scan and model_role policies) - `sync_model_group_roles()` creates NOLOGIN PostgreSQL roles matching model group names (SECURITY DEFINER, with REVOKE EXECUTE FROM PUBLIC) - Migration creates `model-access-public` role explicitly (standard naming convention matching JWT claims and `.models.json`) - Bypass policies for `rls_bypass` role so `inspect` app user can bypass RLS (it does its own access control) - `import_model_configs.py` now syncs roles and grants after import, with role existence checks for dev environments - Migration SQL is inlined (not imported from app code) to ensure immutability across environments - **Terraform note:** `inspect_ro_secret` must be added to `warehouse_read_only_users` in tfvars files (gitignored, applied separately) **What this PR does NOT do:** - Does not enable RLS (`ALTER TABLE ... ENABLE ROW LEVEL SECURITY`) — see follow-up #990 **User roles (v1):** - `inspect_admin` — will bypass RLS (rds_superuser) - `inspect` — bypass policies ready, full read/write - `inspect_ro` — gets `model-access-public` role only - `inspect_ro_secret` — gets ALL model group roles (full researcher access) Linear: PLT-274, related PLT-345 - [x] 16 RLS tests in `tests/core/db/test_rls.py` covering: - Eval/scan with accessible model → visible - Eval/scan with inaccessible model → hidden - Child rows (sample/score/message/sample_model/scanner_result) of hidden parent → hidden - model_role of hidden eval → hidden; model_role of visible eval → visible - NULL model scan → visible - Unknown model → visible (not managed by middleman, treated as public) - Mixed model_roles requiring multiple groups → hidden when user lacks any group - Table owner bypasses RLS - sync_model_group_roles creates NOLOGIN roles and is idempotent - Public groups visible without explicit role grant - Model group without PostgreSQL role hides its models - [x] All DB tests pass (`pytest tests/core/db/ -n auto -vv`) - [x] Alembic migration tests pass (apply, downgrade, upgrade cycle) - [x] `ruff check`, `ruff format`, `basedpyright` — all clean - [ ] After deploy: verify functions and policies exist in database - [ ] After deploy: verify `sync_model_group_roles()` creates expected roles 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat(hawk): add cross-lab scan safeguard (#985) Prevents scanners from one AI lab from reading private model transcripts from a different lab (e.g. an Anthropic scanner cannot scan transcripts from a private OpenAI model). **Depends on**: [middleman update](metr-middleman/middleman-server#238) (Middleman `labs` field + platform/hawk port) must be deployed before this takes full effect. The implementation gracefully degrades when Middleman doesn't return `labs` yet (`labs={}` → check skipped with warning). The original implementation (closed PR #934) was blocked by the "qualified name problem": `.models.json` stores unqualified model names like `"gpt-4o"`, so `parse_model("gpt-4o").lab` returns `None` and the cross-lab check silently skipped every eval-set model. Ask Middleman at scan time instead of storing lab info at eval-set creation time. Middleman already knows each model's lab — we just needed it in the `/model_groups` response. Works for all existing eval sets with no data migration. **`hawk/api/auth/middleman_client.py`** - `ModelGroupsResult(groups, labs)` — new return type for `get_model_groups()` - Graceful fallback: `labs` field has `default_factory=dict`, handles old Middleman versions automatically **`hawk/api/scan_server.py`** - `_validate_cross_lab_scan()`: - Public models (`model-access-public`) always exempt - Lab comparison uses strict string equality — no normalization - Data issues (missing labs, unknown labs) → warning logged to Sentry, scan proceeds (fail-open) - Only actual cross-lab mismatches raise `CrossLabScanError` (403) - Collects all violations before raising - `allow_sensitive_cross_lab_scan` on both `CreateScanRequest` and `ResumeScanRequest` **`hawk/cli/`** - `--allow-sensitive-cross-lab-scan` flag on `scan run` and `scan resume` - Error hint pointing to the flag when a cross-lab error is returned **`hawk/api/problem.py`** - `CrossLabViolation` dataclass + `CrossLabScanError` (403) - [PLT-671](https://linear.app/metr/issue/PLT-671): Switch cross-lab data violations from fail-open (warnings) to fail-closed (errors) once we've validated the Middleman lab data in production - Unit tests in `tests/api/test_scan_server_unit.py` covering: same-lab allowed, cross-lab blocked, public exempt, bypass flag, no scanner models, old Middleman fallback, data issues warn not block, multiple violations, unknown scanner lab still compared - Integration tests in `tests/api/test_create_scan.py` updated for `ModelGroupsResult` return type - Manually tested on dev3: OpenAI scanner vs private gemini-pro → 403 ✅, same + bypass flag → 200 ✅ --------- Co-authored-by: Mischa Spiegelmock <me@mish.dev> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat: add RLS group roles in Terraform (#979) ## Summary - Creates NOLOGIN group roles (`rls_bypass`, `rls_reader`, `model_access_all`) in Terraform - Adds `warehouse_full_access_rw_users` for users that bypass RLS entirely (granted `rls_bypass`) - Adds `warehouse_full_access_ro_users` for read-only users that see all models (granted `rls_reader` + `model_access_all`) - Grants `rls_reader` to regular `read_write_users` and all read-only users (subject to RLS policies) - Moves `inspect` from `read_write_users` to `full_access_rw_users` ## Context Stacked on #962 which refactored migrations to reference these role names instead of hardcoded usernames. ## Test plan - [ ] `tofu plan` shows role creation and grants - [ ] Roles are created before migrations run on fresh deploy - [ ] Existing users get correct role assignments 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Fix warehouse outputs indexing empty user lists (#993) ## Summary - `outputs.tf` referenced `var.read_write_users[0]` which fails when `inspect` is in `full_access_rw_users` instead - Use `local.all_rw_users[0]` and `local.all_ro_users[0]` which combine both regular and full-access users ## Context PR #979 introduced `full_access_rw_users` and moved `inspect` there, leaving `read_write_users` empty. The outputs weren't updated to use the combined locals. ## Test plan - [ ] `tofu plan` succeeds without "collection has no elements" error 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Prevent postgresql_role from revoking RLS grant_role memberships (#994) ## Overview The `roles` attribute on `postgresql_role` is authoritative — on each apply it reconciles to exactly the listed roles, revoking any others. This caused `rls_reader`, `rls_bypass`, and `model_access_all` grants (made by separate `postgresql_grant_role` resources) to be silently revoked whenever the role was modified in a subsequent Terraform apply. This was the root cause of `inspect_ro_risk_report` getting `permission denied for function user_has_model_access` after METR/mp4-deploy#604 was applied. You can see the permissions being added [here](https://metr-github.app.spacelift.io/stack/production-inspect/run/01KM46HN961KEQHYAB8XYMQAAY) and removed again in the [subsequent apply](https://metr-github.app.spacelift.io/stack/production-inspect/run/01KM4TECVP66E6VMGQD13QN7BG). ## Approach and Alternatives Consolidate all role memberships into a computed `local.user_roles` map, so each user's full set of group roles is managed via the authoritative `roles` attribute on `postgresql_role`. This eliminates the conflict between `roles` and separate `postgresql_grant_role` resources. An alternative would be to move everything to `postgresql_grant_role` resources and leave `roles` empty, but the provider may treat an unset `roles` as `[]` and still revoke externally-granted memberships. ## Testing & Validation - `tofu fmt -recursive` passes - Manual verification that `inspect_ro_risk_report` has correct permissions in production (applied directly as immediate fix) ## Checklist - [x] Code follows the project's style guidelines - [x] Self-review completed - [x] Comments added for complex or non-obvious code - [x] Uninformative LLM-generated comments removed - [ ] Documentation updated (if applicable) - [ ] Tests added or updated (if applicable) ## Additional Context The immediate issue was manually fixed by running `GRANT rls_reader TO inspect_ro_risk_report` directly in production. This PR prevents the problem from recurring on future applies. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat: enable row-level security on public tables (#990) ## Summary Activates RLS enforcement on all 8 public tables. The functions, roles, and policies were already created in #962 — this just flips the switch with `ALTER TABLE ... ENABLE ROW LEVEL SECURITY`. Stacked on #962. ## Test plan - [x] Alembic migration tests pass (apply, downgrade, upgrade cycle) - [ ] After deploy: connect as `inspect_ro_secret` → verify all data visible - [ ] After deploy: connect as `inspect_ro` → verify only public-model data visible 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Rasmus Faber-Espensen <rfaber@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix: prevent Terraform from revoking model_access_all group memberships (#995) ## Overview Hotfix for production RLS breakage caused by Spacelift revoking `model_access_all` group memberships on every apply. ## Problem The `roles` attribute on `postgresql_role` is authoritative. `model_access_all` is granted membership in model group roles (e.g. `model-access-fulltimer`, `shiba`, etc.) by Alembic migrations. Since Terraform doesn't list these in `roles`, every Spacelift apply revokes them, breaking RLS model access policies. This was the root cause of repeated `permission denied for function user_has_model_access` errors in production after merging #994. ## Fix Add `lifecycle { ignore_changes = [roles] }` to `model_access_all` so Terraform creates the role but doesn't manage its group memberships. ## Test plan - [x] Verified on staging that model group memberships persist after terraform apply - [x] `tofu fmt` passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * PLT-667: Add disk usage monitoring for k8s nodes (#607) ## Overview Add Datadog disk usage monitoring for k8s nodes. Enables the disk check on node agents and adds a monitor that alerts when any node exceeds 95% disk usage. https://us3.datadoghq.com/monitors/16533905 **Issue:** [PLT-667](https://linear.app/metrevals/issue/PLT-667/add-disk-usage-monitoring-for-k8s-nodes) ## Approach and Alternatives - Configure the disk check via `extraConfd.configDataMap` in the DatadogAgent CRD — this is the standard way to provide check configs through the Datadog Operator - Exclude virtual/pseudo filesystems (`autofs`, loop devices, `/dev/root`) that report 100% usage by design - Add a `node_disk_usage_high` Datadog monitor with 90% warning / 95% critical thresholds, grouped by host and device ## Testing & Validation - [x] Manual testing instructions: - Applied to staging, verified `system.disk.in_use` and `system.disk.used` metrics appear in Datadog - Confirmed staging nodes report ~17% disk usage - Verified production nodes report ~8-9% on real disks (`/dev/nvme*`) - Confirmed false positives from `/dev/root`, overlay, tmpfs, loop devices are excluded ## Checklist - [x] Self-review completed - [x] Tested in staging 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * lock: upgrade aws provider to 6.37.0 (#608) Cherry-picked from mp4-deploy. Only terraform/ (→ core/) portion; terraform_inspect/ has no monorepo equivalent. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Enable ECS event collection and deployment failure alerting (#567) Cherry-picked from mp4-deploy aa2a4d09. Manually applied due to context divergence in tfvars files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: regenerate hawk/uv.lock after sync Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: regenerate Lambda module uv.lock files after sync Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: regenerate middleman/uv.lock after sync Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: merge alembic heads after sync Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Import spec.model into Scan table (#883) ## Summary - Add `model`, `model_generate_config`, and `model_args` nullable columns to the `Scan` table, mirroring the existing `Eval` table pattern - Update the scan import writer to extract these fields from `ScanSpec.model` (a `ModelConfig | None`), using `canonical_model_name()` to strip provider prefixes - Include Alembic migration and two new integration tests (with/without model) ## Test plan - [x] `pytest tests/core/importer/scan/ -n auto -vv` — 47 tests pass - [x] `ruff check .` — clean - [x] `ruff format . --check` — clean - [x] `basedpyright .` — 0 errors, 0 warnings, 0 notes 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> * chore: cherry-pick missing spec.model commit and merge alembic heads The scan.model column was missed during sync #66 (conflict resolution dropped it). The RLS functions reference this column, causing migration failures on fresh databases. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [PLT-587] fix: Return 404 instead of 500 for missing scan records (#927) ## Summary - Fixes Sentry issue HAWK-3N0 where missing scan records returned 500 instead of 404 - Adds exception handler for KeyError from inspect_scout's `get_field()` function - Uses regex matching to only catch specific "not found" errors, not generic KeyErrors ## Context When a user tries to view a scan result that doesn't exist (e.g., stale UI state, bookmarked URL to deleted result), inspect_scout raises `KeyError("'uuid' not found in column")`. This was surfacing as a 500 Internal Server Error. The fix follows the same pattern as `eval_log_server.py` which converts `FileNotFoundError` to 404. ## Test plan - [x] Added tests for both matching (404) and non-matching (500) KeyError cases - [x] All existing tests pass - [x] ruff and basedpyright pass ## Links - Sentry: https://metr-sh.sentry.io/issues/HAWK-3N0 - Linear: https://linear.app/metrevals/issue/PLT-587 🤖 Generated with [Claude Code](https://claude.ai/code) Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * Update inspect-ai and k8s-sandbox for release (#881) - **inspect-ai**: Updated to METR fork commit `4bfe32e7a` (upstream/main at 0.3.179+47 with 4 METR patches: flat view toggle + retry-log enrichment) - **k8s-sandbox**: Updated pin from `metr-fixes` to `metr/combined-prs` (`067730c`) — includes in-cluster-config, websocket keepalive, skip-pod-restart-check - **Viewer**: Published `@metrevals/inspect-log-viewer@0.3.180-beta.20260214043004` All previous `combined_metr_fixes` patches (auth retry, http_client reopen, canonical model names, scanner changes, resolve-attachments, api_logs fix) have been **upstreamed** — that branch can be retired. Only 4 METR-specific patches remain: 1. `bcf1f15ec` — Flat view toggle for transcript viewer (Mischa Spiegelmock) 2. `e49deaa6a` — Enrich log_model_retry with sample context and error summary 3. `db8c51bf7` — SampleContextFilter for enriching SDK retry logs 4. `8ea8ec8bd` — Review fixes for retry-log (filter target, type safety, msg mutation) Moved from `metr-fixes` branch (which had accumulated reverts and stale content) to the cleaner `metr/combined-prs` branch. Several features from the old pin (compose extension, sampleUUID labels, network policy, devcontainer fixes) are now upstream. METR-only patches in the new pin: - feat: detect in-cluster config with kubeconfig fallback (PR #159) - Send WebSocket keepalive frames to prevent idle timeout (PR #156) - Add INSPECT_POD_RESTART_CHECK env var to reduce API server load - [ ] Smoke tests on staging - [ ] Verify eval-set submission works with new inspect-ai - [ ] Verify k8s sandbox creation with new k8s-sandbox * Merge Alembic migration heads (#889) ## Overview Fixes CI failure on #884 caused by multiple Alembic migration heads. ## Approach Two migrations both descended from `f3a4b5c6d7e8`, creating a fork: - `a1b2c3d4e5f7` — add model to scan - `e9f1a2b3c4d5` — add model_role type Added a merge migration (`8c6950acaca1`) that joins both heads, restoring a single linear migration history. ## Testing & validation - [x] `test_migrations_can_be_applied_from_scratch` — passes - [x] `test_migrations_can_be_downgraded_and_upgraded` — passes - [x] `test_migrations_are_up_to_date_with_models` — passes - [x] `test_no_missing_migrations` — passes - [x] `test_no_multiple_heads` — passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * chore: cherry-pick missing ModelRole.type and merge migrations Cherry-picks c4c25f4 (ModelRole.type column + migration) and 4b65d19 (merge migration) that were dropped during sync #13. Merges all alembic heads. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: make duplicate model_group migration a no-op Migration a3b4c5d6e7f8 (cherry-picked from inspect-action) creates the same middleman.model_group/model/model_config tables that c1d2e3f4a5b6 (monorepo original) already creates. Both share the same parent revision b2c3d4e5f6a8, so on a fresh database both run and the second one fails with "relation already exists". Make the cherry-picked version a no-op since the monorepo version handles table creation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: guard model_group migration against pre-existing tables When two migrations (c1d2e3f4a5b6 and a3b4c5d6e7f8) both create the same middleman tables from the same parent revision, the second one to run fails with "relation already exists". Add IF NOT EXISTS checks via information_schema to handle either execution order gracefully. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: Pulumi project rename to hawk and dev env fixes - Rename Pulumi project from metr-platform to hawk (matching PR #17) - Add stagingProject config override for StackReference to still find stg stack under old project name until it's migrated - Fix middleman-model-sync to use Python 3.13 via uvx --python flag Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: regenerate lock files after rebase and remove inspect-ai git source Remove inspect-ai git source override from pyproject.toml (monorepo uses PyPI pin inspect-ai==0.3.200). Regenerate all uv.lock files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: lint errors in merge migrations and test_model_group Remove unused imports from alembic merge migrations, fix import sorting, and fix undefined AsyncSession reference in test_model_group.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: type errors from ModelGroupsResult API change - Add type annotation to yaml.load() results in s3_files.py - Fix create_missing_model_files.py to extract group values from ModelGroupsResult instead of passing the result object directly - Resolve Pulumi.example.yaml merge conflicts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style: format migration and script files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: test failures from ModelGroupsResult API and migration ordering - Update test mocks to return ModelGroupsResult instead of plain sets (get_model_groups now returns a typed result object) - Add depends_on to RLS migration so middleman tables are created first - Revert model FK constraints to RESTRICT to match deployed migrations - Remove stagingProject config override (stg stack now under hawk project) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: update remaining test mocks for ModelGroupsResult and 5-tuple return - Fix test_create_scan_permissions parametrized mock to use ModelGroupsResult - Fix test_resume_scan mock to return 5-tuple from _validate_create_scan_permissions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove unused deploy-dev skill file Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: use bare raise to preserve original traceback in KeyError handler Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: align dashboard tag key with emitted metric tag The dashboard grouped by {eval_set_id} but the DogStatsD hook emits inspect_ai_job_id as the tag key. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Rafael <quantumlove42@gmail.com> Co-authored-by: rasmusfaber <rasmus.faber-espensen@metr.org> Co-authored-by: Rasmus Faber-Espensen <rfaber@gmail.com> Co-authored-by: Paarth Shah <paarth.shah@metr.org> Co-authored-by: Thomas Broadley <thomas@metr.org> Co-authored-by: Sami Jawhar <sami@metr.org>
PaarthShah
added a commit
to METR/hawk
that referenced
this pull request
Apr 3, 2026
* fix: rename hawk auth auth-login to hawk auth login (#974) ## Summary This PR fixes an awkward command naming that was introduced in commit b2efdc4 (PR #684). The auth login command was accidentally named `auth-login` within the auth group, creating the redundant command path `hawk auth auth-login`. This change: - Renames `hawk auth auth-login` to `hawk auth login` for consistency - Maintains the root-level `hawk login` command for backward compatibility - Both `hawk login` and `hawk auth login` now work as expected ## Context The issue was introduced on December 24, 2025 when the auth command group was added. The login command within the group was mistakenly given the name "auth-login" instead of just "login", resulting in the awkward `hawk auth auth-login` command. ## Testing & Validation - [x] All CLI tests pass (`uv run pytest tests/cli/ -n auto`) - [x] Both `hawk login` and `hawk auth login` commands work correctly - [x] Verified with `hawk --help` and `hawk auth --help` - [x] No errors or warnings from basedpyright ## Checklist - [x] Code follows the project's style guidelines (ruff check and format pass) - [x] Self-review completed - [x] Tests pass - [x] Documentation references to `hawk login` remain accurate * fix: add diagnostic logging for Okta token refresh failures (#981) ## Summary - Log the full HTTP response body from Okta when token refresh fails (both `refresh_token.py` and `credential_helper.py`) - Truncate error bodies to 500 chars to avoid huge log entries - Use specific exception types `(OSError, ValueError)` instead of bare `Exception` ## Context Investigating recurring `invalid_refresh_token` errors from Okta affecting eval sets. Previously, only the HTTP status code was logged on refresh failures — the Okta error body (which contains the specific error reason like `invalid_refresh_token`) was discarded. This made it difficult to diagnose the root cause. Related investigation: IAM PR #152 was closed — `refresh_token` is not a valid value for Okta policy rule `grantTypes.include`. ## Test plan - [ ] Deploy to staging - [ ] Trigger a token refresh failure and verify the Okta error body appears in logs - [ ] Verify normal refresh flow still works (no regressions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix(eval_log_reader): add s3:GetObject IAM permission for .models.json (#976) ## Summary - PR #972 added `.models.json` fallback logic to the eval_log_reader Lambda, but the Lambda's IAM role lacks `s3:GetObject` permission on the supporting S3 access point - The access point policy grants the permission on the resource side, but the Lambda role also needs an explicit IAM Allow - Production logs confirm: `"Could not read evals/.../.models.json: AccessDenied"` ## Root cause The S3 access point policy added in #972 grants `s3:GetObject` to the Lambda role for `evals/*/.models.json`, but S3 authorization requires the IAM principal to also have the permission in its own policy. The access point resource policy alone is not sufficient. ## Fix Add `s3:GetObject` to the Lambda role's `policy_statements` in `lambda.tf`, scoped to `evals/*/.models.json` on the supporting access point. ## Test plan - [ ] `tofu plan` shows only the IAM policy change (no other resource modifications) - [ ] Apply to production and verify artifact access works via `aws s3 cp` from the Object Lambda endpoint - [ ] Confirm Lambda logs show `"using .models.json from evals/..."` instead of `AccessDenied` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Remove fargate spot for API (#984) Causing some annoyance on staging --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Cherry pick fixes (#987) Cherry-pick METR/inspect_ai@cbdaa28 and METR/inspect_ai@4b7f380 * Pre-release option for publish version script * Lock all after running --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Hotfix for unclosed connector error silencing (#988) METR/inspect_ai@b47eb00 * security: bump joserfc >=1.6.3 (CVE-2026-27932) (#980) ## Summary Bumps joserfc minimum version from 1.0.4 to 1.6.3 to fix **CVE-2026-27932**. ## Why this is urgent **token-broker-lambda** is the only internet-facing Lambda — it has a public Function URL with `authorization_type = "NONE"`. External attackers can send crafted JWTs to the endpoint, which are passed directly to `joserfc.jwt.decode()` for validation. ## Changes | File | Change | |------|--------| | `terraform/modules/token_broker/pyproject.toml` | `joserfc>=1.0.4` → `joserfc>=1.6.3` | | `pyproject.toml` (api extra) | `joserfc>=1.0.4` → `joserfc>=1.6.3` | | `pyproject.toml` (cli extra) | `joserfc>=1.0.4` → `joserfc>=1.6.3` | ## Context - PR #968 in this repo attempted the same fix but was closed on March 9 - Companion PR: [METR/platform#136](METR/platform#136) - Identified during weekly security triage (2026-03-16) * chore: bump inspect-ai to 9e879d16 (viewer middle-click support) (#989) Bumps `inspect-ai` git pin from `b47eb00c` to `9e879d16` (hotfix HEAD). This brings in middle-click support to open tasks and samples in a new browser tab in the log viewer, cherry-picked from upstream [UKGovernmentBEIS/inspect_ai#3442](UKGovernmentBEIS/inspect_ai#3442) and merged into the `hotfix` branch via [METR/inspect_ai#24](METR/inspect_ai#24). - `pyproject.toml`: bump `inspect-ai` rev - `uv.lock` + all `terraform/modules/*/uv.lock`: regenerated * Fix e2e flakiness from minikube memory exhaustion (#992) ## Summary - Lower runner memory in `.env.local` from 16Gi to 512Mi (sufficient for e2e dummy/simple tasks) - Set explicit `--memory=4096` on `minikube start` for predictable behavior ## Problem Runner pods request 16Gi memory limits but minikube starts with default memory (~2-4GB). When multiple eval sets run concurrently in e2e tests, the second pod fails to schedule with `Insufficient memory`. ## Test plan - [ ] E2e tests pass consistently without "Insufficient memory" errors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Add Datadog metrics hook for rate limit visibility (#978) - Adds `hawk/runner/datadog_metrics.py` — a new inspect_ai hook that emits model usage metrics to DogStatsD - Wired into all three runner entry points (`run_eval_set`, `run_scan`, `run_scan_resume`) - Gated by `INSPECT_DATADOG_METRICS_ENABLED` env var - Updates CiliumNetworkPolicy to allow DogStatsD UDP egress to host:8125 - Includes a Datadog dashboard JSON for visualizing metrics | Metric | Type | Tags | |--------|------|------| | `inspect.model.tokens.input` | count | model, eval_set_id, task_name, run_id | | `inspect.model.tokens.output` | count | model, eval_set_id, task_name, run_id | | `inspect.model.tokens.total` | count | model, eval_set_id, task_name, run_id | | `inspect.model.call_duration` | histogram | model, eval_set_id, task_name, run_id | | `inspect.model.retries` | count | model, eval_set_id, task_name, run_id | | `inspect.eval_set.active` | gauge | eval_set_id | Runner pods send DogStatsD metrics via UDP to the Datadog agent's hostPort (8125) on the node. The CiliumNetworkPolicy needed a `toEntities: host` rule (not `world`) since the traffic goes to the node itself, not outside the cluster. Model names include provider prefixes (e.g. `openai/gpt-4`). The hook strips the prefix so only the model name appears in tags (`model:gpt-4`), avoiding exposure of provider-model associations in Datadog. - Requires [inspect_ai PR `ModelUsageData` eval context fields (`eval_set_id`, `run_id`, `eval_id`, `task_name`, `retries`) - No new Python dependencies — uses a minimal built-in UDP client for DogStatsD protocol | Var | Default | Description | |-----|---------|-------------| | `INSPECT_DATADOG_METRICS_ENABLED` | (unset) | Set to `1` or `true` to enable | | `DOGSTATSD_HOST` | `localhost` | DogStatsD agent host | | `DOGSTATSD_PORT` | `8125` | DogStatsD agent port | - [x] Run with `INSPECT_DATADOG_METRICS_ENABLED=true` and verify metrics appear in Datadog Metrics Explorer - [x] Run without the env var and verify no metrics are sent - [x] Verify provider prefix is stripped from model tags - [ ] Verified end-to-end on staging: `inspect.model.tokens.input/output/total` all appearing with `model:gpt-4o-mini` tag - [ ] Enable DogStatsD hostPort on production DatadogAgent CRD 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat: add RLS functions, roles, and policies for model group access control (#962) Adds the database infrastructure for row-level security (RLS) so read-only warehouse users will only see data for models they have access to. Builds on #951 (model group mapping). **RLS is not yet enabled** — SECURITY`. - Creates RLS policies on all 8 public tables with cascading logic: `eval` and `scan` check model access via `user_has_model_access()`; child tables (`sample`, `score`, `message`, `sample_model`, `scanner_result`) cascade via `EXISTS` against their parent; `model_role` checks model access directly to avoid circular recursion - `user_has_model_access(text[])` SQL function checks `middleman.model → model_group → pg_has_role(current_user, group_name)` - `get_eval_models()` / `get_scan_models()` — SECURITY DEFINER helpers that read `model_role` bypassing RLS (prevents circular recursion between eval/scan and model_role policies) - `sync_model_group_roles()` creates NOLOGIN PostgreSQL roles matching model group names (SECURITY DEFINER, with REVOKE EXECUTE FROM PUBLIC) - Migration creates `model-access-public` role explicitly (standard naming convention matching JWT claims and `.models.json`) - Bypass policies for `rls_bypass` role so `inspect` app user can bypass RLS (it does its own access control) - `import_model_configs.py` now syncs roles and grants after import, with role existence checks for dev environments - Migration SQL is inlined (not imported from app code) to ensure immutability across environments - **Terraform note:** `inspect_ro_secret` must be added to `warehouse_read_only_users` in tfvars files (gitignored, applied separately) **What this PR does NOT do:** - Does not enable RLS (`ALTER TABLE ... ENABLE ROW LEVEL SECURITY`) — see follow-up #990 **User roles (v1):** - `inspect_admin` — will bypass RLS (rds_superuser) - `inspect` — bypass policies ready, full read/write - `inspect_ro` — gets `model-access-public` role only - `inspect_ro_secret` — gets ALL model group roles (full researcher access) Linear: PLT-274, related PLT-345 - [x] 16 RLS tests in `tests/core/db/test_rls.py` covering: - Eval/scan with accessible model → visible - Eval/scan with inaccessible model → hidden - Child rows (sample/score/message/sample_model/scanner_result) of hidden parent → hidden - model_role of hidden eval → hidden; model_role of visible eval → visible - NULL model scan → visible - Unknown model → visible (not managed by middleman, treated as public) - Mixed model_roles requiring multiple groups → hidden when user lacks any group - Table owner bypasses RLS - sync_model_group_roles creates NOLOGIN roles and is idempotent - Public groups visible without explicit role grant - Model group without PostgreSQL role hides its models - [x] All DB tests pass (`pytest tests/core/db/ -n auto -vv`) - [x] Alembic migration tests pass (apply, downgrade, upgrade cycle) - [x] `ruff check`, `ruff format`, `basedpyright` — all clean - [ ] After deploy: verify functions and policies exist in database - [ ] After deploy: verify `sync_model_group_roles()` creates expected roles 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat(hawk): add cross-lab scan safeguard (#985) Prevents scanners from one AI lab from reading private model transcripts from a different lab (e.g. an Anthropic scanner cannot scan transcripts from a private OpenAI model). **Depends on**: [middleman update](metr-middleman/middleman-server#238) (Middleman `labs` field + platform/hawk port) must be deployed before this takes full effect. The implementation gracefully degrades when Middleman doesn't return `labs` yet (`labs={}` → check skipped with warning). The original implementation (closed PR #934) was blocked by the "qualified name problem": `.models.json` stores unqualified model names like `"gpt-4o"`, so `parse_model("gpt-4o").lab` returns `None` and the cross-lab check silently skipped every eval-set model. Ask Middleman at scan time instead of storing lab info at eval-set creation time. Middleman already knows each model's lab — we just needed it in the `/model_groups` response. Works for all existing eval sets with no data migration. **`hawk/api/auth/middleman_client.py`** - `ModelGroupsResult(groups, labs)` — new return type for `get_model_groups()` - Graceful fallback: `labs` field has `default_factory=dict`, handles old Middleman versions automatically **`hawk/api/scan_server.py`** - `_validate_cross_lab_scan()`: - Public models (`model-access-public`) always exempt - Lab comparison uses strict string equality — no normalization - Data issues (missing labs, unknown labs) → warning logged to Sentry, scan proceeds (fail-open) - Only actual cross-lab mismatches raise `CrossLabScanError` (403) - Collects all violations before raising - `allow_sensitive_cross_lab_scan` on both `CreateScanRequest` and `ResumeScanRequest` **`hawk/cli/`** - `--allow-sensitive-cross-lab-scan` flag on `scan run` and `scan resume` - Error hint pointing to the flag when a cross-lab error is returned **`hawk/api/problem.py`** - `CrossLabViolation` dataclass + `CrossLabScanError` (403) - [PLT-671](https://linear.app/metr/issue/PLT-671): Switch cross-lab data violations from fail-open (warnings) to fail-closed (errors) once we've validated the Middleman lab data in production - Unit tests in `tests/api/test_scan_server_unit.py` covering: same-lab allowed, cross-lab blocked, public exempt, bypass flag, no scanner models, old Middleman fallback, data issues warn not block, multiple violations, unknown scanner lab still compared - Integration tests in `tests/api/test_create_scan.py` updated for `ModelGroupsResult` return type - Manually tested on dev3: OpenAI scanner vs private gemini-pro → 403 ✅, same + bypass flag → 200 ✅ --------- Co-authored-by: Mischa Spiegelmock <me@mish.dev> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat: add RLS group roles in Terraform (#979) ## Summary - Creates NOLOGIN group roles (`rls_bypass`, `rls_reader`, `model_access_all`) in Terraform - Adds `warehouse_full_access_rw_users` for users that bypass RLS entirely (granted `rls_bypass`) - Adds `warehouse_full_access_ro_users` for read-only users that see all models (granted `rls_reader` + `model_access_all`) - Grants `rls_reader` to regular `read_write_users` and all read-only users (subject to RLS policies) - Moves `inspect` from `read_write_users` to `full_access_rw_users` ## Context Stacked on #962 which refactored migrations to reference these role names instead of hardcoded usernames. ## Test plan - [ ] `tofu plan` shows role creation and grants - [ ] Roles are created before migrations run on fresh deploy - [ ] Existing users get correct role assignments 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Fix warehouse outputs indexing empty user lists (#993) ## Summary - `outputs.tf` referenced `var.read_write_users[0]` which fails when `inspect` is in `full_access_rw_users` instead - Use `local.all_rw_users[0]` and `local.all_ro_users[0]` which combine both regular and full-access users ## Context PR #979 introduced `full_access_rw_users` and moved `inspect` there, leaving `read_write_users` empty. The outputs weren't updated to use the combined locals. ## Test plan - [ ] `tofu plan` succeeds without "collection has no elements" error 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Prevent postgresql_role from revoking RLS grant_role memberships (#994) ## Overview The `roles` attribute on `postgresql_role` is authoritative — on each apply it reconciles to exactly the listed roles, revoking any others. This caused `rls_reader`, `rls_bypass`, and `model_access_all` grants (made by separate `postgresql_grant_role` resources) to be silently revoked whenever the role was modified in a subsequent Terraform apply. This was the root cause of `inspect_ro_risk_report` getting `permission denied for function user_has_model_access` after METR/mp4-deploy#604 was applied. You can see the permissions being added [here](https://metr-github.app.spacelift.io/stack/production-inspect/run/01KM46HN961KEQHYAB8XYMQAAY) and removed again in the [subsequent apply](https://metr-github.app.spacelift.io/stack/production-inspect/run/01KM4TECVP66E6VMGQD13QN7BG). ## Approach and Alternatives Consolidate all role memberships into a computed `local.user_roles` map, so each user's full set of group roles is managed via the authoritative `roles` attribute on `postgresql_role`. This eliminates the conflict between `roles` and separate `postgresql_grant_role` resources. An alternative would be to move everything to `postgresql_grant_role` resources and leave `roles` empty, but the provider may treat an unset `roles` as `[]` and still revoke externally-granted memberships. ## Testing & Validation - `tofu fmt -recursive` passes - Manual verification that `inspect_ro_risk_report` has correct permissions in production (applied directly as immediate fix) ## Checklist - [x] Code follows the project's style guidelines - [x] Self-review completed - [x] Comments added for complex or non-obvious code - [x] Uninformative LLM-generated comments removed - [ ] Documentation updated (if applicable) - [ ] Tests added or updated (if applicable) ## Additional Context The immediate issue was manually fixed by running `GRANT rls_reader TO inspect_ro_risk_report` directly in production. This PR prevents the problem from recurring on future applies. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat: enable row-level security on public tables (#990) ## Summary Activates RLS enforcement on all 8 public tables. The functions, roles, and policies were already created in #962 — this just flips the switch with `ALTER TABLE ... ENABLE ROW LEVEL SECURITY`. Stacked on #962. ## Test plan - [x] Alembic migration tests pass (apply, downgrade, upgrade cycle) - [ ] After deploy: connect as `inspect_ro_secret` → verify all data visible - [ ] After deploy: connect as `inspect_ro` → verify only public-model data visible 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Rasmus Faber-Espensen <rfaber@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix: prevent Terraform from revoking model_access_all group memberships (#995) ## Overview Hotfix for production RLS breakage caused by Spacelift revoking `model_access_all` group memberships on every apply. ## Problem The `roles` attribute on `postgresql_role` is authoritative. `model_access_all` is granted membership in model group roles (e.g. `model-access-fulltimer`, `shiba`, etc.) by Alembic migrations. Since Terraform doesn't list these in `roles`, every Spacelift apply revokes them, breaking RLS model access policies. This was the root cause of repeated `permission denied for function user_has_model_access` errors in production after merging #994. ## Fix Add `lifecycle { ignore_changes = [roles] }` to `model_access_all` so Terraform creates the role but doesn't manage its group memberships. ## Test plan - [x] Verified on staging that model group memberships persist after terraform apply - [x] `tofu fmt` passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * PLT-667: Add disk usage monitoring for k8s nodes (#607) ## Overview Add Datadog disk usage monitoring for k8s nodes. Enables the disk check on node agents and adds a monitor that alerts when any node exceeds 95% disk usage. https://us3.datadoghq.com/monitors/16533905 **Issue:** [PLT-667](https://linear.app/metrevals/issue/PLT-667/add-disk-usage-monitoring-for-k8s-nodes) ## Approach and Alternatives - Configure the disk check via `extraConfd.configDataMap` in the DatadogAgent CRD — this is the standard way to provide check configs through the Datadog Operator - Exclude virtual/pseudo filesystems (`autofs`, loop devices, `/dev/root`) that report 100% usage by design - Add a `node_disk_usage_high` Datadog monitor with 90% warning / 95% critical thresholds, grouped by host and device ## Testing & Validation - [x] Manual testing instructions: - Applied to staging, verified `system.disk.in_use` and `system.disk.used` metrics appear in Datadog - Confirmed staging nodes report ~17% disk usage - Verified production nodes report ~8-9% on real disks (`/dev/nvme*`) - Confirmed false positives from `/dev/root`, overlay, tmpfs, loop devices are excluded ## Checklist - [x] Self-review completed - [x] Tested in staging 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * lock: upgrade aws provider to 6.37.0 (#608) Cherry-picked from mp4-deploy. Only terraform/ (→ core/) portion; terraform_inspect/ has no monorepo equivalent. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Enable ECS event collection and deployment failure alerting (#567) Cherry-picked from mp4-deploy aa2a4d09. Manually applied due to context divergence in tfvars files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: regenerate hawk/uv.lock after sync Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: regenerate Lambda module uv.lock files after sync Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: regenerate middleman/uv.lock after sync Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: merge alembic heads after sync Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Import spec.model into Scan table (#883) ## Summary - Add `model`, `model_generate_config`, and `model_args` nullable columns to the `Scan` table, mirroring the existing `Eval` table pattern - Update the scan import writer to extract these fields from `ScanSpec.model` (a `ModelConfig | None`), using `canonical_model_name()` to strip provider prefixes - Include Alembic migration and two new integration tests (with/without model) ## Test plan - [x] `pytest tests/core/importer/scan/ -n auto -vv` — 47 tests pass - [x] `ruff check .` — clean - [x] `ruff format . --check` — clean - [x] `basedpyright .` — 0 errors, 0 warnings, 0 notes 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> * chore: cherry-pick missing spec.model commit and merge alembic heads The scan.model column was missed during sync #66 (conflict resolution dropped it). The RLS functions reference this column, causing migration failures on fresh databases. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [PLT-587] fix: Return 404 instead of 500 for missing scan records (#927) ## Summary - Fixes Sentry issue HAWK-3N0 where missing scan records returned 500 instead of 404 - Adds exception handler for KeyError from inspect_scout's `get_field()` function - Uses regex matching to only catch specific "not found" errors, not generic KeyErrors ## Context When a user tries to view a scan result that doesn't exist (e.g., stale UI state, bookmarked URL to deleted result), inspect_scout raises `KeyError("'uuid' not found in column")`. This was surfacing as a 500 Internal Server Error. The fix follows the same pattern as `eval_log_server.py` which converts `FileNotFoundError` to 404. ## Test plan - [x] Added tests for both matching (404) and non-matching (500) KeyError cases - [x] All existing tests pass - [x] ruff and basedpyright pass ## Links - Sentry: https://metr-sh.sentry.io/issues/HAWK-3N0 - Linear: https://linear.app/metrevals/issue/PLT-587 🤖 Generated with [Claude Code](https://claude.ai/code) Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * Update inspect-ai and k8s-sandbox for release (#881) - **inspect-ai**: Updated to METR fork commit `4bfe32e7a` (upstream/main at 0.3.179+47 with 4 METR patches: flat view toggle + retry-log enrichment) - **k8s-sandbox**: Updated pin from `metr-fixes` to `metr/combined-prs` (`067730c`) — includes in-cluster-config, websocket keepalive, skip-pod-restart-check - **Viewer**: Published `@metrevals/inspect-log-viewer@0.3.180-beta.20260214043004` All previous `combined_metr_fixes` patches (auth retry, http_client reopen, canonical model names, scanner changes, resolve-attachments, api_logs fix) have been **upstreamed** — that branch can be retired. Only 4 METR-specific patches remain: 1. `bcf1f15ec` — Flat view toggle for transcript viewer (Mischa Spiegelmock) 2. `e49deaa6a` — Enrich log_model_retry with sample context and error summary 3. `db8c51bf7` — SampleContextFilter for enriching SDK retry logs 4. `8ea8ec8bd` — Review fixes for retry-log (filter target, type safety, msg mutation) Moved from `metr-fixes` branch (which had accumulated reverts and stale content) to the cleaner `metr/combined-prs` branch. Several features from the old pin (compose extension, sampleUUID labels, network policy, devcontainer fixes) are now upstream. METR-only patches in the new pin: - feat: detect in-cluster config with kubeconfig fallback (PR #159) - Send WebSocket keepalive frames to prevent idle timeout (PR #156) - Add INSPECT_POD_RESTART_CHECK env var to reduce API server load - [ ] Smoke tests on staging - [ ] Verify eval-set submission works with new inspect-ai - [ ] Verify k8s sandbox creation with new k8s-sandbox * Merge Alembic migration heads (#889) ## Overview Fixes CI failure on #884 caused by multiple Alembic migration heads. ## Approach Two migrations both descended from `f3a4b5c6d7e8`, creating a fork: - `a1b2c3d4e5f7` — add model to scan - `e9f1a2b3c4d5` — add model_role type Added a merge migration (`8c6950acaca1`) that joins both heads, restoring a single linear migration history. ## Testing & validation - [x] `test_migrations_can_be_applied_from_scratch` — passes - [x] `test_migrations_can_be_downgraded_and_upgraded` — passes - [x] `test_migrations_are_up_to_date_with_models` — passes - [x] `test_no_missing_migrations` — passes - [x] `test_no_multiple_heads` — passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * chore: cherry-pick missing ModelRole.type and merge migrations Cherry-picks c4c25f4 (ModelRole.type column + migration) and 4b65d19 (merge migration) that were dropped during sync #13. Merges all alembic heads. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: make duplicate model_group migration a no-op Migration a3b4c5d6e7f8 (cherry-picked from inspect-action) creates the same middleman.model_group/model/model_config tables that c1d2e3f4a5b6 (monorepo original) already creates. Both share the same parent revision b2c3d4e5f6a8, so on a fresh database both run and the second one fails with "relation already exists". Make the cherry-picked version a no-op since the monorepo version handles table creation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: guard model_group migration against pre-existing tables When two migrations (c1d2e3f4a5b6 and a3b4c5d6e7f8) both create the same middleman tables from the same parent revision, the second one to run fails with "relation already exists". Add IF NOT EXISTS checks via information_schema to handle either execution order gracefully. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: Pulumi project rename to hawk and dev env fixes - Rename Pulumi project from metr-platform to hawk (matching PR #17) - Add stagingProject config override for StackReference to still find stg stack under old project name until it's migrated - Fix middleman-model-sync to use Python 3.13 via uvx --python flag Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: regenerate lock files after rebase and remove inspect-ai git source Remove inspect-ai git source override from pyproject.toml (monorepo uses PyPI pin inspect-ai==0.3.200). Regenerate all uv.lock files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: lint errors in merge migrations and test_model_group Remove unused imports from alembic merge migrations, fix import sorting, and fix undefined AsyncSession reference in test_model_group.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: type errors from ModelGroupsResult API change - Add type annotation to yaml.load() results in s3_files.py - Fix create_missing_model_files.py to extract group values from ModelGroupsResult instead of passing the result object directly - Resolve Pulumi.example.yaml merge conflicts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style: format migration and script files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: test failures from ModelGroupsResult API and migration ordering - Update test mocks to return ModelGroupsResult instead of plain sets (get_model_groups now returns a typed result object) - Add depends_on to RLS migration so middleman tables are created first - Revert model FK constraints to RESTRICT to match deployed migrations - Remove stagingProject config override (stg stack now under hawk project) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: update remaining test mocks for ModelGroupsResult and 5-tuple return - Fix test_create_scan_permissions parametrized mock to use ModelGroupsResult - Fix test_resume_scan mock to return 5-tuple from _validate_create_scan_permissions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove unused deploy-dev skill file Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: use bare raise to preserve original traceback in KeyError handler Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: align dashboard tag key with emitted metric tag The dashboard grouped by {eval_set_id} but the DogStatsD hook emits inspect_ai_job_id as the tag key. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Rafael <quantumlove42@gmail.com> Co-authored-by: rasmusfaber <rasmus.faber-espensen@metr.org> Co-authored-by: Rasmus Faber-Espensen <rfaber@gmail.com> Co-authored-by: Paarth Shah <paarth.shah@metr.org> Co-authored-by: Thomas Broadley <thomas@metr.org> Co-authored-by: Sami Jawhar <sami@metr.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cherry-pick of UKGovernmentBEIS/inspect_ai#3442 (merged upstream
fd12f2c4) into thehotfixbranch.Adds middle-click support to open tasks/samples in a new browser tab in the log viewer.
Changes
onCellMouseDownhandler inLogListGrid,SamplesGrid, andSampleListfor middle-click (button === 1)window.open(url, "_blank")Context
METR/inspect-actioncurrently pinsinspect-aito commitbb99d9f6on this branch. Once merged, update the pin in inspect-action'spyproject.tomlto include this feature.