Policy: document and observe callers who aren't signed in

[habdelra]

In service of CS-13521, which lets a realm's policy admit a caller who isn't signed in. The reads, search and writes changes switched that on. This PR gives operators what they need to watch it and act on it. The reference for card and policy authors lives on CS-13581 rather than in the repo.

It has no merge-order dependency on the boxel-skills chain. It is stacked on #6577 only because it describes what that PR ships, and merges after it.

• .claude/skills/operations-diagnosis/SKILL.md gains a section on the anonymous-request record, including its fields, two plain-text warnings, the traps that lead to wrong conclusions, LogQL recipes, and four investigations:

  • a form that says "Too many requests";
  • spam to trace and block;
  • a form whose submissions are refused;
  • a public page that stopped showing cards.

  The channel table, the correlation-id trap and the dashboard table follow.

• The Policy Decisions dashboard (policy-decisions.json) gains an "Anonymous callers" row:

  • outcomes over time;
  • rate-limited and blocked stats;
  • blocked by reason;
  • top addresses by outcome;
  • writes by acting user, beside refusals by acting-user failure.

  Array fields are extracted with | json name="field[0]", since Loki's JSON parser skips arrays.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-06T18:15:23.296030Z	f2909f8	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

Observability diff (vs staging)

Show diff

diff --git a/tmp/remote-canon.gourOM/dashboards/boxel-status/policy-decisions.json b/tmp/committed-canon.j4uWyu/dashboards/boxel-status/policy-decisions.json
index d954eb2..c0d5599 100644
--- a/tmp/remote-canon.gourOM/dashboards/boxel-status/policy-decisions.json
+++ b/tmp/committed-canon.j4uWyu/dashboards/boxel-status/policy-decisions.json
@@ -9,20 +9,7 @@
   },
   "spec": {
     "annotations": {
-      "list": [
-        {
-          "builtIn": 1,
-          "datasource": {
-            "type": "grafana",
-            "uid": "-- Grafana --"
-          },
-          "enable": true,
-          "hide": true,
-          "iconColor": "rgba(0, 211, 255, 1)",
-          "name": "Annotations & Alerts",
-          "type": "dashboard"
-        }
-      ]
+      "list": []
     },
     "description": "Operations-permission policy decisions, search-lane scoping, snapshot reads, compiles and their costs, from the boxel:operations channel. See the operations-diagnosis skill for how to read it.",
     "editable": true,
@@ -1437,10 +1424,426 @@
         ],
         "title": "Compile duration p95",
         "type": "timeseries"
+      },
+      {
+        "collapsed": false,
+        "gridPos": {
+          "h": 1,
+          "w": 24,
+          "x": 0,
+          "y": 67
+        },
+        "id": 26,
+        "panels": [],
+        "title": "Anonymous callers",
+        "type": "row"
+      },
+      {
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "description": "Records of requests from callers who aren't signed in that a realm's anonymous admission refused, blocked, rate-limited or counted, by realm (one per realm for a federated search): admitted (counted), infra (the platform's own services, never counted), refused (no grant admitted it), blocked (the realm refused the address), rate-limited (the address used up its budget), unavailable (the count couldn't be read). A realm whose policy opens nothing to such callers records nothing.",
+        "fieldConfig": {
+          "defaults": {
+            "color": {
+              "mode": "palette-classic"
+            },
+            "custom": {
+              "drawStyle": "line",
+              "fillOpacity": 60,
+              "lineWidth": 1,
+              "showPoints": "never",
+              "spanNulls": false,
+              "stacking": {
+                "group": "A",
+                "mode": "normal"
+              }
+            },
+            "unit": "short"
+          },
+          "overrides": []
+        },
+        "gridPos": {
+          "h": 8,
+          "w": 12,
+          "x": 0,
+          "y": 68
+        },
+        "id": 27,
+        "interval": "1m",
+        "options": {
+          "legend": {
+            "calcs": [
+              "sum"
+            ],
+            "displayMode": "table",
+            "placement": "bottom",
+            "showLegend": true
+          },
+          "tooltip": {
+            "mode": "multi",
+            "sort": "desc"
+          }
+        },
+        "pluginVersion": "12.4.3",
+        "targets": [
+          {
+            "datasource": {
+              "type": "loki",
+              "uid": "loki"
+            },
+            "editorMode": "code",
+            "expr": "sum by (realmURL, outcome) (count_over_time({service=\"realm-server\", env=\"$env\"} |= \"boxel:operations\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:operations\" | kind=\"anonymous-request\" | realmURL=~\".*${realm}.*\" [$__interval]))",
+            "legendFormat": "{{outcome}} {{realmURL}}",
+            "queryType": "range",
+            "refId": "A"
+          }
+        ],
+        "title": "Anonymous outcomes",
+        "type": "timeseries"
+      },
+      {
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "description": "Requests from callers who aren't signed in answered rate-limited in the last 5 minutes.",
+        "fieldConfig": {
+          "defaults": {
+            "color": {
+              "mode": "thresholds"
+            },
+            "mappings": [],
+            "thresholds": {
+              "mode": "absolute",
+              "steps": [
+                {
+                  "color": "green"
+                },
+                {
+                  "color": "yellow",
+                  "value": 50
+                }
+              ]
+            },
+            "unit": "short"
+          },
+          "overrides": []
+        },
+        "gridPos": {
+          "h": 4,
+          "w": 6,
+          "x": 12,
+          "y": 68
+        },
+        "id": 28,
+        "interval": "1m",
+        "options": {
+          "colorMode": "value",
+          "graphMode": "area",
+          "justifyMode": "center",
+          "orientation": "auto",
+          "reduceOptions": {
+            "calcs": [
+              "lastNotNull"
+            ],
+            "fields": "",
+            "values": false
+          },
+          "textMode": "auto"
+        },
+        "pluginVersion": "12.4.3",
+        "targets": [
+          {
+            "datasource": {
+              "type": "loki",
+              "uid": "loki"
+            },
+            "editorMode": "code",
+            "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} |= \"boxel:operations\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:operations\" | kind=\"anonymous-request\" | realmURL=~\".*${realm}.*\" | outcome=\"rate-limited\" [5m])) or vector(0)",
+            "legendFormat": "",
+            "queryType": "range",
+            "refId": "A"
+          }
+        ],
+        "title": "Rate-limited (5m)",
+        "type": "stat"
+      },
+      {
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "description": "Requests from callers who aren't signed in answered blocked in the last 5 minutes.",
+        "fieldConfig": {
+          "defaults": {
+            "color": {
+              "mode": "thresholds"
+            },
+            "mappings": [],
+            "thresholds": {
+              "mode": "absolute",
+              "steps": [
+                {
+                  "color": "green"
+                },
+                {
+                  "color": "yellow",
+                  "value": 50
+                }
+              ]
+            },
+            "unit": "short"
+          },
+          "overrides": []
+        },
+        "gridPos": {
+          "h": 4,
+          "w": 6,
+          "x": 18,
+          "y": 68
+        },
+        "id": 29,
+        "interval": "1m",
+        "options": {
+          "colorMode": "value",
+          "graphMode": "area",
+          "justifyMode": "center",
+          "orientation": "auto",
+          "reduceOptions": {
+            "calcs": [
+              "lastNotNull"
+            ],
+            "fields": "",
+            "values": false
+          },
+          "textMode": "auto"
+        },
+        "pluginVersion": "12.4.3",
+        "targets": [
+          {
+            "datasource": {
+              "type": "loki",
+              "uid": "loki"
+            },
+            "editorMode": "code",
+            "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} |= \"boxel:operations\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:operations\" | kind=\"anonymous-request\" | realmURL=~\".*${realm}.*\" | outcome=\"blocked\" [5m])) or vector(0)",
+            "legendFormat": "",
+            "queryType": "range",
+            "refId": "A"
+          }
+        ],
+        "title": "Blocked (5m)",
+        "type": "stat"
+      },
+      {
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "description": "blocklist: the realm's anonymousBlocklist names the address. ip-undetermined: no address reached the realm, usually a hop misconfiguration. blocklist-invalid: an entry isn't an address or range, which closes the realm to every such caller.",
+        "fieldConfig": {
+          "defaults": {
+            "custom": {
+              "align": "auto"
+            }
+          },
+          "overrides": []
+        },
+        "gridPos": {
+          "h": 4,
+          "w": 12,
+          "x": 12,
+          "y": 72
+        },
+        "id": 30,
+        "options": {
+          "showHeader": true,
+          "sortBy": [
+            {
+              "desc": true,
+              "displayName": "Value"
+            }
+          ]
+        },
+        "pluginVersion": "12.4.3",
+        "targets": [
+          {
+            "datasource": {
+              "type": "loki",
+              "uid": "loki"
+            },
+            "editorMode": "code",
+            "expr": "sum by (realmURL, blockReason) (count_over_time({service=\"realm-server\", env=\"$env\"} |= \"boxel:operations\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:operations\" | kind=\"anonymous-request\" | realmURL=~\".*${realm}.*\" | outcome=\"blocked\" [$__range]))",
+            "instant": true,
+            "legendFormat": "",
+            "queryType": "instant",
+            "refId": "A"
+          }
+        ],
+        "title": "Blocked by reason",
+        "transformations": [
+          {
+            "id": "labelsToFields",
+            "options": {
+              "mode": "columns"
+            }
+          },
+          {
+            "id": "organize",
+            "options": {
+              "excludeByName": {
+                "Time": true
+              }
+            }
+          }
+        ],
+        "type": "table"
+      },
+      {
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "description": "The addresses behind admissions, refusals and rate limits over the selected range, per realm: who to look at before blocking, and whether many visitors share one address (a hop misconfiguration). rateLimitKey is what the budget is keyed on: the address, or an IPv6 address's /64.",
+        "fieldConfig": {
+          "defaults": {
+            "custom": {
+              "align": "auto"
+            }
+          },
+          "overrides": []
+        },
+        "gridPos": {
+          "h": 8,
+          "w": 12,
+          "x": 0,
+          "y": 76
+        },
+        "id": 31,
+        "options": {
+          "showHeader": true,
+          "sortBy": [
+            {
+              "desc": true,
+              "displayName": "Value"
+            }
+          ]
+        },
+        "pluginVersion": "12.4.3",
+        "targets": [
+          {
+            "datasource": {
+              "type": "loki",
+              "uid": "loki"
+            },
+            "editorMode": "code",
+            "expr": "topk(25, sum by (realmURL, rateLimitKey, clientIP, outcome) (count_over_time({service=\"realm-server\", env=\"$env\"} |= \"boxel:operations\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:operations\" | kind=\"anonymous-request\" | realmURL=~\".*${realm}.*\" | outcome=~\"admitted|refused|rate-limited|blocked\" [$__range])))",
+            "instant": true,
+            "legendFormat": "",
+            "queryType": "instant",
+            "refId": "A"
+          }
+        ],
+        "title": "Top addresses by outcome",
+        "transformations": [
+          {
+            "id": "labelsToFields",
+            "options": {
+              "mode": "columns"
+            }
+          },
+          {
+            "id": "organize",
+            "options": {
+              "excludeByName": {
+                "Time": true
+              }
+            }
+          }
+        ],
+        "type": "table"
+      },
+      {
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "description": "A: admitted writes by the first user they were made as. B: refused writes whose grants named no one who may write the realm, by the failure (key-missing, not-a-matrix-id, no-write).",
+        "fieldConfig": {
+          "defaults": {
+            "custom": {
+              "align": "auto"
+            }
+          },
+          "overrides": []
+        },
+        "gridPos": {
+          "h": 8,
+          "w": 12,
+          "x": 12,
+          "y": 76
+        },
+        "id": 32,
+        "options": {
+          "showHeader": true,
+          "sortBy": [
+            {
+              "desc": true,
+              "displayName": "Value"
+            }
+          ]
+        },
+        "pluginVersion": "12.4.3",
+        "targets": [
+          {
+            "datasource": {
+              "type": "loki",
+              "uid": "loki"
+            },
+            "editorMode": "code",
+            "expr": "sum by (realmURL, actingUser) (count_over_time({service=\"realm-server\", env=\"$env\"} |= \"boxel:operations\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:operations\" | kind=\"anonymous-request\" | realmURL=~\".*${realm}.*\" | outcome=\"admitted\" | json actingUser=\"actingUsers[0]\" | actingUser!=\"\" [$__range]))",
+            "instant": true,
+            "legendFormat": "",
+            "queryType": "instant",
+            "refId": "A"
+          },
+          {
+            "datasource": {
+              "type": "loki",
+              "uid": "loki"
+            },
+            "editorMode": "code",
+            "expr": "sum by (realmURL, failure) (count_over_time({service=\"realm-server\", env=\"$env\"} |= \"boxel:operations\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:operations\" | kind=\"anonymous-request\" | realmURL=~\".*${realm}.*\" | outcome=\"refused\" | json failure=\"actingUserFailures[0].failure\" | failure!=\"\" [$__range]))",
+            "instant": true,
+            "legendFormat": "",
+            "queryType": "instant",
+            "refId": "B"
+          }
+        ],
+        "title": "Anonymous writes by acting user",
+        "transformations": [
+          {
+            "id": "labelsToFields",
+            "options": {
+              "mode": "columns"
+            }
+          },
+          {
+            "id": "organize",
+            "options": {
+              "excludeByName": {
+                "Time": true
+              }
+            }
+          }
+        ],
+        "type": "table"
       }
     ],
     "refresh": "1m",
-    "schemaVersion": 42,
+    "schemaVersion": 41,
     "tags": [
       "policy",
       "permissions",
@@ -1451,7 +1854,7 @@
         {
           "hide": 2,
           "name": "env",
-          "query": "staging",
+          "query": "__ENV__",
           "skipUrlSync": true,
           "type": "constant"
         },

(Run: https://github.com/cardstack/boxel/actions/runs/37537691108)

[habdelra]

[Claude Code 🤖] Reviewed the three files against the code at f2909f8. Checked: #anonymousCallerFor, #checkAnonymousBudget, #chargeAnonymous, #countAnonymous, #chargeCaller, #anonymousCommit, #recordAnonymous and #capabilityLanes; the ANONYMOUS_* dispatch table and the route registrations; anonymous-access.ts (300/60 default, malformed-limit fallback, IPv6 /64 key, mapped IPv4); acting-users.ts; the policy compile issue codes and the eligible-operation list; explain.ts; client-address.ts; multi-realm-authorization.ts (fan-out cap of 2, archived realms skipped, the "could not say" warning); handle-search.ts per-realm counting; and main.ts's BOXEL_ANONYMOUS_RATE_LIMIT handling.

What's right:

• The doc's contract matches the code: eligible operations, the three issue codes, read-at-decide acting users, what counts and when, 429/503 with Retry-After and meta.retryAfterSeconds, blocklist fail-closed, infra exemption, the federated cap, and explain with an empty actor.
• The dashboard JSON parses. Panel ids 26–32 are unique and the grid doesn't overlap.
• Datasource, firelens unwrap, range vs instant and the table transformations match the existing panels.
• The array extraction (| json x="arr[0]", arr[0].field) is valid jsonexpr.
• Example addresses are RFC 5737/3849, with no ticket ids or journey language.

Inline:

• Two medium operator traps: ip-undetermined attributed to an unset hop count, and the spam recipe's operation="create" missing /_operations writes.
• A few low-severity precision fixes.

[habdelra]

[Claude Code 🤖] ip-undetermined can't come from BOXEL_TRUSTED_PROXY_HOPS being unset. Unset means zero hops, and callerAddress in middleware/client-address.ts then uses the socket peer, which is always present. Behind the load balancer, that gives every visitor the load balancer's address. That is the shared-clientIP symptom in the next sentence, not ip-undetermined. ip-undetermined comes from a hop count larger than the number of X-Forwarded-For entries (for example a request that skipped the load balancer), or from an entry that isn't an address. Could this say "set higher than the proxies actually in front", and attach "unset" to the shared-address symptom instead?

[habdelra]

[Claude Code 🤖] Fixed in 789713067d. The trap now separates the two symptoms. ip-undetermined means BOXEL_TRUSTED_PROXY_HOPS is set higher than the proxies in front of the server. Many visitors sharing the load balancer's address means it is unset, since the socket peer is used then. The field description now says when ip-undetermined arises.

[habdelra]

[Claude Code 🤖] operation="create" misses form posts that go through /_operations. ANONYMOUS_OPERATIONS (and ANONYMOUS_CAPABILITY_CHECK) has operations: [], so #anonymousCallerFor records operation as [...opened][0]. That is the first operation the policy opens to such callers (often read), not what the batch wrote. An admitted write always carries actingUsers (#anonymousCommit passes it to #chargeCaller). Could step 1 select on that instead, with | json actingUser="actingUsers[0]" | actingUser!="" as the recipe already does, or add route=~"POST .*"? The same applies to the field description on line 322: operation is "what the route was admitted to" only for single-operation routes. For /_operations and /_capabilities it is an arbitrary opened operation, which is worth saying.

[habdelra]

[Claude Code 🤖] Fixed in 789713067d: the spam investigation selects on the acting user instead of operation. Separately, fixed in 567d707e2e on the reads branch and merged up through the stack: the record names a route that runs whatever its request names (the envelope and the capability check) as operation: "*", rather than the first operation the policy happened to open. The field description says so.

[habdelra]

[Claude Code 🤖] Small one: clientIP isn't raw. It's the canonical formatIP form, so an IPv4-mapped IPv6 address becomes a dotted quad and IPv6 is RFC 5952-compressed. Could it say "(canonical, not reduced to /64)"? An operator matching it against ALB or WAF logs needs to know that.

[habdelra]

[Claude Code 🤖] Fixed in 789713067d: clientIP is described as canonical, not reduced to its /64, with an IPv4-mapped address reading as a dotted quad.

[habdelra]

[Claude Code 🤖] correlationId is what the request carried in x-boxel-logging-correlation-id, which the client mints (middleware/index.ts). The host sends one. A bot, a curl spammer or a third-party form page won't, so for the spam and "Too many requests" investigations it will mostly be null. Could this say so and point at clientIP + route + timestamp as the join when it is absent? Also, this trap's heading (line 62) and the frontmatter description ("no record carries a correlation id") now contradict the exception. Could both say "no policy record"?

[habdelra]

[Claude Code 🤖] Fixed in 789713067d. correlationId is described as what the request carried, which the host sends and bots or third-party pages usually don't, with clientIP + route + timestamp as the join when it is null. The trap heading and the frontmatter now say "no policy record".

[habdelra]

[Claude Code 🤖] The budget is keyed on rateLimitKey, not clientIP (#chargeCaller charges caller.rateLimitKey). For IPv6 visitors, counting per clientIP splits one /64's budget across several rows, so no single address looks like it spent the budget. Could step 2 group by rateLimitKey? The "who is being rate-limited" recipe and the Top addresses by outcome panel would read better with rateLimitKey beside clientIP too.

[habdelra]

[Claude Code 🤖] Fixed in 789713067d. The recipes and the "Too many requests" investigation group by rateLimitKey, beside clientIP where the address matters. The Top addresses panel groups by both.

[habdelra]

[Claude Code 🤖] The Anonymous outcomes panel's query is sum by (outcome), so it isn't per realm. The Realm textbox narrows it, but it never splits by realm. Either add realmURL to the by clause and legend in policy-decisions.json (the recipe on line 81 already does), or drop "per realm" here.

[habdelra]

[Claude Code 🤖] Fixed in 789713067d: the Anonymous outcomes panel groups by realmURL, outcome, with both in the legend, and the skill's table says "by realm and outcome".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f2909f8ba0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve the realm in anonymous outcome series

When the default Realm filter is empty—or matches multiple realms—this aggregation groups only by outcome, so traffic from every realm is merged into indistinguishable series. The accompanying operations guide describes this panel as reporting outcomes “per realm,” and losing realmURL prevents operators from identifying which realm is being blocked or rate-limited; include realmURL in the grouping and legend.

Useful? React with 👍 / 👎.

[habdelra]

[Claude Code 🤖] Fixed in 789713067d: sum by (realmURL, outcome) with {{outcome}} {{realmURL}} as the legend.

[chatgpt-codex-connector]

Qualify the one-record-per-request guarantee

Requests that reach anonymous admission do not always emit this record: for example, the admitted .json spelling can return a 302 redirect with served still false, and #chargeAnonymous emits nothing for that response (the anonymous-read test explicitly expects the redirect to produce no record). As written, this guarantee—and the identical dashboard description—can make operators treat a missing line as evidence that admission was never reached; document the unrecorded outcomes or emit a record for them.

Useful? React with 👍 / 👎.

[habdelra]

[Claude Code 🤖] Fixed in 789713067d. The record description no longer promises one line per request. It lists what does leave a line (every refusal, block, rate limit, count failure and counted request) and what doesn't: a .json redirect, a stylesheet, a capability check on a realm anyone may read, and a search no grant scoped. The panel description says the same.

[chatgpt-codex-connector]

Document refused HEAD probes without claiming a 401

For a HEAD request whose card is missing or denied, #chargeAnonymous deliberately emits a refused record because the request was not served, but headCard returns the 200 realm-identity discovery response rather than a 401. Operators following this definition (and the later claim that every refused caller saw 401) will fail to correlate these telemetry records with the HTTP request, so describe the HEAD discovery exception.

Useful? React with 👍 / 👎.

[habdelra]

[Claude Code 🤖] Fixed in 789713067d: refused now says the caller got the 401, or, for a HEAD, the realm's 200 discovery answer.

[lukemelia]

Approved but would like to see us support named operations for anonymous users in the future. If anything, mutating named operations are more valuable than base update or transform.

[habdelra]

[Claude Code 🤖] Thanks, filed as CS-13605: a grant on a named operation opting in to callers who aren't signed in. Its open question is how a program's nested writes are judged, as the visitor or as the grant's acting user, since actor() stays empty for a visitor.