diff --git a/src/content/docs/guards/capture.mdx b/src/content/docs/guards/capture.mdx index ae11da5a..c4ccebfe 100644 --- a/src/content/docs/guards/capture.mdx +++ b/src/content/docs/guards/capture.mdx @@ -176,6 +176,84 @@ export default { | `occurredAt` / `occurred_at` | Optional timestamp. Defaults to the time of the call. Pre-epoch values cannot be represented and are dropped. | | `waitUntil` | JavaScript only. Platform hook that keeps the invocation alive until the batch is sent. | +## Capture outcomes + +Helpers that wrap an action record `metadata.outcome` on the capture event +they emit. The value is capture and Sequence metadata. It never changes the +decision. `conclusion` remains `ALLOW` or `DENY`. The decision has no +outcome field. + +You don't set `outcome`. The helper writes it after your metadata so a +caller key can't overwrite what the helper recorded. + + + + +JavaScript `guardTool` and `guardAction` record one of the following +values: + +| Value | The action | What it means | +| ----- | ---------- | ------------- | +| `success` | Ran. | The helper executed the action. | +| `error` | Ran, then threw. | The wrapped action threw. | +| `denied` | Did not run. | Policy evaluated `DENY`. | +| `unavailable` | Did not run. | Fail-closed. The check was incomplete. | + +They don't record `degraded`. When `onGuardError` is `"allow"` and the +action runs without a complete judgment, the event records `success`. + + + + +Python checkpoint helpers record one of the following values: + +| Value | The action | What it means | +| ----- | ---------- | ------------- | +| `success` | Ran. | Policy judged all of it. | +| `degraded` | Ran. `on_guard_error` is `allow`. | Policy judged it in part or not at all. | +| `error` | Ran, then threw. | The wrapped action threw. | +| `denied` | Did not run. | Policy evaluated `DENY`. | +| `unavailable` | Did not run. | Fail-closed. The check was incomplete. | + +`on_guard_error` is `"allow"` or `"deny"`. The default is `"deny"`: an +incomplete check blocks the action and records `unavailable`. If the +action runs and then throws, the event records `error`, even when the +judgment was incomplete. + +A `degraded` event uses `decision_id` to tell the two incomplete judgments +apart: + +- Present: policy judged the action in part. +- Absent: policy judged none of it. The helper had no decision, a + failed-open decision with an empty id, or an answer it could not read. + +An unreadable decision isn't passed into capture. The event emits +without `decision_id`. + +The following helpers go through the checkpoint engine, so they record +these outcomes: + +- `guard_action` / `guard_action_sync` +- LangChain `guard_tool` and `ArcjetMiddleware` +- CrewAI `guard_tool` + +`register_arcjet_hooks` records `success` when the action proceeds. It +doesn't record `degraded`. + +A direct `capture()` call doesn't set `outcome`. Observe-only +`ArcjetCaptureHandler` and `ArcjetAsyncCaptureHandler` don't go through +the checkpoint engine. + + + + +The Go SDK doesn't write `metadata.outcome` on `Capture`. Use `Capture` +to record that an allowed action happened. Outcome classification is not +part of the Go helper surface. + + + + ## Call `capture()` without a client handle Passing the client explicitly is the recommended path. When `capture()` is @@ -186,6 +264,8 @@ startup and import the free `capture()` function. See Agent guards - Guard testing and reference +- LangChain agent guard +- CrewAI agent guard - Testing Arcjet - Python SDK capture - Go SDK capture diff --git a/src/content/docs/guards/crewai.mdx b/src/content/docs/guards/crewai.mdx index 588dc178..081d3cc3 100644 --- a/src/content/docs/guards/crewai.mdx +++ b/src/content/docs/guards/crewai.mdx @@ -294,6 +294,13 @@ The core `guard_sync()` call still fails open. It returns `ALLOW`, and `has_failed_open()` returns `True`. The wrappers that sit around an effect fail closed. +`register_arcjet_hooks` records `metadata.outcome` as `success` when the +action proceeds, including when `on_guard_error="allow"` and policy did +not judge the action fully. It doesn't record `degraded`. `guard_tool` +goes through the checkpoint engine, so it records the five checkpoint +outcomes. For more information, see +Capture outcomes. + For more information about fail-open versus fail-closed behavior, see Availability and fail behavior. @@ -390,6 +397,7 @@ register_arcjet_hooks( ## Related - Framework integrations +- Capture events - LangChain agent guard - LangChain JS agent guard - Vercel AI SDK agent guard diff --git a/src/content/docs/guards/framework-integrations.mdx b/src/content/docs/guards/framework-integrations.mdx index 35e8c0b9..86186c79 100644 --- a/src/content/docs/guards/framework-integrations.mdx +++ b/src/content/docs/guards/framework-integrations.mdx @@ -137,7 +137,9 @@ Pick the surface that matches what you hold when the effect runs: - Observe only: `ArcjetCaptureHandler` / `ArcjetAsyncCaptureHandler`. These cannot deny a call. LangChain ignores callback return values. -For more information about install, helpers, fail-closed behavior, and +`guard_tool` and `ArcjetMiddleware` go through the checkpoint engine, so +their capture events include `metadata.outcome`. For more information +about install, helpers, fail-closed behavior, capture outcomes, and correlation, see the LangChain agent guard. @@ -181,8 +183,10 @@ There is no `arcjet[crewai]` extra. Install official [`b1253640`](https://github.com/arcjet/arcjet-py/commit/b1253640ce676b948594beed5fe62450d0e1c77d). Published `arcjet` 0.9.0 does not include it. -For more information about install, helpers, fail-closed behavior, and -`HookAborted`, see the +`register_arcjet_hooks` records `metadata.outcome` as `success` when the +action proceeds. It doesn't record `degraded`. `guard_tool` goes through +the checkpoint engine. For more information about install, helpers, +fail-closed behavior, `HookAborted`, and capture outcomes, see the CrewAI agent guard. ## LangGraph JS diff --git a/src/content/docs/guards/langchain.mdx b/src/content/docs/guards/langchain.mdx index 055f7a53..30adc269 100644 --- a/src/content/docs/guards/langchain.mdx +++ b/src/content/docs/guards/langchain.mdx @@ -299,6 +299,14 @@ The core `guard()` call still fails open. It returns `ALLOW`, and `has_failed_open()` returns `True`. The wrappers that sit around an effect fail closed. +Checkpoint surfaces go through the checkpoint engine, so the capture +event they emit includes `metadata.outcome`. If the action ran only +because `on_guard_error="allow"` and policy did not judge it fully, the +event records `degraded`. The default records `unavailable` and blocks. +For more information about the five values and how `decision_id` +distinguishes a partial judgment, see +Capture outcomes. + ## Correlation Pass one correlation ID into `ainvoke()` so middleware and guarded tools diff --git a/src/content/docs/guards/reference.mdx b/src/content/docs/guards/reference.mdx index e065caa6..ba432af1 100644 --- a/src/content/docs/guards/reference.mdx +++ b/src/content/docs/guards/reference.mdx @@ -179,8 +179,10 @@ Metadata is untrusted and is not redacted – do not put secrets or PII in it. Use `capture()` / `Capture` to record that an allowed action happened. Captures are visibility data: they never change a conclusion and never set -`hasFailedOpen()`. For options, batching, `flush()`, and serverless `waitUntil`, -see Capture events. +`hasFailedOpen()`. Python checkpoint helpers and JavaScript wrappers also +write `metadata.outcome` on the event they emit. For options, batching, +`flush()`, serverless `waitUntil`, and the outcome values, see +Capture events. ## Decisions and policy results diff --git a/src/content/docs/reference/python.mdx b/src/content/docs/reference/python.mdx index 1e9aa728..be2ac4af 100644 --- a/src/content/docs/reference/python.mdx +++ b/src/content/docs/reference/python.mdx @@ -719,6 +719,15 @@ checkpoint helpers. They fail closed by default a standalone `BaseTool` you call yourself. `POST_TOOL_CALL` is not registered. +LangChain checkpoint helpers and CrewAI `guard_tool` record +`metadata.outcome` on the capture event. The value is capture and +Sequence metadata. It doesn't change `conclusion`, which remains +`ALLOW` or `DENY`. For the five values and the `degraded` +discriminator, see +Capture outcomes. +`register_arcjet_hooks` records `success` when the action proceeds. It +doesn't record `degraded`. + For install, examples, correlation, and the configure-before-wrap rule, see the LangChain agent guard. For CrewAI hooks, see the @@ -941,7 +950,10 @@ href="/guards/reference#metadata">Guard metadata. `guard()` decides whether something is allowed. `capture()` records that it happened. It never affects a decision, never raises, and is not awaited even -on the async client. +on the async client. LangChain checkpoint helpers and CrewAI +`guard_tool` set `metadata.outcome` on the events they emit. A direct +`capture()` call doesn't. See +Capture outcomes. ```py aj.capture( diff --git a/src/content/docs/testing.mdx b/src/content/docs/testing.mdx index 6f542a34..db56f3d7 100644 --- a/src/content/docs/testing.mdx +++ b/src/content/docs/testing.mdx @@ -19,7 +19,9 @@ and the in-memory test client. Guard tests must assert that a denied or unavailable decision prevents the side effect, not only that Arcjet returned a particular result. Capture tests record that an allowed action happened, so they must assert that the application recorded the expected `action` after the side -effect ran. See Capture events. +effect ran. When you test a Python checkpoint helper, assert +`metadata.outcome` as well. See +Capture outcomes. ## Test with Newman diff --git a/src/content/docs/troubleshooting.mdx b/src/content/docs/troubleshooting.mdx index 55ab2704..5b0062c4 100644 --- a/src/content/docs/troubleshooting.mdx +++ b/src/content/docs/troubleshooting.mdx @@ -228,6 +228,10 @@ LangChain wrappers do this by default. See Agent guard availability and fail behavior and the LangChain agent guard. +Python checkpoint helpers that proceed because `on_guard_error="allow"` +record `metadata.outcome` as `degraded` on the capture event. See +Capture outcomes. + ### Arcjet uses 127.0.0.1 when the public IP address is missing in development mode Arcjet's automatic IP detection expects a valid public IP address to enable diff --git a/tests/screenshot.test.ts-snapshots/screenshot-guards-framework-integrations-dark-chromium-linux.png b/tests/screenshot.test.ts-snapshots/screenshot-guards-framework-integrations-dark-chromium-linux.png index 60d273cf..2d64d07d 100644 Binary files a/tests/screenshot.test.ts-snapshots/screenshot-guards-framework-integrations-dark-chromium-linux.png and b/tests/screenshot.test.ts-snapshots/screenshot-guards-framework-integrations-dark-chromium-linux.png differ diff --git a/tests/screenshot.test.ts-snapshots/screenshot-guards-framework-integrations-light-chromium-linux.png b/tests/screenshot.test.ts-snapshots/screenshot-guards-framework-integrations-light-chromium-linux.png index 4f1580b7..3b171371 100644 Binary files a/tests/screenshot.test.ts-snapshots/screenshot-guards-framework-integrations-light-chromium-linux.png and b/tests/screenshot.test.ts-snapshots/screenshot-guards-framework-integrations-light-chromium-linux.png differ diff --git a/tests/screenshot.test.ts-snapshots/screenshot-testing-dark-chromium-linux.png b/tests/screenshot.test.ts-snapshots/screenshot-testing-dark-chromium-linux.png index 6a8170ae..e3145459 100644 Binary files a/tests/screenshot.test.ts-snapshots/screenshot-testing-dark-chromium-linux.png and b/tests/screenshot.test.ts-snapshots/screenshot-testing-dark-chromium-linux.png differ diff --git a/tests/screenshot.test.ts-snapshots/screenshot-testing-light-chromium-linux.png b/tests/screenshot.test.ts-snapshots/screenshot-testing-light-chromium-linux.png index a5b81966..8a80d997 100644 Binary files a/tests/screenshot.test.ts-snapshots/screenshot-testing-light-chromium-linux.png and b/tests/screenshot.test.ts-snapshots/screenshot-testing-light-chromium-linux.png differ diff --git a/tests/screenshot.test.ts-snapshots/screenshot-troubleshooting-dark-chromium-linux.png b/tests/screenshot.test.ts-snapshots/screenshot-troubleshooting-dark-chromium-linux.png index b75770ed..e4080b75 100644 Binary files a/tests/screenshot.test.ts-snapshots/screenshot-troubleshooting-dark-chromium-linux.png and b/tests/screenshot.test.ts-snapshots/screenshot-troubleshooting-dark-chromium-linux.png differ diff --git a/tests/screenshot.test.ts-snapshots/screenshot-troubleshooting-light-chromium-linux.png b/tests/screenshot.test.ts-snapshots/screenshot-troubleshooting-light-chromium-linux.png index 37d6918f..71bbf222 100644 Binary files a/tests/screenshot.test.ts-snapshots/screenshot-troubleshooting-light-chromium-linux.png and b/tests/screenshot.test.ts-snapshots/screenshot-troubleshooting-light-chromium-linux.png differ