Agent Incident Forensics
Portable, redactable, hash-chained evidence for agent runs.
Seal it. Ship it. Re-verify it — without trusting the producer’s UI.
Traces show you a story in a dashboard.
SAGE gives you an artifact another team can fail-closed verify offline.
Observability products already exist. The gap is a backend-neutral forensic pack: redacted before hash, content-addressed blobs, custody MAC, optional pinned Ed25519, third-party sage verify.
SAGE is a security tool, not a chat UI, not an RCA leaderboard, and not “AI observability SaaS.”
Latest published release: v2.3.0 (wheel,
sage-verify pyz, checksums, SBOM, Sigstore bundles). A source checkout tracks main; use Install
for either path.
A synthetic client-support workflow recorded through real LangChain (langchain-core==1.8.0), with a
deterministic test model, a tool failure, a sealed and signed pack, and a recipient-verifiable kit.
From the repository root (tested with Python 3.10 and 3.12):
python -m venv .venv # Windows: py -3.12 -m venv .venv
. .venv/bin/activate # PowerShell: .venv\Scripts\Activate.ps1
python -m pip install -e ".[langchain,sign]"
python examples/consultancy_demo.py --scenario tool-error --mode sync --out demo-runs/run-01demo-runs/run-01/handoff/ is the shareable kit and demo-runs/run-01/keys/ stays with the producer.
INCIDENT_SUMMARY.md reports three separate results: workflow outcome, capture completeness, and
evidence verification. Scenarios: success, tool-error, tool-timeout; modes: sync, async.
The output directory must not already exist (or must be empty).
Verify the kit as a recipient would. In the demo the keys are in ../keys relative to the kit:
sage handoff verify demo-runs/run-01/handoff --hmac-key-file demo-runs/run-01/keys/hmac.key --public-key-file demo-runs/run-01/keys/ed25519.public.txt
sage key fingerprint demo-runs/run-01/keys/ed25519.public.txt # same value as pinned_key_fingerprint aboveThe same commands work in PowerShell (\ or / in paths). Exit 0 means the evidence, policy, and
receipt all verified. On failure, error_type names what failed (SignatureMismatchError,
ReceiptMismatchError, PolicyViolationError, ChainIntegrityError).
Before you share a kit, scan its evidence for possible leaks. The scan reports each finding's
location and rule, never the value. Exit 0 means no findings, 1 means findings to review. It does
not cover the convenience notes in the kit (INCIDENT_SUMMARY.md, build_record.json), so read
those yourself. See docs/REDACTION.md.
sage scan demo-runs/run-01/handoff/evidence.sage.tar.gzTo build a kit from your own recording (a sealed .sage.json bundle or a journal directory):
sage handoff path/to/journal --out-dir handoff-kit --hmac-key-file keys/hmac.keyWithout --hmac-key-file, the key comes from the SAGE_PACK_KEY environment variable. Add --sign
(with SAGE_SIGN_PRIVATE_KEY set) and --policy for a signed kit. sage handoff warns when the
leak scan finds something. Pass --fail-on-findings to refuse instead.
A real recipient has no checkout: they get the kit, the wheel, and the two key files separately, and
follow HANDOFF.md. A recipient who cannot install packages can use the single-file verifier from
the release, sage-verify-<version>.pyz, which needs only Python 3.10 or newer:
python sage-verify-<version>.pyz <kit> --hmac-key-file <file> --public-key-file <file>.
pip install -e ".[dev]"
export SAGE_PACK_KEY="replace-me"
python examples/security_verify_loop.py
sage verify evidence.sage.tar.gz \
--policy policies/strict.json \
--witness \
--receipt verify.receipt.jsonAuditor posture (pinned signature):
python examples/auditor_kit.py
# docs: docs/AUDITOR_KIT.md · docs/VERIFY_RUNBOOK.mdSageRecorder records only the spans you open or that an adapter opens for you. Wrapping a call in the
recorder does not trace it by itself.
Manual spans:
from sage import SageRecorder
with SageRecorder(trace_id="incident-42") as rec:
with rec.tool_call("lookup", inputs={"account": account_id}) as span:
span.set_output(result=lookup(account_id))
rec.export("incident.sage.json")LangChain (automatic for the runnables, chat models, and tools invoked with the callback):
from sage import SageRecorder
from sage.adapters.langchain_callback import SageLangChainCallback
with SageRecorder(trace_id="incident-42", framework="langchain") as rec:
handler = SageLangChainCallback(rec)
chain.invoke(inputs, config={"callbacks": [handler]}) # AsyncSageLangChainCallback for ainvoke
handler.assert_complete() # raises if capture was incomplete
rec.export("incident.sage.json")The same handler works for LangGraph: pass it in the graph's config (pip install ".[langgraph]").
Tested scope: langchain-core==1.8.0 (also 1.6.13 and 1.7.7) and langgraph==1.2.21 on Python 3.12.
That covers chains, chat models, tools, retrievers, streaming, batch, RunnableParallel, and LangGraph
graphs with conditional edges, tool nodes, parallel branches, subgraphs, and interrupt/resume, sync and
async, one callback instance per workflow. COMPATIBILITY.md lists exactly what is and is not
validated. Callback errors surface to the caller (raise_error=True), so start in development or staging.
OpenAI Agents SDK (pip install -e ".[openai-agents]", tested with openai-agents==0.24.0):
from agents import Runner, set_trace_processors
from sage.adapters.openai_agents import SageOpenAIAgentsProcessor
processor = SageOpenAIAgentsProcessor(trace_id="incident-42", journal_dir="run.journal")
set_trace_processors([processor]) # add_trace_processor(...) keeps the OpenAI exporter too
with processor:
Runner.run_sync(agent, "Where is order 42?")
processor.assert_complete()
processor.export("incident.sage.json")Agents, model calls, tool calls, handoffs, guardrails, and the SDK's task/turn spans become SAGE spans nested as the SDK nested them.
PydanticAI (pip install -e ".[pydantic-ai]", tested with pydantic-ai-slim==2.68.0):
from sage.adapters.pydantic_ai import SagePydanticAI
sage = SagePydanticAI(trace_id="incident-42", journal_dir="run.journal")
agent = sage.instrument(agent) # or sage.instrument_all()
with sage:
agent.run_sync("Where is order 42?")
sage.assert_complete()
sage.export("incident.sage.json")It reads PydanticAI's OpenTelemetry spans (invoke_agent, chat, execute_tool) through a private
tracer provider; nothing is sent to an external exporter.
Existing OpenTelemetry traces (OTLP JSON, JSON Lines, or protobuf with pip install -e ".[otlp]"):
sage import traces.pb --format otlp --out incident.sage.json
sage verify incident.sage.jsonOpenInference (llm.*, tool.*, retrieval.documents.*) and GenAI (gen_ai.*) attributes become
typed span data. Tested against exports from opentelemetry-sdk==1.45.2. See COMPATIBILITY.md for
what each adapter covers and what is not validated.
For production agent runs, use the default "batch" durability (omit journal_durability=). Use
"span" only when every closed span must hit disk before end_span returns; use "close" for
bulk capture with rare flush_journal() checkpoints.
Pass journal_dir= to write a crash-recoverable journal while the run is in progress. Closed spans
are redacted, offloaded, hash-chained, and appended to spans.jsonl / chain.jsonl.
journal_durability= selects when they reach the disk:
| Mode | When a closed span is on disk | What a crash can lose |
|---|---|---|
"span" |
Before end_span returns (four fsyncs per span) |
Nothing that was closed |
"batch" (default) |
A background thread group-commits with fsync when the oldest waiting span is 50 ms old or 256 spans wait, whichever comes first | Spans closed inside that window, plus the commit under way |
"close" |
Same background writes, but no fsync until flush_journal() or the seal |
Process crash: as "batch". Operating-system crash or power loss: anything since the last flush_journal() |
with SageRecorder(trace_id="incident-42", journal_dir="run.journal", journal_durability="batch") as rec:
...
rec.flush_journal() # optional checkpoint: returns once everything closed so far is fsynced- The window is tunable:
journal_batch_interval_ms=andjournal_batch_max_spans=. It is also bounded: once 4 ×journal_batch_max_spansspans (1,024 by default) are uncommitted,end_spanwaits for the disk. When the disk is slow, the window stretches in time but not past that count. SAGE_JOURNAL_DURABILITY=span|batch|closesets the mode when the argument is not given. The adapters (SageLangChainCallback,SageOpenAIAgentsProcessor,SagePydanticAI,wrap_agent_run,attach_otel_tap,instrument,wrap_run) acceptjournal_durability=too.- Sealing (leaving the
withblock,finalize(),export()) always flushes, fsyncs, and stops the writer thread. A write failure on the writer thread is raised asJournalWriteErrorfrom the nextend_span,flush_journal(), or the seal. It is never dropped. - In
"batch"and"close"mode, read the live files only afterflush_journal();end_spanreturning no longer means the span is on disk. Use"span"if you need that guarantee. - After a crash,
sage audit <journal>(orsage.journal.recover_journal) returns the intact prefix. A journal killed in the middle of a commit can failsage verify-journaluntil recovered. - Offloaded blobs follow the same schedule: in
"batch"and"close"mode they are fsynced in groups before the journal lines that refer to them are committed (and always before the seal). A blob never gets its content-address name before it has been fsynced; until then only the recording process can read it. - The mode changes timing only. Record formats and every hash are identical in all three modes, and journals written by earlier releases verify unchanged.
Measure it on your own hardware: python scripts/measure_overhead.py --mode journal --durability all
prints per-span medians by mode and payload size as JSON. CI runs the same script on Linux and keeps
the result as the journal-benchmark artifact. docs/perf.md has reference numbers
for Linux and Windows. These are measurements, not a service-level commitment.
| Capability | Detail |
|---|---|
| Recorder | Sanitize-on-close, redact-before-hash, CAS offload |
| Journal | Crash-safe live WAL (durability span / batch / close) → sealed manifest + merkle |
| Pack v2 | Portable .sage.tar.gz with custody-bound HMAC |
| Verify | Policies, blob inventory, witness, receipts |
| Signatures | Optional Ed25519 — pinned key required (TOFU refused) |
| Handoff | Offline auditor kit (sage handoff) |
| Adapters | LangChain callback, OpenAI Agents SDK processor, PydanticAI (tested against the pinned real packages); OTLP / OpenInference import; CrewAI/AutoGen wrappers are unit-tested only |
Formats are frozen under the 2.x compatibility covenant.
- Does not prove the agent / LLM / tools told the truth
- Does not guarantee complete capture of arbitrary applications; only instrumented work is recorded
- Does not diagnose root cause automatically
- Redaction applies configured key and pattern rules (plus opt-in shape detectors), and
sage scanchecks a pack before you share it; neither detects every kind of sensitive data. See docs/REDACTION.md - Software HMAC/Ed25519 ≠ HSM identity — see THREAT_MODEL.md
- Not a legal-admissibility or regulatory-compliance claim
- Research (
sage research *) is not forensic custody
Pilot / recipient (no checkout): download
sage_incident_bundles-2.3.0-py3-none-any.whl
from the v2.3.0 release. Verify
SHA256SUMS.json with scripts/verify_release.py when you need attestation checks.
python -m venv .venv && .venv\Scripts\activate # PowerShell; on Unix: source .venv/bin/activate
python -m pip install "sage_incident_bundles-2.3.0-py3-none-any.whl[sign]" # drop [sign] for HMAC-only kits
sage doctorOffline verifier only (Python 3.10+, no pip):
python sage-verify-2.3.0.pyz <handoff-kit> --hmac-key-file keys/hmac.key --public-key-file keys/ed25519.public.txtPyPI publish is optional (off until the maintainer enables trusted publishing — see RELEASE.md).
Developers (checkout on main):
pip install -e ".[dev]" # core + tests + cryptography
pip install -e ".[langchain]" # pinned langchain-core for the callback adapter
pip install -e ".[langgraph]" # LangGraph StateGraph tests / demos
pip install -e ".[openai-agents]" # pinned openai-agents for the tracing processor
pip install -e ".[pydantic-ai]" # pinned pydantic-ai-slim + opentelemetry-sdk
pip install -e ".[otlp]" # pinned opentelemetry-proto for binary OTLP import
pip install -e ".[sign]" # Ed25519 only
pip install -e ".[tui]" # optional inspect TUICore recorder / CAS / verify / CLI remain stdlib-only.
python scripts/release_check.py
pytest -q
python examples/auditor_kit.py
sage doctorCI: Ubuntu · Windows · macOS × Python 3.10 / 3.12.
| Doc | Purpose |
|---|---|
| PILOT_ONBOARDING.md | Send to design partners: install from release, five-minute proof |
| LANGGRAPH_PILOT.md | LangGraph wiring and validated scope |
| PILOT_SALES_CHECKLIST.md | Kyle: demo, legal placeholders, before first invoice |
| VERIFY_RUNBOOK.md | Record → ship → auditor re-verify |
| AUDITOR_KIT.md | Pinned-signature verify posture |
| REDACTION.md | Redaction rules, opt-in detectors, sage scan pre-share check |
| THREAT_MODEL.md | Assets, adversaries, non-goals |
| FORMAT.md | Normative artifact spec; conformance vectors in tests/vectors/ |
| SECURITY.md | Supported versions / reporting |
| RELEASE.md | Tag / publish checklist |
| CONTRIBUTING.md | PR bar for a security tool |
If you run agents in production and want sealed incident packs on one real path — open an issue or reach out. We want brutal feedback, not vanity stars.
FSL-1.1-ALv2 — Functional Source License (Fair Source).
- ✅ Internal use, education, research, self-host for your company
- ❌ Competing commercial product/SaaS without a commercial license
- ⏱ Converts to Apache-2.0 2 years after each version’s availability
Commercial terms: COMMERCIAL.md · License FAQ: fsl.software
