Compliance and red-teaming harness for tool-using LLM agents.
Agentshield runs an agent against adversarial scenarios inside a sandbox, traces every model and tool call, scores the trace against a declarative policy, and writes a signed audit report. A GitHub Action fails a pull request when the agent regresses.
The adapter layer, the sandbox, the scenario suites, tracing, the policy scorer, signed-report verification, and the HTTP service are in place.
One interface, AgentUnderTest.step, asks an agent for its next action and gets back either tool calls or a final answer. The agent does not execute tools. Three adapters sit on that interface: HTTP (any service that returns the next action as JSON), OpenAI SDK (a live client, or a recorded completion in tests), and LangGraph (a graph that yields the next step).
The sandbox is what runs that step loop and executes tools:
- A deterministic starter toolset —
search,read_file,send_email,http_get,db_query— each with a trap variant selected by scenario state, not a global ToolRegistry, the only path from a requested tool call to a result; an unknown tool or schema-invalid arguments come back as a rejection, not a crash- An executor that dispatches every tool call through the registry only, and stops on a final answer, a step limit, a time limit, or an agent crash
- A reported process boundary (
inprocessby default;subprocess, hardened withfirejail --net=nonewhen it is installed) — see the threat model for exactly which guarantees hold today
agentshield scenarios lists the 22 shipped scenarios (id, suite, and expectation names) and does not run them. The runner scores each expectation with the same rules as a policy file. Every run stores an ordered trace, including full tool arguments. Optional OpenTelemetry spans carry the argument hash, not the raw arguments. A trace plus a policy file produces one row per rule. CI replays those agents, fixtures, and recorded judge votes. It does not call a model provider. agentshield verify checks a signature over the report's canonical JSON. The default signer in tests is local, not ML-DSA. The optional qknot extra signs with Ed25519 and ML-DSA-87; that check uses the public keys inside the bundle. The service extra accepts POST /runs and returns the signed report when a worker finishes. agentshield run scores a suite and exits 1 when a rule at or above the chosen severity fails.
Release. Tag 0.1.0 and publish only when that is requested. A five-model live benchmark still needs provider keys.
The local results table is RESULTS.md. The image is the root Dockerfile.
Python 3.11 or newer, from a checkout of this repository:
python -m venv .venvOn Windows, activate with .venv\Scripts\activate. On macOS and Linux, activate with source .venv/bin/activate.
python -m pip install -e ".[dev,docs]"
agentshield --versionThe OpenAI, LangGraph, OpenTelemetry, and service clients are optional: .[openai], .[langgraph], .[otel], and .[service]. The core install does not import them.
- Quickstart
- Adapters
- Sandbox
- Tracing
- Service
- Dashboard
- Threat model
- Writing scenarios
- Policy reference
- Roadmap
mkdocs build --strict builds the site after the docs extra is installed.
ruff check .
ruff format --check .
mypy
pytestContributor workflow, versioning, and the changelog rule are in CONTRIBUTING.md.