An adversarial, self-improving laboratory for computer-use agent reliability.
Break your AI agent before reality does.
AgentForge is a research-engineering framework for evaluating computer-use agents under conditions that ordinary task-success benchmarks often miss: misleading controls, destructive decoys, ambiguity, prompt injection, stale observations, reordered interfaces, repeated execution, and recovery from failure.
The project is deliberately model-agnostic. An LLM is a policy component; AgentForge focuses on the infrastructure around that policy: safe action validation, evidence capture, outcome verification, reliability measurement, root-cause analysis, counterfactual reasoning, tamper-evident auditability, and regression-gated improvement.
A computer-use agent can succeed once and still be unreliable.
A serious reliability evaluation therefore needs to answer more than:
“Did the task finish?”
AgentForge evaluates:
- Outcome: Was the intended task actually completed?
- Process: Were actions valid, efficient, and policy-compliant?
- Safety: Were dangerous operations prevented or avoided?
- Reliability: Does the behavior remain stable across repeated trials?
- Auditability: Can the trajectory and evidence be independently inspected?
- Attribution: Was a failure caused by the agent, the environment, or the evaluation itself?
- Recovery: Can the system detect failure and re-plan rather than blindly continue?
- Improvement: Can proposed changes be compared against historical behavior and rejected when they regress reliability?
┌──────────────────────┐
│ Agent Policy │
│ Baseline / LLM │
└──────────┬───────────┘
│ decision
▼
┌──────────────────────┐
│ Safety + Action │
│ Validator │
└──────────┬───────────┘
│ approved action
▼
┌──────────────────────┐
│ Browser Executor │
│ / Solari Sandbox │
└──────────┬───────────┘
│ observation
▼
┌──────────────────────┐
│ Verifier │
│ outcome + evidence │
└──────────┬───────────┘
│
┌────────────────┼────────────────┐
▼ ▼ ▼
Trajectory Audit Trail Metrics
+ Evidence SHA-256 chain Reliability
│ │
└──────────────┬──────────────────┘
▼
┌──────────────────────┐
│ Root-Cause Analysis │
│ + Counterfactuals │
└──────────┬───────────┘
▼
┌──────────────────────┐
│ Improvement Engine │
│ + Regression Gate │
└──────────────────────┘
Observe → Decide → Validate → Execute → Observe → Verify
│
└→ Recover / Re-plan when required
AgentForge keeps outcome and process evaluation separate.
A run may fail its objective while still demonstrating safe behavior. Conversely, an agent may eventually reach the right page after taking an unsafe action. Those cases must not receive the same score.
The evaluator therefore tracks:
| Dimension | Examples |
|---|---|
| Outcome | success rate, confidence, verification status |
| Process | process success, action efficiency |
| Safety | safety compliance, successful destructive execution |
| Action reliability | action failure rate, failed-run rate |
| Reliability | Wilson confidence interval, outcome consistency, worst-case confidence |
| Auditability | trajectory evidence, hash-chain integrity |
| Diagnosis | controllability, root cause, counterfactual alternatives |
| Improvement | historical comparison, regression accept/reject |
This separation is one of the project's central design principles.
Benchmarks can be mutated to test whether an agent relies on brittle positional or lexical shortcuts.
Current mutation strategies include:
rename-decoyreorder-elementsadd-decoychange-labelchange-layoutinject-instructionadd-constraintincrease-step-count
The benchmark schema also models attack classes such as:
- prompt injection
- misleading labels
- destructive decoys
- ambiguous intent
- irreversible actions
- constraint conflicts
- visual decoys
- stale observations
- action-ordering traps
- confirmation bypass
- hidden risk
AgentForge also audits the benchmark itself before treating results as evidence. It checks task specification, objectives, constraints, success/failure criteria, evidence requirements, adversarial design, risk level, and evaluation mode.
This matters because a flawed evaluator can produce a misleading agent score even when the agent behavior is unchanged.
Safety is enforced before execution, rather than inferred only after a task has completed.
The action-validation layer checks:
- current observed element identity
- semantic target consistency
- action type requirements
- destructive/high-risk controls
- stale or mismatched element identities
- required inputs
- invalid action parameters
Element-based actions carry an identity snapshot containing the observed tag, text, role, and ARIA label. This prevents a stale element reference from silently resolving to a different control after the page changes.
Importantly, a blocked destructive action is not counted as harmful execution. Successful execution of a destructive operation is treated differently from a validator rejection or an execution failure.
Every trajectory can contain:
- actions
- observations
- screenshots
- URLs
- titles
- page text
- outputs
- validation failures
- action failures
- verification results
Audit trails use a SHA-256 hash chain. Each event commits to the previous event hash, and the genesis hash binds the chain to the experiment/task identity and creation metadata.
The verifier can therefore detect modifications to:
- event payloads
- event order
- event sequence numbers
- timestamps
- previous hashes
- task/experiment identity
- final chain state
This turns an execution log into tamper-evident evidence, rather than an ordinary debug log.
When a run fails, AgentForge attempts to distinguish:
Controllable by agent
│
├─ wrong action selection
├─ premature completion
├─ constraint violation
└─ poor recovery
Outside agent control
│
├─ environment unavailable
├─ browser/session failure
└─ transient infrastructure failure
Mixed / uncertain
│
└─ evidence insufficient for a confident attribution
The system also generates counterfactual alternatives and evaluates whether a plausible safer action could have avoided the observed failure.
This makes failure analysis actionable instead of stopping at FAILED.
AgentForge does not automatically declare a change an improvement merely because one run looks better.
The improvement pipeline is:
Failure
↓
Root cause
↓
Improvement proposal
↓
New evaluation
↓
Historical comparison
↓
Regression gate
↓
ACCEPT / REJECT / UNCHANGED
Historical comparisons include outcome, process, safety, action reliability, consistency, and composite evaluation signals.
This is intended to prevent the common failure mode where an agent becomes better at one benchmark while silently becoming less safe or less reliable elsewhere.
The latest limited-model evaluation used four trials. The current snapshot is intentionally reported without hiding failures:
| Metric | Current LLM policy | Deterministic baseline |
|---|---|---|
| Trials | 4 | 4 |
| Success rate | 50% | 75% |
| Average confidence | 47.5% | 72.5% |
| Action failure rate | 0% | 0% |
| Verification failure rate | 50% | 25% |
| Safety compliance | 100% | — |
| Overall evaluation score | 63.50/100 | 81.75/100 |
| Average duration | 25.8 s | 41.2 s |
The current model policy does not outperform the deterministic baseline on this small benchmark. That result is retained as a limitation rather than hidden or tuned away.
At the same time, the execution layer recorded 0% action failures and the safety evaluator recorded 100% safety compliance in the latest four-trial evaluation. The remaining weakness is primarily task-completion/verification reliability, not a demonstrated collapse of the action-safety infrastructure.
Because this is a four-trial snapshot, these figures should be treated as an engineering experiment rather than a statistically definitive model ranking.
AgentForge/
├── benchmarks/
│ ├── environments/ # Browser benchmark environments
│ └── tasks/browser/ # Benchmark specifications
├── packages/
│ ├── agent/
│ │ ├── core/ # Agent loop and baseline policy
│ │ ├── llm/ # LLM policy + structured decisions
│ │ └── validator/ # Action validation + risk engine
│ ├── evaluator/ # Benchmark runner, metrics, RCA, audit
│ ├── shared/ # Shared schemas and trajectory types
│ ├── solari/ # Solari client adapter
│ └── verifier/ # Outcome verification
├── tests/ # Core integrity tests
├── experiments/runs/ # Local experiment artifacts (ignored)
├── .env.example # Configuration template
└── package.json
- Node.js 22+
- npm
- A Solari API key for sandbox/browser evaluation
- A Gemini API key only when running the LLM policy
npm ciCopy .env.example to .env and provide credentials locally.
Never commit .env or API keys.
npm run checkThis runs TypeScript type checking and the automated integrity tests.
npm run evaluatenpm run agent- Safety before execution — risky actions are gated before they reach the environment.
- Evidence over assertions — verification uses recorded observations and explicit evidence.
- Outcome ≠ process — successful completion does not erase unsafe behavior.
- Failure attribution matters — environment failures should not automatically become agent failures.
- Repeated execution matters — one successful trajectory is not reliability.
- Benchmarks need auditing too — evaluator defects can create false conclusions.
- Improvements need regression gates — local gains are not enough.
- Reproducibility over hidden state — trajectories and audit artifacts should make decisions inspectable.
- Model-agnostic infrastructure — the evaluation stack should survive policy/model replacement.
- Honest reporting — limitations and negative results remain visible.
- The current browser benchmark suite is intentionally small and cannot support broad claims about general computer-use reliability.
- The latest LLM comparison contains only four trials and should not be interpreted as a definitive statistical ranking.
- Some benchmark environments depend on external sandbox/browser infrastructure.
- A stronger model/policy may change task-completion results without changing the underlying AgentForge evaluation stack.
- Full desktop interaction and larger real-world workflows remain future extensions.
AgentForge is designed around a broader question:
How should computer-use agents be evaluated when reliability, safety, auditability, and recoverability matter more than one-shot task success?
The framework is intended to support future experiments across models, tasks, adversarial conditions, and repeated trials while keeping the evaluation and evidence layers independent of the policy being tested.
Research-engineering prototype / active development.
The system is functional, type-checked, and equipped with automated integrity tests and CI validation. Experimental results are reported with their limitations rather than presented as production-level guarantees.