From 586282d6a88b310234afef487e7f519108c466bb Mon Sep 17 00:00:00 2001 From: Magyari Sandor Szilard Date: Mon, 8 Jun 2026 11:46:16 +0200 Subject: [PATCH 1/6] chore: first plan for slim vs a2a test Signed-off-by: Magyari Sandor Szilard --- ...slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md | 435 ++++++++++++++++++ benchmarks/agntcy-multi-agent/go.mod | 5 + benchmarks/agntcy-multi-agent/go.sum | 4 + benchmarks/agntcy-multi-agent/plans/README.md | 57 +++ .../plans/domains/k8s-incident-response.yaml | 173 +++++++ .../plans/domains/mobile-nav-assistant.yaml | 140 ++++++ .../plans/domains/urban-traffic-reroute.yaml | 156 +++++++ .../tools/validate_plans/main.go | 104 +++++ 8 files changed, 1074 insertions(+) create mode 100644 .cursor/plans/slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md create mode 100644 benchmarks/agntcy-multi-agent/go.mod create mode 100644 benchmarks/agntcy-multi-agent/go.sum create mode 100644 benchmarks/agntcy-multi-agent/plans/README.md create mode 100644 benchmarks/agntcy-multi-agent/plans/domains/k8s-incident-response.yaml create mode 100644 benchmarks/agntcy-multi-agent/plans/domains/mobile-nav-assistant.yaml create mode 100644 benchmarks/agntcy-multi-agent/plans/domains/urban-traffic-reroute.yaml create mode 100644 benchmarks/agntcy-multi-agent/tools/validate_plans/main.go diff --git a/.cursor/plans/slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md b/.cursor/plans/slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md new file mode 100644 index 00000000..568f81e3 --- /dev/null +++ b/.cursor/plans/slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md @@ -0,0 +1,435 @@ +--- +name: SLIM vs A2A DAG benchmark +overview: Design a new CSIT benchmark package with domain-specific YAML DAG plans (concrete agent/task names), two fully separate executors (A2A gRPC vs SLIM Multicast RPC), and comparable metrics demonstrating SLIM's advantages in fast sync, fast context sharing, and fast failure propagation. +todos: + - id: plan-schema + content: Define shared internal/plan YAML types, validation, and DAG helpers (only shared layer) + status: pending + - id: domain-plans + content: "Phase 1 — hand-author 3 domain YAML plans in plans/domains/ (no plangen)" + status: pending + - id: a2a-impl + content: Standalone A2A stack — cmd/a2a-agent, cmd/a2a-runner, internal/a2a/* (no SLIM imports) + status: pending + - id: slim-impl + content: Standalone SLIM stack — cmd/slim-agent, cmd/slim-runner, proto + internal/slim/* (no A2A imports) + status: pending + - id: metrics-reporting + content: Shared metrics schema + TSV/markdown reporting with sync/context/failure propagation fields + status: pending + - id: taskfile-suite + content: Wire Taskfile, Ginkgo suite, and CI smoke workflow following agntcy-slim patterns + status: pending +isProject: false +--- + +# SLIM vs A2A Multi-Agent DAG Benchmark + +## Goal + +Build a reproducible benchmark in CSIT that shows **when SLIM Multicast RPC is favorable over direct A2A gRPC** in a distributed multi-agent workload: + +- **Phase 1 (plans):** Hand-author YAML execution plans with concrete agent/task names — no code generator +- **Phase 2 (implementation):** Separate A2A and SLIM runners/agents that execute those plans and emit metrics +- Tasks use **concrete domain-specific names** (network probe, pod log scrape, route recalc, etc.) but **simulate/mock** execution via timed sleeps and canned outputs +- Parallel branches become **obsolete** when an upstream dependency fails or times out +- **SLIM advantages under test** (three pillars): + 1. **Fast sync** — agents stay aligned on shared execution state with low coordination latency + 2. **Fast context change sharing** — when upstream results or priorities shift, all affected agents receive updated context in one fan-out (multicast) vs N point-to-point pushes + 3. **Fast failure propagation** — one multicast abort/notify vs K sequential A2A calls to cancel obsolete parallel work +- **A2A and SLIM are completely separate implementations** — separate binaries, packages, and RPC stacks; only YAML plan parsing/validation is shared (`internal/plan`) + +## Recommended defaults (questions skipped) + + +| Decision | Choice | Rationale | +| ------------- | ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Runtime | **Local processes + embedded SLIM dataplane** | Matches `[benchmarks/agntcy-slim](benchmarks/agntcy-slim)`; fast iteration; K8s can be phase 2 | +| Orchestration | **Separate runners per transport** (`a2a-runner`, `slim-runner`) | Each impl owns its own DAG driver; same plan YAML input, no shared scheduler interface | +| Language | **Go-only** | User asked for A2A Go SDK + SLIM Go bindings; existing Python `slima2a` is out of scope | +| SLIM API | **Custom slimrpc protobuf + multicast group client** | Multicast for context push + cancel/notify; upstream pattern in `[client_group.go](/Users/smagyari/dev/slim/data-plane/bindings/go/examples/slimrpc/simple/cmd/client_group/client_group.go)` | +| Shared code | **`internal/plan` only** | YAML load, validate, DAG helpers; no shared transport, agent, or orchestrator code | + + +## Architecture + +```mermaid +flowchart TB + subgraph shared [Shared layer] + PlanYAML[plans/*.yaml] + PlanLib[internal/plan] + PlanYAML --> PlanLib + end + + subgraph a2a [A2A implementation — standalone] + A2ARunner[a2a-runner] + A2AAgents[a2a-agent pool] + A2ARunner --> A2AAgents + end + + subgraph slim [SLIM implementation — standalone] + SlimRunner[slim-runner] + SlimAgents[slim-agent pool] + SlimRunner --> SlimAgents + end + + subgraph bench [Benchmark harness] + Suite[Ginkgo suite_test.go] + Suite --> A2ARunner + Suite --> SlimRunner + A2ARunner --> Metrics[results.tsv + summary.md] + SlimRunner --> Metrics + end + + PlanLib --> A2ARunner + PlanLib --> SlimRunner +``` + +### Why this highlights SLIM + +Both implementations execute the **same YAML plan** (same agents, tasks, timings, dependencies). Implementations are independent but must produce **comparable metrics**. SLIM's three advantages show up at coordination boundaries: + +| Coordination event | A2A gRPC (sequential / point-to-point) | SLIM Multicast RPC | +| ------------------ | -------------------------------------- | ------------------ | +| Execute task on one agent | Unary gRPC `SendMessage` per task | Point-to-point slimrpc unary | +| **Share context update** to K agents (new priority, revised hypothesis, abort reason) | **K sequential** push RPCs | **1 multicast** `PushContext` fan-out | +| **Sync barrier** — all agents acknowledge phase transition | **K sequential** sync RPCs + client-side aggregation | **1 multicast** `SyncPhase` + aggregated stream responses | +| Cancel obsolete parallel tasks (K agents) | **K sequential** cancel RPCs | **1 multicast** `CancelTasks` | +| Notify dependents of failure | **M sequential** notify RPCs | **1 multicast** `NotifyFailure` | + +Metrics like `context_push_ms`, `sync_barrier_ms`, `cancel_propagation_ms`, `obsolete_tasks_completed`, and `total_wall_clock_ms` should show SLIM advantage as agent count, parallel fan-out, and context-change frequency grow. + +## New package layout + +Create `[benchmarks/agntcy-multi-agent/](benchmarks/agntcy-multi-agent/)` and register it in `[benchmarks/Taskfile.yml](benchmarks/Taskfile.yml)`: + +``` +benchmarks/agntcy-multi-agent/ +├── Taskfile.yml +├── internal/ +│ └── plan/ # ONLY shared code: YAML schema, validation, DAG helpers +├── a2a/ # completely separate A2A implementation +│ ├── proto/ # optional A2A task payload JSON schema (not slim proto) +│ ├── cmd/ +│ │ ├── agent/main.go # a2a-agent — generic worker, differs by name/port only +│ │ └── runner/main.go # a2a-runner — loads plan, drives DAG via A2A gRPC +│ └── internal/ +│ ├── executor/ # task mock execution (sleep, timeout, cancel) +│ ├── scheduler/ # DAG state machine for A2A path +│ └── client/ # a2a-go card resolve, gRPC calls +├── slim/ # completely separate SLIM implementation +│ ├── proto/taskdag/v1/taskdag.proto # slimrpc service (Execute, Cancel, PushContext, SyncPhase) +│ ├── cmd/ +│ │ ├── agent/main.go # slim-agent — generic worker on dataplane +│ │ └── runner/main.go # slim-runner — loads plan, drives DAG via slimrpc +│ └── internal/ +│ ├── executor/ # same mock semantics as a2a (duplicated, not imported) +│ ├── scheduler/ # DAG state machine for SLIM path +│ └── client/ # slim-bindings-go p2p + multicast group client +├── tools/ +│ └── report_dashboard.go +├── plans/ +│ └── domains/ # hand-authored plans only (committed) +│ ├── k8s-incident-response.yaml +│ ├── urban-traffic-reroute.yaml +│ └── mobile-nav-assistant.yaml +├── tests/ +│ ├── suite_test.go +│ ├── dag_benchmark_test.go +│ └── helpers_test.go +├── templates/ +└── reports/ +``` + +**Separation rule:** `a2a/` must not import `slim/` or `slim-bindings-go`. `slim/` must not import `a2a/` or `a2a-go`. Both import only `internal/plan`. Mock executor logic may be duplicated intentionally to keep stacks independent (or extracted later behind build tags — not in v1). + +Reuse patterns from: + +- Metrics/reporting: `[benchmarks/agntcy-slim/tests/benchmark_models_test.go](benchmarks/agntcy-slim/tests/benchmark_models_test.go)`, `[benchmark_suite_test.go](benchmarks/agntcy-slim/tests/benchmark_suite_test.go)` +- SLIM lifecycle: `[benchmarks/agntcy-slim/tests/config/local-slim.yaml](benchmarks/agntcy-slim/tests/config/local-slim.yaml)` + `startLocalSlimStack()` helpers +- A2A gRPC server/client: `[integrations/agntcy-a2a/fixtures/go-jsonrpc-server/main.go](integrations/agntcy-a2a/fixtures/go-jsonrpc-server/main.go)` + +Dependencies (new `go.mod` under benchmark package): + +- `github.com/a2aproject/a2a-go/v2` (already pinned in `[integrations/go.mod](integrations/go.mod)`) +- `github.com/agntcy/slim-bindings-go` (same version as `[benchmarks/agntcy-slim/tests/echo-client/go.mod](benchmarks/agntcy-slim/tests/echo-client/go.mod)`) +- `gopkg.in/yaml.v3`, Ginkgo/Gomega, Gonum (for CI stats, optional) + +## YAML plan schema + +Each plan file describes **named agents**, **named tasks** (domain-realistic labels), dependencies, and timing. Tasks mock real work via `completionTimeSec` / `maxCompletionTimeSec` / canned `output`. + +### Schema (unchanged structure, concrete names) + +```yaml +apiVersion: bench.agntcy.io/v1 +kind: ExecutionPlan +metadata: + name: k8s-incident-response + domain: kubernetes-troubleshooting + description: Parallel diagnosis of a failing deployment; branches abort if root cause found early +spec: + defaults: + completionTimeSec: 0.3 + maxCompletionTimeSec: 2.0 + maxRetries: 1 +agents: + - id: cluster-scout + slimName: agntcy/bench/cluster-scout + a2aPort: 9101 + - id: log-miner + slimName: agntcy/bench/log-miner + a2aPort: 9102 + - id: metrics-analyst + slimName: agntcy/bench/metrics-analyst + a2aPort: 9103 + # ... +tasks: + - id: fetch-pod-status + name: "Fetch pod status for checkout-api" + agent: cluster-scout + dependsOn: [] + completionTimeSec: 0.25 + maxCompletionTimeSec: 1.0 + output: "phase=CrashLoopBackOff replicas=3/5" + - id: scrape-container-logs + name: "Scrape last 500 lines from crashing pod" + agent: log-miner + dependsOn: [fetch-pod-status] + completionTimeSec: 0.6 + maxCompletionTimeSec: 1.5 + injectFailure: false + output: "OOMKilled exit=137" + # ... +contextUpdates: # optional — triggers SLIM multicast vs A2A sequential push + - afterTask: fetch-pod-status + payload: "hypothesis=memory-pressure priority=high" + targetAgents: [log-miner, metrics-analyst, event-correlator] +``` + +Fields: +- `agents[].id` — logical role name used in task assignments +- `agents[].slimName` — SLIM dataplane identity (`org/group/app`) +- `agents[].a2aPort` — gRPC port for A2A agent card (A2A impl only) +- `tasks[].name` — human-readable task label (reporting + realism) +- `contextUpdates[]` — models mid-plan context shifts (key SLIM sync/context benchmark axis) + +### Three hand-authored domain plans (committed in `plans/domains/`) + +#### 1. `k8s-incident-response.yaml` — K8s cluster troubleshooting + +**Scenario:** `checkout-api` deployment degraded; agents diagnose in parallel; if OOM root cause confirmed early, cancel speculative branches (network partition check, node drain simulation). + +| Agent | Role | +|-------|------| +| `cluster-scout` | Pod/status/endpoint discovery | +| `log-miner` | Container log extraction | +| `metrics-analyst` | Prometheus metric queries | +| `event-correlator` | K8s event timeline merge | +| `network-prober` | Service mesh / DNS checks | +| `runbook-executor` | Remediation steps (scale, rollback) | + +Example tasks: `fetch-pod-status`, `scrape-container-logs`, `query-memory-metrics`, `correlate-oom-events`, `probe-service-mesh`, `simulate-rollback`, `apply-memory-limit-patch`. DAG depth ~4, fan-out up to 4 parallel probes; one injected failure path on `probe-service-mesh` to exercise cancel propagation. + +#### 2. `urban-traffic-reroute.yaml` — Realtime traffic incident response + +**Scenario:** Highway accident detected; traffic agents recompute routes, update signals, notify transit; if primary corridor reopening ETA drops, cancel stale detour plans. + +| Agent | Role | +|-------|------| +| `incident-detector` | Ingest camera/sensor feed | +| `flow-modeler` | Traffic flow simulation | +| `route-planner` | Dynamic route recalculation | +| `signal-coordinator` | Intersection timing adjustments | +| `transit-dispatcher` | Bus/rail reroute | +| `alert-broadcaster` | Waze/511-style push | + +Example tasks: `detect-lane-closure`, `estimate-queue-length`, `recalc-northbound-routes`, `recalc-southbound-routes`, `adjust-signal-phasing`, `reroute-bus-line-42`, `publish-driver-alerts`. High parallel fan-out (N/S/E/W route recalc); context update when `estimate-queue-length` revises ETA. + +#### 3. `mobile-nav-assistant.yaml` — Android voice assistant (navigate + search) + +**Scenario:** User on phone says *"Take me to the nearest open pharmacy that's on the way to the airport"*; agents parse intent, search POIs, plan route, refine as user moves; if user changes priority ("actually, coffee first"), obsolete search/nav branches cancel. + +| Agent | Role | +|-------|------| +| `intent-parser` | NLU / slot extraction from Android input | +| `poi-searcher` | Places API search (pharmacy, coffee) | +| `route-engine` | Turn-by-turn path computation | +| `traffic-watcher` | Live ETA adjustments | +| `ui-composer` | Android notification / map tile updates | +| `session-tracker` | User location + command history sync | + +Example tasks: `parse-voice-command`, `search-pharmacy-near-route`, `search-coffee-near-route`, `compute-route-to-airport-via-poi`, `refine-route-live-traffic`, `render-map-overlay`, `push-voice-prompt`. Context update when user amends command mid-flight (`priority=coffee-first`); multiple parallel POI searches become obsolete on context change. + +Plan scale (agent count, task count, fan-out, failure injection) is defined **directly in each YAML file**. Add or edit plans by hand; no generator. + +## Agent design (two separate binaries, identical mock semantics) + +### `a2a/cmd/agent` — A2A-only generic worker + +``` +a2a-agent --agent-id=cluster-scout --grpc-port=9101 --card-port=8081 +``` + +- Implements `a2asrv` executor: parse JSON task payload (`taskId`, `completionTimeSec`, `maxCompletionTimeSec`, `context`) +- Mock execution: sleep, enforce deadline, honour cancel messages, return canned `output` +- Exposes gRPC via `a2agrpc.NewHandler`; agent card at `/.well-known/agent-card.json` +- **No SLIM imports** + +All agents share this one binary; they differ only by `--agent-id`, ports, and SLIM name mapping comes from plan YAML (SLIM agents use `slimName` field). + +### `slim/cmd/agent` — SLIM-only generic worker + +``` +slim-agent --slim-name=agntcy/bench/cluster-scout --endpoint=127.0.0.1:46357 +``` + +- Registers slimrpc `TaskDAG` service from `slim/proto/taskdag/v1/taskdag.proto` +- Same mock execution semantics as A2A agent (duplicated code in `slim/internal/executor`) +- Subscribes to `slimName` on dataplane +- **No A2A imports** + +## Runners (two separate DAG drivers) + +Each runner is a **standalone program** with its own scheduler — no shared `Transport` interface. + +### `a2a/cmd/runner` + +1. Load plan via `internal/plan` +2. Spawn `a2a-agent` processes for each agent in plan +3. Run DAG scheduler (`a2a/internal/scheduler`): + - Execute ready tasks via sequential/per-task A2A gRPC `SendMessage` + - On `contextUpdates`: **loop** `SendMessage` to each target agent with updated payload + - On failure/timeout: **loop** cancel RPCs to each affected agent + - Track sync barrier latency as time to complete all sequential round-trips +4. Write `RunMetrics` JSON to stdout / file + +### `slim/cmd/runner` + +1. Load same plan via `internal/plan` +2. Start local SLIM dataplane (or connect to existing) +3. Spawn `slim-agent` processes for each agent +4. Run DAG scheduler (`slim/internal/scheduler`): + - Execute ready tasks via point-to-point slimrpc unary + - On `contextUpdates`: **multicast** `PushContext` to group channel of target agents + - On phase transitions requiring ack: **multicast** `SyncPhase`, aggregate stream responses + - On failure/timeout: **multicast** `CancelTasks` + `NotifyFailure` +5. Write same-shape `RunMetrics` JSON + +Proto (`slim/proto/taskdag/v1/taskdag.proto`): + +```protobuf +service TaskDAG { + rpc ExecuteTask(ExecuteTaskRequest) returns (ExecuteTaskResponse); + rpc CancelTask(CancelTaskRequest) returns (CancelTaskResponse); + rpc PushContext(ContextUpdate) returns (ContextAck); +} +service TaskDAGGroup { + rpc PushContextMulticast(ContextUpdate) returns (stream MulticastContextAck); + rpc SyncPhase(PhaseSyncRequest) returns (stream MulticastPhaseAck); + rpc CancelTasks(CancelTasksRequest) returns (stream MulticastCancelResult); + rpc NotifyFailure(NotifyFailureRequest) returns (stream MulticastNotifyResult); +} +``` + +## Metrics schema + +Both runners emit the **same JSON/TSV shape** (defined in `tests/` or a tiny shared `internal/metrics` struct-only package with no runtime logic — acceptable if needed for report aggregation only). Fields: + +| Field | Description | +| ----- | ----------- | +| `plan_name`, `domain` | Plan id and domain label | +| `implementation` | `a2a-grpc` or `slim-multicast` | +| `agents`, `tasks` | Plan scale | +| `total_wall_clock_ms` | End-to-end plan execution | +| `tasks_completed`, `tasks_failed`, `tasks_timed_out` | Outcome counts | +| `tasks_cancelled`, `obsolete_tasks_completed` | Wasted work after failure/context change | +| `retries_attempted`, `retries_succeeded` | Retry stats | +| `context_push_ms` | Time to deliver context update to all target agents | +| `sync_barrier_ms` | Time for all agents to ack phase sync | +| `cancel_propagation_ms` | Failure → all cancels acked | +| `execute_rpc_count`, `multicast_rpc_count`, `sequential_rpc_count` | RPC overhead breakdown | +| `makespan_ms` | Critical-path duration (successful runs) | + +Comparison report auto-computes per-plan deltas: `(a2a - slim) / a2a` for wall clock, context push, sync barrier, cancel propagation, obsolete tasks. + +## Taskfile and execution + +`[benchmarks/agntcy-multi-agent/Taskfile.yml](benchmarks/agntcy-multi-agent/Taskfile.yml)`: + + +| Task | Purpose | +| -------------------------- | --------------------------------------------------------- | +| `deps:slim-bindings-setup` | Reuse slim benchmark setup | +| `benchmark:suite` | All plans in `plans/domains/` × both implementations | +| `benchmark:ci:smoke` | 3 domain plans × both implementations (CI subset) | +| `benchmark:report` | Build HTML dashboard | + + +Configurable env vars: + +- `PLAN_DIR=plans/domains` (default; all hand-authored plans) +- `IMPLEMENTATIONS=a2a,slim` +- `RUN_BENCHMARK_DAG=1` (gate like `SLIM_RUN_BENCHMARK_SUITE`) + +Example local run: + +```bash +task benchmarks:multi-agent:benchmark:ci:smoke RUN_BENCHMARK_DAG=1 +task benchmarks:multi-agent:benchmark:suite RUN_BENCHMARK_DAG=1 +``` + +## Test suite (Ginkgo) + +`[tests/dag_benchmark_test.go](benchmarks/agntcy-multi-agent/tests/dag_benchmark_test.go)`: + +- Label: `benchmark-dag` +- Skip unless `RUN_BENCHMARK_DAG=1` +- For each plan in `PLAN_DIR`: + 1. Run `a2a-runner --plan=...` → collect metrics + 2. Run `slim-runner --plan=...` (start SLIM stack first) → collect metrics +- Assert all three domain plans complete on both implementations; report deltas (no hard latency assertions) + +## CI integration (phase 2) + +Add workflow modeled on `[.github/workflows/test-benchmarks-slim.yaml](.github/workflows/test-benchmarks-slim.yaml)`: + +- Path filter: `benchmarks/agntcy-multi-agent/`** +- Job runs `benchmark:ci:smoke` only (small plan, ~2 min) +- Publish artifact to test reports site via `[.github/workflows/publish-test-reports-pages.yaml](.github/workflows/publish-test-reports-pages.yaml)` + +## Expected outcome + +Running the domain plans (especially `urban-traffic-reroute` and `mobile-nav-assistant` with frequent context updates) should show: + +- Similar `makespan_ms` on **successful** runs with no context churn (happy path parity) +- Lower `context_push_ms` and `sync_barrier_ms` with **SLIM** when `contextUpdates` fan out to many agents +- Lower `cancel_propagation_ms` and `obsolete_tasks_completed` with **SLIM** when failures abort parallel branches (K8s incident plan) +- Lower `total_wall_clock_ms` for SLIM on context-heavy and failure-heavy plans + +**Narrative:** SLIM Multicast RPC enables **fast sync**, **fast shared context propagation**, and **fast failure propagation** — all critical when many agents execute interdependent realtime tasks and obsolete work must be abandoned quickly. + +## Implementation order + +### Phase 1 — Plans (YAML only, no Go) + +1. Finalize schema in plan doc / `internal/plan` types +2. Hand-author 3 domain plans in `plans/domains/` with full agent lists, task DAGs, timings, `contextUpdates`, and injected failure paths + +### Phase 2 — Implementation + +3. **`internal/plan`** — YAML load, validation, DAG helpers +4. **`a2a/` stack** — agent + runner + scheduler; end-to-end on domain plans +5. **`slim/` stack** — proto, agent + runner + scheduler with multicast; same plans end-to-end +6. **Context sync + failure propagation** — wire `contextUpdates`, cancel obsolete tasks, retries +7. **Metrics + TSV + templates + dashboard** +8. **Taskfile + Ginkgo suite + CI smoke** + +## Out of scope (initial version) + +- **`plangen` / automated plan scaling** — plans are hand-authored; add new YAML files manually +- K8s/Kind deployment (design allows later: same agent/runner binaries in Helm) +- Python `slima2a` agents +- A2A-over-SLIM third transport leg (could be added later as `slim-a2a`) +- Directory/registry integration (`agntcy-dir`) — unrelated to this comparison + diff --git a/benchmarks/agntcy-multi-agent/go.mod b/benchmarks/agntcy-multi-agent/go.mod new file mode 100644 index 00000000..e6de3725 --- /dev/null +++ b/benchmarks/agntcy-multi-agent/go.mod @@ -0,0 +1,5 @@ +module github.com/agntcy/csit/benchmarks/agntcy-multi-agent + +go 1.25.0 + +require gopkg.in/yaml.v3 v3.0.1 diff --git a/benchmarks/agntcy-multi-agent/go.sum b/benchmarks/agntcy-multi-agent/go.sum new file mode 100644 index 00000000..a62c313c --- /dev/null +++ b/benchmarks/agntcy-multi-agent/go.sum @@ -0,0 +1,4 @@ +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/benchmarks/agntcy-multi-agent/plans/README.md b/benchmarks/agntcy-multi-agent/plans/README.md new file mode 100644 index 00000000..8ea79c5e --- /dev/null +++ b/benchmarks/agntcy-multi-agent/plans/README.md @@ -0,0 +1,57 @@ +# Execution plans (Phase 1) + +Hand-authored YAML DAG plans for the SLIM vs A2A multi-agent benchmark. +Each plan defines named agents, interdependent tasks (mocked via timing fields), +optional failure injection, and context updates that exercise sync / context +sharing / failure propagation. + +## Plans + +| File | Domain | Agents | Tasks | Notes | +|------|--------|--------|-------|-------| +| [k8s-incident-response.yaml](domains/k8s-incident-response.yaml) | Kubernetes troubleshooting | 6 | 16 | OOM diagnosis; `probe-service-mesh` injects failure; 2 context updates | +| [urban-traffic-reroute.yaml](domains/urban-traffic-reroute.yaml) | Realtime traffic | 6 | 14 | I-5 closure; N/S/E/W parallel route recalc; detour cancel on ETA revision | +| [mobile-nav-assistant.yaml](domains/mobile-nav-assistant.yaml) | Android voice navigation | 6 | 12 | Pharmacy→airport intent; coffee-first amendment obsoletes pharmacy branch | + +## Schema + +```yaml +apiVersion: bench.agntcy.io/v1 +kind: ExecutionPlan +metadata: + name: + domain: + description: +spec: + defaults: + completionTimeSec: + maxCompletionTimeSec: + maxRetries: +agents: + - id: + slimName: agntcy/bench/ + a2aPort: # unique within plan; k8s=91xx, traffic=92xx, mobile=93xx +tasks: + - id: + name: + agent: + dependsOn: [, ...] + completionTimeSec: + maxCompletionTimeSec: + injectFailure: # optional; default false + output: +contextUpdates: # optional + - afterTask: + payload: + targetAgents: [, ...] +``` + +## Port allocation + +Ports are unique per plan to allow isolated local runs: + +- **k8s-incident-response:** A2A gRPC `9101`–`9106` +- **urban-traffic-reroute:** A2A gRPC `9201`–`9206` +- **mobile-nav-assistant:** A2A gRPC `9301`–`9306` + +SLIM identities use `agntcy/bench/` from each plan's `agents[].slimName`. diff --git a/benchmarks/agntcy-multi-agent/plans/domains/k8s-incident-response.yaml b/benchmarks/agntcy-multi-agent/plans/domains/k8s-incident-response.yaml new file mode 100644 index 00000000..b99f09e5 --- /dev/null +++ b/benchmarks/agntcy-multi-agent/plans/domains/k8s-incident-response.yaml @@ -0,0 +1,173 @@ +apiVersion: bench.agntcy.io/v1 +kind: ExecutionPlan +metadata: + name: k8s-incident-response + domain: kubernetes-troubleshooting + description: > + checkout-api deployment in CrashLoopBackOff; agents diagnose in parallel. + OOM root cause confirmed early cancels speculative network partition and + node-drain branches. probe-service-mesh injects failure to exercise abort propagation. +spec: + defaults: + completionTimeSec: 0.3 + maxCompletionTimeSec: 2.0 + maxRetries: 1 + +agents: + - id: cluster-scout + slimName: agntcy/bench/cluster-scout + a2aPort: 9101 + - id: log-miner + slimName: agntcy/bench/log-miner + a2aPort: 9102 + - id: metrics-analyst + slimName: agntcy/bench/metrics-analyst + a2aPort: 9103 + - id: event-correlator + slimName: agntcy/bench/event-correlator + a2aPort: 9104 + - id: network-prober + slimName: agntcy/bench/network-prober + a2aPort: 9105 + - id: runbook-executor + slimName: agntcy/bench/runbook-executor + a2aPort: 9106 + +tasks: + - id: fetch-pod-status + name: Fetch pod status for checkout-api deployment + agent: cluster-scout + dependsOn: [] + completionTimeSec: 0.25 + maxCompletionTimeSec: 1.0 + output: "deployment=checkout-api phase=CrashLoopBackOff replicas=3/5 namespace=prod" + + - id: list-recent-events + name: List Kubernetes events for checkout-api pods + agent: event-correlator + dependsOn: [fetch-pod-status] + completionTimeSec: 0.35 + maxCompletionTimeSec: 1.2 + output: "events=47 warning=BackOff reason=FailedScheduling count=12" + + - id: scrape-container-logs + name: Scrape last 500 lines from crashing checkout-api pod + agent: log-miner + dependsOn: [fetch-pod-status] + completionTimeSec: 0.55 + maxCompletionTimeSec: 1.5 + output: "container=checkout-api OOMKilled exit=137 lastLine=java.lang.OutOfMemoryError" + + - id: query-memory-metrics + name: Query Prometheus memory usage for checkout-api + agent: metrics-analyst + dependsOn: [fetch-pod-status] + completionTimeSec: 0.4 + maxCompletionTimeSec: 1.3 + output: "metric=container_memory_working_set_bytes value=512Mi limit=512Mi breached=true" + + - id: query-cpu-metrics + name: Query Prometheus CPU throttling for checkout-api + agent: metrics-analyst + dependsOn: [fetch-pod-status] + completionTimeSec: 0.35 + maxCompletionTimeSec: 1.2 + output: "metric=container_cpu_cfs_throttled_seconds_total value=0.02 throttled=false" + + - id: probe-dns-resolution + name: Probe cluster DNS resolution for checkout-api service + agent: network-prober + dependsOn: [fetch-pod-status] + completionTimeSec: 0.45 + maxCompletionTimeSec: 1.4 + output: "service=checkout-api.svc.cluster.local resolved=10.96.142.88 latency_ms=3" + + - id: probe-service-mesh + name: Probe Istio sidecar connectivity for checkout-api pod + agent: network-prober + dependsOn: [fetch-pod-status] + completionTimeSec: 0.5 + maxCompletionTimeSec: 1.0 + injectFailure: true + output: "sidecar=istio-proxy status=503 upstream=checkout-api:8080" + + - id: analyze-gc-logs + name: Analyze JVM GC patterns in container logs + agent: log-miner + dependsOn: [scrape-container-logs] + completionTimeSec: 0.5 + maxCompletionTimeSec: 1.6 + output: "gc=G1Full frequent=true heap=512Mi postGC=498Mi" + + - id: correlate-oom-events + name: Correlate OOMKilled events with pod restart timeline + agent: event-correlator + dependsOn: [scrape-container-logs, list-recent-events] + completionTimeSec: 0.45 + maxCompletionTimeSec: 1.5 + output: "correlation=strong restarts=18 window=15m cause=OOMKilled" + + - id: check-node-capacity + name: Check node allocatable memory on worker pool + agent: cluster-scout + dependsOn: [query-memory-metrics] + completionTimeSec: 0.4 + maxCompletionTimeSec: 1.5 + output: "node=worker-3 allocatable_memory=32Gi pressure=low" + + - id: simulate-node-drain + name: Simulate cordon and drain of suspect worker node + agent: runbook-executor + dependsOn: [check-node-capacity, probe-service-mesh] + completionTimeSec: 0.8 + maxCompletionTimeSec: 2.5 + output: "node=worker-3 cordoned=false simulation=aborted" + + - id: confirm-root-cause-oom + name: Confirm memory limit breach as root cause + agent: event-correlator + dependsOn: [correlate-oom-events, analyze-gc-logs, query-memory-metrics] + completionTimeSec: 0.3 + maxCompletionTimeSec: 1.0 + output: "root_cause=memory_limit_breach confidence=0.97 action=raise_limit" + + - id: apply-memory-limit-patch + name: Patch deployment memory limit from 512Mi to 1Gi + agent: runbook-executor + dependsOn: [confirm-root-cause-oom] + completionTimeSec: 0.6 + maxCompletionTimeSec: 2.0 + output: "deployment=checkout-api memory_limit=1Gi patch=applied rollout=started" + + - id: simulate-rollback + name: Simulate rollback to previous checkout-api revision + agent: runbook-executor + dependsOn: [probe-service-mesh] + completionTimeSec: 0.7 + maxCompletionTimeSec: 2.0 + output: "revision=checkout-api-42 rollback=skipped reason=upstream_probe_failed" + + - id: verify-pod-health + name: Verify checkout-api pods reach Running state + agent: cluster-scout + dependsOn: [apply-memory-limit-patch] + completionTimeSec: 0.5 + maxCompletionTimeSec: 2.5 + output: "deployment=checkout-api phase=Running replicas=5/5 ready=true" + + - id: publish-incident-summary + name: Publish incident timeline to status page + agent: event-correlator + dependsOn: [verify-pod-health] + completionTimeSec: 0.25 + maxCompletionTimeSec: 1.0 + output: "incident=INC-2847 status=resolved duration=23m root_cause=OOMKilled" + +contextUpdates: + - afterTask: fetch-pod-status + payload: "hypothesis=memory-pressure priority=high focus=checkout-api namespace=prod" + targetAgents: [log-miner, metrics-analyst, event-correlator, network-prober] + + - afterTask: confirm-root-cause-oom + payload: "root_cause=memory_limit_breach cancel_speculative=true abort=node-drain,mesh-rollback" + targetAgents: [network-prober, runbook-executor, cluster-scout] diff --git a/benchmarks/agntcy-multi-agent/plans/domains/mobile-nav-assistant.yaml b/benchmarks/agntcy-multi-agent/plans/domains/mobile-nav-assistant.yaml new file mode 100644 index 00000000..7f36f55e --- /dev/null +++ b/benchmarks/agntcy-multi-agent/plans/domains/mobile-nav-assistant.yaml @@ -0,0 +1,140 @@ +apiVersion: bench.agntcy.io/v1 +kind: ExecutionPlan +metadata: + name: mobile-nav-assistant + domain: mobile-voice-navigation + description: > + Android user asks for pharmacy on the way to the airport; agents parse intent, + search POIs, and compute routes. Mid-flight command change to coffee-first + obsoletes pharmacy branch via context update. +spec: + defaults: + completionTimeSec: 0.3 + maxCompletionTimeSec: 2.0 + maxRetries: 1 + +agents: + - id: intent-parser + slimName: agntcy/bench/intent-parser + a2aPort: 9301 + - id: poi-searcher + slimName: agntcy/bench/poi-searcher + a2aPort: 9302 + - id: route-engine + slimName: agntcy/bench/route-engine + a2aPort: 9303 + - id: traffic-watcher + slimName: agntcy/bench/traffic-watcher + a2aPort: 9304 + - id: ui-composer + slimName: agntcy/bench/ui-composer + a2aPort: 9305 + - id: session-tracker + slimName: agntcy/bench/session-tracker + a2aPort: 9306 + +tasks: + - id: parse-voice-command + name: Parse voice command for pharmacy stop en route to airport + agent: intent-parser + dependsOn: [] + completionTimeSec: 0.25 + maxCompletionTimeSec: 1.0 + output: "intent=navigate_with_stop destination=SEA_airport stop=pharmacy open_now=true" + + - id: sync-user-location + name: Sync current GPS location and heading from Android device + agent: session-tracker + dependsOn: [parse-voice-command] + completionTimeSec: 0.15 + maxCompletionTimeSec: 0.6 + output: "lat=47.6062 lon=-122.3321 heading=145 speed_kmh=32 accuracy_m=8" + + - id: geocode-airport-destination + name: Geocode Seattle-Tacoma International Airport destination + agent: poi-searcher + dependsOn: [parse-voice-command] + completionTimeSec: 0.2 + maxCompletionTimeSec: 0.8 + output: "poi=SEA_airport lat=47.4502 lon=-122.3088 place_id=ChIJSeaTac" + + - id: search-pharmacy-near-route + name: Search open pharmacies along route to airport + agent: poi-searcher + dependsOn: [parse-voice-command, sync-user-location] + completionTimeSec: 0.55 + maxCompletionTimeSec: 1.8 + output: "poi=CVS_Queen_Anne lat=47.6234 lon=-122.3571 open=true detour_min=4" + + - id: search-coffee-near-route + name: Search coffee shops along route to airport + agent: poi-searcher + dependsOn: [parse-voice-command, sync-user-location] + completionTimeSec: 0.5 + maxCompletionTimeSec: 1.8 + output: "poi=Starbucks_Belltown lat=47.6145 lon=-122.3498 open=true detour_min=2" + + - id: rank-stop-candidates + name: Rank stop candidates by detour time and user preferences + agent: intent-parser + dependsOn: [search-pharmacy-near-route, search-coffee-near-route] + completionTimeSec: 0.3 + maxCompletionTimeSec: 1.0 + output: "ranked=pharmacy,coffee preferred=pharmacy detour_budget_min=8" + + - id: compute-route-to-airport-via-pharmacy + name: Compute turn-by-turn route to airport via CVS Queen Anne + agent: route-engine + dependsOn: [search-pharmacy-near-route, geocode-airport-destination] + completionTimeSec: 0.6 + maxCompletionTimeSec: 2.0 + output: "route_id=R-pharm-01 distance_km=28.4 eta_min=34 stops=1 via=CVS_Queen_Anne" + + - id: compute-route-to-airport-via-coffee + name: Compute turn-by-turn route to airport via Starbucks Belltown + agent: route-engine + dependsOn: [search-coffee-near-route, geocode-airport-destination] + completionTimeSec: 0.55 + maxCompletionTimeSec: 2.0 + output: "route_id=R-coffee-01 distance_km=27.1 eta_min=31 stops=1 via=Starbucks_Belltown" + + - id: refine-route-live-traffic + name: Refine active route with live traffic and incident data + agent: traffic-watcher + dependsOn: [compute-route-to-airport-via-pharmacy, compute-route-to-airport-via-coffee] + completionTimeSec: 0.45 + maxCompletionTimeSec: 1.5 + output: "route_id=R-coffee-01 eta_min=33 delay_min=+2 incident=I-5_slowdown" + + - id: render-map-overlay + name: Render map overlay and stop markers on Android UI + agent: ui-composer + dependsOn: [refine-route-live-traffic, rank-stop-candidates] + completionTimeSec: 0.35 + maxCompletionTimeSec: 1.2 + output: "overlay=turn_by_turn markers=2 stop=coffee theme=night_mode" + + - id: push-voice-prompt + name: Push voice prompt confirming coffee stop and updated ETA + agent: ui-composer + dependsOn: [render-map-overlay] + completionTimeSec: 0.2 + maxCompletionTimeSec: 0.8 + output: "tts=Routing via Starbucks Belltown. ETA 33 minutes. Say change stop to adjust." + + - id: update-session-history + name: Update session history with amended navigation intent + agent: session-tracker + dependsOn: [push-voice-prompt] + completionTimeSec: 0.15 + maxCompletionTimeSec: 0.6 + output: "session=android-7f3a intent_history=2 priority=coffee-first destination=SEA_airport" + +contextUpdates: + - afterTask: parse-voice-command + payload: "user_amendment=actually_coffee_first priority=coffee-first cancel=pharmacy_branch" + targetAgents: [poi-searcher, route-engine, intent-parser, ui-composer] + + - afterTask: refine-route-live-traffic + payload: "active_route=R-coffee-01 eta_min=33 sync=ui,voice session=android-7f3a" + targetAgents: [ui-composer, session-tracker, traffic-watcher] diff --git a/benchmarks/agntcy-multi-agent/plans/domains/urban-traffic-reroute.yaml b/benchmarks/agntcy-multi-agent/plans/domains/urban-traffic-reroute.yaml new file mode 100644 index 00000000..5856b057 --- /dev/null +++ b/benchmarks/agntcy-multi-agent/plans/domains/urban-traffic-reroute.yaml @@ -0,0 +1,156 @@ +apiVersion: bench.agntcy.io/v1 +kind: ExecutionPlan +metadata: + name: urban-traffic-reroute + domain: realtime-traffic-management + description: > + Highway I-5 northbound lane closure after accident; agents recompute routes, + adjust signals, and reroute transit. Corridor reopening ETA revision cancels + stale detour plans via context update. +spec: + defaults: + completionTimeSec: 0.35 + maxCompletionTimeSec: 2.0 + maxRetries: 1 + +agents: + - id: incident-detector + slimName: agntcy/bench/incident-detector + a2aPort: 9201 + - id: flow-modeler + slimName: agntcy/bench/flow-modeler + a2aPort: 9202 + - id: route-planner + slimName: agntcy/bench/route-planner + a2aPort: 9203 + - id: signal-coordinator + slimName: agntcy/bench/signal-coordinator + a2aPort: 9204 + - id: transit-dispatcher + slimName: agntcy/bench/transit-dispatcher + a2aPort: 9205 + - id: alert-broadcaster + slimName: agntcy/bench/alert-broadcaster + a2aPort: 9206 + +tasks: + - id: detect-lane-closure + name: Detect lane closure from camera and WIM sensor feed + agent: incident-detector + dependsOn: [] + completionTimeSec: 0.2 + maxCompletionTimeSec: 0.8 + output: "highway=I-5 direction=northbound lanes_closed=2 location=mile-172 accident=true" + + - id: estimate-queue-length + name: Estimate queue length and delay on affected corridor + agent: flow-modeler + dependsOn: [detect-lane-closure] + completionTimeSec: 0.45 + maxCompletionTimeSec: 1.5 + output: "queue_km=4.2 delay_min=38 reopening_eta_min=45 corridor=I-5-N" + + - id: model-traffic-impedance + name: Model traffic impedance across downtown grid + agent: flow-modeler + dependsOn: [estimate-queue-length] + completionTimeSec: 0.5 + maxCompletionTimeSec: 1.8 + output: "impedance_matrix=12x12 hotspots=3 max_delay_min=22" + + - id: plan-detour-via-highway-99 + name: Plan corridor-wide detour via Highway 99 + agent: route-planner + dependsOn: [estimate-queue-length] + completionTimeSec: 0.7 + maxCompletionTimeSec: 2.5 + output: "detour=WA-99 added_distance_km=18 eta_min=52 status=proposed" + + - id: plan-detour-via-surface-streets + name: Plan local detour via Aurora and surface streets + agent: route-planner + dependsOn: [estimate-queue-length] + completionTimeSec: 0.65 + maxCompletionTimeSec: 2.5 + output: "detour=aurora-ave added_distance_km=9 eta_min=41 status=proposed" + + - id: recalc-northbound-routes + name: Recalculate northbound dynamic routes for active fleet + agent: route-planner + dependsOn: [model-traffic-impedance] + completionTimeSec: 0.55 + maxCompletionTimeSec: 1.8 + output: "direction=northbound routes_updated=12400 avg_delay_delta_min=+6" + + - id: recalc-southbound-routes + name: Recalculate southbound dynamic routes for active fleet + agent: route-planner + dependsOn: [model-traffic-impedance] + completionTimeSec: 0.5 + maxCompletionTimeSec: 1.8 + output: "direction=southbound routes_updated=11800 avg_delay_delta_min=+4" + + - id: recalc-eastbound-routes + name: Recalculate eastbound dynamic routes for active fleet + agent: route-planner + dependsOn: [model-traffic-impedance] + completionTimeSec: 0.48 + maxCompletionTimeSec: 1.8 + output: "direction=eastbound routes_updated=9200 avg_delay_delta_min=+3" + + - id: recalc-westbound-routes + name: Recalculate westbound dynamic routes for active fleet + agent: route-planner + dependsOn: [model-traffic-impedance] + completionTimeSec: 0.52 + maxCompletionTimeSec: 1.8 + output: "direction=westbound routes_updated=8900 avg_delay_delta_min=+3" + + - id: adjust-signal-phasing + name: Adjust intersection signal phasing on alternate corridors + agent: signal-coordinator + dependsOn: [recalc-northbound-routes, recalc-southbound-routes] + completionTimeSec: 0.4 + maxCompletionTimeSec: 1.5 + output: "intersections=37 green_extension_sec=12 corridor_bias=north-south" + + - id: reroute-bus-line-42 + name: Reroute King County Metro bus line 42 + agent: transit-dispatcher + dependsOn: [recalc-eastbound-routes, plan-detour-via-surface-streets] + completionTimeSec: 0.45 + maxCompletionTimeSec: 1.6 + output: "line=42 stops_bypassed=6 alternate=aurora-ave passengers_notified=820" + + - id: reroute-rail-shuttle + name: Reroute Sound Transit shuttle feeders + agent: transit-dispatcher + dependsOn: [recalc-westbound-routes] + completionTimeSec: 0.42 + maxCompletionTimeSec: 1.6 + output: "shuttle=ST-Express-550 headway_min=8 alternate_hub=Northgate" + + - id: publish-driver-alerts + name: Publish Waze and 511 driver alerts for I-5 closure + agent: alert-broadcaster + dependsOn: [adjust-signal-phasing, reroute-bus-line-42, reroute-rail-shuttle] + completionTimeSec: 0.3 + maxCompletionTimeSec: 1.2 + output: "channels=waze,511 recipients=840000 message=I-5NB_lane_closure_m172" + + - id: sync-incident-dashboard + name: Sync traffic operations center incident dashboard + agent: incident-detector + dependsOn: [publish-driver-alerts] + completionTimeSec: 0.25 + maxCompletionTimeSec: 1.0 + output: "dashboard=TOC-West incident=TRF-9182 status=active mitigation=deployed" + +contextUpdates: + - afterTask: estimate-queue-length + payload: "reopening_eta_min=12 corridor=I-5-N cancel_detours=true priority=primary_corridor" + targetAgents: [route-planner, transit-dispatcher, signal-coordinator] + + - afterTask: model-traffic-impedance + payload: "phase=route_recalc sync=all_directions timestamp=2026-06-04T14:32:00Z" + targetAgents: [route-planner, signal-coordinator, alert-broadcaster] diff --git a/benchmarks/agntcy-multi-agent/tools/validate_plans/main.go b/benchmarks/agntcy-multi-agent/tools/validate_plans/main.go new file mode 100644 index 00000000..953a7a7e --- /dev/null +++ b/benchmarks/agntcy-multi-agent/tools/validate_plans/main.go @@ -0,0 +1,104 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "os" + "path/filepath" + "strings" + + "gopkg.in/yaml.v3" +) + +type plan struct { + Agents []struct { + ID string `yaml:"id"` + } `yaml:"agents"` + Tasks []struct { + ID string `yaml:"id"` + Agent string `yaml:"agent"` + DependsOn []string `yaml:"dependsOn"` + CompletionTimeSec float64 `yaml:"completionTimeSec"` + MaxCompletionTimeSec float64 `yaml:"maxCompletionTimeSec"` + } `yaml:"tasks"` + ContextUpdates []struct { + AfterTask string `yaml:"afterTask"` + TargetAgents []string `yaml:"targetAgents"` + } `yaml:"contextUpdates"` +} + +func main() { + dir := filepath.Join("..", "plans", "domains") + if len(os.Args) > 1 { + dir = os.Args[1] + } + + entries, err := os.ReadDir(dir) + if err != nil { + fmt.Fprintf(os.Stderr, "read dir: %v\n", err) + os.Exit(1) + } + + var errs []string + for _, e := range entries { + if !strings.HasSuffix(e.Name(), ".yaml") { + continue + } + path := filepath.Join(dir, e.Name()) + data, err := os.ReadFile(path) + if err != nil { + errs = append(errs, err.Error()) + continue + } + + var p plan + if err := yaml.Unmarshal(data, &p); err != nil { + errs = append(errs, fmt.Sprintf("%s: parse: %v", e.Name(), err)) + continue + } + + agents := map[string]bool{} + for _, a := range p.Agents { + agents[a.ID] = true + } + tasks := map[string]struct{}{} + for _, t := range p.Tasks { + tasks[t.ID] = struct{}{} + } + + for _, t := range p.Tasks { + if !agents[t.Agent] { + errs = append(errs, fmt.Sprintf("%s: task %s unknown agent %s", e.Name(), t.ID, t.Agent)) + } + for _, d := range t.DependsOn { + if _, ok := tasks[d]; !ok { + errs = append(errs, fmt.Sprintf("%s: task %s unknown dep %s", e.Name(), t.ID, d)) + } + } + if t.MaxCompletionTimeSec < t.CompletionTimeSec { + errs = append(errs, fmt.Sprintf("%s: task %s max < completion", e.Name(), t.ID)) + } + } + for _, cu := range p.ContextUpdates { + if _, ok := tasks[cu.AfterTask]; !ok { + errs = append(errs, fmt.Sprintf("%s: contextUpdate unknown afterTask %s", e.Name(), cu.AfterTask)) + } + for _, a := range cu.TargetAgents { + if !agents[a] { + errs = append(errs, fmt.Sprintf("%s: contextUpdate unknown agent %s", e.Name(), a)) + } + } + } + + fmt.Printf("%s: %d agents, %d tasks, %d contextUpdates\n", e.Name(), len(p.Agents), len(p.Tasks), len(p.ContextUpdates)) + } + + if len(errs) > 0 { + for _, e := range errs { + fmt.Println("ERROR:", e) + } + os.Exit(1) + } +} From 0c0680695801ee673b1ed523ba07acb301c7b945 Mon Sep 17 00:00:00 2001 From: Magyari Sandor Szilard Date: Fri, 19 Jun 2026 10:56:47 +0200 Subject: [PATCH 2/6] feat: experimenting with slim vs a2a Signed-off-by: Magyari Sandor Szilard --- ...slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md | 147 ++++-- .github/workflows/test-slim-vs-a2a.yaml | 55 +++ benchmarks/Taskfile.yml | 5 + benchmarks/agntcy-multi-agent/go.mod | 5 - benchmarks/agntcy-multi-agent/go.sum | 4 - .../tools/validate_plans/main.go | 104 ---- benchmarks/slim-vs-a2a/Taskfile.yml | 399 +++++++++++++++ benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go | 153 ++++++ benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go | 114 +++++ .../slim-vs-a2a/a2a/internal/client/client.go | 291 +++++++++++ .../a2a/internal/executor/executor.go | 97 ++++ .../a2a/internal/protocol/protocol.go | 54 ++ .../a2a/internal/scheduler/scheduler.go | 342 +++++++++++++ benchmarks/slim-vs-a2a/go.mod | 31 ++ benchmarks/slim-vs-a2a/go.sum | 101 ++++ .../slim-vs-a2a/internal/benchlog/benchlog.go | 122 +++++ .../internal/metrics/coordstats.go | 95 ++++ .../internal/metrics/coordstats_test.go | 39 ++ .../slim-vs-a2a/internal/metrics/metrics.go | 142 ++++++ .../slim-vs-a2a/internal/plan/generator.go | 231 +++++++++ .../internal/plan/generator_test.go | 46 ++ benchmarks/slim-vs-a2a/internal/plan/plan.go | 295 +++++++++++ .../plans/README.md | 0 .../plans/domains/k8s-incident-response.yaml | 0 .../plans/domains/mobile-nav-assistant.yaml | 0 .../plans/domains/urban-traffic-reroute.yaml | 0 .../sustainable-resource-10ag-10ms.yaml | 152 ++++++ .../sustainable-resource-10ag-20ms.yaml | 152 ++++++ .../sustainable-resource-17ag-10ms.yaml | 243 +++++++++ .../sustainable-resource-17ag-20ms.yaml | 243 +++++++++ .../sweeps/sustainable-resource-2ag-10ms.yaml | 48 ++ .../sweeps/sustainable-resource-2ag-20ms.yaml | 48 ++ .../sweeps/sustainable-resource-5ag-10ms.yaml | 87 ++++ .../sweeps/sustainable-resource-5ag-20ms.yaml | 87 ++++ benchmarks/slim-vs-a2a/slim/cmd/agent/main.go | 68 +++ .../slim-vs-a2a/slim/cmd/runner/main.go | 124 +++++ .../slim/internal/client/client.go | 462 ++++++++++++++++++ .../slim/internal/executor/executor.go | 100 ++++ .../slim/internal/protocol/protocol.go | 98 ++++ .../slim/internal/scheduler/scheduler.go | 341 +++++++++++++ .../slim/internal/server/handler.go | 25 + .../slim/internal/slimrpc/slimrpc.go | 9 + .../slim-vs-a2a/tests/slim_vs_a2a_test.go | 299 ++++++++++++ benchmarks/slim-vs-a2a/tests/suite_test.go | 16 + benchmarks/slim-vs-a2a/tools/gen_plan/main.go | 52 ++ benchmarks/slim-vs-a2a/tools/report/main.go | 241 +++++++++ .../slim-vs-a2a/tools/validate_plans/main.go | 28 ++ 47 files changed, 5641 insertions(+), 154 deletions(-) create mode 100644 .github/workflows/test-slim-vs-a2a.yaml delete mode 100644 benchmarks/agntcy-multi-agent/go.mod delete mode 100644 benchmarks/agntcy-multi-agent/go.sum delete mode 100644 benchmarks/agntcy-multi-agent/tools/validate_plans/main.go create mode 100644 benchmarks/slim-vs-a2a/Taskfile.yml create mode 100644 benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go create mode 100644 benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go create mode 100644 benchmarks/slim-vs-a2a/a2a/internal/client/client.go create mode 100644 benchmarks/slim-vs-a2a/a2a/internal/executor/executor.go create mode 100644 benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go create mode 100644 benchmarks/slim-vs-a2a/a2a/internal/scheduler/scheduler.go create mode 100644 benchmarks/slim-vs-a2a/go.mod create mode 100644 benchmarks/slim-vs-a2a/go.sum create mode 100644 benchmarks/slim-vs-a2a/internal/benchlog/benchlog.go create mode 100644 benchmarks/slim-vs-a2a/internal/metrics/coordstats.go create mode 100644 benchmarks/slim-vs-a2a/internal/metrics/coordstats_test.go create mode 100644 benchmarks/slim-vs-a2a/internal/metrics/metrics.go create mode 100644 benchmarks/slim-vs-a2a/internal/plan/generator.go create mode 100644 benchmarks/slim-vs-a2a/internal/plan/generator_test.go create mode 100644 benchmarks/slim-vs-a2a/internal/plan/plan.go rename benchmarks/{agntcy-multi-agent => slim-vs-a2a}/plans/README.md (100%) rename benchmarks/{agntcy-multi-agent => slim-vs-a2a}/plans/domains/k8s-incident-response.yaml (100%) rename benchmarks/{agntcy-multi-agent => slim-vs-a2a}/plans/domains/mobile-nav-assistant.yaml (100%) rename benchmarks/{agntcy-multi-agent => slim-vs-a2a}/plans/domains/urban-traffic-reroute.yaml (100%) create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-10ms.yaml create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-20ms.yaml create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-10ms.yaml create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-20ms.yaml create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-10ms.yaml create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-20ms.yaml create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-10ms.yaml create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-20ms.yaml create mode 100644 benchmarks/slim-vs-a2a/slim/cmd/agent/main.go create mode 100644 benchmarks/slim-vs-a2a/slim/cmd/runner/main.go create mode 100644 benchmarks/slim-vs-a2a/slim/internal/client/client.go create mode 100644 benchmarks/slim-vs-a2a/slim/internal/executor/executor.go create mode 100644 benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go create mode 100644 benchmarks/slim-vs-a2a/slim/internal/scheduler/scheduler.go create mode 100644 benchmarks/slim-vs-a2a/slim/internal/server/handler.go create mode 100644 benchmarks/slim-vs-a2a/slim/internal/slimrpc/slimrpc.go create mode 100644 benchmarks/slim-vs-a2a/tests/slim_vs_a2a_test.go create mode 100644 benchmarks/slim-vs-a2a/tests/suite_test.go create mode 100644 benchmarks/slim-vs-a2a/tools/gen_plan/main.go create mode 100644 benchmarks/slim-vs-a2a/tools/report/main.go create mode 100644 benchmarks/slim-vs-a2a/tools/validate_plans/main.go diff --git a/.cursor/plans/slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md b/.cursor/plans/slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md index 568f81e3..724168fb 100644 --- a/.cursor/plans/slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md +++ b/.cursor/plans/slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md @@ -1,13 +1,13 @@ --- -name: SLIM vs A2A DAG benchmark -overview: Design a new CSIT benchmark package with domain-specific YAML DAG plans (concrete agent/task names), two fully separate executors (A2A gRPC vs SLIM Multicast RPC), and comparable metrics demonstrating SLIM's advantages in fast sync, fast context sharing, and fast failure propagation. +name: slim-vs-a2a +overview: CSIT comparison package at benchmarks/slim-vs-a2a — domain YAML DAG plans, two fully separate executors (A2A gRPC vs SLIM Multicast RPC), comparable metrics, and a dedicated test-slim-vs-a2a CI workflow (Phase 3). todos: - id: plan-schema content: Define shared internal/plan YAML types, validation, and DAG helpers (only shared layer) status: pending - id: domain-plans content: "Phase 1 — hand-author 3 domain YAML plans in plans/domains/ (no plangen)" - status: pending + status: completed - id: a2a-impl content: Standalone A2A stack — cmd/a2a-agent, cmd/a2a-runner, internal/a2a/* (no SLIM imports) status: pending @@ -18,19 +18,36 @@ todos: content: Shared metrics schema + TSV/markdown reporting with sync/context/failure propagation fields status: pending - id: taskfile-suite - content: Wire Taskfile, Ginkgo suite, and CI smoke workflow following agntcy-slim patterns + content: "Phase 2 — Taskfile + local Ginkgo suite (no CI yet)" + status: pending + - id: ci-integration + content: "Phase 3 — dedicated test-slim-vs-a2a.yaml workflow, CI smoke, artifacts, pages publish" status: pending + - id: rename-project + content: "Rename benchmarks/agntcy-multi-agent → benchmarks/slim-vs-a2a (folder, go.mod, Taskfile namespace)" + status: completed isProject: false --- -# SLIM vs A2A Multi-Agent DAG Benchmark +# SLIM vs A2A Comparison (`benchmarks/slim-vs-a2a`) + +This package lives under `benchmarks/` for repo layout consistency, but its tasks are **comparative evaluations** (same DAG plan, A2A vs SLIM metrics) — not throughput/latency benchmarks like `agntcy-slim`. Task names use the `compare:` prefix, not `benchmark:`. + +## Project naming -## Goal +| Item | Name | +| ---- | ---- | +| Directory | [`benchmarks/slim-vs-a2a/`](benchmarks/slim-vs-a2a/) | +| Go module | `github.com/agntcy/csit/benchmarks/slim-vs-a2a` | +| Taskfile namespace | `benchmarks:slim-vs-a2a:*` (e.g. `task benchmarks:slim-vs-a2a:compare:suite`) | +| CI workflow | [`.github/workflows/test-slim-vs-a2a.yaml`](.github/workflows/test-slim-vs-a2a.yaml) — **standalone**, not merged into `test-benchmarks-slim.yaml` or `test-integrations.yaml` | +| Report artifact prefix | `slim-vs-a2a-*` | -Build a reproducible benchmark in CSIT that shows **when SLIM Multicast RPC is favorable over direct A2A gRPC** in a distributed multi-agent workload: +Build a reproducible **comparison** in CSIT that shows **when SLIM Multicast RPC is favorable over direct A2A gRPC** in a distributed multi-agent workload: -- **Phase 1 (plans):** Hand-author YAML execution plans with concrete agent/task names — no code generator -- **Phase 2 (implementation):** Separate A2A and SLIM runners/agents that execute those plans and emit metrics +- **Phase 1 (plans):** Hand-author YAML execution plans with concrete agent/task names — no code generator *(done)* +- **Phase 2 (implementation):** Separate A2A and SLIM runners/agents, metrics, local Taskfile + Ginkgo suite +- **Phase 3 (CI):** Dedicated [`test-slim-vs-a2a.yaml`](.github/workflows/test-slim-vs-a2a.yaml) workflow — separate from SLIM benchmarks and A2A integration CI - Tasks use **concrete domain-specific names** (network probe, pod log scrape, route recalc, etc.) but **simulate/mock** execution via timed sleeps and canned outputs - Parallel branches become **obsolete** when an upstream dependency fails or times out - **SLIM advantages under test** (three pillars): @@ -44,7 +61,7 @@ Build a reproducible benchmark in CSIT that shows **when SLIM Multicast RPC is f | Decision | Choice | Rationale | | ------------- | ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Runtime | **Local processes + embedded SLIM dataplane** | Matches `[benchmarks/agntcy-slim](benchmarks/agntcy-slim)`; fast iteration; K8s can be phase 2 | +| Runtime | **Local processes + embedded SLIM dataplane** | Matches `[benchmarks/agntcy-slim](benchmarks/agntcy-slim)`; fast iteration; K8s deployment deferred beyond Phase 3 | | Orchestration | **Separate runners per transport** (`a2a-runner`, `slim-runner`) | Each impl owns its own DAG driver; same plan YAML input, no shared scheduler interface | | Language | **Go-only** | User asked for A2A Go SDK + SLIM Go bindings; existing Python `slima2a` is out of scope | | SLIM API | **Custom slimrpc protobuf + multicast group client** | Multicast for context push + cancel/notify; upstream pattern in `[client_group.go](/Users/smagyari/dev/slim/data-plane/bindings/go/examples/slimrpc/simple/cmd/client_group/client_group.go)` | @@ -101,10 +118,10 @@ Metrics like `context_push_ms`, `sync_barrier_ms`, `cancel_propagation_ms`, `obs ## New package layout -Create `[benchmarks/agntcy-multi-agent/](benchmarks/agntcy-multi-agent/)` and register it in `[benchmarks/Taskfile.yml](benchmarks/Taskfile.yml)`: +Create [`benchmarks/slim-vs-a2a/`](benchmarks/slim-vs-a2a/) and register it in [`benchmarks/Taskfile.yml`](benchmarks/Taskfile.yml) under the `slim-vs-a2a` include key: ``` -benchmarks/agntcy-multi-agent/ +benchmarks/slim-vs-a2a/ ├── Taskfile.yml ├── internal/ │ └── plan/ # ONLY shared code: YAML schema, validation, DAG helpers @@ -135,7 +152,7 @@ benchmarks/agntcy-multi-agent/ │ └── mobile-nav-assistant.yaml ├── tests/ │ ├── suite_test.go -│ ├── dag_benchmark_test.go +│ ├── slim_vs_a2a_test.go │ └── helpers_test.go ├── templates/ └── reports/ @@ -212,7 +229,7 @@ Fields: - `agents[].slimName` — SLIM dataplane identity (`org/group/app`) - `agents[].a2aPort` — gRPC port for A2A agent card (A2A impl only) - `tasks[].name` — human-readable task label (reporting + realism) -- `contextUpdates[]` — models mid-plan context shifts (key SLIM sync/context benchmark axis) +- `contextUpdates[]` — models mid-plan context shifts (key SLIM sync/context comparison axis) ### Three hand-authored domain plans (committed in `plans/domains/`) @@ -353,50 +370,89 @@ Both runners emit the **same JSON/TSV shape** (defined in `tests/` or a tiny sha Comparison report auto-computes per-plan deltas: `(a2a - slim) / a2a` for wall clock, context push, sync barrier, cancel propagation, obsolete tasks. -## Taskfile and execution +## Local execution (Phase 2) -`[benchmarks/agntcy-multi-agent/Taskfile.yml](benchmarks/agntcy-multi-agent/Taskfile.yml)`: +[`benchmarks/slim-vs-a2a/Taskfile.yml`](benchmarks/slim-vs-a2a/Taskfile.yml) — local dev only in Phase 2. +### Task naming (`compare:` not `benchmark:`) -| Task | Purpose | -| -------------------------- | --------------------------------------------------------- | -| `deps:slim-bindings-setup` | Reuse slim benchmark setup | -| `benchmark:suite` | All plans in `plans/domains/` × both implementations | -| `benchmark:ci:smoke` | 3 domain plans × both implementations (CI subset) | -| `benchmark:report` | Build HTML dashboard | - +| Task | Purpose | +| ---- | ------- | +| `deps:slim-bindings-setup` | Reuse slim-bindings setup from `agntcy-slim` | +| `compare:suite` | Run all domain plans on both A2A and SLIM; collect comparison metrics | +| `compare:report` | Build HTML dashboard from `reports/` comparison results | +| `compare:ci:smoke` | *(Phase 3)* Bounded plan matrix for CI (~2 min) | Configurable env vars: - `PLAN_DIR=plans/domains` (default; all hand-authored plans) - `IMPLEMENTATIONS=a2a,slim` -- `RUN_BENCHMARK_DAG=1` (gate like `SLIM_RUN_BENCHMARK_SUITE`) +- `RUN_SLIM_VS_A2A=1` (gate env var for Ginkgo suite; analogous to `SLIM_RUN_BENCHMARK_SUITE` but named for comparison) Example local run: ```bash -task benchmarks:multi-agent:benchmark:ci:smoke RUN_BENCHMARK_DAG=1 -task benchmarks:multi-agent:benchmark:suite RUN_BENCHMARK_DAG=1 +task benchmarks:slim-vs-a2a:compare:suite RUN_SLIM_VS_A2A=1 +task benchmarks:slim-vs-a2a:compare:report ``` -## Test suite (Ginkgo) +## Test suite (Phase 2 — Ginkgo, local) -`[tests/dag_benchmark_test.go](benchmarks/agntcy-multi-agent/tests/dag_benchmark_test.go)`: +[`tests/slim_vs_a2a_test.go`](benchmarks/slim-vs-a2a/tests/slim_vs_a2a_test.go): -- Label: `benchmark-dag` -- Skip unless `RUN_BENCHMARK_DAG=1` +- Label: `slim-vs-a2a` +- Skip unless `RUN_SLIM_VS_A2A=1` - For each plan in `PLAN_DIR`: 1. Run `a2a-runner --plan=...` → collect metrics 2. Run `slim-runner --plan=...` (start SLIM stack first) → collect metrics - Assert all three domain plans complete on both implementations; report deltas (no hard latency assertions) -## CI integration (phase 2) +## CI integration (Phase 3 — dedicated workflow) + +Wire into CI only after Phase 2 runs cleanly locally. Use a **completely separate** workflow — do not add jobs to [`test-benchmarks-slim.yaml`](.github/workflows/test-benchmarks-slim.yaml) or [`test-integrations.yaml`](.github/workflows/test-integrations.yaml). + +**New file:** [`.github/workflows/test-slim-vs-a2a.yaml`](.github/workflows/test-slim-vs-a2a.yaml) + +| Deliverable | Path / task | +| ----------- | ------------- | +| Register in benchmarks root Taskfile | [`benchmarks/Taskfile.yml`](benchmarks/Taskfile.yml) — `includes.slim-vs-a2a` → `./slim-vs-a2a` | +| CI smoke Taskfile task | `compare:ci:smoke` — 3 domain plans × both implementations (~2 min) | +| Standalone CI workflow | `.github/workflows/test-slim-vs-a2a.yaml` | +| Workflow name (GitHub UI) | `test-slim-vs-a2a` | +| Path filter | `benchmarks/slim-vs-a2a/**`, `.github/workflows/test-slim-vs-a2a.yaml` | +| CI job command | `task benchmarks:slim-vs-a2a:compare:ci:smoke` | +| Report artifacts | Upload `reports/` as `slim-vs-a2a-smoke-report` (separate artifact name from SLIM benchmark reports) | +| Pages publish hook | Extend [`publish-test-reports-pages.yaml`](.github/workflows/publish-test-reports-pages.yaml) with a dedicated `--slim-vs-a2a-dir` input; do not fold into existing SLIM smoke/capacity dirs | -Add workflow modeled on `[.github/workflows/test-benchmarks-slim.yaml](.github/workflows/test-benchmarks-slim.yaml)`: +Phase 3 CI smoke example: -- Path filter: `benchmarks/agntcy-multi-agent/`** -- Job runs `benchmark:ci:smoke` only (small plan, ~2 min) -- Publish artifact to test reports site via `[.github/workflows/publish-test-reports-pages.yaml](.github/workflows/publish-test-reports-pages.yaml)` +```bash +task benchmarks:slim-vs-a2a:compare:ci:smoke RUN_SLIM_VS_A2A=1 +``` + +### `test-slim-vs-a2a.yaml` sketch + +```yaml +name: test-slim-vs-a2a +on: + pull_request: + paths: + - benchmarks/slim-vs-a2a/** + - .github/workflows/test-slim-vs-a2a.yaml + workflow_dispatch: +jobs: + slim-vs-a2a-smoke: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Run comparison smoke test + run: task benchmarks:slim-vs-a2a:compare:ci:smoke + - name: Upload report + uses: actions/upload-artifact@v4 + with: + name: slim-vs-a2a-smoke-report + path: benchmarks/slim-vs-a2a/reports/ +``` ## Expected outcome @@ -411,24 +467,33 @@ Running the domain plans (especially `urban-traffic-reroute` and `mobile-nav-ass ## Implementation order -### Phase 1 — Plans (YAML only, no Go) +### Phase 1 — Plans (YAML only) — complete 1. Finalize schema in plan doc / `internal/plan` types -2. Hand-author 3 domain plans in `plans/domains/` with full agent lists, task DAGs, timings, `contextUpdates`, and injected failure paths +2. Hand-author 3 domain plans in [`plans/domains/`](benchmarks/slim-vs-a2a/plans/domains/) with full agent lists, task DAGs, timings, `contextUpdates`, and injected failure paths -### Phase 2 — Implementation +### Phase 2 — Implementation (local only) +0. ~~**Rename** `benchmarks/agntcy-multi-agent/` → `benchmarks/slim-vs-a2a/`~~ *(done)* 3. **`internal/plan`** — YAML load, validation, DAG helpers 4. **`a2a/` stack** — agent + runner + scheduler; end-to-end on domain plans 5. **`slim/` stack** — proto, agent + runner + scheduler with multicast; same plans end-to-end 6. **Context sync + failure propagation** — wire `contextUpdates`, cancel obsolete tasks, retries -7. **Metrics + TSV + templates + dashboard** -8. **Taskfile + Ginkgo suite + CI smoke** +7. **Metrics + TSV + templates + local report dashboard** +8. **Taskfile + Ginkgo suite** — runnable locally via `task benchmarks:slim-vs-a2a:compare:suite` + +### Phase 3 — CI and publishing (dedicated flow) + +9. **`compare:ci:smoke`** Taskfile task (bounded matrix for CI) +10. **Register** `slim-vs-a2a` in [`benchmarks/Taskfile.yml`](benchmarks/Taskfile.yml) +11. **Create** [`.github/workflows/test-slim-vs-a2a.yaml`](.github/workflows/test-slim-vs-a2a.yaml) — standalone workflow, separate triggers and artifacts from SLIM/A2A/integration CI +12. **Artifact upload** — `slim-vs-a2a-smoke-report` from CI runs +13. **Test reports site** — dedicated publish input in `publish-test-reports-pages.yaml` + dashboard section ## Out of scope (initial version) - **`plangen` / automated plan scaling** — plans are hand-authored; add new YAML files manually -- K8s/Kind deployment (design allows later: same agent/runner binaries in Helm) +- K8s/Kind deployment (deferred beyond Phase 3; same binaries can be containerized later) - Python `slima2a` agents - A2A-over-SLIM third transport leg (could be added later as `slim-a2a`) - Directory/registry integration (`agntcy-dir`) — unrelated to this comparison diff --git a/.github/workflows/test-slim-vs-a2a.yaml b/.github/workflows/test-slim-vs-a2a.yaml new file mode 100644 index 00000000..c47daae6 --- /dev/null +++ b/.github/workflows/test-slim-vs-a2a.yaml @@ -0,0 +1,55 @@ +name: test-slim-vs-a2a + +on: + push: + branches: + - main + paths: + - '.github/workflows/test-slim-vs-a2a.yaml' + - 'benchmarks/slim-vs-a2a/**' + - 'benchmarks/Taskfile.yml' + pull_request: + paths: + - '.github/workflows/test-slim-vs-a2a.yaml' + - 'benchmarks/slim-vs-a2a/**' + - 'benchmarks/Taskfile.yml' + workflow_dispatch: + +jobs: + slim-vs-a2a-smoke: + runs-on: ubuntu-latest + + permissions: + contents: read + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Environment + uses: ./.github/actions/setup-env + with: + go: true + + - name: Install slimctl + shell: bash + run: | + task benchmarks:slim-vs-a2a:deps:slimctl-download + echo "$(pwd)/benchmarks/agntcy-slim/bin" >> "$GITHUB_PATH" + + - name: Run SLIM vs A2A comparison smoke + shell: bash + working-directory: benchmarks/slim-vs-a2a + env: + RUN_SLIM_VS_A2A: "1" + run: | + set -o pipefail + task compare:ci:smoke + + - name: Upload comparison report + if: always() + uses: actions/upload-artifact@v4 + with: + name: slim-vs-a2a-smoke-report + path: benchmarks/slim-vs-a2a/reports/ + if-no-files-found: warn diff --git a/benchmarks/Taskfile.yml b/benchmarks/Taskfile.yml index 3f65d070..faee6796 100644 --- a/benchmarks/Taskfile.yml +++ b/benchmarks/Taskfile.yml @@ -17,6 +17,11 @@ includes: dir: ./agntcy-dir excludes: [default] + slim-vs-a2a: + taskfile: ./slim-vs-a2a/Taskfile.yml + dir: ./slim-vs-a2a + excludes: [default] + tasks: default: cmd: task -l \ No newline at end of file diff --git a/benchmarks/agntcy-multi-agent/go.mod b/benchmarks/agntcy-multi-agent/go.mod deleted file mode 100644 index e6de3725..00000000 --- a/benchmarks/agntcy-multi-agent/go.mod +++ /dev/null @@ -1,5 +0,0 @@ -module github.com/agntcy/csit/benchmarks/agntcy-multi-agent - -go 1.25.0 - -require gopkg.in/yaml.v3 v3.0.1 diff --git a/benchmarks/agntcy-multi-agent/go.sum b/benchmarks/agntcy-multi-agent/go.sum deleted file mode 100644 index a62c313c..00000000 --- a/benchmarks/agntcy-multi-agent/go.sum +++ /dev/null @@ -1,4 +0,0 @@ -gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= -gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= -gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/benchmarks/agntcy-multi-agent/tools/validate_plans/main.go b/benchmarks/agntcy-multi-agent/tools/validate_plans/main.go deleted file mode 100644 index 953a7a7e..00000000 --- a/benchmarks/agntcy-multi-agent/tools/validate_plans/main.go +++ /dev/null @@ -1,104 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package main - -import ( - "fmt" - "os" - "path/filepath" - "strings" - - "gopkg.in/yaml.v3" -) - -type plan struct { - Agents []struct { - ID string `yaml:"id"` - } `yaml:"agents"` - Tasks []struct { - ID string `yaml:"id"` - Agent string `yaml:"agent"` - DependsOn []string `yaml:"dependsOn"` - CompletionTimeSec float64 `yaml:"completionTimeSec"` - MaxCompletionTimeSec float64 `yaml:"maxCompletionTimeSec"` - } `yaml:"tasks"` - ContextUpdates []struct { - AfterTask string `yaml:"afterTask"` - TargetAgents []string `yaml:"targetAgents"` - } `yaml:"contextUpdates"` -} - -func main() { - dir := filepath.Join("..", "plans", "domains") - if len(os.Args) > 1 { - dir = os.Args[1] - } - - entries, err := os.ReadDir(dir) - if err != nil { - fmt.Fprintf(os.Stderr, "read dir: %v\n", err) - os.Exit(1) - } - - var errs []string - for _, e := range entries { - if !strings.HasSuffix(e.Name(), ".yaml") { - continue - } - path := filepath.Join(dir, e.Name()) - data, err := os.ReadFile(path) - if err != nil { - errs = append(errs, err.Error()) - continue - } - - var p plan - if err := yaml.Unmarshal(data, &p); err != nil { - errs = append(errs, fmt.Sprintf("%s: parse: %v", e.Name(), err)) - continue - } - - agents := map[string]bool{} - for _, a := range p.Agents { - agents[a.ID] = true - } - tasks := map[string]struct{}{} - for _, t := range p.Tasks { - tasks[t.ID] = struct{}{} - } - - for _, t := range p.Tasks { - if !agents[t.Agent] { - errs = append(errs, fmt.Sprintf("%s: task %s unknown agent %s", e.Name(), t.ID, t.Agent)) - } - for _, d := range t.DependsOn { - if _, ok := tasks[d]; !ok { - errs = append(errs, fmt.Sprintf("%s: task %s unknown dep %s", e.Name(), t.ID, d)) - } - } - if t.MaxCompletionTimeSec < t.CompletionTimeSec { - errs = append(errs, fmt.Sprintf("%s: task %s max < completion", e.Name(), t.ID)) - } - } - for _, cu := range p.ContextUpdates { - if _, ok := tasks[cu.AfterTask]; !ok { - errs = append(errs, fmt.Sprintf("%s: contextUpdate unknown afterTask %s", e.Name(), cu.AfterTask)) - } - for _, a := range cu.TargetAgents { - if !agents[a] { - errs = append(errs, fmt.Sprintf("%s: contextUpdate unknown agent %s", e.Name(), a)) - } - } - } - - fmt.Printf("%s: %d agents, %d tasks, %d contextUpdates\n", e.Name(), len(p.Agents), len(p.Tasks), len(p.ContextUpdates)) - } - - if len(errs) > 0 { - for _, e := range errs { - fmt.Println("ERROR:", e) - } - os.Exit(1) - } -} diff --git a/benchmarks/slim-vs-a2a/Taskfile.yml b/benchmarks/slim-vs-a2a/Taskfile.yml new file mode 100644 index 00000000..fd76e14c --- /dev/null +++ b/benchmarks/slim-vs-a2a/Taskfile.yml @@ -0,0 +1,399 @@ +# Copyright AGNTCY Contributors (https://github.com/agntcy) +# SPDX-License-Identifier: Apache-2.0 + +--- +version: '3' + +silent: true + +includes: + slim: + taskfile: ../agntcy-slim/Taskfile.yml + dir: ../agntcy-slim + excludes: [default] + +vars: + PLAN_DIR: '{{ .PLAN_DIR | default "./plans/domains" }}' + REPORTS_DIR: '{{ .REPORTS_DIR | default "./reports" }}' + RESULTS_TSV: '{{ .RESULTS_TSV | default "./reports/results.tsv" }}' + SWEEP_TSV: '{{ .SWEEP_TSV | default "./reports/sweep.tsv" }}' + BIN_DIR: '{{ .BIN_DIR | default "./bin" }}' + SLIM_ENDPOINT: '{{ .SLIM_ENDPOINT | default "http://127.0.0.1:46357" }}' + # Use a dedicated name: agntcy-slim's included Taskfile also defines SLIMCTL_PATH=./bin/slimctl + # and would otherwise resolve to slim-vs-a2a/bin/slimctl when tasks run in this directory. + COMPARE_SLIMCTL: '{{ .COMPARE_SLIMCTL | default "../agntcy-slim/bin/slimctl" }}' + COMPARE_SLIM_BINDINGS_VERSION: '{{ .COMPARE_SLIM_BINDINGS_VERSION | default "v1.4.0" }}' + COMPARE_SLIMCTL_TAG: '{{ .COMPARE_SLIMCTL_TAG | default "slimctl-v1.4.0" }}' + IMPLEMENTATIONS: '{{ .IMPLEMENTATIONS | default "a2a,slim" }}' + +tasks: + default: + cmd: task -l + + deps:slim-bindings-setup: + desc: Install native SLIM bindings required by slim agents + cmds: + - go run github.com/agntcy/slim-bindings-go/cmd/slim-bindings-setup@{{ .COMPARE_SLIM_BINDINGS_VERSION }} + + deps:slimctl-download: + desc: Download slimctl for local SLIM stack startup + cmds: + - | + set -e + SLIMCTL_ARCH=$(arch) + SLIMCTL_OS=$(uname -s | tr '[:upper:]' '[:lower:]') + SLIMCTL_ABI="" + if [ "$SLIMCTL_ARCH" = "x86_64" ]; then + SLIMCTL_ARCH="amd64" + fi + if [ "$SLIMCTL_OS" = "linux" ]; then + if ldd --version 2>&1 | grep -qi musl; then + SLIMCTL_ABI="-musl" + else + SLIMCTL_ABI="-gnu" + fi + fi + SLIMCTL_URL="https://github.com/agntcy/slim/releases/download/{{ .COMPARE_SLIMCTL_TAG }}/slimctl-$SLIMCTL_OS-$SLIMCTL_ARCH${SLIMCTL_ABI}.tar.gz" + TARGET_DIR=$(dirname {{ .COMPARE_SLIMCTL }}) + mkdir -p "$TARGET_DIR" + TMP_DIR=$(mktemp -d) + trap 'rm -rf "$TMP_DIR"' EXIT + curl --fail --show-error --location -o "$TMP_DIR/slimctl.tar.gz" "$SLIMCTL_URL" + tar -xzf "$TMP_DIR/slimctl.tar.gz" -C "$TMP_DIR" + mv "$TMP_DIR/slimctl" {{ .COMPARE_SLIMCTL }} + chmod +x {{ .COMPARE_SLIMCTL }} + {{ .COMPARE_SLIMCTL }} version || true + + build: + desc: Build a2a and slim agent/runner binaries + deps: + - deps:slim-bindings-setup + cmds: + - mkdir -p {{ .BIN_DIR }} + - go build -o {{ .BIN_DIR }}/a2a-agent ./a2a/cmd/agent + - go build -o {{ .BIN_DIR }}/a2a-runner ./a2a/cmd/runner + - | + CACHE_DIR="$(go env GOPATH)/.cgo-cache/slim-bindings/{{ .COMPARE_SLIM_BINDINGS_VERSION }}" + export CGO_LDFLAGS="-L${CACHE_DIR}" + go build -o {{ .BIN_DIR }}/slim-agent ./slim/cmd/agent + go build -o {{ .BIN_DIR }}/slim-runner ./slim/cmd/runner + + validate:plans: + desc: Validate all domain execution plans + cmds: + - go run ./tools/validate_plans --dir {{ .PLAN_DIR }} + + compare:cleanup: + internal: true + cmds: + - pkill -f '{{ .BIN_DIR }}/a2a-agent' 2>/dev/null || true + - pkill -f '{{ .BIN_DIR }}/slim-agent' 2>/dev/null || true + - pkill -f 'slimctl slim start -c' 2>/dev/null || true + + compare:suite: + desc: Run all domain plans on A2A and SLIM and collect comparison metrics + deps: + - build + - validate:plans + cmds: + - task: compare:cleanup + - rm -f {{ .RESULTS_TSV }} + - | + set -euo pipefail + for plan in {{ .PLAN_DIR }}/*.yaml; do + [ -f "$plan" ] || continue + base=$(basename "$plan" .yaml) + echo "==> A2A $base" + {{ .BIN_DIR }}/a2a-runner \ + --plan "$plan" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ + --output-tsv {{ .RESULTS_TSV }} + done + - task: compare:suite:slim + + compare:suite:slim: + internal: true + deps: + - deps:slimctl-download + cmds: + - task: compare:slim-stack + vars: + PLAN_GLOB: '{{ .PLAN_DIR }}/*.yaml' + + compare:plan: + desc: Run one plan on A2A and SLIM (PLAN=mobile-nav-assistant or path to yaml) + deps: + - build + requires: + vars: [PLAN] + cmds: + - task: compare:cleanup + - rm -f {{ .RESULTS_TSV }} + - | + set -euo pipefail + PLAN_INPUT="{{ .PLAN }}" + if [ -f "$PLAN_INPUT" ]; then + PLAN_PATH="$PLAN_INPUT" + elif [ -f "{{ .PLAN_DIR }}/${PLAN_INPUT}.yaml" ]; then + PLAN_PATH="{{ .PLAN_DIR }}/${PLAN_INPUT}.yaml" + elif [ -f "{{ .PLAN_DIR }}/${PLAN_INPUT}" ]; then + PLAN_PATH="{{ .PLAN_DIR }}/${PLAN_INPUT}" + else + echo "plan not found: ${PLAN_INPUT} (tried file, {{ .PLAN_DIR }}/\${PLAN_INPUT}.yaml)" >&2 + exit 1 + fi + base=$(basename "$PLAN_PATH" .yaml) + echo "==> A2A ${base}" + {{ .BIN_DIR }}/a2a-runner \ + --plan "$PLAN_PATH" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ + --output-tsv {{ .RESULTS_TSV }} + - task: compare:slim-stack + vars: + PLAN_GLOB: '{{ .PLAN }}' + + compare:slim-stack: + internal: true + deps: + - deps:slimctl-download + cmds: + - | + set -euo pipefail + PLAN_INPUT="{{ .PLAN_GLOB }}" + resolve_plan() { + local input="$1" + if [ -f "$input" ]; then + printf '%s' "$input" + return + fi + if [ -f "{{ .PLAN_DIR }}/${input}.yaml" ]; then + printf '%s' "{{ .PLAN_DIR }}/${input}.yaml" + return + fi + if [ -f "{{ .PLAN_DIR }}/${input}" ]; then + printf '%s' "{{ .PLAN_DIR }}/${input}" + return + fi + printf '%s' "$input" + } + DATAPLANE_PORT=46357 + CONTROLLER_PORT=46358 + CONFIG=$(mktemp) + trap 'kill ${SLIM_PID:-} 2>/dev/null || true; rm -f "$CONFIG"; {{ .COMPARE_SLIMCTL }} slim stop 2>/dev/null || true' EXIT + cat > "$CONFIG" </dev/null; then + break + fi + sleep 0.2 + done + if [[ "$PLAN_INPUT" == *"*"* ]]; then + PLAN_LIST=($PLAN_INPUT) + else + PLAN_LIST=("$(resolve_plan "$PLAN_INPUT")") + fi + for plan in "${PLAN_LIST[@]}"; do + [ -f "$plan" ] || continue + base=$(basename "$plan" .yaml) + echo "==> SLIM ${base}" + {{ .BIN_DIR }}/slim-runner \ + --plan "$plan" \ + --endpoint {{ .SLIM_ENDPOINT }} \ + --agent-bin {{ .BIN_DIR }}/slim-agent \ + --coord-mode {{ .COORD_MODE | default "multicast" }} \ + --round-budget-ms {{ .ROUND_BUDGET_MS | default "0" }} \ + --output-json {{ .REPORTS_DIR }}/slim-${base}.json \ + --output-tsv {{ .RESULTS_TSV }} + done + + compare:report: + desc: Build HTML dashboard from comparison TSV results + cmds: + - go run ./tools/report --tsv {{ .RESULTS_TSV }} --sweep-tsv {{ .SWEEP_TSV }} --output {{ .REPORTS_DIR }}/index.html + + compare:sweep: + desc: Run transport sweep (SWEEP_FAMILY, SWEEP_AGENTS, SWEEP_BUDGETS env lists) + deps: + - build + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "sustainable-resource" }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' + SWEEP_BUDGETS: '{{ .SWEEP_BUDGETS | default "10,20" }}' + cmds: + - task: compare:cleanup + - rm -f {{ .SWEEP_TSV }} + - task: compare:sweep:run + + compare:sweep:run: + internal: true + deps: + - deps:slimctl-download + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "sustainable-resource" }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' + SWEEP_BUDGETS: '{{ .SWEEP_BUDGETS | default "10,20" }}' + cmds: + - | + set -euo pipefail + mkdir -p {{ .REPORTS_DIR }} plans/sweeps + IFS=',' read -r -a AGENT_LIST <<< "{{ .SWEEP_AGENTS }}" + IFS=',' read -r -a BUDGET_LIST <<< "{{ .SWEEP_BUDGETS }}" + for agents in "${AGENT_LIST[@]}"; do + for budget in "${BUDGET_LIST[@]}"; do + plan_path="plans/sweeps/{{ .SWEEP_FAMILY }}-${agents}ag-${budget}ms.yaml" + go run ./tools/gen_plan \ + -family {{ .SWEEP_FAMILY }} \ + -agents "$agents" \ + -round-budget-ms "$budget" \ + -output "$plan_path" + base=$(basename "$plan_path" .yaml) + echo "==> sweep A2A $base" + {{ .BIN_DIR }}/a2a-runner \ + --plan "$plan_path" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --round-budget-ms "$budget" \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ + --output-tsv {{ .SWEEP_TSV }} + done + done + - task: compare:sweep:slim + + compare:sweep:slim: + internal: true + deps: + - deps:slimctl-download + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "sustainable-resource" }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' + SWEEP_BUDGETS: '{{ .SWEEP_BUDGETS | default "10,20" }}' + cmds: + - | + set -euo pipefail + IFS=',' read -r -a AGENT_LIST <<< "{{ .SWEEP_AGENTS }}" + IFS=',' read -r -a BUDGET_LIST <<< "{{ .SWEEP_BUDGETS }}" + DATAPLANE_PORT=46357 + CONTROLLER_PORT=46358 + CONFIG=$(mktemp) + trap 'kill ${SLIM_PID:-} 2>/dev/null || true; rm -f "$CONFIG"; {{ .COMPARE_SLIMCTL }} slim stop 2>/dev/null || true' EXIT + cat > "$CONFIG" </dev/null; then + break + fi + sleep 0.2 + done + for agents in "${AGENT_LIST[@]}"; do + for budget in "${BUDGET_LIST[@]}"; do + plan_path="plans/sweeps/{{ .SWEEP_FAMILY }}-${agents}ag-${budget}ms.yaml" + [ -f "$plan_path" ] || continue + base=$(basename "$plan_path" .yaml) + echo "==> sweep SLIM $base" + {{ .BIN_DIR }}/slim-runner \ + --plan "$plan_path" \ + --endpoint {{ .SLIM_ENDPOINT }} \ + --agent-bin {{ .BIN_DIR }}/slim-agent \ + --coord-mode multicast \ + --round-budget-ms "$budget" \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/slim-${base}.json \ + --output-tsv {{ .SWEEP_TSV }} + done + done + + compare:ci:smoke: + desc: CI smoke — domain plans plus one sweep point (~2 min) + deps: + - build + - validate:plans + cmds: + - task: compare:cleanup + - rm -f {{ .RESULTS_TSV }} {{ .SWEEP_TSV }} + - | + set -euo pipefail + for plan in {{ .PLAN_DIR }}/*.yaml; do + [ -f "$plan" ] || continue + base=$(basename "$plan" .yaml) + echo "==> smoke A2A $base" + {{ .BIN_DIR }}/a2a-runner \ + --plan "$plan" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ + --output-tsv {{ .RESULTS_TSV }} + done + - task: compare:slim-stack + vars: + PLAN_GLOB: '{{ .PLAN_DIR }}/*.yaml' + COORD_MODE: multicast + - | + set -euo pipefail + plan_path="plans/sweeps/ci-smoke-sustainable-10ag-20ms.yaml" + mkdir -p plans/sweeps + go run ./tools/gen_plan -family sustainable-resource -agents 10 -round-budget-ms 20 -output "$plan_path" + echo "==> smoke sweep A2A" + {{ .BIN_DIR }}/a2a-runner \ + --plan "$plan_path" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --round-budget-ms 20 \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/a2a-ci-sweep.json \ + --output-tsv {{ .SWEEP_TSV }} + - task: compare:slim-stack + vars: + PLAN_GLOB: 'plans/sweeps/ci-smoke-sustainable-10ag-20ms.yaml' + ROUND_BUDGET_MS: '20' + COORD_MODE: multicast + - task: compare:report + + test: + desc: Run Ginkgo comparison suite (requires RUN_SLIM_VS_A2A=1) + deps: + - build + cmds: + - go test ./tests -v -failfast -test.paniconexit0 -ginkgo.v -ginkgo.label-filter=slim-vs-a2a -ginkgo.timeout 30m -timeout 30m + - task: compare:report diff --git a/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go b/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go new file mode 100644 index 00000000..99c0c27c --- /dev/null +++ b/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go @@ -0,0 +1,153 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "iter" + "log" + "net" + "net/http" + + "github.com/a2aproject/a2a-go/v2/a2a" + a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" + "github.com/a2aproject/a2a-go/v2/a2asrv" + "github.com/a2aproject/a2a-go/v2/a2asrv/taskstore" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/executor" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/protocol" + "google.golang.org/grpc" +) + +type dagExecutor struct { + engine *executor.Engine +} + +func (d *dagExecutor) Execute(ctx context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { + reqText := firstText(execCtx.Message) + req, err := protocol.DecodeRequest(reqText) + if err != nil { + return func(yield func(a2a.Event, error) bool) { + yield(nil, fmt.Errorf("decode request: %w", err)) + } + } + + resp := d.engine.Handle(ctx, req) + body, err := json.Marshal(resp) + if err != nil { + return func(yield func(a2a.Event, error) bool) { + yield(nil, err) + } + } + + return func(yield func(a2a.Event, error) bool) { + task := a2a.NewSubmittedTask(execCtx, execCtx.Message) + state := a2a.TaskStateCompleted + if !resp.OK { + state = a2a.TaskStateFailed + } + task.Status = a2a.TaskStatus{ + State: state, + Message: a2a.NewMessageForTask( + a2a.MessageRoleAgent, + task, + a2a.NewTextPart(string(body)), + ), + } + yield(task, nil) + } +} + +func (d *dagExecutor) Cancel(_ context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { + return func(yield func(a2a.Event, error) bool) { + yield( + a2a.NewStatusUpdateEvent( + execCtx, + a2a.TaskStateCanceled, + a2a.NewMessageForTask( + a2a.MessageRoleAgent, + execCtx, + a2a.NewTextPart(`{"ok":true}`), + ), + ), + nil, + ) + } +} + +func firstText(message *a2a.Message) string { + if message == nil { + return "" + } + for _, part := range message.Parts { + if text := part.Text(); text != "" { + return text + } + } + return "" +} + +func main() { + agentID := flag.String("agent-id", "", "logical agent id") + grpcPort := flag.Int("grpc-port", 0, "A2A gRPC port") + cardPort := flag.Int("card-port", 0, "agent card HTTP port") + flag.Parse() + + if *agentID == "" || *grpcPort == 0 || *cardPort == 0 { + log.Fatal("agent-id, grpc-port, and card-port are required") + } + + handler := a2asrv.NewHandler( + &dagExecutor{engine: executor.New()}, + a2asrv.WithTaskStore(taskstore.NewInMemory(&taskstore.InMemoryStoreConfig{ + Authenticator: func(context.Context) (string, error) { return "slim-vs-a2a", nil }, + })), + ) + + card := &a2a.AgentCard{ + Name: fmt.Sprintf("slim-vs-a2a-%s", *agentID), + Description: "SLIM vs A2A comparison agent", + Version: "1.0.0", + SupportedInterfaces: []*a2a.AgentInterface{ + a2a.NewAgentInterface( + fmt.Sprintf("127.0.0.1:%d", *grpcPort), + a2a.TransportProtocolGRPC, + ), + }, + Capabilities: a2a.AgentCapabilities{Streaming: false}, + DefaultInputModes: []string{"text/plain"}, + DefaultOutputModes: []string{"text/plain"}, + Skills: []a2a.AgentSkill{ + {ID: "dag-task", Name: "DAG task execution", Description: "Mock DAG task worker"}, + }, + } + + cardListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", *cardPort)) + if err != nil { + log.Fatalf("card listen: %v", err) + } + grpcListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", *grpcPort)) + if err != nil { + log.Fatalf("grpc listen: %v", err) + } + + grpcHandler := a2agrpc.NewHandler(handler) + grpcServer := grpc.NewServer() + grpcHandler.RegisterWith(grpcServer) + + go func() { + mux := http.NewServeMux() + mux.Handle(a2asrv.WellKnownAgentCardPath, a2asrv.NewStaticAgentCardHandler(card)) + if err := http.Serve(cardListener, mux); err != nil { + log.Printf("card server stopped: %v", err) + } + }() + + fmt.Printf("A2A_AGENT_READY agent=%s grpc=%d card=%d\n", *agentID, *grpcPort, *cardPort) + if err := grpcServer.Serve(grpcListener); err != nil { + log.Fatalf("grpc serve: %v", err) + } +} diff --git a/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go b/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go new file mode 100644 index 00000000..2468b430 --- /dev/null +++ b/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go @@ -0,0 +1,114 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "log" + "os" + "os/exec" + "time" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/client" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/scheduler" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" +) + +func main() { + planPath := flag.String("plan", "", "path to execution plan yaml") + agentBin := flag.String("agent-bin", "", "path to a2a agent binary") + outputJSON := flag.String("output-json", "", "write run metrics json") + outputTSV := flag.String("output-tsv", "", "append run metrics tsv") + waitReady := flag.Duration("wait-ready", 3*time.Second, "wait for agents to start") + roundBudgetMS := flag.Int64("round-budget-ms", 0, "coordination round budget in ms (overrides plan)") + quiet := flag.Bool("quiet", false, "disable benchmark timing logs") + flag.Parse() + + if *quiet { + benchlog.SetEnabled(false) + } + + if *planPath == "" { + log.Fatal("--plan is required") + } + + p, err := plan.LoadFile(*planPath) + if err != nil { + log.Fatalf("load plan: %v", err) + } + + agentPath := *agentBin + if agentPath == "" { + agentPath = os.Getenv("A2A_AGENT_BIN") + } + if agentPath == "" { + log.Fatal("set --agent-bin or A2A_AGENT_BIN") + } + + procs := startAgents(p, agentPath) + time.Sleep(*waitReady) + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + + cli, err := client.New(ctx, p, client.Config{ + RoundBudgetMS: p.EffectiveRoundBudgetMS(*roundBudgetMS), + }) + if err != nil { + stopAgents(procs) + log.Fatalf("client: %v", err) + } + + result := scheduler.New(p, cli).Run(ctx) + + time.Sleep(200 * time.Millisecond) + stopAgents(procs) + + if *outputJSON != "" { + if err := metrics.WriteJSON(*outputJSON, result); err != nil { + log.Fatalf("write json: %v", err) + } + } + if *outputTSV != "" { + if err := metrics.AppendTSV(*outputTSV, result); err != nil { + log.Fatalf("write tsv: %v", err) + } + } + + data, _ := json.MarshalIndent(result, "", " ") + fmt.Println(string(data)) +} + +func startAgents(p *plan.ExecutionPlan, agentBin string) []*exec.Cmd { + var procs []*exec.Cmd + for _, agent := range p.Agents { + cmd := exec.Command( + agentBin, + "--agent-id", agent.ID, + "--grpc-port", fmt.Sprintf("%d", agent.A2APort), + "--card-port", fmt.Sprintf("%d", p.CardPort(agent)), + ) + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + if err := cmd.Start(); err != nil { + stopAgents(procs) + log.Fatalf("start agent %s: %v", agent.ID, err) + } + procs = append(procs, cmd) + } + return procs +} + +func stopAgents(procs []*exec.Cmd) { + for _, cmd := range procs { + if cmd.Process != nil { + _ = cmd.Process.Kill() + } + } +} diff --git a/benchmarks/slim-vs-a2a/a2a/internal/client/client.go b/benchmarks/slim-vs-a2a/a2a/internal/client/client.go new file mode 100644 index 00000000..72fa84e4 --- /dev/null +++ b/benchmarks/slim-vs-a2a/a2a/internal/client/client.go @@ -0,0 +1,291 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package client + +import ( + "context" + "fmt" + "strings" + "time" + + "github.com/a2aproject/a2a-go/v2/a2a" + "github.com/a2aproject/a2a-go/v2/a2aclient" + "github.com/a2aproject/a2a-go/v2/a2aclient/agentcard" + a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" +) + +type Config struct { + RoundBudgetMS int64 +} + +type Client struct { + clients map[string]*a2aclient.Client + stats *Stats + coordStats *metrics.CoordStats +} + +type Stats struct { + ExecuteRPCCount int + SequentialRPCCount int + MulticastRPCCount int +} + +func New(ctx context.Context, p *plan.ExecutionPlan, cfg Config) (*Client, error) { + clients := map[string]*a2aclient.Client{} + for _, agent := range p.Agents { + baseURL := p.CardBaseURL(agent) + card, err := agentcard.DefaultResolver.Resolve(ctx, baseURL) + if err != nil { + return nil, fmt.Errorf("resolve card for %s: %w", agent.ID, err) + } + cli, err := a2aclient.NewFromCard( + ctx, + card, + a2agrpc.WithGRPCTransport(grpc.WithTransportCredentials(insecure.NewCredentials())), + ) + if err != nil { + return nil, fmt.Errorf("client for %s: %w", agent.ID, err) + } + clients[agent.ID] = cli + } + return &Client{ + clients: clients, + stats: &Stats{}, + coordStats: metrics.NewCoordStats(cfg.RoundBudgetMS), + }, nil +} + +func (c *Client) CoordStats() *metrics.CoordStats { return c.coordStats } + +func (c *Client) Stats() Stats { + return *c.stats +} + +func (c *Client) recordCoord(duration time.Duration, targets, responded, payloadBytes, messagesSent int) { + if c.coordStats != nil { + c.coordStats.RecordCoordOp(duration, targets, responded, payloadBytes, messagesSent) + } +} + +func (c *Client) ExecuteTask(ctx context.Context, agentID string, task plan.Task) (protocol.Response, error) { + req := protocol.Request{ + Op: protocol.OpExecute, + TaskID: task.ID, + CompletionTimeSec: task.CompletionTimeSec, + MaxCompletionTimeSec: task.MaxCompletionTimeSec, + Output: task.Output, + InjectFailure: task.InjectFailure, + } + return c.send(ctx, agentID, req, true) +} + +func (c *Client) CancelTasks(ctx context.Context, agentIDs []string, taskIDs []string) error { + start := time.Now() + var perAgent []string + responded := 0 + for i, agentID := range agentIDs { + req := protocol.Request{Op: protocol.OpCancel, TaskIDs: taskIDs} + callStart := time.Now() + _, err := c.send(ctx, agentID, req, false) + perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) + if err != nil { + c.recordCoord(time.Since(start), len(agentIDs), responded, len(taskIDs)*16, i) + benchlog.RPC(benchlog.ImplA2A, protocol.OpCancel, "sequential", time.Since(start), false, + fmt.Sprintf("agents=%d", len(agentIDs)), + fmt.Sprintf("idx=%d", i+1), + fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), + fmt.Sprintf("err=%v", err), + ) + return err + } + responded++ + c.stats.SequentialRPCCount++ + } + duration := time.Since(start) + c.recordCoord(duration, len(agentIDs), responded, len(taskIDs)*16, len(agentIDs)) + benchlog.RPC(benchlog.ImplA2A, protocol.OpCancel, "sequential", duration, true, + fmt.Sprintf("agents=%d", len(agentIDs)), + fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), + ) + return nil +} + +func (c *Client) PushContext(ctx context.Context, agentIDs []string, payload string) (time.Duration, error) { + start := time.Now() + req := protocol.Request{Op: protocol.OpContext, Payload: payload} + var perAgent []string + responded := 0 + for i, agentID := range agentIDs { + callStart := time.Now() + if _, err := c.send(ctx, agentID, req, false); err != nil { + perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) + c.recordCoord(time.Since(start), len(agentIDs), responded, len(payload), i) + benchlog.RPC(benchlog.ImplA2A, protocol.OpContext, "sequential", time.Since(start), false, + fmt.Sprintf("agents=%d", len(agentIDs)), + fmt.Sprintf("idx=%d", i+1), + fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), + fmt.Sprintf("err=%v", err), + ) + return 0, err + } + perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) + responded++ + c.stats.SequentialRPCCount++ + } + duration := time.Since(start) + c.recordCoord(duration, len(agentIDs), responded, len(payload), len(agentIDs)) + benchlog.RPC(benchlog.ImplA2A, protocol.OpContext, "sequential", duration, true, + fmt.Sprintf("agents=%d", len(agentIDs)), + fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), + ) + return duration, nil +} + +func (c *Client) SyncPhase(ctx context.Context, agentIDs []string, phase string) (time.Duration, error) { + start := time.Now() + req := protocol.Request{Op: protocol.OpSync, Phase: phase} + var perAgent []string + responded := 0 + for i, agentID := range agentIDs { + callStart := time.Now() + if _, err := c.send(ctx, agentID, req, false); err != nil { + perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) + c.recordCoord(time.Since(start), len(agentIDs), responded, len(phase), i) + benchlog.RPC(benchlog.ImplA2A, protocol.OpSync, "sequential", time.Since(start), false, + fmt.Sprintf("agents=%d", len(agentIDs)), + fmt.Sprintf("idx=%d", i+1), + fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), + fmt.Sprintf("err=%v", err), + ) + return 0, err + } + perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) + responded++ + c.stats.SequentialRPCCount++ + } + duration := time.Since(start) + c.recordCoord(duration, len(agentIDs), responded, len(phase), len(agentIDs)) + benchlog.RPC(benchlog.ImplA2A, protocol.OpSync, "sequential", duration, true, + fmt.Sprintf("agents=%d", len(agentIDs)), + fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), + ) + return duration, nil +} + +func (c *Client) NotifyFailure(ctx context.Context, agentIDs []string, failedTaskID string) error { + start := time.Now() + payload := "failure=" + failedTaskID + req := protocol.Request{Op: protocol.OpContext, Payload: payload} + var perAgent []string + responded := 0 + for i, agentID := range agentIDs { + callStart := time.Now() + if _, err := c.send(ctx, agentID, req, false); err != nil { + perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) + c.recordCoord(time.Since(start), len(agentIDs), responded, len(payload), i) + benchlog.RPC(benchlog.ImplA2A, protocol.OpContext, "sequential", time.Since(start), false, + fmt.Sprintf("agents=%d", len(agentIDs)), + fmt.Sprintf("failed_task=%s", failedTaskID), + fmt.Sprintf("idx=%d", i+1), + fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), + fmt.Sprintf("err=%v", err), + ) + return err + } + perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) + responded++ + c.stats.SequentialRPCCount++ + } + duration := time.Since(start) + c.recordCoord(duration, len(agentIDs), responded, len(payload), len(agentIDs)) + benchlog.RPC(benchlog.ImplA2A, protocol.OpContext, "sequential", duration, true, + fmt.Sprintf("agents=%d", len(agentIDs)), + fmt.Sprintf("failed_task=%s", failedTaskID), + fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), + ) + return nil +} + +func (c *Client) send(ctx context.Context, agentID string, req protocol.Request, execute bool) (protocol.Response, error) { + cli, ok := c.clients[agentID] + if !ok { + return protocol.Response{}, fmt.Errorf("unknown agent %q", agentID) + } + text, err := protocol.EncodeRequest(req) + if err != nil { + return protocol.Response{}, err + } + msg := a2a.NewMessage(a2a.MessageRoleUser, a2a.NewTextPart(text)) + callStart := time.Now() + result, err := cli.SendMessage(ctx, &a2a.SendMessageRequest{Message: msg}) + duration := time.Since(callStart) + if err != nil { + if execute { + benchlog.RPC(benchlog.ImplA2A, req.Op, "p2p", duration, false, + fmt.Sprintf("agent=%s", agentID), + fmt.Sprintf("task=%s", req.TaskID), + fmt.Sprintf("err=%v", err), + ) + } + return protocol.Response{}, err + } + if execute { + c.stats.ExecuteRPCCount++ + } + respText, err := responseText(result) + if err != nil { + if execute { + benchlog.RPC(benchlog.ImplA2A, req.Op, "p2p", duration, false, + fmt.Sprintf("agent=%s", agentID), + fmt.Sprintf("task=%s", req.TaskID), + fmt.Sprintf("err=%v", err), + ) + } + return protocol.Response{}, err + } + resp, decErr := protocol.DecodeResponse(respText) + if execute { + benchlog.RPC(benchlog.ImplA2A, req.Op, "p2p", duration, decErr == nil && resp.OK, + fmt.Sprintf("agent=%s", agentID), + fmt.Sprintf("task=%s", req.TaskID), + fmt.Sprintf("resp_ok=%t", resp.OK), + ) + } + if decErr != nil { + return protocol.Response{}, decErr + } + return resp, nil +} + +func responseText(result a2a.SendMessageResult) (string, error) { + switch v := result.(type) { + case *a2a.Message: + return firstText(v), nil + case *a2a.Task: + if v.Status.Message != nil { + return firstText(v.Status.Message), nil + } + return "", fmt.Errorf("task response missing message") + default: + return "", fmt.Errorf("unexpected result type %T", result) + } +} + +func firstText(message *a2a.Message) string { + if message == nil { + return "" + } + for _, part := range message.Parts { + if text := part.Text(); text != "" { + return text + } + } + return "" +} diff --git a/benchmarks/slim-vs-a2a/a2a/internal/executor/executor.go b/benchmarks/slim-vs-a2a/a2a/internal/executor/executor.go new file mode 100644 index 00000000..5b98bbf4 --- /dev/null +++ b/benchmarks/slim-vs-a2a/a2a/internal/executor/executor.go @@ -0,0 +1,97 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "context" + "strings" + "sync" + "time" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/protocol" +) + +type Engine struct { + mu sync.Mutex + cancels map[string]context.CancelFunc + obsolete bool +} + +func New() *Engine { + return &Engine{cancels: map[string]context.CancelFunc{}} +} + +func (e *Engine) Handle(ctx context.Context, req protocol.Request) protocol.Response { + switch req.Op { + case protocol.OpExecute: + return e.execute(ctx, req) + case protocol.OpCancel: + return e.cancel(req) + case protocol.OpContext: + return e.applyContext(req) + case protocol.OpSync: + return protocol.Response{OK: true} + default: + return protocol.Response{OK: false, Error: "unknown op"} + } +} + +func (e *Engine) execute(parent context.Context, req protocol.Request) protocol.Response { + if req.InjectFailure { + return protocol.Response{OK: false, TaskID: req.TaskID, Error: "injected failure"} + } + + ctx, cancel := context.WithCancel(parent) + e.mu.Lock() + e.cancels[req.TaskID] = cancel + e.mu.Unlock() + defer func() { + e.mu.Lock() + delete(e.cancels, req.TaskID) + e.mu.Unlock() + cancel() + }() + + deadline := time.Duration(req.MaxCompletionTimeSec * float64(time.Second)) + if deadline <= 0 { + deadline = time.Duration(req.CompletionTimeSec*float64(time.Second)) + time.Second + } + timer := time.NewTimer(deadline) + defer timer.Stop() + + sleep := time.Duration(req.CompletionTimeSec * float64(time.Second)) + start := time.Now() + select { + case <-time.After(sleep): + case <-ctx.Done(): + return protocol.Response{OK: false, TaskID: req.TaskID, Error: "cancelled"} + case <-timer.C: + return protocol.Response{OK: false, TaskID: req.TaskID, Error: "timeout"} + } + + return protocol.Response{ + OK: true, + TaskID: req.TaskID, + Output: req.Output, + ElapsedSec: time.Since(start).Seconds(), + } +} + +func (e *Engine) cancel(req protocol.Request) protocol.Response { + e.mu.Lock() + defer e.mu.Unlock() + for _, id := range req.TaskIDs { + if cancel, ok := e.cancels[id]; ok { + cancel() + } + } + return protocol.Response{OK: true} +} + +func (e *Engine) applyContext(req protocol.Request) protocol.Response { + if strings.Contains(req.Payload, "cancel") { + e.obsolete = true + } + return protocol.Response{OK: true} +} diff --git a/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go b/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go new file mode 100644 index 00000000..0c48cd2b --- /dev/null +++ b/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go @@ -0,0 +1,54 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package protocol + +import "encoding/json" + +const ( + OpExecute = "execute" + OpCancel = "cancel" + OpContext = "context" + OpSync = "sync" +) + +type Request struct { + Op string `json:"op"` + TaskID string `json:"taskId,omitempty"` + CompletionTimeSec float64 `json:"completionTimeSec,omitempty"` + MaxCompletionTimeSec float64 `json:"maxCompletionTimeSec,omitempty"` + Output string `json:"output,omitempty"` + InjectFailure bool `json:"injectFailure,omitempty"` + TaskIDs []string `json:"taskIds,omitempty"` + Payload string `json:"payload,omitempty"` + Phase string `json:"phase,omitempty"` + FailedTaskID string `json:"failedTaskId,omitempty"` +} + +type Response struct { + OK bool `json:"ok"` + TaskID string `json:"taskId,omitempty"` + Output string `json:"output,omitempty"` + Error string `json:"error,omitempty"` + ElapsedSec float64 `json:"elapsedSec,omitempty"` +} + +func EncodeRequest(req Request) (string, error) { + data, err := json.Marshal(req) + if err != nil { + return "", err + } + return string(data), nil +} + +func DecodeRequest(text string) (Request, error) { + var req Request + err := json.Unmarshal([]byte(text), &req) + return req, err +} + +func DecodeResponse(text string) (Response, error) { + var resp Response + err := json.Unmarshal([]byte(text), &resp) + return resp, err +} diff --git a/benchmarks/slim-vs-a2a/a2a/internal/scheduler/scheduler.go b/benchmarks/slim-vs-a2a/a2a/internal/scheduler/scheduler.go new file mode 100644 index 00000000..37b49c5c --- /dev/null +++ b/benchmarks/slim-vs-a2a/a2a/internal/scheduler/scheduler.go @@ -0,0 +1,342 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package scheduler + +import ( + "context" + "fmt" + "strings" + "sync" + "time" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/client" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" +) + +type taskState int + +const ( + statePending taskState = iota + stateRunning + stateCompleted + stateFailed + stateTimedOut + stateCancelled +) + +type Scheduler struct { + plan *plan.ExecutionPlan + client *client.Client +} + +func New(p *plan.ExecutionPlan, cli *client.Client) *Scheduler { + return &Scheduler{plan: p, client: cli} +} + +func (s *Scheduler) Run(ctx context.Context) metrics.RunResult { + start := time.Now() + benchlog.SetRunStart(start) + result := metrics.RunResult{ + PlanName: s.plan.Metadata.Name, + Domain: s.plan.Metadata.Domain, + Implementation: "a2a-grpc", + Agents: len(s.plan.Agents), + Tasks: len(s.plan.Tasks), + } + + states := map[string]taskState{} + contextFired := map[string]bool{} + var mu sync.Mutex + var wg sync.WaitGroup + var contextPushMS int64 + var syncBarrierMS int64 + var cancelPropagationMS int64 + var obsoleteCompleted int + abort := false + + for _, t := range s.plan.Tasks { + states[t.ID] = statePending + } + + cancelTasks := func(taskIDs []string) { + if len(taskIDs) == 0 { + return + } + byAgent := map[string][]string{} + for _, id := range taskIDs { + task, ok := s.plan.TaskByID(id) + if !ok { + continue + } + byAgent[task.Agent] = append(byAgent[task.Agent], id) + } + startCancel := time.Now() + var cancelErr error + for agentID, ids := range byAgent { + if err := s.client.CancelTasks(ctx, []string{agentID}, ids); err != nil && cancelErr == nil { + cancelErr = err + } + } + duration := time.Since(startCancel) + cancelPropagationMS += duration.Milliseconds() + kv := []string{ + fmt.Sprintf("agents=%d", len(byAgent)), + fmt.Sprintf("tasks=%d", len(taskIDs)), + } + if cancelErr != nil { + kv = append(kv, fmt.Sprintf("err=%v", cancelErr)) + } + benchlog.Coord(benchlog.ImplA2A, "cancel", cancelErr == nil, duration, kv...) + } + + cancelDownstream := func(root string) { + downstream := plan.DownstreamTasks(s.plan.Tasks, root) + var ids []string + mu.Lock() + for id := range downstream { + switch states[id] { + case statePending, stateRunning: + states[id] = stateCancelled + result.TasksCancelled++ + ids = append(ids, id) + } + } + mu.Unlock() + cancelTasks(ids) + } + + handleFailure := func(taskID string, timedOut bool) { + mu.Lock() + if timedOut { + states[taskID] = stateTimedOut + result.TasksTimedOut++ + } else { + states[taskID] = stateFailed + result.TasksFailed++ + } + abort = true + mu.Unlock() + agents := affectedAgents([]string{taskID}, s.plan) + if err := s.client.NotifyFailure(ctx, agents, taskID); err != nil { + benchlog.Coord(benchlog.ImplA2A, "notify_failure", false, 0, + fmt.Sprintf("task=%s", taskID), + fmt.Sprintf("agents=%d", len(agents)), + fmt.Sprintf("err=%v", err), + ) + } + cancelTasks([]string{taskID}) + cancelDownstream(taskID) + } + + applyContext := func(taskID string) { + mu.Lock() + if contextFired[taskID] { + mu.Unlock() + return + } + contextFired[taskID] = true + mu.Unlock() + + for _, cu := range s.plan.ContextUpdatesAfter(taskID) { + d, err := s.client.PushContext(ctx, cu.TargetAgents, cu.Payload) + contextPushMS += d.Milliseconds() + kv := []string{ + fmt.Sprintf("after_task=%s", taskID), + fmt.Sprintf("agents=%d", len(cu.TargetAgents)), + fmt.Sprintf("payload=%q", benchlog.TruncatePayload(cu.Payload)), + } + if err != nil { + kv = append(kv, fmt.Sprintf("err=%v", err)) + } + benchlog.Coord(benchlog.ImplA2A, "context_push", err == nil, d, kv...) + + if strings.Contains(cu.Payload, "sync=") || strings.Contains(cu.Payload, "phase=") { + syncD, syncErr := s.client.SyncPhase(ctx, cu.TargetAgents, cu.Payload) + if syncErr == nil { + syncBarrierMS += syncD.Milliseconds() + } + syncKV := []string{ + fmt.Sprintf("after_task=%s", taskID), + fmt.Sprintf("agents=%d", len(cu.TargetAgents)), + } + if syncErr != nil { + syncKV = append(syncKV, fmt.Sprintf("err=%v", syncErr)) + } + benchlog.Coord(benchlog.ImplA2A, "sync_phase", syncErr == nil, syncD, syncKV...) + } + if strings.Contains(strings.ToLower(cu.Payload), "cancel") { + var toCancel []string + mu.Lock() + for _, t := range s.plan.Tasks { + if states[t.ID] != stateRunning { + continue + } + if shouldCancelByContext(t, cu.Payload) { + states[t.ID] = stateCancelled + result.TasksCancelled++ + toCancel = append(toCancel, t.ID) + obsoleteCompleted++ + } + } + mu.Unlock() + cancelTasks(toCancel) + } + } + } + + launch := func(task plan.Task) { + wg.Add(1) + go func(task plan.Task) { + defer wg.Done() + taskStart := time.Now() + benchlog.Task(benchlog.ImplA2A, "started", task.ID, task.Agent, 0) + resp, err := s.client.ExecuteTask(ctx, task.Agent, task) + taskDuration := time.Since(taskStart) + mu.Lock() + if abort && states[task.ID] == stateRunning { + states[task.ID] = stateCancelled + result.TasksCancelled++ + mu.Unlock() + benchlog.Task(benchlog.ImplA2A, "cancelled", task.ID, task.Agent, taskDuration) + return + } + mu.Unlock() + + if err != nil { + benchlog.Task(benchlog.ImplA2A, "failed", task.ID, task.Agent, taskDuration, + fmt.Sprintf("err=%v", err)) + handleFailure(task.ID, false) + return + } + if !resp.OK { + benchlog.Task(benchlog.ImplA2A, "failed", task.ID, task.Agent, taskDuration, + fmt.Sprintf("resp_error=%s", resp.Error)) + handleFailure(task.ID, resp.Error == "timeout") + return + } + + mu.Lock() + states[task.ID] = stateCompleted + result.TasksCompleted++ + mu.Unlock() + benchlog.Task(benchlog.ImplA2A, "completed", task.ID, task.Agent, taskDuration) + applyContext(task.ID) + }(task) + } + + for { + mu.Lock() + if abort { + mu.Unlock() + break + } + ready := readyTasks(s.plan.Tasks, states) + for _, task := range ready { + states[task.ID] = stateRunning + launch(task) + } + pending := countStates(states, statePending) + running := countStates(states, stateRunning) + mu.Unlock() + + if len(ready) == 0 && pending == 0 && running == 0 { + break + } + time.Sleep(5 * time.Millisecond) + } + + wg.Wait() + stats := s.client.Stats() + coordTimeMS := contextPushMS + syncBarrierMS + cancelPropagationMS + result.TotalWallClockMS = time.Since(start).Milliseconds() + result.ContextPushMS = contextPushMS + result.SyncBarrierMS = syncBarrierMS + result.CancelPropagationMS = cancelPropagationMS + result.ObsoleteTasksCompleted = obsoleteCompleted + result.ExecuteRPCCount = stats.ExecuteRPCCount + result.SequentialRPCCount = stats.SequentialRPCCount + result.MulticastRPCCount = stats.MulticastRPCCount + result.MakespanMS = result.TotalWallClockMS + result.Success = result.TasksFailed == 0 && result.TasksTimedOut == 0 + if s.client.CoordStats() != nil { + s.client.CoordStats().ApplyToRunResult(&result, coordTimeMS) + } + if !result.Success && result.Error == "" { + result.Error = fmt.Sprintf("failed=%d timed_out=%d cancelled=%d", result.TasksFailed, result.TasksTimedOut, result.TasksCancelled) + } + benchlog.Coord(benchlog.ImplA2A, "run_finished", result.Success, time.Since(start), + fmt.Sprintf("tasks_completed=%d", result.TasksCompleted), + fmt.Sprintf("tasks_failed=%d", result.TasksFailed), + fmt.Sprintf("tasks_cancelled=%d", result.TasksCancelled), + fmt.Sprintf("context_push_ms=%d", contextPushMS), + fmt.Sprintf("execute_rpcs=%d", stats.ExecuteRPCCount), + fmt.Sprintf("sequential_rpcs=%d", stats.SequentialRPCCount), + ) + return result +} + +func readyTasks(tasks []plan.Task, states map[string]taskState) []plan.Task { + var ready []plan.Task + for _, task := range tasks { + if states[task.ID] != statePending { + continue + } + ok := true + for _, dep := range task.DependsOn { + if states[dep] != stateCompleted { + ok = false + break + } + } + if ok { + ready = append(ready, task) + } + } + return ready +} + +func countStates(states map[string]taskState, target taskState) int { + n := 0 + for _, st := range states { + if st == target { + n++ + } + } + return n +} + +func affectedAgents(taskIDs []string, p *plan.ExecutionPlan) []string { + set := map[string]struct{}{} + for _, id := range taskIDs { + task, ok := p.TaskByID(id) + if !ok { + continue + } + set[task.Agent] = struct{}{} + } + out := make([]string, 0, len(set)) + for id := range set { + out = append(out, id) + } + return out +} + +func shouldCancelByContext(task plan.Task, payload string) bool { + payload = strings.ToLower(payload) + if strings.Contains(payload, "pharmacy") && strings.Contains(task.ID, "pharmacy") { + return true + } + if strings.Contains(payload, "detour") && strings.Contains(task.ID, "detour") { + return true + } + if strings.Contains(payload, "node-drain") && strings.Contains(task.ID, "node-drain") { + return true + } + if strings.Contains(payload, "mesh-rollback") && strings.Contains(task.ID, "rollback") { + return true + } + return false +} diff --git a/benchmarks/slim-vs-a2a/go.mod b/benchmarks/slim-vs-a2a/go.mod new file mode 100644 index 00000000..01cfcbab --- /dev/null +++ b/benchmarks/slim-vs-a2a/go.mod @@ -0,0 +1,31 @@ +module github.com/agntcy/csit/benchmarks/slim-vs-a2a + +go 1.25.0 + +require ( + github.com/a2aproject/a2a-go/v2 v2.3.0 + github.com/agntcy/slim-bindings-go v1.4.0 + github.com/onsi/ginkgo/v2 v2.29.0 + github.com/onsi/gomega v1.41.0 + google.golang.org/grpc v1.80.0 + gopkg.in/yaml.v3 v3.0.1 +) + +require ( + github.com/Masterminds/semver/v3 v3.4.0 // indirect + github.com/go-logr/logr v1.4.3 // indirect + github.com/go-task/slim-sprig/v3 v3.0.0 // indirect + github.com/google/go-cmp v0.7.0 // indirect + github.com/google/pprof v0.0.0-20260402051712-545e8a4df936 // indirect + github.com/google/uuid v1.6.0 // indirect + go.yaml.in/yaml/v3 v3.0.4 // indirect + golang.org/x/mod v0.35.0 // indirect + golang.org/x/net v0.53.0 // indirect + golang.org/x/sync v0.20.0 // indirect + golang.org/x/sys v0.43.0 // indirect + golang.org/x/text v0.36.0 // indirect + golang.org/x/tools v0.44.0 // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 // indirect + google.golang.org/protobuf v1.36.11 // indirect +) diff --git a/benchmarks/slim-vs-a2a/go.sum b/benchmarks/slim-vs-a2a/go.sum new file mode 100644 index 00000000..1aad676a --- /dev/null +++ b/benchmarks/slim-vs-a2a/go.sum @@ -0,0 +1,101 @@ +github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0= +github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM= +github.com/a2aproject/a2a-go/v2 v2.3.0 h1:wSseKBBlDYBAJLdHZ4puFIL2XyL/5YuMrDw9TXLL1ek= +github.com/a2aproject/a2a-go/v2 v2.3.0/go.mod h1:mkZr8y2bUgAVQsjs/5fHK7xrRlAHDybMEyxWh2tKRC8= +github.com/agntcy/slim-bindings-go v1.4.0 h1:DXAGhe9TWSm33KIgrJFkIcxdqh+/oPWy4Jq5PCI0HvM= +github.com/agntcy/slim-bindings-go v1.4.0/go.mod h1:XK0Ing+REEl8xG79HTMx52XzWK2THuTQA+Y7JTAn428= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/gkampitakis/ciinfo v0.3.2 h1:JcuOPk8ZU7nZQjdUhctuhQofk7BGHuIy0c9Ez8BNhXs= +github.com/gkampitakis/ciinfo v0.3.2/go.mod h1:1NIwaOcFChN4fa/B0hEBdAb6npDlFL8Bwx4dfRLRqAo= +github.com/gkampitakis/go-diff v1.3.2 h1:Qyn0J9XJSDTgnsgHRdz9Zp24RaJeKMUHg2+PDZZdC4M= +github.com/gkampitakis/go-diff v1.3.2/go.mod h1:LLgOrpqleQe26cte8s36HTWcTmMEur6OPYerdAAS9tk= +github.com/gkampitakis/go-snaps v0.5.15 h1:amyJrvM1D33cPHwVrjo9jQxX8g/7E2wYdZ+01KS3zGE= +github.com/gkampitakis/go-snaps v0.5.15/go.mod h1:HNpx/9GoKisdhw9AFOBT1N7DBs9DiHo/hGheFGBZ+mc= +github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= +github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI= +github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8= +github.com/goccy/go-yaml v1.18.0 h1:8W7wMFS12Pcas7KU+VVkaiCng+kG8QiFeFwzFb+rwuw= +github.com/goccy/go-yaml v1.18.0/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/pprof v0.0.0-20260402051712-545e8a4df936 h1:EwtI+Al+DeppwYX2oXJCETMO23COyaKGP6fHVpkpWpg= +github.com/google/pprof v0.0.0-20260402051712-545e8a4df936/go.mod h1:MxpfABSjhmINe3F1It9d+8exIHFvUqtLIRCdOGNXqiI= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/joshdk/go-junit v1.0.0 h1:S86cUKIdwBHWwA6xCmFlf3RTLfVXYQfvanM5Uh+K6GE= +github.com/joshdk/go-junit v1.0.0/go.mod h1:TiiV0PqkaNfFXjEiyjWM3XXrhVyCa1K4Zfga6W52ung= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/maruel/natural v1.1.1 h1:Hja7XhhmvEFhcByqDoHz9QZbkWey+COd9xWfCfn1ioo= +github.com/maruel/natural v1.1.1/go.mod h1:v+Rfd79xlw1AgVBjbO0BEQmptqb5HvL/k9GRHB7ZKEg= +github.com/mfridman/tparse v0.18.0 h1:wh6dzOKaIwkUGyKgOntDW4liXSo37qg5AXbIhkMV3vE= +github.com/mfridman/tparse v0.18.0/go.mod h1:gEvqZTuCgEhPbYk/2lS3Kcxg1GmTxxU7kTC8DvP0i/A= +github.com/onsi/ginkgo/v2 v2.29.0 h1:rfh+ZFjgJhYWRoIqVf3Uwx/W20yLrcrE2h2GmYVRaag= +github.com/onsi/ginkgo/v2 v2.29.0/go.mod h1:+aXOY+vzZ5mu2iI2HpTZUPmM//oQfsNFX6gU9kNcA44= +github.com/onsi/gomega v1.41.0 h1:OwKp4pXNgVxf6sCplzYo794OFNuoL2q2SBMU5NSWOjA= +github.com/onsi/gomega v1.41.0/go.mod h1:M/Uqpu/8qTjtzCLUA2zJHX9Iilrau25x1PdoSRbWh5A= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII= +github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o= +github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= +github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= +github.com/tidwall/gjson v1.18.0 h1:FIDeeyB800efLX89e5a8Y0BNH+LOngJyGrIWxG2FKQY= +github.com/tidwall/gjson v1.18.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk= +github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA= +github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM= +github.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4= +github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU= +github.com/tidwall/sjson v1.2.5 h1:kLy8mja+1c9jlljvWTlSazM7cKDRfJuR/bOJhcY5NcY= +github.com/tidwall/sjson v1.2.5/go.mod h1:Fvgq9kS/6ociJEDnK0Fk1cpYF4FIW6ZF7LAe+6jwd28= +go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= +go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= +go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48= +go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8= +go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0= +go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs= +go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18= +go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE= +go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8= +go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew= +go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI= +go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA= +go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc= +go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= +golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM= +golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU= +golang.org/x/net v0.53.0 h1:d+qAbo5L0orcWAr0a9JweQpjXF19LMXJE8Ey7hwOdUA= +golang.org/x/net v0.53.0/go.mod h1:JvMuJH7rrdiCfbeHoo3fCQU24Lf5JJwT9W3sJFulfgs= +golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= +golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI= +golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/text v0.36.0 h1:JfKh3XmcRPqZPKevfXVpI1wXPTqbkE5f7JA92a55Yxg= +golang.org/x/text v0.36.0/go.mod h1:NIdBknypM8iqVmPiuco0Dh6P5Jcdk8lJL0CUebqK164= +golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c= +golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI= +gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= +gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= +google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4 h1:yOzSCGPx+cp5VO7IxvZ9SBFF7j1tZVcNtlHR2iYKtVo= +google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4/go.mod h1:Q9HWtNeE7tM9npdIsEvqXj1QJIvVoeAV3rtXtS715Cw= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 h1:tEkOQcXgF6dH1G+MVKZrfpYvozGrzb91k6ha7jireSM= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= +google.golang.org/grpc v1.80.0 h1:Xr6m2WmWZLETvUNvIUmeD5OAagMw3FiKmMlTdViWsHM= +google.golang.org/grpc v1.80.0/go.mod h1:ho/dLnxwi3EDJA4Zghp7k2Ec1+c2jqup0bFkw07bwF4= +google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= +google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/benchmarks/slim-vs-a2a/internal/benchlog/benchlog.go b/benchmarks/slim-vs-a2a/internal/benchlog/benchlog.go new file mode 100644 index 00000000..d3e0ef81 --- /dev/null +++ b/benchmarks/slim-vs-a2a/internal/benchlog/benchlog.go @@ -0,0 +1,122 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package benchlog + +import ( + "fmt" + "io" + "log" + "os" + "strings" + "sync" + "time" +) + +const ( + ImplSLIM = "slim-multicast" + ImplA2A = "a2a-grpc" +) + +var ( + mu sync.RWMutex + runStart time.Time + enabled = true + out io.Writer = os.Stderr +) + +// SetRunStart records the scheduler run start for elapsed_ms on each line. +func SetRunStart(t time.Time) { + mu.Lock() + runStart = t + mu.Unlock() +} + +// SetEnabled toggles benchmark logging (default: on). +func SetEnabled(on bool) { + mu.Lock() + enabled = on + mu.Unlock() +} + +func sinceStart() int64 { + mu.RLock() + start := runStart + mu.RUnlock() + if start.IsZero() { + return 0 + } + return time.Since(start).Milliseconds() +} + +func write(kind string, impl string, kv ...string) { + mu.RLock() + on := enabled + mu.RUnlock() + if !on { + return + } + + ts := time.Now().UTC().Format(time.RFC3339Nano) + elapsed := sinceStart() + parts := []string{ + fmt.Sprintf("bench ts=%s elapsed_ms=%d impl=%s kind=%s", ts, elapsed, impl, kind), + } + parts = append(parts, kv...) + log.New(out, "", 0).Println(strings.Join(parts, " ")) +} + +// RPC logs a client RPC with duration and outcome. +func RPC(impl, op, mode string, duration time.Duration, ok bool, kv ...string) { + status := "ok" + if !ok { + status = "error" + } + fields := []string{ + fmt.Sprintf("op=%s", op), + fmt.Sprintf("mode=%s", mode), + fmt.Sprintf("duration_ms=%d", duration.Milliseconds()), + fmt.Sprintf("status=%s", status), + } + fields = append(fields, kv...) + write("rpc", impl, fields...) +} + +// Task logs scheduler task lifecycle events. +func Task(impl, event, taskID, agent string, duration time.Duration, kv ...string) { + fields := []string{ + fmt.Sprintf("event=%s", event), + fmt.Sprintf("task=%s", taskID), + fmt.Sprintf("agent=%s", agent), + } + if duration > 0 { + fields = append(fields, fmt.Sprintf("duration_ms=%d", duration.Milliseconds())) + } + fields = append(fields, kv...) + write("task", impl, fields...) +} + +// TruncatePayload shortens long payload strings for log lines. +func TruncatePayload(payload string) string { + if len(payload) <= 80 { + return payload + } + return payload[:77] + "..." +} + +// Coord logs coordination ops (context push, cancel, sync) from the scheduler. +func Coord(impl, event string, ok bool, duration time.Duration, kv ...string) { + status := "ok" + if !ok { + status = "error" + } + fields := []string{ + fmt.Sprintf("event=%s", event), + fmt.Sprintf("status=%s", status), + } + if duration > 0 { + fields = append(fields, fmt.Sprintf("duration_ms=%d", duration.Milliseconds())) + } + fields = append(fields, kv...) + write("coord", impl, fields...) +} diff --git a/benchmarks/slim-vs-a2a/internal/metrics/coordstats.go b/benchmarks/slim-vs-a2a/internal/metrics/coordstats.go new file mode 100644 index 00000000..2da1bf63 --- /dev/null +++ b/benchmarks/slim-vs-a2a/internal/metrics/coordstats.go @@ -0,0 +1,95 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package metrics + +import ( + "sort" + "sync" + "time" +) + +// CoordStats tracks coordination RPC quality metrics across a run. +type CoordStats struct { + mu sync.Mutex + + RoundBudgetMS int64 + ContextPushOps int + pushDurations []int64 + MissingResponses int + DeadlineMisses int + BytesSent int64 +} + +func NewCoordStats(roundBudgetMS int64) *CoordStats { + return &CoordStats{RoundBudgetMS: roundBudgetMS} +} + +// RecordCoordOp records one coordination operation (context, sync, cancel, notify). +func (c *CoordStats) RecordCoordOp(duration time.Duration, targets int, responded int, payloadBytes int, messagesSent int) { + if c == nil { + return + } + c.mu.Lock() + defer c.mu.Unlock() + + c.ContextPushOps++ + c.pushDurations = append(c.pushDurations, duration.Milliseconds()) + if targets > responded { + c.MissingResponses += targets - responded + } + if c.RoundBudgetMS > 0 && duration.Milliseconds() > c.RoundBudgetMS { + c.DeadlineMisses++ + } + if messagesSent <= 0 { + messagesSent = 1 + } + c.BytesSent += int64(payloadBytes * messagesSent) +} + +// P95ContextPushMS returns the p95 of recorded coordination op durations. +func (c *CoordStats) P95ContextPushMS() int64 { + if c == nil { + return 0 + } + c.mu.Lock() + defer c.mu.Unlock() + return c.p95Locked() +} + +func (c *CoordStats) p95Locked() int64 { + if len(c.pushDurations) == 0 { + return 0 + } + sorted := append([]int64(nil), c.pushDurations...) + sort.Slice(sorted, func(i, j int) bool { return sorted[i] < sorted[j] }) + idx := int(float64(len(sorted)-1) * 0.95) + return sorted[idx] +} + +// Snapshot returns a copy of aggregate counters. +func (c *CoordStats) Snapshot() (ops int, missing, misses int, bytes int64, p95 int64) { + if c == nil { + return 0, 0, 0, 0, 0 + } + c.mu.Lock() + defer c.mu.Unlock() + return c.ContextPushOps, c.MissingResponses, c.DeadlineMisses, c.BytesSent, c.p95Locked() +} + +// ApplyToRunResult copies coordination stats into a run result. +func (c *CoordStats) ApplyToRunResult(r *RunResult, coordTimeMS int64) { + if c == nil || r == nil { + return + } + ops, missing, misses, bytes, p95 := c.Snapshot() + r.ContextPushOps = ops + r.ContextPushP95MS = p95 + r.CoordMissingResponses = missing + r.CoordDeadlineMisses = misses + r.CoordBytesSent = bytes + r.RoundBudgetMS = c.RoundBudgetMS + if r.TotalWallClockMS > 0 && coordTimeMS > 0 { + r.CoordTimeSharePct = float64(coordTimeMS) * 100 / float64(r.TotalWallClockMS) + } +} diff --git a/benchmarks/slim-vs-a2a/internal/metrics/coordstats_test.go b/benchmarks/slim-vs-a2a/internal/metrics/coordstats_test.go new file mode 100644 index 00000000..9fa900bf --- /dev/null +++ b/benchmarks/slim-vs-a2a/internal/metrics/coordstats_test.go @@ -0,0 +1,39 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package metrics_test + +import ( + "testing" + "time" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" +) + +func TestCoordStatsP95AndDeadline(t *testing.T) { + cs := metrics.NewCoordStats(10) + cs.RecordCoordOp(5*time.Millisecond, 3, 3, 100, 1) + cs.RecordCoordOp(15*time.Millisecond, 3, 3, 100, 1) + cs.RecordCoordOp(20*time.Millisecond, 3, 2, 100, 1) + + if cs.P95ContextPushMS() != 15 { + t.Fatalf("p95=%d want 15", cs.P95ContextPushMS()) + } + _, missing, misses, bytes, _ := cs.Snapshot() + if missing != 1 { + t.Fatalf("missing=%d want 1", missing) + } + if misses != 2 { + t.Fatalf("deadline misses=%d want 2", misses) + } + if bytes != 300 { + t.Fatalf("bytes=%d want 300", bytes) + } + + var r metrics.RunResult + r.TotalWallClockMS = 100 + cs.ApplyToRunResult(&r, 30) + if r.CoordTimeSharePct != 30 { + t.Fatalf("share=%f want 30", r.CoordTimeSharePct) + } +} diff --git a/benchmarks/slim-vs-a2a/internal/metrics/metrics.go b/benchmarks/slim-vs-a2a/internal/metrics/metrics.go new file mode 100644 index 00000000..c9ba20e4 --- /dev/null +++ b/benchmarks/slim-vs-a2a/internal/metrics/metrics.go @@ -0,0 +1,142 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package metrics + +import ( + "encoding/csv" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strconv" +) + +type RunResult struct { + PlanName string `json:"plan_name"` + Domain string `json:"domain"` + Implementation string `json:"implementation"` + Agents int `json:"agents"` + Tasks int `json:"tasks"` + TotalWallClockMS int64 `json:"total_wall_clock_ms"` + TasksCompleted int `json:"tasks_completed"` + TasksFailed int `json:"tasks_failed"` + TasksTimedOut int `json:"tasks_timed_out"` + TasksCancelled int `json:"tasks_cancelled"` + ObsoleteTasksCompleted int `json:"obsolete_tasks_completed"` + RetriesAttempted int `json:"retries_attempted"` + RetriesSucceeded int `json:"retries_succeeded"` + ContextPushMS int64 `json:"context_push_ms"` + SyncBarrierMS int64 `json:"sync_barrier_ms"` + CancelPropagationMS int64 `json:"cancel_propagation_ms"` + ExecuteRPCCount int `json:"execute_rpc_count"` + MulticastRPCCount int `json:"multicast_rpc_count"` + SequentialRPCCount int `json:"sequential_rpc_count"` + MakespanMS int64 `json:"makespan_ms"` + ContextPushP95MS int64 `json:"context_push_p95_ms"` + ContextPushOps int `json:"context_push_ops"` + CoordMissingResponses int `json:"coord_missing_responses"` + CoordDeadlineMisses int `json:"coord_deadline_misses"` + CoordBytesSent int64 `json:"coord_bytes_sent"` + CoordTimeSharePct float64 `json:"coord_time_share_pct"` + RoundBudgetMS int64 `json:"round_budget_ms"` + Success bool `json:"success"` + Error string `json:"error,omitempty"` +} + +func WriteJSON(path string, result RunResult) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + data, err := json.MarshalIndent(result, "", " ") + if err != nil { + return err + } + return os.WriteFile(path, data, 0o644) +} + +func AppendTSV(path string, result RunResult) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + writeHeader := true + if info, err := os.Stat(path); err == nil { + writeHeader = info.Size() == 0 + } else if !os.IsNotExist(err) { + return err + } + + file, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + return err + } + defer file.Close() + + w := csv.NewWriter(file) + w.Comma = '\t' + if writeHeader { + if err := w.Write([]string{ + "plan_name", "domain", "implementation", "agents", "tasks", + "total_wall_clock_ms", "tasks_completed", "tasks_failed", "tasks_timed_out", + "tasks_cancelled", "obsolete_tasks_completed", "retries_attempted", "retries_succeeded", + "context_push_ms", "sync_barrier_ms", "cancel_propagation_ms", + "execute_rpc_count", "multicast_rpc_count", "sequential_rpc_count", "makespan_ms", + "context_push_p95_ms", "context_push_ops", "coord_missing_responses", "coord_deadline_misses", + "coord_bytes_sent", "coord_time_share_pct", "round_budget_ms", + "success", "error", + }); err != nil { + return err + } + } + if err := w.Write([]string{ + result.PlanName, + result.Domain, + result.Implementation, + strconv.Itoa(result.Agents), + strconv.Itoa(result.Tasks), + strconv.FormatInt(result.TotalWallClockMS, 10), + strconv.Itoa(result.TasksCompleted), + strconv.Itoa(result.TasksFailed), + strconv.Itoa(result.TasksTimedOut), + strconv.Itoa(result.TasksCancelled), + strconv.Itoa(result.ObsoleteTasksCompleted), + strconv.Itoa(result.RetriesAttempted), + strconv.Itoa(result.RetriesSucceeded), + strconv.FormatInt(result.ContextPushMS, 10), + strconv.FormatInt(result.SyncBarrierMS, 10), + strconv.FormatInt(result.CancelPropagationMS, 10), + strconv.Itoa(result.ExecuteRPCCount), + strconv.Itoa(result.MulticastRPCCount), + strconv.Itoa(result.SequentialRPCCount), + strconv.FormatInt(result.MakespanMS, 10), + strconv.FormatInt(result.ContextPushP95MS, 10), + strconv.Itoa(result.ContextPushOps), + strconv.Itoa(result.CoordMissingResponses), + strconv.Itoa(result.CoordDeadlineMisses), + strconv.FormatInt(result.CoordBytesSent, 10), + strconv.FormatFloat(result.CoordTimeSharePct, 'f', 1, 64), + strconv.FormatInt(result.RoundBudgetMS, 10), + strconv.FormatBool(result.Success), + result.Error, + }); err != nil { + return err + } + w.Flush() + return w.Error() +} + +func WriteSummaryMarkdown(path string, results []RunResult) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + var b []byte + b = append(b, "# SLIM vs A2A Comparison Summary\n\n"...) + b = append(b, "| Plan | Implementation | Wall (ms) | Context push (ms) | Cancel prop (ms) | Completed | Failed | Cancelled | Success |\n"...) + b = append(b, "| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |\n"...) + for _, r := range results { + b = append(b, fmt.Sprintf("| %s | %s | %d | %d | %d | %d | %d | %d | %t |\n", + r.PlanName, r.Implementation, r.TotalWallClockMS, r.ContextPushMS, r.CancelPropagationMS, + r.TasksCompleted, r.TasksFailed, r.TasksCancelled, r.Success)...) + } + return os.WriteFile(path, b, 0o644) +} diff --git a/benchmarks/slim-vs-a2a/internal/plan/generator.go b/benchmarks/slim-vs-a2a/internal/plan/generator.go new file mode 100644 index 00000000..fc921b91 --- /dev/null +++ b/benchmarks/slim-vs-a2a/internal/plan/generator.go @@ -0,0 +1,231 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package plan + +import ( + "fmt" +) + +// CanonicalFamily identifies paper-aligned micro-plan templates. +type CanonicalFamily string + +const ( + FamilyPreferenceAggregation CanonicalFamily = "preference-aggregation" + FamilySupplyChainCascade CanonicalFamily = "supply-chain-cascade" + FamilySustainableResource CanonicalFamily = "sustainable-resource" +) + +// GenerateOptions configures generated execution plans for sweeps. +type GenerateOptions struct { + Family CanonicalFamily + Agents int + FanOut int + RoundBudgetMS int64 + ExecuteSleepSec float64 + Rounds int +} + +// GenerateCanonical builds an ExecutionPlan for transport/coordination sweeps. +func GenerateCanonical(opts GenerateOptions) (*ExecutionPlan, error) { + if opts.Agents < 2 { + return nil, fmt.Errorf("agents must be >= 2") + } + if opts.ExecuteSleepSec <= 0 { + opts.ExecuteSleepSec = 0.01 + } + if opts.Rounds <= 0 { + opts.Rounds = 1 + } + + switch opts.Family { + case FamilyPreferenceAggregation: + return generatePreferenceAggregation(opts) + case FamilySupplyChainCascade: + return generateSupplyChainCascade(opts) + case FamilySustainableResource: + return generateSustainableResource(opts) + default: + return nil, fmt.Errorf("unknown family %q", opts.Family) + } +} + +func generatePreferenceAggregation(opts GenerateOptions) (*ExecutionPlan, error) { + n := opts.Agents + p := basePlan(string(FamilyPreferenceAggregation), "canonical-preference", opts) + agents := make([]Agent, n) + for i := 0; i < n; i++ { + agents[i] = agentAt(i, 9400+i*10) + } + p.Agents = agents + + var tasks []Task + for i := 0; i < n; i++ { + deps := []string{} + if i > 0 { + deps = []string{fmt.Sprintf("propose-%d", i-1)} + } + tasks = append(tasks, Task{ + ID: fmt.Sprintf("propose-%d", i), + Name: fmt.Sprintf("Agent %d local proposal", i), + Agent: agents[i].ID, + DependsOn: deps, + CompletionTimeSec: opts.ExecuteSleepSec, + MaxCompletionTimeSec: opts.ExecuteSleepSec * 4, + Output: fmt.Sprintf("proposal=score_%d", i*2), + }) + } + p.Tasks = tasks + + lastTask := tasks[len(tasks)-1].ID + targets := agentIDs(agents) + if opts.FanOut > 0 && opts.FanOut < len(targets) { + targets = targets[:opts.FanOut] + } + p.ContextUpdates = []ContextUpdate{{ + AfterTask: lastTask, + Payload: "sync=phase=round-finalize proposals=aggregated", + TargetAgents: targets, + }} + return p, p.Validate() +} + +func generateSupplyChainCascade(opts GenerateOptions) (*ExecutionPlan, error) { + n := opts.Agents + p := basePlan(string(FamilySupplyChainCascade), "canonical-supply-chain", opts) + agents := make([]Agent, n) + for i := 0; i < n; i++ { + agents[i] = agentAt(i, 9500+i*10) + } + p.Agents = agents + + var tasks []Task + for i := 0; i < n; i++ { + deps := []string{} + if i > 0 { + deps = []string{fmt.Sprintf("stage-%d", i-1)} + } + tasks = append(tasks, Task{ + ID: fmt.Sprintf("stage-%d", i), + Name: fmt.Sprintf("Supply chain stage %d order", i), + Agent: agents[i].ID, + DependsOn: deps, + CompletionTimeSec: opts.ExecuteSleepSec, + MaxCompletionTimeSec: opts.ExecuteSleepSec * 4, + Output: fmt.Sprintf("order=units_%d", 4+i), + }) + } + p.Tasks = tasks + + // Per-hop context to downstream neighbor(s). + for i := 0; i < n-1; i++ { + targets := []string{agents[i+1].ID} + p.ContextUpdates = append(p.ContextUpdates, ContextUpdate{ + AfterTask: fmt.Sprintf("stage-%d", i), + Payload: fmt.Sprintf("epoch=%d stock=shared", i+1), + TargetAgents: targets, + }) + } + return p, p.Validate() +} + +func generateSustainableResource(opts GenerateOptions) (*ExecutionPlan, error) { + n := opts.Agents + p := basePlan(string(FamilySustainableResource), "canonical-sustainable-resource", opts) + agents := make([]Agent, n) + for i := 0; i < n; i++ { + agents[i] = agentAt(i, 9600+i*10) + } + p.Agents = agents + + allTargets := agentIDs(agents) + fanOut := len(allTargets) + if opts.FanOut > 0 && opts.FanOut < fanOut { + fanOut = opts.FanOut + } + targets := allTargets[:fanOut] + + var tasks []Task + for r := 0; r < opts.Rounds; r++ { + for i := 0; i < n; i++ { + id := fmt.Sprintf("extract-r%d-a%d", r, i) + deps := []string{} + if r > 0 { + deps = append(deps, fmt.Sprintf("extract-r%d-a%d", r-1, n-1)) + } else if i > 0 { + deps = append(deps, fmt.Sprintf("extract-r%d-a%d", r, i-1)) + } + tasks = append(tasks, Task{ + ID: id, + Name: fmt.Sprintf("Round %d agent %d extraction", r, i), + Agent: agents[i].ID, + DependsOn: deps, + CompletionTimeSec: opts.ExecuteSleepSec, + MaxCompletionTimeSec: opts.ExecuteSleepSec * 4, + Output: fmt.Sprintf("extract=2.0 round=%d", r), + }) + } + // Shared stock broadcast after each round's last agent completes. + p.ContextUpdates = append(p.ContextUpdates, ContextUpdate{ + AfterTask: fmt.Sprintf("extract-r%d-a%d", r, n-1), + Payload: fmt.Sprintf("round=%d stock=shared shadow_price=0.35", r), + TargetAgents: append([]string(nil), targets...), + }) + } + p.Tasks = tasks + return p, p.Validate() +} + +func basePlan(domain, name string, opts GenerateOptions) *ExecutionPlan { + return &ExecutionPlan{ + APIVersion: "bench.agntcy.io/v1", + Kind: "ExecutionPlan", + Metadata: Metadata{ + Name: fmt.Sprintf("%s-%dagents", name, opts.Agents), + Domain: domain, + Description: "Generated canonical plan for transport sweeps", + }, + Spec: Spec{ + Defaults: TaskTiming{ + CompletionTimeSec: opts.ExecuteSleepSec, + MaxCompletionTimeSec: opts.ExecuteSleepSec * 4, + }, + RoundBudgetMS: opts.RoundBudgetMS, + Sweep: &SweepSpec{ + Family: string(opts.Family), + Agents: opts.Agents, + FanOut: opts.FanOut, + RoundBudgetMS: opts.RoundBudgetMS, + ExecuteSleepSec: opts.ExecuteSleepSec, + }, + }, + } +} + +func agentAt(index, portBase int) Agent { + id := fmt.Sprintf("agent-%d", index) + return Agent{ + ID: id, + SlimName: fmt.Sprintf("agntcy/bench/%s", id), + A2APort: portBase + index, + } +} + +func agentIDs(agents []Agent) []string { + ids := make([]string, len(agents)) + for i, a := range agents { + ids[i] = a.ID + } + return ids +} + +// EffectiveRoundBudgetMS returns runner override, plan spec, or zero. +func (p *ExecutionPlan) EffectiveRoundBudgetMS(override int64) int64 { + if override > 0 { + return override + } + if p.Spec.RoundBudgetMS > 0 { + return p.Spec.RoundBudgetMS + } + return 0 +} diff --git a/benchmarks/slim-vs-a2a/internal/plan/generator_test.go b/benchmarks/slim-vs-a2a/internal/plan/generator_test.go new file mode 100644 index 00000000..dfd247c7 --- /dev/null +++ b/benchmarks/slim-vs-a2a/internal/plan/generator_test.go @@ -0,0 +1,46 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package plan_test + +import ( + "testing" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" +) + +func TestGenerateCanonicalFamilies(t *testing.T) { + families := []plan.CanonicalFamily{ + plan.FamilyPreferenceAggregation, + plan.FamilySupplyChainCascade, + plan.FamilySustainableResource, + } + for _, family := range families { + p, err := plan.GenerateCanonical(plan.GenerateOptions{ + Family: family, + Agents: 5, + RoundBudgetMS: 20, + ExecuteSleepSec: 0.01, + Rounds: 1, + }) + if err != nil { + t.Fatalf("family %s: %v", family, err) + } + if len(p.Agents) != 5 { + t.Fatalf("family %s: expected 5 agents, got %d", family, len(p.Agents)) + } + if p.Spec.RoundBudgetMS != 20 { + t.Fatalf("family %s: round budget not set", family) + } + } +} + +func TestEffectiveRoundBudgetMS(t *testing.T) { + p := &plan.ExecutionPlan{Spec: plan.Spec{RoundBudgetMS: 15}} + if got := p.EffectiveRoundBudgetMS(0); got != 15 { + t.Fatalf("expected 15, got %d", got) + } + if got := p.EffectiveRoundBudgetMS(10); got != 10 { + t.Fatalf("override expected 10, got %d", got) + } +} diff --git a/benchmarks/slim-vs-a2a/internal/plan/plan.go b/benchmarks/slim-vs-a2a/internal/plan/plan.go new file mode 100644 index 00000000..b42a9c85 --- /dev/null +++ b/benchmarks/slim-vs-a2a/internal/plan/plan.go @@ -0,0 +1,295 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package plan + +import ( + "fmt" + "os" + "path/filepath" + "strings" + + "gopkg.in/yaml.v3" +) + +type ExecutionPlan struct { + APIVersion string `yaml:"apiVersion"` + Kind string `yaml:"kind"` + Metadata Metadata `yaml:"metadata"` + Spec Spec `yaml:"spec"` + Agents []Agent `yaml:"agents"` + Tasks []Task `yaml:"tasks"` + ContextUpdates []ContextUpdate `yaml:"contextUpdates"` +} + +type Metadata struct { + Name string `yaml:"name"` + Domain string `yaml:"domain"` + Description string `yaml:"description"` +} + +type Spec struct { + Defaults TaskTiming `yaml:"defaults"` + MaxRetries int `yaml:"maxRetries"` + RoundBudgetMS int64 `yaml:"roundBudgetMs"` + Sweep *SweepSpec `yaml:"sweep,omitempty"` +} + +// SweepSpec records parameters used to generate or classify sweep plans. +type SweepSpec struct { + Family string `yaml:"family,omitempty"` + Agents int `yaml:"agents,omitempty"` + FanOut int `yaml:"fanOut,omitempty"` + RoundBudgetMS int64 `yaml:"roundBudgetMs,omitempty"` + ExecuteSleepSec float64 `yaml:"executeSleepSec,omitempty"` +} + +type Agent struct { + ID string `yaml:"id"` + SlimName string `yaml:"slimName"` + A2APort int `yaml:"a2aPort"` +} + +type Task struct { + ID string `yaml:"id"` + Name string `yaml:"name"` + Agent string `yaml:"agent"` + DependsOn []string `yaml:"dependsOn"` + CompletionTimeSec float64 `yaml:"completionTimeSec"` + MaxCompletionTimeSec float64 `yaml:"maxCompletionTimeSec"` + InjectFailure bool `yaml:"injectFailure"` + Output string `yaml:"output"` +} + +type TaskTiming struct { + CompletionTimeSec float64 `yaml:"completionTimeSec"` + MaxCompletionTimeSec float64 `yaml:"maxCompletionTimeSec"` +} + +type ContextUpdate struct { + AfterTask string `yaml:"afterTask"` + Payload string `yaml:"payload"` + TargetAgents []string `yaml:"targetAgents"` +} + +func LoadFile(path string) (*ExecutionPlan, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var p ExecutionPlan + if err := yaml.Unmarshal(data, &p); err != nil { + return nil, fmt.Errorf("parse %s: %w", path, err) + } + if err := p.Validate(); err != nil { + return nil, fmt.Errorf("validate %s: %w", filepath.Base(path), err) + } + p.applyDefaults() + return &p, nil +} + +// WriteFile marshals an execution plan to a YAML file. +func WriteFile(path string, p *ExecutionPlan) error { + if err := p.Validate(); err != nil { + return err + } + data, err := yaml.Marshal(p) + if err != nil { + return err + } + if dir := filepath.Dir(path); dir != "" && dir != "." { + if err := os.MkdirAll(dir, 0o755); err != nil { + return err + } + } + return os.WriteFile(path, data, 0o644) +} + +func LoadDir(dir string) ([]*ExecutionPlan, error) { + entries, err := os.ReadDir(dir) + if err != nil { + return nil, err + } + var plans []*ExecutionPlan + for _, entry := range entries { + if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".yaml") { + continue + } + p, err := LoadFile(filepath.Join(dir, entry.Name())) + if err != nil { + return nil, err + } + plans = append(plans, p) + } + if len(plans) == 0 { + return nil, fmt.Errorf("no plan yaml files in %s", dir) + } + return plans, nil +} + +func (p *ExecutionPlan) applyDefaults() { + for i := range p.Tasks { + if p.Tasks[i].CompletionTimeSec == 0 { + p.Tasks[i].CompletionTimeSec = p.Spec.Defaults.CompletionTimeSec + } + if p.Tasks[i].MaxCompletionTimeSec == 0 { + p.Tasks[i].MaxCompletionTimeSec = p.Spec.Defaults.MaxCompletionTimeSec + } + } +} + +func (p *ExecutionPlan) Validate() error { + if p.Metadata.Name == "" { + return fmt.Errorf("metadata.name is required") + } + if len(p.Agents) == 0 { + return fmt.Errorf("at least one agent is required") + } + if len(p.Tasks) == 0 { + return fmt.Errorf("at least one task is required") + } + + agentIDs := map[string]Agent{} + for _, a := range p.Agents { + if a.ID == "" { + return fmt.Errorf("agent id is required") + } + if _, ok := agentIDs[a.ID]; ok { + return fmt.Errorf("duplicate agent id %q", a.ID) + } + agentIDs[a.ID] = a + } + + taskIDs := map[string]struct{}{} + for _, t := range p.Tasks { + if t.ID == "" { + return fmt.Errorf("task id is required") + } + if _, ok := taskIDs[t.ID]; ok { + return fmt.Errorf("duplicate task id %q", t.ID) + } + taskIDs[t.ID] = struct{}{} + } + + for _, t := range p.Tasks { + if _, ok := agentIDs[t.Agent]; !ok { + return fmt.Errorf("task %q references unknown agent %q", t.ID, t.Agent) + } + for _, dep := range t.DependsOn { + if _, ok := taskIDs[dep]; !ok { + return fmt.Errorf("task %q depends on unknown task %q", t.ID, dep) + } + } + if t.MaxCompletionTimeSec > 0 && t.MaxCompletionTimeSec < t.CompletionTimeSec { + return fmt.Errorf("task %q maxCompletionTimeSec < completionTimeSec", t.ID) + } + } + + for _, cu := range p.ContextUpdates { + if _, ok := taskIDs[cu.AfterTask]; !ok { + return fmt.Errorf("contextUpdate references unknown task %q", cu.AfterTask) + } + for _, agentID := range cu.TargetAgents { + if _, ok := agentIDs[agentID]; !ok { + return fmt.Errorf("contextUpdate references unknown agent %q", agentID) + } + } + } + + if err := detectCycle(p.Tasks); err != nil { + return err + } + return nil +} + +func (p *ExecutionPlan) AgentByID(id string) (Agent, bool) { + for _, a := range p.Agents { + if a.ID == id { + return a, true + } + } + return Agent{}, false +} + +func (p *ExecutionPlan) TaskByID(id string) (Task, bool) { + for _, t := range p.Tasks { + if t.ID == id { + return t, true + } + } + return Task{}, false +} + +func (p *ExecutionPlan) ContextUpdatesAfter(taskID string) []ContextUpdate { + var out []ContextUpdate + for _, cu := range p.ContextUpdates { + if cu.AfterTask == taskID { + out = append(out, cu) + } + } + return out +} + +func (p *ExecutionPlan) CardPort(agent Agent) int { + return agent.A2APort - 1000 +} + +func (p *ExecutionPlan) CardBaseURL(agent Agent) string { + return fmt.Sprintf("http://127.0.0.1:%d", p.CardPort(agent)) +} + +func detectCycle(tasks []Task) error { + graph := map[string][]string{} + for _, t := range tasks { + graph[t.ID] = append([]string(nil), t.DependsOn...) + } + visiting := map[string]bool{} + visited := map[string]bool{} + var dfs func(string) error + dfs = func(id string) error { + if visiting[id] { + return fmt.Errorf("cycle detected at task %q", id) + } + if visited[id] { + return nil + } + visiting[id] = true + for _, dep := range graph[id] { + if err := dfs(dep); err != nil { + return err + } + } + visiting[id] = false + visited[id] = true + return nil + } + for id := range graph { + if err := dfs(id); err != nil { + return err + } + } + return nil +} + +func DownstreamTasks(tasks []Task, rootID string) map[string]struct{} { + dependents := map[string][]string{} + for _, t := range tasks { + for _, dep := range t.DependsOn { + dependents[dep] = append(dependents[dep], t.ID) + } + } + out := map[string]struct{}{} + queue := []string{rootID} + for len(queue) > 0 { + cur := queue[0] + queue = queue[1:] + for _, child := range dependents[cur] { + if _, ok := out[child]; ok { + continue + } + out[child] = struct{}{} + queue = append(queue, child) + } + } + return out +} diff --git a/benchmarks/agntcy-multi-agent/plans/README.md b/benchmarks/slim-vs-a2a/plans/README.md similarity index 100% rename from benchmarks/agntcy-multi-agent/plans/README.md rename to benchmarks/slim-vs-a2a/plans/README.md diff --git a/benchmarks/agntcy-multi-agent/plans/domains/k8s-incident-response.yaml b/benchmarks/slim-vs-a2a/plans/domains/k8s-incident-response.yaml similarity index 100% rename from benchmarks/agntcy-multi-agent/plans/domains/k8s-incident-response.yaml rename to benchmarks/slim-vs-a2a/plans/domains/k8s-incident-response.yaml diff --git a/benchmarks/agntcy-multi-agent/plans/domains/mobile-nav-assistant.yaml b/benchmarks/slim-vs-a2a/plans/domains/mobile-nav-assistant.yaml similarity index 100% rename from benchmarks/agntcy-multi-agent/plans/domains/mobile-nav-assistant.yaml rename to benchmarks/slim-vs-a2a/plans/domains/mobile-nav-assistant.yaml diff --git a/benchmarks/agntcy-multi-agent/plans/domains/urban-traffic-reroute.yaml b/benchmarks/slim-vs-a2a/plans/domains/urban-traffic-reroute.yaml similarity index 100% rename from benchmarks/agntcy-multi-agent/plans/domains/urban-traffic-reroute.yaml rename to benchmarks/slim-vs-a2a/plans/domains/urban-traffic-reroute.yaml diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-10ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-10ms.yaml new file mode 100644 index 00000000..5c483fba --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-10ms.yaml @@ -0,0 +1,152 @@ +apiVersion: bench.agntcy.io/v1 +kind: ExecutionPlan +metadata: + name: canonical-sustainable-resource-10agents + domain: sustainable-resource + description: Generated canonical plan for transport sweeps +spec: + defaults: + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + maxRetries: 0 + roundBudgetMs: 10 + sweep: + family: sustainable-resource + agents: 10 + roundBudgetMs: 10 + executeSleepSec: 0.01 +agents: + - id: agent-0 + slimName: agntcy/bench/agent-0 + a2aPort: 9600 + - id: agent-1 + slimName: agntcy/bench/agent-1 + a2aPort: 9611 + - id: agent-2 + slimName: agntcy/bench/agent-2 + a2aPort: 9622 + - id: agent-3 + slimName: agntcy/bench/agent-3 + a2aPort: 9633 + - id: agent-4 + slimName: agntcy/bench/agent-4 + a2aPort: 9644 + - id: agent-5 + slimName: agntcy/bench/agent-5 + a2aPort: 9655 + - id: agent-6 + slimName: agntcy/bench/agent-6 + a2aPort: 9666 + - id: agent-7 + slimName: agntcy/bench/agent-7 + a2aPort: 9677 + - id: agent-8 + slimName: agntcy/bench/agent-8 + a2aPort: 9688 + - id: agent-9 + slimName: agntcy/bench/agent-9 + a2aPort: 9699 +tasks: + - id: extract-r0-a0 + name: Round 0 agent 0 extraction + agent: agent-0 + dependsOn: [] + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a1 + name: Round 0 agent 1 extraction + agent: agent-1 + dependsOn: + - extract-r0-a0 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a2 + name: Round 0 agent 2 extraction + agent: agent-2 + dependsOn: + - extract-r0-a1 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a3 + name: Round 0 agent 3 extraction + agent: agent-3 + dependsOn: + - extract-r0-a2 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a4 + name: Round 0 agent 4 extraction + agent: agent-4 + dependsOn: + - extract-r0-a3 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a5 + name: Round 0 agent 5 extraction + agent: agent-5 + dependsOn: + - extract-r0-a4 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a6 + name: Round 0 agent 6 extraction + agent: agent-6 + dependsOn: + - extract-r0-a5 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a7 + name: Round 0 agent 7 extraction + agent: agent-7 + dependsOn: + - extract-r0-a6 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a8 + name: Round 0 agent 8 extraction + agent: agent-8 + dependsOn: + - extract-r0-a7 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a9 + name: Round 0 agent 9 extraction + agent: agent-9 + dependsOn: + - extract-r0-a8 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 +contextUpdates: + - afterTask: extract-r0-a9 + payload: round=0 stock=shared shadow_price=0.35 + targetAgents: + - agent-0 + - agent-1 + - agent-2 + - agent-3 + - agent-4 + - agent-5 + - agent-6 + - agent-7 + - agent-8 + - agent-9 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-20ms.yaml new file mode 100644 index 00000000..5f35eff4 --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-20ms.yaml @@ -0,0 +1,152 @@ +apiVersion: bench.agntcy.io/v1 +kind: ExecutionPlan +metadata: + name: canonical-sustainable-resource-10agents + domain: sustainable-resource + description: Generated canonical plan for transport sweeps +spec: + defaults: + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + maxRetries: 0 + roundBudgetMs: 20 + sweep: + family: sustainable-resource + agents: 10 + roundBudgetMs: 20 + executeSleepSec: 0.01 +agents: + - id: agent-0 + slimName: agntcy/bench/agent-0 + a2aPort: 9600 + - id: agent-1 + slimName: agntcy/bench/agent-1 + a2aPort: 9611 + - id: agent-2 + slimName: agntcy/bench/agent-2 + a2aPort: 9622 + - id: agent-3 + slimName: agntcy/bench/agent-3 + a2aPort: 9633 + - id: agent-4 + slimName: agntcy/bench/agent-4 + a2aPort: 9644 + - id: agent-5 + slimName: agntcy/bench/agent-5 + a2aPort: 9655 + - id: agent-6 + slimName: agntcy/bench/agent-6 + a2aPort: 9666 + - id: agent-7 + slimName: agntcy/bench/agent-7 + a2aPort: 9677 + - id: agent-8 + slimName: agntcy/bench/agent-8 + a2aPort: 9688 + - id: agent-9 + slimName: agntcy/bench/agent-9 + a2aPort: 9699 +tasks: + - id: extract-r0-a0 + name: Round 0 agent 0 extraction + agent: agent-0 + dependsOn: [] + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a1 + name: Round 0 agent 1 extraction + agent: agent-1 + dependsOn: + - extract-r0-a0 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a2 + name: Round 0 agent 2 extraction + agent: agent-2 + dependsOn: + - extract-r0-a1 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a3 + name: Round 0 agent 3 extraction + agent: agent-3 + dependsOn: + - extract-r0-a2 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a4 + name: Round 0 agent 4 extraction + agent: agent-4 + dependsOn: + - extract-r0-a3 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a5 + name: Round 0 agent 5 extraction + agent: agent-5 + dependsOn: + - extract-r0-a4 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a6 + name: Round 0 agent 6 extraction + agent: agent-6 + dependsOn: + - extract-r0-a5 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a7 + name: Round 0 agent 7 extraction + agent: agent-7 + dependsOn: + - extract-r0-a6 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a8 + name: Round 0 agent 8 extraction + agent: agent-8 + dependsOn: + - extract-r0-a7 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a9 + name: Round 0 agent 9 extraction + agent: agent-9 + dependsOn: + - extract-r0-a8 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 +contextUpdates: + - afterTask: extract-r0-a9 + payload: round=0 stock=shared shadow_price=0.35 + targetAgents: + - agent-0 + - agent-1 + - agent-2 + - agent-3 + - agent-4 + - agent-5 + - agent-6 + - agent-7 + - agent-8 + - agent-9 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-10ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-10ms.yaml new file mode 100644 index 00000000..63faa3da --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-10ms.yaml @@ -0,0 +1,243 @@ +apiVersion: bench.agntcy.io/v1 +kind: ExecutionPlan +metadata: + name: canonical-sustainable-resource-17agents + domain: sustainable-resource + description: Generated canonical plan for transport sweeps +spec: + defaults: + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + maxRetries: 0 + roundBudgetMs: 10 + sweep: + family: sustainable-resource + agents: 17 + roundBudgetMs: 10 + executeSleepSec: 0.01 +agents: + - id: agent-0 + slimName: agntcy/bench/agent-0 + a2aPort: 9600 + - id: agent-1 + slimName: agntcy/bench/agent-1 + a2aPort: 9611 + - id: agent-2 + slimName: agntcy/bench/agent-2 + a2aPort: 9622 + - id: agent-3 + slimName: agntcy/bench/agent-3 + a2aPort: 9633 + - id: agent-4 + slimName: agntcy/bench/agent-4 + a2aPort: 9644 + - id: agent-5 + slimName: agntcy/bench/agent-5 + a2aPort: 9655 + - id: agent-6 + slimName: agntcy/bench/agent-6 + a2aPort: 9666 + - id: agent-7 + slimName: agntcy/bench/agent-7 + a2aPort: 9677 + - id: agent-8 + slimName: agntcy/bench/agent-8 + a2aPort: 9688 + - id: agent-9 + slimName: agntcy/bench/agent-9 + a2aPort: 9699 + - id: agent-10 + slimName: agntcy/bench/agent-10 + a2aPort: 9710 + - id: agent-11 + slimName: agntcy/bench/agent-11 + a2aPort: 9721 + - id: agent-12 + slimName: agntcy/bench/agent-12 + a2aPort: 9732 + - id: agent-13 + slimName: agntcy/bench/agent-13 + a2aPort: 9743 + - id: agent-14 + slimName: agntcy/bench/agent-14 + a2aPort: 9754 + - id: agent-15 + slimName: agntcy/bench/agent-15 + a2aPort: 9765 + - id: agent-16 + slimName: agntcy/bench/agent-16 + a2aPort: 9776 +tasks: + - id: extract-r0-a0 + name: Round 0 agent 0 extraction + agent: agent-0 + dependsOn: [] + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a1 + name: Round 0 agent 1 extraction + agent: agent-1 + dependsOn: + - extract-r0-a0 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a2 + name: Round 0 agent 2 extraction + agent: agent-2 + dependsOn: + - extract-r0-a1 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a3 + name: Round 0 agent 3 extraction + agent: agent-3 + dependsOn: + - extract-r0-a2 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a4 + name: Round 0 agent 4 extraction + agent: agent-4 + dependsOn: + - extract-r0-a3 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a5 + name: Round 0 agent 5 extraction + agent: agent-5 + dependsOn: + - extract-r0-a4 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a6 + name: Round 0 agent 6 extraction + agent: agent-6 + dependsOn: + - extract-r0-a5 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a7 + name: Round 0 agent 7 extraction + agent: agent-7 + dependsOn: + - extract-r0-a6 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a8 + name: Round 0 agent 8 extraction + agent: agent-8 + dependsOn: + - extract-r0-a7 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a9 + name: Round 0 agent 9 extraction + agent: agent-9 + dependsOn: + - extract-r0-a8 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a10 + name: Round 0 agent 10 extraction + agent: agent-10 + dependsOn: + - extract-r0-a9 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a11 + name: Round 0 agent 11 extraction + agent: agent-11 + dependsOn: + - extract-r0-a10 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a12 + name: Round 0 agent 12 extraction + agent: agent-12 + dependsOn: + - extract-r0-a11 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a13 + name: Round 0 agent 13 extraction + agent: agent-13 + dependsOn: + - extract-r0-a12 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a14 + name: Round 0 agent 14 extraction + agent: agent-14 + dependsOn: + - extract-r0-a13 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a15 + name: Round 0 agent 15 extraction + agent: agent-15 + dependsOn: + - extract-r0-a14 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a16 + name: Round 0 agent 16 extraction + agent: agent-16 + dependsOn: + - extract-r0-a15 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 +contextUpdates: + - afterTask: extract-r0-a16 + payload: round=0 stock=shared shadow_price=0.35 + targetAgents: + - agent-0 + - agent-1 + - agent-2 + - agent-3 + - agent-4 + - agent-5 + - agent-6 + - agent-7 + - agent-8 + - agent-9 + - agent-10 + - agent-11 + - agent-12 + - agent-13 + - agent-14 + - agent-15 + - agent-16 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-20ms.yaml new file mode 100644 index 00000000..3fa68a74 --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-20ms.yaml @@ -0,0 +1,243 @@ +apiVersion: bench.agntcy.io/v1 +kind: ExecutionPlan +metadata: + name: canonical-sustainable-resource-17agents + domain: sustainable-resource + description: Generated canonical plan for transport sweeps +spec: + defaults: + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + maxRetries: 0 + roundBudgetMs: 20 + sweep: + family: sustainable-resource + agents: 17 + roundBudgetMs: 20 + executeSleepSec: 0.01 +agents: + - id: agent-0 + slimName: agntcy/bench/agent-0 + a2aPort: 9600 + - id: agent-1 + slimName: agntcy/bench/agent-1 + a2aPort: 9611 + - id: agent-2 + slimName: agntcy/bench/agent-2 + a2aPort: 9622 + - id: agent-3 + slimName: agntcy/bench/agent-3 + a2aPort: 9633 + - id: agent-4 + slimName: agntcy/bench/agent-4 + a2aPort: 9644 + - id: agent-5 + slimName: agntcy/bench/agent-5 + a2aPort: 9655 + - id: agent-6 + slimName: agntcy/bench/agent-6 + a2aPort: 9666 + - id: agent-7 + slimName: agntcy/bench/agent-7 + a2aPort: 9677 + - id: agent-8 + slimName: agntcy/bench/agent-8 + a2aPort: 9688 + - id: agent-9 + slimName: agntcy/bench/agent-9 + a2aPort: 9699 + - id: agent-10 + slimName: agntcy/bench/agent-10 + a2aPort: 9710 + - id: agent-11 + slimName: agntcy/bench/agent-11 + a2aPort: 9721 + - id: agent-12 + slimName: agntcy/bench/agent-12 + a2aPort: 9732 + - id: agent-13 + slimName: agntcy/bench/agent-13 + a2aPort: 9743 + - id: agent-14 + slimName: agntcy/bench/agent-14 + a2aPort: 9754 + - id: agent-15 + slimName: agntcy/bench/agent-15 + a2aPort: 9765 + - id: agent-16 + slimName: agntcy/bench/agent-16 + a2aPort: 9776 +tasks: + - id: extract-r0-a0 + name: Round 0 agent 0 extraction + agent: agent-0 + dependsOn: [] + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a1 + name: Round 0 agent 1 extraction + agent: agent-1 + dependsOn: + - extract-r0-a0 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a2 + name: Round 0 agent 2 extraction + agent: agent-2 + dependsOn: + - extract-r0-a1 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a3 + name: Round 0 agent 3 extraction + agent: agent-3 + dependsOn: + - extract-r0-a2 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a4 + name: Round 0 agent 4 extraction + agent: agent-4 + dependsOn: + - extract-r0-a3 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a5 + name: Round 0 agent 5 extraction + agent: agent-5 + dependsOn: + - extract-r0-a4 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a6 + name: Round 0 agent 6 extraction + agent: agent-6 + dependsOn: + - extract-r0-a5 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a7 + name: Round 0 agent 7 extraction + agent: agent-7 + dependsOn: + - extract-r0-a6 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a8 + name: Round 0 agent 8 extraction + agent: agent-8 + dependsOn: + - extract-r0-a7 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a9 + name: Round 0 agent 9 extraction + agent: agent-9 + dependsOn: + - extract-r0-a8 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a10 + name: Round 0 agent 10 extraction + agent: agent-10 + dependsOn: + - extract-r0-a9 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a11 + name: Round 0 agent 11 extraction + agent: agent-11 + dependsOn: + - extract-r0-a10 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a12 + name: Round 0 agent 12 extraction + agent: agent-12 + dependsOn: + - extract-r0-a11 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a13 + name: Round 0 agent 13 extraction + agent: agent-13 + dependsOn: + - extract-r0-a12 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a14 + name: Round 0 agent 14 extraction + agent: agent-14 + dependsOn: + - extract-r0-a13 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a15 + name: Round 0 agent 15 extraction + agent: agent-15 + dependsOn: + - extract-r0-a14 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a16 + name: Round 0 agent 16 extraction + agent: agent-16 + dependsOn: + - extract-r0-a15 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 +contextUpdates: + - afterTask: extract-r0-a16 + payload: round=0 stock=shared shadow_price=0.35 + targetAgents: + - agent-0 + - agent-1 + - agent-2 + - agent-3 + - agent-4 + - agent-5 + - agent-6 + - agent-7 + - agent-8 + - agent-9 + - agent-10 + - agent-11 + - agent-12 + - agent-13 + - agent-14 + - agent-15 + - agent-16 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-10ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-10ms.yaml new file mode 100644 index 00000000..33b54646 --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-10ms.yaml @@ -0,0 +1,48 @@ +apiVersion: bench.agntcy.io/v1 +kind: ExecutionPlan +metadata: + name: canonical-sustainable-resource-2agents + domain: sustainable-resource + description: Generated canonical plan for transport sweeps +spec: + defaults: + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + maxRetries: 0 + roundBudgetMs: 10 + sweep: + family: sustainable-resource + agents: 2 + roundBudgetMs: 10 + executeSleepSec: 0.01 +agents: + - id: agent-0 + slimName: agntcy/bench/agent-0 + a2aPort: 9600 + - id: agent-1 + slimName: agntcy/bench/agent-1 + a2aPort: 9611 +tasks: + - id: extract-r0-a0 + name: Round 0 agent 0 extraction + agent: agent-0 + dependsOn: [] + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a1 + name: Round 0 agent 1 extraction + agent: agent-1 + dependsOn: + - extract-r0-a0 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 +contextUpdates: + - afterTask: extract-r0-a1 + payload: round=0 stock=shared shadow_price=0.35 + targetAgents: + - agent-0 + - agent-1 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-20ms.yaml new file mode 100644 index 00000000..44301ee5 --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-20ms.yaml @@ -0,0 +1,48 @@ +apiVersion: bench.agntcy.io/v1 +kind: ExecutionPlan +metadata: + name: canonical-sustainable-resource-2agents + domain: sustainable-resource + description: Generated canonical plan for transport sweeps +spec: + defaults: + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + maxRetries: 0 + roundBudgetMs: 20 + sweep: + family: sustainable-resource + agents: 2 + roundBudgetMs: 20 + executeSleepSec: 0.01 +agents: + - id: agent-0 + slimName: agntcy/bench/agent-0 + a2aPort: 9600 + - id: agent-1 + slimName: agntcy/bench/agent-1 + a2aPort: 9611 +tasks: + - id: extract-r0-a0 + name: Round 0 agent 0 extraction + agent: agent-0 + dependsOn: [] + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a1 + name: Round 0 agent 1 extraction + agent: agent-1 + dependsOn: + - extract-r0-a0 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 +contextUpdates: + - afterTask: extract-r0-a1 + payload: round=0 stock=shared shadow_price=0.35 + targetAgents: + - agent-0 + - agent-1 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-10ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-10ms.yaml new file mode 100644 index 00000000..7ecd542a --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-10ms.yaml @@ -0,0 +1,87 @@ +apiVersion: bench.agntcy.io/v1 +kind: ExecutionPlan +metadata: + name: canonical-sustainable-resource-5agents + domain: sustainable-resource + description: Generated canonical plan for transport sweeps +spec: + defaults: + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + maxRetries: 0 + roundBudgetMs: 10 + sweep: + family: sustainable-resource + agents: 5 + roundBudgetMs: 10 + executeSleepSec: 0.01 +agents: + - id: agent-0 + slimName: agntcy/bench/agent-0 + a2aPort: 9600 + - id: agent-1 + slimName: agntcy/bench/agent-1 + a2aPort: 9611 + - id: agent-2 + slimName: agntcy/bench/agent-2 + a2aPort: 9622 + - id: agent-3 + slimName: agntcy/bench/agent-3 + a2aPort: 9633 + - id: agent-4 + slimName: agntcy/bench/agent-4 + a2aPort: 9644 +tasks: + - id: extract-r0-a0 + name: Round 0 agent 0 extraction + agent: agent-0 + dependsOn: [] + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a1 + name: Round 0 agent 1 extraction + agent: agent-1 + dependsOn: + - extract-r0-a0 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a2 + name: Round 0 agent 2 extraction + agent: agent-2 + dependsOn: + - extract-r0-a1 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a3 + name: Round 0 agent 3 extraction + agent: agent-3 + dependsOn: + - extract-r0-a2 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a4 + name: Round 0 agent 4 extraction + agent: agent-4 + dependsOn: + - extract-r0-a3 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 +contextUpdates: + - afterTask: extract-r0-a4 + payload: round=0 stock=shared shadow_price=0.35 + targetAgents: + - agent-0 + - agent-1 + - agent-2 + - agent-3 + - agent-4 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-20ms.yaml new file mode 100644 index 00000000..175189a0 --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-20ms.yaml @@ -0,0 +1,87 @@ +apiVersion: bench.agntcy.io/v1 +kind: ExecutionPlan +metadata: + name: canonical-sustainable-resource-5agents + domain: sustainable-resource + description: Generated canonical plan for transport sweeps +spec: + defaults: + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + maxRetries: 0 + roundBudgetMs: 20 + sweep: + family: sustainable-resource + agents: 5 + roundBudgetMs: 20 + executeSleepSec: 0.01 +agents: + - id: agent-0 + slimName: agntcy/bench/agent-0 + a2aPort: 9600 + - id: agent-1 + slimName: agntcy/bench/agent-1 + a2aPort: 9611 + - id: agent-2 + slimName: agntcy/bench/agent-2 + a2aPort: 9622 + - id: agent-3 + slimName: agntcy/bench/agent-3 + a2aPort: 9633 + - id: agent-4 + slimName: agntcy/bench/agent-4 + a2aPort: 9644 +tasks: + - id: extract-r0-a0 + name: Round 0 agent 0 extraction + agent: agent-0 + dependsOn: [] + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a1 + name: Round 0 agent 1 extraction + agent: agent-1 + dependsOn: + - extract-r0-a0 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a2 + name: Round 0 agent 2 extraction + agent: agent-2 + dependsOn: + - extract-r0-a1 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a3 + name: Round 0 agent 3 extraction + agent: agent-3 + dependsOn: + - extract-r0-a2 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 + - id: extract-r0-a4 + name: Round 0 agent 4 extraction + agent: agent-4 + dependsOn: + - extract-r0-a3 + completionTimeSec: 0.01 + maxCompletionTimeSec: 0.04 + injectFailure: false + output: extract=2.0 round=0 +contextUpdates: + - afterTask: extract-r0-a4 + payload: round=0 stock=shared shadow_price=0.35 + targetAgents: + - agent-0 + - agent-1 + - agent-2 + - agent-3 + - agent-4 diff --git a/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go b/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go new file mode 100644 index 00000000..b08d1110 --- /dev/null +++ b/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go @@ -0,0 +1,68 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "flag" + "fmt" + "log" + "strings" + + slim "github.com/agntcy/slim-bindings-go" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/executor" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/server" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/slimrpc" +) + +const defaultSharedSecret = "demo-shared-secret-min-32-chars!!" + +func main() { + slimName := flag.String("slim-name", "", "SLIM identity org/group/app") + endpoint := flag.String("endpoint", "http://127.0.0.1:46357", "SLIM dataplane endpoint") + flag.Parse() + + if *slimName == "" { + log.Fatal("--slim-name is required") + } + + slim.InitializeWithDefaults() + service := slim.GetGlobalService() + + name, err := nameFromString(*slimName) + if err != nil { + log.Fatalf("parse name: %v", err) + } + + app, err := service.CreateAppWithSecret(name, defaultSharedSecret) + if err != nil { + log.Fatalf("create app: %v", err) + } + defer app.Destroy() + + connID, err := service.Connect(slim.NewInsecureClientConfig(*endpoint)) + if err != nil { + log.Fatalf("connect: %v", err) + } + if err := app.Subscribe(name, &connID); err != nil { + log.Fatalf("subscribe: %v", err) + } + + rpcServer := slim.ServerNewWithConnection(app, name, &connID) + rpcServer.RegisterUnaryUnary(slimrpc.ServiceName, slimrpc.MethodHandle, &server.Handler{ + Engine: executor.New(*slimName), + }) + + fmt.Printf("SLIM_AGENT_READY name=%s\n", *slimName) + if err := rpcServer.Serve(); err != nil { + log.Printf("slimrpc server: %v", err) + } +} + +func nameFromString(value string) (*slim.Name, error) { + parts := strings.Split(value, "/") + if len(parts) != 3 { + return nil, fmt.Errorf("invalid name format: %s", value) + } + return slim.NewName(parts[0], parts[1], parts[2]), nil +} diff --git a/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go b/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go new file mode 100644 index 00000000..4d3e8ca9 --- /dev/null +++ b/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go @@ -0,0 +1,124 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "log" + "os" + "os/exec" + "time" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/client" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/scheduler" +) + +func main() { + planPath := flag.String("plan", "", "path to execution plan yaml") + endpoint := flag.String("endpoint", "http://127.0.0.1:46357", "SLIM dataplane endpoint") + agentBin := flag.String("agent-bin", "", "path to slim agent binary") + outputJSON := flag.String("output-json", "", "write run metrics json") + outputTSV := flag.String("output-tsv", "", "append run metrics tsv") + waitReady := flag.Duration("wait-ready", 3*time.Second, "wait for agents to start") + roundBudgetMS := flag.Int64("round-budget-ms", 0, "coordination round budget in ms (overrides plan)") + coordMode := flag.String("coord-mode", "multicast", "coordination mode: multicast or unicast") + quiet := flag.Bool("quiet", false, "disable benchmark timing logs") + flag.Parse() + + if *quiet { + benchlog.SetEnabled(false) + } + + if *planPath == "" { + log.Fatal("--plan is required") + } + + p, err := plan.LoadFile(*planPath) + if err != nil { + log.Fatalf("load plan: %v", err) + } + + mode := client.CoordModeMulticast + if *coordMode == "unicast" { + mode = client.CoordModeUnicast + } + + agentPath := *agentBin + if agentPath == "" { + agentPath = os.Getenv("SLIM_AGENT_BIN") + } + if agentPath == "" { + log.Fatal("set --agent-bin or SLIM_AGENT_BIN") + } + + procs := startAgents(p, agentPath, *endpoint) + time.Sleep(*waitReady) + + cli, err := client.New(*endpoint, p, client.Config{ + RoundBudgetMS: p.EffectiveRoundBudgetMS(*roundBudgetMS), + CoordMode: mode, + }) + if err != nil { + stopAgents(procs) + log.Fatalf("client: %v", err) + } + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + + if err := cli.SetupGroup(ctx, p); err != nil { + cli.Close() + stopAgents(procs) + log.Fatalf("setup group: %v", err) + } + + result := scheduler.New(p, cli).Run(ctx) + + time.Sleep(200 * time.Millisecond) + cli.Close() + stopAgents(procs) + + if *outputJSON != "" { + if err := metrics.WriteJSON(*outputJSON, result); err != nil { + log.Fatalf("write json: %v", err) + } + } + if *outputTSV != "" { + if err := metrics.AppendTSV(*outputTSV, result); err != nil { + log.Fatalf("write tsv: %v", err) + } + } + + data, _ := json.MarshalIndent(result, "", " ") + fmt.Println(string(data)) +} + +func startAgents(p *plan.ExecutionPlan, agentBin, endpoint string) []*exec.Cmd { + var procs []*exec.Cmd + for _, agent := range p.Agents { + cmd := exec.Command(agentBin, "--slim-name", agent.SlimName, "--endpoint", endpoint) + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + if err := cmd.Start(); err != nil { + stopAgents(procs) + log.Fatalf("start agent %s: %v", agent.ID, err) + } + procs = append(procs, cmd) + } + return procs +} + +func stopAgents(procs []*exec.Cmd) { + for _, cmd := range procs { + if cmd.Process != nil { + _ = cmd.Process.Kill() + } + } +} diff --git a/benchmarks/slim-vs-a2a/slim/internal/client/client.go b/benchmarks/slim-vs-a2a/slim/internal/client/client.go new file mode 100644 index 00000000..cc853373 --- /dev/null +++ b/benchmarks/slim-vs-a2a/slim/internal/client/client.go @@ -0,0 +1,462 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package client + +import ( + "context" + "fmt" + "slices" + "strings" + "sync" + "time" + + slim "github.com/agntcy/slim-bindings-go" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/slimrpc" +) + +const defaultSharedSecret = "demo-shared-secret-min-32-chars!!" + +type CoordMode string + +const ( + CoordModeMulticast CoordMode = "multicast" + CoordModeUnicast CoordMode = "unicast" +) + +type Config struct { + RoundBudgetMS int64 + CoordMode CoordMode +} + +type Client struct { + mu sync.Mutex + app *slim.App + connID uint64 + agentNames map[string]*slim.Name + slimNameByID map[string]string + p2pChannels map[string]*slim.Channel + groupChannels map[string]*slim.Channel + groupLocks map[string]*sync.Mutex + coordMode CoordMode + coordStats *metrics.CoordStats + stats Stats + implLabel string +} + +type Stats struct { + ExecuteRPCCount int + SequentialRPCCount int + MulticastRPCCount int +} + +func New(endpoint string, p *plan.ExecutionPlan, cfg Config) (*Client, error) { + if cfg.CoordMode == "" { + cfg.CoordMode = CoordModeMulticast + } + impl := benchlog.ImplSLIM + if cfg.CoordMode == CoordModeUnicast { + impl = "slim-unicast" + } + + slim.InitializeWithDefaults() + service := slim.GetGlobalService() + + runnerName := slim.NewName("agntcy", "bench", "runner") + app, err := service.CreateAppWithSecret(runnerName, defaultSharedSecret) + if err != nil { + return nil, err + } + + connID, err := service.Connect(slim.NewInsecureClientConfig(endpoint)) + if err != nil { + app.Destroy() + return nil, err + } + if err := app.Subscribe(runnerName, &connID); err != nil { + app.Destroy() + return nil, err + } + + agentNames := map[string]*slim.Name{} + slimNameByID := map[string]string{} + for _, agent := range p.Agents { + name, err := nameFromString(agent.SlimName) + if err != nil { + app.Destroy() + return nil, err + } + agentNames[agent.ID] = name + slimNameByID[agent.ID] = agent.SlimName + if err := app.SetRoute(name, connID); err != nil { + app.Destroy() + return nil, fmt.Errorf("set route %s: %w", agent.SlimName, err) + } + } + + return &Client{ + app: app, + connID: connID, + agentNames: agentNames, + slimNameByID: slimNameByID, + p2pChannels: make(map[string]*slim.Channel), + groupChannels: make(map[string]*slim.Channel), + groupLocks: make(map[string]*sync.Mutex), + coordMode: cfg.CoordMode, + coordStats: metrics.NewCoordStats(cfg.RoundBudgetMS), + implLabel: impl, + }, nil +} + +func (c *Client) CoordStats() *metrics.CoordStats { return c.coordStats } + +func (c *Client) Implementation() string { return c.implLabel } + +func (c *Client) Close() { + c.mu.Lock() + defer c.mu.Unlock() + + for key, ch := range c.groupChannels { + _ = ch.Close(nil) + delete(c.groupChannels, key) + } + for id, ch := range c.p2pChannels { + _ = ch.Close(nil) + delete(c.p2pChannels, id) + } + if c.app != nil { + c.app.Destroy() + c.app = nil + } +} + +func (c *Client) Stats() Stats { return c.stats } + +func (c *Client) SetupGroup(context.Context, *plan.ExecutionPlan) error { return nil } + +func (c *Client) ExecuteTask(ctx context.Context, agentID string, task plan.Task) (protocol.Response, error) { + req := protocol.Request{ + Op: protocol.OpExecute, + TaskID: task.ID, + CompletionTimeSec: task.CompletionTimeSec, + MaxCompletionTimeSec: task.MaxCompletionTimeSec, + Output: task.Output, + InjectFailure: task.InjectFailure, + } + payload, err := protocol.EncodeRequest(req) + if err != nil { + return protocol.Response{}, err + } + start := time.Now() + reply, err := c.callUnary(ctx, agentID, payload, 30*time.Second) + duration := time.Since(start) + if err != nil { + benchlog.RPC(c.implLabel, protocol.OpExecute, "p2p", duration, false, + fmt.Sprintf("agent=%s", agentID), + fmt.Sprintf("task=%s", task.ID), + fmt.Sprintf("err=%v", err), + ) + return protocol.Response{}, err + } + c.stats.ExecuteRPCCount++ + resp, decErr := protocol.DecodeResponse(reply) + benchlog.RPC(c.implLabel, protocol.OpExecute, "p2p", duration, decErr == nil && resp.OK, + fmt.Sprintf("agent=%s", agentID), + fmt.Sprintf("task=%s", task.ID), + fmt.Sprintf("resp_ok=%t", resp.OK), + ) + if decErr != nil { + return protocol.Response{}, decErr + } + return resp, nil +} + +func (c *Client) CancelTasks(ctx context.Context, agentIDs []string, taskIDs []string) error { + return c.coordOp(ctx, agentIDs, protocol.Request{ + Op: protocol.OpCancel, + TaskIDs: taskIDs, + TargetSlimNames: c.slimNamesFor(agentIDs), + }) +} + +func (c *Client) PushContext(ctx context.Context, agentIDs []string, payload string) (time.Duration, error) { + start := time.Now() + err := c.coordOp(ctx, agentIDs, protocol.Request{ + Op: protocol.OpContext, + Payload: payload, + TargetSlimNames: c.slimNamesFor(agentIDs), + }) + return time.Since(start), err +} + +func (c *Client) SyncPhase(ctx context.Context, agentIDs []string, phase string) (time.Duration, error) { + start := time.Now() + err := c.coordOp(ctx, agentIDs, protocol.Request{ + Op: protocol.OpSync, + Phase: phase, + TargetSlimNames: c.slimNamesFor(agentIDs), + }) + return time.Since(start), err +} + +func (c *Client) NotifyFailure(ctx context.Context, agentIDs []string, failedTaskID string) error { + return c.coordOp(ctx, agentIDs, protocol.Request{ + Op: protocol.OpContext, + Payload: "failure=" + failedTaskID, + TargetSlimNames: c.slimNamesFor(agentIDs), + }) +} + +func (c *Client) coordOp(ctx context.Context, agentIDs []string, req protocol.Request) error { + if len(agentIDs) == 0 { + return nil + } + payload, err := protocol.EncodeRequest(req) + if err != nil { + return err + } + if c.coordMode == CoordModeUnicast { + return c.unicastFanOut(ctx, agentIDs, req, payload) + } + return c.multicast(ctx, agentIDs, req, payload) +} + +func (c *Client) unicastFanOut(ctx context.Context, agentIDs []string, req protocol.Request, payload []byte) error { + start := time.Now() + var firstErr error + responded := 0 + for i, agentID := range agentIDs { + callStart := time.Now() + _, err := c.callUnary(ctx, agentID, payload, 10*time.Second) + if err != nil && firstErr == nil { + firstErr = err + } else if err == nil { + responded++ + } + c.stats.SequentialRPCCount++ + if err != nil { + benchlog.RPC(c.implLabel, req.Op, "sequential", time.Since(start), false, + fmt.Sprintf("targets=%d", len(agentIDs)), + fmt.Sprintf("idx=%d", i+1), + fmt.Sprintf("err=%v", err), + ) + c.recordCoord(time.Since(start), len(agentIDs), responded, len(payload), len(agentIDs)) + return err + } + _ = callStart + } + duration := time.Since(start) + c.recordCoord(duration, len(agentIDs), responded, len(payload), len(agentIDs)) + benchlog.RPC(c.implLabel, req.Op, "sequential", duration, true, + fmt.Sprintf("targets=%d", len(agentIDs)), + fmt.Sprintf("responded=%d", responded), + ) + return firstErr +} + +func (c *Client) recordCoord(duration time.Duration, targets, responded, payloadBytes, messagesSent int) { + if c.coordStats != nil { + c.coordStats.RecordCoordOp(duration, targets, responded, payloadBytes, messagesSent) + } +} + +func (c *Client) slimNamesFor(agentIDs []string) []string { + names := make([]string, 0, len(agentIDs)) + for _, id := range agentIDs { + if name, ok := c.slimNameByID[id]; ok { + names = append(names, name) + } + } + return names +} + +func subsetGroupKey(agentIDs []string) string { + ids := append([]string(nil), agentIDs...) + slices.Sort(ids) + return strings.Join(ids, "|") +} + +func (c *Client) groupLock(key string) *sync.Mutex { + c.mu.Lock() + defer c.mu.Unlock() + if l, ok := c.groupLocks[key]; ok { + return l + } + l := &sync.Mutex{} + c.groupLocks[key] = l + return l +} + +func (c *Client) subsetGroup(agentIDs []string) (*slim.Channel, error) { + key := subsetGroupKey(agentIDs) + + c.mu.Lock() + if ch, ok := c.groupChannels[key]; ok { + c.mu.Unlock() + return ch, nil + } + c.mu.Unlock() + + members := make([]*slim.Name, 0, len(agentIDs)) + for _, id := range agentIDs { + name, ok := c.agentNames[id] + if !ok { + return nil, fmt.Errorf("unknown agent %q", id) + } + members = append(members, name) + } + + channel, err := slim.ChannelNewGroupWithConnection(c.app, members, &c.connID) + if err != nil { + return nil, err + } + + c.mu.Lock() + if existing, ok := c.groupChannels[key]; ok { + c.mu.Unlock() + _ = channel.Close(nil) + return existing, nil + } + c.groupChannels[key] = channel + c.mu.Unlock() + return channel, nil +} + +func (c *Client) multicast(ctx context.Context, agentIDs []string, req protocol.Request, payload []byte) error { + key := subsetGroupKey(agentIDs) + lock := c.groupLock(key) + lock.Lock() + defer lock.Unlock() + + group, err := c.subsetGroup(agentIDs) + if err != nil { + return err + } + + timeout := 10 * time.Second + if deadline, ok := ctx.Deadline(); ok { + timeout = time.Until(deadline) + if timeout <= 0 { + return context.DeadlineExceeded + } + } + + start := time.Now() + targets := c.slimNamesFor(agentIDs) + reader, err := group.CallMulticastUnary(slimrpc.ServiceName, slimrpc.MethodHandle, payload, &timeout, nil) + if err != nil { + benchlog.RPC(c.implLabel, req.Op, "multicast", time.Since(start), false, + fmt.Sprintf("targets=%d", len(targets)), + fmt.Sprintf("target_names=%s", strings.Join(targets, ",")), + fmt.Sprintf("err=%v", err), + ) + c.recordCoord(time.Since(start), len(targets), 0, len(payload), 1) + return err + } + defer reader.Destroy() + + c.stats.MulticastRPCCount++ + responded, drainErr := drainMulticast(reader, targets) + duration := time.Since(start) + c.recordCoord(duration, len(targets), responded, len(payload), 1) + kv := []string{ + fmt.Sprintf("targets=%d", len(targets)), + fmt.Sprintf("responded=%d", responded), + fmt.Sprintf("target_names=%s", strings.Join(targets, ",")), + } + if drainErr != nil { + kv = append(kv, fmt.Sprintf("err=%v", drainErr)) + } + benchlog.RPC(c.implLabel, req.Op, "multicast", duration, drainErr == nil, kv...) + return drainErr +} + +func drainMulticast(reader *slim.MulticastResponseReader, targetedSlimNames []string) (int, error) { + targetSet := make(map[string]struct{}, len(targetedSlimNames)) + for _, name := range targetedSlimNames { + targetSet[protocol.NormalizeSlimName(name)] = struct{}{} + } + + var firstErr error + responded := map[string]struct{}{} + + for { + switch msg := reader.Next().(type) { + case slim.MulticastStreamMessageEnd: + for name := range targetSet { + if _, ok := responded[name]; !ok && firstErr == nil { + firstErr = fmt.Errorf("missing multicast response from %q", name) + } + } + return len(responded), firstErr + case slim.MulticastStreamMessageError: + if firstErr == nil && msg.Error != nil { + firstErr = msg.Error.AsError() + } + case slim.MulticastStreamMessageData: + resp, err := protocol.DecodeResponse(msg.Item.Message) + if err != nil && firstErr == nil { + firstErr = err + continue + } + slimName := resp.SlimName + if slimName == "" && msg.Item.Context.Source != nil { + slimName = msg.Item.Context.Source.String() + } + slimName = protocol.NormalizeSlimName(slimName) + if slimName == "" { + continue + } + responded[slimName] = struct{}{} + if _, wanted := targetSet[slimName]; wanted && !resp.OK && firstErr == nil { + firstErr = fmt.Errorf("%s: %s", slimName, resp.Error) + } + } + } +} + +func (c *Client) callUnary(ctx context.Context, agentID string, payload []byte, timeout time.Duration) ([]byte, error) { + channel, err := c.p2pChannel(agentID) + if err != nil { + return nil, err + } + + if deadline, ok := ctx.Deadline(); ok { + remaining := time.Until(deadline) + if remaining > 0 && remaining < timeout { + timeout = remaining + } + } + + return channel.CallUnary(slimrpc.ServiceName, slimrpc.MethodHandle, payload, &timeout, nil) +} + +func (c *Client) p2pChannel(agentID string) (*slim.Channel, error) { + c.mu.Lock() + defer c.mu.Unlock() + + if ch, ok := c.p2pChannels[agentID]; ok { + return ch, nil + } + + dest, ok := c.agentNames[agentID] + if !ok { + return nil, fmt.Errorf("unknown agent %q", agentID) + } + + ch := slim.ChannelNewWithConnection(c.app, dest, &c.connID) + c.p2pChannels[agentID] = ch + return ch, nil +} + +func nameFromString(value string) (*slim.Name, error) { + parts := strings.Split(value, "/") + if len(parts) != 3 { + return nil, fmt.Errorf("invalid name format: %s", value) + } + return slim.NewName(parts[0], parts[1], parts[2]), nil +} diff --git a/benchmarks/slim-vs-a2a/slim/internal/executor/executor.go b/benchmarks/slim-vs-a2a/slim/internal/executor/executor.go new file mode 100644 index 00000000..ccd2079d --- /dev/null +++ b/benchmarks/slim-vs-a2a/slim/internal/executor/executor.go @@ -0,0 +1,100 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "context" + "strings" + "sync" + "time" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/protocol" +) + +type Engine struct { + SlimName string + mu sync.Mutex + cancels map[string]context.CancelFunc +} + +func New(slimName string) *Engine { + return &Engine{SlimName: slimName, cancels: map[string]context.CancelFunc{}} +} + +func (e *Engine) Handle(ctx context.Context, req protocol.Request) protocol.Response { + if len(req.TargetSlimNames) > 0 && !req.Targets(e.SlimName) { + return protocol.Response{OK: true, SlimName: e.SlimName} + } + + switch req.Op { + case protocol.OpExecute: + resp := e.execute(ctx, req) + resp.SlimName = e.SlimName + return resp + case protocol.OpCancel: + resp := e.cancel(req) + resp.SlimName = e.SlimName + return resp + case protocol.OpContext, protocol.OpSync: + return protocol.Response{OK: true, SlimName: e.SlimName} + default: + return protocol.Response{OK: false, SlimName: e.SlimName, Error: "unknown op"} + } +} + +func (e *Engine) execute(parent context.Context, req protocol.Request) protocol.Response { + if req.InjectFailure { + return protocol.Response{OK: false, TaskID: req.TaskID, Error: "injected failure"} + } + + ctx, cancel := context.WithCancel(parent) + e.mu.Lock() + e.cancels[req.TaskID] = cancel + e.mu.Unlock() + defer func() { + e.mu.Lock() + delete(e.cancels, req.TaskID) + e.mu.Unlock() + cancel() + }() + + deadline := time.Duration(req.MaxCompletionTimeSec * float64(time.Second)) + if deadline <= 0 { + deadline = time.Duration(req.CompletionTimeSec*float64(time.Second)) + time.Second + } + timer := time.NewTimer(deadline) + defer timer.Stop() + + sleep := time.Duration(req.CompletionTimeSec * float64(time.Second)) + start := time.Now() + select { + case <-time.After(sleep): + case <-ctx.Done(): + return protocol.Response{OK: false, TaskID: req.TaskID, Error: "cancelled"} + case <-timer.C: + return protocol.Response{OK: false, TaskID: req.TaskID, Error: "timeout"} + } + + return protocol.Response{ + OK: true, + TaskID: req.TaskID, + Output: req.Output, + ElapsedSec: time.Since(start).Seconds(), + } +} + +func (e *Engine) cancel(req protocol.Request) protocol.Response { + e.mu.Lock() + defer e.mu.Unlock() + for _, id := range req.TaskIDs { + if cancel, ok := e.cancels[id]; ok { + cancel() + } + } + return protocol.Response{OK: true} +} + +func (e *Engine) ObsoleteFromPayload(payload string) bool { + return strings.Contains(strings.ToLower(payload), "cancel") +} diff --git a/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go b/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go new file mode 100644 index 00000000..7baf6e8a --- /dev/null +++ b/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go @@ -0,0 +1,98 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package protocol + +import "encoding/json" + +const ( + OpExecute = "execute" + OpCancel = "cancel" + OpContext = "context" + OpSync = "sync" +) + +type Request struct { + Op string `json:"op"` + TaskID string `json:"taskId,omitempty"` + CompletionTimeSec float64 `json:"completionTimeSec,omitempty"` + MaxCompletionTimeSec float64 `json:"maxCompletionTimeSec,omitempty"` + Output string `json:"output,omitempty"` + InjectFailure bool `json:"injectFailure,omitempty"` + TaskIDs []string `json:"taskIds,omitempty"` + Payload string `json:"payload,omitempty"` + Phase string `json:"phase,omitempty"` + FailedTaskID string `json:"failedTaskId,omitempty"` + // TargetSlimNames lists org/group/app SLIM names that should handle a multicast. + // Other group members acknowledge and ignore the request. + TargetSlimNames []string `json:"targetSlimNames,omitempty"` +} + +func (r Request) Targets(slimName string) bool { + if len(r.TargetSlimNames) == 0 { + return true + } + target := NormalizeSlimName(slimName) + for _, name := range r.TargetSlimNames { + if NormalizeSlimName(name) == target { + return true + } + } + return false +} + +// NormalizeSlimName strips SlimRPC instance suffixes so agntcy/bench/foo and +// agntcy/bench/foo/ compare equal. +func NormalizeSlimName(name string) string { + parts := splitSlimName(name) + if len(parts) >= 3 { + return parts[0] + "/" + parts[1] + "/" + parts[2] + } + return name +} + +func splitSlimName(name string) []string { + var parts []string + start := 0 + for i := 0; i < len(name); i++ { + if name[i] == '/' { + if i > start { + parts = append(parts, name[start:i]) + } + start = i + 1 + } + } + if start < len(name) { + parts = append(parts, name[start:]) + } + return parts +} + +type Response struct { + OK bool `json:"ok"` + TaskID string `json:"taskId,omitempty"` + SlimName string `json:"slimName,omitempty"` + Output string `json:"output,omitempty"` + Error string `json:"error,omitempty"` + ElapsedSec float64 `json:"elapsedSec,omitempty"` +} + +func EncodeRequest(req Request) ([]byte, error) { + return json.Marshal(req) +} + +func DecodeRequest(data []byte) (Request, error) { + var req Request + err := json.Unmarshal(data, &req) + return req, err +} + +func EncodeResponse(resp Response) ([]byte, error) { + return json.Marshal(resp) +} + +func DecodeResponse(data []byte) (Response, error) { + var resp Response + err := json.Unmarshal(data, &resp) + return resp, err +} diff --git a/benchmarks/slim-vs-a2a/slim/internal/scheduler/scheduler.go b/benchmarks/slim-vs-a2a/slim/internal/scheduler/scheduler.go new file mode 100644 index 00000000..2a1166c0 --- /dev/null +++ b/benchmarks/slim-vs-a2a/slim/internal/scheduler/scheduler.go @@ -0,0 +1,341 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package scheduler + +import ( + "context" + "fmt" + "strings" + "sync" + "time" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/client" +) + +type taskState int + +const ( + statePending taskState = iota + stateRunning + stateCompleted + stateFailed + stateTimedOut + stateCancelled +) + +type Scheduler struct { + plan *plan.ExecutionPlan + client *client.Client +} + +func New(p *plan.ExecutionPlan, cli *client.Client) *Scheduler { + return &Scheduler{plan: p, client: cli} +} + +func (s *Scheduler) Run(ctx context.Context) metrics.RunResult { + start := time.Now() + benchlog.SetRunStart(start) + result := metrics.RunResult{ + PlanName: s.plan.Metadata.Name, + Domain: s.plan.Metadata.Domain, + Implementation: s.client.Implementation(), + Agents: len(s.plan.Agents), + Tasks: len(s.plan.Tasks), + } + + states := map[string]taskState{} + contextFired := map[string]bool{} + var mu sync.Mutex + var wg sync.WaitGroup + var contextPushMS int64 + var syncBarrierMS int64 + var cancelPropagationMS int64 + var obsoleteCompleted int + abort := false + + for _, t := range s.plan.Tasks { + states[t.ID] = statePending + } + + cancelTasks := func(taskIDs []string) { + if len(taskIDs) == 0 { + return + } + agentSet := map[string]struct{}{} + for _, id := range taskIDs { + task, ok := s.plan.TaskByID(id) + if !ok { + continue + } + agentSet[task.Agent] = struct{}{} + } + agentIDs := make([]string, 0, len(agentSet)) + for id := range agentSet { + agentIDs = append(agentIDs, id) + } + startCancel := time.Now() + err := s.client.CancelTasks(ctx, agentIDs, taskIDs) + duration := time.Since(startCancel) + cancelPropagationMS += duration.Milliseconds() + kv := []string{ + fmt.Sprintf("agents=%d", len(agentIDs)), + fmt.Sprintf("tasks=%d", len(taskIDs)), + } + if err != nil { + kv = append(kv, fmt.Sprintf("err=%v", err)) + } + benchlog.Coord(benchlog.ImplSLIM, "cancel", err == nil, duration, kv...) + } + + cancelDownstream := func(root string) { + downstream := plan.DownstreamTasks(s.plan.Tasks, root) + var ids []string + mu.Lock() + for id := range downstream { + switch states[id] { + case statePending, stateRunning: + states[id] = stateCancelled + result.TasksCancelled++ + ids = append(ids, id) + } + } + mu.Unlock() + cancelTasks(ids) + } + + handleFailure := func(taskID string, timedOut bool) { + mu.Lock() + if timedOut { + states[taskID] = stateTimedOut + result.TasksTimedOut++ + } else { + states[taskID] = stateFailed + result.TasksFailed++ + } + abort = true + mu.Unlock() + agents := affectedAgents([]string{taskID}, s.plan) + if err := s.client.NotifyFailure(ctx, agents, taskID); err != nil { + benchlog.Coord(benchlog.ImplSLIM, "notify_failure", false, 0, + fmt.Sprintf("task=%s", taskID), + fmt.Sprintf("agents=%d", len(agents)), + fmt.Sprintf("err=%v", err), + ) + } + cancelTasks([]string{taskID}) + cancelDownstream(taskID) + } + + applyContext := func(taskID string) { + mu.Lock() + if contextFired[taskID] { + mu.Unlock() + return + } + contextFired[taskID] = true + mu.Unlock() + + for _, cu := range s.plan.ContextUpdatesAfter(taskID) { + d, err := s.client.PushContext(ctx, cu.TargetAgents, cu.Payload) + contextPushMS += d.Milliseconds() + kv := []string{ + fmt.Sprintf("after_task=%s", taskID), + fmt.Sprintf("agents=%d", len(cu.TargetAgents)), + fmt.Sprintf("payload=%q", benchlog.TruncatePayload(cu.Payload)), + } + if err != nil { + kv = append(kv, fmt.Sprintf("err=%v", err)) + } + benchlog.Coord(benchlog.ImplSLIM, "context_push", err == nil, d, kv...) + + if strings.Contains(cu.Payload, "sync=") || strings.Contains(cu.Payload, "phase=") { + syncD, syncErr := s.client.SyncPhase(ctx, cu.TargetAgents, cu.Payload) + if syncErr == nil { + syncBarrierMS += syncD.Milliseconds() + } + syncKV := []string{ + fmt.Sprintf("after_task=%s", taskID), + fmt.Sprintf("agents=%d", len(cu.TargetAgents)), + } + if syncErr != nil { + syncKV = append(syncKV, fmt.Sprintf("err=%v", syncErr)) + } + benchlog.Coord(benchlog.ImplSLIM, "sync_phase", syncErr == nil, syncD, syncKV...) + } + if strings.Contains(strings.ToLower(cu.Payload), "cancel") { + var toCancel []string + mu.Lock() + for _, t := range s.plan.Tasks { + if states[t.ID] != stateRunning { + continue + } + if shouldCancelByContext(t, cu.Payload) { + states[t.ID] = stateCancelled + result.TasksCancelled++ + toCancel = append(toCancel, t.ID) + obsoleteCompleted++ + } + } + mu.Unlock() + cancelTasks(toCancel) + } + } + } + + launch := func(task plan.Task) { + wg.Add(1) + go func(task plan.Task) { + defer wg.Done() + taskStart := time.Now() + benchlog.Task(benchlog.ImplSLIM, "started", task.ID, task.Agent, 0) + resp, err := s.client.ExecuteTask(ctx, task.Agent, task) + taskDuration := time.Since(taskStart) + mu.Lock() + if abort && states[task.ID] == stateRunning { + states[task.ID] = stateCancelled + result.TasksCancelled++ + mu.Unlock() + benchlog.Task(benchlog.ImplSLIM, "cancelled", task.ID, task.Agent, taskDuration) + return + } + mu.Unlock() + + if err != nil { + benchlog.Task(benchlog.ImplSLIM, "failed", task.ID, task.Agent, taskDuration, + fmt.Sprintf("err=%v", err)) + handleFailure(task.ID, false) + return + } + if !resp.OK { + benchlog.Task(benchlog.ImplSLIM, "failed", task.ID, task.Agent, taskDuration, + fmt.Sprintf("resp_error=%s", resp.Error)) + handleFailure(task.ID, resp.Error == "timeout") + return + } + + mu.Lock() + states[task.ID] = stateCompleted + result.TasksCompleted++ + mu.Unlock() + benchlog.Task(benchlog.ImplSLIM, "completed", task.ID, task.Agent, taskDuration) + applyContext(task.ID) + }(task) + } + + for { + mu.Lock() + if abort { + mu.Unlock() + break + } + ready := readyTasks(s.plan.Tasks, states) + for _, task := range ready { + states[task.ID] = stateRunning + launch(task) + } + pending := countStates(states, statePending) + running := countStates(states, stateRunning) + mu.Unlock() + + if len(ready) == 0 && pending == 0 && running == 0 { + break + } + time.Sleep(5 * time.Millisecond) + } + + wg.Wait() + stats := s.client.Stats() + coordTimeMS := contextPushMS + syncBarrierMS + cancelPropagationMS + result.TotalWallClockMS = time.Since(start).Milliseconds() + result.ContextPushMS = contextPushMS + result.SyncBarrierMS = syncBarrierMS + result.CancelPropagationMS = cancelPropagationMS + result.ObsoleteTasksCompleted = obsoleteCompleted + result.ExecuteRPCCount = stats.ExecuteRPCCount + result.SequentialRPCCount = stats.SequentialRPCCount + result.MulticastRPCCount = stats.MulticastRPCCount + result.MakespanMS = result.TotalWallClockMS + result.Success = result.TasksFailed == 0 && result.TasksTimedOut == 0 + if s.client.CoordStats() != nil { + s.client.CoordStats().ApplyToRunResult(&result, coordTimeMS) + } + if !result.Success && result.Error == "" { + result.Error = fmt.Sprintf("failed=%d timed_out=%d cancelled=%d", result.TasksFailed, result.TasksTimedOut, result.TasksCancelled) + } + benchlog.Coord(benchlog.ImplSLIM, "run_finished", result.Success, time.Since(start), + fmt.Sprintf("tasks_completed=%d", result.TasksCompleted), + fmt.Sprintf("tasks_failed=%d", result.TasksFailed), + fmt.Sprintf("tasks_cancelled=%d", result.TasksCancelled), + fmt.Sprintf("context_push_ms=%d", contextPushMS), + fmt.Sprintf("execute_rpcs=%d", stats.ExecuteRPCCount), + fmt.Sprintf("multicast_rpcs=%d", stats.MulticastRPCCount), + ) + return result +} + +func readyTasks(tasks []plan.Task, states map[string]taskState) []plan.Task { + var ready []plan.Task + for _, task := range tasks { + if states[task.ID] != statePending { + continue + } + ok := true + for _, dep := range task.DependsOn { + if states[dep] != stateCompleted { + ok = false + break + } + } + if ok { + ready = append(ready, task) + } + } + return ready +} + +func countStates(states map[string]taskState, target taskState) int { + n := 0 + for _, st := range states { + if st == target { + n++ + } + } + return n +} + +func affectedAgents(taskIDs []string, p *plan.ExecutionPlan) []string { + set := map[string]struct{}{} + for _, id := range taskIDs { + task, ok := p.TaskByID(id) + if !ok { + continue + } + set[task.Agent] = struct{}{} + } + out := make([]string, 0, len(set)) + for id := range set { + out = append(out, id) + } + return out +} + +func shouldCancelByContext(task plan.Task, payload string) bool { + payload = strings.ToLower(payload) + if strings.Contains(payload, "pharmacy") && strings.Contains(task.ID, "pharmacy") { + return true + } + if strings.Contains(payload, "detour") && strings.Contains(task.ID, "detour") { + return true + } + if strings.Contains(payload, "node-drain") && strings.Contains(task.ID, "node-drain") { + return true + } + if strings.Contains(payload, "mesh-rollback") && strings.Contains(task.ID, "rollback") { + return true + } + return false +} diff --git a/benchmarks/slim-vs-a2a/slim/internal/server/handler.go b/benchmarks/slim-vs-a2a/slim/internal/server/handler.go new file mode 100644 index 00000000..64ba8ca0 --- /dev/null +++ b/benchmarks/slim-vs-a2a/slim/internal/server/handler.go @@ -0,0 +1,25 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package server + +import ( + "context" + + slim "github.com/agntcy/slim-bindings-go" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/executor" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/protocol" +) + +type Handler struct { + Engine *executor.Engine +} + +func (h *Handler) Handle(request []byte, _ *slim.Context) ([]byte, error) { + req, err := protocol.DecodeRequest(request) + if err != nil { + return nil, err + } + resp := h.Engine.Handle(context.Background(), req) + return protocol.EncodeResponse(resp) +} diff --git a/benchmarks/slim-vs-a2a/slim/internal/slimrpc/slimrpc.go b/benchmarks/slim-vs-a2a/slim/internal/slimrpc/slimrpc.go new file mode 100644 index 00000000..82814939 --- /dev/null +++ b/benchmarks/slim-vs-a2a/slim/internal/slimrpc/slimrpc.go @@ -0,0 +1,9 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package slimrpc + +const ( + ServiceName = "bench.Agent" + MethodHandle = "Handle" +) diff --git a/benchmarks/slim-vs-a2a/tests/slim_vs_a2a_test.go b/benchmarks/slim-vs-a2a/tests/slim_vs_a2a_test.go new file mode 100644 index 00000000..ef48b1e2 --- /dev/null +++ b/benchmarks/slim-vs-a2a/tests/slim_vs_a2a_test.go @@ -0,0 +1,299 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package tests + +import ( + "encoding/json" + "fmt" + "net" + "os" + "os/exec" + "path/filepath" + "strings" + "time" + + ginkgo "github.com/onsi/ginkgo/v2" + "github.com/onsi/gomega" + "github.com/onsi/gomega/gexec" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" +) + +var ( + binDir string + planDir string + reportsDir string + resultsTSV string + slimctlPath string + slimConfigPath string + slimSession *gexec.Session +) + +var _ = ginkgo.BeforeSuite(func() { + if os.Getenv("RUN_SLIM_VS_A2A") != "1" { + return + } + + var err error + binDir, err = os.MkdirTemp("", "slim-vs-a2a-bin-*") + gomega.Expect(err).NotTo(gomega.HaveOccurred()) + + build := exec.Command("go", "build", "-o", filepath.Join(binDir, "a2a-agent"), "./a2a/cmd/agent") + build.Dir = repoRoot() + gomega.Expect(build.Run()).To(gomega.Succeed()) + + build = exec.Command("go", "build", "-o", filepath.Join(binDir, "a2a-runner"), "./a2a/cmd/runner") + build.Dir = repoRoot() + gomega.Expect(build.Run()).To(gomega.Succeed()) + + setup := exec.Command("go", "run", "github.com/agntcy/slim-bindings-go/cmd/slim-bindings-setup@v1.4.0") + setup.Dir = repoRoot() + gomega.Expect(setup.Run()).To(gomega.Succeed()) + + build = exec.Command("go", "build", "-o", filepath.Join(binDir, "slim-agent"), "./slim/cmd/agent") + build.Dir = repoRoot() + build.Env = append(os.Environ(), slimCgoLDFlags()...) + gomega.Expect(build.Run()).To(gomega.Succeed()) + + build = exec.Command("go", "build", "-o", filepath.Join(binDir, "slim-runner"), "./slim/cmd/runner") + build.Dir = repoRoot() + build.Env = append(os.Environ(), slimCgoLDFlags()...) + gomega.Expect(build.Run()).To(gomega.Succeed()) + + planDir = filepath.Join(repoRoot(), "plans", "domains") + reportsDir = filepath.Join(repoRoot(), "reports") + resultsTSV = filepath.Join(reportsDir, "results.tsv") + gomega.Expect(os.MkdirAll(reportsDir, 0o755)).To(gomega.Succeed()) + slimctlPath = filepath.Join(repoRoot(), "..", "agntcy-slim", "bin", "slimctl") + if _, err := os.Stat(slimctlPath); err != nil { + download := exec.Command("task", "deps:slimctl-download") + download.Dir = repoRoot() + gomega.Expect(download.Run()).To(gomega.Succeed()) + } +}) + +var _ = ginkgo.AfterSuite(func() { + stopSlimStack() + if binDir != "" { + _ = os.RemoveAll(binDir) + } + gexec.CleanupBuildArtifacts() +}) + +var _ = ginkgo.Describe("SLIM vs A2A comparison", ginkgo.Label("slim-vs-a2a"), func() { + ginkgo.BeforeEach(func() { + if os.Getenv("RUN_SLIM_VS_A2A") != "1" { + ginkgo.Skip("set RUN_SLIM_VS_A2A=1 to run comparison suite") + } + }) + + ginkgo.It("runs all domain plans on A2A and SLIM", func() { + plans, err := plan.LoadDir(planDir) + gomega.Expect(err).NotTo(gomega.HaveOccurred()) + gomega.Expect(plans).NotTo(gomega.BeEmpty()) + + _ = os.Remove(resultsTSV) + + for _, p := range plans { + planPath := filepath.Join(planDir, p.Metadata.Name+".yaml") + a2aResult := runA2A(planPath, p.Metadata.Name, 0) + gomega.Expect(a2aResult.PlanName).To(gomega.Equal(p.Metadata.Name)) + gomega.Expect(a2aResult.Implementation).To(gomega.Equal("a2a-grpc")) + gomega.Expect(a2aResult.TotalWallClockMS).To(gomega.BeNumerically(">", 0)) + + startSlimStack() + slimResult := runSLIM(planPath, p.Metadata.Name, "multicast", 0) + stopSlimStack() + + gomega.Expect(slimResult.PlanName).To(gomega.Equal(p.Metadata.Name)) + gomega.Expect(slimResult.Implementation).To(gomega.Equal("slim-multicast")) + gomega.Expect(slimResult.TotalWallClockMS).To(gomega.BeNumerically(">", 0)) + + logComparison(p.Metadata.Name, a2aResult, slimResult) + } + }) + + ginkgo.It("logs sweep crossover hints at 10 agents / 20ms budget", func() { + sweepDir := filepath.Join(repoRoot(), "plans", "sweeps") + gomega.Expect(os.MkdirAll(sweepDir, 0o755)).To(gomega.Succeed()) + planPath := filepath.Join(sweepDir, "ginkgo-sweep-sustainable-10ag-20ms.yaml") + + gen := exec.Command("go", "run", "./tools/gen_plan", + "-family", "sustainable-resource", + "-agents", "10", + "-round-budget-ms", "20", + "-output", planPath, + ) + gen.Dir = repoRoot() + gomega.Expect(gen.Run()).To(gomega.Succeed()) + + a2aResult := runA2A(planPath, "sweep-a2a", 20) + startSlimStack() + slimResult := runSLIM(planPath, "sweep-slim", "multicast", 20) + stopSlimStack() + + ginkgo.GinkgoWriter.Printf( + "sweep agents=10 budget=20ms a2a_p95=%d slim_p95=%d a2a_missing=%d slim_missing=%d\n", + a2aResult.ContextPushP95MS, + slimResult.ContextPushP95MS, + a2aResult.CoordMissingResponses, + slimResult.CoordMissingResponses, + ) + + if slimResult.ContextPushP95MS > a2aResult.ContextPushP95MS { + ginkgo.GinkgoWriter.Printf("note: SLIM p95 higher than A2A at this scale (expected below crossover)\n") + } + }) +}) + +func logComparison(name string, a2aResult, slimResult metrics.RunResult) { + ginkgo.GinkgoWriter.Printf( + "plan=%s a2a_wall=%dms slim_wall=%dms a2a_ctx=%dms slim_ctx=%dms a2a_p95=%d slim_p95=%d a2a_missing=%d slim_missing=%d\n", + name, + a2aResult.TotalWallClockMS, + slimResult.TotalWallClockMS, + a2aResult.ContextPushMS, + slimResult.ContextPushMS, + a2aResult.ContextPushP95MS, + slimResult.ContextPushP95MS, + a2aResult.CoordMissingResponses, + slimResult.CoordMissingResponses, + ) +} + +func runA2A(planPath, jsonName string, roundBudgetMS int64) metrics.RunResult { + jsonPath := filepath.Join(reportsDir, jsonName+".json") + args := []string{ + "--plan", planPath, + "--agent-bin", filepath.Join(binDir, "a2a-agent"), + "--output-json", jsonPath, + "--wait-ready", "4s", + "--quiet", + } + if roundBudgetMS > 0 { + args = append(args, "--round-budget-ms", fmt.Sprintf("%d", roundBudgetMS)) + } + cmd := exec.Command(filepath.Join(binDir, "a2a-runner"), args...) + session, err := gexec.Start(cmd, ginkgo.GinkgoWriter, ginkgo.GinkgoWriter) + gomega.Expect(err).NotTo(gomega.HaveOccurred()) + gomega.Eventually(session, 3*time.Minute).Should(gexec.Exit(0)) + return readResult(jsonPath) +} + +func runSLIM(planPath, jsonName, coordMode string, roundBudgetMS int64) metrics.RunResult { + jsonPath := filepath.Join(reportsDir, jsonName+".json") + args := []string{ + "--plan", planPath, + "--endpoint", "http://127.0.0.1:46357", + "--agent-bin", filepath.Join(binDir, "slim-agent"), + "--coord-mode", coordMode, + "--output-json", jsonPath, + "--wait-ready", "4s", + "--quiet", + } + if roundBudgetMS > 0 { + args = append(args, "--round-budget-ms", fmt.Sprintf("%d", roundBudgetMS)) + } + cmd := exec.Command(filepath.Join(binDir, "slim-runner"), args...) + session, err := gexec.Start(cmd, ginkgo.GinkgoWriter, ginkgo.GinkgoWriter) + gomega.Expect(err).NotTo(gomega.HaveOccurred()) + gomega.Eventually(session, 3*time.Minute).Should(gexec.Exit(0)) + return readResult(jsonPath) +} + +func readResult(path string) metrics.RunResult { + data, err := os.ReadFile(path) + gomega.Expect(err).NotTo(gomega.HaveOccurred()) + var result metrics.RunResult + gomega.Expect(json.Unmarshal(data, &result)).To(gomega.Succeed()) + return result +} + +func repoRoot() string { + wd, err := os.Getwd() + gomega.Expect(err).NotTo(gomega.HaveOccurred()) + for { + if _, err := os.Stat(filepath.Join(wd, "go.mod")); err == nil { + if _, err := os.Stat(filepath.Join(wd, "a2a")); err == nil { + return wd + } + } + parent := filepath.Dir(wd) + if parent == wd { + ginkgo.Fail("could not locate slim-vs-a2a module root") + } + wd = parent + } +} + +func slimCgoLDFlags() []string { + out, err := exec.Command("go", "env", "GOPATH").Output() + if err != nil { + return nil + } + cacheDir := filepath.Join(strings.TrimSpace(string(out)), ".cgo-cache", "slim-bindings", "v1.4.0") + return []string{"CGO_LDFLAGS=-L" + cacheDir} +} + +func startSlimStack() { + stopSlimStack() + dataplanePort := 46357 + controllerPort := 46358 + configFile, err := os.CreateTemp("", "slim-vs-a2a-*.yaml") + gomega.Expect(err).NotTo(gomega.HaveOccurred()) + slimConfigPath = configFile.Name() + config := fmt.Sprintf(`services: + slim/0: + node_id: "slim-vs-a2a" + group_name: "bench" + dataplane: + servers: + - endpoint: "127.0.0.1:%d" + metadata: + local_endpoint: "127.0.0.1" + external_endpoint: "127.0.0.1:%d" + trust_domain: "example.org" + tls: + insecure: true + controller: + servers: + - endpoint: "127.0.0.1:%d" + tls: + insecure: true +`, dataplanePort, dataplanePort, controllerPort) + _, err = configFile.WriteString(config) + gomega.Expect(err).NotTo(gomega.HaveOccurred()) + gomega.Expect(configFile.Close()).To(gomega.Succeed()) + + cmd := exec.Command(slimctlPath, "slim", "start", "-c", slimConfigPath) + session, err := gexec.Start(cmd, ginkgo.GinkgoWriter, ginkgo.GinkgoWriter) + gomega.Expect(err).NotTo(gomega.HaveOccurred()) + slimSession = session + gomega.Expect(waitForPort(fmt.Sprintf("127.0.0.1:%d", dataplanePort), 20*time.Second)).To(gomega.Succeed()) +} + +func stopSlimStack() { + if slimSession != nil { + slimSession.Terminate().Wait(5 * time.Second) + slimSession = nil + } + if slimConfigPath != "" { + _ = os.Remove(slimConfigPath) + slimConfigPath = "" + } +} + +func waitForPort(addr string, timeout time.Duration) error { + deadline := time.Now().Add(timeout) + for time.Now().Before(deadline) { + conn, err := net.DialTimeout("tcp", addr, 250*time.Millisecond) + if err == nil { + _ = conn.Close() + return nil + } + time.Sleep(100 * time.Millisecond) + } + return fmt.Errorf("timed out waiting for %s", addr) +} diff --git a/benchmarks/slim-vs-a2a/tests/suite_test.go b/benchmarks/slim-vs-a2a/tests/suite_test.go new file mode 100644 index 00000000..3671cfcf --- /dev/null +++ b/benchmarks/slim-vs-a2a/tests/suite_test.go @@ -0,0 +1,16 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package tests + +import ( + "testing" + + ginkgo "github.com/onsi/ginkgo/v2" + "github.com/onsi/gomega" +) + +func TestSlimVsA2A(t *testing.T) { + gomega.RegisterFailHandler(ginkgo.Fail) + ginkgo.RunSpecs(t, "SLIM vs A2A Suite") +} diff --git a/benchmarks/slim-vs-a2a/tools/gen_plan/main.go b/benchmarks/slim-vs-a2a/tools/gen_plan/main.go new file mode 100644 index 00000000..561cae39 --- /dev/null +++ b/benchmarks/slim-vs-a2a/tools/gen_plan/main.go @@ -0,0 +1,52 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "flag" + "fmt" + "os" + + "gopkg.in/yaml.v3" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" +) + +func main() { + family := flag.String("family", "sustainable-resource", "canonical family: preference-aggregation, supply-chain-cascade, sustainable-resource") + agents := flag.Int("agents", 5, "number of agents") + fanOut := flag.Int("fan-out", 0, "context fan-out (0 = all agents)") + roundBudgetMS := flag.Int64("round-budget-ms", 20, "coordination round budget ms") + executeSec := flag.Float64("execute-sec", 0.01, "task execute sleep seconds") + rounds := flag.Int("rounds", 1, "rounds for sustainable-resource family") + output := flag.String("output", "", "write plan yaml to path (default stdout)") + flag.Parse() + + p, err := plan.GenerateCanonical(plan.GenerateOptions{ + Family: plan.CanonicalFamily(*family), + Agents: *agents, + FanOut: *fanOut, + RoundBudgetMS: *roundBudgetMS, + ExecuteSleepSec: *executeSec, + Rounds: *rounds, + }) + if err != nil { + fmt.Fprintf(os.Stderr, "generate: %v\n", err) + os.Exit(1) + } + + if *output == "" { + data, err := yaml.Marshal(p) + if err != nil { + fmt.Fprintf(os.Stderr, "marshal: %v\n", err) + os.Exit(1) + } + os.Stdout.Write(data) + return + } + if err := plan.WriteFile(*output, p); err != nil { + fmt.Fprintf(os.Stderr, "write: %v\n", err) + os.Exit(1) + } +} diff --git a/benchmarks/slim-vs-a2a/tools/report/main.go b/benchmarks/slim-vs-a2a/tools/report/main.go new file mode 100644 index 00000000..244037e9 --- /dev/null +++ b/benchmarks/slim-vs-a2a/tools/report/main.go @@ -0,0 +1,241 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/csv" + "flag" + "fmt" + "html/template" + "os" + "path/filepath" + "strconv" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" +) + +type planComparison struct { + PlanName string + A2A metrics.RunResult + SLIM metrics.RunResult + HasA2A bool + HasSLIM bool +} + +func main() { + tsvPath := flag.String("tsv", "./reports/results.tsv", "comparison results tsv") + sweepTSV := flag.String("sweep-tsv", "./reports/sweep.tsv", "optional sweep results tsv") + output := flag.String("output", "./reports/index.html", "html dashboard output") + flag.Parse() + + results, err := readTSV(*tsvPath) + if err != nil { + fmt.Fprintf(os.Stderr, "read tsv: %v\n", err) + os.Exit(1) + } + comparisons := groupByPlan(results) + + var sweepResults []metrics.RunResult + if *sweepTSV != "" { + if sr, err := readTSV(*sweepTSV); err == nil { + sweepResults = sr + } + } + + if err := writeHTML(*output, comparisons, sweepResults); err != nil { + fmt.Fprintf(os.Stderr, "write html: %v\n", err) + os.Exit(1) + } + fmt.Printf("wrote %s\n", *output) +} + +func readTSV(path string) ([]metrics.RunResult, error) { + file, err := os.Open(path) + if err != nil { + return nil, err + } + defer file.Close() + + reader := csv.NewReader(file) + reader.Comma = '\t' + records, err := reader.ReadAll() + if err != nil { + return nil, err + } + if len(records) <= 1 { + return nil, fmt.Errorf("no data rows in %s", path) + } + + var out []metrics.RunResult + for _, row := range records[1:] { + if len(row) < 22 { + continue + } + r := metrics.RunResult{ + PlanName: row[0], + Domain: row[1], + Implementation: row[2], + Error: row[len(row)-1], + } + r.Agents = atoi(row[3]) + r.Tasks = atoi(row[4]) + r.TotalWallClockMS = atoi64(row[5]) + r.TasksCompleted = atoi(row[6]) + r.TasksFailed = atoi(row[7]) + r.TasksTimedOut = atoi(row[8]) + r.TasksCancelled = atoi(row[9]) + r.ObsoleteTasksCompleted = atoi(row[10]) + r.RetriesAttempted = atoi(row[11]) + r.RetriesSucceeded = atoi(row[12]) + r.ContextPushMS = atoi64(row[13]) + r.SyncBarrierMS = atoi64(row[14]) + r.CancelPropagationMS = atoi64(row[15]) + r.ExecuteRPCCount = atoi(row[16]) + r.MulticastRPCCount = atoi(row[17]) + r.SequentialRPCCount = atoi(row[18]) + r.MakespanMS = atoi64(row[19]) + if len(row) >= 29 { + r.ContextPushP95MS = atoi64(row[20]) + r.ContextPushOps = atoi(row[21]) + r.CoordMissingResponses = atoi(row[22]) + r.CoordDeadlineMisses = atoi(row[23]) + r.CoordBytesSent = atoi64(row[24]) + r.CoordTimeSharePct, _ = strconv.ParseFloat(row[25], 64) + r.RoundBudgetMS = atoi64(row[26]) + r.Success = row[27] == "true" + r.Error = row[28] + } else { + r.Success = row[20] == "true" + if len(row) > 21 { + r.Error = row[21] + } + } + out = append(out, r) + } + return out, nil +} + +func groupByPlan(results []metrics.RunResult) []planComparison { + byPlan := map[string]*planComparison{} + for _, r := range results { + pc, ok := byPlan[r.PlanName] + if !ok { + pc = &planComparison{PlanName: r.PlanName} + byPlan[r.PlanName] = pc + } + switch r.Implementation { + case "a2a-grpc": + pc.A2A = r + pc.HasA2A = true + case "slim-multicast", "slim-unicast": + if !pc.HasSLIM || r.Implementation == "slim-multicast" { + pc.SLIM = r + pc.HasSLIM = true + } + } + } + out := make([]planComparison, 0, len(byPlan)) + for _, pc := range byPlan { + out = append(out, *pc) + } + return out +} + +type reportData struct { + Comparisons []planComparison + Sweep []metrics.RunResult +} + +func writeHTML(path string, comparisons []planComparison, sweep []metrics.RunResult) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + tmpl := template.Must(template.New("report").Funcs(template.FuncMap{ + "deltaPct": deltaPct, + "deltaPctInt": deltaPctInt, + }).Parse(htmlTemplate)) + file, err := os.Create(path) + if err != nil { + return err + } + defer file.Close() + return tmpl.Execute(file, reportData{Comparisons: comparisons, Sweep: sweep}) +} + +func deltaPct(a2a, slim int64) string { + if a2a == 0 { + return "n/a" + } + pct := (float64(a2a-slim) / float64(a2a)) * 100 + return fmt.Sprintf("%.1f%%", pct) +} + +func deltaPctInt(a2a, slim int) string { + return deltaPct(int64(a2a), int64(slim)) +} + +func atoi(v string) int { + n, _ := strconv.Atoi(v) + return n +} + +func atoi64(v string) int64 { + n, _ := strconv.ParseInt(v, 10, 64) + return n +} + +const htmlTemplate = ` + + + + SLIM vs A2A Comparison + + + +

SLIM vs A2A DAG Comparison

+

Delta columns show (A2A − SLIM) / A2A. Positive values mean SLIM was faster or used fewer RPCs.

+ {{range .Comparisons}} +

{{.PlanName}}

+ + + + + + + + + + + + + + + + + +
MetricA2ASLIMDelta
Wall clock (ms){{if .HasA2A}}{{.A2A.TotalWallClockMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.TotalWallClockMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.TotalWallClockMS .SLIM.TotalWallClockMS}}{{else}}—{{end}}
Context push (ms){{if .HasA2A}}{{.A2A.ContextPushMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.ContextPushMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.ContextPushMS .SLIM.ContextPushMS}}{{else}}—{{end}}
Context push p95 (ms){{if .HasA2A}}{{.A2A.ContextPushP95MS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.ContextPushP95MS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.ContextPushP95MS .SLIM.ContextPushP95MS}}{{else}}—{{end}}
Missing responses{{if .HasA2A}}{{.A2A.CoordMissingResponses}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.CoordMissingResponses}}{{else}}—{{end}}—
Deadline misses{{if .HasA2A}}{{.A2A.CoordDeadlineMisses}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.CoordDeadlineMisses}}{{else}}—{{end}}—
Coord bytes sent{{if .HasA2A}}{{.A2A.CoordBytesSent}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.CoordBytesSent}}{{else}}—{{end}}—
Cancel propagation (ms){{if .HasA2A}}{{.A2A.CancelPropagationMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.CancelPropagationMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.CancelPropagationMS .SLIM.CancelPropagationMS}}{{else}}—{{end}}
Sequential RPC count{{if .HasA2A}}{{.A2A.SequentialRPCCount}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.SequentialRPCCount}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPctInt .A2A.SequentialRPCCount .SLIM.SequentialRPCCount}}{{else}}—{{end}}
Multicast RPC count{{if .HasA2A}}{{.A2A.MulticastRPCCount}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.MulticastRPCCount}}{{else}}—{{end}}—
Success{{if .HasA2A}}{{.A2A.Success}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.Success}}{{else}}—{{end}}—
+ {{end}} + {{if .Sweep}} +

Sweep results

+ + + + + {{range .Sweep}} + + + + + {{end}} +
PlanImplAgentsBudget msCtx p95MissingDeadline missesBytes
{{.PlanName}}{{.Implementation}}{{.Agents}}{{.RoundBudgetMS}}{{.ContextPushP95MS}}{{.CoordMissingResponses}}{{.CoordDeadlineMisses}}{{.CoordBytesSent}}
+ {{end}} + + +` diff --git a/benchmarks/slim-vs-a2a/tools/validate_plans/main.go b/benchmarks/slim-vs-a2a/tools/validate_plans/main.go new file mode 100644 index 00000000..50737315 --- /dev/null +++ b/benchmarks/slim-vs-a2a/tools/validate_plans/main.go @@ -0,0 +1,28 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "flag" + "fmt" + "os" + "path/filepath" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" +) + +func main() { + dir := flag.String("dir", filepath.Join("plans", "domains"), "directory containing plan yaml files") + flag.Parse() + + plans, err := plan.LoadDir(*dir) + if err != nil { + fmt.Fprintf(os.Stderr, "load plans: %v\n", err) + os.Exit(1) + } + for _, p := range plans { + fmt.Printf("%s: %d agents, %d tasks, %d contextUpdates\n", + p.Metadata.Name, len(p.Agents), len(p.Tasks), len(p.ContextUpdates)) + } +} From 2242d9f4403d806f80fd65613fbd5e51df638865 Mon Sep 17 00:00:00 2001 From: Magyari Sandor Szilard Date: Fri, 19 Jun 2026 15:21:16 +0200 Subject: [PATCH 3/6] feat: slim vs a2a convergence test Signed-off-by: Magyari Sandor Szilard --- .../slim-vs-a2a-v2_consensus_37de84b5.plan.md | 332 +++++++++++++++++ .github/workflows/test-slim-vs-a2a-v2.yaml | 22 ++ benchmarks/Taskfile.yml | 5 + benchmarks/slim-vs-a2a-v2/Taskfile.yml | 348 ++++++++++++++++++ .../slim-vs-a2a-v2/a2a/cmd/agent/main.go | 161 ++++++++ .../slim-vs-a2a-v2/a2a/cmd/runner/main.go | 281 ++++++++++++++ .../a2a/internal/agent/runtime.go | 259 +++++++++++++ .../a2a/internal/client/client.go | 160 ++++++++ .../a2a/internal/protocol/protocol.go | 59 +++ .../a2a/internal/relay/relay.go | 215 +++++++++++ benchmarks/slim-vs-a2a-v2/go.mod | 25 ++ benchmarks/slim-vs-a2a-v2/go.sum | 62 ++++ .../internal/benchlog/benchlog.go | 89 +++++ .../internal/consensus/consensus.go | 313 ++++++++++++++++ .../internal/metrics/metrics.go | 120 ++++++ .../internal/scenario/scenario.go | 246 +++++++++++++ .../internal/scenario/scenario_test.go | 23 ++ benchmarks/slim-vs-a2a-v2/plans/README.md | 67 ++++ .../hypothesis-convergence-10ag-20ms.yaml | 55 +++ .../hypothesis-convergence-17ag-20ms.yaml | 83 +++++ .../hypothesis-convergence-20ag-20ms.yaml | 95 +++++ .../hypothesis-convergence-50ag-20ms.yaml | 215 +++++++++++ .../hypothesis-convergence-5ag-20ms.yaml | 35 ++ .../slim-vs-a2a-v2/slim/cmd/agent/main.go | 51 +++ .../slim-vs-a2a-v2/slim/cmd/runner/main.go | 334 +++++++++++++++++ .../slim/internal/agent/runtime.go | 246 +++++++++++++ .../slim/internal/protocol/protocol.go | 41 +++ .../slim-vs-a2a-v2/tools/gen_scenario/main.go | 45 +++ .../slim-vs-a2a-v2/tools/report/main.go | 207 +++++++++++ .../tools/validate_scenarios/main.go | 41 +++ 30 files changed, 4235 insertions(+) create mode 100644 .cursor/plans/slim-vs-a2a-v2_consensus_37de84b5.plan.md create mode 100644 .github/workflows/test-slim-vs-a2a-v2.yaml create mode 100644 benchmarks/slim-vs-a2a-v2/Taskfile.yml create mode 100644 benchmarks/slim-vs-a2a-v2/a2a/cmd/agent/main.go create mode 100644 benchmarks/slim-vs-a2a-v2/a2a/cmd/runner/main.go create mode 100644 benchmarks/slim-vs-a2a-v2/a2a/internal/agent/runtime.go create mode 100644 benchmarks/slim-vs-a2a-v2/a2a/internal/client/client.go create mode 100644 benchmarks/slim-vs-a2a-v2/a2a/internal/protocol/protocol.go create mode 100644 benchmarks/slim-vs-a2a-v2/a2a/internal/relay/relay.go create mode 100644 benchmarks/slim-vs-a2a-v2/go.mod create mode 100644 benchmarks/slim-vs-a2a-v2/go.sum create mode 100644 benchmarks/slim-vs-a2a-v2/internal/benchlog/benchlog.go create mode 100644 benchmarks/slim-vs-a2a-v2/internal/consensus/consensus.go create mode 100644 benchmarks/slim-vs-a2a-v2/internal/metrics/metrics.go create mode 100644 benchmarks/slim-vs-a2a-v2/internal/scenario/scenario.go create mode 100644 benchmarks/slim-vs-a2a-v2/internal/scenario/scenario_test.go create mode 100644 benchmarks/slim-vs-a2a-v2/plans/README.md create mode 100644 benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml create mode 100644 benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-17ag-20ms.yaml create mode 100644 benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-20ag-20ms.yaml create mode 100644 benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-50ag-20ms.yaml create mode 100644 benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml create mode 100644 benchmarks/slim-vs-a2a-v2/slim/cmd/agent/main.go create mode 100644 benchmarks/slim-vs-a2a-v2/slim/cmd/runner/main.go create mode 100644 benchmarks/slim-vs-a2a-v2/slim/internal/agent/runtime.go create mode 100644 benchmarks/slim-vs-a2a-v2/slim/internal/protocol/protocol.go create mode 100644 benchmarks/slim-vs-a2a-v2/tools/gen_scenario/main.go create mode 100644 benchmarks/slim-vs-a2a-v2/tools/report/main.go create mode 100644 benchmarks/slim-vs-a2a-v2/tools/validate_scenarios/main.go diff --git a/.cursor/plans/slim-vs-a2a-v2_consensus_37de84b5.plan.md b/.cursor/plans/slim-vs-a2a-v2_consensus_37de84b5.plan.md new file mode 100644 index 00000000..50a05836 --- /dev/null +++ b/.cursor/plans/slim-vs-a2a-v2_consensus_37de84b5.plan.md @@ -0,0 +1,332 @@ +--- +name: slim-vs-a2a-v2 consensus +overview: Create a new `benchmarks/slim-vs-a2a-v2` benchmark that compares pure Go A2A coordinator fan-out vs SLIM group multicast streaming on an identical parallel consensus problem, with configurable agent counts, TSV metrics collection, and HTML reporting mirroring v1. +todos: + - id: phase-0-spike + content: Spike SLIM v1.4.0 CallMulticastStream and a2a-go v2.3.0 streaming APIs; prototype propagation latency measurement + status: completed + - id: phase-1-shared + content: Implement internal/scenario, internal/consensus, internal/metrics, tools/gen_scenario + status: completed + - id: phase-2-slim + content: Implement slim-agent, slim-runner, slim/internal/client with group multicast streaming + status: completed + - id: phase-3-a2a + content: Implement a2a-agent (coordinator/worker), a2a-runner, a2a/internal/coordinator with parallel fan-out + status: completed + - id: phase-4-harness + content: Add Taskfile.yml, tools/report HTML, register in benchmarks/Taskfile.yml, CI smoke workflow + status: completed + - id: phase-5-validate + content: Run sweeps, verify SLIM consensus_wall_ms beats A2A at N>=10, document results in plans/README.md + status: in_progress +isProject: false +--- + +# SLIM vs A2A v2 — Consensus Streaming Benchmark + +## Goals + +- **Fair comparison:** identical consensus logic, agent count, think-time, and termination rules for both implementations +- **Highlight SLIM:** group multicast streaming delivers findings to all agents faster than A2A coordinator fan-out +- **Configurable scale:** sweep `agents` (e.g. 2, 5, 10, 17, 32) and coordination pressure (think time, finding rate) +- **Same reporting UX as v1:** Taskfile tasks → append TSV → HTML dashboard +- **Isolation:** new module at [`benchmarks/slim-vs-a2a-v2/`](benchmarks/slim-vs-a2a-v2/); v1 ([`benchmarks/slim-vs-a2a/`](benchmarks/slim-vs-a2a/)) unchanged + +**Primary success metric:** `consensus_wall_ms` — time from run start until all agents report the same agreed value. SLIM should win increasingly as `agents` grows. + +--- + +## Consensus Problem (shared by both implementations) + +### Scenario: Distributed Hypothesis Convergence + +N agents independently evaluate shards of evidence (simulated CPU work). Each agent maintains a local hypothesis `(value, confidence)`. Agents emit **findings** whenever local state changes. The run succeeds when **every agent holds the identical hypothesis**. + +### Simulation rules (deterministic, no LLM) + +- Each agent starts with `value = i % K`, `confidence = 0.1` +- Each think cycle (sleep `think_time_ms`): recompute locally using agent_id + round + received findings; emit if changed +- On receiving a peer finding: merge via deterministic merge function; may re-think and re-emit +- Consensus when all agents hold the same `value` (target derived deterministically from N via `targetMode: majority`) +- Terminate on consensus or `max_rounds` exceeded (failure) + +**Why this fits transport benchmarking:** agents run in parallel and emit findings continuously while working. Transport latency directly affects convergence speed — unlike v1's DAG where context updates happen only at task boundaries. + +### Scenario config (`ConsensusScenario` YAML) + +```yaml +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-10agents +spec: + agents: 10 + thinkTimeMs: 50 + findingEmitDelayMs: 5 + maxRounds: 100 + targetMode: majority + seed: 42 +``` + +Generated sweep files: `plans/sweeps/hypothesis-{N}ag-{think}ms.yaml` via `tools/gen_scenario`. + +--- + +## Transport Architecture + +```mermaid +flowchart TB + subgraph shared [Shared] + Scenario[ConsensusScenario YAML] + Sim[internal/consensus simulator] + Scenario --> Sim + end + + subgraph a2a [A2A hub coordinator] + Coord[coordinator agent-0] + W1[worker agent-1] + W2[worker agent-2] + WN[worker agent-N] + W1 -->|finding RPC| Coord + W2 -->|finding RPC| Coord + WN -->|finding RPC| Coord + Coord -->|parallel fan-out| W1 + Coord -->|parallel fan-out| W2 + Coord -->|parallel fan-out| WN + end + + subgraph slim [SLIM group multicast stream] + Group[SLIM group channel] + SA[agent-0] + SB[agent-1] + SC[agent-N] + SA -->|multicast stream| Group + SB -->|multicast stream| Group + Group -->|immediate delivery| SA + Group -->|immediate delivery| SB + Group -->|immediate delivery| SC + end +``` + +### A2A (`a2a-coordinator-stream`) + +| Aspect | Design | +|--------|--------| +| Topology | Hub-and-spoke: `agent-0` is coordinator; workers `agent-1..N-1` | +| Coordinator | Receives findings, deduplicates, fans out to all other agents in parallel goroutines | +| Streaming | Enable A2A agent card `Streaming: true`; use a2a-go streaming/subscribe APIs (Phase 0 spike) | +| Fallback | Parallel unary `SendMessage` per target if streaming unavailable | +| Reference | Current sequential fan-out in [`a2a/internal/client/client.go`](benchmarks/slim-vs-a2a/a2a/internal/client/client.go) `PushContext` | + +### SLIM (`slim-group-multicast-stream`) + +| Aspect | Design | +|--------|--------| +| Topology | Single SLIM group — all N agents are peers (no coordinator) | +| Finding publish | `group.CallMulticastStream(...)` — one publish, all members receive on open stream | +| Setup | Reuse group pattern from [`slim/internal/client/client.go`](benchmarks/slim-vs-a2a/slim/internal/client/client.go) `subsetGroup` + `SetupGroup` | +| v2 upgrade over v1 | v1 uses `CallMulticastUnary`; v2 uses **multicast streaming** for continuous finding propagation | + +--- + +## Package Layout + +``` +benchmarks/slim-vs-a2a-v2/ +├── go.mod +├── Taskfile.yml +├── plans/ +│ ├── README.md +│ └── sweeps/ +├── internal/ +│ ├── scenario/ # YAML load, validate, GenerateOptions +│ ├── consensus/ # shared simulator: merge, target, termination +│ ├── metrics/ # RunResult + consensus fields + TSV +│ └── benchlog/ # adapt from v1 +├── a2a/ +│ ├── cmd/agent/main.go +│ ├── cmd/runner/main.go +│ └── internal/ +│ ├── client/client.go +│ ├── coordinator/coordinator.go +│ └── protocol/protocol.go +├── slim/ +│ ├── cmd/agent/main.go +│ ├── cmd/runner/main.go +│ └── internal/ +│ ├── client/client.go +│ └── protocol/protocol.go +├── tools/ +│ ├── gen_scenario/main.go +│ └── report/main.go +├── reports/ # gitignored +└── tests/ + ├── suite_test.go + └── consensus_test.go +``` + +Register in [`benchmarks/Taskfile.yml`](benchmarks/Taskfile.yml): + +```yaml + slim-vs-a2a-v2: + taskfile: ./slim-vs-a2a-v2/Taskfile.yml + dir: ./slim-vs-a2a-v2 + excludes: [default] +``` + +**Reuse from v1 (copy, not cross-module import):** `benchlog`, SLIM stack startup Taskfile patterns, `slimctl` download, agent spawning, TSV append pattern from [`internal/metrics/metrics.go`](benchmarks/slim-vs-a2a/internal/metrics/metrics.go). + +### Pinned dependencies (same as v1) + +| Dependency | Version | Notes | +|------------|---------|-------| +| `github.com/agntcy/slim-bindings-go` | **v1.4.0** | Multicast streaming API spike targets this release | +| `slimctl` | **slimctl-v1.4.0** | Local dataplane for SLIM runs | +| `github.com/a2aproject/a2a-go/v2` | **v2.3.0** | A2A coordinator streaming | + +Taskfile vars (mirror [`benchmarks/slim-vs-a2a/Taskfile.yml`](benchmarks/slim-vs-a2a/Taskfile.yml)): + +```yaml +COMPARE_SLIM_BINDINGS_VERSION: '{{ .COMPARE_SLIM_BINDINGS_VERSION | default "v1.4.0" }}' +COMPARE_SLIMCTL_TAG: '{{ .COMPARE_SLIMCTL_TAG | default "slimctl-v1.4.0" }}' +``` + +`go.mod` require line: `github.com/agntcy/slim-bindings-go v1.4.0` + +--- + +## Metrics Schema + +Extend v1 `RunResult` with consensus-specific fields: + +| Field | Description | +|-------|-------------| +| `consensus_wall_ms` | **Primary:** start → all agents agree | +| `consensus_round` | Round number when consensus reached | +| `findings_emitted` | Total findings published | +| `findings_received_total` | Sum across agents | +| `avg_propagation_ms` | Mean time from emit → all others received | +| `p95_propagation_ms` | Tail latency of finding delivery | +| `last_agent_converge_ms` | Straggler: slowest agent to reach consensus | +| `coord_fanout_ms` | Time in coordination transport | +| `stream_rpc_count` | Streaming/multicast ops | +| `unicast_rpc_count` | Point-to-point ops | + +TSV columns (tab-separated, append mode): + +``` +scenario_name domain implementation agents consensus_wall_ms consensus_round +findings_emitted avg_propagation_ms p95_propagation_ms last_agent_converge_ms +coord_fanout_ms stream_rpc_count unicast_rpc_count success error +``` + +--- + +## Taskfile Tasks + +Mirror [`benchmarks/slim-vs-a2a/Taskfile.yml`](benchmarks/slim-vs-a2a/Taskfile.yml): + +| Task | Purpose | +|------|---------| +| `build` | Build `a2a-agent`, `a2a-runner`, `slim-agent`, `slim-runner` | +| `deps:slim-bindings-setup` | CGO bindings | +| `deps:slimctl-download` | Local SLIM dataplane | +| `validate:scenarios` | Validate YAML scenarios | +| `compare:cleanup` | Kill stray agent processes | +| `compare:suite` | All default scenarios: A2A then SLIM → `reports/results.tsv` | +| `compare:plan` | Single scenario: `PLAN=hypothesis-convergence-10agents` | +| `compare:sweep` | Agent/think-time sweep → `reports/sweep.tsv` | +| `compare:report` | `go run ./tools/report --tsv reports/results.tsv --sweep-tsv reports/sweep.tsv -o reports/index.html` | +| `compare:ci:smoke` | Small sweep + report (~2 min) | +| `test` | Ginkgo suite with `RUN_SLIM_VS_A2A_V2=1` | + +Example usage: + +```bash +cd benchmarks/slim-vs-a2a-v2 +task compare:plan PLAN=hypothesis-convergence-10agents +task compare:sweep SWEEP_AGENTS=2,5,10,17 SWEEP_THINK_MS=10,20,50 +task compare:report +``` + +Sweep env vars: `SWEEP_AGENTS` (default `2,5,10,17`), `SWEEP_THINK_MS` (default `10,20,50`), `SWEEP_FAMILY` (default `hypothesis-convergence`). + +--- + +## HTML Report + +Adapt [`tools/report/main.go`](benchmarks/slim-vs-a2a/tools/report/main.go): + +- Per-scenario comparison table: consensus wall ms, last agent converge, avg/p95 propagation, stream RPC count +- Delta columns: `(A2A − SLIM) / A2A` — positive = SLIM faster +- Sweep section: table with agents, think ms, implementation, consensus metrics + +--- + +## Implementation Phases + +### Phase 0 — API spike +- Confirm `slim-bindings-go` **v1.4.0** multicast streaming API (`CallMulticastStream` or equivalent) +- Confirm a2a-go **v2.3.0** streaming/subscribe for coordinator push +- Prototype finding propagation latency measurement + +### Phase 1 — Shared core +- `internal/scenario` — YAML schema + validator +- `internal/consensus` — deterministic simulator +- `internal/metrics` — RunResult + TSV append +- `tools/gen_scenario` — sweep YAML generator + +### Phase 2 — SLIM path +- `slim-agent` — group member, inbound stream handler, think loop +- `slim-runner` — start agents, create group, run until consensus +- `slim/internal/client` — `PublishFindingStream`, propagation timestamps + +### Phase 3 — A2A path +- `a2a-agent` — coordinator/worker roles + streaming handler +- `a2a-runner` — spawn N agents, designate agent-0 as coordinator +- `a2a/internal/coordinator` — receive finding, parallel fan-out + +### Phase 4 — Harness and CI +- Taskfile (all tasks above) +- `tools/report` HTML dashboard +- Optional `.github/workflows/test-slim-vs-a2a-v2.yaml` +- `plans/README.md` + +### Phase 5 — Validation +- Verify SLIM `consensus_wall_ms` < A2A at N >= 10 +- Confirm determinism with fixed `seed` + +--- + +## Key Design Decisions + +| Decision | Choice | Rationale | +|----------|--------|-----------| +| Separate folder | `slim-vs-a2a-v2` | Different workload (consensus vs DAG); v1 CI unchanged | +| A2A coordinator | `agent-0` is coordinator | Models realistic hub pattern; SLIM has no hub | +| Shared simulator | `internal/consensus` | Same merge/terminate logic; only I/O differs | +| SLIM transport | Multicast stream (not unary) | v2 differentiator vs v1 | +| SLIM bindings version | **slim-bindings-go v1.4.0** | Same pin as v1; multicast streaming API validated against this release | +| Agent count fairness | N total processes: A2A = 1 coord + N-1 workers; SLIM = N peers | Document in README | + +--- + +## Relationship to v1 + +| | slim-vs-a2a (v1) | slim-vs-a2a-v2 | +|--|-----------------|----------------| +| Workload | YAML DAG task execution | Continuous consensus loop | +| Coordination trigger | Task completion / contextUpdates | Finding emission during parallel work | +| SLIM transport | Multicast unary | Multicast streaming | +| A2A transport | Sequential unicast fan-out | Coordinator parallel stream fan-out | +| Primary metric | `total_wall_clock_ms`, `context_push_ms` | `consensus_wall_ms`, `avg_propagation_ms` | +| Plan kind | `ExecutionPlan` | `ConsensusScenario` | + +--- + +## Open Questions (Phase 0) + +1. ~~SLIM bindings version~~ **Resolved:** use `slim-bindings-go v1.4.0` and `slimctl-v1.4.0` (same as v1). Phase 0 confirms the exact multicast streaming method name in this release. +2. A2A streaming — a2a-go v2.3.0 server-push suitability vs parallel unary fallback +3. Coordinator fairness — N-1 workers + 1 coordinator vs external runner-level coordinator diff --git a/.github/workflows/test-slim-vs-a2a-v2.yaml b/.github/workflows/test-slim-vs-a2a-v2.yaml new file mode 100644 index 00000000..b4634b39 --- /dev/null +++ b/.github/workflows/test-slim-vs-a2a-v2.yaml @@ -0,0 +1,22 @@ +name: test-slim-vs-a2a-v2 + +on: + workflow_dispatch: + pull_request: + paths: + - 'benchmarks/slim-vs-a2a-v2/**' + - '.github/workflows/test-slim-vs-a2a-v2.yaml' + +jobs: + smoke: + runs-on: ubuntu-latest + defaults: + run: + working-directory: benchmarks/slim-vs-a2a-v2 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-go@v5 + with: + go-version: '1.25' + - name: Run CI smoke + run: task compare:ci:smoke diff --git a/benchmarks/Taskfile.yml b/benchmarks/Taskfile.yml index faee6796..6152f79e 100644 --- a/benchmarks/Taskfile.yml +++ b/benchmarks/Taskfile.yml @@ -22,6 +22,11 @@ includes: dir: ./slim-vs-a2a excludes: [default] + slim-vs-a2a-v2: + taskfile: ./slim-vs-a2a-v2/Taskfile.yml + dir: ./slim-vs-a2a-v2 + excludes: [default] + tasks: default: cmd: task -l \ No newline at end of file diff --git a/benchmarks/slim-vs-a2a-v2/Taskfile.yml b/benchmarks/slim-vs-a2a-v2/Taskfile.yml new file mode 100644 index 00000000..ccae223a --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/Taskfile.yml @@ -0,0 +1,348 @@ +# Copyright AGNTCY Contributors (https://github.com/agntcy) +# SPDX-License-Identifier: Apache-2.0 + +--- +version: '3' + +silent: true + +vars: + SCENARIO_DIR: '{{ .SCENARIO_DIR | default "./plans/sweeps" }}' + REPORTS_DIR: '{{ .REPORTS_DIR | default "./reports" }}' + RESULTS_TSV: '{{ .RESULTS_TSV | default "./reports/results.tsv" }}' + SWEEP_TSV: '{{ .SWEEP_TSV | default "./reports/sweep.tsv" }}' + BIN_DIR: '{{ .BIN_DIR | default "./bin" }}' + SLIM_ENDPOINT: '{{ .SLIM_ENDPOINT | default "http://127.0.0.1:46357" }}' + COMPARE_SLIMCTL: '{{ .COMPARE_SLIMCTL | default "../agntcy-slim/bin/slimctl" }}' + COMPARE_SLIM_BINDINGS_VERSION: '{{ .COMPARE_SLIM_BINDINGS_VERSION | default "v1.4.0" }}' + COMPARE_SLIMCTL_TAG: '{{ .COMPARE_SLIMCTL_TAG | default "slimctl-v1.4.0" }}' + +tasks: + default: + cmd: task -l + + deps:slim-bindings-setup: + desc: Install native SLIM bindings required by slim agents + cmds: + - go run github.com/agntcy/slim-bindings-go/cmd/slim-bindings-setup@{{ .COMPARE_SLIM_BINDINGS_VERSION }} + + deps:slimctl-download: + desc: Download slimctl for local SLIM stack startup + cmds: + - | + set -e + SLIMCTL_ARCH=$(arch) + SLIMCTL_OS=$(uname -s | tr '[:upper:]' '[:lower:]') + SLIMCTL_ABI="" + if [ "$SLIMCTL_ARCH" = "x86_64" ]; then + SLIMCTL_ARCH="amd64" + fi + if [ "$SLIMCTL_OS" = "linux" ]; then + if ldd --version 2>&1 | grep -qi musl; then + SLIMCTL_ABI="-musl" + else + SLIMCTL_ABI="-gnu" + fi + fi + SLIMCTL_URL="https://github.com/agntcy/slim/releases/download/{{ .COMPARE_SLIMCTL_TAG }}/slimctl-$SLIMCTL_OS-$SLIMCTL_ARCH${SLIMCTL_ABI}.tar.gz" + TARGET_DIR=$(dirname {{ .COMPARE_SLIMCTL }}) + mkdir -p "$TARGET_DIR" + TMP_DIR=$(mktemp -d) + trap 'rm -rf "$TMP_DIR"' EXIT + curl --fail --show-error --location -o "$TMP_DIR/slimctl.tar.gz" "$SLIMCTL_URL" + tar -xzf "$TMP_DIR/slimctl.tar.gz" -C "$TMP_DIR" + mv "$TMP_DIR/slimctl" {{ .COMPARE_SLIMCTL }} + chmod +x {{ .COMPARE_SLIMCTL }} + {{ .COMPARE_SLIMCTL }} version || true + + build: + desc: Build a2a and slim agent/runner binaries + deps: + - deps:slim-bindings-setup + cmds: + - mkdir -p {{ .BIN_DIR }} + - go build -o {{ .BIN_DIR }}/a2a-agent ./a2a/cmd/agent + - go build -o {{ .BIN_DIR }}/a2a-runner ./a2a/cmd/runner + - | + CACHE_DIR="$(go env GOPATH)/.cgo-cache/slim-bindings/{{ .COMPARE_SLIM_BINDINGS_VERSION }}" + export CGO_LDFLAGS="-L${CACHE_DIR}" + go build -o {{ .BIN_DIR }}/slim-agent ./slim/cmd/agent + go build -o {{ .BIN_DIR }}/slim-runner ./slim/cmd/runner + + validate:scenarios: + desc: Validate consensus scenario YAML files + cmds: + - go run ./tools/validate_scenarios --dir {{ .SCENARIO_DIR }} + + compare:cleanup: + internal: true + cmds: + - pkill -f '{{ .BIN_DIR }}/a2a-agent' 2>/dev/null || true + - pkill -f '{{ .BIN_DIR }}/slim-agent' 2>/dev/null || true + - pkill -f 'slimctl slim start -c' 2>/dev/null || true + + compare:suite: + desc: Run default consensus scenarios on A2A and SLIM + deps: + - build + - validate:scenarios + cmds: + - task: compare:cleanup + - rm -f {{ .RESULTS_TSV }} + - | + set -euo pipefail + for scenario in {{ .SCENARIO_DIR }}/hypothesis-convergence-*.yaml; do + [ -f "$scenario" ] || continue + base=$(basename "$scenario" .yaml) + echo "==> A2A $base" + {{ .BIN_DIR }}/a2a-runner \ + --scenario "$scenario" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ + --output-tsv {{ .RESULTS_TSV }} || true + done + - task: compare:slim-stack + vars: + SCENARIO_GLOB: '{{ .SCENARIO_DIR }}/hypothesis-convergence-*.yaml' + + compare:plan: + desc: Run one scenario on A2A and SLIM (PLAN=path or basename) + deps: + - build + requires: + vars: [PLAN] + cmds: + - task: compare:cleanup + - rm -f {{ .RESULTS_TSV }} + - | + set -euo pipefail + PLAN_INPUT="{{ .PLAN }}" + if [ -f "$PLAN_INPUT" ]; then + SCENARIO_PATH="$PLAN_INPUT" + elif [ -f "{{ .SCENARIO_DIR }}/${PLAN_INPUT}.yaml" ]; then + SCENARIO_PATH="{{ .SCENARIO_DIR }}/${PLAN_INPUT}.yaml" + else + echo "scenario not found: ${PLAN_INPUT}" >&2 + exit 1 + fi + base=$(basename "$SCENARIO_PATH" .yaml) + echo "==> A2A ${base}" + {{ .BIN_DIR }}/a2a-runner \ + --scenario "$SCENARIO_PATH" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ + --output-tsv {{ .RESULTS_TSV }} + - task: compare:slim-stack + vars: + SCENARIO_GLOB: '{{ .PLAN }}' + + compare:slim-stack: + internal: true + deps: + - deps:slimctl-download + cmds: + - | + set -euo pipefail + SCENARIO_INPUT="{{ .SCENARIO_GLOB }}" + resolve_scenario() { + local input="$1" + if [ -f "$input" ]; then + printf '%s' "$input" + return + fi + if [ -f "{{ .SCENARIO_DIR }}/${input}.yaml" ]; then + printf '%s' "{{ .SCENARIO_DIR }}/${input}.yaml" + return + fi + printf '%s' "$input" + } + DATAPLANE_PORT=46357 + CONTROLLER_PORT=46358 + CONFIG=$(mktemp) + trap 'kill ${SLIM_PID:-} 2>/dev/null || true; rm -f "$CONFIG"; {{ .COMPARE_SLIMCTL }} slim stop 2>/dev/null || true' EXIT + cat > "$CONFIG" </dev/null; then + break + fi + sleep 0.2 + done + if [[ "$SCENARIO_INPUT" == *"*"* ]]; then + SCENARIO_LIST=($SCENARIO_INPUT) + else + SCENARIO_LIST=("$(resolve_scenario "$SCENARIO_INPUT")") + fi + for scenario in "${SCENARIO_LIST[@]}"; do + [ -f "$scenario" ] || continue + base=$(basename "$scenario" .yaml) + echo "==> SLIM ${base}" + {{ .BIN_DIR }}/slim-runner \ + --scenario "$scenario" \ + --endpoint {{ .SLIM_ENDPOINT }} \ + --agent-bin {{ .BIN_DIR }}/slim-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/slim-${base}.json \ + --output-tsv {{ .RESULTS_TSV }} || true + done + + compare:report: + desc: Build HTML dashboard from comparison TSV results + cmds: + - go run ./tools/report --tsv {{ .RESULTS_TSV }} --sweep-tsv {{ .SWEEP_TSV }} --output {{ .REPORTS_DIR }}/index.html + + compare:sweep: + desc: Run agent/think-time sweep (SWEEP_AGENTS, SWEEP_THINK_MS env lists) + deps: + - build + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' + cmds: + - task: compare:cleanup + - rm -f {{ .SWEEP_TSV }} + - task: compare:sweep:run + + compare:sweep:run: + internal: true + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' + cmds: + - | + set -euo pipefail + mkdir -p {{ .REPORTS_DIR }} {{ .SCENARIO_DIR }} + IFS=',' read -r -a AGENT_LIST <<< "{{ .SWEEP_AGENTS }}" + IFS=',' read -r -a THINK_LIST <<< "{{ .SWEEP_THINK_MS }}" + for agents in "${AGENT_LIST[@]}"; do + for think in "${THINK_LIST[@]}"; do + scenario_path="{{ .SCENARIO_DIR }}/{{ .SWEEP_FAMILY }}-${agents}ag-${think}ms.yaml" + go run ./tools/gen_scenario \ + -family {{ .SWEEP_FAMILY }} \ + -agents "$agents" \ + -think-ms "$think" \ + -output "$scenario_path" + base=$(basename "$scenario_path" .yaml) + echo "==> sweep A2A $base" + {{ .BIN_DIR }}/a2a-runner \ + --scenario "$scenario_path" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ + --output-tsv {{ .SWEEP_TSV }} || true + done + done + - task: compare:sweep:slim + + compare:sweep:slim: + internal: true + deps: + - deps:slimctl-download + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' + cmds: + - | + set -euo pipefail + IFS=',' read -r -a AGENT_LIST <<< "{{ .SWEEP_AGENTS }}" + IFS=',' read -r -a THINK_LIST <<< "{{ .SWEEP_THINK_MS }}" + DATAPLANE_PORT=46357 + CONTROLLER_PORT=46358 + CONFIG=$(mktemp) + trap 'kill ${SLIM_PID:-} 2>/dev/null || true; rm -f "$CONFIG"; {{ .COMPARE_SLIMCTL }} slim stop 2>/dev/null || true' EXIT + cat > "$CONFIG" </dev/null; then + break + fi + sleep 0.2 + done + for agents in "${AGENT_LIST[@]}"; do + for think in "${THINK_LIST[@]}"; do + scenario_path="{{ .SCENARIO_DIR }}/{{ .SWEEP_FAMILY }}-${agents}ag-${think}ms.yaml" + [ -f "$scenario_path" ] || continue + base=$(basename "$scenario_path" .yaml) + echo "==> sweep SLIM $base" + {{ .BIN_DIR }}/slim-runner \ + --scenario "$scenario_path" \ + --endpoint {{ .SLIM_ENDPOINT }} \ + --agent-bin {{ .BIN_DIR }}/slim-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/slim-${base}.json \ + --output-tsv {{ .SWEEP_TSV }} || true + done + done + + compare:ci:smoke: + desc: CI smoke — one sweep point plus report (~2 min) + deps: + - build + cmds: + - task: compare:cleanup + - rm -f {{ .RESULTS_TSV }} {{ .SWEEP_TSV }} + - | + set -euo pipefail + mkdir -p {{ .SCENARIO_DIR }} {{ .REPORTS_DIR }} + scenario_path="{{ .SCENARIO_DIR }}/ci-smoke-hypothesis-5ag-20ms.yaml" + go run ./tools/gen_scenario -family hypothesis-convergence -agents 5 -think-ms 20 -output "$scenario_path" + echo "==> smoke A2A" + {{ .BIN_DIR }}/a2a-runner \ + --scenario "$scenario_path" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/a2a-ci-smoke.json \ + --output-tsv {{ .RESULTS_TSV }} + - task: compare:slim-stack + vars: + SCENARIO_GLOB: 'plans/sweeps/ci-smoke-hypothesis-5ag-20ms.yaml' + - task: compare:report + + test: + desc: Run unit tests + cmds: + - go test ./... -count=1 diff --git a/benchmarks/slim-vs-a2a-v2/a2a/cmd/agent/main.go b/benchmarks/slim-vs-a2a-v2/a2a/cmd/agent/main.go new file mode 100644 index 00000000..a98348ff --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/a2a/cmd/agent/main.go @@ -0,0 +1,161 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "iter" + "log" + "net" + "net/http" + + "github.com/a2aproject/a2a-go/v2/a2a" + a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" + "github.com/a2aproject/a2a-go/v2/a2asrv" + "github.com/a2aproject/a2a-go/v2/a2asrv/taskstore" + agentrt "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/agent" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" + "google.golang.org/grpc" +) + +type consensusExecutor struct { + runtime *agentrt.Runtime +} + +func (e *consensusExecutor) Execute(ctx context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { + reqText := firstText(execCtx.Message) + req, err := agentrt.DecodeRequestText(reqText) + if err != nil { + return func(yield func(a2a.Event, error) bool) { + yield(nil, fmt.Errorf("decode request: %w", err)) + } + } + + if req.Op == protocol.OpStreamFindings { + return e.runtime.StreamFindings(ctx, execCtx) + } + + resp := e.runtime.HandleUnary(ctx, req) + body, err := json.Marshal(resp) + if err != nil { + return func(yield func(a2a.Event, error) bool) { + yield(nil, err) + } + } + return func(yield func(a2a.Event, error) bool) { + task := a2a.NewSubmittedTask(execCtx, execCtx.Message) + state := a2a.TaskStateCompleted + if !resp.OK { + state = a2a.TaskStateFailed + } + task.Status = a2a.TaskStatus{ + State: state, + Message: a2a.NewMessageForTask( + a2a.MessageRoleAgent, + task, + a2a.NewTextPart(string(body)), + ), + } + yield(task, nil) + } +} + +func (e *consensusExecutor) Cancel(_ context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { + return func(yield func(a2a.Event, error) bool) { + yield(a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateCanceled, a2a.NewMessageForTask( + a2a.MessageRoleAgent, execCtx, a2a.NewTextPart(`{"ok":true}`), + )), nil) + } +} + +func firstText(message *a2a.Message) string { + if message == nil { + return "" + } + for _, part := range message.Parts { + if text := part.Text(); text != "" { + return text + } + } + return "" +} + +func main() { + agentID := flag.String("agent-id", "", "logical agent id") + grpcPort := flag.Int("grpc-port", 0, "A2A gRPC port") + cardPort := flag.Int("card-port", 0, "agent card HTTP port") + scenarioFile := flag.String("scenario-file", "", "consensus scenario yaml") + agentIndex := flag.Int("agent-index", 0, "agent index") + relayCardURL := flag.String("relay-card-url", "", "runner relay agent card base URL") + flag.Parse() + + if *agentID == "" || *grpcPort == 0 || *scenarioFile == "" || *relayCardURL == "" { + log.Fatal("agent-id, grpc-port, scenario-file, and relay-card-url are required") + } + card := *cardPort + if card == 0 { + card = *grpcPort + 1000 + } + + s, err := scenario.LoadFile(*scenarioFile) + if err != nil { + log.Fatalf("load scenario: %v", err) + } + + rt := agentrt.NewRuntime(s, *agentIndex, *agentID, *relayCardURL) + defer rt.Close() + + handler := a2asrv.NewHandler( + &consensusExecutor{runtime: rt}, + a2asrv.WithTaskStore(taskstore.NewInMemory(&taskstore.InMemoryStoreConfig{ + Authenticator: func(context.Context) (string, error) { return "slim-vs-a2a-v2", nil }, + })), + ) + + agentCard := &a2a.AgentCard{ + Name: fmt.Sprintf("consensus-v2-%s", *agentID), + Description: "SLIM vs A2A v2 consensus agent", + Version: "1.0.0", + SupportedInterfaces: []*a2a.AgentInterface{ + a2a.NewAgentInterface(fmt.Sprintf("127.0.0.1:%d", *grpcPort), a2a.TransportProtocolGRPC), + }, + Capabilities: a2a.AgentCapabilities{Streaming: true}, + DefaultInputModes: []string{"text/plain"}, + DefaultOutputModes: []string{"text/plain"}, + } + + cardListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", card)) + if err != nil { + log.Fatalf("card listen: %v", err) + } + grpcListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", *grpcPort)) + if err != nil { + log.Fatalf("grpc listen: %v", err) + } + + grpcHandler := a2agrpc.NewHandler(handler) + grpcServer := grpc.NewServer() + grpcHandler.RegisterWith(grpcServer) + + go func() { + mux := http.NewServeMux() + mux.Handle(a2asrv.WellKnownAgentCardPath, a2asrv.NewStaticAgentCardHandler(agentCard)) + if err := http.Serve(cardListener, mux); err != nil { + log.Printf("card server stopped: %v", err) + } + }() + + // Subscribe to the runner relay stream as soon as we are up; it retries + // until the relay server is reachable. + rt.StartRelaySubscription(context.Background()) + + fmt.Printf("A2A_AGENT_READY agent=%s grpc=%d card=%d\n", *agentID, *grpcPort, card) + if err := grpcServer.Serve(grpcListener); err != nil { + log.Fatalf("grpc serve: %v", err) + } +} diff --git a/benchmarks/slim-vs-a2a-v2/a2a/cmd/runner/main.go b/benchmarks/slim-vs-a2a-v2/a2a/cmd/runner/main.go new file mode 100644 index 00000000..0e2fa05c --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/a2a/cmd/runner/main.go @@ -0,0 +1,281 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Command runner is the A2A benchmark driver and relay hub. Because A2A has no +// peer multicast, the runner is the explicit relay: it subscribes to every +// agent's findings stream (OpStreamFindings) and fans each finding out to all +// other agents over the relay stream (OpStreamRelay). It also drives start and +// polls snapshots to detect global consensus. +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "log" + "os" + "os/exec" + "time" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/client" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/relay" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/metrics" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" +) + +func main() { + scenarioPath := flag.String("scenario", "", "path to consensus scenario yaml") + agentBin := flag.String("agent-bin", "", "path to a2a-agent binary") + outputJSON := flag.String("output-json", "", "write run metrics json") + outputTSV := flag.String("output-tsv", "", "append run metrics tsv") + waitReady := flag.Duration("wait-ready", 3*time.Second, "wait for agents to start") + relayGRPCPort := flag.Int("relay-grpc-port", 9600, "relay hub gRPC port") + relayCardPort := flag.Int("relay-card-port", 9601, "relay hub agent card port") + quiet := flag.Bool("quiet", false, "disable benchmark logs") + flag.Parse() + + if *quiet { + benchlog.SetEnabled(false) + } + if *scenarioPath == "" { + log.Fatal("--scenario is required") + } + + s, err := scenario.LoadFile(*scenarioPath) + if err != nil { + log.Fatalf("load scenario: %v", err) + } + + agentPath := *agentBin + if agentPath == "" { + agentPath = os.Getenv("A2A_AGENT_BIN") + } + if agentPath == "" { + log.Fatal("set --agent-bin or A2A_AGENT_BIN") + } + + // Start the relay hub first so agents can subscribe as they come up. + hub := relay.NewHub(*relayGRPCPort, *relayCardPort) + if err := hub.Serve(); err != nil { + log.Fatalf("relay serve: %v", err) + } + + procs := startAgents(s, agentPath, *scenarioPath, hub.CardURL()) + defer stopAgents(procs) + time.Sleep(*waitReady) + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + + cli, err := newClientWithRetry(ctx, s) + if err != nil { + stopAgents(procs) + log.Fatalf("client: %v", err) + } + + agentIDs := s.AgentIDs() + agentIndexByID := map[string]int{} + for i, a := range s.Agents { + agentIndexByID[a.ID] = i + } + + // Runner subscribes to every agent's findings stream and relays each finding + // to all other agents. Retries keep the subscription alive across restarts. + for _, id := range agentIDs { + go func(agentID string) { + for { + select { + case <-ctx.Done(): + return + default: + } + err := cli.SubscribeFindings(ctx, agentID, func(f consensus.Finding) { + hub.Broadcast(f) + }) + if err == nil || ctx.Err() != nil { + return + } + time.Sleep(300 * time.Millisecond) + } + }(id) + } + + // Let finding subscriptions establish before agents start producing. + time.Sleep(500 * time.Millisecond) + + runStart := time.Now() + benchlog.SetRunStart(runStart) + if err := cli.StartAll(ctx, agentIDs); err != nil { + stopAgents(procs) + log.Fatalf("start agents: %v", err) + } + + result := metrics.RunResult{ + ScenarioName: s.Metadata.Name, + Domain: s.Metadata.Domain, + Implementation: benchlog.ImplA2A, + Agents: len(s.Agents), + ThinkTimeMs: s.Spec.ThinkTimeMs, + } + + var snapshots []consensus.AgentSnapshot + deadline := time.Now().Add(2 * time.Minute) + for time.Now().Before(deadline) { + snapshots = snapshots[:0] + var pollErr error + for _, id := range agentIDs { + snap, err := cli.Snapshot(ctx, id) + if err != nil { + pollErr = err + break + } + snapshots = append(snapshots, snap) + } + if pollErr == nil { + if ok, _ := consensus.GlobalConsensus(snapshots); ok { + result.Success = true + break + } + } + time.Sleep(20 * time.Millisecond) + } + + if !result.Success && len(snapshots) > 0 { + if ok, _ := consensus.GlobalConsensus(snapshots); ok { + result.Success = true + } + } + + result = aggregateResult(result, runStart, snapshots, hub) + + stopAgents(procs) + + if *outputJSON != "" { + if err := metrics.WriteJSON(*outputJSON, result); err != nil { + log.Fatalf("write json: %v", err) + } + } + if *outputTSV != "" { + if err := metrics.AppendTSV(*outputTSV, result); err != nil { + log.Fatalf("write tsv: %v", err) + } + } + + data, _ := json.MarshalIndent(result, "", " ") + fmt.Println(string(data)) + if !result.Success { + os.Exit(1) + } +} + +func newClientWithRetry(ctx context.Context, s *scenario.ConsensusScenario) (*client.Client, error) { + var lastErr error + for attempt := 0; attempt < 50; attempt++ { + cli, err := client.New(ctx, s) + if err == nil { + return cli, nil + } + lastErr = err + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-time.After(200 * time.Millisecond): + } + } + return nil, lastErr +} + +func aggregateResult(result metrics.RunResult, runStart time.Time, snapshots []consensus.AgentSnapshot, hub *relay.Hub) metrics.RunResult { + if len(snapshots) == 0 { + result.Error = "no agent snapshots" + result.ConsensusWallMS = time.Since(runStart).Milliseconds() + return result + } + + var ( + totalEmitted int + totalApplied int + lastConvergeMS int64 + propDurations []int64 + maxConsensusRound int + maxRound int + ) + + for _, snap := range snapshots { + totalEmitted += snap.FindingsEmitted + totalApplied += snap.FindingsApplied + if snap.ConsensusRound > maxConsensusRound { + maxConsensusRound = snap.ConsensusRound + } + if snap.Round > maxRound { + maxRound = snap.Round + } + if snap.ConvergedAtNs > 0 { + ms := (snap.ConvergedAtNs - runStart.UnixNano()) / int64(time.Millisecond) + if ms > lastConvergeMS { + lastConvergeMS = ms + } + } + if snap.AvgPropagationMs > 0 { + propDurations = append(propDurations, snap.AvgPropagationMs) + } + } + + if result.Success { + result.ConsensusWallMS = time.Since(runStart).Milliseconds() + if lastConvergeMS > 0 { + result.ConsensusWallMS = lastConvergeMS + } + } else { + result.ConsensusWallMS = time.Since(runStart).Milliseconds() + result.Error = "consensus not reached" + } + + result.ConsensusRound = maxConsensusRound + if result.ConsensusRound == 0 { + result.ConsensusRound = maxRound + } + result.FindingsEmitted = totalEmitted + result.FindingsReceivedTotal = totalApplied + result.LastAgentConvergeMS = lastConvergeMS + result.AvgPropagationMS, result.P95PropagationMS = metrics.AggregatePropagation(propDurations) + // Relay hub did all the fan-out work; report its real counters. + streamCount, fanoutMS := hub.Stats() + result.StreamRPCCount = streamCount + result.CoordFanoutMS = fanoutMS + return result +} + +func startAgents(s *scenario.ConsensusScenario, agentBin, scenarioFile, relayCardURL string) []*exec.Cmd { + var procs []*exec.Cmd + for i, agent := range s.Agents { + cmd := exec.Command( + agentBin, + "--agent-id", agent.ID, + "--grpc-port", fmt.Sprintf("%d", agent.A2APort), + "--card-port", fmt.Sprintf("%d", s.CardPort(agent)), + "--scenario-file", scenarioFile, + "--agent-index", fmt.Sprintf("%d", i), + "--relay-card-url", relayCardURL, + ) + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + if err := cmd.Start(); err != nil { + stopAgents(procs) + log.Fatalf("start agent %s: %v", agent.ID, err) + } + procs = append(procs, cmd) + } + return procs +} + +func stopAgents(procs []*exec.Cmd) { + for _, cmd := range procs { + if cmd.Process != nil { + _ = cmd.Process.Kill() + } + } +} diff --git a/benchmarks/slim-vs-a2a-v2/a2a/internal/agent/runtime.go b/benchmarks/slim-vs-a2a-v2/a2a/internal/agent/runtime.go new file mode 100644 index 00000000..6e39e40c --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/a2a/internal/agent/runtime.go @@ -0,0 +1,259 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Package agent implements an A2A consensus agent for the relay topology. +// +// There is no peer-to-peer multicast in A2A, so every finding flows through the +// runner relay over two server-streaming legs: +// - OpStreamFindings: this agent streams the findings it produces to the +// runner (the runner is the stream client, this agent the stream server). +// - OpStreamRelay: this agent subscribes to the runner and receives every +// relayed finding (this agent is the stream client, the runner the server). +// +// OpStart / OpSnapshot remain unary calls the runner drives. +package agent + +import ( + "context" + "encoding/json" + "fmt" + "iter" + "sync" + "sync/atomic" + "time" + + "github.com/a2aproject/a2a-go/v2/a2a" + "github.com/a2aproject/a2a-go/v2/a2aclient" + "github.com/a2aproject/a2a-go/v2/a2aclient/agentcard" + a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" + "github.com/a2aproject/a2a-go/v2/a2asrv" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" +) + +type Runtime struct { + scenario *scenario.ConsensusScenario + agentIndex int + agentID string + relayCardURL string + + engine *consensus.Engine + + outbound chan consensus.Finding + running atomic.Bool + done chan struct{} + startedAt time.Time + relayOnce sync.Once +} + +func NewRuntime(s *scenario.ConsensusScenario, agentIndex int, agentID, relayCardURL string) *Runtime { + return &Runtime{ + scenario: s, + agentIndex: agentIndex, + agentID: agentID, + relayCardURL: relayCardURL, + engine: consensus.NewEngine(s.Spec, agentIndex), + outbound: make(chan consensus.Finding, 1024), + done: make(chan struct{}), + } +} + +// StartRelaySubscription dials the runner relay and subscribes to the relayed +// finding stream (OpStreamRelay). It runs once and retries until the relay is up. +func (r *Runtime) StartRelaySubscription(ctx context.Context) { + r.relayOnce.Do(func() { + go r.relayLoop(ctx) + }) +} + +func (r *Runtime) relayLoop(ctx context.Context) { + cli := r.dialRelay(ctx) + if cli == nil { + return + } + req := protocol.Request{Op: protocol.OpStreamRelay, AgentIndex: r.agentIndex} + text, err := json.Marshal(req) + if err != nil { + return + } + for { + select { + case <-ctx.Done(): + return + default: + } + msg := a2a.NewMessage(a2a.MessageRoleUser, a2a.NewTextPart(string(text))) + for ev, streamErr := range cli.SendStreamingMessage(ctx, &a2a.SendMessageRequest{Message: msg}) { + if streamErr != nil { + break + } + f, ok := findingFromEvent(ev) + if !ok || f.AgentIndex == r.agentIndex { + continue + } + r.applyFinding(f) + } + select { + case <-ctx.Done(): + return + case <-time.After(500 * time.Millisecond): + } + } +} + +func (r *Runtime) dialRelay(ctx context.Context) *a2aclient.Client { + for attempt := 0; attempt < 100; attempt++ { + select { + case <-ctx.Done(): + return nil + default: + } + card, err := agentcard.DefaultResolver.Resolve(ctx, r.relayCardURL) + if err == nil { + cli, cerr := a2aclient.NewFromCard(ctx, card, + a2agrpc.WithGRPCTransport(grpc.WithTransportCredentials(insecure.NewCredentials()))) + if cerr == nil { + return cli + } + } + time.Sleep(200 * time.Millisecond) + } + return nil +} + +// HandleUnary serves the unary control ops (start, snapshot). +func (r *Runtime) HandleUnary(_ context.Context, req protocol.Request) protocol.Response { + switch req.Op { + case protocol.OpStart: + r.StartRun() + return protocol.Response{OK: true} + case protocol.OpSnapshot: + body, err := json.Marshal(r.engine.Snapshot()) + if err != nil { + return protocol.Response{OK: false, Error: err.Error()} + } + return protocol.Response{OK: true, Body: string(body)} + default: + return protocol.Response{OK: false, Error: "unknown op"} + } +} + +// StreamFindings serves OpStreamFindings: a long-lived server stream that emits +// every finding this agent produces. The first event establishes the task; each +// finding is delivered as a non-terminal Working status update so the stream +// stays open. +func (r *Runtime) StreamFindings(ctx context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { + return func(yield func(a2a.Event, error) bool) { + task := a2a.NewSubmittedTask(execCtx, execCtx.Message) + if !yield(task, nil) { + return + } + for { + select { + case <-ctx.Done(): + yield(a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateCompleted, nil), nil) + return + case f := <-r.outbound: + body, err := consensus.EncodeFinding(f) + if err != nil { + continue + } + msg := a2a.NewMessageForTask(a2a.MessageRoleAgent, execCtx, a2a.NewTextPart(string(body))) + ev := a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateWorking, msg) + if !yield(ev, nil) { + return + } + } + } + } +} + +func (r *Runtime) StartRun() { + if r.running.Swap(true) { + return + } + r.startedAt = time.Now() + benchlog.SetRunStart(r.startedAt) + go r.runLoop() +} + +func (r *Runtime) runLoop() { + defer close(r.done) + spec := r.scenario.Spec + think := time.Duration(spec.ThinkTimeMs) * time.Millisecond + emitGap := time.Duration(spec.FindingEmitDelayMs) * time.Millisecond + + for round := 0; round < spec.MaxRounds; round++ { + time.Sleep(think) + + finding, emit := r.engine.Think() + if emit && finding != nil { + select { + case r.outbound <- *finding: + benchlog.Finding(benchlog.ImplA2A, "published", r.agentIndex, finding.FindingID) + case <-time.After(5 * time.Second): + benchlog.Finding(benchlog.ImplA2A, "publish_timeout", r.agentIndex, finding.FindingID) + } + time.Sleep(emitGap) + } + + if r.engine.HasLocalConsensus() { + return + } + } +} + +func (r *Runtime) applyFinding(f consensus.Finding) { + recvNs := time.Now().UnixNano() + r.engine.ApplyFinding(f) + if f.EmittedAt > 0 { + r.engine.RecordPropagation(f.EmittedAt, recvNs) + } + benchlog.Finding(benchlog.ImplA2A, "received", r.agentIndex, f.FindingID, + fmt.Sprintf("from=%d", f.AgentIndex)) +} + +func (r *Runtime) Snapshot() consensus.AgentSnapshot { + return r.engine.Snapshot() +} + +func (r *Runtime) Close() {} + +// findingFromEvent extracts a finding from a streamed event, ignoring the +// initial submitted-task event and any event without a message payload. +func findingFromEvent(ev a2a.Event) (consensus.Finding, bool) { + su, ok := ev.(*a2a.TaskStatusUpdateEvent) + if !ok || su.Status.Message == nil { + return consensus.Finding{}, false + } + text := firstText(su.Status.Message) + if text == "" { + return consensus.Finding{}, false + } + f, err := consensus.DecodeFinding([]byte(text)) + if err != nil { + return consensus.Finding{}, false + } + return f, true +} + +func firstText(message *a2a.Message) string { + if message == nil { + return "" + } + for _, part := range message.Parts { + if text := part.Text(); text != "" { + return text + } + } + return "" +} + +// DecodeRequestText is used by the agent server executor. +func DecodeRequestText(text string) (protocol.Request, error) { + return protocol.DecodeRequest(text) +} diff --git a/benchmarks/slim-vs-a2a-v2/a2a/internal/client/client.go b/benchmarks/slim-vs-a2a-v2/a2a/internal/client/client.go new file mode 100644 index 00000000..4a9cf2fe --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/a2a/internal/client/client.go @@ -0,0 +1,160 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package client + +import ( + "context" + "encoding/json" + "fmt" + + "github.com/a2aproject/a2a-go/v2/a2a" + "github.com/a2aproject/a2a-go/v2/a2aclient" + "github.com/a2aproject/a2a-go/v2/a2aclient/agentcard" + a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" +) + +type Client struct { + clients map[string]*a2aclient.Client +} + +func New(ctx context.Context, s *scenario.ConsensusScenario) (*Client, error) { + clients := map[string]*a2aclient.Client{} + for _, agent := range s.Agents { + baseURL := s.CardBaseURL(agent) + card, err := agentcard.DefaultResolver.Resolve(ctx, baseURL) + if err != nil { + return nil, fmt.Errorf("resolve card for %s: %w", agent.ID, err) + } + cli, err := a2aclient.NewFromCard( + ctx, + card, + a2agrpc.WithGRPCTransport(grpc.WithTransportCredentials(insecure.NewCredentials())), + ) + if err != nil { + return nil, fmt.Errorf("client for %s: %w", agent.ID, err) + } + clients[agent.ID] = cli + } + return &Client{clients: clients}, nil +} + +func (c *Client) StartAll(ctx context.Context, agentIDs []string) error { + for _, id := range agentIDs { + if err := c.send(ctx, id, protocol.Request{Op: protocol.OpStart}); err != nil { + return err + } + } + return nil +} + +func (c *Client) Snapshot(ctx context.Context, agentID string) (consensus.AgentSnapshot, error) { + resp, err := c.sendWithResponse(ctx, agentID, protocol.Request{Op: protocol.OpSnapshot}) + if err != nil { + return consensus.AgentSnapshot{}, err + } + var snap consensus.AgentSnapshot + if err := json.Unmarshal([]byte(resp.Body), &snap); err != nil { + return consensus.AgentSnapshot{}, err + } + return snap, nil +} + +// SubscribeFindings opens the OpStreamFindings server stream to an agent and +// invokes onFinding for every finding the agent produces. It blocks until the +// stream ends or ctx is cancelled, so callers run it in a goroutine. +func (c *Client) SubscribeFindings(ctx context.Context, agentID string, onFinding func(consensus.Finding)) error { + cli, ok := c.clients[agentID] + if !ok { + return fmt.Errorf("unknown agent %q", agentID) + } + reqText, err := json.Marshal(protocol.Request{Op: protocol.OpStreamFindings}) + if err != nil { + return err + } + msg := a2a.NewMessage(a2a.MessageRoleUser, a2a.NewTextPart(string(reqText))) + for ev, streamErr := range cli.SendStreamingMessage(ctx, &a2a.SendMessageRequest{Message: msg}) { + if streamErr != nil { + return streamErr + } + f, ok := findingFromEvent(ev) + if !ok { + continue + } + onFinding(f) + } + return nil +} + +func (c *Client) send(ctx context.Context, agentID string, req protocol.Request) error { + _, err := c.sendWithResponse(ctx, agentID, req) + return err +} + +func (c *Client) sendWithResponse(ctx context.Context, agentID string, req protocol.Request) (protocol.Response, error) { + cli, ok := c.clients[agentID] + if !ok { + return protocol.Response{}, fmt.Errorf("unknown agent %q", agentID) + } + text, err := json.Marshal(req) + if err != nil { + return protocol.Response{}, err + } + msg := a2a.NewMessage(a2a.MessageRoleUser, a2a.NewTextPart(string(text))) + result, err := cli.SendMessage(ctx, &a2a.SendMessageRequest{Message: msg}) + if err != nil { + return protocol.Response{}, err + } + respText, err := responseText(result) + if err != nil { + return protocol.Response{}, err + } + return protocol.DecodeResponse(respText) +} + +func findingFromEvent(ev a2a.Event) (consensus.Finding, bool) { + su, ok := ev.(*a2a.TaskStatusUpdateEvent) + if !ok || su.Status.Message == nil { + return consensus.Finding{}, false + } + text := firstText(su.Status.Message) + if text == "" { + return consensus.Finding{}, false + } + f, err := consensus.DecodeFinding([]byte(text)) + if err != nil { + return consensus.Finding{}, false + } + return f, true +} + +func responseText(result a2a.SendMessageResult) (string, error) { + switch v := result.(type) { + case *a2a.Message: + return firstText(v), nil + case *a2a.Task: + if v.Status.Message != nil { + return firstText(v.Status.Message), nil + } + return "", fmt.Errorf("task response missing message") + default: + return "", fmt.Errorf("unexpected result type %T", result) + } +} + +func firstText(message *a2a.Message) string { + if message == nil { + return "" + } + for _, part := range message.Parts { + if text := part.Text(); text != "" { + return text + } + } + return "" +} diff --git a/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol/protocol.go b/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol/protocol.go new file mode 100644 index 00000000..21f53971 --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol/protocol.go @@ -0,0 +1,59 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Package protocol defines the A2A control/stream ops for the relay topology. +// +// Unlike SLIM's native group session, A2A has no peer multicast: findings must +// pass through the runner, which is the explicit relay hub. Two server-streaming +// legs carry findings: +// - OpStreamFindings: the runner subscribes to each agent (agent streams its +// own findings to the runner). +// - OpStreamRelay: each agent subscribes to the runner (runner streams every +// relayed finding back out to the agents). +// +// OpStart and OpSnapshot remain unary control calls driven by the runner. +package protocol + +import ( + "encoding/json" +) + +const ( + OpStart = "start" + OpSnapshot = "snapshot" + OpStreamFindings = "stream_findings" + OpStreamRelay = "stream_relay" +) + +type Request struct { + Op string `json:"op"` + // AgentIndex identifies the subscriber on OpStreamRelay so the relay can + // avoid echoing an agent's own findings back to it. + AgentIndex int `json:"agentIndex,omitempty"` +} + +type Response struct { + OK bool `json:"ok"` + Error string `json:"error,omitempty"` + Body string `json:"body,omitempty"` +} + +func EncodeRequest(req Request) ([]byte, error) { + return json.Marshal(req) +} + +func DecodeRequest(text string) (Request, error) { + var req Request + err := json.Unmarshal([]byte(text), &req) + return req, err +} + +func EncodeResponse(resp Response) ([]byte, error) { + return json.Marshal(resp) +} + +func DecodeResponse(text string) (Response, error) { + var resp Response + err := json.Unmarshal([]byte(text), &resp) + return resp, err +} diff --git a/benchmarks/slim-vs-a2a-v2/a2a/internal/relay/relay.go b/benchmarks/slim-vs-a2a-v2/a2a/internal/relay/relay.go new file mode 100644 index 00000000..b0ea2b9f --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/a2a/internal/relay/relay.go @@ -0,0 +1,215 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Package relay implements the runner-side relay hub for the A2A topology. +// +// The hub is the central point every finding must pass through, because A2A has +// no peer multicast. It hosts an A2A server exposing OpStreamRelay: each agent +// subscribes and receives a server stream of every relayed finding. The runner +// separately subscribes to each agent's OpStreamFindings stream and calls +// Broadcast for every finding, which fans it out to all other agents' relay +// streams. This relay fan-out is exactly the work native SLIM avoids. +package relay + +import ( + "context" + "fmt" + "iter" + "net" + "net/http" + "sync" + "time" + + "github.com/a2aproject/a2a-go/v2/a2a" + a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" + "github.com/a2aproject/a2a-go/v2/a2asrv" + "github.com/a2aproject/a2a-go/v2/a2asrv/taskstore" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" + "google.golang.org/grpc" +) + +type subscriber struct { + agentIndex int + ch chan consensus.Finding +} + +type Hub struct { + grpcPort int + cardPort int + + mu sync.Mutex + subs map[*subscriber]struct{} + streamRPCCount int + fanoutMS int64 +} + +func NewHub(grpcPort, cardPort int) *Hub { + return &Hub{ + grpcPort: grpcPort, + cardPort: cardPort, + subs: map[*subscriber]struct{}{}, + } +} + +func (h *Hub) CardURL() string { + return fmt.Sprintf("http://127.0.0.1:%d", h.cardPort) +} + +// Serve binds the relay's gRPC and card listeners and serves them in the +// background. It returns once the listeners are bound so callers can hand the +// card URL to agents. +func (h *Hub) Serve() error { + handler := a2asrv.NewHandler( + &relayExecutor{hub: h}, + a2asrv.WithTaskStore(taskstore.NewInMemory(&taskstore.InMemoryStoreConfig{ + Authenticator: func(context.Context) (string, error) { return "slim-vs-a2a-v2", nil }, + })), + ) + + card := &a2a.AgentCard{ + Name: "consensus-v2-relay", + Description: "SLIM vs A2A v2 relay hub", + Version: "1.0.0", + SupportedInterfaces: []*a2a.AgentInterface{ + a2a.NewAgentInterface(fmt.Sprintf("127.0.0.1:%d", h.grpcPort), a2a.TransportProtocolGRPC), + }, + Capabilities: a2a.AgentCapabilities{Streaming: true}, + DefaultInputModes: []string{"text/plain"}, + DefaultOutputModes: []string{"text/plain"}, + } + + cardListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", h.cardPort)) + if err != nil { + return fmt.Errorf("relay card listen: %w", err) + } + grpcListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", h.grpcPort)) + if err != nil { + _ = cardListener.Close() + return fmt.Errorf("relay grpc listen: %w", err) + } + + grpcHandler := a2agrpc.NewHandler(handler) + grpcServer := grpc.NewServer() + grpcHandler.RegisterWith(grpcServer) + + go func() { + mux := http.NewServeMux() + mux.Handle(a2asrv.WellKnownAgentCardPath, a2asrv.NewStaticAgentCardHandler(card)) + _ = http.Serve(cardListener, mux) + }() + go func() { + _ = grpcServer.Serve(grpcListener) + }() + return nil +} + +// Broadcast fans a finding out to every subscriber except its producer. +func (h *Hub) Broadcast(f consensus.Finding) { + start := time.Now() + h.mu.Lock() + targets := make([]*subscriber, 0, len(h.subs)) + for s := range h.subs { + if s.agentIndex == f.AgentIndex { + continue + } + targets = append(targets, s) + } + h.mu.Unlock() + + delivered := 0 + for _, s := range targets { + select { + case s.ch <- f: + delivered++ + case <-time.After(5 * time.Second): + } + } + + h.mu.Lock() + h.streamRPCCount += delivered + h.fanoutMS += time.Since(start).Milliseconds() + h.mu.Unlock() +} + +func (h *Hub) Stats() (streamRPCCount int, fanoutMS int64) { + h.mu.Lock() + defer h.mu.Unlock() + return h.streamRPCCount, h.fanoutMS +} + +func (h *Hub) register(agentIndex int) *subscriber { + s := &subscriber{agentIndex: agentIndex, ch: make(chan consensus.Finding, 1024)} + h.mu.Lock() + h.subs[s] = struct{}{} + h.mu.Unlock() + return s +} + +func (h *Hub) unregister(s *subscriber) { + h.mu.Lock() + delete(h.subs, s) + h.mu.Unlock() +} + +type relayExecutor struct { + hub *Hub +} + +func (e *relayExecutor) Execute(ctx context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { + req, err := protocol.DecodeRequest(firstText(execCtx.Message)) + if err != nil { + return func(yield func(a2a.Event, error) bool) { + yield(nil, fmt.Errorf("decode request: %w", err)) + } + } + if req.Op != protocol.OpStreamRelay { + return func(yield func(a2a.Event, error) bool) { + yield(nil, fmt.Errorf("relay only serves stream_relay, got %q", req.Op)) + } + } + + sub := e.hub.register(req.AgentIndex) + return func(yield func(a2a.Event, error) bool) { + defer e.hub.unregister(sub) + task := a2a.NewSubmittedTask(execCtx, execCtx.Message) + if !yield(task, nil) { + return + } + for { + select { + case <-ctx.Done(): + yield(a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateCompleted, nil), nil) + return + case f := <-sub.ch: + body, err := consensus.EncodeFinding(f) + if err != nil { + continue + } + msg := a2a.NewMessageForTask(a2a.MessageRoleAgent, execCtx, a2a.NewTextPart(string(body))) + ev := a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateWorking, msg) + if !yield(ev, nil) { + return + } + } + } + } +} + +func (e *relayExecutor) Cancel(_ context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { + return func(yield func(a2a.Event, error) bool) { + yield(a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateCanceled, nil), nil) + } +} + +func firstText(message *a2a.Message) string { + if message == nil { + return "" + } + for _, part := range message.Parts { + if text := part.Text(); text != "" { + return text + } + } + return "" +} diff --git a/benchmarks/slim-vs-a2a-v2/go.mod b/benchmarks/slim-vs-a2a-v2/go.mod new file mode 100644 index 00000000..0d354844 --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/go.mod @@ -0,0 +1,25 @@ +module github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2 + +go 1.25.0 + +require ( + github.com/a2aproject/a2a-go/v2 v2.3.0 + github.com/agntcy/slim-bindings-go v1.4.0 + google.golang.org/grpc v1.80.0 + gopkg.in/yaml.v3 v3.0.1 +) + +require ( + github.com/google/uuid v1.6.0 // indirect + github.com/kr/pretty v0.3.1 // indirect + github.com/rogpeppe/go-internal v1.13.1 // indirect + golang.org/x/mod v0.35.0 // indirect + golang.org/x/net v0.53.0 // indirect + golang.org/x/sync v0.20.0 // indirect + golang.org/x/sys v0.43.0 // indirect + golang.org/x/text v0.36.0 // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 // indirect + google.golang.org/protobuf v1.36.11 // indirect + gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 // indirect +) diff --git a/benchmarks/slim-vs-a2a-v2/go.sum b/benchmarks/slim-vs-a2a-v2/go.sum new file mode 100644 index 00000000..d2fc54cd --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/go.sum @@ -0,0 +1,62 @@ +github.com/a2aproject/a2a-go/v2 v2.3.0 h1:wSseKBBlDYBAJLdHZ4puFIL2XyL/5YuMrDw9TXLL1ek= +github.com/a2aproject/a2a-go/v2 v2.3.0/go.mod h1:mkZr8y2bUgAVQsjs/5fHK7xrRlAHDybMEyxWh2tKRC8= +github.com/agntcy/slim-bindings-go v1.4.0 h1:DXAGhe9TWSm33KIgrJFkIcxdqh+/oPWy4Jq5PCI0HvM= +github.com/agntcy/slim-bindings-go v1.4.0/go.mod h1:XK0Ing+REEl8xG79HTMx52XzWK2THuTQA+Y7JTAn428= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= +github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA= +github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= +github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII= +github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o= +go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= +go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= +go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48= +go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8= +go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0= +go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs= +go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18= +go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE= +go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8= +go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew= +go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI= +go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA= +golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM= +golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU= +golang.org/x/net v0.53.0 h1:d+qAbo5L0orcWAr0a9JweQpjXF19LMXJE8Ey7hwOdUA= +golang.org/x/net v0.53.0/go.mod h1:JvMuJH7rrdiCfbeHoo3fCQU24Lf5JJwT9W3sJFulfgs= +golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= +golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI= +golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/text v0.36.0 h1:JfKh3XmcRPqZPKevfXVpI1wXPTqbkE5f7JA92a55Yxg= +golang.org/x/text v0.36.0/go.mod h1:NIdBknypM8iqVmPiuco0Dh6P5Jcdk8lJL0CUebqK164= +gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= +gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= +google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4 h1:yOzSCGPx+cp5VO7IxvZ9SBFF7j1tZVcNtlHR2iYKtVo= +google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4/go.mod h1:Q9HWtNeE7tM9npdIsEvqXj1QJIvVoeAV3rtXtS715Cw= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 h1:tEkOQcXgF6dH1G+MVKZrfpYvozGrzb91k6ha7jireSM= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= +google.golang.org/grpc v1.80.0 h1:Xr6m2WmWZLETvUNvIUmeD5OAagMw3FiKmMlTdViWsHM= +google.golang.org/grpc v1.80.0/go.mod h1:ho/dLnxwi3EDJA4Zghp7k2Ec1+c2jqup0bFkw07bwF4= +google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= +google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/benchmarks/slim-vs-a2a-v2/internal/benchlog/benchlog.go b/benchmarks/slim-vs-a2a-v2/internal/benchlog/benchlog.go new file mode 100644 index 00000000..bdb4999d --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/internal/benchlog/benchlog.go @@ -0,0 +1,89 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package benchlog + +import ( + "fmt" + "io" + "log" + "os" + "strings" + "sync" + "time" +) + +const ( + ImplSLIM = "slim-group-session" + ImplA2A = "a2a-relay-stream" +) + +var ( + mu sync.RWMutex + runStart time.Time + enabled = true + out io.Writer = os.Stderr +) + +func SetRunStart(t time.Time) { + mu.Lock() + runStart = t + mu.Unlock() +} + +func SetEnabled(on bool) { + mu.Lock() + enabled = on + mu.Unlock() +} + +func sinceStart() int64 { + mu.RLock() + start := runStart + mu.RUnlock() + if start.IsZero() { + return 0 + } + return time.Since(start).Milliseconds() +} + +func write(kind, impl string, kv ...string) { + mu.RLock() + on := enabled + mu.RUnlock() + if !on { + return + } + ts := time.Now().UTC().Format(time.RFC3339Nano) + elapsed := sinceStart() + parts := []string{ + fmt.Sprintf("bench ts=%s elapsed_ms=%d impl=%s kind=%s", ts, elapsed, impl, kind), + } + parts = append(parts, kv...) + log.New(out, "", 0).Println(strings.Join(parts, " ")) +} + +func RPC(impl, op, mode string, duration time.Duration, ok bool, kv ...string) { + status := "ok" + if !ok { + status = "error" + } + fields := []string{ + fmt.Sprintf("op=%s", op), + fmt.Sprintf("mode=%s", mode), + fmt.Sprintf("duration_ms=%d", duration.Milliseconds()), + fmt.Sprintf("status=%s", status), + } + fields = append(fields, kv...) + write("rpc", impl, fields...) +} + +func Finding(impl, event string, agentIndex int, findingID int64, kv ...string) { + fields := []string{ + fmt.Sprintf("event=%s", event), + fmt.Sprintf("agent=%d", agentIndex), + fmt.Sprintf("finding_id=%d", findingID), + } + fields = append(fields, kv...) + write("finding", impl, fields...) +} diff --git a/benchmarks/slim-vs-a2a-v2/internal/consensus/consensus.go b/benchmarks/slim-vs-a2a-v2/internal/consensus/consensus.go new file mode 100644 index 00000000..7beb3baa --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/internal/consensus/consensus.go @@ -0,0 +1,313 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package consensus + +import ( + "encoding/json" + "fmt" + "sync" + "time" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" +) + +const consensusConfidence = 0.55 + +type Finding struct { + FindingID int64 `json:"findingId"` + AgentIndex int `json:"agentIndex"` + Round int `json:"round"` + Value int `json:"value"` + Confidence float64 `json:"confidence"` + EmittedAt int64 `json:"emittedAtNs"` +} + +type Config struct { + AgentIndex int + AgentCount int + ValueSpace int + TargetMode string + Seed int64 +} + +type Engine struct { + cfg Config + + mu sync.Mutex + + round int + value int + confidence float64 + targetValue int + lastEmitValue int + lastEmitConf float64 + lastEmitRound int + nextFindingID int64 + distinctSupports map[int]map[int]struct{} + convergedAt int64 + consensusRound int + + propMu sync.Mutex + propDurations []int64 + findingsEmitted int + findingsApplied int +} + +func NewEngine(spec scenario.Spec, agentIndex int) *Engine { + k := spec.ValueSpace + if k <= 0 { + k = 3 + } + target := targetValue(spec, agentIndex) + return &Engine{ + cfg: Config{ + AgentIndex: agentIndex, + AgentCount: spec.Agents, + ValueSpace: k, + TargetMode: spec.TargetMode, + Seed: spec.Seed, + }, + value: agentIndex % k, + confidence: 0.1, + targetValue: target, + lastEmitValue: -1, + distinctSupports: map[int]map[int]struct{}{}, + } +} + +func targetValue(spec scenario.Spec, agentIndex int) int { + k := spec.ValueSpace + if k <= 0 { + k = 3 + } + switch spec.TargetMode { + case scenario.TargetModeMajority: + return (spec.Agents / 2) % k + default: + return agentIndex % k + } +} + +func (e *Engine) TargetValue() int { + return e.targetValue +} + +func (e *Engine) Think() (finding *Finding, emit bool) { + e.mu.Lock() + defer e.mu.Unlock() + + e.round++ + e.refreshConfidenceLocked() + + if e.value == e.targetValue && e.confidence >= consensusConfidence { + if e.convergedAt == 0 { + e.convergedAt = time.Now().UnixNano() + e.consensusRound = e.round + } + } + + if e.lastEmitValue == e.value && e.lastEmitConf == e.confidence && e.lastEmitRound == e.round { + return nil, false + } + + e.lastEmitValue = e.value + e.lastEmitConf = e.confidence + e.lastEmitRound = e.round + e.nextFindingID++ + e.findingsEmitted++ + + f := Finding{ + FindingID: e.nextFindingID, + AgentIndex: e.cfg.AgentIndex, + Round: e.round, + Value: e.value, + Confidence: e.confidence, + EmittedAt: time.Now().UnixNano(), + } + return &f, true +} + +func (e *Engine) refreshConfidenceLocked() { + sources := e.distinctSupports[e.value] + if len(sources)+1 >= (e.cfg.AgentCount+1)/2 { + e.confidence += 0.12 + if e.confidence > 1 { + e.confidence = 1 + } + } + if e.value != e.targetValue && e.round > 2 { + e.value = e.targetValue + e.confidence = maxFloat(e.confidence, 0.35) + } +} + +func (e *Engine) ApplyFinding(f Finding) { + e.mu.Lock() + defer e.mu.Unlock() + + e.findingsApplied++ + if _, ok := e.distinctSupports[f.Value]; !ok { + e.distinctSupports[f.Value] = map[int]struct{}{} + } + e.distinctSupports[f.Value][f.AgentIndex] = struct{}{} + + if f.Confidence >= e.confidence || f.Value == e.targetValue { + e.value = f.Value + e.confidence = maxFloat(e.confidence, f.Confidence) + } + e.refreshConfidenceLocked() + + if e.value == e.targetValue && e.confidence >= consensusConfidence && e.convergedAt == 0 { + e.convergedAt = time.Now().UnixNano() + e.consensusRound = e.round + } +} + +func (e *Engine) RecordPropagation(emitNs int64, recvNs int64) { + if emitNs <= 0 || recvNs <= emitNs { + return + } + e.propMu.Lock() + e.propDurations = append(e.propDurations, (recvNs-emitNs)/int64(time.Millisecond)) + e.propMu.Unlock() +} + +func (e *Engine) HasLocalConsensus() bool { + e.mu.Lock() + defer e.mu.Unlock() + return e.value == e.targetValue && e.confidence >= consensusConfidence +} + +func (e *Engine) LocalState() (value int, confidence float64, round int) { + e.mu.Lock() + defer e.mu.Unlock() + return e.value, e.confidence, e.round +} + +func (e *Engine) ConvergedAtNs() int64 { + e.mu.Lock() + defer e.mu.Unlock() + return e.convergedAt +} + +func (e *Engine) ConsensusRound() int { + e.mu.Lock() + defer e.mu.Unlock() + return e.consensusRound +} + +func (e *Engine) FindingsEmitted() int { + e.mu.Lock() + defer e.mu.Unlock() + return e.findingsEmitted +} + +func (e *Engine) FindingsApplied() int { + e.mu.Lock() + defer e.mu.Unlock() + return e.findingsApplied +} + +func (e *Engine) PropagationStats() (avg, p95 int64) { + e.propMu.Lock() + defer e.propMu.Unlock() + if len(e.propDurations) == 0 { + return 0, 0 + } + var sum int64 + for _, d := range e.propDurations { + sum += d + } + avg = sum / int64(len(e.propDurations)) + p95 = percentile(e.propDurations, 0.95) + return avg, p95 +} + +func EncodeFinding(f Finding) ([]byte, error) { + return json.Marshal(f) +} + +func DecodeFinding(data []byte) (Finding, error) { + var f Finding + err := json.Unmarshal(data, &f) + return f, err +} + +func maxFloat(a, b float64) float64 { + if a > b { + return a + } + return b +} + +func percentile(values []int64, p float64) int64 { + if len(values) == 0 { + return 0 + } + sorted := append([]int64(nil), values...) + for i := 0; i < len(sorted); i++ { + for j := i + 1; j < len(sorted); j++ { + if sorted[j] < sorted[i] { + sorted[i], sorted[j] = sorted[j], sorted[i] + } + } + } + idx := int(float64(len(sorted)-1) * p) + return sorted[idx] +} + +type AgentSnapshot struct { + AgentIndex int `json:"agentIndex"` + Value int `json:"value"` + Confidence float64 `json:"confidence"` + Round int `json:"round"` + HasConsensus bool `json:"hasConsensus"` + ConvergedAtNs int64 `json:"convergedAtNs"` + ConsensusRound int `json:"consensusRound"` + FindingsEmitted int `json:"findingsEmitted"` + FindingsApplied int `json:"findingsApplied"` + AvgPropagationMs int64 `json:"avgPropagationMs"` + P95PropagationMs int64 `json:"p95PropagationMs"` +} + +func (e *Engine) Snapshot() AgentSnapshot { + value, conf, round := e.LocalState() + avg, p95 := e.PropagationStats() + return AgentSnapshot{ + AgentIndex: e.cfg.AgentIndex, + Value: value, + Confidence: conf, + Round: round, + HasConsensus: e.HasLocalConsensus(), + ConvergedAtNs: e.ConvergedAtNs(), + ConsensusRound: e.ConsensusRound(), + FindingsEmitted: e.FindingsEmitted(), + FindingsApplied: e.FindingsApplied(), + AvgPropagationMs: avg, + P95PropagationMs: p95, + } +} + +func GlobalConsensus(snapshots []AgentSnapshot) (ok bool, value int) { + if len(snapshots) == 0 { + return false, 0 + } + if !snapshots[0].HasConsensus { + return false, 0 + } + v := snapshots[0].Value + for _, s := range snapshots[1:] { + if !s.HasConsensus || s.Value != v { + return false, 0 + } + } + return true, v +} + +func ValidateSnapshots(snapshots []AgentSnapshot, expected int) error { + if len(snapshots) != expected { + return fmt.Errorf("expected %d snapshots, got %d", expected, len(snapshots)) + } + return nil +} diff --git a/benchmarks/slim-vs-a2a-v2/internal/metrics/metrics.go b/benchmarks/slim-vs-a2a-v2/internal/metrics/metrics.go new file mode 100644 index 00000000..adc1dff2 --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/internal/metrics/metrics.go @@ -0,0 +1,120 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package metrics + +import ( + "encoding/csv" + "encoding/json" + "os" + "path/filepath" + "strconv" +) + +type RunResult struct { + ScenarioName string `json:"scenario_name"` + Domain string `json:"domain"` + Implementation string `json:"implementation"` + Agents int `json:"agents"` + ThinkTimeMs int64 `json:"think_time_ms"` + ConsensusWallMS int64 `json:"consensus_wall_ms"` + ConsensusRound int `json:"consensus_round"` + FindingsEmitted int `json:"findings_emitted"` + FindingsReceivedTotal int `json:"findings_received_total"` + AvgPropagationMS int64 `json:"avg_propagation_ms"` + P95PropagationMS int64 `json:"p95_propagation_ms"` + LastAgentConvergeMS int64 `json:"last_agent_converge_ms"` + CoordFanoutMS int64 `json:"coord_fanout_ms"` + StreamRPCCount int `json:"stream_rpc_count"` + UnicastRPCCount int `json:"unicast_rpc_count"` + Success bool `json:"success"` + Error string `json:"error,omitempty"` +} + +func WriteJSON(path string, result RunResult) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + data, err := json.MarshalIndent(result, "", " ") + if err != nil { + return err + } + return os.WriteFile(path, data, 0o644) +} + +func AppendTSV(path string, result RunResult) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + writeHeader := true + if info, err := os.Stat(path); err == nil { + writeHeader = info.Size() == 0 + } else if !os.IsNotExist(err) { + return err + } + + file, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + return err + } + defer file.Close() + + w := csv.NewWriter(file) + w.Comma = '\t' + if writeHeader { + if err := w.Write([]string{ + "scenario_name", "domain", "implementation", "agents", "think_time_ms", + "consensus_wall_ms", "consensus_round", "findings_emitted", "findings_received_total", + "avg_propagation_ms", "p95_propagation_ms", "last_agent_converge_ms", + "coord_fanout_ms", "stream_rpc_count", "unicast_rpc_count", + "success", "error", + }); err != nil { + return err + } + } + if err := w.Write([]string{ + result.ScenarioName, + result.Domain, + result.Implementation, + strconv.Itoa(result.Agents), + strconv.FormatInt(result.ThinkTimeMs, 10), + strconv.FormatInt(result.ConsensusWallMS, 10), + strconv.Itoa(result.ConsensusRound), + strconv.Itoa(result.FindingsEmitted), + strconv.Itoa(result.FindingsReceivedTotal), + strconv.FormatInt(result.AvgPropagationMS, 10), + strconv.FormatInt(result.P95PropagationMS, 10), + strconv.FormatInt(result.LastAgentConvergeMS, 10), + strconv.FormatInt(result.CoordFanoutMS, 10), + strconv.Itoa(result.StreamRPCCount), + strconv.Itoa(result.UnicastRPCCount), + strconv.FormatBool(result.Success), + result.Error, + }); err != nil { + return err + } + w.Flush() + return w.Error() +} + +func AggregatePropagation(durations []int64) (avg, p95 int64) { + if len(durations) == 0 { + return 0, 0 + } + var sum int64 + for _, d := range durations { + sum += d + } + avg = sum / int64(len(durations)) + sorted := append([]int64(nil), durations...) + for i := 0; i < len(sorted); i++ { + for j := i + 1; j < len(sorted); j++ { + if sorted[j] < sorted[i] { + sorted[i], sorted[j] = sorted[j], sorted[i] + } + } + } + idx := int(float64(len(sorted)-1) * 0.95) + p95 = sorted[idx] + return avg, p95 +} diff --git a/benchmarks/slim-vs-a2a-v2/internal/scenario/scenario.go b/benchmarks/slim-vs-a2a-v2/internal/scenario/scenario.go new file mode 100644 index 00000000..ab9e6311 --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/internal/scenario/scenario.go @@ -0,0 +1,246 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package scenario + +import ( + "fmt" + "os" + "strings" + + "gopkg.in/yaml.v3" +) + +const ( + TargetModeMajority = "majority" +) + +type ConsensusScenario struct { + APIVersion string `yaml:"apiVersion"` + Kind string `yaml:"kind"` + Metadata Metadata `yaml:"metadata"` + Spec Spec `yaml:"spec"` + Agents []Agent `yaml:"agents"` +} + +type Metadata struct { + Name string `yaml:"name"` + Domain string `yaml:"domain"` + Description string `yaml:"description"` +} + +type Spec struct { + Agents int `yaml:"agents"` + ThinkTimeMs int64 `yaml:"thinkTimeMs"` + FindingEmitDelayMs int64 `yaml:"findingEmitDelayMs"` + MaxRounds int `yaml:"maxRounds"` + TargetMode string `yaml:"targetMode"` + Seed int64 `yaml:"seed"` + ValueSpace int `yaml:"valueSpace"` +} + +type Agent struct { + ID string `yaml:"id"` + SlimName string `yaml:"slimName"` + A2APort int `yaml:"a2aPort"` + CardPort int `yaml:"cardPort,omitempty"` + Role string `yaml:"role,omitempty"` +} + +func LoadFile(path string) (*ConsensusScenario, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var s ConsensusScenario + if err := yaml.Unmarshal(data, &s); err != nil { + return nil, err + } + if err := s.Validate(); err != nil { + return nil, err + } + return &s, nil +} + +func (s *ConsensusScenario) Validate() error { + if s.APIVersion != "bench.agntcy.io/v2" { + return fmt.Errorf("unsupported apiVersion %q", s.APIVersion) + } + if s.Kind != "ConsensusScenario" { + return fmt.Errorf("unsupported kind %q", s.Kind) + } + if s.Metadata.Name == "" { + return fmt.Errorf("metadata.name is required") + } + if s.Spec.Agents < 2 { + return fmt.Errorf("spec.agents must be >= 2") + } + if len(s.Agents) != s.Spec.Agents { + return fmt.Errorf("agents list length %d != spec.agents %d", len(s.Agents), s.Spec.Agents) + } + if s.Spec.ThinkTimeMs <= 0 { + return fmt.Errorf("spec.thinkTimeMs must be > 0") + } + if s.Spec.MaxRounds <= 0 { + return fmt.Errorf("spec.maxRounds must be > 0") + } + if s.Spec.TargetMode == "" { + s.Spec.TargetMode = TargetModeMajority + } + if s.Spec.ValueSpace <= 0 { + s.Spec.ValueSpace = 3 + } + if s.Spec.FindingEmitDelayMs <= 0 { + s.Spec.FindingEmitDelayMs = 1 + } + for i, a := range s.Agents { + if a.ID == "" { + return fmt.Errorf("agents[%d].id is required", i) + } + if a.SlimName == "" { + return fmt.Errorf("agents[%d].slimName is required", i) + } + if a.A2APort <= 0 { + return fmt.Errorf("agents[%d].a2aPort is required", i) + } + } + return nil +} + +func (s *ConsensusScenario) CardPort(agent Agent) int { + if agent.CardPort > 0 { + return agent.CardPort + } + return agent.A2APort + 1000 +} + +func (s *ConsensusScenario) CardBaseURL(agent Agent) string { + return fmt.Sprintf("http://127.0.0.1:%d", s.CardPort(agent)) +} + +func (s *ConsensusScenario) AgentByID(id string) (Agent, bool) { + for _, a := range s.Agents { + if a.ID == id { + return a, true + } + } + return Agent{}, false +} + +func (s *ConsensusScenario) Coordinator() Agent { + for _, a := range s.Agents { + if a.Role == "coordinator" || a.ID == "agent-0" { + return a + } + } + return s.Agents[0] +} + +func (s *ConsensusScenario) WorkerAgents() []Agent { + coord := s.Coordinator() + var out []Agent + for _, a := range s.Agents { + if a.ID != coord.ID { + out = append(out, a) + } + } + return out +} + +func (s *ConsensusScenario) AgentIDs() []string { + ids := make([]string, len(s.Agents)) + for i, a := range s.Agents { + ids[i] = a.ID + } + return ids +} + +func (s *ConsensusScenario) SlimNames() []string { + names := make([]string, len(s.Agents)) + for i, a := range s.Agents { + names[i] = a.SlimName + } + return names +} + +type GenerateOptions struct { + Family string + Agents int + ThinkTimeMs int64 + Seed int64 +} + +func Generate(opts GenerateOptions) (*ConsensusScenario, error) { + if opts.Agents < 2 { + return nil, fmt.Errorf("agents must be >= 2") + } + if opts.ThinkTimeMs <= 0 { + opts.ThinkTimeMs = 10 + } + if opts.Family == "" { + opts.Family = "hypothesis-convergence" + } + if opts.Seed == 0 { + opts.Seed = 42 + } + + name := fmt.Sprintf("%s-%dagents-%dms", opts.Family, opts.Agents, opts.ThinkTimeMs) + agents := make([]Agent, opts.Agents) + for i := 0; i < opts.Agents; i++ { + id := fmt.Sprintf("agent-%d", i) + role := "worker" + if i == 0 { + role = "coordinator" + } + agents[i] = Agent{ + ID: id, + SlimName: fmt.Sprintf("agntcy/bench-v2/%s", id), + A2APort: 9700 + i*11, + Role: role, + } + } + + s := &ConsensusScenario{ + APIVersion: "bench.agntcy.io/v2", + Kind: "ConsensusScenario", + Metadata: Metadata{ + Name: name, + Domain: opts.Family, + Description: "Generated consensus scenario for transport sweeps", + }, + Spec: Spec{ + Agents: opts.Agents, + ThinkTimeMs: opts.ThinkTimeMs, + FindingEmitDelayMs: maxInt64(1, opts.ThinkTimeMs/10), + MaxRounds: 200, + TargetMode: TargetModeMajority, + Seed: opts.Seed, + ValueSpace: 3, + }, + Agents: agents, + } + return s, s.Validate() +} + +func Marshal(s *ConsensusScenario) ([]byte, error) { + return yaml.Marshal(s) +} + +func WriteFile(path string, s *ConsensusScenario) error { + data, err := Marshal(s) + if err != nil { + return err + } + return os.WriteFile(path, data, 0o644) +} + +func maxInt64(a, b int64) int64 { + if a > b { + return a + } + return b +} + +func NormalizeFamily(input string) string { + return strings.TrimSpace(input) +} diff --git a/benchmarks/slim-vs-a2a-v2/internal/scenario/scenario_test.go b/benchmarks/slim-vs-a2a-v2/internal/scenario/scenario_test.go new file mode 100644 index 00000000..f39aad08 --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/internal/scenario/scenario_test.go @@ -0,0 +1,23 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package scenario_test + +import ( + "testing" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" +) + +func TestGenerateAndValidate(t *testing.T) { + s, err := scenario.Generate(scenario.GenerateOptions{Agents: 5, ThinkTimeMs: 10}) + if err != nil { + t.Fatal(err) + } + if len(s.Agents) != 5 { + t.Fatalf("agents=%d", len(s.Agents)) + } + if s.Coordinator().ID != "agent-0" { + t.Fatalf("coordinator=%s", s.Coordinator().ID) + } +} diff --git a/benchmarks/slim-vs-a2a-v2/plans/README.md b/benchmarks/slim-vs-a2a-v2/plans/README.md new file mode 100644 index 00000000..d7cfc76b --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/plans/README.md @@ -0,0 +1,67 @@ +# Consensus scenarios (v2) + +YAML `ConsensusScenario` plans for the SLIM vs A2A v2 consensus streaming benchmark. +Each scenario defines N agents that run a deterministic **distributed hypothesis convergence** +workload: agents think in parallel, emit findings, and must reach identical local consensus. + +## Topologies compared + +The two implementations highlight one structural difference: + +- **SLIM (`slim-group-session`)** — agents join a native SLIM group session and + broadcast findings peer-to-peer; the dataplane fans them out. **There is no + application relay.** The runner only moderates (creates the session, invites + agents, sends one start) and then passively observes snapshots that agents + push to it on convergence. +- **A2A (`a2a-relay-stream`)** — A2A has no peer multicast, so the runner is an + explicit **relay hub**. Two server-streaming legs carry findings: agents stream + findings to the runner (`OpStreamFindings`) and subscribe to the runner's + relayed-finding stream (`OpStreamRelay`). Every finding therefore makes an + extra relay hop, which is the overhead `consensus_wall_ms` is expected to show. + +## Schema + +```yaml +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-10agents-10ms + domain: hypothesis-convergence +spec: + agents: 10 + thinkTimeMs: 10 + findingEmitDelayMs: 1 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: worker # legacy field; both transports now treat all agents as peers + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker +``` + +## Generate sweep scenarios + +```bash +go run ./tools/gen_scenario \ + -family hypothesis-convergence \ + -agents 10 \ + -think-ms 20 \ + -output plans/sweeps/hypothesis-convergence-10ag-20ms.yaml +``` + +## Run comparison + +```bash +task build +task compare:plan PLAN=hypothesis-convergence-5ag-20ms +task compare:report +``` + +Primary metric: `consensus_wall_ms` in `reports/results.tsv`. diff --git a/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml b/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml new file mode 100644 index 00000000..51f4402d --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml @@ -0,0 +1,55 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-10agents-20ms + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 10 + thinkTimeMs: 20 + findingEmitDelayMs: 2 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker + - id: agent-5 + slimName: agntcy/bench-v2/agent-5 + a2aPort: 9755 + role: worker + - id: agent-6 + slimName: agntcy/bench-v2/agent-6 + a2aPort: 9766 + role: worker + - id: agent-7 + slimName: agntcy/bench-v2/agent-7 + a2aPort: 9777 + role: worker + - id: agent-8 + slimName: agntcy/bench-v2/agent-8 + a2aPort: 9788 + role: worker + - id: agent-9 + slimName: agntcy/bench-v2/agent-9 + a2aPort: 9799 + role: worker diff --git a/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-17ag-20ms.yaml b/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-17ag-20ms.yaml new file mode 100644 index 00000000..2100ca6e --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-17ag-20ms.yaml @@ -0,0 +1,83 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-17agents-20ms + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 17 + thinkTimeMs: 20 + findingEmitDelayMs: 2 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker + - id: agent-5 + slimName: agntcy/bench-v2/agent-5 + a2aPort: 9755 + role: worker + - id: agent-6 + slimName: agntcy/bench-v2/agent-6 + a2aPort: 9766 + role: worker + - id: agent-7 + slimName: agntcy/bench-v2/agent-7 + a2aPort: 9777 + role: worker + - id: agent-8 + slimName: agntcy/bench-v2/agent-8 + a2aPort: 9788 + role: worker + - id: agent-9 + slimName: agntcy/bench-v2/agent-9 + a2aPort: 9799 + role: worker + - id: agent-10 + slimName: agntcy/bench-v2/agent-10 + a2aPort: 9810 + role: worker + - id: agent-11 + slimName: agntcy/bench-v2/agent-11 + a2aPort: 9821 + role: worker + - id: agent-12 + slimName: agntcy/bench-v2/agent-12 + a2aPort: 9832 + role: worker + - id: agent-13 + slimName: agntcy/bench-v2/agent-13 + a2aPort: 9843 + role: worker + - id: agent-14 + slimName: agntcy/bench-v2/agent-14 + a2aPort: 9854 + role: worker + - id: agent-15 + slimName: agntcy/bench-v2/agent-15 + a2aPort: 9865 + role: worker + - id: agent-16 + slimName: agntcy/bench-v2/agent-16 + a2aPort: 9876 + role: worker diff --git a/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-20ag-20ms.yaml b/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-20ag-20ms.yaml new file mode 100644 index 00000000..a9142590 --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-20ag-20ms.yaml @@ -0,0 +1,95 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-20agents-20ms + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 20 + thinkTimeMs: 20 + findingEmitDelayMs: 2 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker + - id: agent-5 + slimName: agntcy/bench-v2/agent-5 + a2aPort: 9755 + role: worker + - id: agent-6 + slimName: agntcy/bench-v2/agent-6 + a2aPort: 9766 + role: worker + - id: agent-7 + slimName: agntcy/bench-v2/agent-7 + a2aPort: 9777 + role: worker + - id: agent-8 + slimName: agntcy/bench-v2/agent-8 + a2aPort: 9788 + role: worker + - id: agent-9 + slimName: agntcy/bench-v2/agent-9 + a2aPort: 9799 + role: worker + - id: agent-10 + slimName: agntcy/bench-v2/agent-10 + a2aPort: 9810 + role: worker + - id: agent-11 + slimName: agntcy/bench-v2/agent-11 + a2aPort: 9821 + role: worker + - id: agent-12 + slimName: agntcy/bench-v2/agent-12 + a2aPort: 9832 + role: worker + - id: agent-13 + slimName: agntcy/bench-v2/agent-13 + a2aPort: 9843 + role: worker + - id: agent-14 + slimName: agntcy/bench-v2/agent-14 + a2aPort: 9854 + role: worker + - id: agent-15 + slimName: agntcy/bench-v2/agent-15 + a2aPort: 9865 + role: worker + - id: agent-16 + slimName: agntcy/bench-v2/agent-16 + a2aPort: 9876 + role: worker + - id: agent-17 + slimName: agntcy/bench-v2/agent-17 + a2aPort: 9887 + role: worker + - id: agent-18 + slimName: agntcy/bench-v2/agent-18 + a2aPort: 9898 + role: worker + - id: agent-19 + slimName: agntcy/bench-v2/agent-19 + a2aPort: 9909 + role: worker diff --git a/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-50ag-20ms.yaml b/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-50ag-20ms.yaml new file mode 100644 index 00000000..c36402b9 --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-50ag-20ms.yaml @@ -0,0 +1,215 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-50agents-20ms + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 50 + thinkTimeMs: 20 + findingEmitDelayMs: 2 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker + - id: agent-5 + slimName: agntcy/bench-v2/agent-5 + a2aPort: 9755 + role: worker + - id: agent-6 + slimName: agntcy/bench-v2/agent-6 + a2aPort: 9766 + role: worker + - id: agent-7 + slimName: agntcy/bench-v2/agent-7 + a2aPort: 9777 + role: worker + - id: agent-8 + slimName: agntcy/bench-v2/agent-8 + a2aPort: 9788 + role: worker + - id: agent-9 + slimName: agntcy/bench-v2/agent-9 + a2aPort: 9799 + role: worker + - id: agent-10 + slimName: agntcy/bench-v2/agent-10 + a2aPort: 9810 + role: worker + - id: agent-11 + slimName: agntcy/bench-v2/agent-11 + a2aPort: 9821 + role: worker + - id: agent-12 + slimName: agntcy/bench-v2/agent-12 + a2aPort: 9832 + role: worker + - id: agent-13 + slimName: agntcy/bench-v2/agent-13 + a2aPort: 9843 + role: worker + - id: agent-14 + slimName: agntcy/bench-v2/agent-14 + a2aPort: 9854 + role: worker + - id: agent-15 + slimName: agntcy/bench-v2/agent-15 + a2aPort: 9865 + role: worker + - id: agent-16 + slimName: agntcy/bench-v2/agent-16 + a2aPort: 9876 + role: worker + - id: agent-17 + slimName: agntcy/bench-v2/agent-17 + a2aPort: 9887 + role: worker + - id: agent-18 + slimName: agntcy/bench-v2/agent-18 + a2aPort: 9898 + role: worker + - id: agent-19 + slimName: agntcy/bench-v2/agent-19 + a2aPort: 9909 + role: worker + - id: agent-20 + slimName: agntcy/bench-v2/agent-20 + a2aPort: 9920 + role: worker + - id: agent-21 + slimName: agntcy/bench-v2/agent-21 + a2aPort: 9931 + role: worker + - id: agent-22 + slimName: agntcy/bench-v2/agent-22 + a2aPort: 9942 + role: worker + - id: agent-23 + slimName: agntcy/bench-v2/agent-23 + a2aPort: 9953 + role: worker + - id: agent-24 + slimName: agntcy/bench-v2/agent-24 + a2aPort: 9964 + role: worker + - id: agent-25 + slimName: agntcy/bench-v2/agent-25 + a2aPort: 9975 + role: worker + - id: agent-26 + slimName: agntcy/bench-v2/agent-26 + a2aPort: 9986 + role: worker + - id: agent-27 + slimName: agntcy/bench-v2/agent-27 + a2aPort: 9997 + role: worker + - id: agent-28 + slimName: agntcy/bench-v2/agent-28 + a2aPort: 10008 + role: worker + - id: agent-29 + slimName: agntcy/bench-v2/agent-29 + a2aPort: 10019 + role: worker + - id: agent-30 + slimName: agntcy/bench-v2/agent-30 + a2aPort: 10030 + role: worker + - id: agent-31 + slimName: agntcy/bench-v2/agent-31 + a2aPort: 10041 + role: worker + - id: agent-32 + slimName: agntcy/bench-v2/agent-32 + a2aPort: 10052 + role: worker + - id: agent-33 + slimName: agntcy/bench-v2/agent-33 + a2aPort: 10063 + role: worker + - id: agent-34 + slimName: agntcy/bench-v2/agent-34 + a2aPort: 10074 + role: worker + - id: agent-35 + slimName: agntcy/bench-v2/agent-35 + a2aPort: 10085 + role: worker + - id: agent-36 + slimName: agntcy/bench-v2/agent-36 + a2aPort: 10096 + role: worker + - id: agent-37 + slimName: agntcy/bench-v2/agent-37 + a2aPort: 10107 + role: worker + - id: agent-38 + slimName: agntcy/bench-v2/agent-38 + a2aPort: 10118 + role: worker + - id: agent-39 + slimName: agntcy/bench-v2/agent-39 + a2aPort: 10129 + role: worker + - id: agent-40 + slimName: agntcy/bench-v2/agent-40 + a2aPort: 10140 + role: worker + - id: agent-41 + slimName: agntcy/bench-v2/agent-41 + a2aPort: 10151 + role: worker + - id: agent-42 + slimName: agntcy/bench-v2/agent-42 + a2aPort: 10162 + role: worker + - id: agent-43 + slimName: agntcy/bench-v2/agent-43 + a2aPort: 10173 + role: worker + - id: agent-44 + slimName: agntcy/bench-v2/agent-44 + a2aPort: 10184 + role: worker + - id: agent-45 + slimName: agntcy/bench-v2/agent-45 + a2aPort: 10195 + role: worker + - id: agent-46 + slimName: agntcy/bench-v2/agent-46 + a2aPort: 10206 + role: worker + - id: agent-47 + slimName: agntcy/bench-v2/agent-47 + a2aPort: 10217 + role: worker + - id: agent-48 + slimName: agntcy/bench-v2/agent-48 + a2aPort: 10228 + role: worker + - id: agent-49 + slimName: agntcy/bench-v2/agent-49 + a2aPort: 10239 + role: worker diff --git a/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml b/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml new file mode 100644 index 00000000..e37985ae --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml @@ -0,0 +1,35 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-5agents-20ms + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 5 + thinkTimeMs: 20 + findingEmitDelayMs: 2 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker diff --git a/benchmarks/slim-vs-a2a-v2/slim/cmd/agent/main.go b/benchmarks/slim-vs-a2a-v2/slim/cmd/agent/main.go new file mode 100644 index 00000000..bae30c50 --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/slim/cmd/agent/main.go @@ -0,0 +1,51 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "flag" + "fmt" + "log" + "time" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/slim/internal/agent" +) + +func main() { + slimName := flag.String("slim-name", "", "SLIM identity org/group/app") + endpoint := flag.String("endpoint", "http://127.0.0.1:46357", "SLIM dataplane endpoint") + scenarioFile := flag.String("scenario-file", "", "path to consensus scenario yaml") + agentIndex := flag.Int("agent-index", 0, "agent index in scenario") + flag.Parse() + + if *slimName == "" || *scenarioFile == "" { + log.Fatal("--slim-name and --scenario-file are required") + } + + s, err := scenario.LoadFile(*scenarioFile) + if err != nil { + log.Fatalf("load scenario: %v", err) + } + if *agentIndex < 0 || *agentIndex >= len(s.Agents) { + log.Fatalf("agent-index out of range") + } + + rt := agent.NewRuntime(s, *agentIndex, *slimName, *endpoint) + if err := rt.Setup(); err != nil { + log.Fatalf("setup: %v", err) + } + defer rt.Close() + + fmt.Printf("SLIM_AGENT_READY name=%s index=%d scenario=%s\n", *slimName, *agentIndex, s.Metadata.Name) + + // Block until the moderator invites us into the group session. + if err := rt.Join(60 * time.Second); err != nil { + log.Fatalf("join group session: %v", err) + } + + if err := rt.Run(); err != nil { + log.Printf("receive loop ended: %v", err) + } +} diff --git a/benchmarks/slim-vs-a2a-v2/slim/cmd/runner/main.go b/benchmarks/slim-vs-a2a-v2/slim/cmd/runner/main.go new file mode 100644 index 00000000..3f904251 --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/slim/cmd/runner/main.go @@ -0,0 +1,334 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Command runner is the SLIM benchmark driver. It acts as a group-session +// moderator (creates the session, invites every agent, broadcasts a single +// start) and then becomes a passive observer: it never relays findings. Agents +// broadcast findings to each other directly over the SLIM dataplane and push +// their snapshots to the runner the instant they converge. Global consensus is +// detected event-driven from those pushed snapshots. +package main + +import ( + "encoding/json" + "flag" + "fmt" + "log" + "os" + "os/exec" + "strings" + "time" + + slim "github.com/agntcy/slim-bindings-go" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/metrics" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/slim/internal/protocol" +) + +const ( + runnerSlimName = "agntcy/bench-v2/runner" + groupChannelName = "agntcy/bench-v2/consensus" + defaultSharedSecret = "demo-shared-secret-min-32-chars!!" +) + +func main() { + scenarioPath := flag.String("scenario", "", "path to consensus scenario yaml") + endpoint := flag.String("endpoint", "http://127.0.0.1:46357", "SLIM dataplane endpoint") + agentBin := flag.String("agent-bin", "", "path to slim-agent binary") + outputJSON := flag.String("output-json", "", "write run metrics json") + outputTSV := flag.String("output-tsv", "", "append run metrics tsv") + waitReady := flag.Duration("wait-ready", 3*time.Second, "wait for agents to start") + quiet := flag.Bool("quiet", false, "disable benchmark logs") + flag.Parse() + + if *quiet { + benchlog.SetEnabled(false) + } + if *scenarioPath == "" { + log.Fatal("--scenario is required") + } + + s, err := scenario.LoadFile(*scenarioPath) + if err != nil { + log.Fatalf("load scenario: %v", err) + } + + agentPath := *agentBin + if agentPath == "" { + agentPath = os.Getenv("SLIM_AGENT_BIN") + } + if agentPath == "" { + log.Fatal("set --agent-bin or SLIM_AGENT_BIN") + } + + procs := startAgents(s, agentPath, *endpoint, *scenarioPath) + defer stopAgents(procs) + + mod, err := newModerator(*endpoint, s) + if err != nil { + log.Fatalf("moderator: %v", err) + } + defer mod.Close() + + // Let agents come up and reach ListenForSessionAsync before inviting. + time.Sleep(*waitReady) + if err := mod.InviteAll(); err != nil { + log.Fatalf("invite agents: %v", err) + } + + runStart := time.Now() + benchlog.SetRunStart(runStart) + if err := mod.Start(); err != nil { + log.Fatalf("broadcast start: %v", err) + } + + result := metrics.RunResult{ + ScenarioName: s.Metadata.Name, + Domain: s.Metadata.Domain, + Implementation: benchlog.ImplSLIM, + Agents: len(s.Agents), + ThinkTimeMs: s.Spec.ThinkTimeMs, + } + + latest := map[int]consensus.AgentSnapshot{} + n := len(s.Agents) + deadline := time.Now().Add(2 * time.Minute) + for time.Now().Before(deadline) { + timeout := time.Second + msg, err := mod.session.GetMessageAsync(&timeout) + if err != nil { + if isTimeout(err) { + continue + } + break + } + env, derr := protocol.Decode(msg.Payload) + if derr != nil || env.Kind != protocol.KindSnapshot || env.Snapshot == nil { + continue + } + latest[env.Snapshot.AgentIndex] = *env.Snapshot + if len(latest) == n { + if ok, _ := consensus.GlobalConsensus(snapshotSlice(latest)); ok { + result.Success = true + break + } + } + } + + snapshots := snapshotSlice(latest) + result = aggregateResult(result, runStart, snapshots) + + stopAgents(procs) + + if *outputJSON != "" { + if err := metrics.WriteJSON(*outputJSON, result); err != nil { + log.Fatalf("write json: %v", err) + } + } + if *outputTSV != "" { + if err := metrics.AppendTSV(*outputTSV, result); err != nil { + log.Fatalf("write tsv: %v", err) + } + } + + data, _ := json.MarshalIndent(result, "", " ") + fmt.Println(string(data)) + if !result.Success { + os.Exit(1) + } +} + +// moderator owns the group session: it creates it, invites agents, and +// broadcasts the start signal. It does not relay any traffic. +type moderator struct { + app *slim.App + connID uint64 + session *slim.Session + agents []scenario.Agent +} + +func newModerator(endpoint string, s *scenario.ConsensusScenario) (*moderator, error) { + slim.InitializeWithDefaults() + service := slim.GetGlobalService() + + name, err := slim.NameFromString(runnerSlimName) + if err != nil { + return nil, err + } + app, err := service.CreateAppWithSecret(name, defaultSharedSecret) + if err != nil { + return nil, err + } + connID, err := service.Connect(slim.NewInsecureClientConfig(endpoint)) + if err != nil { + app.Destroy() + return nil, err + } + if err := app.Subscribe(name, &connID); err != nil { + app.Destroy() + return nil, err + } + + channelName, err := slim.NameFromString(groupChannelName) + if err != nil { + app.Destroy() + return nil, err + } + interval := 5 * time.Second + maxRetries := uint32(5) + config := slim.SessionConfig{ + SessionType: slim.SessionTypeGroup, + MaxRetries: &maxRetries, + Interval: &interval, + Metadata: map[string]string{}, + } + session, err := app.CreateSessionAndWaitAsync(config, channelName) + if err != nil { + app.Destroy() + return nil, err + } + // Give the session a moment to establish before inviting. + time.Sleep(100 * time.Millisecond) + + return &moderator{app: app, connID: connID, session: session, agents: s.Agents}, nil +} + +func (m *moderator) InviteAll() error { + for _, a := range m.agents { + name, err := slim.NameFromString(a.SlimName) + if err != nil { + return err + } + if err := m.app.SetRouteAsync(name, m.connID); err != nil { + return fmt.Errorf("set route %s: %w", a.SlimName, err) + } + if err := m.session.InviteAndWaitAsync(name); err != nil { + return fmt.Errorf("invite %s: %w", a.SlimName, err) + } + } + return nil +} + +func (m *moderator) Start() error { + payload, err := protocol.Encode(protocol.Envelope{Kind: protocol.KindStart}) + if err != nil { + return err + } + return m.session.PublishAndWaitAsync(payload, nil, nil) +} + +func (m *moderator) Close() { + if m.session != nil && m.app != nil { + _ = m.app.DeleteSessionAndWaitAsync(m.session) + } + if m.app != nil { + m.app.Destroy() + } +} + +func aggregateResult(result metrics.RunResult, runStart time.Time, snapshots []consensus.AgentSnapshot) metrics.RunResult { + if len(snapshots) == 0 { + result.Error = "no agent snapshots" + result.ConsensusWallMS = time.Since(runStart).Milliseconds() + return result + } + + var ( + totalEmitted int + totalApplied int + lastConvergeMS int64 + propDurations []int64 + maxConsensusRound int + maxRound int + ) + + for _, snap := range snapshots { + totalEmitted += snap.FindingsEmitted + totalApplied += snap.FindingsApplied + if snap.ConsensusRound > maxConsensusRound { + maxConsensusRound = snap.ConsensusRound + } + if snap.Round > maxRound { + maxRound = snap.Round + } + if snap.ConvergedAtNs > 0 { + ms := (snap.ConvergedAtNs - runStart.UnixNano()) / int64(time.Millisecond) + if ms > lastConvergeMS { + lastConvergeMS = ms + } + } + if snap.AvgPropagationMs > 0 { + propDurations = append(propDurations, snap.AvgPropagationMs) + } + } + + if result.Success { + result.ConsensusWallMS = time.Since(runStart).Milliseconds() + if lastConvergeMS > 0 { + result.ConsensusWallMS = lastConvergeMS + } + } else { + result.ConsensusWallMS = time.Since(runStart).Milliseconds() + result.Error = "consensus not reached" + } + + result.ConsensusRound = maxConsensusRound + if result.ConsensusRound == 0 { + result.ConsensusRound = maxRound + } + result.FindingsEmitted = totalEmitted + result.FindingsReceivedTotal = totalApplied + result.LastAgentConvergeMS = lastConvergeMS + result.AvgPropagationMS, result.P95PropagationMS = metrics.AggregatePropagation(propDurations) + // Native group session: findings are broadcast peer-to-peer, no relay. + result.StreamRPCCount = totalEmitted + result.CoordFanoutMS = 0 + return result +} + +func snapshotSlice(m map[int]consensus.AgentSnapshot) []consensus.AgentSnapshot { + out := make([]consensus.AgentSnapshot, 0, len(m)) + for _, snap := range m { + out = append(out, snap) + } + return out +} + +func startAgents(s *scenario.ConsensusScenario, agentBin, endpoint, scenarioFile string) []*exec.Cmd { + var procs []*exec.Cmd + for i, agent := range s.Agents { + cmd := exec.Command( + agentBin, + "--slim-name", agent.SlimName, + "--endpoint", endpoint, + "--scenario-file", scenarioFile, + "--agent-index", fmt.Sprintf("%d", i), + ) + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + if err := cmd.Start(); err != nil { + stopAgents(procs) + log.Fatalf("start agent %s: %v", agent.ID, err) + } + procs = append(procs, cmd) + } + return procs +} + +func stopAgents(procs []*exec.Cmd) { + for _, cmd := range procs { + if cmd.Process != nil { + _ = cmd.Process.Kill() + } + } +} + +func isTimeout(err error) bool { + if err == nil { + return false + } + msg := strings.ToLower(err.Error()) + return strings.Contains(msg, "timed out") || strings.Contains(msg, "timeout") +} diff --git a/benchmarks/slim-vs-a2a-v2/slim/internal/agent/runtime.go b/benchmarks/slim-vs-a2a-v2/slim/internal/agent/runtime.go new file mode 100644 index 00000000..9c9fbb2d --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/slim/internal/agent/runtime.go @@ -0,0 +1,246 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Package agent implements a consensus agent that communicates over a native +// SLIM group session. Unlike the slimrpc-based design, there is NO application +// relay: every agent broadcasts its findings to all peers via PublishAndWait, +// and the SLIM dataplane fans them out. The runner is only a moderator that +// creates the session and a passive observer that consumes snapshots agents +// push to it the moment they converge. +package agent + +import ( + "fmt" + "strings" + "sync" + "sync/atomic" + "time" + + slim "github.com/agntcy/slim-bindings-go" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/slim/internal/protocol" +) + +const defaultSharedSecret = "demo-shared-secret-min-32-chars!!" + +type Runtime struct { + scenario *scenario.ConsensusScenario + agentIndex int + slimName string + endpoint string + + engine *consensus.Engine + app *slim.App + connID uint64 + session *slim.Session + + mu sync.Mutex + runnerCtx *slim.MessageContext + lastPush time.Time + + pubMu sync.Mutex // serializes publishes on the session + + running atomic.Bool + done chan struct{} + startedAt time.Time +} + +func NewRuntime(s *scenario.ConsensusScenario, agentIndex int, slimName, endpoint string) *Runtime { + return &Runtime{ + scenario: s, + agentIndex: agentIndex, + slimName: slimName, + endpoint: endpoint, + engine: consensus.NewEngine(s.Spec, agentIndex), + done: make(chan struct{}), + } +} + +// Setup creates the SLIM app, connects to the dataplane, and subscribes to the +// agent's own name so the moderator can route invitations to it. +func (r *Runtime) Setup() error { + slim.InitializeWithDefaults() + service := slim.GetGlobalService() + + name, err := slim.NameFromString(r.slimName) + if err != nil { + return err + } + + app, err := service.CreateAppWithSecret(name, defaultSharedSecret) + if err != nil { + return err + } + r.app = app + + connID, err := service.Connect(slim.NewInsecureClientConfig(r.endpoint)) + if err != nil { + app.Destroy() + return err + } + r.connID = connID + if err := app.Subscribe(name, &connID); err != nil { + app.Destroy() + return err + } + return nil +} + +// Join blocks until the moderator invites this agent into the group session. +func (r *Runtime) Join(timeout time.Duration) error { + session, err := r.app.ListenForSessionAsync(&timeout) + if err != nil { + return fmt.Errorf("listen for session: %w", err) + } + r.session = session + return nil +} + +// Run is the single receive loop on the group session. It dispatches by +// envelope kind: start launches the think loop, finding is applied locally. +// It blocks until the session ends (the runner kills the process at shutdown). +func (r *Runtime) Run() error { + for { + timeout := time.Second + msg, err := r.session.GetMessageAsync(&timeout) + if err != nil { + if isTimeout(err) { + continue + } + return err + } + env, derr := protocol.Decode(msg.Payload) + if derr != nil { + continue + } + switch env.Kind { + case protocol.KindStart: + ctx := msg.Context + r.mu.Lock() + r.runnerCtx = &ctx + r.mu.Unlock() + r.startRun() + case protocol.KindFinding: + if env.Finding == nil || env.Finding.AgentIndex == r.agentIndex { + continue + } + r.applyFinding(*env.Finding) + // Keep the runner's view fresh if we already converged. + r.maybePushSnapshot(false) + } + } +} + +func (r *Runtime) startRun() { + if r.running.Swap(true) { + return + } + r.startedAt = time.Now() + benchlog.SetRunStart(r.startedAt) + go r.runLoop() +} + +func (r *Runtime) runLoop() { + defer close(r.done) + spec := r.scenario.Spec + think := time.Duration(spec.ThinkTimeMs) * time.Millisecond + emitGap := time.Duration(spec.FindingEmitDelayMs) * time.Millisecond + + for round := 0; round < spec.MaxRounds; round++ { + time.Sleep(think) + + finding, emit := r.engine.Think() + if emit && finding != nil { + if err := r.publishFinding(*finding); err != nil { + benchlog.Finding(benchlog.ImplSLIM, "publish_error", r.agentIndex, finding.FindingID, + fmt.Sprintf("err=%v", err)) + } else { + benchlog.Finding(benchlog.ImplSLIM, "published", r.agentIndex, finding.FindingID) + } + time.Sleep(emitGap) + } + + if r.engine.HasLocalConsensus() { + r.maybePushSnapshot(true) + return + } + } + // Reached max rounds without consensus: still report final state. + r.maybePushSnapshot(true) +} + +// publishFinding broadcasts a finding to every group peer. The SLIM dataplane +// performs the fan-out; there is no relay. +func (r *Runtime) publishFinding(f consensus.Finding) error { + payload, err := protocol.Encode(protocol.Envelope{ + Kind: protocol.KindFinding, + AgentIndex: r.agentIndex, + Finding: &f, + }) + if err != nil { + return err + } + r.pubMu.Lock() + defer r.pubMu.Unlock() + return r.session.PublishAndWaitAsync(payload, nil, nil) +} + +// maybePushSnapshot sends the current snapshot directly to the runner (targeted, +// off the peer broadcast path). force=true always sends; otherwise it is +// throttled so post-convergence finding updates do not spam the runner. +func (r *Runtime) maybePushSnapshot(force bool) { + r.mu.Lock() + ctx := r.runnerCtx + if ctx == nil { + r.mu.Unlock() + return + } + if !force && time.Since(r.lastPush) < 50*time.Millisecond { + r.mu.Unlock() + return + } + r.lastPush = time.Now() + r.mu.Unlock() + + snap := r.engine.Snapshot() + payload, err := protocol.Encode(protocol.Envelope{ + Kind: protocol.KindSnapshot, + AgentIndex: r.agentIndex, + Snapshot: &snap, + }) + if err != nil { + return + } + r.pubMu.Lock() + defer r.pubMu.Unlock() + _ = r.session.PublishToAndWaitAsync(*ctx, payload, nil, nil) +} + +func (r *Runtime) applyFinding(f consensus.Finding) { + recvNs := time.Now().UnixNano() + r.engine.ApplyFinding(f) + if f.EmittedAt > 0 { + r.engine.RecordPropagation(f.EmittedAt, recvNs) + } + benchlog.Finding(benchlog.ImplSLIM, "received", r.agentIndex, f.FindingID, + fmt.Sprintf("from=%d", f.AgentIndex)) +} + +func (r *Runtime) Close() { + if r.session != nil && r.app != nil { + _ = r.app.DeleteSessionAndWaitAsync(r.session) + } + if r.app != nil { + r.app.Destroy() + } +} + +func isTimeout(err error) bool { + if err == nil { + return false + } + msg := strings.ToLower(err.Error()) + return strings.Contains(msg, "timed out") || strings.Contains(msg, "timeout") +} diff --git a/benchmarks/slim-vs-a2a-v2/slim/internal/protocol/protocol.go b/benchmarks/slim-vs-a2a-v2/slim/internal/protocol/protocol.go new file mode 100644 index 00000000..42a48fa4 --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/slim/internal/protocol/protocol.go @@ -0,0 +1,41 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Package protocol defines the message envelope exchanged over the native SLIM +// group session. There is no relay and no request/response RPC: the runner +// broadcasts a single start, agents broadcast findings to all peers, and agents +// push their snapshots to the runner the instant they converge. +package protocol + +import ( + "encoding/json" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" +) + +const ( + // KindStart is broadcast once by the runner to define t0 and unblock agents. + KindStart = "start" + // KindFinding is broadcast by an agent to all group peers. + KindFinding = "finding" + // KindSnapshot is pushed by an agent directly to the runner on convergence. + KindSnapshot = "snapshot" +) + +// Envelope is the single JSON message type carried on the group session. +type Envelope struct { + Kind string `json:"kind"` + AgentIndex int `json:"agentIndex,omitempty"` + Finding *consensus.Finding `json:"finding,omitempty"` + Snapshot *consensus.AgentSnapshot `json:"snapshot,omitempty"` +} + +func Encode(e Envelope) ([]byte, error) { + return json.Marshal(e) +} + +func Decode(data []byte) (Envelope, error) { + var e Envelope + err := json.Unmarshal(data, &e) + return e, err +} diff --git a/benchmarks/slim-vs-a2a-v2/tools/gen_scenario/main.go b/benchmarks/slim-vs-a2a-v2/tools/gen_scenario/main.go new file mode 100644 index 00000000..fbae3a44 --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/tools/gen_scenario/main.go @@ -0,0 +1,45 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "flag" + "fmt" + "log" + "os" + "path/filepath" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" +) + +func main() { + family := flag.String("family", "hypothesis-convergence", "scenario family name") + agents := flag.Int("agents", 10, "number of agents") + thinkMS := flag.Int64("think-ms", 10, "think time per agent in ms") + seed := flag.Int64("seed", 42, "random seed") + output := flag.String("output", "", "output yaml path") + flag.Parse() + + if *output == "" { + log.Fatal("-output is required") + } + + s, err := scenario.Generate(scenario.GenerateOptions{ + Family: *family, + Agents: *agents, + ThinkTimeMs: *thinkMS, + Seed: *seed, + }) + if err != nil { + log.Fatalf("generate: %v", err) + } + + if err := os.MkdirAll(filepath.Dir(*output), 0o755); err != nil { + log.Fatalf("mkdir: %v", err) + } + if err := scenario.WriteFile(*output, s); err != nil { + log.Fatalf("write: %v", err) + } + fmt.Printf("wrote %s\n", *output) +} diff --git a/benchmarks/slim-vs-a2a-v2/tools/report/main.go b/benchmarks/slim-vs-a2a-v2/tools/report/main.go new file mode 100644 index 00000000..7bd51e25 --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/tools/report/main.go @@ -0,0 +1,207 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/csv" + "flag" + "fmt" + "html/template" + "os" + "path/filepath" + "strconv" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/metrics" +) + +type scenarioComparison struct { + ScenarioName string + A2A metrics.RunResult + SLIM metrics.RunResult + HasA2A bool + HasSLIM bool +} + +func main() { + tsvPath := flag.String("tsv", "./reports/results.tsv", "comparison results tsv") + sweepTSV := flag.String("sweep-tsv", "./reports/sweep.tsv", "optional sweep results tsv") + output := flag.String("output", "./reports/index.html", "html dashboard output") + flag.Parse() + + results, err := readTSV(*tsvPath) + if err != nil { + fmt.Fprintf(os.Stderr, "read tsv: %v\n", err) + os.Exit(1) + } + comparisons := groupByScenario(results) + + var sweepResults []metrics.RunResult + if *sweepTSV != "" { + if sr, err := readTSV(*sweepTSV); err == nil { + sweepResults = sr + } + } + + if err := writeHTML(*output, comparisons, sweepResults); err != nil { + fmt.Fprintf(os.Stderr, "write html: %v\n", err) + os.Exit(1) + } + fmt.Printf("wrote %s\n", *output) +} + +func readTSV(path string) ([]metrics.RunResult, error) { + file, err := os.Open(path) + if err != nil { + return nil, err + } + defer file.Close() + + reader := csv.NewReader(file) + reader.Comma = '\t' + records, err := reader.ReadAll() + if err != nil { + return nil, err + } + if len(records) <= 1 { + return nil, fmt.Errorf("no data rows in %s", path) + } + + var out []metrics.RunResult + for _, row := range records[1:] { + if len(row) < 16 { + continue + } + r := metrics.RunResult{ + ScenarioName: row[0], + Domain: row[1], + Implementation: row[2], + Error: row[len(row)-1], + } + r.Agents = atoi(row[3]) + r.ThinkTimeMs = atoi64(row[4]) + r.ConsensusWallMS = atoi64(row[5]) + r.ConsensusRound = atoi(row[6]) + r.FindingsEmitted = atoi(row[7]) + r.FindingsReceivedTotal = atoi(row[8]) + r.AvgPropagationMS = atoi64(row[9]) + r.P95PropagationMS = atoi64(row[10]) + r.LastAgentConvergeMS = atoi64(row[11]) + r.CoordFanoutMS = atoi64(row[12]) + r.StreamRPCCount = atoi(row[13]) + r.UnicastRPCCount = atoi(row[14]) + r.Success = row[15] == "true" + if len(row) > 16 { + r.Error = row[16] + } + out = append(out, r) + } + return out, nil +} + +func groupByScenario(results []metrics.RunResult) []scenarioComparison { + byScenario := map[string]*scenarioComparison{} + for _, r := range results { + sc, ok := byScenario[r.ScenarioName] + if !ok { + sc = &scenarioComparison{ScenarioName: r.ScenarioName} + byScenario[r.ScenarioName] = sc + } + switch r.Implementation { + case "a2a-relay-stream": + sc.A2A = r + sc.HasA2A = true + case "slim-group-session": + sc.SLIM = r + sc.HasSLIM = true + } + } + out := make([]scenarioComparison, 0, len(byScenario)) + for _, sc := range byScenario { + out = append(out, *sc) + } + return out +} + +type reportData struct { + Comparisons []scenarioComparison + Sweep []metrics.RunResult +} + +func writeHTML(path string, comparisons []scenarioComparison, sweep []metrics.RunResult) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + tmpl := template.Must(template.New("report").Funcs(template.FuncMap{ + "deltaPct": deltaPct, + }).Parse(htmlTemplate)) + file, err := os.Create(path) + if err != nil { + return err + } + defer file.Close() + return tmpl.Execute(file, reportData{Comparisons: comparisons, Sweep: sweep}) +} + +func deltaPct(a2a, slim int64) string { + if a2a == 0 { + return "n/a" + } + pct := (float64(a2a-slim) / float64(a2a)) * 100 + return fmt.Sprintf("%.1f%%", pct) +} + +func atoi(v string) int { + n, _ := strconv.Atoi(v) + return n +} + +func atoi64(v string) int64 { + n, _ := strconv.ParseInt(v, 10, 64) + return n +} + +const htmlTemplate = ` + + + + SLIM vs A2A v2 Consensus + + + +

SLIM vs A2A v2 — Consensus Streaming

+

Delta columns show (A2A − SLIM) / A2A. Positive values mean SLIM reached consensus faster.

+ {{range .Comparisons}} +

{{.ScenarioName}}

+ + + + + + + + + +
MetricA2ASLIMDelta
Consensus wall (ms){{if .HasA2A}}{{.A2A.ConsensusWallMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.ConsensusWallMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.ConsensusWallMS .SLIM.ConsensusWallMS}}{{else}}—{{end}}
Last agent converge (ms){{if .HasA2A}}{{.A2A.LastAgentConvergeMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.LastAgentConvergeMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.LastAgentConvergeMS .SLIM.LastAgentConvergeMS}}{{else}}—{{end}}
Avg propagation (ms){{if .HasA2A}}{{.A2A.AvgPropagationMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.AvgPropagationMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.AvgPropagationMS .SLIM.AvgPropagationMS}}{{else}}—{{end}}
P95 propagation (ms){{if .HasA2A}}{{.A2A.P95PropagationMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.P95PropagationMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.P95PropagationMS .SLIM.P95PropagationMS}}{{else}}—{{end}}
Stream RPC count{{if .HasA2A}}{{.A2A.StreamRPCCount}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.StreamRPCCount}}{{else}}—{{end}}—
Unicast RPC count{{if .HasA2A}}{{.A2A.UnicastRPCCount}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.UnicastRPCCount}}{{else}}—{{end}}—
Success{{if .HasA2A}}{{.A2A.Success}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.Success}}{{else}}—{{end}}—
+ {{end}} + {{if .Sweep}} +

Sweep results

+ + + {{range .Sweep}} + + + + + {{end}} +
ScenarioImplAgentsThink msConsensus wallP95 propagationStream RPCs
{{.ScenarioName}}{{.Implementation}}{{.Agents}}{{.ThinkTimeMs}}{{.ConsensusWallMS}}{{.P95PropagationMS}}{{.StreamRPCCount}}
+ {{end}} + + +` diff --git a/benchmarks/slim-vs-a2a-v2/tools/validate_scenarios/main.go b/benchmarks/slim-vs-a2a-v2/tools/validate_scenarios/main.go new file mode 100644 index 00000000..c32898f8 --- /dev/null +++ b/benchmarks/slim-vs-a2a-v2/tools/validate_scenarios/main.go @@ -0,0 +1,41 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "flag" + "fmt" + "log" + "os" + "path/filepath" + "strings" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" +) + +func main() { + dir := flag.String("dir", "./plans/sweeps", "directory containing scenario yaml files") + flag.Parse() + + entries, err := os.ReadDir(*dir) + if err != nil { + log.Fatalf("read dir: %v", err) + } + var failed int + for _, e := range entries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".yaml") { + continue + } + path := filepath.Join(*dir, e.Name()) + if _, err := scenario.LoadFile(path); err != nil { + fmt.Printf("INVALID %s: %v\n", path, err) + failed++ + } else { + fmt.Printf("OK %s\n", path) + } + } + if failed > 0 { + os.Exit(1) + } +} From e09b069246f1bfce5893780edde12c14f9b3d81d Mon Sep 17 00:00:00 2001 From: Magyari Sandor Szilard Date: Mon, 29 Jun 2026 11:04:12 +0200 Subject: [PATCH 4/6] feat: decrease time window and increase number of agents Signed-off-by: Magyari Sandor Szilard --- .../slim-vs-a2a-v2_consensus_37de84b5.plan.md | 332 ------------ .../slim_native_vs_a2a_relay_b1828419.plan.md | 122 +++++ ...slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md | 500 ------------------ .github/workflows/test-slim-vs-a2a-v2.yaml | 22 - benchmarks/Taskfile.yml | 5 - benchmarks/slim-vs-a2a-v2/Taskfile.yml | 348 ------------ .../slim-vs-a2a-v2/a2a/cmd/agent/main.go | 161 ------ .../slim-vs-a2a-v2/a2a/cmd/runner/main.go | 281 ---------- .../a2a/internal/client/client.go | 160 ------ .../a2a/internal/protocol/protocol.go | 59 --- benchmarks/slim-vs-a2a-v2/go.mod | 25 - benchmarks/slim-vs-a2a-v2/go.sum | 62 --- .../internal/benchlog/benchlog.go | 89 ---- .../internal/metrics/metrics.go | 120 ----- benchmarks/slim-vs-a2a-v2/plans/README.md | 67 --- .../slim-vs-a2a-v2/slim/cmd/agent/main.go | 51 -- .../slim-vs-a2a-v2/slim/cmd/runner/main.go | 334 ------------ .../slim/internal/protocol/protocol.go | 41 -- .../slim-vs-a2a-v2/tools/report/main.go | 207 -------- benchmarks/slim-vs-a2a/Taskfile.yml | 267 +++++----- benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go | 82 +-- benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go | 208 +++++++- .../a2a/internal/agent/runtime.go | 8 +- .../slim-vs-a2a/a2a/internal/client/client.go | 257 +++------ .../a2a/internal/executor/executor.go | 97 ---- .../a2a/internal/protocol/protocol.go | 57 +- .../a2a/internal/relay/relay.go | 6 +- .../a2a/internal/scheduler/scheduler.go | 342 ------------ benchmarks/slim-vs-a2a/go.mod | 12 +- benchmarks/slim-vs-a2a/go.sum | 45 +- .../slim-vs-a2a/internal/benchlog/benchlog.go | 47 +- .../internal/consensus/consensus.go | 49 +- .../internal/metrics/coordstats.go | 95 ---- .../internal/metrics/coordstats_test.go | 39 -- .../slim-vs-a2a/internal/metrics/metrics.go | 127 ++--- .../slim-vs-a2a/internal/plan/generator.go | 231 -------- .../internal/plan/generator_test.go | 46 -- benchmarks/slim-vs-a2a/internal/plan/plan.go | 295 ----------- .../internal/scenario/scenario.go | 22 +- .../internal/scenario/scenario_test.go | 2 +- benchmarks/slim-vs-a2a/plans/README.md | 117 ++-- .../plans/domains/k8s-incident-response.yaml | 173 ------ .../plans/domains/mobile-nav-assistant.yaml | 140 ----- .../plans/domains/urban-traffic-reroute.yaml | 156 ------ .../hypothesis-convergence-10ag-20ms.yaml | 0 ...ypothesis-convergence-10ag-5ms-10240b.yaml | 56 ++ .../hypothesis-convergence-10ag-5ms.yaml | 55 ++ .../hypothesis-convergence-17ag-20ms.yaml | 0 .../hypothesis-convergence-20ag-20ms.yaml | 0 .../hypothesis-convergence-50ag-20ms.yaml | 0 ...ypothesis-convergence-50ag-5ms-10240b.yaml | 216 ++++++++ .../hypothesis-convergence-50ag-5ms.yaml | 215 ++++++++ .../hypothesis-convergence-5ag-20ms.yaml | 0 .../sustainable-resource-10ag-10ms.yaml | 152 ------ .../sustainable-resource-10ag-20ms.yaml | 152 ------ .../sustainable-resource-17ag-10ms.yaml | 243 --------- .../sustainable-resource-17ag-20ms.yaml | 243 --------- .../sweeps/sustainable-resource-2ag-10ms.yaml | 48 -- .../sweeps/sustainable-resource-2ag-20ms.yaml | 48 -- .../sweeps/sustainable-resource-5ag-10ms.yaml | 87 --- .../sweeps/sustainable-resource-5ag-20ms.yaml | 87 --- benchmarks/slim-vs-a2a/slim/cmd/agent/main.go | 59 +-- .../slim-vs-a2a/slim/cmd/runner/main.go | 289 ++++++++-- .../slim/internal/agent/runtime.go | 8 +- .../slim/internal/client/client.go | 462 ---------------- .../slim/internal/executor/executor.go | 100 ---- .../slim/internal/protocol/protocol.go | 111 +--- .../slim/internal/scheduler/scheduler.go | 341 ------------ .../slim/internal/server/handler.go | 25 - .../slim/internal/slimrpc/slimrpc.go | 9 - .../slim-vs-a2a/tests/slim_vs_a2a_test.go | 299 ----------- benchmarks/slim-vs-a2a/tests/suite_test.go | 16 - benchmarks/slim-vs-a2a/tools/gen_plan/main.go | 52 -- .../tools/gen_scenario/main.go | 12 +- benchmarks/slim-vs-a2a/tools/report/main.go | 239 +++++---- .../slim-vs-a2a/tools/validate_plans/main.go | 28 - .../tools/validate_scenarios/main.go | 2 +- 77 files changed, 1784 insertions(+), 7776 deletions(-) delete mode 100644 .cursor/plans/slim-vs-a2a-v2_consensus_37de84b5.plan.md create mode 100644 .cursor/plans/slim_native_vs_a2a_relay_b1828419.plan.md delete mode 100644 .cursor/plans/slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md delete mode 100644 .github/workflows/test-slim-vs-a2a-v2.yaml delete mode 100644 benchmarks/slim-vs-a2a-v2/Taskfile.yml delete mode 100644 benchmarks/slim-vs-a2a-v2/a2a/cmd/agent/main.go delete mode 100644 benchmarks/slim-vs-a2a-v2/a2a/cmd/runner/main.go delete mode 100644 benchmarks/slim-vs-a2a-v2/a2a/internal/client/client.go delete mode 100644 benchmarks/slim-vs-a2a-v2/a2a/internal/protocol/protocol.go delete mode 100644 benchmarks/slim-vs-a2a-v2/go.mod delete mode 100644 benchmarks/slim-vs-a2a-v2/go.sum delete mode 100644 benchmarks/slim-vs-a2a-v2/internal/benchlog/benchlog.go delete mode 100644 benchmarks/slim-vs-a2a-v2/internal/metrics/metrics.go delete mode 100644 benchmarks/slim-vs-a2a-v2/plans/README.md delete mode 100644 benchmarks/slim-vs-a2a-v2/slim/cmd/agent/main.go delete mode 100644 benchmarks/slim-vs-a2a-v2/slim/cmd/runner/main.go delete mode 100644 benchmarks/slim-vs-a2a-v2/slim/internal/protocol/protocol.go delete mode 100644 benchmarks/slim-vs-a2a-v2/tools/report/main.go rename benchmarks/{slim-vs-a2a-v2 => slim-vs-a2a}/a2a/internal/agent/runtime.go (96%) delete mode 100644 benchmarks/slim-vs-a2a/a2a/internal/executor/executor.go rename benchmarks/{slim-vs-a2a-v2 => slim-vs-a2a}/a2a/internal/relay/relay.go (97%) delete mode 100644 benchmarks/slim-vs-a2a/a2a/internal/scheduler/scheduler.go rename benchmarks/{slim-vs-a2a-v2 => slim-vs-a2a}/internal/consensus/consensus.go (82%) delete mode 100644 benchmarks/slim-vs-a2a/internal/metrics/coordstats.go delete mode 100644 benchmarks/slim-vs-a2a/internal/metrics/coordstats_test.go delete mode 100644 benchmarks/slim-vs-a2a/internal/plan/generator.go delete mode 100644 benchmarks/slim-vs-a2a/internal/plan/generator_test.go delete mode 100644 benchmarks/slim-vs-a2a/internal/plan/plan.go rename benchmarks/{slim-vs-a2a-v2 => slim-vs-a2a}/internal/scenario/scenario.go (91%) rename benchmarks/{slim-vs-a2a-v2 => slim-vs-a2a}/internal/scenario/scenario_test.go (87%) delete mode 100644 benchmarks/slim-vs-a2a/plans/domains/k8s-incident-response.yaml delete mode 100644 benchmarks/slim-vs-a2a/plans/domains/mobile-nav-assistant.yaml delete mode 100644 benchmarks/slim-vs-a2a/plans/domains/urban-traffic-reroute.yaml rename benchmarks/{slim-vs-a2a-v2 => slim-vs-a2a}/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml (100%) create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms-10240b.yaml create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms.yaml rename benchmarks/{slim-vs-a2a-v2 => slim-vs-a2a}/plans/sweeps/hypothesis-convergence-17ag-20ms.yaml (100%) rename benchmarks/{slim-vs-a2a-v2 => slim-vs-a2a}/plans/sweeps/hypothesis-convergence-20ag-20ms.yaml (100%) rename benchmarks/{slim-vs-a2a-v2 => slim-vs-a2a}/plans/sweeps/hypothesis-convergence-50ag-20ms.yaml (100%) create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms-10240b.yaml create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms.yaml rename benchmarks/{slim-vs-a2a-v2 => slim-vs-a2a}/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml (100%) delete mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-10ms.yaml delete mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-20ms.yaml delete mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-10ms.yaml delete mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-20ms.yaml delete mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-10ms.yaml delete mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-20ms.yaml delete mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-10ms.yaml delete mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-20ms.yaml rename benchmarks/{slim-vs-a2a-v2 => slim-vs-a2a}/slim/internal/agent/runtime.go (95%) delete mode 100644 benchmarks/slim-vs-a2a/slim/internal/client/client.go delete mode 100644 benchmarks/slim-vs-a2a/slim/internal/executor/executor.go delete mode 100644 benchmarks/slim-vs-a2a/slim/internal/scheduler/scheduler.go delete mode 100644 benchmarks/slim-vs-a2a/slim/internal/server/handler.go delete mode 100644 benchmarks/slim-vs-a2a/slim/internal/slimrpc/slimrpc.go delete mode 100644 benchmarks/slim-vs-a2a/tests/slim_vs_a2a_test.go delete mode 100644 benchmarks/slim-vs-a2a/tests/suite_test.go delete mode 100644 benchmarks/slim-vs-a2a/tools/gen_plan/main.go rename benchmarks/{slim-vs-a2a-v2 => slim-vs-a2a}/tools/gen_scenario/main.go (75%) delete mode 100644 benchmarks/slim-vs-a2a/tools/validate_plans/main.go rename benchmarks/{slim-vs-a2a-v2 => slim-vs-a2a}/tools/validate_scenarios/main.go (91%) diff --git a/.cursor/plans/slim-vs-a2a-v2_consensus_37de84b5.plan.md b/.cursor/plans/slim-vs-a2a-v2_consensus_37de84b5.plan.md deleted file mode 100644 index 50a05836..00000000 --- a/.cursor/plans/slim-vs-a2a-v2_consensus_37de84b5.plan.md +++ /dev/null @@ -1,332 +0,0 @@ ---- -name: slim-vs-a2a-v2 consensus -overview: Create a new `benchmarks/slim-vs-a2a-v2` benchmark that compares pure Go A2A coordinator fan-out vs SLIM group multicast streaming on an identical parallel consensus problem, with configurable agent counts, TSV metrics collection, and HTML reporting mirroring v1. -todos: - - id: phase-0-spike - content: Spike SLIM v1.4.0 CallMulticastStream and a2a-go v2.3.0 streaming APIs; prototype propagation latency measurement - status: completed - - id: phase-1-shared - content: Implement internal/scenario, internal/consensus, internal/metrics, tools/gen_scenario - status: completed - - id: phase-2-slim - content: Implement slim-agent, slim-runner, slim/internal/client with group multicast streaming - status: completed - - id: phase-3-a2a - content: Implement a2a-agent (coordinator/worker), a2a-runner, a2a/internal/coordinator with parallel fan-out - status: completed - - id: phase-4-harness - content: Add Taskfile.yml, tools/report HTML, register in benchmarks/Taskfile.yml, CI smoke workflow - status: completed - - id: phase-5-validate - content: Run sweeps, verify SLIM consensus_wall_ms beats A2A at N>=10, document results in plans/README.md - status: in_progress -isProject: false ---- - -# SLIM vs A2A v2 — Consensus Streaming Benchmark - -## Goals - -- **Fair comparison:** identical consensus logic, agent count, think-time, and termination rules for both implementations -- **Highlight SLIM:** group multicast streaming delivers findings to all agents faster than A2A coordinator fan-out -- **Configurable scale:** sweep `agents` (e.g. 2, 5, 10, 17, 32) and coordination pressure (think time, finding rate) -- **Same reporting UX as v1:** Taskfile tasks → append TSV → HTML dashboard -- **Isolation:** new module at [`benchmarks/slim-vs-a2a-v2/`](benchmarks/slim-vs-a2a-v2/); v1 ([`benchmarks/slim-vs-a2a/`](benchmarks/slim-vs-a2a/)) unchanged - -**Primary success metric:** `consensus_wall_ms` — time from run start until all agents report the same agreed value. SLIM should win increasingly as `agents` grows. - ---- - -## Consensus Problem (shared by both implementations) - -### Scenario: Distributed Hypothesis Convergence - -N agents independently evaluate shards of evidence (simulated CPU work). Each agent maintains a local hypothesis `(value, confidence)`. Agents emit **findings** whenever local state changes. The run succeeds when **every agent holds the identical hypothesis**. - -### Simulation rules (deterministic, no LLM) - -- Each agent starts with `value = i % K`, `confidence = 0.1` -- Each think cycle (sleep `think_time_ms`): recompute locally using agent_id + round + received findings; emit if changed -- On receiving a peer finding: merge via deterministic merge function; may re-think and re-emit -- Consensus when all agents hold the same `value` (target derived deterministically from N via `targetMode: majority`) -- Terminate on consensus or `max_rounds` exceeded (failure) - -**Why this fits transport benchmarking:** agents run in parallel and emit findings continuously while working. Transport latency directly affects convergence speed — unlike v1's DAG where context updates happen only at task boundaries. - -### Scenario config (`ConsensusScenario` YAML) - -```yaml -apiVersion: bench.agntcy.io/v2 -kind: ConsensusScenario -metadata: - name: hypothesis-convergence-10agents -spec: - agents: 10 - thinkTimeMs: 50 - findingEmitDelayMs: 5 - maxRounds: 100 - targetMode: majority - seed: 42 -``` - -Generated sweep files: `plans/sweeps/hypothesis-{N}ag-{think}ms.yaml` via `tools/gen_scenario`. - ---- - -## Transport Architecture - -```mermaid -flowchart TB - subgraph shared [Shared] - Scenario[ConsensusScenario YAML] - Sim[internal/consensus simulator] - Scenario --> Sim - end - - subgraph a2a [A2A hub coordinator] - Coord[coordinator agent-0] - W1[worker agent-1] - W2[worker agent-2] - WN[worker agent-N] - W1 -->|finding RPC| Coord - W2 -->|finding RPC| Coord - WN -->|finding RPC| Coord - Coord -->|parallel fan-out| W1 - Coord -->|parallel fan-out| W2 - Coord -->|parallel fan-out| WN - end - - subgraph slim [SLIM group multicast stream] - Group[SLIM group channel] - SA[agent-0] - SB[agent-1] - SC[agent-N] - SA -->|multicast stream| Group - SB -->|multicast stream| Group - Group -->|immediate delivery| SA - Group -->|immediate delivery| SB - Group -->|immediate delivery| SC - end -``` - -### A2A (`a2a-coordinator-stream`) - -| Aspect | Design | -|--------|--------| -| Topology | Hub-and-spoke: `agent-0` is coordinator; workers `agent-1..N-1` | -| Coordinator | Receives findings, deduplicates, fans out to all other agents in parallel goroutines | -| Streaming | Enable A2A agent card `Streaming: true`; use a2a-go streaming/subscribe APIs (Phase 0 spike) | -| Fallback | Parallel unary `SendMessage` per target if streaming unavailable | -| Reference | Current sequential fan-out in [`a2a/internal/client/client.go`](benchmarks/slim-vs-a2a/a2a/internal/client/client.go) `PushContext` | - -### SLIM (`slim-group-multicast-stream`) - -| Aspect | Design | -|--------|--------| -| Topology | Single SLIM group — all N agents are peers (no coordinator) | -| Finding publish | `group.CallMulticastStream(...)` — one publish, all members receive on open stream | -| Setup | Reuse group pattern from [`slim/internal/client/client.go`](benchmarks/slim-vs-a2a/slim/internal/client/client.go) `subsetGroup` + `SetupGroup` | -| v2 upgrade over v1 | v1 uses `CallMulticastUnary`; v2 uses **multicast streaming** for continuous finding propagation | - ---- - -## Package Layout - -``` -benchmarks/slim-vs-a2a-v2/ -├── go.mod -├── Taskfile.yml -├── plans/ -│ ├── README.md -│ └── sweeps/ -├── internal/ -│ ├── scenario/ # YAML load, validate, GenerateOptions -│ ├── consensus/ # shared simulator: merge, target, termination -│ ├── metrics/ # RunResult + consensus fields + TSV -│ └── benchlog/ # adapt from v1 -├── a2a/ -│ ├── cmd/agent/main.go -│ ├── cmd/runner/main.go -│ └── internal/ -│ ├── client/client.go -│ ├── coordinator/coordinator.go -│ └── protocol/protocol.go -├── slim/ -│ ├── cmd/agent/main.go -│ ├── cmd/runner/main.go -│ └── internal/ -│ ├── client/client.go -│ └── protocol/protocol.go -├── tools/ -│ ├── gen_scenario/main.go -│ └── report/main.go -├── reports/ # gitignored -└── tests/ - ├── suite_test.go - └── consensus_test.go -``` - -Register in [`benchmarks/Taskfile.yml`](benchmarks/Taskfile.yml): - -```yaml - slim-vs-a2a-v2: - taskfile: ./slim-vs-a2a-v2/Taskfile.yml - dir: ./slim-vs-a2a-v2 - excludes: [default] -``` - -**Reuse from v1 (copy, not cross-module import):** `benchlog`, SLIM stack startup Taskfile patterns, `slimctl` download, agent spawning, TSV append pattern from [`internal/metrics/metrics.go`](benchmarks/slim-vs-a2a/internal/metrics/metrics.go). - -### Pinned dependencies (same as v1) - -| Dependency | Version | Notes | -|------------|---------|-------| -| `github.com/agntcy/slim-bindings-go` | **v1.4.0** | Multicast streaming API spike targets this release | -| `slimctl` | **slimctl-v1.4.0** | Local dataplane for SLIM runs | -| `github.com/a2aproject/a2a-go/v2` | **v2.3.0** | A2A coordinator streaming | - -Taskfile vars (mirror [`benchmarks/slim-vs-a2a/Taskfile.yml`](benchmarks/slim-vs-a2a/Taskfile.yml)): - -```yaml -COMPARE_SLIM_BINDINGS_VERSION: '{{ .COMPARE_SLIM_BINDINGS_VERSION | default "v1.4.0" }}' -COMPARE_SLIMCTL_TAG: '{{ .COMPARE_SLIMCTL_TAG | default "slimctl-v1.4.0" }}' -``` - -`go.mod` require line: `github.com/agntcy/slim-bindings-go v1.4.0` - ---- - -## Metrics Schema - -Extend v1 `RunResult` with consensus-specific fields: - -| Field | Description | -|-------|-------------| -| `consensus_wall_ms` | **Primary:** start → all agents agree | -| `consensus_round` | Round number when consensus reached | -| `findings_emitted` | Total findings published | -| `findings_received_total` | Sum across agents | -| `avg_propagation_ms` | Mean time from emit → all others received | -| `p95_propagation_ms` | Tail latency of finding delivery | -| `last_agent_converge_ms` | Straggler: slowest agent to reach consensus | -| `coord_fanout_ms` | Time in coordination transport | -| `stream_rpc_count` | Streaming/multicast ops | -| `unicast_rpc_count` | Point-to-point ops | - -TSV columns (tab-separated, append mode): - -``` -scenario_name domain implementation agents consensus_wall_ms consensus_round -findings_emitted avg_propagation_ms p95_propagation_ms last_agent_converge_ms -coord_fanout_ms stream_rpc_count unicast_rpc_count success error -``` - ---- - -## Taskfile Tasks - -Mirror [`benchmarks/slim-vs-a2a/Taskfile.yml`](benchmarks/slim-vs-a2a/Taskfile.yml): - -| Task | Purpose | -|------|---------| -| `build` | Build `a2a-agent`, `a2a-runner`, `slim-agent`, `slim-runner` | -| `deps:slim-bindings-setup` | CGO bindings | -| `deps:slimctl-download` | Local SLIM dataplane | -| `validate:scenarios` | Validate YAML scenarios | -| `compare:cleanup` | Kill stray agent processes | -| `compare:suite` | All default scenarios: A2A then SLIM → `reports/results.tsv` | -| `compare:plan` | Single scenario: `PLAN=hypothesis-convergence-10agents` | -| `compare:sweep` | Agent/think-time sweep → `reports/sweep.tsv` | -| `compare:report` | `go run ./tools/report --tsv reports/results.tsv --sweep-tsv reports/sweep.tsv -o reports/index.html` | -| `compare:ci:smoke` | Small sweep + report (~2 min) | -| `test` | Ginkgo suite with `RUN_SLIM_VS_A2A_V2=1` | - -Example usage: - -```bash -cd benchmarks/slim-vs-a2a-v2 -task compare:plan PLAN=hypothesis-convergence-10agents -task compare:sweep SWEEP_AGENTS=2,5,10,17 SWEEP_THINK_MS=10,20,50 -task compare:report -``` - -Sweep env vars: `SWEEP_AGENTS` (default `2,5,10,17`), `SWEEP_THINK_MS` (default `10,20,50`), `SWEEP_FAMILY` (default `hypothesis-convergence`). - ---- - -## HTML Report - -Adapt [`tools/report/main.go`](benchmarks/slim-vs-a2a/tools/report/main.go): - -- Per-scenario comparison table: consensus wall ms, last agent converge, avg/p95 propagation, stream RPC count -- Delta columns: `(A2A − SLIM) / A2A` — positive = SLIM faster -- Sweep section: table with agents, think ms, implementation, consensus metrics - ---- - -## Implementation Phases - -### Phase 0 — API spike -- Confirm `slim-bindings-go` **v1.4.0** multicast streaming API (`CallMulticastStream` or equivalent) -- Confirm a2a-go **v2.3.0** streaming/subscribe for coordinator push -- Prototype finding propagation latency measurement - -### Phase 1 — Shared core -- `internal/scenario` — YAML schema + validator -- `internal/consensus` — deterministic simulator -- `internal/metrics` — RunResult + TSV append -- `tools/gen_scenario` — sweep YAML generator - -### Phase 2 — SLIM path -- `slim-agent` — group member, inbound stream handler, think loop -- `slim-runner` — start agents, create group, run until consensus -- `slim/internal/client` — `PublishFindingStream`, propagation timestamps - -### Phase 3 — A2A path -- `a2a-agent` — coordinator/worker roles + streaming handler -- `a2a-runner` — spawn N agents, designate agent-0 as coordinator -- `a2a/internal/coordinator` — receive finding, parallel fan-out - -### Phase 4 — Harness and CI -- Taskfile (all tasks above) -- `tools/report` HTML dashboard -- Optional `.github/workflows/test-slim-vs-a2a-v2.yaml` -- `plans/README.md` - -### Phase 5 — Validation -- Verify SLIM `consensus_wall_ms` < A2A at N >= 10 -- Confirm determinism with fixed `seed` - ---- - -## Key Design Decisions - -| Decision | Choice | Rationale | -|----------|--------|-----------| -| Separate folder | `slim-vs-a2a-v2` | Different workload (consensus vs DAG); v1 CI unchanged | -| A2A coordinator | `agent-0` is coordinator | Models realistic hub pattern; SLIM has no hub | -| Shared simulator | `internal/consensus` | Same merge/terminate logic; only I/O differs | -| SLIM transport | Multicast stream (not unary) | v2 differentiator vs v1 | -| SLIM bindings version | **slim-bindings-go v1.4.0** | Same pin as v1; multicast streaming API validated against this release | -| Agent count fairness | N total processes: A2A = 1 coord + N-1 workers; SLIM = N peers | Document in README | - ---- - -## Relationship to v1 - -| | slim-vs-a2a (v1) | slim-vs-a2a-v2 | -|--|-----------------|----------------| -| Workload | YAML DAG task execution | Continuous consensus loop | -| Coordination trigger | Task completion / contextUpdates | Finding emission during parallel work | -| SLIM transport | Multicast unary | Multicast streaming | -| A2A transport | Sequential unicast fan-out | Coordinator parallel stream fan-out | -| Primary metric | `total_wall_clock_ms`, `context_push_ms` | `consensus_wall_ms`, `avg_propagation_ms` | -| Plan kind | `ExecutionPlan` | `ConsensusScenario` | - ---- - -## Open Questions (Phase 0) - -1. ~~SLIM bindings version~~ **Resolved:** use `slim-bindings-go v1.4.0` and `slimctl-v1.4.0` (same as v1). Phase 0 confirms the exact multicast streaming method name in this release. -2. A2A streaming — a2a-go v2.3.0 server-push suitability vs parallel unary fallback -3. Coordinator fairness — N-1 workers + 1 coordinator vs external runner-level coordinator diff --git a/.cursor/plans/slim_native_vs_a2a_relay_b1828419.plan.md b/.cursor/plans/slim_native_vs_a2a_relay_b1828419.plan.md new file mode 100644 index 00000000..00322177 --- /dev/null +++ b/.cursor/plans/slim_native_vs_a2a_relay_b1828419.plan.md @@ -0,0 +1,122 @@ +--- +name: slim native vs a2a relay +overview: Rewrite the slim-vs-a2a benchmark so the SLIM side uses native SLIM group sessions (peer-to-peer multicast, no relay) and the A2A side uses real A2A server-streaming through the runner acting as the relay hub, to demonstrate that native SLIM needs no relay and yields better convergence metrics at scale. +todos: + - id: slim-protocol + content: Extend slim/internal/protocol with JSON Envelope (finding/start/snapshot, push-based); remove slim/internal/slimrpc and unused slim/internal/client + status: completed + - id: slim-agent + content: Rewrite slim agent runtime + cmd/agent/main.go to join the moderator's group session, run a single GetMessage receive loop, broadcast findings via PublishAndWaitAsync, and push a snapshot to the runner via PublishToAndWaitAsync immediately on local consensus (and on change) + status: completed + - id: slim-runner + content: "Rewrite slim cmd/runner/main.go as passive observer/moderator: create group session, invite agents, broadcast a single start, then event-driven receive loop consuming pushed snapshots (no polling), detect GlobalConsensus, aggregate (StreamRPCCount=findings, CoordFanoutMS=0); remove relay hub usage" + status: completed + - id: a2a-protocol + content: Update a2a/internal/protocol with OpStreamFindings/OpStreamRelay stream-init ops; keep unary OpStart/OpSnapshot; drop OpFinding/OpFanout + status: completed + - id: a2a-agent + content: "Rewrite a2a agent runtime + cmd/agent/main.go: host OpStreamFindings server stream + unary ops, subscribe to runner relay stream, push findings to outbound channel; remove coordinator/fanout" + status: completed + - id: a2a-runner + content: "Rewrite a2a cmd/runner/main.go + internal/client as relay: host OpStreamRelay server, subscribe to each agent's findings, relay+count (StreamRPCCount/CoordFanoutMS), drive start/snapshot, pass relay card URL to agents" + status: completed + - id: labels-report + content: Update benchlog impl tags (a2a-relay-stream, slim-group-session) and tools/report matching strings; note relay-free SLIM in plans/README + status: completed + - id: validate + content: task build; task compare:ci:smoke; run a 20/50-agent sweep point and confirm report renders and consensus_wall_ms delta favors SLIM + status: completed +isProject: false +--- + +## SLIM native sessions vs A2A streaming relay + +Goal: in-place rewrite of both transports in `benchmarks/slim-vs-a2a` to prove the thesis: native SLIM group sessions give true many-to-many with no relay; A2A requires a relay hub (the runner) even with real streaming; at N agents under a deadline, SLIM converges faster. The shared scenario/consensus/metrics/report core is reused; only the two transports and metric labels change. + +### Target topologies + +```mermaid +flowchart TB + subgraph slim [SLIM native group session - no relay] + R1["runner (passive observer: start signal + consumes pushed metrics)"] + SA0["agent-0"] + SA1["agent-1"] + SA2["agent-2"] + SA0 ---|"Publish finding -> all (dataplane fan-out)"| SA1 + SA1 --- SA2 + SA0 --- SA2 + SA0 -.->|"push snapshot on convergence"| R1 + SA1 -.-> R1 + SA2 -.-> R1 + end + subgraph a2a [A2A - runner is the relay hub] + R2["runner = RELAY (hosts A2A server)"] + AA0["agent-0"] + AA1["agent-1"] + AA2["agent-2"] + AA0 -->|"stream findings -> runner"| R2 + AA1 --> R2 + AA2 --> R2 + R2 -->|"stream relayed findings -> agent"| AA0 + R2 --> AA1 + R2 --> AA2 + end +``` + +### 1. SLIM side -> native group sessions (no relay) + +Delete the slimrpc relay path and rebuild on native sessions (APIs confirmed present in `slim-bindings-go v1.4.0`: `CreateSessionAndWaitAsync` + `SessionTypeGroup`, `InviteAndWaitAsync`, `ListenForSessionAsync`, `PublishAndWaitAsync`, `PublishToAndWaitAsync`, `GetMessageAsync`, `ParticipantsList`). Pattern reference: `/Users/smagyari/dev/slim/data-plane/bindings/go/examples/group/main.go`. + +- Remove `[slim/internal/relay/hub.go](benchmarks/slim-vs-a2a/slim/internal/relay/hub.go)`, `[slim/internal/slimrpc/slimrpc.go](benchmarks/slim-vs-a2a/slim/internal/slimrpc/slimrpc.go)`, and the unused `[slim/internal/client/client.go](benchmarks/slim-vs-a2a/slim/internal/client/client.go)`. +- Extend `[slim/internal/protocol/protocol.go](benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go)` with a single JSON envelope sent over the session: + - `Envelope{ Kind string; Finding *consensus.Finding; Snapshot *consensus.AgentSnapshot; AgentIndex int }` with `Kind` in `{"finding","start","snapshot"}`. No request/reply kinds — snapshots are pushed, not polled. +- Rewrite `[slim/internal/agent/runtime.go](benchmarks/slim-vs-a2a/slim/internal/agent/runtime.go)` (member): + - Setup: create app `agntcy/bench-v2/agent-N`, connect, subscribe, then `ListenForSessionAsync(timeout)` to join the moderator's group session; keep `*slim.Session`. + - Single receive loop on `session.GetMessageAsync`: dispatch by `Kind`: + - `finding` -> `applyFinding` (skip own `AgentIndex`, `RecordPropagation`). + - `start` -> save the message's `Context` (identifies the runner) for targeted snapshot pushes, then launch `runLoop` (unchanged Think/emit logic). + - `publishFinding` -> `session.PublishAndWaitAsync(envelope{Kind:"finding"})` (broadcast to all peers; no relay). + - Push metrics immediately (no polling): the agent emits a `snapshot` envelope via `PublishToAndWaitAsync(savedRunnerCtx, snapshot)` the instant it reaches local consensus, and again on any later metric change (throttled) and at run end, so the runner always has the latest. Targeting the runner keeps snapshots off the peer broadcast path. +- Rewrite `[slim/cmd/agent/main.go](benchmarks/slim-vs-a2a/slim/cmd/agent/main.go)`: drop `ServerNewWithConnection`/`Register`; just setup runtime, join session, block on the receive loop. Keep `SLIM_AGENT_READY` print. Flags unchanged plus optional `--channel` (default `agntcy/bench-v2/consensus`). +- Rewrite `[slim/cmd/runner/main.go](benchmarks/slim-vs-a2a/slim/cmd/runner/main.go)` (passive observer / moderator): + - `startAgents` (unchanged spawn), then create the group session as moderator and `InviteAndWaitAsync` every agent name; small settle. + - Broadcast a single `start` (defines t0); after that the runner does NOT poll. It runs one receive loop on `GetMessageAsync`, keeping a `map[agentIndex]AgentSnapshot` of the latest pushed snapshot per agent. It ignores `finding` broadcasts it incidentally receives (it never forwards them — agents already got them directly, so there is no relay). + - Global-consensus detection is event-driven: after each pushed snapshot, run `consensus.GlobalConsensus` over the N latest snapshots; on success record wall time and finish. A 2-min deadline timer bounds the loop for the failure case. + - `aggregateResult`: same fields, computed from the collected pushed snapshots; set `StreamRPCCount = FindingsEmitted` (one native multicast per finding), `CoordFanoutMS = 0`, `UnicastRPCCount = 0` to highlight "no relay". + - Note on observation: the runner sees a copy of broadcast findings only because it is a group member, not because it relays them; agent-to-agent delivery happens via the SLIM dataplane fan-out independent of the runner. + +### 2. A2A side -> runner as relay with real streaming + +Make the runner the relay hub; remove the `agent-0` coordinator/fanout logic. Both legs are client-initiated A2A server-streams (`SendStreamingMessage`/executor `iter.Seq2[a2a.Event,error]`, present in `a2a-go v2.3.0`). + +- `[a2a/internal/protocol/protocol.go](benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go)`: add stream-init ops `OpStreamFindings` (runner subscribes to an agent's outbound findings) and `OpStreamRelay` (agent subscribes to runner's relayed findings); keep unary `OpStart`/`OpSnapshot`; drop `OpFinding`/`OpFanout`. +- Rewrite `[a2a/internal/agent/runtime.go](benchmarks/slim-vs-a2a/a2a/internal/agent/runtime.go)`: + - Remove `isCoordinator`, `fanout`, per-peer clients. Agent dials only the runner relay (card URL passed in). + - Hosts an A2A server: streaming `OpStreamFindings` executor drains an internal `findingsCh` and yields each as an `a2a.Event` until ctx done; unary `OpStart` (start `runLoop`) and `OpSnapshot` (return `AgentSnapshot`). + - On start, agent (as client) opens `SendStreamingMessage(OpStreamRelay)` to the runner and applies every relayed finding (`applyFinding`, skip self, `RecordPropagation`). + - `runLoop` pushes produced findings onto `findingsCh` (the publish leg) instead of unary send. +- Rewrite `[a2a/cmd/agent/main.go](benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go)`: executor routes streaming vs unary ops; add `--relay-card-url` (runner's card) flag. +- Rewrite `[a2a/cmd/runner/main.go](benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go)` + `[a2a/internal/client/client.go](benchmarks/slim-vs-a2a/a2a/internal/client/client.go)` (relay): + - Runner hosts its own A2A server + card (new `--relay-grpc-port`/`--relay-card-port`, default e.g. 9690/10690) implementing streaming `OpStreamRelay`: registers each subscribing agent's outbound channel. + - Runner (as client) subscribes via `SendStreamingMessage(OpStreamFindings)` to every agent; on each received finding, push into all agents' relay channels (skip none; agents skip self), incrementing `StreamRPCCount` and measuring `CoordFanoutMS`. + - Pass the runner relay card URL to each spawned agent. Keep unary `OpStart`/`OpSnapshot` to drive/poll agents and `GlobalConsensus`. + - `aggregateResult`: real `StreamRPCCount` (relayed events) and measured `CoordFanoutMS`; `UnicastRPCCount` = control RPCs only. + +### 3. Labels, report, Taskfile + +- Update impl tags in `[internal/benchlog/benchlog.go](benchmarks/slim-vs-a2a/internal/benchlog/benchlog.go)`: `ImplA2A = "a2a-relay-stream"`, `ImplSLIM = "slim-group-session"`; update the matching strings in `[tools/report/main.go](benchmarks/slim-vs-a2a/tools/report/main.go)` `groupByScenario`/labels. +- `[internal/metrics/metrics.go](benchmarks/slim-vs-a2a/internal/metrics/metrics.go)` struct/columns unchanged (semantics adjusted per above). +- `[Taskfile.yml](benchmarks/slim-vs-a2a/Taskfile.yml)`: `build` targets and slimctl dataplane block unchanged; SLIM runner now needs no extra ports; A2A runner gains relay-port flags (defaulted, so `compare:*` need minimal/no change). Add a one-line note to `[plans/README.md](benchmarks/slim-vs-a2a/plans/README.md)` that SLIM is relay-free and A2A relays via the runner. + +### 4. Validate + +- `task build`; `task compare:ci:smoke` (5 agents/20ms) to confirm both stacks converge and `reports/index.html` renders with the new tags. +- Run a sweep point at higher agent counts (e.g. 20 and 50 via existing `plans/sweeps/*`) to capture the `consensus_wall_ms` delta favoring SLIM, with both stacks under the same 2-min deadline. + +### Risks / notes + +- A2A long-lived streaming executor must yield events over time from a channel; if `a2a-go`'s task/streaming lifecycle fights a never-ending stream, fall back to: receive leg stays real streaming (agents subscribe to runner), publish leg becomes unary `SendMessage(OpFinding)` to the runner. Thesis (relay required) holds either way. +- SLIM session self-delivery: keep the existing self-skip by `AgentIndex` guard. +- Push-based snapshots: agents send their metric the instant they converge (no poll wait), so consensus is detected as soon as the last agent's snapshot arrives. The agent's converged snapshot already contains complete findings/propagation stats (it converged by receiving them); a throttled follow-up push plus an end-of-run push cover any late `findingsApplied` drift. Keep MLS off by default. +- Targeted snapshot push relies on the saved `start` message `Context` to address the runner; if reusing a stored context misbehaves, fall back to broadcasting `snapshot` envelopes (peers ignore non-self snapshots). +- Two CGO/native rewrites: build offline after `deps:slim-bindings-setup`; no new external deps. \ No newline at end of file diff --git a/.cursor/plans/slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md b/.cursor/plans/slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md deleted file mode 100644 index 724168fb..00000000 --- a/.cursor/plans/slim_vs_a2a_dag_benchmark_8b6e1f98.plan.md +++ /dev/null @@ -1,500 +0,0 @@ ---- -name: slim-vs-a2a -overview: CSIT comparison package at benchmarks/slim-vs-a2a — domain YAML DAG plans, two fully separate executors (A2A gRPC vs SLIM Multicast RPC), comparable metrics, and a dedicated test-slim-vs-a2a CI workflow (Phase 3). -todos: - - id: plan-schema - content: Define shared internal/plan YAML types, validation, and DAG helpers (only shared layer) - status: pending - - id: domain-plans - content: "Phase 1 — hand-author 3 domain YAML plans in plans/domains/ (no plangen)" - status: completed - - id: a2a-impl - content: Standalone A2A stack — cmd/a2a-agent, cmd/a2a-runner, internal/a2a/* (no SLIM imports) - status: pending - - id: slim-impl - content: Standalone SLIM stack — cmd/slim-agent, cmd/slim-runner, proto + internal/slim/* (no A2A imports) - status: pending - - id: metrics-reporting - content: Shared metrics schema + TSV/markdown reporting with sync/context/failure propagation fields - status: pending - - id: taskfile-suite - content: "Phase 2 — Taskfile + local Ginkgo suite (no CI yet)" - status: pending - - id: ci-integration - content: "Phase 3 — dedicated test-slim-vs-a2a.yaml workflow, CI smoke, artifacts, pages publish" - status: pending - - id: rename-project - content: "Rename benchmarks/agntcy-multi-agent → benchmarks/slim-vs-a2a (folder, go.mod, Taskfile namespace)" - status: completed -isProject: false ---- - -# SLIM vs A2A Comparison (`benchmarks/slim-vs-a2a`) - -This package lives under `benchmarks/` for repo layout consistency, but its tasks are **comparative evaluations** (same DAG plan, A2A vs SLIM metrics) — not throughput/latency benchmarks like `agntcy-slim`. Task names use the `compare:` prefix, not `benchmark:`. - -## Project naming - -| Item | Name | -| ---- | ---- | -| Directory | [`benchmarks/slim-vs-a2a/`](benchmarks/slim-vs-a2a/) | -| Go module | `github.com/agntcy/csit/benchmarks/slim-vs-a2a` | -| Taskfile namespace | `benchmarks:slim-vs-a2a:*` (e.g. `task benchmarks:slim-vs-a2a:compare:suite`) | -| CI workflow | [`.github/workflows/test-slim-vs-a2a.yaml`](.github/workflows/test-slim-vs-a2a.yaml) — **standalone**, not merged into `test-benchmarks-slim.yaml` or `test-integrations.yaml` | -| Report artifact prefix | `slim-vs-a2a-*` | - -Build a reproducible **comparison** in CSIT that shows **when SLIM Multicast RPC is favorable over direct A2A gRPC** in a distributed multi-agent workload: - -- **Phase 1 (plans):** Hand-author YAML execution plans with concrete agent/task names — no code generator *(done)* -- **Phase 2 (implementation):** Separate A2A and SLIM runners/agents, metrics, local Taskfile + Ginkgo suite -- **Phase 3 (CI):** Dedicated [`test-slim-vs-a2a.yaml`](.github/workflows/test-slim-vs-a2a.yaml) workflow — separate from SLIM benchmarks and A2A integration CI -- Tasks use **concrete domain-specific names** (network probe, pod log scrape, route recalc, etc.) but **simulate/mock** execution via timed sleeps and canned outputs -- Parallel branches become **obsolete** when an upstream dependency fails or times out -- **SLIM advantages under test** (three pillars): - 1. **Fast sync** — agents stay aligned on shared execution state with low coordination latency - 2. **Fast context change sharing** — when upstream results or priorities shift, all affected agents receive updated context in one fan-out (multicast) vs N point-to-point pushes - 3. **Fast failure propagation** — one multicast abort/notify vs K sequential A2A calls to cancel obsolete parallel work -- **A2A and SLIM are completely separate implementations** — separate binaries, packages, and RPC stacks; only YAML plan parsing/validation is shared (`internal/plan`) - -## Recommended defaults (questions skipped) - - -| Decision | Choice | Rationale | -| ------------- | ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Runtime | **Local processes + embedded SLIM dataplane** | Matches `[benchmarks/agntcy-slim](benchmarks/agntcy-slim)`; fast iteration; K8s deployment deferred beyond Phase 3 | -| Orchestration | **Separate runners per transport** (`a2a-runner`, `slim-runner`) | Each impl owns its own DAG driver; same plan YAML input, no shared scheduler interface | -| Language | **Go-only** | User asked for A2A Go SDK + SLIM Go bindings; existing Python `slima2a` is out of scope | -| SLIM API | **Custom slimrpc protobuf + multicast group client** | Multicast for context push + cancel/notify; upstream pattern in `[client_group.go](/Users/smagyari/dev/slim/data-plane/bindings/go/examples/slimrpc/simple/cmd/client_group/client_group.go)` | -| Shared code | **`internal/plan` only** | YAML load, validate, DAG helpers; no shared transport, agent, or orchestrator code | - - -## Architecture - -```mermaid -flowchart TB - subgraph shared [Shared layer] - PlanYAML[plans/*.yaml] - PlanLib[internal/plan] - PlanYAML --> PlanLib - end - - subgraph a2a [A2A implementation — standalone] - A2ARunner[a2a-runner] - A2AAgents[a2a-agent pool] - A2ARunner --> A2AAgents - end - - subgraph slim [SLIM implementation — standalone] - SlimRunner[slim-runner] - SlimAgents[slim-agent pool] - SlimRunner --> SlimAgents - end - - subgraph bench [Benchmark harness] - Suite[Ginkgo suite_test.go] - Suite --> A2ARunner - Suite --> SlimRunner - A2ARunner --> Metrics[results.tsv + summary.md] - SlimRunner --> Metrics - end - - PlanLib --> A2ARunner - PlanLib --> SlimRunner -``` - -### Why this highlights SLIM - -Both implementations execute the **same YAML plan** (same agents, tasks, timings, dependencies). Implementations are independent but must produce **comparable metrics**. SLIM's three advantages show up at coordination boundaries: - -| Coordination event | A2A gRPC (sequential / point-to-point) | SLIM Multicast RPC | -| ------------------ | -------------------------------------- | ------------------ | -| Execute task on one agent | Unary gRPC `SendMessage` per task | Point-to-point slimrpc unary | -| **Share context update** to K agents (new priority, revised hypothesis, abort reason) | **K sequential** push RPCs | **1 multicast** `PushContext` fan-out | -| **Sync barrier** — all agents acknowledge phase transition | **K sequential** sync RPCs + client-side aggregation | **1 multicast** `SyncPhase` + aggregated stream responses | -| Cancel obsolete parallel tasks (K agents) | **K sequential** cancel RPCs | **1 multicast** `CancelTasks` | -| Notify dependents of failure | **M sequential** notify RPCs | **1 multicast** `NotifyFailure` | - -Metrics like `context_push_ms`, `sync_barrier_ms`, `cancel_propagation_ms`, `obsolete_tasks_completed`, and `total_wall_clock_ms` should show SLIM advantage as agent count, parallel fan-out, and context-change frequency grow. - -## New package layout - -Create [`benchmarks/slim-vs-a2a/`](benchmarks/slim-vs-a2a/) and register it in [`benchmarks/Taskfile.yml`](benchmarks/Taskfile.yml) under the `slim-vs-a2a` include key: - -``` -benchmarks/slim-vs-a2a/ -├── Taskfile.yml -├── internal/ -│ └── plan/ # ONLY shared code: YAML schema, validation, DAG helpers -├── a2a/ # completely separate A2A implementation -│ ├── proto/ # optional A2A task payload JSON schema (not slim proto) -│ ├── cmd/ -│ │ ├── agent/main.go # a2a-agent — generic worker, differs by name/port only -│ │ └── runner/main.go # a2a-runner — loads plan, drives DAG via A2A gRPC -│ └── internal/ -│ ├── executor/ # task mock execution (sleep, timeout, cancel) -│ ├── scheduler/ # DAG state machine for A2A path -│ └── client/ # a2a-go card resolve, gRPC calls -├── slim/ # completely separate SLIM implementation -│ ├── proto/taskdag/v1/taskdag.proto # slimrpc service (Execute, Cancel, PushContext, SyncPhase) -│ ├── cmd/ -│ │ ├── agent/main.go # slim-agent — generic worker on dataplane -│ │ └── runner/main.go # slim-runner — loads plan, drives DAG via slimrpc -│ └── internal/ -│ ├── executor/ # same mock semantics as a2a (duplicated, not imported) -│ ├── scheduler/ # DAG state machine for SLIM path -│ └── client/ # slim-bindings-go p2p + multicast group client -├── tools/ -│ └── report_dashboard.go -├── plans/ -│ └── domains/ # hand-authored plans only (committed) -│ ├── k8s-incident-response.yaml -│ ├── urban-traffic-reroute.yaml -│ └── mobile-nav-assistant.yaml -├── tests/ -│ ├── suite_test.go -│ ├── slim_vs_a2a_test.go -│ └── helpers_test.go -├── templates/ -└── reports/ -``` - -**Separation rule:** `a2a/` must not import `slim/` or `slim-bindings-go`. `slim/` must not import `a2a/` or `a2a-go`. Both import only `internal/plan`. Mock executor logic may be duplicated intentionally to keep stacks independent (or extracted later behind build tags — not in v1). - -Reuse patterns from: - -- Metrics/reporting: `[benchmarks/agntcy-slim/tests/benchmark_models_test.go](benchmarks/agntcy-slim/tests/benchmark_models_test.go)`, `[benchmark_suite_test.go](benchmarks/agntcy-slim/tests/benchmark_suite_test.go)` -- SLIM lifecycle: `[benchmarks/agntcy-slim/tests/config/local-slim.yaml](benchmarks/agntcy-slim/tests/config/local-slim.yaml)` + `startLocalSlimStack()` helpers -- A2A gRPC server/client: `[integrations/agntcy-a2a/fixtures/go-jsonrpc-server/main.go](integrations/agntcy-a2a/fixtures/go-jsonrpc-server/main.go)` - -Dependencies (new `go.mod` under benchmark package): - -- `github.com/a2aproject/a2a-go/v2` (already pinned in `[integrations/go.mod](integrations/go.mod)`) -- `github.com/agntcy/slim-bindings-go` (same version as `[benchmarks/agntcy-slim/tests/echo-client/go.mod](benchmarks/agntcy-slim/tests/echo-client/go.mod)`) -- `gopkg.in/yaml.v3`, Ginkgo/Gomega, Gonum (for CI stats, optional) - -## YAML plan schema - -Each plan file describes **named agents**, **named tasks** (domain-realistic labels), dependencies, and timing. Tasks mock real work via `completionTimeSec` / `maxCompletionTimeSec` / canned `output`. - -### Schema (unchanged structure, concrete names) - -```yaml -apiVersion: bench.agntcy.io/v1 -kind: ExecutionPlan -metadata: - name: k8s-incident-response - domain: kubernetes-troubleshooting - description: Parallel diagnosis of a failing deployment; branches abort if root cause found early -spec: - defaults: - completionTimeSec: 0.3 - maxCompletionTimeSec: 2.0 - maxRetries: 1 -agents: - - id: cluster-scout - slimName: agntcy/bench/cluster-scout - a2aPort: 9101 - - id: log-miner - slimName: agntcy/bench/log-miner - a2aPort: 9102 - - id: metrics-analyst - slimName: agntcy/bench/metrics-analyst - a2aPort: 9103 - # ... -tasks: - - id: fetch-pod-status - name: "Fetch pod status for checkout-api" - agent: cluster-scout - dependsOn: [] - completionTimeSec: 0.25 - maxCompletionTimeSec: 1.0 - output: "phase=CrashLoopBackOff replicas=3/5" - - id: scrape-container-logs - name: "Scrape last 500 lines from crashing pod" - agent: log-miner - dependsOn: [fetch-pod-status] - completionTimeSec: 0.6 - maxCompletionTimeSec: 1.5 - injectFailure: false - output: "OOMKilled exit=137" - # ... -contextUpdates: # optional — triggers SLIM multicast vs A2A sequential push - - afterTask: fetch-pod-status - payload: "hypothesis=memory-pressure priority=high" - targetAgents: [log-miner, metrics-analyst, event-correlator] -``` - -Fields: -- `agents[].id` — logical role name used in task assignments -- `agents[].slimName` — SLIM dataplane identity (`org/group/app`) -- `agents[].a2aPort` — gRPC port for A2A agent card (A2A impl only) -- `tasks[].name` — human-readable task label (reporting + realism) -- `contextUpdates[]` — models mid-plan context shifts (key SLIM sync/context comparison axis) - -### Three hand-authored domain plans (committed in `plans/domains/`) - -#### 1. `k8s-incident-response.yaml` — K8s cluster troubleshooting - -**Scenario:** `checkout-api` deployment degraded; agents diagnose in parallel; if OOM root cause confirmed early, cancel speculative branches (network partition check, node drain simulation). - -| Agent | Role | -|-------|------| -| `cluster-scout` | Pod/status/endpoint discovery | -| `log-miner` | Container log extraction | -| `metrics-analyst` | Prometheus metric queries | -| `event-correlator` | K8s event timeline merge | -| `network-prober` | Service mesh / DNS checks | -| `runbook-executor` | Remediation steps (scale, rollback) | - -Example tasks: `fetch-pod-status`, `scrape-container-logs`, `query-memory-metrics`, `correlate-oom-events`, `probe-service-mesh`, `simulate-rollback`, `apply-memory-limit-patch`. DAG depth ~4, fan-out up to 4 parallel probes; one injected failure path on `probe-service-mesh` to exercise cancel propagation. - -#### 2. `urban-traffic-reroute.yaml` — Realtime traffic incident response - -**Scenario:** Highway accident detected; traffic agents recompute routes, update signals, notify transit; if primary corridor reopening ETA drops, cancel stale detour plans. - -| Agent | Role | -|-------|------| -| `incident-detector` | Ingest camera/sensor feed | -| `flow-modeler` | Traffic flow simulation | -| `route-planner` | Dynamic route recalculation | -| `signal-coordinator` | Intersection timing adjustments | -| `transit-dispatcher` | Bus/rail reroute | -| `alert-broadcaster` | Waze/511-style push | - -Example tasks: `detect-lane-closure`, `estimate-queue-length`, `recalc-northbound-routes`, `recalc-southbound-routes`, `adjust-signal-phasing`, `reroute-bus-line-42`, `publish-driver-alerts`. High parallel fan-out (N/S/E/W route recalc); context update when `estimate-queue-length` revises ETA. - -#### 3. `mobile-nav-assistant.yaml` — Android voice assistant (navigate + search) - -**Scenario:** User on phone says *"Take me to the nearest open pharmacy that's on the way to the airport"*; agents parse intent, search POIs, plan route, refine as user moves; if user changes priority ("actually, coffee first"), obsolete search/nav branches cancel. - -| Agent | Role | -|-------|------| -| `intent-parser` | NLU / slot extraction from Android input | -| `poi-searcher` | Places API search (pharmacy, coffee) | -| `route-engine` | Turn-by-turn path computation | -| `traffic-watcher` | Live ETA adjustments | -| `ui-composer` | Android notification / map tile updates | -| `session-tracker` | User location + command history sync | - -Example tasks: `parse-voice-command`, `search-pharmacy-near-route`, `search-coffee-near-route`, `compute-route-to-airport-via-poi`, `refine-route-live-traffic`, `render-map-overlay`, `push-voice-prompt`. Context update when user amends command mid-flight (`priority=coffee-first`); multiple parallel POI searches become obsolete on context change. - -Plan scale (agent count, task count, fan-out, failure injection) is defined **directly in each YAML file**. Add or edit plans by hand; no generator. - -## Agent design (two separate binaries, identical mock semantics) - -### `a2a/cmd/agent` — A2A-only generic worker - -``` -a2a-agent --agent-id=cluster-scout --grpc-port=9101 --card-port=8081 -``` - -- Implements `a2asrv` executor: parse JSON task payload (`taskId`, `completionTimeSec`, `maxCompletionTimeSec`, `context`) -- Mock execution: sleep, enforce deadline, honour cancel messages, return canned `output` -- Exposes gRPC via `a2agrpc.NewHandler`; agent card at `/.well-known/agent-card.json` -- **No SLIM imports** - -All agents share this one binary; they differ only by `--agent-id`, ports, and SLIM name mapping comes from plan YAML (SLIM agents use `slimName` field). - -### `slim/cmd/agent` — SLIM-only generic worker - -``` -slim-agent --slim-name=agntcy/bench/cluster-scout --endpoint=127.0.0.1:46357 -``` - -- Registers slimrpc `TaskDAG` service from `slim/proto/taskdag/v1/taskdag.proto` -- Same mock execution semantics as A2A agent (duplicated code in `slim/internal/executor`) -- Subscribes to `slimName` on dataplane -- **No A2A imports** - -## Runners (two separate DAG drivers) - -Each runner is a **standalone program** with its own scheduler — no shared `Transport` interface. - -### `a2a/cmd/runner` - -1. Load plan via `internal/plan` -2. Spawn `a2a-agent` processes for each agent in plan -3. Run DAG scheduler (`a2a/internal/scheduler`): - - Execute ready tasks via sequential/per-task A2A gRPC `SendMessage` - - On `contextUpdates`: **loop** `SendMessage` to each target agent with updated payload - - On failure/timeout: **loop** cancel RPCs to each affected agent - - Track sync barrier latency as time to complete all sequential round-trips -4. Write `RunMetrics` JSON to stdout / file - -### `slim/cmd/runner` - -1. Load same plan via `internal/plan` -2. Start local SLIM dataplane (or connect to existing) -3. Spawn `slim-agent` processes for each agent -4. Run DAG scheduler (`slim/internal/scheduler`): - - Execute ready tasks via point-to-point slimrpc unary - - On `contextUpdates`: **multicast** `PushContext` to group channel of target agents - - On phase transitions requiring ack: **multicast** `SyncPhase`, aggregate stream responses - - On failure/timeout: **multicast** `CancelTasks` + `NotifyFailure` -5. Write same-shape `RunMetrics` JSON - -Proto (`slim/proto/taskdag/v1/taskdag.proto`): - -```protobuf -service TaskDAG { - rpc ExecuteTask(ExecuteTaskRequest) returns (ExecuteTaskResponse); - rpc CancelTask(CancelTaskRequest) returns (CancelTaskResponse); - rpc PushContext(ContextUpdate) returns (ContextAck); -} -service TaskDAGGroup { - rpc PushContextMulticast(ContextUpdate) returns (stream MulticastContextAck); - rpc SyncPhase(PhaseSyncRequest) returns (stream MulticastPhaseAck); - rpc CancelTasks(CancelTasksRequest) returns (stream MulticastCancelResult); - rpc NotifyFailure(NotifyFailureRequest) returns (stream MulticastNotifyResult); -} -``` - -## Metrics schema - -Both runners emit the **same JSON/TSV shape** (defined in `tests/` or a tiny shared `internal/metrics` struct-only package with no runtime logic — acceptable if needed for report aggregation only). Fields: - -| Field | Description | -| ----- | ----------- | -| `plan_name`, `domain` | Plan id and domain label | -| `implementation` | `a2a-grpc` or `slim-multicast` | -| `agents`, `tasks` | Plan scale | -| `total_wall_clock_ms` | End-to-end plan execution | -| `tasks_completed`, `tasks_failed`, `tasks_timed_out` | Outcome counts | -| `tasks_cancelled`, `obsolete_tasks_completed` | Wasted work after failure/context change | -| `retries_attempted`, `retries_succeeded` | Retry stats | -| `context_push_ms` | Time to deliver context update to all target agents | -| `sync_barrier_ms` | Time for all agents to ack phase sync | -| `cancel_propagation_ms` | Failure → all cancels acked | -| `execute_rpc_count`, `multicast_rpc_count`, `sequential_rpc_count` | RPC overhead breakdown | -| `makespan_ms` | Critical-path duration (successful runs) | - -Comparison report auto-computes per-plan deltas: `(a2a - slim) / a2a` for wall clock, context push, sync barrier, cancel propagation, obsolete tasks. - -## Local execution (Phase 2) - -[`benchmarks/slim-vs-a2a/Taskfile.yml`](benchmarks/slim-vs-a2a/Taskfile.yml) — local dev only in Phase 2. - -### Task naming (`compare:` not `benchmark:`) - -| Task | Purpose | -| ---- | ------- | -| `deps:slim-bindings-setup` | Reuse slim-bindings setup from `agntcy-slim` | -| `compare:suite` | Run all domain plans on both A2A and SLIM; collect comparison metrics | -| `compare:report` | Build HTML dashboard from `reports/` comparison results | -| `compare:ci:smoke` | *(Phase 3)* Bounded plan matrix for CI (~2 min) | - -Configurable env vars: - -- `PLAN_DIR=plans/domains` (default; all hand-authored plans) -- `IMPLEMENTATIONS=a2a,slim` -- `RUN_SLIM_VS_A2A=1` (gate env var for Ginkgo suite; analogous to `SLIM_RUN_BENCHMARK_SUITE` but named for comparison) - -Example local run: - -```bash -task benchmarks:slim-vs-a2a:compare:suite RUN_SLIM_VS_A2A=1 -task benchmarks:slim-vs-a2a:compare:report -``` - -## Test suite (Phase 2 — Ginkgo, local) - -[`tests/slim_vs_a2a_test.go`](benchmarks/slim-vs-a2a/tests/slim_vs_a2a_test.go): - -- Label: `slim-vs-a2a` -- Skip unless `RUN_SLIM_VS_A2A=1` -- For each plan in `PLAN_DIR`: - 1. Run `a2a-runner --plan=...` → collect metrics - 2. Run `slim-runner --plan=...` (start SLIM stack first) → collect metrics -- Assert all three domain plans complete on both implementations; report deltas (no hard latency assertions) - -## CI integration (Phase 3 — dedicated workflow) - -Wire into CI only after Phase 2 runs cleanly locally. Use a **completely separate** workflow — do not add jobs to [`test-benchmarks-slim.yaml`](.github/workflows/test-benchmarks-slim.yaml) or [`test-integrations.yaml`](.github/workflows/test-integrations.yaml). - -**New file:** [`.github/workflows/test-slim-vs-a2a.yaml`](.github/workflows/test-slim-vs-a2a.yaml) - -| Deliverable | Path / task | -| ----------- | ------------- | -| Register in benchmarks root Taskfile | [`benchmarks/Taskfile.yml`](benchmarks/Taskfile.yml) — `includes.slim-vs-a2a` → `./slim-vs-a2a` | -| CI smoke Taskfile task | `compare:ci:smoke` — 3 domain plans × both implementations (~2 min) | -| Standalone CI workflow | `.github/workflows/test-slim-vs-a2a.yaml` | -| Workflow name (GitHub UI) | `test-slim-vs-a2a` | -| Path filter | `benchmarks/slim-vs-a2a/**`, `.github/workflows/test-slim-vs-a2a.yaml` | -| CI job command | `task benchmarks:slim-vs-a2a:compare:ci:smoke` | -| Report artifacts | Upload `reports/` as `slim-vs-a2a-smoke-report` (separate artifact name from SLIM benchmark reports) | -| Pages publish hook | Extend [`publish-test-reports-pages.yaml`](.github/workflows/publish-test-reports-pages.yaml) with a dedicated `--slim-vs-a2a-dir` input; do not fold into existing SLIM smoke/capacity dirs | - -Phase 3 CI smoke example: - -```bash -task benchmarks:slim-vs-a2a:compare:ci:smoke RUN_SLIM_VS_A2A=1 -``` - -### `test-slim-vs-a2a.yaml` sketch - -```yaml -name: test-slim-vs-a2a -on: - pull_request: - paths: - - benchmarks/slim-vs-a2a/** - - .github/workflows/test-slim-vs-a2a.yaml - workflow_dispatch: -jobs: - slim-vs-a2a-smoke: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - name: Run comparison smoke test - run: task benchmarks:slim-vs-a2a:compare:ci:smoke - - name: Upload report - uses: actions/upload-artifact@v4 - with: - name: slim-vs-a2a-smoke-report - path: benchmarks/slim-vs-a2a/reports/ -``` - -## Expected outcome - -Running the domain plans (especially `urban-traffic-reroute` and `mobile-nav-assistant` with frequent context updates) should show: - -- Similar `makespan_ms` on **successful** runs with no context churn (happy path parity) -- Lower `context_push_ms` and `sync_barrier_ms` with **SLIM** when `contextUpdates` fan out to many agents -- Lower `cancel_propagation_ms` and `obsolete_tasks_completed` with **SLIM** when failures abort parallel branches (K8s incident plan) -- Lower `total_wall_clock_ms` for SLIM on context-heavy and failure-heavy plans - -**Narrative:** SLIM Multicast RPC enables **fast sync**, **fast shared context propagation**, and **fast failure propagation** — all critical when many agents execute interdependent realtime tasks and obsolete work must be abandoned quickly. - -## Implementation order - -### Phase 1 — Plans (YAML only) — complete - -1. Finalize schema in plan doc / `internal/plan` types -2. Hand-author 3 domain plans in [`plans/domains/`](benchmarks/slim-vs-a2a/plans/domains/) with full agent lists, task DAGs, timings, `contextUpdates`, and injected failure paths - -### Phase 2 — Implementation (local only) - -0. ~~**Rename** `benchmarks/agntcy-multi-agent/` → `benchmarks/slim-vs-a2a/`~~ *(done)* -3. **`internal/plan`** — YAML load, validation, DAG helpers -4. **`a2a/` stack** — agent + runner + scheduler; end-to-end on domain plans -5. **`slim/` stack** — proto, agent + runner + scheduler with multicast; same plans end-to-end -6. **Context sync + failure propagation** — wire `contextUpdates`, cancel obsolete tasks, retries -7. **Metrics + TSV + templates + local report dashboard** -8. **Taskfile + Ginkgo suite** — runnable locally via `task benchmarks:slim-vs-a2a:compare:suite` - -### Phase 3 — CI and publishing (dedicated flow) - -9. **`compare:ci:smoke`** Taskfile task (bounded matrix for CI) -10. **Register** `slim-vs-a2a` in [`benchmarks/Taskfile.yml`](benchmarks/Taskfile.yml) -11. **Create** [`.github/workflows/test-slim-vs-a2a.yaml`](.github/workflows/test-slim-vs-a2a.yaml) — standalone workflow, separate triggers and artifacts from SLIM/A2A/integration CI -12. **Artifact upload** — `slim-vs-a2a-smoke-report` from CI runs -13. **Test reports site** — dedicated publish input in `publish-test-reports-pages.yaml` + dashboard section - -## Out of scope (initial version) - -- **`plangen` / automated plan scaling** — plans are hand-authored; add new YAML files manually -- K8s/Kind deployment (deferred beyond Phase 3; same binaries can be containerized later) -- Python `slima2a` agents -- A2A-over-SLIM third transport leg (could be added later as `slim-a2a`) -- Directory/registry integration (`agntcy-dir`) — unrelated to this comparison - diff --git a/.github/workflows/test-slim-vs-a2a-v2.yaml b/.github/workflows/test-slim-vs-a2a-v2.yaml deleted file mode 100644 index b4634b39..00000000 --- a/.github/workflows/test-slim-vs-a2a-v2.yaml +++ /dev/null @@ -1,22 +0,0 @@ -name: test-slim-vs-a2a-v2 - -on: - workflow_dispatch: - pull_request: - paths: - - 'benchmarks/slim-vs-a2a-v2/**' - - '.github/workflows/test-slim-vs-a2a-v2.yaml' - -jobs: - smoke: - runs-on: ubuntu-latest - defaults: - run: - working-directory: benchmarks/slim-vs-a2a-v2 - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-go@v5 - with: - go-version: '1.25' - - name: Run CI smoke - run: task compare:ci:smoke diff --git a/benchmarks/Taskfile.yml b/benchmarks/Taskfile.yml index 6152f79e..faee6796 100644 --- a/benchmarks/Taskfile.yml +++ b/benchmarks/Taskfile.yml @@ -22,11 +22,6 @@ includes: dir: ./slim-vs-a2a excludes: [default] - slim-vs-a2a-v2: - taskfile: ./slim-vs-a2a-v2/Taskfile.yml - dir: ./slim-vs-a2a-v2 - excludes: [default] - tasks: default: cmd: task -l \ No newline at end of file diff --git a/benchmarks/slim-vs-a2a-v2/Taskfile.yml b/benchmarks/slim-vs-a2a-v2/Taskfile.yml deleted file mode 100644 index ccae223a..00000000 --- a/benchmarks/slim-vs-a2a-v2/Taskfile.yml +++ /dev/null @@ -1,348 +0,0 @@ -# Copyright AGNTCY Contributors (https://github.com/agntcy) -# SPDX-License-Identifier: Apache-2.0 - ---- -version: '3' - -silent: true - -vars: - SCENARIO_DIR: '{{ .SCENARIO_DIR | default "./plans/sweeps" }}' - REPORTS_DIR: '{{ .REPORTS_DIR | default "./reports" }}' - RESULTS_TSV: '{{ .RESULTS_TSV | default "./reports/results.tsv" }}' - SWEEP_TSV: '{{ .SWEEP_TSV | default "./reports/sweep.tsv" }}' - BIN_DIR: '{{ .BIN_DIR | default "./bin" }}' - SLIM_ENDPOINT: '{{ .SLIM_ENDPOINT | default "http://127.0.0.1:46357" }}' - COMPARE_SLIMCTL: '{{ .COMPARE_SLIMCTL | default "../agntcy-slim/bin/slimctl" }}' - COMPARE_SLIM_BINDINGS_VERSION: '{{ .COMPARE_SLIM_BINDINGS_VERSION | default "v1.4.0" }}' - COMPARE_SLIMCTL_TAG: '{{ .COMPARE_SLIMCTL_TAG | default "slimctl-v1.4.0" }}' - -tasks: - default: - cmd: task -l - - deps:slim-bindings-setup: - desc: Install native SLIM bindings required by slim agents - cmds: - - go run github.com/agntcy/slim-bindings-go/cmd/slim-bindings-setup@{{ .COMPARE_SLIM_BINDINGS_VERSION }} - - deps:slimctl-download: - desc: Download slimctl for local SLIM stack startup - cmds: - - | - set -e - SLIMCTL_ARCH=$(arch) - SLIMCTL_OS=$(uname -s | tr '[:upper:]' '[:lower:]') - SLIMCTL_ABI="" - if [ "$SLIMCTL_ARCH" = "x86_64" ]; then - SLIMCTL_ARCH="amd64" - fi - if [ "$SLIMCTL_OS" = "linux" ]; then - if ldd --version 2>&1 | grep -qi musl; then - SLIMCTL_ABI="-musl" - else - SLIMCTL_ABI="-gnu" - fi - fi - SLIMCTL_URL="https://github.com/agntcy/slim/releases/download/{{ .COMPARE_SLIMCTL_TAG }}/slimctl-$SLIMCTL_OS-$SLIMCTL_ARCH${SLIMCTL_ABI}.tar.gz" - TARGET_DIR=$(dirname {{ .COMPARE_SLIMCTL }}) - mkdir -p "$TARGET_DIR" - TMP_DIR=$(mktemp -d) - trap 'rm -rf "$TMP_DIR"' EXIT - curl --fail --show-error --location -o "$TMP_DIR/slimctl.tar.gz" "$SLIMCTL_URL" - tar -xzf "$TMP_DIR/slimctl.tar.gz" -C "$TMP_DIR" - mv "$TMP_DIR/slimctl" {{ .COMPARE_SLIMCTL }} - chmod +x {{ .COMPARE_SLIMCTL }} - {{ .COMPARE_SLIMCTL }} version || true - - build: - desc: Build a2a and slim agent/runner binaries - deps: - - deps:slim-bindings-setup - cmds: - - mkdir -p {{ .BIN_DIR }} - - go build -o {{ .BIN_DIR }}/a2a-agent ./a2a/cmd/agent - - go build -o {{ .BIN_DIR }}/a2a-runner ./a2a/cmd/runner - - | - CACHE_DIR="$(go env GOPATH)/.cgo-cache/slim-bindings/{{ .COMPARE_SLIM_BINDINGS_VERSION }}" - export CGO_LDFLAGS="-L${CACHE_DIR}" - go build -o {{ .BIN_DIR }}/slim-agent ./slim/cmd/agent - go build -o {{ .BIN_DIR }}/slim-runner ./slim/cmd/runner - - validate:scenarios: - desc: Validate consensus scenario YAML files - cmds: - - go run ./tools/validate_scenarios --dir {{ .SCENARIO_DIR }} - - compare:cleanup: - internal: true - cmds: - - pkill -f '{{ .BIN_DIR }}/a2a-agent' 2>/dev/null || true - - pkill -f '{{ .BIN_DIR }}/slim-agent' 2>/dev/null || true - - pkill -f 'slimctl slim start -c' 2>/dev/null || true - - compare:suite: - desc: Run default consensus scenarios on A2A and SLIM - deps: - - build - - validate:scenarios - cmds: - - task: compare:cleanup - - rm -f {{ .RESULTS_TSV }} - - | - set -euo pipefail - for scenario in {{ .SCENARIO_DIR }}/hypothesis-convergence-*.yaml; do - [ -f "$scenario" ] || continue - base=$(basename "$scenario" .yaml) - echo "==> A2A $base" - {{ .BIN_DIR }}/a2a-runner \ - --scenario "$scenario" \ - --agent-bin {{ .BIN_DIR }}/a2a-agent \ - --quiet \ - --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ - --output-tsv {{ .RESULTS_TSV }} || true - done - - task: compare:slim-stack - vars: - SCENARIO_GLOB: '{{ .SCENARIO_DIR }}/hypothesis-convergence-*.yaml' - - compare:plan: - desc: Run one scenario on A2A and SLIM (PLAN=path or basename) - deps: - - build - requires: - vars: [PLAN] - cmds: - - task: compare:cleanup - - rm -f {{ .RESULTS_TSV }} - - | - set -euo pipefail - PLAN_INPUT="{{ .PLAN }}" - if [ -f "$PLAN_INPUT" ]; then - SCENARIO_PATH="$PLAN_INPUT" - elif [ -f "{{ .SCENARIO_DIR }}/${PLAN_INPUT}.yaml" ]; then - SCENARIO_PATH="{{ .SCENARIO_DIR }}/${PLAN_INPUT}.yaml" - else - echo "scenario not found: ${PLAN_INPUT}" >&2 - exit 1 - fi - base=$(basename "$SCENARIO_PATH" .yaml) - echo "==> A2A ${base}" - {{ .BIN_DIR }}/a2a-runner \ - --scenario "$SCENARIO_PATH" \ - --agent-bin {{ .BIN_DIR }}/a2a-agent \ - --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ - --output-tsv {{ .RESULTS_TSV }} - - task: compare:slim-stack - vars: - SCENARIO_GLOB: '{{ .PLAN }}' - - compare:slim-stack: - internal: true - deps: - - deps:slimctl-download - cmds: - - | - set -euo pipefail - SCENARIO_INPUT="{{ .SCENARIO_GLOB }}" - resolve_scenario() { - local input="$1" - if [ -f "$input" ]; then - printf '%s' "$input" - return - fi - if [ -f "{{ .SCENARIO_DIR }}/${input}.yaml" ]; then - printf '%s' "{{ .SCENARIO_DIR }}/${input}.yaml" - return - fi - printf '%s' "$input" - } - DATAPLANE_PORT=46357 - CONTROLLER_PORT=46358 - CONFIG=$(mktemp) - trap 'kill ${SLIM_PID:-} 2>/dev/null || true; rm -f "$CONFIG"; {{ .COMPARE_SLIMCTL }} slim stop 2>/dev/null || true' EXIT - cat > "$CONFIG" </dev/null; then - break - fi - sleep 0.2 - done - if [[ "$SCENARIO_INPUT" == *"*"* ]]; then - SCENARIO_LIST=($SCENARIO_INPUT) - else - SCENARIO_LIST=("$(resolve_scenario "$SCENARIO_INPUT")") - fi - for scenario in "${SCENARIO_LIST[@]}"; do - [ -f "$scenario" ] || continue - base=$(basename "$scenario" .yaml) - echo "==> SLIM ${base}" - {{ .BIN_DIR }}/slim-runner \ - --scenario "$scenario" \ - --endpoint {{ .SLIM_ENDPOINT }} \ - --agent-bin {{ .BIN_DIR }}/slim-agent \ - --quiet \ - --output-json {{ .REPORTS_DIR }}/slim-${base}.json \ - --output-tsv {{ .RESULTS_TSV }} || true - done - - compare:report: - desc: Build HTML dashboard from comparison TSV results - cmds: - - go run ./tools/report --tsv {{ .RESULTS_TSV }} --sweep-tsv {{ .SWEEP_TSV }} --output {{ .REPORTS_DIR }}/index.html - - compare:sweep: - desc: Run agent/think-time sweep (SWEEP_AGENTS, SWEEP_THINK_MS env lists) - deps: - - build - vars: - SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' - SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' - SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' - cmds: - - task: compare:cleanup - - rm -f {{ .SWEEP_TSV }} - - task: compare:sweep:run - - compare:sweep:run: - internal: true - vars: - SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' - SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' - SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' - cmds: - - | - set -euo pipefail - mkdir -p {{ .REPORTS_DIR }} {{ .SCENARIO_DIR }} - IFS=',' read -r -a AGENT_LIST <<< "{{ .SWEEP_AGENTS }}" - IFS=',' read -r -a THINK_LIST <<< "{{ .SWEEP_THINK_MS }}" - for agents in "${AGENT_LIST[@]}"; do - for think in "${THINK_LIST[@]}"; do - scenario_path="{{ .SCENARIO_DIR }}/{{ .SWEEP_FAMILY }}-${agents}ag-${think}ms.yaml" - go run ./tools/gen_scenario \ - -family {{ .SWEEP_FAMILY }} \ - -agents "$agents" \ - -think-ms "$think" \ - -output "$scenario_path" - base=$(basename "$scenario_path" .yaml) - echo "==> sweep A2A $base" - {{ .BIN_DIR }}/a2a-runner \ - --scenario "$scenario_path" \ - --agent-bin {{ .BIN_DIR }}/a2a-agent \ - --quiet \ - --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ - --output-tsv {{ .SWEEP_TSV }} || true - done - done - - task: compare:sweep:slim - - compare:sweep:slim: - internal: true - deps: - - deps:slimctl-download - vars: - SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' - SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' - SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' - cmds: - - | - set -euo pipefail - IFS=',' read -r -a AGENT_LIST <<< "{{ .SWEEP_AGENTS }}" - IFS=',' read -r -a THINK_LIST <<< "{{ .SWEEP_THINK_MS }}" - DATAPLANE_PORT=46357 - CONTROLLER_PORT=46358 - CONFIG=$(mktemp) - trap 'kill ${SLIM_PID:-} 2>/dev/null || true; rm -f "$CONFIG"; {{ .COMPARE_SLIMCTL }} slim stop 2>/dev/null || true' EXIT - cat > "$CONFIG" </dev/null; then - break - fi - sleep 0.2 - done - for agents in "${AGENT_LIST[@]}"; do - for think in "${THINK_LIST[@]}"; do - scenario_path="{{ .SCENARIO_DIR }}/{{ .SWEEP_FAMILY }}-${agents}ag-${think}ms.yaml" - [ -f "$scenario_path" ] || continue - base=$(basename "$scenario_path" .yaml) - echo "==> sweep SLIM $base" - {{ .BIN_DIR }}/slim-runner \ - --scenario "$scenario_path" \ - --endpoint {{ .SLIM_ENDPOINT }} \ - --agent-bin {{ .BIN_DIR }}/slim-agent \ - --quiet \ - --output-json {{ .REPORTS_DIR }}/slim-${base}.json \ - --output-tsv {{ .SWEEP_TSV }} || true - done - done - - compare:ci:smoke: - desc: CI smoke — one sweep point plus report (~2 min) - deps: - - build - cmds: - - task: compare:cleanup - - rm -f {{ .RESULTS_TSV }} {{ .SWEEP_TSV }} - - | - set -euo pipefail - mkdir -p {{ .SCENARIO_DIR }} {{ .REPORTS_DIR }} - scenario_path="{{ .SCENARIO_DIR }}/ci-smoke-hypothesis-5ag-20ms.yaml" - go run ./tools/gen_scenario -family hypothesis-convergence -agents 5 -think-ms 20 -output "$scenario_path" - echo "==> smoke A2A" - {{ .BIN_DIR }}/a2a-runner \ - --scenario "$scenario_path" \ - --agent-bin {{ .BIN_DIR }}/a2a-agent \ - --quiet \ - --output-json {{ .REPORTS_DIR }}/a2a-ci-smoke.json \ - --output-tsv {{ .RESULTS_TSV }} - - task: compare:slim-stack - vars: - SCENARIO_GLOB: 'plans/sweeps/ci-smoke-hypothesis-5ag-20ms.yaml' - - task: compare:report - - test: - desc: Run unit tests - cmds: - - go test ./... -count=1 diff --git a/benchmarks/slim-vs-a2a-v2/a2a/cmd/agent/main.go b/benchmarks/slim-vs-a2a-v2/a2a/cmd/agent/main.go deleted file mode 100644 index a98348ff..00000000 --- a/benchmarks/slim-vs-a2a-v2/a2a/cmd/agent/main.go +++ /dev/null @@ -1,161 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package main - -import ( - "context" - "encoding/json" - "flag" - "fmt" - "iter" - "log" - "net" - "net/http" - - "github.com/a2aproject/a2a-go/v2/a2a" - a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" - "github.com/a2aproject/a2a-go/v2/a2asrv" - "github.com/a2aproject/a2a-go/v2/a2asrv/taskstore" - agentrt "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/agent" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" - "google.golang.org/grpc" -) - -type consensusExecutor struct { - runtime *agentrt.Runtime -} - -func (e *consensusExecutor) Execute(ctx context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { - reqText := firstText(execCtx.Message) - req, err := agentrt.DecodeRequestText(reqText) - if err != nil { - return func(yield func(a2a.Event, error) bool) { - yield(nil, fmt.Errorf("decode request: %w", err)) - } - } - - if req.Op == protocol.OpStreamFindings { - return e.runtime.StreamFindings(ctx, execCtx) - } - - resp := e.runtime.HandleUnary(ctx, req) - body, err := json.Marshal(resp) - if err != nil { - return func(yield func(a2a.Event, error) bool) { - yield(nil, err) - } - } - return func(yield func(a2a.Event, error) bool) { - task := a2a.NewSubmittedTask(execCtx, execCtx.Message) - state := a2a.TaskStateCompleted - if !resp.OK { - state = a2a.TaskStateFailed - } - task.Status = a2a.TaskStatus{ - State: state, - Message: a2a.NewMessageForTask( - a2a.MessageRoleAgent, - task, - a2a.NewTextPart(string(body)), - ), - } - yield(task, nil) - } -} - -func (e *consensusExecutor) Cancel(_ context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { - return func(yield func(a2a.Event, error) bool) { - yield(a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateCanceled, a2a.NewMessageForTask( - a2a.MessageRoleAgent, execCtx, a2a.NewTextPart(`{"ok":true}`), - )), nil) - } -} - -func firstText(message *a2a.Message) string { - if message == nil { - return "" - } - for _, part := range message.Parts { - if text := part.Text(); text != "" { - return text - } - } - return "" -} - -func main() { - agentID := flag.String("agent-id", "", "logical agent id") - grpcPort := flag.Int("grpc-port", 0, "A2A gRPC port") - cardPort := flag.Int("card-port", 0, "agent card HTTP port") - scenarioFile := flag.String("scenario-file", "", "consensus scenario yaml") - agentIndex := flag.Int("agent-index", 0, "agent index") - relayCardURL := flag.String("relay-card-url", "", "runner relay agent card base URL") - flag.Parse() - - if *agentID == "" || *grpcPort == 0 || *scenarioFile == "" || *relayCardURL == "" { - log.Fatal("agent-id, grpc-port, scenario-file, and relay-card-url are required") - } - card := *cardPort - if card == 0 { - card = *grpcPort + 1000 - } - - s, err := scenario.LoadFile(*scenarioFile) - if err != nil { - log.Fatalf("load scenario: %v", err) - } - - rt := agentrt.NewRuntime(s, *agentIndex, *agentID, *relayCardURL) - defer rt.Close() - - handler := a2asrv.NewHandler( - &consensusExecutor{runtime: rt}, - a2asrv.WithTaskStore(taskstore.NewInMemory(&taskstore.InMemoryStoreConfig{ - Authenticator: func(context.Context) (string, error) { return "slim-vs-a2a-v2", nil }, - })), - ) - - agentCard := &a2a.AgentCard{ - Name: fmt.Sprintf("consensus-v2-%s", *agentID), - Description: "SLIM vs A2A v2 consensus agent", - Version: "1.0.0", - SupportedInterfaces: []*a2a.AgentInterface{ - a2a.NewAgentInterface(fmt.Sprintf("127.0.0.1:%d", *grpcPort), a2a.TransportProtocolGRPC), - }, - Capabilities: a2a.AgentCapabilities{Streaming: true}, - DefaultInputModes: []string{"text/plain"}, - DefaultOutputModes: []string{"text/plain"}, - } - - cardListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", card)) - if err != nil { - log.Fatalf("card listen: %v", err) - } - grpcListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", *grpcPort)) - if err != nil { - log.Fatalf("grpc listen: %v", err) - } - - grpcHandler := a2agrpc.NewHandler(handler) - grpcServer := grpc.NewServer() - grpcHandler.RegisterWith(grpcServer) - - go func() { - mux := http.NewServeMux() - mux.Handle(a2asrv.WellKnownAgentCardPath, a2asrv.NewStaticAgentCardHandler(agentCard)) - if err := http.Serve(cardListener, mux); err != nil { - log.Printf("card server stopped: %v", err) - } - }() - - // Subscribe to the runner relay stream as soon as we are up; it retries - // until the relay server is reachable. - rt.StartRelaySubscription(context.Background()) - - fmt.Printf("A2A_AGENT_READY agent=%s grpc=%d card=%d\n", *agentID, *grpcPort, card) - if err := grpcServer.Serve(grpcListener); err != nil { - log.Fatalf("grpc serve: %v", err) - } -} diff --git a/benchmarks/slim-vs-a2a-v2/a2a/cmd/runner/main.go b/benchmarks/slim-vs-a2a-v2/a2a/cmd/runner/main.go deleted file mode 100644 index 0e2fa05c..00000000 --- a/benchmarks/slim-vs-a2a-v2/a2a/cmd/runner/main.go +++ /dev/null @@ -1,281 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -// Command runner is the A2A benchmark driver and relay hub. Because A2A has no -// peer multicast, the runner is the explicit relay: it subscribes to every -// agent's findings stream (OpStreamFindings) and fans each finding out to all -// other agents over the relay stream (OpStreamRelay). It also drives start and -// polls snapshots to detect global consensus. -package main - -import ( - "context" - "encoding/json" - "flag" - "fmt" - "log" - "os" - "os/exec" - "time" - - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/client" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/relay" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/benchlog" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/metrics" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" -) - -func main() { - scenarioPath := flag.String("scenario", "", "path to consensus scenario yaml") - agentBin := flag.String("agent-bin", "", "path to a2a-agent binary") - outputJSON := flag.String("output-json", "", "write run metrics json") - outputTSV := flag.String("output-tsv", "", "append run metrics tsv") - waitReady := flag.Duration("wait-ready", 3*time.Second, "wait for agents to start") - relayGRPCPort := flag.Int("relay-grpc-port", 9600, "relay hub gRPC port") - relayCardPort := flag.Int("relay-card-port", 9601, "relay hub agent card port") - quiet := flag.Bool("quiet", false, "disable benchmark logs") - flag.Parse() - - if *quiet { - benchlog.SetEnabled(false) - } - if *scenarioPath == "" { - log.Fatal("--scenario is required") - } - - s, err := scenario.LoadFile(*scenarioPath) - if err != nil { - log.Fatalf("load scenario: %v", err) - } - - agentPath := *agentBin - if agentPath == "" { - agentPath = os.Getenv("A2A_AGENT_BIN") - } - if agentPath == "" { - log.Fatal("set --agent-bin or A2A_AGENT_BIN") - } - - // Start the relay hub first so agents can subscribe as they come up. - hub := relay.NewHub(*relayGRPCPort, *relayCardPort) - if err := hub.Serve(); err != nil { - log.Fatalf("relay serve: %v", err) - } - - procs := startAgents(s, agentPath, *scenarioPath, hub.CardURL()) - defer stopAgents(procs) - time.Sleep(*waitReady) - - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) - defer cancel() - - cli, err := newClientWithRetry(ctx, s) - if err != nil { - stopAgents(procs) - log.Fatalf("client: %v", err) - } - - agentIDs := s.AgentIDs() - agentIndexByID := map[string]int{} - for i, a := range s.Agents { - agentIndexByID[a.ID] = i - } - - // Runner subscribes to every agent's findings stream and relays each finding - // to all other agents. Retries keep the subscription alive across restarts. - for _, id := range agentIDs { - go func(agentID string) { - for { - select { - case <-ctx.Done(): - return - default: - } - err := cli.SubscribeFindings(ctx, agentID, func(f consensus.Finding) { - hub.Broadcast(f) - }) - if err == nil || ctx.Err() != nil { - return - } - time.Sleep(300 * time.Millisecond) - } - }(id) - } - - // Let finding subscriptions establish before agents start producing. - time.Sleep(500 * time.Millisecond) - - runStart := time.Now() - benchlog.SetRunStart(runStart) - if err := cli.StartAll(ctx, agentIDs); err != nil { - stopAgents(procs) - log.Fatalf("start agents: %v", err) - } - - result := metrics.RunResult{ - ScenarioName: s.Metadata.Name, - Domain: s.Metadata.Domain, - Implementation: benchlog.ImplA2A, - Agents: len(s.Agents), - ThinkTimeMs: s.Spec.ThinkTimeMs, - } - - var snapshots []consensus.AgentSnapshot - deadline := time.Now().Add(2 * time.Minute) - for time.Now().Before(deadline) { - snapshots = snapshots[:0] - var pollErr error - for _, id := range agentIDs { - snap, err := cli.Snapshot(ctx, id) - if err != nil { - pollErr = err - break - } - snapshots = append(snapshots, snap) - } - if pollErr == nil { - if ok, _ := consensus.GlobalConsensus(snapshots); ok { - result.Success = true - break - } - } - time.Sleep(20 * time.Millisecond) - } - - if !result.Success && len(snapshots) > 0 { - if ok, _ := consensus.GlobalConsensus(snapshots); ok { - result.Success = true - } - } - - result = aggregateResult(result, runStart, snapshots, hub) - - stopAgents(procs) - - if *outputJSON != "" { - if err := metrics.WriteJSON(*outputJSON, result); err != nil { - log.Fatalf("write json: %v", err) - } - } - if *outputTSV != "" { - if err := metrics.AppendTSV(*outputTSV, result); err != nil { - log.Fatalf("write tsv: %v", err) - } - } - - data, _ := json.MarshalIndent(result, "", " ") - fmt.Println(string(data)) - if !result.Success { - os.Exit(1) - } -} - -func newClientWithRetry(ctx context.Context, s *scenario.ConsensusScenario) (*client.Client, error) { - var lastErr error - for attempt := 0; attempt < 50; attempt++ { - cli, err := client.New(ctx, s) - if err == nil { - return cli, nil - } - lastErr = err - select { - case <-ctx.Done(): - return nil, ctx.Err() - case <-time.After(200 * time.Millisecond): - } - } - return nil, lastErr -} - -func aggregateResult(result metrics.RunResult, runStart time.Time, snapshots []consensus.AgentSnapshot, hub *relay.Hub) metrics.RunResult { - if len(snapshots) == 0 { - result.Error = "no agent snapshots" - result.ConsensusWallMS = time.Since(runStart).Milliseconds() - return result - } - - var ( - totalEmitted int - totalApplied int - lastConvergeMS int64 - propDurations []int64 - maxConsensusRound int - maxRound int - ) - - for _, snap := range snapshots { - totalEmitted += snap.FindingsEmitted - totalApplied += snap.FindingsApplied - if snap.ConsensusRound > maxConsensusRound { - maxConsensusRound = snap.ConsensusRound - } - if snap.Round > maxRound { - maxRound = snap.Round - } - if snap.ConvergedAtNs > 0 { - ms := (snap.ConvergedAtNs - runStart.UnixNano()) / int64(time.Millisecond) - if ms > lastConvergeMS { - lastConvergeMS = ms - } - } - if snap.AvgPropagationMs > 0 { - propDurations = append(propDurations, snap.AvgPropagationMs) - } - } - - if result.Success { - result.ConsensusWallMS = time.Since(runStart).Milliseconds() - if lastConvergeMS > 0 { - result.ConsensusWallMS = lastConvergeMS - } - } else { - result.ConsensusWallMS = time.Since(runStart).Milliseconds() - result.Error = "consensus not reached" - } - - result.ConsensusRound = maxConsensusRound - if result.ConsensusRound == 0 { - result.ConsensusRound = maxRound - } - result.FindingsEmitted = totalEmitted - result.FindingsReceivedTotal = totalApplied - result.LastAgentConvergeMS = lastConvergeMS - result.AvgPropagationMS, result.P95PropagationMS = metrics.AggregatePropagation(propDurations) - // Relay hub did all the fan-out work; report its real counters. - streamCount, fanoutMS := hub.Stats() - result.StreamRPCCount = streamCount - result.CoordFanoutMS = fanoutMS - return result -} - -func startAgents(s *scenario.ConsensusScenario, agentBin, scenarioFile, relayCardURL string) []*exec.Cmd { - var procs []*exec.Cmd - for i, agent := range s.Agents { - cmd := exec.Command( - agentBin, - "--agent-id", agent.ID, - "--grpc-port", fmt.Sprintf("%d", agent.A2APort), - "--card-port", fmt.Sprintf("%d", s.CardPort(agent)), - "--scenario-file", scenarioFile, - "--agent-index", fmt.Sprintf("%d", i), - "--relay-card-url", relayCardURL, - ) - cmd.Stdout = os.Stdout - cmd.Stderr = os.Stderr - if err := cmd.Start(); err != nil { - stopAgents(procs) - log.Fatalf("start agent %s: %v", agent.ID, err) - } - procs = append(procs, cmd) - } - return procs -} - -func stopAgents(procs []*exec.Cmd) { - for _, cmd := range procs { - if cmd.Process != nil { - _ = cmd.Process.Kill() - } - } -} diff --git a/benchmarks/slim-vs-a2a-v2/a2a/internal/client/client.go b/benchmarks/slim-vs-a2a-v2/a2a/internal/client/client.go deleted file mode 100644 index 4a9cf2fe..00000000 --- a/benchmarks/slim-vs-a2a-v2/a2a/internal/client/client.go +++ /dev/null @@ -1,160 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package client - -import ( - "context" - "encoding/json" - "fmt" - - "github.com/a2aproject/a2a-go/v2/a2a" - "github.com/a2aproject/a2a-go/v2/a2aclient" - "github.com/a2aproject/a2a-go/v2/a2aclient/agentcard" - a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" - "google.golang.org/grpc" - "google.golang.org/grpc/credentials/insecure" -) - -type Client struct { - clients map[string]*a2aclient.Client -} - -func New(ctx context.Context, s *scenario.ConsensusScenario) (*Client, error) { - clients := map[string]*a2aclient.Client{} - for _, agent := range s.Agents { - baseURL := s.CardBaseURL(agent) - card, err := agentcard.DefaultResolver.Resolve(ctx, baseURL) - if err != nil { - return nil, fmt.Errorf("resolve card for %s: %w", agent.ID, err) - } - cli, err := a2aclient.NewFromCard( - ctx, - card, - a2agrpc.WithGRPCTransport(grpc.WithTransportCredentials(insecure.NewCredentials())), - ) - if err != nil { - return nil, fmt.Errorf("client for %s: %w", agent.ID, err) - } - clients[agent.ID] = cli - } - return &Client{clients: clients}, nil -} - -func (c *Client) StartAll(ctx context.Context, agentIDs []string) error { - for _, id := range agentIDs { - if err := c.send(ctx, id, protocol.Request{Op: protocol.OpStart}); err != nil { - return err - } - } - return nil -} - -func (c *Client) Snapshot(ctx context.Context, agentID string) (consensus.AgentSnapshot, error) { - resp, err := c.sendWithResponse(ctx, agentID, protocol.Request{Op: protocol.OpSnapshot}) - if err != nil { - return consensus.AgentSnapshot{}, err - } - var snap consensus.AgentSnapshot - if err := json.Unmarshal([]byte(resp.Body), &snap); err != nil { - return consensus.AgentSnapshot{}, err - } - return snap, nil -} - -// SubscribeFindings opens the OpStreamFindings server stream to an agent and -// invokes onFinding for every finding the agent produces. It blocks until the -// stream ends or ctx is cancelled, so callers run it in a goroutine. -func (c *Client) SubscribeFindings(ctx context.Context, agentID string, onFinding func(consensus.Finding)) error { - cli, ok := c.clients[agentID] - if !ok { - return fmt.Errorf("unknown agent %q", agentID) - } - reqText, err := json.Marshal(protocol.Request{Op: protocol.OpStreamFindings}) - if err != nil { - return err - } - msg := a2a.NewMessage(a2a.MessageRoleUser, a2a.NewTextPart(string(reqText))) - for ev, streamErr := range cli.SendStreamingMessage(ctx, &a2a.SendMessageRequest{Message: msg}) { - if streamErr != nil { - return streamErr - } - f, ok := findingFromEvent(ev) - if !ok { - continue - } - onFinding(f) - } - return nil -} - -func (c *Client) send(ctx context.Context, agentID string, req protocol.Request) error { - _, err := c.sendWithResponse(ctx, agentID, req) - return err -} - -func (c *Client) sendWithResponse(ctx context.Context, agentID string, req protocol.Request) (protocol.Response, error) { - cli, ok := c.clients[agentID] - if !ok { - return protocol.Response{}, fmt.Errorf("unknown agent %q", agentID) - } - text, err := json.Marshal(req) - if err != nil { - return protocol.Response{}, err - } - msg := a2a.NewMessage(a2a.MessageRoleUser, a2a.NewTextPart(string(text))) - result, err := cli.SendMessage(ctx, &a2a.SendMessageRequest{Message: msg}) - if err != nil { - return protocol.Response{}, err - } - respText, err := responseText(result) - if err != nil { - return protocol.Response{}, err - } - return protocol.DecodeResponse(respText) -} - -func findingFromEvent(ev a2a.Event) (consensus.Finding, bool) { - su, ok := ev.(*a2a.TaskStatusUpdateEvent) - if !ok || su.Status.Message == nil { - return consensus.Finding{}, false - } - text := firstText(su.Status.Message) - if text == "" { - return consensus.Finding{}, false - } - f, err := consensus.DecodeFinding([]byte(text)) - if err != nil { - return consensus.Finding{}, false - } - return f, true -} - -func responseText(result a2a.SendMessageResult) (string, error) { - switch v := result.(type) { - case *a2a.Message: - return firstText(v), nil - case *a2a.Task: - if v.Status.Message != nil { - return firstText(v.Status.Message), nil - } - return "", fmt.Errorf("task response missing message") - default: - return "", fmt.Errorf("unexpected result type %T", result) - } -} - -func firstText(message *a2a.Message) string { - if message == nil { - return "" - } - for _, part := range message.Parts { - if text := part.Text(); text != "" { - return text - } - } - return "" -} diff --git a/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol/protocol.go b/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol/protocol.go deleted file mode 100644 index 21f53971..00000000 --- a/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol/protocol.go +++ /dev/null @@ -1,59 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -// Package protocol defines the A2A control/stream ops for the relay topology. -// -// Unlike SLIM's native group session, A2A has no peer multicast: findings must -// pass through the runner, which is the explicit relay hub. Two server-streaming -// legs carry findings: -// - OpStreamFindings: the runner subscribes to each agent (agent streams its -// own findings to the runner). -// - OpStreamRelay: each agent subscribes to the runner (runner streams every -// relayed finding back out to the agents). -// -// OpStart and OpSnapshot remain unary control calls driven by the runner. -package protocol - -import ( - "encoding/json" -) - -const ( - OpStart = "start" - OpSnapshot = "snapshot" - OpStreamFindings = "stream_findings" - OpStreamRelay = "stream_relay" -) - -type Request struct { - Op string `json:"op"` - // AgentIndex identifies the subscriber on OpStreamRelay so the relay can - // avoid echoing an agent's own findings back to it. - AgentIndex int `json:"agentIndex,omitempty"` -} - -type Response struct { - OK bool `json:"ok"` - Error string `json:"error,omitempty"` - Body string `json:"body,omitempty"` -} - -func EncodeRequest(req Request) ([]byte, error) { - return json.Marshal(req) -} - -func DecodeRequest(text string) (Request, error) { - var req Request - err := json.Unmarshal([]byte(text), &req) - return req, err -} - -func EncodeResponse(resp Response) ([]byte, error) { - return json.Marshal(resp) -} - -func DecodeResponse(text string) (Response, error) { - var resp Response - err := json.Unmarshal([]byte(text), &resp) - return resp, err -} diff --git a/benchmarks/slim-vs-a2a-v2/go.mod b/benchmarks/slim-vs-a2a-v2/go.mod deleted file mode 100644 index 0d354844..00000000 --- a/benchmarks/slim-vs-a2a-v2/go.mod +++ /dev/null @@ -1,25 +0,0 @@ -module github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2 - -go 1.25.0 - -require ( - github.com/a2aproject/a2a-go/v2 v2.3.0 - github.com/agntcy/slim-bindings-go v1.4.0 - google.golang.org/grpc v1.80.0 - gopkg.in/yaml.v3 v3.0.1 -) - -require ( - github.com/google/uuid v1.6.0 // indirect - github.com/kr/pretty v0.3.1 // indirect - github.com/rogpeppe/go-internal v1.13.1 // indirect - golang.org/x/mod v0.35.0 // indirect - golang.org/x/net v0.53.0 // indirect - golang.org/x/sync v0.20.0 // indirect - golang.org/x/sys v0.43.0 // indirect - golang.org/x/text v0.36.0 // indirect - google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4 // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 // indirect - google.golang.org/protobuf v1.36.11 // indirect - gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 // indirect -) diff --git a/benchmarks/slim-vs-a2a-v2/go.sum b/benchmarks/slim-vs-a2a-v2/go.sum deleted file mode 100644 index d2fc54cd..00000000 --- a/benchmarks/slim-vs-a2a-v2/go.sum +++ /dev/null @@ -1,62 +0,0 @@ -github.com/a2aproject/a2a-go/v2 v2.3.0 h1:wSseKBBlDYBAJLdHZ4puFIL2XyL/5YuMrDw9TXLL1ek= -github.com/a2aproject/a2a-go/v2 v2.3.0/go.mod h1:mkZr8y2bUgAVQsjs/5fHK7xrRlAHDybMEyxWh2tKRC8= -github.com/agntcy/slim-bindings-go v1.4.0 h1:DXAGhe9TWSm33KIgrJFkIcxdqh+/oPWy4Jq5PCI0HvM= -github.com/agntcy/slim-bindings-go v1.4.0/go.mod h1:XK0Ing+REEl8xG79HTMx52XzWK2THuTQA+Y7JTAn428= -github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= -github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= -github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= -github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= -github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= -github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= -github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= -github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= -github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= -github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= -github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= -github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= -github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= -github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= -github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= -github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA= -github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= -github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII= -github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o= -go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= -go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= -go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48= -go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8= -go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0= -go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs= -go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18= -go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE= -go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8= -go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew= -go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI= -go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA= -golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM= -golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU= -golang.org/x/net v0.53.0 h1:d+qAbo5L0orcWAr0a9JweQpjXF19LMXJE8Ey7hwOdUA= -golang.org/x/net v0.53.0/go.mod h1:JvMuJH7rrdiCfbeHoo3fCQU24Lf5JJwT9W3sJFulfgs= -golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= -golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= -golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI= -golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= -golang.org/x/text v0.36.0 h1:JfKh3XmcRPqZPKevfXVpI1wXPTqbkE5f7JA92a55Yxg= -golang.org/x/text v0.36.0/go.mod h1:NIdBknypM8iqVmPiuco0Dh6P5Jcdk8lJL0CUebqK164= -gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= -gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= -google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4 h1:yOzSCGPx+cp5VO7IxvZ9SBFF7j1tZVcNtlHR2iYKtVo= -google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4/go.mod h1:Q9HWtNeE7tM9npdIsEvqXj1QJIvVoeAV3rtXtS715Cw= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 h1:tEkOQcXgF6dH1G+MVKZrfpYvozGrzb91k6ha7jireSM= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= -google.golang.org/grpc v1.80.0 h1:Xr6m2WmWZLETvUNvIUmeD5OAagMw3FiKmMlTdViWsHM= -google.golang.org/grpc v1.80.0/go.mod h1:ho/dLnxwi3EDJA4Zghp7k2Ec1+c2jqup0bFkw07bwF4= -google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= -google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= -gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= -gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= -gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/benchmarks/slim-vs-a2a-v2/internal/benchlog/benchlog.go b/benchmarks/slim-vs-a2a-v2/internal/benchlog/benchlog.go deleted file mode 100644 index bdb4999d..00000000 --- a/benchmarks/slim-vs-a2a-v2/internal/benchlog/benchlog.go +++ /dev/null @@ -1,89 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package benchlog - -import ( - "fmt" - "io" - "log" - "os" - "strings" - "sync" - "time" -) - -const ( - ImplSLIM = "slim-group-session" - ImplA2A = "a2a-relay-stream" -) - -var ( - mu sync.RWMutex - runStart time.Time - enabled = true - out io.Writer = os.Stderr -) - -func SetRunStart(t time.Time) { - mu.Lock() - runStart = t - mu.Unlock() -} - -func SetEnabled(on bool) { - mu.Lock() - enabled = on - mu.Unlock() -} - -func sinceStart() int64 { - mu.RLock() - start := runStart - mu.RUnlock() - if start.IsZero() { - return 0 - } - return time.Since(start).Milliseconds() -} - -func write(kind, impl string, kv ...string) { - mu.RLock() - on := enabled - mu.RUnlock() - if !on { - return - } - ts := time.Now().UTC().Format(time.RFC3339Nano) - elapsed := sinceStart() - parts := []string{ - fmt.Sprintf("bench ts=%s elapsed_ms=%d impl=%s kind=%s", ts, elapsed, impl, kind), - } - parts = append(parts, kv...) - log.New(out, "", 0).Println(strings.Join(parts, " ")) -} - -func RPC(impl, op, mode string, duration time.Duration, ok bool, kv ...string) { - status := "ok" - if !ok { - status = "error" - } - fields := []string{ - fmt.Sprintf("op=%s", op), - fmt.Sprintf("mode=%s", mode), - fmt.Sprintf("duration_ms=%d", duration.Milliseconds()), - fmt.Sprintf("status=%s", status), - } - fields = append(fields, kv...) - write("rpc", impl, fields...) -} - -func Finding(impl, event string, agentIndex int, findingID int64, kv ...string) { - fields := []string{ - fmt.Sprintf("event=%s", event), - fmt.Sprintf("agent=%d", agentIndex), - fmt.Sprintf("finding_id=%d", findingID), - } - fields = append(fields, kv...) - write("finding", impl, fields...) -} diff --git a/benchmarks/slim-vs-a2a-v2/internal/metrics/metrics.go b/benchmarks/slim-vs-a2a-v2/internal/metrics/metrics.go deleted file mode 100644 index adc1dff2..00000000 --- a/benchmarks/slim-vs-a2a-v2/internal/metrics/metrics.go +++ /dev/null @@ -1,120 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package metrics - -import ( - "encoding/csv" - "encoding/json" - "os" - "path/filepath" - "strconv" -) - -type RunResult struct { - ScenarioName string `json:"scenario_name"` - Domain string `json:"domain"` - Implementation string `json:"implementation"` - Agents int `json:"agents"` - ThinkTimeMs int64 `json:"think_time_ms"` - ConsensusWallMS int64 `json:"consensus_wall_ms"` - ConsensusRound int `json:"consensus_round"` - FindingsEmitted int `json:"findings_emitted"` - FindingsReceivedTotal int `json:"findings_received_total"` - AvgPropagationMS int64 `json:"avg_propagation_ms"` - P95PropagationMS int64 `json:"p95_propagation_ms"` - LastAgentConvergeMS int64 `json:"last_agent_converge_ms"` - CoordFanoutMS int64 `json:"coord_fanout_ms"` - StreamRPCCount int `json:"stream_rpc_count"` - UnicastRPCCount int `json:"unicast_rpc_count"` - Success bool `json:"success"` - Error string `json:"error,omitempty"` -} - -func WriteJSON(path string, result RunResult) error { - if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { - return err - } - data, err := json.MarshalIndent(result, "", " ") - if err != nil { - return err - } - return os.WriteFile(path, data, 0o644) -} - -func AppendTSV(path string, result RunResult) error { - if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { - return err - } - writeHeader := true - if info, err := os.Stat(path); err == nil { - writeHeader = info.Size() == 0 - } else if !os.IsNotExist(err) { - return err - } - - file, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644) - if err != nil { - return err - } - defer file.Close() - - w := csv.NewWriter(file) - w.Comma = '\t' - if writeHeader { - if err := w.Write([]string{ - "scenario_name", "domain", "implementation", "agents", "think_time_ms", - "consensus_wall_ms", "consensus_round", "findings_emitted", "findings_received_total", - "avg_propagation_ms", "p95_propagation_ms", "last_agent_converge_ms", - "coord_fanout_ms", "stream_rpc_count", "unicast_rpc_count", - "success", "error", - }); err != nil { - return err - } - } - if err := w.Write([]string{ - result.ScenarioName, - result.Domain, - result.Implementation, - strconv.Itoa(result.Agents), - strconv.FormatInt(result.ThinkTimeMs, 10), - strconv.FormatInt(result.ConsensusWallMS, 10), - strconv.Itoa(result.ConsensusRound), - strconv.Itoa(result.FindingsEmitted), - strconv.Itoa(result.FindingsReceivedTotal), - strconv.FormatInt(result.AvgPropagationMS, 10), - strconv.FormatInt(result.P95PropagationMS, 10), - strconv.FormatInt(result.LastAgentConvergeMS, 10), - strconv.FormatInt(result.CoordFanoutMS, 10), - strconv.Itoa(result.StreamRPCCount), - strconv.Itoa(result.UnicastRPCCount), - strconv.FormatBool(result.Success), - result.Error, - }); err != nil { - return err - } - w.Flush() - return w.Error() -} - -func AggregatePropagation(durations []int64) (avg, p95 int64) { - if len(durations) == 0 { - return 0, 0 - } - var sum int64 - for _, d := range durations { - sum += d - } - avg = sum / int64(len(durations)) - sorted := append([]int64(nil), durations...) - for i := 0; i < len(sorted); i++ { - for j := i + 1; j < len(sorted); j++ { - if sorted[j] < sorted[i] { - sorted[i], sorted[j] = sorted[j], sorted[i] - } - } - } - idx := int(float64(len(sorted)-1) * 0.95) - p95 = sorted[idx] - return avg, p95 -} diff --git a/benchmarks/slim-vs-a2a-v2/plans/README.md b/benchmarks/slim-vs-a2a-v2/plans/README.md deleted file mode 100644 index d7cfc76b..00000000 --- a/benchmarks/slim-vs-a2a-v2/plans/README.md +++ /dev/null @@ -1,67 +0,0 @@ -# Consensus scenarios (v2) - -YAML `ConsensusScenario` plans for the SLIM vs A2A v2 consensus streaming benchmark. -Each scenario defines N agents that run a deterministic **distributed hypothesis convergence** -workload: agents think in parallel, emit findings, and must reach identical local consensus. - -## Topologies compared - -The two implementations highlight one structural difference: - -- **SLIM (`slim-group-session`)** — agents join a native SLIM group session and - broadcast findings peer-to-peer; the dataplane fans them out. **There is no - application relay.** The runner only moderates (creates the session, invites - agents, sends one start) and then passively observes snapshots that agents - push to it on convergence. -- **A2A (`a2a-relay-stream`)** — A2A has no peer multicast, so the runner is an - explicit **relay hub**. Two server-streaming legs carry findings: agents stream - findings to the runner (`OpStreamFindings`) and subscribe to the runner's - relayed-finding stream (`OpStreamRelay`). Every finding therefore makes an - extra relay hop, which is the overhead `consensus_wall_ms` is expected to show. - -## Schema - -```yaml -apiVersion: bench.agntcy.io/v2 -kind: ConsensusScenario -metadata: - name: hypothesis-convergence-10agents-10ms - domain: hypothesis-convergence -spec: - agents: 10 - thinkTimeMs: 10 - findingEmitDelayMs: 1 - maxRounds: 200 - targetMode: majority - seed: 42 - valueSpace: 3 -agents: - - id: agent-0 - slimName: agntcy/bench-v2/agent-0 - a2aPort: 9700 - role: worker # legacy field; both transports now treat all agents as peers - - id: agent-1 - slimName: agntcy/bench-v2/agent-1 - a2aPort: 9711 - role: worker -``` - -## Generate sweep scenarios - -```bash -go run ./tools/gen_scenario \ - -family hypothesis-convergence \ - -agents 10 \ - -think-ms 20 \ - -output plans/sweeps/hypothesis-convergence-10ag-20ms.yaml -``` - -## Run comparison - -```bash -task build -task compare:plan PLAN=hypothesis-convergence-5ag-20ms -task compare:report -``` - -Primary metric: `consensus_wall_ms` in `reports/results.tsv`. diff --git a/benchmarks/slim-vs-a2a-v2/slim/cmd/agent/main.go b/benchmarks/slim-vs-a2a-v2/slim/cmd/agent/main.go deleted file mode 100644 index bae30c50..00000000 --- a/benchmarks/slim-vs-a2a-v2/slim/cmd/agent/main.go +++ /dev/null @@ -1,51 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package main - -import ( - "flag" - "fmt" - "log" - "time" - - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/slim/internal/agent" -) - -func main() { - slimName := flag.String("slim-name", "", "SLIM identity org/group/app") - endpoint := flag.String("endpoint", "http://127.0.0.1:46357", "SLIM dataplane endpoint") - scenarioFile := flag.String("scenario-file", "", "path to consensus scenario yaml") - agentIndex := flag.Int("agent-index", 0, "agent index in scenario") - flag.Parse() - - if *slimName == "" || *scenarioFile == "" { - log.Fatal("--slim-name and --scenario-file are required") - } - - s, err := scenario.LoadFile(*scenarioFile) - if err != nil { - log.Fatalf("load scenario: %v", err) - } - if *agentIndex < 0 || *agentIndex >= len(s.Agents) { - log.Fatalf("agent-index out of range") - } - - rt := agent.NewRuntime(s, *agentIndex, *slimName, *endpoint) - if err := rt.Setup(); err != nil { - log.Fatalf("setup: %v", err) - } - defer rt.Close() - - fmt.Printf("SLIM_AGENT_READY name=%s index=%d scenario=%s\n", *slimName, *agentIndex, s.Metadata.Name) - - // Block until the moderator invites us into the group session. - if err := rt.Join(60 * time.Second); err != nil { - log.Fatalf("join group session: %v", err) - } - - if err := rt.Run(); err != nil { - log.Printf("receive loop ended: %v", err) - } -} diff --git a/benchmarks/slim-vs-a2a-v2/slim/cmd/runner/main.go b/benchmarks/slim-vs-a2a-v2/slim/cmd/runner/main.go deleted file mode 100644 index 3f904251..00000000 --- a/benchmarks/slim-vs-a2a-v2/slim/cmd/runner/main.go +++ /dev/null @@ -1,334 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -// Command runner is the SLIM benchmark driver. It acts as a group-session -// moderator (creates the session, invites every agent, broadcasts a single -// start) and then becomes a passive observer: it never relays findings. Agents -// broadcast findings to each other directly over the SLIM dataplane and push -// their snapshots to the runner the instant they converge. Global consensus is -// detected event-driven from those pushed snapshots. -package main - -import ( - "encoding/json" - "flag" - "fmt" - "log" - "os" - "os/exec" - "strings" - "time" - - slim "github.com/agntcy/slim-bindings-go" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/benchlog" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/metrics" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/slim/internal/protocol" -) - -const ( - runnerSlimName = "agntcy/bench-v2/runner" - groupChannelName = "agntcy/bench-v2/consensus" - defaultSharedSecret = "demo-shared-secret-min-32-chars!!" -) - -func main() { - scenarioPath := flag.String("scenario", "", "path to consensus scenario yaml") - endpoint := flag.String("endpoint", "http://127.0.0.1:46357", "SLIM dataplane endpoint") - agentBin := flag.String("agent-bin", "", "path to slim-agent binary") - outputJSON := flag.String("output-json", "", "write run metrics json") - outputTSV := flag.String("output-tsv", "", "append run metrics tsv") - waitReady := flag.Duration("wait-ready", 3*time.Second, "wait for agents to start") - quiet := flag.Bool("quiet", false, "disable benchmark logs") - flag.Parse() - - if *quiet { - benchlog.SetEnabled(false) - } - if *scenarioPath == "" { - log.Fatal("--scenario is required") - } - - s, err := scenario.LoadFile(*scenarioPath) - if err != nil { - log.Fatalf("load scenario: %v", err) - } - - agentPath := *agentBin - if agentPath == "" { - agentPath = os.Getenv("SLIM_AGENT_BIN") - } - if agentPath == "" { - log.Fatal("set --agent-bin or SLIM_AGENT_BIN") - } - - procs := startAgents(s, agentPath, *endpoint, *scenarioPath) - defer stopAgents(procs) - - mod, err := newModerator(*endpoint, s) - if err != nil { - log.Fatalf("moderator: %v", err) - } - defer mod.Close() - - // Let agents come up and reach ListenForSessionAsync before inviting. - time.Sleep(*waitReady) - if err := mod.InviteAll(); err != nil { - log.Fatalf("invite agents: %v", err) - } - - runStart := time.Now() - benchlog.SetRunStart(runStart) - if err := mod.Start(); err != nil { - log.Fatalf("broadcast start: %v", err) - } - - result := metrics.RunResult{ - ScenarioName: s.Metadata.Name, - Domain: s.Metadata.Domain, - Implementation: benchlog.ImplSLIM, - Agents: len(s.Agents), - ThinkTimeMs: s.Spec.ThinkTimeMs, - } - - latest := map[int]consensus.AgentSnapshot{} - n := len(s.Agents) - deadline := time.Now().Add(2 * time.Minute) - for time.Now().Before(deadline) { - timeout := time.Second - msg, err := mod.session.GetMessageAsync(&timeout) - if err != nil { - if isTimeout(err) { - continue - } - break - } - env, derr := protocol.Decode(msg.Payload) - if derr != nil || env.Kind != protocol.KindSnapshot || env.Snapshot == nil { - continue - } - latest[env.Snapshot.AgentIndex] = *env.Snapshot - if len(latest) == n { - if ok, _ := consensus.GlobalConsensus(snapshotSlice(latest)); ok { - result.Success = true - break - } - } - } - - snapshots := snapshotSlice(latest) - result = aggregateResult(result, runStart, snapshots) - - stopAgents(procs) - - if *outputJSON != "" { - if err := metrics.WriteJSON(*outputJSON, result); err != nil { - log.Fatalf("write json: %v", err) - } - } - if *outputTSV != "" { - if err := metrics.AppendTSV(*outputTSV, result); err != nil { - log.Fatalf("write tsv: %v", err) - } - } - - data, _ := json.MarshalIndent(result, "", " ") - fmt.Println(string(data)) - if !result.Success { - os.Exit(1) - } -} - -// moderator owns the group session: it creates it, invites agents, and -// broadcasts the start signal. It does not relay any traffic. -type moderator struct { - app *slim.App - connID uint64 - session *slim.Session - agents []scenario.Agent -} - -func newModerator(endpoint string, s *scenario.ConsensusScenario) (*moderator, error) { - slim.InitializeWithDefaults() - service := slim.GetGlobalService() - - name, err := slim.NameFromString(runnerSlimName) - if err != nil { - return nil, err - } - app, err := service.CreateAppWithSecret(name, defaultSharedSecret) - if err != nil { - return nil, err - } - connID, err := service.Connect(slim.NewInsecureClientConfig(endpoint)) - if err != nil { - app.Destroy() - return nil, err - } - if err := app.Subscribe(name, &connID); err != nil { - app.Destroy() - return nil, err - } - - channelName, err := slim.NameFromString(groupChannelName) - if err != nil { - app.Destroy() - return nil, err - } - interval := 5 * time.Second - maxRetries := uint32(5) - config := slim.SessionConfig{ - SessionType: slim.SessionTypeGroup, - MaxRetries: &maxRetries, - Interval: &interval, - Metadata: map[string]string{}, - } - session, err := app.CreateSessionAndWaitAsync(config, channelName) - if err != nil { - app.Destroy() - return nil, err - } - // Give the session a moment to establish before inviting. - time.Sleep(100 * time.Millisecond) - - return &moderator{app: app, connID: connID, session: session, agents: s.Agents}, nil -} - -func (m *moderator) InviteAll() error { - for _, a := range m.agents { - name, err := slim.NameFromString(a.SlimName) - if err != nil { - return err - } - if err := m.app.SetRouteAsync(name, m.connID); err != nil { - return fmt.Errorf("set route %s: %w", a.SlimName, err) - } - if err := m.session.InviteAndWaitAsync(name); err != nil { - return fmt.Errorf("invite %s: %w", a.SlimName, err) - } - } - return nil -} - -func (m *moderator) Start() error { - payload, err := protocol.Encode(protocol.Envelope{Kind: protocol.KindStart}) - if err != nil { - return err - } - return m.session.PublishAndWaitAsync(payload, nil, nil) -} - -func (m *moderator) Close() { - if m.session != nil && m.app != nil { - _ = m.app.DeleteSessionAndWaitAsync(m.session) - } - if m.app != nil { - m.app.Destroy() - } -} - -func aggregateResult(result metrics.RunResult, runStart time.Time, snapshots []consensus.AgentSnapshot) metrics.RunResult { - if len(snapshots) == 0 { - result.Error = "no agent snapshots" - result.ConsensusWallMS = time.Since(runStart).Milliseconds() - return result - } - - var ( - totalEmitted int - totalApplied int - lastConvergeMS int64 - propDurations []int64 - maxConsensusRound int - maxRound int - ) - - for _, snap := range snapshots { - totalEmitted += snap.FindingsEmitted - totalApplied += snap.FindingsApplied - if snap.ConsensusRound > maxConsensusRound { - maxConsensusRound = snap.ConsensusRound - } - if snap.Round > maxRound { - maxRound = snap.Round - } - if snap.ConvergedAtNs > 0 { - ms := (snap.ConvergedAtNs - runStart.UnixNano()) / int64(time.Millisecond) - if ms > lastConvergeMS { - lastConvergeMS = ms - } - } - if snap.AvgPropagationMs > 0 { - propDurations = append(propDurations, snap.AvgPropagationMs) - } - } - - if result.Success { - result.ConsensusWallMS = time.Since(runStart).Milliseconds() - if lastConvergeMS > 0 { - result.ConsensusWallMS = lastConvergeMS - } - } else { - result.ConsensusWallMS = time.Since(runStart).Milliseconds() - result.Error = "consensus not reached" - } - - result.ConsensusRound = maxConsensusRound - if result.ConsensusRound == 0 { - result.ConsensusRound = maxRound - } - result.FindingsEmitted = totalEmitted - result.FindingsReceivedTotal = totalApplied - result.LastAgentConvergeMS = lastConvergeMS - result.AvgPropagationMS, result.P95PropagationMS = metrics.AggregatePropagation(propDurations) - // Native group session: findings are broadcast peer-to-peer, no relay. - result.StreamRPCCount = totalEmitted - result.CoordFanoutMS = 0 - return result -} - -func snapshotSlice(m map[int]consensus.AgentSnapshot) []consensus.AgentSnapshot { - out := make([]consensus.AgentSnapshot, 0, len(m)) - for _, snap := range m { - out = append(out, snap) - } - return out -} - -func startAgents(s *scenario.ConsensusScenario, agentBin, endpoint, scenarioFile string) []*exec.Cmd { - var procs []*exec.Cmd - for i, agent := range s.Agents { - cmd := exec.Command( - agentBin, - "--slim-name", agent.SlimName, - "--endpoint", endpoint, - "--scenario-file", scenarioFile, - "--agent-index", fmt.Sprintf("%d", i), - ) - cmd.Stdout = os.Stdout - cmd.Stderr = os.Stderr - if err := cmd.Start(); err != nil { - stopAgents(procs) - log.Fatalf("start agent %s: %v", agent.ID, err) - } - procs = append(procs, cmd) - } - return procs -} - -func stopAgents(procs []*exec.Cmd) { - for _, cmd := range procs { - if cmd.Process != nil { - _ = cmd.Process.Kill() - } - } -} - -func isTimeout(err error) bool { - if err == nil { - return false - } - msg := strings.ToLower(err.Error()) - return strings.Contains(msg, "timed out") || strings.Contains(msg, "timeout") -} diff --git a/benchmarks/slim-vs-a2a-v2/slim/internal/protocol/protocol.go b/benchmarks/slim-vs-a2a-v2/slim/internal/protocol/protocol.go deleted file mode 100644 index 42a48fa4..00000000 --- a/benchmarks/slim-vs-a2a-v2/slim/internal/protocol/protocol.go +++ /dev/null @@ -1,41 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -// Package protocol defines the message envelope exchanged over the native SLIM -// group session. There is no relay and no request/response RPC: the runner -// broadcasts a single start, agents broadcast findings to all peers, and agents -// push their snapshots to the runner the instant they converge. -package protocol - -import ( - "encoding/json" - - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" -) - -const ( - // KindStart is broadcast once by the runner to define t0 and unblock agents. - KindStart = "start" - // KindFinding is broadcast by an agent to all group peers. - KindFinding = "finding" - // KindSnapshot is pushed by an agent directly to the runner on convergence. - KindSnapshot = "snapshot" -) - -// Envelope is the single JSON message type carried on the group session. -type Envelope struct { - Kind string `json:"kind"` - AgentIndex int `json:"agentIndex,omitempty"` - Finding *consensus.Finding `json:"finding,omitempty"` - Snapshot *consensus.AgentSnapshot `json:"snapshot,omitempty"` -} - -func Encode(e Envelope) ([]byte, error) { - return json.Marshal(e) -} - -func Decode(data []byte) (Envelope, error) { - var e Envelope - err := json.Unmarshal(data, &e) - return e, err -} diff --git a/benchmarks/slim-vs-a2a-v2/tools/report/main.go b/benchmarks/slim-vs-a2a-v2/tools/report/main.go deleted file mode 100644 index 7bd51e25..00000000 --- a/benchmarks/slim-vs-a2a-v2/tools/report/main.go +++ /dev/null @@ -1,207 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package main - -import ( - "encoding/csv" - "flag" - "fmt" - "html/template" - "os" - "path/filepath" - "strconv" - - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/metrics" -) - -type scenarioComparison struct { - ScenarioName string - A2A metrics.RunResult - SLIM metrics.RunResult - HasA2A bool - HasSLIM bool -} - -func main() { - tsvPath := flag.String("tsv", "./reports/results.tsv", "comparison results tsv") - sweepTSV := flag.String("sweep-tsv", "./reports/sweep.tsv", "optional sweep results tsv") - output := flag.String("output", "./reports/index.html", "html dashboard output") - flag.Parse() - - results, err := readTSV(*tsvPath) - if err != nil { - fmt.Fprintf(os.Stderr, "read tsv: %v\n", err) - os.Exit(1) - } - comparisons := groupByScenario(results) - - var sweepResults []metrics.RunResult - if *sweepTSV != "" { - if sr, err := readTSV(*sweepTSV); err == nil { - sweepResults = sr - } - } - - if err := writeHTML(*output, comparisons, sweepResults); err != nil { - fmt.Fprintf(os.Stderr, "write html: %v\n", err) - os.Exit(1) - } - fmt.Printf("wrote %s\n", *output) -} - -func readTSV(path string) ([]metrics.RunResult, error) { - file, err := os.Open(path) - if err != nil { - return nil, err - } - defer file.Close() - - reader := csv.NewReader(file) - reader.Comma = '\t' - records, err := reader.ReadAll() - if err != nil { - return nil, err - } - if len(records) <= 1 { - return nil, fmt.Errorf("no data rows in %s", path) - } - - var out []metrics.RunResult - for _, row := range records[1:] { - if len(row) < 16 { - continue - } - r := metrics.RunResult{ - ScenarioName: row[0], - Domain: row[1], - Implementation: row[2], - Error: row[len(row)-1], - } - r.Agents = atoi(row[3]) - r.ThinkTimeMs = atoi64(row[4]) - r.ConsensusWallMS = atoi64(row[5]) - r.ConsensusRound = atoi(row[6]) - r.FindingsEmitted = atoi(row[7]) - r.FindingsReceivedTotal = atoi(row[8]) - r.AvgPropagationMS = atoi64(row[9]) - r.P95PropagationMS = atoi64(row[10]) - r.LastAgentConvergeMS = atoi64(row[11]) - r.CoordFanoutMS = atoi64(row[12]) - r.StreamRPCCount = atoi(row[13]) - r.UnicastRPCCount = atoi(row[14]) - r.Success = row[15] == "true" - if len(row) > 16 { - r.Error = row[16] - } - out = append(out, r) - } - return out, nil -} - -func groupByScenario(results []metrics.RunResult) []scenarioComparison { - byScenario := map[string]*scenarioComparison{} - for _, r := range results { - sc, ok := byScenario[r.ScenarioName] - if !ok { - sc = &scenarioComparison{ScenarioName: r.ScenarioName} - byScenario[r.ScenarioName] = sc - } - switch r.Implementation { - case "a2a-relay-stream": - sc.A2A = r - sc.HasA2A = true - case "slim-group-session": - sc.SLIM = r - sc.HasSLIM = true - } - } - out := make([]scenarioComparison, 0, len(byScenario)) - for _, sc := range byScenario { - out = append(out, *sc) - } - return out -} - -type reportData struct { - Comparisons []scenarioComparison - Sweep []metrics.RunResult -} - -func writeHTML(path string, comparisons []scenarioComparison, sweep []metrics.RunResult) error { - if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { - return err - } - tmpl := template.Must(template.New("report").Funcs(template.FuncMap{ - "deltaPct": deltaPct, - }).Parse(htmlTemplate)) - file, err := os.Create(path) - if err != nil { - return err - } - defer file.Close() - return tmpl.Execute(file, reportData{Comparisons: comparisons, Sweep: sweep}) -} - -func deltaPct(a2a, slim int64) string { - if a2a == 0 { - return "n/a" - } - pct := (float64(a2a-slim) / float64(a2a)) * 100 - return fmt.Sprintf("%.1f%%", pct) -} - -func atoi(v string) int { - n, _ := strconv.Atoi(v) - return n -} - -func atoi64(v string) int64 { - n, _ := strconv.ParseInt(v, 10, 64) - return n -} - -const htmlTemplate = ` - - - - SLIM vs A2A v2 Consensus - - - -

SLIM vs A2A v2 — Consensus Streaming

-

Delta columns show (A2A − SLIM) / A2A. Positive values mean SLIM reached consensus faster.

- {{range .Comparisons}} -

{{.ScenarioName}}

- - - - - - - - - -
MetricA2ASLIMDelta
Consensus wall (ms){{if .HasA2A}}{{.A2A.ConsensusWallMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.ConsensusWallMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.ConsensusWallMS .SLIM.ConsensusWallMS}}{{else}}—{{end}}
Last agent converge (ms){{if .HasA2A}}{{.A2A.LastAgentConvergeMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.LastAgentConvergeMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.LastAgentConvergeMS .SLIM.LastAgentConvergeMS}}{{else}}—{{end}}
Avg propagation (ms){{if .HasA2A}}{{.A2A.AvgPropagationMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.AvgPropagationMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.AvgPropagationMS .SLIM.AvgPropagationMS}}{{else}}—{{end}}
P95 propagation (ms){{if .HasA2A}}{{.A2A.P95PropagationMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.P95PropagationMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.P95PropagationMS .SLIM.P95PropagationMS}}{{else}}—{{end}}
Stream RPC count{{if .HasA2A}}{{.A2A.StreamRPCCount}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.StreamRPCCount}}{{else}}—{{end}}—
Unicast RPC count{{if .HasA2A}}{{.A2A.UnicastRPCCount}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.UnicastRPCCount}}{{else}}—{{end}}—
Success{{if .HasA2A}}{{.A2A.Success}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.Success}}{{else}}—{{end}}—
- {{end}} - {{if .Sweep}} -

Sweep results

- - - {{range .Sweep}} - - - - - {{end}} -
ScenarioImplAgentsThink msConsensus wallP95 propagationStream RPCs
{{.ScenarioName}}{{.Implementation}}{{.Agents}}{{.ThinkTimeMs}}{{.ConsensusWallMS}}{{.P95PropagationMS}}{{.StreamRPCCount}}
- {{end}} - - -` diff --git a/benchmarks/slim-vs-a2a/Taskfile.yml b/benchmarks/slim-vs-a2a/Taskfile.yml index fd76e14c..57fa8929 100644 --- a/benchmarks/slim-vs-a2a/Taskfile.yml +++ b/benchmarks/slim-vs-a2a/Taskfile.yml @@ -6,25 +6,16 @@ version: '3' silent: true -includes: - slim: - taskfile: ../agntcy-slim/Taskfile.yml - dir: ../agntcy-slim - excludes: [default] - vars: - PLAN_DIR: '{{ .PLAN_DIR | default "./plans/domains" }}' + SCENARIO_DIR: '{{ .SCENARIO_DIR | default "./plans/sweeps" }}' REPORTS_DIR: '{{ .REPORTS_DIR | default "./reports" }}' RESULTS_TSV: '{{ .RESULTS_TSV | default "./reports/results.tsv" }}' SWEEP_TSV: '{{ .SWEEP_TSV | default "./reports/sweep.tsv" }}' BIN_DIR: '{{ .BIN_DIR | default "./bin" }}' SLIM_ENDPOINT: '{{ .SLIM_ENDPOINT | default "http://127.0.0.1:46357" }}' - # Use a dedicated name: agntcy-slim's included Taskfile also defines SLIMCTL_PATH=./bin/slimctl - # and would otherwise resolve to slim-vs-a2a/bin/slimctl when tasks run in this directory. COMPARE_SLIMCTL: '{{ .COMPARE_SLIMCTL | default "../agntcy-slim/bin/slimctl" }}' COMPARE_SLIM_BINDINGS_VERSION: '{{ .COMPARE_SLIM_BINDINGS_VERSION | default "v1.4.0" }}' COMPARE_SLIMCTL_TAG: '{{ .COMPARE_SLIMCTL_TAG | default "slimctl-v1.4.0" }}' - IMPLEMENTATIONS: '{{ .IMPLEMENTATIONS | default "a2a,slim" }}' tasks: default: @@ -78,10 +69,10 @@ tasks: go build -o {{ .BIN_DIR }}/slim-agent ./slim/cmd/agent go build -o {{ .BIN_DIR }}/slim-runner ./slim/cmd/runner - validate:plans: - desc: Validate all domain execution plans + validate:scenarios: + desc: Validate consensus scenario YAML files cmds: - - go run ./tools/validate_plans --dir {{ .PLAN_DIR }} + - go run ./tools/validate_scenarios --dir {{ .SCENARIO_DIR }} compare:cleanup: internal: true @@ -91,38 +82,32 @@ tasks: - pkill -f 'slimctl slim start -c' 2>/dev/null || true compare:suite: - desc: Run all domain plans on A2A and SLIM and collect comparison metrics + desc: Run default consensus scenarios on A2A and SLIM deps: - build - - validate:plans + - validate:scenarios cmds: - task: compare:cleanup - rm -f {{ .RESULTS_TSV }} - | set -euo pipefail - for plan in {{ .PLAN_DIR }}/*.yaml; do - [ -f "$plan" ] || continue - base=$(basename "$plan" .yaml) + for scenario in {{ .SCENARIO_DIR }}/hypothesis-convergence-*.yaml; do + [ -f "$scenario" ] || continue + base=$(basename "$scenario" .yaml) echo "==> A2A $base" {{ .BIN_DIR }}/a2a-runner \ - --plan "$plan" \ + --scenario "$scenario" \ --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --quiet \ --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ - --output-tsv {{ .RESULTS_TSV }} + --output-tsv {{ .RESULTS_TSV }} || true done - - task: compare:suite:slim - - compare:suite:slim: - internal: true - deps: - - deps:slimctl-download - cmds: - task: compare:slim-stack vars: - PLAN_GLOB: '{{ .PLAN_DIR }}/*.yaml' + SCENARIO_GLOB: '{{ .SCENARIO_DIR }}/hypothesis-convergence-*.yaml' compare:plan: - desc: Run one plan on A2A and SLIM (PLAN=mobile-nav-assistant or path to yaml) + desc: Run one scenario on A2A and SLIM (PLAN=path or basename) deps: - build requires: @@ -134,25 +119,23 @@ tasks: set -euo pipefail PLAN_INPUT="{{ .PLAN }}" if [ -f "$PLAN_INPUT" ]; then - PLAN_PATH="$PLAN_INPUT" - elif [ -f "{{ .PLAN_DIR }}/${PLAN_INPUT}.yaml" ]; then - PLAN_PATH="{{ .PLAN_DIR }}/${PLAN_INPUT}.yaml" - elif [ -f "{{ .PLAN_DIR }}/${PLAN_INPUT}" ]; then - PLAN_PATH="{{ .PLAN_DIR }}/${PLAN_INPUT}" + SCENARIO_PATH="$PLAN_INPUT" + elif [ -f "{{ .SCENARIO_DIR }}/${PLAN_INPUT}.yaml" ]; then + SCENARIO_PATH="{{ .SCENARIO_DIR }}/${PLAN_INPUT}.yaml" else - echo "plan not found: ${PLAN_INPUT} (tried file, {{ .PLAN_DIR }}/\${PLAN_INPUT}.yaml)" >&2 + echo "scenario not found: ${PLAN_INPUT}" >&2 exit 1 fi - base=$(basename "$PLAN_PATH" .yaml) + base=$(basename "$SCENARIO_PATH" .yaml) echo "==> A2A ${base}" {{ .BIN_DIR }}/a2a-runner \ - --plan "$PLAN_PATH" \ + --scenario "$SCENARIO_PATH" \ --agent-bin {{ .BIN_DIR }}/a2a-agent \ --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ --output-tsv {{ .RESULTS_TSV }} - task: compare:slim-stack vars: - PLAN_GLOB: '{{ .PLAN }}' + SCENARIO_GLOB: '{{ .PLAN }}' compare:slim-stack: internal: true @@ -161,19 +144,15 @@ tasks: cmds: - | set -euo pipefail - PLAN_INPUT="{{ .PLAN_GLOB }}" - resolve_plan() { + SCENARIO_INPUT="{{ .SCENARIO_GLOB }}" + resolve_scenario() { local input="$1" if [ -f "$input" ]; then printf '%s' "$input" return fi - if [ -f "{{ .PLAN_DIR }}/${input}.yaml" ]; then - printf '%s' "{{ .PLAN_DIR }}/${input}.yaml" - return - fi - if [ -f "{{ .PLAN_DIR }}/${input}" ]; then - printf '%s' "{{ .PLAN_DIR }}/${input}" + if [ -f "{{ .SCENARIO_DIR }}/${input}.yaml" ]; then + printf '%s' "{{ .SCENARIO_DIR }}/${input}.yaml" return fi printf '%s' "$input" @@ -185,8 +164,8 @@ tasks: cat > "$CONFIG" < SLIM ${base}" {{ .BIN_DIR }}/slim-runner \ - --plan "$plan" \ + --scenario "$scenario" \ --endpoint {{ .SLIM_ENDPOINT }} \ --agent-bin {{ .BIN_DIR }}/slim-agent \ - --coord-mode {{ .COORD_MODE | default "multicast" }} \ - --round-budget-ms {{ .ROUND_BUDGET_MS | default "0" }} \ + --quiet \ --output-json {{ .REPORTS_DIR }}/slim-${base}.json \ - --output-tsv {{ .RESULTS_TSV }} + --output-tsv {{ .RESULTS_TSV }} || true done compare:report: @@ -235,66 +213,103 @@ tasks: - go run ./tools/report --tsv {{ .RESULTS_TSV }} --sweep-tsv {{ .SWEEP_TSV }} --output {{ .REPORTS_DIR }}/index.html compare:sweep: - desc: Run transport sweep (SWEEP_FAMILY, SWEEP_AGENTS, SWEEP_BUDGETS env lists) + desc: Run agent/think-time/payload sweep (SWEEP_AGENTS, SWEEP_THINK_MS, SWEEP_PAYLOAD_BYTES env lists) deps: - build vars: - SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "sustainable-resource" }}' + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' - SWEEP_BUDGETS: '{{ .SWEEP_BUDGETS | default "10,20" }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES | default "0" }}' cmds: - task: compare:cleanup - rm -f {{ .SWEEP_TSV }} - task: compare:sweep:run + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES }}' + + compare:sweep:payload: + desc: High-N / short-window / large-payload sweep highlighting SLIM multicast (10kB default) + deps: + - build + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "10,50" }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "5" }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES | default "0,10240" }}' + cmds: + - task: compare:cleanup + - rm -f {{ .SWEEP_TSV }} + - task: compare:sweep:run + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES }}' + - task: compare:report compare:sweep:run: internal: true - deps: - - deps:slimctl-download vars: - SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "sustainable-resource" }}' + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' - SWEEP_BUDGETS: '{{ .SWEEP_BUDGETS | default "10,20" }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES | default "0" }}' cmds: - | set -euo pipefail - mkdir -p {{ .REPORTS_DIR }} plans/sweeps + mkdir -p {{ .REPORTS_DIR }} {{ .SCENARIO_DIR }} IFS=',' read -r -a AGENT_LIST <<< "{{ .SWEEP_AGENTS }}" - IFS=',' read -r -a BUDGET_LIST <<< "{{ .SWEEP_BUDGETS }}" + IFS=',' read -r -a THINK_LIST <<< "{{ .SWEEP_THINK_MS }}" + IFS=',' read -r -a PAYLOAD_LIST <<< "{{ .SWEEP_PAYLOAD_BYTES }}" for agents in "${AGENT_LIST[@]}"; do - for budget in "${BUDGET_LIST[@]}"; do - plan_path="plans/sweeps/{{ .SWEEP_FAMILY }}-${agents}ag-${budget}ms.yaml" - go run ./tools/gen_plan \ - -family {{ .SWEEP_FAMILY }} \ - -agents "$agents" \ - -round-budget-ms "$budget" \ - -output "$plan_path" - base=$(basename "$plan_path" .yaml) - echo "==> sweep A2A $base" - {{ .BIN_DIR }}/a2a-runner \ - --plan "$plan_path" \ - --agent-bin {{ .BIN_DIR }}/a2a-agent \ - --round-budget-ms "$budget" \ - --quiet \ - --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ - --output-tsv {{ .SWEEP_TSV }} + for think in "${THINK_LIST[@]}"; do + for payload in "${PAYLOAD_LIST[@]}"; do + suffix="" + if [ "$payload" != "0" ]; then suffix="-${payload}b"; fi + scenario_path="{{ .SCENARIO_DIR }}/{{ .SWEEP_FAMILY }}-${agents}ag-${think}ms${suffix}.yaml" + go run ./tools/gen_scenario \ + -family {{ .SWEEP_FAMILY }} \ + -agents "$agents" \ + -think-ms "$think" \ + -payload-bytes "$payload" \ + -output "$scenario_path" + base=$(basename "$scenario_path" .yaml) + echo "==> sweep A2A $base" + {{ .BIN_DIR }}/a2a-runner \ + --scenario "$scenario_path" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ + --output-tsv {{ .SWEEP_TSV }} || true + done done done - task: compare:sweep:slim + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES }}' compare:sweep:slim: internal: true deps: - deps:slimctl-download vars: - SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "sustainable-resource" }}' + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' - SWEEP_BUDGETS: '{{ .SWEEP_BUDGETS | default "10,20" }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES | default "0" }}' cmds: - | set -euo pipefail IFS=',' read -r -a AGENT_LIST <<< "{{ .SWEEP_AGENTS }}" - IFS=',' read -r -a BUDGET_LIST <<< "{{ .SWEEP_BUDGETS }}" + IFS=',' read -r -a THINK_LIST <<< "{{ .SWEEP_THINK_MS }}" + IFS=',' read -r -a PAYLOAD_LIST <<< "{{ .SWEEP_PAYLOAD_BYTES }}" DATAPLANE_PORT=46357 CONTROLLER_PORT=46358 CONFIG=$(mktemp) @@ -302,8 +317,8 @@ tasks: cat > "$CONFIG" < sweep SLIM $base" - {{ .BIN_DIR }}/slim-runner \ - --plan "$plan_path" \ - --endpoint {{ .SLIM_ENDPOINT }} \ - --agent-bin {{ .BIN_DIR }}/slim-agent \ - --coord-mode multicast \ - --round-budget-ms "$budget" \ - --quiet \ - --output-json {{ .REPORTS_DIR }}/slim-${base}.json \ - --output-tsv {{ .SWEEP_TSV }} + for think in "${THINK_LIST[@]}"; do + for payload in "${PAYLOAD_LIST[@]}"; do + suffix="" + if [ "$payload" != "0" ]; then suffix="-${payload}b"; fi + scenario_path="{{ .SCENARIO_DIR }}/{{ .SWEEP_FAMILY }}-${agents}ag-${think}ms${suffix}.yaml" + [ -f "$scenario_path" ] || continue + base=$(basename "$scenario_path" .yaml) + echo "==> sweep SLIM $base" + {{ .BIN_DIR }}/slim-runner \ + --scenario "$scenario_path" \ + --endpoint {{ .SLIM_ENDPOINT }} \ + --agent-bin {{ .BIN_DIR }}/slim-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/slim-${base}.json \ + --output-tsv {{ .SWEEP_TSV }} || true + done done done compare:ci:smoke: - desc: CI smoke — domain plans plus one sweep point (~2 min) + desc: CI smoke — one sweep point plus report (~2 min) deps: - build - - validate:plans cmds: - task: compare:cleanup - rm -f {{ .RESULTS_TSV }} {{ .SWEEP_TSV }} - | set -euo pipefail - for plan in {{ .PLAN_DIR }}/*.yaml; do - [ -f "$plan" ] || continue - base=$(basename "$plan" .yaml) - echo "==> smoke A2A $base" - {{ .BIN_DIR }}/a2a-runner \ - --plan "$plan" \ - --agent-bin {{ .BIN_DIR }}/a2a-agent \ - --quiet \ - --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ - --output-tsv {{ .RESULTS_TSV }} - done - - task: compare:slim-stack - vars: - PLAN_GLOB: '{{ .PLAN_DIR }}/*.yaml' - COORD_MODE: multicast - - | - set -euo pipefail - plan_path="plans/sweeps/ci-smoke-sustainable-10ag-20ms.yaml" - mkdir -p plans/sweeps - go run ./tools/gen_plan -family sustainable-resource -agents 10 -round-budget-ms 20 -output "$plan_path" - echo "==> smoke sweep A2A" + mkdir -p {{ .SCENARIO_DIR }} {{ .REPORTS_DIR }} + scenario_path="{{ .SCENARIO_DIR }}/ci-smoke-hypothesis-5ag-20ms.yaml" + go run ./tools/gen_scenario -family hypothesis-convergence -agents 5 -think-ms 20 -output "$scenario_path" + echo "==> smoke A2A" {{ .BIN_DIR }}/a2a-runner \ - --plan "$plan_path" \ + --scenario "$scenario_path" \ --agent-bin {{ .BIN_DIR }}/a2a-agent \ - --round-budget-ms 20 \ --quiet \ - --output-json {{ .REPORTS_DIR }}/a2a-ci-sweep.json \ - --output-tsv {{ .SWEEP_TSV }} + --output-json {{ .REPORTS_DIR }}/a2a-ci-smoke.json \ + --output-tsv {{ .RESULTS_TSV }} - task: compare:slim-stack vars: - PLAN_GLOB: 'plans/sweeps/ci-smoke-sustainable-10ag-20ms.yaml' - ROUND_BUDGET_MS: '20' - COORD_MODE: multicast + SCENARIO_GLOB: 'plans/sweeps/ci-smoke-hypothesis-5ag-20ms.yaml' - task: compare:report test: - desc: Run Ginkgo comparison suite (requires RUN_SLIM_VS_A2A=1) - deps: - - build + desc: Run unit tests cmds: - - go test ./tests -v -failfast -test.paniconexit0 -ginkgo.v -ginkgo.label-filter=slim-vs-a2a -ginkgo.timeout 30m -timeout 30m - - task: compare:report + - go test ./... -count=1 diff --git a/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go b/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go index 99c0c27c..9cc5a187 100644 --- a/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go +++ b/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go @@ -17,32 +17,36 @@ import ( a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" "github.com/a2aproject/a2a-go/v2/a2asrv" "github.com/a2aproject/a2a-go/v2/a2asrv/taskstore" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/executor" + agentrt "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/agent" "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" "google.golang.org/grpc" ) -type dagExecutor struct { - engine *executor.Engine +type consensusExecutor struct { + runtime *agentrt.Runtime } -func (d *dagExecutor) Execute(ctx context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { +func (e *consensusExecutor) Execute(ctx context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { reqText := firstText(execCtx.Message) - req, err := protocol.DecodeRequest(reqText) + req, err := agentrt.DecodeRequestText(reqText) if err != nil { return func(yield func(a2a.Event, error) bool) { yield(nil, fmt.Errorf("decode request: %w", err)) } } - resp := d.engine.Handle(ctx, req) + if req.Op == protocol.OpStreamFindings { + return e.runtime.StreamFindings(ctx, execCtx) + } + + resp := e.runtime.HandleUnary(ctx, req) body, err := json.Marshal(resp) if err != nil { return func(yield func(a2a.Event, error) bool) { yield(nil, err) } } - return func(yield func(a2a.Event, error) bool) { task := a2a.NewSubmittedTask(execCtx, execCtx.Message) state := a2a.TaskStateCompleted @@ -61,20 +65,11 @@ func (d *dagExecutor) Execute(ctx context.Context, execCtx *a2asrv.ExecutorConte } } -func (d *dagExecutor) Cancel(_ context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { +func (e *consensusExecutor) Cancel(_ context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { return func(yield func(a2a.Event, error) bool) { - yield( - a2a.NewStatusUpdateEvent( - execCtx, - a2a.TaskStateCanceled, - a2a.NewMessageForTask( - a2a.MessageRoleAgent, - execCtx, - a2a.NewTextPart(`{"ok":true}`), - ), - ), - nil, - ) + yield(a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateCanceled, a2a.NewMessageForTask( + a2a.MessageRoleAgent, execCtx, a2a.NewTextPart(`{"ok":true}`), + )), nil) } } @@ -94,38 +89,47 @@ func main() { agentID := flag.String("agent-id", "", "logical agent id") grpcPort := flag.Int("grpc-port", 0, "A2A gRPC port") cardPort := flag.Int("card-port", 0, "agent card HTTP port") + scenarioFile := flag.String("scenario-file", "", "consensus scenario yaml") + agentIndex := flag.Int("agent-index", 0, "agent index") + relayCardURL := flag.String("relay-card-url", "", "runner relay agent card base URL") flag.Parse() - if *agentID == "" || *grpcPort == 0 || *cardPort == 0 { - log.Fatal("agent-id, grpc-port, and card-port are required") + if *agentID == "" || *grpcPort == 0 || *scenarioFile == "" || *relayCardURL == "" { + log.Fatal("agent-id, grpc-port, scenario-file, and relay-card-url are required") + } + card := *cardPort + if card == 0 { + card = *grpcPort + 1000 + } + + s, err := scenario.LoadFile(*scenarioFile) + if err != nil { + log.Fatalf("load scenario: %v", err) } + rt := agentrt.NewRuntime(s, *agentIndex, *agentID, *relayCardURL) + defer rt.Close() + handler := a2asrv.NewHandler( - &dagExecutor{engine: executor.New()}, + &consensusExecutor{runtime: rt}, a2asrv.WithTaskStore(taskstore.NewInMemory(&taskstore.InMemoryStoreConfig{ Authenticator: func(context.Context) (string, error) { return "slim-vs-a2a", nil }, })), ) - card := &a2a.AgentCard{ - Name: fmt.Sprintf("slim-vs-a2a-%s", *agentID), - Description: "SLIM vs A2A comparison agent", + agentCard := &a2a.AgentCard{ + Name: fmt.Sprintf("consensus-v2-%s", *agentID), + Description: "SLIM vs A2A v2 consensus agent", Version: "1.0.0", SupportedInterfaces: []*a2a.AgentInterface{ - a2a.NewAgentInterface( - fmt.Sprintf("127.0.0.1:%d", *grpcPort), - a2a.TransportProtocolGRPC, - ), + a2a.NewAgentInterface(fmt.Sprintf("127.0.0.1:%d", *grpcPort), a2a.TransportProtocolGRPC), }, - Capabilities: a2a.AgentCapabilities{Streaming: false}, + Capabilities: a2a.AgentCapabilities{Streaming: true}, DefaultInputModes: []string{"text/plain"}, DefaultOutputModes: []string{"text/plain"}, - Skills: []a2a.AgentSkill{ - {ID: "dag-task", Name: "DAG task execution", Description: "Mock DAG task worker"}, - }, } - cardListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", *cardPort)) + cardListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", card)) if err != nil { log.Fatalf("card listen: %v", err) } @@ -140,13 +144,17 @@ func main() { go func() { mux := http.NewServeMux() - mux.Handle(a2asrv.WellKnownAgentCardPath, a2asrv.NewStaticAgentCardHandler(card)) + mux.Handle(a2asrv.WellKnownAgentCardPath, a2asrv.NewStaticAgentCardHandler(agentCard)) if err := http.Serve(cardListener, mux); err != nil { log.Printf("card server stopped: %v", err) } }() - fmt.Printf("A2A_AGENT_READY agent=%s grpc=%d card=%d\n", *agentID, *grpcPort, *cardPort) + // Subscribe to the runner relay stream as soon as we are up; it retries + // until the relay server is reachable. + rt.StartRelaySubscription(context.Background()) + + fmt.Printf("A2A_AGENT_READY agent=%s grpc=%d card=%d\n", *agentID, *grpcPort, card) if err := grpcServer.Serve(grpcListener); err != nil { log.Fatalf("grpc serve: %v", err) } diff --git a/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go b/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go index 2468b430..97cfe625 100644 --- a/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go +++ b/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go @@ -1,6 +1,11 @@ // Copyright AGNTCY Contributors (https://github.com/agntcy) // SPDX-License-Identifier: Apache-2.0 +// Command runner is the A2A benchmark driver and relay hub. Because A2A has no +// peer multicast, the runner is the explicit relay: it subscribes to every +// agent's findings stream (OpStreamFindings) and fans each finding out to all +// other agents over the relay stream (OpStreamRelay). It also drives start and +// polls snapshots to detect global consensus. package main import ( @@ -14,33 +19,34 @@ import ( "time" "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/client" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/scheduler" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/relay" "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" ) func main() { - planPath := flag.String("plan", "", "path to execution plan yaml") - agentBin := flag.String("agent-bin", "", "path to a2a agent binary") + scenarioPath := flag.String("scenario", "", "path to consensus scenario yaml") + agentBin := flag.String("agent-bin", "", "path to a2a-agent binary") outputJSON := flag.String("output-json", "", "write run metrics json") outputTSV := flag.String("output-tsv", "", "append run metrics tsv") waitReady := flag.Duration("wait-ready", 3*time.Second, "wait for agents to start") - roundBudgetMS := flag.Int64("round-budget-ms", 0, "coordination round budget in ms (overrides plan)") - quiet := flag.Bool("quiet", false, "disable benchmark timing logs") + relayGRPCPort := flag.Int("relay-grpc-port", 9600, "relay hub gRPC port") + relayCardPort := flag.Int("relay-card-port", 9601, "relay hub agent card port") + quiet := flag.Bool("quiet", false, "disable benchmark logs") flag.Parse() if *quiet { benchlog.SetEnabled(false) } - - if *planPath == "" { - log.Fatal("--plan is required") + if *scenarioPath == "" { + log.Fatal("--scenario is required") } - p, err := plan.LoadFile(*planPath) + s, err := scenario.LoadFile(*scenarioPath) if err != nil { - log.Fatalf("load plan: %v", err) + log.Fatalf("load scenario: %v", err) } agentPath := *agentBin @@ -51,23 +57,101 @@ func main() { log.Fatal("set --agent-bin or A2A_AGENT_BIN") } - procs := startAgents(p, agentPath) + // Start the relay hub first so agents can subscribe as they come up. + hub := relay.NewHub(*relayGRPCPort, *relayCardPort) + if err := hub.Serve(); err != nil { + log.Fatalf("relay serve: %v", err) + } + + procs := startAgents(s, agentPath, *scenarioPath, hub.CardURL()) + defer stopAgents(procs) time.Sleep(*waitReady) ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) defer cancel() - cli, err := client.New(ctx, p, client.Config{ - RoundBudgetMS: p.EffectiveRoundBudgetMS(*roundBudgetMS), - }) + cli, err := newClientWithRetry(ctx, s) if err != nil { stopAgents(procs) log.Fatalf("client: %v", err) } - result := scheduler.New(p, cli).Run(ctx) + agentIDs := s.AgentIDs() + agentIndexByID := map[string]int{} + for i, a := range s.Agents { + agentIndexByID[a.ID] = i + } + + // Runner subscribes to every agent's findings stream and relays each finding + // to all other agents. Retries keep the subscription alive across restarts. + for _, id := range agentIDs { + go func(agentID string) { + for { + select { + case <-ctx.Done(): + return + default: + } + err := cli.SubscribeFindings(ctx, agentID, func(f consensus.Finding) { + hub.Broadcast(f) + }) + if err == nil || ctx.Err() != nil { + return + } + time.Sleep(300 * time.Millisecond) + } + }(id) + } + + // Let finding subscriptions establish before agents start producing. + time.Sleep(500 * time.Millisecond) + + runStart := time.Now() + benchlog.SetRunStart(runStart) + if err := cli.StartAll(ctx, agentIDs); err != nil { + stopAgents(procs) + log.Fatalf("start agents: %v", err) + } + + result := metrics.RunResult{ + ScenarioName: s.Metadata.Name, + Domain: s.Metadata.Domain, + Implementation: benchlog.ImplA2A, + Agents: len(s.Agents), + ThinkTimeMs: s.Spec.ThinkTimeMs, + PayloadBytes: s.Spec.PayloadBytes, + } + + var snapshots []consensus.AgentSnapshot + deadline := time.Now().Add(2 * time.Minute) + for time.Now().Before(deadline) { + snapshots = snapshots[:0] + var pollErr error + for _, id := range agentIDs { + snap, err := cli.Snapshot(ctx, id) + if err != nil { + pollErr = err + break + } + snapshots = append(snapshots, snap) + } + if pollErr == nil { + if ok, _ := consensus.GlobalConsensus(snapshots); ok { + result.Success = true + break + } + } + time.Sleep(20 * time.Millisecond) + } + + if !result.Success && len(snapshots) > 0 { + if ok, _ := consensus.GlobalConsensus(snapshots); ok { + result.Success = true + } + } + + result = aggregateResult(result, runStart, snapshots, hub) - time.Sleep(200 * time.Millisecond) stopAgents(procs) if *outputJSON != "" { @@ -83,16 +167,100 @@ func main() { data, _ := json.MarshalIndent(result, "", " ") fmt.Println(string(data)) + if !result.Success { + os.Exit(1) + } +} + +func newClientWithRetry(ctx context.Context, s *scenario.ConsensusScenario) (*client.Client, error) { + var lastErr error + for attempt := 0; attempt < 50; attempt++ { + cli, err := client.New(ctx, s) + if err == nil { + return cli, nil + } + lastErr = err + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-time.After(200 * time.Millisecond): + } + } + return nil, lastErr +} + +func aggregateResult(result metrics.RunResult, runStart time.Time, snapshots []consensus.AgentSnapshot, hub *relay.Hub) metrics.RunResult { + if len(snapshots) == 0 { + result.Error = "no agent snapshots" + result.ConsensusWallMS = time.Since(runStart).Milliseconds() + return result + } + + var ( + totalEmitted int + totalApplied int + lastConvergeMS int64 + propDurations []int64 + maxConsensusRound int + maxRound int + ) + + for _, snap := range snapshots { + totalEmitted += snap.FindingsEmitted + totalApplied += snap.FindingsApplied + if snap.ConsensusRound > maxConsensusRound { + maxConsensusRound = snap.ConsensusRound + } + if snap.Round > maxRound { + maxRound = snap.Round + } + if snap.ConvergedAtNs > 0 { + ms := (snap.ConvergedAtNs - runStart.UnixNano()) / int64(time.Millisecond) + if ms > lastConvergeMS { + lastConvergeMS = ms + } + } + if snap.AvgPropagationMs > 0 { + propDurations = append(propDurations, snap.AvgPropagationMs) + } + } + + if result.Success { + result.ConsensusWallMS = time.Since(runStart).Milliseconds() + if lastConvergeMS > 0 { + result.ConsensusWallMS = lastConvergeMS + } + } else { + result.ConsensusWallMS = time.Since(runStart).Milliseconds() + result.Error = "consensus not reached" + } + + result.ConsensusRound = maxConsensusRound + if result.ConsensusRound == 0 { + result.ConsensusRound = maxRound + } + result.FindingsEmitted = totalEmitted + result.FindingsReceivedTotal = totalApplied + result.LastAgentConvergeMS = lastConvergeMS + result.AvgPropagationMS, result.P95PropagationMS = metrics.AggregatePropagation(propDurations) + // Relay hub did all the fan-out work; report its real counters. + streamCount, fanoutMS := hub.Stats() + result.StreamRPCCount = streamCount + result.CoordFanoutMS = fanoutMS + return result } -func startAgents(p *plan.ExecutionPlan, agentBin string) []*exec.Cmd { +func startAgents(s *scenario.ConsensusScenario, agentBin, scenarioFile, relayCardURL string) []*exec.Cmd { var procs []*exec.Cmd - for _, agent := range p.Agents { + for i, agent := range s.Agents { cmd := exec.Command( agentBin, "--agent-id", agent.ID, "--grpc-port", fmt.Sprintf("%d", agent.A2APort), - "--card-port", fmt.Sprintf("%d", p.CardPort(agent)), + "--card-port", fmt.Sprintf("%d", s.CardPort(agent)), + "--scenario-file", scenarioFile, + "--agent-index", fmt.Sprintf("%d", i), + "--relay-card-url", relayCardURL, ) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr diff --git a/benchmarks/slim-vs-a2a-v2/a2a/internal/agent/runtime.go b/benchmarks/slim-vs-a2a/a2a/internal/agent/runtime.go similarity index 96% rename from benchmarks/slim-vs-a2a-v2/a2a/internal/agent/runtime.go rename to benchmarks/slim-vs-a2a/a2a/internal/agent/runtime.go index 6e39e40c..62e3bd67 100644 --- a/benchmarks/slim-vs-a2a-v2/a2a/internal/agent/runtime.go +++ b/benchmarks/slim-vs-a2a/a2a/internal/agent/runtime.go @@ -27,10 +27,10 @@ import ( "github.com/a2aproject/a2a-go/v2/a2aclient/agentcard" a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" "github.com/a2aproject/a2a-go/v2/a2asrv" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/benchlog" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" "google.golang.org/grpc" "google.golang.org/grpc/credentials/insecure" ) diff --git a/benchmarks/slim-vs-a2a/a2a/internal/client/client.go b/benchmarks/slim-vs-a2a/a2a/internal/client/client.go index 72fa84e4..6dff1d5d 100644 --- a/benchmarks/slim-vs-a2a/a2a/internal/client/client.go +++ b/benchmarks/slim-vs-a2a/a2a/internal/client/client.go @@ -5,42 +5,28 @@ package client import ( "context" + "encoding/json" "fmt" - "strings" - "time" "github.com/a2aproject/a2a-go/v2/a2a" "github.com/a2aproject/a2a-go/v2/a2aclient" "github.com/a2aproject/a2a-go/v2/a2aclient/agentcard" a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/protocol" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" "google.golang.org/grpc" "google.golang.org/grpc/credentials/insecure" ) -type Config struct { - RoundBudgetMS int64 -} - type Client struct { - clients map[string]*a2aclient.Client - stats *Stats - coordStats *metrics.CoordStats + clients map[string]*a2aclient.Client } -type Stats struct { - ExecuteRPCCount int - SequentialRPCCount int - MulticastRPCCount int -} - -func New(ctx context.Context, p *plan.ExecutionPlan, cfg Config) (*Client, error) { +func New(ctx context.Context, s *scenario.ConsensusScenario) (*Client, error) { clients := map[string]*a2aclient.Client{} - for _, agent := range p.Agents { - baseURL := p.CardBaseURL(agent) + for _, agent := range s.Agents { + baseURL := s.CardBaseURL(agent) card, err := agentcard.DefaultResolver.Resolve(ctx, baseURL) if err != nil { return nil, fmt.Errorf("resolve card for %s: %w", agent.ID, err) @@ -55,213 +41,96 @@ func New(ctx context.Context, p *plan.ExecutionPlan, cfg Config) (*Client, error } clients[agent.ID] = cli } - return &Client{ - clients: clients, - stats: &Stats{}, - coordStats: metrics.NewCoordStats(cfg.RoundBudgetMS), - }, nil -} - -func (c *Client) CoordStats() *metrics.CoordStats { return c.coordStats } - -func (c *Client) Stats() Stats { - return *c.stats -} - -func (c *Client) recordCoord(duration time.Duration, targets, responded, payloadBytes, messagesSent int) { - if c.coordStats != nil { - c.coordStats.RecordCoordOp(duration, targets, responded, payloadBytes, messagesSent) - } + return &Client{clients: clients}, nil } -func (c *Client) ExecuteTask(ctx context.Context, agentID string, task plan.Task) (protocol.Response, error) { - req := protocol.Request{ - Op: protocol.OpExecute, - TaskID: task.ID, - CompletionTimeSec: task.CompletionTimeSec, - MaxCompletionTimeSec: task.MaxCompletionTimeSec, - Output: task.Output, - InjectFailure: task.InjectFailure, - } - return c.send(ctx, agentID, req, true) -} - -func (c *Client) CancelTasks(ctx context.Context, agentIDs []string, taskIDs []string) error { - start := time.Now() - var perAgent []string - responded := 0 - for i, agentID := range agentIDs { - req := protocol.Request{Op: protocol.OpCancel, TaskIDs: taskIDs} - callStart := time.Now() - _, err := c.send(ctx, agentID, req, false) - perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) - if err != nil { - c.recordCoord(time.Since(start), len(agentIDs), responded, len(taskIDs)*16, i) - benchlog.RPC(benchlog.ImplA2A, protocol.OpCancel, "sequential", time.Since(start), false, - fmt.Sprintf("agents=%d", len(agentIDs)), - fmt.Sprintf("idx=%d", i+1), - fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), - fmt.Sprintf("err=%v", err), - ) +func (c *Client) StartAll(ctx context.Context, agentIDs []string) error { + for _, id := range agentIDs { + if err := c.send(ctx, id, protocol.Request{Op: protocol.OpStart}); err != nil { return err } - responded++ - c.stats.SequentialRPCCount++ } - duration := time.Since(start) - c.recordCoord(duration, len(agentIDs), responded, len(taskIDs)*16, len(agentIDs)) - benchlog.RPC(benchlog.ImplA2A, protocol.OpCancel, "sequential", duration, true, - fmt.Sprintf("agents=%d", len(agentIDs)), - fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), - ) return nil } -func (c *Client) PushContext(ctx context.Context, agentIDs []string, payload string) (time.Duration, error) { - start := time.Now() - req := protocol.Request{Op: protocol.OpContext, Payload: payload} - var perAgent []string - responded := 0 - for i, agentID := range agentIDs { - callStart := time.Now() - if _, err := c.send(ctx, agentID, req, false); err != nil { - perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) - c.recordCoord(time.Since(start), len(agentIDs), responded, len(payload), i) - benchlog.RPC(benchlog.ImplA2A, protocol.OpContext, "sequential", time.Since(start), false, - fmt.Sprintf("agents=%d", len(agentIDs)), - fmt.Sprintf("idx=%d", i+1), - fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), - fmt.Sprintf("err=%v", err), - ) - return 0, err - } - perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) - responded++ - c.stats.SequentialRPCCount++ +func (c *Client) Snapshot(ctx context.Context, agentID string) (consensus.AgentSnapshot, error) { + resp, err := c.sendWithResponse(ctx, agentID, protocol.Request{Op: protocol.OpSnapshot}) + if err != nil { + return consensus.AgentSnapshot{}, err } - duration := time.Since(start) - c.recordCoord(duration, len(agentIDs), responded, len(payload), len(agentIDs)) - benchlog.RPC(benchlog.ImplA2A, protocol.OpContext, "sequential", duration, true, - fmt.Sprintf("agents=%d", len(agentIDs)), - fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), - ) - return duration, nil -} - -func (c *Client) SyncPhase(ctx context.Context, agentIDs []string, phase string) (time.Duration, error) { - start := time.Now() - req := protocol.Request{Op: protocol.OpSync, Phase: phase} - var perAgent []string - responded := 0 - for i, agentID := range agentIDs { - callStart := time.Now() - if _, err := c.send(ctx, agentID, req, false); err != nil { - perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) - c.recordCoord(time.Since(start), len(agentIDs), responded, len(phase), i) - benchlog.RPC(benchlog.ImplA2A, protocol.OpSync, "sequential", time.Since(start), false, - fmt.Sprintf("agents=%d", len(agentIDs)), - fmt.Sprintf("idx=%d", i+1), - fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), - fmt.Sprintf("err=%v", err), - ) - return 0, err - } - perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) - responded++ - c.stats.SequentialRPCCount++ + var snap consensus.AgentSnapshot + if err := json.Unmarshal([]byte(resp.Body), &snap); err != nil { + return consensus.AgentSnapshot{}, err } - duration := time.Since(start) - c.recordCoord(duration, len(agentIDs), responded, len(phase), len(agentIDs)) - benchlog.RPC(benchlog.ImplA2A, protocol.OpSync, "sequential", duration, true, - fmt.Sprintf("agents=%d", len(agentIDs)), - fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), - ) - return duration, nil + return snap, nil } -func (c *Client) NotifyFailure(ctx context.Context, agentIDs []string, failedTaskID string) error { - start := time.Now() - payload := "failure=" + failedTaskID - req := protocol.Request{Op: protocol.OpContext, Payload: payload} - var perAgent []string - responded := 0 - for i, agentID := range agentIDs { - callStart := time.Now() - if _, err := c.send(ctx, agentID, req, false); err != nil { - perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) - c.recordCoord(time.Since(start), len(agentIDs), responded, len(payload), i) - benchlog.RPC(benchlog.ImplA2A, protocol.OpContext, "sequential", time.Since(start), false, - fmt.Sprintf("agents=%d", len(agentIDs)), - fmt.Sprintf("failed_task=%s", failedTaskID), - fmt.Sprintf("idx=%d", i+1), - fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), - fmt.Sprintf("err=%v", err), - ) - return err +// SubscribeFindings opens the OpStreamFindings server stream to an agent and +// invokes onFinding for every finding the agent produces. It blocks until the +// stream ends or ctx is cancelled, so callers run it in a goroutine. +func (c *Client) SubscribeFindings(ctx context.Context, agentID string, onFinding func(consensus.Finding)) error { + cli, ok := c.clients[agentID] + if !ok { + return fmt.Errorf("unknown agent %q", agentID) + } + reqText, err := json.Marshal(protocol.Request{Op: protocol.OpStreamFindings}) + if err != nil { + return err + } + msg := a2a.NewMessage(a2a.MessageRoleUser, a2a.NewTextPart(string(reqText))) + for ev, streamErr := range cli.SendStreamingMessage(ctx, &a2a.SendMessageRequest{Message: msg}) { + if streamErr != nil { + return streamErr } - perAgent = append(perAgent, fmt.Sprintf("%s:%d", agentID, time.Since(callStart).Milliseconds())) - responded++ - c.stats.SequentialRPCCount++ + f, ok := findingFromEvent(ev) + if !ok { + continue + } + onFinding(f) } - duration := time.Since(start) - c.recordCoord(duration, len(agentIDs), responded, len(payload), len(agentIDs)) - benchlog.RPC(benchlog.ImplA2A, protocol.OpContext, "sequential", duration, true, - fmt.Sprintf("agents=%d", len(agentIDs)), - fmt.Sprintf("failed_task=%s", failedTaskID), - fmt.Sprintf("per_agent_ms=%s", strings.Join(perAgent, ",")), - ) return nil } -func (c *Client) send(ctx context.Context, agentID string, req protocol.Request, execute bool) (protocol.Response, error) { +func (c *Client) send(ctx context.Context, agentID string, req protocol.Request) error { + _, err := c.sendWithResponse(ctx, agentID, req) + return err +} + +func (c *Client) sendWithResponse(ctx context.Context, agentID string, req protocol.Request) (protocol.Response, error) { cli, ok := c.clients[agentID] if !ok { return protocol.Response{}, fmt.Errorf("unknown agent %q", agentID) } - text, err := protocol.EncodeRequest(req) + text, err := json.Marshal(req) if err != nil { return protocol.Response{}, err } - msg := a2a.NewMessage(a2a.MessageRoleUser, a2a.NewTextPart(text)) - callStart := time.Now() + msg := a2a.NewMessage(a2a.MessageRoleUser, a2a.NewTextPart(string(text))) result, err := cli.SendMessage(ctx, &a2a.SendMessageRequest{Message: msg}) - duration := time.Since(callStart) if err != nil { - if execute { - benchlog.RPC(benchlog.ImplA2A, req.Op, "p2p", duration, false, - fmt.Sprintf("agent=%s", agentID), - fmt.Sprintf("task=%s", req.TaskID), - fmt.Sprintf("err=%v", err), - ) - } return protocol.Response{}, err } - if execute { - c.stats.ExecuteRPCCount++ - } respText, err := responseText(result) if err != nil { - if execute { - benchlog.RPC(benchlog.ImplA2A, req.Op, "p2p", duration, false, - fmt.Sprintf("agent=%s", agentID), - fmt.Sprintf("task=%s", req.TaskID), - fmt.Sprintf("err=%v", err), - ) - } return protocol.Response{}, err } - resp, decErr := protocol.DecodeResponse(respText) - if execute { - benchlog.RPC(benchlog.ImplA2A, req.Op, "p2p", duration, decErr == nil && resp.OK, - fmt.Sprintf("agent=%s", agentID), - fmt.Sprintf("task=%s", req.TaskID), - fmt.Sprintf("resp_ok=%t", resp.OK), - ) + return protocol.DecodeResponse(respText) +} + +func findingFromEvent(ev a2a.Event) (consensus.Finding, bool) { + su, ok := ev.(*a2a.TaskStatusUpdateEvent) + if !ok || su.Status.Message == nil { + return consensus.Finding{}, false } - if decErr != nil { - return protocol.Response{}, decErr + text := firstText(su.Status.Message) + if text == "" { + return consensus.Finding{}, false + } + f, err := consensus.DecodeFinding([]byte(text)) + if err != nil { + return consensus.Finding{}, false } - return resp, nil + return f, true } func responseText(result a2a.SendMessageResult) (string, error) { diff --git a/benchmarks/slim-vs-a2a/a2a/internal/executor/executor.go b/benchmarks/slim-vs-a2a/a2a/internal/executor/executor.go deleted file mode 100644 index 5b98bbf4..00000000 --- a/benchmarks/slim-vs-a2a/a2a/internal/executor/executor.go +++ /dev/null @@ -1,97 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package executor - -import ( - "context" - "strings" - "sync" - "time" - - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/protocol" -) - -type Engine struct { - mu sync.Mutex - cancels map[string]context.CancelFunc - obsolete bool -} - -func New() *Engine { - return &Engine{cancels: map[string]context.CancelFunc{}} -} - -func (e *Engine) Handle(ctx context.Context, req protocol.Request) protocol.Response { - switch req.Op { - case protocol.OpExecute: - return e.execute(ctx, req) - case protocol.OpCancel: - return e.cancel(req) - case protocol.OpContext: - return e.applyContext(req) - case protocol.OpSync: - return protocol.Response{OK: true} - default: - return protocol.Response{OK: false, Error: "unknown op"} - } -} - -func (e *Engine) execute(parent context.Context, req protocol.Request) protocol.Response { - if req.InjectFailure { - return protocol.Response{OK: false, TaskID: req.TaskID, Error: "injected failure"} - } - - ctx, cancel := context.WithCancel(parent) - e.mu.Lock() - e.cancels[req.TaskID] = cancel - e.mu.Unlock() - defer func() { - e.mu.Lock() - delete(e.cancels, req.TaskID) - e.mu.Unlock() - cancel() - }() - - deadline := time.Duration(req.MaxCompletionTimeSec * float64(time.Second)) - if deadline <= 0 { - deadline = time.Duration(req.CompletionTimeSec*float64(time.Second)) + time.Second - } - timer := time.NewTimer(deadline) - defer timer.Stop() - - sleep := time.Duration(req.CompletionTimeSec * float64(time.Second)) - start := time.Now() - select { - case <-time.After(sleep): - case <-ctx.Done(): - return protocol.Response{OK: false, TaskID: req.TaskID, Error: "cancelled"} - case <-timer.C: - return protocol.Response{OK: false, TaskID: req.TaskID, Error: "timeout"} - } - - return protocol.Response{ - OK: true, - TaskID: req.TaskID, - Output: req.Output, - ElapsedSec: time.Since(start).Seconds(), - } -} - -func (e *Engine) cancel(req protocol.Request) protocol.Response { - e.mu.Lock() - defer e.mu.Unlock() - for _, id := range req.TaskIDs { - if cancel, ok := e.cancels[id]; ok { - cancel() - } - } - return protocol.Response{OK: true} -} - -func (e *Engine) applyContext(req protocol.Request) protocol.Response { - if strings.Contains(req.Payload, "cancel") { - e.obsolete = true - } - return protocol.Response{OK: true} -} diff --git a/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go b/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go index 0c48cd2b..21f53971 100644 --- a/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go +++ b/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go @@ -1,44 +1,45 @@ // Copyright AGNTCY Contributors (https://github.com/agntcy) // SPDX-License-Identifier: Apache-2.0 +// Package protocol defines the A2A control/stream ops for the relay topology. +// +// Unlike SLIM's native group session, A2A has no peer multicast: findings must +// pass through the runner, which is the explicit relay hub. Two server-streaming +// legs carry findings: +// - OpStreamFindings: the runner subscribes to each agent (agent streams its +// own findings to the runner). +// - OpStreamRelay: each agent subscribes to the runner (runner streams every +// relayed finding back out to the agents). +// +// OpStart and OpSnapshot remain unary control calls driven by the runner. package protocol -import "encoding/json" +import ( + "encoding/json" +) const ( - OpExecute = "execute" - OpCancel = "cancel" - OpContext = "context" - OpSync = "sync" + OpStart = "start" + OpSnapshot = "snapshot" + OpStreamFindings = "stream_findings" + OpStreamRelay = "stream_relay" ) type Request struct { - Op string `json:"op"` - TaskID string `json:"taskId,omitempty"` - CompletionTimeSec float64 `json:"completionTimeSec,omitempty"` - MaxCompletionTimeSec float64 `json:"maxCompletionTimeSec,omitempty"` - Output string `json:"output,omitempty"` - InjectFailure bool `json:"injectFailure,omitempty"` - TaskIDs []string `json:"taskIds,omitempty"` - Payload string `json:"payload,omitempty"` - Phase string `json:"phase,omitempty"` - FailedTaskID string `json:"failedTaskId,omitempty"` + Op string `json:"op"` + // AgentIndex identifies the subscriber on OpStreamRelay so the relay can + // avoid echoing an agent's own findings back to it. + AgentIndex int `json:"agentIndex,omitempty"` } type Response struct { - OK bool `json:"ok"` - TaskID string `json:"taskId,omitempty"` - Output string `json:"output,omitempty"` - Error string `json:"error,omitempty"` - ElapsedSec float64 `json:"elapsedSec,omitempty"` + OK bool `json:"ok"` + Error string `json:"error,omitempty"` + Body string `json:"body,omitempty"` } -func EncodeRequest(req Request) (string, error) { - data, err := json.Marshal(req) - if err != nil { - return "", err - } - return string(data), nil +func EncodeRequest(req Request) ([]byte, error) { + return json.Marshal(req) } func DecodeRequest(text string) (Request, error) { @@ -47,6 +48,10 @@ func DecodeRequest(text string) (Request, error) { return req, err } +func EncodeResponse(resp Response) ([]byte, error) { + return json.Marshal(resp) +} + func DecodeResponse(text string) (Response, error) { var resp Response err := json.Unmarshal([]byte(text), &resp) diff --git a/benchmarks/slim-vs-a2a-v2/a2a/internal/relay/relay.go b/benchmarks/slim-vs-a2a/a2a/internal/relay/relay.go similarity index 97% rename from benchmarks/slim-vs-a2a-v2/a2a/internal/relay/relay.go rename to benchmarks/slim-vs-a2a/a2a/internal/relay/relay.go index b0ea2b9f..0a44c285 100644 --- a/benchmarks/slim-vs-a2a-v2/a2a/internal/relay/relay.go +++ b/benchmarks/slim-vs-a2a/a2a/internal/relay/relay.go @@ -24,8 +24,8 @@ import ( a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" "github.com/a2aproject/a2a-go/v2/a2asrv" "github.com/a2aproject/a2a-go/v2/a2asrv/taskstore" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/a2a/internal/protocol" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" "google.golang.org/grpc" ) @@ -63,7 +63,7 @@ func (h *Hub) Serve() error { handler := a2asrv.NewHandler( &relayExecutor{hub: h}, a2asrv.WithTaskStore(taskstore.NewInMemory(&taskstore.InMemoryStoreConfig{ - Authenticator: func(context.Context) (string, error) { return "slim-vs-a2a-v2", nil }, + Authenticator: func(context.Context) (string, error) { return "slim-vs-a2a", nil }, })), ) diff --git a/benchmarks/slim-vs-a2a/a2a/internal/scheduler/scheduler.go b/benchmarks/slim-vs-a2a/a2a/internal/scheduler/scheduler.go deleted file mode 100644 index 37b49c5c..00000000 --- a/benchmarks/slim-vs-a2a/a2a/internal/scheduler/scheduler.go +++ /dev/null @@ -1,342 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package scheduler - -import ( - "context" - "fmt" - "strings" - "sync" - "time" - - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/client" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" -) - -type taskState int - -const ( - statePending taskState = iota - stateRunning - stateCompleted - stateFailed - stateTimedOut - stateCancelled -) - -type Scheduler struct { - plan *plan.ExecutionPlan - client *client.Client -} - -func New(p *plan.ExecutionPlan, cli *client.Client) *Scheduler { - return &Scheduler{plan: p, client: cli} -} - -func (s *Scheduler) Run(ctx context.Context) metrics.RunResult { - start := time.Now() - benchlog.SetRunStart(start) - result := metrics.RunResult{ - PlanName: s.plan.Metadata.Name, - Domain: s.plan.Metadata.Domain, - Implementation: "a2a-grpc", - Agents: len(s.plan.Agents), - Tasks: len(s.plan.Tasks), - } - - states := map[string]taskState{} - contextFired := map[string]bool{} - var mu sync.Mutex - var wg sync.WaitGroup - var contextPushMS int64 - var syncBarrierMS int64 - var cancelPropagationMS int64 - var obsoleteCompleted int - abort := false - - for _, t := range s.plan.Tasks { - states[t.ID] = statePending - } - - cancelTasks := func(taskIDs []string) { - if len(taskIDs) == 0 { - return - } - byAgent := map[string][]string{} - for _, id := range taskIDs { - task, ok := s.plan.TaskByID(id) - if !ok { - continue - } - byAgent[task.Agent] = append(byAgent[task.Agent], id) - } - startCancel := time.Now() - var cancelErr error - for agentID, ids := range byAgent { - if err := s.client.CancelTasks(ctx, []string{agentID}, ids); err != nil && cancelErr == nil { - cancelErr = err - } - } - duration := time.Since(startCancel) - cancelPropagationMS += duration.Milliseconds() - kv := []string{ - fmt.Sprintf("agents=%d", len(byAgent)), - fmt.Sprintf("tasks=%d", len(taskIDs)), - } - if cancelErr != nil { - kv = append(kv, fmt.Sprintf("err=%v", cancelErr)) - } - benchlog.Coord(benchlog.ImplA2A, "cancel", cancelErr == nil, duration, kv...) - } - - cancelDownstream := func(root string) { - downstream := plan.DownstreamTasks(s.plan.Tasks, root) - var ids []string - mu.Lock() - for id := range downstream { - switch states[id] { - case statePending, stateRunning: - states[id] = stateCancelled - result.TasksCancelled++ - ids = append(ids, id) - } - } - mu.Unlock() - cancelTasks(ids) - } - - handleFailure := func(taskID string, timedOut bool) { - mu.Lock() - if timedOut { - states[taskID] = stateTimedOut - result.TasksTimedOut++ - } else { - states[taskID] = stateFailed - result.TasksFailed++ - } - abort = true - mu.Unlock() - agents := affectedAgents([]string{taskID}, s.plan) - if err := s.client.NotifyFailure(ctx, agents, taskID); err != nil { - benchlog.Coord(benchlog.ImplA2A, "notify_failure", false, 0, - fmt.Sprintf("task=%s", taskID), - fmt.Sprintf("agents=%d", len(agents)), - fmt.Sprintf("err=%v", err), - ) - } - cancelTasks([]string{taskID}) - cancelDownstream(taskID) - } - - applyContext := func(taskID string) { - mu.Lock() - if contextFired[taskID] { - mu.Unlock() - return - } - contextFired[taskID] = true - mu.Unlock() - - for _, cu := range s.plan.ContextUpdatesAfter(taskID) { - d, err := s.client.PushContext(ctx, cu.TargetAgents, cu.Payload) - contextPushMS += d.Milliseconds() - kv := []string{ - fmt.Sprintf("after_task=%s", taskID), - fmt.Sprintf("agents=%d", len(cu.TargetAgents)), - fmt.Sprintf("payload=%q", benchlog.TruncatePayload(cu.Payload)), - } - if err != nil { - kv = append(kv, fmt.Sprintf("err=%v", err)) - } - benchlog.Coord(benchlog.ImplA2A, "context_push", err == nil, d, kv...) - - if strings.Contains(cu.Payload, "sync=") || strings.Contains(cu.Payload, "phase=") { - syncD, syncErr := s.client.SyncPhase(ctx, cu.TargetAgents, cu.Payload) - if syncErr == nil { - syncBarrierMS += syncD.Milliseconds() - } - syncKV := []string{ - fmt.Sprintf("after_task=%s", taskID), - fmt.Sprintf("agents=%d", len(cu.TargetAgents)), - } - if syncErr != nil { - syncKV = append(syncKV, fmt.Sprintf("err=%v", syncErr)) - } - benchlog.Coord(benchlog.ImplA2A, "sync_phase", syncErr == nil, syncD, syncKV...) - } - if strings.Contains(strings.ToLower(cu.Payload), "cancel") { - var toCancel []string - mu.Lock() - for _, t := range s.plan.Tasks { - if states[t.ID] != stateRunning { - continue - } - if shouldCancelByContext(t, cu.Payload) { - states[t.ID] = stateCancelled - result.TasksCancelled++ - toCancel = append(toCancel, t.ID) - obsoleteCompleted++ - } - } - mu.Unlock() - cancelTasks(toCancel) - } - } - } - - launch := func(task plan.Task) { - wg.Add(1) - go func(task plan.Task) { - defer wg.Done() - taskStart := time.Now() - benchlog.Task(benchlog.ImplA2A, "started", task.ID, task.Agent, 0) - resp, err := s.client.ExecuteTask(ctx, task.Agent, task) - taskDuration := time.Since(taskStart) - mu.Lock() - if abort && states[task.ID] == stateRunning { - states[task.ID] = stateCancelled - result.TasksCancelled++ - mu.Unlock() - benchlog.Task(benchlog.ImplA2A, "cancelled", task.ID, task.Agent, taskDuration) - return - } - mu.Unlock() - - if err != nil { - benchlog.Task(benchlog.ImplA2A, "failed", task.ID, task.Agent, taskDuration, - fmt.Sprintf("err=%v", err)) - handleFailure(task.ID, false) - return - } - if !resp.OK { - benchlog.Task(benchlog.ImplA2A, "failed", task.ID, task.Agent, taskDuration, - fmt.Sprintf("resp_error=%s", resp.Error)) - handleFailure(task.ID, resp.Error == "timeout") - return - } - - mu.Lock() - states[task.ID] = stateCompleted - result.TasksCompleted++ - mu.Unlock() - benchlog.Task(benchlog.ImplA2A, "completed", task.ID, task.Agent, taskDuration) - applyContext(task.ID) - }(task) - } - - for { - mu.Lock() - if abort { - mu.Unlock() - break - } - ready := readyTasks(s.plan.Tasks, states) - for _, task := range ready { - states[task.ID] = stateRunning - launch(task) - } - pending := countStates(states, statePending) - running := countStates(states, stateRunning) - mu.Unlock() - - if len(ready) == 0 && pending == 0 && running == 0 { - break - } - time.Sleep(5 * time.Millisecond) - } - - wg.Wait() - stats := s.client.Stats() - coordTimeMS := contextPushMS + syncBarrierMS + cancelPropagationMS - result.TotalWallClockMS = time.Since(start).Milliseconds() - result.ContextPushMS = contextPushMS - result.SyncBarrierMS = syncBarrierMS - result.CancelPropagationMS = cancelPropagationMS - result.ObsoleteTasksCompleted = obsoleteCompleted - result.ExecuteRPCCount = stats.ExecuteRPCCount - result.SequentialRPCCount = stats.SequentialRPCCount - result.MulticastRPCCount = stats.MulticastRPCCount - result.MakespanMS = result.TotalWallClockMS - result.Success = result.TasksFailed == 0 && result.TasksTimedOut == 0 - if s.client.CoordStats() != nil { - s.client.CoordStats().ApplyToRunResult(&result, coordTimeMS) - } - if !result.Success && result.Error == "" { - result.Error = fmt.Sprintf("failed=%d timed_out=%d cancelled=%d", result.TasksFailed, result.TasksTimedOut, result.TasksCancelled) - } - benchlog.Coord(benchlog.ImplA2A, "run_finished", result.Success, time.Since(start), - fmt.Sprintf("tasks_completed=%d", result.TasksCompleted), - fmt.Sprintf("tasks_failed=%d", result.TasksFailed), - fmt.Sprintf("tasks_cancelled=%d", result.TasksCancelled), - fmt.Sprintf("context_push_ms=%d", contextPushMS), - fmt.Sprintf("execute_rpcs=%d", stats.ExecuteRPCCount), - fmt.Sprintf("sequential_rpcs=%d", stats.SequentialRPCCount), - ) - return result -} - -func readyTasks(tasks []plan.Task, states map[string]taskState) []plan.Task { - var ready []plan.Task - for _, task := range tasks { - if states[task.ID] != statePending { - continue - } - ok := true - for _, dep := range task.DependsOn { - if states[dep] != stateCompleted { - ok = false - break - } - } - if ok { - ready = append(ready, task) - } - } - return ready -} - -func countStates(states map[string]taskState, target taskState) int { - n := 0 - for _, st := range states { - if st == target { - n++ - } - } - return n -} - -func affectedAgents(taskIDs []string, p *plan.ExecutionPlan) []string { - set := map[string]struct{}{} - for _, id := range taskIDs { - task, ok := p.TaskByID(id) - if !ok { - continue - } - set[task.Agent] = struct{}{} - } - out := make([]string, 0, len(set)) - for id := range set { - out = append(out, id) - } - return out -} - -func shouldCancelByContext(task plan.Task, payload string) bool { - payload = strings.ToLower(payload) - if strings.Contains(payload, "pharmacy") && strings.Contains(task.ID, "pharmacy") { - return true - } - if strings.Contains(payload, "detour") && strings.Contains(task.ID, "detour") { - return true - } - if strings.Contains(payload, "node-drain") && strings.Contains(task.ID, "node-drain") { - return true - } - if strings.Contains(payload, "mesh-rollback") && strings.Contains(task.ID, "rollback") { - return true - } - return false -} diff --git a/benchmarks/slim-vs-a2a/go.mod b/benchmarks/slim-vs-a2a/go.mod index 01cfcbab..02b62e40 100644 --- a/benchmarks/slim-vs-a2a/go.mod +++ b/benchmarks/slim-vs-a2a/go.mod @@ -5,27 +5,21 @@ go 1.25.0 require ( github.com/a2aproject/a2a-go/v2 v2.3.0 github.com/agntcy/slim-bindings-go v1.4.0 - github.com/onsi/ginkgo/v2 v2.29.0 - github.com/onsi/gomega v1.41.0 google.golang.org/grpc v1.80.0 gopkg.in/yaml.v3 v3.0.1 ) require ( - github.com/Masterminds/semver/v3 v3.4.0 // indirect - github.com/go-logr/logr v1.4.3 // indirect - github.com/go-task/slim-sprig/v3 v3.0.0 // indirect - github.com/google/go-cmp v0.7.0 // indirect - github.com/google/pprof v0.0.0-20260402051712-545e8a4df936 // indirect github.com/google/uuid v1.6.0 // indirect - go.yaml.in/yaml/v3 v3.0.4 // indirect + github.com/kr/pretty v0.3.1 // indirect + github.com/rogpeppe/go-internal v1.13.1 // indirect golang.org/x/mod v0.35.0 // indirect golang.org/x/net v0.53.0 // indirect golang.org/x/sync v0.20.0 // indirect golang.org/x/sys v0.43.0 // indirect golang.org/x/text v0.36.0 // indirect - golang.org/x/tools v0.44.0 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4 // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 // indirect google.golang.org/protobuf v1.36.11 // indirect + gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 // indirect ) diff --git a/benchmarks/slim-vs-a2a/go.sum b/benchmarks/slim-vs-a2a/go.sum index 1aad676a..d2fc54cd 100644 --- a/benchmarks/slim-vs-a2a/go.sum +++ b/benchmarks/slim-vs-a2a/go.sum @@ -1,63 +1,28 @@ -github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0= -github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM= github.com/a2aproject/a2a-go/v2 v2.3.0 h1:wSseKBBlDYBAJLdHZ4puFIL2XyL/5YuMrDw9TXLL1ek= github.com/a2aproject/a2a-go/v2 v2.3.0/go.mod h1:mkZr8y2bUgAVQsjs/5fHK7xrRlAHDybMEyxWh2tKRC8= github.com/agntcy/slim-bindings-go v1.4.0 h1:DXAGhe9TWSm33KIgrJFkIcxdqh+/oPWy4Jq5PCI0HvM= github.com/agntcy/slim-bindings-go v1.4.0/go.mod h1:XK0Ing+REEl8xG79HTMx52XzWK2THuTQA+Y7JTAn428= github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= -github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/gkampitakis/ciinfo v0.3.2 h1:JcuOPk8ZU7nZQjdUhctuhQofk7BGHuIy0c9Ez8BNhXs= -github.com/gkampitakis/ciinfo v0.3.2/go.mod h1:1NIwaOcFChN4fa/B0hEBdAb6npDlFL8Bwx4dfRLRqAo= -github.com/gkampitakis/go-diff v1.3.2 h1:Qyn0J9XJSDTgnsgHRdz9Zp24RaJeKMUHg2+PDZZdC4M= -github.com/gkampitakis/go-diff v1.3.2/go.mod h1:LLgOrpqleQe26cte8s36HTWcTmMEur6OPYerdAAS9tk= -github.com/gkampitakis/go-snaps v0.5.15 h1:amyJrvM1D33cPHwVrjo9jQxX8g/7E2wYdZ+01KS3zGE= -github.com/gkampitakis/go-snaps v0.5.15/go.mod h1:HNpx/9GoKisdhw9AFOBT1N7DBs9DiHo/hGheFGBZ+mc= +github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= -github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI= -github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8= -github.com/goccy/go-yaml v1.18.0 h1:8W7wMFS12Pcas7KU+VVkaiCng+kG8QiFeFwzFb+rwuw= -github.com/goccy/go-yaml v1.18.0/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA= github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= -github.com/google/pprof v0.0.0-20260402051712-545e8a4df936 h1:EwtI+Al+DeppwYX2oXJCETMO23COyaKGP6fHVpkpWpg= -github.com/google/pprof v0.0.0-20260402051712-545e8a4df936/go.mod h1:MxpfABSjhmINe3F1It9d+8exIHFvUqtLIRCdOGNXqiI= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/joshdk/go-junit v1.0.0 h1:S86cUKIdwBHWwA6xCmFlf3RTLfVXYQfvanM5Uh+K6GE= -github.com/joshdk/go-junit v1.0.0/go.mod h1:TiiV0PqkaNfFXjEiyjWM3XXrhVyCa1K4Zfga6W52ung= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= -github.com/maruel/natural v1.1.1 h1:Hja7XhhmvEFhcByqDoHz9QZbkWey+COd9xWfCfn1ioo= -github.com/maruel/natural v1.1.1/go.mod h1:v+Rfd79xlw1AgVBjbO0BEQmptqb5HvL/k9GRHB7ZKEg= -github.com/mfridman/tparse v0.18.0 h1:wh6dzOKaIwkUGyKgOntDW4liXSo37qg5AXbIhkMV3vE= -github.com/mfridman/tparse v0.18.0/go.mod h1:gEvqZTuCgEhPbYk/2lS3Kcxg1GmTxxU7kTC8DvP0i/A= -github.com/onsi/ginkgo/v2 v2.29.0 h1:rfh+ZFjgJhYWRoIqVf3Uwx/W20yLrcrE2h2GmYVRaag= -github.com/onsi/ginkgo/v2 v2.29.0/go.mod h1:+aXOY+vzZ5mu2iI2HpTZUPmM//oQfsNFX6gU9kNcA44= -github.com/onsi/gomega v1.41.0 h1:OwKp4pXNgVxf6sCplzYo794OFNuoL2q2SBMU5NSWOjA= -github.com/onsi/gomega v1.41.0/go.mod h1:M/Uqpu/8qTjtzCLUA2zJHX9Iilrau25x1PdoSRbWh5A= -github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= -github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA= +github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII= github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o= -github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= -github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= -github.com/tidwall/gjson v1.18.0 h1:FIDeeyB800efLX89e5a8Y0BNH+LOngJyGrIWxG2FKQY= -github.com/tidwall/gjson v1.18.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk= -github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA= -github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM= -github.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4= -github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU= -github.com/tidwall/sjson v1.2.5 h1:kLy8mja+1c9jlljvWTlSazM7cKDRfJuR/bOJhcY5NcY= -github.com/tidwall/sjson v1.2.5/go.mod h1:Fvgq9kS/6ociJEDnK0Fk1cpYF4FIW6ZF7LAe+6jwd28= go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48= @@ -70,8 +35,6 @@ go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2W go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew= go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI= go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA= -go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc= -go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM= golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU= golang.org/x/net v0.53.0 h1:d+qAbo5L0orcWAr0a9JweQpjXF19LMXJE8Ey7hwOdUA= @@ -82,8 +45,6 @@ golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI= golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/text v0.36.0 h1:JfKh3XmcRPqZPKevfXVpI1wXPTqbkE5f7JA92a55Yxg= golang.org/x/text v0.36.0/go.mod h1:NIdBknypM8iqVmPiuco0Dh6P5Jcdk8lJL0CUebqK164= -golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c= -golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4 h1:yOzSCGPx+cp5VO7IxvZ9SBFF7j1tZVcNtlHR2iYKtVo= diff --git a/benchmarks/slim-vs-a2a/internal/benchlog/benchlog.go b/benchmarks/slim-vs-a2a/internal/benchlog/benchlog.go index d3e0ef81..bdb4999d 100644 --- a/benchmarks/slim-vs-a2a/internal/benchlog/benchlog.go +++ b/benchmarks/slim-vs-a2a/internal/benchlog/benchlog.go @@ -14,8 +14,8 @@ import ( ) const ( - ImplSLIM = "slim-multicast" - ImplA2A = "a2a-grpc" + ImplSLIM = "slim-group-session" + ImplA2A = "a2a-relay-stream" ) var ( @@ -25,14 +25,12 @@ var ( out io.Writer = os.Stderr ) -// SetRunStart records the scheduler run start for elapsed_ms on each line. func SetRunStart(t time.Time) { mu.Lock() runStart = t mu.Unlock() } -// SetEnabled toggles benchmark logging (default: on). func SetEnabled(on bool) { mu.Lock() enabled = on @@ -49,14 +47,13 @@ func sinceStart() int64 { return time.Since(start).Milliseconds() } -func write(kind string, impl string, kv ...string) { +func write(kind, impl string, kv ...string) { mu.RLock() on := enabled mu.RUnlock() if !on { return } - ts := time.Now().UTC().Format(time.RFC3339Nano) elapsed := sinceStart() parts := []string{ @@ -66,7 +63,6 @@ func write(kind string, impl string, kv ...string) { log.New(out, "", 0).Println(strings.Join(parts, " ")) } -// RPC logs a client RPC with duration and outcome. func RPC(impl, op, mode string, duration time.Duration, ok bool, kv ...string) { status := "ok" if !ok { @@ -82,41 +78,12 @@ func RPC(impl, op, mode string, duration time.Duration, ok bool, kv ...string) { write("rpc", impl, fields...) } -// Task logs scheduler task lifecycle events. -func Task(impl, event, taskID, agent string, duration time.Duration, kv ...string) { +func Finding(impl, event string, agentIndex int, findingID int64, kv ...string) { fields := []string{ fmt.Sprintf("event=%s", event), - fmt.Sprintf("task=%s", taskID), - fmt.Sprintf("agent=%s", agent), - } - if duration > 0 { - fields = append(fields, fmt.Sprintf("duration_ms=%d", duration.Milliseconds())) - } - fields = append(fields, kv...) - write("task", impl, fields...) -} - -// TruncatePayload shortens long payload strings for log lines. -func TruncatePayload(payload string) string { - if len(payload) <= 80 { - return payload - } - return payload[:77] + "..." -} - -// Coord logs coordination ops (context push, cancel, sync) from the scheduler. -func Coord(impl, event string, ok bool, duration time.Duration, kv ...string) { - status := "ok" - if !ok { - status = "error" - } - fields := []string{ - fmt.Sprintf("event=%s", event), - fmt.Sprintf("status=%s", status), - } - if duration > 0 { - fields = append(fields, fmt.Sprintf("duration_ms=%d", duration.Milliseconds())) + fmt.Sprintf("agent=%d", agentIndex), + fmt.Sprintf("finding_id=%d", findingID), } fields = append(fields, kv...) - write("coord", impl, fields...) + write("finding", impl, fields...) } diff --git a/benchmarks/slim-vs-a2a-v2/internal/consensus/consensus.go b/benchmarks/slim-vs-a2a/internal/consensus/consensus.go similarity index 82% rename from benchmarks/slim-vs-a2a-v2/internal/consensus/consensus.go rename to benchmarks/slim-vs-a2a/internal/consensus/consensus.go index 7beb3baa..4015cb86 100644 --- a/benchmarks/slim-vs-a2a-v2/internal/consensus/consensus.go +++ b/benchmarks/slim-vs-a2a/internal/consensus/consensus.go @@ -6,10 +6,11 @@ package consensus import ( "encoding/json" "fmt" + mrand "math/rand" "sync" "time" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" ) const consensusConfidence = 0.55 @@ -21,14 +22,19 @@ type Finding struct { Value int `json:"value"` Confidence float64 `json:"confidence"` EmittedAt int64 `json:"emittedAtNs"` + // Payload is optional fixed-size padding used to stress transport bandwidth. + // It carries no semantic meaning for consensus; it only inflates the wire + // size of every finding so the relay-vs-multicast cost difference shows. + Payload string `json:"payload,omitempty"` } type Config struct { - AgentIndex int - AgentCount int - ValueSpace int - TargetMode string - Seed int64 + AgentIndex int + AgentCount int + ValueSpace int + TargetMode string + Seed int64 + PayloadBytes int } type Engine struct { @@ -47,6 +53,7 @@ type Engine struct { distinctSupports map[int]map[int]struct{} convergedAt int64 consensusRound int + payload string propMu sync.Mutex propDurations []int64 @@ -62,20 +69,39 @@ func NewEngine(spec scenario.Spec, agentIndex int) *Engine { target := targetValue(spec, agentIndex) return &Engine{ cfg: Config{ - AgentIndex: agentIndex, - AgentCount: spec.Agents, - ValueSpace: k, - TargetMode: spec.TargetMode, - Seed: spec.Seed, + AgentIndex: agentIndex, + AgentCount: spec.Agents, + ValueSpace: k, + TargetMode: spec.TargetMode, + Seed: spec.Seed, + PayloadBytes: spec.PayloadBytes, }, value: agentIndex % k, confidence: 0.1, targetValue: target, lastEmitValue: -1, distinctSupports: map[int]map[int]struct{}{}, + payload: makePayload(spec.PayloadBytes, spec.Seed, agentIndex), } } +// makePayload returns deterministic, reproducible padding of n bytes. It is used +// only to inflate finding size for transport stress tests, so math/rand is +// intentional here: the same seed/agent yields identical bytes across both +// transports, keeping the SLIM-vs-A2A comparison fair. Not security-sensitive. +func makePayload(n int, seed int64, agentIndex int) string { + if n <= 0 { + return "" + } + const alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 " + rng := mrand.New(mrand.NewSource(seed + int64(agentIndex)*1_000_003)) + b := make([]byte, n) + for i := range b { + b[i] = alphabet[rng.Intn(len(alphabet))] + } + return string(b) +} + func targetValue(spec scenario.Spec, agentIndex int) int { k := spec.ValueSpace if k <= 0 { @@ -124,6 +150,7 @@ func (e *Engine) Think() (finding *Finding, emit bool) { Value: e.value, Confidence: e.confidence, EmittedAt: time.Now().UnixNano(), + Payload: e.payload, } return &f, true } diff --git a/benchmarks/slim-vs-a2a/internal/metrics/coordstats.go b/benchmarks/slim-vs-a2a/internal/metrics/coordstats.go deleted file mode 100644 index 2da1bf63..00000000 --- a/benchmarks/slim-vs-a2a/internal/metrics/coordstats.go +++ /dev/null @@ -1,95 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package metrics - -import ( - "sort" - "sync" - "time" -) - -// CoordStats tracks coordination RPC quality metrics across a run. -type CoordStats struct { - mu sync.Mutex - - RoundBudgetMS int64 - ContextPushOps int - pushDurations []int64 - MissingResponses int - DeadlineMisses int - BytesSent int64 -} - -func NewCoordStats(roundBudgetMS int64) *CoordStats { - return &CoordStats{RoundBudgetMS: roundBudgetMS} -} - -// RecordCoordOp records one coordination operation (context, sync, cancel, notify). -func (c *CoordStats) RecordCoordOp(duration time.Duration, targets int, responded int, payloadBytes int, messagesSent int) { - if c == nil { - return - } - c.mu.Lock() - defer c.mu.Unlock() - - c.ContextPushOps++ - c.pushDurations = append(c.pushDurations, duration.Milliseconds()) - if targets > responded { - c.MissingResponses += targets - responded - } - if c.RoundBudgetMS > 0 && duration.Milliseconds() > c.RoundBudgetMS { - c.DeadlineMisses++ - } - if messagesSent <= 0 { - messagesSent = 1 - } - c.BytesSent += int64(payloadBytes * messagesSent) -} - -// P95ContextPushMS returns the p95 of recorded coordination op durations. -func (c *CoordStats) P95ContextPushMS() int64 { - if c == nil { - return 0 - } - c.mu.Lock() - defer c.mu.Unlock() - return c.p95Locked() -} - -func (c *CoordStats) p95Locked() int64 { - if len(c.pushDurations) == 0 { - return 0 - } - sorted := append([]int64(nil), c.pushDurations...) - sort.Slice(sorted, func(i, j int) bool { return sorted[i] < sorted[j] }) - idx := int(float64(len(sorted)-1) * 0.95) - return sorted[idx] -} - -// Snapshot returns a copy of aggregate counters. -func (c *CoordStats) Snapshot() (ops int, missing, misses int, bytes int64, p95 int64) { - if c == nil { - return 0, 0, 0, 0, 0 - } - c.mu.Lock() - defer c.mu.Unlock() - return c.ContextPushOps, c.MissingResponses, c.DeadlineMisses, c.BytesSent, c.p95Locked() -} - -// ApplyToRunResult copies coordination stats into a run result. -func (c *CoordStats) ApplyToRunResult(r *RunResult, coordTimeMS int64) { - if c == nil || r == nil { - return - } - ops, missing, misses, bytes, p95 := c.Snapshot() - r.ContextPushOps = ops - r.ContextPushP95MS = p95 - r.CoordMissingResponses = missing - r.CoordDeadlineMisses = misses - r.CoordBytesSent = bytes - r.RoundBudgetMS = c.RoundBudgetMS - if r.TotalWallClockMS > 0 && coordTimeMS > 0 { - r.CoordTimeSharePct = float64(coordTimeMS) * 100 / float64(r.TotalWallClockMS) - } -} diff --git a/benchmarks/slim-vs-a2a/internal/metrics/coordstats_test.go b/benchmarks/slim-vs-a2a/internal/metrics/coordstats_test.go deleted file mode 100644 index 9fa900bf..00000000 --- a/benchmarks/slim-vs-a2a/internal/metrics/coordstats_test.go +++ /dev/null @@ -1,39 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package metrics_test - -import ( - "testing" - "time" - - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" -) - -func TestCoordStatsP95AndDeadline(t *testing.T) { - cs := metrics.NewCoordStats(10) - cs.RecordCoordOp(5*time.Millisecond, 3, 3, 100, 1) - cs.RecordCoordOp(15*time.Millisecond, 3, 3, 100, 1) - cs.RecordCoordOp(20*time.Millisecond, 3, 2, 100, 1) - - if cs.P95ContextPushMS() != 15 { - t.Fatalf("p95=%d want 15", cs.P95ContextPushMS()) - } - _, missing, misses, bytes, _ := cs.Snapshot() - if missing != 1 { - t.Fatalf("missing=%d want 1", missing) - } - if misses != 2 { - t.Fatalf("deadline misses=%d want 2", misses) - } - if bytes != 300 { - t.Fatalf("bytes=%d want 300", bytes) - } - - var r metrics.RunResult - r.TotalWallClockMS = 100 - cs.ApplyToRunResult(&r, 30) - if r.CoordTimeSharePct != 30 { - t.Fatalf("share=%f want 30", r.CoordTimeSharePct) - } -} diff --git a/benchmarks/slim-vs-a2a/internal/metrics/metrics.go b/benchmarks/slim-vs-a2a/internal/metrics/metrics.go index c9ba20e4..d9925cb3 100644 --- a/benchmarks/slim-vs-a2a/internal/metrics/metrics.go +++ b/benchmarks/slim-vs-a2a/internal/metrics/metrics.go @@ -6,42 +6,30 @@ package metrics import ( "encoding/csv" "encoding/json" - "fmt" "os" "path/filepath" "strconv" ) type RunResult struct { - PlanName string `json:"plan_name"` - Domain string `json:"domain"` - Implementation string `json:"implementation"` - Agents int `json:"agents"` - Tasks int `json:"tasks"` - TotalWallClockMS int64 `json:"total_wall_clock_ms"` - TasksCompleted int `json:"tasks_completed"` - TasksFailed int `json:"tasks_failed"` - TasksTimedOut int `json:"tasks_timed_out"` - TasksCancelled int `json:"tasks_cancelled"` - ObsoleteTasksCompleted int `json:"obsolete_tasks_completed"` - RetriesAttempted int `json:"retries_attempted"` - RetriesSucceeded int `json:"retries_succeeded"` - ContextPushMS int64 `json:"context_push_ms"` - SyncBarrierMS int64 `json:"sync_barrier_ms"` - CancelPropagationMS int64 `json:"cancel_propagation_ms"` - ExecuteRPCCount int `json:"execute_rpc_count"` - MulticastRPCCount int `json:"multicast_rpc_count"` - SequentialRPCCount int `json:"sequential_rpc_count"` - MakespanMS int64 `json:"makespan_ms"` - ContextPushP95MS int64 `json:"context_push_p95_ms"` - ContextPushOps int `json:"context_push_ops"` - CoordMissingResponses int `json:"coord_missing_responses"` - CoordDeadlineMisses int `json:"coord_deadline_misses"` - CoordBytesSent int64 `json:"coord_bytes_sent"` - CoordTimeSharePct float64 `json:"coord_time_share_pct"` - RoundBudgetMS int64 `json:"round_budget_ms"` - Success bool `json:"success"` - Error string `json:"error,omitempty"` + ScenarioName string `json:"scenario_name"` + Domain string `json:"domain"` + Implementation string `json:"implementation"` + Agents int `json:"agents"` + ThinkTimeMs int64 `json:"think_time_ms"` + PayloadBytes int `json:"payload_bytes"` + ConsensusWallMS int64 `json:"consensus_wall_ms"` + ConsensusRound int `json:"consensus_round"` + FindingsEmitted int `json:"findings_emitted"` + FindingsReceivedTotal int `json:"findings_received_total"` + AvgPropagationMS int64 `json:"avg_propagation_ms"` + P95PropagationMS int64 `json:"p95_propagation_ms"` + LastAgentConvergeMS int64 `json:"last_agent_converge_ms"` + CoordFanoutMS int64 `json:"coord_fanout_ms"` + StreamRPCCount int `json:"stream_rpc_count"` + UnicastRPCCount int `json:"unicast_rpc_count"` + Success bool `json:"success"` + Error string `json:"error,omitempty"` } func WriteJSON(path string, result RunResult) error { @@ -76,46 +64,33 @@ func AppendTSV(path string, result RunResult) error { w.Comma = '\t' if writeHeader { if err := w.Write([]string{ - "plan_name", "domain", "implementation", "agents", "tasks", - "total_wall_clock_ms", "tasks_completed", "tasks_failed", "tasks_timed_out", - "tasks_cancelled", "obsolete_tasks_completed", "retries_attempted", "retries_succeeded", - "context_push_ms", "sync_barrier_ms", "cancel_propagation_ms", - "execute_rpc_count", "multicast_rpc_count", "sequential_rpc_count", "makespan_ms", - "context_push_p95_ms", "context_push_ops", "coord_missing_responses", "coord_deadline_misses", - "coord_bytes_sent", "coord_time_share_pct", "round_budget_ms", + "scenario_name", "domain", "implementation", "agents", "think_time_ms", + "payload_bytes", + "consensus_wall_ms", "consensus_round", "findings_emitted", "findings_received_total", + "avg_propagation_ms", "p95_propagation_ms", "last_agent_converge_ms", + "coord_fanout_ms", "stream_rpc_count", "unicast_rpc_count", "success", "error", }); err != nil { return err } } if err := w.Write([]string{ - result.PlanName, + result.ScenarioName, result.Domain, result.Implementation, strconv.Itoa(result.Agents), - strconv.Itoa(result.Tasks), - strconv.FormatInt(result.TotalWallClockMS, 10), - strconv.Itoa(result.TasksCompleted), - strconv.Itoa(result.TasksFailed), - strconv.Itoa(result.TasksTimedOut), - strconv.Itoa(result.TasksCancelled), - strconv.Itoa(result.ObsoleteTasksCompleted), - strconv.Itoa(result.RetriesAttempted), - strconv.Itoa(result.RetriesSucceeded), - strconv.FormatInt(result.ContextPushMS, 10), - strconv.FormatInt(result.SyncBarrierMS, 10), - strconv.FormatInt(result.CancelPropagationMS, 10), - strconv.Itoa(result.ExecuteRPCCount), - strconv.Itoa(result.MulticastRPCCount), - strconv.Itoa(result.SequentialRPCCount), - strconv.FormatInt(result.MakespanMS, 10), - strconv.FormatInt(result.ContextPushP95MS, 10), - strconv.Itoa(result.ContextPushOps), - strconv.Itoa(result.CoordMissingResponses), - strconv.Itoa(result.CoordDeadlineMisses), - strconv.FormatInt(result.CoordBytesSent, 10), - strconv.FormatFloat(result.CoordTimeSharePct, 'f', 1, 64), - strconv.FormatInt(result.RoundBudgetMS, 10), + strconv.FormatInt(result.ThinkTimeMs, 10), + strconv.Itoa(result.PayloadBytes), + strconv.FormatInt(result.ConsensusWallMS, 10), + strconv.Itoa(result.ConsensusRound), + strconv.Itoa(result.FindingsEmitted), + strconv.Itoa(result.FindingsReceivedTotal), + strconv.FormatInt(result.AvgPropagationMS, 10), + strconv.FormatInt(result.P95PropagationMS, 10), + strconv.FormatInt(result.LastAgentConvergeMS, 10), + strconv.FormatInt(result.CoordFanoutMS, 10), + strconv.Itoa(result.StreamRPCCount), + strconv.Itoa(result.UnicastRPCCount), strconv.FormatBool(result.Success), result.Error, }); err != nil { @@ -125,18 +100,24 @@ func AppendTSV(path string, result RunResult) error { return w.Error() } -func WriteSummaryMarkdown(path string, results []RunResult) error { - if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { - return err +func AggregatePropagation(durations []int64) (avg, p95 int64) { + if len(durations) == 0 { + return 0, 0 } - var b []byte - b = append(b, "# SLIM vs A2A Comparison Summary\n\n"...) - b = append(b, "| Plan | Implementation | Wall (ms) | Context push (ms) | Cancel prop (ms) | Completed | Failed | Cancelled | Success |\n"...) - b = append(b, "| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |\n"...) - for _, r := range results { - b = append(b, fmt.Sprintf("| %s | %s | %d | %d | %d | %d | %d | %d | %t |\n", - r.PlanName, r.Implementation, r.TotalWallClockMS, r.ContextPushMS, r.CancelPropagationMS, - r.TasksCompleted, r.TasksFailed, r.TasksCancelled, r.Success)...) + var sum int64 + for _, d := range durations { + sum += d + } + avg = sum / int64(len(durations)) + sorted := append([]int64(nil), durations...) + for i := 0; i < len(sorted); i++ { + for j := i + 1; j < len(sorted); j++ { + if sorted[j] < sorted[i] { + sorted[i], sorted[j] = sorted[j], sorted[i] + } + } } - return os.WriteFile(path, b, 0o644) + idx := int(float64(len(sorted)-1) * 0.95) + p95 = sorted[idx] + return avg, p95 } diff --git a/benchmarks/slim-vs-a2a/internal/plan/generator.go b/benchmarks/slim-vs-a2a/internal/plan/generator.go deleted file mode 100644 index fc921b91..00000000 --- a/benchmarks/slim-vs-a2a/internal/plan/generator.go +++ /dev/null @@ -1,231 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package plan - -import ( - "fmt" -) - -// CanonicalFamily identifies paper-aligned micro-plan templates. -type CanonicalFamily string - -const ( - FamilyPreferenceAggregation CanonicalFamily = "preference-aggregation" - FamilySupplyChainCascade CanonicalFamily = "supply-chain-cascade" - FamilySustainableResource CanonicalFamily = "sustainable-resource" -) - -// GenerateOptions configures generated execution plans for sweeps. -type GenerateOptions struct { - Family CanonicalFamily - Agents int - FanOut int - RoundBudgetMS int64 - ExecuteSleepSec float64 - Rounds int -} - -// GenerateCanonical builds an ExecutionPlan for transport/coordination sweeps. -func GenerateCanonical(opts GenerateOptions) (*ExecutionPlan, error) { - if opts.Agents < 2 { - return nil, fmt.Errorf("agents must be >= 2") - } - if opts.ExecuteSleepSec <= 0 { - opts.ExecuteSleepSec = 0.01 - } - if opts.Rounds <= 0 { - opts.Rounds = 1 - } - - switch opts.Family { - case FamilyPreferenceAggregation: - return generatePreferenceAggregation(opts) - case FamilySupplyChainCascade: - return generateSupplyChainCascade(opts) - case FamilySustainableResource: - return generateSustainableResource(opts) - default: - return nil, fmt.Errorf("unknown family %q", opts.Family) - } -} - -func generatePreferenceAggregation(opts GenerateOptions) (*ExecutionPlan, error) { - n := opts.Agents - p := basePlan(string(FamilyPreferenceAggregation), "canonical-preference", opts) - agents := make([]Agent, n) - for i := 0; i < n; i++ { - agents[i] = agentAt(i, 9400+i*10) - } - p.Agents = agents - - var tasks []Task - for i := 0; i < n; i++ { - deps := []string{} - if i > 0 { - deps = []string{fmt.Sprintf("propose-%d", i-1)} - } - tasks = append(tasks, Task{ - ID: fmt.Sprintf("propose-%d", i), - Name: fmt.Sprintf("Agent %d local proposal", i), - Agent: agents[i].ID, - DependsOn: deps, - CompletionTimeSec: opts.ExecuteSleepSec, - MaxCompletionTimeSec: opts.ExecuteSleepSec * 4, - Output: fmt.Sprintf("proposal=score_%d", i*2), - }) - } - p.Tasks = tasks - - lastTask := tasks[len(tasks)-1].ID - targets := agentIDs(agents) - if opts.FanOut > 0 && opts.FanOut < len(targets) { - targets = targets[:opts.FanOut] - } - p.ContextUpdates = []ContextUpdate{{ - AfterTask: lastTask, - Payload: "sync=phase=round-finalize proposals=aggregated", - TargetAgents: targets, - }} - return p, p.Validate() -} - -func generateSupplyChainCascade(opts GenerateOptions) (*ExecutionPlan, error) { - n := opts.Agents - p := basePlan(string(FamilySupplyChainCascade), "canonical-supply-chain", opts) - agents := make([]Agent, n) - for i := 0; i < n; i++ { - agents[i] = agentAt(i, 9500+i*10) - } - p.Agents = agents - - var tasks []Task - for i := 0; i < n; i++ { - deps := []string{} - if i > 0 { - deps = []string{fmt.Sprintf("stage-%d", i-1)} - } - tasks = append(tasks, Task{ - ID: fmt.Sprintf("stage-%d", i), - Name: fmt.Sprintf("Supply chain stage %d order", i), - Agent: agents[i].ID, - DependsOn: deps, - CompletionTimeSec: opts.ExecuteSleepSec, - MaxCompletionTimeSec: opts.ExecuteSleepSec * 4, - Output: fmt.Sprintf("order=units_%d", 4+i), - }) - } - p.Tasks = tasks - - // Per-hop context to downstream neighbor(s). - for i := 0; i < n-1; i++ { - targets := []string{agents[i+1].ID} - p.ContextUpdates = append(p.ContextUpdates, ContextUpdate{ - AfterTask: fmt.Sprintf("stage-%d", i), - Payload: fmt.Sprintf("epoch=%d stock=shared", i+1), - TargetAgents: targets, - }) - } - return p, p.Validate() -} - -func generateSustainableResource(opts GenerateOptions) (*ExecutionPlan, error) { - n := opts.Agents - p := basePlan(string(FamilySustainableResource), "canonical-sustainable-resource", opts) - agents := make([]Agent, n) - for i := 0; i < n; i++ { - agents[i] = agentAt(i, 9600+i*10) - } - p.Agents = agents - - allTargets := agentIDs(agents) - fanOut := len(allTargets) - if opts.FanOut > 0 && opts.FanOut < fanOut { - fanOut = opts.FanOut - } - targets := allTargets[:fanOut] - - var tasks []Task - for r := 0; r < opts.Rounds; r++ { - for i := 0; i < n; i++ { - id := fmt.Sprintf("extract-r%d-a%d", r, i) - deps := []string{} - if r > 0 { - deps = append(deps, fmt.Sprintf("extract-r%d-a%d", r-1, n-1)) - } else if i > 0 { - deps = append(deps, fmt.Sprintf("extract-r%d-a%d", r, i-1)) - } - tasks = append(tasks, Task{ - ID: id, - Name: fmt.Sprintf("Round %d agent %d extraction", r, i), - Agent: agents[i].ID, - DependsOn: deps, - CompletionTimeSec: opts.ExecuteSleepSec, - MaxCompletionTimeSec: opts.ExecuteSleepSec * 4, - Output: fmt.Sprintf("extract=2.0 round=%d", r), - }) - } - // Shared stock broadcast after each round's last agent completes. - p.ContextUpdates = append(p.ContextUpdates, ContextUpdate{ - AfterTask: fmt.Sprintf("extract-r%d-a%d", r, n-1), - Payload: fmt.Sprintf("round=%d stock=shared shadow_price=0.35", r), - TargetAgents: append([]string(nil), targets...), - }) - } - p.Tasks = tasks - return p, p.Validate() -} - -func basePlan(domain, name string, opts GenerateOptions) *ExecutionPlan { - return &ExecutionPlan{ - APIVersion: "bench.agntcy.io/v1", - Kind: "ExecutionPlan", - Metadata: Metadata{ - Name: fmt.Sprintf("%s-%dagents", name, opts.Agents), - Domain: domain, - Description: "Generated canonical plan for transport sweeps", - }, - Spec: Spec{ - Defaults: TaskTiming{ - CompletionTimeSec: opts.ExecuteSleepSec, - MaxCompletionTimeSec: opts.ExecuteSleepSec * 4, - }, - RoundBudgetMS: opts.RoundBudgetMS, - Sweep: &SweepSpec{ - Family: string(opts.Family), - Agents: opts.Agents, - FanOut: opts.FanOut, - RoundBudgetMS: opts.RoundBudgetMS, - ExecuteSleepSec: opts.ExecuteSleepSec, - }, - }, - } -} - -func agentAt(index, portBase int) Agent { - id := fmt.Sprintf("agent-%d", index) - return Agent{ - ID: id, - SlimName: fmt.Sprintf("agntcy/bench/%s", id), - A2APort: portBase + index, - } -} - -func agentIDs(agents []Agent) []string { - ids := make([]string, len(agents)) - for i, a := range agents { - ids[i] = a.ID - } - return ids -} - -// EffectiveRoundBudgetMS returns runner override, plan spec, or zero. -func (p *ExecutionPlan) EffectiveRoundBudgetMS(override int64) int64 { - if override > 0 { - return override - } - if p.Spec.RoundBudgetMS > 0 { - return p.Spec.RoundBudgetMS - } - return 0 -} diff --git a/benchmarks/slim-vs-a2a/internal/plan/generator_test.go b/benchmarks/slim-vs-a2a/internal/plan/generator_test.go deleted file mode 100644 index dfd247c7..00000000 --- a/benchmarks/slim-vs-a2a/internal/plan/generator_test.go +++ /dev/null @@ -1,46 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package plan_test - -import ( - "testing" - - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" -) - -func TestGenerateCanonicalFamilies(t *testing.T) { - families := []plan.CanonicalFamily{ - plan.FamilyPreferenceAggregation, - plan.FamilySupplyChainCascade, - plan.FamilySustainableResource, - } - for _, family := range families { - p, err := plan.GenerateCanonical(plan.GenerateOptions{ - Family: family, - Agents: 5, - RoundBudgetMS: 20, - ExecuteSleepSec: 0.01, - Rounds: 1, - }) - if err != nil { - t.Fatalf("family %s: %v", family, err) - } - if len(p.Agents) != 5 { - t.Fatalf("family %s: expected 5 agents, got %d", family, len(p.Agents)) - } - if p.Spec.RoundBudgetMS != 20 { - t.Fatalf("family %s: round budget not set", family) - } - } -} - -func TestEffectiveRoundBudgetMS(t *testing.T) { - p := &plan.ExecutionPlan{Spec: plan.Spec{RoundBudgetMS: 15}} - if got := p.EffectiveRoundBudgetMS(0); got != 15 { - t.Fatalf("expected 15, got %d", got) - } - if got := p.EffectiveRoundBudgetMS(10); got != 10 { - t.Fatalf("override expected 10, got %d", got) - } -} diff --git a/benchmarks/slim-vs-a2a/internal/plan/plan.go b/benchmarks/slim-vs-a2a/internal/plan/plan.go deleted file mode 100644 index b42a9c85..00000000 --- a/benchmarks/slim-vs-a2a/internal/plan/plan.go +++ /dev/null @@ -1,295 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package plan - -import ( - "fmt" - "os" - "path/filepath" - "strings" - - "gopkg.in/yaml.v3" -) - -type ExecutionPlan struct { - APIVersion string `yaml:"apiVersion"` - Kind string `yaml:"kind"` - Metadata Metadata `yaml:"metadata"` - Spec Spec `yaml:"spec"` - Agents []Agent `yaml:"agents"` - Tasks []Task `yaml:"tasks"` - ContextUpdates []ContextUpdate `yaml:"contextUpdates"` -} - -type Metadata struct { - Name string `yaml:"name"` - Domain string `yaml:"domain"` - Description string `yaml:"description"` -} - -type Spec struct { - Defaults TaskTiming `yaml:"defaults"` - MaxRetries int `yaml:"maxRetries"` - RoundBudgetMS int64 `yaml:"roundBudgetMs"` - Sweep *SweepSpec `yaml:"sweep,omitempty"` -} - -// SweepSpec records parameters used to generate or classify sweep plans. -type SweepSpec struct { - Family string `yaml:"family,omitempty"` - Agents int `yaml:"agents,omitempty"` - FanOut int `yaml:"fanOut,omitempty"` - RoundBudgetMS int64 `yaml:"roundBudgetMs,omitempty"` - ExecuteSleepSec float64 `yaml:"executeSleepSec,omitempty"` -} - -type Agent struct { - ID string `yaml:"id"` - SlimName string `yaml:"slimName"` - A2APort int `yaml:"a2aPort"` -} - -type Task struct { - ID string `yaml:"id"` - Name string `yaml:"name"` - Agent string `yaml:"agent"` - DependsOn []string `yaml:"dependsOn"` - CompletionTimeSec float64 `yaml:"completionTimeSec"` - MaxCompletionTimeSec float64 `yaml:"maxCompletionTimeSec"` - InjectFailure bool `yaml:"injectFailure"` - Output string `yaml:"output"` -} - -type TaskTiming struct { - CompletionTimeSec float64 `yaml:"completionTimeSec"` - MaxCompletionTimeSec float64 `yaml:"maxCompletionTimeSec"` -} - -type ContextUpdate struct { - AfterTask string `yaml:"afterTask"` - Payload string `yaml:"payload"` - TargetAgents []string `yaml:"targetAgents"` -} - -func LoadFile(path string) (*ExecutionPlan, error) { - data, err := os.ReadFile(path) - if err != nil { - return nil, err - } - var p ExecutionPlan - if err := yaml.Unmarshal(data, &p); err != nil { - return nil, fmt.Errorf("parse %s: %w", path, err) - } - if err := p.Validate(); err != nil { - return nil, fmt.Errorf("validate %s: %w", filepath.Base(path), err) - } - p.applyDefaults() - return &p, nil -} - -// WriteFile marshals an execution plan to a YAML file. -func WriteFile(path string, p *ExecutionPlan) error { - if err := p.Validate(); err != nil { - return err - } - data, err := yaml.Marshal(p) - if err != nil { - return err - } - if dir := filepath.Dir(path); dir != "" && dir != "." { - if err := os.MkdirAll(dir, 0o755); err != nil { - return err - } - } - return os.WriteFile(path, data, 0o644) -} - -func LoadDir(dir string) ([]*ExecutionPlan, error) { - entries, err := os.ReadDir(dir) - if err != nil { - return nil, err - } - var plans []*ExecutionPlan - for _, entry := range entries { - if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".yaml") { - continue - } - p, err := LoadFile(filepath.Join(dir, entry.Name())) - if err != nil { - return nil, err - } - plans = append(plans, p) - } - if len(plans) == 0 { - return nil, fmt.Errorf("no plan yaml files in %s", dir) - } - return plans, nil -} - -func (p *ExecutionPlan) applyDefaults() { - for i := range p.Tasks { - if p.Tasks[i].CompletionTimeSec == 0 { - p.Tasks[i].CompletionTimeSec = p.Spec.Defaults.CompletionTimeSec - } - if p.Tasks[i].MaxCompletionTimeSec == 0 { - p.Tasks[i].MaxCompletionTimeSec = p.Spec.Defaults.MaxCompletionTimeSec - } - } -} - -func (p *ExecutionPlan) Validate() error { - if p.Metadata.Name == "" { - return fmt.Errorf("metadata.name is required") - } - if len(p.Agents) == 0 { - return fmt.Errorf("at least one agent is required") - } - if len(p.Tasks) == 0 { - return fmt.Errorf("at least one task is required") - } - - agentIDs := map[string]Agent{} - for _, a := range p.Agents { - if a.ID == "" { - return fmt.Errorf("agent id is required") - } - if _, ok := agentIDs[a.ID]; ok { - return fmt.Errorf("duplicate agent id %q", a.ID) - } - agentIDs[a.ID] = a - } - - taskIDs := map[string]struct{}{} - for _, t := range p.Tasks { - if t.ID == "" { - return fmt.Errorf("task id is required") - } - if _, ok := taskIDs[t.ID]; ok { - return fmt.Errorf("duplicate task id %q", t.ID) - } - taskIDs[t.ID] = struct{}{} - } - - for _, t := range p.Tasks { - if _, ok := agentIDs[t.Agent]; !ok { - return fmt.Errorf("task %q references unknown agent %q", t.ID, t.Agent) - } - for _, dep := range t.DependsOn { - if _, ok := taskIDs[dep]; !ok { - return fmt.Errorf("task %q depends on unknown task %q", t.ID, dep) - } - } - if t.MaxCompletionTimeSec > 0 && t.MaxCompletionTimeSec < t.CompletionTimeSec { - return fmt.Errorf("task %q maxCompletionTimeSec < completionTimeSec", t.ID) - } - } - - for _, cu := range p.ContextUpdates { - if _, ok := taskIDs[cu.AfterTask]; !ok { - return fmt.Errorf("contextUpdate references unknown task %q", cu.AfterTask) - } - for _, agentID := range cu.TargetAgents { - if _, ok := agentIDs[agentID]; !ok { - return fmt.Errorf("contextUpdate references unknown agent %q", agentID) - } - } - } - - if err := detectCycle(p.Tasks); err != nil { - return err - } - return nil -} - -func (p *ExecutionPlan) AgentByID(id string) (Agent, bool) { - for _, a := range p.Agents { - if a.ID == id { - return a, true - } - } - return Agent{}, false -} - -func (p *ExecutionPlan) TaskByID(id string) (Task, bool) { - for _, t := range p.Tasks { - if t.ID == id { - return t, true - } - } - return Task{}, false -} - -func (p *ExecutionPlan) ContextUpdatesAfter(taskID string) []ContextUpdate { - var out []ContextUpdate - for _, cu := range p.ContextUpdates { - if cu.AfterTask == taskID { - out = append(out, cu) - } - } - return out -} - -func (p *ExecutionPlan) CardPort(agent Agent) int { - return agent.A2APort - 1000 -} - -func (p *ExecutionPlan) CardBaseURL(agent Agent) string { - return fmt.Sprintf("http://127.0.0.1:%d", p.CardPort(agent)) -} - -func detectCycle(tasks []Task) error { - graph := map[string][]string{} - for _, t := range tasks { - graph[t.ID] = append([]string(nil), t.DependsOn...) - } - visiting := map[string]bool{} - visited := map[string]bool{} - var dfs func(string) error - dfs = func(id string) error { - if visiting[id] { - return fmt.Errorf("cycle detected at task %q", id) - } - if visited[id] { - return nil - } - visiting[id] = true - for _, dep := range graph[id] { - if err := dfs(dep); err != nil { - return err - } - } - visiting[id] = false - visited[id] = true - return nil - } - for id := range graph { - if err := dfs(id); err != nil { - return err - } - } - return nil -} - -func DownstreamTasks(tasks []Task, rootID string) map[string]struct{} { - dependents := map[string][]string{} - for _, t := range tasks { - for _, dep := range t.DependsOn { - dependents[dep] = append(dependents[dep], t.ID) - } - } - out := map[string]struct{}{} - queue := []string{rootID} - for len(queue) > 0 { - cur := queue[0] - queue = queue[1:] - for _, child := range dependents[cur] { - if _, ok := out[child]; ok { - continue - } - out[child] = struct{}{} - queue = append(queue, child) - } - } - return out -} diff --git a/benchmarks/slim-vs-a2a-v2/internal/scenario/scenario.go b/benchmarks/slim-vs-a2a/internal/scenario/scenario.go similarity index 91% rename from benchmarks/slim-vs-a2a-v2/internal/scenario/scenario.go rename to benchmarks/slim-vs-a2a/internal/scenario/scenario.go index ab9e6311..966485f3 100644 --- a/benchmarks/slim-vs-a2a-v2/internal/scenario/scenario.go +++ b/benchmarks/slim-vs-a2a/internal/scenario/scenario.go @@ -37,6 +37,9 @@ type Spec struct { TargetMode string `yaml:"targetMode"` Seed int64 `yaml:"seed"` ValueSpace int `yaml:"valueSpace"` + // PayloadBytes inflates every finding with fixed-size padding to stress + // transport bandwidth. 0 (default) keeps findings at their minimal size. + PayloadBytes int `yaml:"payloadBytes,omitempty"` } type Agent struct { @@ -93,6 +96,9 @@ func (s *ConsensusScenario) Validate() error { if s.Spec.FindingEmitDelayMs <= 0 { s.Spec.FindingEmitDelayMs = 1 } + if s.Spec.PayloadBytes < 0 { + s.Spec.PayloadBytes = 0 + } for i, a := range s.Agents { if a.ID == "" { return fmt.Errorf("agents[%d].id is required", i) @@ -164,10 +170,11 @@ func (s *ConsensusScenario) SlimNames() []string { } type GenerateOptions struct { - Family string - Agents int - ThinkTimeMs int64 - Seed int64 + Family string + Agents int + ThinkTimeMs int64 + Seed int64 + PayloadBytes int } func Generate(opts GenerateOptions) (*ConsensusScenario, error) { @@ -183,8 +190,14 @@ func Generate(opts GenerateOptions) (*ConsensusScenario, error) { if opts.Seed == 0 { opts.Seed = 42 } + if opts.PayloadBytes < 0 { + opts.PayloadBytes = 0 + } name := fmt.Sprintf("%s-%dagents-%dms", opts.Family, opts.Agents, opts.ThinkTimeMs) + if opts.PayloadBytes > 0 { + name = fmt.Sprintf("%s-%db", name, opts.PayloadBytes) + } agents := make([]Agent, opts.Agents) for i := 0; i < opts.Agents; i++ { id := fmt.Sprintf("agent-%d", i) @@ -216,6 +229,7 @@ func Generate(opts GenerateOptions) (*ConsensusScenario, error) { TargetMode: TargetModeMajority, Seed: opts.Seed, ValueSpace: 3, + PayloadBytes: opts.PayloadBytes, }, Agents: agents, } diff --git a/benchmarks/slim-vs-a2a-v2/internal/scenario/scenario_test.go b/benchmarks/slim-vs-a2a/internal/scenario/scenario_test.go similarity index 87% rename from benchmarks/slim-vs-a2a-v2/internal/scenario/scenario_test.go rename to benchmarks/slim-vs-a2a/internal/scenario/scenario_test.go index f39aad08..3cb6c1a5 100644 --- a/benchmarks/slim-vs-a2a-v2/internal/scenario/scenario_test.go +++ b/benchmarks/slim-vs-a2a/internal/scenario/scenario_test.go @@ -6,7 +6,7 @@ package scenario_test import ( "testing" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" ) func TestGenerateAndValidate(t *testing.T) { diff --git a/benchmarks/slim-vs-a2a/plans/README.md b/benchmarks/slim-vs-a2a/plans/README.md index 8ea79c5e..ef5afb3e 100644 --- a/benchmarks/slim-vs-a2a/plans/README.md +++ b/benchmarks/slim-vs-a2a/plans/README.md @@ -1,57 +1,90 @@ -# Execution plans (Phase 1) +# Consensus scenarios (v2) -Hand-authored YAML DAG plans for the SLIM vs A2A multi-agent benchmark. -Each plan defines named agents, interdependent tasks (mocked via timing fields), -optional failure injection, and context updates that exercise sync / context -sharing / failure propagation. +YAML `ConsensusScenario` plans for the SLIM vs A2A v2 consensus streaming benchmark. +Each scenario defines N agents that run a deterministic **distributed hypothesis convergence** +workload: agents think in parallel, emit findings, and must reach identical local consensus. -## Plans +## Topologies compared -| File | Domain | Agents | Tasks | Notes | -|------|--------|--------|-------|-------| -| [k8s-incident-response.yaml](domains/k8s-incident-response.yaml) | Kubernetes troubleshooting | 6 | 16 | OOM diagnosis; `probe-service-mesh` injects failure; 2 context updates | -| [urban-traffic-reroute.yaml](domains/urban-traffic-reroute.yaml) | Realtime traffic | 6 | 14 | I-5 closure; N/S/E/W parallel route recalc; detour cancel on ETA revision | -| [mobile-nav-assistant.yaml](domains/mobile-nav-assistant.yaml) | Android voice navigation | 6 | 12 | Pharmacy→airport intent; coffee-first amendment obsoletes pharmacy branch | +The two implementations highlight one structural difference: + +- **SLIM (`slim-group-session`)** — agents join a native SLIM group session and + broadcast findings peer-to-peer; the dataplane fans them out. **There is no + application relay.** The runner only moderates (creates the session, invites + agents, sends one start) and then passively observes snapshots that agents + push to it on convergence. +- **A2A (`a2a-relay-stream`)** — A2A has no peer multicast, so the runner is an + explicit **relay hub**. Two server-streaming legs carry findings: agents stream + findings to the runner (`OpStreamFindings`) and subscribe to the runner's + relayed-finding stream (`OpStreamRelay`). Every finding therefore makes an + extra relay hop, which is the overhead `consensus_wall_ms` is expected to show. ## Schema ```yaml -apiVersion: bench.agntcy.io/v1 -kind: ExecutionPlan +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario metadata: - name: - domain: - description: + name: hypothesis-convergence-10agents-10ms + domain: hypothesis-convergence spec: - defaults: - completionTimeSec: - maxCompletionTimeSec: - maxRetries: + agents: 10 + thinkTimeMs: 10 + findingEmitDelayMs: 1 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 + payloadBytes: 0 # optional fixed-size padding added to every finding agents: - - id: - slimName: agntcy/bench/ - a2aPort: # unique within plan; k8s=91xx, traffic=92xx, mobile=93xx -tasks: - - id: - name: - agent: - dependsOn: [, ...] - completionTimeSec: - maxCompletionTimeSec: - injectFailure: # optional; default false - output: -contextUpdates: # optional - - afterTask: - payload: - targetAgents: [, ...] + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: worker # legacy field; both transports now treat all agents as peers + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker ``` -## Port allocation +## Generate sweep scenarios + +```bash +go run ./tools/gen_scenario \ + -family hypothesis-convergence \ + -agents 10 \ + -think-ms 20 \ + -payload-bytes 0 \ + -output plans/sweeps/hypothesis-convergence-10ag-20ms.yaml +``` -Ports are unique per plan to allow isolated local runs: +### Payload size (transport-stress) knob -- **k8s-incident-response:** A2A gRPC `9101`–`9106` -- **urban-traffic-reroute:** A2A gRPC `9201`–`9206` -- **mobile-nav-assistant:** A2A gRPC `9301`–`9306` +`spec.payloadBytes` adds fixed-size deterministic padding to every finding. It is +semantically inert (it does not affect the consensus math or the number of +rounds) — it only inflates the wire size of each finding. Because A2A relays each +finding twice through the central runner (`agent → runner → N−1 agents`) while +SLIM publishes once and the dataplane fans out, larger payloads disproportionately +load the A2A relay. Combine it with high agent counts and a short think window to +make the multicast advantage most visible: + +```bash +# Sweep agents × payload at a 5ms think window (0 vs 10kB), then build the report. +task compare:sweep:payload \ + SWEEP_AGENTS=10,50 SWEEP_THINK_MS=5 SWEEP_PAYLOAD_BYTES=0,10240 +``` + +> Caveat: a2a-go's streaming task store appends each relayed finding to the task +> history and deep-copies the task per update, so very large payloads also amplify +> A2A cost super-linearly. That is a property of the A2A streaming/task model, but +> note it when attributing the delta. + +## Run comparison + +```bash +task build +task compare:plan PLAN=hypothesis-convergence-5ag-20ms +task compare:report +``` -SLIM identities use `agntcy/bench/` from each plan's `agents[].slimName`. +Primary metric: `consensus_wall_ms` in `reports/results.tsv`. diff --git a/benchmarks/slim-vs-a2a/plans/domains/k8s-incident-response.yaml b/benchmarks/slim-vs-a2a/plans/domains/k8s-incident-response.yaml deleted file mode 100644 index b99f09e5..00000000 --- a/benchmarks/slim-vs-a2a/plans/domains/k8s-incident-response.yaml +++ /dev/null @@ -1,173 +0,0 @@ -apiVersion: bench.agntcy.io/v1 -kind: ExecutionPlan -metadata: - name: k8s-incident-response - domain: kubernetes-troubleshooting - description: > - checkout-api deployment in CrashLoopBackOff; agents diagnose in parallel. - OOM root cause confirmed early cancels speculative network partition and - node-drain branches. probe-service-mesh injects failure to exercise abort propagation. -spec: - defaults: - completionTimeSec: 0.3 - maxCompletionTimeSec: 2.0 - maxRetries: 1 - -agents: - - id: cluster-scout - slimName: agntcy/bench/cluster-scout - a2aPort: 9101 - - id: log-miner - slimName: agntcy/bench/log-miner - a2aPort: 9102 - - id: metrics-analyst - slimName: agntcy/bench/metrics-analyst - a2aPort: 9103 - - id: event-correlator - slimName: agntcy/bench/event-correlator - a2aPort: 9104 - - id: network-prober - slimName: agntcy/bench/network-prober - a2aPort: 9105 - - id: runbook-executor - slimName: agntcy/bench/runbook-executor - a2aPort: 9106 - -tasks: - - id: fetch-pod-status - name: Fetch pod status for checkout-api deployment - agent: cluster-scout - dependsOn: [] - completionTimeSec: 0.25 - maxCompletionTimeSec: 1.0 - output: "deployment=checkout-api phase=CrashLoopBackOff replicas=3/5 namespace=prod" - - - id: list-recent-events - name: List Kubernetes events for checkout-api pods - agent: event-correlator - dependsOn: [fetch-pod-status] - completionTimeSec: 0.35 - maxCompletionTimeSec: 1.2 - output: "events=47 warning=BackOff reason=FailedScheduling count=12" - - - id: scrape-container-logs - name: Scrape last 500 lines from crashing checkout-api pod - agent: log-miner - dependsOn: [fetch-pod-status] - completionTimeSec: 0.55 - maxCompletionTimeSec: 1.5 - output: "container=checkout-api OOMKilled exit=137 lastLine=java.lang.OutOfMemoryError" - - - id: query-memory-metrics - name: Query Prometheus memory usage for checkout-api - agent: metrics-analyst - dependsOn: [fetch-pod-status] - completionTimeSec: 0.4 - maxCompletionTimeSec: 1.3 - output: "metric=container_memory_working_set_bytes value=512Mi limit=512Mi breached=true" - - - id: query-cpu-metrics - name: Query Prometheus CPU throttling for checkout-api - agent: metrics-analyst - dependsOn: [fetch-pod-status] - completionTimeSec: 0.35 - maxCompletionTimeSec: 1.2 - output: "metric=container_cpu_cfs_throttled_seconds_total value=0.02 throttled=false" - - - id: probe-dns-resolution - name: Probe cluster DNS resolution for checkout-api service - agent: network-prober - dependsOn: [fetch-pod-status] - completionTimeSec: 0.45 - maxCompletionTimeSec: 1.4 - output: "service=checkout-api.svc.cluster.local resolved=10.96.142.88 latency_ms=3" - - - id: probe-service-mesh - name: Probe Istio sidecar connectivity for checkout-api pod - agent: network-prober - dependsOn: [fetch-pod-status] - completionTimeSec: 0.5 - maxCompletionTimeSec: 1.0 - injectFailure: true - output: "sidecar=istio-proxy status=503 upstream=checkout-api:8080" - - - id: analyze-gc-logs - name: Analyze JVM GC patterns in container logs - agent: log-miner - dependsOn: [scrape-container-logs] - completionTimeSec: 0.5 - maxCompletionTimeSec: 1.6 - output: "gc=G1Full frequent=true heap=512Mi postGC=498Mi" - - - id: correlate-oom-events - name: Correlate OOMKilled events with pod restart timeline - agent: event-correlator - dependsOn: [scrape-container-logs, list-recent-events] - completionTimeSec: 0.45 - maxCompletionTimeSec: 1.5 - output: "correlation=strong restarts=18 window=15m cause=OOMKilled" - - - id: check-node-capacity - name: Check node allocatable memory on worker pool - agent: cluster-scout - dependsOn: [query-memory-metrics] - completionTimeSec: 0.4 - maxCompletionTimeSec: 1.5 - output: "node=worker-3 allocatable_memory=32Gi pressure=low" - - - id: simulate-node-drain - name: Simulate cordon and drain of suspect worker node - agent: runbook-executor - dependsOn: [check-node-capacity, probe-service-mesh] - completionTimeSec: 0.8 - maxCompletionTimeSec: 2.5 - output: "node=worker-3 cordoned=false simulation=aborted" - - - id: confirm-root-cause-oom - name: Confirm memory limit breach as root cause - agent: event-correlator - dependsOn: [correlate-oom-events, analyze-gc-logs, query-memory-metrics] - completionTimeSec: 0.3 - maxCompletionTimeSec: 1.0 - output: "root_cause=memory_limit_breach confidence=0.97 action=raise_limit" - - - id: apply-memory-limit-patch - name: Patch deployment memory limit from 512Mi to 1Gi - agent: runbook-executor - dependsOn: [confirm-root-cause-oom] - completionTimeSec: 0.6 - maxCompletionTimeSec: 2.0 - output: "deployment=checkout-api memory_limit=1Gi patch=applied rollout=started" - - - id: simulate-rollback - name: Simulate rollback to previous checkout-api revision - agent: runbook-executor - dependsOn: [probe-service-mesh] - completionTimeSec: 0.7 - maxCompletionTimeSec: 2.0 - output: "revision=checkout-api-42 rollback=skipped reason=upstream_probe_failed" - - - id: verify-pod-health - name: Verify checkout-api pods reach Running state - agent: cluster-scout - dependsOn: [apply-memory-limit-patch] - completionTimeSec: 0.5 - maxCompletionTimeSec: 2.5 - output: "deployment=checkout-api phase=Running replicas=5/5 ready=true" - - - id: publish-incident-summary - name: Publish incident timeline to status page - agent: event-correlator - dependsOn: [verify-pod-health] - completionTimeSec: 0.25 - maxCompletionTimeSec: 1.0 - output: "incident=INC-2847 status=resolved duration=23m root_cause=OOMKilled" - -contextUpdates: - - afterTask: fetch-pod-status - payload: "hypothesis=memory-pressure priority=high focus=checkout-api namespace=prod" - targetAgents: [log-miner, metrics-analyst, event-correlator, network-prober] - - - afterTask: confirm-root-cause-oom - payload: "root_cause=memory_limit_breach cancel_speculative=true abort=node-drain,mesh-rollback" - targetAgents: [network-prober, runbook-executor, cluster-scout] diff --git a/benchmarks/slim-vs-a2a/plans/domains/mobile-nav-assistant.yaml b/benchmarks/slim-vs-a2a/plans/domains/mobile-nav-assistant.yaml deleted file mode 100644 index 7f36f55e..00000000 --- a/benchmarks/slim-vs-a2a/plans/domains/mobile-nav-assistant.yaml +++ /dev/null @@ -1,140 +0,0 @@ -apiVersion: bench.agntcy.io/v1 -kind: ExecutionPlan -metadata: - name: mobile-nav-assistant - domain: mobile-voice-navigation - description: > - Android user asks for pharmacy on the way to the airport; agents parse intent, - search POIs, and compute routes. Mid-flight command change to coffee-first - obsoletes pharmacy branch via context update. -spec: - defaults: - completionTimeSec: 0.3 - maxCompletionTimeSec: 2.0 - maxRetries: 1 - -agents: - - id: intent-parser - slimName: agntcy/bench/intent-parser - a2aPort: 9301 - - id: poi-searcher - slimName: agntcy/bench/poi-searcher - a2aPort: 9302 - - id: route-engine - slimName: agntcy/bench/route-engine - a2aPort: 9303 - - id: traffic-watcher - slimName: agntcy/bench/traffic-watcher - a2aPort: 9304 - - id: ui-composer - slimName: agntcy/bench/ui-composer - a2aPort: 9305 - - id: session-tracker - slimName: agntcy/bench/session-tracker - a2aPort: 9306 - -tasks: - - id: parse-voice-command - name: Parse voice command for pharmacy stop en route to airport - agent: intent-parser - dependsOn: [] - completionTimeSec: 0.25 - maxCompletionTimeSec: 1.0 - output: "intent=navigate_with_stop destination=SEA_airport stop=pharmacy open_now=true" - - - id: sync-user-location - name: Sync current GPS location and heading from Android device - agent: session-tracker - dependsOn: [parse-voice-command] - completionTimeSec: 0.15 - maxCompletionTimeSec: 0.6 - output: "lat=47.6062 lon=-122.3321 heading=145 speed_kmh=32 accuracy_m=8" - - - id: geocode-airport-destination - name: Geocode Seattle-Tacoma International Airport destination - agent: poi-searcher - dependsOn: [parse-voice-command] - completionTimeSec: 0.2 - maxCompletionTimeSec: 0.8 - output: "poi=SEA_airport lat=47.4502 lon=-122.3088 place_id=ChIJSeaTac" - - - id: search-pharmacy-near-route - name: Search open pharmacies along route to airport - agent: poi-searcher - dependsOn: [parse-voice-command, sync-user-location] - completionTimeSec: 0.55 - maxCompletionTimeSec: 1.8 - output: "poi=CVS_Queen_Anne lat=47.6234 lon=-122.3571 open=true detour_min=4" - - - id: search-coffee-near-route - name: Search coffee shops along route to airport - agent: poi-searcher - dependsOn: [parse-voice-command, sync-user-location] - completionTimeSec: 0.5 - maxCompletionTimeSec: 1.8 - output: "poi=Starbucks_Belltown lat=47.6145 lon=-122.3498 open=true detour_min=2" - - - id: rank-stop-candidates - name: Rank stop candidates by detour time and user preferences - agent: intent-parser - dependsOn: [search-pharmacy-near-route, search-coffee-near-route] - completionTimeSec: 0.3 - maxCompletionTimeSec: 1.0 - output: "ranked=pharmacy,coffee preferred=pharmacy detour_budget_min=8" - - - id: compute-route-to-airport-via-pharmacy - name: Compute turn-by-turn route to airport via CVS Queen Anne - agent: route-engine - dependsOn: [search-pharmacy-near-route, geocode-airport-destination] - completionTimeSec: 0.6 - maxCompletionTimeSec: 2.0 - output: "route_id=R-pharm-01 distance_km=28.4 eta_min=34 stops=1 via=CVS_Queen_Anne" - - - id: compute-route-to-airport-via-coffee - name: Compute turn-by-turn route to airport via Starbucks Belltown - agent: route-engine - dependsOn: [search-coffee-near-route, geocode-airport-destination] - completionTimeSec: 0.55 - maxCompletionTimeSec: 2.0 - output: "route_id=R-coffee-01 distance_km=27.1 eta_min=31 stops=1 via=Starbucks_Belltown" - - - id: refine-route-live-traffic - name: Refine active route with live traffic and incident data - agent: traffic-watcher - dependsOn: [compute-route-to-airport-via-pharmacy, compute-route-to-airport-via-coffee] - completionTimeSec: 0.45 - maxCompletionTimeSec: 1.5 - output: "route_id=R-coffee-01 eta_min=33 delay_min=+2 incident=I-5_slowdown" - - - id: render-map-overlay - name: Render map overlay and stop markers on Android UI - agent: ui-composer - dependsOn: [refine-route-live-traffic, rank-stop-candidates] - completionTimeSec: 0.35 - maxCompletionTimeSec: 1.2 - output: "overlay=turn_by_turn markers=2 stop=coffee theme=night_mode" - - - id: push-voice-prompt - name: Push voice prompt confirming coffee stop and updated ETA - agent: ui-composer - dependsOn: [render-map-overlay] - completionTimeSec: 0.2 - maxCompletionTimeSec: 0.8 - output: "tts=Routing via Starbucks Belltown. ETA 33 minutes. Say change stop to adjust." - - - id: update-session-history - name: Update session history with amended navigation intent - agent: session-tracker - dependsOn: [push-voice-prompt] - completionTimeSec: 0.15 - maxCompletionTimeSec: 0.6 - output: "session=android-7f3a intent_history=2 priority=coffee-first destination=SEA_airport" - -contextUpdates: - - afterTask: parse-voice-command - payload: "user_amendment=actually_coffee_first priority=coffee-first cancel=pharmacy_branch" - targetAgents: [poi-searcher, route-engine, intent-parser, ui-composer] - - - afterTask: refine-route-live-traffic - payload: "active_route=R-coffee-01 eta_min=33 sync=ui,voice session=android-7f3a" - targetAgents: [ui-composer, session-tracker, traffic-watcher] diff --git a/benchmarks/slim-vs-a2a/plans/domains/urban-traffic-reroute.yaml b/benchmarks/slim-vs-a2a/plans/domains/urban-traffic-reroute.yaml deleted file mode 100644 index 5856b057..00000000 --- a/benchmarks/slim-vs-a2a/plans/domains/urban-traffic-reroute.yaml +++ /dev/null @@ -1,156 +0,0 @@ -apiVersion: bench.agntcy.io/v1 -kind: ExecutionPlan -metadata: - name: urban-traffic-reroute - domain: realtime-traffic-management - description: > - Highway I-5 northbound lane closure after accident; agents recompute routes, - adjust signals, and reroute transit. Corridor reopening ETA revision cancels - stale detour plans via context update. -spec: - defaults: - completionTimeSec: 0.35 - maxCompletionTimeSec: 2.0 - maxRetries: 1 - -agents: - - id: incident-detector - slimName: agntcy/bench/incident-detector - a2aPort: 9201 - - id: flow-modeler - slimName: agntcy/bench/flow-modeler - a2aPort: 9202 - - id: route-planner - slimName: agntcy/bench/route-planner - a2aPort: 9203 - - id: signal-coordinator - slimName: agntcy/bench/signal-coordinator - a2aPort: 9204 - - id: transit-dispatcher - slimName: agntcy/bench/transit-dispatcher - a2aPort: 9205 - - id: alert-broadcaster - slimName: agntcy/bench/alert-broadcaster - a2aPort: 9206 - -tasks: - - id: detect-lane-closure - name: Detect lane closure from camera and WIM sensor feed - agent: incident-detector - dependsOn: [] - completionTimeSec: 0.2 - maxCompletionTimeSec: 0.8 - output: "highway=I-5 direction=northbound lanes_closed=2 location=mile-172 accident=true" - - - id: estimate-queue-length - name: Estimate queue length and delay on affected corridor - agent: flow-modeler - dependsOn: [detect-lane-closure] - completionTimeSec: 0.45 - maxCompletionTimeSec: 1.5 - output: "queue_km=4.2 delay_min=38 reopening_eta_min=45 corridor=I-5-N" - - - id: model-traffic-impedance - name: Model traffic impedance across downtown grid - agent: flow-modeler - dependsOn: [estimate-queue-length] - completionTimeSec: 0.5 - maxCompletionTimeSec: 1.8 - output: "impedance_matrix=12x12 hotspots=3 max_delay_min=22" - - - id: plan-detour-via-highway-99 - name: Plan corridor-wide detour via Highway 99 - agent: route-planner - dependsOn: [estimate-queue-length] - completionTimeSec: 0.7 - maxCompletionTimeSec: 2.5 - output: "detour=WA-99 added_distance_km=18 eta_min=52 status=proposed" - - - id: plan-detour-via-surface-streets - name: Plan local detour via Aurora and surface streets - agent: route-planner - dependsOn: [estimate-queue-length] - completionTimeSec: 0.65 - maxCompletionTimeSec: 2.5 - output: "detour=aurora-ave added_distance_km=9 eta_min=41 status=proposed" - - - id: recalc-northbound-routes - name: Recalculate northbound dynamic routes for active fleet - agent: route-planner - dependsOn: [model-traffic-impedance] - completionTimeSec: 0.55 - maxCompletionTimeSec: 1.8 - output: "direction=northbound routes_updated=12400 avg_delay_delta_min=+6" - - - id: recalc-southbound-routes - name: Recalculate southbound dynamic routes for active fleet - agent: route-planner - dependsOn: [model-traffic-impedance] - completionTimeSec: 0.5 - maxCompletionTimeSec: 1.8 - output: "direction=southbound routes_updated=11800 avg_delay_delta_min=+4" - - - id: recalc-eastbound-routes - name: Recalculate eastbound dynamic routes for active fleet - agent: route-planner - dependsOn: [model-traffic-impedance] - completionTimeSec: 0.48 - maxCompletionTimeSec: 1.8 - output: "direction=eastbound routes_updated=9200 avg_delay_delta_min=+3" - - - id: recalc-westbound-routes - name: Recalculate westbound dynamic routes for active fleet - agent: route-planner - dependsOn: [model-traffic-impedance] - completionTimeSec: 0.52 - maxCompletionTimeSec: 1.8 - output: "direction=westbound routes_updated=8900 avg_delay_delta_min=+3" - - - id: adjust-signal-phasing - name: Adjust intersection signal phasing on alternate corridors - agent: signal-coordinator - dependsOn: [recalc-northbound-routes, recalc-southbound-routes] - completionTimeSec: 0.4 - maxCompletionTimeSec: 1.5 - output: "intersections=37 green_extension_sec=12 corridor_bias=north-south" - - - id: reroute-bus-line-42 - name: Reroute King County Metro bus line 42 - agent: transit-dispatcher - dependsOn: [recalc-eastbound-routes, plan-detour-via-surface-streets] - completionTimeSec: 0.45 - maxCompletionTimeSec: 1.6 - output: "line=42 stops_bypassed=6 alternate=aurora-ave passengers_notified=820" - - - id: reroute-rail-shuttle - name: Reroute Sound Transit shuttle feeders - agent: transit-dispatcher - dependsOn: [recalc-westbound-routes] - completionTimeSec: 0.42 - maxCompletionTimeSec: 1.6 - output: "shuttle=ST-Express-550 headway_min=8 alternate_hub=Northgate" - - - id: publish-driver-alerts - name: Publish Waze and 511 driver alerts for I-5 closure - agent: alert-broadcaster - dependsOn: [adjust-signal-phasing, reroute-bus-line-42, reroute-rail-shuttle] - completionTimeSec: 0.3 - maxCompletionTimeSec: 1.2 - output: "channels=waze,511 recipients=840000 message=I-5NB_lane_closure_m172" - - - id: sync-incident-dashboard - name: Sync traffic operations center incident dashboard - agent: incident-detector - dependsOn: [publish-driver-alerts] - completionTimeSec: 0.25 - maxCompletionTimeSec: 1.0 - output: "dashboard=TOC-West incident=TRF-9182 status=active mitigation=deployed" - -contextUpdates: - - afterTask: estimate-queue-length - payload: "reopening_eta_min=12 corridor=I-5-N cancel_detours=true priority=primary_corridor" - targetAgents: [route-planner, transit-dispatcher, signal-coordinator] - - - afterTask: model-traffic-impedance - payload: "phase=route_recalc sync=all_directions timestamp=2026-06-04T14:32:00Z" - targetAgents: [route-planner, signal-coordinator, alert-broadcaster] diff --git a/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml similarity index 100% rename from benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml rename to benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms-10240b.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms-10240b.yaml new file mode 100644 index 00000000..abdd9f0f --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms-10240b.yaml @@ -0,0 +1,56 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-10agents-5ms-10240b + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 10 + thinkTimeMs: 5 + findingEmitDelayMs: 1 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 + payloadBytes: 10240 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker + - id: agent-5 + slimName: agntcy/bench-v2/agent-5 + a2aPort: 9755 + role: worker + - id: agent-6 + slimName: agntcy/bench-v2/agent-6 + a2aPort: 9766 + role: worker + - id: agent-7 + slimName: agntcy/bench-v2/agent-7 + a2aPort: 9777 + role: worker + - id: agent-8 + slimName: agntcy/bench-v2/agent-8 + a2aPort: 9788 + role: worker + - id: agent-9 + slimName: agntcy/bench-v2/agent-9 + a2aPort: 9799 + role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms.yaml new file mode 100644 index 00000000..7240d3f6 --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms.yaml @@ -0,0 +1,55 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-10agents-5ms + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 10 + thinkTimeMs: 5 + findingEmitDelayMs: 1 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker + - id: agent-5 + slimName: agntcy/bench-v2/agent-5 + a2aPort: 9755 + role: worker + - id: agent-6 + slimName: agntcy/bench-v2/agent-6 + a2aPort: 9766 + role: worker + - id: agent-7 + slimName: agntcy/bench-v2/agent-7 + a2aPort: 9777 + role: worker + - id: agent-8 + slimName: agntcy/bench-v2/agent-8 + a2aPort: 9788 + role: worker + - id: agent-9 + slimName: agntcy/bench-v2/agent-9 + a2aPort: 9799 + role: worker diff --git a/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-17ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-17ag-20ms.yaml similarity index 100% rename from benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-17ag-20ms.yaml rename to benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-17ag-20ms.yaml diff --git a/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-20ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-20ag-20ms.yaml similarity index 100% rename from benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-20ag-20ms.yaml rename to benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-20ag-20ms.yaml diff --git a/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-50ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-20ms.yaml similarity index 100% rename from benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-50ag-20ms.yaml rename to benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-20ms.yaml diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms-10240b.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms-10240b.yaml new file mode 100644 index 00000000..898f8725 --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms-10240b.yaml @@ -0,0 +1,216 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-50agents-5ms-10240b + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 50 + thinkTimeMs: 5 + findingEmitDelayMs: 1 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 + payloadBytes: 10240 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker + - id: agent-5 + slimName: agntcy/bench-v2/agent-5 + a2aPort: 9755 + role: worker + - id: agent-6 + slimName: agntcy/bench-v2/agent-6 + a2aPort: 9766 + role: worker + - id: agent-7 + slimName: agntcy/bench-v2/agent-7 + a2aPort: 9777 + role: worker + - id: agent-8 + slimName: agntcy/bench-v2/agent-8 + a2aPort: 9788 + role: worker + - id: agent-9 + slimName: agntcy/bench-v2/agent-9 + a2aPort: 9799 + role: worker + - id: agent-10 + slimName: agntcy/bench-v2/agent-10 + a2aPort: 9810 + role: worker + - id: agent-11 + slimName: agntcy/bench-v2/agent-11 + a2aPort: 9821 + role: worker + - id: agent-12 + slimName: agntcy/bench-v2/agent-12 + a2aPort: 9832 + role: worker + - id: agent-13 + slimName: agntcy/bench-v2/agent-13 + a2aPort: 9843 + role: worker + - id: agent-14 + slimName: agntcy/bench-v2/agent-14 + a2aPort: 9854 + role: worker + - id: agent-15 + slimName: agntcy/bench-v2/agent-15 + a2aPort: 9865 + role: worker + - id: agent-16 + slimName: agntcy/bench-v2/agent-16 + a2aPort: 9876 + role: worker + - id: agent-17 + slimName: agntcy/bench-v2/agent-17 + a2aPort: 9887 + role: worker + - id: agent-18 + slimName: agntcy/bench-v2/agent-18 + a2aPort: 9898 + role: worker + - id: agent-19 + slimName: agntcy/bench-v2/agent-19 + a2aPort: 9909 + role: worker + - id: agent-20 + slimName: agntcy/bench-v2/agent-20 + a2aPort: 9920 + role: worker + - id: agent-21 + slimName: agntcy/bench-v2/agent-21 + a2aPort: 9931 + role: worker + - id: agent-22 + slimName: agntcy/bench-v2/agent-22 + a2aPort: 9942 + role: worker + - id: agent-23 + slimName: agntcy/bench-v2/agent-23 + a2aPort: 9953 + role: worker + - id: agent-24 + slimName: agntcy/bench-v2/agent-24 + a2aPort: 9964 + role: worker + - id: agent-25 + slimName: agntcy/bench-v2/agent-25 + a2aPort: 9975 + role: worker + - id: agent-26 + slimName: agntcy/bench-v2/agent-26 + a2aPort: 9986 + role: worker + - id: agent-27 + slimName: agntcy/bench-v2/agent-27 + a2aPort: 9997 + role: worker + - id: agent-28 + slimName: agntcy/bench-v2/agent-28 + a2aPort: 10008 + role: worker + - id: agent-29 + slimName: agntcy/bench-v2/agent-29 + a2aPort: 10019 + role: worker + - id: agent-30 + slimName: agntcy/bench-v2/agent-30 + a2aPort: 10030 + role: worker + - id: agent-31 + slimName: agntcy/bench-v2/agent-31 + a2aPort: 10041 + role: worker + - id: agent-32 + slimName: agntcy/bench-v2/agent-32 + a2aPort: 10052 + role: worker + - id: agent-33 + slimName: agntcy/bench-v2/agent-33 + a2aPort: 10063 + role: worker + - id: agent-34 + slimName: agntcy/bench-v2/agent-34 + a2aPort: 10074 + role: worker + - id: agent-35 + slimName: agntcy/bench-v2/agent-35 + a2aPort: 10085 + role: worker + - id: agent-36 + slimName: agntcy/bench-v2/agent-36 + a2aPort: 10096 + role: worker + - id: agent-37 + slimName: agntcy/bench-v2/agent-37 + a2aPort: 10107 + role: worker + - id: agent-38 + slimName: agntcy/bench-v2/agent-38 + a2aPort: 10118 + role: worker + - id: agent-39 + slimName: agntcy/bench-v2/agent-39 + a2aPort: 10129 + role: worker + - id: agent-40 + slimName: agntcy/bench-v2/agent-40 + a2aPort: 10140 + role: worker + - id: agent-41 + slimName: agntcy/bench-v2/agent-41 + a2aPort: 10151 + role: worker + - id: agent-42 + slimName: agntcy/bench-v2/agent-42 + a2aPort: 10162 + role: worker + - id: agent-43 + slimName: agntcy/bench-v2/agent-43 + a2aPort: 10173 + role: worker + - id: agent-44 + slimName: agntcy/bench-v2/agent-44 + a2aPort: 10184 + role: worker + - id: agent-45 + slimName: agntcy/bench-v2/agent-45 + a2aPort: 10195 + role: worker + - id: agent-46 + slimName: agntcy/bench-v2/agent-46 + a2aPort: 10206 + role: worker + - id: agent-47 + slimName: agntcy/bench-v2/agent-47 + a2aPort: 10217 + role: worker + - id: agent-48 + slimName: agntcy/bench-v2/agent-48 + a2aPort: 10228 + role: worker + - id: agent-49 + slimName: agntcy/bench-v2/agent-49 + a2aPort: 10239 + role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms.yaml new file mode 100644 index 00000000..2b377dfa --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms.yaml @@ -0,0 +1,215 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-50agents-5ms + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 50 + thinkTimeMs: 5 + findingEmitDelayMs: 1 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker + - id: agent-5 + slimName: agntcy/bench-v2/agent-5 + a2aPort: 9755 + role: worker + - id: agent-6 + slimName: agntcy/bench-v2/agent-6 + a2aPort: 9766 + role: worker + - id: agent-7 + slimName: agntcy/bench-v2/agent-7 + a2aPort: 9777 + role: worker + - id: agent-8 + slimName: agntcy/bench-v2/agent-8 + a2aPort: 9788 + role: worker + - id: agent-9 + slimName: agntcy/bench-v2/agent-9 + a2aPort: 9799 + role: worker + - id: agent-10 + slimName: agntcy/bench-v2/agent-10 + a2aPort: 9810 + role: worker + - id: agent-11 + slimName: agntcy/bench-v2/agent-11 + a2aPort: 9821 + role: worker + - id: agent-12 + slimName: agntcy/bench-v2/agent-12 + a2aPort: 9832 + role: worker + - id: agent-13 + slimName: agntcy/bench-v2/agent-13 + a2aPort: 9843 + role: worker + - id: agent-14 + slimName: agntcy/bench-v2/agent-14 + a2aPort: 9854 + role: worker + - id: agent-15 + slimName: agntcy/bench-v2/agent-15 + a2aPort: 9865 + role: worker + - id: agent-16 + slimName: agntcy/bench-v2/agent-16 + a2aPort: 9876 + role: worker + - id: agent-17 + slimName: agntcy/bench-v2/agent-17 + a2aPort: 9887 + role: worker + - id: agent-18 + slimName: agntcy/bench-v2/agent-18 + a2aPort: 9898 + role: worker + - id: agent-19 + slimName: agntcy/bench-v2/agent-19 + a2aPort: 9909 + role: worker + - id: agent-20 + slimName: agntcy/bench-v2/agent-20 + a2aPort: 9920 + role: worker + - id: agent-21 + slimName: agntcy/bench-v2/agent-21 + a2aPort: 9931 + role: worker + - id: agent-22 + slimName: agntcy/bench-v2/agent-22 + a2aPort: 9942 + role: worker + - id: agent-23 + slimName: agntcy/bench-v2/agent-23 + a2aPort: 9953 + role: worker + - id: agent-24 + slimName: agntcy/bench-v2/agent-24 + a2aPort: 9964 + role: worker + - id: agent-25 + slimName: agntcy/bench-v2/agent-25 + a2aPort: 9975 + role: worker + - id: agent-26 + slimName: agntcy/bench-v2/agent-26 + a2aPort: 9986 + role: worker + - id: agent-27 + slimName: agntcy/bench-v2/agent-27 + a2aPort: 9997 + role: worker + - id: agent-28 + slimName: agntcy/bench-v2/agent-28 + a2aPort: 10008 + role: worker + - id: agent-29 + slimName: agntcy/bench-v2/agent-29 + a2aPort: 10019 + role: worker + - id: agent-30 + slimName: agntcy/bench-v2/agent-30 + a2aPort: 10030 + role: worker + - id: agent-31 + slimName: agntcy/bench-v2/agent-31 + a2aPort: 10041 + role: worker + - id: agent-32 + slimName: agntcy/bench-v2/agent-32 + a2aPort: 10052 + role: worker + - id: agent-33 + slimName: agntcy/bench-v2/agent-33 + a2aPort: 10063 + role: worker + - id: agent-34 + slimName: agntcy/bench-v2/agent-34 + a2aPort: 10074 + role: worker + - id: agent-35 + slimName: agntcy/bench-v2/agent-35 + a2aPort: 10085 + role: worker + - id: agent-36 + slimName: agntcy/bench-v2/agent-36 + a2aPort: 10096 + role: worker + - id: agent-37 + slimName: agntcy/bench-v2/agent-37 + a2aPort: 10107 + role: worker + - id: agent-38 + slimName: agntcy/bench-v2/agent-38 + a2aPort: 10118 + role: worker + - id: agent-39 + slimName: agntcy/bench-v2/agent-39 + a2aPort: 10129 + role: worker + - id: agent-40 + slimName: agntcy/bench-v2/agent-40 + a2aPort: 10140 + role: worker + - id: agent-41 + slimName: agntcy/bench-v2/agent-41 + a2aPort: 10151 + role: worker + - id: agent-42 + slimName: agntcy/bench-v2/agent-42 + a2aPort: 10162 + role: worker + - id: agent-43 + slimName: agntcy/bench-v2/agent-43 + a2aPort: 10173 + role: worker + - id: agent-44 + slimName: agntcy/bench-v2/agent-44 + a2aPort: 10184 + role: worker + - id: agent-45 + slimName: agntcy/bench-v2/agent-45 + a2aPort: 10195 + role: worker + - id: agent-46 + slimName: agntcy/bench-v2/agent-46 + a2aPort: 10206 + role: worker + - id: agent-47 + slimName: agntcy/bench-v2/agent-47 + a2aPort: 10217 + role: worker + - id: agent-48 + slimName: agntcy/bench-v2/agent-48 + a2aPort: 10228 + role: worker + - id: agent-49 + slimName: agntcy/bench-v2/agent-49 + a2aPort: 10239 + role: worker diff --git a/benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml similarity index 100% rename from benchmarks/slim-vs-a2a-v2/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml rename to benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-10ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-10ms.yaml deleted file mode 100644 index 5c483fba..00000000 --- a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-10ms.yaml +++ /dev/null @@ -1,152 +0,0 @@ -apiVersion: bench.agntcy.io/v1 -kind: ExecutionPlan -metadata: - name: canonical-sustainable-resource-10agents - domain: sustainable-resource - description: Generated canonical plan for transport sweeps -spec: - defaults: - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - maxRetries: 0 - roundBudgetMs: 10 - sweep: - family: sustainable-resource - agents: 10 - roundBudgetMs: 10 - executeSleepSec: 0.01 -agents: - - id: agent-0 - slimName: agntcy/bench/agent-0 - a2aPort: 9600 - - id: agent-1 - slimName: agntcy/bench/agent-1 - a2aPort: 9611 - - id: agent-2 - slimName: agntcy/bench/agent-2 - a2aPort: 9622 - - id: agent-3 - slimName: agntcy/bench/agent-3 - a2aPort: 9633 - - id: agent-4 - slimName: agntcy/bench/agent-4 - a2aPort: 9644 - - id: agent-5 - slimName: agntcy/bench/agent-5 - a2aPort: 9655 - - id: agent-6 - slimName: agntcy/bench/agent-6 - a2aPort: 9666 - - id: agent-7 - slimName: agntcy/bench/agent-7 - a2aPort: 9677 - - id: agent-8 - slimName: agntcy/bench/agent-8 - a2aPort: 9688 - - id: agent-9 - slimName: agntcy/bench/agent-9 - a2aPort: 9699 -tasks: - - id: extract-r0-a0 - name: Round 0 agent 0 extraction - agent: agent-0 - dependsOn: [] - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a1 - name: Round 0 agent 1 extraction - agent: agent-1 - dependsOn: - - extract-r0-a0 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a2 - name: Round 0 agent 2 extraction - agent: agent-2 - dependsOn: - - extract-r0-a1 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a3 - name: Round 0 agent 3 extraction - agent: agent-3 - dependsOn: - - extract-r0-a2 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a4 - name: Round 0 agent 4 extraction - agent: agent-4 - dependsOn: - - extract-r0-a3 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a5 - name: Round 0 agent 5 extraction - agent: agent-5 - dependsOn: - - extract-r0-a4 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a6 - name: Round 0 agent 6 extraction - agent: agent-6 - dependsOn: - - extract-r0-a5 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a7 - name: Round 0 agent 7 extraction - agent: agent-7 - dependsOn: - - extract-r0-a6 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a8 - name: Round 0 agent 8 extraction - agent: agent-8 - dependsOn: - - extract-r0-a7 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a9 - name: Round 0 agent 9 extraction - agent: agent-9 - dependsOn: - - extract-r0-a8 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 -contextUpdates: - - afterTask: extract-r0-a9 - payload: round=0 stock=shared shadow_price=0.35 - targetAgents: - - agent-0 - - agent-1 - - agent-2 - - agent-3 - - agent-4 - - agent-5 - - agent-6 - - agent-7 - - agent-8 - - agent-9 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-20ms.yaml deleted file mode 100644 index 5f35eff4..00000000 --- a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-10ag-20ms.yaml +++ /dev/null @@ -1,152 +0,0 @@ -apiVersion: bench.agntcy.io/v1 -kind: ExecutionPlan -metadata: - name: canonical-sustainable-resource-10agents - domain: sustainable-resource - description: Generated canonical plan for transport sweeps -spec: - defaults: - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - maxRetries: 0 - roundBudgetMs: 20 - sweep: - family: sustainable-resource - agents: 10 - roundBudgetMs: 20 - executeSleepSec: 0.01 -agents: - - id: agent-0 - slimName: agntcy/bench/agent-0 - a2aPort: 9600 - - id: agent-1 - slimName: agntcy/bench/agent-1 - a2aPort: 9611 - - id: agent-2 - slimName: agntcy/bench/agent-2 - a2aPort: 9622 - - id: agent-3 - slimName: agntcy/bench/agent-3 - a2aPort: 9633 - - id: agent-4 - slimName: agntcy/bench/agent-4 - a2aPort: 9644 - - id: agent-5 - slimName: agntcy/bench/agent-5 - a2aPort: 9655 - - id: agent-6 - slimName: agntcy/bench/agent-6 - a2aPort: 9666 - - id: agent-7 - slimName: agntcy/bench/agent-7 - a2aPort: 9677 - - id: agent-8 - slimName: agntcy/bench/agent-8 - a2aPort: 9688 - - id: agent-9 - slimName: agntcy/bench/agent-9 - a2aPort: 9699 -tasks: - - id: extract-r0-a0 - name: Round 0 agent 0 extraction - agent: agent-0 - dependsOn: [] - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a1 - name: Round 0 agent 1 extraction - agent: agent-1 - dependsOn: - - extract-r0-a0 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a2 - name: Round 0 agent 2 extraction - agent: agent-2 - dependsOn: - - extract-r0-a1 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a3 - name: Round 0 agent 3 extraction - agent: agent-3 - dependsOn: - - extract-r0-a2 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a4 - name: Round 0 agent 4 extraction - agent: agent-4 - dependsOn: - - extract-r0-a3 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a5 - name: Round 0 agent 5 extraction - agent: agent-5 - dependsOn: - - extract-r0-a4 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a6 - name: Round 0 agent 6 extraction - agent: agent-6 - dependsOn: - - extract-r0-a5 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a7 - name: Round 0 agent 7 extraction - agent: agent-7 - dependsOn: - - extract-r0-a6 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a8 - name: Round 0 agent 8 extraction - agent: agent-8 - dependsOn: - - extract-r0-a7 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a9 - name: Round 0 agent 9 extraction - agent: agent-9 - dependsOn: - - extract-r0-a8 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 -contextUpdates: - - afterTask: extract-r0-a9 - payload: round=0 stock=shared shadow_price=0.35 - targetAgents: - - agent-0 - - agent-1 - - agent-2 - - agent-3 - - agent-4 - - agent-5 - - agent-6 - - agent-7 - - agent-8 - - agent-9 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-10ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-10ms.yaml deleted file mode 100644 index 63faa3da..00000000 --- a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-10ms.yaml +++ /dev/null @@ -1,243 +0,0 @@ -apiVersion: bench.agntcy.io/v1 -kind: ExecutionPlan -metadata: - name: canonical-sustainable-resource-17agents - domain: sustainable-resource - description: Generated canonical plan for transport sweeps -spec: - defaults: - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - maxRetries: 0 - roundBudgetMs: 10 - sweep: - family: sustainable-resource - agents: 17 - roundBudgetMs: 10 - executeSleepSec: 0.01 -agents: - - id: agent-0 - slimName: agntcy/bench/agent-0 - a2aPort: 9600 - - id: agent-1 - slimName: agntcy/bench/agent-1 - a2aPort: 9611 - - id: agent-2 - slimName: agntcy/bench/agent-2 - a2aPort: 9622 - - id: agent-3 - slimName: agntcy/bench/agent-3 - a2aPort: 9633 - - id: agent-4 - slimName: agntcy/bench/agent-4 - a2aPort: 9644 - - id: agent-5 - slimName: agntcy/bench/agent-5 - a2aPort: 9655 - - id: agent-6 - slimName: agntcy/bench/agent-6 - a2aPort: 9666 - - id: agent-7 - slimName: agntcy/bench/agent-7 - a2aPort: 9677 - - id: agent-8 - slimName: agntcy/bench/agent-8 - a2aPort: 9688 - - id: agent-9 - slimName: agntcy/bench/agent-9 - a2aPort: 9699 - - id: agent-10 - slimName: agntcy/bench/agent-10 - a2aPort: 9710 - - id: agent-11 - slimName: agntcy/bench/agent-11 - a2aPort: 9721 - - id: agent-12 - slimName: agntcy/bench/agent-12 - a2aPort: 9732 - - id: agent-13 - slimName: agntcy/bench/agent-13 - a2aPort: 9743 - - id: agent-14 - slimName: agntcy/bench/agent-14 - a2aPort: 9754 - - id: agent-15 - slimName: agntcy/bench/agent-15 - a2aPort: 9765 - - id: agent-16 - slimName: agntcy/bench/agent-16 - a2aPort: 9776 -tasks: - - id: extract-r0-a0 - name: Round 0 agent 0 extraction - agent: agent-0 - dependsOn: [] - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a1 - name: Round 0 agent 1 extraction - agent: agent-1 - dependsOn: - - extract-r0-a0 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a2 - name: Round 0 agent 2 extraction - agent: agent-2 - dependsOn: - - extract-r0-a1 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a3 - name: Round 0 agent 3 extraction - agent: agent-3 - dependsOn: - - extract-r0-a2 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a4 - name: Round 0 agent 4 extraction - agent: agent-4 - dependsOn: - - extract-r0-a3 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a5 - name: Round 0 agent 5 extraction - agent: agent-5 - dependsOn: - - extract-r0-a4 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a6 - name: Round 0 agent 6 extraction - agent: agent-6 - dependsOn: - - extract-r0-a5 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a7 - name: Round 0 agent 7 extraction - agent: agent-7 - dependsOn: - - extract-r0-a6 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a8 - name: Round 0 agent 8 extraction - agent: agent-8 - dependsOn: - - extract-r0-a7 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a9 - name: Round 0 agent 9 extraction - agent: agent-9 - dependsOn: - - extract-r0-a8 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a10 - name: Round 0 agent 10 extraction - agent: agent-10 - dependsOn: - - extract-r0-a9 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a11 - name: Round 0 agent 11 extraction - agent: agent-11 - dependsOn: - - extract-r0-a10 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a12 - name: Round 0 agent 12 extraction - agent: agent-12 - dependsOn: - - extract-r0-a11 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a13 - name: Round 0 agent 13 extraction - agent: agent-13 - dependsOn: - - extract-r0-a12 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a14 - name: Round 0 agent 14 extraction - agent: agent-14 - dependsOn: - - extract-r0-a13 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a15 - name: Round 0 agent 15 extraction - agent: agent-15 - dependsOn: - - extract-r0-a14 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a16 - name: Round 0 agent 16 extraction - agent: agent-16 - dependsOn: - - extract-r0-a15 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 -contextUpdates: - - afterTask: extract-r0-a16 - payload: round=0 stock=shared shadow_price=0.35 - targetAgents: - - agent-0 - - agent-1 - - agent-2 - - agent-3 - - agent-4 - - agent-5 - - agent-6 - - agent-7 - - agent-8 - - agent-9 - - agent-10 - - agent-11 - - agent-12 - - agent-13 - - agent-14 - - agent-15 - - agent-16 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-20ms.yaml deleted file mode 100644 index 3fa68a74..00000000 --- a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-17ag-20ms.yaml +++ /dev/null @@ -1,243 +0,0 @@ -apiVersion: bench.agntcy.io/v1 -kind: ExecutionPlan -metadata: - name: canonical-sustainable-resource-17agents - domain: sustainable-resource - description: Generated canonical plan for transport sweeps -spec: - defaults: - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - maxRetries: 0 - roundBudgetMs: 20 - sweep: - family: sustainable-resource - agents: 17 - roundBudgetMs: 20 - executeSleepSec: 0.01 -agents: - - id: agent-0 - slimName: agntcy/bench/agent-0 - a2aPort: 9600 - - id: agent-1 - slimName: agntcy/bench/agent-1 - a2aPort: 9611 - - id: agent-2 - slimName: agntcy/bench/agent-2 - a2aPort: 9622 - - id: agent-3 - slimName: agntcy/bench/agent-3 - a2aPort: 9633 - - id: agent-4 - slimName: agntcy/bench/agent-4 - a2aPort: 9644 - - id: agent-5 - slimName: agntcy/bench/agent-5 - a2aPort: 9655 - - id: agent-6 - slimName: agntcy/bench/agent-6 - a2aPort: 9666 - - id: agent-7 - slimName: agntcy/bench/agent-7 - a2aPort: 9677 - - id: agent-8 - slimName: agntcy/bench/agent-8 - a2aPort: 9688 - - id: agent-9 - slimName: agntcy/bench/agent-9 - a2aPort: 9699 - - id: agent-10 - slimName: agntcy/bench/agent-10 - a2aPort: 9710 - - id: agent-11 - slimName: agntcy/bench/agent-11 - a2aPort: 9721 - - id: agent-12 - slimName: agntcy/bench/agent-12 - a2aPort: 9732 - - id: agent-13 - slimName: agntcy/bench/agent-13 - a2aPort: 9743 - - id: agent-14 - slimName: agntcy/bench/agent-14 - a2aPort: 9754 - - id: agent-15 - slimName: agntcy/bench/agent-15 - a2aPort: 9765 - - id: agent-16 - slimName: agntcy/bench/agent-16 - a2aPort: 9776 -tasks: - - id: extract-r0-a0 - name: Round 0 agent 0 extraction - agent: agent-0 - dependsOn: [] - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a1 - name: Round 0 agent 1 extraction - agent: agent-1 - dependsOn: - - extract-r0-a0 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a2 - name: Round 0 agent 2 extraction - agent: agent-2 - dependsOn: - - extract-r0-a1 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a3 - name: Round 0 agent 3 extraction - agent: agent-3 - dependsOn: - - extract-r0-a2 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a4 - name: Round 0 agent 4 extraction - agent: agent-4 - dependsOn: - - extract-r0-a3 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a5 - name: Round 0 agent 5 extraction - agent: agent-5 - dependsOn: - - extract-r0-a4 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a6 - name: Round 0 agent 6 extraction - agent: agent-6 - dependsOn: - - extract-r0-a5 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a7 - name: Round 0 agent 7 extraction - agent: agent-7 - dependsOn: - - extract-r0-a6 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a8 - name: Round 0 agent 8 extraction - agent: agent-8 - dependsOn: - - extract-r0-a7 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a9 - name: Round 0 agent 9 extraction - agent: agent-9 - dependsOn: - - extract-r0-a8 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a10 - name: Round 0 agent 10 extraction - agent: agent-10 - dependsOn: - - extract-r0-a9 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a11 - name: Round 0 agent 11 extraction - agent: agent-11 - dependsOn: - - extract-r0-a10 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a12 - name: Round 0 agent 12 extraction - agent: agent-12 - dependsOn: - - extract-r0-a11 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a13 - name: Round 0 agent 13 extraction - agent: agent-13 - dependsOn: - - extract-r0-a12 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a14 - name: Round 0 agent 14 extraction - agent: agent-14 - dependsOn: - - extract-r0-a13 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a15 - name: Round 0 agent 15 extraction - agent: agent-15 - dependsOn: - - extract-r0-a14 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a16 - name: Round 0 agent 16 extraction - agent: agent-16 - dependsOn: - - extract-r0-a15 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 -contextUpdates: - - afterTask: extract-r0-a16 - payload: round=0 stock=shared shadow_price=0.35 - targetAgents: - - agent-0 - - agent-1 - - agent-2 - - agent-3 - - agent-4 - - agent-5 - - agent-6 - - agent-7 - - agent-8 - - agent-9 - - agent-10 - - agent-11 - - agent-12 - - agent-13 - - agent-14 - - agent-15 - - agent-16 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-10ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-10ms.yaml deleted file mode 100644 index 33b54646..00000000 --- a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-10ms.yaml +++ /dev/null @@ -1,48 +0,0 @@ -apiVersion: bench.agntcy.io/v1 -kind: ExecutionPlan -metadata: - name: canonical-sustainable-resource-2agents - domain: sustainable-resource - description: Generated canonical plan for transport sweeps -spec: - defaults: - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - maxRetries: 0 - roundBudgetMs: 10 - sweep: - family: sustainable-resource - agents: 2 - roundBudgetMs: 10 - executeSleepSec: 0.01 -agents: - - id: agent-0 - slimName: agntcy/bench/agent-0 - a2aPort: 9600 - - id: agent-1 - slimName: agntcy/bench/agent-1 - a2aPort: 9611 -tasks: - - id: extract-r0-a0 - name: Round 0 agent 0 extraction - agent: agent-0 - dependsOn: [] - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a1 - name: Round 0 agent 1 extraction - agent: agent-1 - dependsOn: - - extract-r0-a0 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 -contextUpdates: - - afterTask: extract-r0-a1 - payload: round=0 stock=shared shadow_price=0.35 - targetAgents: - - agent-0 - - agent-1 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-20ms.yaml deleted file mode 100644 index 44301ee5..00000000 --- a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-2ag-20ms.yaml +++ /dev/null @@ -1,48 +0,0 @@ -apiVersion: bench.agntcy.io/v1 -kind: ExecutionPlan -metadata: - name: canonical-sustainable-resource-2agents - domain: sustainable-resource - description: Generated canonical plan for transport sweeps -spec: - defaults: - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - maxRetries: 0 - roundBudgetMs: 20 - sweep: - family: sustainable-resource - agents: 2 - roundBudgetMs: 20 - executeSleepSec: 0.01 -agents: - - id: agent-0 - slimName: agntcy/bench/agent-0 - a2aPort: 9600 - - id: agent-1 - slimName: agntcy/bench/agent-1 - a2aPort: 9611 -tasks: - - id: extract-r0-a0 - name: Round 0 agent 0 extraction - agent: agent-0 - dependsOn: [] - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a1 - name: Round 0 agent 1 extraction - agent: agent-1 - dependsOn: - - extract-r0-a0 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 -contextUpdates: - - afterTask: extract-r0-a1 - payload: round=0 stock=shared shadow_price=0.35 - targetAgents: - - agent-0 - - agent-1 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-10ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-10ms.yaml deleted file mode 100644 index 7ecd542a..00000000 --- a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-10ms.yaml +++ /dev/null @@ -1,87 +0,0 @@ -apiVersion: bench.agntcy.io/v1 -kind: ExecutionPlan -metadata: - name: canonical-sustainable-resource-5agents - domain: sustainable-resource - description: Generated canonical plan for transport sweeps -spec: - defaults: - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - maxRetries: 0 - roundBudgetMs: 10 - sweep: - family: sustainable-resource - agents: 5 - roundBudgetMs: 10 - executeSleepSec: 0.01 -agents: - - id: agent-0 - slimName: agntcy/bench/agent-0 - a2aPort: 9600 - - id: agent-1 - slimName: agntcy/bench/agent-1 - a2aPort: 9611 - - id: agent-2 - slimName: agntcy/bench/agent-2 - a2aPort: 9622 - - id: agent-3 - slimName: agntcy/bench/agent-3 - a2aPort: 9633 - - id: agent-4 - slimName: agntcy/bench/agent-4 - a2aPort: 9644 -tasks: - - id: extract-r0-a0 - name: Round 0 agent 0 extraction - agent: agent-0 - dependsOn: [] - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a1 - name: Round 0 agent 1 extraction - agent: agent-1 - dependsOn: - - extract-r0-a0 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a2 - name: Round 0 agent 2 extraction - agent: agent-2 - dependsOn: - - extract-r0-a1 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a3 - name: Round 0 agent 3 extraction - agent: agent-3 - dependsOn: - - extract-r0-a2 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a4 - name: Round 0 agent 4 extraction - agent: agent-4 - dependsOn: - - extract-r0-a3 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 -contextUpdates: - - afterTask: extract-r0-a4 - payload: round=0 stock=shared shadow_price=0.35 - targetAgents: - - agent-0 - - agent-1 - - agent-2 - - agent-3 - - agent-4 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-20ms.yaml deleted file mode 100644 index 175189a0..00000000 --- a/benchmarks/slim-vs-a2a/plans/sweeps/sustainable-resource-5ag-20ms.yaml +++ /dev/null @@ -1,87 +0,0 @@ -apiVersion: bench.agntcy.io/v1 -kind: ExecutionPlan -metadata: - name: canonical-sustainable-resource-5agents - domain: sustainable-resource - description: Generated canonical plan for transport sweeps -spec: - defaults: - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - maxRetries: 0 - roundBudgetMs: 20 - sweep: - family: sustainable-resource - agents: 5 - roundBudgetMs: 20 - executeSleepSec: 0.01 -agents: - - id: agent-0 - slimName: agntcy/bench/agent-0 - a2aPort: 9600 - - id: agent-1 - slimName: agntcy/bench/agent-1 - a2aPort: 9611 - - id: agent-2 - slimName: agntcy/bench/agent-2 - a2aPort: 9622 - - id: agent-3 - slimName: agntcy/bench/agent-3 - a2aPort: 9633 - - id: agent-4 - slimName: agntcy/bench/agent-4 - a2aPort: 9644 -tasks: - - id: extract-r0-a0 - name: Round 0 agent 0 extraction - agent: agent-0 - dependsOn: [] - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a1 - name: Round 0 agent 1 extraction - agent: agent-1 - dependsOn: - - extract-r0-a0 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a2 - name: Round 0 agent 2 extraction - agent: agent-2 - dependsOn: - - extract-r0-a1 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a3 - name: Round 0 agent 3 extraction - agent: agent-3 - dependsOn: - - extract-r0-a2 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 - - id: extract-r0-a4 - name: Round 0 agent 4 extraction - agent: agent-4 - dependsOn: - - extract-r0-a3 - completionTimeSec: 0.01 - maxCompletionTimeSec: 0.04 - injectFailure: false - output: extract=2.0 round=0 -contextUpdates: - - afterTask: extract-r0-a4 - payload: round=0 stock=shared shadow_price=0.35 - targetAgents: - - agent-0 - - agent-1 - - agent-2 - - agent-3 - - agent-4 diff --git a/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go b/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go index b08d1110..ee7db5a7 100644 --- a/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go +++ b/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go @@ -7,62 +7,45 @@ import ( "flag" "fmt" "log" - "strings" + "time" - slim "github.com/agntcy/slim-bindings-go" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/executor" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/server" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/slimrpc" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/agent" ) -const defaultSharedSecret = "demo-shared-secret-min-32-chars!!" - func main() { slimName := flag.String("slim-name", "", "SLIM identity org/group/app") endpoint := flag.String("endpoint", "http://127.0.0.1:46357", "SLIM dataplane endpoint") + scenarioFile := flag.String("scenario-file", "", "path to consensus scenario yaml") + agentIndex := flag.Int("agent-index", 0, "agent index in scenario") flag.Parse() - if *slimName == "" { - log.Fatal("--slim-name is required") + if *slimName == "" || *scenarioFile == "" { + log.Fatal("--slim-name and --scenario-file are required") } - slim.InitializeWithDefaults() - service := slim.GetGlobalService() - - name, err := nameFromString(*slimName) + s, err := scenario.LoadFile(*scenarioFile) if err != nil { - log.Fatalf("parse name: %v", err) + log.Fatalf("load scenario: %v", err) } - - app, err := service.CreateAppWithSecret(name, defaultSharedSecret) - if err != nil { - log.Fatalf("create app: %v", err) + if *agentIndex < 0 || *agentIndex >= len(s.Agents) { + log.Fatalf("agent-index out of range") } - defer app.Destroy() - connID, err := service.Connect(slim.NewInsecureClientConfig(*endpoint)) - if err != nil { - log.Fatalf("connect: %v", err) - } - if err := app.Subscribe(name, &connID); err != nil { - log.Fatalf("subscribe: %v", err) + rt := agent.NewRuntime(s, *agentIndex, *slimName, *endpoint) + if err := rt.Setup(); err != nil { + log.Fatalf("setup: %v", err) } + defer rt.Close() - rpcServer := slim.ServerNewWithConnection(app, name, &connID) - rpcServer.RegisterUnaryUnary(slimrpc.ServiceName, slimrpc.MethodHandle, &server.Handler{ - Engine: executor.New(*slimName), - }) + fmt.Printf("SLIM_AGENT_READY name=%s index=%d scenario=%s\n", *slimName, *agentIndex, s.Metadata.Name) - fmt.Printf("SLIM_AGENT_READY name=%s\n", *slimName) - if err := rpcServer.Serve(); err != nil { - log.Printf("slimrpc server: %v", err) + // Block until the moderator invites us into the group session. + if err := rt.Join(60 * time.Second); err != nil { + log.Fatalf("join group session: %v", err) } -} -func nameFromString(value string) (*slim.Name, error) { - parts := strings.Split(value, "/") - if len(parts) != 3 { - return nil, fmt.Errorf("invalid name format: %s", value) + if err := rt.Run(); err != nil { + log.Printf("receive loop ended: %v", err) } - return slim.NewName(parts[0], parts[1], parts[2]), nil } diff --git a/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go b/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go index 4d3e8ca9..42ca7e41 100644 --- a/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go +++ b/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go @@ -1,53 +1,58 @@ // Copyright AGNTCY Contributors (https://github.com/agntcy) // SPDX-License-Identifier: Apache-2.0 +// Command runner is the SLIM benchmark driver. It acts as a group-session +// moderator (creates the session, invites every agent, broadcasts a single +// start) and then becomes a passive observer: it never relays findings. Agents +// broadcast findings to each other directly over the SLIM dataplane and push +// their snapshots to the runner the instant they converge. Global consensus is +// detected event-driven from those pushed snapshots. package main import ( - "context" "encoding/json" "flag" "fmt" "log" "os" "os/exec" + "strings" "time" + slim "github.com/agntcy/slim-bindings-go" "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/client" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/scheduler" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/protocol" +) + +const ( + runnerSlimName = "agntcy/bench-v2/runner" + groupChannelName = "agntcy/bench-v2/consensus" + defaultSharedSecret = "demo-shared-secret-min-32-chars!!" ) func main() { - planPath := flag.String("plan", "", "path to execution plan yaml") + scenarioPath := flag.String("scenario", "", "path to consensus scenario yaml") endpoint := flag.String("endpoint", "http://127.0.0.1:46357", "SLIM dataplane endpoint") - agentBin := flag.String("agent-bin", "", "path to slim agent binary") + agentBin := flag.String("agent-bin", "", "path to slim-agent binary") outputJSON := flag.String("output-json", "", "write run metrics json") outputTSV := flag.String("output-tsv", "", "append run metrics tsv") waitReady := flag.Duration("wait-ready", 3*time.Second, "wait for agents to start") - roundBudgetMS := flag.Int64("round-budget-ms", 0, "coordination round budget in ms (overrides plan)") - coordMode := flag.String("coord-mode", "multicast", "coordination mode: multicast or unicast") - quiet := flag.Bool("quiet", false, "disable benchmark timing logs") + quiet := flag.Bool("quiet", false, "disable benchmark logs") flag.Parse() if *quiet { benchlog.SetEnabled(false) } - - if *planPath == "" { - log.Fatal("--plan is required") + if *scenarioPath == "" { + log.Fatal("--scenario is required") } - p, err := plan.LoadFile(*planPath) + s, err := scenario.LoadFile(*scenarioPath) if err != nil { - log.Fatalf("load plan: %v", err) - } - - mode := client.CoordModeMulticast - if *coordMode == "unicast" { - mode = client.CoordModeUnicast + log.Fatalf("load scenario: %v", err) } agentPath := *agentBin @@ -58,31 +63,64 @@ func main() { log.Fatal("set --agent-bin or SLIM_AGENT_BIN") } - procs := startAgents(p, agentPath, *endpoint) - time.Sleep(*waitReady) + procs := startAgents(s, agentPath, *endpoint, *scenarioPath) + defer stopAgents(procs) - cli, err := client.New(*endpoint, p, client.Config{ - RoundBudgetMS: p.EffectiveRoundBudgetMS(*roundBudgetMS), - CoordMode: mode, - }) + mod, err := newModerator(*endpoint, s) if err != nil { - stopAgents(procs) - log.Fatalf("client: %v", err) + log.Fatalf("moderator: %v", err) + } + defer mod.Close() + + // Let agents come up and reach ListenForSessionAsync before inviting. + time.Sleep(*waitReady) + if err := mod.InviteAll(); err != nil { + log.Fatalf("invite agents: %v", err) } - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) - defer cancel() + runStart := time.Now() + benchlog.SetRunStart(runStart) + if err := mod.Start(); err != nil { + log.Fatalf("broadcast start: %v", err) + } - if err := cli.SetupGroup(ctx, p); err != nil { - cli.Close() - stopAgents(procs) - log.Fatalf("setup group: %v", err) + result := metrics.RunResult{ + ScenarioName: s.Metadata.Name, + Domain: s.Metadata.Domain, + Implementation: benchlog.ImplSLIM, + Agents: len(s.Agents), + ThinkTimeMs: s.Spec.ThinkTimeMs, + PayloadBytes: s.Spec.PayloadBytes, } - result := scheduler.New(p, cli).Run(ctx) + latest := map[int]consensus.AgentSnapshot{} + n := len(s.Agents) + deadline := time.Now().Add(2 * time.Minute) + for time.Now().Before(deadline) { + timeout := time.Second + msg, err := mod.session.GetMessageAsync(&timeout) + if err != nil { + if isTimeout(err) { + continue + } + break + } + env, derr := protocol.Decode(msg.Payload) + if derr != nil || env.Kind != protocol.KindSnapshot || env.Snapshot == nil { + continue + } + latest[env.Snapshot.AgentIndex] = *env.Snapshot + if len(latest) == n { + if ok, _ := consensus.GlobalConsensus(snapshotSlice(latest)); ok { + result.Success = true + break + } + } + } + + snapshots := snapshotSlice(latest) + result = aggregateResult(result, runStart, snapshots) - time.Sleep(200 * time.Millisecond) - cli.Close() stopAgents(procs) if *outputJSON != "" { @@ -98,12 +136,177 @@ func main() { data, _ := json.MarshalIndent(result, "", " ") fmt.Println(string(data)) + if !result.Success { + os.Exit(1) + } +} + +// moderator owns the group session: it creates it, invites agents, and +// broadcasts the start signal. It does not relay any traffic. +type moderator struct { + app *slim.App + connID uint64 + session *slim.Session + agents []scenario.Agent +} + +func newModerator(endpoint string, s *scenario.ConsensusScenario) (*moderator, error) { + slim.InitializeWithDefaults() + service := slim.GetGlobalService() + + name, err := slim.NameFromString(runnerSlimName) + if err != nil { + return nil, err + } + app, err := service.CreateAppWithSecret(name, defaultSharedSecret) + if err != nil { + return nil, err + } + connID, err := service.Connect(slim.NewInsecureClientConfig(endpoint)) + if err != nil { + app.Destroy() + return nil, err + } + if err := app.Subscribe(name, &connID); err != nil { + app.Destroy() + return nil, err + } + + channelName, err := slim.NameFromString(groupChannelName) + if err != nil { + app.Destroy() + return nil, err + } + interval := 5 * time.Second + maxRetries := uint32(5) + config := slim.SessionConfig{ + SessionType: slim.SessionTypeGroup, + MaxRetries: &maxRetries, + Interval: &interval, + Metadata: map[string]string{}, + } + session, err := app.CreateSessionAndWaitAsync(config, channelName) + if err != nil { + app.Destroy() + return nil, err + } + // Give the session a moment to establish before inviting. + time.Sleep(100 * time.Millisecond) + + return &moderator{app: app, connID: connID, session: session, agents: s.Agents}, nil } -func startAgents(p *plan.ExecutionPlan, agentBin, endpoint string) []*exec.Cmd { +func (m *moderator) InviteAll() error { + for _, a := range m.agents { + name, err := slim.NameFromString(a.SlimName) + if err != nil { + return err + } + if err := m.app.SetRouteAsync(name, m.connID); err != nil { + return fmt.Errorf("set route %s: %w", a.SlimName, err) + } + if err := m.session.InviteAndWaitAsync(name); err != nil { + return fmt.Errorf("invite %s: %w", a.SlimName, err) + } + } + return nil +} + +func (m *moderator) Start() error { + payload, err := protocol.Encode(protocol.Envelope{Kind: protocol.KindStart}) + if err != nil { + return err + } + return m.session.PublishAndWaitAsync(payload, nil, nil) +} + +func (m *moderator) Close() { + if m.session != nil && m.app != nil { + _ = m.app.DeleteSessionAndWaitAsync(m.session) + } + if m.app != nil { + m.app.Destroy() + } +} + +func aggregateResult(result metrics.RunResult, runStart time.Time, snapshots []consensus.AgentSnapshot) metrics.RunResult { + if len(snapshots) == 0 { + result.Error = "no agent snapshots" + result.ConsensusWallMS = time.Since(runStart).Milliseconds() + return result + } + + var ( + totalEmitted int + totalApplied int + lastConvergeMS int64 + propDurations []int64 + maxConsensusRound int + maxRound int + ) + + for _, snap := range snapshots { + totalEmitted += snap.FindingsEmitted + totalApplied += snap.FindingsApplied + if snap.ConsensusRound > maxConsensusRound { + maxConsensusRound = snap.ConsensusRound + } + if snap.Round > maxRound { + maxRound = snap.Round + } + if snap.ConvergedAtNs > 0 { + ms := (snap.ConvergedAtNs - runStart.UnixNano()) / int64(time.Millisecond) + if ms > lastConvergeMS { + lastConvergeMS = ms + } + } + if snap.AvgPropagationMs > 0 { + propDurations = append(propDurations, snap.AvgPropagationMs) + } + } + + if result.Success { + result.ConsensusWallMS = time.Since(runStart).Milliseconds() + if lastConvergeMS > 0 { + result.ConsensusWallMS = lastConvergeMS + } + } else { + result.ConsensusWallMS = time.Since(runStart).Milliseconds() + result.Error = "consensus not reached" + } + + result.ConsensusRound = maxConsensusRound + if result.ConsensusRound == 0 { + result.ConsensusRound = maxRound + } + result.FindingsEmitted = totalEmitted + result.FindingsReceivedTotal = totalApplied + result.LastAgentConvergeMS = lastConvergeMS + result.AvgPropagationMS, result.P95PropagationMS = metrics.AggregatePropagation(propDurations) + // Native group session: findings are broadcast peer-to-peer, no relay. + result.StreamRPCCount = totalEmitted + result.CoordFanoutMS = 0 + return result +} + +func snapshotSlice(m map[int]consensus.AgentSnapshot) []consensus.AgentSnapshot { + out := make([]consensus.AgentSnapshot, 0, len(m)) + for _, snap := range m { + out = append(out, snap) + } + return out +} + +func startAgents(s *scenario.ConsensusScenario, agentBin, endpoint, scenarioFile string) []*exec.Cmd { var procs []*exec.Cmd - for _, agent := range p.Agents { - cmd := exec.Command(agentBin, "--slim-name", agent.SlimName, "--endpoint", endpoint) + for i, agent := range s.Agents { + cmd := exec.Command( + agentBin, + "--slim-name", agent.SlimName, + "--endpoint", endpoint, + "--scenario-file", scenarioFile, + "--agent-index", fmt.Sprintf("%d", i), + ) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr if err := cmd.Start(); err != nil { @@ -122,3 +325,11 @@ func stopAgents(procs []*exec.Cmd) { } } } + +func isTimeout(err error) bool { + if err == nil { + return false + } + msg := strings.ToLower(err.Error()) + return strings.Contains(msg, "timed out") || strings.Contains(msg, "timeout") +} diff --git a/benchmarks/slim-vs-a2a-v2/slim/internal/agent/runtime.go b/benchmarks/slim-vs-a2a/slim/internal/agent/runtime.go similarity index 95% rename from benchmarks/slim-vs-a2a-v2/slim/internal/agent/runtime.go rename to benchmarks/slim-vs-a2a/slim/internal/agent/runtime.go index 9c9fbb2d..4cd82f06 100644 --- a/benchmarks/slim-vs-a2a-v2/slim/internal/agent/runtime.go +++ b/benchmarks/slim-vs-a2a/slim/internal/agent/runtime.go @@ -17,10 +17,10 @@ import ( "time" slim "github.com/agntcy/slim-bindings-go" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/benchlog" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/consensus" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/slim/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/protocol" ) const defaultSharedSecret = "demo-shared-secret-min-32-chars!!" diff --git a/benchmarks/slim-vs-a2a/slim/internal/client/client.go b/benchmarks/slim-vs-a2a/slim/internal/client/client.go deleted file mode 100644 index cc853373..00000000 --- a/benchmarks/slim-vs-a2a/slim/internal/client/client.go +++ /dev/null @@ -1,462 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package client - -import ( - "context" - "fmt" - "slices" - "strings" - "sync" - "time" - - slim "github.com/agntcy/slim-bindings-go" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/protocol" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/slimrpc" -) - -const defaultSharedSecret = "demo-shared-secret-min-32-chars!!" - -type CoordMode string - -const ( - CoordModeMulticast CoordMode = "multicast" - CoordModeUnicast CoordMode = "unicast" -) - -type Config struct { - RoundBudgetMS int64 - CoordMode CoordMode -} - -type Client struct { - mu sync.Mutex - app *slim.App - connID uint64 - agentNames map[string]*slim.Name - slimNameByID map[string]string - p2pChannels map[string]*slim.Channel - groupChannels map[string]*slim.Channel - groupLocks map[string]*sync.Mutex - coordMode CoordMode - coordStats *metrics.CoordStats - stats Stats - implLabel string -} - -type Stats struct { - ExecuteRPCCount int - SequentialRPCCount int - MulticastRPCCount int -} - -func New(endpoint string, p *plan.ExecutionPlan, cfg Config) (*Client, error) { - if cfg.CoordMode == "" { - cfg.CoordMode = CoordModeMulticast - } - impl := benchlog.ImplSLIM - if cfg.CoordMode == CoordModeUnicast { - impl = "slim-unicast" - } - - slim.InitializeWithDefaults() - service := slim.GetGlobalService() - - runnerName := slim.NewName("agntcy", "bench", "runner") - app, err := service.CreateAppWithSecret(runnerName, defaultSharedSecret) - if err != nil { - return nil, err - } - - connID, err := service.Connect(slim.NewInsecureClientConfig(endpoint)) - if err != nil { - app.Destroy() - return nil, err - } - if err := app.Subscribe(runnerName, &connID); err != nil { - app.Destroy() - return nil, err - } - - agentNames := map[string]*slim.Name{} - slimNameByID := map[string]string{} - for _, agent := range p.Agents { - name, err := nameFromString(agent.SlimName) - if err != nil { - app.Destroy() - return nil, err - } - agentNames[agent.ID] = name - slimNameByID[agent.ID] = agent.SlimName - if err := app.SetRoute(name, connID); err != nil { - app.Destroy() - return nil, fmt.Errorf("set route %s: %w", agent.SlimName, err) - } - } - - return &Client{ - app: app, - connID: connID, - agentNames: agentNames, - slimNameByID: slimNameByID, - p2pChannels: make(map[string]*slim.Channel), - groupChannels: make(map[string]*slim.Channel), - groupLocks: make(map[string]*sync.Mutex), - coordMode: cfg.CoordMode, - coordStats: metrics.NewCoordStats(cfg.RoundBudgetMS), - implLabel: impl, - }, nil -} - -func (c *Client) CoordStats() *metrics.CoordStats { return c.coordStats } - -func (c *Client) Implementation() string { return c.implLabel } - -func (c *Client) Close() { - c.mu.Lock() - defer c.mu.Unlock() - - for key, ch := range c.groupChannels { - _ = ch.Close(nil) - delete(c.groupChannels, key) - } - for id, ch := range c.p2pChannels { - _ = ch.Close(nil) - delete(c.p2pChannels, id) - } - if c.app != nil { - c.app.Destroy() - c.app = nil - } -} - -func (c *Client) Stats() Stats { return c.stats } - -func (c *Client) SetupGroup(context.Context, *plan.ExecutionPlan) error { return nil } - -func (c *Client) ExecuteTask(ctx context.Context, agentID string, task plan.Task) (protocol.Response, error) { - req := protocol.Request{ - Op: protocol.OpExecute, - TaskID: task.ID, - CompletionTimeSec: task.CompletionTimeSec, - MaxCompletionTimeSec: task.MaxCompletionTimeSec, - Output: task.Output, - InjectFailure: task.InjectFailure, - } - payload, err := protocol.EncodeRequest(req) - if err != nil { - return protocol.Response{}, err - } - start := time.Now() - reply, err := c.callUnary(ctx, agentID, payload, 30*time.Second) - duration := time.Since(start) - if err != nil { - benchlog.RPC(c.implLabel, protocol.OpExecute, "p2p", duration, false, - fmt.Sprintf("agent=%s", agentID), - fmt.Sprintf("task=%s", task.ID), - fmt.Sprintf("err=%v", err), - ) - return protocol.Response{}, err - } - c.stats.ExecuteRPCCount++ - resp, decErr := protocol.DecodeResponse(reply) - benchlog.RPC(c.implLabel, protocol.OpExecute, "p2p", duration, decErr == nil && resp.OK, - fmt.Sprintf("agent=%s", agentID), - fmt.Sprintf("task=%s", task.ID), - fmt.Sprintf("resp_ok=%t", resp.OK), - ) - if decErr != nil { - return protocol.Response{}, decErr - } - return resp, nil -} - -func (c *Client) CancelTasks(ctx context.Context, agentIDs []string, taskIDs []string) error { - return c.coordOp(ctx, agentIDs, protocol.Request{ - Op: protocol.OpCancel, - TaskIDs: taskIDs, - TargetSlimNames: c.slimNamesFor(agentIDs), - }) -} - -func (c *Client) PushContext(ctx context.Context, agentIDs []string, payload string) (time.Duration, error) { - start := time.Now() - err := c.coordOp(ctx, agentIDs, protocol.Request{ - Op: protocol.OpContext, - Payload: payload, - TargetSlimNames: c.slimNamesFor(agentIDs), - }) - return time.Since(start), err -} - -func (c *Client) SyncPhase(ctx context.Context, agentIDs []string, phase string) (time.Duration, error) { - start := time.Now() - err := c.coordOp(ctx, agentIDs, protocol.Request{ - Op: protocol.OpSync, - Phase: phase, - TargetSlimNames: c.slimNamesFor(agentIDs), - }) - return time.Since(start), err -} - -func (c *Client) NotifyFailure(ctx context.Context, agentIDs []string, failedTaskID string) error { - return c.coordOp(ctx, agentIDs, protocol.Request{ - Op: protocol.OpContext, - Payload: "failure=" + failedTaskID, - TargetSlimNames: c.slimNamesFor(agentIDs), - }) -} - -func (c *Client) coordOp(ctx context.Context, agentIDs []string, req protocol.Request) error { - if len(agentIDs) == 0 { - return nil - } - payload, err := protocol.EncodeRequest(req) - if err != nil { - return err - } - if c.coordMode == CoordModeUnicast { - return c.unicastFanOut(ctx, agentIDs, req, payload) - } - return c.multicast(ctx, agentIDs, req, payload) -} - -func (c *Client) unicastFanOut(ctx context.Context, agentIDs []string, req protocol.Request, payload []byte) error { - start := time.Now() - var firstErr error - responded := 0 - for i, agentID := range agentIDs { - callStart := time.Now() - _, err := c.callUnary(ctx, agentID, payload, 10*time.Second) - if err != nil && firstErr == nil { - firstErr = err - } else if err == nil { - responded++ - } - c.stats.SequentialRPCCount++ - if err != nil { - benchlog.RPC(c.implLabel, req.Op, "sequential", time.Since(start), false, - fmt.Sprintf("targets=%d", len(agentIDs)), - fmt.Sprintf("idx=%d", i+1), - fmt.Sprintf("err=%v", err), - ) - c.recordCoord(time.Since(start), len(agentIDs), responded, len(payload), len(agentIDs)) - return err - } - _ = callStart - } - duration := time.Since(start) - c.recordCoord(duration, len(agentIDs), responded, len(payload), len(agentIDs)) - benchlog.RPC(c.implLabel, req.Op, "sequential", duration, true, - fmt.Sprintf("targets=%d", len(agentIDs)), - fmt.Sprintf("responded=%d", responded), - ) - return firstErr -} - -func (c *Client) recordCoord(duration time.Duration, targets, responded, payloadBytes, messagesSent int) { - if c.coordStats != nil { - c.coordStats.RecordCoordOp(duration, targets, responded, payloadBytes, messagesSent) - } -} - -func (c *Client) slimNamesFor(agentIDs []string) []string { - names := make([]string, 0, len(agentIDs)) - for _, id := range agentIDs { - if name, ok := c.slimNameByID[id]; ok { - names = append(names, name) - } - } - return names -} - -func subsetGroupKey(agentIDs []string) string { - ids := append([]string(nil), agentIDs...) - slices.Sort(ids) - return strings.Join(ids, "|") -} - -func (c *Client) groupLock(key string) *sync.Mutex { - c.mu.Lock() - defer c.mu.Unlock() - if l, ok := c.groupLocks[key]; ok { - return l - } - l := &sync.Mutex{} - c.groupLocks[key] = l - return l -} - -func (c *Client) subsetGroup(agentIDs []string) (*slim.Channel, error) { - key := subsetGroupKey(agentIDs) - - c.mu.Lock() - if ch, ok := c.groupChannels[key]; ok { - c.mu.Unlock() - return ch, nil - } - c.mu.Unlock() - - members := make([]*slim.Name, 0, len(agentIDs)) - for _, id := range agentIDs { - name, ok := c.agentNames[id] - if !ok { - return nil, fmt.Errorf("unknown agent %q", id) - } - members = append(members, name) - } - - channel, err := slim.ChannelNewGroupWithConnection(c.app, members, &c.connID) - if err != nil { - return nil, err - } - - c.mu.Lock() - if existing, ok := c.groupChannels[key]; ok { - c.mu.Unlock() - _ = channel.Close(nil) - return existing, nil - } - c.groupChannels[key] = channel - c.mu.Unlock() - return channel, nil -} - -func (c *Client) multicast(ctx context.Context, agentIDs []string, req protocol.Request, payload []byte) error { - key := subsetGroupKey(agentIDs) - lock := c.groupLock(key) - lock.Lock() - defer lock.Unlock() - - group, err := c.subsetGroup(agentIDs) - if err != nil { - return err - } - - timeout := 10 * time.Second - if deadline, ok := ctx.Deadline(); ok { - timeout = time.Until(deadline) - if timeout <= 0 { - return context.DeadlineExceeded - } - } - - start := time.Now() - targets := c.slimNamesFor(agentIDs) - reader, err := group.CallMulticastUnary(slimrpc.ServiceName, slimrpc.MethodHandle, payload, &timeout, nil) - if err != nil { - benchlog.RPC(c.implLabel, req.Op, "multicast", time.Since(start), false, - fmt.Sprintf("targets=%d", len(targets)), - fmt.Sprintf("target_names=%s", strings.Join(targets, ",")), - fmt.Sprintf("err=%v", err), - ) - c.recordCoord(time.Since(start), len(targets), 0, len(payload), 1) - return err - } - defer reader.Destroy() - - c.stats.MulticastRPCCount++ - responded, drainErr := drainMulticast(reader, targets) - duration := time.Since(start) - c.recordCoord(duration, len(targets), responded, len(payload), 1) - kv := []string{ - fmt.Sprintf("targets=%d", len(targets)), - fmt.Sprintf("responded=%d", responded), - fmt.Sprintf("target_names=%s", strings.Join(targets, ",")), - } - if drainErr != nil { - kv = append(kv, fmt.Sprintf("err=%v", drainErr)) - } - benchlog.RPC(c.implLabel, req.Op, "multicast", duration, drainErr == nil, kv...) - return drainErr -} - -func drainMulticast(reader *slim.MulticastResponseReader, targetedSlimNames []string) (int, error) { - targetSet := make(map[string]struct{}, len(targetedSlimNames)) - for _, name := range targetedSlimNames { - targetSet[protocol.NormalizeSlimName(name)] = struct{}{} - } - - var firstErr error - responded := map[string]struct{}{} - - for { - switch msg := reader.Next().(type) { - case slim.MulticastStreamMessageEnd: - for name := range targetSet { - if _, ok := responded[name]; !ok && firstErr == nil { - firstErr = fmt.Errorf("missing multicast response from %q", name) - } - } - return len(responded), firstErr - case slim.MulticastStreamMessageError: - if firstErr == nil && msg.Error != nil { - firstErr = msg.Error.AsError() - } - case slim.MulticastStreamMessageData: - resp, err := protocol.DecodeResponse(msg.Item.Message) - if err != nil && firstErr == nil { - firstErr = err - continue - } - slimName := resp.SlimName - if slimName == "" && msg.Item.Context.Source != nil { - slimName = msg.Item.Context.Source.String() - } - slimName = protocol.NormalizeSlimName(slimName) - if slimName == "" { - continue - } - responded[slimName] = struct{}{} - if _, wanted := targetSet[slimName]; wanted && !resp.OK && firstErr == nil { - firstErr = fmt.Errorf("%s: %s", slimName, resp.Error) - } - } - } -} - -func (c *Client) callUnary(ctx context.Context, agentID string, payload []byte, timeout time.Duration) ([]byte, error) { - channel, err := c.p2pChannel(agentID) - if err != nil { - return nil, err - } - - if deadline, ok := ctx.Deadline(); ok { - remaining := time.Until(deadline) - if remaining > 0 && remaining < timeout { - timeout = remaining - } - } - - return channel.CallUnary(slimrpc.ServiceName, slimrpc.MethodHandle, payload, &timeout, nil) -} - -func (c *Client) p2pChannel(agentID string) (*slim.Channel, error) { - c.mu.Lock() - defer c.mu.Unlock() - - if ch, ok := c.p2pChannels[agentID]; ok { - return ch, nil - } - - dest, ok := c.agentNames[agentID] - if !ok { - return nil, fmt.Errorf("unknown agent %q", agentID) - } - - ch := slim.ChannelNewWithConnection(c.app, dest, &c.connID) - c.p2pChannels[agentID] = ch - return ch, nil -} - -func nameFromString(value string) (*slim.Name, error) { - parts := strings.Split(value, "/") - if len(parts) != 3 { - return nil, fmt.Errorf("invalid name format: %s", value) - } - return slim.NewName(parts[0], parts[1], parts[2]), nil -} diff --git a/benchmarks/slim-vs-a2a/slim/internal/executor/executor.go b/benchmarks/slim-vs-a2a/slim/internal/executor/executor.go deleted file mode 100644 index ccd2079d..00000000 --- a/benchmarks/slim-vs-a2a/slim/internal/executor/executor.go +++ /dev/null @@ -1,100 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package executor - -import ( - "context" - "strings" - "sync" - "time" - - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/protocol" -) - -type Engine struct { - SlimName string - mu sync.Mutex - cancels map[string]context.CancelFunc -} - -func New(slimName string) *Engine { - return &Engine{SlimName: slimName, cancels: map[string]context.CancelFunc{}} -} - -func (e *Engine) Handle(ctx context.Context, req protocol.Request) protocol.Response { - if len(req.TargetSlimNames) > 0 && !req.Targets(e.SlimName) { - return protocol.Response{OK: true, SlimName: e.SlimName} - } - - switch req.Op { - case protocol.OpExecute: - resp := e.execute(ctx, req) - resp.SlimName = e.SlimName - return resp - case protocol.OpCancel: - resp := e.cancel(req) - resp.SlimName = e.SlimName - return resp - case protocol.OpContext, protocol.OpSync: - return protocol.Response{OK: true, SlimName: e.SlimName} - default: - return protocol.Response{OK: false, SlimName: e.SlimName, Error: "unknown op"} - } -} - -func (e *Engine) execute(parent context.Context, req protocol.Request) protocol.Response { - if req.InjectFailure { - return protocol.Response{OK: false, TaskID: req.TaskID, Error: "injected failure"} - } - - ctx, cancel := context.WithCancel(parent) - e.mu.Lock() - e.cancels[req.TaskID] = cancel - e.mu.Unlock() - defer func() { - e.mu.Lock() - delete(e.cancels, req.TaskID) - e.mu.Unlock() - cancel() - }() - - deadline := time.Duration(req.MaxCompletionTimeSec * float64(time.Second)) - if deadline <= 0 { - deadline = time.Duration(req.CompletionTimeSec*float64(time.Second)) + time.Second - } - timer := time.NewTimer(deadline) - defer timer.Stop() - - sleep := time.Duration(req.CompletionTimeSec * float64(time.Second)) - start := time.Now() - select { - case <-time.After(sleep): - case <-ctx.Done(): - return protocol.Response{OK: false, TaskID: req.TaskID, Error: "cancelled"} - case <-timer.C: - return protocol.Response{OK: false, TaskID: req.TaskID, Error: "timeout"} - } - - return protocol.Response{ - OK: true, - TaskID: req.TaskID, - Output: req.Output, - ElapsedSec: time.Since(start).Seconds(), - } -} - -func (e *Engine) cancel(req protocol.Request) protocol.Response { - e.mu.Lock() - defer e.mu.Unlock() - for _, id := range req.TaskIDs { - if cancel, ok := e.cancels[id]; ok { - cancel() - } - } - return protocol.Response{OK: true} -} - -func (e *Engine) ObsoleteFromPayload(payload string) bool { - return strings.Contains(strings.ToLower(payload), "cancel") -} diff --git a/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go b/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go index 7baf6e8a..ba4e3634 100644 --- a/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go +++ b/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go @@ -1,98 +1,41 @@ // Copyright AGNTCY Contributors (https://github.com/agntcy) // SPDX-License-Identifier: Apache-2.0 +// Package protocol defines the message envelope exchanged over the native SLIM +// group session. There is no relay and no request/response RPC: the runner +// broadcasts a single start, agents broadcast findings to all peers, and agents +// push their snapshots to the runner the instant they converge. package protocol -import "encoding/json" +import ( + "encoding/json" -const ( - OpExecute = "execute" - OpCancel = "cancel" - OpContext = "context" - OpSync = "sync" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" ) -type Request struct { - Op string `json:"op"` - TaskID string `json:"taskId,omitempty"` - CompletionTimeSec float64 `json:"completionTimeSec,omitempty"` - MaxCompletionTimeSec float64 `json:"maxCompletionTimeSec,omitempty"` - Output string `json:"output,omitempty"` - InjectFailure bool `json:"injectFailure,omitempty"` - TaskIDs []string `json:"taskIds,omitempty"` - Payload string `json:"payload,omitempty"` - Phase string `json:"phase,omitempty"` - FailedTaskID string `json:"failedTaskId,omitempty"` - // TargetSlimNames lists org/group/app SLIM names that should handle a multicast. - // Other group members acknowledge and ignore the request. - TargetSlimNames []string `json:"targetSlimNames,omitempty"` -} - -func (r Request) Targets(slimName string) bool { - if len(r.TargetSlimNames) == 0 { - return true - } - target := NormalizeSlimName(slimName) - for _, name := range r.TargetSlimNames { - if NormalizeSlimName(name) == target { - return true - } - } - return false -} - -// NormalizeSlimName strips SlimRPC instance suffixes so agntcy/bench/foo and -// agntcy/bench/foo/ compare equal. -func NormalizeSlimName(name string) string { - parts := splitSlimName(name) - if len(parts) >= 3 { - return parts[0] + "/" + parts[1] + "/" + parts[2] - } - return name -} - -func splitSlimName(name string) []string { - var parts []string - start := 0 - for i := 0; i < len(name); i++ { - if name[i] == '/' { - if i > start { - parts = append(parts, name[start:i]) - } - start = i + 1 - } - } - if start < len(name) { - parts = append(parts, name[start:]) - } - return parts -} - -type Response struct { - OK bool `json:"ok"` - TaskID string `json:"taskId,omitempty"` - SlimName string `json:"slimName,omitempty"` - Output string `json:"output,omitempty"` - Error string `json:"error,omitempty"` - ElapsedSec float64 `json:"elapsedSec,omitempty"` -} - -func EncodeRequest(req Request) ([]byte, error) { - return json.Marshal(req) -} +const ( + // KindStart is broadcast once by the runner to define t0 and unblock agents. + KindStart = "start" + // KindFinding is broadcast by an agent to all group peers. + KindFinding = "finding" + // KindSnapshot is pushed by an agent directly to the runner on convergence. + KindSnapshot = "snapshot" +) -func DecodeRequest(data []byte) (Request, error) { - var req Request - err := json.Unmarshal(data, &req) - return req, err +// Envelope is the single JSON message type carried on the group session. +type Envelope struct { + Kind string `json:"kind"` + AgentIndex int `json:"agentIndex,omitempty"` + Finding *consensus.Finding `json:"finding,omitempty"` + Snapshot *consensus.AgentSnapshot `json:"snapshot,omitempty"` } -func EncodeResponse(resp Response) ([]byte, error) { - return json.Marshal(resp) +func Encode(e Envelope) ([]byte, error) { + return json.Marshal(e) } -func DecodeResponse(data []byte) (Response, error) { - var resp Response - err := json.Unmarshal(data, &resp) - return resp, err +func Decode(data []byte) (Envelope, error) { + var e Envelope + err := json.Unmarshal(data, &e) + return e, err } diff --git a/benchmarks/slim-vs-a2a/slim/internal/scheduler/scheduler.go b/benchmarks/slim-vs-a2a/slim/internal/scheduler/scheduler.go deleted file mode 100644 index 2a1166c0..00000000 --- a/benchmarks/slim-vs-a2a/slim/internal/scheduler/scheduler.go +++ /dev/null @@ -1,341 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package scheduler - -import ( - "context" - "fmt" - "strings" - "sync" - "time" - - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/client" -) - -type taskState int - -const ( - statePending taskState = iota - stateRunning - stateCompleted - stateFailed - stateTimedOut - stateCancelled -) - -type Scheduler struct { - plan *plan.ExecutionPlan - client *client.Client -} - -func New(p *plan.ExecutionPlan, cli *client.Client) *Scheduler { - return &Scheduler{plan: p, client: cli} -} - -func (s *Scheduler) Run(ctx context.Context) metrics.RunResult { - start := time.Now() - benchlog.SetRunStart(start) - result := metrics.RunResult{ - PlanName: s.plan.Metadata.Name, - Domain: s.plan.Metadata.Domain, - Implementation: s.client.Implementation(), - Agents: len(s.plan.Agents), - Tasks: len(s.plan.Tasks), - } - - states := map[string]taskState{} - contextFired := map[string]bool{} - var mu sync.Mutex - var wg sync.WaitGroup - var contextPushMS int64 - var syncBarrierMS int64 - var cancelPropagationMS int64 - var obsoleteCompleted int - abort := false - - for _, t := range s.plan.Tasks { - states[t.ID] = statePending - } - - cancelTasks := func(taskIDs []string) { - if len(taskIDs) == 0 { - return - } - agentSet := map[string]struct{}{} - for _, id := range taskIDs { - task, ok := s.plan.TaskByID(id) - if !ok { - continue - } - agentSet[task.Agent] = struct{}{} - } - agentIDs := make([]string, 0, len(agentSet)) - for id := range agentSet { - agentIDs = append(agentIDs, id) - } - startCancel := time.Now() - err := s.client.CancelTasks(ctx, agentIDs, taskIDs) - duration := time.Since(startCancel) - cancelPropagationMS += duration.Milliseconds() - kv := []string{ - fmt.Sprintf("agents=%d", len(agentIDs)), - fmt.Sprintf("tasks=%d", len(taskIDs)), - } - if err != nil { - kv = append(kv, fmt.Sprintf("err=%v", err)) - } - benchlog.Coord(benchlog.ImplSLIM, "cancel", err == nil, duration, kv...) - } - - cancelDownstream := func(root string) { - downstream := plan.DownstreamTasks(s.plan.Tasks, root) - var ids []string - mu.Lock() - for id := range downstream { - switch states[id] { - case statePending, stateRunning: - states[id] = stateCancelled - result.TasksCancelled++ - ids = append(ids, id) - } - } - mu.Unlock() - cancelTasks(ids) - } - - handleFailure := func(taskID string, timedOut bool) { - mu.Lock() - if timedOut { - states[taskID] = stateTimedOut - result.TasksTimedOut++ - } else { - states[taskID] = stateFailed - result.TasksFailed++ - } - abort = true - mu.Unlock() - agents := affectedAgents([]string{taskID}, s.plan) - if err := s.client.NotifyFailure(ctx, agents, taskID); err != nil { - benchlog.Coord(benchlog.ImplSLIM, "notify_failure", false, 0, - fmt.Sprintf("task=%s", taskID), - fmt.Sprintf("agents=%d", len(agents)), - fmt.Sprintf("err=%v", err), - ) - } - cancelTasks([]string{taskID}) - cancelDownstream(taskID) - } - - applyContext := func(taskID string) { - mu.Lock() - if contextFired[taskID] { - mu.Unlock() - return - } - contextFired[taskID] = true - mu.Unlock() - - for _, cu := range s.plan.ContextUpdatesAfter(taskID) { - d, err := s.client.PushContext(ctx, cu.TargetAgents, cu.Payload) - contextPushMS += d.Milliseconds() - kv := []string{ - fmt.Sprintf("after_task=%s", taskID), - fmt.Sprintf("agents=%d", len(cu.TargetAgents)), - fmt.Sprintf("payload=%q", benchlog.TruncatePayload(cu.Payload)), - } - if err != nil { - kv = append(kv, fmt.Sprintf("err=%v", err)) - } - benchlog.Coord(benchlog.ImplSLIM, "context_push", err == nil, d, kv...) - - if strings.Contains(cu.Payload, "sync=") || strings.Contains(cu.Payload, "phase=") { - syncD, syncErr := s.client.SyncPhase(ctx, cu.TargetAgents, cu.Payload) - if syncErr == nil { - syncBarrierMS += syncD.Milliseconds() - } - syncKV := []string{ - fmt.Sprintf("after_task=%s", taskID), - fmt.Sprintf("agents=%d", len(cu.TargetAgents)), - } - if syncErr != nil { - syncKV = append(syncKV, fmt.Sprintf("err=%v", syncErr)) - } - benchlog.Coord(benchlog.ImplSLIM, "sync_phase", syncErr == nil, syncD, syncKV...) - } - if strings.Contains(strings.ToLower(cu.Payload), "cancel") { - var toCancel []string - mu.Lock() - for _, t := range s.plan.Tasks { - if states[t.ID] != stateRunning { - continue - } - if shouldCancelByContext(t, cu.Payload) { - states[t.ID] = stateCancelled - result.TasksCancelled++ - toCancel = append(toCancel, t.ID) - obsoleteCompleted++ - } - } - mu.Unlock() - cancelTasks(toCancel) - } - } - } - - launch := func(task plan.Task) { - wg.Add(1) - go func(task plan.Task) { - defer wg.Done() - taskStart := time.Now() - benchlog.Task(benchlog.ImplSLIM, "started", task.ID, task.Agent, 0) - resp, err := s.client.ExecuteTask(ctx, task.Agent, task) - taskDuration := time.Since(taskStart) - mu.Lock() - if abort && states[task.ID] == stateRunning { - states[task.ID] = stateCancelled - result.TasksCancelled++ - mu.Unlock() - benchlog.Task(benchlog.ImplSLIM, "cancelled", task.ID, task.Agent, taskDuration) - return - } - mu.Unlock() - - if err != nil { - benchlog.Task(benchlog.ImplSLIM, "failed", task.ID, task.Agent, taskDuration, - fmt.Sprintf("err=%v", err)) - handleFailure(task.ID, false) - return - } - if !resp.OK { - benchlog.Task(benchlog.ImplSLIM, "failed", task.ID, task.Agent, taskDuration, - fmt.Sprintf("resp_error=%s", resp.Error)) - handleFailure(task.ID, resp.Error == "timeout") - return - } - - mu.Lock() - states[task.ID] = stateCompleted - result.TasksCompleted++ - mu.Unlock() - benchlog.Task(benchlog.ImplSLIM, "completed", task.ID, task.Agent, taskDuration) - applyContext(task.ID) - }(task) - } - - for { - mu.Lock() - if abort { - mu.Unlock() - break - } - ready := readyTasks(s.plan.Tasks, states) - for _, task := range ready { - states[task.ID] = stateRunning - launch(task) - } - pending := countStates(states, statePending) - running := countStates(states, stateRunning) - mu.Unlock() - - if len(ready) == 0 && pending == 0 && running == 0 { - break - } - time.Sleep(5 * time.Millisecond) - } - - wg.Wait() - stats := s.client.Stats() - coordTimeMS := contextPushMS + syncBarrierMS + cancelPropagationMS - result.TotalWallClockMS = time.Since(start).Milliseconds() - result.ContextPushMS = contextPushMS - result.SyncBarrierMS = syncBarrierMS - result.CancelPropagationMS = cancelPropagationMS - result.ObsoleteTasksCompleted = obsoleteCompleted - result.ExecuteRPCCount = stats.ExecuteRPCCount - result.SequentialRPCCount = stats.SequentialRPCCount - result.MulticastRPCCount = stats.MulticastRPCCount - result.MakespanMS = result.TotalWallClockMS - result.Success = result.TasksFailed == 0 && result.TasksTimedOut == 0 - if s.client.CoordStats() != nil { - s.client.CoordStats().ApplyToRunResult(&result, coordTimeMS) - } - if !result.Success && result.Error == "" { - result.Error = fmt.Sprintf("failed=%d timed_out=%d cancelled=%d", result.TasksFailed, result.TasksTimedOut, result.TasksCancelled) - } - benchlog.Coord(benchlog.ImplSLIM, "run_finished", result.Success, time.Since(start), - fmt.Sprintf("tasks_completed=%d", result.TasksCompleted), - fmt.Sprintf("tasks_failed=%d", result.TasksFailed), - fmt.Sprintf("tasks_cancelled=%d", result.TasksCancelled), - fmt.Sprintf("context_push_ms=%d", contextPushMS), - fmt.Sprintf("execute_rpcs=%d", stats.ExecuteRPCCount), - fmt.Sprintf("multicast_rpcs=%d", stats.MulticastRPCCount), - ) - return result -} - -func readyTasks(tasks []plan.Task, states map[string]taskState) []plan.Task { - var ready []plan.Task - for _, task := range tasks { - if states[task.ID] != statePending { - continue - } - ok := true - for _, dep := range task.DependsOn { - if states[dep] != stateCompleted { - ok = false - break - } - } - if ok { - ready = append(ready, task) - } - } - return ready -} - -func countStates(states map[string]taskState, target taskState) int { - n := 0 - for _, st := range states { - if st == target { - n++ - } - } - return n -} - -func affectedAgents(taskIDs []string, p *plan.ExecutionPlan) []string { - set := map[string]struct{}{} - for _, id := range taskIDs { - task, ok := p.TaskByID(id) - if !ok { - continue - } - set[task.Agent] = struct{}{} - } - out := make([]string, 0, len(set)) - for id := range set { - out = append(out, id) - } - return out -} - -func shouldCancelByContext(task plan.Task, payload string) bool { - payload = strings.ToLower(payload) - if strings.Contains(payload, "pharmacy") && strings.Contains(task.ID, "pharmacy") { - return true - } - if strings.Contains(payload, "detour") && strings.Contains(task.ID, "detour") { - return true - } - if strings.Contains(payload, "node-drain") && strings.Contains(task.ID, "node-drain") { - return true - } - if strings.Contains(payload, "mesh-rollback") && strings.Contains(task.ID, "rollback") { - return true - } - return false -} diff --git a/benchmarks/slim-vs-a2a/slim/internal/server/handler.go b/benchmarks/slim-vs-a2a/slim/internal/server/handler.go deleted file mode 100644 index 64ba8ca0..00000000 --- a/benchmarks/slim-vs-a2a/slim/internal/server/handler.go +++ /dev/null @@ -1,25 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package server - -import ( - "context" - - slim "github.com/agntcy/slim-bindings-go" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/executor" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/protocol" -) - -type Handler struct { - Engine *executor.Engine -} - -func (h *Handler) Handle(request []byte, _ *slim.Context) ([]byte, error) { - req, err := protocol.DecodeRequest(request) - if err != nil { - return nil, err - } - resp := h.Engine.Handle(context.Background(), req) - return protocol.EncodeResponse(resp) -} diff --git a/benchmarks/slim-vs-a2a/slim/internal/slimrpc/slimrpc.go b/benchmarks/slim-vs-a2a/slim/internal/slimrpc/slimrpc.go deleted file mode 100644 index 82814939..00000000 --- a/benchmarks/slim-vs-a2a/slim/internal/slimrpc/slimrpc.go +++ /dev/null @@ -1,9 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package slimrpc - -const ( - ServiceName = "bench.Agent" - MethodHandle = "Handle" -) diff --git a/benchmarks/slim-vs-a2a/tests/slim_vs_a2a_test.go b/benchmarks/slim-vs-a2a/tests/slim_vs_a2a_test.go deleted file mode 100644 index ef48b1e2..00000000 --- a/benchmarks/slim-vs-a2a/tests/slim_vs_a2a_test.go +++ /dev/null @@ -1,299 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package tests - -import ( - "encoding/json" - "fmt" - "net" - "os" - "os/exec" - "path/filepath" - "strings" - "time" - - ginkgo "github.com/onsi/ginkgo/v2" - "github.com/onsi/gomega" - "github.com/onsi/gomega/gexec" - - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" -) - -var ( - binDir string - planDir string - reportsDir string - resultsTSV string - slimctlPath string - slimConfigPath string - slimSession *gexec.Session -) - -var _ = ginkgo.BeforeSuite(func() { - if os.Getenv("RUN_SLIM_VS_A2A") != "1" { - return - } - - var err error - binDir, err = os.MkdirTemp("", "slim-vs-a2a-bin-*") - gomega.Expect(err).NotTo(gomega.HaveOccurred()) - - build := exec.Command("go", "build", "-o", filepath.Join(binDir, "a2a-agent"), "./a2a/cmd/agent") - build.Dir = repoRoot() - gomega.Expect(build.Run()).To(gomega.Succeed()) - - build = exec.Command("go", "build", "-o", filepath.Join(binDir, "a2a-runner"), "./a2a/cmd/runner") - build.Dir = repoRoot() - gomega.Expect(build.Run()).To(gomega.Succeed()) - - setup := exec.Command("go", "run", "github.com/agntcy/slim-bindings-go/cmd/slim-bindings-setup@v1.4.0") - setup.Dir = repoRoot() - gomega.Expect(setup.Run()).To(gomega.Succeed()) - - build = exec.Command("go", "build", "-o", filepath.Join(binDir, "slim-agent"), "./slim/cmd/agent") - build.Dir = repoRoot() - build.Env = append(os.Environ(), slimCgoLDFlags()...) - gomega.Expect(build.Run()).To(gomega.Succeed()) - - build = exec.Command("go", "build", "-o", filepath.Join(binDir, "slim-runner"), "./slim/cmd/runner") - build.Dir = repoRoot() - build.Env = append(os.Environ(), slimCgoLDFlags()...) - gomega.Expect(build.Run()).To(gomega.Succeed()) - - planDir = filepath.Join(repoRoot(), "plans", "domains") - reportsDir = filepath.Join(repoRoot(), "reports") - resultsTSV = filepath.Join(reportsDir, "results.tsv") - gomega.Expect(os.MkdirAll(reportsDir, 0o755)).To(gomega.Succeed()) - slimctlPath = filepath.Join(repoRoot(), "..", "agntcy-slim", "bin", "slimctl") - if _, err := os.Stat(slimctlPath); err != nil { - download := exec.Command("task", "deps:slimctl-download") - download.Dir = repoRoot() - gomega.Expect(download.Run()).To(gomega.Succeed()) - } -}) - -var _ = ginkgo.AfterSuite(func() { - stopSlimStack() - if binDir != "" { - _ = os.RemoveAll(binDir) - } - gexec.CleanupBuildArtifacts() -}) - -var _ = ginkgo.Describe("SLIM vs A2A comparison", ginkgo.Label("slim-vs-a2a"), func() { - ginkgo.BeforeEach(func() { - if os.Getenv("RUN_SLIM_VS_A2A") != "1" { - ginkgo.Skip("set RUN_SLIM_VS_A2A=1 to run comparison suite") - } - }) - - ginkgo.It("runs all domain plans on A2A and SLIM", func() { - plans, err := plan.LoadDir(planDir) - gomega.Expect(err).NotTo(gomega.HaveOccurred()) - gomega.Expect(plans).NotTo(gomega.BeEmpty()) - - _ = os.Remove(resultsTSV) - - for _, p := range plans { - planPath := filepath.Join(planDir, p.Metadata.Name+".yaml") - a2aResult := runA2A(planPath, p.Metadata.Name, 0) - gomega.Expect(a2aResult.PlanName).To(gomega.Equal(p.Metadata.Name)) - gomega.Expect(a2aResult.Implementation).To(gomega.Equal("a2a-grpc")) - gomega.Expect(a2aResult.TotalWallClockMS).To(gomega.BeNumerically(">", 0)) - - startSlimStack() - slimResult := runSLIM(planPath, p.Metadata.Name, "multicast", 0) - stopSlimStack() - - gomega.Expect(slimResult.PlanName).To(gomega.Equal(p.Metadata.Name)) - gomega.Expect(slimResult.Implementation).To(gomega.Equal("slim-multicast")) - gomega.Expect(slimResult.TotalWallClockMS).To(gomega.BeNumerically(">", 0)) - - logComparison(p.Metadata.Name, a2aResult, slimResult) - } - }) - - ginkgo.It("logs sweep crossover hints at 10 agents / 20ms budget", func() { - sweepDir := filepath.Join(repoRoot(), "plans", "sweeps") - gomega.Expect(os.MkdirAll(sweepDir, 0o755)).To(gomega.Succeed()) - planPath := filepath.Join(sweepDir, "ginkgo-sweep-sustainable-10ag-20ms.yaml") - - gen := exec.Command("go", "run", "./tools/gen_plan", - "-family", "sustainable-resource", - "-agents", "10", - "-round-budget-ms", "20", - "-output", planPath, - ) - gen.Dir = repoRoot() - gomega.Expect(gen.Run()).To(gomega.Succeed()) - - a2aResult := runA2A(planPath, "sweep-a2a", 20) - startSlimStack() - slimResult := runSLIM(planPath, "sweep-slim", "multicast", 20) - stopSlimStack() - - ginkgo.GinkgoWriter.Printf( - "sweep agents=10 budget=20ms a2a_p95=%d slim_p95=%d a2a_missing=%d slim_missing=%d\n", - a2aResult.ContextPushP95MS, - slimResult.ContextPushP95MS, - a2aResult.CoordMissingResponses, - slimResult.CoordMissingResponses, - ) - - if slimResult.ContextPushP95MS > a2aResult.ContextPushP95MS { - ginkgo.GinkgoWriter.Printf("note: SLIM p95 higher than A2A at this scale (expected below crossover)\n") - } - }) -}) - -func logComparison(name string, a2aResult, slimResult metrics.RunResult) { - ginkgo.GinkgoWriter.Printf( - "plan=%s a2a_wall=%dms slim_wall=%dms a2a_ctx=%dms slim_ctx=%dms a2a_p95=%d slim_p95=%d a2a_missing=%d slim_missing=%d\n", - name, - a2aResult.TotalWallClockMS, - slimResult.TotalWallClockMS, - a2aResult.ContextPushMS, - slimResult.ContextPushMS, - a2aResult.ContextPushP95MS, - slimResult.ContextPushP95MS, - a2aResult.CoordMissingResponses, - slimResult.CoordMissingResponses, - ) -} - -func runA2A(planPath, jsonName string, roundBudgetMS int64) metrics.RunResult { - jsonPath := filepath.Join(reportsDir, jsonName+".json") - args := []string{ - "--plan", planPath, - "--agent-bin", filepath.Join(binDir, "a2a-agent"), - "--output-json", jsonPath, - "--wait-ready", "4s", - "--quiet", - } - if roundBudgetMS > 0 { - args = append(args, "--round-budget-ms", fmt.Sprintf("%d", roundBudgetMS)) - } - cmd := exec.Command(filepath.Join(binDir, "a2a-runner"), args...) - session, err := gexec.Start(cmd, ginkgo.GinkgoWriter, ginkgo.GinkgoWriter) - gomega.Expect(err).NotTo(gomega.HaveOccurred()) - gomega.Eventually(session, 3*time.Minute).Should(gexec.Exit(0)) - return readResult(jsonPath) -} - -func runSLIM(planPath, jsonName, coordMode string, roundBudgetMS int64) metrics.RunResult { - jsonPath := filepath.Join(reportsDir, jsonName+".json") - args := []string{ - "--plan", planPath, - "--endpoint", "http://127.0.0.1:46357", - "--agent-bin", filepath.Join(binDir, "slim-agent"), - "--coord-mode", coordMode, - "--output-json", jsonPath, - "--wait-ready", "4s", - "--quiet", - } - if roundBudgetMS > 0 { - args = append(args, "--round-budget-ms", fmt.Sprintf("%d", roundBudgetMS)) - } - cmd := exec.Command(filepath.Join(binDir, "slim-runner"), args...) - session, err := gexec.Start(cmd, ginkgo.GinkgoWriter, ginkgo.GinkgoWriter) - gomega.Expect(err).NotTo(gomega.HaveOccurred()) - gomega.Eventually(session, 3*time.Minute).Should(gexec.Exit(0)) - return readResult(jsonPath) -} - -func readResult(path string) metrics.RunResult { - data, err := os.ReadFile(path) - gomega.Expect(err).NotTo(gomega.HaveOccurred()) - var result metrics.RunResult - gomega.Expect(json.Unmarshal(data, &result)).To(gomega.Succeed()) - return result -} - -func repoRoot() string { - wd, err := os.Getwd() - gomega.Expect(err).NotTo(gomega.HaveOccurred()) - for { - if _, err := os.Stat(filepath.Join(wd, "go.mod")); err == nil { - if _, err := os.Stat(filepath.Join(wd, "a2a")); err == nil { - return wd - } - } - parent := filepath.Dir(wd) - if parent == wd { - ginkgo.Fail("could not locate slim-vs-a2a module root") - } - wd = parent - } -} - -func slimCgoLDFlags() []string { - out, err := exec.Command("go", "env", "GOPATH").Output() - if err != nil { - return nil - } - cacheDir := filepath.Join(strings.TrimSpace(string(out)), ".cgo-cache", "slim-bindings", "v1.4.0") - return []string{"CGO_LDFLAGS=-L" + cacheDir} -} - -func startSlimStack() { - stopSlimStack() - dataplanePort := 46357 - controllerPort := 46358 - configFile, err := os.CreateTemp("", "slim-vs-a2a-*.yaml") - gomega.Expect(err).NotTo(gomega.HaveOccurred()) - slimConfigPath = configFile.Name() - config := fmt.Sprintf(`services: - slim/0: - node_id: "slim-vs-a2a" - group_name: "bench" - dataplane: - servers: - - endpoint: "127.0.0.1:%d" - metadata: - local_endpoint: "127.0.0.1" - external_endpoint: "127.0.0.1:%d" - trust_domain: "example.org" - tls: - insecure: true - controller: - servers: - - endpoint: "127.0.0.1:%d" - tls: - insecure: true -`, dataplanePort, dataplanePort, controllerPort) - _, err = configFile.WriteString(config) - gomega.Expect(err).NotTo(gomega.HaveOccurred()) - gomega.Expect(configFile.Close()).To(gomega.Succeed()) - - cmd := exec.Command(slimctlPath, "slim", "start", "-c", slimConfigPath) - session, err := gexec.Start(cmd, ginkgo.GinkgoWriter, ginkgo.GinkgoWriter) - gomega.Expect(err).NotTo(gomega.HaveOccurred()) - slimSession = session - gomega.Expect(waitForPort(fmt.Sprintf("127.0.0.1:%d", dataplanePort), 20*time.Second)).To(gomega.Succeed()) -} - -func stopSlimStack() { - if slimSession != nil { - slimSession.Terminate().Wait(5 * time.Second) - slimSession = nil - } - if slimConfigPath != "" { - _ = os.Remove(slimConfigPath) - slimConfigPath = "" - } -} - -func waitForPort(addr string, timeout time.Duration) error { - deadline := time.Now().Add(timeout) - for time.Now().Before(deadline) { - conn, err := net.DialTimeout("tcp", addr, 250*time.Millisecond) - if err == nil { - _ = conn.Close() - return nil - } - time.Sleep(100 * time.Millisecond) - } - return fmt.Errorf("timed out waiting for %s", addr) -} diff --git a/benchmarks/slim-vs-a2a/tests/suite_test.go b/benchmarks/slim-vs-a2a/tests/suite_test.go deleted file mode 100644 index 3671cfcf..00000000 --- a/benchmarks/slim-vs-a2a/tests/suite_test.go +++ /dev/null @@ -1,16 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package tests - -import ( - "testing" - - ginkgo "github.com/onsi/ginkgo/v2" - "github.com/onsi/gomega" -) - -func TestSlimVsA2A(t *testing.T) { - gomega.RegisterFailHandler(ginkgo.Fail) - ginkgo.RunSpecs(t, "SLIM vs A2A Suite") -} diff --git a/benchmarks/slim-vs-a2a/tools/gen_plan/main.go b/benchmarks/slim-vs-a2a/tools/gen_plan/main.go deleted file mode 100644 index 561cae39..00000000 --- a/benchmarks/slim-vs-a2a/tools/gen_plan/main.go +++ /dev/null @@ -1,52 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package main - -import ( - "flag" - "fmt" - "os" - - "gopkg.in/yaml.v3" - - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" -) - -func main() { - family := flag.String("family", "sustainable-resource", "canonical family: preference-aggregation, supply-chain-cascade, sustainable-resource") - agents := flag.Int("agents", 5, "number of agents") - fanOut := flag.Int("fan-out", 0, "context fan-out (0 = all agents)") - roundBudgetMS := flag.Int64("round-budget-ms", 20, "coordination round budget ms") - executeSec := flag.Float64("execute-sec", 0.01, "task execute sleep seconds") - rounds := flag.Int("rounds", 1, "rounds for sustainable-resource family") - output := flag.String("output", "", "write plan yaml to path (default stdout)") - flag.Parse() - - p, err := plan.GenerateCanonical(plan.GenerateOptions{ - Family: plan.CanonicalFamily(*family), - Agents: *agents, - FanOut: *fanOut, - RoundBudgetMS: *roundBudgetMS, - ExecuteSleepSec: *executeSec, - Rounds: *rounds, - }) - if err != nil { - fmt.Fprintf(os.Stderr, "generate: %v\n", err) - os.Exit(1) - } - - if *output == "" { - data, err := yaml.Marshal(p) - if err != nil { - fmt.Fprintf(os.Stderr, "marshal: %v\n", err) - os.Exit(1) - } - os.Stdout.Write(data) - return - } - if err := plan.WriteFile(*output, p); err != nil { - fmt.Fprintf(os.Stderr, "write: %v\n", err) - os.Exit(1) - } -} diff --git a/benchmarks/slim-vs-a2a-v2/tools/gen_scenario/main.go b/benchmarks/slim-vs-a2a/tools/gen_scenario/main.go similarity index 75% rename from benchmarks/slim-vs-a2a-v2/tools/gen_scenario/main.go rename to benchmarks/slim-vs-a2a/tools/gen_scenario/main.go index fbae3a44..db300374 100644 --- a/benchmarks/slim-vs-a2a-v2/tools/gen_scenario/main.go +++ b/benchmarks/slim-vs-a2a/tools/gen_scenario/main.go @@ -10,7 +10,7 @@ import ( "os" "path/filepath" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" ) func main() { @@ -18,6 +18,7 @@ func main() { agents := flag.Int("agents", 10, "number of agents") thinkMS := flag.Int64("think-ms", 10, "think time per agent in ms") seed := flag.Int64("seed", 42, "random seed") + payloadBytes := flag.Int("payload-bytes", 0, "fixed-size padding added to every finding (bytes)") output := flag.String("output", "", "output yaml path") flag.Parse() @@ -26,10 +27,11 @@ func main() { } s, err := scenario.Generate(scenario.GenerateOptions{ - Family: *family, - Agents: *agents, - ThinkTimeMs: *thinkMS, - Seed: *seed, + Family: *family, + Agents: *agents, + ThinkTimeMs: *thinkMS, + Seed: *seed, + PayloadBytes: *payloadBytes, }) if err != nil { log.Fatalf("generate: %v", err) diff --git a/benchmarks/slim-vs-a2a/tools/report/main.go b/benchmarks/slim-vs-a2a/tools/report/main.go index 244037e9..7b635ee6 100644 --- a/benchmarks/slim-vs-a2a/tools/report/main.go +++ b/benchmarks/slim-vs-a2a/tools/report/main.go @@ -15,12 +15,12 @@ import ( "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" ) -type planComparison struct { - PlanName string - A2A metrics.RunResult - SLIM metrics.RunResult - HasA2A bool - HasSLIM bool +type scenarioComparison struct { + ScenarioName string + A2A metrics.RunResult + SLIM metrics.RunResult + HasA2A bool + HasSLIM bool } func main() { @@ -34,7 +34,7 @@ func main() { fmt.Fprintf(os.Stderr, "read tsv: %v\n", err) os.Exit(1) } - comparisons := groupByPlan(results) + comparisons := groupByScenario(results) var sweepResults []metrics.RunResult if *sweepTSV != "" { @@ -69,91 +69,71 @@ func readTSV(path string) ([]metrics.RunResult, error) { var out []metrics.RunResult for _, row := range records[1:] { - if len(row) < 22 { + if len(row) < 17 { continue } r := metrics.RunResult{ - PlanName: row[0], + ScenarioName: row[0], Domain: row[1], Implementation: row[2], - Error: row[len(row)-1], } r.Agents = atoi(row[3]) - r.Tasks = atoi(row[4]) - r.TotalWallClockMS = atoi64(row[5]) - r.TasksCompleted = atoi(row[6]) - r.TasksFailed = atoi(row[7]) - r.TasksTimedOut = atoi(row[8]) - r.TasksCancelled = atoi(row[9]) - r.ObsoleteTasksCompleted = atoi(row[10]) - r.RetriesAttempted = atoi(row[11]) - r.RetriesSucceeded = atoi(row[12]) - r.ContextPushMS = atoi64(row[13]) - r.SyncBarrierMS = atoi64(row[14]) - r.CancelPropagationMS = atoi64(row[15]) - r.ExecuteRPCCount = atoi(row[16]) - r.MulticastRPCCount = atoi(row[17]) - r.SequentialRPCCount = atoi(row[18]) - r.MakespanMS = atoi64(row[19]) - if len(row) >= 29 { - r.ContextPushP95MS = atoi64(row[20]) - r.ContextPushOps = atoi(row[21]) - r.CoordMissingResponses = atoi(row[22]) - r.CoordDeadlineMisses = atoi(row[23]) - r.CoordBytesSent = atoi64(row[24]) - r.CoordTimeSharePct, _ = strconv.ParseFloat(row[25], 64) - r.RoundBudgetMS = atoi64(row[26]) - r.Success = row[27] == "true" - r.Error = row[28] - } else { - r.Success = row[20] == "true" - if len(row) > 21 { - r.Error = row[21] - } + r.ThinkTimeMs = atoi64(row[4]) + r.PayloadBytes = atoi(row[5]) + r.ConsensusWallMS = atoi64(row[6]) + r.ConsensusRound = atoi(row[7]) + r.FindingsEmitted = atoi(row[8]) + r.FindingsReceivedTotal = atoi(row[9]) + r.AvgPropagationMS = atoi64(row[10]) + r.P95PropagationMS = atoi64(row[11]) + r.LastAgentConvergeMS = atoi64(row[12]) + r.CoordFanoutMS = atoi64(row[13]) + r.StreamRPCCount = atoi(row[14]) + r.UnicastRPCCount = atoi(row[15]) + r.Success = row[16] == "true" + if len(row) > 17 { + r.Error = row[17] } out = append(out, r) } return out, nil } -func groupByPlan(results []metrics.RunResult) []planComparison { - byPlan := map[string]*planComparison{} +func groupByScenario(results []metrics.RunResult) []scenarioComparison { + byScenario := map[string]*scenarioComparison{} for _, r := range results { - pc, ok := byPlan[r.PlanName] + sc, ok := byScenario[r.ScenarioName] if !ok { - pc = &planComparison{PlanName: r.PlanName} - byPlan[r.PlanName] = pc + sc = &scenarioComparison{ScenarioName: r.ScenarioName} + byScenario[r.ScenarioName] = sc } switch r.Implementation { - case "a2a-grpc": - pc.A2A = r - pc.HasA2A = true - case "slim-multicast", "slim-unicast": - if !pc.HasSLIM || r.Implementation == "slim-multicast" { - pc.SLIM = r - pc.HasSLIM = true - } + case "a2a-relay-stream": + sc.A2A = r + sc.HasA2A = true + case "slim-group-session": + sc.SLIM = r + sc.HasSLIM = true } } - out := make([]planComparison, 0, len(byPlan)) - for _, pc := range byPlan { - out = append(out, *pc) + out := make([]scenarioComparison, 0, len(byScenario)) + for _, sc := range byScenario { + out = append(out, *sc) } return out } type reportData struct { - Comparisons []planComparison + Comparisons []scenarioComparison Sweep []metrics.RunResult } -func writeHTML(path string, comparisons []planComparison, sweep []metrics.RunResult) error { +func writeHTML(path string, comparisons []scenarioComparison, sweep []metrics.RunResult) error { if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { return err } tmpl := template.Must(template.New("report").Funcs(template.FuncMap{ - "deltaPct": deltaPct, - "deltaPctInt": deltaPctInt, + "deltaPct": deltaPct, }).Parse(htmlTemplate)) file, err := os.Create(path) if err != nil { @@ -163,18 +143,18 @@ func writeHTML(path string, comparisons []planComparison, sweep []metrics.RunRes return tmpl.Execute(file, reportData{Comparisons: comparisons, Sweep: sweep}) } +// deltaPct expresses the A2A cost as an improvement relative to SLIM: +// (A2A − SLIM) / SLIM × 100. Unlike a reduction relative to A2A (capped at +// 100%), this is unbounded, so a 42× speedup reads as ~4100% rather than ~98%. +// Positive values mean SLIM was faster/cheaper. func deltaPct(a2a, slim int64) string { - if a2a == 0 { + if slim == 0 { return "n/a" } - pct := (float64(a2a-slim) / float64(a2a)) * 100 + pct := (float64(a2a-slim) / float64(slim)) * 100 return fmt.Sprintf("%.1f%%", pct) } -func deltaPctInt(a2a, slim int) string { - return deltaPct(int64(a2a), int64(slim)) -} - func atoi(v string) int { n, _ := strconv.Atoi(v) return n @@ -189,53 +169,128 @@ const htmlTemplate = ` - SLIM vs A2A Comparison + SLIM vs A2A — Consensus Streaming -

SLIM vs A2A DAG Comparison

-

Delta columns show (A2A − SLIM) / A2A. Positive values mean SLIM was faster or used fewer RPCs.

+

SLIM vs A2A — Consensus Streaming

+
+

This benchmark runs the same distributed hypothesis-convergence workload on two + transports and measures how fast N agents reach identical consensus. SLIM + (slim-group-session) broadcasts each finding once over a native group session and the + dataplane fans it out — there is no relay. A2A (a2a-relay-stream) + has no peer multicast, so the runner is an explicit relay hub: every finding makes two streaming + hops (agent → runner → N−1 agents).

+

The Improvement vs SLIM column is (A2A − SLIM) / SLIM × 100 — how much + more time/work A2A costs relative to SLIM. It is unbounded: e.g. +4000% means A2A took ~41× + as long as SLIM. Positive values favor SLIM. See metric definitions and the + architecture diagram below.

+
{{range .Comparisons}} -

{{.PlanName}}

+

{{.ScenarioName}}

- - - - - - - - - - - - - - - + + + + + + +
MetricA2ASLIMDelta
Wall clock (ms){{if .HasA2A}}{{.A2A.TotalWallClockMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.TotalWallClockMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.TotalWallClockMS .SLIM.TotalWallClockMS}}{{else}}—{{end}}
Context push (ms){{if .HasA2A}}{{.A2A.ContextPushMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.ContextPushMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.ContextPushMS .SLIM.ContextPushMS}}{{else}}—{{end}}
Context push p95 (ms){{if .HasA2A}}{{.A2A.ContextPushP95MS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.ContextPushP95MS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.ContextPushP95MS .SLIM.ContextPushP95MS}}{{else}}—{{end}}
Missing responses{{if .HasA2A}}{{.A2A.CoordMissingResponses}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.CoordMissingResponses}}{{else}}—{{end}}—
Deadline misses{{if .HasA2A}}{{.A2A.CoordDeadlineMisses}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.CoordDeadlineMisses}}{{else}}—{{end}}—
Coord bytes sent{{if .HasA2A}}{{.A2A.CoordBytesSent}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.CoordBytesSent}}{{else}}—{{end}}—
Cancel propagation (ms){{if .HasA2A}}{{.A2A.CancelPropagationMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.CancelPropagationMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.CancelPropagationMS .SLIM.CancelPropagationMS}}{{else}}—{{end}}
Sequential RPC count{{if .HasA2A}}{{.A2A.SequentialRPCCount}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.SequentialRPCCount}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPctInt .A2A.SequentialRPCCount .SLIM.SequentialRPCCount}}{{else}}—{{end}}
Multicast RPC count{{if .HasA2A}}{{.A2A.MulticastRPCCount}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.MulticastRPCCount}}{{else}}—{{end}}—
MetricA2ASLIMImprovement vs SLIM
Consensus wall (ms){{if .HasA2A}}{{.A2A.ConsensusWallMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.ConsensusWallMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.ConsensusWallMS .SLIM.ConsensusWallMS}}{{else}}—{{end}}
Last agent converge (ms){{if .HasA2A}}{{.A2A.LastAgentConvergeMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.LastAgentConvergeMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.LastAgentConvergeMS .SLIM.LastAgentConvergeMS}}{{else}}—{{end}}
Avg propagation (ms){{if .HasA2A}}{{.A2A.AvgPropagationMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.AvgPropagationMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.AvgPropagationMS .SLIM.AvgPropagationMS}}{{else}}—{{end}}
P95 propagation (ms){{if .HasA2A}}{{.A2A.P95PropagationMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.P95PropagationMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.P95PropagationMS .SLIM.P95PropagationMS}}{{else}}—{{end}}
Stream RPC count{{if .HasA2A}}{{.A2A.StreamRPCCount}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.StreamRPCCount}}{{else}}—{{end}}—
Unicast RPC count{{if .HasA2A}}{{.A2A.UnicastRPCCount}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.UnicastRPCCount}}{{else}}—{{end}}—
Success{{if .HasA2A}}{{.A2A.Success}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.Success}}{{else}}—{{end}}—
{{end}} {{if .Sweep}}

Sweep results

- - - + {{range .Sweep}} - - + + {{end}}
PlanImplAgentsBudget msCtx p95MissingDeadline missesBytes
ScenarioImplAgentsThink msPayload BConsensus wallAvg propagationP95 propagationStream RPCs
{{.PlanName}}{{.Implementation}}{{.Agents}}{{.RoundBudgetMS}}{{.ContextPushP95MS}}{{.CoordMissingResponses}}{{.CoordDeadlineMisses}}{{.CoordBytesSent}}{{.ScenarioName}}{{.Implementation}}{{.Agents}}{{.ThinkTimeMs}}{{.PayloadBytes}}{{.ConsensusWallMS}}{{.AvgPropagationMS}}{{.P95PropagationMS}}{{.StreamRPCCount}}
{{end}} + +

Metric definitions

+
+
Consensus wall (ms)
+
Headline metric: wall-clock time from the runner's single start broadcast until every agent + has reached identical local consensus. Excludes process/library startup and teardown.
+
Last agent converge (ms)
+
Time until the slowest agent converged, taken from each agent's own convergence timestamp relative to + start. Highlights stragglers behind the headline number.
+
Avg / P95 propagation (ms)
+
Per-finding delivery latency, from the moment a finding is emitted to when a peer applies it + (average and 95th percentile). SLIM is one multicast hop; A2A adds the extra relay hop, so it grows + with agent count, message rate, and payload size.
+
Stream RPC count
+
Number of finding-carrying messages on the data path. For SLIM this equals findings emitted (one + native multicast each). For A2A it is the relay deliveries, ≈ findings × (N−1), because the + hub re-sends every finding to all other agents.
+
Unicast RPC count
+
Control-plane unary RPCs only (start / snapshot polling), not the data path.
+
Coord fanout (ms)
+
Cumulative time the A2A relay hub spent fanning findings out to peers. 0 for SLIM because + there is no relay — the dataplane does the fan-out.
+
Payload B
+
Fixed-size padding (spec.payloadBytes) added to every finding to stress transport + bandwidth. Semantically inert; it does not change the consensus math or round count.
+
Improvement vs SLIM
+
(A2A − SLIM) / SLIM × 100. Unbounded measure of how much more time/work A2A costs than + SLIM; e.g. +4100% ≈ 42× slower. Positive favors SLIM. (Note: a2a-go's streaming task store + also appends each relayed finding to task history and deep-copies per update, which amplifies A2A cost + super-linearly at large payloads — a property of the A2A streaming/task model.)
+
+ +

Architecture

+

SLIM agents form a peer mesh over a native group session and the runner only observes; A2A routes every + finding through the runner acting as a relay hub.

+
+ 
+flowchart TB
+  subgraph slim["SLIM native group session — no relay"]
+    R1["runner (passive observer: start signal + consumes pushed metrics)"]
+    SA0["agent-0"]
+    SA1["agent-1"]
+    SA2["agent-2"]
+    SA0 ---|"Publish finding → all (dataplane fan-out)"| SA1
+    SA1 --- SA2
+    SA0 --- SA2
+    SA0 -.->|"push snapshot on convergence"| R1
+    SA1 -.-> R1
+    SA2 -.-> R1
+  end
+  subgraph a2a["A2A — runner is the relay hub"]
+    R2["runner = RELAY (hosts A2A server)"]
+    AA0["agent-0"]
+    AA1["agent-1"]
+    AA2["agent-2"]
+    AA0 -->|"stream findings → runner"| R2
+    AA1 --> R2
+    AA2 --> R2
+    R2 -->|"stream relayed findings → agent"| AA0
+    R2 --> AA1
+    R2 --> AA2
+  end
+
+
+ + ` diff --git a/benchmarks/slim-vs-a2a/tools/validate_plans/main.go b/benchmarks/slim-vs-a2a/tools/validate_plans/main.go deleted file mode 100644 index 50737315..00000000 --- a/benchmarks/slim-vs-a2a/tools/validate_plans/main.go +++ /dev/null @@ -1,28 +0,0 @@ -// Copyright AGNTCY Contributors (https://github.com/agntcy) -// SPDX-License-Identifier: Apache-2.0 - -package main - -import ( - "flag" - "fmt" - "os" - "path/filepath" - - "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/plan" -) - -func main() { - dir := flag.String("dir", filepath.Join("plans", "domains"), "directory containing plan yaml files") - flag.Parse() - - plans, err := plan.LoadDir(*dir) - if err != nil { - fmt.Fprintf(os.Stderr, "load plans: %v\n", err) - os.Exit(1) - } - for _, p := range plans { - fmt.Printf("%s: %d agents, %d tasks, %d contextUpdates\n", - p.Metadata.Name, len(p.Agents), len(p.Tasks), len(p.ContextUpdates)) - } -} diff --git a/benchmarks/slim-vs-a2a-v2/tools/validate_scenarios/main.go b/benchmarks/slim-vs-a2a/tools/validate_scenarios/main.go similarity index 91% rename from benchmarks/slim-vs-a2a-v2/tools/validate_scenarios/main.go rename to benchmarks/slim-vs-a2a/tools/validate_scenarios/main.go index c32898f8..fa5e76c5 100644 --- a/benchmarks/slim-vs-a2a-v2/tools/validate_scenarios/main.go +++ b/benchmarks/slim-vs-a2a/tools/validate_scenarios/main.go @@ -11,7 +11,7 @@ import ( "path/filepath" "strings" - "github.com/agntcy/csit/benchmarks/slim-vs-a2a-v2/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" ) func main() { From 83bccaf74575262f22c313ce80dda671172878aa Mon Sep 17 00:00:00 2001 From: Magyari Sandor Szilard Date: Mon, 29 Jun 2026 11:21:05 +0200 Subject: [PATCH 5/6] feat: run tests from gh workflow Signed-off-by: Magyari Sandor Szilard --- .../publish-test-reports-pages/action.yaml | 7 +- .github/scripts/build-pages-section.sh | 12 +++- .github/scripts/build-site-landing-page.sh | 3 + .github/scripts/init-sources-json.sh | 11 ++++ .github/workflows/test-slim-vs-a2a.yaml | 64 ++++++++++++++++--- 5 files changed, 85 insertions(+), 12 deletions(-) diff --git a/.github/actions/publish-test-reports-pages/action.yaml b/.github/actions/publish-test-reports-pages/action.yaml index b40e2a18..86308cb1 100644 --- a/.github/actions/publish-test-reports-pages/action.yaml +++ b/.github/actions/publish-test-reports-pages/action.yaml @@ -8,7 +8,7 @@ description: Download suite artifacts, update gh-pages report dirs and docs/inde inputs: suite: - description: "Suite identifier (a2a, a2a-slimrpc, slim-integration, slim-benchmarks, slim-multicluster-private, directory-conformance)" + description: "Suite identifier (a2a, a2a-slimrpc, slim-integration, slim-benchmarks, slim-vs-a2a, slim-multicluster-private, directory-conformance)" required: true source-run-id: description: "Workflow run ID containing uploaded artifacts" @@ -70,6 +70,10 @@ runs: ./.github/scripts/download-workflow-artifact.sh "$RUN_ID" .pages-staging/basic \ --name slim-benchmark-basic-report ;; + slim-vs-a2a) + ./.github/scripts/download-workflow-artifact.sh "$RUN_ID" .pages-staging \ + --name slim-vs-a2a-report + ;; slim-multicluster-private) ./.github/scripts/download-workflow-artifact.sh "$RUN_ID" .pages-staging \ --name slim-multicluster-private-test-result @@ -111,6 +115,7 @@ runs: a2a-slimrpc) WORKFLOW_ID=test-a2a-slimrpc ;; slim-integration) WORKFLOW_ID=test-slim-integration ;; slim-benchmarks) WORKFLOW_ID=test-slim-benchmarks ;; + slim-vs-a2a) WORKFLOW_ID=test-slim-vs-a2a ;; slim-multicluster-private) WORKFLOW_ID=test-slim-multicluster-private ;; directory-conformance) WORKFLOW_ID=test-directory-conformance ;; *) echo "unknown suite: $SUITE" >&2; exit 1 ;; diff --git a/.github/scripts/build-pages-section.sh b/.github/scripts/build-pages-section.sh index 74c37f3e..fa2428dd 100755 --- a/.github/scripts/build-pages-section.sh +++ b/.github/scripts/build-pages-section.sh @@ -5,7 +5,7 @@ # Build one gh-pages docs section from a staging directory populated by download-artifact. # # Required env: -# SUITE a2a | a2a-slimrpc | slim-integration | slim-benchmarks | slim-multicluster-private | directory-conformance +# SUITE a2a | a2a-slimrpc | slim-integration | slim-benchmarks | slim-vs-a2a | slim-multicluster-private | directory-conformance # STAGING_DIR directory with downloaded artifact contents # SITE_DIR gh-pages docs root (e.g. site/docs; suite dirs a2a/, directory/, …) # REPO_ROOT csit checkout (for go run) @@ -99,6 +99,16 @@ case "$SUITE" in fi ;; + slim-vs-a2a) + # compare:report already produced a self-contained index.html (plus TSVs); + # publish it verbatim under benchmarks/slim-vs-a2a/. + if [[ -f "$STAGING_DIR/index.html" ]]; then + mkdir -p "$SITE_DIR/benchmarks/slim-vs-a2a" + cp -a "$STAGING_DIR/." "$SITE_DIR/benchmarks/slim-vs-a2a/" + has_report=true + fi + ;; + slim-multicluster-private) summary="" if [[ -f "$STAGING_DIR/summary.html" ]]; then diff --git a/.github/scripts/build-site-landing-page.sh b/.github/scripts/build-site-landing-page.sh index 6350f55a..7e81978d 100755 --- a/.github/scripts/build-site-landing-page.sh +++ b/.github/scripts/build-site-landing-page.sh @@ -261,6 +261,9 @@ HTML test-slim-benchmarks) blurb="Throughput and latency benchmark dashboards across modes, payload sizes, and sender counts." ;; + test-slim-vs-a2a) + blurb="SLIM native group-session multicast vs A2A relay-hub streaming: consensus latency, propagation, and RPC counts." + ;; test-slim-multicluster-private) blurb="Two-cluster SPIRE federation verification with private cluster B constraints." ;; diff --git a/.github/scripts/init-sources-json.sh b/.github/scripts/init-sources-json.sh index d1b5cf4d..36ac1ece 100755 --- a/.github/scripts/init-sources-json.sh +++ b/.github/scripts/init-sources-json.sh @@ -64,6 +64,17 @@ cat > "$defaults" <<'JSON' "published_report_run_id": "", "published_report_updated_at": "" }, + { + "id": "test-slim-vs-a2a", + "name": "SLIM vs A2A consensus", + "workflow_file": "test-slim-vs-a2a.yaml", + "report_path": "benchmarks/slim-vs-a2a/", + "last_run_id": "", + "last_run_conclusion": "", + "last_run_updated_at": "", + "published_report_run_id": "", + "published_report_updated_at": "" + }, { "id": "test-slim-multicluster-private", "name": "Slim multicluster private", diff --git a/.github/workflows/test-slim-vs-a2a.yaml b/.github/workflows/test-slim-vs-a2a.yaml index c47daae6..b6891007 100644 --- a/.github/workflows/test-slim-vs-a2a.yaml +++ b/.github/workflows/test-slim-vs-a2a.yaml @@ -1,3 +1,6 @@ +# Copyright AGNTCY Contributors (https://github.com/agntcy) +# SPDX-License-Identifier: Apache-2.0 + name: test-slim-vs-a2a on: @@ -16,8 +19,9 @@ on: workflow_dispatch: jobs: - slim-vs-a2a-smoke: + slim-vs-a2a: runs-on: ubuntu-latest + timeout-minutes: 30 permissions: contents: read @@ -31,25 +35,65 @@ jobs: with: go: true - - name: Install slimctl + - name: Run SLIM vs A2A comparison suite shell: bash + working-directory: benchmarks/slim-vs-a2a run: | - task benchmarks:slim-vs-a2a:deps:slimctl-download - echo "$(pwd)/benchmarks/agntcy-slim/bin" >> "$GITHUB_PATH" + set -o pipefail + task compare:suite - - name: Run SLIM vs A2A comparison smoke + - name: Build comparison report + if: always() shell: bash working-directory: benchmarks/slim-vs-a2a - env: - RUN_SLIM_VS_A2A: "1" + run: task compare:report + + - name: Publish step summary + if: always() + shell: bash run: | - set -o pipefail - task compare:ci:smoke + REPORT_DIR="benchmarks/slim-vs-a2a/reports" + { + echo "# SLIM vs A2A — Consensus Comparison" + echo + echo "- Workflow: ${GITHUB_WORKFLOW}" + echo "- Ref: ${GITHUB_REF}" + echo "- SHA: ${GITHUB_SHA}" + echo + if [ -f "$REPORT_DIR/results.tsv" ]; then + echo "## results.tsv" + echo + echo '```' + cat "$REPORT_DIR/results.tsv" + echo '```' + fi + } >> "$GITHUB_STEP_SUMMARY" - name: Upload comparison report if: always() uses: actions/upload-artifact@v4 with: - name: slim-vs-a2a-smoke-report + name: slim-vs-a2a-report path: benchmarks/slim-vs-a2a/reports/ + retention-days: 30 if-no-files-found: warn + + publish-pages: + needs: [slim-vs-a2a] + if: always() && github.ref == 'refs/heads/main' && needs.slim-vs-a2a.result != 'skipped' + runs-on: ubuntu-latest + timeout-minutes: 10 + concurrency: + group: test-report-pages + cancel-in-progress: false + permissions: + contents: write + actions: read + steps: + - uses: actions/checkout@v4 + + - uses: ./.github/actions/publish-test-reports-pages + with: + suite: slim-vs-a2a + source-run-id: ${{ github.run_id }} + source-run-conclusion: ${{ needs.slim-vs-a2a.result }} From 9550d92423d758c09f22be5a7061abc4f930e004 Mon Sep 17 00:00:00 2001 From: Magyari Sandor Szilard Date: Tue, 7 Jul 2026 13:25:58 +0200 Subject: [PATCH 6/6] feat: run multiple epochs Signed-off-by: Magyari Sandor Szilard --- benchmarks/slim-vs-a2a/Taskfile.yml | 23 +- benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go | 14 +- benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go | 154 ++++++++----- .../slim-vs-a2a/a2a/internal/agent/runtime.go | 104 +++++++-- .../slim-vs-a2a/a2a/internal/client/client.go | 4 +- .../a2a/internal/protocol/protocol.go | 2 + .../slim-vs-a2a/a2a/internal/relay/relay.go | 39 +++- .../internal/consensus/consensus.go | 38 +++ .../slim-vs-a2a/internal/metrics/metrics.go | 11 +- .../slim-vs-a2a/internal/scenario/scenario.go | 33 ++- benchmarks/slim-vs-a2a/plans/README.md | 28 ++- ...pothesis-convergence-10ag-20ms-5120b.yaml} | 10 +- .../hypothesis-convergence-10ag-20ms.yaml | 2 + .../hypothesis-convergence-17ag-20ms.yaml | 83 ------- .../hypothesis-convergence-20ag-20ms.yaml | 95 -------- .../hypothesis-convergence-50ag-20ms.yaml | 215 ----------------- ...ypothesis-convergence-50ag-5ms-10240b.yaml | 216 ------------------ .../hypothesis-convergence-50ag-5ms.yaml | 215 ----------------- ...l => hypothesis-convergence-5ag-10ms.yaml} | 28 +-- ...hypothesis-convergence-5ag-20ms-5120b.yaml | 38 +++ .../hypothesis-convergence-5ag-20ms.yaml | 2 + .../hypothesis-convergence-5ag-30ms.yaml | 37 +++ benchmarks/slim-vs-a2a/slim/cmd/agent/main.go | 10 +- .../slim-vs-a2a/slim/cmd/runner/main.go | 130 +++++++---- .../slim/internal/agent/runtime.go | 61 +++-- .../slim/internal/protocol/protocol.go | 6 +- .../slim-vs-a2a/tools/gen_scenario/main.go | 14 +- benchmarks/slim-vs-a2a/tools/report/main.go | 26 ++- 28 files changed, 602 insertions(+), 1036 deletions(-) rename benchmarks/slim-vs-a2a/plans/sweeps/{hypothesis-convergence-10ag-5ms-10240b.yaml => hypothesis-convergence-10ag-20ms-5120b.yaml} (88%) delete mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-17ag-20ms.yaml delete mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-20ag-20ms.yaml delete mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-20ms.yaml delete mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms-10240b.yaml delete mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms.yaml rename benchmarks/slim-vs-a2a/plans/sweeps/{hypothesis-convergence-10ag-5ms.yaml => hypothesis-convergence-5ag-10ms.yaml} (57%) create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms-5120b.yaml create mode 100644 benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-30ms.yaml diff --git a/benchmarks/slim-vs-a2a/Taskfile.yml b/benchmarks/slim-vs-a2a/Taskfile.yml index 57fa8929..388bb864 100644 --- a/benchmarks/slim-vs-a2a/Taskfile.yml +++ b/benchmarks/slim-vs-a2a/Taskfile.yml @@ -131,6 +131,7 @@ tasks: {{ .BIN_DIR }}/a2a-runner \ --scenario "$SCENARIO_PATH" \ --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --quiet \ --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ --output-tsv {{ .RESULTS_TSV }} - task: compare:slim-stack @@ -181,7 +182,7 @@ tasks: tls: insecure: true EOF - {{ .COMPARE_SLIMCTL }} slim start -c "$CONFIG" & + {{ .COMPARE_SLIMCTL }} slim start -c "$CONFIG" >>{{ .REPORTS_DIR }}/slimctl.log 2>&1 & SLIM_PID=$! for i in $(seq 1 50); do if nc -z 127.0.0.1 "${DATAPLANE_PORT}" 2>/dev/null; then @@ -221,6 +222,8 @@ tasks: SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES | default "0" }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS | default "10" }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS | default "30000" }}' cmds: - task: compare:cleanup - rm -f {{ .SWEEP_TSV }} @@ -230,6 +233,8 @@ tasks: SWEEP_AGENTS: '{{ .SWEEP_AGENTS }}' SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS }}' SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS }}' compare:sweep:payload: desc: High-N / short-window / large-payload sweep highlighting SLIM multicast (10kB default) @@ -240,6 +245,8 @@ tasks: SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "10,50" }}' SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "5" }}' SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES | default "0,10240" }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS | default "10" }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS | default "30000" }}' cmds: - task: compare:cleanup - rm -f {{ .SWEEP_TSV }} @@ -249,6 +256,8 @@ tasks: SWEEP_AGENTS: '{{ .SWEEP_AGENTS }}' SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS }}' SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS }}' - task: compare:report compare:sweep:run: @@ -258,6 +267,8 @@ tasks: SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES | default "0" }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS | default "10" }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS | default "30000" }}' cmds: - | set -euo pipefail @@ -276,6 +287,8 @@ tasks: -agents "$agents" \ -think-ms "$think" \ -payload-bytes "$payload" \ + -epochs {{ .SWEEP_EPOCHS }} \ + -max-epoch-ms {{ .SWEEP_MAX_EPOCH_MS }} \ -output "$scenario_path" base=$(basename "$scenario_path" .yaml) echo "==> sweep A2A $base" @@ -294,6 +307,8 @@ tasks: SWEEP_AGENTS: '{{ .SWEEP_AGENTS }}' SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS }}' SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS }}' compare:sweep:slim: internal: true @@ -304,6 +319,8 @@ tasks: SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES | default "0" }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS | default "10" }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS | default "30000" }}' cmds: - | set -euo pipefail @@ -334,7 +351,7 @@ tasks: tls: insecure: true EOF - {{ .COMPARE_SLIMCTL }} slim start -c "$CONFIG" & + {{ .COMPARE_SLIMCTL }} slim start -c "$CONFIG" >>{{ .REPORTS_DIR }}/slimctl.log 2>&1 & SLIM_PID=$! for i in $(seq 1 50); do if nc -z 127.0.0.1 "${DATAPLANE_PORT}" 2>/dev/null; then @@ -373,7 +390,7 @@ tasks: set -euo pipefail mkdir -p {{ .SCENARIO_DIR }} {{ .REPORTS_DIR }} scenario_path="{{ .SCENARIO_DIR }}/ci-smoke-hypothesis-5ag-20ms.yaml" - go run ./tools/gen_scenario -family hypothesis-convergence -agents 5 -think-ms 20 -output "$scenario_path" + go run ./tools/gen_scenario -family hypothesis-convergence -agents 5 -think-ms 20 -epochs 10 -max-epoch-ms 10000 -output "$scenario_path" echo "==> smoke A2A" {{ .BIN_DIR }}/a2a-runner \ --scenario "$scenario_path" \ diff --git a/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go b/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go index 9cc5a187..91fc4f66 100644 --- a/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go +++ b/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go @@ -19,6 +19,7 @@ import ( "github.com/a2aproject/a2a-go/v2/a2asrv/taskstore" agentrt "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/agent" "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" "google.golang.org/grpc" ) @@ -92,8 +93,13 @@ func main() { scenarioFile := flag.String("scenario-file", "", "consensus scenario yaml") agentIndex := flag.Int("agent-index", 0, "agent index") relayCardURL := flag.String("relay-card-url", "", "runner relay agent card base URL") + quiet := flag.Bool("quiet", false, "disable benchmark logs") flag.Parse() + if *quiet { + benchlog.SetEnabled(false) + } + if *agentID == "" || *grpcPort == 0 || *scenarioFile == "" || *relayCardURL == "" { log.Fatal("agent-id, grpc-port, scenario-file, and relay-card-url are required") } @@ -150,11 +156,9 @@ func main() { } }() - // Subscribe to the runner relay stream as soon as we are up; it retries - // until the relay server is reachable. - rt.StartRelaySubscription(context.Background()) - - fmt.Printf("A2A_AGENT_READY agent=%s grpc=%d card=%d\n", *agentID, *grpcPort, card) + if !*quiet { + fmt.Printf("A2A_AGENT_READY agent=%s grpc=%d card=%d\n", *agentID, *grpcPort, card) + } if err := grpcServer.Serve(grpcListener); err != nil { log.Fatalf("grpc serve: %v", err) } diff --git a/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go b/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go index 97cfe625..b31dd2e5 100644 --- a/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go +++ b/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go @@ -63,11 +63,12 @@ func main() { log.Fatalf("relay serve: %v", err) } - procs := startAgents(s, agentPath, *scenarioPath, hub.CardURL()) + procs := startAgents(s, agentPath, *scenarioPath, hub.CardURL(), *quiet) defer stopAgents(procs) time.Sleep(*waitReady) - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + overall := time.Duration(s.Spec.Epochs)*time.Duration(s.Spec.MaxEpochTimeMs)*time.Millisecond + 5*time.Minute + ctx, cancel := context.WithTimeout(context.Background(), overall) defer cancel() cli, err := newClientWithRetry(ctx, s) @@ -108,10 +109,6 @@ func main() { runStart := time.Now() benchlog.SetRunStart(runStart) - if err := cli.StartAll(ctx, agentIDs); err != nil { - stopAgents(procs) - log.Fatalf("start agents: %v", err) - } result := metrics.RunResult{ ScenarioName: s.Metadata.Name, @@ -120,37 +117,59 @@ func main() { Agents: len(s.Agents), ThinkTimeMs: s.Spec.ThinkTimeMs, PayloadBytes: s.Spec.PayloadBytes, + Epochs: s.Spec.Epochs, } - var snapshots []consensus.AgentSnapshot - deadline := time.Now().Add(2 * time.Minute) - for time.Now().Before(deadline) { - snapshots = snapshots[:0] - var pollErr error - for _, id := range agentIDs { - snap, err := cli.Snapshot(ctx, id) - if err != nil { - pollErr = err - break - } - snapshots = append(snapshots, snap) + maxEpochTime := time.Duration(s.Spec.MaxEpochTimeMs) * time.Millisecond + var ( + wallSamples []int64 + allSnapshots []consensus.AgentSnapshot + ) + for epoch := 0; epoch < s.Spec.Epochs; epoch++ { + log.Printf("epoch %d/%d started (budget %s)", epoch+1, s.Spec.Epochs, maxEpochTime) + hub.SetCurrentEpoch(epoch) + if err := cli.StartAll(ctx, agentIDs, epoch); err != nil { + stopAgents(procs) + log.Fatalf("start agents (epoch %d): %v", epoch, err) } - if pollErr == nil { - if ok, _ := consensus.GlobalConsensus(snapshots); ok { - result.Success = true - break + epochStart := time.Now() + deadline := epochStart.Add(maxEpochTime) + var snapshots []consensus.AgentSnapshot + success := false + for time.Now().Before(deadline) { + snapshots = snapshots[:0] + var pollErr error + for _, id := range agentIDs { + snap, err := cli.Snapshot(ctx, id) + if err != nil { + pollErr = err + break + } + snapshots = append(snapshots, snap) + } + // Only accept consensus once every agent has reset into this epoch. + if pollErr == nil && allForEpoch(snapshots, epoch) { + if ok, _ := consensus.GlobalConsensus(snapshots); ok { + success = true + break + } } + time.Sleep(20 * time.Millisecond) } - time.Sleep(20 * time.Millisecond) - } - - if !result.Success && len(snapshots) > 0 { - if ok, _ := consensus.GlobalConsensus(snapshots); ok { - result.Success = true + allSnapshots = append(allSnapshots, snapshots...) + epochWall := time.Since(epochStart) + if success { + result.EpochsSucceeded++ + wallSamples = append(wallSamples, epochWall.Milliseconds()) + log.Printf("epoch %d/%d ok wall_ms=%d", epoch+1, s.Spec.Epochs, epochWall.Milliseconds()) + } else { + result.EpochsFailed++ + log.Printf("epoch %d/%d failed after %s (budget %s exhausted)", epoch+1, s.Spec.Epochs, epochWall.Round(time.Millisecond), maxEpochTime) } } - result = aggregateResult(result, runStart, snapshots, hub) + result.Success = result.EpochsFailed == 0 + result = aggregateResult(result, wallSamples, allSnapshots, hub) stopAgents(procs) @@ -189,22 +208,41 @@ func newClientWithRetry(ctx context.Context, s *scenario.ConsensusScenario) (*cl return nil, lastErr } -func aggregateResult(result metrics.RunResult, runStart time.Time, snapshots []consensus.AgentSnapshot, hub *relay.Hub) metrics.RunResult { +// allForEpoch reports whether every polled snapshot belongs to the given epoch, +// i.e. all agents have reset into it. Empty input is treated as not-yet-ready. +func allForEpoch(snapshots []consensus.AgentSnapshot, epoch int) bool { + if len(snapshots) == 0 { + return false + } + for _, snap := range snapshots { + if snap.Epoch != epoch { + return false + } + } + return true +} + +// aggregateResult folds the snapshots collected across every epoch into the run +// result. wallSamples holds the wall-clock duration of each successful epoch. +func aggregateResult(result metrics.RunResult, wallSamples []int64, snapshots []consensus.AgentSnapshot, hub *relay.Hub) metrics.RunResult { + // Relay hub did all the fan-out work; report its real counters (cumulative + // across every epoch). + streamCount, fanoutMS := hub.Stats() + result.StreamRPCCount = streamCount + result.CoordFanoutMS = fanoutMS + if len(snapshots) == 0 { result.Error = "no agent snapshots" - result.ConsensusWallMS = time.Since(runStart).Milliseconds() return result } var ( totalEmitted int totalApplied int - lastConvergeMS int64 propDurations []int64 maxConsensusRound int maxRound int ) - for _, snap := range snapshots { totalEmitted += snap.FindingsEmitted totalApplied += snap.FindingsApplied @@ -214,25 +252,16 @@ func aggregateResult(result metrics.RunResult, runStart time.Time, snapshots []c if snap.Round > maxRound { maxRound = snap.Round } - if snap.ConvergedAtNs > 0 { - ms := (snap.ConvergedAtNs - runStart.UnixNano()) / int64(time.Millisecond) - if ms > lastConvergeMS { - lastConvergeMS = ms - } - } if snap.AvgPropagationMs > 0 { propDurations = append(propDurations, snap.AvgPropagationMs) } } - if result.Success { - result.ConsensusWallMS = time.Since(runStart).Milliseconds() - if lastConvergeMS > 0 { - result.ConsensusWallMS = lastConvergeMS - } - } else { - result.ConsensusWallMS = time.Since(runStart).Milliseconds() - result.Error = "consensus not reached" + meanWall, maxWall := summarizeWall(wallSamples) + result.ConsensusWallMS = meanWall + result.LastAgentConvergeMS = maxWall + if result.EpochsSucceeded == 0 { + result.Error = "consensus not reached in any epoch" } result.ConsensusRound = maxConsensusRound @@ -241,27 +270,40 @@ func aggregateResult(result metrics.RunResult, runStart time.Time, snapshots []c } result.FindingsEmitted = totalEmitted result.FindingsReceivedTotal = totalApplied - result.LastAgentConvergeMS = lastConvergeMS result.AvgPropagationMS, result.P95PropagationMS = metrics.AggregatePropagation(propDurations) - // Relay hub did all the fan-out work; report its real counters. - streamCount, fanoutMS := hub.Stats() - result.StreamRPCCount = streamCount - result.CoordFanoutMS = fanoutMS return result } -func startAgents(s *scenario.ConsensusScenario, agentBin, scenarioFile, relayCardURL string) []*exec.Cmd { +// summarizeWall returns the mean and max of the per-epoch wall-clock samples. +func summarizeWall(wallSamples []int64) (mean, max int64) { + if len(wallSamples) == 0 { + return 0, 0 + } + var sum int64 + for _, w := range wallSamples { + sum += w + if w > max { + max = w + } + } + return sum / int64(len(wallSamples)), max +} + +func startAgents(s *scenario.ConsensusScenario, agentBin, scenarioFile, relayCardURL string, quiet bool) []*exec.Cmd { var procs []*exec.Cmd for i, agent := range s.Agents { - cmd := exec.Command( - agentBin, + args := []string{ "--agent-id", agent.ID, "--grpc-port", fmt.Sprintf("%d", agent.A2APort), "--card-port", fmt.Sprintf("%d", s.CardPort(agent)), "--scenario-file", scenarioFile, "--agent-index", fmt.Sprintf("%d", i), "--relay-card-url", relayCardURL, - ) + } + if quiet { + args = append(args, "--quiet") + } + cmd := exec.Command(agentBin, args...) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr if err := cmd.Start(); err != nil { diff --git a/benchmarks/slim-vs-a2a/a2a/internal/agent/runtime.go b/benchmarks/slim-vs-a2a/a2a/internal/agent/runtime.go index 62e3bd67..01330625 100644 --- a/benchmarks/slim-vs-a2a/a2a/internal/agent/runtime.go +++ b/benchmarks/slim-vs-a2a/a2a/internal/agent/runtime.go @@ -19,7 +19,6 @@ import ( "fmt" "iter" "sync" - "sync/atomic" "time" "github.com/a2aproject/a2a-go/v2/a2a" @@ -44,30 +43,62 @@ type Runtime struct { engine *consensus.Engine outbound chan consensus.Finding - running atomic.Bool - done chan struct{} startedAt time.Time - relayOnce sync.Once + + relayMu sync.Mutex + relayCancel context.CancelFunc + + mu sync.Mutex + // lastStartedEpoch dedupes repeated OpStart for the same epoch; + // currentEpoch is the attempt currently running (used to drop stale relayed + // findings and to stop a superseded epoch loop). + lastStartedEpoch int + currentEpoch int } func NewRuntime(s *scenario.ConsensusScenario, agentIndex int, agentID, relayCardURL string) *Runtime { return &Runtime{ - scenario: s, - agentIndex: agentIndex, - agentID: agentID, - relayCardURL: relayCardURL, - engine: consensus.NewEngine(s.Spec, agentIndex), - outbound: make(chan consensus.Finding, 1024), - done: make(chan struct{}), + scenario: s, + agentIndex: agentIndex, + agentID: agentID, + relayCardURL: relayCardURL, + engine: consensus.NewEngine(s.Spec, agentIndex), + outbound: make(chan consensus.Finding, 1024), + lastStartedEpoch: -1, + currentEpoch: -1, + } +} + +// stopRelay ends the current OpStreamRelay subscription, if any. +func (r *Runtime) stopRelay() { + r.relayMu.Lock() + if r.relayCancel != nil { + r.relayCancel() + r.relayCancel = nil } + r.relayMu.Unlock() } -// StartRelaySubscription dials the runner relay and subscribes to the relayed -// finding stream (OpStreamRelay). It runs once and retries until the relay is up. -func (r *Runtime) StartRelaySubscription(ctx context.Context) { - r.relayOnce.Do(func() { - go r.relayLoop(ctx) - }) +// startRelay dials the runner relay and subscribes to OpStreamRelay for the +// current epoch. Each epoch gets a fresh stream so a2a-go task history does +// not accumulate across attempts. +func (r *Runtime) startRelay() { + r.stopRelay() + relayCtx, cancel := context.WithCancel(context.Background()) + r.relayMu.Lock() + r.relayCancel = cancel + r.relayMu.Unlock() + go r.relayLoop(relayCtx) +} + +func (r *Runtime) drainOutbound() { + for { + select { + case <-r.outbound: + default: + return + } + } } func (r *Runtime) relayLoop(ctx context.Context) { @@ -95,6 +126,13 @@ func (r *Runtime) relayLoop(ctx context.Context) { if !ok || f.AgentIndex == r.agentIndex { continue } + // Drop stragglers from a different (usually previous) epoch. + r.mu.Lock() + cur := r.currentEpoch + r.mu.Unlock() + if f.Epoch != cur { + continue + } r.applyFinding(f) } select { @@ -129,7 +167,7 @@ func (r *Runtime) dialRelay(ctx context.Context) *a2aclient.Client { func (r *Runtime) HandleUnary(_ context.Context, req protocol.Request) protocol.Response { switch req.Op { case protocol.OpStart: - r.StartRun() + r.startEpoch(req.Epoch) return protocol.Response{OK: true} case protocol.OpSnapshot: body, err := json.Marshal(r.engine.Snapshot()) @@ -172,22 +210,42 @@ func (r *Runtime) StreamFindings(ctx context.Context, execCtx *a2asrv.ExecutorCo } } -func (r *Runtime) StartRun() { - if r.running.Swap(true) { +// startEpoch resets the engine for a fresh attempt and launches the epoch loop. +// Repeated starts for the same (or older) epoch are ignored so a duplicated +// call cannot double-run an attempt. +func (r *Runtime) startEpoch(epoch int) { + r.mu.Lock() + if epoch <= r.lastStartedEpoch { + r.mu.Unlock() return } + r.lastStartedEpoch = epoch + r.currentEpoch = epoch + r.mu.Unlock() + + r.stopRelay() + r.drainOutbound() + r.engine.Reset(epoch) r.startedAt = time.Now() benchlog.SetRunStart(r.startedAt) - go r.runLoop() + r.startRelay() + go r.runEpoch(epoch) } -func (r *Runtime) runLoop() { - defer close(r.done) +func (r *Runtime) runEpoch(epoch int) { spec := r.scenario.Spec think := time.Duration(spec.ThinkTimeMs) * time.Millisecond emitGap := time.Duration(spec.FindingEmitDelayMs) * time.Millisecond for round := 0; round < spec.MaxRounds; round++ { + // Stop early if a newer epoch has superseded this one. + r.mu.Lock() + superseded := r.currentEpoch != epoch + r.mu.Unlock() + if superseded { + return + } + time.Sleep(think) finding, emit := r.engine.Think() diff --git a/benchmarks/slim-vs-a2a/a2a/internal/client/client.go b/benchmarks/slim-vs-a2a/a2a/internal/client/client.go index 6dff1d5d..a6c537d4 100644 --- a/benchmarks/slim-vs-a2a/a2a/internal/client/client.go +++ b/benchmarks/slim-vs-a2a/a2a/internal/client/client.go @@ -44,9 +44,9 @@ func New(ctx context.Context, s *scenario.ConsensusScenario) (*Client, error) { return &Client{clients: clients}, nil } -func (c *Client) StartAll(ctx context.Context, agentIDs []string) error { +func (c *Client) StartAll(ctx context.Context, agentIDs []string, epoch int) error { for _, id := range agentIDs { - if err := c.send(ctx, id, protocol.Request{Op: protocol.OpStart}); err != nil { + if err := c.send(ctx, id, protocol.Request{Op: protocol.OpStart, Epoch: epoch}); err != nil { return err } } diff --git a/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go b/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go index 21f53971..12a07643 100644 --- a/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go +++ b/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go @@ -30,6 +30,8 @@ type Request struct { // AgentIndex identifies the subscriber on OpStreamRelay so the relay can // avoid echoing an agent's own findings back to it. AgentIndex int `json:"agentIndex,omitempty"` + // Epoch identifies which consensus attempt an OpStart begins. + Epoch int `json:"epoch,omitempty"` } type Response struct { diff --git a/benchmarks/slim-vs-a2a/a2a/internal/relay/relay.go b/benchmarks/slim-vs-a2a/a2a/internal/relay/relay.go index 0a44c285..935303ec 100644 --- a/benchmarks/slim-vs-a2a/a2a/internal/relay/relay.go +++ b/benchmarks/slim-vs-a2a/a2a/internal/relay/relay.go @@ -8,7 +8,9 @@ // subscribes and receives a server stream of every relayed finding. The runner // separately subscribes to each agent's OpStreamFindings stream and calls // Broadcast for every finding, which fans it out to all other agents' relay -// streams. This relay fan-out is exactly the work native SLIM avoids. +// streams. Broadcast drops findings whose epoch does not match the hub's +// current epoch so stragglers from a prior attempt are not relayed. This +// relay fan-out is exactly the work native SLIM avoids. package relay import ( @@ -18,6 +20,7 @@ import ( "net" "net/http" "sync" + "sync/atomic" "time" "github.com/a2aproject/a2a-go/v2/a2a" @@ -42,6 +45,10 @@ type Hub struct { subs map[*subscriber]struct{} streamRPCCount int fanoutMS int64 + // currentEpoch is the consensus attempt the hub will relay. Findings tagged + // with a different epoch are dropped so stragglers from a prior attempt do + // not fan out or grow relay-stream task history. + currentEpoch atomic.Int32 } func NewHub(grpcPort, cardPort int) *Hub { @@ -104,8 +111,32 @@ func (h *Hub) Serve() error { return nil } +// ResetStreams closes every active relay subscription so agents open fresh +// OpStreamRelay tasks on the next epoch instead of growing one task forever. +func (h *Hub) ResetStreams() { + h.mu.Lock() + subs := make([]*subscriber, 0, len(h.subs)) + for s := range h.subs { + subs = append(subs, s) + } + h.subs = map[*subscriber]struct{}{} + h.mu.Unlock() + for _, s := range subs { + close(s.ch) + } +} + +// SetCurrentEpoch advances the relay filter to the given epoch. Only findings +// stamped with this epoch are broadcast; older (or future) epochs are ignored. +func (h *Hub) SetCurrentEpoch(epoch int) { + h.currentEpoch.Store(int32(epoch)) +} + // Broadcast fans a finding out to every subscriber except its producer. func (h *Hub) Broadcast(f consensus.Finding) { + if int(h.currentEpoch.Load()) != f.Epoch { + return + } start := time.Now() h.mu.Lock() targets := make([]*subscriber, 0, len(h.subs)) @@ -181,7 +212,11 @@ func (e *relayExecutor) Execute(ctx context.Context, execCtx *a2asrv.ExecutorCon case <-ctx.Done(): yield(a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateCompleted, nil), nil) return - case f := <-sub.ch: + case f, ok := <-sub.ch: + if !ok { + yield(a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateCompleted, nil), nil) + return + } body, err := consensus.EncodeFinding(f) if err != nil { continue diff --git a/benchmarks/slim-vs-a2a/internal/consensus/consensus.go b/benchmarks/slim-vs-a2a/internal/consensus/consensus.go index 4015cb86..ec08b6d3 100644 --- a/benchmarks/slim-vs-a2a/internal/consensus/consensus.go +++ b/benchmarks/slim-vs-a2a/internal/consensus/consensus.go @@ -18,6 +18,7 @@ const consensusConfidence = 0.55 type Finding struct { FindingID int64 `json:"findingId"` AgentIndex int `json:"agentIndex"` + Epoch int `json:"epoch,omitempty"` Round int `json:"round"` Value int `json:"value"` Confidence float64 `json:"confidence"` @@ -42,6 +43,7 @@ type Engine struct { mu sync.Mutex + epoch int round int value int confidence float64 @@ -85,6 +87,33 @@ func NewEngine(spec scenario.Spec, agentIndex int) *Engine { } } +// Reset restores the engine to its initial per-epoch state so the same +// consensus attempt can be re-run. The topology-defining fields (targetValue, +// payload, cfg) are preserved so every epoch is an identical repeat; only the +// accumulated round/convergence/propagation state is cleared. The new epoch +// index is stamped onto every finding and snapshot the engine produces next. +func (e *Engine) Reset(epoch int) { + e.mu.Lock() + e.epoch = epoch + e.round = 0 + e.value = e.cfg.AgentIndex % e.cfg.ValueSpace + e.confidence = 0.1 + e.lastEmitValue = -1 + e.lastEmitConf = 0 + e.lastEmitRound = 0 + e.nextFindingID = 0 + e.distinctSupports = map[int]map[int]struct{}{} + e.convergedAt = 0 + e.consensusRound = 0 + e.findingsEmitted = 0 + e.findingsApplied = 0 + e.mu.Unlock() + + e.propMu.Lock() + e.propDurations = nil + e.propMu.Unlock() +} + // makePayload returns deterministic, reproducible padding of n bytes. It is used // only to inflate finding size for transport stress tests, so math/rand is // intentional here: the same seed/agent yields identical bytes across both @@ -146,6 +175,7 @@ func (e *Engine) Think() (finding *Finding, emit bool) { f := Finding{ FindingID: e.nextFindingID, AgentIndex: e.cfg.AgentIndex, + Epoch: e.epoch, Round: e.round, Value: e.value, Confidence: e.confidence, @@ -286,6 +316,7 @@ func percentile(values []int64, p float64) int64 { type AgentSnapshot struct { AgentIndex int `json:"agentIndex"` + Epoch int `json:"epoch,omitempty"` Value int `json:"value"` Confidence float64 `json:"confidence"` Round int `json:"round"` @@ -298,11 +329,18 @@ type AgentSnapshot struct { P95PropagationMs int64 `json:"p95PropagationMs"` } +func (e *Engine) Epoch() int { + e.mu.Lock() + defer e.mu.Unlock() + return e.epoch +} + func (e *Engine) Snapshot() AgentSnapshot { value, conf, round := e.LocalState() avg, p95 := e.PropagationStats() return AgentSnapshot{ AgentIndex: e.cfg.AgentIndex, + Epoch: e.Epoch(), Value: value, Confidence: conf, Round: round, diff --git a/benchmarks/slim-vs-a2a/internal/metrics/metrics.go b/benchmarks/slim-vs-a2a/internal/metrics/metrics.go index d9925cb3..a8a30834 100644 --- a/benchmarks/slim-vs-a2a/internal/metrics/metrics.go +++ b/benchmarks/slim-vs-a2a/internal/metrics/metrics.go @@ -27,7 +27,9 @@ type RunResult struct { LastAgentConvergeMS int64 `json:"last_agent_converge_ms"` CoordFanoutMS int64 `json:"coord_fanout_ms"` StreamRPCCount int `json:"stream_rpc_count"` - UnicastRPCCount int `json:"unicast_rpc_count"` + Epochs int `json:"epochs"` + EpochsSucceeded int `json:"epochs_succeeded"` + EpochsFailed int `json:"epochs_failed"` Success bool `json:"success"` Error string `json:"error,omitempty"` } @@ -68,7 +70,8 @@ func AppendTSV(path string, result RunResult) error { "payload_bytes", "consensus_wall_ms", "consensus_round", "findings_emitted", "findings_received_total", "avg_propagation_ms", "p95_propagation_ms", "last_agent_converge_ms", - "coord_fanout_ms", "stream_rpc_count", "unicast_rpc_count", + "coord_fanout_ms", "stream_rpc_count", + "epochs", "epochs_succeeded", "epochs_failed", "success", "error", }); err != nil { return err @@ -90,7 +93,9 @@ func AppendTSV(path string, result RunResult) error { strconv.FormatInt(result.LastAgentConvergeMS, 10), strconv.FormatInt(result.CoordFanoutMS, 10), strconv.Itoa(result.StreamRPCCount), - strconv.Itoa(result.UnicastRPCCount), + strconv.Itoa(result.Epochs), + strconv.Itoa(result.EpochsSucceeded), + strconv.Itoa(result.EpochsFailed), strconv.FormatBool(result.Success), result.Error, }); err != nil { diff --git a/benchmarks/slim-vs-a2a/internal/scenario/scenario.go b/benchmarks/slim-vs-a2a/internal/scenario/scenario.go index 966485f3..3d5bf3dc 100644 --- a/benchmarks/slim-vs-a2a/internal/scenario/scenario.go +++ b/benchmarks/slim-vs-a2a/internal/scenario/scenario.go @@ -40,6 +40,13 @@ type Spec struct { // PayloadBytes inflates every finding with fixed-size padding to stress // transport bandwidth. 0 (default) keeps findings at their minimal size. PayloadBytes int `yaml:"payloadBytes,omitempty"` + // Epochs is how many times the same consensus attempt is repeated. Each + // epoch resets the agents and re-runs; the runner counts successful vs + // failed epochs. Defaults to 10. + Epochs int `yaml:"epochs,omitempty"` + // MaxEpochTimeMs is the per-epoch wall-clock budget: if global consensus is + // not reached within it, the epoch is counted as failed. Defaults to 120000. + MaxEpochTimeMs int64 `yaml:"maxEpochTimeMs,omitempty"` } type Agent struct { @@ -99,6 +106,12 @@ func (s *ConsensusScenario) Validate() error { if s.Spec.PayloadBytes < 0 { s.Spec.PayloadBytes = 0 } + if s.Spec.Epochs <= 0 { + s.Spec.Epochs = 10 + } + if s.Spec.MaxEpochTimeMs <= 0 { + s.Spec.MaxEpochTimeMs = 120000 + } for i, a := range s.Agents { if a.ID == "" { return fmt.Errorf("agents[%d].id is required", i) @@ -170,11 +183,13 @@ func (s *ConsensusScenario) SlimNames() []string { } type GenerateOptions struct { - Family string - Agents int - ThinkTimeMs int64 - Seed int64 - PayloadBytes int + Family string + Agents int + ThinkTimeMs int64 + Seed int64 + PayloadBytes int + Epochs int + MaxEpochTimeMs int64 } func Generate(opts GenerateOptions) (*ConsensusScenario, error) { @@ -193,6 +208,12 @@ func Generate(opts GenerateOptions) (*ConsensusScenario, error) { if opts.PayloadBytes < 0 { opts.PayloadBytes = 0 } + if opts.Epochs <= 0 { + opts.Epochs = 10 + } + if opts.MaxEpochTimeMs <= 0 { + opts.MaxEpochTimeMs = 120000 + } name := fmt.Sprintf("%s-%dagents-%dms", opts.Family, opts.Agents, opts.ThinkTimeMs) if opts.PayloadBytes > 0 { @@ -230,6 +251,8 @@ func Generate(opts GenerateOptions) (*ConsensusScenario, error) { Seed: opts.Seed, ValueSpace: 3, PayloadBytes: opts.PayloadBytes, + Epochs: opts.Epochs, + MaxEpochTimeMs: opts.MaxEpochTimeMs, }, Agents: agents, } diff --git a/benchmarks/slim-vs-a2a/plans/README.md b/benchmarks/slim-vs-a2a/plans/README.md index ef5afb3e..699316d2 100644 --- a/benchmarks/slim-vs-a2a/plans/README.md +++ b/benchmarks/slim-vs-a2a/plans/README.md @@ -35,7 +35,9 @@ spec: targetMode: majority seed: 42 valueSpace: 3 - payloadBytes: 0 # optional fixed-size padding added to every finding + payloadBytes: 0 # optional fixed-size padding added to every finding + epochs: 10 # consensus attempts per run (default 10) + maxEpochTimeMs: 120000 # per-epoch consensus budget; exceeding it fails the epoch (default 120000) agents: - id: agent-0 slimName: agntcy/bench-v2/agent-0 @@ -55,9 +57,33 @@ go run ./tools/gen_scenario \ -agents 10 \ -think-ms 20 \ -payload-bytes 0 \ + -epochs 10 \ + -max-epoch-ms 120000 \ -output plans/sweeps/hypothesis-convergence-10ag-20ms.yaml ``` +### Epochs (reliability knob) + +Each run repeats the *same* consensus attempt over `spec.epochs` epochs. An epoch +succeeds if every agent reaches global consensus within `spec.maxEpochTimeMs` +(each attempt still runs up to `maxRounds`); otherwise it is counted as a failed +epoch. Between epochs the agents reset their engine in place and re-run — the SLIM +group session and A2A relay subscriptions stay up for the whole run, so there is a +single teardown. The runner reports `epochs`, `epochs_succeeded`, and +`epochs_failed`; overall `success` is true only when no epoch failed. This +surfaces reliability differences under load: A2A may miss the per-epoch budget +while SLIM still converges. + +Sweeps expose two variables (defaults shown): `SWEEP_EPOCHS=10` and +`SWEEP_MAX_EPOCH_MS=30000` (a modest budget keeps failed A2A epochs capping +quickly). For example: + +```bash +task compare:sweep:payload \ + SWEEP_AGENTS=10,50 SWEEP_THINK_MS=5 SWEEP_PAYLOAD_BYTES=0,10240 \ + SWEEP_EPOCHS=10 SWEEP_MAX_EPOCH_MS=30000 +``` + ### Payload size (transport-stress) knob `spec.payloadBytes` adds fixed-size deterministic padding to every finding. It is diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms-10240b.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms-5120b.yaml similarity index 88% rename from benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms-10240b.yaml rename to benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms-5120b.yaml index abdd9f0f..185c7e51 100644 --- a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms-10240b.yaml +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms-5120b.yaml @@ -1,18 +1,20 @@ apiVersion: bench.agntcy.io/v2 kind: ConsensusScenario metadata: - name: hypothesis-convergence-10agents-5ms-10240b + name: hypothesis-convergence-10agents-20ms-5120b domain: hypothesis-convergence description: Generated consensus scenario for transport sweeps spec: agents: 10 - thinkTimeMs: 5 - findingEmitDelayMs: 1 + thinkTimeMs: 20 + findingEmitDelayMs: 2 maxRounds: 200 targetMode: majority seed: 42 valueSpace: 3 - payloadBytes: 10240 + payloadBytes: 5120 + epochs: 10 + maxEpochTimeMs: 30000 agents: - id: agent-0 slimName: agntcy/bench-v2/agent-0 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml index 51f4402d..0f6ee819 100644 --- a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml @@ -12,6 +12,8 @@ spec: targetMode: majority seed: 42 valueSpace: 3 + epochs: 10 + maxEpochTimeMs: 30000 agents: - id: agent-0 slimName: agntcy/bench-v2/agent-0 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-17ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-17ag-20ms.yaml deleted file mode 100644 index 2100ca6e..00000000 --- a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-17ag-20ms.yaml +++ /dev/null @@ -1,83 +0,0 @@ -apiVersion: bench.agntcy.io/v2 -kind: ConsensusScenario -metadata: - name: hypothesis-convergence-17agents-20ms - domain: hypothesis-convergence - description: Generated consensus scenario for transport sweeps -spec: - agents: 17 - thinkTimeMs: 20 - findingEmitDelayMs: 2 - maxRounds: 200 - targetMode: majority - seed: 42 - valueSpace: 3 -agents: - - id: agent-0 - slimName: agntcy/bench-v2/agent-0 - a2aPort: 9700 - role: coordinator - - id: agent-1 - slimName: agntcy/bench-v2/agent-1 - a2aPort: 9711 - role: worker - - id: agent-2 - slimName: agntcy/bench-v2/agent-2 - a2aPort: 9722 - role: worker - - id: agent-3 - slimName: agntcy/bench-v2/agent-3 - a2aPort: 9733 - role: worker - - id: agent-4 - slimName: agntcy/bench-v2/agent-4 - a2aPort: 9744 - role: worker - - id: agent-5 - slimName: agntcy/bench-v2/agent-5 - a2aPort: 9755 - role: worker - - id: agent-6 - slimName: agntcy/bench-v2/agent-6 - a2aPort: 9766 - role: worker - - id: agent-7 - slimName: agntcy/bench-v2/agent-7 - a2aPort: 9777 - role: worker - - id: agent-8 - slimName: agntcy/bench-v2/agent-8 - a2aPort: 9788 - role: worker - - id: agent-9 - slimName: agntcy/bench-v2/agent-9 - a2aPort: 9799 - role: worker - - id: agent-10 - slimName: agntcy/bench-v2/agent-10 - a2aPort: 9810 - role: worker - - id: agent-11 - slimName: agntcy/bench-v2/agent-11 - a2aPort: 9821 - role: worker - - id: agent-12 - slimName: agntcy/bench-v2/agent-12 - a2aPort: 9832 - role: worker - - id: agent-13 - slimName: agntcy/bench-v2/agent-13 - a2aPort: 9843 - role: worker - - id: agent-14 - slimName: agntcy/bench-v2/agent-14 - a2aPort: 9854 - role: worker - - id: agent-15 - slimName: agntcy/bench-v2/agent-15 - a2aPort: 9865 - role: worker - - id: agent-16 - slimName: agntcy/bench-v2/agent-16 - a2aPort: 9876 - role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-20ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-20ag-20ms.yaml deleted file mode 100644 index a9142590..00000000 --- a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-20ag-20ms.yaml +++ /dev/null @@ -1,95 +0,0 @@ -apiVersion: bench.agntcy.io/v2 -kind: ConsensusScenario -metadata: - name: hypothesis-convergence-20agents-20ms - domain: hypothesis-convergence - description: Generated consensus scenario for transport sweeps -spec: - agents: 20 - thinkTimeMs: 20 - findingEmitDelayMs: 2 - maxRounds: 200 - targetMode: majority - seed: 42 - valueSpace: 3 -agents: - - id: agent-0 - slimName: agntcy/bench-v2/agent-0 - a2aPort: 9700 - role: coordinator - - id: agent-1 - slimName: agntcy/bench-v2/agent-1 - a2aPort: 9711 - role: worker - - id: agent-2 - slimName: agntcy/bench-v2/agent-2 - a2aPort: 9722 - role: worker - - id: agent-3 - slimName: agntcy/bench-v2/agent-3 - a2aPort: 9733 - role: worker - - id: agent-4 - slimName: agntcy/bench-v2/agent-4 - a2aPort: 9744 - role: worker - - id: agent-5 - slimName: agntcy/bench-v2/agent-5 - a2aPort: 9755 - role: worker - - id: agent-6 - slimName: agntcy/bench-v2/agent-6 - a2aPort: 9766 - role: worker - - id: agent-7 - slimName: agntcy/bench-v2/agent-7 - a2aPort: 9777 - role: worker - - id: agent-8 - slimName: agntcy/bench-v2/agent-8 - a2aPort: 9788 - role: worker - - id: agent-9 - slimName: agntcy/bench-v2/agent-9 - a2aPort: 9799 - role: worker - - id: agent-10 - slimName: agntcy/bench-v2/agent-10 - a2aPort: 9810 - role: worker - - id: agent-11 - slimName: agntcy/bench-v2/agent-11 - a2aPort: 9821 - role: worker - - id: agent-12 - slimName: agntcy/bench-v2/agent-12 - a2aPort: 9832 - role: worker - - id: agent-13 - slimName: agntcy/bench-v2/agent-13 - a2aPort: 9843 - role: worker - - id: agent-14 - slimName: agntcy/bench-v2/agent-14 - a2aPort: 9854 - role: worker - - id: agent-15 - slimName: agntcy/bench-v2/agent-15 - a2aPort: 9865 - role: worker - - id: agent-16 - slimName: agntcy/bench-v2/agent-16 - a2aPort: 9876 - role: worker - - id: agent-17 - slimName: agntcy/bench-v2/agent-17 - a2aPort: 9887 - role: worker - - id: agent-18 - slimName: agntcy/bench-v2/agent-18 - a2aPort: 9898 - role: worker - - id: agent-19 - slimName: agntcy/bench-v2/agent-19 - a2aPort: 9909 - role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-20ms.yaml deleted file mode 100644 index c36402b9..00000000 --- a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-20ms.yaml +++ /dev/null @@ -1,215 +0,0 @@ -apiVersion: bench.agntcy.io/v2 -kind: ConsensusScenario -metadata: - name: hypothesis-convergence-50agents-20ms - domain: hypothesis-convergence - description: Generated consensus scenario for transport sweeps -spec: - agents: 50 - thinkTimeMs: 20 - findingEmitDelayMs: 2 - maxRounds: 200 - targetMode: majority - seed: 42 - valueSpace: 3 -agents: - - id: agent-0 - slimName: agntcy/bench-v2/agent-0 - a2aPort: 9700 - role: coordinator - - id: agent-1 - slimName: agntcy/bench-v2/agent-1 - a2aPort: 9711 - role: worker - - id: agent-2 - slimName: agntcy/bench-v2/agent-2 - a2aPort: 9722 - role: worker - - id: agent-3 - slimName: agntcy/bench-v2/agent-3 - a2aPort: 9733 - role: worker - - id: agent-4 - slimName: agntcy/bench-v2/agent-4 - a2aPort: 9744 - role: worker - - id: agent-5 - slimName: agntcy/bench-v2/agent-5 - a2aPort: 9755 - role: worker - - id: agent-6 - slimName: agntcy/bench-v2/agent-6 - a2aPort: 9766 - role: worker - - id: agent-7 - slimName: agntcy/bench-v2/agent-7 - a2aPort: 9777 - role: worker - - id: agent-8 - slimName: agntcy/bench-v2/agent-8 - a2aPort: 9788 - role: worker - - id: agent-9 - slimName: agntcy/bench-v2/agent-9 - a2aPort: 9799 - role: worker - - id: agent-10 - slimName: agntcy/bench-v2/agent-10 - a2aPort: 9810 - role: worker - - id: agent-11 - slimName: agntcy/bench-v2/agent-11 - a2aPort: 9821 - role: worker - - id: agent-12 - slimName: agntcy/bench-v2/agent-12 - a2aPort: 9832 - role: worker - - id: agent-13 - slimName: agntcy/bench-v2/agent-13 - a2aPort: 9843 - role: worker - - id: agent-14 - slimName: agntcy/bench-v2/agent-14 - a2aPort: 9854 - role: worker - - id: agent-15 - slimName: agntcy/bench-v2/agent-15 - a2aPort: 9865 - role: worker - - id: agent-16 - slimName: agntcy/bench-v2/agent-16 - a2aPort: 9876 - role: worker - - id: agent-17 - slimName: agntcy/bench-v2/agent-17 - a2aPort: 9887 - role: worker - - id: agent-18 - slimName: agntcy/bench-v2/agent-18 - a2aPort: 9898 - role: worker - - id: agent-19 - slimName: agntcy/bench-v2/agent-19 - a2aPort: 9909 - role: worker - - id: agent-20 - slimName: agntcy/bench-v2/agent-20 - a2aPort: 9920 - role: worker - - id: agent-21 - slimName: agntcy/bench-v2/agent-21 - a2aPort: 9931 - role: worker - - id: agent-22 - slimName: agntcy/bench-v2/agent-22 - a2aPort: 9942 - role: worker - - id: agent-23 - slimName: agntcy/bench-v2/agent-23 - a2aPort: 9953 - role: worker - - id: agent-24 - slimName: agntcy/bench-v2/agent-24 - a2aPort: 9964 - role: worker - - id: agent-25 - slimName: agntcy/bench-v2/agent-25 - a2aPort: 9975 - role: worker - - id: agent-26 - slimName: agntcy/bench-v2/agent-26 - a2aPort: 9986 - role: worker - - id: agent-27 - slimName: agntcy/bench-v2/agent-27 - a2aPort: 9997 - role: worker - - id: agent-28 - slimName: agntcy/bench-v2/agent-28 - a2aPort: 10008 - role: worker - - id: agent-29 - slimName: agntcy/bench-v2/agent-29 - a2aPort: 10019 - role: worker - - id: agent-30 - slimName: agntcy/bench-v2/agent-30 - a2aPort: 10030 - role: worker - - id: agent-31 - slimName: agntcy/bench-v2/agent-31 - a2aPort: 10041 - role: worker - - id: agent-32 - slimName: agntcy/bench-v2/agent-32 - a2aPort: 10052 - role: worker - - id: agent-33 - slimName: agntcy/bench-v2/agent-33 - a2aPort: 10063 - role: worker - - id: agent-34 - slimName: agntcy/bench-v2/agent-34 - a2aPort: 10074 - role: worker - - id: agent-35 - slimName: agntcy/bench-v2/agent-35 - a2aPort: 10085 - role: worker - - id: agent-36 - slimName: agntcy/bench-v2/agent-36 - a2aPort: 10096 - role: worker - - id: agent-37 - slimName: agntcy/bench-v2/agent-37 - a2aPort: 10107 - role: worker - - id: agent-38 - slimName: agntcy/bench-v2/agent-38 - a2aPort: 10118 - role: worker - - id: agent-39 - slimName: agntcy/bench-v2/agent-39 - a2aPort: 10129 - role: worker - - id: agent-40 - slimName: agntcy/bench-v2/agent-40 - a2aPort: 10140 - role: worker - - id: agent-41 - slimName: agntcy/bench-v2/agent-41 - a2aPort: 10151 - role: worker - - id: agent-42 - slimName: agntcy/bench-v2/agent-42 - a2aPort: 10162 - role: worker - - id: agent-43 - slimName: agntcy/bench-v2/agent-43 - a2aPort: 10173 - role: worker - - id: agent-44 - slimName: agntcy/bench-v2/agent-44 - a2aPort: 10184 - role: worker - - id: agent-45 - slimName: agntcy/bench-v2/agent-45 - a2aPort: 10195 - role: worker - - id: agent-46 - slimName: agntcy/bench-v2/agent-46 - a2aPort: 10206 - role: worker - - id: agent-47 - slimName: agntcy/bench-v2/agent-47 - a2aPort: 10217 - role: worker - - id: agent-48 - slimName: agntcy/bench-v2/agent-48 - a2aPort: 10228 - role: worker - - id: agent-49 - slimName: agntcy/bench-v2/agent-49 - a2aPort: 10239 - role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms-10240b.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms-10240b.yaml deleted file mode 100644 index 898f8725..00000000 --- a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms-10240b.yaml +++ /dev/null @@ -1,216 +0,0 @@ -apiVersion: bench.agntcy.io/v2 -kind: ConsensusScenario -metadata: - name: hypothesis-convergence-50agents-5ms-10240b - domain: hypothesis-convergence - description: Generated consensus scenario for transport sweeps -spec: - agents: 50 - thinkTimeMs: 5 - findingEmitDelayMs: 1 - maxRounds: 200 - targetMode: majority - seed: 42 - valueSpace: 3 - payloadBytes: 10240 -agents: - - id: agent-0 - slimName: agntcy/bench-v2/agent-0 - a2aPort: 9700 - role: coordinator - - id: agent-1 - slimName: agntcy/bench-v2/agent-1 - a2aPort: 9711 - role: worker - - id: agent-2 - slimName: agntcy/bench-v2/agent-2 - a2aPort: 9722 - role: worker - - id: agent-3 - slimName: agntcy/bench-v2/agent-3 - a2aPort: 9733 - role: worker - - id: agent-4 - slimName: agntcy/bench-v2/agent-4 - a2aPort: 9744 - role: worker - - id: agent-5 - slimName: agntcy/bench-v2/agent-5 - a2aPort: 9755 - role: worker - - id: agent-6 - slimName: agntcy/bench-v2/agent-6 - a2aPort: 9766 - role: worker - - id: agent-7 - slimName: agntcy/bench-v2/agent-7 - a2aPort: 9777 - role: worker - - id: agent-8 - slimName: agntcy/bench-v2/agent-8 - a2aPort: 9788 - role: worker - - id: agent-9 - slimName: agntcy/bench-v2/agent-9 - a2aPort: 9799 - role: worker - - id: agent-10 - slimName: agntcy/bench-v2/agent-10 - a2aPort: 9810 - role: worker - - id: agent-11 - slimName: agntcy/bench-v2/agent-11 - a2aPort: 9821 - role: worker - - id: agent-12 - slimName: agntcy/bench-v2/agent-12 - a2aPort: 9832 - role: worker - - id: agent-13 - slimName: agntcy/bench-v2/agent-13 - a2aPort: 9843 - role: worker - - id: agent-14 - slimName: agntcy/bench-v2/agent-14 - a2aPort: 9854 - role: worker - - id: agent-15 - slimName: agntcy/bench-v2/agent-15 - a2aPort: 9865 - role: worker - - id: agent-16 - slimName: agntcy/bench-v2/agent-16 - a2aPort: 9876 - role: worker - - id: agent-17 - slimName: agntcy/bench-v2/agent-17 - a2aPort: 9887 - role: worker - - id: agent-18 - slimName: agntcy/bench-v2/agent-18 - a2aPort: 9898 - role: worker - - id: agent-19 - slimName: agntcy/bench-v2/agent-19 - a2aPort: 9909 - role: worker - - id: agent-20 - slimName: agntcy/bench-v2/agent-20 - a2aPort: 9920 - role: worker - - id: agent-21 - slimName: agntcy/bench-v2/agent-21 - a2aPort: 9931 - role: worker - - id: agent-22 - slimName: agntcy/bench-v2/agent-22 - a2aPort: 9942 - role: worker - - id: agent-23 - slimName: agntcy/bench-v2/agent-23 - a2aPort: 9953 - role: worker - - id: agent-24 - slimName: agntcy/bench-v2/agent-24 - a2aPort: 9964 - role: worker - - id: agent-25 - slimName: agntcy/bench-v2/agent-25 - a2aPort: 9975 - role: worker - - id: agent-26 - slimName: agntcy/bench-v2/agent-26 - a2aPort: 9986 - role: worker - - id: agent-27 - slimName: agntcy/bench-v2/agent-27 - a2aPort: 9997 - role: worker - - id: agent-28 - slimName: agntcy/bench-v2/agent-28 - a2aPort: 10008 - role: worker - - id: agent-29 - slimName: agntcy/bench-v2/agent-29 - a2aPort: 10019 - role: worker - - id: agent-30 - slimName: agntcy/bench-v2/agent-30 - a2aPort: 10030 - role: worker - - id: agent-31 - slimName: agntcy/bench-v2/agent-31 - a2aPort: 10041 - role: worker - - id: agent-32 - slimName: agntcy/bench-v2/agent-32 - a2aPort: 10052 - role: worker - - id: agent-33 - slimName: agntcy/bench-v2/agent-33 - a2aPort: 10063 - role: worker - - id: agent-34 - slimName: agntcy/bench-v2/agent-34 - a2aPort: 10074 - role: worker - - id: agent-35 - slimName: agntcy/bench-v2/agent-35 - a2aPort: 10085 - role: worker - - id: agent-36 - slimName: agntcy/bench-v2/agent-36 - a2aPort: 10096 - role: worker - - id: agent-37 - slimName: agntcy/bench-v2/agent-37 - a2aPort: 10107 - role: worker - - id: agent-38 - slimName: agntcy/bench-v2/agent-38 - a2aPort: 10118 - role: worker - - id: agent-39 - slimName: agntcy/bench-v2/agent-39 - a2aPort: 10129 - role: worker - - id: agent-40 - slimName: agntcy/bench-v2/agent-40 - a2aPort: 10140 - role: worker - - id: agent-41 - slimName: agntcy/bench-v2/agent-41 - a2aPort: 10151 - role: worker - - id: agent-42 - slimName: agntcy/bench-v2/agent-42 - a2aPort: 10162 - role: worker - - id: agent-43 - slimName: agntcy/bench-v2/agent-43 - a2aPort: 10173 - role: worker - - id: agent-44 - slimName: agntcy/bench-v2/agent-44 - a2aPort: 10184 - role: worker - - id: agent-45 - slimName: agntcy/bench-v2/agent-45 - a2aPort: 10195 - role: worker - - id: agent-46 - slimName: agntcy/bench-v2/agent-46 - a2aPort: 10206 - role: worker - - id: agent-47 - slimName: agntcy/bench-v2/agent-47 - a2aPort: 10217 - role: worker - - id: agent-48 - slimName: agntcy/bench-v2/agent-48 - a2aPort: 10228 - role: worker - - id: agent-49 - slimName: agntcy/bench-v2/agent-49 - a2aPort: 10239 - role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms.yaml deleted file mode 100644 index 2b377dfa..00000000 --- a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-50ag-5ms.yaml +++ /dev/null @@ -1,215 +0,0 @@ -apiVersion: bench.agntcy.io/v2 -kind: ConsensusScenario -metadata: - name: hypothesis-convergence-50agents-5ms - domain: hypothesis-convergence - description: Generated consensus scenario for transport sweeps -spec: - agents: 50 - thinkTimeMs: 5 - findingEmitDelayMs: 1 - maxRounds: 200 - targetMode: majority - seed: 42 - valueSpace: 3 -agents: - - id: agent-0 - slimName: agntcy/bench-v2/agent-0 - a2aPort: 9700 - role: coordinator - - id: agent-1 - slimName: agntcy/bench-v2/agent-1 - a2aPort: 9711 - role: worker - - id: agent-2 - slimName: agntcy/bench-v2/agent-2 - a2aPort: 9722 - role: worker - - id: agent-3 - slimName: agntcy/bench-v2/agent-3 - a2aPort: 9733 - role: worker - - id: agent-4 - slimName: agntcy/bench-v2/agent-4 - a2aPort: 9744 - role: worker - - id: agent-5 - slimName: agntcy/bench-v2/agent-5 - a2aPort: 9755 - role: worker - - id: agent-6 - slimName: agntcy/bench-v2/agent-6 - a2aPort: 9766 - role: worker - - id: agent-7 - slimName: agntcy/bench-v2/agent-7 - a2aPort: 9777 - role: worker - - id: agent-8 - slimName: agntcy/bench-v2/agent-8 - a2aPort: 9788 - role: worker - - id: agent-9 - slimName: agntcy/bench-v2/agent-9 - a2aPort: 9799 - role: worker - - id: agent-10 - slimName: agntcy/bench-v2/agent-10 - a2aPort: 9810 - role: worker - - id: agent-11 - slimName: agntcy/bench-v2/agent-11 - a2aPort: 9821 - role: worker - - id: agent-12 - slimName: agntcy/bench-v2/agent-12 - a2aPort: 9832 - role: worker - - id: agent-13 - slimName: agntcy/bench-v2/agent-13 - a2aPort: 9843 - role: worker - - id: agent-14 - slimName: agntcy/bench-v2/agent-14 - a2aPort: 9854 - role: worker - - id: agent-15 - slimName: agntcy/bench-v2/agent-15 - a2aPort: 9865 - role: worker - - id: agent-16 - slimName: agntcy/bench-v2/agent-16 - a2aPort: 9876 - role: worker - - id: agent-17 - slimName: agntcy/bench-v2/agent-17 - a2aPort: 9887 - role: worker - - id: agent-18 - slimName: agntcy/bench-v2/agent-18 - a2aPort: 9898 - role: worker - - id: agent-19 - slimName: agntcy/bench-v2/agent-19 - a2aPort: 9909 - role: worker - - id: agent-20 - slimName: agntcy/bench-v2/agent-20 - a2aPort: 9920 - role: worker - - id: agent-21 - slimName: agntcy/bench-v2/agent-21 - a2aPort: 9931 - role: worker - - id: agent-22 - slimName: agntcy/bench-v2/agent-22 - a2aPort: 9942 - role: worker - - id: agent-23 - slimName: agntcy/bench-v2/agent-23 - a2aPort: 9953 - role: worker - - id: agent-24 - slimName: agntcy/bench-v2/agent-24 - a2aPort: 9964 - role: worker - - id: agent-25 - slimName: agntcy/bench-v2/agent-25 - a2aPort: 9975 - role: worker - - id: agent-26 - slimName: agntcy/bench-v2/agent-26 - a2aPort: 9986 - role: worker - - id: agent-27 - slimName: agntcy/bench-v2/agent-27 - a2aPort: 9997 - role: worker - - id: agent-28 - slimName: agntcy/bench-v2/agent-28 - a2aPort: 10008 - role: worker - - id: agent-29 - slimName: agntcy/bench-v2/agent-29 - a2aPort: 10019 - role: worker - - id: agent-30 - slimName: agntcy/bench-v2/agent-30 - a2aPort: 10030 - role: worker - - id: agent-31 - slimName: agntcy/bench-v2/agent-31 - a2aPort: 10041 - role: worker - - id: agent-32 - slimName: agntcy/bench-v2/agent-32 - a2aPort: 10052 - role: worker - - id: agent-33 - slimName: agntcy/bench-v2/agent-33 - a2aPort: 10063 - role: worker - - id: agent-34 - slimName: agntcy/bench-v2/agent-34 - a2aPort: 10074 - role: worker - - id: agent-35 - slimName: agntcy/bench-v2/agent-35 - a2aPort: 10085 - role: worker - - id: agent-36 - slimName: agntcy/bench-v2/agent-36 - a2aPort: 10096 - role: worker - - id: agent-37 - slimName: agntcy/bench-v2/agent-37 - a2aPort: 10107 - role: worker - - id: agent-38 - slimName: agntcy/bench-v2/agent-38 - a2aPort: 10118 - role: worker - - id: agent-39 - slimName: agntcy/bench-v2/agent-39 - a2aPort: 10129 - role: worker - - id: agent-40 - slimName: agntcy/bench-v2/agent-40 - a2aPort: 10140 - role: worker - - id: agent-41 - slimName: agntcy/bench-v2/agent-41 - a2aPort: 10151 - role: worker - - id: agent-42 - slimName: agntcy/bench-v2/agent-42 - a2aPort: 10162 - role: worker - - id: agent-43 - slimName: agntcy/bench-v2/agent-43 - a2aPort: 10173 - role: worker - - id: agent-44 - slimName: agntcy/bench-v2/agent-44 - a2aPort: 10184 - role: worker - - id: agent-45 - slimName: agntcy/bench-v2/agent-45 - a2aPort: 10195 - role: worker - - id: agent-46 - slimName: agntcy/bench-v2/agent-46 - a2aPort: 10206 - role: worker - - id: agent-47 - slimName: agntcy/bench-v2/agent-47 - a2aPort: 10217 - role: worker - - id: agent-48 - slimName: agntcy/bench-v2/agent-48 - a2aPort: 10228 - role: worker - - id: agent-49 - slimName: agntcy/bench-v2/agent-49 - a2aPort: 10239 - role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-10ms.yaml similarity index 57% rename from benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms.yaml rename to benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-10ms.yaml index 7240d3f6..679bc25b 100644 --- a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-5ms.yaml +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-10ms.yaml @@ -1,17 +1,19 @@ apiVersion: bench.agntcy.io/v2 kind: ConsensusScenario metadata: - name: hypothesis-convergence-10agents-5ms + name: hypothesis-convergence-5agents-10ms domain: hypothesis-convergence description: Generated consensus scenario for transport sweeps spec: - agents: 10 - thinkTimeMs: 5 + agents: 5 + thinkTimeMs: 10 findingEmitDelayMs: 1 maxRounds: 200 targetMode: majority seed: 42 valueSpace: 3 + epochs: 10 + maxEpochTimeMs: 30000 agents: - id: agent-0 slimName: agntcy/bench-v2/agent-0 @@ -33,23 +35,3 @@ agents: slimName: agntcy/bench-v2/agent-4 a2aPort: 9744 role: worker - - id: agent-5 - slimName: agntcy/bench-v2/agent-5 - a2aPort: 9755 - role: worker - - id: agent-6 - slimName: agntcy/bench-v2/agent-6 - a2aPort: 9766 - role: worker - - id: agent-7 - slimName: agntcy/bench-v2/agent-7 - a2aPort: 9777 - role: worker - - id: agent-8 - slimName: agntcy/bench-v2/agent-8 - a2aPort: 9788 - role: worker - - id: agent-9 - slimName: agntcy/bench-v2/agent-9 - a2aPort: 9799 - role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms-5120b.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms-5120b.yaml new file mode 100644 index 00000000..512982df --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms-5120b.yaml @@ -0,0 +1,38 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-5agents-20ms-5120b + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 5 + thinkTimeMs: 20 + findingEmitDelayMs: 2 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 + payloadBytes: 5120 + epochs: 10 + maxEpochTimeMs: 30000 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml index e37985ae..25ed9840 100644 --- a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml @@ -12,6 +12,8 @@ spec: targetMode: majority seed: 42 valueSpace: 3 + epochs: 10 + maxEpochTimeMs: 30000 agents: - id: agent-0 slimName: agntcy/bench-v2/agent-0 diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-30ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-30ms.yaml new file mode 100644 index 00000000..31d25d43 --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-30ms.yaml @@ -0,0 +1,37 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-5agents-30ms + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 5 + thinkTimeMs: 30 + findingEmitDelayMs: 3 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 + epochs: 10 + maxEpochTimeMs: 30000 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker diff --git a/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go b/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go index ee7db5a7..3bd162bf 100644 --- a/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go +++ b/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go @@ -9,6 +9,7 @@ import ( "log" "time" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/agent" ) @@ -18,8 +19,13 @@ func main() { endpoint := flag.String("endpoint", "http://127.0.0.1:46357", "SLIM dataplane endpoint") scenarioFile := flag.String("scenario-file", "", "path to consensus scenario yaml") agentIndex := flag.Int("agent-index", 0, "agent index in scenario") + quiet := flag.Bool("quiet", false, "disable benchmark logs") flag.Parse() + if *quiet { + benchlog.SetEnabled(false) + } + if *slimName == "" || *scenarioFile == "" { log.Fatal("--slim-name and --scenario-file are required") } @@ -38,7 +44,9 @@ func main() { } defer rt.Close() - fmt.Printf("SLIM_AGENT_READY name=%s index=%d scenario=%s\n", *slimName, *agentIndex, s.Metadata.Name) + if !*quiet { + fmt.Printf("SLIM_AGENT_READY name=%s index=%d scenario=%s\n", *slimName, *agentIndex, s.Metadata.Name) + } // Block until the moderator invites us into the group session. if err := rt.Join(60 * time.Second); err != nil { diff --git a/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go b/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go index 42ca7e41..c31ff588 100644 --- a/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go +++ b/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go @@ -63,7 +63,7 @@ func main() { log.Fatal("set --agent-bin or SLIM_AGENT_BIN") } - procs := startAgents(s, agentPath, *endpoint, *scenarioPath) + procs := startAgents(s, agentPath, *endpoint, *scenarioPath, *quiet) defer stopAgents(procs) mod, err := newModerator(*endpoint, s) @@ -80,9 +80,6 @@ func main() { runStart := time.Now() benchlog.SetRunStart(runStart) - if err := mod.Start(); err != nil { - log.Fatalf("broadcast start: %v", err) - } result := metrics.RunResult{ ScenarioName: s.Metadata.Name, @@ -91,35 +88,63 @@ func main() { Agents: len(s.Agents), ThinkTimeMs: s.Spec.ThinkTimeMs, PayloadBytes: s.Spec.PayloadBytes, + Epochs: s.Spec.Epochs, } - latest := map[int]consensus.AgentSnapshot{} n := len(s.Agents) - deadline := time.Now().Add(2 * time.Minute) - for time.Now().Before(deadline) { - timeout := time.Second - msg, err := mod.session.GetMessageAsync(&timeout) - if err != nil { - if isTimeout(err) { - continue - } - break - } - env, derr := protocol.Decode(msg.Payload) - if derr != nil || env.Kind != protocol.KindSnapshot || env.Snapshot == nil { - continue + maxEpochTime := time.Duration(s.Spec.MaxEpochTimeMs) * time.Millisecond + var ( + wallSamples []int64 + allSnapshots []consensus.AgentSnapshot + ) + for epoch := 0; epoch < s.Spec.Epochs; epoch++ { + log.Printf("epoch %d/%d started (budget %s)", epoch+1, s.Spec.Epochs, maxEpochTime) + if err := mod.Start(epoch); err != nil { + log.Fatalf("broadcast start (epoch %d): %v", epoch, err) } - latest[env.Snapshot.AgentIndex] = *env.Snapshot - if len(latest) == n { - if ok, _ := consensus.GlobalConsensus(snapshotSlice(latest)); ok { - result.Success = true + epochStart := time.Now() + latest := map[int]consensus.AgentSnapshot{} + deadline := epochStart.Add(maxEpochTime) + success := false + for time.Now().Before(deadline) { + timeout := time.Second + msg, err := mod.session.GetMessageAsync(&timeout) + if err != nil { + if isTimeout(err) { + continue + } break } + env, derr := protocol.Decode(msg.Payload) + if derr != nil || env.Kind != protocol.KindSnapshot || env.Snapshot == nil { + continue + } + // Ignore stragglers from other epochs. + if env.Snapshot.Epoch != epoch { + continue + } + latest[env.Snapshot.AgentIndex] = *env.Snapshot + if len(latest) == n { + if ok, _ := consensus.GlobalConsensus(snapshotSlice(latest)); ok { + success = true + break + } + } + } + allSnapshots = append(allSnapshots, snapshotSlice(latest)...) + epochWall := time.Since(epochStart) + if success { + result.EpochsSucceeded++ + wallSamples = append(wallSamples, epochWall.Milliseconds()) + log.Printf("epoch %d/%d ok wall_ms=%d", epoch+1, s.Spec.Epochs, epochWall.Milliseconds()) + } else { + result.EpochsFailed++ + log.Printf("epoch %d/%d failed after %s (budget %s exhausted)", epoch+1, s.Spec.Epochs, epochWall.Round(time.Millisecond), maxEpochTime) } } - snapshots := snapshotSlice(latest) - result = aggregateResult(result, runStart, snapshots) + result.Success = result.EpochsFailed == 0 + result = aggregateResult(result, wallSamples, allSnapshots) stopAgents(procs) @@ -212,8 +237,8 @@ func (m *moderator) InviteAll() error { return nil } -func (m *moderator) Start() error { - payload, err := protocol.Encode(protocol.Envelope{Kind: protocol.KindStart}) +func (m *moderator) Start(epoch int) error { + payload, err := protocol.Encode(protocol.Envelope{Kind: protocol.KindStart, Epoch: epoch}) if err != nil { return err } @@ -229,22 +254,21 @@ func (m *moderator) Close() { } } -func aggregateResult(result metrics.RunResult, runStart time.Time, snapshots []consensus.AgentSnapshot) metrics.RunResult { +// aggregateResult folds the snapshots collected across every epoch into the run +// result. wallSamples holds the wall-clock duration of each successful epoch. +func aggregateResult(result metrics.RunResult, wallSamples []int64, snapshots []consensus.AgentSnapshot) metrics.RunResult { if len(snapshots) == 0 { result.Error = "no agent snapshots" - result.ConsensusWallMS = time.Since(runStart).Milliseconds() return result } var ( totalEmitted int totalApplied int - lastConvergeMS int64 propDurations []int64 maxConsensusRound int maxRound int ) - for _, snap := range snapshots { totalEmitted += snap.FindingsEmitted totalApplied += snap.FindingsApplied @@ -254,25 +278,16 @@ func aggregateResult(result metrics.RunResult, runStart time.Time, snapshots []c if snap.Round > maxRound { maxRound = snap.Round } - if snap.ConvergedAtNs > 0 { - ms := (snap.ConvergedAtNs - runStart.UnixNano()) / int64(time.Millisecond) - if ms > lastConvergeMS { - lastConvergeMS = ms - } - } if snap.AvgPropagationMs > 0 { propDurations = append(propDurations, snap.AvgPropagationMs) } } - if result.Success { - result.ConsensusWallMS = time.Since(runStart).Milliseconds() - if lastConvergeMS > 0 { - result.ConsensusWallMS = lastConvergeMS - } - } else { - result.ConsensusWallMS = time.Since(runStart).Milliseconds() - result.Error = "consensus not reached" + meanWall, maxWall := summarizeWall(wallSamples) + result.ConsensusWallMS = meanWall + result.LastAgentConvergeMS = maxWall + if result.EpochsSucceeded == 0 { + result.Error = "consensus not reached in any epoch" } result.ConsensusRound = maxConsensusRound @@ -281,7 +296,6 @@ func aggregateResult(result metrics.RunResult, runStart time.Time, snapshots []c } result.FindingsEmitted = totalEmitted result.FindingsReceivedTotal = totalApplied - result.LastAgentConvergeMS = lastConvergeMS result.AvgPropagationMS, result.P95PropagationMS = metrics.AggregatePropagation(propDurations) // Native group session: findings are broadcast peer-to-peer, no relay. result.StreamRPCCount = totalEmitted @@ -289,6 +303,21 @@ func aggregateResult(result metrics.RunResult, runStart time.Time, snapshots []c return result } +// summarizeWall returns the mean and max of the per-epoch wall-clock samples. +func summarizeWall(wallSamples []int64) (mean, max int64) { + if len(wallSamples) == 0 { + return 0, 0 + } + var sum int64 + for _, w := range wallSamples { + sum += w + if w > max { + max = w + } + } + return sum / int64(len(wallSamples)), max +} + func snapshotSlice(m map[int]consensus.AgentSnapshot) []consensus.AgentSnapshot { out := make([]consensus.AgentSnapshot, 0, len(m)) for _, snap := range m { @@ -297,16 +326,19 @@ func snapshotSlice(m map[int]consensus.AgentSnapshot) []consensus.AgentSnapshot return out } -func startAgents(s *scenario.ConsensusScenario, agentBin, endpoint, scenarioFile string) []*exec.Cmd { +func startAgents(s *scenario.ConsensusScenario, agentBin, endpoint, scenarioFile string, quiet bool) []*exec.Cmd { var procs []*exec.Cmd for i, agent := range s.Agents { - cmd := exec.Command( - agentBin, + args := []string{ "--slim-name", agent.SlimName, "--endpoint", endpoint, "--scenario-file", scenarioFile, "--agent-index", fmt.Sprintf("%d", i), - ) + } + if quiet { + args = append(args, "--quiet") + } + cmd := exec.Command(agentBin, args...) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr if err := cmd.Start(); err != nil { diff --git a/benchmarks/slim-vs-a2a/slim/internal/agent/runtime.go b/benchmarks/slim-vs-a2a/slim/internal/agent/runtime.go index 4cd82f06..4c53f4b3 100644 --- a/benchmarks/slim-vs-a2a/slim/internal/agent/runtime.go +++ b/benchmarks/slim-vs-a2a/slim/internal/agent/runtime.go @@ -13,7 +13,6 @@ import ( "fmt" "strings" "sync" - "sync/atomic" "time" slim "github.com/agntcy/slim-bindings-go" @@ -39,22 +38,26 @@ type Runtime struct { mu sync.Mutex runnerCtx *slim.MessageContext lastPush time.Time + // lastStartedEpoch dedupes repeated KindStart for the same epoch; + // currentEpoch is the attempt currently running (used to drop stale peer + // findings and to stop a superseded epoch loop). + lastStartedEpoch int + currentEpoch int pubMu sync.Mutex // serializes publishes on the session - running atomic.Bool - done chan struct{} startedAt time.Time } func NewRuntime(s *scenario.ConsensusScenario, agentIndex int, slimName, endpoint string) *Runtime { return &Runtime{ - scenario: s, - agentIndex: agentIndex, - slimName: slimName, - endpoint: endpoint, - engine: consensus.NewEngine(s.Spec, agentIndex), - done: make(chan struct{}), + scenario: s, + agentIndex: agentIndex, + slimName: slimName, + endpoint: endpoint, + engine: consensus.NewEngine(s.Spec, agentIndex), + lastStartedEpoch: -1, + currentEpoch: -1, } } @@ -118,14 +121,18 @@ func (r *Runtime) Run() error { switch env.Kind { case protocol.KindStart: ctx := msg.Context - r.mu.Lock() - r.runnerCtx = &ctx - r.mu.Unlock() - r.startRun() + r.startEpoch(env.Epoch, &ctx) case protocol.KindFinding: if env.Finding == nil || env.Finding.AgentIndex == r.agentIndex { continue } + // Drop stragglers from a different (usually previous) epoch. + r.mu.Lock() + cur := r.currentEpoch + r.mu.Unlock() + if env.Finding.Epoch != cur { + continue + } r.applyFinding(*env.Finding) // Keep the runner's view fresh if we already converged. r.maybePushSnapshot(false) @@ -133,22 +140,40 @@ func (r *Runtime) Run() error { } } -func (r *Runtime) startRun() { - if r.running.Swap(true) { +// startEpoch resets the engine for a fresh attempt and launches the epoch loop. +// Repeated starts for the same (or older) epoch are ignored so a duplicated +// broadcast cannot double-run an attempt. +func (r *Runtime) startEpoch(epoch int, ctx *slim.MessageContext) { + r.mu.Lock() + if epoch <= r.lastStartedEpoch { + r.mu.Unlock() return } + r.lastStartedEpoch = epoch + r.currentEpoch = epoch + r.runnerCtx = ctx + r.mu.Unlock() + + r.engine.Reset(epoch) r.startedAt = time.Now() benchlog.SetRunStart(r.startedAt) - go r.runLoop() + go r.runEpoch(epoch) } -func (r *Runtime) runLoop() { - defer close(r.done) +func (r *Runtime) runEpoch(epoch int) { spec := r.scenario.Spec think := time.Duration(spec.ThinkTimeMs) * time.Millisecond emitGap := time.Duration(spec.FindingEmitDelayMs) * time.Millisecond for round := 0; round < spec.MaxRounds; round++ { + // Stop early if a newer epoch has superseded this one. + r.mu.Lock() + superseded := r.currentEpoch != epoch + r.mu.Unlock() + if superseded { + return + } + time.Sleep(think) finding, emit := r.engine.Think() diff --git a/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go b/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go index ba4e3634..d6343700 100644 --- a/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go +++ b/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go @@ -26,8 +26,10 @@ const ( type Envelope struct { Kind string `json:"kind"` AgentIndex int `json:"agentIndex,omitempty"` - Finding *consensus.Finding `json:"finding,omitempty"` - Snapshot *consensus.AgentSnapshot `json:"snapshot,omitempty"` + // Epoch identifies which consensus attempt a KindStart begins. + Epoch int `json:"epoch,omitempty"` + Finding *consensus.Finding `json:"finding,omitempty"` + Snapshot *consensus.AgentSnapshot `json:"snapshot,omitempty"` } func Encode(e Envelope) ([]byte, error) { diff --git a/benchmarks/slim-vs-a2a/tools/gen_scenario/main.go b/benchmarks/slim-vs-a2a/tools/gen_scenario/main.go index db300374..38a4b2d9 100644 --- a/benchmarks/slim-vs-a2a/tools/gen_scenario/main.go +++ b/benchmarks/slim-vs-a2a/tools/gen_scenario/main.go @@ -19,6 +19,8 @@ func main() { thinkMS := flag.Int64("think-ms", 10, "think time per agent in ms") seed := flag.Int64("seed", 42, "random seed") payloadBytes := flag.Int("payload-bytes", 0, "fixed-size padding added to every finding (bytes)") + epochs := flag.Int("epochs", 0, "number of consensus epochs to run (default 10)") + maxEpochMS := flag.Int64("max-epoch-ms", 0, "per-epoch consensus time budget in ms (default 120000)") output := flag.String("output", "", "output yaml path") flag.Parse() @@ -27,11 +29,13 @@ func main() { } s, err := scenario.Generate(scenario.GenerateOptions{ - Family: *family, - Agents: *agents, - ThinkTimeMs: *thinkMS, - Seed: *seed, - PayloadBytes: *payloadBytes, + Family: *family, + Agents: *agents, + ThinkTimeMs: *thinkMS, + Seed: *seed, + PayloadBytes: *payloadBytes, + Epochs: *epochs, + MaxEpochTimeMs: *maxEpochMS, }) if err != nil { log.Fatalf("generate: %v", err) diff --git a/benchmarks/slim-vs-a2a/tools/report/main.go b/benchmarks/slim-vs-a2a/tools/report/main.go index 7b635ee6..6cac8e48 100644 --- a/benchmarks/slim-vs-a2a/tools/report/main.go +++ b/benchmarks/slim-vs-a2a/tools/report/main.go @@ -69,7 +69,7 @@ func readTSV(path string) ([]metrics.RunResult, error) { var out []metrics.RunResult for _, row := range records[1:] { - if len(row) < 17 { + if len(row) < 19 { continue } r := metrics.RunResult{ @@ -89,10 +89,12 @@ func readTSV(path string) ([]metrics.RunResult, error) { r.LastAgentConvergeMS = atoi64(row[12]) r.CoordFanoutMS = atoi64(row[13]) r.StreamRPCCount = atoi(row[14]) - r.UnicastRPCCount = atoi(row[15]) - r.Success = row[16] == "true" - if len(row) > 17 { - r.Error = row[17] + r.Epochs = atoi(row[15]) + r.EpochsSucceeded = atoi(row[16]) + r.EpochsFailed = atoi(row[17]) + r.Success = row[18] == "true" + if len(row) > 19 { + r.Error = row[19] } out = append(out, r) } @@ -206,18 +208,18 @@ const htmlTemplate = ` Avg propagation (ms){{if .HasA2A}}{{.A2A.AvgPropagationMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.AvgPropagationMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.AvgPropagationMS .SLIM.AvgPropagationMS}}{{else}}—{{end}} P95 propagation (ms){{if .HasA2A}}{{.A2A.P95PropagationMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.P95PropagationMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.P95PropagationMS .SLIM.P95PropagationMS}}{{else}}—{{end}} Stream RPC count{{if .HasA2A}}{{.A2A.StreamRPCCount}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.StreamRPCCount}}{{else}}—{{end}}— - Unicast RPC count{{if .HasA2A}}{{.A2A.UnicastRPCCount}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.UnicastRPCCount}}{{else}}—{{end}}— + Epochs (ok / failed){{if .HasA2A}}{{.A2A.EpochsSucceeded}} / {{.A2A.EpochsFailed}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.EpochsSucceeded}} / {{.SLIM.EpochsFailed}}{{else}}—{{end}}— Success{{if .HasA2A}}{{.A2A.Success}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.Success}}{{else}}—{{end}}— {{end}} {{if .Sweep}}

Sweep results

- + {{range .Sweep}} - + {{end}}
ScenarioImplAgentsThink msPayload BConsensus wallAvg propagationP95 propagationStream RPCs
ScenarioImplAgentsThink msPayload BConsensus wallAvg propagationP95 propagationStream RPCsEpochs ok/failed
{{.ScenarioName}}{{.Implementation}}{{.Agents}}{{.ThinkTimeMs}}{{.PayloadBytes}}{{.ConsensusWallMS}}{{.AvgPropagationMS}}{{.P95PropagationMS}}{{.StreamRPCCount}}{{.ConsensusWallMS}}{{.AvgPropagationMS}}{{.P95PropagationMS}}{{.StreamRPCCount}}{{.EpochsSucceeded}}/{{.EpochsFailed}}
@@ -239,8 +241,12 @@ const htmlTemplate = `
Number of finding-carrying messages on the data path. For SLIM this equals findings emitted (one native multicast each). For A2A it is the relay deliveries, ≈ findings × (N−1), because the hub re-sends every finding to all other agents.
-
Unicast RPC count
-
Control-plane unary RPCs only (start / snapshot polling), not the data path.
+
Epochs (ok / failed)
+
Each run repeats the same consensus attempt over several epochs. An epoch succeeds if every + agent reaches global consensus within spec.maxEpochTimeMs (each attempt runs up to + maxRounds); otherwise it is counted as failed. This surfaces reliability differences: under + load A2A may miss the per-epoch budget while SLIM still converges. Overall Success is true only + when no epoch failed.
Coord fanout (ms)
Cumulative time the A2A relay hub spent fanning findings out to peers. 0 for SLIM because there is no relay — the dataplane does the fan-out.