diff --git a/.cursor/plans/slim_native_vs_a2a_relay_b1828419.plan.md b/.cursor/plans/slim_native_vs_a2a_relay_b1828419.plan.md new file mode 100644 index 00000000..00322177 --- /dev/null +++ b/.cursor/plans/slim_native_vs_a2a_relay_b1828419.plan.md @@ -0,0 +1,122 @@ +--- +name: slim native vs a2a relay +overview: Rewrite the slim-vs-a2a benchmark so the SLIM side uses native SLIM group sessions (peer-to-peer multicast, no relay) and the A2A side uses real A2A server-streaming through the runner acting as the relay hub, to demonstrate that native SLIM needs no relay and yields better convergence metrics at scale. +todos: + - id: slim-protocol + content: Extend slim/internal/protocol with JSON Envelope (finding/start/snapshot, push-based); remove slim/internal/slimrpc and unused slim/internal/client + status: completed + - id: slim-agent + content: Rewrite slim agent runtime + cmd/agent/main.go to join the moderator's group session, run a single GetMessage receive loop, broadcast findings via PublishAndWaitAsync, and push a snapshot to the runner via PublishToAndWaitAsync immediately on local consensus (and on change) + status: completed + - id: slim-runner + content: "Rewrite slim cmd/runner/main.go as passive observer/moderator: create group session, invite agents, broadcast a single start, then event-driven receive loop consuming pushed snapshots (no polling), detect GlobalConsensus, aggregate (StreamRPCCount=findings, CoordFanoutMS=0); remove relay hub usage" + status: completed + - id: a2a-protocol + content: Update a2a/internal/protocol with OpStreamFindings/OpStreamRelay stream-init ops; keep unary OpStart/OpSnapshot; drop OpFinding/OpFanout + status: completed + - id: a2a-agent + content: "Rewrite a2a agent runtime + cmd/agent/main.go: host OpStreamFindings server stream + unary ops, subscribe to runner relay stream, push findings to outbound channel; remove coordinator/fanout" + status: completed + - id: a2a-runner + content: "Rewrite a2a cmd/runner/main.go + internal/client as relay: host OpStreamRelay server, subscribe to each agent's findings, relay+count (StreamRPCCount/CoordFanoutMS), drive start/snapshot, pass relay card URL to agents" + status: completed + - id: labels-report + content: Update benchlog impl tags (a2a-relay-stream, slim-group-session) and tools/report matching strings; note relay-free SLIM in plans/README + status: completed + - id: validate + content: task build; task compare:ci:smoke; run a 20/50-agent sweep point and confirm report renders and consensus_wall_ms delta favors SLIM + status: completed +isProject: false +--- + +## SLIM native sessions vs A2A streaming relay + +Goal: in-place rewrite of both transports in `benchmarks/slim-vs-a2a` to prove the thesis: native SLIM group sessions give true many-to-many with no relay; A2A requires a relay hub (the runner) even with real streaming; at N agents under a deadline, SLIM converges faster. The shared scenario/consensus/metrics/report core is reused; only the two transports and metric labels change. + +### Target topologies + +```mermaid +flowchart TB + subgraph slim [SLIM native group session - no relay] + R1["runner (passive observer: start signal + consumes pushed metrics)"] + SA0["agent-0"] + SA1["agent-1"] + SA2["agent-2"] + SA0 ---|"Publish finding -> all (dataplane fan-out)"| SA1 + SA1 --- SA2 + SA0 --- SA2 + SA0 -.->|"push snapshot on convergence"| R1 + SA1 -.-> R1 + SA2 -.-> R1 + end + subgraph a2a [A2A - runner is the relay hub] + R2["runner = RELAY (hosts A2A server)"] + AA0["agent-0"] + AA1["agent-1"] + AA2["agent-2"] + AA0 -->|"stream findings -> runner"| R2 + AA1 --> R2 + AA2 --> R2 + R2 -->|"stream relayed findings -> agent"| AA0 + R2 --> AA1 + R2 --> AA2 + end +``` + +### 1. SLIM side -> native group sessions (no relay) + +Delete the slimrpc relay path and rebuild on native sessions (APIs confirmed present in `slim-bindings-go v1.4.0`: `CreateSessionAndWaitAsync` + `SessionTypeGroup`, `InviteAndWaitAsync`, `ListenForSessionAsync`, `PublishAndWaitAsync`, `PublishToAndWaitAsync`, `GetMessageAsync`, `ParticipantsList`). Pattern reference: `/Users/smagyari/dev/slim/data-plane/bindings/go/examples/group/main.go`. + +- Remove `[slim/internal/relay/hub.go](benchmarks/slim-vs-a2a/slim/internal/relay/hub.go)`, `[slim/internal/slimrpc/slimrpc.go](benchmarks/slim-vs-a2a/slim/internal/slimrpc/slimrpc.go)`, and the unused `[slim/internal/client/client.go](benchmarks/slim-vs-a2a/slim/internal/client/client.go)`. +- Extend `[slim/internal/protocol/protocol.go](benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go)` with a single JSON envelope sent over the session: + - `Envelope{ Kind string; Finding *consensus.Finding; Snapshot *consensus.AgentSnapshot; AgentIndex int }` with `Kind` in `{"finding","start","snapshot"}`. No request/reply kinds — snapshots are pushed, not polled. +- Rewrite `[slim/internal/agent/runtime.go](benchmarks/slim-vs-a2a/slim/internal/agent/runtime.go)` (member): + - Setup: create app `agntcy/bench-v2/agent-N`, connect, subscribe, then `ListenForSessionAsync(timeout)` to join the moderator's group session; keep `*slim.Session`. + - Single receive loop on `session.GetMessageAsync`: dispatch by `Kind`: + - `finding` -> `applyFinding` (skip own `AgentIndex`, `RecordPropagation`). + - `start` -> save the message's `Context` (identifies the runner) for targeted snapshot pushes, then launch `runLoop` (unchanged Think/emit logic). + - `publishFinding` -> `session.PublishAndWaitAsync(envelope{Kind:"finding"})` (broadcast to all peers; no relay). + - Push metrics immediately (no polling): the agent emits a `snapshot` envelope via `PublishToAndWaitAsync(savedRunnerCtx, snapshot)` the instant it reaches local consensus, and again on any later metric change (throttled) and at run end, so the runner always has the latest. Targeting the runner keeps snapshots off the peer broadcast path. +- Rewrite `[slim/cmd/agent/main.go](benchmarks/slim-vs-a2a/slim/cmd/agent/main.go)`: drop `ServerNewWithConnection`/`Register`; just setup runtime, join session, block on the receive loop. Keep `SLIM_AGENT_READY` print. Flags unchanged plus optional `--channel` (default `agntcy/bench-v2/consensus`). +- Rewrite `[slim/cmd/runner/main.go](benchmarks/slim-vs-a2a/slim/cmd/runner/main.go)` (passive observer / moderator): + - `startAgents` (unchanged spawn), then create the group session as moderator and `InviteAndWaitAsync` every agent name; small settle. + - Broadcast a single `start` (defines t0); after that the runner does NOT poll. It runs one receive loop on `GetMessageAsync`, keeping a `map[agentIndex]AgentSnapshot` of the latest pushed snapshot per agent. It ignores `finding` broadcasts it incidentally receives (it never forwards them — agents already got them directly, so there is no relay). + - Global-consensus detection is event-driven: after each pushed snapshot, run `consensus.GlobalConsensus` over the N latest snapshots; on success record wall time and finish. A 2-min deadline timer bounds the loop for the failure case. + - `aggregateResult`: same fields, computed from the collected pushed snapshots; set `StreamRPCCount = FindingsEmitted` (one native multicast per finding), `CoordFanoutMS = 0`, `UnicastRPCCount = 0` to highlight "no relay". + - Note on observation: the runner sees a copy of broadcast findings only because it is a group member, not because it relays them; agent-to-agent delivery happens via the SLIM dataplane fan-out independent of the runner. + +### 2. A2A side -> runner as relay with real streaming + +Make the runner the relay hub; remove the `agent-0` coordinator/fanout logic. Both legs are client-initiated A2A server-streams (`SendStreamingMessage`/executor `iter.Seq2[a2a.Event,error]`, present in `a2a-go v2.3.0`). + +- `[a2a/internal/protocol/protocol.go](benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go)`: add stream-init ops `OpStreamFindings` (runner subscribes to an agent's outbound findings) and `OpStreamRelay` (agent subscribes to runner's relayed findings); keep unary `OpStart`/`OpSnapshot`; drop `OpFinding`/`OpFanout`. +- Rewrite `[a2a/internal/agent/runtime.go](benchmarks/slim-vs-a2a/a2a/internal/agent/runtime.go)`: + - Remove `isCoordinator`, `fanout`, per-peer clients. Agent dials only the runner relay (card URL passed in). + - Hosts an A2A server: streaming `OpStreamFindings` executor drains an internal `findingsCh` and yields each as an `a2a.Event` until ctx done; unary `OpStart` (start `runLoop`) and `OpSnapshot` (return `AgentSnapshot`). + - On start, agent (as client) opens `SendStreamingMessage(OpStreamRelay)` to the runner and applies every relayed finding (`applyFinding`, skip self, `RecordPropagation`). + - `runLoop` pushes produced findings onto `findingsCh` (the publish leg) instead of unary send. +- Rewrite `[a2a/cmd/agent/main.go](benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go)`: executor routes streaming vs unary ops; add `--relay-card-url` (runner's card) flag. +- Rewrite `[a2a/cmd/runner/main.go](benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go)` + `[a2a/internal/client/client.go](benchmarks/slim-vs-a2a/a2a/internal/client/client.go)` (relay): + - Runner hosts its own A2A server + card (new `--relay-grpc-port`/`--relay-card-port`, default e.g. 9690/10690) implementing streaming `OpStreamRelay`: registers each subscribing agent's outbound channel. + - Runner (as client) subscribes via `SendStreamingMessage(OpStreamFindings)` to every agent; on each received finding, push into all agents' relay channels (skip none; agents skip self), incrementing `StreamRPCCount` and measuring `CoordFanoutMS`. + - Pass the runner relay card URL to each spawned agent. Keep unary `OpStart`/`OpSnapshot` to drive/poll agents and `GlobalConsensus`. + - `aggregateResult`: real `StreamRPCCount` (relayed events) and measured `CoordFanoutMS`; `UnicastRPCCount` = control RPCs only. + +### 3. Labels, report, Taskfile + +- Update impl tags in `[internal/benchlog/benchlog.go](benchmarks/slim-vs-a2a/internal/benchlog/benchlog.go)`: `ImplA2A = "a2a-relay-stream"`, `ImplSLIM = "slim-group-session"`; update the matching strings in `[tools/report/main.go](benchmarks/slim-vs-a2a/tools/report/main.go)` `groupByScenario`/labels. +- `[internal/metrics/metrics.go](benchmarks/slim-vs-a2a/internal/metrics/metrics.go)` struct/columns unchanged (semantics adjusted per above). +- `[Taskfile.yml](benchmarks/slim-vs-a2a/Taskfile.yml)`: `build` targets and slimctl dataplane block unchanged; SLIM runner now needs no extra ports; A2A runner gains relay-port flags (defaulted, so `compare:*` need minimal/no change). Add a one-line note to `[plans/README.md](benchmarks/slim-vs-a2a/plans/README.md)` that SLIM is relay-free and A2A relays via the runner. + +### 4. Validate + +- `task build`; `task compare:ci:smoke` (5 agents/20ms) to confirm both stacks converge and `reports/index.html` renders with the new tags. +- Run a sweep point at higher agent counts (e.g. 20 and 50 via existing `plans/sweeps/*`) to capture the `consensus_wall_ms` delta favoring SLIM, with both stacks under the same 2-min deadline. + +### Risks / notes + +- A2A long-lived streaming executor must yield events over time from a channel; if `a2a-go`'s task/streaming lifecycle fights a never-ending stream, fall back to: receive leg stays real streaming (agents subscribe to runner), publish leg becomes unary `SendMessage(OpFinding)` to the runner. Thesis (relay required) holds either way. +- SLIM session self-delivery: keep the existing self-skip by `AgentIndex` guard. +- Push-based snapshots: agents send their metric the instant they converge (no poll wait), so consensus is detected as soon as the last agent's snapshot arrives. The agent's converged snapshot already contains complete findings/propagation stats (it converged by receiving them); a throttled follow-up push plus an end-of-run push cover any late `findingsApplied` drift. Keep MLS off by default. +- Targeted snapshot push relies on the saved `start` message `Context` to address the runner; if reusing a stored context misbehaves, fall back to broadcasting `snapshot` envelopes (peers ignore non-self snapshots). +- Two CGO/native rewrites: build offline after `deps:slim-bindings-setup`; no new external deps. \ No newline at end of file diff --git a/.github/actions/publish-test-reports-pages/action.yaml b/.github/actions/publish-test-reports-pages/action.yaml index b40e2a18..86308cb1 100644 --- a/.github/actions/publish-test-reports-pages/action.yaml +++ b/.github/actions/publish-test-reports-pages/action.yaml @@ -8,7 +8,7 @@ description: Download suite artifacts, update gh-pages report dirs and docs/inde inputs: suite: - description: "Suite identifier (a2a, a2a-slimrpc, slim-integration, slim-benchmarks, slim-multicluster-private, directory-conformance)" + description: "Suite identifier (a2a, a2a-slimrpc, slim-integration, slim-benchmarks, slim-vs-a2a, slim-multicluster-private, directory-conformance)" required: true source-run-id: description: "Workflow run ID containing uploaded artifacts" @@ -70,6 +70,10 @@ runs: ./.github/scripts/download-workflow-artifact.sh "$RUN_ID" .pages-staging/basic \ --name slim-benchmark-basic-report ;; + slim-vs-a2a) + ./.github/scripts/download-workflow-artifact.sh "$RUN_ID" .pages-staging \ + --name slim-vs-a2a-report + ;; slim-multicluster-private) ./.github/scripts/download-workflow-artifact.sh "$RUN_ID" .pages-staging \ --name slim-multicluster-private-test-result @@ -111,6 +115,7 @@ runs: a2a-slimrpc) WORKFLOW_ID=test-a2a-slimrpc ;; slim-integration) WORKFLOW_ID=test-slim-integration ;; slim-benchmarks) WORKFLOW_ID=test-slim-benchmarks ;; + slim-vs-a2a) WORKFLOW_ID=test-slim-vs-a2a ;; slim-multicluster-private) WORKFLOW_ID=test-slim-multicluster-private ;; directory-conformance) WORKFLOW_ID=test-directory-conformance ;; *) echo "unknown suite: $SUITE" >&2; exit 1 ;; diff --git a/.github/scripts/build-pages-section.sh b/.github/scripts/build-pages-section.sh index 74c37f3e..fa2428dd 100755 --- a/.github/scripts/build-pages-section.sh +++ b/.github/scripts/build-pages-section.sh @@ -5,7 +5,7 @@ # Build one gh-pages docs section from a staging directory populated by download-artifact. # # Required env: -# SUITE a2a | a2a-slimrpc | slim-integration | slim-benchmarks | slim-multicluster-private | directory-conformance +# SUITE a2a | a2a-slimrpc | slim-integration | slim-benchmarks | slim-vs-a2a | slim-multicluster-private | directory-conformance # STAGING_DIR directory with downloaded artifact contents # SITE_DIR gh-pages docs root (e.g. site/docs; suite dirs a2a/, directory/, …) # REPO_ROOT csit checkout (for go run) @@ -99,6 +99,16 @@ case "$SUITE" in fi ;; + slim-vs-a2a) + # compare:report already produced a self-contained index.html (plus TSVs); + # publish it verbatim under benchmarks/slim-vs-a2a/. + if [[ -f "$STAGING_DIR/index.html" ]]; then + mkdir -p "$SITE_DIR/benchmarks/slim-vs-a2a" + cp -a "$STAGING_DIR/." "$SITE_DIR/benchmarks/slim-vs-a2a/" + has_report=true + fi + ;; + slim-multicluster-private) summary="" if [[ -f "$STAGING_DIR/summary.html" ]]; then diff --git a/.github/scripts/build-site-landing-page.sh b/.github/scripts/build-site-landing-page.sh index 6350f55a..7e81978d 100755 --- a/.github/scripts/build-site-landing-page.sh +++ b/.github/scripts/build-site-landing-page.sh @@ -261,6 +261,9 @@ HTML test-slim-benchmarks) blurb="Throughput and latency benchmark dashboards across modes, payload sizes, and sender counts." ;; + test-slim-vs-a2a) + blurb="SLIM native group-session multicast vs A2A relay-hub streaming: consensus latency, propagation, and RPC counts." + ;; test-slim-multicluster-private) blurb="Two-cluster SPIRE federation verification with private cluster B constraints." ;; diff --git a/.github/scripts/init-sources-json.sh b/.github/scripts/init-sources-json.sh index d1b5cf4d..36ac1ece 100755 --- a/.github/scripts/init-sources-json.sh +++ b/.github/scripts/init-sources-json.sh @@ -64,6 +64,17 @@ cat > "$defaults" <<'JSON' "published_report_run_id": "", "published_report_updated_at": "" }, + { + "id": "test-slim-vs-a2a", + "name": "SLIM vs A2A consensus", + "workflow_file": "test-slim-vs-a2a.yaml", + "report_path": "benchmarks/slim-vs-a2a/", + "last_run_id": "", + "last_run_conclusion": "", + "last_run_updated_at": "", + "published_report_run_id": "", + "published_report_updated_at": "" + }, { "id": "test-slim-multicluster-private", "name": "Slim multicluster private", diff --git a/.github/workflows/test-slim-vs-a2a.yaml b/.github/workflows/test-slim-vs-a2a.yaml new file mode 100644 index 00000000..b6891007 --- /dev/null +++ b/.github/workflows/test-slim-vs-a2a.yaml @@ -0,0 +1,99 @@ +# Copyright AGNTCY Contributors (https://github.com/agntcy) +# SPDX-License-Identifier: Apache-2.0 + +name: test-slim-vs-a2a + +on: + push: + branches: + - main + paths: + - '.github/workflows/test-slim-vs-a2a.yaml' + - 'benchmarks/slim-vs-a2a/**' + - 'benchmarks/Taskfile.yml' + pull_request: + paths: + - '.github/workflows/test-slim-vs-a2a.yaml' + - 'benchmarks/slim-vs-a2a/**' + - 'benchmarks/Taskfile.yml' + workflow_dispatch: + +jobs: + slim-vs-a2a: + runs-on: ubuntu-latest + timeout-minutes: 30 + + permissions: + contents: read + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Environment + uses: ./.github/actions/setup-env + with: + go: true + + - name: Run SLIM vs A2A comparison suite + shell: bash + working-directory: benchmarks/slim-vs-a2a + run: | + set -o pipefail + task compare:suite + + - name: Build comparison report + if: always() + shell: bash + working-directory: benchmarks/slim-vs-a2a + run: task compare:report + + - name: Publish step summary + if: always() + shell: bash + run: | + REPORT_DIR="benchmarks/slim-vs-a2a/reports" + { + echo "# SLIM vs A2A — Consensus Comparison" + echo + echo "- Workflow: ${GITHUB_WORKFLOW}" + echo "- Ref: ${GITHUB_REF}" + echo "- SHA: ${GITHUB_SHA}" + echo + if [ -f "$REPORT_DIR/results.tsv" ]; then + echo "## results.tsv" + echo + echo '```' + cat "$REPORT_DIR/results.tsv" + echo '```' + fi + } >> "$GITHUB_STEP_SUMMARY" + + - name: Upload comparison report + if: always() + uses: actions/upload-artifact@v4 + with: + name: slim-vs-a2a-report + path: benchmarks/slim-vs-a2a/reports/ + retention-days: 30 + if-no-files-found: warn + + publish-pages: + needs: [slim-vs-a2a] + if: always() && github.ref == 'refs/heads/main' && needs.slim-vs-a2a.result != 'skipped' + runs-on: ubuntu-latest + timeout-minutes: 10 + concurrency: + group: test-report-pages + cancel-in-progress: false + permissions: + contents: write + actions: read + steps: + - uses: actions/checkout@v4 + + - uses: ./.github/actions/publish-test-reports-pages + with: + suite: slim-vs-a2a + source-run-id: ${{ github.run_id }} + source-run-conclusion: ${{ needs.slim-vs-a2a.result }} diff --git a/benchmarks/Taskfile.yml b/benchmarks/Taskfile.yml index 3f65d070..faee6796 100644 --- a/benchmarks/Taskfile.yml +++ b/benchmarks/Taskfile.yml @@ -17,6 +17,11 @@ includes: dir: ./agntcy-dir excludes: [default] + slim-vs-a2a: + taskfile: ./slim-vs-a2a/Taskfile.yml + dir: ./slim-vs-a2a + excludes: [default] + tasks: default: cmd: task -l \ No newline at end of file diff --git a/benchmarks/slim-vs-a2a/Taskfile.yml b/benchmarks/slim-vs-a2a/Taskfile.yml new file mode 100644 index 00000000..388bb864 --- /dev/null +++ b/benchmarks/slim-vs-a2a/Taskfile.yml @@ -0,0 +1,409 @@ +# Copyright AGNTCY Contributors (https://github.com/agntcy) +# SPDX-License-Identifier: Apache-2.0 + +--- +version: '3' + +silent: true + +vars: + SCENARIO_DIR: '{{ .SCENARIO_DIR | default "./plans/sweeps" }}' + REPORTS_DIR: '{{ .REPORTS_DIR | default "./reports" }}' + RESULTS_TSV: '{{ .RESULTS_TSV | default "./reports/results.tsv" }}' + SWEEP_TSV: '{{ .SWEEP_TSV | default "./reports/sweep.tsv" }}' + BIN_DIR: '{{ .BIN_DIR | default "./bin" }}' + SLIM_ENDPOINT: '{{ .SLIM_ENDPOINT | default "http://127.0.0.1:46357" }}' + COMPARE_SLIMCTL: '{{ .COMPARE_SLIMCTL | default "../agntcy-slim/bin/slimctl" }}' + COMPARE_SLIM_BINDINGS_VERSION: '{{ .COMPARE_SLIM_BINDINGS_VERSION | default "v1.4.0" }}' + COMPARE_SLIMCTL_TAG: '{{ .COMPARE_SLIMCTL_TAG | default "slimctl-v1.4.0" }}' + +tasks: + default: + cmd: task -l + + deps:slim-bindings-setup: + desc: Install native SLIM bindings required by slim agents + cmds: + - go run github.com/agntcy/slim-bindings-go/cmd/slim-bindings-setup@{{ .COMPARE_SLIM_BINDINGS_VERSION }} + + deps:slimctl-download: + desc: Download slimctl for local SLIM stack startup + cmds: + - | + set -e + SLIMCTL_ARCH=$(arch) + SLIMCTL_OS=$(uname -s | tr '[:upper:]' '[:lower:]') + SLIMCTL_ABI="" + if [ "$SLIMCTL_ARCH" = "x86_64" ]; then + SLIMCTL_ARCH="amd64" + fi + if [ "$SLIMCTL_OS" = "linux" ]; then + if ldd --version 2>&1 | grep -qi musl; then + SLIMCTL_ABI="-musl" + else + SLIMCTL_ABI="-gnu" + fi + fi + SLIMCTL_URL="https://github.com/agntcy/slim/releases/download/{{ .COMPARE_SLIMCTL_TAG }}/slimctl-$SLIMCTL_OS-$SLIMCTL_ARCH${SLIMCTL_ABI}.tar.gz" + TARGET_DIR=$(dirname {{ .COMPARE_SLIMCTL }}) + mkdir -p "$TARGET_DIR" + TMP_DIR=$(mktemp -d) + trap 'rm -rf "$TMP_DIR"' EXIT + curl --fail --show-error --location -o "$TMP_DIR/slimctl.tar.gz" "$SLIMCTL_URL" + tar -xzf "$TMP_DIR/slimctl.tar.gz" -C "$TMP_DIR" + mv "$TMP_DIR/slimctl" {{ .COMPARE_SLIMCTL }} + chmod +x {{ .COMPARE_SLIMCTL }} + {{ .COMPARE_SLIMCTL }} version || true + + build: + desc: Build a2a and slim agent/runner binaries + deps: + - deps:slim-bindings-setup + cmds: + - mkdir -p {{ .BIN_DIR }} + - go build -o {{ .BIN_DIR }}/a2a-agent ./a2a/cmd/agent + - go build -o {{ .BIN_DIR }}/a2a-runner ./a2a/cmd/runner + - | + CACHE_DIR="$(go env GOPATH)/.cgo-cache/slim-bindings/{{ .COMPARE_SLIM_BINDINGS_VERSION }}" + export CGO_LDFLAGS="-L${CACHE_DIR}" + go build -o {{ .BIN_DIR }}/slim-agent ./slim/cmd/agent + go build -o {{ .BIN_DIR }}/slim-runner ./slim/cmd/runner + + validate:scenarios: + desc: Validate consensus scenario YAML files + cmds: + - go run ./tools/validate_scenarios --dir {{ .SCENARIO_DIR }} + + compare:cleanup: + internal: true + cmds: + - pkill -f '{{ .BIN_DIR }}/a2a-agent' 2>/dev/null || true + - pkill -f '{{ .BIN_DIR }}/slim-agent' 2>/dev/null || true + - pkill -f 'slimctl slim start -c' 2>/dev/null || true + + compare:suite: + desc: Run default consensus scenarios on A2A and SLIM + deps: + - build + - validate:scenarios + cmds: + - task: compare:cleanup + - rm -f {{ .RESULTS_TSV }} + - | + set -euo pipefail + for scenario in {{ .SCENARIO_DIR }}/hypothesis-convergence-*.yaml; do + [ -f "$scenario" ] || continue + base=$(basename "$scenario" .yaml) + echo "==> A2A $base" + {{ .BIN_DIR }}/a2a-runner \ + --scenario "$scenario" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ + --output-tsv {{ .RESULTS_TSV }} || true + done + - task: compare:slim-stack + vars: + SCENARIO_GLOB: '{{ .SCENARIO_DIR }}/hypothesis-convergence-*.yaml' + + compare:plan: + desc: Run one scenario on A2A and SLIM (PLAN=path or basename) + deps: + - build + requires: + vars: [PLAN] + cmds: + - task: compare:cleanup + - rm -f {{ .RESULTS_TSV }} + - | + set -euo pipefail + PLAN_INPUT="{{ .PLAN }}" + if [ -f "$PLAN_INPUT" ]; then + SCENARIO_PATH="$PLAN_INPUT" + elif [ -f "{{ .SCENARIO_DIR }}/${PLAN_INPUT}.yaml" ]; then + SCENARIO_PATH="{{ .SCENARIO_DIR }}/${PLAN_INPUT}.yaml" + else + echo "scenario not found: ${PLAN_INPUT}" >&2 + exit 1 + fi + base=$(basename "$SCENARIO_PATH" .yaml) + echo "==> A2A ${base}" + {{ .BIN_DIR }}/a2a-runner \ + --scenario "$SCENARIO_PATH" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ + --output-tsv {{ .RESULTS_TSV }} + - task: compare:slim-stack + vars: + SCENARIO_GLOB: '{{ .PLAN }}' + + compare:slim-stack: + internal: true + deps: + - deps:slimctl-download + cmds: + - | + set -euo pipefail + SCENARIO_INPUT="{{ .SCENARIO_GLOB }}" + resolve_scenario() { + local input="$1" + if [ -f "$input" ]; then + printf '%s' "$input" + return + fi + if [ -f "{{ .SCENARIO_DIR }}/${input}.yaml" ]; then + printf '%s' "{{ .SCENARIO_DIR }}/${input}.yaml" + return + fi + printf '%s' "$input" + } + DATAPLANE_PORT=46357 + CONTROLLER_PORT=46358 + CONFIG=$(mktemp) + trap 'kill ${SLIM_PID:-} 2>/dev/null || true; rm -f "$CONFIG"; {{ .COMPARE_SLIMCTL }} slim stop 2>/dev/null || true' EXIT + cat > "$CONFIG" <>{{ .REPORTS_DIR }}/slimctl.log 2>&1 & + SLIM_PID=$! + for i in $(seq 1 50); do + if nc -z 127.0.0.1 "${DATAPLANE_PORT}" 2>/dev/null; then + break + fi + sleep 0.2 + done + if [[ "$SCENARIO_INPUT" == *"*"* ]]; then + SCENARIO_LIST=($SCENARIO_INPUT) + else + SCENARIO_LIST=("$(resolve_scenario "$SCENARIO_INPUT")") + fi + for scenario in "${SCENARIO_LIST[@]}"; do + [ -f "$scenario" ] || continue + base=$(basename "$scenario" .yaml) + echo "==> SLIM ${base}" + {{ .BIN_DIR }}/slim-runner \ + --scenario "$scenario" \ + --endpoint {{ .SLIM_ENDPOINT }} \ + --agent-bin {{ .BIN_DIR }}/slim-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/slim-${base}.json \ + --output-tsv {{ .RESULTS_TSV }} || true + done + + compare:report: + desc: Build HTML dashboard from comparison TSV results + cmds: + - go run ./tools/report --tsv {{ .RESULTS_TSV }} --sweep-tsv {{ .SWEEP_TSV }} --output {{ .REPORTS_DIR }}/index.html + + compare:sweep: + desc: Run agent/think-time/payload sweep (SWEEP_AGENTS, SWEEP_THINK_MS, SWEEP_PAYLOAD_BYTES env lists) + deps: + - build + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES | default "0" }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS | default "10" }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS | default "30000" }}' + cmds: + - task: compare:cleanup + - rm -f {{ .SWEEP_TSV }} + - task: compare:sweep:run + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS }}' + + compare:sweep:payload: + desc: High-N / short-window / large-payload sweep highlighting SLIM multicast (10kB default) + deps: + - build + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "10,50" }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "5" }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES | default "0,10240" }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS | default "10" }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS | default "30000" }}' + cmds: + - task: compare:cleanup + - rm -f {{ .SWEEP_TSV }} + - task: compare:sweep:run + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS }}' + - task: compare:report + + compare:sweep:run: + internal: true + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES | default "0" }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS | default "10" }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS | default "30000" }}' + cmds: + - | + set -euo pipefail + mkdir -p {{ .REPORTS_DIR }} {{ .SCENARIO_DIR }} + IFS=',' read -r -a AGENT_LIST <<< "{{ .SWEEP_AGENTS }}" + IFS=',' read -r -a THINK_LIST <<< "{{ .SWEEP_THINK_MS }}" + IFS=',' read -r -a PAYLOAD_LIST <<< "{{ .SWEEP_PAYLOAD_BYTES }}" + for agents in "${AGENT_LIST[@]}"; do + for think in "${THINK_LIST[@]}"; do + for payload in "${PAYLOAD_LIST[@]}"; do + suffix="" + if [ "$payload" != "0" ]; then suffix="-${payload}b"; fi + scenario_path="{{ .SCENARIO_DIR }}/{{ .SWEEP_FAMILY }}-${agents}ag-${think}ms${suffix}.yaml" + go run ./tools/gen_scenario \ + -family {{ .SWEEP_FAMILY }} \ + -agents "$agents" \ + -think-ms "$think" \ + -payload-bytes "$payload" \ + -epochs {{ .SWEEP_EPOCHS }} \ + -max-epoch-ms {{ .SWEEP_MAX_EPOCH_MS }} \ + -output "$scenario_path" + base=$(basename "$scenario_path" .yaml) + echo "==> sweep A2A $base" + {{ .BIN_DIR }}/a2a-runner \ + --scenario "$scenario_path" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/a2a-${base}.json \ + --output-tsv {{ .SWEEP_TSV }} || true + done + done + done + - task: compare:sweep:slim + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS }}' + + compare:sweep:slim: + internal: true + deps: + - deps:slimctl-download + vars: + SWEEP_FAMILY: '{{ .SWEEP_FAMILY | default "hypothesis-convergence" }}' + SWEEP_AGENTS: '{{ .SWEEP_AGENTS | default "2,5,10,17" }}' + SWEEP_THINK_MS: '{{ .SWEEP_THINK_MS | default "10,20,50" }}' + SWEEP_PAYLOAD_BYTES: '{{ .SWEEP_PAYLOAD_BYTES | default "0" }}' + SWEEP_EPOCHS: '{{ .SWEEP_EPOCHS | default "10" }}' + SWEEP_MAX_EPOCH_MS: '{{ .SWEEP_MAX_EPOCH_MS | default "30000" }}' + cmds: + - | + set -euo pipefail + IFS=',' read -r -a AGENT_LIST <<< "{{ .SWEEP_AGENTS }}" + IFS=',' read -r -a THINK_LIST <<< "{{ .SWEEP_THINK_MS }}" + IFS=',' read -r -a PAYLOAD_LIST <<< "{{ .SWEEP_PAYLOAD_BYTES }}" + DATAPLANE_PORT=46357 + CONTROLLER_PORT=46358 + CONFIG=$(mktemp) + trap 'kill ${SLIM_PID:-} 2>/dev/null || true; rm -f "$CONFIG"; {{ .COMPARE_SLIMCTL }} slim stop 2>/dev/null || true' EXIT + cat > "$CONFIG" <>{{ .REPORTS_DIR }}/slimctl.log 2>&1 & + SLIM_PID=$! + for i in $(seq 1 50); do + if nc -z 127.0.0.1 "${DATAPLANE_PORT}" 2>/dev/null; then + break + fi + sleep 0.2 + done + for agents in "${AGENT_LIST[@]}"; do + for think in "${THINK_LIST[@]}"; do + for payload in "${PAYLOAD_LIST[@]}"; do + suffix="" + if [ "$payload" != "0" ]; then suffix="-${payload}b"; fi + scenario_path="{{ .SCENARIO_DIR }}/{{ .SWEEP_FAMILY }}-${agents}ag-${think}ms${suffix}.yaml" + [ -f "$scenario_path" ] || continue + base=$(basename "$scenario_path" .yaml) + echo "==> sweep SLIM $base" + {{ .BIN_DIR }}/slim-runner \ + --scenario "$scenario_path" \ + --endpoint {{ .SLIM_ENDPOINT }} \ + --agent-bin {{ .BIN_DIR }}/slim-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/slim-${base}.json \ + --output-tsv {{ .SWEEP_TSV }} || true + done + done + done + + compare:ci:smoke: + desc: CI smoke — one sweep point plus report (~2 min) + deps: + - build + cmds: + - task: compare:cleanup + - rm -f {{ .RESULTS_TSV }} {{ .SWEEP_TSV }} + - | + set -euo pipefail + mkdir -p {{ .SCENARIO_DIR }} {{ .REPORTS_DIR }} + scenario_path="{{ .SCENARIO_DIR }}/ci-smoke-hypothesis-5ag-20ms.yaml" + go run ./tools/gen_scenario -family hypothesis-convergence -agents 5 -think-ms 20 -epochs 10 -max-epoch-ms 10000 -output "$scenario_path" + echo "==> smoke A2A" + {{ .BIN_DIR }}/a2a-runner \ + --scenario "$scenario_path" \ + --agent-bin {{ .BIN_DIR }}/a2a-agent \ + --quiet \ + --output-json {{ .REPORTS_DIR }}/a2a-ci-smoke.json \ + --output-tsv {{ .RESULTS_TSV }} + - task: compare:slim-stack + vars: + SCENARIO_GLOB: 'plans/sweeps/ci-smoke-hypothesis-5ag-20ms.yaml' + - task: compare:report + + test: + desc: Run unit tests + cmds: + - go test ./... -count=1 diff --git a/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go b/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go new file mode 100644 index 00000000..91fc4f66 --- /dev/null +++ b/benchmarks/slim-vs-a2a/a2a/cmd/agent/main.go @@ -0,0 +1,165 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "iter" + "log" + "net" + "net/http" + + "github.com/a2aproject/a2a-go/v2/a2a" + a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" + "github.com/a2aproject/a2a-go/v2/a2asrv" + "github.com/a2aproject/a2a-go/v2/a2asrv/taskstore" + agentrt "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/agent" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" + "google.golang.org/grpc" +) + +type consensusExecutor struct { + runtime *agentrt.Runtime +} + +func (e *consensusExecutor) Execute(ctx context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { + reqText := firstText(execCtx.Message) + req, err := agentrt.DecodeRequestText(reqText) + if err != nil { + return func(yield func(a2a.Event, error) bool) { + yield(nil, fmt.Errorf("decode request: %w", err)) + } + } + + if req.Op == protocol.OpStreamFindings { + return e.runtime.StreamFindings(ctx, execCtx) + } + + resp := e.runtime.HandleUnary(ctx, req) + body, err := json.Marshal(resp) + if err != nil { + return func(yield func(a2a.Event, error) bool) { + yield(nil, err) + } + } + return func(yield func(a2a.Event, error) bool) { + task := a2a.NewSubmittedTask(execCtx, execCtx.Message) + state := a2a.TaskStateCompleted + if !resp.OK { + state = a2a.TaskStateFailed + } + task.Status = a2a.TaskStatus{ + State: state, + Message: a2a.NewMessageForTask( + a2a.MessageRoleAgent, + task, + a2a.NewTextPart(string(body)), + ), + } + yield(task, nil) + } +} + +func (e *consensusExecutor) Cancel(_ context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { + return func(yield func(a2a.Event, error) bool) { + yield(a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateCanceled, a2a.NewMessageForTask( + a2a.MessageRoleAgent, execCtx, a2a.NewTextPart(`{"ok":true}`), + )), nil) + } +} + +func firstText(message *a2a.Message) string { + if message == nil { + return "" + } + for _, part := range message.Parts { + if text := part.Text(); text != "" { + return text + } + } + return "" +} + +func main() { + agentID := flag.String("agent-id", "", "logical agent id") + grpcPort := flag.Int("grpc-port", 0, "A2A gRPC port") + cardPort := flag.Int("card-port", 0, "agent card HTTP port") + scenarioFile := flag.String("scenario-file", "", "consensus scenario yaml") + agentIndex := flag.Int("agent-index", 0, "agent index") + relayCardURL := flag.String("relay-card-url", "", "runner relay agent card base URL") + quiet := flag.Bool("quiet", false, "disable benchmark logs") + flag.Parse() + + if *quiet { + benchlog.SetEnabled(false) + } + + if *agentID == "" || *grpcPort == 0 || *scenarioFile == "" || *relayCardURL == "" { + log.Fatal("agent-id, grpc-port, scenario-file, and relay-card-url are required") + } + card := *cardPort + if card == 0 { + card = *grpcPort + 1000 + } + + s, err := scenario.LoadFile(*scenarioFile) + if err != nil { + log.Fatalf("load scenario: %v", err) + } + + rt := agentrt.NewRuntime(s, *agentIndex, *agentID, *relayCardURL) + defer rt.Close() + + handler := a2asrv.NewHandler( + &consensusExecutor{runtime: rt}, + a2asrv.WithTaskStore(taskstore.NewInMemory(&taskstore.InMemoryStoreConfig{ + Authenticator: func(context.Context) (string, error) { return "slim-vs-a2a", nil }, + })), + ) + + agentCard := &a2a.AgentCard{ + Name: fmt.Sprintf("consensus-v2-%s", *agentID), + Description: "SLIM vs A2A v2 consensus agent", + Version: "1.0.0", + SupportedInterfaces: []*a2a.AgentInterface{ + a2a.NewAgentInterface(fmt.Sprintf("127.0.0.1:%d", *grpcPort), a2a.TransportProtocolGRPC), + }, + Capabilities: a2a.AgentCapabilities{Streaming: true}, + DefaultInputModes: []string{"text/plain"}, + DefaultOutputModes: []string{"text/plain"}, + } + + cardListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", card)) + if err != nil { + log.Fatalf("card listen: %v", err) + } + grpcListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", *grpcPort)) + if err != nil { + log.Fatalf("grpc listen: %v", err) + } + + grpcHandler := a2agrpc.NewHandler(handler) + grpcServer := grpc.NewServer() + grpcHandler.RegisterWith(grpcServer) + + go func() { + mux := http.NewServeMux() + mux.Handle(a2asrv.WellKnownAgentCardPath, a2asrv.NewStaticAgentCardHandler(agentCard)) + if err := http.Serve(cardListener, mux); err != nil { + log.Printf("card server stopped: %v", err) + } + }() + + if !*quiet { + fmt.Printf("A2A_AGENT_READY agent=%s grpc=%d card=%d\n", *agentID, *grpcPort, card) + } + if err := grpcServer.Serve(grpcListener); err != nil { + log.Fatalf("grpc serve: %v", err) + } +} diff --git a/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go b/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go new file mode 100644 index 00000000..b31dd2e5 --- /dev/null +++ b/benchmarks/slim-vs-a2a/a2a/cmd/runner/main.go @@ -0,0 +1,324 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Command runner is the A2A benchmark driver and relay hub. Because A2A has no +// peer multicast, the runner is the explicit relay: it subscribes to every +// agent's findings stream (OpStreamFindings) and fans each finding out to all +// other agents over the relay stream (OpStreamRelay). It also drives start and +// polls snapshots to detect global consensus. +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "log" + "os" + "os/exec" + "time" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/client" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/relay" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" +) + +func main() { + scenarioPath := flag.String("scenario", "", "path to consensus scenario yaml") + agentBin := flag.String("agent-bin", "", "path to a2a-agent binary") + outputJSON := flag.String("output-json", "", "write run metrics json") + outputTSV := flag.String("output-tsv", "", "append run metrics tsv") + waitReady := flag.Duration("wait-ready", 3*time.Second, "wait for agents to start") + relayGRPCPort := flag.Int("relay-grpc-port", 9600, "relay hub gRPC port") + relayCardPort := flag.Int("relay-card-port", 9601, "relay hub agent card port") + quiet := flag.Bool("quiet", false, "disable benchmark logs") + flag.Parse() + + if *quiet { + benchlog.SetEnabled(false) + } + if *scenarioPath == "" { + log.Fatal("--scenario is required") + } + + s, err := scenario.LoadFile(*scenarioPath) + if err != nil { + log.Fatalf("load scenario: %v", err) + } + + agentPath := *agentBin + if agentPath == "" { + agentPath = os.Getenv("A2A_AGENT_BIN") + } + if agentPath == "" { + log.Fatal("set --agent-bin or A2A_AGENT_BIN") + } + + // Start the relay hub first so agents can subscribe as they come up. + hub := relay.NewHub(*relayGRPCPort, *relayCardPort) + if err := hub.Serve(); err != nil { + log.Fatalf("relay serve: %v", err) + } + + procs := startAgents(s, agentPath, *scenarioPath, hub.CardURL(), *quiet) + defer stopAgents(procs) + time.Sleep(*waitReady) + + overall := time.Duration(s.Spec.Epochs)*time.Duration(s.Spec.MaxEpochTimeMs)*time.Millisecond + 5*time.Minute + ctx, cancel := context.WithTimeout(context.Background(), overall) + defer cancel() + + cli, err := newClientWithRetry(ctx, s) + if err != nil { + stopAgents(procs) + log.Fatalf("client: %v", err) + } + + agentIDs := s.AgentIDs() + agentIndexByID := map[string]int{} + for i, a := range s.Agents { + agentIndexByID[a.ID] = i + } + + // Runner subscribes to every agent's findings stream and relays each finding + // to all other agents. Retries keep the subscription alive across restarts. + for _, id := range agentIDs { + go func(agentID string) { + for { + select { + case <-ctx.Done(): + return + default: + } + err := cli.SubscribeFindings(ctx, agentID, func(f consensus.Finding) { + hub.Broadcast(f) + }) + if err == nil || ctx.Err() != nil { + return + } + time.Sleep(300 * time.Millisecond) + } + }(id) + } + + // Let finding subscriptions establish before agents start producing. + time.Sleep(500 * time.Millisecond) + + runStart := time.Now() + benchlog.SetRunStart(runStart) + + result := metrics.RunResult{ + ScenarioName: s.Metadata.Name, + Domain: s.Metadata.Domain, + Implementation: benchlog.ImplA2A, + Agents: len(s.Agents), + ThinkTimeMs: s.Spec.ThinkTimeMs, + PayloadBytes: s.Spec.PayloadBytes, + Epochs: s.Spec.Epochs, + } + + maxEpochTime := time.Duration(s.Spec.MaxEpochTimeMs) * time.Millisecond + var ( + wallSamples []int64 + allSnapshots []consensus.AgentSnapshot + ) + for epoch := 0; epoch < s.Spec.Epochs; epoch++ { + log.Printf("epoch %d/%d started (budget %s)", epoch+1, s.Spec.Epochs, maxEpochTime) + hub.SetCurrentEpoch(epoch) + if err := cli.StartAll(ctx, agentIDs, epoch); err != nil { + stopAgents(procs) + log.Fatalf("start agents (epoch %d): %v", epoch, err) + } + epochStart := time.Now() + deadline := epochStart.Add(maxEpochTime) + var snapshots []consensus.AgentSnapshot + success := false + for time.Now().Before(deadline) { + snapshots = snapshots[:0] + var pollErr error + for _, id := range agentIDs { + snap, err := cli.Snapshot(ctx, id) + if err != nil { + pollErr = err + break + } + snapshots = append(snapshots, snap) + } + // Only accept consensus once every agent has reset into this epoch. + if pollErr == nil && allForEpoch(snapshots, epoch) { + if ok, _ := consensus.GlobalConsensus(snapshots); ok { + success = true + break + } + } + time.Sleep(20 * time.Millisecond) + } + allSnapshots = append(allSnapshots, snapshots...) + epochWall := time.Since(epochStart) + if success { + result.EpochsSucceeded++ + wallSamples = append(wallSamples, epochWall.Milliseconds()) + log.Printf("epoch %d/%d ok wall_ms=%d", epoch+1, s.Spec.Epochs, epochWall.Milliseconds()) + } else { + result.EpochsFailed++ + log.Printf("epoch %d/%d failed after %s (budget %s exhausted)", epoch+1, s.Spec.Epochs, epochWall.Round(time.Millisecond), maxEpochTime) + } + } + + result.Success = result.EpochsFailed == 0 + result = aggregateResult(result, wallSamples, allSnapshots, hub) + + stopAgents(procs) + + if *outputJSON != "" { + if err := metrics.WriteJSON(*outputJSON, result); err != nil { + log.Fatalf("write json: %v", err) + } + } + if *outputTSV != "" { + if err := metrics.AppendTSV(*outputTSV, result); err != nil { + log.Fatalf("write tsv: %v", err) + } + } + + data, _ := json.MarshalIndent(result, "", " ") + fmt.Println(string(data)) + if !result.Success { + os.Exit(1) + } +} + +func newClientWithRetry(ctx context.Context, s *scenario.ConsensusScenario) (*client.Client, error) { + var lastErr error + for attempt := 0; attempt < 50; attempt++ { + cli, err := client.New(ctx, s) + if err == nil { + return cli, nil + } + lastErr = err + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-time.After(200 * time.Millisecond): + } + } + return nil, lastErr +} + +// allForEpoch reports whether every polled snapshot belongs to the given epoch, +// i.e. all agents have reset into it. Empty input is treated as not-yet-ready. +func allForEpoch(snapshots []consensus.AgentSnapshot, epoch int) bool { + if len(snapshots) == 0 { + return false + } + for _, snap := range snapshots { + if snap.Epoch != epoch { + return false + } + } + return true +} + +// aggregateResult folds the snapshots collected across every epoch into the run +// result. wallSamples holds the wall-clock duration of each successful epoch. +func aggregateResult(result metrics.RunResult, wallSamples []int64, snapshots []consensus.AgentSnapshot, hub *relay.Hub) metrics.RunResult { + // Relay hub did all the fan-out work; report its real counters (cumulative + // across every epoch). + streamCount, fanoutMS := hub.Stats() + result.StreamRPCCount = streamCount + result.CoordFanoutMS = fanoutMS + + if len(snapshots) == 0 { + result.Error = "no agent snapshots" + return result + } + + var ( + totalEmitted int + totalApplied int + propDurations []int64 + maxConsensusRound int + maxRound int + ) + for _, snap := range snapshots { + totalEmitted += snap.FindingsEmitted + totalApplied += snap.FindingsApplied + if snap.ConsensusRound > maxConsensusRound { + maxConsensusRound = snap.ConsensusRound + } + if snap.Round > maxRound { + maxRound = snap.Round + } + if snap.AvgPropagationMs > 0 { + propDurations = append(propDurations, snap.AvgPropagationMs) + } + } + + meanWall, maxWall := summarizeWall(wallSamples) + result.ConsensusWallMS = meanWall + result.LastAgentConvergeMS = maxWall + if result.EpochsSucceeded == 0 { + result.Error = "consensus not reached in any epoch" + } + + result.ConsensusRound = maxConsensusRound + if result.ConsensusRound == 0 { + result.ConsensusRound = maxRound + } + result.FindingsEmitted = totalEmitted + result.FindingsReceivedTotal = totalApplied + result.AvgPropagationMS, result.P95PropagationMS = metrics.AggregatePropagation(propDurations) + return result +} + +// summarizeWall returns the mean and max of the per-epoch wall-clock samples. +func summarizeWall(wallSamples []int64) (mean, max int64) { + if len(wallSamples) == 0 { + return 0, 0 + } + var sum int64 + for _, w := range wallSamples { + sum += w + if w > max { + max = w + } + } + return sum / int64(len(wallSamples)), max +} + +func startAgents(s *scenario.ConsensusScenario, agentBin, scenarioFile, relayCardURL string, quiet bool) []*exec.Cmd { + var procs []*exec.Cmd + for i, agent := range s.Agents { + args := []string{ + "--agent-id", agent.ID, + "--grpc-port", fmt.Sprintf("%d", agent.A2APort), + "--card-port", fmt.Sprintf("%d", s.CardPort(agent)), + "--scenario-file", scenarioFile, + "--agent-index", fmt.Sprintf("%d", i), + "--relay-card-url", relayCardURL, + } + if quiet { + args = append(args, "--quiet") + } + cmd := exec.Command(agentBin, args...) + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + if err := cmd.Start(); err != nil { + stopAgents(procs) + log.Fatalf("start agent %s: %v", agent.ID, err) + } + procs = append(procs, cmd) + } + return procs +} + +func stopAgents(procs []*exec.Cmd) { + for _, cmd := range procs { + if cmd.Process != nil { + _ = cmd.Process.Kill() + } + } +} diff --git a/benchmarks/slim-vs-a2a/a2a/internal/agent/runtime.go b/benchmarks/slim-vs-a2a/a2a/internal/agent/runtime.go new file mode 100644 index 00000000..01330625 --- /dev/null +++ b/benchmarks/slim-vs-a2a/a2a/internal/agent/runtime.go @@ -0,0 +1,317 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Package agent implements an A2A consensus agent for the relay topology. +// +// There is no peer-to-peer multicast in A2A, so every finding flows through the +// runner relay over two server-streaming legs: +// - OpStreamFindings: this agent streams the findings it produces to the +// runner (the runner is the stream client, this agent the stream server). +// - OpStreamRelay: this agent subscribes to the runner and receives every +// relayed finding (this agent is the stream client, the runner the server). +// +// OpStart / OpSnapshot remain unary calls the runner drives. +package agent + +import ( + "context" + "encoding/json" + "fmt" + "iter" + "sync" + "time" + + "github.com/a2aproject/a2a-go/v2/a2a" + "github.com/a2aproject/a2a-go/v2/a2aclient" + "github.com/a2aproject/a2a-go/v2/a2aclient/agentcard" + a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" + "github.com/a2aproject/a2a-go/v2/a2asrv" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" +) + +type Runtime struct { + scenario *scenario.ConsensusScenario + agentIndex int + agentID string + relayCardURL string + + engine *consensus.Engine + + outbound chan consensus.Finding + startedAt time.Time + + relayMu sync.Mutex + relayCancel context.CancelFunc + + mu sync.Mutex + // lastStartedEpoch dedupes repeated OpStart for the same epoch; + // currentEpoch is the attempt currently running (used to drop stale relayed + // findings and to stop a superseded epoch loop). + lastStartedEpoch int + currentEpoch int +} + +func NewRuntime(s *scenario.ConsensusScenario, agentIndex int, agentID, relayCardURL string) *Runtime { + return &Runtime{ + scenario: s, + agentIndex: agentIndex, + agentID: agentID, + relayCardURL: relayCardURL, + engine: consensus.NewEngine(s.Spec, agentIndex), + outbound: make(chan consensus.Finding, 1024), + lastStartedEpoch: -1, + currentEpoch: -1, + } +} + +// stopRelay ends the current OpStreamRelay subscription, if any. +func (r *Runtime) stopRelay() { + r.relayMu.Lock() + if r.relayCancel != nil { + r.relayCancel() + r.relayCancel = nil + } + r.relayMu.Unlock() +} + +// startRelay dials the runner relay and subscribes to OpStreamRelay for the +// current epoch. Each epoch gets a fresh stream so a2a-go task history does +// not accumulate across attempts. +func (r *Runtime) startRelay() { + r.stopRelay() + relayCtx, cancel := context.WithCancel(context.Background()) + r.relayMu.Lock() + r.relayCancel = cancel + r.relayMu.Unlock() + go r.relayLoop(relayCtx) +} + +func (r *Runtime) drainOutbound() { + for { + select { + case <-r.outbound: + default: + return + } + } +} + +func (r *Runtime) relayLoop(ctx context.Context) { + cli := r.dialRelay(ctx) + if cli == nil { + return + } + req := protocol.Request{Op: protocol.OpStreamRelay, AgentIndex: r.agentIndex} + text, err := json.Marshal(req) + if err != nil { + return + } + for { + select { + case <-ctx.Done(): + return + default: + } + msg := a2a.NewMessage(a2a.MessageRoleUser, a2a.NewTextPart(string(text))) + for ev, streamErr := range cli.SendStreamingMessage(ctx, &a2a.SendMessageRequest{Message: msg}) { + if streamErr != nil { + break + } + f, ok := findingFromEvent(ev) + if !ok || f.AgentIndex == r.agentIndex { + continue + } + // Drop stragglers from a different (usually previous) epoch. + r.mu.Lock() + cur := r.currentEpoch + r.mu.Unlock() + if f.Epoch != cur { + continue + } + r.applyFinding(f) + } + select { + case <-ctx.Done(): + return + case <-time.After(500 * time.Millisecond): + } + } +} + +func (r *Runtime) dialRelay(ctx context.Context) *a2aclient.Client { + for attempt := 0; attempt < 100; attempt++ { + select { + case <-ctx.Done(): + return nil + default: + } + card, err := agentcard.DefaultResolver.Resolve(ctx, r.relayCardURL) + if err == nil { + cli, cerr := a2aclient.NewFromCard(ctx, card, + a2agrpc.WithGRPCTransport(grpc.WithTransportCredentials(insecure.NewCredentials()))) + if cerr == nil { + return cli + } + } + time.Sleep(200 * time.Millisecond) + } + return nil +} + +// HandleUnary serves the unary control ops (start, snapshot). +func (r *Runtime) HandleUnary(_ context.Context, req protocol.Request) protocol.Response { + switch req.Op { + case protocol.OpStart: + r.startEpoch(req.Epoch) + return protocol.Response{OK: true} + case protocol.OpSnapshot: + body, err := json.Marshal(r.engine.Snapshot()) + if err != nil { + return protocol.Response{OK: false, Error: err.Error()} + } + return protocol.Response{OK: true, Body: string(body)} + default: + return protocol.Response{OK: false, Error: "unknown op"} + } +} + +// StreamFindings serves OpStreamFindings: a long-lived server stream that emits +// every finding this agent produces. The first event establishes the task; each +// finding is delivered as a non-terminal Working status update so the stream +// stays open. +func (r *Runtime) StreamFindings(ctx context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { + return func(yield func(a2a.Event, error) bool) { + task := a2a.NewSubmittedTask(execCtx, execCtx.Message) + if !yield(task, nil) { + return + } + for { + select { + case <-ctx.Done(): + yield(a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateCompleted, nil), nil) + return + case f := <-r.outbound: + body, err := consensus.EncodeFinding(f) + if err != nil { + continue + } + msg := a2a.NewMessageForTask(a2a.MessageRoleAgent, execCtx, a2a.NewTextPart(string(body))) + ev := a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateWorking, msg) + if !yield(ev, nil) { + return + } + } + } + } +} + +// startEpoch resets the engine for a fresh attempt and launches the epoch loop. +// Repeated starts for the same (or older) epoch are ignored so a duplicated +// call cannot double-run an attempt. +func (r *Runtime) startEpoch(epoch int) { + r.mu.Lock() + if epoch <= r.lastStartedEpoch { + r.mu.Unlock() + return + } + r.lastStartedEpoch = epoch + r.currentEpoch = epoch + r.mu.Unlock() + + r.stopRelay() + r.drainOutbound() + r.engine.Reset(epoch) + r.startedAt = time.Now() + benchlog.SetRunStart(r.startedAt) + r.startRelay() + go r.runEpoch(epoch) +} + +func (r *Runtime) runEpoch(epoch int) { + spec := r.scenario.Spec + think := time.Duration(spec.ThinkTimeMs) * time.Millisecond + emitGap := time.Duration(spec.FindingEmitDelayMs) * time.Millisecond + + for round := 0; round < spec.MaxRounds; round++ { + // Stop early if a newer epoch has superseded this one. + r.mu.Lock() + superseded := r.currentEpoch != epoch + r.mu.Unlock() + if superseded { + return + } + + time.Sleep(think) + + finding, emit := r.engine.Think() + if emit && finding != nil { + select { + case r.outbound <- *finding: + benchlog.Finding(benchlog.ImplA2A, "published", r.agentIndex, finding.FindingID) + case <-time.After(5 * time.Second): + benchlog.Finding(benchlog.ImplA2A, "publish_timeout", r.agentIndex, finding.FindingID) + } + time.Sleep(emitGap) + } + + if r.engine.HasLocalConsensus() { + return + } + } +} + +func (r *Runtime) applyFinding(f consensus.Finding) { + recvNs := time.Now().UnixNano() + r.engine.ApplyFinding(f) + if f.EmittedAt > 0 { + r.engine.RecordPropagation(f.EmittedAt, recvNs) + } + benchlog.Finding(benchlog.ImplA2A, "received", r.agentIndex, f.FindingID, + fmt.Sprintf("from=%d", f.AgentIndex)) +} + +func (r *Runtime) Snapshot() consensus.AgentSnapshot { + return r.engine.Snapshot() +} + +func (r *Runtime) Close() {} + +// findingFromEvent extracts a finding from a streamed event, ignoring the +// initial submitted-task event and any event without a message payload. +func findingFromEvent(ev a2a.Event) (consensus.Finding, bool) { + su, ok := ev.(*a2a.TaskStatusUpdateEvent) + if !ok || su.Status.Message == nil { + return consensus.Finding{}, false + } + text := firstText(su.Status.Message) + if text == "" { + return consensus.Finding{}, false + } + f, err := consensus.DecodeFinding([]byte(text)) + if err != nil { + return consensus.Finding{}, false + } + return f, true +} + +func firstText(message *a2a.Message) string { + if message == nil { + return "" + } + for _, part := range message.Parts { + if text := part.Text(); text != "" { + return text + } + } + return "" +} + +// DecodeRequestText is used by the agent server executor. +func DecodeRequestText(text string) (protocol.Request, error) { + return protocol.DecodeRequest(text) +} diff --git a/benchmarks/slim-vs-a2a/a2a/internal/client/client.go b/benchmarks/slim-vs-a2a/a2a/internal/client/client.go new file mode 100644 index 00000000..a6c537d4 --- /dev/null +++ b/benchmarks/slim-vs-a2a/a2a/internal/client/client.go @@ -0,0 +1,160 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package client + +import ( + "context" + "encoding/json" + "fmt" + + "github.com/a2aproject/a2a-go/v2/a2a" + "github.com/a2aproject/a2a-go/v2/a2aclient" + "github.com/a2aproject/a2a-go/v2/a2aclient/agentcard" + a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" +) + +type Client struct { + clients map[string]*a2aclient.Client +} + +func New(ctx context.Context, s *scenario.ConsensusScenario) (*Client, error) { + clients := map[string]*a2aclient.Client{} + for _, agent := range s.Agents { + baseURL := s.CardBaseURL(agent) + card, err := agentcard.DefaultResolver.Resolve(ctx, baseURL) + if err != nil { + return nil, fmt.Errorf("resolve card for %s: %w", agent.ID, err) + } + cli, err := a2aclient.NewFromCard( + ctx, + card, + a2agrpc.WithGRPCTransport(grpc.WithTransportCredentials(insecure.NewCredentials())), + ) + if err != nil { + return nil, fmt.Errorf("client for %s: %w", agent.ID, err) + } + clients[agent.ID] = cli + } + return &Client{clients: clients}, nil +} + +func (c *Client) StartAll(ctx context.Context, agentIDs []string, epoch int) error { + for _, id := range agentIDs { + if err := c.send(ctx, id, protocol.Request{Op: protocol.OpStart, Epoch: epoch}); err != nil { + return err + } + } + return nil +} + +func (c *Client) Snapshot(ctx context.Context, agentID string) (consensus.AgentSnapshot, error) { + resp, err := c.sendWithResponse(ctx, agentID, protocol.Request{Op: protocol.OpSnapshot}) + if err != nil { + return consensus.AgentSnapshot{}, err + } + var snap consensus.AgentSnapshot + if err := json.Unmarshal([]byte(resp.Body), &snap); err != nil { + return consensus.AgentSnapshot{}, err + } + return snap, nil +} + +// SubscribeFindings opens the OpStreamFindings server stream to an agent and +// invokes onFinding for every finding the agent produces. It blocks until the +// stream ends or ctx is cancelled, so callers run it in a goroutine. +func (c *Client) SubscribeFindings(ctx context.Context, agentID string, onFinding func(consensus.Finding)) error { + cli, ok := c.clients[agentID] + if !ok { + return fmt.Errorf("unknown agent %q", agentID) + } + reqText, err := json.Marshal(protocol.Request{Op: protocol.OpStreamFindings}) + if err != nil { + return err + } + msg := a2a.NewMessage(a2a.MessageRoleUser, a2a.NewTextPart(string(reqText))) + for ev, streamErr := range cli.SendStreamingMessage(ctx, &a2a.SendMessageRequest{Message: msg}) { + if streamErr != nil { + return streamErr + } + f, ok := findingFromEvent(ev) + if !ok { + continue + } + onFinding(f) + } + return nil +} + +func (c *Client) send(ctx context.Context, agentID string, req protocol.Request) error { + _, err := c.sendWithResponse(ctx, agentID, req) + return err +} + +func (c *Client) sendWithResponse(ctx context.Context, agentID string, req protocol.Request) (protocol.Response, error) { + cli, ok := c.clients[agentID] + if !ok { + return protocol.Response{}, fmt.Errorf("unknown agent %q", agentID) + } + text, err := json.Marshal(req) + if err != nil { + return protocol.Response{}, err + } + msg := a2a.NewMessage(a2a.MessageRoleUser, a2a.NewTextPart(string(text))) + result, err := cli.SendMessage(ctx, &a2a.SendMessageRequest{Message: msg}) + if err != nil { + return protocol.Response{}, err + } + respText, err := responseText(result) + if err != nil { + return protocol.Response{}, err + } + return protocol.DecodeResponse(respText) +} + +func findingFromEvent(ev a2a.Event) (consensus.Finding, bool) { + su, ok := ev.(*a2a.TaskStatusUpdateEvent) + if !ok || su.Status.Message == nil { + return consensus.Finding{}, false + } + text := firstText(su.Status.Message) + if text == "" { + return consensus.Finding{}, false + } + f, err := consensus.DecodeFinding([]byte(text)) + if err != nil { + return consensus.Finding{}, false + } + return f, true +} + +func responseText(result a2a.SendMessageResult) (string, error) { + switch v := result.(type) { + case *a2a.Message: + return firstText(v), nil + case *a2a.Task: + if v.Status.Message != nil { + return firstText(v.Status.Message), nil + } + return "", fmt.Errorf("task response missing message") + default: + return "", fmt.Errorf("unexpected result type %T", result) + } +} + +func firstText(message *a2a.Message) string { + if message == nil { + return "" + } + for _, part := range message.Parts { + if text := part.Text(); text != "" { + return text + } + } + return "" +} diff --git a/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go b/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go new file mode 100644 index 00000000..12a07643 --- /dev/null +++ b/benchmarks/slim-vs-a2a/a2a/internal/protocol/protocol.go @@ -0,0 +1,61 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Package protocol defines the A2A control/stream ops for the relay topology. +// +// Unlike SLIM's native group session, A2A has no peer multicast: findings must +// pass through the runner, which is the explicit relay hub. Two server-streaming +// legs carry findings: +// - OpStreamFindings: the runner subscribes to each agent (agent streams its +// own findings to the runner). +// - OpStreamRelay: each agent subscribes to the runner (runner streams every +// relayed finding back out to the agents). +// +// OpStart and OpSnapshot remain unary control calls driven by the runner. +package protocol + +import ( + "encoding/json" +) + +const ( + OpStart = "start" + OpSnapshot = "snapshot" + OpStreamFindings = "stream_findings" + OpStreamRelay = "stream_relay" +) + +type Request struct { + Op string `json:"op"` + // AgentIndex identifies the subscriber on OpStreamRelay so the relay can + // avoid echoing an agent's own findings back to it. + AgentIndex int `json:"agentIndex,omitempty"` + // Epoch identifies which consensus attempt an OpStart begins. + Epoch int `json:"epoch,omitempty"` +} + +type Response struct { + OK bool `json:"ok"` + Error string `json:"error,omitempty"` + Body string `json:"body,omitempty"` +} + +func EncodeRequest(req Request) ([]byte, error) { + return json.Marshal(req) +} + +func DecodeRequest(text string) (Request, error) { + var req Request + err := json.Unmarshal([]byte(text), &req) + return req, err +} + +func EncodeResponse(resp Response) ([]byte, error) { + return json.Marshal(resp) +} + +func DecodeResponse(text string) (Response, error) { + var resp Response + err := json.Unmarshal([]byte(text), &resp) + return resp, err +} diff --git a/benchmarks/slim-vs-a2a/a2a/internal/relay/relay.go b/benchmarks/slim-vs-a2a/a2a/internal/relay/relay.go new file mode 100644 index 00000000..935303ec --- /dev/null +++ b/benchmarks/slim-vs-a2a/a2a/internal/relay/relay.go @@ -0,0 +1,250 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Package relay implements the runner-side relay hub for the A2A topology. +// +// The hub is the central point every finding must pass through, because A2A has +// no peer multicast. It hosts an A2A server exposing OpStreamRelay: each agent +// subscribes and receives a server stream of every relayed finding. The runner +// separately subscribes to each agent's OpStreamFindings stream and calls +// Broadcast for every finding, which fans it out to all other agents' relay +// streams. Broadcast drops findings whose epoch does not match the hub's +// current epoch so stragglers from a prior attempt are not relayed. This +// relay fan-out is exactly the work native SLIM avoids. +package relay + +import ( + "context" + "fmt" + "iter" + "net" + "net/http" + "sync" + "sync/atomic" + "time" + + "github.com/a2aproject/a2a-go/v2/a2a" + a2agrpc "github.com/a2aproject/a2a-go/v2/a2agrpc/v1" + "github.com/a2aproject/a2a-go/v2/a2asrv" + "github.com/a2aproject/a2a-go/v2/a2asrv/taskstore" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/a2a/internal/protocol" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" + "google.golang.org/grpc" +) + +type subscriber struct { + agentIndex int + ch chan consensus.Finding +} + +type Hub struct { + grpcPort int + cardPort int + + mu sync.Mutex + subs map[*subscriber]struct{} + streamRPCCount int + fanoutMS int64 + // currentEpoch is the consensus attempt the hub will relay. Findings tagged + // with a different epoch are dropped so stragglers from a prior attempt do + // not fan out or grow relay-stream task history. + currentEpoch atomic.Int32 +} + +func NewHub(grpcPort, cardPort int) *Hub { + return &Hub{ + grpcPort: grpcPort, + cardPort: cardPort, + subs: map[*subscriber]struct{}{}, + } +} + +func (h *Hub) CardURL() string { + return fmt.Sprintf("http://127.0.0.1:%d", h.cardPort) +} + +// Serve binds the relay's gRPC and card listeners and serves them in the +// background. It returns once the listeners are bound so callers can hand the +// card URL to agents. +func (h *Hub) Serve() error { + handler := a2asrv.NewHandler( + &relayExecutor{hub: h}, + a2asrv.WithTaskStore(taskstore.NewInMemory(&taskstore.InMemoryStoreConfig{ + Authenticator: func(context.Context) (string, error) { return "slim-vs-a2a", nil }, + })), + ) + + card := &a2a.AgentCard{ + Name: "consensus-v2-relay", + Description: "SLIM vs A2A v2 relay hub", + Version: "1.0.0", + SupportedInterfaces: []*a2a.AgentInterface{ + a2a.NewAgentInterface(fmt.Sprintf("127.0.0.1:%d", h.grpcPort), a2a.TransportProtocolGRPC), + }, + Capabilities: a2a.AgentCapabilities{Streaming: true}, + DefaultInputModes: []string{"text/plain"}, + DefaultOutputModes: []string{"text/plain"}, + } + + cardListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", h.cardPort)) + if err != nil { + return fmt.Errorf("relay card listen: %w", err) + } + grpcListener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", h.grpcPort)) + if err != nil { + _ = cardListener.Close() + return fmt.Errorf("relay grpc listen: %w", err) + } + + grpcHandler := a2agrpc.NewHandler(handler) + grpcServer := grpc.NewServer() + grpcHandler.RegisterWith(grpcServer) + + go func() { + mux := http.NewServeMux() + mux.Handle(a2asrv.WellKnownAgentCardPath, a2asrv.NewStaticAgentCardHandler(card)) + _ = http.Serve(cardListener, mux) + }() + go func() { + _ = grpcServer.Serve(grpcListener) + }() + return nil +} + +// ResetStreams closes every active relay subscription so agents open fresh +// OpStreamRelay tasks on the next epoch instead of growing one task forever. +func (h *Hub) ResetStreams() { + h.mu.Lock() + subs := make([]*subscriber, 0, len(h.subs)) + for s := range h.subs { + subs = append(subs, s) + } + h.subs = map[*subscriber]struct{}{} + h.mu.Unlock() + for _, s := range subs { + close(s.ch) + } +} + +// SetCurrentEpoch advances the relay filter to the given epoch. Only findings +// stamped with this epoch are broadcast; older (or future) epochs are ignored. +func (h *Hub) SetCurrentEpoch(epoch int) { + h.currentEpoch.Store(int32(epoch)) +} + +// Broadcast fans a finding out to every subscriber except its producer. +func (h *Hub) Broadcast(f consensus.Finding) { + if int(h.currentEpoch.Load()) != f.Epoch { + return + } + start := time.Now() + h.mu.Lock() + targets := make([]*subscriber, 0, len(h.subs)) + for s := range h.subs { + if s.agentIndex == f.AgentIndex { + continue + } + targets = append(targets, s) + } + h.mu.Unlock() + + delivered := 0 + for _, s := range targets { + select { + case s.ch <- f: + delivered++ + case <-time.After(5 * time.Second): + } + } + + h.mu.Lock() + h.streamRPCCount += delivered + h.fanoutMS += time.Since(start).Milliseconds() + h.mu.Unlock() +} + +func (h *Hub) Stats() (streamRPCCount int, fanoutMS int64) { + h.mu.Lock() + defer h.mu.Unlock() + return h.streamRPCCount, h.fanoutMS +} + +func (h *Hub) register(agentIndex int) *subscriber { + s := &subscriber{agentIndex: agentIndex, ch: make(chan consensus.Finding, 1024)} + h.mu.Lock() + h.subs[s] = struct{}{} + h.mu.Unlock() + return s +} + +func (h *Hub) unregister(s *subscriber) { + h.mu.Lock() + delete(h.subs, s) + h.mu.Unlock() +} + +type relayExecutor struct { + hub *Hub +} + +func (e *relayExecutor) Execute(ctx context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { + req, err := protocol.DecodeRequest(firstText(execCtx.Message)) + if err != nil { + return func(yield func(a2a.Event, error) bool) { + yield(nil, fmt.Errorf("decode request: %w", err)) + } + } + if req.Op != protocol.OpStreamRelay { + return func(yield func(a2a.Event, error) bool) { + yield(nil, fmt.Errorf("relay only serves stream_relay, got %q", req.Op)) + } + } + + sub := e.hub.register(req.AgentIndex) + return func(yield func(a2a.Event, error) bool) { + defer e.hub.unregister(sub) + task := a2a.NewSubmittedTask(execCtx, execCtx.Message) + if !yield(task, nil) { + return + } + for { + select { + case <-ctx.Done(): + yield(a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateCompleted, nil), nil) + return + case f, ok := <-sub.ch: + if !ok { + yield(a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateCompleted, nil), nil) + return + } + body, err := consensus.EncodeFinding(f) + if err != nil { + continue + } + msg := a2a.NewMessageForTask(a2a.MessageRoleAgent, execCtx, a2a.NewTextPart(string(body))) + ev := a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateWorking, msg) + if !yield(ev, nil) { + return + } + } + } + } +} + +func (e *relayExecutor) Cancel(_ context.Context, execCtx *a2asrv.ExecutorContext) iter.Seq2[a2a.Event, error] { + return func(yield func(a2a.Event, error) bool) { + yield(a2a.NewStatusUpdateEvent(execCtx, a2a.TaskStateCanceled, nil), nil) + } +} + +func firstText(message *a2a.Message) string { + if message == nil { + return "" + } + for _, part := range message.Parts { + if text := part.Text(); text != "" { + return text + } + } + return "" +} diff --git a/benchmarks/slim-vs-a2a/go.mod b/benchmarks/slim-vs-a2a/go.mod new file mode 100644 index 00000000..02b62e40 --- /dev/null +++ b/benchmarks/slim-vs-a2a/go.mod @@ -0,0 +1,25 @@ +module github.com/agntcy/csit/benchmarks/slim-vs-a2a + +go 1.25.0 + +require ( + github.com/a2aproject/a2a-go/v2 v2.3.0 + github.com/agntcy/slim-bindings-go v1.4.0 + google.golang.org/grpc v1.80.0 + gopkg.in/yaml.v3 v3.0.1 +) + +require ( + github.com/google/uuid v1.6.0 // indirect + github.com/kr/pretty v0.3.1 // indirect + github.com/rogpeppe/go-internal v1.13.1 // indirect + golang.org/x/mod v0.35.0 // indirect + golang.org/x/net v0.53.0 // indirect + golang.org/x/sync v0.20.0 // indirect + golang.org/x/sys v0.43.0 // indirect + golang.org/x/text v0.36.0 // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 // indirect + google.golang.org/protobuf v1.36.11 // indirect + gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 // indirect +) diff --git a/benchmarks/slim-vs-a2a/go.sum b/benchmarks/slim-vs-a2a/go.sum new file mode 100644 index 00000000..d2fc54cd --- /dev/null +++ b/benchmarks/slim-vs-a2a/go.sum @@ -0,0 +1,62 @@ +github.com/a2aproject/a2a-go/v2 v2.3.0 h1:wSseKBBlDYBAJLdHZ4puFIL2XyL/5YuMrDw9TXLL1ek= +github.com/a2aproject/a2a-go/v2 v2.3.0/go.mod h1:mkZr8y2bUgAVQsjs/5fHK7xrRlAHDybMEyxWh2tKRC8= +github.com/agntcy/slim-bindings-go v1.4.0 h1:DXAGhe9TWSm33KIgrJFkIcxdqh+/oPWy4Jq5PCI0HvM= +github.com/agntcy/slim-bindings-go v1.4.0/go.mod h1:XK0Ing+REEl8xG79HTMx52XzWK2THuTQA+Y7JTAn428= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= +github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA= +github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= +github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII= +github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o= +go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= +go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= +go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48= +go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8= +go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0= +go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs= +go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18= +go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE= +go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8= +go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew= +go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI= +go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA= +golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM= +golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU= +golang.org/x/net v0.53.0 h1:d+qAbo5L0orcWAr0a9JweQpjXF19LMXJE8Ey7hwOdUA= +golang.org/x/net v0.53.0/go.mod h1:JvMuJH7rrdiCfbeHoo3fCQU24Lf5JJwT9W3sJFulfgs= +golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= +golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI= +golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/text v0.36.0 h1:JfKh3XmcRPqZPKevfXVpI1wXPTqbkE5f7JA92a55Yxg= +golang.org/x/text v0.36.0/go.mod h1:NIdBknypM8iqVmPiuco0Dh6P5Jcdk8lJL0CUebqK164= +gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= +gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= +google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4 h1:yOzSCGPx+cp5VO7IxvZ9SBFF7j1tZVcNtlHR2iYKtVo= +google.golang.org/genproto/googleapis/api v0.0.0-20260427160629-7cedc36a6bc4/go.mod h1:Q9HWtNeE7tM9npdIsEvqXj1QJIvVoeAV3rtXtS715Cw= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 h1:tEkOQcXgF6dH1G+MVKZrfpYvozGrzb91k6ha7jireSM= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= +google.golang.org/grpc v1.80.0 h1:Xr6m2WmWZLETvUNvIUmeD5OAagMw3FiKmMlTdViWsHM= +google.golang.org/grpc v1.80.0/go.mod h1:ho/dLnxwi3EDJA4Zghp7k2Ec1+c2jqup0bFkw07bwF4= +google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= +google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/benchmarks/slim-vs-a2a/internal/benchlog/benchlog.go b/benchmarks/slim-vs-a2a/internal/benchlog/benchlog.go new file mode 100644 index 00000000..bdb4999d --- /dev/null +++ b/benchmarks/slim-vs-a2a/internal/benchlog/benchlog.go @@ -0,0 +1,89 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package benchlog + +import ( + "fmt" + "io" + "log" + "os" + "strings" + "sync" + "time" +) + +const ( + ImplSLIM = "slim-group-session" + ImplA2A = "a2a-relay-stream" +) + +var ( + mu sync.RWMutex + runStart time.Time + enabled = true + out io.Writer = os.Stderr +) + +func SetRunStart(t time.Time) { + mu.Lock() + runStart = t + mu.Unlock() +} + +func SetEnabled(on bool) { + mu.Lock() + enabled = on + mu.Unlock() +} + +func sinceStart() int64 { + mu.RLock() + start := runStart + mu.RUnlock() + if start.IsZero() { + return 0 + } + return time.Since(start).Milliseconds() +} + +func write(kind, impl string, kv ...string) { + mu.RLock() + on := enabled + mu.RUnlock() + if !on { + return + } + ts := time.Now().UTC().Format(time.RFC3339Nano) + elapsed := sinceStart() + parts := []string{ + fmt.Sprintf("bench ts=%s elapsed_ms=%d impl=%s kind=%s", ts, elapsed, impl, kind), + } + parts = append(parts, kv...) + log.New(out, "", 0).Println(strings.Join(parts, " ")) +} + +func RPC(impl, op, mode string, duration time.Duration, ok bool, kv ...string) { + status := "ok" + if !ok { + status = "error" + } + fields := []string{ + fmt.Sprintf("op=%s", op), + fmt.Sprintf("mode=%s", mode), + fmt.Sprintf("duration_ms=%d", duration.Milliseconds()), + fmt.Sprintf("status=%s", status), + } + fields = append(fields, kv...) + write("rpc", impl, fields...) +} + +func Finding(impl, event string, agentIndex int, findingID int64, kv ...string) { + fields := []string{ + fmt.Sprintf("event=%s", event), + fmt.Sprintf("agent=%d", agentIndex), + fmt.Sprintf("finding_id=%d", findingID), + } + fields = append(fields, kv...) + write("finding", impl, fields...) +} diff --git a/benchmarks/slim-vs-a2a/internal/consensus/consensus.go b/benchmarks/slim-vs-a2a/internal/consensus/consensus.go new file mode 100644 index 00000000..ec08b6d3 --- /dev/null +++ b/benchmarks/slim-vs-a2a/internal/consensus/consensus.go @@ -0,0 +1,378 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package consensus + +import ( + "encoding/json" + "fmt" + mrand "math/rand" + "sync" + "time" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" +) + +const consensusConfidence = 0.55 + +type Finding struct { + FindingID int64 `json:"findingId"` + AgentIndex int `json:"agentIndex"` + Epoch int `json:"epoch,omitempty"` + Round int `json:"round"` + Value int `json:"value"` + Confidence float64 `json:"confidence"` + EmittedAt int64 `json:"emittedAtNs"` + // Payload is optional fixed-size padding used to stress transport bandwidth. + // It carries no semantic meaning for consensus; it only inflates the wire + // size of every finding so the relay-vs-multicast cost difference shows. + Payload string `json:"payload,omitempty"` +} + +type Config struct { + AgentIndex int + AgentCount int + ValueSpace int + TargetMode string + Seed int64 + PayloadBytes int +} + +type Engine struct { + cfg Config + + mu sync.Mutex + + epoch int + round int + value int + confidence float64 + targetValue int + lastEmitValue int + lastEmitConf float64 + lastEmitRound int + nextFindingID int64 + distinctSupports map[int]map[int]struct{} + convergedAt int64 + consensusRound int + payload string + + propMu sync.Mutex + propDurations []int64 + findingsEmitted int + findingsApplied int +} + +func NewEngine(spec scenario.Spec, agentIndex int) *Engine { + k := spec.ValueSpace + if k <= 0 { + k = 3 + } + target := targetValue(spec, agentIndex) + return &Engine{ + cfg: Config{ + AgentIndex: agentIndex, + AgentCount: spec.Agents, + ValueSpace: k, + TargetMode: spec.TargetMode, + Seed: spec.Seed, + PayloadBytes: spec.PayloadBytes, + }, + value: agentIndex % k, + confidence: 0.1, + targetValue: target, + lastEmitValue: -1, + distinctSupports: map[int]map[int]struct{}{}, + payload: makePayload(spec.PayloadBytes, spec.Seed, agentIndex), + } +} + +// Reset restores the engine to its initial per-epoch state so the same +// consensus attempt can be re-run. The topology-defining fields (targetValue, +// payload, cfg) are preserved so every epoch is an identical repeat; only the +// accumulated round/convergence/propagation state is cleared. The new epoch +// index is stamped onto every finding and snapshot the engine produces next. +func (e *Engine) Reset(epoch int) { + e.mu.Lock() + e.epoch = epoch + e.round = 0 + e.value = e.cfg.AgentIndex % e.cfg.ValueSpace + e.confidence = 0.1 + e.lastEmitValue = -1 + e.lastEmitConf = 0 + e.lastEmitRound = 0 + e.nextFindingID = 0 + e.distinctSupports = map[int]map[int]struct{}{} + e.convergedAt = 0 + e.consensusRound = 0 + e.findingsEmitted = 0 + e.findingsApplied = 0 + e.mu.Unlock() + + e.propMu.Lock() + e.propDurations = nil + e.propMu.Unlock() +} + +// makePayload returns deterministic, reproducible padding of n bytes. It is used +// only to inflate finding size for transport stress tests, so math/rand is +// intentional here: the same seed/agent yields identical bytes across both +// transports, keeping the SLIM-vs-A2A comparison fair. Not security-sensitive. +func makePayload(n int, seed int64, agentIndex int) string { + if n <= 0 { + return "" + } + const alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 " + rng := mrand.New(mrand.NewSource(seed + int64(agentIndex)*1_000_003)) + b := make([]byte, n) + for i := range b { + b[i] = alphabet[rng.Intn(len(alphabet))] + } + return string(b) +} + +func targetValue(spec scenario.Spec, agentIndex int) int { + k := spec.ValueSpace + if k <= 0 { + k = 3 + } + switch spec.TargetMode { + case scenario.TargetModeMajority: + return (spec.Agents / 2) % k + default: + return agentIndex % k + } +} + +func (e *Engine) TargetValue() int { + return e.targetValue +} + +func (e *Engine) Think() (finding *Finding, emit bool) { + e.mu.Lock() + defer e.mu.Unlock() + + e.round++ + e.refreshConfidenceLocked() + + if e.value == e.targetValue && e.confidence >= consensusConfidence { + if e.convergedAt == 0 { + e.convergedAt = time.Now().UnixNano() + e.consensusRound = e.round + } + } + + if e.lastEmitValue == e.value && e.lastEmitConf == e.confidence && e.lastEmitRound == e.round { + return nil, false + } + + e.lastEmitValue = e.value + e.lastEmitConf = e.confidence + e.lastEmitRound = e.round + e.nextFindingID++ + e.findingsEmitted++ + + f := Finding{ + FindingID: e.nextFindingID, + AgentIndex: e.cfg.AgentIndex, + Epoch: e.epoch, + Round: e.round, + Value: e.value, + Confidence: e.confidence, + EmittedAt: time.Now().UnixNano(), + Payload: e.payload, + } + return &f, true +} + +func (e *Engine) refreshConfidenceLocked() { + sources := e.distinctSupports[e.value] + if len(sources)+1 >= (e.cfg.AgentCount+1)/2 { + e.confidence += 0.12 + if e.confidence > 1 { + e.confidence = 1 + } + } + if e.value != e.targetValue && e.round > 2 { + e.value = e.targetValue + e.confidence = maxFloat(e.confidence, 0.35) + } +} + +func (e *Engine) ApplyFinding(f Finding) { + e.mu.Lock() + defer e.mu.Unlock() + + e.findingsApplied++ + if _, ok := e.distinctSupports[f.Value]; !ok { + e.distinctSupports[f.Value] = map[int]struct{}{} + } + e.distinctSupports[f.Value][f.AgentIndex] = struct{}{} + + if f.Confidence >= e.confidence || f.Value == e.targetValue { + e.value = f.Value + e.confidence = maxFloat(e.confidence, f.Confidence) + } + e.refreshConfidenceLocked() + + if e.value == e.targetValue && e.confidence >= consensusConfidence && e.convergedAt == 0 { + e.convergedAt = time.Now().UnixNano() + e.consensusRound = e.round + } +} + +func (e *Engine) RecordPropagation(emitNs int64, recvNs int64) { + if emitNs <= 0 || recvNs <= emitNs { + return + } + e.propMu.Lock() + e.propDurations = append(e.propDurations, (recvNs-emitNs)/int64(time.Millisecond)) + e.propMu.Unlock() +} + +func (e *Engine) HasLocalConsensus() bool { + e.mu.Lock() + defer e.mu.Unlock() + return e.value == e.targetValue && e.confidence >= consensusConfidence +} + +func (e *Engine) LocalState() (value int, confidence float64, round int) { + e.mu.Lock() + defer e.mu.Unlock() + return e.value, e.confidence, e.round +} + +func (e *Engine) ConvergedAtNs() int64 { + e.mu.Lock() + defer e.mu.Unlock() + return e.convergedAt +} + +func (e *Engine) ConsensusRound() int { + e.mu.Lock() + defer e.mu.Unlock() + return e.consensusRound +} + +func (e *Engine) FindingsEmitted() int { + e.mu.Lock() + defer e.mu.Unlock() + return e.findingsEmitted +} + +func (e *Engine) FindingsApplied() int { + e.mu.Lock() + defer e.mu.Unlock() + return e.findingsApplied +} + +func (e *Engine) PropagationStats() (avg, p95 int64) { + e.propMu.Lock() + defer e.propMu.Unlock() + if len(e.propDurations) == 0 { + return 0, 0 + } + var sum int64 + for _, d := range e.propDurations { + sum += d + } + avg = sum / int64(len(e.propDurations)) + p95 = percentile(e.propDurations, 0.95) + return avg, p95 +} + +func EncodeFinding(f Finding) ([]byte, error) { + return json.Marshal(f) +} + +func DecodeFinding(data []byte) (Finding, error) { + var f Finding + err := json.Unmarshal(data, &f) + return f, err +} + +func maxFloat(a, b float64) float64 { + if a > b { + return a + } + return b +} + +func percentile(values []int64, p float64) int64 { + if len(values) == 0 { + return 0 + } + sorted := append([]int64(nil), values...) + for i := 0; i < len(sorted); i++ { + for j := i + 1; j < len(sorted); j++ { + if sorted[j] < sorted[i] { + sorted[i], sorted[j] = sorted[j], sorted[i] + } + } + } + idx := int(float64(len(sorted)-1) * p) + return sorted[idx] +} + +type AgentSnapshot struct { + AgentIndex int `json:"agentIndex"` + Epoch int `json:"epoch,omitempty"` + Value int `json:"value"` + Confidence float64 `json:"confidence"` + Round int `json:"round"` + HasConsensus bool `json:"hasConsensus"` + ConvergedAtNs int64 `json:"convergedAtNs"` + ConsensusRound int `json:"consensusRound"` + FindingsEmitted int `json:"findingsEmitted"` + FindingsApplied int `json:"findingsApplied"` + AvgPropagationMs int64 `json:"avgPropagationMs"` + P95PropagationMs int64 `json:"p95PropagationMs"` +} + +func (e *Engine) Epoch() int { + e.mu.Lock() + defer e.mu.Unlock() + return e.epoch +} + +func (e *Engine) Snapshot() AgentSnapshot { + value, conf, round := e.LocalState() + avg, p95 := e.PropagationStats() + return AgentSnapshot{ + AgentIndex: e.cfg.AgentIndex, + Epoch: e.Epoch(), + Value: value, + Confidence: conf, + Round: round, + HasConsensus: e.HasLocalConsensus(), + ConvergedAtNs: e.ConvergedAtNs(), + ConsensusRound: e.ConsensusRound(), + FindingsEmitted: e.FindingsEmitted(), + FindingsApplied: e.FindingsApplied(), + AvgPropagationMs: avg, + P95PropagationMs: p95, + } +} + +func GlobalConsensus(snapshots []AgentSnapshot) (ok bool, value int) { + if len(snapshots) == 0 { + return false, 0 + } + if !snapshots[0].HasConsensus { + return false, 0 + } + v := snapshots[0].Value + for _, s := range snapshots[1:] { + if !s.HasConsensus || s.Value != v { + return false, 0 + } + } + return true, v +} + +func ValidateSnapshots(snapshots []AgentSnapshot, expected int) error { + if len(snapshots) != expected { + return fmt.Errorf("expected %d snapshots, got %d", expected, len(snapshots)) + } + return nil +} diff --git a/benchmarks/slim-vs-a2a/internal/metrics/metrics.go b/benchmarks/slim-vs-a2a/internal/metrics/metrics.go new file mode 100644 index 00000000..a8a30834 --- /dev/null +++ b/benchmarks/slim-vs-a2a/internal/metrics/metrics.go @@ -0,0 +1,128 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package metrics + +import ( + "encoding/csv" + "encoding/json" + "os" + "path/filepath" + "strconv" +) + +type RunResult struct { + ScenarioName string `json:"scenario_name"` + Domain string `json:"domain"` + Implementation string `json:"implementation"` + Agents int `json:"agents"` + ThinkTimeMs int64 `json:"think_time_ms"` + PayloadBytes int `json:"payload_bytes"` + ConsensusWallMS int64 `json:"consensus_wall_ms"` + ConsensusRound int `json:"consensus_round"` + FindingsEmitted int `json:"findings_emitted"` + FindingsReceivedTotal int `json:"findings_received_total"` + AvgPropagationMS int64 `json:"avg_propagation_ms"` + P95PropagationMS int64 `json:"p95_propagation_ms"` + LastAgentConvergeMS int64 `json:"last_agent_converge_ms"` + CoordFanoutMS int64 `json:"coord_fanout_ms"` + StreamRPCCount int `json:"stream_rpc_count"` + Epochs int `json:"epochs"` + EpochsSucceeded int `json:"epochs_succeeded"` + EpochsFailed int `json:"epochs_failed"` + Success bool `json:"success"` + Error string `json:"error,omitempty"` +} + +func WriteJSON(path string, result RunResult) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + data, err := json.MarshalIndent(result, "", " ") + if err != nil { + return err + } + return os.WriteFile(path, data, 0o644) +} + +func AppendTSV(path string, result RunResult) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + writeHeader := true + if info, err := os.Stat(path); err == nil { + writeHeader = info.Size() == 0 + } else if !os.IsNotExist(err) { + return err + } + + file, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + return err + } + defer file.Close() + + w := csv.NewWriter(file) + w.Comma = '\t' + if writeHeader { + if err := w.Write([]string{ + "scenario_name", "domain", "implementation", "agents", "think_time_ms", + "payload_bytes", + "consensus_wall_ms", "consensus_round", "findings_emitted", "findings_received_total", + "avg_propagation_ms", "p95_propagation_ms", "last_agent_converge_ms", + "coord_fanout_ms", "stream_rpc_count", + "epochs", "epochs_succeeded", "epochs_failed", + "success", "error", + }); err != nil { + return err + } + } + if err := w.Write([]string{ + result.ScenarioName, + result.Domain, + result.Implementation, + strconv.Itoa(result.Agents), + strconv.FormatInt(result.ThinkTimeMs, 10), + strconv.Itoa(result.PayloadBytes), + strconv.FormatInt(result.ConsensusWallMS, 10), + strconv.Itoa(result.ConsensusRound), + strconv.Itoa(result.FindingsEmitted), + strconv.Itoa(result.FindingsReceivedTotal), + strconv.FormatInt(result.AvgPropagationMS, 10), + strconv.FormatInt(result.P95PropagationMS, 10), + strconv.FormatInt(result.LastAgentConvergeMS, 10), + strconv.FormatInt(result.CoordFanoutMS, 10), + strconv.Itoa(result.StreamRPCCount), + strconv.Itoa(result.Epochs), + strconv.Itoa(result.EpochsSucceeded), + strconv.Itoa(result.EpochsFailed), + strconv.FormatBool(result.Success), + result.Error, + }); err != nil { + return err + } + w.Flush() + return w.Error() +} + +func AggregatePropagation(durations []int64) (avg, p95 int64) { + if len(durations) == 0 { + return 0, 0 + } + var sum int64 + for _, d := range durations { + sum += d + } + avg = sum / int64(len(durations)) + sorted := append([]int64(nil), durations...) + for i := 0; i < len(sorted); i++ { + for j := i + 1; j < len(sorted); j++ { + if sorted[j] < sorted[i] { + sorted[i], sorted[j] = sorted[j], sorted[i] + } + } + } + idx := int(float64(len(sorted)-1) * 0.95) + p95 = sorted[idx] + return avg, p95 +} diff --git a/benchmarks/slim-vs-a2a/internal/scenario/scenario.go b/benchmarks/slim-vs-a2a/internal/scenario/scenario.go new file mode 100644 index 00000000..3d5bf3dc --- /dev/null +++ b/benchmarks/slim-vs-a2a/internal/scenario/scenario.go @@ -0,0 +1,283 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package scenario + +import ( + "fmt" + "os" + "strings" + + "gopkg.in/yaml.v3" +) + +const ( + TargetModeMajority = "majority" +) + +type ConsensusScenario struct { + APIVersion string `yaml:"apiVersion"` + Kind string `yaml:"kind"` + Metadata Metadata `yaml:"metadata"` + Spec Spec `yaml:"spec"` + Agents []Agent `yaml:"agents"` +} + +type Metadata struct { + Name string `yaml:"name"` + Domain string `yaml:"domain"` + Description string `yaml:"description"` +} + +type Spec struct { + Agents int `yaml:"agents"` + ThinkTimeMs int64 `yaml:"thinkTimeMs"` + FindingEmitDelayMs int64 `yaml:"findingEmitDelayMs"` + MaxRounds int `yaml:"maxRounds"` + TargetMode string `yaml:"targetMode"` + Seed int64 `yaml:"seed"` + ValueSpace int `yaml:"valueSpace"` + // PayloadBytes inflates every finding with fixed-size padding to stress + // transport bandwidth. 0 (default) keeps findings at their minimal size. + PayloadBytes int `yaml:"payloadBytes,omitempty"` + // Epochs is how many times the same consensus attempt is repeated. Each + // epoch resets the agents and re-runs; the runner counts successful vs + // failed epochs. Defaults to 10. + Epochs int `yaml:"epochs,omitempty"` + // MaxEpochTimeMs is the per-epoch wall-clock budget: if global consensus is + // not reached within it, the epoch is counted as failed. Defaults to 120000. + MaxEpochTimeMs int64 `yaml:"maxEpochTimeMs,omitempty"` +} + +type Agent struct { + ID string `yaml:"id"` + SlimName string `yaml:"slimName"` + A2APort int `yaml:"a2aPort"` + CardPort int `yaml:"cardPort,omitempty"` + Role string `yaml:"role,omitempty"` +} + +func LoadFile(path string) (*ConsensusScenario, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var s ConsensusScenario + if err := yaml.Unmarshal(data, &s); err != nil { + return nil, err + } + if err := s.Validate(); err != nil { + return nil, err + } + return &s, nil +} + +func (s *ConsensusScenario) Validate() error { + if s.APIVersion != "bench.agntcy.io/v2" { + return fmt.Errorf("unsupported apiVersion %q", s.APIVersion) + } + if s.Kind != "ConsensusScenario" { + return fmt.Errorf("unsupported kind %q", s.Kind) + } + if s.Metadata.Name == "" { + return fmt.Errorf("metadata.name is required") + } + if s.Spec.Agents < 2 { + return fmt.Errorf("spec.agents must be >= 2") + } + if len(s.Agents) != s.Spec.Agents { + return fmt.Errorf("agents list length %d != spec.agents %d", len(s.Agents), s.Spec.Agents) + } + if s.Spec.ThinkTimeMs <= 0 { + return fmt.Errorf("spec.thinkTimeMs must be > 0") + } + if s.Spec.MaxRounds <= 0 { + return fmt.Errorf("spec.maxRounds must be > 0") + } + if s.Spec.TargetMode == "" { + s.Spec.TargetMode = TargetModeMajority + } + if s.Spec.ValueSpace <= 0 { + s.Spec.ValueSpace = 3 + } + if s.Spec.FindingEmitDelayMs <= 0 { + s.Spec.FindingEmitDelayMs = 1 + } + if s.Spec.PayloadBytes < 0 { + s.Spec.PayloadBytes = 0 + } + if s.Spec.Epochs <= 0 { + s.Spec.Epochs = 10 + } + if s.Spec.MaxEpochTimeMs <= 0 { + s.Spec.MaxEpochTimeMs = 120000 + } + for i, a := range s.Agents { + if a.ID == "" { + return fmt.Errorf("agents[%d].id is required", i) + } + if a.SlimName == "" { + return fmt.Errorf("agents[%d].slimName is required", i) + } + if a.A2APort <= 0 { + return fmt.Errorf("agents[%d].a2aPort is required", i) + } + } + return nil +} + +func (s *ConsensusScenario) CardPort(agent Agent) int { + if agent.CardPort > 0 { + return agent.CardPort + } + return agent.A2APort + 1000 +} + +func (s *ConsensusScenario) CardBaseURL(agent Agent) string { + return fmt.Sprintf("http://127.0.0.1:%d", s.CardPort(agent)) +} + +func (s *ConsensusScenario) AgentByID(id string) (Agent, bool) { + for _, a := range s.Agents { + if a.ID == id { + return a, true + } + } + return Agent{}, false +} + +func (s *ConsensusScenario) Coordinator() Agent { + for _, a := range s.Agents { + if a.Role == "coordinator" || a.ID == "agent-0" { + return a + } + } + return s.Agents[0] +} + +func (s *ConsensusScenario) WorkerAgents() []Agent { + coord := s.Coordinator() + var out []Agent + for _, a := range s.Agents { + if a.ID != coord.ID { + out = append(out, a) + } + } + return out +} + +func (s *ConsensusScenario) AgentIDs() []string { + ids := make([]string, len(s.Agents)) + for i, a := range s.Agents { + ids[i] = a.ID + } + return ids +} + +func (s *ConsensusScenario) SlimNames() []string { + names := make([]string, len(s.Agents)) + for i, a := range s.Agents { + names[i] = a.SlimName + } + return names +} + +type GenerateOptions struct { + Family string + Agents int + ThinkTimeMs int64 + Seed int64 + PayloadBytes int + Epochs int + MaxEpochTimeMs int64 +} + +func Generate(opts GenerateOptions) (*ConsensusScenario, error) { + if opts.Agents < 2 { + return nil, fmt.Errorf("agents must be >= 2") + } + if opts.ThinkTimeMs <= 0 { + opts.ThinkTimeMs = 10 + } + if opts.Family == "" { + opts.Family = "hypothesis-convergence" + } + if opts.Seed == 0 { + opts.Seed = 42 + } + if opts.PayloadBytes < 0 { + opts.PayloadBytes = 0 + } + if opts.Epochs <= 0 { + opts.Epochs = 10 + } + if opts.MaxEpochTimeMs <= 0 { + opts.MaxEpochTimeMs = 120000 + } + + name := fmt.Sprintf("%s-%dagents-%dms", opts.Family, opts.Agents, opts.ThinkTimeMs) + if opts.PayloadBytes > 0 { + name = fmt.Sprintf("%s-%db", name, opts.PayloadBytes) + } + agents := make([]Agent, opts.Agents) + for i := 0; i < opts.Agents; i++ { + id := fmt.Sprintf("agent-%d", i) + role := "worker" + if i == 0 { + role = "coordinator" + } + agents[i] = Agent{ + ID: id, + SlimName: fmt.Sprintf("agntcy/bench-v2/%s", id), + A2APort: 9700 + i*11, + Role: role, + } + } + + s := &ConsensusScenario{ + APIVersion: "bench.agntcy.io/v2", + Kind: "ConsensusScenario", + Metadata: Metadata{ + Name: name, + Domain: opts.Family, + Description: "Generated consensus scenario for transport sweeps", + }, + Spec: Spec{ + Agents: opts.Agents, + ThinkTimeMs: opts.ThinkTimeMs, + FindingEmitDelayMs: maxInt64(1, opts.ThinkTimeMs/10), + MaxRounds: 200, + TargetMode: TargetModeMajority, + Seed: opts.Seed, + ValueSpace: 3, + PayloadBytes: opts.PayloadBytes, + Epochs: opts.Epochs, + MaxEpochTimeMs: opts.MaxEpochTimeMs, + }, + Agents: agents, + } + return s, s.Validate() +} + +func Marshal(s *ConsensusScenario) ([]byte, error) { + return yaml.Marshal(s) +} + +func WriteFile(path string, s *ConsensusScenario) error { + data, err := Marshal(s) + if err != nil { + return err + } + return os.WriteFile(path, data, 0o644) +} + +func maxInt64(a, b int64) int64 { + if a > b { + return a + } + return b +} + +func NormalizeFamily(input string) string { + return strings.TrimSpace(input) +} diff --git a/benchmarks/slim-vs-a2a/internal/scenario/scenario_test.go b/benchmarks/slim-vs-a2a/internal/scenario/scenario_test.go new file mode 100644 index 00000000..3cb6c1a5 --- /dev/null +++ b/benchmarks/slim-vs-a2a/internal/scenario/scenario_test.go @@ -0,0 +1,23 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package scenario_test + +import ( + "testing" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" +) + +func TestGenerateAndValidate(t *testing.T) { + s, err := scenario.Generate(scenario.GenerateOptions{Agents: 5, ThinkTimeMs: 10}) + if err != nil { + t.Fatal(err) + } + if len(s.Agents) != 5 { + t.Fatalf("agents=%d", len(s.Agents)) + } + if s.Coordinator().ID != "agent-0" { + t.Fatalf("coordinator=%s", s.Coordinator().ID) + } +} diff --git a/benchmarks/slim-vs-a2a/plans/README.md b/benchmarks/slim-vs-a2a/plans/README.md new file mode 100644 index 00000000..699316d2 --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/README.md @@ -0,0 +1,116 @@ +# Consensus scenarios (v2) + +YAML `ConsensusScenario` plans for the SLIM vs A2A v2 consensus streaming benchmark. +Each scenario defines N agents that run a deterministic **distributed hypothesis convergence** +workload: agents think in parallel, emit findings, and must reach identical local consensus. + +## Topologies compared + +The two implementations highlight one structural difference: + +- **SLIM (`slim-group-session`)** — agents join a native SLIM group session and + broadcast findings peer-to-peer; the dataplane fans them out. **There is no + application relay.** The runner only moderates (creates the session, invites + agents, sends one start) and then passively observes snapshots that agents + push to it on convergence. +- **A2A (`a2a-relay-stream`)** — A2A has no peer multicast, so the runner is an + explicit **relay hub**. Two server-streaming legs carry findings: agents stream + findings to the runner (`OpStreamFindings`) and subscribe to the runner's + relayed-finding stream (`OpStreamRelay`). Every finding therefore makes an + extra relay hop, which is the overhead `consensus_wall_ms` is expected to show. + +## Schema + +```yaml +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-10agents-10ms + domain: hypothesis-convergence +spec: + agents: 10 + thinkTimeMs: 10 + findingEmitDelayMs: 1 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 + payloadBytes: 0 # optional fixed-size padding added to every finding + epochs: 10 # consensus attempts per run (default 10) + maxEpochTimeMs: 120000 # per-epoch consensus budget; exceeding it fails the epoch (default 120000) +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: worker # legacy field; both transports now treat all agents as peers + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker +``` + +## Generate sweep scenarios + +```bash +go run ./tools/gen_scenario \ + -family hypothesis-convergence \ + -agents 10 \ + -think-ms 20 \ + -payload-bytes 0 \ + -epochs 10 \ + -max-epoch-ms 120000 \ + -output plans/sweeps/hypothesis-convergence-10ag-20ms.yaml +``` + +### Epochs (reliability knob) + +Each run repeats the *same* consensus attempt over `spec.epochs` epochs. An epoch +succeeds if every agent reaches global consensus within `spec.maxEpochTimeMs` +(each attempt still runs up to `maxRounds`); otherwise it is counted as a failed +epoch. Between epochs the agents reset their engine in place and re-run — the SLIM +group session and A2A relay subscriptions stay up for the whole run, so there is a +single teardown. The runner reports `epochs`, `epochs_succeeded`, and +`epochs_failed`; overall `success` is true only when no epoch failed. This +surfaces reliability differences under load: A2A may miss the per-epoch budget +while SLIM still converges. + +Sweeps expose two variables (defaults shown): `SWEEP_EPOCHS=10` and +`SWEEP_MAX_EPOCH_MS=30000` (a modest budget keeps failed A2A epochs capping +quickly). For example: + +```bash +task compare:sweep:payload \ + SWEEP_AGENTS=10,50 SWEEP_THINK_MS=5 SWEEP_PAYLOAD_BYTES=0,10240 \ + SWEEP_EPOCHS=10 SWEEP_MAX_EPOCH_MS=30000 +``` + +### Payload size (transport-stress) knob + +`spec.payloadBytes` adds fixed-size deterministic padding to every finding. It is +semantically inert (it does not affect the consensus math or the number of +rounds) — it only inflates the wire size of each finding. Because A2A relays each +finding twice through the central runner (`agent → runner → N−1 agents`) while +SLIM publishes once and the dataplane fans out, larger payloads disproportionately +load the A2A relay. Combine it with high agent counts and a short think window to +make the multicast advantage most visible: + +```bash +# Sweep agents × payload at a 5ms think window (0 vs 10kB), then build the report. +task compare:sweep:payload \ + SWEEP_AGENTS=10,50 SWEEP_THINK_MS=5 SWEEP_PAYLOAD_BYTES=0,10240 +``` + +> Caveat: a2a-go's streaming task store appends each relayed finding to the task +> history and deep-copies the task per update, so very large payloads also amplify +> A2A cost super-linearly. That is a property of the A2A streaming/task model, but +> note it when attributing the delta. + +## Run comparison + +```bash +task build +task compare:plan PLAN=hypothesis-convergence-5ag-20ms +task compare:report +``` + +Primary metric: `consensus_wall_ms` in `reports/results.tsv`. diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms-5120b.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms-5120b.yaml new file mode 100644 index 00000000..185c7e51 --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms-5120b.yaml @@ -0,0 +1,58 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-10agents-20ms-5120b + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 10 + thinkTimeMs: 20 + findingEmitDelayMs: 2 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 + payloadBytes: 5120 + epochs: 10 + maxEpochTimeMs: 30000 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker + - id: agent-5 + slimName: agntcy/bench-v2/agent-5 + a2aPort: 9755 + role: worker + - id: agent-6 + slimName: agntcy/bench-v2/agent-6 + a2aPort: 9766 + role: worker + - id: agent-7 + slimName: agntcy/bench-v2/agent-7 + a2aPort: 9777 + role: worker + - id: agent-8 + slimName: agntcy/bench-v2/agent-8 + a2aPort: 9788 + role: worker + - id: agent-9 + slimName: agntcy/bench-v2/agent-9 + a2aPort: 9799 + role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml new file mode 100644 index 00000000..0f6ee819 --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-10ag-20ms.yaml @@ -0,0 +1,57 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-10agents-20ms + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 10 + thinkTimeMs: 20 + findingEmitDelayMs: 2 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 + epochs: 10 + maxEpochTimeMs: 30000 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker + - id: agent-5 + slimName: agntcy/bench-v2/agent-5 + a2aPort: 9755 + role: worker + - id: agent-6 + slimName: agntcy/bench-v2/agent-6 + a2aPort: 9766 + role: worker + - id: agent-7 + slimName: agntcy/bench-v2/agent-7 + a2aPort: 9777 + role: worker + - id: agent-8 + slimName: agntcy/bench-v2/agent-8 + a2aPort: 9788 + role: worker + - id: agent-9 + slimName: agntcy/bench-v2/agent-9 + a2aPort: 9799 + role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-10ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-10ms.yaml new file mode 100644 index 00000000..679bc25b --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-10ms.yaml @@ -0,0 +1,37 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-5agents-10ms + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 5 + thinkTimeMs: 10 + findingEmitDelayMs: 1 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 + epochs: 10 + maxEpochTimeMs: 30000 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms-5120b.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms-5120b.yaml new file mode 100644 index 00000000..512982df --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms-5120b.yaml @@ -0,0 +1,38 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-5agents-20ms-5120b + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 5 + thinkTimeMs: 20 + findingEmitDelayMs: 2 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 + payloadBytes: 5120 + epochs: 10 + maxEpochTimeMs: 30000 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml new file mode 100644 index 00000000..25ed9840 --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-20ms.yaml @@ -0,0 +1,37 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-5agents-20ms + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 5 + thinkTimeMs: 20 + findingEmitDelayMs: 2 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 + epochs: 10 + maxEpochTimeMs: 30000 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker diff --git a/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-30ms.yaml b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-30ms.yaml new file mode 100644 index 00000000..31d25d43 --- /dev/null +++ b/benchmarks/slim-vs-a2a/plans/sweeps/hypothesis-convergence-5ag-30ms.yaml @@ -0,0 +1,37 @@ +apiVersion: bench.agntcy.io/v2 +kind: ConsensusScenario +metadata: + name: hypothesis-convergence-5agents-30ms + domain: hypothesis-convergence + description: Generated consensus scenario for transport sweeps +spec: + agents: 5 + thinkTimeMs: 30 + findingEmitDelayMs: 3 + maxRounds: 200 + targetMode: majority + seed: 42 + valueSpace: 3 + epochs: 10 + maxEpochTimeMs: 30000 +agents: + - id: agent-0 + slimName: agntcy/bench-v2/agent-0 + a2aPort: 9700 + role: coordinator + - id: agent-1 + slimName: agntcy/bench-v2/agent-1 + a2aPort: 9711 + role: worker + - id: agent-2 + slimName: agntcy/bench-v2/agent-2 + a2aPort: 9722 + role: worker + - id: agent-3 + slimName: agntcy/bench-v2/agent-3 + a2aPort: 9733 + role: worker + - id: agent-4 + slimName: agntcy/bench-v2/agent-4 + a2aPort: 9744 + role: worker diff --git a/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go b/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go new file mode 100644 index 00000000..3bd162bf --- /dev/null +++ b/benchmarks/slim-vs-a2a/slim/cmd/agent/main.go @@ -0,0 +1,59 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "flag" + "fmt" + "log" + "time" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/agent" +) + +func main() { + slimName := flag.String("slim-name", "", "SLIM identity org/group/app") + endpoint := flag.String("endpoint", "http://127.0.0.1:46357", "SLIM dataplane endpoint") + scenarioFile := flag.String("scenario-file", "", "path to consensus scenario yaml") + agentIndex := flag.Int("agent-index", 0, "agent index in scenario") + quiet := flag.Bool("quiet", false, "disable benchmark logs") + flag.Parse() + + if *quiet { + benchlog.SetEnabled(false) + } + + if *slimName == "" || *scenarioFile == "" { + log.Fatal("--slim-name and --scenario-file are required") + } + + s, err := scenario.LoadFile(*scenarioFile) + if err != nil { + log.Fatalf("load scenario: %v", err) + } + if *agentIndex < 0 || *agentIndex >= len(s.Agents) { + log.Fatalf("agent-index out of range") + } + + rt := agent.NewRuntime(s, *agentIndex, *slimName, *endpoint) + if err := rt.Setup(); err != nil { + log.Fatalf("setup: %v", err) + } + defer rt.Close() + + if !*quiet { + fmt.Printf("SLIM_AGENT_READY name=%s index=%d scenario=%s\n", *slimName, *agentIndex, s.Metadata.Name) + } + + // Block until the moderator invites us into the group session. + if err := rt.Join(60 * time.Second); err != nil { + log.Fatalf("join group session: %v", err) + } + + if err := rt.Run(); err != nil { + log.Printf("receive loop ended: %v", err) + } +} diff --git a/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go b/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go new file mode 100644 index 00000000..c31ff588 --- /dev/null +++ b/benchmarks/slim-vs-a2a/slim/cmd/runner/main.go @@ -0,0 +1,367 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Command runner is the SLIM benchmark driver. It acts as a group-session +// moderator (creates the session, invites every agent, broadcasts a single +// start) and then becomes a passive observer: it never relays findings. Agents +// broadcast findings to each other directly over the SLIM dataplane and push +// their snapshots to the runner the instant they converge. Global consensus is +// detected event-driven from those pushed snapshots. +package main + +import ( + "encoding/json" + "flag" + "fmt" + "log" + "os" + "os/exec" + "strings" + "time" + + slim "github.com/agntcy/slim-bindings-go" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/protocol" +) + +const ( + runnerSlimName = "agntcy/bench-v2/runner" + groupChannelName = "agntcy/bench-v2/consensus" + defaultSharedSecret = "demo-shared-secret-min-32-chars!!" +) + +func main() { + scenarioPath := flag.String("scenario", "", "path to consensus scenario yaml") + endpoint := flag.String("endpoint", "http://127.0.0.1:46357", "SLIM dataplane endpoint") + agentBin := flag.String("agent-bin", "", "path to slim-agent binary") + outputJSON := flag.String("output-json", "", "write run metrics json") + outputTSV := flag.String("output-tsv", "", "append run metrics tsv") + waitReady := flag.Duration("wait-ready", 3*time.Second, "wait for agents to start") + quiet := flag.Bool("quiet", false, "disable benchmark logs") + flag.Parse() + + if *quiet { + benchlog.SetEnabled(false) + } + if *scenarioPath == "" { + log.Fatal("--scenario is required") + } + + s, err := scenario.LoadFile(*scenarioPath) + if err != nil { + log.Fatalf("load scenario: %v", err) + } + + agentPath := *agentBin + if agentPath == "" { + agentPath = os.Getenv("SLIM_AGENT_BIN") + } + if agentPath == "" { + log.Fatal("set --agent-bin or SLIM_AGENT_BIN") + } + + procs := startAgents(s, agentPath, *endpoint, *scenarioPath, *quiet) + defer stopAgents(procs) + + mod, err := newModerator(*endpoint, s) + if err != nil { + log.Fatalf("moderator: %v", err) + } + defer mod.Close() + + // Let agents come up and reach ListenForSessionAsync before inviting. + time.Sleep(*waitReady) + if err := mod.InviteAll(); err != nil { + log.Fatalf("invite agents: %v", err) + } + + runStart := time.Now() + benchlog.SetRunStart(runStart) + + result := metrics.RunResult{ + ScenarioName: s.Metadata.Name, + Domain: s.Metadata.Domain, + Implementation: benchlog.ImplSLIM, + Agents: len(s.Agents), + ThinkTimeMs: s.Spec.ThinkTimeMs, + PayloadBytes: s.Spec.PayloadBytes, + Epochs: s.Spec.Epochs, + } + + n := len(s.Agents) + maxEpochTime := time.Duration(s.Spec.MaxEpochTimeMs) * time.Millisecond + var ( + wallSamples []int64 + allSnapshots []consensus.AgentSnapshot + ) + for epoch := 0; epoch < s.Spec.Epochs; epoch++ { + log.Printf("epoch %d/%d started (budget %s)", epoch+1, s.Spec.Epochs, maxEpochTime) + if err := mod.Start(epoch); err != nil { + log.Fatalf("broadcast start (epoch %d): %v", epoch, err) + } + epochStart := time.Now() + latest := map[int]consensus.AgentSnapshot{} + deadline := epochStart.Add(maxEpochTime) + success := false + for time.Now().Before(deadline) { + timeout := time.Second + msg, err := mod.session.GetMessageAsync(&timeout) + if err != nil { + if isTimeout(err) { + continue + } + break + } + env, derr := protocol.Decode(msg.Payload) + if derr != nil || env.Kind != protocol.KindSnapshot || env.Snapshot == nil { + continue + } + // Ignore stragglers from other epochs. + if env.Snapshot.Epoch != epoch { + continue + } + latest[env.Snapshot.AgentIndex] = *env.Snapshot + if len(latest) == n { + if ok, _ := consensus.GlobalConsensus(snapshotSlice(latest)); ok { + success = true + break + } + } + } + allSnapshots = append(allSnapshots, snapshotSlice(latest)...) + epochWall := time.Since(epochStart) + if success { + result.EpochsSucceeded++ + wallSamples = append(wallSamples, epochWall.Milliseconds()) + log.Printf("epoch %d/%d ok wall_ms=%d", epoch+1, s.Spec.Epochs, epochWall.Milliseconds()) + } else { + result.EpochsFailed++ + log.Printf("epoch %d/%d failed after %s (budget %s exhausted)", epoch+1, s.Spec.Epochs, epochWall.Round(time.Millisecond), maxEpochTime) + } + } + + result.Success = result.EpochsFailed == 0 + result = aggregateResult(result, wallSamples, allSnapshots) + + stopAgents(procs) + + if *outputJSON != "" { + if err := metrics.WriteJSON(*outputJSON, result); err != nil { + log.Fatalf("write json: %v", err) + } + } + if *outputTSV != "" { + if err := metrics.AppendTSV(*outputTSV, result); err != nil { + log.Fatalf("write tsv: %v", err) + } + } + + data, _ := json.MarshalIndent(result, "", " ") + fmt.Println(string(data)) + if !result.Success { + os.Exit(1) + } +} + +// moderator owns the group session: it creates it, invites agents, and +// broadcasts the start signal. It does not relay any traffic. +type moderator struct { + app *slim.App + connID uint64 + session *slim.Session + agents []scenario.Agent +} + +func newModerator(endpoint string, s *scenario.ConsensusScenario) (*moderator, error) { + slim.InitializeWithDefaults() + service := slim.GetGlobalService() + + name, err := slim.NameFromString(runnerSlimName) + if err != nil { + return nil, err + } + app, err := service.CreateAppWithSecret(name, defaultSharedSecret) + if err != nil { + return nil, err + } + connID, err := service.Connect(slim.NewInsecureClientConfig(endpoint)) + if err != nil { + app.Destroy() + return nil, err + } + if err := app.Subscribe(name, &connID); err != nil { + app.Destroy() + return nil, err + } + + channelName, err := slim.NameFromString(groupChannelName) + if err != nil { + app.Destroy() + return nil, err + } + interval := 5 * time.Second + maxRetries := uint32(5) + config := slim.SessionConfig{ + SessionType: slim.SessionTypeGroup, + MaxRetries: &maxRetries, + Interval: &interval, + Metadata: map[string]string{}, + } + session, err := app.CreateSessionAndWaitAsync(config, channelName) + if err != nil { + app.Destroy() + return nil, err + } + // Give the session a moment to establish before inviting. + time.Sleep(100 * time.Millisecond) + + return &moderator{app: app, connID: connID, session: session, agents: s.Agents}, nil +} + +func (m *moderator) InviteAll() error { + for _, a := range m.agents { + name, err := slim.NameFromString(a.SlimName) + if err != nil { + return err + } + if err := m.app.SetRouteAsync(name, m.connID); err != nil { + return fmt.Errorf("set route %s: %w", a.SlimName, err) + } + if err := m.session.InviteAndWaitAsync(name); err != nil { + return fmt.Errorf("invite %s: %w", a.SlimName, err) + } + } + return nil +} + +func (m *moderator) Start(epoch int) error { + payload, err := protocol.Encode(protocol.Envelope{Kind: protocol.KindStart, Epoch: epoch}) + if err != nil { + return err + } + return m.session.PublishAndWaitAsync(payload, nil, nil) +} + +func (m *moderator) Close() { + if m.session != nil && m.app != nil { + _ = m.app.DeleteSessionAndWaitAsync(m.session) + } + if m.app != nil { + m.app.Destroy() + } +} + +// aggregateResult folds the snapshots collected across every epoch into the run +// result. wallSamples holds the wall-clock duration of each successful epoch. +func aggregateResult(result metrics.RunResult, wallSamples []int64, snapshots []consensus.AgentSnapshot) metrics.RunResult { + if len(snapshots) == 0 { + result.Error = "no agent snapshots" + return result + } + + var ( + totalEmitted int + totalApplied int + propDurations []int64 + maxConsensusRound int + maxRound int + ) + for _, snap := range snapshots { + totalEmitted += snap.FindingsEmitted + totalApplied += snap.FindingsApplied + if snap.ConsensusRound > maxConsensusRound { + maxConsensusRound = snap.ConsensusRound + } + if snap.Round > maxRound { + maxRound = snap.Round + } + if snap.AvgPropagationMs > 0 { + propDurations = append(propDurations, snap.AvgPropagationMs) + } + } + + meanWall, maxWall := summarizeWall(wallSamples) + result.ConsensusWallMS = meanWall + result.LastAgentConvergeMS = maxWall + if result.EpochsSucceeded == 0 { + result.Error = "consensus not reached in any epoch" + } + + result.ConsensusRound = maxConsensusRound + if result.ConsensusRound == 0 { + result.ConsensusRound = maxRound + } + result.FindingsEmitted = totalEmitted + result.FindingsReceivedTotal = totalApplied + result.AvgPropagationMS, result.P95PropagationMS = metrics.AggregatePropagation(propDurations) + // Native group session: findings are broadcast peer-to-peer, no relay. + result.StreamRPCCount = totalEmitted + result.CoordFanoutMS = 0 + return result +} + +// summarizeWall returns the mean and max of the per-epoch wall-clock samples. +func summarizeWall(wallSamples []int64) (mean, max int64) { + if len(wallSamples) == 0 { + return 0, 0 + } + var sum int64 + for _, w := range wallSamples { + sum += w + if w > max { + max = w + } + } + return sum / int64(len(wallSamples)), max +} + +func snapshotSlice(m map[int]consensus.AgentSnapshot) []consensus.AgentSnapshot { + out := make([]consensus.AgentSnapshot, 0, len(m)) + for _, snap := range m { + out = append(out, snap) + } + return out +} + +func startAgents(s *scenario.ConsensusScenario, agentBin, endpoint, scenarioFile string, quiet bool) []*exec.Cmd { + var procs []*exec.Cmd + for i, agent := range s.Agents { + args := []string{ + "--slim-name", agent.SlimName, + "--endpoint", endpoint, + "--scenario-file", scenarioFile, + "--agent-index", fmt.Sprintf("%d", i), + } + if quiet { + args = append(args, "--quiet") + } + cmd := exec.Command(agentBin, args...) + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + if err := cmd.Start(); err != nil { + stopAgents(procs) + log.Fatalf("start agent %s: %v", agent.ID, err) + } + procs = append(procs, cmd) + } + return procs +} + +func stopAgents(procs []*exec.Cmd) { + for _, cmd := range procs { + if cmd.Process != nil { + _ = cmd.Process.Kill() + } + } +} + +func isTimeout(err error) bool { + if err == nil { + return false + } + msg := strings.ToLower(err.Error()) + return strings.Contains(msg, "timed out") || strings.Contains(msg, "timeout") +} diff --git a/benchmarks/slim-vs-a2a/slim/internal/agent/runtime.go b/benchmarks/slim-vs-a2a/slim/internal/agent/runtime.go new file mode 100644 index 00000000..4c53f4b3 --- /dev/null +++ b/benchmarks/slim-vs-a2a/slim/internal/agent/runtime.go @@ -0,0 +1,271 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Package agent implements a consensus agent that communicates over a native +// SLIM group session. Unlike the slimrpc-based design, there is NO application +// relay: every agent broadcasts its findings to all peers via PublishAndWait, +// and the SLIM dataplane fans them out. The runner is only a moderator that +// creates the session and a passive observer that consumes snapshots agents +// push to it the moment they converge. +package agent + +import ( + "fmt" + "strings" + "sync" + "time" + + slim "github.com/agntcy/slim-bindings-go" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/benchlog" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/slim/internal/protocol" +) + +const defaultSharedSecret = "demo-shared-secret-min-32-chars!!" + +type Runtime struct { + scenario *scenario.ConsensusScenario + agentIndex int + slimName string + endpoint string + + engine *consensus.Engine + app *slim.App + connID uint64 + session *slim.Session + + mu sync.Mutex + runnerCtx *slim.MessageContext + lastPush time.Time + // lastStartedEpoch dedupes repeated KindStart for the same epoch; + // currentEpoch is the attempt currently running (used to drop stale peer + // findings and to stop a superseded epoch loop). + lastStartedEpoch int + currentEpoch int + + pubMu sync.Mutex // serializes publishes on the session + + startedAt time.Time +} + +func NewRuntime(s *scenario.ConsensusScenario, agentIndex int, slimName, endpoint string) *Runtime { + return &Runtime{ + scenario: s, + agentIndex: agentIndex, + slimName: slimName, + endpoint: endpoint, + engine: consensus.NewEngine(s.Spec, agentIndex), + lastStartedEpoch: -1, + currentEpoch: -1, + } +} + +// Setup creates the SLIM app, connects to the dataplane, and subscribes to the +// agent's own name so the moderator can route invitations to it. +func (r *Runtime) Setup() error { + slim.InitializeWithDefaults() + service := slim.GetGlobalService() + + name, err := slim.NameFromString(r.slimName) + if err != nil { + return err + } + + app, err := service.CreateAppWithSecret(name, defaultSharedSecret) + if err != nil { + return err + } + r.app = app + + connID, err := service.Connect(slim.NewInsecureClientConfig(r.endpoint)) + if err != nil { + app.Destroy() + return err + } + r.connID = connID + if err := app.Subscribe(name, &connID); err != nil { + app.Destroy() + return err + } + return nil +} + +// Join blocks until the moderator invites this agent into the group session. +func (r *Runtime) Join(timeout time.Duration) error { + session, err := r.app.ListenForSessionAsync(&timeout) + if err != nil { + return fmt.Errorf("listen for session: %w", err) + } + r.session = session + return nil +} + +// Run is the single receive loop on the group session. It dispatches by +// envelope kind: start launches the think loop, finding is applied locally. +// It blocks until the session ends (the runner kills the process at shutdown). +func (r *Runtime) Run() error { + for { + timeout := time.Second + msg, err := r.session.GetMessageAsync(&timeout) + if err != nil { + if isTimeout(err) { + continue + } + return err + } + env, derr := protocol.Decode(msg.Payload) + if derr != nil { + continue + } + switch env.Kind { + case protocol.KindStart: + ctx := msg.Context + r.startEpoch(env.Epoch, &ctx) + case protocol.KindFinding: + if env.Finding == nil || env.Finding.AgentIndex == r.agentIndex { + continue + } + // Drop stragglers from a different (usually previous) epoch. + r.mu.Lock() + cur := r.currentEpoch + r.mu.Unlock() + if env.Finding.Epoch != cur { + continue + } + r.applyFinding(*env.Finding) + // Keep the runner's view fresh if we already converged. + r.maybePushSnapshot(false) + } + } +} + +// startEpoch resets the engine for a fresh attempt and launches the epoch loop. +// Repeated starts for the same (or older) epoch are ignored so a duplicated +// broadcast cannot double-run an attempt. +func (r *Runtime) startEpoch(epoch int, ctx *slim.MessageContext) { + r.mu.Lock() + if epoch <= r.lastStartedEpoch { + r.mu.Unlock() + return + } + r.lastStartedEpoch = epoch + r.currentEpoch = epoch + r.runnerCtx = ctx + r.mu.Unlock() + + r.engine.Reset(epoch) + r.startedAt = time.Now() + benchlog.SetRunStart(r.startedAt) + go r.runEpoch(epoch) +} + +func (r *Runtime) runEpoch(epoch int) { + spec := r.scenario.Spec + think := time.Duration(spec.ThinkTimeMs) * time.Millisecond + emitGap := time.Duration(spec.FindingEmitDelayMs) * time.Millisecond + + for round := 0; round < spec.MaxRounds; round++ { + // Stop early if a newer epoch has superseded this one. + r.mu.Lock() + superseded := r.currentEpoch != epoch + r.mu.Unlock() + if superseded { + return + } + + time.Sleep(think) + + finding, emit := r.engine.Think() + if emit && finding != nil { + if err := r.publishFinding(*finding); err != nil { + benchlog.Finding(benchlog.ImplSLIM, "publish_error", r.agentIndex, finding.FindingID, + fmt.Sprintf("err=%v", err)) + } else { + benchlog.Finding(benchlog.ImplSLIM, "published", r.agentIndex, finding.FindingID) + } + time.Sleep(emitGap) + } + + if r.engine.HasLocalConsensus() { + r.maybePushSnapshot(true) + return + } + } + // Reached max rounds without consensus: still report final state. + r.maybePushSnapshot(true) +} + +// publishFinding broadcasts a finding to every group peer. The SLIM dataplane +// performs the fan-out; there is no relay. +func (r *Runtime) publishFinding(f consensus.Finding) error { + payload, err := protocol.Encode(protocol.Envelope{ + Kind: protocol.KindFinding, + AgentIndex: r.agentIndex, + Finding: &f, + }) + if err != nil { + return err + } + r.pubMu.Lock() + defer r.pubMu.Unlock() + return r.session.PublishAndWaitAsync(payload, nil, nil) +} + +// maybePushSnapshot sends the current snapshot directly to the runner (targeted, +// off the peer broadcast path). force=true always sends; otherwise it is +// throttled so post-convergence finding updates do not spam the runner. +func (r *Runtime) maybePushSnapshot(force bool) { + r.mu.Lock() + ctx := r.runnerCtx + if ctx == nil { + r.mu.Unlock() + return + } + if !force && time.Since(r.lastPush) < 50*time.Millisecond { + r.mu.Unlock() + return + } + r.lastPush = time.Now() + r.mu.Unlock() + + snap := r.engine.Snapshot() + payload, err := protocol.Encode(protocol.Envelope{ + Kind: protocol.KindSnapshot, + AgentIndex: r.agentIndex, + Snapshot: &snap, + }) + if err != nil { + return + } + r.pubMu.Lock() + defer r.pubMu.Unlock() + _ = r.session.PublishToAndWaitAsync(*ctx, payload, nil, nil) +} + +func (r *Runtime) applyFinding(f consensus.Finding) { + recvNs := time.Now().UnixNano() + r.engine.ApplyFinding(f) + if f.EmittedAt > 0 { + r.engine.RecordPropagation(f.EmittedAt, recvNs) + } + benchlog.Finding(benchlog.ImplSLIM, "received", r.agentIndex, f.FindingID, + fmt.Sprintf("from=%d", f.AgentIndex)) +} + +func (r *Runtime) Close() { + if r.session != nil && r.app != nil { + _ = r.app.DeleteSessionAndWaitAsync(r.session) + } + if r.app != nil { + r.app.Destroy() + } +} + +func isTimeout(err error) bool { + if err == nil { + return false + } + msg := strings.ToLower(err.Error()) + return strings.Contains(msg, "timed out") || strings.Contains(msg, "timeout") +} diff --git a/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go b/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go new file mode 100644 index 00000000..d6343700 --- /dev/null +++ b/benchmarks/slim-vs-a2a/slim/internal/protocol/protocol.go @@ -0,0 +1,43 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +// Package protocol defines the message envelope exchanged over the native SLIM +// group session. There is no relay and no request/response RPC: the runner +// broadcasts a single start, agents broadcast findings to all peers, and agents +// push their snapshots to the runner the instant they converge. +package protocol + +import ( + "encoding/json" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/consensus" +) + +const ( + // KindStart is broadcast once by the runner to define t0 and unblock agents. + KindStart = "start" + // KindFinding is broadcast by an agent to all group peers. + KindFinding = "finding" + // KindSnapshot is pushed by an agent directly to the runner on convergence. + KindSnapshot = "snapshot" +) + +// Envelope is the single JSON message type carried on the group session. +type Envelope struct { + Kind string `json:"kind"` + AgentIndex int `json:"agentIndex,omitempty"` + // Epoch identifies which consensus attempt a KindStart begins. + Epoch int `json:"epoch,omitempty"` + Finding *consensus.Finding `json:"finding,omitempty"` + Snapshot *consensus.AgentSnapshot `json:"snapshot,omitempty"` +} + +func Encode(e Envelope) ([]byte, error) { + return json.Marshal(e) +} + +func Decode(data []byte) (Envelope, error) { + var e Envelope + err := json.Unmarshal(data, &e) + return e, err +} diff --git a/benchmarks/slim-vs-a2a/tools/gen_scenario/main.go b/benchmarks/slim-vs-a2a/tools/gen_scenario/main.go new file mode 100644 index 00000000..38a4b2d9 --- /dev/null +++ b/benchmarks/slim-vs-a2a/tools/gen_scenario/main.go @@ -0,0 +1,51 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "flag" + "fmt" + "log" + "os" + "path/filepath" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" +) + +func main() { + family := flag.String("family", "hypothesis-convergence", "scenario family name") + agents := flag.Int("agents", 10, "number of agents") + thinkMS := flag.Int64("think-ms", 10, "think time per agent in ms") + seed := flag.Int64("seed", 42, "random seed") + payloadBytes := flag.Int("payload-bytes", 0, "fixed-size padding added to every finding (bytes)") + epochs := flag.Int("epochs", 0, "number of consensus epochs to run (default 10)") + maxEpochMS := flag.Int64("max-epoch-ms", 0, "per-epoch consensus time budget in ms (default 120000)") + output := flag.String("output", "", "output yaml path") + flag.Parse() + + if *output == "" { + log.Fatal("-output is required") + } + + s, err := scenario.Generate(scenario.GenerateOptions{ + Family: *family, + Agents: *agents, + ThinkTimeMs: *thinkMS, + Seed: *seed, + PayloadBytes: *payloadBytes, + Epochs: *epochs, + MaxEpochTimeMs: *maxEpochMS, + }) + if err != nil { + log.Fatalf("generate: %v", err) + } + + if err := os.MkdirAll(filepath.Dir(*output), 0o755); err != nil { + log.Fatalf("mkdir: %v", err) + } + if err := scenario.WriteFile(*output, s); err != nil { + log.Fatalf("write: %v", err) + } + fmt.Printf("wrote %s\n", *output) +} diff --git a/benchmarks/slim-vs-a2a/tools/report/main.go b/benchmarks/slim-vs-a2a/tools/report/main.go new file mode 100644 index 00000000..6cac8e48 --- /dev/null +++ b/benchmarks/slim-vs-a2a/tools/report/main.go @@ -0,0 +1,302 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/csv" + "flag" + "fmt" + "html/template" + "os" + "path/filepath" + "strconv" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/metrics" +) + +type scenarioComparison struct { + ScenarioName string + A2A metrics.RunResult + SLIM metrics.RunResult + HasA2A bool + HasSLIM bool +} + +func main() { + tsvPath := flag.String("tsv", "./reports/results.tsv", "comparison results tsv") + sweepTSV := flag.String("sweep-tsv", "./reports/sweep.tsv", "optional sweep results tsv") + output := flag.String("output", "./reports/index.html", "html dashboard output") + flag.Parse() + + results, err := readTSV(*tsvPath) + if err != nil { + fmt.Fprintf(os.Stderr, "read tsv: %v\n", err) + os.Exit(1) + } + comparisons := groupByScenario(results) + + var sweepResults []metrics.RunResult + if *sweepTSV != "" { + if sr, err := readTSV(*sweepTSV); err == nil { + sweepResults = sr + } + } + + if err := writeHTML(*output, comparisons, sweepResults); err != nil { + fmt.Fprintf(os.Stderr, "write html: %v\n", err) + os.Exit(1) + } + fmt.Printf("wrote %s\n", *output) +} + +func readTSV(path string) ([]metrics.RunResult, error) { + file, err := os.Open(path) + if err != nil { + return nil, err + } + defer file.Close() + + reader := csv.NewReader(file) + reader.Comma = '\t' + records, err := reader.ReadAll() + if err != nil { + return nil, err + } + if len(records) <= 1 { + return nil, fmt.Errorf("no data rows in %s", path) + } + + var out []metrics.RunResult + for _, row := range records[1:] { + if len(row) < 19 { + continue + } + r := metrics.RunResult{ + ScenarioName: row[0], + Domain: row[1], + Implementation: row[2], + } + r.Agents = atoi(row[3]) + r.ThinkTimeMs = atoi64(row[4]) + r.PayloadBytes = atoi(row[5]) + r.ConsensusWallMS = atoi64(row[6]) + r.ConsensusRound = atoi(row[7]) + r.FindingsEmitted = atoi(row[8]) + r.FindingsReceivedTotal = atoi(row[9]) + r.AvgPropagationMS = atoi64(row[10]) + r.P95PropagationMS = atoi64(row[11]) + r.LastAgentConvergeMS = atoi64(row[12]) + r.CoordFanoutMS = atoi64(row[13]) + r.StreamRPCCount = atoi(row[14]) + r.Epochs = atoi(row[15]) + r.EpochsSucceeded = atoi(row[16]) + r.EpochsFailed = atoi(row[17]) + r.Success = row[18] == "true" + if len(row) > 19 { + r.Error = row[19] + } + out = append(out, r) + } + return out, nil +} + +func groupByScenario(results []metrics.RunResult) []scenarioComparison { + byScenario := map[string]*scenarioComparison{} + for _, r := range results { + sc, ok := byScenario[r.ScenarioName] + if !ok { + sc = &scenarioComparison{ScenarioName: r.ScenarioName} + byScenario[r.ScenarioName] = sc + } + switch r.Implementation { + case "a2a-relay-stream": + sc.A2A = r + sc.HasA2A = true + case "slim-group-session": + sc.SLIM = r + sc.HasSLIM = true + } + } + out := make([]scenarioComparison, 0, len(byScenario)) + for _, sc := range byScenario { + out = append(out, *sc) + } + return out +} + +type reportData struct { + Comparisons []scenarioComparison + Sweep []metrics.RunResult +} + +func writeHTML(path string, comparisons []scenarioComparison, sweep []metrics.RunResult) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + tmpl := template.Must(template.New("report").Funcs(template.FuncMap{ + "deltaPct": deltaPct, + }).Parse(htmlTemplate)) + file, err := os.Create(path) + if err != nil { + return err + } + defer file.Close() + return tmpl.Execute(file, reportData{Comparisons: comparisons, Sweep: sweep}) +} + +// deltaPct expresses the A2A cost as an improvement relative to SLIM: +// (A2A − SLIM) / SLIM × 100. Unlike a reduction relative to A2A (capped at +// 100%), this is unbounded, so a 42× speedup reads as ~4100% rather than ~98%. +// Positive values mean SLIM was faster/cheaper. +func deltaPct(a2a, slim int64) string { + if slim == 0 { + return "n/a" + } + pct := (float64(a2a-slim) / float64(slim)) * 100 + return fmt.Sprintf("%.1f%%", pct) +} + +func atoi(v string) int { + n, _ := strconv.Atoi(v) + return n +} + +func atoi64(v string) int64 { + n, _ := strconv.ParseInt(v, 10, 64) + return n +} + +const htmlTemplate = ` + + + + SLIM vs A2A — Consensus Streaming + + + +

SLIM vs A2A — Consensus Streaming

+
+

This benchmark runs the same distributed hypothesis-convergence workload on two + transports and measures how fast N agents reach identical consensus. SLIM + (slim-group-session) broadcasts each finding once over a native group session and the + dataplane fans it out — there is no relay. A2A (a2a-relay-stream) + has no peer multicast, so the runner is an explicit relay hub: every finding makes two streaming + hops (agent → runner → N−1 agents).

+

The Improvement vs SLIM column is (A2A − SLIM) / SLIM × 100 — how much + more time/work A2A costs relative to SLIM. It is unbounded: e.g. +4000% means A2A took ~41× + as long as SLIM. Positive values favor SLIM. See metric definitions and the + architecture diagram below.

+
+ {{range .Comparisons}} +

{{.ScenarioName}}

+ + + + + + + + + +
MetricA2ASLIMImprovement vs SLIM
Consensus wall (ms){{if .HasA2A}}{{.A2A.ConsensusWallMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.ConsensusWallMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.ConsensusWallMS .SLIM.ConsensusWallMS}}{{else}}—{{end}}
Last agent converge (ms){{if .HasA2A}}{{.A2A.LastAgentConvergeMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.LastAgentConvergeMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.LastAgentConvergeMS .SLIM.LastAgentConvergeMS}}{{else}}—{{end}}
Avg propagation (ms){{if .HasA2A}}{{.A2A.AvgPropagationMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.AvgPropagationMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.AvgPropagationMS .SLIM.AvgPropagationMS}}{{else}}—{{end}}
P95 propagation (ms){{if .HasA2A}}{{.A2A.P95PropagationMS}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.P95PropagationMS}}{{else}}—{{end}}{{if and .HasA2A .HasSLIM}}{{deltaPct .A2A.P95PropagationMS .SLIM.P95PropagationMS}}{{else}}—{{end}}
Stream RPC count{{if .HasA2A}}{{.A2A.StreamRPCCount}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.StreamRPCCount}}{{else}}—{{end}}—
Epochs (ok / failed){{if .HasA2A}}{{.A2A.EpochsSucceeded}} / {{.A2A.EpochsFailed}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.EpochsSucceeded}} / {{.SLIM.EpochsFailed}}{{else}}—{{end}}—
Success{{if .HasA2A}}{{.A2A.Success}}{{else}}—{{end}}{{if .HasSLIM}}{{.SLIM.Success}}{{else}}—{{end}}—
+ {{end}} + {{if .Sweep}} +

Sweep results

+ + + {{range .Sweep}} + + + + + {{end}} +
ScenarioImplAgentsThink msPayload BConsensus wallAvg propagationP95 propagationStream RPCsEpochs ok/failed
{{.ScenarioName}}{{.Implementation}}{{.Agents}}{{.ThinkTimeMs}}{{.PayloadBytes}}{{.ConsensusWallMS}}{{.AvgPropagationMS}}{{.P95PropagationMS}}{{.StreamRPCCount}}{{.EpochsSucceeded}}/{{.EpochsFailed}}
+ {{end}} + +

Metric definitions

+
+
Consensus wall (ms)
+
Headline metric: wall-clock time from the runner's single start broadcast until every agent + has reached identical local consensus. Excludes process/library startup and teardown.
+
Last agent converge (ms)
+
Time until the slowest agent converged, taken from each agent's own convergence timestamp relative to + start. Highlights stragglers behind the headline number.
+
Avg / P95 propagation (ms)
+
Per-finding delivery latency, from the moment a finding is emitted to when a peer applies it + (average and 95th percentile). SLIM is one multicast hop; A2A adds the extra relay hop, so it grows + with agent count, message rate, and payload size.
+
Stream RPC count
+
Number of finding-carrying messages on the data path. For SLIM this equals findings emitted (one + native multicast each). For A2A it is the relay deliveries, ≈ findings × (N−1), because the + hub re-sends every finding to all other agents.
+
Epochs (ok / failed)
+
Each run repeats the same consensus attempt over several epochs. An epoch succeeds if every + agent reaches global consensus within spec.maxEpochTimeMs (each attempt runs up to + maxRounds); otherwise it is counted as failed. This surfaces reliability differences: under + load A2A may miss the per-epoch budget while SLIM still converges. Overall Success is true only + when no epoch failed.
+
Coord fanout (ms)
+
Cumulative time the A2A relay hub spent fanning findings out to peers. 0 for SLIM because + there is no relay — the dataplane does the fan-out.
+
Payload B
+
Fixed-size padding (spec.payloadBytes) added to every finding to stress transport + bandwidth. Semantically inert; it does not change the consensus math or round count.
+
Improvement vs SLIM
+
(A2A − SLIM) / SLIM × 100. Unbounded measure of how much more time/work A2A costs than + SLIM; e.g. +4100% ≈ 42× slower. Positive favors SLIM. (Note: a2a-go's streaming task store + also appends each relayed finding to task history and deep-copies per update, which amplifies A2A cost + super-linearly at large payloads — a property of the A2A streaming/task model.)
+
+ +

Architecture

+

SLIM agents form a peer mesh over a native group session and the runner only observes; A2A routes every + finding through the runner acting as a relay hub.

+
+ 
+flowchart TB
+  subgraph slim["SLIM native group session — no relay"]
+    R1["runner (passive observer: start signal + consumes pushed metrics)"]
+    SA0["agent-0"]
+    SA1["agent-1"]
+    SA2["agent-2"]
+    SA0 ---|"Publish finding → all (dataplane fan-out)"| SA1
+    SA1 --- SA2
+    SA0 --- SA2
+    SA0 -.->|"push snapshot on convergence"| R1
+    SA1 -.-> R1
+    SA2 -.-> R1
+  end
+  subgraph a2a["A2A — runner is the relay hub"]
+    R2["runner = RELAY (hosts A2A server)"]
+    AA0["agent-0"]
+    AA1["agent-1"]
+    AA2["agent-2"]
+    AA0 -->|"stream findings → runner"| R2
+    AA1 --> R2
+    AA2 --> R2
+    R2 -->|"stream relayed findings → agent"| AA0
+    R2 --> AA1
+    R2 --> AA2
+  end
+
+
+ + + + +` diff --git a/benchmarks/slim-vs-a2a/tools/validate_scenarios/main.go b/benchmarks/slim-vs-a2a/tools/validate_scenarios/main.go new file mode 100644 index 00000000..fa5e76c5 --- /dev/null +++ b/benchmarks/slim-vs-a2a/tools/validate_scenarios/main.go @@ -0,0 +1,41 @@ +// Copyright AGNTCY Contributors (https://github.com/agntcy) +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "flag" + "fmt" + "log" + "os" + "path/filepath" + "strings" + + "github.com/agntcy/csit/benchmarks/slim-vs-a2a/internal/scenario" +) + +func main() { + dir := flag.String("dir", "./plans/sweeps", "directory containing scenario yaml files") + flag.Parse() + + entries, err := os.ReadDir(*dir) + if err != nil { + log.Fatalf("read dir: %v", err) + } + var failed int + for _, e := range entries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".yaml") { + continue + } + path := filepath.Join(*dir, e.Name()) + if _, err := scenario.LoadFile(path); err != nil { + fmt.Printf("INVALID %s: %v\n", path, err) + failed++ + } else { + fmt.Printf("OK %s\n", path) + } + } + if failed > 0 { + os.Exit(1) + } +}