Add Zephyr Consensus & Performance Lab

.github/workflows/ci.yml

@@ -61,6 +61,31 @@ jobs:
       - name: Test
         run: go test ./...
 
+  consensus-lab:
+    name: Consensus & performance lab
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Set up Go
+        uses: actions/setup-go@v7
+        with:
+          go-version-file: go.mod
+          cache: false
+
+      - name: Multi-validator conformance gate
+        run: go test ./internal/api -run '^TestLab' -count=1 -timeout=90s
+
+      - name: Partition recovery stress gate
+        run: go test ./internal/api -run '^TestLabSevenValidatorsStallWithoutQuorumThenRecoverWithPeerSync$' -count=3 -timeout=90s
+
+      - name: Seven-validator finalized-throughput sample
+        run: go test ./internal/api -run '^$' -bench '^BenchmarkLabConsensusFinality7Validators$' -benchtime=3x -count=1 -timeout=120s
+
+      - name: P-256 verification baseline
+        run: go test ./internal/api -run '^$' -bench '^BenchmarkLabP256TransactionVerification$' -benchtime=1s -count=1
+
   wallet:
     name: Wallet build
     runs-on: ubuntu-latest
@@ -85,4 +110,4 @@ jobs:
         run: npm audit --audit-level=high
 
       - name: Type-check and build
-        run: npm run build
+        run: npm run build

docs/performance-lab.md

@@ -0,0 +1,238 @@
+# Zephyr Consensus & Performance Lab
+
+## Purpose
+
+The Consensus & Performance Lab turns Zephyr's scalability target into a repeatable engineering program with two independent responsibilities:
+
+1. **protocol conformance**: prove safety and liveness under deterministic multi-validator faults;
+2. **performance measurement**: measure finalized throughput and finality through the real transaction, consensus, state and persistence paths.
+
+A performance change is not successful if it weakens the conformance matrix.
+
+## Canonical meaning of TPS
+
+For Zephyr, the headline TPS number means:
+
+> **transactions finalized by validator consensus per second**.
+
+The benchmark does not count HTTP requests accepted, mempool insertions, unsigned synthetic operations, or batches that were not individually executed as transactions. A transaction counts only after it appears in a committed block protected by the configured quorum-certificate rules.
+
+## Canonical transfer workload
+
+The first reference workload is a native ZPH transfer with:
+
+- a real P-256 signature;
+- the normal transaction domain and chain ID;
+- normal static and stateful validation;
+- a real mempool entry;
+- real state execution;
+- a deterministic state commitment;
+- proposal and vote dissemination;
+- quorum-certificate formation;
+- committed block persistence.
+
+Client-side key generation/signing is prepared outside the timed consensus benchmark. Signature **verification** remains inside the node path and is benchmarked separately as well.
+
+## Validator matrix
+
+The target matrix is:
+
+- 1 validator: local execution/control baseline;
+- 4 validators: smallest useful multi-validator BFT-style lab;
+- 7 validators: primary development baseline;
+- 16 validators: scaling and dissemination-pressure baseline.
+
+The first checked-in gate focuses on 4 and 7 validators. The harness is parameterized so 1- and 16-validator scenarios can use the same machinery later.
+
+The 7-validator reference configuration uses equal voting power: `10,000` per validator, `70,000` total, with Zephyr's normal quorum calculation producing a `46,667` voting-power threshold.
+
+## Required metrics
+
+Every publishable Zephyr performance result should report at least:
+
+- sustained finalized transactions per second;
+- time to finality p50, p95 and p99;
+- validator count;
+- transactions per finalized block;
+- finalized block payload size;
+- protocol payload bytes per finalized transaction;
+- persisted state size per node;
+- CPU profile;
+- heap/allocation profile;
+- machine CPU, RAM, operating system and Go version;
+- network topology and latency assumptions.
+
+The repository benchmark currently emits:
+
+- `finalized-tx/s`;
+- `finality-p50-ms`;
+- `finality-p95-ms`;
+- `finality-p99-ms`;
+- `protocol-payload-B/finalized-tx`;
+- `finalized-block-B`;
+- `state-B/node`.
+
+CPU, heap, mutex and block profiles come from the standard Go benchmark profiler.
+
+## First CI baseline
+
+The first successful 7-validator measurement on a GitHub-hosted Ubuntu runner is a **development baseline only**, not a Zephyr performance claim and not a controlled-hardware result.
+
+With 32 finalized transfers per block, one observed run on an AMD EPYC 7763 hosted runner reported approximately:
+
+- `42.92 finalized-tx/s`;
+- `628 ms` p50 finality;
+- `1.128 s` p95/p99 finality;
+- `23,045 B` finalized block size;
+- `17,468 B` measured protocol payload per finalized transaction;
+- `162,138 B` persisted state per node after the sample;
+- approximately `88.8 us/op` for the separate P-256 transaction-validation baseline.
+
+A separate verification run was in the same rough range at about `44.3 finalized-tx/s`. Shared-runner variance is expected. These numbers establish a measurable starting point and must not be presented as production capacity.
+
+## Running the lab
+
+Run the complete protocol conformance gate:
+
+```bash
+go test ./internal/api -run '^TestLab' -count=1 -timeout=90s
+```
+
+Stress the partition/heal recovery path repeatedly:
+
+```bash
+go test ./internal/api \
+  -run '^TestLabSevenValidatorsStallWithoutQuorumThenRecoverWithPeerSync$' \
+  -count=5 \
+  -timeout=120s
+```
+
+Run the 7-validator finalized-throughput benchmark across consecutive finalized blocks:
+
+```bash
+go test ./internal/api \
+  -run '^$' \
+  -bench '^BenchmarkLabConsensusFinality7Validators$' \
+  -benchtime=5x \
+  -count=1 \
+  -timeout=120s
+```
+
+Measure P-256 transaction verification separately:
+
+```bash
+go test ./internal/api \
+  -run '^$' \
+  -bench '^BenchmarkLabP256TransactionVerification$' \
+  -benchtime=2s \
+  -count=1
+```
+
+Generate profiles for the end-to-end benchmark:
+
+```bash
+go test ./internal/api \
+  -run '^$' \
+  -bench '^BenchmarkLabConsensusFinality7Validators$' \
+  -benchtime=5x \
+  -cpuprofile=cpu.out \
+  -memprofile=mem.out \
+  -mutexprofile=mutex.out \
+  -blockprofile=block.out \
+  -count=1 \
+  -timeout=120s
+```
+
+Then inspect a profile, for example:
+
+```bash
+go tool pprof -http=:8081 cpu.out
+```
+
+## Fault-injection contract
+
+The lab transport wraps Zephyr's existing `peerTransport`; it does not replace consensus or ledger logic. Faults are injected at the transport boundary while transactions, proposals, votes, certificates, blocks, state roots and persistence remain production implementations.
+
+The initial gate covers:
+
+- 7-validator certified happy-path finality;
+- a 4/3 partition where neither side has quorum: no block may commit while partitioned; after heal, quorum finality must resume and lagging validators must catch up;
+- a 5/2 partition where the quorum side commits and the minority later catches up through normal peer recovery;
+- delayed, duplicated and deliberately vote-before-proposal delivery: all nodes must still converge on one committed tip.
+
+The 4/3 scenario exposed a real recovery gap. After heal, enough validators can sign the same proposal to finalize a block even though only a subset has already materialized that block. Snapshot recovery alone cannot solve every such state immediately because fewer than 2/3 of validators may have the new committed snapshot available.
+
+## Certified block catch-up
+
+Zephyr therefore has a signed-evidence catch-up path before snapshot fallback:
+
+1. an authenticated internal endpoint exposes retained **proposal/vote fragments for a height even before the serving node has materialized the block**;
+2. each peer may contribute only a partial fragment;
+3. the receiver groups fragments by the same canonical proposal/round/block and never combines votes from competing proposals or rounds;
+4. malformed or conflicting peer fragments are ignored without poisoning compatible evidence from other peers;
+5. matching valid votes already persisted by the recovering validator are combined with compatible remote fragments;
+6. every P-256 proposal/vote signature, validator identity and scheduled proposer is independently validated against the receiver's local validator set;
+7. the receiver locally recomputes voting power and requires the normal `2/3+` quorum before any state mutation;
+8. the block is independently executed and its chain continuity, transaction validity, state root and hash are checked before atomic import;
+9. only after those checks does the receiver derive its local commit certificate;
+10. quorum-validated snapshot recovery remains the fallback for deeper repair.
+
+This means a peer can transport evidence but cannot manufacture finality: a recovery import still requires the same validator signatures that would have been necessary for consensus. The transport capability is optional, so the future libp2p/QUIC transport can implement the same contract while HTTP remains the reference transport.
+
+The unit and lab gates explicitly verify that:
+
+- evidence below quorum cannot mutate state;
+- a tampered signature is rejected;
+- a locally persisted matching vote can contribute to the quorum together with remote evidence;
+- partial compatible evidence from multiple peers can be aggregated;
+- repeated 4/3 partition/heal recovery converges without depending on message arrival order.
+
+## Next conformance cases
+
+The matrix should expand to cover:
+
+- validator offline/restart before and after vote;
+- proposer crash during a round;
+- conflicting proposals;
+- conflicting votes;
+- explicit Byzantine peer payloads;
+- corrupted snapshots;
+- wrong-chain validators;
+- longer partitions and repeated heal/fail cycles;
+- the same conformance suite over HTTP and future libp2p/QUIC transport.
+
+## Performance-gate policy
+
+Correctness is a hard gate now. Numerical performance thresholds are deliberately not hard-coded against GitHub-hosted runners because shared-runner variance would make the gate noisy.
+
+The next performance step is to establish a controlled reference machine and retain benchmark history. Once variance is understood, Zephyr can add regression budgets such as:
+
+- no more than N% sustained-TPS regression;
+- no more than N% p95 finality regression;
+- no unbounded growth in protocol bytes per finalized transaction;
+- no unexpected state-size or allocation regression.
+
+## Optimization decision rule
+
+No major performance architecture is selected before profiling the canonical benchmark.
+
+The first profile should attribute time and resource pressure across:
+
+`signature verification -> transaction validation -> mempool -> state execution -> state root -> block serialization -> proposal dissemination -> votes -> persistence`
+
+Parallel signature verification, serialization changes, storage replacement, lock reduction, incremental state commitments and transport/dissemination changes are hypotheses until the profile identifies the actual bottleneck.
+
+## Path toward 1M TPS
+
+The engineering sequence is:
+
+1. consensus and performance lab;
+2. profiling and evidence-backed performance architecture;
+3. scalable storage/state execution;
+4. production libp2p/QUIC transport, while keeping HTTP as the reference transport;
+5. deterministic Rust-first WASM execution and fee metering;
+6. staking/governance;
+7. public devnet;
+8. confidential compute marketplace.
+
+At every stage the canonical finalized-TPS/finality benchmark and the consensus conformance matrix remain the comparison point.

docs/performance-profile-2026-08-19.md

@@ -0,0 +1,100 @@
+# Zephyr Canonical Performance Profile — 2026-08-19
+
+## Scope
+
+This profile uses the canonical Consensus & Performance Lab benchmark on a GitHub-hosted Ubuntu 24.04 runner with Go 1.22.12 and an AMD EPYC 7763 CPU.
+
+The measured workload uses 7 validators, real P-256 transaction validation, real HTTP peer replication, deterministic state execution/state root, quorum-certificate consensus and persisted committed blocks. The profiling run used 5 consecutive benchmark iterations with 32 finalized transfers per block.
+
+This document records engineering evidence, not a production performance claim.
+
+## Observed benchmark sample
+
+The profiling run reported approximately:
+
+- `38.52 finalized-tx/s`;
+- `971.5 ms` p50 finality;
+- `1.106 s` p95/p99 finality;
+- `23,054 B` finalized block size;
+- `15,071 B` measured protocol payload per finalized transaction;
+- `244,576 B` persisted state per node after the five-iteration sample.
+
+The lower TPS than shorter CI samples is useful: the sustained multi-block run increases persisted state and exposes costs that a one-block microbenchmark hides.
+
+## CPU profile
+
+The strongest CPU signal is persistence/serialization:
+
+- `ledger.(*Store).writeState`: about **58.3% cumulative CPU**;
+- `encoding/json.MarshalIndent`: about **54.5% cumulative CPU**;
+- `encoding/json.appendIndent`: about **28.6% flat CPU**;
+- `ledger.(*Store).Accept`: about **31.9% cumulative CPU**;
+- P-256 `tx.VerifySignature`: about **10.1% cumulative CPU**.
+
+The conclusion is that signature verification is material but is not the first bottleneck. Full-state JSON persistence currently costs substantially more CPU than P-256 verification.
+
+## Allocation profile
+
+The allocation profile reinforces the same result. Roughly 2.0 GB were allocated during the profiled process, with:
+
+- `encoding/json.MarshalIndent`: about **40.8% flat allocations** and **73.8% cumulative** through its call tree;
+- `ledger.(*Store).writeState`: about **71.4% cumulative allocations**;
+- `encoding/json.Marshal`: about **20.4% flat allocations**;
+- `bytes.growSlice`: about **11.6% flat allocations**;
+- repeated cloning/snapshot construction also contributes materially as the persisted state grows.
+
+The current persistence model serializes and atomically rewrites a broad `persistedState` structure after high-frequency operations such as transaction acceptance, funding and consensus vote recording. That design is excellent for simple correctness/restart guarantees but does not scale as the hot-path persistence architecture.
+
+## Lock contention
+
+The mutex profile identifies the ledger write lock as another direct consequence of the persistence model:
+
+- `ledger.(*Store).Accept`: about **89.7% cumulative mutex delay**;
+- `sync.(*Mutex).Unlock`: about **82.7% flat mutex delay**;
+- `handleBroadcastTransaction`: about **98.8% cumulative through the affected request path**.
+
+Parallel transaction ingress therefore serializes behind a state-wide critical section that also performs expensive cloning/serialization/persistence work.
+
+## Blocking profile
+
+The blocking profile shows two major classes:
+
+1. ledger lock waiting during concurrent transaction acceptance;
+2. synchronous HTTP transaction fan-out to peers.
+
+Notable cumulative blocking signals include:
+
+- HTTP client/send/round-trip paths around **35–37%**;
+- `httpPeerTransport.postJSON` around **25%**;
+- `Server.broadcastTransaction` around **23%**;
+- ledger `Store.Accept` around **13%**.
+
+This confirms that transaction-by-transaction synchronous replication will become a later networking/dissemination bottleneck, but the persistence/lock problem should be addressed first because it dominates both CPU/allocations and local contention.
+
+## Evidence-backed optimization order
+
+The first optimization sequence is therefore:
+
+1. **Persistence hot path**
+   - remove human-readable `MarshalIndent` from machine state persistence immediately;
+   - stop treating full-state JSON rewrite as the long-term write path;
+   - introduce a durable append/batch-oriented journal or structured state backend so mempool/vote/transaction mutations do not reserialize the whole node state;
+   - preserve atomic committed checkpoints and restart validation.
+2. **Ledger concurrency**
+   - reduce the amount of work performed while holding the global store mutex;
+   - separate validation/read preparation from the serialized commit section where deterministic safety allows;
+   - move toward deterministic batch execution/state updates rather than one durable full-state rewrite per transaction.
+3. **State commitment/storage architecture**
+   - make state commitments incremental rather than rebuilding broad state structures as the chain grows;
+   - introduce a backend abstraction suitable for structured key/value state and deterministic snapshots.
+4. **Networking/dissemination**
+   - replace synchronous transaction-by-transaction HTTP fan-out with batched/asynchronous dissemination semantics;
+   - retain the HTTP transport as the reference implementation while adding libp2p/QUIC later.
+5. **Signature verification parallelism**
+   - parallelize P-256 verification once persistence/locking no longer masks its cost.
+
+## Immediate next experiment
+
+The first low-risk performance change should replace `json.MarshalIndent` with compact deterministic-equivalent JSON for `state.json` persistence and rerun the exact same 7-validator benchmark/profiles. This does not change consensus semantics or on-disk JSON meaning, but it directly tests the largest CPU/allocation signal.
+
+If the expected improvement appears, the next architectural change should replace repeated full-state rewrites with a journal/checkpoint persistence layer. No storage engine should be selected until that benchmark establishes how much of the remaining cost is serialization, file I/O, state cloning and lock hold time.