Worker.Run pulls up to 10 events per XREADGROUP but calls processEvent on them one after another, and ax-controller starts exactly one worker. A reconcile is a chain of sequential Substrate RPCs (CreateAtespace, GetActorTemplate, CreateActorTemplate, CreateActor, egress policy, ResumeActor), and for a cold workspace it also polls /readyz for up to WorkspaceReadyTimeout (15s). So one controller pod handles roughly one new task every 15 seconds when workspaces are cold, and scaling means adding pods.
I prototyped a fix: https://github.com/anzal1/ax/tree/perf/concurrent-worker
Worker.Concurrency (flag --concurrency, default 16). Events are sharded to slots by a hash of atespace/name, so events for the same task stay ordered and never reconcile concurrently, while different tasks run in parallel.- Each slot has a bounded queue, so a task with several queued events doesn't stall the other slots.
- Events are still acked only after processing, and shutdown waits for in-flight work.
Benchmark with a fake Substrate at 10ms per RPC and a 200ms readiness poll standing in for the 15s one (64 tasks):
| concurrency |
warm tasks/s |
cold tasks/s |
| 1 |
15.2 |
3.8 |
| 4 |
57.5 |
14.2 |
| 16 |
160.0 |
40.2 |
| 64 |
309.3 |
79.8 |
go test ./internal/controller -run '^$' -bench WorkerThroughput -benchtime 1x
There's also a test that queues three revisions of each task back to back and asserts no task ever has two Substrate calls in flight. It fails if the sharding is replaced with round-robin. The whole suite passes under -race.
Caveats: the latency comes from a mock, not a real cluster, and concurrency=1 on the branch reads up to 64 events ahead where main reads 10. A natural follow-up is merging duplicate reconcile events for the same task, since a reconcile always reads the latest spec.
Happy to turn this into a PR if you're open to it. CLA is signed.
Worker.Runpulls up to 10 events perXREADGROUPbut callsprocessEventon them one after another, andax-controllerstarts exactly one worker. A reconcile is a chain of sequential Substrate RPCs (CreateAtespace, GetActorTemplate, CreateActorTemplate, CreateActor, egress policy, ResumeActor), and for a cold workspace it also polls/readyzfor up toWorkspaceReadyTimeout(15s). So one controller pod handles roughly one new task every 15 seconds when workspaces are cold, and scaling means adding pods.I prototyped a fix: https://github.com/anzal1/ax/tree/perf/concurrent-worker
Worker.Concurrency(flag--concurrency, default 16). Events are sharded to slots by a hash of atespace/name, so events for the same task stay ordered and never reconcile concurrently, while different tasks run in parallel.Benchmark with a fake Substrate at 10ms per RPC and a 200ms readiness poll standing in for the 15s one (64 tasks):
There's also a test that queues three revisions of each task back to back and asserts no task ever has two Substrate calls in flight. It fails if the sharding is replaced with round-robin. The whole suite passes under
-race.Caveats: the latency comes from a mock, not a real cluster, and concurrency=1 on the branch reads up to 64 events ahead where main reads 10. A natural follow-up is merging duplicate reconcile events for the same task, since a reconcile always reads the latest spec.
Happy to turn this into a PR if you're open to it. CLA is signed.