I benchmarked 6 agent orchestrators on the same scenarios. Taskplane came out on top (plus a per-task model trick with OMP)

[kaka-sangi]

Hey all,

I've been trying to pick an orchestrator for running lots of coding agents in parallel, so I set up a test battery and ran the same scenarios against a bunch of tools. Taskplane ended up first, so I figured I'd share the numbers here, including where it lost points, since that's probably more useful to the maintainers than a pat on the back.

How the test worked

• I picked 21 orchestrators and wrote one harness with the same scenarios for all of them: fan-out, dependency DAG, worker crash, orchestrator killed mid-run (resume), a failing task next to independent ones, a hung agent, 40 tiny tasks for scale, a human approval gate, and a couple more.
• The agents are scripted stand-ins, not real models. Fake claude/codex/pi binaries that behave deterministically, so I could measure the orchestrator itself (scheduling, retry, resume, isolation, concurrency caps) and not model quality or luck.
• Each tool is scored 0-100 per category and rolled into a total. There's a control "baseline" (a dumb scheduler) to check the harness makes sense.
• 6 of the 21 got scored. The rest didn't make it. Some were out of time, fusion was blocked because the sandbox couldn't fetch one of its dependencies, and a few need their own build and adapter that I didn't get to. So this is a 6-tool ranking, not a ranking of everything.

Results

Rank	Tool	Total	Coverage	Correctness	Recovery	Throughput	Failure handling	Operability
1	taskplane	78	95%	100	67	51	100	68
2	langgraph	74	100%	100	33	72	90	80
3	baseline (control)	74	100%	90	18	86	90	100
4	ralphex	67	95%	100	42	23	50	100
5	archon	66	95%	100	33	23	90	80
6	bernstein	46	95%	57	33	8	60	71

Observability scored n/a for most of the tools, Taskplane included, so it's left out of the table.

Why Taskplane won

It's not the fastest. It won because it's the only one that behaved well on the boring, painful stuff:

• Worker crash: recovered in about 0.14 s.
• Orchestrator killed mid-batch: resumed without redoing finished work.
• A task fails: independent tasks keep going and the flaky one gets retried. That's the only tool in the group that got a 100 on failure handling.
• Correctness: 100, meaning results actually landed where they should.

Where it lost points (honest part)

• Speed. The wave model left the 7-task DAG at about 98 s against a 30 s ideal, because later waves wait for the whole previous wave. Throughput was 51.
• Hung agents. It never noticed the hung agent. It just waits on the 60 minute stall timeout. Only ralphex actually killed the hung process, and even that one then looped on it instead of moving on.
• Merge step. The merger is an LLM role, so in my run the scripted stand-in did that job. I did not test the merge agent with a real model.
• Human gate. None of the six tools paused for approval in a headless run, Taskplane included.

How the others did, quickly

• langgraph: fastest real scheduler on fan-out (92% of ideal), but it's a library so all the glue is yours. Its crash run failed because my glue's retry didn't catch a killed subprocess, so that score is partly on me.
• ralphex: sequential by design. Resume is exact because state lives in the plan's checkboxes, but it stops the whole plan on the first failure and doesn't retry a crashed agent.
• archon: runs every ready node at once (8 agents on a cap of 4, 40 on a cap of 8). Fast, but it ignores the concurrency cap, which I scored as zero throughput. Resume after killing the orchestrator lost two tasks.
• bernstein: got the dependency order right, but each wave waits about two minutes, tasks were marked done while commits sat on side branches, and one of two same-file edits was silently lost.

Bonus: per-task models with OMP, no fork needed

Separate from the benchmark, I wanted different models for different tasks (cheap one for easy stuff, strong one for the hard stuff). Right now model choice in Taskplane is per role (worker.model, reviewer.model, merge.model, supervisor.model, reviewModel). Every worker in a batch gets the same model and there's no model field in PROMPT.md.

So I tried running Taskplane on top of OMP (the extended pi) and it worked:

• OMP loads the Taskplane extension fine as the /orch host.

• Taskplane looks for pi-coding-agent/dist/cli.js to start workers. I dropped a ~40 line shim there that reads a line from the task's PROMPT.md, picks the model, and launches OMP as the worker:

  **Model:** provider/model-id
  

  No line means it falls back to worker.model, and if that's empty OMP's default applies.

• I ran a real 7-task dependency batch with OMP as host and worker. All 7 completed and the model server logged the requested model per task: two on fast, one on strong (from the PROMPT.md line) and four on mid (from config default).

Things to know if you try it:

• I tested against a fake model server, not real models, so the actual model calls were never exercised.
• OMP rejected Taskplane's worker tool list (ls, find, request_segment_expansion) and every worker crashed until I mapped ls/find to glob and dropped the rest. request_segment_expansion doesn't exist in OMP, so segment expansion doesn't work under OMP.
• OMP needs Bun 1.3.14+, and Taskplane's preflight wants a pi on PATH, so you need a one-line launcher named pi that runs OMP.
• I didn't test switching models mid-session. The OMP RPC docs list a set_model command, but that's untested.

Suggestion for the project: a first-class optional **Model:** field parsed by Taskplane itself (and ideally per-task reviewer/merger models too). It looks like a small patch to the task parser and the lane runner's model argument, and it would make the shim unnecessary. Happy to send a PR or share the shim if there's interest.

Caveats

• Scripted agents only. This measures orchestration behavior, not how good the resulting code is.
• 6 of 21 scored, single run per scenario, one machine. Treat the totals as directional.
• Some causes (especially for bernstein) I read from logs and may partly come from how my scripted agent exits.

Thanks for building this. Curious whether the stall detection for hung agents and a headless approval gate are on the roadmap, since those were the two gaps I hit hardest.