feat(conductor): Spec 005 — Omadia Conductor (deterministic engine, designer, human-in-the-loop) — US1–US8 implemented + live-tested#321
Conversation
…er & human-in-the-loop Conception for Omadia Conductor: a process layer where the runtime (not the LLM) owns step progression and hand-offs, promoting the existing per-tool postcondition / tool-obligation / deterministic-action atoms to process scope. - spec.md: 9 user stories (engine, durable run lifecycle/resume, triggers, event triggers + connector "Conductor Surface", human steps with durable awaits/reminders/ deadline, principals & role resolver seam, visual+conversational Designer, dry-run, audit), 32 FRs, 10 SCs. - data-model.md: workflow/version/draft, run/run-step, conductor_awaits (+responses), roles/role-assignments (the "baton"), manifest emits: extension, LISTEN/NOTIFY resume, run/await state machines. Status: Draft. In-repo modular (@omadia/conductor-core + kernel wiring + Designer under web-ui/app/admin/conductor/). No connectors built here — only the contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Grounded against the live codebase: phase-by-phase build sequence (US1-US9), reuse map, net-new substrate, 16 resolved integration landmines (A-P), test strategy mapped to SC-001..010, and the 4 owner design decisions (conductor_channel_bindings, conductor_schedules, FOR UPDATE SKIP LOCKED claim, serializable predicate AST).
Phase 0+1 of the Spec 005 integration plan: the deterministic, I/O-free
workflow engine — sibling of @omadia/canvas-core.
- Predicate AST (serializable guards + exit postconditions; no eval) with a
total, deterministic evaluator over {ctx, stepResult}.
- validate(graph, knownRefs?): shape gate (ajv 2020-12) + reachability,
unguarded-cycle detection, deadline-without-fallback, duplicate ids,
fallback-origin, and optional agent/action/role/event reference checks —
each error names the offending node (FR-003).
- nextStep(): postcondition verdict -> matching guarded transition -> declared
fallback -> complete/stuck; identical inputs yield identical decisions
(FR-001/002/006, SC-009).
- Published JSON schema (schema/) kept in parity with the runtime schema by test.
- 46 vitest cases (predicate, engine US1 acceptance, validation, fixtures);
builds clean (tsc) and typechecks. Wired into the workspace build order.
ajv-only runtime dep; engine has no DB/network/LLM I/O (FR-032).
…l slice) Deterministic engine now drives real, persisted runs end to end: - migrations/0001_conductor.sql: full conductor_* schema (workflows/versions/ drafts/runs/run_steps/awaits/await_responses/roles/role_assignments) + the resolved-decision tables (conductor_channel_bindings, conductor_schedules) + claim columns + pg_notify triggers. TEXT+CHECK enums, idempotent. - runConductorMigrations (per-subsystem migrator, mirrors runAuthMigrations). - ConductorWorkflowStore (create + versioned publish) and ConductorRunStore (run + durable per-step record, recordStepAndAdvance persists before advance). - StepEffects seam (so US8 preview reuses this executor) + StubStepEffects. - ConductorRunExecutor.startRun: validate-active-version -> create run -> engine nextStep loop -> persist each step + accumulated context; human steps park as waiting (durable awaits land later), agent/action run to completion. - Operator API /api/v1/operator/conductors (publish/list/status/start-run/trace), graph validated by conductor-core before persist; mounted behind requireAuth. - wireConductor() called from boot inside the graphPool block (inert in-memory). tsc --noEmit clean across the middleware; conductor-core builds.
…pm ci) The Docker image build runs `npm ci`, which requires the lockfile to include the new conductor-core workspace package. Regenerated with node 22.22.3; minimal diff (one workspace entry, no other dep churn).
- routes: coerce req.params (string | string[]) via paramStr(); cast req.body graph through unknown before WorkflowGraph (TS2352). - runExecutor: human-step actor uses primitive principalKind/principalRef fields instead of the Principal interface (not assignable to JsonValue). Full workspace build (node 22.22.3) clean; surfaces only under the stricter lockfile-pinned deps the Docker image uses.
tsc does not emit .sql; runConductorMigrations scans dist/conductor/migrations at boot. Add the explicit COPY (same pattern as auth/routines/profile migrations) so the production image ships 0001_conductor.sql. Fixes ENOENT crash-loop on boot.
Engine semantics corrected: an 'agent' step targets an **Agent (orchestrator
instance)** resolved by slug from the multi-orchestrator registry — not a stub,
not a bare model. Backend:
- conductor-core: add step.prompt (agent message, {{ctx.x}} interpolation) and
step.input (action input); agentId doc'd as the Agent slug.
- RealStepEffects: agent step runs a genuine turn via
registry.get(slug).built.bundle.agent.chat(...) (the schedule-worker path),
answer.text becomes the step result; action step invokes the real connector
tool via dynamicAgentRuntime.invokeAgentTool. StepMeta threads runId for
session scoping. wireConductor now wires RealStepEffects (stub kept for tests).
Operator UI (the missing entry point):
- web-ui/app/conductor/page.tsx: list/publish workflows (graph validated by the
engine before save), start runs, render the durable step trace + result context.
- Nav: top-level 'Conductor' link; api.ts conductor client; en/de i18n (parity OK).
Full middleware build + web-ui typecheck clean; conductor-core 46 tests green.
A real Agent turn on the fallback orchestrator takes ~tens of seconds; driving the run synchronously inside the HTTP request hit a gateway timeout (500) even though the run completed durably in the background. Now: - startRun drives the run in the background and returns immediately (202); the run is durable and observed via the run/trace API. awaitCompletion keeps a synchronous path for tests/fast steps. - the Conductor UI polls GET /:slug/runs/:runId until the run leaves 'running', then renders the final trace + result context (the real agent text). - log the actual error in the publish/run 500 catch blocks. Verified live: fallback Agent produced a real gpt-5.5 answer persisted in conductor_runs.context (no stub).
A double-submitted publish of a new slug raced SELECT-then-INSERT and one request hit conductor_workflows_slug_key (500). Replace with INSERT ... ON CONFLICT (slug) DO UPDATE + a FOR UPDATE lock to serialize version numbering. Status is set only on first create.
Replace the raw-JSON publish form with a visual canvas (@xyflow/react):
- ConductorCanvas: add agent/action/human step nodes, drag to wire transitions,
per-kind inspector (agent slug+prompt, action id+input, human principal/
channel/message/reminder/deadline, optional postcondition + fallback + entry
toggle; edge guard), trigger config (manual/event/cron), load an existing
workflow's graph, Save→validate+publish, Run+poll. node.id stays stable while
data.stepId is renameable (edge-safe).
- backend GET /:slug returns {workflow, graph} for the editor to load;
getConductorWorkflowGraph client; en/de i18n (parity OK).
- page keeps the workflow list + quick-run and mounts the designer below.
web-ui typecheck + i18n clean; conductor middleware typecheck clean.
A double-fired Add-step click created two nodes sharing one generated id (React rendered one, state held two) → duplicate_step_id on publish. Use a monotonic ref counter for node/edge ids and swallow sub-350ms repeat add clicks; bump the counter past loaded ids.
The headline human-in-the-loop substrate. A human step now opens a durable conductor_awaits row and parks the run as 'waiting' (instead of a dead park): - ConductorAwaitStore (create/get/listWaiting/listDue/recordResponse/atomic close). - executor.resolveAwait: records the response, atomically closes the await, then resumes the run — feeds the response as the step result into the engine and drives onward. expireAwait: on deadline, fires the step's in-graph fallback (FR-017). - ConductorAwaitWorker: minute-tick poll of due awaits → expire (deadline path). - ISO-8601 duration parsing (PT6H/PT24H/P1D) for reminder/deadline. - routes: GET /awaits/pending (operator inbox), POST /awaits/:id/respond. - migration 0002: responder_id → TEXT (session identity, no users join for MVP). - UI: 'Pending human steps' inbox with Approve/Reject that resolves + resumes. Atomic resolution via UPDATE ... WHERE status='waiting' (FR-018). MVP: quorum 'any' (first response wins); holder-at-access auth deferred. backend + web-ui typecheck clean; i18n parity ok (1263 keys).
The recurring synthetic/double-click double-fire (already fixed for add-step) also started two runs / two responses. Guard handleRun, handleRespond, and the canvas run with a 600ms ref debounce so one click is one action.
…US4) The trigger class the real use cases depend on (merge/RC-build, ATS invite, calendar). ConductorEventRouter.emit(eventId, payload): scans enabled workflows, matches each active version's event trigger (eventId + optional payload-filter predicate, evaluated by conductor-core), and starts a run per match with the payload as initial context (FR-013). Operator route POST /emit + a UI 'Emit an event' form to fire/test it. (The kernel-side seam a connector calls; plugin ctx.events.emit + manifest emits-autodiscovery is the remaining connector half.) backend + web-ui typecheck clean; i18n parity ok (1268 keys).
Confidence before go-live. executor.previewRun(slug, payload, humanResponses):
simulates the workflow path in-memory with NO persistence and NO side effects —
no conductor_runs/awaits rows, no notification, no durable await. Human steps are
answered inline (supplied response, default {approved:true}); agent steps run a
real turn; action steps are stubbed (irreversible connector actions not executed).
Returns the full simulated step path (actor, postcondition, transition per step).
Route POST /:slug/preview + a 'Dry-run' button in the designer that renders the
simulated path. Reuses the StepEffects seam built for exactly this.
backend + web-ui typecheck clean; i18n parity ok (1272 keys).
A human step can address role:<key> instead of a fixed person; the role resolves live to its current holder(s). ConductorRoleStore (createRole, addHolder/removeHolder = the baton, resolve()=current open assignments — never frozen, FR-022). Migration 0003: holder_id/delegate_id -> TEXT (assign by session identity, no users join). Routes: GET/POST /roles, POST /roles/:key/holders (add|remove). The awaits inbox resolves role principals live to their current holders (FR-023). UI 'Roles & the baton' section: create role, assign/move holders; inbox shows role -> resolved holders. MVP: default resolver only (external RoleResolver registry is a follow-up); quorum 'any'. backend + web-ui typecheck clean; i18n parity ok (1278 keys).
Add an 'Edit' button to each workflow row in the operator list that loads the workflow into the visual designer and scrolls there. ConductorCanvas gains an editRequest prop (slug + per-click nonce) and loads on change, reusing the existing getConductorWorkflowGraph + versioned publish path.
Status update — Spec 005 implemented end-to-end (Draft)This PR started as the Spec 005 design docs and now also carries the full vertical implementation of the Omadia Conductor, grounded against the live codebase. Marking it Draft — it is feature-complete for the P1 scope and live-tested locally, but kept in draft until we pick it back up for review/merge (we're moving to other topics for now). What's in the branch (
|
| Story | What landed |
|---|---|
| US1 Deterministic engine | @omadia/conductor-core — I/O-free predicate-AST engine (serializable guards/postconditions, no eval), validate(graph) shape+graph checks, deterministic nextStep(). 46 vitest green. |
| US2 Durable runs | migrations/0001_conductor.sql (full conductor_* schema + claim cols + pg_notify), per-subsystem migrator, ConductorWorkflowStore (versioned publish), ConductorRunStore (durable per-step record), ConductorRunExecutor with a StepEffects seam; async background drive + UI polling (real agent turns are slow). |
| US3 Manual triggers | Operator API /api/v1/operator/conductors (publish/list/status/run/trace) behind requireAuth. |
| US4 Event triggers | ConductorEventRouter.emit(eventId, payload) → matches active versions' event trigger (+ payload-filter predicate) → starts a run per match with payload as context. Operator POST /emit + UI form. (Connector half — plugin ctx.events.emit + manifest emits: autodiscovery — is the documented remaining piece.) |
| US5 Human-in-the-loop | Durable conductor_awaits; human step parks run waiting; resolveAwait records response + atomically closes + resumes; expireAwait fires in-graph fallback on deadline; minute-tick ConductorAwaitWorker; operator inbox (Approve/Reject). |
| US6 Roles & baton | ConductorRoleStore (create / add-remove holder = the baton / live resolve()); awaits inbox resolves role:<key> to current holders live; "Roles & the baton" UI. |
| US7 Visual designer | ConductorCanvas (@xyflow/react): add agent/action/human nodes, drag-wire transitions, per-kind inspector, trigger config, load existing workflow, Save→validate→publish, Run+poll. + NEW: per-row "Edit" button in the list loads a workflow into the canvas (commit 62edeaf). |
| US8 Dry-run / preview | executor.previewRun(...) simulates the path in-memory with zero side effects (no runs/awaits rows, no notifications; irreversible connector actions stubbed); "Dry-run" button renders the simulated step path. |
Real execution = real orchestrator turns: an agent step targets an Agent (orchestrator instance) resolved by slug from the multi-orchestrator registry (RealStepEffects), answer.text is the step result; action steps invoke the real connector tool. No stubbed execution.
Verification
conductor-core: 46/46 vitest green; middlewaretsc --noEmitclean; web-uitsc+ i18n parity (1279 keys, en/de) clean.- Deployed + live-tested in the local
omadia-testdocker stack (~/sources/omadia, same commit): migrations apply, 11conductor_*tables, route auth-gated. End-to-end smokes via Interceptor (authenticated Chrome,:3333 → Conductor): publish + run (real gpt-5.5 output persisted), event emit → auto-run, human await → Approve → resume → complete, role baton re-resolution, dry-run (no DB writes), and the new Edit flow (loadsvisual-demointo the canvas).
Known limitations / remaining work (future phases, not in scope here)
- US4 connector half — plugin
ctx.events.emit+ manifestemits:autodiscovery +EventCatalogRegistryon both activation paths (landmines B/K). - US9 audit viewer — Conductor-shaped branch of
RunTraceVieweroverconductor_run_steps. - US7 conversational builder (chat co-design) — only the visual canvas exists.
- Reminders (proactive channel notify) — needs the
conductor_channel_bindingssender wired. - Run-resume worker for async/
runningruns after a restart (waiting-on-human already survives via durable awaits). - Cron triggers actually firing (
conductor_schedules+ScheduleWorkertick); quorumall.
Notes
- All implementation commits authored as
Marcel Wege <mwege@byte5.de>. - Leftover demo workflows in the
omadia-testDB are harmless local test data.
…is the fallback Align Spec 005 with the Identity & Role Projection direction (#333) before the local-only role model gets cemented: - Principal = user|role is the platform-wide DEFAULT addressee for any surface targeting a person; user-only is the justified exception (FR-020). - The primary RoleResolver projects holders from external systems of record (Entra groups/app-roles, HR/ERP) correlated on a primary key; the local conductor_role_assignments store is the default / stand-alone fallback, not the long-term system of record (FR-021, US6, data-model). - Note the shipped migration's session-identity (sub/email) holder column. This matches the seam roleStore.ts already anticipates ("external resolver in front; follow-up"). No behavior change — spec/doc only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@Weegy heads-up on the role model while you're building this out 👇 Spec 005 was just updated (commit 420694c) to align with #333 (Identity & Role Projection). Key point for your
Nothing urgent to change — your seam is already shaped right, and migration |
Summary
Conception for Omadia Conductor — a deterministic workflow/process layer where the
runtime, not the LLM, owns step progression and hand-offs. It promotes the existing
per-tool atoms (postconditions + verifier, OB-31 tool-obligation / repeat-failure
guards, the
deterministic_actionfast-path) to process scope, so a multi-step,multi-agent process cannot silently stall the way prompt-only frameworks do.
A workflow is a graph of steps (agentic, deterministic action, or human)
connected by guarded transitions. After each step the Conductor evaluates an exit
postcondition and, when unmet, acts deterministically (re-inject / force a tool
obligation / fire a declared fallback transition) instead of hoping the LLM
self-corrects. A hand-off is a transition the runtime fires, not a prompt line an Agent
can drop.
What's in this PR
Docs only — two spec artifacts under
specs/005-omadia-conductor/. No code changes.Status: Draft.
spec.md— 9 user stories (P1: engine, durable run lifecycle/resume, triggers, eventtriggers + connector "Conductor Surface", human steps with durable awaits/reminders/
deadline, principals & role resolver seam; P2: visual+conversational Designer, dry-run;
P3: audit), 32 functional requirements, 10 success criteria, edge cases, assumptions.
data-model.md— schema for workflow / version / draft, run / run-step, the net-newdurable
conductor_awaits(+ per-holder responses), roles / role-assignments (the"baton"), the manifest
emits:extension,LISTEN/NOTIFYresume hook, and therun/await state machines.
Key decisions captured (clarified with the owner)
@omadia/conductor-core(pure engine) + kernel wiring via theexisting
serviceRegistry+ a Designer underweb-ui/app/admin/conductor/. Noseparate repo; only an HR/ERP role resolver belongs in a swappable plugin.
handler).
quorum: any | all(defaultany) for multi-holder roles.emits:block, eventcatalog,
ctx.events.emit, subscription/filter) ships; connectors themselves are outof scope.
pluggable
RoleResolver; await access follows the current holder at access time (thebaton can move mid-wait).
Scope boundary
Conductor defines and depends only on the contract a connector implements
(
emits:/provides:/ event catalog). Building connectors (GitHub/CI, ATS/HR,calendar, ERP) and the role-movement policy are owned by separate plugin work and the
live instance.
🤖 Generated with Claude Code