axon

A small, single-host DAG runner for typed tasks.

You describe a workflow as a graph of tasks. axon runs each task as soon as its dependencies finish, in parallel where the graph allows, threads each task's output to its dependents, and hands you back every output when it's done. It does not render reports or know about any domain — that's the caller's job.

network ─▶ clickhouse ─▶ await ─▶ prepare ─▶ connection ─▶ source

There are two ways to use it:

• As a binary — define task folders and a workflow.yaml, then run the generic axon runner. No orchestration code.
• As a library — build a Registry and Workflow in Go and drive the Engine yourself (embed it in your own program).

The one idea: the Runner seam

The only contract between the engine and a task is bytes in, bytes out:

type Runner interface {
    Run(ctx context.Context, input []byte) (output []byte, err error)
}

Because the seam is just bytes, a task can run in-process (linked Go code) or out-of-process (a subprocess that reads stdin / writes stdout) and the engine can't tell the difference. That's what makes tasks isolated, individually runnable, and polyglot.

Engine ── input JSON ──▶ Runner ──▶ output JSON
                           ├── in-process : call a Go func
                           └── subprocess : exec a binary (axon.SubprocessRunner)

Tasks are self-contained folders

A task is a folder carrying everything about it — its contract (proto), how to run it (task.yaml), and its source:

tasks/create_network/
  task.proto    # the contract: Input (params + deps) and Output
  task.yaml     # descriptor: name, description, revertable, run overrides
  src/main.go   # the implementation (compiled to ./task)
  task          # the built binary (gitignored; built separately)

The implementation is just one typed function — the SDK handles stdin/stdout, encoding, validation, and the result envelope:

type createNetwork struct{}

func (createNetwork) Execute(ctx context.Context, in *networkpb.Input) (*networkpb.Output, error) {
    name := in.GetNetworkName()
    if err := dock.NetworkCreate(ctx, name); err != nil {
        return nil, err
    }
    return &networkpb.Output{NetworkName: name}, nil
}

// Optional: rolled back (in reverse order) if the run fails.
func (createNetwork) Revert(ctx context.Context, in *networkpb.Input, out *networkpb.Output) error {
    return dock.NetworkRemove(ctx, out.GetNetworkName())
}

func main() { sdk.RunProto[*networkpb.Input, *networkpb.Output](createNetwork{}) }

Run a task standalone — the same way the engine does — for debugging or repro:

$ echo '{"networkName":"demo"}' | ./tasks/create_network/task
{"result":{"status":"OK","output":{"networkName":"demo"}}}

The result envelope

stdout is the protocol channel: a task always emits a Result envelope and exits 0; task errors and panics are carried in the envelope, not the exit code. A non-zero exit is reserved for a hard crash. The SDK does all of this; the engine validates and unwraps output before handing it to dependents.

{"result": {"status": "OK", "output": { ... }}}
{"result": {"status": "FAILED", "error": "..."}}

Logs go to stderr as structured JSON (sdk.Logger()), tagged with the run identity the engine injects — stdout stays clean for the envelope.

Proto contracts derive the graph

A task's .proto is the schema, not the wire — input/output still travel as JSON; the proto is used to validate that JSON and to derive dependencies:

package clickhouse_container;
import "create_network/task.proto";

message Input {
  string container_name = 1;            // scalar -> a param
  create_network.Output network = 2;    // another task's Output -> a dependency
}
message Output { string container_name = 1; string host = 2; }

A singular message field typed as another task's Output is a dependency; scalars/maps/repeated are params. Wiring is by type (the Output message's fully-qualified proto name) and resolved per node: a contract field is filled by the node whose task provides its type. The engine reads the graph straight off the schemas, so a workflow needs no after: for data dependencies — only for pure ordering.

Tasks are definitions, nodes are instances. The same task may back any number of nodes in one workflow (e.g. one smoke-test task pointed at two deployments). Auto-wiring needs exactly one node providing the field's type; with several, the run fails pre-flight naming the candidates, and the workflow picks one explicitly with a bare-node reference:

input:
  network: staging_net        # bind staging_net's whole Output to this field
  container_name: db.name     # <node>.<field> still pulls a single value

Two different tasks may also provide the same Output type — they are interchangeable providers (a registry task can stand in for a local one that speaks the same contract). The engine enforces that a shared type name means a shared shape: same name with mismatched schemas is rejected before anything runs.

Workflows are YAML

A workflow references tasks by folder path (relative to the workflow file) and supplies params + ordering. The registry is derived from the folders it references — there is no separately declared registry.

name: full
nodes:
  - name: network
    task: ../tasks/create_network
    description: shared docker network
    params:
      network_name: demo

  - name: clickhouse
    task: ../tasks/clickhouse_container
    after: [network]          # ordering; the network dep is derived from the contract
    params:
      container_name: demo-clickhouse

Run it: the axon binary

# build the task binaries however you like (e.g. a Makefile), then:
$ axon -workflow workflows/full.yaml
$ axon -workflow workflows/full.yaml -serve              # + web UI on :9988
$ axon -workflow workflows/full.yaml -inject-fail tasks/boom   # exercise rollback

The runner: loads the workflow, compiles each task's .proto at runtime to derive the registry, validates the whole graph, runs it with a live progress table, and (with -serve) serves the built-in web UI. It only runs tasks — building their binaries is your job (so tasks can be any language).

full

+------------+--------------------------+--------+---------+
| Task       | Description              | Status | Elapsed |
+------------+--------------------------+--------+---------+
| network    | shared docker network    | Done   | 0.012s  |
| clickhouse | demo-clickhouse          | Done   | 1.240s  |
+------------+--------------------------+--------+---------+

Built-in web UI

-serve starts a gin server with an embedded SPA (Vue + Vite + Tailwind, in ui/). It serves a live, in-memory task table that polls /api/run, alongside the console progress table (which shows the serve URL in its header). After the run it lingers for -serve-linger (default 30s; 0 exits immediately, negative waits for Ctrl-C) so you can inspect the result, then the process exits on its own. Default :9988, override with -serve-addr.

Task stderr is always captured off-terminal (it would otherwise interleave with the progress table's in-place redraws): the web UI's logs view reads it live, and on failure the errored nodes' logs are printed after the summary.

The SPA is a build artifact: make ui compiles ui/ into internal/web/static/dist, which is embedded via //go:embed. dist is not committed (only a .gitkeep placeholder is), so run make ui before building a release binary or using -serve.

Using it as a library

Everything the binary does is also a small API you can embed:

l, _ := loader.Load("workflows/full.yaml")   // registry derived from task folders
engine := axon.New(l.Registry, axon.WithLogSink(perTaskLogFile))

if err := engine.Validate(l.Workflow); err != nil {   // pre-flight: types, deps, params
    log.Fatal(err)
}

run := engine.Start(ctx, l.Workflow)
go web.ListenAndServe(":9988", run, l.Workflow, l.Descriptions, l.Registry, nil)  // optional UI

for ev := range run.Events() {       // live stream
    log.Printf("%s -> %s", ev.Task, ev.Kind)
}
result := run.Wait()                 // every node's status, timing, output

Run exposes Events() (live), Snapshot() (point-in-time table), and Wait() (the final Result). You can also register tasks by hand (reg.Register(name, runner) / sdk.RegisterProto[...]) and run plain in-process Go tasks (sdk.InProcess) without the folder/loader model — see examples/.

Layout

path	what
engine.go run.go	the scheduler, options, Start/Run/Validate, live state, revert
runner.go	the Runner / Reverter seam + Input envelope
registry.go workflow.go	task registry, Workflow/Node + validation
subprocess.go	SubprocessRunner + RevertableSubprocessRunner (out-of-process)
log.go	per-node log routing (engine side)
contract/	proto reflection: provided/required Output types, dep derivation, schema compatibility, the Result envelope, validation
sdk/	task authoring: ProtoTask/RevertableProto, Serve/RunProto/ProtoMain, Logger
loader/	runtime .proto compilation; build a registry from task folders
ui/	the web UI source (Vue + Vite); make ui builds it into the embed dir
internal/web/	built-in gin server + embedded SPA (static/dist, a build artifact)
internal/ui/	the console progress table
cmd/axon/	the generic workflow runner binary
proto/ gen/	example task schemas and their generated Go (via buf)
examples/	library-embedding examples (in-process and subprocess)

What works today

• Single-host, in-memory scheduling with parallelism; live events/snapshots.
• One Runner seam, two execution models (in-process and subprocess), mixable in one workflow.
• Proto contracts: dependencies derived from the schema by Output type and resolved per node — the same task can back many nodes, and same-typed tasks are interchangeable providers; params and outputs validated via protojson (JSON stays the wire format).
• Result envelope + always-exit-0 task wrapper; the engine treats a non-zero exit as a hard failure.
• Revert/rollback on failure, in reverse completion order — in-process and across the subprocess boundary (--revert). Opt-in per task.
• Pre-flight Validate: unknown tasks, unsatisfied or ambiguous contract fields (with the explicit-binding fix named in the error), same-type-name schema conflicts, cycles, and per-node params checked against the schema before anything runs.
• Runtime .proto compilation (pure Go, no protoc): the registry is derived from the task folders a workflow references — no declared registry.
• Structured logging to stderr (sdk.Logger) with engine-side per-task routing (WithLogSink).
• Built-in web UI (-serve) and a generic runner binary (cmd/axon).
• Tested with go test -race ./....

Deliberately out of scope

No database, no multi-host, no durable coordinator. axon is an embeddable runner, not an orchestration service.

Development

make ui      # build the web UI (Vue + Vite + Tailwind) into internal/web/static/dist
make build   # rebuild the UI, then compile the runner into bin/axon
make test    # go test ./...
buf generate # regenerate example proto Go after editing a .proto