Repositories, routing and triggers: runs start from a git push

README.md

@@ -5,7 +5,8 @@ where every job runs in a fresh virtual machine.
 
 > **Status:** early development. A pipeline runs end to end: the controller schedules its jobs, a worker checks
 > out the exact commit and runs each job's commands through the agent with the secrets the admin allows, and
-> the output is stored. Real VMs and Git providers come next; see [docs/ROADMAP.md](docs/ROADMAP.md).
+> the output is stored. Runs start from a `git push`, to your own git server or to GitHub. Real VMs and
+> Opifica-hosted repositories come next; see [docs/ROADMAP.md](docs/ROADMAP.md).
 > See the [organisation page](https://github.com/opifica) for the architecture.
 
 ## What lives here
@@ -14,15 +15,17 @@ where every job runs in a fresh virtual machine.
 |---|---|---|
 | `internal/pipeline` | The pipeline language: YAML parser, validation, CEL conditions, matrix expansion, planning | Working |
 | `schema/` | JSON Schema for pipeline files, for editor autocompletion | Working |
-| `cmd/opifica` | CLI: `validate`, `plan`, `schema` | Working |
+| `cmd/opifica` | CLI: `validate`, `plan`, `schema`, `trigger`, `hook` | Working |
 | `proto/` | API definitions: worker ↔ controller, worker ↔ guest agent (Protobuf, ConnectRPC) | Draft |
 | `pkg/worker` | Shared worker kit: `Hypervisor` interface, warm VM pool, controller client, Git mirrors, job runner, `worker.Run` | Working |
 | `pkg/worker/workertest` | Fake hypervisor for testing worker code without VMs | Working |
 | `cmd/controller` | Join tokens, worker joining over TLS, the Connect stream, worker revocation | Working |
 | `internal/controller/scheduler` | Runs and jobs, matching jobs to workers, leases, retries, cancellation | Working |
 | `internal/controller/logs` | Job output on the controller's disk | Working |
 | `internal/controller/secrets` | Encrypted secrets, handed only to jobs the admin's policy allows; every request audited | Working |
-| | Git providers, admin API | Not started |
+| `internal/controller/repos` | Repositories, routing rules (event + ref → pool), trigger tokens; the trigger API | Working |
+| `cmd/opifica` `trigger`, `hook` | Start runs from GitHub Actions, or from the git hook of a bare repository | Working |
+| | Hosted repositories (opifica-git), admin API | Not started |
 | `cmd/agent`, `internal/agent` | Runs inside each VM: checkout, running steps, masking secrets in output, killing process trees on cancel and timeout | Working |
 | `cmd/devworker` | A worker without VMs for trying Opifica on one machine (no isolation) | Working |
 | `cmd/signer` | Signing service for release artifacts | Not started |
@@ -140,13 +143,53 @@ bin/controller serve -addr 127.0.0.1:8443 &       # TLS; keep it running
 bin/devworker join -controller https://127.0.0.1:8443 -token opj_... -ca-fingerprint sha256:...
 bin/devworker run  -controller https://127.0.0.1:8443 &
 
-# Until Git providers exist, start runs by hand:
-bin/controller run create -repo acme/app -clone-url "$PWD" -pool trusted \
-    -sha "$(git rev-parse HEAD)" -file .opifica/pipeline.yaml
-bin/controller run show <run-id>
+# Register a repository; prints its trigger token
+bin/controller repo create acme/app -clone-url "$PWD" -route push:main=trusted -route '*=untrusted'
+
+# Start a run for the current branch and commit, and follow it
+export OPIFICA_CONTROLLER=https://127.0.0.1:8443 OPIFICA_CA_FINGERPRINT=sha256:... \
+       OPIFICA_REPO=acme/app OPIFICA_TRIGGER_TOKEN=opr_...
+bin/opifica trigger -wait
+
+bin/controller run list
 bin/controller run logs <run-id>
 ```
 
+The controller needs `-allow-local-repos` to build from a local path like `$PWD`.
+
+### Triggering runs
+
+A run starts when a ref moves: a push to a branch, a tag, or a pull request. The controller reads
+`.opifica/pipeline.yaml` at that commit itself, and the repository's **routes** decide the pool. The first matching
+route wins, and an event no route matches starts nothing:
+
+```sh
+bin/controller repo create acme/app -clone-url ssh://git@git.example/acme/app.git \
+    -route push:main=trusted -route tag:v*=release -route pull_request=untrusted
+```
+
+**Your own git server.** Install the hook in the bare repository as `hooks/post-receive`:
+
+```sh
+#!/bin/sh
+export OPIFICA_CONTROLLER=https://ci.example:8443 OPIFICA_CA_FINGERPRINT=sha256:... OPIFICA_REPO=acme/app
+export OPIFICA_TRIGGER_TOKEN="$(cat /etc/opifica/acme-app.token)"
+exec opifica hook post-receive
+```
+
+Each push then prints the runs it started:
+
+```console
+$ git push origin main
+remote: opifica: main: run 0b6f… started in pool trusted (3 jobs)
+```
+
+Opifica-hosted repositories, where creating a repository sets this up for you, are on the roadmap.
+
+**GitHub.** Add [`examples/github-actions.yml`](examples/github-actions.yml) as a workflow. Its step runs
+`opifica trigger -wait`, which passes or fails with the Opifica run and prints its output. For a private
+repository, register it with `-credential-stdin` and a read-only token.
+
 ### Secrets
 
 The admin decides who gets a secret; a pipeline can only ask for it. The value is read from standard input:

cmd/controller/main.go

@@ -1,9 +1,10 @@
-// Command controller runs the Opifica controller: the API, Git provider
-// integration, pipeline engine, scheduler and secrets broker.
+// Command controller runs the Opifica controller: the API, the repository
+// registry and trigger API, pipeline engine, scheduler and secrets broker.
 //
 // Implemented so far: the database, the built-in certificate authority, join
 // tokens, Join, the Connect stream, the scheduler (runs, jobs, leases,
-// retries, cancellation), job logs and secrets. Git providers come next.
+// retries, cancellation), job logs, secrets, and repositories with routing
+// and triggers.
 package main
 
 import (
@@ -23,13 +24,16 @@ import (
 
 	"github.com/google/uuid"
 
+	opificav1 "github.com/opifica/opifica/gen/opifica/v1"
 	"github.com/opifica/opifica/internal/controller/api"
 	"github.com/opifica/opifica/internal/controller/ca"
 	"github.com/opifica/opifica/internal/controller/logs"
+	"github.com/opifica/opifica/internal/controller/repos"
 	"github.com/opifica/opifica/internal/controller/scheduler"
 	"github.com/opifica/opifica/internal/controller/secrets"
 	"github.com/opifica/opifica/internal/controller/store"
 	"github.com/opifica/opifica/internal/controller/workers"
+	"github.com/opifica/opifica/internal/gitmirror"
 	"github.com/opifica/opifica/internal/server"
 	"github.com/opifica/opifica/internal/version"
 )
@@ -41,7 +45,11 @@ Commands:
   token create   Create a one-time join token for a new build machine
   worker list    List workers
   worker revoke  Revoke a worker; it is disconnected and can't reconnect
-  run create     Start a run from a pipeline file (until Git providers exist)
+  repo create    Register a repository with its routing rules; prints a trigger token
+  repo list      List repositories
+  repo delete    Delete a repository
+  repo token create  Create another trigger token for a repository
+  run create     Start a run from a local pipeline file (for testing)
   run list       List recent runs
   run show       Show a run and its jobs
   run cancel     Cancel a run
@@ -83,6 +91,8 @@ func runWithInput(args []string, stdin io.Reader, stdout, stderr io.Writer) int
 		return runCommand(args[1], args[2:], stdout, stderr)
 	case args[0] == "secret" && len(args) > 1:
 		return secretCommand(args[1], args[2:], stdin, stdout, stderr)
+	case args[0] == "repo" && len(args) > 1:
+		return repoCommand(args[1], args[2:], stdin, stdout, stderr)
 	case args[0] == "version":
 		fmt.Fprintln(stdout, version.Version)
 		return 0
@@ -130,6 +140,7 @@ func serve(args []string, stderr io.Writer) int {
 	var c common
 	c.register(fs)
 	addr := fs.String("addr", ":8443", "address to listen on")
+	allowLocal := fs.Bool("allow-local-repos", false, "allow repositories whose clone URL is a local path (hosted on this machine, or development)")
 	hosts := fs.String("hosts", defaultHosts(), "comma-separated names and IPs workers use to reach the controller")
 	if err := fs.Parse(args); err != nil {
 		return 2
@@ -158,13 +169,23 @@ func serve(args []string, stderr io.Writer) int {
 		log.Error("loading secrets key", "err", err)
 		return 1
 	}
+	logStore := &logs.Store{Dir: filepath.Join(c.dataDir, "logs")}
+	var reposSvc *repos.Service
 	sched := scheduler.New(svc.Store, svc.Sessions(), log.With("component", "scheduler"), scheduler.Config{
-		Logs:    &logs.Store{Dir: filepath.Join(c.dataDir, "logs")},
+		Logs:    logStore,
 		Secrets: &secrets.Manager{Store: svc.Store, Key: key},
+		Credentials: func(ctx context.Context, run uuid.UUID) (*opificav1.Credential, error) {
+			return reposSvc.RunCredential(ctx, run)
+		},
 	})
+	reposSvc = &repos.Service{
+		Store: svc.Store, Key: key, Scheduler: sched,
+		Mirrors: &gitmirror.Mirrors{Dir: filepath.Join(c.dataDir, "mirrors"), AllowLocal: *allowLocal},
+	}
 	svc.Messages = sched
 	go sched.Run(ctx)
-	if err := server.RunTLS(ctx, *addr, api.NewHandler(svc), tlsCfg, log); err != nil {
+	handler := api.NewHandler(api.Deps{Workers: svc, Repos: reposSvc, Logs: logStore})
+	if err := server.RunTLS(ctx, *addr, handler, tlsCfg, log); err != nil {
 		log.Error("server stopped", "err", err)
 		return 1
 	}

cmd/controller/main_test.go

@@ -66,7 +66,7 @@ func TestTokenCreateAndJoinOverTLS(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	go server.ServeTLS(ctx, ln, api.NewHandler(svc), tlsCfg, svc.Log)
+	go server.ServeTLS(ctx, ln, api.NewHandler(api.Deps{Workers: svc}), tlsCfg, svc.Log)
 
 	// The worker has no CA yet: it accepts the controller only if the
 	// certificate chains to a CA with the pinned fingerprint.

cmd/controller/repos.go

@@ -0,0 +1,204 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"flag"
+	"fmt"
+	"io"
+	"log/slog"
+	"path/filepath"
+	"strings"
+	"text/tabwriter"
+	"time"
+
+	"github.com/opifica/opifica/internal/controller/repos"
+	"github.com/opifica/opifica/internal/controller/scheduler"
+	"github.com/opifica/opifica/internal/controller/secrets"
+	"github.com/opifica/opifica/internal/gitmirror"
+)
+
+func repoCommand(sub string, args []string, stdin io.Reader, stdout, stderr io.Writer) int {
+	switch sub {
+	case "create":
+		return repoCreate(args, stdin, stdout, stderr)
+	case "list":
+		return repoList(args, stdout, stderr)
+	case "delete":
+		return repoDelete(args, stdout, stderr)
+	case "token":
+		if len(args) > 0 && args[0] == "create" {
+			return repoTokenCreate(args[1:], stdout, stderr)
+		}
+	}
+	fmt.Fprintf(stderr, "controller: unknown command \"repo %s\"\n\n%s", sub, usage)
+	return 2
+}
+
+func withRepos(c common, stderr io.Writer, fn func(ctx context.Context, r *repos.Service) error) int {
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+	svc, closeFn, err := c.open(ctx)
+	if err != nil {
+		fmt.Fprintf(stderr, "controller: %v\n", err)
+		return 1
+	}
+	defer closeFn()
+	key, err := secrets.LoadOrCreateKey(c.dataDir)
+	if err != nil {
+		fmt.Fprintf(stderr, "controller: loading secrets key: %v\n", err)
+		return 1
+	}
+	quiet := slog.New(slog.NewTextHandler(io.Discard, nil))
+	r := &repos.Service{
+		Store: svc.Store, Key: key,
+		Scheduler: scheduler.New(svc.Store, svc.Sessions(), quiet, scheduler.Config{}),
+		Mirrors:   &gitmirror.Mirrors{Dir: filepath.Join(c.dataDir, "mirrors")},
+	}
+	if err := fn(ctx, r); err != nil {
+		fmt.Fprintf(stderr, "controller: %v\n", err)
+		return 1
+	}
+	return 0
+}
+
+func repoCreate(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
+	fs := flag.NewFlagSet("repo create", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	var c common
+	c.register(fs)
+	cloneURL := fs.String("clone-url", "", "where the controller and workers fetch the code (required)")
+	var routes listFlag
+	fs.Var(&routes, "route", "event[:ref]=pool, first match wins (repeatable; default *=untrusted), e.g. push:main=trusted")
+	credStdin := fs.Bool("credential-stdin", false, "read a read-only clone token from standard input")
+	fs.Usage = func() {
+		fmt.Fprint(stderr, `Usage: controller repo create [options] OWNER/NAME
+
+Registers a repository and prints its first trigger token.
+
+Routes decide which pool runs each event; the first matching route wins and
+an event no route matches starts no run. For example:
+
+  -route push:main=trusted -route tag:v*=release -route '*=untrusted'
+
+Options:
+`)
+		fs.PrintDefaults()
+	}
+	pos, err := parseWithArgs(fs, args)
+	if err != nil {
+		return 2
+	}
+	if len(pos) != 1 || *cloneURL == "" {
+		fs.Usage()
+		return 2
+	}
+	var parsed []repos.Route
+	for _, r := range routes {
+		route, err := repos.ParseRoute(r)
+		if err != nil {
+			fmt.Fprintf(stderr, "controller: %v\n", err)
+			return 2
+		}
+		parsed = append(parsed, route)
+	}
+	var cred []byte
+	if *credStdin {
+		b, err := io.ReadAll(io.LimitReader(stdin, 16<<10))
+		if err != nil || len(bytes.TrimSpace(b)) == 0 {
+			fmt.Fprintln(stderr, "controller: -credential-stdin: no token on standard input")
+			return 2
+		}
+		cred = bytes.TrimSpace(b)
+	}
+	return withRepos(c, stderr, func(ctx context.Context, r *repos.Service) error {
+		repo, err := r.Create(ctx, pos[0], *cloneURL, parsed, cred)
+		if err != nil {
+			return err
+		}
+		token, err := r.CreateToken(ctx, repo.Name, "created with the repository")
+		if err != nil {
+			return err
+		}
+		fmt.Fprintf(stdout, "Repository %s registered.\n\nRoutes (first match wins):\n", repo.Name)
+		for _, rt := range repo.Routes {
+			fmt.Fprintf(stdout, "  %s\n", rt)
+		}
+		fmt.Fprintf(stdout, "\nTrigger token (shown once; store it as OPIFICA_TRIGGER_TOKEN):\n\n  %s\n", token)
+		return nil
+	})
+}
+
+func repoList(args []string, stdout, stderr io.Writer) int {
+	fs := flag.NewFlagSet("repo list", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	var c common
+	c.register(fs)
+	if err := fs.Parse(args); err != nil {
+		return 2
+	}
+	return withRepos(c, stderr, func(ctx context.Context, r *repos.Service) error {
+		list, err := r.List(ctx)
+		if err != nil {
+			return err
+		}
+		tw := tabwriter.NewWriter(stdout, 0, 4, 2, ' ', 0)
+		fmt.Fprintln(tw, "NAME\tCLONE URL\tCREDENTIAL\tROUTES")
+		for _, repo := range list {
+			var rs []string
+			for _, rt := range repo.Routes {
+				rs = append(rs, rt.String())
+			}
+			cred := "no"
+			if repo.HasCredential {
+				cred = "yes"
+			}
+			fmt.Fprintf(tw, "%s\t%s\t%s\t%s\n", repo.Name, repo.CloneURL, cred, strings.Join(rs, " "))
+		}
+		return tw.Flush()
+	})
+}
+
+func repoDelete(args []string, stdout, stderr io.Writer) int {
+	fs := flag.NewFlagSet("repo delete", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	var c common
+	c.register(fs)
+	pos, err := parseWithArgs(fs, args)
+	if err != nil || len(pos) != 1 {
+		fmt.Fprintln(stderr, "Usage: controller repo delete [options] OWNER/NAME")
+		return 2
+	}
+	return withRepos(c, stderr, func(ctx context.Context, r *repos.Service) error {
+		n, err := r.Store.DeleteRepository(ctx, pos[0])
+		if err != nil {
+			return err
+		}
+		if n == 0 {
+			return fmt.Errorf("no repository named %s", pos[0])
+		}
+		fmt.Fprintf(stdout, "Repository %s deleted; its trigger tokens no longer work. Past runs are kept.\n", pos[0])
+		return nil
+	})
+}
+
+func repoTokenCreate(args []string, stdout, stderr io.Writer) int {
+	fs := flag.NewFlagSet("repo token create", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	var c common
+	c.register(fs)
+	desc := fs.String("description", "", "where the token is used, e.g. github-actions")
+	pos, err := parseWithArgs(fs, args)
+	if err != nil || len(pos) != 1 {
+		fmt.Fprintln(stderr, "Usage: controller repo token create [options] OWNER/NAME")
+		return 2
+	}
+	return withRepos(c, stderr, func(ctx context.Context, r *repos.Service) error {
+		token, err := r.CreateToken(ctx, pos[0], *desc)
+		if err != nil {
+			return err
+		}
+		fmt.Fprintf(stdout, "Trigger token for %s (shown once):\n\n  %s\n", pos[0], token)
+		return nil
+	})
+}