Go prototype: deliver Ticket runs and repair failing heads

prototype/go/README.md

@@ -1,9 +1,9 @@
 # Go Ticket-run prototype
 
-An isolated, Linux-only experiment for #1830 / #1832 / #1834. The production Rust
-binary and ADR 0001 stay as they are. This slice stops after the opening
-Harness session, before Delivery, CI watch or Repair. Pickup can select and
-reserve a Ready issue and start that same Ticket run.
+An isolated, Linux-only experiment for #1830 / #1832 / #1833 / #1834. The production
+Rust binary and ADR 0001 stay as they are. This slice runs a Ticket run through
+Delivery, CI watch and bounded Repair sessions. Pickup can select and reserve
+a Ready issue and start that same Ticket run.
 
 Requires Go (the version in `go.mod`), Python 3 for tests/generation, Git,
 authenticated `gh`, and Claude Code (`claude`) for real sessions. Tests use
@@ -22,6 +22,8 @@ Run it from the Launch directory of a disposable repository:
 
 ```sh
 /tmp/thirdshift-go https://github.com/owner/repo/issues/7
+# Lower the default maximum of five Repairs (zero disables Repairs):
+/tmp/thirdshift-go --repair-limit 2 https://github.com/owner/repo/issues/7
 /tmp/thirdshift-go pickup
 ```
 
@@ -52,13 +54,52 @@ malformed events, missing completion, error results, and process failures fail
 the Ticket run. `SIGINT` or `SIGTERM` interrupts its current child process
 group: SIGINT, a two-second grace, SIGTERM, another two seconds, then SIGKILL.
 If the leader exits early, remaining group members are killed immediately.
-Interruption exits 130. Claim ending uses its own bounded context so it still
-finishes after interruption: a closed issue loses its Claim; an open issue
+Interruption exits 130. Failed ticket run salvage and Claim ending each use
+their own bounded context so they still finish after interruption. Salvage
+aborts an unfinished merge and pushes committed work ahead of the Base branch
+to a draft pull request. A closed issue loses its Claim; an open issue
 with no Issue branch or open PR on origin has only this run's label changes
 restored (`ready-for-agent` only if it was present before acquisition, and
 `in-progress` removed only if this run added it). Work left on origin keeps
 its Claim, with `agent-failed` on failed or interrupted runs.
 
+## Delivery and Repair
+
+The opening Harness commits locally; thirdshift pushes the Issue branch and
+opens a draft pull request targeting the selected Base branch. It captures
+the PR number and verifies its open state, repository, branches and published
+head before watching CI or changing readiness. A closed or merged PR stops
+Delivery and salvage without reopening it. Existing same-repository PRs are
+verified and reused. A fork pull request for the same branch name is ignored.
+An opening session with no commits ahead of the Base branch ends without Delivery.
+
+Delivery fetches and merges the Base branch, pushes, then watches check runs
+and commit statuses for the full published head SHA. Pending results are polled.
+An empty or passing listing during the first five seconds stays provisional, so
+a check that registers after a faster one still counts. After that grace,
+passing or absent CI plus a mergeable PR permits readiness. A workflow run that
+is still open on the head stays pending even when the checks already listed
+succeeded, and a completed failing run is red when its jobs never appeared.
+A previously seen set of checks disappearing cannot establish green CI. Inherited failures fail
+the run without spending a Repair on the Base branch's code.
+
+A Base branch conflict or failed head checks start a Repair in the same Worktree
+with refreshed trusted issue context. Repairs commit their changes, including
+any conflicted merge, before Delivery pushes and checks the new head. There are
+at most five Repairs (lowered by `--repair-limit`) and ten reconciliation rounds.
+An unchanged Repair head fails; this prototype does not yet implement the Rust
+runner's Check re-run or Abandoned job re-run policies (ADRs 0010 and 0020).
+Failed Delivery drafts any existing PR before attempting a salvage push, so
+a rejected push still leaves it as a draft. It preserves the Claim and
+adds `agent-failed`; work that could not be pushed remains in the Worktree.
+A draft-conversion error is reported while salvage still attempts to push
+committed work, then attempts draft conversion again.
+
+Run outcomes, Claim transitions, Delivery states and CI states use typed enums.
+`go test ./...` type-checks the production syntax tree and rejects any switch
+that does not explicitly cover every enum constant, including switches with
+a default. Checker fixtures demonstrate missing and newly added variants fail.
+
 ## Pickup
 
 `pickup` lists every page of open `ready-for-agent` issues and tries the lowest
@@ -70,9 +111,10 @@ labels, Base/Security fixes, sub-issues, open blockers, and issues shaped within
 30 minutes. Existing origin Issue branches or PRs in any state require a hand-back.
 
 Only Ticket runs are supported: issues with sub-issues (Specs) are passed over.
-This experiment has no production Pass limits, Sweep, CI checks or notifications,
+This experiment has no production Pass limits, Sweep or notifications,
 and inherits the first slice's refusal to resume an existing local branch or
-Worktree. It uses the checked-out Base branch and takes no Pickup options.
+Worktree. It uses the checked-out Base branch and takes no Pickup-specific
+options. The dispatched Ticket run uses the same `--repair-limit` as a direct run.
 
 For each candidate, ADR-0017 creates `refs/heads/thirdshift/pickup/<number>` on
 origin with an empty create lease and a unique, parentless, empty-tree commit.

prototype/go/ci.go

@@ -0,0 +1,149 @@
+package main
+
+import (
+	"context"
+	"fmt"
+)
+
+type ciObservation struct {
+	state  ciState
+	failed []string
+}
+
+// Read checks and commit statuses for the full published SHA, never the latest
+// workflow on a branch. Only the latest status for each context counts.
+func readCI(ctx context.Context, repo, head string) (ciObservation, error) {
+	var pages []struct {
+		CheckRuns []struct{ Name, Status, Conclusion string } `json:"check_runs"`
+	}
+	if err := ghJSON(ctx, &pages, "api", "repos/"+repo+"/commits/"+head+"/check-runs?filter=latest&per_page=100", "--paginate", "--slurp"); err != nil {
+		return ciObservation{}, err
+	}
+	result := ciObservation{state: ciAbsent}
+	pending, count := false, 0
+	for _, page := range pages {
+		for _, check := range page.CheckRuns {
+			count++
+			if check.Status != "completed" {
+				switch check.Status {
+				case "queued", "in_progress", "waiting", "requested", "pending":
+					pending = true
+				default:
+					return result, fmt.Errorf("unknown CI check status %q", check.Status)
+				}
+				continue
+			}
+			switch check.Conclusion {
+			case "success", "neutral", "skipped":
+			case "failure", "timed_out", "cancelled", "action_required", "stale", "startup_failure":
+				result.failed = append(result.failed, check.Name)
+			default:
+				return result, fmt.Errorf("unknown CI conclusion %q", check.Conclusion)
+			}
+		}
+	}
+	var statuses [][]struct{ Context, State string }
+	if err := ghJSON(ctx, &statuses, "api", "repos/"+repo+"/commits/"+head+"/statuses?per_page=100", "--paginate", "--slurp"); err != nil {
+		return result, err
+	}
+	seen := map[string]bool{}
+	for _, page := range statuses {
+		for _, status := range page {
+			if seen[status.Context] {
+				continue
+			}
+			seen[status.Context] = true
+			count++
+			switch status.State {
+			case "success":
+			case "pending", "expected":
+				pending = true
+			case "failure", "error":
+				result.failed = append(result.failed, status.Context)
+			default:
+				return result, fmt.Errorf("unknown commit status %q", status.State)
+			}
+		}
+	}
+	open, err := workflowRuns(ctx, repo, head, &result)
+	if err != nil {
+		return result, err
+	}
+	// An open run may not have created its jobs yet. Successes already listed
+	// must not hide it. A failure already in hand can start a Repair.
+	if open && len(result.failed) == 0 {
+		pending = true
+	}
+	switch {
+	case pending:
+		result.state = ciPending
+	case len(result.failed) > 0:
+		result.state = ciFailed
+	case count > 0:
+		result.state = ciPassed
+	}
+	return result, nil
+}
+
+// Latest workflow run of each workflow on this SHA. Older runs are superseded.
+// A completed failure is red even when none of its jobs appeared as check runs.
+func workflowRuns(ctx context.Context, repo, head string, result *ciObservation) (bool, error) {
+	var pages []struct {
+		WorkflowRuns []struct {
+			ID         int64   `json:"id"`
+			Name       string  `json:"name"`
+			Status     string  `json:"status"`
+			Conclusion *string `json:"conclusion"`
+			CreatedAt  string  `json:"created_at"`
+			WorkflowID int64   `json:"workflow_id"`
+		} `json:"workflow_runs"`
+	}
+	if err := ghJSON(ctx, &pages, "api", "repos/"+repo+"/actions/runs?head_sha="+head+"&per_page=100", "--paginate", "--slurp"); err != nil {
+		return false, err
+	}
+	type latestRun struct {
+		id         int64
+		name       string
+		status     string
+		conclusion *string
+		created    string
+	}
+	latest := map[int64]latestRun{}
+	for _, page := range pages {
+		for _, run := range page.WorkflowRuns {
+			if run.WorkflowID == 0 || run.ID == 0 || run.CreatedAt == "" || run.Status == "" {
+				return false, fmt.Errorf("unreadable workflow run")
+			}
+			prev, ok := latest[run.WorkflowID]
+			if ok && (prev.created > run.CreatedAt || (prev.created == run.CreatedAt && prev.id >= run.ID)) {
+				continue
+			}
+			latest[run.WorkflowID] = latestRun{run.ID, run.Name, run.Status, run.Conclusion, run.CreatedAt}
+		}
+	}
+	open := false
+	for _, run := range latest {
+		switch run.status {
+		case "queued", "in_progress", "waiting", "requested", "pending":
+			open = true
+		case "completed":
+			if run.conclusion == nil {
+				return false, fmt.Errorf("workflow run %d completed without a conclusion", run.id)
+			}
+			switch *run.conclusion {
+			case "success", "neutral", "skipped":
+			case "failure", "timed_out", "cancelled", "action_required", "stale", "startup_failure":
+				name := run.name
+				if name == "" {
+					name = fmt.Sprintf("workflow %d", run.id)
+				}
+				result.failed = append(result.failed, name)
+			default:
+				return false, fmt.Errorf("unknown workflow conclusion %q", *run.conclusion)
+			}
+		default:
+			return false, fmt.Errorf("unknown workflow status %q", run.status)
+		}
+	}
+	return open, nil
+}

prototype/go/claim.go

@@ -81,9 +81,8 @@ func changeLabels(ctx context.Context, issue issueURL, current issueData, add, r
 	return err
 }
 
-// This slice ends before Delivery: release only when nothing remains on origin.
 // Never reuse the cancelled Ticket-run context here.
-func endClaim(ctx context.Context, launch string, issue issueURL, branch string, initial issueData, failed bool) error {
+func endClaim(ctx context.Context, launch string, issue issueURL, branch string, initial issueData, result runOutcome) error {
 	data, err := readIssue(ctx, issue)
 	if err != nil {
 		return err
@@ -92,6 +91,7 @@ func endClaim(ctx context.Context, launch string, issue issueURL, branch string,
 		return nil
 	}
 	var add, remove []string
+	state := claimClosed
 	if data.State == "open" {
 		refs, err := command(ctx, launch, "git", "ls-remote", "--heads", "origin", "refs/heads/"+branch)
 		if err != nil {
@@ -104,22 +104,30 @@ func endClaim(ctx context.Context, launch string, issue issueURL, branch string,
 			return err
 		}
 		if len(refs) != 0 || len(prs) != 0 {
-			if !failed {
-				return nil
+			state = claimHeld
+			if result.failed() {
+				state = claimFailed
 			}
-			if _, err := command(ctx, "", "gh", "label", "create", "agent-failed", "--repo", issue.repo, "--description", "Failed Ticket run", "--force"); err != nil {
-				return err
-			}
-			add = append(add, "agent-failed")
 		} else {
-			if !initial.hasLabel("in-progress") {
-				remove = append(remove, "in-progress")
-			}
-			if initial.hasLabel("ready-for-agent") {
-				add = append(add, "ready-for-agent")
-			}
+			state = claimReleased
+		}
+	}
+	switch state {
+	case claimHeld:
+		return nil
+	case claimFailed:
+		if _, err := command(ctx, "", "gh", "label", "create", "agent-failed", "--repo", issue.repo, "--description", "Failed Ticket run", "--force"); err != nil {
+			return err
+		}
+		add = append(add, "agent-failed")
+	case claimReleased:
+		if !initial.hasLabel("in-progress") {
+			remove = append(remove, "in-progress")
+		}
+		if initial.hasLabel("ready-for-agent") {
+			add = append(add, "ready-for-agent")
 		}
-	} else {
+	case claimClosed:
 		remove = append(remove, "in-progress")
 		if data.hasLabel("agent-failed") {
 			remove = append(remove, "agent-failed")

prototype/go/cli_test.go

@@ -93,8 +93,8 @@ func (f *fixture) writeState(state, association, mode string) {
 	}
 }
 
-func (f *fixture) run(url string) (string, error) {
-	cmd := exec.Command(binary, url)
+func (f *fixture) run(args ...string) (string, error) {
+	cmd := exec.Command(binary, args...)
 	cmd.Dir, cmd.Env = f.repo, f.env
 	output, err := cmd.CombinedOutput()
 	return string(output), err