diff --git a/.claude/agents/security-reviewer.md b/.claude/agents/security-reviewer.md index ed8c494..6c6f164 100644 --- a/.claude/agents/security-reviewer.md +++ b/.claude/agents/security-reviewer.md @@ -35,8 +35,11 @@ have the bot token. - The state moves forward only after `apply` and `health` succeed. `in_progress` is synced before `apply` runs; found at start, it pauses. - `on_failure`: `rollback` runs the old apply and health, and a failed - rollback writes the pause file; `pause` holds the SHA and does not write - the pause file. The code default is `pause`. + rollback makes the instance broken: only a fix forward (a descendant of + the failed SHA) is asked, always with a tap (also in `auto` mode), no + rollback is tried, and only a good fix forward or root's `--resume` ends + it; `pause` holds the SHA and does not write the pause file. The code + default is `pause`. **No code before the tap (rule 3)** - Before Apply, nothing from the commit runs. git only reads objects. diff --git a/DESIGN.md b/DESIGN.md index 256913d..9c77c18 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -547,10 +547,10 @@ The steps, with the state synced to disk at each arrow: - `rollback`, with an applied SHA: write `in_progress = {phase: "rollback", sha, old_sha}` →. Run `apply` with the last good SHA, then `health`. If that works, clear `in_progress` → and report "failed, - rolled back to OLD". If it fails too, write the pause file, clear - `in_progress` → and report "failed, the rollback failed too, the - instance is paused". With no applied SHA there is nothing to roll back - to: hold the SHA, as with `pause`. + rolled back to OLD". If it fails too, the instance becomes + **broken** (below): write `broken = {since, sha, old_sha, reason}` and + clear `in_progress` in one save →, and report it. With no applied SHA + there is nothing to roll back to: hold the SHA, as with `pause`. - `pause`: do not roll back and do not write the pause file. The SHA becomes the held SHA. Clear `in_progress` →. It is not tried again, every report and `pullgate status` say that the last apply failed, and @@ -565,6 +565,40 @@ tries is asked again on the next run; with `max_failed_tries` failed tries it is not asked again until a new commit (or `--ask-again`). With `pause`, a failed SHA is held at once and the count does not matter. +**Broken.** After a failed rollback the host may be in a half state. A +plain pause would need root on the host for every such case, and most of +them are fixed by a new commit. So the instance becomes broken instead: + +- One message goes out when it starts: the reason (the errors of the apply + and of the rollback, each cut in the middle so that its end stays, + code6x/pullgate#25) and the two ways out: a fix forward, or root on the + host. +- The instance keeps fetching. It asks only for a **fix forward**: a tip + that descends from the failed SHA, and is not the failed SHA or the old + applied SHA. This holds with `allow_older` too. Any other tip is refused + with a notice. The failed SHA is not tried again, whatever + `max_failed_tries` says. +- The question has the lines "⚠ The instance is broken: an apply failed + and pullgate could not roll it back. The host may be in a half state. + This commit is a fix forward." and "Reason: …" at the top. The reason + is in ``, after the fixed line. In `auto` mode the owner is asked too: + nothing is applied by itself on a half state. `questions_per_day` + counts these questions. +- At the tap, root checks the fix forward rule again. A fix forward that + succeeds (apply and `health`) ends the broken state. One that fails is + **not** rolled back: the instance stays broken, the failed fix forward + becomes the new failed SHA and reason, and the next newer commit is + asked with the warning. +- The broken state never ends by itself. It does not stop a pause: a + pause file still stops everything. Root ends it with `pullgate status + NAME --resume`. No command from Telegram can end it. +- During a fix forward, `PULLGATE_FAILED_SHA` is the failed SHA. +- When it starts, every question asked before is dropped (the open one + is closed in the chat), so no tap on a question without the warning + applies. The next run asks again. +- If the save of the broken state fails, `in_progress` stays in the file, + and the next start pauses (a crash, below). + **A crash** leaves `in_progress` in the state. The next run or apply cannot know what happened on the host. It writes the pause file, marks the SHA failed and reports "the apply (or rollback) of SHA was cut off; check the @@ -628,6 +662,9 @@ The question is HTML (`parse_mode = HTML`), with link previews off: ``` pullgate on HOST, instance NAME +⚠ The instance is broken: an apply failed and pullgate could not roll it + back. The host may be in a half state. This commit is a fix forward. +Reason: reason (both only when broken, section 10) ⚠ The last apply (SHA) failed. (only when a SHA is held) ⚠ This commit is not a descendant of … (only with allow_older) Apply this commit? @@ -693,7 +730,8 @@ closed after a cut. - `off`: `pullgate run` and `pullgate apply` do nothing. `pullgate status NAME` prints the mode, the applied SHA, the open question, -the held SHA, the pause file and an apply in progress. Flags: +the held SHA, the broken state ("broken since …, last error …"), the pause +file and an apply in progress. Flags: - `--ask-again`: clear the marks of the last fetched tip (expired, skipped, check failed, refused, failed, held), so the next run asks again. If a @@ -703,7 +741,8 @@ the held SHA, the pause file and an apply in progress. Flags: approval for the old code finds no question and does nothing, even one that the listener wrote before the close; - `--pause`: write the pause file; -- `--resume`: remove the pause file and clear an apply in progress. +- `--resume`: remove the pause file, clear an apply in progress, and end + the broken state (section 10). Run it only after you checked the host. The listener (`pullgate listen --config /etc/pullgate/listener.toml`): diff --git a/README.md b/README.md index f4a2018..ead15c6 100644 --- a/README.md +++ b/README.md @@ -258,7 +258,12 @@ sections 5 and 11. - `on_failure = "rollback"`: pullgate runs `apply` again with the last good commit, then `health`. So `apply` must be safe to run again on an older - tree. If the rollback fails too, the instance is paused. + tree. If the rollback fails too, the instance is **broken**: the host + may be in a half state. pullgate tells you once, with the reason. Then it + asks only for a fix forward (a newer commit on top of the failed one), + with a warning, also in `auto` mode, and tries no rollback. A fix forward + that works ends it, or root runs `pullgate status NAME --resume` after a + look at the host. - `on_failure = "pause"`: no rollback. The failed commit is held and not tried again. A newer commit is still asked, with a warning at the top, so you can fix it forward. diff --git a/THREAT-MODEL.md b/THREAT-MODEL.md index 622db79..727ff24 100644 --- a/THREAT-MODEL.md +++ b/THREAT-MODEL.md @@ -236,9 +236,17 @@ having the host manage itself. *Attack or accident.* `apply` fails, and running it again on the old tree fails too. The host may be half changed. -*Defense.* With `on_failure = "rollback"`, a failed rollback writes the pause -file and reports it. Nothing more runs until the owner looks and runs -`pullgate status NAME --resume`. With `on_failure = "pause"` there is no +*Defense.* With `on_failure = "rollback"`, a failed rollback makes the +instance **broken** (DESIGN.md section 10) and reports it once, with the +reason. Nothing is applied until the owner taps Apply on a **fix +forward**: a newer commit on top of the failed one, asked with a strong +warning. The failed commit and the old commit are never asked again, and +no rollback is tried against the half state. In `auto` mode the owner is +asked too. Only a fix forward that passes `health`, or root with +`pullgate status NAME --resume`, ends it. It never ends by itself, and no +message from Telegram ends it, so a stolen bot token or a compromised +listener (T2) cannot lift it: the most they can do is not deliver a tap. +A pause file still stops everything, also while broken. With `on_failure = "pause"` there is no rollback: the failed SHA is held, and the owner can fix it forward. The applied SHA does not change after a failure. diff --git a/internal/gate/apply.go b/internal/gate/apply.go index 76cc804..3cdaa9c 100644 --- a/internal/gate/apply.go +++ b/internal/gate/apply.go @@ -189,6 +189,23 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) ( e.reply(q.ID, code(sha)+" is applied already.") return Ignored, nil } + if b := st.Broken; b != nil { + // Only a fix forward: a commit on top of the last failed one. + // An auto question is never applied here: the owner must tap. + ok := sha != b.SHA && sha != b.OldSHA && !q.Auto + if ok { + var err error + if ok, err = e.Git.IsAncestor(ctx, b.SHA, sha); err != nil { + return "", err + } + } + if !ok { + st.SetMark(sha, state.MarkRefused, now) + e.Log.Warn("broken: refused, not a fix forward", "sha", sha, "failed", b.SHA) + e.reply(q.ID, "Refused "+code(sha)+": the instance is broken, and only a commit on top of the failed commit "+code(b.SHA)+" is applied.") + return Refused, nil + } + } if st.Applied != "" && !e.Cfg.Limits.AllowOlder { ok, err := e.Git.IsAncestor(ctx, st.Applied, sha) if err != nil { @@ -212,13 +229,14 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) ( } defer t.remove() - err = e.tryApply(ctx, t, "apply", sha, old, st.Held, false) + wasBroken := st.Broken + err = e.tryApply(ctx, t, "apply", sha, old, failedSHA(st), false) if stopped(ctx, err) { return "", err // in_progress stays: the next start pauses } if err == nil { st.Applied, st.AppliedAt = sha, e.now() - st.InProgress, st.Held = nil, "" + st.InProgress, st.Held, st.Broken = nil, "", nil st.Marks = map[string]*state.Mark{} if gerr := e.Git.SetApplied(ctx, sha); gerr != nil { e.Log.Warn("keep the applied ref", "err", gerr) @@ -228,6 +246,10 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) ( return "", err } e.Log.Info("applied", "sha", sha) + if wasBroken != nil { + e.reply(q.ID, "✅ Applied "+code(sha)+". The fix forward worked: the instance is no longer broken.") + return Applied, nil + } e.reply(q.ID, "✅ Applied "+code(sha)+".") return Applied, nil } @@ -237,7 +259,18 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) ( if err := e.Dir.Save(st); err != nil { return "", err } - failure := esc(err) + failure := message.Reason(err.Error()) + if b := st.Broken; b != nil { + // A failed fix forward: no rollback against a half state. + b.SHA, b.Reason = sha, "the fix forward "+message.Short(sha)+" failed: "+message.CutLine(err.Error()) + st.InProgress = nil + if err := e.Dir.Save(st); err != nil { + return "", err + } + e.Log.Error("the fix forward failed; the instance stays broken", "sha", sha) + e.reply(q.ID, "❌ The fix forward "+code(sha)+" failed ("+failure+"). No rollback was tried. The instance stays broken.\n"+e.waysOut()) + return StillBroken, nil + } if e.Cfg.OnFailure == config.OnFailureRollback && old != "" { st.InProgress = &state.InProgress{Phase: "rollback", SHA: sha, OldSHA: old, Started: e.now()} if err := e.Dir.Save(st); err != nil { @@ -255,17 +288,25 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) ( e.reply(q.ID, "❌ The apply of "+code(sha)+" failed ("+failure+"). Rolled back to "+code(old)+".") return RolledBack, nil } - if perr := e.Dir.Pause("the rollback to " + old + " failed after the apply of " + sha + " failed"); perr != nil { - // in_progress stays, so the next start pauses (cutOff). - e.Log.Error("write the pause file", "err", perr) - e.reply(q.ID, "❌ The apply of "+code(sha)+" failed, the rollback failed too, and the pause file could not be written. Check the host now.") - return RollbackError, perr - } + // The broken state (DESIGN.md section 10). If this save fails, + // in_progress stays in the file, and the next start pauses. st.InProgress = nil + st.Broken = &state.Broken{ + Since: e.now(), SHA: sha, OldSHA: old, + Reason: "the apply of " + message.Short(sha) + " failed: " + message.CutLine(err.Error()) + + "; the rollback to " + message.Short(old) + " failed: " + message.CutLine(rerr.Error()), + } + // Questions asked before have no warning (or are auto ones): drop + // them all, so no old tap applies. The next run asks again. + e.dropQuestions(st, "the instance is broken. A fix forward is asked again, with a warning.") if err := e.Dir.Save(st); err != nil { - return "", err + // Keep in_progress, so the next start pauses (a crash). + st.Broken = nil + st.InProgress = &state.InProgress{Phase: "rollback", SHA: sha, OldSHA: old, Started: e.now()} + return RollbackError, err } - e.reply(q.ID, "❌ The apply of "+code(sha)+" failed ("+failure+"), and the rollback to "+code(old)+" failed too ("+esc(rerr)+"). The instance is paused. Check the host, then run as root: pullgate status "+e.Cfg.Name+" --resume") + e.Log.Error("the apply and its rollback failed; the instance is broken", "sha", sha, "old", old) + e.reply(q.ID, "❌ The apply of "+code(sha)+" failed ("+failure+"), and the rollback to "+code(old)+" failed too ("+message.Reason(rerr.Error())+"). The host may be in a half state. The instance is now broken.\n"+e.waysOut()) return RollbackError, nil } // on_failure = "pause", or nothing to roll back to: hold the SHA. @@ -278,6 +319,25 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) ( return Held, nil } +// dropQuestions removes every question from the state, and closes the +// open one in the chat. +func (e *Env) dropQuestions(st *state.State, why string) { + for _, q := range st.Questions { + if m := st.Marks[q.SHA]; m != nil && m.Mark == state.MarkAsked { + delete(st.Marks, q.SHA) + } + if q.Open && !q.Auto { + e.edit(q.ID, "This question for "+code(q.SHA)+" is closed: "+why) + } + } + st.Questions = nil +} + +// waysOut says how the broken state ends. +func (e *Env) waysOut() string { + return "Two ways out: push a fix forward (a commit on top of the failed one; it is asked with a warning), or check the host and run as root: pullgate status " + e.Cfg.Name + " --resume" +} + // stopped reports whether err comes from a stop of pullgate itself. func stopped(ctx context.Context, err error) bool { return err != nil && (ctx.Err() != nil || errors.Is(err, ErrStopped)) @@ -317,7 +377,3 @@ func (e *Env) afterApply(st *state.State) { } st.Questions = keep } - -// esc shows an error in a message, in : it may hold text from the -// repo (DESIGN.md section 12). -func esc(err error) string { return message.Code(err.Error()) } diff --git a/internal/gate/broken_test.go b/internal/gate/broken_test.go new file mode 100644 index 0000000..254098a --- /dev/null +++ b/internal/gate/broken_test.go @@ -0,0 +1,259 @@ +package gate + +import ( + "strings" + "testing" + "time" + + "github.com/code6x/pullgate/internal/config" + "github.com/code6x/pullgate/internal/message" + "github.com/code6x/pullgate/internal/spool" + "github.com/code6x/pullgate/internal/state" +) + +// breakIt applies "one", then fails the apply of "bad" and its rollback. +// It returns the applied and the failed SHA. +func breakIt(t *testing.T, f *fixture) (string, string) { + t.Helper() + a := f.applyNew("one") + b := f.commit("bad", "two", map[string]string{"fail-health": "1"}) + f.run() + f.tap(f.ask().Code, "apply") + f.touch("fail-rollback") + if r := f.apply(); r != RollbackError { + t.Fatalf("apply: %s", r) + } + return a, b +} + +func TestFailedRollbackIsBroken(t *testing.T) { + f := newFixture(t, `on_failure = "rollback"`, "[limits]\nmax_failed_tries = 3\n") + a, b := breakIt(t, f) + st := f.state() + if st.Broken == nil || st.Broken.SHA != b || st.Broken.OldSHA != a || st.InProgress != nil || st.Applied != a { + t.Fatalf("state after a failed rollback: %+v", st) + } + if f.env.Dir.Paused() { + t.Fatal("broken is not a pause: no pause file") + } + out := f.outbox() + if len(out) != 1 || !strings.Contains(out[0].Text, "The instance is now broken") || + !strings.Contains(out[0].Text, "Two ways out") || !strings.Contains(out[0].Text, "exit status 3") { + t.Fatalf("one message with the reason and the ways out, got %+v", out) + } + // The failed commit is not asked again (even with tries left), and + // neither is the old one. + if r := f.run(); r != Nothing { + t.Fatalf("run on the failed tip: %s", r) + } + f.r.Push(a) + if r := f.run(); r != Nothing { + t.Fatalf("run on the old commit: %s", r) + } + if len(f.outbox()) != 0 { + t.Fatal("no message for the failed or the old commit") + } + // The notice went out once: later runs send nothing. + f.r.Push(b) + f.run() + if len(f.outbox()) != 0 { + t.Fatal("the broken notice goes out once") + } + var w strings.Builder + if err := f.env.Status(&w, StatusFlags{}); err != nil { + t.Fatal(err) + } + if !strings.Contains(w.String(), "broken: since 2026-11-11 12:00 UTC, last error: the apply of") { + t.Fatalf("status:\n%s", w.String()) + } +} + +func TestBrokenFixForward(t *testing.T) { + f := newFixture(t, `on_failure = "rollback"`, "") + _, b := breakIt(t, f) + f.outbox() + f.r.Git("rm", "--quiet", "fail-health") + c := f.commit("fix", "three", nil) + if r := f.run(); r != Asked { + t.Fatalf("a fix forward is asked: %s", r) + } + q := f.ask() + if !strings.Contains(q.Text, "This commit is a fix forward.\nReason: the apply of "+message.Short(b)) || + !strings.Contains(q.Text, "This commit is a fix forward.") { + t.Fatalf("the question has no warning:\n%s", q.Text) + } + f.tap(q.Code, "apply") + if r := f.apply(); r != Applied { + t.Fatalf("apply: %s", r) + } + st := f.state() + if st.Broken != nil || st.Applied != c { + t.Fatalf("a good fix forward ends the broken state: %+v", st) + } + if !contains(f.outbox(), "no longer broken") { + t.Fatal("no report of the end") + } + // PULLGATE_FAILED_SHA is the failed commit during the fix forward. + if !strings.Contains(f.read("log"), "apply "+c+" old=") || !strings.Contains(f.read("log"), "failed="+b) { + t.Fatalf("env of the fix forward:\n%s", f.read("log")) + } +} + +func TestBrokenBadFixForwardStaysBroken(t *testing.T) { + f := newFixture(t, `on_failure = "rollback"`, "[limits]\nmax_failed_tries = 3\n") + a, _ := breakIt(t, f) + f.outbox() + c := f.commit("still bad", "three", nil) // fail-health is still there + f.run() + f.tap(f.ask().Code, "apply") + before := strings.Count(f.read("log"), "rb=1") + if r := f.apply(); r != StillBroken { + t.Fatalf("apply: %s", r) + } + if strings.Count(f.read("log"), "rb=1") != before { + t.Fatal("no rollback is tried in the broken state") + } + st := f.state() + if st.Broken == nil || st.Broken.SHA != c || st.Applied != a || !strings.Contains(st.Broken.Reason, "fix forward") { + t.Fatalf("a bad fix forward stays broken: %+v", st.Broken) + } + if !contains(f.outbox(), "The instance stays broken") { + t.Fatal("no report") + } + if r := f.run(); r != Nothing { + t.Fatalf("the failed fix forward is not asked again: %s", r) + } + d := f.commit("next", "four", nil) + if r := f.run(); r != Asked { + t.Fatalf("the next newer commit is asked: %s", r) + } + if q := f.ask(); !strings.Contains(q.Text, "fix forward "+message.Short(c)+" failed") { + t.Fatalf("the warning has the last reason:\n%s", q.Text) + } + _ = d +} + +func TestBrokenRefusesNonDescendant(t *testing.T) { + // Even with allow_older, only a commit on top of the failed one. + f := newFixture(t, `on_failure = "rollback"`, "[limits]\nallow_older = true\n") + a, _ := breakIt(t, f) + f.outbox() + f.r.Push(a) + f.r.Git("reset", "--quiet", "--hard", a) + f.commit("other", "five", nil) + if r := f.run(); r != Refused { + t.Fatalf("run: %s", r) + } + if !contains(f.outbox(), "only a fix forward is asked") { + t.Fatal("no notice") + } +} + +func TestBrokenAsksInAutoMode(t *testing.T) { + f := newFixture(t, `on_failure = "rollback"`, "") + _, _ = breakIt(t, f) + f.outbox() + f.env.Cfg.Mode = config.ModeAuto + f.r.Git("rm", "--quiet", "fail-health") + f.commit("fix", "three", nil) + if r := f.run(); r != Asked { + t.Fatalf("broken in auto mode asks the owner: %s", r) + } + if m := f.ask(); m.Kind != spool.KindAsk { + t.Fatal("no question") + } +} + +func TestRootResumeEndsBroken(t *testing.T) { + f := newFixture(t, `on_failure = "rollback"`, "") + breakIt(t, f) + var w strings.Builder + if err := f.env.Status(&w, StatusFlags{Resume: true}); err != nil { + t.Fatal(err) + } + if f.state().Broken != nil || !strings.Contains(w.String(), "The broken state is ended") { + t.Fatalf("root --resume ends the broken state:\n%s", w.String()) + } +} + +func TestBrokenDropsOlderQuestions(t *testing.T) { + f := newFixture(t, `on_failure = "rollback"`, "") + f.applyNew("one") + f.commit("bad", "two", map[string]string{"fail-health": "1"}) + f.run() + q1 := f.ask() + f.commit("next", "three", nil) // fail-health is still in the tree + f.run() + q2 := f.ask() + f.tap(q1.Code, "apply") + f.touch("fail-rollback") + if r := f.apply(); r != RollbackError { + t.Fatalf("apply: %s", r) + } + if st := f.state(); len(st.Questions) != 0 { + t.Fatalf("the questions from before are dropped: %+v", st.Questions) + } + if !contains(f.outbox(), "is closed: the instance is broken") { + t.Fatal("the open question is not closed in the chat") + } + // The tap on the question without the warning does nothing. + f.tap(q2.Code, "apply") + if r := f.apply(); r != Ignored { + t.Fatalf("an old tap while broken: %s", r) + } + // The next run asks it again, with the warning. + if r := f.run(); r != Asked { + t.Fatalf("run: %s", r) + } + if !strings.Contains(f.ask().Text, "This commit is a fix forward.") { + t.Fatal("no warning") + } + // A pause file still stops everything while broken. + if err := f.env.Dir.Pause("test"); err != nil { + t.Fatal(err) + } + f.commit("more", "four", nil) + if r := f.run(); r != Paused { + t.Fatalf("paused and broken: %s", r) + } +} + +func TestBrokenTapRefusesNonFixForward(t *testing.T) { + for _, which := range []string{"failed", "other", "auto"} { + t.Run(which, func(t *testing.T) { + f := newFixture(t, `on_failure = "rollback"`, "[limits]\nallow_older = true\n") + a, b := breakIt(t, f) + f.outbox() + sha, auto := b, false + switch which { + case "other": + f.r.Git("reset", "--quiet", "--hard", a) + sha = f.commit("other", "five", nil) + f.r.Push(sha) + case "auto": + f.r.Git("rm", "--quiet", "fail-health") + sha, auto = f.commit("fix", "three", nil), true + } + f.run() // fetch it + f.outbox() + // A question that root's state holds, as if asked before. + c, err := state.NewCode() + if err != nil { + t.Fatal(err) + } + st := f.state() + st.Questions = append(st.Questions, state.Question{ID: 99, Code: c, SHA: sha, Base: a, + Asked: f.now, Expires: f.now.Add(time.Hour), Open: true, Auto: auto}) + if err := f.env.Dir.Save(st); err != nil { + t.Fatal(err) + } + f.tap(c, "apply") + if r := f.apply(); r != Refused { + t.Fatalf("apply: %s", r) + } + if st := f.state(); st.Applied != a || st.Broken == nil { + t.Fatalf("state: %+v", st) + } + }) + } +} diff --git a/internal/gate/gate_test.go b/internal/gate/gate_test.go index c42edd0..1ab44bb 100644 --- a/internal/gate/gate_test.go +++ b/internal/gate/gate_test.go @@ -537,28 +537,6 @@ func TestMaxFailedTries(t *testing.T) { } } -func TestFailingRollbackPauses(t *testing.T) { - f := newFixture(t, `on_failure = "rollback"`, "") - f.applyNew("one") - f.commit("bad", "two", map[string]string{"fail-health": "1"}) - f.run() - f.tap(f.ask().Code, "apply") - f.touch("fail-rollback") - if r := f.apply(); r != RollbackError { - t.Fatalf("apply: %s", r) - } - if !f.env.Dir.Paused() || f.state().InProgress != nil { - t.Fatal("a failed rollback must write the pause file") - } - if !contains(f.outbox(), "The instance is paused") { - t.Fatal("no report") - } - f.commit("fix", "three", nil) - if r := f.run(); r != Paused { - t.Fatalf("a paused instance does nothing: %s", r) - } -} - func TestOnFailurePauseHoldsAndAsksNewer(t *testing.T) { f := newFixture(t, `on_failure = "pause"`, "") a := f.applyNew("one") diff --git a/internal/gate/run.go b/internal/gate/run.go index b71e1b0..869903e 100644 --- a/internal/gate/run.go +++ b/internal/gate/run.go @@ -36,6 +36,7 @@ const ( RolledBack Result = "rolled-back" RollbackError Result = "rollback-failed" Held Result = "held" + StillBroken Result = "still-broken" ) func (e *Env) head() string { return message.Head(e.Cfg.Host, e.Cfg.Name) } @@ -220,8 +221,13 @@ func (e *Env) decide(ctx context.Context, st *state.State, tip string) (Result, e.Log.Info("the tip is the held SHA; waiting for a new commit", "sha", tip) return Nothing, nil } + if b := st.Broken; b != nil && (tip == b.SHA || tip == b.OldSHA) { + e.Log.Info("broken: the tip is the failed or the old commit; waiting for a fix forward", "sha", tip) + return Nothing, nil + } if m := st.Marks[tip]; m != nil { - retry := m.Mark == state.MarkFailed && e.Cfg.OnFailure == config.OnFailureRollback && m.Tries < e.Cfg.Limits.MaxFailedTries + // No retry in the broken state: a rollback is never tried there. + retry := st.Broken == nil && m.Mark == state.MarkFailed && e.Cfg.OnFailure == config.OnFailureRollback && m.Tries < e.Cfg.Limits.MaxFailedTries if !retry { e.Log.Info("the tip was handled before; waiting for a new commit", "sha", tip, "mark", m.Mark) return Nothing, nil @@ -243,6 +249,20 @@ func (e *Env) decide(ctx context.Context, st *state.State, tip string) (Result, notDesc = true } } + if b := st.Broken; b != nil { + ok, err := e.Git.IsAncestor(ctx, b.SHA, tip) + if err != nil { + return "", err + } + if !ok { + st.SetMark(tip, state.MarkRefused, now) + e.Log.Warn("broken: refused, the tip does not descend from the failed commit", "tip", tip, "failed", b.SHA) + e.notice("⚠ Refused " + code(tip) + ": the instance is broken, and only a fix forward is asked: a commit on top of the failed commit " + code(b.SHA) + ". Nothing was asked.\nPush a fix on top of it, or check the host and run as root: pullgate status " + e.Cfg.Name + " --resume") + return Refused, nil + } + } + // In the broken state the owner is always asked, also in auto mode. + ask := e.Cfg.Mode == config.ModeAsk || st.Broken != nil // Questions per day. var recent []time.Time for _, t := range st.AskTimes { @@ -251,7 +271,7 @@ func (e *Env) decide(ctx context.Context, st *state.State, tip string) (Result, } } st.AskTimes = recent - if e.Cfg.Mode == config.ModeAsk && len(recent) >= e.Cfg.Limits.QuestionsPerDay { + if ask && len(recent) >= e.Cfg.Limits.QuestionsPerDay { e.Log.Warn("questions_per_day reached; not asking", "tip", tip) if now.Sub(st.LimitNote) >= 24*time.Hour { st.LimitNote = now @@ -276,7 +296,7 @@ func (e *Env) decide(ctx context.Context, st *state.State, tip string) (Result, } return "", fmt.Errorf("export %s: %w", tip, err) } - p.env.Failed = st.Held + p.env.Failed = failedSHA(st) if argv := e.Cfg.Commands.Check; len(argv) > 0 { out, err := e.runSandboxed(ctx, argv, p.env) if err != nil { @@ -289,7 +309,7 @@ func (e *Env) decide(ctx context.Context, st *state.State, tip string) (Result, } q := &message.Question{ Host: e.Cfg.Host, Instance: e.Cfg.Name, SHA: tip, Applied: st.Applied, - Held: st.Held, NotDescendant: notDesc, Files: p.files, Watch: e.Cfg.Watch.Paths, + Held: st.Held, Broken: brokenReason(st), NotDescendant: notDesc, Files: p.files, Watch: e.Cfg.Watch.Paths, Submodules: p.tree.Submodules, } for _, s := range p.tree.Symlinks { @@ -317,7 +337,7 @@ func (e *Env) decide(ctx context.Context, st *state.State, tip string) (Result, nq := state.Question{ ID: st.NextID, Code: c, SHA: tip, Base: st.Applied, Asked: now, Expires: now.Add(e.Cfg.Limits.ApprovalTTL.Duration), Open: true, - Auto: e.Cfg.Mode == config.ModeAuto, + Auto: !ask, } st.NextID++ st.Questions = append(st.Questions, nq) @@ -361,3 +381,19 @@ func shortTrees(out, tree, oldTree string) string { } return out } + +// failedSHA is PULLGATE_FAILED_SHA: the last commit that failed in the +// broken state, else the held SHA. +func failedSHA(st *state.State) string { + if st.Broken != nil { + return st.Broken.SHA + } + return st.Held +} + +func brokenReason(st *state.State) string { + if st.Broken == nil { + return "" + } + return st.Broken.Reason +} diff --git a/internal/gate/status.go b/internal/gate/status.go index 458aed2..c1eb849 100644 --- a/internal/gate/status.go +++ b/internal/gate/status.go @@ -45,6 +45,12 @@ func (e *Env) Status(w io.Writer, f StatusFlags) error { st.InProgress = nil changed = true } + if st.Broken != nil { + // Root checked the host: the broken state ends here. + st.Broken = nil + changed = true + _, _ = fmt.Fprintln(w, "The broken state is ended. A new commit, or --ask-again, asks again.") + } } if f.AskAgain && st.LastTip != "" { delete(st.Marks, st.LastTip) @@ -82,6 +88,9 @@ func (e *Env) Status(w io.Writer, f StatusFlags) error { if m := st.Marks[st.LastTip]; m != nil && st.LastTip != "" { show("tip mark", fmt.Sprintf("%s (tries %d)", m.Mark, m.Tries)) } + if b := st.Broken; b != nil { + show("broken", fmt.Sprintf("since %s, last error: %s", b.Since.Format("2006-01-02 15:04 MST"), b.Reason)) + } if e.Dir.Paused() { show("paused", "yes: "+e.Dir.PauseFile()) } else { diff --git a/internal/message/message.go b/internal/message/message.go index a04f94d..7574ca7 100644 --- a/internal/message/message.go +++ b/internal/message/message.go @@ -103,6 +103,24 @@ func Output(out string, n int) string { return "
" + s + "
" } +// Reason is an error in a message: one line, cut in the middle so its +// start and its end (where most tools print the reason) stay, in +// (code6x/pullgate#25). +func Reason(s string) string { return Code(CutLine(s)) } + +// CutLine is s as one line of at most OutputLine characters, cut in the +// middle. +func CutLine(s string) string { + return cutMiddle(strings.Join(strings.Fields(s), " "), OutputLine) +} + +// BrokenReason is the reason of the broken state in a message. It holds +// two errors, each cut with CutLine already, so it gets more room than +// Reason. +func BrokenReason(s string) string { + return Code(cutMiddle(strings.Join(strings.Fields(s), " "), 2*OutputLine+200)) +} + // cutMiddle cleans one line like clean, but a line longer than max keeps // its first max/4 and its last characters, with "…" between. func cutMiddle(s string, max int) string { diff --git a/internal/message/question.go b/internal/message/question.go index 0bc3307..01657de 100644 --- a/internal/message/question.go +++ b/internal/message/question.go @@ -14,6 +14,7 @@ type Question struct { Host, Instance string SHA, Applied string // Applied is "" on the first apply Held string // the held SHA, if any + Broken string // why the instance is broken, if it is (DESIGN.md section 10) NotDescendant bool // with allow_older Subjects []string TotalCommits int @@ -41,6 +42,10 @@ func Short(sha string) string { // Text builds the question text, cut to fit one message. func (q *Question) Text() string { head := []string{Head(q.Host, q.Instance)} + if q.Broken != "" { + head = append(head, "⚠ The instance is broken: an apply failed and pullgate could not roll it back. The host may be in a half state. This commit is a fix forward.", + "Reason: "+BrokenReason(q.Broken)) + } if q.Held != "" { head = append(head, "⚠ The last apply ("+SHA(q.Held)+") failed.") } diff --git a/internal/state/state.go b/internal/state/state.go index 4788650..73d2611 100644 --- a/internal/state/state.go +++ b/internal/state/state.go @@ -58,6 +58,15 @@ type InProgress struct { Started time.Time `json:"started"` } +// Broken is the state after an apply failed and its rollback failed too +// (DESIGN.md section 10). Only a successful fix forward, or root, ends it. +type Broken struct { + Since time.Time `json:"since"` + SHA string `json:"sha"` // the last commit that failed; a fix forward descends from it + OldSHA string `json:"old_sha"` // the applied commit that the failed rollback tried to restore + Reason string `json:"reason"` // the last error, as plain text +} + // State is the content of state.json. type State struct { Version int `json:"version"` @@ -68,6 +77,7 @@ type State struct { Questions []Question `json:"questions"` Marks map[string]*Mark `json:"marks"` InProgress *InProgress `json:"in_progress,omitempty"` + Broken *Broken `json:"broken,omitempty"` AskTimes []time.Time `json:"ask_times,omitempty"` LimitNote time.Time `json:"limit_note,omitempty"` CheckAt time.Time `json:"check_at,omitzero"` // the last check request that was taken (DESIGN.md section 8)