Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .claude/agents/security-reviewer.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,11 @@ have the bot token.
- The state moves forward only after `apply` and `health` succeed.
`in_progress` is synced before `apply` runs; found at start, it pauses.
- `on_failure`: `rollback` runs the old apply and health, and a failed
rollback writes the pause file; `pause` holds the SHA and does not write
the pause file. The code default is `pause`.
rollback makes the instance broken: only a fix forward (a descendant of
the failed SHA) is asked, always with a tap (also in `auto` mode), no
rollback is tried, and only a good fix forward or root's `--resume` ends
it; `pause` holds the SHA and does not write the pause file. The code
default is `pause`.

**No code before the tap (rule 3)**
- Before Apply, nothing from the commit runs. git only reads objects.
Expand Down
51 changes: 45 additions & 6 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -547,10 +547,10 @@ The steps, with the state synced to disk at each arrow:
- `rollback`, with an applied SHA: write `in_progress = {phase:
"rollback", sha, old_sha}` →. Run `apply` with the last good SHA, then
`health`. If that works, clear `in_progress` → and report "failed,
rolled back to OLD". If it fails too, write the pause file, clear
`in_progress` → and report "failed, the rollback failed too, the
instance is paused". With no applied SHA there is nothing to roll back
to: hold the SHA, as with `pause`.
rolled back to OLD". If it fails too, the instance becomes
**broken** (below): write `broken = {since, sha, old_sha, reason}` and
clear `in_progress` in one save →, and report it. With no applied SHA
there is nothing to roll back to: hold the SHA, as with `pause`.
- `pause`: do not roll back and do not write the pause file. The SHA
becomes the held SHA. Clear `in_progress` →. It is not tried again,
every report and `pullgate status` say that the last apply failed, and
Expand All @@ -565,6 +565,40 @@ tries is asked again on the next run; with `max_failed_tries` failed tries
it is not asked again until a new commit (or `--ask-again`). With `pause`,
a failed SHA is held at once and the count does not matter.

**Broken.** After a failed rollback the host may be in a half state. A
plain pause would need root on the host for every such case, and most of
them are fixed by a new commit. So the instance becomes broken instead:

- One message goes out when it starts: the reason (the errors of the apply
and of the rollback, each cut in the middle so that its end stays,
code6x/pullgate#25) and the two ways out: a fix forward, or root on the
host.
- The instance keeps fetching. It asks only for a **fix forward**: a tip
that descends from the failed SHA, and is not the failed SHA or the old
applied SHA. This holds with `allow_older` too. Any other tip is refused
with a notice. The failed SHA is not tried again, whatever
`max_failed_tries` says.
- The question has the lines "⚠ The instance is broken: an apply failed
and pullgate could not roll it back. The host may be in a half state.
This commit is a fix forward." and "Reason: …" at the top. The reason
is in `<code>`, after the fixed line. In `auto` mode the owner is asked too:
nothing is applied by itself on a half state. `questions_per_day`
counts these questions.
- At the tap, root checks the fix forward rule again. A fix forward that
succeeds (apply and `health`) ends the broken state. One that fails is
**not** rolled back: the instance stays broken, the failed fix forward
becomes the new failed SHA and reason, and the next newer commit is
asked with the warning.
- The broken state never ends by itself. It does not stop a pause: a
pause file still stops everything. Root ends it with `pullgate status
NAME --resume`. No command from Telegram can end it.
- During a fix forward, `PULLGATE_FAILED_SHA` is the failed SHA.
- When it starts, every question asked before is dropped (the open one
is closed in the chat), so no tap on a question without the warning
applies. The next run asks again.
- If the save of the broken state fails, `in_progress` stays in the file,
and the next start pauses (a crash, below).

**A crash** leaves `in_progress` in the state. The next run or apply cannot
know what happened on the host. It writes the pause file, marks the SHA
failed and reports "the apply (or rollback) of SHA was cut off; check the
Expand Down Expand Up @@ -628,6 +662,9 @@ The question is HTML (`parse_mode = HTML`), with link previews off:

```
pullgate on <b>HOST</b>, instance <b>NAME</b>
⚠ The instance is broken: an apply failed and pullgate could not roll it
back. The host may be in a half state. This commit is a fix forward.
Reason: <code>reason</code> (both only when broken, section 10)
⚠ The last apply (SHA) failed. (only when a SHA is held)
⚠ This commit is not a descendant of … (only with allow_older)
Apply this commit?
Expand Down Expand Up @@ -693,7 +730,8 @@ closed after a cut.
- `off`: `pullgate run` and `pullgate apply` do nothing.

`pullgate status NAME` prints the mode, the applied SHA, the open question,
the held SHA, the pause file and an apply in progress. Flags:
the held SHA, the broken state ("broken since …, last error …"), the pause
file and an apply in progress. Flags:

- `--ask-again`: clear the marks of the last fetched tip (expired, skipped,
check failed, refused, failed, held), so the next run asks again. If a
Expand All @@ -703,7 +741,8 @@ the held SHA, the pause file and an apply in progress. Flags:
approval for the old code finds no question and does nothing, even one
that the listener wrote before the close;
- `--pause`: write the pause file;
- `--resume`: remove the pause file and clear an apply in progress.
- `--resume`: remove the pause file, clear an apply in progress, and end
the broken state (section 10). Run it only after you checked the host.

The listener (`pullgate listen --config /etc/pullgate/listener.toml`):

Expand Down
7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,12 @@ sections 5 and 11.

- `on_failure = "rollback"`: pullgate runs `apply` again with the last good
commit, then `health`. So `apply` must be safe to run again on an older
tree. If the rollback fails too, the instance is paused.
tree. If the rollback fails too, the instance is **broken**: the host
may be in a half state. pullgate tells you once, with the reason. Then it
asks only for a fix forward (a newer commit on top of the failed one),
with a warning, also in `auto` mode, and tries no rollback. A fix forward
that works ends it, or root runs `pullgate status NAME --resume` after a
look at the host.
- `on_failure = "pause"`: no rollback. The failed commit is held and not
tried again. A newer commit is still asked, with a warning at the top, so
you can fix it forward.
Expand Down
14 changes: 11 additions & 3 deletions THREAT-MODEL.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,9 +236,17 @@ having the host manage itself.
*Attack or accident.* `apply` fails, and running it again on the old tree
fails too. The host may be half changed.

*Defense.* With `on_failure = "rollback"`, a failed rollback writes the pause
file and reports it. Nothing more runs until the owner looks and runs
`pullgate status NAME --resume`. With `on_failure = "pause"` there is no
*Defense.* With `on_failure = "rollback"`, a failed rollback makes the
instance **broken** (DESIGN.md section 10) and reports it once, with the
reason. Nothing is applied until the owner taps Apply on a **fix
forward**: a newer commit on top of the failed one, asked with a strong
warning. The failed commit and the old commit are never asked again, and
no rollback is tried against the half state. In `auto` mode the owner is
asked too. Only a fix forward that passes `health`, or root with
`pullgate status NAME --resume`, ends it. It never ends by itself, and no
message from Telegram ends it, so a stolen bot token or a compromised
listener (T2) cannot lift it: the most they can do is not deliver a tap.
A pause file still stops everything, also while broken. With `on_failure = "pause"` there is no
rollback: the failed SHA is held, and the owner can fix it forward. The
applied SHA does not change after a failure.

Expand Down
86 changes: 71 additions & 15 deletions internal/gate/apply.go
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,23 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) (
e.reply(q.ID, code(sha)+" is applied already.")
return Ignored, nil
}
if b := st.Broken; b != nil {
// Only a fix forward: a commit on top of the last failed one.
// An auto question is never applied here: the owner must tap.
ok := sha != b.SHA && sha != b.OldSHA && !q.Auto
if ok {
var err error
if ok, err = e.Git.IsAncestor(ctx, b.SHA, sha); err != nil {
return "", err
}
}
if !ok {
st.SetMark(sha, state.MarkRefused, now)
e.Log.Warn("broken: refused, not a fix forward", "sha", sha, "failed", b.SHA)
e.reply(q.ID, "Refused "+code(sha)+": the instance is broken, and only a commit on top of the failed commit "+code(b.SHA)+" is applied.")
return Refused, nil
}
}
if st.Applied != "" && !e.Cfg.Limits.AllowOlder {
ok, err := e.Git.IsAncestor(ctx, st.Applied, sha)
if err != nil {
Expand All @@ -212,13 +229,14 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) (
}
defer t.remove()

err = e.tryApply(ctx, t, "apply", sha, old, st.Held, false)
wasBroken := st.Broken
err = e.tryApply(ctx, t, "apply", sha, old, failedSHA(st), false)
if stopped(ctx, err) {
return "", err // in_progress stays: the next start pauses
}
if err == nil {
st.Applied, st.AppliedAt = sha, e.now()
st.InProgress, st.Held = nil, ""
st.InProgress, st.Held, st.Broken = nil, "", nil
st.Marks = map[string]*state.Mark{}
if gerr := e.Git.SetApplied(ctx, sha); gerr != nil {
e.Log.Warn("keep the applied ref", "err", gerr)
Expand All @@ -228,6 +246,10 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) (
return "", err
}
e.Log.Info("applied", "sha", sha)
if wasBroken != nil {
e.reply(q.ID, "✅ Applied "+code(sha)+". The fix forward worked: the instance is no longer broken.")
return Applied, nil
}
e.reply(q.ID, "✅ Applied "+code(sha)+".")
return Applied, nil
}
Expand All @@ -237,7 +259,18 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) (
if err := e.Dir.Save(st); err != nil {
return "", err
}
failure := esc(err)
failure := message.Reason(err.Error())
if b := st.Broken; b != nil {
// A failed fix forward: no rollback against a half state.
b.SHA, b.Reason = sha, "the fix forward "+message.Short(sha)+" failed: "+message.CutLine(err.Error())
st.InProgress = nil
if err := e.Dir.Save(st); err != nil {
return "", err
}
e.Log.Error("the fix forward failed; the instance stays broken", "sha", sha)
e.reply(q.ID, "❌ The fix forward "+code(sha)+" failed ("+failure+"). No rollback was tried. The instance stays broken.\n"+e.waysOut())
return StillBroken, nil
}
if e.Cfg.OnFailure == config.OnFailureRollback && old != "" {
st.InProgress = &state.InProgress{Phase: "rollback", SHA: sha, OldSHA: old, Started: e.now()}
if err := e.Dir.Save(st); err != nil {
Expand All @@ -255,17 +288,25 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) (
e.reply(q.ID, "❌ The apply of "+code(sha)+" failed ("+failure+"). Rolled back to "+code(old)+".")
return RolledBack, nil
}
if perr := e.Dir.Pause("the rollback to " + old + " failed after the apply of " + sha + " failed"); perr != nil {
// in_progress stays, so the next start pauses (cutOff).
e.Log.Error("write the pause file", "err", perr)
e.reply(q.ID, "❌ The apply of "+code(sha)+" failed, the rollback failed too, and the pause file could not be written. Check the host now.")
return RollbackError, perr
}
// The broken state (DESIGN.md section 10). If this save fails,
// in_progress stays in the file, and the next start pauses.
st.InProgress = nil
st.Broken = &state.Broken{
Since: e.now(), SHA: sha, OldSHA: old,
Reason: "the apply of " + message.Short(sha) + " failed: " + message.CutLine(err.Error()) +
"; the rollback to " + message.Short(old) + " failed: " + message.CutLine(rerr.Error()),
}
// Questions asked before have no warning (or are auto ones): drop
// them all, so no old tap applies. The next run asks again.
e.dropQuestions(st, "the instance is broken. A fix forward is asked again, with a warning.")
if err := e.Dir.Save(st); err != nil {
return "", err
// Keep in_progress, so the next start pauses (a crash).
st.Broken = nil
st.InProgress = &state.InProgress{Phase: "rollback", SHA: sha, OldSHA: old, Started: e.now()}
return RollbackError, err
}
e.reply(q.ID, "❌ The apply of "+code(sha)+" failed ("+failure+"), and the rollback to "+code(old)+" failed too ("+esc(rerr)+"). The instance is paused. Check the host, then run as root: <code>pullgate status "+e.Cfg.Name+" --resume</code>")
e.Log.Error("the apply and its rollback failed; the instance is broken", "sha", sha, "old", old)
e.reply(q.ID, "❌ The apply of "+code(sha)+" failed ("+failure+"), and the rollback to "+code(old)+" failed too ("+message.Reason(rerr.Error())+"). The host may be in a half state. The instance is now broken.\n"+e.waysOut())
return RollbackError, nil
}
// on_failure = "pause", or nothing to roll back to: hold the SHA.
Expand All @@ -278,6 +319,25 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) (
return Held, nil
}

// dropQuestions removes every question from the state, and closes the
// open one in the chat.
func (e *Env) dropQuestions(st *state.State, why string) {
for _, q := range st.Questions {
if m := st.Marks[q.SHA]; m != nil && m.Mark == state.MarkAsked {
delete(st.Marks, q.SHA)
}
if q.Open && !q.Auto {
e.edit(q.ID, "This question for "+code(q.SHA)+" is closed: "+why)
}
}
st.Questions = nil
}

// waysOut says how the broken state ends.
func (e *Env) waysOut() string {
return "Two ways out: push a fix forward (a commit on top of the failed one; it is asked with a warning), or check the host and run as root: <code>pullgate status " + e.Cfg.Name + " --resume</code>"
}

// stopped reports whether err comes from a stop of pullgate itself.
func stopped(ctx context.Context, err error) bool {
return err != nil && (ctx.Err() != nil || errors.Is(err, ErrStopped))
Expand Down Expand Up @@ -317,7 +377,3 @@ func (e *Env) afterApply(st *state.State) {
}
st.Questions = keep
}

// esc shows an error in a message, in <code>: it may hold text from the
// repo (DESIGN.md section 12).
func esc(err error) string { return message.Code(err.Error()) }
Loading