Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 35 additions & 1 deletion DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -704,6 +704,16 @@ They run with an empty environment plus:
- `PULLGATE_FAILED_SHA`: the held SHA, if there is one. The host may be
part way to it, not at `PULLGATE_OLD_SHA`.
- `PULLGATE_ROLLBACK`: `1` during a rollback, else empty.
- `PULLGATE_REPORT_FILE`: only for `apply`, and not in a rollback. The
path of an empty file (root's, mode 0600, in the per-process trees
directory) where `apply` may write short lines for the owner, for
example "skipped step x on purpose". After `apply` (also when it
failed), root reads the file: only a regular file, never a link
(`O_NOFOLLOW`), at most 64 KiB. Its first 10 lines are added to the
`applied` or `failed` message under "Report of apply:", each cleaned and
shown as code (section 12), at most 1500 characters in all; a last line
says how many lines were left out. A missing or empty file adds
nothing. `health` does not get the file.

**In a rollback**, `PULLGATE_SHA` and `PULLGATE_TREE` are the last good SHA
and its tree; `PULLGATE_OLD_SHA` and `PULLGATE_OLD_TREE` are the SHA that
Expand Down Expand Up @@ -749,6 +759,8 @@ Apply this commit?
First apply: there is no applied commit yet. All N files are listed.
or: Since <code>SHORT APPLIED SHA</code>:
⚠ Watched path changed: <code>path</code>
⚠ <code>text of a WARN: line of the summary</code>
⚠ N more warnings from the summary (only with more than 10)
⚠ Symlink out of the tree: <code>path</code> → <code>target</code>
⚠ Submodule skipped: <code>path</code>

Expand Down Expand Up @@ -783,12 +795,34 @@ cut to 200 characters, and then it is HTML-escaped. At most 50 commit
subjects are read. Every message and every edit is sent with link previews
off.

**Warnings from the summary.** The `[watch]` list catches changed paths;
only the owner's `summary` can find other risky changes, because it reads
the tree. A line of the `summary` output that starts with `WARN: ` (exactly
these 6 characters, at the start of the line) is taken out of the `<pre>`
block and shown in the head, after the watched-path warnings, as
`⚠ <code>text</code>`: the text after the prefix, cleaned and cut to 200
characters like any repo text. A CRLF line end is not part of the text,
and a `WARN: ` line with no text is dropped. At most 10 such lines are shown; more give
one line "⚠ N more warnings from the summary". A summary with only `WARN: `
lines has no "Summary:" block. When `summary` failed, its output is not
used, so no warning comes from it. A summary without `WARN: ` lines gives
the same message as before.

The message must fit one Telegram message (4096 characters). pullgate cuts
it to 3800 characters, counted on the escaped text in UTF-16 units: the head
(host, instance, SHA, the warnings) always stays, and when lines are left
out the last line says how many ("… 120 more lines left out"). The cut is on
whole lines, so it never splits an HTML tag or entity. The `<pre>` block is
closed after a cut.
closed after a cut. The body is cut from its end: first the rest of the
summary, then the file list, then the commits. The "Full:" line stays last.
When the head alone is too long, the path warnings (watched paths,
symlinks, submodules) are cut first and counted ("⚠ … and N more
warnings"); the summary warnings stay. Each summary warning line is at most
220 characters after the escape (200 plain characters fit), so 10 of them
take at most about 2250 and the watched paths keep room. Only when the fixed
head lines and the summary warnings alone do not fit (a very long broken
reason) are the last summary warnings counted in the "more warnings from
the summary" line, so that the message still fits.

## 13. Modes, limits, status

Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,8 +284,10 @@ known_hosts = "/etc/pullgate/NAME.known_hosts"
[commands]
check = ["/path/to/check"] # optional; runs as check_user; a failure means no question;
# exit 3 means "nothing to apply on this host": no question, no message
summary = ["/path/to/summary"] # optional; runs as check_user; its output is in the question
apply = ["/path/to/apply"] # required; runs as root after Apply
summary = ["/path/to/summary"] # optional; runs as check_user; its output is in the question;
# a line that starts with "WARN: " is shown at the top of the question
apply = ["/path/to/apply"] # required; runs as root after Apply; the first 10 lines it
# writes to $PULLGATE_REPORT_FILE are added to the applied/failed message
health = ["/path/to/health"] # optional; runs as root after apply
check_user = "pullgate-check"

Expand Down
20 changes: 20 additions & 0 deletions THREAT-MODEL.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,26 @@ or during the review.
reason is not cut away; root's tree paths in it become `NEW` and `OLD`.
The head of the question (host, instance, SHA, the warnings) always
comes first, before any text from the commit.
- **`WARN: ` lines of the summary in the head.** `summary` runs as
`check_user` on the commit's tree, so a hostile commit may make it print
`WARN: ` lines. They are shown only as cleaned code after `⚠ `, at most 10
and a count, after the watched-path warnings. They never push out the
fixed head lines (host, instance, SHA, the broken and held lines). Each
is at most 220 characters after the escape, so all 10 take at most about
2250 of the 3400 that the head may use: the watched-path warnings, root's
own, always keep room. Path warnings that still do not fit are counted
("⚠ … and N more warnings"), as before. *Left:* a hostile summary can
print decoy `WARN: ` lines first and push the owner's real ones into the
"N more warnings from the summary" count; the owner's `summary` should
print its own warnings first. They change nothing that root checks: the tap is bound to
the code and the SHA in root's state.
- **The report file of `apply`.** `apply` runs as root and is the owner's
own command, but it may copy text from the tree. Root makes the file
itself, in its own trees directory, and reads it only when it is still
that same file (same device and inode, a regular file, opened with
`O_NOFOLLOW`), at most 64 KiB. So `apply` cannot make root show another
file by a symlink or a hard link. Its lines are cleaned and shown as code, at most
10 lines and 1500 characters. A rollback and `health` get no report file.
- **Hostile text shown as a link in a trusted message.** A commit picks its
own file names, subjects and command output. Telegram turns text that
looks like a domain or a URL (`README.md`, `login.example.com`), an
Expand Down
90 changes: 75 additions & 15 deletions internal/gate/apply.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (
"io"
"io/fs"
"os"
"path/filepath"
"strings"
"syscall"

Expand Down Expand Up @@ -230,7 +231,7 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) (
defer t.remove()

wasBroken := st.Broken
err = e.tryApply(ctx, t, "apply", sha, old, failedSHA(st), false)
report, err := e.tryApply(ctx, t, "apply", sha, old, failedSHA(st), false)
if stopped(ctx, err) {
return "", err // in_progress stays: the next start pauses
}
Expand All @@ -247,10 +248,10 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) (
}
e.Log.Info("applied", "sha", sha)
if wasBroken != nil {
e.reply(q.ID, "✅ Applied "+code(sha)+". The fix forward worked: the instance is no longer broken.")
e.reply(q.ID, withReport("✅ Applied "+code(sha)+". The fix forward worked: the instance is no longer broken.", report))
return Applied, nil
}
e.reply(q.ID, "✅ Applied "+code(sha)+".")
e.reply(q.ID, withReport("✅ Applied "+code(sha)+".", report))
return Applied, nil
}

Expand All @@ -268,15 +269,15 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) (
return "", err
}
e.Log.Error("the fix forward failed; the instance stays broken", "sha", sha)
e.reply(q.ID, "❌ The fix forward "+code(sha)+" failed ("+failure+"). No rollback was tried. The instance stays broken.\n"+e.waysOut())
e.reply(q.ID, withReport("❌ The fix forward "+code(sha)+" failed ("+failure+"). No rollback was tried. The instance stays broken.\n"+e.waysOut(), report))
return StillBroken, nil
}
if e.Cfg.OnFailure == config.OnFailureRollback && old != "" {
st.InProgress = &state.InProgress{Phase: "rollback", SHA: sha, OldSHA: old, Started: e.now()}
if err := e.Dir.Save(st); err != nil {
return "", err
}
rerr := e.tryApply(ctx, t, "rb", old, sha, sha, true)
_, rerr := e.tryApply(ctx, t, "rb", old, sha, sha, true)
if stopped(ctx, rerr) {
return "", rerr
}
Expand All @@ -285,7 +286,7 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) (
if err := e.Dir.Save(st); err != nil {
return "", err
}
e.reply(q.ID, "❌ The apply of "+code(sha)+" failed ("+failure+"). Rolled back to "+code(old)+".")
e.reply(q.ID, withReport("❌ The apply of "+code(sha)+" failed ("+failure+"). Rolled back to "+code(old)+".", report))
return RolledBack, nil
}
// The broken state (DESIGN.md section 10). If this save fails,
Expand All @@ -306,7 +307,7 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) (
return RollbackError, err
}
e.Log.Error("the apply and its rollback failed; the instance is broken", "sha", sha, "old", old)
e.reply(q.ID, "❌ The apply of "+code(sha)+" failed ("+failure+"), and the rollback to "+code(old)+" failed too ("+message.Reason(rerr.Error())+"). The host may be in a half state. The instance is now broken.\n"+e.waysOut())
e.reply(q.ID, withReport("❌ The apply of "+code(sha)+" failed ("+failure+"), and the rollback to "+code(old)+" failed too ("+message.Reason(rerr.Error())+"). The host may be in a half state. The instance is now broken.\n"+e.waysOut(), report))
return RollbackError, nil
}
// on_failure = "pause", or nothing to roll back to: hold the SHA.
Expand All @@ -315,7 +316,7 @@ func (e *Env) applySHA(ctx context.Context, st *state.State, q state.Question) (
if err := e.Dir.Save(st); err != nil {
return "", err
}
e.reply(q.ID, "❌ The apply of "+code(sha)+" failed ("+failure+"). It is held and not tried again. A newer commit will be asked, so you can fix it forward.")
e.reply(q.ID, withReport("❌ The apply of "+code(sha)+" failed ("+failure+"). It is held and not tried again. A newer commit will be asked, so you can fix it forward.", report))
return Held, nil
}

Expand Down Expand Up @@ -343,23 +344,82 @@ func stopped(ctx context.Context, err error) bool {
return err != nil && (ctx.Err() != nil || errors.Is(err, ErrStopped))
}

// tryApply exports fresh trees and runs apply, then health.
func (e *Env) tryApply(ctx context.Context, t *trees, tag, sha, old, failed string, rollback bool) error {
// tryApply exports fresh trees and runs apply, then health. Outside a
// rollback, apply gets a report file; report is its text for the
// message (DESIGN.md section 11), "" when it is empty.
func (e *Env) tryApply(ctx context.Context, t *trees, tag, sha, old, failed string, rollback bool) (report string, err error) {
p, err := e.prepare(ctx, t, tag, sha, old)
if err != nil {
return fmt.Errorf("prepare: %w", err)
return "", fmt.Errorf("prepare: %w", err)
}
p.env.Failed = failed
p.env.Rollback = rollback
if !rollback {
p.env.Report = filepath.Join(t.dir, tag+"-report")
f, err := os.OpenFile(p.env.Report, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600)
if err != nil {
return "", fmt.Errorf("prepare: make the report file: %w", err)
}
made, serr := f.Stat()
_ = f.Close()
if serr != nil {
return "", fmt.Errorf("prepare: the report file: %w", serr)
}
defer func() { report = e.readReport(p.env.Report, made) }()
}
if err := e.runRoot(ctx, "apply", e.Cfg.Commands.Apply, p.env); err != nil {
return err
return "", err
}
if argv := e.Cfg.Commands.Health; len(argv) > 0 {
if err := e.runRoot(ctx, "health", argv, p.env); err != nil {
return err
h := p.env
h.Report = "" // the report file is for apply only
if err := e.runRoot(ctx, "health", argv, h); err != nil {
return "", err
}
}
return nil
return "", nil
}

// maxReport is the most bytes of the report file that root reads.
const maxReport = 64 << 10

// readReport reads the report file of apply: the same file that root
// made (made), not a link, at most maxReport bytes. A file that cannot be read is logged and
// shown as a short note; it never stops the report of the apply.
func (e *Env) readReport(path string, made os.FileInfo) string {
f, err := os.OpenFile(path, os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0) //nolint:gosec // root made this file in its own trees directory
if errors.Is(err, fs.ErrNotExist) {
return "" // apply removed it: no report
}
if err != nil {
e.Log.Warn("read the report file of apply", "err", err)
return "(The report file of apply could not be read. See the journal.)"
}
defer func() { _ = f.Close() }()
// Only the file that root made: not a link, not another file put in
// its place (a hard link to another file has another inode).
if fi, err := f.Stat(); err != nil || !fi.Mode().IsRegular() || !os.SameFile(fi, made) {
e.Log.Warn("the report file of apply is not the file that root made", "path", path)
return "(The report file of apply is not the file that pullgate made, so it is not shown.)"
}
b, err := io.ReadAll(io.LimitReader(f, maxReport+1))
if err != nil {
e.Log.Warn("read the report file of apply", "err", err)
return "(The report file of apply could not be read. See the journal.)"
}
truncated := len(b) > maxReport
if truncated {
b = b[:maxReport]
}
return message.Report(string(b), truncated)
}

// withReport adds the report of apply to a message.
func withReport(text, report string) string {
if report == "" {
return text
}
return text + "\n" + report
}

// afterApply closes questions whose base is no longer the applied SHA, so
Expand Down
7 changes: 6 additions & 1 deletion internal/gate/env.go
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,7 @@ func (c *capBuf) Write(p []byte) (int, error) {
// cmdEnv is the environment of a command (DESIGN.md section 11).
type cmdEnv struct {
SHA, OldSHA, Tree, OldTree, Changed, Failed string
Report string // PULLGATE_REPORT_FILE; only for apply
Rollback bool
}

Expand All @@ -151,7 +152,7 @@ func (e *Env) environ(c cmdEnv) []string {
if c.Rollback {
rb = "1"
}
return []string{
env := []string{
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"PULLGATE_INSTANCE=" + e.Cfg.Name,
"PULLGATE_SHA=" + c.SHA,
Expand All @@ -162,6 +163,10 @@ func (e *Env) environ(c cmdEnv) []string {
"PULLGATE_FAILED_SHA=" + c.Failed,
"PULLGATE_ROLLBACK=" + rb,
}
if c.Report != "" {
env = append(env, "PULLGATE_REPORT_FILE="+c.Report)
}
return env
}

// command builds the exec.Cmd of argv. With sandbox it runs as check_user
Expand Down
7 changes: 6 additions & 1 deletion internal/gate/fixture_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,16 @@ func newFixture(t *testing.T, top, sections string) *fixture {
c := f.ctrl
f.script("apply.sh", `echo "apply $PULLGATE_SHA old=$PULLGATE_OLD_SHA rb=$PULLGATE_ROLLBACK failed=$PULLGATE_FAILED_SHA" >> `+c+`/log
if [ "$PULLGATE_ROLLBACK" = 1 ] && [ -e `+c+`/fail-rollback ]; then exit 3; fi
echo "report=$PULLGATE_REPORT_FILE" >> `+c+`/env
if [ -n "$PULLGATE_REPORT_FILE" ] && [ -e "$PULLGATE_TREE/report" ]; then cat "$PULLGATE_TREE/report" > "$PULLGATE_REPORT_FILE"; fi
if [ -n "$PULLGATE_REPORT_FILE" ] && [ -e "$PULLGATE_TREE/report-link" ]; then rm -f "$PULLGATE_REPORT_FILE"; ln -s `+c+`/secret "$PULLGATE_REPORT_FILE"; fi
if [ -n "$PULLGATE_REPORT_FILE" ] && [ -e "$PULLGATE_TREE/report-hardlink" ]; then rm -f "$PULLGATE_REPORT_FILE"; ln `+c+`/secret "$PULLGATE_REPORT_FILE"; fi
if [ -e "$PULLGATE_TREE/fail-apply" ]; then exit 2; fi
tr '\0' '\n' < "$PULLGATE_CHANGED_FILES" > `+c+`/changed
cat "$PULLGATE_TREE/value" > `+c+`/live
`)
f.script("health.sh", `if [ -e "$PULLGATE_TREE/fail-health" ]; then exit 1; fi
f.script("health.sh", `echo "health report=$PULLGATE_REPORT_FILE" >> `+c+`/env
if [ -e "$PULLGATE_TREE/fail-health" ]; then exit 1; fi
`)
f.script("check.sh", `id -u > `+c+`/check-uid
echo x >> `+c+`/check-count
Expand Down
Loading