Skip to content

Commit 36fdd67

Browse files
authored
fix(status): stop a leftover ACME resolver from reporting divergence (#236)
* fix(status): judge certificates only from resolvers Onebox configures A resolver key left in acme.json by an older proxy configuration keeps a certificate that nothing renews. ob status judged it like any other and reported a healthy host as diverged once it entered the renewal floor. Certificates now carry the resolver key and store file they came from. Only certificates from letsencrypt and onebox-wildcard can be overdue. Others are listed as orphaned, with the key to remove, and never set Diverged. Both status views share one judging helper and one sort order. Fixes #235 * chore(deps): upgrade Go and site dependencies Go: xo/terminfo v1.3.1, go.yaml.in/yaml/v4 v4.0.0. Site: astro 7.4.7, @astrojs/starlight 0.42.10, @astrojs/markdown-remark 7.3.6. spf13/pflag stays at v1.0.10: v1.1.0 prints every boolean flag as --flag[=true|false] in help output. TypeScript stays on 6.x. * fix(status): judge certificates by the resolvers the static config defines Hardcoding letsencrypt and onebox-wildcard as managed left the #235 symptom in place for a host that dropped its wildcard routes, and for a resolver key sitting in a store file it does not write. Status now reads certificatesResolvers.*.acme.storage from the static configuration it already stages, and renews-or-orphans each certificate by its (resolver, store file) pair. Custom resolvers in a project-owned traefik.yml are judged too. When the configuration cannot be read, no certificate is judged overdue or orphaned. The orphan hint names the store's full host path. * fix(status): judge certificates by the host's applied static config Judging by the locally staged config was wrong whenever the host ran an older one: a resolver removed locally but still renewing on the host was called orphaned, and the hint told the operator to delete a certificate the proxy still serves. Status now reads traefik.yml or traefik.yaml from the host's proxy config directory in the certificate read and reads every store file a resolver there names, as well as acme.json and acme-wildcard.json. When that file is refused or does not parse, the issue is recorded, nothing is called orphaned and certificates are judged by expiry alone, so a real overdue renewal is never hidden. The local config read is back to its original job. The /letsencrypt mount target is one constant shared by the compose template, the static config and the parser. * fix(status): never orphan a certificate whose resolver store is unclear A resolver with a relative storage path, mixed-case keys, or a storage value status cannot place was treated as unconfigured, so its renewing certificate was called orphaned with a hint to delete it. CertificateStores now matches keys without regard to case, resolves a relative storage path from /, and lists every defined resolver, mapping an unplaceable store to "". Certificates of such a resolver are judged by expiry and never orphaned. Any failed read of the applied config is recorded, not raised, so the store reads in the same step still run. The parse issue no longer quotes YAML errors, which can repeat values from the file, and the overdue issue names the resolver and store. * fix(status): drop deletion advice and match Traefik's config lookup Traefik reads its static configuration only at startup, so a proxy not restarted after an apply can still renew under a key the file no longer names. The orphaned hint now says the certificate is not renewed by the applied configuration and names the resolver and store, without telling the operator to delete it. Resolver names match without regard to case. traefik.yaml is read before traefik.yml, the order Traefik searches. The text view prints a config issue before an unreadable store returns, as the JSON snapshot does, and every config issue says certificates are judged by expiry alone.
1 parent 58323d9 commit 36fdd67

10 files changed

Lines changed: 1071 additions & 316 deletions

File tree

‎go.mod‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,8 @@ require (
4040
github.com/rogpeppe/go-internal v1.15.0 // indirect
4141
github.com/sirupsen/logrus v1.10.2 // indirect
4242
github.com/xhit/go-str2duration/v2 v2.2.0 // indirect
43-
github.com/xo/terminfo v1.2.0 // indirect
44-
go.yaml.in/yaml/v4 v4.0.0-rc.6 // indirect
43+
github.com/xo/terminfo v1.3.1 // indirect
44+
go.yaml.in/yaml/v4 v4.0.0 // indirect
4545
golang.org/x/sync v0.24.0 // indirect
4646
golang.org/x/sys v0.49.0 // indirect
4747
golang.org/x/text v0.43.0 // indirect

‎go.sum‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -72,13 +72,13 @@ github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWD
7272
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
7373
github.com/xhit/go-str2duration/v2 v2.2.0 h1:SofrD1D+8IuhhkP+OiG1ij4Zq+FiiYSEY/sORBanDG0=
7474
github.com/xhit/go-str2duration/v2 v2.2.0/go.mod h1:ohY8p+0f07DiV6Em5LKB0s2YpLtXVyJfNt1+BlmyAsU=
75-
github.com/xo/terminfo v1.2.0 h1:d0ZTOCpuGE0lwSAOs0zcJjwz3jWQyqcRt9XbGJpCOl4=
76-
github.com/xo/terminfo v1.2.0/go.mod h1:lGzkSo8Fe7IRh/w+Gqz7n5mDog4FVXCj3gy/DYTBqio=
75+
github.com/xo/terminfo v1.3.1 h1:4ZwPCuu4PEFw0PkNp3Hv/x3gqvgLCzYhurszsj101cM=
76+
github.com/xo/terminfo v1.3.1/go.mod h1:lGzkSo8Fe7IRh/w+Gqz7n5mDog4FVXCj3gy/DYTBqio=
7777
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
7878
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
7979
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
80-
go.yaml.in/yaml/v4 v4.0.0-rc.6 h1:1h7H1ohdUh93/FyE4YaDa1Zh64K6VVbjF4K6WUxMtH4=
81-
go.yaml.in/yaml/v4 v4.0.0-rc.6/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
80+
go.yaml.in/yaml/v4 v4.0.0 h1:/LG1pNO84Z4l603j/xMn69CcS5OYhMQ/f7l2LoQUGuw=
81+
go.yaml.in/yaml/v4 v4.0.0/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
8282
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
8383
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
8484
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=

‎internal/engine/proxystatus.go‎

Lines changed: 182 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,10 @@ import (
44
"context"
55
"fmt"
66
"os"
7+
"sort"
78
"strconv"
89
"strings"
10+
"time"
911

1012
"github.com/labstack/onebox/internal/app"
1113
"github.com/labstack/onebox/internal/proxy"
@@ -19,21 +21,25 @@ const renewalFloorDays = 21
1921
// proxyRaw is the managed proxy's status reads, gathered concurrently with the
2022
// app-side reads.
2123
type proxyRaw struct {
22-
ids []string
23-
health string // proxy container health, parsed from docker ps .Status
24-
discovery bool // isolated Docker discovery controller is running
25-
applied string // config and Compose identity the host applied
26-
owner string // sole application identity from the host owner record
27-
ownerEnv string // environment identity from the owner record
28-
acme []string // raw ACME stores; parsed at render, and keys never leave
29-
localHash string // locally staged config and Compose identity (computed offline)
24+
ids []string
25+
health string // proxy container health, parsed from docker ps .Status
26+
discovery bool // isolated Docker discovery controller is running
27+
applied string // config and Compose identity the host applied
28+
owner string // sole application identity from the host owner record
29+
ownerEnv string // environment identity from the owner record
30+
acme []acmeStore // raw ACME stores; parsed at render, and keys never leave
31+
certStores map[string]string // resolver → ACME store file, from the applied static config; nil when unknown
32+
localHash string // locally staged config and Compose identity (computed offline)
3033
// Why a read could not be trusted, when it could not. Recorded rather
3134
// than raised: gather returns on the first error and Status renders
3235
// nothing after it, so raising costs the operator every other fact about
3336
// the host at exactly the moment it is in a strange state.
3437
ownerIssue string
3538
appliedIssue string
3639
acmeIssue string
40+
// certConfigIssue: the applied static configuration could not be read or
41+
// parsed, so certificates are judged by expiry alone.
42+
certConfigIssue string
3743
}
3844

3945
func (px proxyRaw) ownershipIssue(application, environment string) string {
@@ -154,7 +160,13 @@ func (e *Engine) proxyReads(ctx context.Context, px *proxyRaw) []func() error {
154160
return nil
155161
},
156162
func() error {
157-
for _, name := range []string{"acme.json", "acme-wildcard.json"} {
163+
// The static configuration the host applied says which resolver
164+
// writes which store, and so which stores to read. It is read
165+
// first, in the same step as the stores it names.
166+
if err := e.readCertificateConfig(ctx, hp, px); err != nil {
167+
return err
168+
}
169+
for _, name := range certificateStoreFiles(px.certStores) {
158170
path := hp.Acme + "/" + name
159171
res, err := e.T.Run(ctx, readableFileProbe(path))
160172
if err != nil {
@@ -167,7 +179,7 @@ func (e *Engine) proxyReads(ctx context.Context, px *proxyRaw) []func() error {
167179
if res.ExitCode != 0 {
168180
return statusReadResult("proxy certificate store", res, nil)
169181
}
170-
px.acme = append(px.acme, res.Stdout)
182+
px.acme = append(px.acme, acmeStore{file: name, path: path, body: res.Stdout})
171183
}
172184
return nil
173185
},
@@ -184,6 +196,80 @@ func (e *Engine) proxyReads(ctx context.Context, px *proxyRaw) []func() error {
184196
}
185197
}
186198

199+
// readCertificateConfig reads the static configuration the host applied and
200+
// records which store each of its resolvers writes. The host's file is the
201+
// one that matters: the running proxy renews by it, and a local change that
202+
// was never applied renews nothing yet. A failed read or a configuration that
203+
// does not parse is recorded, not raised, and leaves certStores nil so the
204+
// certificates are still judged by expiry. Only a transport error is raised. No file at all is a host that
205+
// never applied a proxy, which has no certificates to judge.
206+
func (e *Engine) readCertificateConfig(ctx context.Context, hp proxy.Paths, px *proxyRaw) error {
207+
for _, name := range proxy.StaticConfigNames {
208+
path := hp.ConfigDir + "/" + name
209+
res, err := e.T.Run(ctx, readableFileProbe(path))
210+
if err != nil {
211+
return err
212+
}
213+
if issue, refused := statusFileIssue("the applied static configuration", path, res); refused {
214+
px.certConfigIssue = issue + "; certificates are judged by expiry alone"
215+
return nil
216+
}
217+
// Any other failure, such as cat losing a race with ob proxy apply
218+
// swapping the directory, is recorded too: the stores this step reads
219+
// next matter more than the configuration that refines them.
220+
if res.ExitCode != 0 {
221+
px.certConfigIssue = fmt.Sprintf("the applied static configuration at %s could not be read (exit %d); certificates are judged by expiry alone", path, res.ExitCode)
222+
return nil
223+
}
224+
if strings.TrimSpace(res.Stdout) == "" {
225+
continue
226+
}
227+
// The parse error is not quoted: it can repeat values from the file,
228+
// and the message reaches the terminal and the JSON snapshot.
229+
stores, err := proxy.CertificateStores([]byte(res.Stdout))
230+
if err != nil {
231+
px.certConfigIssue = fmt.Sprintf("the applied static configuration at %s is not valid YAML; certificates are judged by expiry alone", path)
232+
return nil
233+
}
234+
px.certStores = stores
235+
return nil
236+
}
237+
return nil
238+
}
239+
240+
// resolverStore looks a resolver up without regard to case, because Traefik
241+
// matches its configuration that way and a store key may differ in case from
242+
// the name the configuration spells. A missed match would call a renewing
243+
// certificate a leftover, so the lenient match is the safe one.
244+
func resolverStore(stores map[string]string, resolver string) (string, bool) {
245+
if file, ok := stores[resolver]; ok {
246+
return file, true
247+
}
248+
for name, file := range stores {
249+
if strings.EqualFold(name, resolver) {
250+
return file, true
251+
}
252+
}
253+
return "", false
254+
}
255+
256+
// certificateStoreFiles lists the store files to read: the two Onebox has
257+
// always written, so that keys left in them stay visible, plus any other file
258+
// the applied configuration points a resolver at.
259+
func certificateStoreFiles(stores map[string]string) []string {
260+
files := []string{"acme.json", "acme-wildcard.json"}
261+
seen := map[string]bool{"acme.json": true, "acme-wildcard.json": true}
262+
var extra []string
263+
for _, file := range stores {
264+
if file != "" && !seen[file] {
265+
seen[file] = true
266+
extra = append(extra, file)
267+
}
268+
}
269+
sort.Strings(extra)
270+
return append(files, extra...)
271+
}
272+
187273
// readableFileProbe reads a status file that may legitimately be absent:
188274
// 0 with the contents on stdout, 2 present but unreadable, 5 absence could not
189275
// be established, 0 and empty when there is nothing there.
@@ -310,36 +396,113 @@ func (e *Engine) renderProxy(px proxyRaw) (bool, error) {
310396
fmt.Fprintf(e.Opts.Out, "proxy %-12s %-10s %s owner: %s\n", proxy.ContainerName, health, state, owner)
311397

312398
// cert runway — the renewal loop is the proxy's one silent failure mode
399+
// Printed before the store check so the text view names the same issues
400+
// as the JSON snapshot when both reads fail.
401+
if px.certConfigIssue != "" {
402+
fmt.Fprintf(e.Opts.Out, " cert resolvers unknown ⚠ (%s)\n", px.certConfigIssue)
403+
diverged = true
404+
}
313405
if px.acmeIssue != "" {
314406
fmt.Fprintf(e.Opts.Out, " cert store unreadable ⚠ (%s)\n", px.acmeIssue)
315407
return true, nil
316408
}
317-
certs, err := proxyCertExpiries(px.acme)
409+
certs, err := judgeCertificates(px.acme, px.certStores, e.Opts.Now())
318410
if err != nil {
319411
fmt.Fprintf(e.Opts.Out, " cert store unreadable ⚠ (%v)\n", err)
320412
return true, nil
321413
}
322414
for _, c := range certs {
323-
days := int(c.NotAfter.Sub(e.Opts.Now()).Hours() / 24)
324415
mark := ""
325-
if days < renewalFloorDays {
416+
switch {
417+
case c.Orphaned:
418+
mark = e.ui.Dim(fmt.Sprintf(" not renewed by the applied configuration (resolver %q in %s)", c.Resolver, c.Store))
419+
case c.Overdue:
326420
mark = e.ui.Warn(" RENEWAL OVERDUE ⚠ (lego renews at 30d — check CF_DNS_API_TOKEN / proxy logs)")
327421
diverged = true
328422
}
329-
e.ui.Println(fmt.Sprintf(" cert %-20s expires %s %s%s", c.Domain, c.NotAfter.UTC().Format("2006-01-02"), e.ui.Dim(fmt.Sprintf("(%dd)", days)), mark))
423+
e.ui.Println(fmt.Sprintf(" cert %-20s expires %s %s%s", c.Domain, c.NotAfter.UTC().Format("2006-01-02"), e.ui.Dim(fmt.Sprintf("(%dd)", c.Days)), mark))
330424
}
331425
return diverged, nil
332426
}
333427

334-
func proxyCertExpiries(stores []string) ([]proxy.CertExpiry, error) {
335-
var out []proxy.CertExpiry
336-
for _, store := range stores {
337-
certs, err := proxy.CertExpiries([]byte(store))
428+
// acmeStore is one ACME store file as read from the host. The body holds
429+
// private keys, so it is parsed in place and never rendered.
430+
type acmeStore struct {
431+
file string // file name, as a resolver's acme.storage names it
432+
path string // where the file lives on the host
433+
body string
434+
}
435+
436+
// judgedCertificate is one certificate from an ACME store with the verdict
437+
// that both status views report.
438+
type judgedCertificate struct {
439+
proxy.CertExpiry
440+
Store string // host path of the store the certificate was read from
441+
Days int // whole days until expiry
442+
Overdue bool // a configured resolver failed to renew it in time
443+
Orphaned bool // no configured resolver writes it to this store
444+
}
445+
446+
// judgeCertificates parses every store and judges each certificate by the
447+
// resolver that issued it. A certificate is renewed only when the static
448+
// configuration defines its resolver with this very store as its storage, so
449+
// only such a certificate can be overdue. A key that fails that test was left
450+
// behind by an older proxy configuration (a renamed resolver, wildcard routes
451+
// that were removed, a storage path that moved). Nothing renews it, and
452+
// judging its expiry would report a healthy host as diverged forever, so it is
453+
// reported as orphaned instead. Status does not advise deleting it: Traefik
454+
// reads its static configuration only at startup, so a proxy that was not
455+
// restarted after an apply may still renew under a key the file no longer
456+
// names.
457+
//
458+
// stores maps each resolver of the applied configuration to its store file.
459+
// A nil map means that configuration is unknown. Then no certificate is
460+
// called orphaned, since that would invite deleting one the proxy may still
461+
// renew, and every certificate is judged by its expiry, since hiding a real
462+
// overdue renewal is the worse mistake.
463+
//
464+
// Certificates are sorted, because the store's resolver keys parse from a map
465+
// and both views must print the same order on every run.
466+
func judgeCertificates(raw []acmeStore, stores map[string]string, now time.Time) ([]judgedCertificate, error) {
467+
var out []judgedCertificate
468+
for _, store := range raw {
469+
certs, err := proxy.CertExpiries([]byte(store.body))
338470
if err != nil {
339471
return nil, err
340472
}
341-
out = append(out, certs...)
473+
for _, cert := range certs {
474+
days := int(cert.NotAfter.Sub(now).Hours() / 24)
475+
// Without the configuration every certificate might be renewed,
476+
// so each is judged by its expiry, as before resolvers were known.
477+
// A resolver whose store CertificateStores could not place ("")
478+
// might write here, so it is given the benefit of the doubt.
479+
renewed := true
480+
if stores != nil {
481+
file, configured := resolverStore(stores, cert.Resolver)
482+
renewed = configured && (file == "" || file == store.file)
483+
}
484+
out = append(out, judgedCertificate{
485+
CertExpiry: cert,
486+
Store: store.path,
487+
Days: days,
488+
Overdue: renewed && days < renewalFloorDays,
489+
Orphaned: !renewed,
490+
})
491+
}
342492
}
493+
sort.Slice(out, func(i, j int) bool {
494+
a, b := out[i], out[j]
495+
if a.Domain != b.Domain {
496+
return a.Domain < b.Domain
497+
}
498+
if a.Resolver != b.Resolver {
499+
return a.Resolver < b.Resolver
500+
}
501+
if a.Store != b.Store {
502+
return a.Store < b.Store
503+
}
504+
return a.NotAfter.Before(b.NotAfter)
505+
})
343506
return out, nil
344507
}
345508

0 commit comments

Comments
 (0)