Skip to content

Self-heal Jenkins Hetzner agents that lost their retention strategy - #392

Merged
nogueiraanderson merged 3 commits into
mainfrom
pkg/hetzner-retention-selfheal
Aug 7, 2026
Merged

Self-heal Jenkins Hetzner agents that lost their retention strategy#392
nogueiraanderson merged 3 commits into
mainfrom
pkg/hetzner-retention-selfheal

Conversation

@nogueiraanderson

Copy link
Copy Markdown
Collaborator

Bug

  • Hetzner agents created from a deserialized cloud config carry no retention strategy, which core Jenkins treats as never-shut-down. The worker becomes immortal, absorbs all label demand for months, and fills its disk (the cloud.cd ENOSPC reported in #opensource-jenkins on 2026-08-07: 16/16 agents were 58 days old, ps80 13/13, pxc 2/2).

Fix

  • New init.groovy.d/hetznerRetentionSelfheal.groovy on the 8 EC2 masters with init delivery: cures any Hetzner agent whose effective retention degraded to Always, at boot and every 10 minutes via a PeriodicWork.
  • Backstop for the plugin-side fix (jenkins-hetzner-cloud-plugin PR 16, lands in v103.percona.29); the guard is a no-op when everything is healthy.

…ion strategy

- New init.groovy.d/hetznerRetentionSelfheal.groovy on all 8 EC2 masters with init delivery
- Cures agents whose effective retention degraded to Always (never reaped) at boot and every 10 minutes
- Backstop for the plugin-side deserialization fix in v103.percona.29; no-op when healthy
@nogueiraanderson
nogueiraanderson requested a review from vorsel August 7, 2026 08:00
…hour-wrap agents

- Live re-evaluation now swaps the PeriodicWork (marker toString) instead of keeping the first script's closure forever
- Hour-wrap agents are excluded from the cure, their policy is immune and idle retention would change its semantics
- Drop the system-property guard the timer swap replaces
…f-heal

- Cure only agents positively identified as idle-period; unreadable policies are skipped and logged, never rewritten
- Per-node try/catch so one throwing node cannot abort the pass, guarded boot call so registration always runs
- Late PeriodicWork registration verified live on pg.cd (probe fired within one period)
@nogueiraanderson
nogueiraanderson merged commit 0813bed into main Aug 7, 2026
8 checks passed
@nogueiraanderson
nogueiraanderson deleted the pkg/hetzner-retention-selfheal branch August 7, 2026 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants