Skip to content

OSAC-1675: Harden haproxy concurrency safety - #22

Open
eliorerz wants to merge 1 commit into
osac-project:mainfrom
eliorerz:OSAC-1675-haproxy-concurrency-safety
Open

OSAC-1675: Harden haproxy concurrency safety#22
eliorerz wants to merge 1 commit into
osac-project:mainfrom
eliorerz:OSAC-1675-haproxy-concurrency-safety

Conversation

@eliorerz

Copy link
Copy Markdown
Contributor

Summary

  • Resilient reload: Add _reload_haproxy() that recovers from stopped haproxy by falling back to systemctl restart when reload fails or service is inactive. Prevents cascading boot failures when haproxy is down.
  • Config validation: Add _write_haproxy_cfg() that validates config with haproxy -c before applying, creates a backup, and uses atomic write-then-rename to prevent corruption on SIGKILL.
  • Robust rollback: Wrap each boot rollback cleanup step in try/except so one failure (e.g. haproxy down) doesn't abort remaining cleanup (VMs, networks, overlays, DNS).
  • Orphan cleanup: Add cleanup-haproxy subcommand that finds orphaned haproxy entries (clone IDs with no matching VM) and removes them along with stale DNS entries.

Risk matrix addressed

# Risk Fix
R1 haproxy down causes cascading failures + broken rollback _reload_haproxy() + resilient rollback
R2 No config validation before reload haproxy -c -f check in _write_haproxy_cfg()
R3 Orphaned entries from killed jobs cleanup-haproxy subcommand
R4 Non-atomic config writes Write to temp file, validate, then mv

Test plan

  • All 129 tests pass (6 new test classes added)
  • Stop haproxy manually, run cluster-tool boot, confirm it auto-recovers
  • Inject a bad config line, confirm validation catches it and restores backup
  • Leave an orphaned clone, run cleanup-haproxy, confirm entries removed

- Add _reload_haproxy() that recovers from stopped haproxy by falling
  back to systemctl restart when reload fails or service is inactive
- Add _write_haproxy_cfg() that validates config with haproxy -c before
  applying, creates backup, and uses atomic write-then-rename
- Make boot rollback resilient: wrap each cleanup step in try/except so
  one failure (e.g. haproxy down) doesn't abort remaining cleanup
- Add cleanup-haproxy subcommand that finds orphaned haproxy entries
  (clone IDs with no matching VM) and removes them
- Add tests for all new functionality (129 tests pass)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant