Skip to content

feat: proactive stack health reconciliation and periodic Docker resource pruning#47

Draft
veerendra2 wants to merge 3 commits into
mainfrom
31-reconcile-stacks-proactively
Draft

feat: proactive stack health reconciliation and periodic Docker resource pruning#47
veerendra2 wants to merge 3 commits into
mainfrom
31-reconcile-stacks-proactively

Conversation

@veerendra2

Copy link
Copy Markdown
Owner

Closes #31

Summary

  • Health reconciliation loop (HEALTH_RECONCILE_INTERVAL, default 5m): periodically checks all managed stacks and redeploys any where Status != Running or any container is exited/dead. Skips stacks with restarting containers (Docker restart policy is doing its job).
  • Periodic Docker resource prune (PRUNE_INTERVAL, default 24h): prunes unused images, volumes, and build cache — but only when all managed stacks are healthy. If any stack is degraded or suspended, the cycle is skipped.
  • Suspend label (composeflux.suspend=true): set via docker update --label-add on any container to pause reconciliation for the entire stack. Both the health loop and prune loop respect it. Useful during maintenance (e.g., stopping a DB for backups without triggering a reconcile).
  • Moved inline dClient.Prune() out of Prune() into the dedicated scheduled loop so pruning is always gated on health.

How to Test

Health reconciliation:

# Start composeflux with a short interval for faster testing
HEALTH_RECONCILE_INTERVAL=30s docker compose up -d

# Stop a container inside a managed stack
docker stop <container>

# Within 30s, verify it comes back up
docker ps | grep <container>

Suspend label:

# Set suspend label on a container
docker update --label-add composeflux.suspend=true <container>

# Stop another container in the same stack
docker stop <another-container-in-same-stack>

# Confirm the health loop does NOT restart it (check logs)
docker logs composeflux | grep -i suspend

Prune loop:

# Start with a short prune interval
PRUNE_INTERVAL=2m PRUNE_RESOURCES=true docker compose up -d

# With all stacks healthy — after 2m check logs for prune firing
docker logs composeflux | grep -i prune

# Stop a managed stack, wait for next tick — confirm prune is skipped
docker compose -p <stack> stop
docker logs composeflux | grep "Skipping Docker resource prune"

🤖 Generated with Claude Code

…rce pruning

- Add health reconciliation loop (HEALTH_RECONCILE_INTERVAL, default 5m) that
  detects and redeploys stacks with Status != Running or any exited/dead container
- Add periodic Docker resource prune loop (PRUNE_INTERVAL, default 24h) that only
  fires when all managed stacks are healthy
- Add composeflux.suspend=true label support: any stack with this label on any
  container is skipped by both loops; prune is aborted entirely if any stack is suspended
- Move inline dClient.Prune() out of Prune() into the dedicated scheduled loop
  so pruning is always gated on stack health
- Update docs: new env vars, new Introduction sections, remove outdated limitation

Closes #31

Co-Authored-By: Claude <noreply@anthropic.com>
@veerendra2 veerendra2 added the build Builds dev docker image label Jul 11, 2026
@veerendra2 veerendra2 linked an issue Jul 11, 2026 that may be closed by this pull request
veerendra2 and others added 2 commits July 11, 2026 21:47
…control

PRUNE_INTERVAL unset/empty disables pruning. PRUNE_RESOURCES is removed.

Co-Authored-By: Claude <noreply@anthropic.com>
- Fix stack existence check before calling Ps() in health and prune loops
  (prevents errors on bootstrap when stacks not yet deployed)
- Fix empty container list handling in isStackHealthy
  (stack with zero containers is unhealthy)
- Add deprecation warning for removed PRUNE_RESOURCES env var
- Fix docs: PRUNE_INTERVAL requires 0 to disable, not unset

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Builds dev docker image

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reconcile stacks proactively

1 participant