Skip to content

fix: prevent stuck power lock when restarting an offline server (k8s)#3

Merged
Exonical merged 1 commit into
mainfrom
fix/k8s-restart-while-off-power-lock
Jun 2, 2026
Merged

fix: prevent stuck power lock when restarting an offline server (k8s)#3
Exonical merged 1 commit into
mainfrom
fix/k8s-restart-while-off-power-lock

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented May 31, 2026

Copy link
Copy Markdown

Summary

Restarting (or stopping) an offline server on the Kubernetes environment left the panel stuck with another power action is currently being processed for this server. The power lock was held for ~10 minutes per attempt, so every subsequent action returned ErrLockerLocked.

Root cause: a stopped server's Pod still exists but in a non-running (Succeeded/Failed) phase — a completed Pod is not auto-removed. On restart, HandlePowerAction acquires the power lock and calls WaitForStop:

  • WaitForStopStop → (signal-stop eggs) → Terminate(SIGTERM), which sees the Pod isn't running and returns without deleting it.
  • WaitForStop then calls waitForPodDeletion, which blocks for the full duration (10 min) waiting for a deletion that never happens — holding the power lock the whole time.

Fix: short-circuit WaitForStop when the Pod is already gone or not running — the server is effectively stopped, so mark it offline and return immediately. A subsequent Start still recreates the Pod via OnBeforeStart, so no stale-Pod cleanup is lost.

func (e *Environment) WaitForStop(ctx, duration, terminate) error {
    if pod, err := e.getPod(ctx); err != nil {
        if isNotFound(err) { e.markOffline(); return nil }   // already gone
    } else if !isPodRunning(pod) {
        e.markOffline(); return nil                          // exists but stopped
    }
    // ... existing stop + waitForPodDeletion flow (only when actually running)
}

Also extracted the repeated stopping → offline transition into a markOffline() helper (reused in Terminate).

Tests

Added TestWaitForStop covering restart-while-offline. The "Pod exists but not running" case blocks for the full timeout without the fix and returns immediately with it. go build ./..., go vet ./environment/kubernetes/..., and go test ./... all pass.

Co-Authored-By: Bryce Anglin <brycemanglin@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 3 additional findings.

Open in Devin Review

@Exonical
Exonical merged commit ac8985b into main Jun 2, 2026
16 checks passed
@Exonical
Exonical deleted the fix/k8s-restart-while-off-power-lock branch June 2, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant