Skip to content

fix: prevent stuck power lock when restarting an offline server (k8s)#2

Closed
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1780260993-fix-restart-while-off
Closed

fix: prevent stuck power lock when restarting an offline server (k8s)#2
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1780260993-fix-restart-while-off

Conversation

@devin-ai-integration

Copy link
Copy Markdown

Summary

Restarting (or stopping) an offline server on the Kubernetes environment left the panel stuck with another power action is currently being processed for this server. The power lock was held for ~10 minutes per attempt, so every subsequent action returned ErrLockerLocked.

Root cause: a stopped server's Pod still exists but in a non-running (Succeeded/Failed) phase — a completed Pod is not auto-removed. On restart, HandlePowerAction acquires the power lock and calls WaitForStop:

  • WaitForStopStop → (signal-stop eggs) → Terminate(SIGTERM), which sees the Pod isn't running and returns without deleting it.
  • WaitForStop then calls waitForPodDeletion, which blocks for the full duration (10 min) waiting for a deletion that never happens — holding the power lock the whole time.

Fix: short-circuit WaitForStop when the Pod is already gone or not running — the server is effectively stopped, so mark it offline and return immediately. A subsequent Start still recreates the Pod via OnBeforeStart, so no stale-Pod cleanup is lost.

func (e *Environment) WaitForStop(ctx, duration, terminate) error {
    if pod, err := e.getPod(ctx); err != nil {
        if isNotFound(err) { e.markOffline(); return nil }   // already gone
    } else if !isPodRunning(pod) {
        e.markOffline(); return nil                          // exists but stopped
    }
    // ... existing stop + waitForPodDeletion flow (only when actually running)
}

Also extracted the repeated stopping → offline transition into a markOffline() helper (reused in Terminate).

Tests

Added TestWaitForStop covering restart-while-offline. The "Pod exists but not running" case blocks for the full timeout without the fix and returns immediately with it. go build ./..., go vet ./environment/kubernetes/..., and go test ./... all pass.

Link to Devin session: https://app.devin.ai/sessions/c6e2874ef8924e68a929e491c2018270
Requested by: @Exonical

Co-Authored-By: Bryce Anglin <brycemanglin@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@Exonical Exonical closed this May 31, 2026
@devin-ai-integration
devin-ai-integration Bot deleted the devin/1780260993-fix-restart-while-off branch May 31, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant