Repository navigation
engine: retry a volume mount while the device is not yet openable - #404
Merged
Merged
Conversation
The guest publishes /sys/block and the /dev node before the kernel can open a hot-plugged disk, so mount occasionally failed with "not a valid block device" (3 of 620 mounts on cocoon-test2). Retry only that error on the existing poll interval within the setup timeout; any other mount failure still returns at once.
A retry that ran into the setup deadline returned the last mount error, so the live volume API answered 500 instead of 504. Wrap ctx.Err() whenever the context has ended, both after a failed attempt and while waiting, keeping the last mount error in the message.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #400.
Cause (cocoon-test2, 620 claim-time mounts across cocoon v0.7.2–v0.7.4 and two rt:24.04 digests): after a hot attach the guest creates
/sys/block/vdXand the/dev/vdXnode before the kernel can open the disk.waitForVolumeDevicereturns on the node, andmountthen fails with/dev/vdc is not a valid block device(exit 32). All three failures came on settled VMs where the device appeared within 13–14 ms of the attach.Change:
mountDeviceretries the mount only on that error, on the existing 10 ms poll interval, until the volume setup timeout. Any other mount failure (wrong filesystem, busy mount point, permission) still returns on the first attempt, asTestMountVolumeStopsAtFailedStagechecks.Cost: none on the normal path (one mount exec as before); a not-ready device costs one extra exec per 10 ms until it opens.
Test:
TestMountVolumeRetriesUntilTheDeviceOpens(fake silkd returns the error twice, the third mount succeeds); it fails when the retry is removed.Gates: golangci-lint
0 issues.(sandboxd, linux and darwin), fmt clean,go test -race ./engine/ ./pool/. Not yet run on hardware: the race is about 1 in 200 mounts, so a hardware check needs a long claim loop; I can run one on cocoon-test2 before merge if you want.