Repository navigation
Watch Docker for crashed, unhealthy and restarting app containers - #10
Merged
Merged
Conversation
app:container-died, app:container-unhealthy, app:container-restarting and app:container-healthy carry app:name and docker:container:name; died also carries the new docker:container:exit-code. The core's alerts listen for them by name.
Pure over the events and the Begin registry: a non-zero die nobody asked for and no operation caused is app:container-died, health_status: unhealthy is app:container-unhealthy, more than three starts of a container within ten minutes is app:container-restarting once an episode, and after one of those ten minutes running and not unhealthy is app:container-healthy. A die or a start within a minute of a kill is the stop or restart somebody asked for.
WatchDocker follows the container events of the compose projects with the Docker client the service already uses, asks the Begin registry whether an operation holds the app, and publishes what the decisions say through PublishEventWrapper, never waiting on the bus. A broken stream is followed again after a pause that doubles up to a minute; what was running is then known again only once it starts. One-off containers are left out. main starts it next to the other background loops.
A container forgotten after ten quiet minutes, one already running when AppManagement starts, and every container after a break in the event stream was taken for stopped: a health status never said otherwise, so an unhealthy container that recovered never published app:container-healthy and the core never resolved the alert. A healthy status now says the container runs, and starts its good run if it was not known to run. Each time the event stream is followed, the watch lists the apps' running containers and resumes from them: running or not, unhealthy or not, running well since now at best. This replaces marking every container stopped after a break. Tests: forgotten then unhealthy then healthy, unhealthy as the watch starts, a health status after an unseen start, a crash that stops the container, and the four ways a container can come out of a break.
The core resolves app:<name>:runtime on any app:container-healthy, but the watch published it for each container: an app whose database recovered while its web container was still unhealthy was told it runs normally again, and Docker reports the remaining trouble only once. tick now groups the troubled containers by app and publishes one app:container-healthy for an app only when every one of them has run, not unhealthy, for ten minutes; it then ends the episode of all of them. The event names the first of those containers by name. Test: a two-container app, one crashed and restarted, the other unhealthy until later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows the Docker event stream and publishes app:container-died (non-zero exit outside any operation), app:container-unhealthy, app:container-restarting (more than 3 starts in 10 min) and app:container-healthy (every container of the app fine again for 10 min), for the core's push alerts. Spec: ReCasaOS/CasaOS docs/superpowers/specs/2026-09-25-push-alerts-design.md.