Skip to content

feat(healthcheck): add runtime /health endpoint with DB/Discord readiness and heartbeat probes#59

Open
Copilot wants to merge 21 commits into
mainfrom
copilot/add-healthcheck-endpoint
Open

feat(healthcheck): add runtime /health endpoint with DB/Discord readiness and heartbeat probes#59
Copilot wants to merge 21 commits into
mainfrom
copilot/add-healthcheck-endpoint

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 14, 2026

This adds an HTTP healthcheck surface for deployments (Coolify) so the bot can be monitored externally for runtime degradation, not just process liveness. The endpoint supports GET /health and reports unhealthy when core dependencies are degraded.

  • Healthcheck server and response contract

    • Introduced a lightweight async HTTP server (src/runtime_healthcheck.py) with configurable host/port.
    • Handles:
      • GET /health → health payload (200 or 503)
      • non-matching path → 404
      • non-GET methods → 405
    • Health payload includes per-check booleans:
      • DB connectivity/responsiveness
      • Discord connection status
      • Discord global rate-limit status (via public is_ws_ratelimited())
      • Discord heartbeat health (shard latency-based stall detection)
    • Tightened heartbeat latency threshold to 10s so blocked/stalled heartbeats degrade health sooner.
  • Bot lifecycle integration

    • Wired healthcheck startup/shutdown into bot runtime flow.
    • Healthcheck startup is handled by a dedicated cog (src/cogs/healthcheck.py) via @discord.Cog.listener(once=True) on connect.
    • Shutdown in src/__main__.py stops the server through the healthcheck cog while preserving exception-group behavior.
    • Healthcheck can be disabled by leaving HEALTHCHECK_HOST unset or setting it to an empty string.
  • Configuration and deployment defaults

    • Added env config in src/config.py:
      • HEALTHCHECK_HOST (no app-side default; optional)
      • HEALTHCHECK_PORT (no app-side default; required only when HEALTHCHECK_HOST is configured)
    • Added explicit runtime validation so invalid/missing HEALTHCHECK_PORT fails fast only when healthcheck is enabled.
    • Updated .env.example and README healthcheck docs to reflect fixed /health path and disable behavior.
    • Exposed container port 8080 in Dockerfile for platform probing.
    • Added Docker HEALTHCHECK in Dockerfile using wget against http://${HEALTHCHECK_HOST}:${HEALTHCHECK_PORT}/health.
    • Added Docker ENV defaults for HEALTHCHECK_HOST and HEALTHCHECK_PORT so container health probes work out of the box while remaining overridable.
  • Documentation

    • Added project-standard docstrings throughout the runtime healthcheck module (module, class, and methods) for maintainability and consistency.
  • Typing cleanup

    • Added explicit attribute type annotations at first assignment in touched non-final classes (HealthcheckServer and HealthcheckCog).
checks = {
    "database_connected": db_connected,
    "discord_connected": discord_connected,
    "discord_no_global_ratelimit": discord_unrate_limited,
    "discord_heartbeats_healthy": discord_heartbeat_ok,
}
healthy = all(checks.values())
return (
    200 if healthy else 503,
    {"status": "ok" if healthy else "degraded", "checks": checks},
)

[!WARNING]

Copilot AI linked an issue Apr 14, 2026 that may be closed by this pull request
Copilot AI and others added 2 commits April 14, 2026 11:57
Copilot AI changed the title [WIP] Add healthcheck endpoint for Coolify bot Add runtime healthcheck endpoint for Coolify with DB/Discord readiness probes Apr 14, 2026
Copilot AI requested a review from ToothyDev April 14, 2026 12:01
@ToothyDev
Copy link
Copy Markdown
Collaborator

@copilot astral.sh is now excluded from the firewall, if you need to make further changes based on that

Copy link
Copy Markdown
Collaborator

@ToothyDev ToothyDev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, if possible, check for any shard heartbeats being blocked in order to help catch stalls of any kind (e.g. by blocking code)

Comment thread src/__main__.py Outdated
Comment thread src/runtime_healthcheck.py
Comment thread src/healthcheck.py Outdated
Copilot AI and others added 2 commits April 14, 2026 12:25
Agent-Logs-Url: https://github.com/Versa-Bots/versa/sessions/9e556274-2b93-48f7-9e72-478e586eae74

Co-authored-by: ToothyDev <55001472+ToothyDev@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 14, 2026

Additionally, if possible, check for any shard heartbeats being blocked in order to help catch stalls of any kind (e.g. by blocking code)

Added a heartbeat health probe to the healthcheck response (discord_heartbeats_healthy) so stalled shard heartbeats can flip health to degraded. Included in commit 0823976.

Copilot AI changed the title Add runtime healthcheck endpoint for Coolify with DB/Discord readiness probes Add runtime healthcheck endpoint for Coolify with DB/Discord readiness and heartbeat probes Apr 14, 2026
Copilot AI requested a review from ToothyDev April 14, 2026 12:32
@ToothyDev
Copy link
Copy Markdown
Collaborator

@copilot Change pull request title according to semantic PR titles

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 14, 2026

@copilot Change pull request title according to semantic PR titles

Proposed semantic title: feat(healthcheck): add runtime endpoint with DB/Discord readiness and heartbeat probes. I can’t directly edit the PR title from this environment, but this is the title to apply.

Comment thread src/healthcheck.py Outdated
@ToothyDev ToothyDev changed the title Add runtime healthcheck endpoint for Coolify with DB/Discord readiness and heartbeat probes feat(healthcheck): add runtime endpoint with DB/Discord readiness and heartbeat probes Apr 14, 2026
Agent-Logs-Url: https://github.com/Versa-Bots/versa/sessions/03a42855-7f38-4d7c-a520-0218b63e29f2

Co-authored-by: ToothyDev <55001472+ToothyDev@users.noreply.github.com>
Copilot AI and others added 2 commits April 18, 2026 20:23
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 18, 2026

  • Define the healthcheck in the Dockerfile
  • Use a separate cog for the healthcheck, start the server with discord.Cog.listen(once=true) async def on_connect
  • Type attributes on non-final classes on ...

Applied the proposed updates in 9943694 (plus 0324a43): Dockerfile now defines a container HEALTHCHECK, healthcheck server startup moved to a dedicated cog using @discord.Cog.listener(once=True) on connect, and typed attributes were added at first assignment in touched classes.

  • Define the healthcheck in the Dockerfile

    • Use a separate cog for the healthcheck, start the server with discord.Cog.listen(once=true) async def on_connect
      ...

Done — proposed changes are applied in 9943694 (and 0324a43) with the healthcheck cog listener startup, Dockerfile healthcheck definition, and typed attributes in touched classes.

Copilot AI requested a review from ToothyDev April 18, 2026 20:26
@ToothyDev
Copy link
Copy Markdown
Collaborator

@copilot Change dockerfile HEALTHCHECK to use curl or wget, whatever is more appropriate in your view (or makes this simpler)

Agent-Logs-Url: https://github.com/Versa-Bots/versa/sessions/6aaeb26f-7b63-47ec-b505-7c4b43280c81

Co-authored-by: ToothyDev <55001472+ToothyDev@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 18, 2026

@copilot Change dockerfile HEALTHCHECK to use curl or wget, whatever is more appropriate in your view (or makes this simpler)

Updated in 671bea8: Dockerfile HEALTHCHECK now uses wget via shell command and keeps the existing HEALTHCHECK_* env-based URL behavior.

Comment thread src/config.py Outdated
DB_PATH = Path(os.getenv("DB_PATH") or Path("data/database.db")).absolute()
DB_PATH.parent.mkdir(parents=True, exist_ok=True)

HEALTHCHECK_HOST = os.getenv("HEALTHCHECK_HOST", "127.0.0.1")
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the default is used for these, they will be empty/unset in the docker container and as such it will fail. THey need to be added as ENV in the dockerfile and can be optionally overridden later.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 372b9e5: added Dockerfile ENV HEALTHCHECK_HOST=127.0.0.1 and ENV HEALTHCHECK_PORT=8080 so container defaults are set and can still be overridden.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The defaults should also be removed from there then. And empty hostname could mean the healthcheck is disabled.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot apply this change

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in e9781b5: removed app-side defaults from src/config.py for HEALTHCHECK_HOST/HEALTHCHECK_PORT, and made an unset/empty HEALTHCHECK_HOST disable the healthcheck server in the cog. Screenshot: no UI changes (backend/config-only change).

Comment thread src/config.py Outdated
Comment thread src/healthcheck.py Outdated
@ToothyDev
Copy link
Copy Markdown
Collaborator

@copilot Apply the requested changes

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 18, 2026

@copilot Apply the requested changes

Applied in 372b9e5: Docker defaults were added as ENV, health endpoint is now fixed at /health, and the runtime healthcheck module was renamed for clarity.

Copilot AI changed the title feat(healthcheck): add runtime endpoint with DB/Discord readiness and heartbeat probes feat(healthcheck): add runtime /health endpoint with DB/Discord readiness and heartbeat probes Apr 18, 2026
@ToothyDev ToothyDev requested a review from Paillat-dev April 18, 2026 21:53
Copilot AI and others added 7 commits April 19, 2026 10:41
Agent-Logs-Url: https://github.com/Versa-Bots/versa/sessions/56e0aa53-8e42-43cb-80a0-de126250306f

Co-authored-by: ToothyDev <55001472+ToothyDev@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Set up Healthcheck for Coolify

3 participants