Skip to content

Show build, role and credential plumbing in a health report - #2123

Merged
ppXD merged 1 commit into
mainfrom
fix/show-build-and-credential-plumbing-health
Oct 18, 2026
Merged

ppXD merged 1 commit into
mainfrom
fix/show-build-and-credential-plumbing-health

Conversation

@ppXD

@ppXD ppXD commented Oct 18, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add GET /health, a JSON report beside the probes. Anyone reads the process's build (version plus commit sha), its Hangfire role (Api or Worker) and each check's name and status, so an API image and a worker image can be compared from outside. A signed-in global Admin also reads each check's description and data (git release, connection topology, repair backlog). On the internet-facing API those are reconnaissance, so they are not printed to an anonymous caller. A check that threw is reported by its exception type, never its message. /health/live and /health/ready keep their bodies and status codes; the new checks carry no "ready" tag and only ever report Degraded, so no orchestrator restarts or unroutes a pod over them. Nothing refuses to start: each finding is logged and reported.
  • refresh-lock: a startup self-test takes one pg_advisory_lock key on a session, tries it from a second one several times, and reads the answer of the holder's own unlock. A second session that gets the key, or an unlock told it never held the key, means a transaction-mode pooler, RDS Proxy or a read-write split: an Error at boot and Degraded in the report. A pass is reported as "no sign of multiplexing", not as proof. The compare-and-set write and the re-read keep refreshed tokens correct either way.
  • git: git --version is logged once in a boot posture line with the build and role. Below 2.31 (the release that reads GIT_CONFIG_COUNT) the report is Degraded, and a tokened clone, push, ls-remote or pasted-credential pack clone that git refused for want of a credential fails with a typed message naming the cause instead of "could not read Username". Workspace clones, pushes and ls-remote throw a WorkspaceException, so existing catch sites and failure codes are unchanged; the pack clone stays a PackImportException that carries it.
  • credential-repair-sweep: whether Hangfire holds the sweep's entry, when it is next due and how its last run ended, and how many disconnected credentials it is waiting to move. The backlog comes from the sweep's own candidate query, now shared with it. The scheduler's stored error (an exception chain that can name the storage host) is never printed; the finding says one was recorded and the text is logged once.

Test plan

  • Unit, full suite: 13953 passed, 1 pre-existing skip. New cases cover the version parser, the self-test against a fake server (refused contender with a missed unlock, bounded unlock retry, a throwing unlock, softened pass wording), the two-tier writer (admin, anonymous, signed-in non-admin, unreadable roles), the sweep's redaction and log-once behaviour, and the old-git error at the pack clone
  • Integration (real Postgres): a session pair refuses the second pg_advisory_lock; a shared session and an unlock that lands on another real session are each reported as not exclusive; the sweep entry is read back from real Hangfire storage, with a stored error never repeated
  • E2E (real host, real pipeline): /health shows the build and role for the Api and Worker roles; anonymous callers and signed-in non-admins get no descriptions or data; a global Admin does; a stored scheduler error holding an address and stack frames reaches neither; /health/live and /health/ready are byte-identical to before
  • Non-breaking: no migration, no env toggle, no config key; a rolling deploy or rollback is a code swap (an old pod answers /health with the existing 404 fallback) and git revert rolls it back
  • Staging: read /health as an Admin on one API pod and one worker pod and compare the build; confirm refresh-lock is Healthy behind the real Postgres path

@ppXD
ppXD force-pushed the fix/move-bindings-when-a-credential-is-disconnected branch from 18ae07b to 519bfee Compare October 18, 2026 11:22
An API pod and a worker pod run the same assembly from two images, and
nothing let an operator see which build each one was on, whether the
OAuth refresh lock was really exclusive on this deployment's Postgres
path, whether the host git could carry a clone token, or whether the
credential repair sweep was scheduled at all. Each of those failed
quietly and read as a bad token or a flaky run.

Add GET /health, a JSON report beside the probes. Anyone, signed in or
not, reads the process's build and Hangfire role and each check's name
and status, which is all it takes to compare an API image with a worker
image. A signed-in global Admin also reads each check's description and
data (git release, connection topology, repair backlog); on the
internet-facing API those are reconnaissance, not for an anonymous
caller. A check that threw is reported by the exception's type, never
its message. /health/live and /health/ready keep their bodies and
status codes; the new checks carry no "ready" tag and only ever report
Degraded, so no orchestrator restarts or unroutes a pod over them.
Nothing refuses to start: each finding is logged and reported instead.

- refresh-lock: a self-test takes one advisory-lock key on a session
  and tries it from a second, several times, then reads the answer of
  the holder's own unlock. A second session that gets the key, or an
  unlock told it never held the key, means a transaction-mode pooler,
  RDS Proxy or a read-write split: an Error at boot and Degraded in the
  report. A pass is reported as "no sign of multiplexing", because a
  pooler that picks a server session per statement can slip past it. An
  unlock that missed is retried a few times so the key is not left on
  the pooler's server session. The finding is reused for five minutes
  so the endpoint cannot make the database work per request.
- git: `git --version` is read once and logged in a boot posture line
  with the build and role. Below 2.31, the release that reads
  GIT_CONFIG_COUNT, the report is Degraded, and a tokened clone, push,
  ls-remote or pasted-credential pack clone that git refused for want
  of a credential now fails with a typed message naming the cause
  instead of "could not read Username". For workspace clones, pushes
  and ls-remote it is a WorkspaceException, so every existing catch site
  and failure code is unchanged; the pack clone stays a
  PackImportException carrying it.
- credential-repair-sweep: whether Hangfire holds the sweep's entry,
  when it is next due and how its last run ended, and how many
  disconnected credentials it is waiting to move. The backlog is
  counted by the sweep's own candidate query, now shared with it. The
  scheduler's stored error is an exception chain that can name the
  storage host, so the finding says only that one was recorded; the full
  text is logged once per distinct error.
@ppXD
ppXD changed the base branch from fix/move-bindings-when-a-credential-is-disconnected to main October 18, 2026 11:23
@ppXD
ppXD force-pushed the fix/show-build-and-credential-plumbing-health branch from 9412742 to 9c3a330 Compare October 18, 2026 11:23
@ppXD
ppXD merged commit 78a4571 into main Oct 18, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant