feat: alert when the nightly or its triage post is missing

[esraagamal6]

In plain words

The nightly run posts to #camunda-hub-nightly-test-results every night, and so does the triage that follows it. If the Slack step fails, the run turns red. But if a run never starts (a disabled schedule, a GitHub outage) or hangs, nothing posts and nothing is red, so a missing post goes unnoticed until someone notices the quiet channel. This PR adds a morning check that looks for exactly that and posts one alert.

What to look at

• .github/workflows/hub-nightly-post-check.yml (new): runs daily at 04:30 UTC, after the nightly (02:00) and its triage have finished. Same structure as the gap digest: Vault Slack token, a script, and the fail-if-slack-failed step at the end. A manual run is a dry run by default.
• scripts/triage/hub-nightly-post-check.ts (new): reads today's runs (UTC) of nightly-camunda-hub.yml and triage-camunda-hub-nightly.yml and decides what is wrong. It reports:
  • the scheduled nightly did not start today, is still running, or ended as anything but success or failure. A manual nightly does not stand in for it;
  • the triage for that nightly did not run, is still running, or ended as anything but success. A manual triage, or the triage of a manual nightly, does not count.
• .github/workflows/triage-camunda-hub-nightly.yml: one line, run-name: Triage of nightly <id>. It puts the nightly's run id in the triage run's title, so the check can tell which nightly each triage run belongs to (the list API does not return the triggering run). Nothing else in that workflow changes.
• The alert goes to #camunda-hub-nightly-test-results, pings hub-medic, links the run and the nightly cookbook. It posts nothing when both ran.
• tests/triage/hub-nightly-post-check.test.ts (new) and two lines added to tests/request-validation/slack-failure-marks-run-failed.test.ts, so the new workflow gets the same Slack-failure and dry-run checks as the others.
• AGENTS.md: a short paragraph describing the check.

What changed

A new scheduled check, plus the run title on the triage workflow. No other change to the nightly or the triage.

Judgement calls:

• A nightly whose tests failed is healthy: a red nightly still posts. For the triage only success is healthy, because the digest is its post: skipped (its job has an if), neutral, failure and cancelled all mean no digest.
• It does not read Slack, so it cannot tell that a post failed to arrive when the run succeeded. It catches runs that did not happen, did not finish or did not do their job. A failed Slack step is already a red run.

Details

• I ran tests/triage and tests/request-validation with CONFIG=camunda-hub (the default config's pinned-spec check fails in my environment): 678 tests pass. Biome is clean and tsc on the tests passes.
• I could not run the workflow itself here. A first manual run (dry run by default) will show what it would post. I looked at the real history of both workflows: every night for the last 30 days has one schedule run of the nightly and one workflow_run run of the triage, both completed.
• Triage runs started before this merges have the old title. If it merges between 02:00 and 04:30 UTC, that morning's check reports a missing triage. Merge before 02:00 or after 04:30 UTC to avoid it.
• The cookbook link has no heading anchor yet; the "When a post is missing" heading comes with docs: one-page how-it-works guide for the Hub team, and trim the Hub guides #711.
• The guides in docs: one-page how-it-works guide for the Hub team, and trim the Hub guides #711 and the handover page still say that nobody is alerted when the nightly post is missing. They need updating when this merges.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G3526HsTZMmihMi94p34Pr

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

🟡 Changes recommended

Same-day unrelated triage runs and skipped conclusions can conceal a missing digest, and the alert currently links to an unavailable anchor.

3 open findings

• Treat skipped and neutral triage conclusions as non-healthy · New
• Require triage to run after the selected nightly · New
• Link to an existing cookbook heading · New

What changed in this PR

Adds a morning safeguard that alerts when Hub nightly or triage posts are missing.

Changes:

• Adds scheduled workflow-run monitoring and Slack alerting.
• Adds unit and workflow-policy coverage.
• Documents the new operational check.

File	Description
.github/​workflows/​hub-nightly-post-check.yml	Schedules and runs the check.
scripts/​triage/​hub-nightly-post-check.ts	Evaluates runs and posts alerts.
tests/​triage/​hub-nightly-post-check.test.ts	Tests evaluation and alert behavior.
tests/​request-validation/​slack-failure-marks-run-failed.test.ts	Covers Slack-failure enforcement.
AGENTS.md	Documents the operational workflow.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Ad-hoc nightly runs can incorrectly replace the scheduled run and cause false or missed alerts.

1 open finding

• Filter nightly candidates to scheduled runs only · New

3 resolved since last review

• Require triage to run after the selected nightly
• Treat skipped and neutral triage conclusions as non-healthy
• Link to an existing cookbook heading

Previously missed (1)

In code that hasn't changed since last review

Extend workflow-wiring invariant to cover the dry-run token guard

.github/​workflows/​hub-nightly-post-check.yml:47

The new manual dry-run token guard is omitted from the existing workflow-wiring invariant in tests/request-validation/slack-failure-marks-run-failed.test.ts:151-160; that test currently covers only the gap digest and coverage report. Add hub-nightly-post-check to that it.each list so removing this guard cannot silently make a dry run depend on Vault.

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

Triage runs are not correlated to their triggering nightly, allowing ad-hoc runs to hide or falsely report failures.

1 open finding

• Correlate triage runs with the triggering nightly run ID · New

1 resolved since last review

• Filter nightly candidates to scheduled runs only

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

The run lookup must paginate to avoid false missing-triage alerts when more than 20 runs exist.

1 open finding

• Paginate workflow runs to avoid false missing-triage alerts · New

1 resolved since last review

• Correlate triage runs with the triggering nightly run ID

🧠 Review effort: Balanced

[Copilot]

🟢 Approval recommended

The implementation matches the stated monitoring behavior and includes comprehensive regression coverage.

0 open findings

1 resolved since last review

• Paginate workflow runs to avoid false missing-triage alerts

🧠 Review effort: Balanced