Skip to content

feat(metrics): track rollback faults restored on the dashboard and a badge - #458

Merged
JacksonFergusonDev merged 1 commit into
mainfrom
feat/rollback-metrics
Oct 6, 2026
Merged

JacksonFergusonDev merged 1 commit into
mainfrom
feat/rollback-metrics

Conversation

@JacksonFergusonDev

@JacksonFergusonDev JacksonFergusonDev commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Summary & Motivation

Rollback is the claim a user can least check for themselves, and it already has an exhaustive nightly suite. This publishes what that suite proves: how many faults were injected and restored, as a README badge (rollback faults restored | N), a section on the metrics dashboard, and a count on jacksonferguson.me, the way the mutation score is handled. It follows #457, which renamed the dashboard to /metrics/.

Planning & Key Decisions

  • A count, not a percentage. Rollback is pass/fail, so a rate would read 100% forever. What grows is the fault space covered (templates, sites, platforms), so the history stores raw counts per operating system and template and the graph shows coverage growing. The total adds every OS, so one fault restored on three platforms counts three times.
  • Failure shows in text, never color. The README badges are all cyan and a URL color overrides the endpoint's, so a failing night makes the badge read N failing. rollback-latest.json updates on any complete run; rollback-history.json appends only fully green runs.
  • Only complete runs publish. Each rollback job uploads a report, even when it fails. The publisher checks every artifact the matrix plans, rejects empty or foreign reports, and writes nothing otherwise, so an unknown result never reads as a pass. A killed or interrupted pytest writes no report.
  • A separate workflow. rollback-metrics.yml runs on workflow_run of scheduled Nightlies, as Nightly Report does, so a publishing failure cannot fail the run a release requires. Manual Nightlies, which may narrow the matrix, never publish.
  • Counting at the source. Fault tests carry a rollback_fault marker and --rollback-report has a small plugin write per-scenario counts. It merges the retry pass, so a fault that failed and then passed counts once, as passed. The matrix entries now name their artifacts.
  • Rejected: a ratio badge; a status-colored badge (breaks the cyan row); publishing from inside Nightly (couples release gating to publishing); counting from junit XML (loses earlier results on the retry run).

Rollout

Nothing publishes until the first scheduled Nightly after merge; the dashboard section and badge show "no complete runs yet" until then. The site already has the dispatch token from the mutation workflow. A follow-up in JacksonFergusonDev.github.io adds the count to the Protostar card.

…badge

Nightly's rollback jobs now count the faults they inject, and a new Rollback
Metrics workflow records the total on gh-pages beside the mutation score, for a
dashboard section, a README badge, and jacksonferguson.me.
@JacksonFergusonDev
JacksonFergusonDev merged commit 737f52e into main Oct 6, 2026
23 checks passed
@JacksonFergusonDev
JacksonFergusonDev deleted the feat/rollback-metrics branch October 6, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant