Skip to content

fix: drop stale schedule metrics after rename - #212

Merged
StekPerepolnen merged 16 commits into
ydb-platform:mainfrom
StekPerepolnen:fix-renamed-schedule-metrics
Nov 12, 2026
Merged

StekPerepolnen merged 16 commits into
ydb-platform:mainfrom
StekPerepolnen:fix-renamed-schedule-metrics

Conversation

@StekPerepolnen

@StekPerepolnen StekPerepolnen commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Renaming a backup schedule leaves metric series with the old schedule_name, which can trigger a false stale-backup alert. Detect renames in the schedule watcher and remove the old gauge series after reporting current metrics in each replica.

Adds a regression test for renaming and preserving backup metrics without a schedule_name label.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Renaming a schedule deletes backup metrics that have no schedule-name label, leaving valid metrics missing if the subsequent backup lookup fails.

1 open issue
What changed in this PR

This PR addresses stale metrics left behind when a backup schedule is renamed, which can cause false stale-backup alerts.

Changes:

  • Detects schedule renames and cleans up old metric series before reporting current metrics.
  • Extends cleanup to include schedule run counters.
  • Adds a regression test for renamed labels and preservation of backup metrics without schedule_name.
File Description
internal/​watchers/​schedule_watcher/​schedule_watcher.go Detects renames and moves cleanup before metric reporting.
internal/​watchers/​schedule_watcher/​schedule_watcher_test.go Tests rename cleanup and backup metric preservation.
internal/​metrics/​metrics.go Removes schedule run counters during cleanup.

Review effort: Balanced

@StekPerepolnen
StekPerepolnen requested a balanced review from Copilot November 9, 2026 19:16
@StekPerepolnen
StekPerepolnen marked this pull request as ready for review November 9, 2026 19:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The metrics mock omits cleanup of schedules_failed_count and schedules_succeeded_count, causing tests to retain stale counters after deletions and accumulate them across renames, unlike production.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Mock cleanup leaves schedule counters behind

internal/​metrics/​metrics_mock.go:190

The mock does not delete schedules_failed_count or schedules_succeeded_count, although the production implementation now deletes both during name cleanup. Consequently, tests using InitializeMockMetricsRegistry retain run counters after deletion and accumulate them across renames, unlike production. Delete both counter entries here as well so the mock reflects the new cleanup behavior.

1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Rename cleanup in the label-less mock deletes newly written schedule gauges, leaving timestamp and RPO values absent until the next cycle and diverging from the real registry’s behavior.

1 open issue

Review effort: Balanced

Comment thread internal/metrics/metrics_mock.go Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the schedule-rename metric cleanup.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@StekPerepolnen
StekPerepolnen merged commit 6a8239d into ydb-platform:main Nov 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants