Skip to content

inhibitor merge cache and index so there's only one lock - #5559

Draft
Spaceman1701 wants to merge 1 commit into
prometheus:mainfrom
Spaceman1701:more-inhibitor-refactor
Draft

inhibitor merge cache and index so there's only one lock#5559
Spaceman1701 wants to merge 1 commit into
prometheus:mainfrom
Spaceman1701:more-inhibitor-refactor

Conversation

@Spaceman1701

Copy link
Copy Markdown
Contributor

This change replaces the store.Alerts with a new cache type in the inhibitor. The cache is essentially a merged store.Alerts and inhibitor index that work under a single lock. This allows operations to modify the cache and index atomically, solving a bunch of possible race conditions in the Inhibitor.

I believe that this resolves the last problems caught by the new tests added in #5449.

[BUGFIX]: Fix a race condition where the inhibitor could get into a bad state and fail to inhibit some alerts in some circumstances

@SoloJacobs

SoloJacobs commented Sep 14, 2026

Copy link
Copy Markdown
Contributor
❯ ./run.sh
=== inhibit
flushes: 238231
Mutes calls: 297539
alerts muted: 201516 (67.73%)
flushes fully muted: 178227 (74.81%)
--- PASS: TestReplay (4.79s)
ok  	github.com/prometheus/alertmanager/inhibit	4.799s
=== patchedinhibit
flushes: 238231
Mutes calls: 297539
alerts muted: 201516 (67.73%)
flushes fully muted: 178227 (74.81%)
--- PASS: TestReplay (4.93s)
ok  	github.com/prometheus/alertmanager/patchedinhibit	4.940s
=== v29inhibit
flushes: 238231
Mutes calls: 297539
alerts muted: 201516 (67.73%)
flushes fully muted: 178227 (74.81%)
--- PASS: TestReplay (10.24s)
ok  	github.com/prometheus/alertmanager/v29inhibit	10.254s
=== 5449updatedinhibit
flushes: 238231
Mutes calls: 297539
alerts muted: 201516 (67.73%)
flushes fully muted: 178227 (74.81%)
--- PASS: TestReplay (5.55s)
ok  	github.com/prometheus/alertmanager/5449updatedinhibit	5.556s
=== 5542updatedinhibit
flushes: 238231
Mutes calls: 297539
alerts muted: 201516 (67.73%)
flushes fully muted: 178227 (74.81%)
--- PASS: TestReplay (5.24s)
ok  	github.com/prometheus/alertmanager/5542updatedinhibit	5.250s
=== 5559updatedinhibit
flushes: 238231
Mutes calls: 297539
alerts muted: 201516 (67.73%)
flushes fully muted: 178227 (74.81%)
--- PASS: TestReplay (4.52s)
ok  	github.com/prometheus/alertmanager/5559updatedinhibit	4.527s

(inhibit test was rebased on main, so that is why this test no longer finds any missing mutes).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants