fix: set deleteQueryIndexInEveryRun=false for chained_findings monitor#1726
Open
thecodingshrimp wants to merge 2 commits into
Open
Conversation
opensearch-project#1725) The chained_findings monitor DataSources was constructed with deleteQueryIndexInEveryRun=true, causing the backing query index (.opensearch-sap-*_chained_findings-000001) to be deleted and recreated on every monitor execution. The regular doc-level monitor (same file, ~line 800) correctly passes null/false for this flag. This compounds with an inverted condition in DocLevelMonitorQueries.kt (opensearch-project/alerting) to produce 23k+ MergeSchedulerConfig log lines/min on clusters with chained findings detectors enabled. Fix: pass false instead of true for deleteQueryIndexInEveryRun in the chained_findings monitor DataSources constructor. Fixes opensearch-project#1725 Signed-off-by: thecodingshrimp <leonard.stutzer@sap.com>
8db5701 to
fd7883d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
chained_findingsmonitor was constructed withdeleteQueryIndexInEveryRun=trueinTransportIndexDetectorAction.java(~line 906), causing its backing query index to be deleted and recreated on every monitor execution cycle.nullfor this flag.truetofalse, preventing the chained findings query index from being torn down and rebuilt on every run.Root cause
createDocLevelMonitorMatchAllRequestbuilds thechained_findingsmonitor'sDataSourceswithdeleteQueryIndexInEveryRun=true. This flag instructs the alerting plugin'sDocLevelMonitorQueriesto delete and recreate the backing query index (e.g.,.opensearch-sap-<detector>-detectors-queries-optimized-<uuid>_chained_findings-000001) on every execution, even though the query index for a chained findings monitor is stable across runs.This compounds with a related bug in opensearch-project/alerting where the condition in
DocLevelMonitorQueries.kt:500is inverted (!=should be==), making every run unconditionally enter the delete+recreate path. The combined effect is 23,000+MergeSchedulerConfiglog lines per minute on clusters running detectors with chained findings enabled (3 master nodes × ~6 log lines per delete+recreate event × execution frequency).Change
File:
src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.javaRelated
DocLevelMonitorQueries.kt:500(targetQueryIndex != monitor.dataSources.queryIndexshould be==) — both bugs must be fixed to fully resolve the log storm.Test plan
createDocLevelMonitorMatchAllRequestconfirmdeleteQueryIndexInEveryRunisfalseon the returned monitor'sDataSources_chained_findingsquery index is not deleted/recreated between runs