Skip to content

fix: set deleteQueryIndexInEveryRun=false for chained_findings monitor#1726

Open
thecodingshrimp wants to merge 2 commits into
opensearch-project:mainfrom
thecodingshrimp:fix/chained-findings-monitor-delete-query-index
Open

fix: set deleteQueryIndexInEveryRun=false for chained_findings monitor#1726
thecodingshrimp wants to merge 2 commits into
opensearch-project:mainfrom
thecodingshrimp:fix/chained-findings-monitor-delete-query-index

Conversation

@thecodingshrimp
Copy link
Copy Markdown

Summary

  • The chained_findings monitor was constructed with deleteQueryIndexInEveryRun=true in TransportIndexDetectorAction.java (~line 906), causing its backing query index to be deleted and recreated on every monitor execution cycle.
  • The regular doc-level monitor (same file, ~line 800) correctly passes null for this flag.
  • This one-character fix changes true to false, preventing the chained findings query index from being torn down and rebuilt on every run.

Root cause

createDocLevelMonitorMatchAllRequest builds the chained_findings monitor's DataSources with deleteQueryIndexInEveryRun=true. This flag instructs the alerting plugin's DocLevelMonitorQueries to delete and recreate the backing query index (e.g., .opensearch-sap-<detector>-detectors-queries-optimized-<uuid>_chained_findings-000001) on every execution, even though the query index for a chained findings monitor is stable across runs.

This compounds with a related bug in opensearch-project/alerting where the condition in DocLevelMonitorQueries.kt:500 is inverted (!= should be ==), making every run unconditionally enter the delete+recreate path. The combined effect is 23,000+ MergeSchedulerConfig log lines per minute on clusters running detectors with chained findings enabled (3 master nodes × ~6 log lines per delete+recreate event × execution frequency).

Change

File: src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

-        true), enableDetectorWithDedicatedQueryIndices, true, PLUGIN_OWNER_FIELD, null, null);
+        true), enableDetectorWithDedicatedQueryIndices, false, PLUGIN_OWNER_FIELD, null, null);

Related

Test plan

  • Unit tests for createDocLevelMonitorMatchAllRequest confirm deleteQueryIndexInEveryRun is false on the returned monitor's DataSources
  • Integration test: create a detector with chained findings enabled, execute it multiple times, verify the _chained_findings query index is not deleted/recreated between runs
  • Verify regular doc-level monitor (non-chained) behavior is unchanged

opensearch-project#1725)

The chained_findings monitor DataSources was constructed with
deleteQueryIndexInEveryRun=true, causing the backing query index
(.opensearch-sap-*_chained_findings-000001) to be deleted and
recreated on every monitor execution. The regular doc-level monitor
(same file, ~line 800) correctly passes null/false for this flag.

This compounds with an inverted condition in DocLevelMonitorQueries.kt
(opensearch-project/alerting) to produce 23k+ MergeSchedulerConfig
log lines/min on clusters with chained findings detectors enabled.

Fix: pass false instead of true for deleteQueryIndexInEveryRun in the
chained_findings monitor DataSources constructor.

Fixes opensearch-project#1725

Signed-off-by: thecodingshrimp <leonard.stutzer@sap.com>
@thecodingshrimp thecodingshrimp force-pushed the fix/chained-findings-monitor-delete-query-index branch from 8db5701 to fd7883d Compare May 21, 2026 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] TransportIndexDetectorAction: chained_findings monitor created with deleteQueryIndexInEveryRun=true causing index churn

1 participant