Claude LAD draft

hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/_index.md

@@ -10,8 +10,11 @@ After setting up pipelines and scaling Workers:
 - You can track the status of your pipelines and components with health graphs and out-of-the-box monitors. See [Monitoring Pipelines][2] for more information.
 - You can also create your monitors, dashboards, and notebooks to monitor your pipelines. See [Pipeline Usage Metrics][3] for a list of metrics.
 - If you run into issues with Observability Pipelines, see [Troubleshooting][4].
+- Observability Pipelines groups your logs into [patterns][5] and uses [Watchdog to detect log anomalies][6] before it sends logs to any destination.
 
 [1]: /observability_pipelines/monitoring_and_troubleshooting/worker_cli_commands/
 [2]: /observability_pipelines/monitoring_and_troubleshooting/monitoring_pipelines/
 [3]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/
-[4]: /observability_pipelines/monitoring_and_troubleshooting/troubleshooting/
+[4]: /observability_pipelines/monitoring_and_troubleshooting/troubleshooting/
+[5]: /observability_pipelines/monitoring_and_troubleshooting/log_patterns/
+[6]: /observability_pipelines/monitoring_and_troubleshooting/log_anomaly_detection/

hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/log_anomaly_detection.md

@@ -0,0 +1,39 @@
+---
+title: Log Anomaly Detection in Observability Pipelines
+description: Learn how Observability Pipelines uses Watchdog to detect log anomalies before logs reach any destination.
+disable_toc: false
+further_reading:
+- link: "/logs/explorer/watchdog_insights/"
+  tag: "Documentation"
+  text: "Watchdog Insights for Logs"
+- link: "/observability_pipelines/monitoring_and_troubleshooting/log_patterns/"
+  tag: "Documentation"
+  text: "Log Patterns in Observability Pipelines"
+- link: "/observability_pipelines/monitoring_and_troubleshooting/monitoring_pipelines/"
+  tag: "Documentation"
+  text: "Monitoring Pipelines"
+---
+
+## Overview
+
+Observability Pipelines uses Watchdog to find log anomalies before it sends any data to a destination. Watchdog looks at [log patterns][1] for unusual or new spikes in warning and error logs.
+
+This detection happens on-stream. You get anomaly alerts even if you route your logs to a different SIEM or log management tool, such as Splunk or Microsoft Sentinel.
+
+For example, a security team routes production logs to Splunk. This team can use Watchdog Insights in Observability Pipelines to find an issue before the logs reach Splunk.
+
+## How anomaly detection works
+
+Observability Pipelines groups your logs into [patterns][1] and aggregates them by the `env`, `service`, `source`, and `status` attributes. Watchdog scans these aggregated patterns for anomalies, such as:
+
+- A new type of warning or error log.
+- A spike in the rate of warning or error logs.
+
+## View anomalies
+
+Observability Pipelines shows Watchdog insights in the following places:
+
+- The Observability Pipelines landing page, which shows anomalies detected across all of your pipelines.
+- Each pipeline's page, which shows anomalies detected in that pipeline's logs.
+
+[1]: /observability_pipelines/monitoring_and_troubleshooting/log_patterns/

hugo/content/en/observability_pipelines/monitoring_and_troubleshooting/log_patterns.md

@@ -0,0 +1,46 @@
+---
+title: Log Patterns
+description: Learn how Observability Pipelines groups similar logs into patterns before the logs reach any destination.
+disable_toc: false
+further_reading:
+- link: "/logs/explorer/analytics/patterns/"
+  tag: "Documentation"
+  text: "Log Patterns in Log Explorer"
+- link: "/observability_pipelines/monitoring_and_troubleshooting/log_anomaly_detection/"
+  tag: "Documentation"
+  text: "Log Anomaly Detection in Observability Pipelines"
+- link: "/observability_pipelines/monitoring_and_troubleshooting/monitoring_pipelines/"
+  tag: "Documentation"
+  text: "Monitoring Pipelines"
+---
+
+## Overview
+
+Observability Pipelines groups your logs into patterns as they stream through a pipeline. A pattern is a group of logs with a similar structure. Observability Pipelines finds these patterns before it sends any data to a destination, such as Splunk or Microsoft Sentinel.
+
+Use patterns to find noisy or repeated log messages. You can also use patterns to spot new or unusual log messages as soon as they appear in the pipeline. This works no matter which destinations you route your logs to.
+
+## How patterns are grouped
+
+Observability Pipelines groups logs by the `message` field. It groups the results further by the `status` and `service` fields.
+
+For patterns to appear, each log must have the following attributes at the root level:
+
+- `message`
+- `service`
+- `status`
+- `env`
+- `source`
+
+If a log doesn't have these attributes, Observability Pipelines doesn't include it in a pattern.
+
+**Note**: Place the processors that add or correct these attributes earlier in your pipeline. Observability Pipelines groups logs into patterns after all other processors run.
+
+## View patterns
+
+Observability Pipelines shows patterns for each pipeline. For each pipeline, you can:
+
+- View the patterns found in that pipeline's logs.
+- View a sample of the logs that make up a pattern.
+
+[1]: /logs/explorer/analytics/patterns/