diff --git a/README.md b/README.md index d5c6e6f..7c966cf 100644 --- a/README.md +++ b/README.md @@ -1,94 +1,67 @@ -# Log10x Benchmarks - -Reproducible benchmarks behind the [Log10x](https://www.log10x.com/?utm_source=github&utm_medium=readme&utm_campaign=benchmarks&utm_content=hero) -engineering blog. Each folder is a self-contained benchmark: the exact commands, the tool -versions, the configs, and the committed reference results, so a reader or an assistant can -rerun it and check every figure a post cites. - -## Principles - -- **Public data.** Every log set is a public dataset or a public release asset, fetched by the - benchmark's own setup steps. Nothing depends on private data, and no benchmark vendors a - large file into git. -- **Exact, not approximate.** Tool versions are pinned, configs are committed, and the driver - is the source of truth. If a number is in a post, the code here produces it. -- **Committed reference results.** Each benchmark commits the artifacts its own scripts wrote, - so a rerun can be diffed against what the post reported. -- **Independently verifiable.** Where practical a benchmark ships a second script that - recomputes its headline numbers from primary files rather than from a summary JSON. -- **Falsifiers are named before the run.** Where a benchmark states a threshold that would - refute a Log10x claim, the script checks it on every run and prints the verdict, including - the runs where the claim lost. - -## Benchmarks - -| Folder | Question it answers | Status | +# log10x benchmarks + +This repository contains public, rerunnable measurements of what [log10x](https://www.log10x.com/?utm_source=github&utm_medium=readme&utm_campaign=benchmarks&utm_content=hero) does and does not do to log volume and cost on Splunk, Datadog, Elasticsearch, ClickHouse and CloudWatch. Most tests that ran on a platform read their figure from the platform itself, such as Splunk's license meter, Datadog's usage metric or ClickHouse's system tables. The two accounting benchmarks price the 10x engine's counts at list price and need no account. + +Each directory has a short README with the result, a RECORD.md with the full method and every number, and the raw result files. For nine of the benchmarks, CI recomputes the committed figures from the kept raw counts on every pull request and every push to main, and fails if the two disagree. Several results favor a native feature or another tool, and those rows say so. + +## Splunk + +| Benchmark | What it measures | Result | +|---|---|---| +| [splunk-license](splunk-license/) | License meter bytes, original versus compact events | 63.91% fewer metered bytes, and every compact event read back as its original | + +## Datadog + +| Benchmark | What it measures | Result | +|---|---|---| +| [datadog-ceiling](datadog-ceiling/) | Standard indexing under a monthly budget as volume grows | Standard stayed at the budget as volume grew 2.9 times, and every line reached Datadog | +| [datadog-containment](datadog-containment/) | Stopping a log surge before it reaches Datadog, with or without a rule | The default limit for services without a rule sent 5,555 Postgres surge lines to S3, none outside the failure | +| [datadog-flex-accounting](datadog-flex-accounting/) | List-price cost of moving events from Standard to Flex | At demo volume, saves $3.40 to $6.04 a month against 30-day indexing and costs more against 15-day | +| [datadog-service-control](datadog-service-control/) | Per-pattern plan versus logger-name filters inside one service | Eight logger-name filters moved 24,440 of 29,043 events, the plan 23,854 | + +## Elasticsearch + +| Benchmark | What it measures | Result | +|---|---|---| +| [elastic-hot-tier-budget](elastic-hot-tier-budget/) | Hot-tier size under a budget during a 2.5x step | Hot stayed at 0.94 to 1.04 times the budget with moved lines on warm, and peaked at 1.43 times with them on hot | +| [elastic-hot-tier](elastic-hot-tier/) | How long moved lines stay on hot | Moved lines left hot 30 to about 110 minutes after writing, and bytes sent to hot still grew with load | +| [elastic-store-four-ways](elastic-store-four-ways/) | Index store of compact events versus native logsdb options | Logsdb with `pattern_text` was smallest, and compact on logsdb took 27.2% to 53.7% more | + +## ClickHouse + +| Benchmark | What it measures | Result | +|---|---|---| +| [clickhouse-clickstack](clickhouse-clickstack/) | Disk saved by compact events in the ClickStack schema | Compact events took 14.9% off, but the SQL decoder misread 78% of rows. Original text with a type column in the sort key took 13.1% off | +| [clickhouse-inflate](clickhouse-inflate/) | Query-time cost of decoding compact events | The fixed SQL decoded 137,418 rows in 548 ms, and the buggy SQL hit the memory cap | +| [clickhouse-cloud-elasticity](clickhouse-cloud-elasticity/) | ClickHouse Cloud compute bill when 65.3% of rows are written | Billed compute fell to 0.749 of the full feed's from a 4-unit start, and stayed the same from a 3-unit start | +| [clickstack-e2e](clickstack-e2e/) | Over-cap rows in an object store, queried together with hot rows | 13,010 rows hot and 24,526 in the store counted back to all 37,536 | +| [clickstack-e2e-gaps](clickstack-e2e-gaps/) | Counts, pruning and compute after moving rows to S3 | Counts matched, time-only queries opened all 180 objects, and a collector rule used less compute | +| [clickstack-rows-gate](clickstack-rows-gate/) | Which ClickHouse system table counts written rows exactly | `system.part_log` matched 22,367 rows exactly, and `system.query_log` did not | +| [clickstack-rows](clickstack-rows/) | Rows written at a set rate as load grew 2.5x | 0.91 to 1.13 of the set rate, with one window over the 1.10 bound, so the run failed as registered | + +## CloudWatch + +| Benchmark | What it measures | Result | |---|---|---| -| [`pattern-identity/`](pattern-identity/) | Does a log line keep the same name over time, and does it get the same name in two places? A four-line replay inside one Drain3 process, two Drain3 instances fed disjoint halves of one stream and two engine 1.1.142 instances fed the same halves, the template string separated from the integer Drain returns beside it, and engine-side coverage probed with invented vocabulary and two LogHub log sets. | Current. Behind *A log pattern name learned from traffic changes as more lines arrive* (draft). | -| [`otel-denominators/`](otel-denominators/) | A reduction figure is quoted against a denominator. What happens to it when the denominator changes? One Kubernetes capture measured three ways: as captured, with our own injected field and the collector's debug output removed, and on the log message text alone with the envelope discarded. | Current. Behind the OTel figures in [*Log compaction, measured*](https://www.log10x.com/blog/log-compaction-measured/). | -| [`clickhouse-inflate/`](clickhouse-inflate/) | What does it cost to expand compacted logs at query time in ClickHouse, and what did an earlier Log10x benchmark actually measure? | Current. Not behind a post. | -| [`clickhouse-clickstack/`](clickhouse-clickstack/) | What does the compact form cost or save on disk in ClickHouse's own recommended log schema, the ClickStack `otel_logs` table, rather than in a single `String` column? Three arms so the row-count change from multi-line folding never hides inside a ratio, three codecs, and a check on whether the compact rows read back as the text they came from. | Current. Not behind a post. | -| [`clickstack-e2e/`](clickstack-e2e/) | Does the whole ClickHouse route work end to end, in containers? The released capture through a collector into the 10x receiver, the rows the receiver marked `offload` written to an object store and the rest inserted by ClickStack's own exporter, both sides read back as one table, the query shapes timed with their object-store requests, and a reconciliation of what left the receiver against what is stored. | Current. Behind [*Reading ClickHouse logs from the table and the bucket in one query*](https://www.log10x.com/blog/clickhouse-offload-merge-table/). | -| [`clickstack-e2e-gaps/`](clickstack-e2e-gaps/) | The six questions the end to end run left open, each over the whole capture: what a day predicate prunes across thirty days of objects, whether Vector can write the cold branch, whether the Retriever indexes and queries that layout, what a container killed mid-stream costs line by line, which dashboard answers change when rows leave the hot table, and how the per-type policy compares with a collector-only policy on insert and merge CPU. Plus a second pass on four of them: Vector 0.58.0 writing Parquet beside JSON with the Parquet reader's own counters, a `GROUP BY` and an `ORDER BY ... LIMIT` through the Merge table, the collector configured as its code owners prescribe, the `TTL TO VOLUME 'cold'` alternative costed on the same capture, and ClickHouse issue 116888 demonstrated with its guard. | Current. Not behind a post. | -| [`clickstack-rows-gate/`](clickstack-rows-gate/) | Which ClickHouse system table counts the rows written into the logs table to the row, with the Receiver regulating in lines with a service total: `system.part_log` NewPart rows equals the hot count in both runs, `system.query_log` written_rows drifts with the exporter's async-insert entries; the totals held to the row, a policy rewrite mid-window reset nothing, and the events unit takes effect only from the config or an engine option. | Current. The meter for the ClickHouse rows run. Not behind a post. | -| [`clickstack-rows/`](clickstack-rows/) | The ClickHouse rows run, metered: rows written into ClickStack held at a rate an hour while load grew 2.5x with a storm and a new service, read on ClickHouse's own `system.part_log`. FAIL as registered: the rate held at 0.99 to 1.04 of R in every complete 10-minute window but one, the floor has no allowance in the 10-minute test, one service's 1,422 new statements in a window passed their always-pass sample, and 33 rows were inserted twice. Rerun with the regulator's sample bound (config `baselineMaxOverTotal` 50): every 4-minute window under its bound, 10-minute windows 0.97 to 1.07 of R/6, every returned record stored once; fails as registered on the 10-minute sum alone. | Three runs. The first failed as registered (one 10-minute window at 1.36, 33 duplicates). With the sample bound, the rate held at 0.97 to 1.07 of R in every complete 10-minute window, none over, no duplicates. Beside a collector rule under the same 2.5x step, the hold read 0.91 to 1.13 (one window over the 1.10 bound, the storm's first) against the rule's 2.95 to 3.42; insert-plus-merge CPU did not separate the arms on this rig. | -| [`clickhouse-cloud-elasticity/`](clickhouse-cloud-elasticity/) | How far does a ClickHouse Cloud service's compute fall, in units and in billed dollars, when the rows written into its logs table fall, above the smallest size, and how long does the autoscaler take? One service in AWS us-east-1, the ClickStack `otel_logs` schema under a fixed query load, two arms that differ only in the rows written (all of the replay, or the 65.3% the Receiver kept), four pairs, read on the replicas' own system tables and on `usageCost`. | Current. Not behind a post. From a 4-unit start the cut took the service from 4 units to 3, 0.75 in units and 0.749 in billed compute, in each of three pairs, 2.9 to 4.3 h after the cut; from a 3-unit start it read no change in 10.25 h. | -| [`splunk-license/`](splunk-license/) | Splunk bills on what its licence meter counts, not on a file size. What does the 10x Receiver in front of Splunk do to that number? The same capture through the same Universal Forwarder into the same instance, twice, raw and compact, measured from `license_usage.log` and from the Monitoring Console's own searches, with every compact event read back through the 10x Splunk app and compared to the text that went in. | Current. Not behind a post. | -| [`datadog-flex-accounting/`](datadog-flex-accounting/) | Datadog bills Standard indexing per million events. When the 10x Receiver moves events from Standard to Flex Logs under the plan the 10x MCP writes, how many events and GB move, and what do they cost at Datadog's list prices? The released engine and MCP on one capture at three targets, counted two independent ways, priced per million events moved, with the custom metrics the Datadog metric output creates. Computed from the engine's counts at list price; no Datadog account needed. | Current. Not behind a post. | -| [`datadog-service-control/`](datadog-service-control/) | Inside one service a Datadog service filter cannot split the volume, so the rules a person writes are filters on the service's logger names. On the kafka container of two public captures of the same code, what does the per-pattern plan the 10x MCP writes move against the fewest `@logger.name` filters that move as many events, and what does each do on the second capture with nothing re-tuned? Computed from the engine's counts; no Datadog account needed. | Current. Not behind a post. | -| [`datadog-containment/`](datadog-containment/) | When a service floods its logs, does the released Receiver, installed with the values the MCP writes for Datadog, hold the flood before Datadog, including in a service with no rule, and what does it hold that it should not? The OpenTelemetry demo's own seeded failures on engine 1.1.142, and the Receiver's memory and CPU on a replay of the run's capture. | Current | -| [`datadog-ceiling/`](datadog-ceiling/) | With a monthly budget X for Datadog Standard indexing, does the released Receiver, driven by the released MCP's hourly budget run, hold Standard at X while log volume grows 2 to 3x, as Datadog's own usage metric counts it? The OpenTelemetry demo into a real Datadog org on engine 1.1.146 and log10x-mcp 1.30.71, with a new service, an error storm and one service held to its own number. | Current. Not behind a post. | -| [`elastic-hot-tier/`](elastic-hot-tier/) | With the Receiver's per-statement byte budget and log10x-mcp's Elasticsearch frozen-tier recipe, how long do the lines it moves stay on the hot tier, can they still be found by pattern hash on frozen, and how much of the volume does a fixed budget keep off hot as load grows, as the cluster's own APIs count it? Self-managed Elasticsearch 9.5.5 with hot and frozen nodes, engine 1.1.147, log10x-mcp 1.30.77, the published OpenTelemetry demo sample replayed for 11 hours. | Current. Not behind a post. | -| [`elastic-store-four-ways/`](elastic-store-four-ways/) | On Elasticsearch 9.5.5, how much index store does the engine's compact form save over a raw index, over logsdb, and over logsdb with `message` as `pattern_text` sorted on its template id, on the same sample? Primary store after a force merge to one segment, `best_compression` everywhere, trial licence, engine 1.1.150, the published OpenTelemetry demo sample, the compact form in standard, logsdb and template-sorted logsdb mode. Logsdb with `pattern_text` holds the sample in 34.7% of the raw index's store, the compact form on logsdb in 53.3% and sorted alike in 44.1%: on Elasticsearch 9.x the compact form is not the smaller store. | Current. Not behind a post. | -| [`elastic-hot-tier-budget/`](elastic-hot-tier-budget/) | Does log10x-mcp's hourly hot-tier run, driving the Receiver, hold an Elasticsearch hot tier at a size S while load grows 2.5 times, as the hot node's own stats count it? Self-managed Elasticsearch 9.5.5 with hot and frozen nodes and the published OpenTelemetry demo sample replayed, in four runs, the last two beside a native reroute by service. With the tier-down index on hot the hold fails its bound of 1.10 S, peaking at 1.17 and 1.23 S in the two single-arm runs and at 1.43 S against the reroute's 1.36 S; with it on warm the hold keeps hot at 0.94 to 1.04 S from 20 minutes after the step, 0 of 25 readings over, while the reroute reaches 1.25 S and is still rising. | Current. Not behind a post. | -| [`cloudwatch-ia-accounting/`](cloudwatch-ia-accounting/) | CloudWatch bills ingestion per GB, and an Infrequent Access log group ingests at half the Standard rate. When the 10x Receiver moves events to Infrequent Access or offloads them to S3 under the plan the 10x MCP writes, how many GB move, and what do they cost at AWS's list prices for us-east-1? Computed from the engine's counts at list price; no AWS account needed. | Current. Not behind a post. | -| [`intent/`](intent/) | Does each pattern hold one logging statement? Every event of two OpenTelemetry-demo captures and a LogHub ZooKeeper capture is labelled with the source statement that wrote it, from the programs' own source, and the 10x engine (1.1.142, the current release, on all three; 1.1.132 and 1.1.133 on the demo captures; 1.1.136 on ZooKeeper) and Drain3 are scored against those labels: events merged with another statement, events split from their statement's main pattern, and exact grouping. | Current. Linked from [*How a Drain pattern group changes as more lines arrive*](https://www.log10x.com/blog/drain-pattern-groups-change/). | -| [`naming-stability/`](naming-stability/) | When the engine moves to its next release, how many events change their pattern name, pattern hash or template hash? Engine 1.1.140, 1.1.141 and 1.1.142, each with the configuration and library its image ships, on the two OpenTelemetry-demo captures of `intent/`, compared event by event. | Current | -| [`loghub-compaction/`](loghub-compaction/) | How much smaller is the engine's compact form than the original logs, on the 14 full public LogHub datasets, with the engine and library as they ship? The compact form is the encoded events plus the template dictionary, measured before any codec. | Current | -| [`drain3-vs-log10x/`](drain3-vs-log10x/) | Log10x against [Drain3](https://github.com/logpai/Drain3) on the [LogHub](https://github.com/logpai/loghub) 2k sets: does a line get the same pattern ID regardless of file and order, and does the reduced form reverse to the original bytes? | Retired as a benchmark. The post it was written for never shipped. Its `tenx-encode.config.yaml` and `tenx-decode.config.yaml` are still cited elsewhere as the published round-trip configs, so the folder stays. | - -For anything about pattern identity, read `pattern-identity/`, not `drain3-vs-log10x/`. The two -ask overlapping questions and only the first is maintained. - -## Layout - -There is no single template. Each folder carries its own README with the exact run steps, and -each resolves its paths relative to itself, so a benchmark can be run from anywhere. - -| Folder | Entry point | Pinned deps | Where results land | External data | -|---|---|---|---|---| -| `pattern-identity/` | `./run.sh` | `requirements.txt` (drain3 0.9.11, Python 3.13) and a pinned engine image digest | `results/` | Release assets from `log-10x/config` tag `otel-sample-v1`, plus two LogHub sets from Zenodo record 3227177, fetched into `data/` on first run | -| `otel-denominators/` | `./run.sh` | pinned engine image digest, plus the round-trip configs in `drain3-vs-log10x/` | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1`, fetched into `data/` on first run | -| `clickhouse-inflate/` | `./run.sh` | pinned ClickHouse Docker image | `results.json` | none, the sample is committed | -| `clickhouse-clickstack/` | `./run.sh` | pinned engine image digest and a ClickHouse Docker image | `results/` | the `otel-sample-v2` release asset from `log-10x/config`, fetched into `data/` on first run | -| `clickstack-e2e/` | `./run.sh` | pinned ClickStack, engine, collector and MinIO images | `results/` | the `otel-sample-v2` release asset from `log-10x/config`, fetched into `data/` on first run | -| `clickstack-e2e-gaps/` | `./gap_*.sh` | the same images by digest, plus Vector by digest; the receiver is the published `edge-10x` image | `results/` | the same `otel-sample-v2` asset, read from `../clickstack-e2e/data` | -| `clickhouse-cloud-elasticity/` | `cloud/arms.py`, after the split, stage and meters in its README | the engine and collector images by digest (`results/images.txt`), Python 3.12+, a `clickhouse` client binary | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1`, checked by sha256; a ClickHouse Cloud organization | -| `splunk-license/` | `./run.sh` | pinned engine, Splunk and forwarder images, and the 10x Splunk app | `results/` | the `otel-sample-v2` release asset from `log-10x/config`, fetched into `data/` on first run | -| `datadog-flex-accounting/` | `./run.sh` | pinned engine image digest, log10x-mcp from npm at a pinned version, and the shared scripts in `accounting-lib/` | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1`, checked by sha256 | -| `datadog-ceiling/` | `rig/run.sh` phases, `rig/budget-phase.sh` | the engine image by digest, log10x-mcp 1.30.71 from npm, the demo at tag 3.1.0 | `results/` | a Datadog org with Flex Logs; the demo's own logs | -| `datadog-service-control/` | `./run.sh` | the same as `datadog-flex-accounting/` | `results/` | the same asset, and `rec1155.log.gz` from `log-10x/benchmarks` tag `intent-data-v1`, both checked by sha256 | -| `datadog-containment/` | `rig/go.sh`; `rig/memory/run.sh` for the memory replay | engine and Fluent Bit images by digest, log10x-mcp from npm at a pinned version, OpenTelemetry demo 3.1.0 | `results/` | the demo's images, pulled by tag | -| `elastic-hot-tier/` | `rig/make-replay.py`, then `rig/phases.sh` | Elasticsearch 9.5.5 and Fluent Bit 4.2 images by tag, the engine image by digest, a log10x-mcp checkout (the recipe and Receiver values are rendered from it) | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1` | -| `elastic-store-four-ways/` | `./run.sh` | the engine and Elasticsearch 9.5.5 images by digest | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1`, checked by sha256 | -| `cloudwatch-ia-accounting/` | `./run.sh` | the same as `datadog-flex-accounting/` | `results/` | the same asset | -| `intent/` | `./run.sh` | engine image digests (1.1.132, 1.1.133, 1.1.136, 1.1.142) and drain3 0.9.11 from `pattern-identity/requirements.txt` | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1` and `rec1155.log.gz` from this repository's `intent-data-v1` release, fetched into `cache/` on first run | -| `naming-stability/` | `./run.sh` | engine image digests (1.1.140, 1.1.141, 1.1.142) | `results/` | the same two captures as `intent/`, checked by sha256 | -| `drain3-vs-log10x/` | see its README | `requirements.txt` | `bench/facts.json`, `bench/results.json` | LogHub 2k sets, fetched into `loghub/` | - -Large inputs are never committed. Each folder's `.gitignore` excludes whatever that benchmark -downloads or generates at run time. - -## Continuous integration - -`.github/workflows/ci.yml` syntax-checks every driver and runs the two mechanism checks that -need no download: `drain3-vs-log10x/smoke_test.py` and `pattern-identity/bench/replay.py`. The -full runs need Docker and several hundred megabytes of public data, so they are run by hand and -their artifacts are committed. `otel-denominators/report.py` is checked the same way: CI -re-renders the committed CSV and fails if the rendered table does not match the committed -`results.md`, so an edited number in the table without a rerun is caught. `splunk-license` -carries the same guard against its own primary file: CI recomputes the licence figures from -the `RolloverSummary` lines the run kept verbatim and fails if `results.json` disagrees. `intent` re-renders `results/results.md` from the committed score files and fails if the tables disagree. `naming-stability` does the same from its committed `compare.json`. +| [cloudwatch-ia-accounting](cloudwatch-ia-accounting/) | Bytes and cost of moving events to Infrequent Access | The plan for a 30% target moved 69.7% of bytes to Infrequent Access | + +## Log patterns, compaction and grouping + +| Benchmark | What it measures | Result | +|---|---|---| +| [pattern-identity](pattern-identity/) | Same pattern name in two places, versus Drain3 | The 10x engine named 299,997 of 300,000 probe lines the same, Drain3 0.6022 of shapes seen 2 to 10 times | +| [drain3-vs-log10x](drain3-vs-log10x/) | Pattern ID stability and byte-exact rebuild versus Drain3 | The 10x engine rebuilt 99.98% of lines and Drain3 66.48%, while Drain3 found fewer templates, closer to the hand labels | +| [intent](intent/) | How often one pattern mixes two logging statements | The 10x engine merged 0.01% to 0.21% of events, fewer than Drain3, which split fewer on two of three captures | +| [naming-stability](naming-stability/) | Pattern names across three engine releases | 0 of 582 and 0 of 2,687 patterns renamed, on two demo captures | +| [loghub-compaction](loghub-compaction/) | Compact form size on 14 full LogHub datasets | 44.7% fewer bytes over all 14, from 58.4% on Hadoop to 4.7% on HealthApp | +| [otel-denominators](otel-denominators/) | How the input's content changes a reduction figure | 64.00% as captured, 62.46% without the debug lines and added tag, 72.35% on application messages alone | +| [e103-grouping](e103-grouping/) | Multi-line events from many containers under one Fluent Bit tag | The 1.1.150 image mixed containers in 1,533 of 5,836 records, a build with the fix in none | + +## How to rerun + +Each directory's README gives the command that reruns the benchmark or recomputes its figures from the committed files, and links the full steps, pinned images and data sources. Most full runs need Docker and a public data download, and two need a platform account (a Datadog org with Flex Logs, a ClickHouse Cloud organization). + +[RECORD.md](RECORD.md) has the principles the benchmarks follow, entry points and pinned dependencies for most of them, and what CI checks. ## License diff --git a/RECORD.md b/RECORD.md new file mode 100644 index 0000000..d5c6e6f --- /dev/null +++ b/RECORD.md @@ -0,0 +1,95 @@ +# Log10x Benchmarks + +Reproducible benchmarks behind the [Log10x](https://www.log10x.com/?utm_source=github&utm_medium=readme&utm_campaign=benchmarks&utm_content=hero) +engineering blog. Each folder is a self-contained benchmark: the exact commands, the tool +versions, the configs, and the committed reference results, so a reader or an assistant can +rerun it and check every figure a post cites. + +## Principles + +- **Public data.** Every log set is a public dataset or a public release asset, fetched by the + benchmark's own setup steps. Nothing depends on private data, and no benchmark vendors a + large file into git. +- **Exact, not approximate.** Tool versions are pinned, configs are committed, and the driver + is the source of truth. If a number is in a post, the code here produces it. +- **Committed reference results.** Each benchmark commits the artifacts its own scripts wrote, + so a rerun can be diffed against what the post reported. +- **Independently verifiable.** Where practical a benchmark ships a second script that + recomputes its headline numbers from primary files rather than from a summary JSON. +- **Falsifiers are named before the run.** Where a benchmark states a threshold that would + refute a Log10x claim, the script checks it on every run and prints the verdict, including + the runs where the claim lost. + +## Benchmarks + +| Folder | Question it answers | Status | +|---|---|---| +| [`pattern-identity/`](pattern-identity/) | Does a log line keep the same name over time, and does it get the same name in two places? A four-line replay inside one Drain3 process, two Drain3 instances fed disjoint halves of one stream and two engine 1.1.142 instances fed the same halves, the template string separated from the integer Drain returns beside it, and engine-side coverage probed with invented vocabulary and two LogHub log sets. | Current. Behind *A log pattern name learned from traffic changes as more lines arrive* (draft). | +| [`otel-denominators/`](otel-denominators/) | A reduction figure is quoted against a denominator. What happens to it when the denominator changes? One Kubernetes capture measured three ways: as captured, with our own injected field and the collector's debug output removed, and on the log message text alone with the envelope discarded. | Current. Behind the OTel figures in [*Log compaction, measured*](https://www.log10x.com/blog/log-compaction-measured/). | +| [`clickhouse-inflate/`](clickhouse-inflate/) | What does it cost to expand compacted logs at query time in ClickHouse, and what did an earlier Log10x benchmark actually measure? | Current. Not behind a post. | +| [`clickhouse-clickstack/`](clickhouse-clickstack/) | What does the compact form cost or save on disk in ClickHouse's own recommended log schema, the ClickStack `otel_logs` table, rather than in a single `String` column? Three arms so the row-count change from multi-line folding never hides inside a ratio, three codecs, and a check on whether the compact rows read back as the text they came from. | Current. Not behind a post. | +| [`clickstack-e2e/`](clickstack-e2e/) | Does the whole ClickHouse route work end to end, in containers? The released capture through a collector into the 10x receiver, the rows the receiver marked `offload` written to an object store and the rest inserted by ClickStack's own exporter, both sides read back as one table, the query shapes timed with their object-store requests, and a reconciliation of what left the receiver against what is stored. | Current. Behind [*Reading ClickHouse logs from the table and the bucket in one query*](https://www.log10x.com/blog/clickhouse-offload-merge-table/). | +| [`clickstack-e2e-gaps/`](clickstack-e2e-gaps/) | The six questions the end to end run left open, each over the whole capture: what a day predicate prunes across thirty days of objects, whether Vector can write the cold branch, whether the Retriever indexes and queries that layout, what a container killed mid-stream costs line by line, which dashboard answers change when rows leave the hot table, and how the per-type policy compares with a collector-only policy on insert and merge CPU. Plus a second pass on four of them: Vector 0.58.0 writing Parquet beside JSON with the Parquet reader's own counters, a `GROUP BY` and an `ORDER BY ... LIMIT` through the Merge table, the collector configured as its code owners prescribe, the `TTL TO VOLUME 'cold'` alternative costed on the same capture, and ClickHouse issue 116888 demonstrated with its guard. | Current. Not behind a post. | +| [`clickstack-rows-gate/`](clickstack-rows-gate/) | Which ClickHouse system table counts the rows written into the logs table to the row, with the Receiver regulating in lines with a service total: `system.part_log` NewPart rows equals the hot count in both runs, `system.query_log` written_rows drifts with the exporter's async-insert entries; the totals held to the row, a policy rewrite mid-window reset nothing, and the events unit takes effect only from the config or an engine option. | Current. The meter for the ClickHouse rows run. Not behind a post. | +| [`clickstack-rows/`](clickstack-rows/) | The ClickHouse rows run, metered: rows written into ClickStack held at a rate an hour while load grew 2.5x with a storm and a new service, read on ClickHouse's own `system.part_log`. FAIL as registered: the rate held at 0.99 to 1.04 of R in every complete 10-minute window but one, the floor has no allowance in the 10-minute test, one service's 1,422 new statements in a window passed their always-pass sample, and 33 rows were inserted twice. Rerun with the regulator's sample bound (config `baselineMaxOverTotal` 50): every 4-minute window under its bound, 10-minute windows 0.97 to 1.07 of R/6, every returned record stored once; fails as registered on the 10-minute sum alone. | Three runs. The first failed as registered (one 10-minute window at 1.36, 33 duplicates). With the sample bound, the rate held at 0.97 to 1.07 of R in every complete 10-minute window, none over, no duplicates. Beside a collector rule under the same 2.5x step, the hold read 0.91 to 1.13 (one window over the 1.10 bound, the storm's first) against the rule's 2.95 to 3.42; insert-plus-merge CPU did not separate the arms on this rig. | +| [`clickhouse-cloud-elasticity/`](clickhouse-cloud-elasticity/) | How far does a ClickHouse Cloud service's compute fall, in units and in billed dollars, when the rows written into its logs table fall, above the smallest size, and how long does the autoscaler take? One service in AWS us-east-1, the ClickStack `otel_logs` schema under a fixed query load, two arms that differ only in the rows written (all of the replay, or the 65.3% the Receiver kept), four pairs, read on the replicas' own system tables and on `usageCost`. | Current. Not behind a post. From a 4-unit start the cut took the service from 4 units to 3, 0.75 in units and 0.749 in billed compute, in each of three pairs, 2.9 to 4.3 h after the cut; from a 3-unit start it read no change in 10.25 h. | +| [`splunk-license/`](splunk-license/) | Splunk bills on what its licence meter counts, not on a file size. What does the 10x Receiver in front of Splunk do to that number? The same capture through the same Universal Forwarder into the same instance, twice, raw and compact, measured from `license_usage.log` and from the Monitoring Console's own searches, with every compact event read back through the 10x Splunk app and compared to the text that went in. | Current. Not behind a post. | +| [`datadog-flex-accounting/`](datadog-flex-accounting/) | Datadog bills Standard indexing per million events. When the 10x Receiver moves events from Standard to Flex Logs under the plan the 10x MCP writes, how many events and GB move, and what do they cost at Datadog's list prices? The released engine and MCP on one capture at three targets, counted two independent ways, priced per million events moved, with the custom metrics the Datadog metric output creates. Computed from the engine's counts at list price; no Datadog account needed. | Current. Not behind a post. | +| [`datadog-service-control/`](datadog-service-control/) | Inside one service a Datadog service filter cannot split the volume, so the rules a person writes are filters on the service's logger names. On the kafka container of two public captures of the same code, what does the per-pattern plan the 10x MCP writes move against the fewest `@logger.name` filters that move as many events, and what does each do on the second capture with nothing re-tuned? Computed from the engine's counts; no Datadog account needed. | Current. Not behind a post. | +| [`datadog-containment/`](datadog-containment/) | When a service floods its logs, does the released Receiver, installed with the values the MCP writes for Datadog, hold the flood before Datadog, including in a service with no rule, and what does it hold that it should not? The OpenTelemetry demo's own seeded failures on engine 1.1.142, and the Receiver's memory and CPU on a replay of the run's capture. | Current | +| [`datadog-ceiling/`](datadog-ceiling/) | With a monthly budget X for Datadog Standard indexing, does the released Receiver, driven by the released MCP's hourly budget run, hold Standard at X while log volume grows 2 to 3x, as Datadog's own usage metric counts it? The OpenTelemetry demo into a real Datadog org on engine 1.1.146 and log10x-mcp 1.30.71, with a new service, an error storm and one service held to its own number. | Current. Not behind a post. | +| [`elastic-hot-tier/`](elastic-hot-tier/) | With the Receiver's per-statement byte budget and log10x-mcp's Elasticsearch frozen-tier recipe, how long do the lines it moves stay on the hot tier, can they still be found by pattern hash on frozen, and how much of the volume does a fixed budget keep off hot as load grows, as the cluster's own APIs count it? Self-managed Elasticsearch 9.5.5 with hot and frozen nodes, engine 1.1.147, log10x-mcp 1.30.77, the published OpenTelemetry demo sample replayed for 11 hours. | Current. Not behind a post. | +| [`elastic-store-four-ways/`](elastic-store-four-ways/) | On Elasticsearch 9.5.5, how much index store does the engine's compact form save over a raw index, over logsdb, and over logsdb with `message` as `pattern_text` sorted on its template id, on the same sample? Primary store after a force merge to one segment, `best_compression` everywhere, trial licence, engine 1.1.150, the published OpenTelemetry demo sample, the compact form in standard, logsdb and template-sorted logsdb mode. Logsdb with `pattern_text` holds the sample in 34.7% of the raw index's store, the compact form on logsdb in 53.3% and sorted alike in 44.1%: on Elasticsearch 9.x the compact form is not the smaller store. | Current. Not behind a post. | +| [`elastic-hot-tier-budget/`](elastic-hot-tier-budget/) | Does log10x-mcp's hourly hot-tier run, driving the Receiver, hold an Elasticsearch hot tier at a size S while load grows 2.5 times, as the hot node's own stats count it? Self-managed Elasticsearch 9.5.5 with hot and frozen nodes and the published OpenTelemetry demo sample replayed, in four runs, the last two beside a native reroute by service. With the tier-down index on hot the hold fails its bound of 1.10 S, peaking at 1.17 and 1.23 S in the two single-arm runs and at 1.43 S against the reroute's 1.36 S; with it on warm the hold keeps hot at 0.94 to 1.04 S from 20 minutes after the step, 0 of 25 readings over, while the reroute reaches 1.25 S and is still rising. | Current. Not behind a post. | +| [`cloudwatch-ia-accounting/`](cloudwatch-ia-accounting/) | CloudWatch bills ingestion per GB, and an Infrequent Access log group ingests at half the Standard rate. When the 10x Receiver moves events to Infrequent Access or offloads them to S3 under the plan the 10x MCP writes, how many GB move, and what do they cost at AWS's list prices for us-east-1? Computed from the engine's counts at list price; no AWS account needed. | Current. Not behind a post. | +| [`intent/`](intent/) | Does each pattern hold one logging statement? Every event of two OpenTelemetry-demo captures and a LogHub ZooKeeper capture is labelled with the source statement that wrote it, from the programs' own source, and the 10x engine (1.1.142, the current release, on all three; 1.1.132 and 1.1.133 on the demo captures; 1.1.136 on ZooKeeper) and Drain3 are scored against those labels: events merged with another statement, events split from their statement's main pattern, and exact grouping. | Current. Linked from [*How a Drain pattern group changes as more lines arrive*](https://www.log10x.com/blog/drain-pattern-groups-change/). | +| [`naming-stability/`](naming-stability/) | When the engine moves to its next release, how many events change their pattern name, pattern hash or template hash? Engine 1.1.140, 1.1.141 and 1.1.142, each with the configuration and library its image ships, on the two OpenTelemetry-demo captures of `intent/`, compared event by event. | Current | +| [`loghub-compaction/`](loghub-compaction/) | How much smaller is the engine's compact form than the original logs, on the 14 full public LogHub datasets, with the engine and library as they ship? The compact form is the encoded events plus the template dictionary, measured before any codec. | Current | +| [`drain3-vs-log10x/`](drain3-vs-log10x/) | Log10x against [Drain3](https://github.com/logpai/Drain3) on the [LogHub](https://github.com/logpai/loghub) 2k sets: does a line get the same pattern ID regardless of file and order, and does the reduced form reverse to the original bytes? | Retired as a benchmark. The post it was written for never shipped. Its `tenx-encode.config.yaml` and `tenx-decode.config.yaml` are still cited elsewhere as the published round-trip configs, so the folder stays. | + +For anything about pattern identity, read `pattern-identity/`, not `drain3-vs-log10x/`. The two +ask overlapping questions and only the first is maintained. + +## Layout + +There is no single template. Each folder carries its own README with the exact run steps, and +each resolves its paths relative to itself, so a benchmark can be run from anywhere. + +| Folder | Entry point | Pinned deps | Where results land | External data | +|---|---|---|---|---| +| `pattern-identity/` | `./run.sh` | `requirements.txt` (drain3 0.9.11, Python 3.13) and a pinned engine image digest | `results/` | Release assets from `log-10x/config` tag `otel-sample-v1`, plus two LogHub sets from Zenodo record 3227177, fetched into `data/` on first run | +| `otel-denominators/` | `./run.sh` | pinned engine image digest, plus the round-trip configs in `drain3-vs-log10x/` | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1`, fetched into `data/` on first run | +| `clickhouse-inflate/` | `./run.sh` | pinned ClickHouse Docker image | `results.json` | none, the sample is committed | +| `clickhouse-clickstack/` | `./run.sh` | pinned engine image digest and a ClickHouse Docker image | `results/` | the `otel-sample-v2` release asset from `log-10x/config`, fetched into `data/` on first run | +| `clickstack-e2e/` | `./run.sh` | pinned ClickStack, engine, collector and MinIO images | `results/` | the `otel-sample-v2` release asset from `log-10x/config`, fetched into `data/` on first run | +| `clickstack-e2e-gaps/` | `./gap_*.sh` | the same images by digest, plus Vector by digest; the receiver is the published `edge-10x` image | `results/` | the same `otel-sample-v2` asset, read from `../clickstack-e2e/data` | +| `clickhouse-cloud-elasticity/` | `cloud/arms.py`, after the split, stage and meters in its README | the engine and collector images by digest (`results/images.txt`), Python 3.12+, a `clickhouse` client binary | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1`, checked by sha256; a ClickHouse Cloud organization | +| `splunk-license/` | `./run.sh` | pinned engine, Splunk and forwarder images, and the 10x Splunk app | `results/` | the `otel-sample-v2` release asset from `log-10x/config`, fetched into `data/` on first run | +| `datadog-flex-accounting/` | `./run.sh` | pinned engine image digest, log10x-mcp from npm at a pinned version, and the shared scripts in `accounting-lib/` | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1`, checked by sha256 | +| `datadog-ceiling/` | `rig/run.sh` phases, `rig/budget-phase.sh` | the engine image by digest, log10x-mcp 1.30.71 from npm, the demo at tag 3.1.0 | `results/` | a Datadog org with Flex Logs; the demo's own logs | +| `datadog-service-control/` | `./run.sh` | the same as `datadog-flex-accounting/` | `results/` | the same asset, and `rec1155.log.gz` from `log-10x/benchmarks` tag `intent-data-v1`, both checked by sha256 | +| `datadog-containment/` | `rig/go.sh`; `rig/memory/run.sh` for the memory replay | engine and Fluent Bit images by digest, log10x-mcp from npm at a pinned version, OpenTelemetry demo 3.1.0 | `results/` | the demo's images, pulled by tag | +| `elastic-hot-tier/` | `rig/make-replay.py`, then `rig/phases.sh` | Elasticsearch 9.5.5 and Fluent Bit 4.2 images by tag, the engine image by digest, a log10x-mcp checkout (the recipe and Receiver values are rendered from it) | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1` | +| `elastic-store-four-ways/` | `./run.sh` | the engine and Elasticsearch 9.5.5 images by digest | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1`, checked by sha256 | +| `cloudwatch-ia-accounting/` | `./run.sh` | the same as `datadog-flex-accounting/` | `results/` | the same asset | +| `intent/` | `./run.sh` | engine image digests (1.1.132, 1.1.133, 1.1.136, 1.1.142) and drain3 0.9.11 from `pattern-identity/requirements.txt` | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1` and `rec1155.log.gz` from this repository's `intent-data-v1` release, fetched into `cache/` on first run | +| `naming-stability/` | `./run.sh` | engine image digests (1.1.140, 1.1.141, 1.1.142) | `results/` | the same two captures as `intent/`, checked by sha256 | +| `drain3-vs-log10x/` | see its README | `requirements.txt` | `bench/facts.json`, `bench/results.json` | LogHub 2k sets, fetched into `loghub/` | + +Large inputs are never committed. Each folder's `.gitignore` excludes whatever that benchmark +downloads or generates at run time. + +## Continuous integration + +`.github/workflows/ci.yml` syntax-checks every driver and runs the two mechanism checks that +need no download: `drain3-vs-log10x/smoke_test.py` and `pattern-identity/bench/replay.py`. The +full runs need Docker and several hundred megabytes of public data, so they are run by hand and +their artifacts are committed. `otel-denominators/report.py` is checked the same way: CI +re-renders the committed CSV and fails if the rendered table does not match the committed +`results.md`, so an edited number in the table without a rerun is caught. `splunk-license` +carries the same guard against its own primary file: CI recomputes the licence figures from +the `RolloverSummary` lines the run kept verbatim and fails if `results.json` disagrees. `intent` re-renders `results/results.md` from the committed score files and fails if the tables disagree. `naming-stability` does the same from its committed `compare.json`. + +## License + +MIT. See [LICENSE](LICENSE). diff --git a/accounting-lib/readme.py b/accounting-lib/readme.py index f44f09c..a51b8cd 100644 --- a/accounting-lib/readme.py +++ b/accounting-lib/readme.py @@ -16,7 +16,7 @@ def apply(text: str, blocks: dict) -> str: found = {m.group(1) for m in BLOCK.finditer(text)} missing = sorted(set(blocks) - found) if missing: - raise SystemExit(f'README.md has no generated block for: {", ".join(missing)}') + raise SystemExit(f'RECORD.md has no generated block for: {", ".join(missing)}') return BLOCK.sub(lambda m: f'\n{blocks[m.group(1)].rstrip()}\n' if m.group(1) in blocks else m.group(0), text) diff --git a/clickhouse-clickstack/README.md b/clickhouse-clickstack/README.md index 9f2ba56..f9006ca 100644 --- a/clickhouse-clickstack/README.md +++ b/clickhouse-clickstack/README.md @@ -1,267 +1,49 @@ # clickhouse-clickstack -What does the 10x compact form cost or save on disk in **ClickHouse's own -recommended log schema**, measured the way a ClickHouse engineer would ask for it. +How much disk do 10x compact events save in ClickHouse's ClickStack `otel_logs` schema? On 197,430 lines from the OpenTelemetry demo, the whole table came out 14.9% smaller, but the SQL functions that decode compact rows in ClickHouse returned the wrong text for 78% of them. Keeping the original text and adding the message type as a column inside the sort key took 13.1% off the table with no decoder and the text index intact. -The question is narrow on purpose. ClickHouse has published its own log -compression work: [178x on nginx with typed columns][nginx] (52x once ordered by -time, "more representative of actual usage"), and [19x to 22x from Drain3 log -clustering, 45x with per-service typed tables][clustering]. Against that, a -compression claim measured on a single `String` column is not a claim a -ClickHouse engineer has any reason to read. So this benchmark uses the -[ClickStack `otel_logs` schema][clickstack] as published: typed columns, -`LowCardinality`, the two attribute maps, all eight skip indexes, `ZSTD(1)`, and -`ORDER BY (toStartOfFiveMinutes(Timestamp), ServiceName, Timestamp)`. +The 10x engine writes each message as a template hash plus its values and folds a multi-line event into one, so the lines became 159,170 compact events. -## Run it - -```bash -./run.sh # everything, reusing whatever is already in data/ -FRESH=1 ./run.sh # drop the cached engine output and rebuild it -KEEP=1 ./run.sh # leave the ClickHouse container up afterwards -``` - -Needs Docker, about 4 GB of free disk and roughly 20 minutes on a warm host. It -writes `results/results.json` and `results/clickstack-storage-.md`. - -## Three arms, because two would hide something - -The engine folds a multi-line event (a stack trace, a config dump) into one -event. That changes the row count, and a single ratio between "raw" and -"compact" would carry both the folding and the templating inside one number -without saying so. Three arms take them apart. - -| Arm | Rows | `Body` | Everything else | -|---|---|---|---| -| `native` | one per captured line | that line's message | from that line's envelope | -| `compact` | one per compact event | `~,` | from the same envelope | -| `native_folded` | one per compact event | the original text of that event | identical to `compact` | - -`native` against `compact` is what a deployment sees. `native_folded` against -`compact` holds the row count and every other column fixed, so it is the -templating layer on its own. `native` against `native_folded` is the folding on -its own. The results file reports all three and states both row counts every -time it states a ratio. - -The compact arm's total also carries the template dictionary, without which the -arm cannot be read. - -## INNER encode, not OUTER - -The public `otel-sample-v2` release ships an OUTER-mode encoding: the template -covers the whole Fluentd JSON record, so the compact event has no envelope left -to map into `ServiceName`, `ResourceAttributes` and `LogAttributes`. Putting that -form in `Body` and then filling the attribute columns from the raw record would -put the Kubernetes metadata on the compact side twice. - -So `run.sh` re-encodes the same released capture in **INNER** mode with -[`tenx-encode-inner.config.yaml`](tenx-encode-inner.config.yaml): the envelope -stays JSON and only the `log` value is compacted. That is also the only mode the -ClickHouse app supports ("The plugin requires INNER encode mode; OUTER encode is -not supported on ClickHouse"), and it is the shape of the corpus in -[`../clickhouse-inflate/corpus/`](../clickhouse-inflate/corpus/). The source -logs are the released capture either way, and its sha256 is checked on every run. - -## How the two arms are aligned - -A compact event covers K consecutive captured lines. K is recovered exactly from -the event's own template, because the engine joins folded lines with a newline -and the template carries K-1 of them. `build_arms.py` refuses to write anything -unless both invariants hold on every record: - -1. the line count reconstructed this way equals the number of lines in the capture; -2. the envelope of each compact record, every field except `log`, is byte-identical - to the envelope of the first captured line it covers. - -## The timestamp - -The Fluentd envelope has no timestamp field; the time is inside the message. This -benchmark does not write a second timestamp parser. It reads the one the engine -already produced: a template with a timestamp slot carries that slot's value as -an epoch integer in the compact event, and the scale comes from the digit count. -84.6% of compact events carry one; for the rest the last value seen for that -container is carried forward. Every arm gets the same value for the same event, -so `ORDER BY` and `PARTITION BY` see the same data in each. - -## The decode path - -The compact form itself is lossless: the engine's round trip on this capture -returns all 215,039,161 bytes byte-identical, which -[`../otel-denominators/`](../otel-denominators/) reproduces in one command. What -is broken is the SQL that reads it back inside ClickHouse, which this benchmark -measures and `reference_decode.py` diagnoses. - -The compact arm is read through the **corrected** inflate functions in -[`../clickhouse-inflate/install-fixed.sql`](../clickhouse-inflate/install-fixed.sql), -not the earlier ones. The earlier `tenx_inflate_core` built its result with an -index lambda that captures the `literals` and `slots` arrays; that is quadratic -and aborts a full-table decode at 7.2 GiB. -[`../clickhouse-inflate/`](../clickhouse-inflate/) has that measurement. The SQL -decoder is not offered for ClickHouse; the product's lever there is offload. +## Result -## The second rung: `run_typed.sh` +Whole table on disk at ZSTD(1), with the 348,040-byte template dictionary in every compact total. -`run.sh` stores the compact event the way the ClickHouse app does today, as one -text string in `Body`. ClickHouse's own [log-clustering post][clustering] measured -that shape at 22x against 19x raw, and 45x once the template sat in a -low-cardinality column and the values in typed columns. `run_typed.sh` climbs -the same ladder on the same events, against the container `KEEP=1 ./run.sh` -leaves up: +| Layout | Rows | Bytes on disk | Smaller than published | Full-text index on Body | +|---|---:|---:|---:|---| +| ClickStack as published | 197,430 | 8,225,547 | 0.0% | kept | +| Compact events in Body | 159,170 | 6,999,850 | 14.9% | kept | +| Compact, hash in sort key | 159,170 | 6,188,001 | 24.8% | kept | +| Compact, typed value arrays | 159,170 | 4,376,741 | 46.8% | none | +| Original text, type in sort key | 197,430 | 7,148,605 | 13.1% | kept | +| Original text, sorted by type | 197,430 | 6,451,087 | 21.6% | kept | -| Layout | What changes | -|---|---| -| `compact_sorted` | the template hash materialised and added to ClickStack's sort key; `Body` unchanged | -| `compact_typed` | `Body` replaced by the hash plus typed value arrays, same sort key | -| `compact_typed_bytemplate` | the typed layout sorted by (service, template, time) | +Against the original text of the same events, the compact form takes 23.55% off the `Body` column. The larger savings come mostly from ClickStack's full-text index on Body shrinking or going away, and sorting by type first weakens the time-range primary key. -`build_typed.py` splits every compact event into timestamp, integer and text -arrays, and refuses to write if any row does not reverse to its compact event -byte for byte; `run_typed.sh` rebuilds the events again inside ClickHouse and -compares hash sums with the text form. Ingest cost is measured table-to-table -with `INSERT ... SELECT`, three copies per layout, so no arm pays a JSON-parsing -cost and the CPU is what ClickHouse spends building and compressing that layout. +- Through the SQL decoder, 78.0% of compact rows matched no captured line, and a text filter took 531 ms against 6 ms on the original rows. A Python decoder with the four rules the SQL lacks returned 24,494 of the 24,501 events without a timestamp slot byte-identical, so the text is intact and the SQL misreads it. log10x does not offer the SQL decoder for ClickHouse. +- Row for row, the compact text cost the same insert CPU as the original (1.64 s). Removing rows cut CPU more than proportionally: half the rows cost 39% of insert-plus-merge CPU and a quarter cost 17%. +- TTL RECOMPRESS does nothing on ClickStack, which declares CODEC(ZSTD(1)) on every column. Without that codec, the original-text table at ZSTD(12) was 29.9% smaller than at ZSTD(1), more than any layout above that keeps the text index. -The result is in `results/typed-layout-.md`. In short: the generic typed -arrays compress the payload worse than the text string, the template hash in the -sort key is the one cheap real win, and the large total savings all come from -ClickStack's full-text index on `Body` shrinking or disappearing rather than from -the data compressing better. +## How it was measured -## The compute question: `run_compute.sh` +The data is `otel-sample-200mb.log` from the otel-sample-v2 release, loaded into the published ClickStack DDL with all eight skip indexes and its sort key, minus the TTL placeholder. ClickHouse 26.8.3.105 ran in one container with 8 GiB and 8 CPUs. Every byte count is read from system.parts and system.parts_columns after OPTIMIZE FINAL. -Storage is not where a ClickHouse bill is. ClickHouse Cloud meters compute per -unit-hour and storage at **$25.30 per compressed TB per month**, so an estate of -ordinary size carries tens of dollars of storage and the compute line is the -bill. `run_compute.sh` asks whether the product can move compute. +## Limits -The claim under test is that compute follows row count, so removing rows is a -lever and making each row smaller is not. The falsifier is stated in the script -before the run: if halving the rows does not roughly halve insert-plus-merge -CPU, the claim is wrong. +- The data is one demo capture in one container. +- ClickHouse's own 45x figure comes from one typed table per service, which this single-table test does not reproduce. +- CPU covers ClickHouse's insert and merge time, not the collector and not the write cost of ZSTD(12). +- No price is attached. -Every arm holds the same original log text in the same schema; only the rows -present change. `sampled_75/50/25` is a seeded uniform sample, which holds the -content mix fixed and isolates row count. `bypattern_75/50/25` removes whole -message types largest first, which is the lever the product pulls. Inserts -arrive in batches of a fixed number of rows, so an arm with half the data sends -half the batches, and merges run on their own and are read from -`system.part_log`. Each arm loads three times after a discarded warm-up, fastest -kept. - -Result: half the rows cost 39% of the CPU and a quarter cost 17%, both below the -row share, because fewer parts merge fewer times. Full detail in -`results/compute-vs-rows-.md`. The arms hold the row shape fixed. Rows of a -different shape cost differently: gap 6 in -[`../clickstack-e2e-gaps/`](../clickstack-e2e-gaps/) compares a collector rule that -stores more, smaller rows with the per-type cap, whose rows are fewer and larger. - -## Per-message-type policy: `run_policy.sh` - -Every other destination gives a log line a cheaper place to go, and the -assumption worth checking is that ClickHouse gives it none, leaving deletion as -the only lever. ClickHouse's own TTL grammar says otherwise: `DELETE`, -`RECOMPRESS codec`, `TO DISK`, `TO VOLUME`, plus `WHERE` and `GROUP BY`. Four -levers, two of which keep every line. What ClickHouse cannot do is aim any of -them at a message type, because nothing in the row says which message it is. - -`run_policy.sh` adds a `templateHash` column and drives all four from one -expression over it, on its own small container in about two minutes. Results in -`results/per-pattern-policy-.md`. In short: moving to a cheaper volume, -rolling up into counts and per-type retention all work per message type; -recompression fires and does nothing, because an explicit column codec beats the -part default and the ClickStack schema declares `CODEC(ZSTD(1))` on every -column, which reproduces hyperdxio/hyperdx issue 2525. - -Two traps it surfaces for anyone writing these policies. A TTL move is decided -for a whole part, so a table not partitioned by time moves nothing at all. And a -column codec silently disables `TTL ... RECOMPRESS` against that column. - -## The shape being proposed: `run_labeled.sh` - -Everything above compares native text against compact text. The recommendation that -fell out of it, keep the original text and add the message type as a column, was never -loaded until this script. Three arms on the same 197,430 rows: the column added, the -column inside ClickStack's sort key, and the table sorted by type first. Text index -intact in all three, no decoder anywhere. Result in `results/labeled-layout-.md`: -the column alone costs 91 KB; inside the sort key it takes 13.1% off the table; sorted by -type first, 21.6%. That is most of what compaction achieved, with none of its costs. - -## Query compute: `run_query_compute.sh` - -Five query shapes against the reduction arms. At this table size every query is a few -milliseconds of fixed cost and the run cannot resolve the row-count effect; it says so -in `results/query-compute-.md` rather than reporting the 85% it measured as a -finding. A table of tens of gigabytes would be needed. - -## Tier-down to object storage: `run_policy_s3.sh` - -`run_policy.sh` used a second directory as the cold volume. This one runs MinIO beside -ClickHouse, declares the bucket as a disk of `type s3`, and repeats the per-type move. -2,700 objects land in the bucket, the rows read back, the table stays whole. Result in -`results/per-pattern-policy-s3-.md`. - -## Two lossless levers on the real capture: `run_lossless.sh` - -The two lossless levers nobody had measured. Insert batch shape: the same 197,430 rows -loaded in batches of 1,000, 5,000, 20,000 and 100,000 rows, insert-plus-merge CPU per -arm, three passes after a warm-up. Recompression: the real `Body` at ZSTD(1), 3, 9 and 12 -with no column codec in the way. Result in `results/lossless-levers-.md`: batching -takes insert-plus-merge CPU from 29 s to 4 s across that range with nothing removed, and -ZSTD(12) takes 19% off the text and 30% off the table. The first is a collector or engine -setting; the second is what ClickStack's per-column codecs block. - -## The label as a query lever at scale: `run_scale.sh` and `run_scale_bytype.sh` - -`run_query_compute.sh` could not resolve query cost at 7 MB. These replicate the labeled -rows a hundred times inside ClickHouse, one shifted day per insert, to 19,743,000 rows, -and ask the same question three ways: a plain text scan, ClickStack's own full-text index -via `hasToken`, and the label. The label is also asked what text cannot answer, the census -of message types, and the text index is asked what the label cannot, a word across -everything. `run_scale_bytype.sh` repeats the label queries with the type second in the -sort key after service, the label's best case, and shows what the time-window query pays for it. -Results in `results/label-query-scale-.md`. In short: ClickStack's text index -answers a token count from its postings in about 20 ms and the label, third in the -ClickStack key, cannot beat that; the label's own ground is the census, which text has no -version of, and the sort-key storage effect holds at this size at 13.6%. The replicated -data compresses better than real data would; the query comparison is unaffected. - -## What it does not measure - -- **Ingest over the wire.** Each arm is loaded from a JSONEachRow file, so the - CPU figure includes parsing a payload that is a different size in each arm. It - is reported as that and not as collector-to-ClickHouse ingest. -- **Retention.** The published DDL ends in a `${TABLES_TTL}` placeholder, which - has no value outside a ClickStack install. The TTL clause is dropped; nothing - else in the DDL is changed. -- **Per-service typed tables.** ClickHouse's 45x figure comes from one table per - service with the variable fields promoted to typed columns. This is the single - `otel_logs` table, which is what ClickStack installs by default. +## Run it -## Layout +``` +cd clickhouse-clickstack +python3 reference_decode.py --self-test # decode rules only, no Docker or download +KEEP=1 ./run.sh # the storage tables, leaves ClickHouse up +./run_typed.sh # compact rows sorted by hash, and typed arrays +./run_labeled.sh # original text plus a message-type column +``` -| File | What | -|---|---| -| `run.sh` | the whole run: fetch, encode, map, load, measure, report | -| `tenx-encode-inner.config.yaml` | the INNER encode config | -| `build_arms.py` | the mapping into the ClickStack schema, and the alignment checks | -| `schema.sql.tpl` | the ClickStack DDL with the codec substituted | -| `report.py` | reads `system.parts`, `system.parts_columns`, `system.query_log` and writes `results/` | -| `run_typed.sh` | the second rung: sort key, typed arrays, clean ingest cost, against the container `run.sh` left up | -| `build_typed.py` | splits compact events into typed arrays, reversing every row before writing | -| `report_typed.py` | writes `results/typed-layout-.md` | -| `run_compute.sh` | does compute follow rows: seven arms, batched inserts, background merges | -| `build_reduction.py` | labels every captured line with its message type and builds the reduction arms | -| `report_compute.py` | writes `results/compute-vs-rows-.md` | -| `run_policy.sh` | the four TTL levers driven per message type, on its own container | -| `run_policy_s3.sh` | the per-type move again, to a MinIO bucket declared as an S3 disk | -| `run_labeled.sh` | native text plus a message-type column, in and out of the sort key | -| `run_query_compute.sh` | five query shapes against the reduction arms | -| `run_lossless.sh` | insert batch shape against merge cost, and recompression levels on the real text | -| `run_scale.sh` | the label against the text scan and the text index on 19.7M rows | -| `run_scale_bytype.sh` | the same with the type first in the sort key | -| `reference_decode.py` | decodes the same events with the four rules `install.sql` lacks, to tell a lost original from a misread one; `--self-test` runs in CI | -| `results/` | `results.json` and the dated results file | +The scripts need Docker and about 4 GB of disk. The storage run downloads the capture and takes roughly 20 minutes, and the other two reuse its container. -[nginx]: https://clickhouse.com/blog/log-compression-170x -[clustering]: https://clickhouse.com/blog/improve-compression-log-clustering -[clickstack]: https://clickhouse.com/docs/use-cases/observability/clickstack/ingesting-data/schemas +The full record, with the decode diagnosis and the compute, TTL, batch-size and query-at-scale runs, is in [RECORD.md](RECORD.md). diff --git a/clickhouse-clickstack/RECORD.md b/clickhouse-clickstack/RECORD.md new file mode 100644 index 0000000..9f2ba56 --- /dev/null +++ b/clickhouse-clickstack/RECORD.md @@ -0,0 +1,267 @@ +# clickhouse-clickstack + +What does the 10x compact form cost or save on disk in **ClickHouse's own +recommended log schema**, measured the way a ClickHouse engineer would ask for it. + +The question is narrow on purpose. ClickHouse has published its own log +compression work: [178x on nginx with typed columns][nginx] (52x once ordered by +time, "more representative of actual usage"), and [19x to 22x from Drain3 log +clustering, 45x with per-service typed tables][clustering]. Against that, a +compression claim measured on a single `String` column is not a claim a +ClickHouse engineer has any reason to read. So this benchmark uses the +[ClickStack `otel_logs` schema][clickstack] as published: typed columns, +`LowCardinality`, the two attribute maps, all eight skip indexes, `ZSTD(1)`, and +`ORDER BY (toStartOfFiveMinutes(Timestamp), ServiceName, Timestamp)`. + +## Run it + +```bash +./run.sh # everything, reusing whatever is already in data/ +FRESH=1 ./run.sh # drop the cached engine output and rebuild it +KEEP=1 ./run.sh # leave the ClickHouse container up afterwards +``` + +Needs Docker, about 4 GB of free disk and roughly 20 minutes on a warm host. It +writes `results/results.json` and `results/clickstack-storage-.md`. + +## Three arms, because two would hide something + +The engine folds a multi-line event (a stack trace, a config dump) into one +event. That changes the row count, and a single ratio between "raw" and +"compact" would carry both the folding and the templating inside one number +without saying so. Three arms take them apart. + +| Arm | Rows | `Body` | Everything else | +|---|---|---|---| +| `native` | one per captured line | that line's message | from that line's envelope | +| `compact` | one per compact event | `~,` | from the same envelope | +| `native_folded` | one per compact event | the original text of that event | identical to `compact` | + +`native` against `compact` is what a deployment sees. `native_folded` against +`compact` holds the row count and every other column fixed, so it is the +templating layer on its own. `native` against `native_folded` is the folding on +its own. The results file reports all three and states both row counts every +time it states a ratio. + +The compact arm's total also carries the template dictionary, without which the +arm cannot be read. + +## INNER encode, not OUTER + +The public `otel-sample-v2` release ships an OUTER-mode encoding: the template +covers the whole Fluentd JSON record, so the compact event has no envelope left +to map into `ServiceName`, `ResourceAttributes` and `LogAttributes`. Putting that +form in `Body` and then filling the attribute columns from the raw record would +put the Kubernetes metadata on the compact side twice. + +So `run.sh` re-encodes the same released capture in **INNER** mode with +[`tenx-encode-inner.config.yaml`](tenx-encode-inner.config.yaml): the envelope +stays JSON and only the `log` value is compacted. That is also the only mode the +ClickHouse app supports ("The plugin requires INNER encode mode; OUTER encode is +not supported on ClickHouse"), and it is the shape of the corpus in +[`../clickhouse-inflate/corpus/`](../clickhouse-inflate/corpus/). The source +logs are the released capture either way, and its sha256 is checked on every run. + +## How the two arms are aligned + +A compact event covers K consecutive captured lines. K is recovered exactly from +the event's own template, because the engine joins folded lines with a newline +and the template carries K-1 of them. `build_arms.py` refuses to write anything +unless both invariants hold on every record: + +1. the line count reconstructed this way equals the number of lines in the capture; +2. the envelope of each compact record, every field except `log`, is byte-identical + to the envelope of the first captured line it covers. + +## The timestamp + +The Fluentd envelope has no timestamp field; the time is inside the message. This +benchmark does not write a second timestamp parser. It reads the one the engine +already produced: a template with a timestamp slot carries that slot's value as +an epoch integer in the compact event, and the scale comes from the digit count. +84.6% of compact events carry one; for the rest the last value seen for that +container is carried forward. Every arm gets the same value for the same event, +so `ORDER BY` and `PARTITION BY` see the same data in each. + +## The decode path + +The compact form itself is lossless: the engine's round trip on this capture +returns all 215,039,161 bytes byte-identical, which +[`../otel-denominators/`](../otel-denominators/) reproduces in one command. What +is broken is the SQL that reads it back inside ClickHouse, which this benchmark +measures and `reference_decode.py` diagnoses. + +The compact arm is read through the **corrected** inflate functions in +[`../clickhouse-inflate/install-fixed.sql`](../clickhouse-inflate/install-fixed.sql), +not the earlier ones. The earlier `tenx_inflate_core` built its result with an +index lambda that captures the `literals` and `slots` arrays; that is quadratic +and aborts a full-table decode at 7.2 GiB. +[`../clickhouse-inflate/`](../clickhouse-inflate/) has that measurement. The SQL +decoder is not offered for ClickHouse; the product's lever there is offload. + +## The second rung: `run_typed.sh` + +`run.sh` stores the compact event the way the ClickHouse app does today, as one +text string in `Body`. ClickHouse's own [log-clustering post][clustering] measured +that shape at 22x against 19x raw, and 45x once the template sat in a +low-cardinality column and the values in typed columns. `run_typed.sh` climbs +the same ladder on the same events, against the container `KEEP=1 ./run.sh` +leaves up: + +| Layout | What changes | +|---|---| +| `compact_sorted` | the template hash materialised and added to ClickStack's sort key; `Body` unchanged | +| `compact_typed` | `Body` replaced by the hash plus typed value arrays, same sort key | +| `compact_typed_bytemplate` | the typed layout sorted by (service, template, time) | + +`build_typed.py` splits every compact event into timestamp, integer and text +arrays, and refuses to write if any row does not reverse to its compact event +byte for byte; `run_typed.sh` rebuilds the events again inside ClickHouse and +compares hash sums with the text form. Ingest cost is measured table-to-table +with `INSERT ... SELECT`, three copies per layout, so no arm pays a JSON-parsing +cost and the CPU is what ClickHouse spends building and compressing that layout. + +The result is in `results/typed-layout-.md`. In short: the generic typed +arrays compress the payload worse than the text string, the template hash in the +sort key is the one cheap real win, and the large total savings all come from +ClickStack's full-text index on `Body` shrinking or disappearing rather than from +the data compressing better. + +## The compute question: `run_compute.sh` + +Storage is not where a ClickHouse bill is. ClickHouse Cloud meters compute per +unit-hour and storage at **$25.30 per compressed TB per month**, so an estate of +ordinary size carries tens of dollars of storage and the compute line is the +bill. `run_compute.sh` asks whether the product can move compute. + +The claim under test is that compute follows row count, so removing rows is a +lever and making each row smaller is not. The falsifier is stated in the script +before the run: if halving the rows does not roughly halve insert-plus-merge +CPU, the claim is wrong. + +Every arm holds the same original log text in the same schema; only the rows +present change. `sampled_75/50/25` is a seeded uniform sample, which holds the +content mix fixed and isolates row count. `bypattern_75/50/25` removes whole +message types largest first, which is the lever the product pulls. Inserts +arrive in batches of a fixed number of rows, so an arm with half the data sends +half the batches, and merges run on their own and are read from +`system.part_log`. Each arm loads three times after a discarded warm-up, fastest +kept. + +Result: half the rows cost 39% of the CPU and a quarter cost 17%, both below the +row share, because fewer parts merge fewer times. Full detail in +`results/compute-vs-rows-.md`. The arms hold the row shape fixed. Rows of a +different shape cost differently: gap 6 in +[`../clickstack-e2e-gaps/`](../clickstack-e2e-gaps/) compares a collector rule that +stores more, smaller rows with the per-type cap, whose rows are fewer and larger. + +## Per-message-type policy: `run_policy.sh` + +Every other destination gives a log line a cheaper place to go, and the +assumption worth checking is that ClickHouse gives it none, leaving deletion as +the only lever. ClickHouse's own TTL grammar says otherwise: `DELETE`, +`RECOMPRESS codec`, `TO DISK`, `TO VOLUME`, plus `WHERE` and `GROUP BY`. Four +levers, two of which keep every line. What ClickHouse cannot do is aim any of +them at a message type, because nothing in the row says which message it is. + +`run_policy.sh` adds a `templateHash` column and drives all four from one +expression over it, on its own small container in about two minutes. Results in +`results/per-pattern-policy-.md`. In short: moving to a cheaper volume, +rolling up into counts and per-type retention all work per message type; +recompression fires and does nothing, because an explicit column codec beats the +part default and the ClickStack schema declares `CODEC(ZSTD(1))` on every +column, which reproduces hyperdxio/hyperdx issue 2525. + +Two traps it surfaces for anyone writing these policies. A TTL move is decided +for a whole part, so a table not partitioned by time moves nothing at all. And a +column codec silently disables `TTL ... RECOMPRESS` against that column. + +## The shape being proposed: `run_labeled.sh` + +Everything above compares native text against compact text. The recommendation that +fell out of it, keep the original text and add the message type as a column, was never +loaded until this script. Three arms on the same 197,430 rows: the column added, the +column inside ClickStack's sort key, and the table sorted by type first. Text index +intact in all three, no decoder anywhere. Result in `results/labeled-layout-.md`: +the column alone costs 91 KB; inside the sort key it takes 13.1% off the table; sorted by +type first, 21.6%. That is most of what compaction achieved, with none of its costs. + +## Query compute: `run_query_compute.sh` + +Five query shapes against the reduction arms. At this table size every query is a few +milliseconds of fixed cost and the run cannot resolve the row-count effect; it says so +in `results/query-compute-.md` rather than reporting the 85% it measured as a +finding. A table of tens of gigabytes would be needed. + +## Tier-down to object storage: `run_policy_s3.sh` + +`run_policy.sh` used a second directory as the cold volume. This one runs MinIO beside +ClickHouse, declares the bucket as a disk of `type s3`, and repeats the per-type move. +2,700 objects land in the bucket, the rows read back, the table stays whole. Result in +`results/per-pattern-policy-s3-.md`. + +## Two lossless levers on the real capture: `run_lossless.sh` + +The two lossless levers nobody had measured. Insert batch shape: the same 197,430 rows +loaded in batches of 1,000, 5,000, 20,000 and 100,000 rows, insert-plus-merge CPU per +arm, three passes after a warm-up. Recompression: the real `Body` at ZSTD(1), 3, 9 and 12 +with no column codec in the way. Result in `results/lossless-levers-.md`: batching +takes insert-plus-merge CPU from 29 s to 4 s across that range with nothing removed, and +ZSTD(12) takes 19% off the text and 30% off the table. The first is a collector or engine +setting; the second is what ClickStack's per-column codecs block. + +## The label as a query lever at scale: `run_scale.sh` and `run_scale_bytype.sh` + +`run_query_compute.sh` could not resolve query cost at 7 MB. These replicate the labeled +rows a hundred times inside ClickHouse, one shifted day per insert, to 19,743,000 rows, +and ask the same question three ways: a plain text scan, ClickStack's own full-text index +via `hasToken`, and the label. The label is also asked what text cannot answer, the census +of message types, and the text index is asked what the label cannot, a word across +everything. `run_scale_bytype.sh` repeats the label queries with the type second in the +sort key after service, the label's best case, and shows what the time-window query pays for it. +Results in `results/label-query-scale-.md`. In short: ClickStack's text index +answers a token count from its postings in about 20 ms and the label, third in the +ClickStack key, cannot beat that; the label's own ground is the census, which text has no +version of, and the sort-key storage effect holds at this size at 13.6%. The replicated +data compresses better than real data would; the query comparison is unaffected. + +## What it does not measure + +- **Ingest over the wire.** Each arm is loaded from a JSONEachRow file, so the + CPU figure includes parsing a payload that is a different size in each arm. It + is reported as that and not as collector-to-ClickHouse ingest. +- **Retention.** The published DDL ends in a `${TABLES_TTL}` placeholder, which + has no value outside a ClickStack install. The TTL clause is dropped; nothing + else in the DDL is changed. +- **Per-service typed tables.** ClickHouse's 45x figure comes from one table per + service with the variable fields promoted to typed columns. This is the single + `otel_logs` table, which is what ClickStack installs by default. + +## Layout + +| File | What | +|---|---| +| `run.sh` | the whole run: fetch, encode, map, load, measure, report | +| `tenx-encode-inner.config.yaml` | the INNER encode config | +| `build_arms.py` | the mapping into the ClickStack schema, and the alignment checks | +| `schema.sql.tpl` | the ClickStack DDL with the codec substituted | +| `report.py` | reads `system.parts`, `system.parts_columns`, `system.query_log` and writes `results/` | +| `run_typed.sh` | the second rung: sort key, typed arrays, clean ingest cost, against the container `run.sh` left up | +| `build_typed.py` | splits compact events into typed arrays, reversing every row before writing | +| `report_typed.py` | writes `results/typed-layout-.md` | +| `run_compute.sh` | does compute follow rows: seven arms, batched inserts, background merges | +| `build_reduction.py` | labels every captured line with its message type and builds the reduction arms | +| `report_compute.py` | writes `results/compute-vs-rows-.md` | +| `run_policy.sh` | the four TTL levers driven per message type, on its own container | +| `run_policy_s3.sh` | the per-type move again, to a MinIO bucket declared as an S3 disk | +| `run_labeled.sh` | native text plus a message-type column, in and out of the sort key | +| `run_query_compute.sh` | five query shapes against the reduction arms | +| `run_lossless.sh` | insert batch shape against merge cost, and recompression levels on the real text | +| `run_scale.sh` | the label against the text scan and the text index on 19.7M rows | +| `run_scale_bytype.sh` | the same with the type first in the sort key | +| `reference_decode.py` | decodes the same events with the four rules `install.sql` lacks, to tell a lost original from a misread one; `--self-test` runs in CI | +| `results/` | `results.json` and the dated results file | + +[nginx]: https://clickhouse.com/blog/log-compression-170x +[clustering]: https://clickhouse.com/blog/improve-compression-log-clustering +[clickstack]: https://clickhouse.com/docs/use-cases/observability/clickstack/ingesting-data/schemas diff --git a/clickhouse-cloud-elasticity/README.md b/clickhouse-cloud-elasticity/README.md index 0294eb4..5f47d5f 100644 --- a/clickhouse-cloud-elasticity/README.md +++ b/clickhouse-cloud-elasticity/README.md @@ -1,37 +1,44 @@ # ClickHouse Cloud elasticity -One benchmark in the [log10x benchmarks](../) repository. It asks how far a ClickHouse Cloud service's compute falls, in units and in billed dollars, when the rows written into its logs table fall, above the smallest size, and how long the autoscaler takes to lower it. +When a ClickHouse Cloud service is sent fewer log rows, how far does its compute bill fall, and how soon? In three of four paired runs, writing 65.3% of the rows took the service from 4 units to 3, and billed compute settled at 0.749 of the full feed's rate. The step came 2.9 to 4.3 hours after the cut, and until then both arms paid for 4 units. In the other, which started at 3 units, the same cut changed nothing in 10.25 hours. -**Result: from a 4-unit start, writing 0.653 of the rows took the service from 4 units to 3, a ratio of 0.75, and billed compute to 0.749, in each of three pairs; the step down was done 2.9 to 4.3 hours after the cut. From a 3-unit start the same cut read no change in 10.25 hours.** At 3 units both arms sit inside the autoscaler's no-action band; at 4 units arm B sits below it. Every figure, with the file and field it was read from, is in [RESULT.md](RESULT.md). +The autoscaler's documented rule explains the difference: between 37.5% and 75% CPU it takes no action. On 4 units the busiest replica ran at a median 0.440 to 0.486 of its CPU with every row and 0.337 to 0.351 with the kept rows, below the band. On 3 units both arms stayed inside the band. -ClickHouse Cloud, AWS us-east-1, Scale tier, ClickHouse 26.6, 2 replicas, vertical autoscaling 8 to 32 GiB per replica, idle after 5 minutes. The ClickStack `otel_logs` schema under seven queries a minute. Arm A writes every record of the 215 MB OTel replay at 70,000 rows/s; arm B writes the 65.3% the Receiver (`log10x/edge-10x:1.1.153`) kept, at 45,716 rows/s. 2026-11-02 19:48 to 2026-11-05 03:47 UTC. The service was deleted at the end (`results/raw/teardown.jsonl`). +## Result -| Pair | Arm A's size | Units, A to B | Ratio | usageCost computeCHC per 15 minutes, A to B | Cut to step down done | -|---|---|---|---|---|---| -| 1 | the autoscaler's | 4.0 to 2.997 | 0.749 | 0.298 to 0.224 | 4.33 h | -| 2 | the autoscaler's, where pair 1 left it | 3.0 to 3.0 | 1.0 | 0.224 to 0.224 | no step in 10.25 h | -| 3 | 16 GiB by config, released at the cut | 4.0 to 3.0 | 0.750 | 0.298 to 0.224 | 4.21 h | -| 4 | 16 GiB by config, released at the cut | 4.0 to 3.0 | 0.750 | 0.298 to 0.224 | 2.91 h | +One unit is 8 GiB and 2 vCPU. Arm A writes every row, arm B the 65.3% the 10x engine kept. -## Rerun +| Pair | Arm A's size set by | Settled units, A to B | Billed $ per 15 min, A to B | Hours from cut to step down | +|---|---|---:|---:|---:| +| 1 | the autoscaler | 4.0 to 2.997 | 0.298 to 0.224 | 4.33 | +| 2 | the autoscaler, after pair 1 | 3.0 to 3.0 | 0.224 to 0.224 | no step in 10.25 | +| 3 | config, released at the cut | 4.0 to 3.0 | 0.298 to 0.224 | 4.21 | +| 4 | config, released at the cut | 4.0 to 3.0 | 0.298 to 0.224 | 2.91 | -Needs a ClickHouse Cloud organization and API key, Docker, Python 3.12 or later and a `clickhouse` client binary (26.8 here). `cloud_api.py` reads the key from SSM or from `CHC_KEY_ID` and `CHC_KEY_SECRET` and never prints it. The scripts keep the service password and client config under `~/git/l1x-co/.claude-tmp/chce-secrets/` by default; `CHC_SECRET_FILE`, `CLIENT` and `CONF` move them. +The billed rate is $1.196 an hour at 4 units against $0.896 at 3. Size fell to 0.75 while rows fell to 0.653, because the service made one step, from 16 GiB to 12 GiB per replica. -1. **Split.** `split/run_slices.sh` replays the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1` (checked by sha256) through four Receivers side by side and joins what they returned into `split/build/wire/returned.json`, in about an hour. -2. **Service.** Put the runner's address in `cloud/service.json`, then run `CHC_SECRET_FILE= python3 cloud_api.py create cloud/service.json`. The service password goes to that file (mode 600), not to the terminal. -3. **Tables.** Apply `cloud/schema.sql` with the client, then `cloud/stage.sh` and `cloud/stage_b.sh`. -4. **Meters**, left running, each with `--service ` and an `--out` under `results/raw/`: `cloud/sample.py`, `cloud/meter.py --every 60 --lag 30` and `cloud/usage_poll.py`. -5. **Feed and queries**, left running: `cloud/load.py` and `cloud/queries.py --overlap 3`. Both follow `results/raw/control.json`, which `cloud/phase.py` writes. -6. **Probe.** `cloud/probe.py` with steps of 2,000 to 70,000 rows/s, 10 minutes each. -7. **Arms.** `cloud/arms.py`. Its docstring gives both forms this run used: `--pairs` for the natural pairs, `--forced` for the pairs with arm A's size set by config, then the read calibration. -8. **Report.** `cloud/arms_report.py --raw results/raw --lag-min 30` for every phase, segment and pair; `cloud/curve.py` for the probe. Both read the committed `.jsonl.gz` files as they are. +- The autoscaler's documented scale-down window is 3 hours. Why pair 4 stepped 1.3 hours sooner than pairs 1 and 3 was not measured. +- Below the load that lifts the full feed off the 2-unit minimum, between 50,000 and 70,000 rows a second, both arms cost $0.60 an hour. -## Files +## How it was measured -| Path | What | -|---|---| -| `RESULT.md` | Every figure, the file and field it came from, and what was not measured | -| `split/` | The engine split | -| `cloud/` | The service request, schema, stage, feed, queries, meters, driver and reports | -| `results/` | The split and stage readings, the service creation response | -| `results/raw/` | Every log the scripts wrote: `events.jsonl` (phases and size changes), `scaling-patches.jsonl`, `usage.jsonl`, `teardown.jsonl`, and the meter, feed, sampler and query logs gzipped | +The service ran on the Scale tier in AWS us-east-1, ClickHouse 26.6, with 2 replicas autoscaling from 8 to 32 GiB each. Both arms wrote into the ClickStack `otel_logs` schema under seven queries a minute. Arm A replayed all 171,807 records of a 215 MB OpenTelemetry sample at 70,000 rows a second, arm B the 112,205 kept rows at 45,716. In pairs 3 and 4, config pinned arm A at 16 GiB per replica until the cut, so each step down was the autoscaler's own. Size and CPU were read every 60 seconds, dollars from the usageCost API every 10 minutes. + +## Limits + +- One service shape in one region, with seven queries a minute instead of a production mix. +- Both arms inserted from staged tables, so neither the engine nor the parsing of network inserts ran live. +- Which replica the autoscaler reads, and how often it decides, was not measured. + +## Run it + +The pair ratios recompute from committed logs: + +``` +cd clickhouse-cloud-elasticity +python3 cloud/arms_report.py --raw results/raw --lag-min 30 +``` + +A full rerun needs a ClickHouse Cloud API key, Docker and a `clickhouse` client, and this run spent $42.61 of credit. The steps are in [RECORD.md](RECORD.md#rerun). + +[RECORD.md](RECORD.md) has the run summary and rerun steps, and [RESULT.md](RESULT.md) every figure with the file it came from. diff --git a/clickhouse-cloud-elasticity/RECORD.md b/clickhouse-cloud-elasticity/RECORD.md new file mode 100644 index 0000000..0294eb4 --- /dev/null +++ b/clickhouse-cloud-elasticity/RECORD.md @@ -0,0 +1,37 @@ +# ClickHouse Cloud elasticity + +One benchmark in the [log10x benchmarks](../) repository. It asks how far a ClickHouse Cloud service's compute falls, in units and in billed dollars, when the rows written into its logs table fall, above the smallest size, and how long the autoscaler takes to lower it. + +**Result: from a 4-unit start, writing 0.653 of the rows took the service from 4 units to 3, a ratio of 0.75, and billed compute to 0.749, in each of three pairs; the step down was done 2.9 to 4.3 hours after the cut. From a 3-unit start the same cut read no change in 10.25 hours.** At 3 units both arms sit inside the autoscaler's no-action band; at 4 units arm B sits below it. Every figure, with the file and field it was read from, is in [RESULT.md](RESULT.md). + +ClickHouse Cloud, AWS us-east-1, Scale tier, ClickHouse 26.6, 2 replicas, vertical autoscaling 8 to 32 GiB per replica, idle after 5 minutes. The ClickStack `otel_logs` schema under seven queries a minute. Arm A writes every record of the 215 MB OTel replay at 70,000 rows/s; arm B writes the 65.3% the Receiver (`log10x/edge-10x:1.1.153`) kept, at 45,716 rows/s. 2026-11-02 19:48 to 2026-11-05 03:47 UTC. The service was deleted at the end (`results/raw/teardown.jsonl`). + +| Pair | Arm A's size | Units, A to B | Ratio | usageCost computeCHC per 15 minutes, A to B | Cut to step down done | +|---|---|---|---|---|---| +| 1 | the autoscaler's | 4.0 to 2.997 | 0.749 | 0.298 to 0.224 | 4.33 h | +| 2 | the autoscaler's, where pair 1 left it | 3.0 to 3.0 | 1.0 | 0.224 to 0.224 | no step in 10.25 h | +| 3 | 16 GiB by config, released at the cut | 4.0 to 3.0 | 0.750 | 0.298 to 0.224 | 4.21 h | +| 4 | 16 GiB by config, released at the cut | 4.0 to 3.0 | 0.750 | 0.298 to 0.224 | 2.91 h | + +## Rerun + +Needs a ClickHouse Cloud organization and API key, Docker, Python 3.12 or later and a `clickhouse` client binary (26.8 here). `cloud_api.py` reads the key from SSM or from `CHC_KEY_ID` and `CHC_KEY_SECRET` and never prints it. The scripts keep the service password and client config under `~/git/l1x-co/.claude-tmp/chce-secrets/` by default; `CHC_SECRET_FILE`, `CLIENT` and `CONF` move them. + +1. **Split.** `split/run_slices.sh` replays the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1` (checked by sha256) through four Receivers side by side and joins what they returned into `split/build/wire/returned.json`, in about an hour. +2. **Service.** Put the runner's address in `cloud/service.json`, then run `CHC_SECRET_FILE= python3 cloud_api.py create cloud/service.json`. The service password goes to that file (mode 600), not to the terminal. +3. **Tables.** Apply `cloud/schema.sql` with the client, then `cloud/stage.sh` and `cloud/stage_b.sh`. +4. **Meters**, left running, each with `--service ` and an `--out` under `results/raw/`: `cloud/sample.py`, `cloud/meter.py --every 60 --lag 30` and `cloud/usage_poll.py`. +5. **Feed and queries**, left running: `cloud/load.py` and `cloud/queries.py --overlap 3`. Both follow `results/raw/control.json`, which `cloud/phase.py` writes. +6. **Probe.** `cloud/probe.py` with steps of 2,000 to 70,000 rows/s, 10 minutes each. +7. **Arms.** `cloud/arms.py`. Its docstring gives both forms this run used: `--pairs` for the natural pairs, `--forced` for the pairs with arm A's size set by config, then the read calibration. +8. **Report.** `cloud/arms_report.py --raw results/raw --lag-min 30` for every phase, segment and pair; `cloud/curve.py` for the probe. Both read the committed `.jsonl.gz` files as they are. + +## Files + +| Path | What | +|---|---| +| `RESULT.md` | Every figure, the file and field it came from, and what was not measured | +| `split/` | The engine split | +| `cloud/` | The service request, schema, stage, feed, queries, meters, driver and reports | +| `results/` | The split and stage readings, the service creation response | +| `results/raw/` | Every log the scripts wrote: `events.jsonl` (phases and size changes), `scaling-patches.jsonl`, `usage.jsonl`, `teardown.jsonl`, and the meter, feed, sampler and query logs gzipped | diff --git a/clickhouse-inflate/README.md b/clickhouse-inflate/README.md index 84f631a..3a910db 100644 --- a/clickhouse-inflate/README.md +++ b/clickhouse-inflate/README.md @@ -1,176 +1,46 @@ # clickhouse-inflate -The harness behind the post *We published a ClickHouse benchmark that measured nothing*. +What does it cost ClickHouse to turn compact log events back into text at query time? Here the events sit in a table, and a view decodes them with SQL functions from the log10x ClickHouse app. With the SQL in `install-fixed.sql`, decoding all 137,418 rows took 548 ms and peaked at 41.1 MiB. The SQL in `install-buggy.sql` could not finish the same query at ClickHouse defaults and aborted at the server's memory cap. -It reproduces, on one command, both halves of that correction: +A `SELECT count()` over the view measures none of this, because ClickHouse answers it from a projection and prunes the decode out of the plan. The two SQL files differ by one line per function. The buggy lambda reaches its arrays by subscript, so ClickHouse copies each array once per mapped element. The fixed one passes them to arrayMap as arguments, which ClickHouse walks element by element. -1. the original benchmark timed `SELECT count()`, and ClickHouse pruned the decode out of the - plan: 1 row, 24 bytes, zero function nodes in the plan; -2. once the decode is actually forced, the shipped SQL dies of a quadratic lambda capture, and - the one-line fix takes the same query to 568 ms with a 41 MiB peak, byte-identical output. +## Result -Every number in the table below was produced by `./run.sh` on the host described under -[What this was measured on](#what-this-was-measured-on). Nothing here is copied from the post. +Figures are from results.json, with times averaged over 3 runs after discarding the first. -## Run it - -```bash -./run.sh # 1 minute end to end on a warm host -SKIP_PREFIX=1 ./run.sh # skip the two pre-fix out-of-memory probes -KEEP=1 ./run.sh # leave the container up to poke at it -``` - -Requires Docker. The script pulls a pinned ClickHouse image, starts a container with a pinned -memory limit, builds the corpus, loads it, installs the SHIPPED (buggy) SQL, probes it, installs -the FIXED SQL, probes that, diffs the two outputs byte for byte, writes `results.json`, and -removes the container. - -The two pre-fix probes run until they hit the memory cap and die. How long that takes varies with -memory pressure on the host: between 2 seconds and 10 minutes in our runs. `SKIP_PREFIX=1` skips -them; the pre-fix decode still runs once (under `max_block_size = 64`, where it survives) because -the byte-identity check needs its output. - -## Results - -Measured with `./run.sh`, ClickHouse 25.8.28.1, 8 GiB container, 8 CPUs, 137,418 rows. -Timings are the average of 3 runs with the first discarded, taken from `system.query_log`. -Full detail in [`results.json`](results.json). - -| What | Query | Observed | -|---|---|---| -| The pruned count | `SELECT count() FROM tenx.events` | 18 ms, **read_rows = 1, read_bytes = 24**, **0 function nodes in the plan** | -| Forced decode, PRE-fix | `SELECT sum(length(decoded_log)) FROM tenx.events` | **aborts**, code 241, peak `memory_usage` **7.20 GiB** | -| Forced decode, PRE-fix | `SELECT decoded_log FROM tenx.events FORMAT Null` | **aborts**, code 241, peak `memory_usage` **7.20 GiB** | -| Forced decode, PRE-fix, bounded settings (below) | `SELECT sum(length(decoded_log)) FROM tenx.events` | completes: **4,290 ms, 2.86 GiB peak, 10.6 MiB read** (3 of 3 runs) | -| Forced decode, POST-fix | `SELECT sum(length(decoded_log)) FROM tenx.events` | **568 ms, 41.1 MiB peak** | -| Forced decode, POST-fix | `SELECT decoded_log FROM tenx.events FORMAT Null` | **608 ms, 40.9 MiB peak** | -| Forced decode, POST-fix | same, `tenx.events_native` (format-preserving view) | 657 ms, 78.0 MiB peak | -| Output equality | pre-fix output vs post-fix output, all 137,418 rows | **byte-identical**, 29,065,810 bytes, same sha256 | -| The quadratic term | counted from the corpus and the dictionary | necessary **800,397** array elements, materialized **189,949,597** | -| The template that dominates it | | **818 literals, 275 rows, 96.9%** of the materialized total | - -Both pre-fix probes fail with: - -``` -Code: 241. DB::Exception: (total) memory limit exceeded: would use 7.49 GiB -(attempt to allocate chunk of 4.00 GiB bytes), current RSS: 3.10 GiB, maximum: 6.89 GiB -``` - -### Making the pre-fix decode finish - -The buggy path only completes if the block size is small enough that no single block carries enough -wide-template rows to blow the cap. These are the settings, and a reader has to paste them; there is -no pre-fix full-table decode at ClickHouse defaults: - -```sql -SELECT sum(length(decoded_log)) FROM tenx.events -SETTINGS max_block_size = 64, max_threads = 4, max_memory_usage = 6000000000; --- 4,290 ms, 2.86 GiB peak, 137,418 rows / 11,112,745 bytes read -``` +| Query | SQL file | Outcome | Time | Peak memory | +|---|---|---|---:|---:| +| `SELECT count()` | buggy | 1 row, 24 bytes read | 16 ms | not recorded | +| `sum(length(decoded_log))` | buggy | aborts, code 241 | n/a | 7.20 GiB | +| `sum(length(decoded_log))`, bounded settings | buggy | completes | 4,290 ms | 2.86 GiB | +| `sum(length(decoded_log))` | fixed | completes | 548 ms | 41.1 MiB | +| `SELECT decoded_log`, `FORMAT Null` | buggy | aborts, code 241 | n/a | 7.20 GiB | +| `SELECT decoded_log`, `FORMAT Null` | fixed | completes | 535 ms | 40.9 MiB | -The same query post-fix reads the same 11,112,745 bytes and runs in 548 ms at 41 MiB, at defaults. -The compact events a full decode reads are **10.6 MiB**, not 8.94 MiB. +The 7.20 GiB is above this host's 6.89 GiB cap because ClickHouse records the usage a refused 4 GiB allocation would have reached. The buggy decode completed in 3 of 3 runs with a block size of 64, 4 threads and a per-query limit of 6000000000 bytes. -### Read the abort threshold correctly +- Both files give byte-identical output over all 137,418 rows, 29,065,810 bytes. +- The decode needs 800,397 array elements and the buggy SQL materializes 189,949,597. One template with 818 literals, on 275 rows, accounts for 96.9% of the materialized elements. +- Replacing the functions alone leaves the views on the old body, because ClickHouse copies a function body into a view when it creates the view. The fixed install file recreates the views. -The 7.20 GiB is the peak `memory_usage` ClickHouse recorded for the query at the moment it gave -up. It is not a constant of the bug, and it is not the memory cap. Two separate numbers: +## How it was measured -- **The cap.** ClickHouse sets `max_server_memory_usage` to - `max_server_memory_usage_to_ram_ratio` (default **0.9**) times the RAM it can see, and it takes - the RAM it can see as the smaller of the cgroup limit and the host's `MemTotal`. `run.sh` pins - the container to 8 GiB, so on any host with at least 8 GiB of RAM the cap is 0.9 x 8 GiB = - **7.2 GiB**. On the Docker VM used here (`MemTotal` 7.66 GiB, below the 8 GiB cgroup limit) - ClickHouse capped itself at **6.89 GiB** instead. `run.sh` prints the cap it actually got and - records it in `results.json`. -- **The peak.** The decode asks for a 4 GiB chunk on top of roughly 3.1 GiB already tracked, so - the tracker records about 7.2 GiB of would-be usage and throws. That is the 7.20 GiB in the - table. +ClickHouse 25.8.28.1 ran in Docker, pinned to 8 GiB and 8 CPUs. The data is 500 compact events the 10x engine wrote from the OpenTelemetry demo's Kubernetes logs, cycled to 137,418 rows. The script runs every query on the buggy SQL, then the decodes on the fixed SQL, and compares the outputs byte for byte. Times and peak memory come from ClickHouse's query log. -The transferable statement is not "it dies at 7.2 GiB". It is: **the pre-fix decode's demand grows -with the square of the template width, so it exceeds whatever cap ClickHouse is running under.** -Without a pinned container it would abort at 0.9 x the machine's RAM, wherever that lands. Pinning -`-m 8g` is what makes the failure identical for every reader. +## Limits -### Decode nodes in the plan +- The test ran on one host with one ClickHouse version. Times move with hardware, and the abort point moves with the memory cap, which ClickHouse sets at 0.9 times the RAM it can see. +- The data set is small, 500 events cycled to fill the table. +- The byte-identity check compares the two files' output against each other, not against the original log text. -The post's "zero decode nodes" for the pruned count reproduces in the strongest possible form: the -`SELECT count()` plan contains **no function nodes at all** and reads from -`_minmax_count_projection`. `run.sh` writes the plans to `.work/explain-*.txt`. - -The forced-decode plan contains **27 function nodes** pre-fix and **25** post-fix (`EXPLAIN -actions = 1`, counting `FUNCTION` entries in the action DAG). Of those, 5 pre-fix and 8 post-fix -are the decode builtins (`dictGetOrDefault`, `arrayMap`, `arrayStringConcat`, `splitByChar`, -`arrayResize`). The figure of 14 that appears in the post matches neither count on this version and -should not be relied on; the load-bearing claim is that the count plan has none of them. - -## What this was measured on - -- Image `clickhouse/clickhouse-server:25.8.28.1`, pinned. -- Container: `-m 8g --cpus 8`. Docker Desktop VM with `MemTotal` 7.66 GiB, so ClickHouse's own cap - came out at 6.89 GiB (see above). -- Query settings: ClickHouse defaults. `max_memory_usage = 0` (per-query limit off, so the server - cap governs), `max_block_size = 65409`, `max_threads = auto(8)`. The one exception is the pre-fix - reference decode used for the byte-identity check, which runs at `max_block_size = 64` because - that is the only way the buggy path finishes at all. - -Absolute timings will move with hardware. The shapes will not: the count reads 24 bytes, the pre-fix -decode exceeds the cap, the post-fix decode runs in hundreds of milliseconds at tens of MiB. - -## The corpus - -`corpus/sample-500.jsonl` is 500 compacted OpenTelemetry-demo Kubernetes log events (the fluentd -JSON envelope, with the log line replaced by `~,`), and -`corpus/templates.json` is the 378 templates they decode against. Both are the sample from -log-10x/clickhouse-app under `demo/sample/`, produced -by the 10x engine from the OpenTelemetry demo's container logs. Checksums in `corpus/SHA256SUMS`. - -`run.sh` cycles that sample to **137,418 rows**, the row count of the table in the post. Cycling -preserves the template mix, so the quadratic term lands exactly where the post says it does: 275 of -the rows carry the 818-literal template (a 21 KB multi-line container startup dump the engine folds -into one event), and that one template accounts for 96.9% of the materialized array elements. - -## The SQL - -| File | What it is | -|---|---| -| `install-buggy.sql` | `tenx-for-clickhouse/install.sql` as it shipped, verbatim from `main` of log-10x/clickhouse-app (`de67ad6`). The decode lambda indexes into its arrays. | -| `install-fixed.sql` | The same file with the fix, verbatim from the `fix/decode-quadratic-capture` branch (`39e0206`). The arrays are passed as `arrayMap` arguments. | -| `fix-functions-only.sql` | Only the two corrected core functions, used to demonstrate that replacing a function does not fix a view built on it. | - -The whole fix is one line per function. Before: +## Run it -```sql -arrayMap(i -> concat(literals[i], ...), range(1, length(literals))) ``` - -The lambda reaches `literals`, `slots` and `values` by subscript, so ClickHouse captures those -columns and replicates each of them once per mapped element: a row with N literals materializes N -copies of its N-element arrays. After: - -```sql -arrayMap((lit, slot, val) -> concat(lit, tenx_substitute_slot(val, slot)), - literals, - arrayResize(slots, length(literals), ''), - arrayResize(arrayResize(values, length(slots), ''), length(literals), '')) +cd clickhouse-inflate +./run.sh # pulls the pinned ClickHouse image and writes results.json +SKIP_PREFIX=1 ./run.sh # skips the two buggy-SQL queries that abort ``` -Arrays passed as arguments are walked element-wise, never replicated. - -`run.sh` also checks the second trap the post describes: after `CREATE OR REPLACE FUNCTION` alone, -`SHOW CREATE VIEW tenx.events` still contains the old body (ClickHouse expands function bodies into -a view's stored AST at `CREATE VIEW` time), and the blow-up persists until the views are recreated. -`results.json` records that check as `view_inlining_trap`. - -## What this harness does not cover +A run needs Docker. The two aborting queries take 2 seconds to 10 minutes, depending on host memory pressure. -- The post reports byte-identity on **224,024 rows**, which is this table plus a second, wider - corpus that is not published. This harness verifies byte-identity on the **137,418 rows** it - builds, in full, and prints the sha256 of both outputs. -- The post's "wider stress corpus" figures (86,606 rows, 74 slots per row, 6.15 GiB / 19.9 s before - and 71.9 MiB / 1.5 s after) come from that same unpublished corpus and are not reproduced here. -- The post reports 7,495 ms with a 3.65 GiB peak for the pre-fix `sum(length(decoded_log))`, and - 8.94 MiB read. None of the three reproduce. At ClickHouse defaults that query does not complete - at all, it aborts. At the bounded settings where it does complete - (`max_block_size = 64, max_threads = 4, max_memory_usage = 6000000000`) it takes **4,290 ms** with - a **2.86 GiB** peak and reads **10.6 MiB**. +The full record is in [RECORD.md](RECORD.md): the abort message, the memory cap, plan node counts, the SQL change and the figures this harness does not reproduce. diff --git a/clickhouse-inflate/RECORD.md b/clickhouse-inflate/RECORD.md new file mode 100644 index 0000000..84f631a --- /dev/null +++ b/clickhouse-inflate/RECORD.md @@ -0,0 +1,176 @@ +# clickhouse-inflate + +The harness behind the post *We published a ClickHouse benchmark that measured nothing*. + +It reproduces, on one command, both halves of that correction: + +1. the original benchmark timed `SELECT count()`, and ClickHouse pruned the decode out of the + plan: 1 row, 24 bytes, zero function nodes in the plan; +2. once the decode is actually forced, the shipped SQL dies of a quadratic lambda capture, and + the one-line fix takes the same query to 568 ms with a 41 MiB peak, byte-identical output. + +Every number in the table below was produced by `./run.sh` on the host described under +[What this was measured on](#what-this-was-measured-on). Nothing here is copied from the post. + +## Run it + +```bash +./run.sh # 1 minute end to end on a warm host +SKIP_PREFIX=1 ./run.sh # skip the two pre-fix out-of-memory probes +KEEP=1 ./run.sh # leave the container up to poke at it +``` + +Requires Docker. The script pulls a pinned ClickHouse image, starts a container with a pinned +memory limit, builds the corpus, loads it, installs the SHIPPED (buggy) SQL, probes it, installs +the FIXED SQL, probes that, diffs the two outputs byte for byte, writes `results.json`, and +removes the container. + +The two pre-fix probes run until they hit the memory cap and die. How long that takes varies with +memory pressure on the host: between 2 seconds and 10 minutes in our runs. `SKIP_PREFIX=1` skips +them; the pre-fix decode still runs once (under `max_block_size = 64`, where it survives) because +the byte-identity check needs its output. + +## Results + +Measured with `./run.sh`, ClickHouse 25.8.28.1, 8 GiB container, 8 CPUs, 137,418 rows. +Timings are the average of 3 runs with the first discarded, taken from `system.query_log`. +Full detail in [`results.json`](results.json). + +| What | Query | Observed | +|---|---|---| +| The pruned count | `SELECT count() FROM tenx.events` | 18 ms, **read_rows = 1, read_bytes = 24**, **0 function nodes in the plan** | +| Forced decode, PRE-fix | `SELECT sum(length(decoded_log)) FROM tenx.events` | **aborts**, code 241, peak `memory_usage` **7.20 GiB** | +| Forced decode, PRE-fix | `SELECT decoded_log FROM tenx.events FORMAT Null` | **aborts**, code 241, peak `memory_usage` **7.20 GiB** | +| Forced decode, PRE-fix, bounded settings (below) | `SELECT sum(length(decoded_log)) FROM tenx.events` | completes: **4,290 ms, 2.86 GiB peak, 10.6 MiB read** (3 of 3 runs) | +| Forced decode, POST-fix | `SELECT sum(length(decoded_log)) FROM tenx.events` | **568 ms, 41.1 MiB peak** | +| Forced decode, POST-fix | `SELECT decoded_log FROM tenx.events FORMAT Null` | **608 ms, 40.9 MiB peak** | +| Forced decode, POST-fix | same, `tenx.events_native` (format-preserving view) | 657 ms, 78.0 MiB peak | +| Output equality | pre-fix output vs post-fix output, all 137,418 rows | **byte-identical**, 29,065,810 bytes, same sha256 | +| The quadratic term | counted from the corpus and the dictionary | necessary **800,397** array elements, materialized **189,949,597** | +| The template that dominates it | | **818 literals, 275 rows, 96.9%** of the materialized total | + +Both pre-fix probes fail with: + +``` +Code: 241. DB::Exception: (total) memory limit exceeded: would use 7.49 GiB +(attempt to allocate chunk of 4.00 GiB bytes), current RSS: 3.10 GiB, maximum: 6.89 GiB +``` + +### Making the pre-fix decode finish + +The buggy path only completes if the block size is small enough that no single block carries enough +wide-template rows to blow the cap. These are the settings, and a reader has to paste them; there is +no pre-fix full-table decode at ClickHouse defaults: + +```sql +SELECT sum(length(decoded_log)) FROM tenx.events +SETTINGS max_block_size = 64, max_threads = 4, max_memory_usage = 6000000000; +-- 4,290 ms, 2.86 GiB peak, 137,418 rows / 11,112,745 bytes read +``` + +The same query post-fix reads the same 11,112,745 bytes and runs in 548 ms at 41 MiB, at defaults. +The compact events a full decode reads are **10.6 MiB**, not 8.94 MiB. + +### Read the abort threshold correctly + +The 7.20 GiB is the peak `memory_usage` ClickHouse recorded for the query at the moment it gave +up. It is not a constant of the bug, and it is not the memory cap. Two separate numbers: + +- **The cap.** ClickHouse sets `max_server_memory_usage` to + `max_server_memory_usage_to_ram_ratio` (default **0.9**) times the RAM it can see, and it takes + the RAM it can see as the smaller of the cgroup limit and the host's `MemTotal`. `run.sh` pins + the container to 8 GiB, so on any host with at least 8 GiB of RAM the cap is 0.9 x 8 GiB = + **7.2 GiB**. On the Docker VM used here (`MemTotal` 7.66 GiB, below the 8 GiB cgroup limit) + ClickHouse capped itself at **6.89 GiB** instead. `run.sh` prints the cap it actually got and + records it in `results.json`. +- **The peak.** The decode asks for a 4 GiB chunk on top of roughly 3.1 GiB already tracked, so + the tracker records about 7.2 GiB of would-be usage and throws. That is the 7.20 GiB in the + table. + +The transferable statement is not "it dies at 7.2 GiB". It is: **the pre-fix decode's demand grows +with the square of the template width, so it exceeds whatever cap ClickHouse is running under.** +Without a pinned container it would abort at 0.9 x the machine's RAM, wherever that lands. Pinning +`-m 8g` is what makes the failure identical for every reader. + +### Decode nodes in the plan + +The post's "zero decode nodes" for the pruned count reproduces in the strongest possible form: the +`SELECT count()` plan contains **no function nodes at all** and reads from +`_minmax_count_projection`. `run.sh` writes the plans to `.work/explain-*.txt`. + +The forced-decode plan contains **27 function nodes** pre-fix and **25** post-fix (`EXPLAIN +actions = 1`, counting `FUNCTION` entries in the action DAG). Of those, 5 pre-fix and 8 post-fix +are the decode builtins (`dictGetOrDefault`, `arrayMap`, `arrayStringConcat`, `splitByChar`, +`arrayResize`). The figure of 14 that appears in the post matches neither count on this version and +should not be relied on; the load-bearing claim is that the count plan has none of them. + +## What this was measured on + +- Image `clickhouse/clickhouse-server:25.8.28.1`, pinned. +- Container: `-m 8g --cpus 8`. Docker Desktop VM with `MemTotal` 7.66 GiB, so ClickHouse's own cap + came out at 6.89 GiB (see above). +- Query settings: ClickHouse defaults. `max_memory_usage = 0` (per-query limit off, so the server + cap governs), `max_block_size = 65409`, `max_threads = auto(8)`. The one exception is the pre-fix + reference decode used for the byte-identity check, which runs at `max_block_size = 64` because + that is the only way the buggy path finishes at all. + +Absolute timings will move with hardware. The shapes will not: the count reads 24 bytes, the pre-fix +decode exceeds the cap, the post-fix decode runs in hundreds of milliseconds at tens of MiB. + +## The corpus + +`corpus/sample-500.jsonl` is 500 compacted OpenTelemetry-demo Kubernetes log events (the fluentd +JSON envelope, with the log line replaced by `~,`), and +`corpus/templates.json` is the 378 templates they decode against. Both are the sample from +log-10x/clickhouse-app under `demo/sample/`, produced +by the 10x engine from the OpenTelemetry demo's container logs. Checksums in `corpus/SHA256SUMS`. + +`run.sh` cycles that sample to **137,418 rows**, the row count of the table in the post. Cycling +preserves the template mix, so the quadratic term lands exactly where the post says it does: 275 of +the rows carry the 818-literal template (a 21 KB multi-line container startup dump the engine folds +into one event), and that one template accounts for 96.9% of the materialized array elements. + +## The SQL + +| File | What it is | +|---|---| +| `install-buggy.sql` | `tenx-for-clickhouse/install.sql` as it shipped, verbatim from `main` of log-10x/clickhouse-app (`de67ad6`). The decode lambda indexes into its arrays. | +| `install-fixed.sql` | The same file with the fix, verbatim from the `fix/decode-quadratic-capture` branch (`39e0206`). The arrays are passed as `arrayMap` arguments. | +| `fix-functions-only.sql` | Only the two corrected core functions, used to demonstrate that replacing a function does not fix a view built on it. | + +The whole fix is one line per function. Before: + +```sql +arrayMap(i -> concat(literals[i], ...), range(1, length(literals))) +``` + +The lambda reaches `literals`, `slots` and `values` by subscript, so ClickHouse captures those +columns and replicates each of them once per mapped element: a row with N literals materializes N +copies of its N-element arrays. After: + +```sql +arrayMap((lit, slot, val) -> concat(lit, tenx_substitute_slot(val, slot)), + literals, + arrayResize(slots, length(literals), ''), + arrayResize(arrayResize(values, length(slots), ''), length(literals), '')) +``` + +Arrays passed as arguments are walked element-wise, never replicated. + +`run.sh` also checks the second trap the post describes: after `CREATE OR REPLACE FUNCTION` alone, +`SHOW CREATE VIEW tenx.events` still contains the old body (ClickHouse expands function bodies into +a view's stored AST at `CREATE VIEW` time), and the blow-up persists until the views are recreated. +`results.json` records that check as `view_inlining_trap`. + +## What this harness does not cover + +- The post reports byte-identity on **224,024 rows**, which is this table plus a second, wider + corpus that is not published. This harness verifies byte-identity on the **137,418 rows** it + builds, in full, and prints the sha256 of both outputs. +- The post's "wider stress corpus" figures (86,606 rows, 74 slots per row, 6.15 GiB / 19.9 s before + and 71.9 MiB / 1.5 s after) come from that same unpublished corpus and are not reproduced here. +- The post reports 7,495 ms with a 3.65 GiB peak for the pre-fix `sum(length(decoded_log))`, and + 8.94 MiB read. None of the three reproduce. At ClickHouse defaults that query does not complete + at all, it aborts. At the bounded settings where it does complete + (`max_block_size = 64, max_threads = 4, max_memory_usage = 6000000000`) it takes **4,290 ms** with + a **2.86 GiB** peak and reads **10.6 MiB**. diff --git a/clickstack-e2e-gaps/README.md b/clickstack-e2e-gaps/README.md index 29c73d2..a92a180 100644 --- a/clickstack-e2e-gaps/README.md +++ b/clickstack-e2e-gaps/README.md @@ -1,127 +1,47 @@ # clickstack-e2e-gaps -Ten scripts over the questions the end to end harness next door left open, each -run against the whole 197,430 line capture rather than the 50,000 line slice. -Six are the first pass, one per gap; three more were added in a second pass after -a record check against primary sources; the tenth was added in a third pass, on -the layout the first one builds. +What changes in ClickStack when part of the logs moves to S3 and a Merge table reads both sides? Counts through the Merge table matched the counts from before the move, while the hot table alone gave a wrong count with no error. Queries that name only a time opened all 180 objects over 30 days. On ClickHouse compute, a plain collector rule by service and severity was cheaper than the 10x per-type policy. -`../clickstack-e2e` is the route: a released capture through an OpenTelemetry -Collector into the 10x receiver, the marked slice into an object store, the rest -into ClickStack's own ClickHouse exporter, both sides read back as one table. -This folder keeps that route and asks the questions that run could not answer. +The 10x engine keeps the first lines of each message type in a window hot and marks the rest, which the collector writes to S3 under a service and day path. ClickHouse can skip an object unopened only on those path segments, since the timestamp and the message type live inside the file. -## Run it - -```bash -TENX_CONFIG_TREE= \ -TENX_MODULES_TREE= \ -CSE_DATA_DIR=../clickstack-e2e/data \ -./gap1_multiday_pruning.sh -``` - -Every script takes the same environment, `lib.sh` documents it, and `LINES` -shortens the feed for a smoke run. Run `../clickstack-e2e/run.sh` once first so -the capture is in `data/` with its sha256 checked; these scripts refuse to run -against a capture that does not match. +## Result -The receiver is `ghcr.io/log-10x/edge-10x:1.1.79`, public and pinned by digest -in `lib.sh`, so nothing here needs a build and nothing here needs a private -repository. `log-10x/config` and `log-10x/modules` are public checkouts too. -Every other image is pinned by digest beside it, and the digests that ran are -recorded in the results file. +From several runs over one 197,430-line capture. -`PATCHED_JAR` remains as an optional override: set it and the receiver runs that -run-cloud shadow jar on a stock JRE image instead of the published binary, the -way `../clickstack-e2e/repro-otel-return/verify-patched.sh` does. That override -is how the runs in `results/clickstack-e2e-gaps-2026-09-15.md` were made. When -those runs were made the shipped `edge-10x` image was a native binary carrying -the three OpenTelemetry return path defects, and with those defects half the -capture comes back unmarked, which would have decided most of these questions -before they were asked. Both fixes are released as 1.1.79, and -`../clickstack-e2e/results/clickstack-e2e-2026-09-15.md` is that image's own run. - -The receiver each results file ran: - -| Results | Receiver | +| Measurement | Result | |---|---| -| `clickstack-e2e-gaps-2026-09-15.md`, `gap1.json` to `gap6.json` | the run-cloud shadow jar on `eclipse-temurin:23-jre`, through `PATCHED_JAR` | -| `clickstack-e2e-close-2026-09-15.md`, `gap1-agg.json`, `gap2b.json`, `gap4-router-owners.json`, `gap4-router-persist.json`, `gap7.json`, `gap8.json`, `gap9.json` | `ghcr.io/log-10x/edge-10x:1.1.79`, the `lib.sh` pin | -| `gap4-custody-before-edge-1.1.147.json` | `ghcr.io/log-10x/edge-10x:1.1.147`, through `CSE_ENGINE_IMAGE` | -| `gap4-custody-after-1.1.150-run1.json`, `gap4-custody-after-1.1.150-run2.json` | the engine 1.1.150 run-cloud jar on `eclipse-temurin:23-jre`, through `PATCHED_JAR` | +| Time-only query over 30 days | 180 GETs, 7,449 ms | +| Same query plus a day predicate | 12 GETs, 468 ms | +| LIMIT 100 through the Merge table | 180 GETs, 5,090 ms | +| Same LIMIT 100, hot table alone | 0 GETs, 7 ms | +| One type's count, hot table alone | 3,688 (36,395 before offload) | +| Same count through the Merge table | 36,395 | +| Insert plus merge CPU, seconds | none 10.1, collector 2.7, per-type 8.1 | +| Lines lost, engine acks after writing | 0 of 197,430 | -A host needs about 7 GB of Docker memory and 3 GB of free disk per run, and -nothing else may be running a second ClickStack: two of them on one Docker -daemon starve each other and the first symptom is a container that dies mid-feed -with no error in the run's own log. +Both LIMIT 100 queries returned the same hundred rows, all of them hot. The collector rule used the least CPU, but its rows carry no message type to count by. -## What each script proves, and what it does not +- With the engine acknowledging only after its lines are written, killing the collector, the engine or both lost no line in two runs and stored at most 997 lines twice. Acknowledging on receipt lost 24,900 to 32,203 lines under the same kills. +- An S3 table with hive partitioning that is read once while its prefix is empty matches nothing on a path predicate, while `count()` looks right (ClickHouse issue 116888). Creating it after the first object exists avoids this. -| Script | Proves | Does not | -|---|---|---| -| `gap1_multiday_pruning.sh` | what a day predicate costs against what a time-only predicate costs, over thirty days of objects, counted in S3 GET and LIST | nothing about a real day's mix: every day is a replica of the run's own cold rows with the timestamps shifted, and the replicated objects are gzipped and written by ClickHouse, not by the collector | -| `gap2_vector_parquet.sh` | that Vector carries the whole cold branch, and what the same queries cost over Vector's objects and over a Parquet copy of the same rows | anything about Parquet in Vector as it was first run: the pinned build was `0.50.0-debian` and the probe set `encoding.codec`, which is the per event serializer. Vector's Parquet option is `batch_encoding.codec` on the `aws_s3` sink, added in v0.55.0 and in official builds from v0.56.0. The Parquet objects in that query table were written by ClickHouse | -| `gap3_retriever_fetch.sh` | whether the cold layout as written is indexable by the shipped Retriever index pipeline, against the same pipeline over the same records one per line | nothing about the Retriever's deployed shape: no SQS, no S3 trigger, no Quarkus service, no fan out. The local accessor the engine ships is what runs | -| `gap4_durable_handoff.sh` | whether a line can be lost or doubled when the routing collector, the receiver, or both are killed mid-stream and restarted, with the collector set up the way the shipped OpenTelemetry Collector recipe sets it up, counted line by line through a sequence number inside the body. The run fails when any arm stores a line nowhere; duplicates are reported beside it | nothing about a crash that loses the collector's checkpoint file or the host, nothing about a partial object in the store, and nothing about the recipe's multi-line joining, which the numbered capture cannot exercise | -| `gap5_query_alert_correctness.sh` | which of three dashboard-shaped answers change when rows leave the hot table, across the hot table before, the hot table after, the Merge table and the counts table, and whether HyperDX's API accepts an alert on each | that HyperDX's own evaluation loop fired: an alert evaluates the window its interval names, and a replayed capture is older than that by the time the loop runs. The firing condition is evaluated against the counts each surface returns and is labelled as such | -| `gap6_incremental_economics.sh` | rows, bytes, insert plus merge CPU and answers for three policies on one ClickHouse: no offload, a collector-only policy by service and severity, and the per-type policy | nothing about money. There is no Cloud autoscaler here, no bill, and no compute term; the arms are comparable to each other and to nothing else | -| `gap2b_vector058_parquet.sh` | what Vector 0.58.0 takes for Parquet, and what the same rows cost as newline delimited JSON and as Parquet, both written by Vector in one run and read through two Merge tables, with the Parquet reader's row group and page counters | that the two containers hold the same shape: the JSON arm writes a free attribute map and the Parquet arm writes named columns, because a schema inferred from a free map carries one field per key seen | -| `gap7_s3_table_before_objects.sh` | ClickHouse issue 116888: what an S3 table with an explicit schema and `use_hive_partitioning` answers when created before any object exists, what the same DDL answers when created after, and whether DETACH and ATTACH recovers the first | anything about how long the empty listing is held, or which other engines cache one. One table, one prefix, one server version, named in the results | -| `gap8_ttl_to_s3.sh` | what den-crane's TTL recipe costs on the same capture: the PUTs, the parts and the merge and move CPU a forced `TTL TO VOLUME 'cold'` charges, and what a slice stamped behind the boundary charges on insert alone | nothing about a bill, and nothing about the background scheduler's own pacing: the move is forced so the whole table moves exactly once and the number does not depend on which parts the scheduler happened to pick | -| `gap9_bounded_window.sh` | what a bound inside the table costs against the same bound written into the query: a view over the cold table carrying `day >= today() - 7`, a second Merge table over hot plus that view, and the two aggregation shapes put to both | nothing about an estate's own ratio of window to retention, and nothing about the rows outside the bound, which that table cannot reach at all | +## How it was measured -## Files +The route is the one in [clickstack-e2e](../clickstack-e2e/), with a local S3-compatible store, on one laptop with 8 GB of Docker memory. For the 30-day queries, one run's cold objects were copied into 30 day partitions, six objects a day. GETs and CPU come from ClickHouse's query and part logs, and lost lines are counted by a sequence number inside every line. The collector rule sends the two busiest services' INFO and DEBUG lines to S3. -| File | What it is | -|---|---| -| `lib.sh` | the compose pattern every script shares, plus `g_cold30_layout`, gap 1's thirty day object layout, shared so gap 9 asks its question of the same one: the pinned images, the network, MinIO, ClickStack, the receiver and the routing collector, plus `g_wait_for_object`, the guard for ClickHouse 116888 | -| `measure.py` | one query, timed on a cold cache, read back out of `system.query_log` with its S3 counters | -| `wire_hashes.py` | the census of what the receiver returned, from the collector's wire tap | -| `make_seq_input.py` | the capture with a sequence number inside every line, for gap 4 | -| `patch_retriever_config.py` | the asserted edits that point the shipped Retriever config at the local file system | -| `hyperdx_alert.py` | the alert attempt over the HyperDX API, every status recorded | -| `render.py` | composes `results/clickstack-e2e-gaps-.md` from the per-gap JSON and the hand-written notes | -| `notes/` | the prose for each gap. The numbers are not in here | -| `results/` | `gap.json` per script, the composed first-pass file, and the second pass's own file | +## Limits -## Reading the results files +- Each setup ran once or a few times. Timings compare only with each other. Merge CPU and doubled lines vary by run. +- The 30 days are copies of one day's cold rows, so request counts are real but the content is not a real month. +- Nothing here measures money, HyperDX's own alert evaluation, or fetching cold objects back through 10x. -`results/clickstack-e2e-gaps-2026-09-15.md` is the first pass, one table per gap. -Every number in it comes from `results/gap.json`, and `render.py` composes it -from those files and the prose in `notes/`. Where a gap is open, its section says -so and says what would close it. - -`results/clickstack-e2e-close-2026-09-15.md` is the second pass, written after a -record check against primary sources found one published claim false and four -questions the first pass never put. Read it for the Vector Parquet retraction, -the Parquet reader's own counters, the aggregation shapes through the Merge -table, the collector run as its code owners prescribe, ClickHouse issue 116888 -and its guard, and what `TTL TO VOLUME 'cold'` costs on the same capture. That file -carries the digests every run used and the command that produced each table, and -its own JSON sits beside the first pass's: `gap1-agg.json`, `gap2b.json`, -`gap4-router-owners.json`, `gap7.json`, `gap8.json`. - -The same file's last section, `Wave 3, 2026-09-15 evening`, is a third pass on the -same harness: the persistent sending queue arm of gap 4 (`gap4-router-persist.json`), -the bounded cold window (`gap9.json`), and what the public Retriever image carries, -which is why gap 3 is still not measured. - -Durable handoff with custody, 2026-10-23. The collector runs the shipped recipe's settings -(file positions and the queue into the receiver in `file_storage`, one sender, retries -without a deadline, batching inside the persisted queue, the synchronous file -emitter), on contrib 0.160.0 and RustFS. Lines stored nowhere, lines stored twice, -and lines the receiver completed without an output (`input_lines_abandoned`), out of -197,430: +## Run it -| Arm | receiver answers on receipt | receiver answers once lines are done, run 1 | receiver answers once lines are done, run 2 | -|---|---|---|---| -| collector killed | 32,203 missing, 0 twice | 0 missing, 900 twice, 0 abandoned | 0 missing, 997 twice, 0 abandoned | -| receiver killed | 31,976 missing, 0 twice | 0 missing, 0 twice, 0 abandoned | 0 missing, 0 twice, 0 abandoned | -| both killed | 24,900 missing, 0 twice | 0 missing, 232 twice, 0 abandoned | 0 missing, 0 twice, 0 abandoned | +``` +cd clickstack-e2e-gaps +../clickstack-e2e/run.sh # once: the base run; fetches the capture, checks its sha256 +./gap1_multiday_pruning.sh # or any gap*.sh; LINES=20000 for a short run +``` -The custody columns run a receiver that answers an export only once its lines are written -back and acknowledged (the run-cloud jar on the JRE image, with -`PATCHED_JAR`). JSON: `gap4-custody-before-edge-1.1.147.json`, -`gap4-custody-after-1.1.150-run1.json`, `gap4-custody-after-1.1.150-run2.json`. +Each script needs Docker with about 7 GB of memory and 3 GB of disk. The acknowledge-after-write runs used engine 1.1.150 through `PATCHED_JAR` (see `lib.sh`). -Where the two disagree, the second is the later measurement and says so. +[RECORD.md](RECORD.md) has the full record: what each script measures and leaves out, the build each run used, and a guide to [results/](results/). diff --git a/clickstack-e2e-gaps/RECORD.md b/clickstack-e2e-gaps/RECORD.md new file mode 100644 index 0000000..29c73d2 --- /dev/null +++ b/clickstack-e2e-gaps/RECORD.md @@ -0,0 +1,127 @@ +# clickstack-e2e-gaps + +Ten scripts over the questions the end to end harness next door left open, each +run against the whole 197,430 line capture rather than the 50,000 line slice. +Six are the first pass, one per gap; three more were added in a second pass after +a record check against primary sources; the tenth was added in a third pass, on +the layout the first one builds. + +`../clickstack-e2e` is the route: a released capture through an OpenTelemetry +Collector into the 10x receiver, the marked slice into an object store, the rest +into ClickStack's own ClickHouse exporter, both sides read back as one table. +This folder keeps that route and asks the questions that run could not answer. + +## Run it + +```bash +TENX_CONFIG_TREE= \ +TENX_MODULES_TREE= \ +CSE_DATA_DIR=../clickstack-e2e/data \ +./gap1_multiday_pruning.sh +``` + +Every script takes the same environment, `lib.sh` documents it, and `LINES` +shortens the feed for a smoke run. Run `../clickstack-e2e/run.sh` once first so +the capture is in `data/` with its sha256 checked; these scripts refuse to run +against a capture that does not match. + +The receiver is `ghcr.io/log-10x/edge-10x:1.1.79`, public and pinned by digest +in `lib.sh`, so nothing here needs a build and nothing here needs a private +repository. `log-10x/config` and `log-10x/modules` are public checkouts too. +Every other image is pinned by digest beside it, and the digests that ran are +recorded in the results file. + +`PATCHED_JAR` remains as an optional override: set it and the receiver runs that +run-cloud shadow jar on a stock JRE image instead of the published binary, the +way `../clickstack-e2e/repro-otel-return/verify-patched.sh` does. That override +is how the runs in `results/clickstack-e2e-gaps-2026-09-15.md` were made. When +those runs were made the shipped `edge-10x` image was a native binary carrying +the three OpenTelemetry return path defects, and with those defects half the +capture comes back unmarked, which would have decided most of these questions +before they were asked. Both fixes are released as 1.1.79, and +`../clickstack-e2e/results/clickstack-e2e-2026-09-15.md` is that image's own run. + +The receiver each results file ran: + +| Results | Receiver | +|---|---| +| `clickstack-e2e-gaps-2026-09-15.md`, `gap1.json` to `gap6.json` | the run-cloud shadow jar on `eclipse-temurin:23-jre`, through `PATCHED_JAR` | +| `clickstack-e2e-close-2026-09-15.md`, `gap1-agg.json`, `gap2b.json`, `gap4-router-owners.json`, `gap4-router-persist.json`, `gap7.json`, `gap8.json`, `gap9.json` | `ghcr.io/log-10x/edge-10x:1.1.79`, the `lib.sh` pin | +| `gap4-custody-before-edge-1.1.147.json` | `ghcr.io/log-10x/edge-10x:1.1.147`, through `CSE_ENGINE_IMAGE` | +| `gap4-custody-after-1.1.150-run1.json`, `gap4-custody-after-1.1.150-run2.json` | the engine 1.1.150 run-cloud jar on `eclipse-temurin:23-jre`, through `PATCHED_JAR` | + +A host needs about 7 GB of Docker memory and 3 GB of free disk per run, and +nothing else may be running a second ClickStack: two of them on one Docker +daemon starve each other and the first symptom is a container that dies mid-feed +with no error in the run's own log. + +## What each script proves, and what it does not + +| Script | Proves | Does not | +|---|---|---| +| `gap1_multiday_pruning.sh` | what a day predicate costs against what a time-only predicate costs, over thirty days of objects, counted in S3 GET and LIST | nothing about a real day's mix: every day is a replica of the run's own cold rows with the timestamps shifted, and the replicated objects are gzipped and written by ClickHouse, not by the collector | +| `gap2_vector_parquet.sh` | that Vector carries the whole cold branch, and what the same queries cost over Vector's objects and over a Parquet copy of the same rows | anything about Parquet in Vector as it was first run: the pinned build was `0.50.0-debian` and the probe set `encoding.codec`, which is the per event serializer. Vector's Parquet option is `batch_encoding.codec` on the `aws_s3` sink, added in v0.55.0 and in official builds from v0.56.0. The Parquet objects in that query table were written by ClickHouse | +| `gap3_retriever_fetch.sh` | whether the cold layout as written is indexable by the shipped Retriever index pipeline, against the same pipeline over the same records one per line | nothing about the Retriever's deployed shape: no SQS, no S3 trigger, no Quarkus service, no fan out. The local accessor the engine ships is what runs | +| `gap4_durable_handoff.sh` | whether a line can be lost or doubled when the routing collector, the receiver, or both are killed mid-stream and restarted, with the collector set up the way the shipped OpenTelemetry Collector recipe sets it up, counted line by line through a sequence number inside the body. The run fails when any arm stores a line nowhere; duplicates are reported beside it | nothing about a crash that loses the collector's checkpoint file or the host, nothing about a partial object in the store, and nothing about the recipe's multi-line joining, which the numbered capture cannot exercise | +| `gap5_query_alert_correctness.sh` | which of three dashboard-shaped answers change when rows leave the hot table, across the hot table before, the hot table after, the Merge table and the counts table, and whether HyperDX's API accepts an alert on each | that HyperDX's own evaluation loop fired: an alert evaluates the window its interval names, and a replayed capture is older than that by the time the loop runs. The firing condition is evaluated against the counts each surface returns and is labelled as such | +| `gap6_incremental_economics.sh` | rows, bytes, insert plus merge CPU and answers for three policies on one ClickHouse: no offload, a collector-only policy by service and severity, and the per-type policy | nothing about money. There is no Cloud autoscaler here, no bill, and no compute term; the arms are comparable to each other and to nothing else | +| `gap2b_vector058_parquet.sh` | what Vector 0.58.0 takes for Parquet, and what the same rows cost as newline delimited JSON and as Parquet, both written by Vector in one run and read through two Merge tables, with the Parquet reader's row group and page counters | that the two containers hold the same shape: the JSON arm writes a free attribute map and the Parquet arm writes named columns, because a schema inferred from a free map carries one field per key seen | +| `gap7_s3_table_before_objects.sh` | ClickHouse issue 116888: what an S3 table with an explicit schema and `use_hive_partitioning` answers when created before any object exists, what the same DDL answers when created after, and whether DETACH and ATTACH recovers the first | anything about how long the empty listing is held, or which other engines cache one. One table, one prefix, one server version, named in the results | +| `gap8_ttl_to_s3.sh` | what den-crane's TTL recipe costs on the same capture: the PUTs, the parts and the merge and move CPU a forced `TTL TO VOLUME 'cold'` charges, and what a slice stamped behind the boundary charges on insert alone | nothing about a bill, and nothing about the background scheduler's own pacing: the move is forced so the whole table moves exactly once and the number does not depend on which parts the scheduler happened to pick | +| `gap9_bounded_window.sh` | what a bound inside the table costs against the same bound written into the query: a view over the cold table carrying `day >= today() - 7`, a second Merge table over hot plus that view, and the two aggregation shapes put to both | nothing about an estate's own ratio of window to retention, and nothing about the rows outside the bound, which that table cannot reach at all | + +## Files + +| File | What it is | +|---|---| +| `lib.sh` | the compose pattern every script shares, plus `g_cold30_layout`, gap 1's thirty day object layout, shared so gap 9 asks its question of the same one: the pinned images, the network, MinIO, ClickStack, the receiver and the routing collector, plus `g_wait_for_object`, the guard for ClickHouse 116888 | +| `measure.py` | one query, timed on a cold cache, read back out of `system.query_log` with its S3 counters | +| `wire_hashes.py` | the census of what the receiver returned, from the collector's wire tap | +| `make_seq_input.py` | the capture with a sequence number inside every line, for gap 4 | +| `patch_retriever_config.py` | the asserted edits that point the shipped Retriever config at the local file system | +| `hyperdx_alert.py` | the alert attempt over the HyperDX API, every status recorded | +| `render.py` | composes `results/clickstack-e2e-gaps-.md` from the per-gap JSON and the hand-written notes | +| `notes/` | the prose for each gap. The numbers are not in here | +| `results/` | `gap.json` per script, the composed first-pass file, and the second pass's own file | + +## Reading the results files + +`results/clickstack-e2e-gaps-2026-09-15.md` is the first pass, one table per gap. +Every number in it comes from `results/gap.json`, and `render.py` composes it +from those files and the prose in `notes/`. Where a gap is open, its section says +so and says what would close it. + +`results/clickstack-e2e-close-2026-09-15.md` is the second pass, written after a +record check against primary sources found one published claim false and four +questions the first pass never put. Read it for the Vector Parquet retraction, +the Parquet reader's own counters, the aggregation shapes through the Merge +table, the collector run as its code owners prescribe, ClickHouse issue 116888 +and its guard, and what `TTL TO VOLUME 'cold'` costs on the same capture. That file +carries the digests every run used and the command that produced each table, and +its own JSON sits beside the first pass's: `gap1-agg.json`, `gap2b.json`, +`gap4-router-owners.json`, `gap7.json`, `gap8.json`. + +The same file's last section, `Wave 3, 2026-09-15 evening`, is a third pass on the +same harness: the persistent sending queue arm of gap 4 (`gap4-router-persist.json`), +the bounded cold window (`gap9.json`), and what the public Retriever image carries, +which is why gap 3 is still not measured. + +Durable handoff with custody, 2026-10-23. The collector runs the shipped recipe's settings +(file positions and the queue into the receiver in `file_storage`, one sender, retries +without a deadline, batching inside the persisted queue, the synchronous file +emitter), on contrib 0.160.0 and RustFS. Lines stored nowhere, lines stored twice, +and lines the receiver completed without an output (`input_lines_abandoned`), out of +197,430: + +| Arm | receiver answers on receipt | receiver answers once lines are done, run 1 | receiver answers once lines are done, run 2 | +|---|---|---|---| +| collector killed | 32,203 missing, 0 twice | 0 missing, 900 twice, 0 abandoned | 0 missing, 997 twice, 0 abandoned | +| receiver killed | 31,976 missing, 0 twice | 0 missing, 0 twice, 0 abandoned | 0 missing, 0 twice, 0 abandoned | +| both killed | 24,900 missing, 0 twice | 0 missing, 232 twice, 0 abandoned | 0 missing, 0 twice, 0 abandoned | + +The custody columns run a receiver that answers an export only once its lines are written +back and acknowledged (the run-cloud jar on the JRE image, with +`PATCHED_JAR`). JSON: `gap4-custody-before-edge-1.1.147.json`, +`gap4-custody-after-1.1.150-run1.json`, `gap4-custody-after-1.1.150-run2.json`. + +Where the two disagree, the second is the later measurement and says so. diff --git a/clickstack-e2e/README.md b/clickstack-e2e/README.md index ed05a98..d3e31ff 100644 --- a/clickstack-e2e/README.md +++ b/clickstack-e2e/README.md @@ -1,108 +1,48 @@ # clickstack-e2e -The whole ClickHouse route, end to end, in containers, in one command: a -released log capture read by an OpenTelemetry Collector, regulated by the 10x -receiver, split by the mark the receiver puts on each record, and read back out -of ClickHouse as one table over the rows that stayed and the rows that left. +Can the rows a capped service sends over its cap go to an object store and still be queried with the rows ClickStack kept, as one ClickHouse table, with none lost? In this run they could. Of the 37,536 records the 10x engine returned, 13,010 went into ClickStack's table and 24,526 went to S3-compatible storage, and counting both sides back gave 37,536 with a gap of 0. -Everything a ClickHouse surface claims should be traceable to a run of this -harness. The numbers live in `results/`, not in this file. +Three services had a byte cap, and the engine marked every record it returned with a route, and the records over a cap were marked offload. A stock OpenTelemetry Collector writes the offload records to the object store under a service and day path and sends the rest to ClickStack. ClickHouse reads the objects in place, and a Merge table reads both sides under one name. -## Run it - -```bash -./run.sh # everything -KEEP=1 ./run.sh # leave the containers up afterwards -LINES=20000 ./run.sh # feed a smaller slice of the capture -``` +## Result -Needs Docker with about 8 GB of memory, a few GB of free disk, and roughly -fifteen minutes. The capture is fetched into `data/` on the first run and its -sha256 is checked on every run. Every container is torn down at the end unless -`KEEP=1`. +Count queries from the 2026-09-15 run, each on a cold cache. The service filter is opentelemetry-collector. -The run writes `results/clickstack-e2e-.md` and `results/results.json`. +| Query | Rows read | S3 GETs | ms | +|---|---:|---:|---:| +| Hot table only, last hour | 12,012 | 0 | 8 | +| Merge, one service, last hour | 25,793 | 6 | 77 | +| Merge, one service, one day | 25,793 | 6 | 43 | +| Merge, time only (HyperDX's shape) | 37,536 | 14 | 118 | +| Merge, text search | 37,536 | 14 | 99 | +| Merge, one pattern hash | 37,536 | 14 | 123 | +| Counts table, by service | 2,539 | 0 | 6 | -## The shape +The time-only filter that HyperDX sends read every row and made one GET per offloaded object. A service filter cut that to the 6 objects under that service. The hot table and the counts table made no object-store request. -``` - slice of the released capture - | - | filelog receiver, JSON envelope parsed, `log` value as the body - v - collector --OTLP--> 10x receiver --OTLP--> collector - (cap + action) | - | routing connector on routeState - +---------+---------+ - | | - routeState=offload everything else - | | - awss3 exporter ClickStack's own - -> MinIO, the OTLP endpoint - service and the | - day in the path v - | ClickStack's ClickHouse - v exporter -> otel_logs - S3 engine table - | - +--> Merge table over both <--+ -``` +- Only the capped services had offloaded rows: 16,856 from opentelemetry-collector, 4,522 from kafka and 3,148 from cart. +- The same 2,495 distinct pattern hashes were on the wire and in storage. Only 37,486 of the 37,536 returned records carried a hash, and a record without one is stored but not counted per type. +- ClickStack's own collector has no S3 exporter, so a second, stock collector does the routing. The harness also adds a day column to ClickStack's `otel_logs` table. -The receiver reads two files: `conf/caps.csv` gives a byte cap per service (this run counted bytes; the rows run counts events), and -`conf/actions.csv` gives that service's disposition for the excess. Both must -exist and neither may be empty, and a service with no row in the cap file is not -regulated at all, which is what makes an uncapped service a control. +## How it was measured -## What a run proves +The input is the first 50,000 lines of the released capture otel-sample-200mb.log. One host runs ClickStack 2.38.0 on ClickHouse 26.5.7.64, the 10x engine, a stock OpenTelemetry Collector 0.160.0 and MinIO. The lossless check counts every record the engine returned against the stored rows, in total and per pattern hash. Query figures come from ClickHouse's own query log, with caches dropped first. -- That the marks the regulator puts on a record survive an OTLP round trip - through the receiver and arrive on the returned record as attributes a - collector can route on. The results file lists every field that arrived and - keeps one returned record verbatim. -- That a stock routing connector splits the returned stream on that mark, that - the marked slice lands in an object store with the service and the day in the - object path, and that the rest is inserted by ClickStack's own exporter into - ClickStack's own table. -- That ClickHouse reads the offloaded objects in place through an S3 engine - table, and reads them together with the hot table through a Merge table, with - `_table` naming which side each row came from. -- That a counts-per-type table answers a dashboard question over both sides - after the split. -- What each query shape costs over the Merge table, in rows read, object-store - requests and time, against the same query over the hot table alone. -- That what the receiver returned reconciles against what is stored, counted in - total and per pattern hash. This is a total, not a per-line identity: a lost - line and a duplicated one cancel in it. `clickstack-e2e-gaps` counts per line - through a sequence number inside the body, and under a restart finds both. +## Limits -## What a run does not prove +- One run, with MinIO in place of S3. The times say nothing about real S3, throughput or ClickHouse Cloud cost. +- No alert is defined or fired, and the queries were written for this test. +- Every object carries one day in its path, because the path records the write time. Pruning across thirty days is measured in [clickstack-e2e-gaps](../clickstack-e2e-gaps/). +- The lossless check is a total, so a lost line and a duplicated one would cancel. The gaps benchmark counts per line. -- Nothing about ClickHouse Cloud. There is no compute term, no bill and no - autoscaler here; the run measures requests and rows, not money. -- Nothing about alerts. No alert is defined, fired or compared. -- Nothing about a real estate's query mix. The query list is written here, not - taken from a customer's `system.query_log`. -- Nothing about multi-day pruning. The objects are written during the run, so - they carry one day in the path. `clickstack-e2e-gaps/gap1_multiday_pruning.sh` - measures it over thirty days of objects. -- Nothing about the Retriever. Value-level fetch-back is a different door and - is not exercised. -- Nothing about scale. The default slice is a few tens of thousands of records, - enough to exercise every hop and too small to say anything about throughput. +## Run it -## Files +``` +cd clickstack-e2e +./run.sh # everything, about fifteen minutes +LINES=20000 ./run.sh # a smaller slice of the capture +``` -| File | What it is | -|---|---| -| `run.sh` | the whole run: containers, feed, schema, measurement, teardown | -| `conf/router.yaml` | the collector config, both pipelines, as it ran | -| `conf/caps.csv`, `conf/actions.csv` | the per-service policy the receiver loads | -| `conf/schema_hot.sql` | the counts table and its view, created before any data flows | -| `conf/schema_cold.sql` | the S3 engine table, the rename view, the Merge table | -| `patch_engine_config.py` | the edits the harness makes to the engine's shipped config, each asserted | -| `report.py` | the wire census, the timed queries, the lossless check, the results file | -| `results/` | committed results of the last run | +A run needs Docker with about 8 GB of memory and downloads the capture on first use. -`patch_engine_config.py` is the file to read first if a run comes back with no -marks on the wire: it names the four edits and fails loudly if the shipped -config has moved under it. +[RECORD.md](RECORD.md) describes the pipeline and what a run does and does not prove, and [the results file](results/clickstack-e2e-2026-09-15.md) has every number and config from this run. diff --git a/clickstack-e2e/RECORD.md b/clickstack-e2e/RECORD.md new file mode 100644 index 0000000..ed05a98 --- /dev/null +++ b/clickstack-e2e/RECORD.md @@ -0,0 +1,108 @@ +# clickstack-e2e + +The whole ClickHouse route, end to end, in containers, in one command: a +released log capture read by an OpenTelemetry Collector, regulated by the 10x +receiver, split by the mark the receiver puts on each record, and read back out +of ClickHouse as one table over the rows that stayed and the rows that left. + +Everything a ClickHouse surface claims should be traceable to a run of this +harness. The numbers live in `results/`, not in this file. + +## Run it + +```bash +./run.sh # everything +KEEP=1 ./run.sh # leave the containers up afterwards +LINES=20000 ./run.sh # feed a smaller slice of the capture +``` + +Needs Docker with about 8 GB of memory, a few GB of free disk, and roughly +fifteen minutes. The capture is fetched into `data/` on the first run and its +sha256 is checked on every run. Every container is torn down at the end unless +`KEEP=1`. + +The run writes `results/clickstack-e2e-.md` and `results/results.json`. + +## The shape + +``` + slice of the released capture + | + | filelog receiver, JSON envelope parsed, `log` value as the body + v + collector --OTLP--> 10x receiver --OTLP--> collector + (cap + action) | + | routing connector on routeState + +---------+---------+ + | | + routeState=offload everything else + | | + awss3 exporter ClickStack's own + -> MinIO, the OTLP endpoint + service and the | + day in the path v + | ClickStack's ClickHouse + v exporter -> otel_logs + S3 engine table + | + +--> Merge table over both <--+ +``` + +The receiver reads two files: `conf/caps.csv` gives a byte cap per service (this run counted bytes; the rows run counts events), and +`conf/actions.csv` gives that service's disposition for the excess. Both must +exist and neither may be empty, and a service with no row in the cap file is not +regulated at all, which is what makes an uncapped service a control. + +## What a run proves + +- That the marks the regulator puts on a record survive an OTLP round trip + through the receiver and arrive on the returned record as attributes a + collector can route on. The results file lists every field that arrived and + keeps one returned record verbatim. +- That a stock routing connector splits the returned stream on that mark, that + the marked slice lands in an object store with the service and the day in the + object path, and that the rest is inserted by ClickStack's own exporter into + ClickStack's own table. +- That ClickHouse reads the offloaded objects in place through an S3 engine + table, and reads them together with the hot table through a Merge table, with + `_table` naming which side each row came from. +- That a counts-per-type table answers a dashboard question over both sides + after the split. +- What each query shape costs over the Merge table, in rows read, object-store + requests and time, against the same query over the hot table alone. +- That what the receiver returned reconciles against what is stored, counted in + total and per pattern hash. This is a total, not a per-line identity: a lost + line and a duplicated one cancel in it. `clickstack-e2e-gaps` counts per line + through a sequence number inside the body, and under a restart finds both. + +## What a run does not prove + +- Nothing about ClickHouse Cloud. There is no compute term, no bill and no + autoscaler here; the run measures requests and rows, not money. +- Nothing about alerts. No alert is defined, fired or compared. +- Nothing about a real estate's query mix. The query list is written here, not + taken from a customer's `system.query_log`. +- Nothing about multi-day pruning. The objects are written during the run, so + they carry one day in the path. `clickstack-e2e-gaps/gap1_multiday_pruning.sh` + measures it over thirty days of objects. +- Nothing about the Retriever. Value-level fetch-back is a different door and + is not exercised. +- Nothing about scale. The default slice is a few tens of thousands of records, + enough to exercise every hop and too small to say anything about throughput. + +## Files + +| File | What it is | +|---|---| +| `run.sh` | the whole run: containers, feed, schema, measurement, teardown | +| `conf/router.yaml` | the collector config, both pipelines, as it ran | +| `conf/caps.csv`, `conf/actions.csv` | the per-service policy the receiver loads | +| `conf/schema_hot.sql` | the counts table and its view, created before any data flows | +| `conf/schema_cold.sql` | the S3 engine table, the rename view, the Merge table | +| `patch_engine_config.py` | the edits the harness makes to the engine's shipped config, each asserted | +| `report.py` | the wire census, the timed queries, the lossless check, the results file | +| `results/` | committed results of the last run | + +`patch_engine_config.py` is the file to read first if a run comes back with no +marks on the wire: it names the four edits and fails loudly if the shipped +config has moved under it. diff --git a/clickstack-rows-gate/README.md b/clickstack-rows-gate/README.md index 85dd7ba..f9121a0 100644 --- a/clickstack-rows-gate/README.md +++ b/clickstack-rows-gate/README.md @@ -1,74 +1,47 @@ -# ClickHouse rows gate: which meter equals the hot count +# ClickHouse rows gate -One question, answered on the `clickstack-e2e` rig file for file: when the Receiver -regulates in lines with a service total and the shipped always-pass sample of 5, -which ClickHouse system table counts the rows written into the logs table to the -row? The answer is the meter the rows run reads every hour. +When the 10x engine limits how many lines a service sends into ClickStack, which ClickHouse system table counts the rows written into the logs table exactly? The NewPart rows in `system.part_log` matched the table's own count of 22,367 rows to the row, in both runs. The insert entries in `system.query_log` did not, so they serve as a cross-check at most. + +ClickStack's exporter inserts asynchronously, and each insert statement lists three tables in the query log (the logs table, a counts table and a 15-minute rollup), so a filter on the logs table cannot pick out its rows. Which of the two insert entry kinds carried the extra rows also changed between runs. ## Result -Two runs on 2026-10-23, engine `ghcr.io/log-10x/edge-10x:1.1.147`, ClickStack -2.38.0 on ClickHouse 26.5.7.64, the whole 197,430-line capture fed in about 216 s. -Results in [`results/`](results/), one file per run date. +The table shows the second run, on 2026-10-23. Hot means the logs table, where the lines the engine passed land. -| Count | Run 2 (events unit) | Against hot | +| Count | Rows | Against the logs table | |---|---:|---| -| Hot rows, `SELECT count() FROM default.otel_logs` | 22,367 | the count to match | -| Wire records marked `pass` | 22,367 | equal | -| `system.parts` active rows | 22,367 | equal | -| `system.part_log` NewPart rows, `otel_logs` | 22,367 | **equal** | -| `system.query_log` `written_rows`, kind `Insert`, `tables` has `otel_logs` | 24,315 | off by +1,948 | -| `system.query_log` `written_rows`, kind `AsyncInsertFlush` | 14,848 | off by -7,519 | - -In run 1 the `Insert` entries happened to equal the hot count and the -`AsyncInsertFlush` entries ran over by the materialized view's rows. ClickStack's -exporter inserts asynchronously, and which of the two entry kinds carries the -downstream rows differs from run to run, so `system.query_log` is a cross-check at -most. `system.part_log` NewPart rows equalled the hot count in both runs and is the -meter. Every insert statement names three tables (`otel_logs`, the counts table and -`otel_logs_kv_rollup_15m`) in `query_log`'s `tables`, which is why a filter on the -logs table alone cannot separate its rows there. - -Hot plus cold equalled the wire: 22,367 + 134,720 = 157,087 records returned by the -Receiver, with every record carrying a route. - -## What else the run measured - -- **The service totals held to the row.** With the regulator counting events, - kafka kept 1,000 of a total of 1,000, cart 400 of 400, and opentelemetry-collector - 2,050 of 2,000: the extra 50 is the error keep count for one storming error - statement, which the rate doc states. -- **A policy-file rewrite mid-window did not reset the window counts.** The totals - file was rewritten in place 4 seconds into the feed; the engine logged the reload - with all four policy files, and no capped service passed another line in the - window. The limits bind inside the window across a reload on this engine. -- **The events unit does not take effect from the environment.** Run 1 passed - `rateReceiverUnit=events` in the container's environment and the caps still counted - bytes (each capped service kept only its 5-line samples). The unit has to be in the - config file or an engine option, which is how `receiverFleetValues` writes it. -- **The e2e harness's config patches on 1.1.147:** its rate edits no longer apply - (1.1.147 ships `warmupMs: 0`), and its splice patch failed on indentation only. This - gate's [`patch_engine_config.py`](patch_engine_config.py) matches the expression - text, writes `unit: events`, and leaves `baselineCount` at the shipped 5. -- **The Receiver's Prometheus counters** carry `k8s_container`, `tenx_user_service`, - `tenx_hash`, `message_pattern`, `severity_level`, `routeState` and `tenx_env`. At the - moment of a scrape they lag the wire (`pass` 16,255 against 22,367), so a run reads - the last complete hour rather than the current one. +| `count()` on `default.otel_logs` | 22,367 | the count to match | +| Lines the engine routed to hot | 22,367 | equal | +| `system.parts`, active rows | 22,367 | equal | +| `system.part_log`, NewPart rows | 22,367 | equal | +| `system.query_log`, kind Insert | 24,315 | off by +1,948 | +| `system.query_log`, kind AsyncInsertFlush | 14,848 | off by -7,519 | + +The part log matched in both runs, so it is the count to use. In the first run the Insert entries happened to equal the hot count and the AsyncInsertFlush entries ran over by the materialized view's rows. + +The run also checked the limits: + +- With the limit counted in lines, kafka kept 1,000 lines against a total of 1,000 and cart kept 400 against 400. The opentelemetry-collector service kept 2,050 against 2,000, and the 50 extra lines are the engine's error keep count for one storming error statement. +- Rewriting the totals file in place 4 seconds into the feed did not reset the window counts, and no limited service passed another line in that window. +- Hot plus cold equalled the records the engine returned: 22,367 + 134,720 = 157,087. + +## How it was measured + +The rig is the one in [clickstack-e2e](../clickstack-e2e/): ClickStack 2.38.0 on ClickHouse 26.5.7.64, the 10x engine in front of it, and an S3-compatible store for the cold side. The whole 197,430-line capture was fed in about 216 seconds. Three services had a line total for the window, and their lines over it went to S3, which a ClickHouse view reads back. The results file lists every query with its answer. + +## Limits + +- The feed is one limit window long, so the test says nothing about money, ClickHouse Cloud or a rate kept over hours. The hourly test that uses this count is [clickstack-rows](../clickstack-rows/). +- Both runs used one ClickStack version, and only the second run's figures are committed. +- The engine's Prometheus counters, scraped 5 seconds after the collector stopped, still lagged the lines sent (16,255 passed against 22,367). A reading of them should use the last complete hour. ## Run it ``` -CSE_DATA_DIR=../clickstack-e2e/data ./run.sh # the capture clickstack-e2e/run.sh downloads -RELOAD_AFTER=40 ./run.sh # seconds into the feed to rewrite totals.csv +# from clickstack-rows-gate/, with the capture that clickstack-e2e/run.sh downloads +CSE_DATA_DIR=../clickstack-e2e/data ./run.sh ``` -MinIO's public images were withdrawn from quay.io and Docker Hub (401 and access -denied on 2026-10-23), so the object store is RustFS on the same port and endpoint -name with the same static credentials, and bucket operations run through the AWS CLI -image. The collector and ClickHouse read it as S3 unchanged. - -## What a run does not prove +A run needs Docker and the capture from clickstack-e2e. The engine image is `ghcr.io/log-10x/edge-10x:1.1.147`. Results land in [results/](results/). -Nothing about money, ClickHouse Cloud, or a rate held over hours: the feed is one -regulator window long. The rows run's metered run is the separate harness that holds -a rate through growth; this gate only picks its meter. +[RECORD.md](RECORD.md) has the full record, including the engine counters, their labels and the config patches. diff --git a/clickstack-rows-gate/RECORD.md b/clickstack-rows-gate/RECORD.md new file mode 100644 index 0000000..85dd7ba --- /dev/null +++ b/clickstack-rows-gate/RECORD.md @@ -0,0 +1,74 @@ +# ClickHouse rows gate: which meter equals the hot count + +One question, answered on the `clickstack-e2e` rig file for file: when the Receiver +regulates in lines with a service total and the shipped always-pass sample of 5, +which ClickHouse system table counts the rows written into the logs table to the +row? The answer is the meter the rows run reads every hour. + +## Result + +Two runs on 2026-10-23, engine `ghcr.io/log-10x/edge-10x:1.1.147`, ClickStack +2.38.0 on ClickHouse 26.5.7.64, the whole 197,430-line capture fed in about 216 s. +Results in [`results/`](results/), one file per run date. + +| Count | Run 2 (events unit) | Against hot | +|---|---:|---| +| Hot rows, `SELECT count() FROM default.otel_logs` | 22,367 | the count to match | +| Wire records marked `pass` | 22,367 | equal | +| `system.parts` active rows | 22,367 | equal | +| `system.part_log` NewPart rows, `otel_logs` | 22,367 | **equal** | +| `system.query_log` `written_rows`, kind `Insert`, `tables` has `otel_logs` | 24,315 | off by +1,948 | +| `system.query_log` `written_rows`, kind `AsyncInsertFlush` | 14,848 | off by -7,519 | + +In run 1 the `Insert` entries happened to equal the hot count and the +`AsyncInsertFlush` entries ran over by the materialized view's rows. ClickStack's +exporter inserts asynchronously, and which of the two entry kinds carries the +downstream rows differs from run to run, so `system.query_log` is a cross-check at +most. `system.part_log` NewPart rows equalled the hot count in both runs and is the +meter. Every insert statement names three tables (`otel_logs`, the counts table and +`otel_logs_kv_rollup_15m`) in `query_log`'s `tables`, which is why a filter on the +logs table alone cannot separate its rows there. + +Hot plus cold equalled the wire: 22,367 + 134,720 = 157,087 records returned by the +Receiver, with every record carrying a route. + +## What else the run measured + +- **The service totals held to the row.** With the regulator counting events, + kafka kept 1,000 of a total of 1,000, cart 400 of 400, and opentelemetry-collector + 2,050 of 2,000: the extra 50 is the error keep count for one storming error + statement, which the rate doc states. +- **A policy-file rewrite mid-window did not reset the window counts.** The totals + file was rewritten in place 4 seconds into the feed; the engine logged the reload + with all four policy files, and no capped service passed another line in the + window. The limits bind inside the window across a reload on this engine. +- **The events unit does not take effect from the environment.** Run 1 passed + `rateReceiverUnit=events` in the container's environment and the caps still counted + bytes (each capped service kept only its 5-line samples). The unit has to be in the + config file or an engine option, which is how `receiverFleetValues` writes it. +- **The e2e harness's config patches on 1.1.147:** its rate edits no longer apply + (1.1.147 ships `warmupMs: 0`), and its splice patch failed on indentation only. This + gate's [`patch_engine_config.py`](patch_engine_config.py) matches the expression + text, writes `unit: events`, and leaves `baselineCount` at the shipped 5. +- **The Receiver's Prometheus counters** carry `k8s_container`, `tenx_user_service`, + `tenx_hash`, `message_pattern`, `severity_level`, `routeState` and `tenx_env`. At the + moment of a scrape they lag the wire (`pass` 16,255 against 22,367), so a run reads + the last complete hour rather than the current one. + +## Run it + +``` +CSE_DATA_DIR=../clickstack-e2e/data ./run.sh # the capture clickstack-e2e/run.sh downloads +RELOAD_AFTER=40 ./run.sh # seconds into the feed to rewrite totals.csv +``` + +MinIO's public images were withdrawn from quay.io and Docker Hub (401 and access +denied on 2026-10-23), so the object store is RustFS on the same port and endpoint +name with the same static credentials, and bucket operations run through the AWS CLI +image. The collector and ClickHouse read it as S3 unchanged. + +## What a run does not prove + +Nothing about money, ClickHouse Cloud, or a rate held over hours: the feed is one +regulator window long. The rows run's metered run is the separate harness that holds +a rate through growth; this gate only picks its meter. diff --git a/clickstack-rows/README.md b/clickstack-rows/README.md index 89ecc4b..963883c 100644 --- a/clickstack-rows/README.md +++ b/clickstack-rows/README.md @@ -1,215 +1,50 @@ -# ClickHouse rows run: rows written into ClickStack held at a rate an hour - -One benchmark in the [log10x benchmarks](../) repository. It is the metered gate for the -rows run in log10x-mcp (`log10x_setup_recurring` with `rows_per_hour`). Each run reads the -rows ClickHouse wrote into the logs table from its own `system.part_log`, and rewrites the -Receiver's per-statement limits and service totals in lines. Lines over a service's total -go to the account's own bucket before ClickHouse receives them. - -The method was written before the run: [`METHOD.md`](METHOD.md). It is time-scaled, with -every clock divided by 6 (a run every 10 minutes, a 10-minute meter window, growth stepped -every 20 minutes). The regulator's 4-minute window and the always-pass sample of 5 are -engine constants and were not scaled. - -## Result, 2026-10-23: FAIL as registered; the rate held flat through the growth - -The engine was `ghcr.io/log-10x/edge-10x:1.1.147`, on ClickStack 2.38.0 (ClickHouse -26.5.7.64). The released OpenTelemetry demo capture was replayed from 14 to 35 lines a -second. The dry run measured 42,000 rows an hour with nothing held, so R = 29,399 rows an -hour (0.7 of that). There were 15 applied runs, the last after the feed stopped. - -| | Criterion | Result | -|---|---|---| -| 1 | Every applied run exits 0 and commits, projecting at or under R | **Pass.** 15 runs, all exit 0, each projecting about 29k rows an hour once the rate bound. | -| 2 | Each 4-minute window at or under R/15 plus the floor's share; each 10-minute meter window at or under R/6 | **Fail.** 23 of 25 four-minute windows were under their bound. 19:04 read 1.52 and 19:08 read 1.06. Every complete 10-minute window read 0.99 to 1.04 of R/6, except 19:00 at 1.36. | -| 3 | Hot plus cold equals the returned records, for the ten largest moved statements | **Fail.** 3 of 10 were exact. 7 were 1 to 11 rows over: 33 rows inserted twice into the hot table across the run (0.04%). The bucket was exact. | -| 4 | A count through the Merge table equals the local count | **Fail against the wire, exact against storage.** For all ten statements the Merge table equalled hot plus cold. The differences are the duplicates in criterion 3. | -| 5 | Every line written comes back; every returned record is in the table or the bucket | **Every returned record is stored:** 194,864 rows held 194,831 returned records, none missing, 33 duplicated. Lines in against records back is not settled here: the Receiver folds multi-line events, 242,741 lines into 194,831 records. | -| 6 | Storm: error lines of the storming statement per window at most its limit plus 50 | **Pass, read by hand.** The largest increase in a 4-minute window was about 117, against a limit of about 73 plus 50. | -| 7 | The injected service is held from its first window by the `*` row | **Pass on the regulator's windows.** Read by clock window, the first reads 96 rows against a `*` total of 49. The regulator's windows start at its first regulated line, and that clock window spans two of them, 47 and 49, each at or under the total. From its second window it held to its own row. | - -**What held.** Load grew 2.5 times, with a 20-minute error storm and a new service. -Through all of it, the rows written into the table stayed at 0.99 to 1.04 of R in every -complete 10-minute window but one, from the first growth step to the end of the hold at -35 lines a second (2.5 times the starting 14). The one, 19:00, read 1.36, and cause 2 below -is why. Before the first applied run they read 1.40 to 1.43 of R. Every line over a total went to the bucket -before ClickHouse received it: 110,392 rows there against 84,472 in the table. - -**Why criterion 2 failed: two causes.** - -1. **The always-pass floor.** Each statement passes 5 lines per window on every node, - whatever its limit. The run subtracts that floor from the allowance, but the floor - changes from window to window with the statements that print. The method allows the - floor's share in the 4-minute bound and not in the 10-minute sum, which is why most - 10-minute windows read 1 to 4% over R/6. -2. **One service printing 1,422 new statements in four minutes.** When the replay looped - the capture, Grafana's migrator ran again. The engine names that statement once per - migration id, so it arrived as 1,422 distinct patterns in one window, each passing its - always-pass sample, and no service total bounds that sample. Grafana passed 1,481 lines - in the window against a usual 140, which is the 1.52. The next applied run, ten minutes - later, sized the window after it. The rate holds against growth in known traffic. A - burst of new statements from one service overshoots by its pattern count times the - sample, until the next run. - -**Two more findings.** -- **The Receiver's Prometheus counters undercount under load.** They agreed with the meter - at the starting load (scale 1.00). At 30 and 35 lines a second they read 2.1 to 6.3 times fewer pass - lines than the meter counted, with a healthy scrape. The rows run takes totals from the - meter and only the share each statement contributes from the counters, so the hold did - not depend on them. The report prints the scale factor every run. -- **ClickStack's insert path wrote 33 rows twice** over 84,439 inserted, with no restart. - An earlier run found the same on a retry of an insert that had already committed - (`clickstack-e2e-gaps`, gap 5). - -## Rerun, 2026-10-24, with the regulator's sample bound: every 4-minute window under its bound; FAIL as registered on the 10-minute sum alone - -Same rig, image, capture, pace steps, storm and injected service as the run above, with one -change in the Receiver's configuration: the always-pass sample counts toward its service's -total, and a service passes at most 50 more sample lines per window once its total is spent -(`baselineMaxOverTotal`, log-10x/config `4253528`, declared in log-10x/modules `cbbc8c32`, both -mounted over the 1.1.147 image's own files for this run; the 1.1.150 image ships them in its configuration; `results-sample-bound/build/` records the -commits and the patches). The dry run measured the same 42,000 rows an hour, so R = 29,399 -again. There were 15 applied runs. Results in `results-sample-bound/`. - -| | Criterion | Result | -|---|---|---| -| 1 | Every applied run exits 0 and commits, projecting at or under R | **Pass.** 15 runs, all exit 0, each projecting about 29k rows an hour once the rate bound. | -| 2 | Each 4-minute window at or under R/15 plus the floor's share; each 10-minute meter window at or under R/6 | **Fail on the 10-minute sum alone.** All 25 four-minute windows were under their bound. The complete 10-minute windows read 0.97 to 1.07 of R/6; 5 of 9 were over by 3 to 7%, the floor's share the bound makes no allowance for. | -| 3 | Hot plus cold equals the returned records, for the ten largest moved statements | **Pass.** All ten exact. No row was inserted twice: 194,524 returned, 83,018 in the table and 111,506 in the bucket. | -| 4 | A count through the Merge table equals the local count | **Pass.** All ten exact. | -| 5 | Every line written comes back; every returned record is in the table or the bucket | **Every returned record is stored, none twice.** 242,357 lines became 194,524 records (the Receiver folds multi-line events). | -| 6 | Storm: error lines of the storming statement per window at most its limit plus 50 | **Pass, read by hand.** The largest increase in a 4-minute window was about 124, against the service's total of 76 to 88 plus 50. | -| 7 | The injected service is held from its first window by the `*` row | **Pass.** 86 rows in its first clock window against a `*` total of 186 in force; from the next applied run it held to its own row of 92, with 43 to 92 a window. | - -**What changed.** The capture loop inside the hold at 35 lines a second, where the first run -passed 1,481 Grafana lines in one window (1,422 new statements, each passing its sample), passed -under 200 in every window here: the rows written in that 10-minute window read 1.03 of R/6, -against 1.36 before. The only window with more than 200 Grafana passes in the whole run is the -startup window, before any total was set (1,588 lines, 1,433 statements). The 10-minute sums -sit 3 to 7% over R/6 for the reason the first run found: the floor is inside R and the -10-minute test has no allowance for it, while the 4-minute test does and holds everywhere. - -**The regulator bound, measured by the gate first.** `tests/regulator/run.sh b1-sample-bound -b2-sample-bound-off b3-sample-bound-mute` in log-10x/config: on the image's own configuration -a 200-statement burst under a 50-line total passes 400 of 400 lines; with the bound it passes -100 (the total plus 50), and `baselineMaxOverTotal: 0` restores 400. - -## Two-arm run: the rows run against a collector rule, under the same growth - -The method, written before the run, is [`METHOD_TWO_ARM.md`](METHOD_TWO_ARM.md). -`ARMS=two rig/phases.sh` runs the rows run as above (arm A, `default.otel_logs`, on the -engine image's own configuration) and, on the same node and the same feed, a second -routing collector with the collector-only rule from `clickstack-e2e-gaps` gap 6 (arm B, -[`rig/router-rule.yaml`](rig/router-rule.yaml)): the two chattiest services' INFO and DEBUG -lines go to a second bucket and the rest into `default.otel_logs_rule`, a copy of the logs -table with the same counts view and without ClickStack's `otel_logs_kv_rollup_15m` view. Arm B -has no Receiver. The two services are the capture's chattiest by lines, counted before the run: -`opentelemetry-collector` (93,875 lines) and `cart` (39,292) of 197,430, with `kafka` third at -31,029. Arm B inserts through the OpenTelemetry Collector's ClickHouse exporter straight into -its table, without ClickStack's own collector and its processing, so each arm's figures against -its own pre-step window compare like for like and the CPU figures of one arm against the -other's do not. The meter reads, per table and per window of ten rig minutes counted from the -start, NewPart rows from `system.part_log`, merge CPU from its MergeParts ProfileEvents and -insert CPU from `system.query_log`. -`rig/score.py --arms two` scores the three criteria and writes the comparison -table. Results go to `results-two-arm/`. - -```bash -ARMS=two MCP= GH_TOKEN=$(gh auth token) rig/phases.sh -``` +# ClickHouse rows run -### Result, 2026-10-25: rows separate the arms, CPU does not; FAIL as registered on criteria 1 and 3 +Can the rows written into a ClickStack logs table be kept at a set rate an hour while log volume grows 2.5 times? On one node, the rows ClickHouse wrote stayed between 0.91 and 1.13 of the set rate in every 10-minute window as load grew, one over the 1.10 the test allowed. A static collector rule on the same feed let its table grow to 2.95 to 3.42 times its starting level at full load. CPU did not separate the two. -The engine image is `ghcr.io/log-10x/edge-10x:1.1.150`, on ClickStack 2.38.0 (ClickHouse -26.5.7.64), with the capture, pace steps, storm and injected service of the rows run. From -minute 40, before the pre-step window, `old_parts_lifetime` is 60 s (ClickHouse's default is -480 s) on the 15 tables holding the most merged-away parts, both arms' logs tables among them; -the setting governs only when merged-away parts leave the disk (`timeline.jsonl`, -`old_parts_lifetime_60`). The dry run measured 43,000 rows an hour, so R = 30,099 (5,016 rows -a window). Results in `results-two-arm/`. +Every 10 minutes a scheduled run reads ClickHouse's own `system.part_log` and sets a limit per log statement in the 10x engine. Lines over the limits, apart from a small sample, go to the account's own bucket instead of ClickHouse. -| | Criterion | Result | -|---|---|---| -| 1 | Arm A's rows per window at or under 1.10 R/6, from the first growth step to the end of the hold | **Fail on one window.** Nine of ten read 0.91 to 1.09; minutes 100 to 110, the storm's first window, read 1.13. The 4-minute test: 2 of 24 windows over R/15 plus F/15, the largest at 1.15. | -| 2 | Arm B's rows per window above 1.5x its pre-step level in every window of the hold | **Pass.** 2,686 rows before the step; 7,932 to 9,179 in the hold, 2.95 to 3.42 times. | -| 3 | Insert plus merge CPU: arm A at or under 1.10 of its pre-step level after the step; arm B above 1.5x in the hold | **Fail on both halves.** Arm A read 0.87 to 1.24 of its 17.1 s, three of ten windows over. Arm B read 0.40 to 1.61 of its 122.5 s. No differentiator on CPU. | +## Result -| Window (minutes) | Arm A rows | Arm A CPU s (insert + merge) | Arm B rows | Arm B CPU s (insert + merge) | -|---|---:|---:|---:|---:| -| 50 to 60, before the step | 4,697 | 17.1 (11.8 + 5.3) | 2,686 | 122.5 (18.6 + 103.9) | -| 120 to 130, hold | 5,377 | 20.9 (13.4 + 7.4) | 7,963 | 197.7 (38.4 + 159.3) | -| 130 to 140, hold | 4,581 | 14.9 (11.1 + 3.8) | 7,932 | 64.7 (32.9 + 31.7) | -| 140 to 150, hold | 5,443 | 18.9 (12.2 + 6.7) | 9,179 | 179.9 (38.0 + 141.9) | -| 150 to 160, hold | 5,052 | 15.2 (10.9 + 4.3) | 8,142 | 48.5 (30.7 + 17.9) | - -**What the CPU shows.** Insert CPU follows the rows: arm A's read 0.92 to 1.14 of its pre-step -level in every window after the step but one (1.40, minutes 90 to 100), and arm B's read 1.65 -to 2.06 in the hold. Merge CPU decides the windows. Arm B's inserts came four to nine times as -often as ClickStack's own collector sends arm A's (1,018 to 2,098 insert entries a window -against 240), it ran 5,125 merges against arm A's 376, and its merge CPU swung between 17.9 and -159.3 s from one hold window to the next. Read per window, criterion 3 measures which merges -the scheduler ran inside each window. - -**Run notes.** -- Delivery: arm B, 242,324 lines written and 242,365 rows in its table and bucket (41 over). - Arm A, 204,399 records returned (the Receiver folds multi-line events) and 205,346 in its table - and bucket (933 over in the table, 14 in the bucket). The tables logged 20 and 18 failed inserts, - all but one in the first 20 minutes, before any scored window. -- Rows cut share in the hold (rows to the bucket over the table's rows plus the bucket's): arm A - 0.72, arm B 0.63. -- The run's disk peaked at 1.21 GB. - -## Rows the rule kept that the rows run moved, from the two-arm run - -The two-arm run's wire tap carries every record arm A returned with its service, statement -hash and route. Arm B's rule is deterministic (lines from `opentelemetry-collector` and `cart` -at INFO or DEBUG to the bucket, the rest to the table), so applying it to the same records -gives the rule's table within 23 rows of what arm B measured (58,970 against 58,947 for -minutes 100 to 160). The question it answers: what did the rows run remove that a static rule -left in place? - -| Window | Records the rule keeps | Of those, the rows run sent to the bucket | Share | Records the rule sends out that the rows run kept | +Rows written in eight 10-minute windows on 2026-10-25, against a set rate R of 30,099 rows an hour (5,016 a window). + +| Minutes | Lines a second | 10x engine rows | Of R/6 | Collector rule rows | |---|---:|---:|---:|---:| -| Minutes 100 to 160 (storm and hold) | 50,378 | 27,155 | 53.9% | 7,691 | -| Minutes 120 to 160 (hold) | 27,361 | 11,814 | 43.2% | | -| Whole run | 80,105 | 30,955 | 38.6% | | - -With the planted storm and the injected service left out, the rows run still moved 30.6% of -what the rule kept. The ten largest moved statements make up 84% of the moved records: -recommendation's `ListRecommendations`, two ad statements and kafka housekeeping lines among -them, all in services the rule keeps whole. The 7,691 records the rule would have sent out -and the rows run kept are its sample and error floors; 1,628 of them are an upload-failure -statement the engine labels ERROR and the rule's severity regex reads as INFO. The tap has -no severity field, so the floor split uses the same capture's labels from the Elastic run. - -## Files - -| Path | What | -|---|---| -| `METHOD.md` | the method, written before the run | -| `rig/` | the harness (`phases.sh` runs it: `MCP= GH_TOKEN=... rig/phases.sh`) | -| `results/score.md`, `results/final.md` | criteria 1, 2, 6 and the reports; criteria 3, 4, 5, 7 | -| `results/timeline.jsonl`, `results/meter.log` | every phase and every meter reading | -| `results/budget-runs/*/` | each run's report, files and commit | -| `results/policy.yaml`, `results/images.txt`, `results/cse-engine.log`, `results/engine-metrics.txt` | the policy, image digests, the Receiver's log and its last scrape | -| `results-two-arm/score.md`, `results-two-arm/two-arm-windows.json` | the two-arm run's three criteria and comparison table, and every window per table as ClickHouse read it at collect | - -The wire tap (428 MB) and the replay feed (251 MB) are not committed. The run writes them -again. - -## Rig notes - -- **ClickStack's collector** opens 4317 only after HyperDX hands it a config, which needs a - registered team. The rig registers inside its start loop, then waits for the port. -- **The meter user.** ClickStack's `default` user refuses HTTP from outside the container. - The rig reads through a `rows_meter` user, defined in `rig/rows_meter.xml`, with SELECT on - `system.part_log` and `system.parts` and SHOW TABLES on `otel_logs`. It has no SELECT on - the logs table. -- **The object store is RustFS.** MinIO's public images were withdrawn from quay.io and - Docker Hub (401, access denied), so the rig runs RustFS with the AWS CLI image. - -## What a run does not prove - -Nothing about money, ClickHouse Cloud or its autoscaler, 300 nodes, durability across a -restart, or write CPU against a collector rule. It holds rows on one node, from one -replayed capture. +| 50 to 60 | 14 | 4,697 | 0.94 | 2,686 | +| 60 to 70 | 19 | 5,492 | 1.09 | 3,572 | +| 80 to 90 | 24 | 5,471 | 1.09 | 5,858 | +| 100 to 110 | 30 | 5,671 | 1.13 | 12,877 | +| 120 to 130 | 35 | 5,377 | 1.07 | 7,963 | +| 130 to 140 | 35 | 4,581 | 0.91 | 7,932 | +| 140 to 150 | 35 | 5,443 | 1.09 | 9,179 | +| 150 to 160 | 35 | 5,052 | 1.01 | 8,142 | + +The registered test required every window from minute 60 to 160 at or under 1.10 of R/6, and nine of ten were. Minutes 100 to 110, the first window of a 20-minute error storm, read 1.13, so the run fails as registered. + +- Against each setup's level in minutes 50 to 60, insert plus merge CPU read 0.87 to 1.24 for the 10x engine and 0.40 to 1.61 for the rule at full load, as merge CPU swung with which merges ran. +- Of the 50,378 records the rule would have kept in minutes 100 to 160, the 10x engine sent 27,155 (53.9%) to the bucket. After the storm, in minutes 120 to 160, the share was 43.2%. + +## How it was measured + +Both setups shared one ClickStack 2.38.0 node and a replay of the OpenTelemetry demo capture. Load stepped from 14 to 35 lines a second every 20 minutes, adding a new service at minute 80 and the storm at 100. Every clock is divided by 6, so limits were reset every 10 minutes instead of hourly. R is 0.7 of a dry run's rate with nothing limited. The rule sends the two chattiest services' INFO and DEBUG lines to a second bucket and the rest to a copy of the logs table. CPU is insert time from `system.query_log` plus merge time from the part log. + +## Limits + +- One node and one replayed capture, with nothing on money, ClickHouse Cloud or durability across a restart. +- The setups insert through different collectors, so CPU compares only within a setup. +- Delivery was not exact: the 10x engine's table had 933 rows more than the engine returned, and the two tables logged 20 and 18 failed inserts, all but one before any scored window. + +## Run it + +``` +# from clickstack-rows/: recompute the comparison from the committed files +python3 rig/score.py results-two-arm --arms two + +# full rerun of both setups +ARMS=two MCP= GH_TOKEN=$(gh auth token) \ + CAPTURE= rig/phases.sh +``` + +The recompute needs only Python 3, and a full rerun takes 180 minutes, Docker and the capture `clickstack-e2e/run.sh` downloads. + +[RECORD.md](RECORD.md) has all three runs with their registered criteria, scores and rig notes. diff --git a/clickstack-rows/RECORD.md b/clickstack-rows/RECORD.md new file mode 100644 index 0000000..89ecc4b --- /dev/null +++ b/clickstack-rows/RECORD.md @@ -0,0 +1,215 @@ +# ClickHouse rows run: rows written into ClickStack held at a rate an hour + +One benchmark in the [log10x benchmarks](../) repository. It is the metered gate for the +rows run in log10x-mcp (`log10x_setup_recurring` with `rows_per_hour`). Each run reads the +rows ClickHouse wrote into the logs table from its own `system.part_log`, and rewrites the +Receiver's per-statement limits and service totals in lines. Lines over a service's total +go to the account's own bucket before ClickHouse receives them. + +The method was written before the run: [`METHOD.md`](METHOD.md). It is time-scaled, with +every clock divided by 6 (a run every 10 minutes, a 10-minute meter window, growth stepped +every 20 minutes). The regulator's 4-minute window and the always-pass sample of 5 are +engine constants and were not scaled. + +## Result, 2026-10-23: FAIL as registered; the rate held flat through the growth + +The engine was `ghcr.io/log-10x/edge-10x:1.1.147`, on ClickStack 2.38.0 (ClickHouse +26.5.7.64). The released OpenTelemetry demo capture was replayed from 14 to 35 lines a +second. The dry run measured 42,000 rows an hour with nothing held, so R = 29,399 rows an +hour (0.7 of that). There were 15 applied runs, the last after the feed stopped. + +| | Criterion | Result | +|---|---|---| +| 1 | Every applied run exits 0 and commits, projecting at or under R | **Pass.** 15 runs, all exit 0, each projecting about 29k rows an hour once the rate bound. | +| 2 | Each 4-minute window at or under R/15 plus the floor's share; each 10-minute meter window at or under R/6 | **Fail.** 23 of 25 four-minute windows were under their bound. 19:04 read 1.52 and 19:08 read 1.06. Every complete 10-minute window read 0.99 to 1.04 of R/6, except 19:00 at 1.36. | +| 3 | Hot plus cold equals the returned records, for the ten largest moved statements | **Fail.** 3 of 10 were exact. 7 were 1 to 11 rows over: 33 rows inserted twice into the hot table across the run (0.04%). The bucket was exact. | +| 4 | A count through the Merge table equals the local count | **Fail against the wire, exact against storage.** For all ten statements the Merge table equalled hot plus cold. The differences are the duplicates in criterion 3. | +| 5 | Every line written comes back; every returned record is in the table or the bucket | **Every returned record is stored:** 194,864 rows held 194,831 returned records, none missing, 33 duplicated. Lines in against records back is not settled here: the Receiver folds multi-line events, 242,741 lines into 194,831 records. | +| 6 | Storm: error lines of the storming statement per window at most its limit plus 50 | **Pass, read by hand.** The largest increase in a 4-minute window was about 117, against a limit of about 73 plus 50. | +| 7 | The injected service is held from its first window by the `*` row | **Pass on the regulator's windows.** Read by clock window, the first reads 96 rows against a `*` total of 49. The regulator's windows start at its first regulated line, and that clock window spans two of them, 47 and 49, each at or under the total. From its second window it held to its own row. | + +**What held.** Load grew 2.5 times, with a 20-minute error storm and a new service. +Through all of it, the rows written into the table stayed at 0.99 to 1.04 of R in every +complete 10-minute window but one, from the first growth step to the end of the hold at +35 lines a second (2.5 times the starting 14). The one, 19:00, read 1.36, and cause 2 below +is why. Before the first applied run they read 1.40 to 1.43 of R. Every line over a total went to the bucket +before ClickHouse received it: 110,392 rows there against 84,472 in the table. + +**Why criterion 2 failed: two causes.** + +1. **The always-pass floor.** Each statement passes 5 lines per window on every node, + whatever its limit. The run subtracts that floor from the allowance, but the floor + changes from window to window with the statements that print. The method allows the + floor's share in the 4-minute bound and not in the 10-minute sum, which is why most + 10-minute windows read 1 to 4% over R/6. +2. **One service printing 1,422 new statements in four minutes.** When the replay looped + the capture, Grafana's migrator ran again. The engine names that statement once per + migration id, so it arrived as 1,422 distinct patterns in one window, each passing its + always-pass sample, and no service total bounds that sample. Grafana passed 1,481 lines + in the window against a usual 140, which is the 1.52. The next applied run, ten minutes + later, sized the window after it. The rate holds against growth in known traffic. A + burst of new statements from one service overshoots by its pattern count times the + sample, until the next run. + +**Two more findings.** +- **The Receiver's Prometheus counters undercount under load.** They agreed with the meter + at the starting load (scale 1.00). At 30 and 35 lines a second they read 2.1 to 6.3 times fewer pass + lines than the meter counted, with a healthy scrape. The rows run takes totals from the + meter and only the share each statement contributes from the counters, so the hold did + not depend on them. The report prints the scale factor every run. +- **ClickStack's insert path wrote 33 rows twice** over 84,439 inserted, with no restart. + An earlier run found the same on a retry of an insert that had already committed + (`clickstack-e2e-gaps`, gap 5). + +## Rerun, 2026-10-24, with the regulator's sample bound: every 4-minute window under its bound; FAIL as registered on the 10-minute sum alone + +Same rig, image, capture, pace steps, storm and injected service as the run above, with one +change in the Receiver's configuration: the always-pass sample counts toward its service's +total, and a service passes at most 50 more sample lines per window once its total is spent +(`baselineMaxOverTotal`, log-10x/config `4253528`, declared in log-10x/modules `cbbc8c32`, both +mounted over the 1.1.147 image's own files for this run; the 1.1.150 image ships them in its configuration; `results-sample-bound/build/` records the +commits and the patches). The dry run measured the same 42,000 rows an hour, so R = 29,399 +again. There were 15 applied runs. Results in `results-sample-bound/`. + +| | Criterion | Result | +|---|---|---| +| 1 | Every applied run exits 0 and commits, projecting at or under R | **Pass.** 15 runs, all exit 0, each projecting about 29k rows an hour once the rate bound. | +| 2 | Each 4-minute window at or under R/15 plus the floor's share; each 10-minute meter window at or under R/6 | **Fail on the 10-minute sum alone.** All 25 four-minute windows were under their bound. The complete 10-minute windows read 0.97 to 1.07 of R/6; 5 of 9 were over by 3 to 7%, the floor's share the bound makes no allowance for. | +| 3 | Hot plus cold equals the returned records, for the ten largest moved statements | **Pass.** All ten exact. No row was inserted twice: 194,524 returned, 83,018 in the table and 111,506 in the bucket. | +| 4 | A count through the Merge table equals the local count | **Pass.** All ten exact. | +| 5 | Every line written comes back; every returned record is in the table or the bucket | **Every returned record is stored, none twice.** 242,357 lines became 194,524 records (the Receiver folds multi-line events). | +| 6 | Storm: error lines of the storming statement per window at most its limit plus 50 | **Pass, read by hand.** The largest increase in a 4-minute window was about 124, against the service's total of 76 to 88 plus 50. | +| 7 | The injected service is held from its first window by the `*` row | **Pass.** 86 rows in its first clock window against a `*` total of 186 in force; from the next applied run it held to its own row of 92, with 43 to 92 a window. | + +**What changed.** The capture loop inside the hold at 35 lines a second, where the first run +passed 1,481 Grafana lines in one window (1,422 new statements, each passing its sample), passed +under 200 in every window here: the rows written in that 10-minute window read 1.03 of R/6, +against 1.36 before. The only window with more than 200 Grafana passes in the whole run is the +startup window, before any total was set (1,588 lines, 1,433 statements). The 10-minute sums +sit 3 to 7% over R/6 for the reason the first run found: the floor is inside R and the +10-minute test has no allowance for it, while the 4-minute test does and holds everywhere. + +**The regulator bound, measured by the gate first.** `tests/regulator/run.sh b1-sample-bound +b2-sample-bound-off b3-sample-bound-mute` in log-10x/config: on the image's own configuration +a 200-statement burst under a 50-line total passes 400 of 400 lines; with the bound it passes +100 (the total plus 50), and `baselineMaxOverTotal: 0` restores 400. + +## Two-arm run: the rows run against a collector rule, under the same growth + +The method, written before the run, is [`METHOD_TWO_ARM.md`](METHOD_TWO_ARM.md). +`ARMS=two rig/phases.sh` runs the rows run as above (arm A, `default.otel_logs`, on the +engine image's own configuration) and, on the same node and the same feed, a second +routing collector with the collector-only rule from `clickstack-e2e-gaps` gap 6 (arm B, +[`rig/router-rule.yaml`](rig/router-rule.yaml)): the two chattiest services' INFO and DEBUG +lines go to a second bucket and the rest into `default.otel_logs_rule`, a copy of the logs +table with the same counts view and without ClickStack's `otel_logs_kv_rollup_15m` view. Arm B +has no Receiver. The two services are the capture's chattiest by lines, counted before the run: +`opentelemetry-collector` (93,875 lines) and `cart` (39,292) of 197,430, with `kafka` third at +31,029. Arm B inserts through the OpenTelemetry Collector's ClickHouse exporter straight into +its table, without ClickStack's own collector and its processing, so each arm's figures against +its own pre-step window compare like for like and the CPU figures of one arm against the +other's do not. The meter reads, per table and per window of ten rig minutes counted from the +start, NewPart rows from `system.part_log`, merge CPU from its MergeParts ProfileEvents and +insert CPU from `system.query_log`. +`rig/score.py --arms two` scores the three criteria and writes the comparison +table. Results go to `results-two-arm/`. + +```bash +ARMS=two MCP= GH_TOKEN=$(gh auth token) rig/phases.sh +``` + +### Result, 2026-10-25: rows separate the arms, CPU does not; FAIL as registered on criteria 1 and 3 + +The engine image is `ghcr.io/log-10x/edge-10x:1.1.150`, on ClickStack 2.38.0 (ClickHouse +26.5.7.64), with the capture, pace steps, storm and injected service of the rows run. From +minute 40, before the pre-step window, `old_parts_lifetime` is 60 s (ClickHouse's default is +480 s) on the 15 tables holding the most merged-away parts, both arms' logs tables among them; +the setting governs only when merged-away parts leave the disk (`timeline.jsonl`, +`old_parts_lifetime_60`). The dry run measured 43,000 rows an hour, so R = 30,099 (5,016 rows +a window). Results in `results-two-arm/`. + +| | Criterion | Result | +|---|---|---| +| 1 | Arm A's rows per window at or under 1.10 R/6, from the first growth step to the end of the hold | **Fail on one window.** Nine of ten read 0.91 to 1.09; minutes 100 to 110, the storm's first window, read 1.13. The 4-minute test: 2 of 24 windows over R/15 plus F/15, the largest at 1.15. | +| 2 | Arm B's rows per window above 1.5x its pre-step level in every window of the hold | **Pass.** 2,686 rows before the step; 7,932 to 9,179 in the hold, 2.95 to 3.42 times. | +| 3 | Insert plus merge CPU: arm A at or under 1.10 of its pre-step level after the step; arm B above 1.5x in the hold | **Fail on both halves.** Arm A read 0.87 to 1.24 of its 17.1 s, three of ten windows over. Arm B read 0.40 to 1.61 of its 122.5 s. No differentiator on CPU. | + +| Window (minutes) | Arm A rows | Arm A CPU s (insert + merge) | Arm B rows | Arm B CPU s (insert + merge) | +|---|---:|---:|---:|---:| +| 50 to 60, before the step | 4,697 | 17.1 (11.8 + 5.3) | 2,686 | 122.5 (18.6 + 103.9) | +| 120 to 130, hold | 5,377 | 20.9 (13.4 + 7.4) | 7,963 | 197.7 (38.4 + 159.3) | +| 130 to 140, hold | 4,581 | 14.9 (11.1 + 3.8) | 7,932 | 64.7 (32.9 + 31.7) | +| 140 to 150, hold | 5,443 | 18.9 (12.2 + 6.7) | 9,179 | 179.9 (38.0 + 141.9) | +| 150 to 160, hold | 5,052 | 15.2 (10.9 + 4.3) | 8,142 | 48.5 (30.7 + 17.9) | + +**What the CPU shows.** Insert CPU follows the rows: arm A's read 0.92 to 1.14 of its pre-step +level in every window after the step but one (1.40, minutes 90 to 100), and arm B's read 1.65 +to 2.06 in the hold. Merge CPU decides the windows. Arm B's inserts came four to nine times as +often as ClickStack's own collector sends arm A's (1,018 to 2,098 insert entries a window +against 240), it ran 5,125 merges against arm A's 376, and its merge CPU swung between 17.9 and +159.3 s from one hold window to the next. Read per window, criterion 3 measures which merges +the scheduler ran inside each window. + +**Run notes.** +- Delivery: arm B, 242,324 lines written and 242,365 rows in its table and bucket (41 over). + Arm A, 204,399 records returned (the Receiver folds multi-line events) and 205,346 in its table + and bucket (933 over in the table, 14 in the bucket). The tables logged 20 and 18 failed inserts, + all but one in the first 20 minutes, before any scored window. +- Rows cut share in the hold (rows to the bucket over the table's rows plus the bucket's): arm A + 0.72, arm B 0.63. +- The run's disk peaked at 1.21 GB. + +## Rows the rule kept that the rows run moved, from the two-arm run + +The two-arm run's wire tap carries every record arm A returned with its service, statement +hash and route. Arm B's rule is deterministic (lines from `opentelemetry-collector` and `cart` +at INFO or DEBUG to the bucket, the rest to the table), so applying it to the same records +gives the rule's table within 23 rows of what arm B measured (58,970 against 58,947 for +minutes 100 to 160). The question it answers: what did the rows run remove that a static rule +left in place? + +| Window | Records the rule keeps | Of those, the rows run sent to the bucket | Share | Records the rule sends out that the rows run kept | +|---|---:|---:|---:|---:| +| Minutes 100 to 160 (storm and hold) | 50,378 | 27,155 | 53.9% | 7,691 | +| Minutes 120 to 160 (hold) | 27,361 | 11,814 | 43.2% | | +| Whole run | 80,105 | 30,955 | 38.6% | | + +With the planted storm and the injected service left out, the rows run still moved 30.6% of +what the rule kept. The ten largest moved statements make up 84% of the moved records: +recommendation's `ListRecommendations`, two ad statements and kafka housekeeping lines among +them, all in services the rule keeps whole. The 7,691 records the rule would have sent out +and the rows run kept are its sample and error floors; 1,628 of them are an upload-failure +statement the engine labels ERROR and the rule's severity regex reads as INFO. The tap has +no severity field, so the floor split uses the same capture's labels from the Elastic run. + +## Files + +| Path | What | +|---|---| +| `METHOD.md` | the method, written before the run | +| `rig/` | the harness (`phases.sh` runs it: `MCP= GH_TOKEN=... rig/phases.sh`) | +| `results/score.md`, `results/final.md` | criteria 1, 2, 6 and the reports; criteria 3, 4, 5, 7 | +| `results/timeline.jsonl`, `results/meter.log` | every phase and every meter reading | +| `results/budget-runs/*/` | each run's report, files and commit | +| `results/policy.yaml`, `results/images.txt`, `results/cse-engine.log`, `results/engine-metrics.txt` | the policy, image digests, the Receiver's log and its last scrape | +| `results-two-arm/score.md`, `results-two-arm/two-arm-windows.json` | the two-arm run's three criteria and comparison table, and every window per table as ClickHouse read it at collect | + +The wire tap (428 MB) and the replay feed (251 MB) are not committed. The run writes them +again. + +## Rig notes + +- **ClickStack's collector** opens 4317 only after HyperDX hands it a config, which needs a + registered team. The rig registers inside its start loop, then waits for the port. +- **The meter user.** ClickStack's `default` user refuses HTTP from outside the container. + The rig reads through a `rows_meter` user, defined in `rig/rows_meter.xml`, with SELECT on + `system.part_log` and `system.parts` and SHOW TABLES on `otel_logs`. It has no SELECT on + the logs table. +- **The object store is RustFS.** MinIO's public images were withdrawn from quay.io and + Docker Hub (401, access denied), so the rig runs RustFS with the AWS CLI image. + +## What a run does not prove + +Nothing about money, ClickHouse Cloud or its autoscaler, 300 nodes, durability across a +restart, or write CPU against a collector rule. It holds rows on one node, from one +replayed capture. diff --git a/cloudwatch-ia-accounting/README.md b/cloudwatch-ia-accounting/README.md index 96bfd41..3f687eb 100644 --- a/cloudwatch-ia-accounting/README.md +++ b/cloudwatch-ia-accounting/README.md @@ -1,125 +1,43 @@ # CloudWatch Logs Infrequent Access accounting -One benchmark in the [log10x benchmarks](../) repository. It answers one question: -when the released engine moves events out of a CloudWatch Standard log group into an -Infrequent Access group, or offloads them to S3, under the plan the released MCP writes, -how many bytes move, and what do they cost at AWS's list prices. +When a log10x plan moves CloudWatch events out of a Standard log group, to Infrequent Access or to S3, how many bytes move and what do they cost? On a ten-hour OpenTelemetry demo capture, the plan for a 30% target moved 69.7% of the bytes to Infrequent Access, where ingest costs $0.25 a GB instead of $0.50. The figures are the 10x engine's counts, priced at us-east-1 list rates. -Everything here is computed from the engine's own counts at list price for us-east-1. -Nothing was metered by AWS, and no AWS account is needed to rerun it. +Infrequent Access halves the ingest rate and keeps the $0.03 per GB-month storage price, so a bill cut made by moving bytes there takes about twice that share of the bytes. -## The result +## Result - -The OpenTelemetry demo capture, 197,430 lines, 157,228 events after grouping, 0.215 GB over 10 hours. Engine `log10x/pipeline-10x:1.1.140`, log10x-mcp 1.30.48 with the S3 retriever declared installed, prices read 2026-10-11. - +Where the engine sent the capture's 0.215 GB at each target. - -**Dollars per GB at list.** Ingest: $0.50 Standard, $0.25 Infrequent Access. Storage: $0.03 per GB-month in both classes, on compressed bytes at AWS's stated 0.15 ratio. Logs Insights: $0.005 per GB scanned. S3 Standard: $0.023 per GB-month and $0.005 per 1,000 PUT requests. A GB moved to Infrequent Access costs $0.25 instead of $0.50 to ingest and the same to store; a GB offloaded costs nothing to ingest and $0.023 a month in S3. The 26 bytes per event in the PutLogEvents batch-size rule are shown as a separate line; no AWS page read bills them. The full table with every URL is in [`results/results.md`](results/results.md). - +| Target | Bill share the plan states | GB to Infrequent Access | GB to S3 | Share of bytes moved | +|---:|---:|---:|---:|---:| +| 30% | 30.4% | 0.150 | 0.000 | 69.7% | +| 50% | 56.9% | 0.067 | 0.099 | 77.4% | +| 70% | 70.6% | 0.002 | 0.165 | 77.7% | -**What moves, per target.** The MCP's plan applied as written, its `caps.csv` and -`actions.csv` on the released engine, with each cap scaled to the one-window replay. The -plan's percentages are of the bill, and Infrequent Access halves the ingest rate, so a cut -in dollars is about twice that share of the bytes: +At 50% and 70% the plan also sends whole containers to S3, where a GB costs nothing to ingest and $0.023 a month to store. At 70% nearly all moved bytes went there. - -| Target | Patterns planned | Containers | GB the plan states to IA | GB the engine marked to IA | GB offloaded | GB in Standard | Share of the bill the plan states | Share of the bytes the engine moved | Errors and warnings moved | -|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| 30% | 22 | 5 | 0.136 | 0.150 | 0.000 | 0.065 | 30.4% | 69.7% | 0 | -| 50% | 55 | 17 | 0.071 | 0.067 | 0.099 | 0.048 | 56.9% | 77.4% | 53 | -| 70% | 55 | 17 | 0.030 | 0.002 | 0.165 | 0.048 | 70.6% | 77.7% | 51 | - +- The 30% target moved no errors or warnings. At 50% and 70%, 53 and 51 error events went to S3 with their container, since the engine keeps only half of the error events over a cap. +- At 30% the engine moved 0.150 GB to Infrequent Access where the plan states 0.136 GB, because the plan works from a 12,000-line sample and the engine counts the whole capture. -Per target, the plan's actions by container: +## How it was measured - -- **30%**: 22 patterns; tier_down on 5 containers. -- **50%**: 55 patterns; offload on 1 container (`opentelemetry-collector`), tier_down on 16 containers; 4 planned patterns carried no container and are not in the files. -- **70%**: 55 patterns; offload on 5 containers (`ad`, `cart`, `kafka`, `opentelemetry-collector`, `recommendation`), tier_down on 12 containers; 4 planned patterns carried no container and are not in the files. - +The data is a public capture of Kubernetes container output from the OpenTelemetry demo, 197,430 lines from 36 containers. For each target, the log10x planning tool sampled it and wrote a byte cap and destination per container, and the engine replayed the whole capture against them. Bytes are the UTF-8 size of each emitted record, which the forwarder sends as the event message CloudWatch bills. Every count was taken from the emitted records and from the engine's aggregators, and the run fails if they differ. -Each tier_down container is capped at the largest pattern it keeps. Offload acts on the -whole container, its cap is the one-byte minimum, so the patterns the plan still lists as -tier_down inside an offloaded container are offloaded with it, which is where the gap -between the stated and marked Infrequent Access columns at the higher targets goes. The -engine's bytes run above the plan's because the plan is stated from a 12,000-line sample -scaled to the file and the engine counts the whole capture. +## Limits -**A month at this capture's volume, an illustration.** The capture's span scaled to a -720-hour month, at the rates above. The volume is one small demo cluster's; the rates are -what carry. +- No AWS bill or usage meter was read. Dollars use list prices read on 2026-10-11. +- The dollar figures are small because the capture comes from one small demo cluster. The share of bytes moved depends on how its logs spread across patterns. +- An Infrequent Access log group lacks metric filters, subscription filters, Live Tail, Anomaly Detection and more, so logs that feed one of these lose it when moved. Logs Insights queries still run, apart from four commands. +- The kept share of each severity is drawn at random per event, so a rerun gives slightly different counts. - -| Target | GB to IA per month | Ingest at the IA rate | Ingest at Standard the same GB would cost | GB offloaded per month | Ingest avoided by offload | -|---:|---:|---:|---:|---:|---:| -| 30% | 10.776 | $2.69 | $5.39 | 0.000 | $0 | -| 50% | 4.832 | $1.21 | $2.42 | 7.150 | $3.58 | -| 70% | 0.127 | $0.03 | $0.06 | 11.895 | $5.95 | - +## Run it -**What Infrequent Access lacks**, from AWS's log class table: Facets, natural language -query assist, Anomaly Detection, Live Tail, field indexing, compare to previous time range, -subscription filters, metric filters, Container Insights and Lambda Insights log ingestion, -embedded metric format, the GetLogEvents and FilterLogEvents API operations, and the Logs -Insights commands `pattern`, `diff`, `filterIndex` and `unmask`. Logs Insights queries -otherwise run, as do CloudWatch pipelines, export to S3, scheduled queries, KMS encryption -and masking. A log group's class is fixed when it is created. - -## Counting conditions - -- CloudWatch bills ingestion by the bytes of each event message. Bytes here are the UTF-8 - size of each record the engine emitted; with the shipped forwarder path the forwarder - sends that record as the event message. A multi-line event is one message whose lines are - joined by newlines. -- The split is forwarder-side: the `tenx.tier_down` tag goes to a log group in the - Infrequent Access class, `tenx.offload` to the S3 output. -- The regulator caps each pattern's bytes per container per 240 s window; an event over its - pattern's cap takes the container's action. Patterns under 5% of their container's bytes - in the window are not capped, the first 5 events of a pattern in a window pass, and the - severity floors keep a tenth of INFO, DEBUG and TRACE, three tenths of WARN and half of - ERROR and CRITICAL. The kept share is drawn at random per event, so a rerun differs within - that noise. -- The replay delivers the capture inside one window; each cap the plan wrote is multiplied - by the span the plan covers over 240 s, 15 here, and nothing else in the plan's files is - changed. A container takes the one action its row names; the planner gives it the strongest - action among its patterns, offload before tier_down. `results/policy-*.json` records the - files and the factor. -- Every count is taken two ways, from the emitted records and from the engine's own - aggregators, and the run fails if they differ. -- A month is 720 hours; the capture covers 10. GB are decimal. - -## Reproduce - -```sh -./run.sh # everything -./run.sh plan replay # some stages only, in the order given +``` +cd cloudwatch-ia-accounting +python3 report.py --check # recompute every figure from the committed counts +./run.sh # the full run: plan, replay, count, report ``` -Stages: `data plan control policy replay count report`. Needs Docker, Node 18 or later -and about 1 GB of disk; the whole run takes under ten minutes. `plan` installs the pinned -log10x-mcp into `.mcp/` and asks it for a plan over MCP stdio with the S3 retriever -declared installed; the server runs the pinned engine image for its own sampling. Every -other stage runs the engine image on the capture. `ACCT_TARGETS` sets the targets (default -`30 50 70`), `ACCT_WINDOW` the span the capture covers (default `10h`). - -| File | What it is | -|---|---| -| `run.sh` | the whole run, in stages | -| `prices.json` | every price used, with its URL and the date read | -| `report.py` | renders `results/results.md` and `results/results.json` from the kept outputs; `--check` fails CI if the committed figures do not follow from them | -| `../accounting-lib/` | the MCP driver, the engine run, the policy mapping and the two-way count, shared with [`datadog-flex-accounting/`](../datadog-flex-accounting/) | -| `results/plan-.json` | the plan the MCP wrote, its notes and both engine files as it rendered them | -| `results/policy-.caps.csv`, `.actions.csv`, `.json` | the files the engine read | -| `results/.tally.json`, `results/control.tally.json` | the counts per route, container, severity and pattern | -| `results/results.md` | every table, with the per-target detail | - -## Data +The check needs only Python 3. The full run needs Docker, Node 18 or later and about 1 GB of disk, takes under ten minutes and needs no AWS account. -`otel-sample-200mb.log` from -[otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1), -215,039,161 bytes, 197,430 lines, sha256 -`aa79b9349a4123d88936fd064f5eccd9fd84f45a1fd03cff3f64c259715c2432`: Kubernetes container -output captured through Fluentd from a cluster running the OpenTelemetry demo, 36 -containers, about ten hours. The same asset the [`splunk-license/`](../splunk-license/) -and [`otel-denominators/`](../otel-denominators/) benchmarks run on. +[RECORD.md](RECORD.md) has the full record: planned against moved bytes per container, every feature Infrequent Access lacks, and the counting rules. diff --git a/cloudwatch-ia-accounting/RECORD.md b/cloudwatch-ia-accounting/RECORD.md new file mode 100644 index 0000000..96bfd41 --- /dev/null +++ b/cloudwatch-ia-accounting/RECORD.md @@ -0,0 +1,125 @@ +# CloudWatch Logs Infrequent Access accounting + +One benchmark in the [log10x benchmarks](../) repository. It answers one question: +when the released engine moves events out of a CloudWatch Standard log group into an +Infrequent Access group, or offloads them to S3, under the plan the released MCP writes, +how many bytes move, and what do they cost at AWS's list prices. + +Everything here is computed from the engine's own counts at list price for us-east-1. +Nothing was metered by AWS, and no AWS account is needed to rerun it. + +## The result + + +The OpenTelemetry demo capture, 197,430 lines, 157,228 events after grouping, 0.215 GB over 10 hours. Engine `log10x/pipeline-10x:1.1.140`, log10x-mcp 1.30.48 with the S3 retriever declared installed, prices read 2026-10-11. + + + +**Dollars per GB at list.** Ingest: $0.50 Standard, $0.25 Infrequent Access. Storage: $0.03 per GB-month in both classes, on compressed bytes at AWS's stated 0.15 ratio. Logs Insights: $0.005 per GB scanned. S3 Standard: $0.023 per GB-month and $0.005 per 1,000 PUT requests. A GB moved to Infrequent Access costs $0.25 instead of $0.50 to ingest and the same to store; a GB offloaded costs nothing to ingest and $0.023 a month in S3. The 26 bytes per event in the PutLogEvents batch-size rule are shown as a separate line; no AWS page read bills them. The full table with every URL is in [`results/results.md`](results/results.md). + + +**What moves, per target.** The MCP's plan applied as written, its `caps.csv` and +`actions.csv` on the released engine, with each cap scaled to the one-window replay. The +plan's percentages are of the bill, and Infrequent Access halves the ingest rate, so a cut +in dollars is about twice that share of the bytes: + + +| Target | Patterns planned | Containers | GB the plan states to IA | GB the engine marked to IA | GB offloaded | GB in Standard | Share of the bill the plan states | Share of the bytes the engine moved | Errors and warnings moved | +|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| 30% | 22 | 5 | 0.136 | 0.150 | 0.000 | 0.065 | 30.4% | 69.7% | 0 | +| 50% | 55 | 17 | 0.071 | 0.067 | 0.099 | 0.048 | 56.9% | 77.4% | 53 | +| 70% | 55 | 17 | 0.030 | 0.002 | 0.165 | 0.048 | 70.6% | 77.7% | 51 | + + +Per target, the plan's actions by container: + + +- **30%**: 22 patterns; tier_down on 5 containers. +- **50%**: 55 patterns; offload on 1 container (`opentelemetry-collector`), tier_down on 16 containers; 4 planned patterns carried no container and are not in the files. +- **70%**: 55 patterns; offload on 5 containers (`ad`, `cart`, `kafka`, `opentelemetry-collector`, `recommendation`), tier_down on 12 containers; 4 planned patterns carried no container and are not in the files. + + +Each tier_down container is capped at the largest pattern it keeps. Offload acts on the +whole container, its cap is the one-byte minimum, so the patterns the plan still lists as +tier_down inside an offloaded container are offloaded with it, which is where the gap +between the stated and marked Infrequent Access columns at the higher targets goes. The +engine's bytes run above the plan's because the plan is stated from a 12,000-line sample +scaled to the file and the engine counts the whole capture. + +**A month at this capture's volume, an illustration.** The capture's span scaled to a +720-hour month, at the rates above. The volume is one small demo cluster's; the rates are +what carry. + + +| Target | GB to IA per month | Ingest at the IA rate | Ingest at Standard the same GB would cost | GB offloaded per month | Ingest avoided by offload | +|---:|---:|---:|---:|---:|---:| +| 30% | 10.776 | $2.69 | $5.39 | 0.000 | $0 | +| 50% | 4.832 | $1.21 | $2.42 | 7.150 | $3.58 | +| 70% | 0.127 | $0.03 | $0.06 | 11.895 | $5.95 | + + +**What Infrequent Access lacks**, from AWS's log class table: Facets, natural language +query assist, Anomaly Detection, Live Tail, field indexing, compare to previous time range, +subscription filters, metric filters, Container Insights and Lambda Insights log ingestion, +embedded metric format, the GetLogEvents and FilterLogEvents API operations, and the Logs +Insights commands `pattern`, `diff`, `filterIndex` and `unmask`. Logs Insights queries +otherwise run, as do CloudWatch pipelines, export to S3, scheduled queries, KMS encryption +and masking. A log group's class is fixed when it is created. + +## Counting conditions + +- CloudWatch bills ingestion by the bytes of each event message. Bytes here are the UTF-8 + size of each record the engine emitted; with the shipped forwarder path the forwarder + sends that record as the event message. A multi-line event is one message whose lines are + joined by newlines. +- The split is forwarder-side: the `tenx.tier_down` tag goes to a log group in the + Infrequent Access class, `tenx.offload` to the S3 output. +- The regulator caps each pattern's bytes per container per 240 s window; an event over its + pattern's cap takes the container's action. Patterns under 5% of their container's bytes + in the window are not capped, the first 5 events of a pattern in a window pass, and the + severity floors keep a tenth of INFO, DEBUG and TRACE, three tenths of WARN and half of + ERROR and CRITICAL. The kept share is drawn at random per event, so a rerun differs within + that noise. +- The replay delivers the capture inside one window; each cap the plan wrote is multiplied + by the span the plan covers over 240 s, 15 here, and nothing else in the plan's files is + changed. A container takes the one action its row names; the planner gives it the strongest + action among its patterns, offload before tier_down. `results/policy-*.json` records the + files and the factor. +- Every count is taken two ways, from the emitted records and from the engine's own + aggregators, and the run fails if they differ. +- A month is 720 hours; the capture covers 10. GB are decimal. + +## Reproduce + +```sh +./run.sh # everything +./run.sh plan replay # some stages only, in the order given +``` + +Stages: `data plan control policy replay count report`. Needs Docker, Node 18 or later +and about 1 GB of disk; the whole run takes under ten minutes. `plan` installs the pinned +log10x-mcp into `.mcp/` and asks it for a plan over MCP stdio with the S3 retriever +declared installed; the server runs the pinned engine image for its own sampling. Every +other stage runs the engine image on the capture. `ACCT_TARGETS` sets the targets (default +`30 50 70`), `ACCT_WINDOW` the span the capture covers (default `10h`). + +| File | What it is | +|---|---| +| `run.sh` | the whole run, in stages | +| `prices.json` | every price used, with its URL and the date read | +| `report.py` | renders `results/results.md` and `results/results.json` from the kept outputs; `--check` fails CI if the committed figures do not follow from them | +| `../accounting-lib/` | the MCP driver, the engine run, the policy mapping and the two-way count, shared with [`datadog-flex-accounting/`](../datadog-flex-accounting/) | +| `results/plan-.json` | the plan the MCP wrote, its notes and both engine files as it rendered them | +| `results/policy-.caps.csv`, `.actions.csv`, `.json` | the files the engine read | +| `results/.tally.json`, `results/control.tally.json` | the counts per route, container, severity and pattern | +| `results/results.md` | every table, with the per-target detail | + +## Data + +`otel-sample-200mb.log` from +[otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1), +215,039,161 bytes, 197,430 lines, sha256 +`aa79b9349a4123d88936fd064f5eccd9fd84f45a1fd03cff3f64c259715c2432`: Kubernetes container +output captured through Fluentd from a cluster running the OpenTelemetry demo, 36 +containers, about ten hours. The same asset the [`splunk-license/`](../splunk-license/) +and [`otel-denominators/`](../otel-denominators/) benchmarks run on. diff --git a/cloudwatch-ia-accounting/report.py b/cloudwatch-ia-accounting/report.py index ba8a09f..7e5cb33 100644 --- a/cloudwatch-ia-accounting/report.py +++ b/cloudwatch-ia-accounting/report.py @@ -107,7 +107,7 @@ def compute(results: pathlib.Path, prices: dict, targets: list, window_hours: fl def readme_blocks(r: dict, prices: dict) -> dict: - """Every figure the README quotes from this run, rendered from results.json and prices.json.""" + """Every figure RECORD.md quotes from this run, rendered from results.json and prices.json.""" c, cap = r['control'], r['capture'] pr = lambda k: prices[k]['price'] B = {} @@ -299,9 +299,9 @@ def main() -> int: r = compute(results, prices, run['targets'], run['window_hours'], run) if a.check: committed = json.load(open(results / 'results.json')) - stale = README.differing(open(here / 'README.md').read(), readme_blocks(r, prices)) + stale = README.differing(open(here / 'RECORD.md').read(), readme_blocks(r, prices)) if stale: - raise SystemExit(f'README.md differs from the results in: {", ".join(stale)}; run report.py to rewrite them') + raise SystemExit(f'RECORD.md differs from the results in: {", ".join(stale)}; run report.py to rewrite them') if committed != r: for k in r: if committed.get(k) != r[k]: @@ -312,9 +312,9 @@ def main() -> int: return 0 json.dump(r, open(results / 'results.json', 'w'), indent=1) open(results / 'results.md', 'w').write(render(r, prices)) - readme = here / 'README.md' + readme = here / 'RECORD.md' readme.write_text(README.apply(readme.read_text(), readme_blocks(r, prices))) - print(f'wrote {results / "results.md"} and the README\'s generated blocks') + print(f'wrote {results / "results.md"} and the generated blocks in RECORD.md') return 0 diff --git a/datadog-ceiling/README.md b/datadog-ceiling/README.md index 671b6bf..a53d103 100644 --- a/datadog-ceiling/README.md +++ b/datadog-ceiling/README.md @@ -1,224 +1,52 @@ # Datadog ceiling -One benchmark in the [log10x benchmarks](../) repository. It answers one question: with a -monthly budget X for Datadog Standard indexing, does the released Receiver, driven by the -released MCP's hourly budget run, hold Standard at X while log volume grows 2 to 3x, as -Datadog's own usage metric counts it? - -Every line went to a real Datadog org. The proof is Datadog's meter, -`datadog.estimated_usage.logs.ingested_events{datadog_is_excluded:false}` by `datadog_index`, -read hourly; numbers taken from it are labelled "meter". The rig's own captures (what the -Receiver returned, split by `routeState`) are labelled "local" and agree with the meter to -within 16 events an hour in every hour both were read. - -## The result - -Engine `log10x/edge-10x:1.1.146`, log10x-mcp 1.30.71, OpenTelemetry demo 3.1.0, one Receiver -for every container, 2026-10-20. - -| | Criterion | Result | -|---|---|---| -| 1 | Hourly Standard (`main`, meter) at or under the allowed rate plus the allowances through the growth; Flex absorbs the rest | **Pass.** 20,123, 21,020, 22,744, 21,138 and 21,080 events (meter, 04:00 to 08:00) against 22,351 allowed and 23,826 to 25,136 with the allowances. Ingested events 2.91x, 2.69x and 2.63x the baseline at 06:00, 07:00 and 08:00; Flex took 42,268, 38,957 and 37,651 (meter). | -| 2 | A statement new mid-run is held from its first window | **Pass.** A new service, `inventory`, with one new statement at 476 lines a window: 31 lines a window in Standard from its first window (the `*` row's total of 31; 35 once and 27 the next), the rest in Flex, until the 07:00 run gave it a row of its own (85). | -| 3 | The budget run runs at least twice, an hour or more apart, and its files change with spend | **Pass.** Six applied runs, hourly from 04:02 to 09:00, each a direct commit to the policy branch with a diff ([`results/budget-runs/`](results/budget-runs/)). | -| 4 | Error lines in Standard per storming error statement per node per window at most the statement's limit plus K = 50, the rule the [regulator page](https://doc.log10x.com/run/receive/rate/) states | **Pass, within one line.** The storming payment statement put 68, 66 and 69 lines a window in Standard against 68 (limit 18 plus 50), the 69 in the window that spans the 06:00 run, and 151, 118 and 119 against 188 once that run raised its limit to 138, with 328 to 460 a window moved to Flex. | -| 5 | One service with `service_budgets` held to its own number | **Pass.** `frontend`, own number $0.88 a month (1,792 events an hour): 1,774, 1,763, 1,760 and 1,770 events (meter, `service:frontend`, 05:00 to 08:00). Its 04:00 hour reads 1,957 because 165 lines passed in the 3 minutes before the first budget run; after it, 1,790 in 57 minutes. Metered spend through 07:59: 16,021 events, $0.027 of the $0.88. | -| 6 | Delivery complete; an error monitor still fires | **Pass.** 381,856 of 381,856 non-blank lines the Receiver received came back in its returned records, and every returned record went to Datadog (pass to `main`, tier_down to `tenx-offload`). A log monitor on the payment error statement in `main` went to Alert at 06:03:43 during the storm and back to OK at 06:22:43, after the storm line stopped at 06:16. | - -Month to date the budget run read 164,330 Standard events ($0.28 of $10.89) at 09:00, against -170,061 on the path X allows (59,139 at 04:02:21 plus 22,351 an hour). - -## How X and the limits were set - -Written before the first budget run in [`results/x-method.txt`](results/x-method.txt) and not -changed during the run; the numbers are in [`results/x.json`](results/x.json). - -- **Baseline.** The Receiver with the install's seed policy (every line passes), 5 load - generator users, no flags. Two full hours, 02:00 and 03:00 UTC: 22,143 and 22,576 Standard - events (meter), so B = 22,359.5 events an hour. -- **X.** Month to date from the budget run's own read at 04:02:21 (a dry run): 59,139 - Standard events, 283.96 hours left in October. X = (59,139 + 22,359.5 x 283.96) x $1.70 per - million (Standard, 15-day retention, list, the MCP's default) = **$10.89 a month**. The - allowed rate for the rest of the month is **22,351 events an hour**. -- **frontend's own number.** Its baseline was 3,592.5 events an hour (meter, `service:frontend`); - held to half of it: (8,898 spent + 1,796 x 283.96) x $1.70 per million = **$0.88 a month**, - 1,792 events an hour. It is part of X; the rest of the account gets $10.01. -- **Allowances** (criterion 1): for every window and every statement that had lines moved in - it, 5 sample lines, and 50 more for an error statement (one node). Per hour they came to - 1,475 to 2,785 events ([`results/hourly.csv`](results/hourly.csv)). -- **Fixed for the run:** keep_count 50, `standard_indexes: [main]`, `default_action: tier_down`, - one node per container, the policy's 24-hour lookback, the Receiver's values as rendered. - -## The hourly table - -Datadog's meter was read at 08:01 UTC and again at 18:04 UTC, which gave the same counts for -01:00 to 07:00 and the first complete count for 08:00. Every complete hour matched the local -count within 16 events. - -| Hour (UTC) | Phase | Standard, meter | Flex, meter | Standard, local | Flex, local | Ingested vs baseline | Allowed + allowances | frontend Standard (allowed 1,792) | -|---|---|---:|---:|---:|---:|---:|---:|---:| -| 01:00 | startup | 13,645 | 10 | 13,639 | 0 | 0.61 | | 1,582 | -| 02:00 | baseline | 22,143 | 0 | 22,144 | 0 | 0.99 | | 3,519 | -| 03:00 | baseline | 22,576 | 0 | 22,570 | 0 | 1.01 | | 3,666 | -| 04:00 | budget from 04:02 | 20,123 | 16,434 | 20,123 | 16,440 | 1.64 | 23,826 | 1,957 | -| 05:00 | budget | 21,020 | 22,090 | 21,028 | 22,093 | 1.93 | 24,166 | 1,774 | -| 06:00 | budget | 22,744 | 42,268 | 22,735 | 42,259 | 2.91 | 25,136 | 1,763 | -| 07:00 | budget | 21,138 | 38,957 | 21,138 | 38,957 | 2.69 | 24,606 | 1,760 | -| 08:00 | budget | 21,080 | 37,651 | 21,080 | 37,635 | 2.63 | 24,196 | 1,770 | - -The 01:00 hour holds the 20 routing-check lines (10 in each index). The 06:00 hour is 393 -events over the allowed rate alone and inside the allowances. - -## What ran, in order (UTC, 2026-10-20) - -| Time | Step | -|---|---| -| 01:20 | Routing check: 10 lines with `routeState:tier_down` and 10 with `pass`; the meter counted 10 in `tenx-offload` and 10 in `main` | -| 01:23 | Rig up with the seed policy; 5 users | -| 02:00 to 04:00 | Baseline | -| 04:02:53 | First budget run, X = $10.89 (commit c1d8669) | -| 04:08, 04:27 | Load 12 users, then 24 (12 gave 1.42x) | -| 05:00 | Budget run (409766a) | -| 05:08 | Load 40 users; the demo slowed (below), back to 24 at 05:21 | -| 05:22 (lines from 05:52) | Replay of the baseline capture at 0.8x its own pace, in parallel with the live demo | -| 05:32 to 06:02 | `paymentFailure` 100%; from 05:52 to 06:16 the same payment error line also sent at 2 a second | -| 06:00 | Budget run (538c500): over budget, each statement held at 2,072 an hour | -| 06:17 to 07:17 | New service `inventory`, 2 lines a second, one statement | -| 07:00, 08:00, 09:00 | Budget runs (74739d5, 028813e, 46dda27) | -| 09:05 | Demo down | - -## Datadog changes - -Two us5 orgs are named "me". The SSM keys `/l1x/tooling/datadog-api-key` and `-pat` reach the -l1x.co user's org (free plan, no Flex, org id 1300662860); the first 20 routing-check lines -(`check:rc-1792458671`) went there and are the only events this run sent to it. The run used -the talw.gx@gmail.com org (public id prefix a982f610, Pro with on-demand usage, Flex Logs -Starter) through `/l1x/tooling/datadog-run-api-key` and `-run-app-key`. - -In the run org, before the run: one index `main` (filter `*`, Standard 15 days), one read-only -integration pipeline (`python`, `source:python`), no exclusion filters, archives, log metrics, -custom destinations or monitors, no hosts, and no log usage this month -([`results/datadog-state/`](results/datadog-state/)). Changes, all left in place: - -1. Index `tenx-offload`, created by Tal in the UI: filter `@routeState:tier_down`, Standard - none, Flex 90 days, ordered before `main`; the API reports a daily limit of 200 million - events on it. Undo: delete the index in the UI. -2. Log monitor 23222383, "tenx ceiling rig: payment error statement in Standard", created at - 05:38:14 with no notification target; its query changed at 06:03:05 to select by - `@kubernetes.container_name` (see below). Undo: `DELETE /api/v1/monitor/23222383`. -3. Events: 326,871 records from the rig (pass and tier_down, local count) and the 20 - routing-check lines (`check:rc2-1792459230`). The meter counted 264,130 through its last - read at 08:01. - -## Findings - -- **Flex events count in the usage metric with `datadog_is_excluded:false`**, tagged - `datadog_index:tenx-offload`. `standard_indexes` is what keeps them out of spend. -- **The meter lags minutes, not hours, on this org**: a complete hour read 1 to 6 minutes - after it ended matched the local count within 10 events. -- **A JSON message can remove the service tag.** Datadog parses a JSON message into - attributes; pino's `service.name` field (payment's lines) replaces the `service` attribute - the forwarder set, and the usage metric counts those lines under `service:N/A`. A - `service_budgets` entry for such a service would find no metered spend under its name. The - monitor's first query (`service:payment`) matched nothing for that reason; selecting by - `@kubernetes.container_name` fixed it. -- **An error storm's Standard lines rise with the run's level.** Over its limit an error - statement passes K = 50 more per window, so its lines in Standard are the limit plus 50: - 68, 66 and 69 with the limit at 18, 151 at most with the limit at 138. They reach the - 5-line sample plus 50 only when the level is at the sample. -- **Statement rates read low during growth.** The budget run divides each statement's events - by the hours its series spans over a 24-hour lookback, so at 07:00 it read 32,447 events an - hour where about 60,000 arrived. Standard stayed bounded because each service's total is - written from the allowance and the totals add up to it. -- **The demo saturates near 2x by load alone.** On an 8-CPU Docker VM, 24 users gave 2.0x and - 40 users gave less than 24 (product-catalog at 194% CPU). Growth beyond 2x came from - replaying the baseline capture alongside the live demo. - -## Rig - -The containment rig ([`../datadog-containment/rig/`](../datadog-containment/rig/)) with these -differences, all in [`rig/`](rig/): - -- **Logs to Datadog.** Fluent Bit sends the Receiver's returned records with `routeState` to - Datadog's logs intake (`datadog` output, gzip): pass and tier_down; offload would stay local - and none occurred. `service` is the container. Local per-route files are kept. -- **The Receiver's values** come from log10x-mcp 1.30.71's own functions - ([`rig/gen-receiver-values.mjs`](rig/gen-receiver-values.mjs)): `receiverFleetValues` with - `regulator_unit: events` and keep count 50, then the Fluent Bit overlay's `renderValues` with - the Datadog metrics backend and `policy_git`. The Receiver ran with `rateReceiverUnit events`, - `rateReceiverKeepCounts` ERROR, CRITICAL and FATAL at 50, `@run/output/metric/datadog`, and - GH_ENABLED pulling `log10x/policy/*.csv` from `log-10x/config` branch - `test/datadog-ceiling-policy` ([`results/values/`](results/values/)). Substituted for a - cluster: env vars the values read from Secrets (GH_TOKEN, DD_API_KEY, DD_APP_KEY) passed from - the shell by name, and a tmpfs at `/tmp` for the chart's emptyDir. -- **Policy branch.** Seeded by the install's own seed step (`results/values/seed.sh`) with the - four files and `regulator-unit` = events; deleted after the run. Both workflows in - `log-10x/config` run on `pull_request` only, so the pushes ran nothing. -- **The budget run** is `tenx-recur` from the 1.30.71 package on a `policy.yaml` written by - the MCP's own `emitPolicyYaml` ([`rig/gen-policy.mjs`](rig/gen-policy.mjs)), gitops delivery, - `nodes_by_container` 1 for every demo container; the emitter's default branch name is the - one substitution. [`rig/budget-run.sh`](rig/budget-run.sh) keeps each run's report, files and - commit diff; [`rig/budget-phase.sh`](rig/budget-phase.sh) ran them on the hourly preset with - the load, flag and injection steps. -- **Load.** The load generator's users (`run.sh load`), the replay of the baseline capture and - the error line at a higher rate (`run.sh replay`, [`rig/forward_send.py`](rig/forward_send.py), - sent to Fluent Bit over the Forward protocol as the Docker driver sends), the new service - (`run.sh inject`), and flags through flagd-ui's own storage write (`run.sh flag`). -- **Guards.** Disk floor 4.5 GB, a stop at 2.5 million events sent, and a hard stop; none - fired. The lowest free disk was 8.6 GB. - -Rig effects worth knowing: the replay sends every container under one tag, so a multi-line -group can take a neighbour's line (about 500 email lines landed inside cart and payment -records; the delivery totals are unaffected); the Receiver's windows start at a fixed phase -(201 s past each 240 s epoch slot, measured from the data), which every per-window figure uses. - -## Versions - -| Part | Version | -|---|---| -| Receiver | `log10x/edge-10x:1.1.146@sha256:4d7be22e5e91ffe344ec9878b45063d21a9c2eb4909799a6d5a7f456db5f0c3a`, its own config and modules | -| MCP | log10x-mcp 1.30.71 from npm (tarball sha256 `a9173584294d1431939faf6066c932a1edf7fbc46958837575e27fafac4569e8`) | -| Demo | opentelemetry-demo 3.1.0, `compose.yaml` + `compose.full.yaml`, `FLAGD_CACHE=disabled` on checkout | -| Forwarder | `fluent/fluent-bit:4.2` | -| Datadog | us5, Pro with on-demand usage, Flex Logs Starter | - -## Files - -| Path | What it is | -|---|---| -| `results/results.json` | Every measure, from `rig/analyze.py` | -| `results/hourly.csv` | The hourly table with the allowances | -| `results/x.json`, `results/x-method.txt` | X, the frontend number, and how they were set | -| `results/policy.yaml` | The policy the budget runs read | -| `results/budget-runs//` | Each run: the report, the four files and the report as committed, the diff | -| `results/policy-branch-log.txt` | The policy branch's commits | -| `results/meter-hourly-0801.json`, `results/meter-hourly-1804.json` | Datadog's meter, hourly, as read at 08:01 and 18:04 | -| `results/captures/*.tsv.gz` | The rig's captures without log text: container, times, route, hash, statement, level, lines | -| `results/datadog-state/` | The org before the run, and the monitor | -| `results/timeline.jsonl` | Every step with its time | -| `results/values/` | The rendered Receiver values and what was applied | - -## Reproduce - -Prerequisites: Docker Desktop with about 8 GB for the VM; the demo at tag 3.1.0; the images -above pulled; `rig/load-generator/Dockerfile` built as `rig/load-generator:3.1.0`; log10x-mcp -1.30.71 unpacked with its dependencies installed; a Datadog org with Flex Logs and a Flex-only -index whose filter is `@routeState:tier_down`, ordered before the Standard index; its API and -application keys in SSM (or edit `rig/ddenv.sh`); `gh` signed in with write access to a policy -repo whose workflows do not run on push. +Does a monthly budget for Datadog Standard indexing stay where it was set when log volume grows two to three times? In this run it did. Log volume grew to 2.9 times the baseline, Standard indexing stayed at the budgeted rate, and every line still reached Datadog. + +The limit is set per log statement. Each hour, month-to-date usage is read from Datadog's own usage metric and a limit is written for every statement. Lines over a statement's limit go to a Flex index, so they stay searchable in Datadog. An error statement keeps 50 extra lines a window in Standard, so monitors on errors still fire. + +## Result + +Events an hour on 2026-10-20 (UTC), as Datadog's usage metric counted them. The budget allowed 22,351 an hour in Standard. + +| Hour | Phase | Standard | Flex | Ingested vs baseline | +|---|---|---:|---:|---:| +| 02:00 | baseline | 22,143 | 0 | 0.99x | +| 03:00 | baseline | 22,576 | 0 | 1.01x | +| 04:00 | budget on | 20,123 | 16,434 | 1.64x | +| 05:00 | budget on | 21,020 | 22,090 | 1.93x | +| 06:00 | budget on | 22,744 | 42,268 | 2.91x | +| 07:00 | budget on | 21,138 | 38,957 | 2.69x | +| 08:00 | budget on | 21,080 | 37,651 | 2.63x | + +Four of the five budgeted hours came in under the allowed rate. The 06:00 hour was 393 events over it, inside the allowance for sample lines and error lines, which came to between 1,475 and 2,785 events an hour. + +Three more checks from the same run: + +- A service that first appeared mid-run was limited from its first window. Of its 476 lines a window, 31 went to Standard until the 07:00 update gave it a limit of its own. +- During an error storm, a log monitor on the storming statement went to Alert at 06:03:43 and back to OK at 06:22:43, after the storm ended. +- 381,856 of 381,856 lines reached Datadog. + +## How it was measured + +The application is the OpenTelemetry demo, sending to a real Datadog org with Standard indexing at 15-day retention and one Flex index. Two baseline hours ran with no limits. Then the budget was switched on and load was raised. + +Every number in the table comes from `datadog.estimated_usage.logs.ingested_events`, filtered to `datadog_is_excluded:false` and split by `datadog_index`. The rig's own counts agreed with that metric to within 16 events an hour. + +## Limits + +- One run of under eight hours, on a demo application, with a budget of $10.89 a month. The run shows the level staying put under growth and says nothing about money saved on a real bill. +- Logs only. Hosts, custom metrics and APM are outside this benchmark. +- Lines over the limit go to Flex, which has its own price and its own query compute. Datadog monitors do not run on Flex logs, and the org needs Flex Logs on its plan. + +## Run it + +The table can be recomputed from the committed files without a Datadog account: ``` -export MCP=/path/to/log10x-mcp-1.30.71 RESULTS=$PWD/datadog-ceiling/results -node rig/gen-receiver-values.mjs $RESULTS/values -GH_TOKEN=$(gh auth token) bash $RESULTS/values/seed.sh -GH_TOKEN=$(gh auth token) rig/ddenv.sh rig/run.sh up && rig/run.sh wait-healthy -# two full hours of baseline, then X from x-method.txt and: -node rig/gen-policy.mjs $RESULTS/policy.yaml frontend $RESULTS/policy-clone -GH_TOKEN=$(gh auth token) rig/ddenv.sh rig/budget-run.sh -B0= USERS_2X=24 USERS_3X=24 GH_TOKEN=$(gh auth token) rig/ddenv.sh rig/budget-phase.sh -rig/run.sh replay start 0.8 # for growth past 2x -rig/ddenv.sh python3 rig/analyze.py # after the last hour has reported +cd datadog-ceiling +python3 rig/analyze.py ``` -`rig/analyze.py` reads `results/data/` when the full captures are there and the committed -`results/captures/` otherwise; without Datadog keys it reads the saved meter file. +A full rerun needs Docker, a Datadog org with Flex Logs and about eight hours. The steps are in [RECORD.md](RECORD.md#reproduce). + +The complete record of the run is in [RECORD.md](RECORD.md): every check, how the budget was set, each hourly budget file and the rig. diff --git a/datadog-ceiling/RECORD.md b/datadog-ceiling/RECORD.md new file mode 100644 index 0000000..671b6bf --- /dev/null +++ b/datadog-ceiling/RECORD.md @@ -0,0 +1,224 @@ +# Datadog ceiling + +One benchmark in the [log10x benchmarks](../) repository. It answers one question: with a +monthly budget X for Datadog Standard indexing, does the released Receiver, driven by the +released MCP's hourly budget run, hold Standard at X while log volume grows 2 to 3x, as +Datadog's own usage metric counts it? + +Every line went to a real Datadog org. The proof is Datadog's meter, +`datadog.estimated_usage.logs.ingested_events{datadog_is_excluded:false}` by `datadog_index`, +read hourly; numbers taken from it are labelled "meter". The rig's own captures (what the +Receiver returned, split by `routeState`) are labelled "local" and agree with the meter to +within 16 events an hour in every hour both were read. + +## The result + +Engine `log10x/edge-10x:1.1.146`, log10x-mcp 1.30.71, OpenTelemetry demo 3.1.0, one Receiver +for every container, 2026-10-20. + +| | Criterion | Result | +|---|---|---| +| 1 | Hourly Standard (`main`, meter) at or under the allowed rate plus the allowances through the growth; Flex absorbs the rest | **Pass.** 20,123, 21,020, 22,744, 21,138 and 21,080 events (meter, 04:00 to 08:00) against 22,351 allowed and 23,826 to 25,136 with the allowances. Ingested events 2.91x, 2.69x and 2.63x the baseline at 06:00, 07:00 and 08:00; Flex took 42,268, 38,957 and 37,651 (meter). | +| 2 | A statement new mid-run is held from its first window | **Pass.** A new service, `inventory`, with one new statement at 476 lines a window: 31 lines a window in Standard from its first window (the `*` row's total of 31; 35 once and 27 the next), the rest in Flex, until the 07:00 run gave it a row of its own (85). | +| 3 | The budget run runs at least twice, an hour or more apart, and its files change with spend | **Pass.** Six applied runs, hourly from 04:02 to 09:00, each a direct commit to the policy branch with a diff ([`results/budget-runs/`](results/budget-runs/)). | +| 4 | Error lines in Standard per storming error statement per node per window at most the statement's limit plus K = 50, the rule the [regulator page](https://doc.log10x.com/run/receive/rate/) states | **Pass, within one line.** The storming payment statement put 68, 66 and 69 lines a window in Standard against 68 (limit 18 plus 50), the 69 in the window that spans the 06:00 run, and 151, 118 and 119 against 188 once that run raised its limit to 138, with 328 to 460 a window moved to Flex. | +| 5 | One service with `service_budgets` held to its own number | **Pass.** `frontend`, own number $0.88 a month (1,792 events an hour): 1,774, 1,763, 1,760 and 1,770 events (meter, `service:frontend`, 05:00 to 08:00). Its 04:00 hour reads 1,957 because 165 lines passed in the 3 minutes before the first budget run; after it, 1,790 in 57 minutes. Metered spend through 07:59: 16,021 events, $0.027 of the $0.88. | +| 6 | Delivery complete; an error monitor still fires | **Pass.** 381,856 of 381,856 non-blank lines the Receiver received came back in its returned records, and every returned record went to Datadog (pass to `main`, tier_down to `tenx-offload`). A log monitor on the payment error statement in `main` went to Alert at 06:03:43 during the storm and back to OK at 06:22:43, after the storm line stopped at 06:16. | + +Month to date the budget run read 164,330 Standard events ($0.28 of $10.89) at 09:00, against +170,061 on the path X allows (59,139 at 04:02:21 plus 22,351 an hour). + +## How X and the limits were set + +Written before the first budget run in [`results/x-method.txt`](results/x-method.txt) and not +changed during the run; the numbers are in [`results/x.json`](results/x.json). + +- **Baseline.** The Receiver with the install's seed policy (every line passes), 5 load + generator users, no flags. Two full hours, 02:00 and 03:00 UTC: 22,143 and 22,576 Standard + events (meter), so B = 22,359.5 events an hour. +- **X.** Month to date from the budget run's own read at 04:02:21 (a dry run): 59,139 + Standard events, 283.96 hours left in October. X = (59,139 + 22,359.5 x 283.96) x $1.70 per + million (Standard, 15-day retention, list, the MCP's default) = **$10.89 a month**. The + allowed rate for the rest of the month is **22,351 events an hour**. +- **frontend's own number.** Its baseline was 3,592.5 events an hour (meter, `service:frontend`); + held to half of it: (8,898 spent + 1,796 x 283.96) x $1.70 per million = **$0.88 a month**, + 1,792 events an hour. It is part of X; the rest of the account gets $10.01. +- **Allowances** (criterion 1): for every window and every statement that had lines moved in + it, 5 sample lines, and 50 more for an error statement (one node). Per hour they came to + 1,475 to 2,785 events ([`results/hourly.csv`](results/hourly.csv)). +- **Fixed for the run:** keep_count 50, `standard_indexes: [main]`, `default_action: tier_down`, + one node per container, the policy's 24-hour lookback, the Receiver's values as rendered. + +## The hourly table + +Datadog's meter was read at 08:01 UTC and again at 18:04 UTC, which gave the same counts for +01:00 to 07:00 and the first complete count for 08:00. Every complete hour matched the local +count within 16 events. + +| Hour (UTC) | Phase | Standard, meter | Flex, meter | Standard, local | Flex, local | Ingested vs baseline | Allowed + allowances | frontend Standard (allowed 1,792) | +|---|---|---:|---:|---:|---:|---:|---:|---:| +| 01:00 | startup | 13,645 | 10 | 13,639 | 0 | 0.61 | | 1,582 | +| 02:00 | baseline | 22,143 | 0 | 22,144 | 0 | 0.99 | | 3,519 | +| 03:00 | baseline | 22,576 | 0 | 22,570 | 0 | 1.01 | | 3,666 | +| 04:00 | budget from 04:02 | 20,123 | 16,434 | 20,123 | 16,440 | 1.64 | 23,826 | 1,957 | +| 05:00 | budget | 21,020 | 22,090 | 21,028 | 22,093 | 1.93 | 24,166 | 1,774 | +| 06:00 | budget | 22,744 | 42,268 | 22,735 | 42,259 | 2.91 | 25,136 | 1,763 | +| 07:00 | budget | 21,138 | 38,957 | 21,138 | 38,957 | 2.69 | 24,606 | 1,760 | +| 08:00 | budget | 21,080 | 37,651 | 21,080 | 37,635 | 2.63 | 24,196 | 1,770 | + +The 01:00 hour holds the 20 routing-check lines (10 in each index). The 06:00 hour is 393 +events over the allowed rate alone and inside the allowances. + +## What ran, in order (UTC, 2026-10-20) + +| Time | Step | +|---|---| +| 01:20 | Routing check: 10 lines with `routeState:tier_down` and 10 with `pass`; the meter counted 10 in `tenx-offload` and 10 in `main` | +| 01:23 | Rig up with the seed policy; 5 users | +| 02:00 to 04:00 | Baseline | +| 04:02:53 | First budget run, X = $10.89 (commit c1d8669) | +| 04:08, 04:27 | Load 12 users, then 24 (12 gave 1.42x) | +| 05:00 | Budget run (409766a) | +| 05:08 | Load 40 users; the demo slowed (below), back to 24 at 05:21 | +| 05:22 (lines from 05:52) | Replay of the baseline capture at 0.8x its own pace, in parallel with the live demo | +| 05:32 to 06:02 | `paymentFailure` 100%; from 05:52 to 06:16 the same payment error line also sent at 2 a second | +| 06:00 | Budget run (538c500): over budget, each statement held at 2,072 an hour | +| 06:17 to 07:17 | New service `inventory`, 2 lines a second, one statement | +| 07:00, 08:00, 09:00 | Budget runs (74739d5, 028813e, 46dda27) | +| 09:05 | Demo down | + +## Datadog changes + +Two us5 orgs are named "me". The SSM keys `/l1x/tooling/datadog-api-key` and `-pat` reach the +l1x.co user's org (free plan, no Flex, org id 1300662860); the first 20 routing-check lines +(`check:rc-1792458671`) went there and are the only events this run sent to it. The run used +the talw.gx@gmail.com org (public id prefix a982f610, Pro with on-demand usage, Flex Logs +Starter) through `/l1x/tooling/datadog-run-api-key` and `-run-app-key`. + +In the run org, before the run: one index `main` (filter `*`, Standard 15 days), one read-only +integration pipeline (`python`, `source:python`), no exclusion filters, archives, log metrics, +custom destinations or monitors, no hosts, and no log usage this month +([`results/datadog-state/`](results/datadog-state/)). Changes, all left in place: + +1. Index `tenx-offload`, created by Tal in the UI: filter `@routeState:tier_down`, Standard + none, Flex 90 days, ordered before `main`; the API reports a daily limit of 200 million + events on it. Undo: delete the index in the UI. +2. Log monitor 23222383, "tenx ceiling rig: payment error statement in Standard", created at + 05:38:14 with no notification target; its query changed at 06:03:05 to select by + `@kubernetes.container_name` (see below). Undo: `DELETE /api/v1/monitor/23222383`. +3. Events: 326,871 records from the rig (pass and tier_down, local count) and the 20 + routing-check lines (`check:rc2-1792459230`). The meter counted 264,130 through its last + read at 08:01. + +## Findings + +- **Flex events count in the usage metric with `datadog_is_excluded:false`**, tagged + `datadog_index:tenx-offload`. `standard_indexes` is what keeps them out of spend. +- **The meter lags minutes, not hours, on this org**: a complete hour read 1 to 6 minutes + after it ended matched the local count within 10 events. +- **A JSON message can remove the service tag.** Datadog parses a JSON message into + attributes; pino's `service.name` field (payment's lines) replaces the `service` attribute + the forwarder set, and the usage metric counts those lines under `service:N/A`. A + `service_budgets` entry for such a service would find no metered spend under its name. The + monitor's first query (`service:payment`) matched nothing for that reason; selecting by + `@kubernetes.container_name` fixed it. +- **An error storm's Standard lines rise with the run's level.** Over its limit an error + statement passes K = 50 more per window, so its lines in Standard are the limit plus 50: + 68, 66 and 69 with the limit at 18, 151 at most with the limit at 138. They reach the + 5-line sample plus 50 only when the level is at the sample. +- **Statement rates read low during growth.** The budget run divides each statement's events + by the hours its series spans over a 24-hour lookback, so at 07:00 it read 32,447 events an + hour where about 60,000 arrived. Standard stayed bounded because each service's total is + written from the allowance and the totals add up to it. +- **The demo saturates near 2x by load alone.** On an 8-CPU Docker VM, 24 users gave 2.0x and + 40 users gave less than 24 (product-catalog at 194% CPU). Growth beyond 2x came from + replaying the baseline capture alongside the live demo. + +## Rig + +The containment rig ([`../datadog-containment/rig/`](../datadog-containment/rig/)) with these +differences, all in [`rig/`](rig/): + +- **Logs to Datadog.** Fluent Bit sends the Receiver's returned records with `routeState` to + Datadog's logs intake (`datadog` output, gzip): pass and tier_down; offload would stay local + and none occurred. `service` is the container. Local per-route files are kept. +- **The Receiver's values** come from log10x-mcp 1.30.71's own functions + ([`rig/gen-receiver-values.mjs`](rig/gen-receiver-values.mjs)): `receiverFleetValues` with + `regulator_unit: events` and keep count 50, then the Fluent Bit overlay's `renderValues` with + the Datadog metrics backend and `policy_git`. The Receiver ran with `rateReceiverUnit events`, + `rateReceiverKeepCounts` ERROR, CRITICAL and FATAL at 50, `@run/output/metric/datadog`, and + GH_ENABLED pulling `log10x/policy/*.csv` from `log-10x/config` branch + `test/datadog-ceiling-policy` ([`results/values/`](results/values/)). Substituted for a + cluster: env vars the values read from Secrets (GH_TOKEN, DD_API_KEY, DD_APP_KEY) passed from + the shell by name, and a tmpfs at `/tmp` for the chart's emptyDir. +- **Policy branch.** Seeded by the install's own seed step (`results/values/seed.sh`) with the + four files and `regulator-unit` = events; deleted after the run. Both workflows in + `log-10x/config` run on `pull_request` only, so the pushes ran nothing. +- **The budget run** is `tenx-recur` from the 1.30.71 package on a `policy.yaml` written by + the MCP's own `emitPolicyYaml` ([`rig/gen-policy.mjs`](rig/gen-policy.mjs)), gitops delivery, + `nodes_by_container` 1 for every demo container; the emitter's default branch name is the + one substitution. [`rig/budget-run.sh`](rig/budget-run.sh) keeps each run's report, files and + commit diff; [`rig/budget-phase.sh`](rig/budget-phase.sh) ran them on the hourly preset with + the load, flag and injection steps. +- **Load.** The load generator's users (`run.sh load`), the replay of the baseline capture and + the error line at a higher rate (`run.sh replay`, [`rig/forward_send.py`](rig/forward_send.py), + sent to Fluent Bit over the Forward protocol as the Docker driver sends), the new service + (`run.sh inject`), and flags through flagd-ui's own storage write (`run.sh flag`). +- **Guards.** Disk floor 4.5 GB, a stop at 2.5 million events sent, and a hard stop; none + fired. The lowest free disk was 8.6 GB. + +Rig effects worth knowing: the replay sends every container under one tag, so a multi-line +group can take a neighbour's line (about 500 email lines landed inside cart and payment +records; the delivery totals are unaffected); the Receiver's windows start at a fixed phase +(201 s past each 240 s epoch slot, measured from the data), which every per-window figure uses. + +## Versions + +| Part | Version | +|---|---| +| Receiver | `log10x/edge-10x:1.1.146@sha256:4d7be22e5e91ffe344ec9878b45063d21a9c2eb4909799a6d5a7f456db5f0c3a`, its own config and modules | +| MCP | log10x-mcp 1.30.71 from npm (tarball sha256 `a9173584294d1431939faf6066c932a1edf7fbc46958837575e27fafac4569e8`) | +| Demo | opentelemetry-demo 3.1.0, `compose.yaml` + `compose.full.yaml`, `FLAGD_CACHE=disabled` on checkout | +| Forwarder | `fluent/fluent-bit:4.2` | +| Datadog | us5, Pro with on-demand usage, Flex Logs Starter | + +## Files + +| Path | What it is | +|---|---| +| `results/results.json` | Every measure, from `rig/analyze.py` | +| `results/hourly.csv` | The hourly table with the allowances | +| `results/x.json`, `results/x-method.txt` | X, the frontend number, and how they were set | +| `results/policy.yaml` | The policy the budget runs read | +| `results/budget-runs//` | Each run: the report, the four files and the report as committed, the diff | +| `results/policy-branch-log.txt` | The policy branch's commits | +| `results/meter-hourly-0801.json`, `results/meter-hourly-1804.json` | Datadog's meter, hourly, as read at 08:01 and 18:04 | +| `results/captures/*.tsv.gz` | The rig's captures without log text: container, times, route, hash, statement, level, lines | +| `results/datadog-state/` | The org before the run, and the monitor | +| `results/timeline.jsonl` | Every step with its time | +| `results/values/` | The rendered Receiver values and what was applied | + +## Reproduce + +Prerequisites: Docker Desktop with about 8 GB for the VM; the demo at tag 3.1.0; the images +above pulled; `rig/load-generator/Dockerfile` built as `rig/load-generator:3.1.0`; log10x-mcp +1.30.71 unpacked with its dependencies installed; a Datadog org with Flex Logs and a Flex-only +index whose filter is `@routeState:tier_down`, ordered before the Standard index; its API and +application keys in SSM (or edit `rig/ddenv.sh`); `gh` signed in with write access to a policy +repo whose workflows do not run on push. + +``` +export MCP=/path/to/log10x-mcp-1.30.71 RESULTS=$PWD/datadog-ceiling/results +node rig/gen-receiver-values.mjs $RESULTS/values +GH_TOKEN=$(gh auth token) bash $RESULTS/values/seed.sh +GH_TOKEN=$(gh auth token) rig/ddenv.sh rig/run.sh up && rig/run.sh wait-healthy +# two full hours of baseline, then X from x-method.txt and: +node rig/gen-policy.mjs $RESULTS/policy.yaml frontend $RESULTS/policy-clone +GH_TOKEN=$(gh auth token) rig/ddenv.sh rig/budget-run.sh +B0= USERS_2X=24 USERS_3X=24 GH_TOKEN=$(gh auth token) rig/ddenv.sh rig/budget-phase.sh +rig/run.sh replay start 0.8 # for growth past 2x +rig/ddenv.sh python3 rig/analyze.py # after the last hour has reported +``` + +`rig/analyze.py` reads `results/data/` when the full captures are there and the committed +`results/captures/` otherwise; without Datadog keys it reads the saved meter file. diff --git a/datadog-flex-accounting/README.md b/datadog-flex-accounting/README.md index 12f02ca..3472e22 100644 --- a/datadog-flex-accounting/README.md +++ b/datadog-flex-accounting/README.md @@ -1,162 +1,47 @@ # Datadog Flex Logs accounting -One benchmark in the [log10x benchmarks](../) repository. It answers one question: -when the released engine moves events out of Datadog's Standard index into Flex Logs -under the plan the released MCP writes, how many events and GB move, and what do those -events cost at Datadog's list prices. - -Everything here is computed from the engine's own counts at list price. Nothing was -metered by Datadog, and no Datadog account is needed to rerun it. - -## The result - - -The OpenTelemetry demo capture, 197,430 lines, 157,228 events after grouping, 0.215 GB over 10 hours. Engine `log10x/pipeline-10x:1.1.140`, log10x-mcp 1.30.48, prices read 2026-10-13. - - - -**What a moved event costs at list.** Standard indexing avoided: $1.70 per million events at 15-day retention and $2.50 at 30-day, annual; $2.55 and $3.75 on demand. What Flex adds depends on the tier the volume falls in. Within Flex Starter, below 2 billion events a month and 10 billion stored, storage and compute are billed together at $0.60 per million stored per month, annual ($0.90 on demand), for at least 3 months: Datadog's Flex docs give 3, 6, 12 or 15 months and its price list 6, 12 or 15. A moved event held 3 months costs at least $1.80 per million over its life. On the scalable tiers above Starter, XS to L, storage is $0.05 per million stored per month, annual ($0.075 on demand), held at least 30 days, plus a flat compute tier Datadog prices on request ("Contact Us for Pricing"). At this capture's volume every target falls in Starter, and tier down to Flex Starter costs more than the 15-day indexing it avoids and less than the 30-day indexing. Ingestion, $0.10 per GB, is paid once on every event and unchanged by a move. The full rate table with every URL is in [`results/results.md`](results/results.md). - - -**What moves, per target.** The MCP's plan applied as written, its `caps.csv` and -`actions.csv` on the released engine, with each cap scaled to the one-window replay: - - -| Target | Patterns planned | Containers capped | Events the plan states | Events the engine marked tier_down | Share of the volume the plan states | Share of the bytes the engine moved | Events in Standard | Errors and warnings to Flex | -|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| 30% | 3 | 1 | 60,783 | 67,505 | 40.3% | 39.9% | 89,723 | 0 | -| 50% | 7 | 4 | 82,585 | 90,555 | 50.6% | 56.1% | 66,673 | 0 | -| 70% | 55 | 17 | 112,944 | 119,833 | 68.4% | 71.2% | 37,395 | 0 | - - -The plan moves whole patterns, so a target lands on the first set of patterns that reaches -it, each container capped at the largest pattern it keeps: - - -- **30%**: 3 patterns in 1 container (`opentelemetry-collector`: `Traces`, `Logs`, `Metrics`). -- **50%**: 7 patterns in 4 containers (`opentelemetry-collector`: `Traces`, `Logs`, `Metrics`; `cart`: `cart_cartstore_ValkeyCartStore`; `recommendation`: `Receive_ListRecommendations_for_product_ids`; `ad`: `oteldemo_AdService_Targeted_ad_request_received_for`, `oteldemo_AdService_no_baggage_found_in_context_trace_id_span_flags`). -- **70%**: 55 patterns in 17 containers; 4 planned patterns carried no container and are not in the files. - - -The engine moves each pattern's excess over its container's cap, less the share the -severity floors keep. Its figure runs above the plan's because the plan is stated from a -12,000-line sample scaled to the file and the engine counts the whole capture, and because a -container capped at one byte (nothing kept) moves every pattern it has. - -**A month at this capture's volume, an illustration.** The capture's span scaled to a -720-hour month, at the rates above, with every moved event held the shortest retention its -Flex tier sells. The volume is one small demo cluster's; what carries is the rate at the tier -a volume falls in. - - -| Target | Events to Flex per month | Flex tier | Flex added at that tier, annual | Standard indexing avoided, 15-day annual | Net, 15-day | Standard indexing avoided, 30-day annual | Net, 30-day | Custom metrics above a Pro allotment, 10 to 300 nodes | -|---:|---:|---|---:|---:|---:|---:|---:|---:| -| 30% | 4,860,360 | Starter | $8.75, held 3 months | $8.26 | -$0.49 | $12.15 | +$3.40 | $0 | -| 50% | 6,519,960 | Starter | $11.74, held 3 months | $11.08 | -$0.66 | $16.30 | +$4.56 | $0 | -| 70% | 8,627,976 | Starter | $15.53, held 3 months | $14.67 | -$0.86 | $21.57 | +$6.04 | $0 | - -Net is the indexing avoided less what Flex adds; a minus sign means the move costs more than it saves. At 6 months, the shortest Starter retention on Datadog's price-list page (its Flex Logs page and pricing page list 3, 6, 12 or 15 months, and the table above holds 3), the Starter rows cost $17.50 at 30%, $23.47 at 50% and $31.06 at 70% a month. - - - -**Custom metrics.** The `metrics` stage replays the capture through the engine's shipped Datadog metric output, posting to a local stand-in for Datadog's intake; the one change to the image's config is the URI it posts to. Each series carried `k8s_container`, `k8s_namespace`, `message_pattern`, `routeState`, `severity_level`, `tenx_hash`, `tenx_reported_name`, `tenx_user_service`, and no host, pod or pipeline id, so every node posts the same series. The output keeps to the 200 tag sets with the most bytes, each from both aggregators: at most 776 series in one post without a move, and 768, 786 and 780 at 30, 50 and 70%. A Pro plan includes 100 custom metrics per host, so at 10, 100 and 300 nodes the line is $0. While ranks settle, tag sets enter and leave the top, and the replay, which delivers 10 hours in under a minute, saw 1,386 to 1,590 distinct series in all; on a live stream that settling happens once. Every tag set from every node would be 10,836 series per node, $5,368, $53,680 and $161,040 a month at $5 per 100. - - -## Native filters at equal events - -The question a Datadog buyer asks is whether a per-pattern plan moves anything their own -index filters would not. So each target has a control arm computed from the same capture, -engine and counts: what a person writes in Datadog today, an index filter per service. - -**Selection rule, fixed before the run.** Datadog bills Standard indexing per million -events, so the arms are matched on events, with bytes alongside. Per target, the control -takes the fewest `service:` filters, largest by events first, whose events reach -at least the events the engine marked tier_down under the plan, and reports the overshoot -without trimming it. A second variant adds `-status:(error OR warn)` to each filter, which -leaves ERROR and WARN events in Standard; CRITICAL maps to Datadog's `critical` status and -is not excluded by that clause. Errors and warnings are counted as the plan's column counts -them: ERROR, WARN, CRITICAL and FATAL. - -**What would show the key adds nothing.** At a target, if the filters with the status -exclusion move no more errors and warnings than the plan, and none of the events of the -patterns the plan keeps under a container's cap (patterns in a capped container that the -engine marked nothing of), then per-pattern control adds nothing on this capture at that -target. The table is published as it comes out. - - -| Target | Arm | Index filters | Events moved | Over the plan | GB moved | Errors and warnings moved | Events of patterns the plan keeps under a cap | Events the plan moves that stay in Standard | -|---:|---|---|---:|---:|---:|---:|---:|---:| -| 30% | per-pattern plan | caps on 1 container | 67,505 | 0 | 0.086 | 0 | 0 | 0 | -| 30% | service filters | `service:opentelemetry-collector` | 89,739 | +22,234 | 0.118 | 3,953 | 4,000 | 0 | -| 30% | service filters, `-status:(error OR warn)` | `service:opentelemetry-collector` | 85,786 | +18,281 | 0.109 | 0 | 47 | 0 | -| 50% | per-pattern plan | caps on 4 containers | 90,555 | 0 | 0.121 | 0 | 0 | 0 | -| 50% | service filters | `service:opentelemetry-collector`, `service:kafka` | 118,778 | +28,223 | 0.149 | 3,954 | 4,000 | 23,161 | -| 50% | service filters, `-status:(error OR warn)` | `service:opentelemetry-collector`, `service:kafka` | 114,824 | +24,269 | 0.140 | 0 | 47 | 23,161 | -| 70% | per-pattern plan | caps on 17 containers | 119,833 | 0 | 0.153 | 0 | 0 | 0 | -| 70% | service filters | `service:opentelemetry-collector`, `service:kafka`, `service:cart` | 138,424 | +18,591 | 0.183 | 3,955 | 4,795 | 11,181 | -| 70% | service filters, `-status:(error OR warn)` | `service:opentelemetry-collector`, `service:kafka`, `service:cart` | 134,469 | +14,636 | 0.174 | 0 | 840 | 11,181 | - - - -- **30%**: the falsifier does not hold. With the status exclusion, the filters move 47 events of 22 patterns the plan keeps under a cap (0.05% of the 85,786 events the filters move). They move 18,281 more events than the plan's 67,505, which the plan leaves in Standard. -- **50%**: the falsifier does not hold. With the status exclusion, the filters move 47 events of 22 patterns the plan keeps under a cap (0.04% of the 114,824 events the filters move). They move 24,269 more events than the plan's 90,555, which the plan leaves in Standard, and leave in Standard 23,161 events the plan moves, in services they do not name. -- **70%**: the falsifier does not hold. With the status exclusion, the filters move 840 events of 435 patterns the plan keeps under a cap (0.62% of the 134,469 events the filters move). They move 14,636 more events than the plan's 119,833, which the plan leaves in Standard, and leave in Standard 11,181 events the plan moves, in services they do not name. - - -## Counting conditions - -- Datadog counts indexed events; with the shipped forwarder path one record is one event, - and the Receiver groups the lines of a multi-line event into one record. -- The Flex route is Datadog's index filter on `@routeState:tier_down`, on a Flex index - listed before the Standard one. Every event the engine marks `tier_down` carries that - field. -- The regulator caps each pattern's bytes per container per 240 s window; an event over its - pattern's cap takes the container's action. Patterns under 5% of their container's bytes - in the window are not capped, the first 5 events of a pattern in a window pass, and the - severity floors keep a tenth of INFO, DEBUG and TRACE, three tenths of WARN and half of - ERROR and CRITICAL. The kept share is drawn at random per event, so a rerun differs within - that noise. -- The replay delivers the capture inside one window; each cap the plan wrote is multiplied - by the span the plan covers over 240 s, 15 here, and nothing else in the plan's files is - changed. `results/policy-*.json` records the files and the factor. -- Every count is taken two ways, from the emitted records and from the engine's own - aggregators, and the run fails if they differ. -- A month is 720 hours; the capture covers 10. GB are decimal. - -## Reproduce - -```sh -./run.sh # everything -./run.sh plan replay # some stages only, in the order given +When the 10x engine moves Datadog log events from Standard indexing to Flex Logs, does the move save money at list price? On a 10-hour OpenTelemetry demo capture of 157,228 events, plans targeting 30%, 50% and 70% of the volume moved 67,505 to 119,833 events, none of them errors or warnings. At this volume every move lands in Flex Starter, which saves money against 30-day Standard indexing and costs more than 15-day indexing. + +On annual pricing, Standard indexing lists at $1.70 per million events for 15-day retention and $2.50 for 30-day. Flex Starter lists at $0.60 per million stored per month for at least 3 months, so a moved event costs at least $1.80 per million. That sits between the two indexing prices. + +## Result + +Net is the Standard indexing avoided minus what Flex Starter adds, per month at this capture's volume, and a minus means the move costs more than it saves. + +| Target | Events moved to Flex | Share of bytes moved | Net vs 15-day | Net vs 30-day | +|---:|---:|---:|---:|---:| +| 30% | 67,505 | 39.9% | -$0.49 | +$3.40 | +| 50% | 90,555 | 56.1% | -$0.66 | +$4.56 | +| 70% | 119,833 | 71.2% | -$0.86 | +$6.04 | + +The dollars are small because the demo cluster is small. + +Against Datadog's own index filters, one per service and matched on events: + +- At 30%, one filter on `service:opentelemetry-collector` with `-status:(error OR warn)` does nearly the same job. Of the 18,281 extra events it moves, 47 come from patterns the plan leaves entirely in Standard. +- At 50% and 70%, the filters leave 23,161 and 11,181 events in Standard that the plan moves. +- Without the status clause, every filter set moves 3,953 to 3,955 errors and warnings to Flex. + +## How it was measured + +The capture is Kubernetes output from 36 OpenTelemetry demo containers, 197,430 lines (157,228 events after multi-line grouping) and 0.215 GB. For each target, `log10x-mcp` sampled 12,000 lines and wrote a plan that moves whole log patterns. The released 10x engine replayed the full capture under that plan, counting every event from its output records and its own aggregators, and the run fails if the two differ. Dollars are those counts at Datadog list prices read on 2026-10-13. Datadog metered nothing. + +## Limits + +- One small demo capture, replayed rather than sent to Datadog. Monthly dollars scale its 10 hours to a 720-hour month. +- Only Flex Starter is priced. Above 2 billion events a month, Datadog prices Flex compute on request. +- Datadog's price list gives Starter retention as 6, 12 or 15 months, and its Flex docs add the 3 the table uses. At 6 months Starter costs $17.50, $23.47 and $31.06 a month, more than the indexing avoided at either retention. + +## Run it + +``` +# from the repository root: recompute every figure from the committed counts and prices +python3 datadog-flex-accounting/report.py --check + +# full rerun +cd datadog-flex-accounting && ./run.sh ``` -Stages: `data plan control policy replay count report`. Needs Docker, Node 18 or later -and about 1 GB of disk; the whole run takes under ten minutes. `plan` installs the pinned -log10x-mcp into `.mcp/` and asks it for a plan over MCP stdio; the server runs the pinned -engine image for its own sampling. Every other stage runs the engine image on the capture. -`ACCT_TARGETS` sets the targets (default `30 50 70`), `ACCT_WINDOW` the span the capture -covers (default `10h`). - -| File | What it is | -|---|---| -| `run.sh` | the whole run, in stages | -| `prices.json` | every price used, with its URL and the date read | -| `report.py` | renders `results/results.md` and `results/results.json` from the kept outputs; `--check` fails CI if the committed figures do not follow from them | -| `../accounting-lib/plan.mjs` | asks the released MCP for its plan over stdio and keeps what it wrote | -| `../accounting-lib/replay.config.yaml` | the engine run: shipped enrichment, regulator and aggregators, file in, file out | -| `../accounting-lib/policy.py` | the engine files the tool wrote, caps scaled to the one-window replay | -| `../accounting-lib/count.py` | the two counts, records and aggregators, required to agree | -| `results/plan-.json` | the plan the MCP wrote, its notes and both engine files as it rendered them | -| `results/policy-.caps.csv`, `.actions.csv`, `.json` | the files the engine read | -| `results/.tally.json`, `results/control.tally.json` | the counts per route, container, severity and pattern | -| `results/results.md` | every table, with the per-target detail | - -## Data - -`otel-sample-200mb.log` from -[otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1), -215,039,161 bytes, 197,430 lines, sha256 -`aa79b9349a4123d88936fd064f5eccd9fd84f45a1fd03cff3f64c259715c2432`: Kubernetes container -output captured through Fluentd from a cluster running the OpenTelemetry demo, 36 -containers, about ten hours. The same asset the [`splunk-license/`](../splunk-license/) -and [`otel-denominators/`](../otel-denominators/) benchmarks run on. +The full rerun needs Docker, Node 18 or later and about 1 GB of disk. A run takes under ten minutes and needs no Datadog account. + +[RECORD.md](RECORD.md) has every price source, the plan and engine files per target, the full filter comparison and the counting conditions. diff --git a/datadog-flex-accounting/RECORD.md b/datadog-flex-accounting/RECORD.md new file mode 100644 index 0000000..12f02ca --- /dev/null +++ b/datadog-flex-accounting/RECORD.md @@ -0,0 +1,162 @@ +# Datadog Flex Logs accounting + +One benchmark in the [log10x benchmarks](../) repository. It answers one question: +when the released engine moves events out of Datadog's Standard index into Flex Logs +under the plan the released MCP writes, how many events and GB move, and what do those +events cost at Datadog's list prices. + +Everything here is computed from the engine's own counts at list price. Nothing was +metered by Datadog, and no Datadog account is needed to rerun it. + +## The result + + +The OpenTelemetry demo capture, 197,430 lines, 157,228 events after grouping, 0.215 GB over 10 hours. Engine `log10x/pipeline-10x:1.1.140`, log10x-mcp 1.30.48, prices read 2026-10-13. + + + +**What a moved event costs at list.** Standard indexing avoided: $1.70 per million events at 15-day retention and $2.50 at 30-day, annual; $2.55 and $3.75 on demand. What Flex adds depends on the tier the volume falls in. Within Flex Starter, below 2 billion events a month and 10 billion stored, storage and compute are billed together at $0.60 per million stored per month, annual ($0.90 on demand), for at least 3 months: Datadog's Flex docs give 3, 6, 12 or 15 months and its price list 6, 12 or 15. A moved event held 3 months costs at least $1.80 per million over its life. On the scalable tiers above Starter, XS to L, storage is $0.05 per million stored per month, annual ($0.075 on demand), held at least 30 days, plus a flat compute tier Datadog prices on request ("Contact Us for Pricing"). At this capture's volume every target falls in Starter, and tier down to Flex Starter costs more than the 15-day indexing it avoids and less than the 30-day indexing. Ingestion, $0.10 per GB, is paid once on every event and unchanged by a move. The full rate table with every URL is in [`results/results.md`](results/results.md). + + +**What moves, per target.** The MCP's plan applied as written, its `caps.csv` and +`actions.csv` on the released engine, with each cap scaled to the one-window replay: + + +| Target | Patterns planned | Containers capped | Events the plan states | Events the engine marked tier_down | Share of the volume the plan states | Share of the bytes the engine moved | Events in Standard | Errors and warnings to Flex | +|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| 30% | 3 | 1 | 60,783 | 67,505 | 40.3% | 39.9% | 89,723 | 0 | +| 50% | 7 | 4 | 82,585 | 90,555 | 50.6% | 56.1% | 66,673 | 0 | +| 70% | 55 | 17 | 112,944 | 119,833 | 68.4% | 71.2% | 37,395 | 0 | + + +The plan moves whole patterns, so a target lands on the first set of patterns that reaches +it, each container capped at the largest pattern it keeps: + + +- **30%**: 3 patterns in 1 container (`opentelemetry-collector`: `Traces`, `Logs`, `Metrics`). +- **50%**: 7 patterns in 4 containers (`opentelemetry-collector`: `Traces`, `Logs`, `Metrics`; `cart`: `cart_cartstore_ValkeyCartStore`; `recommendation`: `Receive_ListRecommendations_for_product_ids`; `ad`: `oteldemo_AdService_Targeted_ad_request_received_for`, `oteldemo_AdService_no_baggage_found_in_context_trace_id_span_flags`). +- **70%**: 55 patterns in 17 containers; 4 planned patterns carried no container and are not in the files. + + +The engine moves each pattern's excess over its container's cap, less the share the +severity floors keep. Its figure runs above the plan's because the plan is stated from a +12,000-line sample scaled to the file and the engine counts the whole capture, and because a +container capped at one byte (nothing kept) moves every pattern it has. + +**A month at this capture's volume, an illustration.** The capture's span scaled to a +720-hour month, at the rates above, with every moved event held the shortest retention its +Flex tier sells. The volume is one small demo cluster's; what carries is the rate at the tier +a volume falls in. + + +| Target | Events to Flex per month | Flex tier | Flex added at that tier, annual | Standard indexing avoided, 15-day annual | Net, 15-day | Standard indexing avoided, 30-day annual | Net, 30-day | Custom metrics above a Pro allotment, 10 to 300 nodes | +|---:|---:|---|---:|---:|---:|---:|---:|---:| +| 30% | 4,860,360 | Starter | $8.75, held 3 months | $8.26 | -$0.49 | $12.15 | +$3.40 | $0 | +| 50% | 6,519,960 | Starter | $11.74, held 3 months | $11.08 | -$0.66 | $16.30 | +$4.56 | $0 | +| 70% | 8,627,976 | Starter | $15.53, held 3 months | $14.67 | -$0.86 | $21.57 | +$6.04 | $0 | + +Net is the indexing avoided less what Flex adds; a minus sign means the move costs more than it saves. At 6 months, the shortest Starter retention on Datadog's price-list page (its Flex Logs page and pricing page list 3, 6, 12 or 15 months, and the table above holds 3), the Starter rows cost $17.50 at 30%, $23.47 at 50% and $31.06 at 70% a month. + + + +**Custom metrics.** The `metrics` stage replays the capture through the engine's shipped Datadog metric output, posting to a local stand-in for Datadog's intake; the one change to the image's config is the URI it posts to. Each series carried `k8s_container`, `k8s_namespace`, `message_pattern`, `routeState`, `severity_level`, `tenx_hash`, `tenx_reported_name`, `tenx_user_service`, and no host, pod or pipeline id, so every node posts the same series. The output keeps to the 200 tag sets with the most bytes, each from both aggregators: at most 776 series in one post without a move, and 768, 786 and 780 at 30, 50 and 70%. A Pro plan includes 100 custom metrics per host, so at 10, 100 and 300 nodes the line is $0. While ranks settle, tag sets enter and leave the top, and the replay, which delivers 10 hours in under a minute, saw 1,386 to 1,590 distinct series in all; on a live stream that settling happens once. Every tag set from every node would be 10,836 series per node, $5,368, $53,680 and $161,040 a month at $5 per 100. + + +## Native filters at equal events + +The question a Datadog buyer asks is whether a per-pattern plan moves anything their own +index filters would not. So each target has a control arm computed from the same capture, +engine and counts: what a person writes in Datadog today, an index filter per service. + +**Selection rule, fixed before the run.** Datadog bills Standard indexing per million +events, so the arms are matched on events, with bytes alongside. Per target, the control +takes the fewest `service:` filters, largest by events first, whose events reach +at least the events the engine marked tier_down under the plan, and reports the overshoot +without trimming it. A second variant adds `-status:(error OR warn)` to each filter, which +leaves ERROR and WARN events in Standard; CRITICAL maps to Datadog's `critical` status and +is not excluded by that clause. Errors and warnings are counted as the plan's column counts +them: ERROR, WARN, CRITICAL and FATAL. + +**What would show the key adds nothing.** At a target, if the filters with the status +exclusion move no more errors and warnings than the plan, and none of the events of the +patterns the plan keeps under a container's cap (patterns in a capped container that the +engine marked nothing of), then per-pattern control adds nothing on this capture at that +target. The table is published as it comes out. + + +| Target | Arm | Index filters | Events moved | Over the plan | GB moved | Errors and warnings moved | Events of patterns the plan keeps under a cap | Events the plan moves that stay in Standard | +|---:|---|---|---:|---:|---:|---:|---:|---:| +| 30% | per-pattern plan | caps on 1 container | 67,505 | 0 | 0.086 | 0 | 0 | 0 | +| 30% | service filters | `service:opentelemetry-collector` | 89,739 | +22,234 | 0.118 | 3,953 | 4,000 | 0 | +| 30% | service filters, `-status:(error OR warn)` | `service:opentelemetry-collector` | 85,786 | +18,281 | 0.109 | 0 | 47 | 0 | +| 50% | per-pattern plan | caps on 4 containers | 90,555 | 0 | 0.121 | 0 | 0 | 0 | +| 50% | service filters | `service:opentelemetry-collector`, `service:kafka` | 118,778 | +28,223 | 0.149 | 3,954 | 4,000 | 23,161 | +| 50% | service filters, `-status:(error OR warn)` | `service:opentelemetry-collector`, `service:kafka` | 114,824 | +24,269 | 0.140 | 0 | 47 | 23,161 | +| 70% | per-pattern plan | caps on 17 containers | 119,833 | 0 | 0.153 | 0 | 0 | 0 | +| 70% | service filters | `service:opentelemetry-collector`, `service:kafka`, `service:cart` | 138,424 | +18,591 | 0.183 | 3,955 | 4,795 | 11,181 | +| 70% | service filters, `-status:(error OR warn)` | `service:opentelemetry-collector`, `service:kafka`, `service:cart` | 134,469 | +14,636 | 0.174 | 0 | 840 | 11,181 | + + + +- **30%**: the falsifier does not hold. With the status exclusion, the filters move 47 events of 22 patterns the plan keeps under a cap (0.05% of the 85,786 events the filters move). They move 18,281 more events than the plan's 67,505, which the plan leaves in Standard. +- **50%**: the falsifier does not hold. With the status exclusion, the filters move 47 events of 22 patterns the plan keeps under a cap (0.04% of the 114,824 events the filters move). They move 24,269 more events than the plan's 90,555, which the plan leaves in Standard, and leave in Standard 23,161 events the plan moves, in services they do not name. +- **70%**: the falsifier does not hold. With the status exclusion, the filters move 840 events of 435 patterns the plan keeps under a cap (0.62% of the 134,469 events the filters move). They move 14,636 more events than the plan's 119,833, which the plan leaves in Standard, and leave in Standard 11,181 events the plan moves, in services they do not name. + + +## Counting conditions + +- Datadog counts indexed events; with the shipped forwarder path one record is one event, + and the Receiver groups the lines of a multi-line event into one record. +- The Flex route is Datadog's index filter on `@routeState:tier_down`, on a Flex index + listed before the Standard one. Every event the engine marks `tier_down` carries that + field. +- The regulator caps each pattern's bytes per container per 240 s window; an event over its + pattern's cap takes the container's action. Patterns under 5% of their container's bytes + in the window are not capped, the first 5 events of a pattern in a window pass, and the + severity floors keep a tenth of INFO, DEBUG and TRACE, three tenths of WARN and half of + ERROR and CRITICAL. The kept share is drawn at random per event, so a rerun differs within + that noise. +- The replay delivers the capture inside one window; each cap the plan wrote is multiplied + by the span the plan covers over 240 s, 15 here, and nothing else in the plan's files is + changed. `results/policy-*.json` records the files and the factor. +- Every count is taken two ways, from the emitted records and from the engine's own + aggregators, and the run fails if they differ. +- A month is 720 hours; the capture covers 10. GB are decimal. + +## Reproduce + +```sh +./run.sh # everything +./run.sh plan replay # some stages only, in the order given +``` + +Stages: `data plan control policy replay count report`. Needs Docker, Node 18 or later +and about 1 GB of disk; the whole run takes under ten minutes. `plan` installs the pinned +log10x-mcp into `.mcp/` and asks it for a plan over MCP stdio; the server runs the pinned +engine image for its own sampling. Every other stage runs the engine image on the capture. +`ACCT_TARGETS` sets the targets (default `30 50 70`), `ACCT_WINDOW` the span the capture +covers (default `10h`). + +| File | What it is | +|---|---| +| `run.sh` | the whole run, in stages | +| `prices.json` | every price used, with its URL and the date read | +| `report.py` | renders `results/results.md` and `results/results.json` from the kept outputs; `--check` fails CI if the committed figures do not follow from them | +| `../accounting-lib/plan.mjs` | asks the released MCP for its plan over stdio and keeps what it wrote | +| `../accounting-lib/replay.config.yaml` | the engine run: shipped enrichment, regulator and aggregators, file in, file out | +| `../accounting-lib/policy.py` | the engine files the tool wrote, caps scaled to the one-window replay | +| `../accounting-lib/count.py` | the two counts, records and aggregators, required to agree | +| `results/plan-.json` | the plan the MCP wrote, its notes and both engine files as it rendered them | +| `results/policy-.caps.csv`, `.actions.csv`, `.json` | the files the engine read | +| `results/.tally.json`, `results/control.tally.json` | the counts per route, container, severity and pattern | +| `results/results.md` | every table, with the per-target detail | + +## Data + +`otel-sample-200mb.log` from +[otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1), +215,039,161 bytes, 197,430 lines, sha256 +`aa79b9349a4123d88936fd064f5eccd9fd84f45a1fd03cff3f64c259715c2432`: Kubernetes container +output captured through Fluentd from a cluster running the OpenTelemetry demo, 36 +containers, about ten hours. The same asset the [`splunk-license/`](../splunk-license/) +and [`otel-denominators/`](../otel-denominators/) benchmarks run on. diff --git a/datadog-flex-accounting/report.py b/datadog-flex-accounting/report.py index 93f444a..010e02d 100644 --- a/datadog-flex-accounting/report.py +++ b/datadog-flex-accounting/report.py @@ -506,9 +506,9 @@ def main() -> int: r = compute(results, prices, run['targets'], run['window_hours'], run) if a.check: committed = json.load(open(results / 'results.json')) - stale = README.differing(open(here / 'README.md').read(), readme_blocks(r, prices)) + stale = README.differing(open(here / 'RECORD.md').read(), readme_blocks(r, prices)) if stale: - raise SystemExit(f'README.md differs from the results in: {", ".join(stale)}; run report.py to rewrite them') + raise SystemExit(f'RECORD.md differs from the results in: {", ".join(stale)}; run report.py to rewrite them') if committed != r: for k in r: if committed.get(k) != r[k]: @@ -519,9 +519,9 @@ def main() -> int: return 0 json.dump(r, open(results / 'results.json', 'w'), indent=1) open(results / 'results.md', 'w').write(render(r, prices)) - readme = here / 'README.md' + readme = here / 'RECORD.md' readme.write_text(README.apply(readme.read_text(), readme_blocks(r, prices))) - print(f'wrote {results / "results.md"} and the README\'s generated blocks') + print(f'wrote {results / "results.md"} and the RECORD\'s generated blocks') return 0 diff --git a/datadog-service-control/README.md b/datadog-service-control/README.md index 96c0b07..d3f493e 100644 --- a/datadog-service-control/README.md +++ b/datadog-service-control/README.md @@ -1,459 +1,46 @@ # Datadog service control -One benchmark in the [log10x benchmarks](../) repository. It asks what a per-pattern plan -moves inside a single service that the rules a Datadog user writes for that service would -not, and how many rules each takes. [`datadog-flex-accounting/`](../datadog-flex-accounting/) -sets the plan against an index filter per service across a whole cluster. Inside one service -a service filter cannot split the volume, so the control here is the filter a person writes -on the service's own logger names. +Inside one Datadog service, can the index filters a user writes move as many events out of Standard as a per-pattern plan? On the OpenTelemetry demo's Kafka logs they can. At the 70% target, eight logger-name filters moved 24,440 of 29,043 events and the plan moved 23,854, and neither moved an error or warning. On this data the control can be built in house, with eight rules against the plan's one. -Everything is computed from the engine's own counts. Nothing was sent to Datadog, and no -Datadog account is needed to rerun it. +The plan is one cap row, a byte limit per log pattern on the kafka container, and events the 10x engine marks over it go to Flex through an index filter. Datadog's Kafka pipeline parses each logger into `logger.name`, so the native rule is `service:kafka @logger.name:`. -## The data +## Result - -The kafka container's records from two public captures of the OpenTelemetry demo. Capture A: 31,029 records, 31,230,658 bytes, 29,043 events after grouping in 424 patterns and 54 logger classes, ten hours. Capture B: 6,230 records, 6,569,531 bytes, 6,226 events in 256 patterns and 64 logger classes, two hours. Engine `log10x/pipeline-10x:1.1.140`, log10x-mcp 1.30.49. - +Events moved out of Standard at the 70% target. B got A's rules unchanged, beside a plan written on B. -Both captures run the same Kafka code (3.9.1): A is demo 2.1.3 shipped by Fluentd, B is demo -2.2.0 shipped by Fluent Bit. On both, the logger classes of segment and snapshot churn on -`__cluster_metadata` carry almost all of kafka's events, every one INFO; `results/results.md` -lists the classes by events. The rest are start-up, partition loading, coordinator election -and config dumps. +| Capture | Rule set | Rules | Events moved | Lifecycle events moved, of 317 | +|---|---|---:|---:|---:| +| A, 29,043 events | per-pattern plan | 1 cap row | 23,854 | n/a | +| A | logger-name filters | 8 | 24,440 | n/a | +| B, 6,226 events | plan rewritten on B | 1 cap row | 3,321 | 0 | +| B | A's cap, unchanged | 1 cap row | 3,493 | 95 | +| B | A's logger-name filters | 8 | 4,658 | 0 | -## The arms +On A the filters leave in Standard 3,046 events the plan moves, from two logger classes they do not name. On B, A's filters moved none of the 317 lifecycle events (patterns B has and A lacks), while A's cap moved 95. -**Per-pattern plan.** The MCP's plan on A's kafka slice at 30, 50, 70 and 90% of the slice, -its `caps.csv` and `actions.csv` applied as written through the released engine. One -container, so one cap row. The engine marks the events over the cap `tier_down`; in Datadog, -an index filter on `@routeState:tier_down` in front of the Standard index sends them to a -Flex index. +Across all services of both captures, after a buyer's usual filters (`-status:(error OR warn)` service filters on three services, DEBUG and TRACE dropped): -**Logger-class filters.** Every Kafka line ends with its log4j logger in parentheses, for -example `(org.apache.kafka.storage.internals.log.LogSegment)`. Datadog's Kafka log pipeline -parses it into the standard attribute `logger.name` -([Kafka integration](https://docs.datadoghq.com/integrations/kafka/), -[standard attributes](https://docs.datadoghq.com/standard-attributes/), both read -2026-10-12), so the filter a Datadog user writes is `service:kafka @logger.name:`, -one per class. Each event's class is read from its own text in the replay's output. A Scala -object logs under its own name: `kafka.log.LocalLog$` is a different logger from -`kafka.log.LocalLog`, and a filter names one or the other. +- The plan moved 5,539 events beyond those filters on A (3.5% of the capture) and 4,882 on B (11.7%). +- On B, six more service filters cut that to 48 events but also move 4,085 events the plan keeps. +- A phrase of each pattern's own words selects it, which takes 13 index filters on A and 22 on B. One phrase on A also catches 2 kept events. -**Selection rule, fixed before the run.** Datadog bills Standard indexing per million -events, so the arms are matched on events. Per target, the control takes the fewest classes, -largest by events first, whose events reach at least the events the engine marked -`tier_down` under the plan, and reports the overshoot without trimming it. A second variant -adds `-status:(error OR warn)` to each filter. Events with no logger class (the image's shell -start-up output) cannot be named by such a filter and are never chosen. +## How it was measured -**On B, nothing re-tuned.** A's chosen classes applied to B's slice; A's cap carried to B's -span and applied through the engine; and, for reference, the plan rewritten on B. +The data is the kafka records of two public OpenTelemetry demo captures on Kafka 3.9.1. A covers ten hours through Fluentd and B two hours through Fluent Bit. The plan was written on A at targets from 30 to 90% and moved 23,836 to 23,905 events at each. The filters follow a rule fixed before the run: the fewest logger classes, largest first, whose events reach the plan's. Each count is read from the emitted records and the engine's own counters, and the run fails if they differ. -## Prediction, written before the run +## Limits -The control matches the plan at every target with at most 9 classes. It wrongly moves the -rare lines that share those classes, about a hundred events (`UnifiedLog`'s producer-state -loading, 52, is the largest). On B the same classes still match, because the code is the -same, and the lifecycle lines B has and A does not (coordinator resignation, offsets -unloading, shutdown, event queue closed) sit in classes the control never named, so both -arms keep them. On the same code, Kafka's logger hierarchy is a good native key and the -comparison is close to a draw on events; what it measures is the rule count, about nine -against one cap row, and the rare-line count. The tables are published as they come out. +- Only two demo captures were tested. A production stream can have more events that only the pattern ID selects. +- Nothing was sent to Datadog, so no Datadog meter was read. +- The rule counts leave out the work of finding each pattern first. -## Capture A +## Run it - -| Target | Arm | Rules | Events moved | Over the plan | GB moved | Errors and warnings moved | Events of patterns the plan keeps under the cap | Events the plan moves that stay in Standard | -|---:|---|---:|---:|---:|---:|---:|---:|---:| -| 30% | per-pattern plan | 1 cap row | 23,905 | 0 | 0.026 | 0 | 0 | 0 | -| 30% | logger-class filters | 8 | 24,440 | +535 | 0.026 | 0 | 0 | 3,056 | -| 30% | logger-class filters, `-status:(error OR warn)` | 8 | 24,440 | +535 | 0.026 | 0 | 0 | 3,056 | -| 50% | per-pattern plan | 1 cap row | 23,836 | 0 | 0.026 | 0 | 0 | 0 | -| 50% | logger-class filters | 8 | 24,440 | +604 | 0.026 | 0 | 0 | 3,077 | -| 50% | logger-class filters, `-status:(error OR warn)` | 8 | 24,440 | +604 | 0.026 | 0 | 0 | 3,077 | -| 70% | per-pattern plan | 1 cap row | 23,854 | 0 | 0.026 | 0 | 0 | 0 | -| 70% | logger-class filters | 8 | 24,440 | +586 | 0.026 | 0 | 0 | 3,046 | -| 70% | logger-class filters, `-status:(error OR warn)` | 8 | 24,440 | +586 | 0.026 | 0 | 0 | 3,046 | -| 90% | per-pattern plan | 1 cap row | 23,901 | 0 | 0.026 | 0 | 0 | 0 | -| 90% | logger-class filters | 8 | 24,440 | +539 | 0.026 | 0 | 0 | 3,059 | -| 90% | logger-class filters, `-status:(error OR warn)` | 8 | 24,440 | +539 | 0.026 | 0 | 0 | 3,059 | - - - -The filters at 30%, 50%, 70%, 90%, largest first: `@logger.name:org.apache.kafka.storage.internals.log.LogSegment`, `@logger.name:kafka.log.UnifiedLog`, `@logger.name:kafka.log.LocalLog`, `@logger.name:org.apache.kafka.storage.internals.log.ProducerStateManager`, `@logger.name:org.apache.kafka.storage.internals.log.SnapshotFile`, `@logger.name:org.apache.kafka.image.publisher.SnapshotEmitter`, `@logger.name:org.apache.kafka.image.publisher.SnapshotGenerator`, `@logger.name:kafka.raft.KafkaMetadataLog`. - - - -- **30%**: the plan, one cap row on `kafka` of 4,593 bytes per window, states 85.5% of the volume and moves 23,905 of 29,043 events. 8 logger-class filters reach that with 24,440, 535 more. Neither moves an error or warning; the filters move 0 events of patterns the plan keeps under the cap and leave in Standard 3,056 events the plan moves (`LocalLog$`, `Snapshots`). -- **50%**: the plan, one cap row on `kafka` of 4,593 bytes per window, states 85.5% of the volume and moves 23,836 of 29,043 events. 8 logger-class filters reach that with 24,440, 604 more. Neither moves an error or warning; the filters move 0 events of patterns the plan keeps under the cap and leave in Standard 3,077 events the plan moves (`LocalLog$`, `Snapshots`). -- **70%**: the plan, one cap row on `kafka` of 4,593 bytes per window, states 85.5% of the volume and moves 23,854 of 29,043 events. 8 logger-class filters reach that with 24,440, 586 more. Neither moves an error or warning; the filters move 0 events of patterns the plan keeps under the cap and leave in Standard 3,046 events the plan moves (`LocalLog$`, `Snapshots`). -- **90%**: the plan, one cap row on `kafka` of 4,593 bytes per window, states 88.5% of the volume, its ceiling, short of the target and moves 23,901 of 29,043 events. 8 logger-class filters reach that with 24,440, 539 more. Neither moves an error or warning; the filters move 0 events of patterns the plan keeps under the cap and leave in Standard 3,059 events the plan moves (`LocalLog$`, `Snapshots`). - - -## Capture B, nothing re-tuned - - -| Target | Arm | Rules | Events moved | Over B's plan | Errors and warnings moved | Lifecycle events moved, of 317 | Events of patterns B's plan keeps under its cap | Events B's plan moves that stay in Standard | -|---:|---|---:|---:|---:|---:|---:|---:|---:| -| 30% | plan rewritten on B | 1 cap row | 3,289 | 0 | 0 | 0 | 0 | 0 | -| 30% | A's cap, unchanged | 1 cap row | 3,505 | +216 | 0 | 103 | 212 | 34 | -| 30% | A's logger-class filters | 8 | 4,658 | +1,369 | 0 | 0 | 0 | 321 | -| 30% | A's logger-class filters, `-status:(error OR warn)` | 8 | 4,658 | +1,369 | 0 | 0 | 0 | 321 | -| 50% | plan rewritten on B | 1 cap row | 3,303 | 0 | 0 | 0 | 0 | 0 | -| 50% | A's cap, unchanged | 1 cap row | 3,531 | +228 | 0 | 101 | 213 | 26 | -| 50% | A's logger-class filters | 8 | 4,658 | +1,355 | 0 | 0 | 0 | 324 | -| 50% | A's logger-class filters, `-status:(error OR warn)` | 8 | 4,658 | +1,355 | 0 | 0 | 0 | 324 | -| 70% | plan rewritten on B | 1 cap row | 3,321 | 0 | 0 | 0 | 0 | 0 | -| 70% | A's cap, unchanged | 1 cap row | 3,493 | +172 | 0 | 95 | 205 | 50 | -| 70% | A's logger-class filters | 8 | 4,658 | +1,337 | 0 | 0 | 0 | 328 | -| 70% | A's logger-class filters, `-status:(error OR warn)` | 8 | 4,658 | +1,337 | 0 | 0 | 0 | 328 | -| 90% | plan rewritten on B | 1 cap row | 3,295 | 0 | 0 | 0 | 0 | 0 | -| 90% | A's cap, unchanged | 1 cap row | 3,555 | +260 | 0 | 106 | 214 | 20 | -| 90% | A's logger-class filters | 8 | 4,658 | +1,363 | 0 | 0 | 0 | 313 | -| 90% | A's logger-class filters, `-status:(error OR warn)` | 8 | 4,658 | +1,363 | 0 | 0 | 0 | 313 | - - - -- **30%**: of B's 6,226 events, A's 8 filters move 4,658 (every one still matches) and none of the 317 lifecycle events; A's cap moves 3,505, 103 of them lifecycle events over the cap; the plan rewritten on B moves 3,289 and 0 lifecycle events. -- **50%**: of B's 6,226 events, A's 8 filters move 4,658 (every one still matches) and none of the 317 lifecycle events; A's cap moves 3,531, 101 of them lifecycle events over the cap; the plan rewritten on B moves 3,303 and 0 lifecycle events. -- **70%**: of B's 6,226 events, A's 8 filters move 4,658 (every one still matches) and none of the 317 lifecycle events; A's cap moves 3,493, 95 of them lifecycle events over the cap; the plan rewritten on B moves 3,321 and 0 lifecycle events. -- **90%**: of B's 6,226 events, A's 8 filters move 4,658 (every one still matches) and none of the 317 lifecycle events; A's cap moves 3,555, 106 of them lifecycle events over the cap; the plan rewritten on B moves 3,295 and 0 lifecycle events. - - -## Against the prediction - - -- **At most 9 classes match the plan at every target.** 8 classes at every target, 535 to 604 events over the plan. Held. -- **The filters move about a hundred rare lines that share those classes.** 0. Every pattern in the chosen classes is churn the plan also moves; the producer-state loading lines log under `kafka.log.UnifiedLog$`, a logger name of its own. Did not hold. -- **On B the same classes still match.** Every class matches on B. Held. -- **Both arms keep the lifecycle lines.** The filters move 0 of 317; A's cap moves 95 to 106. Held for the filters, not for the cap. -- **Close to a draw on events; what differs is the rule count.** 8 filters against one cap row. Held. - - -## The floor after a buyer's own levers - -A buyer reaches a per-pattern plan after the native levers, because the bill still grows. -This section measures what is left for the plan once those levers are in place, on both -whole captures. - -**The levers, applied as written.** An index filter per service with -`-status:(error OR warn)` for `opentelemetry-collector`, `kafka` and `cart`, the set the -service filters need at [`datadog-flex-accounting/`](../datadog-flex-accounting/)'s 70% -target, and DEBUG and TRACE dropped everywhere. What stays in Standard is the floor. - -**The plan.** The MCP's plan at 70% on each whole capture, applied through the engine as -written. It says which patterns the plan moves and which it keeps. - -**The buckets, defined before the count.** Every floor event is in exactly one: - -1. errors and warnings (ERROR, WARN, CRITICAL, FATAL), which both arms keep; -2. INFO and unlevelled events in a service one more service filter could move whole: (2a) - the plan moves every such pattern of the service, or (2b) the plan moves none of it; -3. INFO and unlevelled events in a mixed service, one with both patterns the plan moves and - patterns it keeps, split into the moved patterns (E, the events the plan moves beyond the - buyer's filters) and the kept ones; -4. of bucket 3's moved patterns: those a native attribute cell selects alone, those a cell - selects only together with other moved patterns, and those whose cell also holds patterns - the plan keeps. - -**What a native filter can name, fixed before the count.** `service`, `status`, `source` -and `host` on every log; the attributes Datadog's documented integration pipeline parses for -the container; and the top-level keys of a line that is a JSON object. A pattern's cell is -every attribute value all its events share, and a filter on the cell takes every pattern that -falls in it. Exact values only. The containers that are mixed on either capture, and `kafka`: - - -| Container | Integration page | Attributes the pipeline parses | -|---|---|---| -| `kafka` | [kafka](https://docs.datadoghq.com/integrations/kafka/) | `logger.name` | -| `postgresql` | [postgres](https://docs.datadoghq.com/integrations/postgres/) | `postgres.pid`, `postgres.error_severity`, `postgres.message` | -| `fluentd-10x` | [fluentd](https://docs.datadoghq.com/integrations/fluentd/) | none: the page covers forwarding logs to Datadog through Fluentd; no pipeline for Fluentd's own log lines | -| `opensearch` | [opensearch](https://docs.datadoghq.com/integrations/opensearch/) | none: no integration page (404 on 2026-10-13) | -| `valkey-cart` | [valkey](https://docs.datadoghq.com/integrations/valkey/) | none: no integration page (404 on 2026-10-13) | -| `recommendation` | none | none: a demo application service, Python; Datadog's Python pages document attribute extraction for JSON logs only, and these lines are not JSON | -| `product-reviews` | none | none: a demo application service, Python; plain-text lines | -| `llm` | none | none: a demo application service, Python; plain-text lines | -| `email` | none | none: a demo application service, Ruby; plain-text lines | -| `quote` | none | none: a demo application service, PHP; plain-text lines | - - -**Message text, added 2026-10-13.** A Datadog index filter also takes plain terms and -quoted phrases on the message; what it cannot take is the `*:` full-text syntax across all -attributes ([log search syntax](https://docs.datadoghq.com/logs/explorer/search_syntax/), -read 2026-10-13). So a person can write `service:kafka "Deleted log segment"` as an index -filter once the pattern has been found in Log Patterns. The attribute split in bucket 4 was -computed with message text counted as unavailable and stands as computed; bucket 4 is -recomputed with text below the tables. A moved pattern is isolable by text when a phrase of -its own constant words, the longest run of words every one of its events shares, read from -the events and not from the pattern's name, selects its events and no other pattern's events -in the service; a phrase that also selects events of a pattern the plan keeps is a collision, -counted separately; a pattern whose events share no run of words is left to the ID. Words -are runs of letters and digits, matched without regard to case, so a match here is at least -as easy to get as in Datadog (`text.py`). - -The pattern ID, `@tenx_hash`, is one attribute value per statement, exact where a phrase can -match another statement that shares its words, present from the first line with no discovery -step, and the same key the cap, the metrics and the plan use. The ID is not the only way to -select a pattern in Datadog. - -Each bucket is shown with the frequency band of its patterns: each pattern's events over the -whole capture, binned as [`pattern-identity/`](../pattern-identity/) bins them, so a claim -that the money sits in high-volume lines can be checked against the floor rather than the -whole stream. - -**The result that flips the verdict, stated before the run.** A buyer's assistant set its -own condition on 2026-10-13: the one-day measurement must show E x (S - F x R) well above -$1,490 a month, the list price of log10x's Growth tier for up to 300 nodes, where E is the -events moved beyond the buyer's filters, S the Standard rate avoided, F the Flex rate and R -the months retained. At its buyer's volume, about 21 billion indexed events a month, S - F x -R at 15-day Standard on a scalable Flex tier is about $1.65 per million before compute, so E -must exceed roughly 0.9 billion events a month, 4.3% of the indexed stream. In shares, which -is all a demo capture can give: the verdict flips if bucket 3's moved patterns are 5% or -more of the capture's events and more than half of them only the ID selects. Then the -assistant's own condition sends the buyer to a canary. If bucket 3 is under 5%, or most of it -is selectable natively, building in house stands on the demo. - -**Prediction, written before the run.** The floor on capture A is about 14.5% of events -after the three service filters, less once DEBUG and TRACE are dropped (TRACE is mostly -`ad`); about 6,500 of it is errors, warnings and criticals. The rest is small services: -`recommendation` (one statement, movable whole), `grafana`, `frontend`, `fluentd-10x`, -`valkey-cart`, `opensearch`. Bucket 3 is therefore a few thousand events, 2 to 4% of the -capture, and part of that is isolable by a logfmt or class attribute where Datadog parses -one. The prediction is that the floor does not flip the verdict on the public captures: -building in house stands on the demo, narrowly, and the floor composition that would flip it -belongs to a real estate. The tables are published as they come out. - - -**Capture A.** 157,228 events. The floor: 15,644 events, 0.034 GB, 10% of the capture. The plan at 70% caps 17 containers and plans 55 patterns. - -| Bucket | Events | Share of the capture | Patterns | seen once | 2 to 10 | 11 to 100 | 101 to 1,000 | over 1,000 | -|---|---:|---:|---:|---:|---:|---:|---:|---:| -| 1. Errors and warnings | 6,571 | 4.2% | 67 | 38 | 69 | 281 | 126 | 6,057 | -| 2a. INFO and unlevelled, a service whose every such pattern the plan moves | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | -| 2b. INFO and unlevelled, a service the plan moves none of | 3,120 | 2% | 1,882 | 1,699 | 439 | 359 | 623 | 0 | -| 3. Mixed service, patterns the plan moves (E) | 5,539 | 3.5% | 13 | 0 | 0 | 149 | 843 | 4,547 | -| 3. Mixed service, patterns the plan keeps | 414 | 0.3% | 286 | 247 | 148 | 19 | 0 | 0 | -| 4. Of E, an attribute cell holds the pattern alone | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | -| 4. Of E, an attribute cell holds it with moved patterns only | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | -| 4. Of E, its attribute cell also holds patterns the plan keeps | 5,539 | 3.5% | 13 | 0 | 0 | 149 | 843 | 4,547 | - -**Capture B.** 41,573 events. The floor: 10,892 events, 0.015 GB, 26.2% of the capture. The plan at 70% caps 12 containers and plans 46 patterns. - -| Bucket | Events | Share of the capture | Patterns | seen once | 2 to 10 | 11 to 100 | 101 to 1,000 | over 1,000 | -|---|---:|---:|---:|---:|---:|---:|---:|---:| -| 1. Errors and warnings | 582 | 1.4% | 21 | 16 | 13 | 0 | 553 | 0 | -| 2a. INFO and unlevelled, a service whose every such pattern the plan moves | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | -| 2b. INFO and unlevelled, a service the plan moves none of | 1,314 | 3.2% | 146 | 141 | 4 | 0 | 1,169 | 0 | -| 3. Mixed service, patterns the plan moves (E) | 4,882 | 11.7% | 22 | 0 | 0 | 164 | 4,718 | 0 | -| 3. Mixed service, patterns the plan keeps | 4,114 | 9.9% | 120 | 24 | 20 | 2,720 | 1,350 | 0 | -| 4. Of E, an attribute cell holds the pattern alone | 23 | 0.1% | 1 | 0 | 0 | 23 | 0 | 0 | -| 4. Of E, an attribute cell holds it with moved patterns only | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | -| 4. Of E, its attribute cell also holds patterns the plan keeps | 4,859 | 11.7% | 21 | 0 | 0 | 141 | 4,718 | 0 | - - -The mixed services and what the plan does in each: - - -| Capture | Mixed service | Events the plan moves | Patterns | Events the plan keeps | Patterns | -|---|---|---:|---:|---:|---:| -| A | `recommendation` | 3,430 | 1 | 1 | 1 | -| A | `fluentd-10x` | 1,283 | 3 | 144 | 106 | -| A | `valkey-cart` | 600 | 5 | 6 | 6 | -| A | `opensearch` | 226 | 4 | 263 | 173 | -| B | `llm` | 2,025 | 8 | 452 | 22 | -| B | `product-reviews` | 1,309 | 4 | 3,621 | 67 | -| B | `recommendation` | 602 | 1 | 1 | 1 | -| B | `email` | 391 | 1 | 1 | 1 | -| B | `quote` | 391 | 1 | 1 | 1 | -| B | `valkey-cart` | 116 | 5 | 9 | 9 | -| B | `postgresql` | 48 | 2 | 29 | 19 | - - -**Bucket 4 with text filters counted as native, recomputed 2026-10-13.** The same moved -patterns, split by what a phrase of their own constant words selects. Each pattern's phrase, -with the kept patterns it also matches, is in `results/results.md`. - - -**Capture A.** E: 5,539 events in 13 patterns. - -| Bucket | Events | Share of the capture | Patterns | seen once | 2 to 10 | 11 to 100 | 101 to 1,000 | over 1,000 | -|---|---:|---:|---:|---:|---:|---:|---:|---:| -| 4. Of E, a text phrase selects the pattern alone | 5,492 | 3.5% | 12 | 0 | 0 | 102 | 843 | 4,547 | -| 4. Of E, a text phrase selects it with moved patterns only | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | -| 4. Of E, its phrase also selects patterns the plan keeps (collision) | 47 | 0% | 1 | 0 | 0 | 47 | 0 | 0 | -| 4. Of E, no constant phrase: only the ID | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | - -**Capture B.** E: 4,882 events in 22 patterns. - -| Bucket | Events | Share of the capture | Patterns | seen once | 2 to 10 | 11 to 100 | 101 to 1,000 | over 1,000 | -|---|---:|---:|---:|---:|---:|---:|---:|---:| -| 4. Of E, a text phrase selects the pattern alone | 4,432 | 10.7% | 20 | 0 | 0 | 164 | 4,268 | 0 | -| 4. Of E, a text phrase selects it with moved patterns only | 450 | 1.1% | 2 | 0 | 0 | 0 | 450 | 0 | -| 4. Of E, its phrase also selects patterns the plan keeps (collision) | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | -| 4. Of E, no constant phrase: only the ID | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | - - - -- **Capture A**: bucket 3's moved patterns are 3.5% of the capture, 13 patterns. A phrase of its own constant words selects 12 of them alone, 0 only with other moved patterns, 1 with a collision on patterns the plan keeps (`following_tail_of` in `fluentd-10x`, 2 kept events, pieces of the same statement), and 0 have no constant phrase. Natively that is 13 index filters, each written after its pattern is found, against one cap row; the collisions move 2 kept events. The pre-stated condition does not fire: 0% of E has only the ID. -- **Capture B**: bucket 3's moved patterns are 11.7% of the capture, 22 patterns. A phrase of its own constant words selects 20 of them alone, 2 only with other moved patterns, 0 with a collision on patterns the plan keeps (none), and 0 have no constant phrase. Natively that is 22 index filters, each written after its pattern is found, against one cap row; the collisions move 0 kept events. The pre-stated condition does not fire: 0% of E has only the ID. - - -**Growth from A to B.** A covers ten hours of demo 2.1.3 and B two hours of demo 2.2.0, so -the floor is compared per hour. A service present only at the later tag is new code; inside -a service present at both, a pattern seen in both captures moves with traffic and a pattern -seen in one only is a statement added, removed or reworded. - - -The floor per hour: 1,564 on A, 5,446 on B. - -| Change from A to B, floor events per hour | Events per hour | Where | -|---|---:|---| -| Services in B and not in A, new code | +3,704 | `product-reviews`, `llm` | -| Services in A and not in B's capture | -507 | `grafana`, `fluentd-10x`, `opensearch`, `kube-proxy`, `prometheus-server`, `jaeger`, and others | -| Services in both, patterns seen in both: traffic | +50 | | -| Services in both, patterns seen in B only: statements added or reworded | +1,270 | `accounting`, `payment`, `quote`, `email` | -| Services in both, patterns seen in A only: statements gone or reworded | -636 | `opentelemetry-collector`, `frontend` | -| Patterns inside the filtered services the filters no longer match | 0 | none: a service filter matches its service at both tags | - - -**Against the prediction.** - - -- **The floor on A is about 14.5% after the three service filters, less once DEBUG and TRACE are dropped, about 6,500 of it errors, warnings and criticals.** 10% with DEBUG and TRACE dropped; bucket 1 holds 6,571. Held. -- **Bucket 3 is 2 to 4% of the capture.** A: 3.5%. B: 11.7%. Held on A, not on B. -- **Part of it is isolable by a logfmt or class attribute.** Only `LOG_checkpoint_starting_time` in `postgresql` on B (23 events, by postgres.error_severity, postgres.message, postgres.pid) sits in a native cell without a kept pattern; every other moved pattern shares its cell with patterns the plan keeps. Held for 23 of the 10,421 events. -- **The floor does not flip the verdict on the public captures.** A: does not flip with bucket 4 on attribute cells, does not flip with text filters counted as native. B: flips with bucket 4 on attribute cells, does not flip with text filters counted as native. Did not hold on B with bucket 4 on attribute cells; held once text filters count. - - -## What the flip on B rests on - -The condition as computed fired on B because it trusted the plan's keep decisions. A service -counted as mixed, and its moved patterns as E, whenever the plan kept any INFO or unlevelled -pattern of it, and bucket 4 counted message text as unavailable, so every moved pattern of a -mixed service fell to the ID. On B the two services that carry most of E, `product-reviews` -and `llm`, keep patterns for two reasons the text shows. Pieces of one statement: a value -made of words the engine's library knows (a review, a model's answer) enters the pattern's -name, so one statement is named one pattern per value, and each piece sits under the cap, so -the statement stays whole in Standard while statements with fewer bytes move. Short -statements the byte cap keeps: -the cap counts bytes per window while Datadog bills Standard per event, so a short statement -with as many events as a moved one stays. A service filter with `-status:(error OR warn)` on -such a service moves the whole of it, the fragmented statement included, and moves nothing the -plan kept for a reason other than bytes. - -The kept side of every mixed service, split from the text by `text.py` and `floor.py`. -Pieces of one statement: patterns whose events carry the same `[file:line]` source location -as another pattern of the service, or whose text carries a run of words that opens the name -of at least two patterns of the service and that the text follows with a delimited value (a -bracket, brace, parenthesis, quote, `@` or `/`, or, after at least two words, a colon or an -equals sign). A family is one such location or run; two statements that open the same way -before their value fall in one family, so the family count is at most the statement count. -Short statements the byte cap keeps: the rest with at least as many events in the capture as -the smallest pattern the plan moves in the service. Other: what remains, with the events -whose text carries no timestamp (a configuration dump in `fluentd-10x`, the installer's output -in `postgresql`) counted beside it. A service is movable whole by one filter when its other -patterns are all seen once in the capture. The last column is what that filter moves beyond -the plan. - - -**Capture B.** - -| Mixed service | Events the plan moves (E) | Events the plan keeps | Pieces of one statement | Families | Short statements the byte cap keeps | Other | Of other, no timestamp head | A whole-service filter moves beyond the plan | Movable whole | -|---|---:|---:|---:|---:|---:|---:|---:|---:|---| -| `llm` | 2,025 | 452 | 450 | 2 | 0 | 2 | 0 | 452 | yes | -| `product-reviews` | 1,309 | 3,621 | 2,270 | 2 | 1,350 | 1 | 0 | 3,621 | yes | -| `recommendation` | 602 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | yes | -| `email` | 391 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | yes | -| `quote` | 391 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | yes | -| `valkey-cart` | 116 | 9 | 0 | 0 | 0 | 9 | 0 | 9 | yes | -| `postgresql` | 48 | 29 | 2 | 1 | 0 | 27 | 9 | 34 (5 critical) | no | - -E on B: 4,882 events, 11.7% of the capture, as computed. With the movable-whole services moved by one filter each (`email`, `llm`, `product-reviews`, `quote`, `recommendation`, `valkey-cart`): 48 events, 0.1%; those filters move 4,085 events the plan keeps. - -**Capture A.** - -| Mixed service | Events the plan moves (E) | Events the plan keeps | Pieces of one statement | Families | Short statements the byte cap keeps | Other | Of other, no timestamp head | A whole-service filter moves beyond the plan | Movable whole | -|---|---:|---:|---:|---:|---:|---:|---:|---:|---| -| `recommendation` | 3,430 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | yes | -| `fluentd-10x` | 1,283 | 144 | 36 | 6 | 0 | 108 | 82 | 144 | no | -| `valkey-cart` | 600 | 6 | 0 | 0 | 0 | 6 | 0 | 6 | yes | -| `opensearch` | 226 | 263 | 122 | 7 | 0 | 141 | 1 | 264 (1 critical) | no | - -E on A: 5,539 events, 3.5% of the capture, as computed. With the movable-whole services moved by one filter each (`recommendation`, `valkey-cart`): 1,509 events, 1%; those filters move 7 events the plan keeps. - - -`results/results.md` lists every one of bucket 3's moved patterns with its cell, its phrase, -its band and what selects it, and every kept pattern of each mixed service with its split. - -## Counting conditions - -- Datadog counts indexed events; with the shipped forwarder path one record is one event, - and the Receiver groups the lines of a multi-line event into one record. -- The regulator caps each pattern's bytes per container per 240 s window; an event over its - pattern's cap takes the container's action. Patterns under 5% of their container's bytes - in the window are not capped, the first 5 events of a pattern in a window pass, and the - severity floors keep a tenth of INFO, DEBUG and TRACE, three tenths of WARN and half of - ERROR and CRITICAL. The kept share is drawn at random per event, so a rerun differs within - that noise. One cap row moves whole patterns: every pattern above the cap, which sits at - the largest pattern the container keeps. -- The tool takes a file's span from its records' timestamps. These records carry no - timestamp field, so it takes one hour, and each cap it writes is bytes per 240 s window - of that hour. Replayed on its own capture inside one window, a cap is multiplied by - 3,600 / 240 = 15. Applied to B, A's cap is carried from A's ten hours to B's two: - 15 × 7,200 / 36,000 = 3. `results/policy-*.json` records each file and its factor. -- Lifecycle events are the events of the patterns B's slice has and A's does not. -- The patterns the plan keeps under the cap are those the engine marked nothing of under it. -- Errors and warnings are ERROR, WARN, CRITICAL and FATAL. Datadog's `status:error` and - `status:warn` match ERROR and WARN. -- Every count is taken two ways, from the emitted records and from the engine's own - aggregators, and the run fails if they differ; the class rows are required to equal those - counts. - -## Reproduce - -```sh -./run.sh # everything -./run.sh plan replay # some stages only, in the order given +``` +python3 report.py --check # recompute every table from the committed files +./run.sh # the full run ``` -Stages: `data plan control policy replay count report` for the kafka slices and -`floor-plan floor-replay floor-count` for the floor. Needs Docker, Node 18 or later and -about 2 GB of disk; the whole run takes under twenty minutes. `plan` installs the pinned -log10x-mcp into `.mcp/` and asks it for a plan over MCP stdio; every other stage runs the -pinned engine image on a slice. `ACCT_TARGETS` sets the targets (default `30 50 70 90`). - -| File | What it is | -|---|---| -| `run.sh` | the whole run, in stages | -| `slice.py` | one container's records from a capture, byte for byte | -| `classes.py` | each event's logger class, read from its text, added to the counted rows | -| `arms.py` | the selection rule and every arm, from the class rows | -| `floor.py` | the floor, its buckets, the flip test and the growth split | -| `attrs.py` | each pattern's native attribute cell on a whole capture | -| `text.py` | each pattern's constant phrase on a whole capture, what the phrase matches, and the statement signals in its text | -| `integrations.json` | the attributes each container's documented Datadog pipeline parses, with the pages read | -| `report.py` | renders `results/results.md`, `results/results.json` and the tables above; `--check` fails CI if the committed figures do not follow from the kept outputs | -| `../accounting-lib/` | the plan request, the replay, the policy scaling and the two counts, shared with the other accounting benchmarks | -| `results/plan--.json` | the plan the MCP wrote | -| `results/policy--.*` | the files the engine read, with the factor | -| `results/.tally.json`, `results/.classes.json` | the counts per route, severity and pattern, and per logger class | -| `results/replay-config.diff` | the one field added to the shared replay config: each event's text | - -## Data +The check needs only Python. The full run needs Docker, Node 18 or later, about 2 GB of disk and under twenty minutes, and no Datadog account. -- **Capture A.** `otel-sample-200mb.log` from - [otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1), - 215,039,161 bytes, 197,430 lines, sha256 - `aa79b9349a4123d88936fd064f5eccd9fd84f45a1fd03cff3f64c259715c2432`: about ten hours of the - OpenTelemetry demo through Fluentd, the asset [`datadog-flex-accounting/`](../datadog-flex-accounting/) - runs on. -- **Capture B.** `rec1155.log.gz` from - [intent-data-v1](https://github.com/log-10x/benchmarks/releases/tag/intent-data-v1), sha256 - `d9ebe8c79e3ef6c1ca656599203d588831b6ed72e1f245cc32cd26124b6c4301`, expanded 51,439,789 bytes, - 41,713 lines, sha256 `df78e200d929bc57f826a396fb8e1764717bf6b61544970261f3647163351b77`: two - hours of the unmodified demo images through Fluent Bit, the recording - [`intent/`](../intent/) runs on. -- **The slices.** Each capture's records whose `kubernetes.container_name` is `kafka`, - written byte for byte by `slice.py`; `run.sh` checks their sha256 and - `results/results.md` lists them. +[RECORD.md](RECORD.md) has every target and variant, the all-services analysis and the predictions written before the run. diff --git a/datadog-service-control/RECORD.md b/datadog-service-control/RECORD.md new file mode 100644 index 0000000..96c0b07 --- /dev/null +++ b/datadog-service-control/RECORD.md @@ -0,0 +1,459 @@ +# Datadog service control + +One benchmark in the [log10x benchmarks](../) repository. It asks what a per-pattern plan +moves inside a single service that the rules a Datadog user writes for that service would +not, and how many rules each takes. [`datadog-flex-accounting/`](../datadog-flex-accounting/) +sets the plan against an index filter per service across a whole cluster. Inside one service +a service filter cannot split the volume, so the control here is the filter a person writes +on the service's own logger names. + +Everything is computed from the engine's own counts. Nothing was sent to Datadog, and no +Datadog account is needed to rerun it. + +## The data + + +The kafka container's records from two public captures of the OpenTelemetry demo. Capture A: 31,029 records, 31,230,658 bytes, 29,043 events after grouping in 424 patterns and 54 logger classes, ten hours. Capture B: 6,230 records, 6,569,531 bytes, 6,226 events in 256 patterns and 64 logger classes, two hours. Engine `log10x/pipeline-10x:1.1.140`, log10x-mcp 1.30.49. + + +Both captures run the same Kafka code (3.9.1): A is demo 2.1.3 shipped by Fluentd, B is demo +2.2.0 shipped by Fluent Bit. On both, the logger classes of segment and snapshot churn on +`__cluster_metadata` carry almost all of kafka's events, every one INFO; `results/results.md` +lists the classes by events. The rest are start-up, partition loading, coordinator election +and config dumps. + +## The arms + +**Per-pattern plan.** The MCP's plan on A's kafka slice at 30, 50, 70 and 90% of the slice, +its `caps.csv` and `actions.csv` applied as written through the released engine. One +container, so one cap row. The engine marks the events over the cap `tier_down`; in Datadog, +an index filter on `@routeState:tier_down` in front of the Standard index sends them to a +Flex index. + +**Logger-class filters.** Every Kafka line ends with its log4j logger in parentheses, for +example `(org.apache.kafka.storage.internals.log.LogSegment)`. Datadog's Kafka log pipeline +parses it into the standard attribute `logger.name` +([Kafka integration](https://docs.datadoghq.com/integrations/kafka/), +[standard attributes](https://docs.datadoghq.com/standard-attributes/), both read +2026-10-12), so the filter a Datadog user writes is `service:kafka @logger.name:`, +one per class. Each event's class is read from its own text in the replay's output. A Scala +object logs under its own name: `kafka.log.LocalLog$` is a different logger from +`kafka.log.LocalLog`, and a filter names one or the other. + +**Selection rule, fixed before the run.** Datadog bills Standard indexing per million +events, so the arms are matched on events. Per target, the control takes the fewest classes, +largest by events first, whose events reach at least the events the engine marked +`tier_down` under the plan, and reports the overshoot without trimming it. A second variant +adds `-status:(error OR warn)` to each filter. Events with no logger class (the image's shell +start-up output) cannot be named by such a filter and are never chosen. + +**On B, nothing re-tuned.** A's chosen classes applied to B's slice; A's cap carried to B's +span and applied through the engine; and, for reference, the plan rewritten on B. + +## Prediction, written before the run + +The control matches the plan at every target with at most 9 classes. It wrongly moves the +rare lines that share those classes, about a hundred events (`UnifiedLog`'s producer-state +loading, 52, is the largest). On B the same classes still match, because the code is the +same, and the lifecycle lines B has and A does not (coordinator resignation, offsets +unloading, shutdown, event queue closed) sit in classes the control never named, so both +arms keep them. On the same code, Kafka's logger hierarchy is a good native key and the +comparison is close to a draw on events; what it measures is the rule count, about nine +against one cap row, and the rare-line count. The tables are published as they come out. + +## Capture A + + +| Target | Arm | Rules | Events moved | Over the plan | GB moved | Errors and warnings moved | Events of patterns the plan keeps under the cap | Events the plan moves that stay in Standard | +|---:|---|---:|---:|---:|---:|---:|---:|---:| +| 30% | per-pattern plan | 1 cap row | 23,905 | 0 | 0.026 | 0 | 0 | 0 | +| 30% | logger-class filters | 8 | 24,440 | +535 | 0.026 | 0 | 0 | 3,056 | +| 30% | logger-class filters, `-status:(error OR warn)` | 8 | 24,440 | +535 | 0.026 | 0 | 0 | 3,056 | +| 50% | per-pattern plan | 1 cap row | 23,836 | 0 | 0.026 | 0 | 0 | 0 | +| 50% | logger-class filters | 8 | 24,440 | +604 | 0.026 | 0 | 0 | 3,077 | +| 50% | logger-class filters, `-status:(error OR warn)` | 8 | 24,440 | +604 | 0.026 | 0 | 0 | 3,077 | +| 70% | per-pattern plan | 1 cap row | 23,854 | 0 | 0.026 | 0 | 0 | 0 | +| 70% | logger-class filters | 8 | 24,440 | +586 | 0.026 | 0 | 0 | 3,046 | +| 70% | logger-class filters, `-status:(error OR warn)` | 8 | 24,440 | +586 | 0.026 | 0 | 0 | 3,046 | +| 90% | per-pattern plan | 1 cap row | 23,901 | 0 | 0.026 | 0 | 0 | 0 | +| 90% | logger-class filters | 8 | 24,440 | +539 | 0.026 | 0 | 0 | 3,059 | +| 90% | logger-class filters, `-status:(error OR warn)` | 8 | 24,440 | +539 | 0.026 | 0 | 0 | 3,059 | + + + +The filters at 30%, 50%, 70%, 90%, largest first: `@logger.name:org.apache.kafka.storage.internals.log.LogSegment`, `@logger.name:kafka.log.UnifiedLog`, `@logger.name:kafka.log.LocalLog`, `@logger.name:org.apache.kafka.storage.internals.log.ProducerStateManager`, `@logger.name:org.apache.kafka.storage.internals.log.SnapshotFile`, `@logger.name:org.apache.kafka.image.publisher.SnapshotEmitter`, `@logger.name:org.apache.kafka.image.publisher.SnapshotGenerator`, `@logger.name:kafka.raft.KafkaMetadataLog`. + + + +- **30%**: the plan, one cap row on `kafka` of 4,593 bytes per window, states 85.5% of the volume and moves 23,905 of 29,043 events. 8 logger-class filters reach that with 24,440, 535 more. Neither moves an error or warning; the filters move 0 events of patterns the plan keeps under the cap and leave in Standard 3,056 events the plan moves (`LocalLog$`, `Snapshots`). +- **50%**: the plan, one cap row on `kafka` of 4,593 bytes per window, states 85.5% of the volume and moves 23,836 of 29,043 events. 8 logger-class filters reach that with 24,440, 604 more. Neither moves an error or warning; the filters move 0 events of patterns the plan keeps under the cap and leave in Standard 3,077 events the plan moves (`LocalLog$`, `Snapshots`). +- **70%**: the plan, one cap row on `kafka` of 4,593 bytes per window, states 85.5% of the volume and moves 23,854 of 29,043 events. 8 logger-class filters reach that with 24,440, 586 more. Neither moves an error or warning; the filters move 0 events of patterns the plan keeps under the cap and leave in Standard 3,046 events the plan moves (`LocalLog$`, `Snapshots`). +- **90%**: the plan, one cap row on `kafka` of 4,593 bytes per window, states 88.5% of the volume, its ceiling, short of the target and moves 23,901 of 29,043 events. 8 logger-class filters reach that with 24,440, 539 more. Neither moves an error or warning; the filters move 0 events of patterns the plan keeps under the cap and leave in Standard 3,059 events the plan moves (`LocalLog$`, `Snapshots`). + + +## Capture B, nothing re-tuned + + +| Target | Arm | Rules | Events moved | Over B's plan | Errors and warnings moved | Lifecycle events moved, of 317 | Events of patterns B's plan keeps under its cap | Events B's plan moves that stay in Standard | +|---:|---|---:|---:|---:|---:|---:|---:|---:| +| 30% | plan rewritten on B | 1 cap row | 3,289 | 0 | 0 | 0 | 0 | 0 | +| 30% | A's cap, unchanged | 1 cap row | 3,505 | +216 | 0 | 103 | 212 | 34 | +| 30% | A's logger-class filters | 8 | 4,658 | +1,369 | 0 | 0 | 0 | 321 | +| 30% | A's logger-class filters, `-status:(error OR warn)` | 8 | 4,658 | +1,369 | 0 | 0 | 0 | 321 | +| 50% | plan rewritten on B | 1 cap row | 3,303 | 0 | 0 | 0 | 0 | 0 | +| 50% | A's cap, unchanged | 1 cap row | 3,531 | +228 | 0 | 101 | 213 | 26 | +| 50% | A's logger-class filters | 8 | 4,658 | +1,355 | 0 | 0 | 0 | 324 | +| 50% | A's logger-class filters, `-status:(error OR warn)` | 8 | 4,658 | +1,355 | 0 | 0 | 0 | 324 | +| 70% | plan rewritten on B | 1 cap row | 3,321 | 0 | 0 | 0 | 0 | 0 | +| 70% | A's cap, unchanged | 1 cap row | 3,493 | +172 | 0 | 95 | 205 | 50 | +| 70% | A's logger-class filters | 8 | 4,658 | +1,337 | 0 | 0 | 0 | 328 | +| 70% | A's logger-class filters, `-status:(error OR warn)` | 8 | 4,658 | +1,337 | 0 | 0 | 0 | 328 | +| 90% | plan rewritten on B | 1 cap row | 3,295 | 0 | 0 | 0 | 0 | 0 | +| 90% | A's cap, unchanged | 1 cap row | 3,555 | +260 | 0 | 106 | 214 | 20 | +| 90% | A's logger-class filters | 8 | 4,658 | +1,363 | 0 | 0 | 0 | 313 | +| 90% | A's logger-class filters, `-status:(error OR warn)` | 8 | 4,658 | +1,363 | 0 | 0 | 0 | 313 | + + + +- **30%**: of B's 6,226 events, A's 8 filters move 4,658 (every one still matches) and none of the 317 lifecycle events; A's cap moves 3,505, 103 of them lifecycle events over the cap; the plan rewritten on B moves 3,289 and 0 lifecycle events. +- **50%**: of B's 6,226 events, A's 8 filters move 4,658 (every one still matches) and none of the 317 lifecycle events; A's cap moves 3,531, 101 of them lifecycle events over the cap; the plan rewritten on B moves 3,303 and 0 lifecycle events. +- **70%**: of B's 6,226 events, A's 8 filters move 4,658 (every one still matches) and none of the 317 lifecycle events; A's cap moves 3,493, 95 of them lifecycle events over the cap; the plan rewritten on B moves 3,321 and 0 lifecycle events. +- **90%**: of B's 6,226 events, A's 8 filters move 4,658 (every one still matches) and none of the 317 lifecycle events; A's cap moves 3,555, 106 of them lifecycle events over the cap; the plan rewritten on B moves 3,295 and 0 lifecycle events. + + +## Against the prediction + + +- **At most 9 classes match the plan at every target.** 8 classes at every target, 535 to 604 events over the plan. Held. +- **The filters move about a hundred rare lines that share those classes.** 0. Every pattern in the chosen classes is churn the plan also moves; the producer-state loading lines log under `kafka.log.UnifiedLog$`, a logger name of its own. Did not hold. +- **On B the same classes still match.** Every class matches on B. Held. +- **Both arms keep the lifecycle lines.** The filters move 0 of 317; A's cap moves 95 to 106. Held for the filters, not for the cap. +- **Close to a draw on events; what differs is the rule count.** 8 filters against one cap row. Held. + + +## The floor after a buyer's own levers + +A buyer reaches a per-pattern plan after the native levers, because the bill still grows. +This section measures what is left for the plan once those levers are in place, on both +whole captures. + +**The levers, applied as written.** An index filter per service with +`-status:(error OR warn)` for `opentelemetry-collector`, `kafka` and `cart`, the set the +service filters need at [`datadog-flex-accounting/`](../datadog-flex-accounting/)'s 70% +target, and DEBUG and TRACE dropped everywhere. What stays in Standard is the floor. + +**The plan.** The MCP's plan at 70% on each whole capture, applied through the engine as +written. It says which patterns the plan moves and which it keeps. + +**The buckets, defined before the count.** Every floor event is in exactly one: + +1. errors and warnings (ERROR, WARN, CRITICAL, FATAL), which both arms keep; +2. INFO and unlevelled events in a service one more service filter could move whole: (2a) + the plan moves every such pattern of the service, or (2b) the plan moves none of it; +3. INFO and unlevelled events in a mixed service, one with both patterns the plan moves and + patterns it keeps, split into the moved patterns (E, the events the plan moves beyond the + buyer's filters) and the kept ones; +4. of bucket 3's moved patterns: those a native attribute cell selects alone, those a cell + selects only together with other moved patterns, and those whose cell also holds patterns + the plan keeps. + +**What a native filter can name, fixed before the count.** `service`, `status`, `source` +and `host` on every log; the attributes Datadog's documented integration pipeline parses for +the container; and the top-level keys of a line that is a JSON object. A pattern's cell is +every attribute value all its events share, and a filter on the cell takes every pattern that +falls in it. Exact values only. The containers that are mixed on either capture, and `kafka`: + + +| Container | Integration page | Attributes the pipeline parses | +|---|---|---| +| `kafka` | [kafka](https://docs.datadoghq.com/integrations/kafka/) | `logger.name` | +| `postgresql` | [postgres](https://docs.datadoghq.com/integrations/postgres/) | `postgres.pid`, `postgres.error_severity`, `postgres.message` | +| `fluentd-10x` | [fluentd](https://docs.datadoghq.com/integrations/fluentd/) | none: the page covers forwarding logs to Datadog through Fluentd; no pipeline for Fluentd's own log lines | +| `opensearch` | [opensearch](https://docs.datadoghq.com/integrations/opensearch/) | none: no integration page (404 on 2026-10-13) | +| `valkey-cart` | [valkey](https://docs.datadoghq.com/integrations/valkey/) | none: no integration page (404 on 2026-10-13) | +| `recommendation` | none | none: a demo application service, Python; Datadog's Python pages document attribute extraction for JSON logs only, and these lines are not JSON | +| `product-reviews` | none | none: a demo application service, Python; plain-text lines | +| `llm` | none | none: a demo application service, Python; plain-text lines | +| `email` | none | none: a demo application service, Ruby; plain-text lines | +| `quote` | none | none: a demo application service, PHP; plain-text lines | + + +**Message text, added 2026-10-13.** A Datadog index filter also takes plain terms and +quoted phrases on the message; what it cannot take is the `*:` full-text syntax across all +attributes ([log search syntax](https://docs.datadoghq.com/logs/explorer/search_syntax/), +read 2026-10-13). So a person can write `service:kafka "Deleted log segment"` as an index +filter once the pattern has been found in Log Patterns. The attribute split in bucket 4 was +computed with message text counted as unavailable and stands as computed; bucket 4 is +recomputed with text below the tables. A moved pattern is isolable by text when a phrase of +its own constant words, the longest run of words every one of its events shares, read from +the events and not from the pattern's name, selects its events and no other pattern's events +in the service; a phrase that also selects events of a pattern the plan keeps is a collision, +counted separately; a pattern whose events share no run of words is left to the ID. Words +are runs of letters and digits, matched without regard to case, so a match here is at least +as easy to get as in Datadog (`text.py`). + +The pattern ID, `@tenx_hash`, is one attribute value per statement, exact where a phrase can +match another statement that shares its words, present from the first line with no discovery +step, and the same key the cap, the metrics and the plan use. The ID is not the only way to +select a pattern in Datadog. + +Each bucket is shown with the frequency band of its patterns: each pattern's events over the +whole capture, binned as [`pattern-identity/`](../pattern-identity/) bins them, so a claim +that the money sits in high-volume lines can be checked against the floor rather than the +whole stream. + +**The result that flips the verdict, stated before the run.** A buyer's assistant set its +own condition on 2026-10-13: the one-day measurement must show E x (S - F x R) well above +$1,490 a month, the list price of log10x's Growth tier for up to 300 nodes, where E is the +events moved beyond the buyer's filters, S the Standard rate avoided, F the Flex rate and R +the months retained. At its buyer's volume, about 21 billion indexed events a month, S - F x +R at 15-day Standard on a scalable Flex tier is about $1.65 per million before compute, so E +must exceed roughly 0.9 billion events a month, 4.3% of the indexed stream. In shares, which +is all a demo capture can give: the verdict flips if bucket 3's moved patterns are 5% or +more of the capture's events and more than half of them only the ID selects. Then the +assistant's own condition sends the buyer to a canary. If bucket 3 is under 5%, or most of it +is selectable natively, building in house stands on the demo. + +**Prediction, written before the run.** The floor on capture A is about 14.5% of events +after the three service filters, less once DEBUG and TRACE are dropped (TRACE is mostly +`ad`); about 6,500 of it is errors, warnings and criticals. The rest is small services: +`recommendation` (one statement, movable whole), `grafana`, `frontend`, `fluentd-10x`, +`valkey-cart`, `opensearch`. Bucket 3 is therefore a few thousand events, 2 to 4% of the +capture, and part of that is isolable by a logfmt or class attribute where Datadog parses +one. The prediction is that the floor does not flip the verdict on the public captures: +building in house stands on the demo, narrowly, and the floor composition that would flip it +belongs to a real estate. The tables are published as they come out. + + +**Capture A.** 157,228 events. The floor: 15,644 events, 0.034 GB, 10% of the capture. The plan at 70% caps 17 containers and plans 55 patterns. + +| Bucket | Events | Share of the capture | Patterns | seen once | 2 to 10 | 11 to 100 | 101 to 1,000 | over 1,000 | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| 1. Errors and warnings | 6,571 | 4.2% | 67 | 38 | 69 | 281 | 126 | 6,057 | +| 2a. INFO and unlevelled, a service whose every such pattern the plan moves | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | +| 2b. INFO and unlevelled, a service the plan moves none of | 3,120 | 2% | 1,882 | 1,699 | 439 | 359 | 623 | 0 | +| 3. Mixed service, patterns the plan moves (E) | 5,539 | 3.5% | 13 | 0 | 0 | 149 | 843 | 4,547 | +| 3. Mixed service, patterns the plan keeps | 414 | 0.3% | 286 | 247 | 148 | 19 | 0 | 0 | +| 4. Of E, an attribute cell holds the pattern alone | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | +| 4. Of E, an attribute cell holds it with moved patterns only | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | +| 4. Of E, its attribute cell also holds patterns the plan keeps | 5,539 | 3.5% | 13 | 0 | 0 | 149 | 843 | 4,547 | + +**Capture B.** 41,573 events. The floor: 10,892 events, 0.015 GB, 26.2% of the capture. The plan at 70% caps 12 containers and plans 46 patterns. + +| Bucket | Events | Share of the capture | Patterns | seen once | 2 to 10 | 11 to 100 | 101 to 1,000 | over 1,000 | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| 1. Errors and warnings | 582 | 1.4% | 21 | 16 | 13 | 0 | 553 | 0 | +| 2a. INFO and unlevelled, a service whose every such pattern the plan moves | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | +| 2b. INFO and unlevelled, a service the plan moves none of | 1,314 | 3.2% | 146 | 141 | 4 | 0 | 1,169 | 0 | +| 3. Mixed service, patterns the plan moves (E) | 4,882 | 11.7% | 22 | 0 | 0 | 164 | 4,718 | 0 | +| 3. Mixed service, patterns the plan keeps | 4,114 | 9.9% | 120 | 24 | 20 | 2,720 | 1,350 | 0 | +| 4. Of E, an attribute cell holds the pattern alone | 23 | 0.1% | 1 | 0 | 0 | 23 | 0 | 0 | +| 4. Of E, an attribute cell holds it with moved patterns only | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | +| 4. Of E, its attribute cell also holds patterns the plan keeps | 4,859 | 11.7% | 21 | 0 | 0 | 141 | 4,718 | 0 | + + +The mixed services and what the plan does in each: + + +| Capture | Mixed service | Events the plan moves | Patterns | Events the plan keeps | Patterns | +|---|---|---:|---:|---:|---:| +| A | `recommendation` | 3,430 | 1 | 1 | 1 | +| A | `fluentd-10x` | 1,283 | 3 | 144 | 106 | +| A | `valkey-cart` | 600 | 5 | 6 | 6 | +| A | `opensearch` | 226 | 4 | 263 | 173 | +| B | `llm` | 2,025 | 8 | 452 | 22 | +| B | `product-reviews` | 1,309 | 4 | 3,621 | 67 | +| B | `recommendation` | 602 | 1 | 1 | 1 | +| B | `email` | 391 | 1 | 1 | 1 | +| B | `quote` | 391 | 1 | 1 | 1 | +| B | `valkey-cart` | 116 | 5 | 9 | 9 | +| B | `postgresql` | 48 | 2 | 29 | 19 | + + +**Bucket 4 with text filters counted as native, recomputed 2026-10-13.** The same moved +patterns, split by what a phrase of their own constant words selects. Each pattern's phrase, +with the kept patterns it also matches, is in `results/results.md`. + + +**Capture A.** E: 5,539 events in 13 patterns. + +| Bucket | Events | Share of the capture | Patterns | seen once | 2 to 10 | 11 to 100 | 101 to 1,000 | over 1,000 | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| 4. Of E, a text phrase selects the pattern alone | 5,492 | 3.5% | 12 | 0 | 0 | 102 | 843 | 4,547 | +| 4. Of E, a text phrase selects it with moved patterns only | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | +| 4. Of E, its phrase also selects patterns the plan keeps (collision) | 47 | 0% | 1 | 0 | 0 | 47 | 0 | 0 | +| 4. Of E, no constant phrase: only the ID | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | + +**Capture B.** E: 4,882 events in 22 patterns. + +| Bucket | Events | Share of the capture | Patterns | seen once | 2 to 10 | 11 to 100 | 101 to 1,000 | over 1,000 | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| 4. Of E, a text phrase selects the pattern alone | 4,432 | 10.7% | 20 | 0 | 0 | 164 | 4,268 | 0 | +| 4. Of E, a text phrase selects it with moved patterns only | 450 | 1.1% | 2 | 0 | 0 | 0 | 450 | 0 | +| 4. Of E, its phrase also selects patterns the plan keeps (collision) | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | +| 4. Of E, no constant phrase: only the ID | 0 | 0% | 0 | 0 | 0 | 0 | 0 | 0 | + + + +- **Capture A**: bucket 3's moved patterns are 3.5% of the capture, 13 patterns. A phrase of its own constant words selects 12 of them alone, 0 only with other moved patterns, 1 with a collision on patterns the plan keeps (`following_tail_of` in `fluentd-10x`, 2 kept events, pieces of the same statement), and 0 have no constant phrase. Natively that is 13 index filters, each written after its pattern is found, against one cap row; the collisions move 2 kept events. The pre-stated condition does not fire: 0% of E has only the ID. +- **Capture B**: bucket 3's moved patterns are 11.7% of the capture, 22 patterns. A phrase of its own constant words selects 20 of them alone, 2 only with other moved patterns, 0 with a collision on patterns the plan keeps (none), and 0 have no constant phrase. Natively that is 22 index filters, each written after its pattern is found, against one cap row; the collisions move 0 kept events. The pre-stated condition does not fire: 0% of E has only the ID. + + +**Growth from A to B.** A covers ten hours of demo 2.1.3 and B two hours of demo 2.2.0, so +the floor is compared per hour. A service present only at the later tag is new code; inside +a service present at both, a pattern seen in both captures moves with traffic and a pattern +seen in one only is a statement added, removed or reworded. + + +The floor per hour: 1,564 on A, 5,446 on B. + +| Change from A to B, floor events per hour | Events per hour | Where | +|---|---:|---| +| Services in B and not in A, new code | +3,704 | `product-reviews`, `llm` | +| Services in A and not in B's capture | -507 | `grafana`, `fluentd-10x`, `opensearch`, `kube-proxy`, `prometheus-server`, `jaeger`, and others | +| Services in both, patterns seen in both: traffic | +50 | | +| Services in both, patterns seen in B only: statements added or reworded | +1,270 | `accounting`, `payment`, `quote`, `email` | +| Services in both, patterns seen in A only: statements gone or reworded | -636 | `opentelemetry-collector`, `frontend` | +| Patterns inside the filtered services the filters no longer match | 0 | none: a service filter matches its service at both tags | + + +**Against the prediction.** + + +- **The floor on A is about 14.5% after the three service filters, less once DEBUG and TRACE are dropped, about 6,500 of it errors, warnings and criticals.** 10% with DEBUG and TRACE dropped; bucket 1 holds 6,571. Held. +- **Bucket 3 is 2 to 4% of the capture.** A: 3.5%. B: 11.7%. Held on A, not on B. +- **Part of it is isolable by a logfmt or class attribute.** Only `LOG_checkpoint_starting_time` in `postgresql` on B (23 events, by postgres.error_severity, postgres.message, postgres.pid) sits in a native cell without a kept pattern; every other moved pattern shares its cell with patterns the plan keeps. Held for 23 of the 10,421 events. +- **The floor does not flip the verdict on the public captures.** A: does not flip with bucket 4 on attribute cells, does not flip with text filters counted as native. B: flips with bucket 4 on attribute cells, does not flip with text filters counted as native. Did not hold on B with bucket 4 on attribute cells; held once text filters count. + + +## What the flip on B rests on + +The condition as computed fired on B because it trusted the plan's keep decisions. A service +counted as mixed, and its moved patterns as E, whenever the plan kept any INFO or unlevelled +pattern of it, and bucket 4 counted message text as unavailable, so every moved pattern of a +mixed service fell to the ID. On B the two services that carry most of E, `product-reviews` +and `llm`, keep patterns for two reasons the text shows. Pieces of one statement: a value +made of words the engine's library knows (a review, a model's answer) enters the pattern's +name, so one statement is named one pattern per value, and each piece sits under the cap, so +the statement stays whole in Standard while statements with fewer bytes move. Short +statements the byte cap keeps: +the cap counts bytes per window while Datadog bills Standard per event, so a short statement +with as many events as a moved one stays. A service filter with `-status:(error OR warn)` on +such a service moves the whole of it, the fragmented statement included, and moves nothing the +plan kept for a reason other than bytes. + +The kept side of every mixed service, split from the text by `text.py` and `floor.py`. +Pieces of one statement: patterns whose events carry the same `[file:line]` source location +as another pattern of the service, or whose text carries a run of words that opens the name +of at least two patterns of the service and that the text follows with a delimited value (a +bracket, brace, parenthesis, quote, `@` or `/`, or, after at least two words, a colon or an +equals sign). A family is one such location or run; two statements that open the same way +before their value fall in one family, so the family count is at most the statement count. +Short statements the byte cap keeps: the rest with at least as many events in the capture as +the smallest pattern the plan moves in the service. Other: what remains, with the events +whose text carries no timestamp (a configuration dump in `fluentd-10x`, the installer's output +in `postgresql`) counted beside it. A service is movable whole by one filter when its other +patterns are all seen once in the capture. The last column is what that filter moves beyond +the plan. + + +**Capture B.** + +| Mixed service | Events the plan moves (E) | Events the plan keeps | Pieces of one statement | Families | Short statements the byte cap keeps | Other | Of other, no timestamp head | A whole-service filter moves beyond the plan | Movable whole | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---| +| `llm` | 2,025 | 452 | 450 | 2 | 0 | 2 | 0 | 452 | yes | +| `product-reviews` | 1,309 | 3,621 | 2,270 | 2 | 1,350 | 1 | 0 | 3,621 | yes | +| `recommendation` | 602 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | yes | +| `email` | 391 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | yes | +| `quote` | 391 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | yes | +| `valkey-cart` | 116 | 9 | 0 | 0 | 0 | 9 | 0 | 9 | yes | +| `postgresql` | 48 | 29 | 2 | 1 | 0 | 27 | 9 | 34 (5 critical) | no | + +E on B: 4,882 events, 11.7% of the capture, as computed. With the movable-whole services moved by one filter each (`email`, `llm`, `product-reviews`, `quote`, `recommendation`, `valkey-cart`): 48 events, 0.1%; those filters move 4,085 events the plan keeps. + +**Capture A.** + +| Mixed service | Events the plan moves (E) | Events the plan keeps | Pieces of one statement | Families | Short statements the byte cap keeps | Other | Of other, no timestamp head | A whole-service filter moves beyond the plan | Movable whole | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---| +| `recommendation` | 3,430 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | yes | +| `fluentd-10x` | 1,283 | 144 | 36 | 6 | 0 | 108 | 82 | 144 | no | +| `valkey-cart` | 600 | 6 | 0 | 0 | 0 | 6 | 0 | 6 | yes | +| `opensearch` | 226 | 263 | 122 | 7 | 0 | 141 | 1 | 264 (1 critical) | no | + +E on A: 5,539 events, 3.5% of the capture, as computed. With the movable-whole services moved by one filter each (`recommendation`, `valkey-cart`): 1,509 events, 1%; those filters move 7 events the plan keeps. + + +`results/results.md` lists every one of bucket 3's moved patterns with its cell, its phrase, +its band and what selects it, and every kept pattern of each mixed service with its split. + +## Counting conditions + +- Datadog counts indexed events; with the shipped forwarder path one record is one event, + and the Receiver groups the lines of a multi-line event into one record. +- The regulator caps each pattern's bytes per container per 240 s window; an event over its + pattern's cap takes the container's action. Patterns under 5% of their container's bytes + in the window are not capped, the first 5 events of a pattern in a window pass, and the + severity floors keep a tenth of INFO, DEBUG and TRACE, three tenths of WARN and half of + ERROR and CRITICAL. The kept share is drawn at random per event, so a rerun differs within + that noise. One cap row moves whole patterns: every pattern above the cap, which sits at + the largest pattern the container keeps. +- The tool takes a file's span from its records' timestamps. These records carry no + timestamp field, so it takes one hour, and each cap it writes is bytes per 240 s window + of that hour. Replayed on its own capture inside one window, a cap is multiplied by + 3,600 / 240 = 15. Applied to B, A's cap is carried from A's ten hours to B's two: + 15 × 7,200 / 36,000 = 3. `results/policy-*.json` records each file and its factor. +- Lifecycle events are the events of the patterns B's slice has and A's does not. +- The patterns the plan keeps under the cap are those the engine marked nothing of under it. +- Errors and warnings are ERROR, WARN, CRITICAL and FATAL. Datadog's `status:error` and + `status:warn` match ERROR and WARN. +- Every count is taken two ways, from the emitted records and from the engine's own + aggregators, and the run fails if they differ; the class rows are required to equal those + counts. + +## Reproduce + +```sh +./run.sh # everything +./run.sh plan replay # some stages only, in the order given +``` + +Stages: `data plan control policy replay count report` for the kafka slices and +`floor-plan floor-replay floor-count` for the floor. Needs Docker, Node 18 or later and +about 2 GB of disk; the whole run takes under twenty minutes. `plan` installs the pinned +log10x-mcp into `.mcp/` and asks it for a plan over MCP stdio; every other stage runs the +pinned engine image on a slice. `ACCT_TARGETS` sets the targets (default `30 50 70 90`). + +| File | What it is | +|---|---| +| `run.sh` | the whole run, in stages | +| `slice.py` | one container's records from a capture, byte for byte | +| `classes.py` | each event's logger class, read from its text, added to the counted rows | +| `arms.py` | the selection rule and every arm, from the class rows | +| `floor.py` | the floor, its buckets, the flip test and the growth split | +| `attrs.py` | each pattern's native attribute cell on a whole capture | +| `text.py` | each pattern's constant phrase on a whole capture, what the phrase matches, and the statement signals in its text | +| `integrations.json` | the attributes each container's documented Datadog pipeline parses, with the pages read | +| `report.py` | renders `results/results.md`, `results/results.json` and the tables above; `--check` fails CI if the committed figures do not follow from the kept outputs | +| `../accounting-lib/` | the plan request, the replay, the policy scaling and the two counts, shared with the other accounting benchmarks | +| `results/plan--.json` | the plan the MCP wrote | +| `results/policy--.*` | the files the engine read, with the factor | +| `results/.tally.json`, `results/.classes.json` | the counts per route, severity and pattern, and per logger class | +| `results/replay-config.diff` | the one field added to the shared replay config: each event's text | + +## Data + +- **Capture A.** `otel-sample-200mb.log` from + [otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1), + 215,039,161 bytes, 197,430 lines, sha256 + `aa79b9349a4123d88936fd064f5eccd9fd84f45a1fd03cff3f64c259715c2432`: about ten hours of the + OpenTelemetry demo through Fluentd, the asset [`datadog-flex-accounting/`](../datadog-flex-accounting/) + runs on. +- **Capture B.** `rec1155.log.gz` from + [intent-data-v1](https://github.com/log-10x/benchmarks/releases/tag/intent-data-v1), sha256 + `d9ebe8c79e3ef6c1ca656599203d588831b6ed72e1f245cc32cd26124b6c4301`, expanded 51,439,789 bytes, + 41,713 lines, sha256 `df78e200d929bc57f826a396fb8e1764717bf6b61544970261f3647163351b77`: two + hours of the unmodified demo images through Fluent Bit, the recording + [`intent/`](../intent/) runs on. +- **The slices.** Each capture's records whose `kubernetes.container_name` is `kafka`, + written byte for byte by `slice.py`; `run.sh` checks their sha256 and + `results/results.md` lists them. diff --git a/datadog-service-control/arms.py b/datadog-service-control/arms.py index 27eb72e..d89717d 100644 --- a/datadog-service-control/arms.py +++ b/datadog-service-control/arms.py @@ -2,7 +2,7 @@ The arms on one service, computed from the kept class rows (classes.py), which equal the tallies count.py checked against the engine's own aggregators. -Selection rule (README, "Selection rule"): per target, the fewest logger classes, largest by +Selection rule (RECORD.md, "Selection rule"): per target, the fewest logger classes, largest by events first, whose events reach at least the events the engine marked tier_down under the plan on capture A; the overshoot is reported, not trimmed. The filter for a class is `service:kafka @logger.name:`. Events with no logger class cannot be named by such a diff --git a/datadog-service-control/report.py b/datadog-service-control/report.py index 3892f89..af03aee 100644 --- a/datadog-service-control/report.py +++ b/datadog-service-control/report.py @@ -1,8 +1,8 @@ #!/usr/bin/env python3 """ -Renders results/results.md, results/results.json and the README's generated blocks from the +Renders results/results.md, results/results.json and RECORD.md's generated blocks from the kept class rows, tallies, plans and policy files. `--check` recomputes them and fails if the -committed results.json or README differ, so a figure edited without a rerun is caught in CI. +committed results.json or RECORD.md differ, so a figure edited without a rerun is caught in CI. """ import argparse import json @@ -457,9 +457,9 @@ def main() -> int: blocks = readme_blocks(r) if a.check: committed = json.load(open(results / 'results.json')) - stale = README.differing((HERE / 'README.md').read_text(), blocks) + stale = README.differing((HERE / 'RECORD.md').read_text(), blocks) if stale: - raise SystemExit(f'README.md differs from the results in: {", ".join(stale)}; run report.py to rewrite them') + raise SystemExit(f'RECORD.md differs from the results in: {", ".join(stale)}; run report.py to rewrite them') if committed != json.loads(json.dumps(r)): raise SystemExit('results.json does not follow from the kept outputs') if (results / 'results.md').read_text() != render(r): @@ -470,9 +470,9 @@ def main() -> int: return 0 json.dump(r, open(results / 'results.json', 'w'), indent=1) (results / 'results.md').write_text(render(r)) - readme = HERE / 'README.md' + readme = HERE / 'RECORD.md' readme.write_text(README.apply(readme.read_text(), blocks)) - print(f'wrote {results / "results.md"}, results.json and the README\'s generated blocks') + print(f'wrote {results / "results.md"}, results.json and the generated blocks in RECORD.md') return 0 diff --git a/drain3-vs-log10x/README.md b/drain3-vs-log10x/README.md index 9b32bb9..34f71f1 100644 --- a/drain3-vs-log10x/README.md +++ b/drain3-vs-log10x/README.md @@ -1,148 +1,44 @@ -# log10x vs Drain3 — pattern stability and lossless reduction benchmark +# log10x vs Drain3: pattern stability and lossless reduction -Reproducible harness behind the Log10x blog post *"The pattern ID you can plot: stability -and reversibility measured on 16 real datasets"*. +Does a log line keep the same pattern ID whatever order the lines arrive in, and can the original line be rebuilt byte for byte? This benchmark gives the same lines to log10x and to Drain3, the open-source template miner, on 16 public loghub datasets. log10x kept every ID stable on all 16 and rebuilt 99.98% of lines exactly, against 66.48% for Drain3. Drain3 found fewer templates, closer to the hand-labelled count. -It feeds the **same raw log lines** to both tools across the 16 -[loghub](https://github.com/logpai/loghub) 2k datasets (~32,000 lines) and measures: +## Result -1. **Pattern stability** — does a line get the same pattern ID regardless of file / order? -2. **Losslessness (reversibility)** — can you reconstruct the exact original bytes from the - reduced form? +Figures over the 16 loghub samples of 2,000 lines each. -Representation size / compression is out of scope: this benchmark is about stability and -reversibility, not storage cost. The compaction measurement on the 14 full LogHub datasets, -with this folder's encode config, is in [`loghub-compaction/`](../loghub-compaction/). +| Measure | log10x | Drain3 | +|---|---:|---:| +| Same ID in both orders (worst) | 100% | 83.4% (Zookeeper) | +| Same ID alone and combined (worst) | 100% | 0.8% (Windows) | +| Lines rebuilt byte for byte (mean) | 99.98% | 66.48% | +| Datasets rebuilt byte for byte | 14 of 16 | 5 of 16 | +| Templates found | 4,367 | 1,761 | -Reference results are committed in [`bench/facts.json`](bench/facts.json) (canonical roll-up) -and [`bench/results.json`](bench/results.json) (per-dataset). Rerunning should reproduce them. +The second row compares the ID a line gets when its half is read alone with its ID in the whole file. loghub's hand labels list 1,363 templates. -### Where the stability numbers live +- Drain3's patterns depend on reading order. The same 2,000 Mac lines in two orders gave 37 templates found only in one order and 40 only in the other. log10x found the same set in both orders on every dataset. +- log10x missed exact rebuild on BGL (99.75%) and Thunderbird (99.9%). On the full BGL file of 4,747,963 lines it rebuilt 99.73%. Of the 12,648 lines that differ, 11,904 quote a date with a zone name and come back as the same instant written in another zone. The rest trace to two other defects listed in the record. -Every per-dataset stability figure is under `datasets..stability` in -[`bench/facts.json`](bench/facts.json), for both tools: +## How it was measured -| field | meaning | -|---|---| -| `order_pct` | % of lines given the same pattern ID in `A+B` and `B+A` | -| `ctxA_pct` / `ctxB_pct` | % of lines given the same pattern ID alone (`A`) as in context (`A+B`) | -| `templates_AB` / `templates_BA` | distinct patterns mined from the same 2,000 lines, in each order | -| `set_stable` | whether the two orders produce the *same* set of patterns | -| `set_AB_minus_BA` / `set_BA_minus_AB` | how many patterns exist in one order but not the other | +Both tools read identical bytes, with line endings converted to LF. The 10x engine ran in Docker as a codec: one pass encodes, a second decodes, and the output is compared with the input. Drain3 ran with default settings and has no decoder, so the harness rebuilds each line from its template by token position, which joins tokens with single spaces. Drain3 therefore misses exactly the lines with a tab, a run of spaces or a leading or trailing space. For stability, each dataset is split into halves A and B, and both tools run on A, B, A+B and B+A. -The two set-difference fields are what "the two orders produce different template sets" -means concretely. Mac, Drain3: `templates_AB` = 308, `templates_BA` = 311, -`set_AB_minus_BA` = 37, `set_BA_minus_AB` = 40 — i.e. 37 patterns are mined only when A is -read first and 40 only when B is read first, from the identical 2,000 lines. log10x is -`0`/`0` on every dataset. These are stable run to run (Drain3's mining is deterministic for -a fixed input order). +## Limits -## Prerequisites +- The data is public research logs, plus one full BGL file. +- Storage size is out of scope. Compaction on the full loghub datasets is in [loghub-compaction](../loghub-compaction/). +- Every Mac and Windows line has whitespace that the token-position rebuild cannot restore, so Drain3 scores 0% there. +- For a short template list close to the hand labels, Drain3 is the better fit. log10x found more templates on 14 of 16 datasets, and 136,461 against Drain3's 842 on the full BGL file. -- **Python 3.9+** with Drain3 pinned: `pip install -r requirements.txt` (Drain3 0.9.11). -- **Docker**, to run the log10x engine image `log10x/pipeline-10x:1.1.136` (pinned by digest in the scripts). - No API key is needed — local file I/O runs under the image's built-in limited license. -- The 16 loghub datasets (see Setup). +## Run it -## Setup - -Clone loghub into this folder and normalize line endings to LF (loghub's committed files -are CRLF; both tools must see identical bytes): - -```bash -cd drain3-vs-log10x -git clone --depth 1 https://github.com/logpai/loghub.git -python - <<'PY' -import glob -for f in glob.glob("loghub/*/*_2k.log"): - d = open(f, "rb").read() - open(f, "wb").write(d.replace(b"\r\n", b"\n").replace(b"\r", b"\n")) -PY ``` - -Paths are resolved relative to this folder (the parent of `bench/`). To point the scripts -at data elsewhere, set `BENCH_DIR=/path/to/base` (expects `loghub/` and writes `bench/out/` -under it). - -## Run - -```bash -cd bench -python bench.py # runs Drain3 (default+masked) and log10x on all 16 -> results.json -python stability.py Mac # A / B / A+B / B+A pattern-stability test on one dataset -python stability.py all # ... on all 16 -> bench/stability/stability_.json -python facts.py # consolidate -> facts.json (the canonical numbers) -python analyze.py # print the aggregate comparison table +pip install -r requirements.txt +python smoke_test.py # Drain3 whitespace check, no Docker or data +cd bench # full rerun, needs the loghub setup +python bench.py && python stability.py all && python facts.py ``` -Independent verification (recompute from primary files, not from the summary JSON): - -```bash -python verify_lossless.py # fresh byte-exact recheck of log10x + whitespace-run counts -python rederive.py # recompute every aggregate from raw + outputs + a fresh Drain3 -``` - -## Full-dataset run (optional) - -The 2k samples measure losslessness and stability; `bench/bigfile.py` measures how the -fixed template-dictionary cost amortizes at full scale, on one big loghub dataset, -streaming and memory-bounded. The full datasets are hosted on -[Zenodo (record 8196385)](https://zenodo.org/records/8196385): - -```bash -cd drain3-vs-log10x -mkdir -p bigfile && cd bigfile -curl -L -o BGL.zip "https://zenodo.org/records/8196385/files/BGL.zip?download=1" # ~57 MB zip -unzip BGL.zip # -> BGL.log (743,185,031 bytes, 4,747,963 lines, LF endings) -cd ../bench -python bigfile.py BGL ../bigfile/BGL.log -``` - -Reference results for BGL: - -- log10x (engine 1.1.136, measured 2026-10-07): **4,735,315 of 4,747,963 lines come back - identical (99.73%)**; 136,461 cold templates; encode about 2 minutes. Of the 12,648 lines that - do not: 11,904 quote a date with a zone name (`Fri Jun 17 07:25:00 PDT 2005`), which keeps its - instant and decodes in another zone; 737 are the events of 13 templates whose hash is eight - characters or fewer, which the decoder skips; and 7 follow from the 4 events of one template - whose encoded record ends early, folding the next event into it, so 3 come back changed and 4 - do not come back. (Decode is a separate pass and is slow - here: loading 136,461 templates before expanding 4.7M records is the cost of the cold, - over-segmented dictionary. The figure above used the image's own jar outside Docker, because - writing 700 MB through a macOS bind mount is slower still.) -- Drain3 (0.9.11, measured 2026-07-07): 842 templates; 100% lossless on the sampled first 200,000 - lines (BGL has no collapsible whitespace, so token-aligned reconstruction is byte-exact); mine - ~194 s. - -## What each script does - -| script | role | -|---|---| -| `bench.py` | drives Drain3 (mine + reconstruct by token alignment) and log10x (docker) on every dataset; writes `results.json` and the per-dataset log10x output tree under `bench/out/` | -| `stability.py` | splits a dataset in half and runs both tools on A, B, A+B, B+A; measures order- and context-invariance of pattern IDs, and the per-order template-set differences (uses `tenx-stability.config.yaml`, grouping disabled for 1:1 line↔event) | -| `facts.py` | rolls `results.json` + stability outputs into `facts.json` (incl. the `stability` block above) | -| `analyze.py` | prints the aggregate table | -| `verify_lossless.py` | independent byte-exact reconstruction check + per-dataset whitespace-line counts (the lines drain3 cannot reproduce byte-for-byte) | -| `rederive.py` | independent aggregate recomputation from raw files + log10x outputs + a fresh Drain3 run | -| `bigfile.py` | full-dataset run (streaming): losslessness on every line + template count + timings on a big loghub dataset | -| `smoke_test.py` | self-contained check (no Docker/loghub) of the Drain3 whitespace-collapse mechanism; run in CI | - -## The log10x pipeline configs - -The engine is a codec, run in two passes: - -- `tenx-encode.config.yaml` — **encode**: read a raw log file, write the template - dictionary (`templates.json`) and the encoded stream (`encoded.log`). -- `tenx-decode.config.yaml` — **decode**: read a compact file (`cat templates.json - encoded.log > compact.log`); the engine auto-loads the templates, recognizes the - `~`-prefixed encoded records, and writes their original text (`decoded.log`). -- `tenx-stability.config.yaml` — encode with multi-line grouping disabled (1:1 line↔event), - used by `stability.py` to read one content-addressed hash per line. - -`bench.py` runs both passes per dataset and diffs `decoded.log` against the input. - -`results-artifact.html` is a self-contained visual summary of the results. - -## Versions +A full rerun needs Docker, the engine image `log10x/pipeline-10x:1.1.136`, Drain3 0.9.11 and the loghub clone in [RECORD.md](RECORD.md#setup). Reference numbers are in [bench/facts.json](bench/facts.json). -Drain3 0.9.11 · log10x engine 1.1.136 (`log10x/pipeline-10x:1.1.136@sha256:fccff37a6a41e52eb4df8a108d042b0176210842b7ed25f08b1419df3a37904f`) · loghub 2k samples. +[RECORD.md](RECORD.md) has the field definitions, setup, the full BGL run and what each script does. diff --git a/drain3-vs-log10x/RECORD.md b/drain3-vs-log10x/RECORD.md new file mode 100644 index 0000000..9b32bb9 --- /dev/null +++ b/drain3-vs-log10x/RECORD.md @@ -0,0 +1,148 @@ +# log10x vs Drain3 — pattern stability and lossless reduction benchmark + +Reproducible harness behind the Log10x blog post *"The pattern ID you can plot: stability +and reversibility measured on 16 real datasets"*. + +It feeds the **same raw log lines** to both tools across the 16 +[loghub](https://github.com/logpai/loghub) 2k datasets (~32,000 lines) and measures: + +1. **Pattern stability** — does a line get the same pattern ID regardless of file / order? +2. **Losslessness (reversibility)** — can you reconstruct the exact original bytes from the + reduced form? + +Representation size / compression is out of scope: this benchmark is about stability and +reversibility, not storage cost. The compaction measurement on the 14 full LogHub datasets, +with this folder's encode config, is in [`loghub-compaction/`](../loghub-compaction/). + +Reference results are committed in [`bench/facts.json`](bench/facts.json) (canonical roll-up) +and [`bench/results.json`](bench/results.json) (per-dataset). Rerunning should reproduce them. + +### Where the stability numbers live + +Every per-dataset stability figure is under `datasets..stability` in +[`bench/facts.json`](bench/facts.json), for both tools: + +| field | meaning | +|---|---| +| `order_pct` | % of lines given the same pattern ID in `A+B` and `B+A` | +| `ctxA_pct` / `ctxB_pct` | % of lines given the same pattern ID alone (`A`) as in context (`A+B`) | +| `templates_AB` / `templates_BA` | distinct patterns mined from the same 2,000 lines, in each order | +| `set_stable` | whether the two orders produce the *same* set of patterns | +| `set_AB_minus_BA` / `set_BA_minus_AB` | how many patterns exist in one order but not the other | + +The two set-difference fields are what "the two orders produce different template sets" +means concretely. Mac, Drain3: `templates_AB` = 308, `templates_BA` = 311, +`set_AB_minus_BA` = 37, `set_BA_minus_AB` = 40 — i.e. 37 patterns are mined only when A is +read first and 40 only when B is read first, from the identical 2,000 lines. log10x is +`0`/`0` on every dataset. These are stable run to run (Drain3's mining is deterministic for +a fixed input order). + +## Prerequisites + +- **Python 3.9+** with Drain3 pinned: `pip install -r requirements.txt` (Drain3 0.9.11). +- **Docker**, to run the log10x engine image `log10x/pipeline-10x:1.1.136` (pinned by digest in the scripts). + No API key is needed — local file I/O runs under the image's built-in limited license. +- The 16 loghub datasets (see Setup). + +## Setup + +Clone loghub into this folder and normalize line endings to LF (loghub's committed files +are CRLF; both tools must see identical bytes): + +```bash +cd drain3-vs-log10x +git clone --depth 1 https://github.com/logpai/loghub.git +python - <<'PY' +import glob +for f in glob.glob("loghub/*/*_2k.log"): + d = open(f, "rb").read() + open(f, "wb").write(d.replace(b"\r\n", b"\n").replace(b"\r", b"\n")) +PY +``` + +Paths are resolved relative to this folder (the parent of `bench/`). To point the scripts +at data elsewhere, set `BENCH_DIR=/path/to/base` (expects `loghub/` and writes `bench/out/` +under it). + +## Run + +```bash +cd bench +python bench.py # runs Drain3 (default+masked) and log10x on all 16 -> results.json +python stability.py Mac # A / B / A+B / B+A pattern-stability test on one dataset +python stability.py all # ... on all 16 -> bench/stability/stability_.json +python facts.py # consolidate -> facts.json (the canonical numbers) +python analyze.py # print the aggregate comparison table +``` + +Independent verification (recompute from primary files, not from the summary JSON): + +```bash +python verify_lossless.py # fresh byte-exact recheck of log10x + whitespace-run counts +python rederive.py # recompute every aggregate from raw + outputs + a fresh Drain3 +``` + +## Full-dataset run (optional) + +The 2k samples measure losslessness and stability; `bench/bigfile.py` measures how the +fixed template-dictionary cost amortizes at full scale, on one big loghub dataset, +streaming and memory-bounded. The full datasets are hosted on +[Zenodo (record 8196385)](https://zenodo.org/records/8196385): + +```bash +cd drain3-vs-log10x +mkdir -p bigfile && cd bigfile +curl -L -o BGL.zip "https://zenodo.org/records/8196385/files/BGL.zip?download=1" # ~57 MB zip +unzip BGL.zip # -> BGL.log (743,185,031 bytes, 4,747,963 lines, LF endings) +cd ../bench +python bigfile.py BGL ../bigfile/BGL.log +``` + +Reference results for BGL: + +- log10x (engine 1.1.136, measured 2026-10-07): **4,735,315 of 4,747,963 lines come back + identical (99.73%)**; 136,461 cold templates; encode about 2 minutes. Of the 12,648 lines that + do not: 11,904 quote a date with a zone name (`Fri Jun 17 07:25:00 PDT 2005`), which keeps its + instant and decodes in another zone; 737 are the events of 13 templates whose hash is eight + characters or fewer, which the decoder skips; and 7 follow from the 4 events of one template + whose encoded record ends early, folding the next event into it, so 3 come back changed and 4 + do not come back. (Decode is a separate pass and is slow + here: loading 136,461 templates before expanding 4.7M records is the cost of the cold, + over-segmented dictionary. The figure above used the image's own jar outside Docker, because + writing 700 MB through a macOS bind mount is slower still.) +- Drain3 (0.9.11, measured 2026-07-07): 842 templates; 100% lossless on the sampled first 200,000 + lines (BGL has no collapsible whitespace, so token-aligned reconstruction is byte-exact); mine + ~194 s. + +## What each script does + +| script | role | +|---|---| +| `bench.py` | drives Drain3 (mine + reconstruct by token alignment) and log10x (docker) on every dataset; writes `results.json` and the per-dataset log10x output tree under `bench/out/` | +| `stability.py` | splits a dataset in half and runs both tools on A, B, A+B, B+A; measures order- and context-invariance of pattern IDs, and the per-order template-set differences (uses `tenx-stability.config.yaml`, grouping disabled for 1:1 line↔event) | +| `facts.py` | rolls `results.json` + stability outputs into `facts.json` (incl. the `stability` block above) | +| `analyze.py` | prints the aggregate table | +| `verify_lossless.py` | independent byte-exact reconstruction check + per-dataset whitespace-line counts (the lines drain3 cannot reproduce byte-for-byte) | +| `rederive.py` | independent aggregate recomputation from raw files + log10x outputs + a fresh Drain3 run | +| `bigfile.py` | full-dataset run (streaming): losslessness on every line + template count + timings on a big loghub dataset | +| `smoke_test.py` | self-contained check (no Docker/loghub) of the Drain3 whitespace-collapse mechanism; run in CI | + +## The log10x pipeline configs + +The engine is a codec, run in two passes: + +- `tenx-encode.config.yaml` — **encode**: read a raw log file, write the template + dictionary (`templates.json`) and the encoded stream (`encoded.log`). +- `tenx-decode.config.yaml` — **decode**: read a compact file (`cat templates.json + encoded.log > compact.log`); the engine auto-loads the templates, recognizes the + `~`-prefixed encoded records, and writes their original text (`decoded.log`). +- `tenx-stability.config.yaml` — encode with multi-line grouping disabled (1:1 line↔event), + used by `stability.py` to read one content-addressed hash per line. + +`bench.py` runs both passes per dataset and diffs `decoded.log` against the input. + +`results-artifact.html` is a self-contained visual summary of the results. + +## Versions + +Drain3 0.9.11 · log10x engine 1.1.136 (`log10x/pipeline-10x:1.1.136@sha256:fccff37a6a41e52eb4df8a108d042b0176210842b7ed25f08b1419df3a37904f`) · loghub 2k samples. diff --git a/e103-grouping/README.md b/e103-grouping/README.md index 2342b74..38c49ae 100644 --- a/e103-grouping/README.md +++ b/e103-grouping/README.md @@ -1,100 +1,48 @@ -# E103: multi-line grouping across containers +# Multi-line grouping across containers -A minimal reproduction of engine ask E103 on the shipped engine image. In the `elastic-hot-tier-budget` run, 71 of 848,190 returned records held lines from more than one container: kafka's four startup lines (seq 2751 to 2754) joined the jaeger record at seq 2750 and were filed under jaeger. This rig sends the same record shape over the same path and counts such records. +When several containers send to the 10x engine through Fluent Bit under one tag, does each container's multi-line event come back in its own record? On the shipped 1.1.150 image it did not: 1,533 of 5,836 returned records mixed lines from more than one container. A build with a grouping fix returned no mixed records, and neither build lost or duplicated a line. -## What runs +The shipped Fluent Bit input takes an event's source from the Forward tag, and the step that joins multi-line events counts its flush timeout per source. In the [Elasticsearch hot-tier benchmark](../elastic-hot-tier-budget/) every container shared the tag `app.replay`, and 71 of 848,190 returned records were mixed. This rig repeats that record shape and path without Elasticsearch. -The path is the hot-tier rig's, with Elasticsearch dropped. A sender writes Fluent Forward records to Fluent Bit on 24224. Fluent Bit moves the container into a `kubernetes` map (`fluent-bit/prep.lua`, the rig's `prep`), copies every record to `in.jsonl`, and forwards to the Receiver on `tenx:24224` with one worker and one connection. The Receiver returns records to Fluent Bit's second forward input on 24225, which writes them to `returned.jsonl`. +## Result -The Receiver is started the way `gen-receiver-values.mjs` rendered it for the two-arm runs (`elastic-hot-tier-budget/results-two-arm*/values/compose.receiver.yaml`): +A stack trace counts as whole when its head and all its continuation lines came back alone in one record. -- image `ghcr.io/log-10x/edge-10x:1.1.150@sha256:3e755bbc14780a27e643e09b76dfd7ec9616cab67833f70daf9420eff646d350` -- arguments `@run/input/forwarder/fluentbit @apps/receiver symbolMessageHashField tenx_hash outputEncodeField fullText("tenx_hash","routeState","message_pattern","severity_level","utf8Size")` -- environment `TENX_AIRGAPPED=true` and `TENX_FLUENTBIT_OUTPUT_HOST=fluent-bit` -- 0.5 CPU, 512 MB, tmpfs on `/tmp` +| Case | Mixed, shipped | Mixed, fix | Traces whole, shipped | Traces whole, fix | +|---|---:|---:|---:|---:| +| a: other container inside a trace | 512 | 0 | 0 of 512 | 509 of 512 | +| b: kafka startup lines after jaeger | 509 | 0 | none sent | none sent | +| c: two traces interleaved | 512 | 0 | 0 of 1,024 | 1,021 of 1,024 | +| d: one container (control) | 0 | 0 | 509 of 512 | 509 of 512 | +| e: a, b, c, one tag each | 0 | 0 | 508 of 511 | 508 of 511 | +| Total | 1,533 | 0 | 1,017 of 2,559 | 2,547 of 2,559 | -The rendered `GH_*` variables (the policy branch) are left out, so no cap is set and nothing is held. The image keeps its config at `/etc/tenx/config` (`TENX_CONFIG`) and its modules at `/opt/tenx-edge/lib/app/modules` (`TENX_MODULES`). `E103_CONFIG_DIR` and `E103_MODULES_DIR` mount host directories read-only over them, so a candidate fix can be tested on the same image. `run.json` records the image ID and each directory's git commit. +On the shipped image, cases a and c produced a mixed record in every episode. Case e sends the same patterns with one tag per container and returned none, so the mixing follows the shared tag. With the fix, all 5,868 records containing sent lines came back under the tag they were sent with. -In the shipped modules, the Fluent Bit input takes the event's source from the Forward tag (`pipelines/run/modules/input/forwarder/fluentbit/stream.yaml`, `sourcePattern: \"tag\":\"(.*?)\"`). The group step's flush timeout counts per source (`pipelines/run/units/transform/group/unit.yaml`). The hot-tier sender gave every container the tag `app.replay`. +- Both builds returned all 22,557 lines sent, each exactly once. +- The fix build still split 12 traces, all in episodes paused 5 or 6 s against a 5 s group flush timeout. The six in cases d and e also split on the shipped image. -## Cases +## How it was measured -`send.py` uses `pack` and `Sender` from `elastic-hot-tier-budget/rig/forward_send.py`. Every record carries `container_name`, `container_id`, `source`, `log` and `seq`. Every line ends with the token `e103id=`, which sits inside the JSON on the jaeger line. Each case has its own container names, and cases are 10 s apart. Cases a to d send every container under the tag `app.replay`, the way the hot-tier sender did. +A Python sender writes Fluent Forward records to Fluent Bit 4.2, which forwards them to the 10x engine over one connection and writes what comes back to a file. `score.py` matches each returned line to the line sent through a sequence token at the end of every line. Each case sends 500 episodes back to back and 12 more paused between 1 and 6 s, 22,557 lines in all. The engine ran with 0.5 CPU, 512 MB and no budget set. -- **a.** orders logs an ERROR head. Then billing's timestamped INFO lines (1 to 3) arrive, and then orders' stack trace (3 to 8 lines of `\tat ...`, `Caused by: ...` and `\t... N more`). -- **b.** jaeger logs its JSON line. kafka's four startup lines from the hot-tier run follow (no timestamp, no level), then kafka's own timestamped INFO line. -- **c.** orders and payments are both partway through a stack trace. orders' head and two frames go out, then payments' head, then the remaining frames of both, alternating. -- **d.** Control with one container. orders sends a head and its stack trace, then 0 to 2 INFO lines. -- **e.** Cases a, b and c in rotation, with one tag per container (`docker.`, the fluentd driver's default tag). +## Limits -Each case sends 500 episodes back to back. Lines within an episode go out with no sleep between them, and episodes are 1 ms apart. Twelve paused episodes are spread through each case, with pauses of 1, 2, 3, 4, 5 and 6 s, twice each. The pause comes right after the first container's head, and in case c after orders' second frame. The full run sends 22,557 lines in 6,461 events. `groupFlushTimeout` is 5 s, and on a forwarder input a batch reaches the group step up to about 2 s after its line. +- One run per build of about five minutes, on lines from a sender script. Real container logs were not tested. +- The fix build is a local image of 1.1.150 with the engine jar and modules from the fix commits. +- The shipped run did not record the tag each record came back under, so the tag check covers the fix run only. -## Scoring +## Run it -`score.py /data` splits each returned record's `log` on `"\n"` and matches every line to the line sent by its token. It reports the following, per case and in total: - -- records returned and lines returned -- cross-container records: records whose lines came from more than one sender container -- lines missing and lines duplicated -- stack-trace events returned whole: the head and every continuation line in one record that holds nothing else -- cross-container records by the pause in their episode -- records returned under the wrong tag: Fluent Bit stores the tag a record came back under as `_fb_tag`, and the scorer checks it equals `tenx.` followed by the tag its first line was sent under - -It exits 1 when there is any cross-container record, missing line, duplicated line or wrong tag, and 0 otherwise. - -`score.py --self-test` builds three inputs. A clean input must pass. The README's shape (a jaeger record holding kafka's banner lines) must fail, and so must a record returned under the wrong tag. All three verdicts came out as expected. On the full manifest, a simulated grouping in arrival order per tag gives 1,536 cross-container records and exit 1. A simulated grouping per container gives 0 and exit 0. - -## Running - -Docker must have no other rig running: `run.sh up` refuses to start when any container is up other than `e103-*` and `census-ch`. The rig publishes no host port. A run takes about 5 minutes. +Scoring recomputes from the committed results, in this directory: ``` -./run.sh all # 1.1.150 as shipped, results in results/shipped-1.1.150/ -./run.sh extract /path/to/e103-fix # the image's own config and modules, to patch -E103_CONFIG_DIR=/path/to/e103-fix/config E103_MODULES_DIR=/path/to/e103-fix/modules E103_LABEL=fix ./run.sh all -E103_IMAGE=local/edge-10x:e103 E103_JAVA_HEAP=320M E103_LABEL=fix-1.1.151 ./run.sh all # another image +python3 score.py --self-test +python3 score.py results/shipped-1.1.150/data # exits 1: mixed records +python3 score.py results/fix2-1.1.151/data # exits 0 +./run.sh all # full rerun of 1.1.150 ``` -`E103_JAVA_HEAP` sets `TENX_JAVA_HEAP` for an image that runs the engine jar on a JVM, so the heap fits the 512 MB limit. - -`run.sh all` starts the containers with `docker run -d --name e103-*`, sends, waits for the returned records, saves logs and Fluent Bit's counters, scores, gzips the data and removes the containers and the network (`docker rm -f -v`). - -## Result on 1.1.150 - -Run 2026-10-25 from 22:50 to 22:55 UTC. Fluent Bit did not yet store the returned tag, so this run has no tag check. Results are in `results/shipped-1.1.150/`: `run.json` holds the image ID and launch, and `data/` holds the lines sent, the records Fluent Bit received and returned, and `score.json`. - -**Verdict: FAIL, with 1,533 cross-container records.** Fluent Bit took in 22,557 records, forwarded 22,557 to the Receiver and wrote 5,836 returned records. One of those is the Receiver's own connect message. - -| Case | Lines sent | Records | Lines returned | Cross-container records | Lines missing | Lines duplicated | Stack traces whole | -|---|---:|---:|---:|---:|---:|---:|---:| -| a | 4,428 | 1,554 | 4,428 | 512 | 0 | 0 | 0 of 512 | -| b | 3,072 | 1,036 | 3,072 | 509 | 0 | 0 | none sent | -| c | 6,602 | 1,024 | 6,602 | 512 | 0 | 0 | 0 of 1,024 | -| d | 3,781 | 1,027 | 3,781 | 0 | 0 | 0 | 509 of 512 | -| e | 4,674 | 1,194 | 4,674 | 0 | 0 | 0 | 508 of 511 | -| Total | 22,557 | 5,836 | 22,557 | 1,533 | 0 | 0 | | - -In cases a and c, every episode produced one cross-container record, paused or not. In case b, 509 of 512 episodes did. The three that did not were the two 6 s pauses and one 5 s pause, where the jaeger record was sealed before kafka's lines arrived. Case e sends the same three patterns with one tag per container and returns no cross-container record, so the cross-container records follow the shared tag. - -**Example from case b, the hot-tier shape.** The record with seq 4429 is filed under b-jaeger. It holds seq 4429 (b-jaeger, the JSON line) and seq 4430 to 4433 (b-kafka: `===> User`, `uid=1000(appuser) ...`, `===> Setting default values ...` and `===> Configuring ...`). All five lines were sent under the tag `app.replay`. - -Six stack traces in cases d and e came back split, three in each case. All six were in episodes paused 5 or 6 s after the head, and each came back as 4 to 7 records. - -## Result with the E103 fix - -Run 2026-10-26 from 00:00 to 00:04 UTC. Results are in `results/fix2-1.1.151/`. - -The fix is engine commit `0cd458f0f133daab281bfc9d067e38c1410ce050` (log-10x/engine, branch `fix/e103-sequence-key`: "Key a line's sequence values in any field order"). It ships with modules commit `6c69d90a015adaffc2163e927682d5f6304e504d` (log-10x/modules, branch `fix/e103-sequence-key`: "Sequence patterns capture the field name with its value"). The image `local/edge-10x:e103` (`sha256:8f74b8bbec2ef115fb83eb365172872afa5aec2cea4ae5fd065b49c1d77fe361`) is the 1.1.150 image with the engine jar from that commit and those modules at `/opt/tenx-edge/lib/app/modules`. Its modules match the modules commit file for file. The jar runs on a JVM, so the Receiver had `TENX_JAVA_HEAP=320M`. The arguments, other environment and limits are the same as the 1.1.150 run. - -**Verdict: PASS.** Fluent Bit took in 22,557 records, forwarded 22,557 and wrote 5,869 returned records, one of them the Receiver's connect message. All 5,868 records holding sent lines came back under `tenx.` followed by their sent tag: 4,674 under `tenx.app.replay` and 1,194 under `tenx.docker.`. - -| Case | Lines sent | Records (1.1.150) | Records (fix) | Lines returned | Cross-container records | Lines missing | Lines duplicated | Wrong tag | Stack traces whole (1.1.150) | Stack traces whole (fix) | -|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| a | 4,428 | 1,554 | 1,570 | 4,428 | 0 | 0 | 0 | 0 | 0 of 512 | 509 of 512 | -| b | 3,072 | 1,036 | 1,040 | 3,072 | 0 | 0 | 0 | 0 | none sent | none sent | -| c | 6,602 | 1,024 | 1,037 | 6,602 | 0 | 0 | 0 | 0 | 0 of 1,024 | 1,021 of 1,024 | -| d | 3,781 | 1,027 | 1,027 | 3,781 | 0 | 0 | 0 | 0 | 509 of 512 | 509 of 512 | -| e | 4,674 | 1,194 | 1,194 | 4,674 | 0 | 0 | 0 | 0 | 508 of 511 | 508 of 511 | -| Total | 22,557 | 5,836 | 5,869 | 22,557 | 0 | 0 | 0 | 0 | 1,017 | 2,547 | +Scoring needs only Python 3. A full rerun needs Docker with no other rig running and takes about five minutes. Running a fix build is in [RECORD.md](RECORD.md#running). -Every stack trace returned whole on 1.1.150 is returned whole with the fix. All twelve traces that came back split, three in each of cases a, c, d and e, were in episodes paused 5 or 6 s. Cases d and e split the same six on both runs. No returned record holds stack trace lines from more than one episode. +The complete record is in [RECORD.md](RECORD.md): the engine arguments, every case's line sequence, the scoring rules and each build's image and commits. diff --git a/e103-grouping/RECORD.md b/e103-grouping/RECORD.md new file mode 100644 index 0000000..2342b74 --- /dev/null +++ b/e103-grouping/RECORD.md @@ -0,0 +1,100 @@ +# E103: multi-line grouping across containers + +A minimal reproduction of engine ask E103 on the shipped engine image. In the `elastic-hot-tier-budget` run, 71 of 848,190 returned records held lines from more than one container: kafka's four startup lines (seq 2751 to 2754) joined the jaeger record at seq 2750 and were filed under jaeger. This rig sends the same record shape over the same path and counts such records. + +## What runs + +The path is the hot-tier rig's, with Elasticsearch dropped. A sender writes Fluent Forward records to Fluent Bit on 24224. Fluent Bit moves the container into a `kubernetes` map (`fluent-bit/prep.lua`, the rig's `prep`), copies every record to `in.jsonl`, and forwards to the Receiver on `tenx:24224` with one worker and one connection. The Receiver returns records to Fluent Bit's second forward input on 24225, which writes them to `returned.jsonl`. + +The Receiver is started the way `gen-receiver-values.mjs` rendered it for the two-arm runs (`elastic-hot-tier-budget/results-two-arm*/values/compose.receiver.yaml`): + +- image `ghcr.io/log-10x/edge-10x:1.1.150@sha256:3e755bbc14780a27e643e09b76dfd7ec9616cab67833f70daf9420eff646d350` +- arguments `@run/input/forwarder/fluentbit @apps/receiver symbolMessageHashField tenx_hash outputEncodeField fullText("tenx_hash","routeState","message_pattern","severity_level","utf8Size")` +- environment `TENX_AIRGAPPED=true` and `TENX_FLUENTBIT_OUTPUT_HOST=fluent-bit` +- 0.5 CPU, 512 MB, tmpfs on `/tmp` + +The rendered `GH_*` variables (the policy branch) are left out, so no cap is set and nothing is held. The image keeps its config at `/etc/tenx/config` (`TENX_CONFIG`) and its modules at `/opt/tenx-edge/lib/app/modules` (`TENX_MODULES`). `E103_CONFIG_DIR` and `E103_MODULES_DIR` mount host directories read-only over them, so a candidate fix can be tested on the same image. `run.json` records the image ID and each directory's git commit. + +In the shipped modules, the Fluent Bit input takes the event's source from the Forward tag (`pipelines/run/modules/input/forwarder/fluentbit/stream.yaml`, `sourcePattern: \"tag\":\"(.*?)\"`). The group step's flush timeout counts per source (`pipelines/run/units/transform/group/unit.yaml`). The hot-tier sender gave every container the tag `app.replay`. + +## Cases + +`send.py` uses `pack` and `Sender` from `elastic-hot-tier-budget/rig/forward_send.py`. Every record carries `container_name`, `container_id`, `source`, `log` and `seq`. Every line ends with the token `e103id=`, which sits inside the JSON on the jaeger line. Each case has its own container names, and cases are 10 s apart. Cases a to d send every container under the tag `app.replay`, the way the hot-tier sender did. + +- **a.** orders logs an ERROR head. Then billing's timestamped INFO lines (1 to 3) arrive, and then orders' stack trace (3 to 8 lines of `\tat ...`, `Caused by: ...` and `\t... N more`). +- **b.** jaeger logs its JSON line. kafka's four startup lines from the hot-tier run follow (no timestamp, no level), then kafka's own timestamped INFO line. +- **c.** orders and payments are both partway through a stack trace. orders' head and two frames go out, then payments' head, then the remaining frames of both, alternating. +- **d.** Control with one container. orders sends a head and its stack trace, then 0 to 2 INFO lines. +- **e.** Cases a, b and c in rotation, with one tag per container (`docker.`, the fluentd driver's default tag). + +Each case sends 500 episodes back to back. Lines within an episode go out with no sleep between them, and episodes are 1 ms apart. Twelve paused episodes are spread through each case, with pauses of 1, 2, 3, 4, 5 and 6 s, twice each. The pause comes right after the first container's head, and in case c after orders' second frame. The full run sends 22,557 lines in 6,461 events. `groupFlushTimeout` is 5 s, and on a forwarder input a batch reaches the group step up to about 2 s after its line. + +## Scoring + +`score.py /data` splits each returned record's `log` on `"\n"` and matches every line to the line sent by its token. It reports the following, per case and in total: + +- records returned and lines returned +- cross-container records: records whose lines came from more than one sender container +- lines missing and lines duplicated +- stack-trace events returned whole: the head and every continuation line in one record that holds nothing else +- cross-container records by the pause in their episode +- records returned under the wrong tag: Fluent Bit stores the tag a record came back under as `_fb_tag`, and the scorer checks it equals `tenx.` followed by the tag its first line was sent under + +It exits 1 when there is any cross-container record, missing line, duplicated line or wrong tag, and 0 otherwise. + +`score.py --self-test` builds three inputs. A clean input must pass. The README's shape (a jaeger record holding kafka's banner lines) must fail, and so must a record returned under the wrong tag. All three verdicts came out as expected. On the full manifest, a simulated grouping in arrival order per tag gives 1,536 cross-container records and exit 1. A simulated grouping per container gives 0 and exit 0. + +## Running + +Docker must have no other rig running: `run.sh up` refuses to start when any container is up other than `e103-*` and `census-ch`. The rig publishes no host port. A run takes about 5 minutes. + +``` +./run.sh all # 1.1.150 as shipped, results in results/shipped-1.1.150/ +./run.sh extract /path/to/e103-fix # the image's own config and modules, to patch +E103_CONFIG_DIR=/path/to/e103-fix/config E103_MODULES_DIR=/path/to/e103-fix/modules E103_LABEL=fix ./run.sh all +E103_IMAGE=local/edge-10x:e103 E103_JAVA_HEAP=320M E103_LABEL=fix-1.1.151 ./run.sh all # another image +``` + +`E103_JAVA_HEAP` sets `TENX_JAVA_HEAP` for an image that runs the engine jar on a JVM, so the heap fits the 512 MB limit. + +`run.sh all` starts the containers with `docker run -d --name e103-*`, sends, waits for the returned records, saves logs and Fluent Bit's counters, scores, gzips the data and removes the containers and the network (`docker rm -f -v`). + +## Result on 1.1.150 + +Run 2026-10-25 from 22:50 to 22:55 UTC. Fluent Bit did not yet store the returned tag, so this run has no tag check. Results are in `results/shipped-1.1.150/`: `run.json` holds the image ID and launch, and `data/` holds the lines sent, the records Fluent Bit received and returned, and `score.json`. + +**Verdict: FAIL, with 1,533 cross-container records.** Fluent Bit took in 22,557 records, forwarded 22,557 to the Receiver and wrote 5,836 returned records. One of those is the Receiver's own connect message. + +| Case | Lines sent | Records | Lines returned | Cross-container records | Lines missing | Lines duplicated | Stack traces whole | +|---|---:|---:|---:|---:|---:|---:|---:| +| a | 4,428 | 1,554 | 4,428 | 512 | 0 | 0 | 0 of 512 | +| b | 3,072 | 1,036 | 3,072 | 509 | 0 | 0 | none sent | +| c | 6,602 | 1,024 | 6,602 | 512 | 0 | 0 | 0 of 1,024 | +| d | 3,781 | 1,027 | 3,781 | 0 | 0 | 0 | 509 of 512 | +| e | 4,674 | 1,194 | 4,674 | 0 | 0 | 0 | 508 of 511 | +| Total | 22,557 | 5,836 | 22,557 | 1,533 | 0 | 0 | | + +In cases a and c, every episode produced one cross-container record, paused or not. In case b, 509 of 512 episodes did. The three that did not were the two 6 s pauses and one 5 s pause, where the jaeger record was sealed before kafka's lines arrived. Case e sends the same three patterns with one tag per container and returns no cross-container record, so the cross-container records follow the shared tag. + +**Example from case b, the hot-tier shape.** The record with seq 4429 is filed under b-jaeger. It holds seq 4429 (b-jaeger, the JSON line) and seq 4430 to 4433 (b-kafka: `===> User`, `uid=1000(appuser) ...`, `===> Setting default values ...` and `===> Configuring ...`). All five lines were sent under the tag `app.replay`. + +Six stack traces in cases d and e came back split, three in each case. All six were in episodes paused 5 or 6 s after the head, and each came back as 4 to 7 records. + +## Result with the E103 fix + +Run 2026-10-26 from 00:00 to 00:04 UTC. Results are in `results/fix2-1.1.151/`. + +The fix is engine commit `0cd458f0f133daab281bfc9d067e38c1410ce050` (log-10x/engine, branch `fix/e103-sequence-key`: "Key a line's sequence values in any field order"). It ships with modules commit `6c69d90a015adaffc2163e927682d5f6304e504d` (log-10x/modules, branch `fix/e103-sequence-key`: "Sequence patterns capture the field name with its value"). The image `local/edge-10x:e103` (`sha256:8f74b8bbec2ef115fb83eb365172872afa5aec2cea4ae5fd065b49c1d77fe361`) is the 1.1.150 image with the engine jar from that commit and those modules at `/opt/tenx-edge/lib/app/modules`. Its modules match the modules commit file for file. The jar runs on a JVM, so the Receiver had `TENX_JAVA_HEAP=320M`. The arguments, other environment and limits are the same as the 1.1.150 run. + +**Verdict: PASS.** Fluent Bit took in 22,557 records, forwarded 22,557 and wrote 5,869 returned records, one of them the Receiver's connect message. All 5,868 records holding sent lines came back under `tenx.` followed by their sent tag: 4,674 under `tenx.app.replay` and 1,194 under `tenx.docker.`. + +| Case | Lines sent | Records (1.1.150) | Records (fix) | Lines returned | Cross-container records | Lines missing | Lines duplicated | Wrong tag | Stack traces whole (1.1.150) | Stack traces whole (fix) | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| a | 4,428 | 1,554 | 1,570 | 4,428 | 0 | 0 | 0 | 0 | 0 of 512 | 509 of 512 | +| b | 3,072 | 1,036 | 1,040 | 3,072 | 0 | 0 | 0 | 0 | none sent | none sent | +| c | 6,602 | 1,024 | 1,037 | 6,602 | 0 | 0 | 0 | 0 | 0 of 1,024 | 1,021 of 1,024 | +| d | 3,781 | 1,027 | 1,027 | 3,781 | 0 | 0 | 0 | 0 | 509 of 512 | 509 of 512 | +| e | 4,674 | 1,194 | 1,194 | 4,674 | 0 | 0 | 0 | 0 | 508 of 511 | 508 of 511 | +| Total | 22,557 | 5,836 | 5,869 | 22,557 | 0 | 0 | 0 | 0 | 1,017 | 2,547 | + +Every stack trace returned whole on 1.1.150 is returned whole with the fix. All twelve traces that came back split, three in each of cases a, c, d and e, were in episodes paused 5 or 6 s. Cases d and e split the same six on both runs. No returned record holds stack trace lines from more than one episode. diff --git a/elastic-hot-tier-budget/README.md b/elastic-hot-tier-budget/README.md index 9c2a8d5..3c0e7ee 100644 --- a/elastic-hot-tier-budget/README.md +++ b/elastic-hot-tier-budget/README.md @@ -1,187 +1,48 @@ # Elastic hot-tier budget -One benchmark in the [log10x benchmarks](../) repository. It asks whether log10x-mcp's hourly hot-tier run, driving the Receiver, holds an Elasticsearch hot tier under a size S while load grows 2.5 times, as the hot node's own stats count it. +Can a size budget on an Elasticsearch hot tier stay where it was set when log volume steps up 2.5 times within an hour? In this test the budget stayed when the lines over the limits went to a warm node. From 20 minutes after the step to the hour's end, hot sat between 0.94 and 1.04 times the budget, while a fixed reroute by service rose to 1.66 times its level. With those lines on the hot node, hot peaked at 1.43 times. -**Result: holds at steady state; under a 2.5x step it holds within its bound with the tier-down index placed on warm, and overshoots to 1.43 S with that index on hot.** Four runs. The first held the mean but not the bound; the second, with every time constant divided by 6, held hot at 0.97 to 0.98 of S under steady load and overshot to 1.23 S during a 2.5x step-up compressed into an hour. The first two fail the pre-registered bound on the hot store. Results of the first run in `results/`, of the second in `results-r2/` (section below). A third run put a native reroute by service beside the hold under one 2.5x step (`results-two-arm/`): the hold peaked at 1.43 S, over its bound, and the reroute tracked load, rising to 1.72 times its level before the step. A fourth wrote moved lines on warm (`results-two-arm-warm/`): the hold kept hot at 0.94 to 1.04 S through the step, within its bound, and the reroute rose to 1.25 S and kept rising. +The 10x engine limits bytes per log statement, and a run every 10 minutes rewrites them against the hot node's store. Lines over a limit (moved lines below) go to an index that rolls over every 10 minutes and then mounts on frozen. -Self-managed Elasticsearch 9.5.5 (one hot node, one frozen node, filesystem snapshot repository, trial licence), engine `log10x/edge-10x:1.1.147`, log10x-mcp branch `feat/elastic-hot-tier-budget` at 6af0226, the published OpenTelemetry demo sample replayed, 2026-10-22 19:59 to 2026-10-23 09:59 UTC. `results/` holds the timeline, every hourly run's report and files, the cluster readings and slim captures (no log text). +## Result -| | Criterion | Result | -|---|---|---| -| 1 | Every applied hourly run exits 0 and commits; once the size binds, its projection is at or under S | **Pass.** 13 applied runs, all exit 0, each projecting about 45 MB against S = 44.8 MB. | -| 2 | Run hours 7 to 13: hot store at or under 1.10 S at every reading, mean at or under S | **Fail.** Mean 0.998 S over 36 readings; 6 readings above 1.10 S, peak 1.17 S at 05:19 UTC, during the step-up. | -| 3 | Every rolled tier_down index mounted on frozen at 0 bytes | **Pass.** 38 of 38. | -| 4 | Hash queries on frozen equal the local count, ten largest moved statements | **Pass.** Exact, from 170,470 down to 10,498. | -| 5 | Every line received comes back, every returned record is in Elasticsearch | **Pass.** 1,041,196 lines in and back; 162,451 kept and 672,357 moved records, each equal to the cluster's count. | -| 6 | Storm: error lines kept per 4-minute window at most limit plus 50 | **Pass.** Scored on the regulator's own windows (every 240 seconds from its first regulated line): 55 a window (the 5-line sample plus 50) in each full window; the 146 of the first minute fall in two windows either side of 05:00:01. | +Hot node index store from node stats, around one step from 56 to 140 lines a second, against a 63 MB budget. "Before" is the 20-minute mean before the step. -## The hourly table +| Setup | Moved lines written to | Before | Peak in the hour | End of the hour | +|---|---|---:|---:|---:| +| 10x engine | warm node | 61.6 MB | 65.7 MB | 65.2 MB | +| Reroute by service | warm node | 47.4 MB | 78.7 MB | 78.7 MB | +| 10x engine | hot node | 59.9 MB | 90.0 MB | 64.2 MB | +| Reroute by service | hot node | 50.0 MB | 85.9 MB | 81.8 MB | -S = 44.8 MB, set at run hour 1 as 0.7 of where the run's own dry run said hot would settle with nothing held (64 MB). Hot store is the hot node's `indices.store.size_in_bytes`; the other two columns are `_cat/indices` store for the indices not yet mounted. +The bound was 1.10 times the budget at every reading from 20 minutes after the step. With moved lines on warm, none of 25 readings crossed it. With moved lines on hot, 13 of 24 did, because each moved-lines index stays on hot for its 10 minutes of writes and about 7 more until it mounts. -| UTC | Run hour | Load | Hot store (node stats) | of S | Kept stream on hot | Tier_down waiting on hot | -|---|---:|---|---:|---:|---:|---:| -| 19:59 | 0 | 2x | 0.0 MB | 0.00 | 0.0 MB | 0.0 MB | -| 20:59 | 1 | 2x | 11.0 MB | 0.25 | 10.9 MB | 0.0 MB | -| 21:59 | 2 | 2x | 22.4 MB | 0.50 | 14.5 MB | 7.6 MB | -| 22:59 | 3 | 2x | 24.0 MB | 0.54 | 17.2 MB | 6.4 MB | -| 23:59 | 4 | 2x | 29.0 MB | 0.65 | 23.2 MB | 5.2 MB | -| 00:59 | 5 | 2x | 37.1 MB | 0.83 | 31.2 MB | 5.1 MB | -| 01:59 | 6 | 2x | 39.6 MB | 0.88 | 34.2 MB | 4.8 MB | -| 02:59 | 7 | 2.75x | 33.8 MB | 0.76 | 28.1 MB | 4.9 MB | -| 03:59 | 8 | 3.5x | 39.7 MB | 0.89 | 31.6 MB | 7.2 MB | -| 04:59 | 9 | 4.25x | 45.4 MB | 1.01 | 29.8 MB | 14.2 MB | -| 05:59 | 10 | 5x | 50.6 MB | 1.13 | 30.9 MB | 18.4 MB | -| 06:59 | 11 | 5x | 50.1 MB | 1.12 | 26.9 MB | 22.1 MB | -| 07:59 | 12 | 5x | 47.2 MB | 1.05 | 25.1 MB | 20.7 MB | -| 08:59 | 13 | stopped | 44.8 MB | 1.00 | 22.4 MB | 20.8 MB | +- The reroute grew because the other services' lines stay on hot for 2 hours. +- In the warm run every non-blank line arrived: 1,025,014 in the 10x cluster, 1,025,170 documents in the reroute cluster. -## Why the bound failed +## How it was measured -- **The always-pass floor.** The Receiver passes the first 5 lines of every statement in every window on every node, and 50 error lines over a limit. From run hour 9 the run allowed nothing more on hot, yet 1.8 to 3.2 MB an hour stayed there; 59 to 74% of those lines were the sample, across about 2,300 statements. -- **Moved lines waiting to mount.** A tier_down index rolls every 15 minutes here and mounts about 40 minutes later, so up to 22 MB of moved lines sat on hot during the growth, about half of S. Hot here keeps a kept line 6 hours; with retention of days the same hour of moved lines is about 1% of hot. +Two self-managed Elasticsearch 9.5.5 clusters shared one Docker host, each with a hot node and a warm and frozen node. Fluent Bit replayed the OpenTelemetry demo sample to both, through the 10x engine to the first. The second routed `opentelemetry-collector`, the service with the top half of the bytes, to a stream with the moved lines' lifecycle. The budget was 0.7 of where a dry run without limits said hot would settle. Every 100 seconds a meter read each hot node's `indices.store.size_in_bytes`. The methods were fixed in advance: [TWO_ARM_METHOD.md](TWO_ARM_METHOD.md) and [TWO_ARM_METHOD_WARM.md](TWO_ARM_METHOD_WARM.md). -## The rig +## Limits -`rig/`: two Elasticsearch nodes and Fluent Bit (`compose.yaml`), the Receiver rendered by log10x-mcp with `elasticHotTier` and a policy branch (`gen-receiver-values.mjs`), the MCP's frozen-tier recipe then the time-scaled policies (`es-setup.mjs`: kept lines 6 hours on hot, tier_down rollover 15 minutes), `policy.yaml` from `emitPolicyYaml` (`gen-policy.mjs`), `tenx-recur` every hour (`budget-run.sh`), the replay (`make-replay.py`, `forward_send.py`), the phases (`phases.sh`) and the meter (`meter.py`). The Receiver pulled its files from `log-10x/config` branch `test/elastic-hot-tier-policy`. Its byte limits count the UTF-8 size of the whole record the forwarder sends, not only the message, and the hourly run reads that size from the `utf8Size` field the Receiver stamps on each returned record. +- Every time constant was divided by 6, so kept lines stayed 2 hours on hot and a slow ramp against days of retention is not shown. +- Each placement ran once, about four hours, at megabyte scale on one machine. +- The cost of warm, where moved lines peaked at 20.7 MB, was not measured. +- No bill was measured. RECORD.md prices hot at Elastic's published rates. -## Not shown +## Run it -A bill; Cloud Hosted; production retention lengths; 300 nodes; a step in load larger than 2.5x; a hold over days. +``` +# from elastic-hot-tier-budget/, with Docker running +# set POLICY_REPO= to use your own copy of log-10x/config with branch test/elastic-hot-tier-policy +export RESULTS=$PWD/results-rerun MCP=/path/to/log10x-mcp GH_TOKEN=$(gh auth token) \ + ENGINE_IMAGE=ghcr.io/log-10x/edge-10x:1.1.150 +python3 rig/make-replay.py otel-sample-200mb.log # release otel-sample-v1 of log-10x/config +rig/phases-two-arm-warm.sh # about four hours +python3 rig/score-two-arm.py "$RESULTS" +``` -## The rerun (`results-r2/`), 2026-10-23 11:25 to 15:55 UTC +A rerun needs Docker with room for 7.1 GiB of container limits, a built log10x-mcp checkout at c1b456a and a GitHub token that can push the policy branch. -log10x-mcp `feat/elastic-hot-tier-budget` at 0b45d4a. Every time constant divided by 6: a run every 10 minutes, kept lines 2 hours on hot (10-minute rollover, frozen 110 minutes later), tier_down rollover 3 minutes, ILM poll 100 seconds (changed only because everything else was scaled), meter every 100 seconds, growth in four 20-minute steps from 56 to 140 lines a second. No raw capture: lines in are Fluent Bit's input counter, and the local copies of returned records keep only container, times, hash, route, `utf8Size`, level, name and line count (`results-r2/captures/`). S = 56.7 MB, 0.7 of where the dry run said hot would settle with nothing held (81 MB). - -| | Criterion | Result | -|---|---|---| -| 1 | Runs exit 0 and commit; projection at or under S, or the floor reported | **Pass.** 26 of 26 applied runs; projections 51 to 57 MB; the always-pass floor 79 to 268 MB a day, inside what fit at every run. | -| 2 | Run minutes 130 to 250: hot at or under 1.10 S every reading, mean at or under S | **Fail.** Mean 1.095 S, peak 1.23 S, 30 of 72 readings over 1.10 S. Before growth (minutes 111 to 131) 0.97 to 0.98 S. | -| 3 | Rolled tier_down indices mounted at 0 bytes | **Pass.** 73 of 73. | -| 4 | Hash queries on frozen equal local, ten largest moved statements | **Pass.** 10 of 10. | -| 5 | Lines in equal non-blank lines returned; returned records equal Elasticsearch's counts | **Pass.** Every returned record is in Elasticsearch. 1,301,174 lines in, 1,300,992 back; the 182 not returned are blank lines. The sample holds 26, all in its first 23 minutes of source time, and the replay passed over that stretch seven times (`rig/count-blank-sent.py` on `results/replay/source.jsonl` from `make-replay.py` and `results-r2/timeline.jsonl`). | -| 6 | Storm kept per window at most limit + 50 | **Pass.** On the regulator's own windows (rolling at 14:12:14 and 14:16:14): 121 and 165 kept, each within its limit plus 50 and the 5-line sample; the rest of the second window kept none, as its allowance was spent, and the storm ended before the next one. | - -**Why growth overshoots.** The run steers what new lines do; it cannot move lines already on hot. When load rose 2.5 times within an hour, the moved lines waiting to mount grew from 3 MB to 10 to 19 MB at once (the run's own estimate of that slice was right), while the kept lines already on hot leave only as their indices age out, over 2 hours here. The run tightened its allowance to 86 to 90% and projected 51 to 55 MB, but hot stayed above 1.1 S until the window closed. Relative to the time a line stays on hot, this step-up was about 50 times faster than a year of 2.5x growth against days of retention, but the run does not show what a slower ramp does. - -## Two-arm run, 2026-10-24 - -The question, from the method written before the run (`TWO_ARM_METHOD.md`, 2026-10-24): under a 2.5x step-up in load, does the Receiver's hourly hold keep the hot store at S where a native reroute by service does not? Run 2026-10-24 21:12 to 2026-10-25 01:26 UTC, results in `results-two-arm/`. - -**Result: the hold fails its bound and the reroute tracks load; delivery is exact in both arms.** Arm A peaked at 1.43 S in the scored window. Arm B rose to 1.72 times its level before the step. Within the hour of the hold, arm B's peak hot store (1.36 S) stayed below arm A's (1.43 S). At the end of the hold arm A was back at 1.02 S, while arm B was at 1.30 S and still rising. - -**Arm A, the hold**, is the `results-r2` rig with every time constant divided by 6. `tenx-recur` from log10x-mcp main at d6fa2b1 ran every 10 minutes against the engine `ghcr.io/log-10x/edge-10x:1.1.150` (sha256:3e755bbc14780a27e643e09b76dfd7ec9616cab67833f70daf9420eff646d350). Kept lines stay 2 hours on hot (10-minute rollover, frozen 110 minutes later). The tier_down index rolls every 10 minutes (the recipe's hour divided by 6) and goes frozen 1 minute after rollover. The ILM poll is 100 seconds. S = 63 MB, 0.7 of the 90 MB where the dry run at 0:10 said hot would settle with nothing held. - -**Arm B, the native reroute**, is a second Elasticsearch 9.5.5 cluster with the same image and node sizes (one hot node with a 1 GB heap, one frozen node) and its own snapshot repository. Fluent Bit copies every line before it reaches the Receiver and routes the copy with a rule fixed before the run. The services that made the top half of bytes in the feed's first 20 minutes, at the run's pace, go to `native-overflow`, which carries arm A's tier_down ILM policy copied from cluster A. Every other service goes to `native-app`, which carries arm A's kept policy. One service made the top half: `opentelemetry-collector`, with 68.9% of the bytes (`arm-b-rule.json`). Arm B has no Receiver. Its index templates are arm A's, with the stream names changed. - -Each arm has its own hot node, since the hot-tier run subtracts every index on the hot nodes that is not the Receiver's from S, so arm B's streams on arm A's node would take arm A's allowance. The method's "one Elasticsearch 9.5.5 node" is read as one hot node per arm. - -**Load.** The feed is the `results-r2` feed (`replay/source.jsonl`, sha256 24e02233a797…), sent by one sender for the whole run that numbers every line (`seq`). The start is 56 lines a second. S binds at the first meter reading at or above 0.95 S after the first applied run, or at 2:10 (R after that run), whichever comes first. Here it bound at 2:10, 23:22:19 UTC, with arm A at 0.93 S. Load stayed steady for 40 minutes, then took one step to 140 lines a second at 00:02:22, held for 60 minutes. The feed stopped at 01:02:28 after 1,070,359 lines, followed by a 20-minute drain. The run had no new service and no storm, since the method's load names neither. - -**Meter.** Every 100 seconds the meter read both clusters: the hot node's index store, each stream's store on hot and its frozen mounts (`_cat/indices`), and ILM explain. Each reading first refreshes both arms' indices on hot, since arm B's indices take no searches and a search-idle shard writes nothing to its store until a refresh. Without the refresh, a smoke reading showed 6,223 of arm B's 9,236 indexed documents. Each frozen node's cache is 128 MB (512 MB in `results-r2`), to stay inside the disk budget; the cache serves searches on frozen and nothing else. - -| | Criterion | Result | -|---|---|---| -| 1 | Arm A's hot store at or under 1.10 S at every reading from 20 minutes after the step to the end of the hold | **Fail.** 13 of 24 readings (00:23:55 to 01:02:15) over 1.10 S. Peak 1.43 S (90.0 MB at 00:57), mean 1.15 S. | -| 2 | Arm B's hot store above 1.5 times its pre-step level during the hold (pre-step level: the mean of its readings in the 20 minutes before the step); "no differentiator" if every reading from 20 minutes after the step stays within 1.10 times | **Pass.** Pre-step 50.0 MB. Above 1.5 times from 00:45:35, peak 85.9 MB (1.72 times) at 01:00. Above 1.10 times through the scored window, so the run shows a differentiator. | -| 3 | Every line in each arm on hot or frozen, by sequence number | **Pass.** Arm B: 1,070,359 documents, seq 1 to 1,070,359, none missing or repeated, each in the stream its service names. Arm A: all 1,070,203 non-blank lines are in the 848,190 returned records. Each record carries its first line's seq and holds exactly the non-blank lines from that seq to the next record's (`_lines`). The other 156 lines are blank. | - -The hot store per stream around the step, from the meter (`score.json`, `meter.jsonl.gz`): - -| UTC | Minutes from step | Arm A hot node | A kept (`tenx-app`) | A tier_down waiting | Arm B hot node | B normal (`native-app`) | B overflow (`native-overflow`) | -|---|---:|---:|---:|---:|---:|---:|---:| -| 20 minutes before, mean | | 59.9 MB (0.95 S) | 55.4 MB | 3.7 MB | 50.0 MB | 46.2 MB | 2.9 MB | -| 00:02 | 0 | 60.0 MB (0.95 S) | 57.1 MB | 2.0 MB | 51.0 MB | 46.7 MB | 3.4 MB | -| 00:23 | +22 | 78.9 MB (1.25 S) | 55.3 MB | 22.5 MB | 69.3 MB | 60.3 MB | 7.9 MB | -| 00:43 | +42 | 75.0 MB (1.19 S) | 50.5 MB | 23.4 MB | 73.5 MB | 66.0 MB | 6.3 MB | -| 00:57, arm A's peak | +55 | 90.0 MB (1.43 S) | 48.3 MB | 40.3 MB | 82.1 MB | 73.4 MB | 7.4 MB | -| 01:00, arm B's peak | +58 | 62.1 MB (0.99 S) | 49.3 MB | 11.5 MB | 85.9 MB | 76.3 MB | 8.4 MB | -| 01:02, end of hold | +60 | 64.2 MB (1.02 S) | 48.7 MB | 14.2 MB | 81.8 MB | 77.3 MB | 3.3 MB | - -**Why arm A overshoots.** The kept stream was held: 55.4 MB before the step and 46.0 to 57.7 MB during the hold. The overshoot comes from moved lines waiting to mount, which rose from 3.7 MB before the step to as much as 40.3 MB. Each tier_down index takes 10 minutes of moved lines and then waits about 7 minutes more for its mount (the run's own measure, 0.3 hours). At 2.5 times the load that is 20 to 25 MB. The 1.43 S peak is a single index, `tenx-tierdown-000019`. It read 15.3 MB at its last reading before rollover and 34.4 MB for the next three readings, until the searchable_snapshot action's force-merge step; after that it read 16.4 MB. - -**Why arm B rises.** Arm B's normal stream keeps every line of the services outside the top half for 2 hours, so it grows as lines at the new rate replace those at the old: 46.2 MB before the step and 77.3 MB at the end of the hold. The hold ended before those 2 hours had turned over. The overflow stream holds one 10-minute index plus its mount wait: 2.9 MB before the step and 2.1 to 9.7 MB during the hold. - -**Frozen.** Every rolled index in both arms mounted on frozen at 0 bytes local. Arm A mounted 20 tier_down indices and 11 kept indices; arm B mounted 20 overflow and 11 normal. The snapshot repositories hold 181 MB for arm A and 108 MB for arm B; arm A's documents carry the Receiver's fields. - -**Grouping across containers.** The Receiver groups multi-line events in the order lines arrive, so 71 of the 848,190 returned records hold lines from more than one container. For example, kafka's four startup lines at seq 2751 to 2754 joined the jaeger record at seq 2750, which is filed under jaeger. Every such line is still delivered. - -**Projection (Elastic's published arithmetic applied to the measured hot store per arm, not a measured bill).** Hot cost per TB of hot store per month = 1,000 GB / (disk-to-RAM ratio x 0.85 usable) GB of RAM x $ per GB RAM-hour x 730 hours x 2 zones. The rates come from Elastic's Cloud Hosted price table for AWS us-east-1 at Enterprise (https://cloud.elastic.co/deployment-pricing-table). The ratios come from Elastic's AWS default instance page (https://www.elastic.co/docs/reference/cloud/cloud-hosted/aws-default), which marks them as estimations. `aws.es.datahot.i8g` at $0.0793 and 27:1 gives $5,045 per TB-month; `aws.es.datahot.i3en` at $0.0590 and 74:1 gives $1,369. With S set at 1 TB of hot store, the measured ratios price as follows: - -| | Hot store | i8g, per month | i3en, per month | -|---|---:|---:|---:| -| Arm A, 20 minutes before the step | 0.95 S | $4,799 | $1,303 | -| Arm A, peak in the hold | 1.43 S | $7,207 | $1,956 | -| Arm A, end of the hold | 1.02 S | $5,144 | $1,396 | -| Arm B, 20 minutes before the step | 0.79 S | $4,008 | $1,088 | -| Arm B, peak in the hold | 1.36 S | $6,875 | $1,866 | -| Arm B, end of the hold | 1.30 S | $6,552 | $1,779 | - -What moves to frozen sits in the snapshot repository at $0.033 per GB-month. Elastic's node sizes step, so a difference in hot store saves money only when it crosses a node step. At full-size nodes a step is one node, about 1.5 TB of hot store on a 27:1 instance. A hot tier is sized for its peak, and in this hour arm A's peak was the higher of the two. - -**Smoke**, 2026-10-24 20:43 to 21:09 UTC, every phase at a tenth of its length (`results-two-arm-smoke/`). Both arms received every line: 101,241 fed. Arm B held 101,241 documents; arm A held 101,215 non-blank lines in 78,852 records, exact by the same check. Both meters read, and each arm mounted a frozen index. - -**Rig additions.** `compose.arm-b.yaml` (arm B's cluster), `es-setup-b.mjs` (arm A's ILM policies, poll and templates copied to arm B), `top-half.py` and `render-fb-two-arm.py` with `fluent-bit-two-arm/` (the fixed rule), `forward_send.py paced` and `run.sh feed` (one numbered sender, speed changed in place), `meter.py` with `ES_URL_B` (both clusters, per stream, refresh first), `dump-seq.py` and `score-two-arm.py` (delivery and the three criteria), `phases-two-arm.sh` (the schedule), and `record-build.py` (`build.json`: every image digest and the commits). - -Not shown: a measured Elastic Cloud bill, 300 nodes, compact, autoscaling, or a hold longer than the hot retention. - -## Two-arm run, warm placement, 2026-10-25 - -The question, from the method addendum written before the run (`TWO_ARM_METHOD_WARM.md`, 2026-10-25): with moved lines written to warm instead of hot, does arm A's hot store stay at S through the 2.5x step while arm B's tracks load? Run 2026-10-25 05:33 to 09:34 UTC, results in `results-two-arm-warm/`. - -**Result: the hold keeps hot at S through the step and the reroute tracks load; delivery is exact in both arms.** Arm A's hot store stayed between 0.94 and 1.04 S from 20 minutes after the step to the end of the hold (mean 1.00 S). Arm B rose to 1.66 times its level before the step and was still rising when the hold ended. No tier_down or overflow index was found on a hot node in any of 144 readings. - -**The one change.** Arm A's tier_down index and arm B's overflow stream are written on a warm node and mount on frozen from there. The tier_down template carries `index.routing.allocation.include._tier_preference: data_warm,data_hot`, rendered by log10x-mcp's recipe with `tierDownPlacement: 'warm'`; arm B's overflow template is copied from it. Each arm's frozen node also carries the `data_warm` role (`compose.warm.yaml`, 512 MB heap, 1200 MB limit, unchanged), since a third node per arm would set 9.5 GiB of container limits against the Docker VM's 7.65 GiB. Arm A's kept stream and arm B's normal stream stay on hot. Everything else is the 2026-10-24 run: the same feed (sha256 24e02233a797…), the same rule for arm B (`opentelemetry-collector`, 68.9% of bytes), the same ILM policies, rollover ages and 100-second poll, engine `ghcr.io/log-10x/edge-10x:1.1.150`. - -**The hourly run.** `tenx-recur` from log10x-mcp `feat/elastic-tierdown-warm` at c1b456a (`build.json`). It reads which node each tier_down index's shards sit on, counts only those on a `data_hot` or `data` node in the hot split, and counts moved lines in flight against S only while the tier_down write index sits on hot. Every one of the 24 runs reported the tier_down write index off hot, and all exited 0. - -**Load.** S = 63 MB, 0.7 of the 90 MB the dry run at 0:10 said hot would settle at with nothing held (the same S as the 2026-10-24 run). S bound at 07:29:53 UTC on the first rule, a reading at or above 0.95 S (60.6 MB). Steady for 40 minutes, then one step to 140 lines a second at 08:09:51, held 60 minutes. The feed stopped at 09:09:58 after 1,025,170 lines, followed by a 20-minute drain. - -| | Criterion | Result | -|---|---|---| -| 1 | Arm A's hot store at or under 1.10 S at every reading from 20 minutes after the step to the end of the hold | **Pass.** 25 readings (08:29:51 to 09:09:51), none over 1.10 S. Peak 1.04 S (65.7 MB at 08:49:51), mean 1.00 S. | -| 2 | Arm B's hot store above 1.5 times its pre-step level during the hold; "no differentiator" if every reading from 20 minutes after the step stays within 1.10 times | **Pass.** Pre-step 47.4 MB. Peak 78.7 MB (1.66 times) at the end of the hold, still rising. Above 1.10 times in the scored window, so the run shows a differentiator. | -| 3 | Every line in each arm on hot, warm or frozen, by sequence number | **Pass.** Arm B: 1,025,170 documents, seq 1 to 1,025,170, none missing or repeated, each in the stream its service names. Arm A: all 1,025,014 non-blank lines are in the 811,618 returned records, checked as in the 2026-10-24 run; the other 156 lines are blank. | - -Hot and warm per arm, from the meter (`score.json`, `meter.jsonl.gz`). Warm is reported, not scored. - -| UTC | Minutes from step | Arm A hot node | A kept (`tenx-app`) | A tier_down on warm | Arm B hot node | B normal (`native-app`) | B overflow on warm | -|---|---:|---:|---:|---:|---:|---:|---:| -| 20 minutes before, mean | | 61.6 MB (0.98 S) | 60.8 MB | 3.8 MB | 47.4 MB | 46.6 MB | 2.7 MB | -| 08:09 | 0 | 63.0 MB (1.00 S) | 62.2 MB | 6.9 MB | 47.2 MB | 46.5 MB | 3.6 MB | -| 08:29 | +20 | 62.2 MB (0.99 S) | 60.6 MB | 14.7 MB | 57.0 MB | 55.6 MB | 6.3 MB | -| 08:49, arm A's peak | +40 | 65.7 MB (1.04 S) | 64.7 MB | 10.4 MB | 69.7 MB | 68.3 MB | 4.6 MB | -| 09:09, end of hold and arm B's peak | +60 | 65.2 MB (1.03 S) | 64.0 MB | 7.9 MB | 78.7 MB | 77.6 MB | 3.7 MB | - -Warm peaked at 20.7 MB in arm A (08:34:51) and 9.1 MB in arm B (08:33:11). Each tier_down and overflow index was found mounted about 7 minutes after its rollover, as in the 2026-10-24 run; those minutes are spent on warm. Arm A mounted 19 tier_down and 10 kept indices on frozen; arm B 19 overflow and 10 normal. Arm A's peak in the hold was 1.04 S, against 1.43 S in the 2026-10-24 run; arm B's rise was 1.66 times its pre-step level, against 1.72. - -**Smoke**, 2026-10-25 05:05 to 05:31 UTC, every phase at a tenth of its length, with the dry run at 300 seconds (`results-two-arm-warm-smoke/`). Both arms received every line: arm B held 102,734 documents; arm A held 102,708 non-blank lines, exact by the same check. No tier_down or overflow index was on hot, and each arm mounted a frozen index from warm. Two meters read the same clusters in the smoke (`timeline.jsonl.note`). - -**Rig additions.** `compose.warm.yaml`, `phases-two-arm-warm.sh` (the 2026-10-24 schedule with warm placement; containers and volumes removed before and after), `es-setup.mjs` and `es-setup-b.mjs` (`TD_PLACEMENT=warm` renders the recipe with the option and checks each bootstrap index sits on a warm node), `meter.py` (each shard's node from `_cat/shards`: per stream, store on hot and on warm; per arm, the warm node's store), `score-two-arm.py` (the warm lines beside the unchanged criteria), `run.sh clean`. - -Not shown: the cost of a warm tier, a measured Elastic Cloud bill, 300 nodes, compact, autoscaling, or a hold longer than the hot retention. - -## Grouping across containers on the E103 fix, arm A, 2026-10-26 - -The question: on the rig where the defect was found, does the E103 fix return any record holding lines from more than one container? The two-arm smoke's schedule (every phase at a tenth of its length, the dry run at 240 seconds) ran for arm A alone from 00:10 to 00:36 UTC. Results are in `results-e103-smoke/`. - -The engine is commit `0cd458f0f` (log-10x/engine, branch `fix/e103-sequence-key`) with modules commit `6c69d90a` (log-10x/modules, same branch). They ran in the image `local/edge-10x:e103` (sha256:8f74b8bbec2e…), which runs the engine jar on a JVM, so the Receiver had `TENX_JAVA_HEAP=320M` inside its rendered 512 MB limit. Everything else is the 2026-10-24 smoke: the same feed (sha256 24e02233a797…), log10x-mcp 1.30.84 (d6fa2b1), the same ILM policies and the same policy branch. - -**Result: no cross-container record, and delivery is exact.** - -| Count | Value | -|---|---:| -| Lines fed, by seq | 101,663 | -| Non-blank lines fed | 101,637 | -| Records returned | 79,415, plus the Receiver's connect message | -| Records holding lines from more than one container | 0 | -| Non-blank lines missing (sent, in no record) | 0 | -| Lines duplicated or unmatched (returned, no unused sent line) | 0 | -| Records with lines out of feed order | 0 | -| Elasticsearch documents (`tenx-app`, `tenx-tierdown`) | 79,416, with line counts summing to 101,637 | - -**How the count is made.** Fluent Bit keeps every returned record whole in this run. `score-e103.py` matches each record's lines to the lines sent and counts the records whose lines came from more than one container. The 2026-10-24 count (71 of 848,190, and 11 of 78,852 in that day's smoke) treated each record as the run of seqs up to the next record's seq. That holds only while lines are grouped in arrival order. Here it reads 0, with 2 records whose runs do not match their line counts. The scorer's self-test groups the run's feed with one head rule both ways. In arrival order it finds 4,333 cross-container records by lines and the same 4,333 by the run-based count. Per container it finds none by lines, while the run-based count reports 4,322. - -In this run, kafka's four startup lines (seq 2751 to 2754) came back as four kafka records, one line each. The jaeger record at seq 2750 holds only its own line. - -**Rig additions.** `ARM_A_ONLY=1` in `phases-two-arm.sh` runs arm A alone: no cluster B, no copy to it, and every returned record written whole to `data/returned.jsonl`. `gen-receiver-values.mjs` passes `TENX_JAVA_HEAP` to the Receiver. `score-e103.py` makes the count above. +[RECORD.md](RECORD.md) has every run, its criteria, per-stream tables, the cost projection and the rig. diff --git a/elastic-hot-tier-budget/RECORD.md b/elastic-hot-tier-budget/RECORD.md new file mode 100644 index 0000000..9c2a8d5 --- /dev/null +++ b/elastic-hot-tier-budget/RECORD.md @@ -0,0 +1,187 @@ +# Elastic hot-tier budget + +One benchmark in the [log10x benchmarks](../) repository. It asks whether log10x-mcp's hourly hot-tier run, driving the Receiver, holds an Elasticsearch hot tier under a size S while load grows 2.5 times, as the hot node's own stats count it. + +**Result: holds at steady state; under a 2.5x step it holds within its bound with the tier-down index placed on warm, and overshoots to 1.43 S with that index on hot.** Four runs. The first held the mean but not the bound; the second, with every time constant divided by 6, held hot at 0.97 to 0.98 of S under steady load and overshot to 1.23 S during a 2.5x step-up compressed into an hour. The first two fail the pre-registered bound on the hot store. Results of the first run in `results/`, of the second in `results-r2/` (section below). A third run put a native reroute by service beside the hold under one 2.5x step (`results-two-arm/`): the hold peaked at 1.43 S, over its bound, and the reroute tracked load, rising to 1.72 times its level before the step. A fourth wrote moved lines on warm (`results-two-arm-warm/`): the hold kept hot at 0.94 to 1.04 S through the step, within its bound, and the reroute rose to 1.25 S and kept rising. + +Self-managed Elasticsearch 9.5.5 (one hot node, one frozen node, filesystem snapshot repository, trial licence), engine `log10x/edge-10x:1.1.147`, log10x-mcp branch `feat/elastic-hot-tier-budget` at 6af0226, the published OpenTelemetry demo sample replayed, 2026-10-22 19:59 to 2026-10-23 09:59 UTC. `results/` holds the timeline, every hourly run's report and files, the cluster readings and slim captures (no log text). + +| | Criterion | Result | +|---|---|---| +| 1 | Every applied hourly run exits 0 and commits; once the size binds, its projection is at or under S | **Pass.** 13 applied runs, all exit 0, each projecting about 45 MB against S = 44.8 MB. | +| 2 | Run hours 7 to 13: hot store at or under 1.10 S at every reading, mean at or under S | **Fail.** Mean 0.998 S over 36 readings; 6 readings above 1.10 S, peak 1.17 S at 05:19 UTC, during the step-up. | +| 3 | Every rolled tier_down index mounted on frozen at 0 bytes | **Pass.** 38 of 38. | +| 4 | Hash queries on frozen equal the local count, ten largest moved statements | **Pass.** Exact, from 170,470 down to 10,498. | +| 5 | Every line received comes back, every returned record is in Elasticsearch | **Pass.** 1,041,196 lines in and back; 162,451 kept and 672,357 moved records, each equal to the cluster's count. | +| 6 | Storm: error lines kept per 4-minute window at most limit plus 50 | **Pass.** Scored on the regulator's own windows (every 240 seconds from its first regulated line): 55 a window (the 5-line sample plus 50) in each full window; the 146 of the first minute fall in two windows either side of 05:00:01. | + +## The hourly table + +S = 44.8 MB, set at run hour 1 as 0.7 of where the run's own dry run said hot would settle with nothing held (64 MB). Hot store is the hot node's `indices.store.size_in_bytes`; the other two columns are `_cat/indices` store for the indices not yet mounted. + +| UTC | Run hour | Load | Hot store (node stats) | of S | Kept stream on hot | Tier_down waiting on hot | +|---|---:|---|---:|---:|---:|---:| +| 19:59 | 0 | 2x | 0.0 MB | 0.00 | 0.0 MB | 0.0 MB | +| 20:59 | 1 | 2x | 11.0 MB | 0.25 | 10.9 MB | 0.0 MB | +| 21:59 | 2 | 2x | 22.4 MB | 0.50 | 14.5 MB | 7.6 MB | +| 22:59 | 3 | 2x | 24.0 MB | 0.54 | 17.2 MB | 6.4 MB | +| 23:59 | 4 | 2x | 29.0 MB | 0.65 | 23.2 MB | 5.2 MB | +| 00:59 | 5 | 2x | 37.1 MB | 0.83 | 31.2 MB | 5.1 MB | +| 01:59 | 6 | 2x | 39.6 MB | 0.88 | 34.2 MB | 4.8 MB | +| 02:59 | 7 | 2.75x | 33.8 MB | 0.76 | 28.1 MB | 4.9 MB | +| 03:59 | 8 | 3.5x | 39.7 MB | 0.89 | 31.6 MB | 7.2 MB | +| 04:59 | 9 | 4.25x | 45.4 MB | 1.01 | 29.8 MB | 14.2 MB | +| 05:59 | 10 | 5x | 50.6 MB | 1.13 | 30.9 MB | 18.4 MB | +| 06:59 | 11 | 5x | 50.1 MB | 1.12 | 26.9 MB | 22.1 MB | +| 07:59 | 12 | 5x | 47.2 MB | 1.05 | 25.1 MB | 20.7 MB | +| 08:59 | 13 | stopped | 44.8 MB | 1.00 | 22.4 MB | 20.8 MB | + +## Why the bound failed + +- **The always-pass floor.** The Receiver passes the first 5 lines of every statement in every window on every node, and 50 error lines over a limit. From run hour 9 the run allowed nothing more on hot, yet 1.8 to 3.2 MB an hour stayed there; 59 to 74% of those lines were the sample, across about 2,300 statements. +- **Moved lines waiting to mount.** A tier_down index rolls every 15 minutes here and mounts about 40 minutes later, so up to 22 MB of moved lines sat on hot during the growth, about half of S. Hot here keeps a kept line 6 hours; with retention of days the same hour of moved lines is about 1% of hot. + +## The rig + +`rig/`: two Elasticsearch nodes and Fluent Bit (`compose.yaml`), the Receiver rendered by log10x-mcp with `elasticHotTier` and a policy branch (`gen-receiver-values.mjs`), the MCP's frozen-tier recipe then the time-scaled policies (`es-setup.mjs`: kept lines 6 hours on hot, tier_down rollover 15 minutes), `policy.yaml` from `emitPolicyYaml` (`gen-policy.mjs`), `tenx-recur` every hour (`budget-run.sh`), the replay (`make-replay.py`, `forward_send.py`), the phases (`phases.sh`) and the meter (`meter.py`). The Receiver pulled its files from `log-10x/config` branch `test/elastic-hot-tier-policy`. Its byte limits count the UTF-8 size of the whole record the forwarder sends, not only the message, and the hourly run reads that size from the `utf8Size` field the Receiver stamps on each returned record. + +## Not shown + +A bill; Cloud Hosted; production retention lengths; 300 nodes; a step in load larger than 2.5x; a hold over days. + +## The rerun (`results-r2/`), 2026-10-23 11:25 to 15:55 UTC + +log10x-mcp `feat/elastic-hot-tier-budget` at 0b45d4a. Every time constant divided by 6: a run every 10 minutes, kept lines 2 hours on hot (10-minute rollover, frozen 110 minutes later), tier_down rollover 3 minutes, ILM poll 100 seconds (changed only because everything else was scaled), meter every 100 seconds, growth in four 20-minute steps from 56 to 140 lines a second. No raw capture: lines in are Fluent Bit's input counter, and the local copies of returned records keep only container, times, hash, route, `utf8Size`, level, name and line count (`results-r2/captures/`). S = 56.7 MB, 0.7 of where the dry run said hot would settle with nothing held (81 MB). + +| | Criterion | Result | +|---|---|---| +| 1 | Runs exit 0 and commit; projection at or under S, or the floor reported | **Pass.** 26 of 26 applied runs; projections 51 to 57 MB; the always-pass floor 79 to 268 MB a day, inside what fit at every run. | +| 2 | Run minutes 130 to 250: hot at or under 1.10 S every reading, mean at or under S | **Fail.** Mean 1.095 S, peak 1.23 S, 30 of 72 readings over 1.10 S. Before growth (minutes 111 to 131) 0.97 to 0.98 S. | +| 3 | Rolled tier_down indices mounted at 0 bytes | **Pass.** 73 of 73. | +| 4 | Hash queries on frozen equal local, ten largest moved statements | **Pass.** 10 of 10. | +| 5 | Lines in equal non-blank lines returned; returned records equal Elasticsearch's counts | **Pass.** Every returned record is in Elasticsearch. 1,301,174 lines in, 1,300,992 back; the 182 not returned are blank lines. The sample holds 26, all in its first 23 minutes of source time, and the replay passed over that stretch seven times (`rig/count-blank-sent.py` on `results/replay/source.jsonl` from `make-replay.py` and `results-r2/timeline.jsonl`). | +| 6 | Storm kept per window at most limit + 50 | **Pass.** On the regulator's own windows (rolling at 14:12:14 and 14:16:14): 121 and 165 kept, each within its limit plus 50 and the 5-line sample; the rest of the second window kept none, as its allowance was spent, and the storm ended before the next one. | + +**Why growth overshoots.** The run steers what new lines do; it cannot move lines already on hot. When load rose 2.5 times within an hour, the moved lines waiting to mount grew from 3 MB to 10 to 19 MB at once (the run's own estimate of that slice was right), while the kept lines already on hot leave only as their indices age out, over 2 hours here. The run tightened its allowance to 86 to 90% and projected 51 to 55 MB, but hot stayed above 1.1 S until the window closed. Relative to the time a line stays on hot, this step-up was about 50 times faster than a year of 2.5x growth against days of retention, but the run does not show what a slower ramp does. + +## Two-arm run, 2026-10-24 + +The question, from the method written before the run (`TWO_ARM_METHOD.md`, 2026-10-24): under a 2.5x step-up in load, does the Receiver's hourly hold keep the hot store at S where a native reroute by service does not? Run 2026-10-24 21:12 to 2026-10-25 01:26 UTC, results in `results-two-arm/`. + +**Result: the hold fails its bound and the reroute tracks load; delivery is exact in both arms.** Arm A peaked at 1.43 S in the scored window. Arm B rose to 1.72 times its level before the step. Within the hour of the hold, arm B's peak hot store (1.36 S) stayed below arm A's (1.43 S). At the end of the hold arm A was back at 1.02 S, while arm B was at 1.30 S and still rising. + +**Arm A, the hold**, is the `results-r2` rig with every time constant divided by 6. `tenx-recur` from log10x-mcp main at d6fa2b1 ran every 10 minutes against the engine `ghcr.io/log-10x/edge-10x:1.1.150` (sha256:3e755bbc14780a27e643e09b76dfd7ec9616cab67833f70daf9420eff646d350). Kept lines stay 2 hours on hot (10-minute rollover, frozen 110 minutes later). The tier_down index rolls every 10 minutes (the recipe's hour divided by 6) and goes frozen 1 minute after rollover. The ILM poll is 100 seconds. S = 63 MB, 0.7 of the 90 MB where the dry run at 0:10 said hot would settle with nothing held. + +**Arm B, the native reroute**, is a second Elasticsearch 9.5.5 cluster with the same image and node sizes (one hot node with a 1 GB heap, one frozen node) and its own snapshot repository. Fluent Bit copies every line before it reaches the Receiver and routes the copy with a rule fixed before the run. The services that made the top half of bytes in the feed's first 20 minutes, at the run's pace, go to `native-overflow`, which carries arm A's tier_down ILM policy copied from cluster A. Every other service goes to `native-app`, which carries arm A's kept policy. One service made the top half: `opentelemetry-collector`, with 68.9% of the bytes (`arm-b-rule.json`). Arm B has no Receiver. Its index templates are arm A's, with the stream names changed. + +Each arm has its own hot node, since the hot-tier run subtracts every index on the hot nodes that is not the Receiver's from S, so arm B's streams on arm A's node would take arm A's allowance. The method's "one Elasticsearch 9.5.5 node" is read as one hot node per arm. + +**Load.** The feed is the `results-r2` feed (`replay/source.jsonl`, sha256 24e02233a797…), sent by one sender for the whole run that numbers every line (`seq`). The start is 56 lines a second. S binds at the first meter reading at or above 0.95 S after the first applied run, or at 2:10 (R after that run), whichever comes first. Here it bound at 2:10, 23:22:19 UTC, with arm A at 0.93 S. Load stayed steady for 40 minutes, then took one step to 140 lines a second at 00:02:22, held for 60 minutes. The feed stopped at 01:02:28 after 1,070,359 lines, followed by a 20-minute drain. The run had no new service and no storm, since the method's load names neither. + +**Meter.** Every 100 seconds the meter read both clusters: the hot node's index store, each stream's store on hot and its frozen mounts (`_cat/indices`), and ILM explain. Each reading first refreshes both arms' indices on hot, since arm B's indices take no searches and a search-idle shard writes nothing to its store until a refresh. Without the refresh, a smoke reading showed 6,223 of arm B's 9,236 indexed documents. Each frozen node's cache is 128 MB (512 MB in `results-r2`), to stay inside the disk budget; the cache serves searches on frozen and nothing else. + +| | Criterion | Result | +|---|---|---| +| 1 | Arm A's hot store at or under 1.10 S at every reading from 20 minutes after the step to the end of the hold | **Fail.** 13 of 24 readings (00:23:55 to 01:02:15) over 1.10 S. Peak 1.43 S (90.0 MB at 00:57), mean 1.15 S. | +| 2 | Arm B's hot store above 1.5 times its pre-step level during the hold (pre-step level: the mean of its readings in the 20 minutes before the step); "no differentiator" if every reading from 20 minutes after the step stays within 1.10 times | **Pass.** Pre-step 50.0 MB. Above 1.5 times from 00:45:35, peak 85.9 MB (1.72 times) at 01:00. Above 1.10 times through the scored window, so the run shows a differentiator. | +| 3 | Every line in each arm on hot or frozen, by sequence number | **Pass.** Arm B: 1,070,359 documents, seq 1 to 1,070,359, none missing or repeated, each in the stream its service names. Arm A: all 1,070,203 non-blank lines are in the 848,190 returned records. Each record carries its first line's seq and holds exactly the non-blank lines from that seq to the next record's (`_lines`). The other 156 lines are blank. | + +The hot store per stream around the step, from the meter (`score.json`, `meter.jsonl.gz`): + +| UTC | Minutes from step | Arm A hot node | A kept (`tenx-app`) | A tier_down waiting | Arm B hot node | B normal (`native-app`) | B overflow (`native-overflow`) | +|---|---:|---:|---:|---:|---:|---:|---:| +| 20 minutes before, mean | | 59.9 MB (0.95 S) | 55.4 MB | 3.7 MB | 50.0 MB | 46.2 MB | 2.9 MB | +| 00:02 | 0 | 60.0 MB (0.95 S) | 57.1 MB | 2.0 MB | 51.0 MB | 46.7 MB | 3.4 MB | +| 00:23 | +22 | 78.9 MB (1.25 S) | 55.3 MB | 22.5 MB | 69.3 MB | 60.3 MB | 7.9 MB | +| 00:43 | +42 | 75.0 MB (1.19 S) | 50.5 MB | 23.4 MB | 73.5 MB | 66.0 MB | 6.3 MB | +| 00:57, arm A's peak | +55 | 90.0 MB (1.43 S) | 48.3 MB | 40.3 MB | 82.1 MB | 73.4 MB | 7.4 MB | +| 01:00, arm B's peak | +58 | 62.1 MB (0.99 S) | 49.3 MB | 11.5 MB | 85.9 MB | 76.3 MB | 8.4 MB | +| 01:02, end of hold | +60 | 64.2 MB (1.02 S) | 48.7 MB | 14.2 MB | 81.8 MB | 77.3 MB | 3.3 MB | + +**Why arm A overshoots.** The kept stream was held: 55.4 MB before the step and 46.0 to 57.7 MB during the hold. The overshoot comes from moved lines waiting to mount, which rose from 3.7 MB before the step to as much as 40.3 MB. Each tier_down index takes 10 minutes of moved lines and then waits about 7 minutes more for its mount (the run's own measure, 0.3 hours). At 2.5 times the load that is 20 to 25 MB. The 1.43 S peak is a single index, `tenx-tierdown-000019`. It read 15.3 MB at its last reading before rollover and 34.4 MB for the next three readings, until the searchable_snapshot action's force-merge step; after that it read 16.4 MB. + +**Why arm B rises.** Arm B's normal stream keeps every line of the services outside the top half for 2 hours, so it grows as lines at the new rate replace those at the old: 46.2 MB before the step and 77.3 MB at the end of the hold. The hold ended before those 2 hours had turned over. The overflow stream holds one 10-minute index plus its mount wait: 2.9 MB before the step and 2.1 to 9.7 MB during the hold. + +**Frozen.** Every rolled index in both arms mounted on frozen at 0 bytes local. Arm A mounted 20 tier_down indices and 11 kept indices; arm B mounted 20 overflow and 11 normal. The snapshot repositories hold 181 MB for arm A and 108 MB for arm B; arm A's documents carry the Receiver's fields. + +**Grouping across containers.** The Receiver groups multi-line events in the order lines arrive, so 71 of the 848,190 returned records hold lines from more than one container. For example, kafka's four startup lines at seq 2751 to 2754 joined the jaeger record at seq 2750, which is filed under jaeger. Every such line is still delivered. + +**Projection (Elastic's published arithmetic applied to the measured hot store per arm, not a measured bill).** Hot cost per TB of hot store per month = 1,000 GB / (disk-to-RAM ratio x 0.85 usable) GB of RAM x $ per GB RAM-hour x 730 hours x 2 zones. The rates come from Elastic's Cloud Hosted price table for AWS us-east-1 at Enterprise (https://cloud.elastic.co/deployment-pricing-table). The ratios come from Elastic's AWS default instance page (https://www.elastic.co/docs/reference/cloud/cloud-hosted/aws-default), which marks them as estimations. `aws.es.datahot.i8g` at $0.0793 and 27:1 gives $5,045 per TB-month; `aws.es.datahot.i3en` at $0.0590 and 74:1 gives $1,369. With S set at 1 TB of hot store, the measured ratios price as follows: + +| | Hot store | i8g, per month | i3en, per month | +|---|---:|---:|---:| +| Arm A, 20 minutes before the step | 0.95 S | $4,799 | $1,303 | +| Arm A, peak in the hold | 1.43 S | $7,207 | $1,956 | +| Arm A, end of the hold | 1.02 S | $5,144 | $1,396 | +| Arm B, 20 minutes before the step | 0.79 S | $4,008 | $1,088 | +| Arm B, peak in the hold | 1.36 S | $6,875 | $1,866 | +| Arm B, end of the hold | 1.30 S | $6,552 | $1,779 | + +What moves to frozen sits in the snapshot repository at $0.033 per GB-month. Elastic's node sizes step, so a difference in hot store saves money only when it crosses a node step. At full-size nodes a step is one node, about 1.5 TB of hot store on a 27:1 instance. A hot tier is sized for its peak, and in this hour arm A's peak was the higher of the two. + +**Smoke**, 2026-10-24 20:43 to 21:09 UTC, every phase at a tenth of its length (`results-two-arm-smoke/`). Both arms received every line: 101,241 fed. Arm B held 101,241 documents; arm A held 101,215 non-blank lines in 78,852 records, exact by the same check. Both meters read, and each arm mounted a frozen index. + +**Rig additions.** `compose.arm-b.yaml` (arm B's cluster), `es-setup-b.mjs` (arm A's ILM policies, poll and templates copied to arm B), `top-half.py` and `render-fb-two-arm.py` with `fluent-bit-two-arm/` (the fixed rule), `forward_send.py paced` and `run.sh feed` (one numbered sender, speed changed in place), `meter.py` with `ES_URL_B` (both clusters, per stream, refresh first), `dump-seq.py` and `score-two-arm.py` (delivery and the three criteria), `phases-two-arm.sh` (the schedule), and `record-build.py` (`build.json`: every image digest and the commits). + +Not shown: a measured Elastic Cloud bill, 300 nodes, compact, autoscaling, or a hold longer than the hot retention. + +## Two-arm run, warm placement, 2026-10-25 + +The question, from the method addendum written before the run (`TWO_ARM_METHOD_WARM.md`, 2026-10-25): with moved lines written to warm instead of hot, does arm A's hot store stay at S through the 2.5x step while arm B's tracks load? Run 2026-10-25 05:33 to 09:34 UTC, results in `results-two-arm-warm/`. + +**Result: the hold keeps hot at S through the step and the reroute tracks load; delivery is exact in both arms.** Arm A's hot store stayed between 0.94 and 1.04 S from 20 minutes after the step to the end of the hold (mean 1.00 S). Arm B rose to 1.66 times its level before the step and was still rising when the hold ended. No tier_down or overflow index was found on a hot node in any of 144 readings. + +**The one change.** Arm A's tier_down index and arm B's overflow stream are written on a warm node and mount on frozen from there. The tier_down template carries `index.routing.allocation.include._tier_preference: data_warm,data_hot`, rendered by log10x-mcp's recipe with `tierDownPlacement: 'warm'`; arm B's overflow template is copied from it. Each arm's frozen node also carries the `data_warm` role (`compose.warm.yaml`, 512 MB heap, 1200 MB limit, unchanged), since a third node per arm would set 9.5 GiB of container limits against the Docker VM's 7.65 GiB. Arm A's kept stream and arm B's normal stream stay on hot. Everything else is the 2026-10-24 run: the same feed (sha256 24e02233a797…), the same rule for arm B (`opentelemetry-collector`, 68.9% of bytes), the same ILM policies, rollover ages and 100-second poll, engine `ghcr.io/log-10x/edge-10x:1.1.150`. + +**The hourly run.** `tenx-recur` from log10x-mcp `feat/elastic-tierdown-warm` at c1b456a (`build.json`). It reads which node each tier_down index's shards sit on, counts only those on a `data_hot` or `data` node in the hot split, and counts moved lines in flight against S only while the tier_down write index sits on hot. Every one of the 24 runs reported the tier_down write index off hot, and all exited 0. + +**Load.** S = 63 MB, 0.7 of the 90 MB the dry run at 0:10 said hot would settle at with nothing held (the same S as the 2026-10-24 run). S bound at 07:29:53 UTC on the first rule, a reading at or above 0.95 S (60.6 MB). Steady for 40 minutes, then one step to 140 lines a second at 08:09:51, held 60 minutes. The feed stopped at 09:09:58 after 1,025,170 lines, followed by a 20-minute drain. + +| | Criterion | Result | +|---|---|---| +| 1 | Arm A's hot store at or under 1.10 S at every reading from 20 minutes after the step to the end of the hold | **Pass.** 25 readings (08:29:51 to 09:09:51), none over 1.10 S. Peak 1.04 S (65.7 MB at 08:49:51), mean 1.00 S. | +| 2 | Arm B's hot store above 1.5 times its pre-step level during the hold; "no differentiator" if every reading from 20 minutes after the step stays within 1.10 times | **Pass.** Pre-step 47.4 MB. Peak 78.7 MB (1.66 times) at the end of the hold, still rising. Above 1.10 times in the scored window, so the run shows a differentiator. | +| 3 | Every line in each arm on hot, warm or frozen, by sequence number | **Pass.** Arm B: 1,025,170 documents, seq 1 to 1,025,170, none missing or repeated, each in the stream its service names. Arm A: all 1,025,014 non-blank lines are in the 811,618 returned records, checked as in the 2026-10-24 run; the other 156 lines are blank. | + +Hot and warm per arm, from the meter (`score.json`, `meter.jsonl.gz`). Warm is reported, not scored. + +| UTC | Minutes from step | Arm A hot node | A kept (`tenx-app`) | A tier_down on warm | Arm B hot node | B normal (`native-app`) | B overflow on warm | +|---|---:|---:|---:|---:|---:|---:|---:| +| 20 minutes before, mean | | 61.6 MB (0.98 S) | 60.8 MB | 3.8 MB | 47.4 MB | 46.6 MB | 2.7 MB | +| 08:09 | 0 | 63.0 MB (1.00 S) | 62.2 MB | 6.9 MB | 47.2 MB | 46.5 MB | 3.6 MB | +| 08:29 | +20 | 62.2 MB (0.99 S) | 60.6 MB | 14.7 MB | 57.0 MB | 55.6 MB | 6.3 MB | +| 08:49, arm A's peak | +40 | 65.7 MB (1.04 S) | 64.7 MB | 10.4 MB | 69.7 MB | 68.3 MB | 4.6 MB | +| 09:09, end of hold and arm B's peak | +60 | 65.2 MB (1.03 S) | 64.0 MB | 7.9 MB | 78.7 MB | 77.6 MB | 3.7 MB | + +Warm peaked at 20.7 MB in arm A (08:34:51) and 9.1 MB in arm B (08:33:11). Each tier_down and overflow index was found mounted about 7 minutes after its rollover, as in the 2026-10-24 run; those minutes are spent on warm. Arm A mounted 19 tier_down and 10 kept indices on frozen; arm B 19 overflow and 10 normal. Arm A's peak in the hold was 1.04 S, against 1.43 S in the 2026-10-24 run; arm B's rise was 1.66 times its pre-step level, against 1.72. + +**Smoke**, 2026-10-25 05:05 to 05:31 UTC, every phase at a tenth of its length, with the dry run at 300 seconds (`results-two-arm-warm-smoke/`). Both arms received every line: arm B held 102,734 documents; arm A held 102,708 non-blank lines, exact by the same check. No tier_down or overflow index was on hot, and each arm mounted a frozen index from warm. Two meters read the same clusters in the smoke (`timeline.jsonl.note`). + +**Rig additions.** `compose.warm.yaml`, `phases-two-arm-warm.sh` (the 2026-10-24 schedule with warm placement; containers and volumes removed before and after), `es-setup.mjs` and `es-setup-b.mjs` (`TD_PLACEMENT=warm` renders the recipe with the option and checks each bootstrap index sits on a warm node), `meter.py` (each shard's node from `_cat/shards`: per stream, store on hot and on warm; per arm, the warm node's store), `score-two-arm.py` (the warm lines beside the unchanged criteria), `run.sh clean`. + +Not shown: the cost of a warm tier, a measured Elastic Cloud bill, 300 nodes, compact, autoscaling, or a hold longer than the hot retention. + +## Grouping across containers on the E103 fix, arm A, 2026-10-26 + +The question: on the rig where the defect was found, does the E103 fix return any record holding lines from more than one container? The two-arm smoke's schedule (every phase at a tenth of its length, the dry run at 240 seconds) ran for arm A alone from 00:10 to 00:36 UTC. Results are in `results-e103-smoke/`. + +The engine is commit `0cd458f0f` (log-10x/engine, branch `fix/e103-sequence-key`) with modules commit `6c69d90a` (log-10x/modules, same branch). They ran in the image `local/edge-10x:e103` (sha256:8f74b8bbec2e…), which runs the engine jar on a JVM, so the Receiver had `TENX_JAVA_HEAP=320M` inside its rendered 512 MB limit. Everything else is the 2026-10-24 smoke: the same feed (sha256 24e02233a797…), log10x-mcp 1.30.84 (d6fa2b1), the same ILM policies and the same policy branch. + +**Result: no cross-container record, and delivery is exact.** + +| Count | Value | +|---|---:| +| Lines fed, by seq | 101,663 | +| Non-blank lines fed | 101,637 | +| Records returned | 79,415, plus the Receiver's connect message | +| Records holding lines from more than one container | 0 | +| Non-blank lines missing (sent, in no record) | 0 | +| Lines duplicated or unmatched (returned, no unused sent line) | 0 | +| Records with lines out of feed order | 0 | +| Elasticsearch documents (`tenx-app`, `tenx-tierdown`) | 79,416, with line counts summing to 101,637 | + +**How the count is made.** Fluent Bit keeps every returned record whole in this run. `score-e103.py` matches each record's lines to the lines sent and counts the records whose lines came from more than one container. The 2026-10-24 count (71 of 848,190, and 11 of 78,852 in that day's smoke) treated each record as the run of seqs up to the next record's seq. That holds only while lines are grouped in arrival order. Here it reads 0, with 2 records whose runs do not match their line counts. The scorer's self-test groups the run's feed with one head rule both ways. In arrival order it finds 4,333 cross-container records by lines and the same 4,333 by the run-based count. Per container it finds none by lines, while the run-based count reports 4,322. + +In this run, kafka's four startup lines (seq 2751 to 2754) came back as four kafka records, one line each. The jaeger record at seq 2750 holds only its own line. + +**Rig additions.** `ARM_A_ONLY=1` in `phases-two-arm.sh` runs arm A alone: no cluster B, no copy to it, and every returned record written whole to `data/returned.jsonl`. `gen-receiver-values.mjs` passes `TENX_JAVA_HEAP` to the Receiver. `score-e103.py` makes the count above. diff --git a/elastic-hot-tier/README.md b/elastic-hot-tier/README.md index 1c6e7b4..9211b5f 100644 --- a/elastic-hot-tier/README.md +++ b/elastic-hot-tier/README.md @@ -1,149 +1,52 @@ # Elastic hot tier -One benchmark in the [log10x benchmarks](../) repository. It answers one question: with the -Receiver's per-statement byte budget in place and log10x-mcp's Elasticsearch frozen-tier recipe -(hourly rollover, and a frozen phase set to start one minute after it, which ILM acts on at its -next poll, every 10 minutes by default), how long do the lines it moves stay on the hot tier, -can they still be found by pattern hash once they are on frozen, and how much of the volume does -a fixed budget keep off hot as load grows, as the cluster's own APIs count it? +When lines over a per-statement budget go to an Elasticsearch index that rolls over hourly and moves to frozen, how long do they stay on hot? Each moved line left hot 30 to about 110 minutes after it was written, and all 159,536 moved records reached frozen. A fixed budget moved 59% to 73% of message bytes off hot, but the bytes sent to hot still grew with the load. -Every count of what Elasticsearch holds comes from the cluster, read every 10 minutes -(`_cat/indices`, `_cat/allocation`, `_ilm/explain`, and `_stats/store` hourly) into -[`results/meter.jsonl.gz`](results/meter.jsonl.gz). Counts of what the Receiver returned come -from Fluent Bit's local copies, slimmed to [`results/captures/`](results/captures/) (no log text). +The 10x engine gives each log statement a byte budget per 4-minute window. Fluent Bit sends lines over it to the `tenx-tierdown` alias, whose hourly indices are mounted on frozen as searchable snapshots at 0 bytes of local store. -## The result +## Result -Self-managed Elasticsearch 9.5.5 (one hot node, one frozen node, a filesystem snapshot -repository, trial licence), engine `log10x/edge-10x:1.1.147`, log10x-mcp 1.30.77, the published -OpenTelemetry demo sample replayed at 7 lines a second, 2026-10-21 23:14 to 2026-10-22 10:14 UTC. +Message bytes per hour, 2026-10-22 UTC, with a budget of 22,148 bytes per statement per 4-minute window. -| | Criterion | Result | -|---|---|---| -| 1 | Each tier-down index enters the frozen phase within rollover (1 hour) plus two poll intervals of its creation, at 0 bytes of local store | **Pass.** Indices 2 to 7: frozen phase 80.0 minutes after creation every time (rollover at 70.0, frozen phase 10.0 later), each mounted at 0 bytes. Index 1 sat empty from setup until the budget took hold and rolled over at the first poll after its first line. Index 8 received no line before the load stopped and was never rolled over, as ILM leaves an empty index. | -| 2 | From the capped phase on, the tier-down lines on hot never exceed the index being written plus the one mounting; the lines kept on hot follow the budget | **Pass.** At most 8.6 MB of tier-down data on hot at any reading, in at most two indices. In 4-minute windows counted from the capped start, no statement with lines moved kept more message bytes on hot than the budget, beyond the 5-line sample and the error keep count. | -| 3 | A term query on `tenx_hash` against the frozen indices returns what the local copy holds | **Pass.** The ten largest moved patterns: 49,785, 47,996, 17,700, 16,915, 11,172, 9,932, 3,285, 1,304, 631 and 516 documents on frozen, each equal to the local count. | -| 4 | Every line received is in a returned record, and every returned record is in Elasticsearch | **Pass.** 372,783 non-blank lines in and 372,783 back, in 301,980 records (the Receiver folds a multi-line event into one record): 142,444 kept and 142,444 documents in `tenx-app`; 159,536 moved and 159,536 documents across the tier-down indices. | -| 5 | Storm: error lines of the storming statement kept on hot per window at most its budget plus 50 | **Pass.** 109 to 113 a window against an allowance of 186, with 1,649 to 2,260 a window moved. | +| Hour | Phase | Bytes sent to hot | Bytes moved | Share moved | +|---|---|---:|---:|---:| +| 00:15 | baseline | 5,254,073 | 12,768 | 0% | +| 01:15 | budget on | 2,120,292 | 3,107,510 | 59% | +| 02:15 | budget on | 2,166,152 | 3,079,135 | 59% | +| 03:15 | budget on | 2,167,937 | 3,066,345 | 59% | +| 04:15 | budget on | 2,197,761 | 3,107,231 | 59% | +| 05:15 | budget on, 2.5x load | 4,589,240 | 8,626,759 | 65% | +| 06:15 | budget on, 2.5x load | 4,174,512 | 11,471,373 | 73% | +| 07:15 | budget on, 2.5x load | 4,050,664 | 9,484,012 | 70% | -**How long a moved line stays on hot.** ILM acts at its 10-minute poll, so it entered the frozen -phase 10 minutes after each rollover, but the `searchable_snapshot` action then waited one poll -each for shard history leases, the time-series end check and the force merge, so the original -index left hot and the mounted copy appeared 30 to 40 minutes after rollover (the meter reads -every 10 minutes). Each index lived 100 to 110 minutes on hot. A line written early in an -index's hour stayed up to about 110 minutes; one written just before rollover, 30 to 40. All of -it at ILM's default 10-minute poll (`indices.lifecycle.poll_interval`), which this run did not -change. +With the budget on, no statement kept more than its budget on hot in any window, beyond a 5-line sample and 50 extra error lines for the storming statement. At 2.5 times the load hot still took about 4.3 MB an hour, because the statements under budget grew too. -| Index | Created | Rolled over | Frozen phase | Last seen on hot | First seen mounted | -|---|---|---|---|---|---| -| 000002 | 01:25 | 02:35 | 02:45 | 03:05 | 03:15 | -| 000003 | 02:35 | 03:45 | 03:55 | 04:15 | 04:25 | -| 000004 | 03:45 | 04:55 | 05:05 | 05:25 | 05:35 | -| 000005 | 04:55 | 06:05 | 06:15 | 06:35 | 06:45 | -| 000006 | 06:05 | 07:15 | 07:25 | 07:45 | 07:55 | -| 000007 | 07:15 | 08:25 | 08:35 | 08:55 | 09:05 | +- Six of the eight hourly indices of moved lines spent 100 to 110 minutes on hot. +- All 372,783 non-blank lines sent reached Elasticsearch. Term queries on `tenx_hash` for the ten largest moved patterns matched the local counts on frozen. +- Of the moved bytes, 86% came from three OpenTelemetry Collector statements that no rule named. -**How much a fixed budget keeps off hot.** One budget for every statement, set before the -capped phase where half of the second baseline hour's message bytes were over it, moved 59% of -the message bytes at the starting load and 65% to 73% an hour at 2.5 times it. The budget is per -statement, so growth spread across statements under it still reaches hot: the message bytes kept -on hot went from 5.25 MB in the last baseline hour to about 2.17 MB an hour capped and about -4.3 MB an hour at 2.5 times the load. Holding the hot tier at a size as load grows needs the budget -kept current against the tier; this run held it fixed. +## How it was measured -## Whose statements moved +Elasticsearch 9.5.5 ran self-managed with one hot node, one frozen node and a trial license. The load was the published OpenTelemetry demo sample at 7 lines a second, then 2.5 times that for 3 hours plus a new service and a 20-minute error storm. The budget was set from the second baseline hour so that half of its message bytes were over it. The table counts bytes in Fluent Bit's local copy of what the engine returned. Index sizes and ILM state came from the cluster's own APIs every 10 minutes. -The budget moved 159,536 records, 42 MB of message text, from 15 statements. 86% of those bytes came from three statements of the OpenTelemetry Collector, the debug exporter's `Traces`, `Logs` and `Metrics` lines, and 5% from one Grafana statement (the storm line): code the account runs but does not write. No regular expression, fingerprint rule or per-service list was written for any of them; each line carried the engine's pattern hash from the Receiver, the same on every node running the same engine, library and configuration, and the hash queries in criterion 3 find them on frozen by that hash. The injected service's one new statement (`inventory`, 4%) was held from its first window with no entry of its own. +## Limits -| Container | Records moved | Message MB | Share of bytes | Statements | -|---|---:|---:|---:|---:| -| opentelemetry-collector | 114,736 | 36.23 | 86.3% | 5 | -| grafana | 11,172 | 1.96 | 4.7% | 1 | -| inventory (injected) | 17,700 | 1.69 | 4.0% | 1 | -| cart | 13,848 | 1.55 | 3.7% | 3 | -| recommendation | 1,304 | 0.40 | 1.0% | 1 | -| ad | 768 | 0.12 | 0.3% | 2 | -| kafka | 8 | 0.00 | 0.0% | 2 | +- Hot-node bytes say nothing about a bill, since Elastic Cloud Hosted bills the configured size. +- The budget stayed fixed, so the run does not show hot kept at a set size under growth. +- No comparison was made with Elastic's own tools or logsdb. +- The budget aimed at 50% of the baseline message bytes and moved 59%. The storm statement's kept count fits the engine sizing the whole forwarded record, which the run leaves open. -From [`results/captures/tier_down.tsv.gz`](results/captures/tier_down.tsv.gz), by container, `tenx_hash` and `message_pattern`. +## Run it -## The hourly table +``` +cd elastic-hot-tier/rig +python3 make-replay.py +MCP= python3 detach.py ../results/phases.log ./phases.sh +# after the run: bring the two nodes back on the same volumes and score +RIG_DIR=$PWD RIG_DATA=$PWD/../results/data docker compose -p elastic-rig -f compose.yaml up -d es-hot es-frozen +python3 analyze.py +``` -Hours from the capped start. Message bytes are the UTF-8 size of each line's text in the -records the Receiver returned (local). `tenx-app` growth and the tier-down columns are the -cluster's `store.size` (cluster), which also moves with segment merges, so an hour's growth is -not exactly its documents' size. +A rerun takes about 11 hours and needs Docker with about 5 GB free memory, Node 20, Python 3, a built log10x-mcp checkout and otel-sample-200mb.log from release otel-sample-v1. This run used engine 1.1.147 and log10x-mcp 1.30.77. -| Hour (UTC) | Phase | Message bytes to hot | Message bytes moved | Moved | tenx-app growth on hot | Tier-down on hot at hour end | Tier-down docs on frozen at hour end | -|---|---|---:|---:|---:|---:|---:|---:| -| 00:15 | baseline | 5,254,073 | 12,768 | 0% | 4,791,108 | 249 | 0 | -| 01:15 | capped | 2,120,292 | 3,107,510 | 59% | 3,053,345 | 1,806,542 | 1,731 | -| 02:15 | capped | 2,166,152 | 3,079,135 | 59% | 2,916,215 | 2,971,517 | 1,731 | -| 03:15 | capped | 2,167,937 | 3,066,345 | 59% | 3,099,663 | 4,072,216 | 13,584 | -| 04:15 | capped | 2,197,761 | 3,107,231 | 59% | 4,282,683 | 2,934,268 | 25,238 | -| 05:15 | growth | 4,589,240 | 8,626,759 | 65% | 6,695,066 | 6,379,264 | 37,082 | -| 06:15 | growth | 4,174,512 | 11,471,373 | 73% | 6,574,067 | 8,612,903 | 68,850 | -| 07:15 | growth | 4,050,664 | 9,484,012 | 70% | 6,279,659 | 6,850,134 | 122,388 | -| 08:15 | drain | 0 | 0 | 0% | 1,349,761 | 249 | 159,536 | -| 09:15 | drain | 0 | 0 | 0% | 0 | 249 | 159,536 | - -## How the budget was set - -Written before the run in the method and not changed during it. `rig/cap.py` read the records the -Receiver returned between 00:14 and 01:14 (the second baseline hour), grouped them by container, -pattern and 4-minute window, and found the byte limit at which half of those bytes were over it: -**22,148 bytes per pattern per window per node** ([`results/cap.out`](results/cap.out)). The -Receiver was then relaunched with the values log10x-mcp renders for that fleet default: -`rateReceiverAbsoluteCap 22148`, `rateReceiverDefaultAction tier_down`, `outputOffload=true` -([`results/values/launch.json`](results/values/launch.json); the baseline's, with no budget, in [`results/values-baseline/`](results/values-baseline/)). - -The budget held more than half: 59% of message bytes moved. The regulator sizes a line as the -UTF-8 size of the event's text, and the storm statement's kept count (about 112 a window, a -162-byte message) fits that text being the whole forwarded record rather than the message alone. -`cap.py` counted the message. This is noted, not resolved, here. - -## The rig - -Everything under [`rig/`](rig/). `compose.yaml` (two Elasticsearch nodes and Fluent Bit) plus the -Receiver service that `gen-receiver-values.mjs` writes from log10x-mcp's own renderers; -`es-setup.mjs` starts the trial licence, registers the snapshot repository and sends every request -in `elasticFrozenTierRecipe` verbatim; `fluent-bit/fluent-bit.conf` routes `routeState: tier_down` -to the `tenx-tierdown` alias and everything else to `tenx-app`, as the recipe tells the operator to; -`forward_send.py` replays the sample; `phases.sh` is the run; `meter.py` reads the cluster; -`analyze.py` scores it. - -Where the rig differs from an install, and why: - -- **No Kubernetes.** The rendered values read the policy files from a ConfigMap. The rig mounts - header-only cap, action and total files instead, so the fleet budget and default action decide - every line, and leaves out the opt-in config-generation file. -- **Measurement fields.** The rig adds `symbolMessageHashField tenx_hash` (log10x-mcp writes it - for Datadog only; the Elastic recipe maps the field) and an `outputEncodeField` that keeps - `routeState`, `message_pattern` and `severity_level` on the returned record. -- **The split.** log10x-mcp renders one Elasticsearch output; the recipe's note tells the operator - to send tier_down lines to `tenx-tierdown`. The rig does that with `rewrite_tag`. -- **Load.** The published OpenTelemetry demo sample (`otel-sample-200mb.log`, release - `otel-sample-v1`), not a live demo: 7 lines a second for 6 hours, then 17.5 lines a second for - 3 hours, plus one injected service (2 lines a second) and a 20-minute storm of one real Grafana - `level=error` line at 10 lines a second. - -## What this run does not show - -- Anything about a bill. Elastic Cloud Hosted bills the configured size; this run shows bytes on - a self-managed cluster's hot node. -- A hot tier held at a size under growth. The budget was fixed for the run. -- A comparison with a build on Elastic's own tools, or with logsdb. - -## Reproduce - -Docker with about 5 GB of memory free, Node 20, Python 3, and a log10x-mcp checkout built with -`npm run build`. - - cd elastic-hot-tier/rig - # the replay feed and the storm line, from the published sample - python3 make-replay.py - MCP= python3 detach.py ../results/phases.log ./phases.sh - # after the run: bring the two nodes back on the same volumes and score - RIG_DIR=$PWD RIG_DATA=$PWD/../results/data docker compose -p elastic-rig -f compose.yaml up -d es-hot es-frozen - python3 analyze.py +[RECORD.md](RECORD.md) has every check, the cluster's hourly readings and the rig. diff --git a/elastic-hot-tier/RECORD.md b/elastic-hot-tier/RECORD.md new file mode 100644 index 0000000..1c6e7b4 --- /dev/null +++ b/elastic-hot-tier/RECORD.md @@ -0,0 +1,149 @@ +# Elastic hot tier + +One benchmark in the [log10x benchmarks](../) repository. It answers one question: with the +Receiver's per-statement byte budget in place and log10x-mcp's Elasticsearch frozen-tier recipe +(hourly rollover, and a frozen phase set to start one minute after it, which ILM acts on at its +next poll, every 10 minutes by default), how long do the lines it moves stay on the hot tier, +can they still be found by pattern hash once they are on frozen, and how much of the volume does +a fixed budget keep off hot as load grows, as the cluster's own APIs count it? + +Every count of what Elasticsearch holds comes from the cluster, read every 10 minutes +(`_cat/indices`, `_cat/allocation`, `_ilm/explain`, and `_stats/store` hourly) into +[`results/meter.jsonl.gz`](results/meter.jsonl.gz). Counts of what the Receiver returned come +from Fluent Bit's local copies, slimmed to [`results/captures/`](results/captures/) (no log text). + +## The result + +Self-managed Elasticsearch 9.5.5 (one hot node, one frozen node, a filesystem snapshot +repository, trial licence), engine `log10x/edge-10x:1.1.147`, log10x-mcp 1.30.77, the published +OpenTelemetry demo sample replayed at 7 lines a second, 2026-10-21 23:14 to 2026-10-22 10:14 UTC. + +| | Criterion | Result | +|---|---|---| +| 1 | Each tier-down index enters the frozen phase within rollover (1 hour) plus two poll intervals of its creation, at 0 bytes of local store | **Pass.** Indices 2 to 7: frozen phase 80.0 minutes after creation every time (rollover at 70.0, frozen phase 10.0 later), each mounted at 0 bytes. Index 1 sat empty from setup until the budget took hold and rolled over at the first poll after its first line. Index 8 received no line before the load stopped and was never rolled over, as ILM leaves an empty index. | +| 2 | From the capped phase on, the tier-down lines on hot never exceed the index being written plus the one mounting; the lines kept on hot follow the budget | **Pass.** At most 8.6 MB of tier-down data on hot at any reading, in at most two indices. In 4-minute windows counted from the capped start, no statement with lines moved kept more message bytes on hot than the budget, beyond the 5-line sample and the error keep count. | +| 3 | A term query on `tenx_hash` against the frozen indices returns what the local copy holds | **Pass.** The ten largest moved patterns: 49,785, 47,996, 17,700, 16,915, 11,172, 9,932, 3,285, 1,304, 631 and 516 documents on frozen, each equal to the local count. | +| 4 | Every line received is in a returned record, and every returned record is in Elasticsearch | **Pass.** 372,783 non-blank lines in and 372,783 back, in 301,980 records (the Receiver folds a multi-line event into one record): 142,444 kept and 142,444 documents in `tenx-app`; 159,536 moved and 159,536 documents across the tier-down indices. | +| 5 | Storm: error lines of the storming statement kept on hot per window at most its budget plus 50 | **Pass.** 109 to 113 a window against an allowance of 186, with 1,649 to 2,260 a window moved. | + +**How long a moved line stays on hot.** ILM acts at its 10-minute poll, so it entered the frozen +phase 10 minutes after each rollover, but the `searchable_snapshot` action then waited one poll +each for shard history leases, the time-series end check and the force merge, so the original +index left hot and the mounted copy appeared 30 to 40 minutes after rollover (the meter reads +every 10 minutes). Each index lived 100 to 110 minutes on hot. A line written early in an +index's hour stayed up to about 110 minutes; one written just before rollover, 30 to 40. All of +it at ILM's default 10-minute poll (`indices.lifecycle.poll_interval`), which this run did not +change. + +| Index | Created | Rolled over | Frozen phase | Last seen on hot | First seen mounted | +|---|---|---|---|---|---| +| 000002 | 01:25 | 02:35 | 02:45 | 03:05 | 03:15 | +| 000003 | 02:35 | 03:45 | 03:55 | 04:15 | 04:25 | +| 000004 | 03:45 | 04:55 | 05:05 | 05:25 | 05:35 | +| 000005 | 04:55 | 06:05 | 06:15 | 06:35 | 06:45 | +| 000006 | 06:05 | 07:15 | 07:25 | 07:45 | 07:55 | +| 000007 | 07:15 | 08:25 | 08:35 | 08:55 | 09:05 | + +**How much a fixed budget keeps off hot.** One budget for every statement, set before the +capped phase where half of the second baseline hour's message bytes were over it, moved 59% of +the message bytes at the starting load and 65% to 73% an hour at 2.5 times it. The budget is per +statement, so growth spread across statements under it still reaches hot: the message bytes kept +on hot went from 5.25 MB in the last baseline hour to about 2.17 MB an hour capped and about +4.3 MB an hour at 2.5 times the load. Holding the hot tier at a size as load grows needs the budget +kept current against the tier; this run held it fixed. + +## Whose statements moved + +The budget moved 159,536 records, 42 MB of message text, from 15 statements. 86% of those bytes came from three statements of the OpenTelemetry Collector, the debug exporter's `Traces`, `Logs` and `Metrics` lines, and 5% from one Grafana statement (the storm line): code the account runs but does not write. No regular expression, fingerprint rule or per-service list was written for any of them; each line carried the engine's pattern hash from the Receiver, the same on every node running the same engine, library and configuration, and the hash queries in criterion 3 find them on frozen by that hash. The injected service's one new statement (`inventory`, 4%) was held from its first window with no entry of its own. + +| Container | Records moved | Message MB | Share of bytes | Statements | +|---|---:|---:|---:|---:| +| opentelemetry-collector | 114,736 | 36.23 | 86.3% | 5 | +| grafana | 11,172 | 1.96 | 4.7% | 1 | +| inventory (injected) | 17,700 | 1.69 | 4.0% | 1 | +| cart | 13,848 | 1.55 | 3.7% | 3 | +| recommendation | 1,304 | 0.40 | 1.0% | 1 | +| ad | 768 | 0.12 | 0.3% | 2 | +| kafka | 8 | 0.00 | 0.0% | 2 | + +From [`results/captures/tier_down.tsv.gz`](results/captures/tier_down.tsv.gz), by container, `tenx_hash` and `message_pattern`. + +## The hourly table + +Hours from the capped start. Message bytes are the UTF-8 size of each line's text in the +records the Receiver returned (local). `tenx-app` growth and the tier-down columns are the +cluster's `store.size` (cluster), which also moves with segment merges, so an hour's growth is +not exactly its documents' size. + +| Hour (UTC) | Phase | Message bytes to hot | Message bytes moved | Moved | tenx-app growth on hot | Tier-down on hot at hour end | Tier-down docs on frozen at hour end | +|---|---|---:|---:|---:|---:|---:|---:| +| 00:15 | baseline | 5,254,073 | 12,768 | 0% | 4,791,108 | 249 | 0 | +| 01:15 | capped | 2,120,292 | 3,107,510 | 59% | 3,053,345 | 1,806,542 | 1,731 | +| 02:15 | capped | 2,166,152 | 3,079,135 | 59% | 2,916,215 | 2,971,517 | 1,731 | +| 03:15 | capped | 2,167,937 | 3,066,345 | 59% | 3,099,663 | 4,072,216 | 13,584 | +| 04:15 | capped | 2,197,761 | 3,107,231 | 59% | 4,282,683 | 2,934,268 | 25,238 | +| 05:15 | growth | 4,589,240 | 8,626,759 | 65% | 6,695,066 | 6,379,264 | 37,082 | +| 06:15 | growth | 4,174,512 | 11,471,373 | 73% | 6,574,067 | 8,612,903 | 68,850 | +| 07:15 | growth | 4,050,664 | 9,484,012 | 70% | 6,279,659 | 6,850,134 | 122,388 | +| 08:15 | drain | 0 | 0 | 0% | 1,349,761 | 249 | 159,536 | +| 09:15 | drain | 0 | 0 | 0% | 0 | 249 | 159,536 | + +## How the budget was set + +Written before the run in the method and not changed during it. `rig/cap.py` read the records the +Receiver returned between 00:14 and 01:14 (the second baseline hour), grouped them by container, +pattern and 4-minute window, and found the byte limit at which half of those bytes were over it: +**22,148 bytes per pattern per window per node** ([`results/cap.out`](results/cap.out)). The +Receiver was then relaunched with the values log10x-mcp renders for that fleet default: +`rateReceiverAbsoluteCap 22148`, `rateReceiverDefaultAction tier_down`, `outputOffload=true` +([`results/values/launch.json`](results/values/launch.json); the baseline's, with no budget, in [`results/values-baseline/`](results/values-baseline/)). + +The budget held more than half: 59% of message bytes moved. The regulator sizes a line as the +UTF-8 size of the event's text, and the storm statement's kept count (about 112 a window, a +162-byte message) fits that text being the whole forwarded record rather than the message alone. +`cap.py` counted the message. This is noted, not resolved, here. + +## The rig + +Everything under [`rig/`](rig/). `compose.yaml` (two Elasticsearch nodes and Fluent Bit) plus the +Receiver service that `gen-receiver-values.mjs` writes from log10x-mcp's own renderers; +`es-setup.mjs` starts the trial licence, registers the snapshot repository and sends every request +in `elasticFrozenTierRecipe` verbatim; `fluent-bit/fluent-bit.conf` routes `routeState: tier_down` +to the `tenx-tierdown` alias and everything else to `tenx-app`, as the recipe tells the operator to; +`forward_send.py` replays the sample; `phases.sh` is the run; `meter.py` reads the cluster; +`analyze.py` scores it. + +Where the rig differs from an install, and why: + +- **No Kubernetes.** The rendered values read the policy files from a ConfigMap. The rig mounts + header-only cap, action and total files instead, so the fleet budget and default action decide + every line, and leaves out the opt-in config-generation file. +- **Measurement fields.** The rig adds `symbolMessageHashField tenx_hash` (log10x-mcp writes it + for Datadog only; the Elastic recipe maps the field) and an `outputEncodeField` that keeps + `routeState`, `message_pattern` and `severity_level` on the returned record. +- **The split.** log10x-mcp renders one Elasticsearch output; the recipe's note tells the operator + to send tier_down lines to `tenx-tierdown`. The rig does that with `rewrite_tag`. +- **Load.** The published OpenTelemetry demo sample (`otel-sample-200mb.log`, release + `otel-sample-v1`), not a live demo: 7 lines a second for 6 hours, then 17.5 lines a second for + 3 hours, plus one injected service (2 lines a second) and a 20-minute storm of one real Grafana + `level=error` line at 10 lines a second. + +## What this run does not show + +- Anything about a bill. Elastic Cloud Hosted bills the configured size; this run shows bytes on + a self-managed cluster's hot node. +- A hot tier held at a size under growth. The budget was fixed for the run. +- A comparison with a build on Elastic's own tools, or with logsdb. + +## Reproduce + +Docker with about 5 GB of memory free, Node 20, Python 3, and a log10x-mcp checkout built with +`npm run build`. + + cd elastic-hot-tier/rig + # the replay feed and the storm line, from the published sample + python3 make-replay.py + MCP= python3 detach.py ../results/phases.log ./phases.sh + # after the run: bring the two nodes back on the same volumes and score + RIG_DIR=$PWD RIG_DATA=$PWD/../results/data docker compose -p elastic-rig -f compose.yaml up -d es-hot es-frozen + python3 analyze.py diff --git a/elastic-store-four-ways/README.md b/elastic-store-four-ways/README.md index 67006f4..766d031 100644 --- a/elastic-store-four-ways/README.md +++ b/elastic-store-four-ways/README.md @@ -1,111 +1,47 @@ # Elasticsearch store, four ways -One benchmark in the [log10x benchmarks](../) repository. It answers one question: on -Elasticsearch 9.5.5, how much index store does the engine's compact form save over a raw index, -over logsdb index mode, and over logsdb with `message` mapped as `pattern_text`, on the same -sample? The compact form is indexed in standard mode, in logsdb mode, and in logsdb mode sorted -on its template id; the comparison that decides is compact on logsdb against logsdb with -`pattern_text`. - -Two things to know before reading the table. The raw index holds the parsed record: one field per -key of the shipped JSON, the message in `message`. And document counts differ by design: the three native indexes hold one -document per line (197,430), while the engine folds a multi-line event into one, so the compact -indexes hold one per event (157,228) plus one per template (3,030). Each index is checked -against its own input. - -## The result - -Elasticsearch 9.5.5 (Lucene 10.5.1) on a trial licence, engine -`ghcr.io/log-10x/edge-10x:1.1.150`, the published OpenTelemetry demo sample (215,039,161 B, -197,430 lines). Primary store after a force merge to one segment, `best_compression` on every -index, two repetitions: - -| Variant | Index mode | `message` | Store, rep 1 | Store, rep 2 | Mean | Of (a) | Of (b) | -|---|---|---|---:|---:|---:|---:|---:| -| (a) raw | standard | `text` | 20,108,998 B | 20,106,803 B | 20,107,900 B | 100.0% | 167.6% | -| (b) logsdb | logsdb | `text` | 12,003,158 B | 11,997,590 B | 12,000,374 B | 59.7% | 100.0% | -| (c) logsdb + `pattern_text` | logsdb, sorted on `message.template_id` | `pattern_text` | 6,977,160 B | 6,972,360 B | 6,974,760 B | 34.7% | 58.1% | -| (d) compact | standard | encoded event | 14,909,550 B | 14,901,237 B | 14,905,394 B | 74.1% | 124.2% | -| (e) compact on logsdb | logsdb | encoded event | 10,721,205 B | 10,718,972 B | 10,720,088 B | 53.3% | 89.3% | -| (f) compact on logsdb, sorted | logsdb, sorted on `l1x_tid` | encoded event | 8,875,307 B | 8,872,791 B | 8,874,049 B | 44.1% | 73.9% | - -The compact variants' store includes the template dictionary index (537,262 B). - -**Logsdb with `pattern_text`, sorted on its template id, takes less store than the compact form -in every mode.** Compact on logsdb (e) is 153.7% of (c), 53.7% larger; sorted the same -way on its own template id (f), 127.2% of (c), 27.2% larger. The native pair alone holds -this sample in 34.7% of the raw index's store, 65.3% less. Over (a) and (b) the compact form still saves: (e) is 53.3% of (a) and -89.3% of (b). - -Where the difference sits, from `_disk_usage` (repetition 1): the stored compact text in (f), -`message._original`, is about the size of what `pattern_text` stores for (c), -`message.args` plus `message.template`. What the compact form adds is the inverted index over the -encoded `message`, which indexes every variable value as a term, the container, pod and image -ids folded into each event included. In (c) those values sit in `keyword` doc values instead. - -### Amendment, 2026-10-25 - -The sort setting alone did not reach Lucene: with `index.logsdb.default_sort_on_message_template: -true` and documents that carry no `host.name`, the segments of (c) carried a sort on `@timestamp` -only, and (c) measured 10,649,904 B, 52.9% of (a). That run is in -[`results-setting-only/`](results-setting-only/), where the sort check fails. (c) sets -`index.sort.field` explicitly, as Elastic's `pattern_text` page shows in its "Custom sorting" -example, and (f) gives the compact form the same sort on its own template id. `METHOD.md` records -the amendment at its end. - -### The licence - -`pattern_text` is a subscription feature, and the basic licence accepts it with templating off. -On the default basic licence, before the trial, creating a logsdb index with `message` as -`pattern_text` returned 200 and a document indexed (201), and the mapping came back as `{"type": "pattern_text", "disable_templating": true}` and the source mode as -`STORED`: templating off and no synthetic source. On the trial, (c)'s mapping is plain -`{"type": "pattern_text"}`. Both are in `results/licence.json`. - -### The compact form - -The two encodes of the sample are byte-identical: 157,228 events and -3,030 templates, 74,140,583 B encoded plus 3,281,830 B dictionary, -64.0% under the sample before any codec. The dictionary followed by the events decodes to the -sample byte for byte (decoded output sha256 `aa79b9349a4123d88936fd064f5eccd9fd84f45a1fd03cff3f64c259715c2432`). Hashes of both -encodes are in `results/compact.json`. - -## Method - -[`METHOD.md`](METHOD.md), written before the run, with its amendment: the sample, how the compact -form is made, the index definitions, the measurement, and the checks. All 66 checks pass in -`results/results.md`. - -## Reproduce +On Elasticsearch 9.5.5, does the 10x engine's compact form take less index store than Elasticsearch's own log features? Against plain logsdb it does, but against logsdb with the message field mapped as `pattern_text` it does not. That native setup, sorted on its template id, took the least store of all. Compact on logsdb took 53.7% more, and 27.2% more with the same sort on its own template id. + +Per-field disk usage shows why. The stored text in the sorted compact index is about the size of what `pattern_text` stores. The extra is an inverted index over the encoded message, with every variable value as a term, including the container, pod and image ids in each event. With `pattern_text` those values sit in keyword doc values. + +## Result + +Primary store after a force merge to one segment, as the mean of two repetitions: + +| Variant | Index mode | Store | Of raw | Of logsdb | +|---|---|---:|---:|---:| +| Raw | standard | 20,107,900 B | 100.0% | 167.6% | +| Logsdb | logsdb | 12,000,374 B | 59.7% | 100.0% | +| Logsdb + pattern_text | logsdb, sorted on template id | 6,974,760 B | 34.7% | 58.1% | +| Compact | standard | 14,905,394 B | 74.1% | 124.2% | +| Compact on logsdb | logsdb | 10,720,088 B | 53.3% | 89.3% | +| Compact on logsdb, sorted | logsdb, sorted on template id | 8,874,049 B | 44.1% | 73.9% | + +Compact on logsdb is smaller than raw and plain logsdb, and every compact variant is larger than `pattern_text`. The compact figures include the template dictionary index (537,262 B). + +- The template sort took effect only when set with `index.sort.field`. With `index.logsdb.default_sort_on_message_template` alone, on documents with no host.name field, the segments were sorted on @timestamp only and the `pattern_text` index took 10,649,904 B ([results-setting-only/](results-setting-only/)). + +## How it was measured + +The sample is the published OpenTelemetry demo capture, 215,039,161 B in 197,430 JSON lines as Fluentd ships them. The native indexes store the parsed record, one document per line. The compact indexes store one document per event (157,228, with multi-line events folded into one) plus 3,030 templates. The compact form decoded back to the sample byte for byte. + +Everything ran on one Elasticsearch 9.5.5 node in Docker on a trial license, with one shard and best_compression per index. The repetitions differed by at most 0.07%, and all 66 checks in [results/results.md](results/results.md) pass. + +## Limits + +- The test measures store on one sample and one node, and not query cost, ingest cost or Elastic Cloud pricing. +- Search-time decoding was not tested, because the decoding plugin is built for Elasticsearch 8.17.0, not 9.x. +- Every compact variant carries the metadata inside the encoded event. Keeping the metadata as separate fields was not measured. +- `pattern_text` is a subscription feature and was measured on a trial license. The basic license accepted it with templating off, and that case was not measured. + +## Run it ``` -./run.sh # encode twice, decode check, one Elasticsearch node, six variants twice, report -./run.sh report # re-render results/results.md from results/ +cd elastic-store-four-ways +./run.sh report # re-render results/results.md from the committed results/ +./run.sh # full run: encode, decode check, six variants twice, report ``` -Needs Docker, Python 3 and `curl`; about 2 GB of memory for Elasticsearch and 3 GB of free disk. -The sample is fetched from the `otel-sample-v1` release of `log-10x/config` into `data/` unless -`ES4_SAMPLE` names a local copy; either way its sha256 is checked. The engine and Elasticsearch -images are pinned by digest. Every container is started detached under a fixed name and removed -with `docker rm -f -v`, on exit too. The whole run takes about 10 minutes. - -## What is claimed and what is not - -- Primary store size after a force merge to one segment, per variant, on this sample, with these - images and a trial licence. -- Not query or ingest cost, not search-time decoding with the l1es plugin, not - Elastic Cloud pricing, not a compact form of the message text alone with the metadata kept as - fields. `METHOD.md` lists the rest. - -## Files - -| File | What it is | -|---|---| -| `METHOD.md` | the plan, written before the run, and its amendment | -| `run.sh` | the whole run | -| `bench.py` | the Elasticsearch side: licence, index definitions, loading, measurement, report | -| `results/compact.json` | the compact form: counts and sha256 of both encodes, whether they match, the decoded output's sha256 and the byte comparison with the sample | -| `results/licence.json` | the version, the response to creating a `pattern_text` index on the default licence, the trial | -| `results/rep.json` | each repetition: per index store, documents, segments and their sort, settings, leases, per-field bytes | -| `results/rep/` | each index's mapping and `_disk_usage` output | -| `results/results.json`, `results/results.md` | the table, the ratios and the checks | -| `results-setting-only/` | the run before the amendment, (c) with `default_sort_on_message_template` alone | +The report needs only Python 3. A full run also needs Docker, curl, about 2 GB of memory and 3 GB of free disk, and takes about 10 minutes. + +[METHOD.md](METHOD.md) is the plan written before the run, and [RECORD.md](RECORD.md) is the full record with per-field bytes, the license probe and hashes. diff --git a/elastic-store-four-ways/RECORD.md b/elastic-store-four-ways/RECORD.md new file mode 100644 index 0000000..67006f4 --- /dev/null +++ b/elastic-store-four-ways/RECORD.md @@ -0,0 +1,111 @@ +# Elasticsearch store, four ways + +One benchmark in the [log10x benchmarks](../) repository. It answers one question: on +Elasticsearch 9.5.5, how much index store does the engine's compact form save over a raw index, +over logsdb index mode, and over logsdb with `message` mapped as `pattern_text`, on the same +sample? The compact form is indexed in standard mode, in logsdb mode, and in logsdb mode sorted +on its template id; the comparison that decides is compact on logsdb against logsdb with +`pattern_text`. + +Two things to know before reading the table. The raw index holds the parsed record: one field per +key of the shipped JSON, the message in `message`. And document counts differ by design: the three native indexes hold one +document per line (197,430), while the engine folds a multi-line event into one, so the compact +indexes hold one per event (157,228) plus one per template (3,030). Each index is checked +against its own input. + +## The result + +Elasticsearch 9.5.5 (Lucene 10.5.1) on a trial licence, engine +`ghcr.io/log-10x/edge-10x:1.1.150`, the published OpenTelemetry demo sample (215,039,161 B, +197,430 lines). Primary store after a force merge to one segment, `best_compression` on every +index, two repetitions: + +| Variant | Index mode | `message` | Store, rep 1 | Store, rep 2 | Mean | Of (a) | Of (b) | +|---|---|---|---:|---:|---:|---:|---:| +| (a) raw | standard | `text` | 20,108,998 B | 20,106,803 B | 20,107,900 B | 100.0% | 167.6% | +| (b) logsdb | logsdb | `text` | 12,003,158 B | 11,997,590 B | 12,000,374 B | 59.7% | 100.0% | +| (c) logsdb + `pattern_text` | logsdb, sorted on `message.template_id` | `pattern_text` | 6,977,160 B | 6,972,360 B | 6,974,760 B | 34.7% | 58.1% | +| (d) compact | standard | encoded event | 14,909,550 B | 14,901,237 B | 14,905,394 B | 74.1% | 124.2% | +| (e) compact on logsdb | logsdb | encoded event | 10,721,205 B | 10,718,972 B | 10,720,088 B | 53.3% | 89.3% | +| (f) compact on logsdb, sorted | logsdb, sorted on `l1x_tid` | encoded event | 8,875,307 B | 8,872,791 B | 8,874,049 B | 44.1% | 73.9% | + +The compact variants' store includes the template dictionary index (537,262 B). + +**Logsdb with `pattern_text`, sorted on its template id, takes less store than the compact form +in every mode.** Compact on logsdb (e) is 153.7% of (c), 53.7% larger; sorted the same +way on its own template id (f), 127.2% of (c), 27.2% larger. The native pair alone holds +this sample in 34.7% of the raw index's store, 65.3% less. Over (a) and (b) the compact form still saves: (e) is 53.3% of (a) and +89.3% of (b). + +Where the difference sits, from `_disk_usage` (repetition 1): the stored compact text in (f), +`message._original`, is about the size of what `pattern_text` stores for (c), +`message.args` plus `message.template`. What the compact form adds is the inverted index over the +encoded `message`, which indexes every variable value as a term, the container, pod and image +ids folded into each event included. In (c) those values sit in `keyword` doc values instead. + +### Amendment, 2026-10-25 + +The sort setting alone did not reach Lucene: with `index.logsdb.default_sort_on_message_template: +true` and documents that carry no `host.name`, the segments of (c) carried a sort on `@timestamp` +only, and (c) measured 10,649,904 B, 52.9% of (a). That run is in +[`results-setting-only/`](results-setting-only/), where the sort check fails. (c) sets +`index.sort.field` explicitly, as Elastic's `pattern_text` page shows in its "Custom sorting" +example, and (f) gives the compact form the same sort on its own template id. `METHOD.md` records +the amendment at its end. + +### The licence + +`pattern_text` is a subscription feature, and the basic licence accepts it with templating off. +On the default basic licence, before the trial, creating a logsdb index with `message` as +`pattern_text` returned 200 and a document indexed (201), and the mapping came back as `{"type": "pattern_text", "disable_templating": true}` and the source mode as +`STORED`: templating off and no synthetic source. On the trial, (c)'s mapping is plain +`{"type": "pattern_text"}`. Both are in `results/licence.json`. + +### The compact form + +The two encodes of the sample are byte-identical: 157,228 events and +3,030 templates, 74,140,583 B encoded plus 3,281,830 B dictionary, +64.0% under the sample before any codec. The dictionary followed by the events decodes to the +sample byte for byte (decoded output sha256 `aa79b9349a4123d88936fd064f5eccd9fd84f45a1fd03cff3f64c259715c2432`). Hashes of both +encodes are in `results/compact.json`. + +## Method + +[`METHOD.md`](METHOD.md), written before the run, with its amendment: the sample, how the compact +form is made, the index definitions, the measurement, and the checks. All 66 checks pass in +`results/results.md`. + +## Reproduce + +``` +./run.sh # encode twice, decode check, one Elasticsearch node, six variants twice, report +./run.sh report # re-render results/results.md from results/ +``` + +Needs Docker, Python 3 and `curl`; about 2 GB of memory for Elasticsearch and 3 GB of free disk. +The sample is fetched from the `otel-sample-v1` release of `log-10x/config` into `data/` unless +`ES4_SAMPLE` names a local copy; either way its sha256 is checked. The engine and Elasticsearch +images are pinned by digest. Every container is started detached under a fixed name and removed +with `docker rm -f -v`, on exit too. The whole run takes about 10 minutes. + +## What is claimed and what is not + +- Primary store size after a force merge to one segment, per variant, on this sample, with these + images and a trial licence. +- Not query or ingest cost, not search-time decoding with the l1es plugin, not + Elastic Cloud pricing, not a compact form of the message text alone with the metadata kept as + fields. `METHOD.md` lists the rest. + +## Files + +| File | What it is | +|---|---| +| `METHOD.md` | the plan, written before the run, and its amendment | +| `run.sh` | the whole run | +| `bench.py` | the Elasticsearch side: licence, index definitions, loading, measurement, report | +| `results/compact.json` | the compact form: counts and sha256 of both encodes, whether they match, the decoded output's sha256 and the byte comparison with the sample | +| `results/licence.json` | the version, the response to creating a `pattern_text` index on the default licence, the trial | +| `results/rep.json` | each repetition: per index store, documents, segments and their sort, settings, leases, per-field bytes | +| `results/rep/` | each index's mapping and `_disk_usage` output | +| `results/results.json`, `results/results.md` | the table, the ratios and the checks | +| `results-setting-only/` | the run before the amendment, (c) with `default_sort_on_message_template` alone | diff --git a/intent/README.md b/intent/README.md index b7a0de7..2ae7df4 100644 --- a/intent/README.md +++ b/intent/README.md @@ -1,198 +1,44 @@ # Intent: does each pattern hold one logging statement? -A log pattern is useful when it stands for one thing the code does. A cap, a -sample rate or a route set on a pattern then acts on that one statement. This -benchmark measures that against ground truth. Every labelled event of two -captures of the OpenTelemetry demo (99.58% of otel215's events, 99.42% of rec1155's) carries -the source statement that wrote it, found in the program's own source at the -version that ran. Each tool's patterns are then scored against those labels. A third -capture, LogHub's ZooKeeper log, was [labelled after the engine was fixed](#a-third-capture-loghub-zookeeper): -every one of its events is labelled, the labels were frozen before any tool ran, and each -tool ran once. [METHOD.md](METHOD.md) -describes how a label is made. +A cap or sample rate set on a log pattern is only safe when the pattern stands for one logging statement. This benchmark labels each event with the source line that wrote it and counts how often a tool puts two statements in one pattern. On three captures the 10x engine merged 0.01% to 0.21% of events, fewer than Drain3 in every configuration. Drain3 with masking split fewer events and produced fewer patterns on one of two demo captures and on ZooKeeper. -Three outcomes, per event: +A merged event shares its pattern with a different statement, so a cap on that pattern acts on both. A split event sits outside its statement's main pattern, and that pattern can still be capped on its own. -- **merged**: the event shares its pattern with a different statement that holds most of the pattern's events. This is the defect: two statements under one pattern share one cap, one sample rate, one route. -- **split**: the event sits outside its statement's most frequent pattern. One statement under several patterns can still be capped one pattern at a time, so a split costs rule count, not correctness. -- **exact grouping**: the event's statement has exactly one pattern, and that pattern holds only that statement. +## Result -Two tools are scored on the same events: -- the 10x engine: 1.1.142, the current release, on all three captures; 1.1.132 and 1.1.133 on the two demo captures; 1.1.136 on the ZooKeeper capture; -- [Drain3](https://github.com/logpai/Drain3) 0.9.11, in two arms with the settings of [`../pattern-identity`](../pattern-identity), and at drain3's factory defaults. +Share of events merged with another statement. otel215 and rec1155 are two OpenTelemetry demo captures. -## Results +| Tool | otel215 | rec1155 | ZooKeeper | +|---|---:|---:|---:| +| 10x engine | 0.01% (13) | 0.15% (63) | 0.21% (153) | +| Drain3, example-config masking | 0.05% (83) | 1.78% (734) | 0.75% (555) | +| Drain3, typed-token masking | 0.06% (99) | 1.81% (749) | 0.55% (412) | +| Drain3, factory defaults | 0.15% (239) | 3.34% (1,381) | 1.12% (831) | -`otel215`: 215 MB, 157,228 events, 156,563 labelled. +Drain3's merges on otel215 pair statements that differ in one word, such as Grafana's "starting to provision dashboards" and "finished to provision dashboards". -| tool | merged events | split events | exact grouping | statements with one pattern of their own | patterns | -|---|---:|---:|---:|---:|---:| -| 10x engine 1.1.142 | 0.01% (13) | 4.59% | 91.38% | 77.59% of 299 | 2,277 | -| 10x engine 1.1.133 | 0.01% (13) | 4.59% | 91.38% | 77.59% of 299 | 2,277 | -| 10x engine 1.1.132 | 4.44% (6,944) | 4.59% | 78.83% | 75.92% of 299 | 2,274 | -| Drain3, strong arm | 0.05% (83) | 3.47% | 80.95% | 76.25% of 299 | 865 | -| Drain3, baseline arm | 0.06% (99) | 3.41% | 80.92% | 74.25% of 299 | 812 | -| Drain3, factory defaults | 0.15% (239) | 3.47% | 80.57% | 67.56% of 299 | 686 | -| Drain3, strong arm, per record | 4.48% (7,013) | 3.47% | 76.57% | 76.25% of 299 | 851 | +- Drain3 splits 3.47% of otel215's events against the engine's 4.59%, and with masking 0.32% of ZooKeeper's against 1.21%. The engine splits fewer on rec1155. +- Drain3 produces 865 patterns on otel215 against 2,277, and 58 on ZooKeeper against 97. Two Grafana statements make 1,331 of the engine's, one pattern per migration id. +- Fed one record at a time, as it normally runs, Drain3 merges 4.48% of otel215's events, because every cart service message starts with the same .NET header line. -`rec1155`: 41,573 events, 41,333 labelled. +## How it was measured -| tool | merged events | split events | exact grouping | statements with one pattern of their own | patterns | -|---|---:|---:|---:|---:|---:| -| 10x engine 1.1.142 | 0.15% (63) | 9.95% | 85.94% | 78.4% of 213 | 406 | -| 10x engine 1.1.133 | 0.15% (63) | 9.95% | 85.94% | 78.4% of 213 | 406 | -| 10x engine 1.1.132 | 1.1% (454) | 9.95% | 84.05% | 77.0% of 213 | 404 | -| Drain3, strong arm | 1.78% (734) | 10.37% | 28.42% | 63.38% of 213 | 386 | -| Drain3, baseline arm | 1.81% (749) | 10.49% | 28.26% | 62.91% of 213 | 420 | -| Drain3, factory defaults | 3.34% (1,381) | 10.34% | 26.19% | 57.75% of 213 | 393 | -| Drain3, strong arm, per record | 1.78% (734) | 10.35% | 79.79% | 65.26% of 213 | 377 | +otel215 has 157,228 events from Fluentd, rec1155 has 41,573 from Fluent Bit, and ZooKeeper is LogHub's 74,380-line log from 84 statements. Each event was labeled from source code alone with the file and line that wrote it, and the labels were frozen before any tool's output was read. They cover 99.58% of otel215, 99.42% of rec1155 and all of ZooKeeper. The engine (release 1.1.142) ran as shipped, Drain3 0.9.11 was given the same events, and `bench/score.py` scored both against the labels. -The patterns column counts the patterns of labelled events only (`distinct_patterns` in `bench/score.py`): 2,277 on otel215 and 406 on rec1155 for 1.1.142. [`../naming-stability`](../naming-stability) counts the patterns of every event, with the same 1.1.142 image on the same captures: 2,687 on otel215 and 582 on rec1155. The counts differ by 410 on otel215 and 176 on rec1155. The difference is the patterns that only unlabelled events take. The two runs cut each capture into the same events and give every event the same pattern name. On otel215 the 665 unlabelled events take 411 patterns, and one of them, `Starting`, also names a labelled event, so they add 410. On rec1155 the 240 unlabelled events take 176 patterns that no labelled event shares, so they add 176. In frontend-proxy, which has no labelled events, each unlabelled event takes a pattern of its own: 114 on otel215 and 115 on rec1155, most of them lines of Envoy's startup listing. The next largest are opensearch with 83 patterns, jaeger 58 and prometheus-server 38 on otel215, and postgresql 23 and flagd 13 on rec1155. [`bench/pattern_counts.py`](bench/pattern_counts.py) computes these numbers from the two runs' events ([otel215](results/pattern-counts-1.1.142_otel215.json), [rec1155](results/pattern-counts-1.1.142_rec1155.json)). +## Limits -Without the opentelemetry-collector container (57% of otel215's events, 46% of rec1155's): +- No capture is a clean out-of-sample test. Both demo captures and 2,000 ZooKeeper lines were in view during the engine's rule work. +- Log10x wrote the labels, and each names a source line in `labels/` that can be checked. +- The engine's built-in library lists the source behind 145,612 of otel215's 156,563 labeled events and none of ZooKeeper's. On otel215 statements outside it, the engine merges 0.13% of events against Drain3's 0.86%. +- Only two applications are covered, and the collector's debug exporter wrote 85,739 of otel215's 157,228 events. -| tool | otel215 merged | otel215 exact grouping | rec1155 merged | rec1155 exact grouping | -|---|---:|---:|---:|---:| -| 10x engine 1.1.142 | 0.02% (13) | 79.8% | 0.28% (63) | 73.91% | -| 10x engine 1.1.133 | 0.02% (13) | 79.8% | 0.28% (63) | 73.91% | -| 10x engine 1.1.132 | 10.39% (6,944) | 50.41% | 2.04% (454) | 70.4% | -| Drain3, strong arm | 0.12% (79) | 55.61% | 3.3% (734) | 52.71% | -| Drain3, baseline arm | 0.14% (94) | 55.54% | 3.36% (749) | 52.42% | -| Drain3, factory defaults | 0.35% (233) | 54.72% | 6.2% (1,381) | 48.58% | - -1.1.142 scores the same as 1.1.133 on both demo captures and the same as 1.1.136 on ZooKeeper: each of its score files equals the earlier one byte for byte, the lists of merged patterns and split statements included. - -[`results/results.md`](results/results.md) has every table: -- split shares without the collector; -- scores by whether the statement's source is in the engine's default library; -- the engine's origin and name scores; -- the Drain3 run statistics. - -The `results/*.json` files list every merged pattern with its statements, and every split statement with its top patterns. - -## Reading the numbers - -**Merges.** 1.1.133 and 1.1.142 merge the fewest events on both captures. -- On otel215 the gap to Drain3 is small when Drain3 is given the engine's events: 13 events against 83. Drain3's merges there pair statements that differ in one word, such as Grafana's `starting to provision dashboards` and `finished to provision dashboards`, or OpenSearch's `Starting housekeeping task for auto refresh streaming jobs.` and its `Finished` twin. -- Fed one record at a time, as Drain3 normally runs, it also merges the cart service's three statements (6,930 events). The .NET console logger prints the same header line before each message, so every cart event starts with the same line. 1.1.132 made the same merge; 1.1.133 names such an event from its first record that holds a message. -- On rec1155, Drain3 merges 734 events against 63. The largest are payment's two OpenTelemetry log dumps (390 events each) and two product-reviews statements (225 each). -- At drain3's factory defaults, Drain3 merges more than either arm: 239 events on otel215 and 1,381 on rec1155. With no masking, a template can keep a date, so on otel215 Valkey's `%d changes in %d seconds. Saving...` and `Background saving started by pid %ld` share one template per day (120 events). On rec1155 it adds a merge of the cart service's `GetCartAsync` and `EmptyCartAsync` statements (390 events). - -**Where Drain3 does better.** -- It splits fewer events on otel215: 3.47% against 4.59%, and 8.0% against 10.76% without the collector. -- Without the collector, the two harness arms give more of otel215's statements a pattern of their own: 77.36% and 75.47% against 74.72%. -- It produces about a third as many patterns. -- On otel215 statements whose source the engine's library does not list, its exact grouping is higher: 65.49% against 62.89%. - -Two Grafana statements account for 1,331 of 1.1.133's 2,277 otel215 patterns. The migrator's `Executing migration` (`migrator.go:369`) yields 666 patterns, one per migration id, because the ids are made of words the library knows, and its `Migration successfully executed` twin yields 665. - -**Exact grouping is all or nothing per statement.** One stray event makes every event of its statement count as not exact. Drain3 on rec1155 keeps 7,060 of the 7,063 events of the collector's `Traces` statement in one pattern. The other three sit in single-event patterns built from early lines, before the template generalized. Across the three debug-exporter statements, six such events decide whether 19,054 events count as exact. Fed per record, Drain3 builds none of them, and its exact grouping on rec1155 moves from 28.42% to 79.79% on an input that differs by 140 continuation records. Merged events is the stable comparison. - -## A third capture: LogHub ZooKeeper - -`zookeeper` is LogHub's ZooKeeper log: 74,380 lines from a three-server ZooKeeper 3.4.5 -ensemble, every one labelled, 84 statements. Its labels were frozen and committed before any -tool ran on it (`labels/FROZEN_zookeeper.json`). It is not held out from the engine's development: LogHub's 2,000-line ZooKeeper sample, drawn from this log, was one of 16 LogHub sets the rule work scored by LogHub's template labels, so 2,000 of its lines were in view; no rule was built from the statement labels, which did not exist until this capture was labelled. Each tool then ran once: the released engine 1.1.136 -with its shipped configuration, and Drain3 in its two arms and at its factory defaults. The current release, 1.1.142, ran later on the same frozen labels. [METHOD.md](METHOD.md#the-third-capture-zookeeper) -gives the version evidence and how the labels were made. - -| tool | merged events | split events | exact grouping | statements with one pattern of their own | patterns | -|---|---:|---:|---:|---:|---:| -| 10x engine 1.1.142 | 0.21% (153) | 1.21% | 95.38% | 69.05% of 84 | 97 | -| 10x engine 1.1.136 | 0.21% (153) | 1.21% | 95.38% | 69.05% of 84 | 97 | -| Drain3, strong arm | 0.75% (555) | 0.32% | 91.31% | 50.0% of 84 | 58 | -| Drain3, baseline arm | 0.55% (412) | 0.32% | 93.25% | 55.95% of 84 | 64 | -| Drain3, factory defaults | 1.12% (831) | 1.9% | 76.95% | 42.86% of 84 | 72 | - -Each event is one line, so Drain3 fed one record at a time gives the same figures. The -1.1.89 library does not list ZooKeeper, so every statement falls in the "source not in the -library" group and that split equals the table above. - -**Merges.** -- Thirteen statements print the same message as another statement and differ only in the class and line that log4j prints in brackets: nine `Shutting down`, two `TCP NoDelay set to: true` and two `Got zxid 0x{} expected 0x{}`. A tool that names a line from its level and message gives each group one name, which merges at least 66 of their 125 events. Neither tool separates them. -- 1.1.136 merges those 66 and 87 more. `autopurge.snapRetainCount set to`, `autopurge.purgeInterval set to` and `minSessionTimeout set to` share the name `set_to` (72 events), while `tickTime`, `maxSessionTimeout` and `initLimit` keep their own names. Four `exited loop!` statements, each starting with its class name, share one pattern (15 events). -- Drain3's strong arm puts `Notification time out: {}` (1,322 events) and all six `set to` statements under one template (216 merged events). The baseline arm keeps the time-out apart and merges the six (180). Both arms merge `Client attempting to establish new session` with `Client attempting to renew session` (71 events). -- At factory defaults Drain3 merges 831 events. With no masking, templates such as `2015-07-31 <*> - INFO <*> - <*> <*> <*> <*>` keep the date and hold lines of several statements written that day. It also merges the two `Client attempting to` statements (71 events). - -**Where Drain3 does better.** -- The two harness arms split fewer events: 0.32% against 1.21%. 1.1.136 gives `Server environment:` one pattern per environment key (16 patterns, 504 split events) and the election `Notification:` line one per combination of peer states (9 patterns, 212 split events). At factory defaults Drain3 splits 1.9%. -- It produces 58, 64 or 72 patterns against 97. - -1.1.136 gives more statements a pattern of their own: 58 of 84, against 42 for the strong arm, 47 for the baseline and 36 at factory defaults. - -## Caveats - -- **Two applications.** otel215 and rec1155 are the OpenTelemetry demo: two versions (2.2.0 and 2.1.3), two forwarders (Fluent Bit and Fluentd), two clusters. The ZooKeeper capture is one Java program, ZooKeeper 3.4.5, which prints the class and line of every log call. -- **The collector's debug exporter is half of otel215.** Its three statements are 85,739 of otel215's 157,228 events. The tables without the opentelemetry-collector container show the rest. -- **Unlabelled events are excluded.** 0.58% of rec1155's events and 0.42% of otel215's carry no label and are excluded from every score; [METHOD.md](METHOD.md) lists them by container. Every zookeeper event is labelled. -- **The default library lists much of what runs.** The engine names a line from the library compiled into it: the 1.1.89 library, inside all three engine images. - - Its manifest lists the repository or image behind 144 of otel215's 289 code statements (145,612 of its 156,563 labelled events) and 199 of rec1155's 207 (39,902 of 41,333). These include the OpenTelemetry demo itself, Kafka and the collector. - - The tables' 299 and 213 statements are these 289 and 207 code statements plus 10 and 6 [pseudo statements](METHOD.md#pseudo-statements): events that hold no single statement's text, such as access logs, stack frames and object dumps. - - That cuts both ways: where the source is listed, the engine can find the statement; where it is not (Grafana, fluentd, OpenSearch, Kubernetes), the engine names the line from the words in it that the library knows. - - `results/results.md` splits every score this way. On otel215 statements outside the library, 1.1.133 merges 0.13% of events against Drain3's 0.86%, and Drain3's exact grouping is higher. - - The manifest does not list ZooKeeper, so the ZooKeeper capture is scored entirely outside the library. -- **Event boundaries are the engine's.** An event is one record or a run of records the engine groups (a stack trace, a .NET header and its message). Drain3 is scored on those events, either given each event's text (the default rows) or fed one record at a time (the "per record" rows). -- **Who wrote the labels.** Log10x wrote them, from source code alone, and froze them before any tool's output was read: otel215 and rec1155 on 2026-10-05 (`labels/FROZEN.json`), zookeeper on 2026-10-07 (`labels/FROZEN_zookeeper.json`), before any tool ran on it. Every label names a repository, ref, path and line in `labels/statements.tsv` or `labels/statements_zookeeper.tsv`, so any one can be checked against its source. -- **Which columns are in-sample.** The 1.1.133 column is in-sample. Engine 1.1.133's group-lead rule (a multi-line event is named from its first record that holds a message) was developed after the freeze, with otel215 and rec1155 in view, so its column shows the rule on the data it was built against. 1.1.142 is the current release (2026-10-15) and carries the same rule, so on otel215 and rec1155 it is in-sample in the same sense; the release notes of 1.1.141 report this benchmark's merged and split counts on both captures as a check on that release. The ZooKeeper columns are labelled after the fact but are not a clean out-of-sample test (see its section). 1.1.142 ran on those labels after the 1.1.136 scores were published, and the release notes of 1.1.137 to 1.1.142 name no change made for ZooKeeper. A capture never used in the engine's development would be a clean out-of-sample test; none is scored here. Drain3 ran at the pattern-identity harness's fixed settings, listed below, which were not tuned to these captures, and at drain3's factory defaults. - -## Drain3 settings - -The strong and baseline arms take their settings from `../pattern-identity/bench/identity.py`, imported rather than copied. These are not drain3's factory values: -- depth 6, `sim_th` 0.6, `max_children` 20, `max_clusters` 2,000; -- numeric tokens parametrized; -- baseline arm: typed-token masking; strong arm: the masking block of Drain3's `examples/drain3.ini`. - -The factory-defaults arm uses `TemplateMinerConfig()` as drain3 0.9.11 ships it: -- depth 4, `sim_th` 0.4, `max_children` 100, no cluster cap; -- numeric tokens parametrized; -- no masking. - -Every arm is given the first 1,024 characters of each message. Each run is the harness's reference pass: one instance learns the whole capture in memory, then every message is matched back read-only (`full_search_strategy="fallback"`) and named by the template it matches. No message went unmatched. - -Fed one record at a time, the run reproduces the harness's own reference pass on otel215: 1,481 clusters under the strong arm and 1,465 under the baseline. drain3 0.9.11 on Python 3.13.11. - -## Engines - -| engine | image | library inside | -|---|---|---| -| 1.1.142 (all three captures) | `log10x/pipeline-10x:1.1.142@sha256:22ae67148bd7f6cbdf171c1b8da1c9c6c801290d43488693dfd8527e27138d05` | the 1.1.89 library | -| 1.1.136 (ZooKeeper capture) | `log10x/pipeline-10x:1.1.136@sha256:fccff37a6a41e52eb4df8a108d042b0176210842b7ed25f08b1419df3a37904f` | the 1.1.89 library | -| 1.1.133 | `log10x/pipeline-10x:1.1.133@sha256:fe3dcdefdc2f42fa7117c4f63fe97b28508fbf1356725422703f0dfab55f2360` | the 1.1.89 library | -| 1.1.132 | `log10x/pipeline-10x:1.1.132@sha256:3e21cd41cde6d8263b5fa1b121ff15587a6bfe33ca4b3338b08bbb8c0e7a0c5f` | the 1.1.89 library | - -The library files inside the four images are byte-identical: manifest `4ad0993d5e32b9b6`, symbols `1c539d9496761c8d` (JSON) and `e5544b5f274b876d` (protobuf), first 16 hex digits of SHA-256. - -All four engines run with their shipped configuration through the dev app (`@apps/dev`), which writes each event's pattern name, typed origin and text. `bench/engine.sh` mounts an empty directory over the sample input the image ships, so the engine reads the capture alone. - -## Reproduce - -``` -./run.sh -``` - -This needs Docker, curl and Python 3. `run.sh` does the following: -- downloads otel215 from the `otel-sample-v1` release of log-10x/config, rec1155 from this repository's `intent-data-v1` release and `Zookeeper.tar.gz` from LogHub's [Zenodo record 8196385](https://zenodo.org/records/8196385), checking every hash (the ZooKeeper log's CRLF line ends are converted to LF); -- runs the 1.1.132, 1.1.133 and 1.1.142 images on the demo captures, the 1.1.136 and 1.1.142 images on zookeeper, and the six Drain3 combinations (three arms, each given events or records) on each capture; -- scores every run and writes `results/`. - -On an Intel laptop with Docker Desktop it takes about 55 minutes. The 1.1.142 runs were made on 2026-10-15 with no other container running on the host. - -To score another engine image: +## Run it ``` -bench/engine.sh log10x/pipeline-10x: cache/otel215.log out/ -python3 bench/score.py otel215 out/events.jsonl +python3 intent/bench/summarize.py # print the result tables from the committed scores +cd intent && ./run.sh # full rerun ``` -`bench/score.py` refuses a run whose events are not the labelled events one for one. - -## Files +A full rerun needs Docker, curl and Python 3 and takes about 55 minutes on an Intel laptop. -- `METHOD.md`: how the labels were made, coverage, exclusions, versions. -- `labels/`: per-event labels, the statement registries, text hashes for the alignment check, file names per repository for the origin score, and the freeze records (`FROZEN.json` for otel215 and rec1155, `FROZEN_zookeeper.json` for zookeeper). -- `bench/engine.sh`, `bench/events.config.yaml`: run an engine image over a capture. -- `bench/drain_arm.py`: the Drain3 arms. -- `bench/score.py`: the scorer. -- `bench/summarize.py`: writes `results/results.md`. -- `bench/pattern_counts.py`: compares this benchmark's pattern count with `../naming-stability`'s, from one run of each on the same capture, and writes `results/pattern-counts-1.1.142_.json`. -- `results/`: the committed output of `run.sh` and of `bench/pattern_counts.py`. +[RECORD.md](RECORD.md) has the full score tables, the Drain3 settings and the image digests, and [METHOD.md](METHOD.md) explains how each label was made. diff --git a/intent/RECORD.md b/intent/RECORD.md new file mode 100644 index 0000000..b7a0de7 --- /dev/null +++ b/intent/RECORD.md @@ -0,0 +1,198 @@ +# Intent: does each pattern hold one logging statement? + +A log pattern is useful when it stands for one thing the code does. A cap, a +sample rate or a route set on a pattern then acts on that one statement. This +benchmark measures that against ground truth. Every labelled event of two +captures of the OpenTelemetry demo (99.58% of otel215's events, 99.42% of rec1155's) carries +the source statement that wrote it, found in the program's own source at the +version that ran. Each tool's patterns are then scored against those labels. A third +capture, LogHub's ZooKeeper log, was [labelled after the engine was fixed](#a-third-capture-loghub-zookeeper): +every one of its events is labelled, the labels were frozen before any tool ran, and each +tool ran once. [METHOD.md](METHOD.md) +describes how a label is made. + +Three outcomes, per event: + +- **merged**: the event shares its pattern with a different statement that holds most of the pattern's events. This is the defect: two statements under one pattern share one cap, one sample rate, one route. +- **split**: the event sits outside its statement's most frequent pattern. One statement under several patterns can still be capped one pattern at a time, so a split costs rule count, not correctness. +- **exact grouping**: the event's statement has exactly one pattern, and that pattern holds only that statement. + +Two tools are scored on the same events: +- the 10x engine: 1.1.142, the current release, on all three captures; 1.1.132 and 1.1.133 on the two demo captures; 1.1.136 on the ZooKeeper capture; +- [Drain3](https://github.com/logpai/Drain3) 0.9.11, in two arms with the settings of [`../pattern-identity`](../pattern-identity), and at drain3's factory defaults. + +## Results + +`otel215`: 215 MB, 157,228 events, 156,563 labelled. + +| tool | merged events | split events | exact grouping | statements with one pattern of their own | patterns | +|---|---:|---:|---:|---:|---:| +| 10x engine 1.1.142 | 0.01% (13) | 4.59% | 91.38% | 77.59% of 299 | 2,277 | +| 10x engine 1.1.133 | 0.01% (13) | 4.59% | 91.38% | 77.59% of 299 | 2,277 | +| 10x engine 1.1.132 | 4.44% (6,944) | 4.59% | 78.83% | 75.92% of 299 | 2,274 | +| Drain3, strong arm | 0.05% (83) | 3.47% | 80.95% | 76.25% of 299 | 865 | +| Drain3, baseline arm | 0.06% (99) | 3.41% | 80.92% | 74.25% of 299 | 812 | +| Drain3, factory defaults | 0.15% (239) | 3.47% | 80.57% | 67.56% of 299 | 686 | +| Drain3, strong arm, per record | 4.48% (7,013) | 3.47% | 76.57% | 76.25% of 299 | 851 | + +`rec1155`: 41,573 events, 41,333 labelled. + +| tool | merged events | split events | exact grouping | statements with one pattern of their own | patterns | +|---|---:|---:|---:|---:|---:| +| 10x engine 1.1.142 | 0.15% (63) | 9.95% | 85.94% | 78.4% of 213 | 406 | +| 10x engine 1.1.133 | 0.15% (63) | 9.95% | 85.94% | 78.4% of 213 | 406 | +| 10x engine 1.1.132 | 1.1% (454) | 9.95% | 84.05% | 77.0% of 213 | 404 | +| Drain3, strong arm | 1.78% (734) | 10.37% | 28.42% | 63.38% of 213 | 386 | +| Drain3, baseline arm | 1.81% (749) | 10.49% | 28.26% | 62.91% of 213 | 420 | +| Drain3, factory defaults | 3.34% (1,381) | 10.34% | 26.19% | 57.75% of 213 | 393 | +| Drain3, strong arm, per record | 1.78% (734) | 10.35% | 79.79% | 65.26% of 213 | 377 | + +The patterns column counts the patterns of labelled events only (`distinct_patterns` in `bench/score.py`): 2,277 on otel215 and 406 on rec1155 for 1.1.142. [`../naming-stability`](../naming-stability) counts the patterns of every event, with the same 1.1.142 image on the same captures: 2,687 on otel215 and 582 on rec1155. The counts differ by 410 on otel215 and 176 on rec1155. The difference is the patterns that only unlabelled events take. The two runs cut each capture into the same events and give every event the same pattern name. On otel215 the 665 unlabelled events take 411 patterns, and one of them, `Starting`, also names a labelled event, so they add 410. On rec1155 the 240 unlabelled events take 176 patterns that no labelled event shares, so they add 176. In frontend-proxy, which has no labelled events, each unlabelled event takes a pattern of its own: 114 on otel215 and 115 on rec1155, most of them lines of Envoy's startup listing. The next largest are opensearch with 83 patterns, jaeger 58 and prometheus-server 38 on otel215, and postgresql 23 and flagd 13 on rec1155. [`bench/pattern_counts.py`](bench/pattern_counts.py) computes these numbers from the two runs' events ([otel215](results/pattern-counts-1.1.142_otel215.json), [rec1155](results/pattern-counts-1.1.142_rec1155.json)). + +Without the opentelemetry-collector container (57% of otel215's events, 46% of rec1155's): + +| tool | otel215 merged | otel215 exact grouping | rec1155 merged | rec1155 exact grouping | +|---|---:|---:|---:|---:| +| 10x engine 1.1.142 | 0.02% (13) | 79.8% | 0.28% (63) | 73.91% | +| 10x engine 1.1.133 | 0.02% (13) | 79.8% | 0.28% (63) | 73.91% | +| 10x engine 1.1.132 | 10.39% (6,944) | 50.41% | 2.04% (454) | 70.4% | +| Drain3, strong arm | 0.12% (79) | 55.61% | 3.3% (734) | 52.71% | +| Drain3, baseline arm | 0.14% (94) | 55.54% | 3.36% (749) | 52.42% | +| Drain3, factory defaults | 0.35% (233) | 54.72% | 6.2% (1,381) | 48.58% | + +1.1.142 scores the same as 1.1.133 on both demo captures and the same as 1.1.136 on ZooKeeper: each of its score files equals the earlier one byte for byte, the lists of merged patterns and split statements included. + +[`results/results.md`](results/results.md) has every table: +- split shares without the collector; +- scores by whether the statement's source is in the engine's default library; +- the engine's origin and name scores; +- the Drain3 run statistics. + +The `results/*.json` files list every merged pattern with its statements, and every split statement with its top patterns. + +## Reading the numbers + +**Merges.** 1.1.133 and 1.1.142 merge the fewest events on both captures. +- On otel215 the gap to Drain3 is small when Drain3 is given the engine's events: 13 events against 83. Drain3's merges there pair statements that differ in one word, such as Grafana's `starting to provision dashboards` and `finished to provision dashboards`, or OpenSearch's `Starting housekeeping task for auto refresh streaming jobs.` and its `Finished` twin. +- Fed one record at a time, as Drain3 normally runs, it also merges the cart service's three statements (6,930 events). The .NET console logger prints the same header line before each message, so every cart event starts with the same line. 1.1.132 made the same merge; 1.1.133 names such an event from its first record that holds a message. +- On rec1155, Drain3 merges 734 events against 63. The largest are payment's two OpenTelemetry log dumps (390 events each) and two product-reviews statements (225 each). +- At drain3's factory defaults, Drain3 merges more than either arm: 239 events on otel215 and 1,381 on rec1155. With no masking, a template can keep a date, so on otel215 Valkey's `%d changes in %d seconds. Saving...` and `Background saving started by pid %ld` share one template per day (120 events). On rec1155 it adds a merge of the cart service's `GetCartAsync` and `EmptyCartAsync` statements (390 events). + +**Where Drain3 does better.** +- It splits fewer events on otel215: 3.47% against 4.59%, and 8.0% against 10.76% without the collector. +- Without the collector, the two harness arms give more of otel215's statements a pattern of their own: 77.36% and 75.47% against 74.72%. +- It produces about a third as many patterns. +- On otel215 statements whose source the engine's library does not list, its exact grouping is higher: 65.49% against 62.89%. + +Two Grafana statements account for 1,331 of 1.1.133's 2,277 otel215 patterns. The migrator's `Executing migration` (`migrator.go:369`) yields 666 patterns, one per migration id, because the ids are made of words the library knows, and its `Migration successfully executed` twin yields 665. + +**Exact grouping is all or nothing per statement.** One stray event makes every event of its statement count as not exact. Drain3 on rec1155 keeps 7,060 of the 7,063 events of the collector's `Traces` statement in one pattern. The other three sit in single-event patterns built from early lines, before the template generalized. Across the three debug-exporter statements, six such events decide whether 19,054 events count as exact. Fed per record, Drain3 builds none of them, and its exact grouping on rec1155 moves from 28.42% to 79.79% on an input that differs by 140 continuation records. Merged events is the stable comparison. + +## A third capture: LogHub ZooKeeper + +`zookeeper` is LogHub's ZooKeeper log: 74,380 lines from a three-server ZooKeeper 3.4.5 +ensemble, every one labelled, 84 statements. Its labels were frozen and committed before any +tool ran on it (`labels/FROZEN_zookeeper.json`). It is not held out from the engine's development: LogHub's 2,000-line ZooKeeper sample, drawn from this log, was one of 16 LogHub sets the rule work scored by LogHub's template labels, so 2,000 of its lines were in view; no rule was built from the statement labels, which did not exist until this capture was labelled. Each tool then ran once: the released engine 1.1.136 +with its shipped configuration, and Drain3 in its two arms and at its factory defaults. The current release, 1.1.142, ran later on the same frozen labels. [METHOD.md](METHOD.md#the-third-capture-zookeeper) +gives the version evidence and how the labels were made. + +| tool | merged events | split events | exact grouping | statements with one pattern of their own | patterns | +|---|---:|---:|---:|---:|---:| +| 10x engine 1.1.142 | 0.21% (153) | 1.21% | 95.38% | 69.05% of 84 | 97 | +| 10x engine 1.1.136 | 0.21% (153) | 1.21% | 95.38% | 69.05% of 84 | 97 | +| Drain3, strong arm | 0.75% (555) | 0.32% | 91.31% | 50.0% of 84 | 58 | +| Drain3, baseline arm | 0.55% (412) | 0.32% | 93.25% | 55.95% of 84 | 64 | +| Drain3, factory defaults | 1.12% (831) | 1.9% | 76.95% | 42.86% of 84 | 72 | + +Each event is one line, so Drain3 fed one record at a time gives the same figures. The +1.1.89 library does not list ZooKeeper, so every statement falls in the "source not in the +library" group and that split equals the table above. + +**Merges.** +- Thirteen statements print the same message as another statement and differ only in the class and line that log4j prints in brackets: nine `Shutting down`, two `TCP NoDelay set to: true` and two `Got zxid 0x{} expected 0x{}`. A tool that names a line from its level and message gives each group one name, which merges at least 66 of their 125 events. Neither tool separates them. +- 1.1.136 merges those 66 and 87 more. `autopurge.snapRetainCount set to`, `autopurge.purgeInterval set to` and `minSessionTimeout set to` share the name `set_to` (72 events), while `tickTime`, `maxSessionTimeout` and `initLimit` keep their own names. Four `exited loop!` statements, each starting with its class name, share one pattern (15 events). +- Drain3's strong arm puts `Notification time out: {}` (1,322 events) and all six `set to` statements under one template (216 merged events). The baseline arm keeps the time-out apart and merges the six (180). Both arms merge `Client attempting to establish new session` with `Client attempting to renew session` (71 events). +- At factory defaults Drain3 merges 831 events. With no masking, templates such as `2015-07-31 <*> - INFO <*> - <*> <*> <*> <*>` keep the date and hold lines of several statements written that day. It also merges the two `Client attempting to` statements (71 events). + +**Where Drain3 does better.** +- The two harness arms split fewer events: 0.32% against 1.21%. 1.1.136 gives `Server environment:` one pattern per environment key (16 patterns, 504 split events) and the election `Notification:` line one per combination of peer states (9 patterns, 212 split events). At factory defaults Drain3 splits 1.9%. +- It produces 58, 64 or 72 patterns against 97. + +1.1.136 gives more statements a pattern of their own: 58 of 84, against 42 for the strong arm, 47 for the baseline and 36 at factory defaults. + +## Caveats + +- **Two applications.** otel215 and rec1155 are the OpenTelemetry demo: two versions (2.2.0 and 2.1.3), two forwarders (Fluent Bit and Fluentd), two clusters. The ZooKeeper capture is one Java program, ZooKeeper 3.4.5, which prints the class and line of every log call. +- **The collector's debug exporter is half of otel215.** Its three statements are 85,739 of otel215's 157,228 events. The tables without the opentelemetry-collector container show the rest. +- **Unlabelled events are excluded.** 0.58% of rec1155's events and 0.42% of otel215's carry no label and are excluded from every score; [METHOD.md](METHOD.md) lists them by container. Every zookeeper event is labelled. +- **The default library lists much of what runs.** The engine names a line from the library compiled into it: the 1.1.89 library, inside all three engine images. + - Its manifest lists the repository or image behind 144 of otel215's 289 code statements (145,612 of its 156,563 labelled events) and 199 of rec1155's 207 (39,902 of 41,333). These include the OpenTelemetry demo itself, Kafka and the collector. + - The tables' 299 and 213 statements are these 289 and 207 code statements plus 10 and 6 [pseudo statements](METHOD.md#pseudo-statements): events that hold no single statement's text, such as access logs, stack frames and object dumps. + - That cuts both ways: where the source is listed, the engine can find the statement; where it is not (Grafana, fluentd, OpenSearch, Kubernetes), the engine names the line from the words in it that the library knows. + - `results/results.md` splits every score this way. On otel215 statements outside the library, 1.1.133 merges 0.13% of events against Drain3's 0.86%, and Drain3's exact grouping is higher. + - The manifest does not list ZooKeeper, so the ZooKeeper capture is scored entirely outside the library. +- **Event boundaries are the engine's.** An event is one record or a run of records the engine groups (a stack trace, a .NET header and its message). Drain3 is scored on those events, either given each event's text (the default rows) or fed one record at a time (the "per record" rows). +- **Who wrote the labels.** Log10x wrote them, from source code alone, and froze them before any tool's output was read: otel215 and rec1155 on 2026-10-05 (`labels/FROZEN.json`), zookeeper on 2026-10-07 (`labels/FROZEN_zookeeper.json`), before any tool ran on it. Every label names a repository, ref, path and line in `labels/statements.tsv` or `labels/statements_zookeeper.tsv`, so any one can be checked against its source. +- **Which columns are in-sample.** The 1.1.133 column is in-sample. Engine 1.1.133's group-lead rule (a multi-line event is named from its first record that holds a message) was developed after the freeze, with otel215 and rec1155 in view, so its column shows the rule on the data it was built against. 1.1.142 is the current release (2026-10-15) and carries the same rule, so on otel215 and rec1155 it is in-sample in the same sense; the release notes of 1.1.141 report this benchmark's merged and split counts on both captures as a check on that release. The ZooKeeper columns are labelled after the fact but are not a clean out-of-sample test (see its section). 1.1.142 ran on those labels after the 1.1.136 scores were published, and the release notes of 1.1.137 to 1.1.142 name no change made for ZooKeeper. A capture never used in the engine's development would be a clean out-of-sample test; none is scored here. Drain3 ran at the pattern-identity harness's fixed settings, listed below, which were not tuned to these captures, and at drain3's factory defaults. + +## Drain3 settings + +The strong and baseline arms take their settings from `../pattern-identity/bench/identity.py`, imported rather than copied. These are not drain3's factory values: +- depth 6, `sim_th` 0.6, `max_children` 20, `max_clusters` 2,000; +- numeric tokens parametrized; +- baseline arm: typed-token masking; strong arm: the masking block of Drain3's `examples/drain3.ini`. + +The factory-defaults arm uses `TemplateMinerConfig()` as drain3 0.9.11 ships it: +- depth 4, `sim_th` 0.4, `max_children` 100, no cluster cap; +- numeric tokens parametrized; +- no masking. + +Every arm is given the first 1,024 characters of each message. Each run is the harness's reference pass: one instance learns the whole capture in memory, then every message is matched back read-only (`full_search_strategy="fallback"`) and named by the template it matches. No message went unmatched. + +Fed one record at a time, the run reproduces the harness's own reference pass on otel215: 1,481 clusters under the strong arm and 1,465 under the baseline. drain3 0.9.11 on Python 3.13.11. + +## Engines + +| engine | image | library inside | +|---|---|---| +| 1.1.142 (all three captures) | `log10x/pipeline-10x:1.1.142@sha256:22ae67148bd7f6cbdf171c1b8da1c9c6c801290d43488693dfd8527e27138d05` | the 1.1.89 library | +| 1.1.136 (ZooKeeper capture) | `log10x/pipeline-10x:1.1.136@sha256:fccff37a6a41e52eb4df8a108d042b0176210842b7ed25f08b1419df3a37904f` | the 1.1.89 library | +| 1.1.133 | `log10x/pipeline-10x:1.1.133@sha256:fe3dcdefdc2f42fa7117c4f63fe97b28508fbf1356725422703f0dfab55f2360` | the 1.1.89 library | +| 1.1.132 | `log10x/pipeline-10x:1.1.132@sha256:3e21cd41cde6d8263b5fa1b121ff15587a6bfe33ca4b3338b08bbb8c0e7a0c5f` | the 1.1.89 library | + +The library files inside the four images are byte-identical: manifest `4ad0993d5e32b9b6`, symbols `1c539d9496761c8d` (JSON) and `e5544b5f274b876d` (protobuf), first 16 hex digits of SHA-256. + +All four engines run with their shipped configuration through the dev app (`@apps/dev`), which writes each event's pattern name, typed origin and text. `bench/engine.sh` mounts an empty directory over the sample input the image ships, so the engine reads the capture alone. + +## Reproduce + +``` +./run.sh +``` + +This needs Docker, curl and Python 3. `run.sh` does the following: +- downloads otel215 from the `otel-sample-v1` release of log-10x/config, rec1155 from this repository's `intent-data-v1` release and `Zookeeper.tar.gz` from LogHub's [Zenodo record 8196385](https://zenodo.org/records/8196385), checking every hash (the ZooKeeper log's CRLF line ends are converted to LF); +- runs the 1.1.132, 1.1.133 and 1.1.142 images on the demo captures, the 1.1.136 and 1.1.142 images on zookeeper, and the six Drain3 combinations (three arms, each given events or records) on each capture; +- scores every run and writes `results/`. + +On an Intel laptop with Docker Desktop it takes about 55 minutes. The 1.1.142 runs were made on 2026-10-15 with no other container running on the host. + +To score another engine image: + +``` +bench/engine.sh log10x/pipeline-10x: cache/otel215.log out/ +python3 bench/score.py otel215 out/events.jsonl +``` + +`bench/score.py` refuses a run whose events are not the labelled events one for one. + +## Files + +- `METHOD.md`: how the labels were made, coverage, exclusions, versions. +- `labels/`: per-event labels, the statement registries, text hashes for the alignment check, file names per repository for the origin score, and the freeze records (`FROZEN.json` for otel215 and rec1155, `FROZEN_zookeeper.json` for zookeeper). +- `bench/engine.sh`, `bench/events.config.yaml`: run an engine image over a capture. +- `bench/drain_arm.py`: the Drain3 arms. +- `bench/score.py`: the scorer. +- `bench/summarize.py`: writes `results/results.md`. +- `bench/pattern_counts.py`: compares this benchmark's pattern count with `../naming-stability`'s, from one run of each on the same capture, and writes `results/pattern-counts-1.1.142_.json`. +- `results/`: the committed output of `run.sh` and of `bench/pattern_counts.py`. diff --git a/loghub-compaction/README.md b/loghub-compaction/README.md index b907fa3..4bd1659 100644 --- a/loghub-compaction/README.md +++ b/loghub-compaction/README.md @@ -1,100 +1,48 @@ # LogHub compaction -One benchmark in the [log10x benchmarks](../) repository. It answers one question: how much -smaller is the engine's compact form than the original logs, on the 14 full public LogHub -datasets, with the engine and library as they ship. +How much smaller than the original logs is the 10x engine's compact form, on the 14 full public LogHub datasets? Across all 14 the compact form came to 44.7% fewer bytes, weighted by size. Per dataset the reduction ran from 58.4% on Hadoop down to 4.7% on HealthApp. -The compact form is the encoded events plus the template dictionary needed to expand them -back. The reduction is measured before any codec (gzip, ZSTD), so it stacks with whatever -the storage layer applies after. +The compact form is the encoded events plus the template dictionary needed to expand them back. Both are counted before any codec, so the figure stacks with whatever codec the storage layer applies after, such as gzip or ZSTD. -## The result +## Result -Engine `log10x/pipeline-10x:1.1.133` by digest, the 1.1.89 symbol library it ships, each -corpus cold: +The table shows the four datasets over 100 MB, the highest and lowest of the other ten, and the total. - -| Corpus | Events | Input | Compact (encoded + dictionary) | Reduction | Ratio | Templates | Against the ledger | -|---|---:|---:|---:|---:|---:|---:|---| -| Hadoop | 186,697 | 48,595,595 B | 20,212,363 B | 58.4% | 2.4x | 931 | events 186,697 against 186,695; compact bytes 20,212,363 against 20,198,860; templates 931 against 888 | -| Spark | 32,952,127 | 2,941,224,304 B | 1,248,606,093 B | 57.5% | 2.36x | 3,370 | events 32,952,127 against 32,952,122; compact bytes 1,248,606,093 against 1,248,618,288; templates 3,370 against 3,377 | -| Apache | 52,004 | 5,135,876 B | 2,295,929 B | 55.3% | 2.24x | 371 | equal | -| OpenSSH | 655,147 | 73,417,506 B | 36,854,297 B | 49.8% | 1.99x | 3,032 | equal | -| Proxifier | 21,329 | 2,541,814 B | 1,353,818 B | 46.7% | 1.88x | 1,854 | equal | -| Zookeeper | 74,380 | 10,426,438 B | 5,673,286 B | 45.6% | 1.84x | 157 | equal | -| Linux | 25,567 | 2,349,686 B | 1,303,867 B | 44.5% | 1.8x | 1,262 | compact bytes 1,303,867 against 1,304,303; templates 1,262 against 1,266 | -| HPC | 330,461 | 33,553,503 B | 20,485,336 B | 38.9% | 1.64x | 2,045 | equal | -| BGL | 4,728,216 | 743,185,031 B | 482,351,424 B | 35.1% | 1.54x | 136,461 | equal | -| Mac | 107,220 | 16,879,552 B | 11,391,093 B | 32.5% | 1.48x | 2,466 | equal | -| OpenStack | 207,820 | 61,442,082 B | 42,663,820 B | 30.6% | 1.44x | 8,134 | compact bytes 42,663,820 against 42,663,612; templates 8,134 against 8,133 | -| HDFS | 11,175,629 | 1,577,982,906 B | 1,118,008,927 B | 29.1% | 1.41x | 279 | equal | -| Android | 1,555,005 | 192,270,829 B | 153,830,847 B | 20% | 1.25x | 29,400 | equal | -| HealthApp | 204,648 | 23,529,930 B | 22,432,794 B | 4.7% | 1.05x | 1,341 | equal | +| Dataset | Events | Input bytes | Compact bytes | Reduction | +|---|---:|---:|---:|---:| +| Hadoop | 186,697 | 48,595,595 | 20,212,363 | 58.4% | +| Spark | 32,952,127 | 2,941,224,304 | 1,248,606,093 | 57.5% | +| BGL | 4,728,216 | 743,185,031 | 482,351,424 | 35.1% | +| HDFS | 11,175,629 | 1,577,982,906 | 1,118,008,927 | 29.1% | +| Android | 1,555,005 | 192,270,829 | 153,830,847 | 20% | +| HealthApp | 204,648 | 23,529,930 | 22,432,794 | 4.7% | +| All 14 | 52,276,250 | 5,732,535,052 | 3,167,463,894 | 44.7% | -**All 14:** 52,276,250 events, 5,732,535,052 B in, 3,167,463,894 B compact, byte-weighted reduction 44.7%. The 4 over 100 MB (Spark, BGL, HDFS, Android): 44.9%. - +The four over 100 MB together came to 44.9%. The spread is wide, and HealthApp saves little. All 14 rows, with template counts, are in [results/results.md](results/results.md). -The last column sets each corpus against the measurement the compaction post's figures come -from (the ledger, `ledger.json`). That run used a local build of the same engine version on -macOS; this one uses the released image in a Linux container. Ten of the fourteen reproduce -it byte for byte, and every reduction holds to the decimal place. Where a corpus is many files (Hadoop, Spark, OpenStack) the engine reads -them in directory order, which differs between the two file systems, so multi-line grouping -and the order templates are first seen move slightly. Linux's template count varies from run to run on any setup, because two-digit days -match both `MMM d` and `MMM dd` and the format is chosen per event; its reduction does not. +- Ten of the 14 match, byte for byte, the figures in the log10x post [Log compaction, measured](https://www.log10x.com/blog/log-compaction-measured/), and every reduction matches to the decimal place. +- Hadoop, Spark and OpenStack differ slightly. Each is many files read in directory order, and that order differs between the macOS file system of the post's run and the Linux container here. +- Linux differs because its template count changes from run to run. A two-digit day matches both the MMM d and the MMM dd date formats, and the format is chosen per event. Its reduction stays the same. -## Method +## How it was measured -- **Inputs.** The full datasets of [LogHub](https://github.com/logpai/loghub) from - [Zenodo record 3227177](https://zenodo.org/records/3227177), not the 2,000-line excerpts. - `corpora.tsv` names each file, its md5 as Zenodo publishes it, and the files of the archive - the engine reads (Hadoop's 978 container logs, Spark's 3,852, OpenStack's three). HDFS is - `HDFS_1`. Thunderbird (2.0 GB compressed) and Windows (1.7 GB) are not measured. -- **Engine.** `log10x/pipeline-10x:1.1.133@sha256:fe3dcdefdc2f42fa7117c4f63fe97b28508fbf1356725422703f0dfab55f2360`, - with its default symbol library and no per-corpus compile. -- **Config.** `tenx-encode.config.yaml`, copied unchanged from - [`drain3-vs-log10x/`](../drain3-vs-log10x/) at commit `0fb1d10`, the config the post's - reproduce section names. It writes `encoded.log` (one encoded event per line) and - `templates.json` (one template per line). -- **Cold.** Each corpus starts with an empty template cache, so its whole dictionary is - charged to it. -- **Counts.** Input bytes and lines are summed over the files the engine read; events are the - lines of `encoded.log`, templates the lines of `templates.json`. Reduction is - 1 - (`encoded.log` + `templates.json`) / input. The container runs in UTC; timestamps inside - `encoded.log` follow the time zone, byte counts do not. +The inputs are the full LogHub datasets from Zenodo record 3227177, not the 2,000-line excerpts, each checked against the md5 Zenodo publishes. The released engine image ran in a Linux container with its default symbol library, one encode config and no per-dataset compile. Each dataset started with an empty template cache, so its whole dictionary is charged to it. The reduction is one minus the size of the two output files (`encoded.log` for the events, `templates.json` for the dictionary) over the input bytes the engine read. Both outputs are hashed so a rerun can be compared byte for byte. -`results/.json` keeps each corpus's counts and the sha256 of both output files; -`results/results.md` lists them. The outputs themselves are not kept here (Spark's alone is -1.2 GB); a rerun on the same image compares against the hashes. +## Limits -## Reproduce +- Storage on disk after gzip or ZSTD is not measured. A codec compresses both the original and the compact form. +- Query cost and a comparison with another tool's output are not measured. +- The figures are for these 14 datasets. HDFS is the HDFS_1 set, and the Thunderbird and Windows datasets were not measured. + +## Run it ``` -./run.sh # all 14, then the report and its check -./run.sh Apache Linux # some corpora -./run.sh report # re-render from results/ +cd loghub-compaction +python3 report.py --check # recompute every figure from results/ (no Docker) +./run.sh # all 14 datasets, then the report and its check +./run.sh Apache Linux # some datasets ``` -Needs Docker, Python 3 and about 5 GB free at the peak (Spark). Each archive is downloaded -once into `data/tar/` and checked against Zenodo's md5; each corpus's extracted input and -outputs are deleted once hashed (`LC_KEEP=1` keeps them). `report.py --check` recomputes -`results/results.json` and the table above from `results/.json` and fails on any -difference; CI runs it. - -## What is claimed and what is not - -- The reduction of the compact form against the original bytes, per corpus and in total, on - these files, with this engine and library. -- Not storage after a codec, which compresses both sides; not query cost; not a comparison - with another tool's output. - -## Files +A full run needs Docker, Python 3 and about 5 GB of free disk at the peak (Spark). The script downloads the archives from Zenodo and pins the engine image log10x/pipeline-10x:1.1.133 by digest. -| File | What it is | -|---|---| -| `run.sh` | the whole run, per corpus | -| `corpora.tsv` | each corpus's Zenodo file, md5 and the files the engine reads | -| `tenx-encode.config.yaml` | the encode config, unchanged from `drain3-vs-log10x/` at `0fb1d10` | -| `measure.py` | counts one corpus's input and outputs and hashes the outputs | -| `report.py` | renders `results/results.json`, `results/results.md` and the table above; `--check` fails on drift | -| `ledger.json` | the figures the compaction post carries, from the 2026-10-05 measurement | -| `results/.json` | each corpus's counts and output hashes | +The full record is in [RECORD.md](RECORD.md): every dataset's events, bytes, ratio and template count set against the post's figures, the full method and the file list. diff --git a/loghub-compaction/RECORD.md b/loghub-compaction/RECORD.md new file mode 100644 index 0000000..b907fa3 --- /dev/null +++ b/loghub-compaction/RECORD.md @@ -0,0 +1,100 @@ +# LogHub compaction + +One benchmark in the [log10x benchmarks](../) repository. It answers one question: how much +smaller is the engine's compact form than the original logs, on the 14 full public LogHub +datasets, with the engine and library as they ship. + +The compact form is the encoded events plus the template dictionary needed to expand them +back. The reduction is measured before any codec (gzip, ZSTD), so it stacks with whatever +the storage layer applies after. + +## The result + +Engine `log10x/pipeline-10x:1.1.133` by digest, the 1.1.89 symbol library it ships, each +corpus cold: + + +| Corpus | Events | Input | Compact (encoded + dictionary) | Reduction | Ratio | Templates | Against the ledger | +|---|---:|---:|---:|---:|---:|---:|---| +| Hadoop | 186,697 | 48,595,595 B | 20,212,363 B | 58.4% | 2.4x | 931 | events 186,697 against 186,695; compact bytes 20,212,363 against 20,198,860; templates 931 against 888 | +| Spark | 32,952,127 | 2,941,224,304 B | 1,248,606,093 B | 57.5% | 2.36x | 3,370 | events 32,952,127 against 32,952,122; compact bytes 1,248,606,093 against 1,248,618,288; templates 3,370 against 3,377 | +| Apache | 52,004 | 5,135,876 B | 2,295,929 B | 55.3% | 2.24x | 371 | equal | +| OpenSSH | 655,147 | 73,417,506 B | 36,854,297 B | 49.8% | 1.99x | 3,032 | equal | +| Proxifier | 21,329 | 2,541,814 B | 1,353,818 B | 46.7% | 1.88x | 1,854 | equal | +| Zookeeper | 74,380 | 10,426,438 B | 5,673,286 B | 45.6% | 1.84x | 157 | equal | +| Linux | 25,567 | 2,349,686 B | 1,303,867 B | 44.5% | 1.8x | 1,262 | compact bytes 1,303,867 against 1,304,303; templates 1,262 against 1,266 | +| HPC | 330,461 | 33,553,503 B | 20,485,336 B | 38.9% | 1.64x | 2,045 | equal | +| BGL | 4,728,216 | 743,185,031 B | 482,351,424 B | 35.1% | 1.54x | 136,461 | equal | +| Mac | 107,220 | 16,879,552 B | 11,391,093 B | 32.5% | 1.48x | 2,466 | equal | +| OpenStack | 207,820 | 61,442,082 B | 42,663,820 B | 30.6% | 1.44x | 8,134 | compact bytes 42,663,820 against 42,663,612; templates 8,134 against 8,133 | +| HDFS | 11,175,629 | 1,577,982,906 B | 1,118,008,927 B | 29.1% | 1.41x | 279 | equal | +| Android | 1,555,005 | 192,270,829 B | 153,830,847 B | 20% | 1.25x | 29,400 | equal | +| HealthApp | 204,648 | 23,529,930 B | 22,432,794 B | 4.7% | 1.05x | 1,341 | equal | + +**All 14:** 52,276,250 events, 5,732,535,052 B in, 3,167,463,894 B compact, byte-weighted reduction 44.7%. The 4 over 100 MB (Spark, BGL, HDFS, Android): 44.9%. + + +The last column sets each corpus against the measurement the compaction post's figures come +from (the ledger, `ledger.json`). That run used a local build of the same engine version on +macOS; this one uses the released image in a Linux container. Ten of the fourteen reproduce +it byte for byte, and every reduction holds to the decimal place. Where a corpus is many files (Hadoop, Spark, OpenStack) the engine reads +them in directory order, which differs between the two file systems, so multi-line grouping +and the order templates are first seen move slightly. Linux's template count varies from run to run on any setup, because two-digit days +match both `MMM d` and `MMM dd` and the format is chosen per event; its reduction does not. + +## Method + +- **Inputs.** The full datasets of [LogHub](https://github.com/logpai/loghub) from + [Zenodo record 3227177](https://zenodo.org/records/3227177), not the 2,000-line excerpts. + `corpora.tsv` names each file, its md5 as Zenodo publishes it, and the files of the archive + the engine reads (Hadoop's 978 container logs, Spark's 3,852, OpenStack's three). HDFS is + `HDFS_1`. Thunderbird (2.0 GB compressed) and Windows (1.7 GB) are not measured. +- **Engine.** `log10x/pipeline-10x:1.1.133@sha256:fe3dcdefdc2f42fa7117c4f63fe97b28508fbf1356725422703f0dfab55f2360`, + with its default symbol library and no per-corpus compile. +- **Config.** `tenx-encode.config.yaml`, copied unchanged from + [`drain3-vs-log10x/`](../drain3-vs-log10x/) at commit `0fb1d10`, the config the post's + reproduce section names. It writes `encoded.log` (one encoded event per line) and + `templates.json` (one template per line). +- **Cold.** Each corpus starts with an empty template cache, so its whole dictionary is + charged to it. +- **Counts.** Input bytes and lines are summed over the files the engine read; events are the + lines of `encoded.log`, templates the lines of `templates.json`. Reduction is + 1 - (`encoded.log` + `templates.json`) / input. The container runs in UTC; timestamps inside + `encoded.log` follow the time zone, byte counts do not. + +`results/.json` keeps each corpus's counts and the sha256 of both output files; +`results/results.md` lists them. The outputs themselves are not kept here (Spark's alone is +1.2 GB); a rerun on the same image compares against the hashes. + +## Reproduce + +``` +./run.sh # all 14, then the report and its check +./run.sh Apache Linux # some corpora +./run.sh report # re-render from results/ +``` + +Needs Docker, Python 3 and about 5 GB free at the peak (Spark). Each archive is downloaded +once into `data/tar/` and checked against Zenodo's md5; each corpus's extracted input and +outputs are deleted once hashed (`LC_KEEP=1` keeps them). `report.py --check` recomputes +`results/results.json` and the table above from `results/.json` and fails on any +difference; CI runs it. + +## What is claimed and what is not + +- The reduction of the compact form against the original bytes, per corpus and in total, on + these files, with this engine and library. +- Not storage after a codec, which compresses both sides; not query cost; not a comparison + with another tool's output. + +## Files + +| File | What it is | +|---|---| +| `run.sh` | the whole run, per corpus | +| `corpora.tsv` | each corpus's Zenodo file, md5 and the files the engine reads | +| `tenx-encode.config.yaml` | the encode config, unchanged from `drain3-vs-log10x/` at `0fb1d10` | +| `measure.py` | counts one corpus's input and outputs and hashes the outputs | +| `report.py` | renders `results/results.json`, `results/results.md` and the table above; `--check` fails on drift | +| `ledger.json` | the figures the compaction post carries, from the 2026-10-05 measurement | +| `results/.json` | each corpus's counts and output hashes | diff --git a/loghub-compaction/report.py b/loghub-compaction/report.py index 6815341..4880679 100644 --- a/loghub-compaction/report.py +++ b/loghub-compaction/report.py @@ -1,9 +1,9 @@ #!/usr/bin/env python3 """ -Render results/results.json, results/results.md and the README's table from the per-corpus +Render results/results.json, results/results.md and the RECORD.md table from the per-corpus records (results/.json) and set each against the compaction ledger (ledger.json). -`--check` renders them again and fails if the committed results.json or the README's +`--check` renders them again and fails if the committed results.json or the RECORD.md generated table differ, so a figure edited without a rerun is caught in CI. """ import argparse @@ -14,7 +14,7 @@ HERE = pathlib.Path(__file__).resolve().parent -# README blocks: the text between `` and `` is +# RECORD.md blocks: the text between `` and `` is # written from the results and compared by --check. BLOCK = re.compile(r'\n(.*?)', re.S) @@ -22,7 +22,7 @@ def apply_blocks(text: str, blocks: dict) -> str: missing = sorted(set(blocks) - {m.group(1) for m in BLOCK.finditer(text)}) if missing: - raise SystemExit(f'README.md has no generated block for: {", ".join(missing)}') + raise SystemExit(f'RECORD.md has no generated block for: {", ".join(missing)}') return BLOCK.sub(lambda m: f'\n{blocks[m.group(1)].rstrip()}\n' if m.group(1) in blocks else m.group(0), text) @@ -126,18 +126,18 @@ def main() -> int: committed = json.load(open(HERE / 'results' / 'results.json')) if committed != r: raise SystemExit('results.json does not follow from the per-corpus records') - stale = differing_blocks((HERE / 'README.md').read_text(), blocks) + stale = differing_blocks((HERE / 'RECORD.md').read_text(), blocks) if stale: - raise SystemExit(f'README.md differs from the results in: {", ".join(stale)}; run report.py to rewrite them') + raise SystemExit(f'RECORD.md differs from the results in: {", ".join(stale)}; run report.py to rewrite them') print(f'the committed figures follow from the per-corpus records: {r["aggregate"]["events"]:,} events, ' f'byte-weighted reduction {pct(r["aggregate"]["reduction_pct"])}') return 0 json.dump(r, open(HERE / 'results' / 'results.json', 'w'), indent=1) (HERE / 'results' / 'results.md').write_text(render(r)) - readme = HERE / 'README.md' + readme = HERE / 'RECORD.md' if readme.exists(): readme.write_text(apply_blocks(readme.read_text(), blocks)) - print('wrote results/results.json, results/results.md and the README table') + print('wrote results/results.json, results/results.md and the RECORD.md table') return 0 diff --git a/loghub-compaction/run.sh b/loghub-compaction/run.sh index 82e85fc..3905ada 100755 --- a/loghub-compaction/run.sh +++ b/loghub-compaction/run.sh @@ -6,7 +6,7 @@ # # ./run.sh every corpus in corpora.tsv # ./run.sh Apache Linux some corpora -# ./run.sh report render results.json, results.md and the README table +# ./run.sh report render results.json, results.md and the RECORD.md table # # Environment: # LC_DATA_DIR tarball cache and scratch (default: /data); about 4.5 GB free at peak (Spark) diff --git a/naming-stability/README.md b/naming-stability/README.md index cc62a92..dfd941c 100644 --- a/naming-stability/README.md +++ b/naming-stability/README.md @@ -1,121 +1,44 @@ # Naming stability across releases -One benchmark in the [log10x benchmarks](../) repository. It answers one question: when the -engine moves to its next release, how many events change their pattern name, pattern hash or -template hash, with the configuration and symbol library each release ships? +When the 10x engine moves to its next release, do the same log events keep the same pattern names? Across releases 1.1.140, 1.1.141 and 1.1.142, on two captures of the OpenTelemetry demo, no event changed its pattern name, its pattern hash or its template hash. -This is a folder of its own rather than a section of [`../intent`](../intent). The intent -benchmark scores grouping against source-statement labels, and a rename that keeps every -statement in one pattern leaves its scores unchanged. A rename rate is a property of two -releases and needs no labels, so it is measured here, and each new release adds two rows. +Each event gets a pattern name, a pattern hash (`tenx_hash`, the hash of the name) and a template hash. Each release ran with the configuration and symbol library its own image ships. -## The result +## Result -Engine releases 1.1.140, 1.1.141 and 1.1.142 on the two OpenTelemetry demo captures of -`../intent`, each release with its own image's configuration and library: +| Capture | Releases | Events | Events changed | Patterns renamed | +|---|---|---:|---:|---:| +| rec1155 | 1.1.140 to 1.1.141 | 41,573 | 0 | 0 of 582 | +| rec1155 | 1.1.141 to 1.1.142 | 41,573 | 0 | 0 of 582 | +| otel215 | 1.1.140 to 1.1.141 | 157,228 | 0 | 0 of 2,687 | +| otel215 | 1.1.141 to 1.1.142 | 157,228 | 0 | 0 of 2,687 | -| capture | releases | events | pattern name changed | pattern hash changed | template hash changed | patterns renamed | -|---|---|---:|---:|---:|---:|---:| -| rec1155 | 1.1.140 to 1.1.141 | 41,573 | 0 | 0 | 0 | 0 of 582 | -| rec1155 | 1.1.141 to 1.1.142 | 41,573 | 0 | 0 | 0 | 0 of 582 | -| otel215 | 1.1.140 to 1.1.141 | 157,228 | 0 | 0 | 0 | 0 of 2,687 | -| otel215 | 1.1.141 to 1.1.142 | 157,228 | 0 | 0 | 0 | 0 of 2,687 | +Each pair of releases cut each capture into the same events, so events were compared one for one. An event counts as changed when any of its three names or hashes differs. -Each pair cut each capture into the same events, so events are compared one for one. No event -changed its pattern name, its pattern hash (`tenx_hash`, the hash of the name) or its template -hash, so no pattern is listed as changed ([`results/results.md`](results/results.md)). +- The three images carry the same 1.1.89 symbol library. In the shipped configuration, 18 files differ between 1.1.140 and 1.1.141 and none between 1.1.141 and 1.1.142. +- In 1.1.141, a line opening with the ANSI dim code starts its own event. Neither capture has such a line, so that change had nothing to act on here. +- The intent benchmark reports fewer patterns for the same 1.1.142 image (2,277 on otel215, 406 on rec1155) because it counts only the events it has source labels for. -These pattern counts take every event: 2,687 on otel215 and 582 on rec1155 for 1.1.142. -[`../intent`](../intent) counts the patterns of labelled events only and reports 2,277 on -otel215 and 406 on rec1155 for the same 1.1.142 image. The counts differ by 410 on otel215 and -176 on rec1155. The difference is the patterns that only unlabelled events take. The two runs -cut each capture into the same events and give every event the same pattern name. On otel215 -the 665 unlabelled events take 411 patterns, and one of them, `Starting`, also names a labelled -event, so they add 410. On rec1155 the 240 unlabelled events take 176 patterns that no labelled -event shares, so they add 176. In frontend-proxy, which has no labelled events, each unlabelled -event takes a pattern of its own: 114 on otel215 and 115 on rec1155, most of them lines of -Envoy's startup listing. The next largest are opensearch with 83 patterns, jaeger 58 and -prometheus-server 38 on otel215, and postgresql 23 and flagd 13 on rec1155. -[`../intent/bench/pattern_counts.py`](../intent/bench/pattern_counts.py) computes these numbers -from the two runs' events ([otel215](../intent/results/pattern-counts-1.1.142_otel215.json), -[rec1155](../intent/results/pattern-counts-1.1.142_rec1155.json)). +## How it was measured -## What changed in each release, and what these captures exercise - -The three images carry the same symbol library files, the 1.1.89 library -([`results/library.tsv`](results/library.tsv)). The changes are in the engine and in the -shipped configuration. - -- **1.1.141.** Its [release notes](https://github.com/log-10x/pipeline-releases/releases/tag/1.1.141) - say the engine is unchanged from 1.1.140 and the configuration and modules change. 18 files - of the shipped configuration differ, among them the group, template and rate modules - ([`results/config-diff.txt`](results/config-diff.txt)). The one change in those notes that - touches how events are cut or named is that a line opening with the ANSI dim code, as Rust `tracing` prints a coloured timestamp, - now starts its own event. Neither capture holds such a line (0 records in each), so that - change had nothing to act on here. The same notes report the intent benchmark's merged and - split counts unchanged on these two captures. -- **1.1.142.** Its [release notes](https://github.com/log-10x/pipeline-releases/releases/tag/1.1.142) - list two engine changes; no file of the shipped configuration differs from 1.1.141. The - group flush timer now counts per source, which the notes describe on the forwarder input, - where batches arrive over time; these runs read each capture as a file. A timestamp - fraction that starts with `1000` now keeps its second, which the notes report on 3 of - 67,070 events of other public logs; no event here changed. - -So on these captures, read as files, neither release changed a name, and neither capture -exercises the ANSI case or a forwarder connection's timing. +The two captures are the ones the [intent benchmark](../intent) uses, each checked by SHA-256. Each release's engine image was pinned by digest and run on each capture as a file, writing every event's pattern name, both hashes and its text. A script compared consecutive releases event by event. A pattern of the earlier release counts as renamed when any of its events carries another name in the later one. The runs were made on 2026-10-15. ## Limits -- **Two captures of one application.** Both are the OpenTelemetry demo (versions 2.2.0 and - 2.1.3). Another application, or the ANSI-coloured lines 1.1.141 regroups, can rename where - these do not. -- **File input only.** The 1.1.142 flush change is described for the forwarder input; a run - through a forwarder is not measured here. -- **Three releases.** Releases before 1.1.140 are not compared. In `../intent`, 1.1.132 and - 1.1.133 group otel215 differently (6,944 merged events against 13), so names changed - between those two. Adding an image to `RELEASES` in `run.sh` adds its pair. -- **A library change is not measured.** All three images carry the same library files. A - new library is where names built from library words would move, and none shipped between - these releases. -- **No compatibility policy is stated here.** This folder measures what happened between - releases; it does not promise what future releases will do. +- Both captures come from one application, the OpenTelemetry demo (versions 2.2.0 and 2.1.3). Another application, or the ANSI-colored lines that 1.1.141 regroups, can rename where these do not. +- The engine read each capture as a file. The 1.1.142 release notes describe a flush timer change on the forwarder input, which is not measured here. +- Only three releases are compared. Names did change between two earlier ones: on otel215 the intent benchmark counts 6,944 merged events for 1.1.132 and 13 for 1.1.133. +- A library change is not measured, since all three images carry the same one. The page makes no promise about future releases. -## Method +## Run it -- **Captures.** `otel215` (the `otel-sample-200mb.log` asset of `log-10x/config` release - `otel-sample-v1`) and `rec1155` (this repository's `intent-data-v1` release), checked by - sha256 as `../intent` checks them. -- **Engines.** Each release's `log10x/pipeline-10x` image by digest (in `run.sh`), the dev app - (`@apps/dev`) with the image's shipped configuration, and an output module - (`bench/events.config.yaml`) that writes each event's `message_pattern`, `tenx_hash`, - `templateHash` and text. `bench/engine.sh` mounts an empty directory over the sample input - the image ships, so the engine reads the capture alone. -- **What each image ships.** `run.sh` copies the symbol library and the configuration tree - out of each image and records their SHA-256 (`results/library.tsv`, - `results/config-.sha256`, `results/config-diff.txt`). -- **Comparison.** `bench/compare.py` compares consecutive releases event by event when both - cut the capture into the same events, and record by record otherwise. A pattern of the - earlier release counts as renamed when any of its events carries another name in the later - one. - -The runs were made on 2026-10-15 with no other container running on the host. - -## Reproduce +From this directory: ``` -NS_PRUNE=1 ./run.sh +python3 bench/compare.py --render | diff - results/results.md # check the table against compare.json +NS_PRUNE=1 ./run.sh # full rerun: downloads both captures, pulls the three images ``` -Needs Docker, curl and Python 3. `run.sh` downloads the two captures (or reuses them from -`NS_CACHE`), pulls each image, records what it ships, runs it on both captures, and writes -`results/`. `NS_PRUNE=1` removes each image the script pulled once its runs are done, so one -release image is on disk at a time. On an Intel macOS laptop with an idle host each release's -runs took one to three minutes for both captures, plus the image pull. `python3 bench/compare.py --render` rebuilds -`results/results.md` from the committed `results/compare.json`; CI checks that the two agree. - -## Files +The full rerun needs Docker, curl and Python 3, and took one to three minutes per release on an Intel macOS laptop, plus the image pulls. -- `run.sh`, `bench/engine.sh`, `bench/events.config.yaml`: run each release over each capture. -- `bench/compare.py`: the comparison and `results/results.md`. -- `results/`: `compare.json`, `results.md`, `library.tsv`, `config-.sha256`, - `config-diff.txt`. +The complete record is in [RECORD.md](RECORD.md): the changes each release's notes list, how the pattern counts reconcile with the intent benchmark, the full method and the files. diff --git a/naming-stability/RECORD.md b/naming-stability/RECORD.md new file mode 100644 index 0000000..cc62a92 --- /dev/null +++ b/naming-stability/RECORD.md @@ -0,0 +1,121 @@ +# Naming stability across releases + +One benchmark in the [log10x benchmarks](../) repository. It answers one question: when the +engine moves to its next release, how many events change their pattern name, pattern hash or +template hash, with the configuration and symbol library each release ships? + +This is a folder of its own rather than a section of [`../intent`](../intent). The intent +benchmark scores grouping against source-statement labels, and a rename that keeps every +statement in one pattern leaves its scores unchanged. A rename rate is a property of two +releases and needs no labels, so it is measured here, and each new release adds two rows. + +## The result + +Engine releases 1.1.140, 1.1.141 and 1.1.142 on the two OpenTelemetry demo captures of +`../intent`, each release with its own image's configuration and library: + +| capture | releases | events | pattern name changed | pattern hash changed | template hash changed | patterns renamed | +|---|---|---:|---:|---:|---:|---:| +| rec1155 | 1.1.140 to 1.1.141 | 41,573 | 0 | 0 | 0 | 0 of 582 | +| rec1155 | 1.1.141 to 1.1.142 | 41,573 | 0 | 0 | 0 | 0 of 582 | +| otel215 | 1.1.140 to 1.1.141 | 157,228 | 0 | 0 | 0 | 0 of 2,687 | +| otel215 | 1.1.141 to 1.1.142 | 157,228 | 0 | 0 | 0 | 0 of 2,687 | + +Each pair cut each capture into the same events, so events are compared one for one. No event +changed its pattern name, its pattern hash (`tenx_hash`, the hash of the name) or its template +hash, so no pattern is listed as changed ([`results/results.md`](results/results.md)). + +These pattern counts take every event: 2,687 on otel215 and 582 on rec1155 for 1.1.142. +[`../intent`](../intent) counts the patterns of labelled events only and reports 2,277 on +otel215 and 406 on rec1155 for the same 1.1.142 image. The counts differ by 410 on otel215 and +176 on rec1155. The difference is the patterns that only unlabelled events take. The two runs +cut each capture into the same events and give every event the same pattern name. On otel215 +the 665 unlabelled events take 411 patterns, and one of them, `Starting`, also names a labelled +event, so they add 410. On rec1155 the 240 unlabelled events take 176 patterns that no labelled +event shares, so they add 176. In frontend-proxy, which has no labelled events, each unlabelled +event takes a pattern of its own: 114 on otel215 and 115 on rec1155, most of them lines of +Envoy's startup listing. The next largest are opensearch with 83 patterns, jaeger 58 and +prometheus-server 38 on otel215, and postgresql 23 and flagd 13 on rec1155. +[`../intent/bench/pattern_counts.py`](../intent/bench/pattern_counts.py) computes these numbers +from the two runs' events ([otel215](../intent/results/pattern-counts-1.1.142_otel215.json), +[rec1155](../intent/results/pattern-counts-1.1.142_rec1155.json)). + +## What changed in each release, and what these captures exercise + +The three images carry the same symbol library files, the 1.1.89 library +([`results/library.tsv`](results/library.tsv)). The changes are in the engine and in the +shipped configuration. + +- **1.1.141.** Its [release notes](https://github.com/log-10x/pipeline-releases/releases/tag/1.1.141) + say the engine is unchanged from 1.1.140 and the configuration and modules change. 18 files + of the shipped configuration differ, among them the group, template and rate modules + ([`results/config-diff.txt`](results/config-diff.txt)). The one change in those notes that + touches how events are cut or named is that a line opening with the ANSI dim code, as Rust `tracing` prints a coloured timestamp, + now starts its own event. Neither capture holds such a line (0 records in each), so that + change had nothing to act on here. The same notes report the intent benchmark's merged and + split counts unchanged on these two captures. +- **1.1.142.** Its [release notes](https://github.com/log-10x/pipeline-releases/releases/tag/1.1.142) + list two engine changes; no file of the shipped configuration differs from 1.1.141. The + group flush timer now counts per source, which the notes describe on the forwarder input, + where batches arrive over time; these runs read each capture as a file. A timestamp + fraction that starts with `1000` now keeps its second, which the notes report on 3 of + 67,070 events of other public logs; no event here changed. + +So on these captures, read as files, neither release changed a name, and neither capture +exercises the ANSI case or a forwarder connection's timing. + +## Limits + +- **Two captures of one application.** Both are the OpenTelemetry demo (versions 2.2.0 and + 2.1.3). Another application, or the ANSI-coloured lines 1.1.141 regroups, can rename where + these do not. +- **File input only.** The 1.1.142 flush change is described for the forwarder input; a run + through a forwarder is not measured here. +- **Three releases.** Releases before 1.1.140 are not compared. In `../intent`, 1.1.132 and + 1.1.133 group otel215 differently (6,944 merged events against 13), so names changed + between those two. Adding an image to `RELEASES` in `run.sh` adds its pair. +- **A library change is not measured.** All three images carry the same library files. A + new library is where names built from library words would move, and none shipped between + these releases. +- **No compatibility policy is stated here.** This folder measures what happened between + releases; it does not promise what future releases will do. + +## Method + +- **Captures.** `otel215` (the `otel-sample-200mb.log` asset of `log-10x/config` release + `otel-sample-v1`) and `rec1155` (this repository's `intent-data-v1` release), checked by + sha256 as `../intent` checks them. +- **Engines.** Each release's `log10x/pipeline-10x` image by digest (in `run.sh`), the dev app + (`@apps/dev`) with the image's shipped configuration, and an output module + (`bench/events.config.yaml`) that writes each event's `message_pattern`, `tenx_hash`, + `templateHash` and text. `bench/engine.sh` mounts an empty directory over the sample input + the image ships, so the engine reads the capture alone. +- **What each image ships.** `run.sh` copies the symbol library and the configuration tree + out of each image and records their SHA-256 (`results/library.tsv`, + `results/config-.sha256`, `results/config-diff.txt`). +- **Comparison.** `bench/compare.py` compares consecutive releases event by event when both + cut the capture into the same events, and record by record otherwise. A pattern of the + earlier release counts as renamed when any of its events carries another name in the later + one. + +The runs were made on 2026-10-15 with no other container running on the host. + +## Reproduce + +``` +NS_PRUNE=1 ./run.sh +``` + +Needs Docker, curl and Python 3. `run.sh` downloads the two captures (or reuses them from +`NS_CACHE`), pulls each image, records what it ships, runs it on both captures, and writes +`results/`. `NS_PRUNE=1` removes each image the script pulled once its runs are done, so one +release image is on disk at a time. On an Intel macOS laptop with an idle host each release's +runs took one to three minutes for both captures, plus the image pull. `python3 bench/compare.py --render` rebuilds +`results/results.md` from the committed `results/compare.json`; CI checks that the two agree. + +## Files + +- `run.sh`, `bench/engine.sh`, `bench/events.config.yaml`: run each release over each capture. +- `bench/compare.py`: the comparison and `results/results.md`. +- `results/`: `compare.json`, `results.md`, `library.tsv`, `config-.sha256`, + `config-diff.txt`. diff --git a/otel-denominators/README.md b/otel-denominators/README.md index 5c5a3ce..d9122a8 100644 --- a/otel-denominators/README.md +++ b/otel-denominators/README.md @@ -1,126 +1,41 @@ # OTel sample on three denominators -One benchmark in the [log10x benchmarks](../) repository. It answers one question: -when a reduction figure is quoted for a Kubernetes log stream, what was in the denominator, -and how much does the answer move when you change it. +How much does a log reduction figure depend on what was in the input? On one public capture from a Kubernetes cluster running the OpenTelemetry demo, the 10x engine's figure is 64.00% on the file as captured, 62.46% with the debug output and an added field removed, and 72.35% on the application messages alone. Taking away the Kubernetes metadata around each message raised the figure. -## The question +The capture wraps each container's stdout line in a Fluentd envelope of Kubernetes and Docker metadata. Two objections follow from that. One says the file is padded by a `tenx_tag` field the log10x forwarder adds and by the demo collector's debug exporter. The other says OTLP has no such envelope, so the figure should collapse without it. -The capture as published compacts by **64.00%**. The capture is a Fluentd envelope around -each container's stdout line, so most of its bytes are Kubernetes and Docker metadata rather -than the application's message. Two objections follow, and they point in opposite directions. +## Result -- *You are measuring your own metadata.* The capture carries a `tenx_tag` field this project - injects, and about half its bytes are the demo collector's debug exporter printing every - telemetry batch it handles. -- *You are measuring an envelope, not logs.* An OpenTelemetry practitioner compares this - against OTLP, which carries no per-record Fluentd envelope, and expects the figure to - collapse once the envelope is removed. +| Input | Input bytes | Lines | Compact bytes | Reduction | +|---|---:|---:|---:|---:| +| (a) as captured | 215,039,161 | 197,430 | 77,421,244 | 64.00% | +| (b) tag and debug lines removed | 88,975,911 | 111,691 | 33,401,403 | 62.46% | +| (c) message only | 40,551,755 | 197,430 | 11,212,254 | 72.35% | -Rather than argue either one, this benchmark builds all three denominators from the same -public file and measures them with the same engine, the same configs and the same round-trip -check. +On the message text alone the reduction is 8.35 points higher than on the file as captured, because repeated text is a larger share of an application message than of the metadata around it. Removing the added field and 85,739 collector debug lines took 58.6% of the bytes off the input and moved the figure down by 1.54 points. -## The result +- Every case decoded back to its own input byte for byte. +- An OTLP record carries more than a bare message and much less than this envelope, so (b) and (c) bracket it at 62.46% to 72.35%. -| Case | Input bytes | Lines | Encoded | Templates bytes | Compact | Reduction | Templates | Round trip | -|---|---:|---:|---:|---:|---:|---:|---:|---| -| (a) as captured | 215,039,161 | 197,430 | 74,142,102 | 3,279,142 | 77,421,244 | **64.00%** | 3,028 | byte-identical | -| (b) tag and debug lines removed | 88,975,911 | 111,691 | 30,452,359 | 2,949,044 | 33,401,403 | **62.46%** | 2,978 | byte-identical | -| (c) message only | 40,551,755 | 197,430 | 10,610,578 | 601,676 | 11,212,254 | **72.35%** | 2,625 | byte-identical | +## How it was measured -Reduction is the compact form, the encoded events plus the template dictionary needed to -expand them, against the input, with no gzip or ZSTD on either side. Each row decodes back to -its own input and is compared with `cmp`, so no row borrows another row's round trip. +The input is `otel-sample-200mb.log` from the public [otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1) release, captured through Fluentd from 36 containers over about ten hours. Input (b) removes the added field and the collector's debug exporter lines, and keeps the envelope because a log platform bills it. Input (c) is the message value of each record, one per line. All three ran through the same engine and configs, and each decode was compared with its own input. Compact bytes are the encoded events plus the template dictionary, with no gzip or ZSTD on either side. -**The second objection does not survive the measurement.** On the message text alone, with the -whole envelope gone, the reduction is 8.35 points **higher** than the figure as captured, not -lower. Repeated text is a larger share of an application message than of the Kubernetes -metadata around it, so removing the envelope removes the harder part of the file. +## Limits -**The first objection moves the figure by 1.54 points.** Removing the injected field and -dropping 85,739 collector debug lines, which together take 126,063,250 bytes off the input, -58.6% of it, moves 64.00% to 62.46%. +- One capture from one small demo cluster at roughly 5 lines per second. +- No OTLP capture was measured. The OTLP range above is inferred from two measured points. +- Nothing here says what a particular platform bills. +- Input (b) removes two unrelated things at once and serves as a sensitivity check. Neither removal is a property of the engine. -An OTLP log record carries more than a bare message, since it has a timestamp, severity, body, -attributes and trace ids, and much less than this Fluentd envelope. Rows (b) and (c) therefore -bracket it at **62.46% to 72.35%**. No figure for an OTLP wire is measured here, and none is -claimed. Measuring one needs a capture of OTLP records, which this is not. +## Run it -## What each denominator is - -- **(a) as captured** is the release asset, untouched: one JSON record per line with `log`, - `stream`, `docker` and `kubernetes` keys, plus the `tenx_tag` field the forwarder adds. -- **(b) tag and debug lines removed** removes `tenx_tag` and drops the collector's debug-exporter lines, - keeping the Kubernetes and Docker envelope, because a log platform bills that envelope. A - line is dropped only when its container is `opentelemetry-collector` **and** its message - matches zap's tab-separated debug prefix `\tinfo\t(Traces|Logs|Metrics)\t`. Ordinary - collector error logs are kept, including the 3,745 lines reading `failed to upload metrics`: - those are a misconfiguration in the captured cluster, and a platform bills them. -- **(c) message only** is the `log` value alone, one message per line, envelope discarded. No - record's `log` value contains a newline, 0 of 197,430, so the line count is unambiguous. - -(b) and (c) differ in line count on purpose. (c) keeps every message including the collector's; -only (b) drops them. - -## What is not claimed - -- **No figure for the OpenTelemetry wire.** Both this capture and the numbers above are Fluentd - envelopes. The bracket in the result section is arithmetic on two measured points, not a - measurement. -- **No claim about what a particular platform bills.** Metering rules differ per product and - per contract. -- **One capture, one cluster, about ten hours.** 36 containers from one small demo cluster at - roughly 5 lines per second. A shape that occurs once at 5 lines per second is a different - population from a shape that occurs once at 100,000. -- **(b) is a sensitivity check, not a production configuration.** It removes two unrelated - things at once. The forwarder that produced this capture adds `tenx_tag`; the demo enables - the debug exporter. Neither is a property of the engine being measured. - -## Reproduce - -```sh -./run.sh +``` +cd otel-denominators +python3 report.py --csv results/results.csv --out /tmp/results.md # recompute the table from the committed counts +./run.sh # full rerun of all three cases ``` -Needs Docker and about 1.2 GB of free disk. The script fetches the public capture once (8.4 MB -gzipped), checks its sha256 against the one this table was measured on, builds the two derived -inputs and checks their byte and line counts against the committed run, then encodes and -decodes all three cases. It writes `results/results.csv` and `results/results.md`, and deletes -the bulky per-case output unless `OTD_KEEP=1`. - -`./run.sh c_msg` runs one case. `OTD_DATA_DIR`, `OTD_IMAGE` and `OTD_KEEP` override the cache -location, the engine image and the output retention. - -| File | What it is | -|---|---| -| `run.sh` | fetch, build, encode, decode, compare, report | -| `build_inputs.py` | derives (b) and (c) from (a), and fails if the counts move | -| `report.py` | renders `results/results.md` from the CSV, recomputing the percentage from the byte counts | -| `results/results.csv` | the committed run, one row per case | -| `results/results.md` | the same run as a table, with the image digest and the input sha256 | - -Pinned: engine image `log10x/pipeline-10x@sha256:fccff37a6a41e52eb4df8a108d042b0176210842b7ed25f08b1419df3a37904f` -(`log10x/pipeline-10x:1.1.136`), and the two round-trip configs in -[`../drain3-vs-log10x/`](../drain3-vs-log10x/), which are the configs the compaction post -publishes. - -## Data - -`otel-sample-200mb.log` from [otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1), -215,039,161 bytes, 197,430 lines, sha256 -`aa79b9349a4123d88936fd064f5eccd9fd84f45a1fd03cff3f64c259715c2432`. Kubernetes container output -captured through Fluentd from a cluster running the OpenTelemetry demo 2.1.3, about ten hours -between the 1st and 99th percentile timestamp, 36 containers. The same asset the -[`pattern-identity/`](../pattern-identity/) benchmark runs on. - -### One line differs from the copy the blog post measured - -The post's manifest reports 215,039,166 input bytes for this dataset; the published asset is -215,039,161. The difference is a single line, 9731, whose message names an engine config path: -the released asset reads `@apps/regulator` and the copy behind the manifest reads -`@apps/edge/optimizer`, an older name for the same app. One token, five bytes, in one line of -197,430. +The recompute needs only Python. The full rerun needs Docker and about 1.2 GB of free disk, and runs engine image `log10x/pipeline-10x:1.1.136`, pinned by digest in the script. -Every figure in this benchmark is measured on the published asset with the pinned engine, so a -reader who downloads it reproduces this table rather than the manifest's byte counts. +The complete record is in [RECORD.md](RECORD.md): how each input is built, the full table with template counts, the image digest and the data's sha256. diff --git a/otel-denominators/RECORD.md b/otel-denominators/RECORD.md new file mode 100644 index 0000000..5c5a3ce --- /dev/null +++ b/otel-denominators/RECORD.md @@ -0,0 +1,126 @@ +# OTel sample on three denominators + +One benchmark in the [log10x benchmarks](../) repository. It answers one question: +when a reduction figure is quoted for a Kubernetes log stream, what was in the denominator, +and how much does the answer move when you change it. + +## The question + +The capture as published compacts by **64.00%**. The capture is a Fluentd envelope around +each container's stdout line, so most of its bytes are Kubernetes and Docker metadata rather +than the application's message. Two objections follow, and they point in opposite directions. + +- *You are measuring your own metadata.* The capture carries a `tenx_tag` field this project + injects, and about half its bytes are the demo collector's debug exporter printing every + telemetry batch it handles. +- *You are measuring an envelope, not logs.* An OpenTelemetry practitioner compares this + against OTLP, which carries no per-record Fluentd envelope, and expects the figure to + collapse once the envelope is removed. + +Rather than argue either one, this benchmark builds all three denominators from the same +public file and measures them with the same engine, the same configs and the same round-trip +check. + +## The result + +| Case | Input bytes | Lines | Encoded | Templates bytes | Compact | Reduction | Templates | Round trip | +|---|---:|---:|---:|---:|---:|---:|---:|---| +| (a) as captured | 215,039,161 | 197,430 | 74,142,102 | 3,279,142 | 77,421,244 | **64.00%** | 3,028 | byte-identical | +| (b) tag and debug lines removed | 88,975,911 | 111,691 | 30,452,359 | 2,949,044 | 33,401,403 | **62.46%** | 2,978 | byte-identical | +| (c) message only | 40,551,755 | 197,430 | 10,610,578 | 601,676 | 11,212,254 | **72.35%** | 2,625 | byte-identical | + +Reduction is the compact form, the encoded events plus the template dictionary needed to +expand them, against the input, with no gzip or ZSTD on either side. Each row decodes back to +its own input and is compared with `cmp`, so no row borrows another row's round trip. + +**The second objection does not survive the measurement.** On the message text alone, with the +whole envelope gone, the reduction is 8.35 points **higher** than the figure as captured, not +lower. Repeated text is a larger share of an application message than of the Kubernetes +metadata around it, so removing the envelope removes the harder part of the file. + +**The first objection moves the figure by 1.54 points.** Removing the injected field and +dropping 85,739 collector debug lines, which together take 126,063,250 bytes off the input, +58.6% of it, moves 64.00% to 62.46%. + +An OTLP log record carries more than a bare message, since it has a timestamp, severity, body, +attributes and trace ids, and much less than this Fluentd envelope. Rows (b) and (c) therefore +bracket it at **62.46% to 72.35%**. No figure for an OTLP wire is measured here, and none is +claimed. Measuring one needs a capture of OTLP records, which this is not. + +## What each denominator is + +- **(a) as captured** is the release asset, untouched: one JSON record per line with `log`, + `stream`, `docker` and `kubernetes` keys, plus the `tenx_tag` field the forwarder adds. +- **(b) tag and debug lines removed** removes `tenx_tag` and drops the collector's debug-exporter lines, + keeping the Kubernetes and Docker envelope, because a log platform bills that envelope. A + line is dropped only when its container is `opentelemetry-collector` **and** its message + matches zap's tab-separated debug prefix `\tinfo\t(Traces|Logs|Metrics)\t`. Ordinary + collector error logs are kept, including the 3,745 lines reading `failed to upload metrics`: + those are a misconfiguration in the captured cluster, and a platform bills them. +- **(c) message only** is the `log` value alone, one message per line, envelope discarded. No + record's `log` value contains a newline, 0 of 197,430, so the line count is unambiguous. + +(b) and (c) differ in line count on purpose. (c) keeps every message including the collector's; +only (b) drops them. + +## What is not claimed + +- **No figure for the OpenTelemetry wire.** Both this capture and the numbers above are Fluentd + envelopes. The bracket in the result section is arithmetic on two measured points, not a + measurement. +- **No claim about what a particular platform bills.** Metering rules differ per product and + per contract. +- **One capture, one cluster, about ten hours.** 36 containers from one small demo cluster at + roughly 5 lines per second. A shape that occurs once at 5 lines per second is a different + population from a shape that occurs once at 100,000. +- **(b) is a sensitivity check, not a production configuration.** It removes two unrelated + things at once. The forwarder that produced this capture adds `tenx_tag`; the demo enables + the debug exporter. Neither is a property of the engine being measured. + +## Reproduce + +```sh +./run.sh +``` + +Needs Docker and about 1.2 GB of free disk. The script fetches the public capture once (8.4 MB +gzipped), checks its sha256 against the one this table was measured on, builds the two derived +inputs and checks their byte and line counts against the committed run, then encodes and +decodes all three cases. It writes `results/results.csv` and `results/results.md`, and deletes +the bulky per-case output unless `OTD_KEEP=1`. + +`./run.sh c_msg` runs one case. `OTD_DATA_DIR`, `OTD_IMAGE` and `OTD_KEEP` override the cache +location, the engine image and the output retention. + +| File | What it is | +|---|---| +| `run.sh` | fetch, build, encode, decode, compare, report | +| `build_inputs.py` | derives (b) and (c) from (a), and fails if the counts move | +| `report.py` | renders `results/results.md` from the CSV, recomputing the percentage from the byte counts | +| `results/results.csv` | the committed run, one row per case | +| `results/results.md` | the same run as a table, with the image digest and the input sha256 | + +Pinned: engine image `log10x/pipeline-10x@sha256:fccff37a6a41e52eb4df8a108d042b0176210842b7ed25f08b1419df3a37904f` +(`log10x/pipeline-10x:1.1.136`), and the two round-trip configs in +[`../drain3-vs-log10x/`](../drain3-vs-log10x/), which are the configs the compaction post +publishes. + +## Data + +`otel-sample-200mb.log` from [otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1), +215,039,161 bytes, 197,430 lines, sha256 +`aa79b9349a4123d88936fd064f5eccd9fd84f45a1fd03cff3f64c259715c2432`. Kubernetes container output +captured through Fluentd from a cluster running the OpenTelemetry demo 2.1.3, about ten hours +between the 1st and 99th percentile timestamp, 36 containers. The same asset the +[`pattern-identity/`](../pattern-identity/) benchmark runs on. + +### One line differs from the copy the blog post measured + +The post's manifest reports 215,039,166 input bytes for this dataset; the published asset is +215,039,161. The difference is a single line, 9731, whose message names an engine config path: +the released asset reads `@apps/regulator` and the copy behind the manifest reads +`@apps/edge/optimizer`, an older name for the same app. One token, five bytes, in one line of +197,430. + +Every figure in this benchmark is measured on the published asset with the pinned engine, so a +reader who downloads it reproduces this table rather than the manifest's byte counts. diff --git a/pattern-identity/README.md b/pattern-identity/README.md index 2e9b9d0..35f4cec 100644 --- a/pattern-identity/README.md +++ b/pattern-identity/README.md @@ -1,207 +1,47 @@ # Pattern identity benchmark -One benchmark in the [log10x benchmarks](../) repository. Everything it needs is in this -directory: the scripts under `bench/`, the pinned dependencies, `run.sh`, and the artifacts -under `results/` that each script wrote. +Does a log line get the same pattern name in two places, and keep it over time? Sampling and suppression rules key on that name. Two Drain3 instances, each mining one half of the same stream, gave a line whose shape occurs 2 to 10 times the same template 0.6022 of the time. Two 10x engine instances fed the same halves gave the same pattern name to 299,997 of 300,000 probe lines. -## The question +A Drain3 template is the running intersection of every line in its cluster, so the template changes when a sibling line arrives. In one process, the same bytes replayed after two siblings got a new template, and Drain3 reported no cluster change. The 10x engine takes names from a compiled symbol library instead of mining them. -Does a log line keep the same name over time, and does it get the same name in two different places? +## Result -A pattern name is the key that sampling rules, suppression, dedup keys, cost attribution and new-pattern detection are built on. A template miner produces that name by mining the stream in front of it, so the name is a function of what that process happened to see and when. This benchmark measures what that costs, on public data, states what result would prove the log10x position wrong, and runs those tests instead of asserting them. Two of the pre-registered tests fired against log10x, and both stand as concessions on this page, in the sections where they were tested. +Share of probe lines both instances named the same, mean of 10 trials. -The Drain-side experiments run against one public sample: 197,430 Fluentd/Kubernetes lines from the OpenTelemetry demo, published at [otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1). Measured scale, from `results/data_provenance.txt`: 10.3 hours between the 1st and 99th percentile timestamp, about 5.3 lines per second, 36 Kubernetes containers, the busiest of them 47.5% of the lines. That is one small demo cluster over about ten hours, not a log platform, and a shape that occurs once at 5 lines per second is a different population from a shape that occurs once at 100,000. The engine-side coverage experiments additionally use two public LogHub log sets from Zenodo and a pinned engine Docker image whose digest is recorded in the artifact: `log10x/pipeline-10x:1.1.140`, the image `results/coverage.md` was produced on. Section 2's engine rows run `log10x/pipeline-10x:1.1.142`, digest in `results/engine_split.md`. Every measured number on this page is printed by a script in this directory; the wall-clock estimates under Reproduce are not, and the ledger expectations are external, recorded six weeks earlier as disclosed in section 4. - -## What Drain does well - -Drain3 is the incumbent for a reason, and the concessions come first. - -- **Its templates are stable under reordering.** The same 60,000 lines fed in natural order and shuffled produce template strings that agree 0.9860 of the time (`bench/identifier.py`). The published stability result reproduces here and stands. -- **It works from the stream alone.** There is no build step, no compiler pass, and no dependency on the emitting source. Pointed at an unfamiliar stream it produces useful templates immediately, which makes it the right default when the emitting code is out of reach. -- **Two instances that never exchanged state give a rare line the same name.** Through the production path, `add_log_message`, a line seen once is named identically by both instances 0.9859 of the time (`bench/identity.py`, strong arm). This was a pre-registered falsifier and it fired against log10x. The mechanism is in the Drain3 source: neither instance can generalize a shape it has one example of, so both mint the line's own literal text, and the two strings are equal. An earlier revision of this benchmark reported 0.0 for that row by probing with the read-only `match()` API, whose misses are a coverage property of a lookup call; that number described the probe, not the miner, and the corrected design below replaces it. -- **On frequent shapes it converges.** For shapes occurring over 1,000 times, two independent instances agree 1.0000 of the time, through either path. -- **Centralized query-time mining is out of scope of the split result.** The major SIEM platforms that mine patterns do it in one central pass at query time, over whatever the query returns, so no second instance exists to disagree. The two-forwarder experiment below indicts per-node mining and any design that persists a mined name, not those products as they ship today. Section 1 is the part that reaches a central miner, because a name moving inside one process needs only that one process. -- **Two claims made against it fail on this data, and the scripts say so.** Drain does not shatter stack frames into a long cluster tail here: 4,658 frame lines collapse to 25 shapes. And an exception count turned out to be expressible in Drain's output on this file, through clusters that fire exactly once per burst and never outside one, so that claim is withdrawn in `results/multiline.txt` where it was tested. - -The templates are good. What this benchmark measures is the identity attached to them. - -## 1. A name moves inside one process - -This is the exhibit everything else depends on. It needs no split, no fleet, and no download, it runs in under a second, and `run.sh` runs it first. - -One Drain3 instance, one cluster, four lines from the public sample: a Grafana provisioning line arrives and is named, two sibling outputs of the same statement arrive, and the first line is presented again byte for byte. - -``` -step 1 logger=provisioning.datasources t= level=info msg="inserting datasource from configuration" name=Jaeger uid=webstore-traces -step 4 logger=provisioning.datasources t= level=info msg="inserting datasource from configuration" <*> <*> -``` - -The bytes are identical. The name is not. Step 2 reports `cluster_template_changed`, so the rename is visible to a caller processing the sibling. Step 4, the replay of the original bytes, reports `change_type='none'`, so a pipeline that keys on the template, or on a hash of it, writes a second name for the first line while being told the cluster held steady. Both facts are asserted against drain3 0.9.11, and the script refuses to run under `python -O` so the asserts stay live (`bench/replay.py`, `results/replay.txt`). - -This applies to every deployment, including one central miner with a single model. Anything that persists a mined name across time inherits it: a sampling rule, a suppression rule, a dedup key, a cost attribution, a new-pattern panel that now has a new pattern to report. Hand-written alerts on fields are untouched; alerts keyed on a mined pattern id are not. This is also why the singleton agreement conceded above is narrower than it sounds: the name both instances agree on is the pre-generalization literal, the exact string this exhibit shows moving the moment a sibling arrives. - -## 2. Two instances, the production path - -Two Drain3 instances each receive one random half of the same stream, disjoint, which models two forwarders that never exchange state. The same probe sequence is then fed to both through `add_log_message` itself, in the same order, exactly as if two nodes with different histories started seeing the same traffic. Every probe gets an answer from both sides, because the production path mints a cluster on first sight; there is no coverage term and no `None` anywhere in this table. Full file, 10 trials, 30,000 probes per trial, seed pinned (`bench/identity.py`, `results/identity.md`). - -The probe pass itself is shared traffic, so the two states converge as it proceeds, which works in Drain's favour on net. The divergence reported is what survives that convergence. - -**Drain3 at its strongest honest configuration** (`strong`: drain3's own `examples/drain3.ini` masking block, `FilePersistence` carried across a mid-stream restart, fallback lookup): - -| line frequency | probes/trial | template agreement | stdev | cluster_id agreement | -|---|---:|---:|---:|---:| -| seen once | 136 | 0.9859 | 0.0085 | 0.0007 | -| 2 to 10 | 204 | **0.6022** | 0.0243 | 0.0005 | -| 11 to 100 | 589 | 0.9118 | 0.0158 | 0.0051 | -| 101 to 1,000 | 916 | 0.9788 | 0.0045 | 0.0070 | -| over 1,000 | 28,152 | 1.0000 | 0.0000 | 0.0646 | -| all events | 30,000 | 0.9949 | 0.0003 | **0.0609** | - -Reading it, conceded row first: - -- **Lines seen once agree, and the falsifier fired.** 0.9859, mechanism above, concession recorded in section "What would prove log10x wrong". The claim that a locally mined name is local to the process that mined it is withdrawn for lines seen once. -- **The divergence sits exactly where generalization begins.** At 2-to-10 the two instances give the same line two different names 0.3978 of the time, and at 11-to-100 still 0.0882. `create_template` wildcards the positions where that instance's own siblings happened to vary, so two instances with different histories settle on different intersections of the same statement's outputs. A fleet-wide statement that fires a handful of times per node lands in these strata on each node that mines it independently; the two instances here measure one such pair. -- **The integer never travels.** `cluster_id` agrees 0.0609 over all probes, and 0.0005 in the 2-to-10 stratum, because it is an arrival counter. Hashing the template string is the correct repair for the integer, and the hash column would equal the template column exactly: equal strings hash equal, divergent strings hash divergent. Hashing inherits the divergence. -- **The aggregate carries almost none of this.** The strong reference pass finds 1,481 distinct shapes, 921 of them occurring exactly once, yet the all-events row sits at 0.9949 because the over-1,000 stratum is 94% of probed events. The tail is where the divergence is, and the aggregate is not where the tail is visible. - -`baseline` (typed-token masking, memory only) lands within a few points of `strong` in every stratum: 0.6500 at 2-to-10, 0.9036 at 11-to-100. Neither masking style, nor persistence, nor lookup strategy moves the tail materially; the split does. - -**Buying agreement.** The `shared` arm chains the two instances through one snapshot, so B warm-starts from A's final state. That moves 2-to-10 agreement from 0.6022 to 0.8114 and cluster_id agreement over all events from 0.0609 to 0.9970. Agreement is purchasable, and its price is one shared model that both sides read and write in sequence, which is an architecture rather than a setting. - -### The 10x engine on the same halves - -`bench/engine_split.py` runs this protocol through engine 1.1.142 (`log10x/pipeline-10x:1.1.142@sha256:22ae67148bd7f6cbdf171c1b8da1c9c6c801290d43488693dfd8527e27138d05`): the same seed, split and probe sequence per trial, each half through its own engine container with the image's shipped configuration and symbol library, fed in the order Drain3 receives it, then the probe records. The engine reads the whole record rather than the `log` field and names events, and an indented line joins the event above it, so a probe record is credited with the names of the event that holds it. The pattern hash is `tenx_hash`, the hash of the pattern name. On the probes of each row of the table above: - -| line frequency | probes/trial | Drain3 strong, template | engine, pattern hash | engine, template hash | +| Shape occurs in stream | Probes per trial | Drain3 template | Drain3 cluster_id | Engine pattern hash | |---|---:|---:|---:|---:| -| seen once | 136 | 0.9859 | 1.0000 | 1.0000 | -| 2 to 10 | 204 | 0.6022 | 1.0000 | 1.0000 | -| 11 to 100 | 589 | 0.9118 | 1.0000 | 1.0000 | -| 101 to 1,000 | 916 | 0.9788 | 1.0000 | 0.9962 | -| over 1,000 | 28,152 | 1.0000 | 1.0000 | 1.0000 | -| all events | 30,000 | 0.9949 | 1.0000 | 0.9998 | - -The two instances gave 299,997 of the 300,000 probe records (10 trials of 30,000) the same pattern name and pattern hash. The 3 that differ are the first 3 probes of trial 10: indented lines that joined the last event of each instance's half, and the two halves end with different events. The template hash differs on 44 more probe records, all events of one Grafana statement (`notify_retry_canceled_due_unrecoverable_error_after_attempts_SMTP_not_configured`), in 2 trials: the same two-record event, named the same on both sides, with two different template hashes. `bench/engine_split_context.py` runs that event at the end of shorter stretches of one instance's feed: each input gives the same answer on every run, and the template hash changes with how much history comes before it (`results/engine_split_context.md`). So the template hash of these events depends on what the instance read earlier, and the pattern name does not. - -Restricted to probes whose Drain3 shape occurs in both halves, Drain3's strong arm agrees 0.6555 at 2 to 10 and the engine 1.0000. Binned by the engine's own frequencies the result is the same (`results/engine_split.md`). - -Without the shared probe pass, from each instance's answers for its own half: a record that occurs, byte for byte, in both halves and that both instances named as an event of its own got the same pattern hash 1.0000 and the same template hash 1.0000 of the time (17,481 records per trial). +| once | 136 | 0.9859 | 0.0007 | 1.0000 | +| 2 to 10 times | 204 | 0.6022 | 0.0005 | 1.0000 | +| 11 to 100 times | 589 | 0.9118 | 0.0051 | 1.0000 | +| 101 to 1,000 times | 916 | 0.9788 | 0.0070 | 1.0000 | +| over 1,000 times | 28,152 | 1.0000 | 0.0646 | 1.0000 | +| all probes | 30,000 | 0.9949 | 0.0609 | 1.0000 | -### Query-time lookup, reported separately +A line seen once agrees because neither Drain3 instance can generalize from one example. The divergence starts when Drain3 generalizes, and the all-probes row hides it because shapes seen over 1,000 times are 94% of probes. -The read-only `match()` API answers a different question: not "what would this node name the line" but "does a name already exist here to resolve a query against". drain3 0.9.11 requires a similarity of 1.0 on that path, and an instance that never ingested the shape returns `None`. That is **coverage**, and this benchmark reports it under that word, never as disagreement. Strong arm: coverage at "seen once" is 0.0000, meaning the name exists in exactly one place in the fleet and nothing elsewhere can resolve it; coverage at 2-to-10 is 0.5307. Among probes both instances can already answer, the names still differ 0.2146 at 2-to-10 and 0.0924 at 11-to-100, the same generalization divergence as the primary table, seen from the lookup side. Full tables in `results/identity.md`. +- Fed the same 60,000 lines in two orders, two Drain3 runs agreed on the template 0.9860 of the time and on `cluster_id`, an arrival counter, 0.0001. +- The 3 engine probes that differed were indented lines joined to each half's last event, and the two halves ended differently. +- A second engine identifier, the template hash, differed on 44 probes of one Grafana statement, because for those events it depends on what the instance read before. -## 3. The identifier +## How it was measured -A separate experiment separates the template from the integer Drain returns with it, inside one process. Same 60,000 lines, two arrival orders: the template string agrees 0.9860, the integer `cluster_id` agrees 0.0001, and under cap pressure at `max_clusters=500` the split widens to 0.9848 against 0.0000 (`bench/identifier.py`). `cluster_id` is `self.clusters_counter += 1`, which is arrival rank and nothing else. Drain's own examples print it and dashboards key on it. +The data is 197,430 Fluentd/Kubernetes lines from the OpenTelemetry demo ([otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1)), covering 10.3 hours. Each trial splits the stream into random disjoint halves, like two forwarders that never share state, and after its half each instance names the same 30,000 probe lines. Both ingest every probe, so their states converge, which on net favors Drain3. Drain3 0.9.11 runs its strongest tested configuration against engine 1.1.142, and the fork drain3-improved 0.10.0 gave identical numbers. -## 4. Coverage on the engine side +## Limits -The engine assigns names from a compiled symbol library rather than from the stream. Section 2 runs the half-split on it: a record's pattern name did not depend on what the instance read before, apart from records that joined a different event, while one statement's template hash did. The remaining engine question is coverage, and it splits into two parts, one circular and one not. The OpenTelemetry demo's source is in the engine's default symbol library, so the demo sample is not unfamiliar code to the engine; the two LogHub sets below are. +- The data is one small demo cluster, and the magnitudes belong to this stream. +- A central miner, such as a SIEM that mines at query time, has no second instance to disagree, but the replay result still applies. Drain3 needs no build step and is the right default when the emitting code is out of reach. +- The demo's source is in the engine's default symbol library. On invented words the engine returned an all-variable template named by a hash. +- Of the engine's templates here, 82.8% carry deployment values such as image tags, and agreement across redeploys is unmeasured. -**The circular part is fenced off.** The published templates asset is the dictionary the encoding run wrote for its own output: `bench/engine_side.py` measures that every one of its 2,896 templates is referenced and every reference resolves, and reports that as a consistency gate, not evidence. Reading a resolve rate between two co-generated artifacts as library coverage is unsupported, and this page does not do it. +## Run it -**The non-circular part runs the engine** (`bench/coverage.py`, `results/coverage.md`, pinned image digest recorded in the artifact; the script skips cleanly on machines without Docker): - -- **Invented vocabulary.** Lines whose content words were made up for this benchmark, fed beside ordinary lines. The words cannot sit in any symbol library, because they did not exist until this test was written, which is what closes the "the templates file is definitionally complete for its own encoding run" loophole. On engine 1.1.140, the run committed here, the ordinary line comes back as `$(yyyy-MM-dd HH:mm:ss) INFO Connection established to database` with the readable pattern name `Connection_established_to_database`. The invented line comes back as `$(yyyy-MM-dd HH:mm:ss) $ $ $ $ $` named `template_38924ebe1000f1a0`, a hash of its template's shape with no word from the line, so the coverage failure is visible in the artifact rather than disguised as a confident wrong name. All three occurrences carry the same template hash and name, and a second, independent engine run reproduces both. -- **Unfamiliar streams.** Two public LogHub log sets the symbol library was never built for, downloaded from Zenodo record 3227177: Proxifier, a Windows consumer application, encodes to 21,329 events under 1,854 templates with a median of 12 literal words per template and zero pure-variable templates; the Apache error log encodes to 52,004 events under 371 templates, median 20 literal words, zero pure-variable. Two independent engine runs agree on both files exactly, and the script fails on any difference; the counts also equal the compaction ledger's measurement of the same files on engine 1.1.133. -- **Caution.** The `encoded.log` the analysis app writes carries `pattern=` and `patternHash=` fields for tooling; it is the analysis artifact, and no storage-reduction figure may be computed from it. None appears in this benchmark. - -**What the published artifacts show about granularity**, from `bench/engine_side.py`: 2,896 templates for 159,454 events, 91.4% of them used exactly once, so template assignment does not wait for repetition. On the message body, 2,884 of 2,896 templates carry alphanumeric literal text at a median of 15 tokens, 12 do not, and anchoring does not degrade with rarity (used-once median 15 against over-1,000 median 19.5, ratio 0.77 against a floor of 0.50). One finding cuts against log10x and stays: 82.8% of templates carry deployment values welded into envelope literals, image tags and pod-name fragments, so a redeploy changes those templates. That is the same identity property this benchmark scores Drain on, at the template level. A pattern name holds across such a redeploy where the welded values are not library words; cross-deployment agreement of engine pattern names is unmeasured here. - -## 5. The mechanism - -The Drain-side results follow from four short code paths in `drain3/drain.py`, checkable in about ninety seconds against the pinned drain3 0.9.11. - -- `add_log_message`: an unmatched line creates a cluster immediately, on first sight, holding the full literal tokens. There is no minimum-support threshold. This is why the production path agrees on rare shapes, and why the agreed name is the raw line. -- `create_template(seq1, seq2)` returns `[token2 if token1 == token2 else param_str ...]` and writes the result back. A cluster template is the running intersection of every line that ever matched it, so it only loses literal tokens, and it loses them whenever a sibling arrives. Sibling arrival order is a property of the stream, which is why two streams settle on different intersections. -- `tree_search` partitions on token count at the first level. Two outputs of one source statement with different token counts can never share a cluster, whatever `depth` and `sim_th` are set to. -- `match()` requires a similarity of 1.0 and defaults to `full_search_strategy="never"`. It is a lookup, not a naming call, which is why this benchmark reports its misses as coverage. - -The same intersection mechanism is specified in the ICWS 2017 paper, so this is Drain working as designed, not a bug in drain3. - -## What would prove log10x wrong - -Each falsifier names a quantity that a rerun could move past a threshold fixed in the source before the run. They point in both directions: some fire against log10x, and two already have. - -- **Two independent instances give a line seen once the same name through the production path.** `bench/identity.py`, strong arm, threshold 0.50. Observed **0.9859. TRIGGERED.** The claim that a locally mined name is local to the process that mined it is withdrawn for lines seen once. What survives is narrower: the agreed name is the pre-generalization literal that section 1 shows moving, and the divergence begins one sibling later. -- **The 2-to-10 divergence disappears.** Threshold: production-path template disagreement below 0.02 at 2-to-10, strong arm. Observed 0.3978, not triggered. The check reruns at any similarity threshold via `--sim-th`; a sim_th at which it passes while the reference pass still resolves the stream into at least half the distinct shapes found at sim_th 0.6 retires the divergence claim outright. The script scores that shape-count guard itself: a non-default `--sim-th` run measures the sim_th 0.6 shape count on the same lines, and a pass that merely coarsened the stream does not retire the claim. -- **The integer survives the split.** Threshold: cluster_id agreement at or above 0.50 over all probes, strong arm. Observed 0.0609, not triggered; at 2-to-10 it is 0.0005. -- **Where both instances hold a name at lookup time, they agree.** Threshold: disagreement below 0.02 at 11-to-100, strong arm. Observed 0.0924, not triggered. -- **Byte-identical input keeps its name inside one process.** `bench/replay.py` asserts the template moves and that the replay reports no cluster change; a drain3 release that makes the name hold fails the asserts and collapses the exhibit. -- **Drain's templates drift under reordering.** `bench/identifier.py` asserts template agreement above 0.95 at the 60,000-line reference run; below it, the stability concession at the top of this page is wrong and log10x has no business repeating it. The twin assert fires the other way: cluster_id agreement above 0.01 retires the arrival-rank claim. -- **An invented-vocabulary line ever yields two different template hashes**, across occurrences or across independent engine runs. `bench/coverage.py` checks it on every run; one mismatch retires the engine determinism claim. -- **An invented-vocabulary line comes back with a pattern name other than `template_`, a hash of its shape**. Same script; one occurrence retires the claim that engine coverage failures are visible rather than mislabelled. -- **Two independent engine runs disagree on a LogHub file**, in events or template hashes, or a run mints a material share of pure-variable templates. Observed zero pure-variable templates out of 1,854 and 371, and exact run-to-run agreement; a mismatch fails `bench/coverage.py`. -- **Two engine instances give a probe record different pattern hashes.** `bench/engine_split.py`, threshold 0.99 pattern hash agreement in any stratum, under Drain3's strata. Observed 0.99999 at the lowest (over 1,000, 3 probes), not triggered. -- **Engine templates degrade toward wildcards as shapes get rarer.** `bench/engine_side.py`, used-once median against over-1,000 median, floor 0.50. Observed 0.77, not triggered. -- **An exception count is not expressible in Drain's output.** `bench/multiline.py`. Fired against log10x on this file: proxy clusters exist for every burst and three of them unioned count all 2,316. The claim is withdrawn in `results/multiline.txt`, and what remains is narrower: the header-to-frames link is not a field in Drain's output, and the proxy holds only while each header shape stays exclusive to its exception. - -## Fixed by the design, not tested by it - -- **Production-path agreement for lines seen once is high mechanically.** Neither instance can generalize a shape it has one example of, so both mint the line's own literal text. The concession above is genuine, and this is its mechanism. -- **Lookup coverage for lines seen once is near zero by construction.** The shape occurs in one line, the line lands in one half, and `match()` requires a similarity of 1.0. It is a coverage property of a lookup API and is scored as coverage everywhere on this page. -- **The shared arm's seen-once coverage lands near one half by construction.** A shape seen once is in A's side or B's side with equal probability, and only A's side reaches B through the snapshot. Observed 0.5053, which is the sanity check passing. -- **Engine agreement on the probe pass is what a name computed from the event gives.** Both instances receive the same probe records in the same order, so a probe lands in an event of the same records on both sides; the result says that nothing in an instance's half changed a name, and no more. The same-line figures from the halves are the part without shared traffic. - -## Where this applies - -The split result applies where two or more processes mine independently and their names are expected to line up: multi-region or multi-cluster pipelines each mining locally, stream processors sharded on a key other than the pattern, edge agents clustering locally for sampling, and any design that ships a locally mined name upstream or persists it. Where one central miner holds the only model, at ingest or at query time, there is no second opinion and section 2 does not apply. Section 1 does apply there, because a name moving inside one process needs only that one process, and it reaches anything that persists a mined name across time. - -A random half-split is also the friendliest division available: both instances draw statistically identical samples of one stream. Production splits follow placement, so each node sees different source statements, which is harsher than anything measured here. - -## How this relates to log-parser accuracy benchmarks - -Parser evaluations in the literature (LogHub/LogPAI, and the GA, PA, FGA and FTA metrics) measure single-run accuracy against human-labelled templates, so they need labelled ground truth and they score one miner at a time. No standard metric exists for cross-instance identity. Self-agreement is the instrument used here precisely because it needs no ground truth: shapes come from Drain's own full-stream reference pass and agreement is measured between two Drain instances, so no log10x notion of a pattern enters the Drain side of the measurement anywhere. - -## Prior art, and what is not claimed - -- Lossless template-plus-values storage is not novel. Prior art: CLP/YScope, US10484506B2. -- Template extraction from log text is not novel. Prior art: Drain, Spell, IPLoM. -- Compile-time message identity is not novel. Prior art: NanoLog, ETW event manifests. -- Engine-side identifiers are not unconditionally stable. They hold relative to a pinned symbol library, a pinned engine version, and that library's coverage of the emitting statement. Section 4 adds a measured fourth condition: 82.8% of templates on the published artifact carry deployment values in envelope literals, so a redeploy moves those templates; the pattern name, which keys on the selected words and not the envelope, is unmeasured across deployments. -- Cross-instance agreement for the engine is measured on one protocol and one stream (section 2): two instances fed disjoint halves gave a record that lands in the same event on both the same pattern name. A record grouped under a different line takes that line's name, and the template hash of one statement's events depended on what the instance had read before. -- Immediate naming of unfamiliar streams is not claimed as an engine advantage. Drain3 does the same with less machinery. The measured difference is the failure shape: a template-only name under a deterministic hash, against a literal template that moves when siblings arrive. -- The split result is not claimed against centrally mining query-time products. It binds per-node mining and persisted mined names. - -## Reproduce - -```sh -./run.sh +``` +cd pattern-identity +./run.sh # full run, writes every artifact under results/ +PIB_QUICK=1 ./run.sh # identity experiment at 60,000 lines and 3 trials ``` -That builds a virtualenv, installs the pinned dependencies (drain3==0.9.11 on Python 3.13), runs the replay first, then downloads the public release assets if absent, records their sha256 and line counts to `results/data_provenance.txt`, and runs the remaining experiments cheapest first, writing every artifact under `results/`. Measured on an Intel macOS laptop: replay under 1s, engine_side about 5s, identifier about 25s, multiline about 65s, coverage about 2 minutes with the Docker image present (it skips cleanly without), identity about 19 minutes, engine_split about 35 minutes and engine_split_context about 3 with the Docker image present (both skip cleanly without). `PIB_QUICK=1 ./run.sh` runs the identity experiment at 60,000 lines and 3 trials. `PIB_PYTHON`, `PIB_VENV` and `PIB_DATA_DIR` override the interpreter, the virtualenv location and the download cache. The otel-sample assets are cached gzipped and read in place; the LogHub sets cache under `data/` as well. - -Runs are deterministic and no artifact records a wall-clock time, so `git diff --exit-code results/` after a rerun is a usable gate. Timings go to stderr. - -| script | question | artifact | -|---|---|---| -| `bench/replay.py` | one process: does a line keep its name when siblings arrive later | `results/replay.txt` | -| `bench/identity.py` | two instances, split halves, same probe sequence through the production path: template and identifier agreement by frequency, plus query-time lookup coverage | `results/identity.md`, `results/identity_raw.json` | -| `bench/identifier.py` | two arrival orders: does the template travel, does the integer travel | `results/identifier.txt` | -| `bench/multiline.py` | a stack trace is one event to a reader: how many names does each side give it | `results/multiline.txt` | -| `bench/engine_side.py` | how finely does the engine name this stream, and how much source text does each name pin down | `results/engine_side.md` | -| `bench/coverage.py` | what the engine does with vocabulary its library never held | `results/coverage.md`, `results/coverage_raw.json` | -| `bench/engine_split.py` | the identity protocol through two engine containers: pattern and template hash agreement by frequency, with and without the shared probe pass | `results/engine_split.md`, `results/engine_split_raw.json` | -| `bench/engine_split_context.py` | one probe event whose template hash differed between the two instances, run after shorter stretches of the same history | `results/engine_split_context.md` | - -## The maintained fork - -Upstream drain3 went quiet in April 2026 ([logpai/Drain3#120](https://github.com/logpai/Drain3/issues/120)) and the maintained fork is [drain3-improved](https://github.com/jpodivin/Drain3-improved), which installs the same `drain3` import name. Every Drain-side experiment on this page was rerun against **drain3-improved 0.10.0** with the same seed, configuration and input. - -**Every measured number is identical**, including the per-trial counts in the raw JSON. `results/drain3_improved.md` states how it was run and what came back, and the fork's artifacts sit beside the upstream ones as `results/*_drain3_improved.*`. So the identity results on this page are not a property of an unmaintained release. - -## Known limits - -Stated in the terms the reviewers of this benchmark used, because an objection published by the author is a caveat and the same objection found by a reader is a refutation. +The script needs Python 3.13 and downloads the sample. The Drain3 split takes about 19 minutes, and the engine split takes about 35 and needs Docker. -- **The divergence bands come from one dataset family.** Every frequency-stratified number in section 2 is measured on one OpenTelemetry demo stream. The LogHub sets exercise only the engine coverage exhibit, not the split. The mechanism is source-verified and order-driven, so the direction should transfer; the magnitudes are properties of this stream. -- **The probe pass shares traffic between the instances.** Both ingest every probe, so their states converge during the measurement, and the reported divergence understates, in expectation, what a fleet with no shared traffic would see. The direction is net rather than guaranteed per probe: a probe that matches different clusters in the two instances wildcards one side's template only, so a later probe of a different shape can flip from agree to differ because of the shared ingest. -- **The engine half-split feeds each half shuffled.** The protocol is Drain3's, so each engine instance reads its half in a random order, and a multi-line event's lines land in both halves and next to unrelated records. Fed in stream order, the reference run names 133,173 of 197,430 records as events of their own; each shuffled half names 62,274 of 98,715 on average. The same-line comparison is therefore scored on records each instance named as an event of their own. The engine runs shared the host with another session's Splunk container, using one to five cores, for the reference run and trials 1 to 8; it was stopped for trials 9 and 10 and for `bench/engine_split_context.py`, which reproduced the template-hash difference with the same answer on every repeat. -- **The seen-once stratum has no operational counterpart.** A shape that occurs exactly once in the whole stream is processed by exactly one forwarder and receives exactly one name. The row is reported because the falsifier on it fired against log10x and because the mechanism is instructive, not because a fleet behaves that way. -- **The half-split indicts an architecture many platforms do not run.** A deployment that mines centrally against one model has no cross-instance problem, and a reader running one correctly concludes that section 2 does not apply to them. Section 1 still does. -- **The engine artifacts are vendor-produced.** The coverage experiments run a pinned vendor-built Docker image, digest recorded in `results/coverage.md`, and this benchmark does not build the engine from source. The otel-sample templates asset is the encoding run's own output dictionary, which is why its resolve rate is a gate and not evidence, and the OpenTelemetry demo's source is in the engine's default symbol library. The invented-vocabulary probe exists because it cannot be answered by anything compiled with sight of anything. -- **The encoded artifact is for analysis.** `encoded.log` carries `pattern=` and `patternHash=` fields for tooling. No storage-reduction figure may be computed from it, and none appears here. -- **Part of the engine's template identity is deployment-specific.** 82.8% of templates on the published artifact carry environment values in envelope literals, so a redeployment with a new image tag or pod hash changes those templates. The pattern name keys on the selected words, not the envelope, so it is expected to hold; cross-deployment agreement of pattern names is unmeasured, and this is the concrete reason it could be below 1.0. -- **This sample is a weak surface for multi-line claims.** 2,311 of 2,316 exception bursts come from one container emitting one error shape, so the burst statistics describe one shape repeated, and `results/multiline.txt` says so in its scope note. -- **drain3 is pinned at 0.9.11 and a version bump is a re-run gate, not a routine update.** `bench/replay.py` and `bench/identifier.py` assert current library behaviour, and both refuse to run under `python -O` so the asserts stay live. The maintained fork is covered: see below. +[RECORD.md](RECORD.md) has every experiment and the tests that could prove log10x wrong, two of which fired against it. diff --git a/pattern-identity/RECORD.md b/pattern-identity/RECORD.md new file mode 100644 index 0000000..2e9b9d0 --- /dev/null +++ b/pattern-identity/RECORD.md @@ -0,0 +1,207 @@ +# Pattern identity benchmark + +One benchmark in the [log10x benchmarks](../) repository. Everything it needs is in this +directory: the scripts under `bench/`, the pinned dependencies, `run.sh`, and the artifacts +under `results/` that each script wrote. + +## The question + +Does a log line keep the same name over time, and does it get the same name in two different places? + +A pattern name is the key that sampling rules, suppression, dedup keys, cost attribution and new-pattern detection are built on. A template miner produces that name by mining the stream in front of it, so the name is a function of what that process happened to see and when. This benchmark measures what that costs, on public data, states what result would prove the log10x position wrong, and runs those tests instead of asserting them. Two of the pre-registered tests fired against log10x, and both stand as concessions on this page, in the sections where they were tested. + +The Drain-side experiments run against one public sample: 197,430 Fluentd/Kubernetes lines from the OpenTelemetry demo, published at [otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1). Measured scale, from `results/data_provenance.txt`: 10.3 hours between the 1st and 99th percentile timestamp, about 5.3 lines per second, 36 Kubernetes containers, the busiest of them 47.5% of the lines. That is one small demo cluster over about ten hours, not a log platform, and a shape that occurs once at 5 lines per second is a different population from a shape that occurs once at 100,000. The engine-side coverage experiments additionally use two public LogHub log sets from Zenodo and a pinned engine Docker image whose digest is recorded in the artifact: `log10x/pipeline-10x:1.1.140`, the image `results/coverage.md` was produced on. Section 2's engine rows run `log10x/pipeline-10x:1.1.142`, digest in `results/engine_split.md`. Every measured number on this page is printed by a script in this directory; the wall-clock estimates under Reproduce are not, and the ledger expectations are external, recorded six weeks earlier as disclosed in section 4. + +## What Drain does well + +Drain3 is the incumbent for a reason, and the concessions come first. + +- **Its templates are stable under reordering.** The same 60,000 lines fed in natural order and shuffled produce template strings that agree 0.9860 of the time (`bench/identifier.py`). The published stability result reproduces here and stands. +- **It works from the stream alone.** There is no build step, no compiler pass, and no dependency on the emitting source. Pointed at an unfamiliar stream it produces useful templates immediately, which makes it the right default when the emitting code is out of reach. +- **Two instances that never exchanged state give a rare line the same name.** Through the production path, `add_log_message`, a line seen once is named identically by both instances 0.9859 of the time (`bench/identity.py`, strong arm). This was a pre-registered falsifier and it fired against log10x. The mechanism is in the Drain3 source: neither instance can generalize a shape it has one example of, so both mint the line's own literal text, and the two strings are equal. An earlier revision of this benchmark reported 0.0 for that row by probing with the read-only `match()` API, whose misses are a coverage property of a lookup call; that number described the probe, not the miner, and the corrected design below replaces it. +- **On frequent shapes it converges.** For shapes occurring over 1,000 times, two independent instances agree 1.0000 of the time, through either path. +- **Centralized query-time mining is out of scope of the split result.** The major SIEM platforms that mine patterns do it in one central pass at query time, over whatever the query returns, so no second instance exists to disagree. The two-forwarder experiment below indicts per-node mining and any design that persists a mined name, not those products as they ship today. Section 1 is the part that reaches a central miner, because a name moving inside one process needs only that one process. +- **Two claims made against it fail on this data, and the scripts say so.** Drain does not shatter stack frames into a long cluster tail here: 4,658 frame lines collapse to 25 shapes. And an exception count turned out to be expressible in Drain's output on this file, through clusters that fire exactly once per burst and never outside one, so that claim is withdrawn in `results/multiline.txt` where it was tested. + +The templates are good. What this benchmark measures is the identity attached to them. + +## 1. A name moves inside one process + +This is the exhibit everything else depends on. It needs no split, no fleet, and no download, it runs in under a second, and `run.sh` runs it first. + +One Drain3 instance, one cluster, four lines from the public sample: a Grafana provisioning line arrives and is named, two sibling outputs of the same statement arrive, and the first line is presented again byte for byte. + +``` +step 1 logger=provisioning.datasources t= level=info msg="inserting datasource from configuration" name=Jaeger uid=webstore-traces +step 4 logger=provisioning.datasources t= level=info msg="inserting datasource from configuration" <*> <*> +``` + +The bytes are identical. The name is not. Step 2 reports `cluster_template_changed`, so the rename is visible to a caller processing the sibling. Step 4, the replay of the original bytes, reports `change_type='none'`, so a pipeline that keys on the template, or on a hash of it, writes a second name for the first line while being told the cluster held steady. Both facts are asserted against drain3 0.9.11, and the script refuses to run under `python -O` so the asserts stay live (`bench/replay.py`, `results/replay.txt`). + +This applies to every deployment, including one central miner with a single model. Anything that persists a mined name across time inherits it: a sampling rule, a suppression rule, a dedup key, a cost attribution, a new-pattern panel that now has a new pattern to report. Hand-written alerts on fields are untouched; alerts keyed on a mined pattern id are not. This is also why the singleton agreement conceded above is narrower than it sounds: the name both instances agree on is the pre-generalization literal, the exact string this exhibit shows moving the moment a sibling arrives. + +## 2. Two instances, the production path + +Two Drain3 instances each receive one random half of the same stream, disjoint, which models two forwarders that never exchange state. The same probe sequence is then fed to both through `add_log_message` itself, in the same order, exactly as if two nodes with different histories started seeing the same traffic. Every probe gets an answer from both sides, because the production path mints a cluster on first sight; there is no coverage term and no `None` anywhere in this table. Full file, 10 trials, 30,000 probes per trial, seed pinned (`bench/identity.py`, `results/identity.md`). + +The probe pass itself is shared traffic, so the two states converge as it proceeds, which works in Drain's favour on net. The divergence reported is what survives that convergence. + +**Drain3 at its strongest honest configuration** (`strong`: drain3's own `examples/drain3.ini` masking block, `FilePersistence` carried across a mid-stream restart, fallback lookup): + +| line frequency | probes/trial | template agreement | stdev | cluster_id agreement | +|---|---:|---:|---:|---:| +| seen once | 136 | 0.9859 | 0.0085 | 0.0007 | +| 2 to 10 | 204 | **0.6022** | 0.0243 | 0.0005 | +| 11 to 100 | 589 | 0.9118 | 0.0158 | 0.0051 | +| 101 to 1,000 | 916 | 0.9788 | 0.0045 | 0.0070 | +| over 1,000 | 28,152 | 1.0000 | 0.0000 | 0.0646 | +| all events | 30,000 | 0.9949 | 0.0003 | **0.0609** | + +Reading it, conceded row first: + +- **Lines seen once agree, and the falsifier fired.** 0.9859, mechanism above, concession recorded in section "What would prove log10x wrong". The claim that a locally mined name is local to the process that mined it is withdrawn for lines seen once. +- **The divergence sits exactly where generalization begins.** At 2-to-10 the two instances give the same line two different names 0.3978 of the time, and at 11-to-100 still 0.0882. `create_template` wildcards the positions where that instance's own siblings happened to vary, so two instances with different histories settle on different intersections of the same statement's outputs. A fleet-wide statement that fires a handful of times per node lands in these strata on each node that mines it independently; the two instances here measure one such pair. +- **The integer never travels.** `cluster_id` agrees 0.0609 over all probes, and 0.0005 in the 2-to-10 stratum, because it is an arrival counter. Hashing the template string is the correct repair for the integer, and the hash column would equal the template column exactly: equal strings hash equal, divergent strings hash divergent. Hashing inherits the divergence. +- **The aggregate carries almost none of this.** The strong reference pass finds 1,481 distinct shapes, 921 of them occurring exactly once, yet the all-events row sits at 0.9949 because the over-1,000 stratum is 94% of probed events. The tail is where the divergence is, and the aggregate is not where the tail is visible. + +`baseline` (typed-token masking, memory only) lands within a few points of `strong` in every stratum: 0.6500 at 2-to-10, 0.9036 at 11-to-100. Neither masking style, nor persistence, nor lookup strategy moves the tail materially; the split does. + +**Buying agreement.** The `shared` arm chains the two instances through one snapshot, so B warm-starts from A's final state. That moves 2-to-10 agreement from 0.6022 to 0.8114 and cluster_id agreement over all events from 0.0609 to 0.9970. Agreement is purchasable, and its price is one shared model that both sides read and write in sequence, which is an architecture rather than a setting. + +### The 10x engine on the same halves + +`bench/engine_split.py` runs this protocol through engine 1.1.142 (`log10x/pipeline-10x:1.1.142@sha256:22ae67148bd7f6cbdf171c1b8da1c9c6c801290d43488693dfd8527e27138d05`): the same seed, split and probe sequence per trial, each half through its own engine container with the image's shipped configuration and symbol library, fed in the order Drain3 receives it, then the probe records. The engine reads the whole record rather than the `log` field and names events, and an indented line joins the event above it, so a probe record is credited with the names of the event that holds it. The pattern hash is `tenx_hash`, the hash of the pattern name. On the probes of each row of the table above: + +| line frequency | probes/trial | Drain3 strong, template | engine, pattern hash | engine, template hash | +|---|---:|---:|---:|---:| +| seen once | 136 | 0.9859 | 1.0000 | 1.0000 | +| 2 to 10 | 204 | 0.6022 | 1.0000 | 1.0000 | +| 11 to 100 | 589 | 0.9118 | 1.0000 | 1.0000 | +| 101 to 1,000 | 916 | 0.9788 | 1.0000 | 0.9962 | +| over 1,000 | 28,152 | 1.0000 | 1.0000 | 1.0000 | +| all events | 30,000 | 0.9949 | 1.0000 | 0.9998 | + +The two instances gave 299,997 of the 300,000 probe records (10 trials of 30,000) the same pattern name and pattern hash. The 3 that differ are the first 3 probes of trial 10: indented lines that joined the last event of each instance's half, and the two halves end with different events. The template hash differs on 44 more probe records, all events of one Grafana statement (`notify_retry_canceled_due_unrecoverable_error_after_attempts_SMTP_not_configured`), in 2 trials: the same two-record event, named the same on both sides, with two different template hashes. `bench/engine_split_context.py` runs that event at the end of shorter stretches of one instance's feed: each input gives the same answer on every run, and the template hash changes with how much history comes before it (`results/engine_split_context.md`). So the template hash of these events depends on what the instance read earlier, and the pattern name does not. + +Restricted to probes whose Drain3 shape occurs in both halves, Drain3's strong arm agrees 0.6555 at 2 to 10 and the engine 1.0000. Binned by the engine's own frequencies the result is the same (`results/engine_split.md`). + +Without the shared probe pass, from each instance's answers for its own half: a record that occurs, byte for byte, in both halves and that both instances named as an event of its own got the same pattern hash 1.0000 and the same template hash 1.0000 of the time (17,481 records per trial). + +### Query-time lookup, reported separately + +The read-only `match()` API answers a different question: not "what would this node name the line" but "does a name already exist here to resolve a query against". drain3 0.9.11 requires a similarity of 1.0 on that path, and an instance that never ingested the shape returns `None`. That is **coverage**, and this benchmark reports it under that word, never as disagreement. Strong arm: coverage at "seen once" is 0.0000, meaning the name exists in exactly one place in the fleet and nothing elsewhere can resolve it; coverage at 2-to-10 is 0.5307. Among probes both instances can already answer, the names still differ 0.2146 at 2-to-10 and 0.0924 at 11-to-100, the same generalization divergence as the primary table, seen from the lookup side. Full tables in `results/identity.md`. + +## 3. The identifier + +A separate experiment separates the template from the integer Drain returns with it, inside one process. Same 60,000 lines, two arrival orders: the template string agrees 0.9860, the integer `cluster_id` agrees 0.0001, and under cap pressure at `max_clusters=500` the split widens to 0.9848 against 0.0000 (`bench/identifier.py`). `cluster_id` is `self.clusters_counter += 1`, which is arrival rank and nothing else. Drain's own examples print it and dashboards key on it. + +## 4. Coverage on the engine side + +The engine assigns names from a compiled symbol library rather than from the stream. Section 2 runs the half-split on it: a record's pattern name did not depend on what the instance read before, apart from records that joined a different event, while one statement's template hash did. The remaining engine question is coverage, and it splits into two parts, one circular and one not. The OpenTelemetry demo's source is in the engine's default symbol library, so the demo sample is not unfamiliar code to the engine; the two LogHub sets below are. + +**The circular part is fenced off.** The published templates asset is the dictionary the encoding run wrote for its own output: `bench/engine_side.py` measures that every one of its 2,896 templates is referenced and every reference resolves, and reports that as a consistency gate, not evidence. Reading a resolve rate between two co-generated artifacts as library coverage is unsupported, and this page does not do it. + +**The non-circular part runs the engine** (`bench/coverage.py`, `results/coverage.md`, pinned image digest recorded in the artifact; the script skips cleanly on machines without Docker): + +- **Invented vocabulary.** Lines whose content words were made up for this benchmark, fed beside ordinary lines. The words cannot sit in any symbol library, because they did not exist until this test was written, which is what closes the "the templates file is definitionally complete for its own encoding run" loophole. On engine 1.1.140, the run committed here, the ordinary line comes back as `$(yyyy-MM-dd HH:mm:ss) INFO Connection established to database` with the readable pattern name `Connection_established_to_database`. The invented line comes back as `$(yyyy-MM-dd HH:mm:ss) $ $ $ $ $` named `template_38924ebe1000f1a0`, a hash of its template's shape with no word from the line, so the coverage failure is visible in the artifact rather than disguised as a confident wrong name. All three occurrences carry the same template hash and name, and a second, independent engine run reproduces both. +- **Unfamiliar streams.** Two public LogHub log sets the symbol library was never built for, downloaded from Zenodo record 3227177: Proxifier, a Windows consumer application, encodes to 21,329 events under 1,854 templates with a median of 12 literal words per template and zero pure-variable templates; the Apache error log encodes to 52,004 events under 371 templates, median 20 literal words, zero pure-variable. Two independent engine runs agree on both files exactly, and the script fails on any difference; the counts also equal the compaction ledger's measurement of the same files on engine 1.1.133. +- **Caution.** The `encoded.log` the analysis app writes carries `pattern=` and `patternHash=` fields for tooling; it is the analysis artifact, and no storage-reduction figure may be computed from it. None appears in this benchmark. + +**What the published artifacts show about granularity**, from `bench/engine_side.py`: 2,896 templates for 159,454 events, 91.4% of them used exactly once, so template assignment does not wait for repetition. On the message body, 2,884 of 2,896 templates carry alphanumeric literal text at a median of 15 tokens, 12 do not, and anchoring does not degrade with rarity (used-once median 15 against over-1,000 median 19.5, ratio 0.77 against a floor of 0.50). One finding cuts against log10x and stays: 82.8% of templates carry deployment values welded into envelope literals, image tags and pod-name fragments, so a redeploy changes those templates. That is the same identity property this benchmark scores Drain on, at the template level. A pattern name holds across such a redeploy where the welded values are not library words; cross-deployment agreement of engine pattern names is unmeasured here. + +## 5. The mechanism + +The Drain-side results follow from four short code paths in `drain3/drain.py`, checkable in about ninety seconds against the pinned drain3 0.9.11. + +- `add_log_message`: an unmatched line creates a cluster immediately, on first sight, holding the full literal tokens. There is no minimum-support threshold. This is why the production path agrees on rare shapes, and why the agreed name is the raw line. +- `create_template(seq1, seq2)` returns `[token2 if token1 == token2 else param_str ...]` and writes the result back. A cluster template is the running intersection of every line that ever matched it, so it only loses literal tokens, and it loses them whenever a sibling arrives. Sibling arrival order is a property of the stream, which is why two streams settle on different intersections. +- `tree_search` partitions on token count at the first level. Two outputs of one source statement with different token counts can never share a cluster, whatever `depth` and `sim_th` are set to. +- `match()` requires a similarity of 1.0 and defaults to `full_search_strategy="never"`. It is a lookup, not a naming call, which is why this benchmark reports its misses as coverage. + +The same intersection mechanism is specified in the ICWS 2017 paper, so this is Drain working as designed, not a bug in drain3. + +## What would prove log10x wrong + +Each falsifier names a quantity that a rerun could move past a threshold fixed in the source before the run. They point in both directions: some fire against log10x, and two already have. + +- **Two independent instances give a line seen once the same name through the production path.** `bench/identity.py`, strong arm, threshold 0.50. Observed **0.9859. TRIGGERED.** The claim that a locally mined name is local to the process that mined it is withdrawn for lines seen once. What survives is narrower: the agreed name is the pre-generalization literal that section 1 shows moving, and the divergence begins one sibling later. +- **The 2-to-10 divergence disappears.** Threshold: production-path template disagreement below 0.02 at 2-to-10, strong arm. Observed 0.3978, not triggered. The check reruns at any similarity threshold via `--sim-th`; a sim_th at which it passes while the reference pass still resolves the stream into at least half the distinct shapes found at sim_th 0.6 retires the divergence claim outright. The script scores that shape-count guard itself: a non-default `--sim-th` run measures the sim_th 0.6 shape count on the same lines, and a pass that merely coarsened the stream does not retire the claim. +- **The integer survives the split.** Threshold: cluster_id agreement at or above 0.50 over all probes, strong arm. Observed 0.0609, not triggered; at 2-to-10 it is 0.0005. +- **Where both instances hold a name at lookup time, they agree.** Threshold: disagreement below 0.02 at 11-to-100, strong arm. Observed 0.0924, not triggered. +- **Byte-identical input keeps its name inside one process.** `bench/replay.py` asserts the template moves and that the replay reports no cluster change; a drain3 release that makes the name hold fails the asserts and collapses the exhibit. +- **Drain's templates drift under reordering.** `bench/identifier.py` asserts template agreement above 0.95 at the 60,000-line reference run; below it, the stability concession at the top of this page is wrong and log10x has no business repeating it. The twin assert fires the other way: cluster_id agreement above 0.01 retires the arrival-rank claim. +- **An invented-vocabulary line ever yields two different template hashes**, across occurrences or across independent engine runs. `bench/coverage.py` checks it on every run; one mismatch retires the engine determinism claim. +- **An invented-vocabulary line comes back with a pattern name other than `template_`, a hash of its shape**. Same script; one occurrence retires the claim that engine coverage failures are visible rather than mislabelled. +- **Two independent engine runs disagree on a LogHub file**, in events or template hashes, or a run mints a material share of pure-variable templates. Observed zero pure-variable templates out of 1,854 and 371, and exact run-to-run agreement; a mismatch fails `bench/coverage.py`. +- **Two engine instances give a probe record different pattern hashes.** `bench/engine_split.py`, threshold 0.99 pattern hash agreement in any stratum, under Drain3's strata. Observed 0.99999 at the lowest (over 1,000, 3 probes), not triggered. +- **Engine templates degrade toward wildcards as shapes get rarer.** `bench/engine_side.py`, used-once median against over-1,000 median, floor 0.50. Observed 0.77, not triggered. +- **An exception count is not expressible in Drain's output.** `bench/multiline.py`. Fired against log10x on this file: proxy clusters exist for every burst and three of them unioned count all 2,316. The claim is withdrawn in `results/multiline.txt`, and what remains is narrower: the header-to-frames link is not a field in Drain's output, and the proxy holds only while each header shape stays exclusive to its exception. + +## Fixed by the design, not tested by it + +- **Production-path agreement for lines seen once is high mechanically.** Neither instance can generalize a shape it has one example of, so both mint the line's own literal text. The concession above is genuine, and this is its mechanism. +- **Lookup coverage for lines seen once is near zero by construction.** The shape occurs in one line, the line lands in one half, and `match()` requires a similarity of 1.0. It is a coverage property of a lookup API and is scored as coverage everywhere on this page. +- **The shared arm's seen-once coverage lands near one half by construction.** A shape seen once is in A's side or B's side with equal probability, and only A's side reaches B through the snapshot. Observed 0.5053, which is the sanity check passing. +- **Engine agreement on the probe pass is what a name computed from the event gives.** Both instances receive the same probe records in the same order, so a probe lands in an event of the same records on both sides; the result says that nothing in an instance's half changed a name, and no more. The same-line figures from the halves are the part without shared traffic. + +## Where this applies + +The split result applies where two or more processes mine independently and their names are expected to line up: multi-region or multi-cluster pipelines each mining locally, stream processors sharded on a key other than the pattern, edge agents clustering locally for sampling, and any design that ships a locally mined name upstream or persists it. Where one central miner holds the only model, at ingest or at query time, there is no second opinion and section 2 does not apply. Section 1 does apply there, because a name moving inside one process needs only that one process, and it reaches anything that persists a mined name across time. + +A random half-split is also the friendliest division available: both instances draw statistically identical samples of one stream. Production splits follow placement, so each node sees different source statements, which is harsher than anything measured here. + +## How this relates to log-parser accuracy benchmarks + +Parser evaluations in the literature (LogHub/LogPAI, and the GA, PA, FGA and FTA metrics) measure single-run accuracy against human-labelled templates, so they need labelled ground truth and they score one miner at a time. No standard metric exists for cross-instance identity. Self-agreement is the instrument used here precisely because it needs no ground truth: shapes come from Drain's own full-stream reference pass and agreement is measured between two Drain instances, so no log10x notion of a pattern enters the Drain side of the measurement anywhere. + +## Prior art, and what is not claimed + +- Lossless template-plus-values storage is not novel. Prior art: CLP/YScope, US10484506B2. +- Template extraction from log text is not novel. Prior art: Drain, Spell, IPLoM. +- Compile-time message identity is not novel. Prior art: NanoLog, ETW event manifests. +- Engine-side identifiers are not unconditionally stable. They hold relative to a pinned symbol library, a pinned engine version, and that library's coverage of the emitting statement. Section 4 adds a measured fourth condition: 82.8% of templates on the published artifact carry deployment values in envelope literals, so a redeploy moves those templates; the pattern name, which keys on the selected words and not the envelope, is unmeasured across deployments. +- Cross-instance agreement for the engine is measured on one protocol and one stream (section 2): two instances fed disjoint halves gave a record that lands in the same event on both the same pattern name. A record grouped under a different line takes that line's name, and the template hash of one statement's events depended on what the instance had read before. +- Immediate naming of unfamiliar streams is not claimed as an engine advantage. Drain3 does the same with less machinery. The measured difference is the failure shape: a template-only name under a deterministic hash, against a literal template that moves when siblings arrive. +- The split result is not claimed against centrally mining query-time products. It binds per-node mining and persisted mined names. + +## Reproduce + +```sh +./run.sh +``` + +That builds a virtualenv, installs the pinned dependencies (drain3==0.9.11 on Python 3.13), runs the replay first, then downloads the public release assets if absent, records their sha256 and line counts to `results/data_provenance.txt`, and runs the remaining experiments cheapest first, writing every artifact under `results/`. Measured on an Intel macOS laptop: replay under 1s, engine_side about 5s, identifier about 25s, multiline about 65s, coverage about 2 minutes with the Docker image present (it skips cleanly without), identity about 19 minutes, engine_split about 35 minutes and engine_split_context about 3 with the Docker image present (both skip cleanly without). `PIB_QUICK=1 ./run.sh` runs the identity experiment at 60,000 lines and 3 trials. `PIB_PYTHON`, `PIB_VENV` and `PIB_DATA_DIR` override the interpreter, the virtualenv location and the download cache. The otel-sample assets are cached gzipped and read in place; the LogHub sets cache under `data/` as well. + +Runs are deterministic and no artifact records a wall-clock time, so `git diff --exit-code results/` after a rerun is a usable gate. Timings go to stderr. + +| script | question | artifact | +|---|---|---| +| `bench/replay.py` | one process: does a line keep its name when siblings arrive later | `results/replay.txt` | +| `bench/identity.py` | two instances, split halves, same probe sequence through the production path: template and identifier agreement by frequency, plus query-time lookup coverage | `results/identity.md`, `results/identity_raw.json` | +| `bench/identifier.py` | two arrival orders: does the template travel, does the integer travel | `results/identifier.txt` | +| `bench/multiline.py` | a stack trace is one event to a reader: how many names does each side give it | `results/multiline.txt` | +| `bench/engine_side.py` | how finely does the engine name this stream, and how much source text does each name pin down | `results/engine_side.md` | +| `bench/coverage.py` | what the engine does with vocabulary its library never held | `results/coverage.md`, `results/coverage_raw.json` | +| `bench/engine_split.py` | the identity protocol through two engine containers: pattern and template hash agreement by frequency, with and without the shared probe pass | `results/engine_split.md`, `results/engine_split_raw.json` | +| `bench/engine_split_context.py` | one probe event whose template hash differed between the two instances, run after shorter stretches of the same history | `results/engine_split_context.md` | + +## The maintained fork + +Upstream drain3 went quiet in April 2026 ([logpai/Drain3#120](https://github.com/logpai/Drain3/issues/120)) and the maintained fork is [drain3-improved](https://github.com/jpodivin/Drain3-improved), which installs the same `drain3` import name. Every Drain-side experiment on this page was rerun against **drain3-improved 0.10.0** with the same seed, configuration and input. + +**Every measured number is identical**, including the per-trial counts in the raw JSON. `results/drain3_improved.md` states how it was run and what came back, and the fork's artifacts sit beside the upstream ones as `results/*_drain3_improved.*`. So the identity results on this page are not a property of an unmaintained release. + +## Known limits + +Stated in the terms the reviewers of this benchmark used, because an objection published by the author is a caveat and the same objection found by a reader is a refutation. + +- **The divergence bands come from one dataset family.** Every frequency-stratified number in section 2 is measured on one OpenTelemetry demo stream. The LogHub sets exercise only the engine coverage exhibit, not the split. The mechanism is source-verified and order-driven, so the direction should transfer; the magnitudes are properties of this stream. +- **The probe pass shares traffic between the instances.** Both ingest every probe, so their states converge during the measurement, and the reported divergence understates, in expectation, what a fleet with no shared traffic would see. The direction is net rather than guaranteed per probe: a probe that matches different clusters in the two instances wildcards one side's template only, so a later probe of a different shape can flip from agree to differ because of the shared ingest. +- **The engine half-split feeds each half shuffled.** The protocol is Drain3's, so each engine instance reads its half in a random order, and a multi-line event's lines land in both halves and next to unrelated records. Fed in stream order, the reference run names 133,173 of 197,430 records as events of their own; each shuffled half names 62,274 of 98,715 on average. The same-line comparison is therefore scored on records each instance named as an event of their own. The engine runs shared the host with another session's Splunk container, using one to five cores, for the reference run and trials 1 to 8; it was stopped for trials 9 and 10 and for `bench/engine_split_context.py`, which reproduced the template-hash difference with the same answer on every repeat. +- **The seen-once stratum has no operational counterpart.** A shape that occurs exactly once in the whole stream is processed by exactly one forwarder and receives exactly one name. The row is reported because the falsifier on it fired against log10x and because the mechanism is instructive, not because a fleet behaves that way. +- **The half-split indicts an architecture many platforms do not run.** A deployment that mines centrally against one model has no cross-instance problem, and a reader running one correctly concludes that section 2 does not apply to them. Section 1 still does. +- **The engine artifacts are vendor-produced.** The coverage experiments run a pinned vendor-built Docker image, digest recorded in `results/coverage.md`, and this benchmark does not build the engine from source. The otel-sample templates asset is the encoding run's own output dictionary, which is why its resolve rate is a gate and not evidence, and the OpenTelemetry demo's source is in the engine's default symbol library. The invented-vocabulary probe exists because it cannot be answered by anything compiled with sight of anything. +- **The encoded artifact is for analysis.** `encoded.log` carries `pattern=` and `patternHash=` fields for tooling. No storage-reduction figure may be computed from it, and none appears here. +- **Part of the engine's template identity is deployment-specific.** 82.8% of templates on the published artifact carry environment values in envelope literals, so a redeployment with a new image tag or pod hash changes those templates. The pattern name keys on the selected words, not the envelope, so it is expected to hold; cross-deployment agreement of pattern names is unmeasured, and this is the concrete reason it could be below 1.0. +- **This sample is a weak surface for multi-line claims.** 2,311 of 2,316 exception bursts come from one container emitting one error shape, so the burst statistics describe one shape repeated, and `results/multiline.txt` says so in its scope note. +- **drain3 is pinned at 0.9.11 and a version bump is a re-run gate, not a routine update.** `bench/replay.py` and `bench/identifier.py` assert current library behaviour, and both refuse to run under `python -O` so the asserts stay live. The maintained fork is covered: see below. diff --git a/splunk-license/README.md b/splunk-license/README.md index 8848a77..ddf565a 100644 --- a/splunk-license/README.md +++ b/splunk-license/README.md @@ -1,205 +1,48 @@ -# Splunk licence-metered volume, with and without 10x in front - -One benchmark in the [log10x benchmarks](../) repository. It answers one -question: Splunk bills on what its licence meter counts, so what does putting -the 10x Receiver in front of Splunk do to that number, rather than to a file -size on disk. - -## The question - -A file shrinking on disk and Splunk's licence meter are different -measurements. The meter counts what the indexer ingests, which is neither the -file size on disk nor what the forwarder sent. This run reads the meter. - -The same capture, two arms, one difference. Arm one ships the raw file. -Arm two ships the same file after the 10x Receiver has compacted it, with the -three settings the Splunk app requires: `varMaxRecurIndexes: 0`, -`timestampZone: UTC` and `maxPerObject: 1`. Same forwarder, same -index settings, same instance, same licence day. - -## The result - -Two runs on the same capture, the same Splunk 10.4.3, the same forwarder path and the same -licence meter. The full table for each, with the Monitoring Console's own output and the raw -rollup line, is in [`results-2026-10-10/results.md`](results-2026-10-10/results.md) and -[`results/results.md`](results/results.md). - -| Run | Engine | Splunk app | Baseline, metered bytes | Compact, metered bytes | Licence-metered reduction | -|---|---|---|---:|---:|---:| -| 2026-10-10 | 1.1.139 | 1.1.5 | 214,841,731 | 77,532,468 | **63.91%** | -| 2026-09-18 | 1.1.79 | commit `9a203c6` | 214,841,731 | 80,653,626 | 62.46% | - -**63.91%** on the current engine and app, measured by Splunk's licence meter on Splunk -10.4.3, through a Universal Forwarder, on the Enterprise download trial. The compact arm is -the compact events, 74,338,244 metered bytes, and the template dictionary, 3,194,224. The -same two files measured on disk give 63.87%; the next section explains the gap. - -The 2026-09-18 run's compact arm also carried the app's searchable copy of each template, -2,441,744 metered bytes, which the app now writes unmetered. The rest of the difference is -the engine: 74,338,244 metered bytes of compact events against 75,017,247. - -Splunk's own dated figure for its own tool is "customers using Edge Processor can routinely -reduce ingest volumes by 30-50% without losing analytical value" ([Splunk blog, "From Data -Chaos to Clarity: The Complete Guide to Splunk Data Management", 2026-04-20](https://www.splunk.com/en_us/blog/platform/the-complete-guide-to-splunk-data-management.html)). -Both runs are above that range on this capture. - -**Every compact event expands back to its original text.** In both runs all 157,228 come back -byte-identical, in UTC and in any other timezone, read through the app's own macro and -compared against the text that went in. - -## The three things the licence meter sees that a file ratio does not - -**Line terminators are not billed.** The meter counts the bytes of each event's -raw text, and Splunk has already stripped the line break by then. The capture is -215,039,161 bytes on disk and 214,841,731 metered, exactly its 197,430 newlines -short. The compact arm loses its own newlines the same way, and as a -share of itself it loses more of them, 0.21% against the baseline's 0.09%, -because its bytes are fewer while its records are of the same order. So the -metered ratio comes out slightly better than the file ratio on the same two -files: 63.91% against 63.87% in the 2026-10-10 run, where nothing else is counted. - -**The template dictionary is billed.** The compact events are unreadable without -it, so a deployment sends it, and Splunk meters it like anything else. It is -inside the compact arm here for that reason. - -**The app's searchable copy of each template.** When the app's Consume KV alert -fills the KV store, it also writes each template into Splunk as a searchable -`tenx_dml_pure` event, sent here to the compact arm's own index so the arm is -charged for whatever the meter counts. In the 2026-09-18 run the app wrote that -copy through `/services/receivers/simple` and the meter counted 2,441,744 bytes. -The app writes it with `collect` as sourcetype `stash`, which Splunk does not -meter: 0 bytes in the 2026-10-10 run. - -**The event time is free.** App 1.1.5 sets a compact event's `_time` from the -timestamp in its original line while the event is parsed. The compact events -metered exactly their raw text, 74,338,244 bytes, with that setting on. - -## The daily rollup has no per-arm figure in it - -The obvious place to read GB/day is `index=_internal source=*license_usage.log -type=RolloverSummary`. On Splunk 10.4.3 that rollup is a single line per licence -pool, carrying the day's total `b=` and no `idx`, `st`, `s` or `h`. Split it by -index and nothing comes back. The per-arm split has to come from `type=Usage`, -the per-minute records, binned to the same day. - -Splunk's own console does the same: the Monitoring -Console's Historic License Usage view reads RolloverSummary while it is unsplit -and switches to Usage the moment it is split by index, source, host or -sourcetype. The run reports both, keeps the empty output of the by-index rollup -query so the next reader does not think the run failed, and checks the one -identity that ties them together: the per-index Usage bytes sum to the rollup's -own daily total to the byte, 292,374,199 in the 2026-10-10 run and 295,495,357 in -the 2026-09-18 run. - -## Expansion - -Every one of the 157,228 compact events was read back through the app's own -`tenx-inflate` macro and compared, byte for byte, against the text that went in. -**All 157,228 come back byte-identical** in both runs, with the search head in UTC -and with it in the instance's own timezone. `expansion.json` and -`expansion_local_tz.json` in each results directory carry the two checks. - -The three Receiver settings are what the app's expansion relies on: - -- `varMaxRecurIndexes: 0`: the app does not expand back-referenced templates, - so the compact form carries none. The engine's own round trip on the same - compact form is byte-identical. -- `timestampZone: UTC`: the app renders timestamps in UTC, so the compact form - stores them in UTC. -- `maxPerObject: 1`: the app stores one timestamp format per template. At `1` - the first timestamp in an event keeps its slot and any later one round-trips - as ordinary text. `template_stats.py` fails the run if any template carries - two slots. - -Each run installed the app commit named in its results file. The 2026-09-18 -commit's changes are in the app's `main`: Java quoted literals in timestamp -formats (`log-10x/splunk-app` #11), UTC rendering with exact sub-second digits -(#14), and a KV store keyed on the trimmed template hash (#13). The 2026-10-10 -run installed app 1.1.5 (`log-10x/splunk-app` #41), which dates each compact -event by its original line, so the harness counts and reads compact events over -all time rather than over the last hours. - -## What is claimed and what is not - -- **The volume figures are what this Splunk instance metered**, on one capture, - on one licence day, through the forwarder path named in the results file. They - are not a claim about a customer's estate, whose data, sourcetypes and - forwarder topology are all different. -- **Every compact event expands to its original text on both runs**, byte for - byte, in both timezones, with the three Receiver settings above. -- **One capture.** 36 Kubernetes containers from one small demo cluster, a - Fluentd envelope around each container's stdout line. -- **Nothing about indexed storage, search speed or retention.** The meter is - about ingest. -- **Nothing about Splunk Cloud**, whose metering and ingest path are not this. - -## Reproduce +# Splunk license-metered volume, with and without 10x in front + +Splunk bills on what its license meter counts, which is neither the file size on disk nor what the forwarder sent. This test sends the same capture to Splunk twice, once as it is and once after the 10x engine has turned it into compact events, and reads the meter for each. The compact arm metered 63.91% fewer bytes, and every compact event read back as its original text. + +Compact events cannot be read without their template dictionary, so the dictionary goes to Splunk too and its metered bytes count against the compact arm. The 10x Splunk app expands each compact event back to its original text at search time. + +## Result + +License-metered bytes for the run on 2026-10-10, from Splunk's own license usage log. + +| Arm | What Splunk ingested | Metered bytes | GB/day | +|---|---|---:|---:| +| Baseline | the capture as it is | 214,841,731 | 0.200087 | +| Compact | compact events | 74,338,244 | 0.069233 | +| Compact | template dictionary | 3,194,224 | 0.002975 | +| Compact | total | 77,532,468 | 0.072208 | + +The compact total is 63.91% below the baseline. Measured as files on disk, the same two arms give 63.87%. The gap is line breaks, which the meter does not count and which make up more of the smaller compact arm (0.21% against 0.09%). + +- All 157,228 compact events came back byte-identical to the text that went in, with the search head in UTC and in the instance's own timezone. +- Per Kubernetes container with at least 100 events, the reduction ran from 57.86% for ad to 69.27% for opensearch. These figures leave the dictionary out, which makes each one slightly better than in a deployment. + +## How it was measured + +The data is a 215,039,161 byte capture of 197,430 lines from 36 Kubernetes containers in a small demo cluster, each line in a Fluentd envelope. Both arms went through the same Universal Forwarder to one Splunk Enterprise 10.4.3 instance on the download trial license, into separate indexes on the same license day. On this version the daily `type=RolloverSummary` line carries only the day's total, so the split by index comes from the per-minute `type=Usage` records, as the Monitoring Console does. The per-index bytes add up to the rollup's daily total of 292,374,199 bytes exactly. + +## Limits + +- One capture from one demo cluster on one license day. A production estate has its own data, sourcetypes and forwarder topology. +- The figure is ingest only. Indexed storage, search speed and retention were not measured. +- Splunk Cloud, which meters by a different path, was not tested. + +## Run it + +The license figures can be recomputed from the saved rollup lines without Splunk or Docker: ```sh -./run.sh # everything, then tear the containers down -KEEP=1 ./run.sh # leave them up -./run.sh encode licence # some stages only, in the order given +# from the repository root +python3 splunk-license/report.py --dir splunk-license --data splunk-license/data \ + --results splunk-license/results-2026-10-10 --check + +# the full run: a throwaway Splunk container, torn down at the end +splunk-license/run.sh ``` -The defaults reproduce the 2026-10-10 run: engine `log10x/pipeline-10x:1.1.139` pinned by -digest, Splunk and the Universal Forwarder at 10.4.3, results in `results-2026-10-10/`. The -2026-09-18 run is `E21_ENGINE=ghcr.io/log-10x/edge-10x:1.1.79 E21_RESULTS=$PWD/results` with -the app at commit `9a203c6` in `E21_APP_DIR`. - -Stages are `data encode groundtruth splunk ingest kv expand licence report`. -Needs Docker with about 10 GB of free disk, since the Splunk image alone is -6.5 GB, and roughly two hours, most of it waiting for a licence day to end. -It stands up a throwaway Splunk container on the Enterprise download trial and -destroys it; it touches no existing Splunk. - -### The licence day, and why the container gets a timezone - -`RolloverSummary` is written when the licence day ends, at midnight in the -licence manager's own local time, and a partial day is not a result. A run -started at nine in the morning would otherwise wait fifteen hours for a number. -So `run.sh` gives the container a whole-hour timezone chosen to put its next -local midnight at least 75 minutes ahead of container start, which is long -enough for the whole ingest to sit inside one licence day with room to spare. -The rollover is an ordinary one and the day it summarises holds the entire run; -only its wall-clock position is chosen. `E21_MIN_LEAD` sets the margin. - -### Sizing to the ceiling rather than around it - -The download trial meters 500 MB a day. Both arms together are about 296 MB, so -the whole capture fits inside one licence day on one instance, and the two arms -go into separate indexes on that same day. Nothing here raises, resets or works -around the quota. - -| File | What it is | -|---|---| -| `run.sh` | the whole run, in stages | -| `tenx-encode-splunk.config.yaml` | the published OUTER round-trip encode config | -| `tenx-decode.config.yaml` | the published decode config, for the engine round trip | -| `conf/indexer/`, `conf/uf/` | every Splunk and forwarder setting, as it ran | -| `conf/tenx_config.conf` | the one app setting changed, and why | -| `conf/searches/` | the licence searches, including the Monitoring Console's own | -| `build_groundtruth.py` | what each compact event's original text was, with its invariants | -| `template_stats.py` | fails the run if any template carries more timestamp slots than the app can reconstruct | -| `worst_slice.py` | the reduction per Kubernetes container, so the worst slice is named | -| `verify_expansion.py` | every compact event read back through the app, against that | -| `report.py` | renders `results/results.md` from the kept RolloverSummary lines | -| `results-2026-10-10/` | the 2026-10-10 run, engine 1.1.139 and app 1.1.5 | -| `results/` | the 2026-09-18 run, engine 1.1.79 | - -`report.py` recomputes the licence figures from -`results/license_usage_rollover.log`, the rollup lines kept verbatim, rather -than from a search result, so the table and the primary log agree by -construction. - -## Data - -`otel-sample-200mb.log` from -[otel-sample-v2](https://github.com/log-10x/config/releases/tag/otel-sample-v2), -215,039,161 bytes, 197,430 lines, sha256 -`aa79b9349a4123d88936fd064f5eccd9fd84f45a1fd03cff3f64c259715c2432`. -[otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1) -carries the same file (gzip sha256 `c118e55f…` in both releases). The same -asset the [`otel-denominators/`](../otel-denominators/) and -[`clickhouse-clickstack/`](../clickhouse-clickstack/) benchmarks run on, which -is what lets the licence-metered figure here be set beside the file ratios -there. +A full run needs Docker, about 10 GB of free disk and roughly two hours, most of it waiting for the license day to end. The run touches no existing Splunk. + +The full record, with every Splunk setting, the encode settings the app needs and all the results tables, is in [RECORD.md](RECORD.md). diff --git a/splunk-license/RECORD.md b/splunk-license/RECORD.md new file mode 100644 index 0000000..8848a77 --- /dev/null +++ b/splunk-license/RECORD.md @@ -0,0 +1,205 @@ +# Splunk licence-metered volume, with and without 10x in front + +One benchmark in the [log10x benchmarks](../) repository. It answers one +question: Splunk bills on what its licence meter counts, so what does putting +the 10x Receiver in front of Splunk do to that number, rather than to a file +size on disk. + +## The question + +A file shrinking on disk and Splunk's licence meter are different +measurements. The meter counts what the indexer ingests, which is neither the +file size on disk nor what the forwarder sent. This run reads the meter. + +The same capture, two arms, one difference. Arm one ships the raw file. +Arm two ships the same file after the 10x Receiver has compacted it, with the +three settings the Splunk app requires: `varMaxRecurIndexes: 0`, +`timestampZone: UTC` and `maxPerObject: 1`. Same forwarder, same +index settings, same instance, same licence day. + +## The result + +Two runs on the same capture, the same Splunk 10.4.3, the same forwarder path and the same +licence meter. The full table for each, with the Monitoring Console's own output and the raw +rollup line, is in [`results-2026-10-10/results.md`](results-2026-10-10/results.md) and +[`results/results.md`](results/results.md). + +| Run | Engine | Splunk app | Baseline, metered bytes | Compact, metered bytes | Licence-metered reduction | +|---|---|---|---:|---:|---:| +| 2026-10-10 | 1.1.139 | 1.1.5 | 214,841,731 | 77,532,468 | **63.91%** | +| 2026-09-18 | 1.1.79 | commit `9a203c6` | 214,841,731 | 80,653,626 | 62.46% | + +**63.91%** on the current engine and app, measured by Splunk's licence meter on Splunk +10.4.3, through a Universal Forwarder, on the Enterprise download trial. The compact arm is +the compact events, 74,338,244 metered bytes, and the template dictionary, 3,194,224. The +same two files measured on disk give 63.87%; the next section explains the gap. + +The 2026-09-18 run's compact arm also carried the app's searchable copy of each template, +2,441,744 metered bytes, which the app now writes unmetered. The rest of the difference is +the engine: 74,338,244 metered bytes of compact events against 75,017,247. + +Splunk's own dated figure for its own tool is "customers using Edge Processor can routinely +reduce ingest volumes by 30-50% without losing analytical value" ([Splunk blog, "From Data +Chaos to Clarity: The Complete Guide to Splunk Data Management", 2026-04-20](https://www.splunk.com/en_us/blog/platform/the-complete-guide-to-splunk-data-management.html)). +Both runs are above that range on this capture. + +**Every compact event expands back to its original text.** In both runs all 157,228 come back +byte-identical, in UTC and in any other timezone, read through the app's own macro and +compared against the text that went in. + +## The three things the licence meter sees that a file ratio does not + +**Line terminators are not billed.** The meter counts the bytes of each event's +raw text, and Splunk has already stripped the line break by then. The capture is +215,039,161 bytes on disk and 214,841,731 metered, exactly its 197,430 newlines +short. The compact arm loses its own newlines the same way, and as a +share of itself it loses more of them, 0.21% against the baseline's 0.09%, +because its bytes are fewer while its records are of the same order. So the +metered ratio comes out slightly better than the file ratio on the same two +files: 63.91% against 63.87% in the 2026-10-10 run, where nothing else is counted. + +**The template dictionary is billed.** The compact events are unreadable without +it, so a deployment sends it, and Splunk meters it like anything else. It is +inside the compact arm here for that reason. + +**The app's searchable copy of each template.** When the app's Consume KV alert +fills the KV store, it also writes each template into Splunk as a searchable +`tenx_dml_pure` event, sent here to the compact arm's own index so the arm is +charged for whatever the meter counts. In the 2026-09-18 run the app wrote that +copy through `/services/receivers/simple` and the meter counted 2,441,744 bytes. +The app writes it with `collect` as sourcetype `stash`, which Splunk does not +meter: 0 bytes in the 2026-10-10 run. + +**The event time is free.** App 1.1.5 sets a compact event's `_time` from the +timestamp in its original line while the event is parsed. The compact events +metered exactly their raw text, 74,338,244 bytes, with that setting on. + +## The daily rollup has no per-arm figure in it + +The obvious place to read GB/day is `index=_internal source=*license_usage.log +type=RolloverSummary`. On Splunk 10.4.3 that rollup is a single line per licence +pool, carrying the day's total `b=` and no `idx`, `st`, `s` or `h`. Split it by +index and nothing comes back. The per-arm split has to come from `type=Usage`, +the per-minute records, binned to the same day. + +Splunk's own console does the same: the Monitoring +Console's Historic License Usage view reads RolloverSummary while it is unsplit +and switches to Usage the moment it is split by index, source, host or +sourcetype. The run reports both, keeps the empty output of the by-index rollup +query so the next reader does not think the run failed, and checks the one +identity that ties them together: the per-index Usage bytes sum to the rollup's +own daily total to the byte, 292,374,199 in the 2026-10-10 run and 295,495,357 in +the 2026-09-18 run. + +## Expansion + +Every one of the 157,228 compact events was read back through the app's own +`tenx-inflate` macro and compared, byte for byte, against the text that went in. +**All 157,228 come back byte-identical** in both runs, with the search head in UTC +and with it in the instance's own timezone. `expansion.json` and +`expansion_local_tz.json` in each results directory carry the two checks. + +The three Receiver settings are what the app's expansion relies on: + +- `varMaxRecurIndexes: 0`: the app does not expand back-referenced templates, + so the compact form carries none. The engine's own round trip on the same + compact form is byte-identical. +- `timestampZone: UTC`: the app renders timestamps in UTC, so the compact form + stores them in UTC. +- `maxPerObject: 1`: the app stores one timestamp format per template. At `1` + the first timestamp in an event keeps its slot and any later one round-trips + as ordinary text. `template_stats.py` fails the run if any template carries + two slots. + +Each run installed the app commit named in its results file. The 2026-09-18 +commit's changes are in the app's `main`: Java quoted literals in timestamp +formats (`log-10x/splunk-app` #11), UTC rendering with exact sub-second digits +(#14), and a KV store keyed on the trimmed template hash (#13). The 2026-10-10 +run installed app 1.1.5 (`log-10x/splunk-app` #41), which dates each compact +event by its original line, so the harness counts and reads compact events over +all time rather than over the last hours. + +## What is claimed and what is not + +- **The volume figures are what this Splunk instance metered**, on one capture, + on one licence day, through the forwarder path named in the results file. They + are not a claim about a customer's estate, whose data, sourcetypes and + forwarder topology are all different. +- **Every compact event expands to its original text on both runs**, byte for + byte, in both timezones, with the three Receiver settings above. +- **One capture.** 36 Kubernetes containers from one small demo cluster, a + Fluentd envelope around each container's stdout line. +- **Nothing about indexed storage, search speed or retention.** The meter is + about ingest. +- **Nothing about Splunk Cloud**, whose metering and ingest path are not this. + +## Reproduce + +```sh +./run.sh # everything, then tear the containers down +KEEP=1 ./run.sh # leave them up +./run.sh encode licence # some stages only, in the order given +``` + +The defaults reproduce the 2026-10-10 run: engine `log10x/pipeline-10x:1.1.139` pinned by +digest, Splunk and the Universal Forwarder at 10.4.3, results in `results-2026-10-10/`. The +2026-09-18 run is `E21_ENGINE=ghcr.io/log-10x/edge-10x:1.1.79 E21_RESULTS=$PWD/results` with +the app at commit `9a203c6` in `E21_APP_DIR`. + +Stages are `data encode groundtruth splunk ingest kv expand licence report`. +Needs Docker with about 10 GB of free disk, since the Splunk image alone is +6.5 GB, and roughly two hours, most of it waiting for a licence day to end. +It stands up a throwaway Splunk container on the Enterprise download trial and +destroys it; it touches no existing Splunk. + +### The licence day, and why the container gets a timezone + +`RolloverSummary` is written when the licence day ends, at midnight in the +licence manager's own local time, and a partial day is not a result. A run +started at nine in the morning would otherwise wait fifteen hours for a number. +So `run.sh` gives the container a whole-hour timezone chosen to put its next +local midnight at least 75 minutes ahead of container start, which is long +enough for the whole ingest to sit inside one licence day with room to spare. +The rollover is an ordinary one and the day it summarises holds the entire run; +only its wall-clock position is chosen. `E21_MIN_LEAD` sets the margin. + +### Sizing to the ceiling rather than around it + +The download trial meters 500 MB a day. Both arms together are about 296 MB, so +the whole capture fits inside one licence day on one instance, and the two arms +go into separate indexes on that same day. Nothing here raises, resets or works +around the quota. + +| File | What it is | +|---|---| +| `run.sh` | the whole run, in stages | +| `tenx-encode-splunk.config.yaml` | the published OUTER round-trip encode config | +| `tenx-decode.config.yaml` | the published decode config, for the engine round trip | +| `conf/indexer/`, `conf/uf/` | every Splunk and forwarder setting, as it ran | +| `conf/tenx_config.conf` | the one app setting changed, and why | +| `conf/searches/` | the licence searches, including the Monitoring Console's own | +| `build_groundtruth.py` | what each compact event's original text was, with its invariants | +| `template_stats.py` | fails the run if any template carries more timestamp slots than the app can reconstruct | +| `worst_slice.py` | the reduction per Kubernetes container, so the worst slice is named | +| `verify_expansion.py` | every compact event read back through the app, against that | +| `report.py` | renders `results/results.md` from the kept RolloverSummary lines | +| `results-2026-10-10/` | the 2026-10-10 run, engine 1.1.139 and app 1.1.5 | +| `results/` | the 2026-09-18 run, engine 1.1.79 | + +`report.py` recomputes the licence figures from +`results/license_usage_rollover.log`, the rollup lines kept verbatim, rather +than from a search result, so the table and the primary log agree by +construction. + +## Data + +`otel-sample-200mb.log` from +[otel-sample-v2](https://github.com/log-10x/config/releases/tag/otel-sample-v2), +215,039,161 bytes, 197,430 lines, sha256 +`aa79b9349a4123d88936fd064f5eccd9fd84f45a1fd03cff3f64c259715c2432`. +[otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1) +carries the same file (gzip sha256 `c118e55f…` in both releases). The same +asset the [`otel-denominators/`](../otel-denominators/) and +[`clickhouse-clickstack/`](../clickhouse-clickstack/) benchmarks run on, which +is what lets the licence-metered figure here be set beside the file ratios +there.