From fd6e96f053978929531c182e0bf5ce8faf2cbe6c Mon Sep 17 00:00:00 2001 From: Tal Weiss <73443663+talwgx@users.noreply.github.com> Date: Fri, 13 Nov 2026 10:20:44 -0500 Subject: [PATCH] READMEs for 17 benchmarks in the published blog voice Each front page was rewritten from its RECORD.md in the voice of the published posts, then read against the record by a second reader. Every number traces to RECORD.md or the committed results. Two meaning fixes (clickstack-e2e-gaps, elastic-hot-tier); trial-license asides removed. splunk-license: the run-setup line no longer names the license type. Co-Authored-By: Claude Opus 5.5 --- clickhouse-clickstack/README.md | 45 +++++++++++++----------- clickhouse-cloud-elasticity/README.md | 38 ++++++++++---------- clickhouse-inflate/README.md | 50 ++++++++++++++------------- clickstack-e2e-gaps/README.md | 33 ++++++++---------- clickstack-e2e/README.md | 31 ++++++++--------- clickstack-rows-gate/README.md | 37 +++++++++----------- clickstack-rows/README.md | 34 +++++++++--------- cloudwatch-ia-accounting/README.md | 28 +++++++-------- datadog-flex-accounting/README.md | 41 ++++++++++++---------- datadog-service-control/README.md | 33 +++++++++--------- drain3-vs-log10x/README.md | 37 ++++++++++---------- e103-grouping/README.md | 40 ++++++++++++--------- elastic-hot-tier-budget/README.md | 32 ++++++++--------- elastic-hot-tier/README.md | 38 ++++++++++---------- elastic-store-four-ways/README.md | 38 ++++++++++---------- intent/README.md | 38 +++++++++++--------- naming-stability/README.md | 36 +++++++++---------- splunk-license/README.md | 2 +- 18 files changed, 316 insertions(+), 315 deletions(-) diff --git a/clickhouse-clickstack/README.md b/clickhouse-clickstack/README.md index f9006ca..ccf6286 100644 --- a/clickhouse-clickstack/README.md +++ b/clickhouse-clickstack/README.md @@ -1,38 +1,42 @@ # clickhouse-clickstack -How much disk do 10x compact events save in ClickHouse's ClickStack `otel_logs` schema? On 197,430 lines from the OpenTelemetry demo, the whole table came out 14.9% smaller, but the SQL functions that decode compact rows in ClickHouse returned the wrong text for 78% of them. Keeping the original text and adding the message type as a column inside the sort key took 13.1% off the table with no decoder and the text index intact. +On ClickHouse Cloud most of a log bill is compute, and only a little of it is the cost of keeping rows. This benchmark loaded 197,430 OpenTelemetry demo lines into ClickStack's published `otel_logs` schema, as original text and as 10x compact events. The compact table was 14.9% smaller, but the SQL that decodes it inside ClickHouse returned the wrong text for 78% of rows. Original text with a message-type column in the sort key was 13.1% smaller, with no decoder and the text index intact. -The 10x engine writes each message as a template hash plus its values and folds a multi-line event into one, so the lines became 159,170 compact events. +10x stores the text each log statement repeats once, as a template, and writes each message as the template's hash plus its values. The engine folds a multi-line event into one, so the lines became 159,170 compact events. -## Result +## Table size on disk at ZSTD(1) -Whole table on disk at ZSTD(1), with the 348,040-byte template dictionary in every compact total. +Every compact total includes the 348,040-byte template dictionary. | Layout | Rows | Bytes on disk | Smaller than published | Full-text index on Body | |---|---:|---:|---:|---| -| ClickStack as published | 197,430 | 8,225,547 | 0.0% | kept | -| Compact events in Body | 159,170 | 6,999,850 | 14.9% | kept | -| Compact, hash in sort key | 159,170 | 6,188,001 | 24.8% | kept | +| ClickStack as published | 197,430 | 8,225,547 | reference | kept | +| Original text, one row per event | 159,170 | 7,570,122 | 8% | kept | +| Compact events in Body | 159,170 | 6,999,850 | 14.9% | values only | +| Compact, hash in sort key | 159,170 | 6,188,001 | 24.8% | values only | | Compact, typed value arrays | 159,170 | 4,376,741 | 46.8% | none | | Original text, type in sort key | 197,430 | 7,148,605 | 13.1% | kept | | Original text, sorted by type | 197,430 | 6,451,087 | 21.6% | kept | -Against the original text of the same events, the compact form takes 23.55% off the `Body` column. The larger savings come mostly from ClickStack's full-text index on Body shrinking or going away, and sorting by type first weakens the time-range primary key. +The message is a minority of a ClickStack row, and the skip indexes and attribute maps take most of the bytes. Against the same 159,170 events as original text, the compact form takes 23.55% off Body and 7.53% off the table. The larger savings come from the full-text index on Body shrinking or going away. Sorting by type first weakens the time-range primary key. -- Through the SQL decoder, 78.0% of compact rows matched no captured line, and a text filter took 531 ms against 6 ms on the original rows. A Python decoder with the four rules the SQL lacks returned 24,494 of the 24,501 events without a timestamp slot byte-identical, so the text is intact and the SQL misreads it. log10x does not offer the SQL decoder for ClickHouse. -- Row for row, the compact text cost the same insert CPU as the original (1.64 s). Removing rows cut CPU more than proportionally: half the rows cost 39% of insert-plus-merge CPU and a quarter cost 17%. -- TTL RECOMPRESS does nothing on ClickStack, which declares CODEC(ZSTD(1)) on every column. Without that codec, the original-text table at ZSTD(12) was 29.9% smaller than at ZSTD(1), more than any layout above that keeps the text index. +## Reading compact rows back with SQL -## How it was measured +A one-word filter through the SQL decoder took 531 ms at its fastest, against 6 ms on the original rows. A Python decoder with the four template rules the SQL lacks returned 24,494 of the 24,501 events without a timestamp slot byte-identical, and the other 134,669 were not checked. The text is there, and the SQL misreads it. 10x does not offer the SQL decoder for ClickHouse. -The data is `otel-sample-200mb.log` from the otel-sample-v2 release, loaded into the published ClickStack DDL with all eight skip indexes and its sort key, minus the TTL placeholder. ClickHouse 26.8.3.105 ran in one container with 8 GiB and 8 CPUs. Every byte count is read from system.parts and system.parts_columns after OPTIMIZE FINAL. +## Insert and merge CPU -## Limits +Row for row, compact text cost the same insert CPU as the original, 1.64 s. Half the rows, sampled uniformly, cost 39% of the insert-plus-merge CPU and a quarter cost 17%, because fewer parts merge fewer times. Removing the two largest message types left 73% of the rows and 83% of the CPU, because the remaining rows cost more each to compress. -- The data is one demo capture in one container. -- ClickHouse's own 45x figure comes from one typed table per service, which this single-table test does not reproduce. -- CPU covers ClickHouse's insert and merge time, not the collector and not the write cost of ZSTD(12). -- No price is attached. +## How the tables were built and read + +The capture is otel-sample-200mb.log from the [otel-sample-v2 release](https://github.com/log-10x/config/releases/tag/otel-sample-v2), loaded into the ClickStack DDL as published minus its TTL placeholder, on ClickHouse 26.8.3.105 with 8 GiB and 8 CPUs. Bytes come from system.parts and system.parts_columns after OPTIMIZE FINAL, and CPU from system.query_log and system.part_log. + +## Scope of the run + +- One demo capture was measured, and its largest message type is the collector's debug exporter. +- [ClickHouse's 45x figure](https://clickhouse.com/blog/improve-compression-log-clustering) comes from one typed table per service, which this single-table test does not reproduce. +- CPU is ClickHouse's insert and merge time, without the collector. ## Run it @@ -42,8 +46,9 @@ python3 reference_decode.py --self-test # decode rules only, no Docker or down KEEP=1 ./run.sh # the storage tables, leaves ClickHouse up ./run_typed.sh # compact rows sorted by hash, and typed arrays ./run_labeled.sh # original text plus a message-type column +./run_compute.sh # insert and merge CPU against row count ``` -The scripts need Docker and about 4 GB of disk. The storage run downloads the capture and takes roughly 20 minutes, and the other two reuse its container. +The shell scripts need Docker and about 4 GB of disk. The storage run downloads the capture and takes about 20 minutes. -The full record, with the decode diagnosis and the compute, TTL, batch-size and query-at-scale runs, is in [RECORD.md](RECORD.md). +[RECORD.md](RECORD.md) has the decode diagnosis and the TTL, batch-size, recompression and query runs. diff --git a/clickhouse-cloud-elasticity/README.md b/clickhouse-cloud-elasticity/README.md index 5f47d5f..0452cce 100644 --- a/clickhouse-cloud-elasticity/README.md +++ b/clickhouse-cloud-elasticity/README.md @@ -1,44 +1,42 @@ # ClickHouse Cloud elasticity -When a ClickHouse Cloud service is sent fewer log rows, how far does its compute bill fall, and how soon? In three of four paired runs, writing 65.3% of the rows took the service from 4 units to 3, and billed compute settled at 0.749 of the full feed's rate. The step came 2.9 to 4.3 hours after the cut, and until then both arms paid for 4 units. In the other, which started at 3 units, the same cut changed nothing in 10.25 hours. +ClickHouse Cloud bills compute by the unit, so fewer rows show on the bill only when they let the autoscaler step down a size. One service took two feeds in four pairs of runs: the full feed of a 215 MB OpenTelemetry capture, then the 65.3% of its rows the 10x engine kept. From a 4-unit start, the kept rows took the service to 3 units in three pairs, and billed compute settled at 0.749 of the full feed's rate. The step came 2.9 to 4.3 hours after the cut, and until then both feeds paid for 4 units. From a 3-unit start, the same cut changed nothing in 10.25 hours. -The autoscaler's documented rule explains the difference: between 37.5% and 75% CPU it takes no action. On 4 units the busiest replica ran at a median 0.440 to 0.486 of its CPU with every row and 0.337 to 0.351 with the kept rows, below the band. On 3 units both arms stayed inside the band. +## Units and billed compute in each pair -## Result - -One unit is 8 GiB and 2 vCPU. Arm A writes every row, arm B the 65.3% the 10x engine kept. - -| Pair | Arm A's size set by | Settled units, A to B | Billed $ per 15 min, A to B | Hours from cut to step down | +| Pair | Full feed's size set by | Units, full feed to kept rows | Billed $ per 15 min | Hours from cut to step down | |---|---|---:|---:|---:| | 1 | the autoscaler | 4.0 to 2.997 | 0.298 to 0.224 | 4.33 | | 2 | the autoscaler, after pair 1 | 3.0 to 3.0 | 0.224 to 0.224 | no step in 10.25 | -| 3 | config, released at the cut | 4.0 to 3.0 | 0.298 to 0.224 | 4.21 | -| 4 | config, released at the cut | 4.0 to 3.0 | 0.298 to 0.224 | 2.91 | +| 3 | a 16 GiB minimum | 4.0 to 3.0 | 0.298 to 0.224 | 4.21 | +| 4 | a 16 GiB minimum | 4.0 to 3.0 | 0.298 to 0.224 | 2.91 | + +Billed compute is the usageCost API's 15-minute postings, $1.196 an hour at 4 units against $0.896 at 3. Rows fell to 0.653, and the service made one step, from 16 to 12 GiB per replica, which is 0.75. -The billed rate is $1.196 an hour at 4 units against $0.896 at 3. Size fell to 0.75 while rows fell to 0.653, because the service made one step, from 16 GiB to 12 GiB per replica. +- A service at its minimum saves nothing. Below the load that lifts the full feed off 2 units, between 50,000 and 70,000 rows a second, both feeds cost $0.60 an hour. +- Why pair 4 stepped down 1.3 hours sooner than pairs 1 and 3 was not measured. -- The autoscaler's documented scale-down window is 3 hours. Why pair 4 stepped 1.3 hours sooner than pairs 1 and 3 was not measured. -- Below the load that lifts the full feed off the 2-unit minimum, between 50,000 and 70,000 rows a second, both arms cost $0.60 an hour. +## The autoscaler's CPU band -## How it was measured +By its documentation, the autoscaler takes no action between 37.5% and 75% CPU. On 4 units the busiest replica ran at a median 0.440 to 0.486 of its CPU with the full feed, inside the band, and 0.337 to 0.351 with the kept rows, below it. In pair 2, on 3 units, both feeds stayed inside the band, at 0.637 and 0.455. -The service ran on the Scale tier in AWS us-east-1, ClickHouse 26.6, with 2 replicas autoscaling from 8 to 32 GiB each. Both arms wrote into the ClickStack `otel_logs` schema under seven queries a minute. Arm A replayed all 171,807 records of a 215 MB OpenTelemetry sample at 70,000 rows a second, arm B the 112,205 kept rows at 45,716. In pairs 3 and 4, config pinned arm A at 16 GiB per replica until the cut, so each step down was the autoscaler's own. Size and CPU were read every 60 seconds, dollars from the usageCost API every 10 minutes. +## How the service and its bill were read + +The service ran on the Scale tier in AWS us-east-1, ClickHouse 26.6, with 2 replicas autoscaling from 8 to 32 GiB each, where a unit is 8 GiB and 2 vCPUs. Both feeds wrote into the ClickStack `otel_logs` schema under seven queries a minute, the full feed at 70,000 rows a second and the kept rows at 45,716. In pairs 3 and 4 a 16 GiB minimum set the full feed's size and was lowered to 8 GiB at the cut, so each step down was the autoscaler's own. Size and CPU were read every 60 seconds and billed compute every 10 minutes. ## Limits -- One service shape in one region, with seven queries a minute instead of a production mix. -- Both arms inserted from staged tables, so neither the engine nor the parsing of network inserts ran live. +- One service shape in one region, with a fixed query load. +- Both feeds were inserted from staged tables, so neither the engine nor network insert parsing ran live. - Which replica the autoscaler reads, and how often it decides, was not measured. ## Run it -The pair ratios recompute from committed logs: +The pair ratios recompute offline from the committed logs: ``` cd clickhouse-cloud-elasticity python3 cloud/arms_report.py --raw results/raw --lag-min 30 ``` -A full rerun needs a ClickHouse Cloud API key, Docker and a `clickhouse` client, and this run spent $42.61 of credit. The steps are in [RECORD.md](RECORD.md#rerun). - -[RECORD.md](RECORD.md) has the run summary and rerun steps, and [RESULT.md](RESULT.md) every figure with the file it came from. +A full rerun needs a ClickHouse Cloud API key, Docker and a `clickhouse` client, and this run spent $42.61 of credit. [RECORD.md](RECORD.md#rerun) has the rerun steps, and [RESULT.md](RESULT.md) every figure with its source. diff --git a/clickhouse-inflate/README.md b/clickhouse-inflate/README.md index 3a910db..1ba06f3 100644 --- a/clickhouse-inflate/README.md +++ b/clickhouse-inflate/README.md @@ -1,46 +1,48 @@ # clickhouse-inflate -What does it cost ClickHouse to turn compact log events back into text at query time? Here the events sit in a table, and a view decodes them with SQL functions from the log10x ClickHouse app. With the SQL in `install-fixed.sql`, decoding all 137,418 rows took 548 ms and peaked at 41.1 MiB. The SQL in `install-buggy.sql` could not finish the same query at ClickHouse defaults and aborted at the server's memory cap. +A ClickHouse view runs its SQL each time a query reads it. Here a view decodes compact log events, each a template hash plus the line's values, with SQL functions from the log-10x/clickhouse-app repository. Decoding all 137,418 rows took 548 ms and peaked at 41.1 MiB when the decode lambda took its arrays as arrayMap arguments. With a lambda that reads the same arrays by subscript, the query hit the server's memory cap at ClickHouse defaults and aborted. -A `SELECT count()` over the view measures none of this, because ClickHouse answers it from a projection and prunes the decode out of the plan. The two SQL files differ by one line per function. The buggy lambda reaches its arrays by subscript, so ClickHouse copies each array once per mapped element. The fixed one passes them to arrayMap as arguments, which ClickHouse walks element by element. +A lambda that indexes into its arrays captures them, and ClickHouse copies a captured array once per mapped element, so a row with N literals materializes N copies of its N-element arrays. Arrays passed to arrayMap as arguments are walked element by element. -## Result +A `SELECT count()` over the view measures none of this, because ClickHouse answers it from a built-in count projection and its plan has no function nodes. -Figures are from results.json, with times averaged over 3 runs after discarding the first. +## What ClickHouse's query log recorded -| Query | SQL file | Outcome | Time | Peak memory | +Times average the second and third of three runs. Every figure is from [results.json](results.json). + +| Query | Decode lambda | Outcome | Time | Peak memory | |---|---|---|---:|---:| -| `SELECT count()` | buggy | 1 row, 24 bytes read | 16 ms | not recorded | -| `sum(length(decoded_log))` | buggy | aborts, code 241 | n/a | 7.20 GiB | -| `sum(length(decoded_log))`, bounded settings | buggy | completes | 4,290 ms | 2.86 GiB | -| `sum(length(decoded_log))` | fixed | completes | 548 ms | 41.1 MiB | -| `SELECT decoded_log`, `FORMAT Null` | buggy | aborts, code 241 | n/a | 7.20 GiB | -| `SELECT decoded_log`, `FORMAT Null` | fixed | completes | 535 ms | 40.9 MiB | +| `SELECT count()` | not in the plan | 1 row, 24 bytes read | 16 ms | not recorded | +| `sum(length(decoded_log))` | by subscript | aborts, code 241 | n/a | 7.2 GiB | +| same, bounded settings | by subscript | completes | 4,290 ms | 2.86 GiB | +| `sum(length(decoded_log))` | as arguments | completes | 548 ms | 41.1 MiB | +| `SELECT decoded_log`, `FORMAT Null` | by subscript | aborts, code 241 | n/a | 7.2 GiB | +| `SELECT decoded_log`, `FORMAT Null` | as arguments | completes | 535 ms | 40.9 MiB | +| `sum(length(decoded_log))`, format-preserving view | as arguments | completes | 600 ms | 78 MiB | -The 7.20 GiB is above this host's 6.89 GiB cap because ClickHouse records the usage a refused 4 GiB allocation would have reached. The buggy decode completed in 3 of 3 runs with a block size of 64, 4 threads and a per-query limit of 6000000000 bytes. +The 7.2 GiB is above this host's 6.89 GiB cap because ClickHouse records the usage a refused 4 GiB allocation would have reached. The bounded run, with a block size of 64, 4 threads and a per-query limit of 6000000000 bytes, completed in 3 of 3 runs. -- Both files give byte-identical output over all 137,418 rows, 29,065,810 bytes. -- The decode needs 800,397 array elements and the buggy SQL materializes 189,949,597. One template with 818 literals, on 275 rows, accounts for 96.9% of the materialized elements. -- Replacing the functions alone leaves the views on the old body, because ClickHouse copies a function body into a view when it creates the view. The fixed install file recreates the views. +- Both lambdas return byte-identical output over all 137,418 rows, 29,065,810 bytes. +- The decode needs 800,397 array elements and the subscript lambda materializes 189,949,597, 96.9% of them from one template with 818 literals on 275 rows. +- Replacing the functions alone leaves the views on the old body, because ClickHouse copies a function body into each view at creation. [install-fixed.sql](install-fixed.sql) recreates them. -## How it was measured +## How the queries were run -ClickHouse 25.8.28.1 ran in Docker, pinned to 8 GiB and 8 CPUs. The data is 500 compact events the 10x engine wrote from the OpenTelemetry demo's Kubernetes logs, cycled to 137,418 rows. The script runs every query on the buggy SQL, then the decodes on the fixed SQL, and compares the outputs byte for byte. Times and peak memory come from ClickHouse's query log. +ClickHouse 25.8.28.1 ran in Docker at default query settings, pinned to 8 GiB and 8 CPUs. The data is 500 compact events the 10x engine wrote from the OpenTelemetry demo's Kubernetes logs, repeated to 137,418 rows. The queries ran on the subscript version in [install-buggy.sql](install-buggy.sql), then on [install-fixed.sql](install-fixed.sql). For the byte comparison, the subscript version decodes at a block size of 64 so that it completes. ## Limits -- The test ran on one host with one ClickHouse version. Times move with hardware, and the abort point moves with the memory cap, which ClickHouse sets at 0.9 times the RAM it can see. -- The data set is small, 500 events cycled to fill the table. -- The byte-identity check compares the two files' output against each other, not against the original log text. +- One host and one ClickHouse version were tested. Times move with hardware, and the abort point moves with the memory cap, which ClickHouse sets by default at 0.9 times the RAM it can see. +- The byte comparison checks the two lambdas against each other, never against the original log text. [clickhouse-clickstack](../clickhouse-clickstack/) makes that check, and there this SQL returns the wrong text for most rows. ## Run it ``` cd clickhouse-inflate -./run.sh # pulls the pinned ClickHouse image and writes results.json -SKIP_PREFIX=1 ./run.sh # skips the two buggy-SQL queries that abort +./run.sh # pulls the pinned ClickHouse image and writes results.json +KEEP=1 ./run.sh # the same run, leaving the container up afterwards ``` -A run needs Docker. The two aborting queries take 2 seconds to 10 minutes, depending on host memory pressure. +A run needs Docker, and the two aborting queries take between 2 seconds and 10 minutes, depending on the host's memory pressure. -The full record is in [RECORD.md](RECORD.md): the abort message, the memory cap, plan node counts, the SQL change and the figures this harness does not reproduce. +[RECORD.md](RECORD.md) has the abort message, how ClickHouse sets its memory cap, the plan node counts and the SQL of both lambdas. diff --git a/clickstack-e2e-gaps/README.md b/clickstack-e2e-gaps/README.md index a92a180..8736732 100644 --- a/clickstack-e2e-gaps/README.md +++ b/clickstack-e2e-gaps/README.md @@ -1,38 +1,35 @@ # clickstack-e2e-gaps -What changes in ClickStack when part of the logs moves to S3 and a Merge table reads both sides? Counts through the Merge table matched the counts from before the move, while the hot table alone gave a wrong count with no error. Queries that name only a time opened all 180 objects over 30 days. On ClickHouse compute, a plain collector rule by service and severity was cheaper than the 10x per-type policy. +On ClickHouse, every row that arrives has to be parsed, indexed, written to disk, and merged again each time parts combine. These runs fed a 197,430-line capture to ClickStack, kept each message type's first lines per time window in ClickStack's hot table, sent the rest to a local S3-compatible bucket, and read both through one Merge table. A count of one message type returned 36,395 through the Merge table, as before the move, and 3,688 from the hot table alone, with no error. A collector rule moving the two busiest services' INFO and DEBUG lines cost less insert and merge CPU than the per-type budget, 2.7 seconds against 8.1, but its rows carry no message type to count by. -The 10x engine keeps the first lines of each message type in a window hot and marks the rest, which the collector writes to S3 under a service and day path. ClickHouse can skip an object unopened only on those path segments, since the timestamp and the message type live inside the file. +The 10x engine marks lines over budget, and a stock OpenTelemetry Collector writes them under a service and day path. ClickHouse can skip an object only by its path, so a query bounded only by time opens every object in the bucket. -## Result - -From several runs over one 197,430-line capture. +## What ClickHouse counted | Measurement | Result | |---|---| | Time-only query over 30 days | 180 GETs, 7,449 ms | | Same query plus a day predicate | 12 GETs, 468 ms | -| LIMIT 100 through the Merge table | 180 GETs, 5,090 ms | -| Same LIMIT 100, hot table alone | 0 GETs, 7 ms | -| One type's count, hot table alone | 3,688 (36,395 before offload) | +| Newest 100 rows, Merge table | 180 GETs, 5,090 ms | +| Newest 100 rows, hot table alone | 0 GETs, 7 ms | +| One type's count, hot table alone | 3,688 (36,395 before the move) | | Same count through the Merge table | 36,395 | | Insert plus merge CPU, seconds | none 10.1, collector 2.7, per-type 8.1 | | Lines lost, engine acks after writing | 0 of 197,430 | -Both LIMIT 100 queries returned the same hundred rows, all of them hot. The collector rule used the least CPU, but its rows carry no message type to count by. +Both queries for the newest 100 rows returned the same rows, all hot. The collector rule kept more hot rows than the per-type budget, 72,759 against 43,128, but fewer bytes, 3,522,512 against 4,664,841, and most of the work on ClickStack's table is indexing the body text. -- With the engine acknowledging only after its lines are written, killing the collector, the engine or both lost no line in two runs and stored at most 997 lines twice. Acknowledging on receipt lost 24,900 to 32,203 lines under the same kills. -- An S3 table with hive partitioning that is read once while its prefix is empty matches nothing on a path predicate, while `count()` looks right (ClickHouse issue 116888). Creating it after the first object exists avoids this. +- With the shipped 10x collector recipe and the engine answering after writing, killing the collector, the engine or both lost no line in two runs, stored at most 997 lines twice, and left 2,040 to 2,538 objects. Answering on receipt lost 24,900 to 32,203 lines and left 24 or 25 objects. +- An S3 table with hive partitioning read once while its prefix is empty matches nothing on a path predicate, while `count()` looks right (ClickHouse issue 116888). Creating it after the first object exists avoids this. -## How it was measured +## How the capture went through ClickStack -The route is the one in [clickstack-e2e](../clickstack-e2e/), with a local S3-compatible store, on one laptop with 8 GB of Docker memory. For the 30-day queries, one run's cold objects were copied into 30 day partitions, six objects a day. GETs and CPU come from ClickHouse's query and part logs, and lost lines are counted by a sequence number inside every line. The collector rule sends the two busiest services' INFO and DEBUG lines to S3. +The table draws on several runs of the [clickstack-e2e](../clickstack-e2e/) route, on one laptop with 8 GB of Docker memory. The 30 day partitions are copies of one run's cold objects, so request counts are real and the content is not a real month. GETs and CPU come from ClickHouse's query and part logs, and lost lines from a sequence number inside every line. ## Limits -- Each setup ran once or a few times. Timings compare only with each other. Merge CPU and doubled lines vary by run. -- The 30 days are copies of one day's cold rows, so request counts are real but the content is not a real month. -- Nothing here measures money, HyperDX's own alert evaluation, or fetching cold objects back through 10x. +- Each setup ran once or a few times. Timings compare only with each other, and merge CPU and doubled lines vary by run. +- Money, HyperDX's own alert evaluation and fetching cold objects back through 10x are not measured. ## Run it @@ -42,6 +39,6 @@ cd clickstack-e2e-gaps ./gap1_multiday_pruning.sh # or any gap*.sh; LINES=20000 for a short run ``` -Each script needs Docker with about 7 GB of memory and 3 GB of disk. The acknowledge-after-write runs used engine 1.1.150 through `PATCHED_JAR` (see `lib.sh`). +Each script needs about 7 GB of Docker memory and 3 GB of disk. Gap 3 and the answer-after-write runs need an engine jar in `PATCHED_JAR` (see `lib.sh`), and those runs used 1.1.150. -[RECORD.md](RECORD.md) has the full record: what each script measures and leaves out, the build each run used, and a guide to [results/](results/). +[RECORD.md](RECORD.md) has what each script measures and leaves out, each run's build, and a guide to [results/](results/). diff --git a/clickstack-e2e/README.md b/clickstack-e2e/README.md index d3e31ff..e9acf9f 100644 --- a/clickstack-e2e/README.md +++ b/clickstack-e2e/README.md @@ -1,39 +1,38 @@ # clickstack-e2e -Can the rows a capped service sends over its cap go to an object store and still be queried with the rows ClickStack kept, as one ClickHouse table, with none lost? In this run they could. Of the 37,536 records the 10x engine returned, 13,010 went into ClickStack's table and 24,526 went to S3-compatible storage, and counting both sides back gave 37,536 with a gap of 0. +ClickStack stores log records in one ClickHouse table, `otel_logs`, which HyperDX queries. In this run, records over a byte cap on each of three services went to an S3-compatible bucket, and one Merge table answered queries over both sides. Of the 37,536 records the 10x engine returned, 13,010 went into ClickStack's table and 24,526 to the bucket, and the stored counts matched the returned ones in total and for every pattern hash. -Three services had a byte cap, and the engine marked every record it returned with a route, and the records over a cap were marked offload. A stock OpenTelemetry Collector writes the offload records to the object store under a service and day path and sends the rest to ClickStack. ClickHouse reads the objects in place, and a Merge table reads both sides under one name. +An OpenTelemetry Collector sends each record to the 10x engine over OTLP, and a record over its service's cap comes back marked `offload`. The collector writes those to the bucket under a path naming the service and the day. -## Result +## What each query read from the table and the bucket -Count queries from the 2026-09-15 run, each on a cold cache. The service filter is opentelemetry-collector. +Count queries from the 2026-09-15 run, each on a cold cache, as ClickHouse's query log recorded them. The service filter is opentelemetry-collector. | Query | Rows read | S3 GETs | ms | |---|---:|---:|---:| -| Hot table only, last hour | 12,012 | 0 | 8 | +| ClickStack table, last hour | 12,012 | 0 | 8 | | Merge, one service, last hour | 25,793 | 6 | 77 | | Merge, one service, one day | 25,793 | 6 | 43 | -| Merge, time only (HyperDX's shape) | 37,536 | 14 | 118 | +| Merge, last hour (HyperDX's filter) | 37,536 | 14 | 118 | | Merge, text search | 37,536 | 14 | 99 | -| Merge, one pattern hash | 37,536 | 14 | 123 | +| Merge, one pattern's text | 40,075 | 14 | 136 | +| Merge, hashed rows per side | 37,536 | 14 | 123 | | Counts table, by service | 2,539 | 0 | 6 | -The time-only filter that HyperDX sends read every row and made one GET per offloaded object. A service filter cut that to the 6 objects under that service. The hot table and the counts table made no object-store request. +Each object in the bucket carries the service and the date in its name, and ClickHouse can skip a whole object on its name alone, so a service filter read only that service's 6 objects. The timestamps sit inside the objects, so the time-only filter HyperDX sends opened all 14 and took 118 ms, against 8 ms on ClickStack's table. The one-pattern query also reads the counts table. -- Only the capped services had offloaded rows: 16,856 from opentelemetry-collector, 4,522 from kafka and 3,148 from cart. -- The same 2,495 distinct pattern hashes were on the wire and in storage. Only 37,486 of the 37,536 returned records carried a hash, and a record without one is stored but not counted per type. -- ClickStack's own collector has no S3 exporter, so a second, stock collector does the routing. The harness also adds a day column to ClickStack's `otel_logs` table. +- The counts table, one count per service and pattern per minute, got the offloaded rows' counts from one INSERT ... SELECT after the run. +- Of the 37,536 returned records, 37,486 carried a pattern hash. A record without one is stored but cannot be counted per type. ## How it was measured -The input is the first 50,000 lines of the released capture otel-sample-200mb.log. One host runs ClickStack 2.38.0 on ClickHouse 26.5.7.64, the 10x engine, a stock OpenTelemetry Collector 0.160.0 and MinIO. The lossless check counts every record the engine returned against the stored rows, in total and per pattern hash. Query figures come from ClickHouse's own query log, with caches dropped first. +The input is the first 50,000 lines of the released capture otel-sample-200mb.log, whose envelope carries no timestamp, so each record takes its ingest time. One command starts ClickStack 2.38.0 on ClickHouse 26.5.7.64, the 10x engine, a stock OpenTelemetry Collector 0.160.0 and MinIO in containers. The stock collector routes because ClickStack's own collector has no S3 exporter. The offload encoding keeps only body and attributes, so the collector first copies service, severity and record time into attributes. The harness adds a day column to ClickStack's table. ## Limits -- One run, with MinIO in place of S3. The times say nothing about real S3, throughput or ClickHouse Cloud cost. -- No alert is defined or fired, and the queries were written for this test. +- The run used MinIO in place of S3 and a query list written for this test, and did not measure throughput, ClickHouse Cloud cost or alerts. - Every object carries one day in its path, because the path records the write time. Pruning across thirty days is measured in [clickstack-e2e-gaps](../clickstack-e2e-gaps/). -- The lossless check is a total, so a lost line and a duplicated one would cancel. The gaps benchmark counts per line. +- The lossless check starts from the records the engine returned, not the 50,000 input lines, and a lost line and a duplicated one would cancel in it. ## Run it @@ -45,4 +44,4 @@ LINES=20000 ./run.sh # a smaller slice of the capture A run needs Docker with about 8 GB of memory and downloads the capture on first use. -[RECORD.md](RECORD.md) describes the pipeline and what a run does and does not prove, and [the results file](results/clickstack-e2e-2026-09-15.md) has every number and config from this run. +[RECORD.md](RECORD.md) has the pipeline and what a run does and does not prove, and [the results file](results/clickstack-e2e-2026-09-15.md) has every number and config. diff --git a/clickstack-rows-gate/README.md b/clickstack-rows-gate/README.md index f9121a0..afb8e2b 100644 --- a/clickstack-rows-gate/README.md +++ b/clickstack-rows-gate/README.md @@ -1,39 +1,34 @@ # ClickHouse rows gate -When the 10x engine limits how many lines a service sends into ClickStack, which ClickHouse system table counts the rows written into the logs table exactly? The NewPart rows in `system.part_log` matched the table's own count of 22,367 rows to the row, in both runs. The insert entries in `system.query_log` did not, so they serve as a cross-check at most. +Two ClickHouse system tables record inserts: `system.query_log` has an entry per insert with the rows it wrote, and `system.part_log` has a NewPart entry per new data part with its row count. Both were read against the row count of ClickStack's logs table while the 10x engine limited three services to a set total per window. The NewPart rows equaled the table's count in both runs. In the second run the query log's insert entries were 1,948 rows over or 7,519 under, depending on the entry kind. -ClickStack's exporter inserts asynchronously, and each insert statement lists three tables in the query log (the logs table, a counts table and a 15-minute rollup), so a filter on the logs table cannot pick out its rows. Which of the two insert entry kinds carried the extra rows also changed between runs. +ClickStack's exporter inserts asynchronously, and each insert statement lists three tables in the query log (the logs table, a counts table and a 15-minute rollup), so a filter on the logs table cannot pick out its rows. -## Result +## What the system tables counted -The table shows the second run, on 2026-10-23. Hot means the logs table, where the lines the engine passed land. - -| Count | Rows | Against the logs table | +| Count, second run, 2026-10-23 | Rows | Against the logs table | |---|---:|---| | `count()` on `default.otel_logs` | 22,367 | the count to match | -| Lines the engine routed to hot | 22,367 | equal | +| Events the engine passed | 22,367 | equal | | `system.parts`, active rows | 22,367 | equal | | `system.part_log`, NewPart rows | 22,367 | equal | | `system.query_log`, kind Insert | 24,315 | off by +1,948 | | `system.query_log`, kind AsyncInsertFlush | 14,848 | off by -7,519 | -The part log matched in both runs, so it is the count to use. In the first run the Insert entries happened to equal the hot count and the AsyncInsertFlush entries ran over by the materialized view's rows. - -The run also checked the limits: +The part log also matched in the first run, where the Insert entries happened to equal the table's count and the AsyncInsertFlush entries ran over by a materialized view's rows. Which entry kind carries the other tables' rows changes between runs, so the query log is a cross-check at most. Rows in the table and the store add up to the 157,087 records the engine returned (22,367 + 134,720). -- With the limit counted in lines, kafka kept 1,000 lines against a total of 1,000 and cart kept 400 against 400. The opentelemetry-collector service kept 2,050 against 2,000, and the 50 extra lines are the engine's error keep count for one storming error statement. -- Rewriting the totals file in place 4 seconds into the feed did not reset the window counts, and no limited service passed another line in that window. -- Hot plus cold equalled the records the engine returned: 22,367 + 134,720 = 157,087. +- In the second run, kafka put 1,000 rows in the table against its total of 1,000, cart 400 against 400 and opentelemetry-collector 2,050 against 2,000. The extra 50 are the error lines the engine keeps for one storming error statement. +- Rewriting the totals file in place 4 seconds into the feed did not reset the window counts, and no limited service added another row. +- Scraped 5 seconds after the collector stopped, the engine's Prometheus counters still lagged, at 16,255 passed events against 22,367, so a scheduled run reads the last complete hour. -## How it was measured +## The ClickStack rig and the feed -The rig is the one in [clickstack-e2e](../clickstack-e2e/): ClickStack 2.38.0 on ClickHouse 26.5.7.64, the 10x engine in front of it, and an S3-compatible store for the cold side. The whole 197,430-line capture was fed in about 216 seconds. Three services had a line total for the window, and their lines over it went to S3, which a ClickHouse view reads back. The results file lists every query with its answer. +The rig is the one in [clickstack-e2e](../clickstack-e2e/): ClickStack 2.38.0 on ClickHouse 26.5.7.64, the 10x engine and an S3-compatible store. An OpenTelemetry collector reads the 197,430-line capture and sends its events to the engine, which returns them marked pass or offload. Passed events go into the logs table and offloaded ones to the store, which a ClickHouse view reads back. Every count was read after flushing the system logs, and the [results file](results/clickstack-rows-gate-2026-10-23.md) lists each query and its answer. -## Limits +## What the run does not cover -- The feed is one limit window long, so the test says nothing about money, ClickHouse Cloud or a rate kept over hours. The hourly test that uses this count is [clickstack-rows](../clickstack-rows/). -- Both runs used one ClickStack version, and only the second run's figures are committed. -- The engine's Prometheus counters, scraped 5 seconds after the collector stopped, still lagged the lines sent (16,255 passed against 22,367). A reading of them should use the last complete hour. +- The feed is one limit window long, so the run says nothing about money, ClickHouse Cloud or a rate kept over hours, which [clickstack-rows](../clickstack-rows/) measures from the part log. +- Both runs used one ClickStack version and its exporter's asynchronous inserts, and only the second run's figures are committed. ## Run it @@ -42,6 +37,6 @@ The rig is the one in [clickstack-e2e](../clickstack-e2e/): ClickStack 2.38.0 on CSE_DATA_DIR=../clickstack-e2e/data ./run.sh ``` -A run needs Docker and the capture from clickstack-e2e. The engine image is `ghcr.io/log-10x/edge-10x:1.1.147`. Results land in [results/](results/). +A run needs Docker and the capture from clickstack-e2e, uses the engine image `ghcr.io/log-10x/edge-10x:1.1.147` and writes to [results/](results/). -[RECORD.md](RECORD.md) has the full record, including the engine counters, their labels and the config patches. +[RECORD.md](RECORD.md) has the full record, with the engine's counters, config patches and store setup. diff --git a/clickstack-rows/README.md b/clickstack-rows/README.md index 963883c..6364841 100644 --- a/clickstack-rows/README.md +++ b/clickstack-rows/README.md @@ -1,14 +1,12 @@ # ClickHouse rows run -Can the rows written into a ClickStack logs table be kept at a set rate an hour while log volume grows 2.5 times? On one node, the rows ClickHouse wrote stayed between 0.91 and 1.13 of the set rate in every 10-minute window as load grew, one over the 1.10 the test allowed. A static collector rule on the same feed let its table grow to 2.95 to 3.42 times its starting level at full load. CPU did not separate the two. +Every row that arrives in ClickHouse has to be parsed, indexed, written to disk, and merged again each time parts combine, and `system.part_log` records each new part with its row count. A scheduled run read that log and set limits per log statement and service in the 10x engine, which sent lines over them, apart from a sample, to the account's bucket. As load grew 2.5 times, rows written into ClickStack stayed between 0.91 and 1.13 of the set rate in every 10-minute window, while a collector rule by service and severity let its table grow 2.95 to 3.42 times its starting level. One window went over the 1.10 the registered test allowed, and CPU did not separate the two setups. -Every 10 minutes a scheduled run reads ClickHouse's own `system.part_log` and sets a limit per log statement in the 10x engine. Lines over the limits, apart from a small sample, go to the account's own bucket instead of ClickHouse. +The set rate R is 30,099 rows an hour, 0.7 of a dry run's unlimited rate. Every clock in the rig is divided by 6, so the scheduled run comes every 10 minutes and each window is scored against R/6. -## Result +## Rows written per 10-minute window -Rows written in eight 10-minute windows on 2026-10-25, against a set rate R of 30,099 rows an hour (5,016 a window). - -| Minutes | Lines a second | 10x engine rows | Of R/6 | Collector rule rows | +| Minutes | Lines a second | Rows, 10x engine | Of R/6 | Rows, collector rule | |---|---:|---:|---:|---:| | 50 to 60 | 14 | 4,697 | 0.94 | 2,686 | | 60 to 70 | 19 | 5,492 | 1.09 | 3,572 | @@ -19,20 +17,22 @@ Rows written in eight 10-minute windows on 2026-10-25, against a set rate R of 3 | 140 to 150 | 35 | 5,443 | 1.09 | 9,179 | | 150 to 160 | 35 | 5,052 | 1.01 | 8,142 | -The registered test required every window from minute 60 to 160 at or under 1.10 of R/6, and nine of ten were. Minutes 100 to 110, the first window of a 20-minute error storm, read 1.13, so the run fails as registered. +The second window of each growth step, left out here, read 0.92, 0.91 and 0.95 of R/6. The window over 1.10 is minutes 100 to 110, the first of a 20-minute error storm, so the run fails as [registered](METHOD_TWO_ARM.md). + +- Of the 50,378 records the rule keeps in minutes 100 to 160, the engine sent 27,155 (53.9%) to the bucket, and 43.2% in minutes 120 to 160, after the storm. +- The engine's table had 933 more rows than the engine returned, and the tables logged 20 and 18 failed inserts, all but one before any scored window. + +## Insert and merge CPU per window -- Against each setup's level in minutes 50 to 60, insert plus merge CPU read 0.87 to 1.24 for the 10x engine and 0.40 to 1.61 for the rule at full load, as merge CPU swung with which merges ran. -- Of the 50,378 records the rule would have kept in minutes 100 to 160, the 10x engine sent 27,155 (53.9%) to the bucket. After the storm, in minutes 120 to 160, the share was 43.2%. +The setups insert through different collectors, so each is read against its own level in minutes 50 to 60. After the first growth step the engine's table read 0.87 to 1.24 of it, three of ten windows over 1.10, and at full load the rule's read 0.40 to 1.61. Insert CPU followed the rows, and merge CPU swung with the merges run in each window. -## How it was measured +## How the part log was read -Both setups shared one ClickStack 2.38.0 node and a replay of the OpenTelemetry demo capture. Load stepped from 14 to 35 lines a second every 20 minutes, adding a new service at minute 80 and the storm at 100. Every clock is divided by 6, so limits were reset every 10 minutes instead of hourly. R is 0.7 of a dry run's rate with nothing limited. The rule sends the two chattiest services' INFO and DEBUG lines to a second bucket and the rest to a copy of the logs table. CPU is insert time from `system.query_log` plus merge time from the part log. +Both setups shared one ClickStack 2.38.0 node (ClickHouse 26.5.7.64) and one replay of the OpenTelemetry demo capture. Load rose from 14 to 35 lines a second over minutes 60 to 120 and stayed at 35 to minute 160. The rule sends the two chattiest services' INFO and DEBUG lines to a second bucket. CPU is insert time from `system.query_log` plus merge time from the part log. -## Limits +## Scope of the run -- One node and one replayed capture, with nothing on money, ClickHouse Cloud or durability across a restart. -- The setups insert through different collectors, so CPU compares only within a setup. -- Delivery was not exact: the 10x engine's table had 933 rows more than the engine returned, and the two tables logged 20 and 18 failed inserts, all but one before any scored window. +The run covers one node and one replayed capture, and measures nothing about money, ClickHouse Cloud or durability across a restart. ## Run it @@ -45,6 +45,6 @@ ARMS=two MCP= GH_TOKEN=$(gh auth token) \ CAPTURE= rig/phases.sh ``` -The recompute needs only Python 3, and a full rerun takes 180 minutes, Docker and the capture `clickstack-e2e/run.sh` downloads. +The recompute needs only Python 3 and prints [every window](results-two-arm/score.md). A full rerun takes 180 minutes, Docker, Node.js and the capture `clickstack-e2e/run.sh` downloads. -[RECORD.md](RECORD.md) has all three runs with their registered criteria, scores and rig notes. +[RECORD.md](RECORD.md) has all three runs on this rig, their criteria, scores and rig notes. diff --git a/cloudwatch-ia-accounting/README.md b/cloudwatch-ia-accounting/README.md index 3f687eb..006c996 100644 --- a/cloudwatch-ia-accounting/README.md +++ b/cloudwatch-ia-accounting/README.md @@ -1,12 +1,10 @@ # CloudWatch Logs Infrequent Access accounting -When a log10x plan moves CloudWatch events out of a Standard log group, to Infrequent Access or to S3, how many bytes move and what do they cost? On a ten-hour OpenTelemetry demo capture, the plan for a 30% target moved 69.7% of the bytes to Infrequent Access, where ingest costs $0.25 a GB instead of $0.50. The figures are the 10x engine's counts, priced at us-east-1 list rates. +The expensive line on a CloudWatch Logs bill is ingestion, the charge for each GB ingested, with storage after it costing far less. The Infrequent Access log class ingests at half the Standard rate, $0.25 a GB against $0.50 in us-east-1, and an event offloaded to S3 pays no CloudWatch ingestion. Under a log10x plan to cut the bill 30%, the 10x engine moved 69.7% of the bytes of a ten-hour OpenTelemetry demo capture to Infrequent Access. -Infrequent Access halves the ingest rate and keeps the $0.03 per GB-month storage price, so a bill cut made by moving bytes there takes about twice that share of the bytes. +A log group's class is fixed at creation, so the forwarder sends the events the engine marks to a second log group in the Infrequent Access class. The classes differ in ingestion cost only, so a cut made there takes about twice its share of the bill in bytes. -## Result - -Where the engine sent the capture's 0.215 GB at each target. +## What left the Standard log class | Target | Bill share the plan states | GB to Infrequent Access | GB to S3 | Share of bytes moved | |---:|---:|---:|---:|---:| @@ -14,21 +12,21 @@ Where the engine sent the capture's 0.215 GB at each target. | 50% | 56.9% | 0.067 | 0.099 | 77.4% | | 70% | 70.6% | 0.002 | 0.165 | 77.7% | -At 50% and 70% the plan also sends whole containers to S3, where a GB costs nothing to ingest and $0.023 a month to store. At 70% nearly all moved bytes went there. +At 50% and 70% the plans also send whole containers to S3, at $0.023 per GB-month to store. Both left 0.048 GB in Standard. -- The 30% target moved no errors or warnings. At 50% and 70%, 53 and 51 error events went to S3 with their container, since the engine keeps only half of the error events over a cap. -- At 30% the engine moved 0.150 GB to Infrequent Access where the plan states 0.136 GB, because the plan works from a 12,000-line sample and the engine counts the whole capture. +- The 30% plan moved no errors or warnings. At 50% and 70%, 53 and 51 ERROR events went to S3 with their containers, since the engine keeps half the ERROR events over a cap. +- The plan works from a 12,000-line sample, so at 30% the engine moved 0.150 GB to Infrequent Access against the plan's 0.136 GB. At 70% the plan states 0.030 GB and the engine marked 0.002 GB, because offload acts on the whole container, including patterns the plan lists for Infrequent Access. -## How it was measured +## How the event message bytes were counted -The data is a public capture of Kubernetes container output from the OpenTelemetry demo, 197,430 lines from 36 containers. For each target, the log10x planning tool sampled it and wrote a byte cap and destination per container, and the engine replayed the whole capture against them. Bytes are the UTF-8 size of each emitted record, which the forwarder sends as the event message CloudWatch bills. Every count was taken from the emitted records and from the engine's aggregators, and the run fails if they differ. +The capture is 0.215 GB, 197,430 lines of output from 36 Kubernetes containers in a cluster running the OpenTelemetry demo. For each target, the log10x planning tool wrote a byte cap and an action per container, and the engine replayed the whole capture, moving each pattern's bytes over its cap. Caps apply per 240-second window and the replay fits inside one, so each cap was multiplied by 15 for the hour the plan covers. Bytes are the UTF-8 size of each emitted record, which the forwarder sends as the event message CloudWatch bills. ## Limits -- No AWS bill or usage meter was read. Dollars use list prices read on 2026-10-11. -- The dollar figures are small because the capture comes from one small demo cluster. The share of bytes moved depends on how its logs spread across patterns. -- An Infrequent Access log group lacks metric filters, subscription filters, Live Tail, Anomaly Detection and more, so logs that feed one of these lose it when moved. Logs Insights queries still run, apart from four commands. -- The kept share of each severity is drawn at random per event, so a rerun gives slightly different counts. +- No AWS bill or usage meter was read. Dollars are list prices read on 2026-10-11. +- The GB figures come from one small demo cluster, and only the per-GB rates carry to another account. +- AWS's log class table lists metric filters, subscription filters, Live Tail and Anomaly Detection among the features Infrequent Access lacks. +- Each severity's kept share is drawn at random per event, so reruns differ slightly. ## Run it @@ -40,4 +38,4 @@ python3 report.py --check # recompute every figure from the committed counts The check needs only Python 3. The full run needs Docker, Node 18 or later and about 1 GB of disk, takes under ten minutes and needs no AWS account. -[RECORD.md](RECORD.md) has the full record: planned against moved bytes per container, every feature Infrequent Access lacks, and the counting rules. +[RECORD.md](RECORD.md) has each plan's actions by container, a month at this volume, and every feature Infrequent Access lacks. diff --git a/datadog-flex-accounting/README.md b/datadog-flex-accounting/README.md index 3472e22..437ff9a 100644 --- a/datadog-flex-accounting/README.md +++ b/datadog-flex-accounting/README.md @@ -1,38 +1,41 @@ # Datadog Flex Logs accounting -When the 10x engine moves Datadog log events from Standard indexing to Flex Logs, does the move save money at list price? On a 10-hour OpenTelemetry demo capture of 157,228 events, plans targeting 30%, 50% and 70% of the volume moved 67,505 to 119,833 events, none of them errors or warnings. At this volume every move lands in Flex Starter, which saves money against 30-day Standard indexing and costs more than 15-day indexing. +Datadog prices indexing per million events and ingestion per GB, and a pattern routed to Flex Logs stops paying the standard index rate. The 10x engine replayed a 10-hour OpenTelemetry demo capture under plans moving whole patterns to Flex at 30%, 50% and 70% targets. At annual list prices and this capture's volume, every move fell in Flex Starter, where it saves $3.40 to $6.04 a month against 30-day Standard indexing and costs $0.49 to $0.86 more than 15-day indexing. -On annual pricing, Standard indexing lists at $1.70 per million events for 15-day retention and $2.50 for 30-day. Flex Starter lists at $0.60 per million stored per month for at least 3 months, so a moved event costs at least $1.80 per million. That sits between the two indexing prices. +Standard indexing lists at $1.70 per million events for 15-day retention and $2.50 for 30-day. Flex Starter bills storage and compute together at $0.60 per million stored per month for at least 3 months, so a moved event costs at least $1.80 per million. Ingestion is billed as before. -## Result +## Net per month against Standard indexing -Net is the Standard indexing avoided minus what Flex Starter adds, per month at this capture's volume, and a minus means the move costs more than it saves. +Net is the indexing avoided less what Flex Starter adds, with the capture's 10 hours scaled to a 720-hour month. -| Target | Events moved to Flex | Share of bytes moved | Net vs 15-day | Net vs 30-day | +| Target | Events moved in 10 hours | Share of bytes moved | Net vs 15-day | Net vs 30-day | |---:|---:|---:|---:|---:| | 30% | 67,505 | 39.9% | -$0.49 | +$3.40 | | 50% | 90,555 | 56.1% | -$0.66 | +$4.56 | | 70% | 119,833 | 71.2% | -$0.86 | +$6.04 | -The dollars are small because the demo cluster is small. +Datadog's Flex docs give the 3-month minimum. At 6 months, the shortest retention on Datadog's price list, Starter costs $17.50 to $31.06 a month, more than the indexing avoided at either retention. -Against Datadog's own index filters, one per service and matched on events: +Each target lands on the first set of whole patterns reaching it, three at 30% and 55 at 70%. None of the moved events was an error or a warning. -- At 30%, one filter on `service:opentelemetry-collector` with `-status:(error OR warn)` does nearly the same job. Of the 18,281 extra events it moves, 47 come from patterns the plan leaves entirely in Standard. -- At 50% and 70%, the filters leave 23,161 and 11,181 events in Standard that the plan moves. -- Without the status clause, every filter set moves 3,953 to 3,955 errors and warnings to Flex. +## One Datadog index filter per service, at equal events -## How it was measured +Each target is compared with the fewest `service:` index filters reaching the events the plan moves. -The capture is Kubernetes output from 36 OpenTelemetry demo containers, 197,430 lines (157,228 events after multi-line grouping) and 0.215 GB. For each target, `log10x-mcp` sampled 12,000 lines and wrote a plan that moves whole log patterns. The released 10x engine replayed the full capture under that plan, counting every event from its output records and its own aggregators, and the run fails if the two differ. Dollars are those counts at Datadog list prices read on 2026-10-13. Datadog metered nothing. +- At 30%, one filter on `service:opentelemetry-collector` with `-status:(error OR warn)` moves every event the plan moves plus 18,281 more. All but 47 are the part of the three moved patterns that the plan leaves in Standard. +- At 50% and 70%, the filters leave in Standard 23,161 and 11,181 events the plan moves. +- Without the status clause, the filters move 3,953 to 3,955 errors and warnings to Flex, where Datadog runs no log monitors. -## Limits +## How the events were counted and priced -- One small demo capture, replayed rather than sent to Datadog. Monthly dollars scale its 10 hours to a 720-hour month. -- Only Flex Starter is priced. Above 2 billion events a month, Datadog prices Flex compute on request. -- Datadog's price list gives Starter retention as 6, 12 or 15 months, and its Flex docs add the 3 the table uses. At 6 months Starter costs $17.50, $23.47 and $31.06 a month, more than the indexing avoided at either retention. +The capture is [otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1), 197,430 lines of Kubernetes output from 36 containers, which the engine groups into 157,228 events. The log10x planning tool wrote each plan as a byte cap per container from a 12,000-line sample. The engine replayed the capture in one 240-second window, each cap multiplied by 15 to cover the plan's hour. Events were counted from the output records and the engine's aggregators, and the run fails if they differ. -## Run it +## What the accounting leaves out + +- Datadog metered nothing. The dollars are the engine's counts at list prices read on 2026-10-13. +- The volume is one small demo cluster's. Above 2 billion events a month Datadog prices Flex compute on request, so only Starter is priced. + +## Rerunning the accounting ``` # from the repository root: recompute every figure from the committed counts and prices @@ -42,6 +45,6 @@ python3 datadog-flex-accounting/report.py --check cd datadog-flex-accounting && ./run.sh ``` -The full rerun needs Docker, Node 18 or later and about 1 GB of disk. A run takes under ten minutes and needs no Datadog account. +The full rerun takes under ten minutes and needs Docker, Node 18 or later, about 1 GB of disk and no Datadog account. -[RECORD.md](RECORD.md) has every price source, the plan and engine files per target, the full filter comparison and the counting conditions. +[RECORD.md](RECORD.md) has every rate, the plans, the filter comparison and the counting conditions. diff --git a/datadog-service-control/README.md b/datadog-service-control/README.md index d3f493e..7191763 100644 --- a/datadog-service-control/README.md +++ b/datadog-service-control/README.md @@ -1,46 +1,47 @@ # Datadog service control -Inside one Datadog service, can the index filters a user writes move as many events out of Standard as a per-pattern plan? On the OpenTelemetry demo's Kafka logs they can. At the 70% target, eight logger-name filters moved 24,440 of 29,043 events and the plan moved 23,854, and neither moved an error or warning. On this data the control can be built in house, with eight rules against the plan's one. +Datadog bills Standard indexing per million events. Inside one service a service filter cannot split that volume, so the control there is an index filter on the logger name Datadog's Kafka pipeline parses, `service:kafka @logger.name:`. In a replay of kafka's records from a public OpenTelemetry demo capture, eight such filters moved 24,440 of 29,043 events out of Standard and a per-pattern plan moved 23,854. Neither moved an error or warning, and on this data the control can be built in house. -The plan is one cap row, a byte limit per log pattern on the kafka container, and events the 10x engine marks over it go to Flex through an index filter. Datadog's Kafka pipeline parses each logger into `logger.name`, so the native rule is `service:kafka @logger.name:`. +The plan is one cap row, a byte limit per pattern per 240-second window, and the 10x engine marks events over it for a Flex index. -## Result +## Events moved out of Standard in the kafka service -Events moved out of Standard at the 70% target. B got A's rules unchanged, beside a plan written on B. +The plan was asked to move 70% of the volume, and capture B, a later demo version, got A's rules unchanged. | Capture | Rule set | Rules | Events moved | Lifecycle events moved, of 317 | |---|---|---:|---:|---:| | A, 29,043 events | per-pattern plan | 1 cap row | 23,854 | n/a | | A | logger-name filters | 8 | 24,440 | n/a | -| B, 6,226 events | plan rewritten on B | 1 cap row | 3,321 | 0 | +| B, 6,226 events | plan written on B | 1 cap row | 3,321 | 0 | | B | A's cap, unchanged | 1 cap row | 3,493 | 95 | | B | A's logger-name filters | 8 | 4,658 | 0 | -On A the filters leave in Standard 3,046 events the plan moves, from two logger classes they do not name. On B, A's filters moved none of the 317 lifecycle events (patterns B has and A lacks), while A's cap moved 95. +On A the filters leave in Standard 3,046 events the plan moves, from two classes they do not name. All of A's filters still matched on B. Of B's 317 lifecycle events, from patterns A lacks, A's cap moved 95 and the filters none. -Across all services of both captures, after a buyer's usual filters (`-status:(error OR warn)` service filters on three services, DEBUG and TRACE dropped): +## What the plan moves after service and level filters -- The plan moved 5,539 events beyond those filters on A (3.5% of the capture) and 4,882 on B (11.7%). -- On B, six more service filters cut that to 48 events but also move 4,085 events the plan keeps. -- A phrase of each pattern's own words selects it, which takes 13 index filters on A and 22 on B. One phrase on A also catches 2 kept events. +Of the events left in Standard on both whole captures after `-status:(error OR warn)` filters on three services and DEBUG and TRACE dropped, the plan at 70% moved 5,539 on A, 3.5% of the capture, and 4,882 on B, 11.7%. An index filter on a quoted phrase of each pattern's constant words selects it, 13 filters on A and 22 on B, and one also selects 2 events the plan keeps. A condition fixed before the run called for a trial of the plan only if most of those events needed the pattern ID, and none do. -## How it was measured +On B, six more service filters leave 48 of those events and also move 4,085 the plan keeps, nearly all of them statements split into one pattern per value or short statements the byte cap keeps. -The data is the kafka records of two public OpenTelemetry demo captures on Kafka 3.9.1. A covers ten hours through Fluentd and B two hours through Fluent Bit. The plan was written on A at targets from 30 to 90% and moved 23,836 to 23,905 events at each. The filters follow a rule fixed before the run: the fewest logger classes, largest first, whose events reach the plan's. Each count is read from the emitted records and the engine's own counters, and the run fails if they differ. +## How the events were counted + +Capture A is ten hours through Fluentd and B two hours through Fluent Bit, on Kafka 3.9.1. On A the plan moved 23,836 to 23,905 events at targets from 30 to 90%. The filters follow a rule fixed before the run, the fewest logger classes, largest first, whose events reach the plan's, so the totals on A are close by construction. Each count is read twice from the 10x engine's replay, and the run fails if the two differ. ## Limits -- Only two demo captures were tested. A production stream can have more events that only the pattern ID selects. -- Nothing was sent to Datadog, so no Datadog meter was read. +- Only two demo captures were tested. +- Nothing was sent to Datadog, so no usage metric was read. - The rule counts leave out the work of finding each pattern first. ## Run it ``` +cd datadog-service-control python3 report.py --check # recompute every table from the committed files ./run.sh # the full run ``` -The check needs only Python. The full run needs Docker, Node 18 or later, about 2 GB of disk and under twenty minutes, and no Datadog account. +The full run needs Docker, Node 18 or later, about 2 GB of disk and under twenty minutes. -[RECORD.md](RECORD.md) has every target and variant, the all-services analysis and the predictions written before the run. +[RECORD.md](RECORD.md) has every target, the all-services analysis and the predictions written before the run. diff --git a/drain3-vs-log10x/README.md b/drain3-vs-log10x/README.md index 34f71f1..b71f8a0 100644 --- a/drain3-vs-log10x/README.md +++ b/drain3-vs-log10x/README.md @@ -1,44 +1,45 @@ # log10x vs Drain3: pattern stability and lossless reduction -Does a log line keep the same pattern ID whatever order the lines arrive in, and can the original line be rebuilt byte for byte? This benchmark gives the same lines to log10x and to Drain3, the open-source template miner, on 16 public loghub datasets. log10x kept every ID stable on all 16 and rebuilt 99.98% of lines exactly, against 66.48% for Drain3. Drain3 found fewer templates, closer to the hand-labelled count. +A saved search or an alert points at a log pattern group by its stored text. Drain3, the open-source implementation of the Drain algorithm, rewrites that text as new lines arrive. On 16 public loghub samples of 2,000 lines, read in two orders and in halves, the 10x engine gave each line one pattern ID throughout, and Drain3 gave as few as 83.4% of lines the same template in both orders. The engine rebuilt 99.98% of lines byte for byte and Drain3 66.48%, and every line Drain3 missed differs only in its spacing. -## Result +Drain compares each new line against the text it already stored for that group, and every spot where the two differ is blanked out. -Figures over the 16 loghub samples of 2,000 lines each. +## Pattern IDs and rebuilt lines on 16 samples -| Measure | log10x | Drain3 | +| Measure | 10x engine | Drain3 | |---|---:|---:| | Same ID in both orders (worst) | 100% | 83.4% (Zookeeper) | | Same ID alone and combined (worst) | 100% | 0.8% (Windows) | | Lines rebuilt byte for byte (mean) | 99.98% | 66.48% | -| Datasets rebuilt byte for byte | 14 of 16 | 5 of 16 | -| Templates found | 4,367 | 1,761 | +| Samples rebuilt byte for byte | 14 of 16 | 5 of 16 | +| Templates, all 16 samples | 4,367 | 1,761 | -The second row compares the ID a line gets when its half is read alone with its ID in the whole file. loghub's hand labels list 1,363 templates. +The second row compares a line's ID in its half alone and in the whole file. Drain3's ID is its matched template text, since its cluster numbers are per-run counters, and the engine's is a content-addressed template hash. -- Drain3's patterns depend on reading order. The same 2,000 Mac lines in two orders gave 37 templates found only in one order and 40 only in the other. log10x found the same set in both orders on every dataset. -- log10x missed exact rebuild on BGL (99.75%) and Thunderbird (99.9%). On the full BGL file of 4,747,963 lines it rebuilt 99.73%. Of the 12,648 lines that differ, 11,904 quote a date with a zone name and come back as the same instant written in another zone. The rest trace to two other defects listed in the record. +- The same 2,000 Mac lines in two orders gave Drain3 37 templates only in one order and 40 only in the other. The engine's sets matched on every sample. +- Drain3 rebuilt the BGL and Thunderbird samples in full, where the engine rebuilt 99.75% and 99.9%, and 99.73% of the full BGL file's 4,747,963 lines. Of the 12,648 misses, 11,904 quote a date with a zone name and come back as the same instant in another zone. The decoder skips 737 lines of 13 templates with hashes of eight characters or fewer, and one template whose encoded records end early accounts for 7. +- The 1,363 templates in loghub's hand labels are closer to Drain3's count than to the engine's. On the full BGL file the engine found 136,461 templates against Drain3's 842, and loading them made decode slow. For a short template list, Drain3 is the better fit. -## How it was measured +## How the lines were split, reordered and rebuilt -Both tools read identical bytes, with line endings converted to LF. The 10x engine ran in Docker as a codec: one pass encodes, a second decodes, and the output is compared with the input. Drain3 ran with default settings and has no decoder, so the harness rebuilds each line from its template by token position, which joins tokens with single spaces. Drain3 therefore misses exactly the lines with a tab, a run of spaces or a leading or trailing space. For stability, each dataset is split into halves A and B, and both tools run on A, B, A+B and B+A. +Both tools read identical bytes with LF line endings, and run each sample's first half A and second half B as A, B, A+B and B+A. The engine's decode output is compared with its input. The harness rebuilds each Drain3 line from its matched template by token position with single spaces, which loses every tab, run of spaces and leading or trailing space. -## Limits +## Scope of the test -- The data is public research logs, plus one full BGL file. -- Storage size is out of scope. Compaction on the full loghub datasets is in [loghub-compaction](../loghub-compaction/). -- Every Mac and Windows line has whitespace that the token-position rebuild cannot restore, so Drain3 scores 0% there. -- For a short template list close to the hand labels, Drain3 is the better fit. log10x found more templates on 14 of 16 datasets, and 136,461 against Drain3's 842 on the full BGL file. +- The data is public research logs from loghub. +- Drain3 ran with default settings, and a run with number, hex and IP masking is kept per sample in the facts file. +- Storage size is out of scope, and [loghub-compaction](../loghub-compaction/) measures compact size. -## Run it +## Rerunning the comparison ``` +cd drain3-vs-log10x pip install -r requirements.txt python smoke_test.py # Drain3 whitespace check, no Docker or data cd bench # full rerun, needs the loghub setup python bench.py && python stability.py all && python facts.py ``` -A full rerun needs Docker, the engine image `log10x/pipeline-10x:1.1.136`, Drain3 0.9.11 and the loghub clone in [RECORD.md](RECORD.md#setup). Reference numbers are in [bench/facts.json](bench/facts.json). +A full rerun needs Docker, the engine image `log10x/pipeline-10x:1.1.136` and the loghub clone in [RECORD.md](RECORD.md#setup). Reference numbers are in [bench/facts.json](bench/facts.json). [RECORD.md](RECORD.md) has the field definitions, setup, the full BGL run and what each script does. diff --git a/e103-grouping/README.md b/e103-grouping/README.md index 38c49ae..f54e528 100644 --- a/e103-grouping/README.md +++ b/e103-grouping/README.md @@ -1,12 +1,12 @@ # Multi-line grouping across containers -When several containers send to the 10x engine through Fluent Bit under one tag, does each container's multi-line event come back in its own record? On the shipped 1.1.150 image it did not: 1,533 of 5,836 returned records mixed lines from more than one container. A build with a grouping fix returned no mixed records, and neither build lost or duplicated a line. +Fluent Bit's Forward protocol sends every record with a tag, and the Fluent Bit input in the shipped 10x engine image, 1.1.150, takes each event's source from that tag. The engine joins a multi-line event such as a stack trace into one record, and the step that joins lines counts its flush timeout per source. Across five cases, the shipped image returned 1,533 of 5,836 records mixing lines from more than one container, all in cases that send every container under one tag. A build with a fix for this returned none, and neither build lost or duplicated any of the 22,557 lines sent. -The shipped Fluent Bit input takes an event's source from the Forward tag, and the step that joins multi-line events counts its flush timeout per source. In the [Elasticsearch hot-tier benchmark](../elastic-hot-tier-budget/) every container shared the tag `app.replay`, and 71 of 848,190 returned records were mixed. This rig repeats that record shape and path without Elasticsearch. +The [Elasticsearch hot-tier benchmark](../elastic-hot-tier-budget/RECORD.md) also sent every container under one tag, and 71 of its 848,190 returned records were mixed. -## Result +## Records returned to Fluent Bit, per case -A stack trace counts as whole when its head and all its continuation lines came back alone in one record. +Each case sends 512 episodes, short runs of lines, and 12 of them pause for 1 to 6 s partway through. A stack trace counts as whole when its head and every continuation line came back in one record with nothing else. | Case | Mixed, shipped | Mixed, fix | Traces whole, shipped | Traces whole, fix | |---|---:|---:|---:|---:| @@ -14,29 +14,35 @@ A stack trace counts as whole when its head and all its continuation lines came | b: kafka startup lines after jaeger | 509 | 0 | none sent | none sent | | c: two traces interleaved | 512 | 0 | 0 of 1,024 | 1,021 of 1,024 | | d: one container (control) | 0 | 0 | 509 of 512 | 509 of 512 | -| e: a, b, c, one tag each | 0 | 0 | 508 of 511 | 508 of 511 | +| e: a, b, c, a tag per container | 0 | 0 | 508 of 511 | 508 of 511 | | Total | 1,533 | 0 | 1,017 of 2,559 | 2,547 of 2,559 | -On the shipped image, cases a and c produced a mixed record in every episode. Case e sends the same patterns with one tag per container and returned none, so the mixing follows the shared tag. With the fix, all 5,868 records containing sent lines came back under the tag they were sent with. +On the shipped image, every episode in cases a and c produced a mixed record, paused or not, as did case b except three episodes paused 5 or 6 s. Case e sends the same patterns with one tag per container and returned none, so the mixing follows the shared tag. One mixed record from case b, filed under the jaeger container, contains kafka's four startup lines: -- Both builds returned all 22,557 lines sent, each exactly once. -- The fix build still split 12 traces, all in episodes paused 5 or 6 s against a 5 s group flush timeout. The six in cases d and e also split on the shipped image. +``` +{"level":"info","ts":1792968679.3069091,"caller":"grpc@v1.60.0/clientconn.go:532","msg":"[core][Channel #1] Channel Connectivity change to READY","system":"grpc","grpc_log":true,"e103id":"4429"} +===> User e103id=4430 +uid=1000(appuser) gid=1000(appuser) groups=1000(appuser) e103id=4431 +===> Setting default values of environment variables if not already set. e103id=4432 +===> Configuring ... e103id=4433 +``` -## How it was measured +- With the fix, all 5,868 records with sent lines came back under `tenx.` plus their sent tag. The shipped run did not record returned tags. +- The fix build still split 12 traces, all in episodes paused 5 or 6 s against a 5 s group flush timeout. The six in cases d and e split on the shipped image too. -A Python sender writes Fluent Forward records to Fluent Bit 4.2, which forwards them to the 10x engine over one connection and writes what comes back to a file. `score.py` matches each returned line to the line sent through a sequence token at the end of every line. Each case sends 500 episodes back to back and 12 more paused between 1 and 6 s, 22,557 lines in all. The engine ran with 0.5 CPU, 512 MB and no budget set. +## How the lines were sent and matched -## Limits +A Python sender writes Fluent Forward records to Fluent Bit 4.2, which forwards them to the 10x engine over one connection and writes what comes back to a file. A sequence token ending every line lets `score.py` match each returned line to its sent line. Apart from the 12 pauses, episodes go out 1 ms apart. The engine ran with 0.5 CPU and 512 MB. -- One run per build of about five minutes, on lines from a sender script. Real container logs were not tested. -- The fix build is a local image of 1.1.150 with the engine jar and modules from the fix commits. -- The shipped run did not record the tag each record came back under, so the tag check covers the fix run only. +## Scope of the run -## Run it +- One run per build, about five minutes each, on lines from a sender script, not real container logs. +- The fix build is a local image: the shipped image with the engine jar and modules from the fix commits. -Scoring recomputes from the committed results, in this directory: +## Run it ``` +# from this directory: rescore the committed results, then rerun python3 score.py --self-test python3 score.py results/shipped-1.1.150/data # exits 1: mixed records python3 score.py results/fix2-1.1.151/data # exits 0 @@ -45,4 +51,4 @@ python3 score.py results/fix2-1.1.151/data # exits 0 Scoring needs only Python 3. A full rerun needs Docker with no other rig running and takes about five minutes. Running a fix build is in [RECORD.md](RECORD.md#running). -The complete record is in [RECORD.md](RECORD.md): the engine arguments, every case's line sequence, the scoring rules and each build's image and commits. +The complete record is in [RECORD.md](RECORD.md): engine arguments, every case's line sequence, scoring rules and each build's image and commits. diff --git a/elastic-hot-tier-budget/README.md b/elastic-hot-tier-budget/README.md index 3c0e7ee..8e8f285 100644 --- a/elastic-hot-tier-budget/README.md +++ b/elastic-hot-tier-budget/README.md @@ -1,37 +1,33 @@ # Elastic hot-tier budget -Can a size budget on an Elasticsearch hot tier stay where it was set when log volume steps up 2.5 times within an hour? In this test the budget stayed when the lines over the limits went to a warm node. From 20 minutes after the step to the hour's end, hot sat between 0.94 and 1.04 times the budget, while a fixed reroute by service rose to 1.66 times its level. With those lines on the hot node, hot peaked at 1.43 times. +The size of an Elasticsearch cluster follows how much data sits in its hot tier, because hot nodes carry the local SSD and the memory that keep that data searchable. Here a run every 10 minutes read the hot node's store and set the 10x engine's byte limit per log statement to fit a budget, and lines over a limit moved to an index that ILM sends to frozen. With that index on a warm node, through an hour at 2.5 times the log volume, the hot tier stayed between 0.94 and 1.04 times the budget from 20 minutes after the step. -The 10x engine limits bytes per log statement, and a run every 10 minutes rewrites them against the hot node's store. Lines over a limit (moved lines below) go to an index that rolls over every 10 minutes and then mounts on frozen. +A fixed reroute by service in Fluent Bit, on a second cluster, rose to 1.66 times its level before the step. With the moved lines' index on hot, the engine's hot tier peaked at 1.43 times the budget, above the reroute's peak. -## Result +## What the hot node's index store read -Hot node index store from node stats, around one step from 56 to 140 lines a second, against a 63 MB budget. "Before" is the 20-minute mean before the step. +The feed stepped from 56 to 140 lines a second, and the budget was 63 MB. -| Setup | Moved lines written to | Before | Peak in the hour | End of the hour | +| Setup | Moved lines written to | Mean, 20 minutes before | Peak in the hour | End of the hour | |---|---|---:|---:|---:| | 10x engine | warm node | 61.6 MB | 65.7 MB | 65.2 MB | | Reroute by service | warm node | 47.4 MB | 78.7 MB | 78.7 MB | | 10x engine | hot node | 59.9 MB | 90.0 MB | 64.2 MB | | Reroute by service | hot node | 50.0 MB | 85.9 MB | 81.8 MB | -The bound was 1.10 times the budget at every reading from 20 minutes after the step. With moved lines on warm, none of 25 readings crossed it. With moved lines on hot, 13 of 24 did, because each moved-lines index stays on hot for its 10 minutes of writes and about 7 more until it mounts. +The bound was 1.10 times the budget at every reading from 20 minutes after the step. With moved lines on warm, none of 25 readings went over it. With moved lines on hot, 13 of 24 did, because each moved-lines index stays on hot for its 10 minutes of writes and about 7 more until it mounts. The reroute rose because the other services' lines stay on hot for 2 hours. With either placement, each cluster received every non-blank line. -- The reroute grew because the other services' lines stay on hot for 2 hours. -- In the warm run every non-blank line arrived: 1,025,014 in the 10x cluster, 1,025,170 documents in the reroute cluster. +## How the two clusters were set up and read -## How it was measured +Two self-managed Elasticsearch 9.5.5 clusters shared one Docker host, each with a hot node and a frozen node that also took the warm role in the warm run. Fluent Bit sent a replay of the published OpenTelemetry demo sample through the 10x engine to the first cluster and a copy of every line to the second. For the second cluster it routed `opentelemetry-collector`, 68.9% of the bytes in the first 20 minutes, to an index with the moved lines' lifecycle. Every 100 seconds a meter read each hot node's `indices.store.size_in_bytes`. Both methods were fixed in advance: [TWO_ARM_METHOD.md](TWO_ARM_METHOD.md) and [TWO_ARM_METHOD_WARM.md](TWO_ARM_METHOD_WARM.md). -Two self-managed Elasticsearch 9.5.5 clusters shared one Docker host, each with a hot node and a warm and frozen node. Fluent Bit replayed the OpenTelemetry demo sample to both, through the 10x engine to the first. The second routed `opentelemetry-collector`, the service with the top half of the bytes, to a stream with the moved lines' lifecycle. The budget was 0.7 of where a dry run without limits said hot would settle. Every 100 seconds a meter read each hot node's `indices.store.size_in_bytes`. The methods were fixed in advance: [TWO_ARM_METHOD.md](TWO_ARM_METHOD.md) and [TWO_ARM_METHOD_WARM.md](TWO_ARM_METHOD_WARM.md). +## Time scale and scope -## Limits +- Every time constant was divided by 6, so the hourly run came every 10 minutes and kept lines stayed 2 hours on hot. A slow ramp against days of retention is not shown. +- Each comparison ran once, on one machine. Elastic Cloud Hosted was not tested. +- No bill was measured, and neither was the cost of warm, where moved lines peaked at 20.7 MB. -- Every time constant was divided by 6, so kept lines stayed 2 hours on hot and a slow ramp against days of retention is not shown. -- Each placement ran once, about four hours, at megabyte scale on one machine. -- The cost of warm, where moved lines peaked at 20.7 MB, was not measured. -- No bill was measured. RECORD.md prices hot at Elastic's published rates. - -## Run it +## Rerunning the warm-placement run ``` # from elastic-hot-tier-budget/, with Docker running @@ -45,4 +41,4 @@ python3 rig/score-two-arm.py "$RESULTS" A rerun needs Docker with room for 7.1 GiB of container limits, a built log10x-mcp checkout at c1b456a and a GitHub token that can push the policy branch. -[RECORD.md](RECORD.md) has every run, its criteria, per-stream tables, the cost projection and the rig. +[RECORD.md](RECORD.md) has every run, its criteria, the per-stream tables, the cost projection and the rig. diff --git a/elastic-hot-tier/README.md b/elastic-hot-tier/README.md index 9211b5f..b769ff4 100644 --- a/elastic-hot-tier/README.md +++ b/elastic-hot-tier/README.md @@ -1,12 +1,10 @@ # Elastic hot tier -When lines over a per-statement budget go to an Elasticsearch index that rolls over hourly and moves to frozen, how long do they stay on hot? Each moved line left hot 30 to about 110 minutes after it was written, and all 159,536 moved records reached frozen. A fixed budget moved 59% to 73% of message bytes off hot, but the bytes sent to hot still grew with the load. +The size of an Elasticsearch cluster follows how much data sits in its hot tier, because hot nodes carry the local SSD and the memory that keep that data searchable. Elasticsearch tiers by index, so lines over a log statement's byte budget, marked by the 10x engine, go to an index that ILM rolls over hourly and moves to frozen. On a self-managed 9.5.5 cluster, one fixed budget moved 59% of the message bytes to that index, and 65% to 73% at 2.5 times the load. Each moved line left hot 30 to about 110 minutes after it was written. -The 10x engine gives each log statement a byte budget per 4-minute window. Fluent Bit sends lines over it to the `tenx-tierdown` alias, whose hourly indices are mounted on frozen as searchable snapshots at 0 bytes of local store. +## What the budget moved, hour by hour -## Result - -Message bytes per hour, 2026-10-22 UTC, with a budget of 22,148 bytes per statement per 4-minute window. +The budget was 22,148 bytes per statement per 4-minute window on each node. Message bytes an hour on 2026-10-22 (UTC), from Fluent Bit's copy of the engine's output: | Hour | Phase | Bytes sent to hot | Bytes moved | Share moved | |---|---|---:|---:|---:| @@ -19,24 +17,26 @@ Message bytes per hour, 2026-10-22 UTC, with a budget of 22,148 bytes per statem | 06:15 | budget on, 2.5x load | 4,174,512 | 11,471,373 | 73% | | 07:15 | budget on, 2.5x load | 4,050,664 | 9,484,012 | 70% | -With the budget on, no statement kept more than its budget on hot in any window, beyond a 5-line sample and 50 extra error lines for the storming statement. At 2.5 times the load hot still took about 4.3 MB an hour, because the statements under budget grew too. +No statement kept more than the budget on hot in any window, beyond a 5-line sample and up to 50 extra error lines. The budget is per statement, so growth in statements under it still reached hot. + +## How long moved lines stayed on hot + +At ILM's default 10-minute poll, indices 2 to 7 each rolled over 70 minutes after creation and entered the frozen phase 10 minutes later. The searchable snapshot action then took three more polls, so each index left hot 30 to 40 minutes after rollover, mounted at 0 bytes of local store. Index 1 sat empty until the budget took effect, and index 8 received no line. -- Six of the eight hourly indices of moved lines spent 100 to 110 minutes on hot. -- All 372,783 non-blank lines sent reached Elasticsearch. Term queries on `tenx_hash` for the ten largest moved patterns matched the local counts on frozen. -- Of the moved bytes, 86% came from three OpenTelemetry Collector statements that no rule named. +For the ten largest moved patterns, a term query on `tenx_hash`, the engine's pattern hash, returned the local count from frozen. All 372,783 non-blank lines sent reached Elasticsearch, and all 159,536 moved records ended on frozen. -## How it was measured +## How the cluster was read -Elasticsearch 9.5.5 ran self-managed with one hot node, one frozen node and a trial license. The load was the published OpenTelemetry demo sample at 7 lines a second, then 2.5 times that for 3 hours plus a new service and a 20-minute error storm. The budget was set from the second baseline hour so that half of its message bytes were over it. The table counts bytes in Fluent Bit's local copy of what the engine returned. Index sizes and ILM state came from the cluster's own APIs every 10 minutes. +The cluster had one hot node and one frozen node, and its APIs were read every 10 minutes. The published [OpenTelemetry demo sample](https://github.com/log-10x/config/releases/tag/otel-sample-v1) was replayed into Fluent Bit at 7 lines a second for 6 hours, then 17.5 a second for 3 hours with an injected service and an error storm. -## Limits +## What this run does not show -- Hot-node bytes say nothing about a bill, since Elastic Cloud Hosted bills the configured size. -- The budget stayed fixed, so the run does not show hot kept at a set size under growth. -- No comparison was made with Elastic's own tools or logsdb. -- The budget aimed at 50% of the baseline message bytes and moved 59%. The storm statement's kept count fits the engine sizing the whole forwarded record, which the run leaves open. +- With the budget fixed, the run does not show a hot tier kept at a set size as load grows. +- The budget was set so half of the second baseline hour's message bytes exceeded it, and 59% moved. The storm line's kept count fits the engine sizing the whole forwarded record rather than the message, which the run leaves open. +- The run shows nothing about a bill, since Elastic Cloud Hosted bills the configured size. +- Elastic's own tools and logsdb were not measured. -## Run it +## Rerunning the benchmark ``` cd elastic-hot-tier/rig @@ -47,6 +47,6 @@ RIG_DIR=$PWD RIG_DATA=$PWD/../results/data docker compose -p elastic-rig -f comp python3 analyze.py ``` -A rerun takes about 11 hours and needs Docker with about 5 GB free memory, Node 20, Python 3, a built log10x-mcp checkout and otel-sample-200mb.log from release otel-sample-v1. This run used engine 1.1.147 and log10x-mcp 1.30.77. +A rerun takes 11 hours and needs Docker with about 5 GB of memory free, Node 20, Python 3 and a built log10x-mcp checkout (1.30.77, engine 1.1.147). -[RECORD.md](RECORD.md) has every check, the cluster's hourly readings and the rig. +[RECORD.md](RECORD.md) has every check, each index's ILM timeline, the hourly readings and the rig. diff --git a/elastic-store-four-ways/README.md b/elastic-store-four-ways/README.md index 766d031..7c78900 100644 --- a/elastic-store-four-ways/README.md +++ b/elastic-store-four-ways/README.md @@ -1,38 +1,36 @@ # Elasticsearch store, four ways -On Elasticsearch 9.5.5, does the 10x engine's compact form take less index store than Elasticsearch's own log features? Against plain logsdb it does, but against logsdb with the message field mapped as `pattern_text` it does not. That native setup, sorted on its template id, took the least store of all. Compact on logsdb took 53.7% more, and 27.2% more with the same sort on its own template id. +Elasticsearch stores logs in less space with logsdb index mode and the `pattern_text` field type, which keeps each message as a template plus its arguments. 10x stores each repeated format string once, as a template, and ships each event as a reference to it plus the values that change. On one Elasticsearch 9.5.5 node with the published OpenTelemetry demo capture, logsdb with `pattern_text`, sorted on its template id, took the least store, 34.7% of a raw index. Compact on logsdb took 53.7% more than that, and 27.2% more when sorted the same way on its own template id. -Per-field disk usage shows why. The stored text in the sorted compact index is about the size of what `pattern_text` stores. The extra is an inverted index over the encoded message, with every variable value as a term, including the container, pod and image ids in each event. With `pattern_text` those values sit in keyword doc values. +Per field, the sorted compact index stores about as much message text as `pattern_text` does. The compact form adds the inverted index over the encoded message, with a term for every variable value, including the container, pod and image ids. The `pattern_text` index keeps those values as keyword doc values. -## Result +## Primary store of each index -Primary store after a force merge to one segment, as the mean of two repetitions: +Each figure is the primary store after a force merge to one segment, averaged over two repetitions. -| Variant | Index mode | Store | Of raw | Of logsdb | +| Index | Index mode | Store | Share of raw | Share of logsdb | |---|---|---:|---:|---:| -| Raw | standard | 20,107,900 B | 100.0% | 167.6% | -| Logsdb | logsdb | 12,000,374 B | 59.7% | 100.0% | -| Logsdb + pattern_text | logsdb, sorted on template id | 6,974,760 B | 34.7% | 58.1% | +| Raw | standard | 20,107,900 B | 100% | 167.6% | +| Logsdb | logsdb | 12,000,374 B | 59.7% | 100% | +| Logsdb, pattern_text | logsdb, sorted on template id | 6,974,760 B | 34.7% | 58.1% | | Compact | standard | 14,905,394 B | 74.1% | 124.2% | | Compact on logsdb | logsdb | 10,720,088 B | 53.3% | 89.3% | | Compact on logsdb, sorted | logsdb, sorted on template id | 8,874,049 B | 44.1% | 73.9% | -Compact on logsdb is smaller than raw and plain logsdb, and every compact variant is larger than `pattern_text`. The compact figures include the template dictionary index (537,262 B). +Compact on logsdb is smaller than both raw and logsdb, but in standard mode the compact index is 24.2% larger than logsdb. Every compact figure includes the template dictionary index, 537,262 B in the first repetition. -- The template sort took effect only when set with `index.sort.field`. With `index.logsdb.default_sort_on_message_template` alone, on documents with no host.name field, the segments were sorted on @timestamp only and the `pattern_text` index took 10,649,904 B ([results-setting-only/](results-setting-only/)). +- With only `index.logsdb.default_sort_on_message_template` set, on documents without a host.name field, segments were sorted on @timestamp alone and the `pattern_text` index took 10,649,904 B ([results-setting-only/](results-setting-only/)). The table sets `index.sort.field` instead, as Elastic's "Custom sorting" example does. -## How it was measured +## How the store was read -The sample is the published OpenTelemetry demo capture, 215,039,161 B in 197,430 JSON lines as Fluentd ships them. The native indexes store the parsed record, one document per line. The compact indexes store one document per event (157,228, with multi-line events folded into one) plus 3,030 templates. The compact form decoded back to the sample byte for byte. - -Everything ran on one Elasticsearch 9.5.5 node in Docker on a trial license, with one shard and best_compression per index. The repetitions differed by at most 0.07%, and all 66 checks in [results/results.md](results/results.md) pass. +The sample is the published [OpenTelemetry demo capture](https://github.com/log-10x/config/releases/tag/otel-sample-v1), 215,039,161 B in 197,430 JSON lines as Fluentd ships them. The native indexes store the parsed record, one document per line, and the compact indexes one document per event (157,228, multi-line events folded into one) plus 3,030 templates. The compact form decoded to the sample byte for byte. Each index had one primary shard and best_compression, and its primary store came from the index stats API. Repetitions differed by at most 0.07%, and all 66 checks in [results/results.md](results/results.md) pass. ## Limits -- The test measures store on one sample and one node, and not query cost, ingest cost or Elastic Cloud pricing. -- Search-time decoding was not tested, because the decoding plugin is built for Elasticsearch 8.17.0, not 9.x. -- Every compact variant carries the metadata inside the encoded event. Keeping the metadata as separate fields was not measured. -- `pattern_text` is a subscription feature and was measured on a trial license. The basic license accepted it with templating off, and that case was not measured. +- Query cost, ingest cost and Elastic Cloud pricing were not measured. +- Search-time decoding was not tested, because its plugin is built for Elasticsearch 8.17.0. +- The compact form carries the metadata inside each event, and metadata kept as fields was not measured. +- Every index ran with Elastic's paid features on. On the free basic license, a `pattern_text` index came back with templating off and stored source, which was not measured. ## Run it @@ -42,6 +40,6 @@ cd elastic-store-four-ways ./run.sh # full run: encode, decode check, six variants twice, report ``` -The report needs only Python 3. A full run also needs Docker, curl, about 2 GB of memory and 3 GB of free disk, and takes about 10 minutes. +The report needs Python 3. A full run also needs Docker, curl, about 2 GB of memory and 3 GB of free disk, and takes about 10 minutes. -[METHOD.md](METHOD.md) is the plan written before the run, and [RECORD.md](RECORD.md) is the full record with per-field bytes, the license probe and hashes. +The full record, with each repetition, the license probe and the compact form's hashes, is in [RECORD.md](RECORD.md), and the plan written before the run in [METHOD.md](METHOD.md). diff --git a/intent/README.md b/intent/README.md index 2ae7df4..b59ff1e 100644 --- a/intent/README.md +++ b/intent/README.md @@ -1,12 +1,10 @@ # Intent: does each pattern hold one logging statement? -A cap or sample rate set on a log pattern is only safe when the pattern stands for one logging statement. This benchmark labels each event with the source line that wrote it and counts how often a tool puts two statements in one pattern. On three captures the 10x engine merged 0.01% to 0.21% of events, fewer than Drain3 in every configuration. Drain3 with masking split fewer events and produced fewer patterns on one of two demo captures and on ZooKeeper. +A log platform groups repeated lines and shows a count beside each group. A cap, a sample rate or a route set on a group acts on every statement inside it. Against labels naming the source line behind the events of three captures, the 10x engine put 0.01% to 0.21% of events in another statement's group, fewer than [Drain3](https://github.com/logpai/Drain3) in every configuration. Drain3 with masking split fewer events and produced fewer patterns on two captures. -A merged event shares its pattern with a different statement, so a cap on that pattern acts on both. A split event sits outside its statement's main pattern, and that pattern can still be capped on its own. +## Events grouped with a different statement -## Result - -Share of events merged with another statement. otel215 and rec1155 are two OpenTelemetry demo captures. +otel215 and rec1155 are OpenTelemetry demo captures. | Tool | otel215 | rec1155 | ZooKeeper | |---|---:|---:|---:| @@ -15,22 +13,28 @@ Share of events merged with another statement. otel215 and rec1155 are two OpenT | Drain3, typed-token masking | 0.06% (99) | 1.81% (749) | 0.55% (412) | | Drain3, factory defaults | 0.15% (239) | 3.34% (1,381) | 1.12% (831) | -Drain3's merges on otel215 pair statements that differ in one word, such as Grafana's "starting to provision dashboards" and "finished to provision dashboards". +Drain3's merges on otel215 pair statements that differ in one word, such as Grafana's "starting to provision dashboards" and "finished to provision dashboards". Fed one record at a time, as it normally runs, Drain3 with example-config masking merges 4.48% of otel215's events, 6,930 of them because the cart service's .NET logger prints this header above its three statements: + +``` +info: cart.cartstore.ValkeyCartStore[0] +``` + +The engine names that event from its message line. + +## Split events and pattern counts -- Drain3 splits 3.47% of otel215's events against the engine's 4.59%, and with masking 0.32% of ZooKeeper's against 1.21%. The engine splits fewer on rec1155. -- Drain3 produces 865 patterns on otel215 against 2,277, and 58 on ZooKeeper against 97. Two Grafana statements make 1,331 of the engine's, one pattern per migration id. -- Fed one record at a time, as it normally runs, Drain3 merges 4.48% of otel215's events, because every cart service message starts with the same .NET header line. +A split event sits outside its statement's most frequent pattern, which costs that statement an extra rule. With example-config masking, Drain3 splits 3.47% of otel215's events against the engine's 4.59% and 0.32% of ZooKeeper's against 1.21%. On rec1155 the engine splits fewer, 9.95% against 10.37%. Drain3 produces 865 patterns for otel215's labeled events against 2,277, and 58 for ZooKeeper's against 97. Two Grafana migration statements make 1,331 of the engine's, one per migration id. -## How it was measured +## How the events were labeled -otel215 has 157,228 events from Fluentd, rec1155 has 41,573 from Fluent Bit, and ZooKeeper is LogHub's 74,380-line log from 84 statements. Each event was labeled from source code alone with the file and line that wrote it, and the labels were frozen before any tool's output was read. They cover 99.58% of otel215, 99.42% of rec1155 and all of ZooKeeper. The engine (release 1.1.142) ran as shipped, Drain3 0.9.11 was given the same events, and `bench/score.py` scored both against the labels. +The labels cover 99.58% of otel215's 157,228 events, 99.42% of rec1155's 41,573 and all 74,380 lines of LogHub's ZooKeeper log, come from source code alone, and were frozen before any tool's output was read. Engine 1.1.142 ran as shipped, and Drain3 0.9.11 was given the engine's events with the settings of [pattern-identity](../pattern-identity) or its factory defaults. The scorer gives each pattern to the statement with most of its events and counts the rest as merged. -## Limits +## Scope of the test -- No capture is a clean out-of-sample test. Both demo captures and 2,000 ZooKeeper lines were in view during the engine's rule work. -- Log10x wrote the labels, and each names a source line in `labels/` that can be checked. -- The engine's built-in library lists the source behind 145,612 of otel215's 156,563 labeled events and none of ZooKeeper's. On otel215 statements outside it, the engine merges 0.13% of events against Drain3's 0.86%. -- Only two applications are covered, and the collector's debug exporter wrote 85,739 of otel215's 157,228 events. +- No capture is a clean out-of-sample test. The engine's rule for naming a multi-line event was built with both demo captures in view, and 2,000 ZooKeeper lines were in view during development. +- Log10x wrote the labels, each naming a source line in `labels/`. +- The engine's symbol library lists the source behind 145,612 of otel215's 156,563 labeled events and none of ZooKeeper's. Outside it on otel215, the engine merges 0.13% of events against Drain3's 0.86%, and Drain3 groups more of them one statement to one pattern, 65.49% against 62.89%. +- The captures come from two applications, and three statements of the collector's debug exporter wrote 85,739 of otel215's events. ## Run it @@ -41,4 +45,4 @@ cd intent && ./run.sh # full rerun A full rerun needs Docker, curl and Python 3 and takes about 55 minutes on an Intel laptop. -[RECORD.md](RECORD.md) has the full score tables, the Drain3 settings and the image digests, and [METHOD.md](METHOD.md) explains how each label was made. +[RECORD.md](RECORD.md) has every score table and the Drain3 settings, and [METHOD.md](METHOD.md) explains how each label was made. diff --git a/naming-stability/README.md b/naming-stability/README.md index dfd941c..fd52521 100644 --- a/naming-stability/README.md +++ b/naming-stability/README.md @@ -1,10 +1,10 @@ # Naming stability across releases -When the 10x engine moves to its next release, do the same log events keep the same pattern names? Across releases 1.1.140, 1.1.141 and 1.1.142, on two captures of the OpenTelemetry demo, no event changed its pattern name, its pattern hash or its template hash. +The 10x engine gives each log event a pattern name, a pattern hash (`tenx_hash`, the hash of the name) and a template hash, and a per-pattern budget, a routing rule or a cost report keys on the pattern. Three consecutive engine releases, 1.1.140, 1.1.141 and 1.1.142, ran over two captures of the OpenTelemetry demo. Across 157,228 events in one capture and 41,573 in the other, no event changed its pattern name, pattern hash or template hash between releases. -Each event gets a pattern name, a pattern hash (`tenx_hash`, the hash of the name) and a template hash. Each release ran with the configuration and symbol library its own image ships. +10x gives each line an ID taken from the fixed words the code prints, leaving out every value that is not a word in its library. Nodes running the same pinned library and engine version agree on it. All three images carry the same 1.1.89 library, so this test keeps the library and changes the engine and its shipped configuration. -## Result +## Names the three releases gave the same events | Capture | Releases | Events | Events changed | Patterns renamed | |---|---|---:|---:|---:| @@ -13,32 +13,30 @@ Each event gets a pattern name, a pattern hash (`tenx_hash`, the hash of the nam | otel215 | 1.1.140 to 1.1.141 | 157,228 | 0 | 0 of 2,687 | | otel215 | 1.1.141 to 1.1.142 | 157,228 | 0 | 0 of 2,687 | -Each pair of releases cut each capture into the same events, so events were compared one for one. An event counts as changed when any of its three names or hashes differs. +Each pair of releases cut each capture into the same events, so events were compared one for one. A pattern counts as renamed when any of its events carries another name in the later release. -- The three images carry the same 1.1.89 symbol library. In the shipped configuration, 18 files differ between 1.1.140 and 1.1.141 and none between 1.1.141 and 1.1.142. -- In 1.1.141, a line opening with the ANSI dim code starts its own event. Neither capture has such a line, so that change had nothing to act on here. -- The intent benchmark reports fewer patterns for the same 1.1.142 image (2,277 on otel215, 406 on rec1155) because it counts only the events it has source labels for. +- The 1.1.141 release notes say the engine is unchanged, and 18 files of the shipped configuration differ, among them the group, template and rate modules. In 1.1.142 no configuration file differs, and its notes list two engine changes. +- In 1.1.141 a line opening with the ANSI dim code starts its own event, and neither capture has such a line. -## How it was measured +## How each release was run -The two captures are the ones the [intent benchmark](../intent) uses, each checked by SHA-256. Each release's engine image was pinned by digest and run on each capture as a file, writing every event's pattern name, both hashes and its text. A script compared consecutive releases event by event. A pattern of the earlier release counts as renamed when any of its events carries another name in the later one. The runs were made on 2026-10-15. +The captures are otel215 from the [otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1) release and rec1155 from this repository's [intent-data-v1](https://github.com/log-10x/benchmarks/releases/tag/intent-data-v1) release, both checked by SHA-256. Each release's image, pinned by digest, read each capture as a file and wrote every event's pattern name, both hashes and its text. A script compared consecutive releases event by event. The runs were made on 2026-10-15. -## Limits +## Applications, inputs, releases and libraries outside this test -- Both captures come from one application, the OpenTelemetry demo (versions 2.2.0 and 2.1.3). Another application, or the ANSI-colored lines that 1.1.141 regroups, can rename where these do not. -- The engine read each capture as a file. The 1.1.142 release notes describe a flush timer change on the forwarder input, which is not measured here. -- Only three releases are compared. Names did change between two earlier ones: on otel215 the intent benchmark counts 6,944 merged events for 1.1.132 and 13 for 1.1.133. -- A library change is not measured, since all three images carry the same one. The page makes no promise about future releases. +- Both captures come from one application, the OpenTelemetry demo (versions 2.2.0 and 2.1.3). Another application, or the ANSI-colored lines that 1.1.141 regroups, can change names where these did not. +- The 1.1.142 release notes describe a flush timer change on the forwarder input, and no forwarder run is measured here. +- Releases before 1.1.140 are not compared, and names did change between two of them: on otel215 the [intent benchmark](../intent) counts 6,944 merged events for 1.1.132 and 13 for 1.1.133. +- A library change is not measured, and a new library is where names built from library words would move. These results promise nothing about later releases. ## Run it -From this directory: - ``` +cd naming-stability python3 bench/compare.py --render | diff - results/results.md # check the table against compare.json -NS_PRUNE=1 ./run.sh # full rerun: downloads both captures, pulls the three images +NS_PRUNE=1 ./run.sh # full rerun: downloads both captures, pulls the three images ``` -The full rerun needs Docker, curl and Python 3, and took one to three minutes per release on an Intel macOS laptop, plus the image pulls. +The check needs only Python. The full rerun needs Docker, curl and Python 3, and took one to three minutes per release on an Intel macOS laptop, plus the image pulls. -The complete record is in [RECORD.md](RECORD.md): the changes each release's notes list, how the pattern counts reconcile with the intent benchmark, the full method and the files. +[RECORD.md](RECORD.md) has the changes each release's notes list, how the pattern counts reconcile with the intent benchmark, the full method and the files. diff --git a/splunk-license/README.md b/splunk-license/README.md index d25197c..e096cc5 100644 --- a/splunk-license/README.md +++ b/splunk-license/README.md @@ -22,7 +22,7 @@ The meter counts the text of each event. The capture is 215,039,161 bytes on dis ## How the meter was read -[The capture](https://github.com/log-10x/config/releases/tag/otel-sample-v2) is 215,039,161 bytes and 197,430 lines of output from 36 Kubernetes containers in an OpenTelemetry demo cluster. The engine joins the lines of a multi-line event into one event, so the 197,430 lines are 157,228 events, and the run stops if any line is left unassigned. Both forms went through the same Universal Forwarder into the same Splunk Enterprise 10.4.3 instance on the same license day. The 10x engine writes the compact form of the capture as two files, the compact events and the template dictionary, and the forwarder sends each to an index of its own. Splunk Enterprise runs in a container on the download trial license, which allows 500 MB a day, and both arms fit in one license day. Splunk's license usage log reports usage per index, and the per-index entries add up to the day's rollup total to the byte: 292,374,199. +[The capture](https://github.com/log-10x/config/releases/tag/otel-sample-v2) is 215,039,161 bytes and 197,430 lines of output from 36 Kubernetes containers in an OpenTelemetry demo cluster. The engine joins the lines of a multi-line event into one event, so the 197,430 lines are 157,228 events, and the run stops if any line is left unassigned. Both forms went through the same Universal Forwarder into the same Splunk Enterprise 10.4.3 instance on the same license day. The 10x engine writes the compact form of the capture as two files, the compact events and the template dictionary, and the forwarder sends each to an index of its own. Splunk Enterprise runs in a container, and both arms were metered on the same license day. Splunk's license usage log reports usage per index, and the per-index entries add up to the day's rollup total to the byte: 292,374,199. ## Limits