For comparison: README prose taken from the published posts

datadog-ceiling/README.md

@@ -1,8 +1,8 @@
 # Datadog ceiling
 
-Does a monthly budget for Datadog Standard indexing stay where it was set when log volume grows two to three times? In this run it did. Log volume grew to 2.9 times the baseline, Standard indexing stayed at the budgeted rate, and every line still reached Datadog.
+Datadog's own usage metric showed Standard indexing staying at a set number while ingested events grew 2.6 to 2.9 times on synthetic traffic, and Flex took the rest. A pattern routed to Flex stops paying the standard index rate. The lines stay queryable in Datadog's own Log Explorer, with no rehydration step.
 
-The limit is set per log statement. Each hour, month-to-date usage is read from Datadog's own usage metric and a limit is written for every statement. Lines over a statement's limit go to a Flex index, so they stay searchable in Datadog. An error statement keeps 50 extra lines a window in Standard, so monitors on errors still fire.
+Setting a monthly number of Standard-indexed events comes first. An hourly run reads Datadog's usage metric month to date and writes a cap per pattern and a total per container into the policy the sidecars read, and a container the run has not seen yet keeps the level the known containers share. A pattern's first five events of each window stay in the standard tier, and an error pattern over its cap keeps 50 more there in each window on each node, so a monitor on an error keeps firing.
 
 ## Result
 
@@ -20,23 +20,20 @@ Events an hour on 2026-10-20 (UTC), as Datadog's usage metric counted them. The
 
 Four of the five budgeted hours came in under the allowed rate. The 06:00 hour was 393 events over it, inside the allowance for sample lines and error lines, which came to between 1,475 and 2,785 events an hour.
 
-Three more checks from the same run:
-
-- A service that first appeared mid-run was limited from its first window. Of its 476 lines a window, 31 went to Standard until the 07:00 update gave it a limit of its own.
 - During an error storm, a log monitor on the storming statement went to Alert at 06:03:43 and back to OK at 06:22:43, after the storm ended.
 - 381,856 of 381,856 lines reached Datadog.
 
 ## How it was measured
 
-The application is the OpenTelemetry demo, sending to a real Datadog org with Standard indexing at 15-day retention and one Flex index. Two baseline hours ran with no limits. Then the budget was switched on and load was raised.
+The application is the OpenTelemetry demo, sending to a real Datadog org with Standard indexing at 15-day retention and one Flex index. The engine marks the events that should move with an attribute, and a Flex-only index whose filter matches that marker receives them, because Datadog indexes are first match wins and that index is ordered before the catch-all. Two baseline hours ran with no limits. Then the budget was switched on and load was raised.
 
-Every number in the table comes from `datadog.estimated_usage.logs.ingested_events`, filtered to `datadog_is_excluded:false` and split by `datadog_index`. The rig's own counts agreed with that metric to within 16 events an hour.
+Every number in the table comes from `datadog.estimated_usage.logs.ingested_events`, filtered to `datadog_is_excluded:false` and split by `datadog_index`.
 
 ## Limits
 
-- One run of under eight hours, on a demo application, with a budget of $10.89 a month. The run shows the level staying put under growth and says nothing about money saved on a real bill.
-- Logs only. Hosts, custom metrics and APM are outside this benchmark.
-- Lines over the limit go to Flex, which has its own price and its own query compute. Datadog monitors do not run on Flex logs, and the org needs Flex Logs on its plan.
+- One run of under eight hours, on a demo application, with a budget of $10.89 a month.
+- Ingestion is billed as before, and Flex charges its own lower rate for storage plus compute for the queries that read it.
+- Datadog runs no log monitors on Flex. Queries there run on Flex compute and are slower than the standard index.
 
 ## Run it
 
@@ -50,3 +47,5 @@ python3 rig/analyze.py
 A full rerun needs Docker, a Datadog org with Flex Logs and about eight hours. The steps are in [RECORD.md](RECORD.md#reproduce).
 
 The complete record of the run is in [RECORD.md](RECORD.md): every check, how the budget was set, each hourly budget file and the rig.
+
+The same measurement, with more context: https://www.log10x.com/blog/cut-datadog-log-bill-per-pattern/

loghub-compaction/README.md

@@ -1,40 +1,39 @@
 # LogHub compaction
 
-How much smaller than the original logs is the 10x engine's compact form, on the 14 full public LogHub datasets? Across all 14 the compact form came to 44.7% fewer bytes, weighted by size. Per dataset the reduction ran from 58.4% on Hadoop down to 4.7% on HealthApp.
+A log platform meters ingest by the byte, and most bytes in a log line are repetitive text. 10x stores each line's repeated text once as a template, and ships each line as that template's hash plus the line's values. The 14 LogHub datasets measured here hold 52,276,250 events and 5,732,535,052 bytes of input. Weighted by input bytes, they reduce 44.7%, and the spread across them is wide.
 
-The compact form is the encoded events plus the template dictionary needed to expand them back. Both are counted before any codec, so the figure stacks with whatever codec the storage layer applies after, such as gzip or ZSTD.
+Reduction means the bytes of the compact form, the encoded events plus the template dictionary needed to expand them, against the bytes of the original file, with no gzip or ZSTD on either side.
 
-## Result
-
-The table shows the four datasets over 100 MB, the highest and lowest of the other ten, and the total.
+## What the runs measured
 
 | Dataset | Events | Input bytes | Compact bytes | Reduction |
 |---|---:|---:|---:|---:|
 | Hadoop | 186,697 | 48,595,595 | 20,212,363 | 58.4% |
 | Spark | 32,952,127 | 2,941,224,304 | 1,248,606,093 | 57.5% |
-| BGL | 4,728,216 | 743,185,031 | 482,351,424 | 35.1% |
+| Apache | 52,004 | 5,135,876 | 2,295,929 | 55.3% |
+| OpenStack | 207,820 | 61,442,082 | 42,663,820 | 30.6% |
 | HDFS | 11,175,629 | 1,577,982,906 | 1,118,008,927 | 29.1% |
 | Android | 1,555,005 | 192,270,829 | 153,830,847 | 20% |
 | HealthApp | 204,648 | 23,529,930 | 22,432,794 | 4.7% |
 | All 14 | 52,276,250 | 5,732,535,052 | 3,167,463,894 | 44.7% |
 
-The four over 100 MB together came to 44.9%. The spread is wide, and HealthApp saves little. All 14 rows, with template counts, are in [results/results.md](results/results.md).
+The reduction tracks how much of each line is repeated text and how many lines share each template. HealthApp and Android, at 4.7% and 20%, are the floor. Both are mobile logs whose component names and tags ship as values, beside the timestamps and IDs, so little of each line is repeated text.
 
-- Ten of the 14 match, byte for byte, the figures in the log10x post [Log compaction, measured](https://www.log10x.com/blog/log-compaction-measured/), and every reduction matches to the decimal place.
-- Hadoop, Spark and OpenStack differ slightly. Each is many files read in directory order, and that order differs between the macOS file system of the post's run and the Linux container here.
-- Linux differs because its template count changes from run to run. A two-digit day matches both the MMM d and the MMM dd date formats, and the format is chosen per event. Its reduction stays the same.
+- HDFS and OpenStack sit at the low end of the middle because their lines are mostly values: block IDs, PIDs, and byte counts, with little fixed text between them, and the values ship on every event regardless.
+- Hadoop, Spark, and Apache sit at the high end because they write more repeated message text around their values.
+- All 14 rows, with template counts, are in [results/results.md](results/results.md).
 
-## How it was measured
+## Inputs and method
 
-The inputs are the full LogHub datasets from Zenodo record 3227177, not the 2,000-line excerpts, each checked against the md5 Zenodo publishes. The released engine image ran in a Linux container with its default symbol library, one encode config and no per-dataset compile. Each dataset started with an empty template cache, so its whole dictionary is charged to it. The reduction is one minus the size of the two output files (`encoded.log` for the events, `templates.json` for the dictionary) over the input bytes the engine read. Both outputs are hashed so a rerun can be compared byte for byte.
+Each run is one pass of the engine over one dataset: the original file goes in, and two files come out, the encoded events and the template dictionary. The LogHub datasets are the full files from the project's [Zenodo record](https://zenodo.org/records/3227177) rather than its 2,000-line benchmark excerpts. Every dataset went through the default symbol library that ships in the image. A library can be compiled from a specific codebase, and none of these runs did that.
 
 ## Limits
 
-- Storage on disk after gzip or ZSTD is not measured. A codec compresses both the original and the compact form.
+- A general-purpose compressor can beat these ratios on log text. A self-managed store also compresses what it holds, so the reduction does not carry to disk as-is.
+- The two LogHub sets left out, Thunderbird and Windows, are the collection's largest, so the 44.7% weighted figure covers the 14 included sets and is not a LogHub-wide number.
 - Query cost and a comparison with another tool's output are not measured.
-- The figures are for these 14 datasets. HDFS is the HDFS_1 set, and the Thunderbird and Windows datasets were not measured.
 
-## Run it
+## Reproducing a row
 
 ```
 cd loghub-compaction
@@ -46,3 +45,5 @@ python3 report.py --check   # recompute every figure from results/ (no Docker)
 A full run needs Docker, Python 3 and about 5 GB of free disk at the peak (Spark). The script downloads the archives from Zenodo and pins the engine image log10x/pipeline-10x:1.1.133 by digest.
 
 The full record is in [RECORD.md](RECORD.md): every dataset's events, bytes, ratio and template count set against the post's figures, the full method and the file list.
+
+The same measurement, with more context: https://www.log10x.com/blog/log-compaction-measured/

otel-denominators/README.md

@@ -1,8 +1,8 @@
 # OTel sample on three denominators
 
-How much does a log reduction figure depend on what was in the input? On one public capture from a Kubernetes cluster running the OpenTelemetry demo, the 10x engine's figure is 64.00% on the file as captured, 62.46% with the debug output and an added field removed, and 72.35% on the application messages alone. Taking away the Kubernetes metadata around each message raised the figure.
+The same stream was measured three ways, each decoded byte-identical. The stream is Kubernetes container output, captured through Fluentd from a cluster running the OpenTelemetry demo. As captured, metadata included, the reduction is 64.00%. On the log message text alone, metadata stripped, the reduction is 72.35%.
 
-The capture wraps each container's stdout line in a Fluentd envelope of Kubernetes and Docker metadata. Two objections follow from that. One says the file is padded by a `tenx_tag` field the log10x forwarder adds and by the demo collector's debug exporter. The other says OTLP has no such envelope, so the figure should collapse without it.
+About half the bytes are the demo collector's debug exporter, which the demo enables by default. Reduction means the bytes of the compact form, the encoded events plus the template dictionary needed to expand them, against the bytes of the original file, with no gzip or ZSTD on either side.
 
 ## Result
 
@@ -12,21 +12,19 @@ The capture wraps each container's stdout line in a Fluentd envelope of Kubernet
 | (b) tag and debug lines removed | 88,975,911 | 111,691 | 33,401,403 | 62.46% |
 | (c) message only | 40,551,755 | 197,430 | 11,212,254 | 72.35% |
 
-On the message text alone the reduction is 8.35 points higher than on the file as captured, because repeated text is a larger share of an application message than of the metadata around it. Removing the added field and 85,739 collector debug lines took 58.6% of the bytes off the input and moved the figure down by 1.54 points.
+The first is the file as Fluentd shipped it. With the tag field our forwarder adds and the collector's debug output removed, metadata kept, the reduction is 62.46%. The second removes two things at once and is a sensitivity check.
 
-- Every case decoded back to its own input byte for byte.
 - An OTLP record carries more than a bare message and much less than this envelope, so (b) and (c) bracket it at 62.46% to 72.35%.
 
 ## How it was measured
 
-The input is `otel-sample-200mb.log` from the public [otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1) release, captured through Fluentd from 36 containers over about ten hours. Input (b) removes the added field and the collector's debug exporter lines, and keeps the envelope because a log platform bills it. Input (c) is the message value of each record, one per line. All three ran through the same engine and configs, and each decode was compared with its own input. Compact bytes are the encoded events plus the template dictionary, with no gzip or ZSTD on either side.
+The stream covers 36 Kubernetes containers over about ten hours, the demo's own services (demo version 2.1.3) together with the cluster's infrastructure and system pods, every line carrying its Kubernetes metadata as Fluentd ships it. Input (b) removes the added field and the collector's debug exporter lines, and keeps the envelope because a log platform bills it. Input (c) is the message value of each record, one per line. All three ran through the same engine and configs, and each decode was compared with its own input.
 
 ## Limits
 
 - One capture from one small demo cluster at roughly 5 lines per second.
 - No OTLP capture was measured. The OTLP range above is inferred from two measured points.
 - Nothing here says what a particular platform bills.
-- Input (b) removes two unrelated things at once and serves as a sensitivity check. Neither removal is a property of the engine.
 
 ## Run it
 
@@ -39,3 +37,5 @@ python3 report.py --csv results/results.csv --out /tmp/results.md   # recompute
 The recompute needs only Python. The full rerun needs Docker and about 1.2 GB of free disk, and runs engine image `log10x/pipeline-10x:1.1.136`, pinned by digest in the script.
 
 The complete record is in [RECORD.md](RECORD.md): how each input is built, the full table with template counts, the image digest and the data's sha256.
+
+The same measurement, with more context: https://www.log10x.com/blog/log-compaction-measured/

pattern-identity/README.md

@@ -1,8 +1,8 @@
 # Pattern identity benchmark
 
-Does a log line get the same pattern name in two places, and keep it over time? Sampling and suppression rules key on that name. Two Drain3 instances, each mining one half of the same stream, gave a line whose shape occurs 2 to 10 times the same template 0.6022 of the time. Two 10x engine instances fed the same halves gave the same pattern name to 299,997 of 300,000 probe lines.
+A saved search or an alert points at a log pattern group by its stored text. Drain3, the open-source implementation of the Drain algorithm, rewrites that text as new lines arrive. Two copies fed random halves of one log set disagree on the stored text for many repeated shapes, and two 10x engine instances fed the same halves gave the same pattern name to 299,997 of 300,000 probe lines.
 
-A Drain3 template is the running intersection of every line in its cluster, so the template changes when a sibling line arrives. In one process, the same bytes replayed after two siblings got a new template, and Drain3 reported no cluster change. The 10x engine takes names from a compiled symbol library instead of mining them.
+Drain decides which words in a log line stay the same and which change by comparing each new line against the text it already stored for that group. Every spot where the two lines differ is blanked out. The 10x engine's compiler takes the source, or the binaries and container images the application ships as, and links the message text and names into a symbol library that is settled before the first log line arrives.
 
 ## Result
 
@@ -17,22 +17,19 @@ Share of probe lines both instances named the same, mean of 10 trials.
 | over 1,000 times | 28,152 | 1.0000 | 0.0646 | 1.0000 |
 | all probes | 30,000 | 0.9949 | 0.0609 | 1.0000 |
 
-A line seen once agrees because neither Drain3 instance can generalize from one example. The divergence starts when Drain3 generalizes, and the all-probes row hides it because shapes seen over 1,000 times are 94% of probes.
+A reference pass over the same stream resolved it into 1,481 distinct shapes, 921 of them seen once. On those same test log lines the two copies rarely pick the same group number.
 
-- Fed the same 60,000 lines in two orders, two Drain3 runs agreed on the template 0.9860 of the time and on `cluster_id`, an arrival counter, 0.0001.
-- The 3 engine probes that differed were indented lines joined to each half's last event, and the two halves ended differently.
-- A second engine identifier, the template hash, differed on 44 probes of one Grafana statement, because for those events it depends on what the instance read before.
+- Three consecutive Grafana provisioning log lines were fed to one copy (0.9.11) in order, then the first one again, and the text stored for that group changed between the first call and the fourth. On the fourth call the tool reports `change_type='none'`, and the group number held across all four calls too.
+- An alert check looks a log line up rather than feeding it in, and asked that way a log line seen once almost never has a group in one of the two copies.
 
 ## How it was measured
 
-The data is 197,430 Fluentd/Kubernetes lines from the OpenTelemetry demo ([otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1)), covering 10.3 hours. Each trial splits the stream into random disjoint halves, like two forwarders that never share state, and after its half each instance names the same 30,000 probe lines. Both ingest every probe, so their states converge, which on net favors Drain3. Drain3 0.9.11 runs its strongest tested configuration against engine 1.1.142, and the fork drain3-improved 0.10.0 gave identical numbers.
+The log set is the public [otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1) capture of the OpenTelemetry demo. The log set's 197,430 lines were shuffled, then split down the middle, one random half to each copy, so neither saw the other's. The benchmark calls that random split the friendliest division available to Drain. After its half each instance names the same 30,000 probe lines. Drain3 0.9.11 runs its strongest tested configuration against engine 1.1.142.
 
 ## Limits
 
-- The data is one small demo cluster, and the magnitudes belong to this stream.
-- A central miner, such as a SIEM that mines at query time, has no second instance to disagree, but the replay result still applies. Drain3 needs no build step and is the right default when the emitting code is out of reach.
-- The demo's source is in the engine's default symbol library. On invented words the engine returned an all-variable template named by a hash.
-- Of the engine's templates here, 82.8% carry deployment values such as image tags, and agreement across redeploys is unmeasured.
+- The price is running that compiler over the code that prints the logs, which watching traffic never asks for.
+- When no word in a line is in the library, 10x gives the line a pattern from its shape: `template_` followed by a hash of its punctuation and of where its values sit.
 
 ## Run it
 
@@ -45,3 +42,5 @@ PIB_QUICK=1 ./run.sh     # identity experiment at 60,000 lines and 3 trials
 The script needs Python 3.13 and downloads the sample. The Drain3 split takes about 19 minutes, and the engine split takes about 35 and needs Docker.
 
 [RECORD.md](RECORD.md) has every experiment and the tests that could prove log10x wrong, two of which fired against it.
+
+The same measurement, with more context: https://www.log10x.com/blog/drain-pattern-groups-change/