From 7ba36744b0b67c9ff6e922095a0784eb87ba69fe Mon Sep 17 00:00:00 2001 From: Tal Weiss <73443663+talwgx@users.noreply.github.com> Date: Thu, 12 Nov 2026 23:19:17 -0500 Subject: [PATCH] README prose taken from the published posts, for comparison Co-Authored-By: Claude Opus 5.5 --- datadog-ceiling/README.md | 19 ++++++++-------- loghub-compaction/README.md | 31 +++++++++++++------------- otel-denominators/README.md | 12 +++++----- pattern-identity/README.md | 21 +++++++++--------- splunk-license/README.md | 44 ++++++++++++++++++------------------- 5 files changed, 62 insertions(+), 65 deletions(-) diff --git a/datadog-ceiling/README.md b/datadog-ceiling/README.md index a53d103..b748f3f 100644 --- a/datadog-ceiling/README.md +++ b/datadog-ceiling/README.md @@ -1,8 +1,8 @@ # Datadog ceiling -Does a monthly budget for Datadog Standard indexing stay where it was set when log volume grows two to three times? In this run it did. Log volume grew to 2.9 times the baseline, Standard indexing stayed at the budgeted rate, and every line still reached Datadog. +Datadog's own usage metric showed Standard indexing staying at a set number while ingested events grew 2.6 to 2.9 times on synthetic traffic, and Flex took the rest. A pattern routed to Flex stops paying the standard index rate. The lines stay queryable in Datadog's own Log Explorer, with no rehydration step. -The limit is set per log statement. Each hour, month-to-date usage is read from Datadog's own usage metric and a limit is written for every statement. Lines over a statement's limit go to a Flex index, so they stay searchable in Datadog. An error statement keeps 50 extra lines a window in Standard, so monitors on errors still fire. +Setting a monthly number of Standard-indexed events comes first. An hourly run reads Datadog's usage metric month to date and writes a cap per pattern and a total per container into the policy the sidecars read, and a container the run has not seen yet keeps the level the known containers share. A pattern's first five events of each window stay in the standard tier, and an error pattern over its cap keeps 50 more there in each window on each node, so a monitor on an error keeps firing. ## Result @@ -20,23 +20,20 @@ Events an hour on 2026-10-20 (UTC), as Datadog's usage metric counted them. The Four of the five budgeted hours came in under the allowed rate. The 06:00 hour was 393 events over it, inside the allowance for sample lines and error lines, which came to between 1,475 and 2,785 events an hour. -Three more checks from the same run: - -- A service that first appeared mid-run was limited from its first window. Of its 476 lines a window, 31 went to Standard until the 07:00 update gave it a limit of its own. - During an error storm, a log monitor on the storming statement went to Alert at 06:03:43 and back to OK at 06:22:43, after the storm ended. - 381,856 of 381,856 lines reached Datadog. ## How it was measured -The application is the OpenTelemetry demo, sending to a real Datadog org with Standard indexing at 15-day retention and one Flex index. Two baseline hours ran with no limits. Then the budget was switched on and load was raised. +The application is the OpenTelemetry demo, sending to a real Datadog org with Standard indexing at 15-day retention and one Flex index. The engine marks the events that should move with an attribute, and a Flex-only index whose filter matches that marker receives them, because Datadog indexes are first match wins and that index is ordered before the catch-all. Two baseline hours ran with no limits. Then the budget was switched on and load was raised. -Every number in the table comes from `datadog.estimated_usage.logs.ingested_events`, filtered to `datadog_is_excluded:false` and split by `datadog_index`. The rig's own counts agreed with that metric to within 16 events an hour. +Every number in the table comes from `datadog.estimated_usage.logs.ingested_events`, filtered to `datadog_is_excluded:false` and split by `datadog_index`. ## Limits -- One run of under eight hours, on a demo application, with a budget of $10.89 a month. The run shows the level staying put under growth and says nothing about money saved on a real bill. -- Logs only. Hosts, custom metrics and APM are outside this benchmark. -- Lines over the limit go to Flex, which has its own price and its own query compute. Datadog monitors do not run on Flex logs, and the org needs Flex Logs on its plan. +- One run of under eight hours, on a demo application, with a budget of $10.89 a month. +- Ingestion is billed as before, and Flex charges its own lower rate for storage plus compute for the queries that read it. +- Datadog runs no log monitors on Flex. Queries there run on Flex compute and are slower than the standard index. ## Run it @@ -50,3 +47,5 @@ python3 rig/analyze.py A full rerun needs Docker, a Datadog org with Flex Logs and about eight hours. The steps are in [RECORD.md](RECORD.md#reproduce). The complete record of the run is in [RECORD.md](RECORD.md): every check, how the budget was set, each hourly budget file and the rig. + +The same measurement, with more context: https://www.log10x.com/blog/cut-datadog-log-bill-per-pattern/ diff --git a/loghub-compaction/README.md b/loghub-compaction/README.md index 4bd1659..33bc370 100644 --- a/loghub-compaction/README.md +++ b/loghub-compaction/README.md @@ -1,40 +1,39 @@ # LogHub compaction -How much smaller than the original logs is the 10x engine's compact form, on the 14 full public LogHub datasets? Across all 14 the compact form came to 44.7% fewer bytes, weighted by size. Per dataset the reduction ran from 58.4% on Hadoop down to 4.7% on HealthApp. +A log platform meters ingest by the byte, and most bytes in a log line are repetitive text. 10x stores each line's repeated text once as a template, and ships each line as that template's hash plus the line's values. The 14 LogHub datasets measured here hold 52,276,250 events and 5,732,535,052 bytes of input. Weighted by input bytes, they reduce 44.7%, and the spread across them is wide. -The compact form is the encoded events plus the template dictionary needed to expand them back. Both are counted before any codec, so the figure stacks with whatever codec the storage layer applies after, such as gzip or ZSTD. +Reduction means the bytes of the compact form, the encoded events plus the template dictionary needed to expand them, against the bytes of the original file, with no gzip or ZSTD on either side. -## Result - -The table shows the four datasets over 100 MB, the highest and lowest of the other ten, and the total. +## What the runs measured | Dataset | Events | Input bytes | Compact bytes | Reduction | |---|---:|---:|---:|---:| | Hadoop | 186,697 | 48,595,595 | 20,212,363 | 58.4% | | Spark | 32,952,127 | 2,941,224,304 | 1,248,606,093 | 57.5% | -| BGL | 4,728,216 | 743,185,031 | 482,351,424 | 35.1% | +| Apache | 52,004 | 5,135,876 | 2,295,929 | 55.3% | +| OpenStack | 207,820 | 61,442,082 | 42,663,820 | 30.6% | | HDFS | 11,175,629 | 1,577,982,906 | 1,118,008,927 | 29.1% | | Android | 1,555,005 | 192,270,829 | 153,830,847 | 20% | | HealthApp | 204,648 | 23,529,930 | 22,432,794 | 4.7% | | All 14 | 52,276,250 | 5,732,535,052 | 3,167,463,894 | 44.7% | -The four over 100 MB together came to 44.9%. The spread is wide, and HealthApp saves little. All 14 rows, with template counts, are in [results/results.md](results/results.md). +The reduction tracks how much of each line is repeated text and how many lines share each template. HealthApp and Android, at 4.7% and 20%, are the floor. Both are mobile logs whose component names and tags ship as values, beside the timestamps and IDs, so little of each line is repeated text. -- Ten of the 14 match, byte for byte, the figures in the log10x post [Log compaction, measured](https://www.log10x.com/blog/log-compaction-measured/), and every reduction matches to the decimal place. -- Hadoop, Spark and OpenStack differ slightly. Each is many files read in directory order, and that order differs between the macOS file system of the post's run and the Linux container here. -- Linux differs because its template count changes from run to run. A two-digit day matches both the MMM d and the MMM dd date formats, and the format is chosen per event. Its reduction stays the same. +- HDFS and OpenStack sit at the low end of the middle because their lines are mostly values: block IDs, PIDs, and byte counts, with little fixed text between them, and the values ship on every event regardless. +- Hadoop, Spark, and Apache sit at the high end because they write more repeated message text around their values. +- All 14 rows, with template counts, are in [results/results.md](results/results.md). -## How it was measured +## Inputs and method -The inputs are the full LogHub datasets from Zenodo record 3227177, not the 2,000-line excerpts, each checked against the md5 Zenodo publishes. The released engine image ran in a Linux container with its default symbol library, one encode config and no per-dataset compile. Each dataset started with an empty template cache, so its whole dictionary is charged to it. The reduction is one minus the size of the two output files (`encoded.log` for the events, `templates.json` for the dictionary) over the input bytes the engine read. Both outputs are hashed so a rerun can be compared byte for byte. +Each run is one pass of the engine over one dataset: the original file goes in, and two files come out, the encoded events and the template dictionary. The LogHub datasets are the full files from the project's [Zenodo record](https://zenodo.org/records/3227177) rather than its 2,000-line benchmark excerpts. Every dataset went through the default symbol library that ships in the image. A library can be compiled from a specific codebase, and none of these runs did that. ## Limits -- Storage on disk after gzip or ZSTD is not measured. A codec compresses both the original and the compact form. +- A general-purpose compressor can beat these ratios on log text. A self-managed store also compresses what it holds, so the reduction does not carry to disk as-is. +- The two LogHub sets left out, Thunderbird and Windows, are the collection's largest, so the 44.7% weighted figure covers the 14 included sets and is not a LogHub-wide number. - Query cost and a comparison with another tool's output are not measured. -- The figures are for these 14 datasets. HDFS is the HDFS_1 set, and the Thunderbird and Windows datasets were not measured. -## Run it +## Reproducing a row ``` cd loghub-compaction @@ -46,3 +45,5 @@ python3 report.py --check # recompute every figure from results/ (no Docker) A full run needs Docker, Python 3 and about 5 GB of free disk at the peak (Spark). The script downloads the archives from Zenodo and pins the engine image log10x/pipeline-10x:1.1.133 by digest. The full record is in [RECORD.md](RECORD.md): every dataset's events, bytes, ratio and template count set against the post's figures, the full method and the file list. + +The same measurement, with more context: https://www.log10x.com/blog/log-compaction-measured/ diff --git a/otel-denominators/README.md b/otel-denominators/README.md index d9122a8..62a4fde 100644 --- a/otel-denominators/README.md +++ b/otel-denominators/README.md @@ -1,8 +1,8 @@ # OTel sample on three denominators -How much does a log reduction figure depend on what was in the input? On one public capture from a Kubernetes cluster running the OpenTelemetry demo, the 10x engine's figure is 64.00% on the file as captured, 62.46% with the debug output and an added field removed, and 72.35% on the application messages alone. Taking away the Kubernetes metadata around each message raised the figure. +The same stream was measured three ways, each decoded byte-identical. The stream is Kubernetes container output, captured through Fluentd from a cluster running the OpenTelemetry demo. As captured, metadata included, the reduction is 64.00%. On the log message text alone, metadata stripped, the reduction is 72.35%. -The capture wraps each container's stdout line in a Fluentd envelope of Kubernetes and Docker metadata. Two objections follow from that. One says the file is padded by a `tenx_tag` field the log10x forwarder adds and by the demo collector's debug exporter. The other says OTLP has no such envelope, so the figure should collapse without it. +About half the bytes are the demo collector's debug exporter, which the demo enables by default. Reduction means the bytes of the compact form, the encoded events plus the template dictionary needed to expand them, against the bytes of the original file, with no gzip or ZSTD on either side. ## Result @@ -12,21 +12,19 @@ The capture wraps each container's stdout line in a Fluentd envelope of Kubernet | (b) tag and debug lines removed | 88,975,911 | 111,691 | 33,401,403 | 62.46% | | (c) message only | 40,551,755 | 197,430 | 11,212,254 | 72.35% | -On the message text alone the reduction is 8.35 points higher than on the file as captured, because repeated text is a larger share of an application message than of the metadata around it. Removing the added field and 85,739 collector debug lines took 58.6% of the bytes off the input and moved the figure down by 1.54 points. +The first is the file as Fluentd shipped it. With the tag field our forwarder adds and the collector's debug output removed, metadata kept, the reduction is 62.46%. The second removes two things at once and is a sensitivity check. -- Every case decoded back to its own input byte for byte. - An OTLP record carries more than a bare message and much less than this envelope, so (b) and (c) bracket it at 62.46% to 72.35%. ## How it was measured -The input is `otel-sample-200mb.log` from the public [otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1) release, captured through Fluentd from 36 containers over about ten hours. Input (b) removes the added field and the collector's debug exporter lines, and keeps the envelope because a log platform bills it. Input (c) is the message value of each record, one per line. All three ran through the same engine and configs, and each decode was compared with its own input. Compact bytes are the encoded events plus the template dictionary, with no gzip or ZSTD on either side. +The stream covers 36 Kubernetes containers over about ten hours, the demo's own services (demo version 2.1.3) together with the cluster's infrastructure and system pods, every line carrying its Kubernetes metadata as Fluentd ships it. Input (b) removes the added field and the collector's debug exporter lines, and keeps the envelope because a log platform bills it. Input (c) is the message value of each record, one per line. All three ran through the same engine and configs, and each decode was compared with its own input. ## Limits - One capture from one small demo cluster at roughly 5 lines per second. - No OTLP capture was measured. The OTLP range above is inferred from two measured points. - Nothing here says what a particular platform bills. -- Input (b) removes two unrelated things at once and serves as a sensitivity check. Neither removal is a property of the engine. ## Run it @@ -39,3 +37,5 @@ python3 report.py --csv results/results.csv --out /tmp/results.md # recompute The recompute needs only Python. The full rerun needs Docker and about 1.2 GB of free disk, and runs engine image `log10x/pipeline-10x:1.1.136`, pinned by digest in the script. The complete record is in [RECORD.md](RECORD.md): how each input is built, the full table with template counts, the image digest and the data's sha256. + +The same measurement, with more context: https://www.log10x.com/blog/log-compaction-measured/ diff --git a/pattern-identity/README.md b/pattern-identity/README.md index 35f4cec..23c095a 100644 --- a/pattern-identity/README.md +++ b/pattern-identity/README.md @@ -1,8 +1,8 @@ # Pattern identity benchmark -Does a log line get the same pattern name in two places, and keep it over time? Sampling and suppression rules key on that name. Two Drain3 instances, each mining one half of the same stream, gave a line whose shape occurs 2 to 10 times the same template 0.6022 of the time. Two 10x engine instances fed the same halves gave the same pattern name to 299,997 of 300,000 probe lines. +A saved search or an alert points at a log pattern group by its stored text. Drain3, the open-source implementation of the Drain algorithm, rewrites that text as new lines arrive. Two copies fed random halves of one log set disagree on the stored text for many repeated shapes, and two 10x engine instances fed the same halves gave the same pattern name to 299,997 of 300,000 probe lines. -A Drain3 template is the running intersection of every line in its cluster, so the template changes when a sibling line arrives. In one process, the same bytes replayed after two siblings got a new template, and Drain3 reported no cluster change. The 10x engine takes names from a compiled symbol library instead of mining them. +Drain decides which words in a log line stay the same and which change by comparing each new line against the text it already stored for that group. Every spot where the two lines differ is blanked out. The 10x engine's compiler takes the source, or the binaries and container images the application ships as, and links the message text and names into a symbol library that is settled before the first log line arrives. ## Result @@ -17,22 +17,19 @@ Share of probe lines both instances named the same, mean of 10 trials. | over 1,000 times | 28,152 | 1.0000 | 0.0646 | 1.0000 | | all probes | 30,000 | 0.9949 | 0.0609 | 1.0000 | -A line seen once agrees because neither Drain3 instance can generalize from one example. The divergence starts when Drain3 generalizes, and the all-probes row hides it because shapes seen over 1,000 times are 94% of probes. +A reference pass over the same stream resolved it into 1,481 distinct shapes, 921 of them seen once. On those same test log lines the two copies rarely pick the same group number. -- Fed the same 60,000 lines in two orders, two Drain3 runs agreed on the template 0.9860 of the time and on `cluster_id`, an arrival counter, 0.0001. -- The 3 engine probes that differed were indented lines joined to each half's last event, and the two halves ended differently. -- A second engine identifier, the template hash, differed on 44 probes of one Grafana statement, because for those events it depends on what the instance read before. +- Three consecutive Grafana provisioning log lines were fed to one copy (0.9.11) in order, then the first one again, and the text stored for that group changed between the first call and the fourth. On the fourth call the tool reports `change_type='none'`, and the group number held across all four calls too. +- An alert check looks a log line up rather than feeding it in, and asked that way a log line seen once almost never has a group in one of the two copies. ## How it was measured -The data is 197,430 Fluentd/Kubernetes lines from the OpenTelemetry demo ([otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1)), covering 10.3 hours. Each trial splits the stream into random disjoint halves, like two forwarders that never share state, and after its half each instance names the same 30,000 probe lines. Both ingest every probe, so their states converge, which on net favors Drain3. Drain3 0.9.11 runs its strongest tested configuration against engine 1.1.142, and the fork drain3-improved 0.10.0 gave identical numbers. +The log set is the public [otel-sample-v1](https://github.com/log-10x/config/releases/tag/otel-sample-v1) capture of the OpenTelemetry demo. The log set's 197,430 lines were shuffled, then split down the middle, one random half to each copy, so neither saw the other's. The benchmark calls that random split the friendliest division available to Drain. After its half each instance names the same 30,000 probe lines. Drain3 0.9.11 runs its strongest tested configuration against engine 1.1.142. ## Limits -- The data is one small demo cluster, and the magnitudes belong to this stream. -- A central miner, such as a SIEM that mines at query time, has no second instance to disagree, but the replay result still applies. Drain3 needs no build step and is the right default when the emitting code is out of reach. -- The demo's source is in the engine's default symbol library. On invented words the engine returned an all-variable template named by a hash. -- Of the engine's templates here, 82.8% carry deployment values such as image tags, and agreement across redeploys is unmeasured. +- The price is running that compiler over the code that prints the logs, which watching traffic never asks for. +- When no word in a line is in the library, 10x gives the line a pattern from its shape: `template_` followed by a hash of its punctuation and of where its values sit. ## Run it @@ -45,3 +42,5 @@ PIB_QUICK=1 ./run.sh # identity experiment at 60,000 lines and 3 trials The script needs Python 3.13 and downloads the sample. The Drain3 split takes about 19 minutes, and the engine split takes about 35 and needs Docker. [RECORD.md](RECORD.md) has every experiment and the tests that could prove log10x wrong, two of which fired against it. + +The same measurement, with more context: https://www.log10x.com/blog/drain-pattern-groups-change/ diff --git a/splunk-license/README.md b/splunk-license/README.md index ddf565a..d25197c 100644 --- a/splunk-license/README.md +++ b/splunk-license/README.md @@ -1,38 +1,36 @@ # Splunk license-metered volume, with and without 10x in front -Splunk bills on what its license meter counts, which is neither the file size on disk nor what the forwarder sent. This test sends the same capture to Splunk twice, once as it is and once after the 10x engine has turned it into compact events, and reads the meter for each. The compact arm metered 63.91% fewer bytes, and every compact event read back as its original text. +An ingest-based Splunk Enterprise license is measured in daily indexing volume, the uncompressed data indexed each day. 10x stores the text each log statement repeats once, in a dictionary, and sends every event as its values. The 10x app for Splunk expands each event to its original text at search time. For one public Kubernetes log capture sent through a Universal Forwarder, Splunk's license meter counted 63.91% fewer bytes in compact form, dictionary included. -Compact events cannot be read without their template dictionary, so the dictionary goes to Splunk too and its metered bytes count against the compact arm. The 10x Splunk app expands each compact event back to its original text at search time. +## What the license meter counted -## Result +| Reading on Splunk's license meter | Value | +|---|---| +| Capture sent as is | 214,841,731 bytes | +| Compact events | 74,338,244 bytes | +| Template dictionary | 3,194,224 bytes | +| Compact form in total | 77,532,468 bytes | +| Reduction | 63.91% | +| Lowest container (`ad`) | 57.86% | +| Highest container (`opensearch`) | 69.27% | -License-metered bytes for the run on 2026-10-10, from Splunk's own license usage log. +The meter counts the text of each event. The capture is 215,039,161 bytes on disk and 214,841,731 on the meter, and the difference is its 197,430 line breaks. Each compact event was metered at the length of its own text, with event time set from the line. -| Arm | What Splunk ingested | Metered bytes | GB/day | -|---|---|---:|---:| -| Baseline | the capture as it is | 214,841,731 | 0.200087 | -| Compact | compact events | 74,338,244 | 0.069233 | -| Compact | template dictionary | 3,194,224 | 0.002975 | -| Compact | total | 77,532,468 | 0.072208 | +- The template dictionary is counted like any other data, and its 3,194,224 bytes are inside the compact total. +- The two container rows cover events only, because the dictionary is not split by container, and containers with fewer than 100 events are not ranked. +- Through the app's own macro, 157,228 of 157,228 compact events expanded byte-identical, with the search head set to UTC and again to Etc/GMT-2. -The compact total is 63.91% below the baseline. Measured as files on disk, the same two arms give 63.87%. The gap is line breaks, which the meter does not count and which make up more of the smaller compact arm (0.21% against 0.09%). +## How the meter was read -- All 157,228 compact events came back byte-identical to the text that went in, with the search head in UTC and in the instance's own timezone. -- Per Kubernetes container with at least 100 events, the reduction ran from 57.86% for ad to 69.27% for opensearch. These figures leave the dictionary out, which makes each one slightly better than in a deployment. - -## How it was measured - -The data is a 215,039,161 byte capture of 197,430 lines from 36 Kubernetes containers in a small demo cluster, each line in a Fluentd envelope. Both arms went through the same Universal Forwarder to one Splunk Enterprise 10.4.3 instance on the download trial license, into separate indexes on the same license day. On this version the daily `type=RolloverSummary` line carries only the day's total, so the split by index comes from the per-minute `type=Usage` records, as the Monitoring Console does. The per-index bytes add up to the rollup's daily total of 292,374,199 bytes exactly. +[The capture](https://github.com/log-10x/config/releases/tag/otel-sample-v2) is 215,039,161 bytes and 197,430 lines of output from 36 Kubernetes containers in an OpenTelemetry demo cluster. The engine joins the lines of a multi-line event into one event, so the 197,430 lines are 157,228 events, and the run stops if any line is left unassigned. Both forms went through the same Universal Forwarder into the same Splunk Enterprise 10.4.3 instance on the same license day. The 10x engine writes the compact form of the capture as two files, the compact events and the template dictionary, and the forwarder sends each to an index of its own. Splunk Enterprise runs in a container on the download trial license, which allows 500 MB a day, and both arms fit in one license day. Splunk's license usage log reports usage per index, and the per-index entries add up to the day's rollup total to the byte: 292,374,199. ## Limits -- One capture from one demo cluster on one license day. A production estate has its own data, sourcetypes and forwarder topology. +- The 63.91% belongs to this capture. - The figure is ingest only. Indexed storage, search speed and retention were not measured. -- Splunk Cloud, which meters by a different path, was not tested. - -## Run it +- The meter reading here is from Splunk Enterprise, and Splunk Cloud, workload pricing included, is outside this benchmark. -The license figures can be recomputed from the saved rollup lines without Splunk or Docker: +## Rerunning the benchmark ```sh # from the repository root @@ -43,6 +41,6 @@ python3 splunk-license/report.py --dir splunk-license --data splunk-license/data splunk-license/run.sh ``` -A full run needs Docker, about 10 GB of free disk and roughly two hours, most of it waiting for the license day to end. The run touches no existing Splunk. +The repository's CI recomputes the license figures from the kept rollup lines, so an edited results table fails. The run needs Docker and about 10 GB of disk and takes about two hours, most of it waiting for the license day to end. The full record, with every Splunk setting, the encode settings the app needs and all the results tables, is in [RECORD.md](RECORD.md).