Skip to content

docs: document what the observability stack costs in RPS - #234

Merged
lesnik512 merged 4 commits into
mainfrom
docs/187-performance-page
Sep 20, 2026
Merged

lesnik512 merged 4 commits into
mainfrom
docs/187-performance-page

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Closes #187.

A Performance page under Introduction, sourced from benchmarks/README.md rather than copying it,
plus a re-measurement of the whole write-up and the harness change that makes its "tuned"
configuration something a reader can actually paste.

The page

docs/introduction/performance.md, in the nav, covering what the issue asked for: headline table,
per-instrument table, the four findings that surprised the author, the trade-off table, the
trivial-handler caveat, and an "if you are RPS-constrained, start here" section. Linked from the
Sentry, OpenTelemetry, Prometheus and Structlog sections of the configuration page.

The issue asked for a page listing knobs that did not exist at the time. Three of its four sibling
issues have since shipped, so the page documents opentelemetry_sampler,
opentelemetry_exclude_spans, sentry_auto_session_tracking and sentry_logging_breadcrumb_level
as real settings.

One copy of every number

The page pulls five blocks out of benchmarks/README.md through pymdownx.snippets, which was
already enabled. Markers live in HTML comments so the benchmarks file still renders cleanly on
GitHub, and they are stripped from the published page.

benchmarks/ is outside docs_dir, so this is a deliberate coupling. check_paths: true makes it
fail loudly: I pointed a snippet at a missing file and confirmed the build aborts with
Snippet at path 'benchmarks/MOVED.md' could not be found, then confirmed all five regions actually
render rather than silently emitting nothing.

Re-measured, one session, one machine

Every number in benchmarks/README.md is from today. The environment block needed no change: same
macOS 26.6.2, CPython 3.14.7, same library versions, so this is like-for-like.

before after
real server: bare / full / tuned 7978 / 2389 / 4187 7698 / 2313 / 4164
per-instrument: Prometheus / Sentry / OTel +17.5 / +58.5 / +120.1 +17.7 / +57.6 / +121.1
OTel, both knobs 2.16x 2.14x
Sentry integrations=[] +0.4 −0.1
tuning, on the real server +75% +80%

Every conclusion survived, which is the useful result: the ordering and the ratios are the durable
part, as §8 has always claimed.

Scope note. §8's documented commands do not reproduce the whole file. They miss §3's structlog
table, five of §4b's nine rows, §5's verify.py table, §2's concurrency sweep, the
traces_sample_rate=1.0 figure and §1's Sentry-alone cost. Refreshing only the documented subset
would have left the file mixing two sessions, which is exactly what its own closing caveat says
makes the columns incomparable. So the rest was run too.

The harness no longer monkeypatches

benchmarks/stack_scenarios.py reached "tuned" by patching around features that now exist:

_patch_exclude_send_receive_spans  """...lite-bootstrap never passes it (issue #185)."""
_patch_ratio_sampler               """...no sampler is configurable (issue #184)."""

Both docstrings were false, and _sentry_tuned hand-built what are now config fields. The scenarios
are now plain FastAPIConfig parts; _patch_*, PATCHES and setup() are gone. Verified the
result is identical to what the patches produced: ParentBased at 0.01, exclude_spans set,
auto_session_tracking False, breadcrumb and logs handlers both None with the event handler still
at ERROR.

This matters for the page: it tells readers to set five things, so the benchmark should measure
those five things and not an approximation of them.

Corrections review caught

  • The traces_sample_rate=1.0 line never named its baseline. The figure is measured, but against
    the Sentry suite's own 6241 RPS baseline, not §3's bare 7698. Both review axes read it as an
    arithmetic error, which settles it: the sentence now says which baseline it uses and that the two
    tables are not directly comparable.
  • The tuning table omitted http_methods_to_capture=(). The listed settings summed to ~132 µs
    against ~136 µs measured; the missing one is worth ~35 µs, the second-largest saving and the one
    with the sharpest trade-off. Added, and the column that merely restated its own link target is now
    "what you give up".
  • sentry_auto_session_tracking was documented at ~7 µs and re-measured at 7.7, and a
    pre-existing ~33 µs for exclude_spans re-measured at 34.0. Both reconciled.
  • §1 claimed "roughly half of that is recoverable". Tuning recovers 24 of the 70 points lost,
    about a third. Wrong before this branch; fixed here.

One finding I did not take: "most of it is Sentry's two log handlers" was called an overstatement at
9.6/33.7 = 28%, but that divides by §4c's whole per-request delta, charging the ASGI cost to every
log record. §4c minus §4b is 12.3 µs per record of handler cost against 9.6 recovered, so it is 78%.
The page now shows that subtraction instead of asserting "mostly".

Verification

ruff format, ruff check, ty, 307 tests, mkdocs build --strict, and the same lychee invocation
CI runs (95 links, 0 errors).

Known, not addressed here

  • configuration.md keeps three inline µs figures outside the snippet mechanism. All three are
    reconciled now, but they will drift on the next re-measure. Snippeting single scalars seemed worse
    than the drift.
  • verify.py's docstring advertises python verify.py a b c; its argparse takes one scenario.
  • The lowest-direct CI job runs only scripts/floor_smoke.py, not ty or pytest.

@lesnik512
lesnik512 merged commit 927dc99 into main Sep 20, 2026
13 checks passed
@lesnik512
lesnik512 deleted the docs/187-performance-page branch September 20, 2026 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document what the observability stack costs in RPS

1 participant