docs: add benchmarking blog posts and performance reference page

[SamBarker]

Summary

• Adds two blog posts about benchmarking Kroxylicious proxy overhead:
  • [May 1] "Does my proxy look big in this cluster?" — operator-focused: methodology, passthrough and encryption results, sizing guidance
  • [May 8] "Benchmarking a Kafka proxy: the engineering story" — engineer-focused: OMB harness, flamegraphs (interactive iframes), bugs found in own tooling, cluster incident
• Adds a /performance/ reference page summarising key numbers and linking to both posts
• Adds interactive async-profiler flamegraphs as self-contained HTML assets
• Updates overview.markdown with headline performance figures and a link to the reference page

Status

Draft — the posts are first drafts. Known open items:

• Per-connection scaling section in Post 1 needs the TODO placeholder replaced once 4-core sweep data is available and the scaling picture is better understood
• Post 2 has a stub section for 4-core validation results (pending sweep completion)
• Post 2 tone has not yet received the same voice treatment as Post 1

Test plan

• Run ./run.sh and verify site renders at http://127.0.0.1:4000/
• Check both blog posts render correctly including flamegraph iframes
• Check /performance/ page renders with correct tables
• Check cross-links between posts and to /performance/ work

🤖 Generated with Claude Code

[tombentley]

I wonder how much that's down to the decode predicate thing -- basically we know the filter chain, and what each filter in it wants to intercept, and I think we avoid doing the request/response decoding when we know nothing is interested. That was code that was in there from the beginning, but I don't actually know how relevant it is -- maybe some of the internal filters mean we're decoding requests and response always, in which case 1.4% is impressive. Or maybe we're acting more like a L4 proxy most of the time, in which case 1.4% is not quite as impressive.

[SamBarker]

Great question — this is actually a stronger story than the original prose suggested. The default infrastructure filters (BrokerAddressFilter, TopicNameCacheFilter, ApiVersionsIntersect) are doing genuine L7 work: metadata, FindCoordinator, and API version exchanges are fully decoded for address rewriting and version negotiation. But the high-volume produce/consume traffic hits the decode predicate and passes through without full deserialisation. So the proxy is selectively L7 — real protocol awareness where it needs it, L4-like passthrough on the hot path. The 1.4% is the cost of that design, and it validates it. Updating the prose to make this explicit.

[tombentley]

Did we ever figure out how to reuse the buffers more? I think that was a TODO at one point.

[SamBarker]

Correct — the TODO was never addressed. A BufferPool class existed at one point but was deleted as unused in early 2024. Cipher instances are still created fresh per operation. These remain genuine open optimisation opportunities.

[PaulRMellor]

I read through the first post. Reads well overall, and the tone is nice and approachable. I left a few suggestions, particularly in places where the AI-assisted wording feels a bit noticeable.

[PaulRMellor]

Suggested change

	All good benchmarking stories start with a hunch. Mine was that Kroxylicious is cheap to run — I'd stake my career on it, in fact — but it turns out that "trust me, I wrote it" is not a widely accepted unit of measurement. People want proof. Sensibly.
	All good benchmarking stories start with a hunch. I was confident Kroxylicious was cheap to run — but it turns out that "trust me, I wrote it" is not a widely accepted unit of measurement. People want proof. Sensibly.

[PaulRMellor]

Suggested change

There's a practical question underneath the hunch too. The most common thing operators ask us is some variation of: "How many cores does the proxy need?" Which is really just "is this thing going to slow down my Kafka?" in a polite engineering hat. We'd been giving the classic answer: "it depends on your workload and traffic patterns, so you'll need to test in your environment." Which is true. And also deeply unsatisfying for everyone involved, including us.

There's a practical question underneath the hunch too. The most common thing operators ask us is some variation of: "How many cores does the proxy need?" Which is really just another way of asking: "is this thing going to slow down my Kafka?" We'd been giving the classic answer: "it depends on your workload and traffic patterns, so you'll need to test in your environment." Which is true, but not especially useful.

[PaulRMellor]

Suggested change

	So we stopped saying "it depends", and got off the fence: we built something you can run **yourselves** on your own infrastructure with your own workload, and measured it. Here are some representative numbers from ours.
	So instead of saying "it depends", we built something measurable you can run **yourselves** on your own infrastructure with your own workload, and measured it. Here are some representative numbers from ours.

[PaulRMellor]

"got off the fence" might be a bit colloquial for non-native speakers

[PaulRMellor]

Suggested change

	No, we didn't run this on a laptop — it's a realistic deployment: a 6-node OpenShift cluster on Fyre, IBM's internal cloud platform — a controlled environment. Kroxylicious ran as a single proxy pod with a 1000m CPU limit.
	We ran the benchmarks on a realistic deployment rather than a local development machine: a 6-node OpenShift cluster on Fyre, IBM's internal cloud platform, providing a controlled test environment. Kroxylicious ran as a single proxy pod with a 1000m CPU limit.

[PaulRMellor]

Suggested change

	**The headline: ~0.2 ms additional average publish latency. Throughput is unaffected.**
	**The headline: ~0.2 ms additional average publish latency. Measured throughput was unaffected.**

[PaulRMellor]

Suggested change

Ok, so let's make the proxy smarter — make it do something people actually care about! [Record encryption](https://kroxylicious.io/documentation/0.20.0/html/record-encryption-guide) uses AES-256-GCM to encrypt each record passing through the proxy. AES-256-GCM is going to ask the CPU to work relatively hard on its own, but it's also going to push the proxy to understand each record it receives, unpack it, copy it, encrypt it, and re-pack it before sending it on to the broker. With all that work going on we expect some impact to latency and throughput. To answer our original question we need to identify two things: the latency when everything is going smoothly, and the reduction in throughput all this work causes. Monitoring latency once we go past the throughput inflection point isn't very helpful — it's dominated by the throughput limits and their erratic impacts on the latency of individual requests (a big hello to batching and buffering effects).

[Record encryption](https://kroxylicious.io/documentation/0.20.0/html/record-encryption-guide) is a more representative workload because the proxy actively processes each record. Record encryption uses AES-256-GCM to encrypt each record passing through the proxy. AES-256-GCM is going to ask the CPU to work relatively hard on its own, but it's also going to push the proxy to understand each record it receives, unpack it, copy it, encrypt it, and re-pack it before sending it on to the broker. With all that work going on we expect some impact to latency and throughput. To answer our original question we need to identify two things: the latency when everything is going smoothly, and the reduction in throughput all this work causes. Monitoring latency once we go past the throughput inflection point isn't very helpful — it's dominated by the throughput limits and their erratic impacts on the latency of individual requests (a big hello to batching and buffering effects).

[PaulRMellor]

Is understand the right word here: "push the proxy to understand each record it receives"?
Would parse suffice?

[PaulRMellor]

Suggested change

	A quick note on percentiles for anyone not steeped in performance benchmarking: p99 latency is the value that 99% of requests complete within — meaning 1 in 100 requests takes longer. Averages flatter; the p99 is what your slowest clients actually experience, and it's usually the number that matters.
	A quick note on percentiles for anyone not steeped in performance benchmarking: p99 latency is the value that 99% of requests complete within — meaning 1 in 100 requests takes longer. Average latency can hide tail latency effects; the p99 is what your slowest clients actually experience, and it's usually the number that matters.

[PaulRMellor]

Suggested change

	2. **Latency budget**: well below saturation, expect 0.2–3 ms additional average publish latency and 15–40 ms additional p99. The overhead scales with how hard you're pushing — give yourself headroom and you'll barely notice it.
	2. **Latency budget**: well below saturation, expect 0.2–3 ms additional average publish latency and 15–40 ms additional p99. The overhead scales with how hard you're pushing — providing sufficient headroom keeps the latency overhead relatively small.

[PaulRMellor]

we have mixture of msg/s and msg/sec in the post
we should be consistent

[PaulRMellor]

Suggested change

	These are real results from real hardware, but they don't tell a story for your workload. A few things worth knowing before you put these numbers in a slide deck:
	These are real results from real hardware, but they do not necessarily reflect your workload characteristics. A few things worth knowing before you put these numbers in a slide deck: