diff --git a/.ai/skills/optimize-comet-expression/SKILL.md b/.ai/skills/optimize-comet-expression/SKILL.md index a7d8af89a56..69593e3b4e4 100644 --- a/.ai/skills/optimize-comet-expression/SKILL.md +++ b/.ai/skills/optimize-comet-expression/SKILL.md @@ -1,6 +1,6 @@ --- name: optimize-comet-expression -description: Use when optimizing the performance of an existing native scalar expression in the datafusion-comet-spark-expr crate (native/spark-expr/) — casts, string/JSON/array/math kernels that run per-row or per-batch. Covers benchmarking, keeping output bit-identical, and the no-regression gate. Not for adding new expressions (use implement-comet-expression) or wiring upstream functions (use wire-datafusion-function). +description: Use when optimizing the performance of an existing native scalar expression in the datafusion-comet-spark-expr crate (native/spark-expr/) — casts, string/JSON/array/math kernels that run per-row or per-batch. Covers benchmarking, keeping output bit-identical, and the slower-shape gate. Not for adding new expressions (use implement-comet-expression) or wiring upstream functions (use wire-datafusion-function). argument-hint: --- @@ -60,27 +60,31 @@ being a no-op or a regression.** **second independent sample** (`--baseline main` again) — real changes reproduce, noise does not. A shape whose code path you did not touch cannot truly regress: if it flags, it is noise, and the fix is another sample, not abandoning the change. -7. **Apply the no-regression gate** (below) before writing any PR. +7. **Apply the gate** (below) before writing any PR. 8. **Finish:** `make format`, build, `cargo clippy --all-targets --workspace -- -D warnings`. PR title ``perf: optimize `$ARGUMENTS` (Nx faster)``, paste the criterion output for every shape. 9. **Record the performance audit.** Add a dated `Performance (tuned ...)` line under the expression's heading on the relevant page in - `docs/source/contributor-guide/expression-audits/` (naming the technique, speedup, PR, and - benchmark file) so contributors can see what has already been tuned. Check this page before - starting, too — it tells you whether the expression was already optimized. + `docs/source/contributor-guide/expression-audits/` (naming the technique, speedup, PR, + benchmark file, and any shape that got slower with its cost) so contributors can see what has + already been tuned. Check this page before starting, too — it tells you whether the expression was already optimized. ## The gate: what blocks a submission -- **A shape got meaningfully and reproducibly slower.** Do NOT submit. Even a 90%-faster-no-nulls - win does not justify a 30%-slower-dense-nulls loss. Either gate the fast path behind a per-batch - runtime check that picks the right path, or abandon the change. A Comet PR was closed for exactly - this. "Reproducibly" matters: confirm a flagged regression on a second sample first (step 6) — - a ~2% flag on a code path you did not touch is noise, not a blocker. +- **A shape that real workloads reach often got meaningfully and reproducibly slower.** Do NOT + submit. A 90%-faster-no-nulls win does not justify a 30%-slower-dense-nulls loss, because real + columns have dense nulls. Either gate the fast path behind a per-batch runtime check that picks the + right path, or abandon the change. A Comet PR was closed for exactly this. "Reproducibly" matters: + confirm a flagged regression on a second sample first (step 6) — a ~2% flag on a code path you did + not touch is noise, not a blocker. +- **A slower shape is not named in the PR.** A shape may be slower when it is rare or bounded and the + common case gains. The PR description must give its measured cost, how often real workloads reach + it and why, and the common-case gain. - **No meaningful speedup on any shape.** There is nothing to submit. A change inside criterion's noise threshold is not an improvement. - **Output differs from `main`** on any input, including null placement or error behavior. -- **You only benchmarked one shape.** You have not shown the absence of a regression. +- **You only benchmarked one shape.** You have not shown which shapes got slower. ## Red flags — STOP @@ -90,17 +94,17 @@ being a no-op or a regression.** correctness failure. Diff the null buffer. - "I'll apply the fallible conversion to every slot" → a garbage value under a null must not raise an error Spark does not. Use `try_unary` or skip null slots. -- "One benchmark improved, ship it" → re-read the gate. One improvement plus one regression is not - a win. +- "One benchmark improved, ship it" → re-read the gate. A regression on a common shape is not a + win, and a regression on a rare shape still has to be named in the PR. ## Rationalization table | Excuse | Reality | | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | "Measuring the baseline is overhead, I'll benchmark once at the end" | Without a `main` baseline you cannot report a change %, and you cannot tell a win from noise. Baseline first. | -| "Dense-null regression is an edge case" | Real columns have dense nulls. A regression there is a regression. Gate the path or drop it. | +| "Dense-null regression is an edge case" | Real columns have dense nulls. Gate the path or drop it. A rare shape needs evidence, not an assertion. | | "Existing tests are enough proof, no need to check the null buffer" | Tests may not assert null placement. Confirm bit-identical output explicitly. | -| "This trades a bit of the null case for a big win elsewhere" | That is a trade-off, not an optimization. Only submit a strict improvement (or a correctly-gated per-batch path). | +| "This trades a bit of the null case for a big win elsewhere" | Acceptable only if the slower shape is rare or bounded, the PR names its cost and how often real workloads reach it, and the common case gains. Otherwise gate the path. | | "I'll edit the code, then run the baseline" | `cargo bench` compiles from disk. A baseline built with your change already applied measures the optimized code and hides the real speedup. Baseline on unmodified source; stash the edit if needed. | | "One shape flagged a 2% regression, abandon it" | Criterion compares separate runs; small flags on an untouched code path are cross-run noise. Take a second sample before deciding — real regressions reproduce. | diff --git a/.ai/skills/review-comet-expression-pr/SKILL.md b/.ai/skills/review-comet-expression-pr/SKILL.md index a5a22fe8145..02570c3c957 100644 --- a/.ai/skills/review-comet-expression-pr/SKILL.md +++ b/.ai/skills/review-comet-expression-pr/SKILL.md @@ -34,7 +34,7 @@ bar, and the output format. This skill only covers what is specific to expressio | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `docs/source/contributor-guide/adding_a_new_expression.md` | The serde contract, support levels, when to set the return type explicitly, shimming | | `docs/source/contributor-guide/sql-file-tests.md` | The test framework expression PRs are expected to use, and every directive it supports | -| `docs/source/contributor-guide/optimizing_expressions.md` | The benchmark workflow and the no-regression rule, for PRs that change an existing kernel | +| `docs/source/contributor-guide/optimizing_expressions.md` | The benchmark workflow and the slower-shape rule, for PRs that change an existing kernel | | `docs/source/contributor-guide/timezones.md` | The timestamp label rule, where the session timezone comes from, and how to test it, for datetime expressions, casts, and anything that returns a timestamp | Hold the diff against these. If the PR does something a guide says to do differently, either the PR @@ -218,8 +218,8 @@ carrying. 5. If a benchmark shows Comet slower than Spark, flag it. The PR needs to explain why that is acceptable or include a plan to fix it. -For PRs optimizing an existing kernel, `optimizing_expressions.md` defines the no-regression rule -and the requirement to record a performance audit. Hold the PR to both. +For PRs optimizing an existing kernel, `optimizing_expressions.md` defines the rule for shapes that +get slower and the requirement to record a performance audit. Hold the PR to both. ## 6. Documentation diff --git a/.ai/skills/review-comet-pr/SKILL.md b/.ai/skills/review-comet-pr/SKILL.md index 501ea2aaf1f..35be583a86d 100644 --- a/.ai/skills/review-comet-pr/SKILL.md +++ b/.ai/skills/review-comet-pr/SKILL.md @@ -334,8 +334,10 @@ Two kinds of finding have to block the merge until the author addresses them: a that the PR introduces, and a performance regression. A correctness problem means Comet can return a different answer from Spark, including a result where Spark raises an error, or `Compatible()` over a known divergence. Crashes, hangs, leaks, and lost or corrupted data count too. A performance -regression means an existing path gets slower or uses more memory. A tracking issue for a later fix -does not address either one. +regression means a path that real workloads reach often gets slower or uses more memory, or a PR makes +a shape slower without naming it. A shape that is rare or bounded may be slower when the PR gives its +measured cost, how often real workloads reach it and why, and the gain on the common case. A tracking +issue for a later fix does not address either one. When any finding is one of these, the review is submitted as **Request changes**. `main` needs a single approval to merge, and any committer's approval counts, including one given before this diff --git a/AGENTS.md b/AGENTS.md index 60af781d677..faf4cef297d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -46,6 +46,24 @@ another maintainer's commits. If the lease check rejects the push, inspect and i remote changes before retrying; do not bypass it with `--force`. See [Submitting a Pull Request](docs/source/contributor-guide/development.md#submitting-a-pull-request). +## Code guidelines + +- A PR merges as one squashed commit titled with the PR title, so it should do only what that title says. Put unrelated renames, reformatting, refactors, and drive-by fixes in a + separate PR. +- Default to no comment. Comment only what the code cannot say: invariants, non-obvious constraints, and + why a choice that looks wrong is right. Do not restate the code or describe how it changed. +- A panic that escapes a JNI or `extern "C"` entry point is undefined behavior. Return an error instead, + and see the `review-comet-ffi-pr` skill for the checklist. +- Config keys, their defaults, and user-visible behavior are public API. Renaming or removing a key + covered by the versioning policy needs a deprecated alias and a major release. Keys in the `testing` + category and keys marked `internal()` are exempt. See + [Renaming an Existing Config](docs/source/contributor-guide/config_conventions.md#renaming-an-existing-config). + In a review, name every new or changed config and public class. +- A performance PR shows which change produces the gain and covers both well-predicted input (constant + lengths, all valid) and poorly predicted input (random). A shape may get slower when the common case + gets faster and the PR names its cost and how often real workloads reach it. See + [Optimizing Expressions](docs/source/contributor-guide/optimizing_expressions.md). + ## Checking a change against CI A green pull request does not mean a change is safe to queue. The pull request tier runs the Comet diff --git a/docs/source/contributor-guide/optimizing_expressions.md b/docs/source/contributor-guide/optimizing_expressions.md index 4b11dd04874..3e9d321de5c 100644 --- a/docs/source/contributor-guide/optimizing_expressions.md +++ b/docs/source/contributor-guide/optimizing_expressions.md @@ -50,8 +50,9 @@ improvement, there is nothing to submit. 3. **Apply the optimization**, preserving exact semantics (see [Correctness](#correctness-is-non-negotiable)). 4. **Run the existing unit tests** for the function. They must pass unchanged. Output must be bit-identical to `main`, including null placement and error behavior. -5. **Re-run the benchmark** against the baseline. Confirm a meaningful speedup on at least one - shape **and no meaningful regression on any shape**. +5. **Re-run the benchmark** against the baseline. Confirm a meaningful speedup on the common + shapes, and account for every shape that got slower (see + [Slower shapes need a justification](#slower-shapes-need-a-justification)). 6. **Submit** with the criterion output pasted into the PR description. ## Writing a benchmark @@ -97,6 +98,9 @@ looked at and loses on one you did not. Always include: - **Short vs. long values.** Allocation savings shrink and copy costs grow as values get longer. - **Valid vs. invalid inputs.** Error and fallback paths have different costs. - **ASCII vs. non-ASCII** for string functions with an ASCII fast path. +- **Well-predicted vs. poorly predicted branches.** Constant lengths or all-valid input against + random lengths or values. Branchless code wins on the second and can lose on the first, and + branchy code does the reverse. ### Running benchmarks @@ -166,13 +170,19 @@ Cross-cutting principles behind the table: - **Watch capacity hints.** A `rows * rows` capacity hint (instead of `rows`) silently over-allocates; fix these when you find them. -## The no-regression rule +## Slower shapes need a justification -**Do not submit an optimization if any benchmark shape is meaningfully slower**, even if another -shape is dramatically faster. A change that is 90% faster with no nulls but 30% slower with dense -nulls is a trade-off, not a win, and should either be gated behind a runtime check that picks the -right path per batch or not submitted at all. A Comet optimization PR was closed for exactly this -reason (a 30% dense-null regression alongside a 90% no-null win). +Optimize for the common case. A shape may get slower when the common case gets faster and the PR +justifies it. For each shape that is meaningfully and reproducibly slower, the PR description gives +the measured cost, how often real workloads reach that shape and why, and the gain on the common +case. For example, an error path that fails the query runs at most once per partition, so a slower +error path can be acceptable. + +A shape that real workloads reach often is not an exception. A change that is 90% faster with no +nulls but 30% slower with dense nulls costs the many real columns that have dense nulls, so either +gate the fast path behind a runtime check that picks the right path per batch, or do not submit it. +A Comet optimization PR was closed for exactly this reason. A claim that a shape is rare needs +evidence about real workloads, not an assertion. ## Record the performance audit @@ -189,8 +199,8 @@ expression has no audit entry yet): preallocated `BinaryBuilder`; up to 31% faster on long strings. Benchmark: `benches/unhex.rs`. ``` -Name the technique, the measured speedup, the PR, and the benchmark file. Keep it to one line -per tuning pass so the history stays scannable. +Name the technique, the measured speedup, the PR, the benchmark file, and any shape that got +slower with its cost. Keep it to one line per tuning pass so the history stays scannable. ## PR conventions @@ -199,7 +209,8 @@ Optimization PRs are small and follow a consistent shape: - Title: ``perf: optimize `function_name` (Nx faster)`` with the headline speedup. - Rationale: "Optimize existing expression." - Changes: one or two sentences naming the specific technique and what per-row cost it removes. -- Testing: "Existing tests." plus the pasted criterion output for every shape. +- Testing: "Existing tests." plus the pasted criterion output for every shape, and any shape that + got slower with its cost and justification. - Keep the diff focused on one expression. Do not bundle unrelated changes. Follow the standard [Pre-PR checklist](development.md): `make format`, build, and