Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 19 additions & 15 deletions .ai/skills/optimize-comet-expression/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: optimize-comet-expression
description: Use when optimizing the performance of an existing native scalar expression in the datafusion-comet-spark-expr crate (native/spark-expr/) — casts, string/JSON/array/math kernels that run per-row or per-batch. Covers benchmarking, keeping output bit-identical, and the no-regression gate. Not for adding new expressions (use implement-comet-expression) or wiring upstream functions (use wire-datafusion-function).
description: Use when optimizing the performance of an existing native scalar expression in the datafusion-comet-spark-expr crate (native/spark-expr/) — casts, string/JSON/array/math kernels that run per-row or per-batch. Covers benchmarking, keeping output bit-identical, and the slower-shape gate. Not for adding new expressions (use implement-comet-expression) or wiring upstream functions (use wire-datafusion-function).
argument-hint: <expression-name>
---

Expand Down Expand Up @@ -60,27 +60,31 @@ being a no-op or a regression.**
**second independent sample** (`--baseline main` again) — real changes reproduce, noise does
not. A shape whose code path you did not touch cannot truly regress: if it flags, it is noise,
and the fix is another sample, not abandoning the change.
7. **Apply the no-regression gate** (below) before writing any PR.
7. **Apply the gate** (below) before writing any PR.
8. **Finish:** `make format`, build, `cargo clippy --all-targets --workspace -- -D warnings`.
PR title ``perf: optimize `$ARGUMENTS` (Nx faster)``, paste the criterion output for every
shape.
9. **Record the performance audit.** Add a dated `Performance (tuned ...)` line under the
expression's heading on the relevant page in
`docs/source/contributor-guide/expression-audits/` (naming the technique, speedup, PR, and
benchmark file) so contributors can see what has already been tuned. Check this page before
starting, too — it tells you whether the expression was already optimized.
`docs/source/contributor-guide/expression-audits/` (naming the technique, speedup, PR,
benchmark file, and any shape that got slower with its cost) so contributors can see what has
already been tuned. Check this page before starting, too — it tells you whether the expression was already optimized.

## The gate: what blocks a submission

- **A shape got meaningfully and reproducibly slower.** Do NOT submit. Even a 90%-faster-no-nulls
win does not justify a 30%-slower-dense-nulls loss. Either gate the fast path behind a per-batch
runtime check that picks the right path, or abandon the change. A Comet PR was closed for exactly
this. "Reproducibly" matters: confirm a flagged regression on a second sample first (step 6) —
a ~2% flag on a code path you did not touch is noise, not a blocker.
- **A shape that real workloads reach often got meaningfully and reproducibly slower.** Do NOT
submit. A 90%-faster-no-nulls win does not justify a 30%-slower-dense-nulls loss, because real
columns have dense nulls. Either gate the fast path behind a per-batch runtime check that picks the
right path, or abandon the change. A Comet PR was closed for exactly this. "Reproducibly" matters:
confirm a flagged regression on a second sample first (step 6) — a ~2% flag on a code path you did
not touch is noise, not a blocker.
- **A slower shape is not named in the PR.** A shape may be slower when it is rare or bounded and the
common case gains. The PR description must give its measured cost, how often real workloads reach
it and why, and the common-case gain.
- **No meaningful speedup on any shape.** There is nothing to submit. A change inside criterion's
noise threshold is not an improvement.
- **Output differs from `main`** on any input, including null placement or error behavior.
- **You only benchmarked one shape.** You have not shown the absence of a regression.
- **You only benchmarked one shape.** You have not shown which shapes got slower.

## Red flags — STOP

Expand All @@ -90,17 +94,17 @@ being a no-op or a regression.**
correctness failure. Diff the null buffer.
- "I'll apply the fallible conversion to every slot" → a garbage value under a null must not raise
an error Spark does not. Use `try_unary` or skip null slots.
- "One benchmark improved, ship it" → re-read the gate. One improvement plus one regression is not
a win.
- "One benchmark improved, ship it" → re-read the gate. A regression on a common shape is not a
win, and a regression on a rare shape still has to be named in the PR.

## Rationalization table

| Excuse | Reality |
| -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| "Measuring the baseline is overhead, I'll benchmark once at the end" | Without a `main` baseline you cannot report a change %, and you cannot tell a win from noise. Baseline first. |
| "Dense-null regression is an edge case" | Real columns have dense nulls. A regression there is a regression. Gate the path or drop it. |
| "Dense-null regression is an edge case" | Real columns have dense nulls. Gate the path or drop it. A rare shape needs evidence, not an assertion. |
| "Existing tests are enough proof, no need to check the null buffer" | Tests may not assert null placement. Confirm bit-identical output explicitly. |
| "This trades a bit of the null case for a big win elsewhere" | That is a trade-off, not an optimization. Only submit a strict improvement (or a correctly-gated per-batch path). |
| "This trades a bit of the null case for a big win elsewhere" | Acceptable only if the slower shape is rare or bounded, the PR names its cost and how often real workloads reach it, and the common case gains. Otherwise gate the path. |
| "I'll edit the code, then run the baseline" | `cargo bench` compiles from disk. A baseline built with your change already applied measures the optimized code and hides the real speedup. Baseline on unmodified source; stash the edit if needed. |
| "One shape flagged a 2% regression, abandon it" | Criterion compares separate runs; small flags on an untouched code path are cross-run noise. Take a second sample before deciding — real regressions reproduce. |

Expand Down
6 changes: 3 additions & 3 deletions .ai/skills/review-comet-expression-pr/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ bar, and the output format. This skill only covers what is specific to expressio
| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `docs/source/contributor-guide/adding_a_new_expression.md` | The serde contract, support levels, when to set the return type explicitly, shimming |
| `docs/source/contributor-guide/sql-file-tests.md` | The test framework expression PRs are expected to use, and every directive it supports |
| `docs/source/contributor-guide/optimizing_expressions.md` | The benchmark workflow and the no-regression rule, for PRs that change an existing kernel |
| `docs/source/contributor-guide/optimizing_expressions.md` | The benchmark workflow and the slower-shape rule, for PRs that change an existing kernel |
| `docs/source/contributor-guide/timezones.md` | The timestamp label rule, where the session timezone comes from, and how to test it, for datetime expressions, casts, and anything that returns a timestamp |

Hold the diff against these. If the PR does something a guide says to do differently, either the PR
Expand Down Expand Up @@ -218,8 +218,8 @@ carrying.
5. If a benchmark shows Comet slower than Spark, flag it. The PR needs to explain why that is
acceptable or include a plan to fix it.

For PRs optimizing an existing kernel, `optimizing_expressions.md` defines the no-regression rule
and the requirement to record a performance audit. Hold the PR to both.
For PRs optimizing an existing kernel, `optimizing_expressions.md` defines the rule for shapes that
get slower and the requirement to record a performance audit. Hold the PR to both.

## 6. Documentation

Expand Down
6 changes: 4 additions & 2 deletions .ai/skills/review-comet-pr/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,8 +334,10 @@ Two kinds of finding have to block the merge until the author addresses them: a
that the PR introduces, and a performance regression. A correctness problem means Comet can return a
different answer from Spark, including a result where Spark raises an error, or `Compatible()` over a
known divergence. Crashes, hangs, leaks, and lost or corrupted data count too. A performance
regression means an existing path gets slower or uses more memory. A tracking issue for a later fix
does not address either one.
regression means a path that real workloads reach often gets slower or uses more memory, or a PR makes
a shape slower without naming it. A shape that is rare or bounded may be slower when the PR gives its
measured cost, how often real workloads reach it and why, and the gain on the common case. A tracking
issue for a later fix does not address either one.

When any finding is one of these, the review is submitted as **Request changes**. `main` needs a
single approval to merge, and any committer's approval counts, including one given before this
Expand Down
18 changes: 18 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,24 @@ another maintainer's commits. If the lease check rejects the push, inspect and i
remote changes before retrying; do not bypass it with `--force`. See
[Submitting a Pull Request](docs/source/contributor-guide/development.md#submitting-a-pull-request).

## Code guidelines

- A PR merges as one squashed commit titled with the PR title, so it should do only what that title says. Put unrelated renames, reformatting, refactors, and drive-by fixes in a
separate PR.
- Default to no comment. Comment only what the code cannot say: invariants, non-obvious constraints, and
why a choice that looks wrong is right. Do not restate the code or describe how it changed.
- A panic that escapes a JNI or `extern "C"` entry point is undefined behavior. Return an error instead,
and see the `review-comet-ffi-pr` skill for the checklist.
- Config keys, their defaults, and user-visible behavior are public API. Renaming or removing a key
covered by the versioning policy needs a deprecated alias and a major release. Keys in the `testing`
category and keys marked `internal()` are exempt. See
[Renaming an Existing Config](docs/source/contributor-guide/config_conventions.md#renaming-an-existing-config).
In a review, name every new or changed config and public class.
- A performance PR shows which change produces the gain and covers both well-predicted input (constant
lengths, all valid) and poorly predicted input (random). A shape may get slower when the common case
gets faster and the PR names its cost and how often real workloads reach it. See
[Optimizing Expressions](docs/source/contributor-guide/optimizing_expressions.md).

## Checking a change against CI

A green pull request does not mean a change is safe to queue. The pull request tier runs the Comet
Expand Down
33 changes: 22 additions & 11 deletions docs/source/contributor-guide/optimizing_expressions.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,9 @@ improvement, there is nothing to submit.
3. **Apply the optimization**, preserving exact semantics (see [Correctness](#correctness-is-non-negotiable)).
4. **Run the existing unit tests** for the function. They must pass unchanged. Output must be
bit-identical to `main`, including null placement and error behavior.
5. **Re-run the benchmark** against the baseline. Confirm a meaningful speedup on at least one
shape **and no meaningful regression on any shape**.
5. **Re-run the benchmark** against the baseline. Confirm a meaningful speedup on the common
shapes, and account for every shape that got slower (see
[Slower shapes need a justification](#slower-shapes-need-a-justification)).
6. **Submit** with the criterion output pasted into the PR description.

## Writing a benchmark
Expand Down Expand Up @@ -97,6 +98,9 @@ looked at and loses on one you did not. Always include:
- **Short vs. long values.** Allocation savings shrink and copy costs grow as values get longer.
- **Valid vs. invalid inputs.** Error and fallback paths have different costs.
- **ASCII vs. non-ASCII** for string functions with an ASCII fast path.
- **Well-predicted vs. poorly predicted branches.** Constant lengths or all-valid input against
random lengths or values. Branchless code wins on the second and can lose on the first, and
branchy code does the reverse.

### Running benchmarks

Expand Down Expand Up @@ -166,13 +170,19 @@ Cross-cutting principles behind the table:
- **Watch capacity hints.** A `rows * rows` capacity hint (instead of `rows`) silently
over-allocates; fix these when you find them.

## The no-regression rule
## Slower shapes need a justification

**Do not submit an optimization if any benchmark shape is meaningfully slower**, even if another
shape is dramatically faster. A change that is 90% faster with no nulls but 30% slower with dense
nulls is a trade-off, not a win, and should either be gated behind a runtime check that picks the
right path per batch or not submitted at all. A Comet optimization PR was closed for exactly this
reason (a 30% dense-null regression alongside a 90% no-null win).
Optimize for the common case. A shape may get slower when the common case gets faster and the PR
justifies it. For each shape that is meaningfully and reproducibly slower, the PR description gives
the measured cost, how often real workloads reach that shape and why, and the gain on the common
case. For example, an error path that fails the query runs at most once per partition, so a slower
error path can be acceptable.

A shape that real workloads reach often is not an exception. A change that is 90% faster with no
nulls but 30% slower with dense nulls costs the many real columns that have dense nulls, so either
gate the fast path behind a runtime check that picks the right path per batch, or do not submit it.
A Comet optimization PR was closed for exactly this reason. A claim that a shape is rare needs
evidence about real workloads, not an assertion.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that a merged optimization can make a shape slower, could "Record the performance audit" below ask for that too? Its template records the technique, the speedup, the PR and the benchmark, so the audit entry, which is what the next person tuning the expression reads, would not show that a shape was traded away or why. A clause such as "and any shape that got slower, with its cost" in the template, plus the same in the PR conventions' Testing line, would keep the justification next to the history. Two existing entries in expression-audits/conversion_funcs.md already say "no regression on any shape", so the line has room for it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, added. The audit template's guidance in optimizing_expressions.md and step 9 of the optimize-comet-expression skill now ask for any shape that got slower with its cost, and the PR conventions' Testing line asks for the cost and justification.


## Record the performance audit

Expand All @@ -189,8 +199,8 @@ expression has no audit entry yet):
preallocated `BinaryBuilder`; up to 31% faster on long strings. Benchmark: `benches/unhex.rs`.
```

Name the technique, the measured speedup, the PR, and the benchmark file. Keep it to one line
per tuning pass so the history stays scannable.
Name the technique, the measured speedup, the PR, the benchmark file, and any shape that got
slower with its cost. Keep it to one line per tuning pass so the history stays scannable.

## PR conventions

Expand All @@ -199,7 +209,8 @@ Optimization PRs are small and follow a consistent shape:
- Title: ``perf: optimize `function_name` (Nx faster)`` with the headline speedup.
- Rationale: "Optimize existing expression."
- Changes: one or two sentences naming the specific technique and what per-row cost it removes.
- Testing: "Existing tests." plus the pasted criterion output for every shape.
- Testing: "Existing tests." plus the pasted criterion output for every shape, and any shape that
got slower with its cost and justification.
- Keep the diff focused on one expression. Do not bundle unrelated changes.

Follow the standard [Pre-PR checklist](development.md): `make format`, build, and
Expand Down