Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 8 additions & 9 deletions .ai/skills/audit-comet-expression/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: audit-comet-expression
description: Audit an existing Comet expression for correctness and test coverage. Studies the Spark implementation across versions 3.4.3, 3.5.8, 4.0.1, and 4.1.1, reviews the Comet and DataFusion implementations, identifies missing test coverage, and offers to implement additional tests.
description: Audit an existing Comet expression for correctness and test coverage. Studies the Spark implementation across versions 3.5.8, 4.0.1, and 4.1.1, reviews the Comet and DataFusion implementations, identifies missing test coverage, and offers to implement additional tests.
argument-hint: <expression-name>
---

Expand Down Expand Up @@ -29,7 +29,7 @@ Audit the Comet implementation of the `$ARGUMENTS` expression for correctness an

This audit covers:

1. Spark implementation across versions 3.4.3, 3.5.8, 4.0.1, and 4.1.1
1. Spark implementation across versions 3.5.8, 4.0.1, and 4.1.1
2. Comet Scala serde implementation
3. Comet Rust / DataFusion implementation
4. Existing test coverage (Comet SQL Tests and Comet Scala Tests)
Expand All @@ -43,7 +43,7 @@ Clone specific Spark version tags (use shallow clones to avoid polluting the wor

```bash
set -eu -o pipefail
for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do
for tag in v3.5.8 v4.0.1 v4.1.1; do
dir="/tmp/spark-${tag}"
if [ ! -d "$dir" ]; then
git clone --depth 1 --branch "$tag" https://github.com/apache/spark.git "$dir"
Expand All @@ -56,7 +56,7 @@ done
Search the Catalyst SQL expressions source:

```bash
for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do
for tag in v3.5.8 v4.0.1 v4.1.1; do
dir="/tmp/spark-${tag}"
echo "=== $tag ==="
find "$dir/sql/catalyst/src/main/scala" -name "*.scala" | \
Expand All @@ -67,7 +67,7 @@ done
If the expression is not found in catalyst, also check core:

```bash
for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do
for tag in v3.5.8 v4.0.1 v4.1.1; do
dir="/tmp/spark-${tag}"
echo "=== $tag ==="
find "$dir/sql" -name "*.scala" | \
Expand All @@ -90,7 +90,6 @@ For each Spark version, read the expression file and note:

Produce a concise diff summary of what changed between:

- 3.4.3 → 3.5.8
- 3.5.8 → 4.0.1
- 4.0.1 → 4.1.1

Expand All @@ -107,7 +106,7 @@ Pay attention to:
## Step 2: Locate the Spark Tests

```bash
for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do
for tag in v3.5.8 v4.0.1 v4.1.1; do
dir="/tmp/spark-${tag}"
echo "=== $tag ==="
find "$dir/sql" -name "*.scala" -path "*/test/*" | \
Expand Down Expand Up @@ -732,7 +731,7 @@ used for the expression in `docs/source/user-guide/latest/expressions.md`.
other pages.
- Add (or update) a `## <function_name>` section, keeping sections alphabetically ordered.
- Under that heading, add one bullet per Spark version checked, each including:
- Spark version (e.g. 3.4.3, 3.5.8, 4.0.1, 4.1.1)
- Spark version (e.g. 3.5.8, 4.0.1, 4.1.1)
- Today's date
- A brief note for any version-specific finding (behavioral difference, known
incompatibility); omit the note if nothing notable.
Expand All @@ -744,7 +743,7 @@ used for the expression in `docs/source/user-guide/latest/expressions.md`.
Present the audit as:

1. **Expression Summary** - Brief description of what `$ARGUMENTS` does, its input/output types, and null behavior
2. **Spark Version Differences** - Summary of any behavioral or API differences across Spark 3.4.3, 3.5.8, 4.0.1, and 4.1.1
2. **Spark Version Differences** - Summary of any behavioral or API differences across Spark 3.5.8, 4.0.1, and 4.1.1
3. **Comet Implementation Notes** - Summary of how Comet implements this expression and any concerns
4. **Coverage Gap Analysis** - The gap table from Step 5, plus implementation gaps
5. **Recommendations** - Prioritized list from Step 6
Expand Down
2 changes: 1 addition & 1 deletion .ai/skills/implement-comet-expression/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ make

### 4. Run the audit skill

Once the initial implementation passes its smoke test, run the `audit-comet-expression` skill on `$ARGUMENTS`. It compares the implementation and tests against Spark 3.4.3, 3.5.8, and 4.0.1 and produces a prioritized list of gaps.
Once the initial implementation passes its smoke test, run the `audit-comet-expression` skill on `$ARGUMENTS`. It compares the implementation and tests against Spark 3.5.8, 4.0.1, and 4.1.1 and produces a prioritized list of gaps.

### 5. Implement audit-recommended tests and iterate

Expand Down
4 changes: 2 additions & 2 deletions .ai/skills/review-comet-pr/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,8 +183,8 @@ such as `wrap_second_millisecond_overflow`, never the version, such as `spark_42
fork that backports a Spark fix then changes the one Scala check that sets the flag and leaves the
protobuf and Rust alone. Flag a version-named field, argument, or function, and comments that say
"Spark 4.2" for behavior later releases inherit or that leave out the Spark JIRA.
`Cast.is_spark4_plus` and `get_json_object_spark34` predate the rule and #6891 tracks renaming them,
so flag them only when a PR adds behavior behind them.
`Cast.is_spark4_plus` predates the rule and #6891 tracks renaming it,
so flag it only when a PR adds behavior behind it.

When Spark changed the behavior in a patch release, such as SPARK-55969 or SPARK-54918, a check on
the minor version is wrong for every earlier patch. CI builds only the newest patch of each line, so
Expand Down
2 changes: 1 addition & 1 deletion .ai/skills/wire-datafusion-function/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ make
cd native && cargo clippy --all-targets --workspace -- -D warnings
```

Then run `audit-comet-expression` on `$ARGUMENTS` to compare against Spark 3.4.3 / 3.5.8 / 4.0.1 and surface coverage gaps; iterate on tests.
Then run `audit-comet-expression` on `$ARGUMENTS` to compare against Spark 3.5.8 / 4.0.1 / 4.1.1 and surface coverage gaps; iterate on tests.

The user generally runs tests themselves. If asked for a smoke test:

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pr_build_linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -582,7 +582,7 @@ jobs:
- name: "exec"
value: |
org.apache.comet.exec.CometAggregateSuite
org.apache.comet.exec.CometExec3_4PlusSuite
org.apache.comet.exec.CometExecCompatibilitySuite
org.apache.comet.exec.CometExecSuite
org.apache.comet.exec.CometMergeRowsSuite
org.apache.comet.exec.CometTopKSuite
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pr_build_macos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@ jobs:
- name: "exec"
value: |
org.apache.comet.exec.CometAggregateSuite
org.apache.comet.exec.CometExec3_4PlusSuite
org.apache.comet.exec.CometExecCompatibilitySuite
org.apache.comet.exec.CometExecSuite
org.apache.comet.exec.CometMergeRowsSuite
org.apache.comet.exec.CometTopKSuite
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ docker push localhost:32000/apache/datafusion-comet-tpcbench:latest
export SPARK_MASTER=k8s://https://127.0.0.1:16443
export COMET_DOCKER_IMAGE=localhost:32000/apache/datafusion-comet-tpcbench:latest
# Location of Comet JAR within the Docker image
export COMET_JAR=/opt/spark/jars/comet-spark-spark3.4_2.12-0.5.0-SNAPSHOT.jar
export COMET_JAR=/opt/spark/jars/comet-spark-spark3.5_2.12-0.5.0-SNAPSHOT.jar

$SPARK_HOME/bin/spark-submit \
--master $SPARK_MASTER \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -435,10 +435,9 @@ object DeltaScanSupport {
return providerReason
}

// Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
// Spark 3.4, exec enabled, existence default values, the Variant read confs, and a proto
// representation for every serialized data and partition type); tags its own fallback
// reasons.
// Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, exec enabled,
// existence default values, the Variant read confs, and a proto representation for every
// serialized data and partition type); tags its own fallback reasons.
if (!CometNativeScan.isSupported(scanExec)) {
return Some("Core native scan gates rejected the scan (see reasons above)")
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ import org.apache.spark.sql.types.{ByteType, LongType, StringType, StructField,
import org.apache.spark.sql.util.QueryExecutionListener

import org.apache.comet.CometConf
import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus}
import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus
import org.apache.comet.ExtendedExplainInfo
import org.apache.comet.serde.OperatorOuterClass
import org.apache.comet.serde.operator.CometNativeScan
Expand Down Expand Up @@ -1751,7 +1751,6 @@ class CometDeltaNativeScanSuite extends CometDeltaTestBase {
test(
"DPP stays executable when transition reversion restores the delta scan: " +
s"AQE=$adaptive") {
assume(isSpark35Plus, "Comet AQE DPP query-stage optimizer rules require Spark 3.5+")
withTempPath { factDir =>
withTempPath { dimDir =>
val factPath = factDir.getAbsolutePath
Expand Down
Loading
Loading