[SPARK-58081][INFRA] Document Python test base classes in AGENTS.md#57154
Open
zhengruifeng wants to merge 2 commits into
Open
[SPARK-58081][INFRA] Document Python test base classes in AGENTS.md#57154zhengruifeng wants to merge 2 commits into
zhengruifeng wants to merge 2 commits into
Conversation
### What changes were proposed in this pull request? Add a `## Python Test Base Classes` section to `AGENTS.md` (`CLAUDE.md` is a symlink to it), right after the existing `## Scala Test Base Classes` section and mirroring its structure: a short intro, an ASCII "adds capability on top of the previous" ladder, and a scope/base/notes table. It covers: - the classic ladder `unittest.TestCase` -> `PySparkBaseTestCase` -> `ReusedPySparkTestCase` -> `ReusedSQLTestCase` -> `PandasOnSparkTestCase`, including the fresh-vs-shared `SparkContext` distinction (`PySparkTestCase` vs `ReusedPySparkTestCase`); - a `### Spark Connect test bases` subsection for `PlanOnlyTestFixture`, `ReusedConnectTestCase`, and `ReusedMixedTestCase`, noting the `should_test_connect` auto-skip; - a `### Mixins and helpers` subsection for `SQLTestUtils`, `PySparkErrorTestUtils`, the standalone `assertDataFrameEqual` / `assertSchemaEqual`, and the off-ladder domain bases (`SparkSessionTestCase`, `MLlibTestCase`, `PySparkStreamingTestCase`). ### Why are the changes needed? `AGENTS.md` already guides contributors (and AI agents) to pick the lowest suitable Scala test base class, but has no equivalent for PySpark. The Python test bases under `python/pyspark/testing/` form a similar ladder that is easy to get wrong (e.g. subclassing a heavier base than needed, or missing the Connect-specific bases). Documenting them fills the adjacent gap. ### Does this PR introduce _any_ user-facing change? No. Documentation only. ### How was this patch tested? No code changes. Verified the described class hierarchy and capabilities against the sources under `python/pyspark/testing/` (`utils.py`, `sqlutils.py`, `connectutils.py`, `pandasutils.py`, `mlutils.py`, `mllibutils.py`, `streamingutils.py`), and spot-checked that only em-dashes (U+2014) were added as non-ASCII, consistent with the rest of the file. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8)
Sharpen the ReusedMixedTestCase row: state that its purpose is to directly compare classic vs Connect, exposing a classic `self.spark` and a Connect `self.connect` so a test can run the same operation on each and assert they agree. Generated-by: Claude Code (Opus 4.8)
Member
|
Maybe put it under |
Contributor
Author
@pan3793 '### PySpark Tests' is also in this file, so I add the test stuffs in it. @cloud-fan @HyukjinKwon wdyt? |
zhengruifeng
changed the title
Member
|
maybe let's move when it gets bigger later |
HyukjinKwon
approved these changes
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add a
## Python Test Base Classessection toAGENTS.md(CLAUDE.mdis a symlink to it), right after the existing## Scala Test Base Classessection and mirroring its structure: a short intro, an ASCII "adds capability on top of the previous" ladder, and a scope/base/notes table.It covers:
unittest.TestCase->PySparkBaseTestCase->ReusedPySparkTestCase->ReusedSQLTestCase->PandasOnSparkTestCase, including the fresh-vs-sharedSparkContextdistinction (PySparkTestCasevsReusedPySparkTestCase);### Spark Connect test basessubsection forPlanOnlyTestFixture,ReusedConnectTestCase, andReusedMixedTestCase, noting theshould_test_connectauto-skip;### Mixins and helperssubsection forSQLTestUtils,PySparkErrorTestUtils, the standaloneassertDataFrameEqual/assertSchemaEqual, and the off-ladder domain bases (SparkSessionTestCase,MLlibTestCase,PySparkStreamingTestCase).Why are the changes needed?
AGENTS.mdalready guides contributors (and AI agents) to pick the lowest suitable Scala test base class, but has no equivalent for PySpark. The Python test bases underpython/pyspark/testing/form a similar ladder that is easy to get wrong (e.g. subclassing a heavier base than needed, or missing the Connect-specific bases). Documenting them fills the adjacent gap.Does this PR introduce any user-facing change?
No. Documentation only.
How was this patch tested?
No code changes. Verified the described class hierarchy and capabilities against the sources under
python/pyspark/testing/(utils.py,sqlutils.py,connectutils.py,pandasutils.py,mlutils.py,mllibutils.py,streamingutils.py), and spot-checked that only em-dashes (U+2014) were added as non-ASCII, consistent with the rest of the file.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)