Skip to content

Add support for LISTOF function, array return, and EXPLODE operator in PyDough - #523

Open
knassre-bodo wants to merge 35 commits into
mainfrom
kian/explode
Open

Add support for LISTOF function, array return, and EXPLODE operator in PyDough#523
knassre-bodo wants to merge 35 commits into
mainfrom
kian/explode

Conversation

@knassre-bodo

@knassre-bodo knassre-bodo commented May 26, 2026

Copy link
Copy Markdown
Contributor

Linked ticket

Closes #521
Closes #522

Type of change

  • Bug fix
  • New feature
  • Refactor
  • Docs / config

What changed and why?

Adds the LISTOF function to PyDough to aggregate data into arrays, and the EXPLODE operator to flatten such arrays into multiple rows of table data. Also added support for DataFrame collections containing arrays, which become array literals in the SQL.

  • All of these features are not supported for ANSI or SQLITE because those dialects do not have those capabilities.
  • Array literals are not supported for Snowflake because the dialect does not have that capability.
  • EXPLODE is not yet supported currently for MySQL or Oracle due to additional complications with those dialects.

How I tested this?

  • Multiple new tests on TPCH data with the name prefix array_data_ to test the LISTOF and array literal features.
  • Multiple new tests on TPCH data with the name prefix explode_ to test different variations of the EXPLODE operator, both on arrays and strings. Variations include:
    • Regular collection.EXPLODE(data)
    • Inside of a child context collection.CALCULATE(x=COUNT(EXPLODE(data)))
    • Exploding array data vs string data
    • Different cardinality/distinctness information about the exploded data
    • Exploded data being used for grouping and for window operations
    • Nested explosions (e.g. splitting an already split string with collection.EXPLODE(...).EXPLODE(...))
    • Splitting strings on an empty delimiter
    • DataFrame tables with varying data-types, empty arrays, nulls
  • Error tests for incorrect uses of the EXPLODE operator (e.g. bad arguments)

Notes for reviewers

  • A few changes to how the database connections are handled to account for difference in how the connectors return arrays.
  • Several of the tests have dialect-specific skip marks. Some of these are truly unsupported features, and some are just for now unsupported.

@knassre-bodo
knassre-bodo requested review from a team, hadia206, john-sanchez31 and juankx-bodo and removed request for a team August 17, 2026 17:26
Comment on lines -796 to -798
def user_collection(self) -> PyDoughUserGeneratedCollection:
"""The wrapped user-generated collection."""
return self._parcel[0]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a thing that should be allowed with unqualified nodes, since any method or property name you add becomes something that can no longer be used as a PyDough term name.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add EXPLODE operator to PyDough Add LISTOF function to aggregate data into arrays

1 participant