Skip to content

[Data] Fix WebDataset docs and encoder typing - #65278

Open
viiccwen wants to merge 1 commit into
ray-project:masterfrom
viiccwen:data-webdataset-contract
Open

[Data] Fix WebDataset docs and encoder typing#65278
viiccwen wants to merge 1 commit into
ray-project:masterfrom
viiccwen:data-webdataset-contract

Conversation

@viiccwen

@viiccwen viiccwen commented Aug 7, 2026

Copy link
Copy Markdown

Description

Correct the public Dataset.write_webdataset() contract so it matches the tar-based WebDataset implementation.

  • Replace copied TFRecord descriptions with WebDataset sample and tar archive semantics.
  • Correct output path and default extension documentation.
  • Define reusable encoder aliases for a default format specifier, a sample-transform callable, or a list of those values.
  • Use the aliases in both Dataset.write_webdataset() and WebDatasetDatasink.
  • Add focused tests proving callable and chained-list encoders are accepted and produce the expected archive entries.

This changes documentation and static typing only; existing supported runtime inputs remain valid.

Related issues

Closes #65277.

Additional information

API documentation changes (no real behavior change).

Signed-off-by: viiccwen <vicwen@apache.org>
@viiccwen
viiccwen requested a review from a team as a code owner August 7, 2026 02:38

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refines the type annotations and documentation for the WebDataset writer, introducing type aliases for encoder configurations and updating the docstrings to correctly reference WebDataset tar archives instead of TFRecord files. It also adds tests to verify default, callable, and chained encoders. The review feedback suggests refining the WebDatasetEncoderConfig type alias to use List[WebDatasetEncoderSpec] instead of List[Any], and using this alias consistently in Dataset.write_webdataset instead of an inlined type union.


WebDatasetEncoder = Callable[[Dict[str, Any]], Dict[str, Any]]
WebDatasetEncoderSpec = Union[bool, str, WebDatasetEncoder]
WebDatasetEncoderConfig = Optional[Union[WebDatasetEncoderSpec, List[Any]]]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The type alias WebDatasetEncoderConfig currently uses List[Any] for chained encoders. Since WebDatasetEncoderSpec is already defined to represent valid encoder types, we can make this type annotation much more precise and self-documenting by using List[WebDatasetEncoderSpec] instead of List[Any].

Suggested change
WebDatasetEncoderConfig = Optional[Union[WebDatasetEncoderSpec, List[Any]]]
WebDatasetEncoderConfig = Optional[Union[WebDatasetEncoderSpec, List[WebDatasetEncoderSpec]]]

Comment on lines +5323 to +5330
encoder: Optional[
Union[
bool,
str,
Callable[[Dict[str, Any]], Dict[str, Any]],
List[Any],
]
] = True,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The PR description states that one of the goals is to "Use the aliases in both Dataset.write_webdataset() and WebDatasetDatasink." However, the implementation in Dataset.write_webdataset still uses an inlined (and less precise) type union.\n\nUsing the newly defined WebDatasetEncoderConfig alias here keeps the public API signature clean and consistent with WebDatasetDatasink.\n\nNote: Please also update the imports at the top of python/ray/data/dataset.py to import WebDatasetEncoderConfig from ray.data._internal.datasource.webdataset_datasink.

        encoder: "WebDatasetEncoderConfig" = True,

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's due to the style for Ray-data.

@400Ping 400Ping self-assigned this Aug 7, 2026
@ray-gardener ray-gardener Bot added docs An issue or change related to documentation data Ray Data-related issues community-contribution Contributed by the community labels Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community data Ray Data-related issues docs An issue or change related to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Data] Correct write_webdataset documentation and encoder type contract

2 participants