Skip to content

[Data] Replace the unpack error when a V1 read finds no files - #65256

Open
jackylee-ch wants to merge 2 commits into
ray-project:masterfrom
jackylee-ch:data/empty-read-clear-message
Open

[Data] Replace the unpack error when a V1 read finds no files#65256
jackylee-ch wants to merge 2 commits into
ray-project:masterfrom
jackylee-ch:data/empty-read-clear-message

Conversation

@jackylee-ch

@jackylee-ch jackylee-ch commented Aug 6, 2026

Copy link
Copy Markdown

Description

FileBasedDatasource.__init__ unpacked the listing with zip(*expand_paths(...)), so an empty listing raised an internal error on every V1 file reader. Listing drops names starting with _ or ., so a Spark output directory holding only _SUCCESS reaches it without any typo.

ray.data.read_csv("/path/spark_output/")
# was: ValueError: not enough values to unpack (expected 2, got 0)
# now: ValueError: no files found under ['/path/spark_output']. Check the path
#      and any configured `partition_filter` or `file_extensions` filters.

The message is V2's, reused rather than invented. Missing paths are dropped during listing, so the old ignore_missing_paths branch could not tell absent paths from empty ones; both now report the same reason, with a note appended when the flag is set.

Related issues

None.

Additional information

Tests: empty directory, directory holding only _SUCCESS/a hidden file, all paths missing with the flag set. test_file_based_datasource.py 35 passed/3 skipped (was 31/3); csv/json/text/numpy/binary/webdataset suites pass; pre-commit clean; reverting the change fails all three new tests. Bazel/C++ not run locally.

No open PR touches this file. AI assistance (Claude) was used; I reviewed every changed line and ran the tests above.

`FileBasedDatasource.__init__` unpacked `expand_paths` with
`zip(*...)`, which raises `ValueError: not enough values to unpack
(expected 2, got 0)` whenever listing yields nothing. Reuse the V2
message instead, and materialize the listing once so the existing
`ignore_missing_paths` guard below is reachable.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@jackylee-ch
jackylee-ch requested a review from a team as a code owner August 6, 2026 11:15

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors path expansion in FileBasedDatasource to raise a clear ValueError when no files are found, and adds unit tests to cover empty directories, excluded prefixes, and missing paths. The review feedback suggests refining the error message when ignore_missing_paths is enabled to avoid misleading users if the paths exist but are empty, along with updating the corresponding test assertions.

Comment on lines +193 to +197
if ignore_missing_paths:
raise ValueError(
"None of the provided paths exist. "
"The 'ignore_missing_paths' field is set to True."
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

When ignore_missing_paths is set to True, raising "None of the provided paths exist." can be misleading if the paths actually exist but are empty (or contain only excluded files, such as _SUCCESS or hidden files).

Consider using a more general and accurate error message that covers both cases (missing paths and empty directories).

Suggested change
if ignore_missing_paths:
raise ValueError(
"None of the provided paths exist. "
"The 'ignore_missing_paths' field is set to True."
)
if ignore_missing_paths:
raise ValueError(
f"No files found under {paths!r}. Check that the paths exist and "
"are not empty (note that 'ignore_missing_paths' is set to True)."
)



def test_all_paths_missing_with_ignore_missing_paths(ray_start_regular_shared):
with pytest.raises(ValueError, match="None of the provided paths exist"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Update the test assertion to match the improved error message.

Suggested change
with pytest.raises(ValueError, match="None of the provided paths exist"):
with pytest.raises(ValueError, match="No files found under"):

The `ignore_missing_paths` branch claimed "None of the provided paths
exist", but listing drops missing paths before we get here, so an empty
directory or one holding only excluded files hit the same branch and got
a wrong reason. Report the "no files found" message in both cases and
append a note about the skipping when the flag is set.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@jackylee-ch

jackylee-ch commented Aug 6, 2026

Copy link
Copy Markdown
Author

Confirmed, and the reason was wrong too — fixed in 961bc50.

Missing paths are dropped inside expand_paths, so an empty listing cannot tell absent paths from empty ones. read_csv("/tmp/empty_dir", ignore_missing_paths=True) reported "None of the provided paths exist" for a directory that does exist.

Both cases now report "no files found under ...", with a note appended when the flag is set. Test assertion updated. test_file_based_datasource.py 35 passed/3 skipped; csv/json/numpy suites pass.

@ray-gardener ray-gardener Bot added data Ray Data-related issues community-contribution Contributed by the community labels Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community data Ray Data-related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant