[Data] Replace the unpack error when a V1 read finds no files - #65256
[Data] Replace the unpack error when a V1 read finds no files#65256jackylee-ch wants to merge 2 commits into
Conversation
`FileBasedDatasource.__init__` unpacked `expand_paths` with `zip(*...)`, which raises `ValueError: not enough values to unpack (expected 2, got 0)` whenever listing yields nothing. Reuse the V2 message instead, and materialize the listing once so the existing `ignore_missing_paths` guard below is reachable. Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
There was a problem hiding this comment.
Code Review
This pull request refactors path expansion in FileBasedDatasource to raise a clear ValueError when no files are found, and adds unit tests to cover empty directories, excluded prefixes, and missing paths. The review feedback suggests refining the error message when ignore_missing_paths is enabled to avoid misleading users if the paths exist but are empty, along with updating the corresponding test assertions.
| if ignore_missing_paths: | ||
| raise ValueError( | ||
| "None of the provided paths exist. " | ||
| "The 'ignore_missing_paths' field is set to True." | ||
| ) |
There was a problem hiding this comment.
When ignore_missing_paths is set to True, raising "None of the provided paths exist." can be misleading if the paths actually exist but are empty (or contain only excluded files, such as _SUCCESS or hidden files).
Consider using a more general and accurate error message that covers both cases (missing paths and empty directories).
| if ignore_missing_paths: | |
| raise ValueError( | |
| "None of the provided paths exist. " | |
| "The 'ignore_missing_paths' field is set to True." | |
| ) | |
| if ignore_missing_paths: | |
| raise ValueError( | |
| f"No files found under {paths!r}. Check that the paths exist and " | |
| "are not empty (note that 'ignore_missing_paths' is set to True)." | |
| ) |
|
|
||
|
|
||
| def test_all_paths_missing_with_ignore_missing_paths(ray_start_regular_shared): | ||
| with pytest.raises(ValueError, match="None of the provided paths exist"): |
The `ignore_missing_paths` branch claimed "None of the provided paths exist", but listing drops missing paths before we get here, so an empty directory or one holding only excluded files hit the same branch and got a wrong reason. Report the "no files found" message in both cases and append a note about the skipping when the flag is set. Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
|
Confirmed, and the reason was wrong too — fixed in 961bc50. Missing paths are dropped inside Both cases now report "no files found under ...", with a note appended when the flag is set. Test assertion updated. |
Description
FileBasedDatasource.__init__unpacked the listing withzip(*expand_paths(...)), so an empty listing raised an internal error on every V1 file reader. Listing drops names starting with_or., so a Spark output directory holding only_SUCCESSreaches it without any typo.The message is V2's, reused rather than invented. Missing paths are dropped during listing, so the old
ignore_missing_pathsbranch could not tell absent paths from empty ones; both now report the same reason, with a note appended when the flag is set.Related issues
None.
Additional information
Tests: empty directory, directory holding only
_SUCCESS/a hidden file, all paths missing with the flag set.test_file_based_datasource.py35 passed/3 skipped (was 31/3); csv/json/text/numpy/binary/webdataset suites pass;pre-commitclean; reverting the change fails all three new tests. Bazel/C++ not run locally.No open PR touches this file. AI assistance (Claude) was used; I reviewed every changed line and ran the tests above.