You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(core): name a format override on a Parquet path - #1371
Reading a Parquet file as text no longer suggests giving it a .parquet extension when the path already has one. The read failed because a format was selected over that extension. The hint now says the format was overridden and names --format parquet on the command line, .format(FileFormat::Parquet) in Rust, and format="parquet" in Python. A path that is not already .parquet still gets the extension advice. When the format was selected explicitly, the hint names that format.
When the format was selected explicitly, say so and name it: parquet_suggestion in crates/dataprof-core/src/binary_input.rs. an_explicit_text_format_on_a_parquet_path_names_the_override and test_an_explicit_text_format_on_a_parquet_path_names_the_override.
Keep the extension advice for a file whose extension is not .parquet: the same function. an_explicit_text_format_on_a_misnamed_parquet_file_keeps_the_extension_hint. A misnamed file with no explicit format still uses the previous sentence.
Name all three front ends in the shared hint: --format parquet on the command line, .format(FileFormat::Parquet) in Rust, and format="parquet" in Python. test_check_names_the_format_override_for_a_parquet_path.
Class of defect
A suggestion that ignores the path and the selected format. The fixed sentence was added in #901.
Covered: .parquet and .PARQUET; JSON and CSV as the selected format; a .csv path of Parquet bytes, with and without an explicit format.
Covered, routes: analyze_file, infer_schema, quick_row_count, analyze_structure, async profile_file, Python profile, and python -m dataprof.check. Direct analyze_csv_file and analyze_json_file on a .parquet path no longer ask for the extension. Those readers have no selected format to name, so the hint says the format was overridden without naming CSV or JSON.
Left out: in-memory byte routes have no path extension. A misnamed file read with the format left to its extension still uses the Rust and Python spellings; that case is not the override, and the existing tests require those spellings.
The three front ends share one error string, so the hint names --format parquet on the command line, .format(FileFormat::Parquet) in Rust, and format="parquet" in Python. dataprof.check does not rewrite the hint.
python -m dataprof.check data.parquet --format json with no gate rule stops first on check() states no requirement and never opens the file. The check test passes --min-rows 1, and the binary-input error is what it prints. profile(path, format="json") reaches the hint with no gate rule.
reject_binary_input and DataProfilerError::binary_input keep their signatures. reject_binary_input_with_format is an extra entry point on the doc-hidden binary_input module.
Summary by CodeRabbit
Bug Fixes
Improved error messages when a file’s contents conflict with its extension or selected format. Messages now identify the selected format when relevant and offer clearer guidance, such as selecting Parquet for a Parquet file or correcting its extension.
Extension-specific guidance now works consistently with uppercase extensions and across synchronous and asynchronous profiling.
A Parquet file read as text was told to add a `.parquet` extension
even when the path already had one. That read failed because a
format was selected over the extension. The hint now says the
format was overridden and names `--format parquet`, and it keeps
the extension advice only for a path that is not already
`.parquet`.
ClosesAndreaBozzo#1356
Binary-input errors now use the file extension and any explicit format selection to tailor Parquet guidance. Profiling routes pass format overrides to validation. Rust and Python tests cover the updated suggestions.
Changes: Binary-input error guidance
Layer / File(s)
Summary
Format-aware suggestions and error construction crates/dataprof-core/src/binary_input.rs, crates/dataprof-core/src/errors.rs
Parquet suggestions now account for whether the path has a .parquet extension and whether a non-Parquet format was selected. A new public rejection function accepts an optional format; the existing function delegates without one.
Format propagation and regression coverage crates/dataprof-partial/src/lib.rs, crates/dataprof/src/profiler.rs, tests/binary_input.rs, python/tests/test_binary_input.py
Profiling routes pass applicable format overrides to validation. Tests cover format-specific guidance for Parquet paths, including uppercase extensions, misnamed files, and CLI error output.
Parquet bytes in a file named .csv, with CSV selected explicitly on a partial route, get the generic extension advice rather than a hint naming the selected format. The primary .parquet override hint behaves as described. This is a small diagnostic-text gap and is low risk to merge.
Pre-merge checks: 4
Check name
Status
Explanation
Title check
The title names the change: a format override on a Parquet path now gets a specific hint. It is short, uses the conventional fix(core) prefix, and matches the changeset, which adds format-aware binary…
Linked Issues check
Issue #1356 requires four coding outcomes. (1) Explicit format on a .parquet path: parquet_suggestion in crates/dataprof-core/src/binary_input.rs returns a message stating the format was overrid…
Out of Scope Changes check
All changed areas trace to #1356: binary_input.rs (hint text and format-aware helper), errors.rs (format-aware constructor, with the public binary_input signature unchanged), profiler.rs (sync…
Description check
The description follows the repository template closely. It has a clear 'What changed' summary, a linked issue (Closes #1356), and acceptance criteria that map each criterion to a file or test. It inc…
Full details: Title check
Explanation
The title names the change: a format override on a Parquet path now gets a specific hint. It is short, uses the conventional fix(core) prefix, and matches the changeset, which adds format-aware binary-input suggestions for .parquet paths with an explicit format.
Full details: Linked Issues check
Explanation
Issue #1356 requires four coding outcomes. (1) Explicit format on a .parquet path: parquet_suggestion in crates/dataprof-core/src/binary_input.rs returns a message stating the format was overridden and naming the selected format and --format parquet when explicit_format is set and the path ends in .parquet (case-insensitive). (2) Extension advice kept for non-.parquet paths: the (false, Some) and (false, None) arms keep the .parquet extension advice. (3) CLI spelling: the explicit-format hint names --format parquet; the no-format .parquet arm also uses --format parquet. (4) Explicit format plumbed to the error: reject_binary_input_with_format and binary_input_with_format in errors.rs pass the format through, and profiler.rs and partial/src/lib.rs call the format-aware path. Automated tests in tests/binary_input.rs and python/tests/test_binary_input.py cover these routes, per the PR description and the change summary. Limitation, not a failure: python -m dataprof.check data.parquet --format json with no gate rule stops on check() states no requirement before the file is opened, so the hint appears only when a gate rule is present or on the profile route. This earlier check is pre-existing and not a requirement of #1356. I did not run the test suite; the verdict relies on source inspection and the reported test results.
Full details: Out of Scope Changes check
Explanation
All changed areas trace to #1356: binary_input.rs (hint text and format-aware helper), errors.rs (format-aware constructor, with the public binary_input signature unchanged), profiler.rs (sync and async paths pass the configured format), dataprof-partial/src/lib.rs (passes the explicit format only when it differs from the extension format), and the tests in tests/binary_input.rs and python/tests/test_binary_input.py. The added reject_binary_input_with_format entry point is needed to thread the format through and is documented as such. I did not find unrelated behavior changes in the summaries. The partial change was checked only through the change summary, not the hunk itself.
Full details: Description check
Explanation
The description follows the repository template closely. It has a clear 'What changed' summary, a linked issue (Closes #1356), and acceptance criteria that map each criterion to a file or test. It includes a 'Class of defect' section that covers input routes, neighbouring values, and what was left out with reasons. It also gives a 'How it was verified' section with the commands run and their results, plus 'Anything reviewers should know' notes on breaking-change status, the shared error string, and the signatures that stay unchanged. The template's 'Feature flags' and 'Generated schema' items were removed as not applicable, which is acceptable. The description is complete and specific.
Finishing Touches
Generate unit tests (beta)
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
The reason will be displayed to describe this comment to others. Learn more.
Additional comments: 1
crates/dataprof-partial/src/lib.rs-154-154: 1
154-154: Minor · Functional Correctness · Quick win
Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.
Preserve explicit format selection when it matches the extension.
If export.csv contains Parquet bytes, Profiler::new().format(FileFormat::Csv).infer_schema(...) passes Csv here. Line 154 compares it with the inferred Csv and passes None. The suggestion therefore omits the selected CSV format and returns Rust/Python advice instead of --format parquet.
quick_row_count and analyze_structure have the same failure. Preserve explicit-selection provenance from Profiler; do not infer it from format inequality. Keep inferred calls on None.
The PR objectives require explicit selections to be named with CLI recovery advice.
Review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: 07578ffd-17c8-47af-afb2-0c544d433828
Commits
Reviewing files that changed from the base of the PR and between de354e4 and be44342.
Files selected for processing: 6
crates/dataprof-core/src/binary_input.rs
crates/dataprof-core/src/errors.rs
crates/dataprof-partial/src/lib.rs
crates/dataprof/src/profiler.rs
python/tests/test_binary_input.py
tests/binary_input.rs
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
…eaBozzo#1356)
Selecting CSV for a Parquet file named export.csv was treated as no
selection on infer_schema, quick_row_count, and analyze_structure,
because those routes compared the format with the extension. Pass the
caller's selection through. An inferred read stays unnamed.
The reason will be displayed to describe this comment to others. Learn more.
Thanks @inverteum. The fix is in the right place: parquet_suggestion now gets the path and the selected format, and the tests cover every route the PR body lists, including the async one and both cases of the extension. Treating an explicit .format() that repeats the extension as a selection is a good call, and the PR body explains it. One point inline needs a change before merge, one is small.
The reason will be displayed to describe this comment to others. Learn more.
--format parquet is a CLI flag (needs a change). As the PR body says, this sentence now reaches Rust and Python callers too: Profiler::new().format(FileFormat::Json).analyze_file(..) and dp.profile(path, format="json") are told to select --format parquet, which neither API has. The issue asked for the CLI spelling in the CLI, not in place of the others. The core error cannot tell which front end called it, so the simplest fix is to name all three, as the inferred-format sentence already does:
The format was overridden and set to JSON. Select Parquet (`--format parquet` on the command line, `.format(FileFormat::Parquet)` in Rust, `format="parquet"` in Python), or leave the format out to use the `.parquet` extension.
The same applies to the other two arms (lines 195 and 199); line 195 is reached only from the Rust text readers, where --format means nothing. If you see a better way, for example dataprof.check rewriting the hint, pick it and say why in the PR description.
The reason will be displayed to describe this comment to others. Learn more.
A doc comment is no longer true (small).resolve_format (line 608) says "Every path-taking entry point asks here", and these three methods now branch on format_override themselves. Either update that sentence, or keep calling resolve_format and pass along whether the format was selected, so one place still decides the format.
@inverteum thank you for these four PRs. They are careful work, and the reviews are up on each.
One request for next time: please keep to one open PR at a time, and wait to be assigned on an issue before you start on it. Four PRs in a few hours means four reviews landing at once, and two of them (#1373 and #1374) already change the same tests. Once one of these merges, feel free to pick up the next.
The shared error told Rust and Python callers to select `--format
parquet`, a flag only the command line has. The hint now names that
flag, `.format(FileFormat::Parquet)`, and `format="parquet"`.
`resolve_format` no longer says every path-taking entry point asks
there.
The reason will be displayed to describe this comment to others. Learn more.
Thanks. The hint now names all three spellings, the tests pin the full sentence on every route, and the resolve_format doc says which entry points branch on their own. Looks good to me.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Reading a Parquet file as text no longer suggests giving it a
.parquetextension when the path already has one. The read failed because a format was selected over that extension. The hint now says the format was overridden and names--format parqueton the command line,.format(FileFormat::Parquet)in Rust, andformat="parquet"in Python. A path that is not already.parquetstill gets the extension advice. When the format was selected explicitly, the hint names that format.Related issue: Closes #1356
Acceptance criteria
parquet_suggestionincrates/dataprof-core/src/binary_input.rs.an_explicit_text_format_on_a_parquet_path_names_the_overrideandtest_an_explicit_text_format_on_a_parquet_path_names_the_override..parquet: the same function.an_explicit_text_format_on_a_misnamed_parquet_file_keeps_the_extension_hint. A misnamed file with no explicit format still uses the previous sentence.--format parqueton the command line,.format(FileFormat::Parquet)in Rust, andformat="parquet"in Python.test_check_names_the_format_override_for_a_parquet_path.Class of defect
A suggestion that ignores the path and the selected format. The fixed sentence was added in #901.
.parquetand.PARQUET; JSON and CSV as the selected format; a.csvpath of Parquet bytes, with and without an explicit format.analyze_file,infer_schema,quick_row_count,analyze_structure, asyncprofile_file, Pythonprofile, andpython -m dataprof.check. Directanalyze_csv_fileandanalyze_json_fileon a.parquetpath no longer ask for the extension. Those readers have no selected format to name, so the hint says the format was overridden without naming CSV or JSON.How it was verified
Anything reviewers should know
--format parqueton the command line,.format(FileFormat::Parquet)in Rust, andformat="parquet"in Python.dataprof.checkdoes not rewrite the hint.python -m dataprof.check data.parquet --format jsonwith no gate rule stops first oncheck() states no requirementand never opens the file. The check test passes--min-rows 1, and the binary-input error is what it prints.profile(path, format="json")reaches the hint with no gate rule.reject_binary_inputandDataProfilerError::binary_inputkeep their signatures.reject_binary_input_with_formatis an extra entry point on the doc-hiddenbinary_inputmodule.Summary by CodeRabbit