Skip to content

fix(core): name a format override on a Parquet path - #1371

Merged
AndreaBozzo merged 3 commits into
AndreaBozzo:masterfrom
inverteum:fix/1356-parquet-format-override
Nov 19, 2026
Merged

AndreaBozzo merged 3 commits into
AndreaBozzo:masterfrom
inverteum:fix/1356-parquet-format-override

Conversation

@inverteum

@inverteum inverteum commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Reading a Parquet file as text no longer suggests giving it a .parquet extension when the path already has one. The read failed because a format was selected over that extension. The hint now says the format was overridden and names --format parquet on the command line, .format(FileFormat::Parquet) in Rust, and format="parquet" in Python. A path that is not already .parquet still gets the extension advice. When the format was selected explicitly, the hint names that format.

Related issue: Closes #1356

Acceptance criteria

  • When the format was selected explicitly, say so and name it: parquet_suggestion in crates/dataprof-core/src/binary_input.rs. an_explicit_text_format_on_a_parquet_path_names_the_override and test_an_explicit_text_format_on_a_parquet_path_names_the_override.
  • Keep the extension advice for a file whose extension is not .parquet: the same function. an_explicit_text_format_on_a_misnamed_parquet_file_keeps_the_extension_hint. A misnamed file with no explicit format still uses the previous sentence.
  • Name all three front ends in the shared hint: --format parquet on the command line, .format(FileFormat::Parquet) in Rust, and format="parquet" in Python. test_check_names_the_format_override_for_a_parquet_path.

Class of defect

A suggestion that ignores the path and the selected format. The fixed sentence was added in #901.

  • Covered: .parquet and .PARQUET; JSON and CSV as the selected format; a .csv path of Parquet bytes, with and without an explicit format.
  • Covered, routes: analyze_file, infer_schema, quick_row_count, analyze_structure, async profile_file, Python profile, and python -m dataprof.check. Direct analyze_csv_file and analyze_json_file on a .parquet path no longer ask for the extension. Those readers have no selected format to name, so the hint says the format was overridden without naming CSV or JSON.
  • Left out: in-memory byte routes have no path extension. A misnamed file read with the format left to its extension still uses the Rust and Python spellings; that case is not the override, and the existing tests require those spellings.

How it was verified

$ cargo test --test binary_input
test result: ok. 7 passed; 0 failed
$ cargo test --test binary_input --features async-streaming -- an_explicit_text_format_on_a_parquet_path_names_the_override_on_the_async_route
test result: ok. 1 passed; 0 failed
$ cargo fmt --all -- --check
$ cargo clippy -p dataprof-core --all-targets --features parquet -- -D warnings
$ cargo clippy -p dataprof-core --all-targets -- -D warnings
$ cargo clippy -p dataprof-partial --all-targets --features parquet -- -D warnings
$ cargo clippy -p dataprof --all-targets --tests -- -D warnings
$ uv run --no-sync ruff format python/tests/test_binary_input.py
1 file left unchanged
$ uv run --no-sync ruff check python/tests/test_binary_input.py
All checks passed!
$ uv run --no-sync pytest python/tests/test_binary_input.py -q
6 passed

Anything reviewers should know

  • The three front ends share one error string, so the hint names --format parquet on the command line, .format(FileFormat::Parquet) in Rust, and format="parquet" in Python. dataprof.check does not rewrite the hint.
  • python -m dataprof.check data.parquet --format json with no gate rule stops first on check() states no requirement and never opens the file. The check test passes --min-rows 1, and the binary-input error is what it prints. profile(path, format="json") reaches the hint with no gate rule.
  • reject_binary_input and DataProfilerError::binary_input keep their signatures. reject_binary_input_with_format is an extra entry point on the doc-hidden binary_input module.

Summary by CodeRabbit

  • Bug Fixes
    • Improved error messages when a file’s contents conflict with its extension or selected format. Messages now identify the selected format when relevant and offer clearer guidance, such as selecting Parquet for a Parquet file or correcting its extension.
    • Extension-specific guidance now works consistently with uppercase extensions and across synchronous and asynchronous profiling.

A Parquet file read as text was told to add a `.parquet` extension
even when the path already had one. That read failed because a
format was selected over the extension. The hint now says the
format was overridden and names `--format parquet`, and it keeps
the extension advice only for a path that is not already
`.parquet`.

Closes AndreaBozzo#1356
@github-actions github-actions Bot added area: rust Pull requests that update rust code area: python Python bindings and API area: testing Tests and quality assurance labels Nov 19, 2026
@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

Binary-input errors now use the file extension and any explicit format selection to tailor Parquet guidance. Profiling routes pass format overrides to validation. Rust and Python tests cover the updated suggestions.

Changes: Binary-input error guidance

Layer / File(s) Summary
Format-aware suggestions and error construction
crates/dataprof-core/src/binary_input.rs, crates/dataprof-core/src/errors.rs
Parquet suggestions now account for whether the path has a .parquet extension and whether a non-Parquet format was selected. A new public rejection function accepts an optional format; the existing function delegates without one.
Format propagation and regression coverage
crates/dataprof-partial/src/lib.rs, crates/dataprof/src/profiler.rs, tests/binary_input.rs, python/tests/test_binary_input.py
Profiling routes pass applicable format overrides to validation. Tests cover format-specific guidance for Parquet paths, including uppercase extensions, misnamed files, and CLI error output.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix · Severity of issue fixed: Low

Suggested labels: bug, area: rust, area: python, area: testing

Merge Risk: Low · up to be443

Parquet bytes in a file named .csv, with CSV selected explicitly on a partial route, get the generic extension advice rather than a hint naming the selected format. The primary .parquet override hint behaves as described. This is a small diagnostic-text gap and is low risk to merge.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the change: a format override on a Parquet path now gets a specific hint. It is short, uses the conventional fix(core) prefix, and matches the changeset, which adds format-aware binary…
Linked Issues check Passed Issue #1356 requires four coding outcomes. (1) Explicit format on a .parquet path: parquet_suggestion in crates/dataprof-core/src/binary_input.rs returns a message stating the format was overrid…
Out of Scope Changes check Passed All changed areas trace to #1356: binary_input.rs (hint text and format-aware helper), errors.rs (format-aware constructor, with the public binary_input signature unchanged), profiler.rs (sync…
Description check Passed The description follows the repository template closely. It has a clear 'What changed' summary, a linked issue (Closes #1356), and acceptance criteria that map each criterion to a file or test. It inc…
Full details: Title check

Explanation

The title names the change: a format override on a Parquet path now gets a specific hint. It is short, uses the conventional fix(core) prefix, and matches the changeset, which adds format-aware binary-input suggestions for .parquet paths with an explicit format.

Full details: Linked Issues check

Explanation

Issue #1356 requires four coding outcomes. (1) Explicit format on a .parquet path: parquet_suggestion in crates/dataprof-core/src/binary_input.rs returns a message stating the format was overridden and naming the selected format and --format parquet when explicit_format is set and the path ends in .parquet (case-insensitive). (2) Extension advice kept for non-.parquet paths: the (false, Some) and (false, None) arms keep the .parquet extension advice. (3) CLI spelling: the explicit-format hint names --format parquet; the no-format .parquet arm also uses --format parquet. (4) Explicit format plumbed to the error: reject_binary_input_with_format and binary_input_with_format in errors.rs pass the format through, and profiler.rs and partial/src/lib.rs call the format-aware path. Automated tests in tests/binary_input.rs and python/tests/test_binary_input.py cover these routes, per the PR description and the change summary. Limitation, not a failure: python -m dataprof.check data.parquet --format json with no gate rule stops on check() states no requirement before the file is opened, so the hint appears only when a gate rule is present or on the profile route. This earlier check is pre-existing and not a requirement of #1356. I did not run the test suite; the verdict relies on source inspection and the reported test results.

Full details: Out of Scope Changes check

Explanation

All changed areas trace to #1356: binary_input.rs (hint text and format-aware helper), errors.rs (format-aware constructor, with the public binary_input signature unchanged), profiler.rs (sync and async paths pass the configured format), dataprof-partial/src/lib.rs (passes the explicit format only when it differs from the extension format), and the tests in tests/binary_input.rs and python/tests/test_binary_input.py. The added reject_binary_input_with_format entry point is needed to thread the format through and is documented as such. I did not find unrelated behavior changes in the summaries. The partial change was checked only through the change summary, not the hunk itself.

Full details: Description check

Explanation

The description follows the repository template closely. It has a clear 'What changed' summary, a linked issue (Closes #1356), and acceptance criteria that map each criterion to a file or test. It includes a 'Class of defect' section that covers input routes, neighbouring values, and what was left out with reasons. It also gives a 'How it was verified' section with the commands run and their results, plus 'Anything reviewers should know' notes on breaking-change status, the shared error string, and the signatures that stay unchanged. The template's 'Feature flags' and 'Generated schema' items were removed as not applicable, which is acceptable. The description is complete and specific.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional comments: 1

crates/dataprof-partial/src/lib.rs-154-154: 1

154-154: Minor · Functional Correctness · Quick win

Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Preserve explicit format selection when it matches the extension.

If export.csv contains Parquet bytes, Profiler::new().format(FileFormat::Csv).infer_schema(...) passes Csv here. Line 154 compares it with the inferred Csv and passes None. The suggestion therefore omits the selected CSV format and returns Rust/Python advice instead of --format parquet.

quick_row_count and analyze_structure have the same failure. Preserve explicit-selection provenance from Profiler; do not infer it from format inequality. Keep inferred calls on None.

The PR objectives require explicit selections to be named with CLI recovery advice.

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 07578ffd-17c8-47af-afb2-0c544d433828
Commits

Reviewing files that changed from the base of the PR and between de354e4 and be44342.

Files selected for processing: 6

  • crates/dataprof-core/src/binary_input.rs
  • crates/dataprof-core/src/errors.rs
  • crates/dataprof-partial/src/lib.rs
  • crates/dataprof/src/profiler.rs
  • python/tests/test_binary_input.py
  • tests/binary_input.rs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

…eaBozzo#1356)

Selecting CSV for a Parquet file named export.csv was treated as no
selection on infer_schema, quick_row_count, and analyze_structure,
because those routes compared the format with the extension. Pass the
caller's selection through. An inferred read stays unnamed.

@AndreaBozzo AndreaBozzo left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @inverteum. The fix is in the right place: parquet_suggestion now gets the path and the selected format, and the tests cover every route the PR body lists, including the async one and both cases of the extension. Treating an explicit .format() that repeats the extension as a selection is a good call, and the PR body explains it. One point inline needs a change before merge, one is small.

let already_parquet = has_parquet_extension(path);
match (already_parquet, selected) {
(true, Some(format)) => format!(
"The format was overridden and set to {}; select `--format parquet`, or leave the \

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--format parquet is a CLI flag (needs a change). As the PR body says, this sentence now reaches Rust and Python callers too: Profiler::new().format(FileFormat::Json).analyze_file(..) and dp.profile(path, format="json") are told to select --format parquet, which neither API has. The issue asked for the CLI spelling in the CLI, not in place of the others. The core error cannot tell which front end called it, so the simplest fix is to name all three, as the inferred-format sentence already does:

The format was overridden and set to JSON. Select Parquet (`--format parquet` on the command line, `.format(FileFormat::Parquet)` in Rust, `format="parquet"` in Python), or leave the format out to use the `.parquet` extension.

The same applies to the other two arms (lines 195 and 199); line 195 is reached only from the Rust text readers, where --format means nothing. If you see a better way, for example dataprof.check rewriting the hint, pick it and say why in the PR description.

// `.format()` is a selection even when it repeats the extension.
// `infer_schema_with_format` for an inferred read would name that
// format (#1356).
match self.config.format_override.clone() {

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A doc comment is no longer true (small). resolve_format (line 608) says "Every path-taking entry point asks here", and these three methods now branch on format_override themselves. Either update that sentence, or keep calling resolve_format and pass along whether the format was selected, so one place still decides the format.

@AndreaBozzo

Copy link
Copy Markdown
Owner

@inverteum thank you for these four PRs. They are careful work, and the reviews are up on each.

One request for next time: please keep to one open PR at a time, and wait to be assigned on an issue before you start on it. Four PRs in a few hours means four reviews landing at once, and two of them (#1373 and #1374) already change the same tests. Once one of these merges, feel free to pick up the next.

The shared error told Rust and Python callers to select `--format
parquet`, a flag only the command line has. The hint now names that
flag, `.format(FileFormat::Parquet)`, and `format="parquet"`.
`resolve_format` no longer says every path-taking entry point asks
there.

@AndreaBozzo AndreaBozzo left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The hint now names all three spellings, the tests pin the full sentence on every route, and the resolve_format doc says which entry points branch on their own. Looks good to me.

@AndreaBozzo AndreaBozzo added this to the 0.15.0 milestone Nov 19, 2026
@AndreaBozzo
AndreaBozzo merged commit 28779c0 into AndreaBozzo:master Nov 19, 2026
48 checks passed
@github-project-automation github-project-automation Bot moved this to Done in dataprof Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: python Python bindings and API area: rust Pull requests that update rust code area: testing Tests and quality assurance

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Binary-input error suggests a .parquet extension the file already has when --format overrides it

2 participants