Repository navigation
Infer column dtypes with columns and schema_overrides - #5102
tonyyuyiding wants to merge 7 commits into
Conversation
|
This pull request has been imported. If you are a Meta employee, you can view this in D123393717. (Because this pull request was imported automatically, there will not be any future comments.) |
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
|
This pull request has been automatically marked as stale because it has not had recent activity for more than 2 weeks. If you are still working on this this pull request, please add a comment or push new commits to keep it active. Otherwise, please unassign yourself and allow someone else to take over. Thank you for your contributions! |
|
:) |
|
Sorry for the delay, I'll take a look this week |
rchen152
left a comment
There was a problem hiding this comment.
Thanks, and sorry again for the delayed review! I found two behavior regressions, which we should definitely fix before merging (see in-line comments).
AI review also turned up a few code quality suggestions. Feel free to incorporate any that you think will improve the code, but consider these optional:
-
The new case is a second copy of the read path.
infer_read_csv_schema (polars_specials.rs:1552-1608) is now a two-arm match: the old body re-indented in one arm and a new pipeline in the other. Both arms end with the same row-index and rename steps (lines 1581-1586 and 1600-1606). The two cases only differ in where the starting columns come from. So the match should produce columns and then hand off to one shared tail. Better still, the missing-schema case could build its columns from the columns= names with Unknown dtypes, apply the overrides, and switch the selection to All. Then the existing code runs unchanged. -
The override handling is split into two helpers that each skip the other's variant. apply_polars_csv_mapping_overrides returns early unless the overrides are a Mapping, and apply_polars_csv_sequence_overrides returns true unless they're a Sequence. The missing-schema arm calls both in a row, and each call does nothing for one of the variants. Worse, the given-schema path now drops a Mapping silently through the sequence helper's else { return true }. That is the unreachable-looking-state pattern AGENTS.md warns against. The PR also deletes the comment that stated this rule ("A complete schema changes only for positional dtype overrides."), which was on the removed CsvDtypeOverrides::parse. A single apply function that matches all three variants explicitly would fix this. Each path should then make its choice to ignore a dict visible, and the rule should be stated again in a comment.
-
The width-0 trick is redundant. At lines 1518-1521, a missing schema passes width 0 to polars_csv_selection so that index selections fail. The missing-schema arm rejects anything other than Names anyway (line 1589), so the same rule is enforced twice, and the first enforcement is indirect. Fixing (1) probably removes the need for it.
| }; | ||
| overrides.insert( | ||
| self.polars_column_name(key)?, | ||
| self.polars_dtype_from_expr(value)?, |
There was a problem hiding this comment.
This causes us to require schema_overrides to be resolvable even when schema is passed, which leads to this regression:
from typing import reveal_type
import polars as pl
def f(dt: pl.DataType) -> None:
# today: DataFrame[a: Int64]
# with PR: DataFrame
reveal_type(pl.read_csv("data.csv", schema={"a": pl.Int64}, schema_overrides={"a": dt}))
| let CsvColumnSelection::Names(names) = selection else { | ||
| return None; | ||
| }; | ||
| if matches!(overrides, CsvDtypeOverrides::Unchanged) { |
There was a problem hiding this comment.
This checks only whether overrides were passed, not whether they cover any of the listed columns, which causes pyrefly to treat cases that should be equivalent differently:
from typing import reveal_type
import polars as pl
reveal_type(pl.read_csv("data.csv", columns=["a"])) # DataFrame
reveal_type(pl.read_csv("data.csv", columns=["a"], schema_overrides={})) # DataFrame[a:Unknown]
reveal_type(pl.read_csv("data.csv", columns=["a"], schema_overrides={"z": pl.Int64})) # DataFrame[a:Unknown]
There was a problem hiding this comment.
Do you prefer DataFrame or DataFrame[a:Unknown] for these cases?
There was a problem hiding this comment.
I'd prefer DataFrame[a: Unknown] for all three. IMO an unknown dtype shouldn't affect known columns.
|
This pull request has been automatically marked as stale because it has been waiting for its author for more than 2 weeks. Please take the action it is waiting for, or close the PR if you are no longer working on it. Thank you for your contributions! |
This comment has been minimized.
This comment has been minimized.
|
Diff from mypy_primer, showing the effect of this PR on open source code: ============================================================
SUMMARY
============================================================
Total: +1 new errors, -0 fixed errors
By preset: +0/-0 (default), +0/-0 (strict)
Projects with changes (1):
python-sop: +1 -0
============================================================
FULL DIFF DETAILS
------------------------------------------------------------
python-sop (https://gitlab.com/dkg/python-sop)
+ ERROR sop/__init__.py:331:25-37: The type of attribute `autocomplete` is unknown because it comes from an error [unknown-from-error] |
Summary
Fixes #5036
Motivation
Originally, in
pl.read_csv, Pyrefly only infers the column dtypes ifschemais provided. However, in the case below, where we have bothcolumnsandschema_overrides, there is enough information to infer the precise schema.With this PR, Pyrefly will try to infer the schema as precisely as possible with
columnsandschema_overrides, even thoughschemais not given.Approach
The function
polars_csv_optionsis used to parse the options ofread_csvandscan_csv. In the old implementation, it assumedschemaalways exists andschema_overridesis either a sequence or absent, because the checker relied onschemato perform inference. To add the new feature, I have extended relevantenums and updated some old functions accordingly to handle more possible options. Then I added a new branch ininfer_read_csv_schemato handle the cases whereschemais missing, while the old branch was left unchanged.Test Plan
Tests have been added in
pyrefly/lib/test/polars/dataframe.rs.