Skip to content

Infer column dtypes with columns and schema_overrides - #5102

Open
tonyyuyiding wants to merge 7 commits into
pyrefly-org:mainfrom
tonyyuyiding:feat/read-csv-without-schema
Open

tonyyuyiding wants to merge 7 commits into
pyrefly-org:mainfrom
tonyyuyiding:feat/read-csv-without-schema

Conversation

@tonyyuyiding

Copy link
Copy Markdown
Contributor

Summary

Fixes #5036

  • I am a human, and I fully understand the code I am submitting.

Motivation

Originally, in pl.read_csv, Pyrefly only infers the column dtypes if schema is provided. However, in the case below, where we have both columns and schema_overrides, there is enough information to infer the precise schema.

from typing import reveal_type
import polars as pl

df = pl.read_csv(
    'data.csv',
    columns=['DATE', 'TT BUY'],
    schema_overrides={'DATE': pl.Datetime, 'TT BUY': pl.String},
)
reveal_type(df) # Opaque dataframe. Desired: DataFrame['DATE': Date, 'TT BUY': String]

With this PR, Pyrefly will try to infer the schema as precisely as possible with columns and schema_overrides, even though schema is not given.

Approach

The function polars_csv_options is used to parse the options of read_csv and scan_csv. In the old implementation, it assumed schema always exists and schema_overrides is either a sequence or absent, because the checker relied on schema to perform inference. To add the new feature, I have extended relevant enums and updated some old functions accordingly to handle more possible options. Then I added a new branch in infer_read_csv_schema to handle the cases where schema is missing, while the old branch was left unchanged.

Test Plan

Tests have been added in pyrefly/lib/test/polars/dataframe.rs.

@meta-cla meta-cla Bot added the cla signed label Oct 5, 2026
@github-actions github-actions Bot added the size/l label Oct 5, 2026
@meta-codesync

meta-codesync Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

This pull request has been imported. If you are a Meta employee, you can view this in D123393717. (Because this pull request was imported automatically, there will not be any future comments.)

@github-actions github-actions Bot added size/l and removed size/l labels Oct 5, 2026
@tonyyuyiding
tonyyuyiding marked this pull request as ready for review October 5, 2026 10:11
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

@github-actions

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had recent activity for more than 2 weeks.

If you are still working on this this pull request, please add a comment or push new commits to keep it active. Otherwise, please unassign yourself and allow someone else to take over.

Thank you for your contributions!

@github-actions github-actions Bot added the stale label Oct 21, 2026
@tonyyuyiding

Copy link
Copy Markdown
Contributor Author

:)

@rchen152

rchen152 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Sorry for the delay, I'll take a look this week

@rchen152 rchen152 added size/m and removed size/l labels Nov 2, 2026

@rchen152 rchen152 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, and sorry again for the delayed review! I found two behavior regressions, which we should definitely fix before merging (see in-line comments).

AI review also turned up a few code quality suggestions. Feel free to incorporate any that you think will improve the code, but consider these optional:

  1. The new case is a second copy of the read path.
    infer_read_csv_schema (polars_specials.rs:1552-1608) is now a two-arm match: the old body re-indented in one arm and a new pipeline in the other. Both arms end with the same row-index and rename steps (lines 1581-1586 and 1600-1606). The two cases only differ in where the starting columns come from. So the match should produce columns and then hand off to one shared tail. Better still, the missing-schema case could build its columns from the columns= names with Unknown dtypes, apply the overrides, and switch the selection to All. Then the existing code runs unchanged.

  2. The override handling is split into two helpers that each skip the other's variant. apply_polars_csv_mapping_overrides returns early unless the overrides are a Mapping, and apply_polars_csv_sequence_overrides returns true unless they're a Sequence. The missing-schema arm calls both in a row, and each call does nothing for one of the variants. Worse, the given-schema path now drops a Mapping silently through the sequence helper's else { return true }. That is the unreachable-looking-state pattern AGENTS.md warns against. The PR also deletes the comment that stated this rule ("A complete schema changes only for positional dtype overrides."), which was on the removed CsvDtypeOverrides::parse. A single apply function that matches all three variants explicitly would fix this. Each path should then make its choice to ignore a dict visible, and the rule should be stated again in a comment.

  3. The width-0 trick is redundant. At lines 1518-1521, a missing schema passes width 0 to polars_csv_selection so that index selections fail. The missing-schema arm rejects anything other than Names anyway (line 1589), so the same rule is enforced twice, and the first enforcement is indirect. Fixing (1) probably removes the need for it.

Comment thread pyrefly/lib/alt/polars_specials.rs Outdated
};
overrides.insert(
self.polars_column_name(key)?,
self.polars_dtype_from_expr(value)?,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This causes us to require schema_overrides to be resolvable even when schema is passed, which leads to this regression:

from typing import reveal_type
import polars as pl

def f(dt: pl.DataType) -> None:
    # today: DataFrame[a: Int64]
    # with PR: DataFrame
    reveal_type(pl.read_csv("data.csv", schema={"a": pl.Int64}, schema_overrides={"a": dt}))

Comment thread pyrefly/lib/alt/polars_specials.rs Outdated
let CsvColumnSelection::Names(names) = selection else {
return None;
};
if matches!(overrides, CsvDtypeOverrides::Unchanged) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This checks only whether overrides were passed, not whether they cover any of the listed columns, which causes pyrefly to treat cases that should be equivalent differently:

from typing import reveal_type

import polars as pl

reveal_type(pl.read_csv("data.csv", columns=["a"]))  # DataFrame
reveal_type(pl.read_csv("data.csv", columns=["a"], schema_overrides={}))  # DataFrame[a:Unknown]
reveal_type(pl.read_csv("data.csv", columns=["a"], schema_overrides={"z": pl.Int64}))  # DataFrame[a:Unknown]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you prefer DataFrame or DataFrame[a:Unknown] for these cases?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer DataFrame[a: Unknown] for all three. IMO an unknown dtype shouldn't affect known columns.

@github-actions

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has been waiting for its author for more than 2 weeks.

Please take the action it is waiting for, or close the PR if you are no longer working on it.

Thank you for your contributions!

@github-actions github-actions Bot added the stale label Nov 17, 2026
@github-actions github-actions Bot removed the size/m label Nov 18, 2026
@github-actions

This comment has been minimized.

@github-actions github-actions Bot removed the stale label Nov 19, 2026
@github-actions github-actions Bot added size/m and removed size/m labels Nov 19, 2026
@github-actions

Copy link
Copy Markdown

Diff from mypy_primer, showing the effect of this PR on open source code:

============================================================
SUMMARY
============================================================
Total: +1 new errors, -0 fixed errors
By preset: +0/-0 (default), +0/-0 (strict)

Projects with changes (1):
  python-sop: +1 -0
============================================================

FULL DIFF DETAILS
------------------------------------------------------------

python-sop (https://gitlab.com/dkg/python-sop)
+ ERROR sop/__init__.py:331:25-37: The type of attribute `autocomplete` is unknown because it comes from an error [unknown-from-error]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Type inference in polars read_csv with schema-overrides but without schema

3 participants