Repository navigation
Allow a constructor with union-valued arguments to match a union annotation for #5456 - #5537
asukaminato0721 wants to merge 4 commits into
Conversation
|
✅ This PR meets all requirements from our pull request checklist. Optional recommendations:
Note: this comment updates automatically after changes to the PR. Updates may take up to half an hour. |
|
This pull request has been imported. If you are a Meta employee, you can view this in D129832053. (Because this pull request was imported automatically, there will not be any future comments.) |
This comment has been minimized.
This comment has been minimized.
stroxler
left a comment
There was a problem hiding this comment.
This seems pretty good, I think the overall structure is probably right.
An AI review flags a set of performance and correctness concerns, though. We should fix the perf issue and ideally fix (or at least track with bug= tests if it's not worth fixing) the correctness concerns
recursive expansion cost
Location: call.rs:1289–1303.
A complete source generator for the measured eight-level example:
expression = "Node(lambda arg: None, value)"
for _ in range(7):
expression = f"Node(lambda arg: None, value, {expression})"
source = f"""from typing import Callable
class Node[T]:
def __init__(self, cb: Callable[[T], None], value: T,
child: Node[int] | Node[str] | None = None) -> None: ...
def f(value: int | str) -> None:
root: Node[int] | Node[str] = {expression}
"""
print(source)Optimized, single-thread before/after measurements on the same worker:
- Depth 4: base 0.101s; head 0.125s.
- Depth 6: base 0.184s; head 1.017s.
- Depth 8: base 1.227s; head 32.868s, about 27× slower. Independent Codex confirmation: 1.156s versus 34.243s.
- A simpler rejected case, with
Node(value, child), no callbacks, andvalue: int | str | bytes: depth 8 base 0.515s; head exceeds 20s.
Base emits assignment/child errors on these union examples; it is not being presented as accepting them. The base failure path is already recursive and slow at greater depths. The new expansion amplifies that cost.
Each expansion rechecks unchanged children. Lambda-containing calls bypass prepared inference caching at fbcode/pyrefly/pyrefly/lib/alt/callable.rs:236–238; unsuccessful closed inference also cannot benefit from successful-result caching. The per-invocation argument-combination budget does not bound recursive work.
Improve: preserve contextual checking while reusing compatible outcomes or bounding recursive speculative work. Add benchmarks/tests for contextual bindings and rejected unions, not only successful closed calls. Do not solve this by blindly flattening lambdas without context.
invalid conditional lambda accepted
Location: overload.rs:152–174, now used by constructor expansion.
from typing import Callable
class A[T]:
def __init__(self, cb: Callable[[T], None] | None, value: T) -> None:
if cb is not None:
cb(value)
def f(value: int | str, condition: bool) -> None:
result: A[int] | A[str] = A(
(lambda arg: arg.upper()) if condition else None, value
)- Base:
A[int | str]not assignable toA[int] | A[str]. - Head: zero errors.
- Pyright: explicitly reports that
inthas noupper, plus callback incompatibility. - CPython:
f(1, True)raisesAttributeError: 'int' object has no attribute 'upper'. - The corresponding direct, non-union constructor still reports the invalid integer attribute on both builds.
The top-level expression is a conditional, not a lambda. The probe infers its lambda without parameter context, expands the resulting callable-or-None union into types, and never checks the lambda expression again. This newly removes the error that previously prevented acceptance of invalid code.
Improve: preserve contextual expressions recursively, including conditional/compound expressions containing argument bindings. Conservatively declining an unsafe expansion is preferable to accepting this program.
branch-local cast redundancy
Location: call.rs:1312–1319.
from typing import Callable, cast
class A[T]:
def __init__(self, callback: Callable[[T], int], value: T) -> None: ...
def f(value: int | str) -> None:
result: A[int] | A[str] = A(lambda arg: cast(int, arg), value)- Base: only the expected
bad-assignment. - Head: accepts, but warns
Redundant cast: int is the same type as int. - With
--error redundant-cast, head fails checking this example. - Removing the cast produces
bad-assignmentand an incompatible callback return type on both builds.
The cast is redundant under the int context, but remains necessary to satisfy the callback’s declared int return in the str context. Unioning soft diagnostics from accepted outcomes turns a branch-local observation into a globally invalid recommendation; deduplication does not correct that quantification.
Improve: require redundancy to hold across accepted contexts, without blanket suppression of legitimate diagnostics such as deprecation.
a5b8ade to
6df7a80
Compare
Merging this PR will improve performance by 19.27%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
This comment has been minimized.
This comment has been minimized.
stroxler
left a comment
There was a problem hiding this comment.
Thanks!
At this point based on AI review I still think the overall approach is good, but there are a lot of implementation details to work through. This is a pretty complex change, so I guess it's not surprising we might go through multiple rounds of finding issues:
Here are some problem flagged for a closer look:
Rejected attempts lose caller-bound errors while retaining illegal specialization
Location: new retry loop, call.rs:1247–1299. The existing caller-variable snapshot at 1483–1503 covers preliminary hint comparison, not the subsequent matching and finishing.
class A[T]:
def __init__(self, x: T) -> None:
self.x: T = x
def consume[T: int](item: A[T] | A[str]) -> None:
x = item.x
if not isinstance(x, str):
x.bit_length()
def f(x: int | str | bytes) -> None:
consume(A(x))
f(b'not an int or str')Identical code: base reports an upper-bound specialization error; head reports zero errors. Mypy and Pyright reject the call. CPython 3.14 raises AttributeError: 'bytes' object has no attribute 'bit_length'.
An additional reveal-type probe retains bytes | int | str as the caller's result type on both revisions, but only base reports the illegal bound. Constrained caller parameters and independent mixed pairs show the same lost-error behavior. The fix must include speculative caller-variable/restriction state, not just deferred diagnostic publication.
The budget degrades a separately cached binding
Location: budget ownership, answers_solver.rs:2404–2411, and unconditional spending in the constructor selection loop.
class A[T]:
def __init__(self, value: T) -> None: ...
class Node[T]:
def __init__(self, value: T,
child: Node[int] | Node[str] | None = None) -> None: ...
class Wrap:
def __init__(self, a: Node[int] | Node[str], b: object) -> None: ...
def need(x: A[int] | A[str]) -> int: ...
def f(w: int | str | bytes, v: int | str) -> None:
Wrap(Node(w, Node(w, Node(w, Node(w, Node(w))))), helper(v)) # type: ignore
def helper(v: int | str):
return need(A(v))Head reports an argument error inside helper. Replace only the Wrap(...) line with pass: the unchanged helper then checks with zero errors. Base reports the helper error in both versions. This was reproduced with one checking thread.
An independent binding's return inference can first run under an exhausted constructor budget and retain the degraded result. Separate controls also show that swapping constructor arguments changes whether a later A(v) gets distribution, and 1,001 unrelated plain P() calls can exhaust it. A limit on speculative work is reasonable; letting unrelated or rejected work affect another cached binding is not a local precision limit.
A branch-only lint becomes a false positive
Location: diagnostic merging, call.rs:1310–1347.
from typing import Callable
class A[T]:
def __init__(self, cb: Callable[[T], object], value: T) -> None: ...
def f(value: int | str) -> None:
result: A[int] | A[str] = A(lambda arg: str(arg), value)With unnecessary-type-conversion enabled, head newly reports Unnecessary str() call. The conversion is necessary in the int branch, so that diagnostic is incorrect. The merge intersects only RedundantCast and unions all other diagnostics. Cross-context diagnostic validity needs a consistent policy, not another constructor-specific exception.
Adding an extra hint member prevents a valid constructor match
Location: selection, call.rs:1247–1256, versus the stricter batch acceptance predicate.
For A[T, U] taking value: T and callback: Callable[[U], str], head accepts:
result: A[int, str] | A[str, str] = A(value, lambda x: x.upper())It rejects the identical call when the annotation also includes A[int, int]. The first hint can contain a failing callback-body check, but select returns it before examining later valid hints. The batch then rejects it through is_match.
Base rejects both annotations, so this is incomplete new-feature behavior, not a claimed new base-to-head diagnostic regression. Each input alternative has a valid later hint; no complexity limit explains the rejection. Keyword arguments reproduce it too.
Structural direction: retain constructor-specific __new__/__init__ semantics, but make an attempt's inference state, budget effects, and diagnostics obey one ownership/publication boundary. Reusing argument enumeration alone does not provide that boundary. A broader solver overhaul or synthetic constructor overloads are not justified by this review.
Possible tech debt
AI review also flagged that selection and batch acceptance use different match predicates, and selection also owns expansion, cross-context diagnostic filtering, and budget spending. This might be worth a look, it seems like there might be inconsistent duplication and a bit of tech debt if we don't try to clean this up
6df7a80 to
dcd60d6
Compare
This comment has been minimized.
This comment has been minimized.
dcd60d6 to
7d6dfe1
Compare
This comment has been minimized.
This comment has been minimized.
7d6dfe1 to
532beb1
Compare
This comment has been minimized.
This comment has been minimized.
stroxler
left a comment
There was a problem hiding this comment.
Thanks! It's looking better, but probably still needs some work
In particular, AI review is flagging that there are code organization issues that look like tech debt, and appear to be driving some bad behaviors including incomplete fixes to some of the concerns raised earlier
In addition, there's a perf concern with lambda handling.
Details:
Structural issues
-
One boundary in name only.
construct_with_hinthandles hint filtering, budget scope, root discovery, four snapshot/restore sites, trace deferral, the expansion loop, the caller-evidence join, the replay pass, the diagnostic merge, the lambda-type merge, and the result union. A probe's state is committed in one place (attempt.commit), then partly patched again afterwards (lambda unions, the diagnostic filter, trace replay). Findings #1, #2, and #4 all come from that split. -
Duplicated machinery.
call_intersection(call.rs:~2619–2690) already runs the same speculative attempt: it builds roots from args, the hint, and first-use types, takes a lambda snapshot, callsenter_lambda_binding_context,snapshot_reachable_vars, expands the result and lambda types, restores, and merges lambda types by union.ConstructorStateandconstructor_argument_typescopy that logic. Concrete alternative: extract onespeculate(roots, f) -> Speculation<T>(value, post-state, expanded lambda types, closed traces, diagnostics) with an explicitcommit/merge. Both callers would use it, and a singlemerge_accepted(&[Speculation])would apply one rule each for vars, lambda types, traces, and diagnostics. With that in place,construct_with_hintsplits into select / distribute / merge pieces of about 30–40 lines each. The cost is a moderate refactor that also touches intersection calls; the benefit is that #1, #2, and #4 get fixed in one place instead of three. This keeps constructor__new__/__init__semantics as they are, and it is neither synthetic overloads nor a solver overhaul. -
Budget. A thread-global
Cell<Option<State>>keyed onstack().len(), plus aspeculativeflag, couples the budget to the calc-stack layout. It fixes the bindings case from last round, but results still depend on argument order inside one expanding tree: an expensive sibling starvesA(v). This is deterministic and not a regression against base. PlainP()calls inside the expanding call's own arguments still spend fuel (990 accepted, 1,000 rejected). Two simpler options:- (a) Don't distribute inside another constructor's expansion probe at all. That is one flag, with no cliff and no order dependence, at some cost to nested contextual precision.
- (b) Pass the budget explicitly as part of the
Speculationframe.
Also:
1000is an inline magic number (answers_solver.rs:2460),ConstructorBudget.remainingactually holds the whole state, andowner: bool+previouswould be clearer as an enum.
Restructure should fix: We still get branch-only false positives for some rules
# pyrefly.toml: incompatible-comparison = true
class A[T]:
def __init__(self, cb: Callable[[T], object], value: T) -> None: ...
def f(value: int | str) -> None:
r: A[int] | A[str] = A(lambda arg: arg == "a", value)base: ERROR 5:26-58 `A[int | str]` is not assignable to `A[int] | A[str]` [bad-assignment]
head: ERROR 5:47-50 Comparison `==` between incompatible types `int` and `Literal['a']` [incompatible-comparison]
The comparison is fine for arg: int | str. This is the same class of bug as prior #3.
crates/pyrefly_config/src/error_kind.rs @ head:
786 pub fn requires_all_contexts(self) -> bool {
787 matches!(
788 self,
789 ErrorKind::RedundantCast
790 | ErrorKind::RedundantCondition
791 | ErrorKind::UnnecessaryComparison
792 | ErrorKind::UnnecessaryTypeConversion
793 | ErrorKind::PytorchEfficiencyLintRedundantToCall
794 )- An opt-in list goes stale whenever someone adds a new lint.
RedundantConditionandUnnecessaryComparisonare hard kinds, so any probe that emits them already failsis_match, andreplay_alternativesnever sees them. That makes those two entries dead and contradicts the doc comment's claim that the list is "independent of selection".call_intersection(call.rs:~2770–2797) merges accepted contexts under a different policy (it keeps the first context'sarg_errors), so we would now have two cross-context policies.replay_alternatives<const N>has one caller with N=3, and it keys on(range, kind)rather than on the message.
Suggested direction: the safer default is to invert the list. Keep a diagnostic emitted inside a re-checked contextual expression (a lambda body) only if it appears in every accepted context, and exempt the few kinds that must always survive, such as deprecated. Alternatively, keep the policy local to constructors. Either way, intersection calls and constructors should share one rule.
Restructure should fix: Accepted alternatives overwrite IDE traces
class X:
def meth(self) -> int: ...
class Y:
def meth(self) -> str: ...
def f(value: X | Y) -> None:
result: A[X] | A[Y] = A(lambda arg: arg.meth(), value)Hover on meth:
- base:
((self: X) -> int) | ((self: Y) -> str) - head:
def meth(self: Y) -> str
Go-to-definition: base returns X.meth and Y.meth; head returns Y.meth only. Claude also saw hover on the lambda itself show (arg: str) -> str instead of the union.
lib/alt/call.rs @ head:
1479 for (id, types) in lambda_types {
1480 self.set_lambda_param_type(id, self.unions(types));
1481 }
1482 for traces in expansion_traces.into_iter().chain(traces) {
1483 self.trace_state().replay_traces(&traces);
1484 }Lambda types are merged by union, but traces are replayed with extend, so the last write wins at each range. This is a second instance of the same problem as #1 and #2: each effect channel has its own merge rule.
Perf issue: Width-1 hints become quadratic in sibling lambdas (should-fix)
items: list[A[int]] = [A(lambda x: None), A(lambda x: None), ...] # N calls in one binding| N | base | head |
|---|---|---|
| 3,200 | 0.21 s | 1.33 s (mine) |
| 6,400 | 0.26 s | 4.84 s (Codex) |
Without the annotation, the timings are equal.
lib/alt/call.rs @ head:
1324 let Some(hint) = hint.filter(|hint| hint.types().len() <= MAX_HINT_WIDTH) else {
...
1337 let before = ConstructorState::capture(self, &roots);
...
1344 let mut state = ConstructorState::capture(self, &roots);
1345 for (_, ty) in &mut state.lambda_types {
1346 if let Some(ty) = ty {
1347 *ty = self.solver().expand(ty.clone(), self);ConstructorState::capture calls snapshot_lambda_param_types(), which walks every lambda param owned by the binding (answers_solver.rs:2590). It does that twice per call, and it does it even with a width-1 hint where nothing gets expanded. Suggested direction: width-1 hints (or any call that is not expandable) should keep the base path and call construct(Some(hint)) directly. When speculation does run, snapshot only the lambda params inside this call's argument subtree.
532beb1 to
646adc9
Compare
This comment has been minimized.
This comment has been minimized.
646adc9 to
9b0ea7e
Compare
|
Diff from mypy_primer, showing the effect of this PR on open source code: ============================================================
SUMMARY
============================================================
Total: +14 new errors, -14 fixed errors
By preset: +3/-5 (default), +3/-5 (strict)
Projects with changes (11):
pip: +1 -0
rotki: +0 -1
koda-validate: +2 -1
spark: +2 -0
ibis: +5 -0
bokeh: +1 -10
pandas: +1 -0
pytensor: +0 -1
werkzeug: +0 -1
dedupe: +1 -0
rich: +1 -0
============================================================
FULL DIFF DETAILS
------------------------------------------------------------
pip (https://github.com/pypa/pip)
+ ERROR src/pip/_vendor/rich/progress.py:504:44-82: Unused `# type: ignore` comment [unused-type-ignore]
rotki (https://github.com/rotki/rotki)
- ERROR rotkehlchen/api/services/transactions.py:496:36-500:14: `dict[CHAINS_WITH_TRANSACTIONS_TYPE, list[BTCAddress | ChecksumAddress | SolanaAddress | SubstrateAddress]]` is not assignable to variable `blockchain_addresses` with type `dict[CHAINS_WITH_TRANSACTIONS_TYPE, ListOfBlockchainAddresses]` [bad-assignment]
koda-validate (https://github.com/keithasaurus/koda-validate)
+ ERROR koda_validate/dictionary.py:60:20-43: Returned type `Valid[Just[Unknown]]` is not assignable to declared return type `Invalid | Valid[Maybe]` [bad-return]
+ ERROR koda_validate/dictionary.py:67:20-43: Returned type `Valid[Just[Unknown]]` is not assignable to declared return type `Invalid | Valid[Maybe]` [bad-return]
- ERROR koda_validate/tuple.py:260:16-262:10: Returned type `NTupleValidator[tuple[T1] | tuple[T1, T2] | tuple[T1, T2, T3] | tuple[T1, T2, T3, T4] | tuple[T1, T2, T3, T4, T5] | tuple[T1, T2, T3, T4, T5, T6] | tuple[T1, T2, T3, T4, T5, T6, T7] | tuple[T1, T2, T3, T4, T5, T6, T7, T8]]` is not assignable to declared return type `NTupleValidator[tuple[T1]] | NTupleValidator[tuple[T1, T2]] | NTupleValidator[tuple[T1, T2, T3]] | NTupleValidator[tuple[T1, T2, T3, T4]] | NTupleValidator[tuple[T1, T2, T3, T4, T5]] | NTupleValidator[tuple[T1, T2, T3, T4, T5, T6]] | NTupleValidator[tuple[T1, T2, T3, T4, T5, T6, T7]] | NTupleValidator[tuple[T1, T2, T3, T4, T5, T6, T7, T8]]` [bad-return]
spark (https://github.com/apache/spark)
+ ERROR python/pyspark/pandas/namespace.py:353:34-60: Unused `# type: ignore` comment [unused-type-ignore]
+ ERROR python/pyspark/pandas/namespace.py:1192:58-60: The type of this argument is partially unknown; it contains an implicit `Any` [partially-unknown-type]
ibis (https://github.com/ibis-project/ibis)
+ ERROR ibis/common/tests/test_collections.py:174:12-18: The type of this argument is partially unknown; it contains an implicit `Any` [partially-unknown-type]
+ ERROR ibis/common/tests/test_collections.py:175:12-23: The type of this argument is partially unknown; it contains an implicit `Any` [partially-unknown-type]
+ ERROR ibis/common/tests/test_collections.py:176:12-16: The type of this argument is partially unknown; it contains an implicit `Any` [partially-unknown-type]
+ ERROR ibis/common/tests/test_collections.py:177:12-17: The type of this argument is partially unknown; it contains an implicit `Any` [partially-unknown-type]
+ ERROR ibis/common/tests/test_collections.py:178:12-18: The type of this argument is partially unknown; it contains an implicit `Any` [partially-unknown-type]
bokeh (https://github.com/bokeh/bokeh)
- ERROR src/bokeh/models/filters.py:92:16-94:9: This expression's type is partially unknown; it contains an implicit `Any`. Please provide an explicit type annotation. [partially-unknown-type]
- ERROR src/bokeh/models/filters.py:92:25-56: The type of this argument is partially unknown; it contains an implicit `Any` [partially-unknown-type]
- ERROR src/bokeh/models/filters.py:92:34-55: The type of this argument is partially unknown; it contains an implicit `Any` [partially-unknown-type]
- ERROR src/bokeh/models/graphs.py:84:20-97:9: This expression's type is partially unknown; it contains an implicit `Any`. Please provide an explicit type annotation. [partially-unknown-type]
- ERROR src/bokeh/models/graphs.py:84:46-64: The type of this argument is partially unknown; it contains an implicit `Any` [partially-unknown-type]
- ERROR src/bokeh/models/graphs.py:84:50-60: The type of this argument is partially unknown; it contains an implicit `Any` [partially-unknown-type]
- ERROR src/bokeh/models/graphs.py:84:66-76: The type of this argument is partially unknown; it contains an implicit `Any` [partially-unknown-type]
- ERROR src/bokeh/models/layouts.py:408:86-91: The type of `child` is an explicit `Any` [explicit-any-use]
- ERROR src/bokeh/models/layouts.py:429:86-91: The type of `child` is an explicit `Any` [explicit-any-use]
- ERROR src/bokeh/models/tools.py:677:90-96: Argument `Sequence[Sequence[Any]]` is not assignable to parameter `groups` with type `list[list[Model]]` in function `bokeh.models.misc.group_by.GroupByModels.__init__` [bad-argument-type]
+ ERROR src/bokeh/models/tools.py:677:90-96: Argument `Sequence[Sequence[DataRenderer]]` is not assignable to parameter `groups` with type `list[list[Model]]` in function `bokeh.models.misc.group_by.GroupByModels.__init__` [bad-argument-type]
pandas (https://github.com/pandas-dev/pandas)
+ ERROR pandas/io/common.py:142:30-32: The type of this argument is partially unknown; it contains an implicit `Any` [partially-unknown-type]
pytensor (https://github.com/pymc-devs/pytensor)
- ERROR pytensor/compile/rebuild.py:167:19-32: `list[SymbolicOutput | Variable[Unknown, Unknown]]` is not assignable to variable `outputs` with type `Sequence[SymbolicOutput] | Sequence[Variable[Unknown, Unknown]] | SymbolicOutput | Variable[Unknown, Unknown]` [bad-assignment]
werkzeug (https://github.com/pallets/werkzeug)
- ERROR src/werkzeug/wrappers/response.py:460:33-52: `list[bytes | str]` is not assignable to attribute `response` with type `Iterable[bytes] | Iterable[str]` [bad-assignment]
dedupe (https://github.com/dedupeio/dedupe)
+ ERROR dedupe/api.py:551:47-73: Unused `# type: ignore` comment [unused-type-ignore]
rich (https://github.com/Textualize/rich)
+ ERROR rich/progress.py:497:44-82: Unused `# type: ignore` comment [unused-type-ignore] |
Summary
Fixes stage3 of #5456 (review)
Test Plan