Skip to content

Do not narrow enum equality comparisons when member values are not statically known - #11692

Open
Nilesh Patil (nileshpatil6) wants to merge 4 commits into
microsoft:mainfrom
nileshpatil6:fix/enum-narrowing-unknown-values
Open

Nilesh Patil (nileshpatil6) wants to merge 4 commits into
microsoft:mainfrom
nileshpatil6:fix/enum-narrowing-unknown-values

Conversation

@nileshpatil6

@nileshpatil6 Nilesh Patil (nileshpatil6) commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Bug

#11622 added equality narrowing for primitive-backed enums, so a member of an IntEnum or StrEnum correctly compares equal to its underlying value. When a member's value is not statically known, though, unwrapPrimitiveBackedEnumLiteral falls back to returning the enum literal unchanged. isLiteralValueEqualAtRuntime then compares an EnumLiteral against an int or str literal, which is never equal, so the comparison is treated as statically false and the positive branch is eliminated.

auto() is the common way to hit this:

from enum import IntEnum, auto

class Color(IntEnum):
    RED = auto()
    GREEN = auto()

c: Color = Color.RED
if c == 1:
    reveal_type(c)  # Never, but Color.RED == 1 is True at runtime

The same happens in reverse (x: Literal[1, 3] compared against x == Color.RED), and in a match statement a case on a member whose value is known can unsoundly eliminate siblings whose values are not.

Any computed member value has the same effect, auto() is just the most common form.

Change

isLiteralValueEqualAtRuntime now returns boolean | undefined, where undefined means the comparison cannot be decided statically because a member's underlying value is unknown. This includes nested members whose value comes from another enum member with an unknown value.

An indeterminate member no longer blocks all positive-branch narrowing. For standard enum construction and comparison, the positive branch keeps every member that could match (definite matches plus members with unknown values) and still excludes known nonmatches, so partially known enums keep useful precision. Where custom __eq__/__ne__/__new__, a custom metaclass, or class creation hooks could change the result, narrowing conservatively retains the enum. TypeVar conditions are preserved.

All determinate behaviour from #11622 is unchanged: enums whose member values are all known narrow exactly as before.

Tests

  • enumNarrowing1.py: auto() and computed members in both comparison directions, plus the match-statement sibling-elimination case.
  • enumNarrowingSafety1.py (added by Bill Schnurr (@bschnurr) in d70e5f6): mixed known/unknown IntEnum and StrEnum with narrower consumers, inequality, nested and deeply nested unknown values, known nested nonmatches, custom equality/construction/metaclass, IntFlag, subclass and descriptor hooks, and bound/constrained TypeVars.

Core build, ESLint, Prettier and all eight evaluator suites pass (1,463 tests).

@bschnurr

Bill Schnurr (bschnurr) commented Sep 1, 2026 •

Copy link
Copy Markdown
Member

🔒 Automated review in progress — Bill Schnurr (@bschnurr) is auto-reviewing this PR.

@bschnurr

Copy link
Copy Markdown
Member

Please add the direct match-statement regression described in the PR so the sibling-elimination path is explicitly covered.

Verification: The relevant tests could not be fully run in the isolated environment; this review is not fully verified.

Result: could-not-verify

Summary: Verification could not run because no trusted sandbox image is configured for `microsoft/pyright`, and local execution was not authorized. The PR adds five scenarios to the existing `EnumNarrowing1` sample test, covering unknown `auto()` and computed enum values in both comparison directions. No executable test result was obtained.

Test runs: 1 not run

  • ⚠️ Not run | Targeted test discovery and dependency preflight | git --no-pager status --short && printf '\n--- changed files ---\n' && git --no-pager diff --name-status upstream/main...HEAD && printf '\n--- test diff ---\n' && git --no-pager diff --unified=3 upstream/main...HEAD -- packages/pyright-internal/src/tests packages/pyright-internal/src/analyzer/typeGuards.ts && printf '\n--- dependency/profile check ---\n' && printf 'AUTOMATION_SANDBOX_PROFILE=%s\n' "$AUTOMATION_SANDBOX_PROFILE" && test -x packages/pyright-internal/node_modules/.bin/jest && echo 'package jest: present' || echo 'package jest: missing' && test -f packages/pyright-internal/package.json && node -e "const p=require('./packages/pyright-internal/package.json'); console.log(JSON.stringify({build:p.scripts.build,test:p.scripts.test},null,2))"
⚠️ Targeted test discovery and dependency preflight diagnostic output
Container verification could not start and local execution was not authorized for this PR HEAD: No trusted sandbox image is configured for microsoft/pyright.

@bschnurr Bill Schnurr (bschnurr) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@bschnurr Bill Schnurr (bschnurr) added the review-auto:approved Automated review: no blocking findings (approval posted). label Sep 1, 2026
@nileshpatil6

Copy link
Copy Markdown
Contributor Author

Added the match statement cases in 50ff887.

test_match_int_literal_pattern covers the sibling elimination path directly: ComputedValue.SECOND has a known value of 40 while FIRST is computed, so a case 40: pattern must keep FIRST in play rather than eliminating it. test_match_auto_int_literal_pattern covers the same path when no member value is known.

Both assert that neither branch narrows. On main they report errors (the matching case eliminates the unknown-valued member); with the change the sample is clean.

EnumNarrowing1 passes and tsc --noEmit is clean.

@bschnurr

Copy link
Copy Markdown
Member

Verification: The relevant tests could not be fully run in the isolated environment; this review is not fully verified.

Result: could-not-verify

Summary: No tests executed because the verification container could not start: no trusted `microsoft/pyright` sandbox image was configured, and local execution was unauthorized. The PR adds seven scenarios to the existing `EnumNarrowing1` sample covering unknown `IntEnum` values, reverse comparisons, computed values, and match narrowing. Runtime confidence is therefore limited.

Test runs: 1 not run

  • ⚠️ Not run | Verification environment preflight and test discovery | git status --short && git diff --name-status upstream/main...HEAD && git diff --unified=0 upstream/main...HEAD -- packages/pyright-internal/src/tests && if exist packages\pyright-internal\node_modules\jest\bin\jest.js (echo JEST_PRESENT) else (echo JEST_MISSING) && node -e "const p=require('./packages/pyright-internal/package.json'); console.log(JSON.stringify(p.scripts,null,2))"
⚠️ Verification environment preflight and test discovery diagnostic output
Container verification could not start: no trusted sandbox image is configured for microsoft/pyright, and local execution was not authorized.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@heejaechang

Heejae Chang (heejaechang) commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

🟢 Pyright CLI QA — 50ff8870 — verdict: green

Automated CLI checks compared this change with its base revision.
6 focused check(s) completed.

No new behavior difference was detected.

@heejaechang

Copy link
Copy Markdown
Collaborator

The added test_match_int_literal_pattern and test_match_auto_int_literal_pattern scenarios pass on both the pre-PR baseline (5a958ca8) and PR head (50ff8870), so they don’t demonstrate the claimed match-statement regression.

The distinction is between a direct pattern and an equality comparison in a guard:

# Direct literal pattern: uses the separate pattern-matching narrowing path.
match value:
    case 40:
        ...

# Equality in a guard: exercises the equality helper changed by this PR.
match value:
    case _ if value == 40:
        ...

Could you clarify which match behavior this PR fixes and add a regression test that fails on the baseline and passes with the change?

If the direct-pattern tests are intended as unchanged-behavior controls, keeping them makes sense, but the description should identify them as such.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@bschnurr

Copy link
Copy Markdown
Member

Copilot resolve the merge conflicts in this pull request

@rchiodo Rich Chiodo (rchiodo) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The direct computed-enum comparison is fixed, but a nested computed value still gets its reachable branch discarded:

from enum import IntEnum
from typing import Literal

def computed() -> int: return 7
class Source(IntEnum): VALUE = computed()
class Nested(IntEnum): VALUE = Source.VALUE

def nested(v: Literal[Nested.VALUE]) -> int:
    if v == 7:
        return 'reachable'  # head misses reportReturnType
    return 0

Both base and head consider the branch unreachable and miss reportReturnType, but CPython 3.14.3 reaches it (Nested.VALUE == 7 is true). The direct Source.VALUE == 7 control is incorrectly discarded on base and checked on head. A known nonmatch remains unreachable.

unwrapPrimitiveBackedEnumLiteral treats the inner EnumLiteral as a known literal even when its primitive payload is unknown. Could we conservatively preserve unknown nested values and add coverage?

test_int_var_equals_auto_member also passes on base because its positive branch becomes Never and the assertion never gets checked. On base, the sample reports errors only at lines 103, 128 and 137, none in this test. This local form fails on base (Never) and passes on head:

from enum import IntEnum, auto
from typing import Literal, assert_type
class AutoValue(IntEnum):
    RED = auto()
    GREEN = auto()
def local_form(flag: bool):
    x = 1 if flag else 3
    if x == AutoValue.RED:
        assert_type(x, Literal[1, 3])

Could we add a reachability-sensitive reverse-direction test like this?

One optional precision improvement: for C(IntEnum) with FIRST = get_int(), SECOND = 40 and THIRD = 50, head keeps all of C after v: C is compared to 40. The expanded Literal[C.FIRST, C.SECOND, C.THIRD] narrows to Literal[C.FIRST, C.SECOND]. Keeping all of C is sound. Base was unsound if FIRST evaluates to 40, since SECOND becomes its alias. Maybe the C form could keep definite matches and unknown-valued members while dropping known nonmatches. That refinement hasn't been tried.

@bschnurr

Copy link
Copy Markdown
Member

AI-assisted review.

Following up on the precision case in Rich Chiodo (@rchiodo)'s review: retaining the entire enum also causes a new false positive at an ordinary consumer.

from enum import IntEnum
from typing import Literal, reveal_type


def computed() -> int:
    return 2


class E(IntEnum):
    ONE = 1
    COMPUTED = computed()
    THREE = 3


def consume(value: Literal[E.ONE, E.COMPUTED]) -> str:
    return value.name


def example(value: E) -> str:
    if value == 1:
        reveal_type(value)
        return consume(value)
    return "other"


if __name__ == "__main__":
    for member in E:
        print(example(member))

In basic mode, baseline 5a958ca8 accepts the call, while head 50ff8870 reveals E and reports reportArgumentType at consume(value). The baseline's Literal[E.ONE] narrowing is too narrow if COMPUTED can also equal 1; the desired result should retain that unknown-valued member. But the known THREE member cannot match 1, so retaining it prevents this valid consumer from type-checking. The same issue reproduces with a StrEnum; the negative-branch and all-known controls retain their precision.

Could the positive branch retain the definite matches and unknown-valued members, while excluding known nonmatches, with a consumer test like this one?

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Class B: Pyright cannot determine values added or changed by dynamic enum hooks. Keep those comparisons conservative while retaining the possible literal members for standard enums and preserving TypeVar conditions. Treat nested unknown enum values as indeterminate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bschnurr

Copy link
Copy Markdown
Member

Pushed d70e5f6 to cover nested enum members whose underlying values are unknown and to preserve useful positive-branch precision for partially known enums. For standard construction and comparison, narrowing keeps all possible matching members; for dynamic hooks or custom behavior it conservatively retains the enum. The change also preserves TypeVar conditions.

Validation: the core build, ESLint, Prettier, and all eight evaluator suites passed (1,463 tests), including the focused default and Python 3.11 enum cases. Dynamic subclass and descriptor hooks were checked against CPython. The additional enum-member scan has not been performance-benchmarked.

Please update the PR description: an indeterminate member no longer necessarily prevents all positive-branch narrowing, and the follow-up adds enumNarrowingSafety1.py coverage.

@nileshpatil6

Copy link
Copy Markdown
Contributor Author

Thanks Bill Schnurr (@bschnurr) for the follow-up commit, and Rich Chiodo (@rchiodo) for the nested and consumer cases. I've updated the description to match the new positive-branch behaviour and the enumNarrowingSafety1.py coverage.

@github-actions

Copy link
Copy Markdown
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

@github-actions

Copy link
Copy Markdown
Contributor

Compared candidate 0bb7282627c58d8de8ab83a38e485b8a9fab6940 against its first parent 7e29199f7b3c5bddeda6956588a15552398a66ad.

Release history charts were not rendered for this run. See the workflow run for details.

Type checker benchmark

🟢 No performance regressions detected.

Regression threshold: 20.0%
Statistic: median
Variance guard: >1.0s time and >100.0 MB memory

Package Checker Files checked Time Time delta Peak memory Memory delta Status
ansible pyright 583 22.002s +0.3% 1064.4 MB -0.2% 🟢 Pass
click pyright 17 3.138s -2.0% 359.6 MB +0.5% 🟢 Pass
homeassistant pyright 9850 152.331s -0.5% 6018.2 MB +0.9% 🟢 Pass
numpy pyright 356 30.493s +2.1% 1338.1 MB +10.8% 🟢 Pass
pandas pyright 1459 137.046s -0.5% 3449.3 MB -1.7% 🟢 Pass
pytest pyright 243 12.600s -3.2% 867.2 MB +0.4% 🟢 Pass
requests pyright 19 2.425s -1.2% 347.1 MB -0.3% 🟢 Pass
torch pyright 2400 151.794s +2.4% 4745.6 MB -2.2% 🟢 Pass
transformers pyright 2901 127.760s +0.5% 4855.8 MB +0.1% 🟢 Pass

Pyright stats

Package Parsed/bound Checked Find Read Tokenize Parse Imports Bind Check Cycles
ansible 988 583 0.050s 0.100s 0.380s 0.777s 0.213s 1.007s 18.483s 0.000s
click 111 17 0.000s 0.013s 0.117s 0.210s 0.037s 0.273s 2.220s 0.000s
homeassistant 11154 9850 0.770s 0.770s 2.957s 5.530s 2.110s 6.870s 129.630s 0.000s
numpy 606 356 0.010s 0.053s 0.293s 0.523s 0.210s 0.760s 28.163s 0.000s
pandas 1899 1459 0.070s 0.240s 1.263s 2.733s 0.293s 2.873s 128.587s 0.000s
pytest 545 243 0.020s 0.053s 0.370s 0.517s 0.107s 0.717s 10.393s 0.000s
requests 167 19 0.000s 0.010s 0.120s 0.207s 0.063s 0.263s 1.443s 0.000s
torch 3078 2400 0.187s 0.420s 2.090s 3.783s 0.630s 5.367s 138.913s 0.000s
transformers 3536 2901 0.167s 0.380s 2.200s 3.573s 0.903s 8.940s 112.617s 0.000s

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-auto:approved Automated review: no blocking findings (approval posted).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants