When parsing enums derived from `enum.Enum`, non-canonical values are ignored

[bindreams]

My package does not use pybind11 hacky enums, but instead derives enums from enum.Enum using a popular helper macro.

However, in that case the generated stub does not contain some values, the ones that are considered non-canonical (as defined in this comment). These are zero values (maybe just in enum.Flag) and repeated/alias values.

I haven't looked at the implementation, but the issue is probably related to this: python/cpython#109633. If I'm correct, the issue can probable be fixed by iterating over the __members__ of the enum class.

Example of what I mean:

>>> list(iter(InputMode))
[<InputMode.UNICODE: 1>,
 <InputMode.GS1: 2>,
 <InputMode.ESCAPE: 8>,
 <InputMode.GS1PARENS: 16>,
 <InputMode.GS1NOCHECK: 32>,
 <InputMode.HEIGHTPERROW: 64>,
 <InputMode.FAST: 128>,
 <InputMode.EXTRA_ESCAPE: 256>]

>>> InputMode.__members__
mappingproxy({'DATA': <InputMode.DATA: 0>,
              'UNICODE': <InputMode.UNICODE: 1>,
              'GS1': <InputMode.GS1: 2>,
              'ESCAPE': <InputMode.ESCAPE: 8>,
              'GS1PARENS': <InputMode.GS1PARENS: 16>,
              'GS1NOCHECK': <InputMode.GS1NOCHECK: 32>,
              'HEIGHTPERROW': <InputMode.HEIGHTPERROW: 64>,
              'FAST': <InputMode.FAST: 128>,
              'EXTRA_ESCAPE': <InputMode.EXTRA_ESCAPE: 256>})

As you can see, with iter(), the DATA value is missing.

[sizmailov]

I don't see how it's relevant to stub generation and this repo.

[bindreams]

I think it's relevant because I'm generating stubs for a pybind11 project and they are incorrect :)
I would have used mypy stubgen (which parses enums correctly) but it has multiple other problems.

Do you consider issues related to user-defined type casters out of scope for this project?

[sizmailov]

What is the generated stub, and what is expected?

[bindreams]

Sorry for the back and forth, I should have started with an example.

Here is an example enum in C++

enum class ExampleEnum : int {
    A = 0,
    B = 1,
    C = 1,
    D = 2,
};

Code to generate a pybind11 binding using the helper macro I mentioned above

// Global scope
P11X_DECLARE_ENUM(
    "ExampleEnum",
    "enum.Enum",  // or "enum.Flag", etc.
    {"A", ExampleEnum::A},
    {"B", ExampleEnum::B},
    {"C", ExampleEnum::C},
    {"D", ExampleEnum::D}
)

PYBIND11_MODULE(module_name, m) {
    p11x::bind_enums(m);
}

Here is what pybind11-stubgen generates, when this enum is bound to python's enum.Enum or enum.Flag (or derived classes) respectively:

class ExampleEnum(enum.Enum):
    A: typing.ClassVar[ExampleEnum]  # value = <ExampleEnum.A: 0>
    B: typing.ClassVar[ExampleEnum]  # value = <ExampleEnum.B: 1>
    D: typing.ClassVar[ExampleEnum]  # value = <ExampleEnum.D: 2>

class ExampleEnum(enum.Flag):
    B: typing.ClassVar[ExampleEnum]  # value = <ExampleEnum.B: 1>
    D: typing.ClassVar[ExampleEnum]  # value = <ExampleEnum.D: 2>

You can see that C is missing in both cases because it has a repeated value, and A is missing in the enum.Flag case becase it's value is 0. However, the real enum you get in the module obviously contains all of the elements, and they are accessible via the __members__ dict:

>>> dir(ExampleEnum)
['B', 'D', '__class__', ...]

>>> list(iter(ExampleEnum))
[<ExampleEnum.B: 1>, <ExampleEnum.D: 2>]

>>> ExampleEnum.__members__
mappingproxy({'A': <ExampleEnum.A: 0>,
              'B': <ExampleEnum.B: 1>,
              'C': <ExampleEnum.B: 1>,
              'D': <ExampleEnum.D: 2>})

For completeness, the desired stub would look like this:

class ExampleEnum(enum.Enum):
    A: typing.ClassVar[ExampleEnum]  # value = <ExampleEnum.A: 0>
    B: typing.ClassVar[ExampleEnum]  # value = <ExampleEnum.B: 1>
    C: typing.ClassVar[ExampleEnum]  # value = <ExampleEnum.C: 1>
    D: typing.ClassVar[ExampleEnum]  # value = <ExampleEnum.D: 2>

[dyollb]

@sizmailov
FYI. pybind11 added support for native enums (enum.Enum derived) two weeks ago:
pybind/pybind11#5555

The old py::enum_ has been marked as deprecated. I haven't tried yet, but maybe this issue will become more relevant to pybind11-stubgen if the issue can be reproduced with the new py::native_enum?

[dyollb]

Also, native enums (e.g. enum.IntEnum) support duplicate values (as does pybind11), however, the "second" key is just an alias:

from enum import IntEnum

status_dict = {
    "OK": 0,
    "SUCCESS": 0,  # alias
    "ERROR": 1,
}

Status = IntEnum("Status", status_dict)

print(Status.OK)       # Status.OK
print(Status.SUCCESS)  # Status.OK (alias)
print(Status.ERROR)    # Status.ERROR

[dyollb]

I can confirm this also happens with the new pybind11::native_enum, which uses the standard library enum types as base class.

For the following enum (added in enum.cpp):

enum class ConsoleBorderColor {
    Black = 1,
    White = 2,
    Red = 4,
    Bordeaux = 4
};

#if PYBIND11_VERSION_AT_LEAST(3, 0)

    py::native_enum<demo::sublibA::ConsoleBorderColor>(m, "ConsoleBorderColor",  "enum.Enum")
        .value("Black", demo::sublibA::ConsoleBorderColor::Black)
        .value("White", demo::sublibA::ConsoleBorderColor::White)
        .value("Red", demo::sublibA::ConsoleBorderColor::Red)
        .value("Bordeaux", demo::sublibA::ConsoleBorderColor::Bordeaux) // alias of Red
        .finalize();

#endif

I get the following stubs:

class ConsoleBorderColor(enum.Enum):
    Black: typing.ClassVar[ConsoleBorderColor]  # value = <ConsoleBorderColor.Black: 1>
    Red: typing.ClassVar[ConsoleBorderColor]  # value = <ConsoleBorderColor.Red: 4>
    White: typing.ClassVar[ConsoleBorderColor]  # value = <ConsoleBorderColor.White: 2>