[mypyc] Add librt.strings.isidentifier codepoint primitive#21522
Conversation
True if a codepoint can start a valid identifier (XID_Start, per PEP 3131). The ASCII fast path covers `[A-Za-z_]` inline; non-ASCII codepoints round-trip through PyUnicode_FromOrdinal + PyUnicode_IsIdentifier so the answer matches str.isidentifier on a 1-character string. The non-ASCII path is the first allocating helper in this series, so its body lives out-of-line in codepoint_extra_ops.c (it would otherwise be emitted as a separate copy in every translation unit that includes the header). On OOM it swallows the exception via PyErr_Clear() and returns False, which keeps the function ERR_NEVER. Documented at the call site so callers don't get a surprising silent failure. Stack: depends on the librt.strings.isspace primitive.
add1d44 to
283b2f8
Compare
This comment has been minimized.
This comment has been minimized.
| ModDesc( | ||
| "librt.strings", | ||
| ["strings/librt_strings.c", "codepoint_extra_ops.c"], | ||
| ["codepoint_extra_ops.h"], | ||
| ["strings"], | ||
| ), |
There was a problem hiding this comment.
sorry for not catching this earlier but it looks like the files are dependent on each other the wrong way. the _extra_ops files should be internal to mypyc and shouldn't be needed when building librt modules.
so instead of including codepoint_extra_ops in librt_strings we need it the other way around, with the common functionality in librt_strings. like it's done with the other _extra_ops files.
There was a problem hiding this comment.
No worries, that's on me first!
There was a problem hiding this comment.
thanks! i see there's still #include "codepoint_extra_ops.h" in librt_strings.c and it would be good to clean that up by moving the functions from there to librt_strings.h.
that would make the codepoint_extra_ops files empty so maybe they aren't actually needed and we could just compile the functions statically straight from librt_strings.h? unless you're planning on adding functions later that would be better placed outside of librt_strings.h.
There was a problem hiding this comment.
I was planning on using it to add functions like toupper(), tolower() etc so that they're separated from librt_strings, what do you think? Worth the complexity?
There was a problem hiding this comment.
would there be separate implementations used by the python wrapper in librt_strings and when compiled statically with mypyc? in that case sure, i think it'd make sense to have them in codepoint_extra_ops.
but if they're more similar to the functions added so far where the python wrapper calls the same function that mypyc could generate in the C file then they would have to be in librt_strings. it would be cleaner to keep the separation between the librt modules and C files relevant only in mypyc generated code by not including them by the librt module files.
There was a problem hiding this comment.
I don't think so, they'd be the same function afaict. Do we worry about the librt capsule indirection overhead? I assume these codepoint primitives will participate in hot loops so preferably they'd be inlined instead of going through the librt fp array (?)
I see the overall point though, there's no mypyc callers so there's no real point in having these outside of librt_strings.
There was a problem hiding this comment.
yeah there would be overhead if the functions are exposed through the C_API array. but i think we could avoid it by defining the C functions inline in librt_strings.h instead. that way when compiling with mypyc, the function would be part of the extension and called statically.
i think this would be better than exposing through C_API especially for the functions that are very simple and thus wouldn't bloat the extension object files. edit: and i think it would be fine to define all the functions you've added so far as inline.
There was a problem hiding this comment.
Updated the PR, hopefully with what you had in mind
There was a problem hiding this comment.
yes, i think this will work well. thanks!
- codepoint_extra_ops.h: include CPy.h and move the isidentifier slow
path inline into LibRTStrings_IsIdentifier. Aborts via
CPyError_OutOfMemory on allocation failure (the helper is ERR_NEVER,
so returning a silently-wrong bool under memory pressure was the wrong
contract). Matches the pattern in the sibling _extra_ops.h files (all
bodies static inline, CPy.h included for runtime helpers).
- codepoint_extra_ops.c: reduce to a single-line shim that #includes the
header. Exists only so SourceDep("codepoint_extra_ops.c") pulls the
header into user mypyc extensions in include_runtime_files mode.
- build.py / lib-rt/setup.py: drop codepoint_extra_ops.c from the
librt.strings module sources. The _extra_ops.c files are mypyc-internal
(linked into user extensions via SourceDep in mypyc/ir/deps.py); the
librt.strings Python module shouldn't depend on them, matching how
bytes_extra_ops, str_extra_ops, etc. are organized. librt.strings now
picks up LibRTStrings_IsIdentifier via #include of the header.
This comment has been minimized.
This comment has been minimized.
Per follow-up review on python#21522, the codepoint classifiers belong with the rest of the librt.strings public surface rather than in a shared inline header, since they implement Python-visible librt.strings functions (not mypyc-internal codegen helpers like the other _extra_ops files). Move them out of codepoint_extra_ops.h and into librt_strings.c as proper non-static functions, exposed to mypyc-compiled callers via the capsule API the same way every other LibRTStrings_* function works. This keeps the librt module files independent of mypyc-internal _extra_ops headers, matching the pattern used by BytesWriter_internal etc. The cost is one indirect call per use vs. the previous inlined macro; still substantially faster than the Python method dispatch the primitives are replacing. - librt_strings.h: bump LIBRT_STRINGS_API_VERSION 4->5, LIBRT_STRINGS_API_LEN 14->19. - librt_strings_api.h: add 5 macro entries for LibRTStrings_API[14..18]. - librt_strings.c: define the 5 helpers; register them in the capsule table; drop `#include "codepoint_extra_ops.h"`. - mypyc/ir/deps.py: delete CODEPOINT_EXTRA_OPS. - mypyc/primitives/librt_strings_ops.py: drop the CODEPOINT_EXTRA_OPS dep from the five codepoint primitives. - Delete codepoint_extra_ops.{c,h}.
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
Per the follow-up review on python#21522 (discussion_r3302940360), the codepoint helpers should be defined `static inline` in librt_strings.h rather than routed through the capsule API. The librt_strings.h header is already pulled into both the librt.strings module and every mypyc- compiled extension that uses any librt.strings primitive (via librt_strings_api.h), so static inlines there are visible to both sides with no per-call indirection -- which matters here because the helpers participate in per-character hot loops where the capsule call overhead would dwarf the work of a single Py_UNICODE_IS* macro. This supersedes the capsule approach from the previous commit: - librt_strings.h: include CPy.h and stdint.h; define the five classification helpers (IsSpace, IsDigit, IsAlnum, IsAlpha, IsIdentifier) as `static inline`; revert LIBRT_STRINGS_API_VERSION 5 -> 4 and _API_LEN 19 -> 14. - librt_strings_api.h: drop the five capsule macro entries. - librt_strings.c: drop the five out-of-line function bodies and the matching capsule table entries; the `cp_*` Python wrappers now reach the inlines via the header include.
Two i32 -> i32 case-conversion helpers mirroring the existing codepoint classifiers. ASCII fast paths inline (`a..z <-> A..Z` via add/sub 32); non-ASCII delegates to `str.upper` / `str.lower` on a 1-character string via a shared LibRTStrings_ChangeCase_slow helper. When uppercasing or lowercasing expands to multiple codepoints (e.g. 'ß'.upper() == 'SS', 'fi'.upper() == 'FI'), the helper returns the input unchanged so the signature stays i32 -> i32. Allocation failure aborts via CPyError_OutOfMemory, matching how LibRTStrings_IsIdentifier handles OOM and keeping the helpers ERR_NEVER. Following the inline-in-header pattern landed for isidentifier (python#21522), the bodies live as `static inline` in librt_strings.h so they compile directly into both the librt.strings module and mypyc-emitted code with no capsule indirection. Stack: depends on the librt.strings.isidentifier primitive (python#21522).
Recognizes the AST shape `IndexExpr(str) == StrLiteral` (and the symmetric `StrLiteral == IndexExpr(str)`, plus the `!=` variants) and lowers it to an int compare of codepoints reusing the existing CPyStr_GetItemUnsafeAsInt primitive. Today the pattern lowers to CPyStr_GetItem + CPyStr_EqualLiteral, which allocates or looks up a 1-character PyUnicode object per iteration and goes through a generic string-equality call. After specialization it becomes an inlined PyUnicode_READ plus an int compare -- about 4x faster on bench_str_compare with a 3-compares-per-iteration workload, and closer to ~9x with the more typical 1-compare-per-iteration shape. No annotations required; benefits any code that compares a string index against a 1-character literal. Multi-character / empty literals fall through to the generic path (which still correctly returns False). Bounds checking is preserved -- the helper raises IndexError for out-of-range indices, same as the unspecialized path. Stack: builds on the `ord(s[i])` primitive (python#20578) and the librt.strings codepoint helpers (python#21462, python#21504, python#21509, python#21521, python#21522, python#21553).
Recognizes the AST shape `IndexExpr(str) == StrLiteral` (and the symmetric `StrLiteral == IndexExpr(str)`, plus the `!=` variants) and lowers it to an int compare of codepoints reusing the existing CPyStr_GetItemUnsafeAsInt primitive. Today the pattern lowers to CPyStr_GetItem + CPyStr_EqualLiteral, which allocates or looks up a 1-character PyUnicode object per iteration and goes through a generic string-equality call. After specialization it becomes an inlined PyUnicode_READ plus an int compare -- about 4x faster on bench_str_compare with a 3-compares-per-iteration workload, and closer to ~9x with the more typical 1-compare-per-iteration shape. No annotations required; benefits any code that compares a string index against a 1-character literal. Multi-character / empty literals fall through to the generic path (which still correctly returns False). Bounds checking is preserved -- the helper raises IndexError for out-of-range indices, same as the unspecialized path. Stack: builds on the `ord(s[i])` primitive (python#20578) and the librt.strings codepoint helpers (python#21462, python#21504, python#21509, python#21521, python#21522, python#21553).
Recognizes the AST shape `IndexExpr(str) == StrLiteral` (and the symmetric `StrLiteral == IndexExpr(str)`, plus the `!=` variants) and lowers it to an int compare of codepoints reusing the existing CPyStr_GetItemUnsafeAsInt primitive. Today the pattern lowers to CPyStr_GetItem + CPyStr_EqualLiteral, which allocates or looks up a 1-character PyUnicode object per iteration and goes through a generic string-equality call. After specialization it becomes an inlined PyUnicode_READ plus an int compare -- about 4x faster on bench_str_compare with a 3-compares-per-iteration workload, and closer to ~9x with the more typical 1-compare-per-iteration shape. No annotations required; benefits any code that compares a string index against a 1-character literal. Multi-character / empty literals fall through to the generic path (which still correctly returns False). Bounds checking is preserved -- the helper raises IndexError for out-of-range indices, same as the unspecialized path. Stack: builds on the `ord(s[i])` primitive (python#20578) and the librt.strings codepoint helpers (python#21462, python#21504, python#21509, python#21521, python#21522, python#21553).
## Mypy 2.2
### Support for Closed TypedDicts (PEP 728)
Mypy now supports closed TypedDicts as specified in PEP 728. A closed TypedDict cannot have extra
keys beyond those explicitly defined. This allows the type checker to determine that certain
operations are safe when they otherwise wouldn't be due to the potential presence of unknown keys.
You can use the `closed` keyword argument with `TypedDict`:
```python
HasName = TypedDict("HasName", {"name": str})
HasOnlyName = TypedDict("HasOnlyName", {"name": str}, closed=True)
Movie = TypedDict("Movie", {"name": str, "year": int})
movie: Movie = {"name": "Nimona", "year": 2023}
has_name: HasName = movie # OK: HasName is open (default)
has_only_name: HasOnlyName = movie # Error: HasOnlyName is closed and Movie has extra "year" key
```
Closed TypedDicts enable more precise type checking because the type checker knows exactly which
keys are present. This is particularly useful when working with TypedDict unions or when you want
to ensure that a TypedDict conforms to an exact shape.
The `closed` keyword also enables safe type narrowing with `in` checks:
```python
Book = TypedDict('Book', {'book': str}, closed=True)
DVD = TypedDict('DVD', {'dvd': str}, closed=True)
type Inventory = Book | DVD
def print_type(inventory: Inventory) -> None:
if "book" in inventory:
# Type is narrowed to Book here - safe because DVD is closed
print(inventory["book"])
else:
# Type is narrowed to DVD here
print(inventory["dvd"])
```
The `closed` keyword is also supported in class-based syntax:
```python
class HasOnlyName(TypedDict, closed=True):
name: str
```
Note that closed TypedDicts are structural types, so a closed TypedDict is assignable to an open
TypedDict with the same keys, but not vice versa.
Contributed by Alice (PR [21382](python/mypy#21382)).
### Complete Support for Type Variable Defaults (PEP 696)
Mypy now has complete support for type variable defaults as specified in PEP 696. This allows you to
specify default values for type parameters in generic classes, functions, and type aliases.
Traditional syntax (Python 3.11 and earlier):
```python
T = TypeVar("T", default=int) # This means that if no type is specified T = int
@DataClass
class Box(Generic[T]):
value: T | None = None
reveal_type(Box()) # type is Box[int]
reveal_type(Box(value="Hello World!")) # type is Box[str]
```
New syntax (Python 3.12+):
```python
class Box[T = int]:
def __init__(self, value: T) -> None:
self.value = value
reveal_type(Box()) # type is Box[int]
reveal_type(Box(value="Hello World!")) # type is Box[str]
```
Type variable defaults work with all forms of generics, including classes, functions, and type aliases.
This release completes the implementation by fixing various edge cases involving recursive defaults,
dependencies between type variables, and interactions with variadic generics.
Contributed by Ivan Levkivskyi (PRs [21491](python/mypy#21491),
[21526](python/mypy#21526), [21544](python/mypy#21544)).
### Respect Explicit Return Type of `__new__()`
Mypy now respects explicitly annotated return types in `__new__()` methods. Previously, mypy would
always assume that `__new__()` returns an instance of the current class, ignoring explicit annotations.
With this change, if you explicitly annotate a return type that differs from the implicit type, mypy
will use the explicit annotation:
```python
class Factory:
def __new__(cls) -> Product:
return Product()
reveal_type(Factory()) # type is Product, not Factory
```
Note that mypy still gives an error at the definition site if the explicit annotation is not a
subtype of the current class, since this is technically not type-safe.
For backwards compatibility, there are two exceptions:
- If the return type is `Any`, mypy will still use the current class as the return type.
- If the explicit return type comes from a superclass and is a supertype of the implicit return type,
mypy will use the implicit (more specific) type:
```python
class A:
def __new__(cls) -> A: ...
reveal_type(A()) # type is A
class B:
def __new__(cls) -> B:
return cls()
class C(B): ...
reveal_type(C()) # type is C
```
This fixes several long-standing issues where explicit `__new__()` return types were ignored.
Contributed by Ivan Levkivskyi (PR [21441](python/mypy#21441)).
### TypeForm Support No Longer Experimental
Support for `TypeForm` is no longer experimental. `TypeForm` (introduced in Python 3.14) allows you
to annotate parameters that accept type expressions, providing better type checking for functions
that work with types as values.
```python
from typing import TypeForm
def make_list(tp: TypeForm[T]) -> list[T]:
...
# Correctly typed as list[int]
int_list = make_list(int)
```
`TypeForm` support was previously reverted from mypy 2.1 due to a performance regression, but this
has now been mitigated.
Contributed by Ivan Levkivskyi and Jelle Zijlstra (PRs [21262](python/mypy#21262), [21591](python/mypy#21591),
[21459](python/mypy#21459)).
### Experimental WASM Wheel for Python 3.14
Mypy now ships an experimental WebAssembly (WASM) wheel for Python 3.14. This allows mypy to run
in WASM environments such as Pyodide and browser-based Python implementations.
The WASM wheel is considered experimental and may have limitations compared to native builds. Please
report any issues you encounter when using mypy in WASM environments.
Contributed by Ivan Levkivskyi (PR [21671](python/mypy#21671)).
### Mypyc Free-threading Improvements
- Make function wrappers thread-safe on free-threaded builds (Jukka Lehtosalo, PR [21620](python/mypy#21620))
- Make list remove and index thread-safe on free-threaded builds (Jukka Lehtosalo, PR [21614](python/mypy#21614))
- Fix dict iteration memory safety on free-threaded builds (Jukka Lehtosalo, PR [21617](python/mypy#21617))
- Make some dict primitives thread-safe on free-threading builds (Jukka Lehtosalo, PR [21616](python/mypy#21616))
- Fix free-threading race condition in argument parsing (Jukka Lehtosalo, PR [21613](python/mypy#21613))
- Document free threading and other doc updates (Jukka Lehtosalo, PR [21494](python/mypy#21494))
### `librt.strings` Updates
- Add `librt.strings.toupper` and `librt.strings.tolower` codepoint primitives (Vaggelis Danias, PR [21553](python/mypy#21553))
- Add `librt.strings.isidentifier` codepoint primitive (Vaggelis Danias, PR [21522](python/mypy#21522))
- Add `librt.strings.isalpha` codepoint primitive (Vaggelis Danias, PR [21521](python/mypy#21521))
- Add `librt.strings.isalnum` codepoint primitive (Vaggelis Danias, PR [21509](python/mypy#21509))
- Add `librt.strings.isdigit` codepoint primitive (Vaggelis Danias, PR [21504](python/mypy#21504))
- Add `librt.strings.isspace` char primitive (Vaggelis Danias, PR [21462](python/mypy#21462))
### Mypyc Improvements
- Fix name lookup when class var and module var have the same name (Jukka Lehtosalo, PR [21594](python/mypy#21594))
- Report file and line number on uncaught exceptions (Jukka Lehtosalo, PR [21584](python/mypy#21584))
- Use `other` arg instead of `self` for RHS type (Ryan Heard, PR [21569](python/mypy#21569))
- Use `method_sig` to get the method signature (Ryan Heard, PR [21567](python/mypy#21567))
- Preserve inherited attribute defaults under `separate=True` (Jo, PR [21547](python/mypy#21547))
- Fix missing cross-group header deps in incremental builds (Jo, PR [21490](python/mypy#21490))
- Fix cross-group call to inherited `__mypyc_defaults_setup` (Jo, PR [21481](python/mypy#21481))
- Fix non-deterministic class struct layout under `separate=True` (Vaggelis Danias, PR [21530](python/mypy#21530))
- Specialize `s[i] == 'x'` to a codepoint int compare (Vaggelis Danias, PR [21579](python/mypy#21579))
- Fix reference leak in mypyc bytes concatenation (Colinxu2020, PR [21469](python/mypy#21469))
### Fixes to Crashes
- Fix crash on invalid recursive variadic alias (Ivan Levkivskyi, PR [21572](python/mypy#21572))
- Fix crashes on variadic unpacking in synthetic types (Ivan Levkivskyi, PR [21555](python/mypy#21555))
- Fix crash on unhandled meet variadic tuple vs instance (Ivan Levkivskyi, PR [21558](python/mypy#21558))
- Fix crash on deferred generic class nested in function (Ivan Levkivskyi, PR [21557](python/mypy#21557))
- Fix crash in new-style type alias with variadic unpack (Ivan Levkivskyi, PR [21551](python/mypy#21551))
- Fix various crashes on recursive type variable defaults (Ivan Levkivskyi, PR [21491](python/mypy#21491))
- Fix crash for empty `Annotated` type application (Rayan Salhab, PR [21503](python/mypy#21503))
- Fix crash on `Unpack` used without arguments in class bases (Sai Asish Y, PR [21470](python/mypy#21470))
### Performance Improvements
- Memoize the options snapshot (Kevin Kannammalil, PR [21354](python/mypy#21354))
- Don't include `not_ready_deps` tracking as relating to mypy internals (Kevin Kannammalil, PR [21389](python/mypy#21389))
- Speed up transitive dependency hash for singleton SCCs (Kevin Kannammalil, PR [21390](python/mypy#21390))
- Optimize typeform checks (Jelle Zijlstra, PR [21459](python/mypy#21459))
### Improvements to the Native Parser
- Support `--shadow-file` with `--native-parser` (Jukka Lehtosalo, PR [21623](python/mypy#21623))
- Add Python version checks to native parser (Kevin Kannammalil, PR [21539](python/mypy#21539))
- Allow nativeparse to parse source code directly (bzoracler, PR [21260](python/mypy#21260))
### Other Notable Fixes and Improvements
- Add function definition notes for `too many positional arguments` errors (Kevin Kannammalil, PR [21410](python/mypy#21410))
- Fix the exportjson tool (.ff cache to .json conversion) (Jukka Lehtosalo, PR [21628](python/mypy#21628))
- Support floats in JSON in fixed-format cache (Ivan Levkivskyi, PR [21603](python/mypy#21603))
- Update `TypedDictType.__init__` signature to preserve backward compat (Jukka Lehtosalo, PR [21590](python/mypy#21590))
- Fix constructor calls for union-bounded `TypeVar`s (Jingchen Ye, PR [21571](python/mypy#21571))
- Fix `TypedDict` indexing with literal keys in comprehensions (Jingchen Ye, PR [21556](python/mypy#21556))
- Correctly handle empty tuple index when unpacked (Ivan Levkivskyi, PR [21545](python/mypy#21545))
- Support protocol checks for self-types in tuple types (Ivan Levkivskyi, PR [21535](python/mypy#21535))
- Fix edge cases in variadic tuple subclasses (Ivan Levkivskyi, PR [21518](python/mypy#21518))
- Special-case constructor for tuple types (Ivan Levkivskyi, PR [21502](python/mypy#21502))
- Fix false positive "Expected TypedDict key to be string literal" for `Union[TypedDict, dict[K, V]]` (Zakir Jiwani, PR [21511](python/mypy#21511))
- Use explicit `Never` for type inference (Ivan Levkivskyi, PR [21497](python/mypy#21497))
- Narrow membership in statically known containers (Shantanu, PR [21461](python/mypy#21461))
- Improve negative narrowing for membership checks on tuples (Shantanu, PR [21456](python/mypy#21456))
- Analyze `TypedDict` decorators (Pranav Manglik, PR [21267](python/mypy#21267))
- Start testing Python 3.15 (Marc Mueller, PR [21439](python/mypy#21439))
- Improved handling of `NamedTuple`, `TypedDict`, `Enum`, and regular classes nested in functions (Ivan Levkivskyi, PR [21478](python/mypy#21478))
5th PR of #21418