You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
tokenize: _get_col_offsets can pass a negative byte offset to _PyPegen_byte_offset_to_character_offset_line (1-byte heap over-read) #162142
_get_col_offsets in Python/Python-tokenize.c passes byte_offset == -1 to _PyPegen_byte_offset_to_character_offset_line when a token starts before the current line's start, and that function then reads data[-1], one byte before the line's UTF-8 buffer. An INDENT token does this when the tokenizer still holds earlier lines in its buffer, because INDENT starts at the buffer start (p_start = tok->buf in 3.15.1, tok->buf_offset on main).
On 3.15.1 (before gh-160268) this input reaches it:
When it is a continuation byte (0x80-0xBF) or 0xF8-0xFF, ValueError: Invalid UTF-8 sequence is set and the -1 return is not checked, so it surfaces as SystemError: ... returned a result with an exception set. Filling the heap with array.array('d', [-1.0, -1.0]) objects and deleting every other one before running the snippet triggers this on 3.12.13, 3.13.14, 3.14.6 and 3.15.1.
An AddressSanitizer build of v3.15.1 (PYTHONMALLOC=malloc) reports:
ERROR: AddressSanitizer: heap-buffer-overflow ... READ of size 1
#0 in _PyPegen_byte_offset_to_character_offset_line Parser/pegen.c:33
#1 in _get_col_offsets Python/Python-tokenize.c:229
#2 in tokenizeriter_next Python/Python-tokenize.c:304
... located 1 bytes before 5-byte region (allocated by unicode_fill_utf8)
A pydebug build aborts with Assertion `(res != NULL) ^ (PyErr_Occurred() != NULL)' failed in Python/ceval.c.
gh-160268 removed this trigger: the mismatched bracket is now a TokenError on main, 3.15.2 and 3.14.10. Since gh-153569, main also checks the -1 return, so the SystemError and the assertion are gone there. The one-byte read is still possible: the same-line fast path still runs without byte_offset >= 0, INDENT still starts at the buffer start, and _PyPegen_byte_offset_to_character_offset_line has no bounds check. I found no input that reaches it on main (600,000 generated inputs under ASan), so this is a hardening request. With this guard applied to v3.15.1, the repro and 30,000 fuzz iterations run clean under ASan (INDENT's own start column stays at -1; that is the lexer side):
- if (lineno == end_lineno) {+ if (lineno == end_lineno && byte_offset >= 0) {
Activity