Skip to content

tokenize: _get_col_offsets can pass a negative byte offset to _PyPegen_byte_offset_to_character_offset_line (1-byte heap over-read) #162142

Open

Description

@Nitjsefnie

Bug report

_get_col_offsets in Python/Python-tokenize.c passes byte_offset == -1 to _PyPegen_byte_offset_to_character_offset_line when a token starts before the current line's start, and that function then reads data[-1], one byte before the line's UTF-8 buffer. An INDENT token does this when the tokenizer still holds earlier lines in its buffer, because INDENT starts at the buffer start (p_start = tok->buf in 3.15.1, tok->buf_offset on main).

On 3.15.1 (before gh-160268) this input reaches it:

import io, tokenize

src = 'f"{)\n é\n'
for tok in tokenize.generate_tokens(io.StringIO(src).readline):
    print(tok)

What a release build prints depends on the byte before the buffer. When that byte is ASCII:

TokenInfo(type=5 (INDENT), string='f"{)\n ', start=(2, -1), end=(2, 1), line=' é\n')
TokenInfo(type=1 (NAME), string='é', start=(2, -3), end=(2, -2), line=' é\n')

When it is a continuation byte (0x80-0xBF) or 0xF8-0xFF, ValueError: Invalid UTF-8 sequence is set and the -1 return is not checked, so it surfaces as SystemError: ... returned a result with an exception set. Filling the heap with array.array('d', [-1.0, -1.0]) objects and deleting every other one before running the snippet triggers this on 3.12.13, 3.13.14, 3.14.6 and 3.15.1.

An AddressSanitizer build of v3.15.1 (PYTHONMALLOC=malloc) reports:

ERROR: AddressSanitizer: heap-buffer-overflow ... READ of size 1
    #0 in _PyPegen_byte_offset_to_character_offset_line Parser/pegen.c:33
    #1 in _get_col_offsets Python/Python-tokenize.c:229
    #2 in tokenizeriter_next Python/Python-tokenize.c:304
... located 1 bytes before 5-byte region (allocated by unicode_fill_utf8)

A pydebug build aborts with Assertion `(res != NULL) ^ (PyErr_Occurred() != NULL)' failed in Python/ceval.c.

gh-160268 removed this trigger: the mismatched bracket is now a TokenError on main, 3.15.2 and 3.14.10. Since gh-153569, main also checks the -1 return, so the SystemError and the assertion are gone there. The one-byte read is still possible: the same-line fast path still runs without byte_offset >= 0, INDENT still starts at the buffer start, and _PyPegen_byte_offset_to_character_offset_line has no bounds check. I found no input that reaches it on main (600,000 generated inputs under ASan), so this is a hardening request. With this guard applied to v3.15.1, the repro and 30,000 fuzz iterations run clean under ASan (INDENT's own start column stays at -1; that is the lexer side):

-        if (lineno == end_lineno) {
+        if (lineno == end_lineno && byte_offset >= 0) {

CPython versions tested on:

3.12, 3.13, 3.14, 3.15, CPython main branch

Operating systems tested on:

Linux

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions