Skip to content

ElementTree.parse() fails with "not well-formed" for declared encoding "utf8" #148821

Closed

Description

@lemon24

Bug report

Bug description:

ElementTree.parse() fails with "not well-formed (invalid token)" for wrong declared encoding utf8 iif the XML contains non-ASCII characters, instead of "unknown encoding"; more details after repro:

Repro:

import io, xml.etree.ElementTree as ET

s = """\
<?xml version='1.0' encoding='utf8'?>
<outline text="Comentário" />
"""

def parse(s):
    return ET.parse(io.BytesIO(s.encode()))
    
parse(s)

Output:

>>> parse(s)
Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    parse(s)
  File "<python-input-2>", line 2, in parse
    return ET.parse(io.BytesIO(s.encode()))
  File "/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/xml/etree/ElementTree.py", line 1214, in parse
    tree.parse(source, parser)
  File "/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/xml/etree/ElementTree.py", line 577, in parse
    self._root = parser._parse_whole(source)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 2, column 21
>>>
>>> # utf-8 works fine
>>> parse(s.replace('utf8', 'utf-8')).getroot().get('text')
'Comentário'
>>>
>>> # ascii-only characters work fine, despite the wrong utf8 declared encoding
>>> parse(s.replace('á', 'a')).getroot().get('text')
'Comentario'
>>>
>>> # a truly unknown encoding fails with the correct message
>>> parse(s.replace('utf8', 'xyz'))
Traceback (most recent call last):
  ...
LookupError: unknown encoding: xyz
>>>
>>> # ascii encoding fails with the same message as utf8
>>> # (perhaps utf8 silently falls back to ascii?)
>>> parse(s.replace('utf8', 'ascii'))
Traceback (most recent call last):
  ...
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 2, column 21

Per the XML spec and IANA character sets list, the correct (and only) encoding name is utf-8 (works fine with etree).

Whether to accept utf8 was discussed previously in #46531, which was closed as won't fix (but in that issue, the error message was "unknown encoding", so the current message is a regression); FWIW, LXML does accept utf8 as a valid encoding.

Expected behavior:

  • utf8 encoding fails with "unknown encoding", regardless of whether the input contains non-ASCII characters or not ("in the face of ambiguity, refuse the temptation to guess"), or
  • treat utf8 as utf-8, even if it's not actually correct (str.encode() and LXML supporting it seems to indicate it is a common (mis)spelling)

LXML behavior, for reference:

>>> import lxml.etree as ET
>>> 
>>> # lxml accepts wrong encoding utf8
>>> parse(s).getroot().get('text')
'Comentário'
>>>
>>> # unknown encoding fails as expected
>>> parse(s.replace('utf8', 'xyz'))
Traceback (most recent call last):
  ...
lxml.etree.XMLSyntaxError: Unsupported encoding: xyz, line 1, column 35

CPython versions tested on:

3.14, 3.13, 3.12

Operating systems tested on:

macOS, Linux

Linked PRs

Activity

  1. changed the title [-]ElementTree.parse() fails with "not well-formed" for encoding "utf8"[/-] [+]ElementTree.parse() fails with "not well-formed" for declared encoding "utf8"[/+] on Apr 21, 2026
  2. serhiy-storchaka commented on Apr 22, 2026

    @serhiy-storchaka
    Member

    This is related to #62259.

    Besides the built in set, the parser only supports single-byte encodings. It simply tries to decode bytes(range(256)) and use the result as a byte-to-character mapping. It fails to detect that "utf8" is a multi-byte encoding.

    We can improve detection of multi-byte encodings by using the incremental decoder with every single undecodable byte. If the result is an empty string, the encoding is multi-byte. But the code would be complex, and it will slow down initialization for some encodings (for example for alternative names of ASCII).

    Supporting multi-byte encodings would be even more complex.

    The simplest way is to add special predefined flag to CodecInfo objects to denote single-byte and multi-byte encodings. But this is complicated too.

  3. IvyXu420 commented on Apr 22, 2026

    @IvyXu420
    Contributor

    Yes I agree on that. To fix this issue is quite complex and I haven’t found an easy solution either.

  4. serhiy-storchaka commented on Apr 23, 2026

    @serhiy-storchaka
    Member

    #148911 adds optional private attribute _is_single_byte=False for standard multi-byte text codecs. This is similar as setting _is_text_encoding=False for standard non-text codecs (see #63818). This allows to blacklist them in pyexpact.

    Note that it is not added for some multi-byte codecs ("mbcs", "oem", "cpXXX" fallbacks on Windows) because it is not so easy to determine if the codec is multi-byte. Maybe we will add blacklists and whitelists for "cpXXX" fallbacks in future.

    Minor design questions:

    • Should we also add _is_single_byte=True for standard single-byte codecs? There are much more single-byte codecs (>70) than multi-byte codecs, but this will allow to differenciate them from codecs with undefined attribute like "mbcs".
    • Would not be better to name it _is_multi_byte=True instead of _is_single_byte=False?

    cc @ncoghlan, @malemburg

  5. added 9 commits that reference this issue on May 13, 2026
  6. serhiy-storchaka commented on May 30, 2026

    @serhiy-storchaka
    Member

    #148911 was superseded by #149860.

    Now non-canonical encoding names like utf8 are supported, but only of BMP characters (U+0000 - U+FFFF). This is a limitation of Expat.

  7. added
    3.16new features, bugs and security fixes
    and removed
    3.13only security fixes
    3.14bugs and security fixes
    3.15bugs and security fixes
    on May 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

3.16new features, bugs and security fixesextension-modulesC modules in the Modules dirtopic-XMLtype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions