Repository navigation
Closed
Labels
3.16new features, bugs and security fixesnew features, bugs and security fixesextension-modulesC modules in the Modules dirC modules in the Modules dirtopic-XMLtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
Bug report
Bug description:
ElementTree.parse() fails with "not well-formed (invalid token)" for wrong declared encoding utf8 iif the XML contains non-ASCII characters, instead of "unknown encoding"; more details after repro:
Repro:
import io, xml.etree.ElementTree as ET
s = """\
<?xml version='1.0' encoding='utf8'?>
<outline text="Comentário" />
"""
def parse(s):
return ET.parse(io.BytesIO(s.encode()))
parse(s)Output:
>>> parse(s)
Traceback (most recent call last):
File "<python-input-3>", line 1, in <module>
parse(s)
File "<python-input-2>", line 2, in parse
return ET.parse(io.BytesIO(s.encode()))
File "/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/xml/etree/ElementTree.py", line 1214, in parse
tree.parse(source, parser)
File "/Library/Frameworks/Python.framework/Versions/3.14/lib/python3.14/xml/etree/ElementTree.py", line 577, in parse
self._root = parser._parse_whole(source)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 2, column 21
>>>
>>> # utf-8 works fine
>>> parse(s.replace('utf8', 'utf-8')).getroot().get('text')
'Comentário'
>>>
>>> # ascii-only characters work fine, despite the wrong utf8 declared encoding
>>> parse(s.replace('á', 'a')).getroot().get('text')
'Comentario'
>>>
>>> # a truly unknown encoding fails with the correct message
>>> parse(s.replace('utf8', 'xyz'))
Traceback (most recent call last):
...
LookupError: unknown encoding: xyz
>>>
>>> # ascii encoding fails with the same message as utf8
>>> # (perhaps utf8 silently falls back to ascii?)
>>> parse(s.replace('utf8', 'ascii'))
Traceback (most recent call last):
...
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 2, column 21Per the XML spec and IANA character sets list, the correct (and only) encoding name is utf-8 (works fine with etree).
Whether to accept utf8 was discussed previously in #46531, which was closed as won't fix (but in that issue, the error message was "unknown encoding", so the current message is a regression); FWIW, LXML does accept utf8 as a valid encoding.
Expected behavior:
utf8encoding fails with "unknown encoding", regardless of whether the input contains non-ASCII characters or not ("in the face of ambiguity, refuse the temptation to guess"), or- treat
utf8asutf-8, even if it's not actually correct (str.encode() and LXML supporting it seems to indicate it is a common (mis)spelling)
LXML behavior, for reference:
>>> import lxml.etree as ET
>>>
>>> # lxml accepts wrong encoding utf8
>>> parse(s).getroot().get('text')
'Comentário'
>>>
>>> # unknown encoding fails as expected
>>> parse(s.replace('utf8', 'xyz'))
Traceback (most recent call last):
...
lxml.etree.XMLSyntaxError: Unsupported encoding: xyz, line 1, column 35CPython versions tested on:
3.14, 3.13, 3.12
Operating systems tested on:
macOS, Linux
Linked PRs
- gh-148821: Always reject known multi-byte encodings in pyexpat #148911
- gh-148821: Add more strict tests for XML encodings #149765
- [3.15] gh-148821: Add more strict tests for XML encodings (GH-149765) #149770
- [3.14] gh-148821: Add more strict tests for XML encodings (GH-149765) #149771
- [3.13] gh-148821: Add more strict tests for XML encodings (GH-149765) #149772
- gh-148821: Add more tests for invalid XML encodings #149820
- [3.15] gh-148821: Add more tests for invalid XML encodings (GH-149820) #149821
- [3.14] gh-148821: Add more tests for invalid XML encodings (GH-149820) #149822
- [3.13] gh-148821: Add more tests for invalid XML encodings (GH-149820) #149823
Metadata
Metadata
Assignees
Labels
3.16new features, bugs and security fixesnew features, bugs and security fixesextension-modulesC modules in the Modules dirC modules in the Modules dirtopic-XMLtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Activity
[-]ElementTree.parse() fails with "not well-formed" for encoding "utf8"[/-][+]ElementTree.parse() fails with "not well-formed" for declared encoding "utf8"[/+]serhiy-storchaka commentedon Apr 22, 2026
This is related to #62259.
Besides the built in set, the parser only supports single-byte encodings. It simply tries to decode
bytes(range(256))and use the result as a byte-to-character mapping. It fails to detect that "utf8" is a multi-byte encoding.We can improve detection of multi-byte encodings by using the incremental decoder with every single undecodable byte. If the result is an empty string, the encoding is multi-byte. But the code would be complex, and it will slow down initialization for some encodings (for example for alternative names of ASCII).
Supporting multi-byte encodings would be even more complex.
The simplest way is to add special predefined flag to CodecInfo objects to denote single-byte and multi-byte encodings. But this is complicated too.
IvyXu420 commentedon Apr 22, 2026
Yes I agree on that. To fix this issue is quite complex and I haven’t found an easy solution either.
serhiy-storchaka commentedon Apr 23, 2026
#148911 adds optional private attribute
_is_single_byte=Falsefor standard multi-byte text codecs. This is similar as setting_is_text_encoding=Falsefor standard non-text codecs (see #63818). This allows to blacklist them inpyexpact.Note that it is not added for some multi-byte codecs ("mbcs", "oem", "cpXXX" fallbacks on Windows) because it is not so easy to determine if the codec is multi-byte. Maybe we will add blacklists and whitelists for "cpXXX" fallbacks in future.
Minor design questions:
_is_single_byte=Truefor standard single-byte codecs? There are much more single-byte codecs (>70) than multi-byte codecs, but this will allow to differenciate them from codecs with undefined attribute like "mbcs"._is_multi_byte=Trueinstead of_is_single_byte=False?cc @ncoghlan, @malemburg
pythongh-148821: Add more strict tests for XML encodings
gh-148821: Add more strict tests for XML encodings (GH-149765)
[3.13] gh-148821: Add more strict tests for XML encodings (GH-149765) (…
[3.14] gh-148821: Add more strict tests for XML encodings (GH-149765) (…
[3.15] gh-148821: Add more strict tests for XML encodings (GH-149765) (…
gh-148821: Add more tests for invalid XML encodings (GH-149820)
[3.13] gh-148821: Add more tests for invalid XML encodings (GH-149820) (
[3.14] gh-148821: Add more tests for invalid XML encodings (GH-149820) (
[3.15] gh-148821: Add more tests for invalid XML encodings (GH-149820) (
serhiy-storchaka commentedon May 30, 2026
#148911 was superseded by #149860.
Now non-canonical encoding names like utf8 are supported, but only of BMP characters (U+0000 - U+FFFF). This is a limitation of Expat.