From 27239f276a22f25de0d8a3a8ece6a47f4d4f2828 Mon Sep 17 00:00:00 2001 From: Ying Xu Date: Fri, 22 May 2026 16:46:52 +0800 Subject: [PATCH 1/3] Update `untokenize.rst` documentation to reflect current implementation --- Doc/library/tokenize.rst | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 2eea51734fde03c..77b61e6976d6c62 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -85,17 +85,23 @@ write back the modified script. .. function:: untokenize(iterable) Converts tokens back into Python source code. The *iterable* must return - sequences with at least two elements, the token type and the token string. - Any additional sequence elements are ignored. + sequences with either two or five elements. The result is guaranteed to tokenize back to match the input so that the - conversion is lossless and round-trips are assured. The guarantee applies - only to the token type and token string as the spacing between tokens - (column positions) may change. + conversion is lossless and round-trips are assured. + + If *iterable* returns sequences with two elements (the token type and token + string), the result will tokenize back to the same token types and strings as + the input, but the spacing between tokens (column positions) may change. + + If *iterable* returns sequences with five elements + (``type token string start end line``), the column positions are preserved + and the result will tokenize back to match the input exactly. + It returns bytes, encoded using the :data:`~token.ENCODING` token, which is the first token sequence output by :func:`.tokenize`. If there is no - encoding token in the input, it returns a str instead. + encoding token in the input, it returns a :class:`str` instead. :func:`.tokenize` needs to detect the encoding of source files it tokenizes. The From b3ae1b8aa84bd30b1cc7a4605cb17cf2e45242c8 Mon Sep 17 00:00:00 2001 From: Ivy Xu Date: Thu, 9 Jul 2026 00:27:54 +0800 Subject: [PATCH 2/3] Address review --- Doc/library/tokenize.rst | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 77b61e6976d6c62..b68ead8655ec28e 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -84,24 +84,25 @@ write back the modified script. .. function:: untokenize(iterable) - Converts tokens back into Python source code. The *iterable* must return - sequences with either two or five elements. + Converts tokens back into Python source code. - The result is guaranteed to tokenize back to match the input so that the - conversion is lossless and round-trips are assured. + The *iterable* should return sequences with either two or five elements: - If *iterable* returns sequences with two elements (the token type and token - string), the result will tokenize back to the same token types and strings as - the input, but the spacing between tokens (column positions) may change. + - If *iterable* returns sequences with two elements ``(type, string)``, + the result will tokenize back to the same token types and strings as + the input, but the spacing between tokens (column positions) may change. - If *iterable* returns sequences with five elements - (``type token string start end line``), the column positions are preserved - and the result will tokenize back to match the input exactly. + - If *iterable* returns sequences with five elements + ``(type, string, start, end, line)``, the column positions are preserved + and the result will tokenize back to match the input as closely as possible. + If *iterable* mixes two- and five-element sequences, + as soon as a two-element sequence is encountered, the extra elements + ``(start, end, line)`` of any later five-element sequences are ignored too. It returns bytes, encoded using the :data:`~token.ENCODING` token, which is the first token sequence output by :func:`.tokenize`. If there is no - encoding token in the input, it returns a :class:`str` instead. + encoding token in the input, it returns a str instead. :func:`.tokenize` needs to detect the encoding of source files it tokenizes. The From 49c39d6e2250a851a225a3170525381871a5b382 Mon Sep 17 00:00:00 2001 From: Ivy Xu Date: Thu, 9 Jul 2026 19:10:40 +0800 Subject: [PATCH 3/3] Polish the doc --- Doc/library/tokenize.rst | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index b68ead8655ec28e..dc45908ba955e6c 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -90,16 +90,22 @@ write back the modified script. - If *iterable* returns sequences with two elements ``(type, string)``, the result will tokenize back to the same token types and strings as - the input, but the spacing between tokens (column positions) may change. + the input, but the resulting output is poor. - If *iterable* returns sequences with five elements - ``(type, string, start, end, line)``, the column positions are preserved - and the result will tokenize back to match the input as closely as possible. + ``(type, string, start, end, line)``, + the result will tokenize back to match the input as closely as possible + (tab characters may be replaced with spaces). If *iterable* mixes two- and five-element sequences, as soon as a two-element sequence is encountered, the extra elements ``(start, end, line)`` of any later five-element sequences are ignored too. + The result is guaranteed to tokenize back to match the input so + that the conversion is lossless and round-trips are assured. + The guarantee applies only to the token type and token string as + the spacing between tokens (column positions) may change. + It returns bytes, encoded using the :data:`~token.ENCODING` token, which is the first token sequence output by :func:`.tokenize`. If there is no encoding token in the input, it returns a str instead.