Skip to content

gh-79478: Update untokenize.rst documentation to reflect current implementation #150225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ByteFlowing1337 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

gh-79478: Update untokenize.rst documentation to reflect current implementation #150225

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
27239f2
Update `untokenize.rst` documentation to reflect current implementation
ByteFlowing1337 May 22, 2026
b3ae1b8
Address review
ByteFlowing1337 Jul 8, 2026
49c39d6
Polish the doc
ByteFlowing1337 Jul 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 21 additions & 8 deletions Doc/library/tokenize.rst
View file
Open in desktop

@robsdedude robsdedude Jun 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the docs could be more explicit about whether the iterator may or may not mix 2- and 5-element sequences.

@ByteFlowing1337 ByteFlowing1337 Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a para describing the behavior when they are mixed. :)

Original file line number Diff line number Diff line change
Expand Up @@ -84,14 +84,27 @@ write back the modified script.

.. function:: untokenize(iterable)

Converts tokens back into Python source code. The *iterable* must return
sequences with at least two elements, the token type and the token string.
Any additional sequence elements are ignored.

The result is guaranteed to tokenize back to match the input so that the
conversion is lossless and round-trips are assured. The guarantee applies
only to the token type and token string as the spacing between tokens
(column positions) may change.
Converts tokens back into Python source code.

The *iterable* should return sequences with either two or five elements:

- If *iterable* returns sequences with two elements ``(type, string)``,
the result will tokenize back to the same token types and strings as
the input, but the resulting output is poor.

- If *iterable* returns sequences with five elements
``(type, string, start, end, line)``,
the result will tokenize back to match the input as closely as possible
(tab characters may be replaced with spaces).

Comment thread
ByteFlowing1337 marked this conversation as resolved.
If *iterable* mixes two- and five-element sequences,
as soon as a two-element sequence is encountered, the extra elements
``(start, end, line)`` of any later five-element sequences are ignored too.

The result is guaranteed to tokenize back to match the input so
that the conversion is lossless and round-trips are assured.
The guarantee applies only to the token type and token string as
the spacing between tokens (column positions) may change.

It returns bytes, encoded using the :data:`~token.ENCODING` token, which
is the first token sequence output by :func:`.tokenize`. If there is no
Expand Down
Loading