Merge remote-tracking branch 'origin/main' into httplib-query-method-rfc-10008

Author: mbeijen

Doc/library/asyncio-graph.rst

@@ -4,7 +4,7 @@
 .. _asyncio-graph:
 
 ========================
-Call Graph Introspection
+Call graph introspection
 ========================
 
 **Source code:** :source:`Lib/asyncio/graph.py`
@@ -17,6 +17,12 @@ a suspended *future*.  These utilities and the underlying machinery
 can be used from within a Python program or by external profilers
 and debuggers.
 
+.. seealso::
+
+   :ref:`asyncio-introspection-tools`
+      Command-line tools for inspecting tasks in another running Python
+      process.
+
 .. versionadded:: 3.14
 
 

Doc/library/asyncio-tools.rst

@@ -0,0 +1,157 @@
+.. currentmodule:: asyncio
+
+.. _asyncio-introspection-tools:
+
+================================
+Command-line introspection tools
+================================
+
+**Source code:** :source:`Lib/asyncio/tools.py`
+
+-------------------------------------
+
+The :mod:`!asyncio` module can be invoked as a script via ``python -m
+asyncio`` to inspect the task graph of another running Python process without
+modifying it or restarting it.  The :mod:`!asyncio.tools` submodule implements
+this interface.
+
+The following commands inspect the process identified by ``PID``:
+
+.. code-block:: shell-session
+
+   $ python -m asyncio pstree [--retries N] PID
+   $ python -m asyncio ps [--retries N] PID
+
+The commands read the target process state without executing any code in it.
+They are only available on supported platforms and may require permission to
+inspect another process.  See the :ref:`permission-requirements <permission-requirements>` for details.
+
+.. seealso::
+
+   :ref:`asyncio-graph`
+      Programmatic APIs for inspecting the async call graph of a task or
+      future in the current process.
+
+The command examples below use this program, which creates a task hierarchy
+suitable for inspection and prints its process ID:
+
+.. code-block:: python
+   :caption: example.py
+
+   import asyncio
+   import os
+
+   async def play(track):
+       await asyncio.sleep(3600)
+       print(f"🎵 Finished: {track}")
+
+   async def album(name, tracks):
+       async with asyncio.TaskGroup() as tg:
+           for track in tracks:
+               tg.create_task(play(track), name=track)
+
+   async def main():
+       print(f"PID: {os.getpid()}")
+       async with asyncio.TaskGroup() as tg:
+           tg.create_task(
+               album("Sundowning", ["TNDNBTG", "Levitate"]),
+               name="Sundowning",
+           )
+           tg.create_task(
+               album("TMBTE", ["DYWTYLM", "Aqua Regia"]),
+               name="TMBTE",
+           )
+
+   asyncio.run(main())
+
+Run the program in one terminal and leave it running:
+
+.. code-block:: shell-session
+
+   $ python example.py
+   PID: 12345
+
+Then pass the printed process ID to the commands from another terminal.
+Thread IDs, task IDs, file paths, and line numbers vary between runs and
+source layouts.
+
+.. versionadded:: 3.14
+
+Command-line options
+====================
+
+.. option:: pstree PID
+
+   Display task and coroutine relationships as a tree.  Each task is shown
+   with its full coroutine stack, nested under the task (if any) that is
+   awaiting it.  This subcommand is useful for quickly identifying which branch
+   of a task hierarchy is blocked and where in its coroutine stack execution
+   has paused:
+
+   .. code-block:: shell-session
+
+      $ python -m asyncio pstree 12345
+      └── (T) Task-1
+          └──  main example.py:12
+              └──  TaskGroup.__aexit__ Lib/asyncio/taskgroups.py:75
+                  └──  TaskGroup._aexit Lib/asyncio/taskgroups.py:124
+                      ├── (T) Sundowning
+                      │   └──  album example.py:7
+                      │       └──  TaskGroup.__aexit__ Lib/asyncio/taskgroups.py:75
+                      │           └──  TaskGroup._aexit Lib/asyncio/taskgroups.py:124
+                      │               ├── (T) TNDNBTG
+                      │               │   └──  play example.py:4
+                      │               │       └──  sleep Lib/asyncio/tasks.py:702
+                      │               └── (T) Levitate
+                      │                   └──  play example.py:4
+                      │                       └──  sleep Lib/asyncio/tasks.py:702
+                      └── (T) TMBTE
+                          └──  album example.py:7
+                              └──  TaskGroup.__aexit__ Lib/asyncio/taskgroups.py:75
+                                  └──  TaskGroup._aexit Lib/asyncio/taskgroups.py:124
+                                      ├── (T) DYWTYLM
+                                      │   └──  play example.py:4
+                                      │       └──  sleep Lib/asyncio/tasks.py:702
+                                      └── (T) Aqua Regia
+                                          └──  play example.py:4
+                                              └──  sleep Lib/asyncio/tasks.py:702
+
+   If the await graph contains a cycle, ``pstree`` reports an error instead
+   of printing a tree.  A cycle in the await graph is unusual and typically
+   indicates a programming error:
+
+   .. code-block:: shell-session
+
+      $ python -m asyncio pstree 12345
+      ERROR: await-graph contains cycles - cannot print a tree!
+
+      cycle: Task-2 → Task-3 → Task-2
+
+.. option:: ps PID
+
+   Display a flat table of all pending tasks in the process *PID*.  Each row
+   shows the event-loop thread ID, task ID and name, coroutine stack, and the
+   awaiting task's stack, name, and ID, if any.
+
+   This subcommand prints all tasks regardless of whether the await graph
+   contains cycles:
+
+   .. code-block:: shell-session
+
+      $ python -m asyncio ps 12345
+      tid        task id              task name            coroutine stack                                    awaiter chain                                      awaiter name    awaiter id
+      ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+      18445801   0x10a456060          Task-1               TaskGroup._aexit -> TaskGroup.__aexit__ -> main                                                                       0x0
+      18445801   0x10a439f60          Sundowning           TaskGroup._aexit -> TaskGroup.__aexit__ -> album   TaskGroup._aexit -> TaskGroup.__aexit__ -> main    Task-1          0x10a456060
+      18445801   0x10a439d70          TMBTE                TaskGroup._aexit -> TaskGroup.__aexit__ -> album   TaskGroup._aexit -> TaskGroup.__aexit__ -> main    Task-1          0x10a456060
+      18445801   0x10a2a3a80          TNDNBTG              sleep -> play                                      TaskGroup._aexit -> TaskGroup.__aexit__ -> album   Sundowning      0x10a439f60
+      18445801   0x10a2a38a0          Levitate             sleep -> play                                      TaskGroup._aexit -> TaskGroup.__aexit__ -> album   Sundowning      0x10a439f60
+      18445801   0x10a2d7150          DYWTYLM              sleep -> play                                      TaskGroup._aexit -> TaskGroup.__aexit__ -> album   TMBTE           0x10a439d70
+      18445801   0x10a6bdaa0          Aqua Regia           sleep -> play                                      TaskGroup._aexit -> TaskGroup.__aexit__ -> album   TMBTE           0x10a439d70
+
+.. option:: --retries N
+
+   Retry failed attempts to inspect the target process up to *N* times.  This
+   can help when the target process changes while its state is being read.
+
+   .. versionadded:: 3.15

Doc/library/asyncio.rst

@@ -47,6 +47,13 @@ asyncio provides a set of **high-level** APIs to:
 
 * :ref:`synchronize <asyncio-sync>` concurrent code;
 
+For **introspection**, asyncio provides APIs and tools for:
+
+* inspecting the :ref:`async call graph <asyncio-graph>` of tasks and futures;
+
+* inspecting tasks in another running Python process with
+  :ref:`command-line tools <asyncio-introspection-tools>`;
+
 Additionally, there are **low-level** APIs for
 *library and framework developers* to:
 
@@ -108,7 +115,13 @@ for full functionality and the latest features.
    asyncio-subprocess.rst
    asyncio-queue.rst
    asyncio-exceptions.rst
+
+.. toctree::
+   :caption: Introspection APIs
+   :maxdepth: 1
+
    asyncio-graph.rst
+   asyncio-tools.rst
 
 .. toctree::
    :caption: Low-level APIs

Doc/library/urllib.parse.rst

@@ -649,8 +649,9 @@ task isn't already covered by the URL parsing functions above.
 
    The optional *encoding* and *errors* parameters specify how to deal with
    non-ASCII characters, as accepted by the :meth:`str.encode` method.
-   *encoding* defaults to ``'utf-8'``.
-   *errors* defaults to ``'strict'``, meaning unsupported characters raise a
+   Although these parameters default to ``None`` in the function signature,
+   when processing :class:`str` inputs, *encoding* effectively defaults to ``'utf-8'``
+   and *errors* to ``'strict'``, meaning unsupported characters raise a
    :class:`UnicodeEncodeError`.
    *encoding* and *errors* must not be supplied if *string* is a
    :class:`bytes`, or a :class:`TypeError` is raised.

Lib/test/test_curses.py

@@ -1,6 +1,7 @@
 import functools
 import inspect
 import os
+import platform
 import select
 import string
 import sys
@@ -86,6 +87,18 @@ def wrapped(self, *args, **kwargs):
 BROKEN_NEWTERM = _ncurses_version is not None and _ncurses_version < (6, 5)
 USE_NEWTERM = hasattr(curses, 'newterm') and not BROKEN_NEWTERM
 
+# Older macOS reports a variation selector as a spacing character (wcwidth()
+# == 1) rather than a combining mark, so it cannot share a cell with its base.
+# The failure is confirmed on 14.2 and gone by 26, so skip below 26.
+def _broken_variation_selector_width():
+    if sys.platform == 'darwin':
+        mac_ver = platform.mac_ver()[0]
+        if mac_ver:
+            return tuple(map(int, mac_ver.split('.'))) < (26,)
+    return False
+
+BROKEN_VARIATION_SELECTOR_WIDTH = _broken_variation_selector_width()
+
 # newterm() is used when available (it reports errors instead of exiting), but
 # initscr() is still the fallback, and an unusable $TERM has no terminal to
 # drive either way.
@@ -411,7 +424,8 @@ def test_addch_emoji(self):
         stdscr = self.stdscr
         if self._encodable('\U0001f600'):
             stdscr.addch(0, 0, '\U0001f600')          # single emoji
-        if self._encodable('\u263a\ufe0f'):
+        # Skip the variation selector where the platform reports it as spacing.
+        if not BROKEN_VARIATION_SELECTOR_WIDTH and self._encodable('\u263a\ufe0f'):
             stdscr.addch(1, 0, '\u263a\ufe0f')        # WHITE SMILING FACE + VS-16
         # An emoji ZWJ sequence or an emoji with a modifier is more than one
         # spacing character and cannot share a single cell.

Lib/test/test_ttk/test_style.py

@@ -440,7 +440,8 @@ def test_theme_create_image(self):
 
         b = ttk.Label(self.root, style='TestWidget')
         b.pack(expand=True, fill='both')
-        self.assertEqual(b.winfo_reqwidth(), 134)
+        # The exact width varies with the Tk version and display scaling.
+        self.assertGreater(b.winfo_reqwidth(), 130)
         self.assertEqual(b.winfo_reqheight(), 100)
 
         style.theme_use(curr_theme)

Lib/test/test_ttk/test_widgets.py

@@ -1500,8 +1500,13 @@ def test_configure_height(self):
 
     def test_configure_selectmode(self):
         widget = self.create()
-        self.checkEnumParam(widget, 'selectmode',
-                            'none', 'browse', 'extended')
+        if tk_version >= (9, 1):
+            self.checkEnumParam(widget, 'selectmode',
+                                'none', 'single', 'browse', 'extended',
+                                'multiple')
+        else:
+            self.checkEnumParam(widget, 'selectmode',
+                                'none', 'browse', 'extended')
 
     @requires_tk(8, 7)
     def test_configure_selecttype(self):
@@ -1784,6 +1789,9 @@ def simulate_heading_click(x, y):
         commands = self.tv.master._tclCommands
         self.tv.heading('#0', command=str(self.tv.heading('#0', command=None)))
         self.assertEqual(commands, self.tv.master._tclCommands)
+        # Click elsewhere first, so the second heading click is not reported
+        # as a double click (which does not invoke the command).
+        simulate_mouse_click(self.tv, 5, 50)
         simulate_heading_click(5, 5)
         if not success:
             self.fail("The command associated to the treeview heading wasn't "
@@ -1892,10 +1900,11 @@ def test_insert_item(self):
             value)
 
         # test for values which are not None
+        keep_type = self.wantobjects and tk_version >= (9, 1)
         itemid = self.tv.insert('', 'end', 0)
-        self.assertEqual(itemid, '0')
+        self.assertEqual(itemid, 0 if keep_type else '0')
         itemid = self.tv.insert('', 'end', 0.0)
-        self.assertEqual(itemid, '0.0')
+        self.assertEqual(itemid, 0.0 if keep_type else '0.0')
         # this is because False resolves to 0 and element with 0 iid is already present
         self.assertRaises(tkinter.TclError, self.tv.insert, '', 'end', False)
         self.assertRaises(tkinter.TclError, self.tv.insert, '', 'end', '')
@@ -1955,7 +1964,10 @@ def test_selection(self):
 
         self.tv.insert('', 'end', id=b'bytes\xe2\x82\xac')
         self.tv.selection_set(b'bytes\xe2\x82\xac')
-        self.assertEqual(self.tv.selection(), ('bytes\xe2\x82\xac',))
+        self.assertEqual(self.tv.selection(),
+                         (b'bytes\xe2\x82\xac',)
+                         if self.wantobjects and tk_version >= (9, 1)
+                         else ('bytes\xe2\x82\xac',))
 
         self.tv.selection_set()
         self.assertEqual(self.tv.selection(), ())
@@ -2004,14 +2016,19 @@ def test_virtual_events(self):
     def test_set(self):
         self.tv['columns'] = ['A', 'B']
         item = self.tv.insert('', 'end', values=['a', 'b'])
-        self.assertEqual(self.tv.set(item), {'A': 'a', 'B': 'b'})
+        values = self.tv.set(item)
+        if tk_version >= (9, 1):
+            self.assertEqual(values.pop('#0'), '')
+        self.assertEqual(values, {'A': 'a', 'B': 'b'})
 
         self.tv.set(item, 'B', 'a')
         self.assertEqual(self.tv.item(item, values=None),
                          ('a', 'a') if self.wantobjects else 'a a')
 
         self.tv['columns'] = ['B']
-        self.assertEqual(self.tv.set(item), {'B': 'a'})
+        values = self.tv.set(item)
+        values.pop('#0', None)
+        self.assertEqual(values, {'B': 'a'})
 
         self.tv.set(item, 'B', 'b')
         self.assertEqual(self.tv.set(item, column='B'), 'b')
@@ -2023,7 +2040,9 @@ def test_set(self):
                          123 if self.wantobjects else '123')
         self.assertEqual(self.tv.item(item, values=None),
                          (123, 'a') if self.wantobjects else '123 a')
-        self.assertEqual(self.tv.set(item),
+        values = self.tv.set(item)
+        values.pop('#0', None)
+        self.assertEqual(values,
                          {'B': 123} if self.wantobjects else {'B': '123'})
 
         # inexistent column

Lib/test/test_urllib2.py

@@ -980,6 +980,35 @@ def test_http(self):
             self.assertEqual(req.unredirected_hdrs["Host"], "baz")
             self.assertEqual(req.unredirected_hdrs["Spam"], "foo")
 
+    def test_http_header_priority(self):
+        # gh-47005: regular headers set via add_header() must override
+        # unredirected headers with the same name in do_open(), consistent
+        # with get_header() and header_items().
+        cases = [
+            ("Content-Type", "application/json", "application/x-www-form-urlencoded"),
+            ("Content-Length", "99", "0"),
+            ("Host", "override.example.com", "internal.example.com"),
+            ("Authorization", "Bearer user-token", "Basic stale="),
+            ("Cookie", "a=1", "b=2"),
+            ("User-Agent", "MyApp/1.0", "Python-urllib/test"),
+        ]
+        h = urllib.request.AbstractHTTPHandler()
+        h.parent = MockOpener()
+
+        for key, regular, unredirected in cases:
+            req = Request("http://example.com/", headers={key: regular})
+            req.timeout = None
+            req.add_unredirected_header(key, unredirected)
+
+            http = MockHTTPClass()
+            h.do_open(http, req)
+
+            sent_headers = dict(http.req_headers)
+            self.assertEqual(sent_headers[key], regular)
+            # key is capitalized by add_header() and add_unredirected_header() calls
+            self.assertEqual(req.get_header(key.capitalize()), regular)
+            self.assertEqual(dict(req.header_items())[key.capitalize()], regular)
+
     def test_http_body_file(self):
         # A regular file - chunked encoding is used unless Content Length is
         # already set.

Lib/urllib/request.py

@@ -1293,8 +1293,7 @@ def do_open(self, http_class, req, **http_conn_args):
         h.set_debuglevel(self._debuglevel)
 
         headers = dict(req.unredirected_hdrs)
-        headers.update({k: v for k, v in req.headers.items()
-                        if k not in headers})
+        headers.update(req.headers)
 
         # TODO(jhylton): Should this be redesigned to handle
         # persistent connections?