[Regression]: 11 test failures on WebKit on Debian12 after a8ad57fcb4566146d4be6587c50bd7a095383ee4

[clopez]

Last Good Version

d73823e

First Bad Version

a8ad57f

Steps to reproduce

Use the Debian12 container from the CI (./cli.sh webkit-debian-12) to run the tests with webkit and you get this 11 new failures

  11 failed
    [webkit-library] › tests/library/browsertype-connect.spec.ts:80:5 › launchServer › should connect over wss 
    [webkit-library] › tests/library/browsertype-connect.spec.ts:80:5 › run-server › should connect over wss 
    [webkit-library] › tests/library/debug-controller.spec.ts:85:1 › should pick element ───────────
    [webkit-library] › tests/library/debug-controller.spec.ts:116:1 › should report pages ──────────
    [webkit-library] › tests/library/debug-controller.spec.ts:144:1 › should reset for reuse ───────
    [webkit-library] › tests/library/debug-controller.spec.ts:162:1 › should highlight all ─────────
    [webkit-library] › tests/library/debug-controller.spec.ts:176:1 › should record ────────────────
    [webkit-library] › tests/library/debug-controller.spec.ts:211:1 › should record custom data-testid 
    [webkit-library] › tests/library/debug-controller.spec.ts:270:1 › should highlight inside iframe 
    [webkit-library] › tests/library/debug-controller.spec.ts:296:1 › should highlight aria template 
    [webkit-library] › tests/library/debug-controller.spec.ts:313:1 › should report error in aria template 

This is reproducible with current roll (2286)

The problem is because debian-12 defines localhost as ::1

pwuser@c58ba2e5eaaa:~/playwright$ cat /etc/hosts |grep localhost
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback

pwuser@c58ba2e5eaaa:~/playwright$ ping localhost
PING localhost(localhost (::1)) 56 data bytes
64 bytes from localhost (::1): icmp_seq=1 ttl=64 time=0.022 ms
64 bytes from localhost (::1): icmp_seq=2 ttl=64 time=0.058 ms

The following patch fixes the issue:

index 5eb6502aa..dbb7a601d 100644
--- a/packages/utils/wsServer.ts
+++ b/packages/utils/wsServer.ts
@@ -65,7 +65,7 @@ export class WSServer {
 
     // Default to loopback so the WebSocket RPC is not exposed to the network unless
     // the caller explicitly opts in by passing a host (e.g. '0.0.0.0').
-    hostname ??= 'localhost';
+    hostname ??= '127.0.0.1';
 
     const server = createHttpServer((request, response) => this._onRequest(request, response));
     server.on('error', error => debugLogger.log('server', String(error)));

Expected behavior

Test pass

Actual behavior

Test fail. This is the verbose log of the failures:

pwuser@c58ba2e5eaaa:~/playwright$ DEBUG="pw:browser*" npx playwright test --config=tests/library/playwright.config.ts --project=webkit-*  library/browsertype-connect.spec.ts -g 'should connect over wss'

Running 2 tests using 2 workers
[webkit-library] › tests/library/browsertype-connect.spec.ts:80:5 › run-server › should connect over wss
  pw:browser <ws connecting> wss://localhost:8912/ws +0ms
  pw:browser <ws connected> wss://localhost:8912/ws +13ms
(node:58296) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 1)
(Use `node --trace-warnings ...` to show where the warning was created)
  pw:browser <ws disconnected> wss://localhost:8912/ws code=1005 reason= +5ms
[webkit-library] › tests/library/browsertype-connect.spec.ts:80:5 › launchServer › should connect over wss
  pw:browser <ws connecting> wss://localhost:8908/ws +0ms
  pw:browser <ws connected> wss://localhost:8908/ws +12ms
(node:58295) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 1)
(Use `node --trace-warnings ...` to show where the warning was created)
  pw:browser <ws disconnected> wss://localhost:8908/ws code=1005 reason= +5ms
[webkit-library] › tests/library/browsertype-connect.spec.ts:80:5 › run-server › should connect over wss
====== node /home/pwuser/playwright/packages/playwright-core/cli.js run-server
Listening on ws://localhost:42113/

=========================================
  1) [webkit-library] › tests/library/browsertype-connect.spec.ts:80:5 › run-server › should connect over wss 

    Error: connect ECONNREFUSED 127.0.0.1:42113

    Error: browserType.connect: Target page, context or browser has been closed

      45 |         ...options.headers,
      46 |       };
    > 47 |       browser = await browserType.connect(wsEndpoint, options);
         |                                   ^
      48 |       return browser;
      49 |     });
      50 |     await browser?.close();
        at connect (/home/pwuser/playwright/tests/library/browsertype-connect.spec.ts:47:35)
        at /home/pwuser/playwright/tests/library/browsertype-connect.spec.ts:109:31

    Error Context: test-results/browsertype-connect-run-server-should-connect-over-wss-webkit-library/error-context.md

[webkit-library] › tests/library/browsertype-connect.spec.ts:80:5 › launchServer › should connect over wss
====== node /home/pwuser/playwright/tests/config/remote-server-impl.js {"browserTypeName":"webkit","launchOptions":{"headless":true,"handleSIGINT":true,"handleSIGTERM":true,"handleSIGHUP":true}}
(tempDir=>/tmp/playwright_webkitdev_profile-WQiHRV)
(pid=>58341)
(wsEndpoint=>ws://localhost:33607/8ca6b7701c80fa20efd6762fa64ef10e)
(exitCode=>0)
(signal=>null)

=========================================
  2) [webkit-library] › tests/library/browsertype-connect.spec.ts:80:5 › launchServer › should connect over wss 

    Error: connect ECONNREFUSED 127.0.0.1:33607

    Error: browserType.connect: Target page, context or browser has been closed

      45 |         ...options.headers,
      46 |       };
    > 47 |       browser = await browserType.connect(wsEndpoint, options);
         |                                   ^
      48 |       return browser;
      49 |     });
      50 |     await browser?.close();
        at connect (/home/pwuser/playwright/tests/library/browsertype-connect.spec.ts:47:35)
        at /home/pwuser/playwright/tests/library/browsertype-connect.spec.ts:109:31

    Error Context: test-results/browsertype-connect-launchServer-should-connect-over-wss-webkit-library/error-context.md

  2 failed
    [webkit-library] › tests/library/browsertype-connect.spec.ts:80:5 › launchServer › should connect over wss 
    [webkit-library] › tests/library/browsertype-connect.spec.ts:80:5 › run-server › should connect over wss 

And also:

pwuser@c58ba2e5eaaa:~/playwright$ DEBUG="pw:browser*" npx playwright test --config=tests/library/playwright.config.ts --project=webkit-*  library/debug-controller.spec.ts                   

Running 10 tests using 10 workers
  1) [webkit-library] › tests/library/debug-controller.spec.ts:85:1 › should pick element ──────────

    Error: WebSocket error: connect ECONNREFUSED 127.0.0.1:35695

       at config/debugControllerBackend.ts:57

      55 |       });
      56 |       transport._ws.addEventListener('error', event => {
    > 57 |         reject(new Error('WebSocket error: ' + event.message));
         |                ^
      58 |         transport._ws.close();
      59 |       });
      60 |     });
        at WebSocket.<anonymous> (/home/pwuser/playwright/tests/config/debugControllerBackend.ts:57:16)
        at callListener (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:290:14)
        at WebSocket.onError (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:230:9)
        at emitErrorAndClose (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:1035:13)
        at ClientRequest.<anonymous> (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:880:5)

    Error Context: test-results/debug-controller-should-pick-element-webkit-library/error-context.md

  2) [webkit-library] › tests/library/debug-controller.spec.ts:116:1 › should report pages ─────────

    Error: WebSocket error: connect ECONNREFUSED 127.0.0.1:43225

       at config/debugControllerBackend.ts:57

      55 |       });
      56 |       transport._ws.addEventListener('error', event => {
    > 57 |         reject(new Error('WebSocket error: ' + event.message));
         |                ^
      58 |         transport._ws.close();
      59 |       });
      60 |     });
        at WebSocket.<anonymous> (/home/pwuser/playwright/tests/config/debugControllerBackend.ts:57:16)
        at callListener (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:290:14)
        at WebSocket.onError (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:230:9)
        at emitErrorAndClose (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:1035:13)
        at ClientRequest.<anonymous> (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:880:5)

    Error Context: test-results/debug-controller-should-report-pages-webkit-library/error-context.md

  3) [webkit-library] › tests/library/debug-controller.spec.ts:162:1 › should highlight all ────────

    Error: WebSocket error: connect ECONNREFUSED 127.0.0.1:38389

       at config/debugControllerBackend.ts:57

      55 |       });
      56 |       transport._ws.addEventListener('error', event => {
    > 57 |         reject(new Error('WebSocket error: ' + event.message));
         |                ^
      58 |         transport._ws.close();
      59 |       });
      60 |     });
        at WebSocket.<anonymous> (/home/pwuser/playwright/tests/config/debugControllerBackend.ts:57:16)
        at callListener (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:290:14)
        at WebSocket.onError (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:230:9)
        at emitErrorAndClose (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:1035:13)
        at ClientRequest.<anonymous> (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:880:5)

    Error Context: test-results/debug-controller-should-highlight-all-webkit-library/error-context.md

  4) [webkit-library] › tests/library/debug-controller.spec.ts:211:1 › should record custom data-testid 

    Error: WebSocket error: connect ECONNREFUSED 127.0.0.1:33939

       at config/debugControllerBackend.ts:57

      55 |       });
      56 |       transport._ws.addEventListener('error', event => {
    > 57 |         reject(new Error('WebSocket error: ' + event.message));
         |                ^
      58 |         transport._ws.close();
      59 |       });
      60 |     });
        at WebSocket.<anonymous> (/home/pwuser/playwright/tests/config/debugControllerBackend.ts:57:16)
        at callListener (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:290:14)
        at WebSocket.onError (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:230:9)
        at emitErrorAndClose (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:1035:13)
        at ClientRequest.<anonymous> (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:880:5)

    Error Context: test-results/debug-controller-should-record-custom-data-testid-webkit-library/error-context.md

  5) [webkit-library] › tests/library/debug-controller.spec.ts:144:1 › should reset for reuse ──────

    Error: WebSocket error: connect ECONNREFUSED 127.0.0.1:44577

       at config/debugControllerBackend.ts:57

      55 |       });
      56 |       transport._ws.addEventListener('error', event => {
    > 57 |         reject(new Error('WebSocket error: ' + event.message));
         |                ^
      58 |         transport._ws.close();
      59 |       });
      60 |     });
        at WebSocket.<anonymous> (/home/pwuser/playwright/tests/config/debugControllerBackend.ts:57:16)
        at callListener (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:290:14)
        at WebSocket.onError (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:230:9)
        at emitErrorAndClose (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:1035:13)
        at ClientRequest.<anonymous> (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:880:5)

    Error Context: test-results/debug-controller-should-reset-for-reuse-webkit-library/error-context.md

  6) [webkit-library] › tests/library/debug-controller.spec.ts:176:1 › should record ───────────────

    Error: WebSocket error: connect ECONNREFUSED 127.0.0.1:33095

       at config/debugControllerBackend.ts:57

      55 |       });
      56 |       transport._ws.addEventListener('error', event => {
    > 57 |         reject(new Error('WebSocket error: ' + event.message));
         |                ^
      58 |         transport._ws.close();
      59 |       });
      60 |     });
        at WebSocket.<anonymous> (/home/pwuser/playwright/tests/config/debugControllerBackend.ts:57:16)
        at callListener (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:290:14)
        at WebSocket.onError (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:230:9)
        at emitErrorAndClose (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:1035:13)
        at ClientRequest.<anonymous> (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:880:5)

    Error Context: test-results/debug-controller-should-record-webkit-library/error-context.md

[webkit-library] › tests/library/debug-controller.spec.ts:246:1 › should reset routes before reuse
  pw:browser <ws connecting> ws://localhost:39489/6f6e68288a9a78f4723134d3a4d54c31 +0ms
  7) [webkit-library] › tests/library/debug-controller.spec.ts:296:1 › should highlight aria template 

    Error: WebSocket error: connect ECONNREFUSED 127.0.0.1:37065

       at config/debugControllerBackend.ts:57

      55 |       });
      56 |       transport._ws.addEventListener('error', event => {
    > 57 |         reject(new Error('WebSocket error: ' + event.message));
         |                ^
      58 |         transport._ws.close();
      59 |       });
      60 |     });
        at WebSocket.<anonymous> (/home/pwuser/playwright/tests/config/debugControllerBackend.ts:57:16)
        at callListener (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:290:14)
        at WebSocket.onError (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:230:9)
        at emitErrorAndClose (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:1035:13)
        at ClientRequest.<anonymous> (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:880:5)

    Error Context: test-results/debug-controller-should-highlight-aria-template-webkit-library/error-context.md

  pw:browser <ws connected> ws://localhost:39489/6f6e68288a9a78f4723134d3a4d54c31 +5ms
  8) [webkit-library] › tests/library/debug-controller.spec.ts:270:1 › should highlight inside iframe 

    Error: WebSocket error: connect ECONNREFUSED 127.0.0.1:32869

       at config/debugControllerBackend.ts:57

      55 |       });
      56 |       transport._ws.addEventListener('error', event => {
    > 57 |         reject(new Error('WebSocket error: ' + event.message));
         |                ^
      58 |         transport._ws.close();
      59 |       });
      60 |     });
        at WebSocket.<anonymous> (/home/pwuser/playwright/tests/config/debugControllerBackend.ts:57:16)
        at callListener (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:290:14)
        at WebSocket.onError (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:230:9)
        at emitErrorAndClose (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:1035:13)
        at ClientRequest.<anonymous> (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:880:5)

    Error Context: test-results/debug-controller-should-highlight-inside-iframe-webkit-library/error-context.md

  9) [webkit-library] › tests/library/debug-controller.spec.ts:313:1 › should report error in aria template 

    Error: WebSocket error: connect ECONNREFUSED 127.0.0.1:44745

       at config/debugControllerBackend.ts:57

      55 |       });
      56 |       transport._ws.addEventListener('error', event => {
    > 57 |         reject(new Error('WebSocket error: ' + event.message));
         |                ^
      58 |         transport._ws.close();
      59 |       });
      60 |     });
        at WebSocket.<anonymous> (/home/pwuser/playwright/tests/config/debugControllerBackend.ts:57:16)
        at callListener (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:290:14)
        at WebSocket.onError (/home/pwuser/playwright/node_modules/ws/lib/event-target.js:230:9)
        at emitErrorAndClose (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:1035:13)
        at ClientRequest.<anonymous> (/home/pwuser/playwright/node_modules/ws/lib/websocket.js:880:5)

    Error Context: test-results/debug-controller-should-report-error-in-aria-template-webkit-library/error-context.md

  pw:browser <ws disconnecting> ws://localhost:39489/6f6e68288a9a78f4723134d3a4d54c31 +692ms
  pw:browser <ws connecting> ws://localhost:39489/6f6e68288a9a78f4723134d3a4d54c31 +0ms
  pw:browser <ws disconnected> ws://localhost:39489/6f6e68288a9a78f4723134d3a4d54c31 code=1005 reason= +3ms
  pw:browser <ws connected> ws://localhost:39489/6f6e68288a9a78f4723134d3a4d54c31 +1ms
  pw:browser <ws disconnecting> ws://localhost:39489/6f6e68288a9a78f4723134d3a4d54c31 +1s
  pw:browser <ws disconnected> ws://localhost:39489/6f6e68288a9a78f4723134d3a4d54c31 code=1005 reason= +1ms
  9 failed
    [webkit-library] › tests/library/debug-controller.spec.ts:85:1 › should pick element ───────────
    [webkit-library] › tests/library/debug-controller.spec.ts:116:1 › should report pages ──────────
    [webkit-library] › tests/library/debug-controller.spec.ts:144:1 › should reset for reuse ───────
    [webkit-library] › tests/library/debug-controller.spec.ts:162:1 › should highlight all ─────────
    [webkit-library] › tests/library/debug-controller.spec.ts:176:1 › should record ────────────────
    [webkit-library] › tests/library/debug-controller.spec.ts:211:1 › should record custom data-testid 
    [webkit-library] › tests/library/debug-controller.spec.ts:270:1 › should highlight inside iframe 
    [webkit-library] › tests/library/debug-controller.spec.ts:296:1 › should highlight aria template 
    [webkit-library] › tests/library/debug-controller.spec.ts:313:1 › should report error in aria template 
  1 passed (2.3s)

Additional context

No response

Environment

pwuser@c58ba2e5eaaa:~/playwright$ npx envinfo --preset playwright

  System:
    OS: Linux 6.17 Debian GNU/Linux 12 (bookworm) 12 (bookworm)
    CPU: (24) x64 Intel(R) Core(TM) Ultra 9 275HX
    Memory: 114.32 GB / 125.27 GB
    Container: Yes
  Binaries:
    Node: 22.22.2 - /usr/bin/node
    npm: 10.9.7 - /usr/bin/npm
  Languages:
    Bash: 5.2.15 - /usr/bin/bash

[yury-s]

Interestingly enough, I cannot reproduce it with the same docker image in devcontainers, looks like the resolution order is different. I can reproduce it though with the following command:

NODE_OPTIONS="--dns-result-order=ipv4first --no-network-family-autoselection" npx playwright test --config=tests/library/playwright.config.ts --project=webkit-library library/browsertype-connect.spec.ts -g 'should connect over wss'

[yury-s]

Not reproducible everywhere because it's environmental, not a Node version issue (Node 22.22.2, same as CI, also doesn't repro in a regular dev container).

The CI job runs in container: debian:12 where IPv6 loopback is effectively disabled:

• server.listen(0, 'localhost') binds to ::1 (first entry in /etc/hosts).
• Client connect to ws://localhost:PORT can't reach ::1, Happy Eyeballs falls back to 127.0.0.1 → ECONNREFUSED.

In a dev container with working ::1, both sides agree on ::1 and the test passes.

To repro locally without the GHA container, force the asymmetry:

NODE_OPTIONS="--no-network-family-autoselection --dns-result-order=ipv4first" \
  npx playwright test --config=tests/library/playwright.config.ts \
  --project=webkit-library library/browsertype-connect.spec.ts -g 'should connect over wss'