Failover retries keep targeting an unhealthy primary instead of the healthy backup

[Kewe63]

Summary

With multiProvider.strategy: "failover", a read retries the original primary after it becomes unhealthy instead of selecting the healthy backup. The call rejects despite remaining retry budget and an available backup.

Reproduced using the actual SDK provider and native fetch against two real local HTTP servers. The backup succeeds when called directly. This is separate from #6: caching is not enabled in this reproduction.

Affected revision and environment

• Repository default branch: mainnet-beta, checked again before filing.
• Commit: a34d5fc84580b8ef88d77c7d72fdcc515c467a3a.
• Source package: @psy-protocol/psy-sdk, manifest version 2.0.4.
• Node 22.23.1, pnpm 9.15.9, TypeScript 4.9.5, Jest 29.7.0 / ts-jest 29.4.6.
• Linux x86_64 / WSL2; unchanged workspace lockfile.

Tests run exported provider classes from this checkout, not an independently downloaded registry build. No WASM, wallet, transaction, proving service or live RPC is used.

Expected behavior

The multi-provider example documents automatic failover when the primary is down.

With maxConsecutiveFailures: 1 and maxAttempts: 2, a retryable primary failure should leave one attempt for the healthy backup:

primary → backup → result 42

Actual behavior

primary → primary → rejected read

The native-fetch test configures the primary to return HTTP 503 and the backup to return a synthetic HTTP 200 JSON-RPC result of 42. A separate control reads 42 from that same backup successfully.

The test explicitly sets retryableErrors: ["Error in RPC call"] to match the SDK's HTTP error. It does not assume HTTP 503 is retryable by default.

Reproduction

In a disposable checkout of the pinned commit:

cd psy-ts-sdk
pnpm install --frozen-lockfile
# Save the test below as packages/psy-sdk/src/provider/localTransport.regression.test.ts
unshare -Urn -- sh -c '/usr/sbin/ip link set lo up && exec pnpm --filter @psy-protocol/psy-sdk exec jest --runInBand --no-cache --runTestsByPath src/provider/localTransport.regression.test.ts'

unshare -Urn isolates the Linux test network; loopback is enabled inside it. Install dependencies before entering the namespace. Both listeners use ephemeral loopback ports and are closed after the tests. No requests leave the namespace. Neither fetch nor the provider implementation is mocked.

Complete executed regression test

import { afterAll, beforeAll, describe, expect, it } from "@jest/globals";
import { createServer, Server } from "node:http";
import { CoordinatorEdgeRpcProvider } from "../coord-edge-rpc/client";
import type { ClientConfig } from "./provider";

// Actual native fetch over ephemeral loopback HTTP servers, never an external RPC.
describe("quality failover with native HTTP transport", () => {
    let primary: Server;
    let backup: Server;
    let primaryUrl: string;
    let backupUrl: string;
    let requests: string[] = [];
    async function listen(label: string, status: number): Promise<{ server: Server; url: string }> {
        const server = createServer((request, response) => {
            request.resume();
            requests.push(label);
            response.writeHead(status, { "Content-Type": "application/json" });
            response.end(
                JSON.stringify(
                    status === 200
                        ? { jsonrpc: "2.0", id: "1", result: 42 }
                        : { jsonrpc: "2.0", id: "1", error: { code: -32000, message: "temporarily unavailable" } }
                )
            );
        });
        await new Promise<void>((resolve, reject) => {
            server.once("error", reject);
            server.listen(0, "127.0.0.1", resolve);
        });
        const address = server.address();
        if (!address || typeof address === "string") throw new Error("Expected local TCP address");
        return { server, url: `http://127.0.0.1:${address.port}` };
    }
    beforeAll(async () => {
        const a = await listen("primary", 503);
        primary = a.server;
        primaryUrl = a.url;
        const b = await listen("backup", 200);
        backup = b.server;
        backupUrl = b.url;
    });
    afterAll(async () => {
        for (const server of [primary, backup]) {
            if (!server) continue;
            server.closeAllConnections();
            await new Promise<void>((resolve, reject) => server.close((error) => (error ? reject(error) : resolve())));
        }
    });
    it("control: native fetch reads the healthy backup successfully", async () => {
        requests = [];
        const client = new CoordinatorEdgeRpcProvider(backupUrl);
        try {
            expect(await client.getLatestCheckpointId()).toBe(42);
            expect(requests).toEqual(["backup"]);
        } finally {
            client.destroy();
        }
    });
    it("should use the healthy backup after the configured retryable HTTP failure", async () => {
        requests = [];
        const config: ClientConfig = {
            multiProvider: { strategy: "failover", maxConsecutiveFailures: 1, healthCheckInterval: 1000000 },
            retry: { maxAttempts: 2, baseDelay: 0, maxDelay: 0, jitter: false, retryableErrors: ["Error in RPC call"] },
        };
        const client = new CoordinatorEdgeRpcProvider([primaryUrl, backupUrl], config);
        try {
            const outcome = await client.getLatestCheckpointId().then(
                (value) => ({ value, failed: false }),
                () => ({ value: null, failed: true })
            );
            expect({ ...outcome, requests }).toEqual({ value: 42, failed: false, requests: ["primary", "backup"] });
        } finally {
            client.destroy();
        }
    });
});

Observed output

Rerun immediately before filing:

quality failover with native HTTP transport
  ✓ control: native fetch reads the healthy backup successfully
  ✕ should use the healthy backup after the configured retryable HTTP failure

Expected: { value: 42, failed: false, requests: ["primary", "backup"] }
Received: { value: null, failed: true, requests: ["primary", "primary"] }

Tests: 1 failed, 1 passed, 2 total
Process exit: 1

The same native HTTP test also reproduced in two earlier isolated runs. An additional controlled-transport test independently observed the primary becoming unhealthy while the backup remained healthy, with both attempts still targeting the primary.

Root cause

rpc() selects an endpoint once and passes its URL into rpc_with_url().

The retry loop reuses that URL in its default strategy branch. The catch path updates health, but the next attempt does not reselect an endpoint using the updated health state.

Impact and scope

Suggested priority: Medium / P2. A supported SDK read fails despite a working configured backup and sufficient retry budget. This requires a retryable failure and a configuration that leaves an attempt after the primary becomes unhealthy.

Only read behavior was tested. This does not establish that later independently initiated calls fail, that every strategy/configuration is affected, or that there is any transaction/fund impact.

Related work and duplicate check

#6 concerns cache initialization and has a different root cause; this reproducer omits cache configuration entirely. Fresh all-state tracker checks and searches for failover and retry found no equivalent issue/PR. The earlier audit also inspected accessible PR files, relevant patches and Provider path history; no equivalent fix was found. Private/external histories were not exhaustively reviewed.

Suggested fix direction

For failover, reselect an eligible endpoint after a retryable failure and attribute health changes to the actual endpoint used. Preserve attempt limits, headers and cancellation. Keep this read regression; do not broaden mutation retry behavior without separately defining that contract.