diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d97fae..7eae743 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,12 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Fixed + +- **网页搜索遇到连接层瞬时失败会整批失败,且 `fetch failed` 无法定位原因(#117)。** 报告者在 10 分钟内 14 次 `web_search` 失败 3 次:一次 4 个 query 的调用 2 次全部失败,单 query 12 次失败 1 次,失败后立即重试同一 query 全部成功;失败在拿到 HTTP 状态之前发生,面板记录约 21.7 秒。根因是同一链路上的两个独立事实:undici 的 `fetch` 只在连接/读取阶段失败后抛出裸的 `TypeError: fetch failed`(真实原因在 `error.cause`,本机对不可达地址实测 10.5 秒后为 `UND_ERR_CONNECT_TIMEOUT`,双地址族各等一轮可达约 20 秒),而插件的搜索只发一次、失败即抛;DSH 的 `dsh-tool-web` 在多 query 时又会在第一个失败时中止同批其余请求并整体抛出。现在 `CommandCodeSearchProvider.search()` 对「已收到 HTTP 响应」之外的连接层失败重试:最多 3 次尝试(首发 + 2 次),退避 500 ms/1 s 且带 ±20% 抖动,账号只在进入重试前解析一次;等待过程监听调用方信号,用户取消或工具协程超时仍按 `WEB_ABORTED` 结束,不会睡满退避。所有非取消的 fetch 拒绝都按瞬时处理(含证书、参数类确定性错误),代价是最多约 1.5 秒后抛出同一错误;收到 HTTP 状态(含 401/403/429)不重试——那是服务端的答案——响应体失败也不重试,因为响应头已到、请求已被接受并可能执行,重放有重复计费的代价。最终错误追加原因码(`UND_ERR_CONNECT_TIMEOUT`、`ECONNRESET` 等,复用 `transportFailureReason()`)、目标 origin(`httpOrigin()` 丢弃路径与查询,不回显反代前缀或凭据)、耗时与尝试次数,`fetch failed` 不再同时承担 DNS、代理、超时三种解释。**边界**:多 query 的部分成功仍由 DSH 的 `dsh-tool-web` 决定,不在本插件;重试只覆盖「未收到响应」,因此一次上游已处理但响应丢失的搜索可能被重放(搜索本身只读且幂等)。慢失败时 3 次尝试可能超过 `dsh-tool-web` 默认 30 秒预算,最终只看到 `WEB_ABORTED`;完整算术与取舍见 `docs/agent-reference/accounts-and-retry.md`。**验证**:`tests/web-search.test.ts` 新增 7 条用例(第二次尝试成功且密钥只解析一次、3 次后停止并在错误串带原因/目标/次数、403 不重试、注入生产等待函数的取消用例、默认 `retryWait ?? searchRetryWait` 接线的取消用例、退避基数与抖动边界、传给等待函数的延迟序列);`typecheck` 退出码 0,`npm test` 全量通过,`build` 两段 `Build complete`。未做真实账号的并发搜索复验(需要报告者的网络条件)。 + ## [0.16.4] - 2026-11-02 ### Added diff --git a/docs/agent-reference/accounts-and-retry.md b/docs/agent-reference/accounts-and-retry.md index 115d7fd..80618fa 100644 --- a/docs/agent-reference/accounts-and-retry.md +++ b/docs/agent-reference/accounts-and-retry.md @@ -217,6 +217,34 @@ Task-specific reference moved from the former root `AGENTS.md`. All source and t actually reads (take/restore/re-take, foreign pin, `preexisting`, refused write → memory fallback, a missing settings service, and unload not writing). A frozen runtime that offers no writable selection surface must neither throw nor change anything. + **Transient network failures are retried inside the provider (issue #117).** `dsh-tool-web` runs the queries + of one call concurrently and aborts every sibling as soon as the first fails, so one pre-response + `fetch failed` used to discard the whole multi-query step (the reporter's 4-query calls failed 2/2 while + retrying the same single query by hand succeeded). `search()` now repeats a fetch that received NO response + up to `SEARCH_MAX_ATTEMPTS` (3 = the first try plus two retries) with jittered exponential backoff + (500 ms then 1 s, ±20%), resolving the account once before the first attempt; the wait listens to the + caller's signal, so a user stop or the tool's cooperative deadline still ends as `WEB_ABORTED` instead of + sleeping through. Every non-cancel fetch rejection counts as transient: the runtime does not classify + `error.cause` for us, and a deterministic failure (an expired certificate, an argument the runtime rejects) + only costs the bounded backoff before the same error surfaces — cheaper than a code allowlist that could + drop an unlisted blip. A received HTTP status — 401/403/429 included — is never retried: it is the server's + answer, and replaying a plan/credential rejection helps nobody. A **response-body** failure is not retried + either, but for the opposite reason: the headers arrived, so the request was already accepted and may have + been executed; replaying it risks duplicate billing, and the failure is reported as-is. The terminal error + appends the fetch cause code (`UND_ERR_CONNECT_TIMEOUT`, `ECONNRESET`, …), the target origin from + `httpOrigin()` (path and query dropped, so a reverse-proxy prefix or a credential cannot leak) plus elapsed + time and attempt count, so `fetch failed` alone no longer has to cover DNS, proxy and connect-timeout + causes. Partial results for a multi-query call are still impossible here: the abort-on-first-failure policy + lives in dsh's `dsh-tool-web`, not in this provider. + **The retry budget deliberately brushes the tool's own deadline.** undici's connect timeout is ~10 s, so + three connect-timeout attempts plus the two waits span ~31.5–33 s — past `dsh-tool-web`'s default 30 s + `searchTimeoutMs`. In that slow-failure case the tool deadline fires mid-retry and the call ends as + `WEB_ABORTED`, so the terminal transport diagnosis above is never produced; the reporter's 21.7 s first + failure leaves only ~8 s for the retry, which a fresh connection can still use but a sustained outage + cannot. A shorter per-attempt timeout is not added (it would be a new user-visible contract), and an + elapsed-time rule is not used (it would skip the very slow failure this issue reported). + `tests/web-search.test.ts` pins the retry, the once-only key resolution, the no-retry-on-status rule, the + abortable wait and the diagnostic suffix. - **`TRANSPORT` carries its own bounded budget, and the route policy cannot express it** (`src/transport-retry.ts` + the `agent/request-error` listener in `src/index.ts`; issue #39's second report). `providerRetryPolicy()` is captured once per route, so every code in its whitelist shares one diff --git "a/docs/\345\206\263\347\255\226\350\256\260\345\275\225.md" "b/docs/\345\206\263\347\255\226\350\256\260\345\275\225.md" index 0d90054..b3547a4 100644 --- "a/docs/\345\206\263\347\255\226\350\256\260\345\275\225.md" +++ "b/docs/\345\206\263\347\255\226\350\256\260\345\275\225.md" @@ -191,3 +191,5 @@ 1. **2026-10-31 — 探活改探真实端点;网络与配置写入失败必须带可定位信息(issue #110、#111,用户拍板)。** **来源**:同一位报告者在 0.16.0 上提交两条 issue,均在当前 HEAD 复现。**(a) 探活路径前缀**:`probeCommandCode()` 原先用 `new URL('/', apiBase)` 探 API base 根,而 URL 构造会丢掉路径前缀;`apiBase` 是普通字符串、真实请求走 `${apiBase}${path}`,因此带前缀的反向代理(报告者的 `http://127.0.0.1:18797/bili/https://api.commandcode.ai`)必然误报「本机无法连接」,根路径被其它服务占用返回 200 时又会假报正常。**用户选定**:改探真实端点 `${apiBase}/alpha/whoami`(与登录校验、whoami 与计费同一条链路),用与真实请求相同的字符串拼接保留前缀。**未采纳**:(a1) 只保留前缀继续探根——反代只转发 `/alpha/*`、根路径 404 时仍误报;(a2) 只写文档说明——`apiBase` 没有 origin 校验,用户仍会踩。**(b) 未认证 401/403 算「可达」**:探活刻意不带凭据,`/alpha/whoami` 的预期回答就是 401(本机实测 body 为结构化 `UNAUTHORIZED`),403 同理(网关拦截);判为失败会让每个健康主机显示「连接异常」。2xx/3xx 与 401/403 之外(404 表配置路径错误、429、5xx)仍是 HTTP 失败,超时与传输失败继续分开。**代价**:判定语义不再镜像官方 Upptime monitor 的 `expectedStatusCodes`,模块与 wire 注释已同步改写。**(c) 失败提示带目标与原因**:`getUsage()` 的 `failures` 原先只有 `fetch failed` 或 `HTTP `,无法区分「反代换端口」「代理没起」「API 故障」。新增 `src/transport-failure.ts`:从 fetch 的 `error.cause`(含 AggregateError 子项)取错误码并映射为中文原因、保留原始码(`ECONNREFUSED` →「连接被拒绝(ECONNREFUSED)」),`httpOrigin()` 另附 `scheme://host:port`(丢弃路径与查询,不回显反代前缀或凭据);无码不猜原因,未知码回显 `网络错误()`。**(d) 设置写入提示**:`unconfirmed` 原先固定映射到「密钥未写入」,但该状态来自配置写入确认,与密钥无关——报告者只改 ZDR 与模型可见性却被引向密钥。现按 `credential` 维度分文案:本次保存带了密钥(`not-issued`)才说「配置与密钥均未写入宿主」,否则只说「配置提交未获确认,未写入宿主」。同时透传宿主原文:`SettingsMutationResult` 扩为 `SettingsMutationReport`(`{ result, message? }`),`SettingsWriteResult.hostError` 携带原文,`settingsWriteNotice()` 追加「宿主返回:…」(折叠为一行、限长 240 字符)。报告者命中的 `atomic-write: timed out waiting for the writer lock` 因此可见。**边界**:`result` 仍是唯一写入判定,`message` 从不参与分支;原文可能含宿主路径(如 profile 锁文件),属同一信任域诊断信息且长度受控。**核验**:`npm test` 1153/1153、`typecheck` 与 `build` 通过;新增用例覆盖前缀保留、401 可达、传输原因映射(`tests/transport-failure.test.ts`)、未带密钥不提密钥、宿主原文透传与折叠截断。**端到端(2026-10-31)**:隔离 `DSH_HOME` 起真实宿主(`dsh 0.2.2-alpha.2`,web profile,插件 `link:` 到本仓库构建产物),`apiBase` 指向本地带前缀反代 `http://127.0.0.1:18797/bili/https://api.commandcode.ai`,无头 Chrome 打开设置页实测——反代在线时「本机 连通正常 · 781 ms」,反代日志记录到 `/bili/https://api.commandcode.ai/alpha/whoami`,而旧实现的探测目标(反代根 `/`)返回 404,构成红绿对照;停掉反代后「本机无法连接」,账户卡片显示「连接被拒绝(ECONNREFUSED);目标 http://127.0.0.1:18797」;用存活进程占住 profile 写锁后保存,提示为「配置提交未获确认,未写入宿主;请核对后重试。宿主返回:atomic-write: timed out waiting for the writer lock at …」,未出现「密钥」。**真实凭据复验(2026-10-31,充值后)**:同一隔离环境改用 `~/.dsh/.credentials.yaml` 里的真实 `COMMANDCODE_API_KEY`(文件复制进隔离 `DSH_HOME`、600 权限,全程不打印密钥,收尾即删)——只读接口全部 200(`/alpha/whoami`、`/alpha/billing/credits`、`/alpha/billing/subscriptions`、`/alpha/usage/summary`、`/provider/v1/models`),状态行「本机 连通正常 · 838 ms」,账户卡片显示真实数据且无失败提示(成功路径不会误加「目标/原因」后缀);真实生成跑通,`POST /alpha/generate` 返回 200 流式响应(`text-delta` 为「收到」),会话费用读数 `≈$0.0015` 与上游扣费 `0.001471284` 一致,`monthlyCredits` 由 10 降至 9.998528716。**浏览器登录链路复验(2026-10-31)**:隔离宿主里把 `apiKeyEnv` 指向一个凭据库里不存在的引用 `COMMANDCODE_LOGIN_TEST`,设置页初始显示「未配置」;点「网页登录」后宿主绑定 `127.0.0.1:5959` 并给出授权链接(`https://commandcode.ai/studio/auth/cli?callback=…&state=…`);随后按 Studio 的契约向回调 POST 五字段(真实密钥 + 该 state,`Origin: https://commandcode.ai`),得 `200 {"success":true}`——反代日志同期出现 `GET /alpha/whoami` **200**(即插件用交付的密钥经前缀反代完成了校验),隔离凭据库随即出现 `COMMANDCODE_LOGIN_TEST` 且文件仍为 `600`,界面转为「已登录为 mars-sea · login-e2e」,账户卡片加载出真实数据(Go 套餐、月额度 $10.00)。**未覆盖**:Studio 页面本身的登录——无头浏览器打开授权链接会被重定向到 `https://commandcode.ai/zh/signin?returnTo=…` 并出现 Cloudflare Turnstile 挑战,必须真人在浏览器里完成账号登录,这一步无法自动化;因此本次验证的是「插件侧全链路 + Studio 回调契约」,不是完整的人工登录。**未做**:浏览器 Studio 登录开通链路(需人工在弹窗里完成登录);`COMMANDCODE_API_KEY_3` 已失效(`whoami` 401),未纳入验证。**另记**:余额为 0 时上游返回 `400 BAD_REQUEST` 加正文 `insufficient credits`,插件已能据此做账号轮换判定,但用户文案仍是通用 400 提示,未回显该原因(见 `TODO.local.md`)。 2. **2026-10-31 — 额度不足的提示改由账号侧证据驱动,文案与轮换共用同一份判据。** **来源**:上条端到端实测中,充值前的那次生成被上游以 `400 BAD_REQUEST` 拒绝,正文为 `You have insufficient credits to make this request…`。插件当时的表现是:账号池**已经**按 `insufficient credits` 把该账号判为 `unavailable`(`classifyAccountRejection`),但单账号场景没有可轮换对象,最终落到 `generateHttpError()` 的通用分支,界面显示「服务端拒绝了本次请求,请检查模型、套餐和请求设置」——把用户引向模型、套餐、请求设置三条,一条也查不到点上。**用户拍板**:补这一条文案。**做法**:把「额度用尽」的正文特征抽成 `isInsufficientCreditsText()`(与「模型不在套餐内」的 `isModelOutsidePlanText()` 并列),账号轮换与 `generateHttpError()` 共用;`generateHttpError()` 在通用分支之前识别该形状(代码 `INSUFFICIENT_CREDITS`/`PREMIUM_CREDITS_EXHAUSTED` 或正文特征),返回「账户额度不足,无法完成本次请求——请充值额度,或切换到其他账号」。**边界**:失败码仍取 `httpErrorCode(status)`,重试预算、账号轮换与协议回退一律不变——这是文案,不是分类;上游英文正文继续不回显(沿用该函数末尾的展示规则)。**未采纳**:把 `USAGE_EXCEEDED`(窗口额度)与 `MODEL_NOT_IN_PLAN`(套餐不含该模型)也并进这条文案——两者的用户可行动方向不同,前者该等窗口重置、后者该换模型,混成一句「额度不足」会给出错误建议。**核验**:`tests/adapter.test.ts` 三条用例——实测正文(含 `BAD_REQUEST` 代码)给出可行动文案且不再出现通用提示、仅结构化代码无正文措辞时同样识别、普通 400(`validation_error`)不被该分支吞掉;`npm test` 1156/1156、`typecheck` 退出码 0。**证据边界**:该形状取自 2026-10-31 真实账号的真实响应,但账户充值后已无法复现,故只有合成用例覆盖,未做端到端复验。 + +3. **2026-11-03 — 网页搜索只对「未收到响应」的连接层失败做有界重试,并让错误串带原因与目标(issue #117)。** **来源**:报告者在 0.16.3 上记录 10 分钟内 14 次 `web_search` 失败 3 次——4 个 query 的调用 2 次全部失败,单 query 12 次失败 1 次,失败后立即重试同一 query 全部成功;错误为 `Command Code web search request failed: fetch failed`,面板记录 21.738 s、0 token、$0.0000,发生在拿到 HTTP 状态之前,时间段与 6 个子代理并发检索重叠。**判定**:(a) 不是套餐或凭据问题——那会先给出 401/403 正文;(b) 不是 DSH 的 30 s 工具预算——记录为 21.7 s 且报文是 `fetch failed`;(c) `fetch failed` 是 undici 对连接/读取阶段失败的统一外壳,真实码在 `error.cause`:本机对不可达地址实测 10.5 s 后为 `UND_ERR_CONNECT_TIMEOUT`,双地址族各等一轮可到约 20 s,与 21.7 s 量级一致;DSH 的 `dsh-tool-web` 在多 query 时第一个失败即 abort 兄弟并整体抛出,把一次抖动放大成整批失败。**建议取舍**:采纳 (1) 2–3 次指数退避 + 抖动、(3) 错误串带原因与目标;不采纳 (2) 多 query 部分成功——该策略位于 DSH 的 `dsh-tool-web`,修改它超出本插件边界,应在上游处理;(4) 可配置搜索超时与 keep-alive 未做,前者是新的用户可见契约,后者尚无证据表明连接复用是瓶颈。**约束**:只对「未收到 HTTP 响应」重试,已收到状态(含 401/403/429)不重试——那是服务端的答案;响应体解析/读取失败也不重试,因为响应头已到、请求已被接受并可能执行,重放有重复计费或重复执行的代价。所有非取消的 fetch 拒绝都按瞬时处理(含证书、参数类确定性错误):运行时不替我们分类 `cause`,代价是最多约 1.5 s 后抛出同一错误,换不遗漏未收录的瞬时码。账号解析只在进入重试前做一次;等待监听调用方信号,取消或工具协程超时仍为 `WEB_ABORTED`;退避 500 ms→1 s、±20% 抖动,总尝试 3 次;诊断复用 `transportFailureReason()`/`httpOrigin()`,origin 丢弃路径与查询。**预算边界**:undici 连接超时约 10 s,3 次连接超时尝试加两次退避约 31.5–33 s,已超过 `dsh-tool-web` 默认 30 s 的 `searchTimeoutMs`;慢失败时工具 deadline 落在重试期间,最终以 `WEB_ABORTED` 结束、诊断串不会出现(报告者 21.7 s 首败只剩约 8 s)。未加每次尝试超时(新的用户可见契约),也未按已耗时收敛(会跳过本 issue 的慢失败场景)。**证据边界**:重试用例为合成 fetch,未在报告者的 Windows/代理条件与 6 并发下复验;一次上游已处理但响应丢失的搜索可能被重放,搜索只读且幂等,接受这一代价。 diff --git a/lib/index.d.ts b/lib/index.d.ts index ee72aee..1f794d2 100644 --- a/lib/index.d.ts +++ b/lib/index.d.ts @@ -2220,6 +2220,8 @@ interface CommandCodeSearchProviderDeps { apiBase(): string; /** Injectable fetch for tests; defaults to the global fetch. */ fetchImpl?: typeof fetch; + /** Injectable retry wait for tests; defaults to the module's abortable retry wait. */ + retryWait?: (delayMs: number, signal?: AbortSignal) => Promise; } /** * A `ctx.web` search provider backed by the Command Code Provider API, reusing diff --git a/lib/index.js b/lib/index.js index 752ba71..85cfb03 100644 --- a/lib/index.js +++ b/lib/index.js @@ -11173,6 +11173,28 @@ function unwrapVolatileConfig(config) { * engine range (`engines.dsh`) starts at rc.2, so that reference is the only * shape this module serves — see {@link createCommandCodeSearchSelection}. * +* ## Search retries +* +* `dsh-tool-web` fans several queries out concurrently and aborts the whole +* batch on the FIRST failure (it rethrows that failure; issue #117), so one +* transient connection failure used to discard every result. The endpoint is +* read-only and idempotent, and the failure the issue observed arrived before +* any HTTP status — a network-level `fetch failed`, not a plan/credential +* verdict. This provider therefore repeats a fetch that never received a +* response, up to SEARCH_MAX_ATTEMPTS attempts with jittered +* exponential backoff, and leaves every received status (401/403/429/…) and +* every response-body failure untouched: a received status is the server's +* answer, and a body failure means the request was already accepted and may +* have been executed, so replaying it risks duplicate billing. Cancellation +* always wins — an aborted call (including the tool's cooperative deadline) +* stops before the next attempt and still reports `WEB_ABORTED`. +* +* Every non-cancel `fetch` rejection counts as retryable: the engine does not +* classify the `error.cause` for us, and a deterministic failure (an expired +* certificate, an argument the runtime rejects) merely costs the bounded +* backoff before the same error surfaces. That is cheaper than maintaining a +* transient-code allowlist that silently drops an unlisted blip. +* * @module dsh-commandcode-provider/web-search */ /** Stable id this provider registers under in `ctx.web`. */ @@ -11465,6 +11487,68 @@ const MAX_NUM_RESULTS = 10; const DEFAULT_NUM_RESULTS = 5; /** The endpoint the search POST goes to; `{apiBase}` is prepended. */ const SEARCH_ROUTE = "/alpha/web-search"; +/** Base wait before the second attempt; each later wait doubles. */ +const SEARCH_RETRY_BASE_DELAY_MS = 500; +/** ±fraction applied to each wait so concurrent queries do not retry in lockstep. */ +const SEARCH_RETRY_JITTER_RATIO = .2; +/** +* Wait before the next attempt, doubling from 500 ms with ±20% jitter. +* +* The jitter matters here more than on the model path: a four-query fan-out +* that shares one network hiccup would otherwise reconnect all four at the +* same instant. Exported so the cadence is pinned without waiting on real +* timers. +* +* @param attempt - the failed attempt number, starting at 1. +* @returns the delay in milliseconds. +*/ +function searchRetryDelayMs(attempt) { + const base = SEARCH_RETRY_BASE_DELAY_MS * 2 ** Math.max(0, attempt - 1); + const jitter = 1 + (Math.random() * 2 - 1) * SEARCH_RETRY_JITTER_RATIO; + return Math.round(base * jitter); +} +/** +* Sleep before a retry, resolving early only by rejecting with the caller's +* cancellation. A plain `setTimeout` would let an aborted tool call sit out a +* wait it no longer needs; this keeps the abort responsive and preserves the +* `WEB_ABORTED` taxonomy. +* +* Exported so the cancellation behavior is pinned without waiting out a real +* 500 ms timer: tests can either await this helper directly or wrap it to learn +* when the wait started. +*/ +function searchRetryWait(delayMs, signal) { + if (signal?.aborted === true) return Promise.reject(searchAborted(signal, signal.reason)); + return new Promise((resolve, reject) => { + const onAbort = () => { + clearTimeout(timer); + reject(searchAborted(signal, signal?.reason)); + }; + const timer = setTimeout(() => { + signal?.removeEventListener("abort", onAbort); + resolve(); + }, delayMs); + signal?.addEventListener("abort", onAbort, { once: true }); + }); +} +/** +* The failure a search ends with once every attempt has failed. +* +* `fetch failed` alone cannot distinguish a reset from a connect timeout from +* a dead proxy (issue #117), so the cause code, target origin and elapsed time +* ride along. The origin comes from `httpOrigin()`, which drops paths and +* query strings, so a reverse proxy prefix or a credential never leaks here. +*/ +function transportFailureMessage(error, apiBase, attempts, elapsedMs) { + const origin = httpOrigin(apiBase); + const details = [ + transportFailureReason(error), + origin === void 0 ? void 0 : `目标 ${origin}`, + `耗时 ${elapsedMs}ms`, + `已尝试 ${attempts} 次` + ].filter((detail) => detail !== void 0); + return `Command Code web search request failed: ${readableErrorText(error)}(${details.join(";")})`; +} /** * Clamp a DSH `maxResults` bound into Command Code's 1–10 range, applying the * CLI default of 5 when the caller supplied none. @@ -11524,24 +11608,33 @@ var CommandCodeSearchProvider = class { numResults: clampNumResults(request.maxResults) }; let response; - try { - response = await (this.deps.fetchImpl ?? fetch)(endpoint, { - method: "POST", - headers: { - "Content-Type": "application/json", - ...IDENTITY_ENCODING_HEADER, - Authorization: `Bearer ${key}`, - "x-command-code-version": COMMAND_CODE_CLI_VERSION, - "x-cli-environment": "production", - ...attributionHeaders() - }, - body: JSON.stringify(body), - ...signal !== void 0 ? { signal } : {} - }); - } catch (error) { - if (signal?.aborted === true || isAbortError(error)) throw searchAborted(signal, error); - throw new WebError(`Command Code web search request failed: ${readableErrorText(error)}`, "WEB_PROVIDER_ERROR", { cause: error }); + let lastError; + const startedAt = Date.now(); + for (let attempt = 1; attempt <= 3; attempt += 1) { + throwIfAborted(signal); + try { + response = await (this.deps.fetchImpl ?? fetch)(endpoint, { + method: "POST", + headers: { + "Content-Type": "application/json", + ...IDENTITY_ENCODING_HEADER, + Authorization: `Bearer ${key}`, + "x-command-code-version": COMMAND_CODE_CLI_VERSION, + "x-cli-environment": "production", + ...attributionHeaders() + }, + body: JSON.stringify(body), + ...signal !== void 0 ? { signal } : {} + }); + break; + } catch (error) { + if (signal?.aborted === true || isAbortError(error)) throw searchAborted(signal, error); + lastError = error; + if (attempt === 3) break; + await (this.deps.retryWait ?? searchRetryWait)(searchRetryDelayMs(attempt), signal); + } } + if (response === void 0) throw new WebError(transportFailureMessage(lastError, apiBase, 3, Date.now() - startedAt), "WEB_PROVIDER_ERROR", { cause: lastError }); if (!response.ok) { let message = `Command Code web search failed (HTTP ${response.status})`; try { diff --git a/src/web-search.ts b/src/web-search.ts index 3a89b79..7b772bc 100644 --- a/src/web-search.ts +++ b/src/web-search.ts @@ -16,6 +16,28 @@ * engine range (`engines.dsh`) starts at rc.2, so that reference is the only * shape this module serves — see {@link createCommandCodeSearchSelection}. * + * ## Search retries + * + * `dsh-tool-web` fans several queries out concurrently and aborts the whole + * batch on the FIRST failure (it rethrows that failure; issue #117), so one + * transient connection failure used to discard every result. The endpoint is + * read-only and idempotent, and the failure the issue observed arrived before + * any HTTP status — a network-level `fetch failed`, not a plan/credential + * verdict. This provider therefore repeats a fetch that never received a + * response, up to SEARCH_MAX_ATTEMPTS attempts with jittered + * exponential backoff, and leaves every received status (401/403/429/…) and + * every response-body failure untouched: a received status is the server's + * answer, and a body failure means the request was already accepted and may + * have been executed, so replaying it risks duplicate billing. Cancellation + * always wins — an aborted call (including the tool's cooperative deadline) + * stops before the next attempt and still reports `WEB_ABORTED`. + * + * Every non-cancel `fetch` rejection counts as retryable: the engine does not + * classify the `error.cause` for us, and a deterministic failure (an expired + * certificate, an argument the runtime rejects) merely costs the bounded + * backoff before the same error surfaces. That is cheaper than maintaining a + * transient-code allowlist that silently drops an unlisted blip. + * * @module dsh-commandcode-provider/web-search */ @@ -26,6 +48,7 @@ import { COMMAND_CODE_CLI_VERSION } from './adapter.ts' import { isVolatileRef, volatileWrite } from './config-volatile.ts' import { readableErrorText } from './readable-error.ts' import { IDENTITY_ENCODING_HEADER } from './response-encoding.ts' +import { httpOrigin, transportFailureReason } from './transport-failure.ts' /** Stable id this provider registers under in `ctx.web`. */ export const COMMANDCODE_SEARCH_PROVIDER_ID = 'commandcode' @@ -439,6 +462,92 @@ const DEFAULT_NUM_RESULTS = 5 /** The endpoint the search POST goes to; `{apiBase}` is prepended. */ const SEARCH_ROUTE = '/alpha/web-search' +/** + * Attempts per search: the first try plus two retries. + * + * dsh fans a multi-query call out concurrently and aborts the siblings of the + * first failure, so retrying HERE is what keeps one connection blip from + * discarding a four-query research step (issue #117). Two retries cover the + * common transient cases. The budget deliberately brushes the web tool's + * default 30 s cooperative deadline, so a slow connect-timeout failure can hit + * that deadline mid-retry and surface as `WEB_ABORTED` instead of the terminal + * diagnosis; the full arithmetic and the rejected alternatives live in + * docs/agent-reference/accounts-and-retry.md. + */ +export const SEARCH_MAX_ATTEMPTS = 3 + +/** Base wait before the second attempt; each later wait doubles. */ +const SEARCH_RETRY_BASE_DELAY_MS = 500 + +/** ±fraction applied to each wait so concurrent queries do not retry in lockstep. */ +const SEARCH_RETRY_JITTER_RATIO = 0.2 + +/** + * Wait before the next attempt, doubling from 500 ms with ±20% jitter. + * + * The jitter matters here more than on the model path: a four-query fan-out + * that shares one network hiccup would otherwise reconnect all four at the + * same instant. Exported so the cadence is pinned without waiting on real + * timers. + * + * @param attempt - the failed attempt number, starting at 1. + * @returns the delay in milliseconds. + */ +export function searchRetryDelayMs(attempt: number): number { + const base = SEARCH_RETRY_BASE_DELAY_MS * 2 ** Math.max(0, attempt - 1) + const jitter = 1 + (Math.random() * 2 - 1) * SEARCH_RETRY_JITTER_RATIO + return Math.round(base * jitter) +} + +/** + * Sleep before a retry, resolving early only by rejecting with the caller's + * cancellation. A plain `setTimeout` would let an aborted tool call sit out a + * wait it no longer needs; this keeps the abort responsive and preserves the + * `WEB_ABORTED` taxonomy. + * + * Exported so the cancellation behavior is pinned without waiting out a real + * 500 ms timer: tests can either await this helper directly or wrap it to learn + * when the wait started. + */ +export function searchRetryWait(delayMs: number, signal?: AbortSignal): Promise { + if (signal?.aborted === true) return Promise.reject(searchAborted(signal, signal.reason)) + return new Promise((resolve, reject) => { + const onAbort = (): void => { + clearTimeout(timer) + reject(searchAborted(signal, signal?.reason)) + } + const timer = setTimeout(() => { + signal?.removeEventListener('abort', onAbort) + resolve() + }, delayMs) + signal?.addEventListener('abort', onAbort, { once: true }) + }) +} + +/** + * The failure a search ends with once every attempt has failed. + * + * `fetch failed` alone cannot distinguish a reset from a connect timeout from + * a dead proxy (issue #117), so the cause code, target origin and elapsed time + * ride along. The origin comes from `httpOrigin()`, which drops paths and + * query strings, so a reverse proxy prefix or a credential never leaks here. + */ +function transportFailureMessage( + error: unknown, + apiBase: string, + attempts: number, + elapsedMs: number, +): string { + const origin = httpOrigin(apiBase) + const details = [ + transportFailureReason(error), + origin === undefined ? undefined : `目标 ${origin}`, + `耗时 ${elapsedMs}ms`, + `已尝试 ${attempts} 次`, + ].filter((detail): detail is string => detail !== undefined) + return `Command Code web search request failed: ${readableErrorText(error)}(${details.join(';')})` +} + /** Per-request facts the provider needs, all injected so the class stays cordis-free and testable. */ export interface CommandCodeSearchProviderDeps { /** 使用搜索已捕获的网关解析账号,探测不能在等待后转到另一个网关。 */ @@ -447,6 +556,8 @@ export interface CommandCodeSearchProviderDeps { apiBase(): string /** Injectable fetch for tests; defaults to the global fetch. */ fetchImpl?: typeof fetch + /** Injectable retry wait for tests; defaults to the module's abortable retry wait. */ + retryWait?: (delayMs: number, signal?: AbortSignal) => Promise } /** @@ -523,27 +634,42 @@ export class CommandCodeSearchProvider implements WebSearchProvider { numResults: clampNumResults(request.maxResults), } - let response: Response - try { - response = await (this.deps.fetchImpl ?? fetch)(endpoint, { - method: 'POST', - headers: { - 'Content-Type': 'application/json', - ...IDENTITY_ENCODING_HEADER, - Authorization: `Bearer ${key}`, - 'x-command-code-version': COMMAND_CODE_CLI_VERSION, - 'x-cli-environment': 'production', - ...attributionHeaders(), - }, - body: JSON.stringify(body), - ...signal !== undefined ? { signal } : {}, - }) - } catch (error: unknown) { - if (signal?.aborted === true || isAbortError(error)) throw searchAborted(signal, error) + let response: Response | undefined + let lastError: unknown + const startedAt = Date.now() + for (let attempt = 1; attempt <= SEARCH_MAX_ATTEMPTS; attempt += 1) { + throwIfAborted(signal) + try { + response = await (this.deps.fetchImpl ?? fetch)(endpoint, { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + ...IDENTITY_ENCODING_HEADER, + Authorization: `Bearer ${key}`, + 'x-command-code-version': COMMAND_CODE_CLI_VERSION, + 'x-cli-environment': 'production', + ...attributionHeaders(), + }, + body: JSON.stringify(body), + ...signal !== undefined ? { signal } : {}, + }) + break + } catch (error: unknown) { + // Cancellation is never retried, and a received-response path below is + // never reached: this catch only covers a fetch that got no response. + // Every such rejection counts as transient; see the module comment for + // why a code allowlist would be worse. + if (signal?.aborted === true || isAbortError(error)) throw searchAborted(signal, error) + lastError = error + if (attempt === SEARCH_MAX_ATTEMPTS) break + await (this.deps.retryWait ?? searchRetryWait)(searchRetryDelayMs(attempt), signal) + } + } + if (response === undefined) { throw new WebError( - `Command Code web search request failed: ${readableErrorText(error)}`, + transportFailureMessage(lastError, apiBase, SEARCH_MAX_ATTEMPTS, Date.now() - startedAt), 'WEB_PROVIDER_ERROR', - { cause: error }, + { cause: lastError }, ) } diff --git a/tests/web-search.test.ts b/tests/web-search.test.ts index 1f8c478..eaf3dc3 100644 --- a/tests/web-search.test.ts +++ b/tests/web-search.test.ts @@ -26,7 +26,7 @@ import { fileURLToPath } from 'node:url' import type { Context } from '@deepseek-ai/cordis' import type { WebRuntime } from '@deepseek-ai/dsh-web' import { - CommandCodeSearchProvider, createCommandCodeSearchSelection, + CommandCodeSearchProvider, SEARCH_MAX_ATTEMPTS, createCommandCodeSearchSelection, searchRetryDelayMs, searchRetryWait, } from '../src/web-search.ts' import type { WebSelectionSettings } from '../src/web-search.ts' import { volatileWrite } from '../src/config-volatile.ts' @@ -275,6 +275,179 @@ test('throws WEB_ABORTED when the caller aborts mid-flight', async () => { ) }) +/** Node's `fetch` rejection shape: a bare `TypeError` with the real code one level down. */ +function fetchFailure(code: string): TypeError { + return new TypeError('fetch failed', { + cause: Object.assign(new Error(`synthetic ${code}`), { code }), + }) +} + +test('retries a fetch that got no response and resolves the key only once', async () => { + // issue #117: one transient connection failure used to fail the whole + // multi-query batch, and the same query succeeded when retried by hand. + let calls = 0 + let resolves = 0 + const stub = makeFetch(() => { + calls += 1 + if (calls === 1) throw fetchFailure('UND_ERR_CONNECT_TIMEOUT') + return asResponse(okBody([{ title: 'T', url: 'https://a.example', snippet: 'S' }])) + }) + const provider = new CommandCodeSearchProvider({ + resolveKey: async () => { resolves += 1; return 'cc-key-123' }, + apiBase: () => 'https://api.commandcode.ai', + fetchImpl: stub.fetchImpl, + retryWait: async () => {}, + }) + const result = await provider.search({ query: 'q' }) + + assert.equal(calls, 2) + assert.equal(resolves, 1, '账号解析只在进入重试前做一次') + assert.deepEqual(result.sources, [{ url: 'https://a.example', title: 'T', snippet: 'S' }]) +}) + +test('stops at SEARCH_MAX_ATTEMPTS and names the cause, target and attempt count', async () => { + let calls = 0 + const stub = makeFetch(() => { + calls += 1 + throw fetchFailure('UND_ERR_CONNECT_TIMEOUT') + }) + const provider = new CommandCodeSearchProvider({ + resolveKey: async () => 'cc-key-123', + apiBase: () => 'https://api.commandcode.ai', + fetchImpl: stub.fetchImpl, + retryWait: async () => {}, + }) + await assert.rejects( + () => provider.search({ query: 'q' }), + (error: unknown) => { + const e = error as { code?: string; message?: string } + assert.equal(e.code, 'WEB_PROVIDER_ERROR') + assert.equal(calls, SEARCH_MAX_ATTEMPTS) + assert.match(e.message ?? '', /Command Code web search request failed: fetch failed/) + assert.match(e.message ?? '', /连接超时(UND_ERR_CONNECT_TIMEOUT)/) + assert.match(e.message ?? '', /目标 https:\/\/api\.commandcode\.ai/) + assert.match(e.message ?? '', new RegExp(`已尝试 ${SEARCH_MAX_ATTEMPTS} 次`)) + return true + }, + ) +}) + +test('does not retry once an HTTP status arrived: a 403 is an answer, not a blip', async () => { + let calls = 0 + const stub = makeFetch(() => { + calls += 1 + return asResponse({ status: 403, body: { error: 'upgrade_required' } }) + }) + const provider = new CommandCodeSearchProvider({ + resolveKey: async () => 'cc-key-123', + apiBase: () => 'https://api.commandcode.ai', + fetchImpl: stub.fetchImpl, + retryWait: async () => { throw new Error('a received response must never be retried') }, + }) + await assert.rejects( + () => provider.search({ query: 'q' }), + (error: unknown) => { + assert.equal((error as { code?: string }).code, 'WEB_PROVIDER_ERROR') + return true + }, + ) + assert.equal(calls, 1) +}) + +test('an abort during the retry wait stops it instead of making another request', async () => { + let calls = 0 + const stub = makeFetch(() => { calls += 1; throw fetchFailure('ECONNRESET') }) + const controller = new AbortController() + let waitStarted!: () => void + const started = new Promise(resolve => { waitStarted = resolve }) + const provider = new CommandCodeSearchProvider({ + resolveKey: async () => 'cc-key-123', + apiBase: () => 'https://api.commandcode.ai', + fetchImpl: stub.fetchImpl, + // Delegate to the production wait and signal when it is armed, so the test + // never races a real 500 ms timer against a short abort timer. + retryWait: async (delayMs, signal) => { + waitStarted() + return searchRetryWait(delayMs, signal) + }, + }) + const pending = provider.search({ query: 'q' }, controller.signal) + await started + controller.abort() + await assert.rejects( + () => pending, + (error: unknown) => { + assert.equal((error as { code?: string }).code, 'WEB_ABORTED') + return true + }, + ) + assert.equal(calls, 1, '取消后不得再发请求') +}) + +test('the default retry wait (no injected seam) is abortable', { timeout: 5000 }, async (t) => { + // The injected-seam test above proves the cancellation semantics of the + // production wait; this one keeps the `retryWait ?? searchRetryWait` default + // itself covered. Timers are faked so the 500 ms wait cannot fire, and a + // captured real timer guards against a regression that would hang instead. + const realSetTimeout = globalThis.setTimeout + const realClearTimeout = globalThis.clearTimeout + t.mock.timers.enable({ apis: ['setTimeout'] }) + let calls = 0 + const stub = makeFetch(() => { calls += 1; throw fetchFailure('ECONNRESET') }) + const provider = new CommandCodeSearchProvider({ + resolveKey: async () => 'cc-key-123', + apiBase: () => 'https://api.commandcode.ai', + fetchImpl: stub.fetchImpl, + }) + const controller = new AbortController() + const pending = provider.search({ query: 'q' }, controller.signal) + // Let the first rejection reach the default wait and arm the faked timer. + await new Promise(resolve => setImmediate(resolve)) + let guardTimer: ReturnType | undefined + const guard = new Promise((_, reject) => { + guardTimer = realSetTimeout(() => reject(new Error('默认等待未响应取消')), 1000) + }) + try { + controller.abort() + await assert.rejects( + () => Promise.race([pending, guard]), + (error: unknown) => { + assert.equal((error as { code?: string }).code, 'WEB_ABORTED') + return true + }, + ) + } finally { + if (guardTimer !== undefined) realClearTimeout(guardTimer) + } + assert.equal(calls, 1, '取消后不得再发请求') +}) + +test('retry backoff doubles from 500 ms and stays inside the jitter bound', () => { + for (let round = 0; round < 50; round += 1) { + const first = searchRetryDelayMs(1) + const second = searchRetryDelayMs(2) + assert.ok(first >= 400 && first <= 600, `first wait ${first}`) + assert.ok(second >= 800 && second <= 1200, `second wait ${second}`) + } +}) + +test('the retry wait receives the 500 ms then 1 s sequence, not a fixed delay', async () => { + let calls = 0 + const delays: number[] = [] + const stub = makeFetch(() => { calls += 1; throw fetchFailure('ECONNRESET') }) + const provider = new CommandCodeSearchProvider({ + resolveKey: async () => 'cc-key-123', + apiBase: () => 'https://api.commandcode.ai', + fetchImpl: stub.fetchImpl, + retryWait: async (delayMs) => { delays.push(delayMs) }, + }) + await assert.rejects(() => provider.search({ query: 'q' })) + assert.equal(calls, SEARCH_MAX_ATTEMPTS) + assert.equal(delays.length, SEARCH_MAX_ATTEMPTS - 1) + assert.ok(delays[0]! >= 400 && delays[0]! <= 600, `first wait ${delays[0]}`) + assert.ok(delays[1]! >= 800 && delays[1]! <= 1200, `second wait ${delays[1]}`) +}) + test('available() is false for a blank or non-parseable apiBase', async () => { const bad = new CommandCodeSearchProvider({ resolveKey: async () => 'k', apiBase: () => '', fetchImpl: makeFetch(() => asResponse(okBody([]))).fetchImpl,