[ENG-3418] Cut cloud turn wait before the first reply

[tino097]

https://linear.app/mindsdb/issue/ENG-3418

What changed

• Prefetch cloud connector credentials with bounded concurrency: a cloud turn with OAuth connectors fetches turn-key tokens up to four at a time before registration and prompt building, instead of one auth round trip per connection in sequence
• Bound memory rule retrieval and select rules by index: the rule filter returns {"keep": [n, ...]} under a 3 s deadline; a timeout, malformed, empty, out of range or truncated answer keeps every conditional rule, an explicit {"keep": []} drops them, and Always/Never rules are always kept
• Apply selected reasoning effort in cloud Anton sessions: moved to ENG-3419, removed again below
• Correct the truncation comment and loosen timing asserts in retrieval tests: no behavior change
• Unwrap one code fence around the rule selection before parsing: a fenced answer still filters instead of keeping every rule
• Say which prefetch failures are left for a later read: comment only
• Drop stale rule filter figures and stop pinning the output cap in tests: review follow up, no behavior change
• Prefetch every connection through the bounded pool and test the cache, not the thread: one fetch path; an unexpected error on a single connection is now logged and retried by the later read instead of raised during prefetch
• Move selected reasoning effort out of this change into its own: the net diff no longer touches reasoning effort
• Keep the rule filter output cap at 4096 so reasoning tokens cannot disable it: the cap is unchanged from staging
• Drop outcome shares measured on the old echo filter: comment only
• Report a provider timeout in the rule filter as an error, not a deadline hit: timeout now means the 3 s deadline was reached
• Import test dependencies at the top of the retrieval and prefetch tests: tests only
• Test that a rule selection at the output cap is rejected even when it reports stop: tests only

Why this matters

Follow up to #518 and #469. Connector tokens were fetched one after another (eight connections at 0.2 s each took 1.64 s in a fake benchmark, 0.41 s with prefetch). The rule filter had no deadline of its own, so a slow filter held the first reply token.

Notes for the reviewer

• Squash merge, please. The history adds reasoning effort and then removes it; the net diff has none.
• Credential isolation is unchanged. Workers only call _fetch_uncached. The calling thread writes the cache after the pool joins. No ContextVar, os.environ or secret registry access happens off thread.
• Rule selection changes which rules reach the prompt. An eval run showing answers did not get worse, and staging time to first reply on a turn with many rules, are still to do.
• Analytics keys are unchanged. A new timeout outcome is added; truncation still reports kept_all_no_match, and stop_reason with output_tokens tells a cap hit from a malformed answer.

Ships with

• [ENG-3418] Make the routing budget tunable and measure its full latency cost cowork-server#663 (routing gate budget). Either order.

Checks

Check	Observed result
uv run pytest (full suite, after the effort removal)	5007 passed, 91 skipped
uv run pytest (full suite, after all review fixes)	5008 passed, 91 skipped
uv run pytest tests/test_cortex.py after the last test	90 passed
New regression tests run against the prior code	fail as expected (barrier, index selection, deadline, fence, provider timeout)

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-12T18:02:38.071597Z	9346043	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[ZoranPandovski]

The ENG-3093 items are already on staging: #518 runs the global and project rule filters in parallel and builds the memory prompt once per turn, and #469 removed the router call. So this PR is follow-up latency work, and it would be clearer under its own ticket. Could you also move reasoning_effort into a separate PR? It isn't related to latency and it changes the turn request shared with the controller and cowork-server, so it's easier to review and roll back on its own. The token prefetch and the 3 s limit on the rule filter look good to me; details inline.

[tino097]

Eval results for the rule filter change.

Setup: cowork_evals, latest:sonnet, a 46-rule fixture over the filter budget (42 When rules, each with a checkable effect), 30 tasks x 3 repeats per side. Current staging against staging with this PR merged in.

	PR	staging
checks pass	89/90	89/90
first text p50 / p95	7.4 s / 16.1 s	7.2 s / 16.7 s
rule filter p95	2.7 s	7.8 s
slowest rule filter call	3.0 s	15.7 s

• Answer quality: 0.0 pp difference, 95% CI [-3.3, +3.3] pp. The run could detect a 4.4 pp drop.
• Time to first text: +0.25 s, CI [-0.66, +1.05] s, so no measurable change.
• The 3 s limit fired on 15% of filter calls; those turns kept every rule.

Production, last 7 days (rule_retrieval events, filter calls that chose rules: 276 calls from 21 installs). These are desktop installs; cloud pods reported no filter calls in the window, so the cloud pick is still unmeasured:

median	9.3 s
p95	60.4 s
max	178 s
over 3 s	96%

The cap would fire on nearly every real filter call and bound waits that today run from 20 s to several minutes before the first reply. The calls that do finish take 2.0 to 2.8 s, so the margin is thin; a follow-up should make the filter itself faster rather than only cap it.

[ea-rus]

If we have a lot of rules accumulated - timeout probably will be triggered on every turn
Maybe to add another condition, to not run it if the count of rules is greater than X
And the fix the the problem with count of rules:

• fix the memory compaction on cloud
• brainstorm what we can do if there are lot of rules after compaction: more agressive compaction, split rules into groups and using Jev or llm choose the proper group to find rules