Skip to content

test: cut slow Rust test wall time - #260

Merged
kumanday merged 6 commits into
mainfrom
test/speed-up-slow-rust-tests
Nov 8, 2026
Merged

kumanday merged 6 commits into
mainfrom
test/speed-up-slow-rust-tests

Conversation

@kumanday

@kumanday kumanday commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

Follow-up to #259. Cuts the Rust test wall time by fixing tests that waited on clocks or shells instead of doing work. Every assertion is kept.

  • run Merging tests (30s each → under 1s). Both tests waited out the scheduler's hardcoded 30s running-state refresh before the mock tracker's Merging transition became visible. SchedulerConfig now carries running_state_refresh_interval_ms (default 30s, unchanged for operators), and opensymphony run accepts OPENSYMPHONY_RUNNING_STATE_REFRESH_MS as a positive-millisecond override; invalid values are ignored with a warning. The two tests set it to 50ms. The snapshot poll in their wait helpers drops from 250ms to 50ms. The "exactly one /interrupt" assertion still holds (10/10 local repeats).
  • doctor and install tests (5–10s locally → under 0.5s). real_bash_path() ran bash -lc, which sources the developer's login profile (about 5s on a typical macOS setup). A non-login bash -c resolves the same binary. CI was already fast here.
  • gateway code-graph contract test (21.4s → 8.0s + 13.4s on CI). The two 505-document atlas fixtures (cap and dedupe at the 500-record contract limit) use their own repository ids, so they move to gateway_code_graph_atlas_caps_and_dedupes_large_repositories with a fresh store and run in parallel with the endpoint checks. Total test time is unchanged, but the longest single test is shorter.
  • CI cache bug from chore(ci): speed up Rust CI and report build and test timings #259. A pull request that restores the main cache reused the libduckdb-sys build-script output (so the download was skipped), while rust-cache had pruned target/duckdb-download. The test binaries then failed to link (unable to find library -lduckdb). The first run of this branch hit it. rust-test now caches that directory and uses a new cache key so the next main run saves it.

Docs: docs/operations.md, docs/configuration.md, docs/testing-and-operations.md.

Not changed (production-bound)

These are slow because of production code paths, not test waits:

  • workspace_manager::parent_execution_root_reuses_three_repositories_and_preserves_children: about 4,600 git subprocesses from checkout verification (for example, 152 git fsck runs). It takes 9–11s on Linux CI and 85–95s on macOS, where each spawn goes through the Xcode git shim.
  • target_branch_index_bootstraps_and_advances_immutable_snapshots and the other memory code-graph tests: DuckDB per-row INSERT OR REPLACE (planned as a MERGE, about 0.8ms per statement), five staleness UPDATEs per document, a fresh database instance plus migrate_index per helper call, and per-file git cat-file spawns during indexing. prepare_cached gave no measurable gain. Speeding these up means batching writes and sharing connections in opensymphony-memory.

Evidence

CI rust-test job, latest main push run before this change vs this branch (both nextest ci profile, 4-vCPU runner):

Test Before After
run::run_interrupts_active_codex_stdio_worker_when_tracker_reports_merging 30.4s 0.4s
run::run_recovers_human_review_worker_and_interrupts_when_tracker_reports_merging 30.2s 0.2s
gateway::gateway_serves_code_graph_contract_endpoints 21.4s 8.0s
gateway::gateway_code_graph_atlas_caps_and_dedupes_large_repositories (split out) — 13.4s
doctor::doctor_accepts_present_prerequisites_from_path 0.4s 0.1s
workspace_manager::parent_execution_root_reuses_three_repositories_and_preserves_children 10.9s 9.3s (unchanged code)
opensymphony_memory::code_graph_tests::target_branch_index_bootstraps_and_advances_immutable_snapshots 18.3s 18.9s (unchanged code)
nextest wall time 94.0s 79.4s
summed test time 354.6s 294.2s
tests ≥ 30s / ≥ 5s 2 / 10 0 / 10

Local macOS (Apple Silicon, shared machine under load, so absolute times are noisy):

Test Before After
run Merging tests (each) 30.7–31.0s 0.6–1.2s
doctor::doctor_accepts_present_prerequisites_from_path 9.7s 0.3s
doctor::doctor_bootstraps_missing_managed_local_tooling_into_explicit_configured_dir 9.7s 0.3s
install::* (5 tests) 5.1–5.8s 0.1–0.4s
tests ≥ 30s 3 1 (the workspace test above)

Validation:

  • cargo nextest run --no-default-features --features duckdb-prebuilt --profile ci: 2265/2267 pass locally. The two failures are the known macOS-only migration tests (/var symlink), which fail on main too. All tests pass on CI.
  • cargo fmt --all --check and cargo clippy-dev are clean.
  • New unit test running_state_refresh_override_accepts_only_positive_milliseconds.

…upt tests

The two Merging-supersede run tests waited out the scheduler's hardcoded
30s running-state refresh before observing the tracker transition.
SchedulerConfig now carries the cadence (default 30s) and
OPENSYMPHONY_RUNNING_STATE_REFRESH_MS overrides it for opensymphony run;
the tests set it to 50ms.
doctor and install tests located the real bash with `bash -lc`, which
sources the developer's profile and took about 5s per lookup on macOS.
A non-login shell resolves the same binary.
The contract-endpoint test persisted two 505-document fixtures to cross
the 500-record atlas cap. Those scenarios use their own repository ids,
so they move to a separate test with a fresh store and run in parallel.
A pull request that restored the main cache reused the duckdb
build-script output, which skipped the download, while rust-cache had
pruned target/duckdb-download. Linking the test binaries then failed
with `unable to find library -lduckdb`.
Existing caches predate the cache-directories entry; a new key forces one
cold build that saves the library alongside the target dir.
@kumanday
kumanday marked this pull request as ready for review November 8, 2026 20:39
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-08T20:39:59.131284Z bd68af7 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kumanday
kumanday merged commit c8b8450 into main Nov 8, 2026
7 checks passed
@kumanday
kumanday deleted the test/speed-up-slow-rust-tests branch November 8, 2026 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant