Skip to content

CI and test hardening: no timing assertions, no network in unit tests, release gate trusts CI and caches builds - #48

Merged
scottypate merged 1 commit into
mainfrom
ci-test-hardening
Sep 11, 2026
Merged

scottypate merged 1 commit into
mainfrom
ci-test-hardening

Conversation

@scottypate

Copy link
Copy Markdown
Contributor

Why

Two flaky tests failed CI three times today, and a release takes ~45 minutes: the release workflow re-runs the full suite cold (13 min) and then compiles bundled DuckDB from scratch on three runners (13–15 min each) with caching deliberately off. PR CI itself is ~3 minutes.

Tests

  • Bootstrap canary (engine::tests): the default suite now checks the behaviour (LIMIT 0 creates the empty table with the SELECT's schema). The timing claim — every shape of it failed on a loaded 2-vCPU runner while the gate held — is an #[ignore]d test run by hand when the planner or DuckDB moves.
  • run_git timeout test hung on github.com; it now hangs on a local ssh listener that never answers. No network in unit tests.
  • Integration serve readiness (20 s → 60 s) and SIGTERM drain (5 s → 30 s) bounds sized for a shared runner; the log sentences, not the clock, prove the drain ran.

Release workflow

  • The gate no longer re-runs the suite. It waits for the CI workflow run on the tagged commit (pushed together with the tag by make release) and requires success. Same safety property, 13 minutes less.
  • Each build target gets a Swatinem/rust-cache keyed on the release profile and target, so the dependency tree (DuckDB above all) persists across tags. A warm release links datamk; a dependency bump rebuilds.

Expected release time after this lands and the caches are seeded once: roughly 5 minutes plus the macOS runner's link.

Tests: 914 unit (1 ignored) + 33 integration, clippy clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KLsmJHt8RwDUfoN7iaemBy

… no network in unit tests, release gate trusts CI and caches builds

- bootstrap canary: the default suite checks behaviour only; the timing claim
  is an #[ignore]d test run by hand
- git timeout test hangs on a local ssh listener instead of github.com
- integration readiness/drain bounds sized for a shared 2-vCPU runner
- release.yml: the gate waits for and requires the CI run on the tagged
  commit instead of re-running the suite cold (13 min); per-target
  rust-cache for the release profile so DuckDB is not rebuilt every tag

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KLsmJHt8RwDUfoN7iaemBy
@scottypate
scottypate merged commit d226dfa into main Sep 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant