test(e2e): migrate e2e suite to agentry - #3
Merged
Merged
Conversation
Replace the hand-rolled Vitest suite that spawned `claude` directly with
tests written against agentry ("Playwright for AI Agents"). Each scenario
drives a scheduler slash command through a real Claude agent — the plugin is
loaded via the new `--plugin-dir` passthrough — and asserts on the agent's
reply with the same tolerant, multi-alternative matching as before.
The harness lives in `e2e/` as a self-contained pnpm project that links
agentry via `file:` (with `pnpm.overrides` rewriting its `workspace:*` deps).
This keeps agentry entirely out of the npm-managed root so `npm ci` in CI is
unaffected. Runs live and local-only, matching the prior suite (e2e is not in
CI); `test:e2e` now shells into the e2e project.
Constraint: scheduler is npm + CI uses `npm ci`, which cannot resolve
agentry's workspace:* deps — hence the isolated pnpm sub-project.
Constraint: agentry's directory sandbox isolates cwd, not $HOME; the scheduler
reads global ~/.claude and ~/Library/LaunchAgents, so empty/populated
scenarios assume a machine with no pre-existing scheduled tasks.
Rejected: add agentry to the root package.json | breaks `npm ci` on all jobs
Rejected: $HOME remap for determinism | risks breaking the claude CLI's own auth
Confidence: high
Scope-risk: narrow
Not-tested: full green live run (env-coupled to global scheduler state; the
plugin-load path and assertions are verified live on representative scenarios)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…ER_STATE_DIR
The scheduler's global state (schedules, logs, history) is fixed at ~/.claude,
which makes the read commands impossible to test without touching real machine
state. Add a `getStateDir()` seam honoring `CLAUDE_SCHEDULER_STATE_DIR` (default
unchanged) in the library, and resolve the same env var in the status/list/
history/logs command prompts (Bash `${CLAUDE_SCHEDULER_STATE_DIR:-$HOME/.claude}`
expansion). OS registration (launchd/cron) still uses the real home.
Constraint: the slash commands read literal paths from prompts, not the TS
library — so the env seam has to live in the prompts too, not just config.ts.
Constraint: must not change behavior when the var is unset (default ~/.claude).
Rejected: remap $HOME to isolate state | breaks the claude CLI's own OAuth auth
Confidence: high
Scope-risk: moderate
Directive: keep the command prompts and getStateDir() in sync — both resolve the
same env var; OS-registration paths intentionally do NOT honor it
Not-tested: Linux cron command paths (only the macOS read paths are exercised)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Point CLAUDE_SCHEDULER_STATE_DIR (via agentry's per-run env) at a `state/` subdir of each scenario's sandbox and seed fixtures there, so empty/populated states are hermetic and independent of the machine's real ~/.claude — verified live (8/8 green) on a machine with pre-existing scheduled tasks. Auth-safe: $HOME is untouched, so the claude CLI keeps its credentials. Drop the /scheduler:add scenario: it performs real OS registration the state dir can't isolate (covered by the unit suite instead). The populated history scenario now asserts the seeded records rather than just "runs without error". Rejected: $HOME remap for isolation | drops the claude CLI's OAuth auth Confidence: high Scope-risk: narrow Not-tested: mutating commands (intentionally excluded from the hermetic suite) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A machine with orphaned real launchd plists could make /scheduler:status lead with an "orphaned registrations" report and omit the empty-state line (the OS registration dir isn't isolatable by CLAUDE_SCHEDULER_STATE_DIR). Guarantee the `Tasks: none configured` line is always emitted first (orphan notes stay as a secondary append), and broaden the assertion to accept the orphan-report phrasing. Enable `retries: 2` in the e2e config to absorb the residual, inherent phrasing variance of live model runs. Verified 8/8 green across repeated runs. Rejected: assert only on final phrasing | live output varies run to run Confidence: high Scope-risk: narrow Not-tested: Linux (cron) status path Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Migrate the end-to-end suite from a hand-rolled Vitest harness that spawned
claudedirectly to tests written against agentry ("Playwright for AI Agents"), and make the scenarios deterministic and hermetic.How it works
e2e/. Each scenario drives a scheduler slash command through a real Claude agent and asserts on the reply with tolerant, multi-alternative matching.file:../../agentry/packages/cliwithpnpm.overridesrewriting itsworkspace:*deps to the sibling paths — keeping agentry out of the npm-managed root sonpm ciin CI is unaffected.~/.claudestate that a cwd sandbox can't isolate. This PR adds aCLAUDE_SCHEDULER_STATE_DIRoverride (getStateDir()+ thestatus/list/history/logscommand prompts); each scenario points it at a per-sandboxstate/dir and seeds fixtures there. Auth-safe —$HOMEis untouched, so theclaudeCLI keeps its OAuth credentials (a$HOMEremap drops them).Scope
status/list/history/logs)./scheduler:addscenario — it does real OS registration (launchd/cron) the state dir can't isolate; still covered by the unit suite.Verification
npm run typecheck/lint/test(340) /build— all green (root manifest unchanged,npm ci-safe).Removed
vitest.config.e2e.ts,src/__tests__/e2e/{commands.test.ts,helpers.ts,fixtures.ts};test:e2enow shells intoe2e/.🤖 Generated with Claude Code