test(fixtures): restore skills-custom fixture with assertions - #8680
Conversation
Seed release-notes as one live user skill and flaky-triage as one crystallized candidate under skills/auto/.pending/. The consumer test proves the real live and pending readers split them correctly and exercises both approve and dismiss paths without colliding with packaged builtin skill names. Its first assertions prove the nested .pending directory and .meta.json survive the seeded-home copy; sdist and wheel archive checks separately prove both dot paths survive packaging.
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Additive test-only fixture validated against the real production readers; no new surface, no contract change, trivially reversible. Suggestions
[DESIGN-REVIEWED] 664b38b |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All contract-required checks are done: I counted consumers of the fixture, verified the schema claims against First-Principles-Verdict: CONCERNS The fixture-drift cause this PR names — readers no longer parsing what fixtures carry — is fixed only in the new fixture; What this change shipsIntent: give developers a seedable home that exercises the Skills tab's live-vs-pending split, validated against the real readers — an ADDITION (a "restore" plus its first consumer test).
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 664b38b |
NicholasRBowers
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (6 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only change — restores the skills-custom seeded-home fixture (data files under tests_fixtures/) with a consumer test covering the live/pending skill readers and approve/dismiss paths; no runtime code touched.
Summary
Restores the
skills-customseeded-home fixture and gives it a consumer test, so the fixture isvalidated against the production skills readers instead of sitting unexercised on disk.
The fixture seeds one live user skill (
release-notes) plus one crystallized candidate awaitingreview (
flaky-triage) underskills/auto/.pending/, including the.meta.jsonthe pending listconsumes. That is the smallest home that exercises the Skills tab's live-vs-pending split and both
of its terminal actions.
Consumer test
test/test_fixture_skills_custom.pydrives the realSkillsLoaderagainst the seeded home — nomocks, no reimplemented parsing:
.pending/flaky-triagedirectory and its
.meta.jsonboth exist in the materialized home.list_skills()returns exactly{release-notes};auto/flaky-triageisabsent from the live set, and
list_pending_skills()returns exactly[flaky-triage].name,description,triggers,source(crystallize) andhas_scriptsare asserted against literals, andget_pending_skill()is checked for the sameslug/scriptsvalues, so a silent key rename in the reader fails the test.approve_pending_skill()returnsauto/flaky-triage, empties the pending list, and moves thecandidate into the live set — asserted as the exact set
{release-notes, auto/flaky-triage}.dismiss_pending_skill()is exercised in a second, independent seeded home (approve anddismiss are mutually exclusive on one candidate): it returns
True, empties the pending list, andleaves the live set unchanged.
the names discovered by
_iter_skill_files(_BUILTIN_SKILLS_DIR), so the fixture cannot pass byaccidentally shadowing a skill that ships in the package, and adding a builtin later cannot break
it.
Schema repairs
The fixture is aligned to what the current production readers actually parse, not to an older
fixture's shape:
fixture.yamlcarries the currentschema-version: 2026-04-28and thefixture-name:key thatpod/runtime.pyscans for.config.jsonuses the current workspace keyworkspaces.default.dir, matchingWorkspaceConfig.dirand the loader'sraw_value.get("dir", "workspace"), rather than the legacypathspelling still present in older fixtures..meta.jsoncarries exactly the key set the pending reader consumes —slug,name,source,created_at,description,triggers,has_scripts,scripts— so the pending list rendersfully populated instead of falling back to empty-string defaults.
Packaging: the dot-directory risk is settled empirically
The obvious objection to this fixture is that its payload lives behind two dot paths
(
skills/auto/.pending/flaky-triage/.meta.json), and packaging tooling routinely dropsdot-prefixed entries — which would make the fixture pass from a source checkout and fail from an
installed distribution. That was not left to inspection of
MANIFEST.in: both artifacts were builtand their archives listed.
skills/auto/.pending/flaky-triage/.meta.jsonis present in both thebuilt sdist (
kirocrew-0.7.0.tar.gz) and the built wheel (kirocrew-0.7.0-py3-none-any.whl),alongside the candidate's
SKILL.mdand the liverelease-notes/SKILL.md. Therecursive-include src/kiro_crew/tests_fixtures *line plusinclude_package_data = Truedo coverdot paths in practice.
Verification
.venv/bin/python -m pytest -qover the new test plus every suite that reads the fixture registry:test/test_fixture_skills_custom.pytest/test_pod_scenarios_command.pytest/test_pod_seed_scenarios.pytest/test_seed.pytest_pod_seed_scenarios.pyandtest_pod_scenarios_command.pyare the registry-tolerant scenariostests from #8222, so adding a fixture does not have to touch them.
Style gates on the one touched Python file (
test/test_fixture_skills_custom.py), all clean:isort --check-only,flake8,black --check --target-version py310.Pattern harvest
Not generalizable: fixture restoration validated against current production readers; no new rule.
Backend-only: a test fixture directory (JSON/YAML/markdown data) plus one pytest module. No
dashboard, API, or CLI-output surface changes, so there is nothing to screenshot.