Skip to content

test(fixtures): restore skills-custom fixture with assertions - #8680

Merged
NicholasRBowers merged 1 commit into
mainfrom
fx/fixture-skills-custom
Sep 5, 2026
Merged

test(fixtures): restore skills-custom fixture with assertions#8680
NicholasRBowers merged 1 commit into
mainfrom
fx/fixture-skills-custom

Conversation

@pepmach

@pepmach pepmach commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Restores the skills-custom seeded-home fixture and gives it a consumer test, so the fixture is
validated against the production skills readers instead of sitting unexercised on disk.

The fixture seeds one live user skill (release-notes) plus one crystallized candidate awaiting
review (flaky-triage) under skills/auto/.pending/, including the .meta.json the pending list
consumes. That is the smallest home that exercises the Skills tab's live-vs-pending split and both
of its terminal actions.

Consumer test

test/test_fixture_skills_custom.py drives the real SkillsLoader against the seeded home — no
mocks, no reimplemented parsing:

  • Layout survives the seeded-home copy. First assertions prove the nested .pending/flaky-triage
    directory and its .meta.json both exist in the materialized home.
  • Live/pending split. list_skills() returns exactly {release-notes}; auto/flaky-triage is
    absent from the live set, and list_pending_skills() returns exactly [flaky-triage].
  • Pending metadata round-trips. name, description, triggers, source (crystallize) and
    has_scripts are asserted against literals, and get_pending_skill() is checked for the same
    slug/scripts values, so a silent key rename in the reader fails the test.
  • approve_pending_skill() returns auto/flaky-triage, empties the pending list, and moves the
    candidate into the live set — asserted as the exact set {release-notes, auto/flaky-triage}.
  • dismiss_pending_skill() is exercised in a second, independent seeded home (approve and
    dismiss are mutually exclusive on one candidate): it returns True, empties the pending list, and
    leaves the live set unchanged.
  • No builtin collision. Both the live set and the post-approve set are asserted disjoint from
    the names discovered by _iter_skill_files(_BUILTIN_SKILLS_DIR), so the fixture cannot pass by
    accidentally shadowing a skill that ships in the package, and adding a builtin later cannot break
    it.

Schema repairs

The fixture is aligned to what the current production readers actually parse, not to an older
fixture's shape:

  • fixture.yaml carries the current schema-version: 2026-04-28 and the fixture-name: key that
    pod/runtime.py scans for.
  • config.json uses the current workspace key workspaces.default.dir, matching
    WorkspaceConfig.dir and the loader's raw_value.get("dir", "workspace"), rather than the legacy
    path spelling still present in older fixtures.
  • .meta.json carries exactly the key set the pending reader consumes — slug, name, source,
    created_at, description, triggers, has_scripts, scripts — so the pending list renders
    fully populated instead of falling back to empty-string defaults.

Packaging: the dot-directory risk is settled empirically

The obvious objection to this fixture is that its payload lives behind two dot paths
(skills/auto/.pending/flaky-triage/.meta.json), and packaging tooling routinely drops
dot-prefixed entries — which would make the fixture pass from a source checkout and fail from an
installed distribution. That was not left to inspection of MANIFEST.in: both artifacts were built
and their archives listed. skills/auto/.pending/flaky-triage/.meta.json is present in both the
built sdist (kirocrew-0.7.0.tar.gz) and the built wheel (kirocrew-0.7.0-py3-none-any.whl),
alongside the candidate's SKILL.md and the live release-notes/SKILL.md. The
recursive-include src/kiro_crew/tests_fixtures * line plus include_package_data = True do cover
dot paths in practice.

Verification

.venv/bin/python -m pytest -q over the new test plus every suite that reads the fixture registry:

Suite Tests
test/test_fixture_skills_custom.py 1
test/test_pod_scenarios_command.py 18
test/test_pod_seed_scenarios.py 60
test/test_seed.py 46
Total 125 passed, 0 failed

test_pod_seed_scenarios.py and test_pod_scenarios_command.py are the registry-tolerant scenarios
tests from #8222, so adding a fixture does not have to touch them.

Style gates on the one touched Python file (test/test_fixture_skills_custom.py), all clean:
isort --check-only, flake8, black --check --target-version py310.

Pattern harvest

Not generalizable: fixture restoration validated against current production readers; no new rule.

Backend-only: a test fixture directory (JSON/YAML/markdown data) plus one pytest module. No
dashboard, API, or CLI-output surface changes, so there is nothing to screenshot.

Seed release-notes as one live user skill and flaky-triage as one crystallized candidate under skills/auto/.pending/. The consumer test proves the real live and pending readers split them correctly and exercises both approve and dismiss paths without colliding with packaged builtin skill names. Its first assertions prove the nested .pending directory and .meta.json survive the seeded-home copy; sdist and wheel archive checks separately prove both dot paths survive packaging.
@pepmach
pepmach requested a review from a team as a code owner September 5, 2026 06:27
@pepmach
pepmach requested a review from hoang-phan98 September 5, 2026 06:27
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 5, 2026
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 664b38b6a049f5a5ca89c616d2676aaaceb1839d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 664b38b

Verdict parsed from the review's SHA-scoped output markers for commit 664b38b6a049f5a5ca89c616d2676aaaceb1839d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 664b38b6a049f5a5ca89c616d2676aaaceb1839d: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 664b38b6a049f5a5ca89c616d2676aaaceb1839d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Additive test-only fixture validated against the real production readers; no new surface, no contract change, trivially reversible.

Suggestions

  • The dot-path packaging guarantee rests on a one-time manual archive listing; a future setuptools glob change would drop .pending/.meta.json from the wheel invisibly, since CI runs from a source checkout. A tiny build-and-list assertion (or noting the risk in the fixture description) would make the guarantee durable rather than empirical.
  • The pending candidate's .meta.json is hand-authored rather than produced by the crystallize writer, so a writer-format evolution won't be caught here; regenerating it via the production write path when that format next changes keeps fixture and writer from drifting.

[DESIGN-REVIEWED] 664b38b

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 664b38b6a049f5a5ca89c616d2676aaaceb1839d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 664b38b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 664b38b6a049f5a5ca89c616d2676aaaceb1839d: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 664b38b6a049f5a5ca89c616d2676aaaceb1839d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All contract-required checks are done: I counted consumers of the fixture, verified the schema claims against seed.py, pod/runtime.py, and config/sections.py, and counted sibling fixtures carrying the legacy key. Final review follows.

First-Principles-Verdict: CONCERNS

The fixture-drift cause this PR names — readers no longer parsing what fixtures carry — is fixed only in the new fixture; minimal and rich keep the broken legacy key.

What this change ships

Intent: give developers a seedable home that exercises the Skills tab's live-vs-pending split, validated against the real readers — an ADDITION (a "restore" plus its first consumer test).

  1. skills-custom appears in the kirocrew seed / pod fixture list (auto-scanned by seed.available_fixtures()) — justified
  2. A home seeding one live skill and one pending candidate — justified
  3. New test drives SkillsLoader live/pending/approve/dismiss against the seeded home — justified, pins on-disk .meta.json schema independent of the writer path
  4. Fixture config.json uses the current dir workspace key — justified, but point-fix (see Watch)
  5. Second seeded home exercising only dismiss — partially duplicate of test/test_skill_pending.py::test_dismiss

Watch

  • The PR's own description names the root cause: "the legacy path spelling still present in older fixtures." Counted: 2 sibling fixtures (tests_fixtures/minimal/config.json:23,26, tests_fixtures/rich/config.json:20,23, grep "path"). _migrate_workspaces (config/sections.py:3916) maps {"path": "workspace-extra"} to dir="workspace", so both siblings' extra workspace silently points at the wrong directory while shipping workspace-extra/ content. The fix is two-line-per-file and in scope; landing only the new fixture leaves the diagnosed defect live.
  • The fixture's only counted consumer is its own test (grep skills-custom: fixture manifest + test, nothing else); its demo value via kirocrew seed is asserted, not evidenced. Cheap to keep, but it is the same zero-consumer state that presumably got it deleted last time.

Subtractions

  • Drop the second seeded_home("skills-custom") block (test/test_fixture_skills_custom.py:51-55): dismiss semantics are already pinned by test/test_skill_pending.py:115, and the fixture's on-disk shape is fully proven by the first block.

[FIRST-PRINCIPLES-REVIEWED] 664b38b

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 5, 2026
@NicholasRBowers
NicholasRBowers enabled auto-merge (squash) September 5, 2026 07:28

@NicholasRBowers NicholasRBowers left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (6 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only change — restores the skills-custom seeded-home fixture (data files under tests_fixtures/) with a consumer test covering the live/pending skill readers and approve/dismiss paths; no runtime code touched.

@NicholasRBowers
NicholasRBowers merged commit 4a26786 into main Sep 5, 2026
64 checks passed
@NicholasRBowers
NicholasRBowers deleted the fx/fixture-skills-custom branch September 5, 2026 07:29
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants