diff --git a/.github/skills/add-feature/SKILL.md b/.github/skills/add-feature/SKILL.md index 8f14430b..66ab5b6b 100644 --- a/.github/skills/add-feature/SKILL.md +++ b/.github/skills/add-feature/SKILL.md @@ -39,6 +39,18 @@ bug fixes. (discover → translate → register → execute) and name the modules touched. 4. Seed the **Decisions log** with the initial choices and who made them (Human vs Agent). +5. If the feature adds or modifies a runnable sample, complete the + **Sample / user journey** section before architecture review: + - identify the target learner and the one primary capability the sample teaches; + - write a user-centered story, including the natural-language prompt when the + sample starts from one, and the observable terminal result; + - record an explicit FRD decision to create a dedicated sample or extend an + existing one, with the rationale and migration/testing impact; + - define the exact setup, run, test, and cleanup commands, the concise evidence + the PR will include, and any existing samples affected when code is extracted. +6. Treat the completed sample design as a gate. Do not advance a sample-bearing + feature to architecture review while the learner, capability, journey, + dedicated-vs-existing decision, or validation boundary remains ambiguous. ## Phase 2 — Architecture review (planning mode) *(gate: human sign-off → `status: Finalized`)* @@ -49,7 +61,7 @@ bug fixes. (discovery read-only; registration is the only Azure-aware stage; lazy runner); - whether the public surface stays consistent with `docs/front-matter-spec.md`. 2. Iterate on the FRD until findings are resolved. Append decisions to the log. -3. Get explicit human sign-off, record it in §8, and set `status: Finalized`. +3. Get explicit human sign-off, record it in §9, and set `status: Finalized`. **Do not implement before the FRD is Finalized.** ## Phase 3 — Implementation *(gate: `ruff` + `mypy` clean)* @@ -71,7 +83,39 @@ bug fixes. 2. Add tests under `tests/`, mirroring source modules. For config/authoring changes, add a scenario folder under `tests/fixtures/config_scenarios/`. 3. For bug-adjacent work, add a failing regression test first. -4. Run the full CI-equivalent gate: +4. If the feature adds or modifies a runnable sample, run an independent sample + review (a reviewer other than the sample author, preferably a review sub-agent) + and resolve findings about: + - whether the sample tells the FRD's user-centered story and keeps one primary + capability; + - whether its documented commands produce the expected observable terminal + result from a clean setup; + - whether shared-code extraction or other changes preserve coverage for every + affected existing sample. +5. For behavior where an LLM generates structured execution inputs such as tool + arguments, dependency graphs, or configuration, include an opt-in real-model + E2E that crosses the complete boundary: + **natural-language prompt → model → tool call → parser/runtime → observable + terminal result**. Unit or integration tests that inject model output or call + the parser/runtime directly cannot substitute for this E2E. +6. Give each live E2E dependency a uniquely named Foundry environment variable + documented in the sample and FRD. When the opt-in variables are absent, the + live sample E2E must skip with a clear reason rather than fail; partial + configuration should fail with an actionable message. When they are present, + a sample-owned setup/test script may: + - generate an ignored `local.settings.json`, but must refuse to overwrite an + existing or concurrently created user file; + - run the real-model E2E and capture exact evidence of the authored tool + arguments, dependency graph, or configuration plus the observable result; + - track file ownership and clean up only the files that script created. + Record the exact commands, prompt, relevant model-authored structured input, + and resulting terminal evidence in both the FRD and PR without exposing secrets. + The `samples/workflow-retry-policy/scripts/run-e2e.py` proposed in PR #170 is + a concrete example of this pattern when available, but these gates are generic + and do not depend on that unmerged implementation. +7. Treat independent sample review and the required live boundary as a sample + validation gate. Do not accept bypass-model tests as evidence that it passed. +8. Run the full CI-equivalent gate: ```bash python -m pytest --cache-clear --cov=./src/azure_functions_agents --cov-report=xml --cov-branch tests ``` diff --git a/docs/frds/_template.md b/docs/frds/_template.md index 1cbb4c1f..3123a95e 100644 --- a/docs/frds/_template.md +++ b/docs/frds/_template.md @@ -56,27 +56,74 @@ branch: / > Backward-compatibility notes, deprecations, and migration guidance. -## 5. Decisions log +## 5. Sample / user journey + +> Complete this section when the feature adds or modifies a runnable sample; +> otherwise state "Not applicable" and why. Keep each sample focused on one +> primary capability so a learner can understand what its result demonstrates. + +- **Target learner:** … +- **Primary capability:** … +- **User-centered story / natural-language prompt:** … +- **Expected observable terminal result:** … +- **Dedicated sample vs existing sample:** … +- **Rationale and impact on existing samples:** … + +### Model-backed E2E boundary + +> If an LLM generates structured execution inputs such as tool arguments, +> dependency graphs, or configuration, require an opt-in real-model E2E that +> covers **natural-language prompt → model → tool call → parser/runtime → +> observable terminal result**. Unit or integration tests that bypass the model +> cannot substitute for this boundary. + +- **Real-model E2E required:** Yes / No — … +- **Uniquely named Foundry environment variables:** … +- **Sample-owned setup/test script:** … +- **Exact setup, run, and test commands:** … +- **Exact evidence to capture in the PR:** … (prompt, relevant model-authored + tool arguments / dependency graph / configuration, and observable result) +- **Generated ignored files (for example, `local.settings.json`):** … (refuse + to overwrite an existing or concurrently created user file) +- **Cleanup:** … (track ownership and remove only files created by the + sample-owned script) +- **Missing-environment behavior:** Skip clearly when not opted in; fail with an + actionable message when only part of the required configuration is present. + +## 6. Decisions log > Append-only. Record every non-trivial choice and **who** made it. This is the -> durable record that makes the FRD worth committing. +> durable record that makes the FRD worth committing. When a runnable sample is +> in scope, include the decision to create a dedicated sample or extend an +> existing one. | # | Decision | Options considered | Choice | Decided by | Date | | - | -------- | ------------------ | ------ | ---------- | ---- | | 1 | … | A / B / C | B | Human / Agent | YYYY-MM-DD | -## 6. Test plan +## 7. Test plan > New/changed tests, mirroring source modules under `tests/`. When config or > authoring behavior changes, add a scenario folder under > `tests/fixtures/config_scenarios/`. List the key cases (happy path, errors, -> edge cases) and any new fixtures. +> edge cases) and any new fixtures. For runnable samples, include an independent +> sample review, exact validation commands and evidence, and tests for existing +> samples affected by extraction or shared-code changes. - [ ] Unit: … - [ ] Fixture scenario: `tests/fixtures/config_scenarios//` - [ ] Regression (if fixing a bug): … - -## 7. Docs impact +- [ ] Independent sample review (if applicable): … +- [ ] Existing affected samples: … +- [ ] Opt-in real-model E2E (if required): prompt → model → tool call → + parser/runtime → observable terminal result +- [ ] Missing E2E environment variables skip clearly rather than fail. +- [ ] Exact commands, prompt, structured model output, and terminal evidence + recorded in the FRD and PR without exposing secrets. +- [ ] Sample-owned script refuses to overwrite user files and cleans up only + the files it created. + +## 8. Docs impact > Which docs change and how. @@ -85,7 +132,7 @@ branch: / - [ ] `docs/triggers.md` — trigger types - [ ] `README.md` — user-facing quickstart / examples -## 8. Status & sign-off +## 9. Status & sign-off - **Architecture review (phase 2):** - **Human sign-off:** → set `status: Finalized` before implementing.