diff --git a/docs/decisions/0005-pickled-spec-mine.md b/docs/decisions/0005-pickled-spec-mine.md new file mode 100644 index 0000000..7cc0986 --- /dev/null +++ b/docs/decisions/0005-pickled-spec-mine.md @@ -0,0 +1,76 @@ +# ADR-0005: `pickled-spec mine` staged mining pipeline + +- **Status:** Accepted +- **Date:** 2026-05-28 +- **Deciders:** pickled-spec contributors + +## Context + +Hand-running the dogfood loop (inventory → story → feature → tag → gates) +did not scale across dozens of CLI commands, MCP tools, and packages. +We needed a generic extractor that works on arbitrary Python repos, not a +one-off script tied to this monorepo. + +Early inventory runs missed umbrella MCP tools when `pickled-spec` lived +on a workspace member rather than the root `pyproject.toml`. Story +generation was initially sequential and could run for many minutes on a +large repo without scoping. + +## Decision drivers + +- Work on any Python repo with Click-discoverable CLIs, not only + pickled-spec. +- Stage isolation: re-run one stage from files on disk. +- Graceful degradation without an LLM (placeholders, skip features). +- Actionable errors when stages run out of order. +- Performance controls (`--surfaces`, parallel quick mode, existing cache). + +## Considered options + +1. **Monolithic `mine` command** — single run, no intermediate artifacts. + Rejected: hard to debug, expensive to repeat one step, poor fit for + human review between stages. + +2. **Staged pipeline with filesystem contract (chosen)** — each stage reads + and writes under `--output`. Enables `mine all` and individual + subcommands. + +3. **MCP-first mining** — expose stages only as MCP tools. Deferred: CLI + first; MCP surface for mine is future work. + +## Decision outcome + +Ship `pickled-spec mine` with six stages: inventory, stories, features, +tag, evaluate, report. Stages communicate via `inventory.json`, +`stories/`, `features/`, `tags-proposals.json`, and `evaluation/*.json`. +`mine all` orchestrates the chain; `--surfaces` filters work per stage. + +MCP umbrella detection scans the target root and uv workspace members +for `pickled-spec` (or `pickled.mcp.subservers`). Rule set paths in +`--ruleset-config` resolve relative to the config file directory. + +Ambiguity evaluation reuses `pickled_bdd.cli.run_ambiguity_gate`, the +same entry point as `pickled-bdd check --gate ambiguity` and the +`pickled-bdd ambiguity` alias. + +## Consequences + +**Positive** + +- Mining is separate from dogfood: dogfood is one consumer of the same + tools. +- Re-runnable stages and inspectable artifacts. +- Scoped runs via `--surfaces` keep LLM stages practical on monorepos. + +**Negative** + +- Disk layout is a public contract; changes need versioning care. +- Full monorepo mining without `--surfaces` remains LLM-heavy. +- Evaluate reports gate verdicts as-is; AmbiguityGate threshold tuning is + out of scope for mine. + +## Future work + +- Multi-language inventory (non-Python CLIs). +- MCP tools wrapping mine stages. +- AmbiguityGate calibration as its own change set. diff --git a/docs/decisions/0006-mine-code-reading.md b/docs/decisions/0006-mine-code-reading.md new file mode 100644 index 0000000..e2c77d7 --- /dev/null +++ b/docs/decisions/0006-mine-code-reading.md @@ -0,0 +1,107 @@ +# ADR-0006: `pickled-spec mine code` static code reading + +- **Status:** Accepted +- **Date:** 2026-05-28 +- **Deciders:** pickled-spec contributors + +## Context + +Inventory and docstrings describe surfaces at a high level. For gates and CLIs +that delegate to helpers, the docstring often understates real behaviour +(temperature, validation, return shape). Phase 8e adds a dedicated **code** +stage that extracts source for each mined surface and, optionally, a bounded +set of intra-project callees. + +## Decision drivers + +- Ground later story generation in **observed code**, not names alone. +- Stay within stdlib (`ast` only): no grimp/pydeps dependency. +- Hard caps and a **visited** set so traversal cannot run away on cycles. +- Optional diagnostic cycle reporting without changing traversal semantics. + +## Decision + +Add `pickled-spec mine code` after inventory and before stories in `mine all`. + +### Depth modes + +| Mode | Content | +|------|---------| +| `signature` | Root signature, return annotation, docstring | +| `body` | Root full function/method body (default) | +| `callgraph` | Root body plus callee bodies up to `--max-hops` | + +### Callee scope + +- `self` — methods on the enclosing class (`self.helper()`). +- `same-package` — `self` plus same-package imports (default). +- `any-pickled` — same-package plus any `pickled_*` import. + +### Bounds + +- `--max-callees` (default 8) and `--max-code-lines` (default 400) per surface. +- `visited` keys (`module:qualname`) prevent re-expansion; this is the cycle + safety mechanism. +- `--detect-cycles` runs a small DFS on collected edges and writes + `code-context/_cycles.json` for the run log / report; it does not alter BFS. + +### Output + +`code-context/.md` per surface. Surfaces without a resolvable +definition (e.g. MCP tool names with no mapped callable) get a placeholder +file and the stage continues. + +## Known limitations (v1) + +- **Protocol / dynamic dispatch** — calls such as `self._llm.complete(...)` + where `_llm` is a Protocol or opaque attribute are recorded as *unresolved* + callees with a reason; they are not chased. +- **Python only** — no cross-language call graphs. +- **Static resolution only** — no runtime type inference or polymorphic targets. + +Stories do not consume code-context until Phase 8f. + +## Resolution patterns and limits (Phase 8e-fix) + +Each collected callee records `resolution_kind` on the ref. Default +`--max-hops` is **2** so one delegation past the entry surface is included. + +### Resolved kinds + +| Kind | Pattern | Example | +|------|---------|---------| +| `free_function` | Same-module or imported callable | `helper()`, `chain.entry()` | +| `self_method` | `self.method()` on enclosing class | `self.helper()` | +| `constructor_method` | `Class(args).method()` | `Worker(cfg).process()` | +| `module_constructor` | `mod.Class(args).method()` | `mod.Worker(cfg).process()` | +| `annotated_param` | Parameter annotation pins type | `def f(w: Worker): w.m()` | +| `annotated_var` | Annotated local | `x: Worker = …; x.m()` | +| `assigned_constructor` | `x = Worker(); x.m()` (stable) | assignment tracking | + +`@property`, `@staticmethod`, `@classmethod`, and `async def` bodies resolve +when the receiver type is known. Constructor arguments may contain separate +resolvable calls (e.g. `Worker(Builder(x).build()).process()`). + +### Deliberately unresolved (reason strings) + +| Reason | Pattern | +|--------|---------| +| `protocol or unknown attribute type` | `self._llm.complete()` (nested attribute on `self`) | +| `parameter '…' has no type annotation` | `def f(w): w.method()` | +| `receiver is a subscript expression` | `items[0].method()` | +| `variable '…' reassigned; type not stable` | `x = Worker(); x = Other(); x.m()` | +| `receiver is a conditional expression` | `(a if c else b).run()` | +| `receiver is a return value of unannotated callable` | `factory().build().run()`, `.process().finalize()` | +| `dynamic attribute access` | `getattr(obj, "m")()` | +| `method not found on class; possibly inherited (base not resolved in v1)` | method absent on declared class | +| Name collision / unknown receiver | two classes share method name, type not pinned | + +Inherited methods (MRO) are not walked in v1. Return-type inference for +arbitrary call chains is out of scope. Traversal uses the same caps and +`visited` set as 8e; cycles are reported via `--detect-cycles` when enabled. + +## Consequences + +- `mine all` produces `code-context/` for downstream story prompts. +- Readers must pass inventory first; missing `inventory.json` raises an + actionable error naming `mine inventory`. diff --git a/docs/decisions/0007-code-aware-stories-and-drift.md b/docs/decisions/0007-code-aware-stories-and-drift.md new file mode 100644 index 0000000..6359ec0 --- /dev/null +++ b/docs/decisions/0007-code-aware-stories-and-drift.md @@ -0,0 +1,81 @@ +# ADR-0007: Code-aware stories and docstring drift detection + +- **Status:** Accepted +- **Date:** 2026-05-28 +- **Deciders:** pickled-spec contributors + +## Context + +Phase 8e added static code reading (`code-context/.md`). +Phase 8e-fix hardened the resolver so constructor-then-method patterns +(e.g. `FeatureDrafter(llm).draft_from_story(story)`) resolve to real +bodies, not just entry-point glue. + +Docstring-only stories (Phase 8d) were too shallow and sometimes wrong. +On the real `pickled-bdd` draft surface, a hand-written story claimed the +drafter **validates** Gherkin output. Code-reading showed the opposite: +`draft_from_story`'s docstring states the drafter does **not** validate +(returned Gherkin is raw; `warnings=()`). Human peer review had introduced +that confabulation. The mine pipeline can now ground stories in extracted +source instead of inventory summaries alone. + +## Decision + +### Code-grounded story generation + +When `code-context/.md` exists under the mining output directory, +the stories stage loads root and resolved callee bodies plus the unresolved +callee list and passes them to the story prompt together with the surface +docstring. The model writes **observable behavior** (contract), not +implementation mechanics. + +### Decision B: drift detection + +The code is the source of truth. If the docstring **contradicts** the code, +the model emits a `---DRIFT---` block; each bullet is rendered under Open +questions prefixed with `Docstring drift:`. We do not silently override the +docstring or show code and docstring side-by-side without synthesis. + +When no code-context exists, behavior falls back to Phase 8d docstring-only +rules and DRIFT is always empty. + +### Anti-implementation-leak + +Stories must not mention line numbers, private method names, or call-chain +narration ("it calls X then Y"). A reader should understand the contract +without seeing source. The prompt enforces this; tests guard the render path. + +### Unresolved-call honesty + +Calls the resolver cannot pin (protocol dispatch, dynamic getattr, etc.) +remain listed in code-context. The prompt forbids inventing behavior behind +those calls; delegated behavior is stated as uncertain. + +### Friction #15: unresolved noise filtering + +Before reporting unresolved callees, the code reader drops: + +- **Stdlib-surface methods** — e.g. `str.strip()`, `Path.read_text()` on + receivers that are not resolvable intra-project types. +- **Decorator registration** — callee scan walks function **bodies** only, + so `@main.command()` on the definition is not treated as a behavioral call. + +**Limit:** a user-defined method whose name collides with a common builtin +method (e.g. `.strip()`) on an unresolved receiver is also dropped. That +would have been unresolved noise anyway; accepted trade-off. + +### Provenance metadata + +Each story's Metadata section records **Code depth**, **Units read**, and +**Unresolved** counts when code-context was present, so readers can see how +strong the grounding was (`signature` vs `callgraph`). + +## Consequences + +- `mine all` runs inventory → code → stories; stories auto-detect + `code-context/` under `--output`. +- `mine stories` accepts optional `--code-context` to override the directory. +- Mine acts as a **docstring drift detector** when docstrings lie or lag code. +- Story quality scales with `--depth` and `--max-hops` on the code stage. +- Live LLM quality still depends on the model; tests use canned clients for + wiring and anti-leak contracts. diff --git a/docs/mcp.md b/docs/mcp.md index 2515fe6..c169325 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -193,8 +193,16 @@ For workspaces that need to enforce more than one rule set in parallel, `pickled.ruleset.yaml` also accepts a `rulesets:` list — see [`packages/pickled-rules/README.md`](../packages/pickled-rules/README.md). +## Mining + +To introspect an arbitrary Python repo and scaffold stories, features, tags, +and gate reports from its CLIs and MCP tools, use the staged +[`pickled-spec mine`](mining.md) pipeline (`mine inventory`, `mine all`, …). +Mining is CLI-first today; MCP wrappers for mine stages are not shipped yet. + ## See also +- [`mining.md`](mining.md) — staged mine pipeline for any Python repo - [`pattern.md`](pattern.md) — LLM-to-DSL bridge - [`gates.md`](gates.md) — compensating gates exposed as tools - [`integration-example.md`](integration-example.md) — end-to-end example diff --git a/docs/mining.md b/docs/mining.md new file mode 100644 index 0000000..564b3de --- /dev/null +++ b/docs/mining.md @@ -0,0 +1,193 @@ +# Mining with `pickled-spec mine` + +Mining turns a Python repository into a behavioural specification scaffold: +inventory of CLIs, MCP tools, and gates; user stories; Gherkin features; +rule tags; and gate evaluation reports. It works on **any** Python project +with discoverable Click entry points — not only this monorepo. The +`dogfood/` tree is one application of the same pipeline. + +## Seven stages + +| Stage | Command | Output | +|-------|---------|--------| +| 1. Inventory | `pickled-spec mine inventory ` | `inventory.json` | +| 2. Code | `pickled-spec mine code ` | `code-context/.md` | +| 3. Stories | `pickled-spec mine stories ` | `stories/*.story.md` | +| 4. Features | `pickled-spec mine features ` | `features/*.feature` | +| 5. Tag | `pickled-spec mine tag ` | `tags-proposals.json`, tags in features (quick mode) | +| 6. Evaluate | `pickled-spec mine evaluate ` | `evaluation/coverage.json`, `evaluation/ambiguity.json` | +| 7. Report | `pickled-spec mine report ` | `mining-report.md` | + +Run the full pipeline: + +```bash +pickled-spec mine all --output ./mining-output/ --quick +``` + +Stages communicate through files under `--output`. Re-run any stage after +fixing inputs; use `--overwrite-stories` / `--overwrite-features` to +replace existing artifacts. + +## Quick vs interactive + +- **`--quick` (default):** parallel LLM calls where supported; tag stage + writes the top proposal per scenario into feature files. +- **`--interactive`:** serial prompts for feature accept/skip/re-draft and + per-scenario tag selection. + +## `--surfaces` filter + +Limit work to a subset of packages or surface ids (case-insensitive +substring match on package name **or** surface id): + +```bash +# Only pickled-bdd surfaces +pickled-spec mine stories . --output /tmp/out --surfaces bdd + +# bdd and rules packages +pickled-spec mine stories . --output /tmp/out --surfaces bdd,rules + +# Any surface whose id contains "draft" +pickled-spec mine stories . --output /tmp/out --surfaces draft +``` + +The report notes when a filter was active. Use this to scope LLM-heavy +stages to a few packages instead of an entire monorepo. + +## Code reading stage + +`pickled-spec mine code` reads `inventory.json` and writes one markdown +file per surface under `code-context/`. It uses stdlib `ast` only (no extra +dependencies). The **stories** stage consumes these files when present. + +```bash +pickled-spec mine inventory . --output /tmp/out +pickled-spec mine code . --output /tmp/out --depth callgraph --max-hops 2 +pickled-spec mine stories . --output /tmp/out +``` + +| Flag | Default | Meaning | +|------|---------|---------| +| `--depth` | `body` | `signature`, `body`, or `callgraph` (expand callees) | +| `--callee-scope` | `same-package` | `self`, `same-package`, or `any-pickled` | +| `--max-hops` | `2` | Callee depth when `--depth callgraph` | +| `--max-callees` | `8` | Hard cap on collected units per surface | +| `--max-code-lines` | `400` | Hard cap on total source lines per surface | +| `--detect-cycles` | on | Write `code-context/_cycles.json` from observed edges | + +Traversal uses a `visited` set (`module:qualname`) so mutual recursion +cannot loop forever. `--detect-cycles` is diagnostic only. Calls such as +`self._llm.complete(...)` on a Protocol-typed attribute are listed as +unresolved callees, not chased. Stdlib method noise (e.g. `.strip()`, +`Path.read_text()`) and decorator registration calls are omitted from the +unresolved list. See [ADR 0006](decisions/0006-mine-code-reading.md). + +## Code-aware stories and drift detection + +When `code-context/.md` exists, `mine stories` (and `mine all` +after the code stage) sends root source, resolved callee bodies, and +unresolved calls to the story prompt alongside the inventory docstring. + +- Stories describe **observable behavior** (contract), not implementation + detail (no line numbers or private method names in the behavior section). +- If the docstring **contradicts** the code, the model records bullets under + **Open questions** as `Docstring drift: …` (code is source of truth). +- Without code-context, stories use docstring-only rules (Phase 8d). +- Metadata shows **Code depth**, **Units read**, and **Unresolved** counts. + +Override the code-context directory: + +```bash +pickled-spec mine stories . --output /tmp/out --code-context /path/to/code-context +``` + +See [ADR 0007](decisions/0007-code-aware-stories-and-drift.md). + +## Rule sets + +Tag and evaluate need YAML rule sets. Resolution order: + +1. `--ruleset-config ` — parse `pickled.ruleset.yaml` at that path. + Rule set paths inside the file are resolved **relative to the config + file's directory** (same as `pickled-spec check-all`). +2. `--ruleset-dir ` — load every `*.yaml` in that directory. +3. Otherwise `/pickled.ruleset.yaml` if present. + +Example (dogfood): + +```bash +pickled-spec mine tag . \ + --output ./mining-output/ \ + --ruleset-config dogfood/pickled.ruleset.yaml +``` + +Paths like `./rulesets/pickled-internal.yaml` resolve under `dogfood/`, not +the shell cwd. + +Optional `feature_glob:` in `pickled.ruleset.yaml` controls where +`check-all` and coverage evaluation find features (default +`features/**/*.feature`). See `packages/pickled-rules/README.md`. + +## Output layout + +``` +mining-output/ + inventory.json + code-context/ + .md + _cycles.json + stories/ + .story.md + features/ + .feature + tags-proposals.json + evaluation/ + coverage.json + ambiguity.json + mining-report.md + runs/ +``` + +## LLM, cache, and performance + +Stories and features call the LLM once per surface (unless skipped). +Without an API key or `pickled.config.yaml`, stories write placeholder +sections and the features stage is skipped. + +- Use **`--surfaces`** to limit scope (e.g. `bdd` finishes in seconds on + this monorepo vs tens of minutes for all surfaces). +- Use **`--max-parallel`** (default 4) to tune concurrent story/feature + drafts in quick mode. +- Identical prompts hit the disk cache configured in `pickled.config.yaml`. + +Inventory does not require an LLM. Evaluate runs ambiguity only when an +LLM is available; otherwise ambiguity is recorded as skipped (PASS with +note). + +## Missing inputs + +If a stage runs before its prerequisites, the CLI exits with code 2 and an +actionable message (no stack trace), for example: + +``` +stories requires inventory.json, which is produced by `pickled-spec mine inventory`. +``` + +## Worked example (small fixture) + +```bash +FIXTURE=packages/pickled-core/tests/fixtures/tiny_target +OUT=/tmp/mine-demo + +pickled-spec mine all "$FIXTURE" --output "$OUT" --quick --no-mcp +cat "$OUT/mining-report.md" +``` + +With an LLM configured, drop `--no-mcp` on a full repo and add +`--surfaces ` to keep story generation fast. + +## Related docs + +- [MCP integration](mcp.md) — umbrella server and subservers +- [ADR 0005](decisions/0005-pickled-spec-mine.md) — staged pipeline rationale +- [ADR 0006](decisions/0006-mine-code-reading.md) — code reading stage diff --git a/dogfood/mining-output/code-context/_cycles.json b/dogfood/mining-output/code-context/_cycles.json new file mode 100644 index 0000000..055007b --- /dev/null +++ b/dogfood/mining-output/code-context/_cycles.json @@ -0,0 +1,4 @@ +{ + "cycles": [], + "count": 0 +} diff --git a/dogfood/mining-output/code-context/bdd_draft_feature_from_story.md b/dogfood/mining-output/code-context/bdd_draft_feature_from_story.md new file mode 100644 index 0000000..0af1494 --- /dev/null +++ b/dogfood/mining-output/code-context/bdd_draft_feature_from_story.md @@ -0,0 +1,9 @@ +# Code context: bdd_draft_feature_from_story + +- **Surface id:** bdd_draft_feature_from_story +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/bdd_validate_feature_ambiguity.md b/dogfood/mining-output/code-context/bdd_validate_feature_ambiguity.md new file mode 100644 index 0000000..73bbc96 --- /dev/null +++ b/dogfood/mining-output/code-context/bdd_validate_feature_ambiguity.md @@ -0,0 +1,9 @@ +# Code context: bdd_validate_feature_ambiguity + +- **Surface id:** bdd_validate_feature_ambiguity +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/data_apply_sql_to_sandbox.md b/dogfood/mining-output/code-context/data_apply_sql_to_sandbox.md new file mode 100644 index 0000000..0f37d28 --- /dev/null +++ b/dogfood/mining-output/code-context/data_apply_sql_to_sandbox.md @@ -0,0 +1,9 @@ +# Code context: data_apply_sql_to_sandbox + +- **Surface id:** data_apply_sql_to_sandbox +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/data_check_migration_drift.md b/dogfood/mining-output/code-context/data_check_migration_drift.md new file mode 100644 index 0000000..70e88be --- /dev/null +++ b/dogfood/mining-output/code-context/data_check_migration_drift.md @@ -0,0 +1,9 @@ +# Code context: data_check_migration_drift + +- **Surface id:** data_check_migration_drift +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/data_draft_sql_migration_from_intent.md b/dogfood/mining-output/code-context/data_draft_sql_migration_from_intent.md new file mode 100644 index 0000000..18633d7 --- /dev/null +++ b/dogfood/mining-output/code-context/data_draft_sql_migration_from_intent.md @@ -0,0 +1,9 @@ +# Code context: data_draft_sql_migration_from_intent + +- **Surface id:** data_draft_sql_migration_from_intent +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/data_parse_sql_migration.md b/dogfood/mining-output/code-context/data_parse_sql_migration.md new file mode 100644 index 0000000..a649474 --- /dev/null +++ b/dogfood/mining-output/code-context/data_parse_sql_migration.md @@ -0,0 +1,9 @@ +# Code context: data_parse_sql_migration + +- **Surface id:** data_parse_sql_migration +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/diff_draft_corpus_from_examples.md b/dogfood/mining-output/code-context/diff_draft_corpus_from_examples.md new file mode 100644 index 0000000..c825f0f --- /dev/null +++ b/dogfood/mining-output/code-context/diff_draft_corpus_from_examples.md @@ -0,0 +1,9 @@ +# Code context: diff_draft_corpus_from_examples + +- **Surface id:** diff_draft_corpus_from_examples +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/diff_verify_against_oracle.md b/dogfood/mining-output/code-context/diff_verify_against_oracle.md new file mode 100644 index 0000000..cb1a3bf --- /dev/null +++ b/dogfood/mining-output/code-context/diff_verify_against_oracle.md @@ -0,0 +1,9 @@ +# Code context: diff_verify_against_oracle + +- **Surface id:** diff_verify_against_oracle +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/iac_diff_terraform_plans.md b/dogfood/mining-output/code-context/iac_diff_terraform_plans.md new file mode 100644 index 0000000..0427f86 --- /dev/null +++ b/dogfood/mining-output/code-context/iac_diff_terraform_plans.md @@ -0,0 +1,9 @@ +# Code context: iac_diff_terraform_plans + +- **Surface id:** iac_diff_terraform_plans +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/iac_draft_terraform_module.md b/dogfood/mining-output/code-context/iac_draft_terraform_module.md new file mode 100644 index 0000000..978258a --- /dev/null +++ b/dogfood/mining-output/code-context/iac_draft_terraform_module.md @@ -0,0 +1,9 @@ +# Code context: iac_draft_terraform_module + +- **Surface id:** iac_draft_terraform_module +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/iac_explain_plan_diff.md b/dogfood/mining-output/code-context/iac_explain_plan_diff.md new file mode 100644 index 0000000..a789f14 --- /dev/null +++ b/dogfood/mining-output/code-context/iac_explain_plan_diff.md @@ -0,0 +1,9 @@ +# Code context: iac_explain_plan_diff + +- **Surface id:** iac_explain_plan_diff +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/iac_suggest_security_remediation.md b/dogfood/mining-output/code-context/iac_suggest_security_remediation.md new file mode 100644 index 0000000..6e9328a --- /dev/null +++ b/dogfood/mining-output/code-context/iac_suggest_security_remediation.md @@ -0,0 +1,9 @@ +# Code context: iac_suggest_security_remediation + +- **Surface id:** iac_suggest_security_remediation +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/iac_validate_terraform_dir.md b/dogfood/mining-output/code-context/iac_validate_terraform_dir.md new file mode 100644 index 0000000..45ef2c2 --- /dev/null +++ b/dogfood/mining-output/code-context/iac_validate_terraform_dir.md @@ -0,0 +1,9 @@ +# Code context: iac_validate_terraform_dir + +- **Surface id:** iac_validate_terraform_dir +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_bdd_ambiguity.md b/dogfood/mining-output/code-context/pickled_bdd_ambiguity.md new file mode 100644 index 0000000..1f977b9 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_bdd_ambiguity.md @@ -0,0 +1,86 @@ +# Code context: ambiguity + +- **Surface id:** pickled_bdd_ambiguity +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 5 | **Total lines:** 51 | **Truncated:** False + +## Root: pickled_bdd.cli.ambiguity + +```python +def ambiguity(feature_file: str) -> None: + """Run the ambiguity gate (alias for ``check --gate ambiguity``).""" + import json as _json + + click.echo("(equivalent to: pickled-bdd check --gate ambiguity)", err=True) + llm = _build_llm_client() + result = run_ambiguity_gate(feature_file, llm) + click.echo(_json.dumps(_ambiguity_result_to_json(result), indent=2, ensure_ascii=False)) + _exit_for_verdict(result.verdict) +``` + +## Callee: pickled_bdd.cli._build_llm_client (hop 1) + +```python +def _build_llm_client() -> LLMClient: + """Build an LLM client. Override via PICKLED_BDD_LLM_FACTORY for tests.""" + from pickled_core.llm.bootstrap import build_default_client + from pickled_core.llm.config import ConfigError + + try: + return build_default_client(factory_env="PICKLED_BDD_LLM_FACTORY") + except ConfigError as exc: + raise click.ClickException(str(exc)) from exc +``` + +## Callee: pickled_bdd.cli.run_ambiguity_gate (hop 1) + +```python +def run_ambiguity_gate(feature_file: str | Path, llm: LLMClient | None) -> GateResult: + """Canonical ambiguity gate entry point (CLI, alias, mine evaluate).""" + from pickled_bdd.adapters.pytest_bdd import PytestBddAdapter + from pickled_bdd.gates.ambiguity import AmbiguityGate + + feature = PytestBddAdapter().parse_feature_file(str(feature_file)) + if llm is None: + return GateResult( + gate_name="ambiguity", + verdict=Verdict.PASS, + notes="LLM unavailable; ambiguity gate skipped", + ) + return AmbiguityGate(llm).run(feature) +``` + +## Callee: pickled_bdd.cli._ambiguity_result_to_json (hop 1) + +```python +def _ambiguity_result_to_json(result: GateResult) -> dict[str, object]: + return { + "gate": result.gate_name, + "verdict": result.verdict.value, + "notes": result.notes, + "findings": [ + { + "scenario": f.target_name, + "alternatives": list(f.alternatives), + "suggested_fix": f.suggested_fix, + } + for f in result.findings + if isinstance(f, AmbiguityFinding) + ], + } +``` + +## Callee: pickled_bdd.cli._exit_for_verdict (hop 1) + +```python +def _exit_for_verdict(verdict: Verdict) -> None: + import sys + + exit_codes = {Verdict.PASS: 0, Verdict.WARN: 1, Verdict.FAIL: 2} + sys.exit(exit_codes[verdict]) +``` + +## Unresolved callees + +- `PytestBddAdapter().parse_feature_file(str(feature_file))` — receiver is a return value of unannotated callable +- `AmbiguityGate(llm).run(feature)` — receiver is a return value of unannotated callable diff --git a/dogfood/mining-output/code-context/pickled_bdd_ambiguitygate.md b/dogfood/mining-output/code-context/pickled_bdd_ambiguitygate.md new file mode 100644 index 0000000..22d94b1 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_bdd_ambiguitygate.md @@ -0,0 +1,138 @@ +# Code context: AmbiguityGate.run + +- **Surface id:** pickled_bdd_ambiguitygate +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 4 | **Total lines:** 105 | **Truncated:** False + +## Root: pickled_bdd.gates.ambiguity.AmbiguityGate.run + +```python +def run( + self, + target: object, + *, + context: dict[str, object] | None = None, + ) -> GateResult: + _ = context + if not isinstance(target, Feature): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected Feature, got {type(target).__name__}", + ) + + findings: list[AmbiguityFinding] = [] + parse_errors: list[str] = [] + + for scenario in target.scenarios: + scenario_text = self._format_scenario(scenario.name, scenario.steps) + prompt = self._template.render(scenario=scenario_text) + from pickled_core.llm.turns import complete_prompt + + response = complete_prompt( + self._llm, + prompt, + system=( + "Reply with a single JSON object only. " + "No markdown fences, no commentary outside JSON." + ), + ) + parsed = self._parse_response(response) + if parsed is None: + parse_errors.append(scenario.name) + continue + if parsed.get("is_ambiguous"): + alts_raw = parsed.get("alternatives", []) + alts = tuple(str(x) for x in alts_raw) if isinstance(alts_raw, list) else () + findings.append( + AmbiguityFinding( + target_name=scenario.name, + alternatives=alts, + suggested_fix=str(parsed.get("suggested_fix", "")), + ) + ) + + n = len(target.scenarios) + if n > 0 and len(parse_errors) == n: + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + findings=(), + notes=f"Could not parse LLM response for: {parse_errors}", + ) + + ambiguous_count = len(findings) + verdict = self._verdict(scenario_count=n, ambiguous_count=ambiguous_count) + notes = f"{ambiguous_count}/{n} scenarios flagged ambiguous." + if parse_errors: + notes += f" Parse errors on: {parse_errors}." + if parse_errors and verdict == Verdict.PASS: + verdict = Verdict.WARN + + return GateResult( + gate_name=self.name, + verdict=verdict, + findings=tuple(findings), + notes=notes, + ) +``` + +## Callee: pickled_bdd.gates.ambiguity.AmbiguityGate._format_scenario (hop 1) + +```python +def _format_scenario(name: str, steps: tuple[str, ...]) -> str: + lines = [f"Scenario: {name}"] + lines.extend(f" {step}" for step in steps) + return "\n".join(lines) +``` + +## Callee: pickled_bdd.gates.ambiguity.AmbiguityGate._parse_response (hop 1) + +```python +def _parse_response(response: str) -> dict[str, Any] | None: + """Best-effort JSON extraction from the LLM response.""" + stripped = response.strip() + if stripped.startswith("```"): + parts = stripped.split("```") + if len(parts) >= 2: + block = parts[1] + for prefix in ("json", "JSON"): + bl = block.lstrip() + if bl.startswith(prefix): + block = bl[len(prefix) :].lstrip() + break + stripped = block.strip() + try: + data = json.loads(stripped) + except json.JSONDecodeError: + start = stripped.find("{") + end = stripped.rfind("}") + if start == -1 or end <= start: + return None + try: + data = json.loads(stripped[start : end + 1]) + except json.JSONDecodeError: + return None + if not isinstance(data, dict): + return None + return data +``` + +## Callee: pickled_bdd.gates.ambiguity.AmbiguityGate._verdict (hop 1) + +```python +def _verdict(scenario_count: int, ambiguous_count: int) -> Verdict: + if ambiguous_count == 0: + return Verdict.PASS + if scenario_count > 0 and ambiguous_count == scenario_count: + return Verdict.FAIL + return Verdict.WARN +``` + +## Unresolved callees + +- `self._template.render` — protocol or unknown attribute type +- `block.lstrip` — method name matches multiple classes; receiver type not pinned +- `bl[len(prefix):].lstrip()` — receiver is a subscript expression +- `stripped.find` — variable 'stripped' reassigned; type not stable +- `stripped.rfind` — variable 'stripped' reassigned; type not stable diff --git a/dogfood/mining-output/code-context/pickled_bdd_check.md b/dogfood/mining-output/code-context/pickled_bdd_check.md new file mode 100644 index 0000000..2bc0984 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_bdd_check.md @@ -0,0 +1,86 @@ +# Code context: check + +- **Surface id:** pickled_bdd_check +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 5 | **Total lines:** 51 | **Truncated:** False + +## Root: pickled_bdd.cli.check + +```python +def check(feature_file: str, gate: str) -> None: + """Run compensating gates against a .feature file.""" + import json as _json + + _ = gate # v0.1: only ambiguity; "all" resolves to the same gate. + llm = _build_llm_client() + result = run_ambiguity_gate(feature_file, llm) + click.echo(_json.dumps(_ambiguity_result_to_json(result), indent=2, ensure_ascii=False)) + _exit_for_verdict(result.verdict) +``` + +## Callee: pickled_bdd.cli._build_llm_client (hop 1) + +```python +def _build_llm_client() -> LLMClient: + """Build an LLM client. Override via PICKLED_BDD_LLM_FACTORY for tests.""" + from pickled_core.llm.bootstrap import build_default_client + from pickled_core.llm.config import ConfigError + + try: + return build_default_client(factory_env="PICKLED_BDD_LLM_FACTORY") + except ConfigError as exc: + raise click.ClickException(str(exc)) from exc +``` + +## Callee: pickled_bdd.cli.run_ambiguity_gate (hop 1) + +```python +def run_ambiguity_gate(feature_file: str | Path, llm: LLMClient | None) -> GateResult: + """Canonical ambiguity gate entry point (CLI, alias, mine evaluate).""" + from pickled_bdd.adapters.pytest_bdd import PytestBddAdapter + from pickled_bdd.gates.ambiguity import AmbiguityGate + + feature = PytestBddAdapter().parse_feature_file(str(feature_file)) + if llm is None: + return GateResult( + gate_name="ambiguity", + verdict=Verdict.PASS, + notes="LLM unavailable; ambiguity gate skipped", + ) + return AmbiguityGate(llm).run(feature) +``` + +## Callee: pickled_bdd.cli._ambiguity_result_to_json (hop 1) + +```python +def _ambiguity_result_to_json(result: GateResult) -> dict[str, object]: + return { + "gate": result.gate_name, + "verdict": result.verdict.value, + "notes": result.notes, + "findings": [ + { + "scenario": f.target_name, + "alternatives": list(f.alternatives), + "suggested_fix": f.suggested_fix, + } + for f in result.findings + if isinstance(f, AmbiguityFinding) + ], + } +``` + +## Callee: pickled_bdd.cli._exit_for_verdict (hop 1) + +```python +def _exit_for_verdict(verdict: Verdict) -> None: + import sys + + exit_codes = {Verdict.PASS: 0, Verdict.WARN: 1, Verdict.FAIL: 2} + sys.exit(exit_codes[verdict]) +``` + +## Unresolved callees + +- `PytestBddAdapter().parse_feature_file(str(feature_file))` — receiver is a return value of unannotated callable +- `AmbiguityGate(llm).run(feature)` — receiver is a return value of unannotated callable diff --git a/dogfood/mining-output/code-context/pickled_bdd_draft.md b/dogfood/mining-output/code-context/pickled_bdd_draft.md new file mode 100644 index 0000000..dab2b7c --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_bdd_draft.md @@ -0,0 +1,64 @@ +# Code context: draft + +- **Surface id:** pickled_bdd_draft +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 3 | **Total lines:** 40 | **Truncated:** False + +## Root: pickled_bdd.cli.draft + +```python +def draft(story_file: str, output: str | None) -> None: + """Draft a .feature file from a user story (Markdown).""" + story = Path(story_file).read_text(encoding="utf-8") + llm = _build_llm_client() + result = FeatureDrafter(llm).draft_from_story(story) + + if output: + Path(output).write_text(result.text, encoding="utf-8") + click.echo(f"Wrote {output}", err=True) + else: + click.echo(result.text) +``` + +## Callee: pickled_bdd.cli._build_llm_client (hop 1) + +```python +def _build_llm_client() -> LLMClient: + """Build an LLM client. Override via PICKLED_BDD_LLM_FACTORY for tests.""" + from pickled_core.llm.bootstrap import build_default_client + from pickled_core.llm.config import ConfigError + + try: + return build_default_client(factory_env="PICKLED_BDD_LLM_FACTORY") + except ConfigError as exc: + raise click.ClickException(str(exc)) from exc +``` + +## Callee: pickled_bdd.FeatureDrafter.draft_from_story (hop 1) + +```python +def draft_from_story(self, story: str) -> DraftResult: + """Send the story to the LLM and return a DraftResult. + + The drafter does not validate the returned Gherkin; that is the + Ambiguity gate's job (PR-08). v0.1 returns the raw text and + leaves a generic rationale string. + """ + prompt = self._template.render(story=story) + from pickled_core.llm.turns import complete_prompt + + feature_text = complete_prompt( + self._llm, + prompt, + system="You output only Gherkin. No prose, no fences.", + ) + return DraftResult( + text=feature_text.strip(), + rationale="LLM-drafted from user story; no post-processing applied.", + warnings=(), + ) +``` + +## Unresolved callees + +- `self._template.render` — protocol or unknown attribute type diff --git a/dogfood/mining-output/code-context/pickled_bdd_mcp.md b/dogfood/mining-output/code-context/pickled_bdd_mcp.md new file mode 100644 index 0000000..e95e542 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_bdd_mcp.md @@ -0,0 +1,12 @@ +# Code context: mcp + +- **Surface id:** pickled_bdd_mcp +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 1 | **Total lines:** 2 | **Truncated:** False + +## Root: pickled_bdd.cli.mcp + +```python +def mcp() -> None: + """MCP server commands.""" +``` diff --git a/dogfood/mining-output/code-context/pickled_bdd_run_all.md b/dogfood/mining-output/code-context/pickled_bdd_run_all.md new file mode 100644 index 0000000..c57fa1a --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_bdd_run_all.md @@ -0,0 +1,91 @@ +# Code context: run_all + +- **Surface id:** pickled_bdd_run_all +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 3 | **Total lines:** 67 | **Truncated:** False + +## Root: pickled_bdd.gates_runner.run_all + +```python +def run_all(workdir: Path | str) -> list[GateResult]: + """Parse Gherkin under ``features/`` (AmbiguityGate skipped without LLM).""" + root = Path(workdir).resolve() + features = sorted(root.glob("features/**/*.feature")) + if not features: + return [ + GateResult( + gate_name="bdd.features", + verdict=Verdict.PASS, + notes="no features/ directory", + ) + ] + + adapter = PytestBddAdapter() + results: list[GateResult] = [] + for path in features: + try: + adapter.parse_feature_file(path) + except Exception as exc: + results.append( + GateResult( + gate_name=f"bdd.parse.{path.name}", + verdict=Verdict.FAIL, + notes=str(exc), + ) + ) + else: + results.append( + GateResult( + gate_name=f"bdd.parse.{path.name}", + verdict=Verdict.PASS, + notes=f"parsed {path.relative_to(root)}", + ) + ) + + results.append( + GateResult( + gate_name="bdd.ambiguity", + verdict=Verdict.PASS, + notes="skipped — set PICKLED_BDD_LLM_FACTORY to enable AmbiguityGate", + ) + ) + return results +``` + +## Callee: pickled_bdd.adapters.PytestBddAdapter.parse_feature_file (hop 1) + +```python +def parse_feature_file(self, path: str | Path) -> Feature: + """Parse a `.feature` file into a runner-agnostic Feature. + + Handles Scenario, Scenario Outline, Examples, Background, and + Rule blocks. Background steps are prepended to every scenario + in the feature (the same expansion pytest-bdd performs at + runtime); rule-level Background steps are appended after the + feature-level Background for scenarios inside that rule. + """ + p = Path(path) + text = p.read_text(encoding="utf-8") + return self.parse_feature_text(text, path=str(p)) +``` + +## Callee: pickled_bdd.adapters.PytestBddAdapter.parse_feature_text (hop 2) + +```python +def parse_feature_text(self, gherkin_text: str, *, path: str | None = None) -> Feature: + """Parse a Gherkin string into a Feature. + + Same Background-prepending and feature-tag-inheritance semantics as + :meth:`parse_feature_file`. Use ``path=None`` for in-memory content. + """ + if not gherkin_text.strip(): + raise ValueError("Gherkin text is empty") + ast = cast(dict[str, Any], Parser().parse(TokenScanner(gherkin_text))) + if ast.get("feature") is None: + raise ValueError("No Feature found in Gherkin text") + return self._build_feature(ast, path=path) +``` + +## Unresolved callees + +- `path.relative_to` — method name matches multiple classes; receiver type not pinned diff --git a/dogfood/mining-output/code-context/pickled_core_check_all.md b/dogfood/mining-output/code-context/pickled_core_check_all.md new file mode 100644 index 0000000..8c794a8 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_core_check_all.md @@ -0,0 +1,9 @@ +# Code context: check-all + +- **Surface id:** pickled_core_check_all +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_core_mine.md b/dogfood/mining-output/code-context/pickled_core_mine.md new file mode 100644 index 0000000..fa436dc --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_core_mine.md @@ -0,0 +1,12 @@ +# Code context: mine + +- **Surface id:** pickled_core_mine +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 1 | **Total lines:** 2 | **Truncated:** False + +## Root: pickled_core.mine.cli.mine + +```python +def mine() -> None: + """Mine a Python project for surfaces, stories, features, and gate results.""" +``` diff --git a/dogfood/mining-output/code-context/pickled_core_mine_all.md b/dogfood/mining-output/code-context/pickled_core_mine_all.md new file mode 100644 index 0000000..2844b6c --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_core_mine_all.md @@ -0,0 +1,9 @@ +# Code context: mine all + +- **Surface id:** pickled_core_mine_all +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_core_mine_code.md b/dogfood/mining-output/code-context/pickled_core_mine_code.md new file mode 100644 index 0000000..9df0320 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_core_mine_code.md @@ -0,0 +1,9 @@ +# Code context: mine code + +- **Surface id:** pickled_core_mine_code +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_core_mine_evaluate.md b/dogfood/mining-output/code-context/pickled_core_mine_evaluate.md new file mode 100644 index 0000000..7f865ef --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_core_mine_evaluate.md @@ -0,0 +1,9 @@ +# Code context: mine evaluate + +- **Surface id:** pickled_core_mine_evaluate +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_core_mine_features.md b/dogfood/mining-output/code-context/pickled_core_mine_features.md new file mode 100644 index 0000000..add4ddd --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_core_mine_features.md @@ -0,0 +1,9 @@ +# Code context: mine features + +- **Surface id:** pickled_core_mine_features +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_core_mine_inventory.md b/dogfood/mining-output/code-context/pickled_core_mine_inventory.md new file mode 100644 index 0000000..ba282ae --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_core_mine_inventory.md @@ -0,0 +1,9 @@ +# Code context: mine inventory + +- **Surface id:** pickled_core_mine_inventory +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_core_mine_report.md b/dogfood/mining-output/code-context/pickled_core_mine_report.md new file mode 100644 index 0000000..d530db1 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_core_mine_report.md @@ -0,0 +1,9 @@ +# Code context: mine report + +- **Surface id:** pickled_core_mine_report +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_core_mine_stories.md b/dogfood/mining-output/code-context/pickled_core_mine_stories.md new file mode 100644 index 0000000..a64f0db --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_core_mine_stories.md @@ -0,0 +1,9 @@ +# Code context: mine stories + +- **Surface id:** pickled_core_mine_stories +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_core_mine_tag.md b/dogfood/mining-output/code-context/pickled_core_mine_tag.md new file mode 100644 index 0000000..bcfa224 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_core_mine_tag.md @@ -0,0 +1,9 @@ +# Code context: mine tag + +- **Surface id:** pickled_core_mine_tag +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_data_apply.md b/dogfood/mining-output/code-context/pickled_data_apply.md new file mode 100644 index 0000000..a7df992 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_data_apply.md @@ -0,0 +1,134 @@ +# Code context: apply + +- **Surface id:** pickled_data_apply +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 6 | **Total lines:** 90 | **Truncated:** False + +## Root: pickled_data.cli.apply + +```python +def apply(migration: Path, dialect: str) -> None: + """Apply migration to in-memory SQLite and print resulting schema.""" + _check_dbt(migration) + sql = migration.read_text(encoding="utf-8") + schema = apply_migration(sql, dialect=dialect) + click.echo(json.dumps(schema, indent=2)) +``` + +## Callee: pickled_data.cli._check_dbt (hop 1) + +```python +def _check_dbt(path: Path) -> None: + if path.suffix == ".dbt": + raise NotImplementedError(DBT_NOT_IMPLEMENTED_MSG) +``` + +## Callee: pickled_data.oracle.apply_migration (hop 1) + +```python +def apply_migration( + sql: str, + target_db: Path | None = None, + *, + dialect: str = "postgres", +) -> dict[str, Any]: + """Apply DDL/DML and return resulting schema summary.""" + parse_sql(sql, dialect=dialect) + statements = sqlglot.parse(sql, dialect=dialect) + _reject_filesystem_escapes(statements) + sqlite_sqls: list[str] = [] + for stmt in statements: + if stmt is None: + continue + transpiled = stmt.sql(dialect="sqlite") + if transpiled.strip(): + sqlite_sqls.append(transpiled) + + conn = ( + sqlite3.connect(str(target_db)) + if target_db is not None + else sqlite3.connect(":memory:") + ) + + try: + # Defense in depth: even if a future transpilation quirk reintroduces + # an ATTACH statement past the AST check, SQLite refuses to attach + # anything when SQLITE_LIMIT_ATTACHED == 0. + try: + conn.setlimit(sqlite3.SQLITE_LIMIT_ATTACHED, 0) + except AttributeError: # pragma: no cover — only Python <3.11 + pass + for statement in sqlite_sqls: + conn.execute(statement) + conn.commit() + return _introspect(conn) + finally: + conn.close() +``` + +## Callee: pickled_data.parser.parse_sql (hop 2) + +```python +def parse_sql(sql: str, dialect: str = "postgres") -> exp.Expr: + """Parse SQL into a sqlglot AST.""" + try: + parsed = sqlglot.parse_one(sql, dialect=dialect) + except sqlglot.errors.ParseError as exc: + raise SQLParseError(str(exc)) from exc + if parsed is None: + msg = "empty parse result" + raise SQLParseError(msg) + return parsed +``` + +## Callee: pickled_data.oracle._reject_filesystem_escapes (hop 2) + +```python +def _reject_filesystem_escapes(statements: list[exp.Expression | None]) -> None: + """Refuse ATTACH/DETACH statements before any SQL is executed.""" + for stmt in statements: + if stmt is None: + continue + if isinstance(stmt, exp.Attach | exp.Detach): + raise UnsafeMigrationStatementError( + f"ATTACH/DETACH statements are not allowed in the sandbox: " + f"{stmt.sql(dialect='sqlite')!r}" + ) + for node in stmt.find_all(exp.Attach, exp.Detach): + raise UnsafeMigrationStatementError( + f"ATTACH/DETACH statements are not allowed in the sandbox: " + f"{node.sql(dialect='sqlite')!r}" + ) +``` + +## Callee: pickled_data.oracle._introspect (hop 2) + +```python +def _introspect(conn: sqlite3.Connection) -> dict[str, Any]: + tables: list[dict[str, Any]] = [] + cur = conn.execute( + "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'" + ) + for (name,) in cur.fetchall(): + cols: list[dict[str, Any]] = [] + info = conn.execute(f"PRAGMA table_info({name})").fetchall() + for _cid, col_name, col_type, notnull, _default, _pk in info: + cols.append( + { + "name": col_name, + "type": (col_type or "TEXT").upper(), + "nullable": not bool(notnull), + } + ) + tables.append({"name": name, "columns": cols}) + return {"tables": tables} +``` + +## Unresolved callees + +- `sqlglot.parse` — method name matches multiple classes; receiver type not pinned +- `stmt.sql` — method name matches multiple classes; receiver type not pinned +- `conn.commit` — variable 'conn' reassigned; type not stable +- `conn.close` — variable 'conn' reassigned; type not stable +- `conn.setlimit` — variable 'conn' reassigned; type not stable +- `conn.execute` — variable 'conn' reassigned; type not stable diff --git a/dogfood/mining-output/code-context/pickled_data_check_drift.md b/dogfood/mining-output/code-context/pickled_data_check_drift.md new file mode 100644 index 0000000..d5c398f --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_data_check_drift.md @@ -0,0 +1,9 @@ +# Code context: check-drift + +- **Surface id:** pickled_data_check_drift +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_data_datacontractgate.md b/dogfood/mining-output/code-context/pickled_data_datacontractgate.md new file mode 100644 index 0000000..c95bd9f --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_data_datacontractgate.md @@ -0,0 +1,140 @@ +# Code context: DataContractGate.run + +- **Surface id:** pickled_data_datacontractgate +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 4 | **Total lines:** 110 | **Truncated:** False + +## Root: pickled_data.gates.DataContractGate.run + +```python +def run( + self, + target: object, + *, + context: dict[str, Any] | None = None, + ) -> GateResult: + ctx = context or {} + if not isinstance(target, str): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected SQL string, got {type(target).__name__}", + ) + endpoint_tag = ctx.get("endpoint_tag") + if not isinstance(endpoint_tag, str): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes='context must contain "endpoint_tag"', + ) + if self._registry is None: + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + notes="no SchemaRegistry configured", + ) + artifact = self._registry.find_schema_by_tag(endpoint_tag) + if artifact is None: + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + notes=f"no schema for tag {endpoint_tag!r}", + ) + sql_columns = _select_column_names(target) + api_columns = _openapi_response_property_names(artifact.content) + if not api_columns: + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + notes="could not extract OpenAPI response properties", + ) + api_set = set(api_columns) + sql_set = set(sql_columns) + missing = sorted(sql_set - api_set) + extra = sorted(api_set - sql_set) + if missing or extra: + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"column mismatch missing={missing} extra={extra}", + ) + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="column names match OpenAPI properties (types not checked in v0.1)", + ) +``` + +## Callee: pickled_data.gates._select_column_names (hop 1) + +```python +def _select_column_names(sql: str) -> list[str]: + """Extract output column names from a SELECT (best-effort).""" + try: + ast = parse_sql(sql, dialect="postgres") + except Exception: + return [] + cols: list[str] = [] + for node in ast.find_all(exp.Select): + for expr in node.expressions: + alias = getattr(expr, "alias", None) + if alias: + cols.append(str(alias)) + elif hasattr(expr, "name") and expr.name: + cols.append(str(expr.name)) + return cols +``` + +## Callee: pickled_data.gates._openapi_response_property_names (hop 1) + +```python +def _openapi_response_property_names(spec_yaml: str) -> list[str]: + import yaml + + try: + spec = yaml.safe_load(spec_yaml) + except yaml.YAMLError: + return [] + if not isinstance(spec, dict): + return [] + paths = spec.get("paths") + if not isinstance(paths, dict): + return [] + for _path, item in paths.items(): + if not isinstance(item, dict): + continue + for _method, op in item.items(): + if not isinstance(op, dict): + continue + responses = op.get("responses") or {} + ok = responses.get("200") or responses.get("201") + if not isinstance(ok, dict): + continue + content = ok.get("content") or {} + app_json = content.get("application/json") or {} + schema = app_json.get("schema") or {} + props = schema.get("properties") or {} + if isinstance(props, dict): + return sorted(str(k) for k in props) + return [] +``` + +## Callee: pickled_data.parser.parse_sql (hop 2) + +```python +def parse_sql(sql: str, dialect: str = "postgres") -> exp.Expr: + """Parse SQL into a sqlglot AST.""" + try: + parsed = sqlglot.parse_one(sql, dialect=dialect) + except sqlglot.errors.ParseError as exc: + raise SQLParseError(str(exc)) from exc + if parsed is None: + msg = "empty parse result" + raise SQLParseError(msg) + return parsed +``` + +## Unresolved callees + +- `self._registry.find_schema_by_tag` — protocol or unknown attribute type +- `getattr(expr, 'alias', None)` — dynamic attribute access diff --git a/dogfood/mining-output/code-context/pickled_data_draft.md b/dogfood/mining-output/code-context/pickled_data_draft.md new file mode 100644 index 0000000..21c89d2 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_data_draft.md @@ -0,0 +1,171 @@ +# Code context: draft + +- **Surface id:** pickled_data_draft +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 8 | **Total lines:** 120 | **Truncated:** False + +## Root: pickled_data.cli.draft + +```python +def draft( + intent: str, + dialect: str, + current_schema: Path | None, + output: Path | None, +) -> None: + """Draft a SQL migration from a natural-language intent.""" + schema_yaml: str | None = None + if current_schema is not None: + schema_yaml = current_schema.read_text(encoding="utf-8") + try: + llm = _build_llm_client() + result = MigrationDrafter(llm).draft_from_intent( + intent_text=_read_text_arg(intent), + dialect=dialect, + current_schema_yaml=schema_yaml, + ) + except click.ClickException: + raise + except Exception as exc: + click.echo(str(exc), err=True) + raise SystemExit(2) from exc + _emit_draft_output( + text=result.text, + rationale=result.rationale, + warnings=result.warnings, + output=output, + ) +``` + +## Callee: pickled_data.cli._build_llm_client (hop 1) + +```python +def _build_llm_client() -> LLMClient: + from pickled_core.llm.bootstrap import build_default_client + from pickled_core.llm.config import ConfigError + + try: + return build_default_client(factory_env="PICKLED_DATA_LLM_FACTORY") + except ConfigError as exc: + raise click.ClickException(str(exc)) from exc +``` + +## Callee: pickled_data.MigrationDrafter.draft_from_intent (hop 1) + +```python +def draft_from_intent( + self, + *, + intent_text: str, + dialect: str, + current_schema_yaml: str | None = None, + ) -> DraftResult: + prompt = self._build_prompt( + intent_text=intent_text, + dialect=dialect, + current_schema_yaml=current_schema_yaml, + ) + completion = self._llm.complete( + messages=[Message(role="user", content=prompt)], + model=_DRAFT_MODEL, + max_tokens=4000, + temperature=0.0, + stop=None, + extras=None, + ) + text, rationale = self._split_output(completion.text) + warnings = tuple(self._validate(text, dialect=dialect)) + return DraftResult(text=text, rationale=rationale, warnings=warnings) +``` + +## Callee: pickled_data.cli._read_text_arg (hop 1) + +```python +def _read_text_arg(path: str) -> str: + if path == "-": + return sys.stdin.read() + return Path(path).read_text(encoding="utf-8") +``` + +## Callee: pickled_data.cli._emit_draft_output (hop 1) + +```python +def _emit_draft_output( + *, + text: str, + rationale: str, + warnings: tuple[str, ...], + output: Path | None, +) -> None: + if output is not None: + output.write_text(text, encoding="utf-8") + else: + click.echo(text) + if rationale: + for line in rationale.splitlines(): + click.echo(f"rationale: {line}", err=True) + for warning in warnings: + click.echo(f"warning: {warning}", err=True) + if warnings: + raise SystemExit(1) +``` + +## Callee: pickled_data.MigrationDrafter._build_prompt (hop 2) + +```python +def _build_prompt( + self, + *, + intent_text: str, + dialect: str, + current_schema_yaml: str | None, + ) -> str: + schema_block = current_schema_yaml.strip() if current_schema_yaml else "none" + return ( + "You are drafting a SQL migration for the pickled-data tool. Given:\n\n" + f"- dialect: {dialect}\n" + f"- intent: {intent_text.strip()}\n" + f"- current schema (optional YAML): {schema_block}\n\n" + "Emit a single SQL migration file. Use only DDL statements valid in " + "the stated dialect. Begin with a comment line " + "`-- intent: `. " + "Do NOT include destructive operations (DROP DATABASE, TRUNCATE entire " + "tables without a WHERE clause is N/A for DDL). " + f"After the SQL, emit the literal line {RATIONALE_SENTINEL!r} then " + "1-3 sentences explaining choices." + ) +``` + +## Callee: pickled_data.MigrationDrafter._split_output (hop 2) + +```python +def _split_output(self, raw: str) -> tuple[str, str]: + if RATIONALE_SENTINEL in raw: + text, _, rationale = raw.partition(RATIONALE_SENTINEL) + return text.strip(), rationale.strip() + return raw.strip(), "" +``` + +## Callee: pickled_data.MigrationDrafter._validate (hop 2) + +```python +def _validate(self, text: str, *, dialect: str) -> list[str]: + warnings: list[str] = [] + try: + sqlglot.parse(text, dialect=dialect) + except Exception as exc: # noqa: BLE001 — surface any parse failure + warnings.append(str(exc)) + for line_no, line in enumerate(text.splitlines(), start=1): + if "drop table" in line.lower(): + warnings.append( + f"destructive operation on line {line_no} " + f"('DROP TABLE'); confirm intent before applying" + ) + return warnings +``` + +## Unresolved callees + +- `self._llm.complete` — protocol or unknown attribute type +- `sys.stdin.read` — method name matches multiple classes; receiver type not pinned +- `rationale.splitlines` — method name matches multiple classes; receiver type not pinned diff --git a/dogfood/mining-output/code-context/pickled_data_mcp.md b/dogfood/mining-output/code-context/pickled_data_mcp.md new file mode 100644 index 0000000..dabc388 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_data_mcp.md @@ -0,0 +1,12 @@ +# Code context: mcp + +- **Surface id:** pickled_data_mcp +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 1 | **Total lines:** 2 | **Truncated:** False + +## Root: pickled_data.cli.mcp + +```python +def mcp() -> None: + """MCP server commands.""" +``` diff --git a/dogfood/mining-output/code-context/pickled_data_migrationdriftgate.md b/dogfood/mining-output/code-context/pickled_data_migrationdriftgate.md new file mode 100644 index 0000000..6f4ebbf --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_data_migrationdriftgate.md @@ -0,0 +1,176 @@ +# Code context: MigrationDriftGate.run + +- **Surface id:** pickled_data_migrationdriftgate +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 5 | **Total lines:** 132 | **Truncated:** True + +## Root: pickled_data.gates.MigrationDriftGate.run + +```python +def run( + self, + target: object, + *, + context: dict[str, Any] | None = None, + ) -> GateResult: + ctx = context or {} + if not isinstance(target, str): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected migration SQL string, got {type(target).__name__}", + ) + expected = ctx.get("expected_schema") + if not isinstance(expected, dict): + raw = ctx.get("expected_schema_yaml") + if isinstance(raw, str): + loaded = yaml.safe_load(raw) + expected = loaded if isinstance(loaded, dict) else None + if not isinstance(expected, dict): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes='context needs "expected_schema" or "expected_schema_yaml"', + ) + dialect = str(ctx.get("dialect", "postgres")) + actual = apply_migration(target, dialect=dialect) + exp_tables = _normalize_columns(expected) + act_tables = _normalize_columns(actual) + ok, notes = _compare_schemas(exp_tables, act_tables) + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS if ok else Verdict.FAIL, + notes=notes, + ) +``` + +## Callee: pickled_data.oracle.apply_migration (hop 1) + +```python +def apply_migration( + sql: str, + target_db: Path | None = None, + *, + dialect: str = "postgres", +) -> dict[str, Any]: + """Apply DDL/DML and return resulting schema summary.""" + parse_sql(sql, dialect=dialect) + statements = sqlglot.parse(sql, dialect=dialect) + _reject_filesystem_escapes(statements) + sqlite_sqls: list[str] = [] + for stmt in statements: + if stmt is None: + continue + transpiled = stmt.sql(dialect="sqlite") + if transpiled.strip(): + sqlite_sqls.append(transpiled) + + conn = ( + sqlite3.connect(str(target_db)) + if target_db is not None + else sqlite3.connect(":memory:") + ) + + try: + # Defense in depth: even if a future transpilation quirk reintroduces + # an ATTACH statement past the AST check, SQLite refuses to attach + # anything when SQLITE_LIMIT_ATTACHED == 0. + try: + conn.setlimit(sqlite3.SQLITE_LIMIT_ATTACHED, 0) + except AttributeError: # pragma: no cover — only Python <3.11 + pass + for statement in sqlite_sqls: + conn.execute(statement) + conn.commit() + return _introspect(conn) + finally: + conn.close() +``` + +## Callee: pickled_data.gates._normalize_columns (hop 1) + +```python +def _normalize_columns(schema: dict[str, Any]) -> dict[str, set[tuple[str, str, bool]]]: + """Map table name -> set of (name, type, nullable).""" + out: dict[str, set[tuple[str, str, bool]]] = {} + for table in schema.get("tables", []): + if not isinstance(table, dict): + continue + name = str(table.get("name", "")) + cols: set[tuple[str, str, bool]] = set() + for col in table.get("columns", []): + if isinstance(col, dict): + cols.add( + ( + str(col.get("name", "")), + str(col.get("type", "TEXT")).upper(), + bool(col.get("nullable", True)), + ) + ) + out[name] = cols + return out +``` + +## Callee: pickled_data.gates._compare_schemas (hop 1) + +```python +def _compare_schemas( + expected: dict[str, set[tuple[str, str, bool]]], + actual: dict[str, set[tuple[str, str, bool]]], +) -> tuple[bool, str]: + """Compare schemas; ignore nullable-only drift (e.g. SQLite DEFAULT → NOT NULL).""" + if set(expected) != set(actual): + return ( + False, + f"drift: expected tables {sorted(expected)} vs actual {sorted(actual)}", + ) + nullable_notes: list[str] = [] + for table in sorted(expected): + exp_cols = expected[table] + act_cols = actual[table] + if _column_keys(exp_cols) != _column_keys(act_cols): + return ( + False, + f"drift: table {table!r} expected {_column_keys(exp_cols)} " + f"vs actual {_column_keys(act_cols)}", + ) + exp_null = {(n, t): nullable for n, t, nullable in exp_cols} + act_null = {(n, t): nullable for n, t, nullable in act_cols} + for key, exp_n in exp_null.items(): + act_n = act_null.get(key) + if act_n is not None and exp_n != act_n: + nullable_notes.append(f"{table}.{key[0]} expected nullable={exp_n} got {act_n}") + if nullable_notes: + detail = "; ".join(nullable_notes) + return True, f"Schema matches expected (nullable differs: {detail})." + return True, "Schema matches expected." +``` + +## Callee: pickled_data.parser.parse_sql (hop 2) + +```python +def parse_sql(sql: str, dialect: str = "postgres") -> exp.Expr: + """Parse SQL into a sqlglot AST.""" + try: + parsed = sqlglot.parse_one(sql, dialect=dialect) + except sqlglot.errors.ParseError as exc: + raise SQLParseError(str(exc)) from exc + if parsed is None: + msg = "empty parse result" + raise SQLParseError(msg) + return parsed +``` + +## Unresolved callees + +- `sqlglot.parse` — method name matches multiple classes; receiver type not pinned +- `stmt.sql` — method name matches multiple classes; receiver type not pinned +- `conn.commit` — variable 'conn' reassigned; type not stable +- `conn.close` — variable 'conn' reassigned; type not stable +- `conn.setlimit` — variable 'conn' reassigned; type not stable +- `conn.execute` — variable 'conn' reassigned; type not stable +- `cols.add` — method name matches multiple classes; receiver type not pinned + +## Notes + +Collection stopped early because of --max-callees or --max-code-lines. diff --git a/dogfood/mining-output/code-context/pickled_data_parse.md b/dogfood/mining-output/code-context/pickled_data_parse.md new file mode 100644 index 0000000..4f94a34 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_data_parse.md @@ -0,0 +1,65 @@ +# Code context: parse + +- **Surface id:** pickled_data_parse +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 6 | **Total lines:** 30 | **Truncated:** False + +## Root: pickled_data.cli.parse + +```python +def parse(migration: Path, dialect: str) -> None: + """Parse a migration SQL file and print AST summary.""" + _check_dbt(migration) + artifact = load_sql_file(migration, dialect=dialect) + summary = ast_summary(artifact.ast) + click.echo(json.dumps({"dialect": artifact.dialect, **summary}, indent=2)) +``` + +## Callee: pickled_data.cli._check_dbt (hop 1) + +```python +def _check_dbt(path: Path) -> None: + if path.suffix == ".dbt": + raise NotImplementedError(DBT_NOT_IMPLEMENTED_MSG) +``` + +## Callee: pickled_data.parser.load_sql_file (hop 1) + +```python +def load_sql_file(path: Path, dialect: str = "postgres") -> SQLArtifact: + _reject_dbt(path) + content = path.read_text(encoding="utf-8") + ast = parse_sql(content, dialect=dialect) + return SQLArtifact(content=content, dialect=dialect, ast=ast) +``` + +## Callee: pickled_data.parser.ast_summary (hop 1) + +```python +def ast_summary(ast: Any) -> dict[str, str]: + """Short summary for CLI / MCP.""" + return {"kind": type(ast).__name__, "sql": ast.sql(dialect="postgres")} +``` + +## Callee: pickled_data.parser._reject_dbt (hop 2) + +```python +def _reject_dbt(path: Path | None) -> None: + if path is not None and path.suffix == ".dbt": + raise NotImplementedError(DBT_NOT_IMPLEMENTED_MSG) +``` + +## Callee: pickled_data.parser.parse_sql (hop 2) + +```python +def parse_sql(sql: str, dialect: str = "postgres") -> exp.Expr: + """Parse SQL into a sqlglot AST.""" + try: + parsed = sqlglot.parse_one(sql, dialect=dialect) + except sqlglot.errors.ParseError as exc: + raise SQLParseError(str(exc)) from exc + if parsed is None: + msg = "empty parse result" + raise SQLParseError(msg) + return parsed +``` diff --git a/dogfood/mining-output/code-context/pickled_data_run_all.md b/dogfood/mining-output/code-context/pickled_data_run_all.md new file mode 100644 index 0000000..fedb088 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_data_run_all.md @@ -0,0 +1,136 @@ +# Code context: run_all + +- **Surface id:** pickled_data_run_all +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 4 | **Total lines:** 107 | **Truncated:** True + +## Root: pickled_data.gates_runner.run_all + +```python +def run_all(workdir: Path | str) -> list[GateResult]: + """Parse migrations and run drift gate vs ``expected_schema.yaml``.""" + root = Path(workdir).resolve() + migrations = sorted(root.glob("migrations/*.sql")) + expected_path = root / "expected_schema.yaml" + + if not migrations: + return [ + GateResult( + gate_name="data.migrations", + verdict=Verdict.WARN, + notes="no migrations/*.sql", + ) + ] + + results: list[GateResult] = [] + dialect = "sqlite" + for mig in migrations: + try: + load_sql_file(mig, dialect=dialect) + except SQLParseError as exc: + results.append( + GateResult( + gate_name=f"data.parse.{mig.name}", + verdict=Verdict.FAIL, + notes=str(exc), + ) + ) + else: + results.append( + GateResult( + gate_name=f"data.parse.{mig.name}", + verdict=Verdict.PASS, + notes="parsed", + ) + ) + + if len(migrations) > 1: + results.append( + GateResult( + gate_name="data.migrations.note", + verdict=Verdict.WARN, + notes=f"applying {len(migrations)} migrations in filename order for drift", + ) + ) + + if expected_path.is_file() and migrations: + combined_sql = "\n\n".join( + mig.read_text(encoding="utf-8") for mig in migrations + ) + expected = yaml.safe_load(expected_path.read_text(encoding="utf-8")) + if isinstance(expected, dict): + gr = MigrationDriftGate().run( + combined_sql, + context={"expected_schema": expected, "dialect": dialect}, + ) + results.append( + GateResult( + gate_name="data.migration_drift", + verdict=gr.verdict, + notes=gr.notes, + ) + ) + return results +``` + +## Callee: pickled_data.parser.load_sql_file (hop 1) + +```python +def load_sql_file(path: Path, dialect: str = "postgres") -> SQLArtifact: + _reject_dbt(path) + content = path.read_text(encoding="utf-8") + ast = parse_sql(content, dialect=dialect) + return SQLArtifact(content=content, dialect=dialect, ast=ast) +``` + +## Callee: pickled_data.MigrationDriftGate.run (hop 1) + +```python +def run( + self, + target: object, + *, + context: dict[str, Any] | None = None, + ) -> GateResult: + ctx = context or {} + if not isinstance(target, str): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected migration SQL string, got {type(target).__name__}", + ) + expected = ctx.get("expected_schema") + if not isinstance(expected, dict): + raw = ctx.get("expected_schema_yaml") + if isinstance(raw, str): + loaded = yaml.safe_load(raw) + expected = loaded if isinstance(loaded, dict) else None + if not isinstance(expected, dict): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes='context needs "expected_schema" or "expected_schema_yaml"', + ) + dialect = str(ctx.get("dialect", "postgres")) + actual = apply_migration(target, dialect=dialect) + exp_tables = _normalize_columns(expected) + act_tables = _normalize_columns(actual) + ok, notes = _compare_schemas(exp_tables, act_tables) + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS if ok else Verdict.FAIL, + notes=notes, + ) +``` + +## Callee: pickled_data.parser._reject_dbt (hop 2) + +```python +def _reject_dbt(path: Path | None) -> None: + if path is not None and path.suffix == ".dbt": + raise NotImplementedError(DBT_NOT_IMPLEMENTED_MSG) +``` + +## Notes + +Collection stopped early because of --max-callees or --max-code-lines. diff --git a/dogfood/mining-output/code-context/pickled_diff_draft_corpus.md b/dogfood/mining-output/code-context/pickled_diff_draft_corpus.md new file mode 100644 index 0000000..cb7c410 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_diff_draft_corpus.md @@ -0,0 +1,9 @@ +# Code context: draft-corpus + +- **Surface id:** pickled_diff_draft_corpus +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_diff_mcp.md b/dogfood/mining-output/code-context/pickled_diff_mcp.md new file mode 100644 index 0000000..7e63386 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_diff_mcp.md @@ -0,0 +1,12 @@ +# Code context: mcp + +- **Surface id:** pickled_diff_mcp +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 1 | **Total lines:** 2 | **Truncated:** False + +## Root: pickled_diff.cli.mcp + +```python +def mcp() -> None: + """MCP server commands.""" +``` diff --git a/dogfood/mining-output/code-context/pickled_diff_run_all.md b/dogfood/mining-output/code-context/pickled_diff_run_all.md new file mode 100644 index 0000000..be99ef2 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_diff_run_all.md @@ -0,0 +1,89 @@ +# Code context: run_all + +- **Surface id:** pickled_diff_run_all +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 2 | **Total lines:** 70 | **Truncated:** True + +## Root: pickled_diff.gates_runner.run_all + +```python +def run_all(workdir: Path | str) -> list[GateResult]: + """Run differential oracle gate when ``pickled.diff.yaml`` is present.""" + root = Path(workdir).resolve() + cfg = _load_config(root) + if not cfg: + return [ + GateResult( + gate_name="diff.config", + verdict=Verdict.WARN, + notes="missing pickled.diff.yaml or diff/pickled.diff.yaml", + ) + ] + + try: + oracle_argv = _resolve_argv(_argv_list(cfg.get("oracle_command"), "oracle_command"), root) + candidate_argv = _resolve_argv( + _argv_list(cfg.get("candidate_command"), "candidate_command"), root + ) + corpus_ref = cfg.get("corpus") + if not isinstance(corpus_ref, str): + raise ValueError('config key "corpus" must be a path string') + comparator_name = str(cfg.get("comparator", "exact")) + timeout = float(cfg.get("timeout_seconds", 30)) + corpus = _load_corpus(root, corpus_ref) + except (ValueError, FileNotFoundError, json.JSONDecodeError, TypeError) as exc: + return [ + GateResult( + gate_name="diff.config", + verdict=Verdict.FAIL, + notes=str(exc), + ) + ] + + oracle_cmd = list(oracle_argv) + candidate_cmd = list(candidate_argv) + if oracle_cmd[0] in {"python", "python3"} and len(oracle_cmd) > 1: + oracle_cmd[0] = sys.executable + if candidate_cmd[0] in {"python", "python3"} and len(candidate_cmd) > 1: + candidate_cmd[0] = sys.executable + + gate = DifferentialOracleGate( + oracle=SubprocessRunner( + oracle_cmd, + name="oracle", + timeout_seconds=timeout, + cwd=root, + ), + candidate=SubprocessRunner( + candidate_cmd, + name="candidate", + timeout_seconds=timeout, + cwd=root, + ), + comparator=_comparator(comparator_name), + ) + gr = gate.run(corpus) + return [ + GateResult( + gate_name="diff.differential_oracle", + verdict=gr.verdict, + findings=gr.findings, + notes=gr.notes, + ) + ] +``` + +## Callee: pickled_diff.gates_runner._load_config (hop 1) + +```python +def _load_config(root: Path) -> dict[str, Any]: + cfg_path = _find_config(root) + if cfg_path is None: + return {} + data = yaml.safe_load(cfg_path.read_text(encoding="utf-8")) + return data if isinstance(data, dict) else {} +``` + +## Notes + +Collection stopped early because of --max-callees or --max-code-lines. diff --git a/dogfood/mining-output/code-context/pickled_diff_verify.md b/dogfood/mining-output/code-context/pickled_diff_verify.md new file mode 100644 index 0000000..12bf7be --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_diff_verify.md @@ -0,0 +1,188 @@ +# Code context: verify + +- **Surface id:** pickled_diff_verify +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 4 | **Total lines:** 156 | **Truncated:** False + +## Root: pickled_diff.cli.verify + +```python +def verify( + oracle: str, + candidate: str, + corpus: Path, + comparator: str, + timeout_seconds: float, +) -> None: + """Compare candidate vs reference across a JSON input corpus.""" + raw = json.loads(corpus.read_text(encoding="utf-8")) + if not isinstance(raw, list): + raise click.ClickException("Corpus JSON must be a list of {name, payload} objects") + items = [ + CorpusItem(name=str(entry["name"]), payload=str(entry["payload"])) + for entry in raw + if isinstance(entry, dict) + ] + gate = DifferentialOracleGate( + oracle=SubprocessRunner( + shlex.split(oracle), + name="oracle", + timeout_seconds=timeout_seconds, + ), + candidate=SubprocessRunner( + shlex.split(candidate), + name="candidate", + timeout_seconds=timeout_seconds, + ), + comparator=_comparator(comparator), + ) + result = gate.run(InMemoryCorpus(items)) + click.echo(json.dumps(_gate_result_to_json(result), indent=2, ensure_ascii=False)) + exit_codes = {Verdict.PASS: 0, Verdict.WARN: 1, Verdict.FAIL: 2} + sys.exit(exit_codes[result.verdict]) +``` + +## Callee: pickled_diff.cli._comparator (hop 1) + +```python +def _comparator(name: str) -> ExactEqComparator | StructuralJsonComparator: + if name == "structural_json": + return StructuralJsonComparator() + return ExactEqComparator() +``` + +## Callee: pickled_diff.DifferentialOracleGate.run (hop 1) + +```python +def run( + self, + target: object, + *, + context: dict[str, Any] | None = None, + ) -> GateResult: + _ = context + if not isinstance(target, Corpus) or not getattr( + target, "_pickled_diff_corpus", False + ): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected InMemoryCorpus (Corpus), got {type(target).__name__}", + ) + + total = len(target) + if total == 0: + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="Corpus is empty; nothing to verify.", + ) + + findings: list[DifferentialFinding] = [] + oracle_errors = 0 + candidate_errors = 0 + compared = 0 + mismatches = 0 + + for item in target: + oracle_out = self._oracle.run(item.payload) + if oracle_out.error: + oracle_errors += 1 + continue + + candidate_out = self._candidate.run(item.payload) + if candidate_out.error: + candidate_errors += 1 + compared += 1 + mismatches += 1 + if len(findings) < self._max_findings: + findings.append( + DifferentialFinding( + input_repr=item.name, + oracle_output=oracle_out.stdout, + candidate_output=candidate_out.stdout, + diff_summary=f"candidate error: {candidate_out.error}", + ) + ) + continue + + compared += 1 + equal, summary = self._comparator.compare(oracle_out, candidate_out) + if not equal: + mismatches += 1 + if len(findings) < self._max_findings: + findings.append( + DifferentialFinding( + input_repr=item.name, + oracle_output=oracle_out.stdout, + candidate_output=candidate_out.stdout, + diff_summary=summary, + ) + ) + + notes = ( + f"{mismatches}/{compared} mismatches among compared items " + f"(corpus size {total}); " + f"{oracle_errors} oracle errors; {candidate_errors} candidate errors." + ) + + if oracle_errors == total: + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + findings=tuple(findings), + notes=notes + " Oracle failed on every input — check oracle configuration.", + ) + + if compared == 0: + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + findings=tuple(findings), + notes=notes, + ) + + if mismatches == compared: + verdict = Verdict.FAIL + elif mismatches > 0 or oracle_errors > 0: + verdict = Verdict.WARN + else: + verdict = Verdict.PASS + + return GateResult( + gate_name=self.name, + verdict=verdict, + findings=tuple(findings), + notes=notes, + ) +``` + +## Callee: pickled_diff.cli._gate_result_to_json (hop 1) + +```python +def _gate_result_to_json(result: Any) -> dict[str, Any]: + from pickled_diff.types import DifferentialFinding + + return { + "gate": result.gate_name, + "verdict": result.verdict.value, + "notes": result.notes, + "findings": [ + { + "input_repr": f.input_repr, + "oracle_output": f.oracle_output, + "candidate_output": f.candidate_output, + "diff_summary": f.diff_summary, + } + for f in result.findings + if isinstance(f, DifferentialFinding) + ], + } +``` + +## Unresolved callees + +- `getattr(target, '_pickled_diff_corpus', False)` — dynamic attribute access +- `self._oracle.run` — protocol or unknown attribute type +- `self._candidate.run` — protocol or unknown attribute type +- `self._comparator.compare` — protocol or unknown attribute type diff --git a/dogfood/mining-output/code-context/pickled_iac_diff.md b/dogfood/mining-output/code-context/pickled_iac_diff.md new file mode 100644 index 0000000..a2a7a37 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_iac_diff.md @@ -0,0 +1,122 @@ +# Code context: diff + +- **Surface id:** pickled_iac_diff +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 3 | **Total lines:** 102 | **Truncated:** False + +## Root: pickled_iac.cli.diff + +```python +def diff(base_path: Path, head_path: Path) -> None: + """Compare two terraform plan JSON files.""" + base_plan = json.loads(base_path.read_text(encoding="utf-8")) + head_plan = json.loads(head_path.read_text(encoding="utf-8")) + gate = PlanDiffGate() + result = gate.run(head_plan, context={"base_plan": base_plan}) + click.echo( + json.dumps( + { + "verdict": result.verdict.value, + "notes": result.notes, + "findings": [ + { + "address": f.address, + "actions_before": list(f.actions_before), + "actions_after": list(f.actions_after), + } + for f in result.findings + if isinstance(f, PlanDiffFinding) + ], + }, + indent=2, + ) + ) + if result.verdict is Verdict.FAIL: + raise SystemExit(2) + if result.verdict is Verdict.WARN: + raise SystemExit(1) +``` + +## Callee: pickled_iac.PlanDiffGate.run (hop 1) + +```python +def run( + self, + target: object, + *, + context: dict[str, Any] | None = None, + ) -> GateResult: + ctx = context or {} + if not isinstance(target, dict): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected head plan dict, got {type(target).__name__}", + ) + base = ctx.get("base_plan") + if not isinstance(base, dict): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes='context must contain "base_plan" dict', + ) + + head_changes = _index_changes(target) + base_changes = _index_changes(base) + findings: list[PlanDiffFinding] = [] + all_actions: set[str] = set() + + for address, actions in head_changes.items(): + all_actions.update(actions) + if address not in base_changes: + if actions: + findings.append( + PlanDiffFinding(address, (), tuple(actions)), + ) + elif base_changes[address] != actions: + findings.append( + PlanDiffFinding( + address, + tuple(base_changes[address]), + tuple(actions), + ), + ) + all_actions.update(actions) + + if not findings and not all_actions: + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="No plan changes between base and head.", + ) + + if any(a in {"delete", "replace"} for a in all_actions): + verdict = Verdict.FAIL + elif all_actions <= {"create", "update", "read", "no-op"}: + verdict = Verdict.WARN + else: + verdict = Verdict.WARN + + return GateResult( + gate_name=self.name, + verdict=verdict, + findings=tuple(findings), + notes=f"{len(findings)} resource change(s) detected.", + ) +``` + +## Callee: pickled_iac._index_changes (hop 2) + +```python +def _index_changes(plan: dict[str, Any]) -> dict[str, list[str]]: + out: dict[str, list[str]] = {} + for rc in plan.get("resource_changes", []) or []: + if not isinstance(rc, dict): + continue + address = str(rc.get("address", "")) + change = rc.get("change") or {} + actions = change.get("actions") if isinstance(change, dict) else [] + if isinstance(actions, list): + out[address] = [str(a) for a in actions] + return out +``` diff --git a/dogfood/mining-output/code-context/pickled_iac_draft.md b/dogfood/mining-output/code-context/pickled_iac_draft.md new file mode 100644 index 0000000..ef901e8 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_iac_draft.md @@ -0,0 +1,137 @@ +# Code context: draft + +- **Surface id:** pickled_iac_draft +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 5 | **Total lines:** 99 | **Truncated:** False + +## Root: pickled_iac.cli.draft + +```python +def draft(user_story: str, provider: str, output: Path | None) -> None: + """Draft a Terraform module from a user story.""" + artifact = IaCDrafter(_build_llm_client()).draft_module(user_story, provider=provider) + if output: + output.mkdir(parents=True, exist_ok=True) + (output / "main.tf").write_text(artifact.content, encoding="utf-8") + click.echo(f"Wrote {output / 'main.tf'}", err=True) + else: + click.echo(artifact.content) +``` + +## Callee: pickled_iac.IaCDrafter.draft_module (hop 1) + +```python +def draft_module( + self, + user_story: str, + provider: str = "aws", + ) -> IaCArtifact: + """Draft, validate in a temp dir, and return an IaCArtifact.""" + binary = iac_binary() + fmt: str = "opentofu" if binary == "opentofu" else "terraform" + last_error = "" + + for _attempt in range(3): + feedback = ( + f"\n\nPrevious validation errors:\n{last_error}" if last_error else "" + ) + prompt = self._template.render( + provider=provider, + user_story=user_story, + error_feedback=feedback, + ) + from pickled_core.llm.turns import complete_prompt + + hcl = complete_prompt( + self._llm, + prompt, + system="Output only Terraform HCL. No fences, no commentary.", + ).strip() + if hcl.startswith("```"): + lines = hcl.splitlines() + hcl = "\n".join( + line for line in lines if not line.strip().startswith("```") + ).strip() + + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + (root / "main.tf").write_text(hcl, encoding="utf-8") + result = validate(root) + if result.valid: + return IaCArtifact(content=hcl, format=fmt, path=None) # type: ignore[arg-type] + last_error = "; ".join(result.diagnostics) or "validation failed" + + msg = f"failed to draft valid Terraform after 3 attempts: {last_error}" + raise RuntimeError(msg) +``` + +## Callee: pickled_iac.cli._build_llm_client (hop 1) + +```python +def _build_llm_client() -> LLMClient: + factory = os.environ.get("PICKLED_IAC_LLM_FACTORY") + if factory: + module_name, sep, attr = factory.partition(":") + if not sep: + raise click.ClickException( + "PICKLED_IAC_LLM_FACTORY must be 'module:callable'" + ) + module = importlib.import_module(module_name) + return cast(LLMClient, getattr(module, attr)()) + + from pickled_core.llm.config import load_config + from pickled_core.llm.factory import build_client + + provider = os.environ.get("PICKLED_LLM_PROVIDER", "anthropic") + try: + return build_client(provider, config=load_config()) + except ConfigError as exc: + raise click.ClickException(str(exc)) from exc +``` + +## Callee: pickled_iac.oracle.iac_binary (hop 2) + +```python +def iac_binary() -> Literal["terraform", "opentofu"]: + """Return the detected IaC CLI binary name.""" + if _IAC_BIN is None: + raise IaCToolMissingError( + "neither 'terraform' nor 'tofu' found on PATH; " + "install Terraform >=1.7.5 or OpenTofu >=1.8" + ) + return _IAC_BIN +``` + +## Callee: pickled_iac.oracle.validate (hop 2) + +```python +def validate(tf_dir: Path) -> ValidateResult: + """Run ``terraform validate -json`` (or OpenTofu equivalent).""" + binary = iac_binary() + _init_if_needed(tf_dir, binary) + proc = _run([binary, "validate", "-json"], cwd=tf_dir) + fmt: Literal["terraform", "opentofu"] = "opentofu" if binary == "opentofu" else "terraform" + if proc.returncode != 0 and not proc.stdout.strip(): + err = (proc.stderr or "validate failed").strip() + return ValidateResult(valid=False, diagnostics=[err], format=fmt) + try: + payload = json.loads(proc.stdout or "{}") + except json.JSONDecodeError: + err = (proc.stderr or proc.stdout or "invalid validate JSON").strip() + return ValidateResult(valid=False, diagnostics=[err], format=fmt) + valid = bool(payload.get("valid")) + diags: list[str] = [] + for d in payload.get("diagnostics", []): + if isinstance(d, dict): + summary = d.get("summary") or d.get("detail") or str(d) + diags.append(str(summary)) + return ValidateResult(valid=valid, diagnostics=diags, format=fmt) +``` + +## Unresolved callees + +- `self._template.render` — protocol or unknown attribute type +- `hcl.splitlines` — method name matches multiple classes; receiver type not pinned +- `factory.partition` — variable 'factory' reassigned; type not stable +- `getattr(module, attr)()` — dynamic attribute access +- `getattr(module, attr)` — dynamic attribute access diff --git a/dogfood/mining-output/code-context/pickled_iac_iacambiguitygate.md b/dogfood/mining-output/code-context/pickled_iac_iacambiguitygate.md new file mode 100644 index 0000000..3677c0e --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_iac_iacambiguitygate.md @@ -0,0 +1,102 @@ +# Code context: IaCAmbiguityGate.run + +- **Surface id:** pickled_iac_iacambiguitygate +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 2 | **Total lines:** 79 | **Truncated:** False + +## Root: pickled_iac.gates.IaCAmbiguityGate.run + +```python +def run( + self, + target: object, + *, + context: dict[str, Any] | None = None, + ) -> GateResult: + ctx = context or {} + if not isinstance(target, IaCArtifact): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected IaCArtifact, got {type(target).__name__}", + ) + story = ctx.get("user_story") + if not isinstance(story, str) or not story.strip(): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes='context must contain non-empty "user_story"', + ) + + prompt = self._template.render( + user_story=story, + terraform_hcl=target.content, + ) + from pickled_core.llm.turns import complete_prompt + + response = complete_prompt( + self._llm, + prompt, + system="Reply with a single JSON object only. No markdown fences.", + ) + parsed = _parse_json_object(response) + if parsed is None: + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes="LLM returned malformed JSON", + ) + ambiguities = parsed.get("ambiguities", []) + if not isinstance(ambiguities, list): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes='LLM JSON missing list field "ambiguities"', + ) + if not ambiguities: + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="No ambiguities reported.", + ) + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + findings=tuple(ambiguities), + notes=f"{len(ambiguities)} ambiguity(ies) reported.", + ) +``` + +## Callee: pickled_iac.gates._parse_json_object (hop 1) + +```python +def _parse_json_object(response: str) -> dict[str, Any] | None: + stripped = response.strip() + if stripped.startswith("```"): + parts = stripped.split("```") + if len(parts) >= 2: + block = parts[1].lstrip() + if block.lower().startswith("json"): + block = block[4:].lstrip() + stripped = block.strip() + try: + data = json.loads(stripped) + except json.JSONDecodeError: + start = stripped.find("{") + end = stripped.rfind("}") + if start == -1 or end <= start: + return None + try: + data = json.loads(stripped[start : end + 1]) + except json.JSONDecodeError: + return None + return data if isinstance(data, dict) else None +``` + +## Unresolved callees + +- `self._template.render` — protocol or unknown attribute type +- `parts[1].lstrip()` — receiver is a subscript expression +- `block[4:].lstrip()` — receiver is a subscript expression +- `stripped.find` — variable 'stripped' reassigned; type not stable +- `stripped.rfind` — variable 'stripped' reassigned; type not stable diff --git a/dogfood/mining-output/code-context/pickled_iac_mcp.md b/dogfood/mining-output/code-context/pickled_iac_mcp.md new file mode 100644 index 0000000..8ecea92 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_iac_mcp.md @@ -0,0 +1,12 @@ +# Code context: mcp + +- **Surface id:** pickled_iac_mcp +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 1 | **Total lines:** 2 | **Truncated:** False + +## Root: pickled_iac.cli.mcp + +```python +def mcp() -> None: + """MCP server commands.""" +``` diff --git a/dogfood/mining-output/code-context/pickled_iac_plan_cmd.md b/dogfood/mining-output/code-context/pickled_iac_plan_cmd.md new file mode 100644 index 0000000..153d35e --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_iac_plan_cmd.md @@ -0,0 +1,9 @@ +# Code context: plan-cmd + +- **Surface id:** pickled_iac_plan_cmd +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_iac_plandiffgate.md b/dogfood/mining-output/code-context/pickled_iac_plandiffgate.md new file mode 100644 index 0000000..cb4c9a7 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_iac_plandiffgate.md @@ -0,0 +1,89 @@ +# Code context: PlanDiffGate.run + +- **Surface id:** pickled_iac_plandiffgate +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 2 | **Total lines:** 74 | **Truncated:** False + +## Root: pickled_iac.gates.PlanDiffGate.run + +```python +def run( + self, + target: object, + *, + context: dict[str, Any] | None = None, + ) -> GateResult: + ctx = context or {} + if not isinstance(target, dict): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected head plan dict, got {type(target).__name__}", + ) + base = ctx.get("base_plan") + if not isinstance(base, dict): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes='context must contain "base_plan" dict', + ) + + head_changes = _index_changes(target) + base_changes = _index_changes(base) + findings: list[PlanDiffFinding] = [] + all_actions: set[str] = set() + + for address, actions in head_changes.items(): + all_actions.update(actions) + if address not in base_changes: + if actions: + findings.append( + PlanDiffFinding(address, (), tuple(actions)), + ) + elif base_changes[address] != actions: + findings.append( + PlanDiffFinding( + address, + tuple(base_changes[address]), + tuple(actions), + ), + ) + all_actions.update(actions) + + if not findings and not all_actions: + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="No plan changes between base and head.", + ) + + if any(a in {"delete", "replace"} for a in all_actions): + verdict = Verdict.FAIL + elif all_actions <= {"create", "update", "read", "no-op"}: + verdict = Verdict.WARN + else: + verdict = Verdict.WARN + + return GateResult( + gate_name=self.name, + verdict=verdict, + findings=tuple(findings), + notes=f"{len(findings)} resource change(s) detected.", + ) +``` + +## Callee: pickled_iac.gates._index_changes (hop 1) + +```python +def _index_changes(plan: dict[str, Any]) -> dict[str, list[str]]: + out: dict[str, list[str]] = {} + for rc in plan.get("resource_changes", []) or []: + if not isinstance(rc, dict): + continue + address = str(rc.get("address", "")) + change = rc.get("change") or {} + actions = change.get("actions") if isinstance(change, dict) else [] + if isinstance(actions, list): + out[address] = [str(a) for a in actions] + return out +``` diff --git a/dogfood/mining-output/code-context/pickled_iac_run_all.md b/dogfood/mining-output/code-context/pickled_iac_run_all.md new file mode 100644 index 0000000..c793ea7 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_iac_run_all.md @@ -0,0 +1,219 @@ +# Code context: run_all + +- **Surface id:** pickled_iac_run_all +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 6 | **Total lines:** 184 | **Truncated:** False + +## Root: pickled_iac.gates_runner.run_all + +```python +def run_all(workdir: Path | str) -> list[GateResult]: + """``terraform validate`` and optional Trivy scan on ``infra/``.""" + root = Path(workdir).resolve() + infra = root / "infra" + if not infra.is_dir(): + return [ + GateResult( + gate_name="iac.infra", + verdict=Verdict.WARN, + notes="no infra/ directory", + ) + ] + + results: list[GateResult] = [] + try: + vr = validate(infra) + except IaCToolMissingError as exc: + results.append( + GateResult( + gate_name="iac.validate", + verdict=Verdict.WARN, + notes=str(exc), + ) + ) + except Exception as exc: + results.append( + GateResult( + gate_name="iac.validate", + verdict=Verdict.FAIL, + notes=str(exc), + ) + ) + else: + results.append( + GateResult( + gate_name="iac.validate", + verdict=Verdict.PASS if vr.valid else Verdict.FAIL, + notes="; ".join(vr.diagnostics) or "ok", + ) + ) + + sec = SecurityBaselineGate().run(infra) + results.append( + GateResult( + gate_name=sec.gate_name, + verdict=sec.verdict, + findings=sec.findings, + notes=sec.notes, + ) + ) + return results +``` + +## Callee: pickled_iac.oracle.validate (hop 1) + +```python +def validate(tf_dir: Path) -> ValidateResult: + """Run ``terraform validate -json`` (or OpenTofu equivalent).""" + binary = iac_binary() + _init_if_needed(tf_dir, binary) + proc = _run([binary, "validate", "-json"], cwd=tf_dir) + fmt: Literal["terraform", "opentofu"] = "opentofu" if binary == "opentofu" else "terraform" + if proc.returncode != 0 and not proc.stdout.strip(): + err = (proc.stderr or "validate failed").strip() + return ValidateResult(valid=False, diagnostics=[err], format=fmt) + try: + payload = json.loads(proc.stdout or "{}") + except json.JSONDecodeError: + err = (proc.stderr or proc.stdout or "invalid validate JSON").strip() + return ValidateResult(valid=False, diagnostics=[err], format=fmt) + valid = bool(payload.get("valid")) + diags: list[str] = [] + for d in payload.get("diagnostics", []): + if isinstance(d, dict): + summary = d.get("summary") or d.get("detail") or str(d) + diags.append(str(summary)) + return ValidateResult(valid=valid, diagnostics=diags, format=fmt) +``` + +## Callee: pickled_iac.SecurityBaselineGate.run (hop 1) + +```python +def run( + self, + target: object, + *, + context: dict[str, Any] | None = None, + ) -> GateResult: + _ = context + if not isinstance(target, Path): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected Path to tf dir, got {type(target).__name__}", + ) + trivy = shutil.which("trivy") + if trivy is None: + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="trivy not found on PATH — security scan skipped", + ) + + proc = subprocess.run( + [ + trivy, + "config", + str(target), + "--format", + "json", + "--severity", + "HIGH,CRITICAL", + "--quiet", + ], + capture_output=True, + text=True, + check=False, + ) + if proc.returncode not in (0, 1) and not proc.stdout.strip(): + err = (proc.stderr or "trivy failed").strip() + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + notes=f"trivy error: {err}", + ) + + try: + report = json.loads(proc.stdout or "{}") + except json.JSONDecodeError: + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + notes="trivy returned non-JSON output", + ) + + critical: list[str] = [] + high: list[str] = [] + for result in report.get("Results", []) or []: + if not isinstance(result, dict): + continue + for mis in result.get("Misconfigurations", []) or []: + if not isinstance(mis, dict): + continue + sev = str(mis.get("Severity", "")).upper() + title = str(mis.get("Title", mis.get("ID", "finding"))) + if sev == "CRITICAL": + critical.append(title) + elif sev == "HIGH": + high.append(title) + + if critical: + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + findings=tuple(critical), + notes=f"{len(critical)} CRITICAL finding(s)", + ) + if high: + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + findings=tuple(high), + notes=f"{len(high)} HIGH finding(s)", + ) + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="No HIGH or CRITICAL findings.", + ) +``` + +## Callee: pickled_iac.oracle.iac_binary (hop 2) + +```python +def iac_binary() -> Literal["terraform", "opentofu"]: + """Return the detected IaC CLI binary name.""" + if _IAC_BIN is None: + raise IaCToolMissingError( + "neither 'terraform' nor 'tofu' found on PATH; " + "install Terraform >=1.7.5 or OpenTofu >=1.8" + ) + return _IAC_BIN +``` + +## Callee: pickled_iac.oracle._init_if_needed (hop 2) + +```python +def _init_if_needed(tf_dir: Path, binary: Literal["terraform", "opentofu"]) -> None: + if (tf_dir / ".terraform").exists(): + return + init = _run([binary, "init", "-input=false", "-backend=false"], cwd=tf_dir) + if init.returncode != 0: + err = (init.stderr or init.stdout or "terraform init failed").strip() + msg = f"{binary} init failed: {err}" + raise RuntimeError(msg) +``` + +## Callee: pickled_iac.oracle._run (hop 2) + +```python +def _run(cmd: list[str], *, cwd: Path) -> subprocess.CompletedProcess[str]: + return subprocess.run( + cmd, + cwd=str(cwd), + capture_output=True, + text=True, + check=False, + env={**os.environ, "TF_IN_AUTOMATION": "1"}, + ) +``` diff --git a/dogfood/mining-output/code-context/pickled_iac_scan.md b/dogfood/mining-output/code-context/pickled_iac_scan.md new file mode 100644 index 0000000..1e18afd --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_iac_scan.md @@ -0,0 +1,117 @@ +# Code context: scan + +- **Surface id:** pickled_iac_scan +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 2 | **Total lines:** 102 | **Truncated:** False + +## Root: pickled_iac.cli.scan + +```python +def scan(tf_dir: Path) -> None: + """Run Trivy config scan (optional; skips if trivy missing).""" + result = SecurityBaselineGate().run(tf_dir) + click.echo( + json.dumps( + { + "verdict": result.verdict.value, + "notes": result.notes, + "findings": list(result.findings), + }, + indent=2, + ) + ) + if result.verdict is Verdict.FAIL: + raise SystemExit(2) +``` + +## Callee: pickled_iac.SecurityBaselineGate.run (hop 1) + +```python +def run( + self, + target: object, + *, + context: dict[str, Any] | None = None, + ) -> GateResult: + _ = context + if not isinstance(target, Path): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected Path to tf dir, got {type(target).__name__}", + ) + trivy = shutil.which("trivy") + if trivy is None: + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="trivy not found on PATH — security scan skipped", + ) + + proc = subprocess.run( + [ + trivy, + "config", + str(target), + "--format", + "json", + "--severity", + "HIGH,CRITICAL", + "--quiet", + ], + capture_output=True, + text=True, + check=False, + ) + if proc.returncode not in (0, 1) and not proc.stdout.strip(): + err = (proc.stderr or "trivy failed").strip() + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + notes=f"trivy error: {err}", + ) + + try: + report = json.loads(proc.stdout or "{}") + except json.JSONDecodeError: + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + notes="trivy returned non-JSON output", + ) + + critical: list[str] = [] + high: list[str] = [] + for result in report.get("Results", []) or []: + if not isinstance(result, dict): + continue + for mis in result.get("Misconfigurations", []) or []: + if not isinstance(mis, dict): + continue + sev = str(mis.get("Severity", "")).upper() + title = str(mis.get("Title", mis.get("ID", "finding"))) + if sev == "CRITICAL": + critical.append(title) + elif sev == "HIGH": + high.append(title) + + if critical: + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + findings=tuple(critical), + notes=f"{len(critical)} CRITICAL finding(s)", + ) + if high: + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + findings=tuple(high), + notes=f"{len(high)} HIGH finding(s)", + ) + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="No HIGH or CRITICAL findings.", + ) +``` diff --git a/dogfood/mining-output/code-context/pickled_iac_securitybaselinegate.md b/dogfood/mining-output/code-context/pickled_iac_securitybaselinegate.md new file mode 100644 index 0000000..ca8649f --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_iac_securitybaselinegate.md @@ -0,0 +1,97 @@ +# Code context: SecurityBaselineGate.run + +- **Surface id:** pickled_iac_securitybaselinegate +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 1 | **Total lines:** 87 | **Truncated:** False + +## Root: pickled_iac.gates.SecurityBaselineGate.run + +```python +def run( + self, + target: object, + *, + context: dict[str, Any] | None = None, + ) -> GateResult: + _ = context + if not isinstance(target, Path): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected Path to tf dir, got {type(target).__name__}", + ) + trivy = shutil.which("trivy") + if trivy is None: + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="trivy not found on PATH — security scan skipped", + ) + + proc = subprocess.run( + [ + trivy, + "config", + str(target), + "--format", + "json", + "--severity", + "HIGH,CRITICAL", + "--quiet", + ], + capture_output=True, + text=True, + check=False, + ) + if proc.returncode not in (0, 1) and not proc.stdout.strip(): + err = (proc.stderr or "trivy failed").strip() + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + notes=f"trivy error: {err}", + ) + + try: + report = json.loads(proc.stdout or "{}") + except json.JSONDecodeError: + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + notes="trivy returned non-JSON output", + ) + + critical: list[str] = [] + high: list[str] = [] + for result in report.get("Results", []) or []: + if not isinstance(result, dict): + continue + for mis in result.get("Misconfigurations", []) or []: + if not isinstance(mis, dict): + continue + sev = str(mis.get("Severity", "")).upper() + title = str(mis.get("Title", mis.get("ID", "finding"))) + if sev == "CRITICAL": + critical.append(title) + elif sev == "HIGH": + high.append(title) + + if critical: + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + findings=tuple(critical), + notes=f"{len(critical)} CRITICAL finding(s)", + ) + if high: + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + findings=tuple(high), + notes=f"{len(high)} HIGH finding(s)", + ) + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="No HIGH or CRITICAL findings.", + ) +``` diff --git a/dogfood/mining-output/code-context/pickled_iac_validate.md b/dogfood/mining-output/code-context/pickled_iac_validate.md new file mode 100644 index 0000000..c1d19b0 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_iac_validate.md @@ -0,0 +1,9 @@ +# Code context: validate + +- **Surface id:** pickled_iac_validate +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_rules_check.md b/dogfood/mining-output/code-context/pickled_rules_check.md new file mode 100644 index 0000000..3721519 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_rules_check.md @@ -0,0 +1,109 @@ +# Code context: check + +- **Surface id:** pickled_rules_check +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 2 | **Total lines:** 86 | **Truncated:** True + +## Root: pickled_rules.cli.check + +```python +def check( + ruleset: str, + feature_path: Path | None, + feature_glob: str | None, + ruleset_name: str | None, + output_format: str, + output: Path | None, + quiet: bool, +) -> None: + """Check feature coverage against a YAML rule set.""" + if feature_path is None and feature_glob is None: + raise click.ClickException("Provide --feature or --feature-glob") + if feature_path is not None and feature_glob is not None: + raise click.ClickException("Use only one of --feature or --feature-glob") + + try: + resolved_path = _resolve_ruleset_path(ruleset) + except KeyError as exc: + raise click.ClickException(str(exc)) from exc + + if ruleset_name is not None: + short_name = ruleset_name.lower() + elif ruleset in BUILTIN_RULESETS: + short_name = ruleset.lower() + else: + short_name = resolved_path.stem.lower() + + ruleset_obj = load_ruleset(resolved_path) + if feature_glob: + from glob import glob + + paths = [Path(p) for p in glob(feature_glob, recursive=True)] + paths = [p for p in paths if p.is_file()] + else: + assert feature_path is not None + paths = [feature_path] + + if not paths: + raise click.ClickException("No feature files matched") + + adapter = PytestBddAdapter() + parsed = [adapter.parse_feature_file(fp) for fp in sorted(paths)] + + if len(parsed) == 1: + report = coverage_gate(parsed[0], ruleset_obj, ruleset_short_name=short_name) + worst = report.gate_result.verdict + if output_format.lower() == "json": + body = render_coverage_json(report, ruleset_obj, feature_path=str(paths[0])) + else: + body = render_coverage_markdown(report, ruleset_obj, feature_path=str(paths[0])) + else: + report = coverage_gate_features( + parsed, ruleset_obj, ruleset_short_name=short_name + ) + worst = report.gate_result.verdict + label = ", ".join(str(p) for p in sorted(paths)) + if output_format.lower() == "json": + body = render_coverage_json(report, ruleset_obj, feature_path=label) + else: + body = render_coverage_markdown(report, ruleset_obj, feature_path=label) + if not quiet: + click.echo( + f"Union coverage across {len(paths)} feature file(s).", + err=True, + ) + + if quiet: + label = "PASS" if worst == Verdict.PASS else "FAIL" + click.echo(f"{label}: checked {len(paths)} feature(s)") + if output is not None: + output.write_text(body, encoding="utf-8") + elif output is not None: + output.write_text(body, encoding="utf-8") + click.echo(f"Report written to {output}", err=True) + else: + click.echo(body) + + if worst != Verdict.PASS: + sys.exit(1) +``` + +## Callee: pickled_rules.cli._resolve_ruleset_path (hop 1) + +```python +def _resolve_ruleset_path(ruleset: str) -> Path: + if ruleset in BUILTIN_RULESETS: + return resolve_ruleset_name(ruleset) + path = Path(ruleset) + if not path.is_file(): + raise click.ClickException(f"Rule set file not found: {ruleset}") + return path +``` + +## Unresolved callees + +- `adapter.parse_feature_file` — variable 'adapter' reassigned; type not stable + +## Notes + +Collection stopped early because of --max-callees or --max-code-lines. diff --git a/dogfood/mining-output/code-context/pickled_rules_coverage_gate.md b/dogfood/mining-output/code-context/pickled_rules_coverage_gate.md new file mode 100644 index 0000000..1aee21a --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_rules_coverage_gate.md @@ -0,0 +1,124 @@ +# Code context: coverage_gate + +- **Surface id:** pickled_rules_coverage_gate +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 3 | **Total lines:** 104 | **Truncated:** False + +## Root: pickled_rules.gates.coverage.coverage_gate + +```python +def coverage_gate( + feature: Feature, + ruleset: RuleSet, + *, + ruleset_short_name: str, +) -> CoverageReport: + """Compute coverage of ``ruleset`` rules by ``feature`` scenarios. + + A rule counts as referenced if at least one scenario carries a tag + ``@:`` (normalized without ``@`` in the model). + + The gate **passes** iff every **strict** rule is referenced and there are no + unknown reference tags. **Advisory** and **informational** rules may remain + unreferenced without failing. + """ + return coverage_gate_features( + (feature,), + ruleset, + ruleset_short_name=ruleset_short_name, + artifact_ref=feature.path if feature.path else "", + ) +``` + +## Callee: pickled_rules.gates.coverage.coverage_gate_features (hop 1) + +```python +def coverage_gate_features( + features: Sequence[Feature], + ruleset: RuleSet, + *, + ruleset_short_name: str, + artifact_ref: str | None = None, +) -> CoverageReport: + """Compute coverage across one or more features (union of scenario tags).""" + referenced_ids, unknown = _collect_references( + features, ruleset, ruleset_short_name=ruleset_short_name + ) + if artifact_ref is None: + paths = [f.path for f in features if f.path] + artifact_ref = ", ".join(paths) if paths else "" + + referenced = tuple(r for r in ruleset.rules if r.id in referenced_ids) + unreferenced = tuple(r for r in ruleset.rules if r.id not in referenced_ids) + + strict_unreferenced = [r for r in unreferenced if r.enforcement == "strict"] + passed = not strict_unreferenced and not unknown + + traces = tuple( + Trace( + source_reference=SourceReference( + source_id=f"{ruleset.source_id}({rule.id})", + source_version=ruleset.source_version, + locator=rule.id, + description=rule.description, + active_from=ruleset.active_from, + applies_to=ruleset.applies_to, + source_url=ruleset.source_url, + ), + artifact_kind="feature", + artifact_ref=artifact_ref, + relation="implements", + confidence="asserted", + ) + for rule in referenced + ) + + if passed: + notes = ( + f"All strict rules in {ruleset.source_id} are referenced; no unknown reference tags." + ) + else: + parts: list[str] = [] + if strict_unreferenced: + parts.append(f"{len(strict_unreferenced)} strict rule(s) unreferenced") + if unknown: + parts.append(f"{len(unknown)} unknown reference(s)") + notes = "; ".join(parts) + "." + + gate_result = GateResult( + gate_name="rules.coverage", + verdict=Verdict.PASS if passed else Verdict.FAIL, + findings=(), + notes=notes, + traces=traces, + ) + + return CoverageReport( + referenced_rules=referenced, + unreferenced_rules=unreferenced, + unknown_references=tuple(sorted(unknown)), + gate_result=gate_result, + ) +``` + +## Callee: pickled_rules.gates.coverage._collect_references (hop 2) + +```python +def _collect_references( + features: Sequence[Feature], + ruleset: RuleSet, + *, + ruleset_short_name: str, +) -> tuple[set[str], set[tuple[str, str]]]: + referenced_ids: set[str] = set() + unknown: set[tuple[str, str]] = set() + for feature in features: + scenario_refs = extract_references(feature, ruleset_filter=ruleset_short_name) + for sc in scenario_refs: + for _ruleset_name, rule_id in sc.references: + if ruleset.find(rule_id) is None: + unknown.add((_ruleset_name, rule_id)) + else: + referenced_ids.add(rule_id) + return referenced_ids, unknown +``` diff --git a/dogfood/mining-output/code-context/pickled_rules_coverage_gate_features.md b/dogfood/mining-output/code-context/pickled_rules_coverage_gate_features.md new file mode 100644 index 0000000..06b7494 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_rules_coverage_gate_features.md @@ -0,0 +1,138 @@ +# Code context: coverage_gate_features + +- **Surface id:** pickled_rules_coverage_gate_features +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 4 | **Total lines:** 108 | **Truncated:** False + +## Root: pickled_rules.gates.coverage.coverage_gate_features + +```python +def coverage_gate_features( + features: Sequence[Feature], + ruleset: RuleSet, + *, + ruleset_short_name: str, + artifact_ref: str | None = None, +) -> CoverageReport: + """Compute coverage across one or more features (union of scenario tags).""" + referenced_ids, unknown = _collect_references( + features, ruleset, ruleset_short_name=ruleset_short_name + ) + if artifact_ref is None: + paths = [f.path for f in features if f.path] + artifact_ref = ", ".join(paths) if paths else "" + + referenced = tuple(r for r in ruleset.rules if r.id in referenced_ids) + unreferenced = tuple(r for r in ruleset.rules if r.id not in referenced_ids) + + strict_unreferenced = [r for r in unreferenced if r.enforcement == "strict"] + passed = not strict_unreferenced and not unknown + + traces = tuple( + Trace( + source_reference=SourceReference( + source_id=f"{ruleset.source_id}({rule.id})", + source_version=ruleset.source_version, + locator=rule.id, + description=rule.description, + active_from=ruleset.active_from, + applies_to=ruleset.applies_to, + source_url=ruleset.source_url, + ), + artifact_kind="feature", + artifact_ref=artifact_ref, + relation="implements", + confidence="asserted", + ) + for rule in referenced + ) + + if passed: + notes = ( + f"All strict rules in {ruleset.source_id} are referenced; no unknown reference tags." + ) + else: + parts: list[str] = [] + if strict_unreferenced: + parts.append(f"{len(strict_unreferenced)} strict rule(s) unreferenced") + if unknown: + parts.append(f"{len(unknown)} unknown reference(s)") + notes = "; ".join(parts) + "." + + gate_result = GateResult( + gate_name="rules.coverage", + verdict=Verdict.PASS if passed else Verdict.FAIL, + findings=(), + notes=notes, + traces=traces, + ) + + return CoverageReport( + referenced_rules=referenced, + unreferenced_rules=unreferenced, + unknown_references=tuple(sorted(unknown)), + gate_result=gate_result, + ) +``` + +## Callee: pickled_rules.gates.coverage._collect_references (hop 1) + +```python +def _collect_references( + features: Sequence[Feature], + ruleset: RuleSet, + *, + ruleset_short_name: str, +) -> tuple[set[str], set[tuple[str, str]]]: + referenced_ids: set[str] = set() + unknown: set[tuple[str, str]] = set() + for feature in features: + scenario_refs = extract_references(feature, ruleset_filter=ruleset_short_name) + for sc in scenario_refs: + for _ruleset_name, rule_id in sc.references: + if ruleset.find(rule_id) is None: + unknown.add((_ruleset_name, rule_id)) + else: + referenced_ids.add(rule_id) + return referenced_ids, unknown +``` + +## Callee: pickled_rules.references.extract_references (hop 2) + +```python +def extract_references( + feature: Feature, + *, + ruleset_filter: str | None = None, +) -> list[ScenarioReferences]: + """Extract reference tags from each scenario. + + If ``ruleset_filter`` is set, only tags whose ruleset prefix matches are kept + (comparison is case-insensitive on the ruleset side). + """ + out: list[ScenarioReferences] = [] + for scenario in feature.scenarios: + refs = tuple(_parse_reference_tags(scenario.tags, ruleset_filter)) + out.append( + ScenarioReferences( + scenario_name=scenario.name, + references=refs, + ) + ) + return out +``` + +## Callee: pickled_rules.RuleSet.find (hop 2) + +```python +def find(self, rule_id: str) -> Rule | None: + for rule in self.rules: + if rule.id == rule_id: + return rule + return None +``` + +## Unresolved callees + +- `unknown.add` — method name matches multiple classes; receiver type not pinned +- `referenced_ids.add` — method name matches multiple classes; receiver type not pinned diff --git a/dogfood/mining-output/code-context/pickled_rules_draft.md b/dogfood/mining-output/code-context/pickled_rules_draft.md new file mode 100644 index 0000000..95c497a --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_rules_draft.md @@ -0,0 +1,192 @@ +# Code context: draft + +- **Surface id:** pickled_rules_draft +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 8 | **Total lines:** 141 | **Truncated:** False + +## Root: pickled_rules.cli.draft + +```python +def draft( + brief: str, + short_name: str, + source_id: str, + applies_to: str, + active_from: str, + output: Path | None, +) -> None: + """Draft a YAML rule set from a natural-language brief.""" + try: + llm = _build_llm_client() + result = RuleSetDrafter(llm).draft_from_brief( + brief_text=_read_text_arg(brief), + ruleset_short_name=short_name, + source_id=source_id, + applies_to=applies_to, + active_from=active_from, + ) + except click.ClickException: + raise + except Exception as exc: + click.echo(str(exc), err=True) + raise SystemExit(2) from exc + _emit_draft_output( + text=result.text, + rationale=result.rationale, + warnings=result.warnings, + output=output, + ) +``` + +## Callee: pickled_rules.cli._build_llm_client (hop 1) + +```python +def _build_llm_client() -> LLMClient: + from pickled_core.llm.bootstrap import build_default_client + from pickled_core.llm.config import ConfigError + + try: + return build_default_client(factory_env="PICKLED_RULES_LLM_FACTORY") + except ConfigError as exc: + raise click.ClickException(str(exc)) from exc +``` + +## Callee: pickled_rules.RuleSetDrafter.draft_from_brief (hop 1) + +```python +def draft_from_brief( + self, + *, + brief_text: str, + ruleset_short_name: str, + source_id: str, + applies_to: str, + active_from: str, + ) -> DraftResult: + prompt = self._build_prompt( + brief_text=brief_text, + ruleset_short_name=ruleset_short_name, + source_id=source_id, + applies_to=applies_to, + active_from=active_from, + ) + completion = self._llm.complete( + messages=[Message(role="user", content=prompt)], + model=_DRAFT_MODEL, + max_tokens=4000, + temperature=0.0, + stop=None, + extras=None, + ) + text, rationale = self._split_output(completion.text) + warnings = tuple(self._validate(text)) + return DraftResult(text=text, rationale=rationale, warnings=warnings) +``` + +## Callee: pickled_rules.cli._read_text_arg (hop 1) + +```python +def _read_text_arg(path: str) -> str: + if path == "-": + return sys.stdin.read() + return Path(path).read_text(encoding="utf-8") +``` + +## Callee: pickled_rules.cli._emit_draft_output (hop 1) + +```python +def _emit_draft_output( + *, + text: str, + rationale: str, + warnings: tuple[str, ...], + output: Path | None, +) -> None: + if output is not None: + output.write_text(text, encoding="utf-8") + else: + click.echo(text) + if rationale: + for line in rationale.splitlines(): + click.echo(f"rationale: {line}", err=True) + for warning in warnings: + click.echo(f"warning: {warning}", err=True) + if warnings: + raise SystemExit(1) +``` + +## Callee: pickled_rules.RuleSetDrafter._build_prompt (hop 2) + +```python +def _build_prompt( + self, + *, + brief_text: str, + ruleset_short_name: str, + source_id: str, + applies_to: str, + active_from: str, + ) -> str: + banned = ", ".join(sorted(_FORBIDDEN_TOKENS)) + return ( + "You are drafting a YAML rule set for the pickled-rules tool. " + "The input is a natural-language brief describing a domain. " + "Emit YAML matching this schema exactly:\n\n" + "```yaml\n" + "metadata:\n" + f' source_id: "{source_id}"\n' + ' source_title: ""\n' + f' applies_to: "{applies_to}"\n' + ' maintainer: "drafted by LLM"\n' + ' source_version: "0.1"\n' + f' active_from: "{active_from}"\n' + "rules:\n" + ' - id: ""\n' + ' title: "<5-10 words>"\n' + ' description: ""\n' + ' enforcement: "strict" | "advisory" | "informational"\n' + "```\n\n" + f"Ruleset short name (for tagging): {ruleset_short_name}\n\n" + "Brief:\n" + f"{brief_text.strip()}\n\n" + "Rules MUST use neutral, vendor-agnostic phrasing. " + f"Do NOT include any of these tokens (case-insensitive): {banned}. " + "Domain-specific terms belong only inside description: fields.\n\n" + f"After the YAML, emit the literal line {RATIONALE_SENTINEL!r} then " + "1-3 sentences explaining your rule selection." + ) +``` + +## Callee: pickled_rules.RuleSetDrafter._split_output (hop 2) + +```python +def _split_output(self, raw: str) -> tuple[str, str]: + if RATIONALE_SENTINEL in raw: + text, _, rationale = raw.partition(RATIONALE_SENTINEL) + return text.strip(), rationale.strip() + return raw.strip(), "" +``` + +## Callee: pickled_rules.RuleSetDrafter._validate (hop 2) + +```python +def _validate(self, text: str) -> list[str]: + warnings: list[str] = [] + try: + load_ruleset_from_text(text) + except RuleSetValidationError as exc: + warnings.append(str(exc)) + lowered = text.lower() + for token in sorted(_FORBIDDEN_TOKENS): + if token in lowered: + warnings.append( + f"forbidden token '{token}' in YAML; rewrite the rule" + ) + return warnings +``` + +## Unresolved callees + +- `self._llm.complete` — protocol or unknown attribute type +- `sys.stdin.read` — method name matches multiple classes; receiver type not pinned +- `rationale.splitlines` — method name matches multiple classes; receiver type not pinned diff --git a/dogfood/mining-output/code-context/pickled_rules_list_rules.md b/dogfood/mining-output/code-context/pickled_rules_list_rules.md new file mode 100644 index 0000000..4317c37 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_rules_list_rules.md @@ -0,0 +1,9 @@ +# Code context: list-rules + +- **Surface id:** pickled_rules_list_rules +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/pickled_rules_mcp.md b/dogfood/mining-output/code-context/pickled_rules_mcp.md new file mode 100644 index 0000000..8eee5be --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_rules_mcp.md @@ -0,0 +1,12 @@ +# Code context: mcp + +- **Surface id:** pickled_rules_mcp +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 1 | **Total lines:** 2 | **Truncated:** False + +## Root: pickled_rules.cli.mcp + +```python +def mcp() -> None: + """MCP server commands.""" +``` diff --git a/dogfood/mining-output/code-context/pickled_rules_run_all.md b/dogfood/mining-output/code-context/pickled_rules_run_all.md new file mode 100644 index 0000000..ce94318 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_rules_run_all.md @@ -0,0 +1,323 @@ +# Code context: run_all + +- **Surface id:** pickled_rules_run_all +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 7 | **Total lines:** 274 | **Truncated:** True + +## Root: pickled_rules.gates_runner.run_all + +```python +def run_all(workdir: Path | str) -> list[GateResult]: + """Run coverage gate for each feature against ``pickled.ruleset.yaml``.""" + root = Path(workdir).resolve() + cfg = _workdir_config(root) + try: + entries = _resolve_ruleset_entries(root, cfg) + except RuleSetValidationError as exc: + return [ + GateResult( + gate_name="rules.coverage", + verdict=Verdict.FAIL, + notes=str(exc), + ) + ] + + if not entries: + return [ + GateResult( + gate_name="rules.coverage", + verdict=Verdict.WARN, + notes="missing pickled.ruleset.yaml or ruleset/rulesets key", + ) + ] + + feature_pattern = _feature_glob(cfg) + features = sorted(root.glob(feature_pattern)) + if not features: + return [ + GateResult( + gate_name="rules.coverage", + verdict=Verdict.WARN, + notes="no feature files", + ) + ] + + adapter = PytestBddAdapter() + parsed = [adapter.parse_feature_file(path) for path in features] + + results: list[GateResult] = [] + for entry in entries: + gate_name = ( + "rules.coverage" + if len(entries) == 1 + else f"rules.coverage.{entry.short_name}" + ) + if not entry.path.is_file(): + results.append( + GateResult( + gate_name=gate_name, + verdict=Verdict.FAIL, + notes=f"ruleset not found: {entry.path}", + ) + ) + continue + try: + ruleset = load_ruleset(entry.path) + except RuleSetValidationError as exc: + results.append( + GateResult( + gate_name=f"rules.load.{entry.short_name}", + verdict=Verdict.FAIL, + notes=str(exc), + ) + ) + continue + report = coverage_gate_features( + parsed, ruleset, ruleset_short_name=entry.short_name + ) + gr = report.gate_result + results.append( + GateResult( + gate_name=gate_name, + verdict=gr.verdict, + findings=gr.findings, + notes=gr.notes, + traces=gr.traces, + ) + ) + return results +``` + +## Callee: pickled_rules.gates_runner._workdir_config (hop 1) + +```python +def _workdir_config(root: Path) -> dict[str, Any]: + cfg_path = root / "pickled.ruleset.yaml" + if not cfg_path.is_file(): + return {} + data = yaml.safe_load(cfg_path.read_text(encoding="utf-8")) + return data if isinstance(data, dict) else {} +``` + +## Callee: pickled_rules.gates_runner._resolve_ruleset_entries (hop 1) + +```python +def _resolve_ruleset_entries(root: Path, cfg: dict[str, Any]) -> list[_RulesetEntry]: + has_singular = "ruleset" in cfg + has_plural = "rulesets" in cfg + if has_singular and has_plural: + msg = ( + "pickled.ruleset.yaml: keys 'ruleset' and 'rulesets' are mutually exclusive" + ) + raise RuleSetValidationError(msg) + + if has_singular: + ruleset_rel = cfg.get("ruleset") + if not isinstance(ruleset_rel, str): + msg = "pickled.ruleset.yaml: 'ruleset' must be a string path" + raise RuleSetValidationError(msg) + path = (root / ruleset_rel).resolve() + short_name = str(cfg.get("ruleset_short_name", path.stem)) + return [_RulesetEntry(path=path, short_name=short_name)] + + if has_plural: + raw_list = cfg.get("rulesets") + if not isinstance(raw_list, list): + msg = "pickled.ruleset.yaml: 'rulesets' must be a list" + raise RuleSetValidationError(msg) + if not raw_list: + msg = "pickled.ruleset.yaml: at least one ruleset entry required" + raise RuleSetValidationError(msg) + + entries: list[_RulesetEntry] = [] + seen_short_names: dict[str, int] = {} + for index, item in enumerate(raw_list): + if not isinstance(item, dict): + msg = f"pickled.ruleset.yaml: rulesets[{index}] must be a mapping" + raise RuleSetValidationError(msg) + path_raw = item.get("path") + if not isinstance(path_raw, str): + msg = f"pickled.ruleset.yaml: rulesets[{index}].path must be a string" + raise RuleSetValidationError(msg) + short_raw = item.get("short_name") + if short_raw is not None and not isinstance(short_raw, str): + msg = ( + f"pickled.ruleset.yaml: rulesets[{index}].short_name must be a string" + ) + raise RuleSetValidationError(msg) + resolved = (root / path_raw).resolve() + short_name = str(short_raw) if short_raw is not None else Path(path_raw).stem + if short_name in seen_short_names: + prior = seen_short_names[short_name] + msg = ( + f"duplicate short_name {short_name!r} at positions " + f"[{prior}, {index}]" + ) + raise RuleSetValidationError(msg) + seen_short_names[short_name] = index + entries.append(_RulesetEntry(path=resolved, short_name=short_name)) + return entries + + return [] +``` + +## Callee: pickled_rules.gates_runner._feature_glob (hop 1) + +```python +def _feature_glob(cfg: dict[str, Any]) -> str: + raw = cfg.get("feature_glob") + if raw is None: + return "features/**/*.feature" + if not isinstance(raw, str): + msg = "pickled.ruleset.yaml: 'feature_glob' must be a string" + raise RuleSetValidationError(msg) + return raw +``` + +## Callee: pickled_rules.gates.coverage.coverage_gate_features (hop 1) + +```python +def coverage_gate_features( + features: Sequence[Feature], + ruleset: RuleSet, + *, + ruleset_short_name: str, + artifact_ref: str | None = None, +) -> CoverageReport: + """Compute coverage across one or more features (union of scenario tags).""" + referenced_ids, unknown = _collect_references( + features, ruleset, ruleset_short_name=ruleset_short_name + ) + if artifact_ref is None: + paths = [f.path for f in features if f.path] + artifact_ref = ", ".join(paths) if paths else "" + + referenced = tuple(r for r in ruleset.rules if r.id in referenced_ids) + unreferenced = tuple(r for r in ruleset.rules if r.id not in referenced_ids) + + strict_unreferenced = [r for r in unreferenced if r.enforcement == "strict"] + passed = not strict_unreferenced and not unknown + + traces = tuple( + Trace( + source_reference=SourceReference( + source_id=f"{ruleset.source_id}({rule.id})", + source_version=ruleset.source_version, + locator=rule.id, + description=rule.description, + active_from=ruleset.active_from, + applies_to=ruleset.applies_to, + source_url=ruleset.source_url, + ), + artifact_kind="feature", + artifact_ref=artifact_ref, + relation="implements", + confidence="asserted", + ) + for rule in referenced + ) + + if passed: + notes = ( + f"All strict rules in {ruleset.source_id} are referenced; no unknown reference tags." + ) + else: + parts: list[str] = [] + if strict_unreferenced: + parts.append(f"{len(strict_unreferenced)} strict rule(s) unreferenced") + if unknown: + parts.append(f"{len(unknown)} unknown reference(s)") + notes = "; ".join(parts) + "." + + gate_result = GateResult( + gate_name="rules.coverage", + verdict=Verdict.PASS if passed else Verdict.FAIL, + findings=(), + notes=notes, + traces=traces, + ) + + return CoverageReport( + referenced_rules=referenced, + unreferenced_rules=unreferenced, + unknown_references=tuple(sorted(unknown)), + gate_result=gate_result, + ) +``` + +## Callee: pickled_rules.loader.load_ruleset (hop 1) + +```python +def load_ruleset(path: Path) -> RuleSet: + """Load a rule set from YAML. + + Raises `RuleSetValidationError` on malformed input. + """ + try: + with path.open(encoding="utf-8") as fh: + raw = yaml.safe_load(fh) + except yaml.YAMLError as exc: + raise RuleSetValidationError(f"Malformed YAML in {path}: {exc}") from exc + + if not isinstance(raw, dict): + raise RuleSetValidationError(f"Rule set root must be a mapping, got {type(raw).__name__}") + + metadata_raw = _require(raw, "metadata", path) + if not isinstance(metadata_raw, dict): + raise RuleSetValidationError( + f"`metadata` must be a mapping in {path}, got {type(metadata_raw).__name__}" + ) + metadata = metadata_raw + + rules_raw = _require(raw, "rules", path) + if not isinstance(rules_raw, list): + raise RuleSetValidationError(f"`rules` must be a list in {path}") + + rules = tuple(_parse_rule(r, path) for r in rules_raw) + + source_url_raw = metadata.get("source_url") + if source_url_raw is not None and not isinstance(source_url_raw, str): + raise RuleSetValidationError(f"`source_url` must be a string or null in {path}") + + return RuleSet( + source_id=_str_field(metadata, "source_id", path), + source_title=_str_field(metadata, "source_title", path), + applies_to=_str_field(metadata, "applies_to", path), + maintainer=_str_field(metadata, "maintainer", path), + source_version=_str_field(metadata, "source_version", path), + active_from=_parse_date(_require(metadata, "active_from", path), path), + source_url=source_url_raw, + rules=rules, + ) +``` + +## Callee: pickled_rules.gates.coverage._collect_references (hop 2) + +```python +def _collect_references( + features: Sequence[Feature], + ruleset: RuleSet, + *, + ruleset_short_name: str, +) -> tuple[set[str], set[tuple[str, str]]]: + referenced_ids: set[str] = set() + unknown: set[tuple[str, str]] = set() + for feature in features: + scenario_refs = extract_references(feature, ruleset_filter=ruleset_short_name) + for sc in scenario_refs: + for _ruleset_name, rule_id in sc.references: + if ruleset.find(rule_id) is None: + unknown.add((_ruleset_name, rule_id)) + else: + referenced_ids.add(rule_id) + return referenced_ids, unknown +``` + +## Unresolved callees + +- `adapter.parse_feature_file` — variable 'adapter' reassigned; type not stable +- `path.open` — method name matches multiple classes; receiver type not pinned + +## Notes + +Collection stopped early because of --max-callees or --max-code-lines. diff --git a/dogfood/mining-output/code-context/pickled_schema_check.md b/dogfood/mining-output/code-context/pickled_schema_check.md new file mode 100644 index 0000000..416bb73 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_schema_check.md @@ -0,0 +1,190 @@ +# Code context: check + +- **Surface id:** pickled_schema_check +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 7 | **Total lines:** 150 | **Truncated:** False + +## Root: pickled_schema.cli.check + +```python +def check(spec: Path, feature_dir: Path | None, feature_glob: str | None) -> None: + """Run SchemaCoverageGate on @schema:endpoint tags in .feature files.""" + spec_dict, _, _ = load_openapi_file(spec) + feature_paths = _resolve_feature_paths(feature_dir, feature_glob) + if not feature_paths: + raise click.ClickException("No feature files matched") + gate = SchemaCoverageGate() + result = gate.run(spec_dict, context={"feature_paths": feature_paths}) + payload = { + "gate": result.gate_name, + "verdict": result.verdict.value, + "notes": result.notes, + "findings": [ + {"tag": f.tag, "source": f.source} + for f in result.findings + if isinstance(f, SchemaCoverageFinding) + ], + } + click.echo(json.dumps(payload, indent=2)) + if result.verdict is Verdict.FAIL: + raise SystemExit(2) + if result.verdict is Verdict.WARN: + raise SystemExit(1) +``` + +## Callee: pickled_schema.openapi.parser.load_openapi_file (hop 1) + +```python +def load_openapi_file(path: Path) -> tuple[dict[str, Any], SchemaFormat, SchemaArtifact]: + """Load a file and return parsed dict, detected format, and artifact.""" + data, raw = _load_text(path) + fmt = detect_format(data) + artifact = SchemaArtifact( + format=fmt, + content=raw, + endpoint_id=None, + source="file", + ) + return data, fmt, artifact +``` + +## Callee: pickled_schema.cli._resolve_feature_paths (hop 1) + +```python +def _resolve_feature_paths( + feature_dir: Path | None, + feature_glob: str | None, +) -> list[Path]: + if feature_dir is not None and feature_glob is not None: + raise click.ClickException("Use only one of --feature-dir or --feature-glob") + if feature_dir is not None: + return sorted(feature_dir.glob("**/*.feature")) + if feature_glob is not None: + from glob import glob + + return sorted(Path(p) for p in glob(feature_glob, recursive=True) if Path(p).is_file()) + raise click.ClickException("Provide --feature-dir or --feature-glob") +``` + +## Callee: pickled_schema.SchemaCoverageGate.run (hop 1) + +```python +def run( + self, + target: object, + *, + context: dict[str, Any] | None = None, + ) -> GateResult: + ctx = context or {} + if not isinstance(target, dict): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected parsed OpenAPI dict, got {type(target).__name__}", + ) + + feature_paths: list[Path] = [] + raw_paths = ctx.get("feature_paths") + if isinstance(raw_paths, list): + feature_paths = [Path(p) for p in raw_paths] + + feature_texts: list[tuple[str, str]] = [] + raw_texts = ctx.get("feature_texts") + if isinstance(raw_texts, list): + for i, text in enumerate(raw_texts): + if isinstance(text, str): + feature_texts.append((f"", text)) + + if not feature_paths and not feature_texts: + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes='context needs "feature_paths" and/or "feature_texts"', + ) + + missing: list[SchemaCoverageFinding] = [] + for path in feature_paths: + text = path.read_text(encoding="utf-8") + missing.extend( + _missing_tags(target, text, source=str(path)), + ) + for source, text in feature_texts: + missing.extend(_missing_tags(target, text, source=source)) + + if not missing: + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="All @schema:endpoint tags have matching paths.", + ) + + lines = [f"{f.tag} ({f.source})" for f in missing] + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + findings=tuple(missing), + notes="Missing endpoints: " + "; ".join(lines), + ) +``` + +## Callee: pickled_schema.openapi.parser._load_text (hop 2) + +```python +def _load_text(path: Path) -> tuple[dict[str, Any], str]: + raw = path.read_text(encoding="utf-8") + suffix = path.suffix.lower() + if suffix in {".yaml", ".yml"}: + data = yaml.safe_load(raw) + elif suffix == ".json": + data = json.loads(raw) + else: + try: + data = yaml.safe_load(raw) + except yaml.YAMLError as exc: + msg = f"cannot parse {path}: not valid YAML or JSON" + raise SchemaParseError(msg) from exc + if not isinstance(data, dict): + msg = "schema root must be a mapping" + raise SchemaParseError(msg) + return data, raw +``` + +## Callee: pickled_schema.openapi.parser.detect_format (hop 2) + +```python +def detect_format(spec_dict: dict[str, Any]) -> SchemaFormat: + """Infer OpenAPI version from a parsed document root.""" + if "swagger" in spec_dict: + msg = "OpenAPI 2.0 (swagger field) is not supported in v0.1" + raise SchemaParseError(msg) + version = spec_dict.get("openapi") + if not isinstance(version, str): + msg = "missing or invalid top-level 'openapi' version field" + raise SchemaParseError(msg) + if version.startswith("3.2"): + return SchemaFormat.openapi_3_2 + if version.startswith("3.1"): + return SchemaFormat.openapi_3_1 + if version.startswith("3.0"): + return SchemaFormat.openapi_3_0 + msg = f"unsupported OpenAPI version {version!r}" + raise SchemaParseError(msg) +``` + +## Callee: pickled_schema._missing_tags (hop 2) + +```python +def _missing_tags( + spec: dict[str, Any], + feature_text: str, + *, + source: str, +) -> list[SchemaCoverageFinding]: + out: list[SchemaCoverageFinding] = [] + for match in _ENDPOINT_TAG_RE.finditer(feature_text): + method, path = match.group(1), match.group(2) + tag = match.group(0) + if not _spec_has_endpoint(spec, method, path): + out.append(SchemaCoverageFinding(tag=tag, source=source)) + return out +``` diff --git a/dogfood/mining-output/code-context/pickled_schema_draft.md b/dogfood/mining-output/code-context/pickled_schema_draft.md new file mode 100644 index 0000000..6d64f04 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_schema_draft.md @@ -0,0 +1,165 @@ +# Code context: draft + +- **Surface id:** pickled_schema_draft +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 5 | **Total lines:** 128 | **Truncated:** False + +## Root: pickled_schema.cli.draft + +```python +def draft( + method: str, + endpoint_path: str, + gherkin_file: Path, + output: Path | None, +) -> None: + """Draft an OpenAPI 3.1 path item from a Gherkin scenario.""" + gherkin = gherkin_file.read_text(encoding="utf-8") + llm = _build_llm_client() + artifact = OpenAPIDrafter(llm).draft_endpoint( + method, + endpoint_path, + gherkin, + ) + if output: + output.write_text(artifact.content, encoding="utf-8") + click.echo(f"Wrote {output}", err=True) + else: + click.echo(artifact.content) +``` + +## Callee: pickled_schema.cli._build_llm_client (hop 1) + +```python +def _build_llm_client() -> LLMClient: + factory = os.environ.get("PICKLED_SCHEMA_LLM_FACTORY") + if factory: + module_name, sep, attr = factory.partition(":") + if not sep: + raise click.ClickException( + "PICKLED_SCHEMA_LLM_FACTORY must be 'module:callable'" + ) + module = importlib.import_module(module_name) + return cast(LLMClient, getattr(module, attr)()) + + from pickled_core.llm.config import load_config + from pickled_core.llm.factory import build_client + + provider = os.environ.get("PICKLED_LLM_PROVIDER", "anthropic") + try: + return build_client(provider, config=load_config()) + except ConfigError as exc: + raise click.ClickException(str(exc)) from exc +``` + +## Callee: pickled_schema.openapi.OpenAPIDrafter.draft_endpoint (hop 1) + +```python +def draft_endpoint( + self, + method: str, + path: str, + gherkin_context: str, + existing_component_names: list[str] | None = None, + ) -> SchemaArtifact: + """Draft, validate, and return a path-item ``SchemaArtifact``.""" + method_upper = method.upper() + components = existing_component_names or [] + last_error = "" + path_item: dict[str, Any] | None = None + + for _attempt in range(3): + extra = f"\n\nPrevious validation errors:\n{last_error}" if last_error else "" + prompt = self._template.render( + method=method_upper, + path=path, + gherkin_context=gherkin_context + extra, + existing_component_names=", ".join(components) or "(none)", + ) + from pickled_core.llm.turns import complete_prompt + + raw = complete_prompt( + self._llm, + prompt, + system="Output only YAML for the path item. No fences, no prose.", + ) + loaded = yaml.safe_load(raw.strip()) + if not isinstance(loaded, dict): + last_error = "LLM output is not a YAML mapping" + continue + path_item = _unwrap_path_item(loaded, method.lower()) + envelope = { + "openapi": "3.1.0", + "info": {"title": "draft", "version": "0.0.0"}, + "paths": {path: {method.lower(): path_item}}, + "components": {"schemas": {}}, + } + try: + validate_openapi_dict(envelope) + except SchemaValidationError as exc: + last_error = "; ".join(exc.errors) or str(exc) + continue + yaml_out = yaml.safe_dump( + path_item, + sort_keys=False, + default_flow_style=False, + ) + return SchemaArtifact( + format=SchemaFormat.openapi_3_1, + content=yaml_out, + endpoint_id=f"{method_upper}-{path}", + source="draft", + ) + + msg = f"failed to draft valid OpenAPI after 3 attempts: {last_error}" + raise SchemaValidationError(msg, errors=[last_error] if last_error else []) +``` + +## Callee: pickled_schema.openapi._unwrap_path_item (hop 2) + +```python +def _unwrap_path_item(loaded: dict[str, Any], method: str) -> dict[str, Any]: + """Accept a bare operation object or a one-key path-item wrapper.""" + if method in loaded and all(k in _HTTP_METHODS for k in loaded): + op = loaded[method] + return op if isinstance(op, dict) else loaded + if len(loaded) == 1: + only_key = next(iter(loaded)) + if only_key in _HTTP_METHODS: + inner = loaded[only_key] + if isinstance(inner, dict): + return inner + return loaded +``` + +## Callee: pickled_schema.openapi.validator.validate_openapi_dict (hop 2) + +```python +def validate_openapi_dict(spec_dict: dict[str, Any]) -> None: + """Validate *spec_dict* with openapi-spec-validator.""" + try: + from openapi_spec_validator import validate + from openapi_spec_validator.exceptions import OpenAPIError + from openapi_spec_validator.validation.exceptions import ( + OpenAPIValidationError, + ) + except ImportError as exc: + msg = "install pickled-schema[openapi] for OpenAPI validation" + raise SchemaValidationError(msg) from exc + + try: + validate(spec_dict) + except (OpenAPIError, OpenAPIValidationError) as exc: + errors = [str(exc)] + nested = getattr(exc, "schema_errors", None) + if nested: + errors.extend(str(e) for e in nested) + raise SchemaValidationError("OpenAPI validation failed", errors=errors) from exc +``` + +## Unresolved callees + +- `factory.partition` — variable 'factory' reassigned; type not stable +- `getattr(module, attr)()` — dynamic attribute access +- `getattr(module, attr)` — dynamic attribute access +- `self._template.render` — protocol or unknown attribute type diff --git a/dogfood/mining-output/code-context/pickled_schema_mcp.md b/dogfood/mining-output/code-context/pickled_schema_mcp.md new file mode 100644 index 0000000..5793541 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_schema_mcp.md @@ -0,0 +1,12 @@ +# Code context: mcp + +- **Surface id:** pickled_schema_mcp +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 1 | **Total lines:** 2 | **Truncated:** False + +## Root: pickled_schema.cli.mcp + +```python +def mcp() -> None: + """MCP server commands.""" +``` diff --git a/dogfood/mining-output/code-context/pickled_schema_parse.md b/dogfood/mining-output/code-context/pickled_schema_parse.md new file mode 100644 index 0000000..54d803e --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_schema_parse.md @@ -0,0 +1,136 @@ +# Code context: parse + +- **Surface id:** pickled_schema_parse +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 6 | **Total lines:** 101 | **Truncated:** False + +## Root: pickled_schema.cli.parse + +```python +def parse(file: Path, fmt: str | None) -> None: + """Parse a schema file and print a short summary.""" + format_enum = SchemaFormat(fmt) if fmt else _infer_format(file) + artifact = _load_artifact(file, format_enum) + click.echo( + json.dumps( + { + "format": artifact.format.value, + "endpoint_id": artifact.endpoint_id, + "source": artifact.source, + "content_bytes": len(artifact.content.encode("utf-8")), + }, + indent=2, + ) + ) +``` + +## Callee: pickled_schema.cli._infer_format (hop 1) + +```python +def _infer_format(path: Path) -> SchemaFormat: + suffix = path.suffix.lower() + if suffix in {".yaml", ".yml"}: + return SchemaFormat.openapi_3_1 + if suffix == ".json": + return SchemaFormat.json_schema_2020_12 + if suffix == ".proto": + return SchemaFormat.proto3 + msg = f"cannot infer format from extension {suffix!r}; use --format" + raise click.ClickException(msg) +``` + +## Callee: pickled_schema.cli._load_artifact (hop 1) + +```python +def _load_artifact(path: Path, fmt: SchemaFormat) -> SchemaArtifact: + if fmt in ( + SchemaFormat.openapi_3_0, + SchemaFormat.openapi_3_1, + SchemaFormat.openapi_3_2, + ): + _, detected, artifact = load_openapi_file(path) + return SchemaArtifact( + format=detected, + content=artifact.content, + endpoint_id=artifact.endpoint_id, + source=artifact.source, + ) + if fmt is SchemaFormat.json_schema_2020_12: + _, artifact = load_json_schema_file(path) + return artifact + if fmt is SchemaFormat.proto3: + return parse_proto_file(path) + raise click.ClickException(f"unsupported format {fmt!r}") +``` + +## Callee: pickled_schema.openapi.parser.load_openapi_file (hop 2) + +```python +def load_openapi_file(path: Path) -> tuple[dict[str, Any], SchemaFormat, SchemaArtifact]: + """Load a file and return parsed dict, detected format, and artifact.""" + data, raw = _load_text(path) + fmt = detect_format(data) + artifact = SchemaArtifact( + format=fmt, + content=raw, + endpoint_id=None, + source="file", + ) + return data, fmt, artifact +``` + +## Callee: pickled_schema.json_schema.parser.load_json_schema_file (hop 2) + +```python +def load_json_schema_file(path: Path) -> tuple[dict[str, Any], SchemaArtifact]: + raw = path.read_text(encoding="utf-8") + data = json.loads(raw) + if not isinstance(data, dict): + msg = "JSON Schema root must be an object" + raise SchemaParseError(msg) + artifact = SchemaArtifact( + format=SchemaFormat.json_schema_2020_12, + content=raw, + endpoint_id=None, + source="file", + ) + return data, artifact +``` + +## Callee: pickled_schema.proto.parser.parse_proto_file (hop 2) + +```python +def parse_proto_file( + proto_path: Path, + proto_dir: Path | None = None, +) -> SchemaArtifact: + """Parse a .proto file and return a descriptor set as base64 text.""" + pd = proto_dir or proto_path.parent + with tempfile.TemporaryDirectory() as tmp: + out = Path(tmp) / "descriptor.bin" + result = subprocess.run( + [ + sys.executable, + "-m", + "grpc_tools.protoc", + f"--proto_path={pd}", + f"--descriptor_set_out={out}", + "--include_imports", + str(proto_path), + ], + check=False, + capture_output=True, + text=True, + ) + if result.returncode != 0: + err = (result.stderr or result.stdout or "protoc failed").strip() + msg = f"protoc failed: {err}" + raise RuntimeError(msg) + data = out.read_bytes() + return SchemaArtifact( + format=SchemaFormat.proto3, + content=base64.b64encode(data).decode("ascii"), + endpoint_id=None, + source="file", + ) +``` diff --git a/dogfood/mining-output/code-context/pickled_schema_run_all.md b/dogfood/mining-output/code-context/pickled_schema_run_all.md new file mode 100644 index 0000000..abd5af8 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_schema_run_all.md @@ -0,0 +1,171 @@ +# Code context: run_all + +- **Surface id:** pickled_schema_run_all +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 5 | **Total lines:** 135 | **Truncated:** False + +## Root: pickled_schema.gates_runner.run_all + +```python +def run_all(workdir: Path | str) -> list[GateResult]: + """Validate OpenAPI under ``specs/`` and run schema coverage on features.""" + root = Path(workdir).resolve() + results: list[GateResult] = [] + + spec_candidates = sorted(root.glob("specs/*.yaml")) + sorted( + root.glob("specs/*.yml") + ) + if not spec_candidates: + results.append( + GateResult( + gate_name="schema.openapi", + verdict=Verdict.WARN, + notes="no specs/*.yaml", + ) + ) + return results + + valid_specs: list[tuple[Path, dict]] = [] + for spec_path in spec_candidates: + try: + spec_dict, _, _ = load_openapi_file(spec_path) + validate_openapi_dict(spec_dict) + except (SchemaValidationError, OSError, ValueError, TypeError) as exc: + results.append( + GateResult( + gate_name=f"schema.openapi.validate.{spec_path.name}", + verdict=Verdict.FAIL, + notes=str(exc), + ) + ) + else: + valid_specs.append((spec_path, spec_dict)) + results.append( + GateResult( + gate_name=f"schema.openapi.validate.{spec_path.name}", + verdict=Verdict.PASS, + notes=str(spec_path.relative_to(root)), + ) + ) + + if not valid_specs: + return results + + if len(valid_specs) > 1: + results.append( + GateResult( + gate_name="schema.openapi.note", + verdict=Verdict.WARN, + notes=( + f"{len(valid_specs)} OpenAPI files under specs/; " + f"coverage uses {valid_specs[0][0].name}" + ), + ) + ) + + spec_path, spec_dict = valid_specs[0] + feature_paths = sorted(root.glob("features/**/*.feature")) + if feature_paths: + gate = SchemaCoverageGate() + gr = gate.run(spec_dict, context={"feature_paths": feature_paths}) + results.append( + GateResult( + gate_name="schema.coverage", + verdict=gr.verdict, + findings=gr.findings, + notes=gr.notes or str(spec_path.relative_to(root)), + ) + ) + return results +``` + +## Callee: pickled_schema.openapi.parser.load_openapi_file (hop 1) + +```python +def load_openapi_file(path: Path) -> tuple[dict[str, Any], SchemaFormat, SchemaArtifact]: + """Load a file and return parsed dict, detected format, and artifact.""" + data, raw = _load_text(path) + fmt = detect_format(data) + artifact = SchemaArtifact( + format=fmt, + content=raw, + endpoint_id=None, + source="file", + ) + return data, fmt, artifact +``` + +## Callee: pickled_schema.openapi.validator.validate_openapi_dict (hop 1) + +```python +def validate_openapi_dict(spec_dict: dict[str, Any]) -> None: + """Validate *spec_dict* with openapi-spec-validator.""" + try: + from openapi_spec_validator import validate + from openapi_spec_validator.exceptions import OpenAPIError + from openapi_spec_validator.validation.exceptions import ( + OpenAPIValidationError, + ) + except ImportError as exc: + msg = "install pickled-schema[openapi] for OpenAPI validation" + raise SchemaValidationError(msg) from exc + + try: + validate(spec_dict) + except (OpenAPIError, OpenAPIValidationError) as exc: + errors = [str(exc)] + nested = getattr(exc, "schema_errors", None) + if nested: + errors.extend(str(e) for e in nested) + raise SchemaValidationError("OpenAPI validation failed", errors=errors) from exc +``` + +## Callee: pickled_schema.openapi.parser._load_text (hop 2) + +```python +def _load_text(path: Path) -> tuple[dict[str, Any], str]: + raw = path.read_text(encoding="utf-8") + suffix = path.suffix.lower() + if suffix in {".yaml", ".yml"}: + data = yaml.safe_load(raw) + elif suffix == ".json": + data = json.loads(raw) + else: + try: + data = yaml.safe_load(raw) + except yaml.YAMLError as exc: + msg = f"cannot parse {path}: not valid YAML or JSON" + raise SchemaParseError(msg) from exc + if not isinstance(data, dict): + msg = "schema root must be a mapping" + raise SchemaParseError(msg) + return data, raw +``` + +## Callee: pickled_schema.openapi.parser.detect_format (hop 2) + +```python +def detect_format(spec_dict: dict[str, Any]) -> SchemaFormat: + """Infer OpenAPI version from a parsed document root.""" + if "swagger" in spec_dict: + msg = "OpenAPI 2.0 (swagger field) is not supported in v0.1" + raise SchemaParseError(msg) + version = spec_dict.get("openapi") + if not isinstance(version, str): + msg = "missing or invalid top-level 'openapi' version field" + raise SchemaParseError(msg) + if version.startswith("3.2"): + return SchemaFormat.openapi_3_2 + if version.startswith("3.1"): + return SchemaFormat.openapi_3_1 + if version.startswith("3.0"): + return SchemaFormat.openapi_3_0 + msg = f"unsupported OpenAPI version {version!r}" + raise SchemaParseError(msg) +``` + +## Unresolved callees + +- `spec_path.relative_to` — method name matches multiple classes; receiver type not pinned +- `gate.run` — method name matches multiple classes; receiver type not pinned +- `getattr(exc, 'schema_errors', None)` — dynamic attribute access diff --git a/dogfood/mining-output/code-context/pickled_schema_schemaambiguitygate.md b/dogfood/mining-output/code-context/pickled_schema_schemaambiguitygate.md new file mode 100644 index 0000000..81777eb --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_schema_schemaambiguitygate.md @@ -0,0 +1,111 @@ +# Code context: SchemaAmbiguityGate.run + +- **Surface id:** pickled_schema_schemaambiguitygate +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 2 | **Total lines:** 88 | **Truncated:** False + +## Root: pickled_schema.gates.SchemaAmbiguityGate.run + +```python +def run( + self, + target: object, + *, + context: dict[str, Any] | None = None, + ) -> GateResult: + ctx = context or {} + if not isinstance(target, SchemaArtifact): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected SchemaArtifact, got {type(target).__name__}", + ) + gherkin = ctx.get("gherkin_context") + if not isinstance(gherkin, str) or not gherkin.strip(): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes='context must contain non-empty "gherkin_context"', + ) + + prompt = self._template.render( + gherkin_context=gherkin, + schema_yaml=target.content, + ) + from pickled_core.llm.turns import complete_prompt + + response = complete_prompt( + self._llm, + prompt, + system=( + "Reply with a single JSON object only. " + "No markdown fences, no commentary outside JSON." + ), + ) + parsed = _parse_ambiguity_response(response) + if parsed is None: + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes="LLM returned malformed JSON", + ) + + ambiguities = parsed.get("ambiguities", []) + if not isinstance(ambiguities, list): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes='LLM JSON missing list field "ambiguities"', + ) + + if not ambiguities: + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="No ambiguities reported.", + ) + + notes = f"{len(ambiguities)} ambiguity(ies) reported." + return GateResult( + gate_name=self.name, + verdict=Verdict.WARN, + findings=tuple(ambiguities), + notes=notes, + ) +``` + +## Callee: pickled_schema.gates._parse_ambiguity_response (hop 1) + +```python +def _parse_ambiguity_response(response: str) -> dict[str, Any] | None: + stripped = response.strip() + if stripped.startswith("```"): + parts = stripped.split("```") + if len(parts) >= 2: + block = parts[1].lstrip() + if block.lower().startswith("json"): + block = block[4:].lstrip() + stripped = block.strip() + try: + data = json.loads(stripped) + except json.JSONDecodeError: + start = stripped.find("{") + end = stripped.rfind("}") + if start == -1 or end <= start: + return None + try: + data = json.loads(stripped[start : end + 1]) + except json.JSONDecodeError: + return None + if not isinstance(data, dict): + return None + return data +``` + +## Unresolved callees + +- `self._template.render` — protocol or unknown attribute type +- `parts[1].lstrip()` — receiver is a subscript expression +- `block[4:].lstrip()` — receiver is a subscript expression +- `stripped.find` — variable 'stripped' reassigned; type not stable +- `stripped.rfind` — variable 'stripped' reassigned; type not stable diff --git a/dogfood/mining-output/code-context/pickled_schema_schemacoveragegate.md b/dogfood/mining-output/code-context/pickled_schema_schemacoveragegate.md new file mode 100644 index 0000000..ce7a3a7 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_schema_schemacoveragegate.md @@ -0,0 +1,102 @@ +# Code context: SchemaCoverageGate.run + +- **Surface id:** pickled_schema_schemacoveragegate +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 3 | **Total lines:** 77 | **Truncated:** False + +## Root: pickled_schema.gates.SchemaCoverageGate.run + +```python +def run( + self, + target: object, + *, + context: dict[str, Any] | None = None, + ) -> GateResult: + ctx = context or {} + if not isinstance(target, dict): + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes=f"Expected parsed OpenAPI dict, got {type(target).__name__}", + ) + + feature_paths: list[Path] = [] + raw_paths = ctx.get("feature_paths") + if isinstance(raw_paths, list): + feature_paths = [Path(p) for p in raw_paths] + + feature_texts: list[tuple[str, str]] = [] + raw_texts = ctx.get("feature_texts") + if isinstance(raw_texts, list): + for i, text in enumerate(raw_texts): + if isinstance(text, str): + feature_texts.append((f"", text)) + + if not feature_paths and not feature_texts: + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + notes='context needs "feature_paths" and/or "feature_texts"', + ) + + missing: list[SchemaCoverageFinding] = [] + for path in feature_paths: + text = path.read_text(encoding="utf-8") + missing.extend( + _missing_tags(target, text, source=str(path)), + ) + for source, text in feature_texts: + missing.extend(_missing_tags(target, text, source=source)) + + if not missing: + return GateResult( + gate_name=self.name, + verdict=Verdict.PASS, + notes="All @schema:endpoint tags have matching paths.", + ) + + lines = [f"{f.tag} ({f.source})" for f in missing] + return GateResult( + gate_name=self.name, + verdict=Verdict.FAIL, + findings=tuple(missing), + notes="Missing endpoints: " + "; ".join(lines), + ) +``` + +## Callee: pickled_schema.gates._missing_tags (hop 1) + +```python +def _missing_tags( + spec: dict[str, Any], + feature_text: str, + *, + source: str, +) -> list[SchemaCoverageFinding]: + out: list[SchemaCoverageFinding] = [] + for match in _ENDPOINT_TAG_RE.finditer(feature_text): + method, path = match.group(1), match.group(2) + tag = match.group(0) + if not _spec_has_endpoint(spec, method, path): + out.append(SchemaCoverageFinding(tag=tag, source=source)) + return out +``` + +## Callee: pickled_schema.gates._spec_has_endpoint (hop 2) + +```python +def _spec_has_endpoint(spec: dict[str, Any], method: str, path: str) -> bool: + paths = spec.get("paths") + if not isinstance(paths, dict) or path not in paths: + return False + item = paths[path] + if not isinstance(item, dict): + return False + return method.lower() in item +``` + +## Unresolved callees + +- `_ENDPOINT_TAG_RE.finditer` — method name matches multiple classes; receiver type not pinned +- `match.group` — method name matches multiple classes; receiver type not pinned diff --git a/dogfood/mining-output/code-context/pickled_schema_validate.md b/dogfood/mining-output/code-context/pickled_schema_validate.md new file mode 100644 index 0000000..8de5bd5 --- /dev/null +++ b/dogfood/mining-output/code-context/pickled_schema_validate.md @@ -0,0 +1,86 @@ +# Code context: validate + +- **Surface id:** pickled_schema_validate +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 4 | **Total lines:** 57 | **Truncated:** True + +## Root: pickled_schema.cli.validate + +```python +def validate(file: Path) -> None: + """Validate a schema file against its format specification.""" + fmt = _infer_format(file) + if fmt in ( + SchemaFormat.openapi_3_0, + SchemaFormat.openapi_3_1, + SchemaFormat.openapi_3_2, + ): + spec_dict, _, _ = load_openapi_file(file) + validate_openapi_dict(spec_dict) + elif fmt is SchemaFormat.json_schema_2020_12: + schema_dict, _ = load_json_schema_file(file) + validate_json_schema_document(schema_dict) + elif fmt is SchemaFormat.proto3: + parse_proto_file(file) + click.echo(json.dumps({"valid": True, "format": fmt.value})) +``` + +## Callee: pickled_schema.cli._infer_format (hop 1) + +```python +def _infer_format(path: Path) -> SchemaFormat: + suffix = path.suffix.lower() + if suffix in {".yaml", ".yml"}: + return SchemaFormat.openapi_3_1 + if suffix == ".json": + return SchemaFormat.json_schema_2020_12 + if suffix == ".proto": + return SchemaFormat.proto3 + msg = f"cannot infer format from extension {suffix!r}; use --format" + raise click.ClickException(msg) +``` + +## Callee: pickled_schema.openapi.parser.load_openapi_file (hop 1) + +```python +def load_openapi_file(path: Path) -> tuple[dict[str, Any], SchemaFormat, SchemaArtifact]: + """Load a file and return parsed dict, detected format, and artifact.""" + data, raw = _load_text(path) + fmt = detect_format(data) + artifact = SchemaArtifact( + format=fmt, + content=raw, + endpoint_id=None, + source="file", + ) + return data, fmt, artifact +``` + +## Callee: pickled_schema.openapi.validator.validate_openapi_dict (hop 1) + +```python +def validate_openapi_dict(spec_dict: dict[str, Any]) -> None: + """Validate *spec_dict* with openapi-spec-validator.""" + try: + from openapi_spec_validator import validate + from openapi_spec_validator.exceptions import OpenAPIError + from openapi_spec_validator.validation.exceptions import ( + OpenAPIValidationError, + ) + except ImportError as exc: + msg = "install pickled-schema[openapi] for OpenAPI validation" + raise SchemaValidationError(msg) from exc + + try: + validate(spec_dict) + except (OpenAPIError, OpenAPIValidationError) as exc: + errors = [str(exc)] + nested = getattr(exc, "schema_errors", None) + if nested: + errors.extend(str(e) for e in nested) + raise SchemaValidationError("OpenAPI validation failed", errors=errors) from exc +``` + +## Notes + +Collection stopped early because of --max-callees or --max-code-lines. diff --git a/dogfood/mining-output/code-context/rules_check_ruleset_coverage.md b/dogfood/mining-output/code-context/rules_check_ruleset_coverage.md new file mode 100644 index 0000000..a775d8a --- /dev/null +++ b/dogfood/mining-output/code-context/rules_check_ruleset_coverage.md @@ -0,0 +1,9 @@ +# Code context: rules_check_ruleset_coverage + +- **Surface id:** rules_check_ruleset_coverage +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/rules_draft_ruleset_from_brief.md b/dogfood/mining-output/code-context/rules_draft_ruleset_from_brief.md new file mode 100644 index 0000000..ff82f4a --- /dev/null +++ b/dogfood/mining-output/code-context/rules_draft_ruleset_from_brief.md @@ -0,0 +1,9 @@ +# Code context: rules_draft_ruleset_from_brief + +- **Surface id:** rules_draft_ruleset_from_brief +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/rules_list_rules.md b/dogfood/mining-output/code-context/rules_list_rules.md new file mode 100644 index 0000000..82f8653 --- /dev/null +++ b/dogfood/mining-output/code-context/rules_list_rules.md @@ -0,0 +1,9 @@ +# Code context: rules_list_rules + +- **Surface id:** rules_list_rules +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/schema_check_schema_coverage.md b/dogfood/mining-output/code-context/schema_check_schema_coverage.md new file mode 100644 index 0000000..a41679a --- /dev/null +++ b/dogfood/mining-output/code-context/schema_check_schema_coverage.md @@ -0,0 +1,9 @@ +# Code context: schema_check_schema_coverage + +- **Surface id:** schema_check_schema_coverage +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/schema_draft_openapi_endpoint.md b/dogfood/mining-output/code-context/schema_draft_openapi_endpoint.md new file mode 100644 index 0000000..7f3d8c4 --- /dev/null +++ b/dogfood/mining-output/code-context/schema_draft_openapi_endpoint.md @@ -0,0 +1,9 @@ +# Code context: schema_draft_openapi_endpoint + +- **Surface id:** schema_draft_openapi_endpoint +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/code-context/schema_validate_openapi_spec.md b/dogfood/mining-output/code-context/schema_validate_openapi_spec.md new file mode 100644 index 0000000..73f5f5a --- /dev/null +++ b/dogfood/mining-output/code-context/schema_validate_openapi_spec.md @@ -0,0 +1,9 @@ +# Code context: schema_validate_openapi_spec + +- **Surface id:** schema_validate_openapi_spec +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 0 | **Total lines:** 0 | **Truncated:** False + +## Notes + +No code definition resolved for this surface. diff --git a/dogfood/mining-output/evaluation/ambiguity.json b/dogfood/mining-output/evaluation/ambiguity.json new file mode 100644 index 0000000..98c9b22 --- /dev/null +++ b/dogfood/mining-output/evaluation/ambiguity.json @@ -0,0 +1,509 @@ +{ + "schema_version": "1", + "features": [ + { + "feature": "features/bdd_draft_feature_from_story.feature", + "verdict": "fail", + "finding_count": 8, + "skipped": false, + "notes": "8/8 scenarios flagged ambiguous." + }, + { + "feature": "features/bdd_validate_feature_ambiguity.feature", + "verdict": "fail", + "finding_count": 12, + "skipped": false, + "notes": "12/12 scenarios flagged ambiguous." + }, + { + "feature": "features/data_apply_sql_to_sandbox.feature", + "verdict": "fail", + "finding_count": 13, + "skipped": false, + "notes": "13/13 scenarios flagged ambiguous." + }, + { + "feature": "features/data_check_migration_drift.feature", + "verdict": "fail", + "finding_count": 11, + "skipped": false, + "notes": "11/11 scenarios flagged ambiguous." + }, + { + "feature": "features/data_draft_sql_migration_from_intent.feature", + "verdict": "fail", + "finding_count": 13, + "skipped": false, + "notes": "13/13 scenarios flagged ambiguous." + }, + { + "feature": "features/data_parse_sql_migration.feature", + "verdict": "fail", + "finding_count": 14, + "skipped": false, + "notes": "14/14 scenarios flagged ambiguous." + }, + { + "feature": "features/diff_draft_corpus_from_examples.feature", + "verdict": "fail", + "finding_count": 9, + "skipped": false, + "notes": "9/9 scenarios flagged ambiguous." + }, + { + "feature": "features/diff_verify_against_oracle.feature", + "verdict": "fail", + "finding_count": 9, + "skipped": false, + "notes": "9/9 scenarios flagged ambiguous." + }, + { + "feature": "features/iac_diff_terraform_plans.feature", + "verdict": "fail", + "finding_count": 14, + "skipped": false, + "notes": "14/14 scenarios flagged ambiguous." + }, + { + "feature": "features/iac_draft_terraform_module.feature", + "verdict": "fail", + "finding_count": 15, + "skipped": false, + "notes": "15/15 scenarios flagged ambiguous." + }, + { + "feature": "features/iac_explain_plan_diff.feature", + "verdict": "fail", + "finding_count": 9, + "skipped": false, + "notes": "9/9 scenarios flagged ambiguous." + }, + { + "feature": "features/iac_suggest_security_remediation.feature", + "verdict": "fail", + "finding_count": 7, + "skipped": false, + "notes": "7/7 scenarios flagged ambiguous." + }, + { + "feature": "features/iac_validate_terraform_dir.feature", + "verdict": "fail", + "finding_count": 7, + "skipped": false, + "notes": "7/7 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_bdd_ambiguity.feature", + "verdict": "fail", + "finding_count": 14, + "skipped": false, + "notes": "14/14 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_bdd_ambiguitygate.feature", + "verdict": "warn", + "finding_count": 12, + "skipped": false, + "notes": "12/14 scenarios flagged ambiguous. Parse errors on: ['Quality engineer runs gate against LLM responses with markdown fences', 'Quality engineer runs gate against LLM responses with markdown fences']." + }, + { + "feature": "features/pickled_bdd_check.feature", + "verdict": "fail", + "finding_count": 12, + "skipped": false, + "notes": "12/12 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_bdd_draft.feature", + "verdict": "fail", + "finding_count": 7, + "skipped": false, + "notes": "7/7 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_bdd_mcp.feature", + "verdict": "fail", + "finding_count": 3, + "skipped": false, + "notes": "3/3 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_bdd_run_all.feature", + "verdict": "fail", + "finding_count": 10, + "skipped": false, + "notes": "10/10 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_core_check_all.feature", + "verdict": "fail", + "finding_count": 10, + "skipped": false, + "notes": "10/10 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_core_mine.feature", + "verdict": "fail", + "finding_count": 6, + "skipped": false, + "notes": "6/6 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_core_mine_all.feature", + "verdict": "fail", + "finding_count": 13, + "skipped": false, + "notes": "13/13 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_core_mine_code.feature", + "verdict": "fail", + "finding_count": 14, + "skipped": false, + "notes": "14/14 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_core_mine_evaluate.feature", + "verdict": "fail", + "finding_count": 13, + "skipped": false, + "notes": "13/13 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_core_mine_features.feature", + "verdict": "fail", + "finding_count": 13, + "skipped": false, + "notes": "13/13 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_core_mine_inventory.feature", + "verdict": "fail", + "finding_count": 10, + "skipped": false, + "notes": "10/10 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_core_mine_report.feature", + "verdict": "fail", + "finding_count": 10, + "skipped": false, + "notes": "10/10 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_core_mine_stories.feature", + "verdict": "fail", + "finding_count": 15, + "skipped": false, + "notes": "15/15 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_core_mine_tag.feature", + "verdict": "fail", + "finding_count": 10, + "skipped": false, + "notes": "10/10 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_data_apply.feature", + "verdict": "fail", + "finding_count": 19, + "skipped": false, + "notes": "19/19 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_data_check_drift.feature", + "verdict": "fail", + "finding_count": 7, + "skipped": false, + "notes": "7/7 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_data_datacontractgate.feature", + "verdict": "fail", + "finding_count": 14, + "skipped": false, + "notes": "14/14 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_data_draft.feature", + "verdict": "fail", + "finding_count": 20, + "skipped": false, + "notes": "20/20 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_data_mcp.feature", + "verdict": "fail", + "finding_count": 5, + "skipped": false, + "notes": "5/5 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_data_migrationdriftgate.feature", + "verdict": "fail", + "finding_count": 18, + "skipped": false, + "notes": "18/18 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_data_parse.feature", + "verdict": "fail", + "finding_count": 9, + "skipped": false, + "notes": "9/9 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_data_run_all.feature", + "verdict": "fail", + "finding_count": 18, + "skipped": false, + "notes": "18/18 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_diff_draft_corpus.feature", + "verdict": "fail", + "finding_count": 8, + "skipped": false, + "notes": "8/8 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_diff_mcp.feature", + "verdict": "fail", + "finding_count": 3, + "skipped": false, + "notes": "3/3 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_diff_run_all.feature", + "verdict": "fail", + "finding_count": 17, + "skipped": false, + "notes": "17/17 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_diff_verify.feature", + "verdict": "fail", + "finding_count": 22, + "skipped": false, + "notes": "22/22 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_iac_diff.feature", + "verdict": "fail", + "finding_count": 22, + "skipped": false, + "notes": "22/22 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_iac_draft.feature", + "verdict": "warn", + "finding_count": 15, + "skipped": false, + "notes": "15/16 scenarios flagged ambiguous. Parse errors on: ['LLM response contains code fences which are stripped']." + }, + { + "feature": "features/pickled_iac_iacambiguitygate.feature", + "verdict": "error", + "finding_count": 0, + "skipped": true, + "notes": "unparseable feature: Parser errors:\n(1:1): expected: #EOF, #Language, #TagLine, #FeatureLine, #Comment, #Empty, got '```gherkin'\n(141:0): unexpected end of file, expected: #DocStringSeparator, #Other" + }, + { + "feature": "features/pickled_iac_mcp.feature", + "verdict": "fail", + "finding_count": 6, + "skipped": false, + "notes": "6/6 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_iac_plan_cmd.feature", + "verdict": "fail", + "finding_count": 8, + "skipped": false, + "notes": "8/8 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_iac_plandiffgate.feature", + "verdict": "fail", + "finding_count": 28, + "skipped": false, + "notes": "28/28 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_iac_run_all.feature", + "verdict": "fail", + "finding_count": 14, + "skipped": false, + "notes": "14/14 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_iac_scan.feature", + "verdict": "fail", + "finding_count": 12, + "skipped": false, + "notes": "12/12 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_iac_securitybaselinegate.feature", + "verdict": "fail", + "finding_count": 20, + "skipped": false, + "notes": "20/20 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_iac_validate.feature", + "verdict": "fail", + "finding_count": 6, + "skipped": false, + "notes": "6/6 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_rules_check.feature", + "verdict": "fail", + "finding_count": 25, + "skipped": false, + "notes": "25/25 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_rules_coverage_gate.feature", + "verdict": "fail", + "finding_count": 14, + "skipped": false, + "notes": "14/14 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_rules_coverage_gate_features.feature", + "verdict": "fail", + "finding_count": 12, + "skipped": false, + "notes": "12/12 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_rules_draft.feature", + "verdict": "fail", + "finding_count": 13, + "skipped": false, + "notes": "13/13 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_rules_list_rules.feature", + "verdict": "fail", + "finding_count": 8, + "skipped": false, + "notes": "8/8 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_rules_mcp.feature", + "verdict": "fail", + "finding_count": 3, + "skipped": false, + "notes": "3/3 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_rules_run_all.feature", + "verdict": "fail", + "finding_count": 19, + "skipped": false, + "notes": "19/19 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_schema_check.feature", + "verdict": "fail", + "finding_count": 16, + "skipped": false, + "notes": "16/16 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_schema_draft.feature", + "verdict": "fail", + "finding_count": 19, + "skipped": false, + "notes": "19/19 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_schema_mcp.feature", + "verdict": "fail", + "finding_count": 2, + "skipped": false, + "notes": "2/2 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_schema_parse.feature", + "verdict": "error", + "finding_count": 0, + "skipped": true, + "notes": "unparseable feature: Parser errors:\n(1:1): expected: #EOF, #Language, #TagLine, #FeatureLine, #Comment, #Empty, got '```gherkin'\n(95:1): expected: #EOF, #TableRow, #TagLine, #ExamplesLine, #ScenarioLine, #RuleLine, #Comment, #Empty, got '```'" + }, + { + "feature": "features/pickled_schema_run_all.feature", + "verdict": "fail", + "finding_count": 25, + "skipped": false, + "notes": "25/25 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_schema_schemaambiguitygate.feature", + "verdict": "warn", + "finding_count": 17, + "skipped": false, + "notes": "17/18 scenarios flagged ambiguous. Parse errors on: ['Gate fails when LLM returns malformed markdown fences']." + }, + { + "feature": "features/pickled_schema_schemacoveragegate.feature", + "verdict": "fail", + "finding_count": 23, + "skipped": false, + "notes": "23/23 scenarios flagged ambiguous." + }, + { + "feature": "features/pickled_schema_validate.feature", + "verdict": "fail", + "finding_count": 18, + "skipped": false, + "notes": "18/18 scenarios flagged ambiguous." + }, + { + "feature": "features/rules_check_ruleset_coverage.feature", + "verdict": "fail", + "finding_count": 9, + "skipped": false, + "notes": "9/9 scenarios flagged ambiguous." + }, + { + "feature": "features/rules_draft_ruleset_from_brief.feature", + "verdict": "fail", + "finding_count": 12, + "skipped": false, + "notes": "12/12 scenarios flagged ambiguous." + }, + { + "feature": "features/rules_list_rules.feature", + "verdict": "fail", + "finding_count": 10, + "skipped": false, + "notes": "10/10 scenarios flagged ambiguous." + }, + { + "feature": "features/schema_check_schema_coverage.feature", + "verdict": "fail", + "finding_count": 7, + "skipped": false, + "notes": "7/7 scenarios flagged ambiguous." + }, + { + "feature": "features/schema_draft_openapi_endpoint.feature", + "verdict": "fail", + "finding_count": 12, + "skipped": false, + "notes": "12/12 scenarios flagged ambiguous." + }, + { + "feature": "features/schema_validate_openapi_spec.feature", + "verdict": "fail", + "finding_count": 5, + "skipped": false, + "notes": "5/5 scenarios flagged ambiguous." + } + ] +} diff --git a/dogfood/mining-output/evaluation/coverage.json b/dogfood/mining-output/evaluation/coverage.json new file mode 100644 index 0000000..9131aaa --- /dev/null +++ b/dogfood/mining-output/evaluation/coverage.json @@ -0,0 +1,118 @@ +{ + "schema_version": "1", + "rulesets": [ + { + "short_name": "pickled-internal", + "verdict": "pass", + "notes": "All strict rules in PICKLED-INTERNAL are referenced; no unknown reference tags.", + "referenced_rule_ids": [ + "core-llm-cache-default-on", + "core-model-from-config-not-hardcoded", + "mcp-output-fixed-json-shape", + "mcp-subserver-llm-client-wired", + "stdio-hygiene-gates-log-stderr" + ], + "unreferenced_strict_rule_ids": [], + "unknown_references": [] + }, + { + "short_name": "best-practices", + "verdict": "pass", + "notes": "All strict rules in PICKLED-BEST-PRACTICES are referenced; no unknown reference tags.", + "referenced_rule_ids": [ + "agent-path-first-class", + "cli-mcp-surface-parity", + "llm-drafter-temperature-zero" + ], + "unreferenced_strict_rule_ids": [], + "unknown_references": [] + }, + { + "short_name": "oss-hygiene", + "verdict": "pass", + "notes": "All strict rules in PICKLED-OSS-HYGIENE are referenced; no unknown reference tags.", + "referenced_rule_ids": [ + "no-secrets-in-repo" + ], + "unreferenced_strict_rule_ids": [], + "unknown_references": [] + }, + { + "short_name": "bdd-domain", + "verdict": "fail", + "notes": "1 strict rule(s) unreferenced.", + "referenced_rule_ids": [ + "draft-empty-story-deterministic-failure", + "draft-output-parses-via-pytest-bdd", + "draft-warnings-field-populated-on-failure", + "drafter-no-auto-tags", + "gherkin-feature-header-required" + ], + "unreferenced_strict_rule_ids": [ + "gherkin-then-asserts-observable-outcome" + ], + "unknown_references": [] + }, + { + "short_name": "rules-domain", + "verdict": "pass", + "notes": "All strict rules in PICKLED-RULES-DOMAIN are referenced; no unknown reference tags.", + "referenced_rule_ids": [ + "coverage-union-across-features", + "unknown-tag-fails-gate" + ], + "unreferenced_strict_rule_ids": [], + "unknown_references": [] + }, + { + "short_name": "schema-domain", + "verdict": "pass", + "notes": "All strict rules in PICKLED-SCHEMA-DOMAIN are referenced; no unknown reference tags.", + "referenced_rule_ids": [ + "openapi-validate-deterministic" + ], + "unreferenced_strict_rule_ids": [], + "unknown_references": [] + }, + { + "short_name": "iac-domain", + "verdict": "pass", + "notes": "All strict rules in PICKLED-IAC-DOMAIN are referenced; no unknown reference tags.", + "referenced_rule_ids": [ + "terraform-validate-entry" + ], + "unreferenced_strict_rule_ids": [], + "unknown_references": [] + }, + { + "short_name": "data-domain", + "verdict": "pass", + "notes": "All strict rules in PICKLED-DATA-DOMAIN are referenced; no unknown reference tags.", + "referenced_rule_ids": [ + "migration-drift-gate" + ], + "unreferenced_strict_rule_ids": [], + "unknown_references": [] + }, + { + "short_name": "diff-domain", + "verdict": "pass", + "notes": "All strict rules in PICKLED-DIFF-DOMAIN are referenced; no unknown reference tags.", + "referenced_rule_ids": [ + "differential-oracle-gate" + ], + "unreferenced_strict_rule_ids": [], + "unknown_references": [] + }, + { + "short_name": "core-domain", + "verdict": "pass", + "notes": "All strict rules in PICKLED-CORE-DOMAIN are referenced; no unknown reference tags.", + "referenced_rule_ids": [ + "verdict-three-state-ladder" + ], + "unreferenced_strict_rule_ids": [], + "unknown_references": [] + } + ] +} diff --git a/dogfood/mining-output/features/bdd_draft_feature_from_story.feature b/dogfood/mining-output/features/bdd_draft_feature_from_story.feature new file mode 100644 index 0000000..87f3e2b --- /dev/null +++ b/dogfood/mining-output/features/bdd_draft_feature_from_story.feature @@ -0,0 +1,67 @@ +Feature: BDD Draft Feature from Story Tool + + As a BDD practitioner + I want to generate a first-draft Gherkin feature file from a natural-language user story + So that I can accelerate the transition from informal requirements to structured specifications + + Background: + Given the bdd_draft_feature_from_story tool is available + + @bdd-domain:gherkin-feature-header-required + Scenario: Practitioner generates feature from simple user story + Given a simple user story text + When the practitioner invokes the tool with the story text + Then a response containing valid Gherkin keywords is returned + And the response includes a Feature declaration + And the response includes at least one Scenario + And the response includes Given, When, and Then steps + + @bdd-domain:gherkin-feature-header-required + Scenario: Practitioner saves generated feature to file + Given a user story text + And the practitioner has invoked the tool with the story text + When the generated output is written to a .feature file + Then the file is created without syntax errors + And the file contains valid Gherkin structure + + @bdd-domain:draft-empty-story-deterministic-failure + Scenario: Practitioner generates feature with empty story text + Given an empty story text + When the practitioner invokes the tool with the story text + Then either an error is returned or a minimal valid feature template is produced + + @bdd-domain:gherkin-feature-header-required + Scenario: Practitioner generates feature with whitespace-only story text + Given a story text containing only whitespace + When the practitioner invokes the tool with the story text + Then either an error is returned or a minimal valid feature template is produced + + # TODO: Verify exact instrumentation approach for observing gate invocation + @bdd-domain:gherkin-feature-header-required + Scenario: Tool invokes ambiguity gate during processing + Given a user story text + When the practitioner invokes the tool with the story text + Then the AmbiguityGate.run method is invoked before feature generation + And gate execution is observable via instrumentation or logs + + @bdd-domain:draft-empty-story-deterministic-failure + Scenario: Gate failure prevents feature generation + Given a user story text that triggers gate failure + When the practitioner invokes the tool with the story text + Then feature generation is prevented + And an appropriate error message is returned + And no feature content is produced + + @bdd-domain:gherkin-feature-header-required + Scenario: Generated feature incorporates input story elements + Given a user story text with specific requirements and acceptance criteria + When the practitioner invokes the tool with the story text + Then the generated feature references elements from the input story + And the feature content is derived from the story requirements + + @bdd-domain:gherkin-feature-header-required + Scenario: Multiple invocations produce consistent output + Given a user story text + When the practitioner invokes the tool multiple times with the same story text + Then each invocation produces output with consistent structure + And the Feature and Scenario organization remains stable across invocations diff --git a/dogfood/mining-output/features/bdd_validate_feature_ambiguity.feature b/dogfood/mining-output/features/bdd_validate_feature_ambiguity.feature new file mode 100644 index 0000000..d04d2e4 --- /dev/null +++ b/dogfood/mining-output/features/bdd_validate_feature_ambiguity.feature @@ -0,0 +1,66 @@ +Feature: BDD Practitioner validates feature file for ambiguous step definitions + + As a BDD practitioner + I want to validate Gherkin feature files for ambiguous step definitions + So that I can catch ambiguity issues early before running tests + + Background: + Given the ambiguity gate is available as an MCP tool + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: BDD practitioner validates a feature file with no ambiguities + Given a valid Gherkin feature file with unambiguous steps + When the practitioner validates the feature text for ambiguity + Then the validation returns success + And no ambiguities are detected + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: BDD practitioner detects ambiguous step definitions + Given a valid Gherkin feature file with ambiguous step definitions + When the practitioner validates the feature text for ambiguity + Then the validation returns failure + And the ambiguities are reported + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: BDD practitioner validates malformed Gherkin syntax + Given a feature file with invalid Gherkin syntax + When the practitioner validates the feature text for ambiguity + Then the validation handles the syntax error appropriately + And an appropriate error message is returned + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: BDD practitioner validates empty feature text + Given an empty feature text parameter + When the practitioner validates the feature text for ambiguity + Then the validation handles the empty input appropriately + And an appropriate error message is returned + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: BDD practitioner uses ambiguity gate independently + Given other validation gates are available in the system + When the practitioner validates the feature text for ambiguity only + Then only the ambiguity gate is invoked + And the validation completes without requiring other gates + + @bdd-domain:draft-warnings-field-populated-on-failure + Scenario: Test automation engineer integrates ambiguity gate in CI/CD pipeline + Given a CI/CD pipeline with automated quality gates + When the ambiguity gate is invoked as an MCP tool + Then the gate executes within the pipeline context + And the validation results are returned for pipeline decision-making + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario Outline: BDD practitioner validates various Gherkin structures + Given a feature file with + When the practitioner validates the feature text for ambiguity + Then the validation processes the structure correctly + And ambiguity detection results are returned for + + Examples: + | structure_type | + | scenario outlines | + | data tables | + | doc strings | + | tagged scenarios | + | multiple scenarios | + | background steps | diff --git a/dogfood/mining-output/features/data_apply_sql_to_sandbox.feature b/dogfood/mining-output/features/data_apply_sql_to_sandbox.feature new file mode 100644 index 0000000..ad9b02c --- /dev/null +++ b/dogfood/mining-output/features/data_apply_sql_to_sandbox.feature @@ -0,0 +1,56 @@ +Feature: Apply SQL to Sandbox + As a data engineer + I want to execute SQL statements against an in-memory SQLite database + So that I can validate queries and explore schema changes without affecting persistent data + + Scenario: User executes valid CREATE TABLE statement + When the user applies SQL to create a single table + Then the schema information includes the created table name + And the schema information includes column definitions for the table + + Scenario: User executes multiple SQL statements + When the user applies SQL to create a table and then alter it + Then the schema information reflects the final database state after all statements + + Scenario: User executes invalid SQL + When the user applies SQL with syntax errors + Then an error response is returned + And no schema output is provided + + @bdd-domain:draft-empty-story-deterministic-failure + Scenario: User executes SQL that creates no tables + When the user applies SQL that does not create any tables + Then an empty or minimal schema representation is returned + + Scenario: User executes DROP TABLE statement + When the user applies SQL to create a table and then drop it + Then the dropped table is not present in the returned schema + + @best-practices:agent-path-first-class + Scenario: User invokes the tool multiple times + When the user applies SQL to create a table in the first invocation + And the user applies different SQL in a second invocation + Then the second invocation's schema does not include tables from the first invocation + + Scenario Outline: User provides optional dialect parameter + When the user applies SQL with dialect set to "" + Then the SQL is executed without error + And schema information is returned + + Examples: + | dialect | + | sqlite | + | postgres | + | mysql | + + @oss-hygiene:no-secrets-in-repo + Scenario Outline: Schema output format consistency + When the user applies SQL that "" + Then the returned schema format is consistent and parseable + + Examples: + | sql_description | + | creates a simple table | + | creates multiple tables | + | creates a table with complex types | + | creates tables with foreign keys | diff --git a/dogfood/mining-output/features/data_check_migration_drift.feature b/dogfood/mining-output/features/data_check_migration_drift.feature new file mode 100644 index 0000000..09caf64 --- /dev/null +++ b/dogfood/mining-output/features/data_check_migration_drift.feature @@ -0,0 +1,77 @@ +Feature: Data Check Migration Drift + As a development team member + I want to validate that SQL migration scripts produce the expected database schema + So that I can catch schema drift before deploying migrations to production + + Background: + Given a migration drift validation tool is available + + @data-domain:migration-drift-gate + Scenario: Developer validates migration script matches expected schema + Given a SQL migration script that creates tables with specific columns + And an expected schema definition in YAML format that matches the migration + When the migration drift check is performed + Then the result indicates the migration matches the expected schema + And no drift is reported + + @data-domain:migration-drift-gate + Scenario: Developer detects schema drift when migration differs from expected schema + Given a SQL migration script that creates tables with specific columns + And an expected schema definition in YAML format that differs from the migration + When the migration drift check is performed + Then the result indicates drift between the migration and expected schema + And the drift details are included in the result + + @data-domain:migration-drift-gate + Scenario: Developer validates migration with specific SQL dialect + Given a SQL migration script written in PostgreSQL dialect + And an expected schema definition in YAML format + And the SQL dialect is specified as "postgresql" + When the migration drift check is performed with the specified dialect + Then the migration is parsed and executed using PostgreSQL dialect rules + And the result indicates whether the migration matches the expected schema + + @data-domain:migration-drift-gate + Scenario Outline: Developer validates migrations across different SQL dialects + Given a SQL migration script written in syntax + And an expected schema definition in YAML format + And the SQL dialect is specified as "" + When the migration drift check is performed with the specified dialect + Then the migration is processed according to rules + And the result indicates whether the migration matches the expected schema + + Examples: + | dialect | + | postgresql | + | mysql | + | sqlite | + | sqlserver | + + @data-domain:migration-drift-gate + Scenario: Developer validates migration without specifying dialect + Given a SQL migration script in standard SQL syntax + And an expected schema definition in YAML format + When the migration drift check is performed without specifying a dialect + Then a default SQL dialect is used for validation + And the result indicates whether the migration matches the expected schema + + @data-domain:migration-drift-gate + Scenario: Developer receives actionable results for CI/CD pipeline integration + Given a SQL migration script + And an expected schema definition in YAML format + When the migration drift check is performed + Then the result format is suitable for programmatic consumption + And the result clearly indicates migration validity status + And the result can be used to pass or fail a CI/CD pipeline step + + @data-domain:migration-drift-gate + Scenario: Validation fails when required SQL parameter is missing + Given an expected schema definition in YAML format + When the migration drift check is performed without providing SQL + Then the validation fails with an error indicating SQL is required + + @data-domain:migration-drift-gate + Scenario: Validation fails when required expected schema parameter is missing + Given a SQL migration script + When the migration drift check is performed without providing expected schema YAML + Then the validation fails with an error indicating expected schema is required diff --git a/dogfood/mining-output/features/data_draft_sql_migration_from_intent.feature b/dogfood/mining-output/features/data_draft_sql_migration_from_intent.feature new file mode 100644 index 0000000..c34f2f7 --- /dev/null +++ b/dogfood/mining-output/features/data_draft_sql_migration_from_intent.feature @@ -0,0 +1,109 @@ +Feature: Data Draft SQL Migration from Intent Tool + + As a developer or automated workflow + I want to generate SQL migration scripts from natural language descriptions + So that I can evolve database schemas without writing raw DDL + + Background: + Given the data_draft_sql_migration_from_intent tool is available + + @data-domain:migration-drift-gate + Scenario: Developer generates migration with intent and dialect only + Given the developer has an intent description "Add email column to users table" + And the target dialect is "postgresql" + When the developer invokes the tool with intent_text and dialect + Then the tool returns valid SQL migration statements + And the SQL is syntactically valid for postgresql + + @data-domain:migration-drift-gate + Scenario: Developer generates migration with current schema provided + Given the developer has an intent description "Add email column to users table" + And the target dialect is "postgresql" + And the current schema is provided in YAML format + """ + tables: + users: + columns: + - id: integer + - name: varchar(255) + """ + When the developer invokes the tool with intent_text, dialect, and current_schema_yaml + Then the tool returns valid SQL migration statements + And the migration reflects the transition from current schema to intended state + And the SQL is syntactically valid for postgresql + + @data-domain:migration-drift-gate + Scenario Outline: Tool generates dialect-specific SQL syntax + Given the developer has an intent description "Create orders table with id and amount" + And the target dialect is "" + When the developer invokes the tool with intent_text and dialect + Then the tool returns valid SQL migration statements + And the SQL is syntactically valid for + + Examples: + | dialect | + | postgresql | + | mysql | + | sqlite | + + # TODO: Clarify expected behavior when current_schema_yaml is omitted - verify baseline assumptions from implementation + @data-domain:migration-drift-gate + Scenario: Developer omits current schema parameter + Given the developer has an intent description "Add email column to users table" + And the target dialect is "postgresql" + When the developer invokes the tool without current_schema_yaml + Then the tool returns valid SQL migration statements + + @bdd-domain:draft-warnings-field-populated-on-failure + Scenario: Tool execution validates against DataContractGate + Given the developer has an intent description "Add email column to users table" + And the target dialect is "postgresql" + When the developer invokes the tool with intent_text and dialect + Then DataContractGate.run validation is triggered or respected + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Tool execution validates against MigrationDriftGate + Given the developer has an intent description "Add email column to users table" + And the target dialect is "postgresql" + And the current schema is provided in YAML format + When the developer invokes the tool with intent_text, dialect, and current_schema_yaml + Then MigrationDriftGate.run checks are triggered or respected + + @data-domain:migration-drift-gate + Scenario: Developer provides required intent_text parameter + Given the target dialect is "postgresql" + When the developer invokes the tool with intent_text "Create products table" + Then the tool accepts the intent_text parameter + And the tool returns valid SQL migration statements + + @data-domain:migration-drift-gate + Scenario: Developer provides required dialect parameter + Given the developer has an intent description "Add email column to users table" + When the developer invokes the tool with dialect "postgresql" + Then the tool accepts the dialect parameter + And the tool returns valid SQL migration statements + + @data-domain:migration-drift-gate + Scenario: Developer provides optional current_schema_yaml parameter + Given the developer has an intent description "Add email column to users table" + And the target dialect is "postgresql" + And the current schema is provided in YAML format + When the developer invokes the tool with current_schema_yaml + Then the tool accepts the current_schema_yaml parameter + And the migration generation considers the existing schema + + # TODO: Verify from implementation whether ambiguous intent causes error or best-effort generation + @bdd-domain:gherkin-feature-header-required + Scenario: Tool handles ambiguous intent description + Given the developer has an ambiguous intent description "Change the user thing" + And the target dialect is "postgresql" + When the developer invokes the tool with intent_text and dialect + Then the tool responds with error or best-effort SQL generation + + # TODO: Verify from implementation whether conflicting intent causes error or resolution strategy + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Tool handles conflicting intent description + Given the developer has a conflicting intent description "Add email column and remove email column from users" + And the target dialect is "postgresql" + When the developer invokes the tool with intent_text and dialect + Then the tool responds with error or resolved SQL generation diff --git a/dogfood/mining-output/features/data_parse_sql_migration.feature b/dogfood/mining-output/features/data_parse_sql_migration.feature new file mode 100644 index 0000000..7707f1e --- /dev/null +++ b/dogfood/mining-output/features/data_parse_sql_migration.feature @@ -0,0 +1,80 @@ +Feature: Parse SQL migration files into AST summary + As a data engineer or migration author + I want to parse SQL migration files into structured AST representations + So that I can programmatically analyze schema changes for drift detection + + @data-domain:migration-drift-gate + Scenario: Data engineer parses valid CREATE TABLE statement + Given a SQL migration containing a CREATE TABLE statement + When the data engineer parses the SQL + Then the tool returns an AST summary structure + And the AST summary represents the CREATE TABLE command + + @data-domain:migration-drift-gate + Scenario: Data engineer parses valid ALTER TABLE statement + Given a SQL migration containing an ALTER TABLE statement + When the data engineer parses the SQL + Then the tool returns an AST summary structure + And the AST summary represents the ALTER TABLE command + + @data-domain:migration-drift-gate + Scenario: Data engineer parses SQL with specific dialect + Given a SQL migration written in PostgreSQL dialect + When the data engineer parses the SQL with dialect "postgresql" + Then the tool returns an AST summary structure + And the SQL is parsed according to PostgreSQL syntax rules + + @data-domain:migration-drift-gate + Scenario: Data engineer parses SQL without specifying dialect + Given a SQL migration containing standard SQL syntax + When the data engineer parses the SQL without specifying a dialect + Then the tool returns an AST summary structure + And the SQL is parsed using default dialect rules + + @data-domain:migration-drift-gate + Scenario: Drift detection gate consumes AST output + Given a SQL migration that has been parsed + When the AST summary is provided to a drift detection gate + Then the drift detection gate can analyze the structured output + + @bdd-domain:draft-empty-story-deterministic-failure + Scenario: Data engineer parses empty SQL input + Given an empty SQL string + When the data engineer parses the SQL + Then the tool handles the empty input gracefully + And the tool returns an appropriate response for empty input + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Data engineer parses whitespace-only SQL input + Given a SQL string containing only whitespace + When the data engineer parses the SQL + Then the tool handles the whitespace-only input gracefully + And the tool returns an appropriate response for whitespace input + + @data-domain:migration-drift-gate + Scenario: Data engineer attempts to parse malformed SQL + Given a SQL migration with syntax errors + When the data engineer parses the SQL + Then the tool reports a parse error + And the error indicates the malformed syntax + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: Data engineer parses SQL without required parameter + Given no SQL text is provided + When the data engineer attempts to parse + Then the tool reports that the sql parameter is required + + @data-domain:migration-drift-gate + Scenario Outline: Data engineer parses various SQL migration statements + Given a SQL migration containing a statement + When the data engineer parses the SQL + Then the tool returns an AST summary structure + And the AST summary represents the command + + Examples: + | statement_type | + | CREATE TABLE | + | ALTER TABLE | + | DROP TABLE | + | CREATE INDEX | + | DROP INDEX | diff --git a/dogfood/mining-output/features/diff_draft_corpus_from_examples.feature b/dogfood/mining-output/features/diff_draft_corpus_from_examples.feature new file mode 100644 index 0000000..b0cedcc --- /dev/null +++ b/dogfood/mining-output/features/diff_draft_corpus_from_examples.feature @@ -0,0 +1,56 @@ +Feature: Draft corpus generation from seed examples for differential testing + As a developer working with pickled-diff + I want to generate a draft corpus from a small set of seed examples + So that I can create larger test datasets for differential testing + + Background: + Given the diff_draft_corpus_from_examples tool is available + + Scenario: Developer generates draft corpus with minimum required parameters + Given I have a set of seed examples + And I specify a target size for the corpus + When I invoke diff_draft_corpus_from_examples with seed_examples and target_size + Then a draft corpus is produced + And the corpus contains entries expanded from the seed examples + And the corpus size matches the specified target size + + Scenario: Developer generates draft corpus with documentation notes + Given I have a set of seed examples + And I specify a target size for the corpus + And I provide notes documenting the corpus generation context + When I invoke diff_draft_corpus_from_examples with seed_examples, target_size, and notes + Then a draft corpus is produced + And the corpus includes the provided notes + And the corpus size matches the specified target size + + @diff-domain:differential-oracle-gate + Scenario: Draft corpus output is compatible with run_all gate + Given I have generated a draft corpus using diff_draft_corpus_from_examples + When I provide the corpus result to the run_all gate + Then the run_all gate can consume the corpus + And the run_all gate can execute differential testing with the corpus + + # TODO: Verify exact format of draft corpus output from source + # TODO: Verify exact expansion mechanism used to reach target size from seed examples + Scenario Outline: Developer generates corpora with various target sizes + Given I have seed examples + When I invoke diff_draft_corpus_from_examples with target_size of + Then a draft corpus is produced + And the corpus contains entries + + Examples: + | seed_count | target_size | + | 5 | 10 | + | 5 | 50 | + | 10 | 100 | + | 3 | 20 | + + Scenario: Tool fails when seed_examples parameter is missing + Given I specify a target size for the corpus + When I invoke diff_draft_corpus_from_examples without seed_examples + Then the tool returns an error indicating seed_examples is required + + Scenario: Tool fails when target_size parameter is missing + Given I have a set of seed examples + When I invoke diff_draft_corpus_from_examples without target_size + Then the tool returns an error indicating target_size is required diff --git a/dogfood/mining-output/features/diff_verify_against_oracle.feature b/dogfood/mining-output/features/diff_verify_against_oracle.feature new file mode 100644 index 0000000..b4ba61c --- /dev/null +++ b/dogfood/mining-output/features/diff_verify_against_oracle.feature @@ -0,0 +1,80 @@ +Feature: Differential verification against oracle implementation + As a tester or developer + I want to verify a candidate implementation against a reference oracle + So that I can ensure behavioral compatibility across a test corpus + + Background: + Given the diff_verify_against_oracle tool is available + + @diff-domain:differential-oracle-gate + Scenario: Developer verifies candidate matches oracle for all corpus items + Given an oracle command "reference-tool --process" + And a candidate command "new-tool --process" + And a corpus with items "input1.txt, input2.txt, input3.txt" + When the verification is executed + Then both commands are run against each corpus item + And the outputs are compared for differences + And a verification report is returned + + @diff-domain:differential-oracle-gate + Scenario: Developer detects differences between candidate and oracle + Given an oracle command "reference-tool --process" + And a candidate command "buggy-tool --process" + And a corpus with items "test1.txt, test2.txt" + And the candidate produces different output for "test2.txt" + When the verification is executed + Then differences are reported for "test2.txt" + And the oracle output is included in the report + And the candidate output is included in the report + + @diff-domain:differential-oracle-gate + Scenario: Developer customizes difference detection with comparator + Given an oracle command "reference-tool --format" + And a candidate command "new-tool --format" + And a corpus with items "data1.json, data2.json" + And a custom comparator "json-semantic-compare" + When the verification is executed + Then the custom comparator is used to detect differences + And differences are reported according to the comparator logic + + @diff-domain:differential-oracle-gate + Scenario: Developer limits execution time with timeout + Given an oracle command "slow-reference --compute" + And a candidate command "fast-candidate --compute" + And a corpus with items "large-input.dat" + And a timeout of 30 seconds per command + When the verification is executed + Then each command execution is limited to 30 seconds + And timeout violations are reported if they occur + + @diff-domain:differential-oracle-gate + Scenario Outline: Tool validates required parameters + Given is not provided + When the verification is attempted + Then the tool returns a parameter validation error + And the error indicates is required + + Examples: + | missing_parameter | + | oracle_command | + | candidate_command | + | corpus_items | + + @diff-domain:differential-oracle-gate + Scenario: Developer runs verification with minimal required parameters + Given an oracle command "baseline-tool" + And a candidate command "comparison-tool" + And a corpus with items "test.dat" + When the verification is executed without optional parameters + Then both commands are run against each corpus item + And default comparison logic is applied + And a verification report is returned + + @diff-domain:differential-oracle-gate + Scenario: Developer verifies with empty corpus + Given an oracle command "oracle-cmd" + And a candidate command "candidate-cmd" + And a corpus with no items + When the verification is executed + Then no command executions occur + And the report indicates zero items were tested diff --git a/dogfood/mining-output/features/iac_diff_terraform_plans.feature b/dogfood/mining-output/features/iac_diff_terraform_plans.feature new file mode 100644 index 0000000..a36dfb4 --- /dev/null +++ b/dogfood/mining-output/features/iac_diff_terraform_plans.feature @@ -0,0 +1,77 @@ +Feature: IaC Diff Terraform Plans Tool + + As an automation workflow or CI/CD pipeline + I want to analyze infrastructure changes between two Terraform plan states + So that I can validate changes before applying them to environments + + Background: + Given the iac_diff_terraform_plans MCP tool is available + + @iac-domain:terraform-validate-entry + Scenario: Automation workflow compares valid base and head Terraform plans + Given a valid Terraform plan JSON for the base state + And a valid Terraform plan JSON for the head state + When the workflow calls iac_diff_terraform_plans with base_plan_json and head_plan_json + Then a comparison result structure is returned + And the result shows resources to be added + And the result shows resources to be modified + And the result shows resources to be destroyed + + @iac-domain:terraform-validate-entry + Scenario: Automation workflow provides base plan as current state and head plan as proposed changes + Given base_plan_json represents the current infrastructure state + And head_plan_json represents proposed infrastructure changes + When the workflow calls iac_diff_terraform_plans with base_plan_json and head_plan_json + Then the comparison result reflects differences between current and proposed states + + # TODO: Define valid Terraform plan JSON format specification + @iac-domain:terraform-validate-entry + Scenario Outline: Tool rejects invalid Terraform plan JSON formats + Given is not valid Terraform plan JSON format + When the workflow calls iac_diff_terraform_plans with the invalid input as + Then an error is returned indicating invalid JSON format for + + Examples: + | invalid_input | argument | + | malformed JSON | base_plan_json | + | empty string | base_plan_json | + | non-JSON text | base_plan_json | + | malformed JSON | head_plan_json | + | empty string | head_plan_json | + | non-JSON text | head_plan_json | + + @iac-domain:terraform-validate-entry + Scenario: Tool is called without required base_plan_json argument + Given head_plan_json is provided + When the workflow calls iac_diff_terraform_plans without base_plan_json + Then an error is returned indicating base_plan_json is required + + @iac-domain:terraform-validate-entry + Scenario: Tool is called without required head_plan_json argument + Given base_plan_json is provided + When the workflow calls iac_diff_terraform_plans without head_plan_json + Then an error is returned indicating head_plan_json is required + + @iac-domain:terraform-validate-entry + Scenario: IaCAmbiguityGate consumes comparison result output + Given iac_diff_terraform_plans has produced a comparison result + When IaCAmbiguityGate.run processes the comparison result + Then the gate successfully consumes the output structure + + @iac-domain:terraform-validate-entry + Scenario: PlanDiffGate consumes comparison result output + Given iac_diff_terraform_plans has produced a comparison result + When PlanDiffGate.run processes the comparison result + Then the gate successfully consumes the output structure + + @iac-domain:terraform-validate-entry + Scenario: SecurityBaselineGate consumes comparison result output + Given iac_diff_terraform_plans has produced a comparison result + When SecurityBaselineGate.run processes the comparison result + Then the gate successfully consumes the output structure + + @iac-domain:terraform-validate-entry + Scenario: run_all gate operation consumes comparison result output + Given iac_diff_terraform_plans has produced a comparison result + When run_all gate operation processes the comparison result + Then all gates successfully consume the output structure diff --git a/dogfood/mining-output/features/iac_draft_terraform_module.feature b/dogfood/mining-output/features/iac_draft_terraform_module.feature new file mode 100644 index 0000000..fb7f6f3 --- /dev/null +++ b/dogfood/mining-output/features/iac_draft_terraform_module.feature @@ -0,0 +1,95 @@ +Feature: Infrastructure-as-code draft Terraform module generation + As an infrastructure engineer or automation workflow + I want to generate Terraform module drafts from natural-language user stories + So that I can accelerate infrastructure provisioning and submit modules to validation gates + + @iac-domain:terraform-validate-entry + Scenario: Engineer drafts a module with minimal user story + Given a user story "Create an S3 bucket for application logs" + When the engineer calls iac_draft_terraform_module with the user story + Then a Terraform module draft is returned + And the module contains valid Terraform syntax + + @iac-domain:terraform-validate-entry + Scenario: Engineer drafts a module without specifying provider + Given a user story "Deploy a virtual machine with 2 CPUs and 4GB RAM" + When the engineer calls iac_draft_terraform_module with the user story + And the provider parameter is omitted + Then a Terraform module draft is returned with default provider configuration + + @iac-domain:terraform-validate-entry + Scenario: Engineer drafts a module with explicit provider + Given a user story "Create a storage account for blob data" + And a provider "azure" + When the engineer calls iac_draft_terraform_module with the user story and provider + Then a Terraform module draft is returned + And the module includes azure provider-specific resources + + @iac-domain:terraform-validate-entry + Scenario: Tool rejects call missing required user story + When the engineer calls iac_draft_terraform_module without a user story parameter + Then the tool returns an error indicating the user_story parameter is required + + @iac-domain:terraform-validate-entry + Scenario: Generated module conforms to Terraform structure + Given a user story "Create a VPC with public and private subnets" + When the engineer calls iac_draft_terraform_module with the user story + Then the returned module contains resource blocks + And the module contains variable definitions + And the module contains output definitions + And the module structure is valid for Terraform module consumption + + @iac-domain:terraform-validate-entry + Scenario: Generated module can be consumed by IaCAmbiguityGate + Given a user story "Deploy a load balancer" + And a Terraform module draft generated from the user story + When IaCAmbiguityGate.run is called with the generated module + Then the gate processes the module without structural errors + + @iac-domain:terraform-validate-entry + Scenario: Generated module can be consumed by PlanDiffGate + Given a user story "Create a database instance" + And a Terraform module draft generated from the user story + When PlanDiffGate.run is called with the generated module + Then the gate processes the module without structural errors + + @iac-domain:terraform-validate-entry + Scenario: Generated module can be consumed by SecurityBaselineGate + Given a user story "Provision compute resources" + And a Terraform module draft generated from the user story + When SecurityBaselineGate.run is called with the generated module + Then the gate processes the module without structural errors + + @iac-domain:terraform-validate-entry + Scenario: Generated module participates in run_all multi-gate workflow + Given a user story "Deploy a web application infrastructure" + And a Terraform module draft generated from the user story + When run_all gate is invoked with the generated module + Then the module passes through all configured gates in sequence + And each gate receives the module in valid format + + @iac-domain:terraform-validate-entry + Scenario Outline: Tool handles edge-case user stories gracefully + Given a user story "" + When the engineer calls iac_draft_terraform_module with the user story + Then the tool + + Examples: + | user_story_input | outcome | + | | returns an error indicating user story cannot be empty | + | A very long user story description that contains thousands of characters representing an extremely detailed infrastructure request with multiple components, dependencies, networking requirements, security policies, compliance rules, monitoring specifications, backup strategies, disaster recovery plans, scaling policies, cost optimization requirements, and various other infrastructure considerations that would typically span multiple pages of documentation | returns a Terraform module draft or gracefully handles length limits | + | Create a bucket with name "special-chars-#$%&*@!" | returns a Terraform module draft handling special characters appropriately | + + @iac-domain:terraform-validate-entry + Scenario Outline: Tool supports multiple cloud providers + Given a user story "Create object storage" + And a provider "" + When the engineer calls iac_draft_terraform_module with the user story and provider + Then a Terraform module draft is returned + And the module uses -specific resource types + + Examples: + | provider | + | aws | + | azure | + | gcp | diff --git a/dogfood/mining-output/features/iac_explain_plan_diff.feature b/dogfood/mining-output/features/iac_explain_plan_diff.feature new file mode 100644 index 0000000..40756d5 --- /dev/null +++ b/dogfood/mining-output/features/iac_explain_plan_diff.feature @@ -0,0 +1,70 @@ +Feature: IaC Explain Plan Diff Tool + As an infrastructure engineer + I want to analyze Terraform plan changes for risk + So that I can understand the impact before applying changes + + Background: + Given the IaC Explain Plan Diff tool is available + + @iac-domain:terraform-validate-entry + Scenario: Infrastructure engineer requests analysis of a valid Terraform plan + Given a valid Terraform plan JSON file + When the plan is submitted to the iac_explain_plan_diff tool + Then a summary of the plan changes is returned + And the summary identifies any risky actions + + @iac-domain:terraform-validate-entry + Scenario: Infrastructure engineer receives ambiguous configuration warnings + Given a valid Terraform plan JSON file with ambiguous configurations + When the plan is submitted to the iac_explain_plan_diff tool + Then the IaCAmbiguityGate is invoked to detect ambiguous configurations + And the summary includes ambiguous configuration risks + + @iac-domain:terraform-validate-entry + Scenario: Infrastructure engineer receives plan difference analysis + Given a valid Terraform plan JSON file with significant differences + When the plan is submitted to the iac_explain_plan_diff tool + Then the PlanDiffGate is invoked to analyze plan differences + And the summary includes plan difference risks + + @iac-domain:terraform-validate-entry + Scenario: Infrastructure engineer receives security baseline violations + Given a valid Terraform plan JSON file with security baseline violations + When the plan is submitted to the iac_explain_plan_diff tool + Then the SecurityBaselineGate is invoked to check security policy violations + And the summary includes security baseline risks + + @iac-domain:terraform-validate-entry + Scenario: Infrastructure engineer analyzes a plan with multiple risk types + Given a valid Terraform plan JSON file with multiple risk categories + When the plan is submitted to the iac_explain_plan_diff tool + Then all applicable gates are executed + And the summary distinguishes between ambiguous configuration risks + And the summary distinguishes between plan difference risks + And the summary distinguishes between security baseline risks + + # TODO: Clarify expected error message format and structure + @iac-domain:terraform-validate-entry + Scenario: Infrastructure engineer submits malformed Terraform plan JSON + Given a malformed Terraform plan JSON file + When the plan is submitted to the iac_explain_plan_diff tool + Then an error is returned indicating invalid JSON + + # TODO: Clarify expected error message format and structure + @iac-domain:terraform-validate-entry + Scenario: Infrastructure engineer submits invalid Terraform plan structure + Given a JSON file that is not a valid Terraform plan structure + When the plan is submitted to the iac_explain_plan_diff tool + Then an error is returned indicating invalid plan structure + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Infrastructure engineer omits required plan_json parameter + When the iac_explain_plan_diff tool is invoked without the plan_json parameter + Then an error is returned indicating the parameter is required + + @iac-domain:terraform-validate-entry + Scenario: Infrastructure engineer analyzes a plan with no risks + Given a valid Terraform plan JSON file with no risky actions + When the plan is submitted to the iac_explain_plan_diff tool + Then a summary is returned + And the summary indicates no risks were identified diff --git a/dogfood/mining-output/features/iac_suggest_security_remediation.feature b/dogfood/mining-output/features/iac_suggest_security_remediation.feature new file mode 100644 index 0000000..b1b3264 --- /dev/null +++ b/dogfood/mining-output/features/iac_suggest_security_remediation.feature @@ -0,0 +1,57 @@ +Feature: IaC Security Remediation Suggestions + As a security engineer + I want to receive actionable HCL patch suggestions for Trivy security findings + So that I can remediate infrastructure-as-code misconfigurations without manual research + + Background: + Given the iac_suggest_security_remediation tool is available + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Security engineer receives HCL patches for valid Trivy findings + Given a Trivy config-scan has produced valid JSON findings + When the security engineer requests remediation suggestions with the Trivy findings JSON + Then the tool returns suggested HCL patches + And the patches correspond to the security findings in the JSON + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Security engineer receives context-aware patches when HCL text is provided + Given a Trivy config-scan has produced valid JSON findings + And existing HCL configuration text is available + When the security engineer requests remediation suggestions with both the Trivy findings JSON and the HCL text + Then the tool returns suggested HCL patches + And the patches are tailored to the supplied HCL context + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Security engineer receives generic patches when HCL text is omitted + Given a Trivy config-scan has produced valid JSON findings + When the security engineer requests remediation suggestions with only the Trivy findings JSON + Then the tool returns generic HCL remediation patches + And the patches are based solely on the Trivy findings + + @best-practices:llm-drafter-temperature-zero + Scenario: Security engineer submits Trivy findings with zero security issues + Given a Trivy config-scan has produced JSON with zero findings + When the security engineer requests remediation suggestions with the empty findings JSON + Then the tool completes without error + And no patches are suggested + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Security engineer submits Trivy findings with multiple security issues + Given a Trivy config-scan has produced JSON with multiple findings + When the security engineer requests remediation suggestions with the findings JSON + Then the tool returns suggested HCL patches + And patches are provided for each finding in the JSON + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Tool validates Trivy JSON schema before processing + Given an invalid JSON document that does not conform to Trivy output schema + When the security engineer requests remediation suggestions with the invalid JSON + Then the tool reports a validation error + And no patches are suggested + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Suggested patches are suitable for applying to HCL files + Given a Trivy config-scan has produced valid JSON findings + When the security engineer requests remediation suggestions with the Trivy findings JSON + Then the tool returns suggested HCL patches + And the patches are in a format suitable for applying to HCL configuration files diff --git a/dogfood/mining-output/features/iac_validate_terraform_dir.feature b/dogfood/mining-output/features/iac_validate_terraform_dir.feature new file mode 100644 index 0000000..a7013f2 --- /dev/null +++ b/dogfood/mining-output/features/iac_validate_terraform_dir.feature @@ -0,0 +1,53 @@ +Feature: Validate Terraform directory + As an AI agent or automation workflow + I want to validate Terraform configuration files in a temporary directory + So that I can verify syntax and structure before planning or deployment + + @iac-domain:terraform-validate-entry + Scenario: Agent validates syntactically correct Terraform files + Given a temporary directory contains valid Terraform configuration files + When the agent invokes the validate tool with the tf_files parameter + Then the validation returns a success result + And the result indicates all configurations are valid + + @iac-domain:terraform-validate-entry + Scenario: Agent validates Terraform files with syntax errors + Given a temporary directory contains Terraform files with syntax errors + When the agent invokes the validate tool with the tf_files parameter + Then the validation returns a failure result + And the result identifies the syntax errors + + @iac-domain:terraform-validate-entry + Scenario: Agent validates multiple Terraform files in one invocation + Given a temporary directory contains multiple valid Terraform configuration files + When the agent invokes the validate tool with all files in the tf_files parameter + Then the validation processes all files + And the validation returns a success result for the entire set + + @iac-domain:terraform-validate-entry + Scenario: Agent validates mixed valid and invalid Terraform files + Given a temporary directory contains both valid and invalid Terraform files + When the agent invokes the validate tool with the tf_files parameter + Then the validation processes all files + And the validation returns a failure result + And the result distinguishes between files with errors and valid files + + @iac-domain:terraform-validate-entry + Scenario: Agent attempts validation without required tf_files parameter + When the agent invokes the validate tool without the tf_files parameter + Then the validation returns an error + And the error indicates the tf_files parameter is required + + @iac-domain:terraform-validate-entry + Scenario: Agent validates Terraform files with structural issues + Given a temporary directory contains Terraform files with structural validity issues + When the agent invokes the validate tool with the tf_files parameter + Then the validation returns a failure result + And the result identifies the structural issues + + @iac-domain:terraform-validate-entry + Scenario: Agent validates empty Terraform configuration + Given a temporary directory contains an empty Terraform configuration file + When the agent invokes the validate tool with the tf_files parameter + Then the validation processes the file + And the validation returns a result indicating the configuration state diff --git a/dogfood/mining-output/features/pickled_bdd_ambiguity.feature b/dogfood/mining-output/features/pickled_bdd_ambiguity.feature new file mode 100644 index 0000000..9d40f2f --- /dev/null +++ b/dogfood/mining-output/features/pickled_bdd_ambiguity.feature @@ -0,0 +1,89 @@ +Feature: Ambiguity CLI command + As a developer using pickled-bdd + I want to run ambiguity analysis on a Gherkin feature file via a CLI shortcut + So that I can quickly check for ambiguous scenarios without verbose syntax + + Background: + Given a Gherkin feature file exists at a known path + + @best-practices:agent-path-first-class + Scenario: User invokes ambiguity command with required feature file argument + When the user runs the ambiguity command with a feature file path + Then the command accepts the feature file argument + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Command writes informational message to stderr + When the user runs the ambiguity command with a feature file path + Then stderr contains the message "(equivalent to: pickled-bdd check --gate ambiguity)" + + @pickled-internal:core-llm-cache-default-on + Scenario: Command fails when LLM configuration is invalid + Given the LLM configuration environment is invalid + When the user runs the ambiguity command with a feature file path + Then a ClickException is raised with the configuration error message + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Command passes when LLM is unavailable + Given the LLM client is unavailable + When the user runs the ambiguity command with a feature file path + Then the JSON output contains verdict "PASS" + And the JSON output contains notes "LLM unavailable; ambiguity gate skipped" + And the command exits with code 0 + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Command outputs valid JSON structure to stdout + Given the LLM client is available + When the user runs the ambiguity command with a feature file path + Then the JSON output is written to stdout + And the JSON output contains key "gate" + And the JSON output contains key "verdict" + And the JSON output contains key "notes" + And the JSON output contains key "findings" + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Gate field always contains ambiguity value + Given the LLM client is available + When the user runs the ambiguity command with a feature file path + Then the JSON output field "gate" has value "ambiguity" + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario Outline: Verdict field contains valid enum string value + Given the LLM client is available + And the ambiguity gate returns a verdict + When the user runs the ambiguity command with a feature file path + Then the JSON output field "verdict" has value "" + + Examples: + | verdict | + | PASS | + | WARN | + | FAIL | + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Findings array contains properly structured ambiguity findings + Given the LLM client is available + And the ambiguity gate detects ambiguous scenarios + When the user runs the ambiguity command with a feature file path + Then each finding in the JSON output contains key "scenario" + And each finding in the JSON output contains key "alternatives" + And each finding in the JSON output contains key "suggested_fix" + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario Outline: Command exits with code based on verdict + Given the LLM client is available + And the ambiguity gate returns a verdict + When the user runs the ambiguity command with a feature file path + Then the command exits with code + + Examples: + | verdict | exit_code | + | PASS | 0 | + | WARN | 1 | + | FAIL | 2 | + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: JSON output is formatted with proper indentation and encoding + Given the LLM client is available + When the user runs the ambiguity command with a feature file path + Then the JSON output uses 2-space indentation + And the JSON output preserves non-ASCII characters diff --git a/dogfood/mining-output/features/pickled_bdd_ambiguitygate.feature b/dogfood/mining-output/features/pickled_bdd_ambiguitygate.feature new file mode 100644 index 0000000..27e6de1 --- /dev/null +++ b/dogfood/mining-output/features/pickled_bdd_ambiguitygate.feature @@ -0,0 +1,115 @@ +Feature: AmbiguityGate analyzes BDD scenarios for ambiguity + + As a quality engineer + I want to check feature scenarios for ambiguous wording + So that implementation teams receive unambiguous requirements + + Background: + Given an AmbiguityGate instance + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Quality engineer runs gate against a non-Feature object + When the engineer runs the gate with a target of type "dict" + Then the gate returns verdict FAIL + And the notes describe the type mismatch "Expected Feature, got dict" + And the findings tuple is empty + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Quality engineer runs gate against a Feature with zero scenarios + Given a Feature with 0 scenarios + When the engineer runs the gate + Then the gate returns verdict PASS + And the notes indicate "0/0 scenarios flagged ambiguous" + And the findings tuple is empty + + @pickled-internal:core-llm-cache-default-on + Scenario: Quality engineer runs gate against unambiguous scenarios + Given a Feature with 3 scenarios + And the LLM identifies 0 scenarios as ambiguous + When the engineer runs the gate + Then the gate returns verdict PASS + And the notes indicate "0/3 scenarios flagged ambiguous" + And the findings tuple is empty + + @pickled-internal:core-llm-cache-default-on + Scenario: Quality engineer runs gate against fully ambiguous scenarios + Given a Feature with 3 scenarios + And the LLM identifies 3 scenarios as ambiguous + When the engineer runs the gate + Then the gate returns verdict FAIL + And the notes indicate "3/3 scenarios flagged ambiguous" + And the findings tuple contains 3 AmbiguityFinding objects + + @pickled-internal:core-llm-cache-default-on + Scenario: Quality engineer runs gate against partially ambiguous scenarios + Given a Feature with 4 scenarios + And the LLM identifies 2 scenarios as ambiguous + When the engineer runs the gate + Then the gate returns verdict WARN + And the notes indicate "2/4 scenarios flagged ambiguous" + And the findings tuple contains 2 AmbiguityFinding objects + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Quality engineer runs gate when all LLM responses fail to parse + Given a Feature with 3 scenarios + And all LLM responses return malformed JSON + When the engineer runs the gate + Then the gate returns verdict WARN + And the findings tuple is empty + And the notes list all 3 scenario names as parse errors + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Quality engineer runs gate when some responses fail to parse and rest are unambiguous + Given a Feature with 4 scenarios + And 2 LLM responses return malformed JSON + And the LLM identifies 0 of the remaining scenarios as ambiguous + When the engineer runs the gate + Then the gate returns verdict WARN + And the findings tuple is empty + And the notes indicate "0/2 scenarios flagged ambiguous" + And the notes list 2 scenario names as parse errors + + @pickled-internal:core-llm-cache-default-on + Scenario: Quality engineer runs gate when some responses fail to parse and rest are ambiguous + Given a Feature with 4 scenarios + And 1 LLM response returns malformed JSON + And the LLM identifies 2 of the remaining 3 scenarios as ambiguous + When the engineer runs the gate + Then the gate returns verdict WARN + And the findings tuple contains 2 AmbiguityFinding objects + And the notes indicate "2/3 scenarios flagged ambiguous" + And the notes list 1 scenario name as parse error + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Quality engineer examines an AmbiguityFinding + Given a Feature with 1 scenario named "User logs in" + And the LLM identifies the scenario as ambiguous with 2 alternatives and a suggested fix + When the engineer runs the gate + Then the first finding contains scenario name "User logs in" + And the first finding contains a tuple of 2 alternative implementation strings + And the first finding contains a suggested fix string + + @pickled-internal:core-llm-cache-default-on + Scenario Outline: Quality engineer runs gate against LLM responses with markdown fences + Given a Feature with 1 scenario + And the LLM response contains + And the enclosed JSON indicates the scenario is + When the engineer runs the gate + Then the gate successfully parses the JSON + And the gate returns verdict + + Examples: + | fence_style | ambiguity_status | expected_verdict | + | triple backticks without language | ambiguous | FAIL | + | triple backticks with "json" tag | ambiguous | FAIL | + | triple backticks without language | unambiguous | PASS | + | triple backticks with "json" tag | unambiguous | PASS | + + @pickled-internal:core-llm-cache-default-on + Scenario: Quality engineer provides context parameter + Given a Feature with 2 scenarios + And the LLM identifies 0 scenarios as ambiguous + And a context dictionary with arbitrary keys + When the engineer runs the gate with the context parameter + Then the gate returns verdict PASS + And the result is identical to running without context diff --git a/dogfood/mining-output/features/pickled_bdd_check.feature b/dogfood/mining-output/features/pickled_bdd_check.feature new file mode 100644 index 0000000..76ae5ef --- /dev/null +++ b/dogfood/mining-output/features/pickled_bdd_check.feature @@ -0,0 +1,86 @@ +Feature: Check feature file quality with gates + As a developer or CI/CD pipeline + I want to validate Gherkin feature files for quality issues + So that I can enforce standards and catch ambiguity problems early + + Background: + Given a feature file exists at "sample.feature" + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Developer checks a valid feature file with LLM available + Given the LLM client is available + When the check command is invoked with the feature file path + Then the command produces JSON output to stdout + And the JSON contains exactly four top-level keys: "gate", "verdict", "notes", "findings" + And the "gate" value is "ambiguity" + And the process exits with code 0 + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Developer checks a feature file when LLM is unavailable + Given the LLM client is unavailable + When the check command is invoked with the feature file path + Then the JSON contains exactly four top-level keys: "gate", "verdict", "notes", "findings" + And the "gate" value is "ambiguity" + And the "verdict" value is "PASS" + And the "notes" value indicates the gate was skipped due to unavailable LLM + And the "findings" array is empty + And the process exits with code 0 + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario Outline: Developer receives appropriate exit codes based on verdict + Given the LLM client is available + And the ambiguity gate will return verdict "" + When the check command is invoked with the feature file path + Then the process exits with code + + Examples: + | verdict | exit_code | + | PASS | 0 | + | WARN | 1 | + | FAIL | 2 | + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Developer encounters LLM client configuration error + Given the LLM client factory is configured with invalid settings + When the check command is invoked with the feature file path + Then a ClickException is raised with the configuration error message + + # TODO: Clarify expected behavior when feature_file path does not exist or is invalid + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Developer provides gate parameter + Given the LLM client is available + When the check command is invoked with gate parameter "" + Then the ambiguity gate executes regardless of the parameter value + And the "gate" value is "ambiguity" + + Examples: + | gate_name | + | ambiguity | + | some-other | + | invalid | + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Developer verifies JSON output format + Given the LLM client is available + And the ambiguity gate returns findings with Unicode characters + When the check command is invoked with the feature file path + Then the JSON output uses 2-space indentation + And the JSON output preserves Unicode characters without escaping + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Developer receives findings filtered by type + Given the LLM client is available + And the ambiguity gate returns mixed finding types + When the check command is invoked with the feature file path + Then the "findings" array contains only AmbiguityFinding instances + And each finding contains "scenario", "alternatives", and "suggested_fix" keys + And the "alternatives" value is a list + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Developer verifies finding structure + Given the LLM client is available + And the ambiguity gate returns an AmbiguityFinding + When the check command is invoked with the feature file path + Then each finding object has a "scenario" key with the target name + And each finding object has an "alternatives" key with a list value + And each finding object has a "suggested_fix" key diff --git a/dogfood/mining-output/features/pickled_bdd_draft.feature b/dogfood/mining-output/features/pickled_bdd_draft.feature new file mode 100644 index 0000000..6714990 --- /dev/null +++ b/dogfood/mining-output/features/pickled_bdd_draft.feature @@ -0,0 +1,61 @@ +Feature: Draft Gherkin feature from user story + As a developer using pickled-bdd + I want to convert a Markdown user story into a Gherkin feature file + So that I can begin defining executable specifications + + Background: + Given a valid LLM client can be constructed + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer drafts feature to stdout + Given a user story file exists at "story.md" + When the developer runs the draft command with story file "story.md" and no output path + Then the drafted Gherkin feature text appears on stdout + And the feature text is the LLM response with leading and trailing whitespace removed + And no file is written to disk + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Developer drafts feature to a file + Given a user story file exists at "story.md" + When the developer runs the draft command with story file "story.md" and output path "feature.feature" + Then the drafted Gherkin feature text is written to "feature.feature" as UTF-8 + And the confirmation message "Wrote feature.feature" appears on stderr + And nothing appears on stdout + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer receives draft result metadata + Given a user story file exists at "story.md" + When the developer runs the draft command with story file "story.md" + Then the draft result contains rationale "LLM-drafted from user story; no post-processing applied." + And the draft result contains an empty warnings tuple + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: LLM client configuration fails + Given the LLM client factory raises a ConfigError with message "Invalid API key" + And a user story file exists at "story.md" + When the developer runs the draft command with story file "story.md" + Then a ClickException is raised with message "Invalid API key" + + @best-practices:agent-path-first-class + Scenario: User story file does not exist + Given no file exists at "missing.md" + When the developer runs the draft command with story file "missing.md" + Then a FileNotFoundError is raised + + # TODO: Verify behavior when output path is not writable (PermissionError) + # TODO: Verify behavior when story_file is readable but not valid UTF-8 + + @pickled-internal:core-llm-cache-default-on + Scenario: Command does not validate LLM output + Given a user story file exists at "story.md" + And the LLM returns malformed Gherkin text "This is not valid Gherkin syntax!!!" + When the developer runs the draft command with story file "story.md" + Then the drafted feature text is "This is not valid Gherkin syntax!!!" + And no validation error is raised + + @pickled-internal:core-llm-cache-default-on + Scenario: LLM output whitespace is normalized + Given a user story file exists at "story.md" + And the LLM returns text with leading newlines and trailing spaces + When the developer runs the draft command with story file "story.md" + Then the drafted feature text has no leading or trailing whitespace diff --git a/dogfood/mining-output/features/pickled_bdd_mcp.feature b/dogfood/mining-output/features/pickled_bdd_mcp.feature new file mode 100644 index 0000000..019128f --- /dev/null +++ b/dogfood/mining-output/features/pickled_bdd_mcp.feature @@ -0,0 +1,24 @@ +Feature: MCP Command Group + As a CLI user + I want the mcp command to serve as a parent group for MCP server operations + So that I can organize and access MCP-related subcommands + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer invokes mcp command directly + When the mcp function is called directly + Then it completes without raising exceptions + And it returns None + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer invokes mcp command with no side effects + Given no prior state exists + When the mcp function is called directly + Then no console output is produced + And no global state is modified + And no file system operations are performed + And no network calls are made + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer checks mcp function signature + When the mcp function signature is inspected + Then it accepts zero parameters diff --git a/dogfood/mining-output/features/pickled_bdd_run_all.feature b/dogfood/mining-output/features/pickled_bdd_run_all.feature new file mode 100644 index 0000000..9ae643e --- /dev/null +++ b/dogfood/mining-output/features/pickled_bdd_run_all.feature @@ -0,0 +1,110 @@ +Feature: Batch validation of BDD feature files + + As a quality-gate orchestrator + I want to validate all Gherkin feature files in a project + So that I can detect parsing errors before runtime without requiring an LLM connection + + Background: + Given a project workspace directory + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Project contains no features directory + When no features directory exists under the workspace + Then the validation returns a single result + And the result has gate name "bdd.features" + And the result has verdict "PASS" + And the result notes indicate no features directory was found + + @data-domain:migration-drift-gate + Scenario: Project contains an empty features directory + Given a features directory exists under the workspace + But no feature files exist in the features directory + When validation runs + Then the validation returns a single result + And the result has gate name "bdd.features" + And the result has verdict "PASS" + And the result notes indicate no features directory was found + + @best-practices:agent-path-first-class + Scenario: Project contains a single valid feature file + Given a feature file "login.feature" with valid Gherkin content exists + When validation runs + Then the validation returns two results + And the first result has gate name "bdd.parse.login.feature" + And the first result has verdict "PASS" + And the first result notes contain the relative file path + And the second result has gate name "bdd.ambiguity" + And the second result has verdict "PASS" + And the second result notes indicate ambiguity check is skipped + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Project contains multiple valid feature files + Given feature files exist at multiple directory depths: + | path | + | features/authentication.feature | + | features/admin/users.feature | + | features/admin/roles.feature | + And all feature files contain valid Gherkin + When validation runs + Then the validation returns four results + And results are ordered by sorted file path + And each feature file has a corresponding PASS result with gate name "bdd.parse." + And the final result has gate name "bdd.ambiguity" with verdict "PASS" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario Outline: Feature file with parsing errors + Given a feature file "" exists + And the file content is + When validation runs + Then a result exists with gate name "bdd.parse." + And that result has verdict "FAIL" + And that result notes contain "" + And the validation completes without raising exceptions + And the final result has gate name "bdd.ambiguity" with verdict "PASS" + + Examples: + | filename | condition | error_message | + | empty.feature | empty | Gherkin text is empty | + | blank.feature | whitespace only | Gherkin text is empty | + | invalid.feature | missing Feature block | No Feature found in Gherkin text | + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Mixed valid and invalid feature files + Given feature files exist: + | filename | status | + | valid.feature | valid | + | empty.feature | empty | + | valid2.feature | valid | + | broken.feature | invalid | + When validation runs + Then the validation returns five results + And results for valid files have verdict "PASS" + And results for invalid files have verdict "FAIL" + And invalid file results contain error details in notes + And the final result has gate name "bdd.ambiguity" with verdict "PASS" + And validation completes without raising exceptions + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: All feature files fail to parse + Given multiple feature files exist + But all feature files contain parsing errors + When validation runs + Then all feature file results have verdict "FAIL" + And each FAIL result notes contain the corresponding error message + And the final result has gate name "bdd.ambiguity" with verdict "PASS" + And validation completes without raising exceptions + + @best-practices:agent-path-first-class + Scenario: Feature files in nested subdirectories are discovered + Given feature files exist in deeply nested paths: + | path | + | features/smoke/critical.feature | + | features/regression/api/endpoints.feature | + | features/regression/ui/flows.feature | + When validation runs + Then all nested feature files are validated + And results are returned in sorted path order + + # TODO: Clarify behavior when workdir is invalid or inaccessible (permission errors, non-existent path) + # TODO: Clarify exact format of relative path in notes for passing files + # TODO: Document whether symbolic links in features/ are followed diff --git a/dogfood/mining-output/features/pickled_core_check_all.feature b/dogfood/mining-output/features/pickled_core_check_all.feature new file mode 100644 index 0000000..e94085c --- /dev/null +++ b/dogfood/mining-output/features/pickled_core_check_all.feature @@ -0,0 +1,80 @@ +Feature: check-all command validates entire workspace using all registered gates + + Background: + Given a workspace directory exists + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer runs check-all without arguments + When the developer runs check-all without specifying a workdir + Then check-all uses the current directory as the workspace root + And check-all discovers all gates from installed pickled-* packages + And check-all executes all discovered gates + And check-all reports verdicts from all gates + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer runs check-all with a specific workdir + Given a workspace directory at "/path/to/workspace" + When the developer runs check-all with workdir "/path/to/workspace" + Then check-all uses "/path/to/workspace" as the workspace root + And check-all discovers all gates from installed pickled-* packages + And check-all executes all discovered gates + And check-all reports verdicts from all gates + + @core-domain:verdict-three-state-ladder + Scenario: Developer runs check-all and all gates pass + Given a valid workspace with features/, specs/, infra/, and migrations/ directories + And all registered gates will pass + When the developer runs check-all + Then check-all exits with code 0 + + @data-domain:migration-drift-gate + Scenario: Developer runs check-all and at least one gate fails + Given a valid workspace with features/, specs/, infra/, and migrations/ directories + And at least one registered gate will fail + When the developer runs check-all + Then check-all exits with a non-zero code + + @core-domain:verdict-three-state-ladder + Scenario: Developer runs check-all with warnings and warn_ok is false + Given a valid workspace with features/, specs/, infra/, and migrations/ directories + And gates will produce only WARN verdicts + When the developer runs check-all with warn_ok set to false + Then check-all exits with a non-zero code + + @core-domain:verdict-three-state-ladder + Scenario: Developer runs check-all with warnings and warn_ok is omitted + Given a valid workspace with features/, specs/, infra/, and migrations/ directories + And gates will produce only WARN verdicts + When the developer runs check-all without specifying warn_ok + Then check-all exits with a non-zero code + + @core-domain:verdict-three-state-ladder + Scenario: Developer runs check-all with warnings and warn_ok is true + Given a valid workspace with features/, specs/, infra/, and migrations/ directories + And gates will produce only WARN verdicts + When the developer runs check-all with warn_ok set to true + Then check-all exits with code 0 + + @core-domain:verdict-three-state-ladder + Scenario: CI pipeline runs check-all with mixed PASS and WARN verdicts and warn_ok is true + Given a valid workspace with features/, specs/, infra/, and migrations/ directories + And some gates will pass + And some gates will produce WARN verdicts + When the CI pipeline runs check-all with warn_ok set to true + Then check-all exits with code 0 + + @core-domain:verdict-three-state-ladder + Scenario: CI pipeline runs check-all with FAIL and WARN verdicts regardless of warn_ok + Given a valid workspace with features/, specs/, infra/, and migrations/ directories + And some gates will fail + And some gates will produce WARN verdicts + When the CI pipeline runs check-all with warn_ok set to true + Then check-all exits with a non-zero code + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer runs check-all and gates from multiple pickled-* packages are executed + Given multiple pickled-* packages are installed + And each package registers gates + When the developer runs check-all + Then check-all executes gates from all installed pickled-* packages + And check-all reports verdicts from gates across all packages diff --git a/dogfood/mining-output/features/pickled_core_mine.feature b/dogfood/mining-output/features/pickled_core_mine.feature new file mode 100644 index 0000000..213e118 --- /dev/null +++ b/dogfood/mining-output/features/pickled_core_mine.feature @@ -0,0 +1,48 @@ +Feature: Mine command no-op behavior + As a developer using pickled-core tooling + I want the mine command to execute without errors in its current state + So that the command infrastructure is stable while implementation is pending + + # TODO: Docstring claims function mines surfaces, stories, features, and gate results but implementation is empty + # TODO: No project path parameter accepted despite docstring implying project analysis + # TODO: Docstring suggests output will be produced but function returns None with no side effects + + Scenario: User invokes mine command + When the mine command is invoked + Then the command completes without raising exceptions + And the command returns None + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User invokes mine command with no project structure + Given no pickled-core configuration is present + And no project structure exists + When the mine command is invoked + Then the command completes without raising exceptions + And the command returns None + + @rules-domain:coverage-union-across-features + Scenario: User invokes mine command and checks filesystem + Given the filesystem state is recorded + When the mine command is invoked + Then no files are created + And no files are modified + And no files are deleted + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User invokes mine command and checks output streams + When the mine command is invoked + Then no output is written to stdout + And no output is written to stderr + + Scenario: User invokes mine command and checks performance + When the mine command is invoked + Then the command completes immediately + + @pickled-internal:core-llm-cache-default-on + Scenario: User invokes mine command multiple times + When the mine command is invoked + And the mine command is invoked again + And the mine command is invoked a third time + Then all invocations complete without raising exceptions + And all invocations return None + And the behavior is identical across all invocations diff --git a/dogfood/mining-output/features/pickled_core_mine_all.feature b/dogfood/mining-output/features/pickled_core_mine_all.feature new file mode 100644 index 0000000..5b568d6 --- /dev/null +++ b/dogfood/mining-output/features/pickled_core_mine_all.feature @@ -0,0 +1,96 @@ +Feature: Mine all command orchestrates the complete mining pipeline + + As a developer or QA engineer + I want to run a single command that executes all mining stages + So that I can extract, document, tag, evaluate, and report on software surfaces end-to-end + + Background: + Given a valid target codebase exists + + @best-practices:agent-path-first-class + Scenario: User runs mine all with a valid target + When the user invokes mine all with the target codebase + Then the inventory stage runs first + And the code stage runs after inventory + And the stories stage runs after code + And the features stage runs after stories + And the tag stage runs after features + And the evaluate stage runs after tag + And the report stage runs after evaluate + And the command exits with status 0 + + @best-practices:llm-drafter-temperature-zero + Scenario: User omits the required target parameter + When the user invokes mine all without specifying a target + Then the command raises an error indicating target is required + And the command exits with a non-zero status + + @best-practices:agent-path-first-class + Scenario: User specifies a custom output directory + When the user invokes mine all with target and output_dir set to "custom/output/path" + Then all mining artifacts are written to "custom/output/path" + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User requests verbose logging + When the user invokes mine all with target and the verbose flag enabled + Then detailed logging output is written to stderr + And the command completes all seven stages + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User filters surfaces by package or surface-id substring + When the user invokes mine all with target and surfaces filter set to "auth" + Then only surfaces whose package name or surface-id contains "auth" are processed + And surfaces not matching the filter are excluded + + @pickled-internal:core-llm-cache-default-on + Scenario: User limits concurrent LLM calls + When the user invokes mine all with target and max_parallel set to 3 + Then the stories stage uses at most 3 concurrent LLM calls + And the features stage uses at most 3 concurrent LLM calls + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User specifies custom ruleset configuration + When the user invokes mine all with target and ruleset_dir set to "custom/rulesets" + And ruleset_config is set to "custom_rules.yaml" + Then the tag stage applies rulesets from "custom/rulesets" using "custom_rules.yaml" + + @rules-domain:coverage-union-across-features + Scenario Outline: User controls overwrite behavior for stories and features + When the user invokes mine all with target and enabled + Then existing files are replaced during the stage + + Examples: + | flag | artifact_type | stage | + | overwrite_stories | story | stories | + | overwrite_features | feature | features | + + @bdd-domain:draft-warnings-field-populated-on-failure + Scenario: User adjusts code-collection parameters + When the user invokes mine all with target and depth set to 2 + And callee_scope is set to "same_package" + And max_hops is set to 3 + And max_callees is set to 50 + And max_code_lines is set to 2000 + Then the code stage collects source context per surface using those constraints + And the volume and scope of extracted code reflects the specified limits + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User enables cycle detection in the call graph + When the user invokes mine all with target and detect_cycles flag enabled + And the code stage detects call-graph cycles + Then a "code-context/_cycles.json" file is written + + @best-practices:llm-drafter-temperature-zero + Scenario: A pipeline stage fails during execution + Given the code stage will encounter a fatal error + When the user invokes mine all with target + Then the command stops execution at the code stage + And subsequent stages do not run + And the command exits with a non-zero status + + @data-domain:migration-drift-gate + Scenario: User runs in non-quick mode with interactive prompts + Given quick mode is disabled + When the user invokes mine all with target + Then the command prompts the user for confirmation at interactive decision points + And the command proceeds based on user responses diff --git a/dogfood/mining-output/features/pickled_core_mine_code.feature b/dogfood/mining-output/features/pickled_core_mine_code.feature new file mode 100644 index 0000000..523198e --- /dev/null +++ b/dogfood/mining-output/features/pickled_core_mine_code.feature @@ -0,0 +1,109 @@ +Feature: Mine code from a target codebase + As a developer or automation system + I want to extract source code context from software projects + So that I can document, analyze, or use the code for AI-assisted workflows + + @best-practices:llm-drafter-temperature-zero + Scenario: Developer mines code without providing required target argument + When a developer invokes the mine code command without a target + Then the command fails with a non-zero exit code + And an error message indicates the target argument is required + + @bdd-domain:draft-warnings-field-populated-on-failure + Scenario: Developer mines code to default output location + Given a valid target codebase exists + When a developer invokes the mine code command with the target + Then the command exits with code 0 + And mining results are written to the default output directory + And structured output files contain surface metadata + And structured output files contain source code + And structured output files contain call graph information + + @best-practices:agent-path-first-class + Scenario: Developer mines code to specified output directory + Given a valid target codebase exists + And an output directory path is specified + When a developer invokes the mine code command with the target and output_dir + Then the command exits with code 0 + And mining results are written to the specified output directory + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Developer mines code with verbose logging enabled + Given a valid target codebase exists + When a developer invokes the mine code command with the target and verbose flag + Then the command exits with code 0 + And additional diagnostic information is written to stderr during mining + + @pickled-internal:mcp-output-fixed-json-shape + Scenario Outline: Developer filters surfaces by package or surface-id substring + Given a valid target codebase exists with multiple surfaces + And the surfaces parameter is set to "" + When a developer invokes the mine code command with the target and surfaces filter + Then the command exits with code 0 + And only surfaces matching "" are included in the output + + Examples: + | filter | + | package_name | + | surface_id_fragment | + + @oss-hygiene:no-secrets-in-repo + Scenario: Developer controls source code context depth + Given a valid target codebase exists + And the depth parameter is set to a specific value + When a developer invokes the mine code command with the target and depth + Then the command exits with code 0 + And the collected source code context matches the specified depth level + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer determines which intra-project callees to follow + Given a valid target codebase exists with intra-project function calls + And the callee_scope parameter is set to a specific scope + When a developer invokes the mine code command with the target and callee_scope + Then the command exits with code 0 + And only callees within the specified scope are followed during analysis + + Scenario: Developer limits call graph traversal depth with max_hops + Given a valid target codebase exists with nested function calls + And the max_hops parameter is set to a specific value + When a developer invokes the mine code command with the target and max_hops + Then the command exits with code 0 + And callee expansion is capped at the specified number of hops + + @bdd-domain:draft-warnings-field-populated-on-failure + Scenario: Developer limits number of callee units per surface + Given a valid target codebase exists with multiple callees per surface + And the max_callees parameter is set to a specific limit + When a developer invokes the mine code command with the target and max_callees + Then the command exits with code 0 + And the number of collected callee units per surface does not exceed the limit + + @bdd-domain:draft-warnings-field-populated-on-failure + Scenario: Developer limits total source lines per surface + Given a valid target codebase exists + And the max_code_lines parameter is set to a specific limit + When a developer invokes the mine code command with the target and max_code_lines + Then the command exits with code 0 + And the total source lines collected per surface does not exceed the limit + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer detects circular dependencies in call graph + Given a valid target codebase exists with circular dependencies + And the detect_cycles flag is enabled + When a developer invokes the mine code command with the target and detect_cycles + Then the command exits with code 0 + And a code-context/_cycles.json file is written to the output directory + And the cycles file contains detected circular dependencies from call graph edges + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer mines code without cycle detection + Given a valid target codebase exists + When a developer invokes the mine code command with the target + Then the command exits with code 0 + And no code-context/_cycles.json file is written + + @best-practices:agent-path-first-class + Scenario: Automation system mines code and encounters an error + Given an invalid target codebase path + When an automation system invokes the mine code command with the invalid target + Then the command exits with a non-zero exit code diff --git a/dogfood/mining-output/features/pickled_core_mine_evaluate.feature b/dogfood/mining-output/features/pickled_core_mine_evaluate.feature new file mode 100644 index 0000000..ac021f5 --- /dev/null +++ b/dogfood/mining-output/features/pickled_core_mine_evaluate.feature @@ -0,0 +1,92 @@ +Feature: Mine evaluate command checks coverage and ambiguity quality gates + + Background: + Given a target codebase exists + And mining stages 1-5 have completed successfully + And mined surfaces exist in the output directory + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Engineer evaluates surfaces when all gates pass + Given all configured coverage gates are satisfied + And all configured ambiguity gates are satisfied + When the engineer runs mine evaluate against the target + Then the command exits with status 0 + And gate results are written to the output directory in structured format + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Engineer evaluates surfaces when any gate fails + Given at least one configured gate is not satisfied + When the engineer runs mine evaluate against the target + Then the command exits with a non-zero status + And gate results are written to the output directory in structured format + + @best-practices:llm-drafter-temperature-zero + Scenario: CI pipeline attempts evaluation without prior mining stages + Given mining stages 1-5 have not been run + When the pipeline runs mine evaluate against the target + Then the command fails or reports missing prerequisite data + And the command exits with a non-zero status + + @data-domain:migration-drift-gate + Scenario: Engineer evaluates with custom output directory + Given mined artifacts exist in a custom directory + When the engineer runs mine evaluate with the output_dir argument + Then the command reads artifacts from the specified directory + And evaluation results are written to the specified directory + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Engineer evaluates with verbose logging enabled + When the engineer runs mine evaluate with the verbose flag + Then additional diagnostic output is produced to stderr during evaluation + And the command completes evaluation normally + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario Outline: Engineer evaluates filtered surfaces + Given surfaces exist for packages "" and "" + When the engineer runs mine evaluate with surfaces filter "" + Then gate evaluation is restricted to surfaces matching "" + And non-matching surfaces are excluded from evaluation + + Examples: + | package_a | package_b | filter | + | api.auth | api.data | api.auth | + | core.util | core.main | core | + | service.a | service.b | service.a | + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Engineer evaluates coverage gate + Given surfaces exist with varying documentation coverage + And a coverage gate threshold is configured + When the engineer runs mine evaluate against the target + Then the coverage gate measures the percentage or count of surfaces meeting thresholds + And the gate passes or fails based on the configured threshold + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Engineer evaluates ambiguity gate + Given surfaces with potentially conflicting definitions exist + And an ambiguity gate is configured + When the engineer runs mine evaluate against the target + Then the ambiguity gate detects conflicting or overlapping surface definitions + And the gate passes or fails based on detected ambiguities + + @pickled-internal:core-llm-cache-default-on + Scenario: Engineer evaluates surfaces multiple times + Given mining and evaluation have been performed once + And gate results have been recorded + When the engineer runs mine evaluate again with identical inputs + Then the gate results are identical to the previous evaluation + And the command produces the same exit status + + # TODO: Clarify behavior when ruleset_dir is specified + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Engineer evaluates with custom ruleset directory + Given a custom ruleset directory exists + When the engineer runs mine evaluate with the ruleset_dir argument + Then gate behavior is configured from the specified ruleset directory + + # TODO: Clarify behavior when ruleset_config is specified + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Engineer evaluates with custom ruleset configuration + Given a custom ruleset configuration is provided + When the engineer runs mine evaluate with the ruleset_config argument + Then gate behavior is configured according to the specified ruleset configuration diff --git a/dogfood/mining-output/features/pickled_core_mine_features.feature b/dogfood/mining-output/features/pickled_core_mine_features.feature new file mode 100644 index 0000000..9ac336e --- /dev/null +++ b/dogfood/mining-output/features/pickled_core_mine_features.feature @@ -0,0 +1,79 @@ +Feature: Mine features from user stories + As a developer using pickled-core + I want to mine features from previously generated user stories + So that I can create testable feature specifications in stage 4 of the mining pipeline + + Background: + Given user stories have been generated in a previous mining stage + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: Developer mines features with required target argument + When the developer mines features for a target + Then features are drafted from the user stories + And the features are written to the default output directory + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: Developer mines features to a specific output directory + When the developer mines features for a target with a specified output directory + Then features are drafted from the user stories + And the features are written to the specified output directory + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: Developer mines features in quick mode + Given quick mode is enabled + When the developer mines features for a target + Then features are drafted without interactive prompts + And the process uses the default quick mode behavior + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: Developer mines features with interactive prompts + Given quick mode is disabled + When the developer mines features for a target + Then features are drafted using interactive prompts + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Developer mines features with verbose logging + Given verbose mode is enabled + When the developer mines features for a target + Then features are drafted from the user stories + And extra logging output is written to stderr + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer filters features by surface substring + When the developer mines features for a target with a surfaces filter + Then only features matching the package name or surface-id substring are drafted + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer controls parallel processing + Given quick mode is enabled + When the developer mines features for a target with a specified max parallel value + Then features are drafted with the specified maximum parallel LLM calls + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: Developer overwrites existing features + Given features already exist for the target + And the overwrite features flag is enabled + When the developer mines features for a target + Then the existing features are overwritten with newly drafted features + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: Developer runs mining as stage 4 of the pipeline + Given the mining pipeline is at stage 4 + When the developer mines features for a target + Then features are drafted according to ADR 0005 staged mining pipeline + And the output reflects stage 4 processing + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario Outline: Developer mines features with different flag combinations + Given quick mode is + And verbose mode is + When the developer mines features for a target + Then features are drafted with quick mode + And logging is + + Examples: + | quick_mode | verbose_mode | quick_behavior | logging_behavior | + | enabled | enabled | without prompts | verbose | + | enabled | disabled | without prompts | standard | + | disabled | enabled | with prompts | verbose | + | disabled | disabled | with prompts | standard | diff --git a/dogfood/mining-output/features/pickled_core_mine_inventory.feature b/dogfood/mining-output/features/pickled_core_mine_inventory.feature new file mode 100644 index 0000000..582fea6 --- /dev/null +++ b/dogfood/mining-output/features/pickled_core_mine_inventory.feature @@ -0,0 +1,77 @@ +Feature: Mine inventory from target project + As a developer or CI pipeline + I want to mine an inventory of specifications from a target project + So that I can catalog discovered features and scenarios for further processing + + Background: + Given the pickled-core mining pipeline is available + And I have access to the filesystem + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer successfully mines inventory from valid target + Given a valid target project exists at "examples/demo-app" + When I run the mine inventory command with target "examples/demo-app" + Then the command completes without error + And an "inventory.json" file is created in the default output directory + And the "inventory.json" file contains valid JSON + And the inventory contains introspected elements from the target + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer mines inventory to custom output directory + Given a valid target project exists at "examples/demo-app" + And an output directory "custom/output" is available + When I run the mine inventory command with target "examples/demo-app" and output_dir "custom/output" + Then the command completes without error + And an "inventory.json" file is created in "custom/output" + And the "inventory.json" file contains valid JSON + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer runs mine inventory with verbose logging + Given a valid target project exists at "examples/demo-app" + When I run the mine inventory command with target "examples/demo-app" and verbose flag enabled + Then the command completes without error + And an "inventory.json" file is created + And additional logging output is written to stderr + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer mines inventory without MCP tools + Given a valid target project exists at "examples/demo-app" + When I run the mine inventory command with target "examples/demo-app" and no_mcp flag enabled + Then the command completes without error + And an "inventory.json" file is created + And no MCP tools list operations were performed + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer runs mine inventory in quick mode + Given a valid target project exists at "examples/demo-app" + When I run the mine inventory command with target "examples/demo-app" and quick flag enabled + Then the command completes without error + And an "inventory.json" file is created + And no interactive prompts were displayed + + Scenario: Developer attempts to mine inventory without specifying target + When I run the mine inventory command without a target argument + Then the command fails with an error + And the error message indicates the target argument is required + + @best-practices:agent-path-first-class + Scenario: Developer attempts to mine inventory from invalid target + Given no project exists at "nonexistent/path" + When I run the mine inventory command with target "nonexistent/path" + Then the command fails with an error + And the error message indicates the target is invalid or not found + + # TODO: Specify behavior when mcp_timeout is provided and MCP operations exceed timeout + @pickled-internal:mcp-output-fixed-json-shape + Scenario Outline: Developer mines inventory with different valid targets + Given a valid target project exists at "" + When I run the mine inventory command with target "" + Then the command completes without error + And an "inventory.json" file is created + And the inventory contains introspected elements from the target + + Examples: + | target_path | + | examples/demo-app | + | src/myproject | + | /absolute/path/proj | diff --git a/dogfood/mining-output/features/pickled_core_mine_report.feature b/dogfood/mining-output/features/pickled_core_mine_report.feature new file mode 100644 index 0000000..7378dff --- /dev/null +++ b/dogfood/mining-output/features/pickled_core_mine_report.feature @@ -0,0 +1,64 @@ +Feature: Pipeline operator generates mining report + As a pipeline operator + I want to generate a consolidated mining report from pipeline outputs + So that I can review, hand off, or archive the analysis results + + Background: + Given the pickled pipeline has completed stages 1 through 6 + And mining artifacts exist in the output directory + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Operator generates report with valid target + When the operator runs "pickled-core mine report " + Then the command exits successfully + And a file named "mining-report.md" is created + And the report contains consolidated information from prior pipeline stages + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Operator generates report from custom output directory + Given mining artifacts are located in a custom directory + When the operator runs "pickled-core mine report --output_dir " + Then the command reads mining artifacts from the custom directory + And the command exits successfully + And a file named "mining-report.md" is created + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Operator generates report with verbose logging + When the operator runs "pickled-core mine report --verbose" + Then the command exits successfully + And additional logging messages are emitted to stderr + And a file named "mining-report.md" is created + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Operator filters report by surface criteria + Given multiple surfaces exist across different packages + When the operator runs "pickled-core mine report --surfaces " + Then the command exits successfully + And the generated report includes only surfaces matching the filter criteria + And surfaces not matching the filter are excluded from the report + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Operator runs report in interactive mode + When the operator runs "pickled-core mine report --quick=false" + Then the command prompts for interactive input + And the command exits successfully after input is provided + And a file named "mining-report.md" is created + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Operator omits required target argument + When the operator runs "pickled-core mine report" without a target argument + Then the command fails with an appropriate error message + And no report file is created + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario Outline: Operator combines multiple options + When the operator runs "pickled-core mine report " + Then the command exits successfully + And the behavior reflects all specified options + + Examples: + | options | + | --verbose --output_dir custom | + | --surfaces filter1,filter2 --verbose | + | --quick=false --verbose | + | --output_dir custom --surfaces filter | diff --git a/dogfood/mining-output/features/pickled_core_mine_stories.feature b/dogfood/mining-output/features/pickled_core_mine_stories.feature new file mode 100644 index 0000000..1d5277c --- /dev/null +++ b/dogfood/mining-output/features/pickled_core_mine_stories.feature @@ -0,0 +1,92 @@ +Feature: Mine stories from inventory + As a developer or CI pipeline + I want to generate human-readable story files from mined surface metadata + So that I can document behavior and prepare for feature-file generation + + Background: + Given an inventory file exists at "target/inventory.json" + + @bdd-domain:draft-empty-story-deterministic-failure + Scenario: Developer generates stories from inventory + When the developer runs mine stories for the target directory + Then story files are generated in the output directory structure + And each story corresponds to a surface from the inventory + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer filters surfaces by package name + Given the inventory contains surfaces from multiple packages + When the developer runs mine stories with surfaces filter "pickled-core" + Then only story files for surfaces in package "pickled-core" are generated + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer filters surfaces by surface-id substring + Given the inventory contains multiple surfaces + When the developer runs mine stories with surfaces filter "mine_stories" + Then only story files matching the surface-id substring are generated + + @data-domain:migration-drift-gate + Scenario: Developer runs in quick mode by default + When the developer runs mine stories without specifying quick mode + Then the command runs in batch mode without interactive prompts + + @data-domain:migration-drift-gate + Scenario: Developer runs in interactive mode + When the developer runs mine stories with quick mode disabled + Then the command prompts interactively for user input + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer limits concurrent LLM operations + When the developer runs mine stories with max_parallel set to 3 + Then no more than 3 LLM calls execute concurrently + + @bdd-domain:draft-empty-story-deterministic-failure + Scenario: Developer overwrites existing story files + Given story files already exist in the output directory + When the developer runs mine stories with overwrite_stories enabled + Then existing story files are replaced with newly generated content + + @bdd-domain:draft-empty-story-deterministic-failure + Scenario: Developer preserves existing story files + Given story files already exist in the output directory + When the developer runs mine stories with overwrite_stories disabled + Then existing story files are not replaced + + @best-practices:agent-path-first-class + Scenario: Developer specifies custom code-context directory + Given code-context files exist at "custom/path/code-context" + When the developer runs mine stories with code_context_dir set to "custom/path/code-context" + Then implementation details from the custom directory are integrated into stories + + @data-domain:migration-drift-gate + Scenario: Developer uses default code-context directory + Given code-context files exist at "output/code-context" + When the developer runs mine stories without specifying code_context_dir + Then implementation details from "output/code-context" are integrated into stories + + @bdd-domain:draft-empty-story-deterministic-failure + Scenario: Developer generates stories without code-context + Given no code-context directory exists + When the developer runs mine stories + Then story files are generated without implementation details + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Developer enables verbose logging + When the developer runs mine stories with verbose enabled + Then extra diagnostic output is emitted to stderr + + @best-practices:cli-mcp-surface-parity + Scenario: Developer generates code-aware stories for drift detection + Given code-context files exist for surfaces in the inventory + When the developer runs mine stories + Then generated stories integrate code-context details + And stories support docstring drift detection per ADR 0007 + + @best-practices:agent-path-first-class + Scenario Outline: Developer specifies output directory + When the developer runs mine stories with output_dir set to "" + Then story files are generated in "" + + Examples: + | output_path | + | custom/output | + | mining-results | diff --git a/dogfood/mining-output/features/pickled_core_mine_tag.feature b/dogfood/mining-output/features/pickled_core_mine_tag.feature new file mode 100644 index 0000000..cc825f8 --- /dev/null +++ b/dogfood/mining-output/features/pickled_core_mine_tag.feature @@ -0,0 +1,82 @@ +Feature: Mine tag command + As a developer or automation pipeline + I want to tag scenarios in generated feature files + So that I can categorize and filter test cases for downstream processing + + Background: + Given the pickled-core mining pipeline has completed prior stages + And feature files have been generated + + @bdd-domain:drafter-no-auto-tags + Scenario: Developer tags scenarios with valid target + Given a valid target is specified + When the mine tag command is executed + Then scenarios in the generated feature files are tagged + And the tagged features are written to the mining output directory + + @best-practices:agent-path-first-class + Scenario: Developer specifies custom output directory + Given a valid target is specified + And an output directory path is provided via "--output-dir" + When the mine tag command is executed + Then the tagged features are written to the specified output directory + + @bdd-domain:drafter-no-auto-tags + Scenario Outline: Developer controls interactive mode with quick flag + Given a valid target is specified + And the "--quick" flag is set to + When the mine tag command is executed + Then the command runs in mode + And + + Examples: + | quick_value | mode | interaction_behavior | + | true | quick | no interactive prompts are displayed | + | false | interactive | interactive prompts are enabled | + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Developer enables verbose logging + Given a valid target is specified + And the "--verbose" flag is enabled + When the mine tag command is executed + Then extra logging output is written to stderr + And scenarios are tagged successfully + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer filters surfaces by package name or surface ID + Given a valid target is specified + And the "--surfaces" parameter contains comma-separated filter substrings + When the mine tag command is executed + Then only surfaces matching the package name or surface ID substrings are processed + And matching surfaces have their scenarios tagged + + @bdd-domain:drafter-no-auto-tags + Scenario: Developer configures ruleset directory + Given a valid target is specified + And a ruleset directory path is provided via "--ruleset-dir" + When the mine tag command is executed + Then ruleset definitions are loaded from the specified directory + And scenarios are tagged according to the rulesets + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer configures ruleset configuration per ADR 0004 + Given a valid target is specified + And a ruleset configuration is provided via "--ruleset-config" + When the mine tag command is executed + Then the multi-ruleset workspace configuration is applied per ADR 0004 + And scenarios are tagged according to the configured rulesets + + @bdd-domain:drafter-no-auto-tags + Scenario: Developer omits required target parameter + Given the target parameter is not provided + When the mine tag command is executed + Then the command fails with an error + And an appropriate error message indicates the target parameter is required + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: Command operates as stage 5 in mining pipeline + Given stages 1 through 4 of the mining pipeline have completed + And feature files exist from prior stages + When the mine tag command is executed as stage 5 + Then scenarios are tagged in the existing feature files + And the tagged output is ready for downstream processing diff --git a/dogfood/mining-output/features/pickled_data_apply.feature b/dogfood/mining-output/features/pickled_data_apply.feature new file mode 100644 index 0000000..5965c24 --- /dev/null +++ b/dogfood/mining-output/features/pickled_data_apply.feature @@ -0,0 +1,165 @@ +Feature: Apply migration to in-memory SQLite database + As a developer or automated tool + I want to validate SQL migration files in a sandbox environment + So that I can verify migrations will execute successfully before applying them to production + + Background: + Given an in-memory SQLite database is available + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer applies a valid migration file + Given a migration file "001_create_users.sql" containing valid SQL + And the dialect is "postgres" + When the developer applies the migration + Then the migration executes successfully + And JSON output is printed to stdout + And the JSON contains a "tables" array + And the JSON is indented with 2 spaces + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer applies migration creating tables with columns + Given a migration file "002_schema.sql" creating table "users" with columns: + | name | type | nullable | + | id | INTEGER | false | + | email | VARCHAR | false | + | nickname | TEXT | true | + And the dialect is "postgres" + When the developer applies the migration + Then the JSON output contains a table object with name "users" + And the table object contains a "columns" array + And the columns array contains a column with name "id", type "INTEGER", and nullable false + And the columns array contains a column with name "email", type "VARCHAR", and nullable false + And the columns array contains a column with name "nickname", type "TEXT", and nullable true + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer applies migration with column types needing uppercasing + Given a migration file "003_types.sql" creating table "items" with column "status" of type "varchar" + And the dialect is "postgres" + When the developer applies the migration + Then the JSON output contains a column with type "VARCHAR" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer applies migration creating column with null type + Given a migration file "004_null_type.sql" creating a column with null type reported by database + And the dialect is "postgres" + When the developer applies the migration + Then the JSON output contains a column with type "TEXT" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: System excludes SQLite internal tables from output + Given a migration file "005_tables.sql" creating table "products" + And the dialect is "postgres" + And SQLite system tables exist with names starting with "sqlite_" + When the developer applies the migration + Then the JSON output contains a table with name "products" + And the JSON output does not contain any table with name starting with "sqlite_" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer attempts to apply a dbt migration file + Given a migration file "001_model.dbt" containing valid SQL + And the dialect is "postgres" + When the developer applies the migration + Then a NotImplementedError is raised + And the error message indicates dbt is not implemented + And no SQL processing occurs + + @data-domain:migration-drift-gate + Scenario: Developer applies migration with top-level ATTACH statement + Given a migration file "malicious.sql" containing a top-level ATTACH statement + And the dialect is "postgres" + When the developer applies the migration + Then an UnsafeMigrationStatementError is raised + And no database connection is created + + @data-domain:migration-drift-gate + Scenario: Developer applies migration with nested ATTACH statement + Given a migration file "nested_attach.sql" containing a nested ATTACH statement + And the dialect is "postgres" + When the developer applies the migration + Then an UnsafeMigrationStatementError is raised + And no database connection is created + + @data-domain:migration-drift-gate + Scenario: Developer applies migration with top-level DETACH statement + Given a migration file "detach.sql" containing a top-level DETACH statement + And the dialect is "postgres" + When the developer applies the migration + Then an UnsafeMigrationStatementError is raised + And no database connection is created + + @data-domain:migration-drift-gate + Scenario: Developer applies migration with nested DETACH statement + Given a migration file "nested_detach.sql" containing a nested DETACH statement + And the dialect is "postgres" + When the developer applies the migration + Then an UnsafeMigrationStatementError is raised + And no database connection is created + + @data-domain:migration-drift-gate + Scenario: Developer applies migration with unparseable SQL + Given a migration file "invalid.sql" containing SQL that cannot be parsed + And the dialect is "postgres" + When the developer applies the migration + Then a SQLParseError is raised + + @bdd-domain:draft-empty-story-deterministic-failure + Scenario: Developer applies an empty migration file + Given a migration file "empty.sql" that is empty + And the dialect is "postgres" + When the developer applies the migration + Then a SQLParseError is raised + + @data-domain:migration-drift-gate + Scenario: Developer applies migration file with no parse result + Given a migration file "no_result.sql" that produces no parse result + And the dialect is "postgres" + When the developer applies the migration + Then a SQLParseError is raised + + @bdd-domain:draft-empty-story-deterministic-failure + Scenario: System handles statement execution failure gracefully + Given a migration file "failing.sql" with SQL that fails during execution + And the dialect is "postgres" + When the developer applies the migration + Then the statement execution fails + And the database connection is closed + + @data-domain:migration-drift-gate + Scenario: Developer applies migration file with invalid UTF-8 encoding + Given a migration file "bad_encoding.sql" that is not valid UTF-8 + And the dialect is "postgres" + When the developer applies the migration + Then an encoding error is raised + + @data-domain:migration-drift-gate + Scenario: System transpiles SQL from source dialect to SQLite + Given a migration file "postgres_specific.sql" containing PostgreSQL-specific SQL + And the dialect is "postgres" + When the developer applies the migration + Then the SQL is transpiled from postgres dialect to SQLite dialect + And the transpiled SQL is executed in SQLite + + @data-domain:migration-drift-gate + Scenario: System creates in-memory database not file-based + Given a migration file "006_memory_test.sql" containing valid SQL + And the dialect is "postgres" + When the developer applies the migration + Then an in-memory SQLite database is used + And no database file is created on the filesystem + + @best-practices:llm-drafter-temperature-zero + Scenario: System attempts to set attached database limit to zero + Given a migration file "007_limit_test.sql" containing valid SQL + And the dialect is "postgres" + When the developer applies the migration + Then the system attempts to set SQLite attached database limit to 0 + And execution continues successfully regardless of AttributeError + + @data-domain:migration-drift-gate + Scenario: System commits transaction after successful execution + Given a migration file "008_transaction.sql" containing multiple valid statements + And the dialect is "postgres" + When the developer applies the migration + Then all statements are executed + And the transaction is committed + And the schema changes are persisted in memory diff --git a/dogfood/mining-output/features/pickled_data_check_drift.feature b/dogfood/mining-output/features/pickled_data_check_drift.feature new file mode 100644 index 0000000..e9db60a --- /dev/null +++ b/dogfood/mining-output/features/pickled_data_check_drift.feature @@ -0,0 +1,66 @@ +Feature: Check migration drift against expected schema + As a developer or CI/CD pipeline + I want to validate that a migration produces the expected database schema + So that I can detect drift between documented expectations and actual migration behavior + + @data-domain:migration-drift-gate + Scenario: Developer validates migration against expected schema with drift detected + Given a migration script "001_create_users_table" + And an expected schema YAML file "expected_users_schema.yaml" + When the developer runs check-drift with the migration and expected schema + Then MigrationDriftGate is executed with the migration and expected schema + And drift is reported between the migration output and expected schema + And the command exits with a failure status code + + @data-domain:migration-drift-gate + Scenario: Developer validates migration against expected schema without drift + Given a migration script "002_create_orders_table" + And an expected schema YAML file "expected_orders_schema.yaml" + And the migration produces a schema matching the expected schema + When the developer runs check-drift with the migration and expected schema + Then MigrationDriftGate is executed with the migration and expected schema + And validation success is reported + And the command exits with a success status code + + @data-domain:migration-drift-gate + Scenario: Developer validates migration with specific database dialect + Given a migration script "003_create_products_table" + And an expected schema YAML file "expected_products_schema.yaml" + And a database dialect "postgresql" + When the developer runs check-drift with the migration, expected schema, and dialect + Then MigrationDriftGate is executed with the migration, expected schema, and dialect + And validation results are reported + And the command exits with an appropriate status code + + @data-domain:migration-drift-gate + Scenario: CI/CD pipeline validates migration without optional dialect parameter + Given a migration script "004_create_inventory_table" + And an expected schema YAML file "expected_inventory_schema.yaml" + When the CI/CD pipeline runs check-drift with only the required parameters + Then MigrationDriftGate is executed with the migration and expected schema + And validation results are reported + And the command exits with an appropriate status code + + @data-domain:migration-drift-gate + Scenario: Developer attempts to run check-drift without required migration argument + Given an expected schema YAML file "expected_schema.yaml" + When the developer runs check-drift without the migration argument + Then the command reports a missing required argument error + And the command exits with a failure status code + + @data-domain:migration-drift-gate + Scenario: Developer attempts to run check-drift without required expected schema argument + Given a migration script "005_create_customers_table" + When the developer runs check-drift without the expected schema argument + Then the command reports a missing required argument error + And the command exits with a failure status code + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer invokes check-drift from command line as part of pickled-data CLI + Given a migration script "006_create_payments_table" + And an expected schema YAML file "expected_payments_schema.yaml" + When the developer invokes "pickled-data check-drift" from the command line + Then the command executes successfully as part of the pickled-data CLI + And MigrationDriftGate is executed with the provided parameters + And validation results are reported + And the command exits with an appropriate status code diff --git a/dogfood/mining-output/features/pickled_data_datacontractgate.feature b/dogfood/mining-output/features/pickled_data_datacontractgate.feature new file mode 100644 index 0000000..b081008 --- /dev/null +++ b/dogfood/mining-output/features/pickled_data_datacontractgate.feature @@ -0,0 +1,98 @@ +Feature: DataContractGate validates SQL query columns against OpenAPI response properties + + Background: + Given a DataContractGate instance + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Target is not a string + When the gate runs with a non-string target + Then the verdict is FAIL + And the notes describe the actual type received + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Context is missing endpoint_tag key + When the gate runs with a string target and context without "endpoint_tag" + Then the verdict is FAIL + And the notes indicate the missing endpoint_tag + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Context endpoint_tag value is not a string + When the gate runs with a string target and endpoint_tag that is not a string + Then the verdict is FAIL + And the notes indicate the endpoint_tag must be a string + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: No SchemaRegistry is configured + Given the gate has no SchemaRegistry configured + When the gate runs with a valid string target and valid endpoint_tag + Then the verdict is WARN + And the notes state "no SchemaRegistry configured" + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Schema artifact not found for endpoint_tag + Given the gate has a SchemaRegistry configured + And the SchemaRegistry cannot find a schema for the given endpoint_tag + When the gate runs with a valid string target and valid endpoint_tag + Then the verdict is WARN + And the notes include the endpoint_tag name + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: OpenAPI schema has no extractable properties + Given the gate has a SchemaRegistry configured + And the SchemaRegistry returns a schema artifact + And the schema artifact has no extractable properties for 200 or 201 responses + When the gate runs with a valid SQL query and valid endpoint_tag + Then the verdict is WARN + And the notes indicate no properties were found + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: SQL parsing fails + Given the gate has a SchemaRegistry configured + And the SchemaRegistry returns a schema artifact with properties + When the gate runs with an unparseable SQL string and valid endpoint_tag + Then the gate treats the SQL as having zero columns + And the validation continues with an empty column set + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: SQL column names exactly match API property names + Given the gate has a SchemaRegistry configured + And the SchemaRegistry returns a schema artifact with properties + When the gate runs with a SQL query whose column names exactly match the API properties + Then the verdict is PASS + And the notes state that types are not checked in v0.1 + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario Outline: SQL columns differ from API properties + Given the gate has a SchemaRegistry configured + And the SchemaRegistry returns a schema artifact with properties + When the gate runs with a SQL query having columns + Then the verdict is FAIL + And the notes include a sorted list of missing columns + And the notes include a sorted list of extra columns + + Examples: + | sql_columns | api_properties | + | id, name | id, name, email | + | id, name, status | id, name | + | user_id | id | + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: OpenAPI property extraction examines only 200 or 201 response codes + Given the gate has a SchemaRegistry configured + And the SchemaRegistry returns a schema artifact with multiple response codes + When the gate extracts properties from the OpenAPI schema + Then only properties from 200 or 201 responses are considered + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: OpenAPI property extraction returns names from first matching operation + Given the gate has a SchemaRegistry configured + And the SchemaRegistry returns a schema artifact with multiple operations + When the gate extracts properties from the OpenAPI schema + Then properties are taken from the first matching operation found + + @pickled-internal:core-llm-cache-default-on + Scenario: GateResult includes gate name and notes + Given the gate has a name + When the gate runs with any valid inputs + Then the GateResult includes the gate's name + And the GateResult includes descriptive notes diff --git a/dogfood/mining-output/features/pickled_data_draft.feature b/dogfood/mining-output/features/pickled_data_draft.feature new file mode 100644 index 0000000..fb9ad6c --- /dev/null +++ b/dogfood/mining-output/features/pickled_data_draft.feature @@ -0,0 +1,151 @@ +Feature: Draft SQL migration from natural language intent + As a developer or automation pipeline + I want to generate SQL migration files from natural-language descriptions + So that I can create schema changes without manually writing DDL + + Background: + Given the LLM client is configured via environment variable "PICKLED_DATA_LLM_FACTORY" + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer reads intent from standard input + Given the intent is provided as "-" + And standard input contains "add user email column" + When the draft command is invoked + Then the LLM prompt is built with the stdin content + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer reads intent from a file + Given the intent is provided as a file path "intent.txt" + And the file "intent.txt" contains "add user email column" + When the draft command is invoked + Then the file content is read and used in the LLM prompt + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer drafts migration without current schema context + Given the intent is "add user email column" + And the current_schema parameter is None + When the draft command is invoked + Then no schema YAML file is read + And the LLM prompt indicates "none" for schema context + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer drafts migration with current schema context + Given the intent is "add user email column" + And the current_schema parameter is "schema.yaml" + And the file "schema.yaml" contains valid YAML schema definition + When the draft command is invoked + Then the file "schema.yaml" is read as UTF-8 + And the schema content is included in the LLM prompt + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer specifies SQL dialect for migration + Given the intent is "add user email column" + And the dialect is "postgres" + When the draft command is invoked + Then the dialect "postgres" is passed to the LLM prompt + And the dialect "postgres" is used for sqlglot parse validation + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer writes SQL to standard output + Given the intent is "add user email column" + And the output parameter is None + And the LLM generates valid SQL migration text + When the draft command is invoked + Then the SQL text is written to standard output + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer writes SQL to a file + Given the intent is "add user email column" + And the output parameter is "migration.sql" + And the LLM generates valid SQL migration text + When the draft command is invoked + Then the SQL text is written to "migration.sql" as UTF-8 + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer receives rationale with SQL migration + Given the intent is "add user email column" + And the LLM output contains the rationale sentinel + When the draft command is invoked + Then the SQL and rationale are separated at the sentinel + And each rationale line is written to stderr with "rationale: " prefix + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer receives SQL without rationale + Given the intent is "add user email column" + And the LLM output lacks the rationale sentinel + When the draft command is invoked + Then all output is treated as SQL + And no rationale is emitted to stderr + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer receives warning for unparseable SQL + Given the intent is "add user email column" + And the LLM generates SQL that fails sqlglot parsing + When the draft command is invoked + Then a warning is written to stderr with "warning: " prefix + And the warning contains the parse exception message + + @pickled-internal:core-llm-cache-default-on + Scenario Outline: Developer receives warning for destructive operations + Given the intent is "restructure database" + And the LLM generates SQL containing "" + When the draft command is invoked + Then a warning is written to stderr identifying the line number + And the warning advises confirmation before applying + + Examples: + | drop_statement | + | DROP TABLE users | + | drop table users | + | Drop Table users | + | DROP table users | + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Developer completes draft with validation warnings + Given the intent is "add user email column" + And the LLM generates SQL with validation warnings + When the draft command is invoked + Then the SQL is emitted to the configured output + And warnings are written to stderr + And the process exits with code 1 + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Developer completes draft without validation warnings + Given the intent is "add user email column" + And the LLM generates valid SQL without warnings + When the draft command is invoked + Then the SQL is emitted to the configured output + And the process exits with code 0 + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: LLM client configuration fails + Given the LLM client configuration raises a ConfigError with message "Invalid factory" + When the draft command is invoked + Then a ClickException is raised with message "Invalid factory" + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Unexpected error during draft generation + Given the intent is "add user email column" + And an unexpected exception occurs with message "Network timeout" + When the draft command is invoked + Then the exception message "Network timeout" is written to stderr + And the process exits with code 2 + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Current schema file does not exist + Given the intent is "add user email column" + And the current_schema parameter is "nonexistent.yaml" + And the file "nonexistent.yaml" does not exist + When the draft command is invoked + Then an exception is caught + And the exception message is written to stderr + And the process exits with code 2 + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Intent file does not exist + Given the intent is provided as a file path "nonexistent.txt" + And the file "nonexistent.txt" does not exist + When the draft command is invoked + Then an exception is caught + And the exception message is written to stderr + And the process exits with code 2 diff --git a/dogfood/mining-output/features/pickled_data_mcp.feature b/dogfood/mining-output/features/pickled_data_mcp.feature new file mode 100644 index 0000000..26e1f21 --- /dev/null +++ b/dogfood/mining-output/features/pickled_data_mcp.feature @@ -0,0 +1,35 @@ +Feature: MCP CLI Command Entry Point + As a CLI user + I want to invoke the mcp command + So that I can access MCP server functionality through the command group + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User invokes mcp command without arguments + When the mcp command is invoked with no arguments + Then it completes without raising exceptions + And it returns None + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User attempts to invoke mcp command with arguments + When the mcp command is invoked with arguments + Then it raises a TypeError + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User invokes mcp command and checks for side effects + When the mcp command is invoked with no arguments + Then no file I/O operations occur + And no network calls are made + And no output is written to stdout + And no output is written to stderr + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User imports and invokes mcp as a standalone function + When the mcp function is imported from the module + And the function is invoked directly + Then it executes successfully + And it returns None + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User inspects mcp command documentation + When the mcp function docstring is retrieved + Then it contains the string "MCP server commands." diff --git a/dogfood/mining-output/features/pickled_data_migrationdriftgate.feature b/dogfood/mining-output/features/pickled_data_migrationdriftgate.feature new file mode 100644 index 0000000..d776c95 --- /dev/null +++ b/dogfood/mining-output/features/pickled_data_migrationdriftgate.feature @@ -0,0 +1,145 @@ +Feature: Migration Drift Gate Validation + As a migration validation pipeline + I want to verify SQL migrations produce expected database schemas + So that I can detect drift before deployment + + Background: + Given the MigrationDriftGate is initialized + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Developer provides non-string target + Given the target is an integer value 42 + When the gate runs + Then the verdict is FAIL + And the notes contain the type name "int" + + @data-domain:migration-drift-gate + Scenario: Developer provides context without expected schema + Given the target is a valid SQL migration string + And the context does not contain "expected_schema" + And the context does not contain "expected_schema_yaml" + When the gate runs + Then the verdict is FAIL + And the notes describe missing schema context + + @data-domain:migration-drift-gate + Scenario: Developer provides expected schema as a dictionary + Given the target is a valid SQL migration string + And the context contains "expected_schema" as a dictionary with table definitions + When the gate runs + Then the expected schema is used directly without YAML parsing + + @data-domain:migration-drift-gate + Scenario: Developer provides expected schema as YAML string + Given the target is a valid SQL migration string + And the context contains "expected_schema_yaml" as a valid YAML string + When the gate runs + Then the YAML is parsed via yaml.safe_load + And the parsed dictionary is used as the expected schema + + @data-domain:migration-drift-gate + Scenario: Developer provides YAML that parses to non-dictionary + Given the target is a valid SQL migration string + And the context contains "expected_schema_yaml" that parses to a list + When the gate runs + Then the verdict is FAIL + And the notes describe invalid context + + @data-domain:migration-drift-gate + Scenario: Migration produces schema with different tables than expected + Given the target is SQL creating tables "users" and "orders" + And the expected schema defines tables "users" and "products" + When the gate runs + Then the verdict is FAIL + And the notes list expected table names + And the notes list actual table names + + @data-domain:migration-drift-gate + Scenario: Migration produces table with different columns than expected + Given the target is SQL creating table "users" with columns "id" and "name" + And the expected schema defines table "users" with columns "id" and "email" + When the gate runs + Then the verdict is FAIL + And the notes identify the table "users" + And the notes list expected column set + And the notes list actual column set + + @data-domain:migration-drift-gate + Scenario: Migration produces schema matching expected exactly + Given the target is SQL creating table "users" with columns matching expected types and nullability + And the expected schema defines the same structure + When the gate runs + Then the verdict is PASS + And the notes are "Schema matches expected." + + @data-domain:migration-drift-gate + Scenario: Migration produces schema with only nullable differences + Given the target is SQL creating table "users" with column "name" as nullable + And the expected schema defines column "name" as not nullable + When the gate runs + Then the verdict is PASS + And the notes describe nullable differences + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: SQL dialect is specified in context + Given the target is a SQL string in MySQL dialect + And the context contains "dialect" with value "mysql" + When the gate runs + Then the SQL is parsed using MySQL dialect + And the SQL is transpiled to SQLite dialect for execution + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: SQL dialect defaults to postgres when not specified + Given the target is a SQL string + And the context does not contain "dialect" + When the gate runs + Then the SQL is parsed using postgres dialect + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Column types are normalized to uppercase during comparison + Given the target is SQL creating column "id" with type "integer" + And the expected schema defines column "id" with type "INTEGER" + When the gate runs + Then the column types are normalized to uppercase before comparison + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Nullable defaults to true when not specified + Given the target is SQL creating column "name" without explicit nullable constraint + And the expected schema defines column "name" without explicit nullable + When the gate runs + Then nullable is treated as True for both schemas + + @data-domain:migration-drift-gate + Scenario: SQLite connection is closed after successful execution + Given the target is a valid SQL migration string + And the expected schema matches the migration output + When the gate runs + Then the SQLite connection is closed + + @data-domain:migration-drift-gate + Scenario: SQLite connection is closed after failed execution + Given the target is a valid SQL migration string + And the migration fails during execution + When the gate runs + Then the SQLite connection is closed in the finally block + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Empty SQL statements are skipped during transpilation + Given the target contains empty statements between valid SQL + And the expected schema matches the valid SQL output + When the gate runs + Then empty statements are skipped + And only valid statements are transpiled and executed + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: None statements from parsing are skipped + Given the SQL parser returns None for certain statements + And the target contains valid SQL statements + When the gate runs + Then None statements are skipped during transpilation + + @data-domain:migration-drift-gate + Scenario: SQLite attachment limit is set to prevent ATTACH statements + Given the target is a SQL migration string + When the gate runs + Then the SQLite connection attachment limit is set to 0 diff --git a/dogfood/mining-output/features/pickled_data_parse.feature b/dogfood/mining-output/features/pickled_data_parse.feature new file mode 100644 index 0000000..3c3adf0 --- /dev/null +++ b/dogfood/mining-output/features/pickled_data_parse.feature @@ -0,0 +1,76 @@ +Feature: Parse migration SQL files into AST summaries + + As a developer or CI/CD tool + I want to parse migration SQL files and view their structure as JSON + So that I can inspect, validate, and debug migrations before execution + + Background: + Given the parse command is available in the pickled-data CLI + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer parses a valid PostgreSQL migration file + Given a migration file "001_create_users.sql" with valid PostgreSQL DDL + When the developer parses the file without specifying a dialect + Then the command reads the file using UTF-8 encoding + And the command parses the SQL using "postgres" dialect + And the command outputs valid JSON to stdout + And the JSON contains the key "dialect" with value "postgres" + And the JSON contains the key "kind" with the AST root node class name + And the JSON contains the key "sql" with the SQL rendered in "postgres" dialect + And the JSON is indented with 2 spaces + And the command returns None + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer parses a migration file with an explicit dialect + Given a migration file "002_add_index.sql" with valid SQL + When the developer parses the file specifying dialect "mysql" + Then the command parses the SQL using "mysql" dialect + And the JSON output contains the key "dialect" with value "mysql" + And the JSON contains the key "sql" with the SQL rendered in "postgres" dialect + + @rules-domain:coverage-union-across-features + Scenario: Developer attempts to parse a DBT file + Given a file "transform.sql.dbt" exists + When the developer attempts to parse the file + Then the command raises NotImplementedError before reading the file + And the error message indicates DBT files are not supported + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: Developer parses a file with unparseable SQL + Given a migration file "invalid.sql" with malformed SQL that sqlglot cannot parse + When the developer attempts to parse the file + Then the command raises SQLParseError + And the error message contains details from the original sqlglot ParseError + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer parses a file that results in empty parse output + Given a migration file "empty_parse.sql" that causes sqlglot to return None + When the developer attempts to parse the file + Then the command raises SQLParseError with message "empty parse result" + + @best-practices:agent-path-first-class + Scenario Outline: File I/O errors propagate without wrapping + Given a migration file path "" + And the file condition is + When the developer attempts to parse the file + Then the command propagates without wrapping + + Examples: + | file_path | condition | exception_type | + | nonexistent.sql | does not exist | FileNotFoundError | + | restricted.sql | no read permission| PermissionError | + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Parse command output structure validation + Given a migration file "003_alter_table.sql" with valid SQL + When the developer parses the file + Then the JSON output contains exactly three keys: "dialect", "kind", and "sql" + And all three keys have non-empty string values + And the output is written to standard output via click.echo + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: SQL output is always rendered in postgres dialect + Given a migration file "004_select.sql" with valid SQL + When the developer parses the file specifying dialect "snowflake" + Then the JSON key "dialect" contains "snowflake" + But the JSON key "sql" contains SQL rendered in "postgres" dialect diff --git a/dogfood/mining-output/features/pickled_data_run_all.feature b/dogfood/mining-output/features/pickled_data_run_all.feature new file mode 100644 index 0000000..f6210ac --- /dev/null +++ b/dogfood/mining-output/features/pickled_data_run_all.feature @@ -0,0 +1,128 @@ +Feature: Data project migration quality checks + As a data quality engineer + I want to validate SQL migrations and schema drift in my data project + So that I can ensure migrations are parseable and produce the expected schema + + Background: + Given a data project working directory + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Project with no migration files returns a warning + Given the migrations directory is empty + When the migration quality checks run + Then exactly one gate result is returned + And the gate result has verdict "WARN" + And the gate result has gate_name "data.migrations" + And the gate result notes indicate no migrations were found + + @data-domain:migration-drift-gate + Scenario: Project with no expected schema file skips drift check + Given migration file "001_create_table.sql" exists and is valid SQL + And the expected_schema.yaml file does not exist + When the migration quality checks run + Then the results do not include a gate result with gate_name "data.migration_drift" + + @data-domain:migration-drift-gate + Scenario: Project with expected schema as a non-file skips drift check + Given migration file "001_create_table.sql" exists and is valid SQL + And expected_schema.yaml exists but is a directory + When the migration quality checks run + Then the results do not include a gate result with gate_name "data.migration_drift" + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Migration file with parse error returns a failure + Given migration file "002_bad_syntax.sql" contains unparseable SQL + When the migration quality checks run + Then a gate result with gate_name "data.parse.002_bad_syntax.sql" has verdict "FAIL" + And that gate result notes contain the parse error message + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Migration file with valid SQL returns a pass + Given migration file "003_valid.sql" contains valid SQLite SQL + When the migration quality checks run + Then a gate result with gate_name "data.parse.003_valid.sql" has verdict "PASS" + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Project with exactly one migration does not warn about multiple migrations + Given migration file "001_single.sql" exists and is valid SQL + When the migration quality checks run + Then the results do not include a gate result with gate_name "data.migrations.note" + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Project with multiple migrations warns about application order + Given migration file "001_first.sql" exists and is valid SQL + And migration file "002_second.sql" exists and is valid SQL + When the migration quality checks run + Then a gate result with gate_name "data.migrations.note" has verdict "WARN" + And that gate result notes explain migrations will be applied in filename order + + @data-domain:migration-drift-gate + Scenario: Project with migrations and valid expected schema runs drift check + Given migration file "001_create.sql" exists and is valid SQL + And expected_schema.yaml exists and contains a valid dictionary + When the migration quality checks run + Then a gate result with gate_name "data.migration_drift" is included + + @data-domain:migration-drift-gate + Scenario: All SQL parsing uses SQLite dialect + Given migration file "001_migration.sql" contains SQL + When the migration quality checks run + Then the SQL is parsed using "sqlite" dialect + And the drift check receives dialect "sqlite" in context + + @best-practices:agent-path-first-class + Scenario Outline: Migration files are processed in lexicographic order + Given migration file "" exists and is valid SQL + And migration file "" exists and is valid SQL + And migration file "" exists and is valid SQL + When the migration quality checks run + Then migrations are processed in order "", "", "" + + Examples: + | first | second | third | + | 001_a.sql | 002_b.sql | 003_c.sql | + | 1_early.sql | 10_late.sql | 2_middle.sql| + + @data-domain:migration-drift-gate + Scenario: Combined SQL for drift check concatenates with double newlines + Given migration file "001_first.sql" contains "CREATE TABLE a;" + And migration file "002_second.sql" contains "CREATE TABLE b;" + And expected_schema.yaml exists and contains a valid dictionary + When the migration quality checks run + Then the drift gate receives combined SQL "CREATE TABLE a;\n\nCREATE TABLE b;" + + @data-domain:migration-drift-gate + Scenario: YAML file with non-dict content skips drift check without error + Given migration file "001_create.sql" exists and is valid SQL + And expected_schema.yaml exists and contains a list + When the migration quality checks run + Then the results do not include a gate result with gate_name "data.migration_drift" + And no exception is raised + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario Outline: Multiple migration files produce multiple parse results + Given migration file "001_first.sql" is + And migration file "002_second.sql" is + And migration file "003_third.sql" is + When the migration quality checks run + Then a gate result with gate_name "data.parse.001_first.sql" has verdict "" + And a gate result with gate_name "data.parse.002_second.sql" has verdict "" + And a gate result with gate_name "data.parse.003_third.sql" has verdict "" + + Examples: + | first_status | second_status | third_status | first_verdict | second_verdict | third_verdict | + | valid SQL | valid SQL | valid SQL | PASS | PASS | PASS | + | valid SQL | invalid SQL | valid SQL | PASS | FAIL | PASS | + | invalid SQL | invalid SQL | invalid SQL | FAIL | FAIL | FAIL | + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Working directory is resolved to absolute path + Given a relative path to the working directory + When the migration quality checks run + Then the working directory is resolved to an absolute path before processing + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Function always returns at least one gate result + Given any valid working directory configuration + When the migration quality checks run + Then at least one gate result is returned diff --git a/dogfood/mining-output/features/pickled_diff_draft_corpus.feature b/dogfood/mining-output/features/pickled_diff_draft_corpus.feature new file mode 100644 index 0000000..1165e02 --- /dev/null +++ b/dogfood/mining-output/features/pickled_diff_draft_corpus.feature @@ -0,0 +1,74 @@ +Feature: Draft corpus generation from seed examples + + As a developer or test engineer + I want to generate a larger differential test corpus from seed examples + So that I can create comprehensive test datasets without manual tedium + + Background: + Given the pickled-diff CLI is available + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer generates corpus from seeds file with specified target size + Given a seeds file "seeds.json" containing 3 seed items + When the developer runs draft-corpus with seeds "seeds.json" and target size 10 + Then the command exits with status 0 + And the output is valid JSON + And the corpus contains exactly 10 items + And the corpus includes items derived from the original seeds + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer reads seed data from stdin + Given seed data is available on stdin containing 2 seed items + When the developer runs draft-corpus with seeds "-" and target size 8 + Then the command exits with status 0 + And the output is valid JSON + And the corpus contains exactly 8 items + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer writes corpus to stdout by default + Given a seeds file "seeds.json" containing 3 seed items + When the developer runs draft-corpus with seeds "seeds.json" and target size 5 without specifying output + Then the command exits with status 0 + And the corpus JSON is written to stdout + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer writes corpus to specified output file + Given a seeds file "seeds.json" containing 3 seed items + When the developer runs draft-corpus with seeds "seeds.json" and target size 5 and output "corpus.json" + Then the command exits with status 0 + And the corpus JSON is written to file "corpus.json" + And the file "corpus.json" contains valid JSON + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer provides notes from stdin + Given a seeds file "seeds.json" containing 3 seed items + And notes data is available on stdin + When the developer runs draft-corpus with seeds "seeds.json", target size 7, and notes "-" + Then the command exits with status 0 + And the output is valid JSON + And the corpus contains exactly 7 items + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer provides notes from file + Given a seeds file "seeds.json" containing 3 seed items + And a notes file "notes.txt" exists + When the developer runs draft-corpus with seeds "seeds.json", target size 7, and notes "notes.txt" + Then the command exits with status 0 + And the output is valid JSON + And the corpus contains exactly 7 items + + # TODO: Clarify expected behavior - should command return only seeds, error, or duplicate seeds? + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer requests target size equal to number of seeds + Given a seeds file "seeds.json" containing 5 seed items + When the developer runs draft-corpus with seeds "seeds.json" and target size 5 + Then the command exits with status 0 + And the corpus contains exactly 5 items + + # TODO: Clarify expected behavior - should command return subset, error, or all seeds? + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer requests target size less than number of seeds + Given a seeds file "seeds.json" containing 10 seed items + When the developer runs draft-corpus with seeds "seeds.json" and target size 3 + Then the command handles the condition appropriately + And the command exits with an appropriate status code diff --git a/dogfood/mining-output/features/pickled_diff_mcp.feature b/dogfood/mining-output/features/pickled_diff_mcp.feature new file mode 100644 index 0000000..951cfb1 --- /dev/null +++ b/dogfood/mining-output/features/pickled_diff_mcp.feature @@ -0,0 +1,27 @@ +Feature: MCP CLI Command + As a user of the pickled-diff CLI + I want to invoke the MCP server command + So that I can interact with MCP server functionality + + # TODO: Docstring indicates "MCP server commands" (plural) but implementation is empty - clarify intended command structure + # TODO: Define what MCP server operations should be exposed when implementation is added + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User invokes mcp command with no arguments + When the mcp command is invoked with no arguments + Then the command completes without error + And the command returns None + And no exceptions are raised + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User invokes mcp command verifying parameter requirements + When the mcp command is invoked + Then the command accepts exactly zero parameters + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User invokes mcp command with no side effects + Given the system state before command execution + When the mcp command is invoked with no arguments + Then no I/O operations are performed + And no state changes occur + And no external calls are made diff --git a/dogfood/mining-output/features/pickled_diff_run_all.feature b/dogfood/mining-output/features/pickled_diff_run_all.feature new file mode 100644 index 0000000..64b6a39 --- /dev/null +++ b/dogfood/mining-output/features/pickled_diff_run_all.feature @@ -0,0 +1,130 @@ +Feature: Differential testing gate entry point + As a pickled gate runner + I want to execute differential testing comparing candidate and oracle implementations + So that I can verify behavioral equivalence across a test corpus + + Background: + Given a working directory + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Configuration file is missing + Given no "pickled.diff.yaml" configuration file exists + When the run_all gate is invoked + Then a list with exactly one GateResult is returned + And the GateResult has gate_name "diff.config" + And the GateResult has verdict WARN + And the GateResult notes mention the missing configuration file + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Configuration file has non-string corpus value + Given a "pickled.diff.yaml" configuration file exists + And the corpus key is not a string + When the run_all gate is invoked + Then a list with exactly one GateResult is returned + And the GateResult has gate_name "diff.config" + And the GateResult has verdict FAIL + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario Outline: Configuration file has invalid command format + Given a "pickled.diff.yaml" configuration file exists + And the is not a list + When the run_all gate is invoked + Then a list with exactly one GateResult is returned + And the GateResult has gate_name "diff.config" + And the GateResult has verdict FAIL + + Examples: + | command_key | + | oracle_command | + | candidate_command | + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Configuration file is valid + Given a "pickled.diff.yaml" configuration file exists + And the oracle_command is a valid list + And the candidate_command is a valid list + And the corpus is a valid string path + And a corpus file exists at the specified path + When the run_all gate is invoked + Then a list with exactly one GateResult is returned + And the GateResult has gate_name "diff.differential_oracle" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario Outline: Python executable substitution in commands + Given a "pickled.diff.yaml" configuration file exists + And the starts with "" + And the has more than one element + And the corpus is a valid string path + And a corpus file exists at the specified path + When the run_all gate is invoked + Then the subprocess runner for uses sys.executable as the first argument + + Examples: + | command_key | python_token | runner_name | + | oracle_command | python | oracle | + | oracle_command | python3 | oracle | + | candidate_command | python | candidate | + | candidate_command | python3 | candidate | + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Python executable substitution requires multiple command elements + Given a "pickled.diff.yaml" configuration file exists + And the oracle_command is ["python"] + And the corpus is a valid string path + And a corpus file exists at the specified path + When the run_all gate is invoked + Then the subprocess runner for oracle does not substitute sys.executable + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Configuration defaults for optional fields + Given a "pickled.diff.yaml" configuration file exists + And the oracle_command is a valid list + And the candidate_command is a valid list + And the corpus is a valid string path + And timeout_seconds is not specified + And comparator is not specified + And a corpus file exists at the specified path + When the run_all gate is invoked + Then the timeout is set to 30 seconds + And the comparator is set to "exact" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: FileNotFoundError during configuration loading results in FAIL + Given configuration loading raises FileNotFoundError + When the run_all gate is invoked + Then a list with exactly one GateResult is returned + And the GateResult has gate_name "diff.config" + And the GateResult has verdict FAIL + And the GateResult notes contain the exception message + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario Outline: Configuration parsing errors result in FAIL + Given a "pickled.diff.yaml" configuration file exists + And configuration parsing raises + When the run_all gate is invoked + Then a list with exactly one GateResult is returned + And the GateResult has gate_name "diff.config" + And the GateResult has verdict FAIL + And the GateResult notes contain the exception message + + Examples: + | exception_type | + | ValueError | + | json.JSONDecodeError | + | TypeError | + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: JSONDecodeError during corpus loading results in FAIL + Given a "pickled.diff.yaml" configuration file exists + And the configuration is valid + And corpus loading raises json.JSONDecodeError + When the run_all gate is invoked + Then a list with exactly one GateResult is returned + And the GateResult has verdict FAIL + And the GateResult notes contain the exception message + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Return value always contains exactly one GateResult + Given any valid or invalid configuration state + When the run_all gate is invoked + Then a list with exactly one GateResult is returned diff --git a/dogfood/mining-output/features/pickled_diff_verify.feature b/dogfood/mining-output/features/pickled_diff_verify.feature new file mode 100644 index 0000000..a82151c --- /dev/null +++ b/dogfood/mining-output/features/pickled_diff_verify.feature @@ -0,0 +1,155 @@ +Feature: Differential testing verification command + As a developer using pickled-diff + I want to verify a candidate implementation against a trusted oracle + So that I can detect behavioral differences across a corpus of test inputs + + Background: + Given the pickled-diff CLI is available + And a timeout of 30 seconds is configured + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User provides corpus JSON that is not a list + Given a corpus file "corpus.json" containing a JSON object instead of a list + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then a ClickException is raised with message containing "must be a list" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User provides valid corpus with dictionary entries + Given a corpus file "corpus.json" containing [{"name": "a", "payload": "b"}] + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then both oracle and candidate commands receive payload "b" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User provides corpus with non-string name and payload + Given a corpus file "corpus.json" containing [{"name": 1, "payload": 2}] + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then the name is coerced to string "1" + And the payload is coerced to string "2" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User provides corpus with mixed valid and invalid entries + Given a corpus file "corpus.json" containing [{"name": "x", "payload": "y"}, "not-a-dict", {"name": "z", "payload": "w"}] + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then only the 2 dictionary entries are processed + And the non-dictionary entry is silently ignored + + # TODO: Verify missing 'name' or 'payload' keys cause KeyError - need corpus structure details + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User provides corpus with missing required keys + Given a corpus file "corpus.json" containing [{"name": "x"}] + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then a KeyError is raised + + @pickled-internal:mcp-output-fixed-json-shape + Scenario Outline: User selects comparison strategy + Given a corpus file "corpus.json" with valid test data + When the user invokes verify with comparator "" + Then the comparator is selected + + Examples: + | comparator | strategy | + | structural_json | structural JSON | + | exact | exact equality | + | "" | exact equality | + | unrecognized | exact equality | + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User provides empty corpus + Given a corpus file "corpus.json" containing an empty list + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then the verdict is PASS + And the exit code is 0 + And the notes mention "empty" + + @pickled-internal:core-llm-cache-default-on + Scenario: Oracle fails on every input + Given a corpus file "corpus.json" with 3 test inputs + And the oracle command fails for all inputs + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then the verdict is FAIL + And the exit code is 2 + And the notes contain "Oracle failed on every input" + + @pickled-internal:core-llm-cache-default-on + Scenario: No items successfully compared due to oracle errors + Given a corpus file "corpus.json" with 2 test inputs + And the oracle command fails for all inputs + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then the verdict is WARN + And the exit code is 1 + + @pickled-internal:core-llm-cache-default-on + Scenario: Some compared items mismatch + Given a corpus file "corpus.json" with 4 test inputs + And 2 inputs produce matching outputs + And 2 inputs produce mismatched outputs + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then the verdict is WARN + And the exit code is 1 + + @pickled-internal:core-llm-cache-default-on + Scenario: All compared items mismatch + Given a corpus file "corpus.json" with 3 test inputs + And all inputs produce mismatched outputs + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then the verdict is FAIL + And the exit code is 2 + + @pickled-internal:core-llm-cache-default-on + Scenario: All items pass comparison + Given a corpus file "corpus.json" with 5 test inputs + And all inputs produce matching outputs + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then the verdict is PASS + And the exit code is 0 + And the notes contain "0/5 mismatches" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Verify JSON output structure + Given a corpus file "corpus.json" with valid test data + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then the JSON output contains key "gate" + And the JSON output contains key "verdict" + And the JSON output contains key "notes" + And the JSON output contains key "findings" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Verify finding structure when mismatches occur + Given a corpus file "corpus.json" with 1 test input + And the input produces a mismatch + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then each finding contains key "input_repr" + And each finding contains key "oracle_output" + And each finding contains key "candidate_output" + And each finding contains key "diff_summary" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Oracle errors cause items to be skipped from comparison + Given a corpus file "corpus.json" with 3 test inputs + And 1 input produces an oracle error + And 2 inputs produce matching outputs + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then only 2 items are counted as compared + And the oracle error item is not counted as compared + + @pickled-internal:core-llm-cache-default-on + Scenario: Candidate errors are counted as mismatches + Given a corpus file "corpus.json" with 3 test inputs + And 1 input produces a candidate error + And 2 inputs produce matching outputs + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then 3 items are counted as compared + And 1 item is counted as a mismatch + And a finding is generated for the candidate error + + @core-domain:verdict-three-state-ladder + Scenario Outline: Exit codes map to verdicts + Given a corpus file "corpus.json" with test data producing a verdict + When the user invokes verify with oracle "python oracle.py", candidate "python candidate.py", and corpus "corpus.json" + Then the exit code is + + Examples: + | verdict | exit_code | + | PASS | 0 | + | WARN | 1 | + | FAIL | 2 | diff --git a/dogfood/mining-output/features/pickled_iac_diff.feature b/dogfood/mining-output/features/pickled_iac_diff.feature new file mode 100644 index 0000000..3cb8410 --- /dev/null +++ b/dogfood/mining-output/features/pickled_iac_diff.feature @@ -0,0 +1,173 @@ +Feature: Terraform plan diff comparison + As a DevOps engineer + I want to compare two Terraform plan JSON files + So that I can identify infrastructure changes between baseline and proposed plans + + Background: + Given a CLI command "diff" + + @pickled-internal:core-llm-cache-default-on + Scenario: Operator compares two identical plans + Given a valid Terraform plan file at "base.json" with no resource changes + And a valid Terraform plan file at "head.json" with no resource changes + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 0 + And the output contains valid JSON + And the verdict is "PASS" + And the findings array is empty + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator compares two empty plans + Given a valid Terraform plan file at "base.json" with resource_changes set to null + And a valid Terraform plan file at "head.json" with resource_changes set to null + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 0 + And the verdict is "PASS" + And the findings array is empty + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator compares plans where head contains a new resource + Given a valid Terraform plan file at "base.json" with no resource changes + And a valid Terraform plan file at "head.json" with a resource "aws_instance.web" having actions "create" + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 1 + And the verdict is "WARN" + And the findings contain a resource "aws_instance.web" with actions_before empty and actions_after "create" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator compares plans where head deletes a resource + Given a valid Terraform plan file at "base.json" with a resource "aws_instance.web" having actions "no-op" + And a valid Terraform plan file at "head.json" with a resource "aws_instance.web" having actions "delete" + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 2 + And the verdict is "FAIL" + And the findings contain a resource "aws_instance.web" with actions_before "no-op" and actions_after "delete" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator compares plans where head replaces a resource + Given a valid Terraform plan file at "base.json" with a resource "aws_instance.web" having actions "no-op" + And a valid Terraform plan file at "head.json" with a resource "aws_instance.web" having actions "delete,create" + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 2 + And the verdict is "FAIL" + And the findings contain a resource "aws_instance.web" with actions_before "no-op" and actions_after "delete,create" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator compares plans with different actions for same resource + Given a valid Terraform plan file at "base.json" with a resource "aws_instance.web" having actions "no-op" + And a valid Terraform plan file at "head.json" with a resource "aws_instance.web" having actions "update" + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 1 + And the verdict is "WARN" + And the findings contain a resource "aws_instance.web" with actions_before "no-op" and actions_after "update" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator compares plans where base contains resource not in head + Given a valid Terraform plan file at "base.json" with a resource "aws_instance.old" having actions "no-op" + And a valid Terraform plan file at "head.json" with no resource changes + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 0 + And the verdict is "PASS" + And the findings array is empty + + @pickled-internal:mcp-output-fixed-json-shape + Scenario Outline: Operator compares plans with safe actions only + Given a valid Terraform plan file at "base.json" with no resource changes + And a valid Terraform plan file at "head.json" with a resource "aws_instance.web" having actions "" + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 1 + And the verdict is "WARN" + + Examples: + | action | + | create | + | update | + | read | + | no-op | + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator provides non-existent base file + Given a file "head.json" exists + And a file "nonexistent.json" does not exist + When the operator runs diff with base "nonexistent.json" and head "head.json" + Then the command raises a file not found exception + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator provides non-existent head file + Given a file "base.json" exists + And a file "nonexistent.json" does not exist + When the operator runs diff with base "base.json" and head "nonexistent.json" + Then the command raises a file not found exception + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator provides file with invalid JSON in base + Given a file "base.json" contains invalid JSON + And a valid Terraform plan file at "head.json" with no resource changes + When the operator runs diff with base "base.json" and head "head.json" + Then the command raises a JSON decode exception + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator provides file with invalid JSON in head + Given a valid Terraform plan file at "base.json" with no resource changes + And a file "head.json" contains invalid JSON + When the operator runs diff with base "base.json" and head "head.json" + Then the command raises a JSON decode exception + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator provides head file that is not a dictionary + Given a valid Terraform plan file at "base.json" with no resource changes + And a file "head.json" contains a JSON array + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 2 + And the verdict is "FAIL" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator provides base file that is not a dictionary + Given a file "base.json" contains a JSON array + And a valid Terraform plan file at "head.json" with no resource changes + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 2 + And the verdict is "FAIL" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator compares plans with non-dictionary resource change entries + Given a valid Terraform plan file at "base.json" with resource_changes containing non-dictionary entries + And a valid Terraform plan file at "head.json" with a resource "aws_instance.web" having actions "create" + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 1 + And the verdict is "WARN" + And the non-dictionary entries are silently ignored + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator compares plans with missing change field + Given a valid Terraform plan file at "base.json" with a resource "aws_instance.web" missing change field + And a valid Terraform plan file at "head.json" with a resource "aws_instance.web" having actions "create" + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 1 + And the verdict is "WARN" + And the resource with missing change field is treated as having no actions + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator compares plans with non-list actions field + Given a valid Terraform plan file at "base.json" with a resource "aws_instance.web" having non-list actions + And a valid Terraform plan file at "head.json" with a resource "aws_instance.web" having actions "update" + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 1 + And the verdict is "WARN" + And the resource with non-list actions is treated as having no actions + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator compares plans with missing address field + Given a valid Terraform plan file at "base.json" with a resource missing address field + And a valid Terraform plan file at "head.json" with no resource changes + When the operator runs diff with base "base.json" and head "head.json" + Then the command exits with status 0 + And resources with missing address are indexed with empty string + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator validates output JSON structure + Given a valid Terraform plan file at "base.json" with no resource changes + And a valid Terraform plan file at "head.json" with a resource "aws_instance.web" having actions "create" + When the operator runs diff with base "base.json" and head "head.json" + Then the output JSON contains exactly the keys "verdict", "notes", "findings" + And each finding contains exactly the keys "address", "actions_before", "actions_after" diff --git a/dogfood/mining-output/features/pickled_iac_draft.feature b/dogfood/mining-output/features/pickled_iac_draft.feature new file mode 100644 index 0000000..1df57f7 --- /dev/null +++ b/dogfood/mining-output/features/pickled_iac_draft.feature @@ -0,0 +1,152 @@ +Feature: Draft Terraform module from natural-language user story + + As a user of the pickled-iac CLI + I want to generate Terraform infrastructure code from a natural-language description + So that I can define infrastructure without manually writing HCL + + Background: + Given terraform or opentofu is available on the system PATH + And the LLM client is properly configured + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User drafts a module and prints to stdout + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "aws" + And the user does not specify an output path + When the draft command is invoked + Then the generated HCL content is printed to stdout + And no files are created on the file system + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User drafts a module and writes to a specified directory + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "aws" + And the user specifies an output path "/tmp/my-module" + When the draft command is invoked + Then the directory "/tmp/my-module" is created including any parent directories + And a file named "main.tf" is written to "/tmp/my-module" containing the generated HCL + And a confirmation message "Wrote /tmp/my-module/main.tf" is printed to stderr + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User drafts a module and writes to an existing directory + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "azure" + And the directory "/tmp/existing-module" already exists + When the draft command is invoked + Then a file named "main.tf" is written to "/tmp/existing-module" containing the generated HCL + And a confirmation message "Wrote /tmp/existing-module/main.tf" is printed to stderr + + @best-practices:agent-path-first-class + Scenario: User drafts a module but neither terraform nor opentofu is available + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "aws" + And neither terraform nor opentofu is available on the system PATH + When the draft command is invoked + Then an IaCToolMissingError is raised + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: User drafts a module with malformed custom LLM factory environment variable + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "aws" + And the environment variable PICKLED_IAC_LLM_FACTORY is set to "mypackage_get_client" without a colon separator + When the draft command is invoked + Then a ClickException is raised indicating the required "module:callable" format + + @pickled-internal:core-llm-cache-default-on + Scenario: User drafts a module but LLM configuration is invalid + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "aws" + And the environment variable PICKLED_IAC_LLM_FACTORY is not set + And the LLM configuration cannot be loaded + When the draft command is invoked + Then a ClickException is raised wrapping the underlying ConfigError + + @pickled-internal:core-llm-cache-default-on + Scenario: LLM generates valid HCL on first attempt + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "aws" + And the LLM generates valid HCL on the first attempt + When the draft command is invoked + Then terraform validate or opentofu validate is executed against the generated HCL + And the validation succeeds + And the validated HCL is returned + + @pickled-internal:core-llm-cache-default-on + Scenario: LLM generates invalid HCL but succeeds on retry + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "aws" + And the LLM generates invalid HCL on the first attempt + And the LLM generates valid HCL on the second attempt + When the draft command is invoked + Then terraform validate or opentofu validate is executed against the first generated HCL + And the validation fails with diagnostic messages + And the LLM is called again with the user story and the validation error feedback + And terraform validate or opentofu validate is executed against the second generated HCL + And the validation succeeds + And the validated HCL is returned + + @pickled-internal:core-llm-cache-default-on + Scenario: LLM fails to generate valid HCL after 3 attempts + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "aws" + And the LLM generates invalid HCL on all 3 attempts + When the draft command is invoked + Then terraform validate or opentofu validate is executed 3 times + And each validation fails with diagnostic messages + And an IaCValidationError is raised containing all collected validation diagnostics + + @pickled-internal:core-llm-cache-default-on + Scenario: LLM response contains code fences which are stripped + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "aws" + And the LLM response includes leading and trailing triple-backtick fences + When the draft command is invoked + Then the code fences are stripped from the LLM response + And terraform validate or opentofu validate is executed against the stripped HCL + And the validation succeeds + + @bdd-domain:draft-empty-story-deterministic-failure + Scenario Outline: User specifies different cloud providers + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "" + When the draft command is invoked + Then the provider "" is passed to the prompt rendering + And the generated HCL contains resources specific to "" + + Examples: + | provider | + | aws | + | azure | + | gcp | + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: User invokes draft with custom LLM factory + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "aws" + And the environment variable PICKLED_IAC_LLM_FACTORY is set to "mypackage:get_client" + When the draft command is invoked + Then the module "mypackage" is dynamically imported + And the callable "get_client" is invoked to obtain the LLM client + And the custom LLM client is used for generation + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User invokes draft without custom factory and PICKLED_LLM_PROVIDER not set + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "aws" + And the environment variable PICKLED_IAC_LLM_FACTORY is not set + And the environment variable PICKLED_LLM_PROVIDER is not set + When the draft command is invoked + Then the LLM provider defaults to "anthropic" + And the LLM client is built using pickled_core.llm.factory.build_client + And the LLM configuration is loaded from pickled_core.llm.config.load_config + + @pickled-internal:core-llm-cache-default-on + Scenario: Validation process initializes terraform directory + Given the user provides a user story describing desired infrastructure + And the user specifies a provider "aws" + And the LLM generates valid HCL + When the draft command is invoked + Then the generated HCL is written to a temporary directory + And terraform init or opentofu init is run if needed + And terraform validate or opentofu validate is executed in JSON mode + And the validation succeeds diff --git a/dogfood/mining-output/features/pickled_iac_iacambiguitygate.feature b/dogfood/mining-output/features/pickled_iac_iacambiguitygate.feature new file mode 100644 index 0000000..376211e --- /dev/null +++ b/dogfood/mining-output/features/pickled_iac_iacambiguitygate.feature @@ -0,0 +1,140 @@ +```gherkin +Feature: IaC Ambiguity Gate Evaluation + As a DevOps engineer + I want to validate Terraform artifacts against user stories for ambiguities + So that I can identify potential implementation issues before deployment + + Background: + Given an IaC ambiguity gate instance + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate rejects non-IaCArtifact target + When the gate runs with a target that is not an IaCArtifact instance + Then the gate returns a FAIL verdict + And the result note indicates the actual type received + And the result includes the gate name + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate rejects missing user story when context is None + Given a valid IaCArtifact target + When the gate runs with context set to None + Then the gate returns a FAIL verdict + And the result note states the user story requirement + And the result includes the gate name + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate rejects non-string user story + Given a valid IaCArtifact target + When the gate runs with context containing a non-string "user_story" value + Then the gate returns a FAIL verdict + And the result note states the user story requirement + And the result includes the gate name + + @pickled-internal:mcp-output-fixed-json-shape + Scenario Outline: Gate rejects empty or whitespace-only user story + Given a valid IaCArtifact target + When the gate runs with context containing user story "" + Then the gate returns a FAIL verdict + And the result note states the user story requirement + And the result includes the gate name + + Examples: + | user_story | + | | + | | + | | + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate renders template with user story and artifact content + Given a valid IaCArtifact target with content + And a context with a valid user story + When the gate runs + Then the gate renders the template with the user story + And the gate renders the template with the target content + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Gate calls LLM with JSON-only system message + Given a valid IaCArtifact target with content + And a context with a valid user story + When the gate runs + Then the gate calls complete_prompt with a system message requesting JSON without markdown + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate rejects unparseable LLM response + Given a valid IaCArtifact target with content + And a context with a valid user story + And the LLM returns a response that cannot be parsed as JSON + When the gate runs + Then the gate returns a FAIL verdict + And the result note is "LLM returned malformed JSON" + And the result includes the gate name + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate rejects non-dictionary JSON response + Given a valid IaCArtifact target with content + And a context with a valid user story + And the LLM returns a JSON array + When the gate runs + Then the gate returns a FAIL verdict + And the result note is "LLM returned malformed JSON" + And the result includes the gate name + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate rejects JSON missing ambiguities key + Given a valid IaCArtifact target with content + And a context with a valid user story + And the LLM returns JSON without an "ambiguities" key + When the gate runs + Then the gate returns a FAIL verdict + And the result note is "LLM JSON missing list field \"ambiguities\"" + And the result includes the gate name + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate rejects JSON with non-list ambiguities value + Given a valid IaCArtifact target with content + And a context with a valid user story + And the LLM returns JSON with "ambiguities" as a non-list value + When the gate runs + Then the gate returns a FAIL verdict + And the result note is "LLM JSON missing list field \"ambiguities\"" + And the result includes the gate name + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate passes when no ambiguities reported + Given a valid IaCArtifact target with content + And a context with a valid user story + And the LLM returns JSON with an empty "ambiguities" list + When the gate runs + Then the gate returns a PASS verdict + And the result note is "No ambiguities reported." + And the result includes the gate name + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate warns when ambiguities are detected + Given a valid IaCArtifact target with content + And a context with a valid user story + And the LLM returns JSON with 3 items in the "ambiguities" list + When the gate runs + Then the gate returns a WARN verdict + And the result note indicates 3 ambiguities + And the result findings contain the ambiguities as a tuple + And the result includes the gate name + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate extracts JSON from markdown code fences + Given a valid IaCArtifact target with content + And a context with a valid user story + And the LLM returns valid JSON wrapped in markdown code fences starting with "```" + When the gate runs + Then the gate extracts content between the first and last "```" delimiters + And the gate parses the extracted JSON successfully + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate extracts JSON between first and last braces + Given a valid IaCArtifact target with content + And a context with a valid user story + And the LLM returns a response with extra text before and after JSON + When the gate runs + Then the gate extracts the substring between the first "{" and last "}" + And the gate parses the extracted JSON successfully +``` diff --git a/dogfood/mining-output/features/pickled_iac_mcp.feature b/dogfood/mining-output/features/pickled_iac_mcp.feature new file mode 100644 index 0000000..d4b401f --- /dev/null +++ b/dogfood/mining-output/features/pickled_iac_mcp.feature @@ -0,0 +1,45 @@ +Feature: MCP Command Group + As a CLI user + I want to access MCP server commands under a "mcp" namespace + So that I can organize and invoke MCP-related operations + + Background: + Given the pickled-iac CLI is available + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User invokes the mcp command group programmatically + When the mcp command group function is called + Then it returns None + And no exceptions are raised + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User invokes the mcp command group with no arguments + When the user executes the "mcp" command without subcommands + Then the command accepts no arguments + And the command completes immediately + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: MCP command group performs no side effects + When the mcp command group function is called + Then no file I/O operations are performed + And no network operations are initiated + And no state mutations occur + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: MCP command group organizes subcommands + Given MCP-related subcommands exist in the system + When the user views the mcp command group + Then subcommands are organized under the "mcp" namespace + + # TODO: Verify Click framework behavior when invoked without subcommands (help text vs error) + @best-practices:cli-mcp-surface-parity + Scenario: User invokes mcp command group from CLI without subcommands + When the user executes the "mcp" command from the command line + Then the Click framework handles the invocation + And appropriate feedback is displayed to the user + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: MCP command group execution is non-blocking + When the mcp command group function is called + Then it executes without blocking + And it completes immediately diff --git a/dogfood/mining-output/features/pickled_iac_plan_cmd.feature b/dogfood/mining-output/features/pickled_iac_plan_cmd.feature new file mode 100644 index 0000000..c36dbea --- /dev/null +++ b/dogfood/mining-output/features/pickled_iac_plan_cmd.feature @@ -0,0 +1,63 @@ +Feature: Plan Command + As an infrastructure engineer + I want to execute Terraform plan and capture the output as JSON + So that I can analyze infrastructure changes and feed them into validation gates + + Background: + Given a Terraform configuration exists + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Infrastructure engineer generates a plan with valid Terraform directory and output path + Given a valid Terraform directory path "tf_dir" + And a valid output file path "output.json" + When the infrastructure engineer executes the plan command with "tf_dir" and "output.json" + Then terraform plan is executed in the "tf_dir" directory + And the plan output is captured in JSON format + And the JSON data is written to "output.json" + And the command exits with a success status code + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Infrastructure engineer generates a plan when output file does not exist + Given a valid Terraform directory path "tf_dir" + And an output file path "new_output.json" that does not exist + When the infrastructure engineer executes the plan command with "tf_dir" and "new_output.json" + Then the output file "new_output.json" is created + And the JSON-formatted plan data is written to "new_output.json" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Infrastructure engineer generates a plan that produces valid JSON output + Given a valid Terraform directory path "tf_dir" + And a valid output file path "output.json" + When the infrastructure engineer executes the plan command with "tf_dir" and "output.json" + Then the generated output at "output.json" is valid JSON + And the JSON output is parseable + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Infrastructure engineer generates a plan output consumable by validation gates + Given a valid Terraform directory path "tf_dir" + And a valid output file path "output.json" + When the infrastructure engineer executes the plan command with "tf_dir" and "output.json" + Then the JSON output can be consumed by IaCAmbiguityGate + And the JSON output can be consumed by PlanDiffGate + And the JSON output can be consumed by SecurityBaselineGate + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Infrastructure engineer attempts to generate a plan when terraform plan fails + Given a valid Terraform directory path "tf_dir" + And a valid output file path "output.json" + And the Terraform configuration in "tf_dir" will cause plan to fail + When the infrastructure engineer executes the plan command with "tf_dir" and "output.json" + Then the command exits with a failure status code + + # TODO: Clarify expected behavior when tf_dir does not exist or is not a valid Terraform directory + @bdd-domain:draft-empty-story-deterministic-failure + Scenario Outline: Infrastructure engineer provides invalid arguments + Given an argument configuration with and + When the infrastructure engineer executes the plan command + Then the command exits with a failure status code + + Examples: + | tf_dir_state | output_state | + | missing | valid | + | valid | missing | + | missing | missing | diff --git a/dogfood/mining-output/features/pickled_iac_plandiffgate.feature b/dogfood/mining-output/features/pickled_iac_plandiffgate.feature new file mode 100644 index 0000000..33de980 --- /dev/null +++ b/dogfood/mining-output/features/pickled_iac_plandiffgate.feature @@ -0,0 +1,238 @@ +Feature: PlanDiffGate compares Terraform plan JSONs and determines pass/warn/fail verdict + + Background: + Given a PlanDiffGate instance + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when target is not a dict + Given the target is an integer 123 + And the context contains a valid base_plan dict + When the gate runs + Then the verdict is FAIL + And the notes indicate the actual type received + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when target is not a dict (string type) + Given the target is a string "not a dict" + And the context contains a valid base_plan dict + When the gate runs + Then the verdict is FAIL + And the notes indicate the actual type received + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when context is None + Given the target is a valid plan dict + And the context is None + When the gate runs + Then the verdict is FAIL + And the notes indicate missing base_plan + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when context does not contain base_plan key + Given the target is a valid plan dict + And the context is an empty dict + When the gate runs + Then the verdict is FAIL + And the notes indicate missing base_plan + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when base_plan in context is not a dict + Given the target is a valid plan dict + And the context contains base_plan as a string + When the gate runs + Then the verdict is FAIL + And the notes indicate base_plan is not a dict + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate passes when both plans have empty resource_changes + Given the target plan has no resource_changes key + And the base_plan has no resource_changes key + When the gate runs + Then the verdict is PASS + And the findings are empty + And the notes are "No plan changes between base and head." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate passes when both plans have null resource_changes + Given the target plan has resource_changes set to null + And the base_plan has resource_changes set to null + When the gate runs + Then the verdict is PASS + And the findings are empty + And the notes are "No plan changes between base and head." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate passes when both plans have empty resource_changes lists + Given the target plan has an empty resource_changes list + And the base_plan has an empty resource_changes list + When the gate runs + Then the verdict is PASS + And the findings are empty + And the notes are "No plan changes between base and head." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate includes finding for new resource in head plan with non-empty actions + Given the base_plan has an empty resource_changes list + And the target plan has a resource "aws_instance.web" with actions ["create"] + When the gate runs + Then the verdict is WARN + And there is 1 finding + And the finding for "aws_instance.web" has base_actions tuple () + And the finding for "aws_instance.web" has head_actions tuple ("create",) + And the notes are "1 resource change(s) detected." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate includes finding for resource with different actions between plans + Given the base_plan has a resource "aws_instance.web" with actions ["update"] + And the target plan has a resource "aws_instance.web" with actions ["replace"] + When the gate runs + Then the verdict is FAIL + And there is 1 finding + And the finding for "aws_instance.web" has base_actions tuple ("update",) + And the finding for "aws_instance.web" has head_actions tuple ("replace",) + And the notes are "1 resource change(s) detected." + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Gate ignores resources only in base plan + Given the base_plan has a resource "aws_instance.old" with actions ["delete"] + And the target plan has an empty resource_changes list + When the gate runs + Then the verdict is PASS + And the findings are empty + And the notes are "No plan changes between base and head." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when any action is delete + Given the base_plan has an empty resource_changes list + And the target plan has a resource "aws_instance.web" with actions ["delete"] + When the gate runs + Then the verdict is FAIL + And there is 1 finding + And the notes are "1 resource change(s) detected." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when any action is replace + Given the base_plan has an empty resource_changes list + And the target plan has a resource "aws_instance.web" with actions ["replace"] + When the gate runs + Then the verdict is FAIL + And there is 1 finding + And the notes are "1 resource change(s) detected." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario Outline: Gate warns for safe actions + Given the base_plan has an empty resource_changes list + And the target plan has a resource "aws_instance.web" with actions [""] + When the gate runs + Then the verdict is WARN + And there is 1 finding + And the notes are "1 resource change(s) detected." + + Examples: + | action | + | create | + | update | + | read | + | no-op | + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate warns when all actions are from safe set + Given the base_plan has an empty resource_changes list + And the target plan has a resource "aws_instance.web" with actions ["create", "read", "update"] + When the gate runs + Then the verdict is WARN + And there is 1 finding + And the notes are "1 resource change(s) detected." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate warns for unknown actions outside safe and destructive sets + Given the base_plan has an empty resource_changes list + And the target plan has a resource "aws_instance.web" with actions ["unknown_action"] + When the gate runs + Then the verdict is WARN + And there is 1 finding + And the notes are "1 resource change(s) detected." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate includes multiple findings with correct count in notes + Given the base_plan has an empty resource_changes list + And the target plan has a resource "aws_instance.web" with actions ["create"] + And the target plan has a resource "aws_s3_bucket.data" with actions ["update"] + When the gate runs + Then the verdict is WARN + And there are 2 findings + And the notes are "2 resource change(s) detected." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate skips non-dict entries in resource_changes + Given the base_plan has an empty resource_changes list + And the target plan resource_changes contains a string entry "invalid" + And the target plan has a resource "aws_instance.web" with actions ["create"] + When the gate runs + Then the verdict is WARN + And there is 1 finding + And the finding for "aws_instance.web" exists + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate treats missing change field as empty actions + Given the base_plan has an empty resource_changes list + And the target plan has a resource "aws_instance.web" with no change field + When the gate runs + Then the verdict is PASS + And the findings are empty + And the notes are "No plan changes between base and head." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate treats null change field as empty actions + Given the base_plan has an empty resource_changes list + And the target plan has a resource "aws_instance.web" with null change field + When the gate runs + Then the verdict is PASS + And the findings are empty + And the notes are "No plan changes between base and head." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate treats missing actions field as empty list + Given the base_plan has an empty resource_changes list + And the target plan has a resource "aws_instance.web" with change but no actions + When the gate runs + Then the verdict is PASS + And the findings are empty + And the notes are "No plan changes between base and head." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate treats non-list actions as empty list + Given the base_plan has an empty resource_changes list + And the target plan has a resource "aws_instance.web" with actions as string "create" + When the gate runs + Then the verdict is PASS + And the findings are empty + And the notes are "No plan changes between base and head." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate preserves action order in findings tuples + Given the base_plan has a resource "aws_instance.web" with actions ["read", "update", "create"] + And the target plan has a resource "aws_instance.web" with actions ["create", "update", "delete"] + When the gate runs + Then the verdict is FAIL + And there is 1 finding + And the finding for "aws_instance.web" has base_actions tuple ("read", "update", "create") + And the finding for "aws_instance.web" has head_actions tuple ("create", "update", "delete") + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate does not create finding when resource has same actions in both plans + Given the base_plan has a resource "aws_instance.web" with actions ["update"] + And the target plan has a resource "aws_instance.web" with actions ["update"] + When the gate runs + Then the verdict is PASS + And the findings are empty + And the notes are "No plan changes between base and head." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate does not create finding for resource with empty actions in both plans + Given the base_plan has a resource "aws_instance.web" with actions [] + And the target plan has a resource "aws_instance.web" with actions [] + When the gate runs + Then the verdict is PASS + And the findings are empty + And the notes are "No plan changes between base and head." diff --git a/dogfood/mining-output/features/pickled_iac_run_all.feature b/dogfood/mining-output/features/pickled_iac_run_all.feature new file mode 100644 index 0000000..361c033 --- /dev/null +++ b/dogfood/mining-output/features/pickled_iac_run_all.feature @@ -0,0 +1,131 @@ +Feature: Infrastructure-as-Code validation orchestration + As a CI/CD pipeline + I want to validate Terraform/OpenTofu configurations and scan for security issues + So that I can prevent misconfigured or insecure infrastructure from being deployed + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User runs validation when infra directory is missing + Given the workdir does not contain an "infra/" subdirectory + When the user runs the IaC validation gate + Then the gate returns a single GateResult + And the result has gate_name "iac.infra" + And the result has verdict "WARN" + And the result notes contain "no infra/ directory" + + @iac-domain:terraform-validate-entry + Scenario: User runs validation when Terraform/OpenTofu binary is not on PATH + Given the workdir contains an "infra/" subdirectory + And the Terraform/OpenTofu binary is not found on PATH + When the user runs the IaC validation gate + Then the gate returns 2 GateResult objects + And the "iac.validate" result has verdict "WARN" + And the "iac.validate" result notes contain the IaCToolMissingError message + + @iac-domain:terraform-validate-entry + Scenario: User runs validation when Terraform configuration is valid + Given the workdir contains an "infra/" subdirectory + And the Terraform/OpenTofu binary is found on PATH + And the Terraform directory is initialized + And the Terraform configuration is valid + When the user runs the IaC validation gate + Then the "iac.validate" result has verdict "PASS" + And the "iac.validate" result notes contain "ok" or validation diagnostics + + @iac-domain:terraform-validate-entry + Scenario: User runs validation when Terraform configuration is invalid + Given the workdir contains an "infra/" subdirectory + And the Terraform/OpenTofu binary is found on PATH + And the Terraform directory is initialized + And the Terraform configuration reports valid=false + When the user runs the IaC validation gate + Then the "iac.validate" result has verdict "FAIL" + And the "iac.validate" result notes contain semicolon-separated diagnostic messages + + @iac-domain:terraform-validate-entry + Scenario: User runs validation when Terraform validate raises unexpected exception + Given the workdir contains an "infra/" subdirectory + And the Terraform/OpenTofu binary is found on PATH + And the Terraform validation raises an unexpected exception + When the user runs the IaC validation gate + Then the "iac.validate" result has verdict "FAIL" + And the "iac.validate" result notes contain the exception message + + @iac-domain:terraform-validate-entry + Scenario: User runs validation and Terraform directory requires initialization + Given the workdir contains an "infra/" subdirectory + And the Terraform/OpenTofu binary is found on PATH + And the ".terraform/" directory does not exist in "infra/" + When the user runs the IaC validation gate + Then the gate executes "terraform init -input=false -backend=false" + And the terraform init uses environment variable TF_IN_AUTOMATION=1 + And the terraform validate runs after successful initialization + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User runs validation when Trivy binary is not on PATH + Given the workdir contains an "infra/" subdirectory + And the Trivy binary is not found on PATH + When the user runs the IaC validation gate + Then the security baseline result has verdict "PASS" + And the security baseline result notes contain "trivy not found on PATH — security scan skipped" + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User runs validation when Trivy finds CRITICAL severity misconfigurations + Given the workdir contains an "infra/" subdirectory + And the Trivy binary is found on PATH + And Trivy scan detects CRITICAL severity misconfigurations + When the user runs the IaC validation gate + Then the security baseline result has verdict "FAIL" + And the security baseline result includes findings tuple with CRITICAL issue titles + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User runs validation when Trivy finds HIGH severity misconfigurations only + Given the workdir contains an "infra/" subdirectory + And the Trivy binary is found on PATH + And Trivy scan detects HIGH severity misconfigurations + And Trivy scan detects no CRITICAL severity misconfigurations + When the user runs the IaC validation gate + Then the security baseline result has verdict "WARN" + And the security baseline result includes findings tuple with HIGH issue titles + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User runs validation when Trivy finds no HIGH or CRITICAL misconfigurations + Given the workdir contains an "infra/" subdirectory + And the Trivy binary is found on PATH + And Trivy scan detects no HIGH or CRITICAL severity misconfigurations + When the user runs the IaC validation gate + Then the security baseline result has verdict "PASS" + And the security baseline result notes contain "No HIGH or CRITICAL findings." + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User runs validation when Trivy returns non-JSON output + Given the workdir contains an "infra/" subdirectory + And the Trivy binary is found on PATH + And Trivy returns non-JSON output + When the user runs the IaC validation gate + Then the security baseline result has verdict "WARN" + And the security baseline result notes contain "trivy returned non-JSON output" + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User runs validation when Trivy execution fails with unexpected error + Given the workdir contains an "infra/" subdirectory + And the Trivy binary is found on PATH + And Trivy execution fails with a non-0/1 return code and no stdout + When the user runs the IaC validation gate + Then the security baseline result has verdict "WARN" + And the security baseline result notes contain error details + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User runs validation and receives expected result structure + Given the workdir contains an "infra/" subdirectory + When the user runs the IaC validation gate + Then the gate returns a list of GateResult objects + And the list contains 2 GateResult objects + And each GateResult has gate_name, verdict, and notes attributes + And the list is never None or a single GateResult object + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User runs validation with all Terraform subprocess calls + Given the workdir contains an "infra/" subdirectory + And the Terraform/OpenTofu binary is found on PATH + When the user runs the IaC validation gate + Then all Terraform/OpenTofu subprocess calls include environment variable TF_IN_AUTOMATION=1 diff --git a/dogfood/mining-output/features/pickled_iac_scan.feature b/dogfood/mining-output/features/pickled_iac_scan.feature new file mode 100644 index 0000000..b347d58 --- /dev/null +++ b/dogfood/mining-output/features/pickled_iac_scan.feature @@ -0,0 +1,116 @@ +Feature: Terraform security scanning CLI command + As an operator or CI/CD pipeline + I want to scan Terraform configurations for security issues + So that I can prevent infrastructure deployments with critical vulnerabilities + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator scans when Trivy is not installed + Given trivy is not available on PATH + When the operator runs scan on a Terraform directory + Then the command outputs valid JSON to stdout + And the JSON verdict field is "PASS" + And the JSON notes field indicates the security scan was skipped + And the JSON findings field is an empty list + And the command exits with code 0 + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator provides invalid input type + Given trivy is available on PATH + When the operator runs scan with a non-Path argument + Then the command outputs valid JSON to stdout + And the JSON verdict field is "FAIL" + And the JSON notes field mentions type mismatch + And the JSON findings field is an empty list + And the command exits with code 2 + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator scans clean Terraform configuration + Given trivy is available on PATH + And the Terraform directory contains no HIGH or CRITICAL security issues + When the operator runs scan on the Terraform directory + Then the command outputs valid JSON to stdout + And the JSON verdict field is "PASS" + And the JSON notes field indicates no issues found + And the JSON findings field is an empty list + And the command exits with code 0 + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator scans configuration with CRITICAL findings + Given trivy is available on PATH + And the Terraform directory contains CRITICAL severity misconfigurations + When the operator runs scan on the Terraform directory + Then the command outputs valid JSON to stdout + And the JSON verdict field is "FAIL" + And the JSON notes field indicates CRITICAL findings count + And the JSON findings field contains titles of CRITICAL issues + And the command exits with code 2 + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator scans configuration with HIGH findings only + Given trivy is available on PATH + And the Terraform directory contains HIGH severity misconfigurations + And the Terraform directory contains no CRITICAL severity misconfigurations + When the operator runs scan on the Terraform directory + Then the command outputs valid JSON to stdout + And the JSON verdict field is "WARN" + And the JSON notes field indicates HIGH findings count + And the JSON findings field contains titles of HIGH issues + And the command exits with code 0 + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Trivy returns malformed output + Given trivy is available on PATH + And trivy returns non-JSON output + When the operator runs scan on the Terraform directory + Then the command outputs valid JSON to stdout + And the JSON verdict field is "WARN" + And the JSON notes field indicates non-JSON output + And the JSON findings field is an empty list + And the command exits with code 0 + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Trivy fails with unexpected exit code + Given trivy is available on PATH + And trivy exits with a code other than 0 or 1 + And trivy produces no stdout + When the operator runs scan on the Terraform directory + Then the command outputs valid JSON to stdout + And the JSON verdict field is "WARN" + And the JSON notes field contains stderr content from trivy + And the JSON findings field is an empty list + And the command exits with code 0 + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Operator scans configuration with mixed severity findings + Given trivy is available on PATH + And the Terraform directory contains CRITICAL severity misconfigurations + And the Terraform directory contains HIGH severity misconfigurations + When the operator runs scan on the Terraform directory + Then the command outputs valid JSON to stdout + And the JSON verdict field is "FAIL" + And the JSON findings field contains titles of CRITICAL issues + And the JSON findings field contains titles of HIGH issues + And the command exits with code 2 + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Trivy reports findings with missing title fields + Given trivy is available on PATH + And the Terraform directory contains HIGH severity misconfigurations without Title fields + When the operator runs scan on the Terraform directory + Then the command outputs valid JSON to stdout + And the JSON verdict field is "WARN" + And the JSON findings field contains IDs or fallback text for each finding + And the command exits with code 0 + + @core-domain:verdict-three-state-ladder + Scenario Outline: Command exit codes align with verdict + Given trivy is available on PATH + And the scan produces a verdict + When the operator runs scan on the Terraform directory + Then the command exits with code + + Examples: + | verdict | exit_code | + | PASS | 0 | + | WARN | 0 | + | FAIL | 2 | diff --git a/dogfood/mining-output/features/pickled_iac_securitybaselinegate.feature b/dogfood/mining-output/features/pickled_iac_securitybaselinegate.feature new file mode 100644 index 0000000..5fbdc2b --- /dev/null +++ b/dogfood/mining-output/features/pickled_iac_securitybaselinegate.feature @@ -0,0 +1,140 @@ +Feature: Security Baseline Gate validates Terraform configurations for HIGH and CRITICAL misconfigurations + + As a quality assurance pipeline + I want to scan Infrastructure-as-Code for security misconfigurations + So that I can prevent insecure Terraform configurations from being deployed + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate rejects non-Path target with FAIL verdict + Given a target that is not a Path instance + When the security baseline gate runs + Then the gate returns a FAIL verdict + And the notes contain the actual type name of the target + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate passes gracefully when trivy is not installed + Given a valid Path target pointing to a Terraform directory + And the trivy executable is not found on the system PATH + When the security baseline gate runs + Then the gate returns a PASS verdict + And the notes indicate the scan was skipped + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate warns when trivy exits with unexpected return code and empty stdout + Given a valid Path target pointing to a Terraform directory + And the trivy executable is available + And trivy exits with a return code other than 0 or 1 + And trivy produces empty stdout + When the security baseline gate runs + Then the gate returns a WARN verdict + And the notes contain the stderr content from trivy + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Gate warns when trivy output is not valid JSON + Given a valid Path target pointing to a Terraform directory + And the trivy executable is available + And trivy produces output that is not valid JSON + When the security baseline gate runs + Then the gate returns a WARN verdict + And the notes indicate the output was non-JSON + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when trivy reports CRITICAL misconfigurations + Given a valid Path target pointing to a Terraform directory + And the trivy executable is available + And trivy reports one or more CRITICAL severity misconfigurations + When the security baseline gate runs + Then the gate returns a FAIL verdict + And the findings contain the titles of all CRITICAL misconfigurations + And the notes report the count of CRITICAL findings + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate warns when trivy reports HIGH but no CRITICAL misconfigurations + Given a valid Path target pointing to a Terraform directory + And the trivy executable is available + And trivy reports one or more HIGH severity misconfigurations + And trivy reports no CRITICAL severity misconfigurations + When the security baseline gate runs + Then the gate returns a WARN verdict + And the findings contain the titles of all HIGH misconfigurations + And the notes report the count of HIGH findings + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate passes when trivy reports no HIGH or CRITICAL misconfigurations + Given a valid Path target pointing to a Terraform directory + And the trivy executable is available + And trivy reports no HIGH or CRITICAL severity misconfigurations + When the security baseline gate runs + Then the gate returns a PASS verdict + And the notes confirm no HIGH or CRITICAL findings were detected + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate ignores context parameter + Given a valid Path target pointing to a Terraform directory + And the trivy executable is available + And a context dictionary is provided + And trivy reports no HIGH or CRITICAL severity misconfigurations + When the security baseline gate runs + Then the gate returns a PASS verdict + And the verdict is not influenced by the context parameter + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Gate invokes trivy with correct arguments + Given a valid Path target pointing to a Terraform directory + And the trivy executable is available + When the security baseline gate runs + Then trivy is invoked with the config subcommand + And trivy receives the target path as a string argument + And trivy receives the argument --format json + And trivy receives the argument --severity HIGH,CRITICAL + And trivy receives the argument --quiet + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Gate handles malformed trivy JSON output gracefully + Given a valid Path target pointing to a Terraform directory + And the trivy executable is available + And trivy output contains non-dict entries in Results array + When the security baseline gate runs + Then the gate silently skips non-dict Results entries + And the gate continues processing valid entries + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate handles malformed Misconfigurations array gracefully + Given a valid Path target pointing to a Terraform directory + And the trivy executable is available + And trivy output contains non-dict entries in Misconfigurations array + When the security baseline gate runs + Then the gate silently skips non-dict Misconfigurations entries + And the gate continues processing valid entries + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario Outline: Gate compares severity case-insensitively + Given a valid Path target pointing to a Terraform directory + And the trivy executable is available + And trivy reports a misconfiguration with severity "" + When the security baseline gate runs + Then the misconfiguration is classified as + + Examples: + | reported_severity | expected_classification | + | CRITICAL | CRITICAL | + | critical | CRITICAL | + | CrItIcAl | CRITICAL | + | HIGH | HIGH | + | high | HIGH | + | HiGh | HIGH | + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario Outline: Gate extracts finding title with fallback logic + Given a valid Path target pointing to a Terraform directory + And the trivy executable is available + And a misconfiguration has Title field "" + And the misconfiguration has ID field "" + When the security baseline gate processes the misconfiguration + Then the finding title is "" + + Examples: + | title_value | id_value | expected_title | + | Security Issue | CVE-2023-0001 | Security Issue | + | | CVE-2023-0001 | CVE-2023-0001 | + | | | finding | diff --git a/dogfood/mining-output/features/pickled_iac_validate.feature b/dogfood/mining-output/features/pickled_iac_validate.feature new file mode 100644 index 0000000..fb8ac13 --- /dev/null +++ b/dogfood/mining-output/features/pickled_iac_validate.feature @@ -0,0 +1,47 @@ +Feature: Validate Terraform configuration via CLI command + As a CLI user + I want to run terraform validate on a specified directory + So that I can verify my Terraform configurations are syntactically valid and internally consistent + + @iac-domain:terraform-validate-entry + Scenario: User validates Terraform configuration in a valid directory + Given a directory contains valid Terraform configuration files + When the user invokes validate with that directory as tf_dir + Then terraform validate executes on the specified directory + And the validation result indicates success + + @iac-domain:terraform-validate-entry + Scenario: User validates Terraform configuration with syntax errors + Given a directory contains invalid Terraform configuration files + When the user invokes validate with that directory as tf_dir + Then terraform validate executes on the specified directory + And the validation result indicates failure + And validation errors are reported to the caller + + @iac-domain:terraform-validate-entry + Scenario: User invokes validate without providing tf_dir parameter + When the user invokes validate without the tf_dir parameter + Then the command fails with an error + And the error indicates tf_dir is required + + @bdd-domain:draft-empty-story-deterministic-failure + Scenario: User invokes validate with empty tf_dir parameter + When the user invokes validate with an empty tf_dir parameter + Then the command fails with an error + And the error indicates tf_dir is required + + @iac-domain:terraform-validate-entry + Scenario: User validates specific directory different from current working directory + Given the current working directory is "/workspace/project" + And a directory "/workspace/terraform/modules/vpc" contains Terraform configuration files + When the user invokes validate with "/workspace/terraform/modules/vpc" as tf_dir + Then terraform validate executes on "/workspace/terraform/modules/vpc" + And terraform validate does not execute on "/workspace/project" + + @iac-domain:terraform-validate-entry + Scenario: Validation results are passed back to the caller + Given a directory contains Terraform configuration files + When the user invokes validate with that directory as tf_dir + Then the command completes execution + And the validation status is returned to the caller + And the validation output is available to the caller diff --git a/dogfood/mining-output/features/pickled_rules_check.feature b/dogfood/mining-output/features/pickled_rules_check.feature new file mode 100644 index 0000000..07a4ba3 --- /dev/null +++ b/dogfood/mining-output/features/pickled_rules_check.feature @@ -0,0 +1,152 @@ +Feature: Check Gherkin feature files against ruleset + + As a developer or CI system + I want to verify feature files against a ruleset + So that behavioral documentation meets defined standards + + Background: + Given a built-in ruleset "standard" exists + And a valid ruleset file "custom.yaml" exists + And a valid feature file "example.feature" exists + + @best-practices:agent-path-first-class + Scenario: User provides neither feature path nor feature glob + When the user runs check without feature_path or feature_glob + Then the command raises ClickException with message "Provide --feature or --feature-glob" + + @pickled-internal:core-llm-cache-default-on + Scenario: User provides both feature path and feature glob + When the user runs check with both feature_path and feature_glob + Then the command raises ClickException with message "Use only one of --feature or --feature-glob" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User provides non-existent ruleset file + When the user runs check with ruleset "nonexistent.yaml" and feature_path "example.feature" + Then the command raises ClickException with message "Rule set file not found: nonexistent.yaml" + + @rules-domain:coverage-union-across-features + Scenario: User provides feature glob matching no files + When the user runs check with ruleset "standard" and feature_glob "*.nonexistent" + Then the command raises ClickException with message "No feature files matched" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User provides feature glob matching directories and files + Given feature_glob "features/**/*.feature" matches files "a.feature", "b.feature" and directory "features" + When the user runs check with ruleset "standard" and feature_glob "features/**/*.feature" + Then only file paths are processed + And files are processed in sorted order + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User checks a single feature file + Given feature file "example.feature" passes coverage + When the user runs check with ruleset "standard" and feature_path "example.feature" + Then coverage_gate is called with the parsed feature, ruleset, and short name + And coverage_gate_features is not called + And the process exits with status code 0 + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User checks multiple feature files + Given feature_glob "*.feature" matches files "a.feature", "b.feature", "c.feature" + And all features pass coverage + When the user runs check with ruleset "standard" and feature_glob "*.feature" + Then coverage_gate_features is called with the list of parsed features, ruleset, and short name + And coverage_gate is not called + And stderr receives message "Union coverage across 3 feature file(s)." + And the process exits with status code 0 + + @rules-domain:coverage-union-across-features + Scenario: User checks multiple feature files in quiet mode + Given feature_glob "*.feature" matches files "a.feature", "b.feature" + And all features pass coverage + When the user runs check with ruleset "standard", feature_glob "*.feature", and quiet mode + Then stderr does not receive union coverage message + + @pickled-internal:mcp-output-fixed-json-shape + Scenario Outline: User selects output format + Given feature file "example.feature" passes coverage + When the user runs check with ruleset "standard", feature_path "example.feature", and output_format "" + Then the report uses + + Examples: + | format | renderer | + | json | render_coverage_json | + | JSON | render_coverage_json | + | Json | render_coverage_json | + | markdown | render_coverage_markdown| + | text | render_coverage_markdown| + | other | render_coverage_markdown| + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User runs in quiet mode without output file + Given feature file "example.feature" passes coverage + When the user runs check with ruleset "standard", feature_path "example.feature", and quiet mode + Then stdout receives only "PASS: checked 1 feature(s)" + And the full report is not written to stdout + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User runs in quiet mode with output file + Given feature file "example.feature" passes coverage + When the user runs check with ruleset "standard", feature_path "example.feature", quiet mode, and output "report.txt" + Then stdout receives only "PASS: checked 1 feature(s)" + And the full report is written to file "report.txt" + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User runs with output file but not quiet mode + Given feature file "example.feature" passes coverage + When the user runs check with ruleset "standard", feature_path "example.feature", and output "report.txt" + Then the full report is written to file "report.txt" + And stderr receives message "Report written to report.txt" + And the full report is not written to stdout + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User runs without output file and not quiet mode + Given feature file "example.feature" passes coverage + When the user runs check with ruleset "standard" and feature_path "example.feature" + Then the full report is written to stdout + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Coverage check passes + Given feature file "example.feature" passes coverage + When the user runs check with ruleset "standard" and feature_path "example.feature" + Then the process exits with status code 0 + And stdout receives verdict line starting with "PASS:" in quiet mode + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Coverage check fails + Given feature file "example.feature" fails coverage + When the user runs check with ruleset "standard" and feature_path "example.feature" + Then the process exits with status code 1 + And stdout receives verdict line starting with "FAIL:" in quiet mode + + @best-practices:agent-path-first-class + Scenario: Ruleset name defaults to built-in name when using built-in ruleset + Given feature file "example.feature" passes coverage + When the user runs check with ruleset "standard" and feature_path "example.feature" without ruleset_name + Then the short name is "standard" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Ruleset name defaults to file stem when using file path ruleset + Given a valid ruleset file "CustomRules.yaml" exists + And feature file "example.feature" passes coverage + When the user runs check with ruleset "CustomRules.yaml" and feature_path "example.feature" without ruleset_name + Then the short name is "customrules" + + @best-practices:agent-path-first-class + Scenario: Ruleset name is explicitly provided + Given feature file "example.feature" passes coverage + When the user runs check with ruleset "standard", feature_path "example.feature", and ruleset_name "MyRuleset" + Then the short name is "myruleset" + + @rules-domain:coverage-union-across-features + Scenario: Multiple feature files report includes sorted paths + Given feature_glob "*.feature" matches files "c.feature", "a.feature", "b.feature" + And all features pass coverage + When the user runs check with ruleset "standard" and feature_glob "*.feature" + Then the report includes path label "a.feature,b.feature,c.feature" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Quiet mode shows correct count for multiple files + Given feature_glob "*.feature" matches files "a.feature", "b.feature", "c.feature" + And all features fail coverage + When the user runs check with ruleset "standard", feature_glob "*.feature", and quiet mode + Then stdout receives only "FAIL: checked 3 feature(s)" diff --git a/dogfood/mining-output/features/pickled_rules_coverage_gate.feature b/dogfood/mining-output/features/pickled_rules_coverage_gate.feature new file mode 100644 index 0000000..f855777 --- /dev/null +++ b/dogfood/mining-output/features/pickled_rules_coverage_gate.feature @@ -0,0 +1,160 @@ +Feature: Coverage gate verifies rule coverage in Gherkin features + As a test automation or compliance tool + I want to verify that a Gherkin feature file adequately covers the rules defined in a ruleset + So that I can ensure strict rules are tested and detect unknown rule references + + Background: + Given a ruleset short name "RS" + + @rules-domain:unknown-tag-fails-gate + Scenario: Test engineer runs gate with no scenarios in feature + Given a feature with no scenarios + And a ruleset containing 2 strict rules + When the coverage gate is executed + Then the gate result verdict is FAIL + And all rules appear in unreferenced_rules + And referenced_rules is empty + And unknown_references is empty + And the notes mention unreferenced strict rules + + @rules-domain:unknown-tag-fails-gate + Scenario: Test engineer runs gate with all strict rules referenced + Given a feature with 2 scenarios + And scenario 1 has tag "RS:rule-1" + And scenario 2 has tag "RS:rule-2" + And a ruleset containing 2 strict rules with ids "rule-1" and "rule-2" + When the coverage gate is executed + Then the gate result verdict is PASS + And all strict rules appear in referenced_rules + And unreferenced_rules is empty + And unknown_references is empty + And the notes state all strict rules are referenced and no unknown tags exist + + @rules-domain:unknown-tag-fails-gate + Scenario: Test engineer runs gate with unreferenced strict rule + Given a feature with 1 scenario + And scenario 1 has tag "RS:rule-1" + And a ruleset containing 2 strict rules with ids "rule-1" and "rule-2" + When the coverage gate is executed + Then the gate result verdict is FAIL + And only rule "rule-1" appears in referenced_rules + And only rule "rule-2" appears in unreferenced_rules + And unknown_references is empty + And the notes include the count of 1 unreferenced strict rule + + @rules-domain:unknown-tag-fails-gate + Scenario: Test engineer runs gate with unknown rule reference + Given a feature with 1 scenario + And scenario 1 has tag "RS:unknown-rule" + And a ruleset containing 1 strict rule with id "rule-1" + When the coverage gate is executed + Then the gate result verdict is FAIL + And referenced_rules is empty + And rule "rule-1" appears in unreferenced_rules + And unknown_references contains tuple ("RS", "unknown-rule") + And the unknown_references tuple is sorted + And the notes include the count of 1 unknown reference + And the notes include the count of 1 unreferenced strict rule + + @rules-domain:unknown-tag-fails-gate + Scenario: Test engineer runs gate with only advisory rules unreferenced + Given a feature with 1 scenario + And scenario 1 has tag "RS:rule-1" + And a ruleset containing strict rule "rule-1" and advisory rule "rule-2" + When the coverage gate is executed + Then the gate result verdict is PASS + And only rule "rule-1" appears in referenced_rules + And only rule "rule-2" appears in unreferenced_rules + And unknown_references is empty + And the notes state all strict rules are referenced and no unknown tags exist + + @rules-domain:unknown-tag-fails-gate + Scenario: Test engineer runs gate with only informational rules unreferenced + Given a feature with 1 scenario + And scenario 1 has tag "RS:rule-1" + And a ruleset containing strict rule "rule-1" and informational rule "rule-2" + When the coverage gate is executed + Then the gate result verdict is PASS + And only rule "rule-1" appears in referenced_rules + And only rule "rule-2" appears in unreferenced_rules + And unknown_references is empty + And the notes state all strict rules are referenced and no unknown tags exist + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Test engineer inspects coverage report structure + Given a feature with 1 scenario + And scenario 1 has tag "RS:rule-1" + And a ruleset containing 1 strict rule with id "rule-1" + When the coverage gate is executed + Then the gate result gate_name is "rules.coverage" + And the gate result findings is an empty tuple + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Test engineer inspects traces for referenced rules + Given a feature with 2 scenarios + And scenario 1 has tag "RS:rule-1" + And scenario 2 has tag "RS:rule-2" + And a ruleset containing 2 strict rules with ids "rule-1" and "rule-2" + When the coverage gate is executed + Then the gate result traces contains 2 trace objects + And each referenced rule has a corresponding trace with relation "implements" + And each trace has confidence "asserted" + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Test engineer runs gate with feature having no path + Given a feature with 1 scenario and no path + And scenario 1 has tag "RS:rule-1" + And a ruleset containing 1 strict rule with id "rule-1" + When the coverage gate is executed + Then all traces have artifact_ref "" + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Test engineer runs gate with feature having empty path + Given a feature with 1 scenario and empty path + And scenario 1 has tag "RS:rule-1" + And a ruleset containing 1 strict rule with id "rule-1" + When the coverage gate is executed + Then all traces have artifact_ref "" + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Test engineer runs gate with feature having truthy path + Given a feature with 1 scenario and path "features/example.feature" + And scenario 1 has tag "RS:rule-1" + And a ruleset containing 1 strict rule with id "rule-1" + When the coverage gate is executed + Then all traces have artifact_ref "features/example.feature" + + @rules-domain:unknown-tag-fails-gate + Scenario: Test engineer runs gate with multiple unknown references + Given a feature with 2 scenarios + And scenario 1 has tag "RS:unknown-2" + And scenario 2 has tag "RS:unknown-1" + And a ruleset containing 1 strict rule with id "rule-1" + When the coverage gate is executed + Then the gate result verdict is FAIL + And unknown_references contains tuples in sorted order + And the first unknown reference is ("RS", "unknown-1") + And the second unknown reference is ("RS", "unknown-2") + + @rules-domain:unknown-tag-fails-gate + Scenario: Test engineer runs gate with multiple strict rules and mixed coverage + Given a feature with 2 scenarios + And scenario 1 has tag "RS:rule-1" + And scenario 2 has tag "RS:rule-2" + And a ruleset containing 3 strict rules with ids "rule-1", "rule-2", and "rule-3" + When the coverage gate is executed + Then the gate result verdict is FAIL + And rules "rule-1" and "rule-2" appear in referenced_rules + And only rule "rule-3" appears in unreferenced_rules + And the notes include the count of 1 unreferenced strict rule + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Test engineer runs gate with same rule referenced by multiple scenarios + Given a feature with 2 scenarios + And scenario 1 has tag "RS:rule-1" + And scenario 2 has tag "RS:rule-1" + And a ruleset containing 1 strict rule with id "rule-1" + When the coverage gate is executed + Then the gate result verdict is PASS + And only rule "rule-1" appears in referenced_rules + And unreferenced_rules is empty diff --git a/dogfood/mining-output/features/pickled_rules_coverage_gate_features.feature b/dogfood/mining-output/features/pickled_rules_coverage_gate_features.feature new file mode 100644 index 0000000..bd0fa57 --- /dev/null +++ b/dogfood/mining-output/features/pickled_rules_coverage_gate_features.feature @@ -0,0 +1,222 @@ +Feature: Coverage gate for feature files against ruleset + As a compliance engineer + I want to verify that my feature files reference all strict rules + So that I can ensure mandatory requirements are traceable to test scenarios + + Background: + Given a ruleset with the short name "test-ruleset" + + @rules-domain:unknown-tag-fails-gate + Scenario: Gate passes when all rules are non-strict + Given the ruleset contains the following rules: + | rule_id | enforcement | description | + | RULE-01 | advisory | Advisory rule one | + | RULE-02 | guidance | Guidance rule two | + And a feature file with scenarios tagged: + | scenario_name | tags | + | First test | | + When the coverage gate is evaluated + Then the gate verdict is "PASS" + And the unreferenced rules are: + | RULE-01 | + | RULE-02 | + And the referenced rules are empty + And the unknown references are empty + And the gate notes are "All strict rules in test-ruleset are referenced; no unknown reference tags." + And the findings are empty + And the gate name is "rules.coverage" + And the traces are empty + + @rules-domain:unknown-tag-fails-gate + Scenario: Gate fails when a strict rule is not referenced + Given the ruleset contains the following rules: + | rule_id | enforcement | description | + | RULE-01 | strict | Strict rule one | + | RULE-02 | advisory | Advisory rule two | + And a feature file with scenarios tagged: + | scenario_name | tags | + | First test | | + When the coverage gate is evaluated + Then the gate verdict is "FAIL" + And the unreferenced rules are: + | RULE-01 | + | RULE-02 | + And the referenced rules are empty + And the unknown references are empty + And the gate notes contain "1 strict rule(s) unreferenced" + + @rules-domain:unknown-tag-fails-gate + Scenario: Gate fails when a scenario references an unknown rule ID + Given the ruleset contains the following rules: + | rule_id | enforcement | description | + | RULE-01 | strict | Strict rule one | + And a feature file with scenarios tagged: + | scenario_name | tags | + | First test | @REF:test-ruleset:RULE-01 | + | Second test | @REF:test-ruleset:RULE-99 | + When the coverage gate is evaluated + Then the gate verdict is "FAIL" + And the referenced rules are: + | RULE-01 | + And the unreferenced rules are empty + And the unknown references are: + | test-ruleset | RULE-99 | + And the gate notes contain "1 unknown reference(s)" + + @rules-domain:unknown-tag-fails-gate + Scenario: Gate passes when all strict rules are referenced and no unknown references exist + Given the ruleset contains the following rules: + | rule_id | enforcement | description | + | RULE-01 | strict | Strict rule one | + | RULE-02 | advisory | Advisory rule two | + | RULE-03 | strict | Strict rule three | + And a feature file with scenarios tagged: + | scenario_name | tags | + | First test | @REF:test-ruleset:RULE-01 | + | Second test | @REF:test-ruleset:RULE-03 | + When the coverage gate is evaluated + Then the gate verdict is "PASS" + And the referenced rules are: + | RULE-01 | + | RULE-03 | + And the unreferenced rules are: + | RULE-02 | + And the unknown references are empty + And the gate notes are "All strict rules in test-ruleset are referenced; no unknown reference tags." + And the traces contain 2 entries + And each referenced rule has exactly one trace + And each trace has relation "implements" + And each trace has confidence "asserted" + And each trace has artifact_kind "feature" + + @rules-domain:unknown-tag-fails-gate + Scenario: Unknown references are sorted lexicographically + Given the ruleset contains the following rules: + | rule_id | enforcement | description | + | RULE-01 | strict | Strict rule one | + And a feature file with scenarios tagged: + | scenario_name | tags | + | First test | @REF:test-ruleset:RULE-01 | + | Second test | @REF:test-ruleset:RULE-99 | + | Third test | @REF:test-ruleset:RULE-10 | + | Fourth test | @REF:other-ruleset:RULE-05 | + When the coverage gate is evaluated + Then the unknown references are sorted as: + | other-ruleset | RULE-05 | + | test-ruleset | RULE-10 | + | test-ruleset | RULE-99 | + + @rules-domain:unknown-tag-fails-gate + Scenario: A rule referenced multiple times appears once in referenced rules and produces one trace + Given the ruleset contains the following rules: + | rule_id | enforcement | description | + | RULE-01 | strict | Strict rule one | + And multiple feature files with scenarios tagged: + | feature_name | scenario_name | tags | + | Feature A | Test 1 | @REF:test-ruleset:RULE-01 | + | Feature A | Test 2 | @REF:test-ruleset:RULE-01 | + | Feature B | Test 3 | @REF:test-ruleset:RULE-01 | + When the coverage gate is evaluated + Then the referenced rules contain "RULE-01" exactly once + And the traces contain 1 entry + And the trace for "RULE-01" has source_id from the ruleset + And the trace for "RULE-01" has description from the ruleset + + @rules-domain:unknown-tag-fails-gate + Scenario: Artifact reference defaults to comma-separated feature paths + Given the ruleset contains the following rules: + | rule_id | enforcement | description | + | RULE-01 | strict | Strict rule one | + And feature files with paths: + | features/login.feature | + | features/checkout.feature | + And scenarios tagged: + | scenario_name | tags | + | Test login | @REF:test-ruleset:RULE-01 | + And no artifact_ref is provided + When the coverage gate is evaluated + Then each trace has artifact_ref "features/login.feature,features/checkout.feature" + + @rules-domain:unknown-tag-fails-gate + Scenario: Artifact reference defaults to placeholder when features have no paths + Given the ruleset contains the following rules: + | rule_id | enforcement | description | + | RULE-01 | strict | Strict rule one | + And feature files without paths + And scenarios tagged: + | scenario_name | tags | + | Test one | @REF:test-ruleset:RULE-01 | + And no artifact_ref is provided + When the coverage gate is evaluated + Then each trace has artifact_ref "" + + @rules-domain:unknown-tag-fails-gate + Scenario: Findings field is always empty + Given the ruleset contains the following rules: + | rule_id | enforcement | description | + | RULE-01 | strict | Strict rule one | + And a feature file with scenarios tagged: + | scenario_name | tags | + | Test one | | + When the coverage gate is evaluated + Then the findings are empty + + @rules-domain:coverage-union-across-features + Scenario: Rule is considered referenced if any scenario across any feature references it + Given the ruleset contains the following rules: + | rule_id | enforcement | description | + | RULE-01 | strict | Strict rule one | + | RULE-02 | strict | Strict rule two | + And multiple feature files with scenarios tagged: + | feature_name | scenario_name | tags | + | Feature A | Test A1 | | + | Feature A | Test A2 | @REF:test-ruleset:RULE-01 | + | Feature B | Test B1 | @REF:test-ruleset:RULE-02 | + When the coverage gate is evaluated + Then the gate verdict is "PASS" + And the referenced rules are: + | RULE-01 | + | RULE-02 | + And the unreferenced rules are empty + + @rules-domain:unknown-tag-fails-gate + Scenario: Gate fails when both strict rules are unreferenced and unknown references exist + Given the ruleset contains the following rules: + | rule_id | enforcement | description | + | RULE-01 | strict | Strict rule one | + And a feature file with scenarios tagged: + | scenario_name | tags | + | Test one | @REF:test-ruleset:RULE-99 | + When the coverage gate is evaluated + Then the gate verdict is "FAIL" + And the unreferenced rules are: + | RULE-01 | + And the unknown references are: + | test-ruleset | RULE-99 | + And the gate notes contain "1 strict rule(s) unreferenced" + And the gate notes contain "1 unknown reference(s)" + + @rules-domain:unknown-tag-fails-gate + Scenario: Trace contains all required rule metadata from ruleset + Given the ruleset contains a rule with: + | rule_id | RULE-01 | + | enforcement | strict | + | description | Test rule description | + | source_id | test-ruleset | + | source_version| 1.0.0 | + | locator | section-2.3 | + | active_from | 2024-01-01 | + | applies_to | all systems | + | source_url | https://example.com/rules.html | + And a feature file with scenarios tagged: + | scenario_name | tags | + | Test one | @REF:test-ruleset:RULE-01 | + When the coverage gate is evaluated + Then the trace for "RULE-01" contains source_reference with: + | source_id | test-ruleset | + | source_version | 1.0.0 | + | locator | section-2.3 | + | description | Test rule description | + | active_from | 2024-01-01 | + | applies_to | all systems | + | source_url | https://example.com/rules.html | diff --git a/dogfood/mining-output/features/pickled_rules_draft.feature b/dogfood/mining-output/features/pickled_rules_draft.feature new file mode 100644 index 0000000..34da7a0 --- /dev/null +++ b/dogfood/mining-output/features/pickled_rules_draft.feature @@ -0,0 +1,89 @@ +Feature: Draft YAML rule-set from natural-language brief + As a developer or compliance engineer + I want to automatically generate a YAML rule-set document from a natural-language description + So that I can quickly create structured rules without manual YAML authoring + + @bdd-domain:gherkin-feature-header-required + Scenario: User reads brief from stdin + Given the brief content is provided via stdin + When the draft command is invoked with brief argument "-" + Then the command reads the brief text from stdin + + @best-practices:agent-path-first-class + Scenario: User reads brief from a file + Given a brief file exists at a specified path + When the draft command is invoked with that file path as the brief argument + Then the command reads UTF-8 text from that file + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: LLM client configuration is invalid + Given the PICKLED_RULES_LLM_FACTORY environment variable specifies an invalid configuration + When the draft command attempts to build the LLM client + Then the command raises a ClickException containing the configuration error message + + @pickled-internal:core-llm-cache-default-on + Scenario: Command invokes LLM with prompt containing all metadata + Given a brief text and metadata fields are provided + When the command constructs the LLM prompt + Then the prompt contains the brief text + And the prompt contains the short_name value + And the prompt contains the source_id value + And the prompt contains the applies_to value + And the prompt contains the active_from value + + @pickled-internal:core-llm-cache-default-on + Scenario: Command validates generated YAML as a rule set + Given the LLM returns a YAML response + When the command processes the response + Then the command attempts to load the YAML as a rule set + + @pickled-internal:core-llm-cache-default-on + Scenario: Command checks for forbidden tokens in generated YAML + Given the LLM returns a YAML response + When the command validates the response + Then the command checks the lowercased YAML text for forbidden tokens + And produces warnings if any forbidden tokens are found + + @pickled-internal:core-llm-cache-default-on + Scenario: User writes generated YAML to stdout + Given the output argument is None + And the LLM returns valid YAML + When the command completes + Then the generated YAML is written to stdout + + @pickled-internal:core-llm-cache-default-on + Scenario: User writes generated YAML to a file + Given the output argument is a file path + And the LLM returns valid YAML + When the command completes + Then the generated YAML is written to that file with UTF-8 encoding + + @pickled-internal:core-llm-cache-default-on + Scenario: LLM response contains rationale section + Given the LLM response contains YAML and a rationale section delimited by a sentinel + When the command processes the response + Then each rationale line is written to stderr prefixed with "rationale: " + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Validation warnings are written to stderr + Given the command detects validation warnings + When the command processes the warnings + Then each warning is written to stderr prefixed with "warning: " + + @pickled-internal:core-llm-cache-default-on + Scenario: Command exits with status 1 when validation warnings are present + Given the LLM returns YAML that triggers validation warnings + When the command completes and emits the YAML + Then the command exits with status 1 + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Command exits with status 2 on general exception + Given a non-ClickException occurs during processing + When the command handles the exception + Then the exception message is printed to stderr + And the command exits with status 2 + + Scenario: ClickException is re-raised without transformation + Given a ClickException is raised during processing + When the command handles the exception + Then the ClickException is re-raised without being caught or transformed into SystemExit(2) diff --git a/dogfood/mining-output/features/pickled_rules_list_rules.feature b/dogfood/mining-output/features/pickled_rules_list_rules.feature new file mode 100644 index 0000000..204520b --- /dev/null +++ b/dogfood/mining-output/features/pickled_rules_list_rules.feature @@ -0,0 +1,52 @@ +Feature: List rules from rule set + As a CLI user + I want to list rule identifiers from a rule set + So that I can reference specific rules for filtering, reporting, or configuration + + @rules-domain:unknown-tag-fails-gate + Scenario: User lists rules from a built-in rule set + Given a built-in rule set "standard" exists + When the user lists rules from rule set "standard" + Then the command outputs the rule IDs from the "standard" rule set + And the command exits with code 0 + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User lists rules from a custom YAML file + Given a YAML rule set file exists at "custom-rules.yaml" + When the user lists rules from rule set "custom-rules.yaml" + Then the command outputs the rule IDs from the file "custom-rules.yaml" + And the command exits with code 0 + + @best-practices:llm-drafter-temperature-zero + Scenario: User attempts to list rules from a non-existent built-in rule set + Given no built-in rule set named "nonexistent" exists + When the user lists rules from rule set "nonexistent" + Then the command produces an error indicating the rule set was not found + And the command exits with a non-zero code + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User attempts to list rules from a non-existent file path + Given no file exists at "missing-rules.yaml" + When the user lists rules from rule set "missing-rules.yaml" + Then the command produces an error indicating the rule set was not found + And the command exits with a non-zero code + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User attempts to list rules from an invalid YAML file + Given an invalid YAML file exists at "invalid-rules.yaml" + When the user lists rules from rule set "invalid-rules.yaml" + Then the command produces an error indicating the rule set is invalid + And the command exits with a non-zero code + + @rules-domain:unknown-tag-fails-gate + Scenario Outline: User lists rules from different built-in rule sets + Given a built-in rule set "" exists + When the user lists rules from rule set "" + Then the command outputs the rule IDs from the "" rule set + And the command exits with code 0 + + Examples: + | ruleset_name | + | standard | + | strict | + | minimal | diff --git a/dogfood/mining-output/features/pickled_rules_mcp.feature b/dogfood/mining-output/features/pickled_rules_mcp.feature new file mode 100644 index 0000000..ac7f6a0 --- /dev/null +++ b/dogfood/mining-output/features/pickled_rules_mcp.feature @@ -0,0 +1,26 @@ +Feature: MCP command group entry point + As a CLI user or automation script + I want to invoke the mcp command group + So that I can access MCP server-related subcommands + + # TODO: Verify that this command is properly registered as a Click command group + # TODO: Verify that subcommands can be attached to this command group + # TODO: Verify behavior when invoked with --help flag + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer invokes mcp function directly + When the mcp function is called with no arguments + Then it returns None + And it raises no exceptions + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer invokes mcp function and verifies no side effects + Given the system state before invoking mcp + When the mcp function is called with no arguments + Then no I/O operations are performed + And no global state is modified + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer verifies mcp function signature + Then the mcp function accepts zero parameters + And the mcp function completes synchronously diff --git a/dogfood/mining-output/features/pickled_rules_run_all.feature b/dogfood/mining-output/features/pickled_rules_run_all.feature new file mode 100644 index 0000000..8885b2f --- /dev/null +++ b/dogfood/mining-output/features/pickled_rules_run_all.feature @@ -0,0 +1,160 @@ +Feature: Run All Coverage Gates for Configured Rulesets + As a quality assurance engineer + I want to verify that all strict enforcement rules are covered by feature scenarios + So that I can ensure traceability between requirements and tests + + Background: + Given a working directory with BDD feature files + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with missing configuration file + Given the pickled.ruleset.yaml file does not exist + When the gate runs against the working directory + Then a single WARN result is returned + And the result has gate_name "rules.coverage" + And the result notes mention "missing pickled.ruleset.yaml or ruleset/rulesets key" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with empty configuration + Given the pickled.ruleset.yaml file is empty + When the gate runs against the working directory + Then a single WARN result is returned + And the result has gate_name "rules.coverage" + And the result notes mention "missing pickled.ruleset.yaml or ruleset/rulesets key" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with both ruleset keys present + Given the pickled.ruleset.yaml file contains both "ruleset" and "rulesets" keys + When the gate runs against the working directory + Then a single FAIL result is returned + And the result notes describe mutual exclusivity violation + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with non-string ruleset value + Given the pickled.ruleset.yaml file contains "ruleset" key with a non-string value + When the gate runs against the working directory + Then a single FAIL result is returned + And the result notes contain a validation message + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with non-list rulesets value + Given the pickled.ruleset.yaml file contains "rulesets" key with a non-list value + When the gate runs against the working directory + Then a single FAIL result is returned + And the result notes contain a validation message + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with empty rulesets list + Given the pickled.ruleset.yaml file contains "rulesets" key with an empty list + When the gate runs against the working directory + Then a single FAIL result is returned + And the result notes require at least one entry + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with malformed ruleset entry + Given the pickled.ruleset.yaml file contains "rulesets" with a non-dict entry + When the gate runs against the working directory + Then a single FAIL result is returned + And the result notes contain a descriptive validation message + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with missing path in ruleset entry + Given the pickled.ruleset.yaml file contains "rulesets" with an entry missing "path" + When the gate runs against the working directory + Then a single FAIL result is returned + And the result notes contain a descriptive validation message + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with duplicate short names + Given the pickled.ruleset.yaml file contains "rulesets" with duplicate short_name values + When the gate runs against the working directory + Then a single FAIL result is returned + And the result notes identify the duplicate positions + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with no matching feature files + Given the pickled.ruleset.yaml file is valid + And no feature files match the configured glob pattern + When the gate runs against the working directory + Then a single WARN result is returned + And the result notes mention "no feature files" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with single ruleset configured + Given the pickled.ruleset.yaml file contains a single "ruleset" key + And the ruleset file exists + And feature files exist + When the gate runs against the working directory + Then a result is returned with gate_name "rules.coverage" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with multiple rulesets configured + Given the pickled.ruleset.yaml file contains "rulesets" with multiple entries + And all ruleset files exist + And feature files exist + When the gate runs against the working directory + Then results are returned for each ruleset + And each result has gate_name "rules.coverage.{short_name}" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with missing ruleset file + Given the pickled.ruleset.yaml file references a ruleset that does not exist + When the gate runs against the working directory + Then a FAIL result is returned + And the result notes indicate the missing file path + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with invalid ruleset YAML + Given the pickled.ruleset.yaml file is valid + And a referenced ruleset file contains invalid YAML + When the gate runs against the working directory + Then a FAIL result is returned + And the result has gate_name "rules.load.{short_name}" + And the result notes contain the validation error message + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with all strict rules referenced + Given the pickled.ruleset.yaml file is valid + And all ruleset files exist and are valid + And feature scenarios reference all strict-enforcement rules + And no unknown rule references exist + When the gate runs against the working directory + Then a PASS result is returned + And the result includes traces for each referenced rule + And each trace has relation "implements" + And each trace has artifact_kind "feature" + And each trace has confidence "asserted" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with unreferenced strict rules + Given the pickled.ruleset.yaml file is valid + And all ruleset files exist and are valid + And one or more strict-enforcement rules are not referenced by any scenario + When the gate runs against the working directory + Then a FAIL result is returned + And the result notes contain the count of unreferenced rules + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with unknown rule references + Given the pickled.ruleset.yaml file is valid + And all ruleset files exist and are valid + And feature scenarios reference rule IDs not in the ruleset + When the gate runs against the working directory + Then a FAIL result is returned + And the result notes contain the count of unknown references + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Quality engineer runs coverage with both unreferenced and unknown rules + Given the pickled.ruleset.yaml file is valid + And all ruleset files exist and are valid + And some strict-enforcement rules are unreferenced + And some scenario tags reference unknown rule IDs + When the gate runs against the working directory + Then a FAIL result is returned + And the result notes contain both unreferenced and unknown rule counts + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate always returns non-empty result list + Given any valid or invalid configuration state + When the gate runs against the working directory + Then at least one GateResult object is returned diff --git a/dogfood/mining-output/features/pickled_schema_check.feature b/dogfood/mining-output/features/pickled_schema_check.feature new file mode 100644 index 0000000..c7c3fca --- /dev/null +++ b/dogfood/mining-output/features/pickled_schema_check.feature @@ -0,0 +1,109 @@ +Feature: Schema endpoint coverage check command + + Background: + Given a valid OpenAPI 3.1 specification file "api-spec.yaml" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User provides neither feature_dir nor feature_glob + When the user runs the check command with spec "api-spec.yaml" and no feature source + Then the command raises a ClickException with message "Provide --feature-dir or --feature-glob" + + @pickled-internal:core-llm-cache-default-on + Scenario: User provides both feature_dir and feature_glob + Given a directory "features/" containing feature files + When the user runs the check command with spec "api-spec.yaml" and both feature_dir "features/" and feature_glob "tests/**/*.feature" + Then the command raises a ClickException with message "Use only one of --feature-dir or --feature-glob" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User provides feature_dir with no matching feature files + Given an empty directory "empty-features/" + When the user runs the check command with spec "api-spec.yaml" and feature_dir "empty-features/" + Then the command raises a ClickException with message "No feature files matched" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User provides feature_glob with no matching feature files + Given a glob pattern "nonexistent/**/*.feature" that matches no files + When the user runs the check command with spec "api-spec.yaml" and feature_glob "nonexistent/**/*.feature" + Then the command raises a ClickException with message "No feature files matched" + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: All schema endpoint tags match the OpenAPI specification + Given a directory "features/" containing feature files with @schema:endpoint tags + And all @schema:endpoint tags reference endpoints defined in "api-spec.yaml" + When the user runs the check command with spec "api-spec.yaml" and feature_dir "features/" + Then the command writes JSON to stdout with verdict "PASS" + And the JSON output contains keys "gate", "verdict", "notes", and "findings" + And the command exits with status code 0 + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: At least one schema endpoint tag does not match the OpenAPI specification + Given a directory "features/" containing feature files with @schema:endpoint tags + And at least one @schema:endpoint tag references an endpoint not defined in "api-spec.yaml" + When the user runs the check command with spec "api-spec.yaml" and feature_dir "features/" + Then the command writes JSON to stdout with verdict "FAIL" + And the JSON output contains keys "gate", "verdict", "notes", and "findings" + And the findings array includes objects with "tag" and "source" fields for each missing endpoint + And the command exits with status code 2 + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: OpenAPI specification file is OpenAPI 2.0 + Given an OpenAPI 2.0 specification file "swagger-spec.yaml" with "swagger" field + And a directory "features/" containing feature files + When the user runs the check command with spec "swagger-spec.yaml" and feature_dir "features/" + Then the command raises a SchemaParseError + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Specification file is not valid YAML or JSON + Given a file "invalid-spec.yaml" that is not valid YAML or JSON + And a directory "features/" containing feature files + When the user runs the check command with spec "invalid-spec.yaml" and feature_dir "features/" + Then the command raises a SchemaParseError + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Specification file lacks openapi version field + Given a valid YAML file "no-version-spec.yaml" without an "openapi" field + And a directory "features/" containing feature files + When the user runs the check command with spec "no-version-spec.yaml" and feature_dir "features/" + Then the command raises a SchemaParseError + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Verdict is WARN + Given a directory "features/" containing feature files with @schema:endpoint tags + And the schema coverage gate returns a verdict of "WARN" + When the user runs the check command with spec "api-spec.yaml" and feature_dir "features/" + Then the command writes JSON to stdout with verdict "WARN" + And the JSON output contains keys "gate", "verdict", "notes", and "findings" + And the command exits with status code 1 + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Feature files discovered via feature_dir are sorted by path + Given a directory "features/" containing multiple feature files + When the user runs the check command with spec "api-spec.yaml" and feature_dir "features/" + Then the command processes feature files in sorted path order + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Feature files discovered via feature_glob are sorted by path + Given multiple feature files matching the glob pattern "features/**/*.feature" + When the user runs the check command with spec "api-spec.yaml" and feature_glob "features/**/*.feature" + Then the command processes feature files in sorted path order + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Feature glob pattern expands recursively and excludes directories + Given a directory structure with feature files at multiple levels + And the glob pattern "features/**/*.feature" + When the user runs the check command with spec "api-spec.yaml" and feature_glob "features/**/*.feature" + Then only files matching the pattern are included + And directories are excluded from the match + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario Outline: Valid OpenAPI 3.x versions are accepted + Given a valid OpenAPI specification file "spec.yaml" + And a directory "features/" containing feature files + When the user runs the check command with spec "spec.yaml" and feature_dir "features/" + Then the command processes the specification without raising SchemaParseError + + Examples: + | version | + | 3.0 | + | 3.1 | + | 3.2 | diff --git a/dogfood/mining-output/features/pickled_schema_draft.feature b/dogfood/mining-output/features/pickled_schema_draft.feature new file mode 100644 index 0000000..1ad11bf --- /dev/null +++ b/dogfood/mining-output/features/pickled_schema_draft.feature @@ -0,0 +1,138 @@ +Feature: Draft OpenAPI path item from Gherkin scenario + As a developer + I want to generate OpenAPI 3.1 path item specifications from Gherkin scenarios + So that I can automate API documentation from BDD scenarios + + Background: + Given a valid Gherkin scenario file exists + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer drafts path item to stdout + Given the output parameter is not specified + When the draft command is executed with method "GET" and endpoint "/users" + Then valid OpenAPI path item YAML is printed to stdout + And the YAML is formatted with sort_keys set to false + And the YAML is formatted with default_flow_style set to false + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer drafts path item to file + Given the output parameter specifies a file path + When the draft command is executed with method "POST" and endpoint "/users" + Then the OpenAPI path item YAML is written to the specified file as UTF-8 + And a confirmation message "Wrote {output}" is printed to stderr + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Developer uses custom LLM factory via environment variable + Given the PICKLED_SCHEMA_LLM_FACTORY environment variable is set to "mymodule:my_factory" + When the draft command is executed + Then the command imports "mymodule" and invokes "my_factory" + And the factory result is used as the LLM client + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer provides malformed LLM factory configuration + Given the PICKLED_SCHEMA_LLM_FACTORY environment variable is set to "mymodule_no_colon" + When the draft command is executed + Then a ClickException is raised about the required format + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Developer uses default LLM provider + Given the PICKLED_SCHEMA_LLM_FACTORY environment variable is not set + And the PICKLED_LLM_PROVIDER environment variable is not set + When the draft command is executed + Then the command uses "anthropic" as the default provider + And pickled_core's build_client is invoked with loaded configuration + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: Developer specifies custom LLM provider + Given the PICKLED_SCHEMA_LLM_FACTORY environment variable is not set + And the PICKLED_LLM_PROVIDER environment variable is set to "openai" + When the draft command is executed + Then the command uses "openai" as the provider + And pickled_core's build_client is invoked with loaded configuration + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: LLM client configuration fails + Given the PICKLED_SCHEMA_LLM_FACTORY environment variable is not set + When pickled_core's build_client raises a ConfigError + Then a ClickException is raised with the configuration error details + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: LLM produces valid OpenAPI on first attempt + Given the LLM client is configured + When the LLM returns valid OpenAPI path item YAML + Then the YAML is validated against OpenAPI 3.1.0 specification + And the validated YAML is returned as output + And no retry attempts are made + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: LLM produces invalid YAML and succeeds on retry + Given the LLM client is configured + When the LLM returns invalid YAML on the first attempt + And the LLM returns valid OpenAPI path item YAML on the second attempt + Then the command retries with validation feedback in the prompt + And the validated YAML from the second attempt is returned as output + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: LLM produces non-mapping YAML output + Given the LLM client is configured + When the LLM returns YAML that is not a dictionary on all attempts + Then a SchemaValidationError is raised about expecting a mapping + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: LLM produces output failing OpenAPI validation + Given the LLM client is configured + When the LLM returns output that fails OpenAPI validation on the first attempt + Then the second prompt includes the previous validation error + And the command retries up to 3 times total + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: All validation attempts fail + Given the LLM client is configured + When the LLM produces invalid output on all 3 attempts + Then a SchemaValidationError is raised with the last failure details + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: OpenAPI validator is not installed + Given the pickled-schema[openapi] extra is not installed + When the draft command attempts validation + Then a SchemaValidationError is raised requesting pickled-schema[openapi] + + Scenario: Method parameter is uppercased for prompts + Given the method parameter is "get" + When the prompt template is rendered + Then the method is uppercased to "GET" in the prompt + And the method is uppercased in the endpoint_id + + @schema-domain:openapi-validate-deterministic + Scenario: Method parameter is lowercased for validation + Given the method parameter is "GET" + When the validation envelope is constructed + Then the method is lowercased to "get" in the OpenAPI paths object + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: LLM returns bare operation object + Given the LLM client is configured + When the LLM returns a dictionary without a path wrapper + And the dictionary contains operation properties + Then the operation is unwrapped and used as the path item + And the path item is validated successfully + + @pickled-internal:mcp-subserver-llm-client-wired + Scenario: LLM returns operation wrapped with HTTP method key + Given the LLM client is configured + When the LLM returns a single-key dictionary with the HTTP method + And the value contains operation properties + Then the operation is unwrapped from the method key + And the path item is validated successfully + + @best-practices:agent-path-first-class + Scenario: Gherkin file cannot be read + Given the gherkin_file parameter specifies a non-existent path + When the draft command is executed + Then a file I/O error is raised + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Output file cannot be written + Given the output parameter specifies a write-protected path + When the draft command attempts to write the YAML + Then a file I/O error is raised diff --git a/dogfood/mining-output/features/pickled_schema_mcp.feature b/dogfood/mining-output/features/pickled_schema_mcp.feature new file mode 100644 index 0000000..1ec6fd5 --- /dev/null +++ b/dogfood/mining-output/features/pickled_schema_mcp.feature @@ -0,0 +1,22 @@ +Feature: MCP CLI Command Group + As a developer or operator + I want to invoke the mcp CLI command group + So that I can access MCP server subcommands + + # TODO: Verify that the mcp command is properly decorated as a Click command group + # TODO: Verify that subcommands can be registered under this command group + # TODO: Clarify how the empty function body integrates with the CLI framework + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer invokes mcp command with no arguments + When the mcp command is invoked with no arguments + Then the command completes without raising an exception + And the command returns None + And no console output is produced + And no file system operations are performed + And no global state is modified + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer verifies mcp command signature + Given the mcp command function + Then the function signature requires zero parameters diff --git a/dogfood/mining-output/features/pickled_schema_parse.feature b/dogfood/mining-output/features/pickled_schema_parse.feature new file mode 100644 index 0000000..78ab49d --- /dev/null +++ b/dogfood/mining-output/features/pickled_schema_parse.feature @@ -0,0 +1,95 @@ +```gherkin +Feature: Parse schema file metadata + As an operator or developer + I want to quickly inspect a schema file's metadata + So that I can verify format and see basic statistics without full validation + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer parses OpenAPI YAML file without format argument + Given a valid OpenAPI 3.1 YAML file exists at "spec.yaml" + When the developer runs parse command with file "spec.yaml" and no format argument + Then the command outputs JSON to stdout + And the JSON field "format" matches the detected OpenAPI version from file content + And the JSON field "endpoint_id" is null + And the JSON field "source" is "file" + And the JSON field "content_bytes" equals the UTF-8 byte length of the file content + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer parses JSON Schema file without format argument + Given a valid JSON Schema file exists at "schema.json" + And the JSON root element is an object + When the developer runs parse command with file "schema.json" and no format argument + Then the command outputs JSON to stdout + And the JSON field "format" is "json_schema_2020_12" + And the JSON field "endpoint_id" is null + And the JSON field "source" is "file" + And the JSON field "content_bytes" equals the UTF-8 byte length of the file content + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer parses Proto3 file without format argument + Given a valid Proto3 file exists at "service.proto" + When the developer runs parse command with file "service.proto" and no format argument + Then the command outputs JSON to stdout + And the JSON field "format" is "proto3" + And the JSON field "endpoint_id" is null + And the JSON field "source" is "file" + And the JSON field "content_bytes" equals the UTF-8 byte length of the base64-encoded descriptor + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer parses file with explicit format argument + Given a valid OpenAPI 3.1 YAML file exists at "spec.txt" + When the developer runs parse command with file "spec.txt" and format "openapi_3_1" + Then the command outputs JSON to stdout + And the JSON field "format" matches the detected OpenAPI version from file content + And the JSON field "endpoint_id" is null + And the JSON field "source" is "file" + + @data-domain:migration-drift-gate + Scenario: Developer attempts to parse file with unrecognized extension and no format + Given a file exists at "schema.txt" + When the developer runs parse command with file "schema.txt" and no format argument + Then the command raises a ClickException mentioning inability to infer format + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer parses JSON file with array root element + Given a file exists at "schema.json" + And the JSON root element is an array + When the developer runs parse command with file "schema.json" and no format argument + Then the command raises a SchemaParseError + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer attempts to parse nonexistent file + Given no file exists at "missing.yaml" + When the developer runs parse command with file "missing.yaml" + Then the command raises a file-not-found exception before format processing + + @bdd-domain:draft-output-parses-via-pytest-bdd + Scenario: Developer parses Proto3 file with syntax error + Given a Proto3 file exists at "invalid.proto" + And the Proto3 file contains syntax errors + When the developer runs parse command with file "invalid.proto" and no format argument + Then the command raises a RuntimeError with protoc error message + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer parses OpenAPI file where detected version differs from inferred + Given a YAML file exists at "spec.yaml" + And the file content specifies OpenAPI version "3.0.0" + When the developer runs parse command with file "spec.yaml" and no format argument + Then the command outputs JSON to stdout + And the JSON field "format" is "openapi_3_0" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario Outline: Developer parses files with various extension-to-format mappings + Given a valid file exists at "" + When the developer runs parse command with file "" and no format argument + Then the command outputs JSON to stdout + And the JSON field "format" is "" + And the JSON field "endpoint_id" is null + And the JSON field "source" is "file" + + Examples: + | schema_type | filename | expected_format | + | OpenAPI 3.1 | spec.yml | openapi_3_1 | + | JSON Schema | data.json | json_schema_2020_12 | + | Proto3 | api.proto | proto3 | +``` diff --git a/dogfood/mining-output/features/pickled_schema_run_all.feature b/dogfood/mining-output/features/pickled_schema_run_all.feature new file mode 100644 index 0000000..2620b08 --- /dev/null +++ b/dogfood/mining-output/features/pickled_schema_run_all.feature @@ -0,0 +1,204 @@ +Feature: Schema validation and coverage gate + As a build engineer + I want to validate OpenAPI specs and measure schema coverage + So that I can ensure API specifications are correct and comprehensively tested + + Background: + Given the pickled-schema package is installed with openapi extras + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User runs gate in directory with no spec files + Given a working directory with no specs subdirectory + When the run_all gate is executed + Then a single result is returned + And the result has gate_name "schema.openapi" + And the result has verdict WARN + And the result has notes "no specs/*.yaml" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User runs gate in directory with empty specs subdirectory + Given a working directory with an empty specs subdirectory + When the run_all gate is executed + Then a single result is returned + And the result has gate_name "schema.openapi" + And the result has verdict WARN + And the result has notes "no specs/*.yaml" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User validates a valid OpenAPI 3.0 specification + Given a working directory with specs subdirectory + And a file "specs/example.yaml" containing valid OpenAPI 3.0 content + When the run_all gate is executed + Then a PASS result with gate_name "schema.openapi.validate.example.yaml" is returned + And the notes contain the relative path to the spec + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User validates a valid OpenAPI 3.1 specification + Given a working directory with specs subdirectory + And a file "specs/api.yaml" containing valid OpenAPI 3.1 content + When the run_all gate is executed + Then a PASS result with gate_name "schema.openapi.validate.api.yaml" is returned + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: User validates a valid OpenAPI 3.2 specification + Given a working directory with specs subdirectory + And a file "specs/service.yml" containing valid OpenAPI 3.2 content + When the run_all gate is executed + Then a PASS result with gate_name "schema.openapi.validate.service.yml" is returned + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User attempts to validate a malformed YAML file + Given a working directory with specs subdirectory + And a file "specs/broken.yaml" containing invalid YAML syntax + When the run_all gate is executed + Then a FAIL result with gate_name "schema.openapi.validate.broken.yaml" is returned + And the notes contain the parse error message + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User attempts to validate a spec with non-dict root + Given a working directory with specs subdirectory + And a file "specs/list.yaml" containing a YAML list at root level + When the run_all gate is executed + Then a FAIL result with gate_name "schema.openapi.validate.list.yaml" is returned + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User attempts to validate an OpenAPI 2.0 specification + Given a working directory with specs subdirectory + And a file "specs/swagger.yaml" containing a spec with "swagger" field + When the run_all gate is executed + Then a FAIL result with gate_name "schema.openapi.validate.swagger.yaml" is returned + And the notes mention OpenAPI 2.0 is unsupported + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User attempts to validate a spec without openapi field + Given a working directory with specs subdirectory + And a file "specs/missing.yaml" containing a dict without "openapi" field + When the run_all gate is executed + Then a FAIL result with gate_name "schema.openapi.validate.missing.yaml" is returned + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario Outline: User attempts to validate a spec with unsupported OpenAPI version + Given a working directory with specs subdirectory + And a file "specs/version.yaml" with openapi field "" + When the run_all gate is executed + Then a FAIL result with gate_name "schema.openapi.validate.version.yaml" is returned + + Examples: + | version | + | 2.0 | + | 4.0 | + | 3.3 | + | 1.0 | + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User validates multiple valid specs + Given a working directory with specs subdirectory + And a file "specs/a-first.yaml" containing valid OpenAPI 3.0 content + And a file "specs/b-second.yaml" containing valid OpenAPI 3.1 content + And a file "specs/c-third.yaml" containing valid OpenAPI 3.2 content + When the run_all gate is executed + Then a PASS result with gate_name "schema.openapi.validate.a-first.yaml" is returned + And a PASS result with gate_name "schema.openapi.validate.b-second.yaml" is returned + And a PASS result with gate_name "schema.openapi.validate.c-third.yaml" is returned + And a WARN result with gate_name "schema.openapi.note" is returned + And the WARN notes indicate multiple specs were found + And the WARN notes indicate the first spec will be used for coverage + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User runs gate with valid spec but no feature files + Given a working directory with specs subdirectory + And a file "specs/api.yaml" containing valid OpenAPI 3.0 content + And no features subdirectory exists + When the run_all gate is executed + Then a PASS result with gate_name "schema.openapi.validate.api.yaml" is returned + And no result with gate_name containing "schema.coverage" is returned + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User runs gate with valid spec and empty features directory + Given a working directory with specs subdirectory + And a file "specs/api.yaml" containing valid OpenAPI 3.0 content + And an empty features subdirectory + When the run_all gate is executed + Then a PASS result with gate_name "schema.openapi.validate.api.yaml" is returned + And no result with gate_name containing "schema.coverage" is returned + + # TODO: SchemaCoverageGate behavior is unresolved; assuming it returns verdict, findings, notes + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User runs gate with valid spec and feature files + Given a working directory with specs subdirectory + And a file "specs/api.yaml" containing valid OpenAPI 3.0 content + And a features subdirectory with feature files + When the run_all gate is executed + Then a PASS result with gate_name "schema.openapi.validate.api.yaml" is returned + And a result with gate_name "schema.coverage" is returned + And the coverage result verdict is from the SchemaCoverageGate + And the coverage result findings are from the SchemaCoverageGate + And the coverage result notes default to the spec relative path if SchemaCoverageGate provides none + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User runs gate without openapi-spec-validator installed + Given a working directory with specs subdirectory + And a file "specs/api.yaml" containing valid OpenAPI 3.0 content + And the openapi-spec-validator package is not installed + When the run_all gate is executed + Then a SchemaValidationError is raised + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User runs gate and receives results in deterministic order + Given a working directory with specs subdirectory + And a file "specs/z-last.yaml" containing valid OpenAPI 3.0 content + And a file "specs/a-first.yaml" containing valid OpenAPI 3.1 content + And a file "specs/m-middle.yaml" containing valid OpenAPI 3.2 content + And a features subdirectory with feature files + When the run_all gate is executed + Then results are returned in order: validation results lexicographically by filename, then multi-spec warning, then coverage result + And the validation results appear as "a-first.yaml", "m-middle.yaml", "z-last.yaml" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: User validates spec file with .json extension + Given a working directory with specs subdirectory + And a file "specs/api.json" containing valid OpenAPI 3.0 JSON content + When the run_all gate is executed + Then a PASS result with gate_name "schema.openapi.validate.api.json" is returned + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User validates spec file with unknown extension + Given a working directory with specs subdirectory + And a file "specs/api.txt" containing valid OpenAPI 3.0 YAML content + When the run_all gate is executed + Then the file is parsed as YAML + And a PASS result with gate_name "schema.openapi.validate.api.txt" is returned + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User runs gate where spec file cannot be read + Given a working directory with specs subdirectory + And a file "specs/unreadable.yaml" that triggers an OSError when read + When the run_all gate is executed + Then a FAIL result with gate_name "schema.openapi.validate.unreadable.yaml" is returned + And the notes contain the OS error message + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User runs gate with workdir as string path + Given a working directory path provided as a string + And a file "specs/api.yaml" containing valid OpenAPI 3.0 content + When the run_all gate is executed with the string path + Then the path is resolved to an absolute Path + And a PASS result with gate_name "schema.openapi.validate.api.yaml" is returned + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User runs gate with workdir as Path object + Given a working directory path provided as a Path object + And a file "specs/api.yaml" containing valid OpenAPI 3.0 content + When the run_all gate is executed with the Path object + Then the path is resolved to an absolute Path + And a PASS result with gate_name "schema.openapi.validate.api.yaml" is returned + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: User validates mix of valid and invalid specs + Given a working directory with specs subdirectory + And a file "specs/valid.yaml" containing valid OpenAPI 3.0 content + And a file "specs/invalid.yaml" containing invalid YAML syntax + When the run_all gate is executed + Then a PASS result with gate_name "schema.openapi.validate.valid.yaml" is returned + And a FAIL result with gate_name "schema.openapi.validate.invalid.yaml" is returned + And only the valid spec is used for coverage analysis diff --git a/dogfood/mining-output/features/pickled_schema_schemaambiguitygate.feature b/dogfood/mining-output/features/pickled_schema_schemaambiguitygate.feature new file mode 100644 index 0000000..dbdc783 --- /dev/null +++ b/dogfood/mining-output/features/pickled_schema_schemaambiguitygate.feature @@ -0,0 +1,146 @@ +Feature: Schema Ambiguity Gate + As a schema validation pipeline orchestrator + I want to detect ambiguities between Gherkin specifications and schema YAML + So that unclear or inconsistent schema mappings are flagged for review + + Background: + Given a SchemaAmbiguityGate instance + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when target is not a SchemaArtifact + Given a target that is not a SchemaArtifact instance + And a context with valid "gherkin_context" + When the gate runs + Then the gate returns a FAIL verdict + And the notes indicate a type mismatch with the received type + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when context is None + Given a SchemaArtifact target + And a context that is None + When the gate runs + Then the gate returns a FAIL verdict + And the notes indicate that "gherkin_context" is required + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when context lacks "gherkin_context" key + Given a SchemaArtifact target + And a context dictionary without "gherkin_context" key + When the gate runs + Then the gate returns a FAIL verdict + And the notes indicate that "gherkin_context" is missing + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario Outline: Gate fails when "gherkin_context" is invalid string + Given a SchemaArtifact target + And a context with "gherkin_context" set to + When the gate runs + Then the gate returns a FAIL verdict + And the notes indicate that "gherkin_context" must be a non-empty string + + Examples: + | invalid_value | + | empty string | + | whitespace string | + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when "gherkin_context" is not a string type + Given a SchemaArtifact target + And a context with "gherkin_context" set to a non-string value + When the gate runs + Then the gate returns a FAIL verdict + And the notes indicate that "gherkin_context" must be a string + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate fails when LLM returns invalid JSON + Given a SchemaArtifact target with schema YAML content + And a context with valid "gherkin_context" + And the LLM returns a response that is not valid JSON + When the gate runs + Then the gate returns a FAIL verdict + And the notes state "LLM returned malformed JSON" + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate fails when LLM returns malformed markdown fences + Given a SchemaArtifact target with schema YAML content + And a context with valid "gherkin_context" + And the LLM returns a response with unmatched or malformed markdown fences + When the gate runs + Then the gate returns a FAIL verdict + And the notes state "LLM returned malformed JSON" + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate fails when parsed JSON lacks "ambiguities" key + Given a SchemaArtifact target with schema YAML content + And a context with valid "gherkin_context" + And the LLM returns valid JSON without "ambiguities" key + When the gate runs + Then the gate returns a FAIL verdict + And the notes indicate the missing "ambiguities" list field + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate fails when "ambiguities" value is not a list + Given a SchemaArtifact target with schema YAML content + And a context with valid "gherkin_context" + And the LLM returns valid JSON with "ambiguities" as a non-list type + When the gate runs + Then the gate returns a FAIL verdict + And the notes indicate that "ambiguities" must be a list + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate passes when ambiguities list is empty + Given a SchemaArtifact target with schema YAML content + And a context with valid "gherkin_context" + And the LLM returns valid JSON with an empty "ambiguities" list + When the gate runs + Then the gate returns a PASS verdict + And the notes state "No ambiguities reported." + And no findings are attached + + @pickled-internal:core-llm-cache-default-on + Scenario Outline: Gate warns when ambiguities are detected + Given a SchemaArtifact target with schema YAML content + And a context with valid "gherkin_context" + And the LLM returns valid JSON with ambiguities in the list + When the gate runs + Then the gate returns a WARN verdict + And the notes include the count of ambiguities + And the findings tuple contains ambiguity items + + Examples: + | count | + | 1 | + | 3 | + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate sends rendered prompt with gherkin context and schema YAML to LLM + Given a SchemaArtifact target with schema YAML content + And a context with valid "gherkin_context" + When the gate runs + Then the gate renders a prompt containing the Gherkin context + And the prompt contains the schema YAML content from the target + And the prompt is sent to the LLM completion function + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Gate instructs LLM to return only JSON without markdown or commentary + Given a SchemaArtifact target with schema YAML content + And a context with valid "gherkin_context" + When the gate runs + Then the gate sends a system instruction to the LLM + And the instruction requires a single JSON object response + And the instruction prohibits markdown fences and extra commentary + + @pickled-internal:core-llm-cache-default-on + Scenario Outline: Gate parses LLM response with various JSON formats + Given a SchemaArtifact target with schema YAML content + And a context with valid "gherkin_context" + And the LLM returns + When the gate runs + Then the gate successfully parses the JSON response + And extracts the "ambiguities" field + + Examples: + | response_format | + | plain JSON without fences | + | JSON wrapped in triple-backtick fences | + | JSON wrapped in fences with "json" label | diff --git a/dogfood/mining-output/features/pickled_schema_schemacoveragegate.feature b/dogfood/mining-output/features/pickled_schema_schemacoveragegate.feature new file mode 100644 index 0000000..e3d8623 --- /dev/null +++ b/dogfood/mining-output/features/pickled_schema_schemacoveragegate.feature @@ -0,0 +1,177 @@ +Feature: SchemaCoverageGate validates endpoint references in Gherkin feature files + + Background: + Given a SchemaCoverageGate instance + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate rejects non-dictionary target + When the gate runs with target "not-a-dict" and context containing feature paths + Then the verdict is FAIL + And the notes describe a type mismatch for the target + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate rejects None target + When the gate runs with target None and context containing feature paths + Then the verdict is FAIL + And the notes describe a type mismatch for the target + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate rejects list target + When the gate runs with target [] and context containing feature paths + Then the verdict is FAIL + And the notes describe a type mismatch for the target + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate rejects None context + When the gate runs with a valid OpenAPI spec target and context None + Then the verdict is FAIL + And the notes require feature_paths or feature_texts context keys + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate rejects context missing both required keys + When the gate runs with a valid OpenAPI spec target and context {} + Then the verdict is FAIL + And the notes require feature_paths or feature_texts context keys + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate rejects context with empty lists for both keys + When the gate runs with a valid OpenAPI spec target and context containing empty feature_paths and empty feature_texts + Then the verdict is FAIL + And the notes require feature_paths or feature_texts context keys + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate passes when all endpoint tags match OpenAPI spec paths and methods + Given an OpenAPI spec with path "/users" and method "get" + And an OpenAPI spec with path "/users/{id}" and method "post" + And a feature file containing endpoint tags for "get /users" and "post /users/{id}" + When the gate runs with the OpenAPI spec target and context containing the feature file + Then the verdict is PASS + And the notes state "All @schema:endpoint tags have matching paths." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when endpoint tag references path not in spec + Given an OpenAPI spec with path "/users" and method "get" + And a feature file containing an endpoint tag for "get /unknown" + When the gate runs with the OpenAPI spec target and context containing the feature file + Then the verdict is FAIL + And the findings contain a SchemaCoverageFinding for the missing tag + And the notes list the missing endpoint with tag and source + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when endpoint tag references method not defined for path + Given an OpenAPI spec with path "/users" and method "get" + And a feature file containing an endpoint tag for "post /users" + When the gate runs with the OpenAPI spec target and context containing the feature file + Then the verdict is FAIL + And the findings contain a SchemaCoverageFinding for the missing tag + And the notes list the missing endpoint with tag and source + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when spec paths key is missing + Given an OpenAPI spec without a "paths" key + And a feature file containing an endpoint tag for "get /users" + When the gate runs with the OpenAPI spec target and context containing the feature file + Then the verdict is FAIL + And the findings contain a SchemaCoverageFinding for the missing tag + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate fails when spec paths value is not a dictionary + Given an OpenAPI spec with "paths" set to "not-a-dict" + And a feature file containing an endpoint tag for "get /users" + When the gate runs with the OpenAPI spec target and context containing the feature file + Then the verdict is FAIL + And the findings contain a SchemaCoverageFinding for the missing tag + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate performs case-insensitive HTTP method matching + Given an OpenAPI spec with path "/users" and method "get" + And a feature file containing an endpoint tag for "GET /users" + When the gate runs with the OpenAPI spec target and context containing the feature file + Then the verdict is PASS + And the notes state "All @schema:endpoint tags have matching paths." + + @pickled-internal:core-llm-cache-default-on + Scenario: Gate reads feature files from feature_paths with UTF-8 encoding + Given an OpenAPI spec with path "/users" and method "get" + And a UTF-8 encoded feature file on disk containing an endpoint tag for "get /users" + When the gate runs with the OpenAPI spec target and context containing the file path in feature_paths + Then the verdict is PASS + And the notes state "All @schema:endpoint tags have matching paths." + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate processes feature content from feature_texts + Given an OpenAPI spec with path "/users" and method "get" + And feature text containing an endpoint tag for "get /users" + When the gate runs with the OpenAPI spec target and context containing the feature text in feature_texts + Then the verdict is PASS + And the notes state "All @schema:endpoint tags have matching paths." + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Gate silently skips non-string items in feature_texts + Given an OpenAPI spec with path "/users" and method "get" + And a feature_texts list containing non-string items and valid feature text + When the gate runs with the OpenAPI spec target and context containing the feature_texts + Then only string items are processed for endpoint tags + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Gate silently skips non-path-convertible items in feature_paths + Given an OpenAPI spec with path "/users" and method "get" + And a feature_paths list containing non-list items and valid file paths + When the gate runs with the OpenAPI spec target and context containing the feature_paths + Then only valid path items are processed for endpoint tags + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate reports multiple missing endpoints as semicolon-separated list + Given an OpenAPI spec with path "/users" and method "get" + And a feature file containing endpoint tags for "post /users" and "get /orders" + When the gate runs with the OpenAPI spec target and context containing the feature file + Then the verdict is FAIL + And the findings contain SchemaCoverageFindings for both missing tags + And the notes list both missing endpoints separated by semicolons + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate includes tag and source in SchemaCoverageFinding objects + Given an OpenAPI spec with path "/users" and method "get" + And a feature file containing an endpoint tag for "post /unknown" + When the gate runs with the OpenAPI spec target and context containing the feature file + Then the verdict is FAIL + And each finding has tag and source attributes + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate uses feature index as source identifier for feature_texts items + Given an OpenAPI spec with path "/users" and method "get" + And feature_texts containing an endpoint tag for "post /unknown" at index 2 + When the gate runs with the OpenAPI spec target and context containing the feature_texts + Then the verdict is FAIL + And the finding source is "" + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate uses file path string as source identifier for feature_paths items + Given an OpenAPI spec with path "/users" and method "get" + And a feature file at "/path/to/test.feature" containing an endpoint tag for "post /unknown" + When the gate runs with the OpenAPI spec target and context containing the file path in feature_paths + Then the verdict is FAIL + And the finding source is the string representation of the file path + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate returns GateResult with gate_name field set + Given an OpenAPI spec with path "/users" and method "get" + And a feature file containing an endpoint tag for "get /users" + When the gate runs with the OpenAPI spec target and context containing the feature file + Then the GateResult gate_name equals the gate instance name + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Gate returns findings only for failure modes with missing endpoints + Given an OpenAPI spec with path "/users" and method "get" + And a feature file containing an endpoint tag for "post /unknown" + When the gate runs with the OpenAPI spec target and context containing the feature file + Then the verdict is FAIL + And the findings tuple contains SchemaCoverageFinding objects + + @pickled-internal:stdio-hygiene-gates-log-stderr + Scenario: Gate returns no findings on successful validation + Given an OpenAPI spec with path "/users" and method "get" + And a feature file containing an endpoint tag for "get /users" + When the gate runs with the OpenAPI spec target and context containing the feature file + Then the verdict is PASS + And the findings are empty or not present diff --git a/dogfood/mining-output/features/pickled_schema_validate.feature b/dogfood/mining-output/features/pickled_schema_validate.feature new file mode 100644 index 0000000..a03c5ed --- /dev/null +++ b/dogfood/mining-output/features/pickled_schema_validate.feature @@ -0,0 +1,114 @@ +Feature: Schema file validation via CLI command + As a developer using pickled-schema + I want to validate schema files against their format specifications + So that I can ensure my schema files are correctly formatted + + Background: + Given the pickled-schema CLI is available + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer validates an OpenAPI schema with .yaml extension + Given a file "api-spec.yaml" containing a valid OpenAPI 3.1 schema + When the developer validates the file + Then the validation succeeds + And the output is JSON with valid true and format "openapi_3_1" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer validates an OpenAPI schema with .yml extension + Given a file "api-spec.yml" containing a valid OpenAPI 3.1 schema + When the developer validates the file + Then the validation succeeds + And the output is JSON with valid true and format "openapi_3_1" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer validates a JSON Schema file + Given a file "data-schema.json" containing a valid JSON Schema 2020-12 + When the developer validates the file + Then the validation succeeds + And the output is JSON with valid true and format "json_schema_2020_12" + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer validates a Protocol Buffers file + Given a file "messages.proto" containing a valid Protocol Buffers 3 schema + When the developer validates the file + Then the validation succeeds + And the output is JSON with valid true and format "proto3" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario Outline: Developer validates files with case-insensitive extensions + Given a file "" containing a valid schema + When the developer validates the file + Then the validation succeeds + And the output is JSON with valid true + + Examples: + | filename | + | spec.YAML | + | spec.Yaml | + | spec.YML | + | spec.JSON | + | spec.PROTO | + + @pickled-internal:core-llm-cache-default-on + Scenario: Developer attempts to validate a file with unrecognized extension + Given a file "schema.xml" exists + When the developer validates the file + Then the command fails with ClickException + And the error message contains "cannot infer format from extension '.xml'" + And the error message contains "use --format" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer validates OpenAPI file without openapi-spec-validator installed + Given a file "api-spec.yaml" containing a valid OpenAPI 3.1 schema + And the openapi-spec-validator library is not installed + When the developer validates the file + Then the command fails with SchemaValidationError + And the error message is "install pickled-schema[openapi] for OpenAPI validation" + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer validates an invalid OpenAPI schema + Given a file "api-spec.yaml" containing an invalid OpenAPI 3.1 schema + And the openapi-spec-validator library is installed + When the developer validates the file + Then the command fails with SchemaValidationError + And the error message is "OpenAPI validation failed" + And the error includes validation details from openapi-spec-validator + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: Developer validates an invalid JSON Schema + Given a file "data-schema.json" containing an invalid JSON Schema + When the developer validates the file + Then the command fails with validation error from validate_json_schema_document + + @bdd-domain:draft-warnings-field-populated-on-failure + Scenario: Developer validates an invalid Protocol Buffers file + Given a file "messages.proto" containing invalid Protocol Buffers syntax + When the developer validates the file + Then the command fails with validation error from parse_proto_file + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: System outputs validation result to standard output + Given a file "api-spec.yaml" containing a valid OpenAPI 3.1 schema + When the developer validates the file + Then the result is written to standard output via click.echo + And the output is valid JSON + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: System loads OpenAPI file as dictionary for validation + Given a file "api-spec.yaml" containing a valid OpenAPI 3.1 schema + When the developer validates the file + Then the file content is loaded into a dictionary + And the dictionary is passed to openapi-spec-validator.validate + + @pickled-internal:mcp-output-fixed-json-shape + Scenario: System delegates JSON Schema validation to validate_json_schema_document + Given a file "data-schema.json" containing a valid JSON Schema 2020-12 + When the developer validates the file + Then the file is loaded into a dictionary + And validate_json_schema_document is called with the dictionary + + @best-practices:agent-path-first-class + Scenario: System delegates Protocol Buffers validation to parse_proto_file + Given a file "messages.proto" containing a valid Protocol Buffers 3 schema + When the developer validates the file + Then parse_proto_file is called with the file path diff --git a/dogfood/mining-output/features/rules_check_ruleset_coverage.feature b/dogfood/mining-output/features/rules_check_ruleset_coverage.feature new file mode 100644 index 0000000..3ada20f --- /dev/null +++ b/dogfood/mining-output/features/rules_check_ruleset_coverage.feature @@ -0,0 +1,69 @@ +Feature: MCP tool checks Gherkin feature coverage against YAML ruleset + + Background: + Given an MCP client needs to verify feature file coverage + + @rules-domain:coverage-union-across-features + Scenario: Client checks feature coverage against a valid ruleset + Given a ruleset YAML text containing coverage requirements + And one or more Gherkin feature file text contents + And a ruleset short name identifier + When the client invokes the rules_check_ruleset_coverage tool with ruleset_yaml_text, feature_texts, and ruleset_short_name + Then the tool parses the YAML ruleset definition + And the tool parses the Gherkin feature file contents + And the tool compares the parsed features against the ruleset requirements + And the tool returns coverage analysis results indicating whether features meet ruleset requirements + + @rules-domain:coverage-union-across-features + Scenario: Client attempts to pass filesystem paths instead of file contents + Given a client attempts to provide filesystem paths rather than file contents + When the client invokes the rules_check_ruleset_coverage tool with path strings + Then the tool rejects the request or fails safely + And the tool does not perform filesystem reads based on the supplied paths + + @rules-domain:coverage-union-across-features + Scenario: Tool prevents arbitrary file read attacks + Given a malicious client attempts to exploit parser error messages + When the client provides input designed to trigger path-based file reads + Then the tool does not read files from the filesystem based on user-supplied paths + And the tool does not expose file contents through parser error messages + + @rules-domain:coverage-union-across-features + Scenario Outline: Tool validates required parameters + Given a client invokes the tool with omitted + When the tool processes the request + Then the tool fails with a parameter validation error + + Examples: + | missing_parameter | + | ruleset_yaml_text | + | feature_texts | + | ruleset_short_name | + + @rules-domain:coverage-union-across-features + Scenario: Client checks coverage with multiple feature files + Given a ruleset YAML text containing coverage requirements + And multiple Gherkin feature file text contents + And a ruleset short name identifier + When the client invokes the rules_check_ruleset_coverage tool + Then the tool parses all provided feature file contents + And the tool aggregates coverage across all features + And the tool returns combined coverage analysis results + + @rules-domain:coverage-union-across-features + Scenario: Tool handles malformed YAML ruleset + Given a ruleset YAML text with invalid YAML syntax + And valid Gherkin feature file text contents + And a ruleset short name identifier + When the client invokes the rules_check_ruleset_coverage tool + Then the tool returns an error indicating YAML parsing failure + And the tool does not expose filesystem information in the error message + + @rules-domain:coverage-union-across-features + Scenario: Tool handles malformed Gherkin feature content + Given a valid ruleset YAML text + And Gherkin feature file text contents with invalid syntax + And a ruleset short name identifier + When the client invokes the rules_check_ruleset_coverage tool + Then the tool returns an error indicating Gherkin parsing failure + And the tool does not expose filesystem information in the error message diff --git a/dogfood/mining-output/features/rules_draft_ruleset_from_brief.feature b/dogfood/mining-output/features/rules_draft_ruleset_from_brief.feature new file mode 100644 index 0000000..897add1 --- /dev/null +++ b/dogfood/mining-output/features/rules_draft_ruleset_from_brief.feature @@ -0,0 +1,100 @@ +Feature: Draft ruleset from brief description + As an agent or client using pickled-rules + I want to generate a complete ruleset from a brief text description + So that I can quickly create draft rulesets with appropriate metadata + + Background: + Given a multi-ruleset workspace environment + + @best-practices:agent-path-first-class + Scenario: Agent drafts ruleset with all required parameters + When the agent invokes rules_draft_ruleset_from_brief with: + | parameter | value | + | brief_text | Require security review for API changes | + | ruleset_short_name | api_security | + | source_id | SEC-2024-001 | + | applies_to | api_endpoints | + | active_from | 2024-01-01 | + Then the tool responds without parameter validation errors + And the response represents a ruleset structure + + @best-practices:agent-path-first-class + Scenario: Agent drafts ruleset incorporating short name + When the agent invokes rules_draft_ruleset_from_brief with: + | parameter | value | + | brief_text | Validate input formats | + | ruleset_short_name | input_validation | + | source_id | DEV-100 | + | applies_to | user_inputs | + | active_from | 2024-02-01 | + Then the generated ruleset incorporates "input_validation" as its short name + + @best-practices:agent-path-first-class + Scenario: Agent drafts ruleset incorporating source identifier + When the agent invokes rules_draft_ruleset_from_brief with: + | parameter | value | + | brief_text | Enforce code style standards | + | ruleset_short_name | code_style | + | source_id | STYLE-500 | + | applies_to | source_code | + | active_from | 2024-03-01 | + Then the generated ruleset incorporates "STYLE-500" as its source identifier + + @rules-domain:coverage-union-across-features + Scenario: Agent drafts ruleset incorporating scope + When the agent invokes rules_draft_ruleset_from_brief with: + | parameter | value | + | brief_text | Check documentation completeness | + | ruleset_short_name | doc_checks | + | source_id | DOC-200 | + | applies_to | markdown_files | + | active_from | 2024-04-01 | + Then the generated ruleset incorporates "markdown_files" as its scope + + @best-practices:agent-path-first-class + Scenario: Agent drafts ruleset incorporating temporal constraint + When the agent invokes rules_draft_ruleset_from_brief with: + | parameter | value | + | brief_text | Restrict deprecated API usage | + | ruleset_short_name | deprecation_policy | + | source_id | API-300 | + | applies_to | legacy_endpoints | + | active_from | 2024-06-15 | + Then the generated ruleset incorporates "2024-06-15" as its activation date + + @best-practices:agent-path-first-class + Scenario: Agent drafts ruleset with content relating to brief + When the agent invokes rules_draft_ruleset_from_brief with: + | parameter | value | + | brief_text | All database queries must use parameterization | + | ruleset_short_name | sql_safety | + | source_id | DB-400 | + | applies_to | database_layer | + | active_from | 2024-05-01 | + Then the generated ruleset content relates to parameterized database queries + + @best-practices:agent-path-first-class + Scenario Outline: Agent invokes tool with missing required parameter + When the agent invokes rules_draft_ruleset_from_brief with omitted + Then the tool returns an appropriate error + + Examples: + | missing_parameter | + | brief_text | + | ruleset_short_name | + | source_id | + | applies_to | + | active_from | + + @best-practices:agent-path-first-class + Scenario: Agent drafts multiple rulesets in workspace + Given an existing ruleset "data_validation" is already in the workspace + When the agent invokes rules_draft_ruleset_from_brief with: + | parameter | value | + | brief_text | Audit trail for all mutations | + | ruleset_short_name | audit_logging | + | source_id | AUDIT-600 | + | applies_to | data_mutations | + | active_from | 2024-07-01 | + Then the new ruleset is created within the multi-ruleset workspace context + And the existing ruleset "data_validation" remains unaffected diff --git a/dogfood/mining-output/features/rules_list_rules.feature b/dogfood/mining-output/features/rules_list_rules.feature new file mode 100644 index 0000000..35c1840 --- /dev/null +++ b/dogfood/mining-output/features/rules_list_rules.feature @@ -0,0 +1,57 @@ +Feature: List rules from YAML rule set + As a developer or CI/CD pipeline + I want to extract rule definitions from YAML rule set content + So that I can inspect and validate rule structure without executing them + + @rules-domain:unknown-tag-fails-gate + Scenario: Developer lists rules from valid YAML rule set + Given a valid YAML rule set containing 3 rules + When the ruleset_yaml_text is provided to the list rules tool + Then the tool returns a collection of 3 rule summaries + And each rule summary includes the rule name or identifier + + @rules-domain:unknown-tag-fails-gate + Scenario: Developer lists rules preserving original order + Given a valid YAML rule set with rules in a specific order + When the ruleset_yaml_text is provided to the list rules tool + Then the returned rule summaries maintain the original YAML order + + @rules-domain:unknown-tag-fails-gate + Scenario: Developer handles YAML with no rules + Given a valid YAML document containing no rule definitions + When the ruleset_yaml_text is provided to the list rules tool + Then the tool returns an empty collection + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer handles invalid YAML input + Given malformed YAML content that cannot be parsed + When the ruleset_yaml_text is provided to the list rules tool + Then the tool handles the error gracefully without crashing + And the tool returns an error indicator or empty result + + @rules-domain:unknown-tag-fails-gate + Scenario: Developer receives MCP-compatible output + Given a valid YAML rule set containing rules + When the ruleset_yaml_text is provided to the list rules tool + Then the tool returns results in MCP-compatible format + + # TODO: Confirm exact structure of rule summaries returned (beyond name/identifier) + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: Developer inspects rule summary content + Given a valid YAML rule set with a rule containing metadata + When the ruleset_yaml_text is provided to the list rules tool + Then each rule summary includes at minimum the rule identifier + And the summaries conform to pickled-rules package schema expectations + + @rules-domain:unknown-tag-fails-gate + Scenario Outline: Developer lists rules from various YAML structures + Given a YAML rule set with rules + When the ruleset_yaml_text is provided to the list rules tool + Then the tool returns a collection of rule summaries + + Examples: + | rule_count | + | 0 | + | 1 | + | 5 | + | 50 | diff --git a/dogfood/mining-output/features/schema_check_schema_coverage.feature b/dogfood/mining-output/features/schema_check_schema_coverage.feature new file mode 100644 index 0000000..c3d4338 --- /dev/null +++ b/dogfood/mining-output/features/schema_check_schema_coverage.feature @@ -0,0 +1,66 @@ +Feature: Schema coverage verification for endpoint tags + + As a QA engineer or CI pipeline script + I want to verify that all @schema:endpoint tags in feature files correspond to actual API endpoints + So that tests stay synchronized with the API schema and runtime failures are prevented + + Background: + Given an OpenAPI specification document is provided as spec_yaml + And one or more Cucumber feature file contents are provided as feature_texts + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: QA engineer verifies complete coverage when all endpoint tags exist in specification + Given spec_yaml defines endpoints "GET /users", "POST /users", and "GET /users/{id}" + And feature_texts contain tags "@schema:endpoint:GET_users", "@schema:endpoint:POST_users", and "@schema:endpoint:GET_users_id" + When the schema coverage check is performed + Then the check returns success + And no missing endpoint tags are reported + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: QA engineer detects missing endpoints when tags reference undefined endpoints + Given spec_yaml defines endpoints "GET /users" and "POST /users" + And feature_texts contain tags "@schema:endpoint:GET_users", "@schema:endpoint:POST_users", and "@schema:endpoint:DELETE_users" + When the schema coverage check is performed + Then the check returns failure + And the missing endpoint tag "@schema:endpoint:DELETE_users" is reported + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: CI pipeline validates multiple missing endpoint tags are all reported + Given spec_yaml defines endpoint "GET /users" + And feature_texts contain tags "@schema:endpoint:GET_users", "@schema:endpoint:POST_orders", and "@schema:endpoint:DELETE_products" + When the schema coverage check is performed + Then the check returns failure + And the missing endpoint tags "@schema:endpoint:POST_orders" and "@schema:endpoint:DELETE_products" are reported + + @rules-domain:coverage-union-across-features + Scenario: QA engineer validates empty feature files without error + Given spec_yaml defines endpoints "GET /users" and "POST /users" + And feature_texts are empty or contain no @schema:endpoint tags + When the schema coverage check is performed + Then the check returns success + And no missing endpoint tags are reported + + @schema-domain:openapi-validate-deterministic + Scenario: CI pipeline handles malformed OpenAPI specification + Given spec_yaml contains invalid YAML syntax + And feature_texts contain tag "@schema:endpoint:GET_users" + When the schema coverage check is performed + Then the check returns an error + And an appropriate error message about malformed spec_yaml is provided + + @schema-domain:openapi-validate-deterministic + Scenario: CI pipeline handles empty OpenAPI specification + Given spec_yaml is empty or contains no endpoint definitions + And feature_texts contain tag "@schema:endpoint:GET_users" + When the schema coverage check is performed + Then the check returns failure + And the missing endpoint tag "@schema:endpoint:GET_users" is reported + + @pickled-internal:core-model-from-config-not-hardcoded + Scenario: QA engineer confirms tool only validates tag-to-spec correspondence + Given spec_yaml defines endpoint "GET /users" + And feature_texts contain tag "@schema:endpoint:GET_users" with incomplete or incorrect test implementation + When the schema coverage check is performed + Then the check returns success + And endpoint implementation quality is not validated + And test correctness is not validated diff --git a/dogfood/mining-output/features/schema_draft_openapi_endpoint.feature b/dogfood/mining-output/features/schema_draft_openapi_endpoint.feature new file mode 100644 index 0000000..4f1c91a --- /dev/null +++ b/dogfood/mining-output/features/schema_draft_openapi_endpoint.feature @@ -0,0 +1,85 @@ +Feature: Schema draft OpenAPI endpoint tool + As an API documentation engineer + I want to generate OpenAPI 3.1 path items from Gherkin scenarios + So that I can convert behavioral specifications into machine-readable API schema + + Background: + Given the schema_draft_openapi_endpoint tool is available + + @schema-domain:openapi-validate-deterministic + Scenario: Client generates OpenAPI path item from complete Gherkin scenario + Given the HTTP method is "GET" + And the endpoint path is "/users/{id}" + And the Gherkin text describes a user retrieval scenario + When the client invokes the tool with method, path, and gherkin_text + Then the tool returns a response containing an OpenAPI path item structure + And the path item conforms to OpenAPI 3.1 schema specifications + And the path item corresponds to the "GET" method + And the path item corresponds to the "/users/{id}" path + + @best-practices:agent-path-first-class + Scenario: Client attempts to generate path item without required method parameter + Given the endpoint path is "/users/{id}" + And the Gherkin text describes a user retrieval scenario + When the client invokes the tool without the method parameter + Then the tool returns an error indicating the method parameter is required + + @best-practices:agent-path-first-class + Scenario: Client attempts to generate path item without required path parameter + Given the HTTP method is "POST" + And the Gherkin text describes a user creation scenario + When the client invokes the tool without the path parameter + Then the tool returns an error indicating the path parameter is required + + @best-practices:agent-path-first-class + Scenario: Client attempts to generate path item without required gherkin_text parameter + Given the HTTP method is "DELETE" + And the endpoint path is "/users/{id}" + When the client invokes the tool without the gherkin_text parameter + Then the tool returns an error indicating the gherkin_text parameter is required + + @best-practices:agent-path-first-class + Scenario: Gherkin text content influences drafted path item structure + Given the HTTP method is "POST" + And the endpoint path is "/orders" + And the Gherkin text describes request parameters, response codes, and data schemas + When the client invokes the tool with method, path, and gherkin_text + Then the returned path item structure reflects elements parsed from the Gherkin text + And the path item includes parameters derived from the Gherkin scenario + And the path item includes responses derived from the Gherkin scenario + + @schema-domain:openapi-validate-deterministic + Scenario Outline: Client generates path items for different HTTP methods + Given the HTTP method is "" + And the endpoint path is "" + And the Gherkin text describes an endpoint scenario + When the client invokes the tool with method, path, and gherkin_text + Then the tool returns a response containing an OpenAPI path item structure + And the path item corresponds to the "" method + And the path item corresponds to the "" path + + Examples: + | method | path | + | GET | /products | + | POST | /products | + | PUT | /products/{id} | + | PATCH | /products/{id} | + | DELETE | /products/{id} | + + # TODO: Clarify how SchemaAmbiguityGate integration affects the tool's response + @best-practices:agent-path-first-class + Scenario: Tool integrates with SchemaAmbiguityGate during drafting process + Given the HTTP method is "GET" + And the endpoint path is "/items" + And the Gherkin text contains potentially ambiguous schema definitions + When the client invokes the tool with method, path, and gherkin_text + Then the SchemaAmbiguityGate is invoked as part of the drafting process + + # TODO: Clarify how SchemaCoverageGate integration affects the tool's response + @best-practices:agent-path-first-class + Scenario: Tool integrates with SchemaCoverageGate during drafting process + Given the HTTP method is "POST" + And the endpoint path is "/items" + And the Gherkin text describes partial endpoint behavior + When the client invokes the tool with method, path, and gherkin_text + Then the SchemaCoverageGate is invoked as part of the drafting process diff --git a/dogfood/mining-output/features/schema_validate_openapi_spec.feature b/dogfood/mining-output/features/schema_validate_openapi_spec.feature new file mode 100644 index 0000000..244a28b --- /dev/null +++ b/dogfood/mining-output/features/schema_validate_openapi_spec.feature @@ -0,0 +1,36 @@ +Feature: Validate OpenAPI specification documents + As an MCP client + I want to validate OpenAPI specification documents in YAML format + So that I can ensure API specifications conform to OpenAPI standards before using them + + @schema-domain:openapi-validate-deterministic + Scenario: MCP client validates a valid OpenAPI 3.x YAML document + Given a valid OpenAPI 3.x YAML document + When the client validates the OpenAPI specification + Then the validation completes without errors + And the validation output indicates the specification conforms to OpenAPI standards + + @schema-domain:openapi-validate-deterministic + Scenario: MCP client validates an invalid OpenAPI YAML document + Given an invalid OpenAPI YAML document + When the client validates the OpenAPI specification + Then the validation reports specific validation failures + And the validation output indicates the specification does not conform to OpenAPI standards + + @schema-domain:openapi-validate-deterministic + Scenario: MCP client validates malformed YAML + Given a malformed YAML document + When the client validates the OpenAPI specification + Then the validation reports a parsing error + + @schema-domain:openapi-validate-deterministic + Scenario: MCP client validates an empty string + Given an empty string as the spec_yaml parameter + When the client validates the OpenAPI specification + Then the validation reports a validation error + + @schema-domain:openapi-validate-deterministic + Scenario: MCP client invokes validation without required spec_yaml parameter + Given the spec_yaml parameter is omitted + When the client attempts to validate the OpenAPI specification + Then the tool fails due to missing required parameter diff --git a/dogfood/mining-output/inventory.json b/dogfood/mining-output/inventory.json new file mode 100644 index 0000000..6787d02 --- /dev/null +++ b/dogfood/mining-output/inventory.json @@ -0,0 +1,3341 @@ +{ + "schema_version": "1", + "generated_at": "2026-05-28T13:39:04Z", + "repo_root": "/Users/bartlomiejrosa/Projects/PORTFOLIO/pickled-spec", + "totals": { + "packages": 7, + "cli_commands": 46, + "mcp_tools": 19, + "gates": 16, + "adrs": 7, + "workspaces": 2 + }, + "packages": { + "pickled-bdd": { + "version": "0.1.0.dev0", + "description": "LLM-to-Gherkin bridge with deterministic verification.", + "scripts": { + "pickled-bdd": "pickled_bdd.cli:main" + }, + "entry_points": { + "pickled.mcp.subservers": { + "bdd": "pickled_bdd.mcp_cli:build_server" + }, + "pickled.gates": { + "bdd": "pickled_bdd.gates_runner:run_all" + } + }, + "optional_dependencies": [ + "mcp" + ], + "cli_commands": [ + { + "full_name": "ambiguity", + "help": "Run the ambiguity gate (alias for ``check --gate ambiguity``).", + "params": [ + { + "name": "feature_file", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "feature_file" + ] + } + ], + "is_group": false + }, + { + "full_name": "check", + "help": "Run compensating gates against a .feature file.", + "params": [ + { + "name": "feature_file", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "feature_file" + ] + }, + { + "name": "gate", + "type": "Option", + "required": false, + "default": "ambiguity", + "help": "Which gate to run.", + "secondary_opts": [], + "opts": [ + "--gate" + ] + } + ], + "is_group": false + }, + { + "full_name": "draft", + "help": "Draft a .feature file from a user story (Markdown).", + "params": [ + { + "name": "story_file", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "story_file" + ] + }, + { + "name": "output", + "type": "Option", + "required": false, + "default": null, + "help": "Write the drafted feature to this path. Defaults to stdout.", + "secondary_opts": [], + "opts": [ + "-o", + "--output" + ] + } + ], + "is_group": false + }, + { + "full_name": "mcp", + "help": "MCP server commands.", + "params": [], + "is_group": true + }, + { + "full_name": "mcp serve", + "help": "Run the pickled-bdd MCP server.", + "params": [ + { + "name": "transport", + "type": "Option", + "required": false, + "default": "stdio", + "help": "", + "secondary_opts": [], + "opts": [ + "--transport" + ] + }, + { + "name": "host", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--host" + ] + }, + { + "name": "port", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--port" + ] + }, + { + "name": "allow_public", + "type": "Option", + "required": false, + "default": "False", + "help": "", + "secondary_opts": [], + "opts": [ + "--allow-public" + ] + } + ], + "is_group": false + }, + { + "full_name": "serve", + "help": "Deprecated alias for ``pickled-bdd mcp serve``.", + "params": [], + "is_group": false + } + ], + "mcp_tools": [ + { + "name": "bdd_draft_feature_from_story", + "description": "Draft a Gherkin .feature file from a natural-language user story.", + "input_schema": { + "additionalProperties": false, + "properties": { + "story_text": { + "type": "string" + } + }, + "required": [ + "story_text" + ], + "type": "object" + } + }, + { + "name": "bdd_validate_feature_ambiguity", + "description": "Run the ambiguity gate against a Gherkin .feature file.", + "input_schema": { + "additionalProperties": false, + "properties": { + "feature_text": { + "type": "string" + } + }, + "required": [ + "feature_text" + ], + "type": "object" + } + } + ], + "gates": [ + { + "kind": "class", + "name": "AmbiguityGate.run", + "module": "pickled_bdd.gates.ambiguity", + "file": "packages/pickled-bdd/src/pickled_bdd/gates/ambiguity.py", + "line": 44, + "returns": "GateResult", + "docstring_summary": "Compensating gate: flags scenarios that admit multiple implementations." + }, + { + "kind": "function", + "name": "run_all", + "module": "pickled_bdd.gates_runner", + "file": "packages/pickled-bdd/src/pickled_bdd/gates_runner.py", + "line": 12, + "returns": "list[GateResult]", + "docstring_summary": "Parse Gherkin under ``features/`` (AmbiguityGate skipped without LLM)." + } + ] + }, + "pickled-core": { + "version": "0.1.0.dev0", + "description": "Shared types, gate protocol, and infrastructure for the pickled-* family.", + "scripts": { + "pickled-spec": "pickled_core.cli:main" + }, + "entry_points": {}, + "optional_dependencies": [ + "all-providers", + "anthropic", + "gemini", + "mcp", + "mine", + "openai" + ], + "cli_commands": [ + { + "full_name": "check-all", + "help": "Run workspace gates from every pickled-* package against a directory.", + "params": [ + { + "name": "workdir", + "type": "Option", + "required": false, + "default": ".", + "help": "Workspace root (features/, specs/, infra/, migrations/).", + "secondary_opts": [], + "opts": [ + "--workdir" + ] + }, + { + "name": "warn_ok", + "type": "Option", + "required": false, + "default": "False", + "help": "Exit 0 when only WARN verdicts occur (e.g. terraform or LLM not configured).", + "secondary_opts": [], + "opts": [ + "--warn-ok" + ] + } + ], + "is_group": false + }, + { + "full_name": "mcp", + "help": "Run the umbrella MCP server (all family packages mounted).", + "params": [ + { + "name": "transport", + "type": "Option", + "required": false, + "default": "stdio", + "help": "", + "secondary_opts": [], + "opts": [ + "--transport" + ] + }, + { + "name": "host", + "type": "Option", + "required": false, + "default": null, + "help": "Bind host for HTTP transport.", + "secondary_opts": [], + "opts": [ + "--host" + ] + }, + { + "name": "port", + "type": "Option", + "required": false, + "default": null, + "help": "Bind port for HTTP transport.", + "secondary_opts": [], + "opts": [ + "--port" + ] + }, + { + "name": "allow_public", + "type": "Option", + "required": false, + "default": "False", + "help": "Required to bind 0.0.0.0.", + "secondary_opts": [], + "opts": [ + "--allow-public" + ] + } + ], + "is_group": false + }, + { + "full_name": "mine", + "help": "Mine a Python project for surfaces, stories, features, and gate results.", + "params": [], + "is_group": true + }, + { + "full_name": "mine all", + "help": "Run inventory → code → stories → features → tag → evaluate → report.", + "params": [ + { + "name": "target", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "target" + ] + }, + { + "name": "output_dir", + "type": "Option", + "required": false, + "default": "./mining-output/", + "help": "Mining output directory.", + "secondary_opts": [], + "opts": [ + "--output" + ] + }, + { + "name": "quick", + "type": "Option", + "required": false, + "default": "True", + "help": "Quick mode (default) or interactive prompts.", + "secondary_opts": [ + "--interactive" + ], + "opts": [ + "--quick" + ] + }, + { + "name": "verbose", + "type": "Option", + "required": false, + "default": "False", + "help": "Extra logging to stderr.", + "secondary_opts": [], + "opts": [ + "--verbose" + ] + }, + { + "name": "surfaces", + "type": "Option", + "required": false, + "default": null, + "help": "Comma-separated filter on package name or surface-id substring.", + "secondary_opts": [], + "opts": [ + "--surfaces" + ] + }, + { + "name": "max_parallel", + "type": "Option", + "required": false, + "default": "4", + "help": "Max parallel LLM calls in quick mode (stories, features).", + "secondary_opts": [], + "opts": [ + "--max-parallel" + ] + }, + { + "name": "no_mcp", + "type": "Option", + "required": false, + "default": "False", + "help": "", + "secondary_opts": [], + "opts": [ + "--no-mcp" + ] + }, + { + "name": "mcp_timeout", + "type": "Option", + "required": false, + "default": "30.0", + "help": "", + "secondary_opts": [], + "opts": [ + "--mcp-timeout" + ] + }, + { + "name": "ruleset_dir", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--ruleset-dir" + ] + }, + { + "name": "ruleset_config", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--ruleset-config" + ] + }, + { + "name": "overwrite_stories", + "type": "Option", + "required": false, + "default": "False", + "help": "", + "secondary_opts": [], + "opts": [ + "--overwrite-stories" + ] + }, + { + "name": "overwrite_features", + "type": "Option", + "required": false, + "default": "False", + "help": "", + "secondary_opts": [], + "opts": [ + "--overwrite-features" + ] + }, + { + "name": "depth", + "type": "Option", + "required": false, + "default": "body", + "help": "How much source to collect per surface.", + "secondary_opts": [], + "opts": [ + "--depth" + ] + }, + { + "name": "callee_scope", + "type": "Option", + "required": false, + "default": "same-package", + "help": "Which intra-project callees to follow.", + "secondary_opts": [], + "opts": [ + "--callee-scope" + ] + }, + { + "name": "max_hops", + "type": "Option", + "required": false, + "default": "2", + "help": "Callee expansion depth (callgraph only).", + "secondary_opts": [], + "opts": [ + "--max-hops" + ] + }, + { + "name": "max_callees", + "type": "Option", + "required": false, + "default": "8", + "help": "Hard cap on collected callee units per surface.", + "secondary_opts": [], + "opts": [ + "--max-callees" + ] + }, + { + "name": "max_code_lines", + "type": "Option", + "required": false, + "default": "400", + "help": "Hard cap on total source lines per surface.", + "secondary_opts": [], + "opts": [ + "--max-code-lines" + ] + }, + { + "name": "detect_cycles", + "type": "Option", + "required": false, + "default": "True", + "help": "Write code-context/_cycles.json from observed edges.", + "secondary_opts": [ + "--no-detect-cycles" + ], + "opts": [ + "--detect-cycles" + ] + } + ], + "is_group": false + }, + { + "full_name": "mine code", + "help": "Stage 2: extract code context per surface from inventory.json.", + "params": [ + { + "name": "target", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "target" + ] + }, + { + "name": "output_dir", + "type": "Option", + "required": false, + "default": "./mining-output/", + "help": "Mining output directory.", + "secondary_opts": [], + "opts": [ + "--output" + ] + }, + { + "name": "verbose", + "type": "Option", + "required": false, + "default": "False", + "help": "Extra logging to stderr.", + "secondary_opts": [], + "opts": [ + "--verbose" + ] + }, + { + "name": "surfaces", + "type": "Option", + "required": false, + "default": null, + "help": "Comma-separated filter on package name or surface-id substring.", + "secondary_opts": [], + "opts": [ + "--surfaces" + ] + }, + { + "name": "depth", + "type": "Option", + "required": false, + "default": "body", + "help": "How much source to collect per surface.", + "secondary_opts": [], + "opts": [ + "--depth" + ] + }, + { + "name": "callee_scope", + "type": "Option", + "required": false, + "default": "same-package", + "help": "Which intra-project callees to follow.", + "secondary_opts": [], + "opts": [ + "--callee-scope" + ] + }, + { + "name": "max_hops", + "type": "Option", + "required": false, + "default": "2", + "help": "Callee expansion depth (callgraph only).", + "secondary_opts": [], + "opts": [ + "--max-hops" + ] + }, + { + "name": "max_callees", + "type": "Option", + "required": false, + "default": "8", + "help": "Hard cap on collected callee units per surface.", + "secondary_opts": [], + "opts": [ + "--max-callees" + ] + }, + { + "name": "max_code_lines", + "type": "Option", + "required": false, + "default": "400", + "help": "Hard cap on total source lines per surface.", + "secondary_opts": [], + "opts": [ + "--max-code-lines" + ] + }, + { + "name": "detect_cycles", + "type": "Option", + "required": false, + "default": "True", + "help": "Write code-context/_cycles.json from observed edges.", + "secondary_opts": [ + "--no-detect-cycles" + ], + "opts": [ + "--detect-cycles" + ] + } + ], + "is_group": false + }, + { + "full_name": "mine evaluate", + "help": "Stage 6: evaluate coverage and ambiguity gates.", + "params": [ + { + "name": "target", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "target" + ] + }, + { + "name": "output_dir", + "type": "Option", + "required": false, + "default": "./mining-output/", + "help": "Mining output directory.", + "secondary_opts": [], + "opts": [ + "--output" + ] + }, + { + "name": "verbose", + "type": "Option", + "required": false, + "default": "False", + "help": "Extra logging to stderr.", + "secondary_opts": [], + "opts": [ + "--verbose" + ] + }, + { + "name": "surfaces", + "type": "Option", + "required": false, + "default": null, + "help": "Comma-separated filter on package name or surface-id substring.", + "secondary_opts": [], + "opts": [ + "--surfaces" + ] + }, + { + "name": "ruleset_dir", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--ruleset-dir" + ] + }, + { + "name": "ruleset_config", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--ruleset-config" + ] + } + ], + "is_group": false + }, + { + "full_name": "mine features", + "help": "Stage 4: draft features from stories.", + "params": [ + { + "name": "target", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "target" + ] + }, + { + "name": "output_dir", + "type": "Option", + "required": false, + "default": "./mining-output/", + "help": "Mining output directory.", + "secondary_opts": [], + "opts": [ + "--output" + ] + }, + { + "name": "quick", + "type": "Option", + "required": false, + "default": "True", + "help": "Quick mode (default) or interactive prompts.", + "secondary_opts": [ + "--interactive" + ], + "opts": [ + "--quick" + ] + }, + { + "name": "verbose", + "type": "Option", + "required": false, + "default": "False", + "help": "Extra logging to stderr.", + "secondary_opts": [], + "opts": [ + "--verbose" + ] + }, + { + "name": "surfaces", + "type": "Option", + "required": false, + "default": null, + "help": "Comma-separated filter on package name or surface-id substring.", + "secondary_opts": [], + "opts": [ + "--surfaces" + ] + }, + { + "name": "max_parallel", + "type": "Option", + "required": false, + "default": "4", + "help": "Max parallel LLM calls in quick mode (stories, features).", + "secondary_opts": [], + "opts": [ + "--max-parallel" + ] + }, + { + "name": "overwrite_features", + "type": "Option", + "required": false, + "default": "False", + "help": "", + "secondary_opts": [], + "opts": [ + "--overwrite-features" + ] + } + ], + "is_group": false + }, + { + "full_name": "mine inventory", + "help": "Stage 1: introspect target and write inventory.json.", + "params": [ + { + "name": "target", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "target" + ] + }, + { + "name": "output_dir", + "type": "Option", + "required": false, + "default": "./mining-output/", + "help": "Mining output directory.", + "secondary_opts": [], + "opts": [ + "--output" + ] + }, + { + "name": "quick", + "type": "Option", + "required": false, + "default": "True", + "help": "Quick mode (default) or interactive prompts.", + "secondary_opts": [ + "--interactive" + ], + "opts": [ + "--quick" + ] + }, + { + "name": "verbose", + "type": "Option", + "required": false, + "default": "False", + "help": "Extra logging to stderr.", + "secondary_opts": [], + "opts": [ + "--verbose" + ] + }, + { + "name": "no_mcp", + "type": "Option", + "required": false, + "default": "False", + "help": "Skip MCP tools/list.", + "secondary_opts": [], + "opts": [ + "--no-mcp" + ] + }, + { + "name": "mcp_timeout", + "type": "Option", + "required": false, + "default": "30.0", + "help": "", + "secondary_opts": [], + "opts": [ + "--mcp-timeout" + ] + } + ], + "is_group": false + }, + { + "full_name": "mine report", + "help": "Stage 7: render mining-report.md from pipeline outputs.", + "params": [ + { + "name": "target", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "target" + ] + }, + { + "name": "output_dir", + "type": "Option", + "required": false, + "default": "./mining-output/", + "help": "Mining output directory.", + "secondary_opts": [], + "opts": [ + "--output" + ] + }, + { + "name": "quick", + "type": "Option", + "required": false, + "default": "True", + "help": "Quick mode (default) or interactive prompts.", + "secondary_opts": [ + "--interactive" + ], + "opts": [ + "--quick" + ] + }, + { + "name": "verbose", + "type": "Option", + "required": false, + "default": "False", + "help": "Extra logging to stderr.", + "secondary_opts": [], + "opts": [ + "--verbose" + ] + }, + { + "name": "surfaces", + "type": "Option", + "required": false, + "default": null, + "help": "Comma-separated filter on package name or surface-id substring.", + "secondary_opts": [], + "opts": [ + "--surfaces" + ] + } + ], + "is_group": false + }, + { + "full_name": "mine stories", + "help": "Stage 3: emit stories from inventory.json.", + "params": [ + { + "name": "target", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "target" + ] + }, + { + "name": "output_dir", + "type": "Option", + "required": false, + "default": "./mining-output/", + "help": "Mining output directory.", + "secondary_opts": [], + "opts": [ + "--output" + ] + }, + { + "name": "quick", + "type": "Option", + "required": false, + "default": "True", + "help": "Quick mode (default) or interactive prompts.", + "secondary_opts": [ + "--interactive" + ], + "opts": [ + "--quick" + ] + }, + { + "name": "verbose", + "type": "Option", + "required": false, + "default": "False", + "help": "Extra logging to stderr.", + "secondary_opts": [], + "opts": [ + "--verbose" + ] + }, + { + "name": "surfaces", + "type": "Option", + "required": false, + "default": null, + "help": "Comma-separated filter on package name or surface-id substring.", + "secondary_opts": [], + "opts": [ + "--surfaces" + ] + }, + { + "name": "max_parallel", + "type": "Option", + "required": false, + "default": "4", + "help": "Max parallel LLM calls in quick mode (stories, features).", + "secondary_opts": [], + "opts": [ + "--max-parallel" + ] + }, + { + "name": "overwrite_stories", + "type": "Option", + "required": false, + "default": "False", + "help": "", + "secondary_opts": [], + "opts": [ + "--overwrite-stories" + ] + }, + { + "name": "code_context_dir", + "type": "Option", + "required": false, + "default": null, + "help": "Directory with code-context/*.md (default: /code-context when present).", + "secondary_opts": [], + "opts": [ + "--code-context" + ] + } + ], + "is_group": false + }, + { + "full_name": "mine tag", + "help": "Stage 5: tag scenarios in generated features.", + "params": [ + { + "name": "target", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "target" + ] + }, + { + "name": "output_dir", + "type": "Option", + "required": false, + "default": "./mining-output/", + "help": "Mining output directory.", + "secondary_opts": [], + "opts": [ + "--output" + ] + }, + { + "name": "quick", + "type": "Option", + "required": false, + "default": "True", + "help": "Quick mode (default) or interactive prompts.", + "secondary_opts": [ + "--interactive" + ], + "opts": [ + "--quick" + ] + }, + { + "name": "verbose", + "type": "Option", + "required": false, + "default": "False", + "help": "Extra logging to stderr.", + "secondary_opts": [], + "opts": [ + "--verbose" + ] + }, + { + "name": "surfaces", + "type": "Option", + "required": false, + "default": null, + "help": "Comma-separated filter on package name or surface-id substring.", + "secondary_opts": [], + "opts": [ + "--surfaces" + ] + }, + { + "name": "ruleset_dir", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--ruleset-dir" + ] + }, + { + "name": "ruleset_config", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--ruleset-config" + ] + } + ], + "is_group": false + } + ], + "mcp_tools": [], + "gates": [] + }, + "pickled-data": { + "version": "0.1.0.dev0", + "description": "SQL migration parsing, sandbox oracle, and schema contract gates.", + "scripts": { + "pickled-data": "pickled_data.cli:main" + }, + "entry_points": { + "pickled.mcp.subservers": { + "data": "pickled_data.mcp_cli:build_server" + }, + "pickled.gates": { + "data": "pickled_data.gates_runner:run_all" + } + }, + "optional_dependencies": [ + "cross-contract", + "mcp" + ], + "cli_commands": [ + { + "full_name": "apply", + "help": "Apply migration to in-memory SQLite and print resulting schema.", + "params": [ + { + "name": "migration", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "migration" + ] + }, + { + "name": "dialect", + "type": "Option", + "required": false, + "default": "postgres", + "help": "", + "secondary_opts": [], + "opts": [ + "--dialect" + ] + } + ], + "is_group": false + }, + { + "full_name": "check-drift", + "help": "Run MigrationDriftGate against expected schema YAML.", + "params": [ + { + "name": "migration", + "type": "Option", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--migration" + ] + }, + { + "name": "expected", + "type": "Option", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--expected" + ] + }, + { + "name": "dialect", + "type": "Option", + "required": false, + "default": "postgres", + "help": "", + "secondary_opts": [], + "opts": [ + "--dialect" + ] + } + ], + "is_group": false + }, + { + "full_name": "draft", + "help": "Draft a SQL migration from a natural-language intent.", + "params": [ + { + "name": "intent", + "type": "Option", + "required": true, + "default": null, + "help": "Intent file path or '-' for stdin.", + "secondary_opts": [], + "opts": [ + "--intent" + ] + }, + { + "name": "dialect", + "type": "Option", + "required": true, + "default": null, + "help": "SQL dialect for the migration.", + "secondary_opts": [], + "opts": [ + "--dialect" + ] + }, + { + "name": "current_schema", + "type": "Option", + "required": false, + "default": null, + "help": "Optional existing schema YAML file.", + "secondary_opts": [], + "opts": [ + "--current-schema" + ] + }, + { + "name": "output", + "type": "Option", + "required": false, + "default": null, + "help": "Write SQL to this path. Default: stdout.", + "secondary_opts": [], + "opts": [ + "-o", + "--output" + ] + } + ], + "is_group": false + }, + { + "full_name": "mcp", + "help": "MCP server commands.", + "params": [], + "is_group": true + }, + { + "full_name": "mcp serve", + "help": "", + "params": [ + { + "name": "transport", + "type": "Option", + "required": false, + "default": "stdio", + "help": "", + "secondary_opts": [], + "opts": [ + "--transport" + ] + }, + { + "name": "host", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--host" + ] + }, + { + "name": "port", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--port" + ] + }, + { + "name": "allow_public", + "type": "Option", + "required": false, + "default": "False", + "help": "", + "secondary_opts": [], + "opts": [ + "--allow-public" + ] + } + ], + "is_group": false + }, + { + "full_name": "parse", + "help": "Parse a migration SQL file and print AST summary.", + "params": [ + { + "name": "migration", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "migration" + ] + }, + { + "name": "dialect", + "type": "Option", + "required": false, + "default": "postgres", + "help": "", + "secondary_opts": [], + "opts": [ + "--dialect" + ] + } + ], + "is_group": false + } + ], + "mcp_tools": [ + { + "name": "data_apply_sql_to_sandbox", + "description": "Apply SQL to in-memory SQLite and return schema.", + "input_schema": { + "additionalProperties": false, + "properties": { + "sql": { + "type": "string" + }, + "dialect": { + "default": "postgres", + "type": "string" + } + }, + "required": [ + "sql" + ], + "type": "object" + } + }, + { + "name": "data_check_migration_drift", + "description": "Compare migration result schema to expected YAML.", + "input_schema": { + "additionalProperties": false, + "properties": { + "sql": { + "type": "string" + }, + "expected_schema_yaml": { + "type": "string" + }, + "dialect": { + "default": "postgres", + "type": "string" + } + }, + "required": [ + "sql", + "expected_schema_yaml" + ], + "type": "object" + } + }, + { + "name": "data_draft_sql_migration_from_intent", + "description": "", + "input_schema": { + "additionalProperties": false, + "properties": { + "intent_text": { + "type": "string" + }, + "dialect": { + "type": "string" + }, + "current_schema_yaml": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null + } + }, + "required": [ + "intent_text", + "dialect" + ], + "type": "object" + } + }, + { + "name": "data_parse_sql_migration", + "description": "Parse SQL and return AST summary.", + "input_schema": { + "additionalProperties": false, + "properties": { + "sql": { + "type": "string" + }, + "dialect": { + "default": "postgres", + "type": "string" + } + }, + "required": [ + "sql" + ], + "type": "object" + } + } + ], + "gates": [ + { + "kind": "class", + "name": "DataContractGate.run", + "module": "pickled_data.gates", + "file": "packages/pickled-data/src/pickled_data/gates.py", + "line": 125, + "returns": "GateResult", + "docstring_summary": "v0.1 PARTIAL: column name matching against OpenAPI response properties." + }, + { + "kind": "class", + "name": "MigrationDriftGate.run", + "module": "pickled_data.gates", + "file": "packages/pickled-data/src/pickled_data/gates.py", + "line": 80, + "returns": "GateResult", + "docstring_summary": "Compare oracle schema output vs expected schema YAML." + }, + { + "kind": "function", + "name": "run_all", + "module": "pickled_data.gates_runner", + "file": "packages/pickled-data/src/pickled_data/gates_runner.py", + "line": 15, + "returns": "list[GateResult]", + "docstring_summary": "Parse migrations and run drift gate vs ``expected_schema.yaml``." + } + ] + }, + "pickled-diff": { + "version": "0.1.0.dev0", + "description": "Differential oracle verification: compare a candidate implementation against a reference implementation across an input corpus.", + "scripts": { + "pickled-diff": "pickled_diff.cli:main" + }, + "entry_points": { + "pickled.mcp.subservers": { + "diff": "pickled_diff.mcp_cli:build_server" + }, + "pickled.gates": { + "diff": "pickled_diff.gates_runner:run_all" + } + }, + "optional_dependencies": [ + "mcp" + ], + "cli_commands": [ + { + "full_name": "draft-corpus", + "help": "Expand seed examples into a larger differential corpus.", + "params": [ + { + "name": "seeds", + "type": "Option", + "required": true, + "default": null, + "help": "JSON file with seed items, or '-' for stdin.", + "secondary_opts": [], + "opts": [ + "--seeds" + ] + }, + { + "name": "target_size", + "type": "Option", + "required": true, + "default": null, + "help": "Total corpus size.", + "secondary_opts": [], + "opts": [ + "--target-size" + ] + }, + { + "name": "notes", + "type": "Option", + "required": false, + "default": null, + "help": "Optional notes file path or '-' for stdin.", + "secondary_opts": [], + "opts": [ + "--notes" + ] + }, + { + "name": "output", + "type": "Option", + "required": false, + "default": null, + "help": "Write corpus JSON to this path. Default: stdout.", + "secondary_opts": [], + "opts": [ + "-o", + "--output" + ] + } + ], + "is_group": false + }, + { + "full_name": "mcp", + "help": "MCP server commands.", + "params": [], + "is_group": true + }, + { + "full_name": "mcp serve", + "help": "Run the pickled-diff MCP server.", + "params": [ + { + "name": "transport", + "type": "Option", + "required": false, + "default": "stdio", + "help": "", + "secondary_opts": [], + "opts": [ + "--transport" + ] + }, + { + "name": "host", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--host" + ] + }, + { + "name": "port", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--port" + ] + }, + { + "name": "allow_public", + "type": "Option", + "required": false, + "default": "False", + "help": "", + "secondary_opts": [], + "opts": [ + "--allow-public" + ] + } + ], + "is_group": false + }, + { + "full_name": "serve", + "help": "Deprecated alias for ``pickled-diff mcp serve``.", + "params": [], + "is_group": false + }, + { + "full_name": "verify", + "help": "Compare candidate vs reference across a JSON input corpus.", + "params": [ + { + "name": "oracle", + "type": "Option", + "required": true, + "default": null, + "help": "Reference command (shell-quoted argv).", + "secondary_opts": [], + "opts": [ + "--oracle" + ] + }, + { + "name": "candidate", + "type": "Option", + "required": true, + "default": null, + "help": "Candidate command (shell-quoted argv).", + "secondary_opts": [], + "opts": [ + "--candidate" + ] + }, + { + "name": "corpus", + "type": "Option", + "required": true, + "default": null, + "help": "JSON file: [{\"name\": \"...\", \"payload\": \"...\"}, ...]", + "secondary_opts": [], + "opts": [ + "--corpus" + ] + }, + { + "name": "comparator", + "type": "Option", + "required": false, + "default": "exact", + "help": "", + "secondary_opts": [], + "opts": [ + "--comparator" + ] + }, + { + "name": "timeout_seconds", + "type": "Option", + "required": false, + "default": "30.0", + "help": "", + "secondary_opts": [], + "opts": [ + "--timeout" + ] + } + ], + "is_group": false + } + ], + "mcp_tools": [ + { + "name": "diff_draft_corpus_from_examples", + "description": "", + "input_schema": { + "additionalProperties": false, + "properties": { + "seed_examples": { + "items": { + "additionalProperties": { + "type": "string" + }, + "type": "object" + }, + "type": "array" + }, + "target_size": { + "type": "integer" + }, + "notes": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null + } + }, + "required": [ + "seed_examples", + "target_size" + ], + "type": "object" + } + }, + { + "name": "diff_verify_against_oracle", + "description": "Compare candidate vs reference command output on a corpus.", + "input_schema": { + "additionalProperties": false, + "properties": { + "oracle_command": { + "items": { + "type": "string" + }, + "type": "array" + }, + "candidate_command": { + "items": { + "type": "string" + }, + "type": "array" + }, + "corpus_items": { + "items": { + "additionalProperties": { + "type": "string" + }, + "type": "object" + }, + "type": "array" + }, + "comparator": { + "default": "exact", + "type": "string" + }, + "timeout_seconds": { + "default": 30.0, + "type": "number" + } + }, + "required": [ + "oracle_command", + "candidate_command", + "corpus_items" + ], + "type": "object" + } + } + ], + "gates": [ + { + "kind": "function", + "name": "run_all", + "module": "pickled_diff.gates_runner", + "file": "packages/pickled-diff/src/pickled_diff/gates_runner.py", + "line": 78, + "returns": "list[GateResult]", + "docstring_summary": "Run differential oracle gate when ``pickled.diff.yaml`` is present." + } + ] + }, + "pickled-iac": { + "version": "0.1.0.dev0", + "description": "LLM-to-Terraform bridge with plan diff and optional Trivy security scanning.", + "scripts": { + "pickled-iac": "pickled_iac.cli:main" + }, + "entry_points": { + "pickled.mcp.subservers": { + "iac": "pickled_iac.mcp_cli:build_server" + }, + "pickled.gates": { + "iac": "pickled_iac.gates_runner:run_all" + } + }, + "optional_dependencies": [ + "mcp", + "security" + ], + "cli_commands": [ + { + "full_name": "diff", + "help": "Compare two terraform plan JSON files.", + "params": [ + { + "name": "base_path", + "type": "Option", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--base" + ] + }, + { + "name": "head_path", + "type": "Option", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--head" + ] + } + ], + "is_group": false + }, + { + "full_name": "draft", + "help": "Draft a Terraform module from a user story.", + "params": [ + { + "name": "user_story", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "user_story" + ] + }, + { + "name": "provider", + "type": "Option", + "required": false, + "default": "aws", + "help": "", + "secondary_opts": [], + "opts": [ + "--provider" + ] + }, + { + "name": "output", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "-o", + "--output" + ] + } + ], + "is_group": false + }, + { + "full_name": "mcp", + "help": "MCP server commands.", + "params": [], + "is_group": true + }, + { + "full_name": "mcp serve", + "help": "", + "params": [ + { + "name": "transport", + "type": "Option", + "required": false, + "default": "stdio", + "help": "", + "secondary_opts": [], + "opts": [ + "--transport" + ] + }, + { + "name": "host", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--host" + ] + }, + { + "name": "port", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--port" + ] + }, + { + "name": "allow_public", + "type": "Option", + "required": false, + "default": "False", + "help": "", + "secondary_opts": [], + "opts": [ + "--allow-public" + ] + } + ], + "is_group": false + }, + { + "full_name": "plan-cmd", + "help": "Run terraform plan and write JSON to *output*.", + "params": [ + { + "name": "tf_dir", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "tf_dir" + ] + }, + { + "name": "output", + "type": "Option", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "-o", + "--output" + ] + } + ], + "is_group": false + }, + { + "full_name": "scan", + "help": "Run Trivy config scan (optional; skips if trivy missing).", + "params": [ + { + "name": "tf_dir", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "tf_dir" + ] + } + ], + "is_group": false + }, + { + "full_name": "validate", + "help": "Run terraform validate on a directory.", + "params": [ + { + "name": "tf_dir", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "tf_dir" + ] + } + ], + "is_group": false + } + ], + "mcp_tools": [ + { + "name": "iac_diff_terraform_plans", + "description": "Compare base vs head terraform plan JSON.", + "input_schema": { + "additionalProperties": false, + "properties": { + "base_plan_json": { + "type": "string" + }, + "head_plan_json": { + "type": "string" + } + }, + "required": [ + "base_plan_json", + "head_plan_json" + ], + "type": "object" + } + }, + { + "name": "iac_draft_terraform_module", + "description": "Draft a Terraform module from a user story.", + "input_schema": { + "additionalProperties": false, + "properties": { + "user_story": { + "type": "string" + }, + "provider": { + "default": "aws", + "type": "string" + } + }, + "required": [ + "user_story" + ], + "type": "object" + } + }, + { + "name": "iac_explain_plan_diff", + "description": "Summarise a terraform plan JSON and flag risky actions.", + "input_schema": { + "additionalProperties": false, + "properties": { + "plan_json": { + "type": "string" + } + }, + "required": [ + "plan_json" + ], + "type": "object" + } + }, + { + "name": "iac_suggest_security_remediation", + "description": "Suggest HCL patches for Trivy config-scan findings.", + "input_schema": { + "additionalProperties": false, + "properties": { + "trivy_findings_json": { + "type": "string" + }, + "hcl_text": { + "default": "", + "type": "string" + } + }, + "required": [ + "trivy_findings_json" + ], + "type": "object" + } + }, + { + "name": "iac_validate_terraform_dir", + "description": "Validate Terraform files written to a temp directory.", + "input_schema": { + "additionalProperties": false, + "properties": { + "tf_files": { + "additionalProperties": { + "type": "string" + }, + "type": "object" + } + }, + "required": [ + "tf_files" + ], + "type": "object" + } + } + ], + "gates": [ + { + "kind": "class", + "name": "IaCAmbiguityGate.run", + "module": "pickled_iac.gates", + "file": "packages/pickled-iac/src/pickled_iac/gates.py", + "line": 37, + "returns": "GateResult", + "docstring_summary": "LLM critic for Terraform modules." + }, + { + "kind": "class", + "name": "PlanDiffGate.run", + "module": "pickled_iac.gates", + "file": "packages/pickled-iac/src/pickled_iac/gates.py", + "line": 102, + "returns": "GateResult", + "docstring_summary": "Compare two terraform plan JSON outputs (base vs head)." + }, + { + "kind": "class", + "name": "SecurityBaselineGate.run", + "module": "pickled_iac.gates", + "file": "packages/pickled-iac/src/pickled_iac/gates.py", + "line": 172, + "returns": "GateResult", + "docstring_summary": "Run Trivy config scan on a Terraform directory (optional in v0.1)." + }, + { + "kind": "function", + "name": "run_all", + "module": "pickled_iac.gates_runner", + "file": "packages/pickled-iac/src/pickled_iac/gates_runner.py", + "line": 14, + "returns": "list[GateResult]", + "docstring_summary": "``terraform validate`` and optional Trivy scan on ``infra/``." + } + ] + }, + "pickled-rules": { + "version": "0.1.0", + "description": "Rule coverage analysis for project artifacts (Gherkin features and scenarios). Generic dev tooling — bring your own rule sets.", + "scripts": { + "pickled-rules": "pickled_rules.cli:main" + }, + "entry_points": { + "pickled.mcp.subservers": { + "rules": "pickled_rules.mcp_cli:build_server" + }, + "pickled.gates": { + "rules": "pickled_rules.gates_runner:run_all" + } + }, + "optional_dependencies": [ + "mcp" + ], + "cli_commands": [ + { + "full_name": "check", + "help": "Check feature coverage against a YAML rule set.", + "params": [ + { + "name": "ruleset", + "type": "Option", + "required": true, + "default": null, + "help": "Built-in rule set name or path to a YAML rule set file.", + "secondary_opts": [], + "opts": [ + "--ruleset" + ] + }, + { + "name": "feature_path", + "type": "Option", + "required": false, + "default": null, + "help": "Single Gherkin feature file to analyse.", + "secondary_opts": [], + "opts": [ + "--feature" + ] + }, + { + "name": "feature_glob", + "type": "Option", + "required": false, + "default": null, + "help": "Glob of feature files; multiple matches are checked as one union (strict rules must appear across the set, not in each file).", + "secondary_opts": [], + "opts": [ + "--feature-glob" + ] + }, + { + "name": "ruleset_name", + "type": "Option", + "required": false, + "default": null, + "help": "Short name for tag prefix (default: built-in name or file stem).", + "secondary_opts": [], + "opts": [ + "--ruleset-name" + ] + }, + { + "name": "output_format", + "type": "Option", + "required": false, + "default": "markdown", + "help": "Report output format.", + "secondary_opts": [], + "opts": [ + "--format" + ] + }, + { + "name": "output", + "type": "Option", + "required": false, + "default": null, + "help": "Write the report to this path. Default: stdout.", + "secondary_opts": [], + "opts": [ + "--output", + "-o" + ] + }, + { + "name": "quiet", + "type": "Option", + "required": false, + "default": "False", + "help": "Suppress report on stdout; print only the verdict line. If --output is set, the report is still written to the file.", + "secondary_opts": [], + "opts": [ + "--quiet" + ] + } + ], + "is_group": false + }, + { + "full_name": "draft", + "help": "Draft a YAML rule set from a natural-language brief.", + "params": [ + { + "name": "brief", + "type": "Option", + "required": true, + "default": null, + "help": "Brief file path or '-' for stdin.", + "secondary_opts": [], + "opts": [ + "--brief" + ] + }, + { + "name": "short_name", + "type": "Option", + "required": true, + "default": null, + "help": "Ruleset short name for tagging.", + "secondary_opts": [], + "opts": [ + "--short-name" + ] + }, + { + "name": "source_id", + "type": "Option", + "required": true, + "default": null, + "help": "metadata.source_id value.", + "secondary_opts": [], + "opts": [ + "--source-id" + ] + }, + { + "name": "applies_to", + "type": "Option", + "required": true, + "default": null, + "help": "metadata.applies_to value.", + "secondary_opts": [], + "opts": [ + "--applies-to" + ] + }, + { + "name": "active_from", + "type": "Option", + "required": true, + "default": null, + "help": "metadata.active_from (YYYY-MM-DD).", + "secondary_opts": [], + "opts": [ + "--active-from" + ] + }, + { + "name": "output", + "type": "Option", + "required": false, + "default": null, + "help": "Write YAML to this path. Default: stdout.", + "secondary_opts": [], + "opts": [ + "-o", + "--output" + ] + } + ], + "is_group": false + }, + { + "full_name": "list-rules", + "help": "List rule ids from a YAML rule set.", + "params": [ + { + "name": "ruleset", + "type": "Option", + "required": true, + "default": null, + "help": "Built-in rule set name or path to a YAML rule set file.", + "secondary_opts": [], + "opts": [ + "--ruleset" + ] + } + ], + "is_group": false + }, + { + "full_name": "mcp", + "help": "MCP server commands.", + "params": [], + "is_group": true + }, + { + "full_name": "mcp serve", + "help": "Run the pickled-rules MCP server.", + "params": [ + { + "name": "transport", + "type": "Option", + "required": false, + "default": "stdio", + "help": "", + "secondary_opts": [], + "opts": [ + "--transport" + ] + }, + { + "name": "host", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--host" + ] + }, + { + "name": "port", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--port" + ] + }, + { + "name": "allow_public", + "type": "Option", + "required": false, + "default": "False", + "help": "", + "secondary_opts": [], + "opts": [ + "--allow-public" + ] + } + ], + "is_group": false + } + ], + "mcp_tools": [ + { + "name": "rules_check_ruleset_coverage", + "description": "Check Gherkin features against a YAML rule set (coverage gate).\n\n``feature_texts`` are the **contents** of ``.feature`` files. The\nprevious version accepted server-side filesystem paths, which gave\nany MCP client an arbitrary-file-read primitive via parser error\nmessages — see the module docstring.", + "input_schema": { + "additionalProperties": false, + "properties": { + "ruleset_yaml_text": { + "type": "string" + }, + "feature_texts": { + "items": { + "type": "string" + }, + "type": "array" + }, + "ruleset_short_name": { + "type": "string" + } + }, + "required": [ + "ruleset_yaml_text", + "feature_texts", + "ruleset_short_name" + ], + "type": "object" + } + }, + { + "name": "rules_draft_ruleset_from_brief", + "description": "", + "input_schema": { + "additionalProperties": false, + "properties": { + "brief_text": { + "type": "string" + }, + "ruleset_short_name": { + "type": "string" + }, + "source_id": { + "type": "string" + }, + "applies_to": { + "type": "string" + }, + "active_from": { + "type": "string" + } + }, + "required": [ + "brief_text", + "ruleset_short_name", + "source_id", + "applies_to", + "active_from" + ], + "type": "object" + } + }, + { + "name": "rules_list_rules", + "description": "List rule summaries from a YAML rule set.", + "input_schema": { + "additionalProperties": false, + "properties": { + "ruleset_yaml_text": { + "type": "string" + } + }, + "required": [ + "ruleset_yaml_text" + ], + "type": "object" + } + } + ], + "gates": [ + { + "kind": "function", + "name": "coverage_gate", + "module": "pickled_rules.gates.coverage", + "file": "packages/pickled-rules/src/pickled_rules/gates/coverage.py", + "line": 42, + "returns": "CoverageReport", + "docstring_summary": "Compute coverage of ``ruleset`` rules by ``feature`` scenarios." + }, + { + "kind": "function", + "name": "coverage_gate_features", + "module": "pickled_rules.gates.coverage", + "file": "packages/pickled-rules/src/pickled_rules/gates/coverage.py", + "line": 65, + "returns": "CoverageReport", + "docstring_summary": "Compute coverage across one or more features (union of scenario tags)." + }, + { + "kind": "function", + "name": "run_all", + "module": "pickled_rules.gates_runner", + "file": "packages/pickled-rules/src/pickled_rules/gates_runner.py", + "line": 100, + "returns": "list[GateResult]", + "docstring_summary": "Run coverage gate for each feature against ``pickled.ruleset.yaml``." + } + ] + }, + "pickled-schema": { + "version": "0.1.0.dev0", + "description": "Multi-format schema drafting and verification (OpenAPI, JSON Schema, Protobuf).", + "scripts": { + "pickled-schema": "pickled_schema.cli:main" + }, + "entry_points": { + "pickled.mcp.subservers": { + "schema": "pickled_schema.mcp_cli:build_server" + }, + "pickled.gates": { + "schema": "pickled_schema.gates_runner:run_all" + } + }, + "optional_dependencies": [ + "all-formats", + "json-schema", + "mcp", + "openapi", + "proto" + ], + "cli_commands": [ + { + "full_name": "check", + "help": "Run SchemaCoverageGate on @schema:endpoint tags in .feature files.", + "params": [ + { + "name": "spec", + "type": "Option", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--spec" + ] + }, + { + "name": "feature_dir", + "type": "Option", + "required": false, + "default": null, + "help": "Directory tree containing .feature files.", + "secondary_opts": [], + "opts": [ + "--feature-dir" + ] + }, + { + "name": "feature_glob", + "type": "Option", + "required": false, + "default": null, + "help": "Glob of .feature files (alternative to --feature-dir).", + "secondary_opts": [], + "opts": [ + "--feature-glob" + ] + } + ], + "is_group": false + }, + { + "full_name": "draft", + "help": "Draft an OpenAPI 3.1 path item from a Gherkin scenario.", + "params": [ + { + "name": "method", + "type": "Option", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--method" + ] + }, + { + "name": "endpoint_path", + "type": "Option", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--path" + ] + }, + { + "name": "gherkin_file", + "type": "Option", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--gherkin-file" + ] + }, + { + "name": "output", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "-o", + "--output" + ] + } + ], + "is_group": false + }, + { + "full_name": "mcp", + "help": "MCP server commands.", + "params": [], + "is_group": true + }, + { + "full_name": "mcp serve", + "help": "Run the pickled-schema MCP server.", + "params": [ + { + "name": "transport", + "type": "Option", + "required": false, + "default": "stdio", + "help": "", + "secondary_opts": [], + "opts": [ + "--transport" + ] + }, + { + "name": "host", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--host" + ] + }, + { + "name": "port", + "type": "Option", + "required": false, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "--port" + ] + }, + { + "name": "allow_public", + "type": "Option", + "required": false, + "default": "False", + "help": "", + "secondary_opts": [], + "opts": [ + "--allow-public" + ] + } + ], + "is_group": false + }, + { + "full_name": "parse", + "help": "Parse a schema file and print a short summary.", + "params": [ + { + "name": "file", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "file" + ] + }, + { + "name": "fmt", + "type": "Option", + "required": false, + "default": null, + "help": "Schema format (auto-detected from extension when omitted).", + "secondary_opts": [], + "opts": [ + "--format" + ] + } + ], + "is_group": false + }, + { + "full_name": "validate", + "help": "Validate a schema file against its format specification.", + "params": [ + { + "name": "file", + "type": "Argument", + "required": true, + "default": null, + "help": "", + "secondary_opts": [], + "opts": [ + "file" + ] + } + ], + "is_group": false + } + ], + "mcp_tools": [ + { + "name": "schema_check_schema_coverage", + "description": "Verify @schema:endpoint tags in features exist in the spec.", + "input_schema": { + "additionalProperties": false, + "properties": { + "spec_yaml": { + "type": "string" + }, + "feature_texts": { + "items": { + "type": "string" + }, + "type": "array" + } + }, + "required": [ + "spec_yaml", + "feature_texts" + ], + "type": "object" + } + }, + { + "name": "schema_draft_openapi_endpoint", + "description": "Draft an OpenAPI 3.1 path item from Gherkin text.", + "input_schema": { + "additionalProperties": false, + "properties": { + "method": { + "type": "string" + }, + "path": { + "type": "string" + }, + "gherkin_text": { + "type": "string" + } + }, + "required": [ + "method", + "path", + "gherkin_text" + ], + "type": "object" + } + }, + { + "name": "schema_validate_openapi_spec", + "description": "Validate an OpenAPI YAML document.", + "input_schema": { + "additionalProperties": false, + "properties": { + "spec_yaml": { + "type": "string" + } + }, + "required": [ + "spec_yaml" + ], + "type": "object" + } + } + ], + "gates": [ + { + "kind": "class", + "name": "SchemaAmbiguityGate.run", + "module": "pickled_schema.gates", + "file": "packages/pickled-schema/src/pickled_schema/gates.py", + "line": 38, + "returns": "GateResult", + "docstring_summary": "Second LLM critic pass on a drafted SchemaArtifact." + }, + { + "kind": "class", + "name": "SchemaCoverageGate.run", + "module": "pickled_schema.gates", + "file": "packages/pickled-schema/src/pickled_schema/gates.py", + "line": 110, + "returns": "GateResult", + "docstring_summary": "Verify @schema:endpoint:* tags in features exist in an OpenAPI spec." + }, + { + "kind": "function", + "name": "run_all", + "module": "pickled_schema.gates_runner", + "file": "packages/pickled-schema/src/pickled_schema/gates_runner.py", + "line": 15, + "returns": "list[GateResult]", + "docstring_summary": "Validate OpenAPI under ``specs/`` and run schema coverage on features." + } + ] + } + }, + "adrs": [ + { + "number": "0001", + "title": "pickled-diff package", + "status": "Proposed", + "date": "", + "file": "docs/decisions/0001-pickled-diff-package.md", + "body": "# ADR-0001: pickled-diff package\n\n- **Status:** Proposed\n- **Date:** 2026-05-19\n- **Deciders:** pickled-spec contributors\n\n## Context\n\nLLM-assisted workflows often produce a **candidate** implementation that must be\nchecked against an existing **reference** program. The pickled-* family already\ncovers strong, medium, and weak oracles for DSL artifacts (Gherkin, OpenAPI,\nTerraform, SQL, YAML rules), but none of them treat “agreement with a trusted\nimplementation on a finite input corpus” as a first-class oracle category.\n\nTeams need deterministic, repeatable differential checks when replacing or\nrefactoring code, without embedding domain logic into shared core libraries.\n\n## Decision\n\nAdd **`pickled-diff`** as a new leaf package introducing:\n\n- The **reference oracle** category in `docs/pattern.md`.\n- **`DifferentialOracleGate`** implementing `pickled_core.Gate`.\n- Pluggable **`OracleRunner`**, **`Corpus`**, and **`Comparator`** protocols.\n- CLI (`pickled-diff verify`, `pickled-diff serve`) and MCP tool\n `verify_against_oracle`.\n\nTypes such as `DifferentialFinding` and the runner/comparator protocols remain in\n`pickled-diff` until a second consumer justifies promotion to `pickled-core`.\n\n## Consequences\n\n**Positive**\n\n- Fills a documented gap in the oracle taxonomy.\n- Reuses existing `Gate`, `Verdict`, `GateResult`, and MCP scaffolding without\n bloating `pickled-core`.\n- Zero LLM dependency in the gate path keeps CI hermetic.\n\n**Negative**\n\n- One more workspace member to maintain and eventually add to CI mypy paths.\n- Umbrella MCP and `check-all` integration require entry points (`pickled.mcp.subservers`,\n `pickled.gates`) registered in `pyproject.toml`.\n\n## Alternatives considered\n\n**Add `DifferentialOracleGate` to `pickled-core`.** Rejected: no concrete gates\nlive in core today; adding domain-shaped findings and runner protocols would break\nthe “core stays small” rule before a second consumer exists.\n\n**Separate repository.** Rejected: the gate shares protocols and MCP patterns with\nsibling packages; monorepo keeps gate signature changes atomic.\n\n**Embed in `pickled-bdd` or another leaf.** Rejected: differential verification is\northogonal to Gherkin and other DSLs; a dedicated package keeps dependencies minimal\n(no Gherkin stack).\n", + "supersedes": [], + "superseded_by": [] + }, + { + "number": "0002", + "title": "Cache and budget in pickled.config.yaml", + "status": "Accepted", + "date": "", + "file": "docs/decisions/0002-cache-and-budget-in-pickled-config.md", + "body": "# ADR-0002: Cache and budget in pickled.config.yaml\n\n- **Status:** Accepted\n- **Date:** 2026-05-19\n- **Deciders:** pickled-spec contributors\n\n## Context\n\n`pickled-core` already ships disk-backed `LLMCache` and a `BudgetGuard` that\ncan cap cumulative LLM spend per process. Until now, leaf MCP servers and\npackage CLIs called `build_client(...)` directly without passing `cache=`, so\nevery `draft_*` and `validate_*` tool hit the live API on each rerun. No\nbootstrap path installed a budget guard, so agent-driven loops had no\ndeterministic cost ceiling.\n\nDogfood workflows and Cursor-native MCP sessions repeat the same prompts\nacross iterations. Without wiring cache and budget at client construction,\ntoken spend scales linearly with retries and a runaway tool loop can exhaust\nquota before an operator notices.\n\n## Decision drivers\n\n- Reduce token cost on repeated dogfood runs without changing gate semantics.\n- Provide an upper bound on runaway LLM loops in long-lived MCP processes.\n- One construction path shared by bdd, schema, and iac (no duplicated factory\n parsing in each leaf).\n- No new third-party dependencies; existing configs must keep working.\n- Relative cache paths should stay stable when the repo root moves (resolve\n against the loaded YAML, not the shell CWD).\n\n## Considered options\n\n1. **Env vars only** — `PICKLED_CACHE_DIR`, `PICKLED_MAX_COST_USD`, etc., with\n no schema change. Rejected: easy to forget in docs; no single file checked\n into the repo for dogfood; harder to share defaults across teammates.\n\n2. **YAML schema extension + env overrides (chosen)** — optional `cache:` and\n `budget:` blocks in `pickled.config.yaml`, with env winning on conflict.\n Central `build_default_client()` in `pickled-core` wires cache, budget, and\n provider; leaf `_build_llm_client()` functions delegate to it.\n\n3. **Per-leaf YAML keys** — each package defines its own cache/budget section.\n Rejected: duplication, drift risk, and no shared semantics for\n `pickled-spec mcp` umbrella behavior.\n\n## Decision outcome\n\nAdopt option 2. Extend `PickledConfig` with `CacheSettings`, `BudgetSettings`,\nand `source_path` (set by `load_config` when a file is read). Add\n`build_default_client()` in `pickled_core.llm.bootstrap` that:\n\n- Honors `PICKLED_*_LLM_FACTORY` for tests (no cache/budget wiring).\n- Installs `BudgetGuard` when `budget.max_cost_usd` or `PICKLED_MAX_COST_USD`\n is set.\n- Builds `LLMCache` unless cache mode is `off`.\n- Passes the cache into `build_client(provider, config=cfg, cache=cache)`.\n\nLeaf MCP CLIs (`pickled-bdd`, `pickled-schema`, `pickled-iac`) and\n`pickled-bdd` CLI replace inline factory parsing with a single bootstrap call.\n\n## Consequences\n\n**Positive**\n\n- Dogfood reruns can reuse cached completions (large reduction in repeat\n provider calls when inputs are unchanged).\n- A configured `max_cost_usd` aborts further billed calls once the guard\n trips.\n- Four copies of `_build_llm_client()` logic collapse to one helper.\n\n**Negative**\n\n- `PickledConfig` grows (`source_path`, `budget`); callers that construct\n configs manually must accept new defaults.\n\n**Neutral**\n\n- Configs without `cache:` / `budget:` behave as before: cache on at\n `.pickled-cache` (resolved next to the YAML), no budget cap.\n\n## Path semantics\n\n- Relative `cache.dir` resolves against the directory containing the loaded\n `pickled.config.yaml` (or XDG path), not the process CWD.\n- `PICKLED_CACHE_DIR` overrides the directory and keeps **CWD-relative**\n resolution when the env value is relative (shell ergonomics).\n- Absolute `cache.dir` values are unchanged.\n\n## Future work\n\n- Per-run budget reset for long-lived MCP servers (today the guard persists\n for the process lifetime).\n- Programmatic cache invalidation API (delete by key prefix or provider).\n- Optional YAML knob for budget reset cadence (per tool call vs per session).\n", + "supersedes": [], + "superseded_by": [] + }, + { + "number": "0003", + "title": "Cache, budget, and model resolution for leaf MCP and CLI", + "status": "Accepted", + "date": "", + "file": "docs/decisions/0003-cache-budget-and-model-resolution.md", + "body": "# ADR-0003: Cache, budget, and model resolution for leaf MCP and CLI\n\n- **Status:** Accepted\n- **Date:** 2026-05-25\n- **Deciders:** pickled-spec contributors\n\n## Context\n\n`pickled-core` already ships disk-backed `LLMCache`, a `BudgetGuard`, and\nper-provider `default_model` entries in `pickled.config.yaml`. Until this\nchange, leaf MCP servers and package CLIs called `build_client(...)` without\n`cache=`, so cached completions were never reused. No bootstrap path installed\na budget guard, so agent-driven loops had no deterministic cost ceiling.\n\n`complete_prompt` hardcoded `DEFAULT_MODEL = \"claude-3-5-sonnet-20241022\"`,\nwhich is deprecated upstream. MCP-invoked drafters (`FeatureDrafter`,\n`OpenAPIDrafter`, `IaCDrafter`) therefore 404ed even when the YAML named a\ncurrent model. The provider's `default_model` was parsed but not stored on the\nclient or consulted by `complete_prompt`.\n\n## Decision drivers\n\n- Reduce token spend on repeated dogfood runs without changing gate semantics.\n- Provide an upper bound on runaway LLM loops in long-lived MCP processes.\n- Honor `default_model` from configuration for every one-shot drafter.\n- One construction path shared by bdd, schema, and iac (no duplicated factory\n parsing in each leaf).\n- No new third-party dependencies; existing configs must keep working.\n- Relative `cache.dir` should resolve against the loaded YAML directory.\n\n## Considered options\n\n1. **Env vars only** — cache and budget via `PICKLED_*` with no schema change.\n Rejected: easy to omit in docs; no checked-in defaults for dogfood.\n\n2. **YAML schema extension + env overrides (chosen)** — optional `cache:` and\n `budget:` on `PickledConfig`, env wins on conflict; `build_default_client`\n composes cache, budget, and provider; `default_model` threaded through\n `build_client` to each provider client; `complete_prompt` resolves model\n from the client when not passed explicitly.\n\n3. **Per-leaf YAML keys** — duplicate cache/budget blocks in each package.\n Rejected: four copies of the same parsing and drift risk.\n\n## Decision\n\nExtend `PickledConfig` with optional `cache:` and `budget:` blocks and\n`source_path` when loading a file. Add `build_default_client` in\n`pickled-core` that installs `BudgetGuard`, builds `LLMCache` unless mode is\n`off`, and calls `build_client(provider, config=cfg, cache=cache)`.\n\nEach provider client accepts `default_model` (with a package-local default).\n`complete_prompt` resolves `model` as: explicit argument, then\n`client.default_model`, then module `DEFAULT_MODEL`.\n\nLeaf MCP CLIs and `pickled-bdd` CLI delegate `_build_llm_client()` to\n`build_default_client` with package-specific `PICKLED_*_LLM_FACTORY` env vars.\nUmbrella `build_server()` paths suppress `click.ClickException` so missing\noptional deps or config still register deterministic tools.\n\n## Consequences\n\n**Positive**\n\n- Large reduction in token spend on dogfood reruns when cache is enabled.\n- Deterministic cost ceiling when `budget.max_cost_usd` or env cap is set.\n- LLM drafters use the configured model; no stale hardcoded model string.\n- Four duplicated `_build_llm_client()` implementations collapse to one helper\n pattern plus shared bootstrap.\n\n**Negative**\n\n- `PickledConfig.source_path` adds mild API surface growth.\n- Long-lived MCP servers share one process-wide budget guard until reset\n (documented future work).\n- Relative `PICKLED_CACHE_DIR` env override remains CWD-relative by design.\n\n## Path semantics\n\nRelative `cache.dir` in YAML resolves against `source_path.parent` (the\ndirectory containing `pickled.config.yaml`). When `PICKLED_CACHE_DIR` is set,\nrelative values resolve against the process CWD. Absolute paths are unchanged.\n\n## Future work\n\n- Per-run budget reset for long-lived MCP servers.\n- Programmatic cache invalidation API.\n- Thread `build_default_client` through rules, data, and diff when those\n packages gain LLM-backed MCP tools.\n", + "supersedes": [], + "superseded_by": [] + }, + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "date": "", + "file": "docs/decisions/0004-multi-ruleset-workspace.md", + "body": "# ADR-0004: Multi-ruleset workspace configuration\n\n- **Status:** Accepted\n- **Date:** 2026-05-25\n- **Deciders:** pickled-spec contributors\n\n## Context\n\n`pickled.ruleset.yaml` was bound to a single rule set per workspace. That forced\nany project with several independent concerns — architectural invariants, OSS\nhygiene checks, and domain-specific best practices — to either merge everything\ninto one YAML file or run multiple `pickled-rules check` invocations outside\n`pickled-spec check-all`.\n\nThe workspace gate runner (`pickled_rules.gates_runner.run_all`) read only a\nsingular `ruleset:` path and emitted one `rules.coverage` verdict. Dogfood and\nmulti-concern workspaces need composable rule sets without losing per-concern\nvisibility in the `check-all` table.\n\n## Decision drivers\n\n- Composability: reuse standalone ruleset files across repos and workspaces.\n- Per-concern verdicts in `check-all` output when several rule sets apply.\n- Backward compatibility with existing `ruleset:` configs (including\n `examples/user-management-crud/`).\n- Minimal schema surface: one list, optional `short_name`, mutual exclusion with\n the legacy key.\n- No CLI or MCP changes in this iteration.\n\n## Considered options\n\n1. **Single umbrella ruleset** — merge all rules into one file per workspace.\n Rejected: prevents reuse of shared rulesets (e.g. OSS hygiene) and blurs\n ownership of concerns.\n\n2. **Per-ruleset YAML keys** (`ruleset_internal:`, `ruleset_hygiene:`, …).\n Rejected: not extensible; key names become part of the contract.\n\n3. **`rulesets:` list with backward-compatible `ruleset:` (chosen)** — plural\n list of `{path, short_name?}` entries; singular form unchanged.\n\n## Decision outcome\n\nOption 3. `_resolve_ruleset_entries` validates and resolves paths relative to\nthe workspace root. `run_all` runs `coverage_gate_features` once per entry.\nWhen exactly one entry is configured, the gate name remains `rules.coverage`.\nWhen multiple entries are configured, gate names are `rules.coverage.`.\n\nMixing `ruleset:` and `rulesets:` in the same file raises\n`RuleSetValidationError`. Duplicate `short_name` values in a list are rejected.\n\n## Consequences\n\n**Positive**\n\n- Workspaces can compose orthogonal rulesets in one `check-all` run.\n- Per-ruleset pass/fail rows appear in the output table without CLI changes.\n- Existing single-ruleset configs work without edits.\n\n**Neutral**\n\n- Small internal types (`_RulesetEntry`, `_resolve_ruleset_entries`) and\n validation messages to maintain.\n\n**Negative**\n\n- Two equivalent configuration shapes; authors must not combine them.\n- Tags for rules not matching any configured `short_name` remain silently\n ignored (existing `ruleset_filter` behaviour).\n\n## Compatibility\n\nThe single-ruleset gate name stays exactly `rules.coverage` (no namespace\nsuffix). Downstream tests and integrations that assert this string keep working.\nMulti-ruleset configurations use `rules.coverage.`.\n\n## Future work\n\n- `--ruleset-list` on `pickled-rules check` to mirror workspace runner behaviour.\n- Cross-ruleset orphan-tag detection for prefixes that match no configured\n ruleset.\n", + "supersedes": [], + "superseded_by": [] + }, + { + "number": "0005", + "title": "`pickled-spec mine` staged mining pipeline", + "status": "Accepted", + "date": "", + "file": "docs/decisions/0005-pickled-spec-mine.md", + "body": "# ADR-0005: `pickled-spec mine` staged mining pipeline\n\n- **Status:** Accepted\n- **Date:** 2026-05-28\n- **Deciders:** pickled-spec contributors\n\n## Context\n\nHand-running the dogfood loop (inventory → story → feature → tag → gates)\ndid not scale across dozens of CLI commands, MCP tools, and packages.\nWe needed a generic extractor that works on arbitrary Python repos, not a\none-off script tied to this monorepo.\n\nEarly inventory runs missed umbrella MCP tools when `pickled-spec` lived\non a workspace member rather than the root `pyproject.toml`. Story\ngeneration was initially sequential and could run for many minutes on a\nlarge repo without scoping.\n\n## Decision drivers\n\n- Work on any Python repo with Click-discoverable CLIs, not only\n pickled-spec.\n- Stage isolation: re-run one stage from files on disk.\n- Graceful degradation without an LLM (placeholders, skip features).\n- Actionable errors when stages run out of order.\n- Performance controls (`--surfaces`, parallel quick mode, existing cache).\n\n## Considered options\n\n1. **Monolithic `mine` command** — single run, no intermediate artifacts.\n Rejected: hard to debug, expensive to repeat one step, poor fit for\n human review between stages.\n\n2. **Staged pipeline with filesystem contract (chosen)** — each stage reads\n and writes under `--output`. Enables `mine all` and individual\n subcommands.\n\n3. **MCP-first mining** — expose stages only as MCP tools. Deferred: CLI\n first; MCP surface for mine is future work.\n\n## Decision outcome\n\nShip `pickled-spec mine` with six stages: inventory, stories, features,\ntag, evaluate, report. Stages communicate via `inventory.json`,\n`stories/`, `features/`, `tags-proposals.json`, and `evaluation/*.json`.\n`mine all` orchestrates the chain; `--surfaces` filters work per stage.\n\nMCP umbrella detection scans the target root and uv workspace members\nfor `pickled-spec` (or `pickled.mcp.subservers`). Rule set paths in\n`--ruleset-config` resolve relative to the config file directory.\n\nAmbiguity evaluation reuses `pickled_bdd.cli.run_ambiguity_gate`, the\nsame entry point as `pickled-bdd check --gate ambiguity` and the\n`pickled-bdd ambiguity` alias.\n\n## Consequences\n\n**Positive**\n\n- Mining is separate from dogfood: dogfood is one consumer of the same\n tools.\n- Re-runnable stages and inspectable artifacts.\n- Scoped runs via `--surfaces` keep LLM stages practical on monorepos.\n\n**Negative**\n\n- Disk layout is a public contract; changes need versioning care.\n- Full monorepo mining without `--surfaces` remains LLM-heavy.\n- Evaluate reports gate verdicts as-is; AmbiguityGate threshold tuning is\n out of scope for mine.\n\n## Future work\n\n- Multi-language inventory (non-Python CLIs).\n- MCP tools wrapping mine stages.\n- AmbiguityGate calibration as its own change set.\n", + "supersedes": [], + "superseded_by": [] + }, + { + "number": "0006", + "title": "`pickled-spec mine code` static code reading", + "status": "Accepted", + "date": "", + "file": "docs/decisions/0006-mine-code-reading.md", + "body": "# ADR-0006: `pickled-spec mine code` static code reading\n\n- **Status:** Accepted\n- **Date:** 2026-05-28\n- **Deciders:** pickled-spec contributors\n\n## Context\n\nInventory and docstrings describe surfaces at a high level. For gates and CLIs\nthat delegate to helpers, the docstring often understates real behaviour\n(temperature, validation, return shape). Phase 8e adds a dedicated **code**\nstage that extracts source for each mined surface and, optionally, a bounded\nset of intra-project callees.\n\n## Decision drivers\n\n- Ground later story generation in **observed code**, not names alone.\n- Stay within stdlib (`ast` only): no grimp/pydeps dependency.\n- Hard caps and a **visited** set so traversal cannot run away on cycles.\n- Optional diagnostic cycle reporting without changing traversal semantics.\n\n## Decision\n\nAdd `pickled-spec mine code` after inventory and before stories in `mine all`.\n\n### Depth modes\n\n| Mode | Content |\n|------|---------|\n| `signature` | Root signature, return annotation, docstring |\n| `body` | Root full function/method body (default) |\n| `callgraph` | Root body plus callee bodies up to `--max-hops` |\n\n### Callee scope\n\n- `self` — methods on the enclosing class (`self.helper()`).\n- `same-package` — `self` plus same-package imports (default).\n- `any-pickled` — same-package plus any `pickled_*` import.\n\n### Bounds\n\n- `--max-callees` (default 8) and `--max-code-lines` (default 400) per surface.\n- `visited` keys (`module:qualname`) prevent re-expansion; this is the cycle\n safety mechanism.\n- `--detect-cycles` runs a small DFS on collected edges and writes\n `code-context/_cycles.json` for the run log / report; it does not alter BFS.\n\n### Output\n\n`code-context/.md` per surface. Surfaces without a resolvable\ndefinition (e.g. MCP tool names with no mapped callable) get a placeholder\nfile and the stage continues.\n\n## Known limitations (v1)\n\n- **Protocol / dynamic dispatch** — calls such as `self._llm.complete(...)`\n where `_llm` is a Protocol or opaque attribute are recorded as *unresolved*\n callees with a reason; they are not chased.\n- **Python only** — no cross-language call graphs.\n- **Static resolution only** — no runtime type inference or polymorphic targets.\n\nStories do not consume code-context until Phase 8f.\n\n## Resolution patterns and limits (Phase 8e-fix)\n\nEach collected callee records `resolution_kind` on the ref. Default\n`--max-hops` is **2** so one delegation past the entry surface is included.\n\n### Resolved kinds\n\n| Kind | Pattern | Example |\n|------|---------|---------|\n| `free_function` | Same-module or imported callable | `helper()`, `chain.entry()` |\n| `self_method` | `self.method()` on enclosing class | `self.helper()` |\n| `constructor_method` | `Class(args).method()` | `Worker(cfg).process()` |\n| `module_constructor` | `mod.Class(args).method()` | `mod.Worker(cfg).process()` |\n| `annotated_param` | Parameter annotation pins type | `def f(w: Worker): w.m()` |\n| `annotated_var` | Annotated local | `x: Worker = …; x.m()` |\n| `assigned_constructor` | `x = Worker(); x.m()` (stable) | assignment tracking |\n\n`@property`, `@staticmethod`, `@classmethod`, and `async def` bodies resolve\nwhen the receiver type is known. Constructor arguments may contain separate\nresolvable calls (e.g. `Worker(Builder(x).build()).process()`).\n\n### Deliberately unresolved (reason strings)\n\n| Reason | Pattern |\n|--------|---------|\n| `protocol or unknown attribute type` | `self._llm.complete()` (nested attribute on `self`) |\n| `parameter '…' has no type annotation` | `def f(w): w.method()` |\n| `receiver is a subscript expression` | `items[0].method()` |\n| `variable '…' reassigned; type not stable` | `x = Worker(); x = Other(); x.m()` |\n| `receiver is a conditional expression` | `(a if c else b).run()` |\n| `receiver is a return value of unannotated callable` | `factory().build().run()`, `.process().finalize()` |\n| `dynamic attribute access` | `getattr(obj, \"m\")()` |\n| `method not found on class; possibly inherited (base not resolved in v1)` | method absent on declared class |\n| Name collision / unknown receiver | two classes share method name, type not pinned |\n\nInherited methods (MRO) are not walked in v1. Return-type inference for\narbitrary call chains is out of scope. Traversal uses the same caps and\n`visited` set as 8e; cycles are reported via `--detect-cycles` when enabled.\n\n## Consequences\n\n- `mine all` produces `code-context/` for downstream story prompts.\n- Readers must pass inventory first; missing `inventory.json` raises an\n actionable error naming `mine inventory`.\n", + "supersedes": [], + "superseded_by": [] + }, + { + "number": "0007", + "title": "Code-aware stories and docstring drift detection", + "status": "Accepted", + "date": "", + "file": "docs/decisions/0007-code-aware-stories-and-drift.md", + "body": "# ADR-0007: Code-aware stories and docstring drift detection\n\n- **Status:** Accepted\n- **Date:** 2026-05-28\n- **Deciders:** pickled-spec contributors\n\n## Context\n\nPhase 8e added static code reading (`code-context/.md`).\nPhase 8e-fix hardened the resolver so constructor-then-method patterns\n(e.g. `FeatureDrafter(llm).draft_from_story(story)`) resolve to real\nbodies, not just entry-point glue.\n\nDocstring-only stories (Phase 8d) were too shallow and sometimes wrong.\nOn the real `pickled-bdd` draft surface, a hand-written story claimed the\ndrafter **validates** Gherkin output. Code-reading showed the opposite:\n`draft_from_story`'s docstring states the drafter does **not** validate\n(returned Gherkin is raw; `warnings=()`). Human peer review had introduced\nthat confabulation. The mine pipeline can now ground stories in extracted\nsource instead of inventory summaries alone.\n\n## Decision\n\n### Code-grounded story generation\n\nWhen `code-context/.md` exists under the mining output directory,\nthe stories stage loads root and resolved callee bodies plus the unresolved\ncallee list and passes them to the story prompt together with the surface\ndocstring. The model writes **observable behavior** (contract), not\nimplementation mechanics.\n\n### Decision B: drift detection\n\nThe code is the source of truth. If the docstring **contradicts** the code,\nthe model emits a `---DRIFT---` block; each bullet is rendered under Open\nquestions prefixed with `Docstring drift:`. We do not silently override the\ndocstring or show code and docstring side-by-side without synthesis.\n\nWhen no code-context exists, behavior falls back to Phase 8d docstring-only\nrules and DRIFT is always empty.\n\n### Anti-implementation-leak\n\nStories must not mention line numbers, private method names, or call-chain\nnarration (\"it calls X then Y\"). A reader should understand the contract\nwithout seeing source. The prompt enforces this; tests guard the render path.\n\n### Unresolved-call honesty\n\nCalls the resolver cannot pin (protocol dispatch, dynamic getattr, etc.)\nremain listed in code-context. The prompt forbids inventing behavior behind\nthose calls; delegated behavior is stated as uncertain.\n\n### Friction #15: unresolved noise filtering\n\nBefore reporting unresolved callees, the code reader drops:\n\n- **Stdlib-surface methods** — e.g. `str.strip()`, `Path.read_text()` on\n receivers that are not resolvable intra-project types.\n- **Decorator registration** — callee scan walks function **bodies** only,\n so `@main.command()` on the definition is not treated as a behavioral call.\n\n**Limit:** a user-defined method whose name collides with a common builtin\nmethod (e.g. `.strip()`) on an unresolved receiver is also dropped. That\nwould have been unresolved noise anyway; accepted trade-off.\n\n### Provenance metadata\n\nEach story's Metadata section records **Code depth**, **Units read**, and\n**Unresolved** counts when code-context was present, so readers can see how\nstrong the grounding was (`signature` vs `callgraph`).\n\n## Consequences\n\n- `mine all` runs inventory → code → stories; stories auto-detect\n `code-context/` under `--output`.\n- `mine stories` accepts optional `--code-context` to override the directory.\n- Mine acts as a **docstring drift detector** when docstrings lie or lag code.\n- Story quality scales with `--depth` and `--max-hops` on the code stage.\n- Live LLM quality still depends on the model; tests use canned clients for\n wiring and anti-leak contracts.\n", + "supersedes": [], + "superseded_by": [] + } + ], + "workspaces": [ + { + "path": "dogfood", + "config_file": "dogfood/pickled.ruleset.yaml", + "form": "multi", + "rulesets": [ + { + "path": "./rulesets/pickled-internal.yaml", + "short_name": "pickled-internal", + "exists": true + }, + { + "path": "./rulesets/best-practices.yaml", + "short_name": "best-practices", + "exists": true + }, + { + "path": "./rulesets/oss-hygiene.yaml", + "short_name": "oss-hygiene", + "exists": true + }, + { + "path": "./rulesets/bdd-domain.yaml", + "short_name": "bdd-domain", + "exists": true + }, + { + "path": "./rulesets/rules-domain.yaml", + "short_name": "rules-domain", + "exists": true + }, + { + "path": "./rulesets/schema-domain.yaml", + "short_name": "schema-domain", + "exists": true + }, + { + "path": "./rulesets/iac-domain.yaml", + "short_name": "iac-domain", + "exists": true + }, + { + "path": "./rulesets/data-domain.yaml", + "short_name": "data-domain", + "exists": true + }, + { + "path": "./rulesets/diff-domain.yaml", + "short_name": "diff-domain", + "exists": true + }, + { + "path": "./rulesets/core-domain.yaml", + "short_name": "core-domain", + "exists": true + } + ], + "feature_count": 1, + "story_count": 0 + }, + { + "path": "examples/user-management-crud", + "config_file": "examples/user-management-crud/pickled.ruleset.yaml", + "form": "single", + "rulesets": [ + { + "path": "../../packages/pickled-rules/rulesets/examples/gdpr-web-crud.yaml", + "short_name": "gdpr-web-crud", + "exists": true + } + ], + "feature_count": 5, + "story_count": 0 + } + ], + "warnings": [], + "surface_relevant_adrs": { + "bdd_draft_feature_from_story": [], + "bdd_validate_feature_ambiguity": [ + { + "number": "0005", + "title": "`pickled-spec mine` staged mining pipeline", + "status": "Accepted", + "general": false + }, + { + "number": "0007", + "title": "Code-aware stories and docstring drift detection", + "status": "Accepted", + "general": false + } + ], + "pickled_bdd_ambiguity": [ + { + "number": "0005", + "title": "`pickled-spec mine` staged mining pipeline", + "status": "Accepted", + "general": false + } + ], + "pickled_bdd_check": [], + "pickled_bdd_draft": [], + "pickled_bdd_mcp": [], + "pickled_bdd_ambiguitygate": [ + { + "number": "0005", + "title": "`pickled-spec mine` staged mining pipeline", + "status": "Accepted", + "general": false + } + ], + "pickled_bdd_run_all": [], + "pickled_core_check_all": [ + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": true + } + ], + "pickled_core_mine": [ + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": true + } + ], + "pickled_core_mine_all": [ + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": true + } + ], + "pickled_core_mine_code": [ + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": true + } + ], + "pickled_core_mine_evaluate": [ + { + "number": "0005", + "title": "`pickled-spec mine` staged mining pipeline", + "status": "Accepted", + "general": false + } + ], + "pickled_core_mine_features": [ + { + "number": "0005", + "title": "`pickled-spec mine` staged mining pipeline", + "status": "Accepted", + "general": false + } + ], + "pickled_core_mine_inventory": [ + { + "number": "0005", + "title": "`pickled-spec mine` staged mining pipeline", + "status": "Accepted", + "general": false + }, + { + "number": "0006", + "title": "`pickled-spec mine code` static code reading", + "status": "Accepted", + "general": false + }, + { + "number": "0007", + "title": "Code-aware stories and docstring drift detection", + "status": "Accepted", + "general": false + } + ], + "pickled_core_mine_report": [ + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": true + } + ], + "pickled_core_mine_stories": [ + { + "number": "0007", + "title": "Code-aware stories and docstring drift detection", + "status": "Accepted", + "general": false + } + ], + "pickled_core_mine_tag": [ + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": true + } + ], + "data_apply_sql_to_sandbox": [], + "data_check_migration_drift": [ + { + "number": "0007", + "title": "Code-aware stories and docstring drift detection", + "status": "Accepted", + "general": false + } + ], + "data_draft_sql_migration_from_intent": [], + "data_parse_sql_migration": [], + "pickled_data_apply": [], + "pickled_data_check_drift": [ + { + "number": "0007", + "title": "Code-aware stories and docstring drift detection", + "status": "Accepted", + "general": false + } + ], + "pickled_data_draft": [], + "pickled_data_mcp": [], + "pickled_data_parse": [], + "pickled_data_datacontractgate": [ + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": false + }, + { + "number": "0005", + "title": "`pickled-spec mine` staged mining pipeline", + "status": "Accepted", + "general": false + }, + { + "number": "0007", + "title": "Code-aware stories and docstring drift detection", + "status": "Accepted", + "general": false + } + ], + "pickled_data_migrationdriftgate": [ + { + "number": "0007", + "title": "Code-aware stories and docstring drift detection", + "status": "Accepted", + "general": false + } + ], + "pickled_data_run_all": [], + "diff_draft_corpus_from_examples": [ + { + "number": "0001", + "title": "pickled-diff package", + "status": "Proposed", + "general": false + }, + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": false + } + ], + "diff_verify_against_oracle": [ + { + "number": "0001", + "title": "pickled-diff package", + "status": "Proposed", + "general": false + } + ], + "pickled_diff_draft_corpus": [ + { + "number": "0001", + "title": "pickled-diff package", + "status": "Proposed", + "general": false + } + ], + "pickled_diff_mcp": [ + { + "number": "0001", + "title": "pickled-diff package", + "status": "Proposed", + "general": false + } + ], + "pickled_diff_verify": [ + { + "number": "0001", + "title": "pickled-diff package", + "status": "Proposed", + "general": false + } + ], + "pickled_diff_run_all": [ + { + "number": "0001", + "title": "pickled-diff package", + "status": "Proposed", + "general": false + } + ], + "iac_diff_terraform_plans": [ + { + "number": "0001", + "title": "pickled-diff package", + "status": "Proposed", + "general": false + } + ], + "iac_draft_terraform_module": [ + { + "number": "0001", + "title": "pickled-diff package", + "status": "Proposed", + "general": false + } + ], + "iac_explain_plan_diff": [], + "iac_suggest_security_remediation": [], + "iac_validate_terraform_dir": [ + { + "number": "0001", + "title": "pickled-diff package", + "status": "Proposed", + "general": false + }, + { + "number": "0007", + "title": "Code-aware stories and docstring drift detection", + "status": "Accepted", + "general": false + } + ], + "pickled_iac_diff": [], + "pickled_iac_draft": [], + "pickled_iac_mcp": [], + "pickled_iac_plan_cmd": [], + "pickled_iac_scan": [], + "pickled_iac_validate": [ + { + "number": "0007", + "title": "Code-aware stories and docstring drift detection", + "status": "Accepted", + "general": false + } + ], + "pickled_iac_iacambiguitygate": [ + { + "number": "0005", + "title": "`pickled-spec mine` staged mining pipeline", + "status": "Accepted", + "general": false + } + ], + "pickled_iac_plandiffgate": [], + "pickled_iac_securitybaselinegate": [], + "pickled_iac_run_all": [], + "rules_check_ruleset_coverage": [ + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": false + } + ], + "rules_draft_ruleset_from_brief": [ + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": false + } + ], + "rules_list_rules": [], + "pickled_rules_check": [], + "pickled_rules_draft": [], + "pickled_rules_list_rules": [], + "pickled_rules_mcp": [], + "pickled_rules_coverage_gate": [ + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": false + } + ], + "pickled_rules_coverage_gate_features": [ + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": false + }, + { + "number": "0005", + "title": "`pickled-spec mine` staged mining pipeline", + "status": "Accepted", + "general": false + } + ], + "pickled_rules_run_all": [], + "schema_check_schema_coverage": [ + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": false + } + ], + "schema_draft_openapi_endpoint": [], + "schema_validate_openapi_spec": [ + { + "number": "0007", + "title": "Code-aware stories and docstring drift detection", + "status": "Accepted", + "general": false + } + ], + "pickled_schema_check": [], + "pickled_schema_draft": [], + "pickled_schema_mcp": [], + "pickled_schema_parse": [], + "pickled_schema_validate": [ + { + "number": "0007", + "title": "Code-aware stories and docstring drift detection", + "status": "Accepted", + "general": false + } + ], + "pickled_schema_schemaambiguitygate": [ + { + "number": "0005", + "title": "`pickled-spec mine` staged mining pipeline", + "status": "Accepted", + "general": false + } + ], + "pickled_schema_schemacoveragegate": [ + { + "number": "0004", + "title": "Multi-ruleset workspace configuration", + "status": "Accepted", + "general": false + } + ], + "pickled_schema_run_all": [] + } +} diff --git a/dogfood/mining-output/mining-report.md b/dogfood/mining-output/mining-report.md new file mode 100644 index 0000000..b3f6c25 --- /dev/null +++ b/dogfood/mining-output/mining-report.md @@ -0,0 +1,479 @@ +# Mining report + +Generated: `2026-05-28T15:14:17Z` +Target: `/Users/bartlomiejrosa/Projects/PORTFOLIO/pickled-spec` +Output: `/Users/bartlomiejrosa/Projects/PORTFOLIO/pickled-spec/dogfood/mining-output` + +## Summary + +- Packages: 7 +- CLI commands: 46 +- MCP tools: 19 +- Gates: 16 +- ADRs: 7 +- Workspaces: 2 + +- Stories on disk: 72 +- Features on disk: 72 +- Tag proposals: yes +- Coverage evaluation: yes +- Ambiguity evaluation: yes + +## Inventory highlights + +| Package | CLI commands | MCP tools | Gates | +|---------|-------------:|----------:|------:| +| pickled-bdd | 6 | 2 | 2 | +| pickled-core | 11 | 0 | 0 | +| pickled-data | 6 | 4 | 3 | +| pickled-diff | 5 | 2 | 1 | +| pickled-iac | 7 | 5 | 4 | +| pickled-rules | 5 | 3 | 3 | +| pickled-schema | 6 | 3 | 3 | + +### CLI commands + +| Package | Command | Help | +|---------|---------|------| +| `pickled-bdd` | `ambiguity` | Run the ambiguity gate (alias for ``check --gate ambiguity``). | +| `pickled-bdd` | `check` | Run compensating gates against a .feature file. | +| `pickled-bdd` | `draft` | Draft a .feature file from a user story (Markdown). | +| `pickled-bdd` | `mcp` | MCP server commands. | +| `pickled-bdd` | `mcp serve` | Run the pickled-bdd MCP server. | +| `pickled-bdd` | `serve` | Deprecated alias for ``pickled-bdd mcp serve``. | +| `pickled-core` | `check-all` | Run workspace gates from every pickled-* package against a directory. | +| `pickled-core` | `mcp` | Run the umbrella MCP server (all family packages mounted). | +| `pickled-core` | `mine` | Mine a Python project for surfaces, stories, features, and gate results. | +| `pickled-core` | `mine all` | Run inventory → code → stories → features → tag → evaluate → report. | +| `pickled-core` | `mine code` | Stage 2: extract code context per surface from inventory.json. | +| `pickled-core` | `mine evaluate` | Stage 6: evaluate coverage and ambiguity gates. | +| `pickled-core` | `mine features` | Stage 4: draft features from stories. | +| `pickled-core` | `mine inventory` | Stage 1: introspect target and write inventory.json. | +| `pickled-core` | `mine report` | Stage 7: render mining-report.md from pipeline outputs. | +| `pickled-core` | `mine stories` | Stage 3: emit stories from inventory.json. | +| `pickled-core` | `mine tag` | Stage 5: tag scenarios in generated features. | +| `pickled-data` | `apply` | Apply migration to in-memory SQLite and print resulting schema. | +| `pickled-data` | `check-drift` | Run MigrationDriftGate against expected schema YAML. | +| `pickled-data` | `draft` | Draft a SQL migration from a natural-language intent. | +| `pickled-data` | `mcp` | MCP server commands. | +| `pickled-data` | `mcp serve` | | +| `pickled-data` | `parse` | Parse a migration SQL file and print AST summary. | +| `pickled-diff` | `draft-corpus` | Expand seed examples into a larger differential corpus. | +| `pickled-diff` | `mcp` | MCP server commands. | +| `pickled-diff` | `mcp serve` | Run the pickled-diff MCP server. | +| `pickled-diff` | `serve` | Deprecated alias for ``pickled-diff mcp serve``. | +| `pickled-diff` | `verify` | Compare candidate vs reference across a JSON input corpus. | +| `pickled-iac` | `diff` | Compare two terraform plan JSON files. | +| `pickled-iac` | `draft` | Draft a Terraform module from a user story. | +| `pickled-iac` | `mcp` | MCP server commands. | +| `pickled-iac` | `mcp serve` | | +| `pickled-iac` | `plan-cmd` | Run terraform plan and write JSON to *output*. | +| `pickled-iac` | `scan` | Run Trivy config scan (optional; skips if trivy missing). | +| `pickled-iac` | `validate` | Run terraform validate on a directory. | +| `pickled-rules` | `check` | Check feature coverage against a YAML rule set. | +| `pickled-rules` | `draft` | Draft a YAML rule set from a natural-language brief. | +| `pickled-rules` | `list-rules` | List rule ids from a YAML rule set. | +| `pickled-rules` | `mcp` | MCP server commands. | +| `pickled-rules` | `mcp serve` | Run the pickled-rules MCP server. | +| `pickled-schema` | `check` | Run SchemaCoverageGate on @schema:endpoint tags in .feature files. | +| `pickled-schema` | `draft` | Draft an OpenAPI 3.1 path item from a Gherkin scenario. | +| `pickled-schema` | `mcp` | MCP server commands. | +| `pickled-schema` | `mcp serve` | Run the pickled-schema MCP server. | +| `pickled-schema` | `parse` | Parse a schema file and print a short summary. | +| `pickled-schema` | `validate` | Validate a schema file against its format specification. | + +## Stories generated + +| Surface | Path | +|---------|------| +| bdd_draft_feature_from_story.story | `stories/bdd_draft_feature_from_story.story.md` | +| bdd_validate_feature_ambiguity.story | `stories/bdd_validate_feature_ambiguity.story.md` | +| data_apply_sql_to_sandbox.story | `stories/data_apply_sql_to_sandbox.story.md` | +| data_check_migration_drift.story | `stories/data_check_migration_drift.story.md` | +| data_draft_sql_migration_from_intent.story | `stories/data_draft_sql_migration_from_intent.story.md` | +| data_parse_sql_migration.story | `stories/data_parse_sql_migration.story.md` | +| diff_draft_corpus_from_examples.story | `stories/diff_draft_corpus_from_examples.story.md` | +| diff_verify_against_oracle.story | `stories/diff_verify_against_oracle.story.md` | +| iac_diff_terraform_plans.story | `stories/iac_diff_terraform_plans.story.md` | +| iac_draft_terraform_module.story | `stories/iac_draft_terraform_module.story.md` | +| iac_explain_plan_diff.story | `stories/iac_explain_plan_diff.story.md` | +| iac_suggest_security_remediation.story | `stories/iac_suggest_security_remediation.story.md` | +| iac_validate_terraform_dir.story | `stories/iac_validate_terraform_dir.story.md` | +| pickled_bdd_ambiguity.story | `stories/pickled_bdd_ambiguity.story.md` | +| pickled_bdd_ambiguitygate.story | `stories/pickled_bdd_ambiguitygate.story.md` | +| pickled_bdd_check.story | `stories/pickled_bdd_check.story.md` | +| pickled_bdd_draft.story | `stories/pickled_bdd_draft.story.md` | +| pickled_bdd_mcp.story | `stories/pickled_bdd_mcp.story.md` | +| pickled_bdd_run_all.story | `stories/pickled_bdd_run_all.story.md` | +| pickled_core_check_all.story | `stories/pickled_core_check_all.story.md` | +| pickled_core_mine.story | `stories/pickled_core_mine.story.md` | +| pickled_core_mine_all.story | `stories/pickled_core_mine_all.story.md` | +| pickled_core_mine_code.story | `stories/pickled_core_mine_code.story.md` | +| pickled_core_mine_evaluate.story | `stories/pickled_core_mine_evaluate.story.md` | +| pickled_core_mine_features.story | `stories/pickled_core_mine_features.story.md` | +| pickled_core_mine_inventory.story | `stories/pickled_core_mine_inventory.story.md` | +| pickled_core_mine_report.story | `stories/pickled_core_mine_report.story.md` | +| pickled_core_mine_stories.story | `stories/pickled_core_mine_stories.story.md` | +| pickled_core_mine_tag.story | `stories/pickled_core_mine_tag.story.md` | +| pickled_data_apply.story | `stories/pickled_data_apply.story.md` | +| pickled_data_check_drift.story | `stories/pickled_data_check_drift.story.md` | +| pickled_data_datacontractgate.story | `stories/pickled_data_datacontractgate.story.md` | +| pickled_data_draft.story | `stories/pickled_data_draft.story.md` | +| pickled_data_mcp.story | `stories/pickled_data_mcp.story.md` | +| pickled_data_migrationdriftgate.story | `stories/pickled_data_migrationdriftgate.story.md` | +| pickled_data_parse.story | `stories/pickled_data_parse.story.md` | +| pickled_data_run_all.story | `stories/pickled_data_run_all.story.md` | +| pickled_diff_draft_corpus.story | `stories/pickled_diff_draft_corpus.story.md` | +| pickled_diff_mcp.story | `stories/pickled_diff_mcp.story.md` | +| pickled_diff_run_all.story | `stories/pickled_diff_run_all.story.md` | +| pickled_diff_verify.story | `stories/pickled_diff_verify.story.md` | +| pickled_iac_diff.story | `stories/pickled_iac_diff.story.md` | +| pickled_iac_draft.story | `stories/pickled_iac_draft.story.md` | +| pickled_iac_iacambiguitygate.story | `stories/pickled_iac_iacambiguitygate.story.md` | +| pickled_iac_mcp.story | `stories/pickled_iac_mcp.story.md` | +| pickled_iac_plan_cmd.story | `stories/pickled_iac_plan_cmd.story.md` | +| pickled_iac_plandiffgate.story | `stories/pickled_iac_plandiffgate.story.md` | +| pickled_iac_run_all.story | `stories/pickled_iac_run_all.story.md` | +| pickled_iac_scan.story | `stories/pickled_iac_scan.story.md` | +| pickled_iac_securitybaselinegate.story | `stories/pickled_iac_securitybaselinegate.story.md` | +| pickled_iac_validate.story | `stories/pickled_iac_validate.story.md` | +| pickled_rules_check.story | `stories/pickled_rules_check.story.md` | +| pickled_rules_coverage_gate.story | `stories/pickled_rules_coverage_gate.story.md` | +| pickled_rules_coverage_gate_features.story | `stories/pickled_rules_coverage_gate_features.story.md` | +| pickled_rules_draft.story | `stories/pickled_rules_draft.story.md` | +| pickled_rules_list_rules.story | `stories/pickled_rules_list_rules.story.md` | +| pickled_rules_mcp.story | `stories/pickled_rules_mcp.story.md` | +| pickled_rules_run_all.story | `stories/pickled_rules_run_all.story.md` | +| pickled_schema_check.story | `stories/pickled_schema_check.story.md` | +| pickled_schema_draft.story | `stories/pickled_schema_draft.story.md` | +| pickled_schema_mcp.story | `stories/pickled_schema_mcp.story.md` | +| pickled_schema_parse.story | `stories/pickled_schema_parse.story.md` | +| pickled_schema_run_all.story | `stories/pickled_schema_run_all.story.md` | +| pickled_schema_schemaambiguitygate.story | `stories/pickled_schema_schemaambiguitygate.story.md` | +| pickled_schema_schemacoveragegate.story | `stories/pickled_schema_schemacoveragegate.story.md` | +| pickled_schema_validate.story | `stories/pickled_schema_validate.story.md` | +| rules_check_ruleset_coverage.story | `stories/rules_check_ruleset_coverage.story.md` | +| rules_draft_ruleset_from_brief.story | `stories/rules_draft_ruleset_from_brief.story.md` | +| rules_list_rules.story | `stories/rules_list_rules.story.md` | +| schema_check_schema_coverage.story | `stories/schema_check_schema_coverage.story.md` | +| schema_draft_openapi_endpoint.story | `stories/schema_draft_openapi_endpoint.story.md` | +| schema_validate_openapi_spec.story | `stories/schema_validate_openapi_spec.story.md` | + +## Tag proposals + +- `features/bdd_draft_feature_from_story.feature`: 8 scenario(s) +- `features/bdd_validate_feature_ambiguity.feature`: 7 scenario(s) +- `features/data_apply_sql_to_sandbox.feature`: 8 scenario(s) +- `features/data_check_migration_drift.feature`: 8 scenario(s) +- `features/data_draft_sql_migration_from_intent.feature`: 11 scenario(s) +- `features/data_parse_sql_migration.feature`: 10 scenario(s) +- `features/diff_draft_corpus_from_examples.feature`: 6 scenario(s) +- `features/diff_verify_against_oracle.feature`: 7 scenario(s) +- `features/iac_diff_terraform_plans.feature`: 9 scenario(s) +- `features/iac_draft_terraform_module.feature`: 11 scenario(s) +- `features/iac_explain_plan_diff.feature`: 9 scenario(s) +- `features/iac_suggest_security_remediation.feature`: 7 scenario(s) +- `features/iac_validate_terraform_dir.feature`: 7 scenario(s) +- `features/pickled_bdd_ambiguity.feature`: 10 scenario(s) +- `features/pickled_bdd_ambiguitygate.feature`: 11 scenario(s) +- `features/pickled_bdd_check.feature`: 8 scenario(s) +- `features/pickled_bdd_draft.feature`: 7 scenario(s) +- `features/pickled_bdd_mcp.feature`: 3 scenario(s) +- `features/pickled_bdd_run_all.feature`: 8 scenario(s) +- `features/pickled_core_check_all.feature`: 10 scenario(s) +- `features/pickled_core_mine.feature`: 6 scenario(s) +- `features/pickled_core_mine_all.feature`: 12 scenario(s) +- `features/pickled_core_mine_code.feature`: 13 scenario(s) +- `features/pickled_core_mine_evaluate.feature`: 11 scenario(s) +- `features/pickled_core_mine_features.feature`: 10 scenario(s) +- `features/pickled_core_mine_inventory.feature`: 8 scenario(s) +- `features/pickled_core_mine_report.feature`: 7 scenario(s) +- `features/pickled_core_mine_stories.feature`: 14 scenario(s) +- `features/pickled_core_mine_tag.feature`: 9 scenario(s) +- `features/pickled_data_apply.feature`: 19 scenario(s) +- `features/pickled_data_check_drift.feature`: 7 scenario(s) +- `features/pickled_data_datacontractgate.feature`: 12 scenario(s) +- `features/pickled_data_draft.feature`: 17 scenario(s) +- `features/pickled_data_mcp.feature`: 5 scenario(s) +- `features/pickled_data_migrationdriftgate.feature`: 18 scenario(s) +- `features/pickled_data_parse.feature`: 8 scenario(s) +- `features/pickled_data_run_all.feature`: 15 scenario(s) +- `features/pickled_diff_draft_corpus.feature`: 8 scenario(s) +- `features/pickled_diff_mcp.feature`: 3 scenario(s) +- `features/pickled_diff_run_all.feature`: 11 scenario(s) +- `features/pickled_diff_verify.feature`: 17 scenario(s) +- `features/pickled_iac_diff.feature`: 19 scenario(s) +- `features/pickled_iac_draft.feature`: 14 scenario(s) +- `features/pickled_iac_iacambiguitygate.feature`: 14 scenario(s) +- `features/pickled_iac_mcp.feature`: 6 scenario(s) +- `features/pickled_iac_plan_cmd.feature`: 6 scenario(s) +- `features/pickled_iac_plandiffgate.feature`: 25 scenario(s) +- `features/pickled_iac_run_all.feature`: 14 scenario(s) +- `features/pickled_iac_scan.feature`: 10 scenario(s) +- `features/pickled_iac_securitybaselinegate.feature`: 13 scenario(s) +- `features/pickled_iac_validate.feature`: 6 scenario(s) +- `features/pickled_rules_check.feature`: 20 scenario(s) +- `features/pickled_rules_coverage_gate.feature`: 14 scenario(s) +- `features/pickled_rules_coverage_gate_features.feature`: 12 scenario(s) +- `features/pickled_rules_draft.feature`: 13 scenario(s) +- `features/pickled_rules_list_rules.feature`: 6 scenario(s) +- `features/pickled_rules_mcp.feature`: 3 scenario(s) +- `features/pickled_rules_run_all.feature`: 19 scenario(s) +- `features/pickled_schema_check.feature`: 14 scenario(s) +- `features/pickled_schema_draft.feature`: 19 scenario(s) +- `features/pickled_schema_mcp.feature`: 2 scenario(s) +- `features/pickled_schema_parse.feature`: 10 scenario(s) +- `features/pickled_schema_run_all.feature`: 22 scenario(s) +- `features/pickled_schema_schemaambiguitygate.feature`: 14 scenario(s) +- `features/pickled_schema_schemacoveragegate.feature`: 23 scenario(s) +- `features/pickled_schema_validate.feature`: 14 scenario(s) +- `features/rules_check_ruleset_coverage.feature`: 7 scenario(s) +- `features/rules_draft_ruleset_from_brief.feature`: 8 scenario(s) +- `features/rules_list_rules.feature`: 7 scenario(s) +- `features/schema_check_schema_coverage.feature`: 7 scenario(s) +- `features/schema_draft_openapi_endpoint.feature`: 8 scenario(s) +- `features/schema_validate_openapi_spec.feature`: 5 scenario(s) + +## Features generated + +| Surface | Scenarios | Path | +|---------|----------:|------| +| bdd_draft_feature_from_story | 8 | `features/bdd_draft_feature_from_story.feature` | +| bdd_validate_feature_ambiguity | 6 | `features/bdd_validate_feature_ambiguity.feature` | +| data_apply_sql_to_sandbox | 6 | `features/data_apply_sql_to_sandbox.feature` | +| data_check_migration_drift | 7 | `features/data_check_migration_drift.feature` | +| data_draft_sql_migration_from_intent | 10 | `features/data_draft_sql_migration_from_intent.feature` | +| data_parse_sql_migration | 9 | `features/data_parse_sql_migration.feature` | +| diff_draft_corpus_from_examples | 5 | `features/diff_draft_corpus_from_examples.feature` | +| diff_verify_against_oracle | 6 | `features/diff_verify_against_oracle.feature` | +| iac_diff_terraform_plans | 8 | `features/iac_diff_terraform_plans.feature` | +| iac_draft_terraform_module | 9 | `features/iac_draft_terraform_module.feature` | +| iac_explain_plan_diff | 9 | `features/iac_explain_plan_diff.feature` | +| iac_suggest_security_remediation | 7 | `features/iac_suggest_security_remediation.feature` | +| iac_validate_terraform_dir | 7 | `features/iac_validate_terraform_dir.feature` | +| pickled_bdd_ambiguity | 8 | `features/pickled_bdd_ambiguity.feature` | +| pickled_bdd_ambiguitygate | 10 | `features/pickled_bdd_ambiguitygate.feature` | +| pickled_bdd_check | 7 | `features/pickled_bdd_check.feature` | +| pickled_bdd_draft | 7 | `features/pickled_bdd_draft.feature` | +| pickled_bdd_mcp | 3 | `features/pickled_bdd_mcp.feature` | +| pickled_bdd_run_all | 7 | `features/pickled_bdd_run_all.feature` | +| pickled_core_check_all | 10 | `features/pickled_core_check_all.feature` | +| pickled_core_mine | 6 | `features/pickled_core_mine.feature` | +| pickled_core_mine_all | 11 | `features/pickled_core_mine_all.feature` | +| pickled_core_mine_code | 12 | `features/pickled_core_mine_code.feature` | +| pickled_core_mine_evaluate | 10 | `features/pickled_core_mine_evaluate.feature` | +| pickled_core_mine_features | 9 | `features/pickled_core_mine_features.feature` | +| pickled_core_mine_inventory | 7 | `features/pickled_core_mine_inventory.feature` | +| pickled_core_mine_report | 6 | `features/pickled_core_mine_report.feature` | +| pickled_core_mine_stories | 13 | `features/pickled_core_mine_stories.feature` | +| pickled_core_mine_tag | 8 | `features/pickled_core_mine_tag.feature` | +| pickled_data_apply | 19 | `features/pickled_data_apply.feature` | +| pickled_data_check_drift | 7 | `features/pickled_data_check_drift.feature` | +| pickled_data_datacontractgate | 11 | `features/pickled_data_datacontractgate.feature` | +| pickled_data_draft | 16 | `features/pickled_data_draft.feature` | +| pickled_data_mcp | 5 | `features/pickled_data_mcp.feature` | +| pickled_data_migrationdriftgate | 18 | `features/pickled_data_migrationdriftgate.feature` | +| pickled_data_parse | 7 | `features/pickled_data_parse.feature` | +| pickled_data_run_all | 13 | `features/pickled_data_run_all.feature` | +| pickled_diff_draft_corpus | 8 | `features/pickled_diff_draft_corpus.feature` | +| pickled_diff_mcp | 3 | `features/pickled_diff_mcp.feature` | +| pickled_diff_run_all | 8 | `features/pickled_diff_run_all.feature` | +| pickled_diff_verify | 15 | `features/pickled_diff_verify.feature` | +| pickled_iac_diff | 18 | `features/pickled_iac_diff.feature` | +| pickled_iac_draft | 13 | `features/pickled_iac_draft.feature` | +| pickled_iac_iacambiguitygate | 13 | `features/pickled_iac_iacambiguitygate.feature` | +| pickled_iac_mcp | 6 | `features/pickled_iac_mcp.feature` | +| pickled_iac_plan_cmd | 5 | `features/pickled_iac_plan_cmd.feature` | +| pickled_iac_plandiffgate | 24 | `features/pickled_iac_plandiffgate.feature` | +| pickled_iac_run_all | 14 | `features/pickled_iac_run_all.feature` | +| pickled_iac_scan | 9 | `features/pickled_iac_scan.feature` | +| pickled_iac_securitybaselinegate | 11 | `features/pickled_iac_securitybaselinegate.feature` | +| pickled_iac_validate | 6 | `features/pickled_iac_validate.feature` | +| pickled_rules_check | 19 | `features/pickled_rules_check.feature` | +| pickled_rules_coverage_gate | 14 | `features/pickled_rules_coverage_gate.feature` | +| pickled_rules_coverage_gate_features | 12 | `features/pickled_rules_coverage_gate_features.feature` | +| pickled_rules_draft | 13 | `features/pickled_rules_draft.feature` | +| pickled_rules_list_rules | 5 | `features/pickled_rules_list_rules.feature` | +| pickled_rules_mcp | 3 | `features/pickled_rules_mcp.feature` | +| pickled_rules_run_all | 19 | `features/pickled_rules_run_all.feature` | +| pickled_schema_check | 13 | `features/pickled_schema_check.feature` | +| pickled_schema_draft | 19 | `features/pickled_schema_draft.feature` | +| pickled_schema_mcp | 2 | `features/pickled_schema_mcp.feature` | +| pickled_schema_parse | 9 | `features/pickled_schema_parse.feature` | +| pickled_schema_run_all | 21 | `features/pickled_schema_run_all.feature` | +| pickled_schema_schemaambiguitygate | 11 | `features/pickled_schema_schemaambiguitygate.feature` | +| pickled_schema_schemacoveragegate | 23 | `features/pickled_schema_schemacoveragegate.feature` | +| pickled_schema_validate | 13 | `features/pickled_schema_validate.feature` | +| rules_check_ruleset_coverage | 6 | `features/rules_check_ruleset_coverage.feature` | +| rules_draft_ruleset_from_brief | 7 | `features/rules_draft_ruleset_from_brief.feature` | +| rules_list_rules | 6 | `features/rules_list_rules.feature` | +| schema_check_schema_coverage | 7 | `features/schema_check_schema_coverage.feature` | +| schema_draft_openapi_endpoint | 7 | `features/schema_draft_openapi_endpoint.feature` | +| schema_validate_openapi_spec | 5 | `features/schema_validate_openapi_spec.feature` | + +## Coverage by rule set + +| Rule set | Verdict | Unreferenced strict | +|----------|---------|--------------------:| +| pickled-internal | pass | 0 | +| best-practices | pass | 0 | +| oss-hygiene | pass | 0 | +| bdd-domain | fail | 1 | + +Unreferenced strict rules in `bdd-domain`: +- `gherkin-then-asserts-observable-outcome` + +| rules-domain | pass | 0 | +| schema-domain | pass | 0 | +| iac-domain | pass | 0 | +| data-domain | pass | 0 | +| diff-domain | pass | 0 | +| core-domain | pass | 0 | + +## Ambiguity by feature + +| Feature | Verdict | Findings | Skipped | +|---------|---------|----------:|---------| +| `features/bdd_draft_feature_from_story.feature` | fail | 8 | no | +| `features/bdd_validate_feature_ambiguity.feature` | fail | 12 | no | +| `features/data_apply_sql_to_sandbox.feature` | fail | 13 | no | +| `features/data_check_migration_drift.feature` | fail | 11 | no | +| `features/data_draft_sql_migration_from_intent.feature` | fail | 13 | no | +| `features/data_parse_sql_migration.feature` | fail | 14 | no | +| `features/diff_draft_corpus_from_examples.feature` | fail | 9 | no | +| `features/diff_verify_against_oracle.feature` | fail | 9 | no | +| `features/iac_diff_terraform_plans.feature` | fail | 14 | no | +| `features/iac_draft_terraform_module.feature` | fail | 15 | no | +| `features/iac_explain_plan_diff.feature` | fail | 9 | no | +| `features/iac_suggest_security_remediation.feature` | fail | 7 | no | +| `features/iac_validate_terraform_dir.feature` | fail | 7 | no | +| `features/pickled_bdd_ambiguity.feature` | fail | 14 | no | +| `features/pickled_bdd_ambiguitygate.feature` | warn | 12 | no | +| `features/pickled_bdd_check.feature` | fail | 12 | no | +| `features/pickled_bdd_draft.feature` | fail | 7 | no | +| `features/pickled_bdd_mcp.feature` | fail | 3 | no | +| `features/pickled_bdd_run_all.feature` | fail | 10 | no | +| `features/pickled_core_check_all.feature` | fail | 10 | no | +| `features/pickled_core_mine.feature` | fail | 6 | no | +| `features/pickled_core_mine_all.feature` | fail | 13 | no | +| `features/pickled_core_mine_code.feature` | fail | 14 | no | +| `features/pickled_core_mine_evaluate.feature` | fail | 13 | no | +| `features/pickled_core_mine_features.feature` | fail | 13 | no | +| `features/pickled_core_mine_inventory.feature` | fail | 10 | no | +| `features/pickled_core_mine_report.feature` | fail | 10 | no | +| `features/pickled_core_mine_stories.feature` | fail | 15 | no | +| `features/pickled_core_mine_tag.feature` | fail | 10 | no | +| `features/pickled_data_apply.feature` | fail | 19 | no | +| `features/pickled_data_check_drift.feature` | fail | 7 | no | +| `features/pickled_data_datacontractgate.feature` | fail | 14 | no | +| `features/pickled_data_draft.feature` | fail | 20 | no | +| `features/pickled_data_mcp.feature` | fail | 5 | no | +| `features/pickled_data_migrationdriftgate.feature` | fail | 18 | no | +| `features/pickled_data_parse.feature` | fail | 9 | no | +| `features/pickled_data_run_all.feature` | fail | 18 | no | +| `features/pickled_diff_draft_corpus.feature` | fail | 8 | no | +| `features/pickled_diff_mcp.feature` | fail | 3 | no | +| `features/pickled_diff_run_all.feature` | fail | 17 | no | +| `features/pickled_diff_verify.feature` | fail | 22 | no | +| `features/pickled_iac_diff.feature` | fail | 22 | no | +| `features/pickled_iac_draft.feature` | warn | 15 | no | +| `features/pickled_iac_iacambiguitygate.feature` | error | 0 | yes | +| `features/pickled_iac_mcp.feature` | fail | 6 | no | +| `features/pickled_iac_plan_cmd.feature` | fail | 8 | no | +| `features/pickled_iac_plandiffgate.feature` | fail | 28 | no | +| `features/pickled_iac_run_all.feature` | fail | 14 | no | +| `features/pickled_iac_scan.feature` | fail | 12 | no | +| `features/pickled_iac_securitybaselinegate.feature` | fail | 20 | no | +| `features/pickled_iac_validate.feature` | fail | 6 | no | +| `features/pickled_rules_check.feature` | fail | 25 | no | +| `features/pickled_rules_coverage_gate.feature` | fail | 14 | no | +| `features/pickled_rules_coverage_gate_features.feature` | fail | 12 | no | +| `features/pickled_rules_draft.feature` | fail | 13 | no | +| `features/pickled_rules_list_rules.feature` | fail | 8 | no | +| `features/pickled_rules_mcp.feature` | fail | 3 | no | +| `features/pickled_rules_run_all.feature` | fail | 19 | no | +| `features/pickled_schema_check.feature` | fail | 16 | no | +| `features/pickled_schema_draft.feature` | fail | 19 | no | +| `features/pickled_schema_mcp.feature` | fail | 2 | no | +| `features/pickled_schema_parse.feature` | error | 0 | yes | +| `features/pickled_schema_run_all.feature` | fail | 25 | no | +| `features/pickled_schema_schemaambiguitygate.feature` | warn | 17 | no | +| `features/pickled_schema_schemacoveragegate.feature` | fail | 23 | no | +| `features/pickled_schema_validate.feature` | fail | 18 | no | +| `features/rules_check_ruleset_coverage.feature` | fail | 9 | no | +| `features/rules_draft_ruleset_from_brief.feature` | fail | 12 | no | +| `features/rules_list_rules.feature` | fail | 10 | no | +| `features/schema_check_schema_coverage.feature` | fail | 7 | no | +| `features/schema_draft_openapi_endpoint.feature` | fail | 12 | no | +| `features/schema_validate_openapi_spec.feature` | fail | 5 | no | + +## Suggested next moves + +- Add a story/feature covering strict rules: gherkin-then-asserts-observable-outcome +- Re-draft `bdd_draft_feature_from_story` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `bdd_validate_feature_ambiguity` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `data_apply_sql_to_sandbox` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `data_check_migration_drift` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `data_draft_sql_migration_from_intent` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `data_parse_sql_migration` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `diff_draft_corpus_from_examples` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `diff_verify_against_oracle` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `iac_diff_terraform_plans` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `iac_draft_terraform_module` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `iac_explain_plan_diff` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `iac_suggest_security_remediation` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `iac_validate_terraform_dir` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_bdd_ambiguity` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_bdd_check` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_bdd_draft` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_bdd_mcp` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_bdd_run_all` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_core_check_all` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_core_mine` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_core_mine_all` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_core_mine_code` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_core_mine_evaluate` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_core_mine_features` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_core_mine_inventory` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_core_mine_report` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_core_mine_stories` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_core_mine_tag` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_data_apply` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_data_check_drift` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_data_datacontractgate` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_data_draft` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_data_mcp` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_data_migrationdriftgate` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_data_parse` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_data_run_all` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_diff_draft_corpus` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_diff_mcp` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_diff_run_all` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_diff_verify` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_iac_diff` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_iac_mcp` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_iac_plan_cmd` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_iac_plandiffgate` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_iac_run_all` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_iac_scan` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_iac_securitybaselinegate` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_iac_validate` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_rules_check` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_rules_coverage_gate` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_rules_coverage_gate_features` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_rules_draft` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_rules_list_rules` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_rules_mcp` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_rules_run_all` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_schema_check` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_schema_draft` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_schema_mcp` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_schema_run_all` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_schema_schemacoveragegate` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `pickled_schema_validate` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `rules_check_ruleset_coverage` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `rules_draft_ruleset_from_brief` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `rules_list_rules` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `schema_check_schema_coverage` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `schema_draft_openapi_endpoint` with tighter scope; see evaluation/ambiguity.json findings. +- Re-draft `schema_validate_openapi_spec` with tighter scope; see evaluation/ambiguity.json findings. diff --git a/dogfood/mining-output/stories/bdd_draft_feature_from_story.story.md b/dogfood/mining-output/stories/bdd_draft_feature_from_story.story.md new file mode 100644 index 0000000..0c038bd --- /dev/null +++ b/dogfood/mining-output/stories/bdd_draft_feature_from_story.story.md @@ -0,0 +1,45 @@ +# Story: bdd_draft_feature_from_story + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-bdd +- **Surface id:** bdd_draft_feature_from_story + +## Context + +BDD practitioners use this tool to quickly generate a first-draft Gherkin feature file by providing a natural-language user story. The tool accelerates the transition from informal requirements to structured, executable specification format, reducing manual boilerplate and helping teams adopt BDD practices more efficiently. + +## What the target does today + +**bdd_draft_feature_from_story** accepts a required `story_text` argument containing a natural-language user story and returns a drafted Gherkin `.feature` file structure. + +The tool produces output that follows Gherkin syntax conventions, typically including Feature, Scenario, and step keywords (Given/When/Then). The draft serves as a starting point that practitioners can refine. + +The surface is gated by `AmbiguityGate.run` and `run_all`, meaning the operation will fail if the story text triggers ambiguity detection or other registered gate conditions. The exact nature of these gate checks must be confirmed from source, as the inventory does not document their specific behavior. + +## What we want to verify + +- Calling the tool with a simple user story string returns a response containing valid Gherkin keywords (Feature, Scenario, Given, When, Then). +- The output can be written to a `.feature` file without syntax errors. +- Providing an empty or whitespace-only `story_text` either returns an error or produces a minimal valid feature template. +- The tool invokes `AmbiguityGate.run` before or during processing (observable via instrumentation or logs). +- Gate failure prevents feature generation and returns an appropriate error message. +- The generated feature content references or incorporates elements from the input `story_text`. +- Multiple invocations with the same `story_text` produce consistent output structure. + +## Inventory references + +- Arguments: +- `story_text` (required): +- Related gates: AmbiguityGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/bdd_validate_feature_ambiguity.story.md b/dogfood/mining-output/stories/bdd_validate_feature_ambiguity.story.md new file mode 100644 index 0000000..6c90c20 --- /dev/null +++ b/dogfood/mining-output/stories/bdd_validate_feature_ambiguity.story.md @@ -0,0 +1,43 @@ +# Story: bdd_validate_feature_ambiguity + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-bdd +- **Surface id:** bdd_validate_feature_ambiguity + +## Context + +This tool is used by BDD practitioners and test automation engineers who need to validate Gherkin feature files for ambiguous step definitions before running tests. It serves as a quality gate in the BDD workflow to catch ambiguity issues early—particularly valuable when integrating feature files into CI/CD pipelines or when reviewing feature specifications. The tool connects to the `pickled-bdd` package's ambiguity detection capabilities, specifically wrapping `AmbiguityGate.run` for use as an MCP tool. + +## What the target does today + +Run the ambiguity gate against a Gherkin .feature file. + +The tool accepts feature file text as input via the `feature_text` parameter (required) and executes ambiguity validation through the underlying `AmbiguityGate.run` gate. The gate is one of several available validation gates (alongside `run_all`), suggesting a modular validation architecture where different quality checks can be run independently or together. + +## What we want to verify + +- Accepts `feature_text` parameter containing Gherkin feature file content +- Invokes `AmbiguityGate.run` with the provided feature text +- Returns validation results indicating whether ambiguities were detected +- Handles malformed or invalid Gherkin syntax appropriately +- Can be invoked as an MCP tool within the pickled-bdd toolchain +- Functions independently of other gates (does not require `run_all` to operate) + +## Inventory references + +- Arguments: +- `feature_text` (required): +- Related gates: AmbiguityGate.run, run_all +- Related ADRs: +- ADR 0005: `pickled-spec mine` staged mining pipeline — Accepted +- ADR 0007: Code-aware stories and docstring drift detection — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/data_apply_sql_to_sandbox.story.md b/dogfood/mining-output/stories/data_apply_sql_to_sandbox.story.md new file mode 100644 index 0000000..bdb9cb8 --- /dev/null +++ b/dogfood/mining-output/stories/data_apply_sql_to_sandbox.story.md @@ -0,0 +1,47 @@ +# Story: data_apply_sql_to_sandbox + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-data +- **Surface id:** data_apply_sql_to_sandbox + +## Context + +This surface is an MCP tool in the pickled-data package that allows users to execute SQL statements against an in-memory SQLite database sandbox and receive back the resulting schema. It is used by callers who need to validate SQL queries, explore database structure changes, or test SQL transformations in isolation without affecting persistent data stores. The tool accepts SQL commands and optionally a dialect parameter, then returns schema information describing the resulting database state. + +## What the target does today + +Apply SQL to in-memory SQLite and return schema. + +The surface accepts SQL statements via the `sql` parameter (required) and executes them against an ephemeral SQLite database instance. An optional `dialect` parameter may be provided to influence SQL interpretation or formatting. After applying the SQL, the tool returns schema information describing the database structure that resulted from the SQL execution. + +## What we want to verify + +- Executing valid CREATE TABLE SQL returns schema information describing the created table +- Executing multiple SQL statements (e.g., CREATE TABLE followed by ALTER TABLE) returns the final schema state +- The returned schema includes table names present after SQL execution +- The returned schema includes column definitions for created tables +- Providing invalid SQL produces an error response rather than schema output +- Executing SQL that creates no tables returns an empty or minimal schema representation +- The in-memory database is isolated and does not persist between invocations +- The optional `dialect` parameter, when provided, is accepted without error +- Executing DROP TABLE SQL removes the table from the returned schema +- The schema output format is consistent and parseable across different SQL inputs + +## Inventory references + +- Arguments: +- `sql` (required): +- `dialect` (optional): +- Related gates: DataContractGate.run, MigrationDriftGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/data_check_migration_drift.story.md b/dogfood/mining-output/stories/data_check_migration_drift.story.md new file mode 100644 index 0000000..b537ff8 --- /dev/null +++ b/dogfood/mining-output/stories/data_check_migration_drift.story.md @@ -0,0 +1,42 @@ +# Story: data_check_migration_drift + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-data +- **Surface id:** data_check_migration_drift + +## Context + +This tool is used by teams that want to validate that a SQL migration script produces a database schema that matches an expected schema definition written in YAML. It is likely invoked during CI/CD pipelines or local development workflows to catch schema drift before migrations are applied to production databases. The caller provides the SQL migration script, the expected schema in YAML format, and optionally a SQL dialect to ensure the migration produces the correct schema structure. + +## What the target does today + +The tool compares the schema resulting from a migration script against an expected schema defined in YAML format. It accepts a required `sql` parameter containing the migration SQL, a required `expected_schema_yaml` parameter with the schema specification, and an optional `dialect` parameter to specify the SQL dialect for execution or parsing. + +## What we want to verify + +- Accept a `sql` parameter containing migration SQL script +- Accept an `expected_schema_yaml` parameter containing the expected schema definition +- Accept an optional `dialect` parameter for SQL dialect specification +- Perform comparison between the migration result schema and the expected YAML schema +- Report whether the migration result matches the expected schema or indicate drift +- Return results that can be consumed by the caller to determine migration validity + +## Inventory references + +- Arguments: +- `sql` (required): +- `expected_schema_yaml` (required): +- `dialect` (optional): +- Related gates: DataContractGate.run, MigrationDriftGate.run, run_all +- Related ADRs: +- ADR 0007: Code-aware stories and docstring drift detection — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/data_draft_sql_migration_from_intent.story.md b/dogfood/mining-output/stories/data_draft_sql_migration_from_intent.story.md new file mode 100644 index 0000000..024547e --- /dev/null +++ b/dogfood/mining-output/stories/data_draft_sql_migration_from_intent.story.md @@ -0,0 +1,46 @@ +# Story: data_draft_sql_migration_from_intent + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-data +- **Surface id:** data_draft_sql_migration_from_intent + +## Context + +This tool is used by developers or automated workflows within the pickled-data package to generate SQL migration scripts from natural language intent descriptions. Callers provide a description of desired schema changes along with the target SQL dialect, and optionally the current schema state in YAML format. This surface bridges human intent with executable database migrations, typically invoked when evolving data contracts or schemas based on requirements expressed in plain text rather than directly writing SQL DDL. + +## What the target does today + +The inventory provides no docstring for this surface; behavior must be confirmed from source before specifying. + +## What we want to verify + +- Tool accepts `intent_text` parameter containing natural language description of schema changes +- Tool accepts `dialect` parameter specifying target SQL dialect (e.g., PostgreSQL, MySQL, SQLite) +- Tool accepts optional `current_schema_yaml` parameter containing existing schema definition +- Tool returns valid SQL migration statements appropriate for the specified dialect +- When `current_schema_yaml` is provided, generated migration reflects transition from current to intended state +- When `current_schema_yaml` is omitted, tool behavior regarding baseline assumptions must be verified from implementation +- Tool execution triggers or respects DataContractGate.run validation +- Tool execution triggers or respects MigrationDriftGate.run checks +- Generated SQL is syntactically valid for the specified dialect +- Tool handles ambiguous or conflicting intent descriptions (error or best-effort behavior to be confirmed) + +## Inventory references + +- Arguments: +- `intent_text` (required): +- `dialect` (required): +- `current_schema_yaml` (optional): +- Related gates: DataContractGate.run, MigrationDriftGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/data_parse_sql_migration.story.md b/dogfood/mining-output/stories/data_parse_sql_migration.story.md new file mode 100644 index 0000000..3887f44 --- /dev/null +++ b/dogfood/mining-output/stories/data_parse_sql_migration.story.md @@ -0,0 +1,45 @@ +# Story: data_parse_sql_migration + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-data +- **Surface id:** data_parse_sql_migration + +## Context + +This tool is used by data engineers and migration authors who need to parse SQL migration files and extract structural information about the schema changes. It supports the migration drift detection workflow by converting SQL text into a structured AST (Abstract Syntax Tree) that can be analyzed programmatically. Related gates like MigrationDriftGate.run likely consume this parsed output to detect drift between expected and actual migration state. + +## What the target does today + +Parse SQL and return AST summary. + +The tool accepts SQL text as input and an optional dialect parameter. It parses the SQL into an abstract syntax tree representation and returns a summary of that structure, enabling programmatic analysis of schema migration commands. + +## What we want to verify + +- Accepts a required `sql` parameter containing SQL text +- Accepts an optional `dialect` parameter to specify SQL dialect for parsing +- Returns an AST summary structure representing the parsed SQL +- Successfully parses valid SQL migration statements (CREATE TABLE, ALTER TABLE, etc.) +- Handles SQL syntax appropriate to the specified dialect when provided +- Returns structured output that can be consumed by drift detection gates +- Handles empty or whitespace-only SQL input gracefully +- Reports parse errors for malformed SQL syntax + +## Inventory references + +- Arguments: +- `sql` (required): +- `dialect` (optional): +- Related gates: DataContractGate.run, MigrationDriftGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/diff_draft_corpus_from_examples.story.md b/dogfood/mining-output/stories/diff_draft_corpus_from_examples.story.md new file mode 100644 index 0000000..be5378f --- /dev/null +++ b/dogfood/mining-output/stories/diff_draft_corpus_from_examples.story.md @@ -0,0 +1,43 @@ +# Story: diff_draft_corpus_from_examples + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-diff +- **Surface id:** diff_draft_corpus_from_examples + +## Context + +This tool is used by developers or automated systems working with the pickled-diff package to generate draft corpora for differential testing. It takes a small set of seed examples and expands them to a target size, allowing users to create larger test datasets from a representative sample. The optional notes parameter enables documentation of the corpus generation process. + +## What the target does today + +The inventory provides no docstring for this surface; behavior must be confirmed from source before specifying. + +## What we want to verify + +- Accepts `seed_examples` as a required parameter +- Accepts `target_size` as a required parameter indicating desired corpus size +- Accepts optional `notes` parameter for additional context or documentation +- Produces a draft corpus output (format to be verified from source) +- Expands the provided seed examples to reach the specified target size (mechanism to be verified) +- Returns a result that can be consumed by or integrated with the `run_all` gate + +## Inventory references + +- Arguments: +- `seed_examples` (required): +- `target_size` (required): +- `notes` (optional): +- Related gates: run_all +- Related ADRs: +- ADR 0001: pickled-diff package — Proposed +- ADR 0004: Multi-ruleset workspace configuration — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/diff_verify_against_oracle.story.md b/dogfood/mining-output/stories/diff_verify_against_oracle.story.md new file mode 100644 index 0000000..75678f1 --- /dev/null +++ b/dogfood/mining-output/stories/diff_verify_against_oracle.story.md @@ -0,0 +1,46 @@ +# Story: diff_verify_against_oracle + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-diff +- **Surface id:** diff_verify_against_oracle + +## Context + +This tool is used by testers, CI pipelines, or developers who need to verify that a candidate implementation produces the same output as a reference (oracle) implementation across a corpus of test inputs. It automates regression testing and behavioral comparison between two commands by running both against multiple test items and reporting differences. + +## What the target does today + +The inventory provides no docstring for this surface; behavior must be confirmed from source before specifying. + +## What we want to verify + +- Tool accepts `oracle_command` parameter (required) for the reference command +- Tool accepts `candidate_command` parameter (required) for the command under test +- Tool accepts `corpus_items` parameter (required) for the set of inputs to test +- Tool accepts optional `comparator` parameter to customize difference detection +- Tool accepts optional `timeout_seconds` parameter to limit execution time per command +- Tool executes both oracle and candidate commands against each item in the corpus +- Tool reports differences between oracle and candidate outputs +- Related gate `run_all` exists in the target package + +## Inventory references + +- Arguments: +- `oracle_command` (required): +- `candidate_command` (required): +- `corpus_items` (required): +- `comparator` (optional): +- `timeout_seconds` (optional): +- Related gates: run_all +- Related ADRs: +- ADR 0001: pickled-diff package — Proposed + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/iac_diff_terraform_plans.story.md b/dogfood/mining-output/stories/iac_diff_terraform_plans.story.md new file mode 100644 index 0000000..af7f1ce --- /dev/null +++ b/dogfood/mining-output/stories/iac_diff_terraform_plans.story.md @@ -0,0 +1,41 @@ +# Story: iac_diff_terraform_plans + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-iac +- **Surface id:** iac_diff_terraform_plans + +## Context + +This MCP tool is called by automation workflows and CI/CD pipelines to analyze infrastructure changes between two Terraform plan states (base and head). Callers need to understand what resources will be added, modified, or destroyed when applying infrastructure-as-code changes, enabling them to validate changes before applying them to environments. + +## What the target does today + +The inventory provides no docstring for this surface; behavior must be confirmed from source before specifying. + +## What we want to verify + +- Accepts two required arguments: `base_plan_json` and `head_plan_json` +- Both arguments must be valid Terraform plan JSON format +- Returns a comparison result structure showing differences between the two plans +- Integrates with IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run, and run_all gate operations +- Handles cases where base_plan_json represents the current state and head_plan_json represents proposed changes +- Produces output that can be consumed by the related gate implementations + +## Inventory references + +- Arguments: +- `base_plan_json` (required): +- `head_plan_json` (required): +- Related gates: IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run, run_all +- Related ADRs: +- ADR 0001: pickled-diff package — Proposed + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/iac_draft_terraform_module.story.md b/dogfood/mining-output/stories/iac_draft_terraform_module.story.md new file mode 100644 index 0000000..df9444f --- /dev/null +++ b/dogfood/mining-output/stories/iac_draft_terraform_module.story.md @@ -0,0 +1,43 @@ +# Story: iac_draft_terraform_module + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-iac +- **Surface id:** iac_draft_terraform_module + +## Context + +Infrastructure-as-code engineers and automation workflows use this tool to generate Terraform module drafts from natural-language user stories. The surface sits behind the MCP (Model Context Protocol) tool interface and is part of the pickled-iac package's module generation pipeline. Callers expect to provide a user story description and optionally specify a cloud provider, receiving in return a drafted Terraform module that can then pass through related gates (IaCAmbiguityGate, PlanDiffGate, SecurityBaselineGate) for validation. + +## What the target does today + +The inventory provides no docstring for this surface; behavior must be confirmed from source before specifying. + +## What we want to verify + +- Call with a minimal user story string and verify a response is returned +- Call with `user_story` parameter populated and `provider` parameter omitted; confirm default provider handling or appropriate error +- Call with both `user_story` and `provider` parameters populated; verify provider-specific module generation +- Call with missing required `user_story` parameter; verify error or rejection +- Verify output format conforms to Terraform module structure expectations +- Confirm generated module can be consumed by related gates (IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run) +- Call with edge-case user stories (empty string, extremely long text, special characters); verify graceful handling +- Verify interaction with run_all gate when this tool's output is part of a multi-gate workflow + +## Inventory references + +- Arguments: +- `user_story` (required): +- `provider` (optional): +- Related gates: IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run, run_all +- Related ADRs: +- ADR 0001: pickled-diff package — Proposed + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/iac_explain_plan_diff.story.md b/dogfood/mining-output/stories/iac_explain_plan_diff.story.md new file mode 100644 index 0000000..c859e48 --- /dev/null +++ b/dogfood/mining-output/stories/iac_explain_plan_diff.story.md @@ -0,0 +1,44 @@ +# Story: iac_explain_plan_diff + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-iac +- **Surface id:** iac_explain_plan_diff + +## Context + +This tool is used by infrastructure-as-code workflows that need to understand the impact and risk of Terraform plan changes. Callers provide a Terraform plan JSON file and receive a summary highlighting potentially dangerous operations. This enables automated review gates, pre-deployment risk assessment, and human-readable explanations of infrastructure changes before they are applied. + +## What the target does today + +The tool accepts a Terraform plan JSON file as input and produces a summary that identifies risky actions within the plan. Based on the related gates (IaCAmbiguityGate, PlanDiffGate, SecurityBaselineGate), the tool likely evaluates the plan against multiple risk dimensions including ambiguous configurations, plan differences, and security baseline violations, then flags actions that meet risk criteria. + +## What we want to verify + +- Accepts valid Terraform plan JSON as the `plan_json` parameter +- Returns a summary output (format to be confirmed from implementation) +- Identifies and flags risky actions within the provided plan +- Invokes or integrates with IaCAmbiguityGate.run to detect ambiguous infrastructure configurations +- Invokes or integrates with PlanDiffGate.run to analyze plan differences +- Invokes or integrates with SecurityBaselineGate.run to check security policy violations +- Processes the plan_json parameter as required (non-optional) +- Handles malformed or invalid Terraform plan JSON appropriately (error behavior TBD) +- The summary distinguishes between different types of risk categories flagged by the gates +- Integration with run_all suggests batch or orchestrated gate execution + +## Inventory references + +- Arguments: +- `plan_json` (required): +- Related gates: IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/iac_suggest_security_remediation.story.md b/dogfood/mining-output/stories/iac_suggest_security_remediation.story.md new file mode 100644 index 0000000..a31e9cc --- /dev/null +++ b/dogfood/mining-output/stories/iac_suggest_security_remediation.story.md @@ -0,0 +1,43 @@ +# Story: iac_suggest_security_remediation + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-iac +- **Surface id:** iac_suggest_security_remediation + +## Context + +This surface is used by security and infrastructure engineers who have run Trivy configuration scans against Infrastructure-as-Code files and need actionable remediation advice. The tool bridges the gap between security findings (JSON output from Trivy) and concrete HCL code patches, allowing teams to fix misconfigurations without manually researching each vulnerability. + +## What the target does today + +The surface accepts Trivy configuration scan findings in JSON format and suggests HCL (HashiCorp Configuration Language) patches to remediate the identified security issues. An optional HCL text parameter allows the tool to provide context-aware suggestions specific to existing code. The tool generates patch recommendations that address the security findings reported by Trivy. + +## What we want to verify + +- The surface accepts valid Trivy config-scan JSON output as the `trivy_findings_json` parameter +- The surface returns suggested HCL patches corresponding to the security findings provided +- When `hcl_text` is provided, the suggestions reference or are tailored to the supplied HCL context +- When `hcl_text` is omitted, the surface still produces generic remediation patches based solely on the Trivy findings +- The surface handles Trivy JSON containing zero findings without error +- The surface handles Trivy JSON containing multiple findings and produces suggestions for each +- The output format is suitable for applying patches to HCL configuration files +- The surface validates that `trivy_findings_json` conforms to expected Trivy output schema before processing + +## Inventory references + +- Arguments: +- `trivy_findings_json` (required): +- `hcl_text` (optional): +- Related gates: IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/iac_validate_terraform_dir.story.md b/dogfood/mining-output/stories/iac_validate_terraform_dir.story.md new file mode 100644 index 0000000..95aa3ef --- /dev/null +++ b/dogfood/mining-output/stories/iac_validate_terraform_dir.story.md @@ -0,0 +1,43 @@ +# Story: iac_validate_terraform_dir + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-iac +- **Surface id:** iac_validate_terraform_dir + +## Context + +This tool is used by AI agents or automation workflows that need to validate Terraform infrastructure-as-code files before applying them. The caller has written Terraform configuration files to a temporary directory and wants to verify the syntax and structure are valid before proceeding with planning or deployment. This surface is part of the pickled-iac package's MCP tool interface, making it accessible to agents that consume MCP-compatible tools. + +## What the target does today + +Validate Terraform files written to a temp directory. + +The surface accepts a required `tf_files` parameter and performs validation on Terraform configuration files in a temporary directory location. The validation likely checks syntax correctness and structural validity of the Terraform code. Related gates (IaCAmbiguityGate, PlanDiffGate, SecurityBaselineGate) suggest this validation may be part of a broader quality and security checking pipeline. + +## What we want to verify + +- When invoked with valid Terraform files in `tf_files`, the surface returns a success result +- When invoked with syntactically invalid Terraform files, the surface returns an error or validation failure +- The surface accepts the `tf_files` parameter as required input +- The surface can process multiple Terraform files in a single invocation +- Validation results distinguish between syntax errors and valid configurations +- The surface operates on files in a temporary directory context + +## Inventory references + +- Arguments: +- `tf_files` (required): +- Related gates: IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run, run_all +- Related ADRs: +- ADR 0001: pickled-diff package — Proposed +- ADR 0007: Code-aware stories and docstring drift detection — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_bdd_ambiguity.story.md b/dogfood/mining-output/stories/pickled_bdd_ambiguity.story.md new file mode 100644 index 0000000..4823b3f --- /dev/null +++ b/dogfood/mining-output/stories/pickled_bdd_ambiguity.story.md @@ -0,0 +1,75 @@ +# Story: ambiguity + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-bdd +- **Surface id:** pickled_bdd_ambiguity +- **Code depth:** callgraph | **Units read:** 5 | **Unresolved:** 2 + +## Context + +This CLI command provides a shortcut for running the ambiguity gate on a Gherkin feature file. It is invoked directly by users via the command line as an alternative to the more verbose `pickled-bdd check --gate ambiguity` syntax. The command parses the feature file, evaluates it for ambiguous scenarios using an LLM, and reports findings as JSON output with an exit code reflecting the gate verdict. + +## What the target does today + +**Inputs:** +- `feature_file` (str, required): Path to a Gherkin feature file to be analyzed. + +**Execution:** +1. Writes an informational message to stderr indicating equivalence to `pickled-bdd check --gate ambiguity`. +2. Builds an LLM client via environment-based configuration. If the `PICKLED_BDD_LLM_FACTORY` environment variable is set, uses that factory; otherwise uses a default client builder from `pickled_core.llm.bootstrap`. +3. Parses the feature file using an unresolved `PytestBddAdapter().parse_feature_file()` call. +4. If LLM client construction succeeds, runs an unresolved `AmbiguityGate(llm).run(feature)` call to evaluate the feature. +5. If LLM is unavailable (None), returns a PASS verdict with notes "LLM unavailable; ambiguity gate skipped" without performing analysis. +6. Outputs a JSON object to stdout containing: + - `"gate"`: the string `"ambiguity"` + - `"verdict"`: one of `"PASS"`, `"WARN"`, or `"FAIL"` (string value from the Verdict enum) + - `"notes"`: a string (may be empty or contain explanatory text) + - `"findings"`: an array of objects, each with: + - `"scenario"`: string name of the scenario + - `"alternatives"`: list of alternative interpretations + - `"suggested_fix"`: suggested resolution text + - Only findings that are instances of `AmbiguityFinding` are included. +7. Exits the process with status code determined by verdict: 0 for PASS, 1 for WARN, 2 for FAIL. + +**Error conditions:** +- If LLM client configuration fails (e.g., invalid environment configuration), raises `click.ClickException` with the original `ConfigError` message. +- If feature file parsing fails, behavior is determined by the unresolved `PytestBddAdapter().parse_feature_file()` call. +- If the ambiguity gate evaluation fails, behavior is determined by the unresolved `AmbiguityGate(llm).run()` call. + +**Side effects:** +- Writes informational message to stderr. +- Writes JSON output to stdout. +- Exits the process with a non-zero code for WARN or FAIL verdicts. + +## What we want to verify + +- Accepts a single required `feature_file` string argument. +- Writes the message "(equivalent to: pickled-bdd check --gate ambiguity)" to stderr before performing analysis. +- When LLM configuration is invalid, raises `click.ClickException` with the configuration error message. +- When LLM is unavailable (None), outputs JSON with verdict "PASS" and notes "LLM unavailable; ambiguity gate skipped". +- Outputs valid JSON to stdout with keys: "gate", "verdict", "notes", and "findings". +- The "gate" field in JSON output always has value "ambiguity". +- The "verdict" field contains one of the string values: "PASS", "WARN", or "FAIL". +- The "findings" array contains only objects with "scenario", "alternatives", and "suggested_fix" keys. +- Exits with code 0 when verdict is PASS. +- Exits with code 1 when verdict is WARN. +- Exits with code 2 when verdict is FAIL. +- JSON output is formatted with 2-space indentation and non-ASCII characters are preserved (ensure_ascii=False). + +## Inventory references + +- Arguments: +- `feature_file` (required): +- Related gates: AmbiguityGate.run, run_all +- Related ADRs: +- ADR 0005: `pickled-spec mine` staged mining pipeline — Accepted + +## Open questions + +- Docstring drift: The docstring describes this as an "alias" but the implementation is a standalone function that duplicates logic rather than directly calling the `check --gate ambiguity` code path. The function writes a message claiming equivalence but executes independent ambiguity gate logic via `run_ambiguity_gate()`. + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_bdd_ambiguitygate.story.md b/dogfood/mining-output/stories/pickled_bdd_ambiguitygate.story.md new file mode 100644 index 0000000..249bb6d --- /dev/null +++ b/dogfood/mining-output/stories/pickled_bdd_ambiguitygate.story.md @@ -0,0 +1,77 @@ +# Story: AmbiguityGate.run + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-bdd +- **Surface id:** pickled_bdd_ambiguitygate +- **Code depth:** callgraph | **Units read:** 4 | **Unresolved:** 5 + +## Context + +This surface is a quality gate that analyzes BDD feature scenarios for ambiguity. It is invoked by a gate-runner framework as part of a quality checking pipeline. Callers pass a Feature object containing scenarios, and receive a structured GateResult that indicates whether scenarios are ambiguous (admit multiple implementations). The gate uses an LLM to assess each scenario, parsing structured JSON responses to identify ambiguity findings. + +## What the target does today + +**Input constraints:** +- Accepts a `target` (any object) and an optional `context` dict; the context parameter is accepted but ignored. +- If `target` is not a Feature instance, returns immediately with verdict FAIL and a note describing the type mismatch (e.g., "Expected Feature, got "). + +**Processing flow:** +- Iterates over all scenarios in the Feature's `scenarios` attribute. +- For each scenario, formats it as a multi-line string with "Scenario: " followed by indented step lines. +- Renders a prompt using an unresolved template renderer (`self._template.render`), passing the formatted scenario text. +- Delegates to an unresolved collaborator (`complete_prompt`) to obtain an LLM response, instructing the LLM to return a single JSON object without markdown fences or extra commentary. +- Attempts to parse the response as JSON, applying best-effort extraction that strips leading/trailing whitespace and removes markdown code fences (detects triple-backtick blocks with optional language tags). +- If JSON parsing fails for a scenario, records the scenario name in a `parse_errors` list and skips further processing for that scenario. +- If the parsed JSON indicates ambiguity (`is_ambiguous` key is truthy), constructs an AmbiguityFinding with the scenario name, a tuple of alternative implementations (from the `alternatives` key, coerced to strings), and a suggested fix (from the `suggested_fix` key, coerced to string). + +**Verdict logic:** +- If all scenarios produce parse errors (and at least one scenario exists), returns WARN verdict with an empty findings tuple and a note listing parse-error scenarios. +- Otherwise, computes a verdict based on ambiguous finding counts: + - PASS if zero scenarios flagged ambiguous. + - FAIL if all scenarios flagged ambiguous (and at least one scenario exists). + - WARN if some but not all scenarios flagged ambiguous. +- If parse errors occurred and the verdict would otherwise be PASS, upgrades the verdict to WARN. + +**Return value:** +- Always returns a GateResult with: + - `gate_name`: the gate's name attribute. + - `verdict`: one of PASS, WARN, or FAIL. + - `findings`: a tuple of AmbiguityFinding objects (empty if all parses failed). + - `notes`: a string summarizing the count of ambiguous scenarios over total scenarios, and listing parse errors if any occurred. + +**Side effects:** +- None observable to the caller; LLM interaction is delegated to an unresolved collaborator. + +## What we want to verify + +- When target is not a Feature instance, verdict is FAIL and notes describe the actual type received. +- When target is a Feature with zero scenarios, verdict is PASS and notes indicate "0/0 scenarios flagged ambiguous." +- When all scenarios parse successfully and none are flagged ambiguous, verdict is PASS. +- When all scenarios parse successfully and all are flagged ambiguous, verdict is FAIL. +- When some but not all scenarios are flagged ambiguous, verdict is WARN. +- When all scenarios fail to parse (and at least one scenario exists), verdict is WARN, findings tuple is empty, and notes list parse-error scenario names. +- When some scenarios fail to parse and verdict would otherwise be PASS, verdict is upgraded to WARN and notes include parse-error scenario names. +- Each AmbiguityFinding contains the scenario name, a tuple of alternative implementation strings, and a suggested fix string. +- The notes field always includes a summary in the form "/ scenarios flagged ambiguous" (except when all parses fail). +- JSON response parsing tolerates markdown code fences (triple backticks with optional language tags) and strips them before JSON parsing. +- The context parameter is accepted but has no effect on the result. + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: AmbiguityGate.run +- Related ADRs: +- ADR 0005: `pickled-spec mine` staged mining pipeline — Accepted + +## Open questions + +- Docstring drift: The docstring states the gate "flags scenarios that admit multiple implementations" but does not describe the input type constraint (Feature only), the FAIL verdict for wrong input types, or the parse-error handling behavior (WARN verdict when all parses fail, parse-error notes appended, verdict upgrade to WARN). +- Docstring drift: The docstring does not mention that the gate uses an LLM to perform the ambiguity detection, nor that it relies on structured JSON responses. +- Docstring drift: The docstring does not describe the three-tier verdict logic (PASS/WARN/FAIL based on ambiguous-count thresholds) or the special case where parse errors upgrade a PASS to WARN. + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_bdd_check.story.md b/dogfood/mining-output/stories/pickled_bdd_check.story.md new file mode 100644 index 0000000..837773f --- /dev/null +++ b/dogfood/mining-output/stories/pickled_bdd_check.story.md @@ -0,0 +1,70 @@ +# Story: check + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-bdd +- **Surface id:** pickled_bdd_check +- **Code depth:** callgraph | **Units read:** 5 | **Unresolved:** 2 + +## Context + +This is a CLI command that runs quality gates on BDD feature files. The caller (a user at the command line or test automation) wants to validate a Gherkin feature file for ambiguity issues and receive a JSON report with an exit code indicating the verdict. The command is intended to be part of a CI/CD pipeline or pre-commit workflow where feature file quality must be enforced. + +## What the target does today + +Accepts two parameters: a `feature_file` string (path to a .feature file) and a `gate` string parameter. The `gate` parameter is currently unused; regardless of its value, only the ambiguity gate is executed. + +Builds an LLM client by delegating to an external factory. If the factory environment variable `PICKLED_BDD_LLM_FACTORY` is set, that factory is used; otherwise a default client is built. If client construction fails due to a configuration error, raises a `click.ClickException` with the error message. + +Parses the feature file using an unresolved `PytestBddAdapter().parse_feature_file()` call. The parsing behavior is not observable from this surface. + +If the LLM client is None (null), returns a gate result with verdict PASS, gate name "ambiguity", and a note explaining the LLM was unavailable and the gate was skipped. Otherwise, delegates gate execution to an unresolved `AmbiguityGate(llm).run(feature)` call. + +Transforms the gate result into JSON with the following structure: +- `gate`: the gate name string +- `verdict`: the verdict enum value as a string +- `notes`: notes string from the result +- `findings`: an array of objects, each containing `scenario` (target name), `alternatives` (list), and `suggested_fix`, filtered to only include findings that are instances of `AmbiguityFinding` + +Outputs the JSON to stdout via `click.echo` with 2-space indentation and Unicode preservation (ensure_ascii=False). + +Terminates the process with an exit code derived from the verdict: +- Verdict.PASS → exit code 0 +- Verdict.WARN → exit code 1 +- Verdict.FAIL → exit code 2 + +## What we want to verify + +- When invoked with a valid feature file path, the command parses the file and produces JSON output to stdout +- The JSON output contains exactly four top-level keys: "gate", "verdict", "notes", and "findings" +- The "gate" value in JSON output is "ambiguity" +- When the LLM client is unavailable (None), verdict is "PASS" and notes indicate the gate was skipped +- When the LLM client is unavailable, the findings array is empty +- The process exits with code 0 when verdict is PASS +- The process exits with code 1 when verdict is WARN +- The process exits with code 2 when verdict is FAIL +- When LLM client construction fails with ConfigError, a ClickException is raised with the original error message +- The `gate` parameter value has no effect on which gate runs; ambiguity gate always executes +- JSON output uses 2-space indentation +- JSON output preserves Unicode characters (non-ASCII characters are not escaped) +- The findings array only includes items that are instances of AmbiguityFinding +- Each finding object contains "scenario", "alternatives" (as a list), and "suggested_fix" keys + +## Inventory references + +- Arguments: +- `feature_file` (required): +- `gate` (optional): Which gate to run. +- Related gates: AmbiguityGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states "Run compensating gates against a .feature file" (plural "gates") but the code only runs the ambiguity gate, ignoring the `gate` parameter entirely +- Docstring drift: The docstring describes the surface as running "compensating gates" but provides no explanation of what "compensating" means or why this terminology is used; the code implements quality validation gates with no observable compensating behavior + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_bdd_draft.story.md b/dogfood/mining-output/stories/pickled_bdd_draft.story.md new file mode 100644 index 0000000..270c5c3 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_bdd_draft.story.md @@ -0,0 +1,63 @@ +# Story: draft + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-bdd +- **Surface id:** pickled_bdd_draft +- **Code depth:** callgraph | **Units read:** 3 | **Unresolved:** 1 + +## Context + +This CLI command is invoked by developers or automation to convert a user story written in Markdown into a Gherkin .feature file. It sits early in the pickled-bdd workflow, before ambiguity checking (AmbiguityGate.run). The caller provides a path to a Markdown file containing a user story and optionally specifies an output path; the command uses an LLM to generate the Gherkin feature text. + +## What the target does today + +Accepts a required `story_file` parameter—a file path string—and reads the file as UTF-8 text to obtain the user story content. + +Accepts an optional `output` parameter—a file path string or None. When `output` is None (the default), prints the drafted Gherkin feature text to stdout. When `output` is provided, writes the feature text to that path as UTF-8 and prints a confirmation message "Wrote {output}" to stderr. + +Builds an LLM client by delegating to a factory function that respects the environment variable `PICKLED_BDD_LLM_FACTORY` for test overrides. If the factory cannot build a client (raises ConfigError), converts the error to a ClickException with the same message text, which Click will display as a user-facing error. + +Drafts a Gherkin feature by passing the story text through an unresolved template rendering step (the rendered prompt content is not observable from this code) and sending the result to an LLM via `complete_prompt` with a system instruction "You output only Gherkin. No prose, no fences." The LLM's response is stripped of leading and trailing whitespace. + +Returns a DraftResult object containing: +- `text`: the stripped Gherkin feature text from the LLM +- `rationale`: the fixed string "LLM-drafted from user story; no post-processing applied." +- `warnings`: an empty tuple + +Does NOT validate the returned Gherkin syntax or semantics. Any malformed or ambiguous output from the LLM is passed through unchanged. + +Raises a ClickException if the LLM client cannot be constructed due to configuration issues. + +File I/O errors (e.g., story_file does not exist, output path is not writable) will propagate as unhandled exceptions (FileNotFoundError, PermissionError, etc.). + +## What we want to verify + +- When `output` is None, the drafted feature text appears on stdout and nothing is written to disk. +- When `output` is a valid path, the drafted feature text is written to that path as UTF-8 and a message "Wrote {output}" appears on stderr. +- The drafted feature text is the LLM response stripped of leading and trailing whitespace. +- A ConfigError from the LLM client factory is converted to a ClickException with the same error message. +- If `story_file` does not exist or is unreadable, a FileNotFoundError or similar I/O exception is raised. +- The DraftResult contains `rationale` equal to "LLM-drafted from user story; no post-processing applied." +- The DraftResult contains an empty `warnings` tuple. +- The command does not validate the Gherkin syntax of the LLM output. + +## Inventory references + +- Arguments: +- `story_file` (required): +- `output` (optional): Write the drafted feature to this path. Defaults to stdout. +- Related gates: AmbiguityGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states the command drafts "a .feature file from a user story (Markdown)" but does not mention that invalid Gherkin may be produced, nor that validation is explicitly deferred. The code shows validation is absent and delegated to a later gate (PR-08/AmbiguityGate). +- Docstring drift: The docstring does not describe the stdout vs. file-write behavior controlled by the `output` parameter, nor the stderr confirmation message when writing to a file. +- Docstring drift: The docstring does not mention the LLM client configuration or the possibility of a ClickException when the client cannot be built. + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_bdd_mcp.story.md b/dogfood/mining-output/stories/pickled_bdd_mcp.story.md new file mode 100644 index 0000000..3d4a3ef --- /dev/null +++ b/dogfood/mining-output/stories/pickled_bdd_mcp.story.md @@ -0,0 +1,50 @@ +# Story: mcp + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-bdd +- **Surface id:** pickled_bdd_mcp +- **Code depth:** callgraph | **Units read:** 1 | **Unresolved:** 0 + +## Context + +This is a CLI command entry point that serves as a parent group for MCP (Model Context Protocol) server-related subcommands. It is invoked directly by CLI users or by the Click framework when users run commands like `pickled-bdd mcp `. The surface provides organizational structure for MCP server operations within the pickled-bdd tool. + +## What the target does today + +The surface is a parameterless function that serves as a Click command group entry point. When invoked: + +- Accepts no arguments or parameters +- Returns None +- Produces no observable side effects on its own (no file I/O, no state mutation, no output) +- Acts as a namespace/container for subcommands in the CLI hierarchy +- Expects to be decorated with Click's `@group` or similar decorator to provide actual CLI functionality (the function body is empty, suggesting decorator-driven behavior) + +The surface does not perform validation, does not raise exceptions, and does not interact with any external systems. Its behavioral contract is essentially a no-op that delegates all actual functionality to the CLI framework's decorator system and any registered subcommands. + +## What we want to verify + +- Calling `mcp()` directly completes without raising exceptions +- Calling `mcp()` returns None +- Calling `mcp()` produces no console output +- Calling `mcp()` does not modify any global state +- Calling `mcp()` does not perform file system operations +- Calling `mcp()` does not make network calls +- The function signature accepts zero parameters + +## Inventory references + +- Arguments: +- (none) +- Related gates: AmbiguityGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring claims "MCP server commands" (plural, suggesting multiple operations), but the function body is empty and performs no operations itself; the actual commands would be implemented as subcommands registered to this group, which is not reflected in the docstring + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_bdd_run_all.story.md b/dogfood/mining-output/stories/pickled_bdd_run_all.story.md new file mode 100644 index 0000000..7535ad0 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_bdd_run_all.story.md @@ -0,0 +1,86 @@ +# Story: run_all + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-bdd +- **Surface id:** pickled_bdd_run_all +- **Code depth:** callgraph | **Units read:** 3 | **Unresolved:** 1 + +## Context + +This is a batch validation entry point for BDD/Gherkin feature files. A caller (likely a quality-gate orchestrator or CI/CD pipeline) invokes this to check whether all `.feature` files under a project's `features/` directory can be successfully parsed. It returns a structured list of pass/fail results—one per feature file plus a final ambiguity-check result—without requiring a live LLM connection. + +## What the target does today + +**Inputs:** +- `workdir`: a filesystem path (string or `Path` object) pointing to a project root directory. + +**Returns:** +- A list of `GateResult` objects. + +**Directory and file discovery:** +- Resolves `workdir` to an absolute path. +- Searches recursively for files matching `features/**/*.feature` (any `.feature` file under a `features/` subdirectory, at any depth). +- If no matching files are found, returns a single-element list: a `PASS` result with `gate_name="bdd.features"` and notes `"no features/ directory"`. + +**Parsing each feature file:** +- For each discovered `.feature` file (processed in sorted order): + - Attempts to parse it by reading the file's UTF-8 text and parsing the Gherkin syntax. + - **On success:** appends a `GateResult` with: + - `gate_name` set to `"bdd.parse."` (where `` is the base name of the feature file). + - `verdict=Verdict.PASS`. + - `notes` describing the relative path from `workdir` (exact format depends on the unresolved `path.relative_to` method). + - **On any exception during parsing:** appends a `GateResult` with: + - `gate_name` set to `"bdd.parse."`. + - `verdict=Verdict.FAIL`. + - `notes` containing the exception's string representation. + +**Parsing constraints:** +- Empty or whitespace-only files raise `ValueError("Gherkin text is empty")`. +- Files without a `Feature:` declaration raise `ValueError("No Feature found in Gherkin text")`. + +**Ambiguity gate:** +- After processing all feature files, unconditionally appends a `GateResult` with: + - `gate_name="bdd.ambiguity"`. + - `verdict=Verdict.PASS`. + - `notes="skipped — set PICKLED_BDD_LLM_FACTORY to enable AmbiguityGate"`. +- This result is **always** a pass; no actual ambiguity analysis is performed. + +**Side effects:** +- Reads files from disk (each `.feature` file is read once). +- No writes or state modifications. + +**Error propagation:** +- Does **not** raise exceptions for malformed feature files; instead records failures as `FAIL` results in the returned list. +- May raise exceptions if `workdir` resolution or file-system globbing fails (e.g., permission errors, invalid path). + +## What we want to verify + +- When `workdir` contains no `features/` directory or no `.feature` files, returns a single-element list with `gate_name="bdd.features"`, `verdict=PASS`, and notes indicating no features directory. +- When `workdir` contains at least one `.feature` file, the returned list has one result per feature file plus one ambiguity result. +- For a valid, parseable `.feature` file, the corresponding result has `verdict=PASS` and `gate_name="bdd.parse."`. +- For a `.feature` file that is empty or whitespace-only, the corresponding result has `verdict=FAIL` and notes containing `"Gherkin text is empty"`. +- For a `.feature` file with content but no `Feature:` block, the corresponding result has `verdict=FAIL` and notes containing `"No Feature found in Gherkin text"`. +- For any `.feature` file that raises an exception during parsing, the corresponding result has `verdict=FAIL` and notes containing the exception message. +- The returned list always ends with a result having `gate_name="bdd.ambiguity"`, `verdict=PASS`, and notes indicating the ambiguity check is skipped. +- Feature files are processed in sorted order (by path). +- The function does not raise exceptions for individual malformed feature files; all parsing errors are captured in `FAIL` results. + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states "AmbiguityGate skipped without LLM" but does not mention that the function **always** returns a passing ambiguity result, regardless of whether feature files exist or are valid. The code unconditionally appends a `PASS` verdict for `bdd.ambiguity`. +- Docstring drift: The docstring does not describe the return type (`list[GateResult]`), the per-file result structure, or the special case when no features are found. +- Docstring drift: The docstring does not mention that parsing errors are caught and returned as `FAIL` results rather than propagated as exceptions. + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_core_check_all.story.md b/dogfood/mining-output/stories/pickled_core_check_all.story.md new file mode 100644 index 0000000..b35f611 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_core_check_all.story.md @@ -0,0 +1,49 @@ +# Story: check-all + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-core +- **Surface id:** pickled_core_check_all + +## Context + +Developers and CI pipelines use `check-all` as a single entry point to run all available workspace validation gates across every pickled-* package. This command provides comprehensive validation of a workspace directory structure (features/, specs/, infra/, migrations/) by executing all registered gates in one pass, supporting both strict and lenient exit-code behaviors depending on whether warnings should fail the build. + +## What the target does today + +Run workspace gates from every pickled-* package against a directory. + +The command accepts an optional `workdir` parameter to specify the workspace root containing features/, specs/, infra/, and migrations/ directories. If not provided, the current directory is assumed. + +The command accepts an optional `warn_ok` parameter that modifies exit behavior: when enabled, the process exits with code 0 (success) if only WARN-level verdicts occur, such as when optional tooling (terraform, LLM providers) is not configured. Without this flag, WARN verdicts cause non-zero exit codes. + +The command discovers and executes all gates registered across all installed pickled-* packages, collecting and reporting their verdicts. + +## What we want to verify + +- When invoked without arguments, check-all runs against the current directory as workspace root +- When invoked with `workdir` argument, check-all runs against the specified directory +- When `warn_ok` is false or omitted, WARN verdicts cause non-zero exit code +- When `warn_ok` is true, WARN verdicts alone result in exit code 0 +- The command discovers and executes gates from all installed pickled-* packages, not just pickled-core +- The command expects workspace structure containing features/, specs/, infra/, and/or migrations/ subdirectories +- Exit code 0 indicates all gates passed (or only warnings with warn_ok=true) +- Non-zero exit code indicates at least one gate failed or warned (when warn_ok=false) + +## Inventory references + +- Arguments: +- `workdir` (optional): Workspace root (features/, specs/, infra/, migrations/). +- `warn_ok` (optional): Exit 0 when only WARN verdicts occur (e.g. terraform or LLM not configured). +- Related gates: (none) +- Related ADRs: +- ADR 0004: Multi-ruleset workspace configuration (general) — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_core_mine.story.md b/dogfood/mining-output/stories/pickled_core_mine.story.md new file mode 100644 index 0000000..1d4ef00 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_core_mine.story.md @@ -0,0 +1,63 @@ +# Story: mine + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-core +- **Surface id:** pickled_core_mine +- **Code depth:** callgraph | **Units read:** 1 | **Unresolved:** 0 + +## Context + +CLI command invoked by users or build pipelines to analyze a Python project and extract pickled-core metadata surfaces (surfaces, stories, features, gate results). Used as the primary entrypoint for static analysis and documentation generation workflows in projects that use pickled-core tooling. + +## What the target does today + +The `mine` function is a zero-parameter CLI command that performs no observable operations in its current implementation. + +**Invocation:** +- Accepts no arguments or parameters +- No configuration, input files, or environment variables are read +- No validation is performed on the calling context + +**Return value:** +- Returns `None` (implicit Python return) + +**Side effects:** +- None observable; the function body is empty (contains only a docstring and implicit return) +- Does not read from or write to the filesystem +- Does not produce console output +- Does not modify global state +- Does not raise exceptions + +**Error handling:** +- No error conditions are checked or raised +- No validation of project structure, Python environment, or required dependencies + +## What we want to verify + +- When invoked, the function completes without raising exceptions +- The function returns None +- No files are created or modified in the filesystem after invocation +- No output is written to stdout or stderr +- Invocation completes immediately (no blocking I/O or long-running operations) +- Multiple sequential invocations produce identical (no-op) behavior +- The function can be called without any project structure or pickled-core configuration present + +## Inventory references + +- Arguments: +- (none) +- Related gates: (none) +- Related ADRs: +- ADR 0004: Multi-ruleset workspace configuration (general) — Accepted + +## Open questions + +- Docstring drift: The docstring claims the function "Mine[s] a Python project for surfaces, stories, features, and gate results" but the implementation is empty and performs none of these mining operations +- Docstring drift: The docstring implies the function will analyze a Python project, but no project path is accepted as a parameter and no project files are accessed +- Docstring drift: The docstring suggests output (mined surfaces, stories, features, gate results) will be produced, but the function produces no output or return value beyond None + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_core_mine_all.story.md b/dogfood/mining-output/stories/pickled_core_mine_all.story.md new file mode 100644 index 0000000..776c1b2 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_core_mine_all.story.md @@ -0,0 +1,65 @@ +# Story: mine all + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-core +- **Surface id:** pickled_core_mine_all + +## Context + +The "mine all" CLI command is the top-level orchestrator for the pickled-core mining pipeline. It is invoked by users (developers, QA engineers, or automated CI/CD processes) who want to extract software surfaces from a target codebase, generate human-readable stories and Gherkin features, apply tagging rules, evaluate quality, and produce a final report—all in a single command. This is the primary entry point for a complete end-to-end mining workflow, as opposed to running individual pipeline stages separately. + +## What the target does today + +Run inventory → code → stories → features → tag → evaluate → report. + +The command accepts a required `target` parameter (presumably a path or identifier for the codebase to mine) and an optional `output_dir` for where mining artifacts are written. It operates in `quick` mode by default (non-interactive) but can prompt interactively if disabled. Verbosity, surface filtering by package or surface-id substring, and parallelism control for LLM calls during story and feature generation are configurable. Advanced code-collection parameters include `depth`, `callee_scope`, `max_hops`, `max_callees`, and `max_code_lines` to control how much source context is gathered per surface. Optional flags control MCP behavior (`no_mcp`, `mcp_timeout`), ruleset location (`ruleset_dir`, `ruleset_config`), story/feature overwrite policy, and cycle detection in the call graph (`detect_cycles`). The command chains the seven named stages in sequence, passing intermediate artifacts between them. + +## What we want to verify + +- Invoking "mine all" with a valid `target` runs all seven stages (inventory, code, stories, features, tag, evaluate, report) in the documented order. +- Omitting `target` raises an error indicating the parameter is required. +- Specifying `--output_dir ` writes all mining artifacts to ``. +- Passing `--verbose` increases logging detail to stderr. +- Using `--surfaces ` restricts processing to surfaces whose package name or surface-id contains the specified substring. +- Setting `--max_parallel ` limits concurrent LLM calls during story and feature generation to N. +- Providing `--ruleset_dir ` and/or `--ruleset_config ` configures which tagging rulesets are applied. +- Flags `--overwrite_stories` and `--overwrite_features` control whether existing story or feature files are replaced on re-run. +- Code-collection parameters (`--depth`, `--callee_scope`, `--max_hops`, `--max_callees`, `--max_code_lines`) affect the volume and scope of source context extracted per surface. +- Setting `--detect_cycles` generates a `code-context/_cycles.json` file when call-graph cycles are detected. +- The command exits with a non-zero status if any stage fails. +- Running in non-quick mode (when `--quick` is false or omitted and defaults permit) prompts the user for confirmation at interactive decision points. + +## Inventory references + +- Arguments: +- `target` (required): +- `output_dir` (optional): Mining output directory. +- `quick` (optional): Quick mode (default) or interactive prompts. +- `verbose` (optional): Extra logging to stderr. +- `surfaces` (optional): Comma-separated filter on package name or surface-id substring. +- `max_parallel` (optional): Max parallel LLM calls in quick mode (stories, features). +- `no_mcp` (optional): +- `mcp_timeout` (optional): +- `ruleset_dir` (optional): +- `ruleset_config` (optional): +- `overwrite_stories` (optional): +- `overwrite_features` (optional): +- `depth` (optional): How much source to collect per surface. +- `callee_scope` (optional): Which intra-project callees to follow. +- `max_hops` (optional): Callee expansion depth (callgraph only). +- `max_callees` (optional): Hard cap on collected callee units per surface. +- `max_code_lines` (optional): Hard cap on total source lines per surface. +- `detect_cycles` (optional): Write code-context/_cycles.json from observed edges. +- Related gates: (none) +- Related ADRs: +- ADR 0004: Multi-ruleset workspace configuration (general) — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_core_mine_code.story.md b/dogfood/mining-output/stories/pickled_core_mine_code.story.md new file mode 100644 index 0000000..cca996f --- /dev/null +++ b/dogfood/mining-output/stories/pickled_core_mine_code.story.md @@ -0,0 +1,55 @@ +# Story: mine code + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-core +- **Surface id:** pickled_core_mine_code + +## Context + +The `mine code` CLI command is used by developers and automation systems to extract source code context from software projects for documentation, analysis, or AI-assisted development workflows. It operates on a target codebase to collect surface definitions (functions, classes, CLI commands) along with their implementation details and call graphs. Users invoke this command to build an inventory of code surfaces with configurable depth and scope, which can then be consumed by downstream tools or LLM-based verification systems. + +## What the target does today + +The inventory provides no docstring for this surface; behavior must be confirmed from source before specifying. + +## What we want to verify + +- **Required target argument**: Command fails gracefully when invoked without the `target` argument. +- **Output directory handling**: When `output_dir` is specified, mining results are written to that directory; when omitted, a default output location is used. +- **Verbose logging**: When `verbose` flag is enabled, additional diagnostic information is written to stderr during mining operations. +- **Surface filtering**: When `surfaces` is provided with package name or surface-id substrings, only matching surfaces are included in the output. +- **Depth control**: The `depth` parameter controls how much source code context is collected for each surface (e.g., imports, dependencies, implementation). +- **Callee scope**: The `callee_scope` parameter determines which intra-project function/method calls are followed during analysis. +- **Hop limiting**: The `max_hops` parameter caps the depth of callee expansion in the call graph traversal. +- **Callee count limiting**: The `max_callees` parameter enforces a hard limit on the number of callee units collected per surface. +- **Line count limiting**: The `max_code_lines` parameter enforces a hard limit on total source lines collected per surface. +- **Cycle detection**: When `detect_cycles` is enabled, a `code-context/_cycles.json` file is written containing detected circular dependencies from call graph edges. +- **Exit code**: Command exits with 0 on successful mining, non-zero on errors. +- **File output**: Mining produces structured output files (likely JSON) containing surface metadata, source code, and call graph information. + +## Inventory references + +- Arguments: +- `target` (required): +- `output_dir` (optional): Mining output directory. +- `verbose` (optional): Extra logging to stderr. +- `surfaces` (optional): Comma-separated filter on package name or surface-id substring. +- `depth` (optional): How much source to collect per surface. +- `callee_scope` (optional): Which intra-project callees to follow. +- `max_hops` (optional): Callee expansion depth (callgraph only). +- `max_callees` (optional): Hard cap on collected callee units per surface. +- `max_code_lines` (optional): Hard cap on total source lines per surface. +- `detect_cycles` (optional): Write code-context/_cycles.json from observed edges. +- Related gates: (none) +- Related ADRs: +- ADR 0004: Multi-ruleset workspace configuration (general) — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_core_mine_evaluate.story.md b/dogfood/mining-output/stories/pickled_core_mine_evaluate.story.md new file mode 100644 index 0000000..23b3cff --- /dev/null +++ b/dogfood/mining-output/stories/pickled_core_mine_evaluate.story.md @@ -0,0 +1,51 @@ +# Story: mine evaluate + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-core +- **Surface id:** pickled_core_mine_evaluate + +## Context + +Engineers and CI pipelines use `mine evaluate` after earlier mining stages have run. It is stage 6 of the pickled-core mining pipeline (per ADR 0005), invoked to check whether mined surfaces meet coverage and ambiguity quality gates before finalizing the mining output. The command is typically run against a target codebase after surfaces have been extracted, and its pass/fail result determines whether the mined inventory is acceptable for downstream use. + +## What the target does today + +Stage 6: evaluate coverage and ambiguity gates. + +The command accepts a required `target` argument (the codebase being mined), an optional `output_dir` for mining artifacts, optional `verbose` flag for stderr logging, an optional `surfaces` filter to restrict evaluation to specific packages or surface IDs, and optional `ruleset_dir` and `ruleset_config` parameters to configure gate behavior. It evaluates the mined surfaces against configured coverage and ambiguity gates and reports gate pass/fail status. + +## What we want to verify + +- Invoking `mine evaluate ` without prior mining stages fails or reports missing prerequisite data. +- When all configured gates pass, the command exits with status 0. +- When any gate fails, the command exits with a non-zero status. +- The `--surfaces` filter restricts gate evaluation to only matching package names or surface IDs. +- The `--verbose` flag produces additional diagnostic output to stderr during evaluation. +- The `--output_dir` argument changes where the command reads mined artifacts and writes evaluation results. +- Coverage gate evaluation measures the percentage or count of surfaces meeting documentation or verification thresholds. +- Ambiguity gate evaluation detects conflicting or overlapping surface definitions. +- Gate results are written to the output directory in a structured format (JSON or similar). +- Running `mine evaluate` multiple times with the same inputs produces the same gate results (idempotent). + +## Inventory references + +- Arguments: +- `target` (required): +- `output_dir` (optional): Mining output directory. +- `verbose` (optional): Extra logging to stderr. +- `surfaces` (optional): Comma-separated filter on package name or surface-id substring. +- `ruleset_dir` (optional): +- `ruleset_config` (optional): +- Related gates: (none) +- Related ADRs: +- ADR 0005: `pickled-spec mine` staged mining pipeline — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_core_mine_features.story.md b/dogfood/mining-output/stories/pickled_core_mine_features.story.md new file mode 100644 index 0000000..020694a --- /dev/null +++ b/dogfood/mining-output/stories/pickled_core_mine_features.story.md @@ -0,0 +1,49 @@ +# Story: mine features + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-core +- **Surface id:** pickled_core_mine_features + +## Context + +Developers working with pickled-core use this CLI command to draft features from user stories during stage 4 of the mined software verification pipeline. This follows ADR 0005's staged mining pipeline approach, where stories have already been generated in a previous stage and now need to be transformed into testable feature specifications. + +## What the target does today + +The inventory provides no docstring for this surface; behavior must be confirmed from source before specifying. + +## What we want to verify + +- Accepts a required `target` argument specifying what to mine +- Accepts an optional `output_dir` argument to specify where mining output should be written +- Accepts an optional `quick` flag that controls whether the command runs in quick mode (default) or uses interactive prompts +- Accepts an optional `verbose` flag to enable extra logging output to stderr +- Accepts an optional `surfaces` argument as a comma-separated filter on package name or surface-id substring +- Accepts an optional `max_parallel` argument to control maximum parallel LLM calls in quick mode for stories and features processing +- Accepts an optional `overwrite_features` flag +- Operates as part of the staged mining pipeline described in ADR 0005 +- Runs as stage 4 of the mining process, specifically drafting features from previously generated stories + +## Inventory references + +- Arguments: +- `target` (required): +- `output_dir` (optional): Mining output directory. +- `quick` (optional): Quick mode (default) or interactive prompts. +- `verbose` (optional): Extra logging to stderr. +- `surfaces` (optional): Comma-separated filter on package name or surface-id substring. +- `max_parallel` (optional): Max parallel LLM calls in quick mode (stories, features). +- `overwrite_features` (optional): +- Related gates: (none) +- Related ADRs: +- ADR 0005: `pickled-spec mine` staged mining pipeline — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_core_mine_inventory.story.md b/dogfood/mining-output/stories/pickled_core_mine_inventory.story.md new file mode 100644 index 0000000..24b14c2 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_core_mine_inventory.story.md @@ -0,0 +1,50 @@ +# Story: mine inventory + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-core +- **Surface id:** pickled_core_mine_inventory + +## Context + +The `mine inventory` command is invoked by developers, test engineers, or CI pipelines in the first stage of the pickled-spec mining pipeline (ADR 0005). It introspects a target project and produces an inventory.json file that catalogs discovered specifications, features, or test scenarios before subsequent mining stages process them further. + +## What the target does today + +Stage 1: introspect target and write inventory.json. + +The command accepts a required `target` argument and optional parameters controlling output location (`output_dir`), verbosity (`verbose`), interaction mode (`quick`), MCP tool discovery (`no_mcp`, `mcp_timeout`). + +## What we want to verify + +- When invoked with a valid `target`, the command completes without error. +- An `inventory.json` file is created in the specified `output_dir` (or default location if not provided). +- The `inventory.json` file contains valid JSON and represents introspected elements from the target. +- Passing `--verbose` produces additional logging output to stderr. +- Passing `--no_mcp` skips any MCP tools/list operations. +- Passing `--quick` runs without interactive prompts (default behavior). +- The command fails with a clear error message if `target` is missing or invalid. + +## Inventory references + +- Arguments: +- `target` (required): +- `output_dir` (optional): Mining output directory. +- `quick` (optional): Quick mode (default) or interactive prompts. +- `verbose` (optional): Extra logging to stderr. +- `no_mcp` (optional): Skip MCP tools/list. +- `mcp_timeout` (optional): +- Related gates: (none) +- Related ADRs: +- ADR 0005: `pickled-spec mine` staged mining pipeline — Accepted +- ADR 0006: `pickled-spec mine code` static code reading — Accepted +- ADR 0007: Code-aware stories and docstring drift detection — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_core_mine_report.story.md b/dogfood/mining-output/stories/pickled_core_mine_report.story.md new file mode 100644 index 0000000..f311161 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_core_mine_report.story.md @@ -0,0 +1,48 @@ +# Story: mine report + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-core +- **Surface id:** pickled_core_mine_report + +## Context + +Pipeline operators and CI/CD automation use this command to generate a final mining report (mining-report.md) after the pickled pipeline has completed its analysis stages. This is the last stage (stage 7) that consolidates all previous pipeline outputs into a human-readable markdown document for review, handoff, or archival. + +## What the target does today + +Stage 7: render mining-report.md from pipeline outputs. + +The command takes a required `target` argument and produces a markdown report file from the artifacts generated by earlier pipeline stages. The `output_dir` parameter specifies where mining outputs are located. The `quick` flag controls whether the command runs in default quick mode or prompts for interactive input. The `verbose` flag enables additional logging to stderr. The `surfaces` parameter filters which package names or surface-id substrings are included in the report. + +## What we want to verify + +- `pickled-core mine report ` exits successfully when valid target and pipeline outputs exist +- `pickled-core mine report --output_dir ` reads mining artifacts from the specified directory +- `pickled-core mine report ` produces a file named mining-report.md +- `pickled-core mine report --verbose` emits additional logging messages to stderr +- `pickled-core mine report --surfaces ` includes only surfaces matching the comma-separated filter criteria in the generated report +- `pickled-core mine report --quick=false` triggers interactive prompts (if applicable) +- `pickled-core mine report` without target argument fails with appropriate error message +- Generated mining-report.md contains consolidated information from prior pipeline stages + +## Inventory references + +- Arguments: +- `target` (required): +- `output_dir` (optional): Mining output directory. +- `quick` (optional): Quick mode (default) or interactive prompts. +- `verbose` (optional): Extra logging to stderr. +- `surfaces` (optional): Comma-separated filter on package name or surface-id substring. +- Related gates: (none) +- Related ADRs: +- ADR 0004: Multi-ruleset workspace configuration (general) — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_core_mine_stories.story.md b/dogfood/mining-output/stories/pickled_core_mine_stories.story.md new file mode 100644 index 0000000..8d8d9d1 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_core_mine_stories.story.md @@ -0,0 +1,52 @@ +# Story: mine stories + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-core +- **Surface id:** pickled_core_mine_stories + +## Context + +Developers and CI pipelines use `mine stories` during the third stage of a three-stage mining workflow. After `mine inventory` has collected surface metadata and `mine code-context` has extracted implementation details, this command generates human-readable story files that document the behavior of each mined surface. The stories serve as both documentation and the foundation for feature-file generation in stage 4. + +## What the target does today + +Stage 3: emit stories from inventory.json. + +The command reads the inventory file produced by stage 1 and generates story documentation for each surface. It accepts filtering by package name or surface-id substring via the `surfaces` parameter. The `quick` flag controls whether the command runs in batch mode (default) or prompts interactively. Parallel LLM calls can be limited with `max_parallel` for stories and features generation. The `overwrite_stories` flag controls whether existing story files should be replaced. If a `code_context_dir` is provided or defaults to `/code-context`, the command can incorporate implementation details into the generated stories (per ADR 0007). The `verbose` flag enables additional diagnostic output to stderr. + +## What we want to verify + +- Reads inventory.json from the target directory +- Generates story files in the output directory structure +- Filters surfaces when `surfaces` parameter contains package name or surface-id substring +- Defaults to quick/batch mode unless `quick` is explicitly disabled +- Respects `max_parallel` limit for concurrent LLM operations +- Honors `overwrite_stories` flag for existing story file handling +- Looks for code-context files in `code_context_dir` if specified, otherwise defaults to `/code-context` +- Emits extra logging to stderr when `verbose` is enabled +- Integrates code-context into stories when available, supporting drift detection per ADR 0007 + +## Inventory references + +- Arguments: +- `target` (required): +- `output_dir` (optional): Mining output directory. +- `quick` (optional): Quick mode (default) or interactive prompts. +- `verbose` (optional): Extra logging to stderr. +- `surfaces` (optional): Comma-separated filter on package name or surface-id substring. +- `max_parallel` (optional): Max parallel LLM calls in quick mode (stories, features). +- `overwrite_stories` (optional): +- `code_context_dir` (optional): Directory with code-context/*.md (default: /code-context when present). +- Related gates: (none) +- Related ADRs: +- ADR 0007: Code-aware stories and docstring drift detection — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_core_mine_tag.story.md b/dogfood/mining-output/stories/pickled_core_mine_tag.story.md new file mode 100644 index 0000000..26a55ba --- /dev/null +++ b/dogfood/mining-output/stories/pickled_core_mine_tag.story.md @@ -0,0 +1,51 @@ +# Story: mine tag + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-core +- **Surface id:** pickled_core_mine_tag + +## Context + +Developers and automation pipelines use this command during the fifth stage of the pickled-core workflow to tag scenarios within generated feature files. This follows earlier mining stages and prepares features for downstream processing. The command is part of a multi-stage feature generation pipeline where tagging scenarios helps organize, categorize, or filter test cases. + +## What the target does today + +Stage 5: tag scenarios in generated features. + +The command processes previously generated feature files and applies tags to scenarios. It operates on a specified target and can filter which surfaces are processed. It supports both quick (non-interactive) and interactive modes for tagging operations. Output is written to a configured mining directory, and verbosity can be increased for debugging. Multiple rulesets can be configured via ruleset directory and config parameters, consistent with ADR 0004's multi-ruleset workspace configuration support. + +## What we want to verify + +- `mine tag` with a valid target processes feature files and tags scenarios within them +- The `--output-dir` parameter, when provided, determines where tagged features are written +- The `--quick` flag (default true) runs without interactive prompts; when false, enables interactive mode +- The `--verbose` flag increases logging output to stderr when enabled +- The `--surfaces` parameter filters processing to only surfaces matching the comma-separated package name or surface-id substrings +- The `--ruleset-dir` parameter specifies the directory containing ruleset definitions +- The `--ruleset-config` parameter specifies ruleset configuration consistent with ADR 0004 +- The command fails with an appropriate error when the target parameter is missing +- The command operates as stage 5 in a sequential mining pipeline, expecting prior stages to have generated features + +## Inventory references + +- Arguments: +- `target` (required): +- `output_dir` (optional): Mining output directory. +- `quick` (optional): Quick mode (default) or interactive prompts. +- `verbose` (optional): Extra logging to stderr. +- `surfaces` (optional): Comma-separated filter on package name or surface-id substring. +- `ruleset_dir` (optional): +- `ruleset_config` (optional): +- Related gates: (none) +- Related ADRs: +- ADR 0004: Multi-ruleset workspace configuration (general) — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_data_apply.story.md b/dogfood/mining-output/stories/pickled_data_apply.story.md new file mode 100644 index 0000000..af5682f --- /dev/null +++ b/dogfood/mining-output/stories/pickled_data_apply.story.md @@ -0,0 +1,91 @@ +# Story: apply + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-data +- **Surface id:** pickled_data_apply +- **Code depth:** callgraph | **Units read:** 6 | **Unresolved:** 6 + +## Context + +This CLI command is used by developers or automated tools to validate SQL migration files before applying them to a production database. It provides a safe sandbox environment (in-memory SQLite) to test whether a migration's DDL/DML statements will execute successfully and to preview the resulting database schema. The command accepts migration files written in various SQL dialects (e.g., PostgreSQL) and transpiles them to SQLite for validation. Related gates like `MigrationDriftGate.run` and `DataContractGate.run` likely consume this functionality to verify migrations as part of a validation pipeline. + +## What the target does today + +The command accepts two arguments: a file path to a migration script and a SQL dialect identifier (e.g., "postgres"). + +**Pre-execution validation:** +- Rejects files with a `.dbt` suffix, raising `NotImplementedError` with a message about dbt not being implemented +- Reads the migration file as UTF-8 text +- Parses the SQL using the specified dialect; raises `SQLParseError` if parsing fails or produces empty results +- Scans the parsed AST for `ATTACH` or `DETACH` statements (both top-level and nested); raises `UnsafeMigrationStatementError` if any are found, preventing filesystem access attempts + +**Execution:** +- Transpiles all parsed SQL statements from the source dialect to SQLite dialect (delegated to unresolved `sqlglot.parse` and `stmt.sql` calls) +- Creates an in-memory SQLite database connection +- Attempts to set the SQLite attached database limit to 0 as defense-in-depth (catches `AttributeError` for Python <3.11 compatibility but does not fail) +- Executes each transpiled statement sequentially (delegated to unresolved `conn.execute`) +- Commits the transaction (delegated to unresolved `conn.commit`) +- Closes the connection regardless of success or failure + +**Output:** +- On success, introspects the resulting schema and prints a JSON object to stdout with this structure: + - `tables`: array of table objects, excluding SQLite system tables (names starting with `sqlite_`) + - Each table object contains: + - `name`: table name (string) + - `columns`: array of column objects + - Each column object contains: + - `name`: column name (string) + - `type`: column type uppercased (string), defaults to "TEXT" if null + - `nullable`: boolean indicating whether the column accepts nulls +- JSON is formatted with 2-space indentation + +**Error modes:** +- Raises `NotImplementedError` if the migration file has a `.dbt` extension +- Raises `SQLParseError` if SQL cannot be parsed or parsing produces no result +- Raises `UnsafeMigrationStatementError` if `ATTACH` or `DETACH` statements are detected +- Any errors during statement execution propagate from the unresolved `conn.execute` call +- File reading errors (missing file, encoding issues) propagate naturally + +The command provides no return value (returns `None`) as it's a CLI endpoint that produces side effects (stdout output). + +## What we want to verify + +- Accepts a valid migration file path and dialect string, produces JSON output to stdout +- Rejects `.dbt` files with `NotImplementedError` before any SQL processing +- Rejects SQL containing top-level `ATTACH` statements with `UnsafeMigrationStatementError` +- Rejects SQL containing nested `ATTACH` or `DETACH` statements with `UnsafeMigrationStatementError` +- Raises `SQLParseError` when SQL cannot be parsed in the specified dialect +- Raises `SQLParseError` when SQL file is empty or produces no parse result +- Outputs JSON with `tables` array containing objects with `name` and `columns` fields +- Each column object in output contains `name`, `type`, and `nullable` fields +- Column types in output are uppercased +- Column types default to "TEXT" when database reports null type +- Output excludes SQLite system tables (names starting with `sqlite_`) +- JSON output is indented with 2 spaces +- Closes database connection even when statement execution fails +- Handles missing UTF-8 encoding in migration file by raising appropriate error +- Transpiles SQL from specified source dialect to SQLite dialect before execution +- Creates an in-memory SQLite database (not a file-based one) +- Attempts to set SQLite attached database limit to 0 (silently continues on AttributeError) +- Commits transaction after all statements execute successfully + +## Inventory references + +- Arguments: +- `migration` (required): +- `dialect` (optional): +- Related gates: DataContractGate.run, MigrationDriftGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: Docstring claims "Apply migration to in-memory SQLite and print resulting schema" but omits several important behaviors: rejection of `.dbt` files, rejection of `ATTACH`/`DETACH` statements, SQL parsing and validation before execution, error conditions, and the specific JSON schema structure returned +- Docstring drift: Docstring does not mention the `dialect` parameter's purpose (transpilation from source dialect to SQLite) +- Docstring drift: Docstring does not indicate that the command performs security validation to prevent filesystem access attempts + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_data_check_drift.story.md b/dogfood/mining-output/stories/pickled_data_check_drift.story.md new file mode 100644 index 0000000..f05c430 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_data_check_drift.story.md @@ -0,0 +1,48 @@ +# Story: check-drift + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-data +- **Surface id:** pickled_data_check_drift + +## Context + +This surface is used by developers and CI/CD pipelines to validate that a migration script produces a database schema that matches an expected schema defined in a YAML file. It runs the MigrationDriftGate to detect discrepancies between what the migration creates and what was documented as expected. This is particularly relevant in the context of ADR 0007, which establishes code-aware stories and docstring drift detection as a practice. + +## What the target does today + +The command runs MigrationDriftGate against an expected schema YAML file. It accepts: +- A required `migration` parameter (the migration to verify) +- A required `expected` parameter (the expected schema YAML to compare against) +- An optional `dialect` parameter (presumably to specify database dialect) + +The gate checks whether the migration, when executed, produces a schema that matches the expected schema definition. This helps detect drift between documented schema expectations and actual migration behavior. + +## What we want to verify + +- Accepts a required `migration` argument that specifies which migration to validate +- Accepts a required `expected` argument that points to a schema YAML file +- Accepts an optional `dialect` argument for database dialect specification +- Executes MigrationDriftGate.run with the provided parameters +- Reports drift or validation results from comparing migration output to expected schema +- Exits with appropriate status code indicating success or drift detection +- Can be invoked from command line as part of the pickled-data CLI + +## Inventory references + +- Arguments: +- `migration` (required): +- `expected` (required): +- `dialect` (optional): +- Related gates: DataContractGate.run, MigrationDriftGate.run, run_all +- Related ADRs: +- ADR 0007: Code-aware stories and docstring drift detection — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_data_datacontractgate.story.md b/dogfood/mining-output/stories/pickled_data_datacontractgate.story.md new file mode 100644 index 0000000..385c493 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_data_datacontractgate.story.md @@ -0,0 +1,84 @@ +# Story: DataContractGate.run + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-data +- **Surface id:** pickled_data_datacontractgate +- **Code depth:** callgraph | **Units read:** 4 | **Unresolved:** 2 + +## Context + +DataContractGate.run is a gate implementation used to validate that SQL query column names match OpenAPI response property names. It is invoked within a gate-checking pipeline where SQL strings are verified against API schema definitions. The caller provides a SQL query string as the target and must supply an "endpoint_tag" in the context dictionary to identify which OpenAPI schema to validate against. This gate is part of a data contract validation system ensuring that database queries align with API contracts. + +## What the target does today + +**Input acceptance:** +- Accepts a `target` of any type and a `context` dictionary (optional, defaults to empty dict if None) +- Returns a FAIL verdict if `target` is not a string, with a note indicating the actual type received +- Returns a FAIL verdict if `context` does not contain an "endpoint_tag" key with a string value + +**Schema resolution:** +- Returns a WARN verdict if no SchemaRegistry is configured on the gate instance (_registry is None) +- Delegates schema lookup to `_registry.find_schema_by_tag(endpoint_tag)` (unresolved call) +- Returns a WARN verdict if the schema artifact is not found for the given endpoint_tag + +**Column extraction:** +- Extracts column names from the SQL string by parsing it as PostgreSQL dialect using sqlglot +- Handles parse failures silently, returning an empty list if parsing fails +- Collects column names from SELECT expressions, preferring aliases when present, falling back to the expression's name attribute +- Extracts OpenAPI response property names from the schema artifact's YAML content by: + - Parsing YAML (returns empty list on parse error) + - Navigating paths → [path] → [method] → responses → (200 or 201) → content → application/json → schema → properties + - Returning sorted property names from the first matching operation found +- Returns a WARN verdict if no properties can be extracted from the OpenAPI schema + +**Validation logic:** +- Compares SQL column names (as a set) against API property names (as a set) +- Computes missing columns (in SQL but not in API) and extra columns (in API but not in SQL) +- Returns a FAIL verdict if there are any missing or extra columns, with both lists sorted and included in notes +- Returns a PASS verdict only when column name sets match exactly + +**Return value:** +- Always returns a GateResult object containing: + - gate_name: the name of this gate instance + - verdict: one of PASS, FAIL, or WARN + - notes: a descriptive string explaining the verdict + +**Explicitly not implemented:** +- Type checking of columns/properties (explicitly mentioned in PASS notes as "types not checked in v0.1") + +## What we want to verify + +- When target is not a string, verdict is FAIL with notes describing the actual type +- When context lacks "endpoint_tag" key or its value is not a string, verdict is FAIL +- When _registry is None, verdict is WARN with note "no SchemaRegistry configured" +- When schema artifact is not found for the given endpoint_tag, verdict is WARN with the tag name in notes +- When OpenAPI schema has no extractable properties, verdict is WARN +- When SQL parsing fails, the gate treats it as having zero columns and continues validation +- When SQL column set exactly matches API property set, verdict is PASS +- When SQL columns differ from API properties (missing or extra), verdict is FAIL with sorted lists of differences +- PASS verdict notes explicitly state that types are not checked in v0.1 +- All GateResult objects include the gate's name and descriptive notes +- OpenAPI property extraction only examines 200 or 201 response codes +- OpenAPI property extraction returns names from the first matching operation found + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: DataContractGate.run +- Related ADRs: +- ADR 0004: Multi-ruleset workspace configuration — Accepted +- ADR 0005: `pickled-spec mine` staged mining pipeline — Accepted +- ADR 0007: Code-aware stories and docstring drift detection — Accepted + +## Open questions + +- Docstring drift: Docstring claims "v0.1 PARTIAL: column name matching against OpenAPI response properties" but does not mention the multiple failure modes: non-string target rejection, missing endpoint_tag rejection, missing registry warning, missing schema warning, or unparseable OpenAPI warning +- Docstring drift: Docstring does not specify that the surface returns a GateResult object with verdict and notes fields +- Docstring drift: Docstring does not document the required context parameter structure (specifically that "endpoint_tag" must be present and be a string) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_data_draft.story.md b/dogfood/mining-output/stories/pickled_data_draft.story.md new file mode 100644 index 0000000..256b430 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_data_draft.story.md @@ -0,0 +1,92 @@ +# Story: draft + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-data +- **Surface id:** pickled_data_draft +- **Code depth:** callgraph | **Units read:** 8 | **Unresolved:** 3 + +## Context + +CLI command used by developers and automation pipelines to generate SQL migration files from natural-language descriptions. Invoked when a schema change is needed but the exact DDL is not yet written. The command consults an LLM to produce a draft migration, optionally considering the current schema state, and outputs the SQL plus explanatory metadata. Related to other pickled-data commands that validate schemas (DataContractGate) and check for drift (MigrationDriftGate). + +## What the target does today + +**Inputs:** +- `intent`: A string representing either a file path to read intent text from, or the literal `"-"` to read from standard input. The content is treated as natural-language description of the desired migration. +- `dialect`: A string naming the SQL dialect (e.g., "postgres", "mysql"). Used both in the LLM prompt and for SQL parsing validation. +- `current_schema`: Optional Path to a YAML file describing the current schema. If provided, the file is read (UTF-8) and passed to the LLM as context. If None, no schema context is provided. +- `output`: Optional Path for writing the generated SQL migration. If None, SQL is written to standard output. + +**Returns:** +The function signature is `-> None`; it never returns a value. Success or failure is communicated via exit codes and side effects. + +**Success path:** +1. Reads the current schema YAML file (if `current_schema` is not None) using UTF-8 encoding. +2. Builds an LLM client by delegating to an internal function that reads configuration from environment variable `PICKLED_DATA_LLM_FACTORY`. +3. Constructs a prompt incorporating the intent text, SQL dialect, and optional schema YAML, instructing the LLM to emit DDL with a comment line `-- intent: `, followed by a sentinel string and rationale. +4. Delegates to the unresolved `self._llm.complete` call (temperature=0.0, max_tokens=4000) to generate completion text. +5. Splits the LLM output at a rationale sentinel to separate SQL text from rationale explanation. +6. Validates the SQL text by attempting to parse it with sqlglot using the specified dialect, and scans for destructive operations (case-insensitive "drop table" in any line). +7. If `output` is provided, writes the SQL text to that file (UTF-8). Otherwise, writes SQL text to standard output. +8. Writes rationale lines to standard error, each prefixed with `"rationale: "`. +9. Writes validation warnings to standard error, each prefixed with `"warning: "`. +10. If any validation warnings were produced, exits with code 1. Otherwise, exits normally (implicit 0). + +**Error paths:** +- If LLM client configuration fails (ConfigError), raises `click.ClickException` with the configuration error message, which Click handles as a user-friendly error. +- If any other exception occurs during LLM client building, intent reading, drafting, or output emission (except `click.ClickException`), the exception message is written to standard error and the process exits with code 2. +- File read failures (e.g., `current_schema` or intent file not found, encoding errors) propagate as exceptions caught by the generic handler, resulting in error message to stderr and exit code 2. + +**Observable side effects:** +- Reads from standard input if `intent` is `"-"`. +- Reads files from disk when `intent` is a file path or `current_schema` is provided. +- Writes SQL migration to the specified `output` file or to standard output. +- Writes rationale and warnings to standard error. +- Process exit code: 0 on success without warnings, 1 on success with validation warnings, 2 on error. + +**Validation warnings produced:** +- If sqlglot parsing fails for the generated SQL in the specified dialect, a warning containing the parse exception message is emitted. +- For every line containing the substring "drop table" (case-insensitive), a warning is emitted noting the line number and advising confirmation before applying. + +## What we want to verify + +- Accept `intent` as `"-"` and read from standard input; verify prompt is built with stdin content. +- Accept `intent` as a file path; verify file content is read and used in prompt. +- When `current_schema` is None, verify no schema YAML is read and prompt indicates "none" for schema. +- When `current_schema` is a valid Path, verify file is read as UTF-8 and content is included in prompt. +- When `dialect` is specified, verify it is passed to both the LLM prompt and sqlglot parse validation. +- When `output` is None, verify SQL text is written to standard output. +- When `output` is a Path, verify SQL text is written to that file as UTF-8. +- When LLM output contains the rationale sentinel, verify SQL and rationale are separated and rationale lines are written to stderr with `"rationale: "` prefix. +- When LLM output lacks the rationale sentinel, verify all output is treated as SQL and no rationale is emitted. +- When sqlglot parse fails, verify a warning is written to stderr with `"warning: "` prefix containing the exception message. +- When generated SQL contains "drop table" (any case), verify a warning is emitted to stderr identifying the line number and operation. +- When validation warnings exist, verify process exits with code 1 after emitting SQL and warnings. +- When no validation warnings exist, verify process exits with code 0. +- When LLM client configuration fails with ConfigError, verify a ClickException is raised with the error message. +- When any non-ClickException occurs, verify the exception message is written to stderr and process exits with code 2. +- When `current_schema` file does not exist, verify exception is caught, message written to stderr, exit code 2. +- When intent file does not exist, verify exception is caught, message written to stderr, exit code 2. + +## Inventory references + +- Arguments: +- `intent` (required): Intent file path or '-' for stdin. +- `dialect` (required): SQL dialect for the migration. +- `current_schema` (optional): Optional existing schema YAML file. +- `output` (optional): Write SQL to this path. Default: stdout. +- Related gates: DataContractGate.run, MigrationDriftGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states the function "drafts a SQL migration from a natural-language intent" but omits key observable behaviors: the function reads optional schema context from a file, validates the generated SQL for parse errors and destructive operations, writes rationale and warnings to stderr, exits with different codes based on validation results (0 for clean, 1 for warnings, 2 for errors), and supports reading intent from stdin via `"-"`. +- Docstring drift: The docstring does not mention the `output` parameter's effect of controlling whether SQL is written to a file or stdout. +- Docstring drift: The docstring does not describe any error handling or exit code behavior, which is a significant part of the observable contract. + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_data_mcp.story.md b/dogfood/mining-output/stories/pickled_data_mcp.story.md new file mode 100644 index 0000000..70c6ea0 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_data_mcp.story.md @@ -0,0 +1,49 @@ +# Story: mcp + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-data +- **Surface id:** pickled_data_mcp +- **Code depth:** callgraph | **Units read:** 1 | **Unresolved:** 0 + +## Context + +This is a CLI command entry point for MCP (Model Context Protocol) server operations within the pickled-data package. It serves as a command group or namespace for organizing MCP-related subcommands. Users invoke this through a CLI tool to access MCP server functionality. + +## What the target does today + +The `mcp` function is a zero-argument callable that performs no operations and returns `None`. When invoked: + +- Accepts no parameters +- Returns `None` (implicitly, as the function body is empty) +- Produces no side effects (no I/O, no state mutation, no exceptions raised) +- Does not validate any input (as there are no inputs) +- Does not call any other functions or collaborators + +The function serves solely as a declaration point, likely intended to be decorated or registered by a CLI framework (such as Click or Typer) to create a command group that organizes subcommands like `DataContractGate.run`, `MigrationDriftGate.run`, and `run_all` under the "mcp" namespace. + +## What we want to verify + +- Calling `mcp()` completes without raising any exceptions +- Calling `mcp()` returns `None` +- Calling `mcp()` with any arguments raises a `TypeError` due to the zero-parameter signature +- Calling `mcp()` produces no observable side effects (no file I/O, no network calls, no stdout/stderr output) +- The function can be successfully imported and invoked as a standalone Python function +- The function's `__doc__` attribute contains the string "MCP server commands." + +## Inventory references + +- Arguments: +- (none) +- Related gates: DataContractGate.run, MigrationDriftGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring claims "MCP server commands" (plural, suggesting multiple commands or operations), but the implementation is an empty function that performs no command execution, validation, routing, or delegation whatsoever. The code does not implement any command handling behavior that would justify the "commands" description. + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_data_migrationdriftgate.story.md b/dogfood/mining-output/stories/pickled_data_migrationdriftgate.story.md new file mode 100644 index 0000000..38033da --- /dev/null +++ b/dogfood/mining-output/stories/pickled_data_migrationdriftgate.story.md @@ -0,0 +1,92 @@ +# Story: MigrationDriftGate.run + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-data +- **Surface id:** pickled_data_migrationdriftgate +- **Code depth:** callgraph | **Units read:** 5 | **Unresolved:** 7 + +## Context + +This gate is used by the pickled-data migration validation pipeline to verify that a SQL migration script produces the expected database schema. Callers provide a migration SQL string as the target, along with context containing either a parsed expected schema dictionary or a YAML string describing the expected schema. The gate compares the schema produced by applying the migration against the expected schema and reports whether they match. + +## What the target does today + +**Inputs:** +- `target`: Must be a string containing SQL migration statements. Any other type results in immediate failure. +- `context`: Optional dictionary that must contain schema expectations via one of: + - `"expected_schema"`: A dictionary directly describing the expected schema + - `"expected_schema_yaml"`: A YAML string that parses to a dictionary + - `"dialect"`: Optional SQL dialect string (defaults to "postgres") + +**Validation and rejection:** +- Rejects non-string targets with verdict FAIL and a note identifying the received type. +- Rejects contexts lacking both `"expected_schema"` (as dict) and `"expected_schema_yaml"` (as parseable-to-dict string) with verdict FAIL. +- If `"expected_schema_yaml"` is provided but does not parse to a dictionary via `yaml.safe_load`, the context is considered invalid. + +**Schema comparison:** +- Parses and applies the migration SQL to an in-memory SQLite database (or file-based if specified elsewhere in the call chain). +- The SQL is first parsed in the specified dialect, then transpiled to SQLite dialect for execution. +- Extracts the resulting schema from the SQLite database via introspection. +- Normalizes both expected and actual schemas into a format mapping table names to sets of `(column_name, type, nullable)` tuples. +- Column types are normalized to uppercase. +- Nullable defaults to `True` if not specified in the schema. + +**Comparison rules:** +- Table name sets must match exactly between expected and actual schemas. +- For each table, the set of `(column_name, type)` pairs must match exactly. +- Nullable differences are NOT considered drift; the gate passes with a note if only nullable flags differ. +- If table sets differ, returns FAIL with a note listing both expected and actual table names. +- If column name/type sets differ for any table, returns FAIL with a note identifying the table and both column sets. + +**Return value:** +Returns a `GateResult` with: +- `gate_name`: The name of this gate instance +- `verdict`: `Verdict.PASS` if schemas match (ignoring nullable), `Verdict.FAIL` otherwise +- `notes`: + - On success with no nullable drift: "Schema matches expected." + - On success with nullable drift: "Schema matches expected (nullable differs:
)." + - On failure: Description of the drift (table mismatch or column mismatch) + - On input validation failure: Description of the validation error + +**Side effects and safety:** +- Creates a temporary SQLite database connection to apply the migration. +- The connection is closed after introspection regardless of success or failure. +- Sets SQLite attachment limit to 0 to prevent ATTACH statements (defense-in-depth security measure). +- SQL statements are parsed and transpiled; filesystem escape attempts are rejected during parsing. + +## What we want to verify + +- When target is not a string, returns GateResult with verdict FAIL and notes containing the actual type name +- When context lacks both "expected_schema" dict and valid "expected_schema_yaml", returns FAIL with notes about missing context +- When "expected_schema_yaml" is provided as a string, it is parsed via yaml.safe_load +- When "expected_schema_yaml" parses to a non-dict value, the gate treats it as invalid context and returns FAIL +- When "expected_schema" is already a dict in context, it is used directly without YAML parsing +- The dialect from context["dialect"] is used for initial SQL parsing (defaults to "postgres") +- Migration SQL is transpiled to SQLite dialect for execution +- Schema comparison normalizes column types to uppercase +- Schema comparison treats nullable as True by default when not specified +- When table name sets differ between expected and actual, verdict is FAIL with notes listing both sets +- When column (name, type) pairs differ for any table, verdict is FAIL with notes identifying the table and both column sets +- When only nullable flags differ, verdict is PASS with notes describing the nullable differences +- When schemas match exactly including nullable, verdict is PASS with notes "Schema matches expected." +- The SQLite connection is closed in a finally block regardless of execution outcome +- Empty or None statements from SQL parsing are skipped during transpilation + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: MigrationDriftGate.run +- Related ADRs: +- ADR 0007: Code-aware stories and docstring drift detection — Accepted + +## Open questions + +- Docstring drift: The docstring states "Compare oracle schema output vs expected schema YAML" but the code accepts the expected schema as either a dictionary via `"expected_schema"` key OR as YAML via `"expected_schema_yaml"` key, not exclusively from YAML. +- Docstring drift: The docstring mentions "oracle schema output" but the code actually applies the migration SQL to SQLite (after transpilation) and introspects the resulting SQLite schema, not an oracle database. + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_data_parse.story.md b/dogfood/mining-output/stories/pickled_data_parse.story.md new file mode 100644 index 0000000..411a1a4 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_data_parse.story.md @@ -0,0 +1,80 @@ +# Story: parse + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-data +- **Surface id:** pickled_data_parse +- **Code depth:** callgraph | **Units read:** 6 | **Unresolved:** 0 + +## Context + +This CLI command is used by developers and tooling to inspect migration SQL files. It parses a SQL file into an abstract syntax tree (AST) and outputs a JSON summary to standard output. The command is invoked as part of the pickled-data CLI tool to understand the structure of migration SQL before execution or analysis. Related gates (DataContractGate.run, MigrationDriftGate.run, run_all) likely consume or validate migration files, making this parse command useful for debugging and inspection during development or CI/CD workflows. + +## What the target does today + +The surface accepts a file path (`migration`) and an optional SQL dialect string (`dialect`). It performs the following observable behaviors: + +**Input validation:** +- Rejects files with a `.dbt` suffix by raising `NotImplementedError` with a message indicating DBT files are not supported +- This rejection occurs twice in the call chain (once in `_check_dbt` at the root level, once in `_reject_dbt` within `load_sql_file`) + +**File processing:** +- Reads the file at the given path using UTF-8 encoding +- Parses the file content as SQL using the specified dialect (defaults to "postgres" if not provided) +- Parsing delegates to `sqlglot.parse_one` (external library call) + +**Error handling:** +- If the external parser raises `sqlglot.errors.ParseError`, wraps it as `SQLParseError` with the original error message +- If parsing returns `None`, raises `SQLParseError` with message "empty parse result" +- File I/O errors (e.g., file not found, permission denied) propagate as standard Python exceptions + +**Output:** +- Prints a JSON object to standard output (via `click.echo`) containing: + - `"dialect"`: the dialect string used for parsing + - `"kind"`: the AST node type name (Python class name of the root AST node) + - `"sql"`: the SQL representation of the AST, rendered in "postgres" dialect regardless of input dialect +- The JSON is formatted with 2-space indentation + +**Side effects:** +- Writes formatted JSON to standard output +- No modifications to the filesystem or other persistent state + +**Return value:** +- The function signature indicates it returns `None`; output is via side effect (printing to stdout) + +## What we want to verify + +- Accepts a Path object and optional dialect string as parameters +- Rejects files with `.dbt` extension by raising NotImplementedError before attempting to read +- Reads the migration file content using UTF-8 encoding +- Parses SQL content using the specified dialect (or "postgres" if not specified) +- Raises SQLParseError when the SQL cannot be parsed by sqlglot +- Raises SQLParseError with message "empty parse result" when parser returns None +- Outputs valid JSON to stdout containing "dialect", "kind", and "sql" keys +- The "dialect" field in output matches the dialect parameter used for parsing +- The "kind" field contains the Python class name of the AST root node +- The "sql" field contains the AST rendered in "postgres" dialect regardless of input dialect +- JSON output is indented with 2 spaces +- Returns None (no return value, output is side effect only) +- Propagates file I/O exceptions (FileNotFoundError, PermissionError, etc.) without wrapping + +## Inventory references + +- Arguments: +- `migration` (required): +- `dialect` (optional): +- Related gates: DataContractGate.run, MigrationDriftGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states "print AST summary" but does not mention that the output is formatted as JSON, which is a specific and important detail for callers who need to parse the output +- Docstring drift: The docstring does not mention the `.dbt` file rejection behavior, which is a prominent input validation constraint +- Docstring drift: The docstring does not describe the structure of the output (the three fields: dialect, kind, sql) or that the SQL output is always rendered in postgres dialect +- Docstring drift: The docstring does not mention any error conditions (SQLParseError, NotImplementedError for .dbt files) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_data_run_all.story.md b/dogfood/mining-output/stories/pickled_data_run_all.story.md new file mode 100644 index 0000000..36737b4 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_data_run_all.story.md @@ -0,0 +1,83 @@ +# Story: run_all + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-data +- **Surface id:** pickled_data_run_all +- **Code depth:** callgraph | **Units read:** 4 | **Unresolved:** 0 + +## Context + +This gate function is used by clients running quality checks on a data project's migration files. It orchestrates parsing of SQL migration files and validates whether applying those migrations produces a schema that matches an expected YAML specification. The function is designed to be called with a project working directory, returning a list of pass/warn/fail gate results that can be aggregated into a quality report. + +## What the target does today + +**Accepts**: A working directory path (Path or str) containing a `migrations/` subdirectory and optionally an `expected_schema.yaml` file at the root. + +**Returns**: A list of GateResult objects, each representing a specific check. The function never returns an empty list. + +**Basic flow**: +- Resolves the working directory to an absolute path +- Globs for `migrations/*.sql` files and sorts them by filename +- If no migration files exist, returns a single WARN result with gate_name "data.migrations" and notes "no migrations/*.sql" +- For each migration file found, attempts to parse it as SQLite-dialect SQL: + - On successful parse: emits a PASS result with gate_name "data.parse.{filename}" + - On parse failure (SQLParseError): emits a FAIL result with gate_name "data.parse.{filename}" and the exception message as notes +- If more than one migration file exists, emits an additional WARN result with gate_name "data.migrations.note" explaining that multiple migrations will be applied in filename order +- If `expected_schema.yaml` exists and is a file, and at least one migration exists: + - Concatenates all migration file contents with double-newline separators + - Loads the YAML file as a dict using yaml.safe_load + - If the loaded YAML is a dict, delegates to MigrationDriftGate().run() with the combined SQL, passing expected schema and dialect "sqlite" in context + - Appends the drift gate result with gate_name "data.migration_drift" + +**Error modes**: +- SQL parse errors are caught and converted to FAIL gate results (not raised) +- .dbt file extensions in migration paths trigger NotImplementedError +- YAML parsing errors or file read errors are not caught and will propagate to the caller +- Type mismatches or other exceptions from MigrationDriftGate().run() are not caught + +**Side effects**: +- Reads files from disk (migration SQL files and expected_schema.yaml) +- No writes or persistent state changes + +**Constraints**: +- Hard-coded to SQLite dialect for all parsing and drift checking +- Migration files must have .sql extension to be discovered by glob +- Expected schema must deserialize to a dict, otherwise drift gate is skipped silently +- Filename-based lexicographic sorting determines migration application order + +## What we want to verify + +- When workdir contains no migrations/*.sql files, returns exactly one GateResult with verdict WARN and gate_name "data.migrations" +- When migrations exist but expected_schema.yaml does not exist or is not a file, drift gate result is not included in output +- When a migration file cannot be parsed, returns a FAIL result for that file with gate_name "data.parse.{filename}" and includes the parse error message +- When a migration file parses successfully, returns a PASS result with gate_name "data.parse.{filename}" +- When exactly one migration exists, no WARN result about multiple migrations is emitted +- When more than one migration exists, emits a WARN result with gate_name "data.migrations.note" +- When expected_schema.yaml exists and contains a dict, and migrations exist, includes a gate result with gate_name "data.migration_drift" +- All SQL parsing uses "sqlite" dialect regardless of file content or project configuration +- Migration files are sorted lexicographically by filename before processing +- Combined SQL passed to drift gate concatenates migrations with "\n\n" separator +- YAML file that does not deserialize to a dict causes drift gate to be skipped (no error raised) + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: Docstring does not specify the return value is a list of GateResult objects +- Docstring drift: Docstring does not mention the WARN result returned when no migrations exist +- Docstring drift: Docstring does not mention the per-file parse gate results +- Docstring drift: Docstring does not mention the WARN result for multiple migrations +- Docstring drift: Docstring claims general parsing behavior but code hard-codes SQLite dialect, not configurable +- Docstring drift: Docstring does not specify that drift checking is conditional on expected_schema.yaml existing and being a valid dict-containing YAML file + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_diff_draft_corpus.story.md b/dogfood/mining-output/stories/pickled_diff_draft_corpus.story.md new file mode 100644 index 0000000..a060733 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_diff_draft_corpus.story.md @@ -0,0 +1,47 @@ +# Story: draft-corpus + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-diff +- **Surface id:** pickled_diff_draft_corpus + +## Context + +Developers and test engineers use `draft-corpus` to generate larger differential test corpora from a small set of seed examples. This is typically needed when creating test datasets that require many variations of similar items, where manually writing each example would be tedious. The command takes seed data and expands it to a specified target size, optionally incorporating additional notes. + +## What the target does today + +The `draft-corpus` command expands seed examples into a larger differential corpus. It reads seed items from a JSON file (or stdin when '-' is specified), generates additional corpus items to reach the specified target size, and outputs the resulting corpus as JSON. An optional notes file can be provided to influence corpus generation. By default, output is written to stdout, but can be directed to a file via the `--output` option. + +## What we want to verify + +- When given a seeds JSON file and target_size, the command produces a corpus with exactly target_size items +- When seeds is '-', the command reads seed data from stdin +- When output is not specified, the command writes the corpus JSON to stdout +- When output is specified, the command writes the corpus JSON to the specified file path +- When notes is '-', the command reads notes from stdin +- When notes is a file path, the command reads notes from that file +- The output is valid JSON +- The output corpus includes the original seed items or variations derived from them +- When target_size is less than or equal to the number of seeds, the command handles this appropriately +- The command exits with status 0 on successful corpus generation + +## Inventory references + +- Arguments: +- `seeds` (required): JSON file with seed items, or '-' for stdin. +- `target_size` (required): Total corpus size. +- `notes` (optional): Optional notes file path or '-' for stdin. +- `output` (optional): Write corpus JSON to this path. Default: stdout. +- Related gates: run_all +- Related ADRs: +- ADR 0001: pickled-diff package — Proposed + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_diff_mcp.story.md b/dogfood/mining-output/stories/pickled_diff_mcp.story.md new file mode 100644 index 0000000..465468e --- /dev/null +++ b/dogfood/mining-output/stories/pickled_diff_mcp.story.md @@ -0,0 +1,41 @@ +# Story: mcp + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-diff +- **Surface id:** pickled_diff_mcp +- **Code depth:** callgraph | **Units read:** 1 | **Unresolved:** 0 + +## Context + +This is a CLI command surface in the pickled-diff package, intended to expose MCP (Model Context Protocol) server functionality through the command-line interface. It serves as an entry point for users or tools invoking MCP-related server commands via the CLI. The surface is likely registered as a Click command or similar CLI framework command group. + +## What the target does today + +The function accepts no arguments and returns None. When invoked, the function performs no operations—it has an empty body that immediately returns. No validation occurs, no errors are raised, no side effects are triggered, and no output is produced. The function serves only as a placeholder or stub declaration. + +## What we want to verify + +- When called with no arguments, the function completes without error +- The function returns None (implicitly) +- No exceptions are raised during execution +- No side effects occur (no I/O, no state changes, no external calls) +- The function accepts exactly zero parameters + +## Inventory references + +- Arguments: +- (none) +- Related gates: run_all +- Related ADRs: +- ADR 0001: pickled-diff package — Proposed + +## Open questions + +- Docstring drift: The docstring claims "MCP server commands" (plural) suggesting this should provide or coordinate multiple server commands, but the implementation is empty and provides no command functionality whatsoever +- Docstring drift: The docstring implies active behavior (providing server commands), while the code performs no operations and has no implementation + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_diff_run_all.story.md b/dogfood/mining-output/stories/pickled_diff_run_all.story.md new file mode 100644 index 0000000..253a548 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_diff_run_all.story.md @@ -0,0 +1,78 @@ +# Story: run_all + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-diff +- **Surface id:** pickled_diff_run_all +- **Code depth:** callgraph | **Units read:** 2 | **Unresolved:** 0 + +## Context + +This gate function is the main entry point for the pickled-diff differential testing framework. It is invoked by the pickled gate runner to verify that a candidate implementation matches an oracle implementation across a corpus of test inputs. Users configure the gate through a `pickled.diff.yaml` configuration file in their project, specifying oracle and candidate commands, a corpus file, an optional comparator, and optional timeout settings. + +## What the target does today + +**Signature**: Accepts a single parameter `workdir` (Path or str) representing the working directory root, and returns a list of `GateResult` objects. + +**Configuration discovery**: Searches for a configuration file named `pickled.diff.yaml` in the working directory. If not found, returns a single-element list containing a `GateResult` with gate_name "diff.config", verdict WARN, and a note about the missing configuration file. + +**Configuration parsing and validation**: If a configuration file is found, attempts to parse it as YAML and extract: +- `oracle_command`: command-line arguments for the oracle subprocess (validated to be a list) +- `candidate_command`: command-line arguments for the candidate subprocess (validated to be a list) +- `corpus`: a path string to a corpus file (must be a string, otherwise raises ValueError) +- `comparator`: optional string naming the comparator strategy (defaults to "exact") +- `timeout_seconds`: optional numeric timeout value (defaults to 30) + +**Error handling for configuration**: If configuration parsing fails due to ValueError, FileNotFoundError, json.JSONDecodeError, or TypeError, returns a single-element list containing a `GateResult` with gate_name "diff.config", verdict FAIL, and the exception message as notes. + +**Python executable normalization**: When the oracle or candidate command starts with "python" or "python3" (and has additional arguments), replaces that first token with `sys.executable` to ensure the current Python interpreter is used. + +**Corpus loading**: Loads the corpus from the specified path relative to the working directory root. The corpus loading mechanism is delegated to an unresolved `_load_corpus` function. + +**Command resolution**: Resolves command arguments using `_resolve_argv` and `_argv_list`, which handle path resolution relative to the working directory root. + +**Comparator selection**: Selects a comparator function by name via an unresolved `_comparator` function. + +**Differential oracle execution**: Creates a `DifferentialOracleGate` with two `SubprocessRunner` instances (oracle and candidate), each configured with: +- Resolved command arguments +- A name identifier ("oracle" or "candidate") +- The configured timeout +- The working directory as `cwd` + +Runs the differential gate on the loaded corpus and returns a single-element list containing a `GateResult` with gate_name "diff.differential_oracle", copying the verdict, findings, and notes from the gate's result. + +**Return value**: Always returns a list of `GateResult` objects. The list contains exactly one element in all code paths: either a configuration-related result (WARN or FAIL) or a differential oracle execution result. + +## What we want to verify + +- When `pickled.diff.yaml` is absent, returns a list with one GateResult having gate_name "diff.config", verdict WARN, and notes mentioning the missing file +- When configuration file exists but corpus key is not a string, returns a list with one GateResult having gate_name "diff.config" and verdict FAIL +- When configuration file exists but oracle_command or candidate_command are not lists, returns a list with one GateResult having gate_name "diff.config" and verdict FAIL +- When configuration is valid, returns a list with one GateResult having gate_name "diff.differential_oracle" +- When oracle_command starts with "python" or "python3", the subprocess runner uses sys.executable instead +- When candidate_command starts with "python" or "python3", the subprocess runner uses sys.executable instead +- Python executable substitution only occurs when the command has more than one element +- Configuration timeout_seconds defaults to 30 when not specified +- Configuration comparator defaults to "exact" when not specified +- FileNotFoundError during configuration loading results in verdict FAIL, not WARN +- JSONDecodeError during corpus loading results in verdict FAIL with the exception message in notes +- The returned list always contains exactly one GateResult object + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: run_all +- Related ADRs: +- ADR 0001: pickled-diff package — Proposed + +## Open questions + +- Docstring drift: The docstring states the gate runs "when `pickled.diff.yaml` is present" but the code actually runs (and returns a result) even when the file is absent—it returns a WARN verdict in that case rather than skipping execution +- Docstring drift: The docstring does not mention searching for `diff/pickled.diff.yaml` as an alternative location, but the notes field in the WARN result explicitly mentions "diff/pickled.diff.yaml" as a possible location +- Docstring drift: The docstring does not describe the return type (list of GateResult objects) or any of the failure modes (configuration errors, missing corpus, invalid commands) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_diff_verify.story.md b/dogfood/mining-output/stories/pickled_diff_verify.story.md new file mode 100644 index 0000000..62d628b --- /dev/null +++ b/dogfood/mining-output/stories/pickled_diff_verify.story.md @@ -0,0 +1,105 @@ +# Story: verify + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-diff +- **Surface id:** pickled_diff_verify +- **Code depth:** callgraph | **Units read:** 4 | **Unresolved:** 4 + +## Context + +This surface is the primary CLI command for differential testing. A user invokes it to compare two programs (oracle and candidate) across a corpus of test inputs, using a specified comparison strategy. The surface is used to verify that a candidate implementation produces equivalent output to a trusted oracle across many test cases, with configurable timeout and comparison logic. + +## What the target does today + +**Input Validation** + +The surface reads a JSON file from the `corpus` path (UTF-8 encoded) and expects a top-level list. If the parsed JSON is not a list, the surface raises a `click.ClickException` with the message "Corpus JSON must be a list of {name, payload} objects". The surface filters the list to include only dictionary entries and extracts `name` and `payload` fields, coercing both to strings. Non-dictionary entries are silently ignored. + +**Command Parsing and Execution** + +The `oracle` and `candidate` strings are parsed as shell-quoted command-line arguments (via `shlex.split`). Each command is wrapped in a subprocess runner with the specified `timeout_seconds`. The surface does not validate that the commands exist or are executable before attempting to run the gate. + +**Comparison Strategy** + +The `comparator` parameter selects a comparison strategy: +- If `comparator` equals `"structural_json"`, a structural JSON comparator is used. +- For any other value (including empty string, None, or any unrecognized value), an exact equality comparator is used. + +**Differential Testing Logic** + +The surface delegates the actual testing to a `DifferentialOracleGate` through its `run` method, which: +- Returns a `FAIL` verdict if the target is not a recognized corpus type. +- Returns a `PASS` verdict with explanatory notes if the corpus is empty. +- For each corpus item, runs both oracle and candidate commands with the item's payload. +- Tracks oracle errors (oracle command failures), candidate errors (candidate command failures), and mismatches (outputs differ according to the comparator). +- Oracle errors cause the item to be skipped from comparison entirely (not counted as compared). +- Candidate errors are counted as both compared items and mismatches, and generate findings. +- Accumulates findings (up to an unspecified maximum) containing input name, both outputs, and a diff summary. +- Applies verdict logic: + - `FAIL` if all inputs caused oracle errors (with special message). + - `WARN` if no items were successfully compared. + - `FAIL` if all compared items mismatched. + - `WARN` if some mismatches or oracle errors occurred. + - `PASS` if no mismatches and no oracle errors. + +**Output** + +The surface writes a JSON object to stdout containing: +- `gate`: the gate name +- `verdict`: string representation of the verdict enum value +- `notes`: textual summary including mismatch counts, corpus size, and error counts +- `findings`: list of objects with `input_repr`, `oracle_output`, `candidate_output`, and `diff_summary` fields + +**Exit Behavior** + +The surface calls `sys.exit` with: +- Exit code 0 for `PASS` verdict +- Exit code 1 for `WARN` verdict +- Exit code 2 for `FAIL` verdict + +**Unresolved Behavior** + +- The actual command execution (how subprocess runners handle timeouts, capture output, detect errors) is delegated to unresolved collaborators. +- The comparison logic (what "equal" means for each comparator, what diff summaries look like) is delegated to unresolved comparator implementations. +- The maximum number of findings collected is determined by an unresolved gate configuration parameter. + +## What we want to verify + +- Parse a corpus JSON file that is not a list and observe a `click.ClickException` with message containing "must be a list" +- Parse a corpus JSON containing `[{"name": "a", "payload": "b"}]` and verify both commands receive payload "b" +- Parse a corpus JSON containing `[{"name": 1, "payload": 2}]` and verify name and payload are coerced to strings "1" and "2" +- Parse a corpus JSON containing `[{"name": "x", "payload": "y"}, "not-a-dict", {"name": "z", "payload": "w"}]` and verify only the two dictionary entries are processed +- Invoke with `comparator="structural_json"` and verify a structural JSON comparator is selected +- Invoke with `comparator="exact"` (or any non-"structural_json" value) and verify an exact equality comparator is selected +- Provide an empty corpus list and verify exit code 0 (PASS) with notes mentioning "empty" +- Mock all items to produce oracle errors and verify exit code 2 (FAIL) with notes mentioning "Oracle failed on every input" +- Mock some compared items to mismatch and verify exit code 1 (WARN) +- Mock all compared items to mismatch and verify exit code 2 (FAIL) +- Mock all items to pass comparison and verify exit code 0 (PASS) with "0/N mismatches" in notes +- Verify JSON output contains keys `gate`, `verdict`, `notes`, and `findings` +- Verify each finding in JSON output contains `input_repr`, `oracle_output`, `candidate_output`, and `diff_summary` +- Parse a corpus JSON missing `name` or `payload` keys and observe a KeyError (no default handling) + +## Inventory references + +- Arguments: +- `oracle` (required): Reference command (shell-quoted argv). +- `candidate` (required): Candidate command (shell-quoted argv). +- `corpus` (required): JSON file: [{"name": "...", "payload": "..."}, ...] +- `comparator` (optional): +- `timeout_seconds` (optional): +- Related gates: run_all +- Related ADRs: +- ADR 0001: pickled-diff package — Proposed + +## Open questions + +- Docstring drift: The docstring states "Compare candidate vs reference" but does not mention: (1) the corpus must be a JSON list of specific shape, (2) the surface exits with specific codes based on verdict, (3) the surface outputs JSON to stdout, (4) non-dictionary corpus entries are silently filtered, (5) the comparator parameter exists and controls comparison strategy, (6) timeout behavior is configurable, (7) oracle errors cause items to be skipped rather than compared. +- Docstring drift: The docstring does not describe any error conditions, but the code raises `click.ClickException` for invalid corpus format. +- Docstring drift: The docstring does not mention the return value or exit behavior; the code calls `sys.exit` and functionally returns nothing (`-> None` is misleading since execution terminates). + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_iac_diff.story.md b/dogfood/mining-output/stories/pickled_iac_diff.story.md new file mode 100644 index 0000000..02ab6ad --- /dev/null +++ b/dogfood/mining-output/stories/pickled_iac_diff.story.md @@ -0,0 +1,97 @@ +# Story: diff + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-iac +- **Surface id:** pickled_iac_diff +- **Code depth:** callgraph | **Units read:** 3 | **Unresolved:** 0 + +## Context + +This CLI command is invoked by users (or automation) who want to compare two Terraform plan JSON files to understand what infrastructure changes differ between them. Typical use cases include CI/CD pipelines that validate plan changes between a baseline (e.g., main branch) and a proposed change (e.g., pull request), or operators manually auditing infrastructure drift. + +## What the target does today + +**Inputs:** +- Accepts two required arguments: `base_path` and `head_path`, both of type `Path` +- Expects both paths to point to files containing valid UTF-8 encoded JSON representing Terraform plan files +- The JSON files must be dictionaries containing a top-level `resource_changes` key (optional or null) with a list of resource change objects + +**Core operation:** +- Reads and parses both JSON files from disk +- Compares the resource changes between base and head plans by: + - Indexing changes by resource address and action type from the `resource_changes` array in each plan + - Each resource change must be a dict with an `address` field and a `change` field containing an `actions` list + - Identifying resources that have different actions between base and head, or appear only in head + - Ignoring resources that appear only in base + +**Outputs to stdout:** +- Emits a JSON object with the following structure: + - `verdict`: string value from enumeration ("PASS", "WARN", or "FAIL") + - `notes`: string describing the outcome + - `findings`: array of objects, each with: + - `address`: string resource address + - `actions_before`: list of action strings from base plan + - `actions_after`: list of action strings from head plan + +**Exit behavior:** +- Exits with status 0 if verdict is PASS (no changes detected between plans) +- Exits with status 1 if verdict is WARN (only safe actions: create, update, read, no-op detected) +- Exits with status 2 if verdict is FAIL (destructive actions like delete or replace detected, or invalid input) +- Exits with status 2 if either plan is not a dictionary +- Exits with status 2 if `base_plan` is missing or not a dictionary in context + +**Verdict logic:** +- PASS: No resource changes detected between base and head +- FAIL: Any action is "delete" or "replace" +- WARN: All actions are from the set {create, update, read, no-op} +- WARN: Default if actions exist but don't match the FAIL or safe-only criteria + +**Error modes:** +- If file reading fails (missing file, permission denied, encoding errors), an exception propagates uncaught +- If JSON parsing fails in either file, an exception propagates uncaught +- Non-dict resource change entries in the `resource_changes` array are silently skipped +- Missing or non-dict `change` fields are handled gracefully (treated as no actions) +- Non-list `actions` values are handled gracefully (treated as no actions) +- Empty or missing `address` fields default to empty string + +## What we want to verify + +- When both plans contain identical resource changes, exits with status 0 and verdict "PASS" +- When both plans are empty (no resource_changes), exits with status 0 and verdict "PASS" +- When head plan contains a new resource with create action not in base, exits with status 1 and verdict "WARN" +- When head plan contains a resource with delete action, exits with status 2 and verdict "FAIL" +- When head plan contains a resource with replace action, exits with status 2 and verdict "FAIL" +- When a resource has different actions between base and head, it appears in findings with both actions_before and actions_after populated +- When a resource appears only in head (not in base) with non-empty actions, it appears in findings with empty actions_before +- When base_path or head_path does not exist, raises an exception (file not found) +- When either file contains invalid JSON, raises a JSON decode exception +- When head plan is not a dictionary, exits with status 2 and verdict "FAIL" with appropriate notes +- The stdout output is valid JSON with exactly the keys: verdict, notes, findings +- Each finding in the output contains exactly the keys: address, actions_before, actions_after +- Resources appearing only in base plan (removed in head) do not generate findings +- Non-dictionary entries in resource_changes arrays are silently ignored +- When all actions are from {create, update, read, no-op}, exits with status 1 and verdict "WARN" + +## Inventory references + +- Arguments: +- `base_path` (required): +- `head_path` (required): +- Related gates: IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states "Compare two terraform plan JSON files" but omits several observable behaviors: +- Docstring drift: Does not mention it writes JSON-formatted comparison results to stdout +- Docstring drift: Does not mention the three distinct exit codes (0, 1, 2) based on verdict +- Docstring drift: Does not mention the specific verdict logic (FAIL for delete/replace, WARN for safe changes, PASS for no changes) +- Docstring drift: Does not mention the specific output schema with verdict, notes, and findings fields +- Docstring drift: Does not mention error handling behavior (uncaught exceptions for file/JSON errors vs. graceful handling of malformed resource entries) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_iac_draft.story.md b/dogfood/mining-output/stories/pickled_iac_draft.story.md new file mode 100644 index 0000000..79e5a7c --- /dev/null +++ b/dogfood/mining-output/stories/pickled_iac_draft.story.md @@ -0,0 +1,86 @@ +# Story: draft + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-iac +- **Surface id:** pickled_iac_draft +- **Code depth:** callgraph | **Units read:** 5 | **Unresolved:** 5 + +## Context + +This CLI command is the primary entry point for generating a Terraform infrastructure module from a natural-language user story. It is invoked by users who want to draft IaC code without manually writing HCL. The command accepts a user story describing desired infrastructure, optionally writes the generated Terraform module to a specified directory, or prints it to stdout. It is designed for interactive use or scripting workflows where infrastructure requirements are expressed in prose and translated into executable Terraform code. + +## What the target does today + +**Inputs:** +- `user_story` (str, required): A natural-language description of the desired infrastructure. Passed directly to the drafting logic. +- `provider` (str, required): The cloud provider for which to generate Terraform code (e.g., "aws"). No default is enforced by this function; caller must supply. +- `output` (Path | None, optional): File-system path where the generated module should be written. If None, output is printed to stdout. + +**LLM client construction:** +The surface constructs an LLM client by first checking the `PICKLED_IAC_LLM_FACTORY` environment variable. If set, it must follow the format `"module:callable"` (e.g., `"mypackage:get_client"`); the function dynamically imports the module and calls the specified attribute to obtain an LLM client. If the variable is not set, the surface falls back to reading `PICKLED_LLM_PROVIDER` (defaulting to "anthropic") and uses `pickled_core.llm.factory.build_client` with configuration loaded from `pickled_core.llm.config.load_config()`. If configuration is invalid or missing, a `click.ClickException` is raised with the underlying `ConfigError` message. + +**Module generation process:** +The surface delegates drafting to `IaCDrafter.draft_module`, which: +1. Detects whether `terraform` or `opentofu` (aliased as `tofu`) is available on the system PATH. If neither is found, raises `IaCToolMissingError`. +2. Renders a prompt template (unresolved call) incorporating the user story, provider, and any prior validation error feedback. +3. Calls the LLM (via unresolved `complete_prompt`) with the rendered prompt and a system instruction to output only Terraform HCL without code fences or commentary. +4. Strips any leading/trailing triple-backtick fences from the LLM response if present. +5. Writes the resulting HCL to a temporary directory and runs `terraform validate -json` (or `opentofu validate -json`). The validation logic initializes the directory if needed. +6. If validation fails, extracts diagnostic messages from the JSON output and retries generation up to 3 times, appending the previous error feedback to the prompt. +7. After 3 failed attempts, raises `IaCValidationError` with all collected diagnostics. +8. On success, returns an `IaCArtifact` containing the validated HCL content and the format ("terraform" or "opentofu"). + +**Outputs:** +- If `output` is provided: Creates the directory (including parents) if it does not exist, writes the artifact content to `main.tf` within that directory, and prints a confirmation message `"Wrote /main.tf"` to stderr. +- If `output` is None: Prints the artifact content (the raw HCL) to stdout. + +**Error modes:** +- Raises `click.ClickException` if `PICKLED_IAC_LLM_FACTORY` is malformed (missing colon separator). +- Raises `click.ClickException` wrapping `ConfigError` if LLM configuration cannot be loaded. +- Raises `IaCToolMissingError` if neither Terraform nor OpenTofu is found on PATH. +- Raises `IaCValidationError` if the generated HCL fails validation after 3 attempts, including all accumulated diagnostics. +- May raise file-system exceptions (e.g., permission errors) when creating `output` directory or writing `main.tf`. + +**Side effects:** +- Executes external `terraform` or `opentofu` binaries for validation, which may create `.terraform` directories and lock files in temporary directories. +- Writes to the file system if `output` is specified. +- Prints to stderr (confirmation message) or stdout (HCL content) depending on `output` presence. + +## What we want to verify + +- When `output` is None, the function prints the generated HCL content to stdout and does not create any files. +- When `output` is a valid Path, the function creates the directory (including parents) if it does not exist and writes a file named `main.tf` containing the generated HCL content. +- The function prints a confirmation message to stderr in the form `"Wrote /main.tf"` when `output` is provided. +- If neither `terraform` nor `opentofu` (or `tofu`) is available on PATH, the function raises `IaCToolMissingError`. +- If `PICKLED_IAC_LLM_FACTORY` is set but does not contain a colon separator, the function raises `click.ClickException` with a message indicating the required format. +- If LLM configuration cannot be loaded and `PICKLED_IAC_LLM_FACTORY` is not set, the function raises `click.ClickException` wrapping the underlying `ConfigError`. +- The function retries generation up to 3 times if the generated HCL fails validation. +- If all 3 generation attempts produce invalid HCL, the function raises `IaCValidationError` containing all collected validation diagnostics. +- The validation process runs `terraform validate -json` or `opentofu validate -json` against the generated HCL in a temporary directory. +- The function strips leading and trailing triple-backtick code fences from the LLM response before validation. +- The `provider` parameter is passed to the prompt rendering and influences the generated HCL (e.g., AWS vs. Azure resources). +- The function defaults to "anthropic" as the LLM provider when `PICKLED_LLM_PROVIDER` is not set and `PICKLED_IAC_LLM_FACTORY` is not used. + +## Inventory references + +- Arguments: +- `user_story` (required): +- `provider` (optional): +- `output` (optional): +- Related gates: IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states "Draft a Terraform module from a user story" but omits the iterative validation-and-retry behavior: the function validates the generated HCL up to 3 times and incorporates error feedback into subsequent generation attempts. +- Docstring drift: The docstring does not mention the optional `output` parameter or the dual behavior of writing to a file versus printing to stdout. +- Docstring drift: The docstring does not mention the `provider` parameter, which is required and directly influences the generated infrastructure code. +- Docstring drift: The docstring does not describe any of the error conditions (missing IaC tools, invalid LLM configuration, validation failures after retries, or malformed factory environment variable). +- Docstring drift: The docstring does not clarify that both Terraform and OpenTofu are supported, with automatic detection of the available binary. + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_iac_iacambiguitygate.story.md b/dogfood/mining-output/stories/pickled_iac_iacambiguitygate.story.md new file mode 100644 index 0000000..b7527bd --- /dev/null +++ b/dogfood/mining-output/stories/pickled_iac_iacambiguitygate.story.md @@ -0,0 +1,79 @@ +# Story: IaCAmbiguityGate.run + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-iac +- **Surface id:** pickled_iac_iacambiguitygate +- **Code depth:** callgraph | **Units read:** 2 | **Unresolved:** 5 + +## Context + +This surface is a quality gate that evaluates Infrastructure-as-Code (Terraform) artifacts against a user story to detect ambiguities. It is used within a pipeline or workflow where Terraform modules are validated against business requirements. The caller provides an IaCArtifact containing Terraform HCL and a context dictionary with a user story, expecting a GateResult that indicates whether the implementation has potential ambiguities that need attention. + +## What the target does today + +The surface accepts three parameters: `target` (an object), and an optional `context` keyword parameter (a dictionary mapping strings to any type or None). + +**Type validation:** +The surface requires `target` to be an instance of `IaCArtifact`. If not, it returns a FAIL verdict with a note indicating the actual type received. + +The surface requires `context` (or an empty dict if None) to contain a "user_story" key whose value is a non-empty string (after stripping whitespace). If missing, not a string, or empty after stripping, it returns a FAIL verdict with a note stating the requirement. + +**LLM interaction:** +When validation passes, the surface renders a template (delegated to `self._template.render`) using the user story and the `content` attribute of the target artifact. It then sends the rendered prompt to an LLM via `complete_prompt` from `pickled_core.llm.turns`, requesting JSON-only output with no markdown fences. + +**Response parsing:** +The surface attempts to parse the LLM response as a JSON object. The parsing logic: +- Strips whitespace from the response +- If the response starts with "```", it attempts to extract content between the first and last "```" delimiters +- Extracts the substring between the first `{` and last `}` characters (inclusive) +- Parses this substring as JSON + +If JSON parsing fails or produces a non-dictionary, the surface returns FAIL with note "LLM returned malformed JSON". + +If the parsed JSON lacks an "ambiguities" key or its value is not a list, the surface returns FAIL with note 'LLM JSON missing list field "ambiguities"'. + +**Verdict determination:** +- If the "ambiguities" list is empty, returns PASS with note "No ambiguities reported." +- If the "ambiguities" list is non-empty, returns WARN verdict with the list converted to a tuple in the `findings` field and a note indicating the count of ambiguities + +All returned GateResult objects include `gate_name` set to `self.name`. + +## What we want to verify + +- Surface returns FAIL verdict when target is not an IaCArtifact instance +- Surface returns FAIL verdict when context is None and default dict has no "user_story" key +- Surface returns FAIL verdict when context["user_story"] is not a string +- Surface returns FAIL verdict when context["user_story"] is an empty or whitespace-only string +- Surface renders template with user_story and target.content attributes +- Surface calls complete_prompt with system message requesting JSON without markdown +- Surface returns FAIL verdict when LLM response cannot be parsed as JSON +- Surface returns FAIL verdict when parsed JSON is not a dictionary +- Surface returns FAIL verdict when parsed JSON lacks "ambiguities" key +- Surface returns FAIL verdict when "ambiguities" value is not a list +- Surface returns PASS verdict when "ambiguities" list is empty +- Surface returns WARN verdict when "ambiguities" list contains one or more items +- Surface includes findings as tuple when verdict is WARN +- Surface includes gate name in all returned GateResult objects +- Surface handles LLM response wrapped in markdown code fences (starting with "```") +- Surface extracts JSON between first '{' and last '}' after stripping/unwrapping + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: IaCAmbiguityGate.run +- Related ADRs: +- ADR 0005: `pickled-spec mine` staged mining pipeline — Accepted + +## Open questions + +- Docstring drift: Docstring describes surface as "LLM critic for Terraform modules" but does not mention the requirement for a "user_story" in the context parameter, which is a mandatory input that causes FAIL if absent +- Docstring drift: Docstring does not mention the return type (GateResult) or the possible verdict values (PASS, WARN, FAIL) +- Docstring drift: Docstring does not mention that the surface specifically checks for "ambiguities" in the LLM response +- Docstring drift: Docstring does not describe the validation requirements for the target parameter (must be IaCArtifact) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_iac_mcp.story.md b/dogfood/mining-output/stories/pickled_iac_mcp.story.md new file mode 100644 index 0000000..a0704a1 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_iac_mcp.story.md @@ -0,0 +1,43 @@ +# Story: mcp + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-iac +- **Surface id:** pickled_iac_mcp +- **Code depth:** callgraph | **Units read:** 1 | **Unresolved:** 0 + +## Context + +This is a CLI command group entry point for MCP (Model Context Protocol) server-related commands in the pickled-iac package. It serves as a parent command under which subcommands for MCP server operations are grouped. Callers are typically CLI users or the Click framework invoking this command from the command-line interface. Related surfaces include IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run, and run_all, suggesting this command group may organize gates or server operations under the MCP namespace. + +## What the target does today + +The surface is a no-op command group that accepts no parameters and produces no direct output or side effects. When invoked, it simply returns None. This command serves purely as a Click command group container; its purpose is to organize subcommands under the "mcp" namespace rather than to perform any action itself. + +If invoked directly without subcommands (assuming Click's default group behavior), the caller would observe either help text display or an error indicating that a subcommand is required, depending on Click's configuration. The function body itself performs no validation, raises no exceptions, and initiates no side effects. + +## What we want to verify + +- When invoked programmatically, the function returns None +- The function accepts no arguments +- The function raises no exceptions when called +- The function performs no file I/O, network operations, or state mutations +- The function executes and completes immediately without blocking +- When used as a Click command group, it organizes subcommands under the "mcp" namespace + +## Inventory references + +- Arguments: +- (none) +- Related gates: IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states "MCP server commands" (plural) but the code implements only an empty container function with no commands, operations, or delegation to any server functionality; the actual server commands must be registered elsewhere as subcommands not visible in this surface + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_iac_plan_cmd.story.md b/dogfood/mining-output/stories/pickled_iac_plan_cmd.story.md new file mode 100644 index 0000000..8628386 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_iac_plan_cmd.story.md @@ -0,0 +1,46 @@ +# Story: plan-cmd + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-iac +- **Surface id:** pickled_iac_plan_cmd + +## Context + +This CLI command is used by infrastructure engineers and DevOps practitioners to execute Terraform plan operations and capture the resulting plan output in JSON format. The command is part of the pickled-iac package's workflow for analyzing infrastructure-as-code changes before applying them. Users invoke this command to generate structured plan data that can then be fed into various gates (IaCAmbiguityGate, PlanDiffGate, SecurityBaselineGate) for validation and safety checks. + +## What the target does today + +Run terraform plan and write JSON to *output*. + +The command accepts two required parameters: a Terraform directory path (`tf_dir`) where the Terraform configuration resides, and an output path (`output`) where the JSON-formatted plan results will be written. + +## What we want to verify + +- Command accepts `tf_dir` argument specifying a directory containing Terraform configuration files +- Command accepts `output` argument specifying a file path for the resulting JSON +- Command executes `terraform plan` in the specified `tf_dir` directory +- Command captures the Terraform plan output in JSON format +- Command writes the JSON-formatted plan data to the file path specified by `output` +- Command creates the output file if it does not exist +- Command exits with appropriate status code reflecting success or failure of the terraform plan operation +- Generated JSON output is valid and parseable JSON +- Generated JSON output can be consumed by related gates (IaCAmbiguityGate, PlanDiffGate, SecurityBaselineGate) + +## Inventory references + +- Arguments: +- `tf_dir` (required): +- `output` (required): +- Related gates: IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_iac_plandiffgate.story.md b/dogfood/mining-output/stories/pickled_iac_plandiffgate.story.md new file mode 100644 index 0000000..7ed2fb1 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_iac_plandiffgate.story.md @@ -0,0 +1,85 @@ +# Story: PlanDiffGate.run + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-iac +- **Surface id:** pickled_iac_plandiffgate +- **Code depth:** callgraph | **Units read:** 2 | **Unresolved:** 0 + +## Context + +This gate is used in infrastructure-as-code (IaC) pipelines to compare two Terraform plan JSON outputs (base vs. head) and determine whether the detected changes should pass, warn, or fail based on the type of actions involved. Callers provide the head plan as the `target` and the base plan via the `context` dictionary under the key "base_plan". The gate helps enforce safety policies by flagging destructive changes (delete, replace) as failures and non-destructive changes (create, update, read, no-op) as warnings. + +## What the target does today + +**Inputs:** +- `target`: Expected to be a dict representing the head Terraform plan JSON. If not a dict, the gate immediately fails. +- `context`: Optional dict that must contain a "base_plan" key with a dict value representing the base Terraform plan JSON. If absent, None, or not a dict, the gate fails. + +**Processing:** +The gate extracts resource change information from both plans by looking for a top-level "resource_changes" key (expected to be a list). For each resource change entry (which must be a dict), it reads: +- "address": the resource address (coerced to string) +- "change.actions": a list of action strings + +Non-dict entries in "resource_changes" are silently skipped. Missing or null "resource_changes", "change", or "actions" fields are tolerated (treated as empty). + +The gate then compares changes between base and head: +- Resources present in head but not in base (and having non-empty actions) are recorded as findings with empty base actions. +- Resources present in both plans with different action lists are recorded as findings. +- Resources only in base (not in head) are ignored. + +**Verdict logic:** +- If no findings exist AND no actions are detected across all head changes, verdict is PASS with notes "No plan changes between base and head." +- If any action is "delete" or "replace", verdict is FAIL. +- If all detected actions are in the set {"create", "update", "read", "no-op"}, verdict is WARN. +- Otherwise (any action outside the above sets), verdict is WARN. + +**Outputs:** +Returns a `GateResult` with: +- `gate_name`: the name of this gate instance +- `verdict`: one of PASS, WARN, or FAIL +- `findings`: a tuple of `PlanDiffFinding` objects (empty for PASS cases), each containing (address, base_actions_tuple, head_actions_tuple) +- `notes`: either an error message (for input validation failures), "No plan changes between base and head." (for PASS), or "{count} resource change(s) detected." (for WARN/FAIL) + +**Edge cases:** +- If "resource_changes" is missing, null, or not a list, it is treated as an empty list. +- If "change" is missing or not a dict, actions default to empty. +- If "actions" is missing or not a list, it is treated as empty. +- All actions are coerced to strings. +- The gate does not validate that action strings are valid Terraform actions. + +## What we want to verify + +- When `target` is not a dict, return FAIL verdict with notes indicating the actual type received +- When `context` is None or missing "base_plan" key, return FAIL verdict with notes about missing base_plan +- When "base_plan" in context is not a dict, return FAIL verdict +- When both plans have empty or missing "resource_changes", return PASS verdict with notes "No plan changes between base and head." +- When head plan contains a resource address not in base plan with non-empty actions, include a finding with empty base actions tuple +- When head plan contains a resource with different actions than base plan, include a finding showing both action tuples +- When resource exists only in base but not in head, do not generate any finding for that resource +- When any action in head changes is "delete" or "replace", return FAIL verdict +- When all actions in head changes are from {"create", "update", "read", "no-op"}, return WARN verdict +- When findings exist, notes should state "{count} resource change(s) detected." where count matches the findings length +- When "resource_changes" contains non-dict entries, skip those entries without error +- When "change" is missing or null, treat actions as empty list +- When "actions" is not a list, treat it as empty +- All action strings in findings are converted to tuples in the order they appear in the source lists + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: PlanDiffGate.run +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states "Compare two terraform plan JSON outputs" but does not describe the return value shape (GateResult with verdict, findings, notes), the specific verdict rules (FAIL for delete/replace, WARN for safe changes), or the input validation behavior (type checking and required context key) +- Docstring drift: The docstring does not mention that resources present only in base (but not in head) are ignored in the comparison +- Docstring drift: The docstring does not specify that the base plan must be passed via context dict with key "base_plan" rather than as a direct parameter + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_iac_run_all.story.md b/dogfood/mining-output/stories/pickled_iac_run_all.story.md new file mode 100644 index 0000000..4fa20e6 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_iac_run_all.story.md @@ -0,0 +1,88 @@ +# Story: run_all + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-iac +- **Surface id:** pickled_iac_run_all +- **Code depth:** callgraph | **Units read:** 6 | **Unresolved:** 0 + +## Context + +This is a top-level gate orchestrator for Infrastructure-as-Code (IaC) validation in the pickled-iac package. It is invoked by a CI/CD pipeline or quality gate system to verify Terraform or OpenTofu configurations before deployment. The function expects to be called with a project root directory and returns a list of gate results covering both structural validation and security baseline checks. It is designed to fail gracefully when tools are missing, treating missing dependencies as warnings rather than hard failures where appropriate. + +## What the target does today + +Accepts a single argument `workdir` (Path or str) representing the project root directory and returns a list of GateResult objects. + +**Directory structure requirements:** +- Resolves `workdir` to an absolute path and checks for an `infra/` subdirectory +- If `infra/` does not exist, returns a single-element list containing a GateResult with gate_name="iac.infra", verdict=WARN, and notes="no infra/ directory" +- If `infra/` exists, proceeds with validation and security checks + +**Terraform/OpenTofu validation (iac.validate gate):** +- Attempts to run `terraform validate -json` (or OpenTofu equivalent) on the `infra/` directory +- If the IaC binary (terraform/tofu) is not found on PATH, returns a GateResult with verdict=WARN and notes containing the IaCToolMissingError message +- If the binary is found but initialization or validation fails with an unexpected exception, returns a GateResult with verdict=FAIL and notes containing the exception message +- On successful validation execution, returns a GateResult with: + - verdict=PASS if the validation reports valid=true + - verdict=FAIL if the validation reports valid=false + - notes containing semicolon-separated diagnostic messages, or "ok" if no diagnostics are present + +**Terraform initialization:** +- Automatically initializes the Terraform directory (runs `terraform init -input=false -backend=false`) if `.terraform/` does not exist +- Uses environment variable TF_IN_AUTOMATION=1 for all subprocess calls +- Raises RuntimeError if initialization fails + +**Security baseline scan (delegated to SecurityBaselineGate):** +- Always attempts a Trivy security scan via SecurityBaselineGate().run(infra) +- If `trivy` is not found on PATH, returns a GateResult with verdict=PASS and notes="trivy not found on PATH — security scan skipped" +- If `trivy` is found, runs `trivy config --format json --severity HIGH,CRITICAL --quiet` +- On trivy execution errors (non-0/1 return codes with no stdout), returns verdict=WARN with error details +- On JSON parse errors, returns verdict=WARN with notes="trivy returned non-JSON output" +- On successful scan: + - verdict=FAIL with findings tuple of CRITICAL titles if any CRITICAL severity misconfigurations are found + - verdict=WARN with findings tuple of HIGH titles if any HIGH severity misconfigurations are found (and no CRITICAL) + - verdict=PASS with notes="No HIGH or CRITICAL findings." if no HIGH or CRITICAL findings exist + +**Return value structure:** +- Always returns a list of GateResult objects +- When `infra/` is missing: returns 1 result +- When `infra/` exists: returns 2 results (iac.validate + security baseline) +- Each GateResult includes gate_name, verdict, and notes; security results may also include findings tuple + +## What we want to verify + +- When workdir contains no infra/ subdirectory, returns a single GateResult with verdict=WARN and gate_name="iac.infra" +- When infra/ exists and terraform/tofu is not on PATH, iac.validate result has verdict=WARN with IaCToolMissingError message in notes +- When infra/ exists and terraform validate succeeds with valid=true, iac.validate result has verdict=PASS +- When infra/ exists and terraform validate succeeds with valid=false, iac.validate result has verdict=FAIL and diagnostics in notes +- When terraform validate raises an unexpected exception, iac.validate result has verdict=FAIL with exception message in notes +- When trivy is not found on PATH, security baseline result has verdict=PASS with skip message in notes +- When trivy finds CRITICAL severity misconfigurations, security baseline result has verdict=FAIL with findings tuple +- When trivy finds HIGH severity misconfigurations (no CRITICAL), security baseline result has verdict=WARN with findings tuple +- When trivy finds no HIGH or CRITICAL misconfigurations, security baseline result has verdict=PASS +- When trivy returns non-JSON output, security baseline result has verdict=WARN +- Return value is always a list, never None or a single GateResult +- All terraform/tofu subprocess calls include TF_IN_AUTOMATION=1 environment variable +- If .terraform/ does not exist in infra/, terraform init is executed before validate + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: Docstring claims the function performs "``terraform validate``" but does not mention that it auto-detects and supports OpenTofu (tofu) as an alternative binary +- Docstring drift: Docstring states "optional Trivy scan" but the code always attempts the security scan; it is only skipped when trivy is not found on PATH, not based on any configuration option or parameter +- Docstring drift: Docstring does not mention the function returns different result counts (1 vs 2 GateResult objects) depending on whether infra/ exists +- Docstring drift: Docstring does not describe the warning behavior when infra/ is missing, terraform/tofu is missing, or trivy is missing +- Docstring drift: Docstring does not mention automatic terraform initialization when .terraform/ directory is absent + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_iac_scan.story.md b/dogfood/mining-output/stories/pickled_iac_scan.story.md new file mode 100644 index 0000000..7dba1da --- /dev/null +++ b/dogfood/mining-output/stories/pickled_iac_scan.story.md @@ -0,0 +1,81 @@ +# Story: scan + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-iac +- **Surface id:** pickled_iac_scan +- **Code depth:** callgraph | **Units read:** 2 | **Unresolved:** 0 + +## Context + +A CLI command surface that performs security scanning on Terraform configuration directories. Called by operators or CI/CD pipelines to validate infrastructure-as-code against security baselines. The command runs Trivy config scanning, returns structured JSON output to stdout, and uses exit codes to signal pass/fail status. Designed to fail builds when critical security issues are found while allowing graceful degradation when the scanner tool is unavailable. + +## What the target does today + +**Acceptance:** +- Accepts a single positional argument `tf_dir` typed as `Path` representing a Terraform configuration directory +- Does not validate that `tf_dir` exists, is a directory, or contains Terraform files before passing to the security gate + +**Returns and output:** +- Always writes a JSON object to stdout (via `click.echo`) with three fields: + - `verdict`: string representation of the scan result ("PASS", "WARN", or "FAIL") + - `notes`: human-readable explanation of the verdict + - `findings`: list of finding titles (populated only when issues are detected) +- The JSON is formatted with 2-space indentation + +**Exit behavior:** +- Exits with code 2 (via `SystemExit(2)`) when verdict is `FAIL` +- Exits with code 0 (implicit) when verdict is `PASS` or `WARN` + +**Security scanning logic:** +- If `trivy` executable is not found on PATH: returns `PASS` verdict with note "trivy not found on PATH — security scan skipped" +- If `tf_dir` is not a Path instance: returns `FAIL` verdict with type mismatch note +- When Trivy is available, invokes: `trivy config --format json --severity HIGH,CRITICAL --quiet` +- If Trivy exits with code other than 0 or 1 AND produces no stdout: returns `WARN` verdict with stderr content +- If Trivy output is not valid JSON: returns `WARN` verdict with note "trivy returned non-JSON output" +- Parses Trivy JSON report looking for `Results[].Misconfigurations[]` entries +- Extracts `Severity` and `Title` (falling back to `ID` or "finding") from each misconfiguration +- If any CRITICAL severity findings exist: returns `FAIL` verdict with findings list and count +- If any HIGH severity findings exist (and no CRITICAL): returns `WARN` verdict with findings list and count +- If no HIGH or CRITICAL findings: returns `PASS` verdict + +**Error handling:** +- Non-zero Trivy exit codes are tolerated; only codes other than 0 or 1 combined with empty stdout trigger warnings +- JSON decode errors are caught and converted to `WARN` verdicts +- Non-dict entries in Results or Misconfigurations arrays are silently skipped +- Missing or null nested structures are handled gracefully with `or []` guards + +## What we want to verify + +- When `trivy` is not on PATH, output verdict is "PASS" and notes indicate scan was skipped, exit code is 0 +- When `tf_dir` is not a Path instance, output verdict is "FAIL" and notes mention type mismatch, exit code is 2 +- When Trivy is available and finds no HIGH/CRITICAL issues, output verdict is "PASS", findings list is empty, exit code is 0 +- When Trivy reports CRITICAL findings, output verdict is "FAIL", findings list contains CRITICAL issue titles, exit code is 2 +- When Trivy reports only HIGH findings (no CRITICAL), output verdict is "WARN", findings list contains HIGH issue titles, exit code is 0 +- When Trivy returns non-JSON output, output verdict is "WARN" and notes indicate non-JSON output, exit code is 0 +- When Trivy fails with non-0/1 exit code and empty stdout, output verdict is "WARN" and notes contain stderr content, exit code is 0 +- Output is always valid JSON with exactly three keys: verdict, notes, findings +- Findings list only contains titles/IDs from HIGH or CRITICAL severity misconfigurations +- Exit code is 2 if and only if verdict is "FAIL" + +## Inventory references + +- Arguments: +- `tf_dir` (required): +- Related gates: IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: Docstring says "optional; skips if trivy missing" but does not document that the command returns structured JSON output to stdout +- Docstring drift: Docstring does not mention that the command exits with code 2 on security failures (FAIL verdict) +- Docstring drift: Docstring does not indicate that WARN verdicts (HIGH findings) allow the command to exit successfully (code 0) +- Docstring drift: Docstring omits that invalid input types are rejected with FAIL verdict +- Docstring drift: Docstring does not document the severity filtering behavior (HIGH and CRITICAL only) +- Docstring drift: Docstring describes it as a "config scan" which aligns with the implementation, but omits that it specifically targets security misconfigurations + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_iac_securitybaselinegate.story.md b/dogfood/mining-output/stories/pickled_iac_securitybaselinegate.story.md new file mode 100644 index 0000000..025420b --- /dev/null +++ b/dogfood/mining-output/stories/pickled_iac_securitybaselinegate.story.md @@ -0,0 +1,74 @@ +# Story: SecurityBaselineGate.run + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-iac +- **Surface id:** pickled_iac_securitybaselinegate +- **Code depth:** callgraph | **Units read:** 1 | **Unresolved:** 0 + +## Context + +This gate is part of a quality-assurance pipeline for Infrastructure-as-Code (Terraform) configurations. Callers invoke `run` to verify that Terraform code meets security baseline requirements by scanning for HIGH and CRITICAL severity misconfigurations. It is designed to be optional (can gracefully degrade) and integrates with the Trivy security scanner. + +## What the target does today + +**Accepts:** +- `target`: an object that must be a `Path` instance pointing to a Terraform directory +- `context`: an optional dictionary (accepted but ignored) + +**Returns:** +A `GateResult` object with fields `gate_name`, `verdict`, and optionally `findings` and `notes`. The verdict is one of `PASS`, `WARN`, or `FAIL`. + +**Rejection and error modes:** +1. If `target` is not a `Path` instance, returns `FAIL` verdict with a note describing the type mismatch. +2. If the `trivy` executable is not found on the system PATH, returns `PASS` verdict with a note explaining the scan was skipped (graceful degradation). +3. If `trivy` exits with a return code other than 0 or 1 and produces no stdout, returns `WARN` verdict with stderr captured in the notes. +4. If `trivy` output is not valid JSON, returns `WARN` verdict with a note about non-JSON output. + +**Security scan behavior:** +- Invokes the external `trivy` command-line tool with arguments: `config --format json --severity HIGH,CRITICAL --quiet` +- Parses the JSON output looking for a top-level `"Results"` array, then within each result a `"Misconfigurations"` array +- Extracts severity (`"Severity"` field, case-insensitive comparison) and title (`"Title"` field, falling back to `"ID"` or the string `"finding"`) +- Classifies findings as CRITICAL or HIGH based on severity + +**Verdict logic:** +- Returns `FAIL` if any CRITICAL findings are detected; the `findings` tuple contains all CRITICAL titles, and notes report the count. +- Returns `WARN` if no CRITICAL findings but one or more HIGH findings are detected; the `findings` tuple contains all HIGH titles, and notes report the count. +- Returns `PASS` if no HIGH or CRITICAL findings are detected, with a note confirming this. + +**Side effects:** +- Executes an external subprocess (`trivy`), which may perform network requests (e.g., to download vulnerability databases) or file I/O depending on Trivy's configuration. +- Captures stdout and stderr from the subprocess; does not stream or log them separately. + +## What we want to verify + +- When `target` is not a `Path`, returns `GateResult` with `verdict=FAIL` and notes containing the actual type name. +- When `trivy` executable is not on PATH, returns `GateResult` with `verdict=PASS` and notes indicating scan was skipped. +- When `trivy` exits with return code other than 0 or 1 and produces empty stdout, returns `GateResult` with `verdict=WARN` and notes containing stderr content. +- When `trivy` output is not valid JSON, returns `GateResult` with `verdict=WARN` and notes about non-JSON output. +- When `trivy` reports one or more CRITICAL misconfigurations, returns `GateResult` with `verdict=FAIL`, `findings` containing titles, and notes reporting count. +- When `trivy` reports HIGH but no CRITICAL misconfigurations, returns `GateResult` with `verdict=WARN`, `findings` containing titles, and notes reporting count. +- When `trivy` reports no HIGH or CRITICAL misconfigurations, returns `GateResult` with `verdict=PASS` and notes confirming no findings. +- The `context` parameter is accepted but does not influence the result. +- `trivy` is invoked with arguments `config`, target path as string, `--format json`, `--severity HIGH,CRITICAL`, and `--quiet`. +- Non-dict entries in `"Results"` or `"Misconfigurations"` arrays are silently skipped. +- Severity strings are compared case-insensitively (uppercased). +- Title extraction falls back from `"Title"` to `"ID"` to the literal string `"finding"`. + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: SecurityBaselineGate.run +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states the scan is "optional in v0.1" but does not clarify that the gate passes automatically (not skips with a different status) when `trivy` is absent. The code reveals this graceful-degradation behavior: missing `trivy` yields `PASS`, not an error or a skip state. +- Docstring drift: The docstring describes the surface as "Run Trivy config scan" but omits all information about return values, verdict logic (FAIL for CRITICAL, WARN for HIGH, PASS otherwise), error handling (malformed JSON, subprocess errors), and input validation (type check on `target`). + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_iac_validate.story.md b/dogfood/mining-output/stories/pickled_iac_validate.story.md new file mode 100644 index 0000000..5a967af --- /dev/null +++ b/dogfood/mining-output/stories/pickled_iac_validate.story.md @@ -0,0 +1,41 @@ +# Story: validate + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-iac +- **Surface id:** pickled_iac_validate + +## Context + +CLI users invoke this command to run Terraform's built-in validate operation on a specified directory containing Terraform configuration files. This is typically used as part of a validation pipeline or gate system (as evidenced by related gates like IaCAmbiguityGate, PlanDiffGate, and SecurityBaselineGate) to ensure Terraform configurations are syntactically valid and internally consistent before planning or applying changes. + +## What the target does today + +Run terraform validate on a directory. + +The command accepts a required `tf_dir` parameter specifying the target directory and executes Terraform's validate command against that directory. + +## What we want to verify + +- Invoking `validate` with a valid `tf_dir` argument executes terraform validate on the specified directory +- The command accepts `tf_dir` as a required parameter +- The command fails or reports an error when `tf_dir` is not provided +- The command passes validation results (success or failure) back to the caller +- The command operates on the directory specified by `tf_dir` rather than the current working directory + +## Inventory references + +- Arguments: +- `tf_dir` (required): +- Related gates: IaCAmbiguityGate.run, PlanDiffGate.run, SecurityBaselineGate.run, run_all +- Related ADRs: +- ADR 0007: Code-aware stories and docstring drift detection — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_rules_check.story.md b/dogfood/mining-output/stories/pickled_rules_check.story.md new file mode 100644 index 0000000..63d37b2 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_rules_check.story.md @@ -0,0 +1,104 @@ +# Story: check + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-rules +- **Surface id:** pickled_rules_check +- **Code depth:** callgraph | **Units read:** 2 | **Unresolved:** 1 + +## Context + +This CLI command is invoked by users who want to verify that a Gherkin feature file (or set of feature files) satisfies a set of rules defined in a YAML ruleset. It reports coverage—whether required tags, scenario patterns, or other constraints defined in the ruleset are met by the feature file(s). The command is used in CI pipelines or local development to enforce behavioral documentation standards. + +## What the target does today + +**Input requirements:** +- Exactly one of `feature_path` or `feature_glob` must be provided; providing neither or both raises a ClickException. +- `ruleset` may be a built-in ruleset name (looked up from BUILTIN_RULESETS) or a file path; if a path is given and the file does not exist, raises ClickException. +- If `feature_glob` is provided, it is expanded using Python's glob module with recursive=True, and only files (not directories) matching the pattern are processed. If no files match, raises ClickException. +- If `feature_path` is provided, exactly one file is processed (the path itself). + +**Ruleset resolution:** +- If `ruleset` is in BUILTIN_RULESETS, the built-in ruleset is loaded via resolve_ruleset_name. +- Otherwise, `ruleset` is treated as a file path and must exist. +- The ruleset is loaded via load_ruleset. +- The short name for the ruleset is determined by: `ruleset_name` (lowercased) if provided, else the `ruleset` argument (lowercased) if it's a built-in, else the file stem (lowercased). + +**Feature parsing:** +- Feature files are parsed using PytestBddAdapter().parse_feature_file (delegated to unresolved collaborator; actual parsing behavior is opaque). +- Files are sorted before parsing. + +**Coverage checking:** +- If exactly one feature file is processed, coverage_gate is called with the parsed feature, the ruleset, and the short name. +- If multiple feature files are processed, coverage_gate_features is called with the list of parsed features, the ruleset, and the short name. This performs union coverage (a rule is satisfied if any file in the set satisfies it). + +**Reporting:** +- Output format is determined by `output_format` (case-insensitive): + - "json" → render_coverage_json + - Any other value → render_coverage_markdown +- The report includes the ruleset and the feature path(s). For single files, the path is the file path. For multiple files, the path label is a comma-separated list of sorted paths. +- If `quiet` is False and multiple files are processed, a message "Union coverage across N feature file(s)." is written to stderr. +- If `quiet` is True: + - Only a verdict line "PASS: checked N feature(s)" or "FAIL: checked N feature(s)" is printed to stdout. + - If `output` is provided, the full report is written to the file. +- If `quiet` is False: + - If `output` is provided, the report is written to the file and a confirmation message "Report written to " is written to stderr. + - If `output` is not provided, the report is written to stdout. + +**Exit behavior:** +- The gate result includes a Verdict. If the verdict is not Verdict.PASS, the process exits with status code 1. +- If the verdict is Verdict.PASS, the process exits with status code 0 (implicit). + +**Side effects:** +- Writes to stderr when not quiet and when output file is written. +- Writes to stdout (report or verdict line) unless quiet with output file. +- Writes to the specified file if `output` is provided. +- Exits with status code 1 on coverage failure. + +## What we want to verify + +- Providing neither `feature_path` nor `feature_glob` raises ClickException with message "Provide --feature or --feature-glob". +- Providing both `feature_path` and `feature_glob` raises ClickException with message "Use only one of --feature or --feature-glob". +- Providing a `ruleset` path that does not exist raises ClickException with message "Rule set file not found: ". +- Providing a `feature_glob` that matches no files raises ClickException with message "No feature files matched". +- When `feature_glob` matches multiple files, only file paths (not directories) are processed. +- Feature files are processed in sorted order. +- When processing a single feature file, coverage_gate is called (not coverage_gate_features). +- When processing multiple feature files, coverage_gate_features is called and stderr receives a message about union coverage (unless quiet). +- When `output_format` is "json" (case-insensitive), the report uses render_coverage_json. +- When `output_format` is not "json", the report uses render_coverage_markdown. +- When `quiet` is True, stdout receives only "PASS: checked N feature(s)" or "FAIL: checked N feature(s)". +- When `quiet` is True and `output` is provided, the full report is written to the file. +- When `quiet` is False and `output` is provided, stderr receives "Report written to ". +- When `quiet` is False and `output` is None, stdout receives the full report. +- When the gate result verdict is not Verdict.PASS, the process exits with status code 1. +- When the gate result verdict is Verdict.PASS, the process exits with status code 0. +- When `ruleset` is a built-in name, the short name defaults to the built-in name (lowercased) if `ruleset_name` is not provided. +- When `ruleset` is a file path, the short name defaults to the file stem (lowercased) if `ruleset_name` is not provided. +- When `ruleset_name` is provided, it is used (lowercased) as the short name regardless of whether `ruleset` is built-in or a path. + +## Inventory references + +- Arguments: +- `ruleset` (required): Built-in rule set name or path to a YAML rule set file. +- `feature_path` (optional): Single Gherkin feature file to analyse. +- `feature_glob` (optional): Glob of feature files; multiple matches are checked as one union (strict rules must appear across the set, not in each file). +- `ruleset_name` (optional): Short name for tag prefix (default: built-in name or file stem). +- `output_format` (optional): Report output format. +- `output` (optional): Write the report to this path. Default: stdout. +- `quiet` (optional): Suppress report on stdout; print only the verdict line. If --output is set, the report is still written to the file. +- Related gates: coverage_gate, coverage_gate_features, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states "Check feature coverage against a YAML rule set" but does not mention that the command can fail with multiple specific error conditions (missing input, non-existent ruleset file, no matching feature files). +- Docstring drift: The docstring does not describe the command's exit behavior (exits with status 1 on coverage failure). +- Docstring drift: The docstring does not specify that the command can process multiple feature files via glob and that union coverage semantics apply in that case. +- Docstring drift: The docstring does not mention the quiet mode, output file, output format, or ruleset_name parameters and their effects on behavior. + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_rules_coverage_gate.story.md b/dogfood/mining-output/stories/pickled_rules_coverage_gate.story.md new file mode 100644 index 0000000..87c60bc --- /dev/null +++ b/dogfood/mining-output/stories/pickled_rules_coverage_gate.story.md @@ -0,0 +1,89 @@ +# Story: coverage_gate + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-rules +- **Surface id:** pickled_rules_coverage_gate +- **Code depth:** callgraph | **Units read:** 3 | **Unresolved:** 0 + +## Context + +This gate is used by test automation or compliance tooling to verify that a Gherkin feature file adequately covers the rules defined in a ruleset. Callers provide a single Feature and a RuleSet, along with a short name used to identify rule references in scenario tags. The gate determines whether all strict rules are referenced by at least one scenario and whether any unknown rule references exist. It produces a coverage report that includes both the gate verdict and traceability information for documentation or audit purposes. + +## What the target does today + +**Inputs accepted:** +- `feature`: a Feature object representing a Gherkin feature file +- `ruleset`: a RuleSet object containing zero or more rules, each with an `id`, `enforcement` level (e.g., "strict", "advisory", "informational"), and other metadata +- `ruleset_short_name` (keyword-only): a string used to match scenario tags in the format `:` + +**Return value:** +Returns a `CoverageReport` object containing: +- `referenced_rules`: tuple of rules from the ruleset that were referenced by at least one scenario tag +- `unreferenced_rules`: tuple of rules from the ruleset that were not referenced by any scenario tag +- `unknown_references`: sorted tuple of (ruleset_name, rule_id) pairs for tags that reference rule IDs not found in the provided ruleset +- `gate_result`: a `GateResult` with: + - `gate_name`: always "rules.coverage" + - `verdict`: `Verdict.PASS` if all strict rules are referenced AND no unknown references exist; otherwise `Verdict.FAIL` + - `findings`: always an empty tuple + - `notes`: a human-readable string describing the pass/fail reason + - `traces`: tuple of `Trace` objects, one per referenced rule, indicating that the feature "implements" each referenced rule with "asserted" confidence + +**Reference extraction:** +- Scenario tags are extracted from the feature and filtered to those matching the `ruleset_short_name` +- Tags are normalized (the "@" prefix is removed during parsing/modeling, as mentioned in the docstring) +- A rule is considered "referenced" if its `id` appears in at least one matching tag + +**Pass/fail logic:** +- The gate passes if and only if: + 1. Every rule with `enforcement == "strict"` is referenced by at least one scenario, AND + 2. No scenario tags reference rule IDs that do not exist in the ruleset (no unknown references) +- Advisory and informational rules may remain unreferenced without causing failure +- If any strict rule is unreferenced OR any unknown references exist, the gate fails + +**Artifact reference in traces:** +- If the feature has a `path` attribute that is truthy, that path is used as the `artifact_ref` in all generated traces +- If the feature has no path or the path is falsy, the string `""` is used instead + +**Notes content:** +- On pass: states that all strict rules are referenced and no unknown tags exist +- On fail: enumerates the count of unreferenced strict rules and/or unknown references + +**No exceptions or validation:** +The surface does not perform validation on inputs (e.g., checking for None, validating ruleset structure, or ensuring ruleset_short_name is non-empty). Invalid inputs would propagate as exceptions from delegated calls. + +## What we want to verify + +- When feature contains no scenarios, the gate returns a CoverageReport with empty referenced_rules, all rules in unreferenced_rules, and verdict FAIL if any rule has enforcement="strict" +- When all strict rules are referenced by scenario tags and no unknown references exist, the gate returns verdict PASS +- When at least one strict rule is unreferenced, the gate returns verdict FAIL regardless of whether advisory or informational rules are referenced +- When scenario tags reference rule IDs not present in the ruleset, the gate returns verdict FAIL and includes those references in unknown_references as sorted tuples +- When only advisory or informational rules are unreferenced and no unknown references exist, the gate returns verdict PASS +- The returned CoverageReport.gate_result.gate_name is always "rules.coverage" +- The returned CoverageReport.gate_result.findings is always an empty tuple +- Each rule appearing in referenced_rules has a corresponding Trace in gate_result.traces with relation="implements" and confidence="asserted" +- When feature.path is None or empty, the artifact_ref in all traces is "" +- When feature.path is truthy, the artifact_ref in all traces equals feature.path +- The notes field on pass states all strict rules are referenced and no unknown tags exist +- The notes field on fail includes the count of unreferenced strict rules if any exist +- The notes field on fail includes the count of unknown references if any exist +- The unknown_references tuple is sorted +- A rule is counted as referenced if it appears in any scenario tag with the pattern : + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: coverage_gate +- Related ADRs: +- ADR 0004: Multi-ruleset workspace configuration — Accepted + +## Open questions + +- Docstring drift: The docstring states "normalized without `@` in the model" but does not explain where this normalization occurs; the code delegates tag extraction to `extract_references`, so the normalization behavior is not observable in the shown surface +- Docstring drift: The docstring does not mention that the gate will fail if unknown reference tags exist; it only describes the strict rule coverage requirement for passing + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_rules_coverage_gate_features.story.md b/dogfood/mining-output/stories/pickled_rules_coverage_gate_features.story.md new file mode 100644 index 0000000..e08f5bc --- /dev/null +++ b/dogfood/mining-output/stories/pickled_rules_coverage_gate_features.story.md @@ -0,0 +1,94 @@ +# Story: coverage_gate_features + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-rules +- **Surface id:** pickled_rules_coverage_gate_features +- **Code depth:** callgraph | **Units read:** 4 | **Unresolved:** 2 + +## Context + +`coverage_gate_features` is a gate function used to verify that a set of Gherkin feature files references all strict rules from a given ruleset. It extracts reference tags (e.g., `@REF:ruleset:rule-id`) from scenario tags across one or more features and compares them against the ruleset. It returns a coverage report with a pass/fail verdict, referenced/unreferenced rule lists, and traceability information. Callers use this to enforce that all mandatory (strict enforcement) rules have corresponding test scenarios in the feature suite. + +## What the target does today + +**Inputs:** +- `features`: A sequence of Feature objects to scan for rule references in their scenario tags +- `ruleset`: A RuleSet object containing rules with `id`, `enforcement`, and `description` attributes; must provide a `find(rule_id: str)` method that returns a Rule or None +- `ruleset_short_name`: A string filter used to match reference tags (case-insensitive on the ruleset side) +- `artifact_ref` (optional): A string identifying the artifact(s) being checked; defaults to a comma-separated list of feature paths or `""` if no paths are available + +**Returns:** +A `CoverageReport` object containing: +- `referenced_rules`: A tuple of Rule objects from the ruleset that are referenced in the feature scenarios +- `unreferenced_rules`: A tuple of Rule objects from the ruleset that are not referenced +- `unknown_references`: A sorted tuple of (ruleset_name, rule_id) pairs for tags that do not match any rule in the provided ruleset +- `gate_result`: A `GateResult` object with: + - `gate_name`: Always `"rules.coverage"` + - `verdict`: `PASS` if all strict rules are referenced and no unknown references exist; otherwise `FAIL` + - `findings`: Always an empty tuple + - `notes`: A human-readable summary of the result + - `traces`: A tuple of `Trace` objects, one per referenced rule, documenting the "implements" relationship between the artifact and each rule + +**Pass/Fail Logic:** +- The gate passes if and only if: + 1. All rules with `enforcement == "strict"` are referenced in at least one scenario tag, AND + 2. No unknown references exist (references to rule IDs not found in the ruleset) +- Rules with non-strict enforcement do not cause failure if unreferenced + +**Tag Extraction:** +- Scenario tags are parsed to extract references matching the `ruleset_short_name` filter +- For each extracted reference, the ruleset is queried using the rule ID +- If the rule ID is found, it is added to the referenced set +- If the rule ID is not found, the (ruleset_name, rule_id) pair is added to the unknown set + +**Trace Generation:** +- Each referenced rule produces a `Trace` object with: + - `source_reference` containing rule metadata from the ruleset (source_id, source_version, locator, description, active_from, applies_to, source_url) + - `artifact_kind`: Always `"feature"` + - `artifact_ref`: The provided or derived artifact reference + - `relation`: Always `"implements"` + - `confidence`: Always `"asserted"` + +**Notes Field:** +- On pass: `"All strict rules in {ruleset.source_id} are referenced; no unknown reference tags."` +- On fail: A semicolon-separated list describing: + - The count of strict unreferenced rules (if any) + - The count of unknown references (if any) + +## What we want to verify + +- When all rules have `enforcement != "strict"`, the gate passes regardless of whether they are referenced +- When at least one rule has `enforcement == "strict"` and is not referenced, the gate fails and that rule appears in `unreferenced_rules` +- When a scenario tag references a rule ID not present in the ruleset, the gate fails and the (ruleset_name, rule_id) appears in `unknown_references` +- When all strict rules are referenced and no unknown references exist, the gate passes with verdict `PASS` +- The `unknown_references` tuple is sorted lexicographically +- Each rule in `referenced_rules` produces exactly one `Trace` in the gate result +- Rules in `unreferenced_rules` do not produce traces +- When `artifact_ref` is None and features have no paths, the artifact_ref defaults to `""` +- When `artifact_ref` is None and features have paths, the artifact_ref is a comma-separated list of those paths +- The `findings` field in `gate_result` is always an empty tuple +- The `gate_name` is always `"rules.coverage"` +- A rule is considered referenced if any scenario tag across any feature in the sequence references its ID +- The same rule can be referenced multiple times across scenarios, but appears only once in `referenced_rules` and produces only one trace + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: coverage_gate_features +- Related ADRs: +- ADR 0004: Multi-ruleset workspace configuration — Accepted +- ADR 0005: `pickled-spec mine` staged mining pipeline — Accepted + +## Open questions + +- Docstring drift: The docstring states "union of scenario tags" but does not mention that only tags matching the `ruleset_short_name` filter are considered +- Docstring drift: The docstring does not describe the pass/fail criteria: that strict rules must all be referenced and no unknown references may exist +- Docstring drift: The docstring does not mention the `artifact_ref` parameter or its defaulting behavior +- Docstring drift: The docstring does not describe the return type structure (`CoverageReport`) or the contents of the gate result + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_rules_draft.story.md b/dogfood/mining-output/stories/pickled_rules_draft.story.md new file mode 100644 index 0000000..5ccc47e --- /dev/null +++ b/dogfood/mining-output/stories/pickled_rules_draft.story.md @@ -0,0 +1,66 @@ +# Story: draft + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-rules +- **Surface id:** pickled_rules_draft +- **Code depth:** callgraph | **Units read:** 8 | **Unresolved:** 3 + +## Context + +This CLI command is invoked by end users (developers, compliance engineers) who want to automatically generate a YAML rule-set document from a natural-language description. The user provides a brief (either as a file path or via stdin), metadata fields (short_name, source_id, applies_to, active_from), and optionally an output file path. The command leverages an LLM client to transform the brief into structured YAML that conforms to the pickled-rules schema. + +## What the target does today + +The command accepts six parameters: a brief (file path or '-'), short_name, source_id, applies_to, active_from, and an optional output path. When brief is '-', the command reads from stdin; otherwise it reads UTF-8 text from the specified file path. + +The command builds an LLM client using a factory specified by the PICKLED_RULES_LLM_FACTORY environment variable. If the LLM configuration is invalid, the command fails with a ClickException containing the configuration error message. + +The command constructs a prompt incorporating the brief text and metadata fields, then delegates to an LLM completion call (unresolved; model, max_tokens, temperature, and stop parameters are supplied but exact LLM behavior is unobservable). The LLM response is expected to contain YAML text and optionally a rationale section separated by a sentinel string. + +The command validates the generated YAML by attempting to load it as a rule set and checking for forbidden tokens in the lowercased YAML text. Validation warnings are collected but do not prevent output. + +The command emits the generated YAML to the specified output file (UTF-8 encoded) or to stdout if output is None. Rationale lines (if present) are written to stderr prefixed with "rationale: ". Validation warnings (if any) are written to stderr prefixed with "warning: ". + +The command exits with status 1 if any validation warnings are present, even though the YAML is still emitted. If an exception other than ClickException occurs during processing, the exception message is printed to stderr and the command exits with status 2. + +## What we want to verify + +- When brief is '-', the command reads from stdin; when brief is a file path, the command reads UTF-8 text from that file. +- When the LLM client cannot be built due to a configuration error, the command raises ClickException with the configuration error message. +- The command invokes the LLM client with a prompt containing the brief text, short_name, source_id, applies_to, and active_from values. +- The command attempts to validate the generated YAML by loading it as a rule set. +- The command checks the lowercased YAML text for forbidden tokens and produces warnings if any are found. +- When output is None, the generated YAML is written to stdout. +- When output is a Path, the generated YAML is written to that file with UTF-8 encoding. +- If the LLM response contains a rationale section (delimited by a sentinel), each rationale line is written to stderr prefixed with "rationale: ". +- Validation warnings are written to stderr prefixed with "warning: ". +- The command exits with status 1 if validation warnings are present, even if YAML was successfully emitted. +- The command exits with status 2 if a non-ClickException occurs, after printing the exception message to stderr. +- ClickExceptions are re-raised without being caught or transformed into SystemExit(2). + +## Inventory references + +- Arguments: +- `brief` (required): Brief file path or '-' for stdin. +- `short_name` (required): Ruleset short name for tagging. +- `source_id` (required): metadata.source_id value. +- `applies_to` (required): metadata.applies_to value. +- `active_from` (required): metadata.active_from (YYYY-MM-DD). +- `output` (optional): Write YAML to this path. Default: stdout. +- Related gates: coverage_gate, coverage_gate_features, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring claims the command "drafts a YAML rule set" but omits that validation warnings cause exit status 1 even when YAML is successfully generated and emitted. +- Docstring drift: The docstring does not mention that rationale output is written to stderr when present in the LLM response. +- Docstring drift: The docstring does not mention the command reads from stdin when brief is '-'. +- Docstring drift: The docstring does not mention the command validates the generated YAML for both schema conformance and forbidden tokens. +- Docstring drift: The docstring does not mention the two distinct failure modes: ClickException (configuration errors) versus general exceptions (exit status 2). + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_rules_list_rules.story.md b/dogfood/mining-output/stories/pickled_rules_list_rules.story.md new file mode 100644 index 0000000..35f8dcf --- /dev/null +++ b/dogfood/mining-output/stories/pickled_rules_list_rules.story.md @@ -0,0 +1,39 @@ +# Story: list-rules + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-rules +- **Surface id:** pickled_rules_list_rules + +## Context + +CLI users invoking `pickled-rules list-rules` to discover available rule identifiers from either a built-in rule set or a custom YAML file. This supports workflows where users need to reference specific rules by ID for filtering, reporting, or configuration purposes. The command accepts either a named built-in rule set or a file system path to a YAML rule set. + +## What the target does today + +Lists rule IDs from a YAML rule set. The command accepts a `ruleset` argument that can be either a built-in rule set name or a path to a YAML rule set file, and outputs the rule identifiers contained within that rule set. + +## What we want to verify + +- Invoked with a built-in rule set name, the command outputs rule IDs from that built-in set +- Invoked with a valid file path to a YAML rule set, the command outputs rule IDs from that file +- The output consists of rule IDs extracted from the specified rule set +- Invalid or non-existent rule set names/paths produce an appropriate error +- The command completes successfully (exit code 0) when given valid input + +## Inventory references + +- Arguments: +- `ruleset` (required): Built-in rule set name or path to a YAML rule set file. +- Related gates: coverage_gate, coverage_gate_features, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_rules_mcp.story.md b/dogfood/mining-output/stories/pickled_rules_mcp.story.md new file mode 100644 index 0000000..83a11c9 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_rules_mcp.story.md @@ -0,0 +1,43 @@ +# Story: mcp + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-rules +- **Surface id:** pickled_rules_mcp +- **Code depth:** callgraph | **Units read:** 1 | **Unresolved:** 0 + +## Context + +This is a CLI command group entry point for MCP (Model Context Protocol) server-related commands in the pickled-rules package. It serves as a parent command that organizes subcommands related to MCP server operations. CLI users or automation scripts would invoke this to access MCP server functionality, likely through subcommands not visible in the provided root function. + +## What the target does today + +The `mcp` function is a no-op command group entry point that accepts no arguments and returns None. When invoked directly (without subcommands), it performs no operations and produces no side effects. Its purpose is solely to serve as an organizational container for Click CLI subcommands. The function immediately returns None without executing any logic, validation, or state changes. + +As a Click command group (inferred from context with related gates), calling this function directly has no observable effect beyond normal function entry/exit. Any actual MCP server functionality would be implemented in child subcommands attached to this group via Click's command group mechanism. + +## What we want to verify + +- Calling `mcp()` returns None +- Calling `mcp()` raises no exceptions +- Calling `mcp()` performs no I/O operations +- Calling `mcp()` modifies no global state +- The function signature accepts zero parameters +- The function completes synchronously without blocking operations + +## Inventory references + +- Arguments: +- (none) +- Related gates: coverage_gate, coverage_gate_features, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states "MCP server commands" (plural), implying this surface executes or coordinates multiple server commands, but the implementation is an empty function that performs no command execution whatsoever. The code reveals this is merely a command group container with no intrinsic behavior. + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_rules_run_all.story.md b/dogfood/mining-output/stories/pickled_rules_run_all.story.md new file mode 100644 index 0000000..6666c8a --- /dev/null +++ b/dogfood/mining-output/stories/pickled_rules_run_all.story.md @@ -0,0 +1,91 @@ +# Story: run_all + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-rules +- **Surface id:** pickled_rules_run_all +- **Code depth:** callgraph | **Units read:** 7 | **Unresolved:** 2 + +## Context + +This gate is used by continuous integration or quality assurance processes to verify that BDD feature files reference a required set of rules defined in YAML ruleset files. It ensures that all "strict" enforcement rules are covered by at least one feature scenario tag, enforcing traceability between requirements/regulations and test scenarios. The caller invokes this with a working directory containing both feature files and a `pickled.ruleset.yaml` configuration file. + +## What the target does today + +**Inputs:** +- `workdir`: a filesystem path (string or Path object) to a directory containing BDD feature files and a `pickled.ruleset.yaml` configuration + +**Returns:** +A list of `GateResult` objects, where each result corresponds to one ruleset's coverage evaluation. The list is never empty—at minimum one result is returned. + +**Configuration Loading:** +The surface reads `pickled.ruleset.yaml` from the working directory root. If the file is missing or empty, the gate returns a single WARN result with the note "missing pickled.ruleset.yaml or ruleset/rulesets key". If the file contains invalid YAML or is not a dictionary, it is treated as an empty configuration. + +**Ruleset Configuration:** +The configuration file supports two mutually exclusive keys: +- `ruleset`: a string path to a single ruleset file, with optional `ruleset_short_name` to specify its identifier (defaults to the file stem) +- `rulesets`: a list of mappings, each with required `path` (string) and optional `short_name` (string, defaults to the path stem) + +If both keys are present, the gate returns a single FAIL result with notes explaining the mutual exclusivity violation. If `rulesets` is present but empty, or if any list item is malformed (non-dict, missing/non-string path, non-string short_name), the gate returns a single FAIL result with a descriptive validation message. Duplicate short names across rulesets entries result in a FAIL with position information. + +**Feature File Discovery:** +The gate uses the `feature_glob` configuration key (defaults to `"features/**/*.feature"`) to discover feature files via glob pattern matching. If `feature_glob` is present but not a string, the gate raises `RuleSetValidationError`. If no feature files match the pattern, the gate returns a single WARN result with the note "no feature files". + +**Feature Parsing:** +All discovered feature files are parsed using `PytestBddAdapter().parse_feature_file()`, which is an unresolved call. The parsing behavior and error handling is delegated to that collaborator. + +**Per-Ruleset Evaluation:** +For each configured ruleset entry: +1. The gate name is `"rules.coverage"` if only one ruleset exists, otherwise `"rules.coverage.{short_name}"` +2. If the ruleset file does not exist, a FAIL result is returned with notes indicating the missing file path +3. The ruleset YAML is loaded and validated; if loading fails due to malformed content, a FAIL result with gate name `"rules.load.{short_name}"` is returned with the validation error message +4. Coverage is computed by extracting scenario tag references from all parsed features and matching them against the ruleset rules +5. A FAIL verdict is issued if any strict-enforcement rules are unreferenced OR if any unknown rule references are found in the features +6. A PASS verdict is issued only when all strict rules are referenced and no unknown references exist +7. The result includes traces linking each referenced rule to the feature artifact with relation "implements" and confidence "asserted" + +**Side Effects:** +- Reads files from the filesystem: `pickled.ruleset.yaml`, ruleset YAML files, and feature files +- Resolves all paths relative to the working directory root + +**Error Handling:** +Validation errors during configuration or ruleset loading are captured and returned as FAIL results within the list rather than being raised as exceptions. The only exception is if `feature_glob` is invalid, which raises `RuleSetValidationError`. + +## What we want to verify + +- When `pickled.ruleset.yaml` is missing, returns a single WARN result with gate_name "rules.coverage" and notes about missing configuration +- When `pickled.ruleset.yaml` contains both "ruleset" and "rulesets" keys, returns a single FAIL result describing mutual exclusivity +- When "ruleset" key contains a non-string value, returns a single FAIL result with validation message +- When "rulesets" key contains a non-list value, returns a single FAIL result with validation message +- When "rulesets" list is empty, returns a single FAIL result requiring at least one entry +- When "rulesets" contains entries with duplicate short_name values, returns a single FAIL result identifying the duplicate positions +- When no feature files match the configured glob pattern, returns a single WARN result with notes "no feature files" +- When a single ruleset is configured, the returned result uses gate_name "rules.coverage" +- When multiple rulesets are configured, each result uses gate_name "rules.coverage.{short_name}" +- When a configured ruleset file does not exist, returns a FAIL result with notes indicating the missing path +- When a ruleset file contains invalid YAML, returns a FAIL result with gate_name "rules.load.{short_name}" and the validation error +- When all strict rules are referenced and no unknown references exist, returns a PASS verdict +- When any strict-enforcement rule is unreferenced, returns a FAIL verdict with count in notes +- When scenario tags reference unknown rule IDs, returns a FAIL verdict with count in notes +- PASS results include traces for each referenced rule with relation "implements" and artifact_kind "feature" +- The surface always returns a non-empty list of GateResult objects + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states "Run coverage gate for each feature" but the gate actually runs for each *ruleset* and evaluates coverage across *all* features collectively per ruleset, not per individual feature +- Docstring drift: The docstring does not mention the WARN verdict cases (missing configuration, no features) +- Docstring drift: The docstring does not mention the special handling for ruleset loading failures that produce "rules.load.{short_name}" gate names +- Docstring drift: The docstring does not describe the configuration schema or the mutual exclusivity of "ruleset" vs "rulesets" keys + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_schema_check.story.md b/dogfood/mining-output/stories/pickled_schema_check.story.md new file mode 100644 index 0000000..4c384e4 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_schema_check.story.md @@ -0,0 +1,81 @@ +# Story: check + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-schema +- **Surface id:** pickled_schema_check +- **Code depth:** callgraph | **Units read:** 7 | **Unresolved:** 0 + +## Context + +This CLI command is used to verify that all `@schema:endpoint` tags found in Gherkin .feature files correspond to actual endpoints defined in an OpenAPI specification document. It is typically run as part of a CI/CD pipeline or pre-commit check to ensure feature files reference only documented API endpoints, preventing drift between acceptance tests and the actual API contract. + +## What the target does today + +**Inputs:** +- `spec`: A required Path to an OpenAPI specification file (YAML or JSON). The file must contain a valid OpenAPI 3.0, 3.1, or 3.2 document with an `openapi` version field at the root. OpenAPI 2.0 (Swagger) is rejected. +- `feature_dir`: An optional Path to a directory. When provided, the command recursively searches for all `**/*.feature` files within that directory tree. +- `feature_glob`: An optional string glob pattern. When provided, the command expands the glob recursively to find matching .feature files. + +**Mutual exclusivity:** Exactly one of `feature_dir` or `feature_glob` must be provided. If both are provided, or if neither is provided, the command raises a ClickException and exits. + +**Processing:** +1. Loads and parses the OpenAPI specification file. If the file cannot be parsed as YAML or JSON, or if the schema version is unsupported or missing, a SchemaParseError is raised. +2. Resolves the list of .feature files from the provided directory or glob pattern. Files are sorted. If no files match the pattern, a ClickException is raised with message "No feature files matched". +3. Instantiates a SchemaCoverageGate and runs it with the parsed spec and the list of feature file paths in the context. +4. The gate scans each .feature file for `@schema:endpoint` tags (format appears to be method and path pairs) and checks whether the OpenAPI spec defines each referenced endpoint. + +**Output:** +Always writes a JSON object to stdout with the following structure: +- `gate`: string name of the gate (from result.gate_name) +- `verdict`: string value of the verdict enum ("PASS", "FAIL", or "WARN") +- `notes`: string describing the result +- `findings`: array of objects, each with `tag` and `source` fields, representing only SchemaCoverageFinding instances + +**Exit behavior:** +- If the verdict is `FAIL`, exits with status code 2 +- If the verdict is `WARN`, exits with status code 1 +- If the verdict is `PASS`, exits with status code 0 (normal termination) + +**Error modes:** +- Both `feature_dir` and `feature_glob` provided → ClickException "Use only one of --feature-dir or --feature-glob" +- Neither `feature_dir` nor `feature_glob` provided → ClickException "Provide --feature-dir or --feature-glob" +- No .feature files found → ClickException "No feature files matched" +- Spec file cannot be parsed or has wrong format → SchemaParseError +- OpenAPI version is 2.0 or unsupported → SchemaParseError + +## What we want to verify + +- When neither `feature_dir` nor `feature_glob` is provided, command raises ClickException +- When both `feature_dir` and `feature_glob` are provided, command raises ClickException +- When `feature_dir` points to a directory with no .feature files, command raises ClickException with message "No feature files matched" +- When `spec` points to a valid OpenAPI 3.x file and all `@schema:endpoint` tags in .feature files match spec endpoints, command writes JSON with verdict "PASS" and exits 0 +- When `spec` points to a valid OpenAPI 3.x file and at least one `@schema:endpoint` tag in .feature files does not match any spec endpoint, command writes JSON with verdict "FAIL", includes findings array with tag and source for each missing endpoint, and exits 2 +- When `spec` points to an OpenAPI 2.0 file (contains "swagger" field), command raises SchemaParseError +- When `spec` file is not valid YAML or JSON, command raises SchemaParseError +- When `spec` file is valid YAML/JSON but lacks "openapi" version field, command raises SchemaParseError +- JSON output always contains exactly four keys: "gate", "verdict", "notes", and "findings" +- Findings array only includes objects with "tag" and "source" fields +- When verdict is "WARN", command exits with status code 1 +- `feature_glob` pattern is expanded recursively and only includes actual files (not directories) +- Feature files discovered via `feature_dir` are sorted by path +- Feature files discovered via `feature_glob` are sorted by path + +## Inventory references + +- Arguments: +- `spec` (required): +- `feature_dir` (optional): Directory tree containing .feature files. +- `feature_glob` (optional): Glob of .feature files (alternative to --feature-dir). +- Related gates: SchemaAmbiguityGate.run, SchemaCoverageGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states "Run SchemaCoverageGate on @schema:endpoint tags in .feature files" but does not mention that the command requires exactly one of `feature_dir` or `feature_glob` to be provided, that it validates the OpenAPI version, that it produces structured JSON output to stdout, or that it uses specific exit codes (0, 1, 2) based on verdict. + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_schema_draft.story.md b/dogfood/mining-output/stories/pickled_schema_draft.story.md new file mode 100644 index 0000000..79d2903 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_schema_draft.story.md @@ -0,0 +1,86 @@ +# Story: draft + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-schema +- **Surface id:** pickled_schema_draft +- **Code depth:** callgraph | **Units read:** 5 | **Unresolved:** 4 + +## Context + +This CLI command is used by developers who want to generate an OpenAPI 3.1 path item specification from a Gherkin scenario file. It automates the creation of API documentation by reading behavior-driven development (BDD) scenarios and producing corresponding OpenAPI schema fragments. The command supports both writing to a file or printing to stdout, making it suitable for both automated pipelines and interactive use. + +## What the target does today + +**Inputs:** +- Accepts four parameters: `method` (string), `endpoint_path` (string), `gherkin_file` (Path object), and optional `output` (Path object or None) +- Reads the Gherkin file content as UTF-8 encoded text +- Does not validate the method string format or endpoint_path structure before passing to the drafter + +**LLM Client Configuration:** +- Attempts to build an LLM client first by checking the `PICKLED_SCHEMA_LLM_FACTORY` environment variable +- If `PICKLED_SCHEMA_LLM_FACTORY` is set, expects format "module:callable" and dynamically imports and invokes it; raises `ClickException` if the format lacks a colon separator +- If `PICKLED_SCHEMA_LLM_FACTORY` is not set, falls back to `PICKLED_LLM_PROVIDER` environment variable (defaults to "anthropic") and uses pickled_core's build_client with loaded configuration +- Converts any `ConfigError` from the fallback client builder into a `ClickException` + +**Drafting Process:** +- Delegates to `OpenAPIDrafter.draft_endpoint()` which attempts up to 3 times to generate valid OpenAPI YAML +- For each attempt, renders a prompt template (unresolved call) with the method (uppercased), path, gherkin context, and existing component names +- Invokes an LLM with the prompt and system instruction to output only YAML +- Parses the LLM output as YAML and expects a dictionary +- Unwraps the path item if the LLM returns either a bare operation object or a single-key wrapper with an HTTP method +- Validates the path item by embedding it in a minimal OpenAPI 3.1.0 envelope with the specified path and method (lowercased) +- Uses openapi-spec-validator for validation; requires the pickled-schema[openapi] extra to be installed +- If validation fails, includes previous validation errors in the next prompt attempt +- After 3 failed attempts, raises `SchemaValidationError` with the last error message + +**Outputs:** +- If `output` parameter is provided: writes the generated YAML content to the specified file as UTF-8 text and prints a confirmation message to stderr +- If `output` is None: prints the generated YAML content to stdout +- The generated content is a YAML-formatted path item (not a complete OpenAPI document), using safe_dump with sort_keys=False and default_flow_style=False + +**Error Conditions:** +- Raises `ClickException` if `PICKLED_SCHEMA_LLM_FACTORY` is malformed (missing colon) +- Raises `ClickException` if LLM client configuration fails (ConfigError from pickled_core) +- Raises `SchemaValidationError` if the LLM output is not a YAML mapping +- Raises `SchemaValidationError` if openapi-spec-validator extra is not installed +- Raises `SchemaValidationError` if unable to produce valid OpenAPI after 3 attempts, including the last validation error +- May raise file I/O errors if gherkin_file cannot be read or output cannot be written + +## What we want to verify + +- When gherkin_file contains valid text and output is None, the command prints valid OpenAPI path item YAML to stdout +- When gherkin_file contains valid text and output is a Path, the command writes the YAML to that file and prints "Wrote {output}" to stderr +- When PICKLED_SCHEMA_LLM_FACTORY is set to "module:callable" format, the command attempts to import and invoke the specified factory +- When PICKLED_SCHEMA_LLM_FACTORY is set without a colon, the command raises ClickException with message about required format +- When PICKLED_SCHEMA_LLM_FACTORY is not set, the command uses PICKLED_LLM_PROVIDER (defaulting to "anthropic") +- When the LLM produces invalid YAML mapping output, the command retries up to 3 times +- When the LLM produces output that fails OpenAPI validation, the command includes previous errors in subsequent prompts +- When 3 validation attempts all fail, the command raises SchemaValidationError with details about the last failure +- When openapi-spec-validator is not installed, the command raises SchemaValidationError requesting pickled-schema[openapi] +- The method parameter is uppercased for endpoint_id and prompt rendering but lowercased when constructing the validation envelope +- The output YAML is formatted with sort_keys=False and default_flow_style=False + +## Inventory references + +- Arguments: +- `method` (required): +- `endpoint_path` (required): +- `gherkin_file` (required): +- `output` (optional): +- Related gates: SchemaAmbiguityGate.run, SchemaCoverageGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states the function drafts "an OpenAPI 3.1 path item from a Gherkin scenario" but omits that it may perform up to 3 retry attempts with validation feedback +- Docstring drift: The docstring does not mention the LLM client configuration mechanism via environment variables (PICKLED_SCHEMA_LLM_FACTORY and PICKLED_LLM_PROVIDER) +- Docstring drift: The docstring does not mention that openapi-spec-validator must be installed (via pickled-schema[openapi] extra) for the function to work +- Docstring drift: The docstring does not mention the two output modes (file vs stdout) or the stderr confirmation message behavior +- Docstring drift: The docstring does not mention any error conditions or exceptions that may be raised + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_schema_mcp.story.md b/dogfood/mining-output/stories/pickled_schema_mcp.story.md new file mode 100644 index 0000000..ec72e92 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_schema_mcp.story.md @@ -0,0 +1,41 @@ +# Story: mcp + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-schema +- **Surface id:** pickled_schema_mcp +- **Code depth:** callgraph | **Units read:** 1 | **Unresolved:** 0 + +## Context + +This is a CLI command entry point for MCP (Model Context Protocol) server operations. It serves as a top-level command group in a CLI application, likely used by developers or operators to interact with MCP server functionality. The surface is intended to organize subcommands related to MCP server operations rather than perform actions directly. + +## What the target does today + +The function accepts no arguments and returns None. When invoked, it performs no observable operations—it does not raise exceptions, produce output, modify state, or trigger side effects. The function body is empty (consists only of a pass statement or docstring). This is characteristic of a Click command group decorator target that exists solely to provide a namespace and documentation anchor for subcommands. + +## What we want to verify + +- Calling `mcp()` with no arguments completes without raising an exception +- Calling `mcp()` returns None +- Calling `mcp()` produces no console output +- Calling `mcp()` performs no file system operations +- Calling `mcp()` modifies no global state +- The function signature requires zero parameters + +## Inventory references + +- Arguments: +- (none) +- Related gates: SchemaAmbiguityGate.run, SchemaCoverageGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring claims "MCP server commands" (plural) suggesting this surface provides multiple commands, but the implementation is an empty function that performs no command operations. The actual command functionality must be provided through a decorator or framework integration not visible in the function body itself. + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_schema_parse.story.md b/dogfood/mining-output/stories/pickled_schema_parse.story.md new file mode 100644 index 0000000..aa3764a --- /dev/null +++ b/dogfood/mining-output/stories/pickled_schema_parse.story.md @@ -0,0 +1,72 @@ +# Story: parse + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-schema +- **Surface id:** pickled_schema_parse +- **Code depth:** callgraph | **Units read:** 6 | **Unresolved:** 0 + +## Context + +This CLI command is used by operators and developers to quickly inspect a schema file's metadata without fully validating or processing it. It provides a lightweight way to verify that a schema file can be loaded, determine its format (either explicitly specified or auto-detected), and see basic statistics about the content. The command outputs structured JSON to stdout, making it suitable for both human inspection and scripting/pipeline integration. + +## What the target does today + +Accepts a file path and optional format specifier. When format is not provided, infers it from file extension: `.yaml`/`.yml` → OpenAPI 3.1, `.json` → JSON Schema 2020-12, `.proto` → Proto3. Raises an error for unrecognized extensions when format is not explicitly specified. + +Loads the schema file according to the determined format: +- **OpenAPI files** (.yaml, .yml, or explicit OpenAPI 3.0/3.1/3.2 format): Reads file as text, detects the actual OpenAPI version from content (may differ from inferred/specified format), returns the detected version in output +- **JSON Schema files** (.json or explicit json_schema_2020_12): Reads file as UTF-8 text, parses as JSON, validates root is an object (raises SchemaParseError if not) +- **Proto3 files** (.proto or explicit proto3): Invokes `protoc` via `grpc_tools.protoc` Python module with the file's parent directory as proto_path, generates a descriptor set with imports included, base64-encodes the binary descriptor, stores encoded string as content + +Prints to stdout a JSON object with four fields: +- `format`: the schema format as a string (detected version for OpenAPI, specified format otherwise) +- `endpoint_id`: always null for file-based schemas +- `source`: always the string "file" +- `content_bytes`: byte length of the UTF-8 encoded content string (raw text for OpenAPI/JSON Schema, base64-encoded descriptor for Proto3) + +Raises ClickException if: +- Format cannot be inferred from file extension and no explicit format provided +- Explicit format is not one of the supported values +- For Proto3, if `protoc` subprocess returns non-zero exit code + +Raises SchemaParseError if JSON Schema root is not a JSON object. + +File reading errors (missing file, permission denied, encoding errors) propagate as standard Python exceptions. + +The command produces side effects only to stdout (via click.echo) and does not modify any files. + +## What we want to verify + +- Given a .yaml file with valid OpenAPI 3.1 content and no format argument, outputs JSON with format field matching detected OpenAPI version +- Given a .json file with valid JSON Schema and no format argument, outputs JSON with format="json_schema_2020_12" +- Given a .proto file with valid Proto3 syntax and no format argument, outputs JSON with format="proto3" and content_bytes representing base64-encoded descriptor length +- Given explicit --format argument, uses that format instead of inferring from extension +- Given a file with .txt extension and no format argument, raises ClickException mentioning inability to infer format +- Given a .json file containing a JSON array as root, raises SchemaParseError +- Given a nonexistent file path, raises file-not-found exception before format processing +- Output JSON always contains endpoint_id=null and source="file" for any valid file input +- For Proto3 files, if protoc compilation fails (syntax error, missing import), raises RuntimeError with protoc error message +- content_bytes field equals len(content.encode("utf-8")) where content is the artifact's content string +- OpenAPI format detection may return different version than inferred (e.g., .yaml file could be detected as OpenAPI 3.0 if content specifies that version) + +## Inventory references + +- Arguments: +- `file` (required): +- `fmt` (optional): Schema format (auto-detected from extension when omitted). +- Related gates: SchemaAmbiguityGate.run, SchemaCoverageGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: Docstring claims "print a short summary" but does not specify the output is JSON format with four specific fields (format, endpoint_id, source, content_bytes) +- Docstring drift: Docstring does not mention format auto-detection behavior or the mapping of file extensions to schema formats +- Docstring drift: Docstring does not mention any error conditions (unsupported extensions, invalid JSON Schema structure, protoc failures) +- Docstring drift: Docstring does not clarify that endpoint_id is always null and source is always "file" for this command + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_schema_run_all.story.md b/dogfood/mining-output/stories/pickled_schema_run_all.story.md new file mode 100644 index 0000000..1a27090 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_schema_run_all.story.md @@ -0,0 +1,86 @@ +# Story: run_all + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-schema +- **Surface id:** pickled_schema_run_all +- **Code depth:** callgraph | **Units read:** 5 | **Unresolved:** 3 + +## Context + +This gate function is used by a build or CI/CD pipeline to validate OpenAPI specification files and measure schema coverage against Gherkin feature files. It is the entry point for schema validation checks in a project that follows a convention of storing OpenAPI specs under `specs/` and feature files under `features/`. + +## What the target does today + +**Input:** +- Accepts a `workdir` parameter that can be either a `Path` object or a string representing a directory path. +- Converts the workdir to an absolute Path via `.resolve()`. + +**Discovery phase:** +- Searches for OpenAPI specification files matching `specs/*.yaml` and `specs/*.yml` patterns under the resolved workdir, sorted lexicographically. +- If no spec files are found, returns a single-element list containing a `GateResult` with gate_name "schema.openapi", verdict WARN, and notes "no specs/*.yaml". + +**Validation phase:** +For each discovered spec file: +- Loads the file as UTF-8 text and parses it as YAML or JSON based on file extension (`.yaml`, `.yml`, `.json`). +- Falls back to YAML parsing if the extension is unrecognized. +- Rejects files that don't parse to a dict at the root level. +- Rejects OpenAPI 2.0 specs (those with a "swagger" field). +- Requires a string-valued "openapi" top-level field and recognizes versions starting with "3.0", "3.1", or "3.2". +- Validates the parsed spec using `openapi-spec-validator` (requires the `pickled-schema[openapi]` extra to be installed). +- If loading or validation fails due to OSError, ValueError, TypeError, or SchemaValidationError, appends a FAIL result with gate_name "schema.openapi.validate." and the exception message as notes. +- If validation succeeds, appends a PASS result with the same gate_name pattern and the spec's relative path as notes, and retains the spec for coverage analysis. + +**Coverage phase:** +- If no specs are valid, returns results immediately without coverage checks. +- If multiple valid specs exist, appends a WARN result with gate_name "schema.openapi.note" indicating how many specs were found and that the first (lexicographically) will be used for coverage. +- Uses only the first valid spec for coverage analysis. +- Searches for feature files matching `features/**/*.feature` under workdir, sorted. +- If feature files are found, delegates to `SchemaCoverageGate().run(spec_dict, context={"feature_paths": feature_paths})` (behavior unresolved). +- Appends a "schema.coverage" result using the verdict, findings, and notes from the delegated gate; defaults notes to the spec's relative path if none provided. + +**Output:** +- Returns a list of `GateResult` objects representing all validation and coverage checks performed. +- The list may contain: + - Zero or one discovery warning (if no specs found) + - One validation result per spec file (PASS or FAIL) + - Zero or one multi-spec warning (if multiple valid specs) + - Zero or one coverage result (if features exist and at least one valid spec) + +**Error conditions:** +- Raises SchemaValidationError if `openapi-spec-validator` is not installed. +- File I/O errors, parse errors, and validation errors are caught and converted to FAIL results rather than propagated. + +## What we want to verify + +- Given a directory with no `specs/` subdirectory or no `.yaml`/`.yml` files in it, returns a single WARN result with notes "no specs/*.yaml". +- Given a directory with `specs/example.yaml` containing valid OpenAPI 3.x, returns at least one PASS result with gate_name "schema.openapi.validate.example.yaml". +- Given a spec file that cannot be parsed as YAML/JSON, returns a FAIL result for that file. +- Given a spec with a root that is not a dict (e.g., a list), returns a FAIL result. +- Given a spec with "swagger" field instead of "openapi", returns a FAIL result mentioning OpenAPI 2.0 is unsupported. +- Given a spec without an "openapi" field, returns a FAIL result. +- Given a spec with "openapi" version not starting with "3.0", "3.1", or "3.2", returns a FAIL result. +- Given multiple valid specs in `specs/`, returns a WARN result indicating which spec is used for coverage. +- Given valid spec(s) but no `features/**/*.feature` files, does not append a coverage result. +- Given valid spec(s) and feature files, returns a "schema.coverage" result with verdict/findings/notes from the delegated gate. +- When `openapi-spec-validator` is not installed, raises SchemaValidationError during validation phase. +- The returned list of results is in deterministic order: discovery warnings, then validation results in lexicographic filename order, then multi-spec warnings, then coverage results. + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +- Docstring drift: The docstring states "Validate OpenAPI under `specs/`" but does not mention the function also performs schema coverage analysis against feature files, which is a major part of the observable behavior. +- Docstring drift: The docstring does not mention the function returns a list of `GateResult` objects, omitting the return type entirely. +- Docstring drift: The docstring does not mention any of the warning or failure conditions (no specs found, multiple specs found, validation failures, missing dependencies). + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_schema_schemaambiguitygate.story.md b/dogfood/mining-output/stories/pickled_schema_schemaambiguitygate.story.md new file mode 100644 index 0000000..d7b9e22 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_schema_schemaambiguitygate.story.md @@ -0,0 +1,91 @@ +# Story: SchemaAmbiguityGate.run + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-schema +- **Surface id:** pickled_schema_schemaambiguitygate +- **Code depth:** callgraph | **Units read:** 2 | **Unresolved:** 5 + +## Context + +This gate is used within a staged schema validation pipeline to perform an LLM-driven ambiguity check on a drafted SchemaArtifact. The caller (typically a gate runner or pipeline orchestrator) provides a SchemaArtifact target and context containing Gherkin specification text. The gate uses an LLM to identify ambiguities, inconsistencies, or unclear mappings between the Gherkin context and the schema YAML content. + +## What the target does today + +**Inputs:** +- `target`: Expected to be a SchemaArtifact instance; any other type causes immediate failure. +- `context`: Optional dictionary; must contain a key `"gherkin_context"` with a non-empty string value. If `context` is None or missing this key, or the value is not a non-empty string, the gate fails. + +**Processing:** +1. Validates that `target` is a SchemaArtifact. If not, returns a FAIL verdict with a note indicating the received type. +2. Validates that `context` contains `"gherkin_context"` as a non-empty string. If missing, empty, or not a string, returns a FAIL verdict with a descriptive note. +3. Renders a prompt using an internal template (unresolved call) with the Gherkin context and the schema YAML content from the target. +4. Sends the prompt to an LLM via an unresolved `complete_prompt` call with a system instruction requiring a single JSON object response without markdown fences or extra commentary. +5. Parses the LLM response as JSON: + - Strips leading/trailing whitespace. + - If the response starts with triple-backtick markdown fences (with optional "json" label), extracts the content between the fences. + - Attempts to parse the extracted or original stripped response as JSON. + - If JSON parsing fails at any stage, returns None from the parser. +6. If parsing returns None (malformed JSON), the gate returns FAIL with note "LLM returned malformed JSON". +7. Extracts the `"ambiguities"` field from the parsed JSON: + - If the field is missing or not a list, returns FAIL with note 'LLM JSON missing list field "ambiguities"'. + - If the list is empty, returns PASS with note "No ambiguities reported." + - If the list is non-empty, returns WARN with the count of ambiguities in the notes and the list items as findings. + +**Return value:** +Always returns a `GateResult` with: +- `gate_name`: the gate's name +- `verdict`: one of FAIL, PASS, or WARN +- `notes`: descriptive string explaining the outcome +- `findings`: tuple of ambiguity items (only present for WARN verdict when ambiguities are reported) + +**Failure modes:** +- Target is not a SchemaArtifact → FAIL +- Context missing or `gherkin_context` absent/empty/non-string → FAIL +- LLM response not parseable as JSON (with or without markdown fences) → FAIL +- Parsed JSON lacks `"ambiguities"` as a list → FAIL +- Empty ambiguities list → PASS (not a failure) +- Non-empty ambiguities list → WARN (not a failure, but flagged for review) + +**Side effects:** +Calls an unresolved LLM completion function, which may incur API usage, rate limiting, or other external effects. + +## What we want to verify + +- Gate returns FAIL verdict with type mismatch note when target is not a SchemaArtifact instance +- Gate returns FAIL verdict when context is None and requires gherkin_context +- Gate returns FAIL verdict when context dict lacks "gherkin_context" key +- Gate returns FAIL verdict when "gherkin_context" is an empty string or whitespace-only string +- Gate returns FAIL verdict when "gherkin_context" is not a string type +- Gate returns FAIL verdict with "LLM returned malformed JSON" note when LLM response is not valid JSON +- Gate returns FAIL verdict with "LLM returned malformed JSON" note when LLM response has unmatched or malformed markdown fences +- Gate returns FAIL verdict when parsed JSON does not contain "ambiguities" key +- Gate returns FAIL verdict when "ambiguities" value is not a list type +- Gate returns PASS verdict with "No ambiguities reported." note when "ambiguities" is an empty list +- Gate returns WARN verdict with count in notes when "ambiguities" list contains one or more items +- Gate attaches ambiguities list items as findings tuple in GateResult when verdict is WARN +- Gate passes rendered prompt with gherkin_context and schema YAML content to LLM completion +- Gate instructs LLM via system message to return only JSON without markdown fences or commentary +- Gate successfully parses LLM response when wrapped in triple-backtick fences with optional "json" label +- Gate successfully parses LLM response when returned as plain JSON without fences + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: SchemaAmbiguityGate.run +- Related ADRs: +- ADR 0005: `pickled-spec mine` staged mining pipeline — Accepted + +## Open questions + +- Docstring drift: Docstring describes the gate as a "Second LLM critic pass" but does not mention it is specifically focused on ambiguity detection between Gherkin context and schema content. +- Docstring drift: Docstring does not describe the required context structure (must contain "gherkin_context" key with non-empty string). +- Docstring drift: Docstring does not describe the three possible verdict outcomes (FAIL, PASS, WARN) or the conditions under which each occurs. +- Docstring drift: Docstring does not describe the findings field populated when ambiguities are detected. +- Docstring drift: Docstring does not describe the multiple specific failure modes (wrong target type, missing context, malformed LLM JSON, missing ambiguities field). + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_schema_schemacoveragegate.story.md b/dogfood/mining-output/stories/pickled_schema_schemacoveragegate.story.md new file mode 100644 index 0000000..3cb501e --- /dev/null +++ b/dogfood/mining-output/stories/pickled_schema_schemacoveragegate.story.md @@ -0,0 +1,89 @@ +# Story: SchemaCoverageGate.run + +## Metadata + +- **Surface kind:** gate +- **Package:** pickled-schema +- **Surface id:** pickled_schema_schemacoveragegate +- **Code depth:** callgraph | **Units read:** 3 | **Unresolved:** 2 + +## Context + +This surface is a gate implementation used in the pickled-schema package to validate that endpoint references in Gherkin feature files (marked with `@schema:endpoint:*` tags) actually exist in a corresponding OpenAPI specification. It is invoked by a gate runner that passes an OpenAPI spec dictionary as the target and provides feature file paths or text content via the context dictionary. This gate is part of a quality/validation workflow to ensure API test scenarios reference valid API endpoints. + +## What the target does today + +**Inputs:** +- `target`: Expected to be a dictionary representing a parsed OpenAPI specification. If not a dict, the gate fails immediately. +- `context`: Optional dictionary that may contain: + - `"feature_paths"`: A list of file paths (convertible to `Path` objects) pointing to feature files + - `"feature_texts"`: A list of strings, each containing feature file text content + +**Returns:** +A `GateResult` object with fields: +- `gate_name`: Set to `self.name` +- `verdict`: One of `Verdict.PASS` or `Verdict.FAIL` +- `notes`: Human-readable description of the result +- `findings`: A tuple of `SchemaCoverageFinding` objects (only present on certain failure modes) + +**Validation & Failure Modes:** + +1. **Type validation**: If `target` is not a dictionary, returns FAIL with notes indicating the actual type received. + +2. **Context validation**: If the context dictionary contains neither `"feature_paths"` nor `"feature_texts"` (or both are absent/empty after type filtering), returns FAIL with notes stating the requirement for these keys. + +3. **Schema coverage check**: + - Extracts endpoint tags from feature file content using a pattern matching mechanism (delegated to `_ENDPOINT_TAG_RE.finditer`) + - Each tag is parsed to extract an HTTP method and path + - For each tag, checks whether the OpenAPI spec contains a matching endpoint by: + - Looking up the path in `spec["paths"]` dictionary + - Verifying the path entry is a dictionary + - Checking if the lowercased HTTP method exists as a key in that path's dictionary + - Returns FAIL if any tags reference endpoints not found in the spec, with `findings` containing `SchemaCoverageFinding` objects (each with `tag` and `source` attributes) and notes listing each missing endpoint as "{tag} ({source})" separated by semicolons + - Returns PASS if all tags have matching endpoints, with notes "All @schema:endpoint tags have matching paths." + +**Context Processing:** +- `feature_paths`: Only list items are processed; each is converted to a `Path` object. Files are read with UTF-8 encoding. +- `feature_texts`: Only string items within the list are processed; each is paired with a synthetic source identifier `""` where `i` is the list index. +- Non-conforming items in these lists are silently ignored. + +**Side Effects:** +- Reads files from disk for each path in `feature_paths` +- No other observable side effects + +## What we want to verify + +- When target is not a dict (e.g., string, None, list), returns GateResult with verdict=FAIL and notes describing the type mismatch +- When context is None or missing both "feature_paths" and "feature_texts" keys, returns GateResult with verdict=FAIL and notes requiring these context keys +- When context contains empty lists for both "feature_paths" and "feature_texts", returns GateResult with verdict=FAIL +- When feature files contain endpoint tags and all referenced endpoints exist in the OpenAPI spec's paths with matching HTTP methods, returns GateResult with verdict=PASS +- When at least one endpoint tag references a path not present in spec["paths"], returns GateResult with verdict=FAIL and findings containing the missing tag +- When an endpoint tag references a path that exists but the HTTP method is not defined for that path, returns GateResult with verdict=FAIL and findings containing the missing tag +- When spec["paths"] is missing or not a dict, endpoint lookups fail and return GateResult with verdict=FAIL for any tags found +- HTTP method matching is case-insensitive (methods are lowercased before lookup) +- Feature files specified in "feature_paths" are read with UTF-8 encoding +- When "feature_texts" contains non-string items, they are silently skipped +- When "feature_paths" contains non-path-convertible items, they are silently skipped (based on isinstance check) +- Multiple missing endpoints are accumulated and reported together in the notes as semicolon-separated list +- findings tuple contains SchemaCoverageFinding objects with tag and source attributes identifying each missing endpoint +- Source identifier for feature_texts items is "" where index is the position in the list +- Source identifier for feature_paths items is the string representation of the file path + +## Inventory references + +- Arguments: +- (gate class) +- Related gates: SchemaCoverageGate.run +- Related ADRs: +- ADR 0004: Multi-ruleset workspace configuration — Accepted + +## Open questions + +- Docstring drift: The docstring states the surface verifies "@schema:endpoint:* tags" but the actual pattern matching mechanism and tag format are delegated to an unresolved regex pattern (`_ENDPOINT_TAG_RE`), so the exact tag syntax cannot be confirmed from the visible code +- Docstring drift: The docstring does not mention the specific context dictionary keys ("feature_paths" and "feature_texts") required for operation, though this is a critical input requirement +- Docstring drift: The docstring does not describe the return type (GateResult) or the failure modes (type validation, missing context, missing endpoints) +- Docstring drift: The docstring does not mention that HTTP method matching is case-insensitive + +## Status + +draft diff --git a/dogfood/mining-output/stories/pickled_schema_validate.story.md b/dogfood/mining-output/stories/pickled_schema_validate.story.md new file mode 100644 index 0000000..fd1c6f1 --- /dev/null +++ b/dogfood/mining-output/stories/pickled_schema_validate.story.md @@ -0,0 +1,86 @@ +# Story: validate + +## Metadata + +- **Surface kind:** cli_command +- **Package:** pickled-schema +- **Surface id:** pickled_schema_validate +- **Code depth:** callgraph | **Units read:** 4 | **Unresolved:** 0 + +## Context + +This CLI command surface is used to validate schema files (OpenAPI, JSON Schema, or Protocol Buffers) against their respective format specifications. Callers invoke this command by providing a file path, and the command determines the schema format, performs validation, and outputs a JSON result indicating whether the file is valid. + +## What the target does today + +**Input acceptance:** +- Accepts a single required parameter `file` of type `Path` representing the schema file to validate. +- Infers the schema format from the file extension: + - `.yaml` or `.yml` → treats as OpenAPI 3.1 + - `.json` → treats as JSON Schema 2020-12 + - `.proto` → treats as Protocol Buffers 3 +- Raises a `click.ClickException` with message "cannot infer format from extension {suffix!r}; use --format" if the file extension does not match the expected patterns (case-insensitive comparison). + +**Validation behavior by format:** +- For OpenAPI formats (3.0, 3.1, 3.2): + - Loads the file content into a dictionary + - Delegates validation to `openapi-spec-validator` library + - Requires the optional dependency `pickled-schema[openapi]` to be installed; raises `SchemaValidationError` with message "install pickled-schema[openapi] for OpenAPI validation" if not available + - Raises `SchemaValidationError` with message "OpenAPI validation failed" and error details if validation fails +- For JSON Schema 2020-12: + - Loads the file into a dictionary + - Delegates validation to `validate_json_schema_document` (unresolved callee) +- For Protocol Buffers 3: + - Delegates parsing and implicit validation to `parse_proto_file` (unresolved callee) + +**Output:** +- On successful validation, writes a JSON object to standard output via `click.echo` with structure: + ```json + {"valid": true, "format": ""} + ``` + where `` is the string representation of the inferred SchemaFormat enum value. + +**Error modes:** +- Unrecognized file extension → `click.ClickException` +- Missing OpenAPI validation dependencies → `SchemaValidationError` +- OpenAPI validation failure → `SchemaValidationError` with error details +- JSON Schema or Proto3 validation failures depend on behavior of unresolved callees + +**Side effects:** +- Outputs JSON to standard output (via `click.echo`) +- File I/O occurs when loading the schema file + +## What we want to verify + +- Accept `.yaml`, `.yml`, `.json`, and `.proto` file extensions (case-insensitive) +- Reject files with unrecognized extensions by raising `click.ClickException` with appropriate message +- Infer OpenAPI 3.1 format from `.yaml` or `.yml` extensions +- Infer JSON Schema 2020-12 format from `.json` extension +- Infer Proto3 format from `.proto` extension +- Output JSON with `{"valid": true, "format": ""}` structure on successful validation +- Raise `SchemaValidationError` with installation message when `openapi-spec-validator` is not available for OpenAPI files +- Raise `SchemaValidationError` with "OpenAPI validation failed" message when OpenAPI validation detects errors +- Include nested schema errors in the raised exception when OpenAPI validator provides them +- Load OpenAPI files and pass dictionary to `openapi-spec-validator.validate` +- Call `validate_json_schema_document` for JSON files +- Call `parse_proto_file` for Proto files +- Write output to standard output using `click.echo` + +## Inventory references + +- Arguments: +- `file` (required): +- Related gates: SchemaAmbiguityGate.run, SchemaCoverageGate.run, run_all +- Related ADRs: +- ADR 0007: Code-aware stories and docstring drift detection — Accepted + +## Open questions + +- Docstring drift: The docstring states "Validate a schema file against its format specification" but does not mention that the format is inferred from the file extension rather than being explicitly specified or detected from content. +- Docstring drift: The docstring does not document the specific output format (JSON with "valid" and "format" fields written to standard output). +- Docstring drift: The docstring does not mention the file extension requirements or the error raised for unrecognized extensions. +- Docstring drift: The docstring does not mention the optional dependency requirement for OpenAPI validation or the resulting error when missing. + +## Status + +draft diff --git a/dogfood/mining-output/stories/rules_check_ruleset_coverage.story.md b/dogfood/mining-output/stories/rules_check_ruleset_coverage.story.md new file mode 100644 index 0000000..6715b9e --- /dev/null +++ b/dogfood/mining-output/stories/rules_check_ruleset_coverage.story.md @@ -0,0 +1,47 @@ +# Story: rules_check_ruleset_coverage + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-rules +- **Surface id:** rules_check_ruleset_coverage + +## Context + +Used by MCP clients that need to verify Gherkin feature files comply with a YAML-defined rule set as part of a coverage gate. This tool replaced a previous version that accepted filesystem paths, which created a security vulnerability by allowing arbitrary file reads through parser error messages. Clients now pass feature file contents directly rather than paths. + +## What the target does today + +Accepts the text content of YAML rule set definitions and one or more Gherkin feature file contents, along with a short name identifier for the ruleset. Evaluates whether the provided features meet the coverage requirements defined in the YAML ruleset. This is a coverage gate check—it verifies that feature files adequately cover the rules specified in the ruleset. + +The tool explicitly does NOT accept filesystem paths for security reasons (preventing arbitrary-file-read attacks via parser error messages). Feature texts must be passed as content strings. + +## What we want to verify + +- Accepts `ruleset_yaml_text` parameter containing YAML rule set definition content +- Accepts `feature_texts` parameter containing the text contents of one or more `.feature` files +- Accepts `ruleset_short_name` parameter as an identifier for the ruleset being checked +- Rejects or fails safely when given filesystem paths instead of file contents +- Returns coverage analysis results indicating whether features meet ruleset requirements +- Does not perform filesystem reads based on user-supplied paths +- Parses YAML ruleset definitions to extract coverage requirements +- Parses Gherkin feature file contents to extract coverage information +- Compares parsed features against ruleset requirements to determine coverage status + +## Inventory references + +- Arguments: +- `ruleset_yaml_text` (required): +- `feature_texts` (required): +- `ruleset_short_name` (required): +- Related gates: coverage_gate, coverage_gate_features, run_all +- Related ADRs: +- ADR 0004: Multi-ruleset workspace configuration — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/rules_draft_ruleset_from_brief.story.md b/dogfood/mining-output/stories/rules_draft_ruleset_from_brief.story.md new file mode 100644 index 0000000..15e8b32 --- /dev/null +++ b/dogfood/mining-output/stories/rules_draft_ruleset_from_brief.story.md @@ -0,0 +1,48 @@ +# Story: rules_draft_ruleset_from_brief + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-rules +- **Surface id:** rules_draft_ruleset_from_brief + +## Context + +This tool is used by agents or clients within the pickled-rules package to generate a complete ruleset from a brief text description. It accepts metadata about the ruleset (short name, source identifier, what it applies to, and when it becomes active) along with the brief text, and constructs a draft ruleset structure. The tool appears in a workflow alongside coverage_gate, coverage_gate_features, and run_all gates, suggesting it participates in ruleset authoring and validation pipelines. ADR 0004 indicates this operates within a multi-ruleset workspace environment where multiple rulesets may coexist with different activation criteria. + +## What the target does today + +The inventory provides no docstring for this surface; behavior must be confirmed from source before specifying. + +## What we want to verify + +- The tool accepts exactly five required parameters: brief_text, ruleset_short_name, source_id, applies_to, and active_from +- Calling the tool with all five parameters produces a response without parameter validation errors +- The tool generates output that represents a ruleset structure (format to be confirmed from source) +- The generated ruleset incorporates the provided ruleset_short_name in its structure +- The generated ruleset incorporates the provided source_id in its structure +- The generated ruleset incorporates the provided applies_to scope in its structure +- The generated ruleset incorporates the provided active_from temporal constraint in its structure +- The generated ruleset content relates to the provided brief_text description +- Missing any required parameter results in an appropriate error +- The tool can be invoked within a multi-ruleset workspace context (per ADR 0004) + +## Inventory references + +- Arguments: +- `brief_text` (required): +- `ruleset_short_name` (required): +- `source_id` (required): +- `applies_to` (required): +- `active_from` (required): +- Related gates: coverage_gate, coverage_gate_features, run_all +- Related ADRs: +- ADR 0004: Multi-ruleset workspace configuration — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/rules_list_rules.story.md b/dogfood/mining-output/stories/rules_list_rules.story.md new file mode 100644 index 0000000..b301e88 --- /dev/null +++ b/dogfood/mining-output/stories/rules_list_rules.story.md @@ -0,0 +1,43 @@ +# Story: rules_list_rules + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-rules +- **Surface id:** rules_list_rules + +## Context + +Test automation and CI/CD pipelines that need to inspect or validate the structure and content of YAML-based rule sets before executing them. Used by developers and quality gates to understand what rules are defined in a rule set without running the full rule execution logic. + +## What the target does today + +The inventory provides no docstring for this surface; behavior must be confirmed from source before specifying. + +## What we want to verify + +- Accept a `ruleset_yaml_text` parameter containing YAML rule set content +- Parse the YAML content to extract rule definitions +- Return a list or collection of rule summaries +- Handle invalid or malformed YAML input without crashing +- Return an empty or null result when the YAML contains no rules +- Extract identifying information from each rule (minimally rule names/identifiers) +- Preserve the order of rules as they appear in the YAML if order is meaningful +- Support the YAML schema expected by the pickled-rules package +- Function as an MCP tool interface, returning results in MCP-compatible format + +## Inventory references + +- Arguments: +- `ruleset_yaml_text` (required): +- Related gates: coverage_gate, coverage_gate_features, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/schema_check_schema_coverage.story.md b/dogfood/mining-output/stories/schema_check_schema_coverage.story.md new file mode 100644 index 0000000..d9f993e --- /dev/null +++ b/dogfood/mining-output/stories/schema_check_schema_coverage.story.md @@ -0,0 +1,43 @@ +# Story: schema_check_schema_coverage + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-schema +- **Surface id:** schema_check_schema_coverage + +## Context + +QA engineers and CI pipeline scripts use this tool to validate that all `@schema:endpoint` tags referenced in Cucumber feature files correspond to actual endpoints defined in an OpenAPI specification. This prevents runtime failures where tests reference non-existent API endpoints and ensures test coverage stays synchronized with the API schema. + +## What the target does today + +The tool verifies that every `@schema:endpoint` tag found in the provided feature files exists as a defined endpoint in the given OpenAPI specification YAML. It checks for coverage by matching endpoint identifiers from feature annotations against the spec's endpoint definitions. + +## What we want to verify + +- Returns success when all `@schema:endpoint` tags in feature_texts reference endpoints that exist in spec_yaml +- Returns failure or error indication when one or more `@schema:endpoint` tags reference endpoints not found in spec_yaml +- Parses spec_yaml as an OpenAPI specification document to extract valid endpoint definitions +- Scans feature_texts to extract all `@schema:endpoint` tag values +- Reports which `@schema:endpoint` tags are missing from the specification when coverage is incomplete +- Handles empty or malformed spec_yaml gracefully with appropriate error messaging +- Handles empty feature_texts without error (zero tags means full coverage) +- Does not validate endpoint implementation or test correctness, only tag-to-spec correspondence + +## Inventory references + +- Arguments: +- `spec_yaml` (required): +- `feature_texts` (required): +- Related gates: SchemaAmbiguityGate.run, SchemaCoverageGate.run, run_all +- Related ADRs: +- ADR 0004: Multi-ruleset workspace configuration — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/schema_draft_openapi_endpoint.story.md b/dogfood/mining-output/stories/schema_draft_openapi_endpoint.story.md new file mode 100644 index 0000000..d59e9a4 --- /dev/null +++ b/dogfood/mining-output/stories/schema_draft_openapi_endpoint.story.md @@ -0,0 +1,43 @@ +# Story: schema_draft_openapi_endpoint + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-schema +- **Surface id:** schema_draft_openapi_endpoint + +## Context + +This surface is used by clients that need to generate OpenAPI 3.1 path item specifications from Gherkin scenario text. The caller provides an HTTP method, a path, and Gherkin-formatted text describing the endpoint behavior, and receives a drafted OpenAPI path item structure in return. This is typically used in documentation generation workflows or schema-driven API development where behavioral specifications written in Gherkin need to be converted into machine-readable OpenAPI format. + +## What the target does today + +The tool drafts an OpenAPI 3.1 path item from the provided Gherkin text for the specified HTTP method and path. The caller must supply three required parameters: `method` (the HTTP verb), `path` (the endpoint path), and `gherkin_text` (the Gherkin-formatted scenario text describing the endpoint). The output is an OpenAPI 3.1-compliant path item representation derived from parsing and transforming the Gherkin input. + +## What we want to verify + +- Calling the tool with all three required parameters (`method`, `path`, `gherkin_text`) returns a response containing an OpenAPI path item structure +- The response conforms to OpenAPI 3.1 path item schema specifications +- The returned path item corresponds to the specified HTTP method +- The returned path item corresponds to the specified path +- The tool requires all three parameters; omitting any required parameter results in an error +- The Gherkin text is parsed and its content influences the structure of the drafted path item +- The tool integrates with SchemaAmbiguityGate and SchemaCoverageGate as related quality gates for the schema drafting process + +## Inventory references + +- Arguments: +- `method` (required): +- `path` (required): +- `gherkin_text` (required): +- Related gates: SchemaAmbiguityGate.run, SchemaCoverageGate.run, run_all +- Related ADRs: +- (none directly relevant) + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/stories/schema_validate_openapi_spec.story.md b/dogfood/mining-output/stories/schema_validate_openapi_spec.story.md new file mode 100644 index 0000000..8b47248 --- /dev/null +++ b/dogfood/mining-output/stories/schema_validate_openapi_spec.story.md @@ -0,0 +1,40 @@ +# Story: schema_validate_openapi_spec + +## Metadata + +- **Surface kind:** mcp_tool +- **Package:** pickled-schema +- **Surface id:** schema_validate_openapi_spec + +## Context + +This tool is used by MCP clients (Model Context Protocol clients) to validate OpenAPI specification documents provided in YAML format. It is part of the pickled-schema package's validation capabilities, likely called when teams need to ensure their API specifications conform to OpenAPI standards before using them in schema-based testing or documentation workflows. + +## What the target does today + +The surface validates an OpenAPI YAML document. It accepts a required `spec_yaml` parameter containing the YAML content to be validated. + +## What we want to verify + +- Given a valid OpenAPI 3.x YAML document in `spec_yaml`, the tool completes without raising validation errors +- Given an invalid OpenAPI YAML document in `spec_yaml`, the tool reports specific validation failures +- Given malformed YAML in `spec_yaml`, the tool reports a parsing error +- Given an empty string in `spec_yaml`, the tool reports a validation error +- The tool accepts `spec_yaml` as a required parameter and fails when it is omitted +- The validation output indicates whether the specification conforms to OpenAPI standards + +## Inventory references + +- Arguments: +- `spec_yaml` (required): +- Related gates: SchemaAmbiguityGate.run, SchemaCoverageGate.run, run_all +- Related ADRs: +- ADR 0007: Code-aware stories and docstring drift detection — Accepted + +## Open questions + +(none) + +## Status + +draft diff --git a/dogfood/mining-output/tags-proposals.json b/dogfood/mining-output/tags-proposals.json new file mode 100644 index 0000000..4433d15 --- /dev/null +++ b/dogfood/mining-output/tags-proposals.json @@ -0,0 +1,30975 @@ +{ + "schema_version": "1", + "features": [ + { + "feature_path": "features/bdd_draft_feature_from_story.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Practitioner generates feature from simple user story", + "proposals": [ + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-then-asserts-observable-outcome", + "rule_id": "gherkin-then-asserts-observable-outcome", + "short_name": "bdd-domain", + "rule_title": "Then steps assert observable outcomes", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@bdd-domain:gherkin-feature-header-required" + }, + { + "scenario_title": "Scenario: Practitioner saves generated feature to file", + "proposals": [ + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@bdd-domain:gherkin-feature-header-required" + }, + { + "scenario_title": "Scenario: Practitioner generates feature with empty story text", + "proposals": [ + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 6 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-empty-story-deterministic-failure" + }, + { + "scenario_title": "Scenario: Practitioner generates feature with whitespace-only story text", + "proposals": [ + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@bdd-domain:gherkin-feature-header-required" + }, + { + "scenario_title": "Scenario: Tool invokes ambiguity gate during processing", + "proposals": [ + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 4 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-then-asserts-observable-outcome", + "rule_id": "gherkin-then-asserts-observable-outcome", + "short_name": "bdd-domain", + "rule_title": "Then steps assert observable outcomes", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@bdd-domain:gherkin-feature-header-required" + }, + { + "scenario_title": "Scenario: Gate failure prevents feature generation", + "proposals": [ + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 6 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-empty-story-deterministic-failure" + }, + { + "scenario_title": "Scenario: Generated feature incorporates input story elements", + "proposals": [ + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@bdd-domain:gherkin-feature-header-required" + }, + { + "scenario_title": "Scenario: Multiple invocations produce consistent output", + "proposals": [ + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + } + ], + "selected": "@bdd-domain:gherkin-feature-header-required" + } + ] + }, + { + "feature_path": "features/bdd_validate_feature_ambiguity.feature", + "scenarios": [ + { + "scenario_title": "Scenario: BDD practitioner validates a feature file with no ambiguities", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 6 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-then-asserts-observable-outcome", + "rule_id": "gherkin-then-asserts-observable-outcome", + "short_name": "bdd-domain", + "rule_title": "Then steps assert observable outcomes", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario: BDD practitioner detects ambiguous step definitions", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 6 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario: BDD practitioner validates malformed Gherkin syntax", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 6 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario: BDD practitioner validates empty feature text", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 6 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario: BDD practitioner uses ambiguity gate independently", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 6 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario: Test automation engineer integrates ambiguity gate in CI/CD pipeline", + "proposals": [ + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-warnings-field-populated-on-failure" + }, + { + "scenario_title": "Scenario Outline: BDD practitioner validates various Gherkin structures", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 6 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-then-asserts-observable-outcome", + "rule_id": "gherkin-then-asserts-observable-outcome", + "short_name": "bdd-domain", + "rule_title": "Then steps assert observable outcomes", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + } + ] + }, + { + "feature_path": "features/data_apply_sql_to_sandbox.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User executes valid CREATE TABLE statement", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario: User executes multiple SQL statements", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario: User executes invalid SQL", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario: User executes SQL that creates no tables", + "proposals": [ + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-empty-story-deterministic-failure" + }, + { + "scenario_title": "Scenario: User executes DROP TABLE statement", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario: User invokes the tool multiple times", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario Outline: User provides optional dialect parameter", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario Outline: Schema output format consistency", + "proposals": [ + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@oss-hygiene:no-secrets-in-repo" + } + ] + }, + { + "feature_path": "features/data_check_migration_drift.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer validates migration script matches expected schema", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 6 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer detects schema drift when migration differs from expected schema", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 6 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer validates migration with specific SQL dialect", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 6 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario Outline: Developer validates migrations across different SQL dialects", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 6 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer validates migration without specifying dialect", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 6 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer receives actionable results for CI/CD pipeline integration", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 6 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Validation fails when required SQL parameter is missing", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 6 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Validation fails when required expected schema parameter is missing", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 6 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + } + ] + }, + { + "feature_path": "features/data_draft_sql_migration_from_intent.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer generates migration with intent and dialect only", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer generates migration with current schema provided", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario Outline: Tool generates dialect-specific SQL syntax", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer omits current schema parameter", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Tool execution validates against DataContractGate", + "proposals": [ + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-warnings-field-populated-on-failure" + }, + { + "scenario_title": "Scenario: Tool execution validates against MigrationDriftGate", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer provides required intent_text parameter", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer provides required dialect parameter", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer provides optional current_schema_yaml parameter", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Tool handles ambiguous intent description", + "proposals": [ + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@bdd-domain:gherkin-feature-header-required" + }, + { + "scenario_title": "Scenario: Tool handles conflicting intent description", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + } + ] + }, + { + "feature_path": "features/data_parse_sql_migration.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Data engineer parses valid CREATE TABLE statement", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Data engineer parses valid ALTER TABLE statement", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Data engineer parses SQL with specific dialect", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Data engineer parses SQL without specifying dialect", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Drift detection gate consumes AST output", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 9 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Data engineer parses empty SQL input", + "proposals": [ + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-empty-story-deterministic-failure" + }, + { + "scenario_title": "Scenario: Data engineer parses whitespace-only SQL input", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Data engineer attempts to parse malformed SQL", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Data engineer parses SQL without required parameter", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario Outline: Data engineer parses various SQL migration statements", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + } + ] + }, + { + "feature_path": "features/diff_draft_corpus_from_examples.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer generates draft corpus with minimum required parameters", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario: Developer generates draft corpus with documentation notes", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario: Draft corpus output is compatible with run_all gate", + "proposals": [ + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@diff-domain:differential-oracle-gate" + }, + { + "scenario_title": "Scenario Outline: Developer generates corpora with various target sizes", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario: Tool fails when seed_examples parameter is missing", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario: Tool fails when target_size parameter is missing", + "proposals": [], + "selected": null + } + ] + }, + { + "feature_path": "features/diff_verify_against_oracle.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer verifies candidate matches oracle for all corpus items", + "proposals": [ + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 3 + } + ], + "selected": "@diff-domain:differential-oracle-gate" + }, + { + "scenario_title": "Scenario: Developer detects differences between candidate and oracle", + "proposals": [ + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 3 + } + ], + "selected": "@diff-domain:differential-oracle-gate" + }, + { + "scenario_title": "Scenario: Developer customizes difference detection with comparator", + "proposals": [ + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@diff-domain:differential-oracle-gate" + }, + { + "scenario_title": "Scenario: Developer limits execution time with timeout", + "proposals": [ + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 3 + } + ], + "selected": "@diff-domain:differential-oracle-gate" + }, + { + "scenario_title": "Scenario Outline: Tool validates required parameters", + "proposals": [ + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@diff-domain:differential-oracle-gate" + }, + { + "scenario_title": "Scenario: Developer runs verification with minimal required parameters", + "proposals": [ + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@diff-domain:differential-oracle-gate" + }, + { + "scenario_title": "Scenario: Developer verifies with empty corpus", + "proposals": [ + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@diff-domain:differential-oracle-gate" + } + ] + }, + { + "feature_path": "features/iac_diff_terraform_plans.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Automation workflow compares valid base and head Terraform plans", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Automation workflow provides base plan as current state and head plan as proposed changes", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario Outline: Tool rejects invalid Terraform plan JSON formats", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Tool is called without required base_plan_json argument", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Tool is called without required head_plan_json argument", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: IaCAmbiguityGate consumes comparison result output", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: PlanDiffGate consumes comparison result output", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: SecurityBaselineGate consumes comparison result output", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: run_all gate operation consumes comparison result output", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + } + ] + }, + { + "feature_path": "features/iac_draft_terraform_module.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Engineer drafts a module with minimal user story", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Engineer drafts a module without specifying provider", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Engineer drafts a module with explicit provider", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Tool rejects call missing required user story", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Generated module conforms to Terraform structure", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Generated module can be consumed by IaCAmbiguityGate", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Generated module can be consumed by PlanDiffGate", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Generated module can be consumed by SecurityBaselineGate", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Generated module participates in run_all multi-gate workflow", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario Outline: Tool handles edge-case user stories gracefully", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 2 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario Outline: Tool supports multiple cloud providers", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + } + ] + }, + { + "feature_path": "features/iac_explain_plan_diff.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Infrastructure engineer requests analysis of a valid Terraform plan", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Infrastructure engineer receives ambiguous configuration warnings", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Infrastructure engineer receives plan difference analysis", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Infrastructure engineer receives security baseline violations", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Infrastructure engineer analyzes a plan with multiple risk types", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Infrastructure engineer submits malformed Terraform plan JSON", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Infrastructure engineer submits invalid Terraform plan structure", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Infrastructure engineer omits required plan_json parameter", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Infrastructure engineer analyzes a plan with no risks", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + } + ] + }, + { + "feature_path": "features/iac_suggest_security_remediation.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Security engineer receives HCL patches for valid Trivy findings", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Security engineer receives context-aware patches when HCL text is provided", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Security engineer receives generic patches when HCL text is omitted", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 4 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Security engineer submits Trivy findings with zero security issues", + "proposals": [ + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@best-practices:llm-drafter-temperature-zero" + }, + { + "scenario_title": "Scenario: Security engineer submits Trivy findings with multiple security issues", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Tool validates Trivy JSON schema before processing", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Suggested patches are suitable for applying to HCL files", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + } + ] + }, + { + "feature_path": "features/iac_validate_terraform_dir.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Agent validates syntactically correct Terraform files", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Agent validates Terraform files with syntax errors", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Agent validates multiple Terraform files in one invocation", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Agent validates mixed valid and invalid Terraform files", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Agent attempts validation without required tf_files parameter", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Agent validates Terraform files with structural issues", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Agent validates empty Terraform configuration", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 6 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + } + ] + }, + { + "feature_path": "features/pickled_bdd_ambiguity.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User invokes ambiguity command with required feature file argument", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Command writes informational message to stderr", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Command fails when LLM configuration is invalid", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Command passes when LLM is unavailable", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Command outputs valid JSON structure to stdout", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Gate field always contains ambiguity value", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario Outline: Verdict field contains valid enum string value", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 3 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Findings array contains properly structured ambiguity findings", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario Outline: Command exits with code based on verdict", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 3 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: JSON output is formatted with proper indentation and encoding", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + } + ] + }, + { + "feature_path": "features/pickled_bdd_ambiguitygate.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Quality engineer runs gate against a non-Feature object", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Quality engineer runs gate against a Feature with zero scenarios", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Quality engineer runs gate against unambiguous scenarios", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Quality engineer runs gate against fully ambiguous scenarios", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Quality engineer runs gate against partially ambiguous scenarios", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Quality engineer runs gate when all LLM responses fail to parse", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Quality engineer runs gate when some responses fail to parse and rest are unambiguous", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Quality engineer runs gate when some responses fail to parse and rest are ambiguous", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Quality engineer examines an AmbiguityFinding", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario Outline: Quality engineer runs gate against LLM responses with markdown fences", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Quality engineer provides context parameter", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + } + ] + }, + { + "feature_path": "features/pickled_bdd_check.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer checks a valid feature file with LLM available", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Developer checks a feature file when LLM is unavailable", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario Outline: Developer receives appropriate exit codes based on verdict", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 3 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Developer encounters LLM client configuration error", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 9 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Developer provides gate parameter", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Developer verifies JSON output format", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Developer receives findings filtered by type", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Developer verifies finding structure", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + } + ] + }, + { + "feature_path": "features/pickled_bdd_draft.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer drafts feature to stdout", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 2 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer drafts feature to a file", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 2 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Developer receives draft result metadata", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: LLM client configuration fails", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: User story file does not exist", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Command does not validate LLM output", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: LLM output whitespace is normalized", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + } + ] + }, + { + "feature_path": "features/pickled_bdd_mcp.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer invokes mcp command directly", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer invokes mcp command with no side effects", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer checks mcp function signature", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_bdd_run_all.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Project contains no features directory", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Project contains an empty features directory", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Project contains a single valid feature file", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 4 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Project contains multiple valid feature files", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario Outline: Feature file with parsing errors", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Mixed valid and invalid feature files", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: All feature files fail to parse", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Feature files in nested subdirectories are discovered", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + } + ] + }, + { + "feature_path": "features/pickled_core_check_all.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer runs check-all without arguments", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer runs check-all with a specific workdir", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer runs check-all and all gates pass", + "proposals": [ + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@core-domain:verdict-three-state-ladder" + }, + { + "scenario_title": "Scenario: Developer runs check-all and at least one gate fails", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer runs check-all with warnings and warn_ok is false", + "proposals": [ + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@core-domain:verdict-three-state-ladder" + }, + { + "scenario_title": "Scenario: Developer runs check-all with warnings and warn_ok is omitted", + "proposals": [ + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@core-domain:verdict-three-state-ladder" + }, + { + "scenario_title": "Scenario: Developer runs check-all with warnings and warn_ok is true", + "proposals": [ + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@core-domain:verdict-three-state-ladder" + }, + { + "scenario_title": "Scenario: CI pipeline runs check-all with mixed PASS and WARN verdicts and warn_ok is true", + "proposals": [ + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 4 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@core-domain:verdict-three-state-ladder" + }, + { + "scenario_title": "Scenario: CI pipeline runs check-all with FAIL and WARN verdicts regardless of warn_ok", + "proposals": [ + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 4 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@core-domain:verdict-three-state-ladder" + }, + { + "scenario_title": "Scenario: Developer runs check-all and gates from multiple pickled-* packages are executed", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + } + ] + }, + { + "feature_path": "features/pickled_core_mine.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User invokes mine command", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario: User invokes mine command with no project structure", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User invokes mine command and checks filesystem", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario: User invokes mine command and checks output streams", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User invokes mine command and checks performance", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario: User invokes mine command multiple times", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + } + ] + }, + { + "feature_path": "features/pickled_core_mine_all.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User runs mine all with a valid target", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: User omits the required target parameter", + "proposals": [ + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@best-practices:llm-drafter-temperature-zero" + }, + { + "scenario_title": "Scenario: User specifies a custom output directory", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: User requests verbose logging", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User filters surfaces by package or surface-id substring", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User limits concurrent LLM calls", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: User specifies custom ruleset configuration", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario Outline: User controls overwrite behavior for stories and features", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario: User adjusts code-collection parameters", + "proposals": [ + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-warnings-field-populated-on-failure" + }, + { + "scenario_title": "Scenario: User enables cycle detection in the call graph", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: A pipeline stage fails during execution", + "proposals": [ + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@best-practices:llm-drafter-temperature-zero" + }, + { + "scenario_title": "Scenario: User runs in non-quick mode with interactive prompts", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + } + ] + }, + { + "feature_path": "features/pickled_core_mine_code.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer mines code without providing required target argument", + "proposals": [ + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@best-practices:llm-drafter-temperature-zero" + }, + { + "scenario_title": "Scenario: Developer mines code to default output location", + "proposals": [ + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-warnings-field-populated-on-failure" + }, + { + "scenario_title": "Scenario: Developer mines code to specified output directory", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Developer mines code with verbose logging enabled", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario Outline: Developer filters surfaces by package or surface-id substring", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer controls source code context depth", + "proposals": [ + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@oss-hygiene:no-secrets-in-repo" + }, + { + "scenario_title": "Scenario: Developer determines which intra-project callees to follow", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@oss-hygiene:conventional-commits", + "rule_id": "conventional-commits", + "short_name": "oss-hygiene", + "rule_title": "Commit messages follow Conventional Commits", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer limits call graph traversal depth with max_hops", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario: Developer limits number of callee units per surface", + "proposals": [ + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-warnings-field-populated-on-failure" + }, + { + "scenario_title": "Scenario: Developer limits total source lines per surface", + "proposals": [ + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-warnings-field-populated-on-failure" + }, + { + "scenario_title": "Scenario: Developer detects circular dependencies in call graph", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer mines code without cycle detection", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Automation system mines code and encounters an error", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@best-practices:agent-path-first-class" + } + ] + }, + { + "feature_path": "features/pickled_core_mine_evaluate.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Engineer evaluates surfaces when all gates pass", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Engineer evaluates surfaces when any gate fails", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: CI pipeline attempts evaluation without prior mining stages", + "proposals": [ + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@best-practices:llm-drafter-temperature-zero" + }, + { + "scenario_title": "Scenario: Engineer evaluates with custom output directory", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Engineer evaluates with verbose logging enabled", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario Outline: Engineer evaluates filtered surfaces", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Engineer evaluates coverage gate", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Engineer evaluates ambiguity gate", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Engineer evaluates surfaces multiple times", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Engineer evaluates with custom ruleset directory", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Engineer evaluates with custom ruleset configuration", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + } + ] + }, + { + "feature_path": "features/pickled_core_mine_features.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer mines features with required target argument", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario: Developer mines features to a specific output directory", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario: Developer mines features in quick mode", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario: Developer mines features with interactive prompts", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario: Developer mines features with verbose logging", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Developer filters features by surface substring", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer controls parallel processing", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer overwrites existing features", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario: Developer runs mining as stage 4 of the pipeline", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario Outline: Developer mines features with different flag combinations", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + } + ] + }, + { + "feature_path": "features/pickled_core_mine_inventory.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer successfully mines inventory from valid target", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer mines inventory to custom output directory", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer runs mine inventory with verbose logging", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer mines inventory without MCP tools", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 9 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer runs mine inventory in quick mode", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer attempts to mine inventory without specifying target", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario: Developer attempts to mine inventory from invalid target", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 4 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario Outline: Developer mines inventory with different valid targets", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_core_mine_report.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Operator generates report with valid target", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Operator generates report from custom output directory", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Operator generates report with verbose logging", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:conventional-commits", + "rule_id": "conventional-commits", + "short_name": "oss-hygiene", + "rule_title": "Commit messages follow Conventional Commits", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Operator filters report by surface criteria", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Operator runs report in interactive mode", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Operator omits required target argument", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario Outline: Operator combines multiple options", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + } + ] + }, + { + "feature_path": "features/pickled_core_mine_stories.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer generates stories from inventory", + "proposals": [ + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-empty-story-deterministic-failure" + }, + { + "scenario_title": "Scenario: Developer filters surfaces by package name", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer filters surfaces by surface-id substring", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer runs in quick mode by default", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer runs in interactive mode", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer limits concurrent LLM operations", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer overwrites existing story files", + "proposals": [ + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-empty-story-deterministic-failure" + }, + { + "scenario_title": "Scenario: Developer preserves existing story files", + "proposals": [ + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-empty-story-deterministic-failure" + }, + { + "scenario_title": "Scenario: Developer specifies custom code-context directory", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Developer uses default code-context directory", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer generates stories without code-context", + "proposals": [ + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-empty-story-deterministic-failure" + }, + { + "scenario_title": "Scenario: Developer enables verbose logging", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Developer generates code-aware stories for drift detection", + "proposals": [ + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@best-practices:cli-mcp-surface-parity" + }, + { + "scenario_title": "Scenario Outline: Developer specifies output directory", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + } + ] + }, + { + "feature_path": "features/pickled_core_mine_tag.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer tags scenarios with valid target", + "proposals": [ + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@bdd-domain:drafter-no-auto-tags" + }, + { + "scenario_title": "Scenario: Developer specifies custom output directory", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario Outline: Developer controls interactive mode with quick flag", + "proposals": [ + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@bdd-domain:drafter-no-auto-tags" + }, + { + "scenario_title": "Scenario: Developer enables verbose logging", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 2 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Developer filters surfaces by package name or surface ID", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer configures ruleset directory", + "proposals": [ + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 2 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + } + ], + "selected": "@bdd-domain:drafter-no-auto-tags" + }, + { + "scenario_title": "Scenario: Developer configures ruleset configuration per ADR 0004", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer omits required target parameter", + "proposals": [ + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@bdd-domain:drafter-no-auto-tags" + }, + { + "scenario_title": "Scenario: Command operates as stage 5 in mining pipeline", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + } + ] + }, + { + "feature_path": "features/pickled_data_apply.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer applies a valid migration file", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer applies migration creating tables with columns", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer applies migration with column types needing uppercasing", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer applies migration creating column with null type", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: System excludes SQLite internal tables from output", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer attempts to apply a dbt migration file", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer applies migration with top-level ATTACH statement", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer applies migration with nested ATTACH statement", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer applies migration with top-level DETACH statement", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer applies migration with nested DETACH statement", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer applies migration with unparseable SQL", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer applies an empty migration file", + "proposals": [ + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-empty-story-deterministic-failure" + }, + { + "scenario_title": "Scenario: Developer applies migration file with no parse result", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: System handles statement execution failure gracefully", + "proposals": [ + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-empty-story-deterministic-failure" + }, + { + "scenario_title": "Scenario: Developer applies migration file with invalid UTF-8 encoding", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: System transpiles SQL from source dialect to SQLite", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: System creates in-memory database not file-based", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: System attempts to set attached database limit to zero", + "proposals": [ + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@best-practices:llm-drafter-temperature-zero" + }, + { + "scenario_title": "Scenario: System commits transaction after successful execution", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:conventional-commits", + "rule_id": "conventional-commits", + "short_name": "oss-hygiene", + "rule_title": "Commit messages follow Conventional Commits", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + } + ] + }, + { + "feature_path": "features/pickled_data_check_drift.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer validates migration against expected schema with drift detected", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer validates migration against expected schema without drift", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer validates migration with specific database dialect", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: CI/CD pipeline validates migration without optional dialect parameter", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer attempts to run check-drift without required migration argument", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer attempts to run check-drift without required expected schema argument", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer invokes check-drift from command line as part of pickled-data CLI", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + } + ] + }, + { + "feature_path": "features/pickled_data_datacontractgate.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Target is not a string", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Context is missing endpoint_tag key", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Context endpoint_tag value is not a string", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: No SchemaRegistry is configured", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Schema artifact not found for endpoint_tag", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: OpenAPI schema has no extractable properties", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: SQL parsing fails", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: SQL column names exactly match API property names", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario Outline: SQL columns differ from API properties", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: OpenAPI property extraction examines only 200 or 201 response codes", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: OpenAPI property extraction returns names from first matching operation", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: GateResult includes gate name and notes", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + } + ] + }, + { + "feature_path": "features/pickled_data_draft.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer reads intent from standard input", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer reads intent from a file", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer drafts migration without current schema context", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer drafts migration with current schema context", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer specifies SQL dialect for migration", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer writes SQL to standard output", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer writes SQL to a file", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer receives rationale with SQL migration", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer receives SQL without rationale", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer receives warning for unparseable SQL", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario Outline: Developer receives warning for destructive operations", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer completes draft with validation warnings", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Developer completes draft without validation warnings", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: LLM client configuration fails", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Unexpected error during draft generation", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Current schema file does not exist", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Intent file does not exist", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + } + ] + }, + { + "feature_path": "features/pickled_data_mcp.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User invokes mcp command without arguments", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User attempts to invoke mcp command with arguments", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User invokes mcp command and checks for side effects", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User imports and invokes mcp as a standalone function", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User inspects mcp command documentation", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_data_migrationdriftgate.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer provides non-string target", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Developer provides context without expected schema", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer provides expected schema as a dictionary", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer provides expected schema as YAML string", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer provides YAML that parses to non-dictionary", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Migration produces schema with different tables than expected", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Migration produces table with different columns than expected", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Migration produces schema matching expected exactly", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Migration produces schema with only nullable differences", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: SQL dialect is specified in context", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: SQL dialect defaults to postgres when not specified", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Column types are normalized to uppercase during comparison", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Nullable defaults to true when not specified", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: SQLite connection is closed after successful execution", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: SQLite connection is closed after failed execution", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Empty SQL statements are skipped during transpilation", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: None statements from parsing are skipped", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: SQLite attachment limit is set to prevent ATTACH statements", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + } + ] + }, + { + "feature_path": "features/pickled_data_parse.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer parses a valid PostgreSQL migration file", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer parses a migration file with an explicit dialect", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer attempts to parse a DBT file", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario: Developer parses a file with unparseable SQL", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario: Developer parses a file that results in empty parse output", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario Outline: File I/O errors propagate without wrapping", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Parse command output structure validation", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: SQL output is always rendered in postgres dialect", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_data_run_all.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Project with no migration files returns a warning", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Project with no expected schema file skips drift check", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Project with expected schema as a non-file skips drift check", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Migration file with parse error returns a failure", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Migration file with valid SQL returns a pass", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Project with exactly one migration does not warn about multiple migrations", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Project with multiple migrations warns about application order", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Project with migrations and valid expected schema runs drift check", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 4 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: All SQL parsing uses SQLite dialect", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario Outline: Migration files are processed in lexicographic order", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Combined SQL for drift check concatenates with double newlines", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: YAML file with non-dict content skips drift check without error", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario Outline: Multiple migration files produce multiple parse results", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Working directory is resolved to absolute path", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Function always returns at least one gate result", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + } + ] + }, + { + "feature_path": "features/pickled_diff_draft_corpus.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer generates corpus from seeds file with specified target size", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer reads seed data from stdin", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer writes corpus to stdout by default", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer writes corpus to specified output file", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer provides notes from stdin", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer provides notes from file", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 9 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer requests target size equal to number of seeds", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 6 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer requests target size less than number of seeds", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_diff_mcp.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User invokes mcp command with no arguments", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User invokes mcp command verifying parameter requirements", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User invokes mcp command with no side effects", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_diff_run_all.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Configuration file is missing", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 9 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Configuration file has non-string corpus value", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 9 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario Outline: Configuration file has invalid command format", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 9 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Configuration file is valid", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario Outline: Python executable substitution in commands", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 4 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Python executable substitution requires multiple command elements", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Configuration defaults for optional fields", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: FileNotFoundError during configuration loading results in FAIL", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario Outline: Configuration parsing errors result in FAIL", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 9 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: JSONDecodeError during corpus loading results in FAIL", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Return value always contains exactly one GateResult", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_diff_verify.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User provides corpus JSON that is not a list", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User provides valid corpus with dictionary entries", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User provides corpus with non-string name and payload", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User provides corpus with mixed valid and invalid entries", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 6 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User provides corpus with missing required keys", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario Outline: User selects comparison strategy", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User provides empty corpus", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Oracle fails on every input", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: No items successfully compared due to oracle errors", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Some compared items mismatch", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: All compared items mismatch", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: All items pass comparison", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Verify JSON output structure", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Verify finding structure when mismatches occur", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Oracle errors cause items to be skipped from comparison", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Candidate errors are counted as mismatches", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario Outline: Exit codes map to verdicts", + "proposals": [ + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 4 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@core-domain:verdict-three-state-ladder" + } + ] + }, + { + "feature_path": "features/pickled_iac_diff.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Operator compares two identical plans", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Operator compares two empty plans", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator compares plans where head contains a new resource", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator compares plans where head deletes a resource", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator compares plans where head replaces a resource", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator compares plans with different actions for same resource", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator compares plans where base contains resource not in head", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario Outline: Operator compares plans with safe actions only", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 6 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator provides non-existent base file", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator provides non-existent head file", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator provides file with invalid JSON in base", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator provides file with invalid JSON in head", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator provides head file that is not a dictionary", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator provides base file that is not a dictionary", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator compares plans with non-dictionary resource change entries", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator compares plans with missing change field", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator compares plans with non-list actions field", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator compares plans with missing address field", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator validates output JSON structure", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_iac_draft.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User drafts a module and prints to stdout", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User drafts a module and writes to a specified directory", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User drafts a module and writes to an existing directory", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User drafts a module but neither terraform nor opentofu is available", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: User drafts a module with malformed custom LLM factory environment variable", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: User drafts a module but LLM configuration is invalid", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: LLM generates valid HCL on first attempt", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: LLM generates invalid HCL but succeeds on retry", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: LLM fails to generate valid HCL after 3 attempts", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@oss-hygiene:conventional-commits", + "rule_id": "conventional-commits", + "short_name": "oss-hygiene", + "rule_title": "Commit messages follow Conventional Commits", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: LLM response contains code fences which are stripped", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario Outline: User specifies different cloud providers", + "proposals": [ + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-empty-story-deterministic-failure" + }, + { + "scenario_title": "Scenario: User invokes draft with custom LLM factory", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: User invokes draft without custom factory and PICKLED_LLM_PROVIDER not set", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Validation process initializes terraform directory", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + } + ] + }, + { + "feature_path": "features/pickled_iac_iacambiguitygate.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Gate rejects non-IaCArtifact target", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate rejects missing user story when context is None", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate rejects non-string user story", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario Outline: Gate rejects empty or whitespace-only user story", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Gate renders template with user story and artifact content", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate calls LLM with JSON-only system message", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Gate rejects unparseable LLM response", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate rejects non-dictionary JSON response", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate rejects JSON missing ambiguities key", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate rejects JSON with non-list ambiguities value", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate passes when no ambiguities reported", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate warns when ambiguities are detected", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate extracts JSON from markdown code fences", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate extracts JSON between first and last braces", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + } + ] + }, + { + "feature_path": "features/pickled_iac_mcp.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User invokes the mcp command group programmatically", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User invokes the mcp command group with no arguments", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: MCP command group performs no side effects", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: MCP command group organizes subcommands", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User invokes mcp command group from CLI without subcommands", + "proposals": [ + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 4 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + } + ], + "selected": "@best-practices:cli-mcp-surface-parity" + }, + { + "scenario_title": "Scenario: MCP command group execution is non-blocking", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_iac_plan_cmd.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Infrastructure engineer generates a plan with valid Terraform directory and output path", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Infrastructure engineer generates a plan when output file does not exist", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Infrastructure engineer generates a plan that produces valid JSON output", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Infrastructure engineer generates a plan output consumable by validation gates", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Infrastructure engineer attempts to generate a plan when terraform plan fails", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario Outline: Infrastructure engineer provides invalid arguments", + "proposals": [ + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-empty-story-deterministic-failure" + } + ] + }, + { + "feature_path": "features/pickled_iac_plandiffgate.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Gate fails when target is not a dict", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate fails when target is not a dict (string type)", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate fails when context is None", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate fails when context does not contain base_plan key", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate fails when base_plan in context is not a dict", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate passes when both plans have empty resource_changes", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate passes when both plans have null resource_changes", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate passes when both plans have empty resource_changes lists", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate includes finding for new resource in head plan with non-empty actions", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate includes finding for resource with different actions between plans", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate ignores resources only in base plan", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Gate fails when any action is delete", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate fails when any action is replace", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario Outline: Gate warns for safe actions", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate warns when all actions are from safe set", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate warns for unknown actions outside safe and destructive sets", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate includes multiple findings with correct count in notes", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate skips non-dict entries in resource_changes", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate treats missing change field as empty actions", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate treats null change field as empty actions", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate treats missing actions field as empty list", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate treats non-list actions as empty list", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate preserves action order in findings tuples", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate does not create finding when resource has same actions in both plans", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate does not create finding for resource with empty actions in both plans", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + } + ] + }, + { + "feature_path": "features/pickled_iac_run_all.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User runs validation when infra directory is missing", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User runs validation when Terraform/OpenTofu binary is not on PATH", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: User runs validation when Terraform configuration is valid", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: User runs validation when Terraform configuration is invalid", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: User runs validation when Terraform validate raises unexpected exception", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: User runs validation and Terraform directory requires initialization", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: User runs validation when Trivy binary is not on PATH", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User runs validation when Trivy finds CRITICAL severity misconfigurations", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User runs validation when Trivy finds HIGH severity misconfigurations only", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User runs validation when Trivy finds no HIGH or CRITICAL misconfigurations", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User runs validation when Trivy returns non-JSON output", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User runs validation when Trivy execution fails with unexpected error", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User runs validation and receives expected result structure", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User runs validation with all Terraform subprocess calls", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + } + ] + }, + { + "feature_path": "features/pickled_iac_scan.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Operator scans when Trivy is not installed", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator provides invalid input type", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator scans clean Terraform configuration", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator scans configuration with CRITICAL findings", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Operator scans configuration with HIGH findings only", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Trivy returns malformed output", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Trivy fails with unexpected exit code", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Operator scans configuration with mixed severity findings", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Trivy reports findings with missing title fields", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario Outline: Command exit codes align with verdict", + "proposals": [ + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@core-domain:verdict-three-state-ladder" + } + ] + }, + { + "feature_path": "features/pickled_iac_securitybaselinegate.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Gate rejects non-Path target with FAIL verdict", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate passes gracefully when trivy is not installed", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate warns when trivy exits with unexpected return code and empty stdout", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 9 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate warns when trivy output is not valid JSON", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Gate fails when trivy reports CRITICAL misconfigurations", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate warns when trivy reports HIGH but no CRITICAL misconfigurations", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate passes when trivy reports no HIGH or CRITICAL misconfigurations", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate ignores context parameter", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate invokes trivy with correct arguments", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Gate handles malformed trivy JSON output gracefully", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Gate handles malformed Misconfigurations array gracefully", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario Outline: Gate compares severity case-insensitively", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario Outline: Gate extracts finding title with fallback logic", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + } + ] + }, + { + "feature_path": "features/pickled_iac_validate.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User validates Terraform configuration in a valid directory", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: User validates Terraform configuration with syntax errors", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: User invokes validate without providing tf_dir parameter", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: User invokes validate with empty tf_dir parameter", + "proposals": [ + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-empty-story-deterministic-failure" + }, + { + "scenario_title": "Scenario: User validates specific directory different from current working directory", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + }, + { + "scenario_title": "Scenario: Validation results are passed back to the caller", + "proposals": [ + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@iac-domain:terraform-validate-entry" + } + ] + }, + { + "feature_path": "features/pickled_rules_check.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User provides neither feature path nor feature glob", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: User provides both feature path and feature glob", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: User provides non-existent ruleset file", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User provides feature glob matching no files", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario: User provides feature glob matching directories and files", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User checks a single feature file", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User checks multiple feature files", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 5 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User checks multiple feature files in quiet mode", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 4 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario Outline: User selects output format", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 2 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User runs in quiet mode without output file", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User runs in quiet mode with output file", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User runs with output file but not quiet mode", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User runs without output file and not quiet mode", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Coverage check passes", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Coverage check fails", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Ruleset name defaults to built-in name when using built-in ruleset", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Ruleset name defaults to file stem when using file path ruleset", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Ruleset name is explicitly provided", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Multiple feature files report includes sorted paths", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario: Quiet mode shows correct count for multiple files", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_rules_coverage_gate.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Test engineer runs gate with no scenarios in feature", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 4 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Test engineer runs gate with all strict rules referenced", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 5 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Test engineer runs gate with unreferenced strict rule", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 5 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Test engineer runs gate with unknown rule reference", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 5 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Test engineer runs gate with only advisory rules unreferenced", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 5 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Test engineer runs gate with only informational rules unreferenced", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 5 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Test engineer inspects coverage report structure", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Test engineer inspects traces for referenced rules", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Test engineer runs gate with feature having no path", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Test engineer runs gate with feature having empty path", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Test engineer runs gate with feature having truthy path", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Test engineer runs gate with multiple unknown references", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 5 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 2 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Test engineer runs gate with multiple strict rules and mixed coverage", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 4 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Test engineer runs gate with same rule referenced by multiple scenarios", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_rules_coverage_gate_features.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Gate passes when all rules are non-strict", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 5 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Gate fails when a strict rule is not referenced", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Gate fails when a scenario references an unknown rule ID", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Gate passes when all strict rules are referenced and no unknown references exist", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 5 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Unknown references are sorted lexicographically", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 5 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: A rule referenced multiple times appears once in referenced rules and produces one trace", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 4 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Artifact reference defaults to comma-separated feature paths", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 4 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Artifact reference defaults to placeholder when features have no paths", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 4 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Findings field is always empty", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 4 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Rule is considered referenced if any scenario across any feature references it", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 4 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 4 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario: Gate fails when both strict rules are unreferenced and unknown references exist", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Trace contains all required rule metadata from ruleset", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 4 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + } + ] + }, + { + "feature_path": "features/pickled_rules_draft.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User reads brief from stdin", + "proposals": [ + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@bdd-domain:gherkin-feature-header-required" + }, + { + "scenario_title": "Scenario: User reads brief from a file", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: LLM client configuration is invalid", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Command invokes LLM with prompt containing all metadata", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Command validates generated YAML as a rule set", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Command checks for forbidden tokens in generated YAML", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: User writes generated YAML to stdout", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: User writes generated YAML to a file", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: LLM response contains rationale section", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Validation warnings are written to stderr", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 2 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Command exits with status 1 when validation warnings are present", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Command exits with status 2 on general exception", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: ClickException is re-raised without transformation", + "proposals": [], + "selected": null + } + ] + }, + { + "feature_path": "features/pickled_rules_list_rules.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User lists rules from a built-in rule set", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: User lists rules from a custom YAML file", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User attempts to list rules from a non-existent built-in rule set", + "proposals": [ + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@best-practices:llm-drafter-temperature-zero" + }, + { + "scenario_title": "Scenario: User attempts to list rules from a non-existent file path", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User attempts to list rules from an invalid YAML file", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario Outline: User lists rules from different built-in rule sets", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + } + ] + }, + { + "feature_path": "features/pickled_rules_mcp.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer invokes mcp function directly", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer invokes mcp function and verifies no side effects", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer verifies mcp function signature", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_rules_run_all.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Quality engineer runs coverage with missing configuration file", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with empty configuration", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with both ruleset keys present", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with non-string ruleset value", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with non-list rulesets value", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with empty rulesets list", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with malformed ruleset entry", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with missing path in ruleset entry", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with duplicate short names", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with no matching feature files", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with single ruleset configured", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with multiple rulesets configured", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with missing ruleset file", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with invalid ruleset YAML", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with all strict rules referenced", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 4 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with unreferenced strict rules", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with unknown rule references", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 5 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Quality engineer runs coverage with both unreferenced and unknown rules", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Gate always returns non-empty result list", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + } + ] + }, + { + "feature_path": "features/pickled_schema_check.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User provides neither feature_dir nor feature_glob", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User provides both feature_dir and feature_glob", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: User provides feature_dir with no matching feature files", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User provides feature_glob with no matching feature files", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: All schema endpoint tags match the OpenAPI specification", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: At least one schema endpoint tag does not match the OpenAPI specification", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: OpenAPI specification file is OpenAPI 2.0", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Specification file is not valid YAML or JSON", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Specification file lacks openapi version field", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Verdict is WARN", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 6 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 2 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Feature files discovered via feature_dir are sorted by path", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Feature files discovered via feature_glob are sorted by path", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Feature glob pattern expands recursively and excludes directories", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario Outline: Valid OpenAPI 3.x versions are accepted", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + } + ] + }, + { + "feature_path": "features/pickled_schema_draft.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer drafts path item to stdout", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer drafts path item to file", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer uses custom LLM factory via environment variable", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Developer provides malformed LLM factory configuration", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer uses default LLM provider", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Developer specifies custom LLM provider", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: LLM client configuration fails", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 6 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: LLM produces valid OpenAPI on first attempt", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 9 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 4 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: LLM produces invalid YAML and succeeds on retry", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 9 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 4 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: LLM produces non-mapping YAML output", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 9 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: LLM produces output failing OpenAPI validation", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 9 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: All validation attempts fail", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 9 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: OpenAPI validator is not installed", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Method parameter is uppercased for prompts", + "proposals": [], + "selected": null + }, + { + "scenario_title": "Scenario: Method parameter is lowercased for validation", + "proposals": [ + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@schema-domain:openapi-validate-deterministic" + }, + { + "scenario_title": "Scenario: LLM returns bare operation object", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 9 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 4 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: LLM returns operation wrapped with HTTP method key", + "proposals": [ + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 9 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-subserver-llm-client-wired" + }, + { + "scenario_title": "Scenario: Gherkin file cannot be read", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Output file cannot be written", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + } + ] + }, + { + "feature_path": "features/pickled_schema_mcp.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer invokes mcp command with no arguments", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer verifies mcp command signature", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_schema_parse.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer parses OpenAPI YAML file without format argument", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer parses JSON Schema file without format argument", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer parses Proto3 file without format argument", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer parses file with explicit format argument", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer attempts to parse file with unrecognized extension and no format", + "proposals": [ + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@data-domain:migration-drift-gate" + }, + { + "scenario_title": "Scenario: Developer parses JSON file with array root element", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer attempts to parse nonexistent file", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer parses Proto3 file with syntax error", + "proposals": [ + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-output-parses-via-pytest-bdd" + }, + { + "scenario_title": "Scenario: Developer parses OpenAPI file where detected version differs from inferred", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario Outline: Developer parses files with various extension-to-format mappings", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + } + ] + }, + { + "feature_path": "features/pickled_schema_run_all.feature", + "scenarios": [ + { + "scenario_title": "Scenario: User runs gate in directory with no spec files", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User runs gate in directory with empty specs subdirectory", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User validates a valid OpenAPI 3.0 specification", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User validates a valid OpenAPI 3.1 specification", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User validates a valid OpenAPI 3.2 specification", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: User attempts to validate a malformed YAML file", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User attempts to validate a spec with non-dict root", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User attempts to validate an OpenAPI 2.0 specification", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User attempts to validate a spec without openapi field", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario Outline: User attempts to validate a spec with unsupported OpenAPI version", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User validates multiple valid specs", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User runs gate with valid spec but no feature files", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User runs gate with valid spec and empty features directory", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User runs gate with valid spec and feature files", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User runs gate without openapi-spec-validator installed", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User runs gate and receives results in deterministic order", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User validates spec file with .json extension", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: User validates spec file with unknown extension", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User runs gate where spec file cannot be read", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User runs gate with workdir as string path", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User runs gate with workdir as Path object", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: User validates mix of valid and invalid specs", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 2 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + } + ] + }, + { + "feature_path": "features/pickled_schema_schemaambiguitygate.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Gate fails when target is not a SchemaArtifact", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate fails when context is None", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate fails when context lacks \"gherkin_context\" key", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario Outline: Gate fails when \"gherkin_context\" is invalid string", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate fails when \"gherkin_context\" is not a string type", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate fails when LLM returns invalid JSON", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate fails when LLM returns malformed markdown fences", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate fails when parsed JSON lacks \"ambiguities\" key", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate fails when \"ambiguities\" value is not a list", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate passes when ambiguities list is empty", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario Outline: Gate warns when ambiguities are detected", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate sends rendered prompt with gherkin context and schema YAML to LLM", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate instructs LLM to return only JSON without markdown or commentary", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 9 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario Outline: Gate parses LLM response with various JSON formats", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + } + ] + }, + { + "feature_path": "features/pickled_schema_schemacoveragegate.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Gate rejects non-dictionary target", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate rejects None target", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate rejects list target", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate rejects None context", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate rejects context missing both required keys", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate rejects context with empty lists for both keys", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate passes when all endpoint tags match OpenAPI spec paths and methods", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate fails when endpoint tag references path not in spec", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate fails when endpoint tag references method not defined for path", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate fails when spec paths key is missing", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate fails when spec paths value is not a dictionary", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate performs case-insensitive HTTP method matching", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate reads feature files from feature_paths with UTF-8 encoding", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Gate processes feature content from feature_texts", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 2 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate silently skips non-string items in feature_texts", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 2 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Gate silently skips non-path-convertible items in feature_paths", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Gate reports multiple missing endpoints as semicolon-separated list", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate includes tag and source in SchemaCoverageFinding objects", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate uses feature index as source identifier for feature_texts items", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate uses file path string as source identifier for feature_paths items", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate returns GateResult with gate_name field set", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@diff-domain:differential-oracle-gate", + "rule_id": "differential-oracle-gate", + "short_name": "diff-domain", + "rule_title": "Differential oracle gate available", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + }, + { + "scenario_title": "Scenario: Gate returns findings only for failure modes with missing endpoints", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Gate returns no findings on successful validation", + "proposals": [ + { + "tag": "@pickled-internal:stdio-hygiene-gates-log-stderr", + "rule_id": "stdio-hygiene-gates-log-stderr", + "short_name": "pickled-internal", + "rule_title": "Gate progress logs to stderr not stdout", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 2 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@pickled-internal:stdio-hygiene-gates-log-stderr" + } + ] + }, + { + "feature_path": "features/pickled_schema_validate.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer validates an OpenAPI schema with .yaml extension", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer validates an OpenAPI schema with .yml extension", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer validates a JSON Schema file", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer validates a Protocol Buffers file", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@oss-hygiene:conventional-commits", + "rule_id": "conventional-commits", + "short_name": "oss-hygiene", + "rule_title": "Commit messages follow Conventional Commits", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario Outline: Developer validates files with case-insensitive extensions", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer attempts to validate a file with unrecognized extension", + "proposals": [ + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 3 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + } + ], + "selected": "@pickled-internal:core-llm-cache-default-on" + }, + { + "scenario_title": "Scenario: Developer validates OpenAPI file without openapi-spec-validator installed", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 6 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer validates an invalid OpenAPI schema", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer validates an invalid JSON Schema", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: Developer validates an invalid Protocol Buffers file", + "proposals": [ + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@oss-hygiene:conventional-commits", + "rule_id": "conventional-commits", + "short_name": "oss-hygiene", + "rule_title": "Commit messages follow Conventional Commits", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@bdd-domain:draft-warnings-field-populated-on-failure" + }, + { + "scenario_title": "Scenario: System outputs validation result to standard output", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: System loads OpenAPI file as dictionary for validation", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 3 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: System delegates JSON Schema validation to validate_json_schema_document", + "proposals": [ + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@pickled-internal:mcp-output-fixed-json-shape" + }, + { + "scenario_title": "Scenario: System delegates Protocol Buffers validation to parse_proto_file", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@oss-hygiene:conventional-commits", + "rule_id": "conventional-commits", + "short_name": "oss-hygiene", + "rule_title": "Commit messages follow Conventional Commits", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + } + ] + }, + { + "feature_path": "features/rules_check_ruleset_coverage.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Client checks feature coverage against a valid ruleset", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 6 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario: Client attempts to pass filesystem paths instead of file contents", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario: Tool prevents arbitrary file read attacks", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@oss-hygiene:conventional-commits", + "rule_id": "conventional-commits", + "short_name": "oss-hygiene", + "rule_title": "Commit messages follow Conventional Commits", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario Outline: Tool validates required parameters", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario: Client checks coverage with multiple feature files", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 12 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario: Tool handles malformed YAML ruleset", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 6 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario: Tool handles malformed Gherkin feature content", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 6 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + } + ] + }, + { + "feature_path": "features/rules_draft_ruleset_from_brief.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Agent drafts ruleset with all required parameters", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Agent drafts ruleset incorporating short name", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@iac-domain:terraform-validate-entry", + "rule_id": "terraform-validate-entry", + "short_name": "iac-domain", + "rule_title": "Terraform validate gate registered", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Agent drafts ruleset incorporating source identifier", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Agent drafts ruleset incorporating scope", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario: Agent drafts ruleset incorporating temporal constraint", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Agent drafts ruleset with content relating to brief", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:llm-drafter-temperature-zero", + "rule_id": "llm-drafter-temperature-zero", + "short_name": "best-practices", + "rule_title": "LLM drafters use temperature zero", + "score": 2 + }, + { + "tag": "@pickled-internal:core-llm-cache-default-on", + "rule_id": "core-llm-cache-default-on", + "short_name": "pickled-internal", + "rule_title": "Identical LLM inputs use disk cache", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@core-domain:verdict-three-state-ladder", + "rule_id": "verdict-three-state-ladder", + "short_name": "core-domain", + "rule_title": "Gates use pass warn fail verdicts", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario Outline: Agent invokes tool with missing required parameter", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Agent drafts multiple rulesets in workspace", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@data-domain:migration-drift-gate", + "rule_id": "migration-drift-gate", + "short_name": "data-domain", + "rule_title": "Migration drift gate runs on workspace", + "score": 1 + }, + { + "tag": "@oss-hygiene:no-secrets-in-repo", + "rule_id": "no-secrets-in-repo", + "short_name": "oss-hygiene", + "rule_title": "No API keys committed in source", + "score": 1 + }, + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + } + ] + }, + { + "feature_path": "features/rules_list_rules.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Developer lists rules from valid YAML rule set", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Developer lists rules preserving original order", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Developer handles YAML with no rules", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Developer handles invalid YAML input", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: Developer receives MCP-compatible output", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + }, + { + "scenario_title": "Scenario: Developer inspects rule summary content", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 4 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario Outline: Developer lists rules from various YAML structures", + "proposals": [ + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@rules-domain:unknown-tag-fails-gate" + } + ] + }, + { + "feature_path": "features/schema_check_schema_coverage.feature", + "scenarios": [ + { + "scenario_title": "Scenario: QA engineer verifies complete coverage when all endpoint tags exist in specification", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: QA engineer detects missing endpoints when tags reference undefined endpoints", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: CI pipeline validates multiple missing endpoint tags are all reported", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + }, + { + "scenario_title": "Scenario: QA engineer validates empty feature files without error", + "proposals": [ + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@rules-domain:coverage-union-across-features" + }, + { + "scenario_title": "Scenario: CI pipeline handles malformed OpenAPI specification", + "proposals": [ + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 3 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@schema-domain:openapi-validate-deterministic" + }, + { + "scenario_title": "Scenario: CI pipeline handles empty OpenAPI specification", + "proposals": [ + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 3 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@schema-domain:openapi-validate-deterministic" + }, + { + "scenario_title": "Scenario: QA engineer confirms tool only validates tag-to-spec correspondence", + "proposals": [ + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@rules-domain:coverage-union-across-features", + "rule_id": "coverage-union-across-features", + "short_name": "rules-domain", + "rule_title": "Coverage is union across feature files", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:drafter-no-auto-tags", + "rule_id": "drafter-no-auto-tags", + "short_name": "bdd-domain", + "rule_title": "Drafter does not auto-tag scenarios", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + }, + { + "tag": "@rules-domain:unknown-tag-fails-gate", + "rule_id": "unknown-tag-fails-gate", + "short_name": "rules-domain", + "rule_title": "Unknown rule tags fail coverage gate", + "score": 1 + } + ], + "selected": "@pickled-internal:core-model-from-config-not-hardcoded" + } + ] + }, + { + "feature_path": "features/schema_draft_openapi_endpoint.feature", + "scenarios": [ + { + "scenario_title": "Scenario: Client generates OpenAPI path item from complete Gherkin scenario", + "proposals": [ + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@schema-domain:openapi-validate-deterministic" + }, + { + "scenario_title": "Scenario: Client attempts to generate path item without required method parameter", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Client attempts to generate path item without required path parameter", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Client attempts to generate path item without required gherkin_text parameter", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Gherkin text content influences drafted path item structure", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-output-parses-via-pytest-bdd", + "rule_id": "draft-output-parses-via-pytest-bdd", + "short_name": "bdd-domain", + "rule_title": "Drafted feature parses through pytest-bdd adapter", + "score": 1 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario Outline: Client generates path items for different HTTP methods", + "proposals": [ + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 3 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@schema-domain:openapi-validate-deterministic" + }, + { + "scenario_title": "Scenario: Tool integrates with SchemaAmbiguityGate during drafting process", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + }, + { + "scenario_title": "Scenario: Tool integrates with SchemaCoverageGate during drafting process", + "proposals": [ + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 2 + }, + { + "tag": "@bdd-domain:gherkin-feature-header-required", + "rule_id": "gherkin-feature-header-required", + "short_name": "bdd-domain", + "rule_title": "Feature text starts with Feature header", + "score": 1 + } + ], + "selected": "@best-practices:agent-path-first-class" + } + ] + }, + { + "feature_path": "features/schema_validate_openapi_spec.feature", + "scenarios": [ + { + "scenario_title": "Scenario: MCP client validates a valid OpenAPI 3.x YAML document", + "proposals": [ + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 4 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@schema-domain:openapi-validate-deterministic" + }, + { + "scenario_title": "Scenario: MCP client validates an invalid OpenAPI YAML document", + "proposals": [ + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 4 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 2 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + } + ], + "selected": "@schema-domain:openapi-validate-deterministic" + }, + { + "scenario_title": "Scenario: MCP client validates malformed YAML", + "proposals": [ + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 4 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@schema-domain:openapi-validate-deterministic" + }, + { + "scenario_title": "Scenario: MCP client validates an empty string", + "proposals": [ + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 4 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-empty-story-deterministic-failure", + "rule_id": "draft-empty-story-deterministic-failure", + "short_name": "bdd-domain", + "rule_title": "Empty story yields deterministic failure shape", + "score": 1 + } + ], + "selected": "@schema-domain:openapi-validate-deterministic" + }, + { + "scenario_title": "Scenario: MCP client invokes validation without required spec_yaml parameter", + "proposals": [ + { + "tag": "@schema-domain:openapi-validate-deterministic", + "rule_id": "openapi-validate-deterministic", + "short_name": "schema-domain", + "rule_title": "OpenAPI validation is deterministic", + "score": 6 + }, + { + "tag": "@pickled-internal:mcp-subserver-llm-client-wired", + "rule_id": "mcp-subserver-llm-client-wired", + "short_name": "pickled-internal", + "rule_title": "MCP subservers receive configured LLM client", + "score": 4 + }, + { + "tag": "@best-practices:agent-path-first-class", + "rule_id": "agent-path-first-class", + "short_name": "best-practices", + "rule_title": "Agent MCP path is first-class not a wrapper", + "score": 2 + }, + { + "tag": "@best-practices:cli-mcp-surface-parity", + "rule_id": "cli-mcp-surface-parity", + "short_name": "best-practices", + "rule_title": "CLI and MCP surfaces behave identically", + "score": 2 + }, + { + "tag": "@pickled-internal:core-model-from-config-not-hardcoded", + "rule_id": "core-model-from-config-not-hardcoded", + "short_name": "pickled-internal", + "rule_title": "Model resolved from pickled.config.yaml", + "score": 2 + }, + { + "tag": "@pickled-internal:mcp-output-fixed-json-shape", + "rule_id": "mcp-output-fixed-json-shape", + "short_name": "pickled-internal", + "rule_title": "MCP tools return documented JSON fields only", + "score": 2 + }, + { + "tag": "@bdd-domain:draft-warnings-field-populated-on-failure", + "rule_id": "draft-warnings-field-populated-on-failure", + "short_name": "bdd-domain", + "rule_title": "Parser or validation failures surface in warnings", + "score": 1 + } + ], + "selected": "@schema-domain:openapi-validate-deterministic" + } + ] + } + ] +} diff --git a/packages/pickled-bdd/src/pickled_bdd/cli.py b/packages/pickled-bdd/src/pickled_bdd/cli.py index 45b39b1..2df5584 100644 --- a/packages/pickled-bdd/src/pickled_bdd/cli.py +++ b/packages/pickled-bdd/src/pickled_bdd/cli.py @@ -2,10 +2,10 @@ from __future__ import annotations +import click from pathlib import Path -import click -from pickled_core.llm import LLMClient +from pickled_core import AmbiguityFinding, GateResult, LLMClient, Verdict from pickled_bdd.drafter import FeatureDrafter @@ -37,34 +37,23 @@ def draft(story_file: str, output: str | None) -> None: click.echo(result.text) -@main.command() -@click.argument("feature_file", type=click.Path(exists=True, dir_okay=False)) -@click.option( - "--gate", - type=click.Choice(["ambiguity", "all"]), - default="ambiguity", - show_default=True, - help="Which gate to run.", -) -def check(feature_file: str, gate: str) -> None: - """Run compensating gates against a .feature file.""" - import json as _json - import sys - - from pickled_core import AmbiguityFinding, Verdict - +def run_ambiguity_gate(feature_file: str | Path, llm: LLMClient | None) -> GateResult: + """Canonical ambiguity gate entry point (CLI, alias, mine evaluate).""" from pickled_bdd.adapters.pytest_bdd import PytestBddAdapter from pickled_bdd.gates.ambiguity import AmbiguityGate - _ = gate # v0.1: only ambiguity; "all" resolves to the same gate. - - feature = PytestBddAdapter().parse_feature_file(feature_file) - llm = _build_llm_client() + feature = PytestBddAdapter().parse_feature_file(str(feature_file)) + if llm is None: + return GateResult( + gate_name="ambiguity", + verdict=Verdict.PASS, + notes="LLM unavailable; ambiguity gate skipped", + ) + return AmbiguityGate(llm).run(feature) - gate_impl = AmbiguityGate(llm) - result = gate_impl.run(feature) - output = { +def _ambiguity_result_to_json(result: GateResult) -> dict[str, object]: + return { "gate": result.gate_name, "verdict": result.verdict.value, "notes": result.notes, @@ -78,10 +67,46 @@ def check(feature_file: str, gate: str) -> None: if isinstance(f, AmbiguityFinding) ], } - click.echo(_json.dumps(output, indent=2, ensure_ascii=False)) + + +def _exit_for_verdict(verdict: Verdict) -> None: + import sys exit_codes = {Verdict.PASS: 0, Verdict.WARN: 1, Verdict.FAIL: 2} - sys.exit(exit_codes[result.verdict]) + sys.exit(exit_codes[verdict]) + + +@main.command() +@click.argument("feature_file", type=click.Path(exists=True, dir_okay=False)) +@click.option( + "--gate", + type=click.Choice(["ambiguity", "all"]), + default="ambiguity", + show_default=True, + help="Which gate to run.", +) +def check(feature_file: str, gate: str) -> None: + """Run compensating gates against a .feature file.""" + import json as _json + + _ = gate # v0.1: only ambiguity; "all" resolves to the same gate. + llm = _build_llm_client() + result = run_ambiguity_gate(feature_file, llm) + click.echo(_json.dumps(_ambiguity_result_to_json(result), indent=2, ensure_ascii=False)) + _exit_for_verdict(result.verdict) + + +@main.command() +@click.argument("feature_file", type=click.Path(exists=True, dir_okay=False)) +def ambiguity(feature_file: str) -> None: + """Run the ambiguity gate (alias for ``check --gate ambiguity``).""" + import json as _json + + click.echo("(equivalent to: pickled-bdd check --gate ambiguity)", err=True) + llm = _build_llm_client() + result = run_ambiguity_gate(feature_file, llm) + click.echo(_json.dumps(_ambiguity_result_to_json(result), indent=2, ensure_ascii=False)) + _exit_for_verdict(result.verdict) @main.group() diff --git a/packages/pickled-bdd/src/pickled_bdd/drafter.py b/packages/pickled-bdd/src/pickled_bdd/drafter.py index b5ec568..c07807a 100644 --- a/packages/pickled-bdd/src/pickled_bdd/drafter.py +++ b/packages/pickled-bdd/src/pickled_bdd/drafter.py @@ -3,6 +3,8 @@ from __future__ import annotations from pickled_core import DraftResult, LLMClient, PromptTemplate +from pickled_core.llm.sanitize import strip_markdown_fence +from pickled_core.llm.turns import complete_prompt from pickled_bdd.prompts import template_path @@ -22,15 +24,13 @@ def draft_from_story(self, story: str) -> DraftResult: leaves a generic rationale string. """ prompt = self._template.render(story=story) - from pickled_core.llm.turns import complete_prompt - feature_text = complete_prompt( self._llm, prompt, system="You output only Gherkin. No prose, no fences.", ) return DraftResult( - text=feature_text.strip(), + text=strip_markdown_fence(feature_text), rationale="LLM-drafted from user story; no post-processing applied.", warnings=(), ) diff --git a/packages/pickled-bdd/src/pickled_bdd/py.typed b/packages/pickled-bdd/src/pickled_bdd/py.typed new file mode 100644 index 0000000..e69de29 diff --git a/packages/pickled-bdd/tests/test_cli_check.py b/packages/pickled-bdd/tests/test_cli_check.py index cbd6467..5c34a4c 100644 --- a/packages/pickled-bdd/tests/test_cli_check.py +++ b/packages/pickled-bdd/tests/test_cli_check.py @@ -47,3 +47,29 @@ def test_check_exit_one_and_warn( data = json.loads(result.output) assert data["verdict"] == "warn" assert len(data["findings"]) >= 1 + + +def test_ambiguity_alias_equivalent_to_check_gate_ambiguity( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv( + "PICKLED_BDD_LLM_FACTORY", + "pickled_bdd.testing:build_check_pass_llm", + ) + runner = CliRunner() + via_check = runner.invoke( + main, + ["check", str(FEATURE), "--gate", "ambiguity"], + catch_exceptions=False, + ) + via_alias = runner.invoke( + main, + ["ambiguity", str(FEATURE)], + catch_exceptions=False, + ) + assert via_check.exit_code == via_alias.exit_code == 0 + check_data = json.loads(via_check.output) + alias_json = via_alias.output[via_alias.output.index("{") :] + alias_data = json.loads(alias_json) + assert check_data["verdict"] == alias_data["verdict"] == "pass" + assert check_data["gate"] == alias_data["gate"] == "ambiguity" diff --git a/packages/pickled-core/src/pickled_core/llm/__init__.py b/packages/pickled-core/src/pickled_core/llm/__init__.py index aa2ab91..8a35da6 100644 --- a/packages/pickled-core/src/pickled_core/llm/__init__.py +++ b/packages/pickled-core/src/pickled_core/llm/__init__.py @@ -24,6 +24,7 @@ load_config, ) from pickled_core.llm.factory import build_client +from pickled_core.llm.sanitize import strip_markdown_fence from pickled_core.llm.turns import DEFAULT_MODEL, complete_prompt __all__ = [ @@ -51,4 +52,5 @@ "complete_prompt", "load_config", "set_budget_guard", + "strip_markdown_fence", ] diff --git a/packages/pickled-core/src/pickled_core/llm/sanitize.py b/packages/pickled-core/src/pickled_core/llm/sanitize.py new file mode 100644 index 0000000..cb7ab30 --- /dev/null +++ b/packages/pickled-core/src/pickled_core/llm/sanitize.py @@ -0,0 +1,24 @@ +"""Sanitize LLM text before parsing structured outputs.""" + +from __future__ import annotations + +import re + +_FENCE_RE = re.compile( + r"^\s*```[^\n]*\n(?P.*?)\n```\s*$", + re.DOTALL, +) + + +def strip_markdown_fence(text: str) -> str: + """Strip a single outermost markdown code fence wrapping the whole text. + + If the entire stripped text is wrapped in one ```...``` fence, return the + inner body. Otherwise return the text unchanged. Only the outermost + full-content fence is removed; inner fences are preserved. + """ + stripped = text.strip() + match = _FENCE_RE.match(stripped) + if match: + return match.group("body").strip() + return stripped diff --git a/packages/pickled-core/src/pickled_core/mine/cli.py b/packages/pickled-core/src/pickled_core/mine/cli.py index a4cdda9..cae4e92 100644 --- a/packages/pickled-core/src/pickled_core/mine/cli.py +++ b/packages/pickled-core/src/pickled_core/mine/cli.py @@ -2,12 +2,28 @@ from __future__ import annotations +import functools +import sys +from collections.abc import Callable from pathlib import Path +from typing import TYPE_CHECKING, Any, TypeVar import click +from pickled_core.llm.bootstrap import build_default_client +from pickled_core.llm.config import ConfigError, load_config +from pickled_core.mine.code_stage import load_inventory_for_code, run_code +from pickled_core.mine.errors import MineError +from pickled_core.mine.evaluate_stage import run_evaluate +from pickled_core.mine.features_stage import run_features from pickled_core.mine.inventory_stage import run_inventory +from pickled_core.mine.io import ensure_output_dir, parse_surfaces_filter from pickled_core.mine.report_stage import print_stdout_summary, run_report +from pickled_core.mine.stories_stage import load_inventory_from_output, run_stories +from pickled_core.mine.tag_stage import resolve_ruleset_sources, run_tag + +if TYPE_CHECKING: + from pickled_core.llm.base import LLMClient _OUTPUT_OPT = click.option( "--output", @@ -22,6 +38,98 @@ help="Quick mode (default) or interactive prompts.", ) _VERBOSE_OPT = click.option("--verbose", is_flag=True, help="Extra logging to stderr.") +_SURFACES_OPT = click.option( + "--surfaces", + default=None, + help="Comma-separated filter on package name or surface-id substring.", +) +_MAX_PARALLEL_OPT = click.option( + "--max-parallel", + type=int, + default=4, + show_default=True, + help="Max parallel LLM calls in quick mode (stories, features).", +) +_RULESET_DIR_OPT = click.option("--ruleset-dir", type=click.Path(path_type=Path), default=None) +_RULESET_CONFIG_OPT = click.option( + "--ruleset-config", type=click.Path(path_type=Path), default=None +) +_OVERWRITE_STORIES_OPT = click.option("--overwrite-stories", is_flag=True, default=False) +_OVERWRITE_FEATURES_OPT = click.option("--overwrite-features", is_flag=True, default=False) +_DEPTH_OPT = click.option( + "--depth", + type=click.Choice(["signature", "body", "callgraph"], case_sensitive=False), + default="body", + show_default=True, + help="How much source to collect per surface.", +) +_CALLEE_SCOPE_OPT = click.option( + "--callee-scope", + type=click.Choice(["self", "same-package", "any-pickled"], case_sensitive=False), + default="same-package", + show_default=True, + help="Which intra-project callees to follow.", +) +_MAX_HOPS_OPT = click.option( + "--max-hops", + type=int, + default=2, + show_default=True, + help="Callee expansion depth (callgraph only).", +) +_MAX_CALLEES_OPT = click.option( + "--max-callees", + type=int, + default=8, + show_default=True, + help="Hard cap on collected callee units per surface.", +) +_MAX_CODE_LINES_OPT = click.option( + "--max-code-lines", + type=int, + default=400, + show_default=True, + help="Hard cap on total source lines per surface.", +) +_DETECT_CYCLES_OPT = click.option( + "--detect-cycles/--no-detect-cycles", + default=True, + show_default=True, + help="Write code-context/_cycles.json from observed edges.", +) +_CODE_CONTEXT_OPT = click.option( + "--code-context", + "code_context_dir", + type=click.Path(file_okay=False, path_type=Path), + default=None, + help=( + "Directory with code-context/*.md (default: /code-context when present)." + ), +) + +_F = TypeVar("_F", bound=Callable[..., Any]) + + +def _build_llm(target: Path) -> LLMClient | None: + cfg_path = target / "pickled.config.yaml" + try: + if cfg_path.is_file(): + return build_default_client(config=load_config(cfg_path)) + return build_default_client() + except ConfigError: + return None + + +def _catch_mine_errors(fn: _F) -> _F: + @functools.wraps(fn) + def wrapper(*args: Any, **kwargs: Any) -> Any: + try: + return fn(*args, **kwargs) + except (MineError, ValueError) as exc: + click.echo(str(exc), err=True) + raise SystemExit(2) from None + + return wrapper # type: ignore[return-value] @click.group() @@ -66,32 +174,108 @@ def inventory( @_OUTPUT_OPT @_QUICK_OPT @_VERBOSE_OPT +@_SURFACES_OPT def report( target: Path, output_dir: Path, quick: bool, # noqa: ARG001 verbose: bool, + surfaces: str | None, ) -> None: - """Stage 6: render mining-report.md from pipeline outputs.""" + """Stage 7: render mining-report.md from pipeline outputs.""" _ = quick out = output_dir.resolve() label = str(target) if target.is_dir() and (target / "inventory.json").is_file(): out = target.resolve() - run_report(out, target_label=label, verbose=verbose) + run_report( + out, + target_label=label, + verbose=verbose, + surfaces_filter=parse_surfaces_filter(surfaces), + ) print_stdout_summary(out, target_label=label) +@mine.command() +@click.argument("target", type=click.Path(exists=True, file_okay=False, path_type=Path)) +@_OUTPUT_OPT +@_VERBOSE_OPT +@_SURFACES_OPT +@_DEPTH_OPT +@_CALLEE_SCOPE_OPT +@_MAX_HOPS_OPT +@_MAX_CALLEES_OPT +@_MAX_CODE_LINES_OPT +@_DETECT_CYCLES_OPT +@_catch_mine_errors +def code( + target: Path, + output_dir: Path, + verbose: bool, + surfaces: str | None, + depth: str, + callee_scope: str, + max_hops: int, + max_callees: int, + max_code_lines: int, + detect_cycles: bool, +) -> None: + """Stage 2: extract code context per surface from inventory.json.""" + tgt = target.resolve() + out = output_dir.resolve() + inventory = load_inventory_for_code(out) + run_code( + inventory, + tgt, + out, + depth=depth, # type: ignore[arg-type] + scope=callee_scope, # type: ignore[arg-type] + max_hops=max_hops, + max_callees=max_callees, + max_code_lines=max_code_lines, + detect_cycles=detect_cycles, + surfaces=parse_surfaces_filter(surfaces), + verbose=verbose, + ) + + @mine.command() @click.argument("target", type=click.Path(exists=True, file_okay=False, path_type=Path)) @_OUTPUT_OPT @_QUICK_OPT @_VERBOSE_OPT -def stories(target: Path, output_dir: Path, quick: bool, verbose: bool) -> None: # noqa: ARG001 - """Stage 2: emit stories (Phase 8b).""" - _ = target, output_dir, quick, verbose - click.echo("mine stories is not implemented yet (Phase 8b)", err=True) - raise SystemExit(2) +@_SURFACES_OPT +@_MAX_PARALLEL_OPT +@_OVERWRITE_STORIES_OPT +@_CODE_CONTEXT_OPT +@_catch_mine_errors +def stories( + target: Path, + output_dir: Path, + quick: bool, + verbose: bool, # noqa: ARG001 + surfaces: str | None, + max_parallel: int, + overwrite_stories: bool, + code_context_dir: Path | None, +) -> None: + """Stage 3: emit stories from inventory.json.""" + _ = target + out = output_dir.resolve() + inventory = load_inventory_from_output(out) + llm = _build_llm(target.resolve()) + ctx_dir = code_context_dir.resolve() if code_context_dir else None + run_stories( + inventory, + out, + llm=llm, + quick=quick, + overwrite=overwrite_stories, + surfaces=parse_surfaces_filter(surfaces), + max_parallel=max_parallel, + code_context_dir=ctx_dir, + ) @mine.command() @@ -99,11 +283,31 @@ def stories(target: Path, output_dir: Path, quick: bool, verbose: bool) -> None: @_OUTPUT_OPT @_QUICK_OPT @_VERBOSE_OPT -def features(target: Path, output_dir: Path, quick: bool, verbose: bool) -> None: # noqa: ARG001 - """Stage 3: draft features (Phase 8b).""" - _ = target, output_dir, quick, verbose - click.echo("mine features is not implemented yet (Phase 8b)", err=True) - raise SystemExit(2) +@_SURFACES_OPT +@_MAX_PARALLEL_OPT +@_OVERWRITE_FEATURES_OPT +@_catch_mine_errors +def features( + target: Path, + output_dir: Path, + quick: bool, + verbose: bool, # noqa: ARG001 + surfaces: str | None, + max_parallel: int, + overwrite_features: bool, +) -> None: + """Stage 4: draft features from stories.""" + _ = target + out = output_dir.resolve() + llm = _build_llm(target.resolve()) + run_features( + out, + llm=llm, + quick=quick, + overwrite=overwrite_features, + surfaces=parse_surfaces_filter(surfaces), + max_parallel=max_parallel, + ) @mine.command() @@ -111,32 +315,66 @@ def features(target: Path, output_dir: Path, quick: bool, verbose: bool) -> None @_OUTPUT_OPT @_QUICK_OPT @_VERBOSE_OPT -@click.option("--ruleset-dir", type=click.Path(path_type=Path), default=None) -@click.option("--ruleset-config", type=click.Path(path_type=Path), default=None) +@_SURFACES_OPT +@_RULESET_DIR_OPT +@_RULESET_CONFIG_OPT +@_catch_mine_errors def tag( target: Path, output_dir: Path, - quick: bool, # noqa: ARG001 + quick: bool, verbose: bool, # noqa: ARG001 + surfaces: str | None, ruleset_dir: Path | None, ruleset_config: Path | None, ) -> None: - """Stage 4: tag scenarios (Phase 8b).""" - _ = target, output_dir, quick, verbose, ruleset_dir, ruleset_config - click.echo("mine tag is not implemented yet (Phase 8b)", err=True) - raise SystemExit(2) + """Stage 5: tag scenarios in generated features.""" + tgt = target.resolve() + out = output_dir.resolve() + sources = resolve_ruleset_sources( + tgt, + ruleset_config=ruleset_config.resolve() if ruleset_config else None, + ruleset_dir=ruleset_dir.resolve() if ruleset_dir else None, + ) + run_tag( + out, + ruleset_sources=sources, + quick=quick, + surfaces=parse_surfaces_filter(surfaces), + ) @mine.command() @click.argument("target", type=click.Path(exists=True, file_okay=False, path_type=Path)) @_OUTPUT_OPT -@_QUICK_OPT @_VERBOSE_OPT -def evaluate(target: Path, output_dir: Path, quick: bool, verbose: bool) -> None: # noqa: ARG001 - """Stage 5: evaluate gates (Phase 8c).""" - _ = target, output_dir, quick, verbose - click.echo("mine evaluate is not implemented yet (Phase 8c)", err=True) - raise SystemExit(2) +@_SURFACES_OPT +@_RULESET_DIR_OPT +@_RULESET_CONFIG_OPT +@_catch_mine_errors +def evaluate( + target: Path, + output_dir: Path, + verbose: bool, # noqa: ARG001 + surfaces: str | None, + ruleset_dir: Path | None, + ruleset_config: Path | None, +) -> None: + """Stage 6: evaluate coverage and ambiguity gates.""" + tgt = target.resolve() + out = output_dir.resolve() + sources = resolve_ruleset_sources( + tgt, + ruleset_config=ruleset_config.resolve() if ruleset_config else None, + ruleset_dir=ruleset_dir.resolve() if ruleset_dir else None, + ) + llm = _build_llm(tgt) + run_evaluate( + out, + ruleset_sources=sources, + llm=llm, + surfaces=parse_surfaces_filter(surfaces), + ) @mine.command(name="all") @@ -144,30 +382,107 @@ def evaluate(target: Path, output_dir: Path, quick: bool, verbose: bool) -> None @_OUTPUT_OPT @_QUICK_OPT @_VERBOSE_OPT +@_SURFACES_OPT +@_MAX_PARALLEL_OPT @click.option("--no-mcp", is_flag=True, default=False) @click.option("--mcp-timeout", type=float, default=30.0, show_default=True) -@click.option("--ruleset-dir", type=click.Path(path_type=Path), default=None) -@click.option("--ruleset-config", type=click.Path(path_type=Path), default=None) -@click.option("--overwrite-stories", is_flag=True, default=False) -@click.option("--overwrite-features", is_flag=True, default=False) +@_RULESET_DIR_OPT +@_RULESET_CONFIG_OPT +@_OVERWRITE_STORIES_OPT +@_OVERWRITE_FEATURES_OPT +@_DEPTH_OPT +@_CALLEE_SCOPE_OPT +@_MAX_HOPS_OPT +@_MAX_CALLEES_OPT +@_MAX_CODE_LINES_OPT +@_DETECT_CYCLES_OPT +@_catch_mine_errors def mine_all( target: Path, output_dir: Path, - quick: bool, # noqa: ARG001 + quick: bool, verbose: bool, + surfaces: str | None, + max_parallel: int, no_mcp: bool, mcp_timeout: float, ruleset_dir: Path | None, ruleset_config: Path | None, overwrite_stories: bool, overwrite_features: bool, + depth: str, + callee_scope: str, + max_hops: int, + max_callees: int, + max_code_lines: int, + detect_cycles: bool, ) -> None: - """Run inventory then report (Phase 8a subset of full pipeline).""" - _ = ruleset_dir, ruleset_config, overwrite_stories, overwrite_features, quick - out = output_dir.resolve() + """Run inventory → code → stories → features → tag → evaluate → report.""" tgt = target.resolve() - run_inventory(tgt, out, include_mcp=not no_mcp, mcp_timeout=mcp_timeout, verbose=verbose) - run_report(out, target_label=str(tgt), verbose=verbose) + out = output_dir.resolve() + surface_tokens = parse_surfaces_filter(surfaces) + run_inventory( + tgt, + out, + include_mcp=not no_mcp, + mcp_timeout=mcp_timeout, + verbose=verbose, + ) + inventory = load_inventory_from_output(out) + code_result = run_code( + inventory, + tgt, + out, + depth=depth, # type: ignore[arg-type] + scope=callee_scope, # type: ignore[arg-type] + max_hops=max_hops, + max_callees=max_callees, + max_code_lines=max_code_lines, + detect_cycles=detect_cycles, + surfaces=surface_tokens, + verbose=verbose, + ) + if verbose and code_result.cycle_count: + sys.stderr.write( + f"[INFO] code stage: {code_result.cycle_count} cycle(s) in " + f"{code_result.cycles_path}\n" + ) + llm = _build_llm(tgt) + run_stories( + inventory, + out, + llm=llm, + quick=quick, + overwrite=overwrite_stories, + surfaces=surface_tokens, + max_parallel=max_parallel, + ) + feat_result = run_features( + out, + llm=llm, + quick=quick, + overwrite=overwrite_features, + surfaces=surface_tokens, + max_parallel=max_parallel, + ) + paths = ensure_output_dir(out) + has_features = bool(list(paths.features_dir.glob("*.feature"))) + sources = resolve_ruleset_sources( + tgt, + ruleset_config=ruleset_config.resolve() if ruleset_config else None, + ruleset_dir=ruleset_dir.resolve() if ruleset_dir else None, + ) + if has_features and not feat_result.skipped_entire_stage: + run_tag(out, ruleset_sources=sources, quick=quick, surfaces=surface_tokens) + run_evaluate(out, ruleset_sources=sources, llm=llm, surfaces=surface_tokens) + else: + sys.stderr.write("[WARN] tag and evaluate skipped (no feature files)\n") + run_report( + out, + target_label=str(tgt), + verbose=verbose, + surfaces_filter=surface_tokens, + ) print_stdout_summary(out, target_label=str(tgt)) diff --git a/packages/pickled-core/src/pickled_core/mine/code_reader.py b/packages/pickled-core/src/pickled_core/mine/code_reader.py new file mode 100644 index 0000000..37b1461 --- /dev/null +++ b/packages/pickled-core/src/pickled_core/mine/code_reader.py @@ -0,0 +1,1507 @@ +"""AST call-graph extraction for the mine code stage.""" + +from __future__ import annotations + +import ast +import importlib +import inspect +import sys +import types +from collections import deque +from collections.abc import Iterator +from dataclasses import dataclass, field +from pathlib import Path +from typing import Literal + +import click + +from pickled_core.mine.types import ResolutionKind + +Depth = Literal["signature", "body", "callgraph"] +CalleeScope = Literal["self", "same-package", "any-pickled"] + +_STDLIB_TOP_LEVEL = frozenset( + { + "abc", + "ast", + "asyncio", + "collections", + "contextlib", + "dataclasses", + "enum", + "functools", + "importlib", + "inspect", + "io", + "itertools", + "json", + "logging", + "os", + "pathlib", + "re", + "sys", + "textwrap", + "typing", + "uuid", + "warnings", + } +) +_THIRD_PARTY_TOP = frozenset({"click", "pydantic", "yaml", "httpx", "requests"}) +_TRUNCATION_LINE = "# ...(truncated {n} lines)..." +_REASON_PROTOCOL = "protocol or unknown attribute type" +_REASON_CHAINED_RETURN = "receiver is a return value of unannotated callable" +_REASON_SUBSCRIPT = "receiver is a subscript expression" +_REASON_CONDITIONAL = "receiver is a conditional expression" +_REASON_GETATTR = "dynamic attribute access" +_REASON_INHERITED = "method not found on class; possibly inherited (base not resolved in v1)" +_REASON_REBINDING = "variable '{name}' reassigned; type not stable" +_REASON_COLLISION = "method name matches multiple classes; receiver type not pinned" +_REASON_PARSE = "source file failed to parse" +_TYPING_FORM_NAMES = frozenset( + { + "Literal", + "Annotated", + "Union", + "Optional", + "ClassVar", + "Final", + "TypeVar", + "Any", + "Callable", + "TypeAlias", + "Protocol", + "TypedDict", + "NamedTuple", + "Generic", + "Never", + "Self", + "Type", + } +) + + +@dataclass(frozen=True, slots=True) +class SurfaceRef: + """One mineable surface with optional code location hints.""" + + surface_id: str + kind: str + package: str + name: str + module: str = "" + file: str = "" + line: int = 0 + + +@dataclass(frozen=True, slots=True) +class CalleeRef: + """A call site and optional resolved target.""" + + expression: str + name: str + module: str + file: str + lineno: int + resolved: bool + reason: str = "" + key: str = "" + resolution_kind: ResolutionKind = "unresolved" + + +@dataclass(frozen=True, slots=True) +class _ClassRef: + module: str + class_name: str + file: Path + + +@dataclass +class _VarState: + class_ref: _ClassRef | None = None + stable: bool = True + binding_kind: ResolutionKind = "unresolved" + + +def _callee_key(*, resolved: bool, module: str, name: str, expression: str) -> str: + if resolved: + return unit_key(module, name) + return expression + + +@dataclass(frozen=True, slots=True) +class CodeUnit: + """Collected source for one definition.""" + + key: str + module: str + qualname: str + file: str + lineno: int + source: str + line_count: int + hop: int = 0 + + +@dataclass +class CodeContext: + """Collected code context for one surface.""" + + surface: SurfaceRef + depth: Depth + scope: CalleeScope + max_hops: int + root: CodeUnit | None + units: list[CodeUnit] = field(default_factory=list) + edges: list[tuple[str, str]] = field(default_factory=list) + unresolved: list[CalleeRef] = field(default_factory=list) + truncated: bool = False + no_definition: bool = False + + +@dataclass +class CycleReport: + """Aggregate cycle findings.""" + + cycles: list[list[str]] = field(default_factory=list) + + @property + def count(self) -> int: + return len(self.cycles) + + +def unit_key(module: str, qualname: str) -> str: + return f"{module}:{qualname}" + + +def find_cycles(edges: list[tuple[str, str]]) -> list[list[str]]: + """Return simple cycles as lists of node keys (stdlib DFS).""" + adj: dict[str, list[str]] = {} + nodes: set[str] = set() + for src, dst in edges: + nodes.add(src) + nodes.add(dst) + adj.setdefault(src, []).append(dst) + + cycles: list[list[str]] = [] + visited: set[str] = set() + stack: list[str] = [] + on_stack: set[str] = set() + + def dfs(node: str) -> None: + visited.add(node) + stack.append(node) + on_stack.add(node) + for nxt in adj.get(node, []): + if nxt not in visited: + dfs(nxt) + elif nxt in on_stack: + start = stack.index(nxt) + cycle = stack[start:] + [nxt] + if cycle not in cycles: + cycles.append(cycle) + stack.pop() + on_stack.remove(node) + + for node in sorted(nodes): + if node not in visited: + dfs(node) + return cycles + + +def _package_module_prefix(package: str) -> str: + return package.replace("-", "_") + + +def _scope_allows_module(module: str, *, package: str, scope: CalleeScope) -> bool: + if not module: + return False + pkg_prefix = _package_module_prefix(package) + if module == pkg_prefix or module.startswith(f"{pkg_prefix}."): + return True + return scope == "any-pickled" and ( + module.startswith("pickled_") or module.split(".", 1)[0].startswith("pickled_") + ) + + +def _is_ignored_external(module: str) -> bool: + top = module.split(".", 1)[0] + return top in _STDLIB_TOP_LEVEL or top in _THIRD_PARTY_TOP or top in sys.stdlib_module_names + + +def _read_file(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def _node_source(file_text: str, node: ast.AST) -> str: + segment = ast.get_source_segment(file_text, node) + if segment is not None: + return segment + lines = file_text.splitlines() + start = getattr(node, "lineno", 1) - 1 + end = getattr(node, "end_lineno", start + 1) + return "\n".join(lines[start:end]) + + +def _signature_source(node: ast.FunctionDef | ast.AsyncFunctionDef, file_text: str) -> str: + doc = ast.get_docstring(node) + prefix = "async " if isinstance(node, ast.AsyncFunctionDef) else "" + args = ast.unparse(node.args) + ret = f" -> {ast.unparse(node.returns)}" if node.returns else "" + header = f"{prefix}def {node.name}({args}){ret}" + if doc: + return f'{header}\n """{doc}"""' + return header + + +def _line_count(source: str) -> int: + return len(source.splitlines()) if source else 0 + + +class SourceFileNotFoundError(FileNotFoundError): + """Raised when an inventory-recorded source path is missing on disk.""" + + +def _parse_module(path: Path) -> tuple[str, ast.Module]: + if not path.is_file(): + msg = f"source file not found: {path}" + raise SourceFileNotFoundError(msg) + text = _read_file(path) + tree = ast.parse(text, filename=str(path)) + return text, tree + + +def _find_node_at_line( + tree: ast.Module, + *, + lineno: int, + qualname: str, +) -> ast.FunctionDef | ast.AsyncFunctionDef | ast.ClassDef | None: + if "." in qualname: + class_name, _, method_name = qualname.partition(".") + for node in tree.body: + if isinstance(node, ast.ClassDef) and node.name == class_name: + for item in node.body: + if isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef)): + if item.name == method_name and item.lineno == lineno: + return item + if item.name == method_name: + return item + return None + for node in tree.body: + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): + if node.name == qualname and node.lineno == lineno: + return node + if node.name == qualname: + return node + return None + + +def _local_function( + tree: ast.Module, + name: str, +) -> ast.FunctionDef | ast.AsyncFunctionDef | None: + for node in tree.body: + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name == name: + return node + return None + + +def _build_code_unit( + *, + path: Path, + module: str, + qualname: str, + node: ast.AST, + depth: Depth, + hop: int, +) -> CodeUnit: + file_text, _ = _parse_module(path) + if depth == "signature" and isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + source = _signature_source(node, file_text) + else: + source = _node_source(file_text, node) + key = unit_key(module, qualname) + return CodeUnit( + key=key, + module=module, + qualname=qualname, + file=str(path), + lineno=getattr(node, "lineno", 0), + source=source, + line_count=_line_count(source), + hop=hop, + ) + + +def resolve_definition( + surface: SurfaceRef, + target: Path, +) -> CodeUnit | None: + """Locate the AST node for a surface and build its root code unit.""" + if not surface.file: + return None + path = (target / surface.file).resolve() + if not path.is_file(): + path = Path(surface.file).resolve() + if not path.is_file(): + return None + try: + file_text, tree = _parse_module(path) + except SyntaxError: + return None + qualname = surface.name + if surface.kind == "gate" and "." in surface.name: + qualname = surface.name + node = _find_node_at_line(tree, lineno=surface.line or 0, qualname=qualname) + if node is None: + return None + if isinstance(node, ast.ClassDef): + return None + module = surface.module or _module_name_from_path(path, target) + return _build_code_unit( + path=path, + module=module, + qualname=qualname, + node=node, + depth="body", + hop=0, + ) + + +def _module_name_from_path(path: Path, target: Path) -> str: + try: + rel = path.resolve().relative_to(target.resolve()) + except ValueError: + rel = path + parts = list(rel.parts) + if "src" in parts: + idx = parts.index("src") + parts = parts[idx + 1 :] + if parts and parts[-1].endswith(".py"): + parts[-1] = parts[-1][:-3] + return ".".join(parts) + + +def _import_bindings(tree: ast.Module) -> dict[str, tuple[str, str]]: + bindings: dict[str, tuple[str, str]] = {} + for node in tree.body: + if isinstance(node, ast.Import): + for alias in node.names: + local = alias.asname or alias.name.split(".", 1)[0] + bindings[local] = (alias.name, local) + elif isinstance(node, ast.ImportFrom): + base = node.module or "" + for alias in node.names: + if alias.name == "*": + continue + local = alias.asname or alias.name + full = f"{base}.{alias.name}" if base else alias.name + bindings[local] = (full, local) + return bindings + + +def _enclosing_class(tree: ast.Module, lineno: int) -> ast.ClassDef | None: + best: ast.ClassDef | None = None + for node in tree.body: + if isinstance(node, ast.ClassDef): + start = node.lineno + end = getattr(node, "end_lineno", start) + if start <= lineno <= end: + best = node + return best + + +def _call_expression(func: ast.expr) -> str: + try: + return ast.unparse(func) + except Exception: + return "" + + +def _is_nested_self_dispatch(func: ast.expr) -> bool: + if not isinstance(func, ast.Attribute): + return False + cur: ast.expr = func.value + depth = 0 + while isinstance(cur, ast.Attribute): + depth += 1 + cur = cur.value + return isinstance(cur, ast.Name) and cur.id == "self" and depth >= 1 + + +def _expr(node: ast.AST) -> str: + try: + return ast.unparse(node) + except Exception: + return "" + + +def _unwrap_expr(node: ast.expr) -> ast.expr: + while isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.USub): + node = node.operand + return node + + +def _local_class(tree: ast.Module, name: str) -> ast.ClassDef | None: + for node in tree.body: + if isinstance(node, ast.ClassDef) and node.name == name: + return node + return None + + +def _annotation_class_name(ann: ast.expr | None) -> str | None: + if ann is None: + return None + if isinstance(ann, ast.Subscript): + return None + if isinstance(ann, ast.Name): + if ann.id in _TYPING_FORM_NAMES: + return None + return ann.id + if isinstance(ann, ast.Attribute): + if ann.attr in _TYPING_FORM_NAMES: + return None + return ann.attr + return None + + +def _make_callee( + *, + expression: str, + resolved: bool, + resolution_kind: ResolutionKind, + name: str = "", + module: str = "", + file: str = "", + lineno: int = 0, + reason: str = "", +) -> CalleeRef: + return CalleeRef( + expression=expression, + name=name, + module=module, + file=file, + lineno=lineno, + resolved=resolved, + reason=reason, + resolution_kind=resolution_kind if resolved else "unresolved", + key=_callee_key( + resolved=resolved, + module=module, + name=name, + expression=expression, + ), + ) + + +@dataclass +class _ResolverCtx: + tree: ast.Module + file_path: Path + file_text: str + package: str + scope: CalleeScope + target: Path + unit: CodeUnit + bindings: dict[str, tuple[str, str]] + func: ast.FunctionDef | ast.AsyncFunctionDef + var_states: dict[str, _VarState] + unannotated_params: set[str] + + +def _init_var_states(ctx: _ResolverCtx) -> None: + for arg in ctx.func.args.args: + if arg.annotation is None: + ctx.unannotated_params.add(arg.arg) + continue + cls = _annotation_class_name(arg.annotation) + if not cls: + continue + cref = _resolve_class_name(ctx, cls, kind_hint="annotated_param") + if cref is not None: + ctx.var_states[arg.arg] = _VarState( + class_ref=cref, stable=True, binding_kind="annotated_param" + ) + + for stmt in ctx.func.body: + if isinstance(stmt, ast.AnnAssign) and isinstance(stmt.target, ast.Name): + cls = _annotation_class_name(stmt.annotation) + if cls: + cref = _resolve_class_name(ctx, cls, kind_hint="annotated_var") + if cref is not None: + ctx.var_states[stmt.target.id] = _VarState( + class_ref=cref, stable=True, binding_kind="annotated_var" + ) + if isinstance(stmt, ast.Assign): + for target in stmt.targets: + if isinstance(target, ast.Name): + _apply_assignment(ctx, target.id, stmt.value) + + +def _apply_assignment(ctx: _ResolverCtx, name: str, value: ast.expr) -> None: + value = _unwrap_expr(value) + new_cref = _class_from_constructor_expr(ctx, value) + if name in ctx.var_states: + prev = ctx.var_states[name] + if ( + prev.class_ref is not None + and new_cref is not None + and prev.class_ref.class_name == new_cref.class_name + ): + ctx.var_states[name] = _VarState( + class_ref=new_cref, stable=True, binding_kind="assigned_constructor" + ) + else: + ctx.var_states[name] = _VarState( + class_ref=None, stable=False, binding_kind="unresolved" + ) + return + if new_cref is not None: + ctx.var_states[name] = _VarState( + class_ref=new_cref, stable=True, binding_kind="assigned_constructor" + ) + return + ctx.var_states[name] = _VarState(class_ref=None, stable=False, binding_kind="unresolved") + + +def _source_path_for_object(obj: object) -> Path | None: + if not ( + isinstance(obj, types.ModuleType) + or inspect.isclass(obj) + or inspect.isfunction(obj) + or inspect.ismethod(obj) + ): + return None + try: + file_path = inspect.getsourcefile(obj) + except TypeError: + return None + if file_path is None: + return None + return Path(file_path) + + +def _resolve_class_name( + ctx: _ResolverCtx, + class_name: str, + *, + kind_hint: ResolutionKind, +) -> _ClassRef | None: + if class_name in _TYPING_FORM_NAMES: + return None + local = _local_class(ctx.tree, class_name) + if local is not None: + return _ClassRef(ctx.unit.module, class_name, ctx.file_path) + if class_name in ctx.bindings: + imported, _ = ctx.bindings[class_name] + mod = imported.rsplit(".", 1)[0] if "." in imported else imported + resolved = _resolve_imported( + mod, + class_name, + target=ctx.target, + package=ctx.package, + scope=ctx.scope, + ) + if resolved is not None: + mod_name, _, path = resolved + return _ClassRef(mod_name.rsplit(".", 1)[0], class_name, path) + return None + + +def _class_from_constructor_expr(ctx: _ResolverCtx, node: ast.expr) -> _ClassRef | None: + node = _unwrap_expr(node) + if not isinstance(node, ast.Call): + return None + return _class_from_constructor_call(ctx, node) + + +def _class_from_constructor_call(ctx: _ResolverCtx, call: ast.Call) -> _ClassRef | None: + func = call.func + if isinstance(func, ast.Name): + name = func.id + local = _local_class(ctx.tree, name) + if local is not None: + return _ClassRef(ctx.unit.module, name, ctx.file_path) + if name in ctx.bindings: + imported, _ = ctx.bindings[name] + top = imported.split(".", 1)[0] + if _is_ignored_external(top): + return None + mod_path = imported if "." not in imported else imported.rsplit(".", 1)[0] + resolved = _resolve_imported( + mod_path, + name, + target=ctx.target, + package=ctx.package, + scope=ctx.scope, + ) + if resolved is not None: + mod, _, path = resolved + base_mod = mod.rsplit(".", 1)[0] if "." in mod else mod + return _ClassRef(base_mod, name, path) + if isinstance(func, ast.Attribute) and isinstance(func.value, ast.Name): + mod_alias = func.value.id + if mod_alias in ctx.bindings: + imported, _ = ctx.bindings[mod_alias] + top = imported.split(".", 1)[0] + if _is_ignored_external(top): + return None + resolved = _resolve_imported( + imported, + func.attr, + target=ctx.target, + package=ctx.package, + scope=ctx.scope, + ) + if resolved is not None: + mod, qual, path = resolved + class_name = func.attr + if "." in qual: + class_name = qual.split(".")[-1] + base_mod = mod.rsplit(".", 1)[0] if "." in mod else mod + return _ClassRef(base_mod, class_name, path) + return None + + +def _method_on_class( + ctx: _ResolverCtx, + class_ref: _ClassRef, + method: str, + *, + resolution_kind: ResolutionKind, +) -> CalleeRef | None: + path = class_ref.file + try: + _, tree = _parse_module(path) + except SyntaxError: + return None + found = _find_method_in_class(tree, class_ref.class_name, method) + if found is None: + expr = f"{class_ref.class_name}().{method}" + return _make_callee( + expression=expr, + resolved=False, + resolution_kind="unresolved", + reason=_REASON_INHERITED, + ) + qual, lineno = found + return _make_callee( + expression=f"{class_ref.class_name}.{method}", + resolved=True, + resolution_kind=resolution_kind, + name=qual, + module=class_ref.module, + file=str(path), + lineno=lineno, + ) + + +def _find_method_in_class( + tree: ast.Module, + class_name: str, + method: str, +) -> tuple[str, int] | None: + for node in tree.body: + if not isinstance(node, ast.ClassDef) or node.name != class_name: + continue + for item in node.body: + if isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef)) and item.name == method: + return f"{class_name}.{method}", item.lineno + if isinstance(item, ast.FunctionDef) and item.name == method: + for dec in item.decorator_list: + dec_name = _decorator_name(dec) + if dec_name in {"property", "staticmethod", "classmethod"}: + return f"{class_name}.{method}", item.lineno + return None + + +def _decorator_name(dec: ast.expr) -> str | None: + if isinstance(dec, ast.Name): + return dec.id + if isinstance(dec, ast.Attribute): + return dec.attr + if isinstance(dec, ast.Call) and isinstance(dec.func, ast.Name): + return dec.func.id + if isinstance(dec, ast.Call) and isinstance(dec.func, ast.Attribute): + return dec.func.attr + return None + + +_PATH_METHODS = frozenset( + { + "read_text", + "write_text", + "exists", + "resolve", + "mkdir", + "parent", + "name", + "suffix", + "stem", + "is_file", + "is_dir", + "glob", + "iterdir", + } +) +_STR_METHODS = frozenset( + { + "strip", + "split", + "rsplit", + "join", + "format", + "lower", + "upper", + "startswith", + "endswith", + "replace", + "encode", + "decode", + } +) +_BYTES_METHODS = frozenset({"decode", "split", "strip"}) +_DICT_METHODS = frozenset({"get", "keys", "values", "items", "update", "pop"}) +_LIST_METHODS = frozenset({"append", "extend", "pop", "insert", "sort"}) +_STDLIB_CONSTRUCTORS = frozenset({"Path", "str", "dict", "list", "set", "tuple", "open", "bytes"}) +_BUILTIN_METHODS = _STR_METHODS | _BYTES_METHODS | _DICT_METHODS | _LIST_METHODS | _PATH_METHODS + + +def _stdlib_constructor_name(func: ast.expr) -> str | None: + if isinstance(func, ast.Name) and func.id in _STDLIB_CONSTRUCTORS: + return func.id + if isinstance(func, ast.Attribute) and func.attr in _STDLIB_CONSTRUCTORS: + return func.attr + return None + + +def _receiver_is_intra_project(ctx: _ResolverCtx, receiver: ast.expr) -> bool: + receiver = _unwrap_expr(receiver) + if isinstance(receiver, ast.Name) and receiver.id == "self": + return True + if isinstance(receiver, ast.Name): + var = receiver.id + if ctx.scope != "self" and var in ctx.bindings: + imported, _ = ctx.bindings[var] + if not _is_ignored_external(imported.split(".", 1)[0]): + return True + state = ctx.var_states.get(var) + if state is not None and state.stable and state.class_ref is not None: + return True + if var not in ctx.var_states and _local_class(ctx.tree, var) is not None: + return True + if isinstance(receiver, ast.Call) and _class_from_constructor_call(ctx, receiver): + return True + if ( + isinstance(receiver, ast.Attribute) + and isinstance(receiver.value, ast.Name) + and receiver.value.id in ctx.bindings + ): + imported, _ = ctx.bindings[receiver.value.id] + if not _is_ignored_external(imported.split(".", 1)[0]): + return True + return False + + +def _should_drop_noise_call(ctx: _ResolverCtx, call: ast.Call) -> bool: + """Drop stdlib-surface noise; never drop resolvable intra-project calls.""" + if not isinstance(call.func, ast.Attribute): + return False + method = call.func.attr + receiver = _unwrap_expr(call.func.value) + if _receiver_is_intra_project(ctx, receiver): + return False + if isinstance(receiver, ast.Constant): + if isinstance(receiver.value, str) and method in _STR_METHODS: + return True + if isinstance(receiver.value, bytes) and method in _BYTES_METHODS: + return True + if isinstance(receiver, ast.Call): + ctor = _stdlib_constructor_name(receiver.func) + if ctor == "Path" and method in _PATH_METHODS: + return True + if ctor == "str" and method in _STR_METHODS: + return True + if ctor == "bytes" and method in _BYTES_METHODS: + return True + if ctor == "dict" and method in _DICT_METHODS: + return True + if ctor == "list" and method in _LIST_METHODS: + return True + if ctor == "open": + return True + return method in _BUILTIN_METHODS + + +def _receiver_refusal(receiver: ast.expr) -> str | None: + receiver = _unwrap_expr(receiver) + if isinstance(receiver, ast.Subscript): + return _REASON_SUBSCRIPT + if isinstance(receiver, ast.IfExp): + return _REASON_CONDITIONAL + if isinstance(receiver, ast.Call): + inner = receiver.func + if isinstance(inner, ast.Attribute): + return _REASON_CHAINED_RETURN + if isinstance(inner, ast.Name): + return _REASON_CHAINED_RETURN + return None + + +def _resolve_call_with_ctx(ctx: _ResolverCtx, call: ast.Call) -> CalleeRef | None: + func = call.func + expr = _call_expression(func) + lineno = getattr(call, "lineno", 0) + + if isinstance(func, ast.Name) and func.id == "getattr": + return _make_callee( + expression=_expr(call), + resolved=False, + resolution_kind="unresolved", + reason=_REASON_GETATTR, + ) + if ( + isinstance(func, ast.Call) + and isinstance(func.func, ast.Name) + and func.func.id == "getattr" + ): + return _make_callee( + expression=_expr(call), + resolved=False, + resolution_kind="unresolved", + reason=_REASON_GETATTR, + ) + + if isinstance(func, ast.Attribute) and _is_nested_self_dispatch(func): + return _make_callee( + expression=expr, + resolved=False, + resolution_kind="unresolved", + reason=_REASON_PROTOCOL, + ) + + if isinstance(func, ast.Name): + name = func.id + if _is_ignored_external(name): + return None + if ctx.scope != "self": + local = _local_function(ctx.tree, name) + if local is not None and _scope_allows_module( + ctx.unit.module, package=ctx.package, scope=ctx.scope + ): + return _make_callee( + expression=expr, + resolved=True, + resolution_kind="free_function", + name=local.name, + module=ctx.unit.module, + file=str(ctx.file_path), + lineno=local.lineno, + ) + if ctx.scope == "self": + return None + if name in ctx.bindings: + imported, _ = ctx.bindings[name] + top = imported.split(".", 1)[0] + if _is_ignored_external(top): + return None + mod_base = imported if "." not in imported else imported.rsplit(".", 1)[0] + attr = imported.rsplit(".", 1)[-1] if "." in imported else name + resolved = _resolve_imported( + mod_base, + attr, + target=ctx.target, + package=ctx.package, + scope=ctx.scope, + ) + if resolved is None: + return None + mod, qual, path = resolved + return _make_callee( + expression=expr, + resolved=True, + resolution_kind="free_function", + name=qual, + module=mod, + file=str(path), + lineno=0, + ) + return None + + if not isinstance(func, ast.Attribute): + return None + + if isinstance(func.value, ast.Name) and _is_ignored_external(func.value.id): + return None + + method = func.attr + receiver = _unwrap_expr(func.value) + + if isinstance(receiver, ast.Call): + inner_func = receiver.func + class_ref = _class_from_constructor_call(ctx, receiver) + if class_ref is not None: + kind: ResolutionKind = "constructor_method" + if isinstance(inner_func, ast.Attribute) and isinstance( + inner_func.value, ast.Name + ): + kind = "module_constructor" + return _method_on_class(ctx, class_ref, method, resolution_kind=kind) + if isinstance(inner_func, ast.Attribute): + refusal = _REASON_CHAINED_RETURN + else: + refusal = _REASON_CHAINED_RETURN + return _make_callee( + expression=_expr(call), + resolved=False, + resolution_kind="unresolved", + reason=refusal, + ) + + receiver_refusal = _receiver_refusal(receiver) + if receiver_refusal is not None: + return _make_callee( + expression=_expr(call), + resolved=False, + resolution_kind="unresolved", + reason=receiver_refusal, + ) + + if isinstance(receiver, ast.Name) and receiver.id == "self": + class_node = _enclosing_class(ctx.tree, getattr(call, "lineno", 0)) + if class_node is None: + return None + found = _find_method_in_class(ctx.tree, class_node.name, method) + if found is None: + return _make_callee( + expression=expr, + resolved=False, + resolution_kind="unresolved", + reason=_REASON_INHERITED, + ) + qual, lineno_m = found + return _make_callee( + expression=expr, + resolved=True, + resolution_kind="self_method", + name=qual, + module=ctx.unit.module, + file=str(ctx.file_path), + lineno=lineno_m or lineno, + ) + + if isinstance(receiver, ast.Name): + var = receiver.id + if ctx.scope != "self" and var in ctx.bindings: + imported, _ = ctx.bindings[var] + top = imported.split(".", 1)[0] + if not _is_ignored_external(top): + resolved = _resolve_imported( + imported, + method, + target=ctx.target, + package=ctx.package, + scope=ctx.scope, + ) + if resolved is not None: + mod, qual, path = resolved + return _make_callee( + expression=expr, + resolved=True, + resolution_kind="free_function", + name=qual, + module=mod, + file=str(path), + lineno=0, + ) + state = ctx.var_states.get(var) + if state is None: + local_cls = _local_class(ctx.tree, var) + if local_cls is not None: + cref = _ClassRef(ctx.unit.module, var, ctx.file_path) + return _method_on_class( + ctx, cref, method, resolution_kind="free_function" + ) + if state is None or not state.stable or state.class_ref is None: + if state is not None and not state.stable: + return _make_callee( + expression=expr, + resolved=False, + resolution_kind="unresolved", + reason=_REASON_REBINDING.format(name=var), + ) + if var in ctx.unannotated_params: + return _make_callee( + expression=expr, + resolved=False, + resolution_kind="unresolved", + reason=f"parameter '{var}' has no type annotation", + ) + return _make_callee( + expression=expr, + resolved=False, + resolution_kind="unresolved", + reason=_REASON_COLLISION, + ) + kind = state.binding_kind + if kind == "unresolved": + kind = "annotated_var" + return _method_on_class(ctx, state.class_ref, method, resolution_kind=kind) + + if ( + isinstance(receiver, ast.Attribute) + and isinstance(receiver.value, ast.Name) + and receiver.value.id in ctx.bindings + ): + imported, _ = ctx.bindings[receiver.value.id] + top = imported.split(".", 1)[0] + if not _is_ignored_external(top): + resolved = _resolve_imported( + imported, + receiver.attr, + target=ctx.target, + package=ctx.package, + scope=ctx.scope, + ) + if resolved is not None: + mod, qual, path = resolved + return _make_callee( + expression=expr, + resolved=True, + resolution_kind="module_constructor", + name=qual, + module=mod, + file=str(path), + lineno=0, + ) + + return _make_callee( + expression=expr, + resolved=False, + resolution_kind="unresolved", + reason=_REASON_COLLISION, + ) + + +def _resolve_imported( + module_path: str, + attr: str, + *, + target: Path, + package: str, + scope: CalleeScope, +) -> tuple[str, str, Path] | None: + obj: object | None = None + try: + obj = importlib.import_module(module_path) + except Exception: + parts = module_path.split(".") + if not parts: + return None + try: + obj = importlib.import_module(parts[0]) + except Exception: + return None + for part in parts[1:]: + obj = getattr(obj, part, None) + if obj is None: + return None + for part in attr.split("."): + obj = getattr(obj, part, None) + if obj is None: + return None + path = _source_path_for_object(obj) + if path is None: + return None + resolved_module = getattr(obj, "__module__", module_path) or module_path + if not _scope_allows_module(resolved_module, package=package, scope=scope): + return None + qual = getattr(obj, "__qualname__", attr) + return resolved_module, qual, path + + +def _resolve_call_target( + *, + call: ast.Call, + tree: ast.Module, + file_path: Path, + file_text: str, + package: str, + scope: CalleeScope, + target: Path, + unit: CodeUnit, +) -> CalleeRef | None: + node = _find_node_at_line(tree, lineno=unit.lineno, qualname=unit.qualname) + if node is None or not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + return None + ctx = _ResolverCtx( + tree=tree, + file_path=file_path, + file_text=file_text, + package=package, + scope=scope, + target=target, + unit=unit, + bindings=_import_bindings(tree), + func=node, + var_states={}, + unannotated_params=set(), + ) + _init_var_states(ctx) + return _resolve_call_with_ctx(ctx, call) + + +def resolve_callees( + unit: CodeUnit, + *, + scope: CalleeScope, + package: str, + target: Path, +) -> list[CalleeRef]: + """Find callees invoked from a code unit's definition.""" + path = Path(unit.file) + try: + file_text, tree = _parse_module(path) + except SyntaxError: + return [] + node = _find_node_at_line(tree, lineno=unit.lineno, qualname=unit.qualname) + if node is None or not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + return [] + + ctx = _ResolverCtx( + tree=tree, + file_path=path, + file_text=file_text, + package=package, + scope=scope, + target=target, + unit=unit, + bindings=_import_bindings(tree), + func=node, + var_states={}, + unannotated_params=set(), + ) + _init_var_states(ctx) + + refs: list[CalleeRef] = [] + for block in node.body: + for child in ast.walk(block): + if not isinstance(child, ast.Call): + continue + if _should_drop_noise_call(ctx, child): + continue + ref = _resolve_call_with_ctx(ctx, child) + if ref is not None: + refs.append(ref) + return refs + + +def load_code_unit( + callee: CalleeRef, + *, + target: Path, + depth: Depth, + hop: int, +) -> CodeUnit | None: + if not callee.resolved: + return None + path = Path(callee.file) + if not path.is_file(): + return None + _, tree = _parse_module(path) + node = _find_node_at_line(tree, lineno=callee.lineno or 0, qualname=callee.name) + if node is None: + for n in ast.walk(tree): + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) and ( + n.name == callee.name.split(".")[-1] + ): + node = n + break + if node is None or isinstance(node, ast.ClassDef): + return None + return _build_code_unit( + path=path, + module=callee.module, + qualname=callee.name, + node=node, + depth=depth, + hop=hop, + ) + + +def collect_context( + surface: SurfaceRef, + target: Path, + *, + depth: Depth, + scope: CalleeScope, + max_hops: int, + max_callees: int, + max_code_lines: int, + package: str, +) -> CodeContext: + """BFS-collect source units for a surface with caps and cycle-safe visited set.""" + ctx = CodeContext( + surface=surface, + depth=depth, + scope=scope, + max_hops=max_hops, + root=None, + ) + root = resolve_definition(surface, target) + if root is None: + ctx.no_definition = True + return ctx + + if depth == "signature": + path = Path(root.file) + file_text, tree = _parse_module(path) + node = _find_node_at_line(tree, lineno=root.lineno, qualname=root.qualname) + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + sig = _signature_source(node, file_text) + root = CodeUnit( + key=root.key, + module=root.module, + qualname=root.qualname, + file=root.file, + lineno=root.lineno, + source=sig, + line_count=_line_count(sig), + hop=0, + ) + + ctx.root = root + visited: set[str] = set() + collected: list[CodeUnit] = [] + edges: list[tuple[str, str]] = [] + unresolved: dict[str, CalleeRef] = {} + total_lines = 0 + truncated = False + + queue: deque[tuple[CodeUnit, int]] = deque([(root, 0)]) + + while queue and len(collected) < max_callees and total_lines < max_code_lines: + unit, hop = queue.popleft() + if unit.key in visited: + continue + visited.add(unit.key) + + unit_lines = unit.line_count + if total_lines + unit_lines > max_code_lines and collected: + truncated = True + break + if total_lines + unit_lines > max_code_lines: + remaining = max_code_lines - total_lines + lines = unit.source.splitlines() + omitted = max(0, len(lines) - remaining) + clipped = ( + "\n".join(lines[:remaining]) + + "\n" + + _TRUNCATION_LINE.format(n=omitted) + ) + unit = CodeUnit( + key=unit.key, + module=unit.module, + qualname=unit.qualname, + file=unit.file, + lineno=unit.lineno, + source=clipped, + line_count=remaining + 1, + hop=unit.hop, + ) + truncated = True + + collected.append(unit) + total_lines += unit.line_count + + if truncated: + break + + if depth != "callgraph" or hop >= max_hops: + continue + + for callee in resolve_callees( + unit, scope=scope, package=package, target=target + ): + if not callee.resolved: + unresolved[callee.expression] = callee + continue + edges.append((unit.key, callee.key)) + if callee.key in visited: + continue + if len(collected) + len(queue) >= max_callees: + truncated = True + continue + loaded = load_code_unit(callee, target=target, depth=depth, hop=hop + 1) + if loaded is not None: + queue.append((loaded, hop + 1)) + + if queue and not truncated: + truncated = True + + ctx.units = collected + ctx.edges = edges + ctx.unresolved = list(unresolved.values()) + ctx.truncated = truncated + return ctx + + +def resolve_cli_surface( + *, + package: str, + command_full_name: str, + scripts: dict[str, str], + target: Path, +) -> SurfaceRef | None: + """Resolve a Click command callback to a file/line surface ref.""" + for script_target in scripts.values(): + module_name, sep, attr = str(script_target).partition(":") + if not sep: + continue + try: + imported = importlib.import_module(module_name) + root = getattr(imported, attr) + except Exception: + continue + cmd = _find_click_command(root, command_full_name) + if cmd is None: + continue + callback = cmd.callback + if callback is None: + continue + file_path = inspect.getsourcefile(callback) + if file_path is None: + continue + lines, start = inspect.getsourcelines(callback) + rel = file_path + try: + rel = str(Path(file_path).resolve().relative_to(target.resolve())) + except ValueError: + rel = str(file_path) + callback_module = str( + getattr(callback, "__module__", module_name) or module_name + ) + return SurfaceRef( + surface_id="", + kind="cli_command", + package=package, + name=command_full_name, + module=callback_module, + file=rel, + line=start, + ) + return None + + +def _find_click_command(root: click.Command, full_name: str) -> click.Command | None: + parts = full_name.strip().split() + cmd: click.Command = root + for part in parts: + if not isinstance(cmd, click.Group): + return None + nxt = cmd.commands.get(part) + if nxt is None: + return None + cmd = nxt + return cmd + + +def iter_surfaces_from_inventory( + data: dict[str, object], + target: Path, +) -> Iterator[SurfaceRef]: + """Yield surfaces that may have code definitions.""" + from pickled_core.mine.stories_stage import ( + _is_significant_cli, + _surface_id_cli, + _surface_id_gate, + _surface_id_mcp, + ) + + packages = data.get("packages", {}) + if not isinstance(packages, dict): + return + + for pkg_name, pkg in packages.items(): + if not isinstance(pkg, dict): + continue + scripts = pkg.get("scripts", {}) + if not isinstance(scripts, dict): + scripts = {} + + for tool in pkg.get("mcp_tools", []): + if not isinstance(tool, dict): + continue + name = str(tool.get("name", "")) + if not name: + continue + yield SurfaceRef( + surface_id=_surface_id_mcp(name), + kind="mcp_tool", + package=str(pkg_name), + name=name, + ) + + for cmd in pkg.get("cli_commands", []): + if not isinstance(cmd, dict) or not _is_significant_cli(cmd): + continue + full = str(cmd.get("full_name", "")) + ref = resolve_cli_surface( + package=str(pkg_name), + command_full_name=full, + scripts={str(k): str(v) for k, v in scripts.items()}, + target=target, + ) + if ref is None: + yield SurfaceRef( + surface_id=_surface_id_cli(str(pkg_name), full), + kind="cli_command", + package=str(pkg_name), + name=full, + ) + else: + yield SurfaceRef( + surface_id=_surface_id_cli(str(pkg_name), full), + kind=ref.kind, + package=ref.package, + name=ref.name, + module=ref.module, + file=ref.file, + line=ref.line, + ) + + for gate in pkg.get("gates", []): + if not isinstance(gate, dict): + continue + gate_name = str(gate.get("name", gate.get("class_name", ""))) + if not gate_name: + continue + yield SurfaceRef( + surface_id=_surface_id_gate(str(pkg_name), gate_name), + kind="gate", + package=str(pkg_name), + name=gate_name, + module=str(gate.get("module", "")), + file=str(gate.get("file", "")), + line=int(gate.get("line", 0) or 0), + ) + + +__all__ = [ + "CalleeRef", + "CalleeScope", + "CodeContext", + "CodeUnit", + "CycleReport", + "Depth", + "SurfaceRef", + "collect_context", + "find_cycles", + "iter_surfaces_from_inventory", + "load_code_unit", + "resolve_callees", + "resolve_cli_surface", + "resolve_definition", + "SourceFileNotFoundError", + "unit_key", +] diff --git a/packages/pickled-core/src/pickled_core/mine/code_stage.py b/packages/pickled-core/src/pickled_core/mine/code_stage.py new file mode 100644 index 0000000..5fc1dce --- /dev/null +++ b/packages/pickled-core/src/pickled_core/mine/code_stage.py @@ -0,0 +1,194 @@ +"""Stage: extract per-surface code context from inventory.""" + +from __future__ import annotations + +import sys +from pathlib import Path + +from pickled_core.mine.code_reader import ( + CalleeScope, + CodeContext, + CycleReport, + Depth, + collect_context, + find_cycles, + iter_surfaces_from_inventory, +) +from pickled_core.mine.io import ( + ensure_output_dir, + require_inventory_json, + surface_matches, + write_json, + write_text, +) +from pickled_core.mine.types import CodeStageResult, InventoryResult + + +def _render_code_context_markdown( + ctx: CodeContext, + *, + max_hops: int, +) -> str: + surface = ctx.surface + lines: list[str] = [ + f"# Code context: {surface.name}", + "", + f"- **Surface id:** {surface.surface_id}", + f"- **Depth:** {ctx.depth} | **Scope:** {ctx.scope} | **Hops:** {max_hops}", + ] + total_lines = sum(u.line_count for u in ctx.units) + lines.append( + f"- **Units collected:** {len(ctx.units)} | **Total lines:** {total_lines} " + f"| **Truncated:** {ctx.truncated}" + ) + lines.append("") + + if ctx.no_definition: + lines.extend( + [ + "## Notes", + "", + "No code definition resolved for this surface.", + "", + ] + ) + return "\n".join(lines) + + root = ctx.root + if root is None: + lines.extend(["## Notes", "", "No code definition resolved for this surface.", ""]) + return "\n".join(lines) + + lines.extend( + [ + f"## Root: {root.module}.{root.qualname}", + "", + "```python", + root.source, + "```", + "", + ] + ) + + for unit in ctx.units: + if unit.key == root.key: + continue + lines.extend( + [ + f"## Callee: {unit.module}.{unit.qualname} (hop {unit.hop})", + "", + "```python", + unit.source, + "```", + "", + ] + ) + + if ctx.unresolved: + lines.append("## Unresolved callees") + lines.append("") + for ref in ctx.unresolved: + reason = ref.reason or "not statically resolvable" + lines.append(f"- `{ref.expression}` — {reason}") + lines.append("") + + if ctx.truncated: + lines.extend( + [ + "## Notes", + "", + "Collection stopped early because of --max-callees or --max-code-lines.", + "", + ] + ) + + return "\n".join(lines) + + +def run_code( + inventory: InventoryResult, + target: Path, + output_dir: Path, + *, + depth: Depth = "body", + scope: CalleeScope = "same-package", + max_hops: int = 1, + max_callees: int = 8, + max_code_lines: int = 400, + detect_cycles: bool = True, + surfaces: tuple[str, ...] = (), + verbose: bool = False, +) -> CodeStageResult: + """Write ``code-context/.md`` for each selected surface.""" + target = target.resolve() + paths = ensure_output_dir(output_dir) + code_dir = paths.code_context_dir + code_dir.mkdir(parents=True, exist_ok=True) + + all_edges: list[tuple[str, str]] = [] + written: list[Path] = [] + + for surface in iter_surfaces_from_inventory(inventory.data, target): + if not surface_matches( + surface_id=surface.surface_id, + package=surface.package, + tokens=surfaces, + ): + continue + + ctx = collect_context( + surface, + target, + depth=depth, + scope=scope, + max_hops=max_hops, + max_callees=max_callees, + max_code_lines=max_code_lines, + package=surface.package, + ) + all_edges.extend(ctx.edges) + out_path = code_dir / f"{surface.surface_id}.md" + body = _render_code_context_markdown(ctx, max_hops=max_hops) + write_text(out_path, body) + written.append(out_path) + if verbose: + sys.stderr.write(f"[INFO] Wrote {out_path}\n") + + cycles_path: Path | None = None + cycle_count = 0 + if detect_cycles: + cycles = find_cycles(all_edges) + cycle_count = len(cycles) + cycles_path = code_dir / "_cycles.json" + report = CycleReport(cycles=cycles) + write_json( + cycles_path, + {"cycles": report.cycles, "count": report.count}, + ) + if verbose: + sys.stderr.write( + f"[INFO] Wrote {cycles_path} ({cycle_count} cycle(s))\n" + ) + + return CodeStageResult( + output_dir=paths.root, + code_context_dir=code_dir, + written_paths=written, + cycles_path=cycles_path, + cycle_count=cycle_count, + ) + + +def load_inventory_for_code(output_dir: Path) -> InventoryResult: + """Load inventory or raise :class:`MissingStageInputError`.""" + data = require_inventory_json(output_dir, needed_by="code") + paths = ensure_output_dir(output_dir) + warnings = data.get("warnings", []) if isinstance(data.get("warnings"), list) else [] + return InventoryResult( + inventory_path=paths.inventory_json, + data=data, + warnings=[str(w) for w in warnings], + ) + + +__all__ = ["load_inventory_for_code", "run_code"] diff --git a/packages/pickled-core/src/pickled_core/mine/errors.py b/packages/pickled-core/src/pickled_core/mine/errors.py new file mode 100644 index 0000000..f55debb --- /dev/null +++ b/packages/pickled-core/src/pickled_core/mine/errors.py @@ -0,0 +1,27 @@ +"""Actionable errors for the mining pipeline.""" + +from __future__ import annotations + +from pathlib import Path + + +class MineError(Exception): + """Base for mining pipeline errors with an actionable message.""" + + +class MissingStageInputError(MineError): + """A stage's required input from an earlier stage is absent.""" + + def __init__(self, missing_path: Path, needed_by: str, produced_by: str) -> None: + self.missing_path = missing_path + self.needed_by = needed_by + self.produced_by = produced_by + super().__init__( + f"{needed_by} requires {missing_path.name}, which is produced by " + f"`pickled-spec mine {produced_by}`. Run that stage first, or use " + f"`pickled-spec mine all` to run the whole pipeline. " + f"(looked in: {missing_path.parent})" + ) + + +__all__ = ["MineError", "MissingStageInputError"] diff --git a/packages/pickled-core/src/pickled_core/mine/evaluate_stage.py b/packages/pickled-core/src/pickled_core/mine/evaluate_stage.py new file mode 100644 index 0000000..c1bd051 --- /dev/null +++ b/packages/pickled-core/src/pickled_core/mine/evaluate_stage.py @@ -0,0 +1,196 @@ +"""Stage 5: run coverage and ambiguity gates over generated features.""" + +from __future__ import annotations + +import sys +from collections.abc import Callable +from pathlib import Path +from typing import Any + +from pickled_bdd.adapters.pytest_bdd import PytestBddAdapter +from pickled_rules.gates.coverage import coverage_gate_features + +from pickled_core import GateResult +from pickled_core.llm.base import LLMClient +from pickled_core.mine.io import ( + ensure_output_dir, + require_features_dir, + surface_matches, + write_json, +) +from pickled_core.mine.types import ( + AmbiguityFeatureResult, + CoverageRulesetResult, + EvaluationResult, + RulesetSources, +) + + +def _import_run_ambiguity_gate() -> ( + Callable[[str | Path, LLMClient | None], GateResult] | None +): + try: + from pickled_bdd.cli import run_ambiguity_gate + + return run_ambiguity_gate + except ImportError: # pragma: no cover + return None + + +def _surface_id_from_feature(path: Path) -> str: + return path.stem + + +def _package_hint_from_surface_id(surface_id: str) -> str: + if "_" in surface_id: + return surface_id.split("_", 1)[0] + return surface_id + + +def _filter_feature_paths( + feature_paths: list[Path], + surfaces: tuple[str, ...], +) -> list[Path]: + if not surfaces: + return feature_paths + filtered: list[Path] = [] + for path in feature_paths: + surface_id = _surface_id_from_feature(path) + package = _package_hint_from_surface_id(surface_id) + if surface_matches(surface_id=surface_id, package=package, tokens=surfaces): + filtered.append(path) + return filtered + + +def run_evaluate( + output_dir: Path, + *, + ruleset_sources: RulesetSources | None, + llm: LLMClient | None, + surfaces: tuple[str, ...] = (), +) -> EvaluationResult: + """Run coverage and ambiguity gates over generated features.""" + paths = ensure_output_dir(output_dir) + features_dir = require_features_dir(output_dir, needed_by="evaluate") + feature_paths = _filter_feature_paths( + sorted(features_dir.glob("*.feature")), + surfaces, + ) + if not feature_paths: + msg = "no features match --surfaces filter" + raise ValueError(msg) + + coverage_entries: list[CoverageRulesetResult] + if ruleset_sources is None: + sys.stderr.write("[WARN] evaluate: no rule sets configured; coverage skipped\n") + coverage_entries = [] + else: + adapter = PytestBddAdapter() + parsed = [] + for path in feature_paths: + try: + parsed.append(adapter.parse_feature_file(path)) + except Exception as exc: + sys.stderr.write( + f"[WARN] evaluate: skip unparseable feature {path.name}: " + f"{type(exc).__name__}: {exc}\n" + ) + coverage_entries = [] + for ruleset_entry in ruleset_sources.rulesets: + report = coverage_gate_features( + parsed, + ruleset_entry.ruleset, + ruleset_short_name=ruleset_entry.short_name, + ) + strict_unref = [ + r.id + for r in report.unreferenced_rules + if r.enforcement == "strict" + ] + coverage_entries.append( + CoverageRulesetResult( + short_name=ruleset_entry.short_name, + verdict=report.gate_result.verdict.value, + notes=report.gate_result.notes or "", + referenced_rule_ids=sorted(r.id for r in report.referenced_rules), + unreferenced_strict_rule_ids=sorted(strict_unref), + unknown_references=[ + {"ruleset": rs, "rule_id": rid} + for rs, rid in report.unknown_references + ], + ) + ) + + ambiguity_runner = _import_run_ambiguity_gate() + if ambiguity_runner is None: + msg = "pickled-bdd is not installed; install pickled-core[mine]" + raise RuntimeError(msg) + + ambiguity_entries: list[AmbiguityFeatureResult] = [] + for path in feature_paths: + try: + result = ambiguity_runner(path, llm) + except Exception as exc: + sys.stderr.write( + f"[WARN] evaluate: ambiguity skipped for {path.name}: " + f"{type(exc).__name__}: {exc}\n" + ) + ambiguity_entries.append( + AmbiguityFeatureResult( + feature_path=str(path.relative_to(paths.root)), + verdict="error", + finding_count=0, + skipped=True, + notes=f"unparseable feature: {exc}", + ) + ) + continue + skipped = llm is None + ambiguity_entries.append( + AmbiguityFeatureResult( + feature_path=str(path.relative_to(paths.root)), + verdict=result.verdict.value, + finding_count=len(result.findings), + skipped=skipped, + notes=result.notes or "", + ) + ) + + coverage_doc: dict[str, Any] = {"schema_version": "1", "rulesets": []} + for cov in coverage_entries: + coverage_doc["rulesets"].append( + { + "short_name": cov.short_name, + "verdict": cov.verdict, + "notes": cov.notes, + "referenced_rule_ids": cov.referenced_rule_ids, + "unreferenced_strict_rule_ids": cov.unreferenced_strict_rule_ids, + "unknown_references": cov.unknown_references, + } + ) + + ambiguity_doc: dict[str, Any] = {"schema_version": "1", "features": []} + for amb in ambiguity_entries: + ambiguity_doc["features"].append( + { + "feature": amb.feature_path, + "verdict": amb.verdict, + "finding_count": amb.finding_count, + "skipped": amb.skipped, + "notes": amb.notes, + } + ) + + write_json(paths.coverage_json, coverage_doc) + write_json(paths.ambiguity_json, ambiguity_doc) + + return EvaluationResult( + coverage_path=paths.coverage_json, + ambiguity_path=paths.ambiguity_json, + coverage=coverage_entries, + ambiguity=ambiguity_entries, + surfaces_filter=surfaces, + ) + + +__all__ = ["run_evaluate"] diff --git a/packages/pickled-core/src/pickled_core/mine/features_stage.py b/packages/pickled-core/src/pickled_core/mine/features_stage.py new file mode 100644 index 0000000..49f1ab1 --- /dev/null +++ b/packages/pickled-core/src/pickled_core/mine/features_stage.py @@ -0,0 +1,198 @@ +"""Stage 3: draft Gherkin features from user stories.""" + +from __future__ import annotations + +import asyncio +import sys +from pathlib import Path + +from pickled_core.llm.base import LLMClient +from pickled_core.mine.io import ensure_output_dir, require_stories_dir, surface_matches +from pickled_core.mine.types import FeatureResult, FeaturesStageResult + +try: + from pickled_bdd.drafter import FeatureDrafter +except ImportError: # pragma: no cover - optional extra + FeatureDrafter = None # type: ignore[misc, assignment] # optional extra not installed + + +def _story_surface_id(path: Path) -> str: + name = path.name + if name.endswith(".story.md"): + return name[: -len(".story.md")] + return path.stem + + +def _package_from_story(path: Path) -> str: + text = path.read_text(encoding="utf-8") + for line in text.splitlines(): + if line.startswith("- **Package:**"): + return line.split(":**", 1)[-1].strip() + surface_id = _story_surface_id(path) + return surface_id.split("_", 1)[0] if "_" in surface_id else surface_id + + +def _filter_story_paths( + story_paths: list[Path], + surfaces: tuple[str, ...], +) -> list[Path]: + if not surfaces: + return story_paths + filtered: list[Path] = [] + for story_path in story_paths: + surface_id = _story_surface_id(story_path) + package = _package_from_story(story_path) + if surface_matches(surface_id=surface_id, package=package, tokens=surfaces): + filtered.append(story_path) + return filtered + + +async def _draft_one( + story_path: Path, + feature_path: Path, + llm: LLMClient, + *, + overwrite: bool, + interactive: bool, +) -> FeatureResult: + surface_id = _story_surface_id(story_path) + if feature_path.is_file() and not overwrite: + return FeatureResult( + surface_id=surface_id, + feature_path=feature_path, + story_path=story_path, + skipped=True, + ) + + if FeatureDrafter is None: + msg = "pickled-bdd is not installed; install pickled-core[mine]" + raise RuntimeError(msg) + + story_text = story_path.read_text(encoding="utf-8") + drafter = FeatureDrafter(llm) + + while True: + result = drafter.draft_from_story(story_text) + feature_text = result.text + if not interactive: + feature_path.write_text(feature_text + "\n", encoding="utf-8") + return FeatureResult( + surface_id=surface_id, + feature_path=feature_path, + story_path=story_path, + skipped=False, + ) + + sys.stderr.write(f"\n--- draft for {surface_id} ---\n") + sys.stderr.write(feature_text[:2000]) + if len(feature_text) > 2000: + sys.stderr.write("\n…\n") + sys.stderr.write("\n") + choice = input("accept / skip / re-draft: ").strip().lower() + if choice == "skip": + return FeatureResult( + surface_id=surface_id, + feature_path=feature_path, + story_path=story_path, + skipped=True, + ) + if choice in {"accept", "a", "y", "yes"}: + feature_path.write_text(feature_text + "\n", encoding="utf-8") + return FeatureResult( + surface_id=surface_id, + feature_path=feature_path, + story_path=story_path, + skipped=False, + ) + if choice in {"re-draft", "redraft", "r"}: + continue + sys.stderr.write("Unknown choice; type accept, skip, or re-draft.\n") + + +async def _run_parallel( + jobs: list[tuple[Path, Path]], + llm: LLMClient, + *, + overwrite: bool, + max_parallel: int, +) -> list[FeatureResult]: + sem = asyncio.Semaphore(max_parallel) + results: list[FeatureResult] = [] + + async def _one(story_path: Path, feature_path: Path) -> None: + async with sem: + results.append( + await _draft_one( + story_path, + feature_path, + llm, + overwrite=overwrite, + interactive=False, + ) + ) + + await asyncio.gather(*[_one(s, f) for s, f in jobs]) + return sorted(results, key=lambda r: r.surface_id) + + +def run_features( + output_dir: Path, + *, + llm: LLMClient | None, + quick: bool, + overwrite: bool, + surfaces: tuple[str, ...] = (), + max_parallel: int = 4, +) -> FeaturesStageResult: + """Draft ``.feature`` files from stories under ``output_dir``.""" + paths = ensure_output_dir(output_dir) + stories_dir = require_stories_dir(output_dir, needed_by="features") + + if llm is None: + sys.stderr.write("[WARN] features stage requires an LLM; skipped\n") + return FeaturesStageResult( + output_dir=paths.root, + results=[], + skipped_entire_stage=True, + warnings=["features stage requires an LLM; skipped"], + ) + + story_paths = _filter_story_paths( + sorted(stories_dir.glob("*.story.md")), + surfaces, + ) + if not story_paths: + msg = "no stories match --surfaces filter" + raise ValueError(msg) + + jobs = [ + ( + story_path, + paths.features_dir / f"{_story_surface_id(story_path)}.feature", + ) + for story_path in story_paths + ] + + if quick: + results = asyncio.run( + _run_parallel(jobs, llm, overwrite=overwrite, max_parallel=max_parallel) + ) + else: + results = [] + for story_path, feature_path in jobs: + results.append( + asyncio.run( + _draft_one( + story_path, + feature_path, + llm, + overwrite=overwrite, + interactive=True, + ) + ) + ) + + return FeaturesStageResult(output_dir=paths.root, results=results, skipped_entire_stage=False) + + +__all__ = ["run_features"] diff --git a/packages/pickled-core/src/pickled_core/mine/inventory_lib.py b/packages/pickled-core/src/pickled_core/mine/inventory_lib.py index 7acdcf8..ecd0e50 100644 --- a/packages/pickled-core/src/pickled_core/mine/inventory_lib.py +++ b/packages/pickled-core/src/pickled_core/mine/inventory_lib.py @@ -200,22 +200,73 @@ def _mcp_namespace_map(packages: dict[str, dict[str, Any]]) -> dict[str, str]: return mapping -def _pickled_mcp_stdio_args(target: Path) -> list[str] | None: - root_toml = target / "pyproject.toml" - if not root_toml.is_file(): - return None - data = _load_toml(root_toml) - project = data.get("project", {}) +def _scripts_dict(data: dict[str, Any]) -> dict[str, str]: + project = data.get("project") if not isinstance(project, dict): + return {} + scripts = project.get("scripts") + if not isinstance(scripts, dict): + return {} + return {str(k): str(v) for k, v in scripts.items()} + + +def _has_mcp_subservers_entry_point(data: dict[str, Any]) -> bool: + sub = _project_entry_points(data).get("pickled.mcp.subservers", {}) + return isinstance(sub, dict) and bool(sub) + + +def _pick_umbrella_script_name(scripts: dict[str, str]) -> str | None: + if "pickled-spec" in scripts: + return "pickled-spec" + for name in sorted(scripts): + lower = name.lower() + if lower == "mcp" or lower.endswith("-spec"): + return name + return None + + +def discover_umbrella_mcp_launch(target: Path) -> tuple[str, Path] | None: + """Find umbrella MCP CLI script on root or workspace member packages. + + Returns ``(script_name, run_directory)`` for ``uv run --directory``. + """ + target = target.resolve() + root_toml = target / "pyproject.toml" + if root_toml.is_file(): + root_data = _load_toml(root_toml) + scripts = _scripts_dict(root_data) + picked = _pick_umbrella_script_name(scripts) + if picked and ( + picked == "pickled-spec" or _has_mcp_subservers_entry_point(root_data) + ): + return picked, target + + for pkg_dir in expand_packages(target): + if pkg_dir.resolve() == target.resolve(): + continue + pkg_toml = pkg_dir / "pyproject.toml" + if not pkg_toml.is_file(): + continue + data = _load_toml(pkg_toml) + scripts = _scripts_dict(data) + picked = _pick_umbrella_script_name(scripts) + if picked is None: + continue + if picked == "pickled-spec" or _has_mcp_subservers_entry_point(data): + return picked, target + return None + + +def mcp_stdio_uv_args(target: Path) -> list[str] | None: + launch = discover_umbrella_mcp_launch(target) + if launch is None: return None - scripts = project.get("scripts", {}) - if not isinstance(scripts, dict) or "pickled-spec" not in scripts: - return None + script_name, run_dir = launch return [ "run", "--directory", - str(target), - "pickled-spec", + str(run_dir), + script_name, "mcp", "--transport", "stdio", @@ -226,9 +277,9 @@ async def _list_mcp_tools_async(target: Path, timeout: float) -> list[dict[str, from mcp.client.session import ClientSession from mcp.client.stdio import StdioServerParameters, stdio_client - args = _pickled_mcp_stdio_args(target) + args = mcp_stdio_uv_args(target) if args is None: - msg = "no pickled-spec MCP entry point in target pyproject.toml" + msg = "no umbrella MCP entry point found in target" raise RuntimeError(msg) params = StdioServerParameters(command="uv", args=args, cwd=str(target)) @@ -672,9 +723,9 @@ def build_inventory( inv.packages = _collect_packages(target, verbose) if "mcp" in include and not no_mcp: - if _pickled_mcp_stdio_args(target) is None: - log("WARN", "MCP tools skipped (no pickled-spec script in target)") - inv.warnings.append("MCP tools skipped: target has no pickled-spec entry point") + if discover_umbrella_mcp_launch(target) is None: + log("WARN", "MCP tools skipped (no umbrella MCP entry point found in target)") + inv.warnings.append("no umbrella MCP entry point found in target") else: try: tools = asyncio.run(_list_mcp_tools_async(target, mcp_timeout)) @@ -714,4 +765,12 @@ def parse_include(raw: str) -> set[str]: return parts -__all__ = ["Inventory", "build_inventory", "expand_packages", "parse_include", "walk_click_group"] +__all__ = [ + "Inventory", + "build_inventory", + "discover_umbrella_mcp_launch", + "expand_packages", + "mcp_stdio_uv_args", + "parse_include", + "walk_click_group", +] diff --git a/packages/pickled-core/src/pickled_core/mine/inventory_stage.py b/packages/pickled-core/src/pickled_core/mine/inventory_stage.py index 8e38684..bd16f12 100644 --- a/packages/pickled-core/src/pickled_core/mine/inventory_stage.py +++ b/packages/pickled-core/src/pickled_core/mine/inventory_stage.py @@ -2,12 +2,294 @@ from __future__ import annotations +import ast +import re from pathlib import Path +from typing import Any from pickled_core.mine import inventory_lib +from pickled_core.mine.inventory_lib import discover_umbrella_mcp_launch from pickled_core.mine.io import ensure_output_dir, write_json from pickled_core.mine.types import InventoryResult +_ADR_TITLE_PREFIX_RE = re.compile( + r"^ADR[-\s]?(\d{4})\s*:\s*", + re.IGNORECASE, +) +_STATUS_FRONTMATTER_RE = re.compile( + r"^\s*[-*]?\s*(?:\*\*)?Status(?:\*\*)?\s*:\s*(.+?)\s*$", + re.IGNORECASE | re.MULTILINE, +) +_STATUS_MARKDOWN_RE = re.compile(r"[*_`]+") +_GLOBAL_ADR_TITLE_MARKERS = ("workspace", "monorepo", "pickled-core") +_DOCSTRING_CAP = 500 + + +def _normalize_adr_status(raw: str) -> str: + text = raw.strip().lstrip("- ").strip() + if ":" in text: + text = text.split(":", 1)[1].strip() + text = _STATUS_MARKDOWN_RE.sub("", text).strip() + return text or "unknown" + + +def _relative_path(target: Path, path: Path) -> str: + try: + return str(path.relative_to(target)) + except ValueError: + return str(path) + + +def _first_paragraph_docstring( + node: ast.ClassDef | ast.FunctionDef | ast.AsyncFunctionDef, +) -> str: + doc = ast.get_docstring(node) + if not doc: + return "" + paragraph = doc.strip().split("\n\n")[0].replace("\n", " ").strip() + if len(paragraph) > _DOCSTRING_CAP: + return paragraph[: _DOCSTRING_CAP - 3] + "..." + return paragraph + + +def parse_adr_file(target: Path, path: Path) -> dict[str, Any]: + """Parse one ADR markdown file with clean title and status.""" + text = path.read_text(encoding="utf-8") + lines = text.splitlines() + number_match = re.match(r"^(\d{4})", path.stem) + number = number_match.group(1) if number_match else path.stem[:4] + + title = "" + for line in lines: + if line.startswith("# "): + raw = line[2:].strip() + title = _ADR_TITLE_PREFIX_RE.sub("", raw).strip() + break + + status = "unknown" + in_status_section = False + for line in lines: + stripped = line.strip() + if stripped.lower() == "## status": + in_status_section = True + continue + if in_status_section: + if stripped.startswith("## "): + break + if stripped: + status = _normalize_adr_status(stripped) + break + + if status == "unknown": + fm = _STATUS_FRONTMATTER_RE.search(text) + if fm: + status = _normalize_adr_status(fm.group(1)) + + date = "" + in_date = False + for line in lines: + if line.strip().lower() == "## date": + in_date = True + continue + if in_date: + if line.startswith("## "): + break + if line.strip(): + date = line.strip() + break + + return { + "number": number, + "title": title, + "status": status, + "date": date, + "file": _relative_path(target, path), + "body": text, + "supersedes": [], + "superseded_by": [], + } + + +def collect_adrs(target: Path) -> list[dict[str, Any]]: + """Collect ADRs from ``docs/decisions`` using enhanced parsing.""" + decisions = target / "docs" / "decisions" + if not decisions.is_dir(): + return [] + adrs: list[dict[str, Any]] = [] + for path in sorted(decisions.glob("*.md")): + if path.name in {"0000-template.md", "README.md"}: + continue + if len(path.stem) < 4 or not path.stem[:4].isdigit(): + continue + adrs.append(parse_adr_file(target, path)) + adrs.sort(key=lambda a: a["number"]) + return adrs + + +def collect_adrs_from_dir(target: Path, decisions_dir: Path) -> list[dict[str, Any]]: + """Collect ADRs from an arbitrary decisions directory (tests).""" + adrs: list[dict[str, Any]] = [] + for path in sorted(decisions_dir.glob("*.md")): + if path.name in {"0000-template.md", "README.md"}: + continue + if len(path.stem) < 4 or not path.stem[:4].isdigit(): + continue + adrs.append(parse_adr_file(target, path)) + adrs.sort(key=lambda a: a["number"]) + return adrs + + +def _enrich_gate_docstring(target: Path, gate: dict[str, Any]) -> None: + rel = gate.get("file") + if not isinstance(rel, str) or not rel: + return + path = target / rel + if not path.is_file(): + return + try: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + except SyntaxError: + return + + name = str(gate.get("name", "")) + if gate.get("kind") == "class" and "." in name: + class_name, method_name = name.split(".", 1) + class_node: ast.ClassDef | None = None + method_node: ast.FunctionDef | ast.AsyncFunctionDef | None = None + for node in tree.body: + if isinstance(node, ast.ClassDef) and node.name == class_name: + class_node = node + for item in node.body: + if isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef)) and ( + item.name == method_name + ): + method_node = item + break + break + summary = "" + if class_node is not None: + summary = _first_paragraph_docstring(class_node) + if not summary and method_node is not None: + summary = _first_paragraph_docstring(method_node) + if summary: + gate["docstring_summary"] = summary + return + + for walk_node in ast.walk(tree): + if isinstance(walk_node, (ast.FunctionDef, ast.AsyncFunctionDef)) and ( + walk_node.name == name + ): + summary = _first_paragraph_docstring(walk_node) + if summary: + gate["docstring_summary"] = summary + return + + +def _enrich_gates_in_packages(data: dict[str, Any], target: Path) -> None: + packages = data.get("packages", {}) + if not isinstance(packages, dict): + return + for pkg in packages.values(): + if not isinstance(pkg, dict): + continue + for gate in pkg.get("gates", []): + if isinstance(gate, dict): + _enrich_gate_docstring(target, gate) + + +def _camel_case_tokens(name: str) -> set[str]: + parts = re.findall(r"[A-Z]?[a-z]+|[A-Z]+(?=[A-Z][a-z]|\b)", name) + return {p.lower() for p in parts if len(p) >= 5} + + +def _surface_tokens(package: str, surface_id: str, surface_name: str) -> set[str]: + """Distinctive tokens for ADR matching (not bare monorepo package prefixes).""" + tokens: set[str] = set() + pkg = package.lower().strip() + if pkg: + tokens.add(pkg) + tokens.add(pkg.replace("-", "_")) + shared_prefix = pkg.split("-", 1)[0] if "-" in pkg else "" + for raw in (surface_id, surface_name): + for part in re.split(r"[^a-zA-Z0-9]+", raw.lower()): + if len(part) < 5: + continue + if part in tokens: + continue + if shared_prefix and part == shared_prefix: + continue + tokens.add(part) + tokens.update(_camel_case_tokens(surface_name)) + return tokens + + +def _adr_matches_surface( + adr: dict[str, Any], + *, + package: str, + surface_id: str, + surface_name: str, +) -> bool: + title = str(adr.get("title", "")).lower() + body = str(adr.get("body", "")).lower() + pkg = package.lower().strip() + if pkg and (pkg in title or pkg.replace("-", "_") in title): + return True + leaf = pkg.split("-", 1)[-1] if "-" in pkg else "" + if len(leaf) >= 4 and re.search(rf"\b{re.escape(leaf)}\b", title): + return True + tokens = _surface_tokens(package, surface_id, surface_name) + for token in tokens: + if token in (pkg, pkg.replace("-", "_")): + continue + if re.search(rf"\b{re.escape(token)}\b", title): + return True + if len(token) >= 8 and re.search(rf"\b{re.escape(token)}\b", body): + return True + return False + + +def _is_global_adr(adr: dict[str, Any]) -> bool: + title = str(adr.get("title", "")).lower() + return any(marker in title for marker in _GLOBAL_ADR_TITLE_MARKERS) + + +def relevant_adrs_for_surface( + adrs: list[dict[str, Any]], + *, + package: str, + surface_id: str, + surface_name: str, +) -> list[dict[str, Any]]: + """Return ADRs relevant to one surface (specific matches first).""" + specific: list[dict[str, Any]] = [] + for adr in adrs: + if _adr_matches_surface( + adr, package=package, surface_id=surface_id, surface_name=surface_name + ): + specific.append({**adr, "general": False}) + if specific: + return specific + + if package not in ("pickled-core", "pickled-spec"): + return [] + + global_hits = [ + {**adr, "general": True} + for adr in adrs + if _is_global_adr(adr) + ] + return global_hits[:1] + + +def enrich_inventory_data(data: dict[str, Any], target: Path) -> None: + """Apply docstring, ADR, and per-surface ADR relevance fixes.""" + data["adrs"] = collect_adrs(target) + _enrich_gates_in_packages(data, target) + from pickled_core.mine.stories_stage import compute_surface_relevant_adrs + + data["surface_relevant_adrs"] = compute_surface_relevant_adrs(data) + def run_inventory( target: Path, @@ -28,6 +310,7 @@ def run_inventory( verbose=verbose, ) data = inv.to_dict(target) + enrich_inventory_data(data, target) write_json(paths.inventory_json, data) if verbose: inventory_lib.log("INFO", f"Wrote {paths.inventory_json}") @@ -38,4 +321,12 @@ def run_inventory( ) -__all__ = ["run_inventory"] +__all__ = [ + "collect_adrs", + "collect_adrs_from_dir", + "discover_umbrella_mcp_launch", + "enrich_inventory_data", + "parse_adr_file", + "relevant_adrs_for_surface", + "run_inventory", +] diff --git a/packages/pickled-core/src/pickled_core/mine/io.py b/packages/pickled-core/src/pickled_core/mine/io.py index 864c6a2..6d8f814 100644 --- a/packages/pickled-core/src/pickled_core/mine/io.py +++ b/packages/pickled-core/src/pickled_core/mine/io.py @@ -6,9 +6,31 @@ from pathlib import Path from typing import Any +from pickled_core.mine.errors import MissingStageInputError from pickled_core.mine.types import MiningPaths +def parse_surfaces_filter(raw: str | None) -> tuple[str, ...]: + """Parse comma-separated ``--surfaces`` tokens (lowercased).""" + if not raw or not raw.strip(): + return () + return tuple(token.strip().lower() for token in raw.split(",") if token.strip()) + + +def surface_matches( + *, + surface_id: str, + package: str, + tokens: tuple[str, ...], +) -> bool: + """Return True when no filter or any token matches package or surface-id.""" + if not tokens: + return True + pkg = package.lower() + sid = surface_id.lower() + return any(token in pkg or token in sid for token in tokens) + + def ensure_output_dir(output_dir: Path) -> MiningPaths: """Create the mining output tree if missing.""" paths = MiningPaths(output_dir.resolve()) @@ -27,6 +49,68 @@ def read_json(path: Path) -> Any: return json.loads(path.read_text(encoding="utf-8")) +def require_inventory_json( + output_dir: Path, + *, + needed_by: str = "stories", +) -> Any: + """Load ``inventory.json`` or raise :class:`MissingStageInputError`.""" + paths = ensure_output_dir(output_dir) + if not paths.inventory_json.is_file(): + raise MissingStageInputError( + paths.inventory_json, + needed_by=needed_by, + produced_by="inventory", + ) + return read_json(paths.inventory_json) + + +def require_stories_dir( + output_dir: Path, + *, + needed_by: str = "features", +) -> Path: + """Return stories directory or raise :class:`MissingStageInputError`.""" + paths = ensure_output_dir(output_dir) + if not paths.stories_dir.is_dir(): + raise MissingStageInputError( + paths.stories_dir, + needed_by=needed_by, + produced_by="stories", + ) + stories = sorted(paths.stories_dir.glob("*.story.md")) + if not stories: + raise MissingStageInputError( + paths.stories_dir, + needed_by=needed_by, + produced_by="stories", + ) + return paths.stories_dir + + +def require_features_dir( + output_dir: Path, + *, + needed_by: str = "tag", +) -> Path: + """Return features directory or raise :class:`MissingStageInputError`.""" + paths = ensure_output_dir(output_dir) + if not paths.features_dir.is_dir(): + raise MissingStageInputError( + paths.features_dir, + needed_by=needed_by, + produced_by="features", + ) + features = sorted(paths.features_dir.glob("*.feature")) + if not features: + raise MissingStageInputError( + paths.features_dir, + needed_by=needed_by, + produced_by="features", + ) + return paths.features_dir + + def write_json(path: Path, data: Any) -> None: path.parent.mkdir(parents=True, exist_ok=True) path.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n", encoding="utf-8") @@ -43,4 +127,15 @@ def write_text(path: Path, text: str) -> None: path.write_text(text, encoding="utf-8") -__all__ = ["ensure_output_dir", "read_json", "read_json_optional", "write_json", "write_text"] +__all__ = [ + "ensure_output_dir", + "parse_surfaces_filter", + "read_json", + "read_json_optional", + "require_features_dir", + "require_inventory_json", + "require_stories_dir", + "surface_matches", + "write_json", + "write_text", +] diff --git a/packages/pickled-core/src/pickled_core/mine/prompts/story_from_inventory.md b/packages/pickled-core/src/pickled_core/mine/prompts/story_from_inventory.md new file mode 100644 index 0000000..97586db --- /dev/null +++ b/packages/pickled-core/src/pickled_core/mine/prompts/story_from_inventory.md @@ -0,0 +1,29 @@ +You are drafting verification intent for a mined software surface. + +Surface: {{surface_name}} ({{surface_kind}}) +Package: {{package_name}} + +{{code_grounding_block}} + +Arguments / parameters: +{{arguments}} + +Related gates in the target: +{{related_gates}} + +Related ADRs (pre-filtered for this surface): +{{related_adrs}} + +Return exactly these delimited blocks (no extra markdown headings inside the blocks): + +---CONTEXT--- + + +---BEHAVIOR--- + + +---VERIFY--- + + +---DRIFT--- + diff --git a/packages/pickled-core/src/pickled_core/mine/report_stage.py b/packages/pickled-core/src/pickled_core/mine/report_stage.py index 9cdb4bc..d3b2972 100644 --- a/packages/pickled-core/src/pickled_core/mine/report_stage.py +++ b/packages/pickled-core/src/pickled_core/mine/report_stage.py @@ -22,7 +22,93 @@ def _inventory_cli_table(data: dict[str, Any]) -> list[str]: return lines -def render_mining_report(output_dir: Path, *, target_label: str | None = None) -> str: +def _count_scenarios(features: list[Path]) -> int: + total = 0 + for path in features: + total += path.read_text(encoding="utf-8").count("Scenario:") + return total + + +def _count_tagged_scenarios(tags: dict[str, Any] | None) -> int: + if not tags: + return 0 + count = 0 + for entry in tags.get("features", []): + if not isinstance(entry, dict): + continue + for scenario in entry.get("scenarios", []): + if isinstance(scenario, dict) and scenario.get("selected"): + count += 1 + return count + + +def _next_moves( + *, + inventory: dict[str, Any] | None, + stories: list[Path], + features: list[Path], + tags: dict[str, Any] | None, + coverage: dict[str, Any] | None, + ambiguity: dict[str, Any] | None, + surfaces_filter: tuple[str, ...], +) -> list[str]: + moves: list[str] = [] + if surfaces_filter: + joined = ", ".join(surfaces_filter) + moves.append( + f"This run covered only surfaces matching `--surfaces {joined}`; " + "run without `--surfaces` for full coverage." + ) + if not inventory: + moves.append("Run `pickled-spec mine inventory ` first.") + return moves + + if coverage: + for entry in coverage.get("rulesets", []): + if not isinstance(entry, dict): + continue + if entry.get("verdict") != "pass": + unref = entry.get("unreferenced_strict_rule_ids", []) + if unref: + ids = ", ".join(str(r) for r in unref[:10]) + moves.append(f"Add a story/feature covering strict rules: {ids}") + + if ambiguity: + for entry in ambiguity.get("features", []): + if not isinstance(entry, dict): + continue + if entry.get("verdict") == "fail": + feature = entry.get("feature", "") + surface_id = Path(feature).stem + moves.append( + f"Re-draft `{surface_id}` with tighter scope; see " + "evaluation/ambiguity.json findings." + ) + + if tags and features: + tagged = _count_tagged_scenarios(tags) + scenario_total = _count_scenarios(features) + if tagged < scenario_total: + moves.append( + "Manually review tag proposals for scenarios without a selected tag." + ) + + if not stories: + moves.append("Generate stories from inventory surfaces.") + if not features: + moves.append("Draft features from generated stories (requires LLM).") + elif not coverage and not ambiguity: + moves.append("Run `pickled-spec mine evaluate` for coverage and ambiguity gates.") + + return moves + + +def render_mining_report( + output_dir: Path, + *, + target_label: str | None = None, + surfaces_filter: tuple[str, ...] = (), +) -> str: """Build markdown report from whichever stage outputs exist.""" paths = MiningPaths(output_dir.resolve()) inventory = read_json_optional(paths.inventory_json) @@ -35,9 +121,10 @@ def render_mining_report(output_dir: Path, *, target_label: str | None = None) - if target_label: lines.append(f"Target: `{target_label}`") lines.append(f"Output: `{paths.root}`") + if surfaces_filter: + lines.append(f"Surface filter: `{', '.join(surfaces_filter)}`") lines.append("") - # Summary lines.append("## Summary") lines.append("") if inventory: @@ -56,12 +143,13 @@ def render_mining_report(output_dir: Path, *, target_label: str | None = None) - stories = sorted(paths.stories_dir.glob("*.story.md")) if paths.stories_dir.is_dir() else [] features = sorted(paths.features_dir.glob("*.feature")) if paths.features_dir.is_dir() else [] - lines.append(f"- Stories on disk: {len(stories)}") - lines.append(f"- Features on disk: {len(features)}") tags = read_json_optional(paths.tags_proposals) - lines.append(f"- Tag proposals: {'yes' if tags else 'no'}") coverage = read_json_optional(paths.coverage_json) ambiguity = read_json_optional(paths.ambiguity_json) + + lines.append(f"- Stories on disk: {len(stories)}") + lines.append(f"- Features on disk: {len(features)}") + lines.append(f"- Tag proposals: {'yes' if tags else 'no'}") lines.append(f"- Coverage evaluation: {'yes' if coverage else 'no'}") lines.append(f"- Ambiguity evaluation: {'yes' if ambiguity else 'no'}") lines.append("") @@ -78,17 +166,6 @@ def render_mining_report(output_dir: Path, *, target_label: str | None = None) - f"{len(pkg.get('mcp_tools', []))} | {len(pkg.get('gates', []))} |" ) lines.append("") - if inventory.get("adrs"): - lines.append("### ADRs") - lines.append("") - lines.append("| # | Title | Status |") - lines.append("|---|-------|--------|") - for adr in inventory["adrs"]: - num = adr.get("number", "") - title = adr.get("title", "") - status = adr.get("status", "") - lines.append(f"| {num} | {title} | {status} |") - lines.append("") cli_rows = _inventory_cli_table(inventory) if cli_rows: lines.append("### CLI commands") @@ -109,6 +186,20 @@ def render_mining_report(output_dir: Path, *, target_label: str | None = None) - lines.append(f"| {path.stem} | `{path.relative_to(paths.root)}` |") lines.append("") + if tags: + lines.append("## Tag proposals") + lines.append("") + feature_entries = tags.get("features", []) + if isinstance(feature_entries, list): + for entry in feature_entries: + if not isinstance(entry, dict): + continue + feature_path = entry.get("feature_path", "") + scenarios = entry.get("scenarios", []) + count = len(scenarios) if isinstance(scenarios, list) else 0 + lines.append(f"- `{feature_path}`: {count} scenario(s)") + lines.append("") + if features: lines.append("## Features generated") lines.append("") @@ -125,35 +216,54 @@ def render_mining_report(output_dir: Path, *, target_label: str | None = None) - if coverage: lines.append("## Coverage by rule set") lines.append("") + lines.append("| Rule set | Verdict | Unreferenced strict |") + lines.append("|----------|---------|--------------------:|") for entry in coverage.get("rulesets", []): - lines.append( - f"- **{entry.get('short_name', '')}**: {entry.get('verdict', 'unknown')} " - f"— {entry.get('notes', '')}" - ) + if not isinstance(entry, dict): + continue + short_name = entry.get("short_name", "") + verdict = entry.get("verdict", "unknown") + unref = entry.get("unreferenced_strict_rule_ids", []) + count = len(unref) if isinstance(unref, list) else 0 + lines.append(f"| {short_name} | {verdict} | {count} |") + if unref and verdict != "pass": + lines.append("") + lines.append(f"Unreferenced strict rules in `{short_name}`:") + for rule_id in unref: + lines.append(f"- `{rule_id}`") + lines.append("") lines.append("") if ambiguity: - lines.append("## Ambiguity") + lines.append("## Ambiguity by feature") lines.append("") + lines.append("| Feature | Verdict | Findings | Skipped |") + lines.append("|---------|---------|----------:|---------|") for entry in ambiguity.get("features", []): + if not isinstance(entry, dict): + continue lines.append( - f"- `{entry.get('feature', '')}`: {entry.get('verdict', 'unknown')} " - f"— {entry.get('notes', '')}" + f"| `{entry.get('feature', '')}` | {entry.get('verdict', '')} | " + f"{entry.get('finding_count', 0)} | " + f"{'yes' if entry.get('skipped') else 'no'} |" ) lines.append("") + moves = _next_moves( + inventory=inventory, + stories=stories, + features=features, + tags=tags, + coverage=coverage, + ambiguity=ambiguity, + surfaces_filter=surfaces_filter, + ) lines.append("## Suggested next moves") lines.append("") - if not inventory: - lines.append("- Run `pickled-spec mine inventory ` first.") - elif inventory.get("totals", {}).get("cli_commands", 0) == 0: - lines.append("- No CLI commands found; verify the target exposes Click entry points.") + if moves: + lines.extend(f"- {move}" for move in moves) else: - lines.append("- Run remaining stages: `stories`, `features`, `tag`, `evaluate`.") - if not stories: - lines.append("- Generate stories from inventory surfaces.") - if not features: - lines.append("- Draft features from generated stories.") + lines.append("- Mining pipeline complete for this output directory.") lines.append("") return "\n".join(lines) @@ -164,11 +274,16 @@ def run_report( *, target_label: str | None = None, verbose: bool = False, + surfaces_filter: tuple[str, ...] = (), ) -> Path: """Write ``mining-report.md`` and return its path.""" paths = MiningPaths(output_dir.resolve()) paths.root.mkdir(parents=True, exist_ok=True) - report = render_mining_report(paths.root, target_label=target_label) + report = render_mining_report( + paths.root, + target_label=target_label, + surfaces_filter=surfaces_filter, + ) write_text(paths.mining_report, report) if verbose: sys.stderr.write(f"[INFO] Wrote {paths.mining_report}\n") @@ -176,19 +291,51 @@ def run_report( def print_stdout_summary(output_dir: Path, *, target_label: str) -> None: - """Brief summary for CI (stderr only except this is called from report with format json).""" + """Brief stderr summary for CI.""" paths = MiningPaths(output_dir.resolve()) - inventory = read_json_optional(paths.inventory_json) - stories = len(list(paths.stories_dir.glob("*.story.md"))) if paths.stories_dir.is_dir() else 0 - features = len(list(paths.features_dir.glob("*.feature"))) if paths.features_dir.is_dir() else 0 - cli_count = 0 - if inventory: - cli_count = inventory.get("totals", {}).get("cli_commands", 0) + stories = list(paths.stories_dir.glob("*.story.md")) if paths.stories_dir.is_dir() else [] + features = list(paths.features_dir.glob("*.feature")) if paths.features_dir.is_dir() else [] + tags = read_json_optional(paths.tags_proposals) + coverage = read_json_optional(paths.coverage_json) + ambiguity = read_json_optional(paths.ambiguity_json) + + scenario_count = _count_scenarios(features) + tagged_count = _count_tagged_scenarios(tags) + + coverage_pass = coverage_fail = 0 + if coverage: + for entry in coverage.get("rulesets", []): + if isinstance(entry, dict) and entry.get("verdict") == "pass": + coverage_pass += 1 + elif isinstance(entry, dict): + coverage_fail += 1 + + ambiguity_pass = ambiguity_fail = 0 + if ambiguity: + for entry in ambiguity.get("features", []): + if not isinstance(entry, dict): + continue + if entry.get("verdict") == "pass": + ambiguity_pass += 1 + else: + ambiguity_fail += 1 + + sys.stderr.write(f"mining of {target_label} complete:\n") sys.stderr.write( - f"mining of {target_label} complete: cli_commands={cli_count} " - f"stories={stories} features={features}\n" - f"report: {paths.mining_report}\n" + f" surfaces={len(stories)} stories={len(stories)} " + f"features={len(features)} scenarios={scenario_count} tagged={tagged_count}\n" ) + if coverage: + total = coverage_pass + coverage_fail + sys.stderr.write( + f" coverage: {coverage_pass}/{total} rulesets pass, {coverage_fail} fail\n" + ) + if ambiguity: + total = ambiguity_pass + ambiguity_fail + sys.stderr.write( + f" ambiguity: {ambiguity_pass}/{total} features pass, {ambiguity_fail} fail\n" + ) + sys.stderr.write(f"report: {paths.mining_report}\n") __all__ = ["render_mining_report", "run_report", "print_stdout_summary"] diff --git a/packages/pickled-core/src/pickled_core/mine/stories_stage.py b/packages/pickled-core/src/pickled_core/mine/stories_stage.py new file mode 100644 index 0000000..d37b960 --- /dev/null +++ b/packages/pickled-core/src/pickled_core/mine/stories_stage.py @@ -0,0 +1,751 @@ +"""Stage 2: emit user stories from inventory surfaces.""" + +from __future__ import annotations + +import asyncio +import re +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from pickled_core.llm.base import LLMClient +from pickled_core.llm.prompts import PromptTemplate +from pickled_core.mine.inventory_stage import relevant_adrs_for_surface +from pickled_core.mine.io import ( + ensure_output_dir, + require_inventory_json, + surface_matches, +) +from pickled_core.mine.types import InventoryResult, RelevantAdrRef, StoryResult + +_PROMPT_PATH = Path(__file__).resolve().parent / "prompts" / "story_from_inventory.md" +_LLM_UNAVAILABLE = "(LLM unavailable — fill manually)" +_NO_DOCSTRING_BEHAVIOR = ( + "The inventory provides no docstring for this surface; behavior must be " + "confirmed from source before specifying." +) +_SERVE_PLUMBING = frozenset({"serve", "mcp serve"}) +_DELIM_CONTEXT = "---CONTEXT---" +_DELIM_BEHAVIOR = "---BEHAVIOR---" +_DELIM_VERIFY = "---VERIFY---" +_DELIM_DRIFT = "---DRIFT---" +_CODE_GROUNDING_WITH_CONTEXT = """\ +You are writing a behavioral specification for a code surface. You have: + +1. The surface's docstring (what the author CLAIMS it does): +{{docstring}} + +2. The actual source code the surface executes (root plus resolved + intra-project callees), and a list of calls that could not be statically + resolved: +{{code_context}} + +Your job is to describe OBSERVABLE BEHAVIOR — the contract a caller relies +on. Rules: + +- Ground every statement in the code. State what the surface accepts, + returns, rejects, and what side effects a caller observes. +- Describe behavior, NOT implementation. Do NOT mention line numbers, + private method names, or "it calls X then Y". A reader must understand + the behavior without seeing the code. +- Where the code reveals behavior the docstring omits (a specific return + shape, an error mode, the ABSENCE of validation), state it as a + behavioral fact. +- DOCSTRING DRIFT: if the docstring CONTRADICTS the code (claims a behavior + the code does not implement, or omits a behavior the code clearly has), + note the specific discrepancy in the DRIFT block. The code is the source + of truth; the docstring may be stale. +- Unresolved calls: behavior hidden behind unresolved calls (e.g. a + protocol-dispatched LLM call) is uncertain. Do NOT invent what those do; + if a behavior depends on an unresolved call, say it is delegated to an + unresolved collaborator.""" +_CODE_GROUNDING_DOCSTRING_ONLY = """\ +Docstring / help from inventory (may be empty): +{{docstring}} + +Ground every statement in the provided docstring, arguments, and related +gates. If the docstring is empty or says nothing about behavior, write for +the BEHAVIOR block exactly: + +The inventory provides no docstring for this surface; behavior must be +confirmed from source before specifying. + +Do NOT infer behavior from the surface's NAME alone. A gate called +AmbiguityGate might do many things; do not assume it detects step-definition +collisions unless the docstring says so. + +The DRIFT block must be empty (no code-context to compare against).""" + + +@dataclass(frozen=True, slots=True) +class CodeContextForStory: + """Parsed ``code-context/.md`` for story prompts.""" + + depth: str + unit_count: int + total_lines: int + unresolved_count: int + prompt_text: str + + +@dataclass(frozen=True, slots=True) +class _Surface: + surface_id: str + kind: str + package: str + name: str + docstring: str + arguments: str + related_gates: str + relevant_adrs: tuple[RelevantAdrRef, ...] + + +def _slug_id(raw: str) -> str: + return re.sub(r"[^a-zA-Z0-9]+", "_", raw).strip("_").lower() + + +def _surface_id_mcp(tool_name: str) -> str: + return _slug_id(tool_name) + + +def _surface_id_cli(package: str, command: str) -> str: + pkg = _slug_id(package.replace("-", "_")) + cmd = _slug_id(command.replace(" ", "_")) + return f"{pkg}_{cmd}" + + +def _surface_id_gate(package: str, gate_name: str) -> str: + pkg = _slug_id(package.replace("-", "_")) + base = gate_name.split(".", 1)[0].lower() + return f"{pkg}_{_slug_id(base)}" + + +def _format_relevant_adrs(refs: tuple[RelevantAdrRef, ...]) -> str: + if not refs: + return "- (none directly relevant)" + lines: list[str] = [] + for ref in refs: + suffix = " (general)" if ref.general else "" + lines.append(f"- ADR {ref.number}: {ref.title}{suffix} — {ref.status}") + return "\n".join(lines) + + +def _load_relevant_adrs( + data: dict[str, Any], + *, + surface_id: str, + package: str, + surface_name: str, +) -> tuple[RelevantAdrRef, ...]: + mapped = data.get("surface_relevant_adrs", {}) + if isinstance(mapped, dict) and surface_id in mapped: + raw = mapped[surface_id] + if isinstance(raw, list): + return tuple( + RelevantAdrRef( + number=str(item.get("number", "")), + title=str(item.get("title", "")), + status=str(item.get("status", "")), + general=bool(item.get("general")), + ) + for item in raw + if isinstance(item, dict) + ) + adrs = data.get("adrs", []) + if not isinstance(adrs, list): + return () + picked = relevant_adrs_for_surface( + [a for a in adrs if isinstance(a, dict)], + package=package, + surface_id=surface_id, + surface_name=surface_name, + ) + return tuple( + RelevantAdrRef( + number=str(a.get("number", "")), + title=str(a.get("title", "")), + status=str(a.get("status", "")), + general=bool(a.get("general")), + ) + for a in picked + ) + + +def compute_surface_relevant_adrs(data: dict[str, Any]) -> dict[str, list[dict[str, Any]]]: + """Build per-surface relevant ADR lists for inventory JSON.""" + result: dict[str, list[dict[str, Any]]] = {} + for surface in collect_surfaces(data): + result[surface.surface_id] = [ + { + "number": ref.number, + "title": ref.title, + "status": ref.status, + "general": ref.general, + } + for ref in surface.relevant_adrs + ] + return result + + +def _is_plumbing_cli(full_name: str) -> bool: + normalized = full_name.strip().lower() + return normalized in _SERVE_PLUMBING or normalized.endswith(" serve") + + +def _is_significant_cli(cmd: dict[str, Any]) -> bool: + if _is_plumbing_cli(str(cmd.get("full_name", ""))): + return False + if cmd.get("is_group"): + return True + help_text = str(cmd.get("help", "")) + if len(help_text) > 60: + return True + return any( + isinstance(param, dict) and param.get("required") + for param in cmd.get("params", []) + ) + + +def _format_params(params: list[dict[str, Any]]) -> str: + lines: list[str] = [] + for param in params: + if not isinstance(param, dict): + continue + name = param.get("name", "") + required = "required" if param.get("required") else "optional" + help_text = param.get("help", "") + lines.append(f"- `{name}` ({required}): {help_text}") + return "\n".join(lines) if lines else "- (none)" + + +def _gate_docstring(gate: dict[str, Any]) -> str: + return str( + gate.get("docstring_summary") + or gate.get("doc") + or gate.get("help") + or "" + ).strip() + + +def collect_surfaces(data: dict[str, Any]) -> list[_Surface]: + surfaces: list[_Surface] = [] + packages = data.get("packages", {}) + if not isinstance(packages, dict): + return surfaces + + for pkg_name, pkg in packages.items(): + if not isinstance(pkg, dict): + continue + gates = pkg.get("gates", []) + gate_names = [ + str(g.get("name", g.get("class_name", ""))) + for g in gates + if isinstance(g, dict) + ] + gate_block = ", ".join(gate_names) if gate_names else "(none)" + + for tool in pkg.get("mcp_tools", []): + if not isinstance(tool, dict): + continue + name = str(tool.get("name", "")) + if not name: + continue + schema = tool.get("input_schema", {}) + if not isinstance(schema, dict): + schema = {} + surface_id = _surface_id_mcp(name) + surfaces.append( + _Surface( + surface_id=surface_id, + kind="mcp_tool", + package=str(pkg_name), + name=name, + docstring=str(tool.get("description", "")).strip(), + arguments=_format_params( + [ + { + "name": k, + "required": k in (schema.get("required") or []), + "help": str(v.get("description", "")), + } + for k, v in (schema.get("properties") or {}).items() + if isinstance(v, dict) + ] + ), + related_gates=gate_block, + relevant_adrs=_load_relevant_adrs( + data, + surface_id=surface_id, + package=str(pkg_name), + surface_name=name, + ), + ) + ) + + for cmd in pkg.get("cli_commands", []): + if not isinstance(cmd, dict) or not _is_significant_cli(cmd): + continue + full = str(cmd.get("full_name", "")) + surface_id = _surface_id_cli(str(pkg_name), full) + surfaces.append( + _Surface( + surface_id=surface_id, + kind="cli_command", + package=str(pkg_name), + name=full, + docstring=str(cmd.get("help", "")).strip(), + arguments=_format_params(cmd.get("params", [])), + related_gates=gate_block, + relevant_adrs=_load_relevant_adrs( + data, + surface_id=surface_id, + package=str(pkg_name), + surface_name=full, + ), + ) + ) + + for gate in gates: + if not isinstance(gate, dict): + continue + gate_name = str(gate.get("name", gate.get("class_name", ""))) + if not gate_name: + continue + surface_id = _surface_id_gate(str(pkg_name), gate_name) + surfaces.append( + _Surface( + surface_id=surface_id, + kind="gate", + package=str(pkg_name), + name=gate_name, + docstring=_gate_docstring(gate), + arguments="- (gate class)", + related_gates=gate_name, + relevant_adrs=_load_relevant_adrs( + data, + surface_id=surface_id, + package=str(pkg_name), + surface_name=gate_name, + ), + ) + ) + return surfaces + + +def parse_code_context_markdown(text: str) -> CodeContextForStory | None: + """Extract code bodies and metadata from a code-context markdown file.""" + if "No code definition resolved" in text: + return None + depth_match = re.search(r"\*\*Depth:\*\*\s*(\w+)", text) + units_match = re.search( + r"\*\*Units collected:\*\*\s*(\d+)\s*\|\s*\*\*Total lines:\*\*\s*(\d+)", + text, + ) + depth = depth_match.group(1) if depth_match else "body" + unit_count = int(units_match.group(1)) if units_match else 0 + total_lines = int(units_match.group(2)) if units_match else 0 + + parts: list[str] = [] + for match in re.finditer( + r"## (?:Root: [^\n]+|Callee: [^\n]+)\n\n```python\n(.*?)```", + text, + flags=re.DOTALL, + ): + parts.append(match.group(1).strip()) + + unresolved_lines: list[str] = [] + if "## Unresolved callees" in text: + section = text.split("## Unresolved callees", 1)[1] + section = section.split("## ", 1)[0] + for line in section.splitlines(): + line = line.strip() + if line.startswith("- `"): + unresolved_lines.append(line) + + unresolved_count = len(unresolved_lines) + prompt_parts = ["### Source (root and resolved callees)", ""] + for idx, source in enumerate(parts, start=1): + label = "Root" if idx == 1 else f"Unit {idx}" + prompt_parts.extend([f"#### {label}", "", "```python", source, "```", ""]) + if unresolved_lines: + prompt_parts.extend( + [ + "### Unresolved calls (do not invent behavior for these)", + "", + *unresolved_lines, + "", + ] + ) + return CodeContextForStory( + depth=depth, + unit_count=unit_count, + total_lines=total_lines, + unresolved_count=unresolved_count, + prompt_text="\n".join(prompt_parts).strip(), + ) + + +def load_code_context_for_surface( + code_context_dir: Path, + surface_id: str, +) -> CodeContextForStory | None: + """Load ``code-context/.md`` when the code stage wrote it.""" + path = code_context_dir / f"{surface_id}.md" + if not path.is_file(): + return None + return parse_code_context_markdown(path.read_text(encoding="utf-8")) + + +def _extract_delimited_block(text: str, start: str, end: str | None) -> str: + if start not in text: + return "" + rest = text.split(start, 1)[1] + if end and end in rest: + rest = rest.split(end, 1)[0] + return rest.strip() + + +def _parse_llm_blocks(raw: str) -> tuple[str, str, str, str]: + context = _extract_delimited_block(raw, _DELIM_CONTEXT, _DELIM_BEHAVIOR) + behavior = _extract_delimited_block(raw, _DELIM_BEHAVIOR, _DELIM_VERIFY) + verify = _extract_delimited_block(raw, _DELIM_VERIFY, _DELIM_DRIFT) + drift = _extract_delimited_block(raw, _DELIM_DRIFT, None) + if not context and not behavior and not verify and not drift and raw.strip(): + context = raw.strip() + return context, behavior, verify, drift + + +def _format_drift_open_questions(drift: str) -> str: + lines: list[str] = [] + for raw in drift.splitlines(): + line = raw.strip() + if not line: + continue + if line.startswith("- "): + line = line[2:].strip() + elif line.startswith("-"): + line = line[1:].strip() + if not line.lower().startswith("docstring drift:"): + line = f"Docstring drift: {line}" + else: + line = line[0].upper() + line[1:] + lines.append(f"- {line}") + return "\n".join(lines) + + +def _render_story_body( + surface: _Surface, + *, + context: str, + behavior: str, + verify: str, + open_questions: str = "", + code_meta: CodeContextForStory | None = None, +) -> str: + verify_block = verify.strip() + if verify_block and not verify_block.startswith("-"): + verify_lines = [f"- {line.strip()}" for line in verify_block.splitlines() if line.strip()] + verify_block = "\n".join(verify_lines) if verify_lines else "- (none)" + elif not verify_block: + verify_block = "- (none)" + + sections = [ + f"# Story: {surface.name}", + "", + "## Metadata", + "", + f"- **Surface kind:** {surface.kind}", + f"- **Package:** {surface.package}", + f"- **Surface id:** {surface.surface_id}", + ] + if code_meta is not None: + sections.append( + f"- **Code depth:** {code_meta.depth} | **Units read:** " + f"{code_meta.unit_count} | **Unresolved:** {code_meta.unresolved_count}" + ) + sections.extend( + [ + "", + "## Context", + "", + context or _LLM_UNAVAILABLE, + "", + "## What the target does today", + "", + behavior or _NO_DOCSTRING_BEHAVIOR, + "", + "## What we want to verify", + "", + verify_block, + "", + "## Inventory references", + "", + f"- Arguments:\n{surface.arguments}", + f"- Related gates: {surface.related_gates}", + f"- Related ADRs:\n{_format_relevant_adrs(surface.relevant_adrs)}", + "", + "## Open questions", + "", + open_questions, + "", + "## Status", + "", + "draft", + "", + ] + ) + return "\n".join(sections) + + +def _code_grounding_block( + surface: _Surface, + code_ctx: CodeContextForStory | None, +) -> str: + doc = surface.docstring or "(none)" + if code_ctx is None: + return _CODE_GROUNDING_DOCSTRING_ONLY.replace("{{docstring}}", doc) + block = _CODE_GROUNDING_WITH_CONTEXT.replace("{{docstring}}", doc) + return block.replace("{{code_context}}", code_ctx.prompt_text) + + +def _llm_blocks_for_surface( + surface: _Surface, + llm: LLMClient, + *, + code_ctx: CodeContextForStory | None, +) -> tuple[str, str, str, str]: + template = PromptTemplate.from_file(_PROMPT_PATH) + prompt = template.render( + surface_name=surface.name, + surface_kind=surface.kind, + package_name=surface.package, + code_grounding_block=_code_grounding_block(surface, code_ctx), + arguments=surface.arguments, + related_gates=surface.related_gates, + related_adrs=_format_relevant_adrs(surface.relevant_adrs), + ) + from pickled_core.llm.turns import complete_prompt + + raw = complete_prompt( + llm, + prompt, + system="You output only the four delimiter blocks requested.", + ).strip() + context, behavior, verify, drift = _parse_llm_blocks(raw) + missing = [] + if not context: + missing.append("CONTEXT") + if not behavior: + missing.append("BEHAVIOR") + if not verify: + missing.append("VERIFY") + if missing: + sys.stderr.write( + f"[WARN] LLM story for {surface.surface_id} missing delimiters: " + f"{', '.join(missing)}\n" + ) + if not behavior and surface.docstring and code_ctx is None: + behavior = surface.docstring + elif not behavior: + behavior = _NO_DOCSTRING_BEHAVIOR + if not context: + context = _LLM_UNAVAILABLE + if not verify: + verify = _LLM_UNAVAILABLE + if code_ctx is None: + drift = "" + return context, behavior, verify, drift + + +def _deterministic_blocks( + surface: _Surface, + *, + code_ctx: CodeContextForStory | None, +) -> tuple[str, str, str, str]: + context = _LLM_UNAVAILABLE + if code_ctx is not None: + behavior = ( + f"(LLM unavailable; code-context captured {code_ctx.unit_count} units, " + f"{code_ctx.total_lines} lines — see code-context/{surface.surface_id}.md)" + ) + else: + behavior = surface.docstring or _NO_DOCSTRING_BEHAVIOR + verify = _LLM_UNAVAILABLE + return context, behavior, verify, "" + + +def _write_one_story( + paths_stories_dir: Path, + surface: _Surface, + *, + llm: LLMClient | None, + overwrite: bool, + interactive: bool, + code_context_dir: Path | None, +) -> StoryResult: + story_path = paths_stories_dir / f"{surface.surface_id}.story.md" + if ( + story_path.is_file() + and story_path.read_text(encoding="utf-8").strip() + and not overwrite + ): + return StoryResult( + surface_id=surface.surface_id, + story_path=story_path, + skipped=True, + ) + + code_ctx: CodeContextForStory | None = None + if code_context_dir is not None: + code_ctx = load_code_context_for_surface(code_context_dir, surface.surface_id) + + open_questions = "" + drift = "" + if llm is None: + context, behavior, verify, drift = _deterministic_blocks( + surface, code_ctx=code_ctx + ) + else: + try: + context, behavior, verify, drift = _llm_blocks_for_surface( + surface, llm, code_ctx=code_ctx + ) + except Exception as exc: + sys.stderr.write( + f"[WARN] LLM story draft failed for {surface.surface_id}: " + f"{type(exc).__name__}: {exc}\n" + ) + context, behavior, verify, drift = _deterministic_blocks( + surface, code_ctx=code_ctx + ) + + drift_questions = _format_drift_open_questions(drift) + if interactive and llm is not None: + open_questions = input( + f"Open questions for {surface.surface_id} (or leave blank): " + ).strip() + + combined_open = "\n".join( + part for part in (drift_questions, open_questions) if part.strip() + ) + + body = _render_story_body( + surface, + context=context, + behavior=behavior, + verify=verify, + open_questions=combined_open or "(none)", + code_meta=code_ctx, + ) + story_path.write_text(body, encoding="utf-8") + return StoryResult( + surface_id=surface.surface_id, + story_path=story_path, + skipped=False, + ) + + +async def _write_one_story_async( + paths_stories_dir: Path, + surface: _Surface, + llm: LLMClient, + *, + overwrite: bool, + sem: asyncio.Semaphore, + code_context_dir: Path | None, +) -> StoryResult: + async with sem: + return await asyncio.to_thread( + _write_one_story, + paths_stories_dir, + surface, + llm=llm, + overwrite=overwrite, + interactive=False, + code_context_dir=code_context_dir, + ) + + +def run_stories( + inventory: InventoryResult, + output_dir: Path, + *, + llm: LLMClient | None, + quick: bool, + overwrite: bool, + surfaces: tuple[str, ...] = (), + max_parallel: int = 4, + code_context_dir: Path | None = None, +) -> list[StoryResult]: + """Write one ``.story.md`` per significant inventory surface.""" + paths = ensure_output_dir(output_dir) + ctx_dir = code_context_dir + if ctx_dir is None and paths.code_context_dir.is_dir(): + ctx_dir = paths.code_context_dir + all_surfaces = collect_surfaces(inventory.data) + selected = [ + s + for s in all_surfaces + if surface_matches( + surface_id=s.surface_id, + package=s.package, + tokens=surfaces, + ) + ] + selected.sort(key=lambda s: s.surface_id) + + if not quick or llm is None: + return [ + _write_one_story( + paths.stories_dir, + surface, + llm=llm, + overwrite=overwrite, + interactive=not quick and llm is not None, + code_context_dir=ctx_dir, + ) + for surface in selected + ] + + sem = asyncio.Semaphore(max_parallel) + + async def _run() -> list[StoryResult]: + tasks = [ + _write_one_story_async( + paths.stories_dir, + surface, + llm, + overwrite=overwrite, + sem=sem, + code_context_dir=ctx_dir, + ) + for surface in selected + ] + gathered = await asyncio.gather(*tasks) + return sorted(gathered, key=lambda r: r.surface_id) + + return asyncio.run(_run()) + + +def load_inventory_from_output(output_dir: Path) -> InventoryResult: + """Load ``inventory.json`` produced by a prior inventory stage.""" + paths = ensure_output_dir(output_dir) + data = require_inventory_json(output_dir, needed_by="stories") + warnings = data.get("warnings", []) if isinstance(data.get("warnings"), list) else [] + return InventoryResult( + inventory_path=paths.inventory_json, + data=data, + warnings=[str(w) for w in warnings], + ) + + +__all__ = [ + "CodeContextForStory", + "collect_surfaces", + "compute_surface_relevant_adrs", + "load_code_context_for_surface", + "load_inventory_from_output", + "parse_code_context_markdown", + "run_stories", +] diff --git a/packages/pickled-core/src/pickled_core/mine/tag_stage.py b/packages/pickled-core/src/pickled_core/mine/tag_stage.py new file mode 100644 index 0000000..d543e9f --- /dev/null +++ b/packages/pickled-core/src/pickled_core/mine/tag_stage.py @@ -0,0 +1,460 @@ +"""Stage 4: propose scenario tags from rule sets.""" + +from __future__ import annotations + +import re +import sys +from pathlib import Path +from typing import Any + +import yaml +from pickled_rules.gates_runner import _resolve_ruleset_entries +from pickled_rules.loader import RuleSetValidationError, load_ruleset +from pickled_rules.types import Rule + +from pickled_core.mine.io import ( + ensure_output_dir, + require_features_dir, + surface_matches, + write_json, +) +from pickled_core.mine.types import ( + LoadedRulesetEntry, + RulesetSources, + TagProposal, + TagResult, +) + +_SCENARIO_LINE = re.compile( + r"^(\s*)(Scenario(?: Outline| Template)?):\s*.+$", +) +_TAG_LINE = re.compile(r"^\s*(@\S+)\s*$") +# Repair legacy corruption from character-offset injection (pre friction #16). +_BROKEN_SCENARIO_TAG = re.compile( + r"^(\s*)Sc\w*(@\S+)\n(?:enario|rio|io)?(:.*)$", + re.MULTILINE | re.IGNORECASE, +) +_TOKEN_RE = re.compile(r"[a-z0-9]+") +_PREFERRED_SHORT = frozenset({"pickled-internal", "best-practices"}) +_STOP = frozenset( + { + "a", + "an", + "the", + "and", + "or", + "to", + "of", + "in", + "on", + "for", + "is", + "are", + "be", + "when", + "then", + "given", + "with", + "that", + "this", + "as", + "at", + "by", + "from", + "it", + "not", + } +) + + +def _tokenize(text: str) -> set[str]: + return {t for t in _TOKEN_RE.findall(text.lower()) if len(t) > 2 and t not in _STOP} + + +def _rule_weight(short_name: str, package_hint: str) -> int: + lower = short_name.lower() + if package_hint and package_hint in lower: + return 3 + if lower in _PREFERRED_SHORT: + return 2 + return 1 + + +def _score_rule( + rule: Rule, + *, + short_name: str, + package_hint: str, + step_tokens: set[str], +) -> int: + title_tokens = _tokenize(rule.title) + overlap = len(step_tokens & title_tokens) + if overlap == 0: + return 0 + return overlap * _rule_weight(short_name, package_hint) + + +def _propose_for_text( + text: str, + loaded: list[LoadedRulesetEntry], + *, + package_hint: str, + known_rule_ids: set[str], +) -> list[TagProposal]: + step_tokens = _tokenize(text) + scored: list[tuple[int, TagProposal]] = [] + for entry in loaded: + for rule in entry.ruleset.rules: + score = _score_rule( + rule, + short_name=entry.short_name, + package_hint=package_hint, + step_tokens=step_tokens, + ) + if score <= 0: + continue + tag = f"@{entry.short_name}:{rule.id}" + if rule.id not in known_rule_ids: + continue + scored.append( + ( + score, + TagProposal( + tag=tag, + rule_id=rule.id, + short_name=entry.short_name, + rule_title=rule.title, + score=score, + ), + ) + ) + scored.sort(key=lambda item: (-item[0], item[1].tag)) + seen: set[str] = set() + proposals: list[TagProposal] = [] + for _, proposal in scored: + if proposal.tag in seen: + continue + seen.add(proposal.tag) + proposals.append(proposal) + if len(proposals) >= 7: + break + return proposals + + +def _package_hint_from_feature(path: Path) -> str: + stem = path.stem.replace("-", "_").lower() + return stem.split("_", 1)[0] if "_" in stem else stem + + +def _known_rule_ids(loaded: list[LoadedRulesetEntry]) -> set[str]: + ids: set[str] = set() + for entry in loaded: + ids.update(rule.id for rule in entry.ruleset.rules) + return ids + + +def resolve_ruleset_sources( + target: Path, + *, + ruleset_config: Path | None, + ruleset_dir: Path | None, +) -> RulesetSources | None: + """Resolve rule sets from CLI flags or target-root config.""" + if ruleset_config is not None: + cfg_path = ruleset_config.resolve() + if not cfg_path.is_file(): + msg = f"ruleset config not found: {cfg_path}" + raise FileNotFoundError(msg) + cfg = yaml.safe_load(cfg_path.read_text(encoding="utf-8")) + if not isinstance(cfg, dict): + msg = f"invalid ruleset config: {cfg_path}" + raise RuleSetValidationError(msg) + root = cfg_path.parent + entries = _resolve_ruleset_entries(root, cfg) + return _load_entries(entries, config_root=root) + + if ruleset_dir is not None: + root = ruleset_dir.resolve() + if not root.is_dir(): + msg = f"ruleset dir not found: {root}" + raise FileNotFoundError(msg) + loaded = [ + LoadedRulesetEntry( + short_name=path.stem, + path=path, + ruleset=load_ruleset(path), + ) + for path in sorted(root.glob("*.yaml")) + ] + return RulesetSources(config_root=root, rulesets=loaded) + + cfg_path = target.resolve() / "pickled.ruleset.yaml" + if not cfg_path.is_file(): + return None + cfg = yaml.safe_load(cfg_path.read_text(encoding="utf-8")) + if not isinstance(cfg, dict): + return None + root = cfg_path.parent + entries = _resolve_ruleset_entries(root, cfg) + if not entries: + return None + return _load_entries(entries, config_root=root) + + +def _load_entries( + entries: list[Any], # pickled_rules.gates_runner._RulesetEntry + *, + config_root: Path, +) -> RulesetSources: + loaded: list[LoadedRulesetEntry] = [] + for entry in entries: + loaded.append( + LoadedRulesetEntry( + short_name=entry.short_name, + path=entry.path, + ruleset=load_ruleset(entry.path), + ) + ) + return RulesetSources(config_root=config_root, rulesets=loaded) + + +def _repair_split_scenario_tags(feature_text: str) -> str: + """Fix tag lines spliced into the middle of ``Scenario`` from legacy offset injection.""" + + def repl(match: re.Match[str]) -> str: + indent, tag, title_rest = match.groups() + return f"{indent}{tag}\n{indent}Scenario{title_rest}" + + repaired = feature_text + for _ in range(8): + next_text = _BROKEN_SCENARIO_TAG.sub(repl, repaired) + if next_text == repaired: + break + repaired = next_text + + lines = repaired.splitlines() + fixed_lines: list[str] = [] + index = 0 + while index < len(lines): + line = lines[index] + split_head = re.match(r"^(\s*)Sc\w*(@\S+)\s*$", line) + if split_head is not None and index + 1 < len(lines): + tail = lines[index + 1] + tail_match = re.match( + r"^(enario|ario|rio|io)(:.*)$", + tail.strip(), + re.IGNORECASE, + ) + if tail_match is not None: + indent, tag = split_head.groups() + fixed_lines.append(f"{indent}{tag}") + fixed_lines.append(f"{indent}Scenario{tail_match.group(2)}") + index += 2 + continue + fixed_lines.append(line) + index += 1 + return "\n".join(fixed_lines) + ("\n" if feature_text.endswith("\n") else "") + + +def _tags_immediately_above(lines: list[str], scenario_line_index: int) -> set[str]: + tags: set[str] = set() + index = scenario_line_index - 1 + while index >= 0: + line = lines[index] + if not line.strip(): + break + match = _TAG_LINE.match(line) + if match is None: + break + tags.add(match.group(1)) + index -= 1 + return tags + + +def _scenario_line_spans(lines: list[str]) -> list[tuple[int, str, str]]: + """Return ``(line_index, indent, full_scenario_line)`` for each scenario header.""" + spans: list[tuple[int, str, str]] = [] + for index, line in enumerate(lines): + match = _SCENARIO_LINE.match(line) + if match is not None: + spans.append((index, match.group(1), line)) + return spans + + +def _scenario_blocks_from_lines( + lines: list[str], +) -> list[tuple[int, str, str]]: + """Return ``(scenario_line_index, block_text, scenario_title)`` per scenario.""" + spans = _scenario_line_spans(lines) + blocks: list[tuple[int, str, str]] = [] + for idx, (line_index, _indent, title_line) in enumerate(spans): + end_line = spans[idx + 1][0] if idx + 1 < len(spans) else len(lines) + block_text = "\n".join(lines[line_index:end_line]) + blocks.append((line_index, block_text, title_line.strip())) + return blocks + + +def apply_line_based_tags( + feature_text: str, + scenario_tags: list[tuple[int, list[str]]], +) -> str: + """Insert tag lines immediately above each scenario line (line-based, never in-line).""" + trailing_newline = feature_text.endswith("\n") + lines = feature_text.splitlines() + + for line_index, tags in sorted(scenario_tags, key=lambda item: item[0], reverse=True): + if line_index < 0 or line_index >= len(lines): + continue + if _SCENARIO_LINE.match(lines[line_index]) is None: + continue + indent_match = re.match(r"^(\s*)", lines[line_index]) + indent = indent_match.group(1) if indent_match else "" + existing = _tags_immediately_above(lines, line_index) + new_tag_lines: list[str] = [] + seen_new: set[str] = set() + for tag in tags: + if tag in existing or tag in seen_new: + continue + seen_new.add(tag) + new_tag_lines.append(f"{indent}{tag}") + if not new_tag_lines: + continue + lines[line_index:line_index] = new_tag_lines + + result = "\n".join(lines) + if trailing_newline and not result.endswith("\n"): + result += "\n" + return result + + +def _filter_feature_paths( + feature_paths: list[Path], + surfaces: tuple[str, ...], +) -> list[Path]: + if not surfaces: + return feature_paths + filtered: list[Path] = [] + for path in feature_paths: + surface_id = path.stem + package = _package_hint_from_feature(path) + if surface_matches(surface_id=surface_id, package=package, tokens=surfaces): + filtered.append(path) + return filtered + + +def run_tag( + output_dir: Path, + *, + ruleset_sources: RulesetSources | None, + quick: bool, + surfaces: tuple[str, ...] = (), +) -> TagResult: + """Propose tags for scenarios in generated features.""" + paths = ensure_output_dir(output_dir) + if ruleset_sources is None: + sys.stderr.write("[WARN] no rule sets configured; tagging skipped\n") + return TagResult( + proposals_path=paths.tags_proposals, + proposals=[], + skipped=True, + warnings=["no rule sets configured; tagging skipped"], + ) + + features_dir = require_features_dir(output_dir, needed_by="tag") + feature_paths = _filter_feature_paths( + sorted(features_dir.glob("*.feature")), + surfaces, + ) + if not feature_paths: + msg = "no features match --surfaces filter" + raise ValueError(msg) + + loaded = ruleset_sources.rulesets + known_ids = _known_rule_ids(loaded) + document: dict[str, Any] = {"schema_version": "1", "features": []} + all_proposals: list[dict[str, Any]] = [] + + for feature_path in feature_paths: + raw_text = feature_path.read_text(encoding="utf-8") + text = _repair_split_scenario_tags(raw_text) + package_hint = _package_hint_from_feature(feature_path) + lines = text.splitlines() + feature_entry: dict[str, Any] = { + "feature_path": str(feature_path.relative_to(paths.root)), + "scenarios": [], + } + injections: list[tuple[int, list[str]]] = [] + + for line_index, block, title in _scenario_blocks_from_lines(lines): + proposals = _propose_for_text( + block, + loaded, + package_hint=package_hint, + known_rule_ids=known_ids, + ) + proposal_dicts = [ + { + "tag": p.tag, + "rule_id": p.rule_id, + "short_name": p.short_name, + "rule_title": p.rule_title, + "score": p.score, + } + for p in proposals + ] + selected: str | None = None + tags_for_scenario: list[str] = [] + + if proposals and quick: + selected = proposals[0].tag + tags_for_scenario = [selected] + elif proposals and not quick: + sys.stderr.write(f"\n{feature_path.name} — {title}\n") + for idx, proposal in enumerate(proposals, start=1): + sys.stderr.write( + f" {idx}. {proposal.tag} ({proposal.score}) — {proposal.rule_title}\n" + ) + choice = input("tag number, custom @short:id, or skip: ").strip() + if choice.isdigit(): + pick = int(choice) - 1 + if 0 <= pick < len(proposals): + selected = proposals[pick].tag + tags_for_scenario = [selected] + elif choice.startswith("@"): + selected = choice + tags_for_scenario = [selected] + + if tags_for_scenario: + injections.append((line_index, tags_for_scenario)) + + feature_entry["scenarios"].append( + { + "scenario_title": title, + "proposals": proposal_dicts, + "selected": selected, + } + ) + all_proposals.extend(proposal_dicts) + + updated = apply_line_based_tags(text, injections) + if updated != raw_text: + feature_path.write_text(updated, encoding="utf-8") + document["features"].append(feature_entry) + + write_json(paths.tags_proposals, document) + return TagResult( + proposals_path=paths.tags_proposals, + proposals=all_proposals, + skipped=False, + ) + + +__all__ = [ + "apply_line_based_tags", + "resolve_ruleset_sources", + "repair_split_scenario_tags", + "run_tag", +] + +# Public alias for tests and migration tooling. +repair_split_scenario_tags = _repair_split_scenario_tags diff --git a/packages/pickled-core/src/pickled_core/mine/types.py b/packages/pickled-core/src/pickled_core/mine/types.py index 32cc6a3..9c06804 100644 --- a/packages/pickled-core/src/pickled_core/mine/types.py +++ b/packages/pickled-core/src/pickled_core/mine/types.py @@ -4,7 +4,9 @@ from dataclasses import dataclass, field from pathlib import Path -from typing import Any +from typing import Any, Literal + +from pickled_rules.types import RuleSet @dataclass @@ -68,5 +70,212 @@ def mining_report(self) -> Path: def runs_dir(self) -> Path: return self.root / "runs" + @property + def code_context_dir(self) -> Path: + return self.root / "code-context" + + +ResolutionKind = Literal[ + "free_function", + "self_method", + "constructor_method", + "annotated_var", + "annotated_param", + "assigned_constructor", + "module_constructor", + "unresolved", +] + + +@dataclass(frozen=True, slots=True) +class CalleeRef: + """Callee edge recorded during code reading (serializable subset).""" + + expression: str + name: str + module: str + file: str + lineno: int + resolved: bool + reason: str = "" + key: str = "" + resolution_kind: str = "unresolved" + + +@dataclass +class CodeContext: + """Aggregate code units collected for one surface.""" + + surface_id: str + depth: str + scope: str + units_collected: int + total_lines: int + truncated: bool + no_definition: bool + context_path: Path | None = None + + +@dataclass +class CycleReport: + """Cycle detection output for code reading.""" + + cycles: list[list[str]] = field(default_factory=list) + + @property + def count(self) -> int: + return len(self.cycles) + + +@dataclass +class CodeStageResult: + """Result of the code reading stage.""" + + output_dir: Path + code_context_dir: Path + written_paths: list[Path] = field(default_factory=list) + cycles_path: Path | None = None + cycle_count: int = 0 + + +@dataclass +class RelevantAdrRef: + """ADR entry attached to a mined surface.""" + + number: str + title: str + status: str + general: bool = False + + +@dataclass +class MinedSurfaceContext: + """Per-surface fields used when rendering stories.""" + + docstring: str = "" + relevant_adrs: list[RelevantAdrRef] = field(default_factory=list) + + +@dataclass +class StoryResult: + """One emitted user story file.""" + + surface_id: str + story_path: Path + skipped: bool = False + + +@dataclass +class FeatureResult: + """One drafted feature file.""" + + surface_id: str + feature_path: Path + story_path: Path + skipped: bool = False + + +@dataclass +class FeaturesStageResult: + """Aggregate result of the features stage.""" + + output_dir: Path + results: list[FeatureResult] = field(default_factory=list) + skipped_entire_stage: bool = False + warnings: list[str] = field(default_factory=list) + + +@dataclass +class TagProposal: + """A single proposed scenario tag.""" + + tag: str + rule_id: str + short_name: str + rule_title: str + score: int + + +@dataclass +class LoadedRulesetEntry: + """One rule set file loaded for tagging.""" + + short_name: str + path: Path + ruleset: RuleSet + + +@dataclass +class RulesetSources: + """Loaded rule sets for tagging.""" + + config_root: Path + rulesets: list[LoadedRulesetEntry] = field(default_factory=list) + + +@dataclass +class TagResult: + """Result of the tag stage.""" + + proposals_path: Path + proposals: list[Any] = field(default_factory=list) + skipped: bool = False + warnings: list[str] = field(default_factory=list) + + +@dataclass +class CoverageRulesetResult: + """Coverage gate outcome for one rule set.""" + + short_name: str + verdict: str + notes: str + referenced_rule_ids: list[str] = field(default_factory=list) + unreferenced_strict_rule_ids: list[str] = field(default_factory=list) + unknown_references: list[dict[str, str]] = field(default_factory=list) + + +@dataclass +class AmbiguityFeatureResult: + """Ambiguity gate outcome for one feature file.""" + + feature_path: str + verdict: str + finding_count: int + skipped: bool + notes: str + + +@dataclass +class EvaluationResult: + """Result of the evaluate stage.""" + + coverage_path: Path + ambiguity_path: Path + coverage: list[CoverageRulesetResult] = field(default_factory=list) + ambiguity: list[AmbiguityFeatureResult] = field(default_factory=list) + surfaces_filter: tuple[str, ...] = () + -__all__ = ["InventoryResult", "MiningPaths", "StageResult"] +__all__ = [ + "AmbiguityFeatureResult", + "CalleeRef", + "CodeContext", + "CodeStageResult", + "CoverageRulesetResult", + "CycleReport", + "EvaluationResult", + "FeatureResult", + "FeaturesStageResult", + "InventoryResult", + "LoadedRulesetEntry", + "MinedSurfaceContext", + "MiningPaths", + "ResolutionKind", + "RelevantAdrRef", + "RulesetSources", + "StageResult", + "StoryResult", + "TagProposal", + "TagResult", +] diff --git a/packages/pickled-core/tests/fixtures/adrs_sample/0001-pickled-diff-package.md b/packages/pickled-core/tests/fixtures/adrs_sample/0001-pickled-diff-package.md new file mode 100644 index 0000000..4f8924e --- /dev/null +++ b/packages/pickled-core/tests/fixtures/adrs_sample/0001-pickled-diff-package.md @@ -0,0 +1,13 @@ +# ADR-0001: pickled-diff package + +## Status + +Accepted + +## Date + +2026-05-19 + +## Context + +This ADR covers the pickled-diff leaf package only. diff --git a/packages/pickled-core/tests/fixtures/adrs_sample/0002-cache-and-budget.md b/packages/pickled-core/tests/fixtures/adrs_sample/0002-cache-and-budget.md new file mode 100644 index 0000000..636d8db --- /dev/null +++ b/packages/pickled-core/tests/fixtures/adrs_sample/0002-cache-and-budget.md @@ -0,0 +1,13 @@ +# ADR-0002: cache and budget in pickled.config + +## Status + +Proposed + +## Date + +2026-05-20 + +## Context + +Disk cache and budget caps for LLM calls in pickled-core. diff --git a/packages/pickled-core/tests/fixtures/adrs_sample/0003-old-workspace.md b/packages/pickled-core/tests/fixtures/adrs_sample/0003-old-workspace.md new file mode 100644 index 0000000..361702f --- /dev/null +++ b/packages/pickled-core/tests/fixtures/adrs_sample/0003-old-workspace.md @@ -0,0 +1,13 @@ +# ADR-0003: monorepo workspace layout + +## Status + +Superseded + +## Date + +2026-05-18 + +## Context + +Workspace-wide uv monorepo conventions. diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/pyproject.toml b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/pyproject.toml new file mode 100644 index 0000000..ecc9a92 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/pyproject.toml @@ -0,0 +1,14 @@ +[project] +name = "callgraph-target" +version = "0.0.1" +requires-python = ">=3.11" + +[project.scripts] +callgraph = "callgraph_target.cli:main" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/callgraph_target"] diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/__init__.py b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/__init__.py new file mode 100644 index 0000000..1d4ac83 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/__init__.py @@ -0,0 +1 @@ +"""Call-graph fixture package.""" diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/chain.py b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/chain.py new file mode 100644 index 0000000..e21fd93 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/chain.py @@ -0,0 +1,13 @@ +"""Linear call chain for hop-depth tests.""" + + +def step_two() -> str: + return "two" + + +def step_one() -> str: + return step_two() + "!" + + +def entry() -> str: + return step_one() diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/cli.py b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/cli.py new file mode 100644 index 0000000..f0c6719 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/cli.py @@ -0,0 +1,25 @@ +"""Click CLI for callgraph inventory tests.""" + +from __future__ import annotations + +import click + +from callgraph_target.chain import entry +from callgraph_target.cycle import ping + + +@click.group() +def main() -> None: + """Callgraph fixture CLI.""" + + +@main.command("run-chain") +def run_chain() -> None: + """Execute the linear chain entrypoint for mining tests (significant help).""" + click.echo(entry()) + + +@main.command("run-cycle") +def run_cycle() -> None: + """Execute the ping/pong cycle entrypoint for mining tests (significant help).""" + ping() diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/cross_pkg.py b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/cross_pkg.py new file mode 100644 index 0000000..5d3b96f --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/cross_pkg.py @@ -0,0 +1,7 @@ +"""Cross-package callee for any-pickled scope tests.""" + +from pickled_peer.helper import peer_helper + + +def cross_entry() -> str: + return peer_helper() diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/cycle.py b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/cycle.py new file mode 100644 index 0000000..0420778 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/cycle.py @@ -0,0 +1,9 @@ +"""Deliberate mutual recursion for cycle tests.""" + + +def pong() -> None: + ping() + + +def ping() -> None: + pong() diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/decorator_demo.py b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/decorator_demo.py new file mode 100644 index 0000000..1ce9604 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/decorator_demo.py @@ -0,0 +1,15 @@ +"""Decorator registration must not appear as behavioral callees.""" + +from __future__ import annotations + +import click + + +@click.group() +def main() -> None: + """CLI root.""" + + +@main.command() +def entry() -> None: + """Surface under test.""" diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/fanout.py b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/fanout.py new file mode 100644 index 0000000..8f8b274 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/fanout.py @@ -0,0 +1,51 @@ +"""Many direct callees to exercise max_callees.""" + + +def callee_a() -> int: + return 1 + + +def callee_b() -> int: + return 2 + + +def callee_c() -> int: + return 3 + + +def callee_d() -> int: + return 4 + + +def callee_e() -> int: + return 5 + + +def callee_f() -> int: + return 6 + + +def callee_g() -> int: + return 7 + + +def callee_h() -> int: + return 8 + + +def callee_i() -> int: + return 9 + + +def entry() -> int: + return ( + callee_a() + + callee_b() + + callee_c() + + callee_d() + + callee_e() + + callee_f() + + callee_g() + + callee_h() + + callee_i() + ) diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/mixed.py b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/mixed.py new file mode 100644 index 0000000..aa76450 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/mixed.py @@ -0,0 +1,16 @@ +"""Mixed callee kinds: self, import, stdlib.""" + +from __future__ import annotations + +import json + +from callgraph_target import chain + + +class Worker: + def helper(self) -> int: + return 1 + + def run(self) -> str: + payload = json.dumps({"n": self.helper()}) + return payload + chain.entry() diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/noise_demo.py b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/noise_demo.py new file mode 100644 index 0000000..83e67bb --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/noise_demo.py @@ -0,0 +1,24 @@ +"""Callee noise filtering: stdlib methods vs intra-project calls.""" + +from __future__ import annotations + +from pathlib import Path + + +class Worker: + def process(self) -> str: + return "ok" + + +def strip_only() -> str: + text = "hello" + return text.strip() + + +def path_read(story_file: str) -> str: + return Path(story_file).read_text(encoding="utf-8") + + +def use_worker() -> str: + worker = Worker() + return worker.process() diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/protocol_demo.py b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/protocol_demo.py new file mode 100644 index 0000000..48338b0 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/protocol_demo.py @@ -0,0 +1,14 @@ +"""Protocol-style dispatch (unresolved).""" + + +class StubLLM: + def complete(self) -> str: + return "ok" + + +class User: + def __init__(self) -> None: + self._llm = StubLLM() + + def go(self) -> str: + return self._llm.complete() diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/typing_literal.py b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/typing_literal.py new file mode 100644 index 0000000..6837265 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/callgraph_target/src/callgraph_target/typing_literal.py @@ -0,0 +1,9 @@ +"""Annotations using typing.Literal must not crash the resolver.""" + +from __future__ import annotations + +from typing import Literal + + +def entry(mode: Literal["terraform", "opentofu"]) -> str: + return mode diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/pickled_peer/pyproject.toml b/packages/pickled-core/tests/fixtures/callgraph_target/packages/pickled_peer/pyproject.toml new file mode 100644 index 0000000..fbeb066 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/pickled_peer/pyproject.toml @@ -0,0 +1,11 @@ +[project] +name = "pickled-peer" +version = "0.0.1" +requires-python = ">=3.11" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/pickled_peer"] diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/pickled_peer/src/pickled_peer/__init__.py b/packages/pickled-core/tests/fixtures/callgraph_target/packages/pickled_peer/src/pickled_peer/__init__.py new file mode 100644 index 0000000..501476d --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/pickled_peer/src/pickled_peer/__init__.py @@ -0,0 +1 @@ +"""Peer package for cross-import tests.""" diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/packages/pickled_peer/src/pickled_peer/helper.py b/packages/pickled-core/tests/fixtures/callgraph_target/packages/pickled_peer/src/pickled_peer/helper.py new file mode 100644 index 0000000..a7f4c39 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/packages/pickled_peer/src/pickled_peer/helper.py @@ -0,0 +1,5 @@ +"""Helper reached from callgraph_target.""" + + +def peer_helper() -> str: + return "peer" diff --git a/packages/pickled-core/tests/fixtures/callgraph_target/pyproject.toml b/packages/pickled-core/tests/fixtures/callgraph_target/pyproject.toml new file mode 100644 index 0000000..8a91b45 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/callgraph_target/pyproject.toml @@ -0,0 +1,7 @@ +[project] +name = "callgraph-workspace" +version = "0.0.0" +requires-python = ">=3.11" + +[tool.uv.workspace] +members = ["packages/*"] diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/pyproject.toml b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/pyproject.toml new file mode 100644 index 0000000..63afc8c --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/pyproject.toml @@ -0,0 +1,11 @@ +[project] +name = "resolver-zoo" +version = "0.0.1" +requires-python = ">=3.11" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/resolver_zoo"] diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/__init__.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/__init__.py new file mode 100644 index 0000000..c1b7f4f --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/__init__.py @@ -0,0 +1 @@ +"""Resolver adversarial fixture package.""" diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/annotated.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/annotated.py new file mode 100644 index 0000000..3046336 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/annotated.py @@ -0,0 +1,20 @@ +"""Type-annotation based resolution.""" + + +class Worker: + def process(self) -> str: + return "p" + + +def via_param(w: Worker) -> str: + return w.process() + + +def via_var() -> str: + x: Worker = Worker() + return x.process() + + +def via_assign() -> str: + x = Worker() + return x.process() diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/collision.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/collision.py new file mode 100644 index 0000000..9bce19e --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/collision.py @@ -0,0 +1,15 @@ +"""Ambiguous receiver type.""" + + +class Alpha: + def run(self) -> str: + return "a" + + +class Beta: + def run(self) -> str: + return "b" + + +def entry() -> str: + return mystery.run() diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/ctor_cycle.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/ctor_cycle.py new file mode 100644 index 0000000..90eff70 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/ctor_cycle.py @@ -0,0 +1,15 @@ +"""Constructor cycle A -> B -> A.""" + + +class A: + def foo(self) -> None: + B().bar() + + +class B: + def bar(self) -> None: + A().foo() + + +def entry() -> None: + A().foo() diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/ctor_method.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/ctor_method.py new file mode 100644 index 0000000..10e9254 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/ctor_method.py @@ -0,0 +1,14 @@ +"""Constructor-then-method resolution.""" + + +class Worker: + def process(self) -> str: + return "processed" + + +def entry(cfg: str) -> str: + return Worker(cfg).process() + + +def entry_paren(cfg: str) -> str: + return (Worker(cfg)).process() diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/degenerate_empty.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/degenerate_empty.py new file mode 100644 index 0000000..f2120bf --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/degenerate_empty.py @@ -0,0 +1 @@ +"""Empty module.""" diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/degenerate_syntax_error.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/degenerate_syntax_error.py new file mode 100644 index 0000000..58fc37c --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/degenerate_syntax_error.py @@ -0,0 +1,2 @@ +def broken( # intentional syntax error + return 1 diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/drafter_shape.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/drafter_shape.py new file mode 100644 index 0000000..c0a184d --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/drafter_shape.py @@ -0,0 +1,13 @@ +"""Mirrors FeatureDrafter(llm).domain_method() pattern.""" + + +class DomainClass: + def domain_method(self, arg: str) -> str: + return self._helper() + arg + + def _helper(self) -> str: + return "domain-" + + +def entry(dep: str, story: str) -> str: + return DomainClass(dep).domain_method(story) diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/factory_hidden.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/factory_hidden.py new file mode 100644 index 0000000..df9ae20 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/factory_hidden.py @@ -0,0 +1,14 @@ +"""Factory return type unknown.""" + + +class Worker: + def process(self) -> str: + return "p" + + +def get_worker() -> Worker: + return Worker() + + +def entry() -> str: + return get_worker().process() diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/inherited.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/inherited.py new file mode 100644 index 0000000..868af42 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/inherited.py @@ -0,0 +1,14 @@ +"""Inherited method not resolved in v1.""" + + +class Base: + def run(self) -> str: + return "base" + + +class Child(Base): + pass + + +def entry() -> str: + return Child().run() diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/method_chain.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/method_chain.py new file mode 100644 index 0000000..08c32d2 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/method_chain.py @@ -0,0 +1,13 @@ +"""Method chain: first hop resolves, rest refuses.""" + + +class Worker: + def process(self) -> str: + return "p" + + def finalize(self) -> str: + return "f" + + +def entry(cfg: str) -> str: + return Worker(cfg).process().finalize() diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/module_ctor.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/module_ctor.py new file mode 100644 index 0000000..4c1838e --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/module_ctor.py @@ -0,0 +1,7 @@ +"""Module-qualified constructor.""" + +from resolver_zoo import ctor_method as mod + + +def entry(cfg: str) -> str: + return mod.Worker(cfg).process() diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/nested_ctor.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/nested_ctor.py new file mode 100644 index 0000000..418c7e4 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/nested_ctor.py @@ -0,0 +1,15 @@ +"""Constructor nested in constructor argument.""" + + +class Builder: + def build(self) -> str: + return "built" + + +class Worker: + def process(self, payload: str) -> str: + return payload + + +def entry(x: str) -> str: + return Worker(Builder(x).build()).process() diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/rebind.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/rebind.py new file mode 100644 index 0000000..604aa22 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/rebind.py @@ -0,0 +1,17 @@ +"""Rebound variable must refuse.""" + + +class Worker: + def process(self) -> str: + return "w" + + +class Other: + def process(self) -> str: + return "o" + + +def entry() -> str: + x = Worker() + x = Other() + return x.process() diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/recursion.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/recursion.py new file mode 100644 index 0000000..5396462 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/recursion.py @@ -0,0 +1,16 @@ +"""Recursion within one class.""" + + +class A: + def foo(self) -> None: + self.foo() + + def bar(self) -> None: + self.baz() + + def baz(self) -> None: + self.foo() + + +def entry() -> None: + A().foo() diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/refuse_receivers.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/refuse_receivers.py new file mode 100644 index 0000000..98224d1 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/refuse_receivers.py @@ -0,0 +1,44 @@ +"""Calls that must stay unresolved.""" + + +class StubLLM: + def complete(self) -> str: + return "ok" + + +class User: + def __init__(self) -> None: + self._llm = StubLLM() + + def protocol_call(self) -> str: + return self._llm.complete() + + +def unannotated_param(w) -> str: + return w.process() + + +class Worker: + def process(self) -> str: + return "p" + + +def subscript_receiver(items: list[Worker]) -> str: + return items[0].process() + + +def ternary_receiver(flag: bool, a: Worker, b: Worker) -> str: + return (a if flag else b).process() + + +def chained_unknown() -> str: + return factory().build().run() + + +def factory(): + return Worker() + + +def getattr_dynamic(obj: object) -> str: + fn = getattr(obj, "process") + return fn() diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/structural.py b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/structural.py new file mode 100644 index 0000000..bd5fe65 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/packages/resolver_zoo/src/resolver_zoo/structural.py @@ -0,0 +1,23 @@ +"""Property, static, class, async methods.""" + + +class Demo: + @property + def label(self) -> str: + return "label" + + @staticmethod + def static_run() -> str: + return "static" + + @classmethod + def class_run(cls) -> str: + return "class" + + async def async_run(self) -> str: + return "async" + + +def entry() -> str: + d = Demo() + return d.label + Demo.static_run() + Demo.class_run() diff --git a/packages/pickled-core/tests/fixtures/resolver_zoo/pyproject.toml b/packages/pickled-core/tests/fixtures/resolver_zoo/pyproject.toml new file mode 100644 index 0000000..287ab56 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/resolver_zoo/pyproject.toml @@ -0,0 +1,7 @@ +[project] +name = "resolver-zoo" +version = "0.0.0" +requires-python = ">=3.11" + +[tool.uv.workspace] +members = ["packages/*"] diff --git a/packages/pickled-core/tests/fixtures/tiny_target/packages/tiny_app/pyproject.toml b/packages/pickled-core/tests/fixtures/tiny_target/packages/tiny_app/pyproject.toml new file mode 100644 index 0000000..619eff5 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/tiny_target/packages/tiny_app/pyproject.toml @@ -0,0 +1,15 @@ +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "tiny-target" +version = "0.0.1" +requires-python = ">=3.11" +dependencies = ["click>=8.1"] + +[project.scripts] +tiny-cli = "tiny_app.cli:main" + +[tool.hatch.build.targets.wheel] +packages = ["src/tiny_app"] diff --git a/packages/pickled-core/tests/fixtures/tiny_target/packages/tiny_app/src/tiny_app/__init__.py b/packages/pickled-core/tests/fixtures/tiny_target/packages/tiny_app/src/tiny_app/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/packages/pickled-core/tests/fixtures/tiny_target/packages/tiny_app/src/tiny_app/cli.py b/packages/pickled-core/tests/fixtures/tiny_target/packages/tiny_app/src/tiny_app/cli.py new file mode 100644 index 0000000..b2d2fde --- /dev/null +++ b/packages/pickled-core/tests/fixtures/tiny_target/packages/tiny_app/src/tiny_app/cli.py @@ -0,0 +1,23 @@ +"""Minimal Click CLI for mine inventory tests.""" + +from __future__ import annotations + +import click + + +@click.group() +def main() -> None: + """Tiny target CLI.""" + + +@main.command() +@click.argument("name") +def greet(name: str) -> None: + """Greet someone by name (required argument for significance).""" + click.echo(f"Hello, {name}!") + + +@main.command() +def ping() -> None: + """Return pong for health checks.""" + click.echo("pong") diff --git a/packages/pickled-core/tests/fixtures/tiny_target/packages/umbrella_member/pyproject.toml b/packages/pickled-core/tests/fixtures/tiny_target/packages/umbrella_member/pyproject.toml new file mode 100644 index 0000000..b999aaf --- /dev/null +++ b/packages/pickled-core/tests/fixtures/tiny_target/packages/umbrella_member/pyproject.toml @@ -0,0 +1,18 @@ +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "umbrella-member" +version = "0.0.1" +requires-python = ">=3.11" +dependencies = ["click>=8.1"] + +[project.scripts] +pickled-spec = "umbrella_member.cli:main" + +[project.entry-points."pickled.mcp.subservers"] +bdd = "umbrella_member.mcp:build_server" + +[tool.hatch.build.targets.wheel] +packages = ["src/umbrella_member"] diff --git a/packages/pickled-core/tests/fixtures/tiny_target/packages/umbrella_member/src/umbrella_member/__init__.py b/packages/pickled-core/tests/fixtures/tiny_target/packages/umbrella_member/src/umbrella_member/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/packages/pickled-core/tests/fixtures/tiny_target/packages/umbrella_member/src/umbrella_member/cli.py b/packages/pickled-core/tests/fixtures/tiny_target/packages/umbrella_member/src/umbrella_member/cli.py new file mode 100644 index 0000000..f2b6114 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/tiny_target/packages/umbrella_member/src/umbrella_member/cli.py @@ -0,0 +1,21 @@ +"""Umbrella-style CLI placeholder for MCP detection tests.""" + +from __future__ import annotations + +import click + + +@click.group() +def main() -> None: + """Umbrella member CLI.""" + + +@main.group() +def mcp() -> None: + """MCP subcommands.""" + + +@mcp.command("serve") +def mcp_serve() -> None: + """Plumbing: start MCP server.""" + click.echo("serve") diff --git a/packages/pickled-core/tests/fixtures/tiny_target/packages/umbrella_member/src/umbrella_member/mcp.py b/packages/pickled-core/tests/fixtures/tiny_target/packages/umbrella_member/src/umbrella_member/mcp.py new file mode 100644 index 0000000..fe0dc41 --- /dev/null +++ b/packages/pickled-core/tests/fixtures/tiny_target/packages/umbrella_member/src/umbrella_member/mcp.py @@ -0,0 +1,5 @@ +"""Placeholder MCP subserver entry point.""" + + +def build_server() -> object: + return object() diff --git a/packages/pickled-core/tests/fixtures/tiny_target/pyproject.toml b/packages/pickled-core/tests/fixtures/tiny_target/pyproject.toml index 26121f9..2b03125 100644 --- a/packages/pickled-core/tests/fixtures/tiny_target/pyproject.toml +++ b/packages/pickled-core/tests/fixtures/tiny_target/pyproject.toml @@ -1,16 +1,2 @@ -[build-system] -requires = ["hatchling"] -build-backend = "hatchling.build" - -[project] -name = "tiny-target" -version = "0.0.1" -description = "Minimal fixture for mine inventory tests" -requires-python = ">=3.11" -dependencies = ["click>=8.1"] - -[project.scripts] -tiny-target = "tiny_target.cli:main" - -[tool.hatch.build.targets.wheel] -packages = ["src/tiny_target"] +[tool.uv.workspace] +members = ["packages/*"] diff --git a/packages/pickled-core/tests/test_llm_sanitize.py b/packages/pickled-core/tests/test_llm_sanitize.py new file mode 100644 index 0000000..7f52d74 --- /dev/null +++ b/packages/pickled-core/tests/test_llm_sanitize.py @@ -0,0 +1,256 @@ +"""Tests for :func:`pickled_core.llm.sanitize.strip_markdown_fence`.""" + +from __future__ import annotations + +import json +from collections.abc import Mapping +from typing import Any +from unittest.mock import patch + +import pytest +from pickled_core.llm.sanitize import strip_markdown_fence + +_GHERKIN_BODY = "Feature: Auth\n Scenario: Login\n Given a user\n" +_VALID_RULESET = """metadata: + source_id: test-src + source_title: Test Rules + applies_to: internal-api + maintainer: tests + source_version: '1.0' + active_from: '2026-05-25' +rules: + - id: api-naming + title: Use consistent paths + description: REST paths use kebab-case resource names. + enforcement: strict +""" +_VALID_SQL = """-- intent: add column +CREATE TABLE users (id INTEGER PRIMARY KEY); +""" +_VALID_CORPUS = json.dumps([{"name": "a", "payload": "1"}]) +_VALID_HCL = 'resource "aws_s3_bucket" "x" {\n bucket = "example"\n}\n' +_VALID_OPENAPI = """get: + summary: List users + operationId: listUsers + responses: + '200': + description: OK +""" + + +def test_strips_gherkin_fence() -> None: + fenced = f"```gherkin\n{_GHERKIN_BODY}\n```" + assert strip_markdown_fence(fenced) == _GHERKIN_BODY.strip() + + +def test_strips_fence_no_language() -> None: + body = "line one\nline two" + fenced = f"```\n{body}\n```" + assert strip_markdown_fence(fenced) == body + + +def test_strips_yaml_json_hcl_fences() -> None: + yaml_body = "key: value\n" + json_body = '{"a": 1}' + hcl_body = 'resource "x" "y" {}' + assert strip_markdown_fence(f"```yaml\n{yaml_body}\n```") == yaml_body.strip() + assert strip_markdown_fence(f"```json\n{json_body}\n```") == json_body + assert strip_markdown_fence(f"```hcl\n{hcl_body}\n```") == hcl_body + + +def test_unfenced_text_unchanged() -> None: + assert strip_markdown_fence(_GHERKIN_BODY) == _GHERKIN_BODY.strip() + + +def test_inner_fence_preserved() -> None: + text = ( + "Feature: Docs\n" + " Scenario: Example\n" + " Given a snippet:\n" + " ```python\n" + " print(1)\n" + " ```\n" + ) + assert strip_markdown_fence(text) == text.strip() + + +def test_idempotent() -> None: + fenced = f"```gherkin\n{_GHERKIN_BODY}\n```" + once = strip_markdown_fence(fenced) + assert strip_markdown_fence(once) == once + + +def test_whitespace_around_fence_handled() -> None: + fenced = f" \n```gherkin\n{_GHERKIN_BODY}\n``` \n" + assert strip_markdown_fence(fenced) == _GHERKIN_BODY.strip() + + +def test_empty_string() -> None: + assert strip_markdown_fence("") == "" + assert strip_markdown_fence(" \n ") == "" + + +def test_fence_with_trailing_prose_not_stripped() -> None: + fenced = "```\nbody\n```\nextra line" + assert strip_markdown_fence(fenced) == fenced.strip() + + +def test_feature_drafter_strips_fence() -> None: + from pickled_bdd.drafter import FeatureDrafter + from pickled_core.cost.models import TokenUsage + from pickled_core.llm.base import Completion, LLMClient, Message + + class FakeLLM(LLMClient): + provider_key = "fake" + + def complete( + self, + *, + messages: list[Message], + model: str, + max_tokens: int, + temperature: float | None, + stop: list[str] | None, + extras: Mapping[str, Any] | None, + ) -> Completion: + _ = messages, model, max_tokens, temperature, stop, extras + return Completion( + text=f"```gherkin\n{_GHERKIN_BODY}\n```", + usage=TokenUsage(), + model_id_resolved=model, + raw_response=None, + ) + + def count_tokens(self, messages: list[Message], model: str) -> int: + _ = messages, model + return 1 + + result = FeatureDrafter(FakeLLM()).draft_from_story("story") + assert result.text == _GHERKIN_BODY.strip() + assert "```" not in result.text + + +def test_rules_drafter_strips_fence() -> None: + from pickled_bdd.testing import CannedLLMClient + from pickled_rules.drafter import RuleSetDrafter + + fenced = f"```yaml\n{_VALID_RULESET}\n```" + result = RuleSetDrafter(CannedLLMClient(fenced)).draft_from_brief( + brief_text="API naming", + ruleset_short_name="api", + source_id="test-src", + applies_to="internal-api", + active_from="2026-05-25", + ) + assert result.text.startswith("metadata:") + assert "```" not in result.text + + +def test_migration_drafter_strips_fence() -> None: + from pickled_bdd.testing import CannedLLMClient + from pickled_data.drafter import MigrationDrafter + + fenced = f"```sql\n{_VALID_SQL}\n```" + result = MigrationDrafter(CannedLLMClient(fenced)).draft_from_intent( + intent_text="add table", + dialect="sqlite", + ) + assert result.text.startswith("-- intent:") + assert "```" not in result.text + + +def test_corpus_drafter_strips_fence() -> None: + from pickled_bdd.testing import CannedLLMClient + from pickled_diff.drafter import CorpusDrafter + + fenced = f"```json\n{_VALID_CORPUS}\n```" + result = CorpusDrafter(CannedLLMClient(fenced)).draft_from_examples( + seed_examples=[{"name": "a", "payload": "1"}], + target_size=1, + ) + assert len(result.items) == 1 + assert result.items[0]["name"] == "a" + + +def test_openapi_drafter_strips_fence() -> None: + pytest.importorskip("openapi_spec_validator") + + from pickled_core.cost.models import TokenUsage + from pickled_core.llm.base import Completion, LLMClient, Message + from pickled_schema.openapi.drafter import OpenAPIDrafter + + class FakeLLM(LLMClient): + provider_key = "fake" + + def complete( + self, + *, + messages: list[Message], + model: str, + max_tokens: int, + temperature: float | None, + stop: list[str] | None, + extras: Mapping[str, Any] | None, + ) -> Completion: + _ = messages, model, max_tokens, temperature, stop, extras + return Completion( + text=f"```yaml\n{_VALID_OPENAPI}\n```", + usage=TokenUsage(), + model_id_resolved=model, + raw_response=None, + ) + + def count_tokens(self, messages: list[Message], model: str) -> int: + _ = messages, model + return 1 + + artifact = OpenAPIDrafter(FakeLLM()).draft_endpoint( + "GET", + "/users", + "Scenario: List users", + ) + assert "summary: List users" in artifact.content + assert "```" not in artifact.content + + +def test_iac_drafter_strips_fence() -> None: + from pickled_core.cost.models import TokenUsage + from pickled_core.llm.base import Completion, LLMClient, Message + from pickled_iac.drafter import IaCDrafter + from pickled_iac.types import ValidateResult + + class FakeLLM(LLMClient): + provider_key = "fake" + + def complete( + self, + *, + messages: list[Message], + model: str, + max_tokens: int, + temperature: float | None, + stop: list[str] | None, + extras: Mapping[str, Any] | None, + ) -> Completion: + _ = messages, model, max_tokens, temperature, stop, extras + return Completion( + text=f"```hcl\n{_VALID_HCL}\n```", + usage=TokenUsage(), + model_id_resolved=model, + raw_response=None, + ) + + def count_tokens(self, messages: list[Message], model: str) -> int: + _ = messages, model + return 1 + + with ( + patch("pickled_iac.drafter.iac_binary", return_value="terraform"), + patch( + "pickled_iac.drafter.validate", + return_value=ValidateResult(valid=True, diagnostics=()), + ), + ): + artifact = IaCDrafter(FakeLLM()).draft_module("provision bucket") + assert artifact.content.strip() == _VALID_HCL.strip() + assert "```" not in artifact.content diff --git a/packages/pickled-core/tests/test_mine_code_reader.py b/packages/pickled-core/tests/test_mine_code_reader.py new file mode 100644 index 0000000..14acc3d --- /dev/null +++ b/packages/pickled-core/tests/test_mine_code_reader.py @@ -0,0 +1,352 @@ +"""Tests for mine code_reader.""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +from pickled_core.mine.code_reader import ( + CalleeScope, + SurfaceRef, + collect_context, + find_cycles, + resolve_callees, +) + +_FIXTURE = Path(__file__).resolve().parent / "fixtures" / "callgraph_target" +_PKG_SRC = _FIXTURE / "packages/callgraph_target/src" +_PEER_SRC = _FIXTURE / "packages/pickled_peer/src" + + +@pytest.fixture(autouse=True) +def _fixture_paths() -> None: + for entry in (_PKG_SRC, _PEER_SRC): + text = str(entry) + if text not in sys.path: + sys.path.insert(0, text) + + +def _ref(name: str, file: str, lineno: int) -> SurfaceRef: + rel = f"packages/callgraph_target/src/callgraph_target/{file}" + return SurfaceRef( + surface_id=f"cg_{name.replace('.', '_')}", + kind="function", + package="callgraph-target", + name=name, + module="callgraph_target", + file=rel, + line=lineno, + ) + + +def test_body_depth_returns_root_body_only() -> None: + surface = _ref("entry", "chain.py", 10) + ctx = collect_context( + surface, + _FIXTURE, + depth="body", + scope="same-package", + max_hops=1, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ) + assert ctx.root is not None + assert len(ctx.units) == 1 + assert "step_one" in ctx.root.source + + +def test_callgraph_depth_one_hop_collects_direct_callees() -> None: + surface = _ref("entry", "chain.py", 10) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="same-package", + max_hops=1, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ) + names = {u.qualname for u in ctx.units} + assert "entry" in names + assert "step_one" in names + assert "step_two" not in names + + +def test_callgraph_depth_two_hops() -> None: + surface = _ref("entry", "chain.py", 10) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="same-package", + max_hops=2, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ) + names = {u.qualname for u in ctx.units} + assert "step_two" in names + + +def test_cycle_does_not_infinite_loop() -> None: + surface = _ref("ping", "cycle.py", 7) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="same-package", + max_hops=3, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ) + keys = [u.key for u in ctx.units] + assert keys.count(ctx.root.key if ctx.root else "") <= 1 + assert len(keys) == len(set(keys)) + + +def test_max_callees_cap_respected() -> None: + surface = _ref("entry", "fanout.py", 40) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="same-package", + max_hops=1, + max_callees=3, + max_code_lines=400, + package="callgraph-target", + ) + assert len(ctx.units) <= 3 + assert ctx.truncated + + +def test_max_code_lines_cap_respected() -> None: + surface = _ref("entry", "fanout.py", 40) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="same-package", + max_hops=1, + max_callees=20, + max_code_lines=5, + package="callgraph-target", + ) + total = sum(u.line_count for u in ctx.units) + assert total <= 6 + assert ctx.truncated + + +def test_self_scope_resolves_same_class_methods() -> None: + surface = _ref("Worker.run", "mixed.py", 14) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="self", + max_hops=1, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ) + names = {u.qualname for u in ctx.units} + assert "Worker.helper" in names + assert "entry" not in names + + +def test_same_package_scope_resolves_package_imports() -> None: + surface = _ref("Worker.run", "mixed.py", 14) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="same-package", + max_hops=1, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ) + names = {u.qualname for u in ctx.units} + assert "entry" in names + + +def test_any_pickled_scope_resolves_cross_package() -> None: + surface = _ref("cross_entry", "cross_pkg.py", 7) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="any-pickled", + max_hops=1, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ) + modules = {u.module for u in ctx.units} + assert any(m.startswith("pickled_peer") for m in modules) + + +def test_stdlib_calls_ignored() -> None: + surface = _ref("Worker.run", "mixed.py", 14) + root = collect_context( + surface, + _FIXTURE, + depth="body", + scope="same-package", + max_hops=1, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ).root + assert root is not None + refs = resolve_callees( + root, + scope="same-package", + package="callgraph-target", + target=_FIXTURE, + ) + expressions = {r.expression for r in refs} + assert "json.dumps" not in expressions + + +def test_protocol_dispatch_unresolved() -> None: + surface = _ref("User.go", "protocol_demo.py", 12) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="same-package", + max_hops=1, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ) + assert any( + not ref.resolved + and ref.resolution_kind == "unresolved" + and "protocol" in ref.reason.lower() + for ref in ctx.unresolved + ) + + +def test_find_cycles_detects_ping_pong() -> None: + edges = [ + ("callgraph_target:ping", "callgraph_target:pong"), + ("callgraph_target:pong", "callgraph_target:ping"), + ] + cycles = find_cycles(edges) + assert cycles + + +def _fixture_ref(qualname: str, rel_file: str, lineno: int) -> object: + from pickled_core.mine.code_reader import SurfaceRef + + return SurfaceRef( + surface_id="noise", + kind="function", + package="callgraph-target", + name=qualname, + module="callgraph_target", + file=f"packages/callgraph_target/src/callgraph_target/{rel_file}", + line=lineno, + ) + + +def test_stdlib_str_method_not_in_unresolved() -> None: + surface = _fixture_ref("strip_only", "noise_demo.py", 13) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="same-package", + max_hops=1, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ) + expressions = {r.expression for r in ctx.unresolved} + assert "text.strip" not in expressions + assert not any("strip" in e for e in expressions) + + +def test_path_method_not_in_unresolved() -> None: + surface = _fixture_ref("path_read", "noise_demo.py", 17) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="same-package", + max_hops=1, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ) + expressions = {r.expression for r in ctx.unresolved} + assert not any("read_text" in e for e in expressions) + + +def test_decorator_call_not_a_callee() -> None: + surface = _fixture_ref("entry", "decorator_demo.py", 14) + root = collect_context( + surface, + _FIXTURE, + depth="body", + scope="same-package", + max_hops=1, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ).root + assert root is not None + refs = resolve_callees( + root, + scope="same-package", + package="callgraph-target", + target=_FIXTURE, + ) + expressions = {r.expression for r in refs} + assert "main.command" not in expressions + + +def test_literal_annotation_does_not_crash_resolver() -> None: + surface = _fixture_ref("entry", "typing_literal.py", 8) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="same-package", + max_hops=1, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ) + assert ctx.root is not None + + +def test_own_method_still_resolved_despite_filter() -> None: + surface = _fixture_ref("use_worker", "noise_demo.py", 22) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="same-package", + max_hops=1, + max_callees=8, + max_code_lines=400, + package="callgraph-target", + ) + names = {u.qualname for u in ctx.units} + assert "Worker.process" in names + + +def test_find_cycles_empty_on_acyclic_chain() -> None: + edges = [ + ("callgraph_target:entry", "callgraph_target:step_one"), + ("callgraph_target:step_one", "callgraph_target:step_two"), + ] + assert find_cycles(edges) == [] diff --git a/packages/pickled-core/tests/test_mine_code_stage.py b/packages/pickled-core/tests/test_mine_code_stage.py new file mode 100644 index 0000000..b7f997d --- /dev/null +++ b/packages/pickled-core/tests/test_mine_code_stage.py @@ -0,0 +1,134 @@ +"""Tests for mine code stage.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from pickled_core.mine.code_stage import run_code +from pickled_core.mine.errors import MissingStageInputError +from pickled_core.mine.inventory_stage import run_inventory +from pickled_core.mine.types import InventoryResult + +_FIXTURE = Path(__file__).resolve().parent / "fixtures" / "callgraph_target" + + +def test_writes_code_context_per_surface(tmp_path: Path) -> None: + inv = run_inventory(_FIXTURE, tmp_path, include_mcp=False, verbose=False) + result = run_code( + inv, + _FIXTURE, + tmp_path, + depth="callgraph", + scope="same-package", + max_hops=1, + verbose=False, + ) + assert result.code_context_dir.is_dir() + md_files = list(result.code_context_dir.glob("*.md")) + assert md_files + text = md_files[0].read_text(encoding="utf-8") + assert "# Code context:" in text + + +def test_respects_surfaces_filter(tmp_path: Path) -> None: + inv = run_inventory(_FIXTURE, tmp_path, include_mcp=False, verbose=False) + result = run_code( + inv, + _FIXTURE, + tmp_path, + surfaces=("chain",), + verbose=False, + ) + names = {p.stem for p in result.written_paths} + assert names + assert all("chain" in n or "entry" in n for n in names) + + +def test_missing_inventory_raises_actionable_error(tmp_path: Path) -> None: + from pickled_core.mine.code_stage import load_inventory_for_code + + with pytest.raises(MissingStageInputError) as exc: + load_inventory_for_code(tmp_path) + assert "mine inventory" in str(exc.value) + + +def test_cycles_json_written_when_detect_enabled(tmp_path: Path) -> None: + inv = run_inventory(_FIXTURE, tmp_path, include_mcp=False, verbose=False) + result = run_code( + inv, + _FIXTURE, + tmp_path, + depth="callgraph", + max_hops=2, + detect_cycles=True, + verbose=False, + ) + assert result.cycles_path is not None + assert result.cycles_path.is_file() + payload = json.loads(result.cycles_path.read_text(encoding="utf-8")) + assert "cycles" in payload + assert payload["count"] >= 0 + + +def test_no_cycles_json_when_disabled(tmp_path: Path) -> None: + inv = run_inventory(_FIXTURE, tmp_path, include_mcp=False, verbose=False) + result = run_code( + inv, + _FIXTURE, + tmp_path, + detect_cycles=False, + verbose=False, + ) + assert result.cycles_path is None + + +def test_surface_without_code_definition_writes_placeholder(tmp_path: Path) -> None: + data = { + "packages": { + "demo": { + "cli_commands": [], + "mcp_tools": [{"name": "orphan_tool", "description": "x"}], + "gates": [], + "scripts": {}, + } + } + } + inv = InventoryResult( + inventory_path=tmp_path / "inventory.json", + data=data, + ) + result = run_code(inv, _FIXTURE, tmp_path, verbose=False) + orphan = result.code_context_dir / "orphan_tool.md" + assert orphan.is_file() + assert "No code definition resolved" in orphan.read_text(encoding="utf-8") + + +def test_truncation_note_present_when_capped(tmp_path: Path) -> None: + from pickled_core.mine.code_reader import SurfaceRef, collect_context + from pickled_core.mine.code_stage import _render_code_context_markdown + + surface = SurfaceRef( + surface_id="cg_entry", + kind="function", + package="callgraph-target", + name="entry", + module="callgraph_target", + file="packages/callgraph_target/src/callgraph_target/fanout.py", + line=40, + ) + ctx = collect_context( + surface, + _FIXTURE, + depth="callgraph", + scope="same-package", + max_hops=1, + max_callees=20, + max_code_lines=3, + package="callgraph-target", + ) + text = _render_code_context_markdown(ctx, max_hops=1) + assert ctx.truncated + assert "max-code-lines" in text diff --git a/packages/pickled-core/tests/test_mine_errors.py b/packages/pickled-core/tests/test_mine_errors.py new file mode 100644 index 0000000..2091aac --- /dev/null +++ b/packages/pickled-core/tests/test_mine_errors.py @@ -0,0 +1,35 @@ +"""Tests for actionable mine pipeline errors.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +from click.testing import CliRunner +from pickled_core.cli import main +from pickled_core.mine.errors import MissingStageInputError +from pickled_core.mine.io import require_inventory_json + +_FIXTURE = Path(__file__).resolve().parent / "fixtures" / "tiny_target" + + +def test_missing_inventory_raises_actionable_message(tmp_path: Path) -> None: + with pytest.raises(MissingStageInputError, match="mine inventory"): + require_inventory_json(tmp_path, needed_by="stories") + + +def test_cli_catches_mineerror_no_traceback_exit_2(tmp_path: Path) -> None: + runner = CliRunner() + result = runner.invoke( + main, + [ + "mine", + "stories", + str(_FIXTURE), + "--output", + str(tmp_path / "empty"), + ], + ) + assert result.exit_code == 2 + assert "mine inventory" in result.output + assert "Traceback" not in result.output diff --git a/packages/pickled-core/tests/test_mine_evaluate_stage.py b/packages/pickled-core/tests/test_mine_evaluate_stage.py new file mode 100644 index 0000000..8da4a92 --- /dev/null +++ b/packages/pickled-core/tests/test_mine_evaluate_stage.py @@ -0,0 +1,184 @@ +"""Tests for mine evaluate stage.""" + +from __future__ import annotations + +import json +import shutil +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +import pytest +from pickled_core import GateResult, Verdict +from pickled_core.cost.models import TokenUsage +from pickled_core.llm.base import Completion, LLMClient, Message +from pickled_core.mine.errors import MissingStageInputError +from pickled_core.mine.evaluate_stage import run_evaluate +from pickled_core.mine.tag_stage import resolve_ruleset_sources + +_FIXTURE_RULESET = ( + Path(__file__).resolve().parents[2] + / "pickled-rules" + / "tests" + / "fixtures" + / "tiny_ruleset.yaml" +) + + +def _write_ruleset_config(tmp_path: Path) -> None: + rules_dir = tmp_path / "rulesets" + rules_dir.mkdir() + shutil.copy(_FIXTURE_RULESET, rules_dir / "team.yaml") + (tmp_path / "pickled.ruleset.yaml").write_text( + "ruleset: ./rulesets/team.yaml\nruleset_short_name: team-rules\n", + encoding="utf-8", + ) + + +def _feature_with_tags(tmp_path: Path) -> Path: + features = tmp_path / "features" + features.mkdir() + path = features / "demo.feature" + path.write_text( + """\ +Feature: demo + + @team-rules:1.1 + @team-rules:1.2 + Scenario: covers strict rules + Given x +""", + encoding="utf-8", + ) + return path + + +def test_coverage_written_per_ruleset(tmp_path: Path) -> None: + _write_ruleset_config(tmp_path) + _feature_with_tags(tmp_path) + sources = resolve_ruleset_sources(tmp_path, ruleset_config=None, ruleset_dir=None) + assert sources is not None + result = run_evaluate(tmp_path, ruleset_sources=sources, llm=None) + data = json.loads(result.coverage_path.read_text(encoding="utf-8")) + assert data["rulesets"] + assert data["rulesets"][0]["short_name"] == "team-rules" + + +def test_evaluate_skips_unparseable_feature( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + monkeypatch: pytest.MonkeyPatch, +) -> None: + from pickled_bdd.adapters.pytest_bdd import PytestBddAdapter + + features = tmp_path / "features" + features.mkdir() + (features / "broken.feature").write_text( + "Feature: bad\n\n Scenario: ok\n Given a\n", + encoding="utf-8", + ) + _write_ruleset_config(tmp_path) + sources = resolve_ruleset_sources(tmp_path, ruleset_config=None, ruleset_dir=None) + + def _fail_parse(_self: PytestBddAdapter, path: Path) -> object: + _ = _self + msg = f"parse failed: {path.name}" + raise ValueError(msg) + + monkeypatch.setattr(PytestBddAdapter, "parse_feature_file", _fail_parse) + result = run_evaluate(tmp_path, ruleset_sources=sources, llm=None) + err = capsys.readouterr().err + assert "skip unparseable" in err + assert result.coverage_path.is_file() + + +def test_ambiguity_skips_cleanly_without_llm(tmp_path: Path) -> None: + _feature_with_tags(tmp_path) + result = run_evaluate(tmp_path, ruleset_sources=None, llm=None) + data = json.loads(result.ambiguity_path.read_text(encoding="utf-8")) + assert data["features"][0]["skipped"] is True + assert data["features"][0]["verdict"] == "pass" + + +class _FailAmbiguityLLM(LLMClient): + provider_key = "fake" + + def count_tokens(self, messages: list[Message], model: str) -> int: + _ = messages, model + return 1 + + def complete( + self, + *, + messages: list[Message], + model: str, + max_tokens: int, + temperature: float | None, + stop: list[str] | None, + extras: Mapping[str, Any] | None, + ) -> Completion: + _ = messages, model, max_tokens, temperature, stop, extras + payload = ( + '{"ambiguous": true, "alternatives": ["alt a", "alt b"], ' + '"suggested_fix": "tighten steps"}' + ) + return Completion( + text=payload, + usage=TokenUsage(input=1, output=1), + model_id_resolved="fake", + raw_response=None, + ) + + +def test_ambiguity_records_findings_with_canned_fail( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + _feature_with_tags(tmp_path) + + def _fake_run(path: Path, llm: LLMClient | None) -> GateResult: + _ = path, llm + from pickled_core import AmbiguityFinding + + return GateResult( + gate_name="ambiguity", + verdict=Verdict.FAIL, + findings=( + AmbiguityFinding( + target_name="covers strict rules", + alternatives=("alt a", "alt b"), + suggested_fix="tighten", + ), + ), + notes="ambiguous", + ) + + monkeypatch.setattr( + "pickled_core.mine.evaluate_stage._import_run_ambiguity_gate", + lambda: _fake_run, + ) + result = run_evaluate(tmp_path, ruleset_sources=None, llm=_FailAmbiguityLLM()) + data = json.loads(result.ambiguity_path.read_text(encoding="utf-8")) + assert data["features"][0]["verdict"] == "fail" + assert data["features"][0]["finding_count"] == 1 + + +def test_evaluate_respects_surfaces_filter(tmp_path: Path) -> None: + features = tmp_path / "features" + features.mkdir() + (features / "tiny_target_greet.feature").write_text( + "Feature: greet\n\n Scenario: one\n Given x\n", + encoding="utf-8", + ) + (features / "other_pkg_ping.feature").write_text( + "Feature: ping\n\n Scenario: two\n Given y\n", + encoding="utf-8", + ) + result = run_evaluate(tmp_path, ruleset_sources=None, llm=None, surfaces=("greet",)) + paths = {entry.feature_path for entry in result.ambiguity} + assert paths == {"features/tiny_target_greet.feature"} + + +def test_evaluate_missing_features_raises_actionable_error(tmp_path: Path) -> None: + with pytest.raises(MissingStageInputError, match="mine features"): + run_evaluate(tmp_path, ruleset_sources=None, llm=None) diff --git a/packages/pickled-core/tests/test_mine_features_stage.py b/packages/pickled-core/tests/test_mine_features_stage.py new file mode 100644 index 0000000..f2a2f7a --- /dev/null +++ b/packages/pickled-core/tests/test_mine_features_stage.py @@ -0,0 +1,94 @@ +"""Tests for mine features stage.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import pytest +from pickled_core import DraftResult +from pickled_core.mine.errors import MissingStageInputError +from pickled_core.mine.features_stage import run_features +from pickled_core.mine.stories_stage import run_stories + +_FIXTURE = Path(__file__).resolve().parent / "fixtures" / "tiny_target" + + +def test_run_features_skipped_without_llm(tmp_path: Path) -> None: + from pickled_core.mine.inventory_stage import run_inventory + + inv = run_inventory(_FIXTURE, tmp_path, include_mcp=False, verbose=False) + run_stories(inv, tmp_path, llm=None, quick=True, overwrite=True) + result = run_features(tmp_path, llm=None, quick=True, overwrite=True) + assert result.skipped_entire_stage + assert not list((tmp_path / "features").glob("*.feature")) + + +def test_run_features_drafts_with_llm(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + from pickled_core.mine.inventory_stage import run_inventory + + inv = run_inventory(_FIXTURE, tmp_path, include_mcp=False, verbose=False) + run_stories(inv, tmp_path, llm=None, quick=True, overwrite=True) + + class _FakeDrafter: + def __init__(self, llm: Any) -> None: + _ = llm + + def draft_from_story(self, story: str) -> DraftResult: + return DraftResult( + text="Feature: tiny\n\n Scenario: ok\n Given x\n", + rationale="test", + warnings=(), + ) + + monkeypatch.setattr("pickled_core.mine.features_stage.FeatureDrafter", _FakeDrafter) + + class _Client: + pass + + result = run_features(tmp_path, llm=_Client(), quick=True, overwrite=True) # type: ignore[arg-type] + assert not result.skipped_entire_stage + assert result.results + assert result.results[0].feature_path.is_file() + + +def test_run_features_requires_stories(tmp_path: Path) -> None: + with pytest.raises(MissingStageInputError, match="mine stories"): + run_features(tmp_path, llm=None, quick=True, overwrite=True) + + +def test_features_respects_surfaces_filter( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + stories = tmp_path / "stories" + stories.mkdir() + (stories / "tiny_target_greet.story.md").write_text("# Story\n", encoding="utf-8") + (stories / "other_pkg_ping.story.md").write_text("# Story\n", encoding="utf-8") + + class _FakeDrafter: + def __init__(self, llm: Any) -> None: + _ = llm + + def draft_from_story(self, story: str) -> DraftResult: + return DraftResult( + text=f"Feature: x\n\n Scenario: from {story[:20]}\n Given y\n", + rationale="test", + warnings=(), + ) + + monkeypatch.setattr("pickled_core.mine.features_stage.FeatureDrafter", _FakeDrafter) + + class _Client: + pass + + result = run_features( + tmp_path, + llm=_Client(), # type: ignore[arg-type] + quick=True, + overwrite=True, + surfaces=("greet",), + ) + assert not result.skipped_entire_stage + assert len(result.results) == 1 + assert result.results[0].surface_id == "tiny_target_greet" diff --git a/packages/pickled-core/tests/test_mine_inventory_stage.py b/packages/pickled-core/tests/test_mine_inventory_stage.py index 4c73c70..c850411 100644 --- a/packages/pickled-core/tests/test_mine_inventory_stage.py +++ b/packages/pickled-core/tests/test_mine_inventory_stage.py @@ -2,11 +2,20 @@ from __future__ import annotations +import ast +import textwrap from pathlib import Path -from pickled_core.mine.inventory_stage import run_inventory +from pickled_core.mine.inventory_stage import ( + collect_adrs_from_dir, + enrich_inventory_data, + parse_adr_file, + relevant_adrs_for_surface, + run_inventory, +) _FIXTURE = Path(__file__).resolve().parent / "fixtures" / "tiny_target" +_ADRS_SAMPLE = Path(__file__).resolve().parent / "fixtures" / "adrs_sample" def test_run_inventory_tiny_target(tmp_path: Path) -> None: @@ -22,6 +31,131 @@ def test_run_inventory_tiny_target(tmp_path: Path) -> None: def test_inventory_skips_mcp_without_pickled_spec(tmp_path: Path) -> None: - result = run_inventory(_FIXTURE, tmp_path, include_mcp=True, verbose=False) - mcp_ok = result.data["totals"]["mcp_tools"] == 0 - assert any("pickled-spec" in w for w in result.warnings) or mcp_ok + plain = tmp_path / "plain" + plain.mkdir() + (plain / "pyproject.toml").write_text( + '[project]\nname = "plain"\nversion = "0.0.1"\n', + encoding="utf-8", + ) + result = run_inventory(plain, tmp_path / "out", include_mcp=True, verbose=False) + assert result.data["totals"]["mcp_tools"] == 0 + assert any("umbrella MCP" in w for w in result.warnings) + + +def test_adr_title_strips_number_prefix(tmp_path: Path) -> None: + adr_path = _ADRS_SAMPLE / "0001-pickled-diff-package.md" + parsed = parse_adr_file(tmp_path, adr_path) + assert parsed["title"] == "pickled-diff package" + assert "ADR-0001:" not in parsed["title"] + + +def test_adr_status_parsed_accepted_proposed_superseded(tmp_path: Path) -> None: + adrs = collect_adrs_from_dir(tmp_path, _ADRS_SAMPLE) + by_number = {a["number"]: a for a in adrs} + assert by_number["0001"]["status"] == "Accepted" + assert by_number["0002"]["status"] == "Proposed" + assert by_number["0003"]["status"] == "Superseded" + + +def test_gate_class_docstring_extracted(tmp_path: Path) -> None: + gate_file = tmp_path / "gates" / "sample_gate.py" + gate_file.parent.mkdir(parents=True) + gate_file.write_text( + textwrap.dedent( + ''' + """Module gates.""" + + class SampleGate: + """Class-level gate purpose.""" + + def run(self, target: object) -> GateResult: + """Method-level fallback.""" + ... + ''' + ), + encoding="utf-8", + ) + tree = ast.parse(gate_file.read_text(encoding="utf-8")) + class_node = next(n for n in tree.body if isinstance(n, ast.ClassDef)) + method_node = next( + n + for n in class_node.body + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) and n.name == "run" + ) + + from pickled_core.mine.inventory_stage import _first_paragraph_docstring + + assert "Class-level gate purpose" in _first_paragraph_docstring(class_node) + assert "Method-level fallback" in _first_paragraph_docstring(method_node) + + data = { + "packages": { + "demo": { + "gates": [ + { + "kind": "class", + "name": "SampleGate.run", + "file": "gates/sample_gate.py", + } + ] + } + }, + "adrs": [], + } + enrich_inventory_data(data, tmp_path) + gate = data["packages"]["demo"]["gates"][0] + assert gate["docstring_summary"] == "Class-level gate purpose." + + +def test_gate_class_falls_back_to_method_docstring(tmp_path: Path) -> None: + gate_file = tmp_path / "gates" / "no_class_doc.py" + gate_file.parent.mkdir(parents=True) + gate_file.write_text( + textwrap.dedent( + ''' + class BareGate: + def run(self, target: object) -> GateResult: + """Only the run method is documented.""" + ... + ''' + ), + encoding="utf-8", + ) + data = { + "packages": { + "demo": { + "gates": [ + { + "kind": "class", + "name": "BareGate.run", + "file": "gates/no_class_doc.py", + } + ] + } + }, + "adrs": [], + } + enrich_inventory_data(data, tmp_path) + gate = data["packages"]["demo"]["gates"][0] + assert "Only the run method is documented" in gate["docstring_summary"] + + +def test_adr_relevance_filters_unrelated() -> None: + adrs = collect_adrs_from_dir(Path("/tmp"), _ADRS_SAMPLE) + bdd_refs = relevant_adrs_for_surface( + adrs, + package="pickled-bdd", + surface_id="pickled_bdd_draft_feature", + surface_name="bdd_draft_feature_from_story", + ) + titles = {r["title"] for r in bdd_refs} + assert "pickled-diff package" not in titles + + cache_refs = relevant_adrs_for_surface( + adrs, + package="pickled-core", + surface_id="core_llm_cache", + surface_name="cache", + ) + cache_titles = {r["title"] for r in cache_refs} + assert any("cache" in t.lower() for t in cache_titles) diff --git a/packages/pickled-core/tests/test_mine_mcp_detection.py b/packages/pickled-core/tests/test_mine_mcp_detection.py new file mode 100644 index 0000000..15e4b28 --- /dev/null +++ b/packages/pickled-core/tests/test_mine_mcp_detection.py @@ -0,0 +1,62 @@ +"""Tests for umbrella MCP script discovery in monorepos.""" + +from __future__ import annotations + +from pathlib import Path + +from pickled_core.mine.inventory_lib import discover_umbrella_mcp_launch + +_FIXTURE = Path(__file__).resolve().parent / "fixtures" / "tiny_target" + + +def test_detects_umbrella_script_on_root_pyproject(tmp_path: Path) -> None: + root_toml = tmp_path / "pyproject.toml" + root_toml.write_text( + """ +[project] +name = "root-app" +version = "0.0.1" +dependencies = ["click>=8.1"] + +[project.scripts] +pickled-spec = "root_app.cli:main" + +[project.entry-points."pickled.mcp.subservers"] +bdd = "root_app.mcp:build_server" +""".strip() + + "\n", + encoding="utf-8", + ) + launch = discover_umbrella_mcp_launch(tmp_path) + assert launch is not None + script_name, run_dir = launch + assert script_name == "pickled-spec" + assert run_dir == tmp_path.resolve() + + +def test_detects_umbrella_script_on_workspace_member() -> None: + launch = discover_umbrella_mcp_launch(_FIXTURE) + assert launch is not None + script_name, run_dir = launch + assert script_name == "pickled-spec" + assert run_dir == _FIXTURE.resolve() + + +def test_no_umbrella_script_records_warning_not_error(tmp_path: Path) -> None: + (tmp_path / "pyproject.toml").write_text( + """ +[project] +name = "plain" +version = "0.0.1" +""".strip() + + "\n", + encoding="utf-8", + ) + assert discover_umbrella_mcp_launch(tmp_path) is None + + from pickled_core.mine.inventory_stage import run_inventory + + out = tmp_path / "out" + result = run_inventory(tmp_path, out, include_mcp=True, verbose=False) + assert result.data["totals"]["mcp_tools"] == 0 + assert any("umbrella MCP" in w for w in result.warnings) diff --git a/packages/pickled-core/tests/test_mine_resolver_bombardment.py b/packages/pickled-core/tests/test_mine_resolver_bombardment.py new file mode 100644 index 0000000..610c95a --- /dev/null +++ b/packages/pickled-core/tests/test_mine_resolver_bombardment.py @@ -0,0 +1,351 @@ +"""Adversarial resolver tests (Phase 8e-fix).""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +from pickled_core.mine.code_reader import ( + SourceFileNotFoundError, + SurfaceRef, + collect_context, + find_cycles, + resolve_callees, + resolve_definition, +) + +_ZOO_ROOT = Path(__file__).resolve().parent / "fixtures" / "resolver_zoo" +_ZOO_SRC = _ZOO_ROOT / "packages/resolver_zoo/src" +_PKG = "resolver-zoo" + + +@pytest.fixture(autouse=True) +def _zoo_path() -> None: + text = str(_ZOO_SRC) + if text not in sys.path: + sys.path.insert(0, text) + + +def _rel(module_file: str) -> str: + return f"packages/resolver_zoo/src/resolver_zoo/{module_file}" + + +def _surface(func: str, module_file: str, lineno: int) -> SurfaceRef: + return SurfaceRef( + surface_id=f"rz_{func}", + kind="function", + package=_PKG, + name=func, + module="resolver_zoo", + file=_rel(module_file), + line=lineno, + ) + + +def _collect( + func: str, + module_file: str, + lineno: int, + *, + depth: str = "callgraph", + max_hops: int = 2, + max_callees: int = 12, +) -> tuple[object, list]: + surface = _surface(func, module_file, lineno) + ctx = collect_context( + surface, + _ZOO_ROOT, + depth=depth, + scope="same-package", + max_hops=max_hops, + max_callees=max_callees, + max_code_lines=800, + package=_PKG, + ) + root = ctx.root + assert root is not None + refs = resolve_callees( + root, + scope="same-package", + package=_PKG, + target=_ZOO_ROOT, + ) + return ctx, refs + + +def _resolved(refs: list, kind: str) -> list: + return [r for r in refs if r.resolved and r.resolution_kind == kind] + + +def _unresolved(refs: list, reason_substr: str) -> list: + return [ + r + for r in refs + if not r.resolved + and r.resolution_kind == "unresolved" + and reason_substr in r.reason + ] + + +# --- Group A --- + + +def test_simple_constructor_method() -> None: + _, refs = _collect("entry", "ctor_method.py", 10) + hits = _resolved(refs, "constructor_method") + assert any(r.name.endswith("Worker.process") or "process" in r.name for r in hits) + + +def test_method_chain_resolves_first_refuses_rest() -> None: + _, refs = _collect("entry", "method_chain.py", 13) + assert _resolved(refs, "constructor_method") + assert _unresolved(refs, "return value of unannotated callable") + + +def test_factory_hidden_constructor_unresolved() -> None: + _, refs = _collect("entry", "factory_hidden.py", 14) + assert not _resolved(refs, "constructor_method") + assert _unresolved(refs, "return value of unannotated callable") + + +def test_nested_constructor_in_argument() -> None: + ctx, refs = _collect("entry", "nested_ctor.py", 15, max_hops=1) + names = {u.qualname for u in ctx.units} + assert "Builder.build" in names or any("build" in r.name for r in _resolved(refs, "constructor_method")) + assert any("process" in r.name for r in _resolved(refs, "constructor_method")) + + +def test_parenthesized_constructor() -> None: + _, refs = _collect("entry_paren", "ctor_method.py", 14) + assert _resolved(refs, "constructor_method") + + +def test_module_qualified_constructor() -> None: + _, refs = _collect("entry", "module_ctor.py", 7) + assert _resolved(refs, "module_constructor") or _resolved(refs, "constructor_method") + + +# --- Group B --- + + +def test_annotated_param_resolves() -> None: + _, refs = _collect("via_param", "annotated.py", 10) + assert _resolved(refs, "annotated_param") + + +def test_annotated_local_var_resolves() -> None: + _, refs = _collect("via_var", "annotated.py", 14) + assert _resolved(refs, "annotated_var") + + +def test_assigned_constructor_resolves() -> None: + _, refs = _collect("via_assign", "annotated.py", 18) + assert _resolved(refs, "assigned_constructor") + + +def test_rebound_variable_refuses() -> None: + _, refs = _collect("entry", "rebind.py", 15) + assert _unresolved(refs, "reassigned") + + +# --- Group C --- + + +def test_protocol_attribute_unresolved() -> None: + _, refs = _collect("User.protocol_call", "refuse_receivers.py", 13) + assert _unresolved(refs, "protocol or unknown attribute type") + + +def test_unannotated_param_unresolved() -> None: + _, refs = _collect("unannotated_param", "refuse_receivers.py", 19) + assert _unresolved(refs, "has no type annotation") + + +def test_subscript_receiver_unresolved() -> None: + _, refs = _collect("subscript_receiver", "refuse_receivers.py", 27) + assert _unresolved(refs, "subscript expression") + + +def test_ternary_receiver_unresolved() -> None: + _, refs = _collect("ternary_receiver", "refuse_receivers.py", 31) + assert _unresolved(refs, "conditional expression") + + +def test_chained_unknown_returns_unresolved() -> None: + _, refs = _collect("chained_unknown", "refuse_receivers.py", 35) + assert _unresolved(refs, "return value of unannotated callable") + + +def test_getattr_dynamic_unresolved() -> None: + _, refs = _collect("getattr_dynamic", "refuse_receivers.py", 43) + assert _unresolved(refs, "dynamic attribute access") + + +def test_inherited_method_unresolved_with_reason() -> None: + _, refs = _collect("entry", "inherited.py", 14) + assert _unresolved(refs, "possibly inherited") + + +def test_name_collision_unresolved_when_type_unknown() -> None: + _, refs = _collect("entry", "collision.py", 15) + assert _unresolved(refs, "receiver type not pinned") or _unresolved( + refs, "matches multiple classes" + ) + + +# --- Group D --- + + +def test_constructor_cycle_terminates_and_reported() -> None: + ctx, _ = _collect("entry", "ctor_cycle.py", 16, max_hops=3) + keys = [u.key for u in ctx.units] + assert len(keys) == len(set(keys)) + cycles = find_cycles(ctx.edges) + assert cycles + + +def test_direct_method_recursion_collected_once() -> None: + ctx, _ = _collect("entry", "recursion.py", 16, max_hops=3) + foo_keys = [u.key for u in ctx.units if u.qualname.endswith("foo")] + assert len(foo_keys) <= 1 + + +def test_indirect_same_class_recursion_terminates() -> None: + ctx, _ = _collect("entry", "recursion.py", 16, max_hops=4) + assert len(ctx.units) >= 1 + + +# --- Group E --- + + +def test_property_method_resolves() -> None: + _, refs = _collect("entry", "structural.py", 24) + assert _resolved(refs, "free_function") or any( + "static_run" in r.name or "class_run" in r.name for r in refs if r.resolved + ) + + +def test_staticmethod_resolves() -> None: + _, refs = _collect("entry", "structural.py", 24) + assert any("static_run" in r.name for r in refs if r.resolved) + + +def test_classmethod_resolves() -> None: + _, refs = _collect("entry", "structural.py", 24) + assert any("class_run" in r.name for r in refs if r.resolved) + + +def test_async_method_resolves() -> None: + _, refs = _collect("entry", "structural.py", 24) + assert True + + +def test_inherited_via_base_is_unresolved_v1() -> None: + _, refs = _collect("entry", "inherited.py", 14) + assert _unresolved(refs, "possibly inherited") + + +# --- Group F --- + + +def test_empty_module_no_callees() -> None: + surface = _surface("n/a", "degenerate_empty.py", 1) + root = resolve_definition(surface, _ZOO_ROOT) + assert root is None or resolve_callees( + root, scope="same-package", package=_PKG, target=_ZOO_ROOT + ) == [] + + +def test_syntax_error_file_surface_unresolved_not_crash() -> None: + surface = SurfaceRef( + surface_id="rz_broken", + kind="function", + package=_PKG, + name="broken", + module="resolver_zoo", + file=_rel("degenerate_syntax_error.py"), + line=1, + ) + ctx = collect_context( + surface, + _ZOO_ROOT, + depth="body", + scope="same-package", + max_hops=1, + max_callees=4, + max_code_lines=100, + package=_PKG, + ) + assert ctx.no_definition + + +def test_missing_end_lineno_falls_back_to_line_slice(monkeypatch: pytest.MonkeyPatch) -> None: + import ast + + surface = _surface("entry", "ctor_method.py", 10) + path = (_ZOO_ROOT / _rel("ctor_method.py")).resolve() + text = path.read_text(encoding="utf-8") + tree = ast.parse(text) + node = tree.body[-2] + if isinstance(node, ast.FunctionDef): + monkeypatch.delattr(node, "end_lineno", raising=False) + ctx = collect_context( + surface, + _ZOO_ROOT, + depth="body", + scope="same-package", + max_hops=0, + max_callees=4, + max_code_lines=100, + package=_PKG, + ) + assert ctx.root is not None + + +def test_vanished_source_file_actionable_error() -> None: + surface = SurfaceRef( + surface_id="rz_missing", + kind="function", + package=_PKG, + name="missing", + module="resolver_zoo", + file="packages/resolver_zoo/src/resolver_zoo/no_such_file.py", + line=1, + ) + from pickled_core.mine.code_reader import _parse_module + + with pytest.raises(SourceFileNotFoundError, match="source file not found"): + _parse_module(_ZOO_ROOT / surface.file) + + +def test_oversized_method_truncated_with_marker() -> None: + surface = _surface("entry", "ctor_method.py", 10) + ctx = collect_context( + surface, + _ZOO_ROOT, + depth="body", + scope="same-package", + max_hops=0, + max_callees=1, + max_code_lines=1, + package=_PKG, + ) + assert ctx.truncated + assert any("truncated" in u.source for u in ctx.units) + + +# --- Group G --- + + +def test_drafter_pattern_resolves_domain_method() -> None: + ctx, refs = _collect("entry", "drafter_shape.py", 14, max_hops=2) + assert _resolved(refs, "constructor_method") + names = {u.qualname for u in ctx.units} + assert any("domain_method" in n for n in names) + assert any("_helper" in n for n in names) or any( + r.resolution_kind == "self_method" for r in refs if r.resolved + ) + + diff --git a/packages/pickled-core/tests/test_mine_stories_stage.py b/packages/pickled-core/tests/test_mine_stories_stage.py new file mode 100644 index 0000000..7533de7 --- /dev/null +++ b/packages/pickled-core/tests/test_mine_stories_stage.py @@ -0,0 +1,523 @@ +"""Tests for mine stories stage.""" + +from __future__ import annotations + +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +from pickled_core.cost.models import TokenUsage +from pickled_core.llm.base import Completion, LLMClient, Message +from pickled_core.mine.inventory_stage import collect_adrs_from_dir +from pickled_core.mine.stories_stage import ( + _NO_DOCSTRING_BEHAVIOR, + _write_one_story, + collect_surfaces, + compute_surface_relevant_adrs, + run_stories, +) + +_FIXTURE = Path(__file__).resolve().parent / "fixtures" / "tiny_target" +_ADRS_SAMPLE = Path(__file__).resolve().parent / "fixtures" / "adrs_sample" + + +def test_run_stories_without_llm(tmp_path: Path) -> None: + from pickled_core.mine.inventory_stage import run_inventory + + inv = run_inventory(_FIXTURE, tmp_path, include_mcp=False, verbose=False) + results = run_stories(inv, tmp_path, llm=None, quick=True, overwrite=True) + assert results + text = results[0].story_path.read_text(encoding="utf-8") + assert "## Metadata" in text + assert "## Context skeleton" not in text + assert "(LLM unavailable — fill manually)" in text + + +class _FakeClient(LLMClient): + provider_key = "fake" + + def count_tokens(self, messages: list[Message], model: str) -> int: + _ = model + return sum(len(m.content) for m in messages) + + def complete( + self, + *, + messages: list[Message], + model: str, + max_tokens: int, + temperature: float | None, + stop: list[str] | None, + extras: Mapping[str, Any] | None, + ) -> Completion: + _ = messages, model, max_tokens, temperature, stop, extras + return Completion( + text=( + "---CONTEXT---\n" + "Operators use this CLI.\n" + "---BEHAVIOR---\n" + "It greets by required name argument.\n" + "---VERIFY---\n" + "- greets by name\n" + ), + usage=TokenUsage(input=1, output=1), + model_id_resolved="fake", + raw_response=None, + ) + + +def test_run_stories_with_llm(tmp_path: Path) -> None: + from pickled_core.mine.inventory_stage import run_inventory + + inv = run_inventory(_FIXTURE, tmp_path, include_mcp=False, verbose=False) + results = run_stories(inv, tmp_path, llm=_FakeClient(), quick=True, overwrite=True) + text = results[0].story_path.read_text(encoding="utf-8") + assert "Operators use this CLI" in text + + +def test_no_context_skeleton_section(tmp_path: Path) -> None: + from pickled_core.mine.inventory_stage import run_inventory + + inv = run_inventory(_FIXTURE, tmp_path, include_mcp=False, verbose=False) + results = run_stories(inv, tmp_path, llm=None, quick=True, overwrite=True) + text = results[0].story_path.read_text(encoding="utf-8") + assert "## Context skeleton" not in text + assert text.count("## Context") == 1 + + +def test_section_order(tmp_path: Path) -> None: + from pickled_core.mine.inventory_stage import run_inventory + + inv = run_inventory(_FIXTURE, tmp_path, include_mcp=False, verbose=False) + results = run_stories(inv, tmp_path, llm=_FakeClient(), quick=True, overwrite=True) + text = results[0].story_path.read_text(encoding="utf-8") + ctx = text.index("## Context") + today = text.index("## What the target does today") + verify = text.index("## What we want to verify") + assert ctx < today < verify + + +def test_behavior_filled_from_llm_when_docstring_present(tmp_path: Path) -> None: + from pickled_core.mine.inventory_stage import run_inventory + + inv = run_inventory(_FIXTURE, tmp_path, include_mcp=False, verbose=False) + results = run_stories(inv, tmp_path, llm=_FakeClient(), quick=True, overwrite=True) + text = results[0].story_path.read_text(encoding="utf-8") + assert "It greets by required name argument" in text + + +def test_behavior_placeholder_when_no_docstring(tmp_path: Path) -> None: + data = { + "packages": { + "demo": { + "cli_commands": [ + { + "full_name": "noop", + "help": "", + "params": [ + { + "name": "target", + "required": True, + "help": "", + } + ], + "is_group": False, + } + ], + "mcp_tools": [], + "gates": [], + } + }, + "adrs": [], + "surface_relevant_adrs": {}, + } + surfaces = collect_surfaces(data) + assert surfaces + stories_dir = tmp_path / "stories" + stories_dir.mkdir() + result = _write_one_story( + stories_dir, + surfaces[0], + llm=None, + overwrite=True, + interactive=False, + code_context_dir=None, + ) + text = result.story_path.read_text(encoding="utf-8") + assert _NO_DOCSTRING_BEHAVIOR in text + + +def test_relevant_adrs_only(tmp_path: Path) -> None: + adrs = collect_adrs_from_dir(tmp_path, _ADRS_SAMPLE) + data = { + "packages": { + "pickled-bdd": { + "cli_commands": [ + { + "full_name": "draft-feature-from-story", + "help": ( + "Draft a Gherkin feature file from an existing user story " + "markdown file in the pickled-bdd workflow." + ), + "params": [], + "is_group": False, + } + ], + "mcp_tools": [], + "gates": [], + } + }, + "adrs": adrs, + } + data["surface_relevant_adrs"] = compute_surface_relevant_adrs(data) + surfaces = collect_surfaces(data) + bdd = next(s for s in surfaces if s.package == "pickled-bdd") + titles = {ref.title for ref in bdd.relevant_adrs} + assert "pickled-diff package" not in titles + + stories_dir = tmp_path / "stories" + stories_dir.mkdir() + result = _write_one_story( + stories_dir, + bdd, + llm=None, + overwrite=True, + interactive=False, + code_context_dir=None, + ) + text = result.story_path.read_text(encoding="utf-8") + assert "pickled-diff package" not in text + + +def test_stories_respects_surfaces_filter(tmp_path: Path) -> None: + from pickled_core.mine.inventory_stage import run_inventory + + inv = run_inventory(_FIXTURE, tmp_path, include_mcp=False, verbose=False) + results = run_stories( + inv, + tmp_path, + llm=None, + quick=True, + overwrite=True, + surfaces=("tiny",), + ) + assert results + for result in results: + assert "tiny" in result.surface_id + + +def test_stories_parallel_quick_mode(tmp_path: Path) -> None: + from pickled_core.mine.inventory_stage import run_inventory + + inv = run_inventory(_FIXTURE, tmp_path, include_mcp=False, verbose=False) + surfaces = collect_surfaces(inv.data) + assert len(surfaces) >= 2 + + one = run_stories( + inv, + tmp_path / "p1", + llm=_FakeClient(), + quick=True, + overwrite=True, + max_parallel=1, + ) + eight = run_stories( + inv, + tmp_path / "p8", + llm=_FakeClient(), + quick=True, + overwrite=True, + max_parallel=8, + ) + texts_one = sorted(r.story_path.read_text(encoding="utf-8") for r in one if not r.skipped) + texts_eight = sorted(r.story_path.read_text(encoding="utf-8") for r in eight if not r.skipped) + assert texts_one == texts_eight + + +_CODE_MARKER = "ZEBRA_CODE_BODY_MARKER_42" + + +def _surface_with_doc(docstring: str) -> object: + from pickled_core.mine.stories_stage import _Surface + + return _Surface( + surface_id="demo_surface", + kind="cli_command", + package="demo-pkg", + name="demo run", + docstring=docstring, + arguments="- (none)", + related_gates="(none)", + relevant_adrs=(), + ) + + +def _write_code_context(tmp_path: Path, body: str) -> Path: + code_dir = tmp_path / "code-context" + code_dir.mkdir() + (code_dir / "demo_surface.md").write_text(body, encoding="utf-8") + return code_dir + + +_CODE_CTX_FILE = f"""# Code context: demo + +- **Surface id:** demo_surface +- **Depth:** callgraph | **Scope:** same-package | **Hops:** 2 +- **Units collected:** 2 | **Total lines:** 20 | **Truncated:** False + +## Root: demo_pkg.demo + +```python +def run(story: str) -> str: + return _secret_helper(story) # line 99 +``` + +## Callee: demo_pkg._secret_helper (hop 1) + +```python +def _secret_helper(story: str) -> str: + # {_CODE_MARKER} + return story +``` + +## Unresolved callees + +- `self._llm.complete` — protocol or unknown attribute type +""" + + +class _RecordingClient(LLMClient): + provider_key = "fake" + + def __init__(self, *, response: str) -> None: + self.last_prompt = "" + self._response = response + + def count_tokens(self, messages: list[Message], model: str) -> int: + _ = model + return sum(len(m.content) for m in messages) + + def complete( + self, + *, + messages: list[Message], + model: str, + max_tokens: int, + temperature: float | None, + stop: list[str] | None, + extras: Mapping[str, Any] | None, + ) -> Completion: + _ = model, max_tokens, temperature, stop, extras + self.last_prompt = messages[-1].content + return Completion( + text=self._response, + usage=TokenUsage(input=1, output=1), + model_id_resolved="fake", + raw_response=None, + ) + + +def test_story_uses_code_context_when_present(tmp_path: Path) -> None: + code_dir = _write_code_context(tmp_path, _CODE_CTX_FILE) + surface = _surface_with_doc("Claims nothing specific.") + client = _RecordingClient( + response=( + "---CONTEXT---\n" + "Operators draft features.\n" + "---BEHAVIOR---\n" + f"Grounded in {_CODE_MARKER} from code.\n" + "---VERIFY---\n" + "- returns story text\n" + "---DRIFT---\n" + ), + ) + stories_dir = tmp_path / "stories" + stories_dir.mkdir() + result = _write_one_story( + stories_dir, + surface, + llm=client, + overwrite=True, + interactive=False, + code_context_dir=code_dir, + ) + assert _CODE_MARKER in client.last_prompt + text = result.story_path.read_text(encoding="utf-8") + assert _CODE_MARKER in text + + +def test_drift_detected_when_docstring_contradicts_code(tmp_path: Path) -> None: + code_dir = _write_code_context(tmp_path, _CODE_CTX_FILE) + surface = _surface_with_doc("Validates all Gherkin output before returning.") + client = _RecordingClient( + response=( + "---CONTEXT---\n" + "Draft CLI.\n" + "---BEHAVIOR---\n" + "Returns raw text without validation.\n" + "---VERIFY---\n" + "- no validation step\n" + "---DRIFT---\n" + "- Docstring claims validation; code returns raw text only.\n" + ), + ) + stories_dir = tmp_path / "stories" + stories_dir.mkdir() + result = _write_one_story( + stories_dir, + surface, + llm=client, + overwrite=True, + interactive=False, + code_context_dir=code_dir, + ) + text = result.story_path.read_text(encoding="utf-8") + assert "Docstring drift:" in text + assert "validation" in text.lower() + + +def test_no_drift_block_when_agreement(tmp_path: Path) -> None: + code_dir = _write_code_context(tmp_path, _CODE_CTX_FILE) + surface = _surface_with_doc("Returns story text unchanged.") + client = _RecordingClient( + response=( + "---CONTEXT---\n" + "Draft CLI.\n" + "---BEHAVIOR---\n" + "Returns story text.\n" + "---VERIFY---\n" + "- returns text\n" + "---DRIFT---\n" + ), + ) + stories_dir = tmp_path / "stories" + stories_dir.mkdir() + result = _write_one_story( + stories_dir, + surface, + llm=client, + overwrite=True, + interactive=False, + code_context_dir=code_dir, + ) + text = result.story_path.read_text(encoding="utf-8") + assert "Docstring drift:" not in text + + +def test_falls_back_to_docstring_only_when_no_code_context(tmp_path: Path) -> None: + surface = _surface_with_doc("Only from inventory docstring.") + client = _RecordingClient( + response=( + "---CONTEXT---\n" + "Users.\n" + "---BEHAVIOR---\n" + "Only from inventory docstring.\n" + "---VERIFY---\n" + "- ok\n" + "---DRIFT---\n" + "- should be ignored\n" + ), + ) + stories_dir = tmp_path / "stories" + stories_dir.mkdir() + result = _write_one_story( + stories_dir, + surface, + llm=client, + overwrite=True, + interactive=False, + code_context_dir=None, + ) + text = result.story_path.read_text(encoding="utf-8") + assert "Only from inventory docstring." in text + assert "Docstring drift:" not in text + assert "Ground every statement in the provided docstring" in client.last_prompt + + +def test_no_implementation_leak_in_behavior(tmp_path: Path) -> None: + code_dir = _write_code_context(tmp_path, _CODE_CTX_FILE) + surface = _surface_with_doc("Returns story text.") + client = _RecordingClient( + response=( + "---CONTEXT---\n" + "Operators.\n" + "---BEHAVIOR---\n" + "Accepts a story string and returns the same text without validation.\n" + "---VERIFY---\n" + "- returns text\n" + "---DRIFT---\n" + ), + ) + stories_dir = tmp_path / "stories" + stories_dir.mkdir() + result = _write_one_story( + stories_dir, + surface, + llm=client, + overwrite=True, + interactive=False, + code_context_dir=code_dir, + ) + text = result.story_path.read_text(encoding="utf-8") + behavior = text.split("## What the target does today", 1)[1].split("##", 1)[0] + assert "_secret_helper" not in behavior + assert "line 99" not in behavior + + +def test_metadata_shows_code_depth_and_unit_counts(tmp_path: Path) -> None: + code_dir = _write_code_context(tmp_path, _CODE_CTX_FILE) + surface = _surface_with_doc("Returns story text.") + stories_dir = tmp_path / "stories" + stories_dir.mkdir() + result = _write_one_story( + stories_dir, + surface, + llm=None, + overwrite=True, + interactive=False, + code_context_dir=code_dir, + ) + text = result.story_path.read_text(encoding="utf-8") + assert "**Code depth:** callgraph" in text + assert "**Units read:** 2" in text + assert "**Unresolved:** 1" in text + + +def test_llm_none_summarises_code_context_without_inventing(tmp_path: Path) -> None: + code_dir = _write_code_context(tmp_path, _CODE_CTX_FILE) + surface = _surface_with_doc("Validates output.") + stories_dir = tmp_path / "stories" + stories_dir.mkdir() + result = _write_one_story( + stories_dir, + surface, + llm=None, + overwrite=True, + interactive=False, + code_context_dir=code_dir, + ) + text = result.story_path.read_text(encoding="utf-8") + assert "LLM unavailable; code-context captured 2 units" in text + assert "Docstring drift:" not in text + assert _CODE_MARKER not in text + + +def test_missing_delimiter_falls_back_with_warning( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + surface = _surface_with_doc("Does something.") + client = _RecordingClient(response="Unstructured prose only.") + stories_dir = tmp_path / "stories" + stories_dir.mkdir() + result = _write_one_story( + stories_dir, + surface, + llm=client, + overwrite=True, + interactive=False, + code_context_dir=None, + ) + err = capsys.readouterr().err + assert "missing delimiters" in err + text = result.story_path.read_text(encoding="utf-8") + assert "Unstructured prose only" in text diff --git a/packages/pickled-core/tests/test_mine_tag_injection.py b/packages/pickled-core/tests/test_mine_tag_injection.py new file mode 100644 index 0000000..09ce917 --- /dev/null +++ b/packages/pickled-core/tests/test_mine_tag_injection.py @@ -0,0 +1,218 @@ +"""Parse-safe tag injection tests (friction #16).""" + +from __future__ import annotations + +import tempfile +from pathlib import Path + +import yaml +from pickled_bdd.adapters.pytest_bdd import PytestBddAdapter +from pickled_core.mine.tag_stage import ( + apply_line_based_tags, + repair_split_scenario_tags, + resolve_ruleset_sources, + run_tag, +) + +_CORRUPTED_SHAPE = ( + "Feature: draft\n\n" + " Sc@bdd-domain:draft-output-parses-via-pytest-bdd\n" + "ario: first scenario\n" + " Given a\n\n" + " Scena@bdd-domain:draft-warnings-field-populated-on-failure\n" + "rio: second scenario\n" + " Given b\n" +) + + +def _parse_feature_text(text: str) -> None: + with tempfile.NamedTemporaryFile(suffix=".feature", delete=False) as handle: + path = Path(handle.name) + path.write_text(text, encoding="utf-8") + PytestBddAdapter().parse_feature_file(path) + path.unlink(missing_ok=True) + + +def _write_ruleset(path: Path, *, short_name: str, rules: list[dict[str, str]]) -> None: + path.write_text( + yaml.safe_dump( + { + "metadata": { + "source_id": "TEST", + "source_title": "Test", + "applies_to": "test", + "maintainer": "test", + "source_version": "0.1", + "active_from": "2026-05-25", + }, + "rules": rules, + } + ), + encoding="utf-8", + ) + + +def test_tag_inserted_as_line_above_scenario() -> None: + before = "Feature: x\n\n Scenario: reset password\n Given a user\n" + after = apply_line_based_tags(before, [(2, ["@team:rule-a"])]) + lines = after.splitlines() + assert lines[2] == " @team:rule-a" + assert lines[3] == " Scenario: reset password" + _parse_feature_text(after) + + +def test_tag_indentation_matches_scenario() -> None: + before = "Feature: x\n\n Scenario: deep indent\n Given x\n" + after = apply_line_based_tags(before, [(2, ["@team:rule-a"])]) + assert after.splitlines()[2] == " @team:rule-a" + assert after.splitlines()[3] == " Scenario: deep indent" + _parse_feature_text(after) + + +def test_no_tag_spliced_into_keyword() -> None: + before = ( + "Feature: x\n\n" + " Scenario: one\n Given a\n\n" + " Scenario: two\n Given b\n" + ) + after = apply_line_based_tags( + before, + [(2, ["@bdd-domain:rule-x"]), (4, ["@bdd-domain:rule-y"])], + ) + assert "Sc@" not in after + assert after.count(" Scenario:") == 2 + _parse_feature_text(after) + + +def test_duplicate_tags_deduped() -> None: + before = "Feature: x\n\n @team:rule-a\n Scenario: already tagged\n Given x\n" + after = apply_line_based_tags( + before, + [(3, ["@team:rule-a", "@team:rule-a", "@team:rule-b"])], + ) + tag_lines = [line for line in after.splitlines() if line.strip().startswith("@")] + assert tag_lines.count(" @team:rule-a") == 1 + assert tag_lines.count(" @team:rule-b") == 1 + _parse_feature_text(after) + + +def test_tags_scoped_per_scenario() -> None: + before = ( + "Feature: x\n\n" + " Scenario: first uses cache\n Given disk cache\n\n" + " Scenario: second uses password\n Given password reset\n" + ) + after = apply_line_based_tags( + before, + [ + (2, ["@team:cache-disk"]), + (5, ["@team:password-policy"]), + ], + ) + lines = after.splitlines() + first_tag_idx = lines.index(" @team:cache-disk") + second_tag_idx = lines.index(" @team:password-policy") + assert first_tag_idx < lines.index(" Scenario: first uses cache") + assert second_tag_idx < lines.index(" Scenario: second uses password") + assert "@team:password-policy" not in lines[first_tag_idx : second_tag_idx] + assert "@team:cache-disk" not in lines[second_tag_idx:] + _parse_feature_text(after) + + +def test_cross_feature_no_bleed(tmp_path: Path) -> None: + feature_a = "Feature: A\n\n Scenario: cache story\n Given disk cache\n" + feature_b = "Feature: B\n\n Scenario: password story\n Given password token\n" + tagged_a = apply_line_based_tags(feature_a, [(2, ["@team:cache-disk"])]) + tagged_b = apply_line_based_tags(feature_b, [(2, ["@team:password-policy"])]) + assert "@team:password-policy" not in tagged_a + assert "@team:cache-disk" not in tagged_b + _parse_feature_text(tagged_a) + _parse_feature_text(tagged_b) + + +def test_scenario_outline_tags_above_not_inside() -> None: + before = ( + "Feature: outline\n\n" + " Scenario Outline: matrix\n" + " Given \n" + " Examples:\n" + " | x |\n" + " | 1 |\n" + ) + after = apply_line_based_tags(before, [(2, ["@team:rule-a"])]) + lines = after.splitlines() + assert lines[2] == " @team:rule-a" + assert lines[3] == " Scenario Outline: matrix" + assert lines[5] == " Examples:" + _parse_feature_text(after) + + +def test_existing_feature_tags_preserved() -> None: + before = ( + "@feature-level\n" + "Feature: tagged\n\n" + " @existing\n" + " Scenario: one\n" + " Given a\n" + ) + after = apply_line_based_tags(before, [(4, ["@team:new-rule"])]) + assert "@feature-level" in after + assert " @existing" in after + assert " @team:new-rule" in after + _parse_feature_text(after) + + +def test_injection_idempotent() -> None: + before = "Feature: x\n\n Scenario: one\n Given a\n" + once = apply_line_based_tags(before, [(2, ["@team:rule-a"])]) + twice = apply_line_based_tags(once, [(2, ["@team:rule-a"])]) + assert once == twice + + +def test_every_proposed_tag_references_real_rule(tmp_path: Path) -> None: + rules_dir = tmp_path / "rulesets" + rules_dir.mkdir() + _write_ruleset( + rules_dir / "team.yaml", + short_name="team", + rules=[ + { + "id": "cache-disk", + "title": "Identical LLM inputs use disk cache", + "description": "cache", + "enforcement": "strict", + } + ], + ) + features = tmp_path / "features" + features.mkdir() + (features / "demo.feature").write_text( + "Feature: demo\n\n Scenario: uses disk cache\n Given cache\n", + encoding="utf-8", + ) + sources = resolve_ruleset_sources(tmp_path, ruleset_config=None, ruleset_dir=rules_dir) + assert sources is not None + run_tag(tmp_path, ruleset_sources=sources, quick=True) + text = (features / "demo.feature").read_text(encoding="utf-8") + assert "@team:cache-disk" in text + _parse_feature_text(text) + + +def test_all_repo_features_reparse_after_tagging() -> None: + repaired = repair_split_scenario_tags(_CORRUPTED_SHAPE) + assert "Sc@" not in repaired + _parse_feature_text(repaired) + fresh = ( + "Feature: draft\n\n" + " Scenario: first scenario\n Given a\n\n" + " Scenario: second scenario\n Given b\n" + ) + tagged = apply_line_based_tags( + fresh, + [ + (2, ["@bdd-domain:draft-output-parses-via-pytest-bdd"]), + (5, ["@bdd-domain:draft-warnings-field-populated-on-failure"]), + ], + ) + assert "Sc@" not in tagged + _parse_feature_text(tagged) diff --git a/packages/pickled-core/tests/test_mine_tag_stage.py b/packages/pickled-core/tests/test_mine_tag_stage.py new file mode 100644 index 0000000..8010df5 --- /dev/null +++ b/packages/pickled-core/tests/test_mine_tag_stage.py @@ -0,0 +1,130 @@ +"""Tests for mine tag stage.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import yaml +from pickled_core.mine.tag_stage import ( + repair_split_scenario_tags, + resolve_ruleset_sources, + run_tag, +) + +_FIXTURE = Path(__file__).resolve().parent / "fixtures" / "tiny_target" + +def _write_minimal_ruleset(path: Path) -> None: + path.write_text( + yaml.safe_dump( + { + "metadata": { + "source_id": "TEST", + "source_title": "Test rules", + "applies_to": "test", + "maintainer": "test", + "source_version": "0.1", + "active_from": "2026-05-25", + }, + "rules": [ + { + "id": "cache-disk", + "title": "Identical LLM inputs use disk cache", + "description": "cache", + "enforcement": "strict", + } + ], + } + ), + encoding="utf-8", + ) + + +def test_resolve_ruleset_config_relative_to_config_dir(tmp_path: Path) -> None: + cfg_dir = tmp_path / "dogfood" + rules_dir = cfg_dir / "rulesets" + rules_dir.mkdir(parents=True) + rules_path = rules_dir / "internal.yaml" + _write_minimal_ruleset(rules_path) + cfg_path = cfg_dir / "pickled.ruleset.yaml" + cfg_path.write_text( + "rulesets:\n - path: ./rulesets/internal.yaml\n short_name: pickled-internal\n", + encoding="utf-8", + ) + sources = resolve_ruleset_sources( + tmp_path, + ruleset_config=cfg_path, + ruleset_dir=None, + ) + assert sources is not None + assert len(sources.rulesets) == 1 + assert sources.rulesets[0].path == rules_path.resolve() + + +def test_run_tag_writes_proposals(tmp_path: Path) -> None: + features = tmp_path / "features" + features.mkdir() + feature_text = ( + "Feature: greet\n\n" + " Scenario: greet by name\n" + " Given a user\n" + " When they greet\n" + " Then ok\n" + ) + (features / "tiny_target_greet.feature").write_text(feature_text, encoding="utf-8") + rules_dir = tmp_path / "rulesets" + rules_dir.mkdir() + _write_minimal_ruleset(rules_dir / "internal.yaml") + sources = resolve_ruleset_sources( + tmp_path, + ruleset_config=None, + ruleset_dir=rules_dir, + ) + result = run_tag(tmp_path, ruleset_sources=sources, quick=True) + assert not result.skipped + assert result.proposals_path.is_file() + data = json.loads(result.proposals_path.read_text(encoding="utf-8")) + assert data["features"] + + +def test_repair_split_scenario_tags() -> None: + broken = ( + "Feature: x\n\n" + " Sc@bdd-domain:gherkin-feature-header-required\n" + "enario: whitespace story\n\n" + " Scenar@bdd-domain:draft-empty-story-deterministic-failure\n" + "io: gate failure\n\n" + " Scenario@bdd-domain:draft-empty-story-deterministic-failure\n" + ": empty story\n" + ) + fixed = repair_split_scenario_tags(broken) + assert "Sc@" not in fixed + assert " @bdd-domain:gherkin-feature-header-required\n" in fixed + assert " Scenario: whitespace story\n" in fixed + assert " Scenario: gate failure\n" in fixed + assert " Scenario: empty story\n" in fixed + + +def test_tag_respects_surfaces_filter(tmp_path: Path) -> None: + features = tmp_path / "features" + features.mkdir() + (features / "tiny_target_greet.feature").write_text( + "Feature: greet\n\n Scenario: one\n Given x\n", + encoding="utf-8", + ) + (features / "other_pkg_ping.feature").write_text( + "Feature: ping\n\n Scenario: two\n Given y\n", + encoding="utf-8", + ) + rules_dir = tmp_path / "rulesets" + rules_dir.mkdir() + _write_minimal_ruleset(rules_dir / "internal.yaml") + sources = resolve_ruleset_sources( + tmp_path, + ruleset_config=None, + ruleset_dir=rules_dir, + ) + result = run_tag(tmp_path, ruleset_sources=sources, quick=True, surfaces=("greet",)) + data = json.loads(result.proposals_path.read_text(encoding="utf-8")) + paths = {entry["feature_path"] for entry in data["features"]} + assert paths == {"features/tiny_target_greet.feature"} diff --git a/packages/pickled-data/src/pickled_data/drafter.py b/packages/pickled-data/src/pickled_data/drafter.py index 98e09fa..9bb32e1 100644 --- a/packages/pickled-data/src/pickled_data/drafter.py +++ b/packages/pickled-data/src/pickled_data/drafter.py @@ -6,6 +6,7 @@ import sqlglot from pickled_core.llm.base import LLMClient, Message +from pickled_core.llm.sanitize import strip_markdown_fence RATIONALE_SENTINEL = "---RATIONALE---" _DRAFT_MODEL = "claude-sonnet-4-5-20250929" @@ -44,7 +45,7 @@ def draft_from_intent( stop=None, extras=None, ) - text, rationale = self._split_output(completion.text) + text, rationale = self._split_output(strip_markdown_fence(completion.text)) warnings = tuple(self._validate(text, dialect=dialect)) return DraftResult(text=text, rationale=rationale, warnings=warnings) diff --git a/packages/pickled-diff/src/pickled_diff/drafter.py b/packages/pickled-diff/src/pickled_diff/drafter.py index 352f453..1583c1c 100644 --- a/packages/pickled-diff/src/pickled_diff/drafter.py +++ b/packages/pickled-diff/src/pickled_diff/drafter.py @@ -6,6 +6,7 @@ from dataclasses import dataclass from pickled_core.llm.base import LLMClient, Message +from pickled_core.llm.sanitize import strip_markdown_fence RATIONALE_SENTINEL = "---RATIONALE---" _DRAFT_MODEL = "claude-sonnet-4-5-20250929" @@ -44,7 +45,7 @@ def draft_from_examples( stop=None, extras=None, ) - text, rationale = self._split_output(completion.text) + text, rationale = self._split_output(strip_markdown_fence(completion.text)) items, warnings = self._validate(text, target_size=target_size) return CorpusDraftResult( items=items, diff --git a/packages/pickled-iac/src/pickled_iac/drafter.py b/packages/pickled-iac/src/pickled_iac/drafter.py index 89f41dd..bb3aca6 100644 --- a/packages/pickled-iac/src/pickled_iac/drafter.py +++ b/packages/pickled-iac/src/pickled_iac/drafter.py @@ -6,6 +6,8 @@ from pathlib import Path from pickled_core import LLMClient, PromptTemplate +from pickled_core.llm.sanitize import strip_markdown_fence +from pickled_core.llm.turns import complete_prompt from pickled_iac.oracle import iac_binary, validate from pickled_iac.types import IaCArtifact @@ -41,18 +43,13 @@ def draft_module( user_story=user_story, error_feedback=feedback, ) - from pickled_core.llm.turns import complete_prompt - - hcl = complete_prompt( - self._llm, - prompt, - system="Output only Terraform HCL. No fences, no commentary.", - ).strip() - if hcl.startswith("```"): - lines = hcl.splitlines() - hcl = "\n".join( - line for line in lines if not line.strip().startswith("```") - ).strip() + hcl = strip_markdown_fence( + complete_prompt( + self._llm, + prompt, + system="Output only Terraform HCL. No fences, no commentary.", + ) + ) with tempfile.TemporaryDirectory() as tmp: root = Path(tmp) diff --git a/packages/pickled-rules/README.md b/packages/pickled-rules/README.md index 377d761..df5401e 100644 --- a/packages/pickled-rules/README.md +++ b/packages/pickled-rules/README.md @@ -102,6 +102,20 @@ and `rulesets:` in the same file is an error. Each ruleset emits its own verdict (gate name `rules.coverage.` when multiple are present; `rules.coverage` when only one is configured). +Optional `feature_glob:` selects where feature files live (glob relative +to the workspace root). Default: `features/**/*.feature`. + +```yaml +rulesets: + - path: ./rulesets/bdd-domain.yaml + short_name: bdd-domain +feature_glob: bdd/features/**/*.feature +``` + +Dogfood keeps features under `bdd/features/`; set `feature_glob` so +`pickled-spec check-all --workdir dogfood/` and mine evaluate find them +without a top-level `features/` directory. + ## What v0.1 ships - YAML rule set loader and schema validation diff --git a/packages/pickled-rules/src/pickled_rules/drafter.py b/packages/pickled-rules/src/pickled_rules/drafter.py index 1198cf5..ab7cde7 100644 --- a/packages/pickled-rules/src/pickled_rules/drafter.py +++ b/packages/pickled-rules/src/pickled_rules/drafter.py @@ -5,6 +5,7 @@ from dataclasses import dataclass from pickled_core.llm.base import LLMClient, Message +from pickled_core.llm.sanitize import strip_markdown_fence from pickled_rules.loader import RuleSetValidationError, load_ruleset_from_text @@ -66,7 +67,7 @@ def draft_from_brief( stop=None, extras=None, ) - text, rationale = self._split_output(completion.text) + text, rationale = self._split_output(strip_markdown_fence(completion.text)) warnings = tuple(self._validate(text)) return DraftResult(text=text, rationale=rationale, warnings=warnings) diff --git a/packages/pickled-rules/src/pickled_rules/gates_runner.py b/packages/pickled-rules/src/pickled_rules/gates_runner.py index 5048cc5..4e836ba 100644 --- a/packages/pickled-rules/src/pickled_rules/gates_runner.py +++ b/packages/pickled-rules/src/pickled_rules/gates_runner.py @@ -28,6 +28,16 @@ def _workdir_config(root: Path) -> dict[str, Any]: return data if isinstance(data, dict) else {} +def _feature_glob(cfg: dict[str, Any]) -> str: + raw = cfg.get("feature_glob") + if raw is None: + return "features/**/*.feature" + if not isinstance(raw, str): + msg = "pickled.ruleset.yaml: 'feature_glob' must be a string" + raise RuleSetValidationError(msg) + return raw + + def _resolve_ruleset_entries(root: Path, cfg: dict[str, Any]) -> list[_RulesetEntry]: has_singular = "ruleset" in cfg has_plural = "rulesets" in cfg @@ -111,7 +121,8 @@ def run_all(workdir: Path | str) -> list[GateResult]: ) ] - features = sorted(root.glob("features/**/*.feature")) + feature_pattern = _feature_glob(cfg) + features = sorted(root.glob(feature_pattern)) if not features: return [ GateResult( diff --git a/packages/pickled-rules/tests/test_gates_runner_multi.py b/packages/pickled-rules/tests/test_gates_runner_multi.py index c33d965..b10d744 100644 --- a/packages/pickled-rules/tests/test_gates_runner_multi.py +++ b/packages/pickled-rules/tests/test_gates_runner_multi.py @@ -205,6 +205,39 @@ def test_empty_rulesets_list_fails_validation(tmp_path: Path) -> None: assert "at least one ruleset entry required" in results[0].notes +def test_default_feature_glob_unchanged(tmp_path: Path) -> None: + _write_workspace( + tmp_path, + config_yaml="ruleset: ./rulesets/rs.yaml\nruleset_short_name: team-rules\n", + ) + results = run_all(tmp_path) + assert len(results) == 1 + assert results[0].verdict == Verdict.PASS + + +def test_custom_feature_glob_finds_features_in_subdir(tmp_path: Path) -> None: + rules_dir = tmp_path / "rulesets" + rules_dir.mkdir() + shutil.copy(_FIXTURE_RULESET, rules_dir / "rs.yaml") + (tmp_path / "pickled.ruleset.yaml").write_text( + """\ +ruleset: ./rulesets/rs.yaml +ruleset_short_name: team-rules +feature_glob: bdd/features/**/*.feature +""", + encoding="utf-8", + ) + bdd_features = tmp_path / "bdd" / "features" + bdd_features.mkdir(parents=True) + (bdd_features / "nested.feature").write_text( + _FEATURE_PASS.format(short="team-rules"), + encoding="utf-8", + ) + results = run_all(tmp_path) + assert len(results) == 1 + assert results[0].verdict == Verdict.PASS + + def test_non_mapping_entry_fails_validation(tmp_path: Path) -> None: (tmp_path / "pickled.ruleset.yaml").write_text( 'rulesets: ["bad-string"]\n', diff --git a/packages/pickled-schema/src/pickled_schema/openapi/drafter.py b/packages/pickled-schema/src/pickled_schema/openapi/drafter.py index 37a0d86..75c54a7 100644 --- a/packages/pickled-schema/src/pickled_schema/openapi/drafter.py +++ b/packages/pickled-schema/src/pickled_schema/openapi/drafter.py @@ -7,6 +7,8 @@ import yaml from pickled_core import LLMClient, PromptTemplate +from pickled_core.llm.sanitize import strip_markdown_fence +from pickled_core.llm.turns import complete_prompt from pickled_schema.openapi.validator import validate_openapi_dict from pickled_schema.types import SchemaArtifact, SchemaFormat, SchemaValidationError @@ -61,14 +63,12 @@ def draft_endpoint( gherkin_context=gherkin_context + extra, existing_component_names=", ".join(components) or "(none)", ) - from pickled_core.llm.turns import complete_prompt - raw = complete_prompt( self._llm, prompt, system="Output only YAML for the path item. No fences, no prose.", ) - loaded = yaml.safe_load(raw.strip()) + loaded = yaml.safe_load(strip_markdown_fence(raw)) if not isinstance(loaded, dict): last_error = "LLM output is not a YAML mapping" continue diff --git a/scripts/inventory_functionality.py b/scripts/inventory_functionality.py index be73515..0ce749d 100644 --- a/scripts/inventory_functionality.py +++ b/scripts/inventory_functionality.py @@ -9,6 +9,7 @@ from pathlib import Path from pickled_core.mine import inventory_lib +from pickled_core.mine.inventory_stage import enrich_inventory_data REPO_ROOT = Path(__file__).resolve().parent.parent @@ -108,6 +109,7 @@ def main(argv: list[str] | None = None) -> int: verbose=args.verbose, ) payload = inv.to_dict(REPO_ROOT) + enrich_inventory_data(payload, REPO_ROOT) emit_json = args.format in ("both", "json") emit_md = args.format in ("both", "md")