Feat/add spec mining

docs/decisions/0005-pickled-spec-mine.md

@@ -0,0 +1,76 @@
+# ADR-0005: `pickled-spec mine` staged mining pipeline
+
+- **Status:** Accepted
+- **Date:** 2026-05-28
+- **Deciders:** pickled-spec contributors
+
+## Context
+
+Hand-running the dogfood loop (inventory → story → feature → tag → gates)
+did not scale across dozens of CLI commands, MCP tools, and packages.
+We needed a generic extractor that works on arbitrary Python repos, not a
+one-off script tied to this monorepo.
+
+Early inventory runs missed umbrella MCP tools when `pickled-spec` lived
+on a workspace member rather than the root `pyproject.toml`. Story
+generation was initially sequential and could run for many minutes on a
+large repo without scoping.
+
+## Decision drivers
+
+- Work on any Python repo with Click-discoverable CLIs, not only
+  pickled-spec.
+- Stage isolation: re-run one stage from files on disk.
+- Graceful degradation without an LLM (placeholders, skip features).
+- Actionable errors when stages run out of order.
+- Performance controls (`--surfaces`, parallel quick mode, existing cache).
+
+## Considered options
+
+1. **Monolithic `mine` command** — single run, no intermediate artifacts.
+   Rejected: hard to debug, expensive to repeat one step, poor fit for
+   human review between stages.
+
+2. **Staged pipeline with filesystem contract (chosen)** — each stage reads
+   and writes under `--output`. Enables `mine all` and individual
+   subcommands.
+
+3. **MCP-first mining** — expose stages only as MCP tools. Deferred: CLI
+   first; MCP surface for mine is future work.
+
+## Decision outcome
+
+Ship `pickled-spec mine` with six stages: inventory, stories, features,
+tag, evaluate, report. Stages communicate via `inventory.json`,
+`stories/`, `features/`, `tags-proposals.json`, and `evaluation/*.json`.
+`mine all` orchestrates the chain; `--surfaces` filters work per stage.
+
+MCP umbrella detection scans the target root and uv workspace members
+for `pickled-spec` (or `pickled.mcp.subservers`). Rule set paths in
+`--ruleset-config` resolve relative to the config file directory.
+
+Ambiguity evaluation reuses `pickled_bdd.cli.run_ambiguity_gate`, the
+same entry point as `pickled-bdd check --gate ambiguity` and the
+`pickled-bdd ambiguity` alias.
+
+## Consequences
+
+**Positive**
+
+- Mining is separate from dogfood: dogfood is one consumer of the same
+  tools.
+- Re-runnable stages and inspectable artifacts.
+- Scoped runs via `--surfaces` keep LLM stages practical on monorepos.
+
+**Negative**
+
+- Disk layout is a public contract; changes need versioning care.
+- Full monorepo mining without `--surfaces` remains LLM-heavy.
+- Evaluate reports gate verdicts as-is; AmbiguityGate threshold tuning is
+  out of scope for mine.
+
+## Future work
+
+- Multi-language inventory (non-Python CLIs).
+- MCP tools wrapping mine stages.
+- AmbiguityGate calibration as its own change set.

docs/decisions/0006-mine-code-reading.md

@@ -0,0 +1,107 @@
+# ADR-0006: `pickled-spec mine code` static code reading
+
+- **Status:** Accepted
+- **Date:** 2026-05-28
+- **Deciders:** pickled-spec contributors
+
+## Context
+
+Inventory and docstrings describe surfaces at a high level. For gates and CLIs
+that delegate to helpers, the docstring often understates real behaviour
+(temperature, validation, return shape). Phase 8e adds a dedicated **code**
+stage that extracts source for each mined surface and, optionally, a bounded
+set of intra-project callees.
+
+## Decision drivers
+
+- Ground later story generation in **observed code**, not names alone.
+- Stay within stdlib (`ast` only): no grimp/pydeps dependency.
+- Hard caps and a **visited** set so traversal cannot run away on cycles.
+- Optional diagnostic cycle reporting without changing traversal semantics.
+
+## Decision
+
+Add `pickled-spec mine code` after inventory and before stories in `mine all`.
+
+### Depth modes
+
+| Mode | Content |
+|------|---------|
+| `signature` | Root signature, return annotation, docstring |
+| `body` | Root full function/method body (default) |
+| `callgraph` | Root body plus callee bodies up to `--max-hops` |
+
+### Callee scope
+
+- `self` — methods on the enclosing class (`self.helper()`).
+- `same-package` — `self` plus same-package imports (default).
+- `any-pickled` — same-package plus any `pickled_*` import.
+
+### Bounds
+
+- `--max-callees` (default 8) and `--max-code-lines` (default 400) per surface.
+- `visited` keys (`module:qualname`) prevent re-expansion; this is the cycle
+  safety mechanism.
+- `--detect-cycles` runs a small DFS on collected edges and writes
+  `code-context/_cycles.json` for the run log / report; it does not alter BFS.
+
+### Output
+
+`code-context/<surface-id>.md` per surface. Surfaces without a resolvable
+definition (e.g. MCP tool names with no mapped callable) get a placeholder
+file and the stage continues.
+
+## Known limitations (v1)
+
+- **Protocol / dynamic dispatch** — calls such as `self._llm.complete(...)`
+  where `_llm` is a Protocol or opaque attribute are recorded as *unresolved*
+  callees with a reason; they are not chased.
+- **Python only** — no cross-language call graphs.
+- **Static resolution only** — no runtime type inference or polymorphic targets.
+
+Stories do not consume code-context until Phase 8f.
+
+## Resolution patterns and limits (Phase 8e-fix)
+
+Each collected callee records `resolution_kind` on the ref. Default
+`--max-hops` is **2** so one delegation past the entry surface is included.
+
+### Resolved kinds
+
+| Kind | Pattern | Example |
+|------|---------|---------|
+| `free_function` | Same-module or imported callable | `helper()`, `chain.entry()` |
+| `self_method` | `self.method()` on enclosing class | `self.helper()` |
+| `constructor_method` | `Class(args).method()` | `Worker(cfg).process()` |
+| `module_constructor` | `mod.Class(args).method()` | `mod.Worker(cfg).process()` |
+| `annotated_param` | Parameter annotation pins type | `def f(w: Worker): w.m()` |
+| `annotated_var` | Annotated local | `x: Worker = …; x.m()` |
+| `assigned_constructor` | `x = Worker(); x.m()` (stable) | assignment tracking |
+
+`@property`, `@staticmethod`, `@classmethod`, and `async def` bodies resolve
+when the receiver type is known. Constructor arguments may contain separate
+resolvable calls (e.g. `Worker(Builder(x).build()).process()`).
+
+### Deliberately unresolved (reason strings)
+
+| Reason | Pattern |
+|--------|---------|
+| `protocol or unknown attribute type` | `self._llm.complete()` (nested attribute on `self`) |
+| `parameter '…' has no type annotation` | `def f(w): w.method()` |
+| `receiver is a subscript expression` | `items[0].method()` |
+| `variable '…' reassigned; type not stable` | `x = Worker(); x = Other(); x.m()` |
+| `receiver is a conditional expression` | `(a if c else b).run()` |
+| `receiver is a return value of unannotated callable` | `factory().build().run()`, `.process().finalize()` |
+| `dynamic attribute access` | `getattr(obj, "m")()` |
+| `method not found on class; possibly inherited (base not resolved in v1)` | method absent on declared class |
+| Name collision / unknown receiver | two classes share method name, type not pinned |
+
+Inherited methods (MRO) are not walked in v1. Return-type inference for
+arbitrary call chains is out of scope. Traversal uses the same caps and
+`visited` set as 8e; cycles are reported via `--detect-cycles` when enabled.
+
+## Consequences
+
+- `mine all` produces `code-context/` for downstream story prompts.
+- Readers must pass inventory first; missing `inventory.json` raises an
+  actionable error naming `mine inventory`.

docs/decisions/0007-code-aware-stories-and-drift.md

@@ -0,0 +1,81 @@
+# ADR-0007: Code-aware stories and docstring drift detection
+
+- **Status:** Accepted
+- **Date:** 2026-05-28
+- **Deciders:** pickled-spec contributors
+
+## Context
+
+Phase 8e added static code reading (`code-context/<surface-id>.md`).
+Phase 8e-fix hardened the resolver so constructor-then-method patterns
+(e.g. `FeatureDrafter(llm).draft_from_story(story)`) resolve to real
+bodies, not just entry-point glue.
+
+Docstring-only stories (Phase 8d) were too shallow and sometimes wrong.
+On the real `pickled-bdd` draft surface, a hand-written story claimed the
+drafter **validates** Gherkin output. Code-reading showed the opposite:
+`draft_from_story`'s docstring states the drafter does **not** validate
+(returned Gherkin is raw; `warnings=()`). Human peer review had introduced
+that confabulation. The mine pipeline can now ground stories in extracted
+source instead of inventory summaries alone.
+
+## Decision
+
+### Code-grounded story generation
+
+When `code-context/<surface-id>.md` exists under the mining output directory,
+the stories stage loads root and resolved callee bodies plus the unresolved
+callee list and passes them to the story prompt together with the surface
+docstring. The model writes **observable behavior** (contract), not
+implementation mechanics.
+
+### Decision B: drift detection
+
+The code is the source of truth. If the docstring **contradicts** the code,
+the model emits a `---DRIFT---` block; each bullet is rendered under Open
+questions prefixed with `Docstring drift:`. We do not silently override the
+docstring or show code and docstring side-by-side without synthesis.
+
+When no code-context exists, behavior falls back to Phase 8d docstring-only
+rules and DRIFT is always empty.
+
+### Anti-implementation-leak
+
+Stories must not mention line numbers, private method names, or call-chain
+narration ("it calls X then Y"). A reader should understand the contract
+without seeing source. The prompt enforces this; tests guard the render path.
+
+### Unresolved-call honesty
+
+Calls the resolver cannot pin (protocol dispatch, dynamic getattr, etc.)
+remain listed in code-context. The prompt forbids inventing behavior behind
+those calls; delegated behavior is stated as uncertain.
+
+### Friction #15: unresolved noise filtering
+
+Before reporting unresolved callees, the code reader drops:
+
+- **Stdlib-surface methods** — e.g. `str.strip()`, `Path.read_text()` on
+  receivers that are not resolvable intra-project types.
+- **Decorator registration** — callee scan walks function **bodies** only,
+  so `@main.command()` on the definition is not treated as a behavioral call.
+
+**Limit:** a user-defined method whose name collides with a common builtin
+method (e.g. `.strip()`) on an unresolved receiver is also dropped. That
+would have been unresolved noise anyway; accepted trade-off.
+
+### Provenance metadata
+
+Each story's Metadata section records **Code depth**, **Units read**, and
+**Unresolved** counts when code-context was present, so readers can see how
+strong the grounding was (`signature` vs `callgraph`).
+
+## Consequences
+
+- `mine all` runs inventory → code → stories; stories auto-detect
+  `code-context/` under `--output`.
+- `mine stories` accepts optional `--code-context` to override the directory.
+- Mine acts as a **docstring drift detector** when docstrings lie or lag code.
+- Story quality scales with `--depth` and `--max-hops` on the code stage.
+- Live LLM quality still depends on the model; tests use canned clients for
+  wiring and anti-leak contracts.

docs/mcp.md

@@ -193,8 +193,16 @@ For workspaces that need to enforce more than one rule set in parallel,
 `pickled.ruleset.yaml` also accepts a `rulesets:` list — see
 [`packages/pickled-rules/README.md`](../packages/pickled-rules/README.md).
 
+## Mining
+
+To introspect an arbitrary Python repo and scaffold stories, features, tags,
+and gate reports from its CLIs and MCP tools, use the staged
+[`pickled-spec mine`](mining.md) pipeline (`mine inventory`, `mine all`, …).
+Mining is CLI-first today; MCP wrappers for mine stages are not shipped yet.
+
 ## See also
 
+- [`mining.md`](mining.md) — staged mine pipeline for any Python repo
 - [`pattern.md`](pattern.md) — LLM-to-DSL bridge
 - [`gates.md`](gates.md) — compensating gates exposed as tools
 - [`integration-example.md`](integration-example.md) — end-to-end example