diff --git a/CHANGELOG.md b/CHANGELOG.md index 4457784..d3b2bd4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,14 @@ ## Unreleased +## 2.2.0 + +**Prompt builder controls.** Flowkey's `prompt:` mode can now target Claude Code or generic chat tools without exposing raw built-in system-prompt editing. + +### Added + +- **Prompt builder settings for `prompt:` mode.** The Config tab now exposes bounded prompt-output controls (target agent, action, detail, structure, acceptance criteria, verification, output expectations, and a capped user suffix) without allowing raw edits to the locked built-in system prompt. The default remains Claude Code-compatible and keeps the existing prompt-mode system prompt byte-identical; `generic_chat` adds a production Markdown adapter with target-aware validation and deterministic fallback/preview. Live FastFlowLM eval on `qwen3.5:4b`: `claude_code` 2/2, `generic_chat` 10/10. + ## 2.1.1 **Maintenance.** Fixes a recurring meetings-batch error and clears a batch of small audit/doc items. diff --git a/README.md b/README.md index 6a2cb30..6b96a08 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,11 @@ Flowkey is a Windows desktop assistant that adds local-LLM hotkeys for grammar f Everything runs locally through [FastFlowLM](https://fastflowlm.com) (AMD Ryzen AI NPU) or, on machines without the NPU, through [Ollama](https://ollama.com) (CPU/GPU) as a secondary provider. No cloud service, analytics, or telemetry is used by the app. -Current version: `2.1.1` +Current version: `2.2.0` + +## What's new in 2.2 + +- **Prompt builder settings for `prompt:` mode.** Dashboard → Config now lets you tune prompt output for Claude Code or generic chat targets, choose action/detail/structure, add acceptance criteria or verification sections, and preview the deterministic shape without a model call. Claude Code remains the default; `generic_chat` cleared the live FastFlowLM eval gate on `qwen3.5:4b` (10/10). ## What's new in 2.1 @@ -115,6 +119,7 @@ The dashboard is a web page served by the local daemon — open it from the tray - **Tabs:** Overview, Chat, Telemetry, History, Notes, Meetings, Config, Benchmark. - **Theme:** auto-follows your OS day/night setting; the topbar button cycles auto → light → dark. - **Custom modes:** Config → Custom modes lets you add your own `prefix:` commands (id + system prompt). Changes apply to the running app within a second. +- **Prompt builder:** Config → Prompt builder controls the built-in `prompt:` output style without allowing raw edits to the locked system prompt. - **Models:** pull models with live progress — pick a suggestion or type any name (on Ollama, anything from the [library](https://ollama.com/library) works); set active, remove. Suggestions are hardware-aware: detected RAM/VRAM caps the model size (e.g. 32 GB RAM → ~4B on the NPU; 8 GB VRAM → ~9B on the GPU), oversized models are hidden, and free-typing one asks before pulling. - **Benchmark:** works on both providers — `flm bench` on FastFlowLM (~10–20 min, NPU), timed generations with native metrics on Ollama (~1–3 min, server keeps running). - **Notes:** browse or search your vault; History shows recent runs (text is stored only if history storage is enabled). diff --git a/SPEC.md b/SPEC.md index b3d823b..967551c 100644 --- a/SPEC.md +++ b/SPEC.md @@ -16,15 +16,16 @@ Caveman-encoded (compression, not amputation). Paths / ids / action names / numb - LLM: FastFlowLM NPU @ `:52625` | Ollama @ `:11434`, OpenAI-compat `POST /v1/chat/completions` - dashboard: daemon-served `scripts/ui/web/{index.html,app.js,styles.css}`, CSP `default-src 'self'` - paths: `scripts/paths.py` → USER_ROOT/{config,data,logs}; `_version.py` = version src of truth -- version: `2.1.1` (this branch, maintenance); `2.1.0` released (tag on `7835d4b`); repo `agr77one/Fastflow` +- version: `2.2.0` (prompt_builder release branch); `2.1.0` released (tag on `7835d4b`); repo `agr77one/Fastflow` - run tree = `flowkey-pub2` (worktree, branch `live`=origin/main). old `FastFlowPrompt_Local_Setup`=1.5.0 stale. ## §I interfaces -- cfg blocks: `enabled`, `llm`, `providers.{fastflowlm,ollama}`, `server`, `routing`, `notes`, `chat`, `modes`, `dictionary`, `notifications`, `meetings`, `hotkeys` +- cfg blocks: `enabled`, `llm`, `providers.{fastflowlm,ollama}`, `server`, `routing`, `prompt_builder`, `notes`, `chat`, `modes`, `dictionary`, `notifications`, `meetings`, `hotkeys` - api: `POST /action/` ! header `X-FFP-API: 1` → 200 `{ok,result,error,elapsed_ms}` - api: `GET /` → dashboard; `GET /healthz` → `{ok,version,api,actions}` - action: `config_snapshot` → full cfg; `apply_config_patch {patch}` → merge (whitelist `filter_config_patch`) +- action: `prompt_builder_preview {settings?,sample?}` → deterministic local preview (`⊥` LLM call) - action: `notify_gate {title,message}` → `{show,reason,category}` (logs); `notifications_log {limit}` → rows - action: `quill_status` → `{reachable,enabled,server,server_version}` - action: `quill_search_meetings {query,limit}` → `{meetings:[{id,title,date,duration,participants,url}]}` @@ -41,7 +42,7 @@ Caveman-encoded (compression, not amputation). Paths / ids / action names / numb - data: `data/{meeting_digests,meeting_action_status,meeting_skips,notifications,chat_threads}.jsonl` - autostart: HKCU Run `FastFlowPrompt` → bundled `AutoHotkey64.exe` + `grammarFix.ahk`; `FlowkeyGitSync` → `sync.ps1` - sched: Windows task `FlowkeyGitSync` daily 12:00 → `sync.ps1` (ff-only pull, guarded) -- ACTIONS count = 73 +- ACTIONS count = 75 ## §V invariants @@ -68,6 +69,7 @@ Caveman-encoded (compression, not amputation). Paths / ids / action names / numb - V21: local data (config/data/logs/vendor/certs) ∈ `.gitignore` ∴ pull moves code only, never user data - V22: `sync.ps1` ∃ uncommitted tracked changes → skip pull (⊥ clobber un-pushed WIP) - V23: meeting `NoContentError` & age ≥ 2d → skip-marker (`meeting_skips.jsonl`) ∴ ⊥ re-queue ever; age < 2d → retry (Quill transcript may still sync); non-content errors ⊥ skip-mark. batch errors → `daemon.log` only (⊥ UI panel, per user) +- V24: prompt_builder default cfg ⇒ `CLAUDE_PROMPT_SYSTEM_PROMPT` identity; non-default prompt validation target-aware (XML only when effective structure=xml); built-in `modes.prompt.system_prompt` still locked ## §T tasks @@ -93,6 +95,7 @@ T17|.|[DOCS] installer layout: build script says flattened, installer.md still s T18|.|[DOCS] provider roadmap marks selector/status UX incomplete → update to reflect it exists|— T19|x|[DOCS] first-run wizard text "chat popup" → "Open chat" + hotkey now `^!c` (2.1.1)|B5 T20|x|[DOCS] daemon log location — audited 2.1.1: no stale ref in README/docs; nothing to change|— +T21|x|prompt_builder cfg + claude_code identity + generic_chat adapter + dashboard controls/preview|V17,V24 ``` ## §B bugs diff --git a/config/grammar_hotkey.config.example.json b/config/grammar_hotkey.config.example.json index 42c56d9..108761d 100644 --- a/config/grammar_hotkey.config.example.json +++ b/config/grammar_hotkey.config.example.json @@ -27,6 +27,18 @@ "chunk_size_chars": 1200, "min_chunk_chars": 700 }, + "prompt_builder": { + "target_agent": "claude_code", + "detail_level": "balanced", + "action_mode": "implement", + "structure": "agent_default", + "include_acceptance_criteria": false, + "include_verification": false, + "include_output_format": true, + "preserve_user_constraints": true, + "allow_user_suffix": true, + "user_suffix": "" + }, "dictionary": { "protected_words": [] }, diff --git a/docs/prompt-builder-finish-and-validate.md b/docs/prompt-builder-finish-and-validate.md new file mode 100644 index 0000000..66ac044 --- /dev/null +++ b/docs/prompt-builder-finish-and-validate.md @@ -0,0 +1,286 @@ +# Prompt Builder (2.2.0) — Finish & Validate Runbook + +Date: 2026-07-09 +Companion to: `docs/prompt-logic-refinement-plan.md` (design) — this is the +step-by-step to take the implemented-but-unfinished feature to a shipped 2.2.0. + +## Execution status (2026-07-09) + +Local development/validation complete on branch `feat/2.2.0-prompt-builder`. + +Completed from this runbook: + +- Step 0: branch created; unrelated `docs/history-visibility-plan.md` remains + untracked and excluded. +- Step 1: `python -m ruff check scripts tests tools` clean. +- Step 2: added CI-gated pytest coverage for fallback contract safety, + generic_chat echo detection, and action-mode preservation. +- Step 3: default identity prompt now fails closed when the legacy prompt is + missing; covered by pytest. +- Step 4: `tools/prompt_builder_eval.py` now supports model-free and live-FLM + modes. Live FastFlowLM artifact: + `data\benchmarks\prompt_builder_eval_flm_20260709.json`. +- Step 5: version bumped to `2.2.0`; CHANGELOG/README/SPEC updated; live eval + result recorded (`claude_code` 2/2, `generic_chat` 10/10), so generic_chat is + production rather than experimental. +- Step 6: local gates passed: + `ruff`, `pytest tests -q` (345 passed), bundled Node `--check app.js`, + `py_compile`, model-free eval, live-FLM eval. + +Pending remote/release work after local commit: + +- Step 7 remote portion onward: push, PR, CI, ruleset-toggle merge, tag, + installer build, and GitHub release notes. These require remote credentials/ + release operations and should be run only when ready to publish. + +## Original code-review state (before this pass) + +Implemented and **uncommitted on `live`** (= `origin/main` `a60f7e0`). Verified +correct: identity drop-in (strict, tested, runtime-wired), prompt-mode-only +gating (locked grammar/summarize/tone untouched), contract gate only on +non-default, target-aware echo, validate→repair→fallback, staged rollout +(claude_code + generic_chat), filtered+re-normalized config patch, V5-clean +dashboard. **341 pytest pass; `node --check` app.js clean.** + +Blocking gaps this runbook was written to close: +1. `ruff` not clean — 2 `I001` import-sort errors. +2. Generator quality **unmeasured** — `tools/prompt_builder_eval.py` is model-free + (validates renderers only); no live-FLM numbers exist. +3. Three CI-gated tests missing (gate-fail fallback, generic_chat echo, + action-mode preservation in pytest). +4. Minor: identity path depends on `legacy_system_prompt` being present. +5. Version not bumped (still 2.1.1). + +## Definition of done + +- All V20 gates green: `ruff check scripts tests`, `pytest`, + `node --check scripts/ui/web/app.js`, AHK parse-check (only if a `.ahk` + changed — none currently). +- Live-FLM eval recorded: claude_code default reproduces the 2.1.1 baseline + (it's identity, so this must hold); generic_chat either clears its gate + (≥ 8/10) → production, or is shipped **experimental** with a UI note and + claude_code stays the default. +- Version `2.2.0` in the 4 files + `## 2.2.0` CHANGELOG heading + SPEC version + line; `prompt_builder_preview` confirmed read-only. +- Landed via ruleset-toggle squash-merge PR; tag `v2.2.0`; installer built; + release notes PATCHed. + +--- + +## Step 0 — Feature branch & scope + +The work is loose on `live`. Move only the prompt-builder change onto a branch; +keep the unrelated `docs/history-visibility-plan.md` out. + +```powershell +cd C:\Users\ArseniyGrechenkov\Downloads\transfer\transfer\flowkey-pub2 +git switch -c feat/2.2.0-prompt-builder live +``` + +Include in the eventual commit: `scripts/ffp_prompt_builder.py`, +`tests/test_prompt_builder.py`, `tools/prompt_builder_eval.py`, the modified +`scripts/*`, `scripts/ui/web/*`, `config/*`, `setup/defaults/*`, `tests/*`, +`SPEC.md`, `CHANGELOG.md`, `pyproject.toml`, `installer/fastflowprompt.spec`, +and `docs/prompt-logic-refinement-plan.md`. **Exclude** +`docs/history-visibility-plan.md` (separate feature). + +## Step 1 — Lint fix + +```powershell +python -m ruff check --fix scripts tests tools +python -m ruff check scripts tests # must print "All checks passed!" +``` + +Acceptance: `ruff check scripts tests` clean. (`tools/` is outside the CI gate +but fix it too for hygiene.) + +## Step 2 — Add the three CI-gated tests (`tests/test_prompt_builder.py`) + +All model-free (no LLM), so they run in the normal `pytest` gate. + +1. **Deterministic fallback always satisfies its own adapter contract** (covers + the gate-fail branch's safety): + ```python + def test_fallback_shape_passes_its_own_contract(): + import ffp_prompt_builder as pb + for cfg in ({"target_agent": "claude_code", "action_mode": "review"}, + {"target_agent": "generic_chat", "include_verification": True}): + s = pb.PromptBuilderSettings.from_config(cfg) + intent = pb.resolve_intent(s, "sample request") + shaped = pb.render_fallback(s, intent, "sample request") + assert pb.validate(shaped, s).valid, cfg + ``` + +2. **generic_chat conversion is not misread as an echo** (covers the + target-aware `has_target_structure` coupling): + ```python + def test_generic_markdown_conversion_not_flagged_as_echo(): + import ffp_prompt_builder as pb, grammar_fix + s = pb.PromptBuilderSettings.from_config({"target_agent": "generic_chat"}) + good_md = "## Task\nBuild X.\n## Context\n- repo Y\n## Constraints\n- keep scoped" + assert grammar_fix.is_weak_prompt_echo("build x for repo y", good_md, s) is False + echo = "build x for repo y" # reformatted input, no structure + assert grammar_fix.is_weak_prompt_echo("build x for repo y", echo, s) is True + ``` + +3. **action_mode is preserved** (review ⇒ no edit order; plan ⇒ no code edits): + ```python + def test_action_mode_review_and_plan_do_not_order_edits(): + import ffp_prompt_builder as pb + for mode in ("review", "plan"): + s = pb.PromptBuilderSettings.from_config({"target_agent": "generic_chat", "action_mode": mode}) + text = pb.render_fallback(s, pb.resolve_intent(s, "the PR"), "the PR") + assert "Implement the requested change" not in text + ``` + (Confirm the exact forbidden/required strings against + `ffp_prompt_builder._action_instruction`; `tools/prompt_builder_eval.py` + already encodes these strings — reuse them.) + +Run: `python -m pytest tests/test_prompt_builder.py -q` → all pass. + +## Step 3 — Identity hardening (minor) + +`build_system_prompt` returns identity only when `legacy_system_prompt` is +truthy. Add a guard/test so a missing mode prompt can't silently synthesize: + +- In `tests/test_prompt_builder.py`, assert that with default settings and an + EMPTY `legacy_system_prompt`, the result is still safe (either raises or + returns empty so the caller's `if not system_prompt: raise` at + `ffp_llm_client.py:298` fires). Document the dependency in a one-line comment + on `build_system_prompt`. + +## Step 4 — Measure generator quality (the Phase 5 gate) + +### 4a. Model-free eval (must already pass) +```powershell +python tools\prompt_builder_eval.py # text table; expect all valid=True +python tools\prompt_builder_eval.py --json # capture if you want an artifact +``` + +### 4b. Add a live-FLM mode to `tools/prompt_builder_eval.py` +Add an opt-in path (`--live --base-url [--model ] [--bearer ]`) that, +per case, sends `build_system_prompt(settings, intent, legacy_system_prompt=…)` ++ the case input to the local OpenAI-compatible endpoint, then runs +`validate()` (and the `must_contain`/`must_not_contain` checks) on the **model +output** — not the fallback. Emit per-adapter pass counts. Keep the default +model-free path unchanged. + +### 4c. Run it against FastFlowLM +```powershell +# start FLM (default provider/model) +Start-Process -FilePath flm -ArgumentList @('serve','qwen3.5:4b','--pmode','turbo','--host','127.0.0.1','--port','52625') -WindowStyle Hidden +Start-Sleep -Seconds 12 +python tools\prompt_builder_eval.py --live --base-url http://127.0.0.1:52625 --model qwen3.5:4b --json ` + --out data\benchmarks\prompt_builder_eval_flm_.json +``` + +Acceptance gates (record the numbers in the artifact + CHANGELOG/plan): +- **claude_code cases** ≥ the 2.1.1 baseline. (Default is identity, so a drop + here means something else regressed — investigate before shipping.) +- **generic_chat cases** ≥ 8/10 → ship generic_chat as production. +- generic_chat < 8/10 → keep it, but label it **experimental** in the dashboard + (a note that non-Claude targets may need retries) and keep claude_code the + default. Do NOT block the release on it — the default path is unaffected. + +## Step 5 — Version bump to 2.2.0 + docs + +Bump all four (V18 — CI smoke fails on drift): +- `scripts/_version.py` → `__version__ = "2.2.0"` +- `pyproject.toml` → `version = "2.2.0"` +- `installer/installer.iss` → `#define AppVersion "2.2.0"` +- `README.md` → `Current version: `2.2.0`` (and add a one-line Prompt-builder + note under the Config/dashboard section) + +Also: +- `CHANGELOG.md`: move the "Prompt builder settings…" bullet from `## Unreleased` + into a new `## 2.2.0` section with a short headline; note generic_chat's + production/experimental status per Step 4. +- `SPEC.md`: update the version line to `2.2.0`; confirm V24 + the + `prompt_builder` cfg block + `prompt_builder_preview` action + `ACTIONS count` + already present (they are). +- Confirm `prompt_builder_preview` is **not** in `_WRITE_ACTIONS` + (`scripts/ffp_daemon.py:821`) — it's a deterministic read. +- Cosmetic version strings in `installer/README.md` / `installer/sign.ps1` if + present. + +## Step 6 — Validate (full gate run) + +```powershell +python -m ruff check scripts tests # All checks passed! +python -m pytest tests -q # all pass +# JS (no node on this box — use VS Code's Electron as node): +$env:ELECTRON_RUN_AS_NODE=1 +$p = Start-Process "$env:LOCALAPPDATA\Programs\Microsoft VS Code\Code.exe" ` + -ArgumentList '--check','scripts\ui\web\app.js' -Wait -PassThru -NoNewWindow ` + -RedirectStandardError "$env:TEMP\pb.err" -RedirectStandardOutput "$env:TEMP\pb.out" +"exit=$($p.ExitCode)" # exit=0 +``` +AHK parse-check only if a `.ahk` was touched (none in this change). + +Also spot-check the drop-in by hand: with default config, +`build_system_prompt(default, intent, legacy=CLAUDE_PROMPT_SYSTEM_PROMPT)` == +`CLAUDE_PROMPT_SYSTEM_PROMPT` (already asserted by +`test_default_system_prompt_is_identity`). + +## Step 7 — Commit & push (tooling gotchas baked in) + +```bash +git add -A # review with: git status --short (exclude history-visibility-plan.md if undesired) +git commit -m "2.2.0: prompt-builder settings for prompt mode (claude_code + generic_chat)" +``` +Push — GCM only offers interactive auth (hangs non-interactive), so supply the +token and REDACT it: +```bash +export GIT_TERMINAL_PROMPT=0 +TOKEN=$(printf 'protocol=https\nhost=github.com\nusername=agr77one\n\n' | git credential fill 2>/dev/null | sed -n 's/^password=//p') +git push "https://agr77one:${TOKEN}@github.com/agr77one/Fastflow.git" \ + feat/2.2.0-prompt-builder:feat/2.2.0-prompt-builder 2>&1 | sed "s/${TOKEN}/[REDACTED]/g" +``` +Verify: `git ls-remote --heads origin feat/2.2.0-prompt-builder` (repo is public, +read works anonymously) and `origin/main` still `a60f7e0`. + +## Step 8 — PR + CI + ruleset-toggle squash-merge + +- Open PR base=`main`, head=`feat/2.2.0-prompt-builder` (title/body ≈ CHANGELOG). +- Wait for all CI checks green (Python 3.11/3.12/3.13, AutoHotkey syntax, + Web dashboard JS). +- Merge with the ruleset toggle (ruleset id **17344133** "Protect main"): + GET ruleset → PUT `enforcement=disabled` → `PUT /pulls//merge` + (`merge_method=squash`, **full 40-char** head sha) → **always** PUT + `enforcement=active` → delete branch. Do the GET/PUT JSON in **PowerShell** + (Invoke-RestMethod + ConvertTo-Json) — Git-Bash `/tmp` paths aren't readable by + native Windows Python. Restore enforcement even if the merge fails. + +## Step 9 — Tag, build, release notes + +```bash +git fetch origin --quiet +git switch live && git merge --ff-only origin/main # live -> new main +git tag -a v2.2.0 origin/main -m "Flowkey 2.2.0 — prompt builder" +git push "https://agr77one:${TOKEN}@github.com/agr77one/Fastflow.git" v2.2.0 2>&1 | sed "s/${TOKEN}/[REDACTED]/g" +``` +The tag triggers `release-installer.yml` → `Flowkey-Setup-2.2.0.exe` + a GitHub +Release with a GENERIC body. Monitor the run to completion, verify the asset, +then **PATCH the release body** with real notes: +`Invoke-RestMethod PATCH /repos/agr77one/Fastflow/releases/{id}` from PowerShell +(build the JSON with ConvertTo-Json; token in a header var, redacted). + +## Step 10 — Post-release + +- `git switch live` stays at the new main (already ff-merged). +- Update memory `flowkey-release-flow` current version → 2.2.0. +- Optionally start the codex/cursor adapters as 2.2.x (only after generic_chat + is proven), per the design doc's staged rollout. + +## Gotchas reference (bit us before — don't re-learn) + +- **No `node` on this machine** → JS check via VS Code Electron + (`ELECTRON_RUN_AS_NODE=1 … Code.exe --check`). +- **git-credential-manager is interactive-only** → non-interactive push hangs; + fetch token via `git credential fill`, push to + `https://agr77one:@github.com/...`, redact from all output. +- **Git-Bash `/tmp` / `mktemp -d` paths aren't readable by native Windows + Python** → do GitHub API calls + JSON in PowerShell (Invoke-RestMethod / + ConvertTo-Json), not bash+python. +- **PR merge API** requires the full 40-char head sha. +- **Ruleset toggle** must always re-enable enforcement (even on failure). diff --git a/docs/prompt-logic-refinement-plan.md b/docs/prompt-logic-refinement-plan.md new file mode 100644 index 0000000..8934da5 --- /dev/null +++ b/docs/prompt-logic-refinement-plan.md @@ -0,0 +1,377 @@ +# FastFlow Prompt Logic Refinement Plan (rev 2) + +Date: 2026-07-09 +Target release: **2.2.0** (minor feature — touches the locked built-in prompt path) +Status: LOCAL 2.2.0 RELEASE WORK COMPLETE — remote PR/tag/installer release still pending + +## Implementation status (2026-07-09) + +Implemented in this pass: + +- Backend prompt-builder module: `scripts/ffp_prompt_builder.py` + (`PromptBuilderSettings`, `PromptIntent`, `PromptRenderResult`, per-adapter + deterministic renderers, validators, repair, preview). +- Config: `prompt_builder` defaults, load normalization, patch whitelist, + suffix clamp, config snapshot inclusion, shipped config/example/default seed + updates. Built-in `modes.prompt.system_prompt` remains locked. +- Runtime: prompt mode now uses the builder system prompt; default settings keep + `CLAUDE_PROMPT_SYSTEM_PROMPT` byte-identical. Non-default settings get + target-aware structure checks, retry prompts, deterministic repair, and + fallback. `generic_chat` is the first non-Claude shipped adapter. +- Dashboard: Config tab Prompt builder card with target, action, detail, + structure, acceptance/verification/output toggles, preserve-constraints flag, + capped suffix, and deterministic no-LLM preview. +- Daemon/API: `prompt_builder_preview {settings?, sample?}` action. +- Eval: `tools/prompt_builder_eval.py` model-free and live-FLM adapter/action + contract gate for default Claude and shipped generic-chat cases. +- Tests/manifests: new prompt-builder tests, config/runtime/daemon test updates, + `pyproject.toml` and PyInstaller hidden import updated. +- SPEC/CHANGELOG: `SPEC.md` records `prompt_builder`, the preview action, and + invariant `V24`; `CHANGELOG.md` records the Unreleased feature. + +Verified: + +- `python -m py_compile scripts\ffp_prompt_builder.py scripts\ffp_config.py scripts\ffp_llm_client.py scripts\grammar_fix.py scripts\ffp_daemon.py` +- `python -m pytest tests -q` → 345 passed +- JSON parse check for `config/grammar_hotkey.config*.json` and + `setup/defaults/grammar_hotkey.config*.json` +- `python tools\prompt_builder_eval.py` → all 12 cases valid +- `python tools\prompt_builder_eval.py --live --base-url http://127.0.0.1:52625 --model qwen3.5:4b --bearer flm --json --out data\benchmarks\prompt_builder_eval_flm_20260709.json` + → `claude_code` 2/2, `generic_chat` 10/10 + +Still pending before publishing 2.2.0: + +- Commit/push branch, open PR, wait for CI, ruleset-toggle squash merge, tag + `v2.2.0`, build installer, and patch release notes. +- Codex/Cursor adapters remain 2.2.x follow-up, as planned below. + +## Revision note (what changed from rev 1, after code review) + +Rev 1 was directionally right but had two issues that would bite: + +1. **Its proposed defaults did not reproduce current behavior.** Today's prompt + mode (`CLAUDE_PROMPT_SYSTEM_PROMPT`) emits only `// + /` — no verification, no acceptance criteria. + Rev 1 defaulted `include_verification` and `include_acceptance_criteria` to + true, which would silently add sections to every existing user's prompts on + upgrade and re-invalidate the benchmark numbers. **Fixed:** default settings + now render byte-identical to 2.1.x (see "Drop-in guarantee"). +2. **Generator-capability risk was unquantified.** The local model (FLM + `qwen3.5:4b`) is the *generator*; `target_agent` only controls the *format of + what it produces*. The 2026-07 benchmark showed the single XML shape scores + only ~45–49/50 on the best local models and needs deterministic repair. + Asking the same 4B model to conditionally hit 4 formats × 5 action modes × 3 + detail levels is strictly harder. **Fixed:** staged adapter rollout + (claude_code + generic first), realistic per-adapter gates, defined + gate-fail fallback. + +Also resolved rev-1 open questions and defined machine-checkable per-adapter +contracts up front. Full detail below. + +## Goal + +A prompt-builder layer that turns rough user text into reliable prompts for +coding agents and chat tools, while keeping FastFlow's safety model: built-in +prompts stay protected, user customization is explicit, bounded, and stored as +settings. **FastFlow's prompt mode generates a copy-paste-ready prompt** that the +user pastes into their coding agent; the local model is the *generator*, and +`target_agent` selects the output *format*. + +## Current State (verified against code, 2026-07-09) + +- Built-in prompt mode is Claude-specific: `ffp_config.CLAUDE_PROMPT_SYSTEM_PROMPT` + steers the generator to emit ``, ``, ``, + `` (task = one sentence; constraints testable; no invented + requirements). +- Built-in mode prompts are locked: `_enforce_builtin_mode_prompts()` restores + code defaults on load; `filter_config_patch()` rejects edits to any mode in + `BUILTIN_MODE_IDS = {grammar, prompt, summarize, explain, tone}`. +- Runtime lives in `scripts/ffp_llm_client.py`: prompt mode gets larger output + budgets, anti-echo retries, and three XML-hardcoded helpers — + `has_prompt_structure()` (keys off `` etc.), `is_weak_prompt_echo()` + (early-returns "not echo" *because XML is present* — line 168), and + deterministic `force_prompt_shape()` (renders Claude XML). +- A shipped repair layer (`ffp_llm_client` + `tests/test_prompt_output_repair.py`) + fixes Claude-XML near-misses (label-to-XML, malformed ``). +- The benchmark contract checker `tools/provider_bench.check_prompt_contract()` + is XML-only and is *also* the provider-quality gate — its numbers assume XML. +- Dashboard customization exists only for non-built-in custom modes (`prefix:` + commands). No tuning of built-in prompt style, target agent, detail, or action. + +## Product Decision + +Do not replace prompt mode with one universal static prompt. Add a small +prompt-builder engine: + +1. Resolve intent from **settings + lightweight code heuristics** (never an LLM + classification call — see "Intent resolution"). +2. Apply safe, bounded user settings. +3. Select an agent adapter. +4. Produce (a) a generator **system prompt** and (b) a deterministic **fallback + renderer** for that adapter. +5. Validate the model output against the adapter's contract; repair or fall back. + +The internal structure is semantic, not literal: + +```text +Task / Outcome · Context · Constraints · Acceptance criteria · Verification · Output expectations +``` + +Adapters render that structure differently per target. `universal` is an +internal model only — not a user-facing target until adapters are proven. + +## Drop-in guarantee (non-regression — the hard gate for 2.2.0) + +**With default settings, the builder returns `CLAUDE_PROMPT_SYSTEM_PROMPT` +verbatim (identity path).** Synthesis only happens when settings diverge from +default. This makes the drop-in ironclad and the regression test trivial: + +- `test_prompt_builder_default_is_identity`: default `PromptBuilderSettings` → + generator system prompt `== ffp_config.CLAUDE_PROMPT_SYSTEM_PROMPT`, and the + claude_code deterministic fallback `== force_prompt_shape()` current output. +- Re-run the existing prompt benchmark on the default path; pass rate must not + drop vs 2.1.1. **2.2.0 does not ship if this regresses.** + +Consequently the drop-in defaults are: + +```json +{ + "prompt_builder": { + "target_agent": "claude_code", + "detail_level": "balanced", + "action_mode": "implement", + "structure": "agent_default", + "include_acceptance_criteria": false, + "include_verification": false, + "include_output_format": true, + "preserve_user_constraints": true, + "allow_user_suffix": true, + "user_suffix": "" + } +} +``` + +`include_verification`/`include_acceptance_criteria` default **false** to match +2.1.x. `include_output_format` defaults **true** because the current prompt has +``. Turning verification/acceptance on is an explicit, opt-in +richer prompt (and re-baselines quality for that config only). + +## Staged rollout (adapters) + +Because each new target format multiplies the local generator's failure surface: + +- **2.2.0 ships:** `claude_code` (identity default) + `generic_chat`. Full + `action_mode`, `detail_level`, toggles, capped `user_suffix`, deterministic + preview, dashboard card. +- **2.2.x follow-up (after measurement):** `codex` + `cursor` adapters, each + gated on clearing its own contract eval *with its repair path* before ship. +- `universal` stays internal. + +Log clearly in the UI which targets are available; do not expose codex/cursor +selectors until their adapters land. + +## Agent Differences (unchanged from rev 1 — still accurate) + +Claude Code: explicit direct instructions; for tool use, say the action ("edit", +not "suggest"); emphasize outcome, verification, reference files, measurable +targets, final-answer format; XML sections are a good default but must not force +response-format constraints when the user wants code changes. + +Codex: `AGENTS.md` holds persistent repo guidance; per-task prompts state +objective + constraints + files/patterns + verification commands; Markdown is +enough (no XML); don't duplicate long-standing rules. + +Cursor: User/Team/Project/folder rules + AGENTS.md; reference target files/ +symbols/rules; keep the prompt concise and implementation-oriented (editor +already has context). + +Generic chat: plain Markdown; more context (no persistent repo rules); for code +tasks give a verification expectation rather than "ask clarifying questions". + +Sources: OpenAI Codex AGENTS.md (developers.openai.com/codex/guides/agents-md); +Anthropic prompting best practices +(platform.claude.com/docs/en/build-with-claude/prompt-engineering); +Claude Code prompt library (code.claude.com/docs/en/prompt-library); +Cursor rules (cursor.com/docs/rules). + +## Settings + +Add a `prompt_builder` config block (defaults above). Allowed values: + +| Setting | Values | Purpose | +|---|---|---| +| `target_agent` | `claude_code`, `generic_chat` (2.2.0); `codex`, `cursor` (2.2.x) | Rendering adapter / output format | +| `detail_level` | `concise`, `balanced`, `detailed` | How much context/acceptance detail is generated | +| `action_mode` | `plan`, `implement`, `review`, `debug`, `explain` | Prevents "review" prompts requesting edits, and vice versa | +| `structure` | `agent_default`, `markdown`, `xml`, `checklist` | Override shape without editing the locked system prompt | +| `include_acceptance_criteria` | boolean (default false) | Adds measurable done conditions | +| `include_verification` | boolean (default false) | Adds test/build/run expectation | +| `include_output_format` | boolean (default true) | Adds final response/report expectations | +| `user_suffix` | string, capped ≤ 500 chars | Safe power-user style preferences | + +**`structure` precedence:** `agent_default` → use the adapter's native shape +(claude_code→xml, generic_chat→markdown). An explicit non-default `structure` +overrides the adapter shape; the UI must warn that overriding the native shape +can lower quality on local models (e.g. `target_agent=codex` + `structure=xml` +forces XML the Codex adapter would not otherwise request). + +Never allow arbitrary edits to the built-in prompt-mode system prompt through +dashboard settings. Custom modes remain the escape hatch for fully custom prompts. + +## Intent resolution (code, NOT an LLM call) + +`action_mode` and `target_agent` come from settings. Optional lightweight, +in-process keyword heuristics may *auto-suggest* an action mode (e.g. leading +"review"/"audit" → `review`; "plan"/"design" → `plan`) — deterministic, no model +round-trip, so prompt-mode latency is unchanged. There is **no second LLM call** +for classification (that would double NPU prefill cost and add a failure point). + +## Per-adapter output contracts (machine-checkable — define before building) + +These are the exact validators; "N/10 valid" is meaningless without them. + +- **claude_code** (= existing `check_prompt_contract`): ``, ``, + ``, `` present, in order, non-empty; no `` + residue; not wrapped in a Markdown fence. If `include_verification` on, a + `` section is also required. +- **generic_chat**: Markdown headings `## Task`, `## Context`, `## Constraints` + present; `## Output`/`## Acceptance criteria`/`## Verification` required only + when their toggle is on; not a near-verbatim echo of the input. +- **codex** (2.2.x): `Task:`, `Context:`, `Constraints:` present; `Done when:` + required iff acceptance on; `Final response:` required iff output_format on. +- **cursor** (2.2.x): an imperative action line; a `Focus files/areas:` or + explicit project-rule reference; total length under a concise cap; not an echo. + +All adapters additionally reuse the shared anti-echo check. + +## Implementation Plan + +### Phase 1 — Internal model (`scripts/ffp_prompt_builder.py`) + +- `PromptBuilderSettings`, `PromptIntent`, `PromptRenderResult` dataclasses. +- `build_system_prompt(settings, intent) -> str` — **identity for default + claude_code**; synthesized otherwise. +- `render_fallback(settings, intent, cleaned_input) -> str` — per-adapter + deterministic shape (generalizes today's `force_prompt_shape`). +- `contract_for(target_agent) -> validator` and `validate(text, settings)`. +- Keep user-provided constraints separate from generated constraints + (`preserve_user_constraints`); clamp `user_suffix`. +- Pure/stdlib; no network; unit-testable without a model. + +### Phase 2 — Config (`scripts/ffp_config.py`) + +- Add `prompt_builder` defaults (above). +- `_PATCH_PROMPT_BUILDER_KEYS` + enum validation + `user_suffix` clamp, wired + into `filter_config_patch()`. +- Include in `build_config_snapshot()`. +- Keep `modes.prompt.system_prompt` locked (unchanged). + +Tests: defaults present; invalid enums rejected/normalized; built-in prompt still +unpatchable; `user_suffix` length-capped; **default → identity** (drop-in gate). + +### Phase 3 — Runtime integration (`scripts/ffp_llm_client.py`) + +- Prompt mode pulls the generator system prompt from the builder (identity for + default config). +- `has_prompt_structure()` → `has_target_structure(text, target_agent)`. +- `is_weak_prompt_echo()` uses `has_target_structure` (so a valid Markdown + conversion for generic_chat is not misread as an echo — today's line-168 XML + shortcut would false-positive on non-XML adapters). +- `force_prompt_shape()` and the **shipped repair layer** both become + adapter-parameterized (repair is currently Claude-XML-only). +- Anti-echo preserved across adapters. +- **Gate-fail behavior:** model output fails contract → deterministic repair → + still failing → fall back to the claude_code system prompt (safe default) and + log; never emit an unvalidated shape. + +Tests: claude_code still emits/validates XML; generic_chat validates Markdown and +does not fail for lack of XML; echo/reformat still triggers rescue on every +adapter; gate-fail falls back cleanly. + +### Phase 4 — Dashboard settings (`index.html` + `app.js`) + +- "Prompt builder" card in Config: target agent (only shipped targets), detail + level, action mode, structure, the three include toggles, capped `user_suffix` + textarea. +- **Deterministic preview** button: renders a local sample through + `ffp_prompt_builder` with no LLM call. +- Guardrails: note custom modes remain for fully custom prompts; these are + presets, not raw system-prompt edits; warn when `structure` overrides the + adapter default; safe defaults preselected (claude_code / balanced / implement + / agent_default / verification off). +- DOM via createElement/textContent; no native alert/confirm/prompt (V5). + +### Phase 5 — Eval and quality gates (`tools/prompt_builder_eval.py` — NEW, not provider_bench) + +Keep `provider_bench.py` unchanged (its needle-safe longctx + XML contract just +stabilized and its numbers assume XML). New eval reuses the contract primitives. + +Inputs: review request, implement request, plan-only request, debug request, +generic-chat request (codex/cursor inputs added with those adapters). + +Score per adapter contract: required sections present, action mode preserved +(review ⇒ no edit instruction; plan ⇒ no code edits), no invented constraints, +output format not in conflict with task, not a near-verbatim echo. + +Gates: +- **Drop-in (blocking for 2.2.0):** default claude_code path pass rate ≥ 2.1.1 + baseline (no regression). +- claude_code adapter: ≥ current measured baseline. +- generic_chat adapter: ≥ 8/10 (Markdown is easier for the local model). +- codex / cursor (2.2.x): each ≥ 8/10 **with its repair path** before ship; set + the exact bar when measured. +- No mode may force "exactly N sections" or word limits unless the user asked for + a short answer rather than implementation. + +## Best default behavior + +1. Coding-agent work → implementation prompt, not response-format-only. +2. Review → findings-first, no edit instruction unless user says "fix". +3. Planning → plan-only, forbid code edits. +4. claude_code → XML by default. +5. codex → Markdown task/context/constraints/verify; mention repo `AGENTS.md` + only when project conventions matter. +6. cursor → shorter; reference project rules / target files. +7. Always preserve user-stated constraints; never invent word limits, section + counts, or output formats the user didn't request. +8. Always include verification for implementation/debug prompts *when the + verification toggle is on* (off by default for drop-in; on is the richer + opt-in). + +## Rollout order + +1. Backend `ffp_prompt_builder` + deterministic renderers + per-adapter + contracts + **identity default**. +2. Unit tests incl. the drop-in identity gate. +3. Wire runtime prompt mode to the builder (claude_code + generic_chat); + adapter-aware echo/repair/fallback. +4. Dashboard card + deterministic preview. +5. `tools/prompt_builder_eval.py`; run against the current preferred short-task + provider; confirm no drop-in regression. +6. Ship 2.2.0 (version in the 4 files + CHANGELOG; SPEC `prompt_builder` block + + invariants for the identity guarantee and target-aware validation). +7. Only after measured non-regression: add codex + cursor adapters as 2.2.x. +8. Update `docs/local-llm-provider-benchmark-readme.md` only if measured quality + changes. + +## Resolved open questions (were open in rev 1) + +- **Default target = `claude_code`**, not universal — matches current behavior + and the primary tool; universal-by-default would worsen the drop-in problem. +- **No per-provider target.** `target_agent` = where the user pastes the output + (destination intent); `provider` = which local model generates it. Orthogonal; + do not link them. (A provider-aware *repair* optimization is possible later — + explicitly out of scope for 2.2.0.) +- **Preview is deterministic-only** in 2.2.0 (instant, free, no NPU cost). A + "test with current model" live preview is a later, explicitly opt-in add-on. + +## Risks / notes + +- Highest-regression-risk area: this rewires the *locked, safety-critical* + prompt path characterized by the 2026-07 benchmark. The identity default + + drop-in gate are the guardrail. +- Local 4B generators are the constraint, not the adapters — that is why the + rollout is staged and gates are per-adapter with repair. +- SPEC additions: a `prompt_builder` cfg block; an invariant that default + settings ⇒ `CLAUDE_PROMPT_SYSTEM_PROMPT` identity; an invariant that prompt + validation is target-aware (XML required only for claude_code/xml structure). diff --git a/installer/fastflowprompt.spec b/installer/fastflowprompt.spec index 25209b9..6484617 100644 --- a/installer/fastflowprompt.spec +++ b/installer/fastflowprompt.spec @@ -60,6 +60,7 @@ HIDDEN_IMPORTS = [ "ffp_meetings", "ffp_notifications", "ffp_notify", + "ffp_prompt_builder", "ffp_provider_runtime", "ffp_provider_status", "ffp_pull", diff --git a/installer/installer.iss b/installer/installer.iss index 3488a78..8a8846e 100644 --- a/installer/installer.iss +++ b/installer/installer.iss @@ -41,7 +41,7 @@ #define AppURL "https://github.com/agr77one/Fastflow" #define AppExeName "Flowkey.exe" ; symbolic — actual launchers below ; Keep in lockstep with scripts\_version.py. -#define AppVersion "2.1.1" +#define AppVersion "2.2.0" [Setup] AppId={{8A4F1E6C-9B3D-4E62-9F7A-FASTFLOW140}} diff --git a/installer/sign.ps1 b/installer/sign.ps1 index 0f3b6eb..dbe4c4a 100644 --- a/installer/sign.ps1 +++ b/installer/sign.ps1 @@ -52,7 +52,7 @@ .EXAMPLE # Sign the installer $env:FFP_SIGN_PFX_PASSWORD = "ChangeMe!" - .\sign.ps1 -FilePath ..\out\Flowkey-Setup-2.1.1.exe + .\sign.ps1 -FilePath ..\out\Flowkey-Setup-2.2.0.exe #> [CmdletBinding(DefaultParameterSetName = "Sign")] diff --git a/pyproject.toml b/pyproject.toml index dc00cd4..f66233d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -17,7 +17,7 @@ build-backend = "setuptools.build_meta" [project] name = "fastflowprompt" -version = "2.1.1" +version = "2.2.0" description = "Local-LLM-powered grammar fix, prompt rewrite, chat, and dashboard for Windows." readme = "README.md" requires-python = ">=3.11" @@ -86,6 +86,7 @@ py-modules = [ "ffp_meetings", "ffp_notifications", "ffp_notify", + "ffp_prompt_builder", "ffp_provider_runtime", "ffp_provider_status", "ffp_pull", diff --git a/scripts/_version.py b/scripts/_version.py index 7479418..7c956cf 100644 --- a/scripts/_version.py +++ b/scripts/_version.py @@ -1,3 +1,3 @@ """Single source of truth for the app version. Read by grammar_fix.py.""" -__version__ = "2.1.1" +__version__ = "2.2.0" diff --git a/scripts/ffp_config.py b/scripts/ffp_config.py index ca1c15c..340bb6e 100644 --- a/scripts/ffp_config.py +++ b/scripts/ffp_config.py @@ -12,6 +12,7 @@ from pathlib import Path from urllib.parse import urlparse +import ffp_prompt_builder import paths as _paths log = logging.getLogger("ffp.config") @@ -74,6 +75,7 @@ "chunk_size_chars": 1200, "min_chunk_chars": 700, }, + "prompt_builder": copy.deepcopy(ffp_prompt_builder.DEFAULT_PROMPT_BUILDER_CONFIG), "dictionary": { "protected_words": [], }, @@ -160,6 +162,7 @@ def load_config(config_path: Path) -> dict: merged = copy.deepcopy(DEFAULT_CONFIG) deep_merge(merged, loaded) normalize_llm_config(merged, prefer_legacy=not has_llm_block) + normalize_prompt_builder_config(merged) _enforce_builtin_mode_prompts(merged) return merged @@ -316,6 +319,13 @@ def normalize_llm_config(cfg: dict, *, prefer_legacy: bool = False) -> dict: return cfg +def normalize_prompt_builder_config(cfg: dict) -> dict: + cfg["prompt_builder"] = ffp_prompt_builder.PromptBuilderSettings.from_config( + cfg.get("prompt_builder") + ).to_dict() + return cfg + + _LOOPBACK_HOSTS = frozenset({"127.0.0.1", "localhost", "::1"}) _PATCH_SERVER_KEYS = frozenset({ @@ -343,6 +353,7 @@ def normalize_llm_config(cfg: dict, *, prefer_legacy: bool = False) -> dict: _PATCH_HOTKEYS_KEYS = frozenset({"ask_chat", "capture_note", "grammar_fix", "open_chat"}) _PATCH_TONE_KEYS = frozenset({"preset"}) _PATCH_PROVIDER_PROFILE_KEYS = frozenset({"base_url", "model", "auth_bearer", "timeout_seconds", "auto_start"}) +_PATCH_PROMPT_BUILDER_KEYS = frozenset(ffp_prompt_builder.DEFAULT_PROMPT_BUILDER_CONFIG.keys()) # Notification categories accepted in a patch. Keep in sync with # ffp_notifications.CATEGORIES (a test guards against drift). _PATCH_NOTIF_CATEGORIES = frozenset({ @@ -437,6 +448,17 @@ def _filter_meetings_patch(value: dict) -> dict: out["batch"] = fb return out + +def _filter_prompt_builder_patch(value: dict) -> dict: + raw = {k: v for k, v in value.items() if k in _PATCH_PROMPT_BUILDER_KEYS} + if not raw: + return {} + normalized = ffp_prompt_builder.PromptBuilderSettings.from_config({ + **ffp_prompt_builder.DEFAULT_PROMPT_BUILDER_CONFIG, + **raw, + }).to_dict() + return {k: normalized[k] for k in raw} + # Built-in mode prompts stay locked against patching (system-prompt injection # guard); only tone.preset is patchable among them. User-defined modes accept # label + system_prompt under ids matching _MODE_ID_RE, and a JSON null value @@ -511,6 +533,10 @@ def filter_config_patch(patch: dict) -> dict: filtered = {k: v for k, v in value.items() if k in _PATCH_ROUTING_KEYS} if filtered: out[key] = filtered + elif key == "prompt_builder" and isinstance(value, dict): + filtered = _filter_prompt_builder_patch(value) + if filtered: + out[key] = filtered elif key == "notes" and isinstance(value, dict): filtered = {k: v for k, v in value.items() if k in _PATCH_NOTES_KEYS} if filtered: diff --git a/scripts/ffp_daemon.py b/scripts/ffp_daemon.py index 27ba4c4..41ab9af 100644 --- a/scripts/ffp_daemon.py +++ b/scripts/ffp_daemon.py @@ -318,6 +318,14 @@ def _act_provider_status(_args: dict) -> dict: return grammar_fix.build_config_snapshot()["provider_status"] +def _act_prompt_builder_preview(args: dict) -> dict: + settings = args.get("settings") + return grammar_fix.prompt_builder_preview( + str(args.get("sample") or args.get("value") or ""), + settings if isinstance(settings, dict) else None, + ) + + def _act_models_list(_args: dict) -> dict: return grammar_fix.list_llm_models() @@ -755,6 +763,7 @@ def _act_meeting_week_summary(args: dict) -> dict: "recent_history": _act_recent_history, "config_snapshot": _act_config_snapshot, "provider_status": _act_provider_status, + "prompt_builder_preview": _act_prompt_builder_preview, "models_list": _act_models_list, "models_installed": _act_models_installed, "models_not_installed": _act_models_not_installed, diff --git a/scripts/ffp_llm_client.py b/scripts/ffp_llm_client.py index 38d4133..0ec8364 100644 --- a/scripts/ffp_llm_client.py +++ b/scripts/ffp_llm_client.py @@ -8,6 +8,8 @@ from collections.abc import Callable from dataclasses import dataclass +import ffp_prompt_builder + log = logging.getLogger("ffp.llm") @@ -20,6 +22,7 @@ class LlmRuntimeConfig: routing_cfg: dict protected_words: list[str] modes_cfg: dict + prompt_builder_cfg: dict | None = None def is_prompt_mode(mode: str) -> bool: @@ -158,14 +161,25 @@ def has_prompt_structure(text: str) -> bool: return any(tag in lowered for tag in ("", "", "", "")) -def is_weak_prompt_echo(input_text: str, output_text: str) -> bool: +def has_target_structure( + text: str, + settings: ffp_prompt_builder.PromptBuilderSettings | None = None, +) -> bool: + return ffp_prompt_builder.has_target_structure(text, settings) + + +def is_weak_prompt_echo( + input_text: str, + output_text: str, + settings: ffp_prompt_builder.PromptBuilderSettings | None = None, +) -> bool: """True when the model only grammar-fixed/reformatted or prefixed with 'Prompt:' instead of expanding.""" out = str(output_text or "").strip() inp = str(input_text or "").strip() if not out or not inp: return False - # A real converted prompt carries XML scaffold tags — never weak. - if has_prompt_structure(out): + # A real converted prompt carries the target scaffold — never weak. + if has_target_structure(out, settings): return False # Bare "Prompt: ..." prefix is the classic echo. if re.match(r"^prompt:\s*.+", out.lower()): @@ -207,20 +221,23 @@ def looks_like_prompt_text(text: str) -> bool: return any(marker in lowered for marker in prompt_markers) -def force_prompt_shape(input_text: str) -> str: +def force_prompt_shape( + input_text: str, + settings: ffp_prompt_builder.PromptBuilderSettings | None = None, +) -> str: + settings = settings or ffp_prompt_builder.PromptBuilderSettings() + intent = ffp_prompt_builder.resolve_intent(settings, input_text) cleaned = normalize_output(input_text) - return ( - "\n" - f"Produce a copy-paste-ready Claude prompt for: {cleaned}\n" - "\n" - "\n" - "Use , , and sections; Markdown structure; " - "testable constraints; professional approachable tone; no meta-framing.\n" - "" - ) + return ffp_prompt_builder.render_fallback(settings, intent, cleaned) -def strip_prompt_scaffold_labels(text: str) -> str: +def strip_prompt_scaffold_labels( + text: str, + settings: ffp_prompt_builder.PromptBuilderSettings | None = None, +) -> str: + settings = settings or ffp_prompt_builder.PromptBuilderSettings() + if ffp_prompt_builder.effective_structure(settings) != "xml": + return str(text or "").strip() cleaned = str(text or "") cleaned = re.sub(r"(?im)^\s*\*{0,2}\s*(task|constraints|output format)\s*\*{0,2}\s*:\s*", "", cleaned) cleaned = re.sub(r"(?m)^\s*\*\*\s*$", "", cleaned) @@ -264,12 +281,20 @@ def call_flm( ) -> tuple[str, float, str, str]: mode_cfg = (runtime.modes_cfg or {}).get(mode) or {} system_prompt = str(mode_cfg.get("system_prompt") or "").strip() + prompt_settings = ffp_prompt_builder.PromptBuilderSettings.from_config(runtime.prompt_builder_cfg) + prompt_intent = ffp_prompt_builder.resolve_intent(prompt_settings, input_text) if mode == "tone": preset = str(mode_cfg.get("preset") or "formal").strip().lower() preset_cfg = (mode_cfg.get("presets") or {}).get(preset) or {} preset_prompt = str(preset_cfg.get("system_prompt") or "").strip() if preset_prompt: system_prompt = preset_prompt + if is_prompt_mode(mode): + system_prompt = ffp_prompt_builder.build_system_prompt( + prompt_settings, + prompt_intent, + legacy_system_prompt=system_prompt, + ) if not system_prompt: raise RuntimeError(f"No system_prompt configured for mode '{mode}'.") @@ -365,7 +390,7 @@ def remaining_timeout() -> int: raise RuntimeError("Local LLM returned no usable text.") if is_prompt_mode(mode): - stripped = strip_prompt_scaffold_labels(text) + stripped = strip_prompt_scaffold_labels(text, prompt_settings) if stripped: text = stripped out_norm = re.sub(r"\s+", " ", str(text).lower()).strip() @@ -374,15 +399,10 @@ def remaining_timeout() -> int: near_verbatim = ( (out_norm == in_norm) or (reuse_ratio >= 0.85) - or is_weak_prompt_echo(masked_input, text) + or is_weak_prompt_echo(masked_input, text, prompt_settings) ) if near_verbatim: - anti_echo_prompt = ( - "Rewrite into a Claude-ready prompt with , , and sections. " - "Do not copy the request verbatim or use meta-framing like 'Act as a prompt engineer'. " - "Do not use bare labels like Task: or Constraints: without XML tags. " - "Return only the rewritten prompt text." - ) + anti_echo_prompt = ffp_prompt_builder.build_retry_prompt(prompt_settings, rescue=False) try: retried, retry_model = call_api( model, @@ -401,14 +421,9 @@ def remaining_timeout() -> int: overlap_ratio = word_overlap_ratio(masked_input, text) reuse_ratio = line_reuse_ratio(masked_input, text) near_copy = overlap_ratio >= 0.9 or reuse_ratio >= 0.9 - weak_echo = is_weak_prompt_echo(masked_input, text) - if (near_copy and not has_prompt_structure(text)) or weak_echo: - rescue_prompt = ( - "Rewrite into a stronger Claude-ready prompt for Anthropic models. " - "Use XML sections, testable constraints, and Markdown output format. " - "Do not copy the request verbatim or add meta-commentary. " - "Preserve intent and emoji/smiley symbols. Return only the rewritten prompt text." - ) + weak_echo = is_weak_prompt_echo(masked_input, text, prompt_settings) + if (near_copy and not has_target_structure(text, prompt_settings)) or weak_echo: + rescue_prompt = ffp_prompt_builder.build_retry_prompt(prompt_settings, rescue=True) try: rescued, rescue_model = call_api( model, @@ -421,14 +436,30 @@ def remaining_timeout() -> int: text = rescued model_used = rescue_model else: - text = force_prompt_shape(masked_input) + text = force_prompt_shape(masked_input, prompt_settings) except Exception as exc: log.warning("prompt-rescue call failed, using deterministic prompt shaping: %s", exc) - text = force_prompt_shape(masked_input) + text = force_prompt_shape(masked_input, prompt_settings) - if is_prompt_mode(mode) and is_weak_prompt_echo(masked_input, text): + if is_prompt_mode(mode) and is_weak_prompt_echo(masked_input, text, prompt_settings): log.warning("prompt mode still weak after retries; using deterministic prompt shape") - text = force_prompt_shape(masked_input) + text = force_prompt_shape(masked_input, prompt_settings) + + if is_prompt_mode(mode) and prompt_settings.requires_contract_gate(): + result = ffp_prompt_builder.validate(text, prompt_settings) + if not result.valid: + repaired = ffp_prompt_builder.repair_output(text, prompt_settings) + repaired_result = ffp_prompt_builder.validate(repaired, prompt_settings) + if repaired_result.valid: + text = repaired + else: + log.warning( + "prompt output failed %s/%s contract (%s); using deterministic fallback", + prompt_settings.target_agent, + ffp_prompt_builder.effective_structure(prompt_settings), + ", ".join(result.errors), + ) + text = force_prompt_shape(masked_input, prompt_settings) if not text.strip(): raise RuntimeError("Local LLM returned no usable text.") diff --git a/scripts/ffp_prompt_builder.py b/scripts/ffp_prompt_builder.py new file mode 100644 index 0000000..4b3fdd5 --- /dev/null +++ b/scripts/ffp_prompt_builder.py @@ -0,0 +1,546 @@ +"""Prompt-mode settings, renderers, and output contracts.""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import Any + +TARGET_AGENTS = frozenset({"claude_code", "generic_chat"}) +DETAIL_LEVELS = frozenset({"concise", "balanced", "detailed"}) +ACTION_MODES = frozenset({"plan", "implement", "review", "debug", "explain"}) +STRUCTURES = frozenset({"agent_default", "markdown", "xml", "checklist"}) +USER_SUFFIX_MAX_CHARS = 500 + +DEFAULT_PROMPT_BUILDER_CONFIG = { + "target_agent": "claude_code", + "detail_level": "balanced", + "action_mode": "implement", + "structure": "agent_default", + "include_acceptance_criteria": False, + "include_verification": False, + "include_output_format": True, + "preserve_user_constraints": True, + "allow_user_suffix": True, + "user_suffix": "", +} + + +@dataclass(frozen=True) +class PromptBuilderSettings: + target_agent: str = "claude_code" + detail_level: str = "balanced" + action_mode: str = "implement" + structure: str = "agent_default" + include_acceptance_criteria: bool = False + include_verification: bool = False + include_output_format: bool = True + preserve_user_constraints: bool = True + allow_user_suffix: bool = True + user_suffix: str = "" + + @classmethod + def from_config(cls, value: Any) -> PromptBuilderSettings: + cfg = value if isinstance(value, dict) else {} + defaults = DEFAULT_PROMPT_BUILDER_CONFIG + target_agent = _enum(cfg.get("target_agent"), TARGET_AGENTS, defaults["target_agent"]) + detail_level = _enum(cfg.get("detail_level"), DETAIL_LEVELS, defaults["detail_level"]) + action_mode = _enum(cfg.get("action_mode"), ACTION_MODES, defaults["action_mode"]) + structure = _enum(cfg.get("structure"), STRUCTURES, defaults["structure"]) + allow_suffix = _bool(cfg.get("allow_user_suffix"), defaults["allow_user_suffix"]) + user_suffix = _clean_suffix(cfg.get("user_suffix") if allow_suffix else "") + return cls( + target_agent=target_agent, + detail_level=detail_level, + action_mode=action_mode, + structure=structure, + include_acceptance_criteria=_bool( + cfg.get("include_acceptance_criteria"), + defaults["include_acceptance_criteria"], + ), + include_verification=_bool(cfg.get("include_verification"), defaults["include_verification"]), + include_output_format=_bool(cfg.get("include_output_format"), defaults["include_output_format"]), + preserve_user_constraints=_bool( + cfg.get("preserve_user_constraints"), + defaults["preserve_user_constraints"], + ), + allow_user_suffix=allow_suffix, + user_suffix=user_suffix, + ) + + def to_dict(self) -> dict: + return { + "target_agent": self.target_agent, + "detail_level": self.detail_level, + "action_mode": self.action_mode, + "structure": self.structure, + "include_acceptance_criteria": self.include_acceptance_criteria, + "include_verification": self.include_verification, + "include_output_format": self.include_output_format, + "preserve_user_constraints": self.preserve_user_constraints, + "allow_user_suffix": self.allow_user_suffix, + "user_suffix": self.user_suffix, + } + + def is_identity_default(self) -> bool: + return self.to_dict() == DEFAULT_PROMPT_BUILDER_CONFIG + + def requires_contract_gate(self) -> bool: + return not self.is_identity_default() + + +@dataclass(frozen=True) +class PromptIntent: + action_mode: str = "implement" + + +@dataclass(frozen=True) +class PromptRenderResult: + text: str + target_agent: str + structure: str + valid: bool + errors: tuple[str, ...] = () + + +def resolve_intent(settings: PromptBuilderSettings, input_text: str = "") -> PromptIntent: + del input_text + return PromptIntent(action_mode=settings.action_mode) + + +def effective_structure(settings: PromptBuilderSettings) -> str: + if settings.structure != "agent_default": + return settings.structure + if settings.target_agent == "claude_code": + return "xml" + return "markdown" + + +def build_system_prompt( + settings: PromptBuilderSettings, + intent: PromptIntent, + *, + legacy_system_prompt: str = "", +) -> str: + """Build the generator system prompt. + + The default path intentionally returns the existing Claude prompt verbatim so + upgrades do not silently re-baseline prompt-mode behavior. If that legacy + prompt is missing, return empty and let the caller's missing-prompt guard + fail closed instead of synthesizing a subtly different default. + """ + if settings.is_identity_default(): + return legacy_system_prompt + + shape = effective_structure(settings) + parts = [ + _target_instruction(settings.target_agent), + _action_instruction(intent.action_mode), + _shape_instruction(shape, settings), + _detail_instruction(settings.detail_level), + "Base every constraint on the user's text. Do not invent requirements, word limits, or section counts.", + "Return only the rewritten prompt.", + ] + if settings.preserve_user_constraints: + parts.append("Preserve explicit user constraints and keep them distinguishable from generated guidance.") + if settings.user_suffix: + parts.append(f"User style preference: {settings.user_suffix}") + return " ".join(part for part in parts if part).strip() + + +def build_retry_prompt(settings: PromptBuilderSettings, *, rescue: bool = False) -> str: + shape = effective_structure(settings) + strength = "stronger " if rescue else "" + return ( + f"Rewrite into a {strength}copy-paste-ready prompt. " + f"{_shape_instruction(shape, settings)} " + "Do not copy the request verbatim. Do not use meta-framing or add meta-commentary. " + "Preserve intent. Return only the rewritten prompt." + ) + + +def render_fallback( + settings: PromptBuilderSettings, + intent: PromptIntent, + cleaned_input: str, +) -> str: + cleaned = _normalize_text(cleaned_input) + shape = effective_structure(settings) + if shape == "xml": + text = _render_xml(settings, intent, cleaned) + elif shape == "checklist": + text = _render_checklist(settings, intent, cleaned) + else: + text = _render_markdown(settings, intent, cleaned) + return text.strip() + + +def validate(text: str, settings: PromptBuilderSettings) -> PromptRenderResult: + shape = effective_structure(settings) + if shape == "xml": + valid, errors = _validate_xml(text, settings) + elif shape == "checklist": + valid, errors = _validate_markdown(text, settings, checklist=True) + else: + valid, errors = _validate_markdown(text, settings, checklist=False) + return PromptRenderResult( + text=str(text or "").strip(), + target_agent=settings.target_agent, + structure=shape, + valid=valid, + errors=tuple(errors), + ) + + +def has_target_structure(text: str, settings: PromptBuilderSettings | None = None) -> bool: + settings = settings or PromptBuilderSettings() + lowered = str(text or "").lower() + shape = effective_structure(settings) + if shape == "xml": + return any(tag in lowered for tag in ("", "", "", "")) + if shape == "checklist": + return bool(re.search(r"(?im)^\s*-\s+\[[ x]\]\s+\S+", str(text or ""))) + return all( + re.search(rf"(?im)^\s*##\s+{re.escape(name)}\s*$", str(text or "")) + for name in ("Task", "Context", "Constraints") + ) + + +def repair_output(text: str, settings: PromptBuilderSettings) -> str: + """Repair common near-misses without another model call.""" + raw = str(text or "").strip() + if not raw: + return raw + shape = effective_structure(settings) + if shape == "xml": + return _repair_xml_labels(raw, settings) + if shape in {"markdown", "checklist"}: + return _repair_markdown_labels(raw, settings) + return raw + + +def preview(settings_cfg: Any, sample_text: str) -> dict: + settings = PromptBuilderSettings.from_config(settings_cfg) + intent = resolve_intent(settings, sample_text) + text = render_fallback(settings, intent, sample_text or "Refactor the selected code and verify the change.") + result = validate(text, settings) + return { + "output": text, + "valid": result.valid, + "errors": list(result.errors), + "settings": settings.to_dict(), + "target_agent": result.target_agent, + "structure": result.structure, + } + + +def _enum(value: Any, allowed: frozenset[str], default: str) -> str: + cleaned = str(value or "").strip().lower() + return cleaned if cleaned in allowed else default + + +def _bool(value: Any, default: bool) -> bool: + if value is None: + return default + return bool(value) + + +def _clean_suffix(value: Any) -> str: + cleaned = re.sub(r"\s+", " ", str(value or "").replace("\r", "\n")).strip() + return cleaned[:USER_SUFFIX_MAX_CHARS] + + +def _normalize_text(text: str) -> str: + cleaned = str(text or "").strip() + cleaned = re.sub(r"\s+", " ", cleaned) + return cleaned or "the selected text" + + +def _target_instruction(target_agent: str) -> str: + if target_agent == "generic_chat": + return "Rewrite the user text as a prompt for a generic coding/chat assistant." + return "Rewrite the user text as a Claude Code-ready prompt." + + +def _action_instruction(action_mode: str) -> str: + return { + "plan": "Frame the task as planning only; do not request code edits.", + "implement": "Frame the task as implementation work with a concrete outcome.", + "review": "Frame the task as code review; findings first and no edit request unless the user explicitly asks to fix.", + "debug": "Frame the task as debugging; include reproduction facts, suspected area, and verification.", + "explain": "Frame the task as explanation; focus on clarity and source evidence.", + }.get(action_mode, "Frame the task as implementation work with a concrete outcome.") + + +def _shape_instruction(shape: str, settings: PromptBuilderSettings) -> str: + if shape == "xml": + sections = ["", "", ""] + if settings.include_acceptance_criteria: + sections.append("") + if settings.include_verification: + sections.append("") + if settings.include_output_format: + sections.append("") + return "Use XML sections in this order: " + ", ".join(sections) + "." + if shape == "checklist": + return "Use Markdown headings with concise checklist bullets." + sections = ["## Task", "## Context", "## Constraints"] + if settings.include_acceptance_criteria: + sections.append("## Acceptance criteria") + if settings.include_verification: + sections.append("## Verification") + if settings.include_output_format: + sections.append("## Output") + return "Use Markdown headings in this order: " + ", ".join(sections) + "." + + +def _detail_instruction(detail_level: str) -> str: + if detail_level == "concise": + return "Keep sections terse: one or two lines each." + if detail_level == "detailed": + return "Include enough concrete detail for a coding agent to act without guessing." + return "Keep sections short but specific." + + +def _render_xml(settings: PromptBuilderSettings, intent: PromptIntent, cleaned: str) -> str: + if settings.is_identity_default(): + return ( + "\n" + f"Produce a copy-paste-ready Claude prompt for: {cleaned}\n" + "\n" + "\n" + "Use , , and sections; Markdown structure; " + "testable constraints; professional approachable tone; no meta-framing.\n" + "" + ) + + sections = [ + ("task", _fallback_task(settings, intent, cleaned)), + ("context", f"Source request: {cleaned}"), + ("constraints", _fallback_constraints(settings)), + ] + if settings.include_acceptance_criteria: + sections.append(("acceptance_criteria", _fallback_acceptance(intent))) + if settings.include_verification: + sections.append(("verification", _fallback_verification(intent))) + if settings.include_output_format: + sections.append(("output_format", _fallback_output(settings))) + return "\n".join(f"<{name}>\n{body}\n" for name, body in sections) + + +def _render_markdown(settings: PromptBuilderSettings, intent: PromptIntent, cleaned: str) -> str: + sections = [ + ("Task", _fallback_task(settings, intent, cleaned)), + ("Context", f"Source request: {cleaned}"), + ("Constraints", _fallback_constraints(settings)), + ] + if settings.include_acceptance_criteria: + sections.append(("Acceptance criteria", _fallback_acceptance(intent))) + if settings.include_verification: + sections.append(("Verification", _fallback_verification(intent))) + if settings.include_output_format: + sections.append(("Output", _fallback_output(settings))) + return "\n\n".join(f"## {name}\n{body}" for name, body in sections) + + +def _render_checklist(settings: PromptBuilderSettings, intent: PromptIntent, cleaned: str) -> str: + base = _render_markdown(settings, intent, cleaned) + return re.sub(r"(?m)^(?!##)(.+)$", lambda m: "- [ ] " + m.group(1) if m.group(1).strip() else "", base) + + +def _fallback_task(settings: PromptBuilderSettings, intent: PromptIntent, cleaned: str) -> str: + target = "Claude Code" if settings.target_agent == "claude_code" else "a coding assistant" + action = { + "plan": "Create an execution plan", + "implement": "Implement the requested change", + "review": "Review the requested code or PR", + "debug": "Debug the reported issue", + "explain": "Explain the selected material", + }.get(intent.action_mode, "Implement the requested change") + return f"{action} as a copy-paste-ready prompt for {target}: {cleaned}" + + +def _fallback_constraints(settings: PromptBuilderSettings) -> str: + pieces = [ + "Preserve all user-stated requirements.", + "Do not invent unstated constraints.", + ] + if settings.detail_level == "concise": + pieces.append("Keep the prompt concise.") + elif settings.detail_level == "detailed": + pieces.append("Include relevant files, risks, and assumptions when present.") + if settings.user_suffix: + pieces.append(settings.user_suffix) + return " ".join(pieces) + + +def _fallback_acceptance(intent: PromptIntent) -> str: + if intent.action_mode == "review": + return "Findings are specific, severity-ranked, and tied to evidence." + if intent.action_mode == "plan": + return "The plan is ordered, actionable, and avoids code changes." + return "The requested outcome is implemented and edge cases are addressed." + + +def _fallback_verification(intent: PromptIntent) -> str: + if intent.action_mode == "plan": + return "List verification commands or checks to run after implementation." + if intent.action_mode == "review": + return "Mention tests or checks reviewed, and any remaining gaps." + return "Run the relevant tests or commands and report the result." + + +def _fallback_output(settings: PromptBuilderSettings) -> str: + if settings.target_agent == "generic_chat": + return "Return a concise Markdown answer with the requested deliverable first." + return "Return only the final prompt text with no preamble." + + +def _validate_xml(text: str, settings: PromptBuilderSettings) -> tuple[bool, list[str]]: + raw = str(text or "").strip() + errors: list[str] = [] + if settings.is_identity_default(): + lower = raw.lower() + for tag in ("task", "output_format"): + if f"<{tag}>" not in lower: + errors.append(f"missing <{tag}>") + if re.search(r"(?im)^\s*```", raw): + errors.append("wrapped in code fence") + if "" + close_tag = f"" + pos = lower.find(open_tag) + if pos < 0 or close_tag not in lower: + errors.append(f"missing {open_tag}") + continue + if pos < last_pos: + errors.append(f"{open_tag} out of order") + last_pos = pos + body = re.search(rf"<{tag}>\s*(.*?)\s*", raw, flags=re.IGNORECASE | re.DOTALL) + if not body or not body.group(1).strip(): + errors.append(f"empty {open_tag}") + if re.search(r"(?im)^\s*```", raw): + errors.append("wrapped in code fence") + if " tuple[bool, list[str]]: + raw = str(text or "").strip() + errors: list[str] = [] + required = ["Task", "Context", "Constraints"] + if settings.include_acceptance_criteria: + required.append("Acceptance criteria") + if settings.include_verification: + required.append("Verification") + if settings.include_output_format: + required.append("Output") + last_pos = -1 + for name in required: + match = re.search(rf"(?im)^\s*##\s+{re.escape(name)}\s*$", raw) + if not match: + errors.append(f"missing ## {name}") + continue + if match.start() < last_pos: + errors.append(f"## {name} out of order") + last_pos = match.start() + if checklist and not re.search(r"(?im)^\s*-\s+\[[ x]\]\s+\S+", raw): + errors.append("missing checklist bullets") + if re.search(r"(?im)^\s*```", raw): + errors.append("wrapped in code fence") + if " str: + label_map = { + "task": "task", + "context": "context", + "constraints": "constraints", + "acceptance criteria": "acceptance_criteria", + "verification": "verification", + "output": "output_format", + "output format": "output_format", + } + sections = _extract_label_sections(raw, label_map) + if not sections: + return raw + order = ["task", "context", "constraints"] + if settings.include_acceptance_criteria: + order.append("acceptance_criteria") + if settings.include_verification: + order.append("verification") + if settings.include_output_format: + order.append("output_format") + rendered = [] + for name in order: + body = sections.get(name) + if body: + rendered.append(f"<{name}>\n{body.strip()}\n") + return "\n".join(rendered) if rendered else raw + + +def _repair_markdown_labels(raw: str, settings: PromptBuilderSettings) -> str: + label_map = { + "task": "Task", + "context": "Context", + "constraints": "Constraints", + "acceptance criteria": "Acceptance criteria", + "verification": "Verification", + "output": "Output", + "output format": "Output", + } + sections = _extract_label_sections(raw, label_map) + if not sections: + return raw + order = ["Task", "Context", "Constraints"] + if settings.include_acceptance_criteria: + order.append("Acceptance criteria") + if settings.include_verification: + order.append("Verification") + if settings.include_output_format: + order.append("Output") + rendered = [] + for name in order: + body = sections.get(name) + if body: + rendered.append(f"## {name}\n{body.strip()}") + return "\n\n".join(rendered) if rendered else raw + + +def _extract_label_sections(raw: str, label_map: dict[str, str]) -> dict[str, str]: + pattern = re.compile( + r"(?im)^\s*(?:#{1,3}\s*)?\*{0,2}(" + + "|".join(re.escape(k) for k in sorted(label_map, key=len, reverse=True)) + + r")\*{0,2}\s*:\s*" + ) + matches = list(pattern.finditer(raw)) + sections: dict[str, str] = {} + for idx, match in enumerate(matches): + key = label_map[match.group(1).lower()] + start = match.end() + end = matches[idx + 1].start() if idx + 1 < len(matches) else len(raw) + body = raw[start:end].strip() + if body: + sections[key] = body + return sections diff --git a/scripts/grammar_fix.py b/scripts/grammar_fix.py index efe24c1..443fb31 100644 --- a/scripts/grammar_fix.py +++ b/scripts/grammar_fix.py @@ -23,6 +23,7 @@ import ffp_llm_client import ffp_meetings import ffp_notifications +import ffp_prompt_builder import ffp_provider_runtime import ffp_provider_status import ffp_telemetry @@ -70,7 +71,7 @@ def refresh_runtime_config() -> None: global HISTORY_PATH, HISTORY_STORE_TEXT, SERVER_CFG, SERVER_AUTO_START global SERVER_PERFORMANCE_MODE, SERVER_STARTUP_TIMEOUT_SECONDS global SERVER_EXTRA_ARGS, SERVER_LOG_TO_FILE, SERVER_LOG_FILE - global ROUTING_CFG, DICT_CFG, PROTECTED_WORDS + global ROUTING_CFG, PROMPT_BUILDER_CFG, DICT_CFG, PROTECTED_WORDS CONFIG = load_config() ENABLED = bool(CONFIG.get("enabled", True)) @@ -113,6 +114,7 @@ def refresh_runtime_config() -> None: SERVER_LOG_TO_FILE = bool(SERVER_CFG.get("log_to_file", True)) SERVER_LOG_FILE = str(SERVER_CFG.get("log_file") or "flm_server.log") ROUTING_CFG = CONFIG.get("routing") or {} + PROMPT_BUILDER_CFG = CONFIG.get("prompt_builder") or {} DICT_CFG = CONFIG.get("dictionary") or {} PROTECTED_WORDS = [str(w) for w in (DICT_CFG.get("protected_words") or []) if str(w).strip()] @@ -441,6 +443,7 @@ def _select_runtime(mode: str, input_text: str) -> tuple[str, int, str]: routing_cfg=ROUTING_CFG, protected_words=PROTECTED_WORDS, modes_cfg=CONFIG.get("modes") or {}, + prompt_builder_cfg=PROMPT_BUILDER_CFG, ) return ffp_llm_client.select_runtime(runtime, mode, input_text) @@ -508,6 +511,7 @@ def call_flm(mode: str, input_text: str) -> tuple[str, float, str, str]: routing_cfg=ROUTING_CFG, protected_words=PROTECTED_WORDS, modes_cfg=CONFIG.get("modes") or {}, + prompt_builder_cfg=PROMPT_BUILDER_CFG, ) return ffp_llm_client.call_flm( runtime, @@ -735,6 +739,14 @@ def apply_config_patch(patch: dict) -> str: return "ok" +def prompt_builder_preview(sample_text: str = "", settings_cfg: dict | None = None) -> dict: + cfg = load_config() + return ffp_prompt_builder.preview( + settings_cfg if isinstance(settings_cfg, dict) else (cfg.get("prompt_builder") or {}), + sample_text or "Refactor the selected code and verify the change.", + ) + + def build_config_snapshot() -> dict: """Build the live config snapshot consumed by the AHK dashboard. @@ -747,6 +759,7 @@ def build_config_snapshot() -> dict: llm_cfg = cfg.get("llm") or {} notes_cfg = cfg.get("notes") or {} routing_cfg = cfg.get("routing") or {} + prompt_builder_cfg = cfg.get("prompt_builder") or {} hotkeys_cfg = cfg.get("hotkeys") or {} tone_cfg = ((cfg.get("modes") or {}).get("tone") or {}) server_cfg = cfg.get("server") or {} @@ -810,6 +823,9 @@ def build_config_snapshot() -> dict: "chunk_size_chars": int(routing_cfg.get("chunk_size_chars") or 1200), "min_chunk_chars": int(routing_cfg.get("min_chunk_chars") or 700), }, + "prompt_builder": ffp_prompt_builder.PromptBuilderSettings.from_config( + prompt_builder_cfg + ).to_dict(), "notes": { "vault_dir": str(notes_cfg.get("vault_dir") or r"%USERPROFILE%\Documents\FastFlowPrompt Notes"), "categories": list(notes_cfg.get("categories") or []), @@ -890,6 +906,15 @@ def handle_server_cli() -> bool: if action == "provider_status": print(json.dumps(build_config_snapshot()["provider_status"], ensure_ascii=False)) return True + if action == "prompt_builder_preview": + value = "" + if "--value" in args: + vidx = args.index("--value") + if vidx + 1 >= len(args): + raise RuntimeError("Missing value for --value.") + value = str(args[vidx + 1]) + print(json.dumps(prompt_builder_preview(value), ensure_ascii=False)) + return True if action == "models_list": print(json.dumps(list_llm_models(), ensure_ascii=False)) return True diff --git a/scripts/ui/web/app.js b/scripts/ui/web/app.js index 380d9be..177d1ff 100644 --- a/scripts/ui/web/app.js +++ b/scripts/ui/web/app.js @@ -92,6 +92,18 @@ function isValidHotkey(hk) { const PERF_LABELS = { balanced: "🟡 Balanced", max: "🔴 Max throughput" }; const TONE_LABELS = { formal: "🎩 Formal", casual: "👕 Casual", friendly: "🤝 Friendly" }; +const PROMPT_BUILDER_DEFAULTS = { + target_agent: "claude_code", + detail_level: "balanced", + action_mode: "implement", + structure: "agent_default", + include_acceptance_criteria: false, + include_verification: false, + include_output_format: true, + preserve_user_constraints: true, + allow_user_suffix: true, + user_suffix: "", +}; // ---- LLM providers ----------------------------------------------------------- // The daemon resolves the *effective* provider (configured one, with fallback @@ -552,6 +564,64 @@ async function saveNotes() { // ---- Config ---------------------------------------------------------------- +function populatePromptBuilder(pb) { + const cfg = { ...PROMPT_BUILDER_DEFAULTS, ...(pb || {}) }; + $("pb-target").value = cfg.target_agent; + $("pb-action").value = cfg.action_mode; + $("pb-detail").value = cfg.detail_level; + $("pb-structure").value = cfg.structure; + $("pb-acceptance").checked = !!cfg.include_acceptance_criteria; + $("pb-verification").checked = !!cfg.include_verification; + $("pb-output").checked = cfg.include_output_format !== false; + $("pb-preserve").checked = cfg.preserve_user_constraints !== false; + $("pb-allow-suffix").checked = cfg.allow_user_suffix !== false; + $("pb-suffix").value = cfg.user_suffix || ""; + $("pb-suffix").disabled = !$("pb-allow-suffix").checked; + updatePromptBuilderHint(); +} + +function promptBuilderPatch() { + const suffix = $("pb-allow-suffix").checked ? $("pb-suffix").value.trim().slice(0, 500) : ""; + return { + target_agent: $("pb-target").value || "claude_code", + detail_level: $("pb-detail").value || "balanced", + action_mode: $("pb-action").value || "implement", + structure: $("pb-structure").value || "agent_default", + include_acceptance_criteria: $("pb-acceptance").checked, + include_verification: $("pb-verification").checked, + include_output_format: $("pb-output").checked, + preserve_user_constraints: $("pb-preserve").checked, + allow_user_suffix: $("pb-allow-suffix").checked, + user_suffix: suffix, + }; +} + +function updatePromptBuilderHint() { + const override = ($("pb-structure").value || "agent_default") !== "agent_default"; + $("pb-structure-note").hidden = !override; + $("pb-suffix").disabled = !$("pb-allow-suffix").checked; + const remaining = 500 - ($("pb-suffix").value || "").length; + setStatus("pb-status", remaining < 0 ? "Suffix is over 500 characters; save will trim it." : ""); +} + +async function previewPromptBuilder() { + const preview = $("pb-preview"); + preview.hidden = true; + setStatus("pb-status", "Rendering preview…"); + try { + const result = await action("prompt_builder_preview", { + sample: "Refactor the selected code, fix edge cases, and run the relevant tests.", + settings: promptBuilderPatch(), + }); + preview.textContent = result.output || ""; + preview.hidden = false; + const status = result.valid ? `Preview: ${result.target_agent}/${result.structure}` : `Preview has issues: ${(result.errors || []).join(", ")}`; + setStatus("pb-status", status, !!result.valid); + } catch (e) { + setStatus("pb-status", `Preview failed: ${e.message}`, false); + } +} + // ---- Custom modes ------------------------------------------------------------ let customModes = {}; // id -> {label, system_prompt} (non-builtin only) @@ -656,6 +726,7 @@ async function loadConfig() { $("cfg-long-thr").value = routing.long_threshold_chars ?? 1400; $("cfg-chunk-size").value = routing.chunk_size_chars ?? 1200; $("cfg-min-chunk").value = routing.min_chunk_chars ?? 700; + populatePromptBuilder(cfg.prompt_builder || {}); const tone = (cfg.tone || {}).preset || "formal"; document.querySelectorAll('input[name="tone"]').forEach((r) => (r.checked = r.value === tone)); populateNotifications(cfg.notifications || {}); @@ -866,6 +937,7 @@ async function saveConfig() { chunk_size_chars: Number($("cfg-chunk-size").value) || 1200, min_chunk_chars: Number($("cfg-min-chunk").value) || 700, }, + prompt_builder: promptBuilderPatch(), modes: { tone: { preset: tone ? tone.value : "formal" } }, hotkeys, notifications: notificationsPatch(), @@ -1603,6 +1675,10 @@ document.addEventListener("DOMContentLoaded", () => { $("cm-select").addEventListener("change", fillCustomModeForm); $("cm-save").addEventListener("click", saveCustomMode); $("cm-delete").addEventListener("click", deleteCustomMode); + $("pb-preview-btn").addEventListener("click", previewPromptBuilder); + $("pb-structure").addEventListener("change", updatePromptBuilderHint); + $("pb-allow-suffix").addEventListener("change", updatePromptBuilderHint); + $("pb-suffix").addEventListener("input", updatePromptBuilderHint); $("model-set-active").addEventListener("click", setActiveModel); $("model-remove").addEventListener("click", removeModel); $("pull-btn").addEventListener("click", pullModel); diff --git a/scripts/ui/web/index.html b/scripts/ui/web/index.html index 26b0372..8afc3ef 100644 --- a/scripts/ui/web/index.html +++ b/scripts/ui/web/index.html @@ -343,6 +343,56 @@

Tone preset (tone: prefix)

+
+

Prompt builder

+

Controls the prompt: output shape without editing the locked built-in system prompt. Custom modes remain the fully custom escape hatch.

+
+ + +
+
+ + +
+
+ + +
+
+ + +
+ + + + + + + +
+ + +
+ +

Custom modes

Your own prefix: commands for the grammar hotkey — the mode id is the prefix (e.g. id translate → select text, prefix with translate:). Built-in modes aren't editable here.

diff --git a/setup/defaults/grammar_hotkey.config.example.json b/setup/defaults/grammar_hotkey.config.example.json index 42c56d9..108761d 100644 --- a/setup/defaults/grammar_hotkey.config.example.json +++ b/setup/defaults/grammar_hotkey.config.example.json @@ -27,6 +27,18 @@ "chunk_size_chars": 1200, "min_chunk_chars": 700 }, + "prompt_builder": { + "target_agent": "claude_code", + "detail_level": "balanced", + "action_mode": "implement", + "structure": "agent_default", + "include_acceptance_criteria": false, + "include_verification": false, + "include_output_format": true, + "preserve_user_constraints": true, + "allow_user_suffix": true, + "user_suffix": "" + }, "dictionary": { "protected_words": [] }, diff --git a/setup/defaults/grammar_hotkey.config.json b/setup/defaults/grammar_hotkey.config.json index 64f70eb..a22fb3f 100644 --- a/setup/defaults/grammar_hotkey.config.json +++ b/setup/defaults/grammar_hotkey.config.json @@ -27,6 +27,18 @@ "chunk_size_chars": 1200, "min_chunk_chars": 700 }, + "prompt_builder": { + "target_agent": "claude_code", + "detail_level": "balanced", + "action_mode": "implement", + "structure": "agent_default", + "include_acceptance_criteria": false, + "include_verification": false, + "include_output_format": true, + "preserve_user_constraints": true, + "allow_user_suffix": true, + "user_suffix": "" + }, "dictionary": { "protected_words": [] }, diff --git a/tests/test_ffp_config.py b/tests/test_ffp_config.py index 18f4954..5e847bd 100644 --- a/tests/test_ffp_config.py +++ b/tests/test_ffp_config.py @@ -4,6 +4,7 @@ from pathlib import Path import ffp_config +import ffp_prompt_builder import pytest @@ -275,6 +276,49 @@ def test_filter_config_patch_accepts_provider_profiles(): } +def test_default_config_has_prompt_builder_defaults(): + assert ffp_config.DEFAULT_CONFIG["prompt_builder"] == ffp_prompt_builder.DEFAULT_PROMPT_BUILDER_CONFIG + + +def test_load_config_normalizes_prompt_builder(tmp_path): + cfg_path = tmp_path / "grammar_hotkey.config.json" + cfg_path.write_text( + json.dumps({ + "prompt_builder": { + "target_agent": "codex", + "detail_level": "very long", + "user_suffix": "x" * 700, + } + }), + encoding="utf-8", + ) + + loaded = ffp_config.load_config(cfg_path) + + assert loaded["prompt_builder"]["target_agent"] == "claude_code" + assert loaded["prompt_builder"]["detail_level"] == "balanced" + assert len(loaded["prompt_builder"]["user_suffix"]) == ffp_prompt_builder.USER_SUFFIX_MAX_CHARS + + +def test_filter_config_patch_prompt_builder_partial_and_clamped(): + filtered = ffp_config.filter_config_patch({ + "prompt_builder": { + "target_agent": "generic_chat", + "include_verification": 1, + "user_suffix": "x" * 700, + "system_prompt": "evil", + } + }) + + assert filtered == { + "prompt_builder": { + "target_agent": "generic_chat", + "include_verification": True, + "user_suffix": "x" * ffp_prompt_builder.USER_SUFFIX_MAX_CHARS, + } + } + + def test_builtin_mode_prompts_never_mention_emoji(): # Regression: prompts that told the model to "preserve emoji" got parroted # into outputs as invented emoji instructions (B: small models echo any diff --git a/tests/test_ffp_daemon.py b/tests/test_ffp_daemon.py index 5882415..380083f 100644 --- a/tests/test_ffp_daemon.py +++ b/tests/test_ffp_daemon.py @@ -75,8 +75,8 @@ def test_actions_count_and_expected_names(daemon_module): # meeting_digests_list/meeting_process/meeting_batch_run/meeting_batch_status/ # meeting_ask -> 69; meeting_overview -> 70; meeting_actions_list / # meeting_action_set_status / meeting_week_summary -> 73; meeting_redigest - # (strict re-run of a digest) -> 74. - assert len(daemon_module.ACTIONS) == 74 + # (strict re-run of a digest) -> 74; prompt_builder_preview -> 75. + assert len(daemon_module.ACTIONS) == 75 for a in ("chat_threads_list", "chat_thread_get", "chat_send", "chat_thread_delete", "chat_stage_selection", "chat_take_staged", "note_get", "note_move", "note_delete", @@ -85,10 +85,12 @@ def test_actions_count_and_expected_names(daemon_module): "meeting_digests_list", "meeting_process", "meeting_batch_run", "meeting_batch_status", "meeting_ask", "meeting_overview", "meeting_actions_list", "meeting_action_set_status", "meeting_week_summary", - "meeting_redigest"): + "meeting_redigest", "prompt_builder_preview"): assert a in daemon_module.ACTIONS # notify_gate writes the log + dedupe state, so it must be a WRITE action. assert "notify_gate" in daemon_module._WRITE_ACTIONS + # prompt_builder_preview is deterministic and read-only. + assert "prompt_builder_preview" not in daemon_module._WRITE_ACTIONS # The meeting actions manage their own locking + write a separate file, so # they are intentionally NOT under the global config write-lock (a long batch # must not block config saves / notifications). diff --git a/tests/test_grammar_fix.py b/tests/test_grammar_fix.py index c18c88c..015f7c6 100644 --- a/tests/test_grammar_fix.py +++ b/tests/test_grammar_fix.py @@ -522,6 +522,40 @@ def fake_call(model, system_prompt, user_content, max_tokens, timeout_seconds): assert "" in text +def test_call_flm_prompt_generic_chat_falls_back_to_markdown(fresh_modules, monkeypatch): + grammar_fix = fresh_modules("grammar_fix") + grammar_fix.CONFIG["prompt_builder"] = { + "target_agent": "generic_chat", + "detail_level": "balanced", + "action_mode": "implement", + "structure": "agent_default", + "include_acceptance_criteria": True, + "include_verification": True, + "include_output_format": True, + "preserve_user_constraints": True, + "allow_user_suffix": True, + "user_suffix": "", + } + grammar_fix.PROMPT_BUILDER_CFG = grammar_fix.CONFIG["prompt_builder"] + monkeypatch.setattr(grammar_fix, "is_flm_server_reachable", lambda: True) + prompts = [] + + def fake_call(model, system_prompt, user_content, max_tokens, timeout_seconds): + prompts.append(system_prompt) + return (user_content, grammar_fix.FLM_MODEL) + + monkeypatch.setattr(grammar_fix, "_call_flm_api", fake_call) + + text, _, _, _ = grammar_fix.call_flm("prompt", "fix the failing config test") + + assert "generic coding/chat assistant" in prompts[0] + assert text.startswith("## Task") + settings = grammar_fix.ffp_prompt_builder.PromptBuilderSettings.from_config( + grammar_fix.PROMPT_BUILDER_CFG + ) + assert grammar_fix.ffp_prompt_builder.validate(text, settings).valid is True + + def test_is_weak_prompt_echo_flags_multiline_reformat(fresh_modules): grammar_fix = fresh_modules("grammar_fix") llm = grammar_fix.ffp_llm_client diff --git a/tests/test_prompt_builder.py b/tests/test_prompt_builder.py new file mode 100644 index 0000000..cb1723f --- /dev/null +++ b/tests/test_prompt_builder.py @@ -0,0 +1,110 @@ +from __future__ import annotations + +import ffp_config +import ffp_llm_client +import ffp_prompt_builder + + +def test_default_system_prompt_is_identity(): + settings = ffp_prompt_builder.PromptBuilderSettings.from_config({}) + intent = ffp_prompt_builder.resolve_intent(settings, "fix the PR") + + assert ffp_prompt_builder.build_system_prompt( + settings, + intent, + legacy_system_prompt=ffp_config.CLAUDE_PROMPT_SYSTEM_PROMPT, + ) == ffp_config.CLAUDE_PROMPT_SYSTEM_PROMPT + assert ffp_llm_client.force_prompt_shape("fix the PR") == ffp_prompt_builder.render_fallback( + settings, + intent, + "fix the PR", + ) + assert ffp_prompt_builder.preview({}, "fix the PR")["valid"] is True + + +def test_default_system_prompt_without_legacy_fails_closed(): + settings = ffp_prompt_builder.PromptBuilderSettings.from_config({}) + intent = ffp_prompt_builder.resolve_intent(settings, "fix the PR") + + assert ffp_prompt_builder.build_system_prompt( + settings, + intent, + legacy_system_prompt="", + ) == "" + + +def test_generic_chat_markdown_contract_and_preview(): + settings = ffp_prompt_builder.PromptBuilderSettings.from_config({ + "target_agent": "generic_chat", + "include_acceptance_criteria": True, + "include_verification": True, + }) + intent = ffp_prompt_builder.resolve_intent(settings, "review the merge conflict") + + text = ffp_prompt_builder.render_fallback(settings, intent, "review the merge conflict") + result = ffp_prompt_builder.validate(text, settings) + + assert result.valid is True + assert "## Task" in text + assert "## Context" in text + assert "## Constraints" in text + assert "## Acceptance criteria" in text + assert "## Verification" in text + assert ffp_prompt_builder.has_target_structure(text, settings) is True + + +def test_fallback_shape_passes_its_own_contract(): + for cfg in ( + {"target_agent": "claude_code", "action_mode": "review"}, + {"target_agent": "generic_chat", "include_verification": True}, + ): + settings = ffp_prompt_builder.PromptBuilderSettings.from_config(cfg) + intent = ffp_prompt_builder.resolve_intent(settings, "sample request") + shaped = ffp_prompt_builder.render_fallback(settings, intent, "sample request") + + assert ffp_prompt_builder.validate(shaped, settings).valid, cfg + + +def test_generic_markdown_conversion_not_flagged_as_echo(): + settings = ffp_prompt_builder.PromptBuilderSettings.from_config({"target_agent": "generic_chat"}) + good_md = "## Task\nBuild X.\n## Context\n- repo Y\n## Constraints\n- keep scoped" + + assert ffp_llm_client.is_weak_prompt_echo("build x for repo y", good_md, settings) is False + assert ffp_llm_client.is_weak_prompt_echo("build x for repo y", "build x for repo y", settings) is True + + +def test_action_mode_review_and_plan_do_not_order_edits(): + for mode in ("review", "plan"): + settings = ffp_prompt_builder.PromptBuilderSettings.from_config({ + "target_agent": "generic_chat", + "action_mode": mode, + }) + text = ffp_prompt_builder.render_fallback( + settings, + ffp_prompt_builder.resolve_intent(settings, "the PR"), + "the PR", + ) + + assert "Implement the requested change" not in text + + +def test_prompt_builder_clamps_suffix_and_rejects_unshipped_targets(): + settings = ffp_prompt_builder.PromptBuilderSettings.from_config({ + "target_agent": "codex", + "user_suffix": "x" * 700, + }) + + assert settings.target_agent == "claude_code" + assert len(settings.user_suffix) == ffp_prompt_builder.USER_SUFFIX_MAX_CHARS + + +def test_repair_label_output_to_generic_markdown(): + settings = ffp_prompt_builder.PromptBuilderSettings.from_config({"target_agent": "generic_chat"}) + + repaired = ffp_prompt_builder.repair_output( + "Task: Fix the bug\nContext: API fails\nConstraints: Keep scope small\nOutput: Summary", + settings, + ) + + assert ffp_prompt_builder.validate(repaired, settings).valid is True + assert repaired.startswith("## Task") diff --git a/tools/prompt_builder_eval.py b/tools/prompt_builder_eval.py new file mode 100644 index 0000000..efb3c03 --- /dev/null +++ b/tools/prompt_builder_eval.py @@ -0,0 +1,329 @@ +"""Contract eval for prompt-builder renderers and live local models.""" + +from __future__ import annotations + +import argparse +import json +import sys +import time +import urllib.request +from pathlib import Path +from typing import Any + +ROOT = Path(__file__).resolve().parents[1] +SCRIPTS_DIR = ROOT / "scripts" +if str(SCRIPTS_DIR) not in sys.path: + sys.path.insert(0, str(SCRIPTS_DIR)) + +import ffp_config # noqa: E402 +import ffp_llm_client # noqa: E402 +import ffp_prompt_builder # noqa: E402 + +CASES = [ + { + "name": "claude_default_review_pr", + "input": "Review the LPS-RCV-5050 PR and fix merge issues.", + "settings": {}, + "must_contain": ["", ""], + }, + { + "name": "claude_default_plan_dashboard", + "input": "Create a plan to refine prompt logic for the FastFlow app.", + "settings": {}, + "must_contain": ["", ""], + }, + { + "name": "generic_01_implement_full", + "input": "Implement prompt-builder settings and run tests.", + "settings": { + "target_agent": "generic_chat", + "action_mode": "implement", + "include_acceptance_criteria": True, + "include_verification": True, + }, + "must_contain": ["## Task", "## Context", "## Constraints", "## Verification"], + "must_not_contain": ["Planning only"], + }, + { + "name": "generic_02_review_no_edit_order", + "input": "Review this PR for regressions and missing tests.", + "settings": { + "target_agent": "generic_chat", + "action_mode": "review", + "include_acceptance_criteria": True, + "include_output_format": True, + }, + "must_contain": ["## Task", "## Context", "## Constraints"], + "must_not_contain": ["Implement the requested change"], + }, + { + "name": "generic_03_plan_no_code_edits", + "input": "Plan the dashboard prompt settings work.", + "settings": { + "target_agent": "generic_chat", + "action_mode": "plan", + "include_acceptance_criteria": True, + "include_verification": True, + }, + "must_contain": ["## Task", "## Context", "## Constraints"], + "must_not_contain": ["Implement the requested change"], + }, + { + "name": "generic_04_debug_verification", + "input": "Debug why prompt mode echoes the input.", + "settings": { + "target_agent": "generic_chat", + "action_mode": "debug", + "include_verification": True, + }, + "must_contain": ["## Task", "## Verification"], + }, + { + "name": "generic_05_explain_concise", + "input": "Explain the provider routing logic in plain English.", + "settings": { + "target_agent": "generic_chat", + "action_mode": "explain", + "detail_level": "concise", + }, + "must_contain": ["## Task", "## Context", "## Constraints"], + "must_not_contain": ["Implement the requested change"], + }, + { + "name": "generic_06_basic_default", + "input": "Turn this rough note into a clear coding-agent prompt.", + "settings": { + "target_agent": "generic_chat", + }, + "must_contain": ["## Task", "## Context", "## Constraints"], + }, + { + "name": "generic_07_review_with_verification", + "input": "Review the config patch path and say what tests prove it.", + "settings": { + "target_agent": "generic_chat", + "action_mode": "review", + "include_verification": True, + }, + "must_contain": ["## Task", "## Verification"], + "must_not_contain": ["Implement the requested change"], + }, + { + "name": "generic_08_plan_detailed", + "input": "Plan the 2.2.0 release validation checklist.", + "settings": { + "target_agent": "generic_chat", + "action_mode": "plan", + "detail_level": "detailed", + "include_acceptance_criteria": True, + }, + "must_contain": ["## Task", "## Acceptance criteria"], + "must_not_contain": ["Implement the requested change"], + }, + { + "name": "generic_09_debug_acceptance", + "input": "Debug live-FLM prompt eval failures and capture the cause.", + "settings": { + "target_agent": "generic_chat", + "action_mode": "debug", + "include_acceptance_criteria": True, + "include_verification": True, + }, + "must_contain": ["## Task", "## Acceptance criteria", "## Verification"], + }, + { + "name": "generic_10_implement_output", + "input": "Add a dashboard note explaining non-Claude targets.", + "settings": { + "target_agent": "generic_chat", + "action_mode": "implement", + "include_output_format": True, + }, + "must_contain": ["## Task", "## Output"], + }, +] + + +def _case_settings(case: dict) -> ffp_prompt_builder.PromptBuilderSettings: + return ffp_prompt_builder.PromptBuilderSettings.from_config(case.get("settings") or {}) + + +def _check_output(case: dict, text: str) -> tuple[bool, list[str]]: + settings = _case_settings(case) + contract = ffp_prompt_builder.validate(text, settings) + errors = list(contract.errors) + for needle in case.get("must_contain") or []: + if needle not in text: + errors.append(f"missing expected text: {needle}") + for needle in case.get("must_not_contain") or []: + if needle in text: + errors.append(f"forbidden text present: {needle}") + return not errors, errors + + +def _call_openai_compatible( + *, + base_url: str, + model: str, + bearer: str, + system_prompt: str, + user_content: str, + timeout_seconds: int, + max_tokens: int, +) -> tuple[str, str]: + body = json.dumps( + { + "model": model, + "messages": [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_content}, + ], + "temperature": 0.1, + "max_tokens": max(64, int(max_tokens)), + "stream": False, + } + ).encode("utf-8") + headers = {"Content-Type": "application/json"} + if bearer: + headers["Authorization"] = f"Bearer {bearer}" + req = urllib.request.Request( + base_url.rstrip("/") + "/v1/chat/completions", + data=body, + headers=headers, + method="POST", + ) + with urllib.request.urlopen(req, timeout=max(2, int(timeout_seconds))) as resp: + payload = json.loads(resp.read().decode("utf-8")) + choices = payload.get("choices") or [] + content = "" + if choices: + msg = choices[0].get("message") or {} + content = str(msg.get("content") or "") + return ffp_llm_client.normalize_output(content), str(payload.get("model") or model) + + +def run_case_model_free(case: dict) -> dict: + settings = _case_settings(case) + intent = ffp_prompt_builder.resolve_intent(settings, case["input"]) + text = ffp_prompt_builder.render_fallback(settings, intent, case["input"]) + valid, errors = _check_output(case, text) + return _row(case, settings, valid, errors, text, mode="model_free") + + +def run_case_live(case: dict, args: argparse.Namespace) -> dict: + settings = _case_settings(case) + intent = ffp_prompt_builder.resolve_intent(settings, case["input"]) + system_prompt = ffp_prompt_builder.build_system_prompt( + settings, + intent, + legacy_system_prompt=ffp_config.CLAUDE_PROMPT_SYSTEM_PROMPT, + ) + started = time.time() + try: + text, model_used = _call_openai_compatible( + base_url=args.base_url, + model=args.model, + bearer=args.bearer, + system_prompt=system_prompt, + user_content=case["input"], + timeout_seconds=args.timeout_seconds, + max_tokens=args.max_tokens, + ) + elapsed = round(time.time() - started, 2) + valid, errors = _check_output(case, text) + row = _row(case, settings, valid, errors, text, mode="live") + row["model"] = model_used + row["elapsed_seconds"] = elapsed + return row + except Exception as exc: + row = _row(case, settings, False, [str(exc)], "", mode="live") + row["model"] = args.model + row["elapsed_seconds"] = round(time.time() - started, 2) + return row + + +def _row( + case: dict, + settings: ffp_prompt_builder.PromptBuilderSettings, + valid: bool, + errors: list[str], + text: str, + *, + mode: str, +) -> dict: + return { + "name": case["name"], + "mode": mode, + "target_agent": settings.target_agent, + "structure": ffp_prompt_builder.effective_structure(settings), + "valid": bool(valid), + "errors": errors, + "output_chars": len(text), + "output": text, + } + + +def summarize(rows: list[dict]) -> dict: + adapters: dict[str, dict[str, int]] = {} + for row in rows: + key = str(row["target_agent"]) + bucket = adapters.setdefault(key, {"passed": 0, "total": 0}) + bucket["total"] += 1 + if row["valid"]: + bucket["passed"] += 1 + return { + "ok": all(row["valid"] for row in rows), + "adapters": adapters, + "cases_total": len(rows), + "cases_passed": sum(1 for row in rows if row["valid"]), + } + + +def emit_text(rows: list[dict], summary: dict) -> None: + print("name|mode|target|structure|valid|errors") + for row in rows: + errors = "; ".join(row["errors"]) + print( + f"{row['name']}|{row['mode']}|{row['target_agent']}|" + f"{row['structure']}|{row['valid']}|{errors}" + ) + print("summary") + for target, counts in sorted(summary["adapters"].items()): + print(f"{target}: {counts['passed']}/{counts['total']}") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--json", action="store_true", help="emit JSON instead of a text table") + parser.add_argument("--out", help="write JSON results to this path") + parser.add_argument("--live", action="store_true", help="call a local OpenAI-compatible endpoint") + parser.add_argument("--base-url", default="http://127.0.0.1:52625") + parser.add_argument("--model", default="qwen3.5:4b") + parser.add_argument("--bearer", default="flm") + parser.add_argument("--timeout-seconds", type=int, default=90) + parser.add_argument("--max-tokens", type=int, default=900) + args = parser.parse_args() + + rows = [run_case_live(case, args) if args.live else run_case_model_free(case) for case in CASES] + summary = summarize(rows) + payload: dict[str, Any] = { + **summary, + "live": bool(args.live), + "base_url": args.base_url if args.live else "", + "model": args.model if args.live else "", + "cases": rows, + } + + if args.out: + out_path = Path(args.out) + out_path.parent.mkdir(parents=True, exist_ok=True) + out_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + if args.json: + print(json.dumps(payload, ensure_ascii=False, indent=2)) + else: + emit_text(rows, summary) + return 0 if summary["ok"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main())