[prompt-analysis] Copilot PR Prompt Analysis - 2026-09-01

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-09-01

Summary

Analysis Period: Last 30 days (2026-08-11 to 2026-09-01)
Total PRs: 1000 | Merged: 841 (84.1%) | Closed: 153 (15.3%) | Open: 6 (0.6%)

Note: 1000 is the pre-fetch cap, so this is a sample of the most recent 1000 Copilot PRs rather than a complete population; percentages should be read as directional trends.

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Testing	99	89	89.9%
Feature Addition	279	239	85.7%
Documentation	70	59	84.3%
Refactoring	57	48	84.2%
Other	184	153	83.2%
Bug Fix	311	253	81.4%

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 190.8 words (shorter than closed PRs)
• Most common non-stopword terms: github, workflow, sous/chef (internal agent naming), actions, copilot, agent, gpt
• Merged PRs skew toward concrete, scoped titles ("Replace legacy sandbox security settings...", "Fix stale skill docs and repo-path guard...")

Example successful prompts:

1. PR Replace legacy sandbox security settings with runtime profiles #57692: "Removes sandbox.agent.legacy-security and sandbox.agent.sudo configuration in favor of explicit..." → Merged
2. PR Classify unsupported Codex custom tools as invalid-model failures #57578: "Daily Go Test Parallelizer reported a Codex tools: custom API rejection as an unexpected engine fa..." → Merged
3. PR Fix stale skill docs and repo-path guard for the action-based JS architecture #57579: "The repo still documented a dead JavaScript embedding/bundling architecture in a few high-traffic sk..." → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 236.6 words (~24% longer than merged prompts)
• Recurring theme: CVE/vulnerability remediation and report-only tasks with no code change intent
• Issues identified: vague success criteria, "no code changes" deliverables, exploratory/investigative asks rather than concrete implementation asks

Example unsuccessful prompts:

1. PR Re-implement task-preflight skill for container-vuln/CVE and refactor tasks #57173: "Container-vuln/CVE-remediation tasks doubled in volume with no success-rate improvement — 78% of unm..." → Closed
2. PR No code changes: Serena image CVE scan report #57059: "No code changes: Serena image CVE scan report. The request was to produce a copy-pasteable report documenting how CVEs are detected..." → Closed
3. PR Add Claude Code LSP support #57258: "Claude Code supports LSP through plugins. This extends gh-aw's experimental lsp configuration to t..." → Closed

Key Insights

• Length matters inversely: Merged PR prompts average ~46 fewer words than closed ones (190.8 vs 236.6). Longer, more exploratory prompts appear to correlate with lower merge rates — likely because they signal ambiguous scope or investigative/report-only tasks rather than a single actionable change.
• Testing and feature prompts outperform bug-fix prompts: Testing (89.9%) and feature-addition (85.7%) prompts have the highest success rates, while bug-fix prompts (81.4%) lag behind — possibly because bug reports without clear reproduction steps or root-cause hints are harder for Copilot to resolve definitively.
• CVE/report-only requests underperform: Several closed PRs (e.g., Re-implement task-preflight skill for container-vuln/CVE and refactor tasks #57173, No code changes: Serena image CVE scan report #57059) were framed around vulnerability scanning or "no code changes" report deliverables rather than concrete code changes, suggesting these tasks are poorly suited to the PR-merge workflow and might be better handled as issues or discussions instead.

Recommendations

1. DO: Keep prompts concise and scoped to a single, well-defined change (aim for well under 200 words) with a clear before/after described.
2. DO: For bug fixes, include the specific error message, failing workflow/job name, or reproduction steps to raise the bug-fix success rate above the current 81.4%.
3. AVOID: Framing CVE-scanning, audit, or "no code changes" report requests as PR-generating prompts — route these to issues/discussions instead, since they've shown a notably lower merge rate historically (50% in the prior period).

Historical Trends

Date	PRs	Success Rate	Top Category
2026-09-01	1000	84.1%	Testing (89.9%)
2026-08-31	994	84.3%	Refactor (100%*)

*Refactor category on 2026-08-31 had only 5 samples, too small to be statistically meaningful.

Trend: Overall merge rate is stable (~84%) week-over-week. Category composition shifted notably: bug-fix prompts jumped from 641→311 in raw count share due to the rolling 30-day window, while feature-addition share grew from 181→279. The bug-fix success rate held steady (82.8% → 81.4%), and average prompt length for merged PRs increased slightly (122.3 → 190.8 words), though closed-PR prompts remain consistently longer than merged-PR prompts across both periods.

---

Generated by Copilot PR Prompt Analysis (Run: 33556734674)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · copilot · auto · 21 AIC · ⌖ 7.89 AIC · ⊞ 11K · ◷

• expires on Sep 2, 2026, 12:48 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #58028.