Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@

All notable changes to FixMap are documented here.

## 0.7.0 - 2026-07-22

### Added

- Added bounded, explainable definition-site ranking for distinctive task identifiers and exact code or literal fragments, including truncated literals from issue excerpts.
- Added focused regressions for exact literal extraction, generic-identifier noise, and definition-site ranking.

### Changed

- Improved the frozen six-repository evaluation from 50% / 83% / 83% top-1/3/5 to 67% / 100% / 100%. The unchanged Zod #5944 case now ranks its fixing `regexes.ts` file first.

## 0.6.2 - 2026-07-22

### Security
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ jobs:
with:
fetch-depth: 0
- id: fixmap
uses: aryamthecodebreaker/FixMap@v0.6.2
uses: aryamthecodebreaker/FixMap@v0.7.0
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
```
Expand Down Expand Up @@ -189,7 +189,7 @@ npm run evaluate

The current suite covers Action failures, invalid diffs, authentication, the web demo, workspace test routing, and contributor documentation. The cases and full ranked results are visible in [`benchmarks/`](benchmarks).

A separate cross-repository evaluation runs FixMap against six real, already-fixed issues in permissively licensed repositories (Express, Axios, debug, ky, Zod, Pino) pinned to exact commits, and reports honest top-1/3/5 hit rates — currently 50% / 83% / 83%. The dataset, methodology, ranked outputs, and remaining Zod miss live in [`benchmarks/external/`](benchmarks/external); a scheduled workflow reruns it weekly.
A separate cross-repository evaluation runs FixMap against six real, already-fixed issues in permissively licensed repositories (Express, Axios, debug, ky, Zod, Pino) pinned to exact commits, and reports honest top-1/3/5 hit rates — currently 67% / 100% / 100%. The frozen dataset, methodology, and every ranked output live in [`benchmarks/external/`](benchmarks/external); a scheduled workflow reruns it weekly.

## Repository layout

Expand Down
2 changes: 1 addition & 1 deletion apps/web/app/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ export default function HomePage() {
<a className="button primary" href="https://github.com/aryamthecodebreaker/FixMap">Explore the GitHub repo ↗</a>
<a className="button secondary" href="/fixmap-launch.mp4">Open the video</a>
</div>
<p className="film-note">Original product film · 1280×720 · sound on · current release v0.6.2</p>
<p className="film-note">Original product film · 1280×720 · sound on · current release v0.7.0</p>
</div>
<figure className="film-frame">
<video
Expand Down
12 changes: 6 additions & 6 deletions benchmarks/external/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,16 @@ The first run shallow-clones each repository at its pinned SHA into the OS temp

## Results

Measured 2026-07-18 on the dataset above (FixMap v0.5.1, Node v24, `rankContextFiles` with a top-5 window):
Measured 2026-07-22 on the dataset above (FixMap v0.7.0, Node v24, `rankContextFiles` with a top-5 window):

| Metric | Hit rate |
| --- | --- |
| top-1 | 3/6 (50%) |
| top-3 | 5/6 (83%) |
| top-5 | 5/6 (83%) |
| top-1 | 4/6 (67%) |
| top-3 | 6/6 (100%) |
| top-5 | 6/6 (100%) |

Remaining miss: zod #5944 (fix lives in `regexes.ts`; the 600-character task excerpt ends inside the reported pattern, and ranking surfaces the JSON-schema modules that consume it).
The v0.7.0 ranker adds bounded definition-site evidence for distinctive task identifiers and exact code or literal fragments. That general signal moves the frozen Zod #5944 fixing file, `regexes.ts`, from outside the top five to top one without changing the task, expected path, or selection rule.

The exact per-case top-five rankings are checked in at [`results.json`](results.json).
The exact per-case top-five rankings are checked in at [`results.json`](results.json). Six cases are useful regression evidence, not a general claim that FixMap is 100% accurate.

The `--gate` floors (top-1 ≥ 0.3, top-3 ≥ 0.5, top-5 ≥ 0.5) exist only to catch ranking collapses in the scheduled run. They are deliberately below measured performance and are not accuracy claims or targets.
18 changes: 9 additions & 9 deletions benchmarks/external/results.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{
"cases": 6,
"top1HitRate": 0.5,
"top3HitRate": 0.833,
"top5HitRate": 0.833,
"top1HitRate": 0.667,
"top3HitRate": 1,
"top5HitRate": 1,
"floors": {
"top1": 0.3,
"top3": 0.5,
Expand Down Expand Up @@ -84,15 +84,15 @@
"packages/zod/src/v4/core/regexes.ts"
],
"top5": [
"packages/zod/src/v4/core/regexes.ts",
"packages/zod/src/v4/classic/from-json-schema.ts",
"packages/zod/src/v4/core/json-schema-processors.ts",
"packages/zod/src/v4/core/to-json-schema.ts",
"packages/zod/src/v4/core/checks.ts",
"packages/zod/src/v4/core/json-schema.ts"
"packages/zod/src/v4/mini/schemas.ts",
"packages/zod/src/v4/core/to-json-schema.ts"
],
"top1": false,
"top3": false,
"top5Hit": false
"top1": true,
"top3": true,
"top5Hit": true
},
{
"slug": "pinojs/pino",
Expand Down
30 changes: 30 additions & 0 deletions docs/GROWTH_LOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# FixMap growth log

This log records dated baselines and distribution experiments for the 5,000-star goal. GitHub traffic is a rolling 14-day window, so snapshots are preserved here instead of reconstructed later.

## Baseline — 2026-07-22 10:50 UTC

| Metric | Value |
| --- | ---: |
| GitHub stars | 4 |
| GitHub forks | 0 |
| GitHub views (14 days) | 160 total / 14 unique |
| GitHub clones (14 days) | 551 total / 178 unique |
| npm CLI downloads (last day) | 24 |
| npm CLI downloads (last 7 days) | 652 |
| npm CLI downloads (last 30 days) | 878 |
| Open issues | 1 (#59) |

Top GitHub referrers were GitHub (51 views / 2 unique), LinkedIn Android (3 / 2), LinkedIn web (1 / 1), and `t.co` (1 / 1). The repository overview received 120 views from 11 unique visitors.

### Interpretation

Package and clone activity is materially higher than human repository traffic. The immediate bottleneck is qualified discovery and visitor-to-star conversion, not package availability. The next proof point is the v0.7.0 ranking release: it closes the only frozen external benchmark miss and makes a measurable technical story available for promotion.

## Experiments

| Date/time UTC | Channel/change | Stars before | Stars +24h | Stars +72h | Unique visitors | npm downloads | Qualified feedback |
| --- | --- | ---: | ---: | ---: | ---: | ---: | --- |
| 2026-07-22 10:50 | v0.7.0 definition-site ranking and 6/6 Top-5 proof | 4 | — | — | 14 baseline | 652 weekly | #59 implementation |

Only record real installs, issues, discussions, pull requests, integrations, or technically specific comments as qualified feedback. Never coordinate votes or manufacture engagement.
22 changes: 11 additions & 11 deletions docs/LAUNCH_KIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,23 +19,23 @@ The report ranks likely files with reasons, suggests test routes, and names risk
- CLI, MCP server, and GitHub Action share the same core ranker.
- Public GitHub issue URLs supply both task context and the repository in one input; source is scanned in an isolated anonymous shallow checkout that is removed after analysis.
- A frozen external evaluation uses six real fixed issues at pinned pre-fix commits.
- Current checked-in result: top-1 `3/6` (50%), top-3 `5/6` (83%), top-5 `5/6` (83%).
- The remaining Zod miss and every ranked output are public in [`benchmarks/external/`](../benchmarks/external/).
- Current checked-in result: top-1 `4/6` (67%), top-3 `6/6` (100%), top-5 `6/6` (100%).
- Every ranked output and the frozen selection rule are public in [`benchmarks/external/`](../benchmarks/external/).
- A release cannot publish unless the local CI suite and external evaluation gate pass.

The six-case result is evidence about those six cases, not a general “83% accurate” claim.
The six-case result is evidence about those six cases, not a general accuracy claim.

## Truth guardrails

Do not say:

- “FixMap is 83% accurate” or “finds the right file 83% of the time.”
- “FixMap is 100% accurate” or “always finds the right file.”
- “FixMap runs or verifies the tests.” It suggests test routes; it does not execute them.
- “Any GitHub URL works.” Only canonical public repository and issue URLs are supported.
- “Your code never leaves your machine” without the remote-mode nuance: FixMap downloads public source from GitHub, but does not upload analyzed source to a FixMap service.
- “5,000 stars is expected.” It is the goal, not a promised outcome.

Keep the documented miss visible. If a scheduled evaluation regresses, pause promotion, fix or explain the regression, and update the published result before resuming.
Keep every per-case result and any future miss visible. If a scheduled evaluation regresses, pause promotion, fix or explain the regression, and update the published result before resuming.

## Release gate before any campaign

Expand Down Expand Up @@ -119,7 +119,7 @@ Points for the maintainer to explain personally:
2. The one-sentence solution: deterministic repo context—ranked files, test routes, risks, and diagnostics.
3. The fastest trial: include the one-input public GitHub issue URL command.
4. The technical mechanism: path/content signals, real git diff signals, bounded static import proximity, file-kind priors, and workspace boundaries.
5. The honest evidence: six frozen real bugs, `3/6` top-1 and `5/6` top-3/top-5, with the Zod miss linked.
5. The honest evidence: six frozen real bugs, `4/6` top-1 and `6/6` top-3/top-5, with every per-case ranking linked.
6. The scope: JavaScript/TypeScript today; remote URLs are issue-only; suggested tests are not executed.
7. What changed because the benchmark caught a regression: the floor failed at `2/6` top-3, the thresholds stayed fixed, and general ranking rules lifted it to `5/6`.
8. Ask for technical criticism of the evaluation and useful next signals.
Expand All @@ -140,7 +140,7 @@ These are angles and evidence, not identical copy to syndicate.

- Problem: agents spend context and tokens discovering where to start.
- Demo: run FixMap first on a public repository, then hand the report to the agent.
- Evidence: deterministic, zero model calls, inspectable reasons, `5/6` top-5 on the frozen set.
- Evidence: deterministic, zero model calls, inspectable reasons, `6/6` top-5 on the frozen set.
- Honest caveat: it is a routing aid, not semantic code understanding or a correctness oracle.

### Claude Code and Cursor communities
Expand All @@ -167,7 +167,7 @@ claude mcp add fixmap -- npx -y @aryam/fixmap@latest mcp
1. One pain sentence.
2. The public repository command.
3. A screenshot or short terminal video of the real output.
4. One evidence sentence: six pinned bugs, `3/6` top-1 and `5/6` top-5, one public miss.
4. One evidence sentence: six pinned bugs, `4/6` top-1 and `6/6` top-5, with the full result public.
5. Repository link and a specific feedback question.

Avoid generic feature lists and unsupported superlatives.
Expand All @@ -191,10 +191,10 @@ The latest work includes:
- a shared CLI, MCP server, and GitHub Action workflow
- a GitHub Marketplace listing for the Action
- a production dependency audit gate with no high or critical findings
- 119 automated tests, production builds, smoke tests, and a frozen cross-repository evaluation gate
- 122 automated tests, production builds, smoke tests, and a frozen cross-repository evaluation gate
- a new 24-second product film on the README and live site

The evaluation is intentionally public and modest: across six pinned real bugs, FixMap currently ranks an expected file in the top 1 for 3/6 cases and in the top 3 for 5/6. The miss is published too.
The evaluation is intentionally public and modest: across six pinned real bugs, FixMap now ranks an expected file in the top 1 for 4/6 cases and in the top 3 for 6/6. Every per-case ranking is published, and six cases are not a general accuracy claim.

Watch the film and try the live experience: https://fixmap-flax.vercel.app/#launch-film

Expand All @@ -204,7 +204,7 @@ Marketplace: https://github.com/marketplace/actions/fixmap

npm: https://www.npmjs.com/package/@aryam/fixmap

Release: https://github.com/aryamthecodebreaker/FixMap/releases/tag/v0.6.2
Release: https://github.com/aryamthecodebreaker/FixMap/releases/tag/v0.7.0

I would especially value feedback on the ranking explanations and which repository signals would make FixMap more useful before an agent starts editing.

Expand Down
2 changes: 1 addition & 1 deletion examples/reports/workspace-api-discount.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ FixMap found 3 context files and generated 3 test routes.

## Context Files

- `apps/api/src/orders.ts` (medium confidence, score 13): path matches task terms: order; content matches task terms: discount, order, total, code
- `apps/api/src/orders.ts` (high confidence, score 17): path matches task terms: order; content matches task terms: discount, order, total, code; defines task identifiers: orderTotal
- `README.md` (medium confidence, score 8): content matches task terms: order, total, ignore, unknown, discount, code, value
- `packages/utils/src/currency.ts` (low confidence, score 6): content matches task terms: discount, total

Expand Down
2 changes: 1 addition & 1 deletion examples/reports/workspace-utils-rounding.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ FixMap found 2 context files and generated 3 test routes.

## Context Files

- `packages/utils/src/currency.ts` (medium confidence, score 9): path matches task terms: currency; content matches task terms: round, cent
- `packages/utils/src/currency.ts` (medium confidence, score 13): path matches task terms: currency; content matches task terms: round, cent; defines task identifiers: roundToCents
- `README.md` (low confidence, score 6): content matches task terms: round, cent, keep, fraction, formatt, currency

## Test Route
Expand Down
14 changes: 7 additions & 7 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "fixmap-workspace",
"version": "0.6.2",
"version": "0.7.0",
"private": true,
"description": "Local-first repo context for coding agents: paste a GitHub issue URL to get ranked files, test routes, and risks.",
"license": "MIT",
Expand Down
Loading