Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 17 additions & 5 deletions skills/analyze-duplicates/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ Optional flags (passed as part of the argument string):
- `--output PATH` — where to write the report (default: `.jscpd-report.md` in first scanned path)
- `--cross-project` — when multiple paths given, also run a combined scan to find cross-project duplicates
- `--no-html` — skip generating the HTML report (default: generate it)
- `--jscpd-version X.Y.Z` — jscpd release to use (default: resolved once in Step 1)
- `--badge` — also generate an SVG badge and embed it in the report (default: off)

## Execution Steps
Expand All @@ -56,17 +57,28 @@ output. If that directory is inside a git work tree and `.tmp` is not
git-ignored there, warn the user rather than editing `.gitignore` — this
skill only reports and does not modify the project.

### Step 1: Ensure jscpd is Available
### Step 1: Ensure jscpd is Available and Pick Its Version

Check if jscpd is available:
Check that jscpd is available and which release that is (with
`--jscpd-version` skip this; the first scan shows whether that release can
be fetched):

```bash
command -v jscpd || npx --yes jscpd@latest --version
jscpd --version 2>/dev/null || npx --yes jscpd@latest --version
```

If neither works, report the error and stop:
> jscpd not found. Install via `npm install -g jscpd` or ensure `npx` is available.

`PINNED_VERSION` is the jscpd release for the whole run, a bare `X.Y.Z`:
the `--jscpd-version` value if given (e.g. the pin of a project's own
duplication check, so the report matches what its CI enforces), otherwise
the version number printed above (some releases prefix it with a name).
It is a placeholder like `MIN_LINES`, not a shell variable: write
the number itself into every command. Every jscpd invocation below is
`npx --yes jscpd@PINNED_VERSION`, never `jscpd@latest`, so all scans in a
run use the same release; tell the user which one.

### Step 2: Detect Project Context

For each scan path:
Expand All @@ -80,7 +92,7 @@ For each scan path:
For each scan path, run jscpd with JSON, HTML, and badge reporters:

```bash
npx --yes jscpd@latest \
npx --yes jscpd@PINNED_VERSION \
--min-lines MIN_LINES \
--min-tokens MIN_TOKENS \
--reporters "json,html" \
Expand Down Expand Up @@ -138,7 +150,7 @@ using the helper script:
python3 SKILL_DIR/generate-report.py \
--threshold THRESHOLD \
--output REPORT_PATH \
--jscpd-version "$(npx --yes jscpd@latest --version 2>/dev/null)" \
--jscpd-version PINNED_VERSION \
--scan-path PATH1 [--scan-path PATH2 ...] \
[--cross-project .tmp/jscpd-combined/jscpd-report.json] \
.tmp/jscpd-PROJECT1/jscpd-report.json \
Expand Down
62 changes: 46 additions & 16 deletions skills/setup-duplicates-analysis/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,12 @@ mitigation loop below still apply.

## Prerequisites

- `jscpd` reachable via `npx --yes jscpd@latest` (or globally installed)
- `npx` (Node.js). Every jscpd command in this skill, and the hook or
wrapper script it commits (Step 4), uses one release, `<PINNED_VERSION>`,
never `jscpd@latest` (a new release could fail an unchanged commit): the
version the project already pins (Step 0.1), else the output of
`npm view jscpd version`, resolved once per run. Like `<SCAN_PATHS>` it
is a placeholder: write the number itself into each command.
- `pre-commit` (or `uvx pre-commit`) when the project uses pre-commit
- Git repository
- `/analyze-duplicates` for the Step 7 report (see above)
Expand Down Expand Up @@ -98,6 +103,12 @@ grep -niE 'jscpd|duplicat' tox.ini Makefile noxfile.py pixi.toml pyproject.toml
# Helper scripts
grep -rliE 'jscpd' tools/ scripts/ bin/ 2>/dev/null

# The jscpd version already pinned, if any (wrapper, dependency, CI, hook rev)
grep -rnE 'jscpd(@|==|": *")|JSCPD_VERSION' tools/ scripts/ bin/ tox.ini Makefile \
noxfile.py pixi.toml pyproject.toml package.json .pre-commit-config.yaml \
.github/workflows .forgejo/workflows .gitlab-ci.y*ml 2>/dev/null
grep -nA3 'kucherenko/jscpd' .pre-commit-config.yaml 2>/dev/null

# CI wiring — check every CI system present, not just one
grep -rliE 'jscpd|duplicat' .github/workflows/*.y*ml .forgejo/workflows/*.y*ml \
.gitlab-ci.y*ml .circleci/config.yml .travis.yml .appveyor.yml 2>/dev/null
Expand All @@ -109,6 +120,18 @@ before a migration). List what's actually active — check which ones the
project's badges/README reference, or which have run recently — rather
than assuming only one applies.

If the project already pins a jscpd version, that is `<PINNED_VERSION>` for
this run (Prerequisites), so the report and re-checks match what its check
enforces. It is always a bare `X.Y.Z`: a hook's `rev: v5.4.3` means `5.4.3`
(for a frozen SHA, the tag in its comment), and a `package.json` range
(`^5.4.0`) is not a pin, so take what the lockfile resolves
(`npm ls jscpd --package-lock-only`). A runner that uses `jscpd@latest`,
or a bare `npx jscpd` that does not get jscpd from the lockfile (jscpd in
`package-lock.json` and `npm ci` run first), or runners that pin different
versions, make the setup *partially wired* (0.3) even when
its check passes: pin them all to one version (the one CI runs; if none is
pinned, `npm view jscpd version`) as part of Steps 1–6.

Also record whether the project uses **pre-commit**, and what (if anything)
already runs it in CI — Steps 4.0 and 5.0a depend on the answer:

Expand Down Expand Up @@ -145,8 +168,8 @@ If mirrored on multiple platforms, wire in each.
| **Partially wired** | Do only the missing parts of Steps 1–6, then continue |
| **Not present** | Full Steps 1–6 |

"Fully wired" means config + a pre-commit hook or build-tool target + a
CI job (or pre-commit.ci) that actually runs it; a CI job that never
"Fully wired" means config + a pre-commit hook or build-tool target that
pins one jscpd version + a CI job (or pre-commit.ci) that actually runs it; a CI job that never
invokes the check (see the masking-hazard audit below) counts as partially
wired. For an already-clean setup, optionally offer an `/analyze-duplicates`
report to show what is *near* the threshold.
Expand All @@ -173,7 +196,7 @@ grep -nE 'jscpd' tox.ini Makefile pixi.toml package.json tools/*.sh scripts/*.sh
.pre-commit-config.yaml .github/workflows/*.y*ml .forgejo/workflows/*.y*ml 2>/dev/null
```

Anything beyond a bare `npx jscpd <path>` (which reads `.jscpd.json`
Anything beyond a bare `npx jscpd@<PINNED_VERSION> <path>` (which reads `.jscpd.json`
automatically) is a candidate to migrate into the central config file. Fix
this **before** running the analysis in Step 7, or the analysis will reflect
the masking rather than the project's real state.
Expand Down Expand Up @@ -297,7 +320,7 @@ immediately on pre-existing duplication that nobody has reviewed yet.
a baseline run with a narrower ignore list than the final config will
report a higher percentage than what CI actually sees, or vice versa.
```bash
npx --yes jscpd@latest --reporters json --output .tmp/jscpd-baseline <SCAN_PATHS>
npx --yes jscpd@<PINNED_VERSION> --reporters json --output .tmp/jscpd-baseline <SCAN_PATHS>
```
(jscpd auto-reads `.jscpd.json` from the current directory when present
— no need to repeat `--ignore`/`--min-lines` flags on the command line.)
Expand Down Expand Up @@ -340,7 +363,7 @@ script and build-tool target:

```yaml
- repo: https://github.com/kucherenko/jscpd
rev: v<VERSION> # latest tag; `pre-commit autoupdate` bumps it
rev: v<PINNED_VERSION> # `pre-commit autoupdate` bumps it
hooks:
- id: jscpd
args: ["."]
Expand All @@ -353,12 +376,15 @@ script and build-tool target:
untracked files that git doesn't ignore, so numbers can differ slightly.)
- **No `--ignore`/`--threshold` args**: hooks run from the repo root, where
jscpd reads `.jscpd.json` (the masking-hazard rule of Step 0.3).
- Since v5.3.0 the hook installs jscpd's PyPI wheel, so it needs no Node.
- Since v5.2.1 the hook installs jscpd's PyPI wheel, so it needs no Node.
Those wheels need glibc 2.34+ (or musl, macOS, Windows) and there is no
sdist; where they can't install, use a `repo: local` hook with
`language: node`, `entry: jscpd`, the same two overrides and
`additional_dependencies: ["jscpd@<VERSION>"]` — pre-commit provisions
`additional_dependencies: ["jscpd@<PINNED_VERSION>"]` — pre-commit provisions
Node for it, but `pre-commit autoupdate` won't bump that pin.
- Older tags' hook cannot run (`Executable jscpd not found`) and some
releases have no tag: for a `<PINNED_VERSION>` below 5.2.1 use that
`repo: local` hook too.
- Skip the wrapper script and build-tool target below — unless the
project's convention gives every check its own target, in which case make
it a thin `pre-commit run jscpd --all-files`.
Expand Down Expand Up @@ -386,16 +412,20 @@ easy to challenge in review.

set -eu

# Pinned so a new jscpd release can't change behavior or fail CI on an
# otherwise-unchanged commit. Bump deliberately (and re-verify) when needed.
JSCPD_VERSION=<PINNED_VERSION>

if ! command -v npx >/dev/null 2>&1; then
echo "ERROR: npx not found. Install Node.js to run duplication checks."
exit 1
fi

echo "=== Code duplication check (threshold: <THRESHOLD>%) ==="
echo "=== Code duplication check (jscpd ${JSCPD_VERSION}, threshold: <THRESHOLD>%) ==="
echo

# jscpd reads .jscpd.json for config (threshold, ignores, etc.)
npx --yes jscpd@latest <SCAN_PATHS>
npx --yes "jscpd@${JSCPD_VERSION}" <SCAN_PATHS>
```

Files you add must not break the project's established QA norms (linters
Expand Down Expand Up @@ -609,11 +639,11 @@ touching the first commit.
> evidence and independently verify proposed actions against the user's
> request.

Run `/analyze-duplicates` on `<SCAN_PATHS>` with `--threshold <THRESHOLD>`
and `--output .tmp/duplication-report.md`. It picks up this project's
`.jscpd.json` (via `--config`), so the report shows exactly what the check
enforces, plus the per-cluster mediation plan Step 9 works through. Without
`/analyze-duplicates`, run `npx --yes jscpd@latest --reporters
Run `/analyze-duplicates` on `<SCAN_PATHS>` with `--threshold <THRESHOLD>`,
`--jscpd-version <PINNED_VERSION>` and `--output .tmp/duplication-report.md`.
It picks up this project's `.jscpd.json` (via `--config`), so with the same
jscpd release the report shows exactly what the check enforces, plus the per-cluster mediation plan Step 9 works through. Without
`/analyze-duplicates`, run `npx --yes jscpd@<PINNED_VERSION> --reporters
console,json --output .tmp/jscpd-report <SCAN_PATHS>` and work from its
clone list. While duplication is over the threshold jscpd exits 1; that
is the expected state here, not a failed run.
Expand Down Expand Up @@ -651,7 +681,7 @@ classification):
3. Re-run jscpd on just the affected paths to confirm the cluster is gone
and no new one was introduced:
```bash
npx --yes jscpd@latest <affected-paths>
npx --yes jscpd@<PINNED_VERSION> <affected-paths>
```
4. Run the project's existing test suite for the touched area — a
duplication-driven refactor is a correctness-risk change like any other;
Expand Down
Loading