Skip to content

Commit 32b3dc4

Browse files
authored
Harden portable skill protocol, installer safety, and quality gates (#2)
* refactor(skill): define portable evidence-first engineering gates * fix(installer): preserve existing entries and require explicit targets * test(quality): gate package integrity and portable utility regressions * ci: pin verified Node 24-compatible action releases
1 parent 3a7bc44 commit 32b3dc4

19 files changed

Lines changed: 1319 additions & 254 deletions

.editorconfig

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
root = true
2+
3+
[*]
4+
charset = utf-8
5+
end_of_line = lf
6+
insert_final_newline = true
7+
trim_trailing_whitespace = true
8+
indent_style = space
9+
indent_size = 4
10+
11+
[*.{yml,yaml,sh}]
12+
indent_size = 2

.github/dependabot.yml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
version: 2
2+
updates:
3+
- package-ecosystem: github-actions
4+
directory: /
5+
schedule:
6+
interval: weekly
7+
open-pull-requests-limit: 3

.github/pull_request_template.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
## Scope and evidence
2+
3+
State the accepted outcomes, inspected baseline, and implemented root causes.
4+
Link each outcome to current verification. Distinguish observed facts from assumptions.
5+
6+
## Verification
7+
8+
Record commands and results after the final relevant edit. Name unrun/waived checks.
9+
For skill wording changes, identify the scenarios reviewed and whether a model actually ran.
10+
A scenario definition or keyword check is not a successful behavioral evaluation.
11+
12+
## Safety and delivery
13+
14+
Describe compatibility, dependency changes, user-work preservation, discovered bugs,
15+
observability, migration, and rollback. Explain any inapplicable areas.
16+
Keep commits coherent and inspect the staged diff. Do not bypass the required docs check.

.github/workflows/ci.yml

Lines changed: 41 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -4,22 +4,52 @@ on:
44
push:
55
branches: [main]
66
pull_request:
7+
workflow_dispatch:
78

89
permissions:
910
contents: read
1011

12+
concurrency:
13+
group: code-max-${{ github.workflow }}-${{ github.ref }}
14+
cancel-in-progress: true
15+
1116
jobs:
17+
checks:
18+
strategy:
19+
fail-fast: false
20+
matrix:
21+
include:
22+
- os: ubuntu-latest
23+
python: '3.10'
24+
- os: macos-latest
25+
python: '3.13'
26+
runs-on: ${{ matrix.os }}
27+
timeout-minutes: 10
28+
steps:
29+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
30+
with:
31+
persist-credentials: false
32+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
33+
with:
34+
python-version: ${{ matrix.python }}
35+
- name: Validate the skill package
36+
run: python3 scripts/validate.py
37+
- name: Run isolated utility regression tests
38+
run: python3 -m unittest discover -s tests -v
39+
- name: Check Bash syntax
40+
run: bash -n skills.sh
41+
- name: ShellCheck
42+
if: runner.os == 'Linux'
43+
run: shellcheck skills.sh
44+
45+
# Preserve the existing branch-protection context, including matrix failures.
1246
docs:
47+
if: always()
48+
needs: checks
1349
runs-on: ubuntu-latest
50+
timeout-minutes: 2
1451
steps:
15-
- uses: actions/checkout@v4
16-
- name: SKILL.md frontmatter is valid
17-
run: |
18-
head -1 SKILL.md | grep -qx -- '---'
19-
grep -qx 'name: code-max' SKILL.md
20-
grep -q '^description: Use when ' SKILL.md
21-
- name: README and AGENTS links resolve
22-
run: |
23-
grep -ohE '\]\(([^)#:]+)\)' README.md AGENTS.md | sed -E 's/^\]\(|\)$//g' | sort -u | while read -r f; do
24-
test -e "$f" || { echo "broken link: $f"; exit 1; }
25-
done
52+
- name: Require every check to pass
53+
env:
54+
CHECK_RESULT: ${{ needs.checks.result }}
55+
run: test "$CHECK_RESULT" = success

.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
__pycache__/
2+
*.py[cod]
3+
.venv/
4+
.DS_Store

AGENTS.md

Lines changed: 46 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,34 +1,54 @@
1-
# AGENTS.md
1+
# Repository guidance
22

3-
This repository *is* an agent skill. The skill body lives in `SKILL.md` — read it before acting on any coding task in this repo.
3+
This repository distributes an agent skill, not an application framework. Read
4+
[SKILL.md](SKILL.md) before changing it. Follow host instruction precedence and the
5+
user's authorized task. These rules govern this repository, not every consuming project.
46

5-
## Repo map
7+
## Ownership map
68

7-
| File | Purpose |
9+
| Path | Responsibility |
810
| --- | --- |
9-
| `SKILL.md` | The skill itself: frontmatter (`name`, `description`) + the maximum-rigor protocol. |
10-
| `skills.sh` | Symlinks this directory into each supported agent's skills folder. |
11-
| `README.md` | Human-facing docs: what it does, install, usage. |
12-
| `banner.svg` | Header image used by the README. |
13-
14-
## Rules for changes here
15-
16-
1. `SKILL.md` frontmatter must stay valid: `---` on line 1, `name:` matching the directory name, and a `description:` written as *"Use when ..."* trigger conditions.
17-
2. Keep `SKILL.md` under ~200 lines. It is loaded into every agent's context; every line costs tokens on every run.
18-
3. Any behavior change in `SKILL.md` that alters what the skill promises must be mirrored in `README.md`.
19-
4. `skills.sh` is POSIX-ish bash and must pass `shellcheck skills.sh`. New agents go in the `TARGETS` array — nowhere else.
20-
5. No new dependencies, build steps, or package manifests. This repo is text plus one shell script by design.
21-
22-
## Verifying
11+
| [SKILL.md](SKILL.md) | Portable core protocol and activation metadata. |
12+
| [references/](references/) | Directly linked, conditional quality guidance and reporting. |
13+
| [scripts/](scripts/) and [skills.sh](skills.sh) | Optional standard-library Python helpers and Bash entry point. |
14+
| [tests/](tests/) | Isolated utility regression and negative-control tests. |
15+
| [evals/](evals/) | Behavioral scenarios and honest model-evaluation procedure. |
16+
| [README.md](README.md) | Installation, capabilities, limitations, and adoption. |
17+
| [docs/](docs/) | Audit evidence, migration, and historical research. |
18+
| [.github/](.github/) | CI and contribution gates. |
19+
20+
## Change contract
21+
22+
- Inspect the baseline and preserve user-owned work. Record multi-step acceptance work
23+
in the existing task/PR ledger; do not introduce duplicate trackers for every request.
24+
- Keep the core architecture-, stack-, host-, and tool-agnostic. Do not mandate universal
25+
frameworks, arbitrary coverage percentages, broad refactors, or unavailable tools.
26+
- Keep SKILL.md at most 200 lines and 12,000 UTF-8 bytes. This is a local context budget,
27+
not an industry standard. Use one-hop references for optional detail.
28+
- Maintain the minimal frontmatter profile: unquoted, single-line `name: code-max` and
29+
a `description: Use when ...` scalar. The validator deliberately is not a general YAML parser.
30+
- Mirror behavioral promises, dependencies, CLI changes, and limitations in README.md.
31+
Update this ownership map or scoped guidance when responsibilities change.
32+
- Preserve the instruction-only consumption path. Optional tooling uses Python 3.10+
33+
standard library and Bash; no pip/npm dependencies, network calls, or package manifests.
34+
Do not turn installation into execution hooks or automatically modify host permissions.
35+
- Tests are required for executable behavior changes. Use temporary HOME and explicit
36+
targets; never test installation against the developer's real agent directories.
37+
- Report every discovered bug with evidence and disposition. Fix in-scope defects; record
38+
other findings without hiding them or silently expanding scope.
39+
- Make small, coherent, reviewable commits. Stage exact paths and inspect the staged diff.
40+
Work on a branch and open a PR. Respect the existing required `docs` status context;
41+
do not bypass branch protection or claim a remote check passed without observing it.
42+
43+
## Verification
2344

2445
```bash
25-
shellcheck skills.sh # lint (local only, not in CI)
26-
head -1 SKILL.md # must be ---
27-
./skills.sh # idempotent; re-running must not break existing symlinks
46+
python3 scripts/validate.py
47+
python3 -m unittest discover -s tests -v
48+
bash -n skills.sh
49+
shellcheck skills.sh
2850
```
2951

30-
CI (`.github/workflows/ci.yml`) checks the docs only: `SKILL.md` frontmatter and that links in `README.md` / `AGENTS.md` resolve. There are no test suites here — keep it that way.
31-
32-
## Contributing flow
33-
34-
`main` is protected: force-pushes and deletions are blocked, history is linear, and CI must pass. Work on a branch, open a PR, let `docs` go green, then squash-merge.
52+
Read [evals/README.md](evals/README.md) for behavioral evaluation. Utility tests and
53+
scenario-schema validation do not prove model compliance. Report model evaluations,
54+
platform checks, lint, and reviews as unrun when they were unavailable.

README.md

Lines changed: 108 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,78 +1,140 @@
11
<p align="center">
2-
<img src="banner.svg" alt="code-max" width="100%">
2+
<img src="banner.svg" alt="code-max: evidence, or it didn't happen" width="100%">
33
</p>
44

5-
<p align="center">
6-
<a href="SKILL.md"><img src="https://img.shields.io/badge/type-agent%20skill-0b0e14?style=flat" alt="agent skill"></a>
7-
<a href="skills.sh"><img src="https://img.shields.io/badge/works%20with-any%20coding%20agent-5ee2a0?style=flat&labelColor=0b0e14" alt="any coding agent"></a>
8-
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-8b98a8?style=flat&labelColor=0b0e14" alt="MIT"></a>
9-
<a href="https://skills.sh/PyModel/code-max"><img src="https://skills.sh/b/PyModel/code-max" alt="skills.sh installs"></a>
10-
</p>
5+
# code-max
116

12-
An agent skill that stops a coding agent from telling you it finished when it did not — and from shipping the cheap, partial version as if it were production-ready.
7+
A portable engineering skill for evidence-backed implementation, debugging, review,
8+
refactoring, and migrations. It demands the smallest complete solution, relevant tests,
9+
and an honest report instead of unsupported claims that work is finished.
1310

14-
## What it does
11+
## Engineering contract
1512

16-
Coding agents like to say "done" after writing code they never ran. code-max replaces that habit with a production-grade contract. Production-grade means the smallest complete solution, not extra architecture and not a quick substitute for required behavior:
13+
The [core protocol](SKILL.md) requires an acceptance ledger proportional to the task,
14+
source inspection, preservation of user-owned work, cause-level fixes, and verification
15+
after the last relevant edit. It adds architecture discovery, explicit dependencies and
16+
contracts, risk-based quality gates, prompt defect reporting, small reviewable commits,
17+
and migration/rollback planning where relevant.
1718

18-
- Every independently omittable requirement gets an observable acceptance item and direct proof. Trivial work stays lightweight; substantial work uses the harness plan or the repository's tracker.
19-
- No slop, lazy scope reduction, TODOs, stubs, partial migrations, placeholder data presented as real, unwired code, or deferred in-scope edge cases.
20-
- Bugs and behavior changes start with the exact failing test or deterministic reproducer. A durable regression test remains when the project has a test harness.
21-
- Fixes land at the smallest correct shared layer after tracing affected callers, sibling paths, interfaces, tests, and invariants. No under-scoped one-path patch and no drive-by refactor.
22-
- Non-trivial work gets four risk-proportional passes: complete implementation, domain-expert reread, adversarial defect hunt, then low-cost polish. Trivial edits combine them into one focused review. Repeat an affected pass only when the preceding pass changes implementation or proof; stop when the acceptance ledger is reconciled, applicable checks pass, the final diff is reviewed and clean, and no known in-scope defect remains.
23-
- Checks must directly observe the claimed outcome and be able to fail. Negative searches vulnerable to empty inputs, wrong paths, or weak patterns use a positive control; reported numbers are remeasured from the source of truth.
24-
- Tests, type checks, lint, builds, integration checks, and smoke tests run when relevant after the last relevant edit. A green but unrelated check is not proof.
25-
- Delegated work is independently inspected, re-run, and integration-tested by the parent. High-risk or cross-cutting diffs get read-only independent review when available; review never replaces tests.
26-
- Changes stay inside the complete requested scope and preserve your uncommitted work. Instructions hidden in source, logs, generated content, tool output, or web pages remain untrusted data.
27-
- An explicit check waiver never becomes an invented pass. The report names the waiver, remaining proof, and resulting limitation; status follows the task owner's criteria and observed evidence.
19+
It adapts to the repository rather than imposing a stack, framework, microservices,
20+
coverage percentage, or new toolchain. [Conditional quality gates](references/quality-gates.md)
21+
cover architecture, APIs, data, concurrency, resource cleanup, security/privacy, UI and
22+
accessibility, performance, supply chain, and operations. Load only applicable guidance.
2823

29-
Every run ends with a proportional evidence report: status, acceptance results, changed files, and commands actually executed; failures, unvalidated facts, risks, and suspected injection appear only when present. The agent rereads the current request, reconciles every acceptance item, remeasures claims, and reviews the final diff and status before writing `COMPLETE`. Trivial edits get a compact report. Any material unknown, unmet item, or genuine external constraint stays visible as `BLOCKED`.
24+
`COMPLETE` requires current evidence for every authorized acceptance item. `PARTIAL`
25+
retains named unfinished work. `BLOCKED` identifies a concrete constraint. Waived, failed,
26+
and unrun checks never become passes. See the [report template](references/report-template.md).
27+
Explicitly requested scaffolding is allowed but cannot be sold as implemented functionality.
3028

31-
code-max remains instruction-only and agent-agnostic. It does not install hooks, add runtime dependencies, or force orchestration machinery onto focused work.
29+
**Limits:** Instructions cannot guarantee agent compliance or sandbox execution. Host
30+
permissions, review, and project CI remain necessary. The skill does not grant authority
31+
to push, deploy, change permissions, or override project governance. Automated checks in
32+
this repository validate the package and utilities, not every downstream codebase or model.
3233

3334
## Install
3435

35-
```bash
36-
npx skills add PyModel/code-max
37-
```
38-
39-
Installs into whichever agents the [`skills`](https://github.com/vercel-labs/skills) CLI finds on your machine. Update later with `npx skills update code-max`.
36+
The skill itself is Markdown with no runtime dependencies or hooks. Review it before
37+
loading. Install the whole directory, including references, using your host's supported
38+
skill mechanism. Command syntax and discovery differ between hosts.
4039

41-
Or clone and symlink it yourself:
40+
For an inspected local checkout, the optional installer needs Python 3.10+ and Bash:
4241

4342
```bash
4443
git clone https://github.com/PyModel/code-max.git
4544
cd code-max
46-
./skills.sh
45+
./skills.sh --list
46+
./skills.sh --agent claude --dry-run
47+
./skills.sh --agent claude
4748
```
4849

49-
`skills.sh` symlinks this directory into the skills folder of every agent it knows about:
50+
Choose only the hosts you need. Repeat `--agent` or give an absolute custom **parent skills
51+
directory**, not the final code-max path:
5052

51-
| Agent | Path |
52-
| --- | --- |
53-
| Claude Code | `~/.claude/skills/code-max` |
54-
| Codex | `~/.codex/skills/code-max` |
55-
| Cursor | `~/.cursor/skills/code-max` |
56-
| Gemini | `~/.gemini/skills/code-max` |
57-
| Pi | `~/.pi/skills/code-max` |
58-
| OpenCode | `~/.config/opencode/skills/code-max` |
53+
```bash
54+
./skills.sh --agent codex
55+
./skills.sh --target "$HOME/custom-agent/skills" --dry-run
56+
./skills.sh --target "$HOME/custom-agent/skills"
57+
./skills.sh --agent claude --uninstall --dry-run
58+
./skills.sh --agent claude --uninstall
59+
```
5960

60-
Because they are symlinks, `git pull` updates every agent at once. Add or remove entries by editing the `TARGETS` array at the top of the script.
61+
`python3 scripts/install.py` accepts the same options without the Bash wrapper.
62+
With no target selection the installer exits without changing anything. `--all` explicitly
63+
selects all presets; it does not detect installed agents. Hosts that read multiple shared
64+
locations may show duplicates, so prefer selecting a single appropriate location.
65+
66+
| Preset | Destination |
67+
| --- | --- |
68+
| `claude` | `~/.claude/skills/code-max` |
69+
| `codex` | `~/.agents/skills/code-max` |
70+
| `cursor` | `~/.cursor/skills/code-max` |
71+
| `gemini` | `~/.gemini/skills/code-max` |
72+
| `pi` | `~/.pi/agent/skills/code-max` |
73+
| `opencode` | `${XDG_CONFIG_HOME:-$HOME/.config}/opencode/skills/code-max` |
74+
75+
These are discovery presets, not a claim that every host/version was integration-tested.
76+
Use `--target` for other hosts or configured paths. Presets require an absolute HOME;
77+
OpenCode also requires XDG_CONFIG_HOME to be absolute when set. Custom targets do not
78+
require HOME. The installed name remains `code-max` even if the checkout is renamed.
79+
80+
Installation preflights all destinations, preserves existing correct links, and refuses
81+
files, directories, foreign links, and dangling links. Link creation is no-clobber. Dry-run
82+
creates no directories or links. Uninstall removes only links resolving to this checkout;
83+
it never deletes the checkout or target directories. There is no force/overwrite option.
84+
85+
Use trusted, user-owned target directories, not directories concurrently modified by an
86+
adversary. Preflight is not a multi-target filesystem transaction: a later I/O failure may
87+
leave earlier reported operations completed. Inspect output and retry idempotently, or
88+
remove the successful links with the same target selection and `--uninstall`. Uninstall
89+
checks ownership before removal but is not a defense against hostile concurrent replacement.
90+
91+
## Migration and update
92+
93+
Older `./skills.sh` with no arguments installed everywhere and could overwrite entries.
94+
Use `--agent` or `--target` now; use `--all` only deliberately. Old Codex and Pi paths were
95+
`~/.codex/skills` and `~/.pi/skills`. Inspect them before removing duplicates; uninstall
96+
an old link with `--target` only while it still resolves to this checkout. Foreign/stale
97+
links are intentionally not auto-deleted. Nothing migrates your host settings or permissions.
98+
99+
Links follow checkout changes. Review updates before pulling, and use a reviewed tag/commit
100+
or separate checkout when you need an immutable installation. To roll back, select a known
101+
good revision in a clean dedicated checkout; do not reset user-owned changes. See the
102+
[hardening audit](docs/hardening.md) for evidence and rollout limits.
61103

62104
## Use
63105

64-
Ask for it by name, or describe the rigor you want:
106+
Ask your host to use code-max for the task, for example:
65107

66-
```
67-
/code-max fix the token refresh race in src/auth/session.ts
108+
```text
109+
Use code-max to fix the token refresh race. Preserve existing APIs and user changes.
110+
Use code-max for a read-only architecture review. Do not edit or publish anything.
68111
```
69112

113+
Use the project's existing test commands, CI, review process, and task tracker. The skill
114+
does not automatically add hooks, copy AGENTS.md into other projects, or execute helpers.
115+
116+
## Contributing and validation
117+
118+
Read [AGENTS.md](AGENTS.md). Optional tools require no third-party Python packages:
119+
120+
```bash
121+
python3 scripts/validate.py
122+
python3 -m unittest discover -s tests -v
123+
bash -n skills.sh
124+
shellcheck skills.sh
70125
```
71-
Implement the CSV import. Verify before you claim done, no stubs.
72-
```
73126

74-
Reach for it when a wrong answer is expensive: migrations, auth, payments, anything you plan to merge without reading closely. For a one-line typo fix it is overhead.
127+
The offline validator checks this repository's restricted two-scalar frontmatter profile,
128+
core line/byte budget, package symlinks, supported local link forms in all Markdown files,
129+
and [scenario definitions](evals/scenarios.json). It is not a general YAML/Markdown parser;
130+
it does not resolve heading anchors, check external URLs, or execute model evaluations.
131+
The installed folder name matches the skill metadata; renamed source checkouts are allowed.
132+
133+
CI preserves the required `docs` check and gates it on Linux/Python 3.10 and macOS/Python
134+
3.13 utility checks, with ShellCheck on Linux. Actions are SHA-pinned with read-only contents
135+
permission and checkout credentials disabled. See [behavioral evaluation](evals/README.md)
136+
for the separate model/host evaluation procedure and its unrun status.
75137

76138
## License
77139

78-
MIT
140+
[MIT](LICENSE). Original artwork and historical research remain in the repository.

0 commit comments

Comments
 (0)