docs: surface the Marketplace listing, trim a redundant badge

[uipreliga]

Follow-up to #79. The action is now live on the GitHub Actions Marketplace as coder_eval, but nothing in the docs said so.

What changed

Badge row (README.md) — added a Marketplace badge, removed the Docs badge. Row stays at six.

• The Marketplace badge is placed right after PyPI so the two "how you consume this" channels sit together: pip install and the Action.
• The slug is written coder__eval, not coder_eval — shields.io renders a single _ as a space, so the doubled form is what actually displays correctly.
• Dropping Docs costs no discoverability: the Website badge already covers the domain, and the intro links the docs prominently two lines below the badge row (📚 **Full docs:** coder-eval.com/docs).

Listing link added to the three places that introduce the Action:

File	Section
README.md	Use as a GitHub Action
docs/CI_GATE.md	The GitHub Action
docs/tutorials/02-ci-pipeline.md	Shortcut: the packaged action

Each previously described it only as "a composite action at the repo root" — true, but it hid the discovery surface.

What deliberately did NOT change

Every example keeps uses: UiPath/coder_eval@v0. Being on the Marketplace changes discovery, not consumption — the action is still referenced by repo path and there is nothing to install. docs/CI_GATE.md now says that explicitly so nobody goes hunting for an install step.

Also unchanged: coder-eval remains the CLI command and the PyPI package name. Only the Marketplace listing title is coder_eval (see #79 for why — the hyphenated name is squatted by an unrelated org).

Verification

• Audited action.yml against docs/CI_GATE.md: all 10 inputs and both outputs documented, zero missing, zero stale.
• All 8 CLI flags referenced by the CI docs (--junit-xml, --run-dir, --tags, --model, --experiment, --exclude-tags, --sample, --set) verified present in coder-eval run --help.
• No hardcoded coder-eval==X.Y.Z or @v0.9.x anywhere in the docs — all 5 action references use the moving @v0, consistently.
• All six badge images and their link targets resolve; the Marketplace listing returns 200.
• make lint passes 158 tests — CE028 (docs-index parity) and CE029 (doc YAML examples) both clean, checked because this edits prose adjacent to YAML blocks.

🤖 Generated with Claude Code

[github-actions]

Claude finished @uipreliga's task in 1m 25s —— View job

---

Code Review in Progress

Review Checklist:

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Examine the full diff (git diff origin/main...HEAD)
• Review README.md changes
• Review docs/CI_GATE.md changes
• Review docs/tutorials/02-ci-pipeline.md changes
• Perform cross-file consistency checks
• Check for missing documentation or updates
• Submit final review

[uipreliga]

Review: coder_eval — pr:80 (3 files) axis:1,2,3,4,5,6,7,8

Scope: pr:80 (3 files) axis:1,2,3,4,5,6,7,8 · branch docs/readme-marketplace-badge · 58d3571 · 2026-08-04T19:03Z · workflow variant

Change class: trivial — docs/metadata only: adds a GitHub Marketplace badge + listing links in README.md and two docs pages; no code path, schema, or config is touched

coder_eval is in genuinely excellent shape — all eight axes clean with zero critical/high/medium findings and a self-policing architecture (one declarative merge resolver, plugin SPI, CE001+ lint rules) that makes the quality durable rather than incidental — so the only real residual risk is artifact and doc truthfulness rather than code: an onboarding snippet in docs/CI_GATE.md that implies no prerequisites while contradicting itself 11 lines later, and two intentionally nondeterministic harness behaviors (unseeded stratified sampling, pass-stop early exit) whose effect on a suite's score is explained in prose but never recorded in the run artifact; bottom line, ship as-is and spend the next hour on documentation/provenance hardening, not refactors.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	10 / 10	0	0	0	0	—
2. Type Safety	10 / 10	0	0	0	0	—
3. Test Health	10 / 10	0	0	0	0	—
4. Security	10 / 10	0	0	0	0	—
5. Architecture & Design	10 / 10	0	0	0	0	—
6. Error Handling & Resilience	10 / 10	0	0	0	0	—
7. API Surface & Maintainability	10 / 10	0	0	0	0	—
8. Evaluation Harness Quality	9.9 / 10	0	0	0	1	docs/CI_GATE.md: "there is nothing to install" contradicts the agent-runtime prerequisite stated 11 lines later, and the copy-pasteable snippet omits it

Overall Score: 10 / 10 · Weakest Axis: Evaluation Harness Quality at 9.9 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 0 · 🔵 1 across 8 axes.

Blockers

None.

Non-blocking, but please consider before merge

None.

Nits

1. [Axis 8] docs/CI_GATE.md: "there is nothing to install" contradicts the agent-runtime prerequisite stated 11 lines later, and the copy-pasteable snippet omits it (docs/CI_GATE.md:21) — docs/CI_GATE.md is the onboarding contract for the CI gate, and the new intro ends with an unqualified absolute: line 20-21 read composite action living at the repo root (\action.yml`), so you reference it by / repo path — there is nothing to install:. The uses:snippet immediately below (lines 23-30) is a complete, copy-pasteable job step with noactions/setup-node+npm install -g @anthropic-ai/claude-code— yet lines 32-35 sayThe action is agent-agnostic — it installs `coder-eval` but not any / coding-agent runtime. Tasks using the default `claude-code` agent need the / `claude` CLI on `PATH` (Node + `@anthropic-ai/claude-code`), provided by your job / before this step runs.A CI integrator who reads "there is nothing to install" as "no prerequisites" and copies the snippet gets a run that fails on a missingclaudebinary. The sibling passage in docs/tutorials/02-ci-pipeline.md (lines 170-172) does show the setup-node + npm steps, so the two onboarding surfaces now differ in what they imply. Narrow the claim to what it means — e.g.so you reference it by repo path; there is no Marketplace install step` — and/or add the two prerequisite lines to the CI_GATE.md snippet so it matches the tutorial. Severity is Low because the correcting paragraph is 11 lines away in the same section.

What's Missing

Tests:

• 🟡 🟠 The Marketplace slug coder_eval is now hardcoded in four places across the three changed files (README.md:4 badge target, README.md:101, docs/CI_GATE.md:19, docs/tutorials/02-ci-pipeline.md:165) and is derived from action.yml's name: — which the immediately preceding commit (a9c274d) renamed. Nothing links the two: a future name: edit silently 404s all four links, and no external-link check exists in CI (no lychee / markdown-link-check workflow; CE027–CE031 cover env-var, YAML-example, schema and index parity, not URLs or the badge row). Missing: a CE032-style whole-tree rule asserting every github.com/marketplace/actions/<slug> URL in Markdown equals the slugified action.yml name: (plus the shields label coder__eval escaping), or at minimum a link-check job over changed Markdown. (trigger: README.md)

Parallel paths:

• 🔵 🟡 The Marketplace mention landed only inside ## The GitHub Action (docs/CI_GATE.md:18-19); the page's front-matter description: (lines 2-4, which is what the coder-eval.com site renders as the meta/search snippet) and its lede (line 9) still say only "a composite GitHub Action" — the discovery surface the PR is trying to improve is the one left unchanged. (trigger: docs/CI_GATE.md)
• 🔵 🟡 The word "Marketplace" exists only in three body paragraphs plus the new badge. mkdocs.yml's extra.docs_index blurb for CI_GATE ("the packaged Action, JUnit output, score floor") is the SSOT for README.md:193, docs/index.md:88 and docs/llms.txt:35, and none of those flat index surfaces — including the LLM-facing docs/llms.txt — was regenerated to mention it. One blurb edit + make docs-indexes propagates to all three (hand-editing the generated tables would trip CE028). (trigger: README.md)
• 🔵 🟡 The three rewritten intros now front three snippets that disagree on prerequisites: the tutorial's (lines 170-178) includes actions/setup-node + npm install -g @anthropic-ai/claude-code inline, while README.md:106-113 and docs/CI_GATE.md:23-30 are copy-pasteable steps with the agent-runtime note pushed 11 lines (CI_GATE:32-35) and ~50 lines (README:158-161) away. Same root cause as the Axis 8 finding — fixing the "nothing to install" sentence should also align all three snippets. (trigger: docs/tutorials/02-ci-pipeline.md) (restates: Axis 8: docs/CI_GATE.md "there is nothing to install" contradicts the agent-runtime prerequisite)

Downstream consumers:

• 🔵 🔵 pyproject.toml sets readme = "README.md", so the badge row is also the PyPI project-page header: the Marketplace badge appears and the Docs badge disappears there only when the next release cuts, leaving the published 0.9.4 page inconsistent with the repo in the interim. The Docs badge's role is otherwise covered (README.md:25 keeps the 📚 Full docs link and pyproject.toml:56 keeps Documentation = https://coder-eval.com/docs) — verified safe, but the release lag is unstated. (trigger: README.md)

Daily/nightly:

• 🔵 🔵 No code, workflow, or action.yml change, so the CI-gate and nightly run paths are behaviorally untouched — but docs/CI_GATE.md:18 now asserts publication as fact while .github/workflows/release.yml:251-255 documents the Marketplace checkbox as a one-time manual GitHub-UI step that no job verifies. The docs are now coupled to out-of-band listing state (published + still named coder_eval) with no guard on either side; the PR doesn't state that coupling, and the CE032 rule above would only cover the internal action.yml half. (trigger: docs/CI_GATE.md)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] New CE026 (next free number; CE025 and CE027-CE031 are taken) — "Action quickstart snippets must be self-sufficient". New helper module tests/lint/action_snippet_parity.py reusing the fenced-block extractor pattern already in tests/lint/doc_examples.py, wired as a dedicated @pytest.mark.lint class tests/test_custom_lint.py::TestCE026ActionSnippetPrereqs — NOT a BaseRule in tests/lint/runner.py, since that runner is AST-only over .py files while this rule reasons over Markdown (precedent: CE027/CE028/CE029). Forbidden pattern: the FIRST fenced ```yaml block on any .md page under `docs/` or in `README.md` that contains a `uses: /coder_eval@…` step but does not also contain the agent-runtime prerequisite steps (`actions/setup-node` + `npm install -g @anthropic-ai/claude-code`); later illustrative blocks on the same page (single-input examples such as README.md:144, docs/CI_GATE.md:69) are skipped so the rule stays quiet, and an intentionally-partial primary block can opt out with ``. Derive the required prerequisite set from the executable reference that already exists — the `Action Dogfood` job at `.github/workflows/pr-checks.yml:842-872`, which installs Node + the Claude CLI before `uses: ./` — so docs are pinned to a snippet CI proves green rather than to a hand-maintained constant. ruff/pyright/bandit have zero reach here (pure Markdown), which is exactly why this belongs in the CEnnn doc-surface family. Prevents: docs/CI_GATE.md:23-30 (copy-pasteable primary snippet omits setup-node + `npm install -g @anthropic-ai/claude-code`, so an integrator who copies it fails on a missing `claude` binary) and the identical omission at README.md:107-113; passes docs/tutorials/02-ci-pipeline.md:170-180, which already shows both steps — i.e. it mechanically closes the drift between the two onboarding surfaces the finding calls out.
• [ce-lint] Second clause inside the same CE026 module — "no unqualified zero-install absolute next to an Action snippet". Phrase denylist (nothing to install, no setup required, nothing to configure, no prerequisites, zero setup) applied only to .md prose within ~15 lines of a uses: <owner>/coder_eval@… block, satisfied when the same sentence carries a scoping qualifier (Marketplace, marketplace install step, no PyPI install) or the line carries <!-- lint-skip: absolute-claim: <reason> -->. Boundary stated deliberately: this catches the phrase, not the contradiction — judging that a paragraph 11 lines later states a real prerequisite is semantic reasoning no static rule should attempt, so the rule instead forces the absolute to be scoped where it is written. Prevents: docs/CI_GATE.md:21 — the unqualified "there is nothing to install" that contradicts the agent-agnostic prerequisite note at docs/CI_GATE.md:32-35; forces the narrower wording the finding recommends ("there is no Marketplace install step").

Harness improvements (not statically reachable):

• Fail-fast agent-runtime preflight on the run path. src/coder_eval/cli/utils.py::check_tools already probes shutil.which("claude") but is (a) hard-coded to claude/uv regardless of the resolved agent.type, (b) wired only into plan_command.py, and (c) merely prints a yellow warning. Replace it with preflight_agent_runtime(resolved_tasks) that maps each resolved agent type to its required binary through AgentRegistry (so plugin agents declare their own), is called from coder-eval run before the first sandbox is built, and exits non-zero with the literal remediation lines (actions/setup-node@v4 + npm install -g @anthropic-ai/claude-code), emitted as ::error:: when GITHUB_ACTIONS=true. That turns the doc ambiguity into a 2-second self-explaining failure instead of an opaque mid-run crash. Why not static: Needs the execution-time PATH of the CI runner plus the post-merge resolved agent.type per task; neither exists at lint time. Prevents: docs/CI_GATE.md:21 and :23-30 — the concrete failure mode behind the finding (integrator copies the snippet, run dies on a missing claude binary with no pointer to the fix).
• Dogfood the documented snippet instead of a parallel hand-written copy. Extend the existing Action Dogfood job (.github/workflows/pr-checks.yml:836-872), or add a light docs-snippet-smoke job, that extracts the first uses: …/coder_eval@… yaml block from docs/CI_GATE.md and README.md, rewrites uses: to ./, and executes exactly those steps (against tasks/hello_date.yaml with minimum-task-score: "0.0", as the current job does). Today the workflow is right and the docs are wrong precisely because nothing executes the docs; this makes the published snippet the thing under test. Why not static: Only running the extracted steps in a clean runner proves the prerequisite set is complete; a static rule can assert the two known steps (CE026) but cannot discover a third prerequisite introduced later. Prevents: docs/CI_GATE.md:23-30 and README.md:107-113 — future divergence between the copy-pasteable doc snippet and what a fresh runner actually needs.
• Single-source the Action quickstart snippet the way the docs index is already single-sourced (CE028 / make docs-indexes). Keep one canonical block in docs/_snippets/action-quickstart.yml, inject it into README.md, docs/CI_GATE.md and docs/tutorials/02-ci-pipeline.md between <!-- action-snippet:start --> / <!-- action-snippet:end --> markers via a new make docs-snippets target (mirroring Makefile:33-34), and have CE026 assert the injected regions match the source after dedent-normalization (the tutorial nests the block inside a workflow, so compare dedented text, not bytes). Why not static: The parity check is static, but the generator, markers and new make target are a dev-workflow change; and deciding which snippet is canonical (and that every surface must show prerequisites) is a policy call a lint rule can only enforce once made. Prevents: The class behind this finding: three independently hand-maintained onboarding surfaces for the same Action drifting (docs/CI_GATE.md:24 vs README.md:107 vs docs/tutorials/02-ci-pipeline.md:170-180).

Top 5 Priority Actions

1. Narrow the absolute claim at docs/CI_GATE.md:21 ("there is nothing to install") and add the actions/setup-node + npm install -g @anthropic-ai/claude-code steps to the copy-pasteable snippet at docs/CI_GATE.md:23-30 so it matches docs/tutorials/02-ci-pipeline.md:170-172 — a CI integrator who copies it verbatim gets a non-SUCCESS run from a missing claude binary regardless of agent quality, i.e. the doc changes final_status for identical agent output.
2. Record the effective stratified draw (resolved seed plus the selected row ids) at run level in run.json where the sample is taken at /Users/religa/src/coder_eval/src/coder_eval/orchestration/task_loader.py:403, since dataset.sample_seed=None re-draws every run and a suite's aggregate score therefore moves run-to-run for byte-identical agent behavior with no artifact trace of which rows were scored.
3. Stamp a machine-readable "metrics truncated by pass-stop" marker on the suite rollup / gate output when the run ended via the pass-stop floor path (/Users/religa/src/coder_eval/src/coder_eval/orchestration/early_stop.py:444), so weighted armed-subset gating (armed_criteria_passed) can never be mistaken downstream for authoritative classification P/R/F1 — today that caveat lives only in prose.
4. Add a doc-surface lint rule (CE032, wired as a dedicated @pytest.mark.lint class next to CE027-CE031 in /Users/religa/src/coder_eval/tests/test_custom_lint.py) asserting the CI_GATE.md and tutorial-02 action snippets agree on agent-runtime prerequisites, converting this one-time doc fix into permanent enforcement per the CLAUDE.md "could a lint rule have prevented this?" convention.
5. Extend that same doc-parity check to assert the inputs table at docs/CI_GATE.md:39-52 stays in sync with the ten declared inputs in /Users/religa/src/coder_eval/action.yml:25-77 (currently correct) so the published Action's contract cannot silently drift from its reference page as new inputs land.

---

Stats: 0 🔴 · 0 🟠 · 0 🟡 · 1 🔵 across 8 axes reviewed.

[uipreliga]

Thanks — addressed in 86e048c. Summary of what changed and what I deliberately left out.

Fixed: the Axis 8 nit (priority 1)

You were right that this was the one real defect, and that it was a truthfulness bug rather than a wording nit: docs/CI_GATE.md promised "there is nothing to install" directly above a copy-pasteable step that would die on a missing claude binary, with the correction 11 lines away and the tutorial's sibling snippet already showing the steps.

• Narrowed the absolute to "there is no Marketplace install step" — the claim I actually meant.
• Added actions/setup-node + npm install -g @anthropic-ai/claude-code to the primary snippets in docs/CI_GATE.md and README.md. All three onboarding surfaces now agree with docs/tutorials/02-ci-pipeline.md, which had it right.
• Reworded the README agent-runtime callout to point at those steps ("which is why the example above starts with…") instead of restating a prerequisite ~50 lines below the snippet it applies to.

Fixed: the discovery surfaces the first pass missed

Fair hit — the PR set out to improve discovery and skipped the surfaces that actually do the discovering:

• docs/CI_GATE.md front-matter description: (what the site renders as the meta/search snippet) and the page lede now name the listing.
• mkdocs.yml's extra.docs_index blurb for CI_GATE now says "the Marketplace Action", propagated to README.md, docs/index.md and docs/llms.txt via make docs-indexes (not hand-edited — CE028 would have caught that).

Fixed: the missing test, as CE026

Took the "could a lint rule have prevented this?" prompt literally. New doc-surface rule in tests/lint/action_docs.py + TestCE026ActionDocSurfaces (13 tests), wired as a @pytest.mark.lint class alongside CE027–CE031 rather than a BaseRule — same reason you gave: the runner is AST-only over .py, this reasons over Markdown + YAML. Three clauses:

1. Prerequisite parity — a page's first Action snippet must show the agent-runtime steps; later single-input blocks are skipped so the rule stays quiet. Opt out with <!-- lint-skip: action-prereq: <reason> -->.
2. No unqualified zero-install absolute near such a snippet. As you noted, this catches the phrase, not the contradiction — deciding whether a paragraph 11 lines later states a real prerequisite is semantic reasoning a static rule shouldn't attempt, so it forces the absolute to be scoped where it's written.
3. Marketplace slug parity — every github.com/marketplace/actions/<slug> link and the shields badge label must match action.yml's name:. This closes the four-places-hardcoded gap: the badge check decodes shields escaping (coder__eval → coder_eval), so writing the slug the naive way is now a lint failure rather than a badge that silently reads "coder eval".

One deviation from your sketch, and I think it's the better trade: rather than deriving the required steps by parsing the steps that precede uses: ./ in the dogfood job, REQUIRED_PREREQ_TOKENS is an explicit constant that a separate test pins to that job. Deriving would have swept in ambient steps (actions/checkout) that no doc fragment should be required to show, and the tokens-vs-allowlist hackery costs more than it buys. The rot-protection is identical — rename or drop the step in pr-checks.yml and the test fails, naming both places to update.

Verified the rule flags the pre-fix README.md:106 and docs/CI_GATE.md:21/:23 at exactly the lines this review reported, and is clean afterwards.

Deliberately not in this PR

Top 5 items 2 and 3 are real and I'd like them, but they're pre-existing harness behavior rather than anything this branch introduced, and both change the run artifact — wrong thing to smuggle into a badge PR:

• Record the effective stratified draw (resolved seed + selected row ids) at run level where the sample is taken (orchestration/task_loader.py:403). Agreed on the substance: with dataset.sample_seed=None a suite's aggregate moves run-to-run for byte-identical agent behavior and the artifact has no trace of which rows were scored.
• Stamp a machine-readable "metrics truncated by pass-stop" marker on the suite rollup / gate output (orchestration/early_stop.py:444), so weighted armed-subset gating can't be mistaken downstream for authoritative P/R/F1. That caveat currently lives only in prose.

Also deferring the two larger harness suggestions, both worth doing on their own:

• preflight_agent_runtime(resolved_tasks) replacing the plan-only, hardcoded, warning-level check_tools — turning this doc ambiguity into a 2-second self-explaining failure with ::error:: under GITHUB_ACTIONS. CE026 makes the docs correct; a preflight makes the harness correct regardless of which snippet someone copied. That's the stronger fix and deserves its own PR.
• Dogfooding the documented snippet (extract the first uses: …/coder_eval@… block, rewrite to ./, execute) instead of a parallel hand-written copy. Your framing is exactly right — today the workflow is correct and the docs were wrong precisely because nothing executes the docs. CE026 can only assert the two prerequisites we know about; only a real run discovers a third.

Happy to file those four as issues if you want them tracked.

make format / check / lint / test: clean — 3707 tests, 171 lint. make typecheck reports 3 pre-existing openai_codex import errors (optional extra not installed locally), untouched here.