From af417a15631e432400e32a2b36aad53a0c66c821 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Tue, 4 Aug 2026 11:48:08 -0700 Subject: [PATCH 1/3] docs(readme): add Marketplace badge, drop redundant Docs badge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The action is now listed at github.com/marketplace/actions/coder_eval, so the badge row can point at it. Slug uses `coder__eval` β€” shields.io renders a doubled underscore as a single one. Drops the Docs badge to keep the row at six: it pointed at coder-eval.com/docs, which the Website badge already covers at the domain level and which the intro links prominently two lines below ("πŸ“š Full docs: coder-eval.com/docs"), so nothing becomes harder to find. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ab7599ab..d732a1fb 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # Coder Eval β€” evaluate & benchmark AI coding agents and Claude Code skills [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/) +[![GitHub Marketplace](https://img.shields.io/badge/marketplace-coder__eval-2ea44f.svg)](https://github.com/marketplace/actions/coder_eval) [![Website](https://img.shields.io/badge/website-coder--eval.com-1f6feb.svg)](https://coder-eval.com) -[![Docs](https://img.shields.io/badge/docs-coder--eval.com%2Fdocs-1f6feb.svg)](https://coder-eval.com/docs) [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE) [![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/) [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml) From 58d3571e3a65f434b10c40aba3bb9cca9e491496 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Tue, 4 Aug 2026 11:52:36 -0700 Subject: [PATCH 2/3] docs: surface the Marketplace listing in the Action docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The action is now published as `coder_eval` on the GitHub Actions Marketplace, but nothing in the docs said so β€” the three places that introduce the Action described it only as "a composite action at the repo root", which is true but hid the discovery surface. Adds the listing link to README "Use as a GitHub Action", the CI Gate reference, and tutorial 02's "Shortcut: the packaged action". Each keeps the `uses: UiPath/coder_eval@v0` repo-path form, since that is still how the action is referenced β€” being on the Marketplace changes discovery, not consumption, and there is nothing to install. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 8 +++++--- docs/CI_GATE.md | 6 ++++-- docs/tutorials/02-ci-pipeline.md | 6 ++++-- 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index d732a1fb..c51d12a1 100644 --- a/README.md +++ b/README.md @@ -97,9 +97,11 @@ the full setup. ## Use as a GitHub Action -A composite action at the repo root runs `coder-eval` as a CI gate β€” it installs -the pinned CLI, runs your tasks, writes a JUnit XML report, appends `run.md` to -the job summary, and fails the step on any task/gate failure: +A composite action β€” on the Marketplace as +[**coder_eval**](https://github.com/marketplace/actions/coder_eval) β€” runs +`coder-eval` as a CI gate. It installs the pinned CLI, runs your tasks, writes a +JUnit XML report, appends `run.md` to the job summary, and fails the step on any +task/gate failure: ```yaml - uses: UiPath/coder_eval@v0 # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly diff --git a/docs/CI_GATE.md b/docs/CI_GATE.md index 72791b69..867815b5 100644 --- a/docs/CI_GATE.md +++ b/docs/CI_GATE.md @@ -15,8 +15,10 @@ reference for the Action and the JUnit output. For a step-by-step walkthrough ## The GitHub Action -A composite action lives at the repo root (`action.yml`), so you can reference it -directly: +The action is published on the GitHub Actions Marketplace as +[**coder_eval**](https://github.com/marketplace/actions/coder_eval). It is a +composite action living at the repo root (`action.yml`), so you reference it by +repo path β€” there is nothing to install: ```yaml - uses: UiPath/coder_eval@v0 # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly diff --git a/docs/tutorials/02-ci-pipeline.md b/docs/tutorials/02-ci-pipeline.md index ee960a36..153ca86b 100644 --- a/docs/tutorials/02-ci-pipeline.md +++ b/docs/tutorials/02-ci-pipeline.md @@ -161,8 +161,10 @@ jobs: ## Shortcut: the packaged action The five steps above spell out the mechanics, but Coder Eval also ships a -composite action at the repo root that bundles install + run + JUnit report + -job-summary + fail-on-failure into one step: +composite action β€” on the Marketplace as +[**coder_eval**](https://github.com/marketplace/actions/coder_eval) β€” that +bundles install + run + JUnit report + job-summary + fail-on-failure into one +step: ```yaml - uses: actions/setup-node@v4 # the claude-code agent needs the Claude CLI… From 86e048cff567b980b77995b3cd4a8bdc4ae98f9e Mon Sep 17 00:00:00 2001 From: uipreliga Date: Tue, 4 Aug 2026 13:05:02 -0700 Subject: [PATCH 3/3] docs: make the Action quickstarts self-sufficient, add CE026 to enforce it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses the PR review on #80. The finding: docs/CI_GATE.md said "there is nothing to install" above a copy-pasteable `uses:` step with no agent runtime. The action is agent-agnostic, so an integrator who copied it got a run that dies on a missing `claude` binary β€” the correcting note was 11 lines away, and the tutorial's sibling snippet already showed the steps. - Narrow the claim to "there is no Marketplace install step" and add the `actions/setup-node` + `npm install -g @anthropic-ai/claude-code` steps to the primary snippets in docs/CI_GATE.md and README.md, so all three onboarding surfaces now agree with docs/tutorials/02-ci-pipeline.md. - Reword the README agent-runtime callout to point at those steps instead of restating them ~50 lines later. - Surface the Marketplace listing on the discovery surfaces the first pass missed: CI_GATE.md's front-matter `description:` and lede, plus the `extra.docs_index` blurb (regenerated into README / docs/index.md / docs/llms.txt via `make docs-indexes`). CE026 turns the one-time fix into enforcement (doc-surface rule alongside CE027-CE031, not a BaseRule β€” it reasons over Markdown + YAML): 1. a page's *first* Action snippet must show the agent-runtime prerequisites, pinned to the `action-dogfood` job that proves them green in CI rather than to a hand-maintained constant; 2. a zero-install absolute next to such a snippet must name the channel it means (catches the phrase, not the contradiction β€” semantic reasoning is out of scope for a static rule); 3. every `github.com/marketplace/actions/` link and the shields badge label must match `action.yml`'s `name:`, which a rename would otherwise 404 in four places at once. Verified the rule flags the pre-fix README.md and docs/CI_GATE.md at exactly the reported lines and is clean afterwards. make format/check/lint/test all pass (3707 tests, 171 lint). Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 2 +- README.md | 16 ++- docs/CI_GATE.md | 32 ++++-- docs/index.md | 2 +- docs/llms.txt | 2 +- mkdocs.yml | 2 +- tests/lint/action_docs.py | 232 ++++++++++++++++++++++++++++++++++++++ tests/test_custom_lint.py | 163 ++++++++++++++++++++++++++ 8 files changed, 430 insertions(+), 21 deletions(-) create mode 100644 tests/lint/action_docs.py diff --git a/CLAUDE.md b/CLAUDE.md index 664499d3..db8d2615 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -200,7 +200,7 @@ make lint # custom architectural lint rules (CE001+) make verify # All of the above + coverage check (CI equivalent) ``` -When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time β€” CE027–CE031 β€” are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name.) +When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time β€” CE026–CE031 β€” are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's three onboarding surfaces honest: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, and every `github.com/marketplace/actions/` link plus the shields badge label must match `action.yml`'s `name:`.) Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig` β€” see `tests/lint/doc_schema_parity.py`) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise. diff --git a/README.md b/README.md index c51d12a1..d3b9a2e5 100644 --- a/README.md +++ b/README.md @@ -104,7 +104,11 @@ JUnit XML report, appends `run.md` to the job summary, and fails the step on any task/gate failure: ```yaml -- uses: UiPath/coder_eval@v0 # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly +- uses: actions/setup-node@v4 # the claude-code agent needs the Claude CLI… + with: { node-version: '20' } +- run: npm install -g @anthropic-ai/claude-code + +- uses: UiPath/coder_eval@v0 # …then run the gate (@v1 once 1.0.0 ships; @vX.Y.Z pins exactly) with: tasks: tests/tasks/**/*.yaml model: claude-sonnet-5 @@ -156,9 +160,11 @@ code: the step fails if *either* coder-eval exits non-zero *or* any task's alone. > **Agent runtime is the caller's responsibility.** The action is agent-agnostic β€” -> it installs `coder-eval` but no coding-agent runtime. Tasks using the default -> `claude-code` agent need the `claude` CLI on `PATH` (`actions/setup-node` + -> `npm install -g @anthropic-ai/claude-code`) in the job before the action runs. +> it installs `coder-eval` but no coding-agent runtime, which is why the example +> above starts with `actions/setup-node` + +> `npm install -g @anthropic-ai/claude-code`: the default `claude-code` agent +> needs the `claude` CLI on `PATH` before the action runs. Swap those steps for +> your own agent's runtime as needed. > **Security.** Evaluated tasks execute agent-generated code. Do **not** run this > action under `pull_request_target` with secrets exposed to untrusted fork PRs β€” @@ -190,7 +196,7 @@ alone. | [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset | | [Dialog Mode](docs/DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user | | [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver, with custom images | -| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate β€” the packaged Action, JUnit output, score floor | +| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate β€” the Marketplace Action, JUnit output, score floor | | [Extending Coder Eval](docs/EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI | | [Report Schema](docs/REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json | | [How It Compares](docs/comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts | diff --git a/docs/CI_GATE.md b/docs/CI_GATE.md index 867815b5..857fd9b4 100644 --- a/docs/CI_GATE.md +++ b/docs/CI_GATE.md @@ -1,15 +1,18 @@ --- description: >- - Run Coder Eval as a CI gate β€” the packaged composite GitHub Action, JUnit XML - output for test-report ingestion, and an optional per-task score floor. + Run Coder Eval as a CI gate β€” the coder_eval GitHub Action from the Actions + Marketplace, JUnit XML output for test-report ingestion, and an optional + per-task score floor. --- # CI Gate: GitHub Action & JUnit reports -Coder Eval ships a **packaged CI gate**: a composite GitHub Action that installs -the CLI, runs your tasks, emits a JUnit XML report, appends the run summary to the -job summary, and fails the build on any task/gate failure. This page is the -reference for the Action and the JUnit output. For a step-by-step walkthrough +Coder Eval ships a **packaged CI gate**: a composite GitHub Action β€” on the +Actions Marketplace as +[**coder_eval**](https://github.com/marketplace/actions/coder_eval) β€” that +installs the CLI, runs your tasks, emits a JUnit XML report, appends the run +summary to the job summary, and fails the build on any task/gate failure. This +page is the reference for the Action and the JUnit output. For a walkthrough (including a hand-rolled workflow), see [Tutorial 02 β€” Running Coder Eval in CI](tutorials/02-ci-pipeline.md). @@ -18,10 +21,14 @@ reference for the Action and the JUnit output. For a step-by-step walkthrough The action is published on the GitHub Actions Marketplace as [**coder_eval**](https://github.com/marketplace/actions/coder_eval). It is a composite action living at the repo root (`action.yml`), so you reference it by -repo path β€” there is nothing to install: +repo path β€” there is no Marketplace install step: ```yaml -- uses: UiPath/coder_eval@v0 # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly +- uses: actions/setup-node@v4 # the claude-code agent needs the Claude CLI… + with: { node-version: '20' } +- run: npm install -g @anthropic-ai/claude-code + +- uses: UiPath/coder_eval@v0 # …then run the gate (@v1 once 1.0.0 ships; @vX.Y.Z pins exactly) with: tasks: tests/tasks/**/*.yaml model: claude-sonnet-5 @@ -29,10 +36,11 @@ repo path β€” there is nothing to install: ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }} ``` -The action is **agent-agnostic** β€” it installs `coder-eval` but *not* any -coding-agent runtime. Tasks using the default `claude-code` agent need the -`claude` CLI on `PATH` (Node + `@anthropic-ai/claude-code`), provided by your job -*before* this step runs. +The first two steps are there because the action is **agent-agnostic** β€” it +installs `coder-eval` but *not* any coding-agent runtime. Tasks using the default +`claude-code` agent need the `claude` CLI on `PATH` (Node + +`@anthropic-ai/claude-code`), provided by your job *before* the action runs; swap +those steps for your own agent's runtime as needed. ### Inputs diff --git a/docs/index.md b/docs/index.md index 09656e2a..d928436e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -85,7 +85,7 @@ New here? Start with **[Tutorial 01 β€” Your First Evaluation](tutorials/01-firs | [Bring Your Own Dataset](DATASETS.md) | Fan a single task out over a dataset | | [Dialog Mode](DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user | | [Docker Isolation](DOCKER_ISOLATION.md) | The container sandbox driver, with custom images | -| [CI Gate & GitHub Action](CI_GATE.md) | Run Coder Eval as a CI gate β€” the packaged Action, JUnit output, score floor | +| [CI Gate & GitHub Action](CI_GATE.md) | Run Coder Eval as a CI gate β€” the Marketplace Action, JUnit output, score floor | | [Extending Coder Eval](EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI | | [Report Schema](REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json | | [How It Compares](comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts | diff --git a/docs/llms.txt b/docs/llms.txt index ae33be28..260f6fe0 100644 --- a/docs/llms.txt +++ b/docs/llms.txt @@ -32,7 +32,7 @@ and A/B plumbing. - [Bring Your Own Dataset](https://coder-eval.com/docs/datasets): Fan a single task out over a dataset - [Dialog Mode](https://coder-eval.com/docs/dialog-mode): Evaluate agents in multi-turn conversation via a simulated user - [Docker Isolation](https://coder-eval.com/docs/docker-isolation): The container sandbox driver, with custom images -- [CI Gate & GitHub Action](https://coder-eval.com/docs/ci-gate): Run Coder Eval as a CI gate β€” the packaged Action, JUnit output, score floor +- [CI Gate & GitHub Action](https://coder-eval.com/docs/ci-gate): Run Coder Eval as a CI gate β€” the Marketplace Action, JUnit output, score floor - [Extending Coder Eval](https://coder-eval.com/docs/extending): Author a custom agent, criterion, or model pricing via the plugin SPI - [Report Schema](https://coder-eval.com/docs/report-schema): Field-level reference for run.json / variant.json / task.json - [How It Compares](https://coder-eval.com/docs/comparison): vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts diff --git a/mkdocs.yml b/mkdocs.yml index bf2f2e07..d1da7828 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -87,7 +87,7 @@ extra: DATASETS.md: "Fan a single task out over a dataset" DIALOG_MODE.md: "Evaluate agents in multi-turn conversation via a simulated user" DOCKER_ISOLATION.md: "The container sandbox driver, with custom images" - CI_GATE.md: "Run Coder Eval as a CI gate β€” the packaged Action, JUnit output, score floor" + CI_GATE.md: "Run Coder Eval as a CI gate β€” the Marketplace Action, JUnit output, score floor" EXTENDING.md: "Author a custom agent, criterion, or model pricing via the plugin SPI" REPORT_SCHEMA.md: "Field-level reference for run.json / variant.json / task.json" comparison.md: "vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts" diff --git a/tests/lint/action_docs.py b/tests/lint/action_docs.py new file mode 100644 index 00000000..017fa4d7 --- /dev/null +++ b/tests/lint/action_docs.py @@ -0,0 +1,232 @@ +"""CE026 β€” the GitHub Action's onboarding surfaces must stay truthful and self-sufficient. + +Three doc surfaces introduce the same composite Action (``README.md``, +``docs/CI_GATE.md``, ``docs/tutorials/02-ci-pipeline.md``) and each was hand-maintained, +so they drifted. The motivating bug: ``docs/CI_GATE.md`` claimed "there is nothing to +install" and offered a copy-pasteable ``uses:`` step with no agent runtime β€” the action +is agent-agnostic, so an integrator who copied it got a run that dies on a missing +``claude`` binary. The correcting paragraph was 11 lines away; the tutorial's snippet +showed the prerequisite steps; the reference page's did not. + +Three clauses, all mechanical: + +1. **Prerequisite parity.** The *first* fenced ``yaml`` block on a doc page that + references the action (``uses: /coder_eval@…``) is the page's quickstart, so + it must also show the agent-runtime steps. Later blocks on the same page are + single-input illustrations and are skipped, which is what keeps the rule quiet. +2. **No unqualified zero-install absolute** in prose near such a block. This catches + the *phrase*, not the *contradiction*: judging whether a paragraph 11 lines later + states a real prerequisite is semantic reasoning no static rule should attempt, so + the rule instead forces the absolute to be scoped where it is written + ("no *Marketplace* install step"). +3. **Marketplace slug parity.** Every ``github.com/marketplace/actions/`` link and + the shields badge label must match ``action.yml``'s ``name:`` β€” the listing title, + which a rename would silently 404 in four places at once. + +Like CE027-CE031 this is deliberately NOT a ``BaseRule`` in ``tests/lint/runner.py``: +that runner is AST-only over ``.py`` files, whereas this rule reasons over Markdown and +YAML. It is wired as ``tests/test_custom_lint.py::TestCE026ActionDocSurfaces``. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from pathlib import Path + +import yaml + +from tests.lint.doc_examples import extract_yaml_blocks + + +PREREQ_SKIP_MARKER = "` above the fence", + ) + ) + return findings + + +def find_unscoped_absolute_claims(paths: list[Path]) -> list[Finding]: + """Flag an unqualified "nothing to install" in prose next to an Action snippet.""" + findings: list[Finding] = [] + for path in paths: + text = path.read_text(encoding="utf-8") + lines = text.splitlines() + snippet_lines = [i for i, line in enumerate(lines) if _ACTION_USES.search(line)] + if not snippet_lines: + continue + for i, line in enumerate(lines): + lowered = line.lower() + claim = next((c for c in ABSOLUTE_CLAIMS if c in lowered), None) + if claim is None or CLAIM_SKIP_MARKER in line: + continue + if any(q in lowered for q in CLAIM_QUALIFIERS): + continue + if not any(abs(i - s) <= CLAIM_PROXIMITY_LINES for s in snippet_lines): + continue + findings.append( + Finding( + path, + i + 1, + f'unqualified absolute "{claim}" next to an Action snippet β€” the action installs ' + "no coding-agent runtime, so this reads as 'no prerequisites'. Scope it to the " + f"channel you mean (e.g. 'no Marketplace install step') or add " + f"`{CLAIM_SKIP_MARKER}: -->` on the line", + ) + ) + return findings + + +def action_listing_name(action_yml: Path) -> str: + """The Marketplace listing title declared by ``action.yml``'s ``name:``.""" + data = yaml.safe_load(action_yml.read_text(encoding="utf-8")) + name = data.get("name") + if not isinstance(name, str) or not name.strip(): + raise AssertionError(f"{action_yml} declares no usable `name:` β€” the Marketplace listing title") + return name.strip() + + +def marketplace_slug(listing_name: str) -> str: + """GitHub's Marketplace URL slug for a listing title.""" + slug = re.sub(r"\s+", "-", listing_name.strip().lower()) + return re.sub(r"[^a-z0-9._-]", "", slug) + + +def decode_shields_label(label: str) -> str: + """Decode a shields.io badge label back to its displayed text. + + shields renders ``_`` as a space and ``__`` / ``--`` as a literal ``_`` / ``-``, + which is why the badge for ``coder_eval`` must be written ``coder__eval``. + """ + placeholder_underscore, placeholder_hyphen = "\x00", "\x01" + decoded = label.replace("__", placeholder_underscore).replace("--", placeholder_hyphen) + decoded = decoded.replace("_", " ") + return decoded.replace(placeholder_underscore, "_").replace(placeholder_hyphen, "-") + + +def find_slug_mismatches(paths: list[Path], listing_name: str) -> list[Finding]: + """Flag Marketplace links/badges that disagree with ``action.yml``'s ``name:``.""" + expected_slug = marketplace_slug(listing_name) + findings: list[Finding] = [] + for path in paths: + for i, line in enumerate(path.read_text(encoding="utf-8").splitlines()): + for match in _MARKETPLACE_URL.finditer(line): + if match.group(1) != expected_slug: + findings.append( + Finding( + path, + i + 1, + f"Marketplace link slug {match.group(1)!r} does not match action.yml " + f"`name: {listing_name}` (expected {expected_slug!r}) β€” the link 404s", + ) + ) + for match in _SHIELDS_MARKETPLACE.finditer(line): + shown = decode_shields_label(match.group(1)) + if shown != listing_name: + findings.append( + Finding( + path, + i + 1, + f"Marketplace badge label {match.group(1)!r} displays as {shown!r}, not " + f"action.yml `name: {listing_name}` (shields renders a single `_` as a " + "space, so the doubled form is required)", + ) + ) + return findings + + +def dogfood_prereq_tokens(workflow: Path, job: str = DOGFOOD_JOB) -> set[str]: + """Tokens for every step the dogfood job runs before invoking the local action. + + This is the executable proof that ``REQUIRED_PREREQ_TOKENS`` is still the real + prerequisite set: CI runs these steps before ``uses: ./`` and the job passes. + """ + data = yaml.safe_load(workflow.read_text(encoding="utf-8")) + steps = data["jobs"][job]["steps"] + tokens: set[str] = set() + for step in steps: + uses = step.get("uses", "") + if uses.strip() in {"./", "."}: + break + if uses: + tokens.add(uses.split("@", 1)[0]) + run = step.get("run") + if isinstance(run, str): + tokens.update(line.strip() for line in run.splitlines() if line.strip()) + return tokens diff --git a/tests/test_custom_lint.py b/tests/test_custom_lint.py index a1f4da70..fe9ab6cc 100644 --- a/tests/test_custom_lint.py +++ b/tests/test_custom_lint.py @@ -1208,3 +1208,166 @@ def test_registered_fields_are_actually_consumed_on_the_real_tree(self): consumed = consumed_attr_names(self.SRC) for name in ("n_trials", "max_usd", "stratify_field"): assert name in consumed, f"expected {name!r} to be read as an attribute in src/" + + +@pytest.mark.lint +class TestCE026ActionDocSurfaces: + """CE026 β€” the Action's onboarding surfaces must be truthful and self-sufficient. + + The motivating bug: docs/CI_GATE.md said "there is nothing to install" above a + copy-pasteable `uses:` step with no agent runtime, while the correcting + prerequisite note sat 11 lines below and the tutorial's sibling snippet *did* + show the steps. An integrator who copied it got a run that dies on a missing + `claude` binary. Reasons over Markdown + YAML, so it lives here rather than in + the AST-only runner (precedent: CE027-CE031). + """ + + REPO_ROOT = Path(__file__).parent.parent + ACTION_YML = REPO_ROOT / "action.yml" + PR_CHECKS = REPO_ROOT / ".github" / "workflows" / "pr-checks.yml" + + def test_primary_action_snippets_show_agent_runtime_prereqs(self): + from tests.lint.action_docs import default_doc_paths, find_missing_prereqs + + findings = find_missing_prereqs(default_doc_paths(self.REPO_ROOT)) + assert not findings, ( + "\nAction quickstart snippet(s) that a reader can copy but that omit the agent-runtime " + "prerequisite steps:\n\n" + "\n".join(f" {f}" for f in findings) + ) + + def test_no_unqualified_zero_install_claims_near_action_snippets(self): + from tests.lint.action_docs import default_doc_paths, find_unscoped_absolute_claims + + findings = find_unscoped_absolute_claims(default_doc_paths(self.REPO_ROOT)) + assert not findings, ( + "\nUnqualified zero-install absolute(s) beside an Action snippet β€” scope the claim to the " + "channel it is about:\n\n" + "\n".join(f" {f}" for f in findings) + ) + + def test_marketplace_links_match_the_action_listing_name(self): + from tests.lint.action_docs import action_listing_name, default_doc_paths, find_slug_mismatches + + listing = action_listing_name(self.ACTION_YML) + findings = find_slug_mismatches(default_doc_paths(self.REPO_ROOT), listing) + assert not findings, ( + f"\nMarketplace link/badge(s) inconsistent with action.yml `name: {listing}` β€” a rename " + "404s every one of them:\n\n" + "\n".join(f" {f}" for f in findings) + ) + + def test_required_prereqs_match_the_dogfood_job(self): + # The constant is pinned to the executable reference: the dogfood job proves + # in CI that these steps are what a fresh runner needs before `uses: ./`. + from tests.lint.action_docs import DOGFOOD_JOB, REQUIRED_PREREQ_TOKENS, dogfood_prereq_tokens + + tokens = dogfood_prereq_tokens(self.PR_CHECKS) + for required in REQUIRED_PREREQ_TOKENS: + assert any(required in token for token in tokens), ( + f"{required!r} is documented as a prerequisite but the {DOGFOOD_JOB} job no longer " + f"runs it before `uses: ./` (job steps: {sorted(tokens)}). Update " + "REQUIRED_PREREQ_TOKENS and the doc snippets together." + ) + + def test_catches_a_snippet_missing_prereqs(self, tmp_path: Path): + from tests.lint.action_docs import find_missing_prereqs + + page = tmp_path / "page.md" + page.write_text( + "# Gate\n\n```yaml\n- uses: UiPath/coder_eval@v0\n with:\n tasks: t.yaml\n```\n", + encoding="utf-8", + ) + findings = find_missing_prereqs([page]) + assert len(findings) == 1 + assert "actions/setup-node" in findings[0].message + + def test_only_the_first_action_block_is_checked(self, tmp_path: Path): + # Later single-input illustrations must stay quiet, or the rule becomes a nuisance. + from tests.lint.action_docs import find_missing_prereqs + + page = tmp_path / "page.md" + page.write_text( + "```yaml\n" + "- uses: actions/setup-node@v4\n" + "- run: npm install -g @anthropic-ai/claude-code\n" + "- uses: UiPath/coder_eval@v0\n" + "```\n\n" + "Score floor:\n\n" + '```yaml\n- uses: UiPath/coder_eval@v0\n with:\n minimum-task-score: "0.8"\n```\n', + encoding="utf-8", + ) + assert find_missing_prereqs([page]) == [] + + def test_prereq_skip_marker_opts_out(self, tmp_path: Path): + from tests.lint.action_docs import find_missing_prereqs + + page = tmp_path / "page.md" + page.write_text( + "\n```yaml\n- uses: UiPath/coder_eval@v0\n```\n", + encoding="utf-8", + ) + assert find_missing_prereqs([page]) == [] + + def test_page_without_the_action_is_ignored(self, tmp_path: Path): + from tests.lint.action_docs import find_missing_prereqs + + page = tmp_path / "page.md" + page.write_text("```yaml\n- uses: actions/checkout@v6\n```\n", encoding="utf-8") + assert find_missing_prereqs([page]) == [] + + def test_catches_the_nothing_to_install_regression(self, tmp_path: Path): + # The exact sentence this rule exists for. + from tests.lint.action_docs import find_unscoped_absolute_claims + + page = tmp_path / "page.md" + page.write_text( + "you reference it by repo path β€” there is nothing to install:\n\n" + "```yaml\n- uses: UiPath/coder_eval@v0\n```\n", + encoding="utf-8", + ) + findings = find_unscoped_absolute_claims([page]) + assert len(findings) == 1 + assert "nothing to install" in findings[0].message + + def test_scoped_claim_is_allowed(self, tmp_path: Path): + from tests.lint.action_docs import find_unscoped_absolute_claims + + page = tmp_path / "page.md" + page.write_text( + "you reference it by repo path β€” there is no Marketplace install step:\n\n" + "```yaml\n- uses: UiPath/coder_eval@v0\n```\n", + encoding="utf-8", + ) + assert find_unscoped_absolute_claims([page]) == [] + + def test_claim_far_from_a_snippet_is_ignored(self, tmp_path: Path): + from tests.lint.action_docs import find_unscoped_absolute_claims + + page = tmp_path / "page.md" + page.write_text( + "there is nothing to install\n" + "\n" * 40 + "```yaml\n- uses: UiPath/coder_eval@v0\n```\n", + encoding="utf-8", + ) + assert find_unscoped_absolute_claims([page]) == [] + + def test_catches_a_renamed_listing(self, tmp_path: Path): + from tests.lint.action_docs import find_slug_mismatches + + page = tmp_path / "page.md" + page.write_text("[coder_eval](https://github.com/marketplace/actions/coder_eval)\n", encoding="utf-8") + assert find_slug_mismatches([page], "coder_eval") == [] + assert len(find_slug_mismatches([page], "coder eval x")) == 1 + + def test_shields_label_must_decode_to_the_listing_name(self, tmp_path: Path): + from tests.lint.action_docs import decode_shields_label, find_slug_mismatches + + # A single `_` renders as a space, which is why the badge needs the doubled form. + assert decode_shields_label("coder__eval") == "coder_eval" + assert decode_shields_label("coder_eval") == "coder eval" + + page = tmp_path / "page.md" + page.write_text( + "[![m](https://img.shields.io/badge/marketplace-coder_eval-2ea44f.svg)](https://x)\n", + encoding="utf-8", + ) + findings = find_slug_mismatches([page], "coder_eval") + assert len(findings) == 1 + assert "displays as 'coder eval'" in findings[0].message