diff --git a/CLAUDE.md b/CLAUDE.md index 664499d3..db8d2615 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -200,7 +200,7 @@ make lint # custom architectural lint rules (CE001+) make verify # All of the above + coverage check (CI equivalent) ``` -When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE027–CE031 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name.) +When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's three onboarding surfaces honest: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, and every `github.com/marketplace/actions/` link plus the shields badge label must match `action.yml`'s `name:`.) Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig` — see `tests/lint/doc_schema_parity.py`) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise. diff --git a/README.md b/README.md index ab7599ab..d3b9a2e5 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # Coder Eval — evaluate & benchmark AI coding agents and Claude Code skills [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/) +[![GitHub Marketplace](https://img.shields.io/badge/marketplace-coder__eval-2ea44f.svg)](https://github.com/marketplace/actions/coder_eval) [![Website](https://img.shields.io/badge/website-coder--eval.com-1f6feb.svg)](https://coder-eval.com) -[![Docs](https://img.shields.io/badge/docs-coder--eval.com%2Fdocs-1f6feb.svg)](https://coder-eval.com/docs) [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE) [![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/) [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml) @@ -97,12 +97,18 @@ the full setup. ## Use as a GitHub Action -A composite action at the repo root runs `coder-eval` as a CI gate — it installs -the pinned CLI, runs your tasks, writes a JUnit XML report, appends `run.md` to -the job summary, and fails the step on any task/gate failure: +A composite action — on the Marketplace as +[**coder_eval**](https://github.com/marketplace/actions/coder_eval) — runs +`coder-eval` as a CI gate. It installs the pinned CLI, runs your tasks, writes a +JUnit XML report, appends `run.md` to the job summary, and fails the step on any +task/gate failure: ```yaml -- uses: UiPath/coder_eval@v0 # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly +- uses: actions/setup-node@v4 # the claude-code agent needs the Claude CLI… + with: { node-version: '20' } +- run: npm install -g @anthropic-ai/claude-code + +- uses: UiPath/coder_eval@v0 # …then run the gate (@v1 once 1.0.0 ships; @vX.Y.Z pins exactly) with: tasks: tests/tasks/**/*.yaml model: claude-sonnet-5 @@ -154,9 +160,11 @@ code: the step fails if *either* coder-eval exits non-zero *or* any task's alone. > **Agent runtime is the caller's responsibility.** The action is agent-agnostic — -> it installs `coder-eval` but no coding-agent runtime. Tasks using the default -> `claude-code` agent need the `claude` CLI on `PATH` (`actions/setup-node` + -> `npm install -g @anthropic-ai/claude-code`) in the job before the action runs. +> it installs `coder-eval` but no coding-agent runtime, which is why the example +> above starts with `actions/setup-node` + +> `npm install -g @anthropic-ai/claude-code`: the default `claude-code` agent +> needs the `claude` CLI on `PATH` before the action runs. Swap those steps for +> your own agent's runtime as needed. > **Security.** Evaluated tasks execute agent-generated code. Do **not** run this > action under `pull_request_target` with secrets exposed to untrusted fork PRs — @@ -188,7 +196,7 @@ alone. | [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset | | [Dialog Mode](docs/DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user | | [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver, with custom images | -| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor | +| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, score floor | | [Extending Coder Eval](docs/EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI | | [Report Schema](docs/REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json | | [How It Compares](docs/comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts | diff --git a/docs/CI_GATE.md b/docs/CI_GATE.md index 72791b69..857fd9b4 100644 --- a/docs/CI_GATE.md +++ b/docs/CI_GATE.md @@ -1,25 +1,34 @@ --- description: >- - Run Coder Eval as a CI gate — the packaged composite GitHub Action, JUnit XML - output for test-report ingestion, and an optional per-task score floor. + Run Coder Eval as a CI gate — the coder_eval GitHub Action from the Actions + Marketplace, JUnit XML output for test-report ingestion, and an optional + per-task score floor. --- # CI Gate: GitHub Action & JUnit reports -Coder Eval ships a **packaged CI gate**: a composite GitHub Action that installs -the CLI, runs your tasks, emits a JUnit XML report, appends the run summary to the -job summary, and fails the build on any task/gate failure. This page is the -reference for the Action and the JUnit output. For a step-by-step walkthrough +Coder Eval ships a **packaged CI gate**: a composite GitHub Action — on the +Actions Marketplace as +[**coder_eval**](https://github.com/marketplace/actions/coder_eval) — that +installs the CLI, runs your tasks, emits a JUnit XML report, appends the run +summary to the job summary, and fails the build on any task/gate failure. This +page is the reference for the Action and the JUnit output. For a walkthrough (including a hand-rolled workflow), see [Tutorial 02 — Running Coder Eval in CI](tutorials/02-ci-pipeline.md). ## The GitHub Action -A composite action lives at the repo root (`action.yml`), so you can reference it -directly: +The action is published on the GitHub Actions Marketplace as +[**coder_eval**](https://github.com/marketplace/actions/coder_eval). It is a +composite action living at the repo root (`action.yml`), so you reference it by +repo path — there is no Marketplace install step: ```yaml -- uses: UiPath/coder_eval@v0 # becomes @v1 once 1.0.0 ships; @vX.Y.Z pins exactly +- uses: actions/setup-node@v4 # the claude-code agent needs the Claude CLI… + with: { node-version: '20' } +- run: npm install -g @anthropic-ai/claude-code + +- uses: UiPath/coder_eval@v0 # …then run the gate (@v1 once 1.0.0 ships; @vX.Y.Z pins exactly) with: tasks: tests/tasks/**/*.yaml model: claude-sonnet-5 @@ -27,10 +36,11 @@ directly: ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }} ``` -The action is **agent-agnostic** — it installs `coder-eval` but *not* any -coding-agent runtime. Tasks using the default `claude-code` agent need the -`claude` CLI on `PATH` (Node + `@anthropic-ai/claude-code`), provided by your job -*before* this step runs. +The first two steps are there because the action is **agent-agnostic** — it +installs `coder-eval` but *not* any coding-agent runtime. Tasks using the default +`claude-code` agent need the `claude` CLI on `PATH` (Node + +`@anthropic-ai/claude-code`), provided by your job *before* the action runs; swap +those steps for your own agent's runtime as needed. ### Inputs diff --git a/docs/index.md b/docs/index.md index 09656e2a..d928436e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -85,7 +85,7 @@ New here? Start with **[Tutorial 01 — Your First Evaluation](tutorials/01-firs | [Bring Your Own Dataset](DATASETS.md) | Fan a single task out over a dataset | | [Dialog Mode](DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user | | [Docker Isolation](DOCKER_ISOLATION.md) | The container sandbox driver, with custom images | -| [CI Gate & GitHub Action](CI_GATE.md) | Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor | +| [CI Gate & GitHub Action](CI_GATE.md) | Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, score floor | | [Extending Coder Eval](EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI | | [Report Schema](REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json | | [How It Compares](comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts | diff --git a/docs/llms.txt b/docs/llms.txt index ae33be28..260f6fe0 100644 --- a/docs/llms.txt +++ b/docs/llms.txt @@ -32,7 +32,7 @@ and A/B plumbing. - [Bring Your Own Dataset](https://coder-eval.com/docs/datasets): Fan a single task out over a dataset - [Dialog Mode](https://coder-eval.com/docs/dialog-mode): Evaluate agents in multi-turn conversation via a simulated user - [Docker Isolation](https://coder-eval.com/docs/docker-isolation): The container sandbox driver, with custom images -- [CI Gate & GitHub Action](https://coder-eval.com/docs/ci-gate): Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor +- [CI Gate & GitHub Action](https://coder-eval.com/docs/ci-gate): Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, score floor - [Extending Coder Eval](https://coder-eval.com/docs/extending): Author a custom agent, criterion, or model pricing via the plugin SPI - [Report Schema](https://coder-eval.com/docs/report-schema): Field-level reference for run.json / variant.json / task.json - [How It Compares](https://coder-eval.com/docs/comparison): vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts diff --git a/docs/tutorials/02-ci-pipeline.md b/docs/tutorials/02-ci-pipeline.md index ee960a36..153ca86b 100644 --- a/docs/tutorials/02-ci-pipeline.md +++ b/docs/tutorials/02-ci-pipeline.md @@ -161,8 +161,10 @@ jobs: ## Shortcut: the packaged action The five steps above spell out the mechanics, but Coder Eval also ships a -composite action at the repo root that bundles install + run + JUnit report + -job-summary + fail-on-failure into one step: +composite action — on the Marketplace as +[**coder_eval**](https://github.com/marketplace/actions/coder_eval) — that +bundles install + run + JUnit report + job-summary + fail-on-failure into one +step: ```yaml - uses: actions/setup-node@v4 # the claude-code agent needs the Claude CLI… diff --git a/mkdocs.yml b/mkdocs.yml index bf2f2e07..d1da7828 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -87,7 +87,7 @@ extra: DATASETS.md: "Fan a single task out over a dataset" DIALOG_MODE.md: "Evaluate agents in multi-turn conversation via a simulated user" DOCKER_ISOLATION.md: "The container sandbox driver, with custom images" - CI_GATE.md: "Run Coder Eval as a CI gate — the packaged Action, JUnit output, score floor" + CI_GATE.md: "Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, score floor" EXTENDING.md: "Author a custom agent, criterion, or model pricing via the plugin SPI" REPORT_SCHEMA.md: "Field-level reference for run.json / variant.json / task.json" comparison.md: "vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts" diff --git a/tests/lint/action_docs.py b/tests/lint/action_docs.py new file mode 100644 index 00000000..017fa4d7 --- /dev/null +++ b/tests/lint/action_docs.py @@ -0,0 +1,232 @@ +"""CE026 — the GitHub Action's onboarding surfaces must stay truthful and self-sufficient. + +Three doc surfaces introduce the same composite Action (``README.md``, +``docs/CI_GATE.md``, ``docs/tutorials/02-ci-pipeline.md``) and each was hand-maintained, +so they drifted. The motivating bug: ``docs/CI_GATE.md`` claimed "there is nothing to +install" and offered a copy-pasteable ``uses:`` step with no agent runtime — the action +is agent-agnostic, so an integrator who copied it got a run that dies on a missing +``claude`` binary. The correcting paragraph was 11 lines away; the tutorial's snippet +showed the prerequisite steps; the reference page's did not. + +Three clauses, all mechanical: + +1. **Prerequisite parity.** The *first* fenced ``yaml`` block on a doc page that + references the action (``uses: /coder_eval@…``) is the page's quickstart, so + it must also show the agent-runtime steps. Later blocks on the same page are + single-input illustrations and are skipped, which is what keeps the rule quiet. +2. **No unqualified zero-install absolute** in prose near such a block. This catches + the *phrase*, not the *contradiction*: judging whether a paragraph 11 lines later + states a real prerequisite is semantic reasoning no static rule should attempt, so + the rule instead forces the absolute to be scoped where it is written + ("no *Marketplace* install step"). +3. **Marketplace slug parity.** Every ``github.com/marketplace/actions/`` link and + the shields badge label must match ``action.yml``'s ``name:`` — the listing title, + which a rename would silently 404 in four places at once. + +Like CE027-CE031 this is deliberately NOT a ``BaseRule`` in ``tests/lint/runner.py``: +that runner is AST-only over ``.py`` files, whereas this rule reasons over Markdown and +YAML. It is wired as ``tests/test_custom_lint.py::TestCE026ActionDocSurfaces``. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from pathlib import Path + +import yaml + +from tests.lint.doc_examples import extract_yaml_blocks + + +PREREQ_SKIP_MARKER = "` above the fence", + ) + ) + return findings + + +def find_unscoped_absolute_claims(paths: list[Path]) -> list[Finding]: + """Flag an unqualified "nothing to install" in prose next to an Action snippet.""" + findings: list[Finding] = [] + for path in paths: + text = path.read_text(encoding="utf-8") + lines = text.splitlines() + snippet_lines = [i for i, line in enumerate(lines) if _ACTION_USES.search(line)] + if not snippet_lines: + continue + for i, line in enumerate(lines): + lowered = line.lower() + claim = next((c for c in ABSOLUTE_CLAIMS if c in lowered), None) + if claim is None or CLAIM_SKIP_MARKER in line: + continue + if any(q in lowered for q in CLAIM_QUALIFIERS): + continue + if not any(abs(i - s) <= CLAIM_PROXIMITY_LINES for s in snippet_lines): + continue + findings.append( + Finding( + path, + i + 1, + f'unqualified absolute "{claim}" next to an Action snippet — the action installs ' + "no coding-agent runtime, so this reads as 'no prerequisites'. Scope it to the " + f"channel you mean (e.g. 'no Marketplace install step') or add " + f"`{CLAIM_SKIP_MARKER}: -->` on the line", + ) + ) + return findings + + +def action_listing_name(action_yml: Path) -> str: + """The Marketplace listing title declared by ``action.yml``'s ``name:``.""" + data = yaml.safe_load(action_yml.read_text(encoding="utf-8")) + name = data.get("name") + if not isinstance(name, str) or not name.strip(): + raise AssertionError(f"{action_yml} declares no usable `name:` — the Marketplace listing title") + return name.strip() + + +def marketplace_slug(listing_name: str) -> str: + """GitHub's Marketplace URL slug for a listing title.""" + slug = re.sub(r"\s+", "-", listing_name.strip().lower()) + return re.sub(r"[^a-z0-9._-]", "", slug) + + +def decode_shields_label(label: str) -> str: + """Decode a shields.io badge label back to its displayed text. + + shields renders ``_`` as a space and ``__`` / ``--`` as a literal ``_`` / ``-``, + which is why the badge for ``coder_eval`` must be written ``coder__eval``. + """ + placeholder_underscore, placeholder_hyphen = "\x00", "\x01" + decoded = label.replace("__", placeholder_underscore).replace("--", placeholder_hyphen) + decoded = decoded.replace("_", " ") + return decoded.replace(placeholder_underscore, "_").replace(placeholder_hyphen, "-") + + +def find_slug_mismatches(paths: list[Path], listing_name: str) -> list[Finding]: + """Flag Marketplace links/badges that disagree with ``action.yml``'s ``name:``.""" + expected_slug = marketplace_slug(listing_name) + findings: list[Finding] = [] + for path in paths: + for i, line in enumerate(path.read_text(encoding="utf-8").splitlines()): + for match in _MARKETPLACE_URL.finditer(line): + if match.group(1) != expected_slug: + findings.append( + Finding( + path, + i + 1, + f"Marketplace link slug {match.group(1)!r} does not match action.yml " + f"`name: {listing_name}` (expected {expected_slug!r}) — the link 404s", + ) + ) + for match in _SHIELDS_MARKETPLACE.finditer(line): + shown = decode_shields_label(match.group(1)) + if shown != listing_name: + findings.append( + Finding( + path, + i + 1, + f"Marketplace badge label {match.group(1)!r} displays as {shown!r}, not " + f"action.yml `name: {listing_name}` (shields renders a single `_` as a " + "space, so the doubled form is required)", + ) + ) + return findings + + +def dogfood_prereq_tokens(workflow: Path, job: str = DOGFOOD_JOB) -> set[str]: + """Tokens for every step the dogfood job runs before invoking the local action. + + This is the executable proof that ``REQUIRED_PREREQ_TOKENS`` is still the real + prerequisite set: CI runs these steps before ``uses: ./`` and the job passes. + """ + data = yaml.safe_load(workflow.read_text(encoding="utf-8")) + steps = data["jobs"][job]["steps"] + tokens: set[str] = set() + for step in steps: + uses = step.get("uses", "") + if uses.strip() in {"./", "."}: + break + if uses: + tokens.add(uses.split("@", 1)[0]) + run = step.get("run") + if isinstance(run, str): + tokens.update(line.strip() for line in run.splitlines() if line.strip()) + return tokens diff --git a/tests/test_custom_lint.py b/tests/test_custom_lint.py index a1f4da70..fe9ab6cc 100644 --- a/tests/test_custom_lint.py +++ b/tests/test_custom_lint.py @@ -1208,3 +1208,166 @@ def test_registered_fields_are_actually_consumed_on_the_real_tree(self): consumed = consumed_attr_names(self.SRC) for name in ("n_trials", "max_usd", "stratify_field"): assert name in consumed, f"expected {name!r} to be read as an attribute in src/" + + +@pytest.mark.lint +class TestCE026ActionDocSurfaces: + """CE026 — the Action's onboarding surfaces must be truthful and self-sufficient. + + The motivating bug: docs/CI_GATE.md said "there is nothing to install" above a + copy-pasteable `uses:` step with no agent runtime, while the correcting + prerequisite note sat 11 lines below and the tutorial's sibling snippet *did* + show the steps. An integrator who copied it got a run that dies on a missing + `claude` binary. Reasons over Markdown + YAML, so it lives here rather than in + the AST-only runner (precedent: CE027-CE031). + """ + + REPO_ROOT = Path(__file__).parent.parent + ACTION_YML = REPO_ROOT / "action.yml" + PR_CHECKS = REPO_ROOT / ".github" / "workflows" / "pr-checks.yml" + + def test_primary_action_snippets_show_agent_runtime_prereqs(self): + from tests.lint.action_docs import default_doc_paths, find_missing_prereqs + + findings = find_missing_prereqs(default_doc_paths(self.REPO_ROOT)) + assert not findings, ( + "\nAction quickstart snippet(s) that a reader can copy but that omit the agent-runtime " + "prerequisite steps:\n\n" + "\n".join(f" {f}" for f in findings) + ) + + def test_no_unqualified_zero_install_claims_near_action_snippets(self): + from tests.lint.action_docs import default_doc_paths, find_unscoped_absolute_claims + + findings = find_unscoped_absolute_claims(default_doc_paths(self.REPO_ROOT)) + assert not findings, ( + "\nUnqualified zero-install absolute(s) beside an Action snippet — scope the claim to the " + "channel it is about:\n\n" + "\n".join(f" {f}" for f in findings) + ) + + def test_marketplace_links_match_the_action_listing_name(self): + from tests.lint.action_docs import action_listing_name, default_doc_paths, find_slug_mismatches + + listing = action_listing_name(self.ACTION_YML) + findings = find_slug_mismatches(default_doc_paths(self.REPO_ROOT), listing) + assert not findings, ( + f"\nMarketplace link/badge(s) inconsistent with action.yml `name: {listing}` — a rename " + "404s every one of them:\n\n" + "\n".join(f" {f}" for f in findings) + ) + + def test_required_prereqs_match_the_dogfood_job(self): + # The constant is pinned to the executable reference: the dogfood job proves + # in CI that these steps are what a fresh runner needs before `uses: ./`. + from tests.lint.action_docs import DOGFOOD_JOB, REQUIRED_PREREQ_TOKENS, dogfood_prereq_tokens + + tokens = dogfood_prereq_tokens(self.PR_CHECKS) + for required in REQUIRED_PREREQ_TOKENS: + assert any(required in token for token in tokens), ( + f"{required!r} is documented as a prerequisite but the {DOGFOOD_JOB} job no longer " + f"runs it before `uses: ./` (job steps: {sorted(tokens)}). Update " + "REQUIRED_PREREQ_TOKENS and the doc snippets together." + ) + + def test_catches_a_snippet_missing_prereqs(self, tmp_path: Path): + from tests.lint.action_docs import find_missing_prereqs + + page = tmp_path / "page.md" + page.write_text( + "# Gate\n\n```yaml\n- uses: UiPath/coder_eval@v0\n with:\n tasks: t.yaml\n```\n", + encoding="utf-8", + ) + findings = find_missing_prereqs([page]) + assert len(findings) == 1 + assert "actions/setup-node" in findings[0].message + + def test_only_the_first_action_block_is_checked(self, tmp_path: Path): + # Later single-input illustrations must stay quiet, or the rule becomes a nuisance. + from tests.lint.action_docs import find_missing_prereqs + + page = tmp_path / "page.md" + page.write_text( + "```yaml\n" + "- uses: actions/setup-node@v4\n" + "- run: npm install -g @anthropic-ai/claude-code\n" + "- uses: UiPath/coder_eval@v0\n" + "```\n\n" + "Score floor:\n\n" + '```yaml\n- uses: UiPath/coder_eval@v0\n with:\n minimum-task-score: "0.8"\n```\n', + encoding="utf-8", + ) + assert find_missing_prereqs([page]) == [] + + def test_prereq_skip_marker_opts_out(self, tmp_path: Path): + from tests.lint.action_docs import find_missing_prereqs + + page = tmp_path / "page.md" + page.write_text( + "\n```yaml\n- uses: UiPath/coder_eval@v0\n```\n", + encoding="utf-8", + ) + assert find_missing_prereqs([page]) == [] + + def test_page_without_the_action_is_ignored(self, tmp_path: Path): + from tests.lint.action_docs import find_missing_prereqs + + page = tmp_path / "page.md" + page.write_text("```yaml\n- uses: actions/checkout@v6\n```\n", encoding="utf-8") + assert find_missing_prereqs([page]) == [] + + def test_catches_the_nothing_to_install_regression(self, tmp_path: Path): + # The exact sentence this rule exists for. + from tests.lint.action_docs import find_unscoped_absolute_claims + + page = tmp_path / "page.md" + page.write_text( + "you reference it by repo path — there is nothing to install:\n\n" + "```yaml\n- uses: UiPath/coder_eval@v0\n```\n", + encoding="utf-8", + ) + findings = find_unscoped_absolute_claims([page]) + assert len(findings) == 1 + assert "nothing to install" in findings[0].message + + def test_scoped_claim_is_allowed(self, tmp_path: Path): + from tests.lint.action_docs import find_unscoped_absolute_claims + + page = tmp_path / "page.md" + page.write_text( + "you reference it by repo path — there is no Marketplace install step:\n\n" + "```yaml\n- uses: UiPath/coder_eval@v0\n```\n", + encoding="utf-8", + ) + assert find_unscoped_absolute_claims([page]) == [] + + def test_claim_far_from_a_snippet_is_ignored(self, tmp_path: Path): + from tests.lint.action_docs import find_unscoped_absolute_claims + + page = tmp_path / "page.md" + page.write_text( + "there is nothing to install\n" + "\n" * 40 + "```yaml\n- uses: UiPath/coder_eval@v0\n```\n", + encoding="utf-8", + ) + assert find_unscoped_absolute_claims([page]) == [] + + def test_catches_a_renamed_listing(self, tmp_path: Path): + from tests.lint.action_docs import find_slug_mismatches + + page = tmp_path / "page.md" + page.write_text("[coder_eval](https://github.com/marketplace/actions/coder_eval)\n", encoding="utf-8") + assert find_slug_mismatches([page], "coder_eval") == [] + assert len(find_slug_mismatches([page], "coder eval x")) == 1 + + def test_shields_label_must_decode_to_the_listing_name(self, tmp_path: Path): + from tests.lint.action_docs import decode_shields_label, find_slug_mismatches + + # A single `_` renders as a space, which is why the badge needs the doubled form. + assert decode_shields_label("coder__eval") == "coder_eval" + assert decode_shields_label("coder_eval") == "coder eval" + + page = tmp_path / "page.md" + page.write_text( + "[![m](https://img.shields.io/badge/marketplace-coder_eval-2ea44f.svg)](https://x)\n", + encoding="utf-8", + ) + findings = find_slug_mismatches([page], "coder_eval") + assert len(findings) == 1 + assert "displays as 'coder eval'" in findings[0].message