Add Anchored Review Format standard for AI review systems

standard/README.md

@@ -0,0 +1,114 @@
+# Anchored Review Format
+
+This is a small, open standard for the output of an AI paper-review system. If a system adopts this standard, the benchmark can run it directly, with no custom integration code needed.
+
+A review in this standard is a list of comments, where every comment is anchored to a **verbatim quote** from the paper and paired with an **explanation** of the issue. A system connects in one of two ways:
+
+- **Command line** (open systems the benchmark can run): the system provides a command that takes a paper and writes a review. See `[profile-cli.md](profile-cli.md)`.
+- **Hosted API** (closed systems the benchmark calls over the network): the system exposes an endpoint the benchmark submits papers to and polls for results. See `[profile-api.md](profile-api.md)`.
+
+Both return the same review payload, described below.
+
+## Payload format
+
+A review payload looks like this:
+
+```json
+{
+  "standard_version": "1.0",
+  "comments": [
+    {
+      "quote": "we achieve a 51% improvement over the baseline",
+      "explanation": "The 51% figure is not supported by Table 3, which reports a 5.1% relative gain. This looks like a misplaced decimal."
+    }
+  ]
+}
+```
+
+Two top-level fields are required: `standard_version` and `comments`. Each comment requires `quote` and `explanation`. This is everything the standard requires.
+
+Validate a payload with the bundled checker, which needs only Python:
+
+```bash
+python validate.py your_review.json
+```
+
+A passing payload prints `✓ VALID` and exits 0.
+
+## Why these two fields
+
+The benchmark seeds known errors into clean papers and measures how many a system catches. To decide whether a comment caught a seeded error, the benchmark does two things:
+
+1. Match the comment's `quote` against the region of the paper that was changed. This match is fuzzy, but it works best when the quote is **exactly from the paper text**.
+2. Read the comment's `explanation` and judge whether it identifies the same error that was seeded.
+
+So `quote` should be extracted verbatim from the paper, and `explanation` should say what is wrong and why. Those are the only fields the score depends on.
+
+## Full specification
+
+One payload is one system's review of one paper.
+
+### Top-level fields
+
+
+| Field              | Required    | Type   | Meaning                                                                                                                                  |
+| ------------------ | ----------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
+| `standard_version` | yes         | string | Version of this standard, currently `"1.0"`.                                                                                             |
+| `comments`         | yes         | array  | The list of comments. See below.                                                                                                         |
+| `standard`         | recommended | string | The constant `"anchored-review"`, so the payload is self-identifying.                                                                    |
+| `paper_id`         | recommended | string | An id for the paper, such as an arXiv id or a slug.                                                                                      |
+| `system`           | recommended | string | The name of the review system.                                                                                                           |
+| `model`            | optional    | string | The underlying model id, if there is one.                                                                                                |
+| `overall_feedback` | optional    | string | A high-level assessment of the paper.                                                                                                    |
+| `paragraphs`       | optional    | array  | The paper split into paragraphs. Include this only if the comments use `paragraph_index`. Each item is `{"index": int, "text": string}`. |
+
+
+### Comment fields
+
+
+| Field             | Required    | Type            | Meaning                                                       |
+| ----------------- | ----------- | --------------- | ------------------------------------------------------------- |
+| `quote`           | yes         | string          | A span copied from the paper that the comment is about.       |
+| `explanation`     | yes         | string          | What is wrong with the quoted span and why.                   |
+| `title`           | recommended | string          | A short label for the comment.                                |
+| `severity`        | optional    | string          | A free-text tier such as `major`, `moderate`, or `minor`.     |
+| `paragraph_index` | optional    | integer or null | A 0-based index into `paragraphs`, if that array is provided. |
+| `id`              | optional    | string          | A stable id for the comment.                                  |
+
+
+Extra fields are preserved and ignored by the benchmark, so a system can carry its own metadata without breaking anything.
+
+## Connecting a system
+
+Each profile matches a different way a system runs. Both deliver the same payload above.
+
+- **Open source, or otherwise runnable by the benchmark:** the [command-line profile](profile-cli.md). The benchmark runs the system locally, so there is no hosting or inference cost for its authors, and this is the simplest to adopt.
+- **Closed source:** the [hosted API profile](profile-api.md). The system runs behind an endpoint the benchmark calls. This is the option when the benchmark cannot run the system's code, such as for proprietary systems.
+
+## Examples
+
+- `[examples/minimal.json](examples/minimal.json)`: the smallest valid payload, with only the required fields.
+- `[examples/full.json](examples/full.json)`: every field populated, including paragraphs and per-comment metadata.
+- `[examples/invalid-missing-quote.json](examples/invalid-missing-quote.json)`: a payload that fails, showing what the validator reports.
+
+## Validating
+
+```bash
+python validate.py examples/full.json
+```
+
+The checker reports two kinds of findings. **Errors** mean the payload does not conform and will not score. **Warnings** flag recommended fields that were left out, which are fine to ignore. To enforce the recommended fields too, treat warnings as failures:
+
+```bash
+python validate.py your_review.json --strict
+```
+
+The checker uses only the Python standard library, so any Python 3 install can run it as is.
+
+## Versioning
+
+The `standard_version` field tracks the standard. Version `1.0` is the current release. Additions that keep older payloads valid will bump the minor version (`1.1`, `1.2`). Changes that can break older payloads will bump the major version (`2.0`). A system keeps emitting the version it built against, and its payloads stay readable.
+
+## Questions
+
+This standard grew out of the OpenAIReview project. If a field does not map cleanly onto what a system produces, or a field should be added, open an issue. The standard is meant to stay small, so the bar for new required fields is high, but new optional fields are welcome.

standard/examples/full.json

@@ -0,0 +1,30 @@
+{
+  "standard": "anchored-review",
+  "standard_version": "1.0",
+  "paper_id": "2504.06303",
+  "system": "example-reviewer",
+  "model": "example-model-v1",
+  "overall_feedback": "The paper presents a clear method and strong empirical results. The main concerns are an unsupported headline number in the abstract and an inconsistent definition of the loss function between Section 2 and Appendix A.",
+  "paragraphs": [
+    { "index": 0, "text": "Abstract. We present a method that achieves a 51% improvement over the baseline ..." },
+    { "index": 1, "text": "Section 2. We define the training objective as L = -sum p log p ..." }
+  ],
+  "comments": [
+    {
+      "id": "c0",
+      "title": "Unsupported headline improvement",
+      "quote": "we achieve a 51% improvement over the baseline",
+      "explanation": "The 51% figure is not supported by Table 3, which reports a 5.1% relative gain. This looks like a misplaced decimal or a transcription error.",
+      "severity": "major",
+      "paragraph_index": 0
+    },
+    {
+      "id": "c1",
+      "title": "Loss definition conflates two distributions",
+      "quote": "the loss is defined as L = -sum p log p",
+      "explanation": "The cross-entropy loss should sum over the true distribution times the log of the predicted distribution. As written the two distributions are conflated, which makes this the entropy of p rather than a training loss.",
+      "severity": "moderate",
+      "paragraph_index": 1
+    }
+  ]
+}

standard/examples/invalid-missing-quote.json

@@ -0,0 +1,9 @@
+{
+  "standard_version": "1.0",
+  "comments": [
+    {
+      "title": "This comment is missing its quote",
+      "explanation": "A comment must anchor to a verbatim span from the paper, but this one has no quote field, so it cannot be matched to anything and the validator rejects it."
+    }
+  ]
+}

standard/examples/minimal.json

@@ -0,0 +1,13 @@
+{
+  "standard_version": "1.0",
+  "comments": [
+    {
+      "quote": "we achieve a 51% improvement over the baseline",
+      "explanation": "The 51% figure is not supported by Table 3, which reports a 5.1% relative gain. This looks like a misplaced decimal or a transcription error."
+    },
+    {
+      "quote": "the loss is defined as L = -sum p log p",
+      "explanation": "The cross-entropy loss should sum over the true distribution times the log of the predicted distribution. As written the two distributions are conflated, which makes this the entropy of p rather than a training loss."
+    }
+  ]
+}

standard/profile-api.md

@@ -0,0 +1,93 @@
+# Hosted API profile
+
+This profile is for closed systems the benchmark cannot run itself. The system exposes an HTTP endpoint. The benchmark submits each paper, polls until the review is ready, and reads the result.
+
+A review takes minutes, so this is a job-style API. Submit returns immediately with an id, and the result is fetched by polling.
+
+## Endpoints
+
+### Submit a review
+
+```
+POST /v1/reviews
+Authorization: Bearer <api-key>
+Content-Type: application/json
+
+{
+  "paper": { "text": "<full paper text>" },
+  "options": { }
+}
+```
+
+Response:
+
+```json
+{ "session_id": "abc123", "status": "queued" }
+```
+
+- `paper` carries the paper. Supporting `{"text": "..."}` is the minimum. A system may also accept `{"url": "..."}` or a multipart file upload, and the benchmark sends whichever the system declares.
+- `options` is an open object for any parameters the system exposes (review mode, depth). It can be omitted when there are none.
+- `status` is the job state (see below).
+
+### Fetch a review
+
+```
+GET /v1/reviews/{session_id}
+Authorization: Bearer <api-key>
+```
+
+Response while running:
+
+```json
+{ "session_id": "abc123", "status": "running" }
+```
+
+Response when done:
+
+```json
+{
+  "session_id": "abc123",
+  "status": "completed",
+  "result": {
+    "standard_version": "1.0",
+    "comments": [
+      { "quote": "...", "explanation": "..." }
+    ]
+  }
+}
+```
+
+The `result` object is a payload in the [Anchored Review Format](README.md). Returning the standard field names (`quote`, `explanation`) is what lets the benchmark call the API with no per-system adapter. 
+
+## Status values
+
+
+| Status      | Meaning                                   |
+| ----------- | ----------------------------------------- |
+| `queued`    | Accepted, not started.                    |
+| `running`   | In progress. Keep polling.                |
+| `completed` | Done. `result` holds the payload.         |
+| `failed`    | Terminal error. Include an `error` field. |
+
+
+Any other terminal-sounding value is treated as done or failed on a best-effort basis, but the four above keep the interaction unambiguous.
+
+## Authentication
+
+The example uses a bearer token. When an API uses a different header (for example `x-api-key`), the system declares the header name and the benchmark sets it. One scheme per system is sufficient.
+
+## Reference client
+
+`[reference/review_client.py](reference/review_client.py)` is a standalone client for this profile. It submits a paper, polls until the job is terminal, and returns the validated payload. It uses only the Python standard library, so the system's creators can run it against a staging endpoint to check that their output is compatible with the benchmark:
+
+```bash
+python reference/review_client.py --base-url https://your-api.example.com --api-key <key> paper.txt
+```
+
+It prints the returned payload and the validator's verdict.
+
+## Notes
+
+- Reviews are long-running. The client polls with a default timeout that can be raised for slow systems.
+- The benchmark may submit several papers at once. Each `session_id` is independent.
+

standard/profile-cli.md

@@ -0,0 +1,48 @@
+# Command-line profile
+
+This profile is for systems the benchmark can run itself, such as an open-source repository or a locally installable package. The system provides a command that reviews one paper. The benchmark runs it on each paper and reads the review it writes.
+
+This is the simplest way to connect a system. There is no endpoint to host and no inference cost for the system's authors, since the benchmark supplies the compute.
+
+## The contract
+
+The system provides a single command that:
+
+1. Takes the path to a paper file as input.
+2. Writes one review payload, conforming to the [Anchored Review Format](README.md), to a specified path.
+3. Exits `0` on success and non-zero on failure.
+
+The benchmark invokes the command once per paper:
+
+```bash
+your-review-command <paper-path> --out <output-json-path>
+```
+
+- `<paper-path>` is a file the benchmark provides. The system declares which formats it accepts (for example PDF, Markdown, or LaTeX source), and the benchmark sends that format.
+- `--out <output-json-path>` is where the command writes the payload.
+
+Any configuration the system needs (model choice, API keys for its own backend, review depth) can come from command-line flags or environment variables.
+
+## Format validation
+
+The system's creators can run this check to see if their output format is compatible with the benchmark.
+
+```bash
+python validate.py <output-json-path>
+```
+
+## Worked example
+
+The reference system in this repository, `openaireview`, satisfies this profile with:
+
+```bash
+openaireview review paper.pdf --out review.json
+```
+
+The benchmark runs that command per paper and scores the `review.json` it produces. Another system's command takes its place, with whatever name and flags fit the tool.
+
+## Notes
+
+- The benchmark may run several papers in parallel, so each invocation should be independent and write only to its own `--out` path.
+- A nondeterministic run is fine. The benchmark reports the run it observes. If the command exposes a seed, please document it to help us with reproducibility.
+