Skip to content

Name the platform Standard Environments in the README #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
davidbzyk merged 2 commits into from
Nov 4, 2026
Merged

Name the platform Standard Environments in the README #26

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5c49408
Name the platform Standard Environments in the README
davidbzyk Nov 4, 2026
b792fb7
Rename the two line-wrapped mentions too
davidbzyk Nov 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Code Review Workflow

This is both a research experiment into the most effective way to build a custom code review, and
an example workflow for my business, [General Environments](https://general-environments.ai/).
General Environments is a community-based benchmarking and agentic evaluation platform. It has
an example workflow for my business, [Standard Environments](https://standardenvs.com/).
Standard Environments is a community-based benchmarking and agentic evaluation platform. It has
253 open-source benchmarks: you can fork one to create your own, import your own to evaluate it
quickly across harnesses and models, or work with an expert to build the right agentic workflow
and evaluation set for your business.
Expand All @@ -11,7 +11,7 @@ and evaluation set for your business.
`gpt-6-luna`), given one of three kinds of context: a graphed OKF knowledge base, a vector search
index, or nothing but the code itself (plain). In short, letting the model search the codebase on
its own did as well as either context store, with nothing to build or serve. This repository
holds the public voiceover cases and experiments; I also include the results from General
holds the public voiceover cases and experiments; I also include the results from Standard
Environments' own codebase, as it is a great example of rapid agentic engineering.

**The first cross-model result.** On the 12 public eval tasks, gpt-6.1-sol (high) found about 1.7
Expand All @@ -26,12 +26,12 @@ the side-by-side on these pull requests is CodeRabbit, Copilot and the review ac
**The design.** I went through all my code and security review plugins and kept what I thought
was valuable from each; every reviewer cites its sources in its own folder (see the last section).

The full write-up is on the General Environments blog:
[Incorporating Evaluations and Refinement Loops for Non-Regressive Agentic Workflows](https://general-environments.ai/blog/evaluation-refinement-loops/).
It compares and evaluates: this repository's evals are imported into General Environments and run
The full write-up is on the Standard Environments blog:
[Incorporating Evaluations and Refinement Loops for Non-Regressive Agentic Workflows](https://standardenvs.com/blog/evaluation-refinement-loops/).
It compares and evaluates: this repository's evals are imported into Standard Environments and run
across models and harnesses, for review quality and for cost. Running every task on every model
proved expensive, so part of the work is finding the most economically viable set of tasks and
models. The post also covers a larger, more complex codebase (General Environments itself). This
models. The post also covers a larger, more complex codebase (Standard Environments itself). This
Comment thread
davidbzyk marked this conversation as resolved.
Resolved
README is its summary.

## What is in this repository
Expand All @@ -52,7 +52,7 @@ with reading the code alone (**plain**)?
**Setup.**
- Seven reviewers per review, each its own agent, plus a validator for serious findings.
Model gpt-6-luna, reasoning effort high, Codex CLI.
- Ground truth: 28 real merged pull requests from one private Python monorepo (General
- Ground truth: 28 real merged pull requests from one private Python monorepo (Standard
Environments' own). The "golden" issues are the comments six paid review bots left that the
owner accepted: 111 in all.
- Judge: an LLM judge following the Martian Code Review Bench method (does a finding state "the
Expand Down Expand Up @@ -134,7 +134,7 @@ on the pull request. A finding someone resolved, or answered with a reply (fixed
raised again unless the lines it names change; one left open keeps counting in the summary. Replies
reach the reviewers only from people who can push to the repository, checked with the workflow's own
`GITHUB_TOKEN`, so there is no extra token or GitHub App to set up; a reply from a read or triage
collaborator is ignored. On a nine-commit pull request in General Environments, six findings it
collaborator is ignored. On a nine-commit pull request in Standard Environments, six findings it
raised were answered once and never came back across the five reviews that followed.

To update, run the installer again; never edit `.code-review/` by hand. Model, effort, title and
Expand Down Expand Up @@ -217,7 +217,7 @@ Per trial: `research/results/llm-judge-2026-10-13.csv`.

### Run the evals yourself

In General Environments, import this repository: every task folder under `evals/` becomes part of
In Standard Environments, import this repository: every task folder under `evals/` becomes part of
one benchmark, and the run picks the agent, model and judge. With Harbor directly:

export REWARDKIT_JUDGE=codex REWARDKIT_MODEL=gpt-6-luna \
Expand Down
Loading