Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .githooks/_resolve
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
#
# Then, git hook does not require `uv` or `pip` or any specific environment layout.

recorded="${GIT_DIR:-$(git rev-parse --absolute-git-dir)}/harness-path"
recorded="$(git rev-parse --path-format=absolute --git-common-dir)/harness-path"
if [ ! -r "$recorded" ]; then
echo "loopgate: hooks are not installed. Run 'harness install' in this repo." >&2
exit 1
Expand All @@ -21,3 +21,5 @@ if [ ! -x "$HARNESS" ]; then
echo "loopgate: recorded harness '$HARNESS' is gone. Re-run 'harness install'." >&2
exit 1
fi

export PATH="$(dirname "$HARNESS")${PATH:+:$PATH}"
50 changes: 50 additions & 0 deletions .github/workflows/mutation.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: Mutation Testing

on:
schedule:
- cron: "0 0 * * 0" # Weekly at midnight Sunday
workflow_dispatch: # adds "Run workflow" button

env:
FORCE_COLOR: "1"
CLICOLOR_FORCE: "1"
TERM: "xterm-256color"

permissions:
contents: read

concurrency:
group: mutation-${{ github.ref }}
cancel-in-progress: true

jobs:
mutate:
runs-on: ubuntu-latest
timeout-minutes: 60

steps:
- name: Checkout Code
uses: actions/checkout@v6

- name: Install uv
uses: astral-sh/setup-uv@v6
with:
enable-cache: true

- name: uv Sync Dependencies
run: uv sync

- name: Harness Checks
run: uv run --no-sync harness gate # need tests for Mutmut to target

- name: Run Mutmut
run: mutmut run || true

- name: Export CI/CD Stats
run: mutmut export-cicd-stats

- name: Upload Mutmut JSON Report
uses: actions/upload-artifact@v4
with:
name: mutmut-json-report
path: mutants/mutmut-cicd-stats.json
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,8 @@ scratchpad/*
scratchpad/runs/*
!scratchpad/runs/.gitkeep

# mutmut's mutated copies of the source. Ignoring them is also what keeps ruff and semgrep out.
mutants/

# kept at the last line for easy deletion (real projects need lockfile)
uv.lock
142 changes: 77 additions & 65 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@
<img src=".banner.svg" alt="Blue infinity loop" width="360">

<h1>L∞pGate</h1>
<p>A coding-agent loop harness for Claude, Codex, Copilot, or any CLI agent. A dumb Ralph loop runner tells an agent to "Go!" and hands it a PROMPT. Agents can edit. Gates decide what lands. You set the plan in motion. The loops eat the prompt, and each agent iteration must update specs and commit through guardrails.</p>
<h4>Run coding agents strictly andonly accept changes that pass your quality gates.</h4>
<p>A coding-agent loop harness for Claude, Codex, Copilot, or any CLI agent. A dumb Ralph loop runner tells an agent to "Go!" and hands it a PROMPT. Agents can edit. Gates decide what lands. You set the plan in motion. The loops eat the prompt, and each agent iteration must update specs and commit through guardrails.</p>

![Python](https://img.shields.io/badge/Python-3.11+-3776AB?logo=python&logoColor=white)
![Status](https://img.shields.io/badge/github-repo-blue?logo=github)
Expand All @@ -21,13 +22,12 @@

---

## TL;DR: Getting Started.
## TL;DR

1. `gh repo create my-app-name --template <your-gh-username>/loopgate_harness --private --clone` **or** ['Use This Template'](https://github.com/new?template_name=loopgate_harness&template_owner=rxdt)
2. Install dependancies e.g. `uv sync && source .venv/bin/activate && harness install <your-project-name`.
3. Write your project goal in [docs/plan.md](docs/plan.md)
4. `harness run <agent=claude|codex|agy|copilot> [max_iterations] [max_minutes]`
5. Not what you wanted? Refine [`docs/plan.md`](docs/plan.md) / [`docs/PROMPT.md`](docs/PROMPT.md) and re-run
1. `gh repo create <your-app-name>/<your-app-name> --template rxdt/loopgate_harness --private --clone && cd <your-app-name> && uv run harness install <your-app-name> && source .venv/bin/activate`
2. `harness run codex`

**Requirements**: `pip`, `uv`, or `poetry`. Python 3.11.

---

Expand All @@ -37,10 +37,10 @@
- **Worker-agnostic**: Claude, Codex, Copilot, Agy, or any prompt-reading CLI
- **No lazy**: Agents work, _only if they pass the quality gates you set_ ✅
- **Repo-as-memory workflow**: specs/status/prompt are durable but code is king, leaving you free 😎
- **Built-in stack**: Ruff, Pyright, Pylint, Semgrep, Complexipy, Hypothesis, 100% coverage ☑☑☑
- **Built-in stack**: Ruff, Pyright, Pylint, Semgrep, Complexipy, Hypothesis, Mutmut, 100% coverage ☑☑☑
- **Progressive**: Preflight vs full gate split 🆗
- **Forbidden-path containment**: Don't touch that!-configurable 🛑
- **Installable project template**: `harness install loopgate` gets the repo ready ▶️
- **Installable project template**: `harness install <your-app-name>` gets the repo ready ▶️
- **No-rot**: Fresh-context agent iterations to reduce context rot 🔄
- **Simple**: One command setup gets you git hooks and everything else
- **No-waste**: Timeouts and time-limits for all loops ⏸
Expand All @@ -56,20 +56,27 @@
`docs/PROMPT.md` tells each agent to pick a `spec` and build. `docs/specs/` say _what_ to build. The agent decides _what next_. You keep `docs/plan.md` current, and specs get rewritten from it (agent is told in `docs/PROMPT.md` to update the specs). Each iteration the agent updates its spec and `PROJECT_STATUS`. Ideas from [ghuntley](https://github.com/ghuntley), How to Ralph Wiggum.

> [!TIP]
> If you don't like _ANYTHING_ in this framework, remove it.

## Start a project

1. From the root, run `harness install <your-project-name>` to name the project, install dependencies, set up the three git hooks, and delete excess files.
2. Write your grand vision into `docs/plan.md`.
3. Optionally add the first spec in `docs/specs/`, or have an agent draft the first specs.
4. Put product code under `src/` and list new source directories in `pyproject.toml [tool.coverage.run]`.
5. Strict Ruff rules, type checking, pyright, complexipy, and pytest coverage are set in `pyproject.toml`.
6. Your coding quirks go in [`preferences/preferences.py`](preferences/preferences.py).
7. Run a loop:
> If you don't like _ANYTHING_ in this framework, update it.

### Start a project

1. `gh repo create my-app-name --template <your-gh-username>/loopgate_harness --private --clone` **or**
['Use This Template'](https://github.com/new?template_name=loopgate_harness&template_owner=rxdt)
2. Source your environment (if applicable)
3. From the root, run `harness install <your-project-name>` to name the project, install dependencies, set up the git hooks, and delete excess files. Install dependencies e.g. `uv sync && source .venv/bin/activate && harness install <your-project-name`.
4. `git commit` (the `install` command updates the repo)
5. Write your grand vision in [docs/plan.md](docs/plan.md)
6. Optionally add the first spec in `docs/specs/` (or leave it to the agents to draft the first specs based on your `plan.md`)
7. Product code goes in [`src/`](src/). _(List new source directories in [`pyproject.toml [tool.coverage.run]`](pyproject.toml#toolcoveragerun).)_
8. Run some loops!
`harness run <agent=claude|codex|agy|copilot> [max_iterations] [max_minutes]`
9. Not what you wanted? Refine [`docs/plan.md`](docs/plan.md) / [`docs/PROMPT.md`](docs/PROMPT.md) and re-run
10. Strict Ruff rules, type-checking Pyright, Complexipy, and Pytest coverage are set in [`pyproject.toml`](pyproject.toml).
11. Your coding quirks go in [`preferences/preferences.py`](preferences/preferences.py).
12. Loop!:

```sh
harness run <agent> [max_iterations] [max_minutes] # agent: claude/codex/agy/copilot. ralph loop runner adds prompt
harness run <agent> [max_iterations] [max_minutes] # agent: claude/codex/agy/copilot. ralph loop runner injects prompt
```

### Works with `uv`, `poetry`, or `pip`
Expand All @@ -79,12 +86,12 @@ uv sync
source .venv/bin/activate
harness install <your-project-name>
harness gate
harness run <agent>>
harness run <agent>

poetry install
poetry run harness install <your-project-name>
poetry run harness gate
poetry run harness <agent>
poetry run harness run <agent>

python -m venv .venv
source .venv/bin/activate
Expand All @@ -96,7 +103,7 @@ harness run <agent>

![L∞P architecture engine flow](.loops.svg)

## A L∞PS Loop
## A L∞Pgate Loop

The repo is the only memory. Each iteration is a fresh-context agent.

Expand All @@ -112,25 +119,6 @@ The repo is the only memory. Each iteration is a fresh-context agent.

![L∞PS Agents](.loops_agents.svg)

## Safety

`harness run` launches an autonomous LLM worker with the configured permissions, e.g.
`--permission-mode acceptEdits` or `--sandbox danger-full-access`.

The gate bounds what any **commit** may touch, but the worker itself is **not** sandboxed to this repo unless you set that config. Consider the balance: without access it cannot do much. With machine access it can wreak havoc. Under a permissive mode it can run arbitrary shell. You are authorizing real changes. Choose the worker and permission mode deliberately.

#### The Gate: Tiered Checks

⚡ `harness preflight` (pre-commit) → fast checks.
Ruff lint + check format for everyone, _plus_ **containment** for the agents. Self-heals by un-staging forbidden files.

✅ `harness gate` (CI/PR pre-push). Local checks mirror CI → ruff lint + format report-only, pyright, pylint, semgrep, complexipy, hypothesis, pytest @ 100% cov.

Only humans can bypass triggered gates and commit by adding flag `--no-verify`.

<details>
<summary>

## Directory Layout

</summary>
Expand Down Expand Up @@ -173,22 +161,6 @@ patterns = ["# noqa"] # banned in agent-authored diffs
pytest = "uv sync pytest" # one check command, run by the local gate AND CI
```

## Read this before a first run.

1. **This harness does not sandbox agents.** It _tries_ to harness bad code in loops via gates. Sandboxing agents will, e.g. prevent them from maintaining git, running Playwright, being seen as trustworthy by semgrep leading to cyclical failures, etc.

2. **The gate is a guardrail, not a jail.** Agents are crafty, like people. They will find a way to complete a task at all costs. **Trust nothing and no one.**

3. **Mind your usage limits.** `harness run` works agents to the cap set. You can easily burn through your tokens, context windows, and provider usage limits. **Workers continue running as long as there is work to do.**

4. **`docs/PROMPT.md` tells the worker to push or not**.

5. Protect `main` and run the loop on its own branch.

6. **100% coverage does not mean good tests.** That is quantity, not quality. (Upcoming feature: mutation testing)

7. **Note**: `semgrep --config auto` needs network for semgrep registry rules.

## Commands

Tool commands are defined once, in `[tool.harness.gate.checks]` in [pyproject.toml](pyproject.toml). The local gate and CI both derive them from there.
Expand All @@ -215,19 +187,20 @@ To run LoopGate with Claude :
harness run claude 2 20
```

Note: The worker must be installed and authenticated separately.
Note: The worker must be installed and authenticated separately. **Claude Code exports env vars into every shell it spawns. `RALPH_LOOP=1` can be set globally.**

<details>
<summary>

## Expanding your harness </summary>

- Edit rules at [pyproject.toml](pyproject.toml) for [ruff](https://docs.astral.sh/ruff/), [pylint](https://pypi.org/project/pylint/), [pydoclint](https://pypi.org/project/pydoclint/0.9.1/), [pyright](https://github.com/microsoft/pyright), [pytest](https://docs.pytest.org/en/stable/), [hypothesis](https://hypothesis.readthedocs.io/), [complexipy](https://github.com/rohaquinlop/complexipy)
- Edit rules at [pyproject.toml](pyproject.toml) for [ruff](https://docs.astral.sh/ruff/), [pylint](https://pypi.org/project/pylint/), [pydoclint](https://pypi.org/project/pydoclint/0.9.1/), [pyright](https://github.com/microsoft/pyright), [pytest](https://docs.pytest.org/en/stable/), [hypothesis](https://hypothesis.readthedocs.io/), [complexipy](https://github.com/rohaquinlop/complexipy), [mutmut](https://mutmut.readthedocs.io/)
- Add forbidden files, directories, or patterns in `[tool.harness.gate]` at [pyproject.toml](pyproject.toml)
- Add [Hypothesis](https://hypothesis.readthedocs.io/) tests in any test directory, examples at [test_properties.py](tests/preferences/test_properties.py).
- [semgrep](https://docs.semgrep.dev/semgrep-ci/sample-ci-configs) has no repo config here. It uses registry configs plus Semgrep's built-in defaults which ignore tests.
- Edit `[tool.harness.gate.checks]` in [pyproject.toml](pyproject.toml). [ci.yml](.github/workflows/ci.yml) runs the same `harness gate`.
- Remove existing preferences or add your own at [preferences.py](preferences/preferences.py). Current preferences:
- Run [mutmut](https://mutmut.readthedocs.io/) by hand with `uv run mutmut run`, then `uv run mutmut browse`. A surviving mutant is a covered line no assertion checks. It is not a gate check: `mutmut run` exits 0 even with survivors. Its verdicts are cached against source hashes and ignore test edits, so re-run a single mutant by name once you have written a test for it. Examples at [test_with_mutations.py](tests/preferences/test_with_mutations.py).
- [semgrep](https://docs.semgrep.dev/semgrep-ci/sample-ci-configs) has no repo config here. It uses registry configs / Semgrep's built-in defaults which ignore tests.
- Update `[tool.harness.gate.checks]` in [pyproject.toml](pyproject.toml). [ci.yml](.github/workflows/ci.yml) runs those **same exact** `harness gate` checks.
- Add or remove coding preferences [preferences.py](preferences/preferences.py) that only agents in loops **must** respect. Current preferences:

```py
function_argument_assignment_has_star # agents use non-specific `def fun(*)`
Expand All @@ -237,7 +210,7 @@ dynamic_star_call # Calls to def fun(*items) breaks when you can't tell how man
pointless_class # ensure classes are added for good reasons (carry state, values, methods)
lazy_assert # enforce real assertions, stronger tests
objects_injected_into_runtime_memory # finds calls that manipulate global state (dangerous, tricky)
lambda_found # abolish lambdas for agents to keep their code simpler
lambda_found # abolish lambdas, make agents keep their code simple
lazy_any_type_hints # abolish type `Any` used to bypass strict type-checking
chaotic_continue_statements # abolish unecessary nested continue statements, clean code
complex_comprehension # no needlessly dense list/set/dict comprehensions, prefer linear code
Expand Down Expand Up @@ -266,6 +239,10 @@ You don’t have to. The loop runner, Ralph, and the CLI take a prompt, launch a

The included [`harness/js-scaffold`](harness/js-scaffold/package.json) is a simple JavaScript **example** to expand on. Go to [pyproject.toml line 75](pyproject.toml#L75). Update checks. Put `js` into list `[tool.harness].languages`. Remove `py` if unused.

- **Why not just a shell loop?**

A shell loop only reruns an agent. LoopGate ensures fresh context, durable repo state, time and iteration limits, protected paths, and quality gates that stop bad changes _before_ they land.

```
npm run --prefix harness/js-scaffold gate
npm run --prefix harness/js-scaffold preflight
Expand Down Expand Up @@ -322,3 +299,38 @@ npm run --prefix harness/js-scaffold preflight
</details>

![diagram](.diagram.png)

## Read this before a first run.

1. **This harness does not sandbox agents.** It _tries_ to harness bad code in loops via gates. Sandboxing agents will, e.g. prevent them from maintaining git, running Playwright, being seen as trustworthy by semgrep leading to cyclical failures, etc.

2. **The gate is a guardrail, not a jail.** Agents are crafty, like people. They will find a way to complete a task at all costs. **Trust nothing and no one.**

3. **Mind your usage limits.** `harness run` works agents to the cap set. You can easily burn through your tokens, context windows, and provider usage limits. **Workers continue running as long as there is work to do.**

4. **`docs/PROMPT.md` tells the worker to push or not**.

5. Protect `main` and run the loop on its own branch.

6. **100% coverage does not mean good tests.** That is quantity, not quality. Run `uv run mutmut run` to find covered lines that no assertion actually checks.

7. **Note**: `semgrep --config auto` needs network for semgrep registry rules.

## Safety

`harness run` launches an autonomous LLM worker with the configured permissions, e.g.
`--permission-mode acceptEdits` or `--sandbox danger-full-access`.

The gate bounds what any **commit** may touch, but the worker itself is **not** sandboxed to this repo unless you set that config. Consider the balance: without access it cannot do much. With machine access it can wreak havoc. Under a permissive mode it can run arbitrary shell. You are authorizing real changes. Choose the worker and permission mode deliberately.

#### The Gate: Tiered Checks

⚡ `harness preflight` (pre-commit) → fast checks.
Ruff lint + check format for everyone, _plus_ **containment** for the agents. Self-heals by un-staging forbidden files.

✅ `harness gate` (CI/PR pre-push). Local checks mirror CI → ruff lint + format report-only, pyright, pylint, semgrep, complexipy, hypothesis, pytest @ 100% cov.

Only humans can bypass triggered gates and commit by adding flag `--no-verify`.

<details>
<summary>
6 changes: 4 additions & 2 deletions harness/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,9 @@ def setup_git_hooks(env_bin: Path, is_windows: bool) -> Path:
rprint("\n[cyan2]Setting git hooks[/cyan2] with `git config core.hooksPath .githooks`:")
subprocess.run(("git", "config", "core.hooksPath", ".githooks"), cwd=REPO_ROOT_STR, check=True)
binary = env_bin / ("harness.exe" if is_windows else "harness")
recorded = Path(run_git(["rev-parse", "--absolute-git-dir"]).strip()).resolve() / "harness-path"
recorded = (
Path(run_git(["rev-parse", "--path-format=absolute", "--git-common-dir"]).strip()) / "harness-path"
)
recorded.write_text(f"{binary.as_posix()}\n", encoding="utf-8", newline="\n")
typer.echo(
subprocess.run(
Expand Down Expand Up @@ -228,7 +230,7 @@ def cleanup(cwd: Path, name: str | None) -> bool:
tool.setdefault("pyright", tomlkit.table()).update({"include": ["src", "preferences"]})
tool.setdefault("pytest", tomlkit.table()).setdefault("ini_options", tomlkit.table()).update({
"testpaths": ["tests"],
"pythonpath": ["src"],
"pythonpath": [".", "src"],
})
coverage = tool.setdefault("coverage", tomlkit.table())
coverage.setdefault("run", tomlkit.table()).update({"source": ["src", "preferences"]})
Expand Down
9 changes: 5 additions & 4 deletions harness/gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

All containment lists and check commands come from [tool.harness] in pyproject.toml, read once at
import into the constants below. A check is a (name, argv) pair; its `preflight`/`blocking` flags sort
it into the maps and sets this module runs on. Nothing is hardcoded here.
it into the maps and sets this module runs on.
"""

from __future__ import annotations
Expand All @@ -25,6 +25,8 @@
except ImportError: # humans do what they want with preferences.py
prefs = None

EMPTY_TREE = "4b825dc642cb6eb9a060e54bf8d69288fbee4904" # universal empty tree hash


def run_git(args: list[str], repo: Path | None = None, check: bool = True) -> str:
"""Run a git command in the repo and return its stdout.
Expand Down Expand Up @@ -199,13 +201,12 @@ def prepare_commit_msg(argv: list[str]) -> int:
Returns:
Status code integer 0 or 1 (git blocks commit on code 1)
"""
if os.environ.get("RALPH_LOOP") != "1":
if not os.environ.get("RALPH_LOOP"):
return 0
commit_msg_file: str = argv[1] if len(argv) > 1 else ""
command = argv[2] if len(argv) > 2 else ""
msg = ""
empty_tree = "4b825dc642cb6eb9a060e54bf8d69288fbee4904" # universal empty tree hash
ref = "HEAD" if run_git(["rev-parse", "--verify", "HEAD"], check=False).strip() else empty_tree
ref = "HEAD" if run_git(["rev-parse", "--verify", "HEAD"], check=False).strip() else EMPTY_TREE
if command in {"merge", "squash", "rebase", "reset", "clean", "filter-branch"}:
msg = f"You cannot use that git command `{command}`.\n"
if not run_git(["diff-index", "--cached", "--name-only", f"{ref}"]):
Expand Down
Loading
Loading