Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .githooks/_resolve
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Used by the git hooks so they can run checks.
# e.g.
# . "$(dirname "$0")/_resolve"
# exec "$HARNESS" preflight
#
# Loads the harness executable.
# The harness executable was created in cli.py at `def record_harness`` called with `harness install`.
#
# $HARNESS is the path recorded by `record_harness`.
#
# Then, git hook does not require `uv` or `pip` or any specific environment layout.

recorded="${GIT_DIR:-$(git rev-parse --absolute-git-dir)}/harness-path"
if [ ! -r "$recorded" ]; then
echo "loopgate: hooks are not installed. Run 'harness install' in this repo." >&2
exit 1
fi

HARNESS="$(cat "$recorded")"
if [ ! -x "$HARNESS" ]; then
echo "loopgate: recorded harness '$HARNESS' is gone. Re-run 'harness install'." >&2
exit 1
fi
3 changes: 2 additions & 1 deletion .githooks/pre-commit
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,5 @@
# Bypassing this hook is forbidden by AGENTS.md.
set -eu

.venv/bin/harness preflight
. "$(dirname "$0")/_resolve"
exec "$HARNESS" preflight
3 changes: 2 additions & 1 deletion .githooks/pre-push
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,5 @@
# Bypassing this hook is forbidden by AGENTS.md.
set -eu

.venv/bin/harness gate
. "$(dirname "$0")/_resolve"
exec "$HARNESS" gate
5 changes: 4 additions & 1 deletion .githooks/prepare-commit-msg
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
#!/bin/sh
# Agent containment on the proposed commit message. Runs even under --no-verify.
# Bypassing this hook is forbidden by AGENTS.md.
set -eu

.venv/bin/python -c 'import sys; from harness.gate import prepare_commit_msg; raise SystemExit(prepare_commit_msg(sys.argv))' "$@"
. "$(dirname "$0")/_resolve"
exec "$HARNESS" prepare-commit-msg "$@"
37 changes: 10 additions & 27 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
name: gate

# The unbypassable backstop.
# hook, but this re-runs the same checks on every PR and push, so nothing merges
# without passing them. Branch protection makes this required.
# Unbypassable backstop. Re-runs the same local checks on every PR and push so nothing merges# without passing all. Branch protection makes this required.

on:
pull_request:
Expand All @@ -25,19 +23,25 @@ concurrency:
jobs:
dependency-review:
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
timeout-minutes: 10
steps:
- uses: actions/checkout@v6
- uses: actions/dependency-review-action@v5
with:
fail-on-severity: moderate
# On push there is no PR base/head, so diff the pushed commit range.
base-ref: ${{ github.event.pull_request.base.sha || github.event.before }}
head-ref: ${{ github.event.pull_request.head.sha || github.event.after }}
fail-on-severity: low
fail-on-scopes: runtime, development
vulnerability-check: true
warn-only: false

gate:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
runs-on: ${{ matrix.os }}
timeout-minutes: 45
steps:
- uses: actions/checkout@v6
Expand All @@ -51,24 +55,3 @@ jobs:

- name: Harness Checks
run: uv run --no-sync harness gate

# The full gate above runs once on Linux.
# This job runs on Windows we can't run. It only proves install + collection work.
# A Windows dev should add a real ralph.ps1 test.
loop-runner:
strategy:
fail-fast: false
matrix:
os: [macos-latest, windows-latest]
runs-on: ${{ matrix.os }}
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
- name: Install uv
uses: astral-sh/setup-uv@v6
with:
enable-cache: true
- name: uv Sync Dependencies
run: uv sync
- name: Loop-runner tests (ralph.sh on POSIX; skips on Windows)
run: uv run --no-sync pytest harness/tests/test_ralph.py -q
7 changes: 4 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,14 @@ __pycache__/
.ruff_cache/
.hypothesis/
.coverage
dist/
.DS_Store
.claude
.codex
.vscode/
docs/launch.md
test-results/
**/node_modules/
.python-version
uv.lock
harness-path

.env
.env.*
Expand All @@ -28,3 +26,6 @@ scratchpad/*
!scratchpad/runs/
scratchpad/runs/*
!scratchpad/runs/.gitkeep

# kept at the last line for easy deletion (real projects need lockfile)
uv.lock
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Humans edit

- `pyproject.toml`: Python and tool configuration.
- `harness/`: CLI, gate, loop runner, and harness tests. Wraps the loop that starts agents.
- `src/preferences` can contain user-specific preferences. A live Python example is already there.
- `preferences/` contains user-specific preferences. A live Python example is already there.
- `docs/PROMPT.md`: tells agents how to operate headless in the repo (mechanics)
- [A javascript example](harness/js-scaffold) lives in its own directory within [`harness/`](harness) and can be expanded on.

Expand Down Expand Up @@ -62,7 +62,7 @@ Changes to the [`harness/`](harness) itself should preserve the core contract: *

### Where tests live

The harness's own tests live in [`harness/tests/`](harness/tests), including the Hypothesis property tests in [`test_properties.py`](harness/tests/test_properties.py).
The harness's own tests live in [`harness/tests/`](harness/tests). Hypothesis coverage for the gate and preferences lives in [`test_properties.py`](tests/preferences/test_properties.py).

The full suite runs as part of `harness gate` (at 100% coverage). To run only the harness tests while iterating:

Expand All @@ -80,7 +80,7 @@ The full gate before a pull request [`harness/gate.py line 177`](harness/gate.py

```sh
harness gate
# or, to mimic what an agent will see:
# To mimic what an agent will see add the env var:
RALPH_LOOP=1 harness gate
```

Expand Down
78 changes: 53 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@

## TL;DR: Getting Started.

1. `gh repo create my-app --template <your-gh-username>/loopgate_harness --private --clone` **or** ['Use This Template'](https://github.com/new?template_name=loopgate_harness&template_owner=rxdt)
2. `uv run harness install <your-project-name>`
1. `gh repo create my-app-name --template <your-gh-username>/loopgate_harness --private --clone` **or** ['Use This Template'](https://github.com/new?template_name=loopgate_harness&template_owner=rxdt)
2. Install dependancies e.g. `uv sync && source .venv/bin/activate && harness install <your-project-name`.
3. Write your project goal in [docs/plan.md](docs/plan.md)
4. `harness run <agent=claude|codex|agy|copilot> [max_iterations] [max_minutes]`
5. Not what you wanted? Refine [`docs/plan.md`](docs/plan.md) / [`docs/PROMPT.md`](docs/PROMPT.md) and re-run
Expand All @@ -51,7 +51,7 @@
## Details

> [!IMPORTANT]
> Default configurations In [`pyproject.toml`](pyproject.toml) Update tool settings, add agent calls, remove or include checks... or leave as it.
> Default configurations In [`pyproject.toml`](pyproject.toml) Update tool settings, add agent calls, remove or include checks... or leave as is.

`docs/PROMPT.md` tells each agent to pick a `spec` and build. `docs/specs/` say _what_ to build. The agent decides _what next_. You keep `docs/plan.md` current, and specs get rewritten from it (agent is told in `docs/PROMPT.md` to update the specs). Each iteration the agent updates its spec and `PROJECT_STATUS`. Ideas from [ghuntley](https://github.com/ghuntley), How to Ralph Wiggum.

Expand All @@ -60,18 +60,40 @@

## Start a project

1. From inside the checkout, run `harness install <your-project-name>` to name the project, installs dependencies, and set up the 3 git hook.
1. From the root, run `harness install <your-project-name>` to name the project, install dependencies, set up the three git hooks, and delete excess files.
2. Write your grand vision into `docs/plan.md`.
3. Optionally add the first spec in `docs/specs/`, or have an agent draft the first specs.
4. Put product code under `src/` and list new source directories in `pyproject.toml [tool.coverage.run]`.
5. Strict Ruff rules, type checking, pyright, complexipy, and pytest coverage are set in `pyproject.toml`.
6. Your coding quirks go in [`src/preferences/preferences.py`](src/preferences/preferences.py).
6. Your coding quirks go in [`preferences/preferences.py`](preferences/preferences.py).
7. Run a loop:

```sh
harness run <agent> [max_iterations] [max_minutes] # agent: claude/codex/agy/copilot. ralph loop runner adds prompt
```

### Works with `uv`, `poetry`, or `pip`

```sh
uv sync
source .venv/bin/activate
harness install <your-project-name>
harness gate
harness run <agent>>

poetry install
poetry run harness install <your-project-name>
poetry run harness gate
poetry run harness <agent>

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt -e .
harness install <your-project-name>
harness gate
harness run <agent>
```

![L∞P architecture engine flow](.loops.svg)

## A L∞PS Loop
Expand Down Expand Up @@ -114,29 +136,31 @@ Only humans can bypass triggered gates and commit by adding flag `--no-verify`.
</summary>

```
harness/ the gate, loop runner, CLI, custom user checks (🤖 forbidden)
gate.py mirror the CI locally + preferences.py honored (🤖 forbidden)
tests/ the harness's own tests (🤖 forbidden)
test_properties.py hypothesis tests (🤖 forbidden)
.githooks/ pre-commit / pre-push gate hooks (🤖 forbidden)
.github/ CI that re-runs the gate (🤖 forbidden)
harness/ the gate, loop runner, CLI (🤖 forbidden directory)
gate.py mirror the CI locally + preferences.py honored
cli.py command-line entry point
tests/ the harness's own tests
js-scaffold javascript example to build upon
preferences/ user-defined preferences not covered by tools (🤖 forbidden directory)
tests/
preferences/ (🤖 tests/preferences is forbidden directory)
.githooks/ pre-commit / pre-push gate hooks (🤖 forbidden directory)
.github/ CI that re-runs the gate (🤖 forbidden directory)
pyproject.toml project + tooling config (🤖 forbidden)
AGENTS.md rules for agents working in the repo (🤖 forbidden)
docs/PROMPT.md the standing per-iteration instruction (human maintained)
docs/ PLAN, PROJECT_STATUS, PROMPT (human maintained plan.md)
docs/ PLAN, PROJECT_STATUS, PROMPT (human or agent maintained plan.md)
scratchpad/ scratch dir agents can use for temp files (For the 🤖 to play)
docs/specs/ WHAT to build, one PRIORITY-bannered file per track
docs/specs/ WHAT to build, one PRIORITY-bannered file per track (agent maintained)
src/ your product/source code (add to coverage source)
preferences/
preferences.py user-defined preferences not covered by tools (🤖 forbidden)
```

[`pyproject.toml`](pyproject.toml) is the single source of harness configuration. Humans own it and [`preferences/`](preferences/); both are agent-protected.

If an agent edits a forbidden file, the file will be unstaged (not allowed to commit). A forbidden pattern by an agent (e.g. `# noqa` will also prevent their commit and force them to fix it.)

</details>

[`pyproject.toml`](pyproject.toml) is the single source of harness configuration. Humans own all of it (`pyproject.toml` is agent-forbidden; `harness/preferences.py` is part of `harness/`).

A minimal `[tool.harness.gate]` snippet could look like:

```toml
Expand All @@ -149,19 +173,21 @@ patterns = ["# noqa"] # banned in agent-authored diffs
pytest = "uv sync pytest" # one check command, run by the local gate AND CI
```

## ⚠️ Warnings. Read this before a first run.
## Read this before a first run.

1. **This harness does not sandbox agents.** It _tries_ to harness bad code in loops via gates. Sandboxing agents will, e.g. prevent them from maintaining git, running Playwright, being seen as trustworthy by semgrep leading to cyclical failures, etc.

2. **The gate is a guardrail, not a jail.** Agents are crafty, like people. They will find a way to complete a task at all costs. **Trust nothing and no one.**

3. **Mind your usage limits.** `harness run` works agents to the cap set. You can easily burn through your tokens, context windows, and provider usage limits. **Workers continue running as long as there is work to do.**

4. **`docs/PROMPT.md` tells the worker to push every iteration**. Protect `main` and run the loop on its own branch.
4. **`docs/PROMPT.md` tells the worker to push or not**.

5. Protect `main` and run the loop on its own branch.

5. **100% coverage does not mean good tests.** That is quantity, not quality. (Upcoming feature: mutation testing)
6. **100% coverage does not mean good tests.** That is quantity, not quality. (Upcoming feature: mutation testing)

6. **Note**: `semgrep --config auto` needs network for semgrep registry rules.
7. **Note**: `semgrep --config auto` needs network for semgrep registry rules.

## Commands

Expand All @@ -180,13 +206,15 @@ harness run codex 2 20
harness run agy 3 10
harness run copilot 2 20
```
### Running with claude

### Running with claude

To run LoopGate with Claude :

```sh
harness run claude 2 20
```

Note: The worker must be installed and authenticated separately.

<details>
Expand All @@ -196,10 +224,10 @@ Note: The worker must be installed and authenticated separately.

- Edit rules at [pyproject.toml](pyproject.toml) for [ruff](https://docs.astral.sh/ruff/), [pylint](https://pypi.org/project/pylint/), [pydoclint](https://pypi.org/project/pydoclint/0.9.1/), [pyright](https://github.com/microsoft/pyright), [pytest](https://docs.pytest.org/en/stable/), [hypothesis](https://hypothesis.readthedocs.io/), [complexipy](https://github.com/rohaquinlop/complexipy)
- Add forbidden files, directories, or patterns in `[tool.harness.gate]` at [pyproject.toml](pyproject.toml)
- Add Hypothesis tests in any test directory, examples at [test_properties.py](harness/tests/test_properties.py)
- Add [Hypothesis](https://hypothesis.readthedocs.io/) tests in any test directory, examples at [test_properties.py](tests/preferences/test_properties.py).
- [semgrep](https://docs.semgrep.dev/semgrep-ci/sample-ci-configs) has no repo config here. It uses registry configs plus Semgrep's built-in defaults which ignore tests.
- Edit `[tool.harness.gate.checks]` in [pyproject.toml](pyproject.toml). [ci.yml](.github/workflows/ci.yml) runs the same `harness gate`.
- Removing existing preferences or add your own preferences at [preferences.py](src/preferences/preferences.py). Current preferences:
- Remove existing preferences or add your own at [preferences.py](preferences/preferences.py). Current preferences:

```py
function_argument_assignment_has_star # agents use non-specific `def fun(*)`
Expand Down Expand Up @@ -254,7 +282,7 @@ npm run --prefix harness/js-scaffold preflight
- There is NO worktree/branch creation by design. You can create branches/trees and run a loop in each, then merge _(if you really feel like managing that)_
- Agent duties can be contained to a part of the repo. e.g. Codex-1-frontend uses `docs/specs/frontend.md`, Claude-2-researcher `docs/specs/backend`...

### If you must be a ringleader
### If you want to run a graph

**Recommendations for running several agents at once on one branch (no worktrees):**

Expand Down
Loading
Loading