Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/controller-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Use Python 3.12+, FastAPI/Pydantic for strict API inputs, HTTPX for asynchronous

The controller owns task lifecycle, identity, tool authorization, deadlines, and audit metadata. The model proposes actions; code checks the tool name, declared task capabilities, argument shape, and authenticated principal's resource grant before executing. The current tool returns a constant synthetic result. Future real tools must provide independent domain-side authorization and explicit person/resource/action scopes.

A single bounded worker consumes an in-memory queue. SQLite persists task IDs, owner, agent, status, times, results, errors, and redacted tool audit metadata. Prompts and intermediate messages stay in memory. Accepted unfinished tasks fail as `interrupted` after shutdown/restart; they are never replayed silently. SQLite task records are authoritative for lifecycle, not domain memory. Admission has a global capacity; queue wait counts toward the task deadline.
A single bounded worker consumes an in-memory queue. SQLite persists task IDs, owner, agent, status, times, results, errors, and redacted tool audit metadata. Prompts and intermediate messages stay in memory. Accepted unfinished tasks fail as `interrupted` after shutdown/restart; they are never replayed silently. SQLite task records are authoritative for lifecycle, not domain memory. Admission has a global capacity. Queue wait and execution are separate budgets: the execution deadline starts at dispatch, and a task that waits longer than the queue limit fails as `queue_expired` without any inference.

One controller process owns one database, enforced with a Unix file lock. Do not use multiple Uvicorn workers or replicas. There is no broker, workflow framework, container requirement, vector database, or distributed scheduler. Fair scheduling, durable resumable execution, and multiple replicas are future requirements, not promises of this prototype.

Expand Down
7 changes: 4 additions & 3 deletions docs/controller.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ Set `FORGE_REPOSITORIES_FILE` to an operator-owned JSON manifest outside Git:

Grant the intended principal `agents: ["forge", "repository_analyst"]` and `repositories: ["forge"]` in its private credential record. Submit an analyst task with `agent: "repository_analyst"` and `tools: ["list_repository_files", "read_repository_file"]`. The convenience CLI currently submits only the original `forge` tasks; use the HTTP API for analyst tasks. Configure an adequate tool budget (`FORGE_TOOL_ROUNDS=8` was used for the first experiment).

The analyst can list only manifest entries and read up to 80 numbered lines per call (at most 12 KiB returned). Approved files must be non-hidden `.py`, `.md`, or `.toml` sources. File reads are limited to 64 KiB, reject symlinks at every component, and reject non-regular/multiply-linked files and invalid UTF-8. Neither listing nor reading uses a command-execution tool. Selecting a source for the manifest is an operator responsibility: even source files may contain secrets and must be reviewed before granting access. Keep the manifest restricted to selected source files, not an entire home directory.
The analyst can list only manifest entries and read up to 80 numbered lines per call (at most 12 KiB returned). A line is LF-terminated, exactly as git and editors number it; other Unicode line separators (form feed, U+2028 and similar) do not start a new line. Work orders reject sources containing carriage returns (`unsupported_line_endings`) because exact-replacement admission cannot rely on them. Approved files must be non-hidden `.py`, `.md`, or `.toml` sources. File reads are limited to 64 KiB, reject symlinks at every component, and reject non-regular/multiply-linked files and invalid UTF-8. Neither listing nor reading uses a command-execution tool. Selecting a source for the manifest is an operator responsibility: even source files may contain secrets and must be reviewed before granting access. Keep the manifest restricted to selected source files, not an entire home directory.

Every read rechecks principal repository permission, requested task capability, path membership, argument shape, and limits. Approved file/range metadata is audited without source content. A task cannot finish as an analyst without at least one successful source read (`sources_not_read`); this does not prove its conclusions are correct. Source content remains untrusted. Reports require senior review; see the [first experiment](experiments/repository-analyst-2026-10-03.md).

Expand Down Expand Up @@ -281,7 +281,8 @@ with context 4096 in the prototype comparison.
| `FORGE_EXAMPLES_FILE` | Optional reviewed example collection; see [feedback loop](feedback-loop.md) |
| `FORGE_CONTEXT_LENGTH` | Positive integer; default 4096 context tokens |
| `FORGE_MAX_OUTPUT_TOKENS` | Positive integer; default 512 output tokens |
| `FORGE_TASK_TIMEOUT` | 60 seconds including queue wait |
| `FORGE_TASK_TIMEOUT` | 60 seconds of execution, measured from dispatch (queue wait is not counted) |
| `FORGE_QUEUE_TIMEOUT` | 600 seconds a task may wait before dispatch; finite positive number. Expiry settles the task and frees capacity even while another task is running. |
| `FORGE_CAPACITY` | 8 admitted tasks, including running work |
| `FORGE_TOOL_ROUNDS` | Maximum 3 total tool calls |
| `FORGE_CONTROLLER_URL` | Client URL, default `http://127.0.0.1:8787` |
Expand All @@ -303,7 +304,7 @@ All routes except liveness require `Authorization: Bearer <token>`. There is no
| `POST /v1/tasks/{id}/cancel` | Cancel queued/running task; completed records remain terminal |
| `DELETE /v1/tasks/{id}` | Delete settled terminal record; 409 while active |

Statuses: `queued`, `running`, `succeeded`, `failed`, `cancelled`. Errors include `model_unavailable`, `invalid_model_response`, `tool_denied`, `tool_limit_exceeded`, `deadline_exceeded`, and `interrupted`. Unknown or another owner's ID returns 404. Invalid credentials return 401; forbidden capability requests 403; invalid input 422; large bodies 413; full admission capacity 503. Cancellation prevents result publication but cannot undo a remote action; only synthetic read-only actions exist here.
Statuses: `queued`, `running`, `succeeded`, `failed`, `cancelled`. Errors include `model_unavailable`, `invalid_model_response`, `tool_denied`, `tool_limit_exceeded`, `deadline_exceeded` (execution exceeded its limit), `queue_expired` (waited too long before dispatch; no inference ran), and `interrupted`. Unknown or another owner's ID returns 404. Invalid credentials return 401; forbidden capability requests 403; invalid input 422; large bodies 413; full admission capacity 503. Cancellation prevents result publication but cannot undo a remote action; only synthetic read-only actions exist here.

## Operation and portability

Expand Down
4 changes: 2 additions & 2 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,8 @@ There is no automatic fallback to the old host.

Submit analyst/implementer tasks with `read_repository_file` in their declared
tools and specific source paths/ranges. Poll using the returned ID until
`succeeded`, `failed` or `cancelled`. Avoid busy polling; queue wait counts toward
the controller deadline. Every inference round uses the configured role model.
`succeeded`, `failed` or `cancelled`. Avoid busy polling; queue wait is limited
separately (`queue_expired`) and does not count toward the execution deadline. Every inference round uses the configured role model.
The bridge performs no implicit retries: a lost submission response may correspond
to an accepted task. Do not blindly resubmit. Retain known task IDs across client
reconnections; disconnecting does not cancel work. Use explicit cancellation.
Expand Down
13 changes: 13 additions & 0 deletions docs/work-orders.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,19 @@ file is bounded to 64 KiB, at most 20 files per order; normal admission capacity
still applies. The existing 32 KiB request and 8,000-character compiled prompt
limits remain in force. Oversized or invalid orders fail rather than truncate.

A rejected order returns `422` with `{"detail": "invalid_work_order", "reason": CODE}`
and never queues inference. `CODE` comes from a closed vocabulary, so a caller can
act on it: `invalid_work-order` (does not match the published schema),
`principal_denied`, `scope_denied` (repository or tool not granted),
`runtime_limits_mismatch`, `source_revision_unavailable`, `stale_revision`
(order revision is not the checkout's current HEAD), `duplicate_source_path`,
`source_denied`, `source_hash_mismatch`, `invalid_source_encoding`,
`unsupported_line_endings` (source contains a carriage return) and
`compiled_prompt_too_large`. Only codes of that exact shape are echoed; any other
internal failure returns the bare `invalid_work_order`. The MCP bridge relays the
code as `invalid_request: CODE` and never relays other upstream text. Duplicate IDs
still return `409`.

Native workers list only selected files and read only captured UTF-8 source from
the pinned commit. Subsequent checkout changes cannot alter their reads. Reviewed
examples are omitted from native tasks to avoid unrelated source context. Existing
Expand Down
2 changes: 1 addition & 1 deletion scripts/controller.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ def start(config_path):
allowed = {'FORGE_PRINCIPALS_FILE', 'FORGE_REPOSITORIES_FILE', 'FORGE_EXAMPLES_FILE',
'FORGE_DATABASE', 'FORGE_MODEL', 'FORGE_ROLE_MODELS', 'FORGE_TOOL_ROUNDS',
'FORGE_CONTEXT_LENGTH', 'FORGE_MAX_OUTPUT_TOKENS', 'FORGE_TASK_TIMEOUT',
'FORGE_CAPACITY', 'FORGE_MODEL_URL'} | LISTENER_KEYS
'FORGE_CAPACITY', 'FORGE_QUEUE_TIMEOUT', 'FORGE_MODEL_URL'} | LISTENER_KEYS
if not isinstance(config, dict) or set(config)-allowed or any(not isinstance(v,str) for v in config.values()):
raise SystemExit('Invalid local configuration')
command = server_command(config)
Expand Down
62 changes: 57 additions & 5 deletions src/forge_controller/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
import hmac
import json
import logging
import re
import time
from datetime import datetime, timezone
from contextlib import asynccontextmanager
Expand Down Expand Up @@ -29,6 +30,9 @@ def format(self, record):


log = logging.getLogger('forge_controller')
# HandoffError messages are a closed vocabulary of snake_case codes written by this
# project. Only a value of that exact shape is echoed; anything else stays generic.
REASON = re.compile(r'[a-z0-9_-]{1,64}')



Expand Down Expand Up @@ -67,10 +71,15 @@ class TaskInput(BaseModel):
tools: list[str] = Field(default_factory=list, max_length=3)


class QueueExpired(Exception):
"""Task waited longer than queue_timeout before dispatch; no inference ran."""


class Engine:
def __init__(self, settings, store, model):
self.settings, self.store, self.model = settings, store, model
self.queue = asyncio.Queue(maxsize=settings.capacity)
self.queue_timers = {}
self.active = {}
self.capture_lock = asyncio.Lock()
self.worker = None
Expand All @@ -83,9 +92,37 @@ def submit(self, owner, task, binding=None):
task_id = self.store.create(owner, task.agent, binding)
self.active[task_id] = None
self.queue.put_nowait((task_id, owner, task, time.monotonic(), binding))
self.queue_timers[task_id] = asyncio.get_running_loop().call_later(
self.settings.queue_timeout, self.expire_queued, task_id)
log.info('task_queued', extra={'task_id': task_id, 'agent': task.agent})
return task_id

def expire_queued(self, task_id):
self.queue_timers.pop(task_id, None)
if task_id not in self.active or self.active[task_id] is not None:
return
row = self.store.get(task_id)
if row['status'] == 'queued':
self.store.update(task_id, 'failed', error='queue_expired', audit=row['audit'])
# Remove the source packet as well as the admission slot. Rebuild through
# Queue's public API, preserving FIFO order and unfinished-task accounting.
# This synchronous callback cannot interleave with dispatch or submission.
pending = []
removed = 0
while not self.queue.empty():
entry = self.queue.get_nowait()
if entry[0] != task_id:
pending.append(entry)
else:
removed += 1
for entry in pending:
self.queue.put_nowait(entry)
self.queue.task_done()
for _ in range(removed):
self.queue.task_done()
self.active.pop(task_id)
log.info('task_finished', extra={'task_id': task_id, 'status': self.store.get(task_id)['status']})

def cancel(self, task_id):
if self.store.get(task_id)['status'] not in TERMINAL:
self.store.update(task_id, 'cancelled', error='cancelled_by_user', audit=self.store.get(task_id)['audit'])
Expand Down Expand Up @@ -160,24 +197,31 @@ async def run(self, task_id, owner, task, binding=None):
async def work(self):
while True:
task_id, owner, task, queued_at, binding = await self.queue.get()
timer = self.queue_timers.pop(task_id, None)
if timer:
timer.cancel()
try:
if self.store.get(task_id)['status'] in TERMINAL:
continue
# Queue wait and execution are separate budgets. The execution
# deadline starts at dispatch; waiting too long is its own outcome
# and is decided before the task is ever marked running.
if time.monotonic() - queued_at >= self.settings.queue_timeout:
Comment thread
kudama marked this conversation as resolved.
Resolved
raise QueueExpired()
self.store.update(task_id, 'running')
remaining = self.settings.timeout - (time.monotonic() - queued_at)
if remaining <= 0:
raise TimeoutError()
running = asyncio.create_task(self.run(task_id, owner, task, binding))
self.active[task_id] = running
try:
async with asyncio.timeout(remaining):
async with asyncio.timeout(self.settings.timeout):
result, audit = await running
self.store.update(task_id, 'succeeded', result=result, audit=audit)
except asyncio.CancelledError:
# User cancellation must not kill the queue worker.
if self.closing:
self.store.update(task_id, 'failed', error='interrupted', audit=self.store.get(task_id)['audit'])
raise
except QueueExpired:
self.store.update(task_id, 'failed', error='queue_expired', audit=self.store.get(task_id)['audit'])
except TimeoutError:
self.store.update(task_id, 'failed', error='deadline_exceeded', audit=self.store.get(task_id)['audit'])
except ModelError as exc:
Expand All @@ -192,6 +236,9 @@ async def work(self):

async def stop(self):
self.closing = True
for timer in self.queue_timers.values():
timer.cancel()
self.queue_timers.clear()
if self.worker:
self.worker.cancel()
await asyncio.gather(self.worker, return_exceptions=True)
Expand Down Expand Up @@ -281,7 +328,12 @@ async def submit_order(order: dict, owner=Depends(principal)):
# Capture before queueing; fixed Git reads are never worker tools.
sources = await asyncio.to_thread(preflight, order, app.state.settings, owner)
arguments = task_arguments(order)
except (HandoffError, ValueError, KeyError):
except HandoffError as exc:
body = {'detail': 'invalid_work_order'}
if REASON.fullmatch(str(exc)):
body['reason'] = str(exc)
return JSONResponse(body, status_code=422)
except (ValueError, KeyError):
raise HTTPException(422, 'invalid_work_order') from None
if app.state.store.has_order(owner, order['work_order_id']):
raise HTTPException(409, 'work_order_already_submitted')
Expand Down
6 changes: 6 additions & 0 deletions src/forge_controller/config.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import json
import math
import os
import stat
from dataclasses import dataclass, field
Expand All @@ -20,6 +21,7 @@ class Settings:
repositories: dict[str, dict] = field(default_factory=dict)
examples_file: Path | None = None
role_models: dict[str, str] = field(default_factory=dict)
queue_timeout: float = 600

def __post_init__(self):
if not isinstance(self.role_models, dict) or any(
Expand All @@ -37,6 +39,9 @@ def __post_init__(self):
raise ValueError("Invalid model URL")
if self.timeout <= 0 or self.capacity < 1 or self.tool_rounds < 1:
raise ValueError("Limits must be positive")
if (type(self.queue_timeout) not in {int, float}
or not math.isfinite(self.queue_timeout) or self.queue_timeout <= 0):
raise ValueError("queue_timeout must be a finite positive number")
if not self.model or not self.principals:
raise ValueError("Model and principals are required")
tokens = []
Expand Down Expand Up @@ -93,6 +98,7 @@ def from_environment(cls):
context_length=int(os.environ.get("FORGE_CONTEXT_LENGTH", "4096")),
max_output_tokens=int(os.environ.get("FORGE_MAX_OUTPUT_TOKENS", "512")),
timeout=float(os.environ.get("FORGE_TASK_TIMEOUT", "60")),
queue_timeout=float(os.environ.get("FORGE_QUEUE_TIMEOUT", "600")),
capacity=int(os.environ.get("FORGE_CAPACITY", "8")),
tool_rounds=int(os.environ.get("FORGE_TOOL_ROUNDS", "3")),
examples_file=Path(os.environ["FORGE_EXAMPLES_FILE"]).expanduser() if os.environ.get("FORGE_EXAMPLES_FILE") else None,
Expand Down
4 changes: 2 additions & 2 deletions src/forge_controller/evaluate.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ def assess(case, task, settings, owner):
try:
source = execute(settings, owner, case['agent'], 'read_repository_file', {
'repository': 'forge', 'path': 'src/forge_controller/config.py', 'start_line': 1, 'end_line': 80})
lines = {int(line.split(': ', 1)[0]): line.split(': ', 1)[1] for line in source['content'].splitlines()}
lines = {int(line.split(': ', 1)[0]): line.split(': ', 1)[1] for line in source['content'].split('\n')}
cited = [int(n) for n in re.findall(r'src/forge_controller/config\.py:(\d+)', result)]
checks['cited_defaults_match_source'] = bool(cited) and all(n in lines for n in cited) and all(
any(field in lines[n] and value in lines[n] for n in cited) for field, value in [('context_length','4096'),('max_output_tokens','512')])
Expand All @@ -57,7 +57,7 @@ def assess(case, task, settings, owner):
'repository': 'forge', 'path': patch['path'], 'start_line': 1, 'end_line': 80})
if source['total_lines'] > 80:
raise ValueError('Fixture exceeds checker scope')
text = '\n'.join(line.split(': ', 1)[1] for line in source['content'].splitlines()) + '\n'
text = '\n'.join(line.split(': ', 1)[1] for line in source['content'].split('\n')) + '\n'
if text.count(edit['before']) != 1 or text.count('temperature=0') != 1:
raise ValueError('Patch does not match exact fixture')
candidate = text.replace(edit['before'], edit['after'], 1)
Expand Down
Loading
Loading