Skip to content

Add a bounded coding factory with offline replay and paired reports - #1050

Draft
zlareb1 wants to merge 18 commits into
mainfrom
feat/flask-software-factory
Draft

zlareb1 wants to merge 18 commits into
mainfrom
feat/flask-software-factory

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Related: #1052

Changelog category (leave one):

  • New Feature

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

Add a bounded coding factory example with application-owned context retrieval, offline replay, independent behavioral grading and paired attempt reports.

Why

A coding application needs context policy outside the model's tool choices, objective checks after edits and accounting that retains failed attempts. The example compares baseline, host-orchestrated retrieval and optional agent-selected retrieval with the same source tools and budgets.

Scope and risk

The example supplies source-only checkouts, bounded edit/test/revise loops, replay fixtures, injected provider/SDK bindings, physical transport observation, durable pre-dispatch reservations, worker supervision and paired analysis. CI collects its offline and stub-client contracts. The generated operation reach manifest includes the example's call sites.

Candidate code cannot authorize its own passing grade through process status or test reports. Historical grading uses isolated primitive observations and evaluator-owned assertions. Coverage is narrow authored public behavior; imported official test selectors remain diagnostic. Infrastructure and cleanup faults invalidate grading, while candidate crashes remain failed checks that permit revision.

The offline CLI constructs no live client. Actual immutable indexes, verified input-token bounds, credential/transport qualification, complete billing, a trusted live driver and exact protocol/spend approval remain required before a live cohort. Scripted replay and stubbed I/O establish no model quality, savings, adoption or live performance result.

Compatibility, rollout and rollback

  • Wire: no operations, fields or defaults change.
  • Operator steps: none for replay. Live use requires the documented independent qualification and exact approval.
  • Rollback: revert the example, CI collection and generated reach entries; no migrations or stored data changes.

Tests

  • Focused factory contracts, actual SDK serialization with stubbed I/O, reach-manifest checks and documentation/public-copy checks pass through the required resource supervisor.
  • Offline smoke runs all three authored fixtures in each mode, including source edits and failing-to-passing objective checks. It is scripted replay.
  • Retained historical preparation evidence covers base/target/reference controls, hostile candidate output, source-write refusal and cleanup. It does not qualify model resolutions or official-selector coverage.
  • Scoped formatting, lint and whitespace checks are included. Full repository gates run in hosted CI; no heavy local suite or paid benchmark is part of this PR.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated: Codex wrote most of the implementation. Focused contracts and independent objective controls check behavior; internal independent reviews inspect the final source.

  • I can explain the diff and its interaction with the example's controller, SDK bindings and graders.

Checklist

  • Full hosted format, lint, typecheck, unit and integration gates pass on the final head.
  • The operation reach manifest is generated; no wire, schema, plugin or frontend change is included.
  • Replay, authored historical controls and live qualification are distinguished.
  • Commits are signed off.

The application owns context lookup, source checks and attempt accounting.
Scripted fixtures exercise the workflow without live provider calls.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep full source authority behind bounded agent reads. Preserve request counts,
usage and failed result manifests when transport, output limits or graders fail.

Add exact-base source export, fixed focused container grading, immutable tests,
owned cleanup receipts, scope verification, quotas and controller deadlines.
Live cohorts remain disabled pending integration and protocol qualification.

Validation: 36 contracts, 9 historical controls, 18 isolation probes, 270
focused documentation checks and 9 replay attempts. No live model benchmark.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain interrupted attempts and unknown billing across restarts. Keep generation
prices separate from total cost until server and transport billing is available.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Join dispatch and response receipts to usage evidence without retaining other
headers or response content. Keep missing and malformed IDs unknown.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain failures, missing attempts and unknown billing in paired reports. Reuse
the task bootstrap helper and separate A/A references from primary comparisons.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 1:34am UTC

Request Review

@zlareb1
zlareb1 marked this pull request as draft November 20, 2026 14:18
Reject untrusted historical grades before clients or hidden evaluation.
Check physical baseline exposure, collect offline contracts in CI, regenerate
the example reach manifest, and retain
cold-worker reservations. A replacement oracle and live eligibility remain
required before a paid benchmark.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep authored assertions and final grades on the host. Pin exact source,
image, probe seed and control receipts. Candidate crashes and output limits
remain failed checks that allow revision; transport and cleanup errors halt.
The narrow probes do not claim official test coverage or live model results.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Use source-only behavioral preparation and distinguish authored host checks
from candidate-controlled pytest diagnostics. Keep actual index, token,
billing and exact protocol approval requirements explicit.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep grader faults distinct from candidate failures and agent tool errors, even when later hidden checks pass. Remove unreachable report-based qualification work while retaining its explicit refusal.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Distinguish exhausted candidate checks from grader infrastructure failures so bounded trajectories do not halt later cold attempts. Infrastructure timeouts still invalidate grading.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep only bounded visible diagnostics and refuse hidden grading before filesystem or container work. The diagnostic adapter no longer retains private hidden patches or selectors.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Remove the hidden-test application claim from the diagnostic adapter description. Hidden grading uses the separate behavioral probes.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…sk-software-factory

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
An attach deadline establishes no candidate execution outcome. Retain the transport timeout and stop evaluation when the outcome is unknown.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…sk-software-factory

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant