Repository navigation
Conversation
The application owns context lookup, source checks and attempt accounting. Scripted fixtures exercise the workflow without live provider calls. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep full source authority behind bounded agent reads. Preserve request counts, usage and failed result manifests when transport, output limits or graders fail. Add exact-base source export, fixed focused container grading, immutable tests, owned cleanup receipts, scope verification, quotas and controller deadlines. Live cohorts remain disabled pending integration and protocol qualification. Validation: 36 contracts, 9 historical controls, 18 isolation probes, 270 focused documentation checks and 9 replay attempts. No live model benchmark. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain interrupted attempts and unknown billing across restarts. Keep generation prices separate from total cost until server and transport billing is available. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Join dispatch and response receipts to usage evidence without retaining other headers or response content. Keep missing and malformed IDs unknown. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain failures, missing attempts and unknown billing in paired reports. Reuse the task bootstrap helper and separate A/A references from primary comparisons. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
zlareb1
marked this pull request as draft
November 20, 2026 14:18
4 of 5 tasks
Reject untrusted historical grades before clients or hidden evaluation. Check physical baseline exposure, collect offline contracts in CI, regenerate the example reach manifest, and retain cold-worker reservations. A replacement oracle and live eligibility remain required before a paid benchmark. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep authored assertions and final grades on the host. Pin exact source, image, probe seed and control receipts. Candidate crashes and output limits remain failed checks that allow revision; transport and cleanup errors halt. The narrow probes do not claim official test coverage or live model results. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Use source-only behavioral preparation and distinguish authored host checks from candidate-controlled pytest diagnostics. Keep actual index, token, billing and exact protocol approval requirements explicit. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep grader faults distinct from candidate failures and agent tool errors, even when later hidden checks pass. Remove unreachable report-based qualification work while retaining its explicit refusal. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Distinguish exhausted candidate checks from grader infrastructure failures so bounded trajectories do not halt later cold attempts. Infrastructure timeouts still invalidate grading. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep only bounded visible diagnostics and refuse hidden grading before filesystem or container work. The diagnostic adapter no longer retains private hidden patches or selectors. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
6 of 8 tasks
Remove the hidden-test application claim from the diagnostic adapter description. Hidden grading uses the separate behavioral probes. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…sk-software-factory Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
An attach deadline establishes no candidate execution outcome. Retain the transport timeout and stop evaluation when the outcome is unknown. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…sk-software-factory Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #1052
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
Add a bounded coding factory example with application-owned context retrieval, offline replay, independent behavioral grading and paired attempt reports.
Why
A coding application needs context policy outside the model's tool choices, objective checks after edits and accounting that retains failed attempts. The example compares baseline, host-orchestrated retrieval and optional agent-selected retrieval with the same source tools and budgets.
Scope and risk
The example supplies source-only checkouts, bounded edit/test/revise loops, replay fixtures, injected provider/SDK bindings, physical transport observation, durable pre-dispatch reservations, worker supervision and paired analysis. CI collects its offline and stub-client contracts. The generated operation reach manifest includes the example's call sites.
Candidate code cannot authorize its own passing grade through process status or test reports. Historical grading uses isolated primitive observations and evaluator-owned assertions. Coverage is narrow authored public behavior; imported official test selectors remain diagnostic. Infrastructure and cleanup faults invalidate grading, while candidate crashes remain failed checks that permit revision.
The offline CLI constructs no live client. Actual immutable indexes, verified input-token bounds, credential/transport qualification, complete billing, a trusted live driver and exact protocol/spend approval remain required before a live cohort. Scripted replay and stubbed I/O establish no model quality, savings, adoption or live performance result.
Compatibility, rollout and rollback
Tests
AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated: Codex wrote most of the implementation. Focused contracts and independent objective controls check behavior; internal independent reviews inspect the final source.
I can explain the diff and its interaction with the example's controller, SDK bindings and graders.
Checklist