Repository navigation
Conversation
Make the three-arm design, request ceilings and unresolved qualification requirements reviewable before exact protocol and spending approval. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
4 of 5 tasks
Preserve the preparation proposal restrictions against production reuse and provider-backed retrieval work. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Treat candidate-controlled reports as diagnostic and freeze the host probes explicitly. Generate the documentation catalog entry alongside the proposal. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
6 of 8 tasks
…tware-factory-benchmark-plan Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…tware-factory-benchmark-plan Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #1050
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
Not required.
Why
The factory pilot needs a reviewable protocol before inference. The proposal specifies 33 cold attempts across baseline, orchestrated and natural arms, paired repeats, A/A references and a persisted seed-741 schedule.
Scope and risk
Add the proposed protocol, schedule and generated documentation catalog entry. The plan defines source isolation, evaluator-owned behavioral checks, request/token/time ceilings, complete billing and exploratory task-cluster analysis. It explicitly distinguishes narrow authored probes from candidate-controlled official test reports.
The conditional $30 total ceiling and $26.345088 generation reservation depend on unverified token and price assumptions. A qualified live driver, immutable index authorization, input-token bounds, actual pricing and complete billing remain prerequisites. Publishing or merging the plan authorizes no inference, ingestion or spending. The final digest and total ceiling require separate exact approval.
Compatibility, rollout and rollback
Tests
AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated: Codex prepared the protocol and schedule. Internal independent review checks the final diff; no human line-by-line review is claimed.
I can explain the protocol, schedule and their limits.
Checklist