Skip to content

Propose a paired software factory benchmark plan - #1052

Draft
zlareb1 wants to merge 8 commits into
mainfrom
docs/software-factory-benchmark-plan
Draft

zlareb1 wants to merge 8 commits into
mainfrom
docs/software-factory-benchmark-plan

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Related: #1050

Changelog category (leave one):

  • Documentation

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

Not required.

Why

The factory pilot needs a reviewable protocol before inference. The proposal specifies 33 cold attempts across baseline, orchestrated and natural arms, paired repeats, A/A references and a persisted seed-741 schedule.

Scope and risk

Add the proposed protocol, schedule and generated documentation catalog entry. The plan defines source isolation, evaluator-owned behavioral checks, request/token/time ceilings, complete billing and exploratory task-cluster analysis. It explicitly distinguishes narrow authored probes from candidate-controlled official test reports.

The conditional $30 total ceiling and $26.345088 generation reservation depend on unverified token and price assumptions. A qualified live driver, immutable index authorization, input-token bounds, actual pricing and complete billing remain prerequisites. Publishing or merging the plan authorizes no inference, ingestion or spending. The final digest and total ceiling require separate exact approval.

Compatibility, rollout and rollback

  • Wire: no operations, fields or defaults change.
  • Operator steps: none. No benchmark runs as part of this PR.
  • Rollback: remove the protocol, schedule and generated catalog entry; no stored data changes.

Tests

  • Markdown/JSON formatting and documentation-catalog generation run through the required resource supervisor.
  • Documentation-link, public-copy and generated-reference checks pass. Prose and whitespace checks are included.
  • The embedded schedule digest identifies the canonical 33-slot proposal. No paid or live result is claimed.
  • Full repository checks run in hosted CI; no heavy local suite is required.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated: Codex prepared the protocol and schedule. Internal independent review checks the final diff; no human line-by-line review is claimed.

  • I can explain the protocol, schedule and their limits.

Checklist

  • Full hosted checks pass on the final head.
  • The documentation catalog is generated; no wire, schema, plugin or frontend change is included.
  • Actual live qualification and exact spend approval remain separate prerequisites.
  • Commits are signed off.

Make the three-arm design, request ceilings and unresolved qualification requirements reviewable before exact protocol and spending approval.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 1:28am UTC

Request Review

Preserve the preparation proposal restrictions against production reuse and provider-backed retrieval work.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Treat candidate-controlled reports as diagnostic and freeze the host probes explicitly. Generate the documentation catalog entry alongside the proposal.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…tware-factory-benchmark-plan

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…tware-factory-benchmark-plan

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant