From 7b8b79b5e978782c418d6dc73def4ec4ee35b6d5 Mon Sep 17 00:00:00 2001 From: zlareb1 Date: Fri, 20 Nov 2026 14:18:28 +0000 Subject: [PATCH 1/3] Propose a paired software factory pilot and frozen attempt order Make the three-arm design, request ceilings and unresolved qualification requirements reviewable before exact protocol and spending approval. Signed-off-by: zlareb1 --- docs/benchmarks/software-factory-pilot.md | 169 +++++++++++++ .../benchmarks/software-factory-schedule.json | 238 ++++++++++++++++++ 2 files changed, 407 insertions(+) create mode 100644 docs/benchmarks/software-factory-pilot.md create mode 100644 docs/benchmarks/software-factory-schedule.json diff --git a/docs/benchmarks/software-factory-pilot.md b/docs/benchmarks/software-factory-pilot.md new file mode 100644 index 0000000000..87e2b3f0f0 --- /dev/null +++ b/docs/benchmarks/software-factory-pilot.md @@ -0,0 +1,169 @@ +# Proposed software factory pilot + +This draft measures whether application-owned context retrieval changes objective +resolution, total calls, tokens, cost per successful resolution and elapsed time. +It compares three cold arms under the same tools, graders and resource limits. +It measures controlled context utility; three tasks cannot establish adoption, +savings or a general quality improvement. + +The plan is not frozen or approved. Publishing or merging it authorizes no model +calls, index ingestion or paid benchmark. No live result is reported here. + +## Implementation and tasks + +The [factory implementation](https://github.com/ClickHouse/loom/pull/1050) supplies +offline replay, historical qualification, attempt accounting and paired analysis. +A qualified live runner and deployment-specific evidence remain prerequisites. +The example's offline replay fixtures are separate from the historical pilot. + +Use the three historical cases in the implementation's +[`historical-candidates.json`](https://github.com/ClickHouse/loom/blob/feat/flask-software-factory/examples/software_factory/historical-candidates.json): +`pallets__flask-4045`, `pallets__flask-4575` and `pallets__flask-4935`. +Each task pins its exact public source base, dependency image digest, visible +selectors and evaluator-private hidden selectors. Retain all failures. Do not +replace tasks or choose the best attempt after observing results. + +Before running, independently verify visible-base pass, hidden-base fail and +reference pass controls on every exact dependency image. Missing selectors, +collection errors and skipped tests invalidate qualification. Imported controls +and image identity alone do not qualify a different runner or grading path. + +## Arms and schedule + +| Arm | Application behavior | Agent exposure | +| ------------ | ------------------------------------------------------------------------------------------- | --------------------------------------------- | +| Baseline | No context client construction or retrieval | No Loom schema, instructions or hooks | +| Orchestrated | Exact intake lookup, impact before changed-source verification, failure retrieval as needed | Ordinary source tools; no selected-call quota | +| Natural | No automatic context hooks | Optional exact lookup with zero minimum calls | + +All arms share the same provider/model, native response handling, source tools, +objective graders and per-attempt limits. Only retrieved context and its resulting +trajectory may differ. The treatment cannot change the source base, hidden tests, +prices, schedule, provider or retained attempts. + +Use 3 tasks × 3 paired repeats × 3 arms = 27 primary attempts. Add 3 tasks × 2 +baseline A/A copies = 6 reference attempts, for 33 attempts total. The 15 baseline +attempts allow zero Loom requests. All attempts start cold; no cross-task memory +writes, warm reuse or chronological learning is part of this pilot. + +The [proposed schedule](software-factory-schedule.json) uses seed 741 and persists +all 33 slots before calls. Its `schedule_sha256` covers the canonical attempt +list. Freeze this order with the final protocol; a seed reproduces the schedule +and analysis, not the model's responses. Retain failures, nonuse and budget stops +without replacement attempts, automatic retries or post-hoc exclusions. + +## Proposed limits + +| Resource | Per attempt | Whole primary and A/A cohort | +| ------------------------------ | ---------------------------------------------------------- | ---------------------------- | +| Generation requests | 12 | 396 | +| Input tokens | 20,000 per generation request, subject to a verified bound | 7,920,000 | +| Output tokens | 2,048 per generation request | 811,008 | +| Agent tools | 30 | 990 | +| Logical Loom reads | 6 in each context arm; zero in baseline | 108 | +| Physical Loom HTTP requests | 18 in each context arm; zero in baseline | 324 | +| Serialized transcript | 64,000 characters per generation request | Enforced on every request | +| Accepted context | 12,000 characters total | Enforced on every attempt | +| Visible tests | 3 invocations, 30 seconds each | No replacement runs | +| Independent final hidden grade | 1 invocation, 30 seconds | No model judge | +| Agent policy deadline | 180 seconds | Serial bounded attempts | +| Hard controller deadline | 210 seconds, including hidden grading | Retain interrupted attempts | + +The physical Loom cap assumes one retrieval and two single-request scope checks +for each logical read. Auth, discovery, retries, failed dispatches and any extra +checks consume the same cap. Stop before another physical request would exceed +it; additional traffic requires a revised protocol and approval. + +The generation request cap excludes a token-counting service. If a verified +counting mechanism needs network requests, freeze a separate ceiling of at most +396 counting requests, their endpoint, latency and cost before approval. Do not +silently replace an unsupported counting API or treat characters as token counts. +The proposed 20,000-input-token bound remains unqualified. + +After stopping a controller, reconcile at most four recorded, owned containers. +Each has a 10-second inspection limit and a 10-second removal limit, for at most +80 seconds of additional cleanup. Record end-to-end elapsed time including setup, +independent grading and cleanup. Incomplete cleanup halts the cohort. A timeout +or unresolved in-flight request remains a failure with unknown usage. + +## Provider and spending freeze + +Use one exact model through a direct Converse provider, with existing authorized +configuration, standard inference, no temperature default and no explicit cache +checkpoints. Preserve native reasoning blocks. Freeze its exact model identifier, +region, endpoint, API behavior and complete dependency hashes before approval. +Do not introduce an alternate gateway, provider or credential route to complete a +run. No infrastructure, permission or credential changes are authorized here. + +The working price assumption is $2.20 per million input tokens and $11 per million +output tokens. These assumptions require account-specific verification for the +exact model, region, inference class and cache semantics. They are not a promise +of current pricing or a complete bill. + +At the proposed token caps, generation reserves at most +`7,920,000 / 1,000,000 × 2.20 + 811,008 / 1,000,000 × 11 = $26.345088`. +The proposed total ceiling is **$30**, conditional on bounding all counting, +server, transport, auth, cache and infrastructure overhead within the remaining +**$3.654912**. Unknown overhead is unknown, not zero. If verified token bounds or +all-in costs do not fit, revise the plan before requesting approval. + +No dollar amount in this draft authorizes spending. Freeze the final protocol +digest, schedule, dependency locks, price table and whole-run reservation, then +obtain actual owner approval referencing that digest and ceiling. Approval of +benchmark intent or of a PR does not replace that approval. + +## Qualification required before the first request + +1. Pin the merged server/SDK integration and factory controller commits. Check the + actual integration and its required CI evidence separately. +2. Freeze all public task inputs, hidden grader selectors and patches, source + allowlists, tool schemas, prompts, runtime locks, controller and trusted worker + hashes. Keep hidden tests, reference fixes and evaluator credentials outside + agent access. +3. Qualify a source-only checkout per attempt, with no Git history, future + objects, sibling mounts, network or credential access. Candidate code runs in + an isolated focused grader. Verify traversal, symlink, import, subprocess and + egress isolation. Export only allowlisted implementation edits into a separate + independent final grader; preserve evaluator tests and runner configuration. +4. Supply authorized immutable index scopes for all three exact source bases, + using reader credentials. Verify repository, full commit, server build, + permissions and every required index tier before and after reads. External + immutability evidence is mandatory; matching watermarks alone are insufficient. + Restricted zero counts mean unknown. No index creation or grants are implied. +5. Qualify the actual controller/worker and client transport paths, hard deadlines, + durable pre-dispatch reservations and cleanup. Baseline constructs no context + client. Instrument auth and metadata side traffic or refuse it. Failed or + interrupted attempts retain their reservations and require reconciliation. +6. Verify the input-token bound and complete usage/pricing semantics. Join client + request IDs to model and server billing evidence. Count failed and retried + physical dispatches. Never record credentials or arbitrary response headers. +7. Obtain exact protocol and total-spend approval after all evidence exists. + Refuse inference when a required field or receipt is missing. + +Offline contracts, stubbed transports, scripted replay and proposed schedules +are preparation evidence. They establish no live model resolution, complete +billing, index authorization or spending permission. + +## Measurement and analysis + +Retain every scheduled attempt and its independent objective result, invalid-run +reason, source and patch hashes, provider/index provenance, failures, nonuse, +abstentions and stale, repeated or empty contexts. Separate accepted and failed +calls, host-selected and agent-selected calls, logical reads, physical dispatches +and retries. Avoid counting an agent-selected Loom call twice in tool totals. + +Keep raw input, output and cache usage, priced and unknown cost, component latency +and total elapsed time. Cost per successful resolution includes failed attempts; +zero successes has no finite estimate. Unknown usage or pricing suppresses an +all-in cost estimate. Whole-cohort cost includes A/A references. + +Pair by task and repeat. Average repeats within each task and use seed 741 for +paired bootstrap intervals that resample task clusters with arms kept together. +Report A/A variation separately. Missing attempts remain visible and suppress +partial-cohort intervals. Three task clusters support exploratory intervals, +with no significance, percentage-gain, general adoption or savings claim. + +Freeze original outputs before fixing any observed failure. Diagnose a concrete +pattern, add a focused regression, fix the smallest cause and compare under a +preserved equivalent protocol. Never tune against hidden solutions. Any rerun +needs its own reserved budget and approval; it is not a free replacement attempt. diff --git a/docs/benchmarks/software-factory-schedule.json b/docs/benchmarks/software-factory-schedule.json new file mode 100644 index 0000000000..0df8f42ac3 --- /dev/null +++ b/docs/benchmarks/software-factory-schedule.json @@ -0,0 +1,238 @@ +{ + "status": "PROPOSED_NOT_APPROVED", + "seed": 741, + "attempts": [ + { + "order": 0, + "task_id": "pallets__flask-4575", + "repeat": 1, + "arm": "natural", + "mode": "natural" + }, + { + "order": 1, + "task_id": "pallets__flask-4575", + "repeat": 1, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 2, + "task_id": "pallets__flask-4575", + "repeat": 1, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 3, + "task_id": "pallets__flask-4935", + "repeat": "aa", + "arm": "baseline-a", + "mode": "baseline" + }, + { + "order": 4, + "task_id": "pallets__flask-4935", + "repeat": "aa", + "arm": "baseline-b", + "mode": "baseline" + }, + { + "order": 5, + "task_id": "pallets__flask-4575", + "repeat": "aa", + "arm": "baseline-b", + "mode": "baseline" + }, + { + "order": 6, + "task_id": "pallets__flask-4575", + "repeat": "aa", + "arm": "baseline-a", + "mode": "baseline" + }, + { + "order": 7, + "task_id": "pallets__flask-4045", + "repeat": 1, + "arm": "natural", + "mode": "natural" + }, + { + "order": 8, + "task_id": "pallets__flask-4045", + "repeat": 1, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 9, + "task_id": "pallets__flask-4045", + "repeat": 1, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 10, + "task_id": "pallets__flask-4575", + "repeat": 2, + "arm": "natural", + "mode": "natural" + }, + { + "order": 11, + "task_id": "pallets__flask-4575", + "repeat": 2, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 12, + "task_id": "pallets__flask-4575", + "repeat": 2, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 13, + "task_id": "pallets__flask-4045", + "repeat": "aa", + "arm": "baseline-a", + "mode": "baseline" + }, + { + "order": 14, + "task_id": "pallets__flask-4045", + "repeat": "aa", + "arm": "baseline-b", + "mode": "baseline" + }, + { + "order": 15, + "task_id": "pallets__flask-4045", + "repeat": 2, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 16, + "task_id": "pallets__flask-4045", + "repeat": 2, + "arm": "natural", + "mode": "natural" + }, + { + "order": 17, + "task_id": "pallets__flask-4045", + "repeat": 2, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 18, + "task_id": "pallets__flask-4935", + "repeat": 1, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 19, + "task_id": "pallets__flask-4935", + "repeat": 1, + "arm": "natural", + "mode": "natural" + }, + { + "order": 20, + "task_id": "pallets__flask-4935", + "repeat": 1, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 21, + "task_id": "pallets__flask-4935", + "repeat": 0, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 22, + "task_id": "pallets__flask-4935", + "repeat": 0, + "arm": "natural", + "mode": "natural" + }, + { + "order": 23, + "task_id": "pallets__flask-4935", + "repeat": 0, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 24, + "task_id": "pallets__flask-4045", + "repeat": 0, + "arm": "natural", + "mode": "natural" + }, + { + "order": 25, + "task_id": "pallets__flask-4045", + "repeat": 0, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 26, + "task_id": "pallets__flask-4045", + "repeat": 0, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 27, + "task_id": "pallets__flask-4935", + "repeat": 2, + "arm": "natural", + "mode": "natural" + }, + { + "order": 28, + "task_id": "pallets__flask-4935", + "repeat": 2, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 29, + "task_id": "pallets__flask-4935", + "repeat": 2, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 30, + "task_id": "pallets__flask-4575", + "repeat": 0, + "arm": "natural", + "mode": "natural" + }, + { + "order": 31, + "task_id": "pallets__flask-4575", + "repeat": 0, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 32, + "task_id": "pallets__flask-4575", + "repeat": 0, + "arm": "baseline", + "mode": "baseline" + } + ], + "schedule_sha256": "83f5ba7bc0fd54fc6c5cd75326cbd91e5b2755d7cb8537d3268220496e8910e3" +} From a9120e863c1aede59d04f9790edcfcc30bb14e04 Mon Sep 17 00:00:00 2001 From: zlareb1 Date: Fri, 20 Nov 2026 14:20:33 +0000 Subject: [PATCH 2/3] Keep the pilot on private infrastructure and exact retrieval Preserve the preparation proposal restrictions against production reuse and provider-backed retrieval work. Signed-off-by: zlareb1 --- docs/benchmarks/software-factory-pilot.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/benchmarks/software-factory-pilot.md b/docs/benchmarks/software-factory-pilot.md index 87e2b3f0f0..3ecd812316 100644 --- a/docs/benchmarks/software-factory-pilot.md +++ b/docs/benchmarks/software-factory-pilot.md @@ -125,8 +125,10 @@ benchmark intent or of a PR does not replace that approval. an isolated focused grader. Verify traversal, symlink, import, subprocess and egress isolation. Export only allowlisted implementation edits into a separate independent final grader; preserve evaluator tests and runner configuration. -4. Supply authorized immutable index scopes for all three exact source bases, - using reader credentials. Verify repository, full commit, server build, +4. Use a private server and database owned by this pilot; never reuse a production + server. Supply authorized immutable index scopes for all three exact source + bases using reader credentials. Use exact retrieval only, with no embeddings, + conceptual search or model enrichment. Verify repository, full commit, server build, permissions and every required index tier before and after reads. External immutability evidence is mandatory; matching watermarks alone are insufficient. Restricted zero counts mean unknown. No index creation or grants are implied. From feb53a60410589531aa123a203b5e9c1079b800f Mon Sep 17 00:00:00 2001 From: zlareb1 Date: Fri, 20 Nov 2026 20:28:15 +0000 Subject: [PATCH 3/3] Keep the pilot aligned with evaluator-owned behavioral grading Treat candidate-controlled reports as diagnostic and freeze the host probes explicitly. Generate the documentation catalog entry alongside the proposal. Signed-off-by: zlareb1 --- docs/benchmarks/software-factory-pilot.md | 40 +++++++++++++++-------- llms.txt | 1 + 2 files changed, 27 insertions(+), 14 deletions(-) diff --git a/docs/benchmarks/software-factory-pilot.md b/docs/benchmarks/software-factory-pilot.md index 3ecd812316..997d914f90 100644 --- a/docs/benchmarks/software-factory-pilot.md +++ b/docs/benchmarks/software-factory-pilot.md @@ -12,21 +12,33 @@ calls, index ingestion or paid benchmark. No live result is reported here. ## Implementation and tasks The [factory implementation](https://github.com/ClickHouse/loom/pull/1050) supplies -offline replay, historical qualification, attempt accounting and paired analysis. -A qualified live runner and deployment-specific evidence remain prerequisites. +offline replay, a bounded controller, injected provider/SDK bindings, authored +host behavioral checks, attempt accounting and paired analysis. A qualified +live driver and deployment-specific evidence remain prerequisites. The example's offline replay fixtures are separate from the historical pilot. Use the three historical cases in the implementation's [`historical-candidates.json`](https://github.com/ClickHouse/loom/blob/feat/flask-software-factory/examples/software_factory/historical-candidates.json): `pallets__flask-4045`, `pallets__flask-4575` and `pallets__flask-4935`. -Each task pins its exact public source base, dependency image digest, visible -selectors and evaluator-private hidden selectors. Retain all failures. Do not -replace tasks or choose the best attempt after observing results. - -Before running, independently verify visible-base pass, hidden-base fail and -reference pass controls on every exact dependency image. Missing selectors, -collection errors and skipped tests invalidate qualification. Imported controls -and image identity alone do not qualify a different runner or grading path. +Each task pins its exact public source base and dependency image digest. The +imported official test selectors are diagnostic inputs: candidate Python controls +its own test process, exit status and reports, so those reports cannot authorize +a passing final grade. Retain all failures. Do not replace tasks or choose the +best attempt after observing results. + +The proposed objective grader uses narrow authored public-behavior probes: +dotted blueprint names, redirect method/delegation/override, and nested +subdomains. The sandbox returns bounded primitive observations; evaluator-owned +assertions and the final verdict stay outside candidate mounts. This coverage is +not equivalent to the imported official selectors. The final protocol must +explicitly choose this coverage and freeze its probe seed, bridge and assertions. + +Before running, independently verify passing visible neighbors on the base, a +failing target check on the base, and passing target/neighbors with the reference +on every exact image. Candidate syntax/import crashes or malformed output remain +failed checks so revision can continue; infrastructure, state or cleanup faults +invalidate grading. Recheck forgery, source-write and output-limit controls. +Imported receipts and image identity alone do not qualify another grading path. ## Arms and schedule @@ -116,10 +128,10 @@ benchmark intent or of a PR does not replace that approval. 1. Pin the merged server/SDK integration and factory controller commits. Check the actual integration and its required CI evidence separately. -2. Freeze all public task inputs, hidden grader selectors and patches, source - allowlists, tool schemas, prompts, runtime locks, controller and trusted worker - hashes. Keep hidden tests, reference fixes and evaluator credentials outside - agent access. +2. Freeze public task inputs, the host probe seed and assertions, observation + bridge, source allowlists, tool schemas, prompts, runtime locks, controller and + trusted worker hashes. Keep evaluator assertions, reference fixes and + credentials outside agent access. Official test reports remain diagnostic. 3. Qualify a source-only checkout per attempt, with no Git history, future objects, sibling mounts, network or credential access. Candidate code runs in an isolated focused grader. Verify traversal, symlink, import, subprocess and diff --git a/llms.txt b/llms.txt index 8c958c7a15..20f24355a4 100644 --- a/llms.txt +++ b/llms.txt @@ -189,6 +189,7 @@ Start with [the docs map](docs/index.md). To install Loom, a coding agent reads - [Running the benchmarks](docs/benchmarks/running.md): Run every command from the repository root with the checkout's interpreter and the root first on `PYTHONPATH`: - [SDK issue resolution](docs/benchmarks/sdk-issue-resolution.md): The SDK issue harness measures proposed patches with and without Loom code lookup. - [Skill eval](docs/benchmarks/skill-eval.md): The skill eval grades the `loom-memory` skill by the namespace an agent leaves behind. +- [Proposed software factory pilot](docs/benchmarks/software-factory-pilot.md): This draft measures whether application-owned context retrieval changes objective resolution, total calls, tokens, cost per successful resolution and elapsed time. ## Optional