diff --git a/docs/benchmarks/software-factory-pilot.md b/docs/benchmarks/software-factory-pilot.md new file mode 100644 index 0000000000..997d914f90 --- /dev/null +++ b/docs/benchmarks/software-factory-pilot.md @@ -0,0 +1,183 @@ +# Proposed software factory pilot + +This draft measures whether application-owned context retrieval changes objective +resolution, total calls, tokens, cost per successful resolution and elapsed time. +It compares three cold arms under the same tools, graders and resource limits. +It measures controlled context utility; three tasks cannot establish adoption, +savings or a general quality improvement. + +The plan is not frozen or approved. Publishing or merging it authorizes no model +calls, index ingestion or paid benchmark. No live result is reported here. + +## Implementation and tasks + +The [factory implementation](https://github.com/ClickHouse/loom/pull/1050) supplies +offline replay, a bounded controller, injected provider/SDK bindings, authored +host behavioral checks, attempt accounting and paired analysis. A qualified +live driver and deployment-specific evidence remain prerequisites. +The example's offline replay fixtures are separate from the historical pilot. + +Use the three historical cases in the implementation's +[`historical-candidates.json`](https://github.com/ClickHouse/loom/blob/feat/flask-software-factory/examples/software_factory/historical-candidates.json): +`pallets__flask-4045`, `pallets__flask-4575` and `pallets__flask-4935`. +Each task pins its exact public source base and dependency image digest. The +imported official test selectors are diagnostic inputs: candidate Python controls +its own test process, exit status and reports, so those reports cannot authorize +a passing final grade. Retain all failures. Do not replace tasks or choose the +best attempt after observing results. + +The proposed objective grader uses narrow authored public-behavior probes: +dotted blueprint names, redirect method/delegation/override, and nested +subdomains. The sandbox returns bounded primitive observations; evaluator-owned +assertions and the final verdict stay outside candidate mounts. This coverage is +not equivalent to the imported official selectors. The final protocol must +explicitly choose this coverage and freeze its probe seed, bridge and assertions. + +Before running, independently verify passing visible neighbors on the base, a +failing target check on the base, and passing target/neighbors with the reference +on every exact image. Candidate syntax/import crashes or malformed output remain +failed checks so revision can continue; infrastructure, state or cleanup faults +invalidate grading. Recheck forgery, source-write and output-limit controls. +Imported receipts and image identity alone do not qualify another grading path. + +## Arms and schedule + +| Arm | Application behavior | Agent exposure | +| ------------ | ------------------------------------------------------------------------------------------- | --------------------------------------------- | +| Baseline | No context client construction or retrieval | No Loom schema, instructions or hooks | +| Orchestrated | Exact intake lookup, impact before changed-source verification, failure retrieval as needed | Ordinary source tools; no selected-call quota | +| Natural | No automatic context hooks | Optional exact lookup with zero minimum calls | + +All arms share the same provider/model, native response handling, source tools, +objective graders and per-attempt limits. Only retrieved context and its resulting +trajectory may differ. The treatment cannot change the source base, hidden tests, +prices, schedule, provider or retained attempts. + +Use 3 tasks × 3 paired repeats × 3 arms = 27 primary attempts. Add 3 tasks × 2 +baseline A/A copies = 6 reference attempts, for 33 attempts total. The 15 baseline +attempts allow zero Loom requests. All attempts start cold; no cross-task memory +writes, warm reuse or chronological learning is part of this pilot. + +The [proposed schedule](software-factory-schedule.json) uses seed 741 and persists +all 33 slots before calls. Its `schedule_sha256` covers the canonical attempt +list. Freeze this order with the final protocol; a seed reproduces the schedule +and analysis, not the model's responses. Retain failures, nonuse and budget stops +without replacement attempts, automatic retries or post-hoc exclusions. + +## Proposed limits + +| Resource | Per attempt | Whole primary and A/A cohort | +| ------------------------------ | ---------------------------------------------------------- | ---------------------------- | +| Generation requests | 12 | 396 | +| Input tokens | 20,000 per generation request, subject to a verified bound | 7,920,000 | +| Output tokens | 2,048 per generation request | 811,008 | +| Agent tools | 30 | 990 | +| Logical Loom reads | 6 in each context arm; zero in baseline | 108 | +| Physical Loom HTTP requests | 18 in each context arm; zero in baseline | 324 | +| Serialized transcript | 64,000 characters per generation request | Enforced on every request | +| Accepted context | 12,000 characters total | Enforced on every attempt | +| Visible tests | 3 invocations, 30 seconds each | No replacement runs | +| Independent final hidden grade | 1 invocation, 30 seconds | No model judge | +| Agent policy deadline | 180 seconds | Serial bounded attempts | +| Hard controller deadline | 210 seconds, including hidden grading | Retain interrupted attempts | + +The physical Loom cap assumes one retrieval and two single-request scope checks +for each logical read. Auth, discovery, retries, failed dispatches and any extra +checks consume the same cap. Stop before another physical request would exceed +it; additional traffic requires a revised protocol and approval. + +The generation request cap excludes a token-counting service. If a verified +counting mechanism needs network requests, freeze a separate ceiling of at most +396 counting requests, their endpoint, latency and cost before approval. Do not +silently replace an unsupported counting API or treat characters as token counts. +The proposed 20,000-input-token bound remains unqualified. + +After stopping a controller, reconcile at most four recorded, owned containers. +Each has a 10-second inspection limit and a 10-second removal limit, for at most +80 seconds of additional cleanup. Record end-to-end elapsed time including setup, +independent grading and cleanup. Incomplete cleanup halts the cohort. A timeout +or unresolved in-flight request remains a failure with unknown usage. + +## Provider and spending freeze + +Use one exact model through a direct Converse provider, with existing authorized +configuration, standard inference, no temperature default and no explicit cache +checkpoints. Preserve native reasoning blocks. Freeze its exact model identifier, +region, endpoint, API behavior and complete dependency hashes before approval. +Do not introduce an alternate gateway, provider or credential route to complete a +run. No infrastructure, permission or credential changes are authorized here. + +The working price assumption is $2.20 per million input tokens and $11 per million +output tokens. These assumptions require account-specific verification for the +exact model, region, inference class and cache semantics. They are not a promise +of current pricing or a complete bill. + +At the proposed token caps, generation reserves at most +`7,920,000 / 1,000,000 × 2.20 + 811,008 / 1,000,000 × 11 = $26.345088`. +The proposed total ceiling is **$30**, conditional on bounding all counting, +server, transport, auth, cache and infrastructure overhead within the remaining +**$3.654912**. Unknown overhead is unknown, not zero. If verified token bounds or +all-in costs do not fit, revise the plan before requesting approval. + +No dollar amount in this draft authorizes spending. Freeze the final protocol +digest, schedule, dependency locks, price table and whole-run reservation, then +obtain actual owner approval referencing that digest and ceiling. Approval of +benchmark intent or of a PR does not replace that approval. + +## Qualification required before the first request + +1. Pin the merged server/SDK integration and factory controller commits. Check the + actual integration and its required CI evidence separately. +2. Freeze public task inputs, the host probe seed and assertions, observation + bridge, source allowlists, tool schemas, prompts, runtime locks, controller and + trusted worker hashes. Keep evaluator assertions, reference fixes and + credentials outside agent access. Official test reports remain diagnostic. +3. Qualify a source-only checkout per attempt, with no Git history, future + objects, sibling mounts, network or credential access. Candidate code runs in + an isolated focused grader. Verify traversal, symlink, import, subprocess and + egress isolation. Export only allowlisted implementation edits into a separate + independent final grader; preserve evaluator tests and runner configuration. +4. Use a private server and database owned by this pilot; never reuse a production + server. Supply authorized immutable index scopes for all three exact source + bases using reader credentials. Use exact retrieval only, with no embeddings, + conceptual search or model enrichment. Verify repository, full commit, server build, + permissions and every required index tier before and after reads. External + immutability evidence is mandatory; matching watermarks alone are insufficient. + Restricted zero counts mean unknown. No index creation or grants are implied. +5. Qualify the actual controller/worker and client transport paths, hard deadlines, + durable pre-dispatch reservations and cleanup. Baseline constructs no context + client. Instrument auth and metadata side traffic or refuse it. Failed or + interrupted attempts retain their reservations and require reconciliation. +6. Verify the input-token bound and complete usage/pricing semantics. Join client + request IDs to model and server billing evidence. Count failed and retried + physical dispatches. Never record credentials or arbitrary response headers. +7. Obtain exact protocol and total-spend approval after all evidence exists. + Refuse inference when a required field or receipt is missing. + +Offline contracts, stubbed transports, scripted replay and proposed schedules +are preparation evidence. They establish no live model resolution, complete +billing, index authorization or spending permission. + +## Measurement and analysis + +Retain every scheduled attempt and its independent objective result, invalid-run +reason, source and patch hashes, provider/index provenance, failures, nonuse, +abstentions and stale, repeated or empty contexts. Separate accepted and failed +calls, host-selected and agent-selected calls, logical reads, physical dispatches +and retries. Avoid counting an agent-selected Loom call twice in tool totals. + +Keep raw input, output and cache usage, priced and unknown cost, component latency +and total elapsed time. Cost per successful resolution includes failed attempts; +zero successes has no finite estimate. Unknown usage or pricing suppresses an +all-in cost estimate. Whole-cohort cost includes A/A references. + +Pair by task and repeat. Average repeats within each task and use seed 741 for +paired bootstrap intervals that resample task clusters with arms kept together. +Report A/A variation separately. Missing attempts remain visible and suppress +partial-cohort intervals. Three task clusters support exploratory intervals, +with no significance, percentage-gain, general adoption or savings claim. + +Freeze original outputs before fixing any observed failure. Diagnose a concrete +pattern, add a focused regression, fix the smallest cause and compare under a +preserved equivalent protocol. Never tune against hidden solutions. Any rerun +needs its own reserved budget and approval; it is not a free replacement attempt. diff --git a/docs/benchmarks/software-factory-schedule.json b/docs/benchmarks/software-factory-schedule.json new file mode 100644 index 0000000000..0df8f42ac3 --- /dev/null +++ b/docs/benchmarks/software-factory-schedule.json @@ -0,0 +1,238 @@ +{ + "status": "PROPOSED_NOT_APPROVED", + "seed": 741, + "attempts": [ + { + "order": 0, + "task_id": "pallets__flask-4575", + "repeat": 1, + "arm": "natural", + "mode": "natural" + }, + { + "order": 1, + "task_id": "pallets__flask-4575", + "repeat": 1, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 2, + "task_id": "pallets__flask-4575", + "repeat": 1, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 3, + "task_id": "pallets__flask-4935", + "repeat": "aa", + "arm": "baseline-a", + "mode": "baseline" + }, + { + "order": 4, + "task_id": "pallets__flask-4935", + "repeat": "aa", + "arm": "baseline-b", + "mode": "baseline" + }, + { + "order": 5, + "task_id": "pallets__flask-4575", + "repeat": "aa", + "arm": "baseline-b", + "mode": "baseline" + }, + { + "order": 6, + "task_id": "pallets__flask-4575", + "repeat": "aa", + "arm": "baseline-a", + "mode": "baseline" + }, + { + "order": 7, + "task_id": "pallets__flask-4045", + "repeat": 1, + "arm": "natural", + "mode": "natural" + }, + { + "order": 8, + "task_id": "pallets__flask-4045", + "repeat": 1, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 9, + "task_id": "pallets__flask-4045", + "repeat": 1, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 10, + "task_id": "pallets__flask-4575", + "repeat": 2, + "arm": "natural", + "mode": "natural" + }, + { + "order": 11, + "task_id": "pallets__flask-4575", + "repeat": 2, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 12, + "task_id": "pallets__flask-4575", + "repeat": 2, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 13, + "task_id": "pallets__flask-4045", + "repeat": "aa", + "arm": "baseline-a", + "mode": "baseline" + }, + { + "order": 14, + "task_id": "pallets__flask-4045", + "repeat": "aa", + "arm": "baseline-b", + "mode": "baseline" + }, + { + "order": 15, + "task_id": "pallets__flask-4045", + "repeat": 2, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 16, + "task_id": "pallets__flask-4045", + "repeat": 2, + "arm": "natural", + "mode": "natural" + }, + { + "order": 17, + "task_id": "pallets__flask-4045", + "repeat": 2, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 18, + "task_id": "pallets__flask-4935", + "repeat": 1, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 19, + "task_id": "pallets__flask-4935", + "repeat": 1, + "arm": "natural", + "mode": "natural" + }, + { + "order": 20, + "task_id": "pallets__flask-4935", + "repeat": 1, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 21, + "task_id": "pallets__flask-4935", + "repeat": 0, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 22, + "task_id": "pallets__flask-4935", + "repeat": 0, + "arm": "natural", + "mode": "natural" + }, + { + "order": 23, + "task_id": "pallets__flask-4935", + "repeat": 0, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 24, + "task_id": "pallets__flask-4045", + "repeat": 0, + "arm": "natural", + "mode": "natural" + }, + { + "order": 25, + "task_id": "pallets__flask-4045", + "repeat": 0, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 26, + "task_id": "pallets__flask-4045", + "repeat": 0, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 27, + "task_id": "pallets__flask-4935", + "repeat": 2, + "arm": "natural", + "mode": "natural" + }, + { + "order": 28, + "task_id": "pallets__flask-4935", + "repeat": 2, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 29, + "task_id": "pallets__flask-4935", + "repeat": 2, + "arm": "baseline", + "mode": "baseline" + }, + { + "order": 30, + "task_id": "pallets__flask-4575", + "repeat": 0, + "arm": "natural", + "mode": "natural" + }, + { + "order": 31, + "task_id": "pallets__flask-4575", + "repeat": 0, + "arm": "orchestrated", + "mode": "orchestrated" + }, + { + "order": 32, + "task_id": "pallets__flask-4575", + "repeat": 0, + "arm": "baseline", + "mode": "baseline" + } + ], + "schedule_sha256": "83f5ba7bc0fd54fc6c5cd75326cbd91e5b2755d7cb8537d3268220496e8910e3" +} diff --git a/llms.txt b/llms.txt index a1a49a4b6a..bcfaf76c15 100644 --- a/llms.txt +++ b/llms.txt @@ -189,6 +189,7 @@ Start with [the docs map](docs/index.md). To install Loom, a coding agent reads - [Running the benchmarks](docs/benchmarks/running.md): Run every command from the repository root with the checkout's interpreter and the root first on `PYTHONPATH`: - [SDK issue resolution](docs/benchmarks/sdk-issue-resolution.md): The SDK issue harness measures proposed patches with and without Loom code lookup. - [Skill eval](docs/benchmarks/skill-eval.md): The skill eval grades the `loom-memory` skill by the namespace an agent leaves behind. +- [Proposed software factory pilot](docs/benchmarks/software-factory-pilot.md): This draft measures whether application-owned context retrieval changes objective resolution, total calls, tokens, cost per successful resolution and elapsed time. ## Optional