Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
8bfe187
Coding factory runs bounded tasks with offline replay
zlareb1 Nov 19, 2026
24b6e78
Qualify historical factory tasks and retain failed attempt accounting
zlareb1 Nov 19, 2026
e2b6df9
Persist cohort reservations before provider dispatch
zlareb1 Nov 19, 2026
cf32ac4
Retain request IDs in factory transport events
zlareb1 Nov 19, 2026
de4bbc3
Analyze factory pilot attempts by task and repeat
zlareb1 Nov 20, 2026
74ea1d5
Refuse candidate-controlled grades and supervise factory attempts
zlareb1 Nov 20, 2026
dbfdeb0
Grade historical API behavior outside candidate execution
zlareb1 Nov 20, 2026
6715218
Describe the implemented host oracle in the pilot protocol
zlareb1 Nov 20, 2026
26269f4
Compose the coding factory with current main
zlareb1 Nov 20, 2026
54056f0
Compose the factory with current test diagnostics
zlareb1 Nov 20, 2026
5f0517e
Halt cohorts after visible grader infrastructure failures
zlareb1 Nov 20, 2026
adae29c
Keep visible grading budgets scoped to one attempt
zlareb1 Nov 20, 2026
35a37bb
Compose the factory with current source contracts
zlareb1 Nov 20, 2026
5ecca85
Remove unreachable hidden grading from historical diagnostics
zlareb1 Nov 20, 2026
a45c648
Describe historical test runs as visible diagnostics
zlareb1 Nov 21, 2026
76a669a
Merge commit '9af0d68f50b0a4b4ea927f03076fe07d737d55c8' into feat/fla…
zlareb1 Nov 21, 2026
b08ef6f
Treat observation transport deadlines as invalid grading
zlareb1 Nov 21, 2026
5b1bac0
Merge commit '89d7b5ea085416482dc7c5728fa9cb47a3bee540' into feat/fla…
zlareb1 Nov 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,10 @@ jobs:
# The published benchmark harnesses' scoring and dataset readers.
- name: pytest bench harnesses
run: pytest bench -q -p randomly --randomly-seed="${{ github.run_id }}${{ github.run_attempt }}"
- name: Offline software factory contracts
run: |
python -m unittest discover -s examples/software_factory/tests -v
PYTHONPATH="$PWD/sdk/python:$PWD" python -m unittest discover -s examples/software_factory/client_contracts -v
# The generic ingest script's dry run goes through this checkout's CLI.
- name: generic integration sample
run: PYTHON=python bash integrations/generic/test.sh
Expand Down
119 changes: 119 additions & 0 deletions examples/software_factory/PILOT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# Proposed live pilot protocol

This protocol is a proposal. It authorizes no provider calls, ingestion, judge calls
or historical candidate execution. The offline replay results cannot qualify it.

## Question and arms

Measure whether application-owned exact context changes objective resolution,
total cost per successful resolution, and latency on pinned historical tasks.
This measures controlled context utility. It does not establish organic adoption.

Use three arms: baseline, orchestrated Loom, and natural agent-selected Loom.
Baseline receives no Loom backend, schemas, instructions, hooks or SDK setup.
The natural arm has an optional exact lookup tool and no minimum call count.
The orchestrated arm uses the intake, changed-diff and failure-context hooks.
All arms share the same agent tools, provider, tests and task budgets.

Candidate tasks are 4045 (name validation), 4575 (redirect customization feature),
and 4935 (nested subdomain routing), as pinned in `historical-candidates.json`.
The imported receipts describe diagnostic controls; the whole prior cohort is
incomplete. Candidate-written reports do not establish independent correctness.
Case 5063 is excluded because inspected controls are incomplete. Do not infer
qualification from an image tag or a prior summary.

## Qualification and freeze before approval

`HistoricalGrader` uses candidate-controlled pytest reports and remains diagnostic.
The factory and worker refuse it before client construction, and its qualification
CLI refuses it before container execution. `BehavioralGrader` supplies a separate
host oracle with authored public-behavior probes. Its assertions and final grades
stay outside candidate execution. A pilot must explicitly choose this narrower
coverage and freeze its code, seeds and independent control receipts; it cannot
claim the official pytest selectors were executed. Imported diagnostic controls
and filesystem probes do not qualify either oracle for a live protocol.

1. Freeze the implementation commit, all agent-visible inputs, tool schemas,
prompts, output limits, exact provider model id, API version and complete
dependency locks. Select a direct Bedrock Converse model supported by the
account and approve that exact id. Exclude GPT Luna, internal gateways and
Cohere. Do not infer capabilities from an alias. Send no temperature default.
2. Freeze a currently available dependency image by digest for each historical
base. Independently run visible-base pass, hidden base fail and reference
pass controls against the chosen oracle. Require every frozen probe to return
a valid observation; missing or malformed observations cannot pass. Official
pytest selectors remain diagnostic. Reference patches stay evaluator-private.
Imported controls and image digests are evidence to inspect, not permission
to reuse an invalid environment. No full local suites on the shared host.
3. Qualify a sandbox: one source checkout, no `.git`, alternates, future objects,
reference fixes, grader files, credential files, network or sibling mounts.
Candidate tests run without evaluator authority. Mount only immutable runtime
dependencies. Verify traversal, symlink, import, subprocess and egress attacks
fail. Keep assertions and final grades on the host, outside candidate mounts.
Apply only allowlisted source changes to each observation checkout; expose no
hidden expectations, grader program or reference patch.
4. Authorize existing immutable index namespaces for each repository/base using
reader credentials only. Verify full commit and repository identity before
and after reads. Zero restricted counts mean unknown. Do not grant access or
create an index. Missing scope, mutable watermarks or provider-dependent name
resolution blocks the affected arm. The supplied SDK adapter needs this
deployment-specific verifier, with every physical check in its receipt.
5. Freeze a price table and provider/retrieval billing evidence. Unknown cost is
unknown, including server embedding/rerank overhead. Reject a paid cohort if
a conservative whole-run reservation cannot be priced. Disable retries in
the model SDK and count all scope checks, network attempts and failures.
6. Show the exact frozen protocol and maximum reservation to the owner and obtain
new approval. No broad earlier authorization starts this cohort.

## Counts and budgets proposed for approval

Pair by task and repeat. Use 3 tasks × 3 repeats × 3 arms = 27 primary attempts.
Add baseline A/A using 3 tasks × 2 identical baseline copies = 6 attempts, for
33 total. Shuffle arm order within each task/repeat using a published seed and
persist the complete schedule before running. Keep all failures and nonuse;
no best-of selection, retries as replacement attempts, or post-hoc task removal.

Per attempt: at most 12 provider turns, 30 agent tools, 6 logical context reads,
2,048 output tokens per turn, 64,000 serialized transcript characters per turn,
12,000 context characters total, 180 seconds wall time in the qualified runner,
and 3 candidate test invocations of at most 30 seconds each. The authored demo
uses its own defaults; freeze the historical runner's distinct budgets first.
The SDK has no automatic retries: at most 18 physical Loom requests per attempt
if every logical read needs two one-request scope checks. If a verifier needs
more, revise the exact protocol before approval. Max provider requests: 396;
max output tokens: 811,008; max logical Loom requests: 108; max physical Loom
requests under that verifier: 324. The 18 nonbaseline attempts each allow six
logical reads; the 15 baseline attempts have zero retrievals. Price input tokens using the
frozen tokenizer and full transcript, including repeated tool results. Propose a
$25 total reservation only if that worst-case computation fits; otherwise reduce
the design and show the revised plan. No spend is authorized by this number.
No judge is needed: objective graders decide patch correctness.

## Analysis and boundaries

Report objective resolutions, regressions, invalid runs, all costs including
failed attempts, cost per successful resolution, latency, host calls, selected
calls, physical requests/retries, provider input/output/cache tokens, unknown
usage and retrieval overhead. Join client request ids to server billing evidence.
Report costs as unknown when the join or pricing is incomplete.

Use paired task-cluster bootstrap confidence intervals with a frozen seed and
resample tasks, retaining paired arms and repeats inside each cluster. Report
A/A variation separately. Three task clusters are a workflow pilot, not grounds
for significance or percentage-gain claims. Expand independent tasks before any
adoption, savings or general quality conclusion.

Each attempt starts cold. There is no cross-issue memory write path or warm
learning arm. A future chronological study needs a separate freeze and may save
only independently verified facts. Workflow determinism means event routing;
model trajectories and patches remain stochastic.

## Authored oracle proposal

The optional host behavioral oracle covers the three public bug surfaces with
parameterized API observations. Assertions and final grades remain outside the
candidate container. It requires fresh base/reference controls and independent
review. Its narrower authored coverage is not the official pytest selector
coverage described by the diagnostic artifacts. A pilot using this oracle must
freeze that change explicitly, including probe seeds and grader/receipt hashes,
before approval; no existing draft or diagnostic control authorizes substitution.
Loading
Loading