Partial preview success is treated as full fix verification

[matthewp]

When a reporter says a preview fixed only part of an issue, Factory can interpret that as full confirmation and open a PR.

Example: withastro/compiler-rs#139 (comment)

The reporter said :has() was fixed but :is() was still broken. Factory nevertheless marked the fix verified and opened withastro/compiler-rs#142.

Expected behavior:

• A partial fix is not treated as verified.
• The unresolved behavior is sent back through triage without requiring another comment from the reporter.
• Factory tells the reporter that it is retrying rather than silently leaving the issue waiting.

[astro-factory]

• Reproduced: Yes
• Exploration: Yes
• Unit Test: Yes — /triage/gh-19/fix-verifier-partial-repro.test.ts

The bug is in Factory's fix-verification pipeline. The FixVerifier agent only knows three verdict states (confirmed, rejected, inconclusive) and its prompt never mentions mixed/partial feedback. When a reporter says something like ":has() is fixed but :is() is still broken," the model is guided toward the positive-confirmation bucket, so Factory opens a PR even though part of the issue is unresolved.

I wasn't able to find a fix, but I identified some areas that may be relevant. The cleanest change is to add a partial verdict status to src/triage/contracts.ts, teach the FixVerifier prompt in src/triage/agents/fix-verifier.ts to classify mixed comments as partial, and add a branch in src/triage/workflow.ts's verifyFix that skips the PR, posts a retry comment, and routes the issue back to needsTriage for another triage run. The code paths and line numbers are in the full report below.

Full Triage Report

Reproduction Report: Issue #19

Issue Summary

• Title: Partial preview success is treated as full fix verification
• Repository: withastro/factory
• Branch under test: factory/fix-19

When a reporter says a preview release fixed only part of an issue, Factory can interpret that as a full confirmation and open a pull request. The canonical example is a comment that says:

":has() is fixed but :is() is still broken."

Factory then classified the fix as verified and opened a PR, even though the reporter explicitly stated the other half of the bug still reproduced.

Expected behavior

1. A partial fix is not treated as verified.
2. The unresolved behavior is sent back through triage without requiring another comment from the reporter.
3. Factory tells the reporter that it is retrying rather than silently leaving the issue waiting.

Environment

• Factory checkout: /repo
• Triage working directory: /triage/gh-19
• Node.js: >=22.19.0
• Package manager: pnpm
• Test runner: vitest v4.1.10
• Relevant source files:
  • /repo/src/triage/agents/fix-verifier.ts
  • /repo/src/triage/contracts.ts
  • /repo/src/triage/workflow.ts
  • /repo/src/triage/labels.ts
  • /repo/src/triage/fsm.ts

Reproduction Steps

1. Reviewed the fix verification agent prompt and verdict schema.
2. Created a failing reproduction test at /triage/gh-19/fix-verifier-partial-repro.test.ts that exercises the fixVerdictSchema with a partial status.
3. Ran the reproduction test using the project's vitest setup (forced via a sibling config file in triageDir).

Commands executed

mkdir -p /triage/gh-19
pnpm --dir /repo exec vitest run --config /triage/gh-19/vitest.config.ts

Test details

import { expect, test } from 'vitest';
import * as v from 'valibot';
import { fixVerdictSchema } from '../repo/src/triage/contracts.ts';

test('FixVerdict schema should accept a partial-fix verdict (current bug: it does not)', () => {
	const partial = {
		status: 'partial',
		reasoning: 'The preview fixed :has() but :is() is still broken.',
		pr: null,
	};
	expect(() => v.parse(fixVerdictSchema, partial)).not.toThrow();
});

Results

 FAIL  ../triage/gh-19/fix-verifier-partial-repro.test.ts > FixVerdict schema should accept a partial-fix verdict (current bug: it does not)
AssertionError: expected [Function] to not throw an error but 'ValiError: Invalid type: Expected ("confirmed" | "rejected" | "inconclusive") but received "partial"' was thrown

The schema currently only allows:

• status: 'confirmed' → opens a PR and marks the issue fix verified
• status: 'rejected' → marks the issue fix rejected
• status: 'inconclusive' → leaves the issue in fix pending with no further action

There is no partial status, so the data model cannot express "part works, part is still broken."

Code Observations

FixVerifier prompt lacks partial-fix guidance

/repo/src/triage/agents/fix-verifier.ts instructs the model to classify only three buckets:

• positive confirmation → status confirmed
• negative confirmation → status rejected
• neither → status inconclusive

It never tells the model that a comment can be mixed (one part fixed, another still broken), nor does the schema provide a place to record which part is unresolved.

verifyFix only branches on three outcomes

/repo/src/triage/workflow.ts calls FixVerifier and then acts on the verdict with this logic:

• inconclusive → returns fix-inconclusive, leaving the issue waiting.
• rejected → swaps label to fix-rejected.
• confirmed → opens or finds a PR and swaps label to fix-verified.

There is no path that converts a partial success into a new triage run or tells the reporter that Factory is retrying.

State labels support retry, but the workflow does not use them for partial fixes

/repo/src/triage/labels.ts already defines needsTriage (and fixRejected is retriageable), so the state machine could cycle an issue back into triage. /repo/src/triage/fsm.ts shows the issue would route back to the full pipeline if its label were needsTriage, but verifyFix has no branch to make that transition.

Root-Cause Theory

The bug is a gap in the classification contract and both the agent and workflow layers:

1. Agent layer: The FixVerifier prompt/schema afford only binary confirmed/rejected/inconclusive outcomes. A comment that contains positive language for one symptom but negative language for another is likely to be labelled confirmed, because the prompt asks "is this a positive confirmation?" without caveats for partial success.
2. Workflow layer: Even if the model realized the comment was mixed, it has no partial status to return and no ingest path to start a new reproduction/fix attempt on the still-broken behavior.

Reproducibility Verdict

Reproduced. The codebase cannot represent a partial fix in the fixVerdictSchema, and neither the FixVerifier prompt nor the verifyFix workflow handles partial preview success. A reporter comment like the example would therefore be classified as confirmed and a PR would be opened.

---

Diagnosis Report

Confidence: high

Root Cause

The bug spans the data model, the classification agent, and the workflow handler.

1. Data model cannot express partial success

File: /repo/src/triage/contracts.ts (around line 82)

export const fixVerdictSchema = v.object({
	status: v.picklist(['confirmed', 'rejected', 'inconclusive']),
	reasoning: v.pipe(v.string(), v.maxLength(2_000)),
	pr: v.nullable(
		v.object({
			title: v.pipe(v.string(), v.trim(), v.minLength(1), v.maxLength(200)),
			body: v.pipe(v.string(), v.trim(), v.minLength(1), v.maxLength(20_000)),
		}),
	),
});

The verdict schema only permits three states. There is no partial status and no field to capture which symptoms are still broken, so even a perfectly calibrated LLM has no structured way to report a partial result.

2. FixVerifier prompt never mentions partial/mixed confirmation

File: /repo/src/triage/agents/fix-verifier.ts (lines 88–120)

The prompt asks the model to decide between:

• positive confirmation → confirmed
• negative confirmation → rejected
• neither → inconclusive

Diagnostic run of the prompt text confirmed none of the keywords partial, mixed, or only part appear:

[DIAGNOSE] prompt keywords — partial: false
[DIAGNOSE] prompt keywords — mixed: false
[DIAGNOSE] prompt keywords — only part: false

A comment such as ":has() is fixed but :is() is still broken" contains positive language and satisfies the "positive confirmation" heuristic in the prompt. Without an explicit instruction that mixed results must not be treated as confirmed, the model is likely to return status: 'confirmed' and draft a PR.

3. verifyFix branches exhaustively only over the three existing statuses

File: /repo/src/triage/workflow.ts (verifyFix method, lines 854–960)

After reading the verdict:

• inconclusive → returns fix-inconclusive (lines 914–916)
• rejected → swaps label to fix-rejected (lines 918–931)
• otherwise treats it as confirmed → opens a PR and swaps label to fix-verified (lines 936–960)

There is no branch for a partial result, so even if the agent correctly detected a mixed comment and returned something other than confirmed, the schema would reject it and the workflow would fail.

Diagnosis Detail

The root cause is therefore not a single logic bug, but a missing capability: the system was designed for binary fix-verification feedback. To fix it properly, the contract needs a partial status plus a way to communicate the unresolved behavior, the agent prompt needs to classify mixed comments as partial, and the workflow needs to route a partial result back into the triage pipeline with a retry comment to the reporter.

Affected Files and Lines

• /repo/src/triage/contracts.ts — fixVerdictSchema (line ~82)
• /repo/src/triage/agents/fix-verifier.ts — classification prompt and submit_fix_verification tool (lines 88–120, 64–76)
• /repo/src/triage/workflow.ts — verifyFix method verdict branching (lines 854–960)
• /repo/src/triage/labels.ts — needsTriage label exists and is retriageable (supports the retry path once implemented)

Suggested Fix Approach

1. Extend FixVerdict with a partial status (and possibly a remainingIssue field describing what is still broken).
2. Update the FixVerifier prompt to recognize mixed/partial comments and instruct the model to return partial when the reporter says some aspect still fails.
3. Update verifyFix in workflow.ts to handle partial:
   • Do not open a PR.
   • Post a comment telling the reporter Factory is retrying the unresolved part.
   • Swap the issue label back to needsTriage so the next delivery re-runs the full triage pipeline on the remaining behavior.

This satisfies all three expected behaviors listed in the issue.

---

Verification Report

Claim

The reporter claims that when a user says a preview release fixed only part of the issue, Factory should:

1. Not treat the fix as fully verified,
2. Send the unresolved behavior back through triage without waiting for another comment,
3. Tell the reporter that Factory is retrying.

Current Behavior

Factory's fix-verification path supports only three verdicts:

• confirmed → opens a PR and applies triage: fix verified
• rejected → applies triage: fix rejected
• inconclusive → leaves the issue in triage: fix pending

A comment containing both positive and negative feedback has no structured representation, so it is treated as the binary cases above.

Evidence Reviewed

• README.md (lines 69–71) documents the FixVerifier flow as only confirmed → open PR + fix verified; rejected → fix rejected. It does not mention partial confirmation as an intentional supported case.
• Source code (src/triage/agents/fix-verifier.ts) uses a prompt that asks only about positive/negative confirmation, with no guidance for mixed/partial comments.
• Data model (src/triage/contracts.ts) hard-codes the verdict status picklist to ['confirmed', 'rejected', 'inconclusive'].
• Git history: the FixVerifier agent and three-status verdict were introduced together in the initial scaffold commit (746f318). No later commit added partial-fix handling, and the preview-release feature (fb72a0d) did not extend the verdict model.
• Local code search: no occurrence of partial, mixed, or only part was found in the triage state machine or agent code.

Verdict

bug — with high confidence.

The three-status design appears to be an oversight rather than an explicit decision to collapse partial success into full verification. The prompt's positive-confirmation examples would naturally match a comment like ":has() is fixed but :is() is still broken," leading the model to return confirmed and open a PR. The workflow then has no path to treat the still-broken aspect differently. This aligns with the real incident cited in the issue where a PR was opened after a partial confirmation.

This report was made by an LLM. The analysis may be wrong, and the potential fix might not work, but is intended as a starting point for exploring the issue.

[astro-factory]

Triage complete

• Prepare the workspace: ready
• Install dependencies: complete
• Reproduce the issue: confirmed
• Diagnose the cause: high confidence
• Verify the diagnosis: bug
• Attempt a fix: no verified fix
• Publish the result: set triage: unable to fix

---

• Reproduced: Yes
• Exploration: Yes
• Unit Test: Yes — /repo/tests/fix-verifier-partial-repro.test.ts
• Fix: I wasn't able to find a fix, but I identified some areas that may be relevant.

The bug is in Factory's fix-verification pipeline. The FixVerifier agent only knows three verdict states (confirmed, rejected, inconclusive) and its prompt never mentions mixed/partial feedback. When a reporter says something like ":has() is fixed but :is() is still broken," the model is guided toward the positive-confirmation bucket, so Factory opens a PR even though part of the issue is unresolved.

I wasn't able to find a fix, but I identified some areas that may be relevant. The cleanest change is to add a partial verdict status to src/triage/contracts.ts, teach the FixVerifier prompt in src/triage/agents/fix-verifier.ts to classify mixed comments as partial, and add a branch in src/triage/workflow.ts's verifyFix that skips the PR, posts a retry comment, and routes the issue back to needsTriage for another triage run. The code paths and line numbers are in the full report below.

Full Triage Report

Reproduction Report: Issue #19

Issue Summary

• Title: Partial preview success is treated as full fix verification
• Repository: withastro/factory
• Branch under test: factory/fix-19

When a reporter says a preview release fixed only part of an issue, Factory can interpret that as a full confirmation and open a pull request. The canonical example is a comment that says:

":has() is fixed but :is() is still broken."

Factory then classified the fix as verified and opened a PR, even though the reporter explicitly stated the other half of the bug still reproduced.

Expected behavior

1. A partial fix is not treated as verified.
2. The unresolved behavior is sent back through triage without requiring another comment from the reporter.
3. Factory tells the reporter that it is retrying rather than silently leaving the issue waiting.

Environment

• Factory checkout: /repo
• Triage working directory: /triage/gh-19
• Node.js: >=22.19.0
• Package manager: pnpm
• Test runner: vitest v4.1.10
• Relevant source files:
  • /repo/src/triage/agents/fix-verifier.ts
  • /repo/src/triage/contracts.ts
  • /repo/src/triage/workflow.ts
  • /repo/src/triage/labels.ts
  • /repo/src/triage/fsm.ts

Reproduction Steps

1. Reviewed the fix verification agent prompt and verdict schema.
2. Created a failing reproduction test at /repo/tests/fix-verifier-partial-repro.test.ts that exercises the fixVerdictSchema with a partial status.
3. Ran the reproduction test using the project's vitest setup.

Commands executed

pnpm --dir /repo exec vitest run --config /repo/vitest.config.ts tests/fix-verifier-partial-repro.test.ts

Test details

import { expect, test } from 'vitest';
import * as v from 'valibot';
import { fixVerdictSchema } from '../src/triage/contracts.ts';

test('FixVerdict schema should accept a partial-fix verdict (current bug: it does not)', () => {
	const partial = {
		status: 'partial',
		reasoning: 'The preview fixed :has() but :is() is still broken.',
		pr: null,
	};
	expect(() => v.parse(fixVerdictSchema, partial)).not.toThrow();
});

Results

FAIL  tests/fix-verifier-partial-repro.test.ts > FixVerdict schema should accept a partial-fix verdict (current bug: it does not)
AssertionError: expected [Function] to not throw an error but 'ValiError: Invalid type: Expected ("confirmed" | "rejected" | "inconclusive") but received "partial"' was thrown

The schema currently only allows:

• status: 'confirmed' → opens a PR and marks the issue fix verified
• status: 'rejected' → marks the issue fix rejected
• status: 'inconclusive' → leaves the issue in fix pending with no further action

There is no partial status, so the data model cannot express "part works, part still reproduces." The FixVerifier prompt in /repo/src/triage/agents/fix-verifier.ts also has no concept of mixed/partial feedback; it only asks the model to classify a comment as positive confirmation, negative confirmation, or inconclusive. This nudges it toward the positive bucket when one part of the reported issue is confirmed fixed.

Additional observations

• /repo/src/triage/workflow.ts verifyFix branches on verdict.status and maps confirmed to opening/finding a PR, rejected to fix-rejected, and inconclusive to fix-inconclusive.
• There is no label resolution, routing, or retry behavior for a partial fix.
• The expected fix path is likely:
  1. Add partial to fixVerdictSchema in contracts.ts (with pr: null).
  2. Update FixVerifier prompt to classify mixed feedback as partial and to ask for a follow-up question about the still-broken behavior.
  3. Handle partial in workflow.ts verifyFix by swapping fix pending back to needsTriage and posting a comment explaining the retry.

Reproduced?

Yes.

---

Diagnosis Report: Issue #19

Root Cause

The fix-verification pipeline has no concept of a "partial" confirmation. Three related gaps combine to produce the reported behavior:

1. The data model rejects the partial status.

   • File: /repo/src/triage/contracts.ts
   • Line 85: status: v.picklist(['confirmed', 'rejected', 'inconclusive'])
   • The fixVerdictSchema defines exactly three verdicts. Any attempt to submit status: 'partial' fails validation before it can reach downstream logic.

2. The verifier prompt only asks the model to choose positive, negative, or inconclusive.

   • File: /repo/src/triage/agents/fix-verifier.ts
   • Lines 82–115: task instructions list positive confirmation examples, negative confirmation examples, and comments that are neither. There is no category for "part fixed, part still broken." Because the positive bucket is partially satisfied by the comment, the model is guided toward confirmed.

3. verifyFix only branches on the three existing statuses.

   • File: /repo/src/triage/workflow.ts
   • Method verifyFix begins at line 1301.
   • inconclusive returns fix-inconclusive (lines 1339–1341).
   • rejected swaps the label to fixRejected and returns fix-rejected (lines 1343–1355).
   • The remaining code assumes confirmed: it opens or finds a pull request, swaps fixPending to fixVerified, and posts a verification comment (lines 1357–1385). There is no branch for a partial result, so once the model emits confirmed the PR is opened unconditionally.

Why this causes the reported bug

A comment such as:

":has() is fixed but :is() is still broken."

contains both positive and negative signal. Without a dedicated "partial" category, the model treats the positive part as sufficient for confirmed. The schema accepts confirmed with pr content, so verifyFix proceeds to open a PR and mark the issue fixVerified, even though the reporter explicitly said one of the broken behaviors still reproduces.

Affected file paths and line numbers

File	Relevant lines	Why
/repo/src/triage/contracts.ts	84–95	fixVerdictSchema enum lacks partial.
/repo/src/triage/agents/fix-verifier.ts	82–115, 30–35	Prompt doesn't classify mixed comments; output schema is the verdict.
/repo/src/triage/workflow.ts	1301–1385	verifyFix has no partial branch.
/repo/src/triage/labels.ts	87–99	retriageableLabels and terminalLabels define which labels can re-enter triage; fixPending is not retriagable by the generic retriage logic.

Suggested fix approach

1. Add partial to fixVerdictSchema.status in contracts.ts and require pr: null for that status.
2. Update the FixVerifier prompt to recognize mixed/partial feedback, with clear examples, and to emit partial when one part is fixed and another still reproduces.
3. In workflow.ts verifyFix, handle verdict.status === 'partial' by:
   • Skipping the PR creation.
   • Swapping the issue label from fixPending to needsTriage.
   • Posting a comment telling the reporter the remaining behavior is being retriaged, rather than leaving the issue silent.
4. Add/update tests for schema parsing and for the verifyFix branch.

Confidence

High — the reproduction test, schema definition, prompt text, and workflow branching all align and directly explain the reported behavior.

---

Verification Report: Issue #19

Reporter's Claim

When a reporter says a preview release fixed only part of their issue, Factory should:

1. Not treat the fix as fully verified.
2. Send the unresolved behavior back through triage without waiting for another comment.
3. Tell the reporter that it is retrying, rather than leaving the issue silent.

Evidence Reviewed

• Source code intent signals:
  • /repo/src/triage/contracts.ts line 85: the verdict enum only permits confirmed, rejected, inconclusive. There is no partial value.
  • /repo/src/triage/agents/fix-verifier.ts lines 82–115: the prompt instructs the model to classify comments as positive confirmation, negative confirmation, or neither. It gives no category for mixed feedback.
  • /repo/src/triage/workflow.ts lines 1339–1385: verifyFix only branches on the three statuses. The confirmed path unconditionally opens/finds a PR and swaps the label to fixVerified.
• README (README.md): describes the FixVerifier loop as confirmed → open fix PR + fix verified; rejected → fix rejected. It does not mention partial feedback or indicate that partial results are intentionally unsupported.
• Test coverage: No existing test asserts behavior for partial or mixed fix feedback.
• Git history: The three verdict states were introduced in the initial triage scaffold without documented rationale for excluding mixed feedback.

Assessment

The current behavior appears to be an unintended consequence of a simplified feedback model (positive / negative / inconclusive) rather than a deliberate design choice to ignore partial confirmations. Nothing in the code, docs, or tests indicates that the developers considered and rejected handling of mixed feedback. Because the prompt biases the model toward the positive bucket when only one part of the issue is confirmed fixed, Factory can incorrectly open a PR for a fix that the reporter explicitly says is incomplete.

Verdict

bug — the behavior is incorrect and harmful; it is not documented or clearly intended.

Confidence

high — the schema, prompt, workflow, and README all consistently show that only binary confirmation/rejection is modeled, while the real-world scenario described in the issue is unhandled and leads to a false positive.

This report was made by an LLM. The analysis may be wrong, and the potential fix might not work, but is intended as a starting point for exploring the issue.