Repository navigation
Add bounded cover-suppressed evidence to Sites critique #1289
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sidneyswift
wants to merge
7
commits into
main
Choose a base branch
from
codex/sites-cover-suppressed-review-2094
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
2457033
Add bounded cover-suppressed evidence to Sites visual review
sidneyswift 906558d
Use source-bound browser masks immune to author CSS
sidneyswift faee6c5
Reject cover captures with mismatched browser mask targets
sidneyswift 313606a
Reject unsupported cover visibility and repair verdicts
sidneyswift 2cde154
Bound dynamic capture and gate structured cover-only issues
sidneyswift 3ddb33d
Canonicalize cover evidence hashes and verify runner inputs
sidneyswift 2f28d70
Verify schema-valid ordinary and mixed review issues
sidneyswift File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| import { expect, it, vi } from "vitest"; | ||
| import { collectCoverHiddenEvidence } from "../production/collectCoverHiddenEvidence"; | ||
|
|
||
| it("accepts a bounded report with eight long signed artwork URLs in each viewport", async () => { | ||
| const reports = ["mobile", "desktop"].map(name => ({ | ||
| name, | ||
| status: "captured", | ||
| reason: "Exact source", | ||
| masked: Array.from({ length: 8 }, (_, index) => ({ | ||
| index, | ||
| src: "https://assets.example/cover?signature=" + "x".repeat(3900), | ||
| box: { x: 0, y: 0, width: 100, height: 100 }, | ||
| })), | ||
| })); | ||
| const bytes = Buffer.from(JSON.stringify(reports)); | ||
| expect(bytes.length).toBeGreaterThan(30000); | ||
| const result = await collectCoverHiddenEvidence({ | ||
| runCommand: vi.fn(async () => ({ exitCode: 0 })), | ||
| readFileToBuffer: vi.fn(async ({ path }) => | ||
| path === "cover-review-report.json" ? bytes : Buffer.from("png"), | ||
| ), | ||
| } as unknown as Parameters<typeof collectCoverHiddenEvidence>[0]); | ||
| expect(result.reports).toEqual(reports); | ||
| expect(result.images).toHaveLength(2); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,335 @@ | ||
| import { beforeEach, expect, it, vi } from "vitest"; | ||
| import { reviewStep } from "@/app/workflows/sites/reviewStep"; | ||
| import type { SiteSnapshot } from "../schema"; | ||
| import type { CreativeDirection } from "../production/schema"; | ||
| const m = vi.hoisted(() => ({ generate: vi.fn(), read: vi.fn(), write: vi.fn(), create: vi.fn() })); | ||
| vi.mock("../production/generateProductionObject", () => ({ generateProductionObject: m.generate })); | ||
| vi.mock("../production/compileJourney", () => ({ | ||
| compileJourney: vi.fn(async () => ({ steps: [] })), | ||
| })); | ||
| vi.mock("../production/reviewOpeningSequence", () => ({ | ||
| reviewOpeningSequence: vi.fn(async () => ({ | ||
| verdict: "pass", | ||
| issues: [], | ||
| summary: "Clear", | ||
| observations: [], | ||
| })), | ||
| })); | ||
| vi.mock("@vercel/sandbox", () => ({ Sandbox: { create: m.create } })); | ||
| const url = "https://assets.example/cover.png"; | ||
| const snapshot = { | ||
| assets: [{ url, type: "image" }], | ||
| design: { experience: { html: '<img src="' + url + '">', css: "", javascript: "" } }, | ||
| brandWorld: { specification: { direction: { coverHiddenTest: "angular framing" } } }, | ||
| } as unknown as SiteSnapshot; | ||
| const reports = ["mobile", "desktop"].map(name => ({ | ||
| name, | ||
| journeyPassed: true, | ||
| errors: [], | ||
| overflow: false, | ||
| coverHidden: { | ||
| status: "captured", | ||
| reason: "Exact inline cover mask", | ||
| masked: [{ index: 0, src: url, box: { x: 0, y: 0, width: 100, height: 100 } }], | ||
| }, | ||
| })); | ||
| beforeEach(() => { | ||
| vi.resetAllMocks(); | ||
| m.create.mockResolvedValue({ | ||
| runCommand: vi.fn(async () => ({ exitCode: 0 })), | ||
| extendTimeout: vi.fn(), | ||
| updateNetworkPolicy: vi.fn(), | ||
| writeFiles: m.write, | ||
| readFileToBuffer: m.read, | ||
| stop: vi.fn(), | ||
| }); | ||
| m.read.mockImplementation(async ({ path }) => | ||
| path === "review.json" | ||
| ? Buffer.from(JSON.stringify(reports)) | ||
| : path === "cover-review-report.json" | ||
| ? Buffer.from(JSON.stringify(reports.map(r => ({ name: r.name, ...r.coverHidden })))) | ||
| : /^(mobile|desktop)(-active|-first-action|-cover-hidden)?\.png$/.test(path) | ||
| ? Buffer.from(path) | ||
| : null, | ||
| ); | ||
| m.generate.mockResolvedValue({ | ||
| verdict: "pass", | ||
| issues: [], | ||
| summary: "Distinctive framing", | ||
| coverHiddenCritique: { | ||
| assessment: "expresses-world", | ||
| reasons: ["Angular framing remains"], | ||
| improvements: [], | ||
| }, | ||
| }); | ||
| }); | ||
| it("connects actual workflow review through rendering and preserves normal/opening image order without another critique call", async () => { | ||
| const result = await reviewStep( | ||
| snapshot, | ||
| { contract: {} } as CreativeDirection, | ||
| "account", | ||
| "site", | ||
| undefined, | ||
| { baseRevision: 7, releaseArtwork: url }, | ||
| ); | ||
| expect(m.generate).toHaveBeenCalledOnce(); | ||
| const args = m.generate.mock.calls[0]; | ||
| expect(args[3].map((x: string) => Buffer.from(x.split(",")[1], "base64").toString())).toEqual([ | ||
| "mobile.png", | ||
| "mobile-active.png", | ||
| "desktop.png", | ||
| "desktop-active.png", | ||
| "mobile-first-action.png", | ||
| "desktop-first-action.png", | ||
| "mobile-cover-hidden.png", | ||
| "desktop-cover-hidden.png", | ||
| ]); | ||
| expect(args[2].coverHidden.viewports.map((x: { imageIndex: number }) => x.imageIndex)).toEqual([ | ||
| 6, 7, | ||
| ]); | ||
| expect(result.verification?.coverHidden).toMatchObject({ | ||
| baseRevision: 7, | ||
| critique: { assessment: "expresses-world" }, | ||
| viewports: [{ status: "captured" }, { status: "captured" }], | ||
| }); | ||
| expect(result.verification?.coverHidden?.snapshotSha256).toMatch(/^[a-f0-9]{64}$/); | ||
| expect(result.verification?.humanApproval).toBe("not-recorded"); | ||
| const files = m.write.mock.calls[0][0]; | ||
| expect( | ||
| JSON.parse( | ||
| files.find((f: { path: string }) => f.path === "cover-review.json").content.toString(), | ||
| ).artworkUrl, | ||
| ).toBe(url); | ||
| }); | ||
| it("records missing capture as unverified even if model claims the cover-hidden criterion passed", async () => { | ||
| const read = m.read.getMockImplementation()!; | ||
| m.read.mockImplementation(async arg => | ||
| arg.path === "desktop-cover-hidden.png" ? null : read(arg), | ||
| ); | ||
| const result = await reviewStep( | ||
| snapshot, | ||
| { contract: {} } as CreativeDirection, | ||
| "account", | ||
| "site", | ||
| undefined, | ||
| { baseRevision: 7, releaseArtwork: url }, | ||
| ); | ||
| expect(result.verification?.coverHidden?.critique.assessment).toBe("unverified"); | ||
| expect(result.verification?.coverHidden?.viewports[1].status).toBe("unverified"); | ||
| expect(result.verification?.viewports.every(v => v.journeyPassed)).toBe(true); | ||
| }); | ||
| it("keeps subjective repairs distinct from passing objective journey checks", async () => { | ||
| m.generate.mockResolvedValue({ | ||
| verdict: "pass", | ||
| issues: [], | ||
| summary: "Journey worked", | ||
| coverHiddenCritique: { | ||
| assessment: "needs-improvement", | ||
| reasons: ["Only the cover supplies the angular language"], | ||
| improvements: ["Use the saved angular framing in the controls"], | ||
| }, | ||
| }); | ||
| const result = await reviewStep( | ||
| snapshot, | ||
| { contract: {} } as CreativeDirection, | ||
| "account", | ||
| "site", | ||
| undefined, | ||
| { baseRevision: 7, releaseArtwork: url }, | ||
| ); | ||
| expect(result.verdict).toBe("revise"); | ||
| expect(result.issues).toContainEqual( | ||
| expect.objectContaining({ | ||
| severity: "visual", | ||
| module: "implementation", | ||
| detail: "Only the cover supplies the angular language", | ||
| }), | ||
| ); | ||
| expect(result.verification?.viewports.every(v => v.journeyPassed)).toBe(true); | ||
| expect(m.generate).toHaveBeenCalledOnce(); | ||
| }); | ||
| it.each(["malformed", "capture-failure"])( | ||
| "retains normal journey evidence after supplemental %s", | ||
| async failure => { | ||
| const read = m.read.getMockImplementation()!; | ||
| m.read.mockImplementation(async arg => | ||
| arg.path === "cover-review-report.json" | ||
| ? Buffer.from(failure === "malformed" ? "{}" : "[]") | ||
| : read(arg), | ||
| ); | ||
| const result = await reviewStep( | ||
| snapshot, | ||
| { contract: {} } as CreativeDirection, | ||
| "account", | ||
| "site", | ||
| undefined, | ||
| { baseRevision: 7, releaseArtwork: url }, | ||
| ); | ||
| expect(result.verification?.coverHidden?.critique.assessment).toBe("unverified"); | ||
| expect(result.verification?.viewports.every(v => v.journeyPassed)).toBe(true); | ||
| expect(m.generate.mock.calls[0][3]).toHaveLength(6); | ||
| }, | ||
| ); | ||
| it.each(["pass", "revise"])( | ||
| "does not promote unsupported negative cover critique into a repair from %s", | ||
| async verdict => { | ||
| const read = m.read.getMockImplementation()!; | ||
| m.read.mockImplementation(async arg => | ||
| arg.path === "desktop-cover-hidden.png" ? null : read(arg), | ||
| ); | ||
| m.generate.mockResolvedValue({ | ||
| verdict, | ||
| issues: [], | ||
| summary: "Journey works", | ||
| coverHiddenCritique: { | ||
| assessment: "needs-improvement", | ||
| reasons: ["Guessed defect"], | ||
| improvements: ["Change controls"], | ||
| }, | ||
| }); | ||
| const result = await reviewStep( | ||
| snapshot, | ||
| { contract: {} } as CreativeDirection, | ||
| "account", | ||
| "site", | ||
| undefined, | ||
| { baseRevision: 7, releaseArtwork: url }, | ||
| ); | ||
| expect(result.verification?.coverHidden?.critique.assessment).toBe("unverified"); | ||
| expect(result.verdict).toBe("pass"); | ||
| expect(result.issues).toEqual([]); | ||
| }, | ||
| ); | ||
|
|
||
| it("preserves ordinary defects when supplemental cover evidence is missing", async () => { | ||
| const read = m.read.getMockImplementation()!; | ||
| m.read.mockImplementation(async arg => | ||
| arg.path === "desktop-cover-hidden.png" ? null : read(arg), | ||
| ); | ||
| const issue = { | ||
|
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
|
||
| severity: "visual", | ||
| module: "implementation", | ||
| detail: "Normal view text overlaps controls", | ||
| fix: "Add spacing", | ||
| }; | ||
| m.generate.mockResolvedValue({ | ||
| verdict: "revise", | ||
| issues: [{ ...issue, evidence: "ordinary" }], | ||
| summary: "Ordinary defect", | ||
| coverHiddenCritique: { | ||
| assessment: "needs-improvement", | ||
| reasons: ["Unsupported"], | ||
| improvements: [], | ||
| }, | ||
| }); | ||
| const result = await reviewStep( | ||
| snapshot, | ||
| { contract: {} } as CreativeDirection, | ||
| "account", | ||
| "site", | ||
| undefined, | ||
| { baseRevision: 7, releaseArtwork: url }, | ||
| ); | ||
| expect(result.verification?.coverHidden?.critique.assessment).toBe("unverified"); | ||
| expect(result.verdict).toBe("revise"); | ||
| expect(result.issues).toEqual([issue]); | ||
| }); | ||
|
|
||
| it("retains runtime failure admission when discarding an unsupported cover-only verdict", async () => { | ||
| const read = m.read.getMockImplementation()!; | ||
| m.read.mockImplementation(async arg => { | ||
| if (arg.path === "desktop-cover-hidden.png") return null; | ||
| if (arg.path === "review.json") | ||
| return Buffer.from( | ||
| JSON.stringify( | ||
| reports.map(r => ({ ...r, journeyPassed: false, errors: ["Script error"] })), | ||
| ), | ||
| ); | ||
| return read(arg); | ||
| }); | ||
| m.generate.mockResolvedValue({ | ||
| verdict: "revise", | ||
| issues: [], | ||
| summary: "Cover-only opinion", | ||
| coverHiddenCritique: { | ||
| assessment: "needs-improvement", | ||
| reasons: ["Unsupported"], | ||
| improvements: [], | ||
| }, | ||
| }); | ||
| const result = await reviewStep( | ||
| snapshot, | ||
| { contract: {} } as CreativeDirection, | ||
| "account", | ||
| "site", | ||
| undefined, | ||
| { baseRevision: 7, releaseArtwork: url }, | ||
| ); | ||
| expect(result.verification?.coverHidden?.critique.assessment).toBe("unverified"); | ||
| expect(result.verdict).toBe("revise"); | ||
| expect(result.issues).toEqual( | ||
| expect.arrayContaining([ | ||
| expect.objectContaining({ | ||
| severity: "blocking", | ||
| detail: expect.stringContaining("Script error"), | ||
| }), | ||
| ]), | ||
| ); | ||
| }); | ||
|
|
||
| it.each([false, true])( | ||
| "discards structured cover-only issues while retaining ordinary issue=%s", | ||
| async hasOrdinary => { | ||
| const read = m.read.getMockImplementation()!; | ||
| m.read.mockImplementation(async arg => | ||
| arg.path === "desktop-cover-hidden.png" ? null : read(arg), | ||
| ); | ||
| const coverIssue = { | ||
| severity: "visual", | ||
| module: "implementation", | ||
| detail: "Cover hidden opinion", | ||
| fix: "Change frame", | ||
| evidence: "cover-hidden", | ||
| }; | ||
| m.generate.mockResolvedValue({ | ||
| verdict: "revise", | ||
| issues: [ | ||
| coverIssue, | ||
| ...(hasOrdinary | ||
| ? [ | ||
| { | ||
| severity: "visual", | ||
| module: "implementation", | ||
| detail: "Overlapping normal controls", | ||
| fix: "Add spacing", | ||
| evidence: "ordinary", | ||
| }, | ||
| ] | ||
| : []), | ||
| ], | ||
| summary: "Cover-only", | ||
| coverHiddenCritique: { assessment: "unverified", reasons: ["Missing"], improvements: [] }, | ||
| }); | ||
| const result = await reviewStep( | ||
| snapshot, | ||
| { contract: {} } as CreativeDirection, | ||
| "account", | ||
| "site", | ||
| undefined, | ||
| { baseRevision: 7, releaseArtwork: url }, | ||
| ); | ||
| expect(result.verdict).toBe(hasOrdinary ? "revise" : "pass"); | ||
| expect(result.issues).toEqual( | ||
| hasOrdinary | ||
| ? [ | ||
| { | ||
| severity: "visual", | ||
| module: "implementation", | ||
| detail: "Overlapping normal controls", | ||
| fix: "Add spacing", | ||
| }, | ||
| ] | ||
| : [], | ||
| ); | ||
| }, | ||
| ); | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.