Skip to content

Commit 401157e

Browse files
committed
feat(effects): check an effect's code off the main thread, and price the tiers on an M1
A Metal compile takes 120 ms for a new source on an M1, up to 375 ms for the first in a process (#1206), against 10 ms for FXC. checkShaderEffect ran it on the main process's thread. - The addon's checkShaderEffect is an AsyncTask on the libuv pool, like segmentFrame. The service and the checker type return a promise; the service awaits it inside its try, so a rejecting addon still reads as "not checked", as a throwing one did. - runDocumentTool asks first, for the code addEffect or setEffectDefinition will store (shaderEffectToCheck), then hands the answer to the executor, which stays synchronous, as the cursor read and the auto-zoom plan already do. Edits off: nothing is asked. - addEffect's description and the MCP contract resource give each tier in ms on the RTX 4070 Ti and on the M1, as the spec's C.7 asks.
1 parent 8431e83 commit 401157e

12 files changed

Lines changed: 300 additions & 38 deletions

File tree

‎crates/compositor-view-napi/src/lib.rs‎

Lines changed: 33 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -401,20 +401,41 @@ pub struct ShaderEffectCheck {
401401
pub iterations: Option<f64>,
402402
}
403403

404+
pub struct CheckShaderEffectTask {
405+
code: String,
406+
params_json: String,
407+
}
408+
409+
impl Task for CheckShaderEffectTask {
410+
type Output = ShaderEffectCheck;
411+
type JsValue = ShaderEffectCheck;
412+
413+
fn compute(&mut self) -> Result<Self::Output> {
414+
Ok(match openscreen_compositor::shader_fx::check_for_agent_json(&self.code, &self.params_json) {
415+
Ok(report) => ShaderEffectCheck {
416+
error: None,
417+
tier: Some(report.tier.name().to_string()),
418+
frame_reads: Some(report.frame_reads as f64),
419+
iterations: Some(report.iterations as f64),
420+
},
421+
Err(error) => ShaderEffectCheck { error: Some(error), tier: None, frame_reads: None, iterations: None },
422+
})
423+
}
424+
425+
fn resolve(&mut self, _env: Env, check: Self::Output) -> Result<Self::JsValue> {
426+
Ok(check)
427+
}
428+
}
429+
404430
/// Vérifie le WGSL d'un effet shader : validation, bornes des boucles, compilation réelle pour le
405-
/// backend de la machine quand elle existe (FXC sur Windows), et palier de coût. `params_json` :
406-
/// les paramètres déclarés, `[{ "name": "strength", "type": "number" }, …]`.
431+
/// backend de la machine (FXC sur Windows, Metal sur macOS, SPIR-V sur Linux), et palier de coût.
432+
/// `params_json` : les paramètres déclarés, `[{ "name": "strength", "type": "number" }, …]`.
433+
///
434+
/// `AsyncTask`, comme `segment_frame` : la compilation Metal d'une source neuve prend 120 ms, et
435+
/// jusqu'à 375 ms pour la première du processus. Ce temps n'a rien à faire sur le thread principal.
407436
#[napi]
408-
pub fn check_shader_effect(code: String, params_json: String) -> ShaderEffectCheck {
409-
match openscreen_compositor::shader_fx::check_for_agent_json(&code, &params_json) {
410-
Ok(report) => ShaderEffectCheck {
411-
error: None,
412-
tier: Some(report.tier.name().to_string()),
413-
frame_reads: Some(report.frame_reads as f64),
414-
iterations: Some(report.iterations as f64),
415-
},
416-
Err(error) => ShaderEffectCheck { error: Some(error), tier: None, frame_reads: None, iterations: None },
417-
}
437+
pub fn check_shader_effect(code: String, params_json: String) -> AsyncTask<CheckShaderEffectTask> {
438+
AsyncTask::new(CheckShaderEffectTask { code, params_json })
418439
}
419440

420441
/// Installe la scène de l'app (JSON `SceneDescription`) sur la vue : layout preset piloté par

‎docs/shader-effects.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -416,6 +416,9 @@ Il y a deux compilations, à deux moments :
416416
`setEffectDefinition`), le processus principal appelle `checkShaderEffect(code, params)` de l'addon.
417417
- **La compilation est réelle,** pour le backend de la machine, en plus de la validation
418418
naga. « Accepté » veut donc dire « se dessinera ».
419+
- **Elle tourne sur le pool de libuv** (une `AsyncTask`) : la compilation Metal d'un code neuf
420+
prend 120 ms, et jusqu'à 375 ms pour la première du processus. `runDocumentTool` attend la
421+
réponse, puis la passe à l'exécuteur des outils, qui reste synchrone.
419422
- **Le coût est estimé** depuis le code (C.7), sans rendu ni mesure sur la machine.
420423
- **L'erreur revient à l'agent** avec la ligne et la colonne de SON code.
421424
2. **À la pose de la scène** (`Compositor::set_scene`). Chaque code nouveau part compiler sur un
@@ -743,9 +746,8 @@ module et pipeline compris ; les paliers en C.7.
743746
| Vue live contre MP4 (BT.709), image entière | 3,1 / 255 avec effet, 3,0 sans |
744747
| Vue live contre MP4, sur les pixels où l'effet agit | 13 à 16 / 255 avec effet, 14 à 16 sans effet sur les mêmes pixels : le bruit du codec sur des bords nets |
745748

746-
La compilation, dix fois plus lente qu'avec FXC, reste hors du rendu. `checkShaderEffect`, lui,
747-
compile sur le thread qui l'appelle : le processus principal y passe ces 120 ms à chaque code
748-
neuf.
749+
La compilation, dix fois plus lente qu'avec FXC, reste hors du rendu. Elle tourne aussi à
750+
l'écriture (`checkShaderEffect`), d'où le pool de libuv plutôt que le thread principal (D.4).
749751

750752
Le détail est dans
751753
[rendering-performance.md](../technical-documentation/engineering/rendering-performance.md).

‎electron/ai-edition/agent-tools.test.ts‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@ import {
1313
executeAgentTool,
1414
isMutatingTool,
1515
MUTATING_TOOL_NAMES,
16+
shaderEffectCheckLookup,
17+
shaderEffectToCheck,
1618
} from "./agent-tools";
1719
import { measureTextInShippedFonts } from "./text-measure";
1820

@@ -3924,3 +3926,37 @@ describe("shader effects", () => {
39243926
expect(JSON.parse(added.resultJson)).toMatchObject({ validated: false, cost: null });
39253927
});
39263928
});
3929+
3930+
describe("the executor's view of a check asked beforehand", () => {
3931+
const source = {
3932+
code: "fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv); }",
3933+
params: [{ name: "level", type: "number" as const }],
3934+
};
3935+
const check = { tier: "light" as const, frameReads: 1, iterations: 0 };
3936+
3937+
it("is the answer for the same code and layout, and nothing for any other", () => {
3938+
const lookup = shaderEffectCheckLookup(source, check);
3939+
expect(lookup(source.code, [{ name: "level", type: "number" }])).toBe(check);
3940+
expect(lookup(`${source.code}\n`, source.params)).toBeNull();
3941+
expect(lookup(source.code, [{ name: "level", type: "color" }])).toBeNull();
3942+
expect(lookup(source.code, [])).toBeNull();
3943+
});
3944+
3945+
it("is asked only by the two tools that compile", () => {
3946+
const document = fixtureDocument();
3947+
expect(
3948+
shaderEffectToCheck(document, "addEffect", { startSec: 0, endSec: 1, effectId: "fx_1" }),
3949+
).toBeNull();
3950+
expect(shaderEffectToCheck(document, "addTrim", { startSec: 0, endSec: 1 })).toBeNull();
3951+
expect(
3952+
shaderEffectToCheck(document, "addEffect", {
3953+
startSec: 0,
3954+
endSec: 1,
3955+
effect: { name: "A", description: "", code: source.code },
3956+
}),
3957+
).toEqual({ code: source.code, params: [] });
3958+
expect(
3959+
shaderEffectToCheck(document, "setEffectDefinition", { effectId: "fx_missing", code: "x" }),
3960+
).toBeNull();
3961+
});
3962+
});

‎electron/ai-edition/agent-tools.ts‎

Lines changed: 53 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ import {
6161
findShaderEffect,
6262
SHADER_EFFECT_CONTRACT,
6363
type ShaderEffectCheck,
64-
type ShaderEffectChecker,
64+
type ShaderEffectCheckLookup,
6565
shaderEffectCostAdvice,
6666
shaderEffectTierOf,
6767
storedShaderEffectValues,
@@ -1526,7 +1526,7 @@ export function documentSnapshotForModel(
15261526
* wired, its cost cached. An error reads as the agent must act on it. */
15271527
function checkedEffectDefinition(
15281528
candidate: unknown,
1529-
checker: ShaderEffectChecker | undefined,
1529+
checker: ShaderEffectCheckLookup | undefined,
15301530
): { definition: AxcutShaderEffect; check: ShaderEffectCheck | null } | { error: string } {
15311531
const parsed = shaderEffectSchema.safeParse(candidate);
15321532
if (!parsed.success) {
@@ -1546,6 +1546,53 @@ function checkedEffectDefinition(
15461546
return { definition: { ...definition, cost }, check };
15471547
}
15481548

1549+
type ShaderEffectSource = {
1550+
code: string;
1551+
params: Array<{ name: string; type: AxcutShaderEffect["params"][number]["type"] }>;
1552+
};
1553+
1554+
/** The code and parameter layout `addEffect` or `setEffectDefinition` will have the compositor
1555+
* check, read from the same arguments the executor reads; `null` for a call that compiles nothing.
1556+
* The check is asynchronous (a Metal compile takes 120 ms and more) and the executor is not: the
1557+
* async wrapper asks first, then hands the answer over (`shaderEffectCheckLookup`). */
1558+
export function shaderEffectToCheck(
1559+
document: AxcutDocument,
1560+
name: string,
1561+
args: unknown,
1562+
): ShaderEffectSource | null {
1563+
const layout = (
1564+
params: ReadonlyArray<Pick<AxcutShaderEffect["params"][number], "name" | "type">>,
1565+
) => params.map(({ name, type }) => ({ name, type }));
1566+
if (name === "addEffect") {
1567+
const parsed = addEffectArgs.safeParse(args);
1568+
if (!parsed.success || !parsed.data.effect) return null;
1569+
return { code: parsed.data.effect.code, params: layout(parsed.data.effect.params ?? []) };
1570+
}
1571+
if (name === "setEffectDefinition") {
1572+
const parsed = setEffectDefinitionArgs.safeParse(args);
1573+
if (!parsed.success) return null;
1574+
const { effectId, code, params } = parsed.data;
1575+
const existing = findShaderEffect(document, effectId);
1576+
if (!existing || (code === undefined && params === undefined)) return null;
1577+
return { code: code ?? existing.code, params: layout(params ?? existing.params) };
1578+
}
1579+
return null;
1580+
}
1581+
1582+
/** The executor's view of an answer asked for `source`: that answer for the same code and
1583+
* layout, `null` (not checked) for anything else. */
1584+
export function shaderEffectCheckLookup(
1585+
source: ShaderEffectSource,
1586+
check: ShaderEffectCheck | null,
1587+
): ShaderEffectCheckLookup {
1588+
return (code, params) =>
1589+
code === source.code &&
1590+
params.length === source.params.length &&
1591+
params.every((p, i) => p.name === source.params[i].name && p.type === source.params[i].type)
1592+
? check
1593+
: null;
1594+
}
1595+
15491596
/** Two effects of one lane cannot overlap: the refusal names what is in the way, and the way out. */
15501597
function effectLaneBlocked(
15511598
ranges: AxcutShaderEffectRange[],
@@ -1733,9 +1780,10 @@ export interface AgentToolOptions {
17331780
/** Whether this machine can cut out the camera background, asked only by the two tools that
17341781
* report it. Absent where nothing can ask the compositor: reported as not checked. */
17351782
cameraSegmentation?: () => SegmentationSupport;
1736-
/** Asks the compositor to check an effect's WGSL (`checkShaderEffect`). Absent where nothing
1737-
* can: the effect is stored unchecked, and the result says `validated: false`. */
1738-
checkShaderEffect?: ShaderEffectChecker;
1783+
/** The compositor's check of the WGSL this call writes (`checkShaderEffect`), asked beforehand
1784+
* by the async wrapper (`shaderEffectToCheck`). Absent where nothing can: the effect is stored
1785+
* unchecked, and the result says `validated: false`. */
1786+
checkShaderEffect?: ShaderEffectCheckLookup;
17391787
/** `addAutoZooms`' plan, made by the async wrapper from the recorded clicks. Absent means
17401788
* no cursor reader is wired here, which the tool reports as "could not look". */
17411789
autoZooms?: AutoZoomPlan;

‎electron/ai-edition/deep-agent/service.test.ts‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -780,3 +780,104 @@ describe("addAutoZooms reads the clicks through the runtime's cursor reader", ()
780780
expect(result.resultJson).toContain("could not be read");
781781
});
782782
});
783+
784+
describe("an effect's code is checked off the main thread, before the executor reads it", () => {
785+
const LEVEL = { type: "number", name: "level", label: "Level", default: 1, min: 0, max: 2 };
786+
const effect = (code: string) => ({
787+
name: "Dim",
788+
description: "Dims the picture.",
789+
code,
790+
params: [LEVEL],
791+
});
792+
// The compositor's check takes a while (a Metal compile is 120 ms and more): a promise that
793+
// settles later, refusing `nope` and pricing the rest at 20 reads.
794+
const slowChecker = (asked: Array<{ code: string; params: unknown }>) => ({
795+
checkShaderEffect: async (
796+
code: string,
797+
params: ReadonlyArray<{ name: string; type: string }>,
798+
) => {
799+
asked.push({ code, params });
800+
await new Promise((resolve) => setTimeout(resolve, 5));
801+
return code.includes("nope")
802+
? { error: "error: no definition in scope for identifier: `nope`" }
803+
: { tier: "medium" as const, frameReads: 20, iterations: 20 };
804+
},
805+
});
806+
807+
it("waits for the compositor's answer, accepting or refusing on it", async () => {
808+
const asked: Array<{ code: string; params: unknown }> = [];
809+
const runtime = slowChecker(asked);
810+
const accepted = await runDocumentTool(
811+
fixtureDocument(),
812+
"addEffect",
813+
{
814+
startSec: 1,
815+
endSec: 2,
816+
effect: effect("fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv); }"),
817+
},
818+
true,
819+
runtime,
820+
);
821+
expect(accepted.ok).toBe(true);
822+
const result = JSON.parse(accepted.resultJson);
823+
expect(result.validated).toBe(true);
824+
expect(result.cost).toMatchObject({ tier: "medium", frameReads: 20 });
825+
expect(asked[0].params).toEqual([{ name: "level", type: "number" }]);
826+
827+
const refused = await runDocumentTool(
828+
fixtureDocument(),
829+
"addEffect",
830+
{ startSec: 1, endSec: 2, effect: effect("fn effect(uv: vec2<f32>) -> vec4<f32> { nope }") },
831+
true,
832+
runtime,
833+
);
834+
expect(refused.ok).toBe(false);
835+
expect(refused.resultJson).toContain("does not compile");
836+
});
837+
838+
it("checks a new code against the definition's parameters when only the code changes", async () => {
839+
const asked: Array<{ code: string; params: unknown }> = [];
840+
const runtime = slowChecker(asked);
841+
const added = await runDocumentTool(
842+
fixtureDocument(),
843+
"addEffect",
844+
{
845+
startSec: 1,
846+
endSec: 2,
847+
effect: effect("fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv); }"),
848+
},
849+
true,
850+
runtime,
851+
);
852+
const { effectId } = JSON.parse(added.resultJson);
853+
const document = added.document as AxcutDocument;
854+
const code = "fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv) * params.level; }";
855+
const changed = await runDocumentTool(
856+
document,
857+
"setEffectDefinition",
858+
{ effectId, code },
859+
true,
860+
runtime,
861+
);
862+
expect(changed.ok).toBe(true);
863+
expect(asked.at(-1)).toEqual({ code, params: [{ name: "level", type: "number" }] });
864+
});
865+
866+
it("asks nothing when edits are off, or for a tool that compiles nothing", async () => {
867+
const asked: Array<{ code: string; params: unknown }> = [];
868+
const runtime = slowChecker(asked);
869+
await runDocumentTool(
870+
fixtureDocument(),
871+
"addEffect",
872+
{
873+
startSec: 1,
874+
endSec: 2,
875+
effect: effect("fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv); }"),
876+
},
877+
false,
878+
runtime,
879+
);
880+
await runDocumentTool(fixtureDocument(), "addTrim", { startSec: 1, endSec: 2 }, true, runtime);
881+
expect(asked).toEqual([]);
882+
});
883+
});

0 commit comments

Comments
 (0)