diff --git a/README.md b/README.md index b5701a1..1736bae 100644 --- a/README.md +++ b/README.md @@ -1,85 +1,48 @@ -Audio Graph JS — KHR_audio_graph Runtime (Node/Web) +Audio Graph JS - Layered glTF Audio Runtime (Node/Web) **What It Is** -- **Runtime:** A minimal Web Audio–based runtime for graphs shaped like the glTF KHR_audio_graph proposal. -- **Parity tools:** Side‑by‑side runner + comparator to verify parity between native runtime GraphSpecs and KHR containers. -- **Emitters:** Placement‑agnostic emitters with instance expansion from glTF node bindings. +- **Runtime:** A minimal Web Audio-based runtime for layered glTF audio proposals built on `KHR_audio_emitter`, with compatibility support for legacy `KHR_audio_graph` containers. +- **Parity tools:** Side-by-side runner and comparator to verify parity between native runtime GraphSpecs and glTF containers. +- **Emitters:** Placement-agnostic emitters with instance expansion from glTF node or scene bindings. **Requirements** -- **Node.js:** 18+ (includes `globalThis.fetch`). -- Optional: macOS/Linux/Windows shell to run example scripts. +- **Node.js:** 18+. **Install & Build** - Install deps: `npm install` - TypeScript build: `npm run build` -**Quick Start: Run Examples** -- Render one example (writes WAV + trace): `npm run example:run-graph -- examples/graphs/osc.json` -- Run all core examples: `npm run examples` -- Compare KHR vs runtime graphs (traces only): `npm run example:compare-khr` -- Strict compare (traces + WAV checksums): `npm run example:compare-khr:strict` - - Note: “seven-nation-army” WAV strict is intentionally skipped; traces still match. +**Quick Start** +- Render one example: `npm run example:run-graph -- examples/graphs/osc.json` +- Run tests: `npm test` **Emitter Instance Expansion** -- The base graph builds `emitter` nodes as shared input buses (no auto‑connect to destination). -- Instance expansion creates per‑instance panner and post‑gain, connecting bus → panner → gain → destination. -- Two ways to provide instances: - - **glTF node bindings:** Attach `extensions.KHR_audio_graph = { emitter: }` or `{ emitters: [, ...] }` to glTF nodes. The example runner extracts node T/R/S “as is” and expands instances at runtime. - - **Runtime test harness:** Add a top‑level `__emitterInstances` to a runtime GraphSpec, e.g. `{ emitterNodeId: "emit", translation: [1,0,0] }`. - -**Run The New Spatial Emitter Parity Example** -- Runtime graph with instance expansion hint: `node examples/run-graph.mjs examples/graphs/spatial-emitter-instanced.json` -- glTF + KHR graph with node binding: `node examples/run-graph.mjs examples/graphs-khr/spatial-emitter-instanced-khr.json` -- Comparator picks both up automatically. - -**API Usage (Node)** -- Build a graph and apply instances programmatically: - -``` -import wae from 'web-audio-engine'; -import { buildGraphAsync, applyEmitterInstances } from './dist/index.js'; - -const { OfflineAudioContext } = wae; -const spec = { - nodes: [ - { id: 'src', kind: 'oscillator', params: { type: 'sine', frequency: 330, startTime: 0 } }, - { id: 'emit', kind: 'emitter', params: { emitterType: 'spatial', gain: 1 } }, - ], - connections: [ { from: { node: 'src' }, to: { node: 'emit' } } ], -}; - -const sr = 48000; -const ctx = new OfflineAudioContext(2, sr * 2, sr); -const built = await buildGraphAsync(ctx, spec); - -applyEmitterInstances(built, spec, [ - { emitterNodeId: 'emit', translation: [1, 0, 0] }, -]); - -const audioBuffer = await ctx.startRendering(); -``` - -**glTF/KHR Mapping In Runner** -- The example runner (`examples/run-graph.mjs`) accepts either: - - A runtime GraphSpec: `{ nodes, connections, outputs? }` - - A glTF JSON with `extensions.KHR_audio_graph` container and optional node bindings (scalar `emitter` or array `emitters`). -- For mappings, it also: - - Synthesizes deterministic noise or IR where needed to ensure parity. - - Applies simple musical automation for select presets. - -**Testing & Linting** -- Unit tests: `npm test` -- Graph lint: enforced by `buildGraph(Async)` usage in the runner; you can invoke `lintGraph(spec)` manually. +- The base graph builds `emitter` nodes as shared input buses with no direct auto-connect to the destination. +- Instance expansion creates the final panner and post-gain stages for each binding. +- Multiple upstream graph outputs may target the same emitter bus. The runtime mixes those inputs by default before spatialization or final gain. This is informative runtime guidance and matches the intended layered spec behavior. +- Layered glTF bindings: + - Node-level: `extensions.KHR_audio_emitter = { emitter: }` or `{ emitters: [, ...] }` + - Scene-level for global emitters: `extensions.KHR_audio_emitter = { emitters: [, ...] }` +- Runtime test harness: + - Add `__emitterInstances` to a runtime GraphSpec, for example `{ emitterNodeId: "emit", translation: [1, 0, 0] }` + +**Runner Input Modes** +- Runtime GraphSpec: `{ nodes, connections, outputs? }` +- Layered glTF: `KHR_audio_emitter`, optionally `KHR_audio_graph` and `KHR_audio_environment` +- Legacy glTF: `extensions.KHR_audio_graph` **Useful Scripts** -- `npm run spec:validate` — validate KHR graphs using the included schema and examples. -- `npm run spec:validate:gltf` — validate glTF examples. +- `npm run spec:validate` +- `npm run spec:validate:gltf` +- `node tools/spec-validate/validate-layered.mjs ` **Notes** -- Listener design and KHR_animation_pointer integration are TODO (documented under `docs/` and spec repo notes). -- Stereo panner uses an approximation in this runtime; parity tests account for it. +- Listener application is available for layered inputs. +- `KHR_animation_pointer` integration is still TODO. +- Stereo panner uses an approximation in this runtime where needed. **Where Things Live** -- Runtime code: `src/runtime`, nodes under `src/nodes`. -- Examples and tools: `examples/*`, `tools/*`. -- Doc updates: `docs/USAGE.md`, `docs/PLAN_AND_STATUS.md`, `docs/STATE_OF_PROJECT.md`. +- Runtime code: `src/runtime` +- Serialization and parsing: `src/serialization` +- Examples: `examples` +- Notes: `docs` diff --git a/docs/KHR_AUDIO_GRAPH_NODE_SUMMARY.md b/docs/KHR_AUDIO_GRAPH_NODE_SUMMARY.md index 0a1d04f..966bc08 100644 --- a/docs/KHR_AUDIO_GRAPH_NODE_SUMMARY.md +++ b/docs/KHR_AUDIO_GRAPH_NODE_SUMMARY.md @@ -1,47 +1,39 @@ -# KHR_audio_graph Node Mapping (Tracking) - -- Source → AudioBufferSourceNode (implemented) - - id, data(audio|osc), priority, gain, state, autoPlay, loop, loopStart(ms), loopEnd(ms), playbackSpeed, duration(ms), offset(ms), when(s), channelInterpretation -- Audio Data (partial) - - bufferView, uri, mimeType, encodingProperties(bitsPerSample, duration, samples, sampleRate, channels) -- Oscillator Data → OscillatorNode (implemented) - - type, frequency, pulseWidth (static via PeriodicWave; no modulation) -- Emitter (implemented) - - id, emitterType(global|spatial), gain, spatialProperties(spatializationModel, attenuation) - - Maps to GainNode + (PannerNode/StereoPanner) -- Listener → AudioListener (skipped) - -Processors -- Gain → GainNode (implemented) - - gain, interpolation, duration(ms) -- Delay → DelayNode (implemented) - - delayTime(ms) -- Pitch Shifter (deferred) - - pitch(semitones) -- Channel Splitter → ChannelSplitterNode (implemented) -- Channel Merger → ChannelMergerNode (implemented) -- Channel Mixer → GainNode with channelCount (implemented) - - outputChannels -- Audio Mixer (composite) (not_implemented) -- Audio Mixer → GainNode (summing) (implemented) -- Filters (implemented via BiquadFilterNode) - - lowpass: frequency, qualityFactor, bypass - - highpass: frequency, qualityFactor, bypass - - bandpass: frequency, qualityFactor, bypass - - lowshelf: frequency, gain, bypass - - highshelf: frequency, gain, bypass - - peaking: frequency, qualityFactor, gain, bypass - - notch: frequency, qualityFactor, bypass - - allpass: frequency, qualityFactor, bypass -- Reverb (IR-based) (implemented) - - ConvolverNode + wet/dry mix; algorithmic parameters out of scope for now - -Notes -- Unit conversions: ms ↔ s for Delay/Source timing. -- Bypass: requires routing toggle; not native on Web Audio nodes. - - Bypass: supported two ways — build-time rewiring (spec JSON) and runtime toggling via wrapper (setBypass). For filters/delay/convolver/waveshaper, graphs are built with a dry/wet wrapper for runtime bypass. -- Pitch shifter & Reverb: require custom DSP or Worklets; not standard nodes. -- Spatialization: Emitter maps to PannerNode; listener bound to camera. - - Gain smoothing: honors `interpolation` (linear/custom) + `duration` (ms) when provided. - -See docs/KHR_AUDIO_GRAPH_NODE_MAP.json for machine-readable detail. +# Layered Audio Runtime Mapping + +- `KHR_audio_emitter.audio[]` + - asset-level audio payloads + - optional `KHR_audio_graph.encoding` metadata + +- `KHR_audio_emitter.sources[]` + - maps to runtime audio-buffer-source setup + - extended playback fields from `extensions.KHR_audio_graph` + +- `KHR_audio_emitter.emitters[]` + - maps to shared runtime emitter buses + - `type: global` -> gain-only output instance + - `type: positional` -> panner plus post-gain output instance + +- `KHR_audio_graph.graphs[]` + - parsed individually, then merged for execution + - shared emitter targets remain shared across merged graphs + - when multiple outputs target the same emitter, signals are summed by default + +- `KHR_audio_environment.listeners[]` + - maps to `AudioListener` + - transform comes from the bound glTF node + +- `KHR_audio_environment.environments[]` + - scene-level environment currently supported + - reverb is implemented through a real wet/dry environment bus + +## Important Conversions + +- glTF time in the layered runtime path is interpreted in seconds +- glTF cone angles are interpreted in radians +- Web Audio panner cone angles use degrees, so the runtime converts radians to degrees + +## Current Non-Goals + +- node-localized environment zones +- `KHR_animation_pointer` integration +- full spec-sync cleanup of older historical notes diff --git a/docs/PLAN_AND_STATUS.md b/docs/PLAN_AND_STATUS.md index 4ba5f07..1beab15 100644 --- a/docs/PLAN_AND_STATUS.md +++ b/docs/PLAN_AND_STATUS.md @@ -1,33 +1,33 @@ -# Plan & Status — Listener/Emitter Spec + Parser +# Plan And Status -Status: in progress +Status: current layered runtime milestone complete -Phases +## Completed -- [x] Docs: Complex Graph (USAGE section) -- [ ] Spec: Listener language (README; KHR_animation_pointer TODO) -- [ ] Spec: Emitter → Node binding (README + node-level extension schema) -- [ ] Spec: Commit + spec validation (no example breakage) -- [ ] Runtime: glTF parser for emitter bindings (separate class/file) -- [ ] Test: Focused spatial emitter parity (one case; identical transforms) -- [ ] Docs: Spatial Emitter Binding (USAGE) + STATE_OF_PROJECT update +- layered parser for `KHR_audio_emitter`, `KHR_audio_graph`, and `KHR_audio_environment` +- node-level and scene-level emitter binding extraction +- merged execution for multiple layered graphs +- additive default mixing on shared emitter buses +- listener application in the runner +- scene-level environment routing in the runner +- radians-to-degrees conversion for positional emitter cone angles +- layered validator wildcard support +- real glTF fixture tests for layered examples -Listener — Scope (this pass) -- Not a graph node; scene/node-level concept; single Listener per scene. -- KHR_animation_pointer applicability is deferred (TODO). We will specify accessible properties and pointer paths in a later pass. +## Current Scope -Emitter Binding — Node-level (this pass) -- Do not add node ids inside graph emitter objects. -- Bind emitters at glTF node level via a node extension: - - Scalar form: `extensions.KHR_audio_graph = { "emitter": }` - - Array form: `extensions.KHR_audio_graph = { "emitters": [, ...] }` -- Each glTF node creates instances for each referenced emitter id. Multiple nodes may reference the same emitter id. -- Runtime optimization: share upstream audio graph routing; instantiate only the final per-instance spatial stage (PannerNode and any post‑panner gain). +- layered format is the preferred implementation path +- legacy `KHR_audio_graph` runner support remains for compatibility only +- scene-level environment is supported +- node-localized environment zones are deferred -Parser (this pass) -- Separate glTF parser external to audio graph parsing. -- For examples, read node transforms (translation/rotation/scale) “as is” — no world/hierarchy computation. +## Open Items -Test (next step) -- Add one focused spatial emitter test to compare KHR-bound instances vs a native baseline with identical transforms. +- `KHR_animation_pointer` integration +- node-localized environment zones and overlap rules +- additional layered fixture coverage for more complex mixed assets +## Reviewer Notes + +- informative runtime behavior now explicitly supports default signal summing when multiple graph outputs target the same emitter +- this should be mirrored in spec prose as informative guidance, not as hidden implementation behavior diff --git a/docs/STATE_OF_PROJECT.md b/docs/STATE_OF_PROJECT.md index 592b4e9..a6c7fc5 100644 --- a/docs/STATE_OF_PROJECT.md +++ b/docs/STATE_OF_PROJECT.md @@ -1,107 +1,66 @@ -# AudioGraphJS — Current State and Learnings (Snapshot) - -Date: 2025‑09‑01 - -## Scope Overview -- Implemented end‑to‑end tooling and examples to align a runtime GraphSpec with KHR_audio_graph container graphs and verify parity via traces. -- Updated spec text (README) and polished schemas inside `spec-repo` (embedded) to reflect the agreed graph model and conventions. -- Added validators, a graph runner, a comparator, and a set of aligned examples (runtime and KHR) including musical presets. - -## Spec/Text and Schema Updates -- README (spec): - - Extension container shape: `audioData[]` + `graphs[]` with nodes, connections, and optional `outputs[]` sinks. - - Conventions: all times in ms in the spec; runtime converts ms→s. - - Node discriminator: `{ kind, params, label? }` with explicit index/port connections. - - Listener is scene‑level; emitter is a 1‑in/0‑out sink; graphs must have a sink (emitter or `outputs[]`). - - Replaced legacy “procedurals/nodegraph” sections. -- Schemas (select fixes): - - Highshelf title/limits corrected; `source.gain` clamped [0..1]; `source.state` enum; `emitter.gain` [0..1]. -- Spec linter (tools/spec-validate/validate.mjs): - - DAG cycle detection, sink requirement, emitter degrees, splitter/merger/mixer arities (sink exceptions for outputs[]). - -## Runtime/Tools -- Graph builders: `buildGraph`, `buildGraphAsync` now auto‑connect `outputs[]` sinks; emitters also auto‑connect. -- Linter: `lintGraph(spec)` (DAG, sinks, emitter degrees, basic arity with sink exceptions). -- Runner CLI: `examples/run-graph.mjs` - - Accepts runtime GraphSpec or KHR container. - - Mapping rules (KHR → runtime): - - Labels preserved as node ids; connections/outputs indices remapped to ids. - - Oscillator type number → Web Audio type (0 sine, 1 square, 2 sawtooth, 3 triangle). - - Source data.audioData → buffer uri (from extensions.KHR_audio_graph.audioData[]). - - Reverb → Convolver; uri resolved from `audioData` or synthesized IR fallback. - - Runtime GraphSpec enrichments: - - If a convolver has no `uri`, inject a small mono IR (synthesized). - - For musical presets, inject noise buffers (data URIs) for snare/cymbal if missing. - - Trace logging via `createMemoryTrace()`; numbers normalized to 3 decimals. -- KHR validator CLI: `tools/spec-validate/validate-gltf.mjs` - - Validates `extensions.KHR_audio_graph` in glTF files and runs lints (DAG, sinks, degrees/arity). -- Comparator: `examples/compare-khr-runtime.mjs` - - Runs matched pairs from `examples/graphs/*.json` vs `examples/graphs-khr/*-khr.json`, diffs traces (top 20 lines of diffs). - - Strict mode (`--strict-wav` or `STRICT_WAV=1`) compares WAV checksums; noise is generated deterministically via seeded PRNG. Currently skips only the complex `seven-nation-army` pair. - -## Examples (Aligned Pairs) -- Core - - simple-outputs, split-merge-mix, with-emitter, delay, convolver-mix, spatial-emitter. -- Musical - - drum (kick‑style), snare, cymbal, bass, seven-nation-army (riff). -- Each has a runtime GraphSpec JSON and a KHR container JSON with matching topology and labels. -- Baseline traces: `examples/trace-*.txt` and `examples/trace-*_khr.txt`. - -## Sinks Preference -- Adopted global emitter sinks for parity and clarity. Graphs with a single `outputs[]` entry were migrated to a global emitter (emitterType `global`), preserving connectivity. -- Helper: `tools/update-emitters.mjs` automates this migration for both runtime and KHR graphs. - -## Stereo Panner Approximation -- KHR lacks a native stereo panner. We approximate equal‑power panning via split → gains → merge. -- Coefficients for pan x ∈ [-1, 1]: L = cos((x+1)π/4), R = sin((x+1)π/4). -- See examples: `stereo-panner.json` and `stereo-panner-khr.json`. - -## Musical Preset Automation (Runner) -- drum: body/click envelopes and sweeps. -- snare: tone pitch down + noise decay. -- cymbal: long decay envelope on `gainEnv`. -- bass: ADSR‑like envelope; filter ramps skipped in runner to avoid wrapped node handle. -- seven-nation-army: note scheduling with per‑note amp envelopes; filter ramps skipped. - -## Current Parity Status (Comparator) -- All aligned pairs report “traces match”: - - bass, convolver-mix, cymbal, delay, drum, seven-nation-army, simple-outputs, snare, spatial-emitter, split-merge-mix, with-emitter. -- WAV checksums are produced by the runner but not strictly enforced in comparator (intentionally non‑fatal). - -## Key Learnings -- Preserve human‑readable `label` in KHR nodes to achieve stable id‑based traces; remap connection indices accordingly. -- Spec ms→s conversion must be centralized and consistent (source start/offset/duration/loop; delay times, etc.). -- When comparing disparate ecosystems (container vs runtime), inject minimal synthetic content (IR/noise) to stabilize parity, but keep this behind tooling (runner) and document it. -- For wrapped processor nodes (bypass wrappers), direct parameter automation requires exposing or proxying the inner node; otherwise skip in trace‑parity flows or add a typed API. - -## How to Use -- Run a single graph (runtime or KHR): - - `npm run example:run-graph -- examples/graphs/simple-outputs.json` -- Compare all KHR vs runtime pairs: - - `npm run example:compare-khr` -- Run test harness to generate/compare traces and MD5 baselines: - - `npm run example:test-graphs` -- Validate glTF’s KHR_audio_graph sections: - - `npm run spec:validate:gltf -- path/to/scene.gltf` -- Validate schemas + example set: - - `npm run spec:validate` - -## Next Steps -- Spec/Docs - - Add “Web Audio Mapping” notes per node into spec README (from `docs/spec-sync/PROPOSED_TEXT_UPDATES.md`). - - Dedupe legacy/duplicated prose; keep Listener and animation guidance crisp. -- Runtime - - Expose inner nodes (or a param proxy) for wrapped processors to allow filter param automation in examples. - - Add optional strict WAV checksum compare mode to comparator for CI. -- Validator - - Extend `validate-gltf.mjs` to support `.glb` by extracting JSON chunk. - - Add JSON output mode for CI ingestion. -- Examples - - Add aligned pairs for lead/pad/song if desired; refine musical presets and envelopes. - -## File Map (Key) -- Spec docs/schemas: `spec-repo/extensions/2.0/Khronos/KHR_audio_graph/` -- Validators: `tools/spec-validate/` -- Runner + comparator: `examples/run-graph.mjs`, `examples/compare-khr-runtime.mjs` -- Aligned examples: `examples/graphs/` and `examples/graphs-khr/` -- Traces: `examples/trace-*.txt` +# AudioGraphJS - Current State + +Date: 2026-04-02 + +## Summary + +The active implementation direction is the layered model: + +- `KHR_audio_emitter` as the base layer +- `KHR_audio_graph` for routing and processing +- `KHR_audio_environment` for listener and environment semantics + +Legacy `KHR_audio_graph` container support remains in the runner for compatibility, but the layered path is now the primary path. + +## Implemented + +- Layered parser: + - `parseLayeredExtensions()` reads `KHR_audio_emitter`, optional `KHR_audio_graph`, and optional `KHR_audio_environment` + - scene-level and node-level emitter bindings are extracted +- Graph merging: + - multiple layered graphs are merged into one runtime graph + - shared emitters stay shared, which enables additive fan-in +- Default emitter mixing: + - multiple graph outputs may target the same emitter + - the runtime sums those inputs on the shared emitter bus by default +- Listener application: + - scene listener transform is applied to the Web Audio listener +- Environment application: + - scene-level environment creates a real wet/dry routing bus + - emitters route through that bus when environment processing is active +- Positional emitter fix: + - glTF cone angles are converted from radians to Web Audio degrees +- Validation: + - layered validator works with explicit files and wildcard patterns + +## Tests + +- Unit and integration suite: `54` passing tests +- Real layered glTF fixture coverage includes: + - `simple-emitter-only.json` + - `spatial-with-environment.json` + - `same-emitter-multi-graph.json` + +## Current Runtime Behavior + +- Layered fixtures in `examples/graphs-layered` are treated as real glTF JSON with extensions. +- Scene-level global emitters are supported. +- Node-level positional emitters are supported. +- Scene-level environment is supported. +- Node-localized environment zones are still out of scope. + +## Informative Runtime Rule + +When multiple upstream graph outputs target the same emitter, their signals are mixed by default before emitter gain and spatialization. This is the runtime's informative behavior and should be reflected in the proposal language. + +## Remaining Gaps + +- Node-localized environment zones are not implemented. +- `KHR_animation_pointer` integration is not implemented. +- The older legacy-spec docs under `docs/spec-sync` still contain historical notes and are not the source of truth. + +## Recommended Next Steps + +- add one or two more layered examples that combine source bindings and oscillator-only graph sources in the same asset +- decide whether node-localized environment zones belong in the next milestone or should stay deferred +- tighten any remaining spec prose so it matches the layered runtime exactly diff --git a/docs/USAGE.md b/docs/USAGE.md index 6895cf9..ceff6b4 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1,58 +1,37 @@ -# AudioGraphJS Runtime — Usage Notes +# AudioGraphJS Runtime - Usage Notes - GraphSpec - - `nodes`: array of `{ id, kind, params }` using runtime kinds (e.g., `audio-buffer-source`, `oscillator`, `gain`, `biquad-filter`, `delay`, `convolver`, `stereo-panner`, `panner`, `channel-splitter`, `channel-merger`, `channel-mixer`, `audio-mixer`, `emitter`, `wave-shaper`). - - `connections`: edges `{ from: { node, output? }, to: { node, input? } }`. Omitted `output`/`input` target the default port. - - `outputs?`: optional list of node ids to be connected to the destination. This supports global (non‑spatial) sinks. + - `nodes`: array of `{ id, kind, params }` + - `connections`: edges `{ from: { node, output? }, to: { node, input? } }` + - `outputs?`: optional node ids that connect directly to destination - Sinks - - Prefer `emitter` nodes as sinks: they auto‑connect to the destination (1‑in/0‑out). For global (non‑spatial) sinks, use a global emitter (emitterType: "global") instead of `outputs[]`. - - Migration helper: `node tools/update-emitters.mjs` converts graphs with a single `outputs[]` entry into a global emitter sink (preserves connections). Use with care for multi‑output graphs. + - `emitter` nodes represent shared emitter buses. + - Multiple upstream edges may target the same emitter bus. Signals are summed by default before spatialization or final gain. This is the runtime's informative default mixing rule for layered graphs. + - Use `outputs[]` only for direct global sinks that do not need emitter semantics. - Time Units - - Runtime node params use seconds for Web Audio API calls. When consuming KHR_audio_graph (ms), convert ms→s. See `examples/parse-gltf.mjs` for a mapper. - - Deterministic noise: the runner synthesizes noise using a seeded PRNG derived from the file name (seedBase). KHR `audioData` may use `GENERATE_NOISE:seconds:amp` which the runner maps to a deterministic data URI, ensuring strict WAV comparisons when desired. + - Runtime node params use seconds. + - Layered glTF is also interpreted in seconds. + - Legacy `KHR_audio_graph` container examples still map milliseconds to seconds in the runner. - Linting - - Import `lintGraph` from the runtime and validate graphs before building: - ```js - import { lintGraph } from 'audio-graph-js'; - const { errors, warnings } = lintGraph(spec); - if (errors.length) throw new Error(errors.join('\n')); - ``` - - Checks: DAG (no cycles), sink presence (emitter or outputs[]), emitter degree (in=1, out=0), basic arity for splitter/merger/mixer. - -- Spec Validation & glTF Validator (CLI) - - Schema + lints for examples: `npm run spec:validate` - - Validate real glTF files (only `extensions.KHR_audio_graph` is examined): - - `npm run spec:validate:gltf path/to/scene.gltf [more.gltf]` - - Prints JSON Schema errors and linter findings per file; ignores non‑audio parts of the glTF. - -- Comparator strict mode - - Compare KHR vs runtime pairs (traces only): `npm run example:compare-khr` - - Strict WAV checksum: `npm run example:compare-khr:strict` (skips only complex `seven-nation-army` for now). - -## Complex Graph -- Pattern: osc → lowpass → delay → (dry + convolver→wet) → audio-mixer → emitter (global sink) -- Purpose: demonstrates wet/dry routing, summed mixing, and a clear sink pattern. -- Examples: - - Runtime: `examples/graphs/complex.json` - - KHR: `examples/graphs-khr/complex-khr.json` - -## Spatial Emitter Binding -- Bind emitters to glTF nodes via a node extension. A node may reference a single emitter id or an array of emitter ids. -- Scalar form: - - `"extensions": { "KHR_audio_graph": { "emitter": } }` -- Array form: - - `"extensions": { "KHR_audio_graph": { "emitters": [ , ... ] } }` -- Instance semantics: - - Each glTF node creates one emitter instance per referenced emitter id, driven by that node’s transform (translation/rotation/scale). - - Multiple nodes may reference the same emitter id; upstream graph routing is shared, and only the final spatial stage (panner + optional post‑panner gain) is instantiated per instance. - -## Stereo Panner Approximation -- KHR currently has no native `stereo-panner` node. For parity, we approximate equal‑power panning using: - - Split mono -> two gains -> merge to stereo. - - For pan `x` in [-1, 1]: - - Left gain `L = cos((x + 1) * π / 4)` - - Right gain `R = sin((x + 1) * π / 4)` -- See `examples/graphs/stereo-panner.json` and `examples/graphs-khr/stereo-panner-khr.json`. + - `lintGraph(spec)` checks DAG validity, sink presence, emitter degree (`in >= 1`, `out = 0`), and basic routing arity. + - `lintLayeredGraph(graph, audioEmitter)` checks layered graph structure and base-layer references. + +- Layered Binding Model + - Node-level emitter binding: + - `"extensions": { "KHR_audio_emitter": { "emitter": } }` + - `"extensions": { "KHR_audio_emitter": { "emitters": [, ...] } }` + - Scene-level emitter binding for global emitters: + - `"extensions": { "KHR_audio_emitter": { "emitters": [, ...] } }` + - Layered graphs may fan in to the same emitter. The runtime combines those inputs on the shared emitter bus. + +- Layered Environment Model + - Listener binding is read from `KHR_audio_environment` on nodes. + - Environment binding is read from `KHR_audio_environment` on scenes. + - Current runtime scope: scene-level environment only. + +- Validation + - Layered examples: `node tools/spec-validate/validate-layered.mjs ` + - Legacy graph schema validation: `npm run spec:validate` diff --git a/examples/graphs-layered/same-emitter-multi-graph.json b/examples/graphs-layered/same-emitter-multi-graph.json new file mode 100644 index 0000000..ab6d0d3 --- /dev/null +++ b/examples/graphs-layered/same-emitter-multi-graph.json @@ -0,0 +1,50 @@ +{ + "asset": { "version": "2.0" }, + "extensionsUsed": ["KHR_audio_emitter", "KHR_audio_graph"], + "extensions": { + "KHR_audio_emitter": { + "audio": [], + "sources": [], + "emitters": [ + { "type": "global", "gain": 1.0, "sources": [] } + ] + }, + "KHR_audio_graph": { + "graphs": [ + { + "name": "voice-a", + "nodes": [ + { "kind": "oscillator", "params": { "type": "sine", "frequency": 440 }, "label": "oscA" }, + { "kind": "gain", "params": { "gain": 0.3 }, "label": "gainA" } + ], + "connections": [ + { "from": { "node": 0 }, "to": { "node": 1 } } + ], + "outputs": [ + { "node": 1, "emitter": 0 } + ] + }, + { + "name": "voice-b", + "nodes": [ + { "kind": "oscillator", "params": { "type": "sine", "frequency": 440 }, "label": "oscB" }, + { "kind": "gain", "params": { "gain": 0.3 }, "label": "gainB" } + ], + "connections": [ + { "from": { "node": 0 }, "to": { "node": 1 } } + ], + "outputs": [ + { "node": 1, "emitter": 0 } + ] + } + ] + } + }, + "scenes": [ + { + "extensions": { + "KHR_audio_emitter": { "emitters": [0] } + } + } + ] +} diff --git a/examples/run-graph.mjs b/examples/run-graph.mjs index 1f76e55..4ae25db 100644 --- a/examples/run-graph.mjs +++ b/examples/run-graph.mjs @@ -5,7 +5,19 @@ import path from 'node:path'; import { fileURLToPath } from 'node:url'; import crypto from 'node:crypto'; import wae from 'web-audio-engine'; -import { buildGraphAsync, createMemoryTrace, lintGraph, extractEmitterBindings, applyEmitterInstances, parseLayeredExtensions, applyEmitterInstancesFromExtension } from '../dist/index.js'; +import { + buildGraphAsync, + createMemoryTrace, + lintGraph, + extractEmitterBindings, + applyEmitterInstances, + parseLayeredExtensions, + applyEmitterInstancesFromExtension, + applyEnvironment, + applyListener, + mergeGraphSpecs, + loadAudioBuffer, +} from '../dist/index.js'; const { OfflineAudioContext } = wae; const __filename = fileURLToPath(import.meta.url); @@ -196,7 +208,7 @@ async function main() { } else if (json?.extensions?.KHR_audio_emitter) { // 2. Layered format (has extensions.KHR_audio_emitter) layeredResult = parseLayeredExtensions(json); - spec = layeredResult.graphs[0]; + spec = mergeGraphSpecs(layeredResult.graphs); // Resolve GENERATE_NOISE URIs for audio-buffer-source nodes for (const n of spec.nodes) { if (n.kind === 'audio-buffer-source' && n.params?.uri && typeof n.params.uri === 'string' && n.params.uri.startsWith('GENERATE_NOISE')) { @@ -241,6 +253,20 @@ async function main() { const ctx = new OfflineAudioContext(2, sr * 2, sr); const trace = createMemoryTrace(); const built = await buildGraphAsync(ctx, spec, trace); + let layeredDestination = ctx.destination; + if (layeredResult?.listener) { + applyListener(ctx, layeredResult.listener.listener, layeredResult.listener.transform, trace); + } + if (layeredResult?.environment) { + layeredDestination = await applyEnvironment( + ctx, + layeredResult.environment.environment, + built, + layeredResult.audioEmitter, + async (uri, audioContext) => loadAudioBuffer(audioContext, uri), + trace, + ); + } // If input was a layered format, expand emitter instances from parsed result if (layeredResult && layeredResult.emitterBindings.length > 0) { const audioEmitter = layeredResult.audioEmitter; @@ -252,7 +278,7 @@ async function main() { scale: b.scale, })).filter(b => !!b.emitter); const spatModel = layeredResult.listener?.listener?.spatializationModel; - applyEmitterInstancesFromExtension(built, resolved, spatModel, trace); + applyEmitterInstancesFromExtension(built, resolved, spatModel, trace, layeredDestination); } // If input was a legacy glTF with node-level emitter bindings, expand instances now if (json?.nodes && json?.extensions?.KHR_audio_graph && !json?.extensions?.KHR_audio_emitter) { diff --git a/examples/trace-same_emitter_multi_graph.txt b/examples/trace-same_emitter_multi_graph.txt new file mode 100644 index 0000000..017c486 --- /dev/null +++ b/examples/trace-same_emitter_multi_graph.txt @@ -0,0 +1,18 @@ +createOscillator id=g0__oscA +Oscillator[g0__oscA].type=sine +Oscillator[g0__oscA].frequency.setValueAtTime(440, 0) +Oscillator[g0__oscA].start(0) +createGain id=g0__gainA +Gain[g0__gainA].gain.setValueAtTime(0.300, 0) +createEmitterBus id=emitter_0 +createOscillator id=g1__oscB +Oscillator[g1__oscB].type=sine +Oscillator[g1__oscB].frequency.setValueAtTime(440, 0) +Oscillator[g1__oscB].start(0) +createGain id=g1__gainB +Gain[g1__gainB].gain.setValueAtTime(0.300, 0) +connect g0__oscA -> g0__gainA +connect g0__gainA -> emitter_0[in 0] +connect g1__oscB -> g1__gainB +connect g1__gainB -> emitter_0[in 0] +createEmitterInstanceExt id=emitter_0 -> gain -> destination \ No newline at end of file diff --git a/examples/trace-simple_emitter_only.txt b/examples/trace-simple_emitter_only.txt index d6d4156..23dd050 100644 --- a/examples/trace-simple_emitter_only.txt +++ b/examples/trace-simple_emitter_only.txt @@ -2,4 +2,5 @@ createEmitterBus id=emitter_0 createBufferSource id=src_0_e0 (uri) AudioBufferSource[src_0_e0].start(0) AudioBufferSource[src_0_e0].buffer <- uri(data:audio/wav;base64,UklGRiR3AQBXQVZFZm10IBAAAAABAAEAgLsAAAB3AQ…) -connect src_0_e0 -> emitter_0 \ No newline at end of file +connect src_0_e0 -> emitter_0 +createEmitterInstanceExt id=emitter_0 -> gain -> destination \ No newline at end of file diff --git a/examples/trace-spatial_with_environment.txt b/examples/trace-spatial_with_environment.txt index bae4752..69aff20 100644 --- a/examples/trace-spatial_with_environment.txt +++ b/examples/trace-spatial_with_environment.txt @@ -12,4 +12,7 @@ createEmitterBus id=emitter_0 connect src_0_g0[out 0] -> vol[in 0] connect vol -> lpf connect lpf -> emitter_0[in 0] +applyListener spatializationModel=HRTF +applyEnvironment type=parametric mix=0.300 +environment: parametric reverb decayTime=1.500 reflectionDelay=0.020 createEmitterInstanceExt id=emitter_0 -> panner+gain -> destination \ No newline at end of file diff --git a/package.json b/package.json index b394f4c..e0ac78a 100644 --- a/package.json +++ b/package.json @@ -44,7 +44,8 @@ "examples": "npm-run-all -s build example:osc example:osc-pwm example:gain example:biquad example:delay example:buffer example:convolver example:convolver-mix example:stereo example:panner example:splitter example:merger example:channel-mixer example:audio-mixer example:splitter-merger-mixer example:emitter example:complex", "dev": "npm run build && vite", "spec:validate": "node tools/spec-validate/validate.mjs", - "spec:validate:gltf": "node tools/spec-validate/validate-gltf.mjs" + "spec:validate:gltf": "node tools/spec-validate/validate-gltf.mjs", + "spec:validate:layered": "node tools/spec-validate/validate-layered.mjs examples/graphs-layered/*.json" }, "engines": { "node": ">=18" diff --git a/src/index.ts b/src/index.ts index c0391ca..e4e8696 100644 --- a/src/index.ts +++ b/src/index.ts @@ -7,6 +7,7 @@ export { createMemoryTrace } from './runtime/trace.js'; export { setBypass } from './runtime/bypassControl.js'; export { extractEmitterBindings } from './serialization/gltf-emitters.js'; export { applyEmitterInstances, applyEmitterInstancesFromExtension } from './runtime/emitters.js'; -export { parseLayeredExtensions } from './serialization/parse-layered.js'; +export { parseLayeredExtensions, mergeGraphSpecs } from './serialization/parse-layered.js'; export { applyEnvironment } from './runtime/environment.js'; export { applyListener } from './runtime/listener.js'; +export { loadAudioBuffer } from './assets/loadAudioBuffer.js'; diff --git a/src/runtime/emitters.ts b/src/runtime/emitters.ts index 38483cd..252952d 100644 --- a/src/runtime/emitters.ts +++ b/src/runtime/emitters.ts @@ -8,6 +8,10 @@ export interface ResolvedEmitterBinding { scale?: [number, number, number]; } +function radiansToDegrees(angleRadians: number): number { + return angleRadians * (180 / Math.PI); +} + // Minimal quat->forward vector util (glTF convention: forward = [0,0,-1]) function rotateVecByQuat(v: [number, number, number], q: [number, number, number, number]): [number, number, number] { const [x, y, z, w] = q; @@ -28,10 +32,12 @@ export function applyEmitterInstances( built: BuiltGraph, spec: GraphSpec, bindings: ResolvedEmitterBinding[], - trace?: TraceLogger + trace?: TraceLogger, + destination?: AudioNode ) { const ctx = built.context as any; const inputs = built._inputs!; + const targetDestination = destination ?? ctx.destination; for (const b of bindings) { const bus = inputs.get(b.emitterNodeId); @@ -54,8 +60,8 @@ export function applyEmitterInstances( if (typeof att.refDistance === 'number') pan.refDistance = att.refDistance; if (typeof att.maxDistance === 'number') pan.maxDistance = att.maxDistance; if (typeof att.rolloffFactor === 'number') pan.rolloffFactor = att.rolloffFactor; - if (typeof att.coneInnerAngle === 'number') pan.coneInnerAngle = att.coneInnerAngle; - if (typeof att.coneOuterAngle === 'number') pan.coneOuterAngle = att.coneOuterAngle; + if (typeof att.coneInnerAngle === 'number') pan.coneInnerAngle = radiansToDegrees(att.coneInnerAngle); + if (typeof att.coneOuterAngle === 'number') pan.coneOuterAngle = radiansToDegrees(att.coneOuterAngle); if (typeof att.coneOuterGain === 'number') pan.coneOuterGain = att.coneOuterGain; // Apply transform: position from translation; orientation from rotation if (b.translation && (pan as any).positionX) { @@ -85,7 +91,7 @@ export function applyEmitterInstances( bus.connect(postGain); trace?.log?.(`createEmitterInstance id=${b.emitterNodeId} -> gain -> destination`); } - postGain.connect(ctx.destination); + postGain.connect(targetDestination); } } @@ -102,9 +108,11 @@ export function applyEmitterInstancesFromExtension( bindings: ExtensionEmitterBinding[], defaultSpatializationModel?: string, trace?: TraceLogger, + destination?: AudioNode, ) { const ctx = built.context as any; const inputs = built._inputs!; + const targetDestination = destination ?? ctx.destination; for (const b of bindings) { const bus = inputs.get(b.emitterNodeId); @@ -130,8 +138,8 @@ export function applyEmitterInstancesFromExtension( if (typeof pos.refDistance === 'number') pan.refDistance = pos.refDistance; if (typeof pos.maxDistance === 'number') pan.maxDistance = pos.maxDistance; if (typeof pos.rolloffFactor === 'number') pan.rolloffFactor = pos.rolloffFactor; - if (typeof pos.coneInnerAngle === 'number') pan.coneInnerAngle = pos.coneInnerAngle; - if (typeof pos.coneOuterAngle === 'number') pan.coneOuterAngle = pos.coneOuterAngle; + if (typeof pos.coneInnerAngle === 'number') pan.coneInnerAngle = radiansToDegrees(pos.coneInnerAngle); + if (typeof pos.coneOuterAngle === 'number') pan.coneOuterAngle = radiansToDegrees(pos.coneOuterAngle); if (typeof pos.coneOuterGain === 'number') pan.coneOuterGain = pos.coneOuterGain; } @@ -158,6 +166,6 @@ export function applyEmitterInstancesFromExtension( trace?.log?.(`createEmitterInstanceExt id=${b.emitterNodeId} -> gain -> destination`); } - postGain.connect(ctx.destination); + postGain.connect(targetDestination); } } diff --git a/src/runtime/environment.ts b/src/runtime/environment.ts index 3b3f4ff..b86a718 100644 --- a/src/runtime/environment.ts +++ b/src/runtime/environment.ts @@ -8,21 +8,21 @@ export async function applyEnvironment( audioEmitter?: KHRAudioEmitterExtension, loadBuffer?: (uri: string, ctx: BaseAudioContext) => Promise, trace?: TraceLogger, -): Promise { +): Promise { const reverb = environment.reverb; + const ctx = context as any; + if (!reverb) { - trace?.log?.(`applyEnvironment: no reverb configured`); - return; + trace?.log?.('applyEnvironment: no reverb configured'); + return ctx.destination; } const mix = reverb.mix ?? 0.5; trace?.log?.(`applyEnvironment type=${reverb.type ?? 'parametric'} mix=${mix}`); - // Find the final output node (last emitter or last node connected to destination) - // We insert between graph output and destination using a wet/dry parallel path - const ctx = context as any; + const input: GainNode = ctx.createGain(); + input.gain.value = 1.0; - // Create wet/dry mix nodes const dryGain: GainNode = ctx.createGain(); dryGain.gain.value = 1.0 - mix; @@ -34,17 +34,21 @@ export async function applyEnvironment( let reverbNode: AudioNode; + input.connect(dryGain); + dryGain.connect(outputMerge); + if (reverb.type === 'impulseResponse' && typeof reverb.audio === 'number' && audioEmitter) { - // IR-based reverb using ConvolverNode const convolver: ConvolverNode = ctx.createConvolver(); const audioData = audioEmitter.audio[reverb.audio]; if (audioData?.uri && loadBuffer) { convolver.buffer = await loadBuffer(audioData.uri, context); trace?.log?.(`environment: loaded IR from audio[${reverb.audio}]`); } + input.connect(convolver); + convolver.connect(wetGain); reverbNode = convolver; + trace?.log?.('environment: IR reverb applied'); } else { - // Parametric reverb approximation using a simple delay + feedback chain const decayTime = reverb.decayTime ?? 1.5; const reflectionDelay = reverb.reflectionDelay ?? 0.02; const reverbDelay = reverb.reverbDelay ?? 0.04; @@ -52,43 +56,26 @@ export async function applyEnvironment( const delay: DelayNode = ctx.createDelay(Math.max(decayTime, 1.0)); delay.delayTime.value = reflectionDelay + reverbDelay; - // Simple feedback approximation const feedbackGain: GainNode = ctx.createGain(); - // Convert decay time to feedback gain (approximate RT60) const loopTime = reflectionDelay + reverbDelay; feedbackGain.gain.value = loopTime > 0 ? Math.pow(0.001, loopTime / Math.max(decayTime, 0.01)) : 0; - // Early reflections gain const earlyGain: GainNode = ctx.createGain(); earlyGain.gain.value = reverb.earlyReflectionsGain ?? 1.0; - // Simple chain: input → delay → feedback loop, delay → earlyGain → output + input.connect(delay); delay.connect(feedbackGain); feedbackGain.connect(delay); delay.connect(earlyGain); - - reverbNode = delay; - // For parametric, the output comes from earlyGain earlyGain.connect(wetGain); - // Wire dry + wet to output - dryGain.connect(outputMerge); - wetGain.connect(outputMerge); - outputMerge.connect(ctx.destination); + reverbNode = delay; trace?.log?.(`environment: parametric reverb decayTime=${decayTime} reflectionDelay=${reflectionDelay}`); - - // Store the environment nodes on the built graph for reference - (builtGraph as any)._environment = { dryGain, wetGain, reverbNode: delay, outputMerge }; - return; } - // For IR-based reverb, wire: input → dry → outputMerge, input → reverb → wet → outputMerge - reverbNode.connect(wetGain); - dryGain.connect(outputMerge); wetGain.connect(outputMerge); outputMerge.connect(ctx.destination); - trace?.log?.(`environment: IR reverb applied`); - - (builtGraph as any)._environment = { dryGain, wetGain, reverbNode, outputMerge }; + builtGraph._environment = { input, dryGain, wetGain, reverbNode, outputMerge }; + return input; } diff --git a/src/runtime/lint.ts b/src/runtime/lint.ts index 11326e5..64164fa 100644 --- a/src/runtime/lint.ts +++ b/src/runtime/lint.ts @@ -32,7 +32,7 @@ export function lintGraph(spec: GraphSpec): LintResult { // Emitter degree for (const n of spec.nodes) { if (n.kind === 'emitter') { - if ((indeg.get(n.id) || 0) !== 1) errors.push(`Emitter ${n.id} must have exactly one input`); + if ((indeg.get(n.id) || 0) < 1) errors.push(`Emitter ${n.id} must have at least one input`); if ((outdeg.get(n.id) || 0) !== 0) errors.push(`Emitter ${n.id} must have zero outputs`); } } diff --git a/src/serialization/parse-layered.ts b/src/serialization/parse-layered.ts index 40c138f..ce75384 100644 --- a/src/serialization/parse-layered.ts +++ b/src/serialization/parse-layered.ts @@ -269,6 +269,18 @@ export function parseLayeredExtensions(gltf: GltfDocument): LayeredParseResult { } } + // Extract scene-level emitter bindings for global emitters. + const gltfScenes = gltf.scenes || []; + for (let i = 0; i < gltfScenes.length; i++) { + const sceneEmitterExt = gltfScenes[i].extensions?.KHR_audio_emitter; + if (!sceneEmitterExt?.emitters) continue; + for (const emitterId of sceneEmitterExt.emitters) { + if (typeof emitterId === 'number') { + emitterBindings.push({ sceneIndex: i, emitterId }); + } + } + } + // Extract listener from KHR_audio_environment on nodes let listenerResult: LayeredParseResult['listener']; if (audioEnv?.listeners) { @@ -318,3 +330,78 @@ export function parseLayeredExtensions(gltf: GltfDocument): LayeredParseResult { audioEmitter, }; } + +export function mergeGraphSpecs(specs: GraphSpec[]): GraphSpec { + if (specs.length === 0) { + return { nodes: [], connections: [] }; + } + if (specs.length === 1) { + return specs[0]; + } + + const nodes: GraphNodeSpec[] = []; + const connections: GraphConnectionSpec[] = []; + const outputs: string[] = []; + const emitterNodes = new Map(); + + function mapNodeId(specIndex: number, node: GraphNodeSpec): string { + if (node.kind === 'emitter') { + return node.id; + } + return `g${specIndex}__${node.id}`; + } + + for (let specIndex = 0; specIndex < specs.length; specIndex++) { + const spec = specs[specIndex]; + const idMap = new Map(); + + for (const node of spec.nodes) { + const mappedId = mapNodeId(specIndex, node); + idMap.set(node.id, mappedId); + + if (node.kind === 'emitter') { + const existing = emitterNodes.get(mappedId); + if (!existing) { + const cloned = { ...node, params: node.params ? { ...node.params } : undefined }; + emitterNodes.set(mappedId, cloned); + nodes.push(cloned); + } + continue; + } + + nodes.push({ + ...node, + id: mappedId, + params: node.params ? { ...node.params } : undefined, + }); + } + + for (const connection of spec.connections) { + connections.push({ + from: { + node: idMap.get(connection.from.node) ?? connection.from.node, + output: connection.from.output, + }, + to: { + node: idMap.get(connection.to.node) ?? connection.to.node, + input: connection.to.input, + }, + }); + } + + if (spec.outputs) { + for (const output of spec.outputs) { + const mappedOutput = idMap.get(output) ?? output; + if (!outputs.includes(mappedOutput)) { + outputs.push(mappedOutput); + } + } + } + } + + return { + nodes, + connections, + outputs: outputs.length > 0 ? outputs : undefined, + }; +} diff --git a/src/types.ts b/src/types.ts index b254593..9c8ae28 100644 --- a/src/types.ts +++ b/src/types.ts @@ -49,6 +49,13 @@ export interface BuiltGraph { _inputs?: Map; _outputs?: Map; _bypass?: Map; + _environment?: { + input: AudioNode; + dryGain: GainNode; + wetGain: GainNode; + reverbNode: AudioNode; + outputMerge: AudioNode; + }; } // --------------------------------------------------------------------------- @@ -271,7 +278,8 @@ export interface LayeredParseResult { graphs: GraphSpec[]; /** Emitter binding info extracted from glTF nodes */ emitterBindings: { - nodeIndex: number; + nodeIndex?: number; + sceneIndex?: number; emitterId: number; translation?: [number, number, number]; rotation?: [number, number, number, number]; diff --git a/tests/emitters-extension.test.ts b/tests/emitters-extension.test.ts index 7ea89fd..3cea0fa 100644 --- a/tests/emitters-extension.test.ts +++ b/tests/emitters-extension.test.ts @@ -117,8 +117,8 @@ describe('applyEmitterInstancesFromExtension', () => { ]); const panner = ctx.createPanner.mock.results[0].value; - expect(panner.coneInnerAngle).toBe(1.57); - expect(panner.coneOuterAngle).toBe(3.14); + expect(panner.coneInnerAngle).toBeCloseTo(89.954, 2); + expect(panner.coneOuterAngle).toBeCloseTo(179.909, 2); expect(panner.coneOuterGain).toBe(0.2); }); @@ -171,4 +171,22 @@ describe('applyEmitterInstancesFromExtension', () => { expect(trace.log).toHaveBeenCalledWith('warn: emitter bus not found for nonexistent'); expect(ctx.createGain).not.toHaveBeenCalled(); }); + + it('uses provided destination node instead of context destination', () => { + const ctx = mockContext(); + const built = mockBuiltGraph(ctx, 'emitter_0'); + const emitter: AudioEmitter = { type: 'global', gain: 0.7, sources: [] }; + const customDestination = { connect: vi.fn() } as any; + + applyEmitterInstancesFromExtension( + built, + [{ emitterNodeId: 'emitter_0', emitter }], + undefined, + undefined, + customDestination, + ); + + const postGain = ctx.createGain.mock.results[0].value; + expect(postGain.connect).toHaveBeenCalledWith(customDestination); + }); }); diff --git a/tests/environment.test.ts b/tests/environment.test.ts index f5d1d19..730b7e4 100644 --- a/tests/environment.test.ts +++ b/tests/environment.test.ts @@ -60,12 +60,13 @@ describe('applyEnvironment', () => { }; const trace = { log: vi.fn(), getLines: () => [] }; - await applyEnvironment(ctx, env, built, undefined, undefined, trace); + const input = await applyEnvironment(ctx, env, built, undefined, undefined, trace); - // Should have created gain nodes for dry/wet/output + delay + // Should have created input + dry/wet/output gains plus delay expect(ctx.createGain).toHaveBeenCalled(); expect(ctx.createDelay).toHaveBeenCalled(); expect((built as any)._environment).toBeDefined(); + expect(input).toBe((built as any)._environment.input); expect(trace.log).toHaveBeenCalled(); }); @@ -87,10 +88,11 @@ describe('applyEnvironment', () => { const fakeBuffer = {} as AudioBuffer; const loadBuffer = vi.fn().mockResolvedValue(fakeBuffer); - await applyEnvironment(ctx, env, built, audioEmitter, loadBuffer); + const input = await applyEnvironment(ctx, env, built, audioEmitter, loadBuffer); expect(ctx.createConvolver).toHaveBeenCalled(); expect(loadBuffer).toHaveBeenCalledWith('ir.wav', ctx); + expect(input).toBe((built as any)._environment.input); }); it('wet/dry mix: dry gain = 1-mix, wet gain = mix', async () => { @@ -108,22 +110,39 @@ describe('applyEnvironment', () => { await applyEnvironment(ctx, env, built); - // First gain is dry, second is wet - const dryGain = gains[0]; - const wetGain = gains[1]; + // Gains are created in order: input, dry, wet, output merge, feedback, early reflections + const dryGain = gains[1]; + const wetGain = gains[2]; expect(dryGain.gain.value).toBeCloseTo(0.7, 5); expect(wetGain.gain.value).toBeCloseTo(0.3, 5); }); + it('routes audio through environment input instead of allocating detached nodes', async () => { + const ctx = mockContext(); + const built = mockBuiltGraph(ctx); + const env: Environment = { + reverb: { type: 'parametric', mix: 0.4 }, + }; + + const input = await applyEnvironment(ctx, env, built); + const environmentState = (built as any)._environment; + + expect(environmentState).toBeDefined(); + expect(environmentState.input).toBe(input); + expect(environmentState.input.connect).toHaveBeenCalled(); + expect(environmentState.outputMerge.connect).toHaveBeenCalledWith(ctx.destination); + }); + it('no-op when no reverb configured', async () => { const ctx = mockContext(); const built = mockBuiltGraph(ctx); const env: Environment = {}; const trace = { log: vi.fn(), getLines: () => [] }; - await applyEnvironment(ctx, env, built, undefined, undefined, trace); + const input = await applyEnvironment(ctx, env, built, undefined, undefined, trace); expect(ctx.createGain).not.toHaveBeenCalled(); expect(trace.log).toHaveBeenCalledWith('applyEnvironment: no reverb configured'); + expect(input).toBe(ctx.destination); }); }); diff --git a/tests/layered-example-fixtures.test.ts b/tests/layered-example-fixtures.test.ts new file mode 100644 index 0000000..4450bc6 --- /dev/null +++ b/tests/layered-example-fixtures.test.ts @@ -0,0 +1,162 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { describe, it, expect } from 'vitest'; +import { buildGraphAsync } from '../src/runtime/buildGraphAsync'; +import { createMemoryTrace } from '../src/runtime/trace'; +import { parseLayeredExtensions, mergeGraphSpecs } from '../src/serialization/parse-layered'; +import { applyEmitterInstancesFromExtension } from '../src/runtime/emitters'; +import { applyListener } from '../src/runtime/listener'; +import { applyEnvironment } from '../src/runtime/environment'; +import { loadAudioBuffer } from '../src/assets/loadAudioBuffer'; +import type { GltfDocument } from '../src/types'; + +function readFixture(name: string): GltfDocument { + const file = path.resolve(__dirname, '..', 'examples', 'graphs-layered', name); + return JSON.parse(fs.readFileSync(file, 'utf-8')); +} + +function floatTo16BitPCM(float32: number): number { + const s = Math.max(-1, Math.min(1, float32)); + return s < 0 ? s * 0x8000 : s * 0x7fff; +} + +function writeWavPCM16LE(samples: Float32Array, sampleRate: number, numChannels: number): Buffer { + const bytesPerSample = 2; + const blockAlign = numChannels * bytesPerSample; + const byteRate = sampleRate * blockAlign; + const dataSize = samples.length * bytesPerSample; + const buffer = Buffer.alloc(44 + dataSize); + let offset = 0; + buffer.write('RIFF', offset); offset += 4; + buffer.writeUInt32LE(36 + dataSize, offset); offset += 4; + buffer.write('WAVE', offset); offset += 4; + buffer.write('fmt ', offset); offset += 4; + buffer.writeUInt32LE(16, offset); offset += 4; + buffer.writeUInt16LE(1, offset); offset += 2; + buffer.writeUInt16LE(numChannels, offset); offset += 2; + buffer.writeUInt32LE(sampleRate, offset); offset += 4; + buffer.writeUInt32LE(byteRate, offset); offset += 4; + buffer.writeUInt16LE(blockAlign, offset); offset += 2; + buffer.writeUInt16LE(16, offset); offset += 2; + buffer.write('data', offset); offset += 4; + buffer.writeUInt32LE(dataSize, offset); offset += 4; + for (let i = 0; i < samples.length; i++) { + buffer.writeInt16LE(floatTo16BitPCM(samples[i]), 44 + i * 2); + } + return buffer; +} + +function hashString(s: string): number { + let h = 2166136261 >>> 0; + for (let i = 0; i < s.length; i++) { + h ^= s.charCodeAt(i); + h = Math.imul(h, 16777619); + } + return h >>> 0; +} + +function mulberry32(seed: number): () => number { + let t = seed >>> 0; + return function () { + t += 0x6D2B79F5; + let r = Math.imul(t ^ (t >>> 15), 1 | t); + r ^= r + Math.imul(r ^ (r >>> 7), 61 | r); + return ((r ^ (r >>> 14)) >>> 0) / 4294967296; + }; +} + +function makeNoiseDataUri(seedBase: string, seconds = 1.0, amp = 0.5, sampleRate = 48000): string { + const length = Math.floor(sampleRate * seconds); + const rng = mulberry32(hashString(seedBase)); + const samples = new Float32Array(length); + for (let i = 0; i < length; i++) { + samples[i] = (rng() * 2 - 1) * amp; + } + const wav = writeWavPCM16LE(samples, sampleRate, 1); + return `data:audio/wav;base64,${wav.toString('base64')}`; +} + +function resolveGeneratedNoise(gltf: GltfDocument, seedBase: string): void { + const audio = gltf.extensions?.KHR_audio_emitter?.audio ?? []; + for (const entry of audio) { + if (typeof entry.uri === 'string' && entry.uri.startsWith('GENERATE_NOISE')) { + const parts = entry.uri.split(':'); + const seconds = parts[1] ? Number(parts[1]) : 1.0; + const amp = parts[2] ? Number(parts[2]) : 0.5; + entry.uri = makeNoiseDataUri(seedBase, seconds, amp); + } + } +} + +async function renderLayeredFixture(name: string) { + const gltf = readFixture(name); + resolveGeneratedNoise(gltf, name); + const layered = parseLayeredExtensions(gltf); + const spec = mergeGraphSpecs(layered.graphs); + const sampleRate = spec.sampleRate || 48000; + const ctx = new OfflineAudioContext(2, sampleRate * 2, sampleRate); + const trace = createMemoryTrace(); + const built = await buildGraphAsync(ctx, spec, trace); + let destination: AudioNode = ctx.destination; + + if (layered.listener) { + applyListener(ctx, layered.listener.listener, layered.listener.transform, trace); + } + if (layered.environment) { + destination = await applyEnvironment( + ctx, + layered.environment.environment, + built, + layered.audioEmitter, + async (uri, audioContext) => loadAudioBuffer(audioContext, uri), + trace, + ); + } + + if (layered.emitterBindings.length > 0) { + const resolved = layered.emitterBindings.map((binding) => ({ + emitterNodeId: `emitter_${binding.emitterId}`, + emitter: layered.audioEmitter.emitters[binding.emitterId], + translation: binding.translation, + rotation: binding.rotation, + scale: binding.scale, + })); + applyEmitterInstancesFromExtension( + built, + resolved, + layered.listener?.listener?.spatializationModel, + trace, + destination, + ); + } + + const rendered = await ctx.startRendering(); + const left = rendered.getChannelData(0); + let sum = 0; + for (let i = 0; i < left.length; i++) sum += left[i] * left[i]; + const rms = Math.sqrt(sum / left.length); + + return { rms, trace: trace.getLines(), spec }; +} + +describe('layered glTF fixture integration', () => { + it('renders simple-emitter-only fixture with actual KHR_audio_emitter bindings', async () => { + const rendered = await renderLayeredFixture('simple-emitter-only.json'); + expect(rendered.rms).toBeGreaterThan(0.01); + expect(rendered.trace.some((line) => line.includes('createEmitterInstanceExt id=emitter_0'))).toBe(true); + }); + + it('renders spatial-with-environment fixture and applies listener/environment', async () => { + const rendered = await renderLayeredFixture('spatial-with-environment.json'); + expect(rendered.rms).toBeGreaterThan(0.001); + expect(rendered.trace.some((line) => line.includes('applyListener spatializationModel=HRTF'))).toBe(true); + expect(rendered.trace.some((line) => line.includes('applyEnvironment type=parametric mix=0.3'))).toBe(true); + }); + + it('renders same-emitter-multi-graph fixture and mixes graphs on the shared emitter bus', async () => { + const mixed = await renderLayeredFixture('same-emitter-multi-graph.json'); + + expect(mixed.spec.connections.filter((connection) => connection.to.node === 'emitter_0')).toHaveLength(2); + expect(mixed.rms).toBeGreaterThan(0.35); + }); +}); diff --git a/tests/parse-layered.test.ts b/tests/parse-layered.test.ts index bb7b661..9ede68b 100644 --- a/tests/parse-layered.test.ts +++ b/tests/parse-layered.test.ts @@ -1,5 +1,5 @@ import { describe, it, expect } from 'vitest'; -import { parseLayeredExtensions } from '../src/serialization/parse-layered'; +import { parseLayeredExtensions, mergeGraphSpecs } from '../src/serialization/parse-layered'; import type { GltfDocument } from '../src/types'; describe('parseLayeredExtensions', () => { @@ -137,6 +137,30 @@ describe('parseLayeredExtensions', () => { expect(result.graphs).toHaveLength(2); }); + it('mergeGraphSpecs preserves shared emitter buses for additive mixing', () => { + const merged = mergeGraphSpecs([ + { + nodes: [ + { id: 'gainA', kind: 'gain', params: { gain: 1.0 } }, + { id: 'emitter_0', kind: 'emitter', params: { emitterType: 'global', gain: 1.0 } }, + ], + connections: [{ from: { node: 'gainA' }, to: { node: 'emitter_0' } }], + }, + { + nodes: [ + { id: 'gainB', kind: 'gain', params: { gain: 0.5 } }, + { id: 'emitter_0', kind: 'emitter', params: { emitterType: 'global', gain: 1.0 } }, + ], + connections: [{ from: { node: 'gainB' }, to: { node: 'emitter_0' } }], + }, + ]); + + expect(merged.nodes.filter(n => n.id === 'emitter_0')).toHaveLength(1); + expect(merged.connections.filter(c => c.to.node === 'emitter_0')).toHaveLength(2); + expect(merged.nodes.some(n => n.id === 'g0__gainA')).toBe(true); + expect(merged.nodes.some(n => n.id === 'g1__gainB')).toBe(true); + }); + it('handles multiple inputs/outputs per graph', () => { const gltf: GltfDocument = { extensionsUsed: ['KHR_audio_emitter', 'KHR_audio_graph'], @@ -281,6 +305,25 @@ describe('parseLayeredExtensions', () => { expect(result.environment!.sceneIndex).toBe(0); }); + it('extracts scene-level emitter bindings from KHR_audio_emitter on scenes', () => { + const gltf: GltfDocument = { + extensionsUsed: ['KHR_audio_emitter'], + extensions: { + KHR_audio_emitter: { + audio: [{ uri: 'test.wav' }], + sources: [{ audio: 0, gain: 0.5, autoplay: true }], + emitters: [{ type: 'global', gain: 1.0, sources: [0] }], + }, + }, + scenes: [{ extensions: { KHR_audio_emitter: { emitters: [0] } } }], + }; + + const result = parseLayeredExtensions(gltf); + expect(result.emitterBindings).toHaveLength(1); + expect(result.emitterBindings[0].sceneIndex).toBe(0); + expect(result.emitterBindings[0].emitterId).toBe(0); + }); + it('throws when KHR_audio_emitter is missing', () => { const gltf = { extensionsUsed: [] } as any; expect(() => parseLayeredExtensions(gltf)).toThrow('Missing required extension: KHR_audio_emitter'); diff --git a/tools/spec-validate/validate-layered.mjs b/tools/spec-validate/validate-layered.mjs index 2c2c639..5c44292 100644 --- a/tools/spec-validate/validate-layered.mjs +++ b/tools/spec-validate/validate-layered.mjs @@ -5,6 +5,19 @@ import fs from 'node:fs'; import path from 'node:path'; +function expandInput(arg) { + if (!arg.includes('*')) return [arg]; + const normalized = path.resolve(arg); + const dir = path.dirname(normalized); + const pattern = path.basename(normalized) + .replace(/[.+^${}()|[\]\\]/g, '\\$&') + .replace(/\*/g, '.*'); + const regex = new RegExp(`^${pattern}$`); + return fs.readdirSync(dir) + .filter((entry) => regex.test(entry)) + .map((entry) => path.join(dir, entry)); +} + function validateAudioEmitter(ext) { const errors = []; if (!Array.isArray(ext.audio)) errors.push('KHR_audio_emitter.audio must be an array'); @@ -132,7 +145,7 @@ function validateEnvironment(ext) { } function main() { - const files = process.argv.slice(2); + const files = process.argv.slice(2).flatMap(expandInput); if (files.length === 0) { console.log('Usage: node tools/spec-validate/validate-layered.mjs [file2.json ...]'); process.exit(1);