[Product Gap] Add real-audio MIR accuracy acceptance benchmarks #770

Buyer-visible gap

BandScope has strong unit, contract, parity, and security gates, but the protected merge path does not yet prove that an actual audio file traversing the production intake and analysis pipeline yields the expected musical result. A buyer cannot distinguish “the code executes” from “the rehearsal guidance is measurably accurate.”

This issue establishes a reproducible, rights-safe real-audio acceptance layer for harmony, tempo/beat, structure, source separation, role range, and rehearsal cue accuracy. The test must exercise decoded PCM audio through the same public analysis boundary used by the desktop product; mocked feature matrices alone are not acceptance evidence.

Product outcome

Every release candidate publishes an accuracy manifest answering:

• what audio was analyzed;
• what ground truth was expected;
• which exact engine/backend/version produced the result;
• which recognized MIR metric was used;
• whether accuracy improved, regressed, or remained within the registered tolerance;
• whether CPU and GPU paths agree where both exist; and
• which known limitations require human confirmation in the rehearsal UI.

Test tiers

Tier 1 — deterministic redistributable PCM fixtures

Generate or check in tiny, license-clean WAV/FLAC fixtures with immutable SHA-256 manifests. These must be real decoded waveforms, not direct chroma/onset arrays. Tier 1 is a deterministic unit/regression floor; it is not sufficient commercial/scientific acceptance evidence by itself.

Required cases:

• sustained single notes across instrument/vocal register boundaries;
• major, minor, seventh, no-chord, inversion, and enharmonic chord segments;
• a known chord progression with exact change times;
• stable tempo, half/double-tempo ambiguity, swing, pickup, stop, and tempo-change cases;
• verse/chorus/bridge-style structural boundaries and repeated sections;
• bass, vocal, drums, and harmonic stems mixed at known gains;
• simultaneous roles whose ranges overlap and then separate;
• silence, clipping, low level, short input, sample-rate conversion, mono/stereo, and bounded background noise.

The fixture generator, waveform parameters, annotations, checksums, and expected metric ranges must be versioned. Generation should be deterministic and independent of network access.

Tier 2 — redistributable public corpus slice

Use only audio whose redistribution and automated evaluation rights are documented. Record the license, source URL/DOI, exact file hash, annotation provenance, split, and any transformation. Do not commit copyrighted commercial recordings or annotations that require unavailable audio.

Tier 3 — private commercial-readiness benchmark

Run a scheduled/manual benchmark against a separately licensed private corpus. Store only aggregate metrics, bounded error exemplars, configuration hashes, and provenance-safe artifacts in GitHub. The workflow must fail closed when the corpus credential or manifest is absent and must never substitute synthetic evidence while claiming private-corpus success.

Metrics and acceptance contracts

Harmony

• Report segment-based chord symbol recall and corpus-duration-weighted WCSR.
• Report at least root, major/minor, and seventh-vocabulary mappings rather than one opaque score.
• Preserve no-chord handling and exact annotation/estimate time intervals.
• Add boundary error summaries so a temporally shifted but harmonically correct output is visible.

Beat and tempo

• Report beat precision, recall, and F-measure with the registered evaluation implementation and tolerance. The default acceptance profile uses mir_eval.beat.f_measure semantics with the documented 0.07 s (±70 ms) window; any alternative tolerance must be separately named and must not replace the default silently.
• Report continuity-aware beat metrics where applicable so a high local F-measure cannot hide long-run tracking failures.
• Report tempo Acc1 and Acc2 separately so octave-related errors are not hidden.
• Include explicit half-time/double-time fixtures and confidence calibration.
• Treat Chiu et al. (2025) as contemporary peer-reviewed evidence that ±70 ms remains conventional in beat-tracking work, while the pinned mir_eval implementation and Raffel et al. (2014) remain the executable metric authority for this repository.

Structure

• Report boundary precision/recall/F-measure at documented strict and relaxed windows.
• Report a segment-label agreement metric for repeated verse/chorus/bridge regions.
• Verify section ordering, repeated-section identity, and pickup/tag preservation.

Source separation

• Report SI-SDR per stem as the primary scale-invariant objective separation metric using the pinned repository implementation and Le Roux et al. (2019) definition.
• Where comparison with MUSDB/SiSEC-style literature is required, also report the repository-pinned BSS Eval v4 / museval-compatible SDR as a separately labelled compatibility metric. Do not substitute BSS Eval v4 SDR for SI-SDR or collapse the two into one score.
• Record stem-specific artifacts and add a bounded human-listening validation protocol before making perceptual-quality claims; neither SI-SDR nor BSS Eval v4 alone establishes perceptual superiority.
• Verify mixture consistency and absence of NaN/Inf outputs.

Rehearsal-specific outputs

• Range endpoints: note/semitone error and exact out-of-range classification.
• Entry/dropout/stop/pickup cues: event precision, recall, and timing error.
• Role overlap: interval IoU or equivalent overlap error against known role activity.
• Confidence: calibration curve/Brier-style score where a probabilistic output exists.

Regression policy

• Establish the first protected baseline without inventing unsupported “industry standard” thresholds.
• Thereafter fail the protected gate on a statistically or practically meaningful regression defined per metric and fixture family.
• Publish bootstrap or repeated-run uncertainty for nondeterministic stages.
• Keep dataset-level and track-level metrics so aggregate improvements cannot hide severe song-specific failures.
• Require an explicit reviewed waiver with expiry for any accepted regression; no silent threshold reduction.

CPU/GPU and implementation boundary

• Production numeric kernels remain Rust-owned where the corresponding production path is Rust-backed.
• Run CPU reference and available GPU backend on the same exact fixtures and compare outputs/metrics within registered numerical tolerances.
• Do not skip a configured GPU acceptance lane and report it as passing.
• Minimize thread oversubscription and record backend, thread count, device, precision, peak RSS/VRAM, and elapsed time in the manifest.

Security and rights notes

• Treat audio, annotations, metadata, decoders, model artifacts, and benchmark manifests as untrusted input.
• Enforce bounded duration, channels, sample rate, decoded size, file count, and output artifact size.
• Use no shell interpolation or generic execution capability.
• Never publish private audio, absolute local paths, credentials, or copyrighted excerpts in logs/artifacts.
• Keep local-first runtime behavior; the acceptance workflow may use separately authorized benchmark storage but must not add a network dependency to ordinary analysis.

Required repository changes

• Add an accuracy manifest schema and deterministic fixture manifest.
• Add a single documented command for the complete acceptance run.
• Add permanent tests proving actual file decode and end-to-end analysis, metric correctness, malformed-manifest failure, checksum mismatch failure, and report determinism.
• Add production statement and branch coverage 100% plus public docstrings 100% for new repository-owned code.
• Add docs/doctoring/real-audio-accuracy-acceptance.md with metric definitions, claim boundaries, licenses, operational interpretation, rollback policy, and APA 7th references.
• Update AGENTS.md, ARCHITECTURE.md, CHANGELOG.md, and release acceptance documentation.
• Export machine-readable JSON and accessible HTML exact-value reports.

Merge and release gate

Do not claim release readiness from unit tests alone after this acceptance layer exists. Commercial/scientific acceptance requires Tier 2 and/or separately licensed Tier 3 rights-cleared real recordings traversing the production decode → MIR → rehearsal-insight boundary; Tier 1 synthetic/deterministic fixtures cannot substitute for that evidence. Merge the implementation only after exact-head repository CI, central coverage, security/supply-chain checks, realistic audio acceptance, CPU/GPU parity where configured, current-head automated review, zero unresolved threads, and qualifying independent approval all succeed without bypass.

References — APA 7th

Chiu, C.-Y., Liu, L., Weiß, C., & Müller, M. (2025). Cross-modal approaches to beat tracking: A case study on Chopin Mazurkas. Transactions of the International Society for Music Information Retrieval, 8(1), 55–69. https://doi.org/10.5334/tismir.238

Le Roux, J., Wisdom, S., Erdogan, H., & Hershey, J. R. (2019). SDR—Half-baked or well done? In 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://doi.org/10.1109/ICASSP.2019.8683855

Odekerken, D., Koops, H. V., & Volk, A. (2021). Improving audio chord estimation by alignment and integration of crowd-sourced symbolic music. Transactions of the International Society for Music Information Retrieval, 4(1), 141–155. https://doi.org/10.5334/tismir.81

Raffel, C., McFee, B., Humphrey, E. J., Salamon, J., Nieto, O., Liang, D., & Ellis, D. P. W. (2014). MIR_EVAL: A transparent implementation of common MIR metrics. In Proceedings of the 15th International Society for Music Information Retrieval Conference (pp. 367–372).

Schreiber, H., & Müller, M. (2020). Music tempo estimation: Are we done yet? Transactions of the International Society for Music Information Retrieval, 3(1), 111–125. https://doi.org/10.5334/tismir.43

Stöter, F.-R., Liutkus, A., & Ito, N. (2018). The 2018 signal separation evaluation campaign. In Latent Variable Analysis and Signal Separation (pp. 293–305). https://doi.org/10.1007/978-3-319-93764-9_28

Claim boundary

Passing this issue’s benchmark will support specific, versioned accuracy claims on the registered fixtures and corpora. It will not establish universal musical correctness, genre/culture invariance, or perceptual superiority without separate representative data and human validation.