feat(mobile): show what a clip sounds like - #262
Conversation
An audio clip was a coloured block with a filename on it, and a video clip showed frames and nothing of its sound. Either way there was nothing to aim at: finding a beat, a word or a gap meant scrubbing and listening, one guess at a time. Video clips got frames in #246 for exactly this reason; sound never did. Peaks are decoded, because nothing in a file's metadata says where it is loud. Both readers take the clip's own window and fold samples into buckets as they arrive, so a long file costs a pass rather than a copy of itself in memory, and both take the peak per bucket rather than the average — the loudest moment in a bar is what the eye is looking for, and averaging a busy passage flattens it into a wall. Drawn on video clips too, and for the better reason: the phone cannot import an audio file at all, so almost every sound in a cut is a video clip's own. There it is a strip along the bottom of the frames — the frames say which shot this is, the waveform says where the words are. Normalised against the loudest bar rather than full scale, or a quiet recording draws as a flat line and says "no sound here" about a clip that has plenty. Silence keeps a hairline instead of a gap, so a clip never looks like it failed to load. Best-effort like the thumbnails: queued one at a time, cached at module level because a clip is unmounted every time it scrolls out of view, and a file that will not decode leaves the clip exactly as it was. Verified on the emulator with a clip that is silent for three seconds and a 440Hz tone for three: the strip is a hairline under the first half and tall bars under the second. The iOS reader is exercised by `ios-export` against a file written the same way. Closes #261
sjungwon03
added
priority:high
sjungwon03-ai
left a comment
sjungwon03-ai
left a comment
There was a problem hiding this comment.
The exact-head ios-export check fails: PeaksTests.testReadsLoudWhereTheToneIsAndQuietWhereItIs reads a 0.3617 peak in the intended-silent half versus 0.6176 in the tone half, so the new iOS waveform reader is not reliably assigning samples to the requested window. Please correct the bucketing/window handling and make this test pass.
CI measured the fault the job was added to catch: the iOS reader saw a third of full scale in a half that was written silent. Both readers took the loudest sample in a decoded buffer and assigned it to whichever bucket the buffer *started* in. A decoder hands back chunks that straddle bucket boundaries, so a loud passage smeared backwards over a quiet one — and the fault was invisible until something measured a file whose loudness was known. A sample's moment is its buffer's timestamp plus its offset at the sampling rate. Both readers use that now, and both skip anything that falls outside the clip's window instead of clamping it to an edge bucket, which was the same smear by another route. Android had it too and is fixed alongside, though only the iOS reader is measured by CI.
|
Fixed, and thank you — that reading was a real fault, not a threshold problem. Both readers took the loudest sample in a decoded buffer and filed it under whichever bucket the buffer started in. Decoders hand back chunks that straddle bucket boundaries, so a loud passage smeared backwards over a quiet one; 0.36 in a half written silent is exactly what that looks like. A sample's moment is its buffer's timestamp plus its offset at the sampling rate, and that is what decides where it lands now. Anything falling outside the clip's window is skipped rather than clamped into an edge bucket, which was the same smear by another route. Android had the identical fault and is fixed alongside — same reasoning, same shape of code — though only the iOS reader is measured by CI. Now on this head:
This is the second real bug |
sjungwon03-ai
left a comment
sjungwon03-ai
left a comment
There was a problem hiding this comment.
Looks good: the waveform read path is now time-bucketed end-to-end and covered by added platform/native tests for both Android and iOS readers. The required check is currently passing on this head SHA, so the previous iOS bucketing window defect is resolved at the current commit.
2 participants
An audio clip was a coloured block with a filename on it, and a video clip showed frames and nothing of its sound. Either way there was nothing to aim at: finding a beat, a word or a gap meant scrubbing and listening, one guess at a time. Video clips got frames in #246 for exactly this reason; sound never did.
Decoded, not estimated
Nothing in a file's metadata says where it is loud, so both readers decode. Each takes the clip's own window and folds samples into buckets as they arrive — a long file costs a pass rather than a copy of itself in memory — and each takes the peak per bucket, not the average: the loudest moment in a bar is what the eye is looking for, and averaging a busy passage flattens it into a wall.
MediaExtractor+MediaCodecinto PCM.AVAssetReader, in a file with noExpoModulesCorein it — soios-exportruns it for real against a clip written silent-then-tone.Drawn on video clips too
The phone cannot import an audio file at all (the picker is videos-only), so almost every sound in a cut is a video clip's own. On a video clip the waveform is a strip along the bottom of the frames: the frames say which shot this is, the waveform says where the words are. On an audio clip it takes the whole height, because there is nothing else to show.
Normalised against the loudest bar rather than full scale — a quiet recording would otherwise draw as a flat line and say "no sound here" about a clip that has plenty. Silence keeps a hairline instead of a gap, so a clip never looks like it failed to load.
Best-effort like the thumbnails: reads are queued one at a time, cached at module level because a clip is unmounted every time it scrolls out of view, and a file that will not decode leaves the clip exactly as it was.
Verified
On the emulator, with a clip that is silent for three seconds and a 440 Hz tone for three: the strip is a hairline under the first half and tall bars under the second.
985 tests · typecheck on both surfaces ·
npm run build· Kotlin compiles.Noted, not fixed
The phone cannot import audio. That is a real gap — it means an audio track is only reachable through generated speech — and it is its own piece of work rather than a line in this one.
Closes #261