✨ feat(video-engine)!: a browser video engine, and annotations that draw on the frame it presents - #93
Open
cfviotti wants to merge 346 commits into
Open
✨ feat(video-engine)!: a browser video engine, and annotations that draw on the frame it presents#93cfviotti wants to merge 346 commits into
cfviotti wants to merge 346 commits into
Conversation
The sample picker offered three basketball clips that a viewer could not tell apart. Two were the same nine seconds of the same game and differed only in which model run produced their detections and whether the demo played the source file or a 30fps transcode of it. There is one now, and it is the better of the two: five annotation kinds on the clip's own 25fps frames, where the removed fixture had four on a resampled proxy. The three documentation playgrounds, the annotation renderers, the homepage basketball demo and the tracking post processor, all open on it. Nothing on those pages looks busier or emptier than before. The merged fixture draws 10.9 detections a frame at its own confidence gate against the removed one's 11.0, and the per-second profile matches: 4.4 at the opening rising to a plateau of 11.8 to 13.1. Two pages needed care to keep looking right: - The polylines page pins its confidence gate to zero. That page scopes itself to the ball's one derived trace, and the fixture's 0.5 gate hides 200 of the 224 trace segments, so the page drew the ball with no trail at all. - The tracking page stopped inventing a frame count. It read "0/270" while loading, which was the removed fixture's length quoted as a fact. It reads "0/0" until the real number arrives. The clip's media also stopped being tracked twice. `basketball_sample.mp4` was committed in two fixture directories; the second was a Git LFS pointer, so a clone was fetching the same 22MB payload a second time for nothing. Both fixtures share the one copy. `benchmark/masks/run.mjs` located that media through the manifest's provenance record, which names a path that no longer exists, so it now reads `fixture.meta.json`, which is what the demo itself uses. The removed fixture's pose run moves to `basketball_regions`, the only thing that still reads it, and the fixture builder's defaults follow the fixture that survives. Its README now carries the geometry coverage and the provenance the removed one held, including the two model runs that cannot be reproduced from this repo as it stands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Anyone installing `supervision` and reaching for video got told to install
`supervision-js-video-engine`, and that package did not exist. The release
workflow built and published one tarball, the browser package, and the engine
sat at version 0.0.0 with nothing to publish it.
The release now ships both. The engine goes first, so the browser package never
reaches the registry naming a peer that cannot be installed, and it publishes
through the same guards the browser package already uses: the manifest version
is the source of truth, an already-published version is a silent no-op rather
than a failure, and a prerelease and its dist-tag have to agree.
The engine starts at 0.1.0, matching how this project released its first
browser version, and the browser package moves to 0.1.8. It had been sitting at
0.1.7, which is what npm already serves, so the next release would have been
refused as a republish.
The optional peer range narrows from `"*"` to `"^0.1.0"`. The old range would
have accepted a future incompatible major, and the failure would have surfaced
at run time inside `openVideoEngineMediaSource` rather than at install.
One step still needs a person, once. npm will not attach a trusted publisher to
a package name that does not exist yet, so the very first engine release fails
until someone registers the name from their own machine:
npm trust github supervision-js-video-engine \
--file publish-npm.yml \
--repository roboflow/supervision-js \
--environment npm-publish
The engine also gets the LICENSE and README that npm always ships regardless of
the `files` list, so its package page is not blank. Its entry-point table was
checked against the manifest's own `exports` map rather than written from
memory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The halo glow vanished entirely on any scene where some masked detections are filtered out of the picture. Every player kept their silhouette, and not one of them glowed. A single wide mask was enough to do it. The id mask raster holds one identity per pixel, so the last mask written to a pixel owns it, and the halo reads the identities back to find the silhouettes it paints. The layer that prepares that raster admitted every detection carrying a mask, including the ones the halo declines to paint. A full-frame detection at low confidence therefore claimed every pixel on screen, buried the identities of the players above it, and left the halo with a palette full of entries and no pixels to match them. The preparation and the paint now ask the same question through one predicate, so they cannot disagree again. A halo that paints nothing, whether because it has no mask, no instruction, no opacity or no spread, also claims nothing. That last case was live: a demo halo style reports its configured opacity unconditionally, so at zero glow opacity every masked detection was still claiming raster pixels while painting nothing at all. Preparing that coverage is expensive, so the scene reuses it until the set of detections the halo admits actually changes. It compares the two styles over the detections currently buffered instead of assuming any restyle is a new set, which is what a style whose only member is an arbitrary function allows. Moving the spread slider through twelve steps cooked mask coverage 204 times before and cooks it zero times now; driving glow opacity off zero still costs the 17 cooks that genuinely have to come back. One gap stays open and is documented where it lives: a prepared artifact can outlive the buffered window, so a restyle that moves the admission boundary on a frame the buffer has rolled past keeps a stale artifact. Closing it exactly needs `MaskHaloStyle` to carry the identity `MaskStyle` already carries. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scrub group reported four seek timings above a population nobody could see, and two counters the engine broadcasts every tick were read by nothing at all. A reader looking at four timings and a zero had no way to tell "zero because nothing asked" from "zero because the counter is broken". A `Cursor seeks` row now reads them, split as exact and key. Reading it against `Seeks` in the group below answers a question that has cost real time twice: a seek issued while the video plays re-anchors playback instead of moving the cursor, so it lands in neither count and times nowhere. Paused, seven seeks read seven exact. Playing, the same seven seeks read zero here and seven there. Both ledgers are on screen and visibly disjoint. No engine counter was changed to make the panel look busier. The engine was already counting these correctly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three ways the demo eval could report a number that was not true. **A retried scenario measured a warmer page than its first try.** When a scenario was disturbed and retried, the retry ran on the page the first attempt had already warmed, so it looked dramatically better and the harness kept the better number. Measured on the drag scenario, attempt one to attempt two went 32.6ms to 8.1ms stale and 50 to 91 frames a second, on the same build in the same minute. A retry now reloads first, and the two attempts land together: 32.3 to 27.1ms and 50.8 to 62.2fps. The reload costs about 0.6 seconds and only a disturbed attempt pays it. `cadence` keeps its page on purpose, because it selects its own fixture and a reload would drop the demo back to the default clip while every number still named the other one. **The paints scenario could not see a pause that keeps drawing.** It waited six seconds before it started tracing, so anything that decayed after a pause was already over. It now starts the trace first and pauses inside the window. Twenty passes put the settling burst at 167 to 177ms and 11 to 15 paints, with zero paints once settled, so the new budget sits five times wider than the widest pass: the gate is for a pause that keeps drawing, not for the transition. **No recorded number said which tree it came from.** A report could be compared against a baseline taken on different code with nothing to catch it. Reports now carry the commit, whether the tree was dirty, and the fixture the scenario ran on, and the baseline comparison warns before it prints a single delta. The ten guessed noise floors are untouched. Picking numbers without measuring is the failure being fixed here, and the paints scenario's neighbours now read slightly outside two of them, which makes those floors the next thing to measure rather than the next thing to widen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Eleven documents in this repo were false at the same time and every gate stayed green. A fixture README told a reader to run a script that has never existed. The root README named a release that had already shipped. A rebuild command silently read a different input than the one it named. The gate checked that links resolve and that the API facades cover every export. It never read a claim. It now checks six kinds of claim across all 81 tracked Markdown files: - a path a document names has to exist - an `npm run` script it shows has to be declared, workspace forms included - a flag it passes has to be one that script actually parses - a checksum it quotes beside a path has to match that file - a version it states beside a package has to match that manifest - a module it imports has to export what it imports Each was proven able to fail by injecting the failure and watching the gate catch it, including the two real ones above. The link check widened from a subset to all 95 links in the corpus. Six live violations turned up, all in planning documents: a module that never existed, a proposed filename read as an existing path, three references to a module that was renamed before it shipped, and an API sketch importing three symbols under names the package does not use. Counted claims like "nine tsc projects" are not checked. A number in a sentence has no mechanical link to the set it counts, and a gate that guesses is worse than none. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`npm run verify` was failing on this branch before any of today's work, on two counts that had nothing to do with each other. The mask benchmarks declared `setTimeout` and `clearTimeout` in a `/* global */` comment. This branch had already added those to the shared eslint globals, so every one of them was reported as redeclaring a built-in. The comments keep only the globals the config does not supply. And a fixture tool had drifted out of Prettier's shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The branch had upstream's code but not upstream's history. Whoever integrated the 23 commits upstream shipped after the fork did it by copying files, so git recorded no second parent. Every one of those commits still counted as absent, GitHub computed the pull request's diff from the fork point, and the branch would not have merged at all. It merges now, and the tree it produces differs from this branch by eight files. That is the whole point: our side was already a superset, and this records the history to prove it rather than asserting it. Resolving 52 conflicts, the rule was never "which side wins" but "does upstream have something we lack". Every hunk that answered no took ours; the ones that answered yes are ported below. **Ported from upstream** - The docs for `pruneFrames` and `appendLiveFrame`. We carry both and had no written account of either. - The annotation roadmap's current renderer vocabulary and its guidance on the smallest public addition to reach for. - Eight tests: four interaction-presentation tests, two `basketball_regions` fixture tests, a halo-only renderer list, and screen-sized region assets holding steady across a paused zoom. - Two stylesheet rules that fix real gaps here. Without the first, the tracking playground cannot scroll on a narrow viewport. Without the second, hiding a class had no visual affordance even though the markup already emits the modifier. - Three behaviours the resolution would otherwise have dropped: a hidden detection is no longer pickable, region badges redraw on a viewport-style change instead of holding stale geometry, and the editing overlay draws keypoints with the style its host configured. Upstream added 45 stylesheet selectors since the fork and only those two rules were missing here; everything else names a class these components never emit. **What resolving this taught, recorded because it nearly went wrong** Ten places where git auto-applied an upstream hunk outside every conflict region, because our copy-based integration had moved the same code elsewhere. Keeping the HEAD half of each marker would have shipped a duplicated key, four duplicated function definitions, a duplicate block-scoped constant that does not compile, and a reference to a variable this branch never declares. Files were resolved by whole-file replacement, never by patching the marked regions. **What this merge does not take, and why** Two upstream regression tests for seeking while buffering. The production fix they guard is already here; only the coverage is lost. They cannot be ported as written because they drive the renderer into buffering through the playback gate this branch removed, and the option that gate reads is still accepted and ignored, so they would compile and never reach the state they assert. Two upstream tests asserting that playback waits for detection coverage, which this branch's own tests assert it never does. They are direct opposites and cannot coexist. Upstream's editing-gesture hide, which stops the base layers drawing a detection while a gesture previews it. This branch keeps drawing it and moves it instead. That difference stays a deliberate decision rather than a merge artifact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pressing play when the renderer cannot start it did nothing visible. No error, no state change, no hint that anything had been attempted. The renderer's own `togglePlayback()` decides pause or play and drops the rejection from the play it starts. That decision has to stay where it is: it has a branch for a drag in flight, where the producer sits mechanically paused and reads as not playing, so a caller that reads the state and calls play itself would resume a clip the viewer had just paused mid-drag. So the reporting goes onto `play` instead, on the renderer the demo adopts. The play that `togglePlayback()` starts and drops now reaches the same error line every other failure in the demo uses, and callers that already handle the rejection keep handling it. Also drops a `createImageBitmap` stub from a session test. Its comment said it was there to make the pipeline take a mask path that this branch does not have, and a counting probe confirmed the global is called zero times on that path. The test still bites: removing the halo renderer from the presentation fails it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five options let a consumer ask playback to wait until detections and prepared masks cover the moment being played. All five compiled, all five were documented as doing nothing, and all five did nothing. A consumer who set one got silence. They work again, and the default is unchanged: the picture still moves first and annotations still catch up behind it. Setting nothing gets exactly what setting nothing got yesterday. Ask for the gate and playback waits, on play and through the playback loop after a seek, which is what it did before it was removed. The reason for the option is that both behaviours are legitimate. A viewer scrubbing through footage wants the picture immediately and can accept annotations arriving a beat later. A reviewer stepping frame by frame to judge a model would rather wait than see a frame with nothing drawn on it. That choice belongs to the application, not to us. **The trap this nearly shipped with.** Two session defaults still resolved the gate as enabled. They were harmless while nothing read them. Reviving the option without touching them would have turned waiting on for every media session and every render preparation, which is the opposite of the intent. Both now resolve off, and their lookahead numbers stay, so an application that opts in still inherits sensible tuning. Five existing tests set the flag and asserted that nothing waited, which was only true while the flag was inert. Each now tests the real default with nothing set, and the gated case sits beside it. Four tests come back that could not exist while the gate was gone, including two regressions covering a seek taken while buffering. The fix they guard was never lost, but nothing had been able to reach the buffering state to prove it. Documentation stops describing a no-op. Every surface that names the gate now says what it does, what it does not do, and that it ships off, and the contract test that pins those surfaces was rewritten to check for that instead. Two limits worth stating. The gate is a pull-path feature: a push producer never builds the controller that owns the wait, so a push session that enables it gets no gate, exactly as before. And enabling it for detections without a lookahead is inert, because the required coverage ends where playback already is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… canvas This is a computer-vision library, and its performance gate spent its headline budget on how often the repo's own demo repainted a control bar. Ten metrics measured the demo application rather than the library: DOM paint, style-recalc and layout rates, the settling time after a pause, a keyboard shortcut scenario whose entire code path lives in a demo component, and a playhead position parsed out of a demo element's CSS transform against a limit calibrated to that component's half-pixel quantizer. None of them says anything about what a consumer installs. They are gone. The paint budget was worse than useless. It excluded the canvas by matching a paint event's rectangle against the canvas box, and that never matched once, so the number it judged was always the whole page. Chrome hands the node name directly, and in a playing window all 366 paint events named a demo element: the timecode, a cell value, a timeline segment, the inspector column. Not one named the canvas. The rectangles show why the match could not work, since a paint clip is a cull rect and not a damaged region: the root document reported 3000x2300 on a 1500x1150 viewport. **What replaced it answers the question that was actually worth asking.** A canvas presenting video has to paint once per presented frame; painting more than that is waste. Nothing compared the two, though the harness collected both. It does now, and the answer is that the renderer draws exactly once per presented frame: seven windows, ratio 1.0000 every time, and zero draws while paused. The budget is 1.05 with no tolerance. Measuring that also priced the thing the paint gate was standing in front of. In a six second window at 27.1 percent main-thread occupancy, every paint event combined costs 0.106ms per frame, while handing each decoded frame across the worker boundary costs 1.862ms. The gate was watching something 17.6 times cheaper than the cost beside it, and that cost is now written down where the next reader will find it. One metric was retargeted rather than deleted. Whether the playhead drifts while the transport is stopped is a real question, so it now reads the library's own clock instead of a demo element's transform. Its limit was re-derived from measurement and came out at zero, because a stopped transport's time is a stored number and does not jitter. Five surviving metrics have library numerators scaled by demo input, and each now says so where its number is read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
…t it does Two public shapes that made a reader work harder than the code does. **A highlight had two ways to be styled, and they shadowed each other.** Four fields on the base interaction style set the hover and selection rectangle directly, while `hovered.boxStyle` and `selected.boxStyle` did the same job through the ordinary style path. Setting only one state silently switched the four fields off for that state and left them running for the other, which is a half-migrated config that compiles and quietly draws two different highlights. The four are gone. The remaining path has full parity, including the thing that could have made it inadequate: a box style handed to both states can still tell them apart, because the renderer forwards hover and selection into the style context. It also reaches mask, label, keypoint, polygon and polyline highlights, which the removed fields never did. A default highlight looks exactly as it did. **`requiredForPlayback` had nothing to do with playback.** It picks which detection sources a composite source waits for when it reports a range as covered. Every document that mentioned it spent its second paragraph explaining that the name was wrong, which is a strong signal to change the name rather than keep apologising for it. It is `requiredForCoverage`. Coverage is the word these files already use for what that wait is about, the boolean keeps its polarity and its default, and it no longer reads like a second setting on the playback gate sitting beside it. The paragraphs that existed to walk the old name back are gone, and what is left says what the flag does. **Both are breaking, and one fails quietly.** A TypeScript consumer gets a compile error either way, which is the kind that fixes itself in a minute. A plain JavaScript consumer passing the removed style fields reverts to the built-in highlight. A plain JavaScript consumer who had set the renamed flag to false starts waiting on that source again, and with the playback gate enabled that means playback starts waiting too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three changes on this branch alter what an application already using `supervision` gets without changing a line of its own code, and a patch number would not say so. The playback gate ships off by default, where upstream shipped it on, so an application that never configured it now sees the picture move before the annotations do. Four interaction-style fields are gone. And a detection-source flag changed its name. Two of those fail quietly in plain JavaScript. Removed style fields revert a custom highlight to the built-in one. A renamed flag reverts to its default, which under an enabled gate means playback starts waiting where it did not. Under the 0.x convention a minor is the signal for that, so this is 0.2.0 instead of 0.1.8. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An earlier pass replaced a paint gate with three canvas metrics, on the reading that the question worth asking was whether the renderer draws more than once per presented frame. It does not, and it cannot: that ratio reads exactly one on every window ever measured, because the loop is driven one draw per frame by construction. The repaint that actually matters happens in the browser's compositor, one level below anything this harness observes, and it is a property of drawing each frame on the main thread instead of in a worker against a transferred canvas. That is a separate piece of work with its own decision to make, and no metric here should imply it has been measured. So all three go. The presented-frame rate was a window average of a number the cadence scenario already gates three sharper ways, including against the engine's own ledger. The paused render count asserted the loop is idle while stopped, which is a cost question wearing a fidelity name; whether a stopped transport holds its clock is already gated at zero drift. Swept the residue with them. A source contract pinned a demo component's playhead geometry on the grounds of main-thread paint load, which this harness prices at 0.106ms per frame. A comment justified pinning the Demo view with a paint census, and now names the reason that still stands: the Debug view's readouts land inside every frame time and long task sampled. The one number worth keeping is the price of the deferred work, so it moves to the document that describes the presentation boundary, as a recorded measurement rather than a gate's justification. It remains disputed: an independent pass measured the same handler an order lower, and both readings were taken on a machine running many jobs at once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
In the region effects sample the big-head lens jumped off a player's head and
snapped back, frame after frame. It was most obvious when almost nothing was
moving, and some lenses floated over the crowd with no player under them at all.
Every head the model actually saw was fine. The wrong ones were the frames where
the model saw nothing and the head position had to be invented.
Inventing it used the wrong reference. A missing head was moved by however far
the player's bounding box had moved, and that box is set by whichever limb
reaches furthest, usually a raised arm. Vertically the two barely relate. So the
lens tracked an arm instead of a head, and because each invented frame copied
from the single nearest real one, four frames of invention drifted five times as
far as one.
A missing head now sits between the two real observations on either side of it,
which is where it was. And a frame with no player detection at all no longer
invents one by averaging the players before and after, which is what put lenses
in the crowd.
invented heads before 7.2px off, 15.7% badly placed
after 2.8px off, 2.8% badly placed
real heads 2.8px off, 3.5% badly placed
Invented heads are now placed slightly better than observed ones, which is the
point at which they stop being visible as a defect.
Long gaps are no longer filled. Through four frames the fills are
indistinguishable from real observations; at five and beyond the head travels
five or six of its own widths during the camera pan, and no placement rule
recovers that. Four estimators were compared on the same frames and every one of
them was wrong about half the time at seven frames, so those fills are dropped
rather than guessed. Frames keep at least two heads throughout.
Only invented heads changed. Every other detection in the fixture, and every
head the model saw, is byte-identical.
The rebuild runs from the committed fixture, needs no model and no API key, and
is idempotent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dragging a detection drew it twice. Resizing one drew the old size and the new size at the same time, so a resize showed two boxes with only one of them under the pointer. Upstream solved this by hiding the detection from the base layers for the length of the gesture and letting the preview overlay draw it alone. Half of that arrived here and half did not. The overlay half is present, and its own comment still explains that it draws the box fill because the source is hidden. Nothing hid the source. A move looked less broken only by accident: it takes a shortcut that slides the existing box, so the two landed on top of each other and read as one box with a doubled outline. A resize has no such shortcut, so both were visible. The hide is back, and the layers that have no preview to draw stay out of it: labels and region badges keep drawing and follow a move as they did. Four other call sites had to stay on the unhidden state or the fix would have worked against itself. Two feed the focus layer, which would otherwise have filtered out the very detection it was asked to follow. One decides what can be picked, and hiding there would have dropped the gesture's own selection halfway through the drag. With the detection hidden, the focus cut-out now follows the gesture instead of staying at the position the drag started from, and a hidden detection is no longer pickable. A keypoint style set through a presentation update now reaches the overlay. It was accepted at construction and ignored afterwards, so nothing a host set after the first frame ever arrived, and the setter that was meant to deliver it had no callers at all. One more thing, found while checking the above: mask preparation is invalidated on a visibility change only if the scene names one of a hand-written list of style kinds. The list was written when there was one source of prepared masks and never grew when two more arrived, so a halo-only or region-coverage-only scene kept a stale raster and its hidden detections kept claiming pixels. The condition now asks the resolver instead of restating what it knows. A regression test covers the hide, and a second covers a mask preview not triggering it, since masks have no overlay to draw and must stay visible. Also restores an upstream test for the label surviving a gesture, lost when a file was resolved wholesale during the merge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The number input and the toggle in the quality controls had square corners while everything around them was rounded. They asked for a radius token the stylesheet never defined, so the browser resolved it to zero. They now use the token that exists. Also removes a playback store that was never wired to anything. It was added in the same change that rewrote the control bar, superseded before it shipped by the live-readout writer that has eight call sites, and never imported once. And a pose tool's usage example pointed at a fixture directory that no longer exists. It names the surviving one. That example lives in a Python docstring, which is why the documentation gate, which reads Markdown, could not see it. Six unused custom properties come out of the root block. A custom property that no rule and no script reads has no computed effect, so this cannot move a pixel; the extracted class set is byte-identical before and after. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Written after the fact, the branch had grown a second way to do work upstream already did, because the new engine feeds the same machinery from a different direction and each seam got its own answer. **Refilling the detection buffer.** Two rules, fifty lines apart in one file, computing the same threshold from the same option, neither mentioning the other. One came with upstream and fired from the pull path; the other was written here because the push path never reached it. The push path now uses the original, which also gets it coalescing and supersede handling it never had. Below two seconds of lookahead the old rule refilled on essentially every playhead move; that is now one refill at half the lookahead, like everywhere else. **A timeline the layers read.** A twenty-eight line wrapper claimed to withhold frames the prepared window did not cover. It withheld nothing: composed, its one non-delegating method was the identity. Its own test asserted the two were equal. The comment above its consumer described a filter that never existed, and that comment is gone rather than reworded. **Decoding an uploaded file.** The demo opened every upload twice, in two demuxers, concurrently, on a branch whose whole premise is that one engine owns decode. It now reads frames through the session it already has. The second mediabunny use stays, because encoding a still image into a one frame clip is a real thing no library entry offers. **Reporting a failed play.** Three mechanisms, one string. One is enough. **Recording that the playhead moved.** Two setters and two near-identical recorders, one per playback path. The legacy path now emits state on a time change, so a loop reset or a seek moves the readout before the frame lands, which is what the other path already did. **Formatting a playback rate.** The same value rendered as `8.0x` in one place and `8x` two lines later. The measured rate keeps its decimal, because it is a float and needs one; the commanded rate does not. Also removes a third frame-selection rule with no callers, a transport method with no callers, an option inert in both of its own branches, a duplicated session block, and a second component sharing a name with one in the same directory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Changing a marker, ellipse, box-corner or mask-halo style through `setPresentation` drew nothing. The style was stored, the vector layer was updated, a render was requested, and the scheduler declined it. The scheduler decides by comparing a signature of the presentation fields that matter. That signature was a hand-written list, missing exactly the four styles upstream added while this branch was forked. For those four the signature never moved, so the scheduler concluded nothing had changed. The demo hid it completely, which is why it survived: the demo rebuilds its renderer array on every call, so the array's identity always changes and every render lands. An application that keeps its renderer list and swaps one style sees a still picture. The signature now derives its style half from the renderer registry, which is where the mapping from renderer kind to style field already lives and which already carried a helper for exactly this, with a note saying consumers should read it instead of repeating the mapping. A renderer kind added later joins the signature by existing. One hand-written entry stays and one goes. Mask opacity stays, because it is the one value a host is invited to change inside a style object it keeps, and comparing objects by identity cannot see that. A visibility version goes, because it only ever moves when the visibility object itself has already moved. Also folds the two copies of the annotation draw order this branch had added into one declaration. There were five copies in all, and they had already drifted: two of upstream's disagree about whether focus draws before or after the interaction presentation. Resolving the merge required hand-adding a layer to one of them. The remaining three are upstream's and are a separate job, because collapsing them means picking a side in that drift and reshaping a public diagnostics type. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dragging the timeline, most often backwards, turned the picture solid black
while the timeline, the readouts and the buffer lanes all kept working. The
console carried one line:
Destroyed texture used in a submit. While calling Queue.Submit
A texture was freed while the GPU still held a command that referenced it. The
driver rejects the whole command buffer, so nothing in the scene reaches the
canvas: black picture, live interface.
The scene composites decoded frames into one GPU texture and swaps that texture
whenever the decode changes size, which is exactly what scrubbing does as it
alternates between a low resolution preview and a full resolution frame. On each
swap it restated the texture's size to the renderer. It read the old size back
from the object it had already overwritten, so instead of restating the
resolution it multiplied it: one, then a sixth, then a fortieth, on down.
Once that number is wrong the stated size can happen to match what was stated
before. The renderer reuses a texture binding as long as the stated size does not
change, so those swaps were completely silent: no new binding, no invalidation,
and the next draw ran against a binding pointing at the texture just freed.
The compositor now owns the statement of its own texture's size, and frees the
retired texture only after nothing can still point at it. The invariant is that
the stated size always equals the size of the texture it describes. A swap only
happens when the size actually differs, so with the statement truthful the
binding is always renewed and a stale one cannot survive. No guard and no
deferral.
Measured on the nine second clip, thirty backward drags each: one black frame and
one validation error before, none after. None forwards, and none on the seventy
second portrait clip either way. Canvas brightness across the thirty runs stays
between 107 and 111 where the failure read 2.
The reason it took so long to see: the two suspects were both wrong. Destroying a
texture with a queued copy still unsubmitted produces no error at all, which the
demo does hundreds of times a run, and the video engine creates no textures on
this path. It is a stale binding, not a stale copy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…at checks documents now reads code comments Two changes, related only by both being about claims a reader can trust. **The playback gate only works on one of the two ways this library plays video, and none of the five surfaces describing it said so.** A host could enable it on an engine-backed session, read documentation promising playback would wait, and get nothing: no wait, no buffering report, no error. The reason is structural. The wait lives in the renderer's own sample pump, so it can only hold a source the renderer pulls decoded samples from. A source that presents its own frames runs the playhead itself and is never asked to hold. That is what the video-engine source does, and it is what most video sessions run on. Enumerating every renderer option read past the point where a self-presenting source returns shows those two gate fields are the complete set that path never consults. Every surface now names which sources are held, names the engine source rather than describing a category, and says that enabling it elsewhere is silent. The streaming recipe gains the alternative: await the detection source's own range wait, then play. One sentence in that recipe was outright false, claiming detection coverage never raises the blocked-playback signal; with the gate honoured, the wait sets buffering and buffering blocks playback. Also removes two `muted` options declared and never read. One documented itself as a no-op, the other said nothing, so a host setting it got silence either way. Audio is not in this release, and the option's absence is a compile error rather than a quiet lie. **The documentation gate could not see a claim written in code.** It read the 81 Markdown files and nothing else, so a usage example in a Python docstring pointing at a deleted fixture directory went unnoticed for days and was found by hand. It now reads comments too: 363 source files across TypeScript, JavaScript, Python, shell, and the script blocks of every manifest. The same six claim checks run over them, parsing TypeScript with its own compiler so a regex or a string containing a URL cannot be mistaken for a comment. Comments are treated as prose rather than as shell transcripts, which is what makes the motivating case catch. One check is new and only applies to executables: a flag a script shows in its own usage comments must be one its own argument parser reads, with the accepted set computed from the file with its comments blanked so an example cannot vouch for itself. The first sweep produced eighteen hits, seventeen of them noise, and each was eliminated by a rule rather than an exception: runtime flags left of a script path belong to the runtime, a manifest script resolves in its own manifest, and prose must quote an invocation the way it already has to quote a path. Every class was proven able to fail by injection, including the docstring that motivated the work. Two known escapes in the Markdown checker were narrowed. A path a fenced block writes now only excuses reads after the line that writes it, and the bullet verbs that mark a file as proposed only work inside plan documents. That second one mattered immediately: without it, one comment line would have been a way to silence the gate anywhere in the codebase. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ution Five renderers each carried their own copy of the same WebGPU vertex program. Four were byte-identical, 1041 bytes each; the fifth differs by the two lines that carry a tint its fragment stage does not take. They are written once now, in two variants, and every one of the ten shader stages across those five files still hashes exactly as it did. The GLSL beside them duplicates the same way and stays as it is. That is not squeamishness about upstream code: all eight copies, here and on main, hash identically, so this branch has never edited a byte of them. Today each of those hunks merges as context and can never conflict. Collapsing them would rewrite all four and turn any future upstream edit to that program into a conflict here, to deduplicate code this branch does not maintain. The WebGPU side is the opposite case, since none of it exists upstream at all. The test that requires every shader to carry a program for both backends finds them by reading each renderer's source, so a program moved into a shared module would have vanished from it silently. It follows the import now, and both failure modes were checked by breaking them: a missing program, and a shared export renamed out from under its user. **Separately, two places computed the same display fit and disagreed about the ceiling.** The engine caps decode resolution at twice the device pixel ratio. The mask layer, handed a box that states no ceiling, applied none, so on a three times display the picture decoded at 2x while the mask rasters cooked at 3x. Only the one caller that always states a ceiling kept them in step. Both now resolve it the same way. A caller that omits it gets the engine's ceiling, so the rasters land on the grid the decode already used: at three times that is two thirds the width and four ninths the texels. At or below twice, nothing changes. The engine keeps its own copy of the number, because it depends on nothing else in this repo and giving it a dependency to share a constant costs more than the duplication does. One cap was deliberately not applied. The identity plane cooked beside an RGBA composite is uncapped, and capping it would tear the halo: the halo sizes its canvas from the composite's dimensions and walks the identity bytes linearly, so two planes of different sizes put every row after the first at the wrong offset. It would also almost never fire, since that path is reached only when the scaled raster already failed. Both call sites now say so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ay round a loop On a looping clip the renderer reported how far ahead it had prepared annotations by measuring to a frame the playhead had already passed, going forward the long way round. Seventy seconds of horse trail reported 66.86 seconds of readiness for 211 frames covering seven. That number is not a readout. It is compared against the lookahead a session asks for before playback is considered ready, so a wrong one can hold or release the gate for the wrong reason. The window that feeds it kept every frame it had ever seen when the clip loops, including ones with no detections, which are never pruned. Sorted by distance around the loop, a frame passed minutes ago sits at the far end and looks like the furthest thing prepared. The window now takes its members from the frames the buffer actually holds, and differs from the non-looping case only in the order it walks them. A frame just past the loop point stays, because the buffer genuinely reaches across the wrap and plans for it. A frame from the previous lap goes, because it does not. Both simpler fixes were tried against the test that pins a run crossing the wrap, and both broke it the same way: seven prepared frames became three. They discard exactly the frames the core loaded across the loop point, so every lap would start cold. Stopping the run at the wrap has a second problem, since the readiness check does not walk the run, so the two sides would disagree at every loop. The demo stops correcting the number. It had a module capping the reach by the frame count, whose own comment recorded the symptom, and a test pinning the correction. What stays is the demo's own arithmetic, since the library publishes a target as a count and a progress bar needs a length. The debug panel had the same pair on screen four times, two readings raw and two corrected, so it printed the very number the correction existed to replace. It has one reader now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cfviotti
changed the title
… hop has one number Nine metrics had been added over several commits and never recorded, so the harness was comparing 21 of its 30 against a baseline and silently ignoring the rest. Seven of the nine are the cadence family, which is the scenario written to catch a stall. Recorded from seven passes on a clean tree, with both scenarios that were failing when the old file was written now passing, and with the clip written down, which the old file did not do. Two of the twenty comparable metrics moved outside their own noise floor and the rest sit inside it. The thirty noise floors stay as they are. Twenty-nine already carry the distribution that set them, twenty-nine to a hundred and fifty-five passes each, and seven passes is a smaller sample than any of them. Four metrics did spread wider than their floor over those seven, and those four are the ones the harness already documents as inheriting whatever state the ten scenarios before them left in the page. Widening a floor to fit a noisier sample is the move this harness exists to prevent. **The per-frame cost of handing a decoded frame to the main thread is settled at about 1.1 to 1.4 milliseconds**, and the two instruments that appeared to disagree turn out to bracket it. One times the whole browser task, including deserializing the transferred frame before any of our code runs. The other starts at our first line. The difference between them is that deserialization, about a quarter of a millisecond. The 1.862 figure was wrong for a reason worth recording: it divided every message in the window by the number of frames. Those messages are not one population. About three hundred and fifty are trivial, one hundred and eighty are the frame hop itself, and six are worker replies draining promise chains at forty to sixty milliseconds each. Those six carry more than half the total, so the average moved with how many replies happened to land, not with the cost of a frame. Annotations cost three to seven hundredths of a millisecond per frame, not the third of a millisecond previously reported, corroborated by two layer configurations landing within a sixth of a millisecond of each other. And the GPU task the old table listed on the main thread is not on it; it belongs to the GPU process. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five places listed which annotation layers draw and in what order. Two were this branch's and were folded earlier. The remaining three were upstream's, and they had drifted: one drew the focus veil before the hover and selection affordances, another drew them the other way round. The veil is right first, and the tie is broken by what the two produce rather than by which spelling appears more often. The veil sits in a lower slot than the affordances, so the affordances paint over it, and the declaration lists the two in the order their output stacks. Neither reads what the other writes, so the swap moves nothing today; the layer that does decide hover and selection already runs before both. The inversion was never a decision. It exists only inside the bracket that timed interaction and its presentation together, which forced them to be adjacent. The timing now wraps each step individually, so the reason is gone and the reported buckets are unchanged. No public type changed shape. The viewport redraw, which runs on zoom, pan and resize, now walks the same declaration with the two steps it never had left out. Masks and hover picking take no viewport scale, so a scale change cannot alter what they draw, and the picking step is the one that decides what is hovered and selected. Running it on every pan would re-decide them. The one thing that moved is labels, which now draw after the veil instead of before. Nothing reads labels, their slot is fixed, and it is what makes the viewport redraw a strict subsequence of the declaration, which is what lets a test hold it there. Four two-step redraws elsewhere are left alone. They repaint the two layers a state change affects, not the annotation stack, and they already draw the veil first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ee smaller loose ends **The two fixture tools could not be piped together.** The frame extractor writes one manifest file; the pose runner wanted a directory of numbered images. So the pose half of a rebuild had no committed route, and a fixture's README said so. The pose runner reads the manifest now, the same file the segmentation runner already reads, taking each frame's index from the record instead of parsing it out of a filename. Nothing new is written to disk and nothing is re-encoded, which the alternatives both required. Everything around the model call is tested offline against stubs, 38 checks: argument parsing and all three of its rejections, every manifest shape and every malformed frame, real base64 decoding, and the whole run asserting the output header, the frame count, the checksum and one record per frame. The one line that has never run is the model call itself, because this machine has no install of it, and the script says so where a reader will see it. No driver was written for the hosted pose run that produced the committed file. Nothing in this repository states that endpoint's request or response shape, so the driver would have been an invented contract presented as a rebuild path. The README says which parts of a rebuild reproduce the committed data and which replace it. **A request builder with no callers stays**, because it is the only statement in the repository of what to POST to a proxy route the dev server registers unconditionally. Deleting it would leave a live endpoint that injects an API key with nothing describing its input. It now says that about itself. **A class name with no rule goes.** No rule ever existed for it in the stylesheet's history, and its sibling that needs no special treatment carries no modifier either, so writing one would have meant inventing a visual difference nobody asked for. **A textarea gets the same font as the input beside it.** The reset that makes form controls inherit the page font named buttons and inputs and not textareas, so the prompt box rendered in the browser default while the field next to it did not. **And the documentation gate stops trusting a fabricated output path.** A fence that writes a file may excuse later reads of it, which is right, but any string that looked like a command was trusted to do the writing. Now the command has to resolve to a script in this repository and that script has to actually read the flag. Shell redirection stays trusted, since it creates its file whatever runs to its left. The narrower rule that would close the rest of the hole was measured again and is still wrong: treating a destination as a claim about its parent fails three lines of a README that uses a placeholder sample name. That measurement is recorded where the next person will look, so nobody repeats it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…othing In Firefox the picture never appeared. One clip showed an error banner, another showed a grey rectangle with the annotations drawn correctly on top of it. The second is the worse of the two: it reads as a working library over a broken video. The scene hands a decoded frame straight to the graphics device to copy into a texture. Chrome converts a decoded frame. Firefox's converter takes a bitmap, an image, a canvas or an offscreen canvas, and refuses the frame with a type error thrown from inside the present. The present abandons everything after the upload, so the sprite keeps a texture nothing ever wrote to, while the redraw at a resting playhead carries on putting annotations over it. The scene now asks the device once, when it is built, whether it takes a decoded frame, and routes the frames through the staging canvas every non-accelerated scene already uses when it does not. The question is a property of the browser and not of the video, so it is asked once and never per frame: a browser that takes the frame pays a single one-pixel copy each time media opens, and nothing while it plays. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seven places across the public guides, the recipes and the internal contract described a PNG-encoded identity mask. This release replaced that with a raw raster prepared at display resolution, and the source carries no reference to the old form at all. Two of those documents were edited on this branch without the description being swept with them, which is how a reader ends up with guidance that describes a representation the library has not used for weeks. The plan documents are left alone. Their checkboxes record what was decided at the time and are true as history. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gate that waits for masks only ever held a reader the renderer pulls frames from. On the engine, which paces itself, it held the very first frame and nothing after, so a machine that could not keep up played on with no masks at all and said nothing about it. Switching every gate on did not change that: the per-frame hold was wired to detections alone. Masks can now stop a running engine too, so a clip whose masks fall behind slows down and keeps its annotations, which is what the reader that pulls has always done. A hold that waits forever would be worse than the fault it fixes, so it gives up after two seconds and lets the picture go on without masks, saying so rather than going quiet. It gives up only on preparation that has finished nothing at all: a slow one re-arms the gate every time it finishes a frame, however far behind it still is. Preparation that only gets a frame out while the picture is stopped counts too, which is what happens when drawing and decoding share a busy processor. Pausing during a hold no longer leaves the reader frozen. A wait that fails, and a pause that lands while a play is still waiting, both give the reader back, so the next drag moves the playhead instead of restarting a video that was paused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Playback could stop with nothing on screen to explain it. The only sign was a small ring on the play button, which is what the overlay falls back to when no notice could be built, so the viewer was left guessing. Three ways that happened, all now named. Fetching the video was invisible. The notices only ever described masks and detections, so a stop waiting on the clip's own bytes had nothing to report: on a real recording, two seconds after a scrub went entirely unexplained while the file was still arriving. A notice needed a quarter second of unbroken waiting, and the count started again every time a wait cleared for a frame. The reader that pulls holds many short waits rather than one long one, so the count never matured and no notice could appear however long the stutter ran, while the engine's single long hold showed one immediately. The same library, the same gate, opposite behaviour. A wait that clears for a moment and returns is now one wait. A hold on the frame about to be shown, while the frame on screen was ready, produced no notice at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Splitting the previous two commits dropped the code that reports a stop waiting on the clip's own bytes, while keeping the tests that cover it. A stop on a source read fell back to the generic buffering notice, which is what left the wait unexplained in the first place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…f it Every detection carried by a streaming model was measured by painting its mask into a frame-sized buffer and scanning every pixel of it for the edges. At 1920x1080 that is a two-megabyte allocation and two million reads to produce four numbers the run lengths already carry. The runs are walked instead. A frame-filling mask goes from 5.69 ms to 0.03 ms; a heavily fragmented one from 6.57 ms to 2.38 ms. The answer is the same in both, checked against the old path. This ran on the thread that draws, so a model streaming its results took a third to two thirds of every tenth of a second away from the work that keeps the picture moving. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…p ahead How much drawn mask sits in front of the playhead was counted as an unbroken run, stopping at the first frame nobody had finished. A window that was almost entirely drawn reported nothing ahead when one frame near the playhead was outstanding, so the video stopped with a full bar of drawn masks on screen behind it. A model that streams its results puts a fresh undrawn frame into that window several times a second, so the count could never climb back to what the gate asked for. Measured on a clip driven that way: the picture was stopped for 15.6% of the time in five freezes, the longest most of a second. It is now stopped between nothing and 6% in freezes of 80ms, and where enough is drawn ahead it never stops at all. The run now steps over a frame something is already drawing, since that one arrives on its own, and still ends at a gap nobody is working on. A frame the viewer is about to see still stops the picture, as before. The two edges of the wait were also far apart and in the wrong unit. Stopping cost a quarter second of clip and starting again asked for a whole second of it, so every stop had to bank about twenty-three more drawn frames than the one that triggered it. Both are now wall clock and the second is the first plus a margin, scaled by how fast the clip is playing: a stop buys about six frames instead of twenty-three, and asking for a deeper bank no longer buys a longer stop. The two are held apart at every speed and bank, so the pair can never meet and flap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The play button kept its play triangle while the video was stopped waiting for something, with only a thin line orbiting the square as a hint. It read as a video that could be played rather than one already trying, and it was the only sign at all whenever no notice named the wait. The button now shows a turning ring in place of the triangle while the picture is waiting, and stands still for anyone who has asked for less motion. The gate's two edges are separate controls, since they are now separate numbers, and the ceiling says what it does: it buys no drawn frames, it only shortens a stop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Users can now seek, scrub, step, pause, and recover without the picture, playhead, or annotations diverging.\n\nOwn presentation completion through actual paint, bound decoder and source work, and keep package and docs contracts aligned. The regression suite covers stale generations, transfer failures, terminal states, variable-frame-rate identity, residency, and both pull and push backends.
People debugging playback could see that the picture paused, but not whether navigation, media reads, detection coverage, or mask preparation caused it. The demo also hid effective wait bounds and collapsed independent detection and mask policies into one pipeline branch. - put mask readiness targets on the timeline and live blockers in Status - record detection and mask waits independently in Pipeline - expose effective wait ceilings and mask stop/resume thresholds - align defaults, docs, fixture labels, and tests with per-frame behavior Co-Authored-By: Codex Opus 4.6 (1M context) <noreply@anthropic.com>
When playback falls behind, stopping now settles the hidden clock on the last painted frame. Resuming or stepping no longer skips catch-up frames users never saw.
Under load, playback or a paused frame could keep the previous frame mask while the current raster was still preparing. This displayed a plausible but incorrect annotation over the picture. A pending mask now leaves its frame bare until its own raster lands. Tests cover adjacent frames while playing and paused, and the stale-state readout remains as an invariant tripwire.
At high playback rates on Android, dropped presentation work could leave the visible pixels paired with another frame’s annotations and make pause or the next step jump. Materialize the selected frame before transfer, coalesce before the display refresh, and advance playback only after the scene renders that frame. Navigation generations prevent stale acknowledgements from reviving an older position.
Keep the diagnostics tap test aligned with the host presentation contract so the full workspace verification can typecheck the demo. The fixture now exposes an acknowledgement spy and proves the tap only forwards it without claiming the frame reached the screen.
Dragging on Android could repeatedly start neighbor decode walks during brief gaps, burning CPU and causing seconds of tail jank. Wait for 100 ms of quiet before speculative scrub prefetch. Exact foreground landing and playback stay unchanged, and diagnostics distinguish a parked timer from active decoding.
Android H.264 playback could pair a decoder buffer that had already been reused with annotations for its earlier timestamp. Snapshot affected decoder output before queueing it, preserve that ownership through presentation, and avoid a redundant transfer copy so playback can drop whole compositions under pressure without showing false ones.
Collaborator
|
[Jarbas Local João] — REQUEST CHANGES at Gist: https://gist.github.com/joaomarcoscrs/48d5daba624ce9bc154fab6906f1971a |
Opening a video from a ReadableStream threw DataCloneError before playback started, so a streamed source never played at all. A stream cannot be structured-cloned, and every worker command was posted with an empty transfer list, so the load message had no way to carry one. The load now hands the stream to the worker on the transfer list, which leaves the worker holding the only readable end. A post the port refuses rejects the caller instead of leaving a promise waiting on a reply that never arrives. The engine-backed media source reads the video twice, once for the frames it presents and once for the thumbnails and single-frame grabs its sample sink answers. It takes a URL or a Blob for that reason, and refuses a stream before it opens anything, so the bytes are still there to hand somewhere else. A host holding a stream drives WebVideoEngine directly, where the stream has one reader. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Importing supervision/web-video-engine also evaluated the browser package's root entry, so an application that wanted only the video engine ran the whole renderer graph it never asked for. The adapter both entries share now ships as its own emitted module, and the subpath names that module rather than the root. Both entries still export the same adapter, so the two import paths stay interchangeable. The boundary tests walk each published entry's static import graph instead of reading the entry file, because the shared adapter chunk sits a step away from the entry that names it. The walk throws on an import statement it cannot read, so a walk that stopped early can never report a boundary as intact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The boundary tests prove the engine subpath never reaches the browser root entry by following every import out of an entry. A text pattern read those imports one line at a time, so a binding list spread over several lines was invisible to it. The walk would stop early and report the boundary intact because it had stopped looking, which is the one failure this check must not have. The walk parses the emitted JavaScript with rollup/parseAst, already a build dependency. It reads every static form, ignores the dynamic import the package splits its heavy chunks at, and cannot miss a statement it can parse. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The published package emits three entries, and the adapter entry is where both import paths end. Nothing asserted it ships, so a build that dropped it, or the chunk it re-exports, would have reached a consumer with both "supervision" and "supervision/web-video-engine" failing to resolve. The tarball suite asserts that entry, its declarations and its source map ship, and that the shared chunk they re-export is in the archive. That chunk is named with a content hash, so the test reads the name out of the entry that re-exports it rather than carrying a copy that would rot on the next build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The engine documentation described a worker boundary where everything crossed as a copy, and a published layout of two entries. Neither holds: a stream source crosses by transfer, which leaves the worker holding the only readable end, and the adapter ships as a third emitted entry that the root entry and the engine subpath both re-export. It also corrects three claims a reader could have acted on. The engine-backed media source takes a URL or a Blob, not any video source, because it reads the video twice. A stream's declared container type reaches no host through metadata, because nothing reads it. And the published dynamic import names a chunk-relative path rather than the package subpath, which the packaging document had wrong before this branch as well. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
npm versions are immutable, and 0.2.0-next.0 is already published from an earlier commit, so a preview carrying the video engine work needs a number of its own. Without it the publish workflow refuses the upload and consumers on the next tag keep resolving the older build. The manifest, the lockfile, the documentation toolbar and the README state the version together, which docs:check pins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A publish npm accepted was reported as a failed run. The post-publish check polls the dist-tag for 60 seconds, npm's read endpoint took longer than that to reflect it, and the run went red over a version that was already live and correct. Its log named only the version it saw, so nobody reading the failure could tell a rejected upload from a slow read, and those two need opposite responses. The poll now runs for five minutes, and a timeout reports whether the version reached the registry, naming republishing as the wrong move when it did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI on the PR branch was red on every push: 5a5ec46 raised the dist-tag propagation wait from 12 attempts to 60 but left the contract test asserting the old loop and the old "after 60 seconds" message, so `verify` failed before it reached anything else. The test now pins the loop the workflow actually runs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The problem
supervisionalready plays a video. It opens the file through mediabunny, and the renderer asks thesource for a sample at a time the renderer picked.
What it does not have is exact frame identity. The frame index it reports is
round((mediaTime - firstTimestamp) * estimatedFrameRate), and its own type documents that as anestimate. That arithmetic names the wrong frame on:
The failure is quiet. The pixels stay correct and only the annotations move. It survives pausing and it
looks like bad model output.
This pull request adds a browser video engine. The engine, not the renderer, decides which frame is
on screen. It publishes that frame with its own media time and its identity in the container's
timebase, and every annotation layer draws against that one value.
Who this is for
renderer sources and the renderer keeps picking the time. What does reach them is the
maxDevicePixelRatiodefault in What breaks, which caps every presentation surface at 2.The pipeline
Which path a source takes
flowchart TD A["Source opened"] --> B{"Which renderer source<br/>did the caller pass?"} B -->|"the default one"| PULL["Pull path"] B -->|"createWebVideoEngineMediaRendererSource"| C{"Engine chunk loaded?"} C -->|"no"| ERR["Throws, and names supervision/web-video-engine"] C -->|"yes"| D{"H.264 with an avcC record?"} D -->|"yes"| S["One decode session, held across seeks"] D -->|"no"| K["mediabunny sinks, re-positioned per request"] S --> PUSH["Push path"] K --> PUSH PULL --> P1["The renderer picks the time.<br/>It reads the sample timestamp."] PUSH --> P2["The engine picks the time.<br/>It publishes the presented frame."]The library does not read the media and choose. The caller chooses by which source it passes. A
video file passed as a URL opens through mediabunny and takes the pull path.
The pull path is what existing consumers use. Its shape is unchanged: the renderer picks the time
and reads the sample timestamp, as it does on
main. Its behaviour is not. A container that openswith no parsed track now fails as
UnsupportedFormat, wheremainfailed it asNoVideoTrack.errorKindis the field to branch on, so a consumer that reads it sees this. The default changes inWhat breaks reach the pull path as well.
What the engine does
packages/video-engineis a new workspace package. It is private, so npm never publishes it. Itsbuild is staged into
supervision, and consumers import it fromsupervision/web-video-engine. Itdoes not depend on the renderer.
avcCrecord. Other codecs decode through mediabunny's sinks, which re-position on each request.armTrace(windowMs)sizes the ring from the broadcast rate.analysisentry point opens a source and pulls frames without a player.Presented-frame identity
The frame table is built from each packet's own timestamp. That is what makes identity survive a
fractional rate or an unusual timebase.
The position published for a frame comes from the packet that was submitted. It does not come from
the timestamp the platform decoder returns. A decoder that counts from its own origin, or reorders,
therefore cannot slide the annotations off the picture.
A tripwire in the present throws if any layer is handed a media time other than the presented one.
It is armed in every build, production included. It costs about eleven comparisons per presented
frame.
Two rate-derived indexes remain. Both are documented as estimates, not identity:
estimatedFrameIndexin the renderer state, and theNearestFrameIndexdetection selection mode.Temporal detections
Detections are temporal data, independent of decoding. They can be precomputed, appended while
playback runs, or composed from several sources. Overlapping results update the active range without
rebuilding the whole annotation state.
Two producers exist in this repository: precomputed fixtures, and a remote model that pulls frames
from the engine's sample sink.
Prepared annotation rendering
The renderer prepares annotation artifacts ahead of the playhead. It keeps prepared frames on both
sides of it in a bounded cache, so a reversing scrub finds work already done.
On the push path, rendering is event-driven. Pixi's ticker is unused. The scene draws only on a
change: a new presented frame, a detection change, a prepared artifact landing, a hover or selection
change, or a presentation change. A paused scene nobody touches submits no frames. The pull path
still repaints on the ticker.
Masks
A worker builds one byte per pixel holding a detection id. A shader colours those ids from a palette
on the GPU. Where that raster cannot be built, the same worker produces an RGBA composite.
Boxes, labels and vectors draw as before; the hover silhouette is the one thing lost, because
the ids it needs are what the raster carries. The mask layer reports that state rather than
leaving it silent.
The palette holds 80 entries. One entry is the background, so a raster can name 79 detections. It is
keyed on detection index: a mask writes its detection's index plus one. A frame past the ceiling
falls back to the RGBA composite. That path walks each mask's runs rather than the whole plane
once per detection, which on 81 masks over 1920x1080 costs 19 ms for fills and 69 ms with
outlines, where walking the plane cost 151 and 337. The raster path is 1.5 ms. On the 2113-frame horse trail clip, 75
frames used to fall back and no longer do.
A host can declare the box it paints masks into, through
renderPreparation.maskFrame.display. The raster is then built at the size that box can show. Leftunset, masks are built at the detections' own resolution.
The push path runs on WebGPU where images stay on WebGL. Every shader therefore carries a WGSL
variant, and a test requires each shader to carry a program for both backends.
What is new in the public API
The root
supervisionsurface grows from 405 to 425 exported names: 20 added and none removed.The separate
supervision/web-video-engineentrypoint now publishes an explicit list of 48 names.Five internal names formerly leaked by its wildcard barrel were removed before release. These are
different surfaces and should not be combined into one addition count.
WebVideoEngineErrorCodeDecodeUnsupportedtoRateUnsupported. Reached atsupervision/web-video-engine.createWebVideoEngineMediaRendererSource,openWebVideoEngineMediaSource,WebVideoEngineMediaSourcePresentedFrameChannel,PresentedFrameSource,PresentedFramePlayheadand their signal typesPreparedAnnotationWindowSnapshot,PreparedAnnotationWindowFrame,PlaybackGateReachresolveMediaSessionDefaults,ResolvedMediaSessionDefaultsDecodedMediaSourcedeclares both drive modes, and both are public.sampleSinkanswersgetSample(timestamp)for a time the renderer picked.engineis aPresentedFrameChannel: thesource hands each selected frame to the host, which atomically composites matching annotation layers
and acknowledges the frame once displayed. A host with its own decoder can implement the push path
rather than only consume this engine's.
sampleSinkstays required either way, and the engine suppliesa real one over its batch analysis path, which serves thumbnails and one-off frame grabs.
Out of scope
own pull request.
Type of Change
Validation
How to run it
Current inventory and validation scope
npx vitest runnpx vitest runpackages/video-engine/src/*.test.tsnpm run docs:checkMETRICSintools/demo-eval/baseline.mjsFRAME_TIMELINE.MAX_FRAMESMAX_ID_MASK_PALETTE_ENTRIESPLAYBACK_RATEPLAYHEAD_QUANTIZATION_TOLERANCE_SECONDSThe engine's 39 test files cover decoding, timelines, cache behaviour, scrub trajectories, playback
scheduling, frame ownership, worker communication and presentation. They run in Node against fake
browser APIs and a recorded packet table.
Focused regressions cover exact seek landing, frame ownership through cache and teardown, playback
cadence through rate changes and hitches, and both gates on pull and push sources.
Where the numbers come from
Every performance number in this description comes from one Apple M3 Max, 16 cores, 64 GB, in
Chrome, against the 70-second horse trail clip at 30fps through the WebGPU renderer. The clip
carries 2,113 frames and 98,115 detections, which is 46 a frame. These are not cross-device
baselines.
Android testing established correctness, not a performance baseline. On Galaxy S23 Chrome, the
accepted Android H.264 ownership route produced exact raw-pixel/frame identity in 24/24 samples at
1x and 8x, in 14/14 pause and step checks, and in a post-restart 24/24 confirmation while retaining
one playback decoder. Its paired 8x playback slope was 6.588x versus 6.655x for the control. Galaxy
S26 testing helped reproduce the original failure. Other Android browsers, rotated H.264, canvas
presentation, and wide-gamut paths remain unverified. Android HEVC scrub performance remains a
separate unresolved performance lane.
The browser evaluation harness
npm run eval:demodrives the running player and gates 30 metrics against a recorded baseline. Itexits non-zero on a regression. The baseline records the machine, the commit, the clip and whether
the tree was dirty. The repository ships no baseline file and gitignores it: one recorded on a
given processor is only meaningful on that processor, so it stays local.
Eleven families:
sync,latency,layers,cadence,throttle,battery,blanking,drag,playhead,backscrub,focus.The
layersfamily carries a hard budget: zero frames over 34 ms, in every layer combination.Playback rate, presented-frame identity, cache behaviour and cache memory ceilings are covered by
engine unit tests instead. The harness only measures playback at 1x.
Reviewer checklist
npm ci && npm run verifyfrom a clean clone.open the HEVC fixture.
annotations match the video pixels.
setPresentationwhile keeping the same detection array.MediaRendereror interaction-stylecode.
Notes For Reviewers
Answers to review feedback
1. "Polyline rendering broke on a docs page."
You were right, and it is fixed. It was the fixture, not the polyline renderer.
The polylines page embeds the demo with the
basketball_sam3fixture. The page filters toclassName === "basketball"andmetadata.trajectoryTrackId === "basketball-track:0".SAM3 returns a whole-scene answer for that prompt alongside the ball. The fixture's trajectory
step accepted the whole-scene mask as the tracked ball and stamped the track id on it. The page's
filter then kept it faithfully. The precise shape of the defect:
basketball-track:0detections with rect exactly1920x1080af35486rebuilt the trace and refuses any candidate covering 50% or more of the frame.demo/src/fixtures/demo-fixtures.test.tsnow gates it:widestFrameCoveragemust stay under 0.5.That assertion evaluates to 1.0 on the old data, so it is the regression gate for exactly this.
At HEAD the trail is a ball trail: 216 polylines, all on the ball track, footprint 0.013% to 0.221%
of the frame.
The polyline renderer itself is untouched by this branch apart from the new
shadowStrokedefault,which landed after your report.
2. "Make 'buffered by detections' part of the
createMediaSessionAPI."Done.
createMediaSessiontakesplaybackGate, a plain boolean. You either want that playbackmode or you do not, which is the shape you asked for.
It is an umbrella switch over two gates. Either gate can still be set on its own, through
detections.playbackGateandrenderer.renderPreparation.playbackGate.playbackGate: trueNeither default changed. Both were already resolved this way on
main. What was missing was a wayto say yes or no to the whole thing in one place, and a gate that reached a source presenting its
own frames at all: on
mainthe wait lived in the renderer's sample pump, which such a source neverenters.
The docs page you saw playing bare now waits. The masks page embeds the demo. The demo opens a
sample on the Mediabunny media path, which the renderer pulls samples from. A pull source is held at
every frame whenever any gate is on, so
playbackGateReachreportsEveryFrame. The sample passesno session gate, so render preparation is the gate holding it. The detection gate stays off, because
a sample ships its annotations with it.
Both gates reach every frame on both paths:
flowchart TD P["play()"] --> G{"playbackGate"} G -->|"off"| RUN["Frames arrive at once"] G -->|"on, pull path"| PULL["Hold each decoded sample before draw"] G -->|"on, push path"| PUSH["Stop the producer when coverage or artifacts are missing"] PULL --> READY["Present when the wait settles"] PUSH --> READYThe pull path holds each decoded sample between reading and drawing it. The push path stops and
starts the producer, so the detection and render-preparation gates both cover ongoing playback.
Each gate's
maxWaitSecondsbounds its own wait. A pause or a scrub supersedes an active wait, soreadiness landing later does not start a picture the viewer stopped.
3. "
StreamVideoSourceis posted to the worker without transfer ownership", and "theweb-video-enginesubpath statically evaluates the browser root entry."Both fixed. A
ReadableStreamcannot be structured-cloned, so the load hands it to the worker onthe message's transfer list, which leaves the worker holding the only readable end. A post the port
refuses rejects the caller rather than leaving a promise for the hang timeout to settle.
openWebVideoEngineMediaSourcereads the video twice, once for the frames it presents and once forthe thumbnails and single-frame grabs its sample sink answers, so it takes a URL or a
Blobandrefuses a stream before it opens anything. A host holding a one-shot stream drives
WebVideoEnginedirectly, where the stream has one reader.
The adapter both entries share is emitted as its own module, and the subpath names that module rather
than the root, so importing
supervision/web-video-enginedoes not evaluate the browser package'sroot entry. Both entries still export one adapter, identity-equal, so the two import paths stay
interchangeable.
What breaks
This takes
supervisionto 0.2.0-next.0, published on thenexttag.lateststays on 0.1.7 until0.2.0 goes out from
main. The pinned public surface goes from 405 exported names onmainto 425:20 added, none removed. The engine's own names are not among them: they reach consumers at the
supervision/web-video-enginesubpath. Every break below is a change to the shape of a type, or to what adefault does. Rows are ordered by how easily each slips past a consumer.
BaseInteractionStyleOptions:shape,cornerRadius,stroke,fill. All four were already@deprecatedonmain.hovered.boxStyleandselected.boxStyle, which reach mask, label, keypoint, polygon and polyline highlights too.requiredForPlaybackis nowrequiredForCoverage.falsereverts to the defaulttrue, so the composed source waits on that entry again.protectedresolvers are gone fromBaseInteractionStyle:resolveBoxInstruction,resolveShape,resolveCornerRadius,resolveStroke,resolveFill.noImplicitOverride. A subclass that overrode one keeps compiling and stops being called.hovered.boxStyleandselected.boxStyle.UnsupportedFormat, wheremainfailed it asNoVideoTrack.NoVideoTrackstops matching that file and falls through to your generic handler.UnsupportedFormatas well. A container whose tracks read and carry no video still fails asNoVideoTrack.maxCachedChunksfor a fixed cap.MediaRenderergains four required members:togglePlayback(),scrub(),getRenderCount(),getPreparedAnnotationWindow().Pick<MediaRenderer, ...>. Anyone who only callscreateMediaRenderer()is untouched.maxDevicePixelRatioleft unset now caps the presentation surface at 2, wheremainrasterized at the display's own ratio.window.devicePixelRatioexplicitly for the old behaviour. The cap is what puts the surface, the mask rasters and the decode on one grid. A mask raster can only be sampled nearest, so a grid it did not share showed as stair-stepped edges.shadowStroke: nulltoBasePolylineStyleto draw the path bare.mainre-derived every 0.5 s.detections.buffer.refreshIntervalSecondsfor the old cadence. Streams are unchanged at 0.25 s.mainbuffered ten ahead and half a second behind. Core's own defaults move the same way, from five and half a second.main's; what changed is how much ground behind the playhead stays buffered. A narrower lookahead was measured and rejected: over 48 runs six seconds ahead lost to ten in 11 of the 12 backward cells and tied in all 12 forward ones, so the window was widened rather than shifted. If you measured memory, the window is 15 seconds against 10.5.detections: { buffer: { bufferAheadSeconds: 10, bufferBehindSeconds: 0.5 } }.VideoSource.idis removed fromUrlVideoSource,BlobVideoSourceandStreamVideoSource.supervision, and reach consumers only atsupervision/web-video-engine, so no released consumer can be holding it.playbackGateis not on this list, and that is deliberate. The render-preparation gate alreadydefaulted to enabled on
main, and the detection gate already defaulted on for appendable sessions.Both are unchanged. What is new is the
playbackGateboolean itself: an off switch, and a way toturn the detection half on for a session that is not appendable. Nothing an existing consumer does
starts behaving differently.
Two more are changes in output rather than removals.
maincompares theplayhead against the frame's media time exactly. On a source whose frame timestamps are not whole
milliseconds, a playhead that rounds down selected the previous detection frame. Sources on
exact-millisecond timestamps are unchanged.
NearestFrameIndexmode the grid step is measured from the buffered frames' own mediatimes.
frameRateis the fallback when the buffered indexes cannot give a step. With no indexedframe at all the mode does not apply, and selection matches by interval instead. A caller whose
rate matched the clip sees no change. A caller who passed a nominal rate the clip does not run at
was previously walked off the grid by the accumulating difference.
MediaRendererStategains five optional fields, so an existing renderer still satisfies the type.drawnMaskFrameTimemaskHeldStaleplaybackGateReachOfforEveryFrame.seekingplaybackStatecannot say so.scrubbingseekinganswers for the transport. The transport settles one message before the landed framereaches the main thread. A host that needs "is the right picture up" must compare the presented
frame's own media time instead. A scrub sets
seekingon every tick, so a host that draws a waitindicator must read
scrubbingfirst.Deprecated
MediaRendererOptions.mutedMediaSessionRendererOptions.mutedDetectionFrameSelectionOptions.frameIndexOriginTimeNeither
mutedoption was ever read, so nothing sounded different before or after. Audio playbackis deferred.
The main-thread cost
Every annotation is drawn on the page's own thread. The engine decodes off it. The picture and the
boxes, masks, labels, polygons, keypoints and focus over it are composited by Pixi on the main
thread, in one synchronous block per presented frame.
Measured on the reference machine and clip, playing from
t=5s, three runs of a 6.0-second windowholding 180 presented frames:
presentVideoFrame, entry to returnThe frame period is 33 ms. Annotations cost 0.03 to 0.07 ms of the block.
What you see when the budget runs out is the picture falling behind. You never see annotations from
the wrong moment: the frame and every layer over it are drawn from one media time, in one block
nothing can interrupt.
A host application shares this thread with its own work. The direction is to make the block smaller
rather than move it to a worker, and the ceiling on what moving it would buy is known: the block is
1.08 to 1.14 ms of the 7.23 to 7.50 ms the thread is busy, so the rest of the thread bounds the win.
docs/internal/video-engine-presentation.mddocuments the mechanism. The figures above come froma CDP profiling run over the demo, which is not committed.
Tradeoffs
supervision, on its own import path.import()keeps it out of the bundle, so an app that only creates a media session emits no engine asset.<video>element are refused here.WebVideoEngineErrorCodenames which limit was hit.demo/fixturesis 306 MB tracked over 107 files. Every clone and every CI run pays it.Known limitations
<video>elementplays the file, but
VideoDecoderreports the testedhvc1andhev1configurations unsupported.The engine refuses the file at load with
DecodeUnsupported, before any frame is presented. Thebuilt-in URL/File source has no software fallback. This is scoped evidence, not a claim about every
HEVC profile or non-Chromium browser: Safari 18.6 reports both tested configurations supported and
plays the same file. The 9-second basketball fixture is H.264 and plays in Firefox.
Safari 18.6 reaches this fallback because it has no WebGPU. Firefox reaches it because its WebGPU
queue rejects a decoded frame. On the tested Safari 18.6/reference-Mac route, staging dominated the
recorded playback wall time; this is a scoped measurement, not a cross-device browser guarantee.
Eligible Android H.264 decoder-session output may instead materialize owned pixels before WebGPU
performs the final upload.
Passing
renderPreparation.maskFrame.displayis what makes the raster follow what the screen canshow. The demo passes one. The presentation numbers above are optimistic for an integration that
has not opted in.
draws another frame's mask: an unprepared mask is cleared, preparation is scheduled, and atomic
presentation keeps every drawn layer on the presented frame's identity.
StreamVideoSourcevariant is declared but no test or demo exercises it. A stream cannotbe re-opened, so the decoder-recovery path degrades instead of rebuilding on one.
Three defects that ship on
maintodayAll three are fixed here, and none of the fixes is on
main. The code each one lives in was therefirst: region effects and their fixture landed on
mainbefore this branch, the prepared-windowtimeline has been there since the shape-primitives work, and the interaction layer has followed a
selected detection across frames since before this branch opened.
The region-effects lens jumped off a player's head, frame after frame. Some lenses floated over
the crowd with nobody under them. A head the model did not see was moved by however far the player's
whole bounding box moved, and that box is set by whichever limb reaches furthest, usually a raised
arm. An invented head now sits between the two real observations on either side of it.
A detection selected while scrubbing vanished for good the first time its annotations were late.
Scrubbing backward is where they are most often late, so the selection usually died within a frame or
two of the first drag, and picking the detection again was the only way back. An absent frame and a
detection that had genuinely left the video both rebased to nothing, and the caller wrote that empty
result over the selection. The follow step now leaves a selection alone while data is missing and
adjudicates on the next frame that has any.
On a looping clip the prepared render window ranked a frame from the previous lap as the furthest
thing prepared. Seventy seconds of footage reported 66.86 seconds of readiness for 211 frames
covering seven. That number is not a readout: it is compared against the lookahead a session asks
for before playback is considered ready, so a wrong value can hold or release the gate for the wrong
reason.
Reading the diff
The pull path is unchanged. The push path, the transport, the frame-present walk and the
prepared-annotation window are new files, reached only through a presented-frame channel. Today only
the video engine drives that channel. The pull path keeps its three ticker callbacks and its draw
order. That is the split worth holding in mind while reading the renderer diff.
Almost every deletion is fixture data. 2,191,256 of 2,197,601 deleted lines sit under
demo/fixtures, because the detection payloads are no longer pretty-printed. Outside thosefixtures the diff is 425 files, 88,464 insertions against 6,345 deletions. That is the code to
review.
The fixture data itself differs from
main. The SAM3 fixtures are generated against the sourcevideos at their native frame rate rather than a resampled proxy. The clearest case is the basketball
sample. On
mainits manifest reads 270 frames at 30fps againstbasketball_sample.normalized.webm.Here it reads 225 frames at 25fps against
basketball_sample.mp4, the clip's own rate. Loading everyfixture in the demo picker covers this better than reading the diff does.
What the fixtures cost a clone.
demo/fixturesis 306 MB tracked over 107 files, in arepository whose
.gitis 772 MB.horse_trail1min-horse-video.mov, 128 MB, the media the demo playsbasketball_sam3raw-sam3.jsonl, 11 MBbasketball_samplebasketball_sample.mp4, 22 MBbasketball_regionshead-detections.json, 9 MBhorse_trail/raw-sam3.jsonlis 44 MB of raw model output kept for provenance beside the 59 MB ofchunked detections derived from it. Nothing loads it at runtime. It is worth deciding deliberately,
since it is what every reviewer and every CI run pays to clone.
Packaging and release
The engine does not publish on its own.
packages/video-engineis a private workspace, and itsbrowser build is staged into
supervisionunderdist/web-video-engine. Consumers reach it byimport path:
The subpaths are
supervision/web-video-engine,supervision/web-video-engine/analysisandsupervision/web-video-engine/worker.createWebVideoEngineMediaRendererSourceandopenWebVideoEngineMediaSourceare exported from the package root as well, and are the same functionin both places.
There is no second install and no optional peer dependency. Installing
supervisioninstalls theengine, because the staged build is inside the tarball. The tarball grows from 654,101 to 1,732,755
bytes, and every consumer pays that download even if it never imports the engine. The bundle cost
stays conditional. The engine is reached by a dynamic
import(), so an app that imports onlycreateMediaSessionemits 1,750,666 bytes and no engine asset, while adding the engine adapteremits 3,278,684 bytes with the engine in its own 1,503,131-byte chunk. Still images and camera input
never load it. Opening a video file does. If that chunk does not load, the video path throws an
error naming
supervision/web-video-engineand saying the engine is a lazily loaded chunk ofsupervision, rather than a bundler stack trace naming a hashed asset.The release workflow publishes one package. It builds the video-engine workspace, stages that build
into
dist/web-video-engine, and deletes the engine'sfile:devDependency from the packedmanifest. It then builds the portable tarball, smoke-tests it in a clean consumer, and publishes
supervision. A releasedsupervisiontherefore names no engine package and no engine version.After the upload the workflow polls
npm view supervision@<dist_tag>up to twelve times atfive-second intervals, until the dist-tag resolves to the version it just published. The workflow
publishes from
main, or from arelease/*branch whendist_tagisnext.No release step needs a person.
supervisionis already on npm, so its trusted publisher isalready attached. The workflow publishes the generated tarball with
npm publishand authenticatesthrough OIDC. It needs no
npm loginand noNPM_TOKEN. The engine is private and is neverpublished, so there is no second name to register.
Two things that will not warn you
A custom
workerFactorymust match the host's version. The mask preparation protocol changed.The artifact kind is
idMaskrather thanpngIdMask, the payload field israsterrather thanpng, and the job carries amaxRasterWidth. None of those types is exported, so nothing warns.Point the factory at
supervision/render-preparation-workerand this cannot happen.Content Security Policy is unaffected. This package already spawns classic blob workers for mask
preparation and for tracking. The engine's worker needs the same directive and no new one.
Documentation status
docs/publicis the published documentation and it is checked against the code.npm run docs:checkruns 31 checks: every path a document names exists, every npm script it runs is declared, every flag
matches the script that reads it, every checksum matches the file beside it, every version matches
the manifest, every symbol it imports is exported, and every copyable integration example
typechecks. All 31 pass.
Eighteen files under
docs/publicchange here:guides/browser-support.mdapi/video-engine.tsguides/media-sessions.md,guides/detections-and-rendering.md,guides/media-preparation.md,recipes/streaming-detections.md,recipes/multiple-detection-sources.mdguides/application-integration.mdguides/public-api.md,concepts.md,annotation-renderers/polylines.mdapi/media-preparation.ts,api/rendering.ts,api/sessions.tsguides/presentation-styles.md,recipes/interactive-picking.md,recipes/progressive-upload-normalization.mdtypedoc-icons.jsWhat this pull request does not have
the artifact worth recording: load a fixture, scrub backwards with masks on, and watch every
annotation stay on its frame.
at
1a4db5erecorded 91 ms p95 backward-scrub settlement, 3.7 ms p95 seek, 53.5 ms p95 step,zero reported drops, and zero reported engine stalls. It had no retained comparison baseline and
predates the accepted scrub-scheduling and Android-ownership changes, so it is not performance
evidence for the current head.
.gitignoreexcludestools/demo-eval/report.jsonandtools/demo-eval/baseline.json, because those numbers only mean anything on the machine thatrecorded them. Reproduce with
npm run eval:demo -- --url 'http://localhost:5173/?mediaPath=engine'.Android testing covers frame identity and playback cadence, not comparative scrub or composition
cost on slower hardware.
on one side only. The other side needs a harness story that lives in the engine repository.