Fix Webshart video sample and cache handling - #3009
Draft
bghira wants to merge 2 commits into
Draft
Conversation
added 2 commits
August 6, 2026 18:35
Normalize malformed webshart:/ sample IDs before path handling so cache keys and backend lookups resolve to the same logical sample. Honor webshart_shard_cache_gb=0 by disabling whole-shard caching instead of falling back to the default cache size, and reject negative cache sizes. Add video-aware metadata and sample loading for Webshart shards, including ffprobe-based bucket derivation when indexed metadata is incomplete. Reset compressed torch streams before fallback loading so cache reads do not depend on the caller's stream position. Document the zero shard-cache behavior and cover URI normalization, cache keys, builder settings, metadata, and video-loading paths with tests.
Add sparse range probing for remote Webshart video samples so ffprobe can read container metadata without downloading full TAR members when offset and length metadata are available. Prepare Webshart video bucket metadata in bounded parallel chunks using the backend download parallelism setting, while preserving the existing periodic cache flush behavior. Make sample limiting and shuffled distributed bucket splitting independent of discovery order so every rank partitions the same serialized cache contents. Reload refreshed bucket caches on every rank before splitting and prevent rank-local prompt records from being split a second time during text embed cache preparation. Cover range reads, sparse probing, deterministic sample limiting, deterministic bucket shuffling, bucket cache reloads, and rank-local text embedding cache behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes several Webshart data-path, cache, metadata, and distributed preparation edge cases that were found while separating the H3 branch into independently reviewable work.
webshart:/sample IDs back to canonicalwebshart://identifiers before cache key and backend lookup logic runs.webshart_shard_cache_gb=0as an explicit request to disable whole-shard caching, while rejecting negative cache sizes.Validation
Ran on the PR branch:
Result: 206 tests passed.
Also checked the committed files for
/Users/kashand/Users/path leaks; none were found.