Skip to content

Fix Webshart video sample and cache handling - #3009

Draft
bghira wants to merge 2 commits into
mainfrom
fix/webshart-video-cache
Draft

Fix Webshart video sample and cache handling#3009
bghira wants to merge 2 commits into
mainfrom
fix/webshart-video-cache

Conversation

@bghira

@bghira bghira commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Summary

This PR fixes several Webshart data-path, cache, metadata, and distributed preparation edge cases that were found while separating the H3 branch into independently reviewable work.

  • Normalize path-normalized webshart:/ sample IDs back to canonical webshart:// identifiers before cache key and backend lookup logic runs.
  • Preserve webshart_shard_cache_gb=0 as an explicit request to disable whole-shard caching, while rejecting negative cache sizes.
  • Add video-aware sample loading and metadata handling for Webshart shards, including ffprobe-backed bucket metadata when indexed metadata is incomplete.
  • Add sparse HTTP range probing for remote TAR members when offset and length metadata are available, so ffprobe can inspect container metadata without downloading the full video sample.
  • Prepare Webshart video bucket metadata in bounded parallel chunks using the backend download parallelism setting.
  • Make max-sample limiting and shuffled distributed bucket splitting independent of discovery/cache reload order so ranks partition the same serialized cache contents.
  • Reload refreshed bucket caches before splitting and avoid splitting rank-local prompt records a second time during text embed cache preparation.
  • Reset compressed torch cache streams before fallback loading so cache reads do not depend on the caller's stream position.
  • Document the zero shard-cache behavior.

Validation

Ran on the PR branch:

.venv/bin/python -m unittest -v -f tests.test_webshart_backend tests.test_backend_builders tests.test_factory_edge_cases tests.test_metadata_backend tests.test_text_embeds tests.test_utils_pathing

Result: 206 tests passed.

Also checked the committed files for /Users/kash and /Users/ path leaks; none were found.

bghira added 2 commits August 6, 2026 18:35
Normalize malformed webshart:/ sample IDs before path handling so cache keys and backend lookups resolve to the same logical sample.

Honor webshart_shard_cache_gb=0 by disabling whole-shard caching instead of falling back to the default cache size, and reject negative cache sizes.

Add video-aware metadata and sample loading for Webshart shards, including ffprobe-based bucket derivation when indexed metadata is incomplete.

Reset compressed torch streams before fallback loading so cache reads do not depend on the caller's stream position.

Document the zero shard-cache behavior and cover URI normalization, cache keys, builder settings, metadata, and video-loading paths with tests.
Add sparse range probing for remote Webshart video samples so ffprobe can read container metadata without downloading full TAR members when offset and length metadata are available.

Prepare Webshart video bucket metadata in bounded parallel chunks using the backend download parallelism setting, while preserving the existing periodic cache flush behavior.

Make sample limiting and shuffled distributed bucket splitting independent of discovery order so every rank partitions the same serialized cache contents.

Reload refreshed bucket caches on every rank before splitting and prevent rank-local prompt records from being split a second time during text embed cache preparation.

Cover range reads, sparse probing, deterministic sample limiting, deterministic bucket shuffling, bucket cache reloads, and rank-local text embedding cache behavior.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant