Parallelize and speed up the thumbnail pipeline - #153
Merged
Conversation
processGalleryThumbnails processed media files strictly one at a time, which dominates runtime on large galleries. Thumbnailing is IO/CPU bound and Sharp and ffmpeg release the JS thread, so the work parallelizes well. Add a small dependency-free mapWithConcurrency worker-pool helper (order-preserving) and use it to process each section's media with a pool sized to the available cores (cpus - 1). Per-file error isolation is unchanged. On a multi-core machine a large gallery now builds several times faster. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Thumbnails were always encoded as AVIF with Sharp's default settings. AVIF gives small files but is slow to encode, which dominates build time on large galleries. Expose format (avif/webp/jpeg), quality and encoder effort through the existing 4-level config hierarchy (CLI > gallery.json > theme > defaults) so users can trade size for speed (e.g. webp with low effort) on big collections. - Extend ThumbnailConfigSchema and the gallery.json schema with the new optional fields so they validate and round-trip. - resizeImage/createImageThumbnails/createVideoThumbnails pass the encode options through to Sharp, clamping effort for webp and dropping it for jpeg. - Derive the thumbnail file extension from the format, and thread the resolved format into getSubgalleryThumbnailPath so sub-gallery thumbnail links match. - Add --thumbnail-format/--thumbnail-quality/--thumbnail-effort flags to the build and thumbnails commands; build persists them to gallery.json like the existing size/edge flags. Defaults are unchanged (avif, Sharp defaults), so existing galleries produce identical output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
For every image, the pipeline read the source from disk twice (once per thumbnail size, because Sharp re-reads a file-path input for each output) and then re-read and decoded the just-written thumbnail a third time to compute the BlurHash. Read the original into a Buffer once and reuse it for metadata, both thumbnail encodes and the BlurHash, and read EXIF from the same buffer. The BlurHash is now computed from the in-memory original instead of the written thumbnail, which is visually equivalent. The video frame is likewise read once and decoded from memory for both outputs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gallery.json (which records each file's lastMediaTimestamp) was only written once, after every thumbnail in the gallery finished. If a long run was interrupted - Ctrl-C, a crash, an OOM - all of that bookkeeping was lost and the next run regenerated every thumbnail from scratch even though the images were already on disk. Write gallery.json every 16 processed files and again on SIGINT/SIGTERM, using an atomic temp-file-and-rename so an interruption can never leave a truncated gallery.json. Combined with the existing mtime skip check, an interrupted run now resumes where it left off. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: feb48c0ab2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
haltakov
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Speeds up thumbnail generation (
spg thumbnailsand the build's thumbnail step) and makes thumbnail output configurable.max(2, cpus - 1)) instead of one at a time. Sharp and ffmpeg release the JS thread, so large galleries finish several times faster, while per-file error isolation is preserved.avif(default),webporjpeg, with configurablequality(1–100) and encodereffort. Settable via gallery.json, theme config or CLI flags, merged with the hierarchy CLI > gallery.json > theme > defaults.SIGINT/SIGTERM, writing atomically (temp file + rename). An interrupted run resumes instead of regenerating thumbnails already on disk.New CLI flags (
thumbnailsandbuild)--thumbnail-format <avif|webp|jpeg>--thumbnail-quality <1-100>--thumbnail-effort <number>— higher is slower but produces smaller filesTests
concurrency.test.ts— bounded concurrency and input-order preservationthumbnail-config.test.ts— format/quality/effort resolution across the config hierarchythumbnails-progress.test.ts— incremental persistence and resume-after-interruption