Skip to content

Allow conversion to be applied to books already in the library - #160

Open
elric667 wants to merge 5 commits into
Chaptarr:developfrom
elric667:feature/convert-existing-library-files
Open

elric667 wants to merge 5 commits into
Chaptarr:developfrom
elric667:feature/convert-existing-library-files

Conversation

@elric667

Copy link
Copy Markdown

Description

MP3 → M4B conversion only ran for files arriving from a tracked download client, so a
collection that predates Chaptarr could never be normalised to one format.
ImportApprovedBooks.ConvertBookGroupIfNeeded returned early whenever there was no
DownloadClientItem, and nothing else could reach the converter.

This implements both suggestions from #109:

  1. The download id is now synthesised when no download client owns the import. It was
    only ever a correlation key for progress tracking and the .chaptarr-conversions work
    folder, so it is derived from the book and edition instead (chaptarr-local-{bookId}-{editionId}).
    Keeping it stable means a retried conversion still finds the artifact retained by the
    previous attempt. On its own this makes manual import convert — see Behaviour changes.

  2. A ConvertBookFiles command (plus a bulk ConvertAuthor), scoped like RenameFiles.
    BookFileConversionService takes existing BookFile rows, rebuilds import decisions with
    the book and edition already known, and hands them back to the import pipeline flagged as a
    library conversion. That reuses ffmpeg detection, the workspace free-space check, the
    concurrency semaphore, tagging, chapter insertion, destination naming and the recycle bin
    rather than duplicating any of it.

Also added:

  • GET /api/v1/convert?authorId=&bookId= — previews what would convert, and why a file would
    be skipped (no conversion target on the profile, target not allowed by the profile, already
    M4B, missing from disk, Calibre-managed).
  • A Convert Files toolbar action on the author and book detail pages, with a preview modal
    modelled on the existing Organize/Retag modals.

Three design decisions worth reviewing:

  • A library conversion replaces only the files it converted. Reusing the manual-import path
    was tempting, but manualReplaceExisting sweeps in every other file on the book —
    converting an audiobook would have deleted the book's eBook. Hence the new
    LocalBook.IsLibraryConversion flag, with replacement scoped to the conversion's own source
    paths.
  • It bypasses quality gating. The profile that wants everything as M4B is usually the same
    profile that disallows MP3, so gating would reject the library's own files before they
    reached the converter. This matches how manual imports and forced grabs are already treated.
  • Selection is per book, not per file. Every convertible file of a book becomes one output
    file, so converting 5 of 18 tracks would leave the book half MP3 and half M4B. The modal
    selects whole books, and the service expands a partial files selection to the whole edition
    (pulling in only siblings that would convert anyway).

Detached conversion jobs remain limited to tracked downloads. They signal completion by pushing
ProcessMonitoredDownloadsCommand, which has no way to resume a library conversion, so manual
imports and library conversions convert inline on the calling thread — the path manual imports
would already have taken.

Behaviour changes

  • Manual imports now convert when the quality profile has a conversion target. This follows
    directly from fix (1) and is what ConvertToQualityId reads like it promises, but it is a
    change for anyone currently relying on manual import bypassing conversion. Happy to gate it
    behind a separate opt-in if you would rather it stayed opt-in.
  • DownloadedBooksScan of a folder with no tracked download now converts too, for the same
    reason — it is the other caller that passes a null DownloadClientItem. This matches the
    profile setting's own help text ("After download, convert compatible files…"). Note that the
    hasRejectedTrackedDownloadDecisions guard only applies to tracked downloads, so an untracked
    folder with some unmatched files will convert the files that did match.
  • Library rescans are not affected. DiskScanService goes through ImportOrchestrator and
    never calls ImportApprovedBooks.Import, so a scheduled rescan will not start converting an
    existing library on its own. ImportApprovedBooks.Import has exactly three callers:
    ManualImportService, DownloadedBooksImportService, and the new BookFileConversionService.
  • ConvertBookFiles / ConvertAuthor declare RequiresDiskAccess and IsLongRunning, so a
    large conversion run serialises against other disk commands (rescan, rename, download import)
    the same way RenameAuthor does. That is deliberate — it is writing into the library — but
    it does mean a long run holds the disk lane.

On the "adjacent" point in #109: ConvertMp3ToM4b is already mostly neutralised — migration 076
clears it and QualityProfileResource derives it from ConvertToQualityId. Only the DB column
and two en.json strings linger. I left it alone rather than touch migrations in this PR.

Fixes #109

Database Migration

NO. No schema changes, no new migration. ConversionJob rows are unchanged — library
conversions do not create them.

How was this tested?

Built and tested from source on Linux (.NET 10.0.401, Node 22):

  • dotnet build src/NzbDrone.Core/Chaptarr.Core.csproj — clean
  • dotnet build src/Chaptarr.Api.V1/Chaptarr.Api.V1.csproj — clean
  • dotnet test src/Chaptarr.Core.Test/Chaptarr.Core.Test.csproj — 3033 passed, 0 failed,
    including the 13 new tests
  • yarn build — compiles clean
  • yarn typecheck — clean
  • yarn check-translations — all translate() keys present in en.json
  • yarn lint — no new errors introduced (verified per-file against the pre-change baseline;
    the new frontend/src/Convert/ files are lint-clean)
  • stylelint — clean on the new frontend/src/Convert/*.css

New test fixture BookFileConversionServiceFixture (13 tests) covers eligibility and the
skip reasons, per-edition grouping, the IsLibraryConversion / replaceExisting contract,
whole-book expansion from a partial selection, and not pulling in non-convertible siblings.

End to end, in Docker. Built the image with the repo's own Dockerfile.build from this
branch, exported without .git (the same as building from a GitHub tarball). The only changes
were two local build workarounds, neither part of this PR: my sandbox's proxy CA certificate,
and the NuGetAudit workaround described below. Ran it with a fresh config, and drove it through both the API and the real UI in headless Chromium. The image
includes the real ffmpeg, m4b-tool and mp4v2 tools, so these are genuine conversions:

Scenario Result
Library conversion, API. 3-file MP3 book already in the library; profile Convert To set afterwards; ConvertBookFiles sent with only 1 of the 3 files selected One 60.0s M4B with 3 chapters at the file boundaries and correct title/artist/album tags. All 3 MP3s removed (whole-book expansion works) and their rows deleted. Originals landed in the Recycle Bin. No .chaptarr-conversions folder left behind.
Ebook safety. The MP3s and an EPUB of the same book ended up on the same edition (manual import put them there) EPUB untouched, on disk and in the DB. Under the old same-edition replacement rule this EPUB would have been deleted; this is the case the new filesToReplace narrowing exists for.
Upgrades disabled. Default Audiobook profile has Upgrade Allowed off Library conversion still proceeds, as intended for an explicit user action.
Manual import converts (fix 1). Target set, then 2 MP3s manually imported Imported directly as one 30.0s M4B with 2 chapters. Source MP3s are left in the incoming folder (non-destructive, same as the download path).
Book page UI. Convert Files → modal → Convert UI posts {"name":"ConvertBookFiles","authorId":1,"bookId":3,"files":[…4 ids]}; the book goes from 4 MP3 files to 1 M4B.
Author page UI. Mix of convertible and already-converted books Already-converted books show Already M4B with a disabled checkbox. Convert posts without a bookId and converts only the selected book.
Preview reasons. No target set / target set / already M4B "Quality profile 'Audiobook' has no conversion target set" → convertible → "Already M4B". EPUBs are omitted from the list.

The UI run caught one bug of mine, a mis-styled select-all checkbox in the modal, fixed in the
second commit.

Notes for reviewers:

  • Dockerfile.build fails on develop right now, independent of this PR. dotnet restore
    stops with NU1902: Warning As Error: Package 'Microsoft.Build.Tasks.Git' 8.0.0 has a known moderate severity vulnerability (GHSA-23fw-v26w-5fgq). It comes in via
    Microsoft.SourceLink.GitHub 8.0.0 plus TreatWarningsAsErrors, and it fails the same way on
    the untouched develop commit. To build for testing I added /p:NuGetAudit=false to that one
    restore line locally; it is not part of this PR. Bumping SourceLink probably fixes it, and is
    worth its own PR.
  • Profile changes can take up to 10 seconds to show in the preview, because of
    AuthorService's 10-second author cache. That is pre-existing and affects every
    profile-dependent feature.
  • Originals in the Recycle Bin keep the pre-existing upgrade-staging suffix
    (….mp3.chaptarr-upgrade~<guid>), so restoring one means renaming it.
  • If the converted M4B's folder name differs from the MP3s' folder, the old folder is left
    empty. Cosmetic; happy to add cleanup if you want it here.
  • Only tested with synthetic tone MP3s, not a real multi-hour audiobook. Conversion time and
    memory on long books are the existing converter's behaviour, which this PR does not change.

Screenshots (UI changes only)

Convert Files on the book page, next to Preview Rename and Preview Retag. Before: 4 MP3 files.

image

The preview modal at author level: one convertible book, others already M4B with the reason shown.

image

After converting: the book has a single M4B.

image

A note on AI: taking you up on the disclosure paragraph in the template — this change was
written with Claude Code (Claude Agent SDK), using
Claude Opus 5 (claude-opus-5) at xhigh reasoning effort. The investigation, design
decisions, implementation, tests and this description were all AI-generated; I reviewed them
before opening the PR. Please scrutinise it as hard as you like — the three design decisions
called out above and the manual-import behaviour change are the places I'd start. The
end-to-end runs above used synthetic MP3s, so a real multi-hour audiobook is the remaining gap.

elric667 and others added 2 commits September 19, 2026 07:00
MP3 -> M4B conversion only ran for files arriving from a tracked download
client, so a collection that predates Chaptarr could never be normalised to
one format. ConvertBookGroupIfNeeded returned early whenever there was no
DownloadClientItem, and nothing else could reach the converter.

The download id was only ever a correlation key, so it is now synthesised
from the book and edition when no download client owns the import. That alone
lets a manual import convert. On top of it, a ConvertBookFiles command (and a
bulk ConvertAuthor) hands existing BookFile rows back to the import pipeline
as a library conversion, reusing ffmpeg detection, the workspace free-space
check, the concurrency semaphore, tagging, chapters, destination naming and
the recycle bin rather than duplicating any of it.

A library conversion replaces exactly the files it converted. It does not
inherit the manual-import behaviour of replacing every other file on the
book, so converting an audiobook leaves the book's eBook and any second
audiobook edition alone. It also bypasses quality gating, since the profile
that wants everything as M4B is usually the one that disallows MP3.

Selection is per book in the UI and expanded to the whole edition in the
service: every convertible file of a book becomes one output file, so
converting a subset would leave the book half MP3 and half M4B.

- GET /api/v1/convert?authorId=&bookId= previews what would convert and why
  a file would be skipped
- Convert Files toolbar action on the author and book detail pages
- Detached conversion jobs stay limited to tracked downloads, which is the
  only path that has a way to resume the import

Fixes Chaptarr#109

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GXPbRG8Geef4zMeMQgpQ8H
The footer select-all checkbox passed a custom className to CheckInput
without composing the base input class, so the checkbox lost its size and
rendered as a thin sliver. Compose from CheckInput.css the same way the
Organize modal does, and fix the property order stylelint flagged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GXPbRG8Geef4zMeMQgpQ8H
@robertlordhood

Copy link
Copy Markdown
Contributor

Ty for this. One downside of this that I see is that I believe this will reset all stats for these books in downstrem consumers/libraries like ABS, etc. So for instance, if you're half way through your audiobook and convert it to m4b from mp3 in chaptarr, your progress is gone. But, if you use ABS (not sure, I'll have to check, or you can and reply that's helpful!) if you convert that same book it keeps/stores your progress?

I'm sure there's ways around it, or some users just won't care, but something to consider.

@elric667

Copy link
Copy Markdown
Author

I will have to test that thoroughly, but my use of ABS is long as the directory doesn't change, causing it to re-index or create a new entry in ABS. I don't lose my progress. But I've done this inside of ABS using its ability to merge M4B.

A library conversion wrote the M4B to the folder the naming scheme calls
for. When a library predates Chaptarr, that is usually a different folder
from the one the MP3s were in, and tools that key books by folder see a new
book: Audiobookshelf marked the old item missing and created a new one with
no listening progress. Converting in place, which is what Audiobookshelf's
own M4B tool does, keeps the same item and the listener's position.

Library conversions now write to the folder the source files share, or to
their parent when they sit in disc subfolders (CD1, Disc 2), and apply
naming to the file name only. Anything that does not form one book folder
inside the author's root folder falls back to normal naming. Downloads and
manual imports are unchanged, and Organize still moves a book to the
naming scheme on request.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GXPbRG8Geef4zMeMQgpQ8H
@elric667

Copy link
Copy Markdown
Author

Thanks, good catch. I tested it against Audiobookshelf, and you were right for one specific case. That case is now fixed in the PR.

Short version: Audiobookshelf ties listening progress to the book's folder (its library item), not to the audio files inside it. Progress survives a conversion as long as the M4B stays in the folder the MP3s were in. It's lost when the M4B lands in a different folder, because ABS then sees a brand-new book, and the old one shows as missing.

The PR originally put the converted M4B where Chaptarr's naming scheme says the book should live. For a library that predates Chaptarr (the main audience for #109), that's often a different folder name from the existing one, so progress was lost in exactly the case that matters most. I've pushed a fix: library conversions now happen in place, the same way ABS's own M4B tool does it.

Results

ABS 2.36.1, same library folder as Chaptarr. Each book had progress set at 1:15 of 2:00 before converting:

Scenario Where the M4B landed ABS after conversion Progress
Chaptarr, book already in a Chaptarr-named folder same folder same item, now 1 M4B ✅ kept at 1:15
Chaptarr before the fix, book in a pre-Chaptarr folder (… (Unabridged) [MP3]) new Chaptarr-named folder old item missing, new item created ❌ new item has none
Chaptarr after the fix, same pre-Chaptarr layout same folder same item, now 1 M4B ✅ kept at 1:15
ABS's own M4B encoder (for comparison) same folder same item, now 1 M4B ✅ kept at 1:15

So after the fix, converting in Chaptarr behaves the same as converting in ABS as far as progress goes.

What changed

New commit: Keep library conversions in the book's existing folder

  • A library conversion writes the M4B into the folder the source files are already in. Naming is applied to the file name only.
  • Disc subfolders (CD1, Disc 2, …) resolve to their parent book folder, which is how ABS reads them as one book.
  • Anything that doesn't form one book folder inside the root folder (files spread across sibling folders, outside the root, loose in the root) falls back to normal naming.
  • Downloads and manual imports are unchanged. Organize/Rename still moves a book into the naming scheme when you ask it to.

How it was tested

  1. Built the Docker image from this branch using the repo's Dockerfile.build, so the conversions used the real ffmpeg, m4b-tool and mp4v2.
  2. Ran Audiobookshelf 2.36.1 in a second container, pointed at the same library folder, with its folder watcher on (the default).
  3. Put MP3 audiobooks (4 × 30-second files each) into Chaptarr's library, including one moved into a pre-Chaptarr-style folder name with Chaptarr's records pointing at it there.
  4. Scanned them into ABS and set listening progress to 75 seconds (62.5%) on each through ABS's API.
  5. Converted with Chaptarr's new ConvertBookFiles command (what the Convert Files button sends), and one book with ABS's own M4B encoder for comparison.
  6. Checked ABS twice: after its folder watcher reacted, and again after a full library scan. Both gave the same results. For each book I recorded whether the original item ID survived, what files it held, and the progress stored against it.
  7. Reran the pre-Chaptarr-folder case on a fresh book with the fix applied.
  8. Checked the converted file's timing: 120.05s against 120.00s of MP3s, with chapters at the original file boundaries. A saved position lands within a fraction of a second of the same spot in the story. m4b-tool adds about 0.05s per file boundary, so a book with dozens of files could drift a few seconds by the end.
  9. Added unit tests for the in-place rule and its fallbacks. The full suite passes: 3039 tests, 0 failures.

Caveats

  • I only tested Audiobookshelf. Plex, Booksonic, Jellyfin and the rest weren't tested. Anything that keys books by folder should behave the same way, but I haven't confirmed that.
  • Running Organize/Rename afterwards to move a book into the naming scheme will still look like a new book to ABS. That's true of any rename today and isn't specific to conversion.
  • Where the originals go differs: Chaptarr sends the MP3s to its Recycle Bin (if one is configured), while ABS's encoder moves them into ABS's metadata folder.

(Tested with Claude Code, same as the rest of the PR.)

@robertlordhood

Copy link
Copy Markdown
Contributor

ty for the dd. I'll take a look!

A library conversion ran as a task with a single static message, so a long
book gave no sign of how far along it was, and there was no way to stop it:
the conversion handler ignored the task's cancellation token, so Cancel in
System > Tasks had no effect.

The converter already reports its progress to the conversion tracker, but
from its own threads, where the task's status message cannot see it. While
a book converts, poll the tracker and mirror the percentage into the task
message, so it shows in the sidebar and in System > Tasks.

The handlers now take the task's cancellation token and pass it to the
import, which links it to the converter's own cancellation. Cancelling the
task stops m4b-tool, cleans up the work folder, and leaves the book's
original files untouched, since nothing is replaced until a conversion
finishes. A multi-book run stops before starting the next book.

The tracking id is now built by one shared helper, so the import pipeline
and the conversion service cannot disagree on it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GXPbRG8Geef4zMeMQgpQ8H
@elric667

Copy link
Copy Markdown
Author

Pushed one more commit: Show conversion progress and let a running conversion be cancelled

I found this while testing on my own Synology library. A long conversion only showed "Converting 'Harvest' to M4B (1/1)", with no progress. Cancel in System → Tasks did nothing, because the conversion handler ignored the task's cancellation token.

What changed

  • Progress: the task message now includes the converter's percentage, e.g. "Converting 'Harvest' to M4B (1/1) - 42%". It shows in the sidebar, and on the System → Tasks status icon on hover. The converter already reports progress to the conversion tracker, but from its own threads, so the service polls the tracker and mirrors the percentage into the task message.
  • Cancel: the handlers now take the task's cancellation token and pass it to the import, which links it to the converter's own cancellation. Cancel in System → Tasks stops m4b-tool, cleans up the work folder, and leaves the original files untouched, since nothing is replaced until a conversion finishes. A multi-book run stops before starting the next book.
  • The conversion tracking id now comes from one shared helper (LocalConversionId), so the import pipeline and the conversion service can't drift apart.

Tested in Docker with the real ffmpeg/m4b-tool, using a 2-hour, 6-file MP3 book:

  • Progress went 1% → 15% → 63%, then I cancelled via System → Tasks → × → Yes, Cancel.
  • The task showed Cancelled within 1 second, m4b-tool was stopped, and no converter processes were left.
  • The MP3s were untouched on disk and in the database, with no partial M4B or work folder left behind.
  • Converting the same book again afterwards completed normally: a 2.00 h M4B with 6 chapters.
  • 4 new unit tests; the full suite passes (3043).

The percentage jumps rather than creeping, since m4b-tool encodes several files in parallel, and it holds in the 90s while it merges and writes chapters.

(Built and tested with Claude Code, same as the rest of the PR.)

A multi-file conversion is merged into one M4B that has no part number, so
the import names it "Title.m4b". The destination preview copied the first
source file's part fields (1 of N), so it was named "Title - 01.m4b" (or
"Title (1).m4b"). The conflict check before conversion looked at that path,
found nothing, and a taken destination was only found after a full
conversion.

The preview now carries the same part values as the converted file. The
check finds the real destination before any conversion runs, and the work
folder output gets the final name.

Fixes Chaptarr#284

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GXPbRG8Geef4zMeMQgpQ8H
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MP3 to M4B conversion cannot be applied to an existing library, only to new downloads

2 participants