Unicode-safe text handling: fix #51, the panic beside it, and one normalization form - #52
Merged
Merged
Conversation
…g titles Issue #51 fixed the byte slice that split a `ü`. This covers the rest of the class it belonged to. A `clippy::string_slice` sweep over the app crates enumerated all 28 string index sites. Indices from `find`/`rsplit_once`/`split_at`, and offsets past an ASCII prefix already proven by `starts_with`, are boundary-safe. One site was not: `has_drive_letter` checked characters 1 and 2 but never constrained character 0, so `Ü:\Musik` reached a `path_str[1..]` that splits it in half. The deeper problem is that the same name has more than one spelling. `Füßen` is NFC from a Windows tagger and NFD from macOS, canonically equivalent and sharing no bytes, so `LIKE` and FTS silently said no. Worse, FTS shredded a decomposed term into two tokens, because a combining mark is `Mn` and the tokenizer splits on anything non-alphanumeric — a user searching for their own music found nothing. Stored text is now folded to NFC at the single write funnel and search terms are folded on the way in. Paths are deliberately excluded: a path is a filesystem key, and on ext4 the two spellings are two different files. Rows written before this release are folded once by migration v8, via an `nfc()` SQL scalar, rather than left unfindable until a rescan. Tests pad every sample through eight alignments, because a flat list of awkward strings does not catch an offset bug — the panic needs the slice point to land inside a character, which depends on the string's length.
vyrti
changed the title
vyrti
force-pushed
the
fix/issue-51-unicode-filenames
branch
from
September 14, 2026 19:18
78d04bd to
d246f45
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Issue #51 reported a panic on a German umlaut in a FLAC filename. This PR fixes that exact crash and closes the unchecked-UTF-8-slicing failure class across the application targets.
Why the old tests missed it
The existing large filename suites exercised filesystem path normalization. The panic lived in a separate Samsung-only DIDL title renderer and required all of these conditions together: Samsung client profile, filename fallback or extension handling, and a multibyte character at the byte offset computed from the extension length. Its renderer tests used only ASCII (
movie.mp4,clip.mp4), so the path suites never entered the failing branch.The reported panic, and the rest of its class
clippy::string_slicea crate-level denial and an explicit CI error, so future unchecked UTF-8 slicing cannot merge unnoticedÜ:\Musikcould reach a byte slice insideÜin Windows drive-letter detectionOne normalization form for stored and searched text
The same name has more than one spelling.
Füßenis NFC from a Windows tagger and NFD from macOS—canonically equivalent but byte-different. Stored display text is folded to NFC at the database write funnel, and queries are folded on both FTS andLIKEpaths. Existing rows are folded once by schema migration v8.Paths are deliberately not normalized this way: on a byte-exact filesystem, NFC and NFD can be different files. The display filename is normalized; the filesystem key remains byte-identical.
Why the new tests are materially different
A list of scripts is insufficient for an offset panic. The exhaustive Samsung test places each of the 1,112,064 valid Unicode scalar values exactly where the old suffix calculation would bisect every multibyte scalar. The shared corpus also repeats representative scripts and normalization forms at eight byte alignments, exercising continuation-byte positions.
No finite test can enumerate every possible future string or future code path. The structural guarantee is stronger: the fixed renderer no longer computes a byte offset, and CI now rejects direct string slicing throughout the app targets.
Verification
cargo fmt --all -- --checkcargo clippy -p vuio-core -p vuio-cli -p vuio-cast -p vuio-web --all-targets --all-features --locked -- -D clippy::all -D clippy::string_slice -D warningscargo test --workspace --lockedFixes #51