Skip to content

Fix UTF-8 char boundary panic in snippet_from_doc - #6

Merged
mrorigo merged 1 commit into
mainfrom
fix/snippet-utf8-char-boundary
Oct 10, 2026
Merged

mrorigo merged 1 commit into
mainfrom
fix/snippet-utf8-char-boundary

Conversation

@mrorigo

@mrorigo mrorigo commented Oct 10, 2026

Copy link
Copy Markdown
Owner

Problem

okf search panicked on documents containing non-ASCII text:

thread 'main' panicked at src/storage.rs:1056:20:
end byte index 330 is not a char boundary; it is inside 'ℂ' (bytes 328..331 of string)

snippet_from_doc computed byte offsets (pos - 40, pos + len + 80) from a lowercase match on the document body and sliced body directly. Those offsets can land inside a multi-byte UTF-8 character, and Rust panics when slicing at a non-char-boundary index.

Fix

  • Added floor_char_boundary / ceil_char_boundary helpers that snap an arbitrary byte index to the nearest UTF-8 character boundary.
  • snippet_from_doc now snaps both start and end, and guards against a reversed range.
  • Added regression tests covering multibyte bodies for the direct match, offset-shifted match, and no-match fallback paths.

Verification

cargo fmt, cargo clippy --all-targets, and cargo test all pass.

snippet_from_doc computed byte offsets (pos - 40, pos + len + 80) from a
lowercase match and sliced the document body directly. Those offsets can
land inside a multi-byte UTF-8 character, causing a slice-index panic on
documents containing non-ASCII text.

Snap both the start and end offsets to the nearest character boundary and
guard against a reversed range. Add regression tests covering multibyte
bodies for the match, offset-shifted match, and no-match fallback paths.
@mrorigo
mrorigo merged commit a2a288a into main Oct 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant