Skip to content

Fix Unicode handling in Shen string primitives - #19

Merged
pyrex41 merged 1 commit into
mainfrom
fix/unicode-string-primitives
Aug 26, 2026
Merged

pyrex41 merged 1 commit into
mainfrom
fix/unicode-string-primitives

Conversation

@pyrex41

@pyrex41 pyrex41 commented Aug 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • make pos index Unicode scalar values instead of UTF-8 bytes
  • make tlstr remove one complete Unicode scalar instead of slicing at byte 1
  • make string->n return the first scalar value, consistent with n->string
  • add multibyte regression coverage for emoji and accented characters

Problem

(tlstr "😀é") previously sliced the UTF-8 encoding after its first byte and failed with tlstr: produced non-UTF-8 result. pos and string->n used the same byte-oriented assumption, producing inconsistent behavior for multibyte text.

Verification

  • cargo test -p shen-rust --test primitives_coverage string_ops string_op_errors
  • cargo fmt --all -- --check
  • rebuilt release launcher:
    • (tlstr "😀é") => "é"
    • (string->n "😀") => 128512
  • broader cargo test -p shen-rust run was green through all reported library and initial AOT suites; the long-running command was interrupted after its output consumer detached.

@pyrex41
pyrex41 merged commit 4f0c8e0 into main Aug 26, 2026
1 of 2 checks passed
@pyrex41
pyrex41 deleted the fix/unicode-string-primitives branch August 26, 2026 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant