Make string primitives count Unicode code points (UTF-8)

[pyrex41]

Fixes #80.

What changes

The string primitives used to treat a Shen string as raw Lua bytes, so they broke on any character outside ASCII. Now a string is a sequence of Unicode code points, stored as UTF-8:

• (n->string N) accepts whole numbers from 0 to 0x10FFFF, except surrogates, and errors on anything else.
• string->n, pos, tlstr, hdstr, explode and string-length count characters, not bytes.
• hash hashes code points, so it agrees with =.
• The reader decodes UTF-8 source, so "Zürich" reads as 6 characters and prints back unchanged.
• Invalid UTF-8 never raises an error and never loses data. Each bad byte counts as one character, and string->n returns its byte value. (cn (hdstr S) (tlstr S)) always gives back S. This is documented in the new README section "Strings and Unicode".
• read-byte, write-byte and read-file-as-bytelist still work on raw bytes.

ASCII-only strings stay on a plain byte fast path. compiler.lua now compiles (string->n (pos S I)) and (string->n (hdstr S)) to STRNPOS, which reads the code without allocating a one-character string. The kernel cache format moves from SHENKC4 to SHENKC5, which invalidates the old kernel, fasl and stdlib-image caches.

Files: prims.lua, compiler.lua, boot.lua, README.md, bin/yggdrasil-build.lua (comments only), the new test/utf8_strings_spec.lua (122 checks), and a new bifrost/ suite with unicode-codepoints and unicode-literals programs.

Tests

Command	LuaJIT 2.1	Lua 5.1	Lua 5.4
run-kernel-tests.lua	134/134	134/134	134/134
test/utf8_strings_spec.lua	122/0	122/0	122/0
make test	1210 pass, 0 fail	–	1062 pass, 59 fail

The 59 Lua 5.4 failures also happen on main (fc97577): they are in specs that need LuaJIT.

Bifrost run on --impls shen-lua,shen-go: shen-lua passes 7/7. shen-go fails 4 cases because of its own behaviour: it writes characters above 255 as raw bytes, uses a non-kernel string hash, and reads source as Latin-1.

Downstream check: Yggdrasil's (a40d8c5) lua build of a real rule set used to fail with bad argument #1 to 'cf.rune'. It now passes parity (58 checks).

Performance

Medians of interleaved runs with bench/ab/run.py, main against this branch:

Workload	main	branch
Warm boot	0.110 s	0.107 s
Cold boot	1.10 s	0.99 s (noise)
Kernel suite	3.59 s	3.56 s
micro, deep, fib	–	within noise

An ASCII loop that calls pos 4 million times takes 0.010 s against 0.007 s on main, about 0.75 ns more per call.

How other ports compare

On strings built with n->string, shen-lua, shen-go and Tarver's S42 SBCL build all agree. Tarver's build and shen-go read source files as Latin-1, so a literal "ü" becomes two characters there. This PR deliberately reads UTF-8 instead.

Open questions for review

1. Should source be read as Latin-1 to match Tarver exactly? This PR uses UTF-8, because Latin-1 turns literals that work today into garbled characters.
2. Should string->n reject strings longer than one character, as Tarver's build does? It still returns the first character's code.
3. Should n->string on a surrogate return U+FFFD, as Tarver's build does? It errors here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DHL2X2i5C1mWW2f5ypuSaw

---

Generated by Claude Code