fglock · fglock · Mar 4, 2026 · Mar 4, 2026 · Mar 4, 2026 · Mar 4, 2026
diff --git a/.cognition/skills/fix-pat-sprintf/SKILL.md b/.cognition/skills/fix-pat-sprintf/SKILL.md
@@ -0,0 +1,178 @@
+---
+name: fix-pat-sprintf
+description: Fix re/pat.t and op/sprintf2.t test regressions on fix-exiftool-cli branch
+argument-hint: "[test-name or specific failure]"
+triggers:
+  - user
+  - model
+---
+
+# Fix pat.t and sprintf2.t Regressions
+
+You are fixing test regressions in `re/pat.t` (-17 tests) and `op/sprintf2.t` (-3 tests) on the `fix-exiftool-cli` branch of PerlOnJava.
+
+## Hard Constraints
+
+1. **No AST refactoring fallback.** The `LargeBlockRefactorer` / AST splitter must NOT be restored. This is non-negotiable.
+2. **Fix the interpreter.** The bytecode interpreter must achieve feature parity with the JVM compiler. Both backends must produce identical results for all Perl constructs.
+3. **Match the baseline exactly.** Target is the master baseline scores — no more, no less:
+   - `re/pat.t`: 1056/1296
+   - `op/sprintf2.t`: 1652/1655
+4. **Do NOT modify shared runtime** (`RuntimeRegex.java`, `RegexFlags.java`, `RegexPreprocessor.java`, etc.). The runtime is shared between both backends. Fixes must be in the interpreter code.
+
+## Why the Interpreter Is Involved
+
+Large subroutines that exceed the JVM 64KB method limit fall back to the bytecode interpreter via `EmitterMethodCreator.createRuntimeCode()`.
+
+- **pat.t**: The `run_tests` subroutine (lines 38-2652, ~2614 lines) falls back to interpreter. All 1296 tests run through it. Confirmed with `JPERL_SHOW_FALLBACK=1`.
+- **sprintf2.t**: Same mechanism — large test body falls back to interpreter.
+
+## Baseline vs Branch
+
+| Test | Master baseline (397ba45d) | Branch HEAD | Delta |
+|------|---------------------------|-------------|-------|
+| re/pat.t | 1056/1296 | 1039/1296 | -17 |
+| op/sprintf2.t | 1652/1655 | 1649/1655 | -3 |
+
+## Methodology
+
+For each failing test:
+
+1. **Extract** the specific Perl code from the test file
+2. **Compare** JVM vs interpreter output:
+   ```bash
+   ./jperl -E 'extracted code'              # JVM backend (correct behavior)
+   ./jperl --interpreter -E 'extracted code' # Interpreter (may differ)
+   ```
+3. **When they differ**: identify the root cause in the interpreter code (BytecodeCompiler, BytecodeInterpreter, etc.) and fix it
+4. **When they don't differ standalone**: the failure depends on context from earlier tests in the same large function. Investigate what prior state affects the result — look at regex state, variable scoping, match variables, pos(), etc.
+5. **Verify** the fix doesn't break other tests
+
+## Running the Tests
+
+```bash
+# Build
+make build
+
+# Run individual tests via test runner (sets correct ENV vars)
+perl dev/tools/perl_test_runner.pl perl5_t/t/re/pat.t
+perl dev/tools/perl_test_runner.pl perl5_t/t/op/sprintf2.t
+
+# Run manually with correct ENV
+cd perl5_t/t
+PERL_SKIP_BIG_MEM_TESTS=1 JPERL_UNIMPLEMENTED=warn JPERL_OPTS="-Xss256m" ../../jperl re/pat.t
+PERL_SKIP_BIG_MEM_TESTS=1 JPERL_UNIMPLEMENTED=warn ../../jperl op/sprintf2.t
+
+# Compare JVM vs interpreter for a specific construct
+./jperl -E 'code'
+./jperl --interpreter -E 'code'
+
+# Check if a test file uses interpreter fallback
+cd perl5_t/t && JPERL_SHOW_FALLBACK=1 ../../jperl re/pat.t 2>&1 | grep 'interpreter backend'
+
+# Get interpreter bytecodes for a construct
+./jperl --interpreter --disassemble -E 'code' 2>&1
+```
+
+## pat.t: Exact Regressions (18 PASS->FAIL, 1 FAIL->PASS, net -17)
+
+### Tests that went from PASS to FAIL
+
+| # | Test Description | pat.t Line | Category |
+|---|-----------------|------------|----------|
+| 1 | Stack may be bad | 508 | regex match |
+| 2 | $^N, @- and @+ are read-only | 845-851 | eval STRING special vars |
+| 3-4 | \G testing (x2) | 858, 866 | \G anchor |
+| 5 | \b is not special | 1089 | word boundary |
+| 6-8 | \s, [[:space:]] and [[:blank:]] (x3) | 1223-1225 | POSIX classes |
+| 9 | got a latin string - rt75680 | 1252 | latin/unicode |
+| 10-11 | RT #3516 A, B | 1329, 1335 | \G loop |
+| 12 | Qr3 bare | ~1490 | qr// overload |
+| 13 | Qr3 bare - with use re eval | ~1498 | qr// eval |
+| 14 | Eval-group not allowed at runtime | 524 | regex eval |
+| 15-18 | Branch reset pattern 1-4 | 2392-2409 | branch reset |
+
+### Test that went from FAIL to PASS
+
+| Test Description | Category |
+|-----------------|----------|
+| 1 '', '1', '12' (Eval-group) | regex eval |
+
+## Interpreter Architecture
+
+```
+Source -> Lexer -> Parser -> AST --+--> JVM Compiler (EmitterMethodCreator) -> JVM bytecode
+                                  \--> BytecodeCompiler -> InterpretedCode -> BytecodeInterpreter
+```
+
+Both backends share the same runtime (RuntimeRegex, RuntimeScalar, etc.). The difference is ONLY in how the AST is lowered to executable form. The interpreter must handle every construct identically to the JVM compiler.
+
+### Key interpreter files
+
+| File | Role |
+|------|------|
+| `backend/bytecode/BytecodeCompiler.java` | AST -> interpreter bytecodes |
+| `backend/bytecode/BytecodeInterpreter.java` | Main dispatch loop |
+| `backend/bytecode/InterpretedCode.java` | Code object + disassembler |
+| `backend/bytecode/Opcodes.java` | Opcode constants |
+| `backend/bytecode/CompileAssignment.java` | Assignment compilation |
+| `backend/bytecode/CompileBinaryOperator.java` | Binary ops compilation |
+| `backend/bytecode/CompileOperator.java` | Unary/misc ops compilation |
+| `backend/bytecode/SlowOpcodeHandler.java` | Rarely-used op handlers |
+| `backend/bytecode/OpcodeHandlerExtended.java` | CREATE_CLOSURE, STORE_GLOB, etc. |
+| `backend/bytecode/MiscOpcodeHandler.java` | Misc operations |
+| `backend/bytecode/EvalStringHandler.java` | eval STRING compilation for interpreter |
+
+All paths relative to `src/main/java/org/perlonjava/`.
+
+### Key source files (do NOT modify)
+
+| Area | File | Notes |
+|------|------|-------|
+| Regex runtime | `runtime/regex/RuntimeRegex.java` | DO NOT MODIFY |
+| Regex flags | `runtime/regex/RegexFlags.java` | DO NOT MODIFY |
+| Regex preprocessor | `runtime/regex/RegexPreprocessor.java` | DO NOT MODIFY |
+
+All paths relative to `src/main/java/org/perlonjava/`.
+
+## Verification Steps
+
+After any fix:
+
+```bash
+# 1. Build must pass
+make build
+
+# 2. Unit tests must pass
+make test-unit
+
+# 3. Check pat.t — must match baseline (1056/1296)
+perl dev/tools/perl_test_runner.pl perl5_t/t/re/pat.t
+
+# 4. Check sprintf2.t — must match baseline (1652/1655)
+perl dev/tools/perl_test_runner.pl perl5_t/t/op/sprintf2.t
+
+# 5. No regressions in other key tests
+perl dev/tools/perl_test_runner.pl perl5_t/t/op/pack.t
+perl dev/tools/perl_test_runner.pl perl5_t/t/re/pat_rt_report.t
+```
+
+## Debugging Tips
+
+### Compare raw output between baseline and branch
+```bash
+# Save branch output
+cd perl5_t/t && PERL_SKIP_BIG_MEM_TESTS=1 JPERL_UNIMPLEMENTED=warn JPERL_OPTS="-Xss256m" ../../jperl re/pat.t > /tmp/pat_branch.txt 2>&1
+
+# Compare by test name against saved baseline
+LC_ALL=C diff \
+  <(LC_ALL=C grep -E '^(ok|not ok)' /tmp/pat_base_raw.txt | LC_ALL=C sed 's/^ok [0-9]* - /PASS: /;s/^not ok [0-9]* - /FAIL: /' | LC_ALL=C sort) \
+  <(LC_ALL=C grep -E '^(ok|not ok)' /tmp/pat_branch.txt | LC_ALL=C sed 's/^ok [0-9]* - /PASS: /;s/^not ok [0-9]* - /FAIL: /' | LC_ALL=C sort) \
+  | grep '^[<>]'
+```
+
+### Test specific construct through both backends
+```bash
+./jperl -E 'my $s="abcde"; pos $s=2; say $s =~ /^\G/ ? "match" : "no"'
+./jperl --interpreter -E 'my $s="abcde"; pos $s=2; say $s =~ /^\G/ ? "match" : "no"'
+```
diff --git a/.cognition/skills/interpreter-parity/SKILL.md b/.cognition/skills/interpreter-parity/SKILL.md
@@ -200,6 +200,48 @@ Test::More → Test::Builder → Test::Builder::Formatter → Test2::Formatter::
 ```
 The failure is a ClassCastException in `Test/Builder/Formatter.pm` BEGIN block where `*OUT_STD = Test2::Formatter::TAP->can('OUT_STD')` — method call result (RuntimeList) is stored to glob (expects RuntimeScalar).
 
+## Design Decision: JVM Emitter Must Not Mutate the AST
+
+When the JVM backend fails with `MethodTooLargeException` (or `VerifyError`, etc.), `createRuntimeCode()` in `EmitterMethodCreator.java` falls back to the interpreter via `compileToInterpreter(ast, ...)`. The same fallback exists in `PerlLanguageProvider.compileToExecutable()`.
+
+**Problem**: The JVM emitter (EmitterVisitor and helpers) mutates the AST during code generation. If JVM compilation fails partway through, the interpreter receives a corrupted AST, producing wrong results. This is the root cause of mixed-mode failures (e.g., pack.t gets 45 extra failures when the main script falls back to interpreter after partial JVM emission).
+
+**Rule**: The JVM emitter must NEVER permanently mutate AST nodes. All mutations must either:
+1. Be avoided entirely (work on local copies), OR
+2. Use save/restore in try/finally (already done in `EmitLogicalOperator.java`)
+
+### Known AST mutation sites
+
+| File | Line(s) | What it mutates | Status |
+|------|---------|-----------------|--------|
+| `EmitOperator.java` | ~373 | `operand.elements.addFirst(operand.handle)` in `handleSystemBuiltin` — adds handle to elements list, never removed | **DANGEROUS** |
+| `Dereference.java` | ~347,442,511,579,911 | `nodeRight.elements.set(0, new StringNode(...))` — converts IdentifierNode to StringNode for hash autoquoting. `nodeRight` comes from `asListNode()` which creates a new ListNode but shares the same `elements` list | **DANGEROUS** — mutates shared elements list |
+| `EmitLogicalOperator.java` | ~188,300,340 | Temporarily rewrites `declaration.operator`/`.operand` | **SAFE** — uses save/restore in try/finally |
+| `EmitControlFlow.java` | ~280 | `argsNode.elements.add(atUnderscore)` | **SAFE** — `argsNode` is a freshly created ListNode |
+| `EmitOperator.java` | ~398,410 | `handleSpliceBuiltin` removes/restores first element | **SAFE** — uses try/finally restore |
+| Annotations (`setAnnotation`) | various | Sets `blockIsSubroutine`, `skipRegexSaveRestore`, `isDeclaredReference` | **Likely safe** — annotations are additive hints, but verify interpreter handles them |
+
+### How to fix dangerous sites
+
+**`handleSystemBuiltin` (EmitOperator.java:373)**: Wrap in try/finally to remove the added element after accept():
+```java
+if (operand.handle != null) {
+    hasHandle = true;
+    operand.elements.addFirst(operand.handle);
+}
+try {
+    operand.accept(emitterVisitor.with(RuntimeContextType.LIST));
+} finally {
+    if (hasHandle) {
+        operand.elements.removeFirst();
+    }
+}
+```
+
+**Dereference.java autoquoting**: `asListNode()` creates a new ListNode but passes the SAME `elements` list reference. The `elements.set(0, ...)` call mutates the original HashLiteralNode's elements. Fix by either:
+- Making `asListNode()` copy the elements list: `new ListNode(new ArrayList<>(elements), tokenIndex)`
+- Or saving/restoring the original element in try/finally
+
 ## Lessons Learned
 
 ### InterpretedCode constructor drops metadata

diff --git a/.cognition/skills/migrate-jna/SKILL.md b/.cognition/skills/migrate-jna/SKILL.md
@@ -0,0 +1,122 @@
+---
+name: migrate-jna
+description: Migrate from JNA to a modern native access library (eliminate sun.misc.Unsafe warnings)
+argument-hint: "[library choice or file to migrate]"
+triggers:
+  - user
+---
+
+# Migrate JNA to Modern Native Access Library
+
+## Problem
+
+JNA 5.18.1 uses `sun.misc.Unsafe::staticFieldBase` internally, which produces deprecation warnings on Java 21+ and will break in future JDK releases. The project needs to migrate to a library that uses supported APIs.
+
+## Candidate Replacement Libraries
+
+The choice of replacement library is TBD. Evaluate these options:
+
+### Option A: jnr-posix
+- **Maven**: `com.github.jnr:jnr-posix`
+- **Pros**: Purpose-built for POSIX ops, used by JRuby (production-proven), clean high-level API (`FileStat`, `kill()`, `waitpid()`, `umask()`, `utime()`), built on jnr-ffi (no `sun.misc.Unsafe`)
+- **Cons**: Third-party dependency, may not cover Windows-specific calls
+
+### Option B: Java Foreign Function & Memory API (FFM)
+- **Module**: `java.lang.foreign` (JDK built-in)
+- **Pros**: No third-party dependency, official JDK solution, no deprecated APIs
+- **Cons**: Stable only since Java 22 (preview in 21), verbose low-level API, requires manual struct layout definitions
+- **Note**: If the project bumps minimum to Java 22, this becomes viable without preview flags
+
+### Option C: jnr-ffi (without jnr-posix)
+- **Maven**: `com.github.jnr:jnr-ffi`
+- **Pros**: Modern JNA alternative, no `sun.misc.Unsafe`, flexible
+- **Cons**: Lower-level than jnr-posix, requires manual bindings (similar effort to FFM)
+
+## Current JNA Usage
+
+10 files use JNA. All paths relative to `src/main/java/org/perlonjava/`.
+
+### Native interface definitions
+
+| File | JNA Usage |
+|------|-----------|
+| `runtime/nativ/PosixLibrary.java` | POSIX C library bindings: `stat`, `lstat`, `chmod`, `chown`, `getpid`, `getppid`, `setpgid`, `getpgid`, `setsid`, `tcsetpgrp`, `tcgetpgrp`, `getpgrp`, `setpgrp` |
+| `runtime/nativ/WindowsLibrary.java` | Windows kernel32 bindings: `GetCurrentProcessId`, `_getpid` |
+| `runtime/nativ/NativeUtils.java` | JNA Platform utilities: `getpid()`, `getuid()`, `geteuid()`, `getgid()`, `getegid()`, plus `CLibrary` for `getpriority`/`setpriority`/`alarm`/`getlogin` |
+| `runtime/nativ/ExtendedNativeUtils.java` | Additional POSIX: `getpwuid`, `getpwnam`, `getgrnam`, `getgrgid` (passwd/group lookups) |
+
+### Consumers (files that call native operations)
+
+| File | Operations Used |
+|------|----------------|
+| `runtime/operators/Stat.java` | `PosixLibrary.stat()`, `PosixLibrary.lstat()` — all 13 stat fields (dev, ino, mode, nlink, uid, gid, rdev, size, atime, mtime, ctime, blksize, blocks) |
+| `runtime/operators/Operator.java` | `PosixLibrary.chmod()`, `PosixLibrary.chown()`, `NativeUtils` for pid/uid/gid |
+| `runtime/operators/KillOperator.java` | `PosixLibrary.kill()` for sending signals, `NativeUtils.getpid()` |
+| `runtime/operators/WaitpidOperator.java` | JNA `CLibrary.waitpid()` with `WNOHANG`/`WUNTRACED` flags, macros `WIFEXITED`/`WEXITSTATUS`/`WIFSIGNALED`/`WTERMSIG`/`WIFSTOPPED`/`WSTOPSIG` |
+| `runtime/operators/UmaskOperator.java` | JNA `CLibrary.umask()` |
+| `runtime/operators/UtimeOperator.java` | JNA `CLibrary.utimes()` with `timeval` struct |
+
+## Migration Strategy
+
+### Phase 1: Replace native interface definitions
+1. Create new interface files using the chosen library
+2. Keep the same method signatures where possible
+3. Ensure struct mappings (stat, timeval, passwd, group) are complete
+
+### Phase 2: Update consumers one by one
+Migrate in this order (least to most complex):
+1. `UmaskOperator.java` — single `umask()` call
+2. `KillOperator.java` — `kill()` + `getpid()`
+3. `UtimeOperator.java` — `utimes()` with struct
+4. `Operator.java` — `chmod()`, `chown()`, pid/uid/gid
+5. `WaitpidOperator.java` — `waitpid()` with flag macros
+6. `Stat.java` — `stat()`/`lstat()` with 13-field struct
+7. `NativeUtils.java` / `ExtendedNativeUtils.java` — passwd/group lookups
+
+### Phase 3: Remove JNA dependency
+1. Remove JNA imports from all files
+2. Remove JNA from `build.gradle` and `pom.xml`
+3. Remove `--enable-native-access=ALL-UNNAMED` from `jperl` launcher (if no longer needed)
+4. Verify the `sun.misc.Unsafe` warning is gone
+
+## Testing
+
+After each file migration:
+```bash
+make                    # Must pass
+make test-all           # Check for regressions
+```
+
+Key tests that exercise native operations:
+- `perl5_t/t/op/stat.t` — stat/lstat fields
+- `perl5_t/t/io/fs.t` — chmod, chown, utime
+- `perl5_t/t/op/fork.t` — kill, waitpid
+- `src/test/resources/unit/glob.t` — readdir (uses stat internally)
+
+## Build Configuration
+
+### Current JNA in gradle
+```
+# gradle/libs.versions.toml
+jna = "5.18.1"
+jna = { module = "net.java.dev.jna:jna", version.ref = "jna" }
+jna-platform = { module = "net.java.dev.jna:jna-platform", version.ref = "jna" }
+```
+
+### Current JNA in pom.xml
+```xml
+<dependency>
+    <groupId>net.java.dev.jna</groupId>
+    <artifactId>jna</artifactId>
+</dependency>
+<dependency>
+    <groupId>net.java.dev.jna</groupId>
+    <artifactId>jna-platform</artifactId>
+</dependency>
+```
+
+## Platform Considerations
+
+- **macOS/Linux**: Full POSIX support required (stat, lstat, kill, waitpid, chmod, chown, umask, utime, passwd/group lookups)
+- **Windows**: Limited support via `kernel32` (`GetCurrentProcessId`), `msvcrt` (`_getpid`, stat)
+- The replacement must handle both platforms, or gracefully degrade on Windows (as JNA currently does)
diff --git a/jperl b/jperl
@@ -24,5 +24,5 @@ else
 fi
 
 # Launch Java
-java ${JPERL_OPTS} -cp "$CLASSPATH:$JAR_PATH" org.perlonjava.app.cli.Main "$@"
+java --enable-native-access=ALL-UNNAMED ${JPERL_OPTS} -cp "$CLASSPATH:$JAR_PATH" org.perlonjava.app.cli.Main "$@"