diff --git a/docs/SPEC.md b/docs/SPEC.md index 3f1ecb819..01344b8cc 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -89,6 +89,15 @@ fail, then implement. from the prefix a property names, or from a Compose resources directory). The `createDistributable` run itself is a manual check — `docs/platforms/desktop-packaged-app.md`. +- **L-10** Desktop: the library uses no JDK module beyond `java.base`, so it runs on a `jlink`ed + runtime such as the one Compose Desktop's `createDistributable` bundles (issue #77). In particular + native memory is reached through `java.lang.foreign`, never `sun.misc.Unsafe` (module + `jdk.unsupported`): the proxy heap type (`ProxyTypeFactory.createProxyType`, `_pm_proxy_base`) is + built, and its handle slot read and cleared on deallocation, on a runtime limited to `java.base`. + `Status: implemented` — `PM/desktopTest/.../ffi/JlinkedRuntimeProxyTypeTest.kt` (a child JVM with + `--limit-modules java.base` installs the proxy base, and a dropped subclass instance releases its + handle). The packaged sample's own log is a manual check — + `docs/platforms/desktop-packaged-app.md`. ## 2. Low-level C API (downcall surface) diff --git a/docs/platforms/desktop-packaged-app.md b/docs/platforms/desktop-packaged-app.md index 7ba3c6a41..ca24c6889 100644 --- a/docs/platforms/desktop-packaged-app.md +++ b/docs/platforms/desktop-packaged-app.md @@ -94,7 +94,17 @@ and pass `-Dpython.multiplatform.home=`; jpackage expands `$APPDIR` inside - The library needs `--enable-preview` on JDK 21 (`java.lang.foreign` is a preview API there) and `--enable-native-access=ALL-UNNAMED`. A packaged app gets JVM options only from `compose.desktop.application { jvmArgs(...) }`; the sample's `build.gradle.kts` sets them for - `JavaExec` tasks only. + `JavaExec` tasks only. Observed on 2026-10-03 (issue #77): the packaged sample on its bundled + 21.0.12 runtime, with neither flag, initialises CPython and runs every step; the only trace is + the JDK's `WARNING: A restricted method in java.lang.foreign.Linker has been called`. The backend + reaches `java.lang.foreign` reflectively (`Panama.kt`), so no class file of ours is a preview + class file. Not a promise for later JDKs, where the restricted-method warning becomes an error. +- The bundled runtime is `jlink`ed and holds only the modules Compose was told about -- the sample's + `runtime/Contents/Home/release` lists `java.base java.datatransfer java.xml java.prefs + java.desktop java.logging jdk.crypto.ec`. The library therefore uses nothing outside `java.base` + (SPEC L-10). It used `sun.misc.Unsafe` (module `jdk.unsupported`) to build the proxy heap type, + which in the packaged app failed every proxy install with `ExceptionInInitializerError: null` and + every Kotlin-namespace import after it with `No module named 'org'` (issue #77). ## How to check it @@ -115,7 +125,8 @@ env -u PYTHONHOME -u PYTHON_MULTIPLATFORM_LIBPYTHON \ "$APP/Contents/MacOS/org.thisisthepy.python.multiplatform.demo" > .tmp/packaged-run.log 2>&1 & ``` -Expected in `.tmp/packaged-run.log`: `runtime : 3.14.7 ...` and the `await` lines (which import -`asyncio`), with no `Fatal Python error`. The window stays open; close it or kill the process. +Expected in `.tmp/packaged-run.log`: `runtime : 3.14.7 ...`, `proxies : installed: ...` (not +`ExceptionInInitializerError`), the `Greeter` lines, and the `await` lines (which import `asyncio`), +with no `Fatal Python error` and no `No module named`. The window stays open; close it or kill the process. Running from `/` as the working directory (`cd / && env -u PYTHONHOME "$OLDPWD/$APP/..."`) also checks that nothing is extracted into the working directory. diff --git a/python-multiplatform/src/desktopMain/README.md b/python-multiplatform/src/desktopMain/README.md index 5087fcf94..a09a2b612 100644 --- a/python-multiplatform/src/desktopMain/README.md +++ b/python-multiplatform/src/desktopMain/README.md @@ -125,6 +125,23 @@ staged one's `__pycache__` reflects whatever ran against it, so neither is packa `__pycache__` into the app image on first use where it can. See `docs/platforms/desktop-packaged-app.md`. +## Nothing outside `java.base` (SPEC L-10) + +A packaged app does not run on the JDK the build used. Compose Desktop's `createDistributable` +`jlink`s a runtime holding only the modules it was told about; the sample's image has `java.base +java.datatransfer java.xml java.prefs java.desktop java.logging jdk.crypto.ec` and nothing else. +`sun.misc.Unsafe` is in `jdk.unsupported`, so `Class.forName("sun.misc.Unsafe")` there throws +`ClassNotFoundException` -- which `ProxyType`'s initialiser did, so every proxy install in the +packaged sample failed with `ExceptionInInitializerError: null` and every import of a Kotlin +namespace after it with `No module named 'org'` (issue #77). Under Gradle every module is +observable and nothing showed it. + +Native memory is reached through `Panama` (`java.lang.foreign`, in `java.base`): +`allocateBytesFreeable` for a struct image built in a heap `ByteBuffer`, `readPointerSlot` / +`writePointerSlot` for one 8-byte word. `JlinkedRuntimeProxyTypeTest` (desktopTest) runs a child JVM +with `--limit-modules java.base`, which makes a full JDK behave like a jlinked image without building +one. Tests may still use `Unsafe` (`CycleCollectionTest` does): they run on the full JDK. + ## The consumer's own Python is a classpath resource, and `python/` collides with our own package `toolchain`'s `stagePythonBundleDesktop` puts a consumer's payload at the **root of the jar** as diff --git a/python-multiplatform/src/desktopMain/kotlin/python/multiplatform/ffi/ProxyTypeFactory.kt b/python-multiplatform/src/desktopMain/kotlin/python/multiplatform/ffi/ProxyTypeFactory.kt index 30ee0d097..a0419221b 100644 --- a/python-multiplatform/src/desktopMain/kotlin/python/multiplatform/ffi/ProxyTypeFactory.kt +++ b/python-multiplatform/src/desktopMain/kotlin/python/multiplatform/ffi/ProxyTypeFactory.kt @@ -1,6 +1,8 @@ package python.multiplatform.ffi import java.lang.invoke.MethodHandles +import java.nio.ByteBuffer +import java.nio.ByteOrder import python.native.ffi.Panama import python.native.ffi.bindings import python.native.ffi.downcallII_I @@ -25,21 +27,15 @@ private const val PY_RELATIVE_OFFSET = 8 object ProxyType { - // Resolved once at class initialisation rather than per callback. - // - // These used to be looked up inside each slot: Class.forName, getDeclaredField, - // setAccessible and getMethod on every single call, then Method.invoke with a boxed Long. - // tp_clear runs only during a collection so that was merely wasteful; tp_dealloc runs on - // *every* proxy death, which is the common case, so the same shape there would put four - // reflective lookups on the hot path of the object model. - private val unsafeClass = Class.forName("sun.misc.Unsafe") - private val theUnsafe = unsafeClass.getDeclaredField("theUnsafe").apply { isAccessible = true }.get(null) - private val getLongMethod = unsafeClass.getMethod("getLong", Long::class.javaPrimitiveType) - private val putLongMethod = - unsafeClass.getMethod("putLong", Long::class.javaPrimitiveType, Long::class.javaPrimitiveType) - - private fun peekLong(addr: Long): Long = getLongMethod.invoke(theUnsafe, addr) as Long - private fun pokeLong(addr: Long, value: Long) { putLongMethod.invoke(theUnsafe, addr, value) } + // Through java.lang.foreign (java.base), not sun.misc.Unsafe. Unsafe lives in the + // `jdk.unsupported` module, which a jlink'ed runtime does not contain unless asked to -- and + // Compose Desktop's `createDistributable` image does not ask. There, resolving it here made + // this object's throw ClassNotFoundException, every proxy install died with + // `ExceptionInInitializerError: null`, and every Kotlin-namespace import after it with + // `No module named 'org'` (issue #77, SPEC L-10). Nothing is resolved reflectively per call: + // both are handles Panama built once. + private fun peekLong(addr: Long): Long = Panama.readPointerSlot(addr) + private fun pokeLong(addr: Long, value: Long) { Panama.writePointerSlot(addr, value) } /** * Takes the [HandleTable] root out of [selfPtr]'s handle slot and drops it, returning the @@ -171,34 +167,9 @@ actual object ProxyTypeFactory { actual fun createProxyType(): Long { if (ProxyType.proxyTypePtr != 0L) return ProxyType.proxyTypePtr - val unsafeClass = Class.forName("sun.misc.Unsafe") - val theUnsafe = unsafeClass.getDeclaredField("theUnsafe").apply { isAccessible = true }.get(null) - val allocateMemory = unsafeClass.getMethod("allocateMemory", Long::class.javaPrimitiveType) - val putInt = unsafeClass.getMethod("putInt", Long::class.javaPrimitiveType, Int::class.javaPrimitiveType) - val putLong = unsafeClass.getMethod("putLong", Long::class.javaPrimitiveType, Long::class.javaPrimitiveType) - - // Room for eight 16-byte PyType_Slot entries; five are used -- traverse, clear, dealloc, - // members, sentinel -- with room to spare so a sixth does not silently write past the block. - val slotsAddr = allocateMemory.invoke(theUnsafe, 16L * 8) as Long - // Slot 0: Py_tp_traverse (71) - putInt.invoke(theUnsafe, slotsAddr, 71) - val traverseStub = Panama.createUpcallStubIII_I(MethodHandles.lookup().unreflect(ProxyType::class.java.getMethod("tp_traverse", Long::class.javaPrimitiveType, Long::class.javaPrimitiveType, Long::class.javaPrimitiveType))) - putLong.invoke(theUnsafe, slotsAddr + 8, traverseStub) - - // Slot 1: Py_tp_clear (51) - putInt.invoke(theUnsafe, slotsAddr + 16, 51) - val clearStub = Panama.createUpcallStubI_I(MethodHandles.lookup().unreflect(ProxyType::class.java.getMethod("tp_clear", Long::class.javaPrimitiveType))) - putLong.invoke(theUnsafe, slotsAddr + 24, clearStub) - - // Slot 2: Py_tp_dealloc (52). Not 50 -- that is Py_tp_call, which an earlier comment here - // named by mistake. Without this the type inherits subtype_dealloc, which frees the - // object correctly and knows nothing about the handle it was carrying. - putInt.invoke(theUnsafe, slotsAddr + 32, 52) - val deallocStub = Panama.createUpcallStubI_V(MethodHandles.lookup().unreflect(ProxyType::class.java.getMethod("tp_dealloc", Long::class.javaPrimitiveType))) - putLong.invoke(theUnsafe, slotsAddr + 40, deallocStub) - - // Slot 3: Py_tp_members (72) -- the handle slot, exposed to Python itself. `PyMemberDef` on - // every 64-bit target this ships for (macos-aarch64/x86_64, linux-x86_64, windows-x86_64): + // Slot 3's payload -- Py_tp_members (72), the handle slot exposed to Python itself. + // `PyMemberDef` on every 64-bit target this ships for (macos-aarch64/x86_64, linux-x86_64, + // windows-x86_64): // // struct PyMemberDef { const char *name; int type; Py_ssize_t offset; int flags; // const char *doc; }; @@ -206,6 +177,7 @@ actual object ProxyTypeFactory { // name@0(8) type@8(4) [pad 4] offset@16(8) flags@24(4) [pad 4] doc@32(8) = 40 bytes, // and the array needs a second, all-zero entry -- PyMemberDef's own required NUL-name // terminator, the same contract PyMethodDef and PyType_Slot arrays already carry here. + // [nativeStruct] starts from zeroes, so the terminator is written by not writing it. // // `Py_RELATIVE_OFFSET` (3.12+, `descrobject.h`) is what makes `offset = 0` mean "the start // of *this type's* relative data" rather than "byte 0 of the object" -- the same base @@ -216,32 +188,50 @@ actual object ProxyTypeFactory { // make every member on every base type in the MRO wrong by however much the subclass added // -- which is precisely the multiple/deep-inheritance case this type exists to allow // (`Py_TPFLAGS_BASETYPE`, and `work/untrack`'s proof that a subclass survives deallocation). - val membersAddr = allocateMemory.invoke(theUnsafe, 40L * 2) as Long - putLong.invoke(theUnsafe, membersAddr, Panama.allocateUtf8String("_pm_handle")) - putInt.invoke(theUnsafe, membersAddr + 8, PY_T_LONG) - putLong.invoke(theUnsafe, membersAddr + 16, 0L) - putInt.invoke(theUnsafe, membersAddr + 24, PY_RELATIVE_OFFSET) - putLong.invoke(theUnsafe, membersAddr + 32, 0L) - putLong.invoke(theUnsafe, membersAddr + 40, 0L) // sentinel: name - putInt.invoke(theUnsafe, membersAddr + 48, 0) // sentinel: type - putLong.invoke(theUnsafe, membersAddr + 56, 0L) // sentinel: offset - putInt.invoke(theUnsafe, membersAddr + 64, 0) // sentinel: flags - putLong.invoke(theUnsafe, membersAddr + 72, 0L) // sentinel: doc + val membersAddr = nativeStruct(40 * 2) { + putLong(0, Panama.allocateUtf8String("_pm_handle")) + putInt(8, PY_T_LONG) + putLong(16, 0L) + putInt(24, PY_RELATIVE_OFFSET) + putLong(32, 0L) + // 40..79: the sentinel entry, all zero. + } - putInt.invoke(theUnsafe, slotsAddr + 48, PY_TP_MEMBERS) - putLong.invoke(theUnsafe, slotsAddr + 56, membersAddr) + val traverseStub = Panama.createUpcallStubIII_I(MethodHandles.lookup().unreflect(ProxyType::class.java.getMethod("tp_traverse", Long::class.javaPrimitiveType, Long::class.javaPrimitiveType, Long::class.javaPrimitiveType))) + val clearStub = Panama.createUpcallStubI_I(MethodHandles.lookup().unreflect(ProxyType::class.java.getMethod("tp_clear", Long::class.javaPrimitiveType))) + val deallocStub = Panama.createUpcallStubI_V(MethodHandles.lookup().unreflect(ProxyType::class.java.getMethod("tp_dealloc", Long::class.javaPrimitiveType))) - // Slot 4: sentinel - putInt.invoke(theUnsafe, slotsAddr + 64, 0) - putLong.invoke(theUnsafe, slotsAddr + 72, 0L) + // Room for eight 16-byte PyType_Slot entries ({ int slot; void *pfunc; }); five are used -- + // traverse, clear, dealloc, members, sentinel -- with room to spare so a sixth does not + // silently write past the block. + val slotsAddr = nativeStruct(16 * 8) { + // Slot 0: Py_tp_traverse (71) + putInt(0, 71) + putLong(8, traverseStub) + // Slot 1: Py_tp_clear (51) + putInt(16, 51) + putLong(24, clearStub) + // Slot 2: Py_tp_dealloc (52). Not 50 -- that is Py_tp_call, which an earlier comment + // here named by mistake. Without this the type inherits subtype_dealloc, which frees + // the object correctly and knows nothing about the handle it was carrying. + putInt(32, 52) + putLong(40, deallocStub) + // Slot 3: Py_tp_members (72) + putInt(48, PY_TP_MEMBERS) + putLong(56, membersAddr) + // Slot 4 (64..79): the sentinel, all zero. + } - val specAddr = allocateMemory.invoke(theUnsafe, 32L) as Long - putLong.invoke(theUnsafe, specAddr, Panama.allocateUtf8String("KotlinProxy")) - // negative basicsize means it's appended to the end of the base object (PyObject) - putInt.invoke(theUnsafe, specAddr + 8, -8) // 8 bytes for the handle - putInt.invoke(theUnsafe, specAddr + 12, 0) - putInt.invoke(theUnsafe, specAddr + 16, (1 shl 10) or (1 shl 14)) // Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC - putLong.invoke(theUnsafe, specAddr + 24, slotsAddr) + // PyType_Spec { const char *name; int basicsize; int itemsize; unsigned int flags; + // PyType_Slot *slots; } + val specAddr = nativeStruct(32) { + putLong(0, Panama.allocateUtf8String("KotlinProxy")) + // negative basicsize means it's appended to the end of the base object (PyObject) + putInt(8, -8) // 8 bytes for the handle + putInt(12, 0) + putInt(16, (1 shl 10) or (1 shl 14)) // Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC + putLong(24, slotsAddr) + } ProxyType.proxyTypePtr = python.multiplatform.ffi.withGIL { bindings.PyType_FromSpec(specAddr) @@ -249,6 +239,21 @@ actual object ProxyTypeFactory { return ProxyType.proxyTypePtr } + /** + * A zero-filled native block of [size] bytes holding what [fill] wrote, in native byte order. + * + * Built as a heap image and copied out in one step by [Panama.allocateBytesFreeable] (native + * `malloc`), rather than poked field by field through `sun.misc.Unsafe` -- see + * the note above `ProxyType.peekLong` for why Unsafe is not available in a packaged app. Never freed: + * `PyType_FromSpec` may keep pointing into the slot and member arrays for the life of the type, + * and the type lives for the life of the process. + */ + private fun nativeStruct(size: Int, fill: ByteBuffer.() -> Unit): Long { + val image = ByteBuffer.allocate(size).order(ByteOrder.nativeOrder()) + image.fill() + return Panama.allocateBytesFreeable(image.array()) + } + /** * Publishes [createProxyType]'s type into `__main__` as `_pm_proxy_base`. * diff --git a/python-multiplatform/src/desktopMain/kotlin/python/native/ffi/Panama.kt b/python-multiplatform/src/desktopMain/kotlin/python/native/ffi/Panama.kt index 475b9c9b7..d16c8a634 100644 --- a/python-multiplatform/src/desktopMain/kotlin/python/native/ffi/Panama.kt +++ b/python-multiplatform/src/desktopMain/kotlin/python/native/ffi/Panama.kt @@ -75,9 +75,21 @@ internal object Panama { * Release with [freePointerSlot]. */ val allocatePointerSlot: () -> Long - /** Read back the pointer a callee wrote into a slot obtained from [allocatePointerSlot]. */ + /** Read back the pointer a callee wrote into a slot obtained from [allocatePointerSlot]. + * + * Reads any 8 native bytes at the address, in the platform's byte order, so it is also the + * desktop's `*(int64_t *)addr` -- `ProxyType` reads its handle slot through it. */ val readPointerSlot: (Long) -> Long + /** Write a pointer-sized value (`*(int64_t *)addr = value`, native byte order). + * + * The other half of [readPointerSlot]. It exists so that nothing on desktop reaches native + * memory through `sun.misc.Unsafe`: that class lives in the `jdk.unsupported` module, which a + * `jlink`ed runtime -- Compose Desktop's `createDistributable` among them -- does not contain + * unless asked to (issue #77, SPEC L-10). `java.lang.foreign` is in `java.base`, which every + * runtime has. */ + val writePointerSlot: (Long, Long) -> Unit + /** Release a slot obtained from [allocatePointerSlot]. */ val freePointerSlot: (Long) -> Unit @@ -146,6 +158,7 @@ internal object Panama { freeUtf8Address = data.freeUtf8Address allocatePointerSlot = data.allocatePointerSlot readPointerSlot = data.readPointerSlot + writePointerSlot = data.writePointerSlot freePointerSlot = data.freePointerSlot allocateBytesFreeable = data.allocateBytesFreeable unboundDowncallHandle = data.unboundDowncallHandle @@ -165,6 +178,7 @@ internal object Panama { freeUtf8Address = data.freeUtf8Address allocatePointerSlot = data.allocatePointerSlot readPointerSlot = data.readPointerSlot + writePointerSlot = data.writePointerSlot freePointerSlot = data.freePointerSlot allocateBytesFreeable = data.allocateBytesFreeable unboundDowncallHandle = data.unboundDowncallHandle @@ -188,6 +202,7 @@ internal object Panama { val freeUtf8Address: (Long) -> Unit, val allocatePointerSlot: () -> Long, val readPointerSlot: (Long) -> Long, + val writePointerSlot: (Long, Long) -> Unit, val freePointerSlot: (Long) -> Unit, val allocateBytesFreeable: (ByteArray) -> Long, val unboundDowncallHandle: (Int, Int, ReturnKind) -> MethodHandle, @@ -484,6 +499,15 @@ internal object Panama { java.nio.ByteBuffer.wrap(bytes).order(java.nio.ByteOrder.nativeOrder()).getLong(0) } + // The same copy as `readSlot`, running the other way: byte[] -> native. + val writeSlot: (Long, Long) -> Unit = { addr, value -> + val bytes = ByteArray(POINTER_BYTES.toInt()) + java.nio.ByteBuffer.wrap(bytes).order(java.nio.ByteOrder.nativeOrder()).putLong(0, value) + val srcSeg: Any = ofArrayExact.invokeExact(bytes) as Any + val dstSeg: Any = addressToSegmentExact.invokeExact(addr, POINTER_BYTES) as Any + copyExact.invokeExact(srcSeg, 0L, dstSeg, 0L, POINTER_BYTES) as Unit + } + val freeSlot: (Long) -> Unit = freeAddr // ---- Arbitrary bytes into a malloc buffer (non-UTF-8 C strings: wchar_t) ---- @@ -586,7 +610,7 @@ internal object Panama { segmentAddressExact.invokeExact(stub) as Long } - return ModernData(allocStr, readStr, findSym, findAddr, allocFreeable, freeAddr, allocSlot, readSlot, freeSlot, allocBytes, buildShape, buildUpcallStub, buildUpcallStubII_L, buildUpcallStubIII_I, buildUpcallStubI_I, buildUpcallStubI_V) + return ModernData(allocStr, readStr, findSym, findAddr, allocFreeable, freeAddr, allocSlot, readSlot, writeSlot, freeSlot, allocBytes, buildShape, buildUpcallStub, buildUpcallStubII_L, buildUpcallStubIII_I, buildUpcallStubI_I, buildUpcallStubI_V) } private fun adaptModernHandle( @@ -657,6 +681,7 @@ internal object Panama { val freeUtf8Address: (Long) -> Unit, val allocatePointerSlot: () -> Long, val readPointerSlot: (Long) -> Long, + val writePointerSlot: (Long, Long) -> Unit, val freePointerSlot: (Long) -> Unit, val allocateBytesFreeable: (ByteArray) -> Long, val unboundDowncallHandle: (Int, Int, ReturnKind) -> MethodHandle, @@ -843,6 +868,7 @@ internal object Panama { } val allocSlot: () -> Long = { slotsUnsupported() } val readSlot: (Long) -> Long = { slotsUnsupported() } + val writeSlot: (Long, Long) -> Unit = { _, _ -> slotsUnsupported() } val freeSlot: (Long) -> Unit = { slotsUnsupported() } // Same reason as the slots: this backend writes native memory only through `toCString`, @@ -946,7 +972,7 @@ internal object Panama { toRawLongMethod.invoke(segAddressMethod.invoke(stub)) as Long } - return IncubatorData(allocStr, readStr, findSym, findAddr, allocFreeable, freeAddr, allocSlot, readSlot, freeSlot, allocBytes, buildShape, buildUpcallStub, buildUpcallStubII_L, buildUpcallStubIII_I, buildUpcallStubI_I, buildUpcallStubI_V) + return IncubatorData(allocStr, readStr, findSym, findAddr, allocFreeable, freeAddr, allocSlot, readSlot, writeSlot, freeSlot, allocBytes, buildShape, buildUpcallStub, buildUpcallStubII_L, buildUpcallStubIII_I, buildUpcallStubI_I, buildUpcallStubI_V) } private fun adaptIncubatorHandle( diff --git a/python-multiplatform/src/desktopTest/kotlin/python/multiplatform/ffi/JlinkedRuntimeProxyTypeTest.kt b/python-multiplatform/src/desktopTest/kotlin/python/multiplatform/ffi/JlinkedRuntimeProxyTypeTest.kt new file mode 100644 index 000000000..33dad2e24 --- /dev/null +++ b/python-multiplatform/src/desktopTest/kotlin/python/multiplatform/ffi/JlinkedRuntimeProxyTypeTest.kt @@ -0,0 +1,168 @@ +package python.multiplatform.ffi + +import python.multiplatform.reflection.HandleTable +import java.io.File +import java.nio.file.Files +import java.util.concurrent.TimeUnit +import kotlin.system.exitProcess +import kotlin.test.AfterTest +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertNotNull +import kotlin.test.assertTrue + +/** + * SPEC L-10 (issue #77): the desktop library runs on a Java runtime that contains **only + * `java.base`** -- in particular, the proxy heap type is built and torn down without + * `sun.misc.Unsafe`. + * + * Why a child JVM with `--limit-modules`. A packaged desktop app does not run on the JDK the build + * used: Compose Desktop's `createDistributable` `jlink`s a runtime holding only the modules it was + * told about -- `java.base java.datatransfer java.xml java.prefs java.desktop java.logging + * jdk.crypto.ec` in the sample's app image (its `runtime/Contents/Home/release`). `sun.misc.Unsafe` + * lives in `jdk.unsupported`, which is not among them. The test JVM is a full JDK where every module + * is observable, so in-process tests could never see this; `--limit-modules` makes a full JDK + * behave like the jlinked image without building one. `java.base` alone is stricter than Compose's + * set (which is its transitive closure plus AWT), and is the runtime every `jlink` image has. + * + * Observed before the fix, in the packaged sample: `proxies : ExceptionInInitializerError: null`, + * thrown from `ProxyType.`'s `Class.forName("sun.misc.Unsafe")` (ClassNotFoundException), + * then `No module named 'org'` from every step that imports a Kotlin namespace the proxy install + * never built. The same jars on a full JDK printed `proxies : installed: ...`. + * + * Expected red before the fix: [theProxyTypeWorksOnAJavaBaseOnlyRuntime] fails with the child's + * `PROBE_FAILED java.lang.ExceptionInInitializerError ... Caused by: + * java.lang.ClassNotFoundException: sun.misc.Unsafe` and exit code [JlinkedRuntimeProbe.FAILED_EXIT_CODE]. + * [theProbeItselfWorksOnTheFullJdk] is the control, green before and after: it runs the same probe + * without `--limit-modules`, so a red in the first test is the module set and not a broken probe. + */ +class JlinkedRuntimeProxyTypeTest { + + private val scratchDirs = mutableListOf() + + @AfterTest + fun cleanup() { + scratchDirs.forEach { dir -> + if (dir.exists()) { + Files.walk(dir.toPath()).use { paths -> + paths.sorted(Comparator.reverseOrder()).forEach { Files.delete(it) } + } + } + } + scratchDirs.clear() + } + + /** Under `build/`, not the system temporary directory -- AGENTS.md rule 2. */ + private fun newScratchDir(): File = + File("build/tmp/jlinkedRuntimeProxyTypeTest/${java.util.UUID.randomUUID()}").absoluteFile + .also { it.mkdirs(); scratchDirs += it } + + private class Outcome(val exitCode: Int, val output: String) + + /** + * Launches [JlinkedRuntimeProbe] in a child JVM. + * + * `PYTHONHOME` is inherited from this test JVM and `-Djava.library.path=.` lets the child load + * the classpath copy of `libpython` it extracts into its working directory -- exactly what + * `PackagedPythonHomeLaunchTest`'s environment guard does. Both are held fixed on purpose: the + * only thing this test varies is which JDK modules exist, so that is the only thing a red can + * be about. (Where a packaged app finds its prefix is L-9's test.) + */ + private fun launchProbe(limitModules: String?): Outcome { + val workDir = newScratchDir() + val log = File(workDir, "probe.log") + val java = File(System.getProperty("java.home"), "bin/java").path + val command = buildList { + add(java) + if (limitModules != null) { + add("--limit-modules") + add(limitModules) + } + // java.lang.foreign is a preview API on JDK 21, which is what desktopTest runs on. + add("--enable-preview") + add("--enable-native-access=ALL-UNNAMED") + add("-Djava.library.path=.") + add("-cp") + add(System.getProperty("java.class.path")) + add(JlinkedRuntimeProbe::class.java.name) + } + val process = ProcessBuilder(command) + .directory(workDir) + .redirectErrorStream(true) + .redirectOutput(log) + .start() + if (!process.waitFor(120, TimeUnit.SECONDS)) { + process.destroyForcibly() + return Outcome(-1, "timed out after 120 s\n" + log.readText()) + } + return Outcome(process.exitValue(), log.readText()) + } + + private fun assertProbeOk(outcome: Outcome) { + assertNotNull(System.getenv("PYTHONHOME"), "PYTHONHOME is unset for desktopTest; the child cannot start CPython") + assertEquals(0, outcome.exitCode, outcome.output) + assertTrue( + outcome.output.lineSequence().any { it == "PROBE_OK" }, + "the child never reported PROBE_OK:\n${outcome.output}", + ) + } + + @Test + fun theProxyTypeWorksOnAJavaBaseOnlyRuntime() { + assertProbeOk(launchProbe(limitModules = "java.base")) + } + + @Test + fun theProbeItselfWorksOnTheFullJdk() { + assertProbeOk(launchProbe(limitModules = null)) + } +} + +/** + * The child's `main`: the step the packaged sample died in, and the two native-memory accesses + * behind it. + * + * 1. `ProxyTypeFactory.installGcBase()` -- what `PythonProxySource.install()` runs first. It builds + * the `PyType_Spec`/slot/member arrays in native memory and publishes `_pm_proxy_base`. + * 2. A Python subclass instance that carries a real [HandleTable] handle in `_pm_handle` and is then + * dropped. Its `tp_dealloc` *reads* the handle slot and *writes* 0 into it before releasing the + * handle -- the peek and poke that went through `sun.misc.Unsafe`. A handle that still resolves + * afterwards means the dealloc never released it. + * + * Prints `PROBE_OK`, or `PROBE_FAILED ` and exits with [FAILED_EXIT_CODE]. + */ +object JlinkedRuntimeProbe { + const val FAILED_EXIT_CODE = 4 + + @JvmStatic + fun main(args: Array) { + try { + Python3.initialize(silent = true) + check(ProxyTypeFactory.installGcBase()) { "installGcBase() returned false" } + + val handle = HandleTable.register(Any()).raw + Python3.exec( + """ + import gc + class _JlinkProbeProxy(_pm_proxy_base): + pass + _jp = _JlinkProbeProxy() + _jp._pm_handle = $handle + assert _jp._pm_handle == $handle, _jp._pm_handle + del _jp + gc.collect() + """.trimIndent() + ) + check(HandleTable.resolveRaw(handle) == null) { + "tp_dealloc did not release handle $handle: the handle slot was not read or not cleared" + } + } catch (t: Throwable) { + println("PROBE_FAILED ${t.stackTraceToString()}") + System.out.flush() + exitProcess(FAILED_EXIT_CODE) + } + println("PROBE_OK") + System.out.flush() + exitProcess(0) + } +} diff --git a/python-multiplatform/src/nativeMain/kotlin/python/multiplatform/ffi/ProxyTypeFactory.kt b/python-multiplatform/src/nativeMain/kotlin/python/multiplatform/ffi/ProxyTypeFactory.kt index bb4d6fccd..439a20fad 100644 --- a/python-multiplatform/src/nativeMain/kotlin/python/multiplatform/ffi/ProxyTypeFactory.kt +++ b/python-multiplatform/src/nativeMain/kotlin/python/multiplatform/ffi/ProxyTypeFactory.kt @@ -273,8 +273,8 @@ private fun proxyDealloc(self: CPointer?) { * as a global ([proxyTypeAddress]). * * Everything else is easier here than on desktop. `PyType_Spec` and `PyType_Slot` come out of - * cinterop as real structs, so the spec is filled in by field name instead of by hand-computed - * offsets through `sun.misc.Unsafe`. + * cinterop as real structs, so the spec is filled in by field name instead of at hand-computed + * offsets in a byte image that desktop copies out through `java.lang.foreign`. * * ### Threads CPython created *