|
| 1 | +# Desktop: the standard library in a packaged application |
| 2 | + |
| 3 | +SPEC L-9, issue #60, ROADMAP §15.4. |
| 4 | + |
| 5 | +During development `stagePythonHome` downloads a python-build-standalone prefix once per machine and |
| 6 | +the plugin sets `PYTHONHOME` on every `JavaExec` and `Test` task (SPEC L-4). A packaged application |
| 7 | +is started by a launcher, not by Gradle, and nothing put `PYTHONHOME` in its environment, so |
| 8 | +`Py_Initialize()` aborted with `Failed to import encodings module`. This page records how a packaged |
| 9 | +app now gets its prefix, why it was done this way, and how to check it. |
| 10 | + |
| 11 | +## What happens |
| 12 | + |
| 13 | +**Build time** (`python-multiplatform-gradle-plugin`, `PythonBindingsPlugin.configurePackagedPythonHome`). |
| 14 | +Every Compose Desktop `prepare*AppResources` task — the `Sync` that `createDistributable`, |
| 15 | +`packageDmg`/`packageMsi`/`packageDeb` and their `Release` variants take their resources from — |
| 16 | +depends on `stagePythonHome` and copies part of the staged prefix into `python-multiplatform-home/`: |
| 17 | + |
| 18 | +| Host | Copied (relative to the prefix) | |
| 19 | +|---|---| |
| 20 | +| macOS, Linux | `lib/python<X.Y>/**`, `lib/libpython<X.Y>.*` (`<X.Y>t` for a free-threaded build) | |
| 21 | +| Windows | `Lib/**`, `DLLs/**`, `*.dll` | |
| 22 | + |
| 23 | +`**/__pycache__/**`, `include/`, `bin/`, `lib/pkgconfig` and Tcl/Tk are not copied. For 3.14.7 |
| 24 | +macos-aarch64 that is 1,195 files and 39.6 MB (19.4 MB of it `libpython3.14.dylib`) of a 68.9 MB |
| 25 | +prefix, measured on the staged prefix of this machine. |
| 26 | + |
| 27 | +In the app image the directory is `$APPDIR/resources/python-multiplatform-home`; on macOS |
| 28 | +`<App>.app/Contents/app/resources/python-multiplatform-home`, on Linux |
| 29 | +`<app>/lib/app/resources/python-multiplatform-home`. |
| 30 | + |
| 31 | +**Run time** (`python-multiplatform`). `PackagedPythonHome.resolve()` (`jvmMain`) takes the first of: |
| 32 | + |
| 33 | +1. `PYTHONHOME` in the environment — then nothing else happens; CPython reads it itself. |
| 34 | +2. The system property `python.multiplatform.home`, used even if the path does not exist, so that a |
| 35 | + wrong value is reported by name. |
| 36 | +3. `<compose.application.resources.dir>/python-multiplatform-home`, only if that directory exists. |
| 37 | + Compose sets `-Dcompose.application.resources.dir=$APPDIR/resources` on every launcher it builds. |
| 38 | + |
| 39 | +`Python3.initialize()` calls `applyPackagedPythonHome()` right after the `PYTHONHOME` pre-flight |
| 40 | +check (SPEC L-3). On desktop it checks the resolved prefix with the same `PythonHomeCheck` |
| 41 | +(throwing a catchable `IllegalStateException` that names the path and the property it came from) |
| 42 | +and passes it to `Py_SetPythonHome` as a `wchar_t` string. Before that, when the bindings load, |
| 43 | +`manager.loadLibPython` loads `libpython` from the same prefix instead of extracting the classpath |
| 44 | +copy into the working directory. |
| 45 | + |
| 46 | +## Why this design |
| 47 | + |
| 48 | +**Into the app image, not a jar resource extracted at startup.** The consumer's `python/` payload is |
| 49 | +a jar resource that `ClasspathPayload` extracts to a temp directory with a digest stamp. The same |
| 50 | +mechanism for the stdlib was considered and not taken: |
| 51 | + |
| 52 | +- The stdlib is 1,195 files / 39.6 MB uncompressed. Extracted from a jar it is shipped twice (once |
| 53 | + compressed in the app, once extracted per user), and the first launch pays for writing all of it. |
| 54 | + Put into the app image, it is on disk at install time: no extraction ever, so there is no stamp to |
| 55 | + check on each launch and no cold-start cost at all. |
| 56 | +- The payload's cache lives under `java.io.tmpdir`, which operating systems clean up on their own |
| 57 | + schedule. A stamp that survives a partial clean-up which removed part of `encodings/` would be an |
| 58 | + abort at `Py_Initialize` rather than a re-extraction. (Not observed here; a reason not to take the |
| 59 | + risk for the one tree whose absence is fatal.) |
| 60 | +- `libpython` has to be a file for `System.load` anyway. Loading it from the classpath copy means |
| 61 | + extracting 19.4 MB into the working directory and finding it through `java.library.path` — this |
| 62 | + repo's tests pass `-Djava.library.path=.`, a launched app has no such flag and may have `/` as its |
| 63 | + working directory. Loading it from the packaged prefix also pairs it with its own stdlib. |
| 64 | +- jpackage builds an image for the host it runs on, so only one platform's prefix is ever packaged; |
| 65 | + the cost `PythonHomeStaging`'s KDoc measured against shipping four platforms' stdlibs in the |
| 66 | + *library* jar does not arise. |
| 67 | + |
| 68 | +**Into Compose's resources task, not `nativeDistributions.appResourcesRootDir`.** That property is a |
| 69 | +single directory the consumer may already use for their own files; setting it from the plugin would |
| 70 | +replace theirs. Adding a `from` to the `Sync` that assembles the resources composes with whatever |
| 71 | +the consumer configured, and matching the task by name keeps the plugin free of a compile-time |
| 72 | +dependency on the Compose Gradle plugin (as the wasm wiring matches `wasmJsProcessResources`). |
| 73 | + |
| 74 | +**`Py_SetPythonHome`, not an environment variable.** The launcher accepts `--java-options` and no |
| 75 | +environment, and a JVM cannot set its own (`desktopMain/README.md`). `Py_SetPythonHome` is part of |
| 76 | +the Stable ABI (deprecated since 3.11 in favour of `PyConfig.home`, whose struct layout the Stable |
| 77 | +ABI does not promise) and writes CPython's path configuration directly, so `Py_Initialize()` uses it |
| 78 | +whenever `PYTHONHOME` is unset. |
| 79 | + |
| 80 | +## Other packagers |
| 81 | + |
| 82 | +Anything that is not Compose's `createDistributable` (plain `jpackage`, Conveyor, an uber jar) gets |
| 83 | +no copy from the plugin. Copy the same subset of the staged prefix (`stagePythonHome`'s output, |
| 84 | +`~/.gradle/python-multiplatform/python-home/<version>+<release>/<platform>/python`) next to the app |
| 85 | +and pass `-Dpython.multiplatform.home=<path>`; jpackage expands `$APPDIR` inside `--java-options`. |
| 86 | + |
| 87 | +## Limits |
| 88 | + |
| 89 | +- No `.pyc` is shipped. CPython compiles what it imports on first use and writes `__pycache__` into |
| 90 | + the app image where it can; where it cannot (a read-only install), it compiles on every launch. |
| 91 | +- `pythonBindings { packagePythonHome.set(false) }` turns the copy off. `stagePythonHome.set(false)` |
| 92 | + does not: the packaged prefix must be the one the library was built against regardless of what a |
| 93 | + developer's own `PYTHONHOME` points at, so the task is registered (lazily) either way. |
| 94 | +- The library needs `--enable-preview` on JDK 21 (`java.lang.foreign` is a preview API there) and |
| 95 | + `--enable-native-access=ALL-UNNAMED`. A packaged app gets JVM options only from |
| 96 | + `compose.desktop.application { jvmArgs(...) }`; the sample's `build.gradle.kts` sets them for |
| 97 | + `JavaExec` tasks only. |
| 98 | + |
| 99 | +## How to check it |
| 100 | + |
| 101 | +Run on the build host, from the repository root, with no `PYTHONHOME` in the launched app's |
| 102 | +environment: |
| 103 | + |
| 104 | +```bash |
| 105 | +./gradlew :sample:createDistributable --console=plain > .tmp/dist.log 2>&1; echo "EXIT=$?" |
| 106 | + |
| 107 | +# The prefix is in the image (macOS paths; Linux: sample/build/compose/binaries/main/app/<name>/lib/app/resources) |
| 108 | +APP=sample/build/compose/binaries/main/app/org.thisisthepy.python.multiplatform.demo.app |
| 109 | +ls "$APP/Contents/app/resources/python-multiplatform-home/lib" # libpython3.14.dylib python3.14 |
| 110 | +test -f "$APP/Contents/app/resources/python-multiplatform-home/lib/python3.14/encodings/__init__.py" && echo STDLIB_OK |
| 111 | +find "$APP/Contents/app/resources/python-multiplatform-home" -name __pycache__ | wc -l # 0 |
| 112 | + |
| 113 | +# Launch with PYTHONHOME removed; the sample prints its runtime line and runs asyncio before the window |
| 114 | +env -u PYTHONHOME -u PYTHON_MULTIPLATFORM_LIBPYTHON \ |
| 115 | + "$APP/Contents/MacOS/org.thisisthepy.python.multiplatform.demo" > .tmp/packaged-run.log 2>&1 & |
| 116 | +``` |
| 117 | + |
| 118 | +Expected in `.tmp/packaged-run.log`: `runtime : 3.14.7 ...` and the `await` lines (which import |
| 119 | +`asyncio`), with no `Fatal Python error`. The window stays open; close it or kill the process. |
| 120 | +Running from `/` as the working directory (`cd / && env -u PYTHONHOME "$OLDPWD/$APP/..."`) also |
| 121 | +checks that nothing is extracted into the working directory. |
0 commit comments