Skip to content

Commit e25804c

Browse files
authored
Merge pull request #236 from thisisthepy/feat/klib-object-handles
Klib walker: carry Kotlin objects as handles; run the klib fixture on the iOS simulator (#234)
2 parents 8acb347 + 366c57e commit e25804c

8 files changed

Lines changed: 357 additions & 48 deletions

File tree

‎docs/SPEC.md‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -387,10 +387,24 @@ All in `PM/commonTest`, so they run wherever the interpreter loads.
387387
`toString()` of it; `null` stays `None` for `CharSequence?`); a `CharSequence` parameter is still
388388
declined (#73). `Status: implemented`, `GP/artifact/TextStateBindingTest.kt`,
389389
`python-multiplatform-compose/fixtures/compose/.../TextFieldStateRenderTest.kt`.
390-
- **B-2** The walker on **klibs** (Kotlin/Native libraries). `Status: partial`,
391-
`GP/artifact/KlibScannerTest.kt` and `python-multiplatform-gradle-plugin/fixtures/klib-artifact` assert that the scanned klib's
392-
declarations are declined with reasons; no klib declaration is bound at run time yet. It reads
393-
top-level functions only: no class constructor, property or member function (B-1, #215).
390+
- **B-2** The walker on **klibs** (Kotlin/Native libraries). `Status: partial`. It reads
391+
top-level functions only: no class constructor, property or member function yet (B-1, #215).
392+
Since #234 a **Kotlin object crosses as a handle**: a type whose class is a public, non-generic,
393+
non-value `class`, `interface`, `enum class` or `object` of the walked klib is an `OBJECT` slot
394+
or result, non-null or nullable, cast in generated source as the jar producer casts one, and a
395+
handle result carries its ancestry from the klib's supertypes. Handles are the common
396+
trampoline's and `HandleTable`'s, under the jar side's rules: passing one back into Kotlin takes
397+
no second slot, and dropping it releases it exactly once. Still declined, each with its reason:
398+
a type of another klib (`kotlin.coroutines.CoroutineContext`), a class of package `kotlin` other
399+
than the boundary's primitives, an extension (the pinned compiler's klib ABI has no receiver type),
400+
and any `@Deprecated` declaration (the ABI has no level, and a `HIDDEN` one, such as
401+
`kotlinx.coroutines.Job0`, cannot be called from source). From `kotlinx-coroutines-core` this binds
402+
`Job`, `SupervisorJob`, `MainScope`, `Mutex`, `Semaphore`, `newSingleThreadContext` and
403+
`newFixedThreadPoolContext`. `GP/artifact/KlibScannerTest.kt`, `GP/artifact/KlibObjectHandleTest.kt`;
404+
`python-multiplatform-gradle-plugin/fixtures/klib-artifact` builds `androidNativeArm64` and
405+
`iosSimulatorArm64` from one table in `nativeMain`, and its suite (`WalkedKlibArtifactHandleTest`,
406+
`WalkedKlibArtifactTableTest`, `WalkedKlibArtifactPythonImportTest`) runs through Gradle on the iOS
407+
simulator (`iosSimulatorArm64Test`); on androidNative it is linked by the build and run by hand.
394408
- **B-3** Default arguments can be omitted from Python (a call per subset of defaulted parameters,
395409
ambiguous omissions refused). `Status: implemented`, `GP/artifact/DefaultOmissionTest.kt`,
396410
`python-multiplatform-gradle-plugin/fixtures/artifact/.../WalkedArtifactDefaultOmissionTest.kt`, `PM/commonTest/.../PythonxDefaultsTest.kt`.

‎python-multiplatform-gradle-plugin/fixtures/klib-artifact/build.gradle.kts‎

Lines changed: 46 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -4,17 +4,18 @@
44
* go into.
55
*
66
* `:python-multiplatform-gradle-plugin-fixtures:artifact` is `docs/design/ecosystem.md` §5b's second producer over a **jar** -- this is
7-
* the same producer over a **klib**, ROADMAP §16e's "investigated, not implemented" half. One target
8-
* only, and it is `androidNativeArm64` rather than `desktop`: a jar is what a JVM target resolves, a
9-
* klib is what a Kotlin/Native target resolves, and `PythonArtifactBindingsTask
7+
* the same producer over a **klib**. Kotlin/Native targets rather than `desktop`: a jar is what a JVM
8+
* target resolves, a klib is what a Kotlin/Native target resolves, and `PythonArtifactBindingsTask
109
* .artifactConfiguration`/`.artifactSourceSet` are single-valued (ROADMAP §16f #1) -- one module can
1110
* wire one walked configuration, so this fixture is the klib producer's own module rather than a
12-
* second target bolted onto `:artifact`.
11+
* second target bolted onto `:artifact`. Since issue #234 it builds `androidNativeArm64` and
12+
* `iosSimulatorArm64`: the androidNativeArm64 klib is walked, the table is generated into the shared
13+
* `nativeMain`, and the suite in `nativeTest` runs through Gradle on the simulator.
1314
*
1415
* `kotlinx-coroutines-core` is the klib walked, for the same reason JUnit is `:artifact`'s: a real,
1516
* independently-versioned third-party artefact nothing here controls or cooperates with.
1617
*
17-
* ### The suite is linked by the build and run by hand
18+
* ### On androidNativeArm64 the suite is linked by the build and run by hand (`iosSimulatorArm64Test` runs it)
1819
*
1920
* KGP gives `androidNativeArm64` a `linkDebugTestAndroidNativeArm64` and **no run task** -- it
2021
* registers one only where it knows how to reach a host, and an Android device is neither the build
@@ -38,17 +39,32 @@ plugins {
3839
id("io.github.thisisthepy.python.multiplatform.bindings")
3940
}
4041

42+
val pythonDownloadDir = project(":python-multiplatform").layout.buildDirectory.dir("python-standalone").get().asFile
43+
val fixturePythonVersion = project.findProperty("pythonVersion")?.toString() ?: project.rootProject.version.toString()
44+
val fixturePythonLibVersion = fixturePythonVersion.split('.').subList(0, 2).joinToString(".")
45+
4146
kotlin {
47+
// Applied now rather than after evaluation, so `nativeMain`/`nativeTest` exist for the blocks below
48+
// and for `pythonBindings.artifactSourceSet` (issue #234).
49+
applyDefaultHierarchyTemplate()
50+
4251
androidNativeArm64 {
4352
// Mirrors `:python-multiplatform-ksp:fixtures:app`'s own androidNativeArm64 linker config: the test binary needs
4453
// the same CPython symbols resolved at final link time, since `python-multiplatform`'s klib
4554
// only declares them via cinterop and does not itself embed a static libpython.
46-
val downloadDir = project(":python-multiplatform").layout.buildDirectory.dir("python-standalone").get().asFile
47-
val pyVersion = project.findProperty("pythonVersion")?.toString() ?: project.rootProject.version.toString()
48-
val libVersion = pyVersion.split('.').subList(0, 2).joinToString(".")
49-
val targetExtractDir = "$downloadDir/extracted/$pyVersion/android-aarch64/prefix"
55+
val targetExtractDir = "$pythonDownloadDir/extracted/$fixturePythonVersion/android-aarch64/prefix"
5056
binaries.getTest(org.jetbrains.kotlin.gradle.plugin.mpp.NativeBuildType.DEBUG).linkerOpts.addAll(
51-
listOf("-L$targetExtractDir/lib/", "-lpython$libVersion", "-Wl,--allow-shlib-undefined"),
57+
listOf("-L$targetExtractDir/lib/", "-lpython$fixturePythonLibVersion", "-Wl,--allow-shlib-undefined"),
58+
)
59+
}
60+
// Issue #234: the simulator target, so the suite runs through Gradle (`iosSimulatorArm64Test`)
61+
// rather than only being linked. Linked the way `:python-multiplatform`'s own simulator test
62+
// binary is: against the xcframework's simulator slice, with an explicit `-rpath`, because
63+
// `Python.framework` has an @rpath-relative install name and is not copied beside the binary.
64+
iosSimulatorArm64 {
65+
val slice = "$pythonDownloadDir/extracted/$fixturePythonVersion/ios/Python.xcframework/ios-arm64_x86_64-simulator"
66+
binaries.getTest(org.jetbrains.kotlin.gradle.plugin.mpp.NativeBuildType.DEBUG).linkerOpts.addAll(
67+
listOf("-framework", "Python", "-F$slice", "-rpath", slice),
5268
)
5369
}
5470

@@ -58,33 +74,47 @@ kotlin {
5874
implementation(projects.pythonMultiplatform)
5975
}
6076
}
61-
val androidNativeArm64Main by getting {
77+
// Both native targets share the walked table and the suite (issue #234): the table is
78+
// generated into `nativeMain`, and the tests live in `nativeTest`.
79+
val nativeMain by getting {
6280
dependencies {
6381
// A real, separately-published klib. Nothing about it knows it is going to be
6482
// walked -- KlibScanner reads its ABI the same way it would read anyone else's.
6583
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.8.1")
6684
}
6785
}
68-
val androidNativeArm64Test by getting {
86+
val nativeTest by getting {
6987
dependencies {
7088
implementation(libs.kotlin.test)
7189
}
7290
}
7391
}
7492
}
7593

94+
// The iOS `Python.framework` carries no standard library; `:python-multiplatform` unpacks the
95+
// simulator's, and the test binary is pointed at it the way that module points its own.
96+
tasks.withType<org.jetbrains.kotlin.gradle.targets.native.tasks.KotlinNativeSimulatorTest>().configureEach {
97+
dependsOn(":python-multiplatform:extractIosSimulatorStdlib")
98+
val pythonHome = project(":python-multiplatform").layout.buildDirectory.dir("python-stdlib/ios-simulator").get().asFile.absolutePath
99+
// simctl only forwards variables into the spawned process when they carry this prefix.
100+
environment("SIMCTL_CHILD_PYTHONHOME", pythonHome)
101+
environment("PYTHONHOME", pythonHome)
102+
}
103+
76104
pythonBindings {
77105
role.set("app")
78106
processor.set(projects.pythonMultiplatformKsp)
79107

108+
// The androidNativeArm64 klib is walked (what `KlibScannerTest` walks too); the table lands in
109+
// `nativeMain`, so the simulator target compiles the same calls against its own klib.
80110
artifactConfiguration.set("androidNativeArm64CompileKlibraries")
81-
artifactSourceSet.set("androidNativeArm64Main")
111+
artifactSourceSet.set("nativeMain")
82112
// Broad on purpose, unlike `:artifact`'s narrow `junit.runner`/`junit.framework`: coroutines'
83113
// public surface is almost entirely `CoroutineScope`/`Job`/`Flow`-typed extensions, plus a couple
84114
// of `@PublishedApi internal` declarations that are binary-visible but not source-visible outside
85-
// the library that owns them -- none of those are bindable today, so the whole namespace yields
86-
// **zero** declarations. `WalkedKlibArtifactTableTest` pins that as the correct number, not a gap:
87-
// an earlier version bound the `@PublishedApi internal` one and failed to compile the first time
115+
// the library that owns them. Those stay declined; since issue #234 the factories whose types are
116+
// the klib's own classes bind (`Job`, `Mutex`, ...), and `WalkedKlibArtifactTableTest` pins the list.
117+
// An earlier version bound the `@PublishedApi internal` one and failed to compile the first time
88118
// its output was actually built rather than only unit-tested -- `KlibScanWorkAction`'s KDoc has
89119
// the story. See that file's own KDoc for what the walker declines and why.
90120
artifactIncludePackages.set(listOf("kotlinx.coroutines"))
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
package fixture.artifact
2+
3+
import python.multiplatform.ffi.Python3
4+
import python.multiplatform.ffi.upcall.PythonProxySource
5+
import python.multiplatform.generated.FunctionTable
6+
import python.multiplatform.generated.artifacts.ArtifactTable
7+
import python.multiplatform.reflection.HandleTable
8+
import python.multiplatform.reflection.ObjectReference
9+
import python.multiplatform.reflection.UpcallTable
10+
import python.native.ffi.UpcallEntry
11+
import kotlin.test.AfterTest
12+
import kotlin.test.BeforeTest
13+
import kotlin.test.Test
14+
import kotlin.test.assertEquals
15+
import kotlin.test.assertFalse
16+
import kotlin.test.assertNotNull
17+
import kotlin.test.assertNull
18+
19+
/**
20+
* Issue #234 on Kotlin/Native: a Kotlin object a walked klib declaration returns crosses as a handle,
21+
* goes back into Kotlin as that object, and follows the jar side's lifetime rules -- the common
22+
* trampoline's and `HandleTable`'s, which the klib boundary uses unchanged.
23+
*
24+
* - A handle Python holds is one `HandleTable` slot; passing it back into Kotlin (`Job(parent)`)
25+
* neither releases it nor takes a second slot.
26+
* - Dropping every Python reference releases each slot exactly once: the live count returns to its
27+
* baseline, the raw handle no longer resolves, and a second release is refused (no double release).
28+
*
29+
* Red before #234: `Job` is not importable, because the walker declined every `kotlinx.coroutines`
30+
* declaration.
31+
*/
32+
class WalkedKlibArtifactHandleTest {
33+
34+
@BeforeTest
35+
fun installTheWalkersTableIntoAnInterpreter() {
36+
Python3.initialize(silent = true)
37+
UpcallTable.clear()
38+
UpcallTable.install(FunctionTable.fragments + ArtifactTable.fragments)
39+
val mainGlobals = Python3.import("__main__").getAttr("__dict__")
40+
check(UpcallEntry.publish(mainGlobals.pointer)) { "the upcall bootstrap could not be published" }
41+
PythonProxySource.install()
42+
}
43+
44+
@AfterTest
45+
fun cleanup() {
46+
UpcallTable.clear()
47+
}
48+
49+
@Test
50+
fun aHandleGoesBackIntoKotlinAndIsReleasedExactlyOnce() {
51+
Python3.exec("import gc\ngc.collect()")
52+
val baseline = HandleTable.liveCount
53+
Python3.exec(
54+
"""
55+
from kotlinx.coroutines import Job
56+
_kh_parent = Job(None)
57+
_kh_raw = _kh_parent._pm_handle
58+
_kh_children = [Job(_kh_parent) for _ in range(3)]
59+
assert _kh_parent._pm_handle == _kh_raw, 'passing the handle into Kotlin changed it'
60+
""".trimIndent(),
61+
)
62+
// One slot for the parent and one per child: passing the parent in took no slot of its own.
63+
assertEquals(baseline + 4, HandleTable.liveCount)
64+
val raw = Python3.import("__main__").getAttr("_kh_raw").toString().toLong()
65+
assertNotNull(HandleTable.resolveRaw(raw), "the parent's handle should still resolve while Python holds it")
66+
67+
Python3.exec("del _kh_parent, _kh_children\nimport gc\ngc.collect()")
68+
assertEquals(baseline, HandleTable.liveCount, "every slot is released exactly once")
69+
assertNull(HandleTable.resolveRaw(raw), "a released handle must not resolve")
70+
assertFalse(HandleTable.release(ObjectReference(raw)), "a second release must be refused")
71+
}
72+
73+
@Test
74+
fun aNullableHandleParameterTakesNone() {
75+
Python3.exec(
76+
"""
77+
from kotlinx.coroutines import Job
78+
_kh_job = Job(None)
79+
assert _kh_job is not None
80+
del _kh_job
81+
""".trimIndent(),
82+
)
83+
}
84+
}

python-multiplatform-gradle-plugin/fixtures/klib-artifact/src/androidNativeArm64Test/kotlin/fixture/artifact/WalkedKlibArtifactPythonImportTest.kt renamed to python-multiplatform-gradle-plugin/fixtures/klib-artifact/src/nativeTest/kotlin/fixture/artifact/WalkedKlibArtifactPythonImportTest.kt

File renamed without changes.

python-multiplatform-gradle-plugin/fixtures/klib-artifact/src/androidNativeArm64Test/kotlin/fixture/artifact/WalkedKlibArtifactTableTest.kt renamed to python-multiplatform-gradle-plugin/fixtures/klib-artifact/src/nativeTest/kotlin/fixture/artifact/WalkedKlibArtifactTableTest.kt

Lines changed: 21 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -55,30 +55,33 @@ class WalkedKlibArtifactTableTest {
5555
}
5656

5757
/**
58-
* The fixed text: everything `KlibScanner` bound out of the whole `kotlinx-coroutines-core`
59-
* klib under the `kotlinx.coroutines` namespace -- nothing, correctly. This class's own KDoc has
60-
* the reason; `KlibScannerTest.aRealThirdPartyKlibIsScannedWithoutCrashingAndDeclinesEveryDeclaration`
61-
* pins the same fact walking the klib directly rather than through the build. If the two ever
62-
* disagree, the wiring between `KlibScanner` and this generated table is what broke.
63-
*
64-
* A generated fragment file is only ever written when a walked artefact contributes at least one
65-
* entry (`PythonArtifactBindingsTask.generate`'s `if (entries.isEmpty()) return@forEach`), so an
66-
* empty klib means `ArtifactTable.fragments` itself has zero elements -- not one fragment with an
67-
* empty `entries()`.
58+
* The fixed text: everything `KlibScanner` bound out of the whole `kotlinx-coroutines-core` klib
59+
* under the `kotlinx.coroutines` namespace. Zero before issue #234, when no Kotlin object could
60+
* cross the klib boundary; since then the factories whose types are the klib's own classes.
61+
* `KlibScannerTest.aRealThirdPartyKlibIsScannedWithoutCrashingAndBindsItsHandleFactories` pins the
62+
* same list walking the klib directly; if the two disagree, the wiring between `KlibScanner` and
63+
* this generated table is what broke.
6864
*/
6965
@Test
70-
fun theWalkerCorrectlyDeclinesEveryDeclarationInTheKlib() {
71-
assertEquals(emptyList(), ArtifactTable.fragments)
66+
fun theWalkerBindsExactlyTheKlibsHandleFactories() {
7267
ArtifactTable.registerInto()
73-
assertEquals(emptyList(), UpcallTable.entries())
68+
assertEquals(
69+
listOf(
70+
"kotlinx.coroutines.Job",
71+
"kotlinx.coroutines.MainScope",
72+
"kotlinx.coroutines.SupervisorJob",
73+
"kotlinx.coroutines.newFixedThreadPoolContext",
74+
"kotlinx.coroutines.newSingleThreadContext",
75+
"kotlinx.coroutines.sync.Mutex",
76+
"kotlinx.coroutines.sync.Semaphore",
77+
),
78+
UpcallTable.entries().map { it.name }.filter { it.startsWith("kotlinx.") }.sorted(),
79+
)
7480
}
7581

76-
/**
77-
* `registerInto` on an empty `ArtifactTable` must not fail, and must not resolve a name that was
78-
* never bound -- the two failure modes a table with zero fragments could otherwise hide.
79-
*/
82+
/** `registerInto` must not resolve a name that was never bound: `checkIndexOverflow` stays declined. */
8083
@Test
81-
fun registeringAnEmptyTableIsHarmlessAndResolvesNothing() {
84+
fun aDeclinedNameStillResolvesToNothing() {
8285
ArtifactTable.registerInto()
8386
val entry = UpcallTable.resolve("kotlinx.coroutines.flow.internal.checkIndexOverflow")
8487
assertTrue(!entry.isValid, "checkIndexOverflow must not resolve: it is @PublishedApi internal and must stay declined")

0 commit comments

Comments
 (0)