Repository navigation
Conversation
First step of replacing the embedded Alpine system with the native toolchain libraries. This adds the driver the app needs in order to build without a guest; the guest code is still present and still what XForgeEnvironment hands to the Build screen, so nothing behaves differently yet — but everything the native path needs now exists. - `NativeBuildPlan` / `NativeBuildPlanFactory`: read a package directory into a fully resolved plan (module, sources, resources, frameworks, search paths). It refuses, by name, the two things an in-process driver cannot do — SwiftPM dependency resolution and asset catalogs — instead of failing halfway. - `NativeToolchainInvocation`: pure argument construction for `swift-frontend` and `ld64.lld`. Pure on purpose: the compiler only exists in a device build, so a wrong argument list is otherwise undiscoverable until a build runs on a phone. The frontend convention is documented where it is implemented — the driver hands `performFrontend` everything after `-frontend`, so neither the program name nor `-frontend` may appear. - `NativeBuildExecutor`: the `BuildExecutor` the pipeline will use. Compiles Swift and C/Obj-C through the bridge, links with LLD, assembles `.app`, packages an `.ipa`, and keeps the UI responsive by running the blocking compiler calls off the main actor. - `NativeToolchainCapabilities`: reports clang / LLD / swift-frontend / SDK honestly. Clang and LLD alone can build, so the Swift frontend is not required to start — only to compile Swift, which it says plainly. - `NativeSDK.install(fromRemote:)`: install a Darwin SDK from an explicit URL, so a caller that already resolved the asset does not resolve it twice. - `NativeToolchainBridge.mm`: accept a CLI-shaped `-frontend …` argument list by stripping the flag, rather than handing `parseArgs` an argument it rejects. - Projects move to the app's own container: `Project.rootURL` derives the project directory from its name, and `XForgeEnvironment.projectsDirectory` is the single place that decides where that is. `rootPath` still holds the guest-shaped string this model was built around; removing it is the next step, and two stored locations for one project is how a build ends up reading a directory nobody created. Verified by 15 new unit tests (`NativeBuildTests`) covering the plan factory's acceptance and refusals, the frontend/linker argument lists, the main.swift and Swift-runtime cases, and search-path filtering. They run on a simulator, where the compiler is absent, which is exactly why the tested half is argument construction.
XForge no longer boots a Linux guest. The toolchain that builds an app is the
LLVM/Clang/LLD and Swift libraries linked into the process, and the app's own
code does everything the guest used to be asked for.
Removed outright (with the guest they belonged to):
- `App/EmbeddedVM/**` — the ish-arm64 engine bridge, the fakefs rootfs installer,
the emulator, the console/shell sessions. The app links no Linux kernel now.
- `App/Build/EmbeddedLinuxExecutor.swift` and `App/Build/SDKInstaller.swift` —
replaced by `NativeBuildExecutor` and `NativeSDK`.
- `App/Services/{SystemComponents,GuestNetwork,ToolchainManager,IPAConfigureSignService}.swift`,
`HostDNS.{c,h}`, `DownloadManager.swift` — guest provisioning, guest DNS,
the in-guest component installer and the zsign job.
- `App/Views/Terminal/**`, `EngineLogView`, `DownloadsView` — the terminal had
nothing behind it once the guest was gone, and the download screen existed to
drive `curl` inside it.
- `AppTests/{TerminalSessionTests,TerminalKeyConfigurationTests,TerminalBufferTests}.swift`
and everything in `ReleaseResolutionTests` that asserted guest behaviour.
Replaced rather than deleted:
- **Build path.** `NativeBuildExecutor` compiles each translation unit through the
bridge, links with LLD, assembles the `.app`, and packages the `.ipa`, streaming
every step into the Build screen's console. `BuildManager`'s stages lost their
guest commands: the SDK stage installs the bundle into the app container, and
"Configure" now validates the project directory instead of `mkdir`-ing inside a
guest.
- **Signing.** `IPASigningJob` + `AppBundleSigner` sign an IPA with XKit, in
process. The private key is imported through the Security framework and used in
memory: the `.p12` password file that `zsign` needed on disk is gone with it.
- **Project files.** `ProjectFiles` replaces `GuestProjectFiles`, with the same
relative-path discipline (a `..` is rejected, not normalized). The file browser,
the source browser/editor and the Package.swift editor all use it.
- **Export.** `ProjectExporter` zips the project directory with ZIPFoundation
(skipping `.xforge-build`/`.build`) instead of `tar`ring it inside the guest.
- **Log.** `XForgeLog` is an ordinary file the user can share; the log file used
to be the engine's kernel messages through the ish bridge.
- **Import.** A folder picked in Files is copied in, since an iOS process cannot
run `git`. The dependencies editor is gone: a project that declares SwiftPM
dependencies cannot be resolved without a package manager, which cannot run
here, so `NativeBuildPlanFactory` refuses such a project with a message that
names the packages — an editor that could only produce unbuildable projects was
worse than no editor.
`project.yml` no longer builds an engine, links `-lish`/`-lz`/`-liconv`, sets
`GUEST_ARM64`, or needs `Vendor/ish-arm64*` to exist. A plain clone builds.
**What this costs, stated plainly:** the Swift frontend libraries are not part of
the toolchain artifact yet (`.github/workflows/native-toolchain.yml` builds Clang
and LLD only), so a project whose sources are Swift still cannot be compiled —
the build now fails at the Swift compile step with `swiftFrontendMissing`, naming
the count of files, instead of silently needing a guest. C/Objective-C targets
compile and link today. The guest is not coming back as a fallback: it was 1.6 GB
of carefully provisioned Linux, and the honest replacement is a toolchain artifact
that carries the frontend.
The Build screen's stages, the Toolchain screen (capabilities + SDK + smoke test)
and the Settings screen (log, project files, diagnostics) all report from the
in-process toolchain now.
The code removal in the previous commit left the CI still provisioning the guest it no longer uses: a 1.6 GB rootfs download, a meson build of the ish-arm64 engine, and a validation step that checks the guest's zsign installer — which failed as soon as the installer was deleted, which is how this was found. `Build XForge IPA` is now fourteen steps instead of twenty-two, and builds the app rather than the guest: - no rootfs download or verification, no engine checkout/cache/build, no guest tooling validation; - the build tools step installs `xcodegen` and nothing else (meson, ninja, llvm, lld and shellcheck were all for the engine); - the "built app" and "final IPA" checks keep their real content — arm64 binary, identity, entitlements wiring, `Assets.car`, one `.app` in `Payload/`, no `__MACOSX`, the executable is byte-identical after packaging — and lose the rootfs assertions that only described the guest; - the "app must be self-contained" check now greps for the executable rather than `alpine-rootfs.zip`. Also removed: - `EmbeddedLinux/**` — the engine build, the rootfs builder/verifier, the guest provisioning and login scripts, the zsign installer and its patch. - `Tools/engine-smoke/**` — the harness that compiled programs inside the guest. - `Vendor/ish-arm64` (and `.gitmodules`): a submodule whose sources nothing reads any more. `make bootstrap` no longer inits it; `make toolchain` installs the compiler bundle the app actually links. - `.github/workflows/build-rootfs.yml` (publishes the rootfs) and `.github/workflows/ios-build.yml`: the latter staged `EmbeddedLinux/fetch-rootfs.sh` and would fail now, and `build-ipa.yml` replaced it in September. `ios-share.yml` stays — it is the simulator-sharing capability for another app and never referenced the guest. Two Swift 6 fixes found while re-reading the new code, both of which the release build would not have reported: the signer needs `import Security` for `SecPKCS12Import`, and a static `ISO8601DateFormatter` in `XForgeLog` is a non-`Sendable` global that strict concurrency rejects — the formatter is built per line now. The key material is also labelled PKCS#1 in its PEM envelope, because that is what `SecKeyCopyExternalRepresentation` returns.
Deleting DownloadManager took the error type with it, but XForgeReleases and NativeSDK both throw it. Recovered from the deleted file, minus the .notDownloaded case that only the download manager could produce.
Passing AppBundleSigner.Identity into an async signer is a data-race error under strict concurrency because [String: Any] is not Sendable. The entitlements now travel as property-list bytes and are decoded inside the signer, which keeps the non-Sendable value in the scope that consumes it — and the struct is honestly Sendable rather than marked @unchecked.
FileManager.DirectoryEnumerator cannot be iterated from an asynchronous context — Swift 6 rejects the for-in outright — so the walk moved into a synchronous helper. The path helper also shared its name with a parameter, which made one call site read as an attempt to call a String.
The README, the design notes and the changelog still described an app that boots Alpine, ships a rootfs, installs its toolchain inside a guest and signs with zsign. None of that is true any more, so the documentation now describes what the code does — including what it cannot do yet: - **README** — the pipeline (project → build plan → in-process compile/link → `.app` → unsigned `.ipa` → XKit signing), what works today (C/Objective-C) and what does not (Swift, SwiftPM dependencies, asset catalogs, Apple ID provisioning, on-device install), how to install the compiler bundle, and the three workflows that remain. - **CHANGELOG** — a 0.6.0 entry that opens by saying this is a breaking change: the guest and the Terminal are gone, and nothing replaces them. - **Docs/DESIGN.md** — the architecture without a guest, and a correction: projects are ordinary container directories and *are* backed up. The old text argued the opposite, because they used to live inside the guest filesystem. - **Docs/IPA-BUILD.md**, **Docs/NATIVE-TOOLCHAIN.md**, **Docs/XCODE-ALTERNATIVE.md** — the stages, the bundle (Clang and Mach-O LLD; **no Swift frontend yet**), and an honest status for each remaining gap. - **Docs/ISH-ARM64-INTEGRATION.md** — deleted with the integration it documented. Follow-ups the rewrite exposed, all fixed here: - the Toolchain screen's smoke test only compiled while its own text claimed it linked — it now compiles *and* links (the C fixture gained a `main`), and it exercises the Swift frontend too when that is linked, rather than leaving `smokeCompileSwift` unreachable; - the package stage was titled "Package & sign .ipa" but never signs; signing is the separate XKit screen; - `Makefile` named the wrong artifact (`-ios-arm64` vs `-arm64-ios`) in the `gh run download` hint; - `project.yml` pointed at a workflow file that does not exist, and still wired a `Support/Resources` directory nothing fills — removed, with its `.gitignore` rules and the engine/rootfs ignore blocks; - `Tools/test-verify-toolchain.sh` tested the deleted rootfs verifier and could not run at all; - comments that described the deleted build executor, and a settings tab labelled "Tab 5" in an app with three tabs.
…omments The review of the rewritten documentation turned up four leftovers: - NativeToolchainView and ToolchainView showed the same three things — compiler status, SDK install/remove, smoke tests — so the older one is gone. The surviving screen reports capabilities, installs or imports the SDK, and runs a compile-and-link smoke test (plus the Swift one when the frontend is linked). - `make bootstrap` depended on ARCHIVE and therefore failed on a plain clone; the target is gone rather than left as a trap. `make gen` is what a clone needs. - Project.rootPath and NativeSDK still described themselves in terms of the guest (a guest path, SwiftPM's SDK store inside a toolchain installation) that no longer exists. - the docs' own follow-ups (smoke test that claimed to link, a stage titled 'Package & sign', a stale workflow name, the artifact-name mismatch in the Makefile hint, the orphaned verify-rootfs harness, the empty Support/Resources resource directory) landed with the documentation commit.
The toolchain bundle is Clang + Mach-O LLD, which is why a Swift project cannot be compiled on device. This adds the missing half: an opt-in stage that builds `swiftFrontendTool` — the static library holding swift::performFrontend — for arm64-apple-ios, merges it into the same archive, and stages Swift's headers so the bridge's Swift code path compiles. The pieces that are not obvious: - **Swift needs its own LLVM.** The frontend is developed against swiftlang/llvm-project (the swiftasync calling convention, the coroutine passes, the C++ API it is compiled against), so the Swift path checks out that fork at the paired swift/release/6.2 branch. Upstream is kept for the LLVM-only path. - **`-DSWIFT_INSTALL_COMPONENTS=` is the fix for the configure failure** that stops a Swift-for-iOS configure dead: Swift's CMake writes that variable as a *cache* entry, so an empty value on the command line survives its default and no component is installed, which means `swift_install_in_component` returns early and its `install(TARGETS ...)` never runs. CMake refuses that rule for an executable with no BUNDLE destination on an iOS target, and Swift's host tools have none. - **The version has to match the SDK's runtime.** The darwin SDK the app compiles against was built from an Xcode, and Xcode 26.6 ships Swift 6.2.4, so the frontend is pinned to swift-6.2.4-RELEASE; a mismatch means the frontend cannot read the standard library's modules. It is an input, not a constant. - **`configure_only`** exists because the configure is where this fails, and waiting hours to learn that is a waste: it stops after configuring (and after the header compile-check, which proves the bridge finds Swift's headers) so the expensive part is only paid once the shape is right. - The iOS builds are cached under a key that deliberately ignores this file — editing the workflow is what iterating on it looks like, and it must not rebuild a compiler that did not change. `cache_salt` forces one.
The bundle's manifest says whether it includes Swift's frontend libraries, and the bridge needs the matching define: without -DXFORGE_HAS_SWIFT_FRONTEND the Swift entry point compiles to a stub and the app reports 'swift-frontend: missing' for a bundle that has it. Two fixes found while testing the three paths (no bundle / Clang and LLD only / with the frontend) against a fixture bundle: the archive list came from process substitution, which needs /dev/fd, and the revision line came from a grep pipeline that leaves a first read of the manifest awkward on a bare shell. Both are plain redirections now, and the fixture test asserts the resulting XFORGE_NATIVE_CFLAGS for each case.
The first Swift configure stopped at SwiftSharedCMakeConfig.cmake:62: Attempting to cross-compile swift standalone but no native LLVM build found. Please cross-compile LLVM as well. That check only runs when `LLVM_TABLEGEN` is unset; setting it (and CLANG_TABLEGEN) to the host tools this workflow already builds is what it is asking for. actions/cache also only writes on success, so a run that failed in the Swift configure stored nothing and would have rebuilt LLVM from scratch on every iteration. Restore and save are separate steps now, and the save runs with `if: always()` before the step most likely to fail.
The configure got past the tablegen requirement and stopped on two Linux-isms: cmake/modules/Libdispatch.cmake:28: libdispatch requires a newer clang compiler lib/Basic/CMakeLists.txt:10: Could NOT find UUID (missing: UUID_LIBRARY) Both branches are selected by `CMAKE_SYSTEM_NAME STREQUAL "Darwin"`, which CMAKE_SYSTEM_NAME=iOS fails. Swift's own build-script-impl configures iOS the same way this now does — sysroot, architectures, deployment target, compiler target triple and SWIFT_HOST_VARIANT_SDK, with no system name — so the target is unchanged and only the platform *branch* is corrected. One consequence of that: the build is no longer "cross-compiling" as far as CMake is concerned, so Swift's CMake takes the host branch and looks for its native tools in LLVM_TOOLS_BINARY_DIR — the iOS LLVM build's bin, which LLVM_BUILD_TOOLS=OFF leaves empty. The host tools are mirrored there now.
With the platform branch corrected, the Swift configure now completes
("Configuring done") and fails while generating:
CMake Error at lib/Tooling/libSwiftScan/CMakeLists.txt:19 (add_dependencies):
The dependency target "clang" of target "libSwiftScan" does not exist.
That comes from CLANG_BUILD_TOOLS=OFF: a unified build normally defines the
`clang` executable target, and Swift's CMake depends on it existing even when
nobody builds it. Turning it on defines the target and nothing builds it — the
Swift build still asks ninja for one static library.
configure_only reached the end of the Swift configure for iPhoneOS: -- Configuring done (9.1s) and then failed in the bridge compile-check on 'clang/Basic/DiagnosticGroups.inc' file not found — a header the Clang build generates, which configure_only does not run. The check belongs to a real build, so it is skipped with the rest of the build steps, and the configure-only path reports what it actually produced (the tablegen'd SwiftOptions archive, and the listing of swift-build/lib).
The Swift frontend built (937/937 edges, lib/libswiftFrontendTool.a), and then my own verification failed it: 29 archives, all arm64, and 'nm -gU … | grep performFrontend' found nothing. The symbol is defined in lib/FrontendTool/FrontendTool.cpp, so it is in that archive — the check was wrong, not the build, and it stopped the job five seconds after an hour of work. - the symbol search now uses the host llvm-nm when it exists, prints what it found, and *warns* rather than failing. The gate is the link below it, which compiles the bridge with XFORGE_HAS_SWIFT_FRONTEND and fails on a genuinely missing symbol — a functional check instead of a fragile one; - the architecture check stays an error, because a mixed-architecture bundle is worse than one that fails here; - the cache is split. The old single save ran before swift-build existed, so the 52-minute Swift build was being thrown away on every iteration; LLVM's half and Swift's are separate entries now, each saved right after it is produced.
The Swift frontend builds now (937/937 edges, cached), and the job failed in the bridge compile-check with two independent problems: 1. Apple's SDK predefines IBAction and IBOutlet as macros, and clang's generated AttrList.inc expands ATTR(IBAction) into `class IBAction##Attr;` — the macro is pasted into the token paste and the header cannot compile. Both are undefined before any clang header is included, which is what clang's own headers do on the revisions where this bites. 2. Swift's LLVM fork is the *older* clang API generation: DiagnosticOptions is still reference-counted, TextDiagnosticPrinter takes a DiagnosticOptions*, and CompilerInstance has no constructor taking an invocation. Upstream LLVM is the newer one: a plain value taken by reference, invocation in the constructor. The bundle can be built from either — this workflow builds the LLVM-only bundle from upstream and the Swift-carrying one from the fork — so both have to compile. The split is a template parameter rather than an `if constexpr` in place: outside a template the discarded branch is still type-checked, which would defeat the point. The marker for the generation is the base class (is_base_of RefCountedBase), not a member — Retain() is protected, so a member-based trait returns false for both and silently picks the wrong branch.
Both caches were restored and ninja rebuilt anyway: a restored output carries its original timestamp, the checkout gives the sources 'now', and ninja trusts timestamps — so an hour of LLVM and another of Swift were repeated for nothing. The restored trees are touched after extraction, which is safe precisely because any change to a source ref changes the cache key.
The previous attempt still failed, and the log says why in one line: the build was in 'xf_clang_compile_impl<false>' — the *newer* API — against the fork's headers. Two mistakes, both mine: - the dispatch was 'if (xf_clang_uses_refcounted_diagnostics())', a *runtime* condition, so both specialisations were instantiated and the one that does not belong to this clang had to compile anyway. The whole entry point is now a template and the choice is its template argument, a constant expression, so exactly one instantiation exists; - the trait was is_base_of on RefCountedBase. The printer's constructor is the honest test — the older generation takes a DiagnosticOptions*, the newer one a reference, and a pointer never converts to a reference. The lesson generalises: 'if constexpr' outside a template discards nothing, so a generation split has to be a template parameter from the start.
The early touch ran before CMake's configure, and configure rewrites generated headers (llvm/Config/…). A header newer than the outputs invalidates nearly every edge that includes it, so ninja rebuilt the world anyway. The touches now sit immediately before the two build steps, where nothing regenerates inputs in between.
Every run so far has built the app *without* a toolchain: the bundle is a separate workflow's artifact, so prepare-xcode.sh writes the disabled xcconfig and the bridge compiles to the 'not available' stub. Nothing in CI has ever made the app link LLVM, Clang, LLD or Swift. `with_toolchain: true` downloads the artifact, installs it where the xcconfig points, prints the manifest and the generated xcconfig, and builds on from there — which is also the only way to find out what linking the merged archive costs.
The template dispatch still failed, and the log explains why: the errors were in the *newer* API branch against the fork's headers even though the trait should have selected the other one. The reason is the rule I misread: a discarded `if constexpr` branch inside a template is exempt from instantiation only for constructs that depend on the template parameter. Mine did not — `DiagnosticOptions diagOpts;` is a concrete type — so both halves were checked and one of them could never compile. There is no way to hold two API generations in one function that does not depend on them. So there is one generation now, and it is the fork's (reference-counted `DiagnosticOptions`, pointer-taking printer, `setInvocation`), which is also the one Swift itself is built against. The workflow checks out `swiftlang/llvm-project` for *every* bundle instead of upstream for the LLVM-only one, which is what makes a single bridge possible — and it pins a revision that was previously an unpinned checkout of llvm-project main. This deletes the matching machinery rather than adding to it: the trait, the two templates and the dispatcher are gone, and the compile path is a straight line.
The staging step copies the build tree's Swift headers on top of the source
tree's, and those two disagree about one path: the build generates a *file* at
`swift/bridging` (the C++ interop header, included as <swift/bridging>) while
`swift/include/swift/` has a *directory* of the same name, holding ASTGen.h and
MacroEvaluation.h. On a case-insensitive filesystem that is a single path, so the
copy dies with
cp: ...: Is a directory
and stages nothing. It has not bitten yet only because step 20 is skipped while
the Swift frontend check above it fails; the moment that step passes, this one
fails instead. Same fix as the one the unified toolchain workflow needed.
Sources go to `dist/include`, the build tree to `dist/include-generated`, and
consumers search the generated root first — the order a normal Swift/LLVM
cross-build uses. `cp -R -L` is applied for a second reason: the build tree
mirrors headers as symlinks into this checkout, and a bundle shipping those links
is dead on any other machine.
Guarded with a required-roots check and a symlink check so a partial or
non-self-contained bundle cannot be published.
`cache_salt` exists so an LLVM tree can be discarded when its source changes — which it just did, upstream for the fork. Applying it to the Swift tree as well invalidated a 45-minute build that did not change: the Swift tree is a function of the Swift refs and the SDK, both in its key already.
The check that keeps failing needs no libraries — the headers of the LLVM and Swift the bundle is built from, the headers those builds generate, and a compiler — and all of that is already in the two caches the toolchain workflow writes. This workflow restores them, reconfigures so the generated headers match the sources, touches the trees and compiles the bridge: about ten minutes, against the 40-90 the toolchain workflow spends building before it gets there. It deliberately duplicates the compile-check command: the two are meant to be identical, and a viewer of either shows the same contract. It does not link anything, so it says nothing about the archive or the app — it answers one question, which is the question that has been costing hours.
R0GUEEE hit the same three signatures from the other branch (4fd80cc) and wrote the constructor argument out explicitly rather than relying on the default. Taking that form, plus the __has_include guard on PCHContainerOperations.h it needs. Their commit and 5d0c729 are the same fix reached independently; the difference is that this branch also pins every bundle to that LLVM fork, without which the LLVM-only bundle is a different API generation from the bridge that compiles against it.
It reached the Swift configure in two minutes and failed there — my own workflow's fault, not the bridge's: Swift's configure executes a native cmark and resolves its sibling sources by path, and this workflow checked out neither. Both are mirrored from the toolchain workflow now, and the configure args match it argument for argument, so a failure here means an API problem rather than a missing argument.
The bridge compiled (step 23 passed for the first time) and staging failed at the link smoke with: "_cmark_iter_free", referenced from: parseDocumentImpl(swift::markup::MarkupContext&, …) "_cmark_node_get_type", … ld: symbol(s) not found for architecture arm64 Swift's Markup — the doc-comment parser — calls into cmark, and this workflow only ever built cmark as a *host* tool for Swift's configure step. Nothing built it for iPhoneOS, so nothing provided those symbols; the same is true of cmark-gfm-extensions, which is a separate archive. Both are built for the device now and merged into the bundle. The second half of the commit is the reason the first half was found late: the fast loop only compiled the bridge, so anything wrong with staging or linking was discovered by the 90-minute toolchain run — which is exactly what happened, twice. It now stages and links too, with the same commands, and prints the undefined symbols on failure (the raw log buries them under forty lines of libtool notes).
Same trap as the Swift configure, one layer down:
CMake Error at src/CMakeLists.txt:64 (install):
install TARGETS given no BUNDLE DESTINATION for MACOSX_BUNDLE executable target "cmark-gfm"
On an iOS platform CMake refuses install(TARGETS) for an executable without a bundle
destination, and cmark installs its command-line tool. Configured as Darwin with the
iPhoneOS sysroot and an arm64-apple-ios compiler target — the shape Swift's own iOS
builds use — the rule is valid, the objects are still arm64 iOS, and nothing is
installed.
The Darwin configure worked and ninja rejected the target: cmark's archives are 'libcmark-gfm' and 'libcmark-gfm-extensions' (the bare names are the command-line tool, which only the host build needs). Also drops two cmark options this version does not define — static archives are its default, and the unused-variable warnings were the only effect.
On a pull_request run `inputs.with_swift` is empty, so the cmark *checkout* is skipped while my new cmark-for-iOS step ran anyway — 'The source directory swift-cmark does not exist'. It now carries the same condition as the Swift checkouts and stages, which is the condition that actually decides whether cmark is needed: Swift's Markup is the only consumer.
Caches were the wrong substrate for the fast loop and I should have seen it sooner: the toolchain workflow's saves take six seconds for multi-gigabyte trees, which is not a save, and the fast loop reported 'llvm cache hit:' empty — so it failed a step the toolchain run passes with no generated headers at all. The toolchain workflow now packages the static archives, cmark and the generated headers into an artifact (with `always()`, so a failure after the builds still publishes them — the case the fast loop exists for), and the fast loop downloads it. Object files and CMake metadata stay out: most of the bytes, none of the use.
Staging failed on undefined JIT symbols that were referenced from an object that *is* in the archive — the signature of a merge that lost objects. It had: libtool: warning duplicate member name 'X86.cpp.o' from 'libclangCodeGen.a' and … `libtool -static` de-duplicates members by name, across clang's libraries and Swift's alike, and a dropped member is a silently missing object. The archive is now merged with GNU MRI (`llvm-ar -M` + `addlib`, which copies every member, duplicates included), and the step asserts that the merged member count is at least the sum of the inputs' — so this class of bug fails where it happens instead of surfacing as a missing-library link error two steps later. The fast loop keeps staging archives unmerged and force-loading them in its link smoke, which is deliberately order-independent: it is a check, not a product.
Every run rebuilt LLVM (37 min) and Swift (34 min) from scratch even when both caches reported a hit. The restored trees were stamped with `find | touch` after configure, and that stamps files in batches milliseconds apart, so a generator input could end up newer than build.ninja: ninja re-ran CMake, CMake rewrote llvm/Config/*.h, and all 2601 edges were dirty again. The runs show it — restore "hit" at 00:45:08, `[1/2601]` at 00:45:25. * age the source checkouts to a fixed past date instead, so every file the cache restores is newer than every source — true for the object edges and for the generator edge alike; * skip the configure when the restored tree is already configured and built (checked before use, never assumed); * save only trees the build actually finished. The old `always()` save stored whatever existed when the build failed, and since a cache key can never be rewritten, every later run "hit" that partial tree and rebuilt anyway; * new keys (cache_salt 3, Swift -v2) to get past the stored ones. Add the two libraries only Swift references, which the force-loaded smoke link fails on: LLVMOrcJIT (swift's JIT in SwiftMaterializationUnit.cpp) and clangDependencyScanning (ClangImporter's module dependency scanning). Print every undefined symbol on that failure instead of the last 60 lines.
`llvm-ar` and `llvm-nm` come out of the same step as the TableGen pair, and the packaging step merges the archives with the restored `llvm-ar`. Checking only the TableGen tools let a restored tree missing it pass, and the run died three seconds into staging with "llvm-host-build/bin/llvm-ar: No such file or directory" — after an hour of building. Check all four, and refresh the llvm-build/bin mirror on the cached path too (previously the early exit skipped it).
The whole point of the toolchain build is that it only has to happen once per recipe. A workflow artifact cannot carry that: it expires, and it cannot be fetched by name, so "install the toolchain" meant "find the right run first". Publish instead, tagged by what is inside: `toolchain-<llvm sha12>-<swift ref|noswift>-ios<target>-sdk<sdk>`. Two builds of the same inputs land on the same tag, and a bundle that could not be substituted for another (no Swift frontend) can never be handed over as "the newest one" -- only hand-dispatched runs publish, so the LLVM-only half cannot take that slot. - native-toolchain.yml records the tag, the Xcode version and the SDK version in manifest.txt, and uploads the tarball to the release (clobbering a previous upload of the same tag: rebuilding identical inputs is normal). - install-bundle.sh gained `--release [tag]` (and `--release artifact` for a bundle built but not published) so a clone that never ran the workflow can still install the compiler: `make toolchain-release`. - build-ipa.yml installs from the newest toolchain-* release, takes an optional `toolchain_tag`, and warns when the bundle's Swift revision or SDK disagrees with the Xcode it is being linked into -- the frontend reads the deployment SDK's standard library modules, so that mismatch is the failure mode nobody can debug on a device. Docs updated where they claimed the bundle was a workflow artifact, and where they described the old `libtool -static` flatten (the merge is `llvm-ar -M` with an MRI script, because libtool drops members with duplicate names).
Installing the bundle is now an ordinary local action (`make toolchain-release`), and it unpacks hundreds of megabytes of archives into the checkout, where every file showed up as untracked. They are reproducible from the release the manifest names, so they are ignored; Vendor/NativeToolchain/.gitkeep stays tracked so a clone still has the directory to install into.
… sources The first app build that linked the published bundle failed on `Vendor/NativeToolchain/include/llvm/Support/DataTypes.h:19: 'llvm-c/DataTypes.h' file not found`. `llvm/include/llvm-c/` was never staged: the staging copied `llvm/`, `clang/` and `lld/`, which reads like the whole tree and is not. The reason it went unnoticed is the shape of the check: the bridge compile-check compiles against the *source* tree, where `llvm-c/` is present. Nothing compiled the bridge against the staged bundle until a host app did, 20 minutes into an app build. - Stage every top-level directory under each project's `include/`, sources and generated both, instead of naming the ones that looked right. `llvm-c` is required from now on, so leaving it behind fails the packaging step. - The staging step now compiles NativeToolchainBridge.mm against `dist/include` and `dist/include-generated` and links the smoke test with *that* object, so a header the bundle is missing fails here in seconds. - build-ipa's bundle check no longer cries wolf about the runner's Swift version: that Swift only bootstrapped the frontend's build. What has to match is the Swift inside the Darwin SDK the app installs at runtime, which this step cannot see, so it prints both and says which one governs.
`prepare-xcode.sh` builds XFORGE_NATIVE_LDFLAGS from the archives in the bundle,
escaping `$(SRCROOT)` so Xcode keeps it. It escaped the parameter expansion too:
ARCHIVES+=("\$(SRCROOT)/\${lib#"$ROOT/"}")
so the *literal* text `${lib#/Users/runner/work/XForge/XForge/}` went into the
xcconfig. Xcode reads that as a variable reference, finds nothing, and hands the
linker `$(SRCROOT)/` — the archive path with the archive missing:
ld: file cannot be mmap()ed, errno=22 path=/Users/runner/work/XForge/XForge/
Every build with the compiler linked in failed there, which is why no app build
had ever linked the bundle. `${lib#…}` is now expanded here, and the script fails
outright if the file it is about to write still contains a `${…}` or an empty
archive path, so this cannot reach a linker again.
Also `continue-on-error` on both cache saves: an entry whose key already exists
cannot be reserved, and that is not a reason to fail a build that succeeded.
Verified against a fixture bundle for all three paths (no bundle, bundle with no
archives, bundle with one archive) and against an injected `${…}`.
…hing
`EXTRA_DEFINES[@]: unbound variable`. `shell: bash` on a macOS runner is macOS's
bash 3.2, where `set -u` plus an *empty* `"${ARRAY[@]}"` is an error — and those
arrays are empty exactly when the Swift half was not built, which is every pull
request and push run. So the half of CI that runs unattended was red for a reason
that had nothing to do with the bridge, while dispatched `with_swift=true` runs
passed. The flags are accumulated in a string now.
Also stop publishing a `noswift` bundle. It is a complete deliverable for C and
Objective-C targets, but `build-ipa.yml` resolves "the newest toolchain-* release",
and a newer bundle with no frontend in it is the wrong thing to hand over. It
stays available as a workflow artifact.
The SDK the compiler reads has to come from somewhere, and until now the only somewhere was a bundle we publish or one you built on a Mac with `xtool sdk build`. That is the wrong shape for a phone: the app is the tool. `XipArchive` reads Apple's .xip — xar header and table of contents, the pbzx stream in its Content member, the odc cpio archive that decompresses to — in a single streaming pass, one block at a time. Apple's own `Compression` decodes the LZMA2 blocks, the same decoder `unxip` uses on Apple platforms, so nothing is vendored. `DarwinSDKBuilder` turns the entries it wants into the bundle xtool's builder would have produced: xtool's path list (minus prebuilt-modules, with hard links kept as hard links and symlinks as symlinks) and its swift-sdk.json, restricted to iPhoneOS.platform because XForge only ever compiles for the device. `NativeSDK` installs an imported bundle by moving it, and deletes the copy the document picker made, which for a xip is most of the free space on the device. The Toolchain screen takes .xip, .zip and folders, checks free space before it starts, streams progress through an AsyncStream into a @mainactor ObservableObject (the shape the rest of the app uses for state a long job updates), and cleans up afterwards. The tests build a xip layer by layer — odc cpio in pbzx blocks in a xar — with the TOC checksum that a sloppier offset reader would take for the data's, and check the LZMA decode against a blob produced by liblzma, so the decoder is validated against a different implementation than its own encoder. None of it needed a real 11 GB xip. Verified locally: both cpio layouts against real `cpio -H odc` and `-H newc` output, and the whole pipeline (xar offset base, block framing, path filter, bundle layout, swift-sdk.json) against a mirror of the same fixture — which is how the bundle path that dropped `Developer/` was caught before CI could.
CI compiled the app target far enough to say three things, and a re-read said a fourth: - `Payload.init` took a private `BlockStream`, so it has to be private too. - cpio sizes were parsed into `UInt32`. `odc` gives them eleven octal digits, so a member over 4 GiB fails the import — the failure is loud, but the field is 33 bits wide and reading it as 32 is simply wrong. - The test's `String(contentsOf:)` had no encoding (deprecated) and the error assertions pattern-matched an `any Error` without casting first. - `compression_decode_buffer` was handed `[UInt8](data)`, relying on the implicit array-to-pointer conversion. It is passed a real pointer now, via `withUnsafeBytes`, in the reader and in the test's encoders both.
The second CI round got past the first error and stopped here: `Self.isWanted` inside `DarwinSDKBuilder.Writer` resolves to `Writer`, which has no such member. The outer type is named in full now.
- `end - handle.offsetInFile` mixes `Int64` and `UInt64`, and the four places that recorded the file position assigned a `UInt64` to an `Int64`. Mixed comparisons are fine in Swift; mixed arithmetic is not, which is why only these showed up. - `Payload.init` was `private`, which for a member of a *nested* type is not visible to the enclosing type either — so the walk that constructs payloads could not call it. `fileprivate` on the init and on `BlockStream` puts it where the walk can reach it. - The big-endian helper assembled bytes through `withUnsafeMutableBytes` with an element type the compiler could not infer. It shifts now, which is also easier to read and gets the byte order right by construction.
The previous pass fixed the two readers and the progress cursor and missed this one, which is what the compiler stopped on next. Every `handle.offsetInFile` is now converted at the point of use, and none is assigned raw.
…e method Thirteen of them in the path tables — the app target compiles now, and this is what the test target stopped on. The rest of the file already wrote `Self.`; these two lists were the ones I wrote before deciding to.
My blanket qualification of the path constants reached the two lines that *build* those constants. Local `let`s in a function body are fine (and the three the sweep flagged are exactly that); a stored property's initializer is not, because the type is still being built. Bare names there, as before.
…e wrong The tests ran for the first time. Five of the seven new ones passed, including the xar offset test and the full cpio walk, so the reader's plumbing is right. The three failures: - **The compressed-block fixture was raw LZMA2.** A pbzx block is an **xz** stream: `pbzx.c` checks for the `\xFD7zXZ` magic and refuses anything else, and Apple's `COMPRESSION_LZMA` — which is what decodes it — rejected the raw stream this test fed it, exactly as it should. The fixture is now 787 bytes of odc cpio compressed to 256 bytes by liblzma through Python's `lzma` module, wrapped in one pbzx block in a xar, and the test walks *that*: framing, xz decode and cpio parse together, against a stream this process did not produce. The format notes in `XipArchive` said LZMA2 too; they say xz now. - `Toolchains/…/usr/lib/swift_static/iphoneos` is *wanted* — it is the static runtime directory — and the expectation table listed it as dropped. - The bundle has five files (manifest, header, runtime, its hard link, the symlink); the assertion demanded more than five.
…true Deletions, each with the evidence: - **`.github/workflows/ios-share.yml` + `builder.json`.** Its own comments tell you to keep its `js toolchain` blocks identical to `ios-build.yml` and `runner.sh`; neither file is in the repository. It drives the Flutter / React Native / Expo / KMP branches of a multi-framework builder, and it has never run here — `gh run list` for that workflow returns zero runs, so no `ios-share/*` tag has ever been pushed. This is a Swift app with an in-process toolchain. Recoverable from git if the MobAI simulator flow is wanted back. - **`.github/workflows/bridge-api-check.yml`.** A fast loop for the bridge compile-check when reaching it cost 40–90 minutes. It now costs eleven seconds inside a two-minute warm run — and the workflow kept its own copy of the LLVM configure recipe, which is a second place to update. Corrections, found while looking for the above: - `README.md`, `Docs/XCODE-ALTERNATIVE.md` and the roadmap still said the Swift frontend is not in the toolchain bundle. It is, in a `with_swift` bundle, and the app links it: what has not happened is a Swift *build* on a device, which is now what they say. - `Support/XForge.entitlements` justified both of its keys by the embedded Linux guest — a guest that no longer exists. The keys stay (a compiler's working set and its maps are the reason now), the reasoning says so. - `App/Services/InstallAssertion.swift` was the one file nothing referenced. It is not dead weight: it holds the idle timer and a background task assertion for a long job, which is exactly what an `.xip` import is (minutes of CPU, and iOS suspends an app whose screen locks). The import uses it now, so it is referenced — and the import gets guarded.
A device says "still isn't importing" and there is nothing to read back: the screen
shows one line and the log said nothing at all about the import. So:
- `XForgeLog` now records what was picked (name, size, extension, first four bytes,
whether the sandbox let us read it), where `Content` sits in the archive, what came
out of it, and — on failure — the error *with its case and values*, which is what
the screen's one line cannot carry. The Settings screen shares that file.
- The picker accepts `.data` as well as `.zip` and folders. A `.xip` has no UTI the
app can rely on, and a file that matches no allowed type is greyed out in Files —
which is indistinguishable from "the import does nothing". The reader then says
exactly what is wrong with an unrelated file.
- Which import a file gets is decided by its first bytes rather than its extension,
so a xip without the extension, or a zip named `.xip`, does the right thing, and a
file that is neither gets `notABundle` naming it instead of "no SDK in there".
- A missing `swift_static/iphoneos` no longer fails the import. It is a warning: the
headers and tbd stubs are what C, Objective-C and Objective-C++ targets need, and
refusing those because a Swift runtime is absent takes away the main reason to
import a newer SDK at all. The result carries the fact and the screen prints it.
- An archive with no `Contents/Developer` reports `noXcodeInside` rather than
"this Xcode has no iPhoneOS SDK", which is a different and better sentence.
The screen now reports what landed ("…: 20,143 files, 1.4 GB") rather than "done",
because an import that takes minutes needs an answer the user can check.
Reading a device log that has four "holding the app awake for sdk-import" lines and nothing else, two holes stand out in the path itself: - The picker's completion returned on anything that was not a success. An iCloud file the picker cannot download, or a provider that refuses to copy a 10 GB archive, produced nothing on screen and nothing in the log. It now says both. - Progress existed only on screen. "Started and then nothing" is what a stall looks like in a log someone emails you, so the last milestone (25%, 50%, …) is recorded now — four lines per import at most, and enough to tell a stall from a refusal.
Inspected the central directory of the known-good published darwin.artifactbundle.zip (456 MB, darwin-sdk-7). It has no swift/<platform>/prebuilt-modules entries for the device bundle; our filter only excluded the non-existent-at-that-level swift/prebuilt-modules and would have copied hundreds of iPhoneOS prebuilt modules from a real Xcode.xip. Those are generated per-platform/configuration intermediates, not inputs the Darwin SDK needs. Exclude prebuilt-modules by path component, and pin the exact iphoneos path in the fixture test. This cuts the import and matches the bundle XForge already knows is valid.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #15 (it contains all of that branch's commits; this PR is the eight on top).
XForge no longer boots a Linux guest. The compiler that builds an app is the LLVM/Clang/LLD — and, once its libraries are in the toolchain artifact, Swift's frontend — linked into the process through the existing C bridge.
What replaces what
xtoolinside Alpine, driven through a guest shellNativeBuildExecutor: each translation unit through the bridge,ld64.lldlink,.appassembled,.ipapackagedapk info -e …,swift --version,xtool --versionin the guestNativeToolchainCapabilities— what is actually linked, reported honestlyswift sdk installinside the guestdarwin.artifactbundle, downloaded or imported into the app container (NativeSDK)zsignin the guest, with a password file on diskAppBundleSigner,IPASigningJob); the.p12goes in throughSecPKCS12Importand is used in memoryGuestProjectFilesover the guest filesystemProjectFilesover<Documents>/projects, same relative-path disciplinetarin the guest, copied outProjectExporter, a ZIPFoundation archiveRemoved
App/EmbeddedVM/**,EmbeddedLinux/**,Tools/engine-smoke/**, the Terminal tab,ToolchainManager,SystemComponents,GuestNetwork,HostDNS,DownloadManager,IPAConfigureSignService,SDKInstaller,EmbeddedLinuxExecutor,DirectorySize, theVendor/ish-arm64submodule,build-rootfs.yml,ios-build.yml, and the terminal tests.build-ipa.ymldrops from 22 steps to 14 and no longer downloads a 1.6 GB rootfs.Deliberate capability losses, not oversights
git.actoolis macOS-only.Verification
CI with
run_tests=trueon every push: 30 unit tests / 0 failures, Release archive, built-app verification, IPA packaged and verified. Twelve of those tests are new (NativeBuildTests) and cover the plan factory's acceptance and refusals and the exactswift-frontend/ld64.lldargument lists — the half a simulator can check, and the half that decides whether a build works.Not verified: anything at runtime. CI proves the pipeline compiles; it cannot prove that in-process clang/LLD produce a loadable app, that the SDK bundle installs, or that XKit signing succeeds on a device.
Numbers
Documentation
README, CHANGELOG (0.6.0, opening with the breaking change),
Docs/DESIGN.md,Docs/IPA-BUILD.md,Docs/NATIVE-TOOLCHAIN.mdandDocs/XCODE-ALTERNATIVE.mdrewritten;Docs/ISH-ARM64-INTEGRATION.mddeleted with the integration it described.