Skip to content

emit-lib.md

Latest commit

 

History

History
1473 lines (1264 loc) · 78.5 KB

emit-lib.md

File metadata and controls

1473 lines (1264 loc) · 78.5 KB
Raw

Embedding Aether as a Shared Library (--emit=lib)

aetherc --emit=lib (or ae build --emit=lib) compiles a .ae file into a dynamic library (.so on Linux, .dylib on macOS) with ABI-stable entry points that any FFI-capable language can call: C, Java (Panama), Python (ctypes or SWIG), Go (cgo), Ruby, Rust (extern "C"), and so on.

This document covers the v1 contract. For the broader design rationale, see aether-embedded-in-host-applications.md.

Quick example

// config.ae
import std.map

build_config(env: string, port: int) {
    m = map_new()
    map_put(m, "env", env)
    map_put(m, "port", port)
    return m
}

Build:

ae build --emit=lib config.ae
# produces libconfig.so (or libconfig.dylib on macOS)

Consume from C:

#include "aether_config.h"
#include <dlfcn.h>

typedef AetherValue* (*build_fn)(const char*, int32_t);

int main(void) {
    void* h = dlopen("./libconfig.so", RTLD_NOW);
    build_fn build = dlsym(h, "aether_build_config");
    AetherValue* cfg = build("prod", 8080);

    const char* env  = aether_config_get_string(cfg, "env");
    int32_t     port = aether_config_get_int(cfg, "port", 0);
    printf("%s on %d\n", env, port);   // prod on 8080

    aether_config_free(cfg);
    dlclose(h);
}

The flag matrix

Flag Effect
--emit=exe (default) Current behaviour, produces an executable.
--emit=lib No C main() in the output (a program's main() becomes aether_main / aether_main_exit); every top-level Aether function gets an aether_<name>() C-ABI alias; built with -fPIC -shared.
--emit=both Produces both an executable AND a shared library from one source. ae build --emit=both foo.ae -o foo writes foo (the exe) and foo.dylib / foo.so (the lib) side by side. With no -o, defaults are <base> (exe) and lib<base>.<ext> (lib). Internally dispatches cmd_build twice (once per emit mode); the lib pass appends the platform lib extension to the user's -o so the second pass doesn't overwrite the first.
--emit=csrc Emits the portable generated C source (foo.c), a catalog header (foo.h, the aether_<name>() prototypes), and a machine-readable catalog (foo.catalog.json), and stops. No gcc, no native artifact. Same catalog codegen as --emit=lib; the difference is you get source plus a describable ABI surface, not a host .so.
--emit=obj Compiles straight to a single relocatable object file (foo.o), no link step. Same catalog codegen as --emit=lib, so the object exports both the bare Aether names and the aether_<name>() C-ABI aliases. For dropping .ae into an existing C link line.

--emit=obj: one object file for an existing C build

--emit=csrc hands you generated C that your build then has to know how to compile. --emit=obj does that step for you and stops at the object:

ae build --emit=obj lib.ae -o lib.o
cc main.c lib.o $(ae cflags --libs) -o app

This is what makes an incremental make correct without checking generated C into the tree. A checked-in *_ae.c is a stale-artifact trap: edit the .ae, restore the .c to keep a diff clean, and the next plain make links the old code with nothing to warn you. With an object rule the .ae is the only source, and the timestamp the build compares is the one that matters:

%.o: %.ae
	ae build --emit=obj $< -o $@

app: main.o lib.o
	$(CC) $^ $(shell ae cflags --libs) -o $@

The C compiler used for the object is $AE_CC, else $CC, else gcc when present, else cc, the same resolution order as the other build paths, so a cross build picks up the same toolchain. Because the object carries the aether_<name>() aliases, a C caller can #include a header generated by --emit=csrc (or declare the prototypes itself) and call in directly. Note that a -> string export returns a string the caller owns, as a const void* handle: see Strings an export returns.

--emit=csrc: distribute source, not N native libs

A single native lib can't be universal across OSes, ELF vs Mach-O vs PE/COFF are different loader formats. But the pre-native artifact can be: the generated C is portable, so --emit=csrc moves the "which platform" decision from distribution time (ship 6 natives) to the consumer's build:

ae build --emit=csrc mylib.ae -o mylib
# writes mylib.c (portable C) + mylib.h (aether_<name> prototypes)
#      + mylib.catalog.json (the describable ABI surface)

# compile it wherever, against the runtime:
cc -fPIC -shared mylib.c $(ae cflags) -o libmylib.so   # this host
# or feed mylib.c to WASM, or static-link it into a compiled host, or
# publish {mylib.c, mylib.h, mylib.catalog.json} as a content-addressable
# source package.

The .h is a normal C header (include guard + extern "C"); a consumer #includes it and calls aether_add(...) / aether_greet(...) directly. This is the enabling primitive for compile-on-install bindings (Python C-extension / Ruby-gem style) and a "source registry" story where every binding compiles the same hash-pinned source in its own toolchain. Deliberate follow-ups: single-file amalgamation and expressing the required runtime .c set for a fully standalone external compile.

The JSON catalog (<name>.catalog.json)

The same aether_lib_meta() symbol catalog the .c carries in .rodata (see the reflection section below) is also written as a JSON document, so a binding generator in any language can consume it without dlopening a native artifact. It is a faithful serialization of the C struct, driven by the identical codegen tables, so the two can never drift; it is deterministic (source order, stable version strings) and human-diffable, which makes the source artifact content-addressable. Fields:

Field Meaning
schema_version "1.0" functions only, "1.1" adds closures, "1.2" adds constants, "1.3" adds structs and source signatures (the highest feature level present)
aether_version Compiler version stamp
primary_source The .ae the artifact was built from (mirrors the path passed to the compiler: pass a relative path for a reproducible, machine-independent value)
capabilities The --with grants this artifact was built with (e.g. ["fs","net"]), empty when capability-empty. Because the emitted C only contains code paths for granted capabilities, this is the syscall surface a consumer can inspect before compiling the source
functions[] aether_name, c_symbol (the dlsym/link name), signature, source_file, source_line; at schema 1.3 also source_signature, the signature as Aether source ("(m: *Model, p: Vec3) -> void")
closures[] Closure surface reachable from exports: name, role (builder / param / literal), enclosing_export, signature, captures[] (name + type), source_file, source_line
constants[] Exported scalar/string consts: name, type, value
structs[] Schema 1.3: the structs the exports use, name, kind ("", "extern", "extern_packed"), fields[] (name, type as Aether source, flags), source_file, source_line
host_abi Schema 1.11: the host ABI level of libaether.h the library was built at (abi-policy.md)
runtime_abi Schema 1.11: the runtime ABI it was compiled for (AETHER_RUNTIME_ABI_ID, such as "0.856.0/abi1")
requires[] Schema 1.11: what its importer must do (outcome_check, string_release, shared_runtime, runtime_abi, actors); an importer that does not know one refuses the library 
{
  "schema_version": "1.0",
  "aether_version": "0.0.0-dev",
  "primary_source": "mylib.ae",
  "capabilities": [],
  "functions": [
    { "aether_name": "add", "c_symbol": "aether_add", "signature": "(int, int) -> int", "source_file": "mylib.ae", "source_line": 4 }
  ],
  "closures": [],
  "constants": []
}

Naming

Every exported Aether function becomes aether_<name> in the library:

Aether declaration C symbol
sum(a: int, b: int) { ... } aether_sum
greet(name: string) { ... } aether_greet
build_config(env, port) aether_build_config

The internal sum, greet, etc. symbols are also present in the library with external linkage. Callers should use the aether_<name> entry points for ABI stability, the un-prefixed names are an implementation detail and may be hidden in a future version. See Symbol visibility matrix below for the full picture across emit modes.

A function written as several clauses (fact(0) -> ..., fact(n: int) -> ...) is one export: one aether_fact, one catalog record and one header prototype, with the set's signature: a parameter for each position, of the type the clauses give it (a literal pattern such as fact(0) has its literal's type), and the set's return type (#2665). A @c_callback("sym") function is listed in the catalog under sym, and its aether_<name> stub calls sym (#2666); a clause set annotated @c_callback takes the symbol its first annotated clause binds.

A program's main(): aether_main and aether_main_exit

A library built from a .ae that defines main() keeps it. Under --emit=lib (and --emit=staticlib, --emit=obj, --emit=csrc, which share its codegen) the program's entry point is exported as a pair:

int  aether_main(int argc, char** argv);   /* run main(), return its result */
void aether_main_exit(void);               /* stop what main() started      */

aether_main runs the same prologue an executable's main runs (argument setup, the opt-in Capsicum sandbox, and, when the program has actors or imports a library that does, the scheduler started with the main thread marked as not a scheduler thread), then main()'s body, and returns. Its result is main()'s int (0 when main() returns nothing). Actors the body spawned keep running.

aether_main_exit runs the executable's epilogue: it waits for the actors to go quiet, stops and joins the scheduler threads, and prints the message-pool report where the executable would.

An executable is that pair run back to back, from the same emitter:

int main(int argc, char** argv) {      /* what --emit=exe amounts to */
    int rc = aether_main(argc, argv);
    aether_main_exit();
    return rc;
}

with one ordering difference: main()'s defers (an explicit defer, and the automatic frees of its local strings and sequences) belong to aether_main's stack frame, so in a library they run when aether_main returns, before the actors are drained, where an executable runs them after. Don't defer the release of something main() hands to its actors.

When a host calls each. aether_main when it wants the program to start, aether_main_exit when it is done with it. On Android, where the app is the library:

/* AetherActivity's JNI side */
JNIEXPORT void JNICALL Java_..._onCreate(JNIEnv* env, jobject self) {
    aether_main(0, NULL);     /* builds the UI, returns into Android's loop */
}
JNIEXPORT void JNICALL Java_..._onDestroy(JNIEnv* env, jobject self) {
    aether_main_exit();       /* drain the program's actors */
}

A desktop host does the same around its own loop, or calls both back to back to behave like the executable.

Linking it into an executable needs no C main(). The generated C (what aetherc --emit=lib writes, and --emit=csrc), and the --emit=obj and --emit=staticlib output, also define that back-to-back pair as a weak main():

__attribute__((weak)) int main(int argc, char** argv) {
    int rc = aether_main(argc, argv);
    aether_main_exit();
    return rc;
}

So any build tool that links the object into a program (aeb's c.program, a Makefile, cc app.o $(ae cflags --libs)) gets the program's entry point, and its exit code, without a hand-written C file. A host that has its own main() keeps it: a strong definition beats a weak one. A shared library that ae links itself (--emit=lib, native or --target=) leaves it out (it exports no entry point), as does wasm (emscripten runs a module's main on load, which would turn a library into a program) and a compiler without weak definitions. -DAETHER_NO_LIB_MAIN leaves it out anywhere else.

On Windows (MinGW) the entry is an archive member instead. A COFF weak definition is a weak external, and GNU ld (2.46 at least) still searched the archives for main: it pulled libmingw32.a's crtexewin.o, a strong main() that calls WinMain, and the link failed with undefined reference to 'WinMain'. So the object carries no main() there, and the same three lines are the one member of libaether_main.a, installed beside libaether.a. ae cflags --libs names it first (-L<lib> -laether_main -laether ...), and an archive member is linked only when the link still needs the symbol it defines, so:

  • cc app.o $(ae cflags --libs) links the program's own entry, as elsewhere;
  • a host with its own main() keeps it (the member is never pulled), with nothing to change: the same command line works on every platform;
  • a GUI host that enters through WinMain names the C runtime's entry first, cc app.o host.c -lmingw32 $(ae cflags --libs), so that main() is resolved (to the one that calls WinMain) before libaether_main.a is searched. Without that it starts in the program's main().

A link line written by hand rather than from ae cflags --libs adds -laether_main before -laether itself. The shared runtime (aether.dll) and every DLL ae links export no main.

Calling them more than once.

  • aether_main while the program is already running (no aether_main_exit since) is rejected: it prints aether_main: already running; call aether_main_exit() first on stderr, runs nothing, and returns -1.
  • aether_main_exit with nothing running is a no-op, so calling it twice is safe, as is calling it when aether_main never ran.
  • After aether_main_exit, aether_main may run the program again on a fresh scheduler: an Android activity destroyed and recreated in the same process. The previous run's scheduler tables are freed; the actor objects it spawned are not (an executable leaves those to process exit too).
  • Call both from one thread, the host's main or UI thread. They are not synchronized against each other.
  • Stop the host's own threads that send to the program's actors before aether_main_exit: its drain waits for the actors to go idle. A send from such a thread while the program is not running is dropped, counted by scheduler_stopped_sends() and reported once (#2912).

Finding them. They are plain exported symbols: dlsym(h, "aether_main"), a weak reference, or the --emit=csrc header, which declares both when the program has a main(). They are not catalog functions (aether_lib_meta lists the library's callable functions, and an Aether program importing the library must not bind a main). A library without main() has neither. A wasm --emit=lib exports the catalog by default; add --export=main --export=main_exit (with the rest of the list you want) to export the pair. --emit=both keeps the executable's C main and adds no pair.

Reserved names. Every top-level function exports as aether_<name>, so a function called main_exit would export as aether_main_exit. In a program that defines main(), --emit=lib rejects it with a compile error naming both. Rename it, or end its name with _ to keep it private. Without a main(), or in an executable, main_exit is an ordinary name.

Type mapping

Parameter and return types on exported functions are mapped to a fixed public ABI:

Aether type Public C type Notes
int int32_t Fixed width for cross-language clarity
long int64_t The Aether source keyword for 64-bit integers is long; the lexer token TOKEN_INT64 backs both long and int64_t internally
float double IEEE 754 binary64, as everywhere in Aether
f32 float IEEE 754 binary32
bool int32_t 0 = false, 1 = true
string (parameter) const char* Borrowed for the call
string (result) const void* A string the caller owns; see below
ptr / list / map AetherValue* Opaque handle; walk with aether_config_* accessors
*Struct AetherValue* Opaque to a C host; an Aether importer gets the typed pointer back (the catalog's source signature) and reaches the fields
Struct (by value) the struct, by name Its layout is in the catalog's struct records, which an Aether importer re-declares; an --emit=csrc header leaves these prototypes out and says why (#2297)
fn(A, B) -> R void* A C function pointer
fn (a closure) _AeClosure Its function and captured environment, by value. Both ways: a library calls the closure a program passes, and a program calls one the library returns, captures included. Only an Aether importer declares _AeClosure and its calling convention, so an --emit=csrc header leaves these prototypes out, as for a struct by value (#2297)

Strings an export returns

A -> string export hands its caller one reference to a counted string, as any Aether function does (R3 in the ownership model). The caller reads it with aether_string_data() and aether_string_length() and gives it back to the library's runtime, which made it, through the library's release function, aether_<lib>_string_release:

const void* s = aether_greet("world");
printf("%.*s\n", (int)aether_string_length(s), aether_string_data(s));
aether_greet_string_release(s);

<lib> is the library's name: its package (--package), else its entry file's stem (greet.ae is greet), so every target built from one source exports the same function; a --namespace library is named for its library file (libgreet.so is greet). Every character outside [A-Za-z0-9] is written as _. A host never derives it: the catalog names it (string_release, schema 1.9, and "string_release" in the JSON), and ae lib-info prints it as String release:. An --emit=csrc header declares it. The bindings ae build --namespace generates read and release the string for you, and a binary import copies it into the program's runtime and releases the library's reference.

Functions whose parameters or returns use types outside this table (tuples, optionals, actor refs, enums) compile but don't get an aether_<name> alias. aetherc prints a warning naming the skipped function. You can still use those types internally; they just can't cross the FFI boundary yet.

Two private-helper conventions also opt out of the aether_<name> surface:

  • Trailing-underscore: a function whose name ends in _ (e.g. record_start_, helper_, parse_line_) is treated as file-local. It's emitted with C static storage and gets no aether_* alias. This lets two .ae files in the same project declare their own record_start_ without colliding at link time and without leaking either into the public ABI. The convention is enforced by codegen, so adding the _ is enough, no annotation.
  • Tuple-returning helpers: helper(n: int) -> { return n, n+1 } returns a _tuple_int_int struct that doesn't fit the public ABI table. The function itself is fine; the alias is skipped. Wrap with a single-value-returning function if it needs to be exposed across the library boundary.

The AetherValue* accessor API

Composite Aether values, maps, lists, generic ptrs, come back to the host as AetherValue* handles. The host walks them with the functions declared in runtime/aether_config.h:

// Map accessors
const char*  aether_config_get_string(AetherValue* root, const char* key);
int32_t      aether_config_get_int   (AetherValue* root, const char* key, int32_t default_value);
int64_t      aether_config_get_int64 (AetherValue* root, const char* key, int64_t default_value);
float        aether_config_get_float (AetherValue* root, const char* key, float  default_value);
int32_t      aether_config_get_bool  (AetherValue* root, const char* key, int32_t default_value);
AetherValue* aether_config_get_map   (AetherValue* root, const char* key);
AetherValue* aether_config_get_list  (AetherValue* root, const char* key);
int32_t      aether_config_has       (AetherValue* root, const char* key);

// List accessors
int32_t      aether_config_list_size       (AetherValue* list);
AetherValue* aether_config_list_get        (AetherValue* list, int32_t index);
const char*  aether_config_list_get_string (AetherValue* list, int32_t index);
int32_t      aether_config_list_get_int    (AetherValue* list, int32_t index, int32_t default_value);
int64_t      aether_config_list_get_int64  (AetherValue* list, int32_t index, int64_t default_value);
float        aether_config_list_get_float  (AetherValue* list, int32_t index, float   default_value);
int32_t      aether_config_list_get_bool   (AetherValue* list, int32_t index, int32_t default_value);

// Lifetime
void aether_config_free(AetherValue* root);

Ownership

Return value Ownership
Root AetherValue* from an aether_<name>() call Owned, call aether_config_free(root)
Nested map/list handle from _get_map / _get_list / _list_get Borrowed, valid until the root is freed; do NOT free individually
const char* from _get_string / _list_get_string Borrowed, points into the tree; copy if you need it past aether_config_free

Behaviour of missing keys / out-of-range indices

  • Typed getters return the default_value the caller supplied.
  • _get_string / _get_map / _get_list / _list_get_string return NULL.
  • _list_size(NULL) returns 0; _has(NULL, k) returns 0.

Type contract

Aether maps and lists are untyped internally, values are stored as opaque void* with no runtime tag. The accessors reinterpret the stored value as the requested type; they do not verify it. That makes the script-to-host interface a straight FFI contract: document what the script writes at each key, and the host must read it back with the matching accessor. If the script stored an int at "port" and the host asks for a string, the host gets garbage, treat this exactly the way you would treat any other cross-language value exchange.

Capability-empty default

--emit=lib rejects .ae files that import:

  • std.net, std.http and its children, std.tcp, std.udp networking
  • std.fs, std.file, std.dir filesystem
  • std.os, std.dl process, environment, shell, and dynamic loading
  • contrib.host.* foreign-language runtimes

The rationale: a library embedded in a host process shouldn't have ambient network, filesystem, or process-spawning access. Those are capabilities the host grants; the script should only compute and return data. The host mediates I/O through whatever its own runtime provides.

std.map, std.list, std.string, std.json, std.math, and the other capability-free standard modules are allowed.

It also rejects an extern declared in the program's own code (the entry file, an @extern("…") binding, or a local module it imports). An extern names a C function directly, so extern system(cmd: string) -> int would reach what import std.os is gated to stop. Externs declared by other std.* modules are not the program's and are unaffected. Foreign-language host modules sit behind the contrib.host.* gate above. Opt in with --with=extern.

Opting in: --with=<capabilities>

Projects that are the host, code that compiles .ae and handwritten C into one binary, rather than embedding Aether as an untrusted user script, opt into specific capability categories:

ae build --emit=lib --with=fs           file.ae   # std.fs, std.file, std.dir
ae build --emit=lib --with=net          file.ae   # std.net/http/tcp/udp families
ae build --emit=lib --with=os           file.ae   # std.os, std.dl
ae build --emit=lib --with=fs,os        file.ae   # multiple, comma-separated
ae build --emit=lib --with=extern       file.ae   # own C externs, contrib.host.*
ae build --emit=lib --with=first-party  file.ae   # alias for fs,net,os,extern
ae build --emit=lib --with=all          file.ae   # alias for fs,net,os,extern

The gate stays default-deny: a build without --with=fs still rejects import std.fs. Unknown capability names are a hard error (typos shouldn't silently leave a gate closed). The categories mirror the banned-import groupings above, three buckets, chosen coarsely enough that opting in is an auditable event in a project's build invocation.

--with=first-party and --with=all both expand to fs,net,os,extern. The two names are equivalent; pick whichever expresses intent better in your project. first-party reads as "this Aether code is trusted-as-first-party, give it everything"; all reads as a literal shorthand for the full set. Using either one is appropriate for tools like build systems and SDK generators where the .ae files ship with the toolchain itself; it is not appropriate for plugin / user- script scenarios, see "When NOT to use this" below.

When to use this. Systems code where the .ae files are first-party and version-controlled the same way as the .c files. The Aether-side file_open_raw is no different from the C side calling fopen there's no privilege boundary to police.

When NOT to use this. Anywhere the library could run untrusted Aether (user scripts, a plugin loader, a DSL evaluator). Leave the default in place; the host mediates I/O on the script's behalf.

Symbol visibility matrix

What gets static storage class versus external linkage in the generated .c file, by function origin × emit mode. This determines which symbols collide when multiple .c files compiled from different .ae sources are linked into one binary.

Function origin --emit=exe --emit=lib
Local (defined in this .ae) external aether_<name> alias exported; its own name hidden (#2937)
Local with trailing _ (helper_) static static (no aether_<name> alias)
Local with tuple return type external external (no aether_<name> alias)
Imported Aether wrapper static static (no aether_<name> alias)
extern declaration (any module) declaration only, refers to external symbol from libaether.a or another TU
Stdlib C externs (std/*/aether_*.c) linked from libaether.a; no per-TU duplication

Two consequences worth highlighting:

Imported Aether wrappers are private to each TU. When module A and module B both import std.string, both generated .c files contain static AETHER_MAYBE_UNUSED const char* string_copy(...) a private copy each. The attribute is there because a TU that calls part of a module leaves the rest of the private copies uncalled, and a consumer building the generated .c with -Wall -Werror would otherwise fail on -Wunused-function in code it did not write. Linking the two .o files together produces no duplicate-symbol errors, even on linkers that don't support -Wl,--allow-multiple-definition (notably macOS ld64). This is deliberate; see compiler/aether_module.c where cloned imported functions get clone->is_imported = 1, and compiler/codegen/codegen_func.c where the static AETHER_MAYBE_UNUSED prefix is emitted for functions guarded by func->is_imported || trailing_underscore_private.

A library calls its own functions, not a host's. A function with an aether_<name> alias is emitted with hidden visibility under its own name (AETHER_LIB_LOCAL): it still links across the library's translation units, but it is not exported, so on ELF, where a call through an exported name binds to the first definition in the process, a host or another library defining the same name cannot take over the library's calls (#2937). A function without an alias (a builder, a tuple returner) and a @c_callback keep their exported names.

Locally-defined functions retain external linkage. If two .ae files both define a function with the same name (e.g. both define helper), they DO collide at link time. --emit=lib adds an aether_<name> alias on top, but the un-aliased local symbol is still external. This is the symmetric opposite of the imported case and is intentional, local functions are the unit of inter-TU linking; the user program structures around their names.

Implication for downstream tools. Multi-TU binaries that import the same SDK module across many .ae files do not need -Wl,--allow-multiple-definition. The static-marking on imported wrappers handles deduplication at the source level. Tools building this shape can drop the link flag and gain macOS compatibility for free, ld64 (Apple's linker) silently rejects --allow-multiple-definition, and the static-marking removes the need for it.

Using SWIG to generate Java/Python/Ruby/Go bindings

The repo ships a minimal SWIG interface at runtime/aether_config.i. It wraps aether_config.h and marks aether_config_free as the destructor so target-language GC integrates cleanly.

# Python
swig -python -o aether_config_wrap.c runtime/aether_config.i
gcc -fPIC -shared $(python3-config --includes) aether_config_wrap.c \
    libconfig.so $(python3-config --ldflags) -o _aether_config.so

# Java
swig -java -package com.example.aether runtime/aether_config.i

# Ruby, Go, C#, ... same pattern with -ruby, -go, -csharp.

The generated bindings wrap AetherValue* as a proxy class per target language, Python gets AetherValue, Java gets a class AetherValue, etc. Users see idiomatic API calls; the opaque pointer is hidden.

A -> string export returns a handle the caller gives back (Strings an export returns), so wrap it with a helper that copies the bytes out, calls the release, and returns the copy as %newobject. See tests/integration/emit_lib_swig/ for a worked Python round-trip, that helper included.

Per-call resource caps

Hosts loading untrusted Aether scripts bound the script's resource use via two TLS-backed knobs in include/libaether.h:

#include <libaether.h>

aether_set_memory_cap(64 * 1024 * 1024);   // 64 MiB ceiling
aether_set_call_deadline(50);              // 50 ms wall-clock

call_into_aether();                        // returns inside both bounds

if (aether_last_outcome() == AETHER_DEADLINE) {
    /* The deadline fired mid-call: the call ended at a safe point
     * and returned its zero value. Any partial state is the
     * host's to interpret. */
}

Memory cap. Process-wide. The cap-aware allocators (std/string, std/collections, runtime/memory/{arena,pool}) check + account against a single _Atomic uint64_t counter. The counter tracks current usage, not high-water-mark, long-running guests that allocate-and-free in a loop don't trip the cap on cumulative churn. Allocations past the cap return NULL through the existing error-string convention; stdlib functions surface "out of memory: cap exceeded" up the call chain rather than aborting. Pass 0 to disable.

Allocation failure ends the call (#2956). Inside an export, an allocation that fails, past the cap or with the allocator out of memory, still fails cleanly where it happens (the std operation returns its error, its own memory freed), and the call ends at its next safe point, or when it returns, with AETHER_MEMORY_CAP or AETHER_OUT_OF_MEMORY and the zero value: its defers run, and a try in the script does not catch it. The call does not unwind from inside the allocator, which would skip the C code between it and the safe point. An allocation the code cannot go on without ends the call at once with AETHER_OUT_OF_MEMORY: a message's payload copy, make([]T, n), and every allocation the generated code makes for itself (a captured variable's cell, ref, lazy, a closure's environment, a closure boxed into a ptr, heap.new, #3377). Outside an export a send stops the process as before, and the others are a panic naming what ran out (they used to dereference NULL). A malloc the program declares and calls itself still returns NULL, as C's does, and ends the call when it next reaches a safe point or returns. Neither the outcome nor the unwind allocates: the reasons are static strings and the journal is drained in place. One failure still stops the process, inside an export or not: a message the scheduler has no memory to queue behind an actor's earlier messages, since it has no place that keeps their order.

Wall-clock deadline. Per-thread monotonic (CLOCK_MONOTONIC). The codegen emits a safe point, if (aether_caps_armed && aether_exec_poll()) { __aether_abort_call(); break; }, at every for / while loop head under --emit=lib not under --emit=exe (zero overhead on non-sandboxed builds). aether_caps_armed is a global that goes nonzero once any thread arms a deadline, trips one or cancels a call, and stays so; until then a loop head costs one load, with no call and no thread-local read. It is an atomic read with a relaxed load (a plain load on x86 and ARM), so a loop that started before another thread's cancel sees it. (The call reads thread-locals, and on macOS each of those is itself a call: in a ported interpreter that was a fifth of the run time.) Pass 0 to disable; pass a fresh ms value to clear the sticky flag and re-arm.

Interruptions (#2956). Inside an export, the first safe point past the deadline ends the call: the interruption unwinds like a panic, so every frame on the way runs its defers and releases its owned locals, and the export returns its zero value with the outcome AETHER_DEADLINE. A try in the script runs its own cleanup and passes the interruption on instead of running its handler: a script cannot swallow its deadline. Another thread can end a call the same way: aether_exec_token() on the calling thread names the running call (or, outside one, its next), and aether_exec_cancel(token) from any thread ends it at its next safe point with AETHER_CANCELLED. A call nested inside the cancelled one ends first; the cancelled one at its own next safe point, once the host code between them has returned. A cancel of a call that already ended does nothing. Outside any export (a library's actor, say) a loop head breaks at a passed deadline, as it did before. An actor stepped on the call's thread inside it (a send's inline step, a cooperative ask's poll) is outside it too: the call's deadline and cancel do not reach its loops, a try in it catches its own panics, and the call is cut at its own next safe point once the step returns (#3392). The drops its step runs after its handler (of the message a panicking handler ran on, and of the messages waiting for an actor the step ended) are outside the call as well (#3416). Code the runtime runs inside the call in a frame of its own (the drop of an ending actor's messages, when the call ends an actor there and then) is cut by the interruption, and the interruption goes on to the call: it ends at the call's next safe point, or when it returns, and a later call's try is not touched by it (#3404). Such a frame tells an interruption from a panic by what its own code raised: one already on its way when the code began (a defer the interruption's unwind runs ends an actor, whose drops run there) goes on as it was, and an ordinary panic in a drop is the drop's, reported as any is, so the call still ends with the interruption's outcome and a try on the way passes it on (#3415).

The safe points are loop heads, and json.parse, which checks every 4096 values it reads and then fails with "stopped: the call's deadline passed or it was cancelled" (the export then ends with the outcome). Nothing else stops a call, so these are bounded only by their own length:

  • straight-line code, however long;
  • recursion without a loop: a safe point at every function's entry would cut it, but it made fib(30) through its stub 1.18 ms against 0.69 ms, so there is none;
  • C-backed std operations other than the JSON parse (string split, join and replace, json.stringify, a regex match, compression), each bounded by its input;
  • extern C functions, and host services the script calls: they run to their return, and the next safe point after it ends the call.

The trip is also observable via aether_deadline_tripped(). Retrying a tripped deadline is the host's policy: re-arm + call again with a more generous budget, or surface Deadline to the user.

Execution accounts (#2954)

The cap and the deadline above belong to the process and to one call. A host that runs many scripts on one thread, each yielding to it and resumed later (a game's scripts, an editor's plugins), needs a budget that belongs to the script: an execution account.

AetherExecLimits lim = { 256 * 1024, 5000, AETHER_DEADLINE_ACTIVE, NULL };
AetherExecAccount* script = aether_exec_account_new(&lim);

AetherExecAccount* was;
aether_exec_use(script, &was);     // this thread's calls charge `script`
int r = aether_on_frame(dt);
if (aether_last_outcome() == AETHER_MEMORY_CAP) { /* past its 256 KiB */ }
aether_exec_use(was, NULL);        // back to the host's own

if (aether_exec_account_tripped(script)) drop(script);
aether_exec_account_free(script);
  • Memory. While an account is in use on a thread, every cap-aware allocation made there (strings, collections, JSON, the std buffers) is charged to it and to its parent chain. One that would take any of them past its memory_bytes is refused as the process cap refuses one: the call ends at its next safe point with AETHER_MEMORY_CAP, and the account's aether_exec_account_tripped has AETHER_ACCOUNT_TRIPPED_MEMORY. The process cap stays the outer ceiling.
  • A block stays the account's that allocated it until it is freed, wherever that happens: in a call under another account, or in the host outside any call. So a string a script hands its host keeps counting against the script until the host releases it, and two scripts on one thread never charge each other.
  • Deadlines. A call under an account has the earliest deadline on the account's chain, or the call deadline (aether_set_call_deadline) if that is earlier. AETHER_DEADLINE_ELAPSED (the default) counts wall time from aether_exec_account_new, so the time a script spends suspended counts; AETHER_DEADLINE_ACTIVE counts only the time spent in calls under the account, once however calls nest: a call a host service makes inside another, under the same account or a descendant of it, adds no time of its own to it (#3409). Past it the call ends with AETHER_DEADLINE, and the account keeps AETHER_ACCOUNT_TRIPPED_DEADLINE.
  • A parent only narrows. An account made with a parent charges what it holds to the parent too, and its deadline is never later than the parent's: a host gives each plugin an account and each of the plugin's scripts a child of it. A parent that is no live account of these accounts (one the host freed, or one another runtime copy made without sharing them) makes aether_exec_account_new return NULL (#3412): the child is refused, never made wider than the parent meant.
  • Nesting. An execution puts back the account that was in use when it began, whatever the code it ran set: a host service that runs another script under its own account, inside a call, leaves the outer call's account as it was.
  • Threads. An account is in use on one thread at a time: aether_exec_use returns -1 for one in use on another thread, and it may move to another thread once let go there. Allocations on other threads (an actor's handler) are not charged to it, nor those of an actor stepped on this thread while it is in use (a send's inline step, a cooperative ask's poll), the drops its step runs after its handler included (#3416): a step runs in its own context (actor-concurrency.md, "The delivery contract"). The refusal on another thread is a threaded build's: a cooperative build (AETHER_NO_THREADING) runs one thread per program by contract, so there is no other thread to refuse.
  • Ownership. The account is the host's from aether_exec_account_new to aether_exec_account_free; the runtime keeps nothing of limits past the call, writes previous and keeps nothing of it. Freeing an account detaches its blocks, which then count only against the process cap: its ancestors are credited with what they were charged for them (#3408). Its children go under its own parent, which their blocks were charged to all along, and keep narrowing under it. Nothing in the runtime names a freed account (#3407): in use on the calling thread, it is in use no more; in use on another thread, nothing more is charged to it there, and that thread lets go of it at its next allocation, call or aether_exec_use; an execution that began under it puts no account back when it returns. A freed account is the host's no more: passing it again is not supported.
  • An Aether host has the same in std.host: host.account_new(bytes, ms, host.DEADLINE_ELAPSED, parent), host.account_use(a), host.account_in_use(), host.account_used(a), host.account_tripped(a) and host.account_free(a).

Runtime copies. A library built without --shared-runtime has a runtime of its own, and the accounts are still one per process: every copy works through the accounts of the first loaded image that has them, found once, at its first account call or execution (the rule the sandbox's levels follow). There is one lock, one table of blocks and one account in use per thread, so an account is charged whichever copy allocates, and is refused on another thread whichever copy is asked.

  • The copy that owns the accounts must stay loaded while another runs: unloading it first is not supported. Any other copy may be unloaded: it leaves the accounts as it goes (#3411), and is never written into after.
  • Each copy marks its image with a record the loader maps with it (a PE section, an ELF note, a Mach-O section), and the copies read the records in load order, the program first. So a program's own copy is found though its executable exports nothing (on Windows, and on Linux without -rdynamic, as a C host linked with ae cflags --libs is), whenever the libraries are loaded (#3445). Records are read on Windows, macOS, Linux and FreeBSD, NetBSD and OpenBSD; elsewhere (Cygwin, illumos, DragonFly) copies look each other up by name in the global scope, and where two do not find each other, a copy refuses (-1) to put in use an account the other made, rather than charge one it does not guard. Copies in a namespace of their own (glibc's dlmopen) find none outside it.
  • The accounts carry a version and the sizes of what the copies pass each other. A copy that finds accounts of another layout (another Aether version) does not share them: it says so once on stderr, naming --shared-runtime; its aether_exec_account_new returns NULL and its aether_exec_use -1 for an account; its allocations are charged to none; and it reads nothing of the other's but the version and sizes.

What it costs. While no account exists nothing is charged, and a cap-aware allocation still tests one word before it calls malloc (the word also holds the process cap and the test hook); an execution's entry adds one load, and an actor step one at each end (the delivery contract's item 5, in actor-concurrency.md).

While an account exists anywhere in the process (#3410):

  • A thread with no account in use takes no lock. Its cap-aware allocation or free reads its account in use and one word of a filter of the addresses the accounts recorded (32 KiB); only an address whose bit is set (another account's block, or one sharing its bit) takes the accounts' lock to look the table up. Actor cores and HTTP workers pay nothing for a script's account. A clear bit is enough: a block is recorded under the lock with its bit set first, before the allocator returns it, and whatever thread frees it got the pointer after that, so it sees the bit; a free that finds the bit clear is of a block no account was charged for. The bits are cleared only when the table holds no block. A false positive, a free whose bit another block set, costs a table probe under the lock: with n blocks recorded about n / 262,144 of such frees (8% with 20,000).
  • A thread with an account in use takes the lock for each allocation and free (a table probe and a walk of the account's chain), so threads that each have one contend on it. A waiter pauses, then yields.
  • Freeing an account is one pass over the table, under the lock.

Measured on Windows x86_64 (MinGW, the msvcrt heap), a 64-byte aether_caps_malloc and aether_caps_free, best of 5 passes of 2 million on each thread:

Setting 1 thread 3 threads
no account exists 27 ns 81-145 ns
none exists, one more thread allocating (heap contention only) 74-99 ns 123-198 ns
an account in use on that other thread, holding 20,000 blocks 61-82 ns (199-252 before #3410) 102-156 ns (697-716 before)
each thread its own account 46 ns (41 before) 374-407 ns (532-570 before)

Freeing an account of 40,000 live blocks takes 0.5 ms (1.1 s before).

Failures and nested calls at the boundary

Every aether_<name> export (and every package builder export) runs inside an execution frame (#2955, #2956). How each call ended is its outcome:

#include <libaether.h>

int r = aether_parse(text);
switch (aether_last_outcome()) {
case AETHER_OK:        use(r); break;
case AETHER_PANICKED:  log("script failed: %s", aether_last_outcome_text()); break;
case AETHER_DEADLINE:  /* cut by its deadline: r is not a result */ break;
}
  • A panic ends the call, not the process. A panic the export does not catch used to print "panic outside any try/catch or actor" and abort the host. It now ends the call with AETHER_PANICKED, the export returns the zero value of its result (NULL for a string, which the release accepts), and aether_last_outcome_text() gives the panic's message (copied into a fixed buffer, cut at 1023 bytes: nothing is allocated to report it). The library takes the next call at once. Nothing done before the failure is rolled back: host effects, module state, messages sent.
  • A zero value with an outcome other than AETHER_OK is not a result. Release only what an AETHER_OK call returned. AETHER_DEADLINE and AETHER_CANCELLED mean a safe point ended the call ("Per-call resource caps" above): it returned its zero value, its defers run. A call that ran to its return with a stop noted after its last safe point (an allocation it survived, a JSON parse that stopped, the deadline's flag) ends with that outcome and returns its zero value as well (#3388): the stub releases what the function returned, a string, or the strings and closures of a struct or closure by value. A ptr or *Struct result owns nothing (R4), so what it points to is not released, as on a panic. An Aether importer gives such a string back to the library before it raises the failure, whichever version built the library.
  • Nested calls. An export that calls a host service, which calls an export again on the same thread, runs two executions. The inner one sees what the service set for it (caller info, aether_set_call_deadline), has its own outcome, and whatever it leaves behind is undone when it returns, on every exit: the outer one's caller info, deadline (and trip flag) and sandbox levels are as they were (lib-caller-info.md, containment-sandbox.md). Executions nest up to 32 deep on one thread.
  • An Aether program importing the library in binary form gets a failed call raised again as a panic in its own runtime, with the same reason, so a try around the call catches it; host.last_outcome() and host.last_outcome_text() in std.host read the outcome without one. The stub reads it from the library's own runtime through the accessor the catalog names (outcome, schema 1.10), since a library with a runtime of its own records it there.
  • What a call costs. Catching a panic needs the stub to save its context, where a panic can return to, on every call; nothing else is paid unless something asks for it (a nested call, a deadline ever armed, a sandbox level on some thread, caller info changed during the call). On x86 and x86-64 the stub saves it with __builtin_setjmp (0.57 ns; the C library's setjmp is 2.85 ns on Windows), elsewhere with setjmp. Measured on Windows MinGW, a C host calling in a loop (medians of 9 interleaved rounds against main): an export that calls a function costs 5.7 ns a call against 0.4 ns, about 2 ns of it the registers Win64 makes the stub keep (rbx, rbp, rsi, rdi, r12 to r15, xmm6 to xmm15) for a panic's return; fib(30) through its stub is 0.699 ms against 0.701 ms, the difference lost in the recursion. An export that can neither panic, allocate, loop nor call anything (scalar parameters and locals, + - *, comparisons, if/else, return) gets no frame and records its OK outcome only: 0.57 ns against 0.19 ns.
  • A library made for another runtime runs nothing (#2957). A --shared-runtime library on a runtime of another ABI than the one it was compiled for (a shared runtime replaced in place), or --emit=csrc code linked against another Aether's runtime, finds out at its first call: every export returns its zero value with AETHER_INCOMPATIBLE (6), none of its code runs, and the reason, naming both runtimes, is printed once and in aether_last_outcome_text(). A host asks first with aether_<lib>_check(const char** why): 0, or -1 with the reason. The same C compiled against another Aether's headers does not compile. A host treats an outcome it does not know as a failure. abi-policy.md has what is supported across versions.
  • The host API is in every library. A library built with its own runtime carries aether_set_call_deadline, aether_deadline_tripped, aether_set_memory_cap, aether_set_caller, aether_clear_caller, aether_last_outcome, aether_last_outcome_text, aether_exec_token, aether_exec_cancel and the execution accounts' aether_exec_account_new, aether_exec_account_free, aether_exec_use, aether_exec_account_used and aether_exec_account_tripped even when its code calls none of them, so a host that loads it can call them in its runtime (a library DLL had no aether_set_call_deadline before). On the shared runtime they are the shared runtime's.

What's out of scope for v1

  1. Callbacks held live, a host can't pass a closure into Aether and have it retained. If you want this (the ARexx / rules-engine model), track the "Shape B" design note in aether-embedded-in-host-applications.md.
  2. Per-function capability grants, --with= flags are coarse (fs / net / os). Fine-grained gates like "allow file_open but not dir_delete" don't match any concrete threat model for the default-deny shape, and every additional flag is API surface.
  3. Typed returns beyond void*, functions returning map/list come back as AetherValue* with no schema. Host knows the shape.

Kind discrimination + deep-free (aether_value_kind, aether_config_free_deep)

Aether's collections are intentionally untyped at the storage level, a map's value slot is void*, and the host knows the schema. That historically meant schema-mismatch bugs (script stores an int where a map was expected) degraded into host-side segfaults. v1 closes that gap with a magic-tagged kind discriminator.

Every HashMap and ArrayList carries a uint32_t _kind_magic as its leading field, set at construction and cleared on free. The runtime exposes:

typedef enum {
    AETHER_KIND_UNKNOWN = 0,
    AETHER_KIND_MAP     = 1,
    AETHER_KIND_LIST    = 2
} AetherKind;

AetherKind aether_value_kind(AetherValue* v);
int32_t    aether_value_is_map(AetherValue* v);
int32_t    aether_value_is_list(AetherValue* v);

These are safe to call on any pointer the host holds, including scalars that were intptr-cast to AetherValue* (e.g. (void*)(intptr_t)42 from a schema-mismatched _get_map call). The probe applies two layers of safety:

  1. Low-address guard, the zero page + first 64 KiB are unmapped on every supported OS (Linux's vm.mmap_min_addr default is 65536; macOS reserves more; Windows likewise). Any pointer below 0x10000 is filtered before any deref attempt, so common intptr-cast scalars never reach the magic check.
  2. 32-bit magic check, for higher-address values that survive the guard, the leading 4 bytes either match one of the two known constants (AETHER_KIND_LIST_MAGIC / AETHER_KIND_MAP_MAGIC) or the predicate reports UNKNOWN. False-match probability for a non-container value is 2 / 2³² ≈ 5 × 10⁻¹⁰.

aether_config_free_deep(root) builds on the predicate to walk a nested tree of containers and free each one in post-order. Scalars encountered along the way are left untouched (the host owns those separately). aether_config_free (shallow) auto-detects whether the root is a map or a list via the same predicate, so it now works on either container shape uniformly.

AetherValue* root = aether_build_config("prod", 8080);

if (aether_value_is_map(root)) {
    /* Defensive, confirm what the script actually returned. */
}

AetherValue* db = aether_config_get_map(root, "db");
if (aether_value_kind(db) == AETHER_KIND_MAP) {
    const char* host = aether_config_get_string(db, "host");
    /* … */
}

aether_config_free_deep(root);   /* nested map + list freed too */

Defense-in-depth: list_free and map_free clear the magic before releasing the struct's memory, so a use-after-free probe via aether_value_kind reads the cleared magic and reports UNKNOWN rather than a stale match on the freed-but-still-readable original value.

Reflection: the symbol catalog (aether_lib_meta)

Every --emit=lib artifact exports a single reflection entry point:

const AetherLibMeta* aether_lib_meta(void);

declared in runtime/aether_lib_meta.h. The returned struct describes every exported function, Aether name, C symbol, public signature, source file, source line, in a layout-stable, allocation-free form that any FFI consumer (Python ctypes, Java Panama, Ruby Fiddle, Node-API, hand-rolled dlsym) can walk directly.

The entry point is emitted weak (__attribute__((weak)) under GCC/Clang, #1590), so an orchestrator that links several --emit=lib TUs into one artifact (aeb's regen shape: one C per module, one final link) needs no -Wl,--allow-multiple-definition: the multi-TU static-clone link model of 0.539 holds for lib TUs too, on ld64 as well as GNU ld. In a lone .so nothing changes: one definition, same dlsym contract. In a deliberate multi-TU link the first TU's catalog wins; treat aether_lib_meta as meaningful only on single-module artifacts.

typedef struct {
    const char* aether_name;     /* "double_int", "std.fs.copy"      */
    const char* c_symbol;         /* unmangled symbol to dlsym         */
    const char* signature;        /* "(int) -> int"                    */
    const char* source_file;      /* the .ae that defined it           */
    int         source_line;      /* 1-based                            */
} AetherLibFunction;

typedef struct {
    const char* schema_version;   /* "1.0" funcs; "1.1" closures; "1.2" consts; "1.3" structs */
    const char* aether_version;   /* compiler version                  */
    const char* primary_source;   /* the .ae passed to aetherc         */
    int                       function_count;
    const AetherLibFunction*  functions;
    int                       closure_count;   /* v2: closure-surface records */
    const AetherLibClosure*   closures;        /* NULL when closure_count==0 */
    int                       constant_count;  /* v3: exported-constant records */
    const AetherLibConstant*  constants;       /* NULL when constant_count==0 */
    int                       struct_count;    /* v4: struct records          */
    const AetherLibStruct*    structs;         /* NULL when struct_count==0   */
    const char* const*        source_signatures; /* v4: parallel to functions */
} AetherLibMeta;

No JSON, no parsing, no dynamic allocation, the struct is static const in the artifact and the catalog literally lives in .rodata. Schema is versioned; within 1.x only additive fields appear at the struct tail, so a "1.0" reader walks a "1.1" or "1.2" artifact unchanged (it reads function_count/functions and ignores the closure and constant slots; those fields were always present, just NULL before). schema_version is "1.0" for function-only artifacts, "1.1" once closure records are present, "1.2" once exported-constant records (constant_count/constants, of type AetherLibConstant) are present, and "1.3" once struct records or source signatures are (below), and so on up to "1.9" (runtime/aether_lib_meta.h lists them). Since "1.10" every catalog has outcome, the library's outcome accessor (above). Since "1.11" (#2957) every catalog also has host_abi, the host ABI level it was built at, runtime_abi, the runtime ABI it was compiled for, and requires, a comma-separated list of what its importer must do; every library built now is "1.11". An importer reads any minor of schema 1 and refuses another major, and refuses a library that requires something it does not know, naming it and the fix (abi-policy.md). A reader touches a field only when the artifact's minor version is at least the one that appended it: an older artifact's struct ends before the newer slots. The function table includes every Aether-defined function the linker exports; functions skipped from the aether_<name> alias surface (unsupported types, tuple returns, trailing-underscore privates) are also omitted from it.

Closure-context records (v2, schema "1.1")

The flattened function table above is the foreign-FFI surface; it drops closures, real types, and the builder-DSL shape. The closures array is the complement: the closure surface a downstream Aether consumer needs to reconstruct a closure-with-context builder DSL at full fidelity, rather than the lowest-common-denominator C ABI a foreign host links against. Each record:

typedef struct { const char* name; const char* type; } AetherLibCapture;

typedef struct {
    const char* name;              /* param / closure name, or ""       */
    const char* role;              /* "builder"|"param"|"trailing-block"|"literal" */
    const char* enclosing_export;  /* the export it is reachable from   */
    const char* signature;         /* "|int, string| -> bool"           */
    int                     capture_count;
    const AetherLibCapture* captures;   /* NULL when capture_count == 0  */
    const char* source_file;
    int         source_line;
} AetherLibClosure;

Three sources populate it, all reachable from the set of top-level, non-imported, non-_-suffixed functions (which is broader than the ABI function table, a builder or a fn-parameter function is exactly what the ABI gate excludes, yet its bare symbol is still linkable by an Aether consumer):

  • builder, an export that takes an injected _ctx (the builder keyword or the _ctx: ptr first-param convention). The "call me with a trailing block" signal; capture_count is 0.
  • param, a closure-typed (fn) parameter of an export. name is the parameter, signature its |...| -> R shape.
  • trailing-block, a trailing-block closure literal in an export's body.
  • literal, any other hoisted closure literal in an export's body. For both of these, captures lists the enclosing variables the closure closes over with their rendered Aether types.

Closure-literal return types are best-effort: rendered when resolved, ? when the body type-checks lazily and the type isn't known at emit time. Captures and parameter types are exact.

Struct records and source signatures (v4, schema "1.3", #2297)

The display signature flattens a typed pointer to ptr and spells a struct struct Vec3, which is enough to call through a C ABI and not enough to declare the function on the Aether side. Two additions carry what an Aether importer needs:

typedef struct { const char* name; const char* type; const char* flags; } AetherLibField;

typedef struct {
    const char* name;              /* "Model"                              */
    const char* kind;              /* "", "extern" or "extern_packed"      */
    int                    field_count;
    const AetherLibField*  fields; /* type as Aether source: "*Vec3", "fn(ptr, float) -> int" */
    const char* source_file;
    int         source_line;
} AetherLibStruct;
  • structs lists every struct the library defines and every struct an export's signature reaches (through fields, transitively), including one merged in from a module the library imports. A struct that cannot be re-declared (an opaque or header-defined extern type, a union or bit-width field, a field of enum, sum or distinct type) is left out with a compile-time warning naming why.
  • A struct's C name. Struct types are module-scoped, and an executable emits a module's struct as <ns>_<Name>. A library keeps the plain name for a struct its module exports (all of them when the module has no exports list), since a C host or header may spell it; only unexported module structs are mangled there. So two modules of one library that both export a struct of one name still clash, as every same-named pair did before struct types were module-scoped.
  • source_signatures[i] is functions[i]'s signature as Aether source, with parameter names: "(m: *Model, p: Vec3) -> void". A parameter's @keep or @into is part of it ("(p: @keep ptr) -> void", #3013): the extern ae declares for an importer carries it, so the importer is checked as a source import is (a borrowed ptr given to a @keep parameter, or a string given to one without string.keep, is E0902), and a C host reads the contract: keep the argument alive while the library holds it (@keep) or while the result is held (@into). An entry is "" when an export has no source spelling.

Both are emitted only when an artifact has struct records or a signature the display form flattens (a typed pointer, a struct, a function pointer, a @keep or @into parameter), so a library with none of them stays byte-identical to schema 1.2.

Consuming a published library from Aether (import)

The records above are not just for inspection, they are what lets an Aether program consume a precompiled --emit=lib artifact as a first-class import, with the call site reading as if the library were compiled in the same cycle:

import std.map        // builder default factory (map_new) lives here
import gizmo          // resolves libgizmo.so, no gizmo.ae source needed

main() {
    println(gizmo.greet("world"))     // a function export
    s = gizmo.section("intro") { }    // a builder (trailing-block DSL) export
    println(s)
}

When ae run / ae build / ae check sees an import foo that has no foo source module but a libfoo.so / foo.so on the search path, it reads that artifact's aether_lib_meta() catalog and synthesizes a small Aether interface stub: an @extern("<c_symbol>") name(...) -> R for each function export (typed by its source signature from schema 1.3, a parameter's @keep and @into included), a struct declaration for each struct record, and a trailing-block builder wrapper for each builder record (forwarding the block's config map to the library function's (..., _builder) entry point). The stub is dropped onto the module search path, so the existing source-import machinery (typecheck, namespace prefixing, builder registration) rehydrates the library, no special call-site syntax. The artifact is linked in (absolute path + -rpath), and the host's -rdynamic + static libaether satisfy the .so's runtime symbols.

Scope of the current implementation:

  • Function exports and builder DSL entry points are callable. A builder consumer also import std.map (the default map_new config factory is a std.map facility used at the call site).
  • Structs cross (#2297): a struct passes by value, a *Struct reaches its fields (m.position.x), and a struct with a function-pointer field keeps its layout. The importer declares each struct from the catalog, so its layout is the library's.
  • Module state lives once. A module-level var in the library is the library's single instance for every importer: a second binary library that imports the first, and a host that imports both, all reach the same var.
  • Higher-order exports (a function taking a closure fn parameter) are described in the metadata but not yet callable across the binary boundary, passing an Aether closure into an imported function is a follow-on (the closure ABI across the boundary).
  • Every platform. On Windows the catalog is read by mapping the DLL without running its initialisation code, the DLL is linked directly, and ae build copies each imported DLL next to the program it builds (PE has no rpath); ae run puts their directories on the program's PATH.

One runtime for a program and its libraries (--shared-runtime, schema "1.5")

A library built the default way carries its own copy of the runtime, linked in statically: one scheduler, one set of panic frames, one config per image. That keeps a library self-contained for a C or Python host, but a program that imports Aether libraries in binary form then runs on several runtimes. On Linux and FreeBSD ELF symbol interposition happens to bind a library's runtime calls to the program's copy; macOS's two-level namespace and Windows' PE keep them apart, so a panic raised inside a library never reaches the program's catch there.

--shared-runtime links against the runtime as a shared library instead, aether.dll (with libaether.dll.a), libaether.dylib or libaether.so, which the toolchain builds and installs in lib/aether/shared/ (build/shared/ in a source tree), beside the static archive:

ae build --emit=lib --shared-runtime engine.ae -o libengine.so
ae build --emit=lib --shared-runtime script.ae -o libscript.so   # imports engine
ae build app.ae -o app                                            # imports both
  • The library's catalog records it (runtime = "shared", schema 1.5; ae lib-info prints Runtime: shared), and a program importing such a library links the shared runtime too, without being asked.
  • Program and libraries then run on one runtime: a panic inside a library unwinds to the program's catch on every platform, and actors, config and resource caps are the program's.
  • The shared runtime is linked by path (never -laether, which keeps finding the static archive): an rpath to its directory on Linux and macOS, and on Windows aether.dll is copied next to the program (ae build) or put on PATH (ae run). The Windows DLL is self-contained: it needs only system DLLs and the UCRT.
  • A program that imports one library built on the shared runtime and another built static gets a warning naming the static one: rebuild it with --shared-runtime.
  • Each copy of the runtime frees only what it made. A string an export returns crosses as a copy (above), but a struct holding a string (returned by value, or reached through a typed pointer) and a builder that uses its config have none. Where the copies stay apart (macOS, Windows), importing a static library that has such an export is an error naming each one, with the rebuild: --shared-runtime.
  • A sandbox's grants are each thread's, in the runtime: on one runtime a library's export called from inside another's sandbox.enforce block is held to the block's grants. Where the copies stay apart, the inner copy cannot read them and refuses what the block would check, with a message naming --shared-runtime (containment-sandbox.md).
  • Hooks a program supplies to the runtime by @c_callback (the pure-TLS client and server) reach a shared runtime by registration: every @c_callback registers itself by name when its image loads, and the runtime looks a hook up before falling back to its own definition.

A library that runs actors (schema "1.7")

A library's exports may spawn actors that the program never defines:

ae build --emit=lib --shared-runtime crowd.ae -o libcrowd.so   # spawns actors
ae build app.ae -o app                                          # has none
  • The scheduler starts on the first spawn. A program's main() initializes the scheduler only when the program has actors. Otherwise the first spawn initializes it and starts its threads. This holds for an Aether program and for a C host. A second scheduler_init() while one is live is a no-op, so it cannot wipe the actor tables of actors already running. scheduler_shutdown() and scheduler_cleanup() end the lifecycle.
  • The catalog says so. The library's catalog has actors = 1, and ae lib-info prints Actors: yes. This also covers a library with no actors of its own that imports one which has them. A program that imports such a library has main() run the scheduler as it does for its own actors. On the way out, main() drains the scheduler and joins it, so a message the library sent fire-and-forget is still delivered.
  • Draining needs one runtime. The drain reaches the library's actors where the program and the library share one runtime: --shared-runtime everywhere, or ELF interposition with the default static runtime (Linux, FreeBSD). On macOS and Windows a static library's actors run on that library's own scheduler. ae warns that messages still in flight when the program exits are not delivered, and names the library to rebuild with --shared-runtime.
  • A C host calls scheduler_shutdown() before it exits to deliver what is still in flight.

A package as one library (--package, schema "1.4")

A package of many modules (ae3d.core, ae3d.gl, ...) builds into one library, which programs then import module by module (#2297):

ae build --emit=lib --package ae3d --lib src -o libae3d.so
  • What is built. The package's modules are its directory under ., src or a --lib / dependency directory (src/ae3d/): that directory if it holds a module.ae, and every directory below it that does (src/ae3d/core/module.ae is ae3d.core). ae writes an entry that imports all of them and builds it with --lib-package=ae3d. The default output name is lib<package>, with dots as _.
  • What is exported. Each module's public functions and constants are exported: the names in its exports (...) list, or all of them when it has no list, minus _-suffixed privates. A function gets a stable C symbol named after its module, aether_ae3d_core__model_new. The compiler's own merged name (core_model_new) depends on which other modules a build loads, so it is never the ABI. The package's structs are recorded as in schema 1.3, and module vars live once, in the library.
  • The catalog. function_modules and constant_modules are parallel to the function and constant tables and give the module each export belongs to. aether_name is the bare name (model_new). ae lib-info prints a module: line under each export.
  • Importing. import ae3d.core with no ae3d/core source on the path resolves to the package library: ae tries each proper prefix of the import, longest first (ae3d → libae3d), and takes the library whose catalog lists ae3d.core. The interface it builds for that module declares the module's functions and constants and the library's structs, so core.model_new(...) reads as it does against source. Importing several modules of one package links the library once.
  • Builders of a package module are exported too (#2349, schema 1.6). A builder takes its parameters plus the trailing block's void* _builder config, so the library exports a wrapper with that signature, aether_ae3d_ui__panel. Its catalog record carries the module (closure_modules), and a module's interface declares it as a trailing-block builder, so ui.panel("hud") { ... } works against the library. An importer of a builder also import std.map, as for a single-module library. A builder with a pattern parameter is the one form left out, with a warning naming it.

ae lib-info <path> inspect any artifact

The ae CLI ships a turnkey reader that dlopens a .so/.dylib, calls aether_lib_meta, and prints a human-readable dump:

$ ae lib-info build/libscript.so
Aether Library: build/libscript.so
  Schema:        1.0
  Aether:        0.134.0-dev
  Source:        script.ae
  Functions:     3

  - aether_script_handle(ptr, ptr, ptr) -> void
        @ script.ae:3
  - double_int(int) -> int
        c_symbol: aether_double_int
        @ script.ae:7
  - greet(string) -> string
        c_symbol: aether_greet
        @ script.ae:11

The c_symbol: line is suppressed when the symbol equals aether_<name> (the default for plain exports, printing it would just be noise). @c_callback-marked functions whose Aether name is the C symbol show no c_symbol: line either; they are their own export.

On Windows ae lib-info maps the DLL without running its initialisation code and reads the same catalog; any host reaches it with LoadLibrary + GetProcAddress("aether_lib_meta"). A DLL that ae links exports the entry point strong: PE cannot export a weak definition, so the generated code is compiled with -DAETHER_LIB_META_WEAK= there (#2297).

What this enables for FFI hosts

Any embedder can build a typed binding without re-parsing the source:

import ctypes
lib = ctypes.CDLL("./libscript.so")
class AetherLibFunction(ctypes.Structure):
    _fields_ = [("aether_name", ctypes.c_char_p),
                ("c_symbol",    ctypes.c_char_p),
                ("signature",   ctypes.c_char_p),
                ("source_file", ctypes.c_char_p),
                ("source_line", ctypes.c_int)]
class AetherLibMeta(ctypes.Structure):
    _fields_ = [("schema_version", ctypes.c_char_p),
                ("aether_version", ctypes.c_char_p),
                ("primary_source", ctypes.c_char_p),
                ("function_count", ctypes.c_int),
                ("functions",      ctypes.POINTER(AetherLibFunction)),
                ("closure_count",  ctypes.c_int),
                ("closures",       ctypes.c_void_p)]
lib.aether_lib_meta.restype = ctypes.POINTER(AetherLibMeta)
meta = lib.aether_lib_meta()[0]
for i in range(meta.function_count):
    fn = meta.functions[i]
    print(fn.aether_name.decode(), fn.signature.decode())

Same struct walks from Java Panama, Ruby Fiddle, Go cgo. The flat-C layout is the lowest common denominator across every FFI in the wild.

Working tests

The integration suite under tests/integration/ covers:

Test What it proves
emit_lib/ Primitive round-trip through dlopen
emit_lib_composite/ Nested map + list round-trip via accessors
emit_lib_lists/ List-of-ints, list-of-strings, empty list, out-of-range
emit_lib_primitives/ long, bool, float across the boundary
emit_lib_unsupported/ Unsupported param types warn + skip stub
emit_lib_clause_sets/ Clause sets (guards, a literal first clause, @c_callback) and an export-wrapped function: each listed once in the catalog and the header with its set's signature, every catalog symbol exported, each callable from a C host (#2665, #2666)
emit_lib_banned/ All five capability-heavy imports rejected
emit_lib_dual_build/ Same source → exe AND lib via separate invocations
emit_lib_swig/ SWIG Python round-trip (skips if swig missing)
emit_lib_with_capability/ --with=fs,net,os opt-ins; --with=first-party and --with=all aliases
lib_meta/ aether_lib_meta + ae lib-info round-trip, schema, source, function count, three signatures, c_symbol gating, source refs
binary_import_package/ --package builds two modules into one library: stable aether_<module>__<name> symbols, schema 1.4 module records, export-list and _-suffix privacy, a host importing both modules by name from the binary (#2297)
binary_import_structs/ Schema 1.3 struct records and source signatures: a struct by value, *Struct field access, a function-pointer field, closures both ways with their captures, one module state shared by a host and a second binary library; static runtime on Linux/FreeBSD (a library panic caught; elsewhere the import is refused, since *Model holds a string) and --shared-runtime (caught everywhere, schema 1.5), Windows included (#2297)
binary_import_ownership/ Strings an export or a builder returns are given back to the library's runtime: 0 growth and no leaks(1) findings for a binary import (static and shared runtime) and a C host releasing through the catalog's string_release; a struct holding a string and a builder freeing its config run on one runtime and are refused across two (#2931, #2861)
binary_import_keep/ A @keep ptr and an @into ptr export cross the catalog's source signature, and each importer gets the same E0902 verdict from ae check against the library as against its source (a string or a borrowed ptr to the @keep parameter, an @into result that escapes); the accepted one runs (#3013)
binary_import_actors/ A library's actors in a program with none of its own: the first spawn starts the scheduler, schema 1.7 actors passes through a second library, and the program's exit drains a fire-and-forget message (shared runtime everywhere, static on Linux/FreeBSD; elsewhere a warning) (#2297)
emit_lib_keeps_main/ A program's main() as aether_main / aether_main_exit: a C host runs it (args, an ask/reply, its return value), the actors outlive aether_main and aether_main_exit drains them, a second aether_main is rejected, a rerun works without leaking scheduler tables (macOS leaks), the executable prints the same lines, no pair without main(), the reserved main_exit, the --emit=csrc header, and aarch64-linux / Android cross builds
emit_lib_kind_safe/ Kind-discriminator predicates + deep-free safety, adversarial low-address probe ((AetherValue*)42) survives, kind correctly classifies map/list/scalar slots, deep-free walks nested map+list+scalars, magic-clear-on-free defends UAF probes

Run them with the standard make test-ae or individually:

tests/integration/emit_lib_composite/test_emit_lib_composite.sh