OpenBSD and Windows: make ci portability batch

new_changelogs/20261109T225917Z-changed-openbsd-test-portability.md

@@ -0,0 +1 @@
+- **The shell test suite runs on OpenBSD** (`tests/lib/portable.sh`, `tests/lib/syslibs.sh`, `tests/lib/test_openssl_conf.sh`, and the tests that use them). Hand-written C link lines name `-ldl`/`-lrt` only on Linux (`$AE_DL_LIBS`, `$AE_RT_LIBS`); `head -c`, `sha256sum`/`shasum`, `tar -T`, `nm -D`, `od -tu` (OpenBSD zero-pads it) and `file` (which prints `data` for wasm, COFF and Mach-O, and `AArch64` for `aarch64`) go through `head_c`, `sha256_of`, `copy_listed`, `dyn_syms`, `od_uint` and `file_desc`; the 75 `grep "a\|b"` alternations, which OpenBSD's grep reads as a literal `|` (so a "warning filter" let warnings into compared output), are `grep -E`; `\b` and `\s` are POSIX classes; `grep -r --exclude` is `find` (OpenBSD matches `--exclude` against the whole path). `x509 -req` is told `-sha256`: LibreSSL signs with SHA-1 by default and the pure TLS stack rightly refuses that. Tests whose subject cannot exist on the host skip and say why: `ae version install` where there is no release build, musl targets where zig cannot build musl's libc.a, dual-stack `::` where sockets are IPv6-only, the poisoning allocator where libc has no `malloc_usable_size`, FORTIFY where libc has no `__*_chk`, `spawn_sandboxed` off Linux and FreeBSD, and the checks that need OpenSSL's `ssl_conf`, `s_server -cert_chain`, `s_client -sigalgs` or Ed25519 certificates in TLS.

new_changelogs/20261109T225917Z-fixed-openbsd-harness-product.md

@@ -0,0 +1 @@
+- **OpenBSD: four toolchain and runtime gaps the test suite exposed** (`tools/ae.c`, `tools/ae_checksec.c`, `std/os/aether_os.c`, `std/net/aether_http.c`). A program importing `std.audio` no longer fails to link with "unable to find library -ldl": OpenBSD keeps `dlopen` in libc, and `ae` now leaves the `-ldl` that codegen's link table names for `std.audio` off the link line there (as it already does on MinGW). `ae checksec` reads OpenBSD's RETGUARD return-address cookies (`__retguard_*`, and `__stack_smash_handler`) as the stack canary; it reported `canary no` for every OpenBSD binary because that clang does not emit `__stack_chk_fail` for a plain stack array. A spawned child no longer keeps a close-on-exec pipe that took a closed stdout's number as its stdout on OpenBSD and NetBSD: their pipes all report device 0, inode 0, and the pipe time that told them apart is the kernel's coarse clock (10 ms), so a spawn pipe made within a tick of the process start was taken for the stream itself; a stream that was not close-on-exec at start can no longer match a close-on-exec one (`os_spawn_stdio_slot` failed there). The OpenSSL-backed HTTPS client now honours `SSL_CERT_FILE` and `SSL_CERT_DIR` when built against LibreSSL, which ignores both in `SSL_CTX_set_default_verify_paths` (the pure-Aether client and OpenSSL already did).

tests/compiler/test_msvc_compat.sh

@@ -483,7 +483,7 @@ if [ -n "$use_gcc" ] && [ "${MSVC_COMPAT_SWEEP:-0}" = 1 ]; then
         for b in "$SWEEP_TMP"/bad_*; do
             n="${b#"$SWEEP_TMP"/bad_}"
             echo "    $(cat "$b")"
-            grep -B1 -A2 "braced-groups\|fatal error" "$SWEEP_TMP/$n.log" 2>/dev/null | head -8 | sed 's/^/      /'
+            grep -B1 -A2 -E "braced-groups|fatal error" "$SWEEP_TMP/$n.log" 2>/dev/null | head -8 | sed 's/^/      /'
         done
         echo "  [FAIL] $bad of the sweep's programs have GCC extensions on the non-GCC path"
         exit 1

tests/integration/abi_matrix/test_abi_matrix.sh

@@ -41,6 +41,7 @@
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+. "$ROOT/tests/lib/syslibs.sh"
 AE="$ROOT/build/ae"
 CC="${CC:-cc}"
 export AETHER_STACK_TRACE=0
@@ -269,7 +270,7 @@ preflight: refused, the library was made for Aether runtime "*) ;;
             esac
             if (cd "$TMP/oldsh" && cp "$SCRIPT_DIR/app.ae" . && AETHER_HOME="$ROOT" "$AE" build app.ae -o app > app_build.log 2>&1); then
                 bad "this importer took the $old_version shared-runtime library"
-            elif ! grep -q "host ABI level 0\|was made for the Aether runtime" "$TMP/oldsh/app_build.log"; then
+            elif ! grep -q -E "host ABI level 0|was made for the Aether runtime" "$TMP/oldsh/app_build.log"; then
                 bad "this importer's refusal of the $old_version shared-runtime library says why" "$TMP/oldsh/app_build.log"
                 grep -A2 '^note: .* could not be loaded' "$TMP/oldsh/app_build.log" | sed 's/^/        /'
             fi

tests/integration/abi_policy/test_abi_policy.sh

@@ -17,6 +17,7 @@
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+. "$ROOT/tests/lib/syslibs.sh"
 AE="$ROOT/build/ae"
 AETHERC="$ROOT/build/aetherc"
 CC="${CC:-cc}"

tests/integration/actor_ref_field_reject/test_actor_ref_field_reject.sh

@@ -24,7 +24,7 @@ refused() {
         fail "aetherc accepted $1" "$TMPDIR/err.log"
     fi
     grep -q "$2" "$TMPDIR/err.log" || fail "$1: no error containing '$2'" "$TMPDIR/err.log"
-    if grep -q "request for member\|not a structure or union" "$TMPDIR/err.log"; then
+    if grep -q -E "request for member|not a structure or union" "$TMPDIR/err.log"; then
         fail "$1 reached the C compiler" "$TMPDIR/err.log"
     fi
 }

tests/integration/actor_state_ref_name_not_type/test_actor_state_ref_name_not_type.sh

@@ -69,7 +69,7 @@ AE
 
 want="v 30
 hops 3 3"
-got="$("$AE" run "$tmp/main.ae" 2>&1 | tr -d '\r' | grep -v "^warning\|^ *-->\|^ *[0-9]* |\|^ *|\|^Type checking\|^$")"
+got="$("$AE" run "$tmp/main.ae" 2>&1 | tr -d '\r' | grep -v -E "^warning|^ *-->|^ *[0-9]* [|]|^ *[|]|^Type checking|^$")"
 if [ "$got" != "$want" ]; then
     echo "  [FAIL] actor_state_ref_name_not_type: wrong output"
     printf 'got:\n%s\nwant:\n%s\n' "$got" "$want" | sed 's/^/        /'
@@ -172,7 +172,7 @@ main() {
     println("hits ${s.hits} ${t.hits} tally ${m} greeted ${gh} ${hh}")
 }
 AE
-got="$("$AE" run "$tmp/uses.ae" 2>&1 | tr -d '\r' | grep -v "^warning\|^ *-->\|^ *[0-9]* |\|^ *|\|^Type checking\|^$")"
+got="$("$AE" run "$tmp/uses.ae" 2>&1 | tr -d '\r' | grep -v -E "^warning|^ *-->|^ *[0-9]* [|]|^ *[|]|^Type checking|^$")"
 if [ "$got" != "hits 10 2 tally 42 greeted 1 1" ]; then
     echo "  [FAIL] actor_state_ref_name_not_type: a reference field used from outside the actor"
     printf 'got:\n%s\nwant:\nhits 10 2 tally 42 greeted 1 1\n' "$got" | sed 's/^/        /'

tests/integration/actor_state_typed_declarations/test_actor_state_typed_declarations.sh

@@ -74,7 +74,7 @@ AE
 want="b8 4 h16 9
 big 50 null 1
 q null"
-got="$("$AE" run "$tmp/main.ae" 2>&1 | tr -d '\r' | grep -v "^warning\|^ *-->\|^ *[0-9]* |\|^ *|\|^Type checking\|^$")"
+got="$("$AE" run "$tmp/main.ae" 2>&1 | tr -d '\r' | grep -v -E "^warning|^ *-->|^ *[0-9]* [|]|^ *[|]|^Type checking|^$")"
 if [ "$got" != "$want" ]; then
     echo "  [FAIL] actor_state_typed_declarations: wrong output"
     printf 'got:\n%s\nwant:\n%s\n' "$got" "$want" | sed 's/^/        /'

tests/integration/ae_add_binary_package/test_ae_add_binary_package.sh

@@ -31,6 +31,7 @@
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+. "$ROOT/tests/lib/portable.sh"
 AE="$ROOT/build/ae"
 [ -n "${EXE_EXT:-}" ] && AE="$AE$EXE_EXT"
 
@@ -43,9 +44,7 @@ case "$(uname -s)" in
     *) command -v curl >/dev/null 2>&1 ||
            { echo "  [SKIP] ae_add_binary_package: curl needed for the file:// forge"; exit 0; } ;;
 esac
-if command -v sha256sum >/dev/null 2>&1; then SHA="sha256sum"
-elif command -v shasum >/dev/null 2>&1; then SHA="shasum -a 256"
-else echo "  [SKIP] ae_add_binary_package: no sha256 tool"; exit 0; fi
+SHA=sha256_of
 
 # The binary-package path is per-host-triple; on a host we can't name a triple
 # for, ae_host_triple() returns NULL and everything correctly falls to git.
@@ -107,7 +106,7 @@ mk_binpkg() {
     asset="$STEM-$tag-$TRIPLE$EXT"
     # A small but real binary payload (not a real dlopen target — this test
     # exercises ae add's FETCH + INSTALL, not loading).
-    head -c 4096 /dev/urandom > "$d/$asset"
+    head_c 4096 /dev/urandom > "$d/$asset"
     if [ "$mode" = "badsum" ]; then
         echo "0000000000000000000000000000000000000000000000000000000000000000  $asset" > "$d/$asset.sha256"
     elif [ "$mode" = "nosum" ]; then
@@ -122,7 +121,7 @@ mk_binpkg() {
         *)            FT="linux-x86_64";  FEXT=".so" ;;
     esac
     fasset="$STEM-$tag-$FT$FEXT"
-    head -c 4096 /dev/urandom > "$d/$fasset"
+    head_c 4096 /dev/urandom > "$d/$fasset"
     ( cd "$d" && $SHA "$fasset" | awk -v n="$fasset" '{print $1"  "n}' > "$fasset.sha256" )
 }
 mk_binpkg v1.0.0 ok

tests/integration/ae_add_release_artifact/test_ae_add_release_artifact.sh

@@ -25,16 +25,15 @@
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+. "$ROOT/tests/lib/portable.sh"
 . "$ROOT/tests/lib/server_jobs.sh"
 AE="$ROOT/build/ae"
 [ -n "${EXE_EXT:-}" ] && AE="$AE$EXE_EXT"
 
 [ -x "$AE" ] || { echo "  [SKIP] ae_add_release_artifact: ae not built"; exit 0; }
 command -v python3 >/dev/null 2>&1 || { echo "  [SKIP] ae_add_release_artifact: python3 needed to serve fixtures"; exit 0; }
 command -v tar >/dev/null 2>&1 || { echo "  [SKIP] ae_add_release_artifact: tar needed"; exit 0; }
-if command -v sha256sum >/dev/null 2>&1; then SHA="sha256sum"
-elif command -v shasum >/dev/null 2>&1; then SHA="shasum -a 256"
-else echo "  [SKIP] ae_add_release_artifact: no sha256 tool"; exit 0; fi
+SHA=sha256_of
 
 # The artifact path is per-host-triple; on a host we publish no assets
 # for, ae_host_triple() returns NULL and everything correctly falls back

tests/integration/ae_download_api_fallback/test_ae_download_api_fallback.sh

@@ -19,6 +19,7 @@
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+. "$ROOT/tests/lib/portable.sh"
 . "$ROOT/tests/lib/server_jobs.sh"
 AE="$ROOT/build/ae"
 [ -n "${EXE_EXT:-}" ] && AE="$AE$EXE_EXT"
@@ -40,6 +41,9 @@ fail() {
 
 PLAT="$("$AE" version 2>&1 | sed -n 's/^Platform:[[:space:]]*//p' | head -1)"
 [ -n "$PLAT" ] || { echo "  [SKIP] ae_download_api_fallback: cannot read the platform from 'ae version'"; exit 0; }
+# A host with no release build (OpenBSD, NetBSD, ...) has no asset to serve:
+# `ae version` says "unsupported" and `ae version install` takes the source path.
+[ "$PLAT" = unsupported ] && { echo "  [SKIP] ae_download_api_fallback: no release build for this host ($(uname -s))"; exit 0; }
 case "$PLAT" in windows*) echo "  [SKIP] ae_download_api_fallback: zip archives, not covered here"; exit 0 ;; esac
 
 # A release archive that passes ae's post-extract checks: flat layout, a
@@ -49,8 +53,8 @@ ASSET="aether-9.9.9-$PLAT.tar.gz"
 mkdir -p "$TMP/pkg/bin" "$TMP/pkg/lib/aether" "$TMP/pkg/share/aether" "$TMP/rel"
 printf '#!/bin/sh\necho aetherc 9.9.9\n' > "$TMP/pkg/bin/aetherc"
 chmod 755 "$TMP/pkg/bin/aetherc"; cp "$TMP/pkg/bin/aetherc" "$TMP/pkg/bin/ae"
-{ printf '!<arch>\n'; head -c 70000 /dev/urandom; } > "$TMP/pkg/lib/aether/libaether.a"
-head -c 20000 /dev/urandom > "$TMP/pkg/share/aether/pad.bin"
+{ printf '!<arch>\n'; head_c 70000 /dev/urandom; } > "$TMP/pkg/lib/aether/libaether.a"
+head_c 20000 /dev/urandom > "$TMP/pkg/share/aether/pad.bin"
 echo 9.9.9 > "$TMP/pkg/VERSION"
 ( cd "$TMP/pkg" && COPYFILE_DISABLE=1 tar -czf "$TMP/rel/$ASSET" . )
 

tests/integration/ae_help/test_ae_help.sh

@@ -6,6 +6,7 @@
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+. "$ROOT/tests/lib/portable.sh"
 AE="$ROOT/build/ae"
 
 TMPDIR="$(mktemp -d)"
@@ -140,7 +141,7 @@ main() {
 }
 EOF
 out=$("$AE" help "$TMPDIR/json.ae" --json 2>&1)
-first=$(printf '%s' "$out" | head -c 1)
+first=$(printf '%s' "$out" | head_c 1)
 last=$(printf '%s' "$out" | tr -d '\n' | tail -c 1)
 if [ "$first" = "{" ] && [ "$last" = "}" ]; then
     echo "  [PASS] --json emits a single JSON object"

tests/integration/aea_artifacts/test_aea_artifacts.sh

@@ -11,6 +11,7 @@
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+. "$ROOT/tests/lib/portable.sh"
 AETHERC="$ROOT/build/aetherc"
 [ -f "$AETHERC.exe" ] && AETHERC="$AETHERC.exe"
 EXE=""
@@ -187,7 +188,7 @@ if grep -q "not using .*/md2.aea: source changed since the artifact was built" "
 else bad "edited source not detected"; cat "$T/edit_trace.txt"; fi
 
 # 6. A damaged artifact is ignored, not trusted.
-head -c 200 "$MODS/std/cryptography/sha3.aea" > "$T/short.aea"
+head_c 200 "$MODS/std/cryptography/sha3.aea" > "$T/short.aea"
 cp "$T/short.aea" "$MODS/std/cryptography/sha3.aea"
 aec_trace app.ae "$T/damaged.c" 2>"$T/dmg_trace.txt"
 from_source app.ae "$T/damaged_src.c" 2>/dev/null

tests/integration/aether_cfn_proc_address/test_aether_cfn_proc_address.sh

@@ -22,6 +22,7 @@ set -e
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+. "$ROOT/tests/lib/syslibs.sh"
 AETHERC="$ROOT/build/aetherc"
 
 if [ ! -x "$AETHERC" ]; then
@@ -142,7 +143,7 @@ for d in "$ROOT" "$ROOT/runtime" "$ROOT/runtime/actors" "$ROOT/runtime/scheduler
 done
 
 if ! ${CC:-cc} $INCLUDES -o "$tmpdir/app_bin" "$tmpdir/app.gen.c" "$tmpdir/glue.c" \
-        "$ROOT/build/libaether.a" -lpthread -ldl -lm 2>"$tmpdir/cc.err"; then
+        "$ROOT/build/libaether.a" -lpthread $AE_DL_LIBS -lm 2>"$tmpdir/cc.err"; then
     echo "  [FAIL] aether_cfn_proc_address: C compile returned non-zero"
     sed 's/^/    /' "$tmpdir/cc.err"
     exit 1

tests/integration/aether_extern_param_annotation/test_aether_extern_param_annotation.sh

@@ -14,6 +14,7 @@
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+. "$ROOT/tests/lib/syslibs.sh"
 AETHERC="$ROOT/build/aetherc"
 
 if [ ! -x "$AETHERC" ]; then
@@ -89,7 +90,7 @@ INC=$(find "$PREFIX/include/aether" -type d 2>/dev/null | sed 's|^|-I|' | tr '\n
 ${CC:-cc} -O2 -Wall -Wextra -Werror $INC \
     "$tmpdir/positive.c" "$tmpdir/helper.c" \
     -L"$PREFIX/lib/aether" -laether \
-    -pthread -lm -ldl \
+    -pthread -lm $AE_DL_LIBS \
     -o "$tmpdir/positive" >/dev/null 2>"$tmpdir/link.err" || {
     echo "  [FAIL] positive case link failed"
     head -3 "$tmpdir/link.err"

tests/integration/aether_extern_struct/test_aether_extern_struct.sh

@@ -21,6 +21,7 @@ set -e
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+. "$ROOT/tests/lib/syslibs.sh"
 AETHERC="$ROOT/build/aetherc"
 AE="$ROOT/build/ae"
 
@@ -292,7 +293,7 @@ if ! ${CC:-cc} -c -o "$tmpdir/case4_glue.o" "$tmpdir/case4_glue.c" \
     exit 1
 fi
 if ! ${CC:-cc} -o "$tmpdir/case4_bin" "$tmpdir/case4.gen.o" "$tmpdir/case4_glue.o" \
-        "$ROOT/build/libaether.a" -lpthread -ldl -lm 2>"$tmpdir/case4.gcc.err"; then
+        "$ROOT/build/libaether.a" -lpthread $AE_DL_LIBS -lm 2>"$tmpdir/case4.gcc.err"; then
     echo "  [FAIL] case 4: gcc failed linking"
     cat "$tmpdir/case4.gcc.err" | sed 's/^/    /'
     exit 1

tests/integration/aether_fnptr_call/test_aether_fnptr_call.sh

@@ -18,6 +18,7 @@ set -e
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+. "$ROOT/tests/lib/syslibs.sh"
 AETHERC="$ROOT/build/aetherc"
 
 if [ ! -x "$AETHERC" ]; then
@@ -188,7 +189,7 @@ done
 
 if ! ${CC:-cc} $INCLUDES -o "$tmpdir/case4_bin" \
         "$tmpdir/case4.gen.c" "$tmpdir/case4_glue.c" \
-        "$ROOT/build/libaether.a" -lpthread -ldl -lm 2>"$tmpdir/case4.gcc.err"; then
+        "$ROOT/build/libaether.a" -lpthread $AE_DL_LIBS -lm 2>"$tmpdir/case4.gcc.err"; then
     echo "  [FAIL] case 4: gcc returned non-zero"
     cat "$tmpdir/case4.gcc.err" | sed 's/^/    /'
     exit 1

tests/integration/array_field_length/test_array_field_length.sh

@@ -90,7 +90,7 @@ main() {
     println("${v.xs[3]} ${w.xs[1]} ${w.xs[3]} ${s.names[0]} ${string.length(s.names[2])} ${t.names[2]}")
 }
 AE
-got="$("$AE" run "$tmp/ok.ae" 2>&1 | tr -d '\r' | grep -v "^warning\|^ *-->\|^ *[0-9]* |\|^ *|\|^Type checking\|^$")"
+got="$("$AE" run "$tmp/ok.ae" 2>&1 | tr -d '\r' | grep -v -E "^warning|^ *-->|^ *[0-9]* [|]|^ *[|]|^Type checking|^$")"
 if [ "$got" != "4 6 0 x 0 c" ]; then
     echo "  [FAIL] array_field_length: arrays of the field's length and shorter literals"
     printf 'got:\n%s\nwant:\n4 6 0 x 0 c\n' "$got" | sed 's/^/        /'

tests/integration/assign_to_call_result/test_assign_to_call_result.sh

@@ -128,7 +128,7 @@ main() {
     println("${q.x} ${q.y} ${g.cells[1]}")
 }
 AE
-got="$("$AE" run good.ae 2>&1 | tr -d '\r' | grep -v "^warning\|^ *-->\|^ *[0-9]* |\|^ *|\|^Type checking\|^$")"
+got="$("$AE" run good.ae 2>&1 | tr -d '\r' | grep -v -E "^warning|^ *-->|^ *[0-9]* [|]|^ *[|]|^Type checking|^$")"
 if [ "$got" != "5 7 20" ]; then
     echo "  [FAIL] assign_to_call_result: an assignment through a returned pointer stopped working"
     printf 'got:\n%s\nwant:\n5 7 20\n' "$got" | sed 's/^/        /'

tests/integration/binary_import_ownership/test_binary_import_ownership.sh

@@ -17,6 +17,7 @@
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+. "$ROOT/tests/lib/syslibs.sh"
 AE="$ROOT/build/ae"
 [ -x "$AE" ] || [ -x "$AE.exe" ] || { echo "  [SKIP] binary_import_ownership: ae not built"; exit 0; }
 CC="${CC:-cc}"

tests/integration/block_nesting_cc/test_block_nesting_cc.sh

@@ -59,7 +59,7 @@ runs matches "matches 0"
 
 # The deepest ( [ { of the C of the matches, outside literals and comments.
 AETHER_HOME="$ROOT" "$AETHERC" "$WORK/matches.ae" "$WORK/matches.c" >/dev/null 2>&1
-d="$(awk 'BEGIN { d = 0; m = 0 }
+d="$(LC_ALL=C awk 'BEGIN { d = 0; m = 0 }
           /^[ \t]*#/ { next }
           { n = length($0); str = 0
             for (i = 1; i <= n; i++) {