diff --git a/docs/guides/database-validation.md b/docs/guides/database-validation.md index 7717db533..2844db231 100644 --- a/docs/guides/database-validation.md +++ b/docs/guides/database-validation.md @@ -1,6 +1,6 @@ # Database service acceptance -This record separates public-module service/failure checks, positive workflows through actual plugin DSOs, and server-observed request cancellation through those DSOs. Each scope has its own tested revisions and environment; a passing result applies only to that scope. +This record separates public-module service/failure checks, positive plugin-DSO workflows, server-observed request cancellation and owned-server termination. Each scope has its own tested revisions and environment; a passing result applies only to that scope. ## Public modules against real services @@ -94,6 +94,48 @@ cmake --build database-build --target ark_database_public_plugin_cancellation_te `check-database` compiles the program without running the real-service modes. Both owned servers were shut down normally using authenticated commands; the exact processes were independently confirmed absent and private data, credentials and TLS keys were removed. Server termination was not a test action. +## Real plugin server termination + +Both modes of `ark_database_public_plugin_server_termination_tests` passed on **2026-10-26**. Each C++ host and independent fault actor exited `0`; each owned server's intentional `SIGKILL` exit was `137`. + +| Environment | Value | +| --- | --- | +| Machine | Mac16,11 · Apple M4 Pro · 14 cores · 48 GiB | +| System | macOS 27.0.1 (26A434) · arm64 | +| Build | Apple Clang 21.0.0 · C++20 · Boost.Asio · Debug · SSL enabled · OpenSSL 3.6.4 | +| Executed ark | `0722b8bdd2e2c240657a3ceea63428cbbff0f821` | +| arknet | CI pin `a0dfe09cd543e75cfb5996e537ce2fa49f2a758f` | +| Services | Redis 7.4.2 over authenticated TCP; MySQL Community Server 8.4.13 with required verified TLS | + +The host uses public interfaces and actual hidden-symbol plugin DSOs. An independent service connection must observe the exact active request before releasing a private fault barrier; a polling delay does not establish activity. + +| Mode | Observed read-only request before the fault | Original native result after server termination | +| --- | --- | --- | +| `redis` | `CLIENT LIST` identified the exact named client blocked in `XREAD` | `client_errc::disconnected`, `command_may_have_run=true`, and EOF transport status | +| `mysql` | `PROCESSLIST` identified the tagged `SELECT SLEEP(60)` connection in `User sleep` | `connection_errc::transport`, `outcome_unknown=true`, and TLS stream truncation | + +Before signaling the single owned PID, the separate shell actor resolves private paths physically and rejects escapes or symlink coordination files. It rechecks ownership, PID/start time, executable, private data working directory, PID file and that PID's loopback listener; MySQL also checks its private defaults-file argument. These guards apply to the disposable instances prepared for this test, not shared or installed services. + +Both modes reject task cancellation, operation timeout, SQL server error and coroutine exceptions. Request and observer roots complete once on the owner thread/shared executor and retire before cooperative stop. The controlled tests complete natural shutdown within their observation and pumping bounds; repeated stop succeeds, and owning native results/errors remain readable after module/manager/runtime destruction and native DSO close. `manager.stop()` still cannot interrupt blocking hooks; no hard or preemptive production deadline is added. Native transport details are platform-specific. + +The parent reaped the owned server, actor and host children, independently confirmed the exact processes absent, and removed private data, credentials and TLS keys. There was no server restart/recovery, persistence or write-effect assertion. + +Use the [standalone C++ termination guide](../../tests/database/server_termination.md) for private service/actor coordination and endpoint variables. The actor requires `lsof`, `ps` and `rg`. Build the opt-in program with the SSL-enabled database configuration, then run a mode only with its owned fault actor prepared as described in that guide: + +```sh +cmake --build database-build --target ark_database_public_plugin_server_termination_tests --parallel 1 +./database-build/tests/database/ark_database_public_plugin_server_termination_tests redis +./database-build/tests/database/ark_database_public_plugin_server_termination_tests mysql +``` + +The independent actor guards passed twenty invalid-path/marker cases and one intercepted valid-path control. This separate entry starts no server and inspects or signals no real process: + +```sh +bash tests/database/owned_server_fault_actor_guards.sh +``` + +`check-database` compiles the opt-in program without executing its service modes or signaling servers. Compile-only CI success is not another real-service run. + ## Acceptance boundaries -The DSO workflows exercise normal plugin teardown and native close calls. Successful `FreeLibrary` or `dlclose` does not prove that the operating system physically removed the library image. Request cancellation acceptance is limited to the selected server-observed operations on this host. Transaction cancellation, Redis TLS, other cancellation/retry races, server termination, OS image removal, other platforms/deployment environments and cross-machine capacity remain unverified. These records do not establish arbitrary hot-update safety or production capacity. +The DSO workflows exercise normal plugin teardown and native close calls. Successful `FreeLibrary` or `dlclose` does not prove that the operating system physically removed the library image. Request cancellation and server-termination acceptance are limited to the recorded operations on this host. Restart/recovery, persistence and write effects, transaction cancellation, Redis TLS, other cancellation/fault races, OS image removal, other platforms/deployment environments and cross-machine capacity remain unverified. These records do not establish arbitrary hot-update safety or production capacity. diff --git a/docs/guides/database-validation_CN.md b/docs/guides/database-validation_CN.md index 2f1def0cc..e3ea46cd3 100644 --- a/docs/guides/database-validation_CN.md +++ b/docs/guides/database-validation_CN.md @@ -1,6 +1,6 @@ # 数据库服务验收 -本页分别记录公开模块的真实服务正常/故障检查、真实插件 DSO 正常流程,以及经服务端观察确认的 DSO 请求取消。每类检查保留各自的测试版本与环境;通过结果仅适用于对应范围。 +本页分别记录公开模块的真实服务正常/故障检查、插件 DSO 正常流程、经服务端观察确认的请求取消,以及自有服务终止。每类检查保留各自的测试版本与环境;通过结果仅适用于对应范围。 ## 公开模块与真实服务 @@ -94,6 +94,48 @@ cmake --build database-build --target ark_database_public_plugin_cancellation_te `check-database` 仅编译该程序,不运行真实服务模式。两个自有服务均通过认证命令正常关闭;精确进程已独立确认不存在,私有数据、凭据与 TLS 密钥已删除。服务端终止不是本次测试动作。 +## 真实插件服务端终止 + +**2026-10-26**,`ark_database_public_plugin_server_termination_tests` 的两种模式均通过。C++ 宿主与独立故障脚本的退出码均为 `0`;两个自有服务的预期 `SIGKILL` 退出状态均为 `137`。 + +| 环境 | 配置 | +| --- | --- | +| 机器 | Mac16,11 · Apple M4 Pro · 14 核 · 48 GiB | +| 系统 | macOS 27.0.1(26A434)· arm64 | +| 构建 | Apple Clang 21.0.0 · C++20 · Boost.Asio · Debug · 启用 SSL · OpenSSL 3.6.4 | +| 实际执行的 ark | `0722b8bdd2e2c240657a3ceea63428cbbff0f821` | +| arknet | CI 固定版本 `a0dfe09cd543e75cfb5996e537ce2fa49f2a758f` | +| 服务 | Redis 7.4.2:TCP 与密码认证;MySQL Community Server 8.4.13:强制 TLS 与证书校验 | + +宿主使用公开接口和真实隐藏符号插件 DSO。独立服务连接必须先观察到精确匹配的活动请求,才释放私有故障屏障;轮询间隔不能证明请求正在执行。 + +| 模式 | 故障前观察到的只读请求 | 服务终止后的原生结果 | +| --- | --- | --- | +| `redis` | `CLIENT LIST` 确认名称完全匹配的客户端正在 `XREAD` 中阻塞 | 保留 `client_errc::disconnected`、`command_may_have_run=true` 与 EOF 传输状态 | +| `mysql` | `PROCESSLIST` 确认带标记的 `SELECT SLEEP(60)` 连接处于 `User sleep` | 保留 `connection_errc::transport`、`outcome_unknown=true` 与 TLS stream truncation | + +独立 shell 脚本在向单个自有 PID 发送信号前,先按物理路径解析私有文件,拒绝路径逃逸或符号链接协调文件;随后重新核对归属、PID/启动时间、可执行文件、私有数据工作目录、PID 文件及该 PID 的回环监听。MySQL 还核对私有 defaults-file 参数。这些防护仅用于按测试要求准备的一次性实例,不面向共享服务或已安装服务。 + +两个模式均拒绝将任务取消、操作超时、SQL 服务错误或协程异常作为通过结果。请求和观察根任务在所属线程/共享执行器上各完成一次,并在协作式 stop 前退出。受控测试在各自的观察与驱动时限内完成自然停机;重复 stop 成功,模块/管理器/运行时销毁及原生 DSO 关闭后,拥有型结果/错误仍可读取。`manager.stop()` 仍不能中断阻塞钩子,本次不增加强制或抢占式生产期限。原生传输错误细节由平台决定。 + +父进程已回收自有服务、故障脚本与宿主子进程,独立确认精确进程不存在,并删除私有数据、凭据与 TLS 密钥。本次未验证服务重启恢复、持久化或写入效果。 + +按[独立 C++ 服务终止指南](../../tests/database/server_termination_CN.md)准备私有服务、故障脚本协调与端点变量。故障脚本需要 `lsof`、`ps` 和 `rg`。完成启用 SSL 的数据库构建配置后,先按指南准备对应的自有故障脚本,再运行所选模式: + +```sh +cmake --build database-build --target ark_database_public_plugin_server_termination_tests --parallel 1 +./database-build/tests/database/ark_database_public_plugin_server_termination_tests redis +./database-build/tests/database/ark_database_public_plugin_server_termination_tests mysql +``` + +独立脚本防护已通过 20 个无效路径/标记用例及一个有效路径拦截对照。下述独立入口不启动服务,也不检查真实进程或向其发送信号: + +```sh +bash tests/database/owned_server_fault_actor_guards.sh +``` + +`check-database` 仅编译该可选程序,不执行服务模式,也不向服务发送信号。仅编译通过的 CI 不代表另一次真实服务运行。 + ## 验收边界 -DSO 流程覆盖正常插件销毁与原生关闭调用。`FreeLibrary` 或 `dlclose` 成功不证明操作系统已物理移除库映像。本机请求取消验收仅覆盖记录中经服务端观察确认的操作。事务取消、Redis TLS、其他取消/重试竞争、服务端终止、操作系统库映像移除、其他平台/部署环境与跨机器容量仍未验收。这些记录均不证明任意热更新安全或生产容量。 +DSO 流程覆盖正常插件销毁与原生关闭调用。`FreeLibrary` 或 `dlclose` 成功不证明操作系统已物理移除库映像。本机请求取消与服务端终止验收仅覆盖记录中的操作。重启恢复、持久化与写入效果、事务取消、Redis TLS、其他取消/故障竞争、操作系统库映像移除、其他平台/部署环境与跨机器容量仍未验收。这些记录均不证明任意热更新安全或生产容量。 diff --git a/docs/project/roadmap.md b/docs/project/roadmap.md index 5d246c8c2..e082873b8 100644 --- a/docs/project/roadmap.md +++ b/docs/project/roadmap.md @@ -8,13 +8,13 @@ This page distinguishes supported source behavior from remaining design and depl - Public [MySQL](../reference/mysql.md) and [Redis](../reference/redis.md) modules use arknet native clients through Asio coroutine entry points. Inputs and results own their storage; manager-owned task groups resume on the module owner thread. MySQL includes prepared statements and transactions; Redis includes commands and pipelines. Legacy database drivers are no longer the public plugin interface. - Startup callbacks propagate the first failure. The launcher reports the failed phase and exits with code `1` before the update loop. Shutdown stops managed task admission, drains accepted tasks and waits for module readiness. A configured cooperative wait timeout retains the process/resources and disables unloading. These behaviors do not provide partial-startup rollback or a hard process-stop deadline. See [application startup and shutdown](../guides/applications.md) and [managed tasks](../guides/tasks.md). -Public database service/failure checks, [real-plugin DSO positive workflows](../guides/database-validation.md#real-plugin-dsos-against-real-services) and [bounded server-observed request cancellation](../guides/database-validation.md#real-plugin-request-cancellation) passed against isolated Redis and MySQL servers on macOS. The [database service acceptance record](../guides/database-validation.md) keeps their distinct tested revisions, environments and scope; those results do not cover every deployment or source revision. +Public database service/failure checks, [real-plugin DSO positive workflows](../guides/database-validation.md#real-plugin-dsos-against-real-services), [bounded request cancellation](../guides/database-validation.md#real-plugin-request-cancellation) and [observed read-only server termination](../guides/database-validation.md#real-plugin-server-termination) passed against isolated Redis and MySQL servers on macOS. The [database service acceptance record](../guides/database-validation.md) keeps their distinct tested revisions, environments and scope; those results do not cover every deployment or source revision. ## Remaining acceptance and design | Area | Remaining work and acceptance boundary | | --- | --- | -| Database deployment | Transaction cancellation, Redis TLS, other cancellation/retry races, server termination, other platforms and OS library image removal still need acceptance. Add target-environment and cross-machine capacity evidence before making deployment or capacity claims. Local protocol endpoints and construction checks do not replace these runs. | +| Database deployment | Restart/recovery, persistence and write effects, transaction cancellation, Redis TLS, other cancellation/fault races, other platforms and OS library image removal still need acceptance. Add target-environment and cross-machine capacity evidence before making deployment or capacity claims. Local protocol endpoints and construction checks do not replace these runs. | | Startup and shutdown | Partial-startup rollback is not provided. A common dependency-aware shutdown protocol remains design work; managed task drain alone does not prove that arbitrary callbacks, native workers or background cleanup have ended. Define ownership and admission phases before extending unload guarantees. | | Optional plugins | Kafka construction/export checks do not test broker delivery, flush behavior or service interoperability; its driver build needs independent platform acceptance. OSS SDK remains a scaffold without client/request functionality. Physic retains the explicit unsupported legacy PhysX 3 configuration failure. See [optional plugin boundaries](../reference/directory-structure.md). | | C++20 modernization | Review each remaining module and optional integration against its own API, build and runtime requirements. The language baseline, header layout and focused construction checks do not establish completion of the whole modernization. | diff --git a/docs/project/roadmap_CN.md b/docs/project/roadmap_CN.md index c10ed938b..9226ed471 100644 --- a/docs/project/roadmap_CN.md +++ b/docs/project/roadmap_CN.md @@ -8,13 +8,13 @@ - 公开 [MySQL](../reference/mysql_CN.md) 与 [Redis](../reference/redis_CN.md) 模块通过 Asio 协程入口使用 arknet 原生客户端。输入与结果拥有自身存储;管理器拥有的任务组在模块所属线程恢复。MySQL 提供预处理语句与事务,Redis 提供命令与流水线。旧数据库驱动已不再作为公开插件接口。 - 启动回调的首个失败会向上传递。启动器报告失败阶段,在进入更新循环前以退出码 `1` 终止。停止时关闭托管任务接收,排空已接收任务并等待模块就绪。配置的协作式等待超时会保留进程和资源,并禁止卸载。这些行为不提供部分启动回滚,也不构成强制进程退出期限。见[应用启动与停止](../guides/applications_CN.md)和[托管任务](../guides/tasks_CN.md)。 -公开数据库接口的正常/故障检查、[真实插件 DSO 正常流程](../guides/database-validation_CN.md#真实插件动态库与真实服务),以及[经服务端观察确认的有界请求取消](../guides/database-validation_CN.md#真实插件请求取消),已在 macOS 上通过隔离 Redis 与 MySQL 真实服务验收。[数据库服务验收记录](../guides/database-validation_CN.md)分别保留各自的测试版本、环境与范围;这些结果不覆盖所有部署环境或源码版本。 +公开数据库接口的正常/故障检查、[真实插件 DSO 正常流程](../guides/database-validation_CN.md#真实插件动态库与真实服务)、[有界请求取消](../guides/database-validation_CN.md#真实插件请求取消),以及[经观察确认的只读请求服务终止](../guides/database-validation_CN.md#真实插件服务端终止),已在 macOS 上通过隔离 Redis 与 MySQL 真实服务验收。[数据库服务验收记录](../guides/database-validation_CN.md)分别保留各自的测试版本、环境与范围;这些结果不覆盖所有部署环境或源码版本。 ## 剩余验收与设计 | 领域 | 剩余工作与验收边界 | | --- | --- | -| 数据库部署 | 事务取消、Redis TLS、其他取消/重试竞争、服务端终止、其他平台与操作系统库映像移除仍需验收。发布部署或容量结论前,补充目标环境与跨机器容量证据。本机协议端点与构造检查不能替代这些验收。 | +| 数据库部署 | 重启恢复、持久化与写入效果、事务取消、Redis TLS、其他取消/故障竞争、其他平台与操作系统库映像移除仍需验收。发布部署或容量结论前,补充目标环境与跨机器容量证据。本机协议端点与构造检查不能替代这些验收。 | | 启动与停机 | 框架不提供部分启动回滚。通用的依赖感知停机协议仍需设计;托管任务排空不代表任意回调、原生工作线程或后台清理均已结束。扩展卸载保证前,先明确所有权与接收阶段。 | | 可选插件 | Kafka 构造/导出检查不验证 broker 投递、flush 行为或服务互通;驱动构建仍需独立平台验收。OSS SDK 保持骨架状态,不提供客户端/请求功能。Physic 保留旧版 PhysX 3 不受支持的显式配置失败。见[可选插件边界](../reference/directory-structure_CN.md)。 | | C++20 改造 | 按各模块与可选集成的 API、构建和运行要求继续审查。语言基线、头文件布局和局部构造检查不足以证明全部改造完成。 | diff --git a/tests/database/CMakeLists.txt b/tests/database/CMakeLists.txt index 1d8271e16..230ed44b9 100644 --- a/tests/database/CMakeLists.txt +++ b/tests/database/CMakeLists.txt @@ -70,6 +70,13 @@ set_property(TARGET ark_database_public_plugin_cancellation_tests PROPERTY LINK_ target_compile_definitions(ark_database_public_plugin_cancellation_tests PRIVATE ARK_DATABASE_PLUGIN_DIR="$/") add_dependencies(ark_database_public_plugin_cancellation_tests ark_database_redis_plugin) +# Explicit owned-server fault acceptance; never execute this program in check-database. +ark_database_test(ark_database_public_plugin_server_termination_tests public_plugin_server_termination.cpp) +set_property(TARGET ark_database_public_plugin_server_termination_tests PROPERTY INCLUDE_DIRECTORIES "") +set_property(TARGET ark_database_public_plugin_server_termination_tests PROPERTY LINK_LIBRARIES ark::redis Threads::Threads) +target_compile_definitions(ark_database_public_plugin_server_termination_tests PRIVATE + ARK_DATABASE_PLUGIN_DIR="$/") +add_dependencies(ark_database_public_plugin_server_termination_tests ark_database_redis_plugin) # Compile and load each sample's real module and plugin assembly with a local # provider that rejects configuration before coroutine or network work. function(ark_database_sample_check database) @@ -116,7 +123,7 @@ add_custom_target(check-database # Compile real-service acceptance in ordinary CI without running any server work. add_dependencies(check-database ark_database_redis_sample_check ark_database_public_service_tests ark_database_public_failure_service_tests ark_database_public_plugin_service_tests - ark_database_public_plugin_cancellation_tests check-database-samples) + ark_database_public_plugin_cancellation_tests ark_database_public_plugin_server_termination_tests check-database-samples) if(ARKNET_ENABLE_SSL) target_link_libraries(ark_database_public_failure_service_tests PRIVATE ark::mysql) @@ -150,6 +157,8 @@ if(ARKNET_ENABLE_SSL) target_link_libraries(ark_database_public_plugin_service_tests PRIVATE ark::mysql) target_link_libraries(ark_database_public_plugin_cancellation_tests PRIVATE ark::mysql) add_dependencies(ark_database_public_plugin_cancellation_tests ark_database_mysql_plugin) + target_link_libraries(ark_database_public_plugin_server_termination_tests PRIVATE ark::mysql) + add_dependencies(ark_database_public_plugin_server_termination_tests ark_database_mysql_plugin) ark_database_sample_check(mysql) add_custom_target(check-database-samples-mysql COMMAND $ diff --git a/tests/database/README.md b/tests/database/README.md index 3a75fd3a7..87f13f2dd 100644 --- a/tests/database/README.md +++ b/tests/database/README.md @@ -355,3 +355,10 @@ cmake --build database-build --target check-database-samples --parallel 2 Each runner loads the actual DSO through the manager's normal entry and exit symbols. A fixture registers the public database interface and rejects its first configuration request. The sample must resolve that interface, preserve its `primary` (MySQL) or `sample` (Redis) connection name, and report a controlled init failure whose diagnostic names the concrete sample module. The real plugin exit must return successfully, the manager must reach its stopped state, and repeated stop must succeed. The runner uses public manager APIs; it does not inspect the module registry after shutdown or independently prove its erased state. Compile-time checks preserve typed bases and inherited default update behavior; assembly explicitly selects `module_update::disabled`. These tests replace module-only compilation: plugin entry/exit sources are required dependencies even when application/examples are disabled. They preserve original source licenses and do not enable other examples. The rejecting provider does not run sample coroutines or contact services; live MySQL/Redis workflow and deployment acceptance remain separate. + +## Owned-server termination acceptance + +The opt-in [public-plugin server-termination guide](server_termination.md) covers +server-observed blocking read-only work, a separately verified owned-PID fault +actor, native transport failure and exactly-once local retirement. Ordinary +checks compile the program without executing a server fault. diff --git a/tests/database/README_CN.md b/tests/database/README_CN.md index 8506d3643..52c73dcb5 100644 --- a/tests/database/README_CN.md +++ b/tests/database/README_CN.md @@ -283,3 +283,9 @@ cmake --build database-build --target check-database-samples --parallel 2 每个 runner 都通过管理器的标准入口与退出符号加载真实 DSO。fixture 注册公开数据库接口,并拒绝第一次配置请求。示例必须查找到该接口,保留 `primary`(MySQL)或 `sample`(Redis)连接名,并报告受控的 init 失败,诊断中包含具体示例模块名。真实插件退出必须成功返回,管理器必须进入 stopped 状态,重复 stop 必须成功。runner 仅使用管理器公开 API,不在停机后检查模块注册表,也不独立证明注册表条目已被移除。编译期检查保留类型基类与继承的默认 update 行为;装配显式使用 `module_update::disabled`。 这些测试替代仅编译 module.cpp 的检查:即使应用与示例关闭,插件入口/退出源文件仍是必需依赖。保留源码原有许可证,不启用其他示例。拒绝配置的 provider 不运行示例协程,也不连接服务;真实 MySQL/Redis 业务流程与部署仍需单独验收。 + +## 自有服务异常退出验收 + +可选的[公开插件服务异常退出指南](server_termination_CN.md)覆盖真实服务观察到的只读阻塞请求、 +独立核验自有 PID 后的故障注入、原生传输失败及本地仅完成一次的退出。 +常规检查只编译程序,不执行服务故障。 diff --git a/tests/database/owned_server_fault_actor.sh b/tests/database/owned_server_fault_actor.sh new file mode 100755 index 000000000..132da0247 --- /dev/null +++ b/tests/database/owned_server_fault_actor.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +# Explicitly opt-in actor for the public-plugin server-termination program. +# Requires bash, lsof, ps, rg and the fresh private service layout described in the guide. +set -euo pipefail +umask 077 +[[ $# == 7 ]] || { echo 'Usage: owned_server_fault_actor.sh redis|mysql PID executable config datadir pidfile port' >&2; exit 2; } +mode=$1 process_id=$2 executable=$3 config=$4 datadir=$5 pidfile=$6 service_port=$7 +[[ "$mode" == redis || "$mode" == mysql ]] || exit 1 +[[ "$process_id" =~ ^[0-9]+$ && "$process_id" -gt 1 && "$service_port" =~ ^[0-9]+$ ]] || exit 1 +: "${ARK_DATABASE_FAULT_DIR:?}" "${ARK_DATABASE_FAULT_TOKEN:?}" +private_root=$ARK_DATABASE_FAULT_DIR +token=$ARK_DATABASE_FAULT_TOKEN +# Resolve directory aliases and every existing path's parent components before +# inspecting any process. Leaf symlinks are refused rather than followed. +canonical_path() { + local candidate=$1 + [[ -e "$candidate" && ! -L "$candidate" ]] || return 1 + if [[ -d "$candidate" ]]; then + (cd -P -- "$candidate" && pwd -P) + else + (cd -P -- "$(dirname "$candidate")" && printf '%s/%s\n' "$(pwd -P)" "$(basename "$candidate")") + fi +} +[[ "$private_root" == /tmp/ark-dso-termination-owned.* && -d "$private_root" && ! -L "$private_root" ]] || exit 1 +canonical_tmp=$(cd -P /tmp && pwd -P) +canonical_root=$(canonical_path "$private_root") || exit 1 +[[ "$(dirname "$canonical_root")" == "$canonical_tmp" && "$(basename "$canonical_root")" == ark-dso-termination-owned.* ]] || exit 1 +canonical_config=$(canonical_path "$config") || exit 1 +canonical_datadir=$(canonical_path "$datadir") || exit 1 +canonical_pidfile=$(canonical_path "$pidfile") || exit 1 +verify_owned_paths() { + [[ "$(canonical_path "$private_root")" == "$canonical_root" ]] || return 1 + local candidate resolved + for candidate in "$config" "$datadir" "$pidfile"; do + resolved=$(canonical_path "$candidate") || return 1 + [[ "$resolved" == "$canonical_root/"* ]] || return 1 + done + [[ "$(canonical_path "$config")" == "$canonical_config" && + "$(canonical_path "$datadir")" == "$canonical_datadir" && + "$(canonical_path "$pidfile")" == "$canonical_pidfile" ]] || return 1 + [[ -f "$config" && -d "$datadir" && -f "$pidfile" ]] || return 1 + # These files are direct children of the canonical root. Reject even a + # dangling link, including stale output markers that redirection could follow. + for candidate in ownership "$mode.identity" "$mode.ready" "$mode.ready.tmp" "$mode.fired" "$mode.fired.tmp"; do + [[ ! -L "$canonical_root/$candidate" ]] || return 1 + if [[ -e "$canonical_root/$candidate" ]]; then + [[ -f "$canonical_root/$candidate" ]] || return 1 + fi + done + [[ -f "$canonical_root/ownership" && -f "$canonical_root/$mode.identity" && + ! -e "$canonical_root/$mode.fired" && ! -e "$canonical_root/$mode.fired.tmp" ]] || return 1 +} +verify_owned_paths || exit 1 +[[ "$token" =~ ^[0-9a-f]{64}$ && "$(cat "$canonical_root/ownership")" == "$token" ]] || exit 1 +canonical_executable=$(cd -P -- "$(dirname "$executable")" && printf '%s/%s' "$(pwd -P)" "$(basename "$executable")") +identity=$canonical_root/$mode.identity +[[ -f "$identity" && ! -L "$identity" ]] || exit 1 +[[ "$(sed -n '1p' "$identity")" == "$token" && "$(sed -n '2p' "$identity")" == "$process_id" ]] || exit 1 +[[ "$(sed -n '3p' "$identity")" == "$canonical_executable" && "$(sed -n '4p' "$identity")" == "$datadir" ]] || exit 1 +expected_start=$(sed -n '5p' "$identity") +[[ -n "$expected_start" ]] || exit 1 +if [[ "$mode" == redis ]]; then + rg -Fx "dir $datadir" "$config" >/dev/null + rg -Fx "pidfile $pidfile" "$config" >/dev/null +else + rg -Fx "datadir=$datadir" "$config" >/dev/null + rg -Fx "pid-file=$pidfile" "$config" >/dev/null +fi +verify_identity() { + verify_owned_paths || return 1 + [[ "$(cat "$canonical_root/ownership")" == "$token" && "$(cat "$pidfile")" == "$process_id" ]] || return 1 + kill -0 "$process_id" 2>/dev/null || return 1 + [[ "$(ps -p "$process_id" -o lstart=)" == "$expected_start" ]] || return 1 + lsof -nP -a -p "$process_id" -d txt -Fn 2>/dev/null | rg -Fx "n$canonical_executable" >/dev/null || return 1 + # Both owned servers chdir to their exclusive data directory. This also + # validates Redis after it replaces its original argv with a process title. + lsof -nP -a -p "$process_id" -d cwd -Fn 2>/dev/null | rg -Fx "n$canonical_datadir" >/dev/null || return 1 + lsof -nP -a -p "$process_id" -iTCP:"$service_port" -sTCP:LISTEN 2>/dev/null \ + | rg -F "127.0.0.1:$service_port" >/dev/null || return 1 + if [[ "$mode" == mysql ]]; then + ps -p "$process_id" -o command= | rg -F -- "--defaults-file=$config" >/dev/null || return 1 + fi +} +verify_identity || exit 1 +ready=$canonical_root/$mode.ready +for ((step=0; step<150; ++step)); do + verify_owned_paths || exit 1 + if [[ -f "$ready" ]]; then + [[ "$(sed -n '1p' "$ready")" == "$token" && "$(sed -n '2p' "$ready")" =~ ^[0-9]+$ ]] || exit 1 + verify_identity || exit 1 + printf '%s server-active id=%s; verified owned pid=%s; injecting SIGKILL\n' \ + "$mode" "$(sed -n '2p' "$ready")" "$process_id" + kill -KILL "$process_id" + (set -o noclobber; printf '%s\n%s\nSIGKILL\n' "$token" "$process_id" > "$canonical_root/$mode.fired.tmp") + mv "$canonical_root/$mode.fired.tmp" "$canonical_root/$mode.fired" + exit 0 + fi + # Poll throttling, never evidence that server work is active. + sleep 0.1 +done +echo 'No authenticated server-active barrier; refusing to signal any process.' >&2 +exit 1 diff --git a/tests/database/owned_server_fault_actor_guards.sh b/tests/database/owned_server_fault_actor_guards.sh new file mode 100755 index 000000000..70be24803 --- /dev/null +++ b/tests/database/owned_server_fault_actor_guards.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# Non-signaling path-boundary regressions: no server is started or inspected. +set -euo pipefail +umask 077 +actor=${1:-"$(cd "$(dirname "$0")" && pwd -P)/owned_server_fault_actor.sh"} +fixture_root= fixture_outside= +cleanup() { + [[ -z "$fixture_root" ]] || rm -r -- "$fixture_root" + [[ -z "$fixture_outside" ]] || rm -r -- "$fixture_outside" +} +trap cleanup EXIT + +check_case() { + local scenario=$1 actor_root datadir config pidfile token canonical_executable status=0 + fixture_root=$(mktemp -d /tmp/ark-dso-termination-owned.guard.XXXXXX) + fixture_outside=$(mktemp -d /tmp/ark-dso-termination-guard-outside.XXXXXX) + actor_root=$fixture_root + datadir=$fixture_root/data + config=$fixture_root/config + pidfile=$fixture_root/pid + token=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef + mkdir "$datadir" "$fixture_outside/data" "$fixture_outside/inner" + canonical_executable=$(cd /bin && printf '%s/sleep' "$(pwd -P)") + printf '%s\n' "$token" > "$fixture_root/ownership" + printf '%s\n' "$token" > "$fixture_outside/ownership" + # The fake PID is never used to inspect or signal a real process. BASH_ENV + # intercepts process/signal/wait commands; every unexpected call fails fast. + printf '2147483647\n' > "$pidfile" + printf '2147483647\n' > "$fixture_outside/pid" + ln -s "$fixture_outside" "$fixture_root/escape" + ln -s "$fixture_outside/data" "$fixture_root/escape_data" + case "$scenario" in + config_traversal) config=$fixture_root/../$(basename "$fixture_outside")/config ;; + datadir_traversal) datadir=$fixture_root/../$(basename "$fixture_outside")/data ;; + pidfile_traversal) pidfile=$fixture_root/../$(basename "$fixture_outside")/pid ;; + config_parent_symlink) config=$fixture_root/escape/config ;; + datadir_parent_symlink) datadir=$fixture_root/escape/data ;; + pidfile_parent_symlink) pidfile=$fixture_root/escape/pid ;; + config_symlink_dotdot) config=$fixture_root/escape_data/../config ;; + datadir_symlink_dotdot) datadir=$fixture_root/escape_data/../data ;; + pidfile_symlink_dotdot) pidfile=$fixture_root/escape_data/../pid ;; + root_parent_symlink) + actor_root=$fixture_root/escape/inner + datadir=$actor_root/data; config=$actor_root/config; pidfile=$actor_root/pid + mkdir "$datadir" + printf '%s\n' "$token" > "$actor_root/ownership" + printf '2147483647\n' > "$pidfile" + ;; + esac + printf 'dir %s\npidfile %s\n' "$datadir" "$pidfile" > "$config" + printf '%s\n2147483647\n%s\n%s\nfake-start-time\n' "$token" "$canonical_executable" "$datadir" \ + > "$actor_root/redis.identity" + case "$scenario" in + config_leaf_symlink) + cp "$config" "$fixture_outside/config"; rm "$config"; ln -s "$fixture_outside/config" "$config" ;; + datadir_leaf_symlink) + rmdir "$datadir"; ln -s "$fixture_outside/data" "$datadir" ;; + pidfile_leaf_symlink) + rm "$pidfile"; ln -s "$fixture_outside/pid" "$pidfile" ;; + ownership_symlink) + rm "$fixture_root/ownership"; ln -s "$fixture_outside/ownership" "$fixture_root/ownership" ;; + identity_symlink) + mv "$fixture_root/redis.identity" "$fixture_outside/identity" + ln -s "$fixture_outside/identity" "$fixture_root/redis.identity" ;; + ready_symlink) ln -s "$fixture_outside/missing" "$fixture_root/redis.ready" ;; + ready_tmp_symlink) ln -s "$fixture_outside/missing" "$fixture_root/redis.ready.tmp" ;; + fired_symlink) ln -s "$fixture_outside/missing" "$fixture_root/redis.fired" ;; + fired_tmp_symlink) ln -s "$fixture_outside/missing" "$fixture_root/redis.fired.tmp" ;; + stale_fired) printf 'stale\n' > "$fixture_root/redis.fired" ;; + esac + cat > "$fixture_outside/intercept.sh" <<'INTERCEPT' +kill() { printf 'kill\n' >> "$ARK_ACTOR_GUARD_CALLS"; return 99; } +ps() { printf 'ps\n' >> "$ARK_ACTOR_GUARD_CALLS"; return 99; } +lsof() { printf 'lsof\n' >> "$ARK_ACTOR_GUARD_CALLS"; return 99; } +sleep() { printf 'sleep\n' >> "$ARK_ACTOR_GUARD_CALLS"; return 99; } +INTERCEPT + if ARK_DATABASE_FAULT_DIR=$actor_root ARK_DATABASE_FAULT_TOKEN=$token \ + ARK_ACTOR_GUARD_CALLS=$fixture_outside/calls BASH_ENV=$fixture_outside/intercept.sh \ + bash "$actor" redis 2147483647 "$canonical_executable" "$config" "$datadir" "$pidfile" 49999 \ + > "$fixture_outside/actor.log" 2>&1; then :; else status=$?; fi + if [[ "$scenario" == control ]]; then + [[ $status != 0 && "$(cat "$fixture_outside/calls")" == kill ]] || return 1 + printf 'valid private paths reach the intercepted process check; no process inspected or signaled\n' + else + if [[ $status == 0 || -e "$fixture_outside/calls" ]]; then + printf '%s escaped the pre-inspection path guard (actor exit %s)\n' "$scenario" "$status" >&2 + [[ ! -f "$fixture_outside/calls" ]] || cat "$fixture_outside/calls" >&2 + return 1 + fi + [[ ! -f "$fixture_root/redis.fired" || "$scenario" == stale_fired ]] || return 1 + printf '%s rejected before process inspection/signaling\n' "$scenario" + fi + cleanup + fixture_root= fixture_outside= +} + +check_case control +for scenario in config_traversal datadir_traversal pidfile_traversal \ + config_parent_symlink datadir_parent_symlink pidfile_parent_symlink root_parent_symlink \ + config_symlink_dotdot datadir_symlink_dotdot pidfile_symlink_dotdot \ + config_leaf_symlink datadir_leaf_symlink pidfile_leaf_symlink ownership_symlink identity_symlink \ + ready_symlink ready_tmp_symlink fired_symlink fired_tmp_symlink stale_fired; do + check_case "$scenario" +done +printf '20 invalid-path/marker guards and one non-signaling control passed\n' diff --git a/tests/database/public_plugin_server_termination.cpp b/tests/database/public_plugin_server_termination.cpp new file mode 100644 index 000000000..916edb8f8 --- /dev/null +++ b/tests/database/public_plugin_server_termination.cpp @@ -0,0 +1,501 @@ +// Copyright (c) 2026 OpenArkStudio. Licensed under Apache-2.0. See LICENSE. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ark/plugin_manager.hpp" +#include "ark/redis/redis_module.hpp" +#include "ark/task.hpp" + +#if defined(ARKNET_ENABLE_SSL) +#include "ark/mysql/mysql_module.hpp" +#endif + +namespace { + +using namespace std::chrono_literals; + +class acceptance_failure : public std::runtime_error +{ +public: + using std::runtime_error::runtime_error; +}; + +void require(bool condition, const char* message) { + if (!condition) + { + throw acceptance_failure(message); + } +} + +std::string environment(const char* name, bool allow_empty = false) { + const auto* value = std::getenv(name); + require(value && (allow_empty || *value), "Missing database termination acceptance environment"); + return value; +} + +std::uint16_t port(const char* name) { + const auto text = environment(name); + unsigned number{}; + const auto parsed = std::from_chars(text.data(), text.data() + text.size(), number); + require(parsed.ec == std::errc{} && parsed.ptr == text.data() + text.size() && number && number <= 65535, + "Database termination port must be an integer between 1 and 65535"); + return static_cast(number); +} + +struct completion +{ + unsigned count{}; + bool on_owner{}; + std::exception_ptr error; +}; + +// Public interfaces only: no implementation/factory object is linked into this host. +class plugin_host +{ +public: + ark::plugin_manager manager; + ark::task_group* tasks{}; + + explicit plugin_host(const char* library) { + ark::logger::init(nullptr); + ark::logger::instance()->set_log_func([](spdlog::level::level_enum, const std::string&) {}); + require(manager.start(0, ARK_DATABASE_PLUGIN_DIR, "", {{library, ""}}), "Database DSO startup failed"); + require(manager.add_module("termination_consumer", &consumer_), "Cannot register termination consumer"); + runtime_ = manager.get_task_runtime(); + tasks = &manager.create_task_group(consumer_); + } + + ~plugin_host() { + if (!stopped_) + { + try + { stop(); } + catch (...) + { + std::cerr << "Database termination acceptance could not naturally drain\n" << std::flush; + std::_Exit(1); + } + } + } + + void check_executor() const { + require(std::this_thread::get_id() == owner_ && runtime_->get_executor().running_in_this_thread(), + "DSO termination continuation escaped owner thread/shared executor"); + } + + ark::task_group::task_id start(ark::task<> workflow, completion& record) { + const auto id = tasks->start(std::move(workflow), [this, &record](std::exception_ptr error) { + record.on_owner = std::this_thread::get_id() == owner_ && runtime_->get_executor().running_in_this_thread(); + record.error = std::move(error); + ++record.count; + }); + require(id != 0, "Database termination workflow admission failed"); + return id; + } + + static void check_completion(const completion& record) { + if (record.error) + { + std::rethrow_exception(record.error); + } + require(record.count == 1 && record.on_owner, "DSO root completion repeated or escaped its owner"); + } + + void pump_until(const std::function& done) { + const auto deadline = std::chrono::steady_clock::now() + 25s; + while (!done()) + { + require(std::chrono::steady_clock::now() < deadline, "Database termination acceptance exceeded bound"); + require(manager.update(0), "Database termination owner update failed"); + // Throttling only; server-visible state establishes termination readiness. + std::this_thread::sleep_for(1ms); + } + } + + void preflight(ark::task<> workflow) { + auto& record = preflight_completion_; + start(std::move(workflow), record); + pump_until([&] { return record.count != 0; }); + check_completion(record); + require(tasks->pending() == 0, "Preflight did not retire naturally"); + } + + void stop() { + const auto deadline = std::chrono::steady_clock::now() + 30s; + while (!(stopped_ = manager.stop(0))) + { + require(manager.get_shutdown_state() != ark::plugin_manager::shutdown_state::failed && + std::chrono::steady_clock::now() < deadline, + "Interrupted DSO requests did not naturally retire within the shutdown bound"); + require(manager.update(0), "Database termination drain update failed"); + std::this_thread::sleep_for(1ms); + } + require(manager.get_shutdown_state() == ark::plugin_manager::shutdown_state::stopped, + "Database termination shutdown did not reach stopped"); + require(tasks->closed() && tasks->pending() == 0 && manager.pending_tasks() == 0, + "DSO termination stop retained pending managed tasks"); + require(manager.stop(0), "Repeated database termination stop was not idempotent"); + manager.remove_module("termination_consumer"); + } + +private: + completion preflight_completion_; + ark::module consumer_; + std::shared_ptr runtime_; + const std::thread::id owner_{std::this_thread::get_id()}; + bool stopped_{}; +}; + +ark::task<> observer_interval() { + asio::steady_timer timer(co_await asio::this_coro::executor); + timer.expires_after(20ms); + co_await timer.async_wait(asio::use_awaitable); +} + +// The shell actor owns the service and signal authority. This host only publishes +// an observed server-side request ID through a fresh, private token barrier. +class fault_barrier +{ +public: + explicit fault_barrier(std::string protocol) + : directory_(environment("ARK_DATABASE_FAULT_DIR")) + , token_(environment("ARK_DATABASE_FAULT_TOKEN")) + , protocol_(std::move(protocol)) { + require(directory_.is_absolute() && std::filesystem::is_directory(directory_), "Fault directory must exist"); + require(token_.size() == 64 && token_.find_first_not_of("0123456789abcdef") == std::string::npos, + "Fault token must be a fresh 256-bit hexadecimal value"); + require(read(directory_ / "ownership") == token_, "Private fault ownership marker did not match"); + require(!std::filesystem::exists(path("ready")) && !std::filesystem::exists(path("fired")), + "Fault barrier must not reuse prior markers"); + } + + void observed(const std::string& id) { + require(!announced_ && !id.empty(), "Fault barrier repeated or lost server request identity"); + const auto temporary = path("ready.tmp"); + { + std::ofstream output(temporary); + output << token_ << '\n' << id << '\n'; + output.close(); + require(bool(output), "Cannot publish server-active fault marker"); + } + std::filesystem::rename(temporary, path("ready")); + announced_ = true; + std::cout << protocol_ << " server-observed active request id=" << id << "; fault barrier published\n" + << std::flush; + } + + bool fired() const { + return announced_ && std::filesystem::exists(path("fired")) && read(path("fired")) == token_; + } + +private: + static std::string read(const std::filesystem::path& path) { + std::ifstream input(path); + std::string line; + std::getline(input, line); + require(bool(input), "Cannot read private fault marker"); + return line; + } + + std::filesystem::path path(const char* suffix) const { + return directory_ / (protocol_ + "." + suffix); + } + + std::filesystem::path directory_; + std::string token_, protocol_; + bool announced_{}; +}; + +void redis_success(const arknet::redis::result& result) { + require(bool(result) && result.replies.size() == 1, "Explicit Redis observer/recovery operation failed"); +} + +std::string client_field(std::string_view line, std::string_view name) { + const auto marker = std::string(name) + "="; + for (std::size_t position{}; position < line.size();) + { + const auto end = line.find(' ', position); + const auto field = line.substr(position, end == std::string_view::npos ? end : end - position); + if (field.starts_with(marker)) + { + return std::string(field.substr(marker.size())); + } + if (end == std::string_view::npos) + { + break; + } + position = end + 1; + } + return {}; +} + +std::optional blocked_client(const arknet::redis::result& listing, const std::string& tag) { + redis_success(listing); + const auto& reply = listing.replies[0]; + require(reply.type == arknet::redis::value_type::bulk_string || + reply.type == arknet::redis::value_type::verbatim_string, + "Redis CLIENT LIST result was not text"); + std::string_view remaining = reply.text; + if (reply.type == arknet::redis::value_type::verbatim_string) + { + require(remaining.starts_with("txt:"), "Redis CLIENT LIST verbatim result had an unexpected format"); + remaining.remove_prefix(4); + } + std::optional found; + while (!remaining.empty()) + { + const auto end = remaining.find('\n'); + const auto line = remaining.substr(0, end); + if (client_field(line, "name") == tag && client_field(line, "cmd") == "xread" && + client_field(line, "flags").find('b') != std::string::npos) + { + require(!found, "Redis observer saw concurrent replay of its tagged read-only command"); + found = client_field(line, "id"); + require(!found->empty(), "Redis observer could not identify the blocked client"); + } + if (end == std::string_view::npos) + { + break; + } + remaining.remove_prefix(end + 1); + } + return found; +} + +ark::task<> redis_preflight(plugin_host& host, ark::redis_module& database, const std::string& tag) { + arknet::redis::command exists{"EXISTS", tag}; + auto absent = co_await database.command("observer", std::move(exists), 5s); + host.check_executor(); + redis_success(absent); + require(absent.replies[0].type == arknet::redis::value_type::integer && absent.replies[0].integer == 0, + "Exclusive Redis termination stream already exists"); + arknet::redis::command ping{"PING"}; + auto ready = co_await database.command("primary", std::move(ping), 5s); + host.check_executor(); + redis_success(ready); + require(ready.replies[0].text == "PONG", "Redis termination preflight failed"); +} + +ark::task<> redis_request( + plugin_host& host, ark::redis_module& database, std::string tag, std::optional& result) { + arknet::redis::command command{"XREAD", "BLOCK", "60000", "STREAMS", tag, "$"}; + result = co_await database.command("primary", std::move(command), 20s); + host.check_executor(); +} + +ark::task<> redis_observe(plugin_host& host, ark::redis_module& database, fault_barrier& barrier, + const completion& request, const std::optional& result, std::string tag) { + const auto deadline = std::chrono::steady_clock::now() + 10s; + for (;;) + { + require(!request.count && !result && std::chrono::steady_clock::now() < deadline, + "Redis request retired before server-active observation or observation exceeded bound"); + arknet::redis::command clients{"CLIENT", "LIST"}; + auto listing = co_await database.command("observer", std::move(clients), 3s); + host.check_executor(); + if (const auto id = blocked_client(listing, tag)) + { + require(!request.count && !result, "Redis request completed before fault readiness"); + barrier.observed(*id); + co_return; + } + co_await observer_interval(); + host.check_executor(); + } +} + +void redis_failure(const arknet::redis::result& result) { + require(result.error == arknet::redis::client_errc::disconnected && result.command_may_have_run && + result.transport_error && result.transport_error != asio::error::operation_aborted && + result.transport_error != asio::error::timed_out && result.replies.empty(), + "Redis server termination did not preserve disconnected/sent native transport failure"); +} + +void check_redis() { + fault_barrier barrier("redis"); + ark::redis_options options; + options.pool.connection.host = environment("ARK_DATABASE_REDIS_HOST"); + require(options.pool.connection.host == "127.0.0.1", "Redis termination requires an owned IPv4 loopback service"); + options.pool.connection.port = std::to_string(port("ARK_DATABASE_REDIS_PORT")); + options.pool.connection.password = environment("ARK_DATABASE_REDIS_PASSWORD"); + options.pool.max_connections = 1; + options.pool.client_configuration.connect_timeout = 3s; + const auto tag = + "ark_dso_termination_" + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count()); + options.pool.connection.client_name = tag; + completion request, observer; + std::optional result; + { + plugin_host host("ark_database_redis_plugin"); + auto* database = host.manager.find_module(); + require(database && database->configure("primary", options), "Cannot configure Redis termination primary"); + options.pool.connection.client_name = tag + "_observer"; + require(database->configure("observer", options), "Cannot configure Redis termination observer"); + host.preflight(redis_preflight(host, *database, tag)); + const auto request_id = host.start(redis_request(host, *database, tag, result), request); + host.start(redis_observe(host, *database, barrier, request, result, tag), observer); + host.pump_until([&] { return request.count && observer.count && barrier.fired(); }); + plugin_host::check_completion(observer); + plugin_host::check_completion(request); + require(result.has_value(), "Redis server failure lost native result"); + redis_failure(*result); + std::cout << "redis native server failure code=" << static_cast(result->error) + << " may_have_run=" << result->command_may_have_run + << " transport=" << result->transport_error.value() + << " message=" << result->transport_error.message() << '\n'; + require(host.tasks->pending() == 0 && host.manager.pending_tasks() == 0, + "Redis request did not naturally retire before stop"); + require(!host.tasks->cancel(request_id), "Retired Redis request still accepted cancellation"); + host.stop(); + require(request.count == 1 && observer.count == 1, "Redis completion repeated during shutdown"); + } + redis_failure(*result); + require(!result->transport_error.message().empty(), "Redis owned error lost lifetime after DSO unload"); +} + +#if defined(ARKNET_ENABLE_SSL) +ark::task<> mysql_preflight(plugin_host& host, ark::mysql_module& database) { + for (const auto* name : {"primary", "observer"}) + { + auto ready = co_await database.query(name, "SELECT 1", 5s); + host.check_executor(); + require(bool(ready), "MySQL termination preflight failed"); + } +} + +ark::task<> mysql_request( + plugin_host& host, ark::mysql_module& database, std::string tag, std::optional& result) { + result = co_await database.query("primary", "SELECT SLEEP(60) /*" + tag + "*/", 20s); + host.check_executor(); +} + +ark::task<> mysql_observe(plugin_host& host, ark::mysql_module& database, fault_barrier& barrier, + const completion& request, const std::optional& result, std::string tag) { + const auto deadline = std::chrono::steady_clock::now() + 10s; + const auto sql = "SELECT CAST(ID AS SIGNED), STATE FROM information_schema.processlist " + "WHERE ID <> CONNECTION_ID() AND INFO LIKE '%" + + tag + "%'"; + for (;;) + { + require(!request.count && !result && std::chrono::steady_clock::now() < deadline, + "MySQL request retired before server-active observation or observation exceeded bound"); + auto listing = co_await database.query("observer", sql, 3s); + host.check_executor(); + require(bool(listing) && listing.result.rows.size() <= 1, "MySQL independent active observer failed"); + if (!listing.result.rows.empty()) + { + require(listing.result.rows[0].size() == 2, "MySQL observer lost its ID/state fields"); + const auto* phase = std::get_if(&listing.result.rows[0][1]); + if (phase && *phase == "User sleep") + { + const auto id = std::get(listing.result.rows[0][0]); + require(!request.count && !result, "MySQL request completed before fault readiness"); + barrier.observed(std::to_string(id)); + co_return; + } + } + co_await observer_interval(); + host.check_executor(); + } +} + +void mysql_failure(const ark::mysql_reply& result) { + const auto& status = result.result.status; + require(status.code == arknet::mysql::connection_errc::transport && status.outcome_unknown && + status.transport_error && status.transport_error != asio::error::operation_aborted && + status.transport_error != asio::error::timed_out && !status.server_code, + "MySQL server termination did not preserve transport/unknown native failure"); +} + +void check_mysql() { + fault_barrier barrier("mysql"); + ark::mysql_options options; + options.pool.connection.host = environment("ARK_DATABASE_MYSQL_HOST"); + require(options.pool.connection.host == "127.0.0.1", "MySQL termination requires an owned IPv4 loopback service"); + options.pool.connection.port = port("ARK_DATABASE_MYSQL_PORT"); + options.pool.connection.username = environment("ARK_DATABASE_MYSQL_USER"); + options.pool.connection.password = environment("ARK_DATABASE_MYSQL_PASSWORD"); + options.pool.connection.database = environment("ARK_DATABASE_MYSQL_DATABASE"); + options.pool.connection.ca_file = environment("ARK_DATABASE_MYSQL_CA"); + options.pool.connection.tls = arknet::mysql::tls_mode::required; + options.pool.connection.timeout = 5s; + options.pool.max_connections = 1; + const auto tag = + "ark_dso_termination_" + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count()); + completion request, observer; + std::optional result; + { + plugin_host host("ark_database_mysql_plugin"); + auto* database = host.manager.find_module(); + require(database && database->configure("primary", options), "Cannot configure MySQL termination primary"); + require(database->configure("observer", options), "Cannot configure MySQL termination observer"); + host.preflight(mysql_preflight(host, *database)); + const auto request_id = host.start(mysql_request(host, *database, tag, result), request); + host.start(mysql_observe(host, *database, barrier, request, result, tag), observer); + host.pump_until([&] { return request.count && observer.count && barrier.fired(); }); + plugin_host::check_completion(observer); + plugin_host::check_completion(request); + require(result.has_value(), "MySQL server failure lost native result"); + mysql_failure(*result); + const auto& status = result->result.status; + std::cout << "mysql native server failure code=" << static_cast(status.code) + << " outcome_unknown=" << status.outcome_unknown << " transport=" << status.transport_error.value() + << " message=" << status.transport_error.message() << '\n'; + require(host.tasks->pending() == 0 && host.manager.pending_tasks() == 0, + "MySQL request did not naturally retire before stop"); + require(!host.tasks->cancel(request_id), "Retired MySQL request still accepted cancellation"); + host.stop(); + require(request.count == 1 && observer.count == 1, "MySQL completion repeated during shutdown"); + } + mysql_failure(*result); + require( + !result->result.status.transport_error.message().empty(), "MySQL owned error lost lifetime after DSO unload"); +} +#endif + +} // namespace + +int main(int argc, char** argv) { + try + { + require(argc == 2, "Usage: ark_database_public_plugin_server_termination_tests redis|mysql"); + const std::string_view protocol(argv[1]); + if (protocol == "redis") + { + check_redis(); + } +#if defined(ARKNET_ENABLE_SSL) + else if (protocol == "mysql") + { + check_mysql(); + } +#endif + else + { + throw acceptance_failure("Requested server termination protocol is not enabled"); + } + std::cout << protocol << " real DSO server termination, owner completion and natural retirement passed\n"; + return 0; + } + catch (const acceptance_failure& error) + { std::cerr << error.what() << '\n'; } + catch (...) + { std::cerr << "Database DSO server termination acceptance raised an unexpected coroutine/native exception\n"; } + return 1; +} diff --git a/tests/database/server_termination.md b/tests/database/server_termination.md new file mode 100644 index 000000000..36edcc30d --- /dev/null +++ b/tests/database/server_termination.md @@ -0,0 +1,132 @@ +# Owned-server termination through public plugin DSOs + +[简体中文](server_termination_CN.md) + +`ark_database_public_plugin_server_termination_tests redis|mysql` loads the actual +hidden-symbol plugin DSO using only public API targets. `check-database` compiles +this opt-in program but never starts it or signals a server. MySQL requires +`ARKNET_ENABLE_SSL=ON` and trusted TLS. The separate Bash actor requires `lsof`, +`ps`, `rg` and the private service layout below; it targets owned local Unix +processes and does not provide a Windows fault injector. + +Build with the [database test configuration](README.md#build): + +```sh +cmake --build database-build \ + --target ark_database_public_plugin_server_termination_tests --parallel 1 +``` + +## Workload and assertions + +Redis sends a single read-only `XREAD BLOCK 60000 STREAMS $`. +Preflight checks that the stream key does not exist; the test creates no stream +or values. A separate authenticated observer connection reads `CLIENT LIST` and +must find the exact named primary connection with `cmd=xread`, blocked flag and +server client ID. MySQL sends `SELECT SLEEP(60) /**/` against a private +schema. A separate trusted-TLS connection to the same owned server must observe +that exact tag and connection ID in `information_schema.processlist`, in +`User sleep`. Only then does the C++ host publish the private fault barrier. +Observer polling delays are throttling, never proof of active server work. + +The separate shell actor rechecks the ownership token, captured PID/start time, +canonical executable text mapping, private data directory working directory, +PID file and PID-specific IPv4 loopback listener before sending `SIGKILL` to that +single PID. Before any process inspection, the actor resolves the private root +and config/data/PID-file paths physically, including `..` and parent symlinks, +and rejects paths outside the canonical root. It rejects symlink ownership, +identity and coordination files, rechecks path identities before signaling, +and refuses stale output markers. MySQL's defaults-file argument and the private configuration/data +mapping are checked too. The parent must collect that child's intentional +signal exit, the actor's successful receipt and the host exit; a marker alone +does not prove process termination. + +The pending request has a 20-second operation timeout; the server block is +60 seconds. Expected failures are Redis `disconnected` with +`command_may_have_run=true`, and MySQL `transport` with +`outcome_unknown=true`, each retaining a nonzero native transport error. +Cancellation, operation timeout and MySQL server SQL errors are rejected. +These are service results; a coroutine exception fails the test. The request and +observer roots complete exactly once on the owner's shared executor, retire +before stop, and preserve the owning native result after module/manager +teardown and DSO unload. Cooperative stop is bounded and idempotent. No pending +request is cancelled to inject the failure; the retired task ID rejects cancel. + +## Private actor coordination + +Use fresh disposable service instances and private credentials, on IPv4 +loopback only. MySQL needs a fresh schema/account with SELECT permission and +required trusted TLS with the loopback IP in the certificate. Do not point the +actor at a shared or installed service. This test does not restart servers or +inspect persistence, transaction rollback or write effects. + +Before launching a server, create a new private root and token: + +```sh +umask 077 +private_root=$(mktemp -d /tmp/ark-dso-termination-owned.XXXXXX) +export ARK_DATABASE_FAULT_DIR=$private_root +export ARK_DATABASE_FAULT_TOKEN=$(openssl rand -hex 32) +printf '%s\n' "$ARK_DATABASE_FAULT_TOKEN" > "$private_root/ownership" +``` + +Create config/data/PID-file paths under this root. Redis's config must contain +exact `dir ` and `pidfile ` lines; MySQL's config must contain +`datadir=` and `pid-file=` and start with +`--defaults-file=`. Launch in the foreground as a child of the +orchestration shell, capture `$!`, and verify readiness with authenticated +protocol requests and the PID's listener. Do not use elapsed sleep as readiness. +Both servers must use their private data directory as their working directory. + +After readiness, record the identity from that owned child. The executable path +must be canonical (resolve the directory with `pwd -P`), while `datadir`, +`config`, `pidfile` use the original private-root spelling: + +```sh +printf '%s\n%s\n%s\n%s\n%s\n' "$ARK_DATABASE_FAULT_TOKEN" "$service_pid" \ + "$canonical_executable" "$datadir" "$(ps -p "$service_pid" -o lstart=)" \ + > "$private_root/$mode.identity" +# mode is redis or mysql; port belongs only to this owned child. +bash tests/database/owned_server_fault_actor.sh "$mode" "$service_pid" \ + "$canonical_executable" "$config" "$datadir" "$pidfile" "$port" > "$actor_log" 2>&1 & +actor_pid=$! +``` + +Run the C++ host concurrently with the actor. Supply the same endpoint variables +as [public DSO service acceptance](README.md#real-plugin-dsos-against-real-services): +Redis host/port/password; MySQL host/port/user/password/ +database/CA. Keep credentials in private files or environment, never command +arguments or committed logs. Set `mode` to the protocol being tested: + +```sh +./database-build/tests/database/ark_database_public_plugin_server_termination_tests "$mode" +host_status=$? +wait "$actor_pid"; actor_status=$? +wait "$service_pid"; service_status=$? +# Require host_status=0, actor_status=0, service_status=137 (SIGKILL on Bash). +``` + +Capture nonzero signal exits explicitly if the orchestration uses `set -e`. +Wrap the whole run in a bounded shell supervisor. Missing/mismatched/stale +markers fail; the actor refuses to signal without a valid active barrier within +15 seconds. Keep a cleanup trap: stop an un-faulted verified child through its +authenticated normal shutdown; after either path, reap all owned children, +verify their exact PIDs are absent, then remove the token-guarded private root, +including credentials, data and private TLS keys. Never reuse fault markers. + +This covers the observed peer-loss request and local retirement. It does not +establish restart/recovery, persistence, automatic replay behavior, all fault +races, deployed-service acceptance, cross-host behavior or capacity. The precise +native transport error depends on the platform; no exact EOF/reset numeric value +is required. Preserve local identities/results/cleanup evidence outside Git. + +The path guards have a separate non-signaling regression entry point: + +```sh +bash tests/database/owned_server_fault_actor_guards.sh +``` + +It starts no server. A fake PID and intercepted process/signal/wait commands +make unexpected inspection or waiting fail immediately. Twenty invalid path or +marker cases cover traversal, symlink-parent escapes (including symlink plus +`..`), leaf/coordination symlinks and stale receipts; a valid-path control must +reach the intercepted process check. No actual process is inspected or signaled. diff --git a/tests/database/server_termination_CN.md b/tests/database/server_termination_CN.md new file mode 100644 index 000000000..e37a03091 --- /dev/null +++ b/tests/database/server_termination_CN.md @@ -0,0 +1,114 @@ +# 通过公开插件 DSO 验证自有服务异常退出 + +[English](server_termination.md) + +`ark_database_public_plugin_server_termination_tests redis|mysql` 仅使用公开 API +target,加载实际的隐藏符号插件 DSO。`check-database` 只编译该程序,不执行它,也不向 +数据库进程发送信号。MySQL 需要 `ARKNET_ENABLE_SSL=ON` 和受信任 TLS。 +独立 Bash 故障执行脚本需要 `lsof`、`ps`、`rg` 及下述私有服务目录; +它面向本地自有 Unix 进程,不提供 Windows 故障注入。 + +按[数据库测试配置](README_CN.md#构建)构建: + +```sh +cmake --build database-build \ + --target ark_database_public_plugin_server_termination_tests --parallel 1 +``` + +## 请求与断言 + +Redis 发送单条只读命令 `XREAD BLOCK 60000 STREAMS $`。 +预检确认独占流键不存在;测试不创建流或数据。独立认证观察连接读取 `CLIENT LIST`, +必须找到名称完全匹配的主连接,并确认 `cmd=xread`、阻塞标志和服务端 client ID。 +MySQL 在私有 schema 中发送 `SELECT SLEEP(60) /**/`。 +同一自有服务上的独立受信任 TLS 连接,必须在 `information_schema.processlist` +中观察到该标签及 connection ID,且状态为 `User sleep`。 +只有完成真实服务观察后,C++ 程序才发布私有故障屏障。 +轮询间隔仅用于节流,不能作为请求正在执行的证据。 + +独立 shell 脚本再次核对归属令牌、捕获的 PID 和启动时间、规范化可执行文件映射、 +私有数据工作目录、PID 文件及该 PID 的 IPv4 回环监听端口,才向这个 PID 发送 +`SIGKILL`。在任何进程检查前,脚本先按物理路径解析私有根目录以及配置、数据和 PID 文件, +包括 `..` 和父目录符号链接,并拒绝规范化后位于根目录之外的路径。 +归属、身份及协调标记不能是符号链接;发送信号前重新核对路径身份,拒绝旧的输出标记。 +MySQL 还核对 defaults-file 参数,以及私有配置中的数据目录映射。 +父 shell 必须收集服务子进程的预期信号退出状态、故障脚本成功凭据及测试程序退出状态; +单个标记文件不能证明进程已退出。 + +待完成请求的操作超时为 20 秒,服务端阻塞为 60 秒。预期 Redis 返回 +`disconnected`、`command_may_have_run=true`;MySQL 返回 `transport`、 +`outcome_unknown=true`,两者均保留非零原生传输错误。 +取消、操作超时及 MySQL SQL 服务错误均不符合该场景。 +这些失败通过服务结果返回,协程异常会使测试失败。请求和观察根任务必须在拥有线程的 +共享执行器上各完成一次,在 stop 前自然退出;拥有型原生结果在模块、manager 销毁及 +DSO 卸载后仍可访问。协作式 stop 有界且幂等。 +测试不通过取消待完成任务注入故障;已退出的任务 ID 必须拒绝 cancel。 + +## 私有故障屏障 + +仅使用新建的一次性服务、私有凭据和 IPv4 回环监听。MySQL 使用新建的 schema、 +仅具 SELECT 权限的账号,以及证书包含回环 IP 的受信任 TLS。 +不得使用共享服务或已安装的服务。本测试不重启服务,不检查持久化、事务回滚或写入效果。 + +启动服务前,新建私有目录和令牌: + +```sh +umask 077 +private_root=$(mktemp -d /tmp/ark-dso-termination-owned.XXXXXX) +export ARK_DATABASE_FAULT_DIR=$private_root +export ARK_DATABASE_FAULT_TOKEN=$(openssl rand -hex 32) +printf '%s\n' "$ARK_DATABASE_FAULT_TOKEN" > "$private_root/ownership" +``` + +配置、数据和 PID 文件路径都必须位于该目录下。Redis 配置需包含完全匹配的 +`dir ` 和 `pidfile ` 行;MySQL 需包含 `datadir=`、 +`pid-file=`,并通过 `--defaults-file=` 启动。 +以前台模式作为编排 shell 的子进程运行,捕获 `$!`,通过认证请求和该 PID 的监听端口 +检查就绪,不能用等待时间代替。两种服务均须以其私有数据目录作为工作目录。 + +就绪后,记录这个自有子进程的身份。可执行文件路径必须规范化(目录通过 `pwd -P` +解析);`datadir`、`config`、`pidfile` 保留私有根目录原始写法: + +```sh +printf '%s\n%s\n%s\n%s\n%s\n' "$ARK_DATABASE_FAULT_TOKEN" "$service_pid" \ + "$canonical_executable" "$datadir" "$(ps -p "$service_pid" -o lstart=)" \ + > "$private_root/$mode.identity" +# mode 为 redis 或 mysql;port 仅属于这个自有子进程。 +bash tests/database/owned_server_fault_actor.sh "$mode" "$service_pid" \ + "$canonical_executable" "$config" "$datadir" "$pidfile" "$port" > "$actor_log" 2>&1 & +actor_pid=$! +``` + +与故障脚本并行运行 C++ 程序,提供与[公开 DSO 服务验收](README_CN.md#真实插件动态库与真实服务) +相同的端点变量:Redis host/port/password; +MySQL host/port/user/password/database/CA。凭据仅放入私有文件或环境变量, +不进入命令行参数或提交的日志。`mode` 选择协议: + +```sh +./database-build/tests/database/ark_database_public_plugin_server_termination_tests "$mode" +host_status=$? +wait "$actor_pid"; actor_status=$? +wait "$service_pid"; service_status=$? +# 要求 host_status=0、actor_status=0、service_status=137(Bash 中的 SIGKILL)。 +``` + +编排启用 `set -e` 时,需要显式捕获预期的非零信号退出。为整个流程增加有界 shell +监督。缺失、不匹配或旧标记均会失败;故障脚本在 15 秒内未得到有效活动屏障时拒绝发送信号。 +保留清理 trap:对尚未注入故障的、身份核验通过的子进程执行认证正常停机; +无论成功还是失败,均回收全部自有子进程、核实其确切 PID 已消失,再按令牌核验删除私有目录, +包括凭据、数据和 TLS 私钥。不得复用故障标记。 + +该场景仅覆盖观察到的对端丢失请求及本地退出,不代表重启恢复、持久化、自动重放行为、 +所有故障竞态、已部署服务、跨主机或容量验收。原生传输错误由平台决定,不要求固定的 +EOF/reset 数值。身份、结果和清理原始证据保留在 Git 之外。 + +路径防护提供独立的无信号回归入口: + +```sh +bash tests/database/owned_server_fault_actor_guards.sh +``` + +该脚本不启动服务。测试使用虚构 PID,并拦截进程检查、信号和等待命令, +一旦发生非预期检查或等待即立即失败。20 个无效路径或标记用例覆盖目录穿越、 +父目录符号链接逃逸(包括符号链接与 `..` 的组合)、末级及协调文件符号链接、旧凭据; +有效路径对照必须到达被拦截的进程检查。不检查实际进程,也不发送实际信号。