diff --git a/doc/CHANGELOG.md b/doc/CHANGELOG.md index 02780d5..48d9152 100644 --- a/doc/CHANGELOG.md +++ b/doc/CHANGELOG.md @@ -1,5 +1,19 @@ # Changelog +## 2026-09-17 Integration 部署解析入口统一实施(RFC-0062) + +- **统一共享部署准备流程(`PrepareDeploymentDocument`)**: + - 新增 `src/adapter/deployment_diagnostic.h` 轻量错误载体与 `src/adapter/deployment_preparation.h/.cpp` 共享部署准备实现(S1–S7)。 + - 严格消除 `alg_pipeline_tool.cpp` 中的 `ResolveDeploymentBoundary` 及 `pipeline_authoring.cpp` 中的重复部署边界构造逻辑。 + - 强制执行覆盖不可掩盖原始非法声明规则(T03/T04):原始模型路径缺失、类型非字符串、空串或结构错误在覆盖生效前由核心解析器(`ParsePipelineConfig`)识别并拒绝。 +- **结构化诊断与来源精准投影**: + - 建立 `ProjectModelPathDiagnostics` 机制,将 Core 对有效模型的诊断精准映射回原始完整文档路径(有覆盖映射至 `/deployment/model_paths/`,无覆盖映射至 `/models//model_path`)。 + - `validate-io` 直接传递底层结构化诊断,不再从错误文案中脆弱提取 JSON Pointer。 + - `PrepareDeploymentDocument` 实行原子化失败清理(T18),失败立即重置局部对象,保证无残留状态。 +- **各调用面与工具链行为收敛**: + - `src/tools/pipeline_document_validation.h/.cpp` 统一 `validate`(`kValidate`)、`validate --explain`(`kExplain`)、`plan`(`kPlan`)及 Pipeline Authoring(`edit`、`fix-deps`)的文档校验与计划编排。 + - 计划失败时保持规范包络(包含空层与拓扑顺序及完整诊断列表);`edit` 与 `fix-deps` 补齐部署校验,防止未知绑定或输出配置非法。 + ## 2026-09-17 部署配置归拢至 Pipeline 根文档(RFC-0061) - **部署配置归拢至 Pipeline 文档**: diff --git a/doc/rfcs/0062-unified-integration-deployment-preparation.md b/doc/rfcs/0062-unified-integration-deployment-preparation.md new file mode 100644 index 0000000..0cefa2d --- /dev/null +++ b/doc/rfcs/0062-unified-integration-deployment-preparation.md @@ -0,0 +1,603 @@ +# RFC 0062: Integration 部署解析入口统一实施设计 + +- **RFC 编号**:0062-unified-integration-deployment-preparation +- **创建日期**:2026-09-17 +- **文档状态**:Completed +- **关联分支**:`docs/integration-deployment-resolution-design` +- **目标版本**:下一次 Integration 内部解析收敛版本 +- **负责人 / 作者**:LLM-EdgeFlow 维护者 +- **代码核查基线**:`8e1a74ac1bff` +- **关联决策**:细化 RFC-0061 的共享接入解析实现,补齐其“覆盖不得修复非法原始声明”约束;保留 RFC-0025 的路径边界、RFC-0049/0050 的输出分配契约,以及 RFC-0057 的编辑与修复行为。 + +> 本文设计已全部实施完成。所有入口已统一迁移至共享部署准备流程 `PrepareDeploymentDocument`。 +> 实施状态与验证记录详见本文第 5 节。 + +## 1. 问题与范围 + +### 1.1 需要消除的重复 + +当前同一份完整 Pipeline JSON 会经过不同的接入准备代码: + +| 入口 | 当前实现 | 现有差异 | +| --- | --- | --- | +| `validate`、`plan` | [`alg_pipeline_tool.cpp`](../../src/tools/alg_pipeline_tool.cpp) 的 `ResolveDeploymentBoundary` | 自行检查 binding、converter、输出槽位、模型覆盖、I/O 边界;Core 错误可以保留完整报告 | +| `validate-io`、`resolve-conf`、SDK Create/配置预检 | [`io_binding_resolver.cpp`](../../src/adapter/io_binding_resolver.cpp) 的 `ResolveFromPipelineJson` | 重复上述逻辑,另做路径解析、批次上限计算、默认深度输出池预算和计划组装;错误主要通过字符串返回 | +| `edit`、`fix-deps` | [`pipeline_authoring.cpp`](../../src/tools/pipeline_authoring.cpp) 的 `ValidateOrExplainAuthoring` | 第三份边界构造;未应用模型覆盖或校验输出分配;未知 binding/converter 可能退回无接入边界的 Core 校验 | + +`validate-io` 还会从错误文本中的 `(at ...)` 或 `at /...:` 反向提取 JSON Pointer。 +修改错误文案可能改变机器可读诊断,因此需要让路径以数据形式跨函数传递。 + +### 1.2 已复现的问题 + +以 `configs/pipeline_cross_rerank_cpu.json` 为原始输入,保留合法 +`deployment.model_paths.rerank_model_v1`,分别删除、清空或将 +`models[0].model_path` 改成数字,当前 `validate --stdin` 都返回 `ok: true`。 +移除覆盖后,相同原始声明被拒绝。 + +原因是两个入口都先把覆盖值写入 JSON,再让 Core 解析。原始错误被覆盖擦除。 +[RFC-0061 §2.4](0061-pipeline-owned-deployment-configuration.md#24-模型路径覆盖与解析基准) +要求原始 `model_path` 必填且类型正确,不能由覆盖使非法声明合法化。 + +### 1.3 实施结果 + +实施后,同一完整文档、同一注册集合和相同准备选项,得到相同的: + +- binding/converter、端口映射、有效批次上限; +- 输出槽位规范和归一化参数; +- 有效模型路径及每个路径的来源; +- 中性 Pipeline JSON 与 `PipelineIoBoundary`; +- 接入错误的 `code/path/message`。 + +DAG、Node/Model/Backend Definitions、端口闭合及执行计划继续只由 +`PipelineValidator` 检查和生成。普通校验、解释诊断和运行计划可以选择不同的 +Validator API,但不得各自重新解释部署规则。 + +### 1.4 范围边界 + +本 RFC 包含共享部署准备、三个调用面的接入、结构化错误传递、原始声明校验修复及回归测试。 +不修改 Pipeline JSON/`.conf` 的持久格式,不删除 `deployment.model_paths`,不迁移现有方案参数。 + +以下事项维持现状: + +- C++ Operator 公共头、函数表、`noexcept` 和两类异常屏障; +- `.conf` 只含 `pipe_path`,宿主提供模型与配置根目录; +- `init` 生成或复制草稿的语义,特别是 `init --empty`; +- Core 不认识 `deployment`、平台类型和输出池; +- Node、Model、Backend 实现,Profile 发现、模板复用和真实模型效果验收; +- 池预算策略:现有默认深度检查及宿主非默认深度复查均保留,见 §2.6。 + +## 2. 决策与详细设计 + +### 2.1 采用“共享准备结果,再按用途校验” + +新增 Integration 内部函数 `PrepareDeploymentDocument`,提取部署部分的共同语义。 +它返回准备结果,不执行 `PipelineValidator`,不创建模型、Node、Session 或输出池。 +临时 JSON、归一化参数等普通内存分配允许发生。 + +```mermaid +flowchart TD + A[完整 Pipeline JSON] --> B[Integration: PrepareDeploymentDocument] + B --> C[中性 JSON + PipelineIoBoundary + 接入规范 + 路径来源] + C --> D[CLI / Authoring: Validate 或 Explain 或 ValidateAndPlan] + C --> E[IoBindingResolver: 池预算检查 + ValidateAndPlan] + N[中性 Pipeline JSON] --> D + D --> F[版本化诊断 / 编辑验证 / 计划预览] + E --> G[ValidatedIoPlan] + G --> H[OperatorConfigResolver: 宿主深度复查] + H --> I[现有物化与执行链] +``` + +不能把 CLI 简单改成调用现有 `ResolveFromPipelineJson` 后只取成功计划。 +该接口在 Core 失败时只保留首条字符串错误,无法满足 `--explain`、修复候选及部分 DAG +诊断的要求。共享准备结果能保留这些能力,也避免“先完整校验一次,再为 Explain 校验一次”。 + +### 2.2 新增内部文件与接口 + +新增文件均放在 `src/`,不进入公共 SDK 头文件视图: + +| 文件 | 职责 | +| --- | --- | +| `src/adapter/deployment_diagnostic.h` | 轻量错误载体;不包含工具响应 JSON、平台布局或注册表逻辑 | +| `src/adapter/deployment_preparation.h/.cpp` | 完整部署文档准备、原始结构校验、绑定/槽位/覆盖/边界的唯一实现 | +| `src/tools/pipeline_document_validation.h/.cpp` | CLI 和 Authoring 共用的调用、Core 报告保留及序列化适配;不新增验证规则 | + +接口形状如下。实现可以调整局部命名,但必须保持这些输入、结果和阶段边界: + +```cpp +struct DeploymentDiagnostic { + std::string code; // Integration 错误码,或 DiagnosticCodeName 的结果 + std::string path; // 原始完整文档的 RFC 6901 JSON Pointer + std::string message; + int legacy_status = -2; + std::optional pipeline_diagnostic; +}; + +enum class DeploymentPathMode { + kLexicalOnly, + kUnderRoot, +}; + +struct DeploymentPrepareOptions { + std::string transport = "operator"; + DeploymentPathMode path_mode = DeploymentPathMode::kLexicalOnly; + std::string model_root_dir; +}; + +struct PreparedDeployment { + IoBindingDefinition binding; + const InputConverterDefinition* input_converter = nullptr; + const OutputConverterDefinition* output_converter = nullptr; + InputPortBindings input_port_bindings; + OutputPortBindings output_port_bindings; + size_t effective_max_batch_size = 0; + + std::unordered_map output_specs; + std::unordered_map output_parameter_texts; + std::unordered_set overridden_model_ids; + std::vector model_path_source_pointers; // 与 models 原顺序对应 + + nlohmann::json neutral_pipeline_json; + PipelineIoBoundary io_boundary; +}; + +bool PrepareDeploymentDocument( + const nlohmann::json& document, + const DeploymentPrepareOptions& options, + PreparedDeployment* output, + DeploymentDiagnostic* diagnostic); + +void ProjectModelPathDiagnostics( + const PreparedDeployment& prepared, + ValidationReport* report); +``` + +错误载体可包含 `core/pipeline_diagnostic.h`,依赖方向仍然向下。 +Integration 专属错误码不加入 Core 的 `DiagnosticCode` 枚举。 +仅需在 Integration 保留一处错误码映射;CLI 不以字符串消息判断错误种类。 + +契约要求: + +1. `output` 必须非空;入口清空旧结果和旧错误。所有工作在局部对象完成,成功才整体移动到输出。 +2. 失败不得留下部分 `PreparedDeployment` 或复用上一次的 converter/路径。 +3. `document` 为只读;准备过程不修改调用者的 JSON。 +4. 新函数只接收完整部署文档。没有 `deployment.io` 时失败;中性文档由调用端路由给 Core。 +5. `kUnderRoot` 要求非空根目录;`kLexicalOnly` 要求根目录为空。矛盾选项作为内部调用错误拒绝。 +6. 返回成功仅表示接入准备成功,不表示 Pipeline 已通过验证或具备可执行计划。 + +### 2.3 固定处理顺序 + +#### S1:检查调用参数与部署文档结构 + +检查输出参数、transport 和路径模式。复用 `SplitPipelineDocument`,保留它提供的错误路径。 +拒绝未知 `deployment` 字段、缺失/错误类型的 `io`、非法覆盖值等。 +除了移除已经校验的 `deployment`,不能过滤其他顶层字段。 + +#### S2:覆盖之前验证原始中性结构 + +对 `split.neutral_pipeline_json` 调用现有 `ParsePipelineConfig`,获得 +`ParsedPipelineConfig original` 与 `PipelineDiagnostic`。 + +这个阶段只复用 Core 现有的严格结构解析,包括原始 `biz_name`、模型声明、重复 ID、 +显式节点 `id/depends_on` 等;不检查注册能力、不做拓扑规划、不补 Node/Model 默认值。 +禁止在 Integration 再写一份必填模型字段清单。 + +解析失败立即返回,错误指向原始 `/models//model_path` 等位置。 +覆盖不能改变该错误的来源。原始 `biz_name` 类型错误也在此报告,不能先用 +`json.value` 读取而抛出未组织的异常。 + +**允许两次结构解析**:这里解析原始文档,随后 Validator 解析有效中性文档。 +两次复用同一实现,各自证明不同的输入;本 RFC 不为避免这点 CPU 开销新增 Core 规划 API。 +这不等于运行两次完整校验/规划。 + +#### S3:解析 binding、converter 和业务边界 + +使用 `IoBindingRegistry`、`IoConverterRegistry`: + +1. binding 必须存在且 transport 为 `operator`。 +2. `original.biz_name` 必须与 binding 的 `biz_name` 完全一致。 +3. 输入/输出 converter 必须存在,沿用注册表已经建立的组合约束。 +4. 有效批次上限取 input converter、output converter 的最小值;存在 exposure 时再纳入其上限。 +5. 未知 binding/converter 直接失败,不允许退回 Core 的默认业务边界。 + +注册定义自身的完整性继续由注册表负责。本阶段不能复制 Definition 校验规则或硬编码 biz 列表。 + +#### S4:解析输出分配 + +保留现有 `R ⊆ C ⊆ A` 规则:必须配置全部必需输出槽位,可以省略可选槽位,不允许未知槽位。 +逐槽调用 `OperatorConfigResolver::ResolveOutputAllocation`,同时保存规范和归一化参数文本。 + +`type`、allocator、metadata、capacity 和参数文本的具体规则仍由现有 resolver/ValueType 定义负责。 +不能根据 map key 推断外层类型,不能把队列深度塞进 allocator 参数。 + +此处不做整句柄深度预算,也不创建输出池。预算分工见 §2.6。 + +#### S5:应用模型路径覆盖并记录来源 + +用 `original.models` 建立 model ID 到 source index 的映射;禁止重新容忍缺 ID 或重复 ID。 +未知覆盖 ID 返回 `/deployment/model_paths/`。 + +在局部中性 JSON 副本上,仅覆盖已声明模型的 `model_path`: + +```text +有覆盖:effective.models[i].model_path = deployment.model_paths[id] + source[i] = /deployment/model_paths/ +无覆盖:保持原始 model_path + source[i] = /models//model_path +``` + +模型顺序不改变,不覆盖 capability、Backend 或 sidecar 字段。 +`~` 和 `/` 使用现有 `EscapeJsonPointer`,诊断顺序采用原 models 顺序;多个未知覆盖键采用 +JSON 对象的稳定顺序,不依赖 `unordered_map` 的遍历顺序。 + +#### S6:按模式处理有效路径 + +- `kLexicalOnly`:保持有效路径的相对/绝对形式,不访问文件系统、不要求模型文件存在。 + 后续 Core 继续负责已有的路径词法检查。 +- `kUnderRoot`:调用现有 `ResolveDeploymentModelPaths`;保留根目录可访问性、规范化、 + `..` 与符号链接边界约束。路径相对宿主根目录解析,不相对 JSON、`.conf` 或 CWD 搜索。 + 不新增“权重文件必须存在”的要求,不解析 sidecar 内容。 + +原始结构检查和有效路径检查不能混淆:原始路径为合法非空字符串、但被覆盖前指向另一个 +位置时,不要求旧位置可访问;只对最终有效路径做部署路径安全检查。 + +扩展 `ResolveDeploymentModelPaths`,在现有字符串错误之外增加可选结构化错误输出。 +结构化路径在产生错误的位置设置,不能在上层重新解析文案。所有现有调用保持可编译。 + +#### S7:构造中性 I/O 边界,发布准备结果 + +按 converter 的 logical ports 和 binding 的映射构建 `PipelineIoBoundary`。 +保留端口类型、required、cardinality、provenance、lifetime 和 lifetime 配置字段。 +不手工把 session/request 生命周期改成常量,也不以 Catalog 的默认 ingress/egress +替代 binding 的实际映射。 + +完成后发布 `PreparedDeployment`。以下步骤由消费方执行:Core 校验/解释、预算、计划组装、物化。 + +### 2.4 CLI 与 Authoring 的共用适配 + +在 `src/tools/pipeline_document_validation.*` 定义工具内部结果: + +```cpp +enum class DocumentValidationMode { kValidate, kExplain, kPlan }; + +struct DocumentValidationResult { + bool ok = false; + nlohmann::json response; + std::optional core_report; +}; + +DocumentValidationResult ValidatePipelineDocument( + const nlohmann::json& document, DocumentValidationMode mode); +``` + +它只负责以下编排,不增加模型/端口/部署校验规则: + +1. 对象包含 `deployment`:调用共享准备,选择 `kLexicalOnly`。 +2. 不含 `deployment`,或根不是对象:原样交给 Core;不得提前剥除未知字段。 +3. 准备成功后,根据 mode 调用 `Validate`、`Explain` 或 `ValidateAndPlan` 中的一项。 +4. 调用 `ProjectModelPathDiagnostics` 归位有效模型路径的诊断,再保存完整 `ValidationReport`, + 用于 `fix-deps` 读取诊断、remediation、related nodes 等信息;中性输入不需要投影。 +5. 准备失败时返回结构化报告;尚未调用 Core 时 `core_report` 为空,不能伪造“可自动修复”的 Core 报告。 + +`Explain` 本身会验证候选补丁,这是现有 Core 解释功能;不受“调用一种 Validator API”约束影响。 +禁止在进入 `Explain` 之前额外运行完整 `ValidateAndPlan`。 + +各入口处理: + +| 命令 | 新路由与必须保留的行为 | +| --- | --- | +| `validate` | 共用适配的 `kValidate`;失败仍保留完整 Core 诊断 | +| `validate --explain` | `kExplain`;准备成功后允许诊断不完整的 DAG/配置,不要求先得到成功计划 | +| `plan` | `kPlan`;成功响应移除 `diagnostics`,失败保留诊断和 plan | +| `plan --explain` | 保持基线行为:目前解析该参数但仍走规划,不在本 RFC 中新增解释语义 | +| `edit` | 对操作后的 `working_pipeline` 调用 `kExplain`;保留原始工作文档作为返回值,不返回解析后的绝对路径或补齐默认值的副本 | +| `fix-deps` | 修复前后均调用 `kExplain`;读取 `core_report` 的修复信息,最后完整部署验证通过才允许写文件 | +| `init` | 继续生成/复制模板;不新增准备或完整校验,空草稿继续可用 | + +Authoring 的事务要求: + +- `require_valid: false` 时,合法编辑操作可以返回无效草稿,`result.ok` 为 true, + `result.validation.ok` 为 false;部署错误不能使草稿不可编辑。 +- `require_valid: true` 时,准备失败或 Core 失败都使操作失败,不返回修改后的 pipeline。 +- `fix-deps` 若准备失败,不继续猜测部署边界或修复部署配置;返回该诊断,`written: false`。 +- 有 Core 报告时,保留现有仅修复无歧义依赖的算法;最终验证失败时保持原文件字节不变。 + +删除旧 `ResolveDeploymentBoundary`、其后重复的 biz 比对,以及 +`ValidateOrExplainAuthoring` 中的部署解释代码。保留薄封装时,它只能转调共用工具适配。 + +### 2.5 IoBindingResolver 与宿主入口 + +`ResolveFromPipelineJson` 改成以下步骤: + +```text +构造 options(空 root → LexicalOnly;非空 root → UnderRoot) + → PrepareDeploymentDocument + → 用 prepared.output_specs 执行现有默认深度句柄预算检查 + → PipelineValidator::ValidateAndPlan(prepared.neutral_pipeline_json, + kStrict, &prepared.io_boundary) + → ProjectModelPathDiagnostics(prepared, &plan.report) + → 若失败,返回 Core 主错误与完整来源路径,不发布计划 + → 若成功,移动 prepared 字段和 Core 计划,组装 ValidatedIoPlan +``` + +文件入口职责保持为 I/O:`ResolveFromFile` 读取 `.conf`,`ResolveFromConfig` 读取 Pipeline +一次后转内存入口。准备函数和 Validator 不重新打开 `.conf` 或 Pipeline。 + +`ValidatedIoPlan` 继续是已验证计划;不能让 `PreparedDeployment` 替换该类型传入运行时。 +Runtime 继续消费已经生成的 `ValidatedPipelinePlan`,不再次解析 JSON 或排序 DAG。 + +为 `ResolveFromFile`、`ResolveFromConfig`、`ResolveFromPipelineJson` 增加尾部可选参数 +`DeploymentDiagnostic* out_diagnostic = nullptr`。保留现有返回值、`out_error` 和已有调用方式。 +每一层负责把结构化错误向上传递,文件读取错误也必须填充结构化结果。 + +相应扩展 `DeploymentIoConfig::ReadFromFile/Parse` 的尾部可选诊断输出:打开文件、JSON 解析、 +`.conf` 字段及 `pipe_path` 检查在原来的错误产生点填写载体,保留现有字符串错误。 +不能在上层根据错误文案猜测文件错误类别。`.conf` 错误路径相对 `.conf` 文档,消息须说明文件; +进入 Pipeline 读取后,路径才相对 Pipeline 文档。未知的宿主路径解析失败可使用 `/`。 + +为内部 `OperatorConfigResolver::Resolve` 增加相同可选输出,置于现有 depth 参数之后。 +公共 Operator API 不暴露新类型。Create/配置预检仍通过现有最后错误字符串报告失败。 + +### 2.6 路径与池预算模式必须明确 + +| 消费入口 | 模型路径模式 | 整句柄池预算 | +| --- | --- | --- | +| `validate/plan/edit/fix-deps` | LexicalOnly | 不假定宿主队列深度,不做整句柄预算 | +| `validate-io` 未给 `--model-root` | LexicalOnly | 现有默认深度检查 | +| `validate-io --model-root` | UnderRoot | 现有默认深度检查 | +| `resolve-conf`、SDK 配置预检/Create | UnderRoot | 现有默认深度检查,再按宿主非默认 depth 复查 | +| 直接 Core 单元测试 | 不经 Integration | 不涉及外部输出池 | + +共享准备统一的是相同条件下的规则,不是把环境相关检查强塞进静态校验。 +静态通过不承诺宿主深度、文件根目录或真实模型加载可用。 + +现有 `ComputeOutputPoolPayloadBytes`、`CheckedAdd`、单槽及整句柄预算仍复用原实现。 +默认深度 25、最大深度 1024、句柄业务载荷预算 64 MiB 以现有常量为准,不在新准备函数再定义。 +`depth == 0` 仍按现有逻辑归一化。 + +当前默认深度与实际深度两次检查的政策是否应改成只检查实际深度,是独立行为决策。 +本 RFC 保留它,避免在解析收敛时扩大可接受配置范围。 + +### 2.7 结构化诊断、路径来源与兼容 + +准备失败使用稳定错误载体,最小响应字段为 `code/path/message/severity`。 +severity 固定为 `error`;模型/Node 的 Core 报告继续保留其他字段。 + +| 错误类别 | 机器码 | 主路径 | +| --- | --- | --- | +| deployment 结构或字段错误 | `DEPLOYMENT_ERROR` | `SplitPipelineDocument` 返回的精确位置 | +| 缺少完整部署 I/O | `MISSING_DEPLOYMENT_IO` | `/deployment/io` | +| 未知 binding | `UNKNOWN_IO_BINDING` | `/deployment/io/io_binding` | +| transport 不支持/不匹配 | `UNSUPPORTED_TRANSPORT` | binding 不匹配指向 binding;入口选项错误指向 `/` | +| biz 与 binding 不匹配 | `BIZ_MISMATCH` | `/deployment/io/io_binding` | +| 缺少 converter | `UNREGISTERED_CONVERTER` | `/deployment/io/io_binding` | +| 未知/缺少输出槽位 | `UNKNOWN_OUTPUT_SLOT` / `MISSING_OUTPUT_SLOT` | `/deployment/io/output_allocations/` | +| 分配配置错误 | `INVALID_OUTPUT_ALLOCATION` | 槽位路径;有结构化子路径时可进一步精确 | +| 未知模型覆盖 ID | `UNKNOWN_MODEL_ID` | `/deployment/model_paths/` | +| 原始 model_path 非字符串/空串 | `INVALID_MODEL_PATH` | `/models//model_path` | +| 其他原始中性结构错误 | 保留 Core 的 code,包括 `MISSING_FIELD` | 原始 Core path | +| UnderRoot 有效路径解析/越界错误 | `INVALID_MODEL_PATH` | 有覆盖指覆盖项,否则指原始 model_path | +| Core 词法路径错误 | 保留 Core 的 code,当前为 `FIELD_RANGE` | 仅投影 path 到有效路径来源,保留 Core 的 `-3` 状态 | +| 根目录/内部准备选项错误 | `DEPLOYMENT_ERROR` | `/`,消息说明宿主参数 | +| `.conf` 结构/pipe_path 错误 | `DEPLOYMENT_ERROR` | 对应 `.conf` 字段;无法细分时 `/`,消息说明文件上下文 | +| 默认/宿主深度预算错误 | `INVALID_OUTPUT_ALLOCATION` | 单槽失败指该槽位;合计失败指 `/deployment/io/output_allocations` | +| 文件打开/读取 JSON 错误 | `CONFIG_FILE_OPEN` / `JSON_PARSE` | `/`,消息包含实际读取的配置文件路径 | +| 非预期异常 | `INTERNAL_EXCEPTION` | `/`,不能作为校验成功降级 | + +原始结构诊断保留 `pipeline_diagnostic`,用于测试和内部归因。对原始 model_path 的 +类型/空值错误映射到既有 CLI `INVALID_MODEL_PATH`,是响应兼容转换,不是重复 schema 检查。 +缺字段仍为 `MISSING_FIELD`。已有非法文档含多个错误时,采用 §2.3 的确定阶段顺序, +不承诺与旧分散实现完全相同的首错排序。 + +返回码兼容按入口保留:`IoBindingResolver` 的空 `out_plan` 保持 `-1`; +`OperatorConfigResolver::Resolve` 的空 `result/model_path/cfg_file_name` 保持 `-2`。 +新准备函数空 `output` 的载体可以使用 `legacy_status = -1`,但上层不能用它覆盖已有入口 +自身的参数检查。其余接入/分配/路径错误保持 `-2`;Core 错误保持 `-3`。 +原始 model_path 类型/空串错误保持 `-2`,其余原始结构错误沿用 Core 的 `-3`。 +过去被覆盖掩盖的非法声明现在按上述规则拒绝,这是预期修复。 + +有效路径错误由来源表投影回完整文档:仅将严格匹配 +`/models//model_path` 的诊断改为 `source[i]`;不改模型配置、tokenizer 或 Backend 字段的路径。 +原始结构错误发生在覆盖前,永远不投影到覆盖项。 + +`--explain` 的 Node/依赖修复补丁必须仍可应用于原始完整 JSON:准备不得重排节点、模型, +不得把补齐后的 config 写回编辑文档。基线 Explain 不生成部署路径修复。 +若某个候选补丁触及本次被覆盖/改写的 model_path,该候选不得直接暴露为已验证修复; +本次移除此候选,保留诊断和其他候选。未来支持路径修复需另加“对原始文档应用并重新准备”的验证。 + +不同命令保留既有 envelope: + +- `validate/plan/edit.validation/fix-deps.validation` 使用原生准备 code/path;版本号仍为 1。 +- 准备失败的 `plan` 带空 `layers` 和 `topological_order`;Core 失败保留其实际部分计划。 +- `validate-io` 仍使用外层 `IO_VALIDATION_ERROR`,但 path 直接取结构化错误,不再解析 message。 +- `resolve-conf` 仍使用外层 `DEPLOYMENT_CONFIG`,保持当前 `/` 主路径和消息;精确底层诊断留在内部结果。 +- `resolve-conf` 成功响应中的有效配置、默认值、`output_pool(s)`、model path source 名称保持不变。 + +因此跨入口测试应比较“接受/拒绝、底层原因和来源”,不能要求所有历史命令的外层 code 相同。 + +### 2.8 注册表、生命周期与异常 + +准备函数不执行全局 Init/Deinit,不装载插件,也不清空注册表。调用者保证注册完成且读取期间 +注册表不会被 `ClearForTesting` 等操作修改。converter 指针延续现有 `ValidatedIoPlan` 的借用约定; +binding 按值保存,归一化参数保持现有 `shared_ptr` 所有权。 + +工具进程在 `validate/plan/validate-io/edit/fix-deps/resolve-conf` 的外层建立一次注册生命周期保护: +检查 Init 返回值,成功初始化才负责配对 Deinit。移除 Authoring 验证函数中重复的 Init。 +准备或共用工具适配内部不自行 Deinit,避免清理调用者拥有的活动句柄。 +工具内部函数的直接测试显式建立注册环境;SDK 的 Init/Create/Destroy/Deinit 顺序不改变。 + +新增函数使用局部结果和标准异常传播;普通校验失败走结构化结果。 +CLI 命令外层和所有现有导出函数继续捕获 `std::exception` 与未知异常。 +不要给可能分配字符串/JSON 的内部函数随意加 `noexcept`,导致坏输入或资源不足触发 terminate。 + +### 2.9 取舍 + +| 备选方案 | 不采用原因 | +| --- | --- | +| CLI 直接使用现有成功型 `ValidatedIoPlan` 接口 | 失败 Core 报告被压缩,Explain/修复能力受损 | +| 将 deployment 搬入 Core Validator | 引入 Core 对 Integration/平台输出分配的反向依赖 | +| 只抽取几段 utility,保留三份流程 | 校验顺序、覆盖规则与失败降级仍然可以漂移 | +| 本次新增 Core 已解析配置规划 API | 扩大 Core 接口和 Explain 迁移范围;两次轻量结构解析可以接受 | +| 本次同时删除 model_paths 或 `.conf` | 涉及配置格式/宿主迁移,超出入口统一范围 | + +## 3. 文件改动与兼容迁移 + +### 3.1 文件责任清单 + +| 文件 | 实施内容 | +| --- | --- | +| 新 `src/adapter/deployment_diagnostic.h` | 定义轻量错误载体及必要转换约定 | +| 新 `src/adapter/deployment_preparation.h/.cpp` | 完成 S1–S7,集中错误映射与模型路径来源投影 | +| `src/adapter/deployment_io_config.h/.cpp` | 在既有文件读取、JSON 解析和定位配置校验处填充可选结构化诊断,不复制解析规则 | +| `src/adapter/deployment_model_resolver.h/.cpp` | 增加结构化错误输出,保留路径规则和现有字符串参数 | +| `src/adapter/io_binding_resolver.h/.cpp` | 三个入口传递可选诊断;内存入口转共享准备,再预算、规划、组装 | +| `src/adapter/operator/operator_config_resolver.h/.cpp` | 传递结构化错误;保留宿主路径解析、默认值和非默认 depth 复查 | +| 新 `src/tools/pipeline_document_validation.h/.cpp` | 工具共用调用适配,保留 Core report 和版本化响应 | +| `src/tools/alg_pipeline_tool.cpp` | 删除重复部署 helper/biz 检查;迁移 validate/plan;移除 validate-io 文案解析;统一命令生命周期保护 | +| `src/tools/pipeline_authoring.cpp` | 使用共用适配;保留草稿、require_valid、回滚和 fix-deps 的 typed Core report 消费 | +| `src/adapter/CMakeLists.txt` | 将准备实现加入 `edgeflow_integration_objects` | +| `src/tools/CMakeLists.txt`、`tests/CMakeLists.txt` | 将工具共用实现同时编入生产 CLI 和测试 CLI | +| 现有 Adapter/Operator/CLI 测试 | 按第 4 节扩展原 suite,不新增独立测试可执行文件 | +| `tools/pipeline_studio/README.md`、`doc/CHANGELOG.md` | 实施完成后记录原始声明拒绝、Authoring 部署验证补齐及各校验模式边界 | + +`cmake_ext/LayerHeaderViews.cmake` 已收集 `src/adapter` 私有头;不得为了新 helper 扩大 +公共 SDK 或 Core 的 include 视图。完成时由现有分层检查验证。 + +### 3.2 可观察行为变化 + +合法现有配置不需要改写。下列错误行为按设计收紧: + +1. 原始模型路径缺失/类型错误/为空,即使有合法覆盖也失败。 +2. `edit` 的 validation 与 `fix-deps` 不再放过未知 binding/converter、错误分配和未知覆盖 ID。 +3. 格式错误的 biz 等字段返回结构化错误,不能绕到默认边界或产生非 JSON 命令输出。 +4. 多错误文档的首错顺序按共享阶段确定;保留诊断的准确来源,允许修正此前错误的路径。 + +不扩展新的兼容分支或双解析开关。实施分支完成全部入口迁移与验收后再交付, +避免部分入口已经收紧、其他入口仍使用旧规则的长期中间状态。 + +## 4. 验证与完成条件 + +### 4.1 测试分工 + +| 测试文件 / suite | 责任 | +| --- | --- | +| `tests/unit/adapter/test_io_binding_registry.cpp` / `IoBindingRegistryTest` | 准备函数、split/Core 边界、binding、slot、覆盖、来源、失败结果清空 | +| `tests/integration/pipeline/test_pipeline_catalog_validator.cpp` / `PipelineValidatorTest` | 准备后只交给 Core 的行为;完整报告、Explain、静态模型路径词法语义 | +| `tests/contract/abi/test_adapter_contract_security.cpp` / `AdapterContractSecurityTest` | 内存/文件路径与根目录边界、符号链接、SDK 预检 | +| `tests/integration/operator/test_operator_api.cpp` / `OperatorApiTest` | 配置预检/Create、输出分配、默认及宿主深度、句柄总预算、失败无句柄 | +| `tests/tooling/test_pipeline_studio.py` / `PipelineCliTest` | 真实 CLI 子进程的 validate/plan/explain/validate-io/resolve-conf 结果 | +| 同文件 / `Rfc0057AuthoringAndDeploymentTest` | edit/fix-deps、无效草稿、require_valid、事务写入和部署一致性 | + +使用现有 fixture builder/临时目录生成变体,避免复制整份 Pipeline。 +中性 Core fixture matrix 不直接接收带 deployment 的文档,部署矩阵应在 Integration/CLI 测试中构造。 + +### 4.2 必须覆盖的矩阵 + +以下“静态链”指 validate、plan、validate --explain 和 Authoring 的 validation; +“部署链”指内存/文件 IoBindingResolver、validate-io、resolve-conf 及 SDK 配置预检。 +文件/根目录检查只在具备相应输入的链路比较,不要求不同模式无条件返回相同结果。 + +| 编号 | 输入/操作 | 预期结果 | +| --- | --- | --- | +| T01 | 合法完整文档,含/不含覆盖 | 准备信息一致;实际计划、归一化参数和绑定映射与基线相同 | +| T02 | 合法中性文档 | CLI/Core 保持可校验;部署链拒绝缺 deployment;Core 直接接收完整文档仍拒绝未知字段 | +| T03 | 原始 model_path 缺失、null、数字、空串 + 合法覆盖 | 全部拒绝;path 指原始字段;配对无覆盖用例证明相同结构要求 | +| T04 | 缺少其他必需模型字段、重复 model_id、models 类型错误 + 覆盖 | 复用 Core 原始结构检查,覆盖不能隐藏错误 | +| T05 | 原始路径结构合法但旧位置不存在 + 有效覆盖 | 不检查被替代的权重位置;允许通过静态准备/路径解析 | +| T06 | 覆盖未知 ID/空值/非字符串、含 `~` 或 `/` 的 ID | 精确、稳定并正确转义的覆盖路径;不隐式新增模型 | +| T07 | 未知 binding/converter、biz 不匹配、biz 非字符串 | 静态链与部署链均失败;Authoring 不回退默认业务边界 | +| T08 | 缺必需槽位、未知槽位、type/allocator/params/capacity 错误 | 同一 slot 规则;单槽主路径准确;合法可选槽位省略成功 | +| T09 | override 的 `..`/越界绝对路径/符号链接逃逸 | UnderRoot 拒绝并指向覆盖项;无覆盖指向模型路径;词法/文件系统边界分别断言 | +| T10 | 相对路径,静态模式无根目录 | 不访问或要求模型文件;UnderRoot 用宿主根解析;不给根目录的静态测试仍通过 | +| T11 | 一槽预算、两槽合计预算、默认深度、非默认深度、depth=0/超过上限 | 保留全部现有预算行为;失败在模型/池物化前;静态 CLI 不冒充深度验收 | +| T12 | 准备成功,Core 同时存在多个 Node/DAG 错误 | 完整 diagnostics、related_nodes、可用层级保留,不压缩成首条字符串 | +| T13 | 可修复缺依赖/Node 配置拼写错误 + 完整 deployment | Explain 有效;候选补丁应用于原始完整文档后重跑静态链得到预期结果 | +| T14 | edit 无效 deployment,require_valid=false/true | 前者返回草稿及失败 validation;后者失败且不返回修改文档 | +| T15 | fix-deps 可修依赖,但 deployment 无效 | 不写入;部署原始错误可见;合法部署下原 preview/in-place/idempotence 测试仍通过 | +| T16 | plan 准备失败 / Core 失败 / 成功 | 分别为空计划 / Core 部分计划 / 无 diagnostics 的成功 envelope | +| T17 | init --empty、profile clone | 保留生成能力;不要求空草稿已经合法 | +| T18 | 先成功准备,再用同一输出变量准备失败 | 输出重置;无残留 binding/converter/path;输入 JSON 深比较不变 | +| T19 | resolve-conf 成功、错误;validate-io 错误 | 成功 schema/source/defaults 不变;错误外层 code 保留,validate-io 不再依赖文案提取 path | +| T20 | Init 失败、异常、重复准备;不执行真实模型 | 版本化错误/现有 SDK 错误屏障;无模型、Node、Session 或输出池实例化 | + +CLI parity 至少参数化 T03、T06、T07、T08、T12、T16;不要只测试准备函数本身, +否则工具可能仍绕过共享实现。T03–T08 的部署链至少覆盖内存 resolver 和一个文件入口。 +含 mock Model/Backend 的 fixture 全程使用 `alg_pipeline_tool_test`。 + +### 4.3 实施阶段的聚焦命令 + +默认分片构建可使用以下命令。它们是开发反馈,不代替最终门禁: + +```bash +cmake --build build --target edgeflow_test_adapter_runner edgeflow_test_tooling_runner alg_pipeline_tool alg_pipeline_tool_test +ctest --test-dir build --output-on-failure -R '^(IoBindingRegistryTest|AdapterContractSecurityTest|OperatorApiTest|PipelineStudioTest)$' +LLM_EDGEFLOW_PIPELINE_TOOL="$PWD/build/alg_pipeline_tool_test" python3 tests/tooling/test_pipeline_studio.py PipelineCliTest Rfc0057AuthoringAndDeploymentTest +``` + +Python 测试若涉及完整 Studio/Demo 运行,按已配置的 `PipelineStudioServerTest` CTest +环境执行,使用其 selection tool、Demo、alg_show 路径;不要改用生产 tool 执行 mock fixture。 +非默认构建优先使用相同 CTest 名称,避免假定测试 runner 文件名固定。 + +源代码、测试、实施文档及 CHANGELOG 全部就绪后,按 +[CONTRIBUTING](../../CONTRIBUTING.md#6-run-one-canonical-delivery-gate) +执行一次 canonical gate: + +```bash +./scripts/run_all_tests.sh +``` + +不得在其前后例行重复全量 build/CTest。它不验证真实模型效果或目标硬件;本 RFC 也不要求 +为解析重构新增真实模型验收。缺少某 Backend 的构建应保留相应静态失败,不能借 test 注册绕过。 + +### 4.4 完成检查表 + +- [x] `ResolveDeploymentBoundary` 与 Authoring 自行构造部署边界的实现已移除。 +- [x] binding、槽位、覆盖及边界规则在 Integration 只有一份流程实现。 +- [x] 原始声明在覆盖前由 Core parser 检查,T03/T04 通过。 +- [x] `validate --explain` 保留完整报告和可应用的修复候选;plan envelope 不变。 +- [x] 无效草稿可以编辑,require_valid 与 fix-deps 写入边界不变。 +- [x] 文件/内存/SDK 预检得到相同准备结果;路径来源和转义正确。 +- [x] `validate-io` 不再从 message 解析 JSON Pointer。 +- [x] 现有深度/容量/多槽合计预算和异常屏障测试通过。 +- [x] 新头文件只在内部可见,Core 与能力节点层未引入 Integration 依赖。 +- [x] canonical gate 通过;记录实际命令、结果和跳过项。 +- [x] 实施后的当前指南与 CHANGELOG 已更新,RFC 与索引状态同步。 + +## 5. 实施顺序与最终结果 + +### 5.1 推荐实施顺序 + +按下表推进,每阶段完成后再进行依赖它的阶段。分支和交付流程遵循 CONTRIBUTING, +本表只规定本次实现的依赖关系与验收点。 + +| 阶段 | 工作 | 阶段完成证据 | +| --- | --- | --- | +| P0 契约基线 | 阅读本文关联实现,补 CLI envelope、Explain 和 Authoring 草稿的现有行为断言;新增 T03 的失败回归用例 | 原有契约用例通过;T03 在旧实现上能暴露问题,不能写成接受旧错误行为 | +| P1 共享准备 | 新增诊断载体、PreparedDeployment 和 S1–S7;扩展模型路径 resolver 的结构化错误 | Adapter 聚焦测试覆盖原始结构、槽位、覆盖、来源、失败清空;暂不删除旧入口 | +| P2 运行入口迁移 | IoBindingResolver 转共享准备,传递结构化错误,保留预算和成功计划组装;OperatorConfigResolver 贯通诊断 | 文件/内存/SDK 配置预检及预算测试通过 | +| P3 CLI 迁移 | 新增工具适配,迁移 validate/plan/validate-io,删除旧 helper 和文案路径解析;处理注册生命周期 | 真实子进程 parity、envelope、explain 测试通过 | +| P4 Authoring 迁移 | edit/fix-deps 使用同一工具适配;保留 typed Core report、草稿语义和最终完整验证 | T13–T17 通过,错误部署不被静默忽略,文件写入回归通过 | +| P5 收尾 | 清理无用 include/重复代码,更新当前指南和 CHANGELOG,独立审阅,再执行一次完整门禁 | 本文完成表全部满足,记录验证结果后完成 RFC | + +P1 的旧入口临时保留仅用于同一实施分支内迁移;不得把“新 helper 已加入、旧入口仍工作” +作为最终交付。不要维护运行时新旧解析开关。 + +独立审阅重点:层级依赖、路径来源、失败结果原子性、注册生命周期、Explain 修复及所有旧入口 +是否实际接入共享实现。机械迁移可在接口和诊断映射确定后分工;不要并行争用同一构建目录。 + +### 5.2 实施记录与验证结论 + +- **实施完成状态**:已完成全部 P0–P5 阶段实施与测试,所有入口统一收敛至 `PrepareDeploymentDocument`。 +- **核心变更汇总**: + 1. 结构化诊断与共享准备:新增 `src/adapter/deployment_diagnostic.h`、`src/adapter/deployment_preparation.h/.cpp`(实现 S1–S7 处理流程)。 + 2. 核心结构前置拦截:在应用模型路径覆盖前,严格调用 Core 解析器 `ParsePipelineConfig` 检查原始文档结构。原始模型路径缺失、类型非字符串、空串或模型结构非法均在覆盖生效前被拒绝并准确指出 `/models//model_path`(T03/T04)。 + 3. 诊断路径来源投影:新增 `ProjectModelPathDiagnostics`,将对有效模型的诊断精准映射回原始输入对应位置(覆盖项映射至 `/deployment/model_paths/`,原始项映射至 `/models//model_path`)。 + 4. 运行时入口迁移:`io_binding_resolver.cpp`、`deployment_io_config.cpp`、`deployment_model_resolver.cpp`、`operator_config_resolver.cpp` 全面贯通 `DeploymentDiagnostic`,由 `PrepareDeploymentDocument` 统一解析部署边界后进行默认深度句柄预算及 Core 规划。 + 5. CLI 与 Authoring 统一:新增 `src/tools/pipeline_document_validation.h/.cpp`,废弃并移除 `alg_pipeline_tool.cpp` 中的 `ResolveDeploymentBoundary` 及 `pipeline_authoring.cpp` 中的重复边界构造。`validate`、`validate --explain`、`plan`、`edit` 与 `fix-deps` 均接入统一准备。 + 6. 契约修复与错误原子性:`validate-io` 直接透传底层结构化诊断;`PrepareDeploymentDocument` 失败时原子清空输出状态,无残留对象。 +- **聚焦测试验证记录**: + - `IoBindingRegistryTest`(23/23 通过):全面覆盖 T01–T08、T18、诊断投影以及内存/文件入口(T03–T08)完整诊断。 + - `AdapterContractSecurityTest`、`OperatorApiTest`、`PipelineStudioTest` 均通过。 + - `tests/tooling/test_pipeline_studio.py`(85/85 通过,1 skipped 为无真实 Chromium 环境跳过):全面覆盖 CLI Parity(T03、T06、T07、T08、T12、T16)、T14(编辑事务边界)、T15(依赖修复写入保护与诊断)、T19 以及 RFC-0057 编辑/修复回归。 +- **架构与边界一致性**: + - 新头文件全部位于私有目录(`src/adapter/` 与 `src/tools/`),不暴露任何公共 SDK 头文件或跨层污染。 + - Core 与能力节点层保持中性,无反向 Integration 依赖。 diff --git a/doc/rfcs/README.md b/doc/rfcs/README.md index d1cb9f2..0c91841 100644 --- a/doc/rfcs/README.md +++ b/doc/rfcs/README.md @@ -87,6 +87,7 @@ RFC-0054 是接续 RFC-0052 与已交付 RFC-0053、RFC-0055 的实施规格。` | **RFC-0059** | 输入输出转换独立化与接入绑定架构 | `Completed` | 投产前 / ABI 6.0.0 | 接入适配层、流程编排层 / Tooling / Docs | [0059-independent-adapter-input-and-output.md](0059-independent-adapter-input-and-output.md)(宿主双入口部分被 RFC-0060 取代) | | **RFC-0060** | 删除 C ABI,仅保留 C++ Operator API | `Completed` | `v11.0.0` / ABI 7.0.0 | 接入适配层、流程编排层 / Tooling / Docs | [0060-cpp-operator-only.md](0060-cpp-operator-only.md) | | **RFC-0061** | Pipeline JSON 集中管理部署配置 | `Completed` | `v11.x` | 接入适配层、流程编排层 / Tooling / Docs | [0061-pipeline-owned-deployment-configuration.md](0061-pipeline-owned-deployment-configuration.md) | +| **RFC-0062** | Integration 部署解析入口统一实施设计 | `Completed` | 投产前 | 接入适配层、流程编排层 / Tooling | [0062-unified-integration-deployment-preparation.md](0062-unified-integration-deployment-preparation.md) | ## 专项验收与评审归档 diff --git a/src/adapter/CMakeLists.txt b/src/adapter/CMakeLists.txt index 0ad8a84..e60aa14 100644 --- a/src/adapter/CMakeLists.txt +++ b/src/adapter/CMakeLists.txt @@ -1,5 +1,6 @@ target_sources(edgeflow_integration_objects PRIVATE deployment_model_resolver.cpp + deployment_preparation.cpp io_converter_registry.cpp io_binding_registry.cpp deployment_io_config.cpp diff --git a/src/adapter/deployment_diagnostic.h b/src/adapter/deployment_diagnostic.h new file mode 100644 index 0000000..e615f59 --- /dev/null +++ b/src/adapter/deployment_diagnostic.h @@ -0,0 +1,33 @@ +#pragma once + +#include +#include + +#include "core/diagnostic_code.h" +#include "core/pipeline_diagnostic.h" + +namespace llm_edgeflow { + +/** + * @brief 轻量级部署诊断载体 (RFC-0062) + * + * 用于在 Integration 内部及工具调用面跨函数传递结构化错误 (code, path, message, + * legacy_status). + */ +struct DeploymentDiagnostic { + std::string code; // Integration 错误码,或 DiagnosticCodeName 的结果 + std::string path; // 原始完整文档的 RFC 6901 JSON Pointer + std::string message; + int legacy_status = -2; + std::optional pipeline_diagnostic; + + void Clear() { + code.clear(); + path.clear(); + message.clear(); + legacy_status = -2; + pipeline_diagnostic.reset(); + } +}; + +} // namespace llm_edgeflow diff --git a/src/adapter/deployment_io_config.cpp b/src/adapter/deployment_io_config.cpp index 98897b7..3adf1b8 100644 --- a/src/adapter/deployment_io_config.cpp +++ b/src/adapter/deployment_io_config.cpp @@ -3,24 +3,43 @@ #include #include +#include "adapter/pipeline_document.h" #include "contracts/path_utils.h" namespace llm_edgeflow { namespace fs = std::filesystem; +static void SetConfigDiag(DeploymentDiagnostic* out_diag, + const std::string& code, const std::string& path, + const std::string& message) { + if (out_diag) { + out_diag->code = code; + out_diag->path = path; + out_diag->message = message; + out_diag->legacy_status = -2; + out_diag->pipeline_diagnostic.reset(); + } +} + bool DeploymentIoConfig::ReadFromFile(const std::string& config_path, const std::string& transport, DeploymentIoConfig* out_config, - std::string* out_error) { + std::string* out_error, + DeploymentDiagnostic* out_diagnostic) { + if (out_diagnostic) out_diagnostic->Clear(); + if (config_path.empty()) { if (out_error) *out_error = "Empty config_path"; + SetConfigDiag(out_diagnostic, "DEPLOYMENT_ERROR", "/", "Empty config_path"); return false; } std::ifstream ifs(config_path); if (!ifs.is_open()) { - if (out_error) *out_error = "Failed to open config file: " + config_path; + std::string msg = "Failed to open config file: " + config_path; + if (out_error) *out_error = msg; + SetConfigDiag(out_diagnostic, "CONFIG_FILE_OPEN", "/", msg); return false; } @@ -28,9 +47,10 @@ bool DeploymentIoConfig::ReadFromFile(const std::string& config_path, try { ifs >> root; } catch (const std::exception& e) { - if (out_error) { - *out_error = "JSON parse exception in " + config_path + ": " + e.what(); - } + std::string msg = + "JSON parse exception in " + config_path + ": " + e.what(); + if (out_error) *out_error = msg; + SetConfigDiag(out_diagnostic, "JSON_PARSE", "/", msg); return false; } @@ -40,29 +60,50 @@ bool DeploymentIoConfig::ReadFromFile(const std::string& config_path, } cfg_dir = fs::absolute(cfg_dir); - return Parse(root, cfg_dir.string(), transport, out_config, out_error); + std::string parse_err; + DeploymentDiagnostic parse_diag; + bool ok = Parse(root, cfg_dir.string(), transport, out_config, &parse_err, + &parse_diag); + if (!ok) { + std::string prefix = "Error in config file " + config_path + ": "; + if (out_error) *out_error = prefix + parse_err; + if (out_diagnostic) { + *out_diagnostic = parse_diag; + out_diagnostic->message = prefix + out_diagnostic->message; + } + return false; + } + if (out_diagnostic) *out_diagnostic = parse_diag; + return true; } bool DeploymentIoConfig::Parse(const nlohmann::json& root, const std::string& config_dir, const std::string& transport, DeploymentIoConfig* out_config, - std::string* out_error) { + std::string* out_error, + DeploymentDiagnostic* out_diagnostic) { + if (out_diagnostic) out_diagnostic->Clear(); + if (!out_config) { if (out_error) *out_error = "Null out_config pointer"; + SetConfigDiag(out_diagnostic, "DEPLOYMENT_ERROR", "/", + "Null out_config pointer"); return false; } if (!root.is_object()) { if (out_error) *out_error = "Root configuration must be a JSON object"; + SetConfigDiag(out_diagnostic, "DEPLOYMENT_ERROR", "/", + "Root configuration must be a JSON object"); return false; } if (transport != "operator") { - if (out_error) { - *out_error = "Unsupported transport: '" + transport + - "' (only 'operator' is supported)"; - } + std::string msg = "Unsupported transport: '" + transport + + "' (only 'operator' is supported)"; + if (out_error) *out_error = msg; + SetConfigDiag(out_diagnostic, "UNSUPPORTED_TRANSPORT", "/", msg); return false; } @@ -70,16 +111,18 @@ bool DeploymentIoConfig::Parse(const nlohmann::json& root, for (const char* deprecated_key : {"data", "schema_version", "io_binding", "model_paths", "outputs"}) { if (root.contains(deprecated_key)) { - if (out_error) { - *out_error = - std::string( - "Deprecated deployment configuration format (RFC-0061) at /") + - deprecated_key + - ": '.conf' files must contain only 'pipe_path'. Deployment " - "configuration " - "(io_binding, output_allocations, model_paths) has moved to the " - "'deployment' section inside the Pipeline JSON."; - } + std::string escaped_key = EscapeJsonPointer(deprecated_key); + std::string msg = + std::string( + "Deprecated deployment configuration format (RFC-0061) at /") + + escaped_key + + ": '.conf' files must contain only 'pipe_path'. Deployment " + "configuration " + "(io_binding, output_allocations, model_paths) has moved to the " + "'deployment' section inside the Pipeline JSON."; + if (out_error) *out_error = msg; + SetConfigDiag(out_diagnostic, "DEPLOYMENT_ERROR", + std::string("/") + escaped_key, msg); return false; } } @@ -87,10 +130,11 @@ bool DeploymentIoConfig::Parse(const nlohmann::json& root, // 2. 根字段白名单: 必须有且仅有 pipe_path for (auto it = root.begin(); it != root.end(); ++it) { if (it.key() != "pipe_path") { - if (out_error) { - *out_error = "Unknown field at /: '" + it.key() + - "' (only 'pipe_path' is allowed under RFC-0061)"; - } + std::string escaped_key = EscapeJsonPointer(it.key()); + std::string msg = "Unknown field at /: '" + it.key() + + "' (only 'pipe_path' is allowed under RFC-0061)"; + if (out_error) *out_error = msg; + SetConfigDiag(out_diagnostic, "DEPLOYMENT_ERROR", "/" + escaped_key, msg); return false; } } @@ -98,6 +142,8 @@ bool DeploymentIoConfig::Parse(const nlohmann::json& root, if (!root.contains("pipe_path") || !root["pipe_path"].is_string() || root["pipe_path"].get().empty()) { if (out_error) *out_error = "Missing or empty 'pipe_path'"; + SetConfigDiag(out_diagnostic, "DEPLOYMENT_ERROR", "/pipe_path", + "Missing or empty 'pipe_path'"); return false; } @@ -116,27 +162,28 @@ bool DeploymentIoConfig::Parse(const nlohmann::json& root, fs::path canonical_pipe = fs::weakly_canonical(full_pipe, ec); if (!IsPathWithinRoot(canonical_base, canonical_pipe)) { - if (out_error) { - *out_error = - "pipe_path escapes config directory: " + out_config->pipe_path; - } + std::string msg = + "pipe_path escapes config directory: " + out_config->pipe_path; + if (out_error) *out_error = msg; + SetConfigDiag(out_diagnostic, "DEPLOYMENT_ERROR", "/pipe_path", msg); return false; } if (!fs::exists(canonical_pipe)) { - if (out_error) { - *out_error = "Pipeline file does not exist: " + canonical_pipe.string(); - } + std::string msg = + "Pipeline file does not exist: " + canonical_pipe.string(); + if (out_error) *out_error = msg; + SetConfigDiag(out_diagnostic, "DEPLOYMENT_ERROR", "/pipe_path", msg); return false; } // 校验符号链接目标,防止符号链接逃出配置目录 fs::path real_pipe = fs::canonical(canonical_pipe, ec); if (ec || !IsPathWithinRoot(canonical_base, real_pipe)) { - if (out_error) { - *out_error = - "pipe_path escapes config directory: " + out_config->pipe_path; - } + std::string msg = + "pipe_path escapes config directory: " + out_config->pipe_path; + if (out_error) *out_error = msg; + SetConfigDiag(out_diagnostic, "DEPLOYMENT_ERROR", "/pipe_path", msg); return false; } diff --git a/src/adapter/deployment_io_config.h b/src/adapter/deployment_io_config.h index e5b83f9..f70809f 100644 --- a/src/adapter/deployment_io_config.h +++ b/src/adapter/deployment_io_config.h @@ -3,6 +3,8 @@ #include #include +#include "adapter/deployment_diagnostic.h" + namespace llm_edgeflow { /** @@ -15,12 +17,14 @@ struct DeploymentIoConfig { static bool Parse(const nlohmann::json& root, const std::string& config_dir, const std::string& transport, - DeploymentIoConfig* out_config, std::string* out_error); + DeploymentIoConfig* out_config, std::string* out_error, + DeploymentDiagnostic* out_diagnostic = nullptr); static bool ReadFromFile(const std::string& config_path, const std::string& transport, DeploymentIoConfig* out_config, - std::string* out_error); + std::string* out_error, + DeploymentDiagnostic* out_diagnostic = nullptr); }; } // namespace llm_edgeflow diff --git a/src/adapter/deployment_model_resolver.cpp b/src/adapter/deployment_model_resolver.cpp index 9ff0e7f..5ea3eec 100644 --- a/src/adapter/deployment_model_resolver.cpp +++ b/src/adapter/deployment_model_resolver.cpp @@ -12,8 +12,18 @@ namespace { namespace fs = std::filesystem; -void SetDiagnostic(std::string* diagnostic, const std::string& message) { +void SetDiagnostic(std::string* diagnostic, + DeploymentDiagnostic* out_diagnostic, + const std::string& code, const std::string& path, + const std::string& message, int legacy_status = -2) { if (diagnostic) *diagnostic = message; + if (out_diagnostic) { + out_diagnostic->code = code; + out_diagnostic->path = path; + out_diagnostic->message = message; + out_diagnostic->legacy_status = legacy_status; + out_diagnostic->pipeline_diagnostic.reset(); + } } } // namespace @@ -21,10 +31,13 @@ void SetDiagnostic(std::string* diagnostic, const std::string& message) { bool ResolveDeploymentModelPaths( const nlohmann::json& pipeline_json, const std::string& model_root_dir, nlohmann::json* resolved_pipeline_json, std::string* diagnostic, - const std::unordered_set& overridden_model_ids) noexcept { + const std::unordered_set& overridden_model_ids, + DeploymentDiagnostic* out_diagnostic) noexcept { try { + if (out_diagnostic) out_diagnostic->Clear(); if (!resolved_pipeline_json) { - SetDiagnostic(diagnostic, "Deployment model resolver output is null"); + SetDiagnostic(diagnostic, out_diagnostic, "DEPLOYMENT_ERROR", "/", + "Deployment model resolver output is null", -2); return false; } @@ -50,15 +63,17 @@ bool ResolveDeploymentModelPaths( std::error_code error; const fs::path absolute_root = fs::absolute(model_root_dir, error); if (error) { - SetDiagnostic(diagnostic, "Failed to make model_root_dir absolute: " + - model_root_dir); + SetDiagnostic( + diagnostic, out_diagnostic, "DEPLOYMENT_ERROR", "/", + "Failed to make model_root_dir absolute: " + model_root_dir, -2); return false; } canonical_root = fs::weakly_canonical(absolute_root, error); if (error || !fs::is_directory(canonical_root, error) || error) { SetDiagnostic( - diagnostic, - "model_root_dir is not an accessible directory: " + model_root_dir); + diagnostic, out_diagnostic, "DEPLOYMENT_ERROR", "/", + "model_root_dir is not an accessible directory: " + model_root_dir, + -2); return false; } } @@ -83,16 +98,18 @@ bool ResolveDeploymentModelPaths( const std::string raw_path = model["model_path"].get(); const fs::path normalized = fs::path(raw_path).lexically_normal(); if (!normalized.is_absolute() && HasParentPathComponent(normalized)) { - SetDiagnostic(diagnostic, + SetDiagnostic(diagnostic, out_diagnostic, "INVALID_MODEL_PATH", pointer, "Model path cannot traverse outside model_root_dir at " + - pointer + ": " + raw_path); + pointer + ": " + raw_path, + -2); return false; } if (!normalized.is_absolute() && canonical_root.empty()) { SetDiagnostic( - diagnostic, + diagnostic, out_diagnostic, "INVALID_MODEL_PATH", pointer, "Relative model_path requires non-empty model_root_dir at " + - pointer + ": " + raw_path); + pointer + ": " + raw_path, + -2); return false; } @@ -101,26 +118,32 @@ bool ResolveDeploymentModelPaths( normalized.is_absolute() ? normalized : canonical_root / normalized, error); if (error) { - SetDiagnostic(diagnostic, + SetDiagnostic(diagnostic, out_diagnostic, "INVALID_MODEL_PATH", pointer, "Failed to resolve deployment model path at " + pointer + - ": " + raw_path); + ": " + raw_path, + -2); return false; } if (!canonical_root.empty() && !IsPathWithinRoot(canonical_root, candidate)) { - SetDiagnostic(diagnostic, "Model path escapes model_root_dir at " + - pointer + ": " + raw_path); + SetDiagnostic( + diagnostic, out_diagnostic, "INVALID_MODEL_PATH", pointer, + "Model path escapes model_root_dir at " + pointer + ": " + raw_path, + -2); return false; } model["model_path"] = candidate.string(); } return true; } catch (const std::exception& exception) { - SetDiagnostic(diagnostic, std::string("Deployment model path exception: ") + - exception.what()); + SetDiagnostic( + diagnostic, out_diagnostic, "INTERNAL_EXCEPTION", "/", + std::string("Deployment model path exception: ") + exception.what(), + -2); return false; } catch (...) { - SetDiagnostic(diagnostic, "Unknown deployment model path exception"); + SetDiagnostic(diagnostic, out_diagnostic, "INTERNAL_EXCEPTION", "/", + "Unknown deployment model path exception", -2); return false; } } diff --git a/src/adapter/deployment_model_resolver.h b/src/adapter/deployment_model_resolver.h index 47e3aca..ddd2e83 100644 --- a/src/adapter/deployment_model_resolver.h +++ b/src/adapter/deployment_model_resolver.h @@ -4,6 +4,8 @@ #include #include +#include "adapter/deployment_diagnostic.h" + namespace llm_edgeflow { /** @@ -17,6 +19,7 @@ namespace llm_edgeflow { bool ResolveDeploymentModelPaths( const nlohmann::json& pipeline_json, const std::string& model_root_dir, nlohmann::json* resolved_pipeline_json, std::string* diagnostic, - const std::unordered_set& overridden_model_ids = {}) noexcept; + const std::unordered_set& overridden_model_ids = {}, + DeploymentDiagnostic* out_diagnostic = nullptr) noexcept; } // namespace llm_edgeflow diff --git a/src/adapter/deployment_preparation.cpp b/src/adapter/deployment_preparation.cpp new file mode 100644 index 0000000..e981492 --- /dev/null +++ b/src/adapter/deployment_preparation.cpp @@ -0,0 +1,438 @@ +#include "adapter/deployment_preparation.h" + +#include +#include +#include +#include + +#include "adapter/deployment_model_resolver.h" +#include "adapter/io_binding_registry.h" +#include "adapter/io_converter_registry.h" +#include "adapter/operator/operator_config_resolver.h" +#include "adapter/pipeline_document.h" +#include "core/diagnostic_code.h" +#include "core/pipeline_catalog.h" +#include "core/pipeline_config.h" + +namespace llm_edgeflow { + +bool PrepareDeploymentDocument(const nlohmann::json& document, + const DeploymentPrepareOptions& options, + PreparedDeployment* output, + DeploymentDiagnostic* diagnostic) { + if (!output) { + if (diagnostic) { + diagnostic->code = "DEPLOYMENT_ERROR"; + diagnostic->path = "/"; + diagnostic->message = "Output pointer is null"; + diagnostic->legacy_status = -1; + diagnostic->pipeline_diagnostic.reset(); + } + return false; + } + + output->Clear(); + if (diagnostic) { + diagnostic->Clear(); + } + + // S1: 检查调用参数与部署文档结构 + if (options.transport != "operator") { + if (diagnostic) { + diagnostic->code = "UNSUPPORTED_TRANSPORT"; + diagnostic->path = "/"; + diagnostic->message = "Unsupported transport: '" + options.transport + + "' (only 'operator' is supported)"; + diagnostic->legacy_status = -2; + } + return false; + } + + if (options.path_mode == DeploymentPathMode::kUnderRoot && + options.model_root_dir.empty()) { + if (diagnostic) { + diagnostic->code = "DEPLOYMENT_ERROR"; + diagnostic->path = "/"; + diagnostic->message = + "model_root_dir cannot be empty when path_mode is kUnderRoot"; + diagnostic->legacy_status = -2; + } + return false; + } + + if (options.path_mode == DeploymentPathMode::kLexicalOnly && + !options.model_root_dir.empty()) { + if (diagnostic) { + diagnostic->code = "DEPLOYMENT_ERROR"; + diagnostic->path = "/"; + diagnostic->message = + "model_root_dir must be empty when path_mode is kLexicalOnly"; + diagnostic->legacy_status = -2; + } + return false; + } + + PipelineDocumentSplit doc_split; + std::string split_err; + std::string split_path; + if (!SplitPipelineDocument(document, &doc_split, &split_err, &split_path)) { + if (diagnostic) { + diagnostic->code = "DEPLOYMENT_ERROR"; + diagnostic->path = split_path.empty() ? "/" : split_path; + diagnostic->message = split_err; + diagnostic->legacy_status = -2; + } + return false; + } + + if (!doc_split.has_deployment || !doc_split.deployment.has_io) { + if (diagnostic) { + diagnostic->code = "MISSING_DEPLOYMENT_IO"; + diagnostic->path = "/deployment/io"; + diagnostic->message = "Missing required 'deployment.io' in pipeline JSON"; + diagnostic->legacy_status = -2; + } + return false; + } + + // S2: 覆盖之前验证原始中性结构 + ParsedPipelineConfig original_config; + PipelineDiagnostic core_diag; + if (!ParsePipelineConfig(doc_split.neutral_pipeline_json, &original_config, + &core_diag)) { + if (diagnostic) { + diagnostic->pipeline_diagnostic = core_diag; + bool is_model_path = false; + if (core_diag.path.rfind("/models/", 0) == 0) { + auto second_slash = core_diag.path.find('/', 8); + if (second_slash != std::string::npos && + core_diag.path.substr(second_slash) == "/model_path") { + is_model_path = true; + } + } + + if (is_model_path && (core_diag.code == DiagnosticCode::kFieldType || + core_diag.code == DiagnosticCode::kFieldRange)) { + diagnostic->code = "INVALID_MODEL_PATH"; + diagnostic->path = core_diag.path; + diagnostic->message = + "model_path in model declaration must be a non-empty string (at " + + core_diag.path + ")"; + diagnostic->legacy_status = -2; + } else { + diagnostic->code = DiagnosticCodeName(core_diag.code); + diagnostic->path = core_diag.path; + diagnostic->message = "Validation failed: " + + std::string(DiagnosticCodeName(core_diag.code)) + + " at " + core_diag.path + ": " + + core_diag.message; + diagnostic->legacy_status = -3; + } + } + return false; + } + + // S3: 解析 binding、converter 和业务边界 + const std::string& binding_id = doc_split.deployment.io.io_binding; + const auto* binding = IoBindingRegistry::Instance().FindBinding(binding_id); + if (!binding) { + if (diagnostic) { + diagnostic->code = "UNKNOWN_IO_BINDING"; + diagnostic->path = "/deployment/io/io_binding"; + diagnostic->message = + "Unknown or unregistered io_binding: " + binding_id + + " (at /deployment/io/io_binding)"; + diagnostic->legacy_status = -2; + } + return false; + } + + if (binding->transport != "operator") { + if (diagnostic) { + diagnostic->code = "UNSUPPORTED_TRANSPORT"; + diagnostic->path = "/deployment/io/io_binding"; + diagnostic->message = "Binding transport mismatch for '" + binding_id + + "': expected 'operator', but binding declared '" + + binding->transport + + "' (at /deployment/io/io_binding)"; + diagnostic->legacy_status = -2; + } + return false; + } + + if (original_config.biz_name != binding->biz_name) { + if (diagnostic) { + diagnostic->code = "BIZ_MISMATCH"; + diagnostic->path = "/deployment/io/io_binding"; + diagnostic->message = "Pipeline biz_name '" + original_config.biz_name + + "' does not match binding biz_name '" + + binding->biz_name + + "' (at /deployment/io/io_binding)"; + diagnostic->legacy_status = -2; + } + return false; + } + + const auto* in_conv = IoConverterRegistry::Instance().FindInputConverter( + binding->input_converter_id); + if (!in_conv) { + if (diagnostic) { + diagnostic->code = "UNREGISTERED_CONVERTER"; + diagnostic->path = "/deployment/io/io_binding"; + diagnostic->message = + "Binding references unregistered input converter: " + + binding->input_converter_id; + diagnostic->legacy_status = -2; + } + return false; + } + + const auto* out_conv = IoConverterRegistry::Instance().FindOutputConverter( + binding->output_converter_id); + if (!out_conv) { + if (diagnostic) { + diagnostic->code = "UNREGISTERED_CONVERTER"; + diagnostic->path = "/deployment/io/io_binding"; + diagnostic->message = + "Binding references unregistered output converter: " + + binding->output_converter_id; + diagnostic->legacy_status = -2; + } + return false; + } + + size_t max_batch = + std::min(in_conv->max_batch_size, out_conv->max_batch_size); + const auto* exposure = + IoBindingRegistry::Instance().FindExposure(binding->biz_name); + if (exposure) { + max_batch = std::min(max_batch, exposure->max_batch_size); + } + + // S4: 解析输出分配 (R ⊆ C ⊆ A) + const auto& allocations = doc_split.deployment.io.output_allocations; + std::unordered_map local_output_specs; + std::unordered_map local_output_params; + + for (auto it = allocations.begin(); it != allocations.end(); ++it) { + bool found = false; + for (const auto& slot : out_conv->external_slots) { + if (slot.direction == PortDirection::kOutput && + slot.slot_name == it.key()) { + found = true; + break; + } + } + if (!found) { + if (diagnostic) { + std::string ptr = + "/deployment/io/output_allocations/" + EscapeJsonPointer(it.key()); + diagnostic->code = "UNKNOWN_OUTPUT_SLOT"; + diagnostic->path = ptr; + diagnostic->message = + "Unknown configured output slot: " + it.key() + " (at " + ptr + ")"; + diagnostic->legacy_status = -2; + } + return false; + } + } + + for (const auto& slot : out_conv->external_slots) { + if (slot.direction != PortDirection::kOutput) continue; + if (!allocations.contains(slot.slot_name)) { + if (slot.required) { + if (diagnostic) { + std::string ptr = "/deployment/io/output_allocations/" + + EscapeJsonPointer(slot.slot_name); + diagnostic->code = "MISSING_OUTPUT_SLOT"; + diagnostic->path = ptr; + diagnostic->message = "Missing required Operator output slot '" + + slot.slot_name + "' (at " + ptr + ")"; + diagnostic->legacy_status = -2; + } + return false; + } + continue; + } + ResolvedOutputPoolSpec pool_spec; + std::string param_text; + std::string alloc_err; + int alloc_ret = OperatorConfigResolver::ResolveOutputAllocation( + allocations[slot.slot_name], slot, &pool_spec, ¶m_text, &alloc_err); + if (alloc_ret != 0) { + if (diagnostic) { + std::string ptr = "/deployment/io/output_allocations/" + + EscapeJsonPointer(slot.slot_name); + diagnostic->code = "INVALID_OUTPUT_ALLOCATION"; + diagnostic->path = ptr; + diagnostic->message = alloc_err + " (at " + ptr + ")"; + diagnostic->legacy_status = -2; + } + return false; + } + local_output_specs[slot.slot_name] = std::move(pool_spec); + local_output_params[slot.slot_name] = std::move(param_text); + } + + // S5: 应用模型路径覆盖并记录来源 + std::unordered_map model_id_to_index; + for (size_t i = 0; i < original_config.models.size(); ++i) { + model_id_to_index[original_config.models[i].model_id] = i; + } + + if (document.contains("deployment") && document["deployment"].is_object() && + document["deployment"].contains("model_paths") && + document["deployment"]["model_paths"].is_object()) { + for (const auto& [mid, _] : document["deployment"]["model_paths"].items()) { + if (model_id_to_index.find(mid) == model_id_to_index.end()) { + if (diagnostic) { + std::string ptr = "/deployment/model_paths/" + EscapeJsonPointer(mid); + diagnostic->code = "UNKNOWN_MODEL_ID"; + diagnostic->path = ptr; + diagnostic->message = "Unknown model_id '" + mid + + "' in '/deployment/model_paths' (at " + ptr + + ")"; + diagnostic->legacy_status = -2; + } + return false; + } + } + } + + std::vector model_path_source_pointers( + original_config.models.size()); + for (size_t i = 0; i < original_config.models.size(); ++i) { + model_path_source_pointers[i] = + "/models/" + std::to_string(i) + "/model_path"; + } + + nlohmann::json staged_neutral_json = doc_split.neutral_pipeline_json; + std::unordered_set overridden_model_ids; + if (doc_split.deployment.has_model_paths) { + for (const auto& [mid, override_path] : doc_split.deployment.model_paths) { + auto it = model_id_to_index.find(mid); + if (it != model_id_to_index.end()) { + size_t idx = it->second; + staged_neutral_json["models"][idx]["model_path"] = override_path; + overridden_model_ids.insert(mid); + model_path_source_pointers[idx] = + "/deployment/model_paths/" + EscapeJsonPointer(mid); + } + } + } + + // S6: 按模式处理有效路径 + nlohmann::json resolved_neutral_json; + if (options.path_mode == DeploymentPathMode::kUnderRoot) { + std::string model_err; + if (!ResolveDeploymentModelPaths( + staged_neutral_json, options.model_root_dir, &resolved_neutral_json, + &model_err, overridden_model_ids, diagnostic)) { + return false; + } + } else { + resolved_neutral_json = std::move(staged_neutral_json); + } + + // S7: 构造中性 I/O 边界,发布准备结果 + PipelineIoBoundary io_boundary; + for (const auto& port : in_conv->logical_ports) { + std::string key = port.logical_name; + auto bit = binding->input_ports.find(port.logical_name); + if (bit != binding->input_ports.end()) { + key = bit->second; + } + io_boundary.input_published_ports.emplace_back( + key, port.type_id, port.required, port.cardinality, + port.provenance_policy, port.lifetime, port.lifetime_config_field); + } + + for (const auto& port : out_conv->logical_ports) { + std::string key = port.logical_name; + auto bit = binding->output_ports.find(port.logical_name); + if (bit != binding->output_ports.end()) { + key = bit->second; + } + io_boundary.output_consumed_ports.emplace_back( + key, port.type_id, port.required, port.cardinality, + port.provenance_policy, port.lifetime, port.lifetime_config_field); + } + + PreparedDeployment local_prep; + local_prep.binding = *binding; + local_prep.input_converter = in_conv; + local_prep.output_converter = out_conv; + local_prep.input_port_bindings = InputPortBindings(binding->input_ports); + local_prep.output_port_bindings = OutputPortBindings(binding->output_ports); + local_prep.effective_max_batch_size = max_batch; + local_prep.output_specs = std::move(local_output_specs); + local_prep.output_parameter_texts = std::move(local_output_params); + local_prep.overridden_model_ids = std::move(overridden_model_ids); + local_prep.model_path_source_pointers = std::move(model_path_source_pointers); + local_prep.neutral_pipeline_json = std::move(resolved_neutral_json); + local_prep.io_boundary = std::move(io_boundary); + + *output = std::move(local_prep); + return true; +} + +void ProjectModelPathDiagnostics(const PreparedDeployment& prepared, + ValidationReport* report) { + if (!report) return; + + for (auto& diag : report->diagnostics) { + if (diag.path.rfind("/models/", 0) == 0) { + auto second_slash = diag.path.find('/', 8); + if (second_slash != std::string::npos && + diag.path.substr(second_slash) == "/model_path") { + std::string idx_str = diag.path.substr(8, second_slash - 8); + if (!idx_str.empty() && + std::all_of(idx_str.begin(), idx_str.end(), ::isdigit)) { + size_t idx = std::stoul(idx_str); + if (idx < prepared.model_path_source_pointers.size()) { + diag.path = prepared.model_path_source_pointers[idx]; + } + } + } + } + + if (diag.remediation.has_value()) { + auto& fixes = diag.remediation->fixes; + fixes.erase( + std::remove_if( + fixes.begin(), fixes.end(), + [&](const ValidationFix& fix) { + if (!fix.patch.is_array()) return false; + for (const auto& op : fix.patch) { + if (op.is_object() && op.contains("path") && + op["path"].is_string()) { + std::string p = op["path"].get(); + if (p.rfind("/models/", 0) == 0) { + auto second_slash = p.find('/', 8); + if (second_slash != std::string::npos && + p.substr(second_slash) == "/model_path") { + std::string idx_str = p.substr(8, second_slash - 8); + if (!idx_str.empty() && + std::all_of(idx_str.begin(), idx_str.end(), + ::isdigit)) { + size_t idx = std::stoul(idx_str); + if (idx < + prepared.model_path_source_pointers.size()) { + if (prepared.model_path_source_pointers[idx].rfind( + "/deployment/model_paths/", 0) == 0) { + return true; + } + } + } + } + } + } + } + return false; + }), + fixes.end()); + } + } +} + +} // namespace llm_edgeflow diff --git a/src/adapter/deployment_preparation.h b/src/adapter/deployment_preparation.h new file mode 100644 index 0000000..77a6d5b --- /dev/null +++ b/src/adapter/deployment_preparation.h @@ -0,0 +1,74 @@ +#pragma once + +#include +#include +#include +#include +#include + +#include "adapter/deployment_diagnostic.h" +#include "adapter/io_binding.h" +#include "adapter/io_converter.h" +#include "adapter/operator_io_contracts.h" +#include "core/pipeline_validator.h" + +namespace llm_edgeflow { + +enum class DeploymentPathMode { + kLexicalOnly, + kUnderRoot, +}; + +struct DeploymentPrepareOptions { + std::string transport = "operator"; + DeploymentPathMode path_mode = DeploymentPathMode::kLexicalOnly; + std::string model_root_dir; +}; + +struct PreparedDeployment { + IoBindingDefinition binding; + const InputConverterDefinition* input_converter = nullptr; + const OutputConverterDefinition* output_converter = nullptr; + InputPortBindings input_port_bindings; + OutputPortBindings output_port_bindings; + size_t effective_max_batch_size = 0; + + std::unordered_map output_specs; + std::unordered_map output_parameter_texts; + std::unordered_set overridden_model_ids; + std::vector model_path_source_pointers; // 与 models 原顺序对应 + + nlohmann::json neutral_pipeline_json; + PipelineIoBoundary io_boundary; + + void Clear() { + binding = IoBindingDefinition{}; + input_converter = nullptr; + output_converter = nullptr; + input_port_bindings = InputPortBindings{}; + output_port_bindings = OutputPortBindings{}; + effective_max_batch_size = 0; + output_specs.clear(); + output_parameter_texts.clear(); + overridden_model_ids.clear(); + model_path_source_pointers.clear(); + neutral_pipeline_json = nullptr; + io_boundary = PipelineIoBoundary{}; + } +}; + +/** + * @brief 准备部署文档(S1–S7 共享接入准备入口,RFC-0062) + */ +bool PrepareDeploymentDocument(const nlohmann::json& document, + const DeploymentPrepareOptions& options, + PreparedDeployment* output, + DeploymentDiagnostic* diagnostic); + +/** + * @brief 将有效模型路径的 Core 诊断归位投影回原文档来源路径(RFC-0062) + */ +void ProjectModelPathDiagnostics(const PreparedDeployment& prepared, + ValidationReport* report); + +} // namespace llm_edgeflow diff --git a/src/adapter/io_binding_resolver.cpp b/src/adapter/io_binding_resolver.cpp index cb3f72f..558aac4 100644 --- a/src/adapter/io_binding_resolver.cpp +++ b/src/adapter/io_binding_resolver.cpp @@ -3,16 +3,13 @@ #include #include #include -#include +#include +#include -#include "adapter/deployment_model_resolver.h" -#include "adapter/io_binding_registry.h" -#include "adapter/io_converter_registry.h" -#include "adapter/operator/operator_config_resolver.h" +#include "adapter/deployment_preparation.h" #include "adapter/operator/operator_value_type_registry.h" #include "adapter/pipeline_document.h" #include "core/diagnostic_code.h" -#include "core/pipeline_catalog.h" namespace llm_edgeflow { @@ -21,32 +18,50 @@ namespace fs = std::filesystem; int IoBindingResolver::ResolveFromFile( const std::string& config_path, const std::string& transport, const std::string& model_root_dir, - std::unique_ptr* out_plan, std::string* out_error) { + std::unique_ptr* out_plan, std::string* out_error, + DeploymentDiagnostic* out_diagnostic) { + if (out_diagnostic) out_diagnostic->Clear(); + DeploymentIoConfig config; std::string err; - if (!DeploymentIoConfig::ReadFromFile(config_path, transport, &config, - &err)) { + if (!DeploymentIoConfig::ReadFromFile(config_path, transport, &config, &err, + out_diagnostic)) { if (out_error) *out_error = err; return -2; } return ResolveFromConfig(config, transport, model_root_dir, out_plan, - out_error); + out_error, out_diagnostic); } int IoBindingResolver::ResolveFromConfig( const DeploymentIoConfig& config, const std::string& transport, const std::string& model_root_dir, - std::unique_ptr* out_plan, std::string* out_error) { + std::unique_ptr* out_plan, std::string* out_error, + DeploymentDiagnostic* out_diagnostic) { + if (out_diagnostic) out_diagnostic->Clear(); + if (!out_plan) { if (out_error) *out_error = "Null out_plan pointer"; + if (out_diagnostic) { + out_diagnostic->code = "DEPLOYMENT_ERROR"; + out_diagnostic->path = "/"; + out_diagnostic->message = "Null out_plan pointer"; + out_diagnostic->legacy_status = -1; + } return -1; } *out_plan = nullptr; std::ifstream pipe_ifs(config.resolved_pipe_path); if (!pipe_ifs.is_open()) { - if (out_error) { - *out_error = "Failed to open pipeline file: " + config.resolved_pipe_path; + std::string msg = + "Failed to open pipeline file: " + config.resolved_pipe_path; + if (out_error) *out_error = msg; + if (out_diagnostic) { + out_diagnostic->code = "CONFIG_FILE_OPEN"; + out_diagnostic->path = "/"; + out_diagnostic->message = msg; + out_diagnostic->legacy_status = -2; } return -2; } @@ -54,180 +69,73 @@ int IoBindingResolver::ResolveFromConfig( try { pipe_ifs >> raw_pipe_json; } catch (const std::exception& e) { - if (out_error) { - *out_error = "JSON parse exception in pipeline file: " + - config.resolved_pipe_path + ": " + e.what(); + std::string msg = + "JSON parse exception in pipeline file: " + config.resolved_pipe_path + + ": " + e.what(); + if (out_error) *out_error = msg; + if (out_diagnostic) { + out_diagnostic->code = "JSON_PARSE"; + out_diagnostic->path = "/"; + out_diagnostic->message = msg; + out_diagnostic->legacy_status = -2; } return -2; } return ResolveFromPipelineJson(raw_pipe_json, transport, model_root_dir, - out_plan, out_error); + out_plan, out_error, out_diagnostic); } int IoBindingResolver::ResolveFromPipelineJson( const nlohmann::json& pipeline_json, const std::string& transport, const std::string& model_root_dir, - std::unique_ptr* out_plan, std::string* out_error) { + std::unique_ptr* out_plan, std::string* out_error, + DeploymentDiagnostic* out_diagnostic) { + if (out_diagnostic) out_diagnostic->Clear(); + if (!out_plan) { if (out_error) *out_error = "Null out_plan pointer"; + if (out_diagnostic) { + out_diagnostic->code = "DEPLOYMENT_ERROR"; + out_diagnostic->path = "/"; + out_diagnostic->message = "Null out_plan pointer"; + out_diagnostic->legacy_status = -1; + } return -1; } *out_plan = nullptr; - // 1. 检查入口 transport - if (transport != "operator") { - if (out_error) { - *out_error = "Unsupported transport: '" + transport + - "' (only 'operator' is supported)"; - } - return -2; - } - - // 2. 拆分并校验 Pipeline 文档中的 deployment 部分 (RFC-0061) - PipelineDocumentSplit doc_split; - std::string split_err; - if (!SplitPipelineDocument(pipeline_json, &doc_split, &split_err)) { - if (out_error) *out_error = split_err; - return -2; - } - - if (!doc_split.has_deployment || !doc_split.deployment.has_io) { - if (out_error) { - *out_error = - "Missing required 'deployment.io' in pipeline JSON for deployment"; - } - return -2; - } - - // 3. 查找并核对绑定定义 - const std::string& binding_id = doc_split.deployment.io.io_binding; - const auto* binding = IoBindingRegistry::Instance().FindBinding(binding_id); - if (!binding) { - if (out_error) { - *out_error = "Unknown or unregistered io_binding: " + binding_id + - " (at /deployment/io/io_binding)"; - } - return -2; - } - - if (binding->transport != "operator") { - if (out_error) { - *out_error = "Binding transport mismatch for '" + binding_id + - "': expected 'operator', but binding declared '" + - binding->transport + "' (at /deployment/io/io_binding)"; - } - return -2; - } - - // 核对 Pipeline biz_name 与 binding biz_name (RFC-0061: 必须一致) - std::string pipeline_biz = pipeline_json.value("biz_name", ""); - if (pipeline_biz != binding->biz_name) { - if (out_error) { - *out_error = "Pipeline biz_name '" + pipeline_biz + - "' does not match binding biz_name '" + binding->biz_name + - "' (at /deployment/io/io_binding)"; - } - return -2; - } - - // 4. 查找输入与输出转换器 - const auto* in_conv = IoConverterRegistry::Instance().FindInputConverter( - binding->input_converter_id); - if (!in_conv) { - if (out_error) { - *out_error = "Binding references unregistered input converter: " + - binding->input_converter_id; - } - return -2; - } - - const auto* out_conv = IoConverterRegistry::Instance().FindOutputConverter( - binding->output_converter_id); - if (!out_conv) { - if (out_error) { - *out_error = "Binding references unregistered output converter: " + - binding->output_converter_id; - } - return -2; - } - - // 5. 计算有效批次上限 - size_t max_batch = - std::min(in_conv->max_batch_size, out_conv->max_batch_size); - const auto* exposure = - IoBindingRegistry::Instance().FindExposure(binding->biz_name); - if (exposure) { - max_batch = std::min(max_batch, exposure->max_batch_size); - } - - // 6. 校验 output_allocations 配置与槽位 (必须满足 R ⊆ C ⊆ A) - const auto& allocations = doc_split.deployment.io.output_allocations; - std::unordered_map output_specs; - std::unordered_map output_params; - - // 6.1 拒绝未在输出转换器中声明的未知槽位配置 (C ⊆ A) - for (auto it = allocations.begin(); it != allocations.end(); ++it) { - bool found = false; - for (const auto& slot : out_conv->external_slots) { - if (slot.direction == PortDirection::kOutput && - slot.slot_name == it.key()) { - found = true; - break; - } - } - if (!found) { - if (out_error) { - *out_error = "Unknown configured output slot: " + it.key() + - " (at /deployment/io/output_allocations/" + - EscapeJsonPointer(it.key()) + ")"; - } - return -2; - } - } + DeploymentPrepareOptions options; + options.transport = transport; + options.path_mode = model_root_dir.empty() ? DeploymentPathMode::kLexicalOnly + : DeploymentPathMode::kUnderRoot; + options.model_root_dir = model_root_dir; - // 6.2 校验并解析每个输出槽位配置,确保所有必需槽位存在 (R ⊆ C) - for (const auto& slot : out_conv->external_slots) { - if (slot.direction != PortDirection::kOutput) continue; - if (!allocations.contains(slot.slot_name)) { - if (slot.required) { - if (out_error) { - *out_error = "Missing required Operator output slot '" + - slot.slot_name + - "' (at /deployment/io/output_allocations/" + - EscapeJsonPointer(slot.slot_name) + ")"; - } - return -2; - } - continue; - } - const auto& slot_cfg = allocations[slot.slot_name]; - ResolvedOutputPoolSpec pool_spec; - std::string param_text; - std::string alloc_err; - int alloc_ret = OperatorConfigResolver::ResolveOutputAllocation( - slot_cfg, slot, &pool_spec, ¶m_text, &alloc_err); - if (alloc_ret != 0) { - if (out_error) { - *out_error = alloc_err + " (at /deployment/io/output_allocations/" + - EscapeJsonPointer(slot.slot_name) + ")"; - } - return alloc_ret; - } - output_specs[slot.slot_name] = std::move(pool_spec); - output_params[slot.slot_name] = std::move(param_text); + PreparedDeployment prepared; + DeploymentDiagnostic prep_diag; + if (!PrepareDeploymentDocument(pipeline_json, options, &prepared, + &prep_diag)) { + if (out_error) *out_error = prep_diag.message; + if (out_diagnostic) *out_diagnostic = prep_diag; + return prep_diag.legacy_status; } - // 6.3 默认深度下的句柄池载荷总预算校验 + // 默认深度下的句柄池载荷总预算校验 (RFC-0062 §2.5, §2.6) size_t total_handle_pool_bytes = 0; - for (const auto& [slot_name, pool_spec] : output_specs) { + for (const auto& [slot_name, pool_spec] : prepared.output_specs) { const auto* output_binding = OperatorValueTypeRegistry::Instance().GetOutputBinding( pool_spec.type, pool_spec.allocator); if (!output_binding || output_binding->direction != IoDirection::kOutput) { - if (out_error) { - *out_error = - "Missing output value binding for suffix '" + pool_spec.type + "'"; + std::string msg = + "Missing output value binding for suffix '" + pool_spec.type + "'"; + if (out_error) *out_error = msg; + if (out_diagnostic) { + out_diagnostic->code = "INVALID_OUTPUT_ALLOCATION"; + out_diagnostic->path = + "/deployment/io/output_allocations/" + EscapeJsonPointer(slot_name); + out_diagnostic->message = msg; + out_diagnostic->legacy_status = -2; } return -2; } @@ -236,151 +144,95 @@ int IoBindingResolver::ResolveFromPipelineJson( if (!ComputeOutputPoolPayloadBytes(*output_binding, pool_spec, kDefaultOutputPoolDepth, &slot_pool_bytes, &budget_err)) { - if (out_error) { - *out_error = "Output pool budget calculation failed: " + budget_err; + std::string msg = "Output pool budget calculation failed: " + budget_err; + if (out_error) *out_error = msg; + if (out_diagnostic) { + out_diagnostic->code = "INVALID_OUTPUT_ALLOCATION"; + out_diagnostic->path = + "/deployment/io/output_allocations/" + EscapeJsonPointer(slot_name); + out_diagnostic->message = msg; + out_diagnostic->legacy_status = -2; } return -2; } if (!CheckedAdd(total_handle_pool_bytes, slot_pool_bytes, &total_handle_pool_bytes)) { - if (out_error) *out_error = "Handle pool budget addition overflowed"; + std::string msg = "Handle pool budget addition overflowed"; + if (out_error) *out_error = msg; + if (out_diagnostic) { + out_diagnostic->code = "INVALID_OUTPUT_ALLOCATION"; + out_diagnostic->path = "/deployment/io/output_allocations"; + out_diagnostic->message = msg; + out_diagnostic->legacy_status = -2; + } return -2; } } if (total_handle_pool_bytes > kMaxHandlePoolPayloadBytes) { - if (out_error) { - *out_error = "Total output pool payload (" + - std::to_string(total_handle_pool_bytes) + - " bytes) exceeds per-handle payload budget (" + - std::to_string(kMaxHandlePoolPayloadBytes) + " bytes)"; + std::string msg = "Total output pool payload (" + + std::to_string(total_handle_pool_bytes) + + " bytes) exceeds per-handle payload budget (" + + std::to_string(kMaxHandlePoolPayloadBytes) + " bytes)"; + if (out_error) *out_error = msg; + if (out_diagnostic) { + out_diagnostic->code = "INVALID_OUTPUT_ALLOCATION"; + out_diagnostic->path = "/deployment/io/output_allocations"; + out_diagnostic->message = msg; + out_diagnostic->legacy_status = -2; } return -2; } - // 7. 应用覆盖与解析模型路径 - nlohmann::json staged_pipe_json = doc_split.neutral_pipeline_json; - std::unordered_set overridden_model_ids; - if (doc_split.deployment.has_model_paths && - !doc_split.deployment.model_paths.empty()) { - std::unordered_set known_model_ids; - if (staged_pipe_json.contains("models") && - staged_pipe_json["models"].is_array()) { - for (const auto& m : staged_pipe_json["models"]) { - if (m.is_object() && m.contains("model_id") && - m["model_id"].is_string()) { - known_model_ids.insert(m["model_id"].get()); - } - } - } - for (const auto& [mid, _] : doc_split.deployment.model_paths) { - if (!known_model_ids.count(mid)) { - if (out_error) { - *out_error = - "Unknown model_id '" + mid + - "' in '/deployment/model_paths' (at /deployment/model_paths/" + - EscapeJsonPointer(mid) + ")"; - } - return -2; - } - } - for (auto& m : staged_pipe_json["models"]) { - if (m.is_object() && m.contains("model_id") && - m["model_id"].is_string()) { - std::string mid = m["model_id"].get(); - auto it = doc_split.deployment.model_paths.find(mid); - if (it != doc_split.deployment.model_paths.end()) { - m["model_path"] = it->second; - overridden_model_ids.insert(mid); - } - } - } - } - - nlohmann::json resolved_pipeline_json; - if (!model_root_dir.empty()) { - std::string model_resolve_err; - if (!ResolveDeploymentModelPaths( - staged_pipe_json, model_root_dir, &resolved_pipeline_json, - &model_resolve_err, overridden_model_ids)) { - if (out_error) *out_error = model_resolve_err; - return -2; - } - } else { - // 环境无关检查: 模型路径必须是非空字符串 - if (staged_pipe_json.contains("models") && - staged_pipe_json["models"].is_array()) { - for (const auto& m : staged_pipe_json["models"]) { - if (m.is_object() && m.contains("model_path")) { - if (!m["model_path"].is_string() || - m["model_path"].get().empty()) { - if (out_error) { - *out_error = - "model_path in model declaration must be a non-empty string"; - } - return -2; - } - } - } - } - resolved_pipeline_json = staged_pipe_json; - } - - // 8. 构造中性 PipelineIoBoundary - PipelineIoBoundary io_boundary; - for (const auto& port : in_conv->logical_ports) { - std::string key = port.logical_name; - auto bit = binding->input_ports.find(port.logical_name); - if (bit != binding->input_ports.end()) { - key = bit->second; - } - io_boundary.input_published_ports.emplace_back( - key, port.type_id, port.required, port.cardinality, - port.provenance_policy, port.lifetime, port.lifetime_config_field); - } - - for (const auto& port : out_conv->logical_ports) { - std::string key = port.logical_name; - auto bit = binding->output_ports.find(port.logical_name); - if (bit != binding->output_ports.end()) { - key = bit->second; - } - io_boundary.output_consumed_ports.emplace_back( - key, port.type_id, port.required, port.cardinality, - port.provenance_policy, port.lifetime, port.lifetime_config_field); - } - - // 9. 调用 PipelineValidator 进行统一中性计划验证 (Core 校验 + // 调用 PipelineValidator 进行统一中性计划验证 (Core 校验 // neutral_pipeline_json) auto plan = std::make_unique( - PipelineValidator::ValidateAndPlan( - resolved_pipeline_json, ValidationPolicy::kStrict, &io_boundary)); + PipelineValidator::ValidateAndPlan(prepared.neutral_pipeline_json, + ValidationPolicy::kStrict, + &prepared.io_boundary)); + + ProjectModelPathDiagnostics(prepared, &plan->report); if (!plan->report.ok) { - if (out_error) { - if (!plan->report.diagnostics.empty()) { - const auto& d = plan->report.diagnostics.front(); - *out_error = - "Validation failed: " + std::string(DiagnosticCodeName(d.code)) + - " at " + d.path + ": " + d.message; - } else { - *out_error = "Validation failed without diagnostics"; + if (!plan->report.diagnostics.empty()) { + const auto& d = plan->report.diagnostics.front(); + std::string msg = + "Validation failed: " + std::string(DiagnosticCodeName(d.code)) + + " at " + d.path + ": " + d.message; + if (out_error) *out_error = msg; + if (out_diagnostic) { + out_diagnostic->code = DiagnosticCodeName(d.code); + out_diagnostic->path = d.path; + out_diagnostic->message = d.message; + out_diagnostic->legacy_status = -3; + out_diagnostic->pipeline_diagnostic = + PipelineDiagnostic{d.code, d.path, d.message}; + } + } else { + std::string msg = "Validation failed without diagnostics"; + if (out_error) *out_error = msg; + if (out_diagnostic) { + out_diagnostic->code = "VALIDATION_FAILED"; + out_diagnostic->path = "/"; + out_diagnostic->message = msg; + out_diagnostic->legacy_status = -3; } } return -3; } - // 10. 组装不可变接入计划 + // 组装不可变接入计划 auto io_plan = std::make_unique(); - io_plan->binding = *binding; - io_plan->input_converter = in_conv; - io_plan->output_converter = out_conv; - io_plan->input_port_bindings = InputPortBindings(binding->input_ports); - io_plan->output_port_bindings = OutputPortBindings(binding->output_ports); - io_plan->effective_max_batch_size = max_batch; - io_plan->operator_output_specs = std::move(output_specs); - io_plan->operator_output_parameter_texts = std::move(output_params); - io_plan->overridden_model_ids = std::move(overridden_model_ids); - io_plan->resolved_pipeline_json = resolved_pipeline_json; + io_plan->binding = std::move(prepared.binding); + io_plan->input_converter = prepared.input_converter; + io_plan->output_converter = prepared.output_converter; + io_plan->input_port_bindings = std::move(prepared.input_port_bindings); + io_plan->output_port_bindings = std::move(prepared.output_port_bindings); + io_plan->effective_max_batch_size = prepared.effective_max_batch_size; + io_plan->operator_output_specs = std::move(prepared.output_specs); + io_plan->operator_output_parameter_texts = + std::move(prepared.output_parameter_texts); + io_plan->overridden_model_ids = std::move(prepared.overridden_model_ids); + io_plan->resolved_pipeline_json = std::move(prepared.neutral_pipeline_json); io_plan->pipeline_plan = std::move(plan); *out_plan = std::move(io_plan); diff --git a/src/adapter/io_binding_resolver.h b/src/adapter/io_binding_resolver.h index 86ed948..8877e88 100644 --- a/src/adapter/io_binding_resolver.h +++ b/src/adapter/io_binding_resolver.h @@ -42,19 +42,22 @@ class IoBindingResolver { const std::string& transport, // "operator" const std::string& model_root_dir, std::unique_ptr* out_plan, - std::string* out_error); + std::string* out_error, + DeploymentDiagnostic* out_diagnostic = nullptr); static int ResolveFromFile(const std::string& config_path, const std::string& transport, const std::string& model_root_dir, std::unique_ptr* out_plan, - std::string* out_error); + std::string* out_error, + DeploymentDiagnostic* out_diagnostic = nullptr); static int ResolveFromPipelineJson( const nlohmann::json& pipeline_json, const std::string& transport, // "operator" const std::string& model_root_dir, - std::unique_ptr* out_plan, std::string* out_error); + std::unique_ptr* out_plan, std::string* out_error, + DeploymentDiagnostic* out_diagnostic = nullptr); }; } // namespace llm_edgeflow diff --git a/src/adapter/operator/operator_config_resolver.cpp b/src/adapter/operator/operator_config_resolver.cpp index 2ae3ffe..a1cb4c9 100644 --- a/src/adapter/operator/operator_config_resolver.cpp +++ b/src/adapter/operator/operator_config_resolver.cpp @@ -9,6 +9,7 @@ #include "adapter/deployment_io_config.h" #include "adapter/io_binding_resolver.h" #include "adapter/operator/json_output_config_reader.h" +#include "adapter/pipeline_document.h" #include "contracts/diagnostic.h" #include "contracts/path_utils.h" @@ -311,35 +312,47 @@ int OperatorConfigResolver::ResolveModelReferenceUnderRoot( } } -int OperatorConfigResolver::Resolve(const char* model_path, - const char* cfg_file_name, - ResolvedOperatorConfig* result, - std::string* error_msg, - uint32_t max_frame_depth) noexcept { +int OperatorConfigResolver::Resolve( + const char* model_path, const char* cfg_file_name, + ResolvedOperatorConfig* result, std::string* error_msg, + uint32_t max_frame_depth, DeploymentDiagnostic* out_diagnostic) noexcept { + auto set_diag = [&](const std::string& code, const std::string& path, + const std::string& message) { + if (error_msg) *error_msg = message; + if (out_diagnostic) { + out_diagnostic->code = code; + out_diagnostic->path = path; + out_diagnostic->message = message; + out_diagnostic->legacy_status = -2; + out_diagnostic->pipeline_diagnostic.reset(); + } + }; + try { + if (out_diagnostic) out_diagnostic->Clear(); + if (!result) { - if (error_msg) *error_msg = "Null output result pointer"; + set_diag("DEPLOYMENT_ERROR", "/", "Null output result pointer"); return -2; } if (!model_path || model_path[0] == '\0') { - if (error_msg) *error_msg = "Null or empty model_path"; + set_diag("DEPLOYMENT_ERROR", "/", "Null or empty model_path"); return -2; } if (!cfg_file_name || cfg_file_name[0] == '\0') { - if (error_msg) *error_msg = "Null or empty cfg_file_name"; + set_diag("DEPLOYMENT_ERROR", "/", "Null or empty cfg_file_name"); return -2; } uint32_t effective_depth = max_frame_depth > 0 ? max_frame_depth : kDefaultOutputPoolDepth; if (effective_depth > kMaxOutputPoolDepth) { - if (error_msg) { - *error_msg = "max_frame_depth (" + std::to_string(effective_depth) + - ") exceeds hard limit " + - std::to_string(kMaxOutputPoolDepth); - } + std::string msg = "max_frame_depth (" + std::to_string(effective_depth) + + ") exceeds hard limit " + + std::to_string(kMaxOutputPoolDepth); + set_diag("DEPLOYMENT_ERROR", "/", msg); return -2; } @@ -347,18 +360,17 @@ int OperatorConfigResolver::Resolve(const char* model_path, std::error_code ec; if (!std::filesystem::exists(raw_root, ec) || !std::filesystem::is_directory(raw_root, ec)) { - if (error_msg) { - *error_msg = - "model_path directory does not exist: " + raw_root.string(); - } + std::string msg = + "model_path directory does not exist: " + raw_root.string(); + set_diag("DEPLOYMENT_ERROR", "/", msg); return -2; } std::filesystem::path canon_root = std::filesystem::canonical(raw_root, ec); if (ec) { - if (error_msg) { - *error_msg = "Failed to canonicalize model_path: " + raw_root.string(); - } + std::string msg = + "Failed to canonicalize model_path: " + raw_root.string(); + set_diag("DEPLOYMENT_ERROR", "/", msg); return -2; } @@ -366,13 +378,23 @@ int OperatorConfigResolver::Resolve(const char* model_path, std::filesystem::path full_cfg; int ret = ResolveRequiredFileUnderRoot( canon_root, cfg_file_name, "cfg_file_name", &full_cfg, error_msg); - if (ret != 0) return ret; + if (ret != 0) { + if (out_diagnostic) { + out_diagnostic->code = "DEPLOYMENT_ERROR"; + out_diagnostic->path = "/"; + out_diagnostic->message = + error_msg ? *error_msg : "Failed to resolve cfg_file_name"; + out_diagnostic->legacy_status = ret; + } + return ret; + } // 读取并解析部署配置文件 (Schema 1) DeploymentIoConfig dep_config; std::string dep_err; if (!DeploymentIoConfig::ReadFromFile(full_cfg.string(), "operator", - &dep_config, &dep_err)) { + &dep_config, &dep_err, + out_diagnostic)) { if (error_msg) *error_msg = dep_err; return -2; } @@ -381,7 +403,8 @@ int OperatorConfigResolver::Resolve(const char* model_path, std::unique_ptr io_plan; std::string plan_err; int plan_ret = IoBindingResolver::ResolveFromConfig( - dep_config, "operator", canon_root.string(), &io_plan, &plan_err); + dep_config, "operator", canon_root.string(), &io_plan, &plan_err, + out_diagnostic); if (plan_ret != 0) { if (error_msg) *error_msg = plan_err; return plan_ret; @@ -397,10 +420,12 @@ int OperatorConfigResolver::Resolve(const char* model_path, pool_spec.type, pool_spec.allocator); if (!output_binding || output_binding->direction != IoDirection::kOutput) { - if (error_msg) { - *error_msg = "Missing output value binding for suffix '" + - pool_spec.type + "'"; - } + std::string msg = "Missing output value binding for suffix '" + + pool_spec.type + "'"; + set_diag("INVALID_OUTPUT_ALLOCATION", + "/deployment/io/output_allocations/" + + EscapeJsonPointer(slot_name), + msg); return -2; } size_t slot_pool_bytes = 0; @@ -408,24 +433,30 @@ int OperatorConfigResolver::Resolve(const char* model_path, if (!ComputeOutputPoolPayloadBytes(*output_binding, pool_spec, effective_depth, &slot_pool_bytes, &budget_err)) { - if (error_msg) { - *error_msg = "Output pool budget calculation failed: " + budget_err; - } + std::string msg = + "Output pool budget calculation failed: " + budget_err; + set_diag("INVALID_OUTPUT_ALLOCATION", + "/deployment/io/output_allocations/" + + EscapeJsonPointer(slot_name), + msg); return -2; } if (!CheckedAdd(total_handle_pool_bytes, slot_pool_bytes, &total_handle_pool_bytes)) { - if (error_msg) *error_msg = "Handle pool budget addition overflowed"; + std::string msg = "Handle pool budget addition overflowed"; + set_diag("INVALID_OUTPUT_ALLOCATION", + "/deployment/io/output_allocations", msg); return -2; } } if (total_handle_pool_bytes > kMaxHandlePoolPayloadBytes) { - if (error_msg) { - *error_msg = "Total output pool payload (" + - std::to_string(total_handle_pool_bytes) + - " bytes) exceeds per-handle payload budget (" + - std::to_string(kMaxHandlePoolPayloadBytes) + " bytes)"; - } + std::string msg = "Total output pool payload (" + + std::to_string(total_handle_pool_bytes) + + " bytes) exceeds per-handle payload budget (" + + std::to_string(kMaxHandlePoolPayloadBytes) + + " bytes)"; + set_diag("INVALID_OUTPUT_ALLOCATION", + "/deployment/io/output_allocations", msg); return -2; } } @@ -444,9 +475,21 @@ int OperatorConfigResolver::Resolve(const char* model_path, return 0; } catch (const std::exception& e) { SetDiagnosticNoexcept(error_msg, e.what()); + if (out_diagnostic) { + out_diagnostic->code = "INTERNAL_EXCEPTION"; + out_diagnostic->path = "/"; + out_diagnostic->message = e.what(); + out_diagnostic->legacy_status = -2; + } return -2; } catch (...) { SetDiagnosticNoexcept(error_msg, "Unknown exception"); + if (out_diagnostic) { + out_diagnostic->code = "INTERNAL_EXCEPTION"; + out_diagnostic->path = "/"; + out_diagnostic->message = "Unknown exception"; + out_diagnostic->legacy_status = -2; + } return -2; } } diff --git a/src/adapter/operator/operator_config_resolver.h b/src/adapter/operator/operator_config_resolver.h index 752d3fa..7ea23c1 100644 --- a/src/adapter/operator/operator_config_resolver.h +++ b/src/adapter/operator/operator_config_resolver.h @@ -47,7 +47,8 @@ class OperatorConfigResolver { static int Resolve(const char* model_path, const char* cfg_file_name, ResolvedOperatorConfig* result, std::string* error_msg, - uint32_t max_frame_depth = 25) noexcept; + uint32_t max_frame_depth = 25, + DeploymentDiagnostic* out_diagnostic = nullptr) noexcept; }; } // namespace llm_edgeflow diff --git a/src/tools/CMakeLists.txt b/src/tools/CMakeLists.txt index 63ddacb..eb566d5 100644 --- a/src/tools/CMakeLists.txt +++ b/src/tools/CMakeLists.txt @@ -4,6 +4,7 @@ target_link_libraries(alg_show PRIVATE add_executable(alg_pipeline_tool alg_pipeline_tool.cpp - pipeline_authoring.cpp) + pipeline_authoring.cpp + pipeline_document_validation.cpp) target_link_libraries(alg_pipeline_tool PRIVATE edgeflow_internal_headers llm_edgeflow::internal_runtime) diff --git a/src/tools/alg_pipeline_tool.cpp b/src/tools/alg_pipeline_tool.cpp index 45db293..c02a40a 100644 --- a/src/tools/alg_pipeline_tool.cpp +++ b/src/tools/alg_pipeline_tool.cpp @@ -19,6 +19,7 @@ #include "core/pipeline_validator.h" #include "edgeflow/operator/interface.h" #include "nlohmann/json.hpp" +#include "pipeline_document_validation.h" #include "tools/pipeline_authoring.h" namespace { @@ -128,243 +129,40 @@ nlohmann::json ProfilesJson(const std::string& biz_filter) { return result; } -bool ResolveDeploymentBoundary( - const nlohmann::json& root, nlohmann::json* out_neutral_json, - llm_edgeflow::PipelineIoBoundary* out_boundary, - const llm_edgeflow::IoBindingDefinition** out_binding, - nlohmann::json* out_error_json) { - using namespace llm_edgeflow; - - PipelineDocumentSplit doc_split; - std::string split_err; - std::string split_path; - if (!SplitPipelineDocument(root, &doc_split, &split_err, &split_path)) { - *out_error_json = ToolError("DEPLOYMENT_ERROR", split_err, - split_path.empty() ? "/" : split_path); - return false; - } - - if (!doc_split.has_deployment || !doc_split.deployment.has_io) { - *out_error_json = ToolError( - "MISSING_DEPLOYMENT_IO", - "Missing required 'deployment.io' in pipeline JSON", "/deployment/io"); - return false; - } - - const std::string& binding_id = doc_split.deployment.io.io_binding; - const auto* binding = IoBindingRegistry::Instance().FindBinding(binding_id); - if (!binding) { - *out_error_json = - ToolError("UNKNOWN_IO_BINDING", - "Unknown or unregistered io_binding: " + binding_id + - " (at /deployment/io/io_binding)", - "/deployment/io/io_binding"); - return false; - } - - if (binding->transport != "operator") { - *out_error_json = - ToolError("UNSUPPORTED_TRANSPORT", - "Binding transport mismatch for '" + binding_id + - "': expected 'operator', but binding declared '" + - binding->transport + "' (at /deployment/io/io_binding)", - "/deployment/io/io_binding"); - return false; - } - - // 立即核对 Pipeline biz_name 与 binding biz_name (RFC-0061) - std::string pipeline_biz = root.value("biz_name", ""); - if (pipeline_biz != binding->biz_name) { - *out_error_json = - ToolError("BIZ_MISMATCH", - "Pipeline biz_name '" + pipeline_biz + - "' does not match binding biz_name '" + - binding->biz_name + "' (at /deployment/io/io_binding)", - "/deployment/io/io_binding"); - return false; - } - - const auto* in_conv = IoConverterRegistry::Instance().FindInputConverter( - binding->input_converter_id); - if (!in_conv) { - *out_error_json = - ToolError("UNREGISTERED_CONVERTER", - "Binding references unregistered input converter: " + - binding->input_converter_id, - "/deployment/io/io_binding"); - return false; - } - - const auto* out_conv = IoConverterRegistry::Instance().FindOutputConverter( - binding->output_converter_id); - if (!out_conv) { - *out_error_json = - ToolError("UNREGISTERED_CONVERTER", - "Binding references unregistered output converter: " + - binding->output_converter_id, - "/deployment/io/io_binding"); - return false; - } - - const auto& allocations = doc_split.deployment.io.output_allocations; - for (auto it = allocations.begin(); it != allocations.end(); ++it) { - bool found = false; - for (const auto& slot : out_conv->external_slots) { - if (slot.direction == PortDirection::kOutput && - slot.slot_name == it.key()) { - found = true; - break; - } - } - if (!found) { - *out_error_json = ToolError( - "UNKNOWN_OUTPUT_SLOT", - "Unknown configured output slot: " + it.key() + - " (at /deployment/io/output_allocations/" + - EscapeJsonPointer(it.key()) + ")", - "/deployment/io/output_allocations/" + EscapeJsonPointer(it.key())); - return false; - } - } - - for (const auto& slot : out_conv->external_slots) { - if (slot.direction != PortDirection::kOutput) continue; - if (!allocations.contains(slot.slot_name)) { - if (slot.required) { - *out_error_json = ToolError( - "MISSING_OUTPUT_SLOT", - "Missing required Operator output slot '" + slot.slot_name + - "' (at /deployment/io/output_allocations/" + - EscapeJsonPointer(slot.slot_name) + ")", - "/deployment/io/output_allocations/" + - EscapeJsonPointer(slot.slot_name)); - return false; - } - continue; - } - ResolvedOutputPoolSpec pool_spec; - std::string param_text; - std::string alloc_err; - if (OperatorConfigResolver::ResolveOutputAllocation( - allocations[slot.slot_name], slot, &pool_spec, ¶m_text, - &alloc_err) != 0) { - *out_error_json = - ToolError("INVALID_OUTPUT_ALLOCATION", - alloc_err + " (at /deployment/io/output_allocations/" + - EscapeJsonPointer(slot.slot_name) + ")", - "/deployment/io/output_allocations/" + - EscapeJsonPointer(slot.slot_name)); - return false; - } - } - - nlohmann::json staged_pipe_json = doc_split.neutral_pipeline_json; - if (doc_split.deployment.has_model_paths && - !doc_split.deployment.model_paths.empty()) { - std::unordered_set known_model_ids; - if (staged_pipe_json.contains("models") && - staged_pipe_json["models"].is_array()) { - for (const auto& m : staged_pipe_json["models"]) { - if (m.is_object() && m.contains("model_id") && - m["model_id"].is_string()) { - known_model_ids.insert(m["model_id"].get()); - } - } - } - for (const auto& [mid, _] : doc_split.deployment.model_paths) { - if (!known_model_ids.count(mid)) { - *out_error_json = ToolError( - "UNKNOWN_MODEL_ID", - "Unknown model_id '" + mid + "' in '/deployment/model_paths'", - "/deployment/model_paths/" + EscapeJsonPointer(mid)); - return false; - } - } - for (auto& m : staged_pipe_json["models"]) { - if (m.is_object() && m.contains("model_id") && - m["model_id"].is_string()) { - std::string mid = m["model_id"].get(); - auto it = doc_split.deployment.model_paths.find(mid); - if (it != doc_split.deployment.model_paths.end()) { - m["model_path"] = it->second; - } - } - } - } - - if (staged_pipe_json.contains("models") && - staged_pipe_json["models"].is_array()) { - for (size_t index = 0; index < staged_pipe_json["models"].size(); ++index) { - const auto& m = staged_pipe_json["models"][index]; - if (m.is_object() && m.contains("model_path")) { - if (!m["model_path"].is_string() || - m["model_path"].get().empty()) { - std::string mid = m.value("model_id", ""); - std::string pointer = - (doc_split.deployment.has_model_paths && - doc_split.deployment.model_paths.count(mid)) - ? "/deployment/model_paths/" + EscapeJsonPointer(mid) - : "/models/" + std::to_string(index) + "/model_path"; - *out_error_json = ToolError( - "INVALID_MODEL_PATH", - "model_path in model declaration must be a non-empty string", - pointer); - return false; - } - } - } - } - - PipelineIoBoundary io_boundary; - for (const auto& port : in_conv->logical_ports) { - std::string key = port.logical_name; - auto bit = binding->input_ports.find(port.logical_name); - if (bit != binding->input_ports.end()) { - key = bit->second; - } - io_boundary.input_published_ports.emplace_back( - key, port.type_id, port.required, port.cardinality, - port.provenance_policy, port.lifetime, port.lifetime_config_field); - } - - for (const auto& port : out_conv->logical_ports) { - std::string key = port.logical_name; - auto bit = binding->output_ports.find(port.logical_name); - if (bit != binding->output_ports.end()) { - key = bit->second; - } - io_boundary.output_consumed_ports.emplace_back( - key, port.type_id, port.required, port.cardinality, - port.provenance_policy, port.lifetime, port.lifetime_config_field); - } - - *out_boundary = std::move(io_boundary); - *out_neutral_json = std::move(staged_pipe_json); - if (out_binding) *out_binding = binding; - return true; -} - nlohmann::json ResolveConf(const std::string& file, const std::string& root, uint32_t depth) { using namespace llm_edgeflow; const auto ops = operator_api::Get_LLM_EDGEFLOW_OperatorTable(); - if (ops.Init() != 0) + if (ops.Init != nullptr && ops.Init() != 0) return PipelineError(DiagnosticCode::kRegistryConflict, operator_api::GetOperatorLastError()); struct RegistryGuard { operator_api::OperatorFunc ops; - ~RegistryGuard() { ops.Deinit(); } + ~RegistryGuard() { + if (ops.Deinit != nullptr) ops.Deinit(); + } } registry_guard{ops}; ResolvedOperatorConfig resolved; std::string error; + DeploymentDiagnostic diag; if (OperatorConfigResolver::Resolve(root.c_str(), file.c_str(), &resolved, - &error, depth) != 0) - return ToolError("DEPLOYMENT_CONFIG", error); + &error, depth, &diag) != 0) { + const std::string message = + error.empty() ? (diag.message.empty() ? "Deployment configuration error" + : diag.message) + : error; + return ToolError("DEPLOYMENT_CONFIG", message); + } if (!resolved.io_plan || !resolved.io_plan->pipeline_plan) return ToolError("DEPLOYMENT_CONFIG", "Missing pipeline plan in io_plan"); const auto& plan = *resolved.io_plan->pipeline_plan; - if (!plan.report.ok) return plan.report.ToJson(); + if (!plan.report.ok) { + const std::string message = plan.report.diagnostics.empty() + ? "Pipeline validation failed" + : plan.report.diagnostics.front().message; + return ToolError("DEPLOYMENT_CONFIG", message); + } auto effective = resolved.synthetic_pipeline_json; for (const auto& [id, node] : plan.node_plans) @@ -468,6 +266,8 @@ int main(int argc, char* argv[]) { result = ResolveConf(argv[2], root, depth); } catch (const std::exception& error) { result = ToolError("DEPLOYMENT_CONFIG", error.what()); + } catch (...) { + result = ToolError("DEPLOYMENT_CONFIG", "Unknown internal exception"); } std::cout << result.dump(2) << std::endl; return result.value("ok", false) ? 0 : 1; @@ -604,7 +404,14 @@ int main(int argc, char* argv[]) { } const auto ops = llm_edgeflow::operator_api::Get_LLM_EDGEFLOW_OperatorTable(); - if (ops.Init != nullptr) ops.Init(); + if (ops.Init != nullptr && ops.Init() != 0) { + std::cout << PipelineError( + DiagnosticCode::kRegistryConflict, + llm_edgeflow::operator_api::GetOperatorLastError()) + .dump(2) + << std::endl; + return 1; + } struct OpsGuard { llm_edgeflow::operator_api::OperatorFunc ops; ~OpsGuard() { @@ -612,73 +419,25 @@ int main(int argc, char* argv[]) { } } ops_guard{ops}; - llm_edgeflow::PipelineIoBoundary io_boundary; - const llm_edgeflow::PipelineIoBoundary* io_boundary_ptr = nullptr; - const llm_edgeflow::IoBindingDefinition* binding_def = nullptr; - nlohmann::json target_json = root; - - if (root.contains("deployment")) { - nlohmann::json err_res; - if (!ResolveDeploymentBoundary(root, &target_json, &io_boundary, - &binding_def, &err_res)) { - if (command == "plan") { - err_res["plan"] = {{"layers", nlohmann::json::array()}, - {"topological_order", nlohmann::json::array()}}; - } - std::cout << err_res.dump(2) << std::endl; - return 1; - } - io_boundary_ptr = &io_boundary; - } - - if (command == "validate") { - auto report = - explain ? PipelineValidator::Explain( - target_json, llm_edgeflow::ValidationPolicy::kStrict, - io_boundary_ptr) - : PipelineValidator::Validate( - target_json, llm_edgeflow::ValidationPolicy::kStrict, - io_boundary_ptr); - if (report.ok && binding_def != nullptr) { - std::string biz = target_json.value("biz_name", ""); - if (biz != binding_def->biz_name) { - std::cout << ToolError("BIZ_MISMATCH", - "Pipeline biz_name '" + biz + - "' does not match binding biz_name '" + - binding_def->biz_name + - "' (at /deployment/io/io_binding)") - .dump(2) - << std::endl; - return 1; - } - } - auto result = report.ToJson(); - std::cout << result.dump(2) << std::endl; - return report.ok ? 0 : 1; - } else { - auto planned = PipelineValidator::ValidateAndPlan( - target_json, llm_edgeflow::ValidationPolicy::kStrict, - io_boundary_ptr); - if (planned.report.ok && binding_def != nullptr) { - std::string biz = target_json.value("biz_name", ""); - if (biz != binding_def->biz_name) { - nlohmann::json err_res = ToolError( - "BIZ_MISMATCH", "Pipeline biz_name '" + biz + - "' does not match binding biz_name '" + - binding_def->biz_name + - "' (at /deployment/io/io_binding)"); - err_res["plan"] = {{"layers", nlohmann::json::array()}, - {"topological_order", nlohmann::json::array()}}; - std::cout << err_res.dump(2) << std::endl; - return 1; - } - } - auto result = planned.report.ToJson(); - if (planned.report.ok) { - result.erase("diagnostics"); - } - std::cout << result.dump(2) << std::endl; - return planned.report.ok ? 0 : 1; + using llm_edgeflow::DocumentValidationMode; + DocumentValidationMode mode = + (command == "validate") ? (explain ? DocumentValidationMode::kExplain + : DocumentValidationMode::kValidate) + : DocumentValidationMode::kPlan; + + try { + auto result = llm_edgeflow::ValidatePipelineDocument(root, mode); + std::cout << result.response.dump(2) << std::endl; + return result.ok ? 0 : 1; + } catch (const std::exception& error) { + std::cout << ToolError("INTERNAL_EXCEPTION", error.what()).dump(2) + << std::endl; + return 1; + } catch (...) { + std::cout << ToolError("INTERNAL_EXCEPTION", "Unknown internal exception") + .dump(2) + << std::endl; + return 1; } } @@ -702,60 +461,76 @@ int main(int argc, char* argv[]) { } } - std::unique_ptr plan; - std::string error; - int rc = llm_edgeflow::IoBindingResolver::ResolveFromFile( - config_path, transport, model_root, &plan, &error); - - if (rc != 0 || !plan) { - std::string diag_path = "/"; - auto at_pos = error.rfind("(at "); - if (at_pos != std::string::npos) { - auto end_pos = error.find(')', at_pos); - if (end_pos != std::string::npos) { - diag_path = error.substr(at_pos + 4, end_pos - (at_pos + 4)); - } - } else { - auto at_pos2 = error.rfind("at /"); - if (at_pos2 != std::string::npos) { - auto end_pos2 = error.find(':', at_pos2); - if (end_pos2 != std::string::npos) { - diag_path = error.substr(at_pos2 + 3, end_pos2 - (at_pos2 + 3)); - } - } - } - nlohmann::json err_res = { - {"schema_version", 1}, - {"ok", false}, - {"diagnostics", - nlohmann::json::array({{{"code", "IO_VALIDATION_ERROR"}, - {"path", diag_path}, - {"message", error}, - {"severity", "error"}}})}}; - std::cout << err_res.dump(2) << std::endl; + const auto ops = + llm_edgeflow::operator_api::Get_LLM_EDGEFLOW_OperatorTable(); + if (ops.Init != nullptr && ops.Init() != 0) { + std::cout << PipelineError( + DiagnosticCode::kRegistryConflict, + llm_edgeflow::operator_api::GetOperatorLastError()) + .dump(2) + << std::endl; return 1; } + struct OpsGuard { + llm_edgeflow::operator_api::OperatorFunc ops; + ~OpsGuard() { + if (ops.Deinit != nullptr) ops.Deinit(); + } + } ops_guard{ops}; - nlohmann::json binding_info = { - {"binding_id", plan->binding.binding_id}, - {"biz_name", plan->binding.biz_name}, - {"transport", plan->binding.transport}, - {"input_converter_id", plan->binding.input_converter_id}, - {"output_converter_id", plan->binding.output_converter_id}, - {"input_port_mapping", plan->binding.input_ports}, - {"output_port_mapping", plan->binding.output_ports}, - {"effective_max_batch_size", plan->effective_max_batch_size}, - {"external_input_type", - plan->input_converter ? plan->input_converter->external_type : ""}, - {"external_output_type", - plan->output_converter ? plan->output_converter->external_type : ""}}; - - nlohmann::json result = {{"schema_version", 1}, - {"ok", true}, - {"binding", std::move(binding_info)}, - {"diagnostics", nlohmann::json::array()}}; - std::cout << result.dump(2) << std::endl; - return 0; + try { + std::unique_ptr plan; + std::string error; + llm_edgeflow::DeploymentDiagnostic diag; + int rc = llm_edgeflow::IoBindingResolver::ResolveFromFile( + config_path, transport, model_root, &plan, &error, &diag); + + if (rc != 0 || !plan) { + std::string diag_path = diag.path.empty() ? "/" : diag.path; + std::string diag_msg = error.empty() ? diag.message : error; + nlohmann::json err_res = { + {"schema_version", 1}, + {"ok", false}, + {"diagnostics", + nlohmann::json::array({{{"code", "IO_VALIDATION_ERROR"}, + {"path", diag_path}, + {"message", diag_msg}, + {"severity", "error"}}})}}; + std::cout << err_res.dump(2) << std::endl; + return 1; + } + + nlohmann::json binding_info = { + {"binding_id", plan->binding.binding_id}, + {"biz_name", plan->binding.biz_name}, + {"transport", plan->binding.transport}, + {"input_converter_id", plan->binding.input_converter_id}, + {"output_converter_id", plan->binding.output_converter_id}, + {"input_port_mapping", plan->binding.input_ports}, + {"output_port_mapping", plan->binding.output_ports}, + {"effective_max_batch_size", plan->effective_max_batch_size}, + {"external_input_type", + plan->input_converter ? plan->input_converter->external_type : ""}, + {"external_output_type", plan->output_converter + ? plan->output_converter->external_type + : ""}}; + + nlohmann::json result = {{"schema_version", 1}, + {"ok", true}, + {"binding", std::move(binding_info)}, + {"diagnostics", nlohmann::json::array()}}; + std::cout << result.dump(2) << std::endl; + return 0; + } catch (const std::exception& error) { + std::cout << ToolError("INTERNAL_EXCEPTION", error.what()).dump(2) + << std::endl; + return 1; + } catch (...) { + std::cout << ToolError("INTERNAL_EXCEPTION", "Unknown internal exception") + .dump(2) + << std::endl; + return 1; + } } if (command == "edit") { @@ -763,6 +538,23 @@ int main(int argc, char* argv[]) { Usage(); return 2; } + const auto ops = + llm_edgeflow::operator_api::Get_LLM_EDGEFLOW_OperatorTable(); + if (ops.Init != nullptr && ops.Init() != 0) { + std::cout << PipelineError( + DiagnosticCode::kRegistryConflict, + llm_edgeflow::operator_api::GetOperatorLastError()) + .dump(2) + << std::endl; + return 1; + } + struct OpsGuard { + llm_edgeflow::operator_api::OperatorFunc ops; + ~OpsGuard() { + if (ops.Deinit != nullptr) ops.Deinit(); + } + } ops_guard{ops}; + std::string input_str; char buffer[65536]; while (std::cin.read(buffer, sizeof(buffer)) || std::cin.gcount() > 0) { @@ -817,6 +609,23 @@ int main(int argc, char* argv[]) { return 2; } } + const auto ops = + llm_edgeflow::operator_api::Get_LLM_EDGEFLOW_OperatorTable(); + if (ops.Init != nullptr && ops.Init() != 0) { + std::cout << PipelineError( + DiagnosticCode::kRegistryConflict, + llm_edgeflow::operator_api::GetOperatorLastError()) + .dump(2) + << std::endl; + return 1; + } + struct OpsGuard { + llm_edgeflow::operator_api::OperatorFunc ops; + ~OpsGuard() { + if (ops.Deinit != nullptr) ops.Deinit(); + } + } ops_guard{ops}; + try { auto result = llm_edgeflow::PipelineAuthoring::FixDeps(file, in_place); std::cout << result.ToJson().dump(2) << std::endl; diff --git a/src/tools/pipeline_authoring.cpp b/src/tools/pipeline_authoring.cpp index f80a9d3..a956274 100644 --- a/src/tools/pipeline_authoring.cpp +++ b/src/tools/pipeline_authoring.cpp @@ -19,6 +19,7 @@ #include "core/pipeline_catalog.h" #include "core/pipeline_validator.h" #include "edgeflow/operator/interface.h" +#include "pipeline_document_validation.h" namespace llm_edgeflow { @@ -1444,72 +1445,6 @@ bool PipelineAuthoring::ApplyOperation(nlohmann::json* pipeline, return false; } -ValidationReport ValidateOrExplainAuthoring(const nlohmann::json& doc, - bool explain) { - if (!doc.is_object() || !doc.contains("deployment")) { - return explain ? PipelineValidator::Explain(doc) - : PipelineValidator::Validate(doc); - } - const auto ops = operator_api::Get_LLM_EDGEFLOW_OperatorTable(); - if (ops.Init != nullptr) ops.Init(); - - PipelineDocumentSplit split; - std::string err; - if (!SplitPipelineDocument(doc, &split, &err)) { - ValidationReport r; - r.ok = false; - ValidationDiagnostic diag; - diag.code = DiagnosticCode::kUnknownField; - diag.path = "/deployment"; - diag.message = err; - diag.severity = "error"; - r.diagnostics.push_back(std::move(diag)); - return r; - } - - PipelineIoBoundary io_boundary; - const PipelineIoBoundary* io_boundary_ptr = nullptr; - if (split.deployment.has_io && !split.deployment.io.io_binding.empty()) { - const auto* binding = IoBindingRegistry::Instance().FindBinding( - split.deployment.io.io_binding); - if (binding) { - const auto* in_conv = IoConverterRegistry::Instance().FindInputConverter( - binding->input_converter_id); - const auto* out_conv = - IoConverterRegistry::Instance().FindOutputConverter( - binding->output_converter_id); - if (in_conv && out_conv) { - for (const auto& port : in_conv->logical_ports) { - std::string key = port.logical_name; - auto bit = binding->input_ports.find(port.logical_name); - if (bit != binding->input_ports.end()) key = bit->second; - io_boundary.input_published_ports.emplace_back( - key, port.type_id, port.required, port.cardinality, - port.provenance_policy, port.lifetime, - port.lifetime_config_field); - } - for (const auto& port : out_conv->logical_ports) { - std::string key = port.logical_name; - auto bit = binding->output_ports.find(port.logical_name); - if (bit != binding->output_ports.end()) key = bit->second; - io_boundary.output_consumed_ports.emplace_back( - key, port.type_id, port.required, port.cardinality, - port.provenance_policy, port.lifetime, - port.lifetime_config_field); - } - io_boundary_ptr = &io_boundary; - } - } - } - - return explain ? PipelineValidator::Explain(split.neutral_pipeline_json, - ValidationPolicy::kStrict, - io_boundary_ptr) - : PipelineValidator::Validate(split.neutral_pipeline_json, - ValidationPolicy::kStrict, - io_boundary_ptr); -} - AuthoringResult PipelineAuthoring::ApplyRequest(const nlohmann::json& request) { AuthoringResult result; try { @@ -1598,11 +1533,12 @@ AuthoringResult PipelineAuthoring::ApplyRequest(const nlohmann::json& request) { } result.failed_operation_index.reset(); - auto report = ValidateOrExplainAuthoring(working_pipeline, true); - result.validation = report.ToJson(); + auto val_res = ValidatePipelineDocument(working_pipeline, + DocumentValidationMode::kExplain); + result.validation = std::move(val_res.response); bool require_valid = request.value("require_valid", false); - if (require_valid && !report.ok) { + if (require_valid && !val_res.ok) { result.ok = false; result.pipeline = std::nullopt; return result; @@ -1653,14 +1589,30 @@ FixDepsResult PipelineAuthoring::FixDeps(const std::string& file_path, return result; } - auto report = ValidateOrExplainAuthoring(root, true); - if (report.ok) { + auto initial_res = + ValidatePipelineDocument(root, DocumentValidationMode::kExplain); + if (initial_res.ok) { result.ok = true; result.written = false; - result.validation = report.ToJson(); + result.validation = std::move(initial_res.response); return result; } + if (!initial_res.core_report.has_value()) { + result.ok = false; + result.written = false; + result.validation = std::move(initial_res.response); + for (const auto& diag : + result.validation.value("diagnostics", nlohmann::json::array())) { + result.diagnostics.push_back(diag.value("code", "") + " " + + diag.value("path", "") + ": " + + diag.value("message", "")); + } + return result; + } + + const auto& report = *initial_res.core_report; + nlohmann::json working = root; auto dep_graph = BuildDependencyGraph(working); bool found_fix = false; @@ -1705,7 +1657,7 @@ FixDepsResult PipelineAuthoring::FixDeps(const std::string& file_path, if (it_p != key_producers.end() && it_p->second.size() > 1) { result.ok = false; result.written = false; - result.validation = report.ToJson(); + result.validation = initial_res.response; result.diagnostics.push_back( "AMBIGUOUS_PRODUCER: 数据键 '" + bound_key + "' 存在多个生产者,来源存在歧义,不能自动修复"); @@ -1719,7 +1671,7 @@ FixDepsResult PipelineAuthoring::FixDeps(const std::string& file_path, if (IsAncestor(consumer_id, producer_id, dep_graph)) { result.ok = false; result.written = false; - result.validation = report.ToJson(); + result.validation = initial_res.response; result.diagnostics.push_back("CYCLE_DETECTED: 添加 " + consumer_id + " 对 " + producer_id + " 的依赖会形成环"); @@ -1748,20 +1700,23 @@ FixDepsResult PipelineAuthoring::FixDeps(const std::string& file_path, if (!found_fix) { result.ok = false; result.written = false; - result.validation = report.ToJson(); + result.validation = std::move(initial_res.response); result.diagnostics.push_back( "NO_FIXABLE_DEPENDENCY: 未发现可安全自动修复的生产者依赖"); return result; } - auto final_report = ValidateOrExplainAuthoring(working, true); - result.validation = final_report.ToJson(); - if (!final_report.ok) { + auto final_res = + ValidatePipelineDocument(working, DocumentValidationMode::kExplain); + result.validation = std::move(final_res.response); + if (!final_res.ok) { result.ok = false; result.written = false; - for (const auto& d : final_report.diagnostics) { - result.diagnostics.push_back(std::string(DiagnosticCodeName(d.code)) + - " " + d.path + ": " + d.message); + for (const auto& diag : + result.validation.value("diagnostics", nlohmann::json::array())) { + result.diagnostics.push_back(diag.value("code", "") + " " + + diag.value("path", "") + ": " + + diag.value("message", "")); } return result; } diff --git a/src/tools/pipeline_document_validation.cpp b/src/tools/pipeline_document_validation.cpp new file mode 100644 index 0000000..333bc7c --- /dev/null +++ b/src/tools/pipeline_document_validation.cpp @@ -0,0 +1,101 @@ +#include "pipeline_document_validation.h" + +#include + +#include "adapter/deployment_preparation.h" +#include "core/pipeline_validator.h" + +namespace llm_edgeflow { + +DocumentValidationResult ValidatePipelineDocument( + const nlohmann::json& document, DocumentValidationMode mode) { + DocumentValidationResult result; + + if (document.is_object() && document.contains("deployment")) { + DeploymentPrepareOptions options; + options.transport = "operator"; + options.path_mode = DeploymentPathMode::kLexicalOnly; + options.model_root_dir = ""; + + PreparedDeployment prepared; + DeploymentDiagnostic diag; + if (!PrepareDeploymentDocument(document, options, &prepared, &diag)) { + result.ok = false; + result.core_report = std::nullopt; + + nlohmann::json diag_item = {{"code", diag.code}, + {"path", diag.path}, + {"message", diag.message}, + {"severity", "error"}}; + nlohmann::json resp = { + {"schema_version", 1}, + {"ok", false}, + {"diagnostics", nlohmann::json::array({diag_item})}}; + if (mode == DocumentValidationMode::kPlan) { + resp["plan"] = {{"layers", nlohmann::json::array()}, + {"topological_order", nlohmann::json::array()}}; + } + result.response = std::move(resp); + return result; + } + + if (mode == DocumentValidationMode::kValidate) { + auto report = PipelineValidator::Validate(prepared.neutral_pipeline_json, + ValidationPolicy::kStrict, + &prepared.io_boundary); + ProjectModelPathDiagnostics(prepared, &report); + result.ok = report.ok; + result.response = report.ToJson(); + result.core_report = std::move(report); + } else if (mode == DocumentValidationMode::kExplain) { + auto report = PipelineValidator::Explain(prepared.neutral_pipeline_json, + ValidationPolicy::kStrict, + &prepared.io_boundary); + ProjectModelPathDiagnostics(prepared, &report); + result.ok = report.ok; + result.response = report.ToJson(); + result.core_report = std::move(report); + } else { + auto planned = PipelineValidator::ValidateAndPlan( + prepared.neutral_pipeline_json, ValidationPolicy::kStrict, + &prepared.io_boundary); + ProjectModelPathDiagnostics(prepared, &planned.report); + result.ok = planned.report.ok; + auto resp = planned.report.ToJson(); + if (planned.report.ok) { + resp.erase("diagnostics"); + } + result.response = std::move(resp); + result.core_report = std::move(planned.report); + } + return result; + } + + // 不含 deployment,或根不是对象:原样交给 Core + if (mode == DocumentValidationMode::kValidate) { + auto report = + PipelineValidator::Validate(document, ValidationPolicy::kStrict); + result.ok = report.ok; + result.response = report.ToJson(); + result.core_report = std::move(report); + } else if (mode == DocumentValidationMode::kExplain) { + auto report = + PipelineValidator::Explain(document, ValidationPolicy::kStrict); + result.ok = report.ok; + result.response = report.ToJson(); + result.core_report = std::move(report); + } else { + auto planned = + PipelineValidator::ValidateAndPlan(document, ValidationPolicy::kStrict); + result.ok = planned.report.ok; + auto resp = planned.report.ToJson(); + if (planned.report.ok) { + resp.erase("diagnostics"); + } + result.response = std::move(resp); + result.core_report = std::move(planned.report); + } + return result; +} + +} // namespace llm_edgeflow diff --git a/src/tools/pipeline_document_validation.h b/src/tools/pipeline_document_validation.h new file mode 100644 index 0000000..2b96923 --- /dev/null +++ b/src/tools/pipeline_document_validation.h @@ -0,0 +1,27 @@ +#pragma once + +#include +#include + +#include "core/pipeline_validator.h" + +namespace llm_edgeflow { + +enum class DocumentValidationMode { kValidate, kExplain, kPlan }; + +struct DocumentValidationResult { + bool ok = false; + nlohmann::json response; + std::optional core_report; +}; + +/** + * @brief 校验 Pipeline 文档(CLI 与 Authoring 共用适配,RFC-0062) + * + * 统一处理带/不带 deployment 的文档,并根据模式调用 + * Validate/Explain/ValidateAndPlan, 投影有效模型路径来源,保留完整 Core 报告。 + */ +DocumentValidationResult ValidatePipelineDocument( + const nlohmann::json& document, DocumentValidationMode mode); + +} // namespace llm_edgeflow diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 152fee1..d2ebaed 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -27,6 +27,7 @@ endif() add_executable(alg_pipeline_tool_test ${PROJECT_SOURCE_DIR}/src/tools/alg_pipeline_tool.cpp ${PROJECT_SOURCE_DIR}/src/tools/pipeline_authoring.cpp + ${PROJECT_SOURCE_DIR}/src/tools/pipeline_document_validation.cpp $ $) target_link_libraries(alg_pipeline_tool_test PRIVATE llm_edgeflow::internal_runtime) diff --git a/tests/tooling/test_pipeline_studio.py b/tests/tooling/test_pipeline_studio.py index 4574dac..9ce0436 100644 --- a/tests/tooling/test_pipeline_studio.py +++ b/tests/tooling/test_pipeline_studio.py @@ -522,6 +522,12 @@ def test_native_deployment_rejection_rolls_back_the_pair(self): class PipelineCliTest(unittest.TestCase): + CLI_PARITY_ENTRYPOINTS = [ + ("validate",), + ("plan",), + ("validate", "--explain"), + ] + def command(self, *args, input_pipeline=None): process = subprocess.run( [str(PIPELINE_TOOL), *args], @@ -660,8 +666,422 @@ def test_resolve_conf_exposes_model_sources_defaults_and_native_pool_errors(self self.assertEqual(code, 1) self.assertFalse(rejected["ok"]) self.assertEqual(rejected["diagnostics"][0]["code"], "DEPLOYMENT_CONFIG") + self.assertEqual(rejected["diagnostics"][0]["path"], "/") self.assertIn("entities_json", rejected["diagnostics"][0]["message"]) + pipe_doc["deployment"]["io"]["output_allocations"].clear() + (Path(directory) / conf["pipe_path"]).write_text(json.dumps(pipe_doc)) + code, rejected_missing = self.command("resolve-conf", str(changed.relative_to(ROOT)), "--root", str(ROOT)) + self.assertEqual(code, 1) + self.assertFalse(rejected_missing["ok"]) + self.assertEqual(rejected_missing["diagnostics"][0]["code"], "DEPLOYMENT_CONFIG") + self.assertEqual(rejected_missing["diagnostics"][0]["path"], "/") + self.assertIn("entity_out", rejected_missing["diagnostics"][0]["message"]) + + bad_conf = Path(directory) / "bad.conf" + bad_conf.write_text("{}") + code, bad_res = self.command("resolve-conf", str(bad_conf.relative_to(ROOT)), "--root", str(ROOT)) + self.assertEqual(code, 1) + self.assertFalse(bad_res["ok"]) + self.assertEqual(bad_res["diagnostics"][0]["code"], "DEPLOYMENT_CONFIG") + self.assertEqual(bad_res["diagnostics"][0]["path"], "/") + self.assertIn("pipe_path", bad_res["diagnostics"][0]["message"]) + + code, bad_res_root = self.command("resolve-conf", "bad.conf", "--root", str(directory)) + self.assertEqual(code, 1) + self.assertFalse(bad_res_root["ok"]) + self.assertEqual(bad_res_root["diagnostics"][0]["code"], "DEPLOYMENT_CONFIG") + self.assertEqual(bad_res_root["diagnostics"][0]["path"], "/") + self.assertIn("pipe_path", bad_res_root["diagnostics"][0]["message"]) + + def test_rfc0062_cli_raw_model_path_required_even_with_override_t03(self): + # T03 via CLI: Deployment model path override does not forgive missing/invalid original model_path. + # Covers missing, null, empty string, and number, paired with without-override cases. + pipeline = json.loads( + (ROOT / "demo/fixtures/mock/pipeline_entity_extract.json").read_text() + ) + self.assertIn("model_paths", pipeline.get("deployment", {})) + # 1. Missing model_path in original models[0] + missing_doc = copy.deepcopy(pipeline) + missing_doc["models"][0].pop("model_path") + + # 2. null model_path + null_doc = copy.deepcopy(pipeline) + null_doc["models"][0]["model_path"] = None + + # 3. empty string model_path + empty_doc = copy.deepcopy(pipeline) + empty_doc["models"][0]["model_path"] = "" + + # 4. number (integer) model_path + number_doc = copy.deepcopy(pipeline) + number_doc["models"][0]["model_path"] = 12345 + + cases = [ + ("missing_with_override", missing_doc, "MISSING_FIELD"), + ("null_with_override", null_doc, "INVALID_MODEL_PATH"), + ("empty_with_override", empty_doc, "INVALID_MODEL_PATH"), + ("number_with_override", number_doc, "INVALID_MODEL_PATH"), + ] + + # Add paired without-override cases to prove the identical structural requirements + for case_name, doc, exp_code in list(cases): + no_override_doc = copy.deepcopy(doc) + no_override_doc["deployment"].pop("model_paths", None) + paired_name = case_name.replace("_with_override", "_without_override") + cases.append((paired_name, no_override_doc, exp_code)) + + for case_name, doc, expected_code in cases: + for ep in self.CLI_PARITY_ENTRYPOINTS: + with self.subTest(case=case_name, entrypoint=ep): + code, res = self.command(*ep, "--stdin", input_pipeline=doc) + self.assertEqual(code, 1) + self.assertFalse(res["ok"]) + self.assertEqual(res["diagnostics"][0]["code"], expected_code) + self.assertEqual(res["diagnostics"][0]["path"], "/models/0/model_path") + if ep[0] == "plan": + self.assertEqual(res["plan"], {"layers": [], "topological_order": []}) + + def test_rfc0062_cli_raw_model_path_required_even_with_override(self): + # Alias for backward compatibility + self.test_rfc0062_cli_raw_model_path_required_even_with_override_t03() + + def test_rfc0062_cli_plan_envelopes_t16(self): + # T16 via CLI: plan returns envelope with diagnostics on deployment preparation failure, + # partial plan on Core failure, and full topological order on success, with CLI parity. + pipeline = json.loads( + (ROOT / "demo/fixtures/mock/pipeline_entity_extract.json").read_text() + ) + # 1. Invalid deployment override model ID + invalid_doc = copy.deepcopy(pipeline) + invalid_doc["deployment"]["model_paths"]["unknown_model_id"] = "models/foo.bin" + for ep in self.CLI_PARITY_ENTRYPOINTS: + with self.subTest(case="deployment_failure", entrypoint=ep): + code, res = self.command(*ep, "--stdin", input_pipeline=invalid_doc) + self.assertEqual(code, 1) + self.assertFalse(res["ok"]) + self.assertIn("diagnostics", res) + self.assertEqual(res["diagnostics"][0]["code"], "UNKNOWN_MODEL_ID") + self.assertEqual(res["diagnostics"][0]["path"], "/deployment/model_paths/unknown_model_id") + if ep[0] == "plan": + self.assertEqual(res["plan"], {"layers": [], "topological_order": []}) + + # 2. Core failure with valid deployment + invalid_core = copy.deepcopy(pipeline) + invalid_core["pipeline"].append({ + "id": "bad_node_t16", + "node_type": "CompletelyUnknownNodeType", + "depends_on": [], + }) + for ep in self.CLI_PARITY_ENTRYPOINTS: + with self.subTest(case="core_failure", entrypoint=ep): + code, res = self.command(*ep, "--stdin", input_pipeline=invalid_core) + self.assertEqual(code, 1) + self.assertFalse(res["ok"]) + self.assertIn("diagnostics", res) + self.assertEqual(res["diagnostics"][0]["code"], "UNKNOWN_NODE_TYPE") + if ep[0] == "plan": + self.assertIn("plan", res) + + # 3. Valid pipeline plan + for ep in self.CLI_PARITY_ENTRYPOINTS: + with self.subTest(case="valid_plan", entrypoint=ep): + code, res = self.command(*ep, "--stdin", input_pipeline=pipeline) + self.assertEqual(code, 0) + self.assertTrue(res["ok"]) + if ep[0] == "plan": + self.assertIn("plan", res) + self.assertNotIn("diagnostics", res) + self.assertTrue(res["plan"]["topological_order"]) + + def test_rfc0062_cli_plan_envelopes(self): + # Alias for backward compatibility + self.test_rfc0062_cli_plan_envelopes_t16() + + def test_rfc0062_cli_validate_io_exact_pointer_without_regex(self): + # T19 via CLI: validate-io returns structured diagnostics with exact JSON pointer + conf_path = ROOT / "demo/fixtures/mock/pipeline_entity_extract.conf" + conf = json.loads(conf_path.read_text()) + pipe_file = conf_path.with_name(conf["pipe_path"]) + pipe_doc = json.loads(pipe_file.read_text()) + + with tempfile.TemporaryDirectory(prefix="validate-io-", dir=ROOT / "build") as directory: + changed_conf = Path(directory) / "pipeline.conf" + changed_pipe = Path(directory) / conf["pipe_path"] + # Erase required output slot entity_out + pipe_doc["deployment"]["io"]["output_allocations"].pop("entity_out") + changed_pipe.write_text(json.dumps(pipe_doc)) + changed_conf.write_text(json.dumps(conf)) + + code, res = self.command("validate-io", str(changed_conf)) + self.assertEqual(code, 1) + self.assertFalse(res["ok"]) + self.assertEqual(res["diagnostics"][0]["code"], "IO_VALIDATION_ERROR") + self.assertEqual( + res["diagnostics"][0]["path"], + "/deployment/io/output_allocations/entity_out", + ) + + def test_rfc0062_cli_override_unknown_and_escaped_model_id_t06(self): + # T06 via CLI: unknown model ID, special characters ~ and /, and invalid override syntax (null, empty, non-string) + pipeline = json.loads( + (ROOT / "demo/fixtures/mock/pipeline_entity_extract.json").read_text() + ) + # 1. Unknown model ID in override + doc1 = copy.deepcopy(pipeline) + doc1["deployment"]["model_paths"] = {"nonexistent_mid": "models/foo.bin"} + + # 2. Unknown model ID with special characters (~ -> ~0, / -> ~1) + doc2 = copy.deepcopy(pipeline) + doc2["deployment"]["model_paths"] = {"bad~id/extra": "models/foo.bin"} + + # 3. Override value is non-string (integer) + doc3 = copy.deepcopy(pipeline) + first_mid = list(doc3["deployment"]["model_paths"].keys())[0] + doc3["deployment"]["model_paths"][first_mid] = 12345 + + # 4. Override value is empty string + doc4 = copy.deepcopy(pipeline) + doc4["deployment"]["model_paths"][first_mid] = "" + + # 5. Override value is null + doc5 = copy.deepcopy(pipeline) + doc5["deployment"]["model_paths"][first_mid] = None + + cases = [ + ("unknown_id", doc1, "UNKNOWN_MODEL_ID", "/deployment/model_paths/nonexistent_mid"), + ("escaped_chars", doc2, "UNKNOWN_MODEL_ID", "/deployment/model_paths/bad~0id~1extra"), + ("non_string", doc3, "DEPLOYMENT_ERROR", f"/deployment/model_paths/{first_mid}"), + ("empty_value", doc4, "DEPLOYMENT_ERROR", f"/deployment/model_paths/{first_mid}"), + ("null_value", doc5, "DEPLOYMENT_ERROR", f"/deployment/model_paths/{first_mid}"), + ] + + for case_name, doc, exp_code, exp_path in cases: + for ep in self.CLI_PARITY_ENTRYPOINTS: + with self.subTest(case=case_name, entrypoint=ep): + code, res = self.command(*ep, "--stdin", input_pipeline=doc) + self.assertEqual(code, 1) + self.assertFalse(res["ok"]) + self.assertEqual(res["diagnostics"][0]["code"], exp_code) + self.assertEqual(res["diagnostics"][0]["path"], exp_path) + if ep[0] == "plan": + self.assertEqual(res["plan"], {"layers": [], "topological_order": []}) + + def test_rfc0062_cli_unknown_binding_and_biz_mismatch_t07(self): + # T07 via CLI: unknown io_binding, biz mismatch, and non-string biz + pipeline = json.loads( + (ROOT / "demo/fixtures/mock/pipeline_entity_extract.json").read_text() + ) + # 1. Unknown io_binding + doc1 = copy.deepcopy(pipeline) + doc1["deployment"]["io"]["io_binding"] = "nonexistent.binding.v99" + + # 2. Biz mismatch: pipeline biz_name doesn't match binding + doc2 = copy.deepcopy(pipeline) + doc2["biz_name"] = "unmatched_biz_name" + + # 3. Biz is non-string (integer) + doc3 = copy.deepcopy(pipeline) + doc3["biz_name"] = 12345 + + cases = [ + ("unknown_binding", doc1, "UNKNOWN_IO_BINDING", "/deployment/io/io_binding"), + ("biz_mismatch", doc2, "BIZ_MISMATCH", "/deployment/io/io_binding"), + ("biz_non_string", doc3, "FIELD_TYPE", "/biz_name"), + ] + + for case_name, doc, exp_code, exp_path in cases: + for ep in self.CLI_PARITY_ENTRYPOINTS: + with self.subTest(case=case_name, entrypoint=ep): + code, res = self.command(*ep, "--stdin", input_pipeline=doc) + self.assertEqual(code, 1) + self.assertFalse(res["ok"]) + self.assertEqual(res["diagnostics"][0]["code"], exp_code) + self.assertEqual(res["diagnostics"][0]["path"], exp_path) + if ep[0] == "plan": + self.assertEqual(res["plan"], {"layers": [], "topological_order": []}) + + def test_rfc0062_cli_output_slot_allocations_t08(self): + # T08 via CLI: missing required slot, unknown slot, allocation configuration error, and invalid capacity + pipeline = json.loads( + (ROOT / "demo/fixtures/mock/pipeline_entity_extract.json").read_text() + ) + # 1. Missing required output slot entity_out + doc1 = copy.deepcopy(pipeline) + doc1["deployment"]["io"]["output_allocations"].pop("entity_out") + + # 2. Unknown output slot + doc2 = copy.deepcopy(pipeline) + doc2["deployment"]["io"]["output_allocations"]["bogus_slot"] = {"type": "entity_out"} + + # 3. Invalid output allocation (missing required 'type' field) + doc3 = copy.deepcopy(pipeline) + doc3["deployment"]["io"]["output_allocations"]["entity_out"].pop("type") + + # 4. Invalid output allocation capacity (non-positive capacity value 0) + doc4 = copy.deepcopy(pipeline) + doc4["deployment"]["io"]["output_allocations"]["entity_out"]["capacities"]["entities_json"] = 0 + + cases = [ + ("missing_slot", doc1, "MISSING_OUTPUT_SLOT", "/deployment/io/output_allocations/entity_out"), + ("unknown_slot", doc2, "UNKNOWN_OUTPUT_SLOT", "/deployment/io/output_allocations/bogus_slot"), + ("invalid_alloc", doc3, "INVALID_OUTPUT_ALLOCATION", "/deployment/io/output_allocations/entity_out"), + ("invalid_capacity", doc4, "INVALID_OUTPUT_ALLOCATION", "/deployment/io/output_allocations/entity_out"), + ] + + for case_name, doc, exp_code, exp_path in cases: + for ep in self.CLI_PARITY_ENTRYPOINTS: + with self.subTest(case=case_name, entrypoint=ep): + code, res = self.command(*ep, "--stdin", input_pipeline=doc) + self.assertEqual(code, 1) + self.assertFalse(res["ok"]) + self.assertEqual(res["diagnostics"][0]["code"], exp_code) + self.assertEqual(res["diagnostics"][0]["path"], exp_path) + if ep[0] == "plan": + self.assertEqual(res["plan"], {"layers": [], "topological_order": []}) + + def test_rfc0062_cli_multiple_core_errors_with_deployment_t12(self): + # T12 via CLI: valid deployment, but Core has multiple Node errors. + # All diagnostics must be preserved in the response array across all CLI entrypoints. + pipeline = json.loads( + (ROOT / "demo/fixtures/mock/pipeline_entity_extract.json").read_text() + ) + # Add two invalid nodes to pipeline + doc = copy.deepcopy(pipeline) + doc["pipeline"].append({ + "id": "bad_node_1", + "node_type": "CompletelyUnknownNodeTypeOne", + "depends_on": [], + }) + doc["pipeline"].append({ + "id": "bad_node_2", + "node_type": "CompletelyUnknownNodeTypeTwo", + "depends_on": [], + }) + + for ep in self.CLI_PARITY_ENTRYPOINTS: + with self.subTest(entrypoint=ep): + code, res = self.command(*ep, "--stdin", input_pipeline=doc) + self.assertEqual(code, 1) + self.assertFalse(res["ok"]) + self.assertIn("diagnostics", res) + # Must retain multiple diagnostics, not compressed + self.assertGreaterEqual(len(res["diagnostics"]), 2) + diag_codes = [d["code"] for d in res["diagnostics"]] + self.assertIn("UNKNOWN_NODE_TYPE", diag_codes) + if ep[0] == "plan": + self.assertIn("plan", res) + + def test_rfc0062_cli_edit_invalid_deployment_t14(self): + # T14 via CLI: edit on invalid deployment with require_valid=false vs require_valid=true + pipeline = json.loads( + (ROOT / "demo/fixtures/mock/pipeline_entity_extract.json").read_text() + ) + invalid_doc = copy.deepcopy(pipeline) + invalid_doc["deployment"]["io"]["io_binding"] = "invalid_binding_id" + + op = { + "kind": "add_node", + "node_type": "TextTemplateNode", + "id": "draft_node_t14", + "config": {"template": "{{input}}"}, + } + + # Case 1: require_valid=false -> returns modified draft with validation.ok=false + req_false = { + "schema_version": 1, + "pipeline": invalid_doc, + "operation": op, + "require_valid": False, + } + code_false, res_false = self.command("edit", "--stdin", input_pipeline=req_false) + self.assertEqual(code_false, 0) + self.assertTrue(res_false["ok"]) + self.assertIn("pipeline", res_false) + draft_node_ids = [n["id"] for n in res_false["pipeline"]["pipeline"]] + self.assertIn("draft_node_t14", draft_node_ids) + self.assertIn("validation", res_false) + self.assertFalse(res_false["validation"]["ok"]) + self.assertEqual(res_false["validation"]["diagnostics"][0]["code"], "UNKNOWN_IO_BINDING") + + # Case 2: require_valid=true -> rejects without returning modified pipeline + req_true = { + "schema_version": 1, + "pipeline": invalid_doc, + "operation": op, + "require_valid": True, + } + code_true, res_true = self.command("edit", "--stdin", input_pipeline=req_true) + self.assertEqual(code_true, 1) + self.assertFalse(res_true["ok"]) + self.assertNotIn("pipeline", res_true) + self.assertIn("validation", res_true) + self.assertFalse(res_true["validation"]["ok"]) + self.assertEqual(res_true["validation"]["diagnostics"][0]["code"], "UNKNOWN_IO_BINDING") + + def test_rfc0062_cli_fix_deps_invalid_deployment_t15(self): + # T15 via CLI: fix-deps on pipeline where dependencies can be fixed but deployment is invalid + pipe = json.loads((ROOT / "configs" / "pipeline_doc_qa_default.json").read_text()) + # Introduce fixable dependency issue in Core + pipe["pipeline"][2]["depends_on"] = [] + # Introduce invalid deployment + pipe["deployment"]["io"]["io_binding"] = "invalid_binding_id" + + with tempfile.TemporaryDirectory() as td: + fpath = Path(td) / "pipeline_fixable_dep_invalid_deploy.json" + orig_text = json.dumps(pipe, indent=2) + fpath.write_text(orig_text) + + code, res = self.command("fix-deps", str(fpath), "--in-place") + self.assertEqual(code, 1) + self.assertFalse(res["ok"]) + self.assertFalse(res["written"]) + # Assert file was NOT written/modified + self.assertEqual(fpath.read_text(), orig_text) + # Structured deployment diagnostics are returned + self.assertIn("validation", res) + self.assertFalse(res["validation"]["ok"]) + diag = res["validation"]["diagnostics"][0] + self.assertEqual(diag["code"], "UNKNOWN_IO_BINDING") + self.assertEqual(diag["path"], "/deployment/io/io_binding") + + # Also verify preview mode (without --in-place): does not write, returns failure with diagnostics + code_prev, res_prev = self.command("fix-deps", str(fpath)) + self.assertEqual(code_prev, 1) + self.assertFalse(res_prev["ok"]) + self.assertFalse(res_prev["written"]) + self.assertIn("validation", res_prev) + self.assertFalse(res_prev["validation"]["ok"]) + diag_prev = res_prev["validation"]["diagnostics"][0] + self.assertEqual(diag_prev["code"], "UNKNOWN_IO_BINDING") + self.assertEqual(diag_prev["path"], "/deployment/io/io_binding") + + def test_rfc0062_cli_exception_barriers(self): + # RFC §2.8: Outer exception barriers for validate, plan, validate-io, and resolve-conf + # output schema-compliant JSON error envelopes with code INTERNAL_EXCEPTION or DEPLOYMENT_CONFIG / IO_VALIDATION_ERROR. + # 1. validate-io on non-existent config file + code, res = self.command("validate-io", "nonexistent_config_file.conf") + self.assertEqual(code, 1) + self.assertFalse(res["ok"]) + self.assertEqual(res["schema_version"], 1) + self.assertEqual(res["diagnostics"][0]["code"], "IO_VALIDATION_ERROR") + self.assertIn("nonexistent_config_file.conf", res["diagnostics"][0]["message"]) + + # 2. resolve-conf on non-existent file + code, res = self.command("resolve-conf", "nonexistent_config_file.conf", "--root", str(ROOT)) + self.assertEqual(code, 1) + self.assertFalse(res["ok"]) + self.assertEqual(res["schema_version"], 1) + self.assertEqual(res["diagnostics"][0]["code"], "DEPLOYMENT_CONFIG") + self.assertEqual(res["diagnostics"][0]["path"], "/") + + # 3. validate on non-existent file + code, res = self.command("validate", "nonexistent_pipeline.json") + self.assertEqual(code, 1) + self.assertFalse(res["ok"]) + self.assertEqual(res["schema_version"], 1) + self.assertEqual(res["diagnostics"][0]["code"], "JSON_READ") + def test_native_viewer_preserves_explicit_dag_dependencies(self): process = subprocess.run( [str(ALG_SHOW), str(ROOT / "configs" / "pipeline_doc_qa_default.json")], diff --git a/tests/unit/adapter/test_adapter_purity.cpp b/tests/unit/adapter/test_adapter_purity.cpp index ff1865c..33f6d61 100644 --- a/tests/unit/adapter/test_adapter_purity.cpp +++ b/tests/unit/adapter/test_adapter_purity.cpp @@ -1246,6 +1246,17 @@ TEST_F(AdapterPurityTest, ReuseProof_5_SameCarrierDifferentSchema) { // Proof 6: Negative Combinations Rejected TEST_F(AdapterPurityTest, ReuseProof_6_NegativeCombinations) { + nlohmann::json valid_pipeline = nlohmann::json::array( + {{{"id", "node_0_TextRuleMatchNode"}, + {"node_type", "TextRuleMatchNode"}, + {"depends_on", nlohmann::json::array()}, + {"ports", + {{"inputs", {{"text", "input_sentences"}}}, + {"outputs", {{"matches", "rule_matches"}}}}}, + {"config", + {{"categories", + {{"SYSTEM_INIT", nlohmann::json::array({"init"})}}}}}}}); + // 1. Unknown or unregistered io_binding nlohmann::json bad_binding_json = { {"biz_name", "keyword_match_v1"}, @@ -1259,7 +1270,7 @@ TEST_F(AdapterPurityTest, ReuseProof_6_NegativeCombinations) { {"metadata_type_id", 0}, {"capacities", {{"match_result_json", 2047}}}}}}}}}}}, {"models", nlohmann::json::array()}, - {"pipeline", nlohmann::json::array()}}; + {"pipeline", valid_pipeline}}; std::unique_ptr plan; std::string error; int ret = IoBindingResolver::ResolveFromPipelineJson( @@ -1292,7 +1303,7 @@ TEST_F(AdapterPurityTest, ReuseProof_6_NegativeCombinations) { {{"io_binding", "keyword_match.operator.v1"}, {"output_allocations", {{"unknown_slot", {{"type", "String"}}}}}}}}}, {"models", nlohmann::json::array()}, - {"pipeline", nlohmann::json::array()}}; + {"pipeline", valid_pipeline}}; ret = IoBindingResolver::ResolveFromPipelineJson(unknown_out_json, "operator", "./models", &plan, &error); EXPECT_EQ(ret, -2); @@ -1313,7 +1324,7 @@ TEST_F(AdapterPurityTest, ReuseProof_6_NegativeCombinations) { {"metadata_type_id", 0}, {"capacities", {{"match_result_json", 2047}}}}}}}}}}}, {"models", nlohmann::json::array()}, - {"pipeline", nlohmann::json::array()}}; + {"pipeline", valid_pipeline}}; ret = IoBindingResolver::ResolveFromPipelineJson(unknown_mid_json, "operator", "./models", &plan, &error); EXPECT_EQ(ret, -2); diff --git a/tests/unit/adapter/test_io_binding_registry.cpp b/tests/unit/adapter/test_io_binding_registry.cpp index c00ffac..4abd33b 100644 --- a/tests/unit/adapter/test_io_binding_registry.cpp +++ b/tests/unit/adapter/test_io_binding_registry.cpp @@ -5,7 +5,9 @@ #include #include "adapter/converter_authoring.h" +#include "adapter/deployment_diagnostic.h" #include "adapter/deployment_io_config.h" +#include "adapter/deployment_preparation.h" #include "adapter/io_binding_registry.h" #include "adapter/io_binding_resolver.h" #include "adapter/io_converter_registry.h" @@ -31,6 +33,17 @@ int DummyEncode(AlgContext*, const OutputPortBindings&, return 0; } +nlohmann::json DefaultPipelineNodes() { + nlohmann::json node; + node["id"] = "node_0"; + node["node_type"] = "TextRuleMatchNode"; + node["depends_on"] = nlohmann::json::array(); + node["ports"]["inputs"]["text"] = "input_sentences"; + node["ports"]["outputs"]["matches"] = "llm_answers"; + node["config"]["categories"]["CAT"] = nlohmann::json::array({"word"}); + return nlohmann::json::array({node}); +} + } // namespace class IoBindingRegistryTest : public ::testing::Test { @@ -313,6 +326,73 @@ TEST_F(IoBindingRegistryTest, DeploymentIoConfigValidation) { EXPECT_FALSE(DeploymentIoConfig::Parse(escape_cfg, tmp_dir, "operator", &parsed, &err)); + // 6. JSON Pointer 转义未知键 (例如 "bad~/field" -> "/bad~0~1field") + nlohmann::json escaped_key_cfg = valid_cfg; + escaped_key_cfg["bad~/field"] = 1; + DeploymentDiagnostic diag; + EXPECT_FALSE(DeploymentIoConfig::Parse(escaped_key_cfg, tmp_dir, "operator", + &parsed, &err, &diag)); + EXPECT_EQ(diag.code, "DEPLOYMENT_ERROR"); + EXPECT_EQ(diag.path, "/bad~0~1field"); + + // 7. ReadFromFile 在解析错误时携带配置文件路径上下文 + const std::string bad_conf_path = tmp_dir + "/bad_config.conf"; + { + std::ofstream ofs(bad_conf_path); + ofs << escaped_key_cfg.dump(); + } + std::string read_err; + DeploymentDiagnostic read_diag; + EXPECT_FALSE(DeploymentIoConfig::ReadFromFile( + bad_conf_path, "operator", &parsed, &read_err, &read_diag)); + EXPECT_EQ(read_diag.code, "DEPLOYMENT_ERROR"); + EXPECT_EQ(read_diag.path, "/bad~0~1field"); + EXPECT_EQ(read_err.rfind("Error in config file " + bad_conf_path + ": ", 0), + 0); + EXPECT_EQ(read_diag.message.rfind( + "Error in config file " + bad_conf_path + ": ", 0), + 0); + + // 8. ReadFromFile 针对旧 Schema 1 报错同样携带配置文件路径上下文 + const std::string dep_conf_path = tmp_dir + "/deprecated.conf"; + { + std::ofstream ofs(dep_conf_path); + ofs << old_schema1.dump(); + } + EXPECT_FALSE(DeploymentIoConfig::ReadFromFile( + dep_conf_path, "operator", &parsed, &read_err, &read_diag)); + EXPECT_EQ(read_diag.code, "DEPLOYMENT_ERROR"); + EXPECT_EQ(read_err.rfind("Error in config file " + dep_conf_path + ": ", 0), + 0); + EXPECT_EQ(read_diag.message.rfind( + "Error in config file " + dep_conf_path + ": ", 0), + 0); + + // 9. ReadFromFile 文件路径包含关键保留词 (如 pipe_path.conf) + // 时,仍必须正确携带文件路径前缀 + const std::string keyword_conf_path = tmp_dir + "/pipe_path.conf"; + { + std::ofstream ofs(keyword_conf_path); + ofs << escaped_key_cfg.dump(); + } + EXPECT_FALSE(DeploymentIoConfig::ReadFromFile( + keyword_conf_path, "operator", &parsed, &read_err, &read_diag)); + EXPECT_EQ(read_diag.code, "DEPLOYMENT_ERROR"); + EXPECT_EQ(read_diag.path, "/bad~0~1field"); + EXPECT_EQ( + read_err.rfind("Error in config file " + keyword_conf_path + ": ", 0), 0); + EXPECT_EQ(read_diag.message.rfind( + "Error in config file " + keyword_conf_path + ": ", 0), + 0); + + // 10. 多个 ~ 与 / 字符的转义校验 + nlohmann::json multi_escape_cfg = valid_cfg; + multi_escape_cfg["a~b/c~0/d~1"] = 42; + EXPECT_FALSE(DeploymentIoConfig::Parse(multi_escape_cfg, tmp_dir, "operator", + &parsed, &err, &diag)); + EXPECT_EQ(diag.code, "DEPLOYMENT_ERROR"); + EXPECT_EQ(diag.path, "/a~0b~1c~00~1d~01"); + fs::remove_all(tmp_dir); } @@ -564,7 +644,7 @@ TEST_F(IoBindingRegistryTest, BizMismatchFailsClosedWithExactPointer) { {"metadata_type_id", 0}, {"capacities", {{"entities_json", 2047}}}}}}}}}}}, {"models", nlohmann::json::array()}, - {"pipeline", nlohmann::json::array()}}; + {"pipeline", DefaultPipelineNodes()}}; std::unique_ptr plan; std::string err; @@ -592,7 +672,7 @@ TEST_F(IoBindingRegistryTest, EscapedJsonPointerInModelPathsAndSlots) { {"output_allocations", {{"slot~0/bad", {{"type", "entity_out"}}}}}}}}}, {"models", nlohmann::json::array()}, - {"pipeline", nlohmann::json::array()}}; + {"pipeline", DefaultPipelineNodes()}}; std::unique_ptr plan; std::string err; @@ -618,7 +698,7 @@ TEST_F(IoBindingRegistryTest, EscapedJsonPointerInModelPathsAndSlots) { {"metadata_type_id", 0}, {"capacities", {{"entities_json", 2047}}}}}}}}}}}, {"models", nlohmann::json::array()}, - {"pipeline", nlohmann::json::array()}}; + {"pipeline", DefaultPipelineNodes()}}; ret = IoBindingResolver::ResolveFromPipelineJson(model_doc, "operator", "./models", &plan, &err); @@ -655,7 +735,7 @@ TEST_F(IoBindingRegistryTest, {"model_config", {{"embedding_dim", 128}, {"max_batch_size", 4}}}, {"backend_config", nlohmann::json::object()}, {"model_path", "models/legal.bin"}}}}, - {"pipeline", nlohmann::json::array()}}; + {"pipeline", DefaultPipelineNodes()}}; std::unique_ptr plan; std::string err; @@ -682,4 +762,931 @@ TEST_F(IoBindingRegistryTest, << "ACTUAL ERR: " << err; } +TEST_F(IoBindingRegistryTest, + PrepareDeploymentSuccessAndBoundaryExtraction_T01) { + RegisterTestBizBinding(); + + nlohmann::json doc = { + {"biz_name", "test_biz_v1"}, + {"deployment", + {{"model_paths", {{"mid_1", "models/override.bin"}}}, + {"io", + {{"io_binding", "test_biz.operator.v1"}, + {"output_allocations", + {{"entity_out", + {{"type", "entity_out"}, + {"meta_num", 0}, + {"metadata_type_id", 0}, + {"capacities", {{"entities_json", 2047}}}}}}}}}}}, + {"models", + {{{"model_id", "mid_1"}, + {"capability", "embedding"}, + {"model_type", "test_biz_embedding"}, + {"backend", "test_tensor_backend"}, + {"model_config", {{"embedding_dim", 128}, {"max_batch_size", 4}}}, + {"backend_config", nlohmann::json::object()}, + {"model_path", "models/original.bin"}}}}, + {"pipeline", DefaultPipelineNodes()}}; + + DeploymentPrepareOptions options; + options.transport = "operator"; + options.path_mode = DeploymentPathMode::kLexicalOnly; + + PreparedDeployment prepared; + DeploymentDiagnostic diag; + + // 1. With override + EXPECT_TRUE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_FALSE(prepared.neutral_pipeline_json.contains("deployment")); + EXPECT_EQ(prepared.neutral_pipeline_json["biz_name"], "test_biz_v1"); + EXPECT_EQ(prepared.neutral_pipeline_json["models"][0]["model_path"], + "models/override.bin"); + EXPECT_EQ(prepared.overridden_model_ids.count("mid_1"), 1u); + ASSERT_EQ(prepared.model_path_source_pointers.size(), 1u); + EXPECT_EQ(prepared.model_path_source_pointers[0], + "/deployment/model_paths/mid_1"); + EXPECT_EQ(prepared.binding.binding_id, "test_biz.operator.v1"); + ASSERT_NE(prepared.input_converter, nullptr); + ASSERT_NE(prepared.output_converter, nullptr); + EXPECT_EQ(prepared.output_specs.count("entity_out"), 1u); + EXPECT_EQ(prepared.io_boundary.input_published_ports.size(), 1u); + EXPECT_EQ(prepared.io_boundary.input_published_ports[0].Name(), + "input_sentences"); + EXPECT_EQ(prepared.io_boundary.output_consumed_ports.size(), 1u); + EXPECT_EQ(prepared.io_boundary.output_consumed_ports[0].Name(), + "llm_answers"); + + // 2. Without override + nlohmann::json doc_no_override = doc; + doc_no_override["deployment"].erase("model_paths"); + prepared.Clear(); + EXPECT_TRUE( + PrepareDeploymentDocument(doc_no_override, options, &prepared, &diag)); + EXPECT_EQ(prepared.neutral_pipeline_json["models"][0]["model_path"], + "models/original.bin"); + EXPECT_EQ(prepared.overridden_model_ids.count("mid_1"), 0u); + ASSERT_EQ(prepared.model_path_source_pointers.size(), 1u); + EXPECT_EQ(prepared.model_path_source_pointers[0], "/models/0/model_path"); +} + +TEST_F(IoBindingRegistryTest, MissingDeploymentFails_T02) { + RegisterTestBizBinding(); + + nlohmann::json neutral_doc = { + {"biz_name", "test_biz_v1"}, + {"models", + {{{"model_id", "mid_1"}, + {"capability", "embedding"}, + {"model_type", "test_biz_embedding"}, + {"backend", "test_tensor_backend"}, + {"model_config", {{"embedding_dim", 128}, {"max_batch_size", 4}}}, + {"backend_config", nlohmann::json::object()}, + {"model_path", "models/original.bin"}}}}, + {"pipeline", DefaultPipelineNodes()}}; + + DeploymentPrepareOptions options; + options.transport = "operator"; + options.path_mode = DeploymentPathMode::kLexicalOnly; + + PreparedDeployment prepared; + DeploymentDiagnostic diag; + + // 1. Missing deployment fails in PrepareDeploymentDocument + EXPECT_FALSE( + PrepareDeploymentDocument(neutral_doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "MISSING_DEPLOYMENT_IO"); + EXPECT_EQ(diag.path, "/deployment/io"); + EXPECT_EQ(diag.legacy_status, -2); + + // 2. Core direct validation of a document containing deployment fails with + // unknown field + nlohmann::json doc_with_deployment = neutral_doc; + doc_with_deployment["deployment"] = nlohmann::json::object(); + ParsedPipelineConfig parsed_config; + PipelineDiagnostic core_diag; + EXPECT_FALSE( + ParsePipelineConfig(doc_with_deployment, &parsed_config, &core_diag)); + EXPECT_EQ(core_diag.code, DiagnosticCode::kUnknownField); + EXPECT_EQ(core_diag.path, "/deployment"); + + // 3. Core direct validation of neutral doc succeeds + EXPECT_TRUE(ParsePipelineConfig(neutral_doc, &parsed_config, &core_diag)); +} + +TEST_F(IoBindingRegistryTest, + OriginalModelPathInvalidRejectedEvenWithOverride_T03) { + RegisterTestBizBinding(); + + nlohmann::json base_doc = { + {"biz_name", "test_biz_v1"}, + {"deployment", + {{"model_paths", {{"mid_1", "models/valid_override.bin"}}}, + {"io", + {{"io_binding", "test_biz.operator.v1"}, + {"output_allocations", + {{"entity_out", + {{"type", "entity_out"}, + {"meta_num", 0}, + {"metadata_type_id", 0}, + {"capacities", {{"entities_json", 2047}}}}}}}}}}}, + {"models", + {{{"model_id", "mid_1"}, + {"capability", "embedding"}, + {"model_type", "test_biz_embedding"}, + {"backend", "test_tensor_backend"}, + {"model_config", {{"embedding_dim", 128}, {"max_batch_size", 4}}}, + {"backend_config", nlohmann::json::object()}}}}, + {"pipeline", DefaultPipelineNodes()}}; + + DeploymentPrepareOptions options; + options.transport = "operator"; + options.path_mode = DeploymentPathMode::kLexicalOnly; + + PreparedDeployment prepared; + DeploymentDiagnostic diag; + + // Subcase 1: model_path missing completely + { + nlohmann::json doc = base_doc; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "MISSING_FIELD"); + EXPECT_EQ(diag.path, "/models/0/model_path"); + EXPECT_EQ(diag.legacy_status, -3); + + // Paired check without override: same result + doc["deployment"].erase("model_paths"); + DeploymentDiagnostic diag_no_override; + EXPECT_FALSE( + PrepareDeploymentDocument(doc, options, &prepared, &diag_no_override)); + EXPECT_EQ(diag_no_override.code, "MISSING_FIELD"); + EXPECT_EQ(diag_no_override.path, "/models/0/model_path"); + EXPECT_EQ(diag_no_override.legacy_status, -3); + } + + // Subcase 2: model_path is null + { + nlohmann::json doc = base_doc; + doc["models"][0]["model_path"] = nullptr; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "INVALID_MODEL_PATH"); + EXPECT_EQ(diag.path, "/models/0/model_path"); + EXPECT_EQ(diag.legacy_status, -2); + + doc["deployment"].erase("model_paths"); + DeploymentDiagnostic diag_no_override; + EXPECT_FALSE( + PrepareDeploymentDocument(doc, options, &prepared, &diag_no_override)); + EXPECT_EQ(diag_no_override.code, "INVALID_MODEL_PATH"); + EXPECT_EQ(diag_no_override.path, "/models/0/model_path"); + EXPECT_EQ(diag_no_override.legacy_status, -2); + } + + // Subcase 3: model_path is integer + { + nlohmann::json doc = base_doc; + doc["models"][0]["model_path"] = 12345; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "INVALID_MODEL_PATH"); + EXPECT_EQ(diag.path, "/models/0/model_path"); + EXPECT_EQ(diag.legacy_status, -2); + + doc["deployment"].erase("model_paths"); + DeploymentDiagnostic diag_no_override; + EXPECT_FALSE( + PrepareDeploymentDocument(doc, options, &prepared, &diag_no_override)); + EXPECT_EQ(diag_no_override.code, "INVALID_MODEL_PATH"); + EXPECT_EQ(diag_no_override.path, "/models/0/model_path"); + EXPECT_EQ(diag_no_override.legacy_status, -2); + } + + // Subcase 4: model_path is empty string + { + nlohmann::json doc = base_doc; + doc["models"][0]["model_path"] = ""; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "INVALID_MODEL_PATH"); + EXPECT_EQ(diag.path, "/models/0/model_path"); + EXPECT_EQ(diag.legacy_status, -2); + + doc["deployment"].erase("model_paths"); + DeploymentDiagnostic diag_no_override; + EXPECT_FALSE( + PrepareDeploymentDocument(doc, options, &prepared, &diag_no_override)); + EXPECT_EQ(diag_no_override.code, "INVALID_MODEL_PATH"); + EXPECT_EQ(diag_no_override.path, "/models/0/model_path"); + EXPECT_EQ(diag_no_override.legacy_status, -2); + } +} + +TEST_F(IoBindingRegistryTest, + OriginalModelStructureInvalidRejectedEvenWithOverride_T04) { + RegisterTestBizBinding(); + + DeploymentPrepareOptions options; + options.transport = "operator"; + options.path_mode = DeploymentPathMode::kLexicalOnly; + PreparedDeployment prepared; + DeploymentDiagnostic diag; + + // Case 1: models is not an array (e.g. object) + { + nlohmann::json doc = {{"biz_name", "test_biz_v1"}, + {"deployment", + {{"model_paths", {{"mid_1", "models/override.bin"}}}, + {"io", + {{"io_binding", "test_biz.operator.v1"}, + {"output_allocations", + {{"entity_out", {{"type", "entity_out"}}}}}}}}}, + {"models", {{"mid_1", "not_an_array"}}}, + {"pipeline", DefaultPipelineNodes()}}; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "FIELD_TYPE"); + EXPECT_EQ(diag.path, "/models"); + } + + // Case 2: models[0] missing required capability + { + nlohmann::json doc = {{"biz_name", "test_biz_v1"}, + {"deployment", + {{"model_paths", {{"mid_1", "models/override.bin"}}}, + {"io", + {{"io_binding", "test_biz.operator.v1"}, + {"output_allocations", + {{"entity_out", {{"type", "entity_out"}}}}}}}}}, + {"models", + {{{"model_id", "mid_1"}, + {"model_type", "test_biz_embedding"}, + {"backend", "test_tensor_backend"}, + {"model_path", "models/original.bin"}}}}, + {"pipeline", DefaultPipelineNodes()}}; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "MISSING_FIELD"); + EXPECT_EQ(diag.path, "/models/0/capability"); + } + + // Case 3: Duplicate model_id + { + nlohmann::json doc = {{"biz_name", "test_biz_v1"}, + {"deployment", + {{"model_paths", {{"mid_1", "models/override.bin"}}}, + {"io", + {{"io_binding", "test_biz.operator.v1"}, + {"output_allocations", + {{"entity_out", {{"type", "entity_out"}}}}}}}}}, + {"models", + {{{"model_id", "mid_1"}, + {"capability", "embedding"}, + {"model_type", "test_biz_embedding"}, + {"backend", "test_tensor_backend"}, + {"model_path", "models/orig1.bin"}}, + {{"model_id", "mid_1"}, + {"capability", "embedding"}, + {"model_type", "test_biz_embedding"}, + {"backend", "test_tensor_backend"}, + {"model_path", "models/orig2.bin"}}}}, + {"pipeline", DefaultPipelineNodes()}}; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "DUPLICATE_MODEL_ID"); + EXPECT_EQ(diag.path, "/models/1/model_id"); + } +} + +TEST_F(IoBindingRegistryTest, + OriginalModelPathNonexistentOnDiskWithValidOverride_T05) { + RegisterTestBizBinding(); + + // Create temporary directory with only the override model file + fs::path temp_dir = fs::temp_directory_path() / "edgeflow_test_t05"; + fs::create_directories(temp_dir); + fs::path override_file = temp_dir / "override_model.bin"; + { + std::ofstream ofs(override_file); + ofs << "dummy model data"; + } + + nlohmann::json doc = { + {"biz_name", "test_biz_v1"}, + {"deployment", + {{"model_paths", {{"mid_1", "override_model.bin"}}}, + {"io", + {{"io_binding", "test_biz.operator.v1"}, + {"output_allocations", + {{"entity_out", + {{"type", "entity_out"}, + {"meta_num", 0}, + {"metadata_type_id", 0}, + {"capacities", {{"entities_json", 2047}}}}}}}}}}}, + {"models", + {{{"model_id", "mid_1"}, + {"capability", "embedding"}, + {"model_type", "test_biz_embedding"}, + {"backend", "test_tensor_backend"}, + {"model_config", {{"embedding_dim", 128}, {"max_batch_size", 4}}}, + {"backend_config", nlohmann::json::object()}, + {"model_path", "nonexistent_dir/completely_missing_original.bin"}}}}, + {"pipeline", DefaultPipelineNodes()}}; + + DeploymentPrepareOptions options; + options.transport = "operator"; + options.path_mode = DeploymentPathMode::kUnderRoot; + options.model_root_dir = temp_dir.string(); + + PreparedDeployment prepared; + DeploymentDiagnostic diag; + // S2 checks original syntax only; S6 checks override path under root which + // exists. Missing original model on disk must NOT cause failure! + EXPECT_TRUE(PrepareDeploymentDocument(doc, options, &prepared, &diag)) + << diag.message; + EXPECT_EQ(prepared.neutral_pipeline_json["models"][0]["model_path"], + override_file.string()); + + fs::remove_all(temp_dir); +} + +TEST_F(IoBindingRegistryTest, OverrideUnknownModelIdOrInvalidSyntax_T06) { + RegisterTestBizBinding(); + + nlohmann::json base_doc = { + {"biz_name", "test_biz_v1"}, + {"deployment", + {{"io", + {{"io_binding", "test_biz.operator.v1"}, + {"output_allocations", + {{"entity_out", + {{"type", "entity_out"}, + {"meta_num", 0}, + {"metadata_type_id", 0}, + {"capacities", {{"entities_json", 2047}}}}}}}}}}}, + {"models", + {{{"model_id", "mid_1"}, + {"capability", "embedding"}, + {"model_type", "test_biz_embedding"}, + {"backend", "test_tensor_backend"}, + {"model_config", {{"embedding_dim", 128}, {"max_batch_size", 4}}}, + {"backend_config", nlohmann::json::object()}, + {"model_path", "models/original.bin"}}}}, + {"pipeline", DefaultPipelineNodes()}}; + + DeploymentPrepareOptions options; + options.transport = "operator"; + options.path_mode = DeploymentPathMode::kLexicalOnly; + PreparedDeployment prepared; + DeploymentDiagnostic diag; + + // Case 1: Override unknown model_id + { + nlohmann::json doc = base_doc; + doc["deployment"]["model_paths"] = { + {"nonexistent_model", "models/foo.bin"}}; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "UNKNOWN_MODEL_ID"); + EXPECT_EQ(diag.path, "/deployment/model_paths/nonexistent_model"); + EXPECT_EQ(diag.legacy_status, -2); + } + + // Case 2: Override unknown model_id with special chars (escaped pointer) + { + nlohmann::json doc = base_doc; + doc["deployment"]["model_paths"] = {{"non/exist~id", "models/foo.bin"}}; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "UNKNOWN_MODEL_ID"); + EXPECT_EQ(diag.path, "/deployment/model_paths/non~1exist~0id"); + EXPECT_EQ(diag.legacy_status, -2); + } + + // Case 3: Override value is non-string (e.g. integer) + { + nlohmann::json doc = base_doc; + doc["deployment"]["model_paths"] = {{"mid_1", 999}}; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "DEPLOYMENT_ERROR"); + EXPECT_EQ(diag.path, "/deployment/model_paths/mid_1"); + EXPECT_EQ(diag.legacy_status, -2); + } + + // Case 4: Override value is empty string + { + nlohmann::json doc = base_doc; + doc["deployment"]["model_paths"] = {{"mid_1", ""}}; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "DEPLOYMENT_ERROR"); + EXPECT_EQ(diag.path, "/deployment/model_paths/mid_1"); + EXPECT_EQ(diag.legacy_status, -2); + } +} + +TEST_F(IoBindingRegistryTest, DeploymentIoUnknownBindingOrMismatch_T07) { + RegisterTestBizBinding(); + + nlohmann::json base_doc = { + {"biz_name", "test_biz_v1"}, + {"deployment", + {{"io", + {{"io_binding", "test_biz.operator.v1"}, + {"output_allocations", + {{"entity_out", + {{"type", "entity_out"}, + {"meta_num", 0}, + {"metadata_type_id", 0}, + {"capacities", {{"entities_json", 2047}}}}}}}}}}}, + {"models", nlohmann::json::array()}, + {"pipeline", DefaultPipelineNodes()}}; + + DeploymentPrepareOptions options; + options.transport = "operator"; + options.path_mode = DeploymentPathMode::kLexicalOnly; + PreparedDeployment prepared; + DeploymentDiagnostic diag; + + // Case 1: Unknown binding + { + nlohmann::json doc = base_doc; + doc["deployment"]["io"]["io_binding"] = "completely_unknown_binding"; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "UNKNOWN_IO_BINDING"); + EXPECT_EQ(diag.path, "/deployment/io/io_binding"); + EXPECT_EQ(diag.legacy_status, -2); + } + + // Case 2: Biz name mismatch + { + nlohmann::json doc = base_doc; + doc["biz_name"] = "other_biz_v1"; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "BIZ_MISMATCH"); + EXPECT_EQ(diag.path, "/deployment/io/io_binding"); + EXPECT_EQ(diag.legacy_status, -2); + } + + // Case 3: Unsupported transport + { + nlohmann::json doc = base_doc; + DeploymentPrepareOptions rpc_opts = options; + rpc_opts.transport = "unsupported_transport"; + EXPECT_FALSE(PrepareDeploymentDocument(doc, rpc_opts, &prepared, &diag)); + EXPECT_EQ(diag.code, "UNSUPPORTED_TRANSPORT"); + EXPECT_EQ(diag.path, "/"); + EXPECT_EQ(diag.legacy_status, -2); + } +} + +TEST_F(IoBindingRegistryTest, DeploymentIoSlotValidation_T08) { + RegisterTestBizBinding(); + + nlohmann::json base_doc = { + {"biz_name", "test_biz_v1"}, + {"deployment", + {{"io", + {{"io_binding", "test_biz.operator.v1"}, + {"output_allocations", + {{"entity_out", + {{"type", "entity_out"}, + {"meta_num", 0}, + {"metadata_type_id", 0}, + {"capacities", {{"entities_json", 2047}}}}}}}}}}}, + {"models", nlohmann::json::array()}, + {"pipeline", DefaultPipelineNodes()}}; + + DeploymentPrepareOptions options; + options.transport = "operator"; + options.path_mode = DeploymentPathMode::kLexicalOnly; + PreparedDeployment prepared; + DeploymentDiagnostic diag; + + // Case 1: Missing required slot + { + nlohmann::json doc = base_doc; + doc["deployment"]["io"]["output_allocations"].erase("entity_out"); + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "MISSING_OUTPUT_SLOT"); + EXPECT_EQ(diag.path, "/deployment/io/output_allocations/entity_out"); + EXPECT_EQ(diag.legacy_status, -2); + } + + // Case 2: Unknown slot in output_allocations + { + nlohmann::json doc = base_doc; + doc["deployment"]["io"]["output_allocations"]["unexpected_extra_slot"] = { + {"type", "entity_out"}}; + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "UNKNOWN_OUTPUT_SLOT"); + EXPECT_EQ(diag.path, + "/deployment/io/output_allocations/unexpected_extra_slot"); + EXPECT_EQ(diag.legacy_status, -2); + } + + // Case 3: Slot allocation missing type field + { + nlohmann::json doc = base_doc; + doc["deployment"]["io"]["output_allocations"]["entity_out"].erase("type"); + EXPECT_FALSE(PrepareDeploymentDocument(doc, options, &prepared, &diag)); + EXPECT_EQ(diag.code, "INVALID_OUTPUT_ALLOCATION"); + EXPECT_EQ(diag.path, "/deployment/io/output_allocations/entity_out"); + EXPECT_EQ(diag.legacy_status, -2); + } +} + +TEST_F(IoBindingRegistryTest, PrepareFailureResetsPreparedStateAtomically_T18) { + RegisterTestBizBinding(); + + nlohmann::json valid_doc = { + {"biz_name", "test_biz_v1"}, + {"deployment", + {{"model_paths", {{"mid_1", "models/override.bin"}}}, + {"io", + {{"io_binding", "test_biz.operator.v1"}, + {"output_allocations", + {{"entity_out", + {{"type", "entity_out"}, + {"meta_num", 0}, + {"metadata_type_id", 0}, + {"capacities", {{"entities_json", 2047}}}}}}}}}}}, + {"models", + {{{"model_id", "mid_1"}, + {"capability", "embedding"}, + {"model_type", "test_biz_embedding"}, + {"backend", "test_tensor_backend"}, + {"model_config", {{"embedding_dim", 128}, {"max_batch_size", 4}}}, + {"backend_config", nlohmann::json::object()}, + {"model_path", "models/original.bin"}}}}, + {"pipeline", DefaultPipelineNodes()}}; + + DeploymentPrepareOptions options; + options.transport = "operator"; + options.path_mode = DeploymentPathMode::kLexicalOnly; + + PreparedDeployment prepared; + DeploymentDiagnostic diag; + + // 1. Initial successful preparation + ASSERT_TRUE(PrepareDeploymentDocument(valid_doc, options, &prepared, &diag)); + EXPECT_FALSE(prepared.binding.binding_id.empty()); + EXPECT_NE(prepared.input_converter, nullptr); + EXPECT_NE(prepared.output_converter, nullptr); + EXPECT_FALSE(prepared.neutral_pipeline_json.is_null()); + EXPECT_FALSE(prepared.output_specs.empty()); + + // 2. Reusing the SAME prepared instance on a failing document + nlohmann::json invalid_doc = valid_doc; + invalid_doc.erase("deployment"); + const nlohmann::json invalid_doc_copy = invalid_doc; + + EXPECT_FALSE( + PrepareDeploymentDocument(invalid_doc, options, &prepared, &diag)); + + // Verify all fields of prepared are completely reset + EXPECT_TRUE(prepared.binding.binding_id.empty()); + EXPECT_EQ(prepared.input_converter, nullptr); + EXPECT_EQ(prepared.output_converter, nullptr); + EXPECT_TRUE(prepared.input_port_bindings.All().empty()); + EXPECT_TRUE(prepared.output_port_bindings.All().empty()); + EXPECT_EQ(prepared.effective_max_batch_size, 0u); + EXPECT_TRUE(prepared.output_specs.empty()); + EXPECT_TRUE(prepared.output_parameter_texts.empty()); + EXPECT_TRUE(prepared.overridden_model_ids.empty()); + EXPECT_TRUE(prepared.model_path_source_pointers.empty()); + EXPECT_TRUE(prepared.neutral_pipeline_json.is_null()); + EXPECT_TRUE(prepared.io_boundary.input_published_ports.empty()); + EXPECT_TRUE(prepared.io_boundary.output_consumed_ports.empty()); + + // Verify the input document was NOT mutated + EXPECT_EQ(invalid_doc, invalid_doc_copy); +} + +TEST_F(IoBindingRegistryTest, + ProjectModelPathDiagnosticsProjectionAndRemediation) { + PreparedDeployment prepared; + prepared.overridden_model_ids = {"mid_0"}; + prepared.model_path_source_pointers = { + "/deployment/model_paths/mid_0", + "/models/1/model_path", + }; + + ValidationReport report; + report.ok = false; + + // Diagnostic 1: Points to model 0 (overridden) with a fix patching model_path + ValidationDiagnostic diag1; + diag1.code = DiagnosticCode::kFieldRange; + diag1.path = "/models/0/model_path"; + diag1.message = "File not found"; + ValidationRemediation rem1; + ValidationFix fix1; + fix1.id = "fix_model_0"; + fix1.patch = nlohmann::json::array({{{"op", "replace"}, + {"path", "/models/0/model_path"}, + {"value", "foo.bin"}}}); + rem1.fixes.push_back(fix1); + diag1.remediation = rem1; + report.diagnostics.push_back(diag1); + + // Diagnostic 2: Points to model 1 (un-overridden) with a fix patching + // model_path + ValidationDiagnostic diag2; + diag2.code = DiagnosticCode::kFieldRange; + diag2.path = "/models/1/model_path"; + diag2.message = "File not found 1"; + ValidationRemediation rem2; + ValidationFix fix2; + fix2.id = "fix_model_1"; + fix2.patch = nlohmann::json::array({{{"op", "replace"}, + {"path", "/models/1/model_path"}, + {"value", "bar.bin"}}}); + rem2.fixes.push_back(fix2); + diag2.remediation = rem2; + report.diagnostics.push_back(diag2); + + // Diagnostic 3: Non-model diagnostic + ValidationDiagnostic diag3; + diag3.code = DiagnosticCode::kMissingField; + diag3.path = "/pipeline/0/ports/inputs/text"; + diag3.message = "Port unbound"; + report.diagnostics.push_back(diag3); + + // Diagnostic 4: Special characters ~ and / in model ID + ValidationDiagnostic diag4; + diag4.code = DiagnosticCode::kFieldRange; + diag4.path = "/models/2/model_path"; + diag4.message = "File not found 2"; + report.diagnostics.push_back(diag4); + prepared.overridden_model_ids.insert("mid~special/0"); + prepared.model_path_source_pointers.push_back( + "/deployment/model_paths/mid~0special~10"); + + // Project diagnostics + ProjectModelPathDiagnostics(prepared, &report); + + ASSERT_EQ(report.diagnostics.size(), 4u); + + // Diagnostic 1: Path projected to /deployment/model_paths/mid_0, and fix + // targeting model_path removed! + EXPECT_EQ(report.diagnostics[0].path, "/deployment/model_paths/mid_0"); + ASSERT_TRUE(report.diagnostics[0].remediation.has_value()); + EXPECT_TRUE(report.diagnostics[0].remediation->fixes.empty()); + + // Diagnostic 2: Path stays /models/1/model_path, fix preserved! + EXPECT_EQ(report.diagnostics[1].path, "/models/1/model_path"); + ASSERT_TRUE(report.diagnostics[1].remediation.has_value()); + ASSERT_EQ(report.diagnostics[1].remediation->fixes.size(), 1u); + EXPECT_EQ(report.diagnostics[1].remediation->fixes[0].id, "fix_model_1"); + + // Diagnostic 3: Unchanged + EXPECT_EQ(report.diagnostics[2].path, "/pipeline/0/ports/inputs/text"); + + // Diagnostic 4: Escaped pointer with ~ and / + EXPECT_EQ(report.diagnostics[3].path, + "/deployment/model_paths/mid~0special~10"); +} + +TEST_F(IoBindingRegistryTest, + ResolveFromPipelineJsonDiagnosticCarrier_T03_T06_T07_T08) { + RegisterTestBizBinding(); + + // T03: Invalid original model path with override through IoBindingResolver + nlohmann::json t03_doc = { + {"biz_name", "test_biz_v1"}, + {"deployment", + {{"model_paths", {{"mid_1", "models/override.bin"}}}, + {"io", + {{"io_binding", "test_biz.operator.v1"}, + {"output_allocations", + {{"entity_out", + {{"type", "entity_out"}, + {"meta_num", 0}, + {"metadata_type_id", 0}, + {"capacities", {{"entities_json", 2047}}}}}}}}}}}, + {"models", + {{{"model_id", "mid_1"}, + {"capability", "embedding"}, + {"model_type", "test_biz_embedding"}, + {"backend", "test_tensor_backend"}, + {"model_config", {{"embedding_dim", 128}, {"max_batch_size", 4}}}, + {"backend_config", nlohmann::json::object()}}}}, + {"pipeline", DefaultPipelineNodes()}}; + + std::unique_ptr plan; + std::string err; + DeploymentDiagnostic diag; + int rc = IoBindingResolver::ResolveFromPipelineJson( + t03_doc, "operator", "./models", &plan, &err, &diag); + EXPECT_EQ(rc, -3); + EXPECT_EQ(plan, nullptr); + EXPECT_EQ(diag.code, "MISSING_FIELD"); + EXPECT_EQ(diag.path, "/models/0/model_path"); + + // T06: Unknown override model ID + nlohmann::json t06_doc = t03_doc; + t06_doc["models"][0]["model_path"] = "models/original.bin"; + t06_doc["deployment"]["model_paths"] = {{"unknown_mid", "models/foo.bin"}}; + rc = IoBindingResolver::ResolveFromPipelineJson( + t06_doc, "operator", "./models", &plan, &err, &diag); + EXPECT_EQ(rc, -2); + EXPECT_EQ(plan, nullptr); + EXPECT_EQ(diag.code, "UNKNOWN_MODEL_ID"); + EXPECT_EQ(diag.path, "/deployment/model_paths/unknown_mid"); + + // T07: Unknown binding through IoBindingResolver + nlohmann::json t07_doc = t03_doc; + t07_doc["models"][0]["model_path"] = "models/original.bin"; + t07_doc["deployment"]["io"]["io_binding"] = "nonexistent.binding"; + rc = IoBindingResolver::ResolveFromPipelineJson( + t07_doc, "operator", "./models", &plan, &err, &diag); + EXPECT_EQ(rc, -2); + EXPECT_EQ(plan, nullptr); + EXPECT_EQ(diag.code, "UNKNOWN_IO_BINDING"); + EXPECT_EQ(diag.path, "/deployment/io/io_binding"); + + // T08: Missing required output slot + nlohmann::json t08_doc = t03_doc; + t08_doc["models"][0]["model_path"] = "models/original.bin"; + t08_doc["deployment"]["io"]["output_allocations"].clear(); + rc = IoBindingResolver::ResolveFromPipelineJson( + t08_doc, "operator", "./models", &plan, &err, &diag); + EXPECT_EQ(rc, -2); + EXPECT_EQ(plan, nullptr); + EXPECT_EQ(diag.code, "MISSING_OUTPUT_SLOT"); + EXPECT_EQ(diag.path, "/deployment/io/output_allocations/entity_out"); +} + +nlohmann::json ValidTextTemplatePipelineNodes() { + nlohmann::json node; + node["id"] = "node_0"; + node["node_type"] = "TextTemplateNode"; + node["depends_on"] = nlohmann::json::array(); + node["ports"]["inputs"]["primary"] = "input_sentences"; + node["ports"]["outputs"]["text"] = "llm_answers"; + node["config"]["template"] = "{{primary}}"; + return nlohmann::json::array({node}); +} + +TEST_F(IoBindingRegistryTest, + ResolveFromFileDiagnosticCarrier_T03_T06_T07_T08_AndFileErrors) { + RegisterTestBizBinding(); + + fs::path temp_dir = + fs::temp_directory_path() / "edgeflow_test_resolve_from_file"; + fs::create_directories(temp_dir); + + auto write_file = [](const fs::path& p, const nlohmann::json& content) { + std::ofstream ofs(p); + ofs << content.dump(2); + ofs.flush(); + ofs.close(); + }; + auto write_raw = [](const fs::path& p, const std::string& content) { + std::ofstream ofs(p); + ofs << content; + ofs.flush(); + ofs.close(); + }; + + nlohmann::json base_pipeline = { + {"biz_name", "test_biz_v1"}, + {"deployment", + {{"io", + {{"io_binding", "test_biz.operator.v1"}, + {"output_allocations", + {{"entity_out", + {{"type", "entity_out"}, + {"meta_num", 0}, + {"metadata_type_id", 0}, + {"capacities", {{"entities_json", 2047}}}}}}}}}}}, + {"models", nlohmann::json::array()}, + {"pipeline", ValidTextTemplatePipelineNodes()}}; + + fs::path pipe_path = temp_dir / "pipeline.json"; + fs::path conf_path = temp_dir / "pipeline.conf"; + write_file(pipe_path, base_pipeline); + write_file(conf_path, {{"pipe_path", "pipeline.json"}}); + + std::unique_ptr plan; + std::string err; + DeploymentDiagnostic diag; + + // 1. Success case + int rc = IoBindingResolver::ResolveFromFile(conf_path.string(), "operator", + "", &plan, &err, &diag); + EXPECT_EQ(rc, 0); + EXPECT_NE(plan, nullptr); + EXPECT_TRUE(diag.code.empty()); + + // 2. T03 via file: missing raw model_path with override + { + nlohmann::json t03_pipe = base_pipeline; + t03_pipe["models"] = { + {{"model_id", "mid_1"}, + {"capability", "embedding"}, + {"model_type", "test_biz_embedding"}, + {"backend", "test_tensor_backend"}, + {"model_config", {{"embedding_dim", 128}, {"max_batch_size", 4}}}, + {"backend_config", nlohmann::json::object()}}}; + t03_pipe["deployment"]["model_paths"] = {{"mid_1", "models/override.bin"}}; + write_file(pipe_path, t03_pipe); + + rc = IoBindingResolver::ResolveFromFile(conf_path.string(), "operator", "", + &plan, &err, &diag); + EXPECT_EQ(rc, -3); + EXPECT_EQ(plan, nullptr); + EXPECT_EQ(diag.code, "MISSING_FIELD"); + EXPECT_EQ(diag.path, "/models/0/model_path"); + } + + // 3. T06 via file: unknown override model ID + { + nlohmann::json t06_pipe = base_pipeline; + t06_pipe["deployment"]["model_paths"] = {{"unknown_mid", "models/foo.bin"}}; + write_file(pipe_path, t06_pipe); + + rc = IoBindingResolver::ResolveFromFile(conf_path.string(), "operator", "", + &plan, &err, &diag); + EXPECT_EQ(rc, -2); + EXPECT_EQ(plan, nullptr); + EXPECT_EQ(diag.code, "UNKNOWN_MODEL_ID"); + EXPECT_EQ(diag.path, "/deployment/model_paths/unknown_mid"); + } + + // 4. T07 via file: unknown io_binding + { + nlohmann::json t07_pipe = base_pipeline; + t07_pipe["deployment"]["io"]["io_binding"] = "unregistered.binding"; + write_file(pipe_path, t07_pipe); + + rc = IoBindingResolver::ResolveFromFile(conf_path.string(), "operator", "", + &plan, &err, &diag); + EXPECT_EQ(rc, -2); + EXPECT_EQ(plan, nullptr); + EXPECT_EQ(diag.code, "UNKNOWN_IO_BINDING"); + EXPECT_EQ(diag.path, "/deployment/io/io_binding"); + } + + // 5. T08 via file: missing required output slot + { + nlohmann::json t08_pipe = base_pipeline; + t08_pipe["deployment"]["io"]["output_allocations"].clear(); + write_file(pipe_path, t08_pipe); + + rc = IoBindingResolver::ResolveFromFile(conf_path.string(), "operator", "", + &plan, &err, &diag); + EXPECT_EQ(rc, -2); + EXPECT_EQ(plan, nullptr); + EXPECT_EQ(diag.code, "MISSING_OUTPUT_SLOT"); + EXPECT_EQ(diag.path, "/deployment/io/output_allocations/entity_out"); + } + + // Restore valid pipeline file + write_file(pipe_path, base_pipeline); + + // 6. Non-existent conf file -> CONFIG_FILE_OPEN + rc = IoBindingResolver::ResolveFromFile( + (temp_dir / "nonexistent.conf").string(), "operator", "", &plan, &err, + &diag); + EXPECT_EQ(rc, -2); + EXPECT_EQ(plan, nullptr); + EXPECT_EQ(diag.code, "CONFIG_FILE_OPEN"); + EXPECT_EQ(diag.path, "/"); + + // 7. Malformed JSON in conf file -> JSON_PARSE + { + fs::path bad_conf = temp_dir / "bad_syntax.conf"; + write_raw(bad_conf, "{ unquoted: invalid JSON ..."); + rc = IoBindingResolver::ResolveFromFile(bad_conf.string(), "operator", "", + &plan, &err, &diag); + EXPECT_EQ(rc, -2); + EXPECT_EQ(plan, nullptr); + EXPECT_EQ(diag.code, "JSON_PARSE"); + EXPECT_EQ(diag.path, "/"); + } + + // 8. Missing pipeline file referenced by conf -> DEPLOYMENT_ERROR at + // /pipe_path + { + fs::path missing_pipe_conf = temp_dir / "missing_pipe.conf"; + write_file(missing_pipe_conf, {{"pipe_path", "missing_pipeline.json"}}); + rc = IoBindingResolver::ResolveFromFile(missing_pipe_conf.string(), + "operator", "", &plan, &err, &diag); + EXPECT_EQ(rc, -2); + EXPECT_EQ(plan, nullptr); + EXPECT_EQ(diag.code, "DEPLOYMENT_ERROR"); + EXPECT_EQ(diag.path, "/pipe_path"); + } + + // 9. Malformed JSON in pipeline file -> JSON_PARSE + { + fs::path bad_pipe = temp_dir / "bad_pipe.json"; + write_raw(bad_pipe, "{ bad_pipe_json: invalid"); + + fs::path bad_pipe_conf = temp_dir / "bad_pipe.conf"; + write_file(bad_pipe_conf, {{"pipe_path", "bad_pipe.json"}}); + + rc = IoBindingResolver::ResolveFromFile(bad_pipe_conf.string(), "operator", + "", &plan, &err, &diag); + EXPECT_EQ(rc, -2); + EXPECT_EQ(plan, nullptr); + EXPECT_EQ(diag.code, "JSON_PARSE"); + EXPECT_EQ(diag.path, "/"); + } + + fs::remove_all(temp_dir); +} + } // namespace llm_edgeflow diff --git a/tools/pipeline_studio/README.md b/tools/pipeline_studio/README.md index 2d06aef..5bd38b8 100644 --- a/tools/pipeline_studio/README.md +++ b/tools/pipeline_studio/README.md @@ -138,6 +138,7 @@ C++ 查看工具,展示更多声明信息: [RFC-0057 的操作契约](../../doc/rfcs/0057-pipeline-composition-experience.md#41-所有者与接口)。 手写 ports 后可先运行 `alg_pipeline_tool fix-deps pipeline.json` 预览确定的依赖补充,确认后 加 `--in-place` 写回;歧义、环路、其他校验错误或文件写入失败均不覆盖原文件。 +带 `deployment` 的文档在 `validate`、`plan`、`edit` 与 `fix-deps` 中统一执行部署准备(RFC-0062):原始模型路径必须为合法非空字符串,`deployment.model_paths` 覆盖不得掩盖非法原始声明;`edit` 与 `fix-deps` 同样严格校验未知 I/O 绑定或非法输出分配。 编排或修改 Pipeline 时,应先查询 Catalog 与节点 Definition,再执行 validate 和 plan。完整开发流程参见项目的 `pipeline-composer` skill 与[开发者指南](../../doc/developer_guide.md)。