diff --git a/docs/architecture/local-model-templates.md b/docs/architecture/local-model-templates.md new file mode 100644 index 0000000..4f23409 --- /dev/null +++ b/docs/architecture/local-model-templates.md @@ -0,0 +1,107 @@ +# Console 自定义模型模板 + +## 背景 + +Console #64 要求用户在模型库新增可部署的模型模板。用户提供的实际页面只有静态 JSON 目录，运行 Console 0.1.2；Swiss 返回旧式 `{index, name, ref, source}` 目录和 `{entry, ...}` 模型接口。现有 #73 需要独立可写 catalog-server 和令牌，不能覆盖该环境。本次仅修改 Console。 + +## 目标 + +- 在同一个模型库页面新增模板，保存后与原有模型一起可见，并可进入正常部署流程。 +- Console 自己持久化和发布模板，不要求用户额外搭建可写目录服务。 +- 原有目录和现有 18 个模型保持可访问，已有部署来源关联保持有效。 +- 兼容已部署 Swiss 的 v1 静态目录协议与旧式响应，以及新式 Console/Swiss 响应。 + +### 非目标 + +- 修改 Swiss、model-catalog 或 helm-charts 项目代码，覆盖原始远程 JSON，下载模型权重或安装推理引擎。 +- 创建真实示例模型、自动部署工作负载、v2 LLMService 发布、修改已有模板版本或重写历史。 + +## 架构 + +```text +模型库 → 新增模型模板 → Console 鉴权与校验 + → Console 运行时 ConfigMap 原子保存 + → Console 发布只读 v1 目录 + → 既有 Swiss profile API 注册 console-local + → 模型库显示自定义卡片 → 既有 Swiss 计划/差异/Helm 部署 +``` + +### 组件模块 + +| 组件 | 职责 | 数据归属 | +|---|---|---| +| 模型库与新增侧栏 | 结构化模型信息、镜像/GPU/参数，支持参考现有方案 | 用户表单 | +| Console 模板 API | 权限、模板校验、不可变版本、发布与部分失败处理 | Console | +| 运行时 ConfigMap | 精确版本 YAML、元信息、递增发布 generation | 固定 Console 命名空间 | +| 只读目录入口 | 根据 generation 发布 v1 index 与版本 YAML | Console，GET/HEAD | +| Swiss 既有接口 | 注册目录、读取模板、生成计划和部署 | 既有部署后端 | + +### API 与配置 + +拟议 API： + +| API | 契约 | +|---|---| +| `GET /api/model-templates` | `enabled,catalogName,catalogURL,generation,index,modelGenerations,registeredGeneration,registrationError?`；index 为 v1，未能确定注册状态时 registeredGeneration=null | +| `POST /api/model-templates/check` | `{metadata:{name,displayName?,description?,family?,tags?,source:{hf}},recipeYAML}`；只校验，不保存、不修改 profile；返回 valid/model/version/variantCount | +| `POST /api/model-templates` | 同上；创建新模型。重复名字拒绝 409；成功连接目录返回 201，已保存但连接暂未完成返回 202；响应含 `created,model,version,catalogName,generation,catalogURL,ready,registeredGeneration,reason?` | +| `POST /api/model-templates/publish` | 重试把最新 generation 注册到部署后端，不重复创建模板；返回 ready 与 registeredGeneration | +| `GET/HEAD /catalog/console/g/{generation}/index.json` | 指定发布快照的静态 v1 目录 | +| `GET/HEAD /catalog/console/g/{generation}/models/{name}/{name}-{version}.yaml` | 指定快照内的精确不可变 recipe | + +服务端固定配置 `modelTemplates:{enabled,namespace,configMap,publishURL,backend,catalogName}`。backend 默认 swiss，要求静态 `/api/deploy` 后端；catalogName 默认 console-local。启用时 namespace/configMap/publishURL 必须明确；publishURL 为部署服务可达的 Console 目录根地址，无凭据、查询和 fragment。浏览器不能提供写入命名空间、上游 URL 或发布地址。 + +Console Helm chart 在存在匹配的部署后端时默认启用本地模板，为 namespace/configMap 自动填入本次 Console release 的信息。目录使用独立 `server.catalogAddr`（默认 `:8081`），通过仅 ClusterIP 的 `-catalog` Service 提供，chart 的 `modelTemplates.port` 默认 8081；publishURL 指向该内部 Service。普通 Web/NodePort 对目录路径返回 404。没有部署后端的 Console-only 安装及缺少此节的裸配置保持关闭。chart 创建固定命名空间内的运行时存储 Role，但不渲染或覆盖存储数据 ConfigMap；升级保留用户数据。关闭功能不删除数据。 + +### 持久化与发布 + +- 使用一个运行时 ConfigMap 的 state.json 保存不可变模板记录和 generation，总量最多约 900 KiB，单次输入有界，大小超限明确拒绝。 +- 新名字使用 resourceVersion CAS 保存，冲突有限重试；同名不覆盖。每条记录固定创建 generation；旧快照只包含不晚于该 generation 的记录。 +- 版本 recipe 使用 `catalog.swiss/v1`，包含 name/version/servedName/variants；HF 等模型来源保存在 index 元信息。索引 path 和 digest 必须对应精确保存的 YAML 字节。 +- 保留旧 generation 的目录和版本读取；generation 是本地发布快照编号，不冒充 Git commit。 +- Swiss 对目录 URL 有缓存，因此每次发布使用新的 generation URL，避免保存后仍读取旧目录。 +- 新注册的 console-local 不取代原目录。若 profile 之前依赖服务配置的默认目录，先从既有 cluster/catalog 信息补回原目录项，再追加本地目录，保留默认选择及其余 YAML。 +- 使用既有 profile GET/PUT；旧式 parser 不接受 writable/schemaVersion，因此本地 v1 目录项只写 name/url/default。已有同名外部目录不覆盖。 +- 发布前后核对当前 state/profile，有限重试，避免把更新的 generation 降级；profile API 无 CAS，不能承诺与另一个管理员的站点编辑具备跨系统事务。 +- 模板保存与 profile 注册分离。已保存而未注册时不能报完成或自动部署；页面显示待连接、保留表单结果，并提供重试连接。成功状态需核对 Swiss 已读取对应目录 generation。 + +### 权限与边界 + +- 写 API 沿用 Console 会话和密码重置保护，检查 modeltemplates 的 create/update 权限及所选 backend 的 get/update 权限；读 API 检查 modeltemplates get 和 backend get。 +- UI 权限为 modeltemplates.view/create，管理员既有通配权限可用；UI 可见性不是授权边界。 +- K8s 只访问固定 namespace/configMap；get/update 限定资源名，create 仅在该 namespace。数据由运行时拥有，Helm 不重置用户模板。 +- Swiss 静态 HTTP fetcher 不能附加 Console 登录凭据，因此内部专用目录 listener 提供窄范围匿名只读 GET/HEAD，仅发布到 ClusterIP Service；普通 Web/NodePort listener 不提供这些目录路径。内部 listener 不提供模板写 API、OAuth 或 metrics，不接受任意文件路径和任意 URL 代理。发布的模型元信息/recipe 不适合放置明文凭据，应使用部署 Secret 引用。 +- 校验名字、版本、变体唯一性、sglang/vllm 引擎与 chart、镜像、GPU 需求及 YAML 文档；限制请求、读取、记录和响应大小，设置超时，不跟随上游重定向，不记录模板正文或凭据。 +- v1 模板通过已有 Helm 路径部署；新式 UI 对 v1 的既有 LLMService 限制保持有效。 + +### 前端兼容 + +在 Console 的 inferences 模块增加 API 适配器，统一旧 `{index}`/`{entry}` 与新响应；旧 Swiss 缺少 history 接口时从 v1 versions 读取历史。只调整 Console 模块，复制的 Swiss UI 保持原样。 + +默认模型库把当前默认目录与 Console 自定义模板放在同一列表，卡片保留来源并使用实际 catalog 部署。显式选择目录时按目录过滤；来源选择器以“默认与自定义”标明默认视图。新增后刷新本地、cluster/profile/catalog 查询；未连接的模板仍可见并明确限制部署。 + +## 场景 + +### 接入 + +升级 Console 自己的 chart 和镜像；无需改造外部静态目录。管理员可覆盖 modelTemplates 的存储和发布地址。新配置 Role 验证通过后，模型库出现“新增模型模板”。原目录仍正常返回并显示原模型。 + +### 新增私有模型模板 + +填写模型名、权重来源和部署方案，可参考已有模型的引擎/镜像/GPU/values。点击校验只检查格式；点击保存后 ConfigMap 保存模板，目录注册成功则卡片可进入详情和正常部署表单。创建模板本身不下载权重、不修改推理工作负载。 + +### 连接失败与重试 + +如果已有模板保存成功、部署后端暂时不可达，返回 202 并显示“待连接目录”；页面仍显示新模板，重试连接不重复创建。并发同名创建返回 409，存储失败不报告保存成功。原目录来源和既有模型不因本地目录失败而消失。 + +## 验证 + +实现需覆盖：旧/新响应适配，参考模板填充与本地卡片来源，校验/创建/部分失败/重试，权限和固定存储边界，CAS/重复创建/数据大小，generation 快照与精确 digest，profile 原目录保留和旧 parser 可接受字段，旧与新 Swiss 读取兼容。真实用户目录只读检查，不写入虚构模型；实际部署验收另行明确记录。 + +已使用旧版 Swiss `dfdc4be` 命令行实际读取 `State.Index/Recipe` 发布的 HTTP 目录，并生成 `plan.swiss/v1` 计划；没有执行 install/apply。可在 Console 仓库根目录重现： + +```sh +SWISS_CATALOG_COMPAT_BIN=/path/to/swiss go test ./internal/modeltemplate -run TestPublishedTemplateComposesWithLegacySwiss -v +``` + +未配置外部工具时该补充测试跳过；常规 HTTP、存储和权限测试仍运行。 diff --git a/docs/console-design.md b/docs/console-design.md index 362719e..16d798d 100644 --- a/docs/console-design.md +++ b/docs/console-design.md @@ -337,6 +337,26 @@ DeploySheet -> GET /api/llm/route-options[?route=] | Text | Per-key locale translations where present; new options retain engine text | | Ownership | Inferences adapts the Swiss metadata without editing the copied upstream module | +### Console-managed model templates + +Native template creation covers static/read-only catalog installations (console #64; [design](architecture/local-model-templates.md), #84). The writable external catalog editor remains available separately. + +```text +Model Library -> authenticated template API -> runtime ConfigMap CAS + -> internal-only snapshot catalog -> existing Swiss profile registration + -> original + custom cards -> existing Helm plan/diff/deploy +``` + +| Boundary | Decision | +|---|---| +| Ownership | Console owns persistence, publication and UI; other projects' code and original catalog files stay untouched | +| Storage | Fixed namespace/ConfigMap, bounded state, immutable model names/recipes and generation snapshots; Helm does not replace runtime data | +| Publication | Anonymous GET/HEAD on a dedicated ClusterIP-only catalog listener; web/NodePort returns 404 for catalog paths | +| Registration | Existing Swiss profile API, preserving upstream catalogs, implicit single-catalog default and other YAML; generation URL avoids stale index cache | +| Partial success | 201 means saved and readable; 202 means saved/pending with a retry action and disabled deployment | +| Compatibility | Console adapter handles old index/entry envelopes and v1 history as well as current wire forms; mirrored Swiss UI unchanged | +| Verification | Existing Swiss CLI can read published index/recipe bytes and compose a real plan without applying workloads | + ### Workload reads (`/api/k8s/...`) Events and logs are Kubernetes data, not swiss's, so console serves them itself diff --git a/examples/console.yaml b/examples/console.yaml index 9620d60..20a3128 100644 --- a/examples/console.yaml +++ b/examples/console.yaml @@ -9,6 +9,8 @@ server: addr: ":8080" # Prometheus metrics, on a port of their own. "off" disables them. metricsAddr: ":9090" + # Separate internal reader, defaults to :8081 only with modelTemplates enabled. + # catalogAddr: ":8081" auth: # HS256 signing key. In a cluster this comes from a Secret via env; never # commit a real one. Shared with Rise Global on upgrade. @@ -42,6 +44,17 @@ backends: gateway: profile: llm/site-profile +# Bare configuration keeps templates disabled; the Helm chart enables them when +# Swiss is connected and supplies these fixed storage/publication defaults. +# Publish only nonsecret recipes; Swiss reads this subtree without credentials. +modelTemplates: + enabled: false + namespace: console + configMap: console-model-templates + publishURL: http://console-catalog.console.svc:8081/catalog/console + backend: swiss + catalogName: console-local + # The router: programs call models at /v1 with API keys admins issue on # the router's API keys page. console keeps their hashes in this Secret (creating # it) and forwards /v1 to the backend named here with that backend's own diff --git a/helm/console/README.md b/helm/console/README.md index 7bbea9a..1957202 100644 --- a/helm/console/README.md +++ b/helm/console/README.md @@ -11,7 +11,7 @@ ## What it does not do -- **It does not install or modify an existing Swiss or inference gateway** — it only reads their ConfigMaps and Secrets. +- **It does not install or replace an existing Swiss or inference gateway.** The connection reads their ConfigMaps and Secrets; local-template publication registers its catalog through Swiss's existing profile API. - **It does not need `install.sh`** — that is an optional helper in the source repository (discovers Swiss and the gateway, generates values, runs an end-to-end check after install). It is not in the chart package and this guide does not use it. See `./install.sh --help`. ## Quick start @@ -131,6 +131,68 @@ helm upgrade --install console oci://ghcr.io/modelsphere/charts/console \ --wait --timeout 10m ``` +## Custom model templates + +Swiss-connected chart installations enable Console's local model templates by default. Console stores and publishes the catalog itself; no writable catalog-server or catalog token is needed. Console-only installations keep the feature effectively off until the selected backend is a static `/api/deploy` backend. Set `modelTemplates.enabled=false` to disable it explicitly; bare Console configuration without this section is disabled. + +```yaml +modelTemplates: + enabled: true + backend: swiss + catalogName: console-local + port: 8081 # dedicated internal catalog listener and Service + namespace: "" # release namespace + configMap: "" # -model-templates + publishURL: "" # http://-catalog..svc:8081/catalog/console +``` + +| Boundary | Behavior | +|---|---| +| Storage | one runtime-owned ConfigMap, `state.json`, at most 900 KiB; up to 1,000 immutable model names | +| Kubernetes grants | ConfigMap create in the configured namespace; get/update restricted to the configured resource name | +| Chart upgrades | do not render, reset, or delete the runtime data ConfigMap | +| Internal reads | dedicated catalog listener and ClusterIP-only Service; GET/HEAD snapshots and exact recipe paths; no native/write APIs | +| Web listener | `/catalog/console` and all descendants return 404, including anonymous requests; NodePort/LoadBalancer web Services expose only the web port | +| Input | at most 192 KiB JSON and 128 KiB recipe YAML; one strict v1 recipe, validated names, exact chart version and GPU requirements | +| Deployment | normal Swiss Helm plan/diff/deploy after registration; saving a template does not download weights or change workloads | + +`requires.gpus` is a positive safe integer per pod; it is not capped at eight. Multi-node recipes retain `nodes`, vendor, topology and RDMA metadata; LWS requires `values.lws.size` to match `requires.nodes`. Optional `image.digest` accepts a SHA-256 pin alongside repository/tag. Chart versions must be exact pins, so a reference recipe using a version range needs a selected version before saving. + +With `rbac.create=false`, supply the storage grants through your own namespace-scoped Role. ConfigMap creation cannot be restricted by resourceName in Kubernetes; reads and updates are restricted. The default publication URL uses the separate `-catalog` ClusterIP Service, which is never added to the public NodePort/LoadBalancer Service. Swiss's static fetcher cannot send Console credentials, so this internal reader accepts anonymous GET/HEAD. Keep that listener internal and use Secret references instead of plaintext credentials in recipes. + +In bare configuration, `server.catalogAddr` defaults to `:8081` only while templates are enabled; otherwise the listener is off. An enabled provider requires a valid dedicated port distinct from web and metrics ports. The chart sets it from `modelTemplates.port`. Custom `publishURL` values must reach the internal listener, rather than the web endpoint. + +Console permissions are independent of Kubernetes grants. Bind a platform-scoped IAMRole with the following rules alongside existing model/deployment access: + +```yaml +spec: + uiPermissions: [modeltemplates.view, modeltemplates.create] + rules: + - apiGroups: [iam.theriseunion.io] + resources: [modeltemplates] + resourceNames: [console-local] + verbs: [get, create, update] + - apiGroups: [iam.theriseunion.io] + resources: [backends] + resourceNames: [swiss] + verbs: [get, update] +``` + +`create` authorizes validation and creation; `update` authorizes retrying catalog publication. Backend get/update is checked before a new template is saved. UI permission alone does not authorize APIs. + +```text +create form -> validate -> save immutable model -> publish generation URL + -> register through Swiss profile API + -> verify Swiss reads the snapshot -> deploy form +``` + +- The existing default catalog is preserved when the profile previously relied on Swiss's configured fallback. Registration appends or updates only the Console catalog; a foreign catalog using the same name blocks registration. +- Profile registration uses only name, URL and default fields for the local entry, compatible with legacy Swiss. It preserves the other YAML settings and comments. Swiss must expose a writable ConfigMap-backed profile API; this does not require enabling workload deployment. +- A saved template returns HTTP 201 only after Swiss reads its generation. HTTP 202 means saved but waiting for connection; the model remains listed, and publication can be retried without creating it again. +- Every publication URL includes its generation to avoid Swiss's index cache. Old snapshots and exact recipe bytes remain readable. Existing model names cannot be replaced or deleted through these APIs. +- Registration re-reads profile and storage state with bounded retries. The existing Swiss profile API has no compare-and-swap operation, so this cannot be a transaction with concurrent administrator edits. Avoid simultaneous site-profile changes while registering. +- This provider publishes schema v1 for the existing Helm deployment path. It does not enable schema v2 LLMService deployment. + ## Check and sign in ```bash diff --git a/helm/console/templates/_helpers.tpl b/helm/console/templates/_helpers.tpl index 8c19f12..8089ae7 100644 --- a/helm/console/templates/_helpers.tpl +++ b/helm/console/templates/_helpers.tpl @@ -89,3 +89,30 @@ app.kubernetes.io/instance: {{ .Release.Name }} {{ include "console.fullname" . }}-api-keys {{- end -}} +{{- define "console.modelTemplates.enabled" -}} +{{- if .Values.modelTemplates.enabled -}} +{{- $backend := .Values.modelTemplates.backend | default "swiss" -}} +{{- $found := and (eq $backend "swiss") (not (empty (include "console.swiss.backend" .))) -}} +{{- range .Values.backends -}} +{{- if and (eq .name $backend) (eq .prefix "/api/deploy") .url (not .gateway) -}}{{- $found = true -}}{{- end -}} +{{- end -}} +{{- if $found -}}true{{- end -}} +{{- end -}} +{{- end -}} + +{{- define "console.modelTemplates.namespace" -}} +{{- .Values.modelTemplates.namespace | default .Release.Namespace -}} +{{- end -}} + +{{- define "console.modelTemplates.configMap" -}} +{{- .Values.modelTemplates.configMap | default (printf "%s-model-templates" (include "console.fullname" .)) -}} +{{- end -}} + +{{- define "console.modelTemplates.service" -}} +{{- printf "%s-catalog" (include "console.fullname" . | trunc 55 | trimSuffix "-") -}} +{{- end -}} + +{{- define "console.modelTemplates.publishURL" -}} +{{- .Values.modelTemplates.publishURL | default (printf "http://%s.%s.svc:%d/catalog/console" (include "console.modelTemplates.service" .) .Release.Namespace (int .Values.modelTemplates.port)) -}} +{{- end -}} + diff --git a/helm/console/templates/catalog-service.yaml b/helm/console/templates/catalog-service.yaml new file mode 100644 index 0000000..a7034f6 --- /dev/null +++ b/helm/console/templates/catalog-service.yaml @@ -0,0 +1,18 @@ +{{- if include "console.modelTemplates.enabled" . }} +apiVersion: v1 +kind: Service +metadata: + name: {{ include "console.modelTemplates.service" . }} + labels: + {{- include "console.labels" . | nindent 4 }} + app.kubernetes.io/component: catalog +spec: + type: ClusterIP + selector: + {{- include "console.selectorLabels" . | nindent 4 }} + ports: + - name: catalog + port: {{ .Values.modelTemplates.port }} + targetPort: catalog + protocol: TCP +{{- end }} diff --git a/helm/console/templates/configmap.yaml b/helm/console/templates/configmap.yaml index f6fccd8..9392e2a 100644 --- a/helm/console/templates/configmap.yaml +++ b/helm/console/templates/configmap.yaml @@ -9,6 +9,7 @@ data: server: addr: ":8080" metricsAddr: {{ printf ":%d" (int .Values.metrics.port) | quote }} + catalogAddr: {{ if include "console.modelTemplates.enabled" . }}{{ printf ":%d" (int .Values.modelTemplates.port) | quote }}{{ else }}"off"{{ end }} auth: {{- if .Values.auth.disabled }} disabled: true @@ -16,6 +17,13 @@ data: issuer: {{ .Values.auth.issuer | quote }} tokenTTL: {{ .Values.auth.tokenTTL | quote }} # jwtSecret is injected from the Secret via env, not rendered here. + modelTemplates: + enabled: {{ if include "console.modelTemplates.enabled" . }}true{{ else }}false{{ end }} + namespace: {{ include "console.modelTemplates.namespace" . | quote }} + configMap: {{ include "console.modelTemplates.configMap" . | quote }} + publishURL: {{ include "console.modelTemplates.publishURL" . | quote }} + backend: {{ .Values.modelTemplates.backend | quote }} + catalogName: {{ .Values.modelTemplates.catalogName | quote }} backends: {{- range .Values.backends }} - name: {{ .name | quote }} diff --git a/helm/console/templates/deployment.yaml b/helm/console/templates/deployment.yaml index cf97498..72e0f6e 100644 --- a/helm/console/templates/deployment.yaml +++ b/helm/console/templates/deployment.yaml @@ -69,6 +69,10 @@ spec: containerPort: 8080 - name: metrics containerPort: {{ .Values.metrics.port }} + {{- if include "console.modelTemplates.enabled" . }} + - name: catalog + containerPort: {{ .Values.modelTemplates.port }} + {{- end }} readinessProbe: httpGet: { path: /readyz, port: http } livenessProbe: diff --git a/helm/console/templates/model-templates-rbac.yaml b/helm/console/templates/model-templates-rbac.yaml new file mode 100644 index 0000000..a109dce --- /dev/null +++ b/helm/console/templates/model-templates-rbac.yaml @@ -0,0 +1,33 @@ +{{- if and .Values.rbac.create (include "console.modelTemplates.enabled" .) }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ include "console.fullname" . }}-model-templates + namespace: {{ include "console.modelTemplates.namespace" . | quote }} + labels: + {{- include "console.labels" . | nindent 4 }} +rules: + - apiGroups: [""] + resources: ["configmaps"] + verbs: ["create"] + - apiGroups: [""] + resources: ["configmaps"] + resourceNames: [{{ include "console.modelTemplates.configMap" . | quote }}] + verbs: ["get", "update"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ include "console.fullname" . }}-model-templates + namespace: {{ include "console.modelTemplates.namespace" . | quote }} + labels: + {{- include "console.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ include "console.fullname" . }}-model-templates +subjects: + - kind: ServiceAccount + name: {{ include "console.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/helm/console/values.yaml b/helm/console/values.yaml index df397f1..392c1d8 100644 --- a/helm/console/values.yaml +++ b/helm/console/values.yaml @@ -71,6 +71,22 @@ admin: # swissd (model deployment) is not listed here: see swiss and externalSwiss. backends: [] +# Console owns immutable custom model templates and publishes a v1 catalog. +# Effective only when backend names a configured static /api/deploy service; +# Console-only installations stay valid without Swiss. No external catalog-server +# or write token is needed. Runtime data is retained across chart upgrades. +modelTemplates: + enabled: true + backend: swiss + catalogName: console-local + # Dedicated internal listener; never added to the public UI Service. + port: 8081 + # Empty values use this release namespace, -model-templates and the + # separate -catalog ClusterIP Service's /catalog/console URL. + namespace: "" + configMap: "" + publishURL: "" + # swissd, the model deployment backend. Pick one: # # swiss.enabled: true install it with this release, from the swiss chart diff --git a/internal/config/catalog_listener_chart_test.go b/internal/config/catalog_listener_chart_test.go new file mode 100644 index 0000000..590f39a --- /dev/null +++ b/internal/config/catalog_listener_chart_test.go @@ -0,0 +1,101 @@ +package config + +import ( + "io" + "strings" + "testing" + + "gopkg.in/yaml.v3" +) + +func TestHelmCatalogHasOnlyAnInternalService(t *testing.T) { + for _, serviceType := range []string{"NodePort", "LoadBalancer"} { + args := []string{"--set", "externalSwiss.url=http://swiss.swiss.svc/api", "--set", "service.type=" + serviceType} + out, err := helmTemplate(t, args...) + if err != nil { + t.Fatalf("render %v %s", err, out) + } + dec := yaml.NewDecoder(strings.NewReader(out)) + catalogs, main, catalogPorts := 0, 0, 0 + for { + var doc struct { + Kind string `yaml:"kind"` + Metadata struct { + Name string `yaml:"name"` + } `yaml:"metadata"` + Spec struct { + Type string `yaml:"type"` + Ports []struct { + Name string `yaml:"name"` + Port int `yaml:"port"` + TargetPort string `yaml:"targetPort"` + NodePort int `yaml:"nodePort"` + } `yaml:"ports"` + Template struct { + Spec struct { + Containers []struct { + Ports []struct { + Name string `yaml:"name"` + ContainerPort int `yaml:"containerPort"` + } `yaml:"ports"` + } `yaml:"containers"` + } `yaml:"spec"` + } `yaml:"template"` + } `yaml:"spec"` + } + err := dec.Decode(&doc) + if err == io.EOF { + break + } + if err != nil { + t.Fatal(err) + } + if doc.Kind == "Service" { + if doc.Metadata.Name == "console-console-catalog" { + catalogs++ + if doc.Spec.Type != "ClusterIP" || len(doc.Spec.Ports) != 1 || doc.Spec.Ports[0].Port != 8081 || doc.Spec.Ports[0].TargetPort != "catalog" || doc.Spec.Ports[0].NodePort != 0 { + t.Fatalf("catalog service exposed externally: %+v", doc) + } + } else { + for _, port := range doc.Spec.Ports { + if port.Name == "catalog" || port.TargetPort == "catalog" { + t.Fatalf("catalog leaked into service %s", doc.Metadata.Name) + } + } + } + if doc.Metadata.Name == "console-console" { + main++ + if doc.Spec.Type != serviceType || len(doc.Spec.Ports) != 1 || doc.Spec.Ports[0].TargetPort != "http" { + t.Fatalf("main service changed: %+v", doc) + } + } + } + if doc.Kind == "Deployment" && doc.Metadata.Name == "console-console" { + for _, container := range doc.Spec.Template.Spec.Containers { + for _, port := range container.Ports { + if port.Name == "catalog" { + catalogPorts++ + if port.ContainerPort != 8081 { + t.Fatalf("catalog container port=%d", port.ContainerPort) + } + } + } + } + } + } + if catalogs != 1 || main != 1 || catalogPorts != 1 { + t.Fatalf("services catalog=%d main=%d container=%d", catalogs, main, catalogPorts) + } + } + off, err := helmTemplate(t, "--set", "externalSwiss.url=http://swiss.swiss.svc/api", "--set", "modelTemplates.enabled=false") + if err != nil { + t.Fatalf("render %v %s", err, off) + } + if strings.Contains(off, "name: console-console-catalog") || strings.Contains(off, "name: catalog") { + t.Fatal("disabled templates exposed an internal listener/service") + } + c := renderChartConfig(t, "--set", "externalSwiss.url=http://swiss.swiss.svc/api", "--set", "modelTemplates.port=8082") + if c.Server.CatalogAddr != ":8082" || c.ModelTemplates.PublishURL != "http://console-console-catalog.modelsphere.svc:8082/catalog/console" { + t.Fatalf("custom catalog port mismatch=%+v %+v", c.Server, c.ModelTemplates) + } +} diff --git a/internal/config/config.go b/internal/config/config.go index a6ea990..7f2f189 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -20,10 +20,11 @@ import ( ) type Config struct { - Cluster Cluster `yaml:"cluster,omitempty"` - Server Server `yaml:"server,omitempty"` - Backends []Backend `yaml:"backends,omitempty"` - Router Router `yaml:"router,omitempty"` + Cluster Cluster `yaml:"cluster,omitempty"` + Server Server `yaml:"server,omitempty"` + Backends []Backend `yaml:"backends,omitempty"` + Router Router `yaml:"router,omitempty"` + ModelTemplates ModelTemplates `yaml:"modelTemplates,omitempty"` Origin string `yaml:"-"` } @@ -60,6 +61,7 @@ type Server struct { // MetricsAddr serves Prometheus metrics, apart from Addr so they are not // reachable wherever the UI is. Default ":9090"; "off" disables it. MetricsAddr string `yaml:"metricsAddr,omitempty"` + CatalogAddr string `yaml:"catalogAddr,omitempty"` Auth Auth `yaml:"auth,omitempty"` } @@ -197,6 +199,7 @@ func Find(explicit string) string { } func (c *Config) applyDefaults() { + c.ModelTemplates = c.ModelTemplates.WithDefaults() if c.Server.Addr == "" { c.Server.Addr = ":8080" } @@ -209,6 +212,9 @@ func (c *Config) applyDefaults() { if c.Server.MetricsAddr == "" { c.Server.MetricsAddr = DefaultMetricsAddr } + if c.Server.CatalogAddr == "" { + c.Server.CatalogAddr = c.CatalogListenAddr() + } if c.Router.Backend == "" { c.Router.Backend = DefaultRouterBackend } @@ -252,11 +258,11 @@ func (c *Config) Validate() error { return fmt.Errorf("%s: router.maxBodyBytes must not be negative", c.origin()) } } - return nil + return c.validateModelTemplates() } // reservedPrefixes are console's own API; a backend there would shadow them. -var reservedPrefixes = []string{"/api/iam", "/api/me", "/api/router"} +var reservedPrefixes = []string{"/api/iam", "/api/me", "/api/router", "/api/model-templates"} func validateBackend(b Backend) error { // Only /api/* passes the auth middleware's guard; anything else is served diff --git a/internal/config/model_templates.go b/internal/config/model_templates.go new file mode 100644 index 0000000..e198557 --- /dev/null +++ b/internal/config/model_templates.go @@ -0,0 +1,100 @@ +package config + +import ( + "fmt" + "net" + "net/url" + "regexp" + "strconv" + "strings" + + "k8s.io/apimachinery/pkg/util/validation" +) + +var catalogNameRE = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]*$`) + +type ModelTemplates struct { + Enabled bool `yaml:"enabled,omitempty"` + Namespace string `yaml:"namespace,omitempty"` + ConfigMap string `yaml:"configMap,omitempty"` + PublishURL string `yaml:"publishURL,omitempty"` + Backend string `yaml:"backend,omitempty"` + CatalogName string `yaml:"catalogName,omitempty"` +} + +func (m ModelTemplates) WithDefaults() ModelTemplates { + if m.Backend == "" { + m.Backend = "swiss" + } + if m.CatalogName == "" { + m.CatalogName = "console-local" + } + m.PublishURL = strings.TrimSuffix(m.PublishURL, "/") + return m +} + +const DefaultCatalogAddr = ":8081" + +func (c *Config) CatalogListenAddr() string { + if !c.ModelTemplates.Enabled { + return "off" + } + if c.Server.CatalogAddr == "" { + return DefaultCatalogAddr + } + return c.Server.CatalogAddr +} + +func (c *Config) validateCatalogListener() error { + addr := c.CatalogListenAddr() + _, port, err := net.SplitHostPort(addr) + number, parseErr := strconv.Atoi(port) + if err != nil || parseErr != nil || number < 1 || number > 65535 { + return fmt.Errorf("%s: enabled modelTemplates requires server.catalogAddr with a valid dedicated port", c.origin()) + } + web := c.Server.Addr + if web == "" { + web = ":8080" + } + metrics := c.Server.MetricsAddr + if metrics == "" { + metrics = DefaultMetricsAddr + } + for _, other := range []string{web, metrics} { + if other == "off" { + continue + } + _, otherPort, err := net.SplitHostPort(other) + otherNumber, otherErr := strconv.Atoi(otherPort) + if err == nil && otherErr == nil && number == otherNumber { + return fmt.Errorf("%s: server.catalogAddr must use a different port from web and metrics listeners", c.origin()) + } + } + return nil +} + +func (c *Config) validateModelTemplates() error { + m := c.ModelTemplates.WithDefaults() + if !m.Enabled { + return nil + } + if err := c.validateCatalogListener(); err != nil { + return err + } + if m.Namespace == "" || len(validation.IsDNS1123Label(m.Namespace)) != 0 || m.ConfigMap == "" || len(validation.IsDNS1123Subdomain(m.ConfigMap)) != 0 { + return fmt.Errorf("%s: modelTemplates needs an explicit valid namespace and configMap", c.origin()) + } + if m.CatalogName == "" || len(m.CatalogName) > 63 || !catalogNameRE.MatchString(m.CatalogName) { + return fmt.Errorf("%s: modelTemplates.catalogName is invalid", c.origin()) + } + u, err := url.Parse(m.PublishURL) + if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" || u.User != nil || u.RawQuery != "" || u.ForceQuery || u.Fragment != "" || u.Path != "/catalog/console" || u.RawPath != "" { + return fmt.Errorf("%s: modelTemplates.publishURL must be an absolute http(s) Console /catalog/console URL without credentials, query or fragment", c.origin()) + } + for _, b := range c.Backends { + if b.Name == m.Backend && b.Gateway == nil && b.URL != "" && b.Prefix == "/api/deploy" { + return nil + } + } + return fmt.Errorf("%s: modelTemplates.backend must name a static /api/deploy Swiss backend", c.origin()) +} diff --git a/internal/config/model_templates_test.go b/internal/config/model_templates_test.go new file mode 100644 index 0000000..8ed7572 --- /dev/null +++ b/internal/config/model_templates_test.go @@ -0,0 +1,142 @@ +package config + +import ( + "io" + "strings" + "testing" + + "gopkg.in/yaml.v3" +) + +func templateConfig() *Config { + return &Config{Server: Server{Auth: Auth{Disabled: true}}, Backends: []Backend{{Name: "swiss", Prefix: "/api/deploy", URL: "http://swiss.models.svc/api"}}, ModelTemplates: ModelTemplates{Enabled: true, Namespace: "console-ns", ConfigMap: "templates", PublishURL: "http://console-catalog.console-ns.svc:8081/catalog/console"}} +} +func TestModelTemplateConfiguration(t *testing.T) { + bare := &Config{Server: Server{Auth: Auth{Disabled: true}}} + bare.applyDefaults() + if bare.ModelTemplates.Enabled || bare.ModelTemplates.Backend != "swiss" || bare.ModelTemplates.CatalogName != "console-local" { + t.Fatalf("defaults=%+v", bare.ModelTemplates) + } + if err := bare.Validate(); err != nil { + t.Fatal(err) + } + if bare.Server.CatalogAddr != "off" || bare.CatalogListenAddr() != "off" { + t.Fatal("bare configuration starts a catalog listener") + } + enabled := templateConfig() + enabled.applyDefaults() + if enabled.Server.CatalogAddr != DefaultCatalogAddr { + t.Fatalf("catalog default=%s", enabled.Server.CatalogAddr) + } + if err := templateConfig().Validate(); err != nil { + t.Fatal(err) + } + tests := map[string]func(*Config){ + "listener off": func(c *Config) { c.Server.CatalogAddr = "off" }, "listener invalid": func(c *Config) { c.Server.CatalogAddr = "8081" }, "listener zero": func(c *Config) { c.Server.CatalogAddr = ":0" }, "web conflict": func(c *Config) { c.Server.CatalogAddr = "127.0.0.1:8080" }, "metrics conflict": func(c *Config) { c.Server.CatalogAddr = ":9090" }, + "namespace missing": func(c *Config) { c.ModelTemplates.Namespace = "" }, "namespace unsafe": func(c *Config) { c.ModelTemplates.Namespace = "*" }, "configmap missing": func(c *Config) { c.ModelTemplates.ConfigMap = "" }, + "userinfo": func(c *Config) { c.ModelTemplates.PublishURL = "http://secret:password@console/catalog/console" }, "query": func(c *Config) { c.ModelTemplates.PublishURL += "?key=value" }, "fragment": func(c *Config) { c.ModelTemplates.PublishURL += "#fragment" }, "path": func(c *Config) { c.ModelTemplates.PublishURL = "http://console/other" }, + "catalog name": func(c *Config) { c.ModelTemplates.CatalogName = "../escape" }, "missing backend": func(c *Config) { c.Backends = nil }, "wrong module": func(c *Config) { c.Backends[0].Prefix = "/api/other" }, "reserved route": func(c *Config) { c.Backends[0].Prefix = "/api/model-templates" }, + } + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + c := templateConfig() + mutate(c) + if err := c.Validate(); err == nil { + t.Fatal("invalid configuration accepted") + } + }) + } +} +func TestHelmModelTemplateDefaultsAndScopedStorage(t *testing.T) { + bare := renderChartConfig(t) + if bare.ModelTemplates.Enabled || bare.Server.CatalogAddr != "off" { + t.Fatal("Console-only install enabled templates without Swiss") + } + args := []string{"--set", "externalSwiss.url=http://swiss.swiss.svc/api"} + c := renderChartConfig(t, args...) + c.Server.Auth.JWTSecret = "test" + if err := c.Validate(); err != nil { + t.Fatal(err) + } + if c.Server.CatalogAddr != ":8081" { + t.Fatalf("catalog listener=%s", c.Server.CatalogAddr) + } + if !c.ModelTemplates.Enabled || c.ModelTemplates.Namespace != "modelsphere" || c.ModelTemplates.ConfigMap != "console-console-model-templates" || c.ModelTemplates.PublishURL != "http://console-console-catalog.modelsphere.svc:8081/catalog/console" { + t.Fatalf("templates=%+v", c.ModelTemplates) + } + out, err := helmTemplate(t, args...) + if err != nil { + t.Fatalf("render %v %s", err, out) + } + dec := yaml.NewDecoder(strings.NewReader(out)) + roles, bindings := 0, 0 + for { + var doc struct { + Kind string `yaml:"kind"` + Metadata struct{ Name, Namespace string } `yaml:"metadata"` + Rules []struct { + APIGroups []string `yaml:"apiGroups"` + Resources []string `yaml:"resources"` + ResourceNames []string `yaml:"resourceNames"` + Verbs []string `yaml:"verbs"` + } `yaml:"rules"` + Data map[string]string `yaml:"data"` + } + err := dec.Decode(&doc) + if err == io.EOF { + break + } + if err != nil { + t.Fatal(err) + } + if doc.Kind == "ConfigMap" && doc.Data["state.json"] != "" { + t.Fatal("chart owns mutable runtime catalog data") + } + if doc.Metadata.Name != "console-console-model-templates" { + continue + } + if doc.Metadata.Namespace != "modelsphere" { + t.Fatalf("namespace=%s", doc.Metadata.Namespace) + } + switch doc.Kind { + case "Role": + roles++ + if len(doc.Rules) != 2 { + t.Fatalf("rules=%v", doc.Rules) + } + for _, rule := range doc.Rules { + if strings.Join(rule.Resources, ",") != "configmaps" || strings.Join(rule.APIGroups, ",") != "" { + t.Fatalf("unexpected rule=%v", rule) + } + verbs := strings.Join(rule.Verbs, ",") + if verbs == "get,update" { + if strings.Join(rule.ResourceNames, ",") != "console-console-model-templates" { + t.Fatal("unscoped storage reads/updates") + } + } else if verbs != "create" { + t.Fatalf("unsafe storage verbs=%s", verbs) + } + } + case "RoleBinding": + bindings++ + default: + t.Fatalf("unexpected mutable storage object kind=%s", doc.Kind) + } + } + if roles != 1 || bindings != 1 { + t.Fatalf("roles=%d bindings=%d", roles, bindings) + } + for _, extra := range [][]string{{"--set", "modelTemplates.enabled=false"}, {"--set", "rbac.create=false"}} { + out, err := helmTemplate(t, append(append([]string{}, args...), extra...)...) + if err != nil { + t.Fatalf("render %v %s", err, out) + } + if strings.Contains(out, "name: console-console-model-templates\n") { + t.Fatal("storage Role created with feature/custom RBAC disabled") + } + } + c = renderChartConfig(t, append(args, "--set", "modelTemplates.namespace=custom", "--set", "modelTemplates.configMap=custom-state", "--set", "modelTemplates.publishURL=http://console.internal/catalog/console")...) + if c.ModelTemplates.Namespace != "custom" || c.ModelTemplates.ConfigMap != "custom-state" || c.ModelTemplates.PublishURL != "http://console.internal/catalog/console" { + t.Fatalf("overrides=%+v", c.ModelTemplates) + } +} diff --git a/internal/modeltemplate/legacy_cli_test.go b/internal/modeltemplate/legacy_cli_test.go new file mode 100644 index 0000000..137e6bb --- /dev/null +++ b/internal/modeltemplate/legacy_cli_test.go @@ -0,0 +1,83 @@ +package modeltemplate + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "testing" + "time" + + "gopkg.in/yaml.v3" +) + +func TestPublishedTemplateComposesWithLegacySwiss(t *testing.T) { + binary := os.Getenv("SWISS_CATALOG_COMPAT_BIN") + if binary == "" { + t.Skip("set SWISS_CATALOG_COMPAT_BIN to verify an existing Swiss reader") + } + in := testInput("custom") + if inputFile := os.Getenv("SWISS_CATALOG_COMPAT_INPUT"); inputFile != "" { + raw, err := os.ReadFile(inputFile) + if err != nil || json.Unmarshal(raw, &in) != nil { + t.Fatal("cannot read the generated template input") + } + } + recipe, err := Validate(in) + if err != nil { + t.Fatal(err) + } + name := in.Metadata.Name + variant := recipe.Variants[0] + state := Empty() + if err := state.Add(in, "operator", "now"); err != nil { + t.Fatal(err) + } + index, err := state.Index(1) + if err != nil { + t.Fatal(err) + } + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/catalog/console/g/1/index.json": + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(index) + case "/catalog/console/g/1/models/" + name + "/" + name + "-" + recipe.Version + ".yaml": + body, _ := state.Recipe(1, name, name+"-"+recipe.Version+".yaml") + w.Header().Set("Content-Type", "application/yaml") + _, _ = w.Write(body) + default: + http.NotFound(w, r) + } + })) + defer server.Close() + directory := t.TempDir() + profile := filepath.Join(directory, "profile.yaml") + if err := os.WriteFile(profile, []byte("name: compatibility\nnamespace: models\nchartRepo: https://charts.example\nmodel:\n pathTemplate: /models/{{hf}}\n"), 0600); err != nil { + t.Fatal(err) + } + planFile := filepath.Join(directory, "plan.yaml") + ctx, cancel := context.WithTimeout(t.Context(), 20*time.Second) + defer cancel() + command := exec.CommandContext(ctx, binary, "--catalog", server.URL+"/catalog/console/g/1/index.json", "--profile", profile, "plan", "--model", name, "--variant", variant.ID, "--output", planFile) + command.Dir = directory + if output, err := command.CombinedOutput(); err != nil { + t.Fatalf("existing Swiss reader rejected the published template: %v\n%s", err, output) + } + raw, err := os.ReadFile(planFile) + if err != nil { + t.Fatal(err) + } + var plan map[string]any + if err := yaml.Unmarshal(raw, &plan); err != nil { + t.Fatal(err) + } + source, _ := plan["source"].(map[string]any) + chart, _ := plan["chart"].(map[string]any) + if plan["apiVersion"] != "plan.swiss/v1" || source["model"] != name || source["variant"] != variant.ID || chart["name"] != variant.Chart.Name || chart["version"] != variant.Chart.Version { + t.Fatalf("unexpected composed identity: source=%v chart=%v", source, chart) + } +} diff --git a/internal/modeltemplate/store.go b/internal/modeltemplate/store.go new file mode 100644 index 0000000..815ec20 --- /dev/null +++ b/internal/modeltemplate/store.go @@ -0,0 +1,89 @@ +package modeltemplate + +import ( + "context" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/dynamic" +) + +var configMaps = schema.GroupVersionResource{Version: "v1", Resource: "configmaps"} + +const ownerAnnotation = "console.modelsphere.io/model-templates" + +type Store struct { + client dynamic.ResourceInterface + name string +} + +func NewStore(client dynamic.Interface, namespace, name string) *Store { + return &Store{client: client.Resource(configMaps).Namespace(namespace), name: name} +} + +func (s *Store) read(ctx context.Context) (*State, *unstructured.Unstructured, error) { + obj, err := s.client.Get(ctx, s.name, metav1.GetOptions{}) + if apierrors.IsNotFound(err) { + return Empty(), nil, nil + } + if err != nil { + return nil, nil, err + } + if obj.GetLabels()["app.kubernetes.io/managed-by"] != "console" || obj.GetAnnotations()[ownerAnnotation] != "true" { + return nil, nil, ErrInvalidState + } + raw, ok, err := unstructured.NestedString(obj.Object, "data", "state.json") + if err != nil || !ok { + return nil, nil, ErrInvalidState + } + state, err := Decode(raw) + return state, obj, err +} +func (s *Store) Read(ctx context.Context) (*State, error) { + state, _, err := s.read(ctx) + return state, err +} + +func (s *Store) Create(ctx context.Context, in Input, actor string) (*State, error) { + if _, err := Validate(in); err != nil { + return nil, err + } + for attempt := 0; attempt < 5; attempt++ { + state, obj, err := s.read(ctx) + if err != nil { + return nil, err + } + if err := state.Add(in, actor, time.Now().UTC().Format(time.RFC3339)); err != nil { + return nil, err + } + raw, err := state.Encode() + if err != nil { + return nil, err + } + create := obj == nil + if create { + obj = &unstructured.Unstructured{Object: map[string]any{"apiVersion": "v1", "kind": "ConfigMap", "metadata": map[string]any{"name": s.name}}} + obj.SetLabels(map[string]string{"app.kubernetes.io/managed-by": "console"}) + obj.SetAnnotations(map[string]string{ownerAnnotation: "true"}) + } + if err := unstructured.SetNestedField(obj.Object, raw, "data", "state.json"); err != nil { + return nil, err + } + if create { + _, err = s.client.Create(ctx, obj, metav1.CreateOptions{}) + } else { + _, err = s.client.Update(ctx, obj, metav1.UpdateOptions{}) + } + if apierrors.IsAlreadyExists(err) || apierrors.IsConflict(err) { + continue + } + if err != nil { + return nil, err + } + return state, nil + } + return nil, apierrors.NewConflict(configMaps.GroupResource(), s.name, ErrDuplicate) +} diff --git a/internal/modeltemplate/template.go b/internal/modeltemplate/template.go new file mode 100644 index 0000000..dfaa298 --- /dev/null +++ b/internal/modeltemplate/template.go @@ -0,0 +1,338 @@ +package modeltemplate + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "regexp" + "strings" + + "gopkg.in/yaml.v3" +) + +const MaxInputBytes = 128 << 10 +const MaxStateBytes = 900 << 10 +const maxSafeInteger = 1<<53 - 1 + +var nameRE = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]{0,62}$`) +var variantRE = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,62}$`) +var versionRE = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$`) +var hfRE = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*$`) +var digestRE = regexp.MustCompile(`^sha256:[a-f0-9]{64}$`) +var imageRepositoryRE = regexp.MustCompile(`^[a-z0-9]+([._-][a-z0-9]+)*(:[0-9]+)?(/[a-z0-9]+([._-]+[a-z0-9]+)*)*$`) +var imageTagRE = regexp.MustCompile(`^[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$`) + +var ErrDuplicate = errors.New("a template with this model name already exists") +var ErrTooLarge = errors.New("model template storage exceeds its size limit") +var ErrInvalidState = errors.New("model template storage is invalid") + +func ValidName(name string) bool { return nameRE.MatchString(name) } +func ValidVersion(version string) bool { return len(version) <= 64 && versionRE.MatchString(version) } + +type Source struct { + HF string `json:"hf"` +} +type Metadata struct { + Name string `json:"name"` + DisplayName string `json:"displayName,omitempty"` + Description string `json:"description,omitempty"` + Family string `json:"family,omitempty"` + Tags []string `json:"tags,omitempty"` + Source Source `json:"source"` +} +type Input struct { + Metadata Metadata `json:"metadata"` + RecipeYAML string `json:"recipeYAML"` +} +type Recipe struct { + APIVersion string `yaml:"apiVersion" json:"apiVersion"` + Name string `yaml:"name" json:"name"` + Version string `yaml:"version" json:"version"` + ServedName string `yaml:"servedName,omitempty" json:"servedName,omitempty"` + Variants []Variant `yaml:"variants" json:"variants"` +} +type Chart struct { + Name string `yaml:"name" json:"name"` + Version string `yaml:"version" json:"version"` +} +type Image struct { + Repository string `yaml:"repository" json:"repository"` + Tag string `yaml:"tag" json:"tag"` + Digest string `yaml:"digest,omitempty" json:"digest,omitempty"` +} +type Requires struct { + GPUs int `yaml:"gpus" json:"gpus"` + Vendor string `yaml:"vendor,omitempty" json:"vendor,omitempty"` + Nodes int `yaml:"nodes,omitempty" json:"nodes,omitempty"` + Topology string `yaml:"topology,omitempty" json:"topology,omitempty"` + GPUProduct []string `yaml:"gpuProduct,omitempty" json:"gpuProduct,omitempty"` + MinGPUMemoryGiB float64 `yaml:"minGpuMemoryGiB,omitempty" json:"minGpuMemoryGiB,omitempty"` + RDMA bool `yaml:"rdma,omitempty" json:"rdma,omitempty"` +} +type Variant struct { + ID string `yaml:"id" json:"id"` + Description string `yaml:"description,omitempty" json:"description,omitempty"` + Link string `yaml:"link,omitempty" json:"link,omitempty"` + Default bool `yaml:"default,omitempty" json:"default,omitempty"` + Engine string `yaml:"engine" json:"engine"` + Chart Chart `yaml:"chart" json:"chart"` + Image *Image `yaml:"image,omitempty" json:"image,omitempty"` + Requires Requires `yaml:"requires" json:"requires"` + Values map[string]any `yaml:"values,omitempty" json:"values,omitempty"` +} + +type VariantSummary struct { + ID string `json:"id"` + Description string `json:"description,omitempty"` + Link string `json:"link,omitempty"` + Default bool `json:"default,omitempty"` + Engine string `json:"engine"` + Chart Chart `json:"chart"` + Requires Requires `json:"requires"` +} +type IndexVersion struct { + Version string `json:"version"` + Path string `json:"path"` + Digest string `json:"digest"` + Variants []VariantSummary `json:"variants"` +} +type IndexModel struct { + Metadata + Latest string `json:"latest"` + Versions []IndexVersion `json:"versions"` +} +type Index struct { + APIVersion string `json:"apiVersion"` + Count int `json:"count"` + Models []IndexModel `json:"models"` +} +type Record struct { + Input + CreationGen int64 `json:"creationGen"` + CreatedBy string `json:"createdBy"` + CreatedAt string `json:"createdAt"` +} +type State struct { + Version int `json:"version"` + Generation int64 `json:"generation"` + Records []Record `json:"records"` +} + +func Empty() *State { return &State{Version: 1, Records: []Record{}} } + +func Validate(in Input) (*Recipe, error) { + m := in.Metadata + if !ValidName(m.Name) || len(m.DisplayName) > 256 || len(m.Description) > 4096 || len(m.Family) > 128 || len(m.Tags) > 32 || len(m.Source.HF) > 256 || !hfRE.MatchString(m.Source.HF) { + return nil, errors.New("invalid model metadata; name and source.hf (org/name) are required") + } + for _, tag := range m.Tags { + if tag == "" || len(tag) > 128 { + return nil, errors.New("invalid model tags") + } + } + if in.RecipeYAML == "" || len(in.RecipeYAML) > MaxInputBytes { + return nil, errors.New("recipe YAML is empty or exceeds the size limit") + } + var node yaml.Node + decoder := yaml.NewDecoder(strings.NewReader(in.RecipeYAML)) + if err := decoder.Decode(&node); err != nil { + return nil, errors.New("invalid recipe YAML") + } + if decoder.Decode(new(yaml.Node)) != io.EOF { + return nil, errors.New("exactly one YAML document is required") + } + nodes := 0 + if !safeYAML(&node, 0, &nodes) { + return nil, errors.New("recipe YAML aliases, custom tags, complex keys or excessive nesting are not allowed") + } + var p Recipe + decoder = yaml.NewDecoder(strings.NewReader(in.RecipeYAML)) + decoder.KnownFields(true) + if err := decoder.Decode(&p); err != nil { + return nil, errors.New("recipe contains invalid fields or field types") + } + if p.APIVersion != "catalog.swiss/v1" || p.Name != m.Name || !ValidVersion(p.Version) || len(p.Variants) == 0 || len(p.Variants) > 32 { + return nil, errors.New("recipe must use catalog.swiss/v1, match the model name, and declare a semantic version and variants") + } + if len(p.ServedName) > 128 || strings.ContainsAny(p.ServedName, " \t\r\n") { + return nil, errors.New("invalid served model name") + } + seen := map[string]bool{} + defaults := 0 + for _, v := range p.Variants { + if !variantRE.MatchString(v.ID) || seen[v.ID] { + return nil, errors.New("variant ids must be valid and unique") + } + seen[v.ID] = true + if v.Default { + defaults++ + } + if (v.Engine != "sglang" && v.Engine != "vllm") || v.Chart.Name != v.Engine || !ValidVersion(v.Chart.Version) { + return nil, errors.New("variants require sglang/vllm and a matching chart with an exact semantic version") + } + if len(v.Description) > 4096 || len(v.Link) > 2048 { + return nil, errors.New("variant description or link exceeds the size limit") + } + if v.Image != nil && (!imageRepositoryRE.MatchString(v.Image.Repository) || len(v.Image.Repository) > 512 || !imageTagRE.MatchString(v.Image.Tag) || v.Image.Digest != "" && !digestRE.MatchString(v.Image.Digest)) { + return nil, errors.New("invalid engine image") + } + r := v.Requires + if r.GPUs < 1 || int64(r.GPUs) > maxSafeInteger || r.Nodes < 0 || int64(r.Nodes) > maxSafeInteger || r.MinGPUMemoryGiB < 0 || len(r.GPUProduct) > 32 { + return nil, errors.New("invalid GPU requirements") + } + switch r.Vendor { + case "", "nvidia", "ascend", "cambricon", "hygon", "amd": + default: + return nil, errors.New("unknown accelerator vendor") + } + switch r.Topology { + case "", "single-node": + case "lws": + lws, _ := v.Values["lws"].(map[string]any) + size, _ := lws["size"].(int) + if r.Nodes < 2 || size != r.Nodes { + return nil, errors.New("LWS requires matching requires.nodes and values.lws.size") + } + default: + return nil, errors.New("unknown GPU topology") + } + for _, product := range r.GPUProduct { + if product == "" || len(product) > 256 { + return nil, errors.New("invalid GPU product") + } + } + } + if defaults > 1 { + return nil, errors.New("at most one variant may be the default") + } + if _, err := json.Marshal(p); err != nil { + return nil, errors.New("recipe values must be JSON compatible") + } + return &p, nil +} + +func safeYAML(n *yaml.Node, depth int, count *int) bool { + *count++ + if depth > 48 || *count > 16000 || n.Kind == yaml.AliasNode { + return false + } + switch n.Tag { + case "", "!!map", "!!seq", "!!str", "!!int", "!!float", "!!bool", "!!null": + default: + return false + } + if n.Kind == yaml.MappingNode { + for i := 0; i < len(n.Content); i += 2 { + if n.Content[i].Kind != yaml.ScalarNode || n.Content[i].Tag != "!!str" { + return false + } + } + } + for _, child := range n.Content { + if !safeYAML(child, depth+1, count) { + return false + } + } + return true +} + +func Decode(raw string) (*State, error) { + if len(raw) > MaxStateBytes { + return nil, ErrTooLarge + } + var s State + dec := json.NewDecoder(strings.NewReader(raw)) + dec.DisallowUnknownFields() + if dec.Decode(&s) != nil || dec.Decode(new(any)) != io.EOF || s.Version != 1 || s.Generation < 0 || s.Generation > 1000 || int64(len(s.Records)) != s.Generation { + return nil, ErrInvalidState + } + names := map[string]bool{} + generations := map[int64]bool{} + for _, r := range s.Records { + if names[r.Metadata.Name] || r.CreationGen < 1 || r.CreationGen > s.Generation || generations[r.CreationGen] { + return nil, ErrInvalidState + } + if _, err := Validate(r.Input); err != nil { + return nil, ErrInvalidState + } + names[r.Metadata.Name] = true + generations[r.CreationGen] = true + } + return &s, nil +} +func (s *State) Encode() (string, error) { + raw, err := json.Marshal(s) + if err != nil { + return "", err + } + if len(raw) > MaxStateBytes { + return "", ErrTooLarge + } + return string(raw), nil +} +func (s *State) Add(in Input, actor, stamp string) error { + for _, record := range s.Records { + if record.Metadata.Name == in.Metadata.Name { + return ErrDuplicate + } + } + if s.Generation >= 1000 { + return ErrTooLarge + } + s.Generation++ + s.Records = append(s.Records, Record{Input: in, CreationGen: s.Generation, CreatedBy: actor, CreatedAt: stamp}) + return nil +} +func (s *State) Index(generation int64) (*Index, error) { + if generation < 0 || generation > s.Generation { + return nil, errors.New("unknown catalog generation") + } + idx := &Index{APIVersion: "catalog.swiss/v1", Models: []IndexModel{}} + for _, r := range s.Records { + if r.CreationGen > generation { + continue + } + p, err := Validate(r.Input) + if err != nil { + return nil, ErrInvalidState + } + variants := []VariantSummary{} + for _, v := range p.Variants { + variants = append(variants, VariantSummary{ID: v.ID, Description: v.Description, Link: v.Link, Default: v.Default, Engine: v.Engine, Chart: v.Chart, Requires: v.Requires}) + } + sum := sha256.Sum256([]byte(r.RecipeYAML)) + idx.Models = append(idx.Models, IndexModel{Metadata: r.Metadata, Latest: p.Version, Versions: []IndexVersion{{Version: p.Version, Path: fmt.Sprintf("models/%s/%s-%s.yaml", p.Name, p.Name, p.Version), Digest: "sha256:" + hex.EncodeToString(sum[:]), Variants: variants}}}) + } + idx.Count = len(idx.Models) + return idx, nil +} +func (s *State) Recipe(generation int64, name, filename string) ([]byte, bool) { + if generation < 0 || generation > s.Generation || !ValidName(name) { + return nil, false + } + for _, r := range s.Records { + if r.CreationGen <= generation && r.Metadata.Name == name { + p, err := Validate(r.Input) + if err != nil { + return nil, false + } + if filename == name+"-"+p.Version+".yaml" { + return []byte(r.RecipeYAML), true + } + } + } + return nil, false +} +func EqualIndex(raw []byte, want *Index) bool { + var got Index + if json.Unmarshal(raw, &got) != nil { + return false + } + a, _ := json.Marshal(&got) + b, _ := json.Marshal(want) + return bytes.Equal(a, b) +} diff --git a/internal/modeltemplate/template_test.go b/internal/modeltemplate/template_test.go new file mode 100644 index 0000000..93cc55b --- /dev/null +++ b/internal/modeltemplate/template_test.go @@ -0,0 +1,243 @@ +package modeltemplate + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "strings" + "testing" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + dynamicfake "k8s.io/client-go/dynamic/fake" + k8stesting "k8s.io/client-go/testing" +) + +func testInput(name string) Input { + return Input{Metadata: Metadata{Name: name, DisplayName: "Test model", Source: Source{HF: "org/model"}}, RecipeYAML: fmt.Sprintf("# preserve these exact bytes\napiVersion: catalog.swiss/v1\nname: %s\nversion: 1.0.0\nvariants:\n - id: vllm-gpu\n engine: vllm\n chart: {name: vllm, version: 0.7.1}\n image: {repository: registry.example/vllm, tag: stable}\n requires: {gpus: 1, vendor: nvidia}\n values:\n extraArgs: [--trust-remote-code]\n", name)} +} + +func TestGenerationSnapshotsAndExactLegacyIndex(t *testing.T) { + state := Empty() + first := testInput("first") + if err := state.Add(first, "admin", "now"); err != nil { + t.Fatal(err) + } + if err := state.Add(testInput("second"), "admin", "later"); err != nil { + t.Fatal(err) + } + index, err := state.Index(1) + if err != nil { + t.Fatal(err) + } + if index.Count != 1 || index.Models[0].Name != "first" || index.Models[0].Source.HF != "org/model" { + t.Fatalf("index=%+v", index) + } + sum := sha256.Sum256([]byte(first.RecipeYAML)) + v := index.Models[0].Versions[0] + if v.Digest != "sha256:"+hex.EncodeToString(sum[:]) || v.Path != "models/first/first-1.0.0.yaml" { + t.Fatalf("pin=%+v", v) + } + raw, _ := json.Marshal(index) + var wire map[string]json.RawMessage + _ = json.Unmarshal(raw, &wire) + if len(wire) != 3 || wire["apiVersion"] == nil || wire["count"] == nil || wire["models"] == nil { + t.Fatalf("legacy index fields=%v", wire) + } + if strings.Contains(string(raw), "extraArgs") || strings.Contains(string(raw), "image") || strings.Contains(first.RecipeYAML, "source:") { + t.Fatal("listing or recipe mixes metadata and deploy values") + } + doc, ok := state.Recipe(1, "first", "first-1.0.0.yaml") + if !ok || string(doc) != first.RecipeYAML { + t.Fatal("recipe bytes changed") + } + if _, ok := state.Recipe(1, "second", "second-1.0.0.yaml"); ok { + t.Fatal("old snapshot exposes a later record") + } + if _, err := state.Index(3); err == nil { + t.Fatal("future snapshot accepted") + } + before := state.Generation + if !errors.Is(state.Add(first, "other", "later"), ErrDuplicate) || state.Generation != before { + t.Fatal("duplicate mutated state") + } + encoded, err := state.Encode() + if err != nil { + t.Fatal(err) + } + decoded, err := Decode(encoded) + if err != nil || decoded.Generation != 2 { + t.Fatalf("state=%v err=%v", decoded, err) + } +} + +func TestPositiveGPUCountsAndImageDigest(t *testing.T) { + in := testInput("large") + in.RecipeYAML = strings.Replace(in.RecipeYAML, "gpus: 1, vendor: nvidia", "gpus: 16, vendor: amd, nodes: 512, topology: lws, rdma: true", 1) + in.RecipeYAML = strings.Replace(in.RecipeYAML, " values:\n", " values:\n lws: {size: 512}\n", 1) + digest := "sha256:" + strings.Repeat("a", 64) + in.RecipeYAML = strings.Replace(in.RecipeYAML, "tag: stable}", "tag: stable, digest: "+digest+"}", 1) + recipe, err := Validate(in) + if err != nil { + t.Fatal(err) + } + variant := recipe.Variants[0] + if variant.Requires.GPUs != 16 || variant.Requires.Nodes != 512 || !variant.Requires.RDMA || variant.Image.Digest != digest { + t.Fatalf("recipe fields lost: %+v", variant) + } + state := Empty() + if err := state.Add(in, "admin", "now"); err != nil { + t.Fatal(err) + } + encoded, err := state.Encode() + if err != nil { + t.Fatal(err) + } + state, err = Decode(encoded) + if err != nil { + t.Fatal(err) + } + index, err := state.Index(1) + if err != nil { + t.Fatal(err) + } + if index.Models[0].Versions[0].Variants[0].Requires.GPUs != 16 { + t.Fatal("index changed GPU requirements") + } + doc, ok := state.Recipe(1, "large", "large-1.0.0.yaml") + if !ok || string(doc) != in.RecipeYAML { + t.Fatal("image digest or recipe bytes were not preserved") + } +} + +func TestStrictTemplateValidation(t *testing.T) { + base := testInput("model") + if _, err := Validate(base); err != nil { + t.Fatal(err) + } + cases := map[string]func(*Input){ + "unknown field": func(in *Input) { in.RecipeYAML += "namespace: other\n" }, + "source in recipe": func(in *Input) { in.RecipeYAML += "source: {hf: other/model}\n" }, + "multiple documents": func(in *Input) { in.RecipeYAML += "---\n{}\n" }, + "name mismatch": func(in *Input) { in.Metadata.Name = "other" }, + "unsafe name": func(in *Input) { in.Metadata.Name = "../model" }, + "invalid HF": func(in *Input) { in.Metadata.Source.HF = "http://example/model" }, + "engine chart mismatch": func(in *Input) { in.RecipeYAML = strings.Replace(in.RecipeYAML, "name: vllm", "name: sglang", 1) }, + "chart range": func(in *Input) { + in.RecipeYAML = strings.Replace(in.RecipeYAML, "version: 0.7.1", "version: latest", 1) + }, + "zero GPUs": func(in *Input) { in.RecipeYAML = strings.Replace(in.RecipeYAML, "gpus: 1", "gpus: 0", 1) }, + "unknown vendor": func(in *Input) { + in.RecipeYAML = strings.Replace(in.RecipeYAML, "vendor: nvidia", "vendor: made-up", 1) + }, + "invalid image": func(in *Input) { + in.RecipeYAML = strings.Replace(in.RecipeYAML, "registry.example/vllm", "https://registry.example/vllm", 1) + }, + "duplicate key": func(in *Input) { in.RecipeYAML += "version: 2.0.0\n" }, + "alias": func(in *Input) { + in.RecipeYAML = strings.Replace(in.RecipeYAML, "[--trust-remote-code]", "&args [--trust-remote-code]\n other: *args", 1) + }, + "binary": func(in *Input) { + in.RecipeYAML = strings.Replace(in.RecipeYAML, "[--trust-remote-code]", "!!binary SGVsbG8=", 1) + }, + "nonstring map key": func(in *Input) { + in.RecipeYAML = strings.Replace(in.RecipeYAML, "extraArgs: [--trust-remote-code]", "1: something", 1) + }, + "oversized": func(in *Input) { in.RecipeYAML = strings.Repeat("x", MaxInputBytes+1) }, + } + for name, change := range cases { + t.Run(name, func(t *testing.T) { + in := base + change(&in) + if _, err := Validate(in); err == nil { + t.Fatal("invalid template accepted") + } + }) + } +} + +func TestStateBoundsAndCorruption(t *testing.T) { + for _, raw := range []string{`{}`, `{"version":2,"generation":0,"records":[]}`, `{"version":1,"generation":2,"records":[]}`, strings.Repeat("x", MaxStateBytes+1)} { + if _, err := Decode(raw); err == nil { + t.Fatal("invalid state accepted") + } + } + s := Empty() + for i := 0; i < 9; i++ { + in := testInput(fmt.Sprintf("model-%d", i)) + in.RecipeYAML += "# " + strings.Repeat("x", 110<<10) + "\n" + if err := s.Add(in, "admin", "now"); err != nil { + t.Fatal(err) + } + } + if _, err := s.Encode(); !errors.Is(err, ErrTooLarge) { + t.Fatalf("size err=%v", err) + } +} + +func fakeStore(objects ...runtime.Object) (*Store, *dynamicfake.FakeDynamicClient) { + dyn := dynamicfake.NewSimpleDynamicClientWithCustomListKinds(runtime.NewScheme(), map[schema.GroupVersionResource]string{configMaps: "ConfigMapList"}, objects...) + return NewStore(dyn, "console-ns", "templates"), dyn +} + +func TestStoreCASAndCreateOnlyModelNames(t *testing.T) { + store, dyn := fakeStore() + state, err := store.Create(t.Context(), testInput("first"), "admin") + if err != nil || state.Generation != 1 { + t.Fatalf("create=%v %v", state, err) + } + conflicts := 0 + dyn.PrependReactor("update", "configmaps", func(action k8stesting.Action) (bool, runtime.Object, error) { + if conflicts == 0 { + conflicts++ + return true, nil, apierrors.NewConflict(configMaps.GroupResource(), "templates", errors.New("concurrent save")) + } + return false, nil, nil + }) + state, err = store.Create(t.Context(), testInput("second"), "admin") + if err != nil || state.Generation != 2 || conflicts != 1 { + t.Fatalf("CAS=%v %v", state, err) + } + if _, err := store.Create(t.Context(), testInput("first"), "other"); !errors.Is(err, ErrDuplicate) { + t.Fatalf("duplicate=%v", err) + } + for _, action := range dyn.Actions() { + if action.GetNamespace() != "console-ns" || action.GetResource() != configMaps { + t.Fatalf("unsafe storage scope: %#v", action) + } + } + obj, err := dyn.Resource(configMaps).Namespace("console-ns").Get(t.Context(), "templates", metav1.GetOptions{}) + if err != nil { + t.Fatal(err) + } + if obj.GetLabels()["app.kubernetes.io/managed-by"] != "console" || obj.GetAnnotations()[ownerAnnotation] != "true" { + t.Fatal("runtime ownership missing") + } +} + +func TestStoreRefusesForeignConfigMapAndRetriesBoundedly(t *testing.T) { + foreign := &unstructured.Unstructured{Object: map[string]any{"apiVersion": "v1", "kind": "ConfigMap", "metadata": map[string]any{"name": "templates", "namespace": "console-ns"}, "data": map[string]any{"state.json": `{"version":1,"generation":0,"records":[]}`}}} + store, dyn := fakeStore(foreign) + if _, err := store.Create(t.Context(), testInput("first"), "admin"); !errors.Is(err, ErrInvalidState) { + t.Fatalf("foreign=%v", err) + } + for _, a := range dyn.Actions() { + if a.GetVerb() != "get" { + t.Fatal("foreign object changed") + } + } + store, dyn = fakeStore() + creates := 0 + dyn.PrependReactor("create", "configmaps", func(k8stesting.Action) (bool, runtime.Object, error) { + creates++ + return true, nil, apierrors.NewAlreadyExists(configMaps.GroupResource(), "templates") + }) + if _, err := store.Create(t.Context(), testInput("first"), "admin"); !apierrors.IsConflict(err) || creates != 5 { + t.Fatalf("retry err=%v count=%d", err, creates) + } +} diff --git a/internal/server/auth.go b/internal/server/auth.go index 6917664..de75217 100644 --- a/internal/server/auth.go +++ b/internal/server/auth.go @@ -35,6 +35,9 @@ var anonymousAdmin = &iam.Identity{Name: "admin", Groups: []string{"system:maste // error envelope rather than the login HTML. func (s *Server) authenticate(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/api/model-templates" || strings.HasPrefix(r.URL.Path, "/api/model-templates/") { + w.Header().Set("Cache-Control", "no-store") + } if s.cfg.Server.Auth.Disabled { next.ServeHTTP(w, r.WithContext(withIdentity(r.Context(), anonymousAdmin))) return diff --git a/internal/server/model_template_catalog.go b/internal/server/model_template_catalog.go new file mode 100644 index 0000000..2568d69 --- /dev/null +++ b/internal/server/model_template_catalog.go @@ -0,0 +1,122 @@ +package server + +import ( + "bytes" + "encoding/json" + "slices" + + "github.com/modelsphere/console/internal/modeltemplate" +) + +type templateCatalogModel struct { + modeltemplate.Metadata + DefaultVariant string `json:"defaultVariant"` + Variants []templateCatalogVariant `json:"variants"` +} + +type templateCatalogVariant struct { + modeltemplate.VariantSummary + Version string `json:"version"` + Digest string `json:"digest"` +} + +func templateCatalogMatches(result map[string]json.RawMessage, want *modeltemplate.Index) bool { + schema := "" + if raw, ok := result["schema"]; ok { + if json.Unmarshal(raw, &schema) != nil || schema != "v1" { + return false + } + } + hasIndex := false + if raw, ok := result["index"]; ok && !bytes.Equal(bytes.TrimSpace(raw), []byte("null")) { + if !modeltemplate.EqualIndex(raw, want) { + return false + } + hasIndex = true + } + if raw, ok := result["models"]; ok { + return schema == "v1" && templateCatalogModelsMatch(raw, want) + } + return hasIndex +} + +func templateCatalogModelsMatch(raw json.RawMessage, want *modeltemplate.Index) bool { + var models []templateCatalogModel + if json.Unmarshal(raw, &models) != nil || models == nil || len(models) != len(want.Models) { + return false + } + byName := make(map[string]templateCatalogModel, len(models)) + for _, model := range models { + if _, exists := byName[model.Name]; exists { + return false + } + byName[model.Name] = model + } + for _, expected := range want.Models { + model, ok := byName[expected.Name] + if !ok || !templateMetadataMatches(model.Metadata, expected.Metadata) { + return false + } + var latest *modeltemplate.IndexVersion + for i := range expected.Versions { + if expected.Versions[i].Version == expected.Latest { + latest = &expected.Versions[i] + break + } + } + if latest == nil || len(latest.Variants) == 0 || len(model.Variants) != len(latest.Variants) { + return false + } + defaultID := latest.Variants[0].ID + for _, variant := range latest.Variants { + if variant.Default { + defaultID = variant.ID + break + } + } + if model.DefaultVariant != defaultID { + return false + } + variants := make(map[string]templateCatalogVariant, len(model.Variants)) + for _, variant := range model.Variants { + if _, exists := variants[variant.ID]; exists { + return false + } + variants[variant.ID] = variant + } + for _, expectedVariant := range latest.Variants { + variant, ok := variants[expectedVariant.ID] + if !ok || variant.Version != latest.Version || variant.Digest != latest.Digest || variant.Engine != expectedVariant.Engine || variant.Chart != expectedVariant.Chart || !templateRequirementsMatch(variant.Requires, expectedVariant.Requires) { + return false + } + } + } + return true +} + +func templateMetadataMatches(got, want modeltemplate.Metadata) bool { + return got.Name == want.Name && got.DisplayName == want.DisplayName && got.Description == want.Description && got.Family == want.Family && got.Source.HF == want.Source.HF && templateStringsEqual(got.Tags, want.Tags) +} +func templateStringsEqual(a, b []string) bool { + a = slices.Clone(a) + b = slices.Clone(b) + slices.Sort(a) + slices.Sort(b) + return slices.Equal(a, b) +} +func templateRequirementsMatch(a, b modeltemplate.Requires) bool { + normalize := func(r modeltemplate.Requires) modeltemplate.Requires { + if r.Vendor == "" { + r.Vendor = "nvidia" + } + if r.Nodes == 0 { + r.Nodes = 1 + } + if r.Topology == "" { + r.Topology = "single-node" + } + return r + } + a, b = normalize(a), normalize(b) + return a.GPUs == b.GPUs && a.Vendor == b.Vendor && a.Nodes == b.Nodes && a.Topology == b.Topology && a.MinGPUMemoryGiB == b.MinGPUMemoryGiB && a.RDMA == b.RDMA && templateStringsEqual(a.GPUProduct, b.GPUProduct) +} diff --git a/internal/server/model_template_catalog_test.go b/internal/server/model_template_catalog_test.go new file mode 100644 index 0000000..f5d28fa --- /dev/null +++ b/internal/server/model_template_catalog_test.go @@ -0,0 +1,170 @@ +package server + +import ( + "encoding/json" + "slices" + "testing" + + "github.com/modelsphere/console/internal/modeltemplate" +) + +func modernTemplateEnvelope(index modeltemplate.Index) map[string]any { + models := []any{} + for _, model := range index.Models { + raw, _ := json.Marshal(model.Metadata) + var metadata map[string]any + _ = json.Unmarshal(raw, &metadata) + version := model.Versions[0] + defaultID := version.Variants[0].ID + variants := []any{} + for _, variant := range version.Variants { + raw, _ := json.Marshal(variant) + var fields map[string]any + _ = json.Unmarshal(raw, &fields) + fields["version"] = version.Version + fields["digest"] = version.Digest + variants = append(variants, fields) + if variant.Default { + defaultID = variant.ID + } + } + metadata["variants"] = variants + metadata["defaultVariant"] = defaultID + models = append(models, metadata) + } + return map[string]any{"schema": "v1", "models": models} +} +func templateEnvelopeRaw(envelope map[string]any) map[string]json.RawMessage { + raw, _ := json.Marshal(envelope) + var result map[string]json.RawMessage + _ = json.Unmarshal(raw, &result) + return result +} +func firstModernModel(envelope map[string]any) map[string]any { + return envelope["models"].([]any)[0].(map[string]any) +} +func firstModernVariant(envelope map[string]any) map[string]any { + return firstModernModel(envelope)["variants"].([]any)[0].(map[string]any) +} + +func TestModelTemplatesModernV1CreateReadAndPublishRetry(t *testing.T) { + _, _, swiss, h := newTemplateTestServer(t) + swiss.modern = true + rec := do(h, "POST", "/api/model-templates", "", templateBody("first")) + if rec.Code != 201 { + t.Fatalf("modern create %d %s", rec.Code, rec.Body.String()) + } + list := decodeTemplateResult(t, do(h, "GET", "/api/model-templates", "", "")) + if list["registeredGeneration"] != float64(1) || list["registrationError"] != nil { + t.Fatalf("modern read=%v", list) + } + if published := do(h, "POST", "/api/model-templates/publish", "", "{}"); published.Code != 200 { + t.Fatalf("already published %d %s", published.Code, published.Body.String()) + } + if swiss.puts != 1 { + t.Fatalf("already verified registration rewrote profile %d times", swiss.puts) + } + swiss.unavailable = true + if pending := do(h, "POST", "/api/model-templates", "", templateBody("second")); pending.Code != 202 { + t.Fatalf("modern pending %d %s", pending.Code, pending.Body.String()) + } + swiss.unavailable = false + retry := do(h, "POST", "/api/model-templates/publish", "", "{}") + if retry.Code != 200 { + t.Fatalf("modern retry %d %s", retry.Code, retry.Body.String()) + } + result := decodeTemplateResult(t, retry) + if result["ready"] != true || result["registeredGeneration"] != float64(2) { + t.Fatalf("modern retry=%v", result) + } + list = decodeTemplateResult(t, do(h, "GET", "/api/model-templates", "", "")) + if list["registeredGeneration"] != float64(2) { + t.Fatalf("modern read after retry=%v", list) + } +} + +func TestTemplateCatalogMatchesNormalizedV1Semantics(t *testing.T) { + state := modeltemplate.Empty() + in := templateTestInput("first") + in.Metadata.Tags = []string{"chat", "private"} + if err := state.Add(in, "admin", "now"); err != nil { + t.Fatal(err) + } + if err := state.Add(templateTestInput("second"), "admin", "now"); err != nil { + t.Fatal(err) + } + index, err := state.Index(2) + if err != nil { + t.Fatal(err) + } + index.Models[0].Versions[0].Variants = append(index.Models[0].Versions[0].Variants, modeltemplate.VariantSummary{ID: "alternate", Engine: "vllm", Default: true, Chart: modeltemplate.Chart{Name: "vllm", Version: "0.7.1"}, Requires: modeltemplate.Requires{GPUs: 2}}) + envelope := modernTemplateEnvelope(*index) + model := firstModernModel(envelope) + model["tags"] = []any{"private", "chat"} + model["tuning"] = []any{} + variants := model["variants"].([]any) + variants[0].(map[string]any)["default"] = false + requirements := variants[0].(map[string]any)["requires"].(map[string]any) + requirements["vendor"] = "nvidia" + requirements["nodes"] = 1 + requirements["topology"] = "single-node" + requirements["gpuProduct"] = []any{} + slices.Reverse(variants) + slices.Reverse(envelope["models"].([]any)) + if !templateCatalogMatches(templateEnvelopeRaw(envelope), index) { + t.Fatal("equivalent ordering, nil/empty and explicit default requirements were rejected") + } + // The model's effective default is checked independently of optional variant flags. + first := envelope["models"].([]any)[1].(map[string]any) + first["defaultVariant"] = "vllm-gpu" + if templateCatalogMatches(templateEnvelopeRaw(envelope), index) { + t.Fatal("different effective default variant accepted") + } + empty, _ := modeltemplate.Empty().Index(0) + if !templateCatalogMatches(templateEnvelopeRaw(modernTemplateEnvelope(*empty)), empty) { + t.Fatal("empty normalized v1 catalog rejected") + } +} + +func TestModelTemplatesRejectModernCatalogContentMismatches(t *testing.T) { + changes := map[string]func(map[string]any){ + "schema v2": func(e map[string]any) { e["schema"] = "v2" }, + "missing schema": func(e map[string]any) { delete(e, "schema") }, + "wrong catalog": func(e map[string]any) { e["name"] = "other" }, + "wrong source": func(e map[string]any) { e["source"] = "http://other.example/index.json" }, + "malformed models": func(e map[string]any) { e["models"] = "invalid" }, + "missing models": func(e map[string]any) { delete(e, "models") }, + "null models": func(e map[string]any) { e["models"] = nil }, + "model name": func(e map[string]any) { firstModernModel(e)["name"] = "other" }, + "model metadata": func(e map[string]any) { firstModernModel(e)["description"] = "different" }, + "HF identity": func(e map[string]any) { firstModernModel(e)["source"] = map[string]any{"hf": "other/model"} }, + "default variant": func(e map[string]any) { firstModernModel(e)["defaultVariant"] = "other" }, + "variant identity": func(e map[string]any) { firstModernVariant(e)["id"] = "other" }, + "version pin": func(e map[string]any) { firstModernVariant(e)["version"] = "2.0.0" }, + "digest pin": func(e map[string]any) { firstModernVariant(e)["digest"] = "sha256:wrong" }, + "engine": func(e map[string]any) { firstModernVariant(e)["engine"] = "sglang" }, + "chart": func(e map[string]any) { + firstModernVariant(e)["chart"] = map[string]any{"name": "vllm", "version": "1.0.0"} + }, + "GPU requirements": func(e map[string]any) { firstModernVariant(e)["requires"].(map[string]any)["gpus"] = 2 }, + "duplicate model": func(e map[string]any) { e["models"] = append(e["models"].([]any), firstModernModel(e)) }, + "invalid legacy index alongside models": func(e map[string]any) { + e["index"] = map[string]any{"apiVersion": "catalog.swiss/v1", "count": 0, "models": []any{}} + }, + } + for name, change := range changes { + t.Run(name, func(t *testing.T) { + _, _, swiss, h := newTemplateTestServer(t) + swiss.modern = true + swiss.modernChange = change + rec := do(h, "POST", "/api/model-templates", "", templateBody("first")) + if rec.Code != 202 { + t.Fatalf("mismatch %d %s", rec.Code, rec.Body.String()) + } + result := decodeTemplateResult(t, rec) + if result["ready"] != false || result["registeredGeneration"] != nil || result["reason"] != "registration_unverified" { + t.Fatalf("unverified modern response accepted=%v", result) + } + }) + } +} diff --git a/internal/server/model_template_error.go b/internal/server/model_template_error.go new file mode 100644 index 0000000..58b969c --- /dev/null +++ b/internal/server/model_template_error.go @@ -0,0 +1,78 @@ +package server + +import ( + "context" + "errors" + "fmt" + "net" + + apierrors "k8s.io/apimachinery/pkg/api/errors" +) + +type templateRegistrationError struct { + reason string + operation string + cause error +} + +func (e *templateRegistrationError) Error() string { return safeTemplateReason(e.reason) } +func (e *templateRegistrationError) Unwrap() error { return e.cause } +func templateFailure(reason, operation string, cause error) error { + return &templateRegistrationError{reason: reason, operation: operation, cause: cause} +} + +type templateHTTPStatusError struct{ status int } + +func (e *templateHTTPStatusError) Error() string { + return fmt.Sprintf("upstream HTTP status %d", e.status) +} + +func safeTemplateReason(reason string) string { + switch reason { + case "backend_unavailable", "backend_rejected", "backend_response_invalid", "profile_too_large", "profile_unavailable", "profile_invalid", "catalog_name_conflict", "catalog_not_registered", "registration_changed", "registration_unverified", "registration_timeout", "storage_unavailable", "existing_catalog_unknown": + return reason + default: + return "registration_failed" + } +} + +type templateRegistrationDiagnostic struct { + reason string + operation string + causeType string + status int + timeout bool +} + +func templateFailureDiagnostic(err error) templateRegistrationDiagnostic { + d := templateRegistrationDiagnostic{reason: "registration_failed", operation: "registration", causeType: fmt.Sprintf("%T", err)} + var failure *templateRegistrationError + if errors.As(err, &failure) { + d.reason = safeTemplateReason(failure.reason) + d.operation = failure.operation + d.causeType = fmt.Sprintf("%T", failure.cause) + } + var status *templateHTTPStatusError + if errors.As(err, &status) && status.status >= 100 && status.status <= 599 { + d.status = status.status + } + var apiStatus apierrors.APIStatus + if d.status == 0 && errors.As(err, &apiStatus) { + code := int(apiStatus.Status().Code) + if code >= 100 && code <= 599 { + d.status = code + } + } + var network net.Error + d.timeout = errors.Is(err, context.DeadlineExceeded) || (errors.As(err, &network) && network.Timeout()) + return d +} + +func (s *Server) reportTemplateFailure(ctx context.Context, err error) string { + if err == nil { + return "" + } + d := templateFailureDiagnostic(err) + s.log.WarnContext(ctx, "model template registration pending", "reason", d.reason, "operation", d.operation, "cause_type", d.causeType, "status", d.status, "timeout", d.timeout) + return d.reason +} diff --git a/internal/server/model_template_registration.go b/internal/server/model_template_registration.go new file mode 100644 index 0000000..c53750d --- /dev/null +++ b/internal/server/model_template_registration.go @@ -0,0 +1,383 @@ +package server + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "strconv" + "strings" + "time" + + "github.com/modelsphere/console/internal/modeltemplate" + "gopkg.in/yaml.v3" +) + +const templateOperationTimeout = 45 * time.Second +const maxTemplateUpstreamBytes = 2 << 20 + +func (s *Server) templateRequest(ctx context.Context, r *http.Request, method, path string, body any) (map[string]json.RawMessage, error) { + name := s.cfg.ModelTemplates.WithDefaults().Backend + resolve := s.backendTargets()[name] + if resolve == nil { + return nil, templateFailure("backend_unavailable", "backend_resolve", errors.New("configured backend has no resolver")) + } + target, err := resolve(ctx) + if err != nil { + return nil, templateFailure("backend_unavailable", "backend_resolve", err) + } + var raw []byte + if body != nil { + raw, err = json.Marshal(body) + if err != nil { + return nil, templateFailure("profile_too_large", "request_encode", err) + } + if len(raw) > maxTemplateUpstreamBytes { + return nil, templateFailure("profile_too_large", "request_encode", errors.New("upstream request exceeds size limit")) + } + } + u := target.url + suffix, query, _ := strings.Cut(path, "?") + u.Path = strings.TrimSuffix(u.Path, "/") + "/" + suffix + u.RawPath = "" + u.RawQuery = query + u.Fragment = "" + request, err := http.NewRequestWithContext(ctx, method, u.String(), bytes.NewReader(raw)) + if err != nil { + return nil, templateFailure("backend_unavailable", "request_build", err) + } + request.Header.Set("Content-Type", "application/json") + if id := identityFrom(r.Context()); id != nil { + request.Header.Set("X-Remote-User", id.Name) + for _, group := range id.Groups { + request.Header.Add("X-Remote-Group", group) + } + } + if token := bearerToken(r); token != "" { + request.Header.Set("Authorization", "Bearer "+token) + } + if target.key != "" { + request.Header.Set(target.header, target.key) + } + client := &http.Client{Timeout: 10 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }} + response, err := client.Do(request) + if err != nil { + return nil, templateFailure("backend_unavailable", "backend_request", err) + } + defer response.Body.Close() + if response.StatusCode != http.StatusOK { + return nil, templateFailure("backend_rejected", "backend_response", &templateHTTPStatusError{status: response.StatusCode}) + } + raw, err = io.ReadAll(io.LimitReader(response.Body, maxTemplateUpstreamBytes+1)) + if err != nil { + return nil, templateFailure("backend_response_invalid", "response_read", err) + } + if len(raw) > maxTemplateUpstreamBytes { + return nil, templateFailure("backend_response_invalid", "response_read", errors.New("upstream response exceeds size limit")) + } + var result map[string]json.RawMessage + if err := json.Unmarshal(raw, &result); err != nil { + return nil, templateFailure("backend_response_invalid", "response_decode", err) + } + if result == nil { + return nil, templateFailure("backend_response_invalid", "response_decode", errors.New("upstream response is not a JSON object")) + } + return result, nil +} + +func (s *Server) templateProfile(ctx context.Context, r *http.Request) (string, *yaml.Node, error) { + result, err := s.templateRequest(ctx, r, http.MethodGet, "profile", nil) + if err != nil { + return "", nil, err + } + var text string + if err := json.Unmarshal(result["yaml"], &text); err != nil { + return "", nil, templateFailure("profile_unavailable", "profile_decode", err) + } + if text == "" { + return "", nil, templateFailure("profile_unavailable", "profile_decode", errors.New("profile YAML is empty")) + } + var node yaml.Node + dec := yaml.NewDecoder(strings.NewReader(text)) + if err := dec.Decode(&node); err != nil { + return "", nil, templateFailure("profile_invalid", "profile_decode", err) + } + if err := dec.Decode(new(yaml.Node)); err != io.EOF { + if err == nil { + err = errors.New("profile contains multiple YAML documents") + } + return "", nil, templateFailure("profile_invalid", "profile_decode", err) + } + if len(node.Content) != 1 || node.Content[0].Kind != yaml.MappingNode { + return "", nil, templateFailure("profile_invalid", "profile_decode", errors.New("profile is not a mapping")) + } + var check map[string]any + if err := node.Decode(&check); err != nil { + return "", nil, templateFailure("profile_invalid", "profile_decode", err) + } + return text, &node, nil +} + +func yamlField(mapping *yaml.Node, key string) *yaml.Node { + for i := 0; i+1 < len(mapping.Content); i += 2 { + if mapping.Content[i].Value == key { + return mapping.Content[i+1] + } + } + return nil +} +func setYAMLField(mapping *yaml.Node, key string, value *yaml.Node) { + for i := 0; i+1 < len(mapping.Content); i += 2 { + if mapping.Content[i].Value == key { + mapping.Content[i+1] = value + return + } + } + mapping.Content = append(mapping.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: key}, value) +} +func removeYAMLField(mapping *yaml.Node, key string) { + for i := 0; i+1 < len(mapping.Content); i += 2 { + if mapping.Content[i].Value == key { + mapping.Content = append(mapping.Content[:i], mapping.Content[i+2:]...) + return + } + } +} +func catalogNode(name, location string, default_ bool) *yaml.Node { + node := &yaml.Node{} + _ = node.Encode(map[string]any{"name": name, "url": location, "default": default_}) + return node +} + +func (s *Server) templateGeneration(location string) (int64, bool) { + prefix := s.cfg.ModelTemplates.WithDefaults().PublishURL + "/g/" + rest, ok := strings.CutPrefix(location, prefix) + if !ok { + return 0, false + } + digits, ok := strings.CutSuffix(rest, "/index.json") + if !ok { + return 0, false + } + n, err := strconv.ParseInt(digits, 10, 64) + return n, err == nil && n >= 0 && strconv.FormatInt(n, 10) == digits +} +func (s *Server) templateCatalogURL(generation int64) string { + if s.cfg.ModelTemplates.PublishURL == "" { + return "" + } + return fmt.Sprintf("%s/g/%d/index.json", s.cfg.ModelTemplates.WithDefaults().PublishURL, generation) +} + +func (s *Server) templateProfileEntry(profile *yaml.Node) (*yaml.Node, error) { + root := profile.Content[0] + entries := yamlField(root, "catalogs") + if entries == nil || entries.Tag == "!!null" { + return nil, nil + } + if entries.Kind != yaml.SequenceNode { + return nil, templateFailure("profile_invalid", "profile_catalogs", errors.New("invalid catalog YAML structure")) + } + var own *yaml.Node + for _, entry := range entries.Content { + if entry.Kind != yaml.MappingNode { + return nil, templateFailure("profile_invalid", "profile_catalogs", errors.New("invalid catalog YAML structure")) + } + name := yamlField(entry, "name") + if name == nil || name.Value != s.cfg.ModelTemplates.WithDefaults().CatalogName { + continue + } + location := yamlField(entry, "url") + if own != nil || location == nil { + return nil, templateFailure("catalog_name_conflict", "catalog_ownership", errors.New("catalog name belongs to another source or is duplicated")) + } + if _, ok := s.templateGeneration(location.Value); !ok { + return nil, templateFailure("catalog_name_conflict", "catalog_ownership", errors.New("catalog name belongs to another source or is duplicated")) + } + own = entry + } + return own, nil +} + +func (s *Server) templateRegistration(ctx context.Context, r *http.Request, state *modeltemplate.State) (*int64, error) { + _, profile, err := s.templateProfile(ctx, r) + if err != nil { + return nil, err + } + own, err := s.templateProfileEntry(profile) + if err != nil { + return nil, err + } + if own == nil { + return nil, templateFailure("catalog_not_registered", "catalog_lookup", errors.New("profile has no local catalog entry")) + } + location := yamlField(own, "url").Value + generation, _ := s.templateGeneration(location) + idx, err := state.Index(generation) + if err != nil { + return nil, templateFailure("registration_changed", "snapshot_lookup", err) + } + result, err := s.templateRequest(ctx, r, http.MethodGet, "catalog?catalog="+s.cfg.ModelTemplates.WithDefaults().CatalogName, nil) + if err != nil { + return nil, err + } + var source, name string + if err := json.Unmarshal(result["source"], &source); err != nil { + return nil, templateFailure("registration_unverified", "catalog_identity", err) + } + if err := json.Unmarshal(result["name"], &name); err != nil { + return nil, templateFailure("registration_unverified", "catalog_identity", err) + } + if source != location || name != s.cfg.ModelTemplates.WithDefaults().CatalogName || !templateCatalogMatches(result, idx) { + return nil, templateFailure("registration_unverified", "catalog_verify", errors.New("catalog identity or snapshot contents did not match")) + } + return &generation, nil +} + +func (s *Server) templateUpdatedProfile(ctx context.Context, r *http.Request, profile *yaml.Node, generation int64) (string, error) { + root := profile.Content[0] + own, err := s.templateProfileEntry(profile) + if err != nil { + return "", err + } + if own != nil { + current, _ := s.templateGeneration(yamlField(own, "url").Value) + if current > generation { + return "", templateFailure("registration_changed", "generation_check", errors.New("profile generation is ahead of local state")) + } + } + entries := yamlField(root, "catalogs") + if entries == nil || entries.Tag == "!!null" || len(entries.Content) == 0 { + entries = &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"} + legacy := yamlField(root, "catalog") + if legacy != nil && strings.TrimSpace(legacy.Value) != "" && legacy.Tag != "!!null" { + entries.Content = append(entries.Content, catalogNode("default", legacy.Value, true)) + } else { + result, err := s.templateRequest(ctx, r, http.MethodGet, "cluster", nil) + if err != nil { + return "", err + } + var cluster struct { + Catalog string `json:"catalog"` + Catalogs []struct { + Name string `json:"name"` + URL string `json:"url"` + Default bool `json:"default"` + } `json:"catalogs"` + } + raw, err := json.Marshal(result) + if err != nil { + return "", templateFailure("existing_catalog_unknown", "cluster_decode", err) + } + if err := json.Unmarshal(raw, &cluster); err != nil { + return "", templateFailure("existing_catalog_unknown", "cluster_decode", err) + } + for _, c := range cluster.Catalogs { + if c.Name == "" || c.URL == "" || c.Name == s.cfg.ModelTemplates.WithDefaults().CatalogName { + return "", templateFailure("existing_catalog_unknown", "cluster_catalogs", errors.New("cluster catalog information is incomplete or conflicts")) + } + entries.Content = append(entries.Content, catalogNode(c.Name, c.URL, c.Default || len(cluster.Catalogs) == 1)) + } + if len(entries.Content) == 0 { + if cluster.Catalog == "" { + return "", templateFailure("existing_catalog_unknown", "cluster_catalogs", errors.New("cluster catalog information is incomplete or conflicts")) + } + entries.Content = append(entries.Content, catalogNode("default", cluster.Catalog, true)) + } + } + setYAMLField(root, "catalogs", entries) + removeYAMLField(root, "catalog") + } + replacement := catalogNode(s.cfg.ModelTemplates.WithDefaults().CatalogName, s.templateCatalogURL(generation), false) + if own == nil { + for _, entry := range entries.Content { + if name := yamlField(entry, "name"); name != nil && name.Value == s.cfg.ModelTemplates.WithDefaults().CatalogName { + return "", templateFailure("catalog_name_conflict", "catalog_ownership", errors.New("catalog name belongs to another source or is duplicated")) + } + } + } + if own == nil { + if len(entries.Content) == 1 { + value := &yaml.Node{} + _ = value.Encode(true) + setYAMLField(entries.Content[0], "default", value) + } + entries.Content = append(entries.Content, replacement) + } else { + yamlField(own, "url").Value = s.templateCatalogURL(generation) + } + var out bytes.Buffer + enc := yaml.NewEncoder(&out) + enc.SetIndent(2) + if err := enc.Encode(profile); err != nil { + return "", templateFailure("profile_invalid", "profile_encode", err) + } + _ = enc.Close() + return out.String(), nil +} + +func (s *Server) publishTemplates(ctx context.Context, r *http.Request) (*int64, error) { + select { + case s.templatePublishGate <- struct{}{}: + defer func() { <-s.templatePublishGate }() + case <-ctx.Done(): + return nil, templateFailure("registration_timeout", "publication_gate", ctx.Err()) + } + store := s.templateStore() + for attempt := 0; attempt < 3; attempt++ { + state, err := store.Read(ctx) + if err != nil { + return nil, templateFailure("storage_unavailable", "state_read", err) + } + before, profile, err := s.templateProfile(ctx, r) + if err != nil { + return nil, err + } + own, err := s.templateProfileEntry(profile) + if err != nil { + return nil, err + } + if own != nil { + generation, _ := s.templateGeneration(yamlField(own, "url").Value) + if generation == state.Generation { + verified, err := s.templateRegistration(ctx, r, state) + if err == nil { + return verified, nil + } + } + } + desired, err := s.templateUpdatedProfile(ctx, r, profile, state.Generation) + if err != nil { + return nil, err + } + latest, err := store.Read(ctx) + if err != nil { + return nil, templateFailure("storage_unavailable", "state_read", err) + } + if latest.Generation != state.Generation { + continue + } + current, _, err := s.templateProfile(ctx, r) + if err != nil { + return nil, err + } + if current != before { + continue + } + if _, err := s.templateRequest(ctx, r, http.MethodPut, "profile", map[string]string{"yaml": desired}); err != nil { + return nil, err + } + latest, err = store.Read(ctx) + if err != nil { + return nil, templateFailure("storage_unavailable", "state_read", err) + } + if latest.Generation != state.Generation { + continue + } + return s.templateRegistration(ctx, r, latest) + } + return nil, templateFailure("registration_changed", "publication_reconcile", errors.New("profile or state changed during each publication attempt")) +} diff --git a/internal/server/model_template_review_test.go b/internal/server/model_template_review_test.go new file mode 100644 index 0000000..20a1f07 --- /dev/null +++ b/internal/server/model_template_review_test.go @@ -0,0 +1,243 @@ +package server + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "io" + "log/slog" + "net" + "net/http" + "net/http/httptest" + "net/url" + "strings" + "testing" + "time" + + "github.com/modelsphere/console/internal/iam" + "gopkg.in/yaml.v3" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + k8stesting "k8s.io/client-go/testing" +) + +func TestTemplatePublicationPreservesLocalDefaultAndCatalogNode(t *testing.T) { + srv, _, swiss, h := newTemplateTestServer(t) + if rec := do(h, "POST", "/api/model-templates", "", templateBody("first")); rec.Code != 201 { + t.Fatal(rec.Body.String()) + } + swiss.mu.Lock() + var profile yaml.Node + if err := yaml.Unmarshal([]byte(swiss.profile), &profile); err != nil { + swiss.mu.Unlock() + t.Fatal(err) + } + own, err := srv.templateProfileEntry(&profile) + if err != nil { + swiss.mu.Unlock() + t.Fatal(err) + } + entries := yamlField(profile.Content[0], "catalogs") + for _, entry := range entries.Content { + value := &yaml.Node{} + _ = value.Encode(entry == own) + setYAMLField(entry, "default", value) + } + own.HeadComment = "operator selected this catalog" + location := yamlField(own, "url") + location.LineComment = "keep URL comment" + location.Style = yaml.DoubleQuotedStyle + setYAMLField(own, "futureOption", &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: "retained"}) + raw, err := yaml.Marshal(&profile) + if err != nil { + swiss.mu.Unlock() + t.Fatal(err) + } + swiss.profile = string(raw) + swiss.mu.Unlock() + assertCatalog := func(generation int64) { + t.Helper() + swiss.mu.Lock() + raw := swiss.profile + swiss.mu.Unlock() + var doc yaml.Node + if err := yaml.Unmarshal([]byte(raw), &doc); err != nil { + t.Fatal(err) + } + entry, err := srv.templateProfileEntry(&doc) + if err != nil { + t.Fatal(err) + } + if entry == nil || yamlField(entry, "default").Value != "true" || yamlField(entry, "futureOption").Value != "retained" || !strings.Contains(raw, "operator selected this catalog") || !strings.Contains(raw, "keep URL comment") { + t.Fatalf("local preference/fields/comments lost: %s", raw) + } + location := yamlField(entry, "url") + if location.Value != srv.templateCatalogURL(generation) || location.Style != yaml.DoubleQuotedStyle { + t.Fatalf("URL not updated in place: %s", raw) + } + for _, other := range yamlField(doc.Content[0], "catalogs").Content { + if other != entry && yamlField(other, "default").Value != "false" { + t.Fatalf("original default unexpectedly changed: %s", raw) + } + } + } + if rec := do(h, "POST", "/api/model-templates", "", templateBody("second")); rec.Code != 201 { + t.Fatalf("second create %d %s", rec.Code, rec.Body.String()) + } + assertCatalog(2) + swiss.mu.Lock() + swiss.unavailable = true + swiss.mu.Unlock() + if rec := do(h, "POST", "/api/model-templates", "", templateBody("third")); rec.Code != 202 { + t.Fatalf("pending create %d %s", rec.Code, rec.Body.String()) + } + swiss.mu.Lock() + swiss.unavailable = false + swiss.mu.Unlock() + if rec := do(h, "POST", "/api/model-templates/publish", "", "{}"); rec.Code != 200 { + t.Fatalf("retry %d %s", rec.Code, rec.Body.String()) + } + assertCatalog(3) +} + +func templateReviewRequest(t *testing.T) *http.Request { + t.Helper() + r := httptest.NewRequest("GET", "/api/model-templates", nil) + return r.WithContext(withIdentity(t.Context(), &iam.Identity{Name: "admin", Groups: []string{"system:masters"}})) +} +func templateReviewLog(srv *Server) *bytes.Buffer { + buf := &bytes.Buffer{} + srv.log = slog.New(slog.NewJSONHandler(buf, nil)) + return buf +} +func assertTemplateDiagnosticsSafe(t *testing.T, srv *Server, err error, reason string, private ...string) templateRegistrationDiagnostic { + t.Helper() + buf := templateReviewLog(srv) + if got := srv.reportTemplateFailure(t.Context(), err); got != reason { + t.Fatalf("reason=%s want=%s", got, reason) + } + if err.Error() != reason && reason != "registration_failed" { + t.Fatalf("error string is not a safe reason: %s", err.Error()) + } + for _, secret := range private { + if strings.Contains(buf.String(), secret) { + t.Fatalf("private error context leaked to diagnostics: %s", buf.String()) + } + } + var line map[string]any + if json.Unmarshal(buf.Bytes(), &line) != nil || line["reason"] != reason || line["cause_type"] == nil || line["operation"] == nil { + t.Fatalf("missing safe diagnostics: %s", buf.String()) + } + return templateFailureDiagnostic(err) +} + +func TestTemplateRegistrationRetainsNetworkCauseSafely(t *testing.T) { + srv, _, _, _ := newTemplateTestServer(t) + cause := &net.DNSError{Err: "private lookup details", Name: "private-template.example", IsTimeout: true} + srv.targets["swiss"] = func(context.Context) (*backendTarget, error) { return nil, cause } + _, err := srv.templateRequest(t.Context(), templateReviewRequest(t), "GET", "profile", nil) + if !errors.Is(err, cause) { + t.Fatal("network resolver cause was discarded") + } + var network *net.DNSError + if !errors.As(err, &network) { + t.Fatal("network error type was discarded") + } + diagnostic := assertTemplateDiagnosticsSafe(t, srv, err, "backend_unavailable", cause.Name, cause.Err) + if !diagnostic.timeout || diagnostic.operation != "backend_resolve" { + t.Fatalf("network diagnostics=%+v", diagnostic) + } +} + +func TestTemplateRegistrationRetainsJSONAndYAMLParseCauses(t *testing.T) { + t.Run("JSON", func(t *testing.T) { + srv, _, _, _ := newTemplateTestServer(t) + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = io.WriteString(w, `{"private-body-value":`) })) + defer upstream.Close() + target, _ := url.Parse(upstream.URL + "/api") + target.User = url.UserPassword("private-user", "private-password") + srv.targets["swiss"] = func(context.Context) (*backendTarget, error) { + return &backendTarget{url: *target, header: "Authorization", key: "Bearer private-backend-token"}, nil + } + _, err := srv.templateRequest(t.Context(), templateReviewRequest(t), "GET", "profile", nil) + var syntax *json.SyntaxError + if !errors.As(err, &syntax) { + t.Fatalf("JSON cause lost: %T", errors.Unwrap(err)) + } + assertTemplateDiagnosticsSafe(t, srv, err, "backend_response_invalid", upstream.URL, "private-user", "private-password", "private-backend-token", "private-body-value") + }) + t.Run("YAML", func(t *testing.T) { + srv, _, swiss, _ := newTemplateTestServer(t) + swiss.mu.Lock() + swiss.profile = originalTemplateProfile + "name: private-profile-value\n" + swiss.mu.Unlock() + _, _, err := srv.templateProfile(t.Context(), templateReviewRequest(t)) + var parse *yaml.TypeError + if !errors.As(err, &parse) { + t.Fatalf("YAML cause lost: %T", errors.Unwrap(err)) + } + assertTemplateDiagnosticsSafe(t, srv, err, "profile_invalid", "private-profile-value", originalTemplateProfile) + }) +} + +func TestTemplateRegistrationHTTPStatusDiagnosticsAndPublicMapping(t *testing.T) { + srv, _, swiss, h := newTemplateTestServer(t) + swiss.mu.Lock() + swiss.unavailable = true + swiss.mu.Unlock() + _, err := srv.templateRequest(t.Context(), templateReviewRequest(t), "GET", "profile", nil) + var status *templateHTTPStatusError + if !errors.As(err, &status) || status.status != 503 { + t.Fatal("HTTP status context was discarded") + } + diagnostic := assertTemplateDiagnosticsSafe(t, srv, err, "backend_rejected", "upstream secret values") + if diagnostic.status != 503 || diagnostic.timeout { + t.Fatalf("HTTP diagnostics=%+v", diagnostic) + } + buf := templateReviewLog(srv) + rec := do(h, "POST", "/api/model-templates", "", templateBody("first")) + result := decodeTemplateResult(t, rec) + if rec.Code != 202 || result["reason"] != "backend_rejected" || result["created"] != true { + t.Fatalf("public mapping %d %s", rec.Code, rec.Body.String()) + } + if strings.Contains(rec.Body.String(), "upstream secret values") || strings.Contains(buf.String(), "upstream secret values") { + t.Fatal("upstream error body leaked") + } +} + +func TestTemplatePublicationRetainsTimeoutAndStorageCauses(t *testing.T) { + t.Run("publication timeout", func(t *testing.T) { + srv, _, _, _ := newTemplateTestServer(t) + srv.templatePublishGate <- struct{}{} + defer func() { <-srv.templatePublishGate }() + ctx, cancel := context.WithDeadline(t.Context(), time.Now().Add(-time.Second)) + defer cancel() + _, err := srv.publishTemplates(ctx, templateReviewRequest(t)) + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatal("publication cancellation cause lost") + } + if !assertTemplateDiagnosticsSafe(t, srv, err, "registration_timeout").timeout { + t.Fatal("publication timeout not diagnosed") + } + }) + t.Run("cluster status", func(t *testing.T) { + srv, dyn, _, _ := newTemplateTestServer(t) + cause := apierrors.NewForbidden(schema.GroupResource{Resource: "configmaps"}, "templates", errors.New("private cluster denial")) + dyn.PrependReactor("get", "configmaps", func(k8stesting.Action) (bool, runtime.Object, error) { return true, nil, cause }) + _, err := srv.publishTemplates(t.Context(), templateReviewRequest(t)) + if !errors.Is(err, cause) { + t.Fatal("storage cause lost") + } + diagnostic := assertTemplateDiagnosticsSafe(t, srv, err, "storage_unavailable", "private cluster denial") + if diagnostic.status != 403 || diagnostic.operation != "state_read" { + t.Fatalf("cluster diagnostics=%+v", diagnostic) + } + }) +} + +func TestUnknownTemplateRegistrationErrorCannotBecomeWireText(t *testing.T) { + srv, _, _, _ := newTemplateTestServer(t) + assertTemplateDiagnosticsSafe(t, srv, errors.New("http://private-user:private-token@example/private-body"), "registration_failed", "private-user", "private-token", "private-body") +} diff --git a/internal/server/model_templates.go b/internal/server/model_templates.go new file mode 100644 index 0000000..00e6f31 --- /dev/null +++ b/internal/server/model_templates.go @@ -0,0 +1,253 @@ +package server + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "io" + "net/http" + "strconv" + "time" + + "github.com/modelsphere/console/internal/modeltemplate" + apierrors "k8s.io/apimachinery/pkg/api/errors" +) + +func (s *Server) templateStore() *modeltemplate.Store { + m := s.cfg.ModelTemplates.WithDefaults() + return modeltemplate.NewStore(s.kube.Dynamic(), m.Namespace, m.ConfigMap) +} +func (s *Server) mountModelTemplates(mux *http.ServeMux) { + mux.HandleFunc("GET /api/model-templates", s.handleModelTemplates) + mux.HandleFunc("POST /api/model-templates/check", s.handleModelTemplateCheck) + mux.HandleFunc("POST /api/model-templates", s.handleCreateModelTemplate) + mux.HandleFunc("POST /api/model-templates/publish", s.handlePublishModelTemplates) +} +func (s *Server) TemplateCatalogHandler() http.Handler { + mux := http.NewServeMux() + s.mountTemplateCatalog(mux) + mux.HandleFunc("/", http.NotFound) + return s.recover(s.logRequests(mux)) +} + +func (s *Server) mountTemplateCatalog(mux *http.ServeMux) { + mux.HandleFunc("GET /catalog/console/g/{generation}/index.json", s.handleTemplateCatalog) + mux.HandleFunc("GET /catalog/console/g/{generation}/models/{model}/{file}", s.handleTemplateCatalog) + mux.HandleFunc("/catalog/console/", func(w http.ResponseWriter, r *http.Request) { http.NotFound(w, r) }) +} + +func (s *Server) authorizeTemplates(w http.ResponseWriter, r *http.Request, verb string) bool { + w.Header().Set("Cache-Control", "no-store") + m := s.cfg.ModelTemplates.WithDefaults() + if !s.authorize(w, r, verb, "modeltemplates", m.CatalogName) || !s.authorize(w, r, "get", backendResource, m.Backend) { + return false + } + if verb != "get" && !s.authorize(w, r, "update", backendResource, m.Backend) { + return false + } + return true +} +func (s *Server) enabledTemplates(w http.ResponseWriter) bool { + if !s.cfg.ModelTemplates.Enabled { + writeError(w, http.StatusNotFound, "model templates are disabled") + return false + } + return s.haveKube(w) +} +func templateInput(w http.ResponseWriter, r *http.Request) (modeltemplate.Input, *modeltemplate.Recipe, bool) { + var in modeltemplate.Input + _ = http.NewResponseController(w).SetReadDeadline(time.Now().Add(10 * time.Second)) + defer http.NewResponseController(w).SetReadDeadline(time.Time{}) + r.Body = http.MaxBytesReader(w, r.Body, 192<<10) + dec := json.NewDecoder(r.Body) + dec.DisallowUnknownFields() + if dec.Decode(&in) != nil || dec.Decode(new(any)) != io.EOF { + writeError(w, http.StatusBadRequest, "invalid template request") + return in, nil, false + } + recipe, err := modeltemplate.Validate(in) + if err != nil { + writeError(w, http.StatusBadRequest, err.Error()) + return in, nil, false + } + return in, recipe, true +} +func (s *Server) handleModelTemplates(w http.ResponseWriter, r *http.Request) { + if !s.authorizeTemplates(w, r, "get") { + return + } + m := s.cfg.ModelTemplates.WithDefaults() + ctx, cancel := context.WithTimeout(r.Context(), templateOperationTimeout) + defer cancel() + state := modeltemplate.Empty() + if m.Enabled { + if !s.haveKube(w) { + return + } + var err error + state, err = s.templateStore().Read(ctx) + if err != nil { + writeTemplateStorageError(w, err) + return + } + } + index, err := state.Index(state.Generation) + if err != nil { + writeTemplateStorageError(w, err) + return + } + var registered *int64 + reason := "" + if m.Enabled { + var registrationErr error + registered, registrationErr = s.templateRegistration(ctx, r, state) + reason = s.reportTemplateFailure(ctx, registrationErr) + } + generations := map[string]int64{} + for _, record := range state.Records { + generations[record.Metadata.Name] = record.CreationGen + } + result := map[string]any{"enabled": m.Enabled, "catalogName": m.CatalogName, "catalogURL": s.templateCatalogURL(state.Generation), "generation": state.Generation, "index": index, "registeredGeneration": registered, "modelGenerations": generations} + if reason != "" { + result["registrationError"] = reason + } + writeJSON(w, http.StatusOK, result) +} +func (s *Server) handleModelTemplateCheck(w http.ResponseWriter, r *http.Request) { + if !s.authorizeTemplates(w, r, "create") || !s.enabledTemplates(w) { + return + } + _, recipe, ok := templateInput(w, r) + if !ok { + return + } + writeJSON(w, http.StatusOK, map[string]any{"valid": true, "model": recipe.Name, "version": recipe.Version, "variantCount": len(recipe.Variants)}) +} +func (s *Server) handleCreateModelTemplate(w http.ResponseWriter, r *http.Request) { + if !s.authorizeTemplates(w, r, "create") || !s.enabledTemplates(w) { + return + } + in, recipe, ok := templateInput(w, r) + if !ok { + return + } + ctx, cancel := context.WithTimeout(r.Context(), templateOperationTimeout) + defer cancel() + state, err := s.templateStore().Create(ctx, in, identityFrom(r.Context()).Name) + if err != nil { + writeTemplateStorageError(w, err) + return + } + registered, registrationErr := s.publishTemplates(ctx, r) + reason := s.reportTemplateFailure(ctx, registrationErr) + ready := registered != nil && *registered >= state.Generation && reason == "" + status := http.StatusAccepted + if ready { + status = http.StatusCreated + } + result := s.templatePublishResult(state.Generation, registered, reason) + result["created"] = true + result["model"] = recipe.Name + result["version"] = recipe.Version + writeJSON(w, status, result) +} +func (s *Server) templatePublishResult(generation int64, registered *int64, reason string) map[string]any { + result := map[string]any{"catalogName": s.cfg.ModelTemplates.WithDefaults().CatalogName, "generation": generation, "catalogURL": s.templateCatalogURL(generation), "ready": registered != nil && *registered >= generation && reason == "", "registeredGeneration": registered} + if reason != "" { + result["reason"] = reason + } + return result +} +func (s *Server) handlePublishModelTemplates(w http.ResponseWriter, r *http.Request) { + if !s.authorizeTemplates(w, r, "update") || !s.enabledTemplates(w) { + return + } + _ = http.NewResponseController(w).SetReadDeadline(time.Now().Add(10 * time.Second)) + defer http.NewResponseController(w).SetReadDeadline(time.Time{}) + r.Body = http.MaxBytesReader(w, r.Body, 1024) + raw, err := io.ReadAll(r.Body) + var payload map[string]json.RawMessage + if err != nil || len(bytes.TrimSpace(raw)) > 0 && (json.Unmarshal(raw, &payload) != nil || len(payload) != 0) { + writeError(w, http.StatusBadRequest, "publish accepts no template payload") + return + } + ctx, cancel := context.WithTimeout(r.Context(), templateOperationTimeout) + defer cancel() + registered, registrationErr := s.publishTemplates(ctx, r) + reason := s.reportTemplateFailure(ctx, registrationErr) + state, err := s.templateStore().Read(ctx) + if err != nil { + writeTemplateStorageError(w, err) + return + } + status := http.StatusOK + if registered == nil || *registered < state.Generation || reason != "" { + status = http.StatusAccepted + } + writeJSON(w, status, s.templatePublishResult(state.Generation, registered, reason)) +} +func (s *Server) handleTemplateCatalog(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Cache-Control", "no-store") + if !s.cfg.ModelTemplates.Enabled { + http.NotFound(w, r) + return + } + digits := r.PathValue("generation") + generation, err := strconv.ParseInt(digits, 10, 64) + if err != nil || generation < 0 || generation > 1000 || strconv.FormatInt(generation, 10) != digits { + http.NotFound(w, r) + return + } + if s.kube == nil { + writeError(w, http.StatusServiceUnavailable, "template storage is unavailable") + return + } + ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second) + defer cancel() + state, err := s.templateStore().Read(ctx) + if err != nil { + writeError(w, http.StatusServiceUnavailable, "template storage is unavailable") + return + } + var body []byte + if r.PathValue("model") == "" { + index, err := state.Index(generation) + if err != nil { + http.NotFound(w, r) + return + } + body, err = json.Marshal(index) + if err != nil { + writeError(w, http.StatusInternalServerError, "cannot encode catalog") + return + } + w.Header().Set("Content-Type", "application/json") + } else { + var ok bool + body, ok = state.Recipe(generation, r.PathValue("model"), r.PathValue("file")) + if !ok { + http.NotFound(w, r) + return + } + w.Header().Set("Content-Type", "application/yaml") + } + if len(body) > 2<<20 { + writeError(w, http.StatusServiceUnavailable, "catalog exceeds response limit") + return + } + w.Header().Set("Content-Length", strconv.Itoa(len(body))) + if r.Method != http.MethodHead { + _, _ = w.Write(body) + } +} +func writeTemplateStorageError(w http.ResponseWriter, err error) { + switch { + case errors.Is(err, modeltemplate.ErrDuplicate), apierrors.IsConflict(err): + writeError(w, http.StatusConflict, "template name or storage revision conflicts; refresh and retry") + case errors.Is(err, modeltemplate.ErrTooLarge): + writeError(w, http.StatusRequestEntityTooLarge, err.Error()) + default: + writeError(w, http.StatusBadGateway, "model template storage is unavailable or invalid") + } +} diff --git a/internal/server/model_templates_test.go b/internal/server/model_templates_test.go new file mode 100644 index 0000000..5d490a4 --- /dev/null +++ b/internal/server/model_templates_test.go @@ -0,0 +1,483 @@ +package server + +import ( + "encoding/json" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "testing/fstest" + + "github.com/modelsphere/console/internal/cluster" + "github.com/modelsphere/console/internal/config" + "github.com/modelsphere/console/internal/iam" + "github.com/modelsphere/console/internal/modeltemplate" + "gopkg.in/yaml.v3" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + dynamicfake "k8s.io/client-go/dynamic/fake" +) + +var templateMapsGVR = schema.GroupVersionResource{Version: "v1", Resource: "configmaps"} + +const originalTemplateProfile = "# keep operator comment\nname: cluster\nnamespace: models\nmodel:\n pathTemplate: /models/{{hf}}\nchartRepo: https://charts.example\nextra:\n marker: preserve\n" + +type templateSwiss struct { + mu sync.Mutex // Guards profile, response controls, counters and hooks during requests. + profile string + unavailable bool + puts int + stale bool + badBody bool + authorization string + profileGets int + changeOnRecheck bool + onPut func() + redirect string + oversized bool + modern bool + modernChange func(map[string]any) +} + +func (f *templateSwiss) serve(w http.ResponseWriter, r *http.Request) { + f.mu.Lock() + defer f.mu.Unlock() + if f.redirect != "" { + w.Header().Set("Location", f.redirect) + http.Error(w, "private redirect body", 302) + return + } + if f.oversized { + _, _ = w.Write([]byte(strings.Repeat("x", maxTemplateUpstreamBytes+1))) + return + } + if f.unavailable { + http.Error(w, "upstream secret values", http.StatusServiceUnavailable) + return + } + if f.authorization != "" && r.Header.Get("Authorization") != f.authorization { + http.Error(w, "wrong token", 401) + return + } + if r.Header.Get("Cookie") != "" || r.Header.Get("X-Remote-User") == "" { + http.Error(w, "missing proxy auth", 401) + return + } + switch { + case r.Method == "GET" && r.URL.Path == "/api/profile": + f.profileGets++ + if f.changeOnRecheck && f.profileGets == 2 { + f.profile += "# concurrent administrator edit\n" + } + writeJSON(w, 200, map[string]any{"yaml": f.profile}) + case r.Method == "PUT" && r.URL.Path == "/api/profile": + var req struct { + YAML string `json:"yaml"` + } + if json.NewDecoder(r.Body).Decode(&req) != nil { + http.Error(w, "invalid body", 400) + return + } + var profile struct { + Catalogs []struct { + Name string `yaml:"name"` + URL string `yaml:"url"` + Default bool `yaml:"default"` + } `yaml:"catalogs"` + } + if yaml.Unmarshal([]byte(req.YAML), &profile) != nil { + http.Error(w, "invalid profile", 400) + return + } + if strings.Contains(req.YAML, "writable:") || strings.Contains(req.YAML, "schemaVersion:") { + http.Error(w, "legacy parser rejected", 400) + return + } + f.profile = req.YAML + f.puts++ + if f.onPut != nil { + f.onPut() + } + writeJSON(w, 200, map[string]any{"yaml": f.profile}) + case r.Method == "GET" && r.URL.Path == "/api/cluster": + writeJSON(w, 200, map[string]any{"catalog": "http://original.example/index.json", "catalogFrom": "config", "catalogs": []any{map[string]any{"name": "default", "url": "http://original.example/index.json"}}}) + case r.Method == "GET" && r.URL.Path == "/api/catalog": + var profile struct { + Catalogs []struct { + Name string `yaml:"name"` + URL string `yaml:"url"` + } `yaml:"catalogs"` + } + _ = yaml.Unmarshal([]byte(f.profile), &profile) + for _, repo := range profile.Catalogs { + if repo.Name == r.URL.Query().Get("catalog") { + response, err := http.Get(repo.URL) + if err != nil { + http.Error(w, "cannot read provider", 502) + return + } + defer response.Body.Close() + raw, _ := io.ReadAll(response.Body) + source := repo.URL + if f.stale { + source = "http://stale.example/index.json" + } + if f.badBody { + raw = []byte(`{"apiVersion":"catalog.swiss/v1","count":0,"models":[]}`) + } + if f.modern { + var index modeltemplate.Index + if json.Unmarshal(raw, &index) != nil { + http.Error(w, "invalid provider index", 502) + return + } + envelope := modernTemplateEnvelope(index) + envelope["name"] = repo.Name + envelope["source"] = source + if f.modernChange != nil { + f.modernChange(envelope) + } + writeJSON(w, 200, envelope) + } else { + writeJSON(w, 200, map[string]any{"name": repo.Name, "source": source, "index": json.RawMessage(raw)}) + } + return + } + } + http.Error(w, "unknown catalog", 404) + default: + http.NotFound(w, r) + } +} + +func templateTestInput(name string) modeltemplate.Input { + return modeltemplate.Input{Metadata: modeltemplate.Metadata{Name: name, DisplayName: name, Source: modeltemplate.Source{HF: "org/model"}}, RecipeYAML: "# exact recipe\napiVersion: catalog.swiss/v1\nname: " + name + "\nversion: 1.0.0\nvariants:\n - id: vllm-gpu\n engine: vllm\n chart: {name: vllm, version: 0.7.1}\n image: {repository: example/vllm, tag: stable}\n requires: {gpus: 1}\n values: {extraArgs: [--trust-remote-code]}\n"} +} +func templateBody(name string) string { + raw, _ := json.Marshal(templateTestInput(name)) + return string(raw) +} +func newTemplateTestServer(t *testing.T) (*Server, *dynamicfake.FakeDynamicClient, *templateSwiss, http.Handler) { + t.Helper() + swiss := &templateSwiss{profile: originalTemplateProfile} + upstream := httptest.NewServer(http.HandlerFunc(swiss.serve)) + t.Cleanup(upstream.Close) + cfg := &config.Config{Server: config.Server{Auth: config.Auth{Disabled: true}}, Backends: []config.Backend{{Name: "swiss", Prefix: "/api/deploy", URL: upstream.URL + "/api", APIKeyEnv: "TEMPLATE_TEST_KEY"}}, ModelTemplates: config.ModelTemplates{Enabled: true, Namespace: "console-ns", ConfigMap: "templates", PublishURL: "http://placeholder/catalog/console"}} + dyn := dynamicfake.NewSimpleDynamicClientWithCustomListKinds(runtime.NewScheme(), map[schema.GroupVersionResource]string{templateMapsGVR: "ConfigMapList", iam.RolesGVR: "IAMRoleList", iam.RoleBindingsGVR: "IAMRoleBindingList"}) + log := slog.New(slog.DiscardHandler) + srv := New(cfg, cluster.NewKubeFrom(dyn), log, "test") + store := iam.NewStore(dyn) + srv.SetIAM(store, nil, iam.NewAuthenticator(store, nil, log), iam.NewAuthorizer(store)) + public := httptest.NewServer(srv.TemplateCatalogHandler()) + t.Cleanup(public.Close) + cfg.ModelTemplates.PublishURL = public.URL + "/catalog/console" + return srv, dyn, swiss, srv.Handler() +} +func decodeTemplateResult(t *testing.T, rec *httptest.ResponseRecorder) map[string]any { + t.Helper() + var result map[string]any + if err := json.Unmarshal(rec.Body.Bytes(), &result); err != nil { + t.Fatal(err) + } + return result +} + +func TestModelTemplatesCreateRegistersLegacyAndPreservesSource(t *testing.T) { + t.Setenv("TEMPLATE_TEST_KEY", "backend-only") + srv, dyn, swiss, h := newTemplateTestServer(t) + swiss.authorization = "Bearer backend-only" + checked := do(h, "POST", "/api/model-templates/check", "caller", templateBody("first")) + if checked.Code != 200 || len(dyn.Actions()) != 0 || swiss.puts != 0 { + t.Fatalf("check %d %s actions=%v", checked.Code, checked.Body.String(), dyn.Actions()) + } + rec := do(h, "POST", "/api/model-templates", "caller", templateBody("first")) + if rec.Code != 201 { + t.Fatalf("create %d %s", rec.Code, rec.Body.String()) + } + result := decodeTemplateResult(t, rec) + if result["created"] != true || result["ready"] != true || result["generation"] != float64(1) || result["registeredGeneration"] != float64(1) || result["catalogName"] != "console-local" { + t.Fatalf("create result=%v", result) + } + if !strings.Contains(swiss.profile, "# keep operator comment") || !strings.Contains(swiss.profile, "marker: preserve") || !strings.Contains(swiss.profile, "http://original.example/index.json") || !strings.Contains(swiss.profile, "default: true") || !strings.Contains(swiss.profile, "default: false") { + t.Fatalf("profile lost settings: %s", swiss.profile) + } + list := do(h, "GET", "/api/model-templates", "", "") + if list.Code != 200 { + t.Fatal(list.Body.String()) + } + listed := decodeTemplateResult(t, list) + if listed["registeredGeneration"] != float64(1) || listed["modelGenerations"].(map[string]any)["first"] != float64(1) { + t.Fatalf("list=%v", listed) + } + for _, a := range dyn.Actions() { + if a.GetResource() != templateMapsGVR || a.GetNamespace() != "console-ns" { + t.Fatalf("wrong storage scope: %#v", a) + } + } + puts := swiss.puts + if duplicate := do(h, "POST", "/api/model-templates", "", templateBody("first")); duplicate.Code != 409 || swiss.puts != puts { + t.Fatalf("duplicate %d %s", duplicate.Code, duplicate.Body.String()) + } + if second := do(h, "POST", "/api/model-templates", "", templateBody("second")); second.Code != 201 { + t.Fatal(second.Body.String()) + } + old := do(srv.TemplateCatalogHandler(), "GET", "/catalog/console/g/1/index.json", "", "") + var idx modeltemplate.Index + _ = json.Unmarshal(old.Body.Bytes(), &idx) + if old.Code != 200 || idx.Count != 1 || idx.Models[0].Name != "first" { + t.Fatalf("old snapshot=%s", old.Body.String()) + } + current := do(srv.TemplateCatalogHandler(), "GET", "/catalog/console/g/2/index.json", "", "") + _ = json.Unmarshal(current.Body.Bytes(), &idx) + if idx.Count != 2 { + t.Fatalf("current=%s", current.Body.String()) + } + recipe := do(srv.TemplateCatalogHandler(), "GET", "/catalog/console/g/1/models/first/first-1.0.0.yaml", "", "") + if recipe.Code != 200 || recipe.Body.String() != templateTestInput("first").RecipeYAML { + t.Fatal("published recipe bytes differ") + } + if later := do(srv.TemplateCatalogHandler(), "GET", "/catalog/console/g/1/models/second/second-1.0.0.yaml", "", ""); later.Code != 404 { + t.Fatalf("future recipe visible: %d", later.Code) + } + head := do(srv.TemplateCatalogHandler(), "HEAD", "/catalog/console/g/1/index.json", "", "") + if head.Code != 200 || head.Body.Len() != 0 || head.Header().Get("Content-Length") == "" { + t.Fatalf("HEAD=%d %s", head.Code, head.Body.String()) + } +} + +func TestModelTemplateSavedPendingAndPublishRetry(t *testing.T) { + _, _, swiss, h := newTemplateTestServer(t) + swiss.unavailable = true + rec := do(h, "POST", "/api/model-templates", "", templateBody("first")) + if rec.Code != 202 { + t.Fatalf("pending %d %s", rec.Code, rec.Body.String()) + } + result := decodeTemplateResult(t, rec) + if result["created"] != true || result["ready"] != false || result["registeredGeneration"] != nil || strings.Contains(rec.Body.String(), "secret") { + t.Fatalf("pending=%v", result) + } + list := decodeTemplateResult(t, do(h, "GET", "/api/model-templates", "", "")) + if list["index"].(map[string]any)["count"] != float64(1) || list["registrationError"] == nil { + t.Fatalf("saved template hidden: %v", list) + } + swiss.unavailable = false + retry := do(h, "POST", "/api/model-templates/publish", "", "{}") + if retry.Code != 200 { + t.Fatalf("retry %d %s", retry.Code, retry.Body.String()) + } + result = decodeTemplateResult(t, retry) + if result["generation"] != float64(1) || result["ready"] != true { + t.Fatalf("retry=%v", result) + } +} + +func TestModelTemplatesRejectForeignCatalogAndVerifySwissRead(t *testing.T) { + for _, mode := range []string{"foreign", "stale source", "stale index"} { + t.Run(mode, func(t *testing.T) { + _, _, swiss, h := newTemplateTestServer(t) + switch mode { + case "foreign": + swiss.profile += "catalogs:\n - name: console-local\n url: http://foreign.example/index.json\n" + case "stale source": + swiss.stale = true + case "stale index": + swiss.badBody = true + } + rec := do(h, "POST", "/api/model-templates", "", templateBody("first")) + if rec.Code != 202 { + t.Fatalf("got %d %s", rec.Code, rec.Body.String()) + } + result := decodeTemplateResult(t, rec) + if result["ready"] != false || result["registeredGeneration"] != nil { + t.Fatalf("unverified ready=%v", result) + } + if mode == "foreign" && (swiss.puts != 0 || result["reason"] != "catalog_name_conflict") { + t.Fatalf("foreign overwritten=%v", result) + } + }) + } +} + +func TestModelTemplatesPreserveLegacyCatalogAndMultipleCatalogs(t *testing.T) { + for _, profile := range []string{originalTemplateProfile + "catalogs:\n - name: original\n url: http://original.example/index.json\n", originalTemplateProfile + "catalog: http://legacy.example/index.json\n", originalTemplateProfile + "catalogs:\n - name: original\n url: http://original.example/index.json\n default: true\n - name: other\n url: http://other.example/index.json\n"} { + _, _, swiss, h := newTemplateTestServer(t) + swiss.profile = profile + rec := do(h, "POST", "/api/model-templates", "", templateBody("first")) + if rec.Code != 201 { + t.Fatalf("got %d %s", rec.Code, rec.Body.String()) + } + if !strings.Contains(swiss.profile, "default: true") { + t.Fatalf("original implicit default lost: %s", swiss.profile) + } + if strings.Contains(profile, "legacy.example") && !strings.Contains(swiss.profile, "legacy.example") { + t.Fatal("legacy source lost") + } + if strings.Contains(profile, "other.example") && !strings.Contains(swiss.profile, "other.example") { + t.Fatal("second source lost") + } + } +} + +func TestModelTemplatesRejectInvalidInputWithoutPersistence(t *testing.T) { + for _, body := range []string{`{}`, templateBody("first") + `{}`, strings.Replace(templateBody("first"), `"metadata":`, `"namespace":"other","metadata":`, 1), `{"recipeYAML":"` + strings.Repeat("x", 192<<10) + `"}`} { + _, dyn, swiss, h := newTemplateTestServer(t) + rec := do(h, "POST", "/api/model-templates", "", body) + if rec.Code != 400 || len(dyn.Actions()) != 0 || swiss.puts != 0 { + t.Fatalf("got %d %s actions=%v", rec.Code, rec.Body.String(), dyn.Actions()) + } + } +} + +func TestModelTemplatePermissionsPrecedeStorage(t *testing.T) { + for _, mode := range []string{"ui only", "no backend update", "create", "publish only", "read only"} { + t.Run(mode, func(t *testing.T) { + srv, dyn, _, _ := newTemplateTestServer(t) + rules := []any{} + if mode != "ui only" { + verbs := []any{"get", "update"} + if mode == "no backend update" { + verbs = []any{"get"} + } + rules = append(rules, map[string]any{"apiGroups": []any{iam.Group}, "resources": []any{"backends"}, "resourceNames": []any{"swiss"}, "verbs": verbs}) + native := []any{"create"} + if mode == "publish only" { + native = []any{"update"} + } + if mode == "read only" { + native = []any{"get"} + } + rules = append(rules, map[string]any{"apiGroups": []any{iam.Group}, "resources": []any{"modeltemplates"}, "resourceNames": []any{"console-local"}, "verbs": native}) + } + role := globalObj("IAMRole", "template-writer", iam.ScopePlatform, iam.ScopeGlobal, map[string]any{"rules": rules, "uiPermissions": []any{"modeltemplates.create"}}) + if err := dyn.Tracker().Add(role); err != nil { + t.Fatal(err) + } + if err := dyn.Tracker().Add(globalBinding("template-admin", "template-writer", iam.ScopePlatform, iam.ScopeGlobal)); err != nil { + t.Fatal(err) + } + request := httptest.NewRequest("POST", "/api/model-templates", strings.NewReader(templateBody("first"))) + request = request.WithContext(withIdentity(request.Context(), &iam.Identity{Name: "admin"})) + rec := httptest.NewRecorder() + srv.handleCreateModelTemplate(rec, request) + if mode == "create" { + if rec.Code != 201 { + t.Fatalf("create %d %s", rec.Code, rec.Body.String()) + } + } else { + if rec.Code != 403 { + t.Fatalf("permissions %d %s", rec.Code, rec.Body.String()) + } + for _, a := range dyn.Actions() { + if a.GetResource() == templateMapsGVR { + t.Fatalf("unauthorized storage access %#v", a) + } + } + } + }) + } +} + +func TestTemplateRegistrationReconcilesConcurrentChanges(t *testing.T) { + srv, _, swiss, h := newTemplateTestServer(t) + swiss.changeOnRecheck = true + swiss.onPut = func() { + if swiss.puts == 1 { + if _, err := srv.templateStore().Create(t.Context(), templateTestInput("concurrent"), "other"); err != nil { + t.Error(err) + } + } + } + rec := do(h, "POST", "/api/model-templates", "", templateBody("first")) + if rec.Code != 201 { + t.Fatalf("reconcile %d %s", rec.Code, rec.Body.String()) + } + result := decodeTemplateResult(t, rec) + if result["registeredGeneration"] != float64(2) || !strings.Contains(swiss.profile, "# concurrent administrator edit") || !strings.Contains(swiss.profile, "/g/2/index.json") { + t.Fatalf("reconcile=%v profile=%s", result, swiss.profile) + } +} + +func TestTemplateRegistrationRejectsRedirectsAndLargeResponses(t *testing.T) { + for _, mode := range []string{"redirect", "oversized"} { + t.Run(mode, func(t *testing.T) { + followed := false + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + followed = true + http.Error(w, "private target body", 500) + })) + defer target.Close() + _, _, swiss, h := newTemplateTestServer(t) + if mode == "redirect" { + swiss.redirect = target.URL + } else { + swiss.oversized = true + } + rec := do(h, "POST", "/api/model-templates", "", templateBody("first")) + if rec.Code != 202 || followed || strings.Contains(rec.Body.String(), "private") { + t.Fatalf("upstream %d %s followed=%v", rec.Code, rec.Body.String(), followed) + } + }) + } +} + +func TestTemplateCatalogListenerBoundary(t *testing.T) { + srv, dyn, _, h := newTemplateTestServer(t) + if rec := do(h, "POST", "/api/model-templates", "", templateBody("first")); rec.Code != 201 { + t.Fatal(rec.Body.String()) + } + srv.cfg.Server.Auth.Disabled = false + srv.SetWeb(fstest.MapFS{"index.html": &fstest.MapFile{Data: []byte("Console")}}) + h = srv.Handler() + catalog := srv.TemplateCatalogHandler() + for _, path := range []string{"/catalog/console", "/catalog/console/", "/catalog/console/g/1/index.json", "/catalog/console/g/1/models/first/first-1.0.0.yaml"} { + for _, method := range []string{"GET", "HEAD"} { + if rec := do(h, method, path, "", ""); rec.Code != 404 { + t.Fatalf("web listener exposed %s %s: %d", method, path, rec.Code) + } + } + } + if rec := do(catalog, "GET", "/catalog/console/g/1/index.json", "", ""); rec.Code != 200 || !strings.Contains(rec.Body.String(), "first") { + t.Fatalf("internal catalog %d %s", rec.Code, rec.Body.String()) + } + if rec := do(catalog, "HEAD", "/catalog/console/g/1/models/first/first-1.0.0.yaml", "", ""); rec.Code != 200 || rec.Body.Len() != 0 { + t.Fatalf("internal HEAD %d %s", rec.Code, rec.Body.String()) + } + before := len(dyn.Actions()) + for _, request := range []struct{ method, path string }{{"GET", "/api/model-templates"}, {"POST", "/api/model-templates"}, {"POST", "/api/model-templates/check"}, {"POST", "/api/model-templates/publish"}, {"POST", "/oauth/token"}, {"GET", "/metrics"}, {"GET", "/healthz"}, {"POST", "/catalog/console/g/1/index.json"}} { + if rec := do(catalog, request.method, request.path, "", templateBody("other")); rec.Code != 404 { + t.Fatalf("catalog listener exposed %s %s: %d", request.method, request.path, rec.Code) + } + } + if len(dyn.Actions()) != before { + t.Fatal("noncatalog requests accessed template storage") + } + srv.cfg.ModelTemplates.Enabled = false + if rec := do(catalog, "GET", "/catalog/console/g/1/index.json", "", ""); rec.Code != 404 || len(dyn.Actions()) != before { + t.Fatal("disabled catalog listener read storage") + } +} + +func TestModelTemplateAuthenticationAndPublicBounds(t *testing.T) { + cfg := &config.Config{ModelTemplates: config.ModelTemplates{Enabled: true, Namespace: "console-ns", ConfigMap: "templates", PublishURL: "http://console/catalog/console"}} + srv := testServerWithConfig(t, cfg) + h := srv.Handler() + if rec := do(h, "POST", "/api/model-templates", "", templateBody("first")); rec.Code != 401 || rec.Header().Get("Cache-Control") != "no-store" { + t.Fatalf("auth=%d", rec.Code) + } + setRequirePasswordReset(t, srv, true) + token := login(t, h, "admin", "admin-pw") + if rec := do(h, "POST", "/api/model-templates", token, templateBody("first")); rec.Code != 403 { + t.Fatalf("password reset=%d", rec.Code) + } + if rec := do(srv.TemplateCatalogHandler(), "GET", "/catalog/console/g/0/index.json", "", ""); rec.Code != 200 { + t.Fatalf("anonymous static read=%d %s", rec.Code, rec.Body.String()) + } + for _, path := range []string{"/catalog/console/g/-1/index.json", "/catalog/console/g/01/index.json", "/catalog/console/g/1001/index.json", "/catalog/console/g/0/anything", "/catalog/console/g/0/models/first/not-a-version.yaml"} { + if rec := do(srv.TemplateCatalogHandler(), "GET", path, "", ""); rec.Code != 404 { + t.Fatalf("path %s status=%d", path, rec.Code) + } + } +} diff --git a/internal/server/server.go b/internal/server/server.go index bc69241..fe04dd6 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -40,8 +40,9 @@ type Server struct { registry *prometheus.Registry models *modelIndex - targetsOnce sync.Once - targets map[string]targetResolver + targetsOnce sync.Once + targets map[string]targetResolver + templatePublishGate chan struct{} draining atomic.Bool } @@ -49,7 +50,7 @@ type Server struct { func New(cfg *config.Config, kube *cluster.Kube, log *slog.Logger, version string) *Server { reg := prometheus.NewRegistry() reg.MustRegister(collectors.NewGoCollector(), collectors.NewProcessCollector(collectors.ProcessCollectorOpts{})) - return &Server{cfg: cfg, kube: kube, log: log, version: version, registry: reg, models: newModelIndex(log)} + return &Server{cfg: cfg, kube: kube, log: log, version: version, registry: reg, models: newModelIndex(log), templatePublishGate: make(chan struct{}, 1)} } // SetWeb installs the SPA filesystem. Without one, console is API only. @@ -92,6 +93,8 @@ func (s *Server) Handler() http.Handler { mux := http.NewServeMux() mux.HandleFunc("GET /healthz", s.handleHealthz) mux.HandleFunc("GET /readyz", s.handleReadyz) + mux.HandleFunc("/catalog/console", http.NotFound) + mux.HandleFunc("/catalog/console/", http.NotFound) if s.authn != nil { mux.HandleFunc("POST /oauth/token", s.handleToken) @@ -119,6 +122,7 @@ func (s *Server) Handler() http.Handler { // hence inside this block. s.mountBackends(mux) s.mountWorkload(mux) + s.mountModelTemplates(mux) if s.router != nil { s.mountRouter(mux) @@ -140,7 +144,8 @@ func (s *Server) Run(ctx context.Context) error { ReadHeaderTimeout: 10 * time.Second, IdleTimeout: 60 * time.Second, } - errc := make(chan error, 2) + defer srv.Close() + errc := make(chan error, 3) go func() { s.log.Info("console listening", "addr", s.cfg.Server.Addr, "version", s.version) if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed { @@ -157,6 +162,16 @@ func (s *Server) Run(ctx context.Context) error { }() defer metrics.Close() } + if addr := s.cfg.CatalogListenAddr(); addr != "off" { + catalog := &http.Server{Addr: addr, Handler: s.TemplateCatalogHandler(), ReadHeaderTimeout: 10 * time.Second, ReadTimeout: 10 * time.Second, WriteTimeout: 20 * time.Second, IdleTimeout: 60 * time.Second} + go func() { + s.log.Info("catalog listening", "addr", addr) + if err := catalog.ListenAndServe(); err != nil && err != http.ErrServerClosed { + errc <- err + } + }() + defer catalog.Close() + } select { case err := <-errc: return err @@ -217,6 +232,8 @@ type statusRecorder struct { status int } +func (r *statusRecorder) Unwrap() http.ResponseWriter { return r.ResponseWriter } + func (r *statusRecorder) WriteHeader(code int) { r.status = code r.ResponseWriter.WriteHeader(code) diff --git a/web/package-lock.json b/web/package-lock.json index c05f4d0..fdfee82 100644 --- a/web/package-lock.json +++ b/web/package-lock.json @@ -24,7 +24,8 @@ "recharts": "^3.10.1", "rehype-highlight": "^7.0.2", "remark-gfm": "^4.0.1", - "tailwind-merge": "^3.7.0" + "tailwind-merge": "^3.7.0", + "yaml": "^2.9.1" }, "devDependencies": { "@tailwindcss/typography": "^0.5.20", @@ -1417,6 +1418,7 @@ "cpu": [ "ppc64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1433,6 +1435,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1449,6 +1452,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1465,6 +1469,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1481,6 +1486,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1497,6 +1503,7 @@ "cpu": [ "arm" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1513,6 +1520,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1529,6 +1537,7 @@ "cpu": [ "loong64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1545,6 +1554,7 @@ "cpu": [ "mips64el" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1561,6 +1571,7 @@ "cpu": [ "ppc64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1577,6 +1588,7 @@ "cpu": [ "riscv64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1593,6 +1605,7 @@ "cpu": [ "s390x" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1609,6 +1622,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1625,6 +1639,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1641,6 +1656,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1657,6 +1673,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1673,6 +1690,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1689,6 +1707,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1705,6 +1724,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -1721,6 +1741,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "Apache-2.0", "optional": true, "os": [ @@ -5093,6 +5114,21 @@ "node": ">=20.11" } }, + "node_modules/yaml": { + "version": "2.9.1", + "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.9.1.tgz", + "integrity": "sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw==", + "license": "ISC", + "bin": { + "yaml": "bin.mjs" + }, + "engines": { + "node": ">= 14.6" + }, + "funding": { + "url": "https://github.com/sponsors/eemeli" + } + }, "node_modules/zwitch": { "version": "2.0.4", "resolved": "https://registry.npmjs.org/zwitch/-/zwitch-2.0.4.tgz", diff --git a/web/package.json b/web/package.json index 3d4c87b..7fd8a57 100644 --- a/web/package.json +++ b/web/package.json @@ -30,7 +30,8 @@ "recharts": "^3.10.1", "rehype-highlight": "^7.0.2", "remark-gfm": "^4.0.1", - "tailwind-merge": "^3.7.0" + "tailwind-merge": "^3.7.0", + "yaml": "^2.9.1" }, "devDependencies": { "@tailwindcss/typography": "^0.5.20", diff --git a/web/src/modules/docs/content/en-US/deploy.md b/web/src/modules/docs/content/en-US/deploy.md index 8aba883..ab1591a 100644 --- a/web/src/modules/docs/content/en-US/deploy.md +++ b/web/src/modules/docs/content/en-US/deploy.md @@ -78,6 +78,19 @@ From a row's actions in the list, or the top of the detail page: "Behind catalog" in the list means the catalog has a newer version to upgrade to; "Untracked" means the release was not deployed by Swiss and cannot be upgraded here. +## Add a custom model template + +Choose **New model template** in the [Model Library](/inferences/catalog) header. Fill in model identity, weights source and a deployment recipe. An existing recipe can prefill engine, image, GPU and runtime values; enter the new model's identity and weights source separately. + +1. Fill in model information, template version and recipe ID. +2. Select an exact chart version, image and hardware requirements, then review runtime YAML. Use Secret references for sensitive values. +3. **Check** validates the input; **Save template** persists it in Console's custom catalog and connects it to the deployment backend through the existing site API. +4. The default view shows the original models together with new **Custom** cards. Open a card and use the existing recipe-selection and diff-review deployment flow. + +Console owns persistence and preserves templates across chart upgrades. Existing catalogs and their default selection remain. Created versions cannot be overwritten; duplicate model names conflict. A saved template whose catalog connection is incomplete remains visible as **Catalog connection pending**. **Retry catalog connection** publishes it without creating it again; pending templates cannot deploy. + +An administrator must enable Console's storage/internal catalog service and grant template/backend permissions; the existing administrator wildcard works. Publication uses the legacy static catalog format and Helm deployment path. Writable external-catalog editing below is a separate entry point. + ## Edit a catalog's variants A catalog-server catalog can be edited from a model's page. In the [Site profile](/inferences/site-profile) editor, tick the catalog **Writable** and enter its catalog-server **Write token**; swissd keeps the token in its own Secret. Any later change to the catalog other than **Default** (its URL, its name, unticking **Writable**) removes the token: enter it again with the change. Saving checks the token with the catalog server first, a new catalog's too: if the server does not accept it or cannot be reached, nothing is saved and the editor says why. Each change is one commit to the catalog. diff --git a/web/src/modules/docs/content/zh-CN/deploy.md b/web/src/modules/docs/content/zh-CN/deploy.md index 8bed1ba..040e973 100644 --- a/web/src/modules/docs/content/zh-CN/deploy.md +++ b/web/src/modules/docs/content/zh-CN/deploy.md @@ -77,6 +77,19 @@ 列表上的「落后目录」表示目录里有更新的版本，可以升级；「未纳管」表示这个 release 不是通过 swiss 部署的，不能在这里升级。 +## 新增自定义模型模板 + +在 [模型库](/inferences/catalog) 标题栏点击 **新增模型模板**，填写模型标识、权重来源和部署方案。可选一个现有方案作为参考，预填引擎、镜像、GPU 和运行参数；新模型的标识与权重来源独立填写。 + +1. 填写模型信息、模板版本和方案 ID。 +2. 填写精确 chart 版本、镜像和硬件需求，检查运行参数 YAML；敏感信息使用 Secret 引用。 +3. **校验** 检查格式；**保存模板** 将模板保存到 Console 的自定义目录，并通过现有站点接口连接部署服务。 +4. 保存后，默认视图显示原目录模型和带 **自定义** 标识的新卡片。打开卡片，按既有方案选择和预览流程部署。 + +Console 自己持久化模板，chart 升级保留数据；原目录及默认选择保留。当前创建版本不可覆盖，同名模型返回冲突。若模板已保存但目录连接未完成，卡片显示 **待连接目录**；点击 **重试连接目录**，不会重复创建模板。未连接模板不能部署。 + +需要管理员启用 Console 的模板存储与内部目录服务，并授予模板及部署后端权限；现有管理员通配权限可用。当前发布格式兼容旧版静态目录，通过 Helm 路径部署。下面的可写外部目录操作是另一套目录编辑入口。 + ## 编辑目录中的变体 catalog-server 目录可以在模型详情页直接编辑。在 [站点配置](/inferences/site-profile) 的编辑器中把该目录勾选为 **可写**，并填入 catalog-server 的 **写入令牌**；令牌由 swissd 保存在它自己的 Secret 中。之后修改该目录除 **默认** 之外的任何设置（URL、名称、取消勾选 **可写**）都会移除令牌：修改时请重新输入。保存时会先向 catalog-server 校验令牌（新添加的目录也一样）：令牌不被接受或无法连接时，不会保存任何内容，编辑器会说明原因。每次修改都是目录的一次提交。 diff --git a/web/src/modules/inferences/CatalogPage.tsx b/web/src/modules/inferences/CatalogPage.tsx index e767711..a7654ea 100644 --- a/web/src/modules/inferences/CatalogPage.tsx +++ b/web/src/modules/inferences/CatalogPage.tsx @@ -1,6 +1,6 @@ import { useEffect, useMemo, useRef, useState } from "react"; import { Link, useNavigate, useSearchParams } from "react-router"; -import { useQuery, useQueryClient } from "@tanstack/react-query"; +import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; import { Alert, AlertDescription, @@ -23,8 +23,9 @@ import { cn, } from "@modelsphere/ui"; import { Boxes, Library, MoreHorizontal, TriangleAlert } from "lucide-react"; -import { useModulePath } from "@/shell"; -import { api, type CatalogInfo } from "@swiss/lib/api"; +import { useModulePath, usePermissions } from "@/shell"; +import type { CatalogInfo } from "@swiss/lib/api"; +import { api } from "@/modules/inferences/catalogApi"; import { facetsOf, formatUplift, sorters, type Facets } from "@swiss/lib/catalog"; import { useCatalogChoice } from "@swiss/components/CatalogChoice"; import { useT } from "@/modules/inferences/i18n"; @@ -32,6 +33,9 @@ import { DeploySheet, type DeployTarget } from "@/modules/inferences/components/ import { writableCatalogs } from "@/modules/inferences/catalog-write-lib"; import { SETTINGS_FILE } from "@/modules/inferences/catalog-write-api"; import { NewModelSheet } from "@/modules/inferences/components/NewModelSheet"; +import { NewTemplateSheet } from "@/modules/inferences/components/NewTemplateSheet"; +import { readLocalTemplates, publishTemplates, templateReady, type TemplateResult } from "@/modules/inferences/localTemplatesApi"; +import { libraryModels, ownsTemplateCatalog } from "@/modules/inferences/localTemplatesLib"; import { FileSheet } from "@/modules/inferences/components/FileSheet"; import { SetupSheet } from "@/modules/inferences/components/SetupSheet"; import { useCatalogWrite } from "@/modules/inferences/components/useCatalogWrite"; @@ -44,12 +48,17 @@ export function CatalogPage() { const p = useModulePath(); const [params, setParams] = useSearchParams(); const choice = useCatalogChoice(); + const { has } = usePermissions(); + const local = useQuery({ queryKey: ["local-model-templates"], queryFn: readLocalTemplates, enabled: has("modeltemplates.view") || has("modeltemplates.create"), retry: false }); + const configuredCatalog = choice.catalogs.find((candidate) => candidate.name === choice.selected); + const selectedLocal = ownsTemplateCatalog(local.data, configuredCatalog); const catalog = useQuery({ queryKey: ["catalog", choice.selected], queryFn: () => api.catalog(choice.selected), - enabled: !!choice.selected, + enabled: !!choice.selected && !selectedLocal, }); - const all = useMemo(() => catalog.data?.models.map(facetsOf) ?? [], [catalog.data]); + const entries = useMemo(() => libraryModels(catalog.data, local.data, choice.selected, choice.named, configuredCatalog), [catalog.data, local.data, choice.selected, choice.named, configuredCatalog]); + const all = useMemo(() => entries.map((entry) => ({ ...facetsOf(entry.model), catalog: entry.catalog, custom: entry.custom, ready: entry.ready })), [entries]); const options = useMemo(() => facetOptions(all), [all]); const f = filtersFrom(params); const set = (patch: Partial) => setParams((prev) => withFilters(prev, patch), { replace: true }); @@ -62,9 +71,20 @@ export function CatalogPage() { const profile = useQuery({ queryKey: ["profile"], queryFn: api.profile, retry: false }); const writable = writableCatalogs(profile.data?.profile); const w = useCatalogWrite(choice.selected); - const [sheet, setSheet] = useState<"model" | "settings" | "setup" | null>(null); + const [sheet, setSheet] = useState<"model" | "settings" | "setup" | "template" | null>(null); const qc = useQueryClient(); const navigate = useNavigate(); + const [notice, setNotice] = useState(""); + const refresh = () => { for (const key of [["local-model-templates"], ["cluster"], ["profile"], ["catalog"]]) void qc.invalidateQueries({ queryKey: key }); }; + const publish = useMutation({ mutationFn: publishTemplates, onSuccess: (result) => { setNotice(t(result.ready ? "localTemplates.connected" : "localTemplates.pending")); refresh(); } }); + const saved = (result: TemplateResult) => { + setSheet(null); + setNotice(t(result.ready ? "localTemplates.saved" : "localTemplates.savedPending", { name: result.model ?? "" })); + setParams((previous) => { const next = withoutFilters(previous); next.delete("catalog"); return next; }); + refresh(); + }; + const pending = local.data?.enabled && local.data.catalog.models.some((model) => !templateReady(local.data!, model.name)); + const sourceError = choice.error ?? (!selectedLocal ? catalog.error : null) ?? local.error; const copyFrom = catalog.data?.models.find((m) => m.variants.length); const onDeploy = cluster.data?.allowDeploy ? (model: string, catalog: string) => setDeploying({ kind: "deploy", model, catalog }) : undefined; @@ -79,25 +99,26 @@ export function CatalogPage() { return (
- } /> + } actions={has("modeltemplates.create") && } />
{choice.several && ( ({ - value: c.name, - label: c.name, - description: c.schema === "v1" ? t("catalog.legacy") : undefined, - badge: writable.includes(c.name) ? : undefined, - }))} - value={choice.selected} - onValueChange={(v) => v && choice.choose(v)} + title={local.data?.enabled && !choice.named ? t("localTemplates.defaultView") : t("catalog.catalog")} + options={[ + ...(local.data?.enabled ? [{ value: "__default", label: t("localTemplates.defaultView") }] : []), + ...choice.catalogs.map((c) => ({ value: c.name, label: c.name, description: c.schema === "v1" ? t("catalog.legacy") : undefined, badge: writable.includes(c.name) ? : undefined })), + ]} + value={local.data?.enabled ? choice.named : choice.selected} + onValueChange={(value) => { + if (local.data?.enabled && (value === "__default" || !value)) setParams((previous) => { const next = new URLSearchParams(previous); next.delete("catalog"); return next; }); + else if (value) choice.choose(value); + }} /> )} - {catalog.data?.schema === "v1" && ( + {catalog.data?.schema === "v1" && !selectedLocal && (!local.data?.enabled || !!choice.named) && ( {t("catalog.legacy")} @@ -137,7 +158,7 @@ export function CatalogPage() { {t("catalogWrite.settings")} )} @@ -162,28 +183,21 @@ export function CatalogPage() {
-
- {choice.isPending || catalog.isPending && !!choice.selected ? ( -
- -
- ) : choice.error || catalog.error ? ( - - - {String((choice.error ?? catalog.error)?.message)} - - ) : !choice.selected ? ( +
+ {notice && {notice}} + {sourceError && {sourceError.message} } + {pending && {t("localTemplates.pending")} {has("modeltemplates.create") && }} + {publish.error && {publish.error.message}} + {(choice.isPending || !selectedLocal && catalog.isPending && !!choice.selected) && !all.length ? ( +
+ ) : sourceError && !all.length ? null : !choice.selected && !all.length ? ( ) : shown.length === 0 ? ( - - - {t(filtered ? "catalog.noMatch" : "catalog.empty")} - - + {t(filtered ? "catalog.noMatch" : "catalog.empty")} ) : (
{shown.map((x) => ( - + ))}
)} @@ -191,6 +205,7 @@ export function CatalogPage() {
setDeploying(null)} /> + {sheet === "template" && entry.ready)} onClose={() => setSheet(null)} onSaved={saved} />} {sheet === "model" && ( onChange(e.target.value)} placeholder={placeholder} className="w-64" />; } -function ModelCard({ x, catalog, to, onDeploy }: { x: Facets; catalog: string; to: string; onDeploy?: (model: string, catalog: string) => void }) { +function ModelCard({ x, catalog, to, onDeploy, custom = false, pending = false }: { x: Facets; catalog: string; to: string; onDeploy?: (model: string, catalog: string) => void; custom?: boolean; pending?: boolean }) { const t = useT(); const navigate = useNavigate(); const m = x.model; - const open = () => navigate(to); + const open = () => { if (!pending) navigate(to); }; return (
(e.key === "Enter" || e.key === " ") && (e.preventDefault(), open())} className={cn( "group flex cursor-pointer flex-col gap-3 rounded-lg border bg-card p-4 transition-colors hover:border-primary/40 focus-visible:outline-2 focus-visible:outline-ring", x.deprecated && "opacity-70", + pending && "cursor-default hover:border-border", )} >
@@ -279,12 +295,12 @@ function ModelCard({ x, catalog, to, onDeploy }: { x: Facets; catalog: string; t
- e.stopPropagation()} className="block font-medium break-words hover:underline"> - {m.displayName || m.name} - + {pending ? {m.displayName || m.name} : e.stopPropagation()} className="block font-medium break-words hover:underline">{m.displayName || m.name}} {m.family &&
{m.family}
} + {custom && {t("localTemplates.custom")}} + {pending && {t("localTemplates.pendingLabel")}}
- + {!pending && } onClick={(e) => e.stopPropagation()}> @@ -292,7 +308,7 @@ function ModelCard({ x, catalog, to, onDeploy }: { x: Facets; catalog: string; t {t("actions.detail")} {onDeploy && onDeploy(m.name, catalog)}>{t("model.deploy")}} - + }
diff --git a/web/src/modules/inferences/ModelPage.tsx b/web/src/modules/inferences/ModelPage.tsx index 6c164d1..b9b31c4 100644 --- a/web/src/modules/inferences/ModelPage.tsx +++ b/web/src/modules/inferences/ModelPage.tsx @@ -24,7 +24,8 @@ import { } from "@modelsphere/ui"; import { MoreHorizontal, TriangleAlert } from "lucide-react"; import { useModulePath } from "@/shell"; -import { api, type ModelVariant, type Node } from "@swiss/lib/api"; +import type { ModelVariant, Node } from "@swiss/lib/api"; +import { api } from "@/modules/inferences/catalogApi"; import { comparison, formatUplift, httpLink, reportLink, sharedVersion, variantKind, workloadSummary, type Kind } from "@swiss/lib/catalog"; import { gpuModels, vendorLabel } from "@swiss/lib/gpu"; import { useCatalogChoice } from "@swiss/components/CatalogChoice"; diff --git a/web/src/modules/inferences/catalogApi.test.ts b/web/src/modules/inferences/catalogApi.test.ts new file mode 100644 index 0000000..3a6ea60 --- /dev/null +++ b/web/src/modules/inferences/catalogApi.test.ts @@ -0,0 +1,66 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { onUnauthorized } from "@swiss/lib/api"; +import { api, normalizeCatalog } from "@/modules/inferences/catalogApi"; + +const fetchMock = vi.fn(); +const variant = { id: "gpu", engine: "sglang", default: true, chart: { name: "sglang", version: "0.7.1" }, requires: { gpus: 2 } }; +const legacy = { name: "default", ref: "catalog-ref", source: "https://catalog.example/index.json", index: { + apiVersion: "catalog.swiss/v1", site: "https://catalog.example/", count: 1, models: [{ + name: "model", displayName: "Example model", source: { hf: "example/model" }, latest: "1.1.0", + versions: [{ version: "1.1.0", path: "models/model/model-1.1.0.yaml", digest: "sha256:new", variants: [variant] }, { version: "1.0.0", path: "models/model/model-1.0.0.yaml", digest: "sha256:old", variants: [variant] }], + tuning: [{ version: "1.1.0", baseline: "gpu", optimized: "tuned", uplift: 10 }], + }], +} }; +const json = (body: unknown, status = 200) => new Response(JSON.stringify(body), { status }); + +beforeEach(() => { + fetchMock.mockReset(); + vi.stubGlobal("fetch", fetchMock); + vi.stubGlobal("document", { cookie: "" }); +}); +afterEach(() => { vi.unstubAllGlobals(); onUnauthorized(null); }); + +describe("legacy and current catalog compatibility", () => { + it("reads the deployed index envelope without changing template identity or GPU/chart choices", async () => { + fetchMock.mockResolvedValue(json(legacy)); + const result = await api.catalog("default"); + expect(result).toMatchObject({ name: "default", ref: "catalog-ref", schema: "v1", source: legacy.source }); + expect(result.models[0]).toMatchObject({ name: "model", defaultVariant: "gpu", source: { hf: "example/model" }, variants: [{ ...variant, version: "1.1.0", digest: "sha256:new" }] }); + expect(result.models[0].tuning?.[0]).toMatchObject({ baseline: { id: "gpu", version: "1.1.0" }, optimized: { id: "tuned", version: "1.1.0" } }); + }); + + it("preserves the current normalized API response", () => { + const current = { name: "modern", ref: "ref", source: "https://catalog.example", schema: "v2", models: [] }; + expect(normalizeCatalog(current)).toBe(current); + }); + + it("loads the chosen legacy entry with its image, values, served name and site path", async () => { + fetchMock.mockResolvedValueOnce(json({ ref: "entry-ref", localPath: "/models/model", pathTemplate: "/models/{{name}}", imageRepository: { gpu: "mirror.example/engine" }, entry: { name: "model", version: "1.0.0", digest: "sha256:old", servedName: "client-name", variants: [{ ...variant, image: { repository: "example/engine", tag: "v1" }, values: { extraArgs: ["--custom"] } }] } })).mockResolvedValueOnce(json(legacy)); + const result = await api.model("model", { catalog: "default", variant: "gpu", version: "1.0.0" }); + expect(result).toMatchObject({ schema: "v1", localPath: "/models/model", imageRepository: { gpu: "mirror.example/engine" }, variant: { id: "gpu", version: "1.0.0", digest: "sha256:old", servedName: "client-name", image: { tag: "v1" }, values: { extraArgs: ["--custom"] } } }); + expect(result.model.variants[0].version).toBe("1.1.0"); + }); + + it("does not invent a selected variant absent from the legacy version", async () => { + fetchMock.mockResolvedValueOnce(json({ entry: { name: "model", version: "1.0.0", variants: [variant] } })).mockResolvedValueOnce(json(legacy)); + await expect(api.model("model", { variant: "missing" })).rejects.toThrow("variant"); + }); + + it("uses v1 versions when the deployed backend has no history endpoint", async () => { + fetchMock.mockResolvedValueOnce(json({ error: "not found" }, 404)).mockResolvedValueOnce(json(legacy)); + expect(await api.history("model", { catalog: "default", variant: "gpu" })).toMatchObject({ schema: "v1", model: "model", variant: "gpu", revisions: [{ version: "1.1.0", digest: "sha256:new", current: true }, { version: "1.0.0", digest: "sha256:old", current: false }] }); + }); + + it("does not hide auth failures as an old API, and preserves session expiration handling", async () => { + const expired = vi.fn(); + onUnauthorized(expired); + fetchMock.mockResolvedValue(json({ error: "unauthenticated" }, 401)); + await expect(api.history("model")).rejects.toMatchObject({ status: 401 }); + expect(expired).toHaveBeenCalledTimes(1); + expect(fetchMock).toHaveBeenCalledTimes(1); + }); + + it("rejects an unsupported catalog response instead of showing a false empty model library", () => { + expect(() => normalizeCatalog({})).toThrow("not supported"); + }); +}); diff --git a/web/src/modules/inferences/catalogApi.ts b/web/src/modules/inferences/catalogApi.ts new file mode 100644 index 0000000..2199a3d --- /dev/null +++ b/web/src/modules/inferences/catalogApi.ts @@ -0,0 +1,75 @@ +import { api as swissApi, ApiError, type CatalogModel, type CatalogResponse, type HistoryResponse, type ModelResponse, type ModelVariant, type Tuning, type Variant } from "@swiss/lib/api"; + +type LegacyVariant = Omit & { image?: Variant["image"]; values?: Variant["values"] }; +interface LegacyVersion { version: string; digest: string; path: string; variants: LegacyVariant[] } +interface LegacyModel extends Omit { + latest: string; + versions: LegacyVersion[]; + tuning?: { version: string; baseline: string; optimized: string; uplift?: number; workloads?: Tuning["workloads"]; report?: string }[]; +} +interface LegacyCatalog { + name: string; + ref: string; + source: string; + index: { apiVersion: string; site?: string; count: number; models: LegacyModel[] }; +} +const object = (value: unknown): value is Record => !!value && typeof value === "object" && !Array.isArray(value); +const legacy = (value: unknown): value is LegacyCatalog => object(value) && object(value.index) && Array.isArray(value.index.models); + +function legacyModel(model: LegacyModel): CatalogModel { + const latest = model.versions.find((version) => version.version === model.latest); + if (!latest || !Array.isArray(latest.variants)) throw new Error("Catalog model has no readable latest version"); + return { + name: model.name, displayName: model.displayName, description: model.description, family: model.family, + tags: model.tags, deprecated: model.deprecated, source: model.source, + defaultVariant: (latest.variants.find((variant) => variant.default) ?? latest.variants[0])?.id ?? "", + variants: latest.variants.map((variant) => ({ ...variant, version: latest.version, digest: latest.digest })), + tuning: model.tuning?.map((tuning) => ({ ...tuning, baseline: { id: tuning.baseline, version: tuning.version }, optimized: { id: tuning.optimized, version: tuning.version } })), + }; +} + +export function normalizeCatalog(value: unknown): CatalogResponse { + if (legacy(value)) return { name: value.name, ref: value.ref, source: value.source, site: value.index.site, schema: "v1", models: value.index.models.map(legacyModel) }; + if (object(value) && (value.schema === "v1" || value.schema === "v2") && Array.isArray(value.models)) return value as unknown as CatalogResponse; + throw new Error("Catalog response is not supported"); +} + +async function catalog(name?: string): Promise { + return normalizeCatalog(await swissApi.catalog(name)); +} + +async function model(name: string, options: Parameters[1] = {}): Promise { + const body = await swissApi.model(name, options) as unknown; + if (object(body) && object(body.model) && object(body.variant)) return body as unknown as ModelResponse; + if (!object(body) || !object(body.entry) || !Array.isArray(body.entry.variants)) throw new Error("Model response is not supported"); + const entry = body.entry as unknown as { name: string; version: string; digest?: string; servedName?: string; variants: LegacyVariant[] }; + const index = await catalog(options.catalog); + const metadata = index.models.find((candidate) => candidate.name === name); + if (!metadata || entry.name !== name) throw new Error("Model is not present in the selected catalog"); + const variant = options.variant ? entry.variants.find((candidate) => candidate.id === options.variant) : entry.variants.find((candidate) => candidate.default) ?? entry.variants[0]; + if (!variant) throw new Error("Model variant is not present in the selected version"); + return { + ref: String(body.ref ?? index.ref), schema: "v1", model: metadata, + variant: { ...variant, version: entry.version, digest: entry.digest ?? metadata.variants.find((candidate) => candidate.id === variant.id)?.digest ?? "", servedName: entry.servedName || entry.name }, + localPath: typeof body.localPath === "string" ? body.localPath : undefined, + pathTemplate: typeof body.pathTemplate === "string" ? body.pathTemplate : undefined, + imageRepository: object(body.imageRepository) ? body.imageRepository as Record : undefined, + }; +} + +async function history(name: string, options: Parameters[1] = {}): Promise { + try { + return await swissApi.history(name, options); + } catch (error) { + if (!(error instanceof ApiError) || ![404, 405].includes(error.status)) throw error; + const body = await swissApi.catalog(options.catalog) as unknown; + if (!legacy(body)) throw error; + const indexed = body.index.models.find((candidate) => candidate.name === name); + if (!indexed) throw error; + const current = legacyModel(indexed); + const variant = options.variant || current.defaultVariant; + return { ref: body.ref, schema: "v1", model: name, variant, revisions: indexed.versions.filter((version) => version.variants.some((candidate) => candidate.id === variant)).map((version) => ({ version: version.version, digest: version.digest, current: version.version === indexed.latest })) }; + } +} + +export const api = { ...swissApi, catalog, model, history }; diff --git a/web/src/modules/inferences/components/DeploySheet.tsx b/web/src/modules/inferences/components/DeploySheet.tsx index f720311..b82a57f 100644 --- a/web/src/modules/inferences/components/DeploySheet.tsx +++ b/web/src/modules/inferences/components/DeploySheet.tsx @@ -26,7 +26,8 @@ import { } from "@modelsphere/ui"; import { ArrowRight, FlaskConical, TriangleAlert } from "lucide-react"; import { useModulePath } from "@/shell"; -import { api, deployApi, type ApplyResult, type ApplyWith, type CatalogModel, type DiffResult, type Plan } from "@swiss/lib/api"; +import { deployApi, type ApplyResult, type ApplyWith, type CatalogModel, type DiffResult, type Plan } from "@swiss/lib/api"; +import { api } from "@/modules/inferences/catalogApi"; import { EMPTY, formFromPlan, imageOf, planRequest, reseed, siteSwitchDefaults, switchDefaults, type Form, type SwitchDefaults } from "@swiss/components/DeploySettings"; import { releaseCatalog } from "@swiss/components/CatalogChoice"; import { Provenance } from "@swiss/components/Provenance"; diff --git a/web/src/modules/inferences/components/NewTemplateSheet.tsx b/web/src/modules/inferences/components/NewTemplateSheet.tsx new file mode 100644 index 0000000..af45735 --- /dev/null +++ b/web/src/modules/inferences/components/NewTemplateSheet.tsx @@ -0,0 +1,82 @@ +import { useState } from "react"; +import { useMutation } from "@tanstack/react-query"; +import { Alert, AlertDescription, Button, FieldInput, FieldSelect, FieldTextarea, FloatingField, FormSection, Sheet, SheetBody, SheetCancelButton, SheetContent, SheetDescription, SheetFooter, SheetHeader, SheetTitle, Switch, useDirty } from "@modelsphere/ui"; +import { CircleCheck, TriangleAlert } from "lucide-react"; +import type { CatalogModel } from "@swiss/lib/api"; +import { api } from "@/modules/inferences/catalogApi"; +import { useT } from "@/modules/inferences/i18n"; +import { checkTemplate, createTemplate, type TemplateResult } from "@/modules/inferences/localTemplatesApi"; +import { EMPTY_TEMPLATE, fromTemplateVariant, templateInput, TemplateFormError, type TemplateForm } from "@/modules/inferences/templateForm"; + +export interface TemplateReference { catalog: string; model: CatalogModel } + +export function NewTemplateSheet({ references, onClose, onSaved }: { references: TemplateReference[]; onClose: () => void; onSaved: (result: TemplateResult) => void }) { + const t = useT(); + const [form, setForm] = useState({ ...EMPTY_TEMPLATE }); + const [reference, setReference] = useState("__none"); + const dirty = useDirty(form, EMPTY_TEMPLATE); + const check = useMutation({ mutationFn: () => checkTemplate(templateInput(form)) }); + const save = useMutation({ mutationFn: () => createTemplate(templateInput(form)), onSuccess: onSaved }); + const clone = useMutation({ + mutationFn: ({ catalog, model, variant, version }: { catalog: string; model: string; variant: string; version: string }) => api.model(model, { catalog, variant, version }), + onSuccess: (result) => setForm((previous) => ({ ...previous, ...fromTemplateVariant(result.variant) })), + }); + const busy = check.isPending || save.isPending || clone.isPending; + const change = (patch: Partial) => { setForm((previous) => ({ ...previous, ...patch })); check.reset(); save.reset(); clone.reset(); }; + const choices = references.flatMap(({ catalog, model }) => model.variants.map((variant) => ({ value: JSON.stringify([catalog, model.name, variant.id, variant.version]), label: `${model.displayName || model.name} · ${variant.id}`, catalog, model: model.name, variant: variant.id, version: variant.version }))); + const error = save.error ?? check.error ?? clone.error; + const message = error instanceof TemplateFormError ? t(`localTemplates.errors.${error.code}`) : error?.message; + + return ( + !open && !busy && onClose()} dirty={dirty && !busy} onDiscard={onClose}> + + {t("localTemplates.createTitle")}{t("localTemplates.createDescription")} + + {error && {message}} + {check.data?.valid && {t("localTemplates.checked")}} +
+ + { setReference(value); check.reset(); save.reset(); const selected = choices.find((choice) => choice.value === value); if (selected) clone.mutate(selected); }} options={[{ value: "__none", label: t("localTemplates.noReference") }, ...choices]} /> + + {clone.isPending &&

{t("localTemplates.loadingReference")}

} + +
+ + + + + + + + + ); +} + +export function TemplateFields({ form, onChange }: { form: TemplateForm; onChange: (patch: Partial) => void }) { + const t = useT(); + type TextField = Exclude; + const input = (key: TextField, required = false, hint?: string) => ( + + onChange({ [key]: key === "gpus" || key === "nodes" ? event.target.value.replace(/\D/g, "") : event.target.value, ...(key === "imageRepository" || key === "imageTag" ? { imageDigest: "" } : {}) })} inputMode={key === "gpus" || key === "nodes" ? "numeric" : undefined} /> + + ); + return <> +
+ {input("name", true, "name")}{input("displayName")}{input("hf", true, "hf")}{input("servedName", false, "servedName")}{input("family")}{input("tags", false, "tags")} + onChange({ description: event.target.value })} rows={2} /> +
+
+ {input("version", true)}{input("variantId", true)} + onChange({ engine: engine as TemplateForm["engine"], chartVersion: "", imageRepository: "", imageTag: "", imageDigest: "" })} options={[{ value: "sglang", label: "SGLang" }, { value: "vllm", label: "vLLM" }]} /> + {input("chartVersion", true, "chartVersion")}{input("imageRepository", true)}{input("imageTag", true)}{input("imageDigest", false, "imageDigest")} +
+
+ {input("gpus", true)}{input("nodes")}{input("minGpuMemoryGiB")} + onChange({ vendor })} options={[{ value: "nvidia", label: "NVIDIA" }, { value: "amd", label: "AMD" }, { value: "ascend", label: "Ascend" }, { value: "cambricon", label: "Cambricon" }, { value: "hygon", label: "Hygon" }]} /> + {input("gpuProducts", false, "tags")} + onChange({ topology: topology === "__none" ? "" : topology })} options={[{ value: "__none", label: t("localTemplates.topologyDefault") }, { value: "single-node", label: t("localTemplates.singleNode") }, { value: "lws", label: t("localTemplates.multiNode") }]} /> + onChange({ rdma })} /> +
+ onChange({ valuesYAML: event.target.value })} rows={12} spellCheck={false} className="font-mono text-xs" /> + ; +} diff --git a/web/src/modules/inferences/localTemplates.test.tsx b/web/src/modules/inferences/localTemplates.test.tsx new file mode 100644 index 0000000..cb77077 --- /dev/null +++ b/web/src/modules/inferences/localTemplates.test.tsx @@ -0,0 +1,117 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { renderToStaticMarkup } from "react-dom/server"; +import { parse } from "yaml"; +import type { CatalogResponse, Variant } from "@swiss/lib/api"; +import { TemplateFields } from "@/modules/inferences/components/NewTemplateSheet"; +import { createTemplate, publishTemplates, readLocalTemplates, templateReady } from "@/modules/inferences/localTemplatesApi"; +import { libraryModels, ownsTemplateCatalog } from "@/modules/inferences/localTemplatesLib"; +import { EMPTY_TEMPLATE, fromTemplateVariant, templateInput, TemplateFormError } from "@/modules/inferences/templateForm"; + +const form = { ...EMPTY_TEMPLATE, name: "private-model", displayName: "Private model", hf: "example/private-model", chartVersion: "0.7.1", imageRepository: "example/engine", imageTag: "v1", gpus: "2", gpuProducts: "H100, H100", tags: "private, fine-tuned", valuesYAML: 'env:\n FLAG: "false"\nextraArgs:\n - --custom\n' }; +const input = templateInput(form); +const recipe = parse(input.recipeYAML); +const index = { apiVersion: "catalog.swiss/v1", count: 1, models: [{ ...input.metadata, latest: "1.0.0", versions: [{ version: "1.0.0", path: "models/private-model/private-model-1.0.0.yaml", digest: "sha256:recipe", variants: recipe.variants.map(({ image: _image, values: _values, ...variant }: Record) => variant) }] }] }; +const native = { enabled: true, catalogName: "console-local", catalogURL: "http://console.example/catalog/console/g/1/index.json", generation: 1, registeredGeneration: 1, modelGenerations: { "private-model": 1 }, index }; +const fetchMock = vi.fn(); +const json = (body: unknown, status = 200) => new Response(JSON.stringify(body), { status }); +beforeEach(() => { fetchMock.mockReset(); vi.stubGlobal("fetch", fetchMock); vi.stubGlobal("document", { cookie: "token=operator" }); }); +afterEach(() => vi.unstubAllGlobals()); + +describe("native model template authoring", () => { + it("creates actual model metadata and a deployable v1 recipe rather than a release record", () => { + expect(input.metadata).toMatchObject({ name: "private-model", source: { hf: "example/private-model" }, tags: ["private", "fine-tuned"] }); + expect(recipe).toMatchObject({ apiVersion: "catalog.swiss/v1", name: "private-model", version: "1.0.0", variants: [{ id: "default", engine: "sglang", chart: { name: "sglang", version: "0.7.1" }, image: { repository: "example/engine", tag: "v1" }, requires: { gpus: 2, gpuProduct: ["H100"] }, values: { env: { FLAG: "false" }, extraArgs: ["--custom"] } }] }); + expect(recipe).not.toHaveProperty("source"); + expect(recipe).not.toHaveProperty("release"); + }); + + it("preserves copied runtime, image digest and multi-node hardware without replacing new model identity", () => { + const variant = { id: "sglang-tp", version: "2.0.0", digest: "sha256:source-recipe", servedName: "source-model", engine: "sglang", chart: { name: "sglang", version: "0.7.1" }, image: { repository: "example/engine", tag: "v1", digest: `sha256:${"1".repeat(64)}` }, requires: { gpus: 16, nodes: 512, minGpuMemoryGiB: 80, vendor: "amd", gpuProduct: ["example-card"], topology: "lws", rdma: true }, values: { lws: { enabled: true, size: 512 }, extraArgs: ["--custom"] } } as Variant; + const patch = fromTemplateVariant(variant); + expect(patch).not.toHaveProperty("name"); + expect(patch).not.toHaveProperty("hf"); + const copied = templateInput({ ...form, ...patch }); + expect(copied.metadata.name).toBe("private-model"); + expect(parse(copied.recipeYAML).variants[0]).toMatchObject({ image: variant.image, requires: variant.requires, values: variant.values }); + }); + + it("lets an empty parameter editor use chart defaults", () => { + expect(parse(templateInput({ ...form, valuesYAML: "" }).recipeYAML).variants[0].values).toEqual({}); + }); + + it.each(["extraArgs: [", "a: 1\na: 2", "[1, 2]", "a: &loop [*loop]", "a: © {value: 1}\nb: *copy", "a: .inf"])("rejects an invalid parameter document before saving: %s", (valuesYAML) => { + expect(() => templateInput({ ...form, valuesYAML })).toThrow(TemplateFormError); + }); + + it("renders a visible model template form with weight source, image, GPU and runtime fields", () => { + const html = renderToStaticMarkup( {}} />); + for (const label of ["模型标识", "权重来源", "推理引擎", "镜像仓库", "每节点 GPU 数", "运行参数 YAML"]) expect(html).toContain(label); + expect(html).toContain("example/private-model"); + }); + + it("keeps the existing library functional when an older Console lacks native storage", async () => { + fetchMock.mockResolvedValue(json({ error: "not found" }, 404)); + expect(await readLocalTemplates()).toMatchObject({ enabled: false, catalog: { models: [] } }); + }); + + it("authenticates native creation and does not send a storage namespace or publication target", async () => { + fetchMock.mockResolvedValue(json({ created: true, ready: true, model: "private-model", generation: 1 }, 201)); + expect(await createTemplate(input)).toMatchObject({ created: true, ready: true }); + const [url, options] = fetchMock.mock.calls[0] as [string, RequestInit]; + expect(url).toBe("/api/model-templates"); + expect(options.method).toBe("POST"); + expect(new Headers(options.headers).get("Authorization")).toBe("Bearer operator"); + const body = JSON.parse(String(options.body)); + expect(body).toEqual(input); + expect(body).not.toHaveProperty("namespace"); + expect(body).not.toHaveProperty("publishURL"); + }); + + it("preserves a 202 saved-pending result and retries publication without creating again", async () => { + fetchMock.mockResolvedValueOnce(json({ created: true, ready: false, model: "private-model", generation: 1, registeredGeneration: null }, 202)).mockResolvedValueOnce(json({ ready: true, generation: 1, registeredGeneration: 1 })); + expect(await createTemplate(input)).toMatchObject({ created: true, ready: false }); + expect(await publishTemplates()).toMatchObject({ ready: true }); + expect(fetchMock.mock.calls[1][0]).toBe("/api/model-templates/publish"); + expect(JSON.parse(String(fetchMock.mock.calls[1][1].body))).toEqual({}); + }); + + it("combines original and native cards in the default view without shadowing duplicate names", async () => { + fetchMock.mockResolvedValue(json(native)); + const local = await readLocalTemplates(); + const remote: CatalogResponse = { ...local.catalog, name: "default", source: "https://catalog.example/index.json" }; + const rows = libraryModels(remote, local, "default", ""); + expect(rows.map((row) => [row.catalog, row.model.name])).toEqual([["default", "private-model"], ["console-local", "private-model"]]); + expect(remote.models).toHaveLength(1); + expect(libraryModels(remote, local, "default", "default")).toHaveLength(1); + expect(libraryModels(remote, local, "console-local", "console-local", { name: local.catalogName, source: local.catalogURL })).toEqual([{ catalog: "console-local", model: local.catalog.models[0], custom: true, ready: true }]); + }); + + it("keeps a foreign catalog named console-local accessible before native registration", async () => { + fetchMock.mockResolvedValue(json(native)); + const data = await readLocalTemplates(); + const foreign: CatalogResponse = { ...data.catalog, name: "console-local", source: "https://foreign.example/index.json" }; + const empty = { ...data, registrationError: "catalog_name_conflict", registeredGeneration: null, catalog: { ...data.catalog, models: [] } }; + expect(libraryModels(foreign, empty, "console-local", "console-local")).toEqual([{ catalog: "console-local", model: foreign.models[0], custom: false, ready: true }]); + expect(libraryModels(foreign, empty, "console-local", "")).toHaveLength(1); + }); + + it("recognizes owned generations without treating a matching name as ownership", async () => { + fetchMock.mockResolvedValue(json({ ...native, generation: 2, catalogURL: native.catalogURL.replace('/g/1/', '/g/2/') })); + const data = await readLocalTemplates(); + expect(ownsTemplateCatalog(data, { name: "console-local", source: native.catalogURL })).toBe(true); + for (const source of ["https://foreign.example/index.json", native.catalogURL.replace('/catalog/console/', '/catalog/foreign/'), native.catalogURL + '?other=1', "not-a-url"]) { + expect(ownsTemplateCatalog(data, { name: "console-local", source })).toBe(false); + } + expect(ownsTemplateCatalog({ ...data, registrationError: "catalog_name_conflict" }, { name: "console-local", source: native.catalogURL })).toBe(false); + const foreign = { ...data.catalog, source: "https://foreign.example/index.json" }; + expect(libraryModels(foreign, data, "console-local", "console-local")[0].custom).toBe(false); + }); + + it("keeps older published templates deployable while a newer generation is pending", async () => { + fetchMock.mockResolvedValue(json({ ...native, generation: 2, registeredGeneration: 1, modelGenerations: { "private-model": 1, "new-model": 2 } })); + const local = await readLocalTemplates(); + expect(templateReady(local, "private-model")).toBe(true); + expect(templateReady(local, "new-model")).toBe(false); + expect(templateReady({ ...local, registeredGeneration: null }, "private-model")).toBe(false); + }); +}); diff --git a/web/src/modules/inferences/localTemplatesApi.ts b/web/src/modules/inferences/localTemplatesApi.ts new file mode 100644 index 0000000..7bb2fbd --- /dev/null +++ b/web/src/modules/inferences/localTemplatesApi.ts @@ -0,0 +1,59 @@ +import { ApiError, request } from "@/shell"; +import type { CatalogResponse } from "@swiss/lib/api"; +import { normalizeCatalog } from "@/modules/inferences/catalogApi"; + +export interface TemplateMetadata { + name: string; + displayName?: string; + description?: string; + family?: string; + tags?: string[]; + source: { hf: string }; +} +export interface TemplateInput { metadata: TemplateMetadata; recipeYAML: string } +export interface TemplateResult { + created?: boolean; + model?: string; + version?: string; + catalogName: string; + generation: number; + catalogURL: string; + ready: boolean; + registeredGeneration: number | null; + reason?: string; +} +export interface LocalTemplates { + enabled: boolean; + catalogName: string; + catalogURL: string; + generation: number; + registeredGeneration: number | null; + registrationError?: string; + modelGenerations: Record; + catalog: CatalogResponse; +} + +export function disabledTemplates(): LocalTemplates { + return { enabled: false, catalogName: "console-local", catalogURL: "", generation: 0, registeredGeneration: null, modelGenerations: {}, catalog: { name: "console-local", ref: "", source: "", schema: "v1", models: [] } }; +} + +export async function readLocalTemplates(): Promise { + let body: Omit & { index?: unknown }; + try { + body = await request("GET", "/api/model-templates"); + } catch (error) { + if (error instanceof ApiError && error.status === 404) return disabledTemplates(); + throw error; + } + if (!body.enabled) return disabledTemplates(); + return { ...body, modelGenerations: body.modelGenerations ?? {}, catalog: normalizeCatalog({ name: body.catalogName, ref: `generation:${body.generation}`, source: body.catalogURL, index: body.index }) }; +} + +export const checkTemplate = (input: TemplateInput) => request<{ valid: boolean; model: string; version: string; variantCount: number }>("POST", "/api/model-templates/check", input); +export const createTemplate = (input: TemplateInput) => request("POST", "/api/model-templates", input); +export const publishTemplates = () => request("POST", "/api/model-templates/publish", {}); + +export function templateReady(data: LocalTemplates, name: string): boolean { + const generation = data.modelGenerations[name]; + return Number.isSafeInteger(generation) && generation > 0 && data.registeredGeneration !== null && generation <= data.registeredGeneration; +} diff --git a/web/src/modules/inferences/localTemplatesLib.ts b/web/src/modules/inferences/localTemplatesLib.ts new file mode 100644 index 0000000..35b2d9e --- /dev/null +++ b/web/src/modules/inferences/localTemplatesLib.ts @@ -0,0 +1,28 @@ +import type { CatalogInfo, CatalogModel, CatalogResponse } from "@swiss/lib/api"; +import { templateReady, type LocalTemplates } from "@/modules/inferences/localTemplatesApi"; + +export interface LibraryModel { catalog: string; model: CatalogModel; custom: boolean; ready: boolean } +type CatalogLocation = Pick & Partial>; + +function publicationIdentity(value: string | undefined): string | undefined { + if (!value) return; + try { + const url = new URL(value); + if (!["http:", "https:"].includes(url.protocol) || url.username || url.password || url.search || url.hash) return; + const suffix = /\/g\/(?:0|[1-9]\d*)\/index\.json$/.exec(url.pathname); + if (suffix) return url.origin + url.pathname.slice(0, suffix.index); + } catch { return; } +} + +export function ownsTemplateCatalog(local: LocalTemplates | undefined, catalog: CatalogLocation | undefined): boolean { + if (!local?.enabled || !catalog || catalog.name !== local.catalogName || local.registrationError === "catalog_name_conflict") return false; + const expected = publicationIdentity(local.catalogURL); + return expected !== undefined && expected === publicationIdentity(catalog.source || catalog.url); +} + +export function libraryModels(remote: CatalogResponse | undefined, local: LocalTemplates | undefined, selected: string, named: string, configured?: CatalogLocation): LibraryModel[] { + const custom = local?.enabled ? local.catalog.models.map((model) => ({ catalog: local.catalogName, model, custom: true, ready: templateReady(local, model.name) })) : []; + if (selected === local?.catalogName && ownsTemplateCatalog(local, configured ?? remote)) return custom; + const published = remote?.models.map((model) => ({ catalog: remote.name || selected, model, custom: false, ready: true })) ?? []; + return named ? published : [...published, ...custom]; +} diff --git a/web/src/modules/inferences/locales/en-US.json b/web/src/modules/inferences/locales/en-US.json index 54b216d..752136a 100644 --- a/web/src/modules/inferences/locales/en-US.json +++ b/web/src/modules/inferences/locales/en-US.json @@ -324,6 +324,52 @@ "deprecated": "Deprecated: {reason}", "noVariants": "This version has no variants" }, + "localTemplates": { + "add": "New model template", + "addExternal": "Add to this catalog", + "createTitle": "New model template", + "createDescription": "Save a reusable deployment template, then deploy it from the model library.", + "reference": "Reference deployment recipe (optional)", + "referenceHint": "Choose an existing recipe to prefill its engine, image, GPUs and values.", + "noReference": "Start from scratch", + "loadingReference": "Reading reference recipe…", + "check": "Check", "checking": "Checking…", "checked": "The template passed validation and can be saved.", + "save": "Save template", "saving": "Saving…", + "disabled": "Custom template storage is not enabled in this Console. An administrator must upgrade or configure Console.", + "defaultView": "Default and custom", "custom": "Custom", "pendingLabel": "Catalog connection pending", + "pending": "Some custom templates are saved but cannot be read by the deployment backend yet. Connect the catalog before deploying them.", + "saved": "Model template {name} was saved and added to this model library.", + "savedPending": "Model template {name} was saved. Retry the catalog connection from the list before deploying.", + "connected": "The custom catalog is connected and its templates can enter the deployment flow.", + "retryRead": "Retry reading", "retryPublish": "Retry catalog connection", "connecting": "Connecting…", + "topologyDefault": "No topology constraint", "singleNode": "Single node", "multiNode": "Multi-node LeaderWorkerSet", + "sections": { "model": "Model information", "recipe": "Deployment recipe", "hardware": "Hardware requirements", "values": "Engine and runtime values" }, + "fields": { + "name": "Model identifier", "displayName": "Display name", "description": "Description", "hf": "Weights source", + "family": "Model family", "tags": "Tags", "version": "Template version", "servedName": "Served model name", + "variantId": "Recipe ID", "engine": "Inference engine", "chartVersion": "Chart version", + "imageRepository": "Image repository", "imageTag": "Image tag", "imageDigest": "Image digest (optional)", + "gpus": "GPUs per node", "nodes": "Nodes (optional)", "minGpuMemoryGiB": "Minimum GPU memory (GiB, optional)", "vendor": "Accelerator vendor", "gpuProducts": "GPU products (optional)", + "topology": "Topology constraint", "rdma": "Requires RDMA", "valuesYAML": "Runtime values YAML" + }, + "hints": { + "name": "Lowercase letters, digits, dots, underscores and hyphens, starting with a letter or digit; at most 63 characters.", + "hf": "Weights source in org/model format, used to resolve the model path at this site.", + "servedName": "The model name clients call. Empty uses the model identifier.", + "tags": "Separate multiple values with commas.", + "engine": "Review the image, chart and engine arguments after changing engines.", + "chartVersion": "An available exact version, for example 0.7.1. Replace a reference recipe's version range with a specific version.", + "imageDigest": "Optional sha256 digest to pin the image. Changing repository or tag clears the previous digest.", + "topology": "Multi-node recipes need matching node count and values.lws.size.", + "values": "Engine arguments, resources and other values. Deployment services read these recipes; use Secret references for sensitive data." + }, + "errors": { + "required": "Fill in the model identifier, weights source, version, recipe, chart and image.", + "name": "Invalid model identifier or recipe ID; at most 63 characters. Recipe IDs use lowercase letters, digits and hyphens.", + "hardware": "GPU and node counts must be positive whole numbers; minimum memory must be nonnegative.", + "values": "Runtime values must be a valid YAML mapping without duplicate keys, YAML aliases, non-finite numbers or oversized content." + } + }, "newModel": { "action": "New model", "title": "New model in {catalog}", diff --git a/web/src/modules/inferences/locales/zh-CN.json b/web/src/modules/inferences/locales/zh-CN.json index 5e9b747..d34d537 100644 --- a/web/src/modules/inferences/locales/zh-CN.json +++ b/web/src/modules/inferences/locales/zh-CN.json @@ -313,6 +313,61 @@ "deprecated": "已弃用：{reason}", "noVariants": "这个版本没有变体" }, + "localTemplates": { + "add": "新增模型模板", + "addExternal": "新增到当前目录", + "createTitle": "新增模型模板", + "createDescription": "保存可重复使用的部署模板，之后可在模型库选择部署。", + "reference": "参考部署方案（可选）", + "referenceHint": "选择已有方案可预填引擎、镜像、GPU 和参数。", + "noReference": "从零填写", + "loadingReference": "正在读取参考方案…", + "check": "校验", + "checking": "正在校验…", + "checked": "模板校验通过，可以保存。", + "save": "保存模板", + "saving": "正在保存…", + "disabled": "当前 Console 尚未启用自定义模板存储，请管理员升级或配置 Console。", + "defaultView": "默认与自定义", + "custom": "自定义", + "pendingLabel": "待连接目录", + "pending": "部分自定义模板已保存，但部署服务还不能读取。连接成功后可部署。", + "saved": "模型模板 {name} 已保存并加入当前模型库。", + "savedPending": "模型模板 {name} 已保存，目录尚未连接，请在列表重试连接。", + "connected": "自定义模板目录已连接，模板可以进入部署流程。", + "retryRead": "重试读取", + "retryPublish": "重试连接目录", + "connecting": "正在连接…", + "topologyDefault": "不限制拓扑", + "singleNode": "单节点", + "multiNode": "多节点 LeaderWorkerSet", + "sections": { "model": "模型信息", "recipe": "部署方案", "hardware": "硬件需求", "values": "引擎与运行参数" }, + "fields": { + "name": "模型标识", "displayName": "显示名称", "description": "说明", "hf": "权重来源", + "family": "模型系列", "tags": "标签", "version": "模板版本", "servedName": "服务模型名", + "variantId": "部署方案 ID", "engine": "推理引擎", "chartVersion": "Chart 版本", + "imageRepository": "镜像仓库", "imageTag": "镜像标签", "imageDigest": "镜像摘要（可选）", + "gpus": "每节点 GPU 数", "nodes": "节点数（可选）", "minGpuMemoryGiB": "最低 GPU 显存（GiB，可选）", "vendor": "加速卡厂商", + "gpuProducts": "GPU 型号（可选）", "topology": "拓扑约束", "rdma": "需要 RDMA", "valuesYAML": "运行参数 YAML" + }, + "hints": { + "name": "使用小写字母、数字、点、下划线和短横线，以字母或数字开头，最多 63 个字符。", + "hf": "填写 org/model 格式的权重来源，用于解析站点上的模型路径。", + "servedName": "客户端调用时使用的模型名，留空使用模型标识。", + "tags": "多个值用逗号分隔。", + "engine": "更换引擎后请核对镜像、chart 和引擎参数。", + "chartVersion": "填写已有的精确版本号，例如 0.7.1；参考方案中的版本范围需改为具体版本。", + "imageDigest": "可填写 sha256 摘要锁定镜像；修改仓库或标签时会清除旧摘要。", + "topology": "多节点方案需同时填写节点数和 values.lws.size。", + "values": "填写引擎启动参数、资源等 values。模板会提供给部署服务，敏感信息请使用 Secret 引用。" + }, + "errors": { + "required": "请填写模型标识、权重来源、版本、部署方案、chart 和镜像。", + "name": "模型标识或方案 ID 不符合格式，最多 63 个字符；方案 ID 仅使用小写字母、数字和短横线。", + "hardware": "GPU 数和节点数必须是正整数，最低显存必须是非负数。", + "values": "运行参数必须是有效的 YAML 映射，不支持重复键、YAML 引用、非有限数值或过大的内容。" + } + }, "newModel": { "action": "新增模型", "title": "在 {catalog} 中新增模型", diff --git a/web/src/modules/inferences/templateForm.ts b/web/src/modules/inferences/templateForm.ts new file mode 100644 index 0000000..2c1089b --- /dev/null +++ b/web/src/modules/inferences/templateForm.ts @@ -0,0 +1,63 @@ +import { isScalar, parseDocument, stringify, visit } from "yaml"; +import type { Variant } from "@swiss/lib/api"; +import type { TemplateInput } from "@/modules/inferences/localTemplatesApi"; + +export interface TemplateForm { + name: string; displayName: string; description: string; hf: string; family: string; tags: string; + version: string; servedName: string; variantId: string; engine: "sglang" | "vllm"; + chartVersion: string; imageRepository: string; imageTag: string; imageDigest: string; + gpus: string; nodes: string; minGpuMemoryGiB: string; vendor: string; gpuProducts: string; topology: string; rdma: boolean; valuesYAML: string; +} +export const EMPTY_TEMPLATE: TemplateForm = { + name: "", displayName: "", description: "", hf: "", family: "", tags: "", version: "1.0.0", servedName: "", variantId: "default", engine: "sglang", + chartVersion: "", imageRepository: "", imageTag: "", imageDigest: "", gpus: "1", nodes: "", minGpuMemoryGiB: "", vendor: "nvidia", gpuProducts: "", topology: "", rdma: false, valuesYAML: "{}\n", +}; +export class TemplateFormError extends Error { + constructor(public code: "required" | "name" | "hardware" | "values") { super(code); } +} +const optional = (value: string) => value.trim() || undefined; +const list = (value: string) => [...new Set(value.split(/[,\n]/).map((item) => item.trim()).filter(Boolean))]; +const count = (value: string) => /^\d+$/.test(value) && Number.isSafeInteger(Number(value)) && Number(value) > 0; + +function templateValues(text: string): Record { + if (new TextEncoder().encode(text).length > 128 << 10) throw new TemplateFormError("values"); + const document = parseDocument(text.trim() ? text : "{}", { intAsBigInt: true, uniqueKeys: true }); + if (document.errors.length || document.warnings.length) throw new TemplateFormError("values"); + visit(document, { Alias() { throw new TemplateFormError("values"); }, Pair(_, pair) { if (!isScalar(pair.key) || typeof pair.key.value !== "string") throw new TemplateFormError("values"); } }); + let tree: unknown; + try { tree = document.toJS({ maxAliasCount: 50 }); } catch { throw new TemplateFormError("values"); } + if (!tree || typeof tree !== "object" || Array.isArray(tree)) throw new TemplateFormError("values"); + const seen = new Set(); + let nodes = 0; + const check = (value: unknown, depth: number) => { + if (++nodes > 50_000 || depth > 64 || typeof value === "number" && !Number.isFinite(value)) throw new TemplateFormError("values"); + if (value && typeof value === "object") { + if (seen.has(value)) throw new TemplateFormError("values"); + seen.add(value); + for (const child of Object.values(value)) check(child, depth + 1); + seen.delete(value); + } + }; + check(tree, 0); + return tree as Record; +} + +export function templateInput(form: TemplateForm): TemplateInput { + if (![form.name, form.hf, form.version, form.variantId, form.chartVersion, form.imageRepository, form.imageTag].every((value) => value.trim())) throw new TemplateFormError("required"); + if (!/^[a-z0-9][a-z0-9._-]{0,62}$/.test(form.name.trim()) || !/^[a-z0-9][a-z0-9-]{0,62}$/.test(form.variantId.trim())) throw new TemplateFormError("name"); + if (!count(form.gpus) || form.nodes && !count(form.nodes) || form.minGpuMemoryGiB.trim() && (!Number.isFinite(Number(form.minGpuMemoryGiB)) || Number(form.minGpuMemoryGiB) < 0)) throw new TemplateFormError("hardware"); + const products = list(form.gpuProducts); + return { + metadata: { name: form.name.trim(), displayName: optional(form.displayName), description: optional(form.description), family: optional(form.family), tags: list(form.tags), source: { hf: form.hf.trim() } }, + recipeYAML: stringify({ + apiVersion: "catalog.swiss/v1", name: form.name.trim(), version: form.version.trim(), servedName: optional(form.servedName), + variants: [{ id: form.variantId.trim(), engine: form.engine, default: true, chart: { name: form.engine, version: form.chartVersion.trim() }, image: { repository: form.imageRepository.trim(), tag: form.imageTag.trim(), digest: optional(form.imageDigest) }, requires: { gpus: Number(form.gpus), nodes: form.nodes ? Number(form.nodes) : undefined, minGpuMemoryGiB: form.minGpuMemoryGiB.trim() ? Number(form.minGpuMemoryGiB) : undefined, vendor: form.vendor, gpuProduct: products.length ? products : undefined, topology: optional(form.topology), rdma: form.rdma || undefined }, values: templateValues(form.valuesYAML) }], + }, { lineWidth: 0 }), + }; +} + +export function fromTemplateVariant(variant: Variant): Partial { + if (variant.engine !== "sglang" && variant.engine !== "vllm") throw new Error("Unsupported reference engine"); + const memory = (variant.requires as typeof variant.requires & { minGpuMemoryGiB?: number }).minGpuMemoryGiB; + return { variantId: variant.id, engine: variant.engine, chartVersion: variant.chart.version, imageRepository: variant.image?.repository ?? "", imageTag: variant.image?.tag ?? "", imageDigest: variant.image?.digest ?? "", gpus: String(variant.requires.gpus), nodes: variant.requires.nodes ? String(variant.requires.nodes) : "", minGpuMemoryGiB: memory == null ? "" : String(memory), vendor: variant.requires.vendor ?? "nvidia", gpuProducts: variant.requires.gpuProduct?.join(", ") ?? "", topology: variant.requires.topology ?? "", rdma: !!variant.requires.rdma, valuesYAML: stringify(variant.values ?? {}, { lineWidth: 0 }) }; +} diff --git a/web/src/modules/inferences/useSupportedTotals.ts b/web/src/modules/inferences/useSupportedTotals.ts index 9849c70..fbd1180 100644 --- a/web/src/modules/inferences/useSupportedTotals.ts +++ b/web/src/modules/inferences/useSupportedTotals.ts @@ -1,5 +1,6 @@ import { useQueries, type UseQueryResult } from "@tanstack/react-query"; -import { api, type ClusterInfo } from "@swiss/lib/api"; +import type { ClusterInfo } from "@swiss/lib/api"; +import { api } from "@/modules/inferences/catalogApi"; import { supportedTotals } from "@/modules/inferences/dashboard-lib"; export function useSupportedTotals(cluster: UseQueryResult) {