fix: qualify production smoke against published runtime [skip deploy]

.github/workflows/ci.yml

@@ -171,6 +171,7 @@ jobs:
               - '.github/workflows/ci.yml'
               - 'scripts/ci-scope.mjs'
               - 'scripts/production-rag-contract.test.mjs'
+              - 'scripts/release-smoke*.mjs'
               - 'scripts/update-deployment-images.mjs'
               - 'scripts/trigger-komodo-rollout.mjs'
               - 'scripts/prepare-forum-rollout.mjs'
@@ -286,7 +287,7 @@ jobs:
         working-directory: third_party/open-notebook
         run: uv run --frozen --python 3.12 pytest -q
       - if: matrix.component == 'deploy-contract'
-        run: node --test scripts/production-rag-contract.test.mjs
+        run: node --test scripts/production-rag-contract.test.mjs scripts/release-smoke.test.mjs
       - if: matrix.component == 'release'
         run: npm run version:check
 

docs/evidence/production-admission-20261017/README.md

@@ -0,0 +1,81 @@
+# Production admission after control-plane migration
+
+The user authorized production delivery of merged PRs164–175 and subsequent
+fixes after the separate control-plane migration completed. This receipt records
+the available preflight, not a successful production deployment.
+
+The migration handoff reports txy3 as the current control-plane host, three
+healthy nodes, eight running stacks, an empty pending set and successful backup
+update `6ad33425e06d5e1f798292e9`. Those authenticated facts have **not been
+independently re-read by this executor**. The repository's aly ownership prose is
+stale; do not use it to overwrite the migrated resource configuration.
+
+## Observed state
+
+- Source main: `06ba56169ae8796bf6e067f4eecca0064145a816` (PR175 merged).
+- Target public runtime:3.5.2, schema12, controlPlane11; product migration0043.
+- Independent current full SPA session/human-message recovery:6/6, with
+  [retained evidence](../spa-3.5.2/README.md). Captcha login, model-assisted SPA,
+  complete product Worker and3.5.2 endurance are not claimed.
+- `https://ops.christmas1314.xyz` returned HTTPS200 with TLS verification.
+- Public production `/api/meta` returned commit
+  `9332947e099d27b7716236a99604fc548059f6ad`, runtime3.4.7/schema11.
+  Health and dependency health returned200/ok. Public `/api/readyz` returned401;
+  the actual internal readiness and Worker status require authorized inspection.
+- Six affected Compose files passed `docker compose config --quiet` with
+  explicitly synthetic required values. This proves syntax, not production
+  secret completeness. No production credentials were read or printed.
+- Deployment/smoke contracts:22/22. The new smoke cases failed before the fix;
+  the CI selector test also failed before wiring the cases into scoped CI.
+
+The old release smoke falsely required `reasoningRuntime:null`, accepting a
+retired runtime and rejecting the real public runtime contract. The fix verifies
+the exact installed package's public `releaseVersions`, retains commit/product
+version, tenant and authenticated WebSocket checks, and omits HTTP error bodies.
+Its isolated transport fixture is a script contract test, not a live smoke pass.
+
+## Blocking access and remaining release sequence
+
+This cloud executor lacks Sigillo, the Komodo CLI, Komodo API/webhook environment
+injection, Cloudflare environment injection and an existing authorized smoke
+session. The production skill explicitly requires stopping when credentials are
+missing. No credential was created, copied into a file, or extracted from CI.
+No signed webhook, ResourceSync write, deployment or database mutation occurred.
+
+1. Restore the existing authorized cloud Sigillo access; resolve the current
+   `LingxiLoop Control Plane` project and `prod` environment. Check identity and
+   official Komodo CLI2.3.3 through Sigillo against the fixed HTTPS entrypoint.
+2. Read current txy3-backed servers, exact owned stacks, the ordered rollout,
+   update history and resource-sync source. Compare the live source with the
+   reviewed repository before any sync; preserve unrelated services and
+   wegolibrary. Do not run two releases concurrently.
+3. Read the actual production migration/runtime receipt and active task counts.
+   If0027 is still pending, drain interactive, handoff, Mission and Canvas tasks
+   before membership changes. For0043, stop both old workers, confirm no live
+   leases and schema11 provenance, retain data/backup evidence and run the
+   forward migration once. No database reset or legacy adoption is allowed.
+   The existing forum preflight requires schema38 or newer; if the live schema
+   is older, diagnose that boundary before invoking the rollout script.
+4. After checks, use the existing GitHub Actions release path to publish and pin
+   immutable images, then the reviewed `lingxiloop-production-rollout` procedure.
+   Current source pins still reference earlier server/gateway images because the
+   development merges intentionally skipped deployment; do not trigger the old
+   procedure without preparing and verifying the new image references.
+   No control-plane source changed in the164–175 development range; confirm the
+   live Worker version before deciding whether Cloudflare deployment is needed.
+5. Retain the Komodo update ID and require `Complete` with `success:true`.
+   Verify both app/Worker image identities, package3.5.2/schema12/migration43,
+   healthy affected services, public exact commit metadata, and run the repaired
+   smoke using an existing authorized session and tenant. Verify message
+   acceptance/IM durability, a bounded model action and the next human turn only
+   in the authorized test tenant with approved provider/pricing limits.
+
+The original3.5.1 Web/API-only soak, its evidence watcher and the single existing
+independent24h review were left untouched. None qualifies3.5.2 durability.
+
+See [admission receipt](admission.json), [public observation](public-preflight.json),
+[Compose checks](compose-validation.json), [red tests](smoke-red.log),
+[red CI selection](selector-red.log) and [green22-case result](contracts-green.log).
+
+Only trailing whitespace is normalized in the retained test logs. Raw command
+logs remain under `/tmp/lingxiloop-release-*` in this executor.

docs/evidence/production-admission-20261017/admission.json

@@ -0,0 +1,38 @@
+{
+  "recordedAt": "2026-10-17T08:52:01.711881+00:00",
+  "sourceMain": "06ba56169ae8796bf6e067f4eecca0064145a816",
+  "runtime": "3.5.2",
+  "schema": 12,
+  "controlPlane": 11,
+  "migration": "0043_lingxios_3_5_0.sql",
+  "komodoPublicHttps": 200,
+  "komodoIdentityVerified": false,
+  "productionMutationStarted": false,
+  "credentialsPresence": {
+    "KOMODO_API_KEY": false,
+    "KOMODO_API_SECRET": false,
+    "KOMODO_WEBHOOK_URL": false,
+    "KOMODO_WEBHOOK_SECRET": false,
+    "CLOUDFLARE_API_TOKEN": false,
+    "LINGXILOOP_SMOKE_COOKIE": false
+  },
+  "tooling": {
+    "sigillo": false,
+    "km": false,
+    "wrangler": "4.127.1",
+    "gh": "2.46.0",
+    "git": "2.52.0",
+    "docker": "28.4.0"
+  },
+  "checks": {
+    "composeFilesPassed": 6,
+    "deploymentAndSmokeContractPassed": 22,
+    "spaPassed": 6
+  },
+  "rollout": {
+    "updateId": null,
+    "complete": false,
+    "success": null
+  },
+  "scope": "Production admission only. No production write or deployment was attempted."
+}

docs/evidence/production-admission-20261017/compose-validation.json

@@ -0,0 +1,32 @@
+[
+  {
+    "file": "deploy/komodo/lingxiloop-app-a/compose.yml",
+    "exitCode": 0,
+    "output": ""
+  },
+  {
+    "file": "deploy/komodo/lingxiloop-app-b/compose.yml",
+    "exitCode": 0,
+    "output": ""
+  },
+  {
+    "file": "deploy/komodo/lingxiloop-core-state/compose.yml",
+    "exitCode": 0,
+    "output": ""
+  },
+  {
+    "file": "deploy/komodo/lingxiloop-knowledge-agent/compose.yml",
+    "exitCode": 0,
+    "output": ""
+  },
+  {
+    "file": "deploy/komodo/server-b-ingress/compose.yml",
+    "exitCode": 0,
+    "output": ""
+  },
+  {
+    "file": "deploy/komodo/uptime/compose.yml",
+    "exitCode": 0,
+    "output": ""
+  }
+]

docs/evidence/production-admission-20261017/contracts-green.log

@@ -0,0 +1,30 @@
+✔ production stops both old workers before an application can enqueue new effect kinds (1.518111ms)
+✔ server image includes the complete shared runtime import graph (360.702098ms)
+✔ production Open Notebook receives only the explicit RAG environment (0.394844ms)
+✔ packaged and published stacks select the RAG-only image (0.54587ms)
+✔ CI component matrix runs only checks enabled by the changed scope (0.516556ms)
+✔ the production image runs only the RAG API and worker (0.468464ms)
+✔ removed Open Notebook capabilities cannot be re-enabled by deployment configuration (0.285669ms)
+✔ Komodo knowledge services receive writable storage and the control plane URL (0.271819ms)
+✔ Komodo runs isolated workers on both app nodes with bounded drain and readiness (0.701904ms)
+✔ the gateway uses the备案 ingress and the Worker uses its admin domain (0.469977ms)
+✔ live chat streams balance independently across both nodes through unbuffered proxy hops (0.451819ms)
+✔ Komodo control plane and ingress keep management sockets private (0.337797ms)
+✔ main publishes changed images and rolls out a complete immutable release (0.534914ms)
+✔ all deployable LingxiLoop images use CI-managed unique tags (0.808254ms)
+✔ CI selects checks and image publishing by component (0.468874ms)
+✔ test and CI changes select checks without publishing; heavy experiments require manual scope (0.457076ms)
+✔ workflow path filters route real changed paths to their owning checks (29.660654ms)
+✔ actual workflow conditions block publishing and deployment after a failed or cancelled gate (2.148298ms)
+✔ production smoke accepts the exact public runtime and reaches authenticated WebSocket (146.347669ms)
+✔ production smoke rejects retired or mismatched runtime before tenant access (821.167458ms)
+✔ production smoke still enforces the concrete release commit (137.419549ms)
+✔ production smoke HTTP failure never prints server body or session secrets (123.926573ms)
+ℹ tests 22
+ℹ suites 0
+ℹ pass 22
+ℹ fail 0
+ℹ cancelled 0
+ℹ skipped 0
+ℹ todo 0
+ℹ duration_ms 1391.061163

docs/evidence/production-admission-20261017/public-preflight.json

@@ -0,0 +1,50 @@
+{
+  "checkedAt": "2026-10-17T08:50:45.544883+00:00",
+  "results": [
+    {
+      "path": "/api/meta",
+      "httpStatus": "200",
+      "curlExit": 0,
+      "metadata": {
+        "product": "LingxiLoop",
+        "version": "0.1.0-beta",
+        "commitSha": "9332947e099d27b7716236a99604fc548059f6ad",
+        "reasoningRuntime": {
+          "name": "@lyyzka/lingxios",
+          "runtime": "3.4.7",
+          "controlPlane": 11,
+          "kernel": 2,
+          "schema": 11,
+          "assistantMessage": 2
+        }
+      }
+    },
+    {
+      "path": "/api/health",
+      "httpStatus": "200",
+      "curlExit": 0,
+      "health": {
+        "ok": true
+      }
+    },
+    {
+      "path": "/api/readyz",
+      "httpStatus": "401",
+      "curlExit": 0,
+      "health": {}
+    },
+    {
+      "path": "/api/health/dependencies",
+      "httpStatus": "200",
+      "curlExit": 0,
+      "health": {
+        "ok": true,
+        "dependencies": {
+          "database": true,
+          "redis": true,
+          "openNotebook": true
+        }
+      }
+    }
+  ]
+}

docs/evidence/production-admission-20261017/selector-red.log

@@ -0,0 +1,30 @@
+✖ workflow path filters route real changed paths to their owning checks (25.90811ms)
+ℹ tests 1
+ℹ suites 0
+ℹ pass 0
+ℹ fail 1
+ℹ cancelled 0
+ℹ skipped 0
+ℹ todo 0
+ℹ duration_ms 293.145053
+
+✖ failing tests:
+
+test at scripts/production-rag-contract.test.mjs:371:1
+✖ workflow path filters route real changed paths to their owning checks (25.90811ms)
+  AssertionError [ERR_ASSERTION]: scripts/release-smoke.mjs
+
+  false !== true
+
+      at TestContext.<anonymous> (file:///workspace/LingxiLoop-development/scripts/production-rag-contract.test.mjs:401:12)
+      at Test.runInAsyncScope (node:async_hooks:227:14)
+      at Test.run (node:internal/test_runner/test:1382:25)
+      at Test.start (node:internal/test_runner/test:1242:17)
+      at startSubtestAfterBootstrap (node:internal/test_runner/harness:387:17) {
+    generatedMessage: false,
+    code: 'ERR_ASSERTION',
+    actual: false,
+    expected: true,
+    operator: 'strictEqual',
+    diff: 'simple'
+  }

docs/evidence/production-admission-20261017/smoke-red.log

@@ -0,0 +1,89 @@
+✖ production smoke accepts the exact public runtime and reaches authenticated WebSocket (73.756022ms)
+✖ production smoke rejects retired or mismatched runtime before tenant access (75.602386ms)
+✖ production smoke still enforces the concrete release commit (68.32594ms)
+✖ production smoke HTTP failure never prints server body or session secrets (66.769597ms)
+ℹ tests 4
+ℹ suites 0
+ℹ pass 0
+ℹ fail 4
+ℹ cancelled 0
+ℹ skipped 0
+ℹ todo 0
+ℹ duration_ms 430.742551
+
+✖ failing tests:
+
+test at scripts/release-smoke.test.mjs:63:1
+✖ production smoke accepts the exact public runtime and reaches authenticated WebSocket (73.756022ms)
+  AssertionError [ERR_ASSERTION]: ✓ /api/health → 200 (33ms)
+  ✓ /api/health/dependencies → 200 (0ms)
+  ✓ /api/meta → 200 (0ms)
+  Smoke failed: /api/meta did not report the LingxiLoop runtime-pending contract
+
+
+  1 !== 0
+
+      at TestContext.<anonymous> (file:///workspace/LingxiLoop-development/scripts/release-smoke.test.mjs:65:10)
+      at async Test.run (node:internal/test_runner/test:1389:7)
+      at async startSubtestAfterBootstrap (node:internal/test_runner/harness:387:3) {
+    generatedMessage: false,
+    code: 'ERR_ASSERTION',
+    actual: 1,
+    expected: 0,
+    operator: 'strictEqual',
+    diff: 'simple'
+  }
+
+test at scripts/release-smoke.test.mjs:69:1
+✖ production smoke rejects retired or mismatched runtime before tenant access (75.602386ms)
+  AssertionError [ERR_ASSERTION]: null
+
+  0 !== 1
+
+      at TestContext.<anonymous> (file:///workspace/LingxiLoop-development/scripts/release-smoke.test.mjs:72:12)
+      at async Test.run (node:internal/test_runner/test:1389:7)
+      at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
+    generatedMessage: false,
+    code: 'ERR_ASSERTION',
+    actual: 0,
+    expected: 1,
+    operator: 'strictEqual',
+    diff: 'simple'
+  }
+
+test at scripts/release-smoke.test.mjs:77:1
+✖ production smoke still enforces the concrete release commit (68.32594ms)
+  AssertionError [ERR_ASSERTION]: The input did not match the regular expression /commit mismatch/. Input:
+
+  '✓ /api/health → 200 (33ms)\n' +
+    '✓ /api/health/dependencies → 200 (0ms)\n' +
+    '✓ /api/meta → 200 (1ms)\n' +
+    'Smoke failed: /api/meta did not report the LingxiLoop runtime-pending contract\n'
+
+      at TestContext.<anonymous> (file:///workspace/LingxiLoop-development/scripts/release-smoke.test.mjs:80:10)
+      at async Test.run (node:internal/test_runner/test:1389:7)
+      at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
+    generatedMessage: true,
+    code: 'ERR_ASSERTION',
+    actual: '✓ /api/health → 200 (33ms)\n✓ /api/health/dependencies → 200 (0ms)\n✓ /api/meta → 200 (1ms)\nSmoke failed: /api/meta did not report the LingxiLoop runtime-pending contract\n',
+    expected: /commit mismatch/,
+    operator: 'match',
+    diff: 'simple'
+  }
+
+test at scripts/release-smoke.test.mjs:83:1
+✖ production smoke HTTP failure never prints server body or session secrets (66.769597ms)
+  AssertionError [ERR_ASSERTION]: The input was expected to not match the regular expression /synthetic-secret/. Input:
+
+  'Smoke failed: /api/health → 503: synthetic-secret-response-body\n'
+
+      at TestContext.<anonymous> (file:///workspace/LingxiLoop-development/scripts/release-smoke.test.mjs:87:10)
+      at async Test.run (node:internal/test_runner/test:1389:7)
+      at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
+    generatedMessage: true,
+    code: 'ERR_ASSERTION',
+    actual: 'Smoke failed: /api/health → 503: synthetic-secret-response-body\n',
+    expected: /synthetic-secret/,
+    operator: 'doesNotMatch',
+    diff: 'simple'
+  }

scripts/production-rag-contract.test.mjs

@@ -392,6 +392,8 @@ test('workflow path filters route real changed paths to their owning checks', ()
     ['src/lib/canvasLayout.ts', 'integration', 'server'],
     ['package-lock.json', 'integration', 'server'],
     ['.github/workflows/ci.yml', 'deploy_contract', ''],
+    ['scripts/release-smoke.mjs', 'deploy_contract', ''],
+    ['scripts/release-smoke.test.mjs', 'deploy_contract', ''],
     ['deploy/komodo/lingxiloop-app-a/compose.yml', 'deploy_contract', ''],
     ['third_party/open-notebook/tests/test_rag_contract.py', 'open_notebook', ''],
   ]) {