ops: move control-plane ownership and recovery skill to txy3 [skip deploy]

.agents/skills/operate-komodo-production/agents/openai.yaml

@@ -1,3 +1,3 @@
 interface:
   display_name: "Operate Komodo Production"
-  short_description: "Operate LingxiLoop production through Komodo"
+  short_description: "Operate Komodo releases and control-plane recovery on txy3"

.agents/skills/operate-komodo-production/references/control-plane.md

@@ -0,0 +1,57 @@
+# Control-plane maintenance
+
+Use for authorized bootstrap, outage recovery and host migration. It does not authorize deploying product stacks through SSH.
+
+## Current installation
+
+Control-plane migration was accepted on 2026-10-17. Core/MongoDB, keys and backup history run on txy3; cold-copy checksums matched. DNS points to txy3, public HTTPS and the original API credentials work, all three active agents are connected, and all eight product stacks are running. A `Backup Core Database` procedure executed through the public API completed successfully and wrote a new backup on txy3. Historical blocked-port evidence remains in the migration artifacts; it is no longer a current blocker.
+
+- Host: `txy3`, SSH `root`, `49.232.41.248`; Komodo Server name `manager`.
+- Reviewed manifests: `deploy/komodo/core/` and `deploy/komodo/control-ingress/`.
+- Host copies: `/opt/komodo/bootstrap/core/` and `/opt/komodo/bootstrap/control-ingress/`.
+- Core/Mongo containers: `komodo-core-1`, `komodo-mongo-1`; ingress: `komodo-traefik`.
+- Persistent state: `komodo_mongo-data`, `komodo_mongo-config`, `komodo_keys`, `control-ingress_traefik-acme`, and `/etc/komodo/backups`.
+- Core binds `127.0.0.1:9120`; Mongo has no published port. Only Traefik publishes 80/443. No Docker socket is exposed to Core or Traefik.
+- Periphery: `/usr/local/bin/periphery`, `/etc/komodo/periphery.config.toml`, `periphery.service`; outbound TLS to `https://ops.christmas1314.xyz`, `connect_as = "manager"`, inbound listener disabled.
+- Core and Mongo image digests are pinned to the migration source versions. Pull those exact images through the registered accelerator before creating containers.
+- Retired source: aly's Core/Mongo containers are stopped with restart policy `no`; `periphery.service` and its Komodo Server record are disabled. Its original data and keys remain for rollback. Original ingress manifests are in `/opt/komodo/rollback-20261016/`; live aly ingress serves only wegolibrary. A decommissioned aly must not become a prerequisite for future releases.
+
+## Validate or recover txy3
+
+Read host service/container status and compare installed manifest hashes before changing anything. A stopped, intact installation normally needs only `docker start`/`restart` for the exact affected bootstrap container, or `systemctl restart periphery`; these preserve the existing runtime environment.
+
+For reviewed Compose changes or recreation, copy only the reviewed non-secret manifests to the host paths above. Resolve the Sigillo project/environment, then use [bootstrap.cjs](../scripts/bootstrap.cjs):
+
+```powershell
+npm exec --yes --package=sigillo@0.14.2 -- sigillo run -p <project-id> -c prod -- C:\nvm4w\nodejs\node.exe .agents\skills\operate-komodo-production\scripts\bootstrap.cjs validate core
+npm exec --yes --package=sigillo@0.14.2 -- sigillo run -p <project-id> -c prod -- C:\nvm4w\nodejs\node.exe .agents\skills\operate-komodo-production\scripts\bootstrap.cjs validate control-ingress
+```
+
+`create` creates stopped containers for a restore. `up` starts/reconciles the named bootstrap project; it refuses to start Core without existing MongoDB data and the Core private key. Both use `--pull never` so a missing pinned image fails visibly. The helper accepts only `core` and `control-ingress` on `txy3`, uses batch SSH with normal host-key checks and sends only required environment variables as JSON on stdin to Python. It does not persist credentials in an env file or command arguments. The ACME contact defaults to `no-reply@lingxilearn.cn`. The destination certificate volume is external and must be restored before starting ingress.
+
+The manifests intentionally live outside product `resources.toml`: Core must remain recoverable while the Komodo API is down, and a product rollout must not replace the control plane that is executing it.
+
+If public ingress is unavailable, an authorized operator can inspect the local HTTPS API over SSH with `curl --resolve ops.christmas1314.xyz:443:127.0.0.1`. Keep TLS certificate verification enabled and supply authentication headers through stdin from a Sigillo child, never shell arguments or files. This proves only the local control plane; it does not prove public availability or agent connectivity.
+
+## Moving Core to another host
+
+Adapt host-bound helper code only after inspecting the requested new target. Do not treat this reference as permission for an unrequested migration.
+
+1. Inventory source containers, volumes, backup path, scheduled procedures, agents, stack IDs/images, webhook configuration and DNS. Identify unrelated services and shared ingress dependencies. Resolve DNS from an authoritative/public resolver or a remote host; local proxy fake-IP answers are not origin addresses.
+2. Prepare Docker, the exact source image versions, reviewed manifests and a **fresh** Periphery identity on the destination. For onboarding, use a short-lived non-privileged key held only in process memory, verify the server name/IP, revoke the onboarding key and run the agent under systemd. Retain the old host as a distinct server if it still serves another application.
+3. Separate destination control ingress from any source application ingress. Transfer only the required TLS certificate/ACME account and Core key material using encrypted host-to-host streams; no secret-bearing local archive or command output. Preserve ownership and restrictive file permissions. Verify external 80/443 reachability and HTTPS using `curl --resolve` **without disabling certificate verification** before taking the source down.
+4. Verify that the DNS account is available and the exact record can be changed. For `ops.christmas1314.xyz`, change only its Alibaba DNS A record to the new IP; keep `golib.christmas1314.xyz` on aly. Do not attempt this with a Cloudflare token. If login is needed, request it while finishing independent preparation; do not start an outage while access is unresolved.
+5. Wait for active Komodo operations to finish. Run `BackupCoreDatabase` through the API and require a terminal successful update. Stop source Core to prevent concurrent writes, then cleanly stop Mongo. Preserve a rollback copy. Copy the **stopped** Mongo volumes, Core keys and backup history through encrypted streams into empty destination volumes. Do not copy live WiredTiger files.
+6. Start destination Mongo/Core, require the original API credentials to work and compare resource IDs/counts. Keep the source Core/Mongo stopped. Change the authoritative DNS record once the new origin is ready, then verify DNS propagation and agent reconnection. Domain, API credentials, signing secret and procedure IDs remain stable, so Worker/CI callers need no redeployment.
+7. Check all managed servers and product stacks, compare image inventories, and run a narrow control operation such as `BackupCoreDatabase` or the backup procedure. Require its successful terminal update and new backup files on the destination. Test the public control URL, product route and any preserved source application route.
+8. Disable restart policies for retired source Core/Mongo. Remove the old Komodo route/network dependency from aly's ingress only after DNS cutover is verified. Keep stopped source data and rollback manifests unless deletion was specifically requested. Never run `down -v` or global prune.
+
+If the operator has already switched DNS and will fix cloud ingress separately, complete the authorized durable-data migration and verify the local authenticated HTTPS API and a successful backup update. Report public HTTPS and agent reconnection as pending until tested; do not replace TLS with HTTP or work around cloud ingress restrictions using product deployment tunnels.
+
+If validation fails before cutover, keep the source authoritative. After cutover, avoid rolling back to a stale database: stop new writes, assess changes and transfer current state back if needed before changing DNS. A rollback must leave exactly one writable Core.
+
+If rollback needs aly's agent, explicitly re-enable its Komodo Server and systemd service after restoring the authoritative Core. Do not revive retired services just because the legacy Server is disabled. Migrating Core does not migrate wegolibrary; aly shutdown will affect that separate application unless separately moved.
+
+## Completion evidence
+
+Save sanitized before/after inventories and update IDs, DNS A/AAAA answers, certificate-validated HTTP status, destination service/restart state, and checks that the old Core is stopped and unrelated services remain healthy. Do not save raw environments, database dumps, private keys or full API resources as local evidence.

.agents/skills/operate-komodo-production/scripts/bootstrap.cjs

@@ -0,0 +1,33 @@
+const { spawnSync } = require('node:child_process')
+const { existsSync } = require('node:fs')
+
+const ssh = 'C:\\Windows\\System32\\OpenSSH\\ssh.exe'
+const [action, target, ...extra] = process.argv.slice(2)
+const keys = target === 'core'
+  ? ['KOMODO_DATABASE_USERNAME', 'KOMODO_DATABASE_PASSWORD', 'KOMODO_ADMIN_PASSWORD', 'KOMODO_JWT_SECRET', 'KOMODO_WEBHOOK_SECRET']
+  : []
+if (!['validate', 'create', 'up'].includes(action) || !['core', 'control-ingress'].includes(target) || extra.length) {
+  console.error('Usage: bootstrap.cjs <validate|create|up> <core|control-ingress> (txy3 only).')
+  process.exit(1)
+} else if (keys.some(key => !process.env[key])) {
+  console.error('Missing bootstrap credentials; run through Sigillo prod.')
+  process.exit(1)
+} else if (!existsSync(ssh)) {
+  console.error('Expected Windows OpenSSH binary is missing.')
+  process.exit(1)
+} else {
+  const env = target === 'core'
+    ? Object.fromEntries(keys.map(key => [key, process.env[key]]))
+    : { ACME_EMAIL: process.env.ACME_EMAIL || 'no-reply@lingxilearn.cn' }
+  const command = ['docker', 'compose', '-f', `/opt/komodo/bootstrap/${target}/compose.yml`,
+    ...(action === 'validate' ? ['config', '--quiet'] : action === 'create' ? ['create', '--pull', 'never'] : ['up', '-d', '--pull', 'never'])]
+  // JSON stdin avoids putting credentials in shell source, process arguments, or env files.
+  const requireState = action === 'up' && target === 'core'
+  const remote = "python3 -c 'import json,os,subprocess,sys; p=json.load(sys.stdin); assert not p[\"requireState\"] or all(os.path.getsize(f)>0 if os.path.isfile(f) else False for f in [\"/var/lib/docker/volumes/komodo_mongo-data/_data/WiredTiger\",\"/var/lib/docker/volumes/komodo_keys/_data/core.key\"]), \"Refusing to initialize empty production Core: restore MongoDB and Core keys first\"; os.environ.update(p[\"env\"]); sys.exit(subprocess.call(p[\"command\"]))'"
+  const result = spawnSync(ssh, ['-o', 'BatchMode=yes', '-o', 'ConnectTimeout=15', 'txy3', remote], {
+    input: JSON.stringify({ env, command, requireState }), stdio: ['pipe', 'inherit', 'inherit'],
+  })
+  if (result.error) console.error('Unable to start control-plane bootstrap over SSH.')
+  if (result.status === 0) console.log(`txy3 ${target}: ${action} completed`)
+  process.exit(result.status ?? 1)
+}

.agents/skills/operate-komodo-production/scripts/bootstrap.test.cjs

@@ -0,0 +1,50 @@
+const assert = require('node:assert/strict')
+const fs = require('node:fs')
+const vm = require('node:vm')
+
+// Fail closed on wrong targets, missing credentials, missing SSH, and child failures;
+// credentials must reach only stdin, never command arguments or a local file.
+function check(args, env, status = 0, exists = true) {
+  let invocation
+  let exitCode
+  let error
+  let output
+  vm.runInNewContext(fs.readFileSync(`${__dirname}/bootstrap.cjs`, 'utf8'), {
+    require: name => name === 'node:fs' ? { existsSync: () => exists } : {
+      spawnSync: (...args) => { invocation = args; return { status } },
+    },
+    process: { env, argv: ['node', 'bootstrap.cjs', ...args], exit: code => { exitCode = code } },
+    console: { error: message => { error = message }, log: message => { output = message } },
+  })
+  return { invocation, exitCode, error, output }
+}
+const env = Object.fromEntries(['KOMODO_DATABASE_USERNAME', 'KOMODO_DATABASE_PASSWORD', 'KOMODO_ADMIN_PASSWORD', 'KOMODO_JWT_SECRET', 'KOMODO_WEBHOOK_SECRET'].map(key => [key, `test-${key}-$'\n`]))
+env.UNRELATED_SECRET = 'must-not-transfer'
+const ok = check(['validate', 'core'], env, 7)
+assert.equal(ok.exitCode, 7)
+assert.equal(ok.invocation[0], 'C:\\Windows\\System32\\OpenSSH\\ssh.exe')
+assert.deepEqual(Array.from(ok.invocation[1].slice(0, 5)), ['-o', 'BatchMode=yes', '-o', 'ConnectTimeout=15', 'txy3'])
+assert(!ok.invocation[1].join(' ').includes('test-'))
+const payload = JSON.parse(ok.invocation[2].input)
+assert.deepEqual(payload.command, ['docker', 'compose', '-f', '/opt/komodo/bootstrap/core/compose.yml', 'config', '--quiet'])
+assert.equal(payload.requireState, false)
+assert.deepEqual(payload.env, Object.fromEntries(Object.entries(env).filter(([key]) => key !== 'UNRELATED_SECRET')))
+assert.deepEqual(Array.from(ok.invocation[2].stdio), ['pipe', 'inherit', 'inherit'])
+assert.deepEqual(JSON.parse(check(['create', 'core'], env).invocation[2].input).command,
+  ['docker', 'compose', '-f', '/opt/komodo/bootstrap/core/compose.yml', 'create', '--pull', 'never'])
+assert.equal(JSON.parse(check(['up', 'core'], env).invocation[2].input).requireState, true)
+const ingress = check(['up', 'control-ingress'], {})
+assert.equal(ingress.exitCode, 0)
+assert.equal(ingress.output, 'txy3 control-ingress: up completed')
+assert.deepEqual(JSON.parse(ingress.invocation[2].input), {
+  env: { ACME_EMAIL: 'no-reply@lingxilearn.cn' },
+  requireState: false,
+  command: ['docker', 'compose', '-f', '/opt/komodo/bootstrap/control-ingress/compose.yml', 'up', '-d', '--pull', 'never'],
+})
+for (const result of [check(['up', 'core'], {}), check(['down', 'core'], env), check(['up', 'product'], env), check(['up', 'core', 'aly'], env), check(['up', 'core'], env, 0, false)]) {
+  assert.equal(result.exitCode, 1)
+  assert.equal(result.invocation, undefined)
+  assert(result.error && !result.error.includes('test-'))
+}
+assert.equal(check(['up', 'core'], env, null).exitCode, 1)
+console.log('Control-plane bootstrap checks passed')

deploy/komodo/README.md

@@ -1,8 +1,27 @@
 # Komodo production
 
-Komodo v2.3.3 is the only production deployment control plane. `aly` runs
-Core and MongoDB; `txy2` is `shanghai-a`, and `txy` is `shanghai-b`. Periphery
-runs as a systemd service on all three hosts and connects outbound to Core.
+**Control-plane migration accepted (2026-10-17):** Public HTTPS/API, all three
+active agents, all eight running stacks and a successful database-backup
+procedure have been verified on txy3. DNS points to `49.232.41.248`. Evidence
+is in `artifacts/control-plane-migration-20261016/` from the repository root.
+The independent wegolibrary application remains on aly and is outside this
+deployment-control-plane migration.
+
+Komodo v2.3.3 is the only production deployment control plane. `txy3`
+(`49.232.41.248`, Komodo `manager`) runs Core, MongoDB and control ingress;
+`txy2` is `shanghai-a`, and `txy` is `shanghai-b`. The legacy `aly` Server
+record and its Periphery service are disabled; its existing wegolibrary is
+independent. Periphery runs as a systemd service on the three active hosts
+and connects outbound to Core. Releases do not depend on aly staying online.
+
+The stable origin is `https://ops.christmas1314.xyz`; its DNS is managed in
+Alibaba Cloud DNS. `core/` and `control-ingress/` are bootstrap infrastructure
+on txy3 at `/opt/komodo/bootstrap/`, outside the product resource sync.
+`aly-ingress/` preserves wegolibrary on aly, including the historical Compose
+project name `control-ingress` and certificate volume `aly-ingress_traefik-acme`.
+Do not apply the txy3 ingress manifest to aly. See the
+[control-plane maintenance skill](../../.agents/skills/operate-komodo-production/references/control-plane.md)
+for bootstrap, backups, cutover and rollback.
 
 `resources.toml` owns the LingxiLoop stacks. Runtime variables and secrets are
 stored in Komodo after being supplied from Sigillo; no `.env` file is committed.

deploy/komodo/aly-ingress/compose.yml

@@ -0,0 +1,37 @@
+# Preserve the existing Compose project and certificate volume on aly.
+name: control-ingress
+
+services:
+  traefik:
+    image: accel.way2api.fun/docker.io/library/traefik:v3.7.1
+    container_name: aly-traefik
+    command:
+      - --providers.file.directory=/etc/traefik/dynamic
+      - --providers.file.watch=true
+      - --entrypoints.web.address=:80
+      - --entrypoints.web.http.redirections.entrypoint.to=websecure
+      - --entrypoints.web.http.redirections.entrypoint.scheme=https
+      - --entrypoints.websecure.address=:443
+      - --certificatesresolvers.letsencrypt.acme.email=${ACME_EMAIL:?ACME_EMAIL is required}
+      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
+      - --certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web
+      - --log.level=WARN
+      - --accesslog=true
+    ports:
+      - 80:80
+      - 443:443
+    volumes:
+      - ./dynamic.yml:/etc/traefik/dynamic/routes.yml:ro
+      - traefik-acme:/letsencrypt
+    networks: [wegolibrary]
+    restart: unless-stopped
+
+volumes:
+  traefik-acme:
+    external: true
+    name: aly-ingress_traefik-acme
+
+networks:
+  wegolibrary:
+    external: true
+    name: wegolibrary_default

deploy/komodo/aly-ingress/dynamic.yml

@@ -0,0 +1,11 @@
+http:
+  routers:
+    wegolibrary:
+      rule: Host(`golib.christmas1314.xyz`)
+      entryPoints: [websecure]
+      service: wegolibrary
+      tls: { certResolver: letsencrypt }
+  services:
+    wegolibrary:
+      loadBalancer:
+        servers: [{ url: http://frontend:80 }]

deploy/komodo/control-ingress/compose.yml

@@ -3,7 +3,7 @@ name: control-ingress
 services:
   traefik:
     image: accel.way2api.fun/docker.io/library/traefik:v3.7.1
-    container_name: aly-traefik
+    container_name: komodo-traefik
     command:
       - --providers.file.directory=/etc/traefik/dynamic
       - --providers.file.watch=true
@@ -19,23 +19,18 @@ services:
     ports:
       - 80:80
       - 443:443
-    extra_hosts:
-      - host.docker.internal:host-gateway
     volumes:
       - ./dynamic.yml:/etc/traefik/dynamic/routes.yml:ro
       - traefik-acme:/letsencrypt
-    networks: [default, komodo, wegolibrary]
+    networks: [komodo]
     restart: unless-stopped
 
 volumes:
   traefik-acme:
     external: true
-    name: aly-ingress_traefik-acme
+    name: control-ingress_traefik-acme
 
 networks:
   komodo:
     external: true
     name: komodo_default
-  wegolibrary:
-    external: true
-    name: wegolibrary_default

deploy/komodo/control-ingress/dynamic.yml

@@ -5,15 +5,7 @@ http:
       entryPoints: [websecure]
       service: komodo
       tls: { certResolver: letsencrypt }
-    wegolibrary:
-      rule: Host(`golib.christmas1314.xyz`)
-      entryPoints: [websecure]
-      service: wegolibrary
-      tls: { certResolver: letsencrypt }
   services:
     komodo:
       loadBalancer:
         servers: [{ url: http://core:9120 }]
-    wegolibrary:
-      loadBalancer:
-        servers: [{ url: http://frontend:80 }]

deploy/komodo/core/compose.yml

@@ -2,7 +2,7 @@ name: komodo
 
 services:
   mongo:
-    image: mongo:8.0
+    image: accel.way2api.fun/docker.io/library/mongo:8.0.32@sha256:6c108efd306c66a7eb58e0b4042c2fcffb4a4980dac111039c59fd714db6a69b
     command: --quiet --wiredTigerCacheSizeGB 0.25
     restart: unless-stopped
     environment:
@@ -15,7 +15,7 @@ services:
       komodo.skip: ""
 
   core:
-    image: ghcr.io/moghtech/komodo-core:2.3.3
+    image: accel.way2api.fun/ghcr.io/moghtech/komodo-core:2.3.3@sha256:bca73d0eee143066228fb9b80c38a5a2726e573ad2758f7aa92b5c91a8aef1c7
     init: true
     restart: unless-stopped
     depends_on: [mongo]