From 204602c16c3f26cc220128a0974b28a031f394f1 Mon Sep 17 00:00:00 2001 From: Lyyzka <144704183+lyyzka@users.noreply.github.com> Date: Sat, 17 Oct 2026 19:07:35 +0800 Subject: [PATCH 1/2] ops: document and preserve txy3 control-plane migration [skip deploy] --- .../skills/operate-komodo-production/SKILL.md | 89 ++++++++++--------- .../agents/openai.yaml | 2 +- .../references/control-plane.md | 57 ++++++++++++ .../scripts/bootstrap.cjs | 33 +++++++ .../scripts/bootstrap.test.cjs | 50 +++++++++++ deploy/komodo/README.md | 36 +++++--- deploy/komodo/aly-ingress/compose.yml | 37 ++++++++ deploy/komodo/aly-ingress/dynamic.yml | 11 +++ deploy/komodo/control-ingress/compose.yml | 11 +-- deploy/komodo/control-ingress/dynamic.yml | 8 -- deploy/komodo/core/compose.yml | 4 +- scripts/production-rag-contract.test.mjs | 5 +- 12 files changed, 269 insertions(+), 74 deletions(-) create mode 100644 .agents/skills/operate-komodo-production/references/control-plane.md create mode 100644 .agents/skills/operate-komodo-production/scripts/bootstrap.cjs create mode 100644 .agents/skills/operate-komodo-production/scripts/bootstrap.test.cjs create mode 100644 deploy/komodo/aly-ingress/compose.yml create mode 100644 deploy/komodo/aly-ingress/dynamic.yml diff --git a/.agents/skills/operate-komodo-production/SKILL.md b/.agents/skills/operate-komodo-production/SKILL.md index 02d51fb0b..9df3f067b 100644 --- a/.agents/skills/operate-komodo-production/SKILL.md +++ b/.agents/skills/operate-komodo-production/SKILL.md @@ -1,69 +1,76 @@ --- name: operate-komodo-production -description: Operate LingxiLoop production through Komodo, Sigillo, Wrangler, GitHub, Docker validation, and HTTP checks for releases, verification, recovery, and targeted cleanup. +description: Release, inspect, and recover LingxiLoop production through Komodo on txy3 and Sigillo. Includes control-plane host maintenance, migration, and the associated ingress and DNS. --- # Operate Komodo Production -Work from the LingxiLoop repository root. Use Komodo CLI/API state for every deployment and verification. Do not use the Komodo UI, browser automation, host SSH, or direct Docker commands to deploy product Compose files. +Work from the LingxiLoop repository root. Product releases use Komodo CLI/API state, not the Komodo UI or direct host Compose deployments. For an authorized control-plane bootstrap, recovery, or host migration, read [control-plane maintenance](references/control-plane.md); those operations need SSH because Core cannot orchestrate its own outage. -## CLI and credentials +## Ownership and topology -- Use the official Komodo CLI v2.3.3. On this Windows host, use [scripts/km.cjs](scripts/km.cjs) through Sigillo; it runs the installed Linux binary through WSL without a container. -- Use `sigillo` for operator credentials, `wrangler` for Cloudflare, `gh` for GitHub, `git` for source control, `docker compose ... config` for local validation, and `curl` for public HTTP checks. -- Sigillo is the only persistent credential source. Resolve the current ID of its `LingxiLoop Control Plane` project with `sigillo projects`, and use its `prod` environment; never hardcode a project ID. -- Run authenticated commands through `sigillo run -p -c prod -- `. Resolve executable paths first. Map `KOMODO_API_KEY` and `KOMODO_API_SECRET` to `KOMODO_CLI_KEY` and `KOMODO_CLI_SECRET` only inside the ephemeral child process; set `KOMODO_CLI_HOST=https://ops.christmas1314.xyz` there. -- Never put secrets in arguments, files, local CLI profiles, logs, or chat. Keep Sigillo redaction enabled and never use raw secret output or disabled redaction. +**Migration accepted (2026-10-17):** Core, MongoDB, keys and backups run on txy3. Public HTTPS and authenticated API access are verified, `manager`/`shanghai-a`/`shanghai-b` are connected, all eight stacks are running, and the public API's `Backup Core Database` procedure completed successfully. DNS points to `49.232.41.248`. See `artifacts/control-plane-migration-20261016/after.json` and `migration-status.json` for evidence. This completed scope is the deployment control plane; the independent wegolibrary application remains on aly. -### Windows command +| SSH host | Komodo Server | Responsibility | +| --- | --- | --- | +| `txy3` (`49.232.41.248`) | `manager` | Komodo Core 2.3.3, MongoDB 8.0.32, control ingress, Core keys and backups | +| `txy2` (`182.254.156.84`) | `shanghai-a` | Product state and application A | +| `txy` (`111.229.65.23`) | `shanghai-b` | Public product ingress, knowledge, application B, landing, LingxiLit, uptime | +| `aly` (`47.93.133.55`) | `aly` (disabled) | Existing wegolibrary and its separate ingress; retired Core data retained for rollback; not a release dependency | -After resolving the current project ID with `sigillo projects` and checking `sigillo environments -p `, use this one command from the repository root (replace the project placeholder): +- The stable API/UI origin is `https://ops.christmas1314.xyz`. Its DNS is in **Alibaba Cloud DNS**, not the Cloudflare account used for LingxiLoop Workers. Keep this origin, signing secrets and resource IDs stable during host moves unless the user requests a domain change. +- Periphery 2.3.3 runs through systemd on the three active hosts, connecting outbound to Core. The aly service and legacy Server are disabled. Each host owns its own Periphery private key; do not clone agent identities. Preserve Core's `komodo_keys` volume when relocating Core. +- `deploy/komodo/resources.toml` owns the eight product stacks and the ordered `lingxiloop-production-rollout` procedure. Core and control ingress are bootstrap infrastructure outside that resource sync. +- `deploy/komodo/core/` and `control-ingress/` belong on **txy3**. `deploy/komodo/aly-ingress/` belongs on **aly** and preserves its historical Compose project name `control-ingress`, container `aly-traefik`, certificate volume and wegolibrary network. Do not install the txy3 ingress manifest on aly. +- Preserve `/opt/apps/wegolibrary` and its Compose resources. Memos, Arcane and the old Agent OS/harness are retired; do not recreate them. +- Preserve PostgreSQL product state, WuKongIM durable IM, Redis coordination, Open Notebook/SurrealDB ownership, authorization, signed callbacks and the shared LLM ledger. +- Keep the registered `accel.way2api.fun` GitHub/GHCR/image accelerator. Do not introduce another proxy or upgrade infrastructure versions as part of a migration. + +## Credentials and Windows CLI + +Sigillo is the persistent operator credential source. On this Windows host use the official CLI from `npm exec --yes --package=sigillo@0.14.2 -- sigillo ...`, without changing global PATH. Resolve `LingxiLoop Control Plane` using `projects`, then confirm its `prod` environment with `environments -p `. Never hardcode the project ID. + +Use the official Komodo CLI 2.3.3 through [scripts/km.cjs](scripts/km.cjs). It maps injected API credentials into the WSL child and fixes the stable HTTPS origin: ```powershell -sigillo run -p -c prod -- C:\nvm4w\nodejs\node.exe .agents\skills\operate-komodo-production\scripts\km.cjs list servers --format json +npm exec --yes --package=sigillo@0.14.2 -- sigillo run -p -c prod -- C:\nvm4w\nodejs\node.exe .agents\skills\operate-komodo-production\scripts\km.cjs list servers --format json ``` -Replace only the final CLI arguments for `--version`, `list stacks --format json`, `list procedures --format json`, or an authorized CLI operation. Preserve Windows backslashes in the executable and script paths. Use argument arrays rather than `--command`, inline shell assignments, or nested PowerShell quoting. Require the expected version/resource output as well as exit code zero; this Sigillo build can produce empty output when paths use forward slashes or its output is captured in a PowerShell pipeline. +Replace only the final CLI arguments. Resolve executable paths first, preserve Windows backslashes, and use argument arrays. Do not use `--command`, nested shell assignments, cached CLI logins or local credential profiles. Keep Sigillo redaction enabled. Never print secret values, copy them into arguments or local files, or emit full resource configurations containing environments. A zero exit code with empty output is not success: this Sigillo build can lose output when paths use forward slashes or its output is piped through PowerShell. -The wrapper uses absolute WSL/binary paths so Sigillo's reduced PATH cannot hide `wsl.exe`; maps injected API credentials in the child environment; and replaces any existing CLI entries in `WSLENV` with unflagged names. `/w` entries do not reach Linux. Missing credentials/binaries and CLI failures exit nonzero. Do not patch global PATH or fall back to cached CLI credentials. +The wrapper replaces existing Komodo names in `WSLENV` with unflagged entries; `/w` entries do not reach Linux. Do not patch global PATH to repair a command. -Maintain `.agents/skills/operate-komodo-production` as the repository source; synchronize its `SKILL.md` and `scripts/` to the discovered local copy at `C:\Users\34395\.codex\skills\operate-komodo-production` after changes. There is no automatic sync. Run `node .agents\skills\operate-komodo-production\scripts\km.test.cjs` and a Sigillo read-only CLI command after changing the wrapper. +Use a PTY for authenticated commands on this Windows host. If Sigillo stalls before spawning its child or reports `TlsInitializationFailed`, confirm the child did not start before ending/retrying that attempt. Do not leave duplicate migration jobs pending or disable redaction to diagnose transport errors. A CLI version response alone does not verify credential injection; require actual authenticated resource output. -## Mandatory preflight +## Preflight for the operation -Before a production mutation: +1. Inspect the working tree, affected manifests and live Komodo resources. Preserve unrelated local changes. +2. Confirm the Sigillo version, `me --json`, project/environment identity, and the wrapper's `--version` through Sigillo. List servers, stacks and procedures; require the expected names, addresses and versions. Releases require healthy `manager`, `shanghai-a` and `shanghai-b` connections. The retired aly record is intentionally disabled and must not block releases; require it online only for an operation that actually targets aly. +3. Check only tools used by this operation: `git`/`gh` for source and workflows, Docker for Compose validation, and Wrangler for Cloudflare. If Cloudflare is affected, run `wrangler whoami --json` through Sigillo, not a cached login. Resolve the authoritative DNS provider before changing any record. +4. If credentials or expected resources are missing, stop the dependent mutation and finish independent preparation. Ask only for the missing access or information; an already authorized migration/release does not need another approval. -1. Confirm `sigillo --version`, the wrapper's `--version` through Sigillo, `wrangler --version`, `gh --version`, `git --version`, and `docker --version`. -2. Confirm `sigillo me --json`, resolve `LingxiLoop Control Plane`, and confirm its `prod` environment. -3. Through `sigillo run`, confirm the Komodo host and identity by listing the named servers, stacks, and procedures with `km`; confirm Cloudflare with `wrangler whoami --json` when Cloudflare is in scope. -4. Stop if credentials are missing, the host is not `https://ops.christmas1314.xyz`, or the expected resources are absent. A local cached login does not satisfy this gate. +## Product release and verification -## Production ownership +1. Validate each changed Compose file with `docker compose -f config --quiet`, injecting credentials through Sigillo. Do not emit interpolated Compose output. +2. Use repository scripts and `git`/`gh` for authorized source/workflow operations. GitHub Actions publishes immutable images, pins manifests, deploys the Worker when required and triggers the signed rollout. A control-plane host move does not require an unrelated product release. +3. The repository trigger is `scripts/trigger-komodo-rollout.mjs`, run through Sigillo with its required webhook and API variables. It captures the Komodo update ID and polls `GetUpdate` for `status: Complete` and `success: true`. Do not assume the webhook URL exists in a particular credential store; verify variable presence without printing values. +4. For an authorized manual rollout, use `km run procedure lingxiloop-production-rollout -y` when supported by the installed CLI. Capture its update ID and verify completion through the API. HTTP acceptance alone is insufficient. +5. Inspect affected stack action states, individual service health, update history and narrow logs, then check the relevant HTTPS routes with `curl --fail-with-body --silent --show-error`. A successful update or running stack does not prove that its Workers are ready: require both Worker healthchecks, Web readiness, expected runtime/schema versions and actual image digests. Accept authentication redirects only where documented. -- Komodo v2.3.3 is the deployment control plane. `aly` hosts Core and MongoDB; `txy2` is `shanghai-a`; `txy` is `shanghai-b`. -- `deploy/komodo/resources.toml` owns production resources. The ordered procedure is `lingxiloop-production-rollout`. -- The main owned stacks are `server-b-ingress`, `lingxiloop-core-state`, `lingxiloop-knowledge-agent`, `lingxiloop-app-a`, `lingxiloop-app-b`, and `uptime`. The procedure also coordinates `landing` and `lingxilit`. -- Preserve `/opt/apps/wegolibrary` and all of its Compose resources. Memos and other retired resources must not be recreated. -- Preserve PostgreSQL product state, WuKongIM durable IM, Redis ephemeral coordination, the Open Notebook/SurrealDB schema, product authorization, signed callbacks, and the shared LLM ledger. -- Keep the registered GitHub/GHCR accelerator. Do not add an alternate proxy. +Before a runtime schema upgrade, verify a snapshot-consistent backup by restoring it into an isolated database, drain all old Workers and verify no live leases, then follow the ordered migration/deployment procedure. Compare installed schema provenance to the exact packaged schema. Never reset production state or restart an older Worker against an upgraded schema. -## Release and verification +Use existing API shapes from `scripts/trigger-komodo-rollout.mjs` and `workers/control-plane/src/komodo.ts`; consult the matching Komodo version's API for other operations. Never print complete secret-bearing API responses. -1. Inspect `git status`, the affected manifests, `deploy/komodo/resources.toml`, and the corresponding Komodo stack state. -2. Validate each changed Compose file with `docker compose -f config --quiet`. -3. Use repository scripts and `git`/`gh` for the authorized source and workflow operations. GitHub Actions publishes immutable commit-tagged images, pins manifests, deploys the Worker when required, and triggers the signed rollout. -4. For the repository release path, run `scripts/trigger-komodo-rollout.mjs` only through Sigillo. It triggers the signed procedure, obtains the Komodo update ID, and polls `/read/GetUpdate` until `status` is `Complete` and `success` is true. -5. For an explicitly authorized manual rollout, use `km run procedure lingxiloop-production-rollout -y` when supported by the installed v2 CLI. Capture the returned update and verify its terminal state through the Komodo API; webhook acceptance alone is not success. -6. Inspect only affected stacks, services, update history, and narrow logs. Then check the corresponding HTTPS route with `curl --fail-with-body --silent --show-error`. +## Recovery and cleanup -Use the repository's existing Komodo API shapes rather than inventing requests: `scripts/trigger-komodo-rollout.mjs` owns procedure update polling, and `workers/control-plane/src/komodo.ts` owns stack inspection, updates, logs, and actions. +Diagnose the failed Komodo update and owning service, fix forward, and rerun only the failed operation or the ordered procedure when its dependencies require it. Product Compose deployments still go through Komodo even when using SSH to diagnose infrastructure. Core outage recovery follows the linked maintenance reference. -## Recovery and cleanup +For durable IM/WAL recovery, stop the affected writer and retain a complete cold backup before changing its reader. Do not truncate committed data, rewrite checksums or replace the volume to make startup pass. Prefer a narrowly pinned, tested upstream repair and compare original WAL prefixes, immutable data files, node identity and committed/applied positions before and after recovery. + +Identify exact cleanup targets first. Stopping a superseded Core after a verified migration is part of cutover; deleting retained databases, backups, keys or unrelated resources needs explicit scope. Never run global prune or `compose down -v`. Do not start both old and new Core against independently writable databases. -- Diagnose failures from the Komodo update, affected stack action state, service state, and narrow logs. Fix forward and rerun only the failed owning operation or the ordered procedure when its dependency chain requires it. -- Before cleanup, list and identify every exact target in Komodo. Delete only explicitly authorized legacy targets and never run a global prune. -- Do not bypass Komodo by deploying Compose directly on a host. +## Maintain this skill -## Success criteria +`.agents/skills/operate-komodo-production` is the source. Synchronize `SKILL.md`, `references/`, `scripts/` and `agents/openai.yaml` to the discovered local copy at `C:\Users\34395\.codex\skills\operate-komodo-production`; there is no automatic sync. -Require Compose validation, a terminal successful Komodo update, healthy affected stack/services, and the corresponding HTTPS result. Treat expected authentication redirects as healthy only for routes documented that way in the repository. +Run `node .agents\skills\operate-komodo-production\scripts\km.test.cjs` and `bootstrap.test.cjs` after changing their wrappers, then a real Sigillo read-only command or bootstrap `validate`. Validate the skill with skill-creator's `quick_validate.py`. Record non-secret verification artifacts including update IDs, server identities, service/image inventories and HTTP results. diff --git a/.agents/skills/operate-komodo-production/agents/openai.yaml b/.agents/skills/operate-komodo-production/agents/openai.yaml index dd4079a5e..51bfb6b07 100644 --- a/.agents/skills/operate-komodo-production/agents/openai.yaml +++ b/.agents/skills/operate-komodo-production/agents/openai.yaml @@ -1,3 +1,3 @@ interface: display_name: "Operate Komodo Production" - short_description: "Operate LingxiLoop production through Komodo" + short_description: "Operate Komodo releases and control-plane recovery on txy3" diff --git a/.agents/skills/operate-komodo-production/references/control-plane.md b/.agents/skills/operate-komodo-production/references/control-plane.md new file mode 100644 index 000000000..0db2e9bd1 --- /dev/null +++ b/.agents/skills/operate-komodo-production/references/control-plane.md @@ -0,0 +1,57 @@ +# Control-plane maintenance + +Use for authorized bootstrap, outage recovery and host migration. It does not authorize deploying product stacks through SSH. + +## Current installation + +Control-plane migration was accepted on 2026-10-17. Core/MongoDB, keys and backup history run on txy3; cold-copy checksums matched. DNS points to txy3, public HTTPS and the original API credentials work, all three active agents are connected, and all eight product stacks are running. A `Backup Core Database` procedure executed through the public API completed successfully and wrote a new backup on txy3. Historical blocked-port evidence remains in the migration artifacts; it is no longer a current blocker. + +- Host: `txy3`, SSH `root`, `49.232.41.248`; Komodo Server name `manager`. +- Reviewed manifests: `deploy/komodo/core/` and `deploy/komodo/control-ingress/`. +- Host copies: `/opt/komodo/bootstrap/core/` and `/opt/komodo/bootstrap/control-ingress/`. +- Core/Mongo containers: `komodo-core-1`, `komodo-mongo-1`; ingress: `komodo-traefik`. +- Persistent state: `komodo_mongo-data`, `komodo_mongo-config`, `komodo_keys`, `control-ingress_traefik-acme`, and `/etc/komodo/backups`. +- Core binds `127.0.0.1:9120`; Mongo has no published port. Only Traefik publishes 80/443. No Docker socket is exposed to Core or Traefik. +- Periphery: `/usr/local/bin/periphery`, `/etc/komodo/periphery.config.toml`, `periphery.service`; outbound TLS to `https://ops.christmas1314.xyz`, `connect_as = "manager"`, inbound listener disabled. +- Core and Mongo image digests are pinned to the migration source versions. Pull those exact images through the registered accelerator before creating containers. +- Retired source: aly's Core/Mongo containers are stopped with restart policy `no`; `periphery.service` and its Komodo Server record are disabled. Its original data and keys remain for rollback. Original ingress manifests are in `/opt/komodo/rollback-20261016/`; live aly ingress serves only wegolibrary. A decommissioned aly must not become a prerequisite for future releases. + +## Validate or recover txy3 + +Read host service/container status and compare installed manifest hashes before changing anything. A stopped, intact installation normally needs only `docker start`/`restart` for the exact affected bootstrap container, or `systemctl restart periphery`; these preserve the existing runtime environment. + +For reviewed Compose changes or recreation, copy only the reviewed non-secret manifests to the host paths above. Resolve the Sigillo project/environment, then use [bootstrap.cjs](../scripts/bootstrap.cjs): + +```powershell +npm exec --yes --package=sigillo@0.14.2 -- sigillo run -p -c prod -- C:\nvm4w\nodejs\node.exe .agents\skills\operate-komodo-production\scripts\bootstrap.cjs validate core +npm exec --yes --package=sigillo@0.14.2 -- sigillo run -p -c prod -- C:\nvm4w\nodejs\node.exe .agents\skills\operate-komodo-production\scripts\bootstrap.cjs validate control-ingress +``` + +`create` creates stopped containers for a restore. `up` starts/reconciles the named bootstrap project; it refuses to start Core without existing MongoDB data and the Core private key. Both use `--pull never` so a missing pinned image fails visibly. The helper accepts only `core` and `control-ingress` on `txy3`, uses batch SSH with normal host-key checks and sends only required environment variables as JSON on stdin to Python. It does not persist credentials in an env file or command arguments. The ACME contact defaults to `no-reply@lingxilearn.cn`. The destination certificate volume is external and must be restored before starting ingress. + +The manifests intentionally live outside product `resources.toml`: Core must remain recoverable while the Komodo API is down, and a product rollout must not replace the control plane that is executing it. + +If public ingress is unavailable, an authorized operator can inspect the local HTTPS API over SSH with `curl --resolve ops.christmas1314.xyz:443:127.0.0.1`. Keep TLS certificate verification enabled and supply authentication headers through stdin from a Sigillo child, never shell arguments or files. This proves only the local control plane; it does not prove public availability or agent connectivity. + +## Moving Core to another host + +Adapt host-bound helper code only after inspecting the requested new target. Do not treat this reference as permission for an unrequested migration. + +1. Inventory source containers, volumes, backup path, scheduled procedures, agents, stack IDs/images, webhook configuration and DNS. Identify unrelated services and shared ingress dependencies. Resolve DNS from an authoritative/public resolver or a remote host; local proxy fake-IP answers are not origin addresses. +2. Prepare Docker, the exact source image versions, reviewed manifests and a **fresh** Periphery identity on the destination. For onboarding, use a short-lived non-privileged key held only in process memory, verify the server name/IP, revoke the onboarding key and run the agent under systemd. Retain the old host as a distinct server if it still serves another application. +3. Separate destination control ingress from any source application ingress. Transfer only the required TLS certificate/ACME account and Core key material using encrypted host-to-host streams; no secret-bearing local archive or command output. Preserve ownership and restrictive file permissions. Verify external 80/443 reachability and HTTPS using `curl --resolve` **without disabling certificate verification** before taking the source down. +4. Verify that the DNS account is available and the exact record can be changed. For `ops.christmas1314.xyz`, change only its Alibaba DNS A record to the new IP; keep `golib.christmas1314.xyz` on aly. Do not attempt this with a Cloudflare token. If login is needed, request it while finishing independent preparation; do not start an outage while access is unresolved. +5. Wait for active Komodo operations to finish. Run `BackupCoreDatabase` through the API and require a terminal successful update. Stop source Core to prevent concurrent writes, then cleanly stop Mongo. Preserve a rollback copy. Copy the **stopped** Mongo volumes, Core keys and backup history through encrypted streams into empty destination volumes. Do not copy live WiredTiger files. +6. Start destination Mongo/Core, require the original API credentials to work and compare resource IDs/counts. Keep the source Core/Mongo stopped. Change the authoritative DNS record once the new origin is ready, then verify DNS propagation and agent reconnection. Domain, API credentials, signing secret and procedure IDs remain stable, so Worker/CI callers need no redeployment. +7. Check all managed servers and product stacks, compare image inventories, and run a narrow control operation such as `BackupCoreDatabase` or the backup procedure. Require its successful terminal update and new backup files on the destination. Test the public control URL, product route and any preserved source application route. +8. Disable restart policies for retired source Core/Mongo. Remove the old Komodo route/network dependency from aly's ingress only after DNS cutover is verified. Keep stopped source data and rollback manifests unless deletion was specifically requested. Never run `down -v` or global prune. + +If the operator has already switched DNS and will fix cloud ingress separately, complete the authorized durable-data migration and verify the local authenticated HTTPS API and a successful backup update. Report public HTTPS and agent reconnection as pending until tested; do not replace TLS with HTTP or work around cloud ingress restrictions using product deployment tunnels. + +If validation fails before cutover, keep the source authoritative. After cutover, avoid rolling back to a stale database: stop new writes, assess changes and transfer current state back if needed before changing DNS. A rollback must leave exactly one writable Core. + +If rollback needs aly's agent, explicitly re-enable its Komodo Server and systemd service after restoring the authoritative Core. Do not revive retired services just because the legacy Server is disabled. Migrating Core does not migrate wegolibrary; aly shutdown will affect that separate application unless separately moved. + +## Completion evidence + +Save sanitized before/after inventories and update IDs, DNS A/AAAA answers, certificate-validated HTTP status, destination service/restart state, and checks that the old Core is stopped and unrelated services remain healthy. Do not save raw environments, database dumps, private keys or full API resources as local evidence. diff --git a/.agents/skills/operate-komodo-production/scripts/bootstrap.cjs b/.agents/skills/operate-komodo-production/scripts/bootstrap.cjs new file mode 100644 index 000000000..2026a98e4 --- /dev/null +++ b/.agents/skills/operate-komodo-production/scripts/bootstrap.cjs @@ -0,0 +1,33 @@ +const { spawnSync } = require('node:child_process') +const { existsSync } = require('node:fs') + +const ssh = 'C:\\Windows\\System32\\OpenSSH\\ssh.exe' +const [action, target, ...extra] = process.argv.slice(2) +const keys = target === 'core' + ? ['KOMODO_DATABASE_USERNAME', 'KOMODO_DATABASE_PASSWORD', 'KOMODO_ADMIN_PASSWORD', 'KOMODO_JWT_SECRET', 'KOMODO_WEBHOOK_SECRET'] + : [] +if (!['validate', 'create', 'up'].includes(action) || !['core', 'control-ingress'].includes(target) || extra.length) { + console.error('Usage: bootstrap.cjs (txy3 only).') + process.exit(1) +} else if (keys.some(key => !process.env[key])) { + console.error('Missing bootstrap credentials; run through Sigillo prod.') + process.exit(1) +} else if (!existsSync(ssh)) { + console.error('Expected Windows OpenSSH binary is missing.') + process.exit(1) +} else { + const env = target === 'core' + ? Object.fromEntries(keys.map(key => [key, process.env[key]])) + : { ACME_EMAIL: process.env.ACME_EMAIL || 'no-reply@lingxilearn.cn' } + const command = ['docker', 'compose', '-f', `/opt/komodo/bootstrap/${target}/compose.yml`, + ...(action === 'validate' ? ['config', '--quiet'] : action === 'create' ? ['create', '--pull', 'never'] : ['up', '-d', '--pull', 'never'])] + // JSON stdin avoids putting credentials in shell source, process arguments, or env files. + const requireState = action === 'up' && target === 'core' + const remote = "python3 -c 'import json,os,subprocess,sys; p=json.load(sys.stdin); assert not p[\"requireState\"] or all(os.path.getsize(f)>0 if os.path.isfile(f) else False for f in [\"/var/lib/docker/volumes/komodo_mongo-data/_data/WiredTiger\",\"/var/lib/docker/volumes/komodo_keys/_data/core.key\"]), \"Refusing to initialize empty production Core: restore MongoDB and Core keys first\"; os.environ.update(p[\"env\"]); sys.exit(subprocess.call(p[\"command\"]))'" + const result = spawnSync(ssh, ['-o', 'BatchMode=yes', '-o', 'ConnectTimeout=15', 'txy3', remote], { + input: JSON.stringify({ env, command, requireState }), stdio: ['pipe', 'inherit', 'inherit'], + }) + if (result.error) console.error('Unable to start control-plane bootstrap over SSH.') + if (result.status === 0) console.log(`txy3 ${target}: ${action} completed`) + process.exit(result.status ?? 1) +} diff --git a/.agents/skills/operate-komodo-production/scripts/bootstrap.test.cjs b/.agents/skills/operate-komodo-production/scripts/bootstrap.test.cjs new file mode 100644 index 000000000..7406b9be9 --- /dev/null +++ b/.agents/skills/operate-komodo-production/scripts/bootstrap.test.cjs @@ -0,0 +1,50 @@ +const assert = require('node:assert/strict') +const fs = require('node:fs') +const vm = require('node:vm') + +// Fail closed on wrong targets, missing credentials, missing SSH, and child failures; +// credentials must reach only stdin, never command arguments or a local file. +function check(args, env, status = 0, exists = true) { + let invocation + let exitCode + let error + let output + vm.runInNewContext(fs.readFileSync(`${__dirname}/bootstrap.cjs`, 'utf8'), { + require: name => name === 'node:fs' ? { existsSync: () => exists } : { + spawnSync: (...args) => { invocation = args; return { status } }, + }, + process: { env, argv: ['node', 'bootstrap.cjs', ...args], exit: code => { exitCode = code } }, + console: { error: message => { error = message }, log: message => { output = message } }, + }) + return { invocation, exitCode, error, output } +} +const env = Object.fromEntries(['KOMODO_DATABASE_USERNAME', 'KOMODO_DATABASE_PASSWORD', 'KOMODO_ADMIN_PASSWORD', 'KOMODO_JWT_SECRET', 'KOMODO_WEBHOOK_SECRET'].map(key => [key, `test-${key}-$'\n`])) +env.UNRELATED_SECRET = 'must-not-transfer' +const ok = check(['validate', 'core'], env, 7) +assert.equal(ok.exitCode, 7) +assert.equal(ok.invocation[0], 'C:\\Windows\\System32\\OpenSSH\\ssh.exe') +assert.deepEqual(Array.from(ok.invocation[1].slice(0, 5)), ['-o', 'BatchMode=yes', '-o', 'ConnectTimeout=15', 'txy3']) +assert(!ok.invocation[1].join(' ').includes('test-')) +const payload = JSON.parse(ok.invocation[2].input) +assert.deepEqual(payload.command, ['docker', 'compose', '-f', '/opt/komodo/bootstrap/core/compose.yml', 'config', '--quiet']) +assert.equal(payload.requireState, false) +assert.deepEqual(payload.env, Object.fromEntries(Object.entries(env).filter(([key]) => key !== 'UNRELATED_SECRET'))) +assert.deepEqual(Array.from(ok.invocation[2].stdio), ['pipe', 'inherit', 'inherit']) +assert.deepEqual(JSON.parse(check(['create', 'core'], env).invocation[2].input).command, + ['docker', 'compose', '-f', '/opt/komodo/bootstrap/core/compose.yml', 'create', '--pull', 'never']) +assert.equal(JSON.parse(check(['up', 'core'], env).invocation[2].input).requireState, true) +const ingress = check(['up', 'control-ingress'], {}) +assert.equal(ingress.exitCode, 0) +assert.equal(ingress.output, 'txy3 control-ingress: up completed') +assert.deepEqual(JSON.parse(ingress.invocation[2].input), { + env: { ACME_EMAIL: 'no-reply@lingxilearn.cn' }, + requireState: false, + command: ['docker', 'compose', '-f', '/opt/komodo/bootstrap/control-ingress/compose.yml', 'up', '-d', '--pull', 'never'], +}) +for (const result of [check(['up', 'core'], {}), check(['down', 'core'], env), check(['up', 'product'], env), check(['up', 'core', 'aly'], env), check(['up', 'core'], env, 0, false)]) { + assert.equal(result.exitCode, 1) + assert.equal(result.invocation, undefined) + assert(result.error && !result.error.includes('test-')) +} +assert.equal(check(['up', 'core'], env, null).exitCode, 1) +console.log('Control-plane bootstrap checks passed') diff --git a/deploy/komodo/README.md b/deploy/komodo/README.md index b1adb585d..d2748ea77 100644 --- a/deploy/komodo/README.md +++ b/deploy/komodo/README.md @@ -1,20 +1,32 @@ # Komodo production -Komodo v2.3.3 is the only production deployment control plane. `aly` runs -Core and MongoDB; `txy2` is `shanghai-a`, and `txy` is `shanghai-b`. Periphery -runs as a systemd service on all three hosts and connects outbound to Core. +**Control-plane migration accepted (2026-10-17):** Public HTTPS/API, all three +active agents, all eight running stacks and a successful database-backup +procedure have been verified on txy3. DNS points to `49.232.41.248`. Evidence +is in `artifacts/control-plane-migration-20261016/` from the repository root. +The independent wegolibrary application remains on aly and is outside this +deployment-control-plane migration. + +Komodo v2.3.3 is the only production deployment control plane. `txy3` +(`49.232.41.248`, Komodo `manager`) runs Core, MongoDB and control ingress; +`txy2` is `shanghai-a`, and `txy` is `shanghai-b`. The legacy `aly` Server +record and its Periphery service are disabled; its existing wegolibrary is +independent. Periphery runs as a systemd service on the three active hosts +and connects outbound to Core. Releases do not depend on aly staying online. + +The stable origin is `https://ops.christmas1314.xyz`; its DNS is managed in +Alibaba Cloud DNS. `core/` and `control-ingress/` are bootstrap infrastructure +on txy3 at `/opt/komodo/bootstrap/`, outside the product resource sync. +`aly-ingress/` preserves wegolibrary on aly, including the historical Compose +project name `control-ingress` and certificate volume `aly-ingress_traefik-acme`. +Do not apply the txy3 ingress manifest to aly. See the +[control-plane maintenance skill](../../.agents/skills/operate-komodo-production/references/control-plane.md) +for bootstrap, backups, cutover and rollback. `resources.toml` owns the LingxiLoop stacks. Runtime variables and secrets are stored in Komodo after being supplied from Sigillo; no `.env` file is committed. -The release workflow pins immutable images, sends the checked-out resource file -through Komodo’s authenticated configuration API, verifies the pending sync, and -runs the ordered `lingxiloop-production-rollout` procedure. GitHub is the source of -truth; Core does not need to clone the repository to synchronize configuration. -Do not edit the ResourceSync file contents manually: the next release replaces -them with the reviewed repository version. Stack source files still use their -configured repositories on Periphery. The existing ResourceSync is owned by the -release script, so this file declares only its managed resources; declaring the -sync itself attempts to create a duplicate and can overwrite its API source. +The release workflow pins immutable images, syncs this resource file, and runs +the ordered `lingxiloop-production-rollout` procedure. Before applying `0027_learning_collaboration`, drain affected interactive, handoff, Mission and Canvas runs before changing the default rooms' members. diff --git a/deploy/komodo/aly-ingress/compose.yml b/deploy/komodo/aly-ingress/compose.yml new file mode 100644 index 000000000..e8efa4416 --- /dev/null +++ b/deploy/komodo/aly-ingress/compose.yml @@ -0,0 +1,37 @@ +# Preserve the existing Compose project and certificate volume on aly. +name: control-ingress + +services: + traefik: + image: accel.way2api.fun/docker.io/library/traefik:v3.7.1 + container_name: aly-traefik + command: + - --providers.file.directory=/etc/traefik/dynamic + - --providers.file.watch=true + - --entrypoints.web.address=:80 + - --entrypoints.web.http.redirections.entrypoint.to=websecure + - --entrypoints.web.http.redirections.entrypoint.scheme=https + - --entrypoints.websecure.address=:443 + - --certificatesresolvers.letsencrypt.acme.email=${ACME_EMAIL:?ACME_EMAIL is required} + - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json + - --certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web + - --log.level=WARN + - --accesslog=true + ports: + - 80:80 + - 443:443 + volumes: + - ./dynamic.yml:/etc/traefik/dynamic/routes.yml:ro + - traefik-acme:/letsencrypt + networks: [wegolibrary] + restart: unless-stopped + +volumes: + traefik-acme: + external: true + name: aly-ingress_traefik-acme + +networks: + wegolibrary: + external: true + name: wegolibrary_default diff --git a/deploy/komodo/aly-ingress/dynamic.yml b/deploy/komodo/aly-ingress/dynamic.yml new file mode 100644 index 000000000..11a97ebb4 --- /dev/null +++ b/deploy/komodo/aly-ingress/dynamic.yml @@ -0,0 +1,11 @@ +http: + routers: + wegolibrary: + rule: Host(`golib.christmas1314.xyz`) + entryPoints: [websecure] + service: wegolibrary + tls: { certResolver: letsencrypt } + services: + wegolibrary: + loadBalancer: + servers: [{ url: http://frontend:80 }] diff --git a/deploy/komodo/control-ingress/compose.yml b/deploy/komodo/control-ingress/compose.yml index 3e1971c0d..6fd987b86 100644 --- a/deploy/komodo/control-ingress/compose.yml +++ b/deploy/komodo/control-ingress/compose.yml @@ -3,7 +3,7 @@ name: control-ingress services: traefik: image: accel.way2api.fun/docker.io/library/traefik:v3.7.1 - container_name: aly-traefik + container_name: komodo-traefik command: - --providers.file.directory=/etc/traefik/dynamic - --providers.file.watch=true @@ -19,23 +19,18 @@ services: ports: - 80:80 - 443:443 - extra_hosts: - - host.docker.internal:host-gateway volumes: - ./dynamic.yml:/etc/traefik/dynamic/routes.yml:ro - traefik-acme:/letsencrypt - networks: [default, komodo, wegolibrary] + networks: [komodo] restart: unless-stopped volumes: traefik-acme: external: true - name: aly-ingress_traefik-acme + name: control-ingress_traefik-acme networks: komodo: external: true name: komodo_default - wegolibrary: - external: true - name: wegolibrary_default diff --git a/deploy/komodo/control-ingress/dynamic.yml b/deploy/komodo/control-ingress/dynamic.yml index 25cf93297..c83d44012 100644 --- a/deploy/komodo/control-ingress/dynamic.yml +++ b/deploy/komodo/control-ingress/dynamic.yml @@ -5,15 +5,7 @@ http: entryPoints: [websecure] service: komodo tls: { certResolver: letsencrypt } - wegolibrary: - rule: Host(`golib.christmas1314.xyz`) - entryPoints: [websecure] - service: wegolibrary - tls: { certResolver: letsencrypt } services: komodo: loadBalancer: servers: [{ url: http://core:9120 }] - wegolibrary: - loadBalancer: - servers: [{ url: http://frontend:80 }] diff --git a/deploy/komodo/core/compose.yml b/deploy/komodo/core/compose.yml index 8d68df568..ca00cfd95 100644 --- a/deploy/komodo/core/compose.yml +++ b/deploy/komodo/core/compose.yml @@ -2,7 +2,7 @@ name: komodo services: mongo: - image: mongo:8.0 + image: accel.way2api.fun/docker.io/library/mongo:8.0.32@sha256:6c108efd306c66a7eb58e0b4042c2fcffb4a4980dac111039c59fd714db6a69b command: --quiet --wiredTigerCacheSizeGB 0.25 restart: unless-stopped environment: @@ -15,7 +15,7 @@ services: komodo.skip: "" core: - image: ghcr.io/moghtech/komodo-core:2.3.3 + image: accel.way2api.fun/ghcr.io/moghtech/komodo-core:2.3.3@sha256:bca73d0eee143066228fb9b80c38a5a2726e573ad2758f7aa92b5c91a8aef1c7 init: true restart: unless-stopped depends_on: [mongo] diff --git a/scripts/production-rag-contract.test.mjs b/scripts/production-rag-contract.test.mjs index 2ffa15dcb..ed114d742 100644 --- a/scripts/production-rag-contract.test.mjs +++ b/scripts/production-rag-contract.test.mjs @@ -233,14 +233,15 @@ test('live chat streams balance independently across both nodes through unbuffer test('Komodo control plane and ingress keep management sockets private', () => { const manager = read('deploy/komodo/core/compose.yml') - const alyIngress = read('deploy/komodo/control-ingress/compose.yml') + const controlIngress = read('deploy/komodo/control-ingress/compose.yml') + const alyIngress = read('deploy/komodo/aly-ingress/compose.yml') const appIngress = read('deploy/komodo/server-b-ingress/compose.yml') const appRoutes = read('deploy/komodo/server-b-ingress/dynamic.yml') assert.match(manager, /komodo-core:2\.3\.3/) assert.match(manager, /127\.0\.0\.1:9120:9120/) assert.doesNotMatch(manager, /docker\.sock/) - assert.doesNotMatch(`${alyIngress}\n${appIngress}`, /docker\.sock|providers\.docker/) + assert.doesNotMatch(`${controlIngress}\n${alyIngress}\n${appIngress}`, /docker\.sock|providers\.docker/) assert.match(appIngress, /80:80[\s\S]*443:443/) for (const host of ['lingxilearn.cn', 'www.lingxilearn.cn', 'loop.lingxilearn.cn', 'im.lingxilearn.cn', 'openlit.lingxilearn.cn', 'uptime.lingxilearn.cn']) { assert.match(appRoutes, new RegExp(host.replaceAll('.', '\\.'))) From be25dcaf960bc9165a786413bd4044ddaa0af6dd Mon Sep 17 00:00:00 2001 From: Lyyzka <144704183+lyyzka@users.noreply.github.com> Date: Sat, 17 Oct 2026 19:08:56 +0800 Subject: [PATCH 2/2] docs: retain authenticated resource sync ownership [skip deploy] --- deploy/komodo/README.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/deploy/komodo/README.md b/deploy/komodo/README.md index d2748ea77..87241fa68 100644 --- a/deploy/komodo/README.md +++ b/deploy/komodo/README.md @@ -25,8 +25,15 @@ for bootstrap, backups, cutover and rollback. `resources.toml` owns the LingxiLoop stacks. Runtime variables and secrets are stored in Komodo after being supplied from Sigillo; no `.env` file is committed. -The release workflow pins immutable images, syncs this resource file, and runs -the ordered `lingxiloop-production-rollout` procedure. +The release workflow pins immutable images, sends the checked-out resource file +through Komodo’s authenticated configuration API, verifies the pending sync, and +runs the ordered `lingxiloop-production-rollout` procedure. GitHub is the source of +truth; Core does not need to clone the repository to synchronize configuration. +Do not edit the ResourceSync file contents manually: the next release replaces +them with the reviewed repository version. Stack source files still use their +configured repositories on Periphery. The existing ResourceSync is owned by the +release script, so this file declares only its managed resources; declaring the +sync itself attempts to create a duplicate and can overwrite its API source. Before applying `0027_learning_collaboration`, drain affected interactive, handoff, Mission and Canvas runs before changing the default rooms' members.