Task stable

.agents/skills/console-ui-e2e/SKILL.md

@@ -54,6 +54,10 @@ does not need that reference.
 
 ## Run
 
+Follow [test and CI progression](../debug-test/SKILL.md#test-and-ci-progression)
+when sequencing suites or CI jobs. Fix a failure with its affected spec/group,
+then resume the remaining selection; use this skill for browser diagnosis.
+
 ```sh
 pixi run clean-env
 pixi run test-container-e2e --scope console.operations --case <registered-owner> --concurrency 1 --node-cpus 0.25 --cpus 8 --memory-mib 16384

.agents/skills/debug-test/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: debug-test
-description: Diagnose a non-browser failing test from first divergence to root cause.
+description: Run CROWDB test and CI jobs incrementally, and diagnose non-browser failures from first divergence to root cause.
 ---
 
 <!-- Copyright 2026-present Gian <crow.db@outlook.com> -->
@@ -11,6 +11,47 @@ description: Diagnose a non-browser failing test from first divergence to root c
 Do not increase timeouts, suppress errors, weaken assertions, add caller-side
 retries, or otherwise make only the symptom pass.
 
+## Test and CI progression
+
+For expensive suites or multiple jobs, preserve an ordered list of commands,
+completed results, logs, relevant source/image inputs, and the next unrun item.
+Use existing runner records where available.
+
+- Run until an error, then fix it before advancing the affected sequence. Do
+  not first run every job against a known defect. Prioritize bugs that make
+  subsequent tests slow, hang, or fail through the same dependency.
+- For container E2E, use `--fail-fast` to stop queued cases while letting
+  already owned fixtures finish and clean up. Read failure records throughout
+  the log, not only its tail or completed-case count. Do not interrupt both a
+  launcher and its child; forwarded duplicate signals can break cleanup.
+- Reproduce and repair the failure with its test or smallest meaningful test
+  group. Run that group's regression and relevant quality gates after the fix.
+- Resume the remaining tests/jobs from the saved position. Retain earlier
+  passing results; repeat them only when the fix changes their behavior or
+  inputs. Preserve original ordering and coverage when selecting a remainder.
+- Independent work already in flight may finish when it is unaffected. Stop
+  dispatching work that depends on the defect or would repeat its slowdown.
+- Complete shared build prerequisites before parallel test jobs. Never run
+  concurrent builders against the same output directory; use independent
+  output directories when parallel builds are necessary. Perform shared
+  environment cleanup before dispatch; do not repeat `clean-env` while other
+  local jobs still own running fixtures. Host browser fixtures that require
+  stable Docker networks also share that constraint across local CI jobs;
+  keep their browser phases separate from other jobs' bridge creation/removal.
+- Before long suites, inspect actual fixture slots and declared CPU/memory
+  costs. An outer parallel job count does not prove cases run concurrently.
+  Distinguish build, fixture startup, client execution, and cleanup time.
+- Reuse valid builds and images. Rebuild when their relevant inputs change;
+  avoid unrelated feature/profile changes and repeated rebuilds while debugging.
+- After the requested jobs have each been repaired and locally verified, run
+  the requested or required overall verification once as a closing gate. If
+  that gate finds another defect, repair its group and resume the affected
+  remainder rather than restarting valid completed groups from the beginning.
+- Report actual coverage, failures, and unrun/blocked work. A focused pass does
+  not by itself establish that a whole suite or CI job passed.
+
+## Diagnose a failure
+
 1. Check relevant environment variables, stale processes, logs, persisted
    state, and build artifacts.
 2. Reproduce the exact test. If it passes alone, compare the affected suite

.github/workflows/ci.yml

@@ -305,7 +305,7 @@ jobs:
         name: verified-oci-image
         path: target
     - name: Import shared release image
-      run: pixi run image-import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
+      run: pixi run --manifest-path container/oci-image/pixi.toml import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
     - name: Run complete S3 acceptance
       run: pixi run test-e2e-s3
     - name: Capture runner state on failure
@@ -374,7 +374,7 @@ jobs:
         name: verified-oci-image
         path: target
     - name: Import shared release image
-      run: pixi run image-import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
+      run: pixi run --manifest-path container/oci-image/pixi.toml import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
     - name: Run Iceberg acceptance
       env:
         E2E_PARTITION: ${{ matrix.partition }}
@@ -597,7 +597,7 @@ jobs:
         pixi run sudo apt-get update
         pixi run sudo apt-get install -y protobuf-compiler
     - name: Import the shared verified image
-      run: pixi run image-import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
+      run: pixi run --manifest-path container/oci-image/pixi.toml import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
     - name: Verify centralized KV fixtures and protocol layer
       run: |
         pixi run test-container-e2e-fixture
@@ -650,7 +650,7 @@ jobs:
         name: verified-oci-image
         path: target
     - name: Import formal release image
-      run: pixi run image-import target/crowdb.oci.tar --image crowdb-node:dev
+      run: pixi run --manifest-path container/oci-image/pixi.toml import target/crowdb.oci.tar --image crowdb-node:dev
     - name: Verify installed system browser
       run: |
         for b in /snap/bin/chromium /usr/bin/chromium /usr/bin/chromium-browser /usr/bin/google-chrome /usr/bin/google-chrome-stable /usr/bin/microsoft-edge; do
@@ -681,15 +681,17 @@ jobs:
     timeout-minutes: 60
     strategy:
       fail-fast: false
-      max-parallel: 3
+      max-parallel: 4
       matrix:
         include:
         - group: disk
           domains: diskdb diskio
         - group: chunk
           domains: chunkdb chunk-kv chunk-stream chunk-client
-        - group: platform
-          domains: dataset deployment
+        - group: dataset
+          domains: dataset
+        - group: deployment
+          domains: deployment
     permissions:
       contents: read
     steps:
@@ -709,7 +711,17 @@ jobs:
         pixi run sudo apt-get update
         pixi run sudo apt-get install -y protobuf-compiler
     - name: Import shared release image
-      run: pixi run image-import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
+      run: pixi run --manifest-path container/oci-image/pixi.toml import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
+    - uses: Swatinem/rust-cache@v2
+      with:
+        shared-key: container-storage-${{ matrix.group }}-${{ hashFiles('pixi.lock') }}
+        cache-targets: false
+        cache-directories: |
+          target/debug
+          sdk/target
+        cmd-format: "pixi run {0}"
+        cache-workspace-crates: true
+        cache-on-failure: true
     - name: Verify container domain
       env:
         E2E_DOMAINS: ${{ matrix.domains }}

.github/workflows/docker-preview.yml

@@ -7,6 +7,9 @@ concurrency:
   group: crowdb-container-preview-${{ github.ref_name }}
   cancel-in-progress: false
 
+env:
+  CROWDB_E2E_PRINT_SERVICE_ERRORS: "1"
+
 jobs:
   construct:
     runs-on: ubuntu-24.04
@@ -48,7 +51,7 @@ jobs:
         pixi run sudo apt-get clean
         pixi run df -h /
     - name: Stage runtime without a container daemon
-      run: pixi run stage-single-node-container
+      run: pixi run bash container/oci-image/stage.sh
     - name: Install locked standalone builder
       run: pixi install --locked --manifest-path container/oci-image/pixi.toml
     - name: Build OCI without Docker
@@ -87,7 +90,7 @@ jobs:
         name: verified-oci-image
         path: target
     - name: Import exact OCI artifact into Docker
-      run: pixi run image-import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
+      run: pixi run --manifest-path container/oci-image/pixi.toml import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
     - name: Verify Docker startup and recovery
       env:
         CROWDB_PREVIEW_TEST_ARTIFACTS: ${{ runner.temp }}/crowdb-preview-artifacts
@@ -99,19 +102,24 @@ jobs:
         pixi run clean-env
         pixi run --skip-deps test-single-node-container
         pixi run test-container-e2e --profile bridge-admission --concurrency 1 --node-cpus 1 --cpus 4 --memory-mib 6144
-        pixi run test-container-e2e --profile host --concurrency 1 --cpus 4 --memory-mib 4096
+        pixi run sudo --preserve-env=CROWDB_E2E_PRINT_SERVICE_ERRORS "$(command -v pixi)" run --as-is test-container-e2e --profile host --concurrency 1 --cpus 4 --memory-mib 4096
         pixi install --locked --manifest-path container/oci-image/pixi.toml -e runtime
-        pixi run test-container-e2e --profile containerd-cluster --concurrency 1 --node-cpus 1 --cpus 4 --memory-mib 6144
+        pixi run test-container-e2e --profile containerd-cluster --concurrency 1 --node-cpus 1 --cpus 4 --memory-mib 8192
     - name: Record verified digest
       id: artifact
       run: pixi run python container/oci-image/ci-receipt.py target/crowdb.oci.tar
+    - name: Export bounded host and container diagnostics
+      if: always()
+      run: |
+        pixi run sudo "$(command -v pixi)" run --as-is python e2e-test/publish.py --source .crowdb-runtime/artifacts --output target/oci-acceptance-public
+        pixi run sudo "$(command -v pixi)" run --as-is python e2e-test/publish.py --source "${{ runner.temp }}/crowdb-preview-artifacts" --output target/oci-preview-public
     - uses: actions/upload-artifact@v4
       if: failure()
       with:
         name: docker-acceptance-diagnostics
         path: |
-          ${{ runner.temp }}/crowdb-preview-artifacts
-          .crowdb-runtime/artifacts
+          target/oci-acceptance-public
+          target/oci-preview-public
         if-no-files-found: ignore
 
   containerd:
@@ -142,7 +150,7 @@ jobs:
     - name: Verify containerd host-network startup and recovery
       run: |
         set -o pipefail
-        pixi run test-container-e2e --profile containerd-host --concurrency 1 --cpus 3 --memory-mib 2048 --archive "$PWD/target/crowdb.oci.tar" 2>&1 | pixi run tee target/containerd-acceptance.log
+        pixi run sudo --preserve-env=CROWDB_E2E_PRINT_SERVICE_ERRORS "$(command -v pixi)" run --as-is test-container-e2e --profile containerd-host --concurrency 1 --cpus 3 --memory-mib 3072 --archive "$PWD/target/crowdb.oci.tar" 2>&1 | pixi run tee target/containerd-acceptance.log
     - uses: actions/upload-artifact@v4
       if: failure()
       with:
@@ -178,7 +186,7 @@ jobs:
         pixi run sudo apt-get update
         pixi run sudo apt-get install -y protobuf-compiler
     - name: Import the shared verified image
-      run: pixi run image-import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
+      run: pixi run --manifest-path container/oci-image/pixi.toml import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
     - name: Verify centralized KV fixtures and protocol layer
       run: |
         pixi run test-container-e2e-fixture

.github/workflows/iceberg-container-ecosystem.yml

@@ -38,7 +38,7 @@ jobs:
           pixi-version: latest
       - uses: Swatinem/rust-cache@v2
       - name: Build formal release image
-        run: pixi run build-single-node-container
+        run: pixi run oci-build --load
       - name: Prepare locked clients
         run: |
           pixi install --locked -e iceberg-ecosystem

.github/workflows/release-container.yml

@@ -7,6 +7,9 @@ concurrency:
   group: crowdb-container-${{ github.ref_name }}
   cancel-in-progress: false
 
+env:
+  CROWDB_E2E_PRINT_SERVICE_ERRORS: "1"
+
 jobs:
   construct:
     runs-on: ubuntu-24.04
@@ -50,7 +53,7 @@ jobs:
         pixi run sudo apt-get clean
         pixi run df -h /
     - name: Stage runtime without a container daemon
-      run: pixi run stage-single-node-container
+      run: pixi run bash container/oci-image/stage.sh
     - name: Install locked standalone builder
       run: pixi install --locked --manifest-path container/oci-image/pixi.toml
     - name: Build OCI without Docker
@@ -89,7 +92,7 @@ jobs:
         name: verified-oci-image
         path: target
     - name: Import exact OCI artifact into Docker
-      run: pixi run image-import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
+      run: pixi run --manifest-path container/oci-image/pixi.toml import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
     - name: Verify Docker startup and recovery
       env:
         CROWDB_PREVIEW_TEST_ARTIFACTS: ${{ runner.temp }}/crowdb-preview-artifacts
@@ -101,19 +104,24 @@ jobs:
         pixi run clean-env
         pixi run --skip-deps test-single-node-container
         pixi run test-container-e2e --profile bridge-admission --concurrency 1 --node-cpus 1 --cpus 4 --memory-mib 6144
-        pixi run test-container-e2e --profile host --concurrency 1 --cpus 4 --memory-mib 4096
+        pixi run sudo --preserve-env=CROWDB_E2E_PRINT_SERVICE_ERRORS "$(command -v pixi)" run --as-is test-container-e2e --profile host --concurrency 1 --cpus 4 --memory-mib 4096
         pixi install --locked --manifest-path container/oci-image/pixi.toml -e runtime
-        pixi run test-container-e2e --profile containerd-cluster --concurrency 1 --node-cpus 1 --cpus 4 --memory-mib 6144
+        pixi run test-container-e2e --profile containerd-cluster --concurrency 1 --node-cpus 1 --cpus 4 --memory-mib 8192
     - name: Record verified digest
       id: artifact
       run: pixi run python container/oci-image/ci-receipt.py target/crowdb.oci.tar
+    - name: Export bounded host and container diagnostics
+      if: always()
+      run: |
+        pixi run sudo "$(command -v pixi)" run --as-is python e2e-test/publish.py --source .crowdb-runtime/artifacts --output target/oci-acceptance-public
+        pixi run sudo "$(command -v pixi)" run --as-is python e2e-test/publish.py --source "${{ runner.temp }}/crowdb-preview-artifacts" --output target/oci-preview-public
     - uses: actions/upload-artifact@v4
       if: failure()
       with:
         name: docker-acceptance-diagnostics
         path: |
-          ${{ runner.temp }}/crowdb-preview-artifacts
-          .crowdb-runtime/artifacts
+          target/oci-acceptance-public
+          target/oci-preview-public
         if-no-files-found: ignore
 
   containerd:
@@ -144,7 +152,7 @@ jobs:
     - name: Verify containerd host-network startup and recovery
       run: |
         set -o pipefail
-        pixi run test-container-e2e --profile containerd-host --concurrency 1 --cpus 3 --memory-mib 2048 --archive "$PWD/target/crowdb.oci.tar" 2>&1 | pixi run tee target/containerd-acceptance.log
+        pixi run sudo --preserve-env=CROWDB_E2E_PRINT_SERVICE_ERRORS "$(command -v pixi)" run --as-is test-container-e2e --profile containerd-host --concurrency 1 --cpus 3 --memory-mib 3072 --archive "$PWD/target/crowdb.oci.tar" 2>&1 | pixi run tee target/containerd-acceptance.log
     - uses: actions/upload-artifact@v4
       if: failure()
       with:
@@ -181,7 +189,7 @@ jobs:
         pixi run sudo apt-get update
         pixi run sudo apt-get install -y protobuf-compiler
     - name: Import the shared verified image
-      run: pixi run image-import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
+      run: pixi run --manifest-path container/oci-image/pixi.toml import "$PWD/target/crowdb.oci.tar" --image crowdb-node:dev
     - name: Verify centralized KV fixtures and protocol layer
       run: |
         pixi run test-container-e2e-fixture

AGENTS.md

@@ -52,7 +52,8 @@ and transport exposed through FFI.
   aliases or compatibility symlinks such as `.devin`.
 - Code changes: `/coding`; add `/console-ui-e2e` only for visible UI or
   Playwright work.
-- Test diagnosis: `/debug-test`, or `/console-ui-e2e` for browser/UI failures.
+- Test execution and CI progression: `/debug-test`; use `/console-ui-e2e` for
+  browser/UI diagnosis.
 - Docs: one matched `/doc-*` guide; `/doc` only for general or unclear targets.
 - Requirements: `/implement-requirement`; open the detail directly and consult
   the backlog index only for selection, ordering, or status.