Conversation
Collaborator
Author
|
/ok to test 5da275b |
Collaborator
Author
|
/ok to test b36a1b9 |
Collaborator
Author
|
/ok to test 88ab705 |
Add an idempotent Python driver for the kind, NFS CSI, SSH, MariaDB, and Slinky fixture. Setup and start reconcile a complete running environment. Stop deletes the disposable cluster while preserving reusable host state and external NFS data. Capture the sanitized single-host feasibility study and implementation handoff. Add a bounded test action that builds and validates the deployment tarball, then runs one-node and two-node filesystem sweeps through SSH and Slurm. Run the SSH entry point on the host, and transfer the same archive to the Slinky login node for Slurm execution. Install the standard file utility in the test images, validate generated environments, retain diagnostic logs, and assert results and cleanup. Fix two SSH dispatch variable-shadowing bugs exposed by the regression cases.
Add full integration environment teardown Add an idempotent teardown lifecycle action that removes marker-owned NFS configuration and data while preserving installed tools. Track ownership for host state, loop devices, firewall rules, and locally built image tags. Use full teardown for CI cleanup and document its distinction from disposable stop. Run fixture lifecycle and sweep tests concurrently on amd64 and arm64, with a final job requiring both matrix legs to pass. Exercise idempotent setup and teardown, stop/start recovery, and enforce that sweeps run as the provisioned pre-sudo user instead of root. Validate exact one- and two-node workload metadata and ordered worker selection for SSH and Slurm. Log Slurm's per-cell host subset so the harness can prove prefix ordering. Copy completed results to the host, run extract-elbencho.sh, and require reports to expose both node counts. Document strengthened coverage and resource prerequisites, improve missing-command failure diagnostics, and retain bounded artifacts for troubleshooting.
Add an explicit sbx-shared storage backend alongside the existing NFS backend. Persist backend selection, limit automatic selection to recognized host capability gaps, and refuse backend changes until teardown. Run the SBX profile with the tested kind and Kubernetes versions, conditionally map the missing kmsg device, propagate the Docker SBX proxy CA, and expose a repository-backed directory through static RWX volumes. Prove cross-node and host visibility while retaining the same SSH, Slurm, tarball, sweep, result, and reporting coverage as the NFS profile. Use the digest-pinned upstream Elbencho image when release assets are unavailable in SBX. Package its dynamic runtime only in generated test artifacts, stage it for SSH workers, and keep third-party binaries out of the repository. Accept Elbencho's adjacent streamed JSON records and terminate loader-launched services by their listening port. Harden first-boot Slinky behavior by retrying only the recognized webhook startup race and refreshing the configless login client after accounting is ready. Add the reporting Python prerequisite and prevent shell startup noise from contaminating structured result discovery. Document backend selection, lifecycle and cleanup guarantees, environmental fidelity, and the Docker-SBX-specific compatibility behavior. Document the remaining filesystem sweep coverage gaps and rank future work across environment, command-line, and reporting behavior. Make the Docker SBX profile use a Kubernetes-compatible kubectl and reconcile Slinky values on every setup. Ignore terminating LoginSet pods during rollout and require ownership markers before mutating retained state or export directories. Package Elbencho with its discovered dynamic loader, use explicit SSH identities, and increase the bounded workload enough to produce measurable throughput. Handle valid all-zero scaling reports and add focused reporting and lifecycle regression tests. Restore configured Slurm worker order after allocation canonicalization so ORDER_NODES cell selection remains deterministic.
dbishop
force-pushed
the
add-integration-tests
branch
from
September 20, 2026 02:13
88ab705 to
84dd77f
Compare
Separate substrate and scenario selection, schedule scenarios deterministically, and cache deployment archives from one immutable source snapshot consumed by the existing archive builder. Add crash-recoverable SSH home transitions and fast regression coverage for selection, caching, reconciliation, and extensible assertions.
Exercise bounded Elbencho workloads through real SSH and Slurm paths on both CI architectures. Cover buffered and direct I/O, failure and resume, retained data, live capture, multidimensional sweeps, single-file and weighted-root behavior, shared SSH homes, and Slurm scheduling controls. Stage failure injection independently for each substrate and preserve its one-time marker through resume. Validate execution coordinates, lifecycle state, workload totals, native command semantics, reporting output, and scheduler evidence while allowing extensible artifacts and report fields. Isolate and clean scenario-owned storage, restage the SBX runtime after SSH home transitions, select NFS explicitly in CI, and document the expanded scenario catalog. Add specification and failure-staging regression tests.
Cover node-range parsing, Cartesian reification, configuration precedence, SSH host selection, destructive path guards, single-file constraints, weighted sizing, and Slurm argument boundaries with fast shell tests. Exercise reporting filters, CSV round trips, output modes, and live-report option validation without requiring a running fixture. Fix associative TEST_DIRS detection so legacy TEST_DIR cannot override an explicit map. Reject partially invalid node specifications before any execution is reified, and coerce cached report fields using resolved type annotations so numeric filters work after --from-csv. Make CSV input exclusive with raw result directories and reject malformed or no-match filters to prevent successful but misleading reports. Document the resulting reporting and fast-test contracts.
Keep host operator IDs out of pod-side storage setup and preserve all-squashed NFS ownership. Stage the SSH failure-injection delegate and runtime on the local coordinator so the wrapper can exercise the intended failure and resume lifecycle. Make fixture validation substrate-aware, including Slurm-only report extraction while SSH is unavailable. Remove only harness-owned NFS configuration during teardown and preserve pre-existing services and unrelated exports. Require workload metadata for bounded result contracts and validate aggregate and live report filtering independently. Make deployment cache mode coverage independent of umask and add focused regressions for these lifecycle and reporting boundaries. Document the integration invariants and require repository-pinned Python tooling for authoritative lint and test results. Run integration workloads as fixed non-root identity Provision a fixed tester UID/GID 2000 in the Slinky login and compute images, register its Slurm account, and require both the coordinator and two-node srun fan-out to prove the expected identity. Run Slurm sweeps, PVC staging, cleanup, and SSH worker staging as that account with restrictive umasks. Remove client-side ownership changes that fail against all-squashed NFS exports and keep host account IDs confined to local fixture state. Repair failure injection's cross-host staging location, avoid Slurm indices during SSH-only runs, and ignore terminating Slinky workers during rollout validation. Add sentinel-ID regression coverage, document ownership and teardown contracts, and validate the complete SSH and Slurm scenario catalog. Make Slurm account setup idempotent Query existing Slurm accounts, associations, and user defaults before mutating accounting state. Add only missing records, repair a stale default account, and verify the exact tester/storage-test relationship after reconciliation. Cover retained accounting state with a regression that runs the reconciler twice and proves the second pass performs no mutation. This allows repeated setup to accept the state created by its first pass. Run integration lifecycle as ordinary user Keep kubeconfig, keys, downloaded clients, caches, manifests, logs, and test runs user-owned from creation under the repository state tree. Reject root for every lifecycle action and let the NFS backend invoke sudo only for package installation and host-system operations. Make the Docker SBX path fully unprivileged. Use sticky shared roots instead of host-side chown, isolate Helm state from stale root-owned caches, and remove UID-mapped data through the private Docker engine. Install pinned client copies in the user state tree instead of globally. Run the GitHub lifecycle as the runner account and validate the root rejection boundary. Document the privilege model and cover idempotent state bootstrap, sudo-free SBX prerequisites, and diagnostics. Fix filesystem device validation Compare st_dev values from stat -c %d instead of interpreting statfs free-inode counts as filesystem identities. Apply the correction to SSH scriptlet/direct and Slurm sbatch/srun probes, preserve paths with spaces, and report the actual distinct-from-root contract. Add shell regressions that exercise every dispatch route against distinct and matching device identities. Align design, requirements, and repository context with the validation semantics. Direct agents to wrap future commit messages at about 72 characters. Create default integration state parent Allow the known repository-local state path to create its ignored tmp parent in a fresh checkout before bootstrapping user-owned state. Continue rejecting custom state paths unless their parent already exists, preserving the destructive-cleanup safety boundary. Cover clean-checkout bootstrap and custom-path rejection with focused driver safety tests, and document the distinction. Fix integration scenario validation and teardown Search standard sbin paths so ordinary-user NFS setup can find privileged utilities, and record NFS service ownership before package installation can activate it. Keep cleanup safe when private kind or exportfs clients are absent after interrupted or repeated teardown. Read Slurm worker-order evidence from copied execution logs and validate the real Elbencho binary before installing failure-injection wrappers. Use non-sticky writable SBX fixture directories so replacement pods can clean files when Docker SBX remaps bind-mounted ownership. Add focused regressions for tool lookup, ownership sequencing, copied Slurm evidence, wrapper ordering, SBX modes, and missing-client cleanup. Harden SSH fixture transition convergence Require StatefulSet generation and revision convergence before accepting ready SSH workers. Use unique home and RWX probes, retry bounded cross-node visibility, and clean probes from both pods on every exit. Guard NFS teardown when interrupted package installation leaves valid ownership state without exportfs, and cover both lifecycle boundaries with focused regression tests.
dbishop
force-pushed
the
add-integration-tests
branch
from
September 21, 2026 00:52
3ff260f to
cb8be2f
Compare
Preserve positive sub-0.1 ms latency histogram bounds instead of clamping them above the observed data. This avoids a collapsed log axis causing tight-layout plot rendering to allocate an enormous canvas and trigger the OOM killer during the SBX single-file scenario. Retry temporary NFS migration unmounts with a bounded delay and verify with findmnt that the mount is gone before TemporaryDirectory cleanup. Fail safely with actionable diagnostics when the mount remains busy or its removal cannot be verified. Add regression coverage for low-latency histogram bounds, transient busy unmounts, and permanently busy mounts. Document both operational invariants in the repository context. Simplify integration deployment packaging Remove integration-only flags from the deployment builder. Use the same zero-option path used by end users and tolerate normal warnings for unavailable object or cross-architecture tools. Keep cache identities tied to the exact tracked working-tree snapshot, seeded Elbencho binary, runtime, architecture, and recipe. Invoke the builder once from an exact disposable copy. Its download and s3test side effects cannot change the manifested input, and the verified archive is reused across scenarios and runs. Honor pending tracked-file deletions in deployment snapshots and license checks. Cover deleted inputs, builder side effects, zero-argument invocation, and cache invalidation with regression tests. Remove superseded integration feasibility and coverage-gap research notes. Condense CONTEXT.md additions while retaining fixture, privilege, lifecycle, cache, coverage, and CI invariants.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First stab at both integration tests and trying to run them in our GitHub runners.