Skip to content

Latest commit

 

History

History
339 lines (271 loc) · 20.4 KB

testing.md

File metadata and controls

339 lines (271 loc) · 20.4 KB

Testing: Clean-Checkout Commands and Quality Metrics

This reference lists verification commands, CI owners, coverage floors, and warning policy. CI enforces the complete suites through .github/workflows/_quality.yml. Use local setup to select a targeted development check.

Command matrix

The matrix is the single command reference. The paths job selects CI jobs through the quality ownership contract. Unless stated otherwise, the owning job is in .github/workflows/_quality.yml.

Local development uses specific test modules or cases for changed behavior. The lane commands below reproduce complete package checks; do not run them all to verify a local edit. make test runs every row marked yes in its column. It must not be combined with its child targets, because that repeats the same suites. CI owns full suites unless the user explicitly requests a complete local run.

The implement workflow retains make test as its configured full-completion boundary. That configuration does not require running the complete suite locally during ordinary development.

Package targets synchronize dependencies with uv sync or npm ci. Helper targets supply their own tool options. make help lists the targets. Base prerequisites are uv and Node 20.19 or later with npm. The prerequisites column lists additional tools or services.

Unit and package lanes

Lane Lane command Prerequisites CI job make test
Platform (SQLite, local only) make test-platform None None; CI runs the PostgreSQL lane yes
Adapter SDK and packaging make test-adapter-sdk None Tests (adapter SDK) yes
Engine provisioner make test-provisioner None Tests (provisioner) yes
Packer helpers make test-packer None; Packer cases skip without Packer Tests (packer) yes
Installation CLI make test-installation None Tests (installation) yes
Bootstrap scripts make test-bootstrap None (unit selection) Tests (bootstrap) yes
Layer-import checker make test-check-layer-imports None Tests (check_layer_imports) yes
Helm chart contracts make test-charts Helm Tests (charts) yes
Repository guard make test-adr-guard None Tests (adr_guard) yes
GCP scripts uv run pytest tests/ in scripts/gcp None (unit selection) Tests (gcp scripts) no
OpenVPN uv run pytest tests/ in shifter/engine/openvpn None Tests (openvpn) no
Ops helper scripts uv run pytest tests/ in each scripts/<helper> None Tests (<helper>) no
Stack-smoke helpers python3 -m unittest discover -s scripts/stack-smoke -p 'test_*.py' None Lint and security (stack smoke) no
UAT harness units uv run pytest tests/ -q in each uat/<harness> None Tests (<harness>) no
Terraform contracts terraform init -backend=false, then terraform test, in the suite directory Terraform Contract test (<suite>) no

The ops helpers are handle_sd_replacement, check_rds_pending_modifications and assert_portal_inspection. The UAT harnesses are event-load-harness and range-functional-smoke. Their unit tests do not contact a deployment.

CI also runs the integration marker for bootstrap and GCP scripts. The local commands select only unit tests for those packages. See external tool integration tests.

Service and image lanes

Lane Lane command Prerequisites CI job
Platform (PostgreSQL, coverage floor) make test-platform-postgres PostgreSQL on :5432 Tests (shifter_platform, PostgreSQL n/6), Coverage (shifter_platform)
Platform Redis channel layer make test-platform-redis Redis on :6379 Tests (shifter_platform, Redis channel layer)
OpenFGA authorization ./scripts/run_openfga_integration.sh in shifter/shifter_platform Docker Tests (OpenFGA 1.20.0, PostgreSQL)
Subnet migration proof uv run pytest engine/tests/test_subnet_allocation_migrations_postgres.py in shifter/shifter_platform Docker Tests (migration proof)
Guacamole authority mvn verify in shifter/engine/guacamole/authority-extension, then the CI job's integration tests Maven, Docker Tests and compilation (Guacamole authority)
guacd image The CI job's image build, then bash shifter/engine/guacamole/test-entrypoint.sh Docker Tests (guacd image)
Built-image stack smoke bash scripts/stack-smoke/stack_smoke.sh Docker Stack smoke (built image)
GCP deployer image The CI job's fixture release manifest and image build, then docker run --rm bigrae-deployer:ci gcp-deploy --help and its Trivy scan Docker Tests (GCP deployer image)

The PostgreSQL and OpenFGA lanes run some of the same tests. They are not duplicates: each lane proves database or authorization behavior that the other lane cannot prove. The SQLite lane is a fast local lane; CI does not run it.

Frontend and browser lanes

Lane Lane command Prerequisites CI job make test
Legacy platform JavaScript (Jest) make test-js None Tests JavaScript (shifter_platform) yes
React SPA lint, types, Vitest, build make test-spa None SPA (shifter_platform frontend) yes
API contract and generated client The command below, in shifter/shifter_platform None API contract (shifter_platform) no
Authenticated SPA journeys make test-platform-e2e Chromium (the target installs it) SPA E2E (shifter_platform frontend) no
Browser accessibility make test-platform-a11y Chromium (the target installs it) Accessibility (deploy.yml) no

The API contract check needs the test posture, as in CI. The secret key is a synthetic, non-secret value:

TESTING=1 TEST_DB_BACKEND=sqlite DJANGO_DEBUG=true \
  DJANGO_SECRET_KEY=shifter-local-make-test-key \
  uv run python manage.py api_contract --check

The two browser targets build the SPA and use SQLite. CI uses PostgreSQL. Each target deletes shifter/shifter_platform/db.sqlite3. Do not run them at the same time or against an active development server. Both lanes fail on a Content Security Policy violation. See browser security policy.

MCP lanes

Lane Lane command Prerequisites CI job make test
MCP server packages make test-mcp None Lint JavaScript (mcp/<package>), Tests JavaScript (mcp/<package>) yes

make test-mcp runs npm ci, npm run lint and npm test in each package under mcp/. Lint allows zero warnings.

Policy and static checks

Check Lane command CI job
Architecture guard, imports, whitespace, changed prose make policy Architecture (adr conformance), Architecture (import linter), Docs prose (changed Markdown) in vale.yml
Commit hooks pre-commit run None; the hooks run locally before each commit
Workflows, Terraform, Kubernetes The native commands in AGENTS.md Lint (GitHub Actions), Lint (terraform), Lint and security (Helm chart), Lint (k8s schemas)

make policy runs the ADR guard once. Do not also run the configured lint command (adr_guard.py --all --level ci) in the same pass. Vale checks only the committed Markdown changes against origin/dev.

Live qualification

These checks run against a deployed tenant. They need operator authority and are not pull-request gates. A pass proves only the scope that its runbook states.

Check Command or entrypoint Runbook or status
Post-deploy range smoke manage.py run_post_deploy_smoke, run by the deploy workflows Advisory: a failure opens an issue and does not block the deploy
Participant terminal and Guacamole session uat/range-functional-smoke Range functional smoke
Event load envelope event-load-harness GCP event capacity
Range escape boundaries manage.py run_range_escape_validation Range escape validation
Range audit telemetry scripts/gcp/verify_range_audit_telemetry.py Range audit telemetry

Range readiness or an open port does not prove a working terminal session. An accepted destroy request does not prove that teardown completed.

For a targeted check, pass the exact test module or case to uv run pytest from its package directory.

The platform's root conftest.py defaults TESTING=1 for pytest. Test settings default to DEBUG=True, SQLite, and LOCAL_PROVISIONER=False. Explicit environment values can override the debug and database defaults. The platform CI job explicitly sets TESTING=1, TEST_DB_BACKEND=postgres, and DJANGO_DEBUG=true.

CI platform lane

CI runs the platform suite one time, against PostgreSQL. pytest-split divides the suite into six parallel shards. It uses the least_duration algorithm with no durations file, so it deals the tests out to the shards in turn. Each xdist worker clones its test database from one migrated template database. The Coverage (shifter_platform) job then does these steps:

  1. It combines the coverage data from all shards.
  2. It enforces the whole-package coverage floor.
  3. It makes sure that at least one postgres-marked test ran in a shard (scripts/check_postgres_lane_evidence.py).

A skipped postgres-marked test fails its shard. Keep each test fast. One slow test makes its shard slow, because the shards do not use recorded durations.

Test posture

config._runtime_env.IS_TEST_RUN is true when TESTING=1 or pytest is the entry point. Under it the settings default to the test posture so a bare uv run pytest does not inherit the production HTTPS posture a fresh checkout would otherwise get:

  • DEBUG defaults to True (config/settings.py). An explicit DJANGO_DEBUG still wins; production (IS_TEST_RUN false) is unchanged and defaults to False.
  • LOCAL_PROVISIONER is forced off (config/_cloud.py) so range-lifecycle tests never shell out to a real provisioner subprocess (which leaks). A test that exercises that path sets settings.LOCAL_PROVISIONER via the pytest-django settings fixture.
  • TEST_DB_BACKEND defaults to sqlite (config/_database_settings.py).

These are covered by tests/platform/test_clean_checkout_posture.py. Explicit ambient settings can override defaults. Use TEST_DB_BACKEND=sqlite for a targeted local run when the shell selects PostgreSQL.

Keep cloud test doubles within the provider adapter boundary. The tenant-pack and event-content fixtures select AWS and patch its storage constructor. They keep the shared storage factory and content resolver intact. A module imported during a fixture cannot retain a temporary replacement after teardown.

Content tests exercise the real parser and digest checks against synthetic stored bytes. When fixing an order-dependent failure, run the suspected predecessor and failing case together in one process. Also check that the next operation uses its own adapter after the patch ends. This follows Python's guidance on where to patch.

Coverage: two complementary floors

Production coverage measures owned code only. Each publishing package sets [tool.coverage.run] source and omits the whole test tree, conftest hooks, and migrations in its pyproject.toml, so tests can no longer cover themselves (the platform metric was 94% self-covering, ~88% production-only). CI runs pytest --cov (no path) so package config, not the command line, owns the measured set.

Absolute per-package floor, enforced by [tool.coverage.report] fail_under in each package. Floors are per-package (a repository aggregate would let a high-coverage package mask a regression in a smaller one). Each floor sits one point below the measured production-only baseline to absorb run-to-run variance. Floors only stay level or rise. A reduction requires a reviewed, time-bounded exception.

Package Production-only baseline fail_under
shifter/shifter_platform 88% 87
shifter/engine/provisioner 80% 79
shifter/installation 97% 96
scripts/bootstrap 86% 85
scripts/check_layer_imports 96% 95
shifter/packer 76% 75

SPA (frontend) floor. The Django-hosted React SPA (shifter/shifter_platform/frontend) defines coverage floors in vite.config.ts. test.coverage.thresholds covers statements, branches, functions, and lines. Each floor sits one point below its measured baseline, as in the Python packages. This absorbs run-to-run variance; floors only rise. The measured source set is src/**/*.{ts,tsx} excluding tests, test support, and the generated src/api/schema.d.ts; the first-party entrypoint and composition root stay measured.

The SPA (shifter_platform frontend) CI job uploads Vitest LCOV from frontend/coverage/lcov.info. The sonarcloud job restores it. SPA changed lines use the same changed-code gate below (#1526).

Changed-code floor: the SonarCloud raes-strict quality gate requires 80% changed-line coverage. It fails for new_coverage < 80, new_violations > 0, or new duplicated-lines density above 3%. It also fails for any new rating worse than A or new security hotspots that are not fully reviewed. PR analysis waits for the gate (sonar.qualitygate.wait=true). The conditions are recorded in sonar-project.properties; each owned Python package publishes its configured coverage report for Sonar to consume.

The sonar-rules-changed-lines pre-commit hook finds most new_violations before a push. It applies the Python rules that fail the gate most often to the lines that changed against origin/dev. It does not run tests, so it does not check new_coverage.

The released OpenFGA/PostgreSQL lane also publishes coverage-openfga.xml, including the native application and policy SQL duty tests. Sonar consumes both platform reports so service-only behavior contributes real coverage evidence. This supplemental report has no whole-package floor because it exercises a service-specific subset. The Coverage (shifter_platform) job still enforces the unchanged 87% package floor. Sonar still enforces 80% changed-code coverage. Run it locally from shifter/shifter_platform with ./scripts/run_openfga_integration.sh --cov --cov-report=xml:coverage-openfga.xml --cov-fail-under=0.

Warning policy

Warning classification lives in shifter/shifter_platform/pyproject.toml [tool.pytest.ini_options] filterwarnings, so a direct pytest and CI agree regardless of test order or xdist worker.

  • ResourceWarning and unawaited-coroutine RuntimeWarning are errors. They are correctness defects; fix the originating fixture/mock (close/await the resource) rather than downgrade the filter. Converting to ignore, adding a sleep, forcing GC, or disabling xdist is not a fix.
  • DeprecationWarning is an error by default. First-party deprecations, and any new or changed third-party deprecation, fail until fixed or explicitly baselined. Do not add a blanket ignore::DeprecationWarning.
  • Third-party deprecation allowances are narrow and owned. Each is an ignore filter matched by exact message, with an adjacent owner and removal condition. The current set is the django-health-check (<5.0) class-based check deprecations, tracked for migration + removal in #1601.
  • Tests that intentionally assert a warning use pytest.warns at the behavior boundary; that is an assertion, not a baseline exception.

Authenticated SPA end-to-end (Playwright)

make test-platform-e2e runs the authenticated browser journeys against the real Django-hosted built SPA (#1526). It is hermetic: it builds the SPA, migrates and seeds a job-local database, and drives Playwright through normal Django sessions and CSRF. The CI job shifter-platform-e2e in .github/workflows/_quality.yml feeds the PR Gate. It runs the journeys through scripts/run_openfga_integration.sh --browser, which starts a job-local PostgreSQL and OpenFGA server, migrates, and seeds. The make target uses SQLite locally.

The dev/test-only seed_e2e management command establishes the synthetic actors, their accounts and their native policy grants; dev_login establishes their sessions. The dev/test-only seed_e2e_ctf command runs after seed_e2e. It adds one CTF event in the organizer's account, publishes the event's authorization relationships, and releases inbox messages for the inbox participant actor. The Playwright config's auth.setup.ts logs each actor in once and reuses its storageState. No cloud, provisioner, IdP, or live-range dependency is used: LOCAL_PROVISIONER/ENGINE_TASK_* stay unset, so a range launch enqueues nothing and contacts no cloud.

Hermetic scope includes login, authentication revocation, workspace lifecycle, shared-audit history, and authorization denial. Workspace operations cover create, rename, archive, and restore. Denial covers client advice and the authoritative API 403. The journeys cover CTF organizer, participant, and threat-research role landings. The participant inbox includes stored and reflected XSS probes under the enforced CSP.

Authenticated Playwright traces are retained only on a local retry and are never uploaded as CI artifacts (ADR-055-R7). Reproduce failures locally with make test-platform-e2e.

Browser accessibility (Playwright + axe)

make test-platform-a11y runs @axe-core/playwright over the ADR-055 surface matrix (frontend/e2e/a11y/) against the same hermetic stack, with the WCAG 2.2 A/AA tag set. The Accessibility job in .github/workflows/deploy.yml runs it on every pull request as a direct PR Gate dependency (it does not inherit the frontend path skip, per ADR-055-R2), against a job-local PostgreSQL.

The debt baseline is an exact set of privacy-safe fingerprints in frontend/e2e/a11y/baseline.json. The scan fails on any new or resolved finding. The accessibility-baseline ADR guard permits the committed baseline only to shrink against the trusted base branch. Additions require an exact-fingerprint waiver in docs/adr/exceptions.yaml naming ADR-055 under R4/R6.

Browser security policy

The portal enforces the ADR-036 Content Security Policy. The browser lanes record each securitypolicyviolation event through frontend/e2e/support/csp.ts. A journey, accessibility scan or live Django page fails on any violation.

e2e/journeys/csp.spec.ts checks the enforced header and blocked inline script and handlers. It also checks that same-origin WebSockets connect and off-origin connections are blocked. e2e/journeys/django-pages.spec.ts covers the live Django-rendered pages. These tests use Chromium only. The deployed identity provider, MFA and range terminal paths are outside the hermetic lanes; #2117 owns that tier. RB024 is the operator procedure for a blocked feature.

External tool integration tests

GCP scripts, bootstrap, adapter SDK, and Packer register the integration pytest marker. Those checks require external rendering/build commands or isolated package/TLS environments. Use uv run pytest -m "not integration" from a package directory for its tool-independent unit selection, or uv run pytest -m integration to run the external-tool checks explicitly. GCP scripts and bootstrap default to the unit selection; their CI jobs explicitly select both with -m "integration or not integration". SDK and Packer default runs retain both selections. No regression checks are removed from CI.

The bootstrap and GCP script CI jobs install the same digest-verified Helm build; the GCP script job also installs the pinned kubectl client. Bootstrap verifies its OpenSSL prerequisite. SDK wheel conformance requires uv and the runtime extra's cached dependencies; guest TLS integration requires OpenSSL. Packer validation requires Packer and its plugins. These checks do not deploy cloud resources.

shifter/packer/tests/test_scripts.sh and the TestScriptStructureLint and TestScriptContentLint classes are lint-only checks. Their success does not prove provisioning behavior or qualify an image. The script execution suites use controlled command shims; built-image behavior still requires image qualification.