Skip to content

Integration tests against IRIS containers for release-preparation PRs - #359

Draft
isc-klu wants to merge 41 commits into
intersystems-community:masterfrom
isc-klu:integration-tests
Draft

isc-klu wants to merge 41 commits into
intersystems-community:masterfrom
isc-klu:integration-tests

Conversation

@isc-klu

@isc-klu isc-klu commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Reading order

  1. test-fixtures/README.md — the design: cases, checks, containers, how to run locally.
  2. src/test/cases.ts — that matrix as data; src/test/suite/extension.test.ts — the checks.
  3. src/test/runTest.ts and .github/workflows/prepare-release.yml — how they get run.

Companion of intersystems-community/vscode-objectscript#1870; both repos run the same matrix, each installing the other extension's Marketplace release.

What changes

  • New prepare-release.yml workflow runs the suite against IRIS containers on prepare-* PRs and manual dispatch.
  • The placeholder npm test (a single assert.ok("All good"), run on every push) is replaced, and the step leaves main.yml and prerelease.yml.
  • @vscode/test-electron 2.5 → 3.1 (2.x looks for a binary current VS Code no longer ships on macOS), plus skipLibCheck for its typings.

Decisions worth knowing

  • Podman only. The workflow removes the runner's Docker Compose (the ObjectScript extension tries it first), and runTest.ts pins PODMAN_COMPOSE_PROVIDER=podman-compose (Docker Desktop's plugins hijack podman compose).
  • Isolation. Fresh --user-data-dir per case; the suite asserts it is running the build under test, not the Marketplace copy the ObjectScript extension depends on.
  • Lapsed sessions. 10 s /api/atelier timeout; every check reruns as the first request after it lapses.

Verified by mutation

Reverting #347 fails the check that a password stored in settings reaches API consumers; reverting #352 fails the -anonymous namespace listings.

Known failing on CI

All against the Marketplace ObjectScript 3.8.5; they clear once 3.8.6 ships (runTest.ts installs it with --pre-release).

Not covered

  • The authentication provider's prompt and secret storage: they need typed input or seeded secrets, and vscode.authentication.getSession from a test hits VS Code's consent dialog.
  • Incorrect account detected #353 (fix split across both extensions, no self-contained repro) and OAuth2 OAuth2 VSCode #335 (needs an identity provider).

Replace the placeholder suite with checks that every intersystems.servers
configuration connects without prompting, both through the extension's own
API/REST layer and through the released ObjectScript extension. Runs in a
new prepare-release.yml workflow on prepare-* PRs and on demand.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
isc-klu and others added 3 commits September 14, 2026 15:37
Run the IRIS containers under Podman and install the ObjectScript extension's
pre-release, whose Podman support the docker-compose case needs. Drop the
runner's Docker Compose so resolution goes through Podman. The docker-compose
cases fail until the ObjectScript Podman build reaches the Marketplace; the
post-timeout round-trip no longer re-verifies delete-sync, unreliable on the
current release.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
isc-klu and others added 24 commits September 15, 2026 11:25
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
named/anonymous, matching the case suffixes and server entries, so the
service field goes away. Pin podman compose to podman-compose: Docker
Desktop's CLI plugins otherwise make it run Docker Compose.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every check gets its own idle so it is the first request on a lapsed
session; before, the round trip re-authenticated and the rest ran fresh.
Round trip moves last as the most expensive.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The released ObjectScript build doesn't re-wire delete-sync after a
lapse, so the active -named cases fail here until 3.8.6 ships.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
vscode-objectscript#1872 fixed the stale connection for synced deletes;
the released build still has it, so -inactive cases fail here until 3.8.6.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
isc-klu and others added 13 commits September 15, 2026 16:51
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e type

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The matrix is a LAUNCHES constant; the servers, kinds and toggleable set
are plain data; workspaceFile hoists the shared settings. runTest drops
what the README documents.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e whole

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant