Skip to content

test(simulate): give each simulate command its own integration suite - #508

Draft
jcogilvie wants to merge 28 commits into
crossplane:mainfrom
jcogilvie:jco-simulate-integration-split
Draft

test(simulate): give each simulate command its own integration suite#508
jcogilvie wants to merge 28 commits into
crossplane:mainfrom
jcogilvie:jco-simulate-integration-split

Conversation

@jcogilvie

@jcogilvie jcogilvie commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

Description of your changes

Note

Stacked on #487, so until it merges, this PR's diff includes it. The commits for this PR are the top five.

This is the follow-up to two of @haarchri's comments on #486. All the simulate integration cases lived in cmd/crossplane/project. Now each command owns its suite, and the simulate.ProjectResult type is renamed to fit what it is.

Suites per command:

  • cmd/crossplane/composition: TestCompDiffIntegration and the fixtures only it uses.
  • cmd/crossplane/xr: TestDiffIntegration and the fixtures only it uses.
  • cmd/crossplane/project: keeps the project simulate cases, the plan-output and provider-plan tests.
  • Shared harness: internal/testutils/integration/simharness, behind the integration build tag. It holds the argv translation, the run helpers, the case types and the least-privilege RBAC rules.
  • Shared fixtures: the CRDs and every fixture more than one suite uses are in internal/testutils/integration/testdata.

Files moved with git mv, so git log --follow keeps their history. No case is added, removed or changed. The test list across the three packages matches the one before the move.

-p 1 for the integration tests: once the cases were split into three packages, go test ran them in parallel, and three suites rendering through Docker at once timed out. nix/apps.nix now runs integration packages one at a time, and each package's cases still run in parallel. Before the move, the simulate cases were all in one package, so CI's integration job should take about as long as before.

ProjectResult becomes Result: every simulate command emits it for -o json and -o yaml, not only project simulate. NewProjectResult becomes NewResult. The JSON output is unchanged.

Related:

I have:

  • Read and followed Crossplane's contribution process.
  • Run ./nix.sh flake check to ensure this PR is ready for review.
  • Added or updated unit tests. A pure move and rename.
  • Linked a PR or a docs tracking issue to document this change. No user-facing change.
  • Added backport release-x.y labels to auto-backport this PR.

🤖 Generated with Claude Code

jcogilvie and others added 19 commits November 6, 2026 20:35
@jcogilvie @claude
feat(diff): restore the XR-only diff path
ef0875e 
Restore PerformDiff with its input validator, ValidatedInput, the
WithInputValidatorFactory option, their mocks, test helpers and tests
from the state before the XR-only path was removed. Keep the
TestManagedByManagerWithNoName case that the managedBy/boundToClaim
table port added and the restored tests lack.

Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jcogilvie @claude
feat(simulate): classify composite resources and claims for xr simulate
8f1d2d8 
Inputs that are neither Compositions nor project files are composite
resource candidates, which checkComposites confirms against the
cluster's XRDs. Every rejection uses RejectInput's template, so
composition and xr simulate point at each other symmetrically. The
runner gains a Composites input that runs PerformDiff and renders
through renderXRReport.

Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jcogilvie @claude
feat(xr): add xr simulate
c4e0dc0 
xr simulate previews composite resource or claim files against the
cluster with its Compositions and functions, through the shared
simulate runner. It takes no selection flags or --max-concurrency.

Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jcogilvie @claude
test(xr): restore the xr simulate integration tests
c3c77f3 
Restore TestDiffIntegration, TestSimulateProviderPlan, the XR plan-output
cases of TestSimulatePlanOutput and the remaining testdata. The harness
now runs XRDiffTest cases as `xr simulate` with -f per file, and the
TestCompDiffIntegration comments that point at TestDiffIntegration are
restored.

Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jcogilvie @claude
test(xr): restore the XR-only integration harness code
cf0e562 
Restore simArgs.FunctionCredentials with its --function-credentials argv,
which the XR cases of TestDiffIntegration use. simulateArgv maps
XRDiffTest to xr simulate and still fails on any other unknown test type.
The admission-policy gate uses integration.AwaitAdmissionDenial.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie @claude
test(simulate): show composite resource runs ignore the project file
10b3a25 
Move the build step of Run into Flags.build and test that both
Composition and composite resource inputs take imageConfigs only from
--image-configs, never from ./crossplane-project.yaml.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie @claude
docs(xr): say xr simulate -f takes a file or directory
fd45af6 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie @claude
docs(simulate): cross-reference all three simulate commands
80f48a2 
Point project, composition and xr simulate help at each other, and
resource simulate help at the three. Mark xr simulate as added in the
design doc and name the right commands in the integration testdata README.

Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jcogilvie @claude
fix(simulate): say why xr simulate rejects an input no XRD defines
6f39964 
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jcogilvie @claude
fix(simulate): reject a kind: List at load in every simulate command
c0dc566 
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jcogilvie @claude
test(simulate): restore renderXRReport coverage in TestSimulateRender…
0b71dbc 
…Reports

Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jcogilvie @claude
docs(simulate): point every simulate help at resource simulate
2be0bfb 
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jcogilvie @claude
docs(simulate): make the output help and design doc true for xr simulate
8d99792 
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jcogilvie @claude
docs(simulate): fix stale integration harness and testdata README com…
0618d88 
…ments

Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jcogilvie @claude
test(simulate): make the reject symmetry check real and test diffComp…
4d13413 
…osites

Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jcogilvie @claude
test(simulate): fold the project-file image config test into the buil…
f0a099c 
…d test

Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jcogilvie @claude
refactor(xr): validate -f inputs in xr simulate and take a comma-sepa…
106327e 
…rated list

xr simulate owns which inputs it accepts, and -f matches resource simulate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie @claude
fix(simulate): create the package client only when planning
0209b02 
With --skip-plan nothing fetches packages, and --cache-dir is already
reported as ignored, so don't create (and fail on) the cache directory.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie @claude
docs(xr): lead xr simulate's help with the permissions it needs
840facb 
Match composition and project simulate: the create and patch permissions
come in an Important callout straight after the summary, with the reason.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie
jcogilvie force-pushed the jco-simulate-integration-split branch from c64aaa8 to 43fb7f1 Compare November 7, 2026 01:47
jcogilvie and others added 9 commits November 7, 2026 00:10
@jcogilvie @claude
fix(simulate): match each assertion expectation to a distinct result
80be2bd 
Errors, resource plans, validation failures and field errors were matched
first-wins, so two expectations could be satisfied by one result while the
count check still passed. Track used results as the warnings matcher does,
and match per-XR errorIDs by ref so same-named XRs in different namespaces
do not fail spuriously.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie @claude
fix(simulate): check Before for deleted fields; verify storage version
a7b1611 
WithFieldAbsent on a delete read only leaf changes, so an empty map or
list in Before passed. WithDryRunStorageVersion was silently unchecked
without WithDryRunSkipped. WithXRs matches regardless of order, so say so.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie @claude
fix(diff): name the namespace in XR section headers and count only XR…
008d87f 
…s in the footer

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie @claude
fix(simulate): default input namespaces to the kubeconfig context's
fd3f780 
A claim or namespaced composite resource with no metadata.namespace now
takes the context namespace, as kubectl apply would. Cluster-scoped
composite resources are left alone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie @claude
test(simulate): move the shared integration harness and testdata to i…
3912113 
…nternal/testutils

The simulate argv translation, case runner and fixtures shared by the
composition, xr and project integration suites move to simharness and
internal/testutils/integration/testdata so each command can own its suite.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie @claude
test(composition): own the composition integration suite
15fc396 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie @claude
test(xr): own the xr integration suite
c622f16 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie @claude
build(nix): run integration packages one at a time
ab093a8 
Each suite already runs its cases in parallel; three suites rendering
through Docker at once starve each other into timeouts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie @claude
refactor(simulate): rename ProjectResult to Result
5ad7bcc 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie
jcogilvie force-pushed the jco-simulate-integration-split branch from 43fb7f1 to 5ad7bcc Compare November 7, 2026 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tampakrap tampakrap Awaiting requested review from tampakrap tampakrap will be requested when the pull request is marked ready for review tampakrap is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jcogilvie