Skip to content

OCPNODE-4494: e2e test case for RHCOS upgrade from 9 → 10#31393

Open
asahay19 wants to merge 1 commit into
openshift:mainfrom
asahay19:rhcos-upgrade
Open

OCPNODE-4494: e2e test case for RHCOS upgrade from 9 → 10#31393
asahay19 wants to merge 1 commit into
openshift:mainfrom
asahay19:rhcos-upgrade

Conversation

@asahay19

@asahay19 asahay19 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the crun happy-path e2e test for RHCOS 9 → 10 osImageStream upgrades, complementing the runc guard test merged in #31266 (UC6A).
On OCP 5.0–5.2 clusters with dual OS streams and crun as the default runtime, moving a pool from rhel-9 to rhel-10 should succeed without triggering the runc-on-RHCOS-10 guard. This test validates that recommended upgrade path.

What the test does

  1. Creates an isolated MachineConfigPool pinned to rhel-9 (no runc ContainerRuntimeConfig)
  2. Labels one pure worker into the pool and waits for rollout on RHCOS 9 with crun
  3. Patches the pool to rhel-10
  4. Asserts:
    • MCP reaches ready without RenderDegraded / Degraded
    • Node reboots to RHCOS 10 and stays Ready
    • Runtime remains crun (no runc CRI-O drop-in)
    • co/machine-config is not Upgradeable=False (reason DegradedPool)

Changes

  • test/extended/node/runc_upgrade_cases.go — new It + helpers; shared AfterEach parameterized by pool name

  • test/extended/node/runc_upgrade_cases.md — documents both UC6A and UC5/UC14

  • Epic: OCPNODE-4013

Locally executed on Openshift cluster 5.0, it got executed successfully:

./openshift-tests run-test \
  "[Suite:openshift/disruptive-longrunning][sig-node][Serial][Disruptive][OCPFeatureGate:OSStreams] runc RHCOS 10 upgrade guard allows RHCOS 9 to 10 osImageStream upgrade when default runtime is crun"
  
    ------------------------------

  Ran 1 of 1 Specs in 506.229 seconds
  SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
[
  {
    "name": "[Suite:openshift/disruptive-longrunning][sig-node][Serial][Disruptive][OCPFeatureGate:OSStreams] runc RHCOS 10 upgrade guard allows RHCOS 9 to 10 osImageStream upgrade when default runtime is crun",
    "lifecycle": "informing",
    "duration": 506253,
    "startTime": "2026-07-15 12:13:34.449119 UTC",
    "endTime": "2026-07-15 12:22:00.702706 UTC",
    "result": "passed",

Summary by CodeRabbit

  • Tests

    • Expanded end-to-end OS image upgrade coverage to verify RHCOS 9→10 upgrades when the default container runtime is crun.
    • Added a new crun happy-path upgrade scenario, including creation of a pinned MCP, node labeling into it, and verification that the pool doesn’t enter a degraded state.
    • Strengthened upgrade health checks (including reboot to the target OS image) and ensured the default runtime remains crun after the upgrade.
    • Refined runtime verification (polling-based) and improved shared cleanup/rollback handling across scenarios.
  • Documentation

    • Reworked the RHCOS 9→10 upgrade test cases doc, adding/clarifying UC6A and UC5/UC14 details, expected outcomes, and timing guidance.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 16, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 16, 2026

Copy link
Copy Markdown

@asahay19: This pull request references OCPNODE-4494 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Adds the crun happy-path e2e test for RHCOS 9 → 10 osImageStream upgrades, complementing the runc guard test merged in #31266 (UC6A).
On OCP 5.0–5.2 clusters with dual OS streams and crun as the default runtime, moving a pool from rhel-9 to rhel-10 should succeed without triggering the runc-on-RHCOS-10 guard. This test validates that recommended upgrade path.

What the test does

  1. Creates an isolated MachineConfigPool pinned to rhel-9 (no runc ContainerRuntimeConfig)
  2. Labels one pure worker into the pool and waits for rollout on RHCOS 9 with crun
  3. Patches the pool to rhel-10
  4. Asserts:
  • MCP reaches ready without RenderDegraded / Degraded
  • Node reboots to RHCOS 10 and stays Ready
  • Runtime remains crun (no runc CRI-O drop-in)
  • co/machine-config is not Upgradeable=False (reason DegradedPool)

Changes

  • test/extended/node/runc_upgrade_cases.go — new It + helpers; shared AfterEach parameterized by pool name

  • test/extended/node/runc_upgrade_cases.md — documents both UC6A and UC5/UC14

  • Epic: OCPNODE-4013

Locally executed on Openshift cluster 5.0, it got executed successfully:

./openshift-tests run-test \
 "[Suite:openshift/disruptive-longrunning][sig-node][Serial][Disruptive][OCPFeatureGate:OSStreams] runc RHCOS 10 upgrade guard allows RHCOS 9 to 10 osImageStream upgrade when default runtime is crun"
 
   ------------------------------

 Ran 1 of 1 Specs in 506.229 seconds
 SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
[
 {
   "name": "[Suite:openshift/disruptive-longrunning][sig-node][Serial][Disruptive][OCPFeatureGate:OSStreams] runc RHCOS 10 upgrade guard allows RHCOS 9 to 10 osImageStream upgrade when default runtime is crun",
   "lifecycle": "informing",
   "duration": 506253,
   "startTime": "2026-07-15 12:13:34.449119 UTC",
   "endTime": "2026-07-15 12:22:00.702706 UTC",
   "result": "passed",

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: asahay19
Once this PR has been reviewed and has the lgtm label, please assign rphillips for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci
openshift-ci Bot requested review from bitoku and sairameshv July 16, 2026 05:29
@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 98e8ec87-e16b-4a96-b17c-fdbf0f26c496

📥 Commits

Reviewing files that changed from the base of the PR and between c6aab91 and 264d40c.

📒 Files selected for processing (2)
  • test/extended/node/runc_upgrade_cases.go
  • test/extended/node/runc_upgrade_cases.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • test/extended/node/runc_upgrade_cases.md
  • test/extended/node/runc_upgrade_cases.go

Walkthrough

The upgrade test suite now covers runc guard behavior and the crun happy path for RHCOS 9-to-10 upgrades. Cleanup is parameterized by the active test pool, runtime checks use polling helpers, and documentation describes both scenarios and health assertions.

Changes

Runtime upgrade test coverage

Layer / File(s) Summary
Runtime scenarios and pool lifecycle
test/extended/node/runc_upgrade_cases.go
Adds the crun upgrade scenario, validates pool and cluster health, and parameterizes cleanup for the active pool and optional CRC removal.
Runtime polling and readiness helpers
test/extended/node/runc_upgrade_cases.go
Separates polling helpers for runc and crun runtime checks from immediate node readiness assertions, including transient debug-unavailable handling.
Upgrade case documentation
test/extended/node/runc_upgrade_cases.md
Documents the runc guard and crun happy-path cases, commands, condition timing, runtime estimates, and runtime configuration details.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant UpgradeTest
  participant MachineConfigPool
  participant WorkerNode
  participant ClusterOperator
  UpgradeTest->>MachineConfigPool: Create runtime-specific upgrade pool
  UpgradeTest->>WorkerNode: Label worker and verify RHCOS 9 runtime
  UpgradeTest->>MachineConfigPool: Set osImageStream to rhel-10
  MachineConfigPool->>WorkerNode: Roll out RHCOS 10 configuration
  UpgradeTest->>MachineConfigPool: Assert RenderDegraded is absent
  UpgradeTest->>ClusterOperator: Check DegradedPool does not block upgradeability
Loading

Suggested reviewers: sairameshv

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Test Structure And Quality ⚠️ Warning The first It mixes guard-block, recovery, and optional crun-upgrade verification; several setup assertions are bare HaveOccurred() checks without context. Split the scenarios into separate Its and add failure messages to the bare setup expectations (client creation, microshift/topology checks, etc.).
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the main change: adding an e2e test case for RHCOS 9→10 upgrade coverage.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PASS: The only Ginkgo titles in the changed file are static string literals; no pod/node/namespace names, timestamps, UUIDs, or fmt.Sprintf-built titles appear.
Microshift Test Compatibility ✅ Passed PASS: the new Ginkgo suite is guarded by exutil.IsMicroShiftCluster()+g.Skip(), so its MCO/ClusterOperator/MachineConfig usage won't run on MicroShift.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS: both new It blocks sit under a Describe that skips configv1.SingleReplicaTopology, so the test is guarded from SNO.
Topology-Aware Scheduling Compatibility ✅ Passed The PR only adds e2e test/docs for MCP upgrade flows; it doesn't add or modify deployment manifests, operators, controllers, or scheduling constraints.
Ote Binary Stdout Contract ✅ Passed No stdout writes were added in process-level code; the file has no main/init/TestMain/BeforeSuite setup and no fmt.Print/logging at suite scope.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No IPv4-only assumptions or external connectivity were added; the test uses only cluster APIs and OSImageStream/MCP operations.
No-Weak-Crypto ✅ Passed Changed files contain only MCO/runtime checks; no crypto packages, weak algorithms, custom crypto, or secret/token comparisons were found.
Container-Privileges ✅ Passed Changed files are only a Go e2e test and markdown; no privileged/securityContext fields or manifest settings were added.
No-Sensitive-Data-In-Logs ✅ Passed New logs only print node/pool names and status/debug info; no secrets, tokens, PII, or customer data found.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jul 16, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@bitoku bitoku left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall lgtm,
One comment about the style.
There's some functions like assert*, wait*, expect*, verify*.
I don't mind either, but want them to be consistent or follow a rule (for example, expect calls Expect in the function, something like that).

Comment on lines +62 to +64
nodeName = ""
testPoolName = ""
cleanupCRC = false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I don't think they are needed. They should be initialized when it's defined.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped redundant BeforeEach resets for nodeName / testPoolName / cleanupCRC in the latest commit


g.By("Upgrading RHCOS version to RHCOS 10 via osImageStream")
o.Expect(setPoolOSImageStream(ctx, mcClient, crunRHCOS10UpgradePool, streamRHEL10)).To(o.Succeed())
o.Expect(waitForMCP(ctx, mcClient, crunRHCOS10UpgradePool, 45*time.Minute)).To(o.Succeed(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think 45 mins is too long. It's supposed to be finished much earlier.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reduced post-rhel-10 MCP and node waits from 45m to 30m

"pool should roll out to rhel-10 without runc guard RenderDegraded")

g.By("Verifying node rolled out to RHCOS 10 with crun and pool stayed healthy")
o.Expect(waitForNodeRHELMajorVersion(ctx, oc, nodeName, "10", 45*time.Minute)).To(o.Succeed(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think 45 mins is too long. It's supposed to be finished much earlier.

if upgradeable != nil &&
upgradeable.Status == configv1.ConditionFalse &&
upgradeable.Reason == degradedPoolUpgradeableReason {
return fmt.Errorf("ClusterOperator %s Upgradeable=False (reason %s) after crun rhel-10 upgrade: %q",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too specific error message, this may be confusing in the future when someone uses this function.

Suggested change
return fmt.Errorf("ClusterOperator %s Upgradeable=False (reason %s) after crun rhel-10 upgrade: %q",
return fmt.Errorf("ClusterOperator %s Upgradeable=False (reason %s): %q",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generalized assertMachineConfigNotBlockedByDegradedPool error message in the latest commit

@bitoku

bitoku commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-1of2 periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-2of2

@openshift-ci

openshift-ci Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

@bitoku: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-1of2
  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-2of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/07d107d0-8112-11f1-980f-e0e7979d9599-0

@asahay19
asahay19 force-pushed the rhcos-upgrade branch 2 times, most recently from dfe3764 to c6aab91 Compare July 17, 2026 11:42
@asahay19

Copy link
Copy Markdown
Contributor Author

/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-2of2

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@asahay19: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-2of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/bc3277f0-81d4-11f1-91a5-e78e2058153d-0

@asahay19

Copy link
Copy Markdown
Contributor Author

/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-1of2

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@asahay19: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-disruptive-longrunning-techpreview-1of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/cab5adb0-81d4-11f1-9135-6cfa88222756-0

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@asahay19

Copy link
Copy Markdown
Contributor Author

/retest

1 similar comment
@asahay19

Copy link
Copy Markdown
Contributor Author

/retest

@openshift-ci

openshift-ci Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

@asahay19: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gcp-ovn c6aab91 link true /test e2e-gcp-ovn
ci/prow/e2e-aws-ovn-serial-2of2 c6aab91 link true /test e2e-aws-ovn-serial-2of2
ci/prow/e2e-aws-ovn-microshift c6aab91 link true /test e2e-aws-ovn-microshift
ci/prow/e2e-metal-ipi-ovn-ipv6 c6aab91 link true /test e2e-metal-ipi-ovn-ipv6
ci/prow/e2e-aws-ovn-fips c6aab91 link true /test e2e-aws-ovn-fips

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants