OCPBUGS-93879: Create different test buckets for heavy CI jobs #31377
OCPBUGS-93879: Create different test buckets for heavy CI jobs #31377xueqzhan wants to merge 1 commit into
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: automatic mode |
openshift-ci
Bot
added
the
do-not-merge/hold
|
/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThis PR partitions selected Kubernetes and OpenShift tests into CPU-heavy buckets, adds discovery and ChangesCPU-heavy test scheduling
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/0c146d90-7bd2-11f1-8628-d6f865f51c18-0 |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: xueqzhan The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
pkg/test/ginkgo/cmd_runsuite.go (1)
691-786: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftRepeated bucket-execution boilerplate could be extracted into a helper.
The
copyTests → recordTestBucketInterval → q.Execute → EndInterval → log → appendsequence is now duplicated ~14 times in this function (4 new instances added here). Extracting a small helper would reduce duplication and make future bucket additions less error-prone.♻️ Example helper extraction
func executeTestBucket(ctx context.Context, q *parallelTestQueue, recorder monitor.Recorder, name string, tests []*testCase, parallelism int, cfg testOutputConfig, abortFn abortFunc) []*testCase { testsCopy := copyTests(tests) intervalID, startTime := recordTestBucketInterval(recorder, name) q.Execute(ctx, testsCopy, parallelism, cfg, abortFn) recorder.EndInterval(intervalID, time.Now()) logrus.Infof("Completed %s test bucket in %v", name, time.Since(startTime)) return testsCopy }Usage:
- draTestsCopy := copyTests(draTests) - draIntervalID, draStartTime := recordTestBucketInterval(monitorEventRecorder, "DRA") - q.Execute(testCtx, draTestsCopy, parallelism, testOutputConfig, abortFn) - monitorEventRecorder.EndInterval(draIntervalID, time.Now()) - logrus.Infof("Completed DRA test bucket in %v", time.Since(draStartTime)) - tests = append(tests, draTestsCopy...) + tests = append(tests, executeTestBucket(testCtx, q, monitorEventRecorder, "DRA", draTests, parallelism, testOutputConfig, abortFn)...)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/test/ginkgo/cmd_runsuite.go` around lines 691 - 786, The bucket execution flow in this function is repeated many times and should be extracted into a shared helper. Move the copyTests, recordTestBucketInterval, q.Execute, monitorEventRecorder.EndInterval, logrus.Infof, and append sequence into a reusable helper like executeTestBucket, then replace each bucket block (for example Kubernetes, DRA, InPlaceResize, Probing, Storage, Network, Netpol, Builds, OpenShift, DeploymentConfig, MustGather) with calls to it. Keep the helper responsible for returning the copied tests so the caller can continue appending them to tests.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@pkg/test/ginkgo/cmd_runsuite.go`:
- Around line 691-786: The bucket execution flow in this function is repeated
many times and should be extracted into a shared helper. Move the copyTests,
recordTestBucketInterval, q.Execute, monitorEventRecorder.EndInterval,
logrus.Infof, and append sequence into a reusable helper like executeTestBucket,
then replace each bucket block (for example Kubernetes, DRA, InPlaceResize,
Probing, Storage, Network, Netpol, Builds, OpenShift, DeploymentConfig,
MustGather) with calls to it. Keep the helper responsible for returning the
copied tests so the caller can continue appending them to tests.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 62231f2a-4f19-4df7-9419-d5e1cf9a1f93
📒 Files selected for processing (1)
pkg/test/ginkgo/cmd_runsuite.go
openshift-ci
Bot
added
the
ready-for-human-review
|
/payload-aggregate periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview 10 |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/a8f9bc00-7bd7-11f1-9f83-ccfaed31241a-0 |
|
Scheduling required tests: |
|
/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/96088da0-7caf-11f1-9351-44f6e94c965d-0 |
|
/payload-aggregate periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview 10 |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/5569f8e0-7cb1-11f1-8cff-afa80418945c-0 |
|
/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/5badb0e0-7d63-11f1-8c41-d227074856d9-0 |
|
/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/5de72620-7d63-11f1-9955-6ab1f637f7c9-0 |
|
/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/5fcf2000-7d63-11f1-91fc-dc7e147706c0-0 |
|
/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/f85932f0-7e1e-11f1-9e51-697b9f4b4700-0 |
|
/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview |
1 similar comment
|
/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/fadc8270-7e1e-11f1-838a-6af43a5a1ff6-0 |
|
/payload-job periodic-ci-openshift-release-main-ci-5.0-e2e-azure-ovn-upgrade |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/474014e0-838f-11f1-98fa-7cd607985a1e-0 |
|
/payload-aggregate periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview 5 |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/5769ceb0-838f-11f1-8709-dd6334c9ac84-0 |
|
Scheduling required tests: |
|
/retest-required |
|
/payload-job periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-techpreview |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9bdd8580-83c8-11f1-8cdf-b48fdc4f2b9b-0 |
|
/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview |
1 similar comment
|
/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/ab4bde90-83c8-11f1-91c7-3c2aa556f6aa-0 |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/ad2a3b80-83c8-11f1-845d-b0cbe7358598-0 |
|
/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview |
|
@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/afbb46a0-83c8-11f1-8225-30d1e6c61d5a-0 |
xueqzhan
changed the title
openshift-ci-robot
added
jira/valid-reference
|
@xueqzhan: This pull request references Jira Issue OCPBUGS-93879, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
d2ecbbe to
64d5b02
Compare
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@xueqzhan: This pull request references Jira Issue OCPBUGS-93879, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/retest-required |
|
@xueqzhan: This pull request references Jira Issue OCPBUGS-93879, which is valid. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Scheduling required tests: |
|
/retest-required |
|
@xueqzhan: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
Separate CPU-intensive tests into dedicated buckets with reduced parallelism to prevent API server/etcd overload during e2e test execution on heavy jobs.
Problem
The k8s 1.35 rebase massively expanded several test groups (DRA +46%, Pod InPlace Resize +32 table-driven entries, Pod Generation graduated to GA). When these CPU-intensive tests
run simultaneously at full parallelism (30), they overwhelm the API server and etcd, causing cascading context deadline exceeded and etcdserver: request timed out failures. This primarily affects FIPS + TechPreview upgrade jobs, which have the highest test counts and most enabled features.
Solution
Introduce a testBucketCreator interface with two implementations:
The strategy is selected at runtime based on the job name: jobs containing both "fips" and "techpreview" use the heavy bucketing strategy.
Heavy job bucket layout
Kube phase:
OpenShift phase:
Other buckets (Storage, NetworkK8s, Builds) run at 2/3 parallelism instead of 1/2 to reduce overhead.
Impact
I have seen huge improvement for fips tp jobs with this bucket rearrangement. But the overall job duration also increased between 15m to 20m. But given the current state of the jobs, I think it is an ok comprimise.
Summary by CodeRabbit