Skip to content

PMM-15197 Add on-demand min-* agent templates for RC/release testing. - #394

Open
talhabinrizwan wants to merge 1 commit into
mainfrom
PMM-15197-on-demand-instances-for-rc-testing
Open

PMM-15197 Add on-demand min-* agent templates for RC/release testing.#394
talhabinrizwan wants to merge 1 commit into
mainfrom
PMM-15197-on-demand-instances-for-rc-testing

Conversation

@talhabinrizwan

@talhabinrizwan talhabinrizwan commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Adds on-demand variants of all 18 min-* package-testing agent templates (9 x64 + 9 arm64) to the PMM Jenkins master's EC2 cloud config. Today every min-* label is spot-only, which causes RC/release package testing to get interrupted by spot evictions.

  • getTemplate() gains an onDemand flag. When true: no SpotConfiguration (→ on-demand pricing), label gets a -ondemand suffix (e.g. min-ol-8-x64-ondemand), and the instance cap is set to 8 rather than the spot templates' 15.
  • Registers on-demand variants for all 9 x64 + 9 arm64 OSes, using each OS's primary instance type only — the m6a.large/m7i.large fallback pools added under PMM-15066 exist specifically to route around spot capacity/eviction issues, which don't apply once we're not on spot.

On-demand instances cost more per hour than spot, and RC/release testing is infrequent and bounded (Peter's estimate: ~72 concurrent at peak for a single package-test matrix run). Capping at 8 per OS keeps the ceiling reasonable while still comfortably covering real usage; it can be raised later if RC testing ever needs more headroom.

One thing I deliberately left out of is migrating the existing UI-configured -ondemand labels (agent-amd64-ondemand, cli-ondemand, agent-arm64-ondemand) into code. Those still live purely in the Jenkins UI, not in IaC, and folding them into percona-cd-platform properly felt like a bigger, riskier lift than I wanted to bundle into this change.

Also, I'll set up docker-ondemand directly from the Jenkins UI myself, matching how the others were originally configured, no code change needed there on your end.

Test plan

  • jenkins iac deploy applies cleanly against the PMM master
  • Confirm the 18 new -ondemand labels appear as available node templates in Jenkins
  • Once the jenkins-pipelines companion change lands, trigger a package-testing job with USE_ONDEMAND=true and confirm it schedules onto the new on-demand pool instead of spot

@nogueiraanderson nogueiraanderson left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two changes, one optional:

  1. Give the on-demand templates their own description (line 386):
onDemand ? OSType + '-ondemand' : OSType,   // String description

The plugin counts instances per description and lumps spot and on-demand together, so running spot instances eat the on-demand cap of 8. During an RC window with 8+ spot instances of an OS, the on-demand template provisions nothing. Distinct descriptions separate the pools. Best merged before the consumer PR (jenkins-pipelines PR 4325), while no instances exist under the old description.

  1. Make the on-demand agents label-exclusive (line 385):
onDemand ? Node.Mode.EXCLUSIVE : Node.Mode.NORMAL,   // Node.Mode mode

Otherwise any unlabeled job can schedule onto the paid on-demand pool.

  1. Optional: build the spot and on-demand registrations from one shared OS list, so the next OS added cannot silently miss its on-demand variant.

Everything else checks out: null spotConfig correctly selects on-demand provisioning, the constructor arguments line up, and the labels match the consumer PR.

@nogueiraanderson

Copy link
Copy Markdown
Collaborator

Validated both variants in isolation on the pmm master (Script Console harness that constructs the templates against the live ec2-plugin build without registering anything):

Check PR as-is With the two changes
On-demand template really on-demand (spotConfig == null) pass pass
Caps (8 on-demand, 15 spot) pass pass
Distinct description per variant (independent cap pools) fail pass
On-demand agents label-exclusive fail pass

The diff as-is constructs cleanly on the live plugin version (constructor arity, the new 4th parameter, all map lookups), and the two requested one-liners flip the last two rows to pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants