feat: Add feature to enable dynamic ec2 config via workflow labels by edersonbrilhante · Pull Request #5003 · github-aws-runners/terraform-aws-github-runner

edersonbrilhante · 2026-01-19T20:24:29Z

Summary

This PR resumes and completes the work started in #4529.

It also allows to use any other dynamic labels with prefix ghr-. Giving support for unique labels per job or per group of jobs

It ensures that EC2-specific config can be defined via run-ons

How to test:

Use your regular labels, and add ghr-ec2-instance-type and ghr-ec2-image-id

run-ons:
  - <regular-labels>
  - ghr-ec2-instance-type:<different-instance-type>
  - ghr-ec2-image-id:<different-ami-id>

In this case:

The runner is resolved from <regular-labels>
The EC2 instance type and AMI ID are taken exactly from the provided labels

npalm · 2026-01-30T20:23:04Z

@edersonbrilhante great to see this PR.

andrecastro · 2026-02-06T17:34:50Z

This is a really interesting feature!

Just one suggestion from my side: would it be possible to support a whitelist of allowed instance types?

Also, it could be really powerful to have some kind of feature-flag / policy control over which parts of the configuration are allowed to be dynamic. For example, in my org we don’t want developers to be able to select arbitrary AMIs (only a pre-approved set), but it would be awesome to still allow them to choose the instance type for workflow jobs, as long as it’s constrained to an allowed list.

Maybe the "feature flag" is not even necessary as long as we could define the "allowed values" for each configuration, with this we could list only the pre-approved AMIs.

edersonbrilhante · 2026-02-06T18:32:25Z

@andrecastro I liked and makes a lot of sense to me. I just need more time to think about the implementation. And tbis PR is already big enough XD. I could create a following for adding this restricted values feature

stuartp44

I am happy to approve, but I do have a statement about incorrect labels and the effect on the process.

stuartp44 · 2026-02-20T13:30:50Z

I also agree with what was previously mentioned; we probably need a safelist, as we don't want lateral movement when a compromised pipeline is used, especially with the VPC setting. Maybe worth some "allowed_instance_type" setting or something to that effect that can be checked against, and if not in the list, ignored.

edersonbrilhante · 2026-03-09T17:40:38Z

@stuartp44 We can add the safelist, but I think we need a deeper discussion about the implementation

npalm · 2026-03-17T18:30:03Z

@edersonbrilhante tried to test the feature but so far not got it wokring

Used labels: runs-on: [self-hosted, x64, linux, ubuntu-latest, ghr-ec2-instance-type:c5.2xlarge, ghr-ec2-ebs-volume-size:200] But need to check what is worng. I enabled the feature as well rebuild the lambda stack.

edersonbrilhante · 2026-03-17T19:03:02Z

Can you print the logs from dispatch-to-runner? Check if the labels were accepted or not.

npalm

Great feature. Maybe we can add a small saftety net in the disspachter. A clear warning in the docs. And refer in the variable where users can enable to the risk in the docs.

edersonbrilhante · 2026-05-08T12:23:04Z

@npalm can you review

Co-authored-by: Niek Palm <npalm@users.noreply.github.com>

feedback processed - good to merge.

Brend-Smits

LGTM, tested it and it seems to work. Documentation looking great.
Thanks a lot!

Let's improve it from here by adding allow/deny lists 👍🏼

🤖 I have created a release *beep* *boop* --- ## [7.7.0](v7.6.1...v7.7.0) (2026-06-11) ### Features * Add feature to enable dynamic ec2 config via workflow labels ([#5003](#5003)) ([c68445d](c68445d)) * add support for macos runners ([#4930](#4930)) ([3e179a3](3e179a3)) * Introduce Amazon Linux 2023 ARM image ([#4780](#4780)) ([e572ae5](e572ae5)) * relax cpu_options schema and add amd_sev_snp + nested_virtualization support ([#5039](#5039)) ([5a3746d](5a3746d)) * **runner-role:** Enable using separate IAM role for runners ([#4875](#4875)) ([6642e57](6642e57)) ### Bug Fixes * **ci:** sign auto-generated docs commits ([#5154](#5154)) ([a6af4d2](a6af4d2)) * **runners:** wire job_retry.lambda_memory_size and lambda_timeout ([#5120](#5120)) ([404785e](404785e)) * **scale-up:** Add ec2:TerminateInstances permission to scale-up Lambda IAM policy ([#5152](#5152)) ([94c4e12](94c4e12)) * **scale-up:** prevent negative TotalTargetCapacity when runners exceed maximum ([#5062](#5062)) ([9ab7410](9ab7410)) * **webhook:** Fix publish events to EventBridge ([#5143](#5143)) ([a72b737](a72b737)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>

piotrj · 2026-06-11T09:18:35Z

@edersonbrilhante @Brend-Smits I'm working on getting rid of conflicts in #5077 and there is a Priority value that kind of clashes with this feature.

I wonder what's the use case of Priority being set by dynamic label? Priority makes sense mostly (and possibly only) when you define an EC2 fleet that has multiple instance_types and have prioritised allocation type. Then during a scale-up it tries to launch the instance type with the highest priority (and then going down the list in case of no available capacity).

For the case of dynamic labels where you create a separate fleet and that fleet has only 1 instance type it doesn't really make sense to define the priority.

edersonbrilhante · 2026-06-11T13:03:21Z

@piotrj the logic is the same as today: "the first match", then uses the dynamic labels to override the launch template.
Your feature is compatible. The non-dynamic labels will find the first match, using the priority config, then it will override the ec2 configs passed by ec2-*

…5314) ## Description Dynamic EC2 labels allow a workflow to request attribute-based instance selection using `InstanceRequirements`. This is useful when a workload needs a specific CPU and memory shape while allowing EC2 Fleet to choose among several eligible instance families for Spot capacity. Today, `generateFleetOverrides` combines every configured static `InstanceType` with the dynamic `InstanceRequirements` object. AWS treats these fields as mutually exclusive within one Fleet override, so `CreateFleet` rejects the request. The dynamic labels parse successfully, but no runner can launch. This change gives the two selection modes separate request paths: - Fixed selection continues to produce one override per subnet and instance type. - Attribute-based selection produces one `InstanceRequirements`-only override per subnet, allowing EC2 Fleet to choose from the eligible families. - Configurations containing both `InstanceType` and `InstanceRequirements` are rejected before an AWS request is sent. Existing fixed-instance behavior is unchanged. ## Test Plan - Added regression coverage proving requirements-only overrides omit `InstanceType`. - Added regression coverage proving conflicting fixed and attribute-based selection sends no `CreateFleet` request. - Ran the focused EC2 runner suite: 71 tests passed. - Ran the compute-provider suite: 10 files, 272 tests passed. - Ran the compute-provider Prettier and ESLint checks. - Ran `git diff --check`. Validated the request against AWS using a temporary launch template without runner userdata or an IAM profile. A requirements-only Spot Fleet request for 4 vCPU / 8 GiB across several allowed compute families completed successfully and selected `c7i-flex.xlarge` with no Fleet errors. The instance and temporary launch template were deleted afterward. The project typecheck currently reports existing Vitest 4 configuration errors in untouched configuration files. Those unrelated formatting and type changes are intentionally excluded from this PR. ## Related Issues Fixes #5315. Related to #1400 and follows the dynamic EC2 label support introduced in #5003.

edersonbrilhante marked this pull request as ready for review January 20, 2026 16:16

edersonbrilhante requested review from a team as code owners January 20, 2026 16:16

edersonbrilhante force-pushed the feat-ec2-dynamic-config branch 5 times, most recently from 5dab5e4 to afdfb36 Compare February 5, 2026 13:24

edersonbrilhante-sbg-cisco mentioned this pull request Feb 11, 2026

isJobQueued check races with SQS visibility timeout and label-based runner assignment, silently dropping jobs #5026

Closed

stuartp44 reviewed Feb 20, 2026

View reviewed changes

Comment thread lambdas/functions/webhook/src/runners/dispatch.test.ts

Comment thread lambdas/functions/control-plane/src/aws/runners.test.ts

edersonbrilhante force-pushed the feat-ec2-dynamic-config branch from afdfb36 to 0e4f870 Compare March 6, 2026 23:13

edersonbrilhante force-pushed the feat-ec2-dynamic-config branch from aae08eb to 750a9ff Compare March 9, 2026 20:53

edersonbrilhante requested a review from stuartp44 March 9, 2026 20:57

edersonbrilhante force-pushed the feat-ec2-dynamic-config branch 4 times, most recently from ad2f612 to e67a37e Compare March 11, 2026 19:41

npalm previously requested changes Apr 9, 2026

View reviewed changes

Comment thread lambdas/functions/webhook/src/runners/dispatch.ts Outdated

Comment thread docs/configuration.md

Comment thread lambdas/functions/webhook/src/runners/dispatch.ts

edersonbrilhante requested a review from npalm June 10, 2026 11:37

npalm and others added 3 commits June 10, 2026 13:40

feat: Add feature to enable dynamic instance types via workflow labels

bf78641

docs: auto update terraform docs

fa10204

feat: allow to use dynamic instance type in multiple events

c592a46

edersonbrilhante and others added 16 commits June 10, 2026 13:50

style: fix formatting

b3a695e

docs: auto update terraform docs

15ae8ee

fix: fix wrong variable

4b076d6

fix: replace old envvar for new envvar

c556a1e

style: fix tflint

e40f63d

fix: fix envvar in dispatcher

e739284

test: remove wrong tests

ff4bcc4

docs: add documentation and mask as experimental

7e6a5df

docs: update variable description

cfa8ce6

docs: auto update terraform docs

6b5660d

docs: auto update terraform docs

b0213a0

test dynamic labels

e18e4db

test dynamic labels

7c70b17

Update docs/configuration.md

28e2f18

Co-authored-by: Niek Palm <npalm@users.noreply.github.com>

feat: add sanitize function dynamic labels

26573ef

docs: auto update terraform docs

aef7049

edersonbrilhante force-pushed the feat-ec2-dynamic-config branch from eca2581 to e6e19ce Compare June 10, 2026 11:50

style: fix format issue

f987e00

edersonbrilhante force-pushed the feat-ec2-dynamic-config branch from e6e19ce to f987e00 Compare June 10, 2026 11:57

Brend-Smits approved these changes Jun 10, 2026

View reviewed changes

Brend-Smits merged commit c68445d into github-aws-runners:main Jun 10, 2026
42 checks passed

runners-releaser Bot mentioned this pull request Jun 10, 2026

chore(main): release 7.7.0 #5151

Merged

piotrj mentioned this pull request Jun 11, 2026

feat(runners): support OnDemandOptions.AllocationStrategy in EC2 Fleet #5077

Merged

edersonbrilhante deleted the feat-ec2-dynamic-config branch June 22, 2026 09:21

slavab89 mentioned this pull request Aug 28, 2026

fix(runners): support instance requirements in EC2 Fleet overrides #5314

Merged

Uh oh!

Conversation

edersonbrilhante commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to test:

Uh oh!

npalm commented Jan 30, 2026

Uh oh!

andrecastro commented Feb 6, 2026

Uh oh!

edersonbrilhante commented Feb 6, 2026

Uh oh!

stuartp44 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

stuartp44 commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edersonbrilhante commented Mar 9, 2026

Uh oh!

npalm commented Mar 17, 2026

Uh oh!

edersonbrilhante commented Mar 17, 2026

Uh oh!

npalm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

edersonbrilhante commented May 8, 2026

Uh oh!

Brend-Smits left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

piotrj commented Jun 11, 2026

Uh oh!

edersonbrilhante commented Jun 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

edersonbrilhante commented Jan 19, 2026 •

edited

Loading

stuartp44 commented Feb 20, 2026 •

edited

Loading