Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,10 +223,12 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh)
| <a name="input_runners_scale_up_lambda_memory_size"></a> [runners\_scale\_up\_lambda\_memory\_size](#input\_runners\_scale\_up\_lambda\_memory\_size) | Memory size limit in MB for scale-up lambda. | `number` | `512` | no |
| <a name="input_runners_scale_up_lambda_timeout"></a> [runners\_scale\_up\_lambda\_timeout](#input\_runners\_scale\_up\_lambda\_timeout) | Time out for the scale up lambda in seconds. | `number` | `30` | no |
| <a name="input_runners_ssm_housekeeper"></a> [runners\_ssm\_housekeeper](#input\_runners\_ssm\_housekeeper) | Configuration for the SSM housekeeper lambda. This lambda deletes token / JIT config from SSM.<br/><br/> `schedule_expression`: is used to configure the schedule for the lambda.<br/> `enabled`: enable or disable the lambda trigger via the EventBridge.<br/> `lambda_memory_size`: lambda memory size limit.<br/> `lambda_timeout`: timeout for the lambda in seconds.<br/> `config`: configuration for the lambda function. Token path will be read by default from the module. | <pre>object({<br/> schedule_expression = optional(string, "rate(1 day)")<br/> enabled = optional(bool, true)<br/> lambda_memory_size = optional(number, 512)<br/> lambda_timeout = optional(number, 60)<br/> config = object({<br/> tokenPath = optional(string)<br/> minimumDaysOld = optional(number, 1)<br/> dryRun = optional(bool, false)<br/> })<br/> })</pre> | <pre>{<br/> "config": {}<br/>}</pre> | no |
| <a name="input_scale_down_idle_confirmation_seconds"></a> [scale\_down\_idle\_confirmation\_seconds](#input\_scale\_down\_idle\_confirmation\_seconds) | Number of seconds a runner must consistently report not-busy before scale-down terminates it. GitHub's busy flag can be stale (it can read false for a runner that is actively executing a job), so a single not-busy reading is not sufficient evidence a runner is idle. Set to at least one scale-down schedule interval to require two consecutive not-busy evaluations; a busy reading resets the window. 0 keeps the previous single-reading behaviour. | `number` | `0` | no |
| <a name="input_scale_down_schedule_expression"></a> [scale\_down\_schedule\_expression](#input\_scale\_down\_schedule\_expression) | Scheduler expression to check every x for scale down. | `string` | `"cron(*/5 * * * ? *)"` | no |
| <a name="input_scale_errors"></a> [scale\_errors](#input\_scale\_errors) | List of AWS error codes that should trigger retry during scale up. This list replaces the module default scale-up retry errors | `list(string)` | <pre>[<br/> "UnfulfillableCapacity",<br/> "MaxSpotInstanceCountExceeded",<br/> "TargetCapacityLimitExceededException",<br/> "RequestLimitExceeded",<br/> "ResourceLimitExceeded",<br/> "MaxSpotInstanceCountExceeded",<br/> "MaxSpotFleetRequestCountExceeded",<br/> "InsufficientInstanceCapacity",<br/> "InsufficientCapacityOnHost"<br/>]</pre> | no |
| <a name="input_scale_up_reserved_concurrent_executions"></a> [scale\_up\_reserved\_concurrent\_executions](#input\_scale\_up\_reserved\_concurrent\_executions) | Amount of reserved concurrent executions for the scale-up lambda function. A value of 0 disables lambda from being triggered and -1 removes any concurrency limitations. | `number` | `1` | no |
| <a name="input_ssm_paths"></a> [ssm\_paths](#input\_ssm\_paths) | The root path used in SSM to store configuration and secrets. | <pre>object({<br/> root = optional(string, "github-action-runners")<br/> app = optional(string, "app")<br/> runners = optional(string, "runners")<br/> webhook = optional(string, "webhook")<br/> use_prefix = optional(bool, true)<br/> })</pre> | `{}` | no |
| <a name="input_ssm_token_ttl_seconds"></a> [ssm\_token\_ttl\_seconds](#input\_ssm\_token\_ttl\_seconds) | Optional TTL in seconds for the SSM parameters holding the runner registration token / JIT config. When set, the parameters are created with an SSM expiration policy so SSM deletes them itself after the TTL passes. Requires the Advanced parameter tier for every token parameter, which incurs additional costs. Expiration is enforced asynchronously by SSM; the SSM housekeeper lambda remains as a backstop. Must be a positive number, and should comfortably exceed the runner boot time so the config does not expire before the instance reads it. | `number` | `null` | no |
| <a name="input_state_event_rule_binaries_syncer"></a> [state\_event\_rule\_binaries\_syncer](#input\_state\_event\_rule\_binaries\_syncer) | Option to disable EventBridge Lambda trigger for the binary syncer, useful to stop automatic updates of binary distribution | `string` | `"ENABLED"` | no |
| <a name="input_subnet_ids"></a> [subnet\_ids](#input\_subnet\_ids) | List of subnets in which the action runner instances will be launched. The subnets need to exist in the configured VPC (`vpc_id`), and must reside in different availability zones (see https://github.com/github-aws-runners/terraform-aws-github-runner/issues/2904) | `list(string)` | n/a | yes |
| <a name="input_syncer_lambda_s3_key"></a> [syncer\_lambda\_s3\_key](#input\_syncer\_lambda\_s3\_key) | S3 key for syncer lambda function. Required if using an S3 bucket to specify lambdas. | `string` | `null` | no |
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ To be able to support a number of use-cases, the module has quite a lot of confi

## AWS SSM Parameters

The module uses the AWS System Manager Parameter Store to store configuration for the runners, as well as registration tokens and secrets for the Lambdas. Paths for the parameters can be configured via the variable `ssm_paths`. The location of the configuration parameters is retrieved by the runners via the instance tag `ghr:ssm_config_path`. The following default paths will be used. Tokens or JIT config stored in the token path will be deleted after retrieval by instance, data not deleted after a day will be deleted by a SSM housekeeper lambda.
The module uses the AWS System Manager Parameter Store to store configuration for the runners, as well as registration tokens and secrets for the Lambdas. Paths for the parameters can be configured via the variable `ssm_paths`. The location of the configuration parameters is retrieved by the runners via the instance tag `ghr:ssm_config_path`. The following default paths will be used. Tokens or JIT config stored in the token path will be deleted after retrieval by instance, data not deleted after a day will be deleted by a SSM housekeeper lambda. Alternatively you can set `ssm_token_ttl_seconds` to attach a native SSM expiration policy to the token / JIT config parameters so SSM deletes leftovers itself after the TTL passes. Be aware that parameter policies require the Advanced parameter tier for every token parameter, which incurs additional costs, and that expiration is enforced asynchronously by SSM. The housekeeper lambda remains active as a backstop.

Furthermore, to accommodate larger JIT configurations or other stored values, the module implements automatic tier selection for SSM parameters:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -417,6 +417,48 @@ describe('scaleUp with GHES', () => {
});
});

it('adds an expiration policy to the JIT config when SSM_TOKEN_TTL_SECONDS is set', async () => {
process.env.SSM_TOKEN_TTL_SECONDS = '3600';
await scaleUpModule.scaleUp(TEST_DATA);
expect(mockSSMClient).toHaveReceivedNthSpecificCommandWith(1, PutParameterCommand, {
Name: '/github-action-runners/default/runners/config/i-12345',
Value: 'TEST_JIT_CONFIG_ORG',
Type: 'SecureString',
Tier: 'Advanced',
Policies: expect.stringContaining('"Type":"Expiration"') as unknown as string,
});
});

it('adds an expiration policy to the registration token when SSM_TOKEN_TTL_SECONDS is set', async () => {
process.env.ENABLE_EPHEMERAL_RUNNERS = 'false';
process.env.SSM_TOKEN_TTL_SECONDS = '3600';
await scaleUpModule.scaleUp(TEST_DATA);
expect(mockOctokit.actions.createRegistrationTokenForOrg).toBeCalled();
expect(mockSSMClient).toHaveReceivedNthSpecificCommandWith(1, PutParameterCommand, {
Name: '/github-action-runners/default/runners/config/i-12345',
Type: 'SecureString',
Tier: 'Advanced',
Policies: expect.stringContaining('"Type":"Expiration"') as unknown as string,
});
});

it('does not add an expiration policy when SSM_TOKEN_TTL_SECONDS is not set', async () => {
await scaleUpModule.scaleUp(TEST_DATA);
expect(mockSSMClient).toHaveReceivedNthSpecificCommandWith(1, PutParameterCommand, {
Name: '/github-action-runners/default/runners/config/i-12345',
Value: 'TEST_JIT_CONFIG_ORG',
Type: 'SecureString',
Tier: 'Standard',
Policies: undefined,
});
});

it('rejects an invalid SSM_TOKEN_TTL_SECONDS before creating runners', async () => {
process.env.SSM_TOKEN_TTL_SECONDS = 'not-a-number';
await expect(scaleUpModule.scaleUp(TEST_DATA)).rejects.toThrow('SSM_TOKEN_TTL_SECONDS must be a positive number');
expect(mockSSMClient).not.toHaveReceivedCommand(PutParameterCommand);
});

it('quotes runner labels with semicolon separators in non-ephemeral runner config', async () => {
process.env.ENABLE_EPHEMERAL_RUNNERS = 'false';
process.env.RUNNERS_MAXIMUM_COUNT = '2';
Expand Down
46 changes: 46 additions & 0 deletions lambdas/libs/aws-ssm-util/src/index.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,52 @@ describe('Test getParameter and putParameter', () => {
Tier: expectedTier,
});
});

it('Puts parameters without an expiration policy when no TTL is given', async () => {
// Arrange
mockSSMClient.on(PutParameterCommand).resolves({ $metadata: { httpStatusCode: 200 } });

// Act
await putParameter('testParam', 'test', false);

// Assert
expect(mockSSMClient).toHaveReceivedCommandWith(PutParameterCommand, {
Name: 'testParam',
Value: 'test',
Type: 'String',
Tier: 'Standard',
Policies: undefined,
});
});

it('Puts parameters with an expiration policy and forces Advanced tier when a TTL is given', async () => {
// Arrange
vi.useFakeTimers();
vi.setSystemTime(new Date('2026-01-01T00:00:00.000Z'));
mockSSMClient.on(PutParameterCommand).resolves({ $metadata: { httpStatusCode: 200 } });

try {
// Act
await putParameter('testParam', 'test', true, { ttlSeconds: 3600 });

// Assert
expect(mockSSMClient).toHaveReceivedCommandWith(PutParameterCommand, {
Name: 'testParam',
Value: 'test',
Type: 'SecureString',
Tier: 'Advanced',
Policies: JSON.stringify([
{
Type: 'Expiration',
Version: '1.0',
Attributes: { Timestamp: '2026-01-01T01:00:00.000Z' },
},
]),
});
} finally {
vi.useRealTimers();
}
});
});

describe('Test getParameters (batch)', () => {
Expand Down
21 changes: 19 additions & 2 deletions lambdas/libs/aws-ssm-util/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -109,20 +109,37 @@ export async function putParameter(
parameter_name: string,
parameter_value: string,
secure: boolean,
options: { tags?: Tag[] } = {},
options: { tags?: Tag[]; ttlSeconds?: number } = {},
): Promise<void> {
const client = ssmClient();

// Determine tier based on parameter_value size
const valueSizeBytes = Buffer.byteLength(parameter_value, 'utf8');

// Parameter policies (e.g. Expiration) are only supported on the Advanced
// tier, so a TTL forces the tier regardless of the value size. Expiration is
// enforced asynchronously by SSM: treat it as cleanup, not a security boundary.
const expiration =
options.ttlSeconds !== undefined
? JSON.stringify([
{
Type: 'Expiration',
Version: '1.0',
Attributes: {
Timestamp: new Date(Date.now() + options.ttlSeconds * 1000).toISOString(),
},
},
])
: undefined;

await client.send(
new PutParameterCommand({
Name: parameter_name,
Value: parameter_value,
Type: secure ? 'SecureString' : 'String',
Tags: options.tags,
Tier: valueSizeBytes >= SSM_ADVANCED_TIER_THRESHOLD ? 'Advanced' : 'Standard',
Tier: expiration || valueSizeBytes >= SSM_ADVANCED_TIER_THRESHOLD ? 'Advanced' : 'Standard',
Policies: expiration,
}),
);
}
1 change: 1 addition & 0 deletions lambdas/libs/storage-providers/aws/ssm/environment.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ declare global {
SSM_PARAMETER_STORE_TAGS?: string;
SSM_CONFIG_PATH?: string;
SSM_TOKEN_PATH?: string;
SSM_TOKEN_TTL_SECONDS?: string;
PARAMETER_GITHUB_APP_ID_NAME?: string;
PARAMETER_GITHUB_APP_KEY_BASE64_NAME?: string;
PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME?: string;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ describe('aws_ssm runner config store', () => {
vi.clearAllMocks();
process.env = { ...cleanEnv };
delete process.env.SSM_PARAMETER_STORE_TAGS;
delete process.env.SSM_TOKEN_TTL_SECONDS;
process.env.SSM_TOKEN_PATH = '/runner/tokens';
});

Expand Down Expand Up @@ -63,6 +64,21 @@ describe('aws_ssm runner config store', () => {
});
});

it('passes the configured token TTL to SSM', async () => {
process.env.SSM_TOKEN_TTL_SECONDS = '3600';
await createAwsSsmRunnerConfigStore().create({ runnerId: 'runner-1', value: 'jit-config' });
expect(putParameterMock).toHaveBeenCalledWith('/runner/tokens/runner-1', 'jit-config', true, {
tags: [],
ttlSeconds: 3600,
});
});

it('rejects an invalid token TTL before writing', () => {
process.env.SSM_TOKEN_TTL_SECONDS = 'not-a-number';
expect(() => createAwsSsmRunnerConfigStore()).toThrow('SSM_TOKEN_TTL_SECONDS must be a positive number');
expect(putParameterMock).not.toHaveBeenCalled();
});

it.each([undefined, '', ' '])('rejects missing or blank SSM_TOKEN_PATH %j before writing', (tokenPath) => {
setTokenPath(tokenPath);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,13 @@ import type {} from './environment';
import { createAwsSsmStorageLogger, getErrorNames } from './logger';
import { loadSsmParameterStoreTagsFromEnvironment } from './parameter-store-tags';

import { parseSsmTokenTtlSeconds } from './token-ttl';

const logger = createAwsSsmStorageLogger('runner-config-store');

export interface AwsSsmRunnerConfigStoreConfig {
tokenPath: string;
tokenTtlSeconds?: number;
parameterStoreTags: ReadonlyArray<Readonly<{ Key: string; Value: string }>>;
}

Expand All @@ -28,6 +31,7 @@ export function createAwsSsmRunnerConfigStore(config?: AwsSsmRunnerConfigStoreCo

return new AwsSsmRunnerConfigStore({
tokenPath,
tokenTtlSeconds: parseSsmTokenTtlSeconds(process.env.SSM_TOKEN_TTL_SECONDS),
parameterStoreTags: loadSsmParameterStoreTagsFromEnvironment(),
});
}
Expand All @@ -46,6 +50,7 @@ class AwsSsmRunnerConfigStore implements RunnerConfigStore {

try {
await putParameter(parameterName, record.value, true, {
ttlSeconds: this.config.tokenTtlSeconds,
tags: [
...(options.metadata ?? []).map(({ key, value }) => ({ Key: key, Value: value })),
...this.config.parameterStoreTags,
Expand Down
19 changes: 19 additions & 0 deletions lambdas/libs/storage-providers/aws/ssm/token-ttl.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import { describe, expect, it } from 'vitest';

import { parseSsmTokenTtlSeconds } from './token-ttl';

describe('parseSsmTokenTtlSeconds', () => {
it.each([
[undefined, undefined],
['', undefined],
[' ', undefined],
['3600', 3600],
['1', 1],
])('parses %j to %j', (input, expected) => {
expect(parseSsmTokenTtlSeconds(input)).toBe(expected);
});

it.each([['not-a-number'], ['0'], ['-10']])('throws on invalid value %j', (input) => {
expect(() => parseSsmTokenTtlSeconds(input)).toThrow('SSM_TOKEN_TTL_SECONDS must be a positive number');
});
});
10 changes: 10 additions & 0 deletions lambdas/libs/storage-providers/aws/ssm/token-ttl.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
export function parseSsmTokenTtlSeconds(ttl: string | undefined): number | undefined {
if (!ttl || ttl.trim() === '') {
return undefined;
}
const ttlSeconds = parseInt(ttl);
if (isNaN(ttlSeconds) || ttlSeconds <= 0) {
throw new Error(`SSM_TOKEN_TTL_SECONDS must be a positive number, got "${ttl}"`);
}
return ttlSeconds;
}
4 changes: 4 additions & 0 deletions lambdas/libs/storage-providers/storage-providers.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
import { describe, expect, it, vi } from 'vitest';

import { createAwsSsmRunnerConfigStore } from './aws/ssm/runner-config-store';

import { createStorageProviders } from './storage-providers';

vi.mock('./aws/ssm/runner-config-store', () => ({
Expand All @@ -20,6 +22,7 @@ describe('createStorageProviders', () => {
const environment = Object.freeze({
RUNNER_CONFIG_STORAGE_PROVIDER: 'AWS_SSM',
SSM_TOKEN_PATH: '/runners/tokens',
SSM_TOKEN_TTL_SECONDS: '3600',
SSM_CONFIG_PATH: '/runners/config',
SSM_PARAMETER_STORE_TAGS: JSON.stringify([{ Key: 'Environment', Value: 'test' }]),
PARAMETER_GITHUB_APP_ID_NAME: 'app-id',
Expand All @@ -32,6 +35,7 @@ describe('createStorageProviders', () => {

const storage = createStorageProviders(environment);

expect(createAwsSsmRunnerConfigStore).toHaveBeenCalledWith(expect.objectContaining({ tokenTtlSeconds: 3600 }));
expect(storage).toEqual({
runnerConfig: expect.any(Object),
runnerGroupCache: expect.any(Object),
Expand Down
5 changes: 4 additions & 1 deletion lambdas/libs/storage-providers/storage-providers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ import { loadRunnerConfigConsumerConfigFromEnvironment } from './runner-config-c
import { loadSsmParameterStoreTagsFromEnvironment } from './aws/ssm/parameter-store-tags';
import { resolveRunnerConfigStorageProvider, runnerConfigStorageProvider } from './provider';

import { parseSsmTokenTtlSeconds } from './aws/ssm/token-ttl';

type Environment = Readonly<Record<string, string | undefined>>;

const logger = createChildLogger('storage-providers');
Expand All @@ -19,6 +21,7 @@ export function createStorageProviders(environment: Environment = process.env):
}

const tokenPath = required(environment.SSM_TOKEN_PATH, 'SSM_TOKEN_PATH');
const tokenTtlSeconds = parseSsmTokenTtlSeconds(environment.SSM_TOKEN_TTL_SECONDS);
const configPath = required(environment.SSM_CONFIG_PATH, 'SSM_CONFIG_PATH');
const parameterStoreTags = loadSsmParameterStoreTagsFromEnvironment(environment);
const consumerConfig = loadRunnerConfigConsumerConfigFromEnvironment(environment);
Expand All @@ -29,7 +32,7 @@ export function createStorageProviders(environment: Environment = process.env):
});

return {
runnerConfig: createAwsSsmRunnerConfigStore({ tokenPath, parameterStoreTags }),
runnerConfig: createAwsSsmRunnerConfigStore({ tokenPath, tokenTtlSeconds, parameterStoreTags }),
runnerGroupCache: createAwsSsmRunnerGroupCacheStore({ configPath, parameterStoreTags }),
consumer: createAwsSsmRunnerConfigConsumer({ SSM_TOKEN_PATH: tokenPath }, consumerConfig),
...createCommonStorage(environment),
Expand Down
1 change: 1 addition & 0 deletions main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,7 @@ module "runners" {
tokens = "${var.ssm_paths.runners}/tokens"
config = "${var.ssm_paths.runners}/config"
}
ssm_token_ttl_seconds = var.ssm_token_ttl_seconds

s3_runner_binaries = var.enable_runner_binaries_syncer ? {
arn = module.runner_binaries[0].bucket.arn
Expand Down
Loading
Loading