fix(perf): Avoid server-side tag filtering in listEC2Runners DescribeInstances call (~5x latency reduction)

[wadherv]

Description

The Scale-Up Lambda's runner-lookup call (listEC2Runners in lambdas/libs/compute-providers/aws/ec2/src/control-plane/runners.ts) dominates total invocation latency. X-Ray traces show total invocation duration consistently ~30s, while the actual provisioning logic (credential lookups, CreateFleet) is sub-2s:

Step	Duration
SSM GetParameters	0.05s
EC2 DescribeInstances	30.04s
EC2 CreateFleet	1.83s
SSM PutParameter	0.08s

DescribeInstances — the "how many runners do we already have" check — accounts for ~93% of the entire invocation. This pattern reproduced consistently across 10+ sampled traces over a 24-hour window (26–36s every time).

Motivation / Gap

constructFilters() (runners.ts) builds an EC2 Filters array combining instance-state-name with up to four tag:ghr:* filters (environment, Type, Owner, Application, optionally orphan), and getRunners() passes all of them to DescribeInstancesCommand — so EC2 does tag matching server-side.

Root-cause investigation ruled out the usual suspects:

• Not pagination — result sets are in the hundreds, well under AWS's ~1,000-per-page default.
• Not Lambda networking (VPC/ENI) — the same 24–33s delay reproduced identically calling the same API locally, outside the Lambda's VPC, via both boto3 and the AWS CLI.
• Not primarily about match count — a query with fewer filters but fewer matches still ran 5x faster.
• Each additional tag filter pays the same tag-index lookup cost; the current implementation stacks four, compounding into the ~30s seen today.
• Tried two AWS APIs purpose-built for tag lookups (DescribeTags, Resource Groups Tagging API) — both performed the same or worse (31.5s and ~155s respectively). This confirms the fix isn't "use a different tag-lookup API," it's "avoid server-side tag filtering in this call entirely."

Proposed Fix

Fetch by instance-state-name only (a fast, native attribute), then filter by tag in application code, instead of asking EC2 to filter by tags server-side:

// Before: EC2 filters by instance-state-name AND up to 4 tag:ghr:* filters
const ec2Filters = constructFilters(filters); // includes tag:ghr:environment, tag:ghr:Type, etc.
const instances = await ec2.send(new DescribeInstancesCommand({ Filters: ec2Filters }));

// After: EC2 filters by instance-state-name only; tags matched client-side
const stateFilter = [{ Name: 'instance-state-name', Values: ec2Statuses }];
const tagFilters = constructTagFilters(filters); // same tag logic, now applied in JS
const instances = await ec2.send(new DescribeInstancesCommand({ Filters: stateFilter }));
const matched = filterInstancesByTags(instances, tagFilters);

• Current: DescribeInstances with 4 tag filters — ~24–33s
• Proposed: DescribeInstances with only instance-state-name, tags filtered client-side — ~6.5s
• Since Instance.Tags is already returned on every instance in the response, no extra API calls are needed — the tag data is just filtered in-process instead of server-side.

Verification / Proof of Concept

Measured locally with real data:

• State-only fetch (no tag filters): 6.4s for ~950 instances.
• Client-side tag filtering of those 950 instances: <2ms.
• Total: ~6.5s vs. ~24–33s today — a ~5x improvement.

Suggested Rollout

This is a drop-in internal change to listEC2Runners/constructFilters/getRunners — the function signature and Ec2ListRunnerFilters input shape are unchanged, only the internal filtering strategy. No Terraform variable or feature flag needed; behavior is identical from the caller's perspective (pool.ts, scale-up.ts, scale-down.ts), just faster.

Environment

• Module version: (current main as of report date)
• Applies to: lambdas/libs/compute-providers/aws/ec2/src/control-plane/runners.ts — listEC2Runners, constructFilters, getRunners