Description
Since v7.10.0 (#5055), the SpotInterruptionWarning CloudWatch metric is emitted for all EC2 instance shutdowns — not just actual spot interruption warnings. This causes the metric to be inflated by 100-350x in environments with high instance turnover.
Root Cause
PR #5055 added an EventBridge rule (${prefix}-instance-termination) that matches EC2 Instance State-change Notification events with state shutting-down and targets the same interruptionWarning Lambda handler. The intent was to trigger runner deregistration on any termination.
However, the termination-warning.ts handler unconditionally emits the SpotInterruptionWarning metric for all events when ENABLE_METRICS_SPOT_WARNING=true, without checking the event's detail-type. This means normal instance shutdowns (job completions, scale-down, manual termination) are incorrectly counted as spot interruptions.
Impact
In our production environment after upgrading to v7.10.0:
| Period |
Lambda Invocations/day |
SpotInterruptionWarning Sum/day |
| Before (normal) |
116-272 |
9-44 |
| After v7.10.0 |
54,787-93,945 |
7,672-16,446 |
The prod-multi-instance-termination EventBridge rule's TriggeredRules metric confirms it accounts for nearly all the inflated invocations:
- Aug 3: 54,935 (vs 54,787 total Lambda invocations)
- Aug 4: 93,602 (vs 93,945 total Lambda invocations)
Expected Behavior
The SpotInterruptionWarning metric should only be emitted when the event is an actual EC2 Spot Instance Interruption Warning, not for EC2 Instance State-change Notification events. Runner deregistration should still trigger for both event types.
Proposed Fix
In termination-warning.ts, gate metric emission on the event's detail-type:
const isSpotInterruption = event['detail-type'] === 'EC2 Spot Instance Interruption Warning';
const metricName = isSpotInterruption && config.createSpotWarningMetric ? 'SpotInterruptionWarning' : undefined;
metricEvent(instance, event, metricName, logger);
await deregisterRunner(instance, config);
Reproduction
- Deploy v7.10.0 with
enable_runner_deregistration = true and enable_metrics_spot_warning = true
- Any instance shutdown (e.g., normal runner job completion) will emit a
SpotInterruptionWarning metric
- Observe the metric Sum growing proportionally to total instance terminations rather than actual spot interruptions
Environment
- Module version: v7.10.0
- AWS Region: eu-west-1
- Runner type: multi-runner (ephemeral, spot)
Description
Since v7.10.0 (#5055), the
SpotInterruptionWarningCloudWatch metric is emitted for all EC2 instance shutdowns — not just actual spot interruption warnings. This causes the metric to be inflated by 100-350x in environments with high instance turnover.Root Cause
PR #5055 added an EventBridge rule (
${prefix}-instance-termination) that matchesEC2 Instance State-change Notificationevents with stateshutting-downand targets the sameinterruptionWarningLambda handler. The intent was to trigger runner deregistration on any termination.However, the
termination-warning.tshandler unconditionally emits theSpotInterruptionWarningmetric for all events whenENABLE_METRICS_SPOT_WARNING=true, without checking the event'sdetail-type. This means normal instance shutdowns (job completions, scale-down, manual termination) are incorrectly counted as spot interruptions.Impact
In our production environment after upgrading to v7.10.0:
The
prod-multi-instance-terminationEventBridge rule'sTriggeredRulesmetric confirms it accounts for nearly all the inflated invocations:Expected Behavior
The
SpotInterruptionWarningmetric should only be emitted when the event is an actualEC2 Spot Instance Interruption Warning, not forEC2 Instance State-change Notificationevents. Runner deregistration should still trigger for both event types.Proposed Fix
In
termination-warning.ts, gate metric emission on the event'sdetail-type:Reproduction
enable_runner_deregistration = trueandenable_metrics_spot_warning = trueSpotInterruptionWarningmetricEnvironment