Skip to content

bug(termination-watcher): SpotInterruptionWarning metric emitted for all instance shutdowns, not just spot interruptions #5244

Description

@Brend-Smits

Description

Since v7.10.0 (#5055), the SpotInterruptionWarning CloudWatch metric is emitted for all EC2 instance shutdowns — not just actual spot interruption warnings. This causes the metric to be inflated by 100-350x in environments with high instance turnover.

Root Cause

PR #5055 added an EventBridge rule (${prefix}-instance-termination) that matches EC2 Instance State-change Notification events with state shutting-down and targets the same interruptionWarning Lambda handler. The intent was to trigger runner deregistration on any termination.

However, the termination-warning.ts handler unconditionally emits the SpotInterruptionWarning metric for all events when ENABLE_METRICS_SPOT_WARNING=true, without checking the event's detail-type. This means normal instance shutdowns (job completions, scale-down, manual termination) are incorrectly counted as spot interruptions.

Impact

In our production environment after upgrading to v7.10.0:

Period Lambda Invocations/day SpotInterruptionWarning Sum/day
Before (normal) 116-272 9-44
After v7.10.0 54,787-93,945 7,672-16,446

The prod-multi-instance-termination EventBridge rule's TriggeredRules metric confirms it accounts for nearly all the inflated invocations:

  • Aug 3: 54,935 (vs 54,787 total Lambda invocations)
  • Aug 4: 93,602 (vs 93,945 total Lambda invocations)

Expected Behavior

The SpotInterruptionWarning metric should only be emitted when the event is an actual EC2 Spot Instance Interruption Warning, not for EC2 Instance State-change Notification events. Runner deregistration should still trigger for both event types.

Proposed Fix

In termination-warning.ts, gate metric emission on the event's detail-type:

const isSpotInterruption = event['detail-type'] === 'EC2 Spot Instance Interruption Warning';
const metricName = isSpotInterruption && config.createSpotWarningMetric ? 'SpotInterruptionWarning' : undefined;
metricEvent(instance, event, metricName, logger);
await deregisterRunner(instance, config);

Reproduction

  1. Deploy v7.10.0 with enable_runner_deregistration = true and enable_metrics_spot_warning = true
  2. Any instance shutdown (e.g., normal runner job completion) will emit a SpotInterruptionWarning metric
  3. Observe the metric Sum growing proportionally to total instance terminations rather than actual spot interruptions

Environment

  • Module version: v7.10.0
  • AWS Region: eu-west-1
  • Runner type: multi-runner (ephemeral, spot)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions