feat: Add X-Ray instrumentation for GitHub event delivery latency to webhook Lambda

[wadherv]

Description

The webhook Lambda (lambdas/functions/webhook/src/webhook/index.ts) is the entry point for every GitHub workflow_job event, but its X-Ray instrumentation only covers Lambda-internal and downstream-AWS-SDK timing (the auto-instrumented Lambda segment, plus SQS/EventBridge subsegments). There is no visibility into the delivery/ingestion latency between when GitHub generated the event (workflow_job.created_at) and when the webhook Lambda actually began processing it. This is not "Lambda processing lag" — the Lambda's own execution time is already visible today. What's missing is the end-to-end gap before the Lambda even starts, which could stem from GitHub's own delivery queue, network transit, API Gateway, or queuing/cold-start on the ingestion side. This metric can't attribute the delay to either side by itself — that ambiguity is precisely the gap this instrumentation is meant to close.

We hit this directly: a workflow run was triggered by GitHub at 23:28:59 but wasn't processed by our webhook Lambda until 23:37:15 — an ~8 minute gap — with no GitHub-side outage reported for that window. Diagnosing it required manually correlating the workflow run's GitHub timestamp against CloudWatch logs; there was no dashboard/trace/monitor that surfaced this lag directly.

Motivation / Gap

• readWorkflowJobEvent() already extracts event.workflow_job.created_at and logs it as a persistent key, but nothing computes or surfaces the delta against actual invocation time.
• X-Ray's existing auto-instrumentation (Lambda facade segment + AWS SDK subsegments for SQS/SSM) cannot express this, since GitHub is an external, non-instrumented caller — there's no trace-context propagation from GitHub into the webhook Lambda.
• Without this, teams operating self-hosted runner infra have to manually diagnose GitHub-vs-pipeline delay on every incident instead of alerting/dashboarding on it directly.

Proposed Fix

Add a small instrumentation helper called from readWorkflowJobEvent() (used by both publishForRunners and publishOnEventBridge) that:

1. Computes the lag between workflow_job.created_at and Date.now().
2. Adds it as an X-Ray annotation on the current segment (queryable/alertable).
3. Creates a remote-namespace subsegment named github, backdated to the event's created_at, so the service map shows a distinct github node feeding into the webhook Lambda with a span visually representing the real end-to-end delay.

This entire feature is disabled by default and gated behind a single boolean, since backdating trace timestamps is an unusual pattern that not every consumer of the module will want on by default. It's controlled via a Terraform module variable, surfaced to the Lambda as an environment variable:

variable "webhook_xray_github_latency_enabled" {
  description = "Add X-Ray instrumentation (annotation + synthetic 'github' node) measuring the delay between a GitHub event's created_at timestamp and webhook Lambda invocation. Disabled by default."
  type        = bool
  default     = false
}

// This module already wraps aws-xray-sdk-core behind a shared Powertools
// Tracer singleton (see lambdas/libs/aws-powertools-util/src/tracer/index.ts),
// so the instrumentation uses that instead of a raw aws-xray-sdk-core import.
import { tracer } from '@aws-github-runner/aws-powertools-util';

function instrumentGithubLatency(githubCreatedAt: string): void {
  if (process.env.WEBHOOK_XRAY_GITHUB_LATENCY_ENABLED !== 'true') return;

  const segment = tracer.getSegment();
  if (!segment) return;

  const createdAtMs = new Date(githubCreatedAt).getTime();
  const lagMs = Date.now() - createdAtMs;

  // Annotation on the current segment, so the lag is queryable/alertable
  // directly without needing the synthetic subsegment below.
  tracer.putAnnotation('event_lag_ms', lagMs);

  const githubNode = segment.addNewSubsegment('github');
  githubNode.namespace = 'remote';
  githubNode.start_time = createdAtMs / 1000; // X-Ray uses epoch seconds
  githubNode.addAnnotation('workflow_job_created_at', githubCreatedAt);
  githubNode.addAnnotation('event_lag_ms', lagMs);
  githubNode.close();
}

// called from readWorkflowJobEvent(), right before it returns the parsed event:
instrumentGithubLatency(event.workflow_job.created_at);

When the flag is false (the default), instrumentGithubLatency() returns immediately — no annotation, no subsegment, zero behavior change and zero added risk for existing consumers of the module.

Verification / Proof of Concept

We validated that X-Ray accepts a backdated subsegment (rather than rejecting or clipping it) using a standalone test Lambda that fabricates an 8-minute-old "github" subsegment on every invocation. Result, pulled via aws xray batch-get-traces:

xray-lag-theory-test (Lambda facade)
  └─ xray-lag-theory-test (Function segment, real duration: ~2ms)
       └─ github (subsegment, namespace: remote)
            start: ...T04:19:48Z  →  end: ...T04:27:48Z   (exactly 480s)
            annotations: lag_seconds=480.0, fake_event_created_at=...

github (promoted to standalone segment, origin: None, parent_id → function segment)
   start/end: same 480s span

• Trace-level Duration correctly rolled up to 480.022s (not clipped to the ~2ms real execution).
• IsPartial: False, no fault/error flags — a clean, valid, queryable trace.
• X-Ray promoted the subsegment to a standalone segment with its own Id, linked via parent_id — this is what renders as a distinct connected node in the Service Map.
• Caveat: since there's no real AWS resource or http block behind it, the node's origin is None, so it likely renders without a specific service icon in the console (cosmetic only — the trace data and timing are correct).

Suggested Rollout

Ship the whole feature behind webhook_xray_github_latency_enabled, defaulting to false. Consumers who want this visibility opt in explicitly; everyone else sees no change in behavior, trace volume, or shape.

Environment

• Module version: v7.11.0 (current main tag as of this report)
• Applies to: lambdas/functions/webhook/src/webhook/index.ts, both publishForRunners and publishOnEventBridge entry points