Skip to content

runbook.md

Latest commit

 

History

History
155 lines (100 loc) · 7.75 KB

runbook.md

File metadata and controls

155 lines (100 loc) · 7.75 KB
Raw

Credit Risk Intelligence Platform - Operations Runbook

Version: 1.0
Last updated: 2026-06-26
On-call escalation: PagerDuty integration via credit-risk-drift-alerts SNS topic.

1. p95 Latency SLO Breach (> 150ms)

Alert: Grafana alert p95_latency_breach fires when p95 > 150ms for > 2 minutes.

Diagnosis steps:

  1. Check the Grafana "p95 Inference Latency" panel for the onset time.
  2. Check circuit_breaker_state panel. If state = OPEN (1), feature store is degraded - see Section 2.
  3. Check EKS pod count: kubectl get hpa credit-risk-inference -n credit-risk. If HPA is at min replicas during a traffic surge, the pod count may be the bottleneck.
  4. Check Feast retrieval latency histogram in Grafana. If feast_retrieval_latency_seconds p99 > 50ms, Redis is slow.
  5. Check ONNX inference time by looking at inference_latency_seconds breakdown in traces.

Remediation:

  • Redis slow: scale the ElastiCache cluster or check for memory pressure / eviction. Run redis-cli info stats | grep evicted_keys.
  • HPA not scaling fast enough: check HPA events with kubectl describe hpa credit-risk-inference -n credit-risk.
  • ONNX slow: check if INT8 quantized model is loaded (model_version in logs). If FP32 model is loaded, retrain and re-export.
  • Rollback: ./scripts/promote-model.sh --rollback reverts the SageMaker endpoint to the previous production model version.

2. Feature Store Circuit Breaker Open

Alert: circuit_breaker_state{dependency="feast_redis"} = 1 for > 1 minute.

Impact: Scoring requests use cached features (staleness up to 10 minutes). Accuracy is degraded but the service continues operating. circuit_breaker_open: true is set in all scoring responses.

Diagnosis steps:

  1. Check ElastiCache CloudWatch metrics: CacheHits, CacheMisses, Evictions, EngineCPUUtilization.
  2. Check Redis connection count: redis-cli -h <endpoint> info clients | grep connected_clients.
  3. Check Feast materialization logs in CloudWatch: /aws/lambda/feast-materialize.

Remediation:

  • Redis memory full: increase ElastiCache node size or reduce feature TTL in configs/production.yaml (serving.feature_cache.ttl_seconds).
  • Redis endpoint unreachable: check ElastiCache security group allows EKS node group subnet.
  • After Redis recovers, the circuit breaker will auto-close after reset_timeout_seconds (60s default). No manual action needed.
  • If circuit remains open: kubectl rollout restart deployment/credit-risk-inference -n credit-risk to clear the pod-local circuit state.

3. Drift Threshold Breach - Retraining Not Triggering

Alert: drift_psi_score{feature_name="..."} > 0.2 but no entry in retrain_trigger_total within 5 minutes.

Diagnosis steps:

  1. Check the drift detector Lambda logs: aws logs tail /aws/lambda/credit-risk-retrain-trigger --since 1h.
  2. Check whether the SNS message was delivered: SNS console -> credit-risk-drift-alerts topic -> Subscriptions -> Delivery status.
  3. Check whether a pipeline is already running: aws sagemaker list-pipeline-executions --pipeline-name credit-risk-retraining --query 'PipelineExecutionSummaries[?PipelineExecutionStatus==Executing]'.

Remediation:

  • Lambda not triggered: check SNS subscription is active and the Lambda has sagemaker:StartPipelineExecution IAM permission.
  • Pipeline already running: wait for the current execution to complete. Check progress: aws sagemaker describe-pipeline-execution --pipeline-execution-arn <arn>.
  • Manual trigger: curl -X POST https://<service>/v1/admin/retrain -H "X-Admin-Token: <token>" -d '{"reason": "manual drift response", "triggered_by": "oncall"}'.

4. Model Validation Gate Failure - Retraining Blocked

Alert: SageMaker Pipeline execution failed at the ModelValidation step.

Diagnosis steps:

  1. Open the SageMaker Studio pipeline execution graph for the failed run.
  2. Check the ModelValidation step logs. The failure reason will be one of:
    • AUC below baseline: new model did not improve over current production by 1%.
    • Quantization degradation: INT8 quantization degraded AUC by more than 0.5%.
    • Data validation failure: Great Expectations suite failed on the training dataset.

Remediation:

  • AUC below baseline: examine the MLflow experiment run. Check if training data quality degraded (Great Expectations report). If the new data genuinely produces a weaker model, investigate data pipeline for upstream quality issues.
  • Quantization degradation: re-run quantization with a larger or more representative calibration dataset. Calibration set path is in configs/production.yaml (training.quantization.calibration_batch_size).
  • Data validation failure: check Great Expectations HTML report in S3 (s3://credit-risk-prod-artifacts/ge-reports/). The failing expectation identifies the data quality issue.

5. Scoring API Pod OOMKill

Alert: kubectl get events -n credit-risk | grep OOMKill or pod restart counter increasing.

Impact: Pods that OOMKill are restarted by Kubernetes. During restart, in-flight requests fail. If all pods OOMKill simultaneously, the service is down until at least one pod restarts.

Diagnosis steps:

  1. kubectl top pods -n credit-risk - check memory usage.
  2. kubectl describe pod <pod-name> -n credit-risk - check the OOMKill event and which container was killed.
  3. Check if the ChromaDB HNSW index is loaded. If the case collection grew beyond the pod's memory limit, the index load causes OOMKill.

Remediation:

  • Increase resources.limits.memory in infra/k8s/deployment.yaml (currently 4Gi). Apply with kubectl apply -f infra/k8s/deployment.yaml.
  • If ChromaDB index is the cause: reduce the collection size or move to a dedicated ChromaDB server pod rather than in-process loading.
  • Immediate: kubectl rollout restart deployment/credit-risk-inference -n credit-risk to restart pods one at a time (rolling restart).

6. Zero-Downtime Model Rollout

Procedure for promoting a new model to production:

# 1. Validate the new model against holdout set.
./scripts/promote-model.sh --model-version <version> --stage staging

# 2. After manual review of MLflow metrics, promote to production.
./scripts/promote-model.sh --model-version <version> --stage production

# 3. Update the SageMaker endpoint (rolling deployment, zero downtime).
# This is triggered automatically by promote-model.sh on production promotion.

# 4. Monitor p95 latency and decision distribution for 15 minutes post-deploy.
# If anomalies: rollback.
./scripts/promote-model.sh --rollback

Rollback criteria:

  • p95 latency increases by > 20% after deployment.
  • Decision distribution shifts by > 15% (approval rate changes unexpectedly).
  • Any Prometheus alert fires within 15 minutes of deployment.

7. Human Review Queue Backlog

Alert: agent_routing_total{outcome="escalate"} rate increases by > 2x baseline.

Impact: Escalated applications wait in the human review queue. SLA for human review is 24 business hours.

Diagnosis steps:

  1. Check if a model drift event occurred recently (Grafana drift panels).
  2. Check if there was a data pipeline issue that caused all applications to have confidence < 0.75 (e.g., Feast returning zero features due to materialization failure).
  3. Check the escalation reason distribution in CloudWatch Logs Insights: 
    fields escalation_reason | stats count() by escalation_reason

Remediation:

  • Feast materialization failure (all features zero): run ./scripts/run-backfill.sh to trigger an immediate materialization from the offline store.
  • Model confidence systematically low: this signals concept drift. Trigger manual retraining via /v1/admin/retrain.
  • Genuine surge in borderline applications: normal operation. Monitor for return to baseline over 24-48 hours.