feat: add optional Glue retry with exponential backoff and HTTP-level metrics #144
Draft
jamespfaulkner wants to merge 3 commits into
Draft
feat: add optional Glue retry with exponential backoff and HTTP-level metrics #144jamespfaulkner wants to merge 3 commits into
jamespfaulkner wants to merge 3 commits into
Conversation
… metrics Glue calls occasionally fail with ConcurrentModificationException or timeouts. Retries are off by default (safe for HMS threads) and enabled via GLUE_RETRY_ENABLED=true for Dronefly. A RequestHandler2-based GlueMetricRequestHandler records per-operation call duration and error counts via Micrometer for full observability. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ation - Switch GlueMetricRequestHandler to beforeAttempt/afterAttempt hooks so metrics fire per HTTP attempt (including retries) rather than once per logical call - Clarify 'retries disabled' log: SDK default retries still apply when custom policy is off; the custom policy adds CME retry on top - Fix GlueSyncCli to share a single AWSGlue client across ApiaryGlueSync, GluePartitionService, and GlueDatabaseService so the 600s request timeout applies to all table operations - Rename maxAttempts -> maxRetries throughout to match SDK semantics (maxErrorRetry is a retry count, not total attempts) - Unify retry metric tag namespace: use getErrorCode() for AmazonServiceException in both retry condition branches Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…counter only Per-call duration and error total metrics via RequestHandler2 were removed in favour of the existing ApiaryGlueSync success/failure counters, which already provide sufficient operational coverage. The glue_listener_retry_attempt counter from GlueClientFactory is retained to surface CME and throttle retry activity. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GlueClientFactoryto centraliseAWSGlueclient construction with optional exponential-backoff retry supportGLUE_RETRY_ENABLED=true. Retries onConcurrentModificationExceptionand standard transientAWS errors using
PredefinedRetryPolicies.DEFAULT_BACKOFF_STRATEGYGlueMetricRequestHandler(AWS SDK v1RequestHandler2) to record HTTP-level call metrics via Micrometer:glue_client_call_duration{operation,result},glue_client_error_total{operation,error_code}, andglue_listener_retry_attempt{exception_type}New env vars
GLUE_RETRY_ENABLEDtrueto enable retries (Dronefly only)GLUE_RETRY_MAX_ATTEMPTS3New metrics
glue_client_call_durationoperation,resultglue_client_error_totaloperation,error_codeglue_listener_retry_attemptexception_typeTest plan
GlueClientFactoryTest— retry condition,ConcurrentModificationExceptionhandling, metric recording, disabled-by-defaultGlueMetricRequestHandlerTest— success/error tagging, operation name derivation, zero-duration fallbackMetricServiceTest— 3 new methods covered🤖 Generated with Claude Code