fix: stop calling GetSession per query inside the Glue session - #693
Open
meniluca wants to merge 2 commits into
Open
fix: stop calling GetSession per query inside the Glue session#693meniluca wants to merge 2 commits into
meniluca wants to merge 2 commits into
Conversation
The use_arrow result path resolved the session's SecurityConfiguration on every query, building a fresh boto3 client each time and calling GetSession under botocore's legacy retry policy (4 attempts, no backoff). On busy runs that exhausted the Glue API rate limit: ThrottlingException: An error occurred (ThrottlingException) when calling the GetSession operation (reached max retries: 4): Rate exceeded SecurityConfiguration is fixed for a session's lifetime, so it is now resolved once per session rather than once per query, and the in-session clients are cached and built with the adapter's boto_retry_mode / boto_retry_max_attempts. Because cursor() re-sends SQLPROXY before every query into the same interpreter, the cache sentinels are initialised only when unset; assigning them unconditionally would drop the cache before each query and leave the call rate unchanged. Measured on Glue 5.0, 2 x G.1X, eu-central-1, HEAD vs this change: GetSession 1 per query -> 1 per session 1 row 351ms -> 259ms (-26%) 1k rows x 1KB 396ms -> 270ms (-32%) 15k rows x 1KB 733ms -> 658ms (-10%) Affects use_arrow: true only; the stdout path is unchanged.
meniluca
commented
Sep 1, 2026
Member
|
/test glue 7c4aa44 |
|
✅ Python model tests triggered by @rumeshkrish for |
|
✅ S3 Tables tests triggered by @rumeshkrish for |
|
✅ Integration tests triggered by @rumeshkrish for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
resolves: #692
Description
The use_arrow result path resolved the session's SecurityConfiguration on every query, building a fresh boto3 client each time and calling GetSession under botocore's legacy retry policy (4 attempts, no backoff). On busy runs that exhausted the Glue API rate limit:
SecurityConfiguration is fixed for a session's lifetime, so it is now resolved once per session rather than once per query, and the in-session clients are cached and built with the adapter's
boto_retry_mode/boto_retry_max_attempts. Because cursor() re-sends SQLPROXY before every query into the same interpreter, the cache sentinels are initialised only when unset; assigning them unconditionally would drop the cache before each query and leave the call rate unchanged.Measured on Glue 5.0, 2 x G.1X, eu-central-1, HEAD vs this change:
Affects use_arrow: true only; the stdout path is unchanged.
Checklist
CHANGELOG.mdand added information about my change to the "dbt-glue next" section.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.