Repository navigation
Conversation
Checkpoint cleanup can occupy every JobManager IO worker and starve adaptive graph creation. Production captures showed 104-127 seconds before supplier entry, followed by about 0.32 seconds of supplier work. Give only graph construction and state assignment a fixed shared recovery executor. Keep checkpoint-trigger initialization and cleanup on general IO to preserve their feedback, and retain supplier sequencing, job MDC, shared ownership and shutdown ordering. Document the additional capacity. Cover real checkpoint restore and task progress under blocked cleanup, continued checkpoint throttling, and successful late-supplier fencing. Validate routing, feedback and shutdown negative controls without adding diagnostic logging or changing checkpoint formats and retention. Generated-by: pi 1.4.0
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Addresses FLINK-41115.
We encountered production rescales where AdaptiveScheduler waited 104–127 seconds for a graph-creation worker, although executing the supplier took about 0.32 seconds. Thread dumps showed the general JobManager IO workers occupied by checkpoint-file deletion.
Change
Run only background graph construction and state assignment on a JobManager-owned, shared recovery executor. Checkpoint-trigger initialization and cleanup keep their existing IO executor and feedback relationship. Predecessor sequencing, cancellation, state ownership and shutdown ordering are preserved.
The fixed recovery pool uses
jobmanager.io-pool.size, adding up to N shared workers; configuration docs describe this. The queue remains unbounded, and synchronous storage calls inside recovery can still block. No diagnostic logging, dependency, public API signature, serializer, per-record or S3 connector changes.Verification
Real MiniCluster tests saturate general IO with deletion and verify restored-task progress before releasing it, while new checkpoint initialization still waits. Negative controls detect incorrect routing, lost checkpoint feedback and premature shutdown.
Affected runtime/checkpoint suites and core, streaming, Kubernetes and documentation verification pass on Apache master. Configuration documentation was regenerated.
Generated-by: pi 1.4.0