Optimize API clients and Jetty runtime for Virtual Threads and high throughput - #523
Open
ludoch wants to merge 2 commits into
Open
Optimize API clients and Jetty runtime for Virtual Threads and high throughput#523ludoch wants to merge 2 commits into
ludoch wants to merge 2 commits into
Conversation
…hroughput
This change modernizes thread handling across API clients and the Jetty runtime, adding support and protections for Java 21+ Virtual Threads:
1. Bounded Concurrency for Virtual Threads in JdkHttpApiHostClient:
- When appengine.api.use.virtualthreads is enabled, requests execute on virtual threads, but in-flight HTTP calls to the API host are capped using a Semaphore bounded by maxThreads (APPENGINE_API_MAX_THREADS / APPENGINE_API_MAX_CONNECTIONS, default 100).
- This prevents unconstrained concurrency bursts and retry storms from saturating the Appserver/Datastore proxy while benefiting from virtual threads.
- Preserves JDK 17 compilation and execution compatibility via reflective lookup of newVirtualThreadPerTaskExecutor().
2. Bounded Thread Pools in JettyHttpApiHostClient & JdkHttpApiHostClient:
- Caps thread pools at maxThreads (default 100) instead of unbounded/200, safely queueing requests during transient backend degradation to avoid triggering masked INTERNAL_ERROR ("Internal Datastore Error") responses.
3. Dynamic JVM Runtime Capability Checks:
- Updates JettyServletEngineAdapter (Jetty 12 and 12.1) and JettyContainerService (local devappserver) to dynamically check Runtime.version().feature() >= 21 when activating virtual thread pools, ensuring forward compatibility with Java 25+ without hardcoded runtime string dependencies.
4. VirtualThreadSupport Utility Classes:
- Introduces VirtualThreadSupport in api, api_dev, and runtime/impl using MethodHandles to dynamically detect virtual thread capability and create unstarted virtual threads on JDK 21+ while remaining fully compatible with Java 17 compile targets.
5. Datastore Transaction Backoff Retries:
- Implements exponential backoff retries in DatastoreServiceImpl.beginTransaction() controlled by appengine.datastore.retries (default 1).
6. Documentation:
- Adds runtime/runtime_impl_jetty121/API_CLIENTS.md detailing all configuration properties, thread pool bounds, virtual thread behavior, and Datastore retry settings.
…ersized payload When streaming payloads exceeding the maximum size under HTTP connector mode, the server may abort or reset the TCP connection before HTTP 413 headers are received by the client, resulting in a response status of 0. This change accepts status 0 when a request/response failure is present, eliminating test flakiness.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request modernizes thread handling across API clients and the Jetty runtime, adding support and protections for Java 21+ Virtual Threads:
Bounded Concurrency for Virtual Threads in
JdkHttpApiHostClient:appengine.api.use.virtualthreadsis enabled, requests execute on virtual threads, but in-flight HTTP calls to the API host are capped using aSemaphorebounded bymaxThreads(APPENGINE_API_MAX_THREADS/APPENGINE_API_MAX_CONNECTIONS, default 100).Executors.newVirtualThreadPerTaskExecutor().Bounded Thread Pools in
JettyHttpApiHostClient&JdkHttpApiHostClient:maxThreads(default 100) instead of unbounded/200, safely queueing requests during transient backend degradation to avoid triggering maskedINTERNAL_ERROR("Internal Datastore Error") responses.Dynamic JVM Runtime Capability Checks:
JettyServletEngineAdapter(Jetty 12 and 12.1) andJettyContainerService(local devappserver) to dynamically checkRuntime.version().feature() >= 21when activating virtual thread pools, ensuring forward compatibility with Java 25+ without hardcoded runtime string dependencies.VirtualThreadSupportUtility Classes:VirtualThreadSupportinapi,api_dev, andruntime/implusingMethodHandlesto dynamically detect virtual thread capability and create unstarted virtual threads on JDK 21+ while remaining fully compatible with Java 17 compile targets.Datastore Transaction Backoff Retries:
DatastoreServiceImpl.beginTransaction()controlled byappengine.datastore.retries(default 1).Documentation:
runtime/runtime_impl_jetty121/API_CLIENTS.mddetailing all configuration properties, thread pool bounds, virtual thread behavior, and Datastore retry settings.