Skip to content

Optimize API clients and Jetty runtime for Virtual Threads and high throughput - #523

Open
ludoch wants to merge 2 commits into
mainfrom
pr/virtualthreads-performance
Open

Optimize API clients and Jetty runtime for Virtual Threads and high throughput#523
ludoch wants to merge 2 commits into
mainfrom
pr/virtualthreads-performance

Conversation

@ludoch

@ludoch ludoch commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

This pull request modernizes thread handling across API clients and the Jetty runtime, adding support and protections for Java 21+ Virtual Threads:

  1. Bounded Concurrency for Virtual Threads in JdkHttpApiHostClient:

    • When appengine.api.use.virtualthreads is enabled, requests execute on virtual threads, but in-flight HTTP calls to the API host are capped using a Semaphore bounded by maxThreads (APPENGINE_API_MAX_THREADS / APPENGINE_API_MAX_CONNECTIONS, default 100).
    • This prevents unconstrained concurrency bursts and retry storms from saturating the Appserver/Datastore proxy while benefiting from virtual threads.
    • Preserves JDK 17 compilation and execution compatibility via reflective lookup of Executors.newVirtualThreadPerTaskExecutor().
  2. Bounded Thread Pools in JettyHttpApiHostClient & JdkHttpApiHostClient:

    • Caps thread pools at maxThreads (default 100) instead of unbounded/200, safely queueing requests during transient backend degradation to avoid triggering masked INTERNAL_ERROR ("Internal Datastore Error") responses.
  3. Dynamic JVM Runtime Capability Checks:

    • Updates JettyServletEngineAdapter (Jetty 12 and 12.1) and JettyContainerService (local devappserver) to dynamically check Runtime.version().feature() >= 21 when activating virtual thread pools, ensuring forward compatibility with Java 25+ without hardcoded runtime string dependencies.
  4. VirtualThreadSupport Utility Classes:

    • Introduces VirtualThreadSupport in api, api_dev, and runtime/impl using MethodHandles to dynamically detect virtual thread capability and create unstarted virtual threads on JDK 21+ while remaining fully compatible with Java 17 compile targets.
  5. Datastore Transaction Backoff Retries:

    • Implements exponential backoff retries in DatastoreServiceImpl.beginTransaction() controlled by appengine.datastore.retries (default 1).
  6. Documentation:

    • Adds runtime/runtime_impl_jetty121/API_CLIENTS.md detailing all configuration properties, thread pool bounds, virtual thread behavior, and Datastore retry settings.

…hroughput

This change modernizes thread handling across API clients and the Jetty runtime, adding support and protections for Java 21+ Virtual Threads:

1. Bounded Concurrency for Virtual Threads in JdkHttpApiHostClient:
   - When appengine.api.use.virtualthreads is enabled, requests execute on virtual threads, but in-flight HTTP calls to the API host are capped using a Semaphore bounded by maxThreads (APPENGINE_API_MAX_THREADS / APPENGINE_API_MAX_CONNECTIONS, default 100).
   - This prevents unconstrained concurrency bursts and retry storms from saturating the Appserver/Datastore proxy while benefiting from virtual threads.
   - Preserves JDK 17 compilation and execution compatibility via reflective lookup of newVirtualThreadPerTaskExecutor().

2. Bounded Thread Pools in JettyHttpApiHostClient & JdkHttpApiHostClient:
   - Caps thread pools at maxThreads (default 100) instead of unbounded/200, safely queueing requests during transient backend degradation to avoid triggering masked INTERNAL_ERROR ("Internal Datastore Error") responses.

3. Dynamic JVM Runtime Capability Checks:
   - Updates JettyServletEngineAdapter (Jetty 12 and 12.1) and JettyContainerService (local devappserver) to dynamically check Runtime.version().feature() >= 21 when activating virtual thread pools, ensuring forward compatibility with Java 25+ without hardcoded runtime string dependencies.

4. VirtualThreadSupport Utility Classes:
   - Introduces VirtualThreadSupport in api, api_dev, and runtime/impl using MethodHandles to dynamically detect virtual thread capability and create unstarted virtual threads on JDK 21+ while remaining fully compatible with Java 17 compile targets.

5. Datastore Transaction Backoff Retries:
   - Implements exponential backoff retries in DatastoreServiceImpl.beginTransaction() controlled by appengine.datastore.retries (default 1).

6. Documentation:
   - Adds runtime/runtime_impl_jetty121/API_CLIENTS.md detailing all configuration properties, thread pool bounds, virtual thread behavior, and Datastore retry settings.
…ersized payload

When streaming payloads exceeding the maximum size under HTTP connector mode, the server may abort or reset the TCP connection before HTTP 413 headers are received by the client, resulting in a response status of 0. This change accepts status 0 when a request/response failure is present, eliminating test flakiness.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant