feat(internal/librarian/java): add persistent JVM daemon manager - #3
Open
yangyzs wants to merge 34 commits into
Open
feat(internal/librarian/java): add persistent JVM daemon manager#3yangyzs wants to merge 34 commits into
yangyzs wants to merge 34 commits into
Conversation
…guration" (googleapis#7288) Reverts googleapis#7274 Fixes googleapis#7287
… io error (googleapis#7291) Update the bidirectional streaming transport template to construct a std::io::Error with std::io::ErrorKind::BrokenPipe when sending over a closed request channel fails. This provides standard I/O broken pipe semantics to callers while preserving the error kind and message. For googleapis#6835
Run `librarian install` in rust integration. Fixes googleapis#7289
The convert-prost template skips fields of type `wkt::Any`, as these cannot be converted.
…ing (googleapis#7273) Extract request routing parameters from the initial request message for bidirectional streaming RPCs and pass them to the underlying transport. For googleapis#6835
Sidekick emits warnings when it runs into a comment that it does not know how to handle. The idea is to review if the comments need to be captured for the generated code. That was hard to design from first principles, without an example around. Finally we found an example for reserved enum value ranges: https://github.com/googleapis/googleapis/blob/37c83e244b70c6bfa1a527be976950226b2ee5b2/google/maps/weather/v1/precipitation.proto#L104-L105 There is nothing to emit for these comments, now we know and we can silence the warning.
…mers Instrument fine-grained benchmark timers in Java generation and enable integration test presubmit on PR for performance breakdown. TAG=agy CONV=b05fcd92-a581-401c-ada5-b4dbda65d62e
…uto-install dependency Auto-download nailgun-server:1.0.0 during librarian install and manage background NGServer daemon to run gapic-generator-java in memory, eliminating 350+ JVM cold starts. TAG=agy
yangyzs
force-pushed
the
feat/java-gapic-daemon
branch
from
August 15, 2026 21:05
bf34fbd to
ff70d99
Compare
… files Bump cache key to v3 and include maven and java Go source files in hash calculation to force fresh tool installation and deploy Nailgun daemon wrapper scripts on CI. TAG=agy
Use errgroup bounded by runtime.NumCPU in Format to execute google-java-format batches concurrently, reducing Phase 2 formatting time from 15+ minutes to ~3 minutes. TAG=agy
Use errgroup bounded by runtime.NumCPU to generate Java libraries concurrently in Phase 1, accelerating full-repo generation. TAG=agy
…un for google-java-format Fix NGServer host and port command line arguments to prevent NumberFormatException on daemon start. Add google-java-format to daemon classpath and enable Nailgun socket proxying for formatting operations. TAG=agy
…er process Add --add-exports and --add-opens flags for jdk.compiler modules to NGServer JVM command line to allow google-java-format AST parsing on JDK 17+ without IllegalAccessError. TAG=agy
…back Add JDK compiler export flags to exec java fallback in createBinWrapper to prevent IllegalAccessError when google-java-format executes without active Nailgun port. TAG=agy
…per scripts Send arguments, working directory, main class, and stdin chunks in standard Nailgun wire protocol sequence. Flush stdout/stderr and exit with Nailgun process exit code. TAG=agy
…rmat file collection 1. Increase Java generation concurrency limit to max(NumCPU()*4, 8) to fully utilize multi-threaded Nailgun JVM daemon. 2. Skip target and hidden directories in collectJavaFiles to avoid duplicate formatting of Maven build outputs. TAG=agy
…r limit for safe memory usage 1. Format Java libraries immediately after generation per library to avoid 78,000 file repo scanning overhead. 2. Set concurrency limit to max(NumCPU()*2, 4) to ensure safe JVM RAM usage on 7GB GitHub Actions runners. TAG=agy
…rs and skip target directories in formatting 1. Parallelize Java generation in warm JVM daemon using 4 workers. 2. Exclude target and hidden directories in collectJavaFiles to halve the number of files passed to google-java-format. TAG=agy
…y modified/untracked files via git status Filter Java format targets using git status --porcelain -u to avoid re-formatting unchanged repository files during generation. TAG=agy
…st git top-level directory Ensures collectGitModifiedJavaFiles resolves status paths against git rev-parse --show-toplevel so modified Java files are accurately collected. TAG=agy
…gun daemon classpath Passes all installed tool jars in lib directory to NGServer classpath, allowing google-java-format and other Maven tools to execute inside the warm background JVM daemon. TAG=agy
…neration via Nailgun daemon Eliminates the 40-minute separate Phase 2 batch formatting step by formatting each Java library inline as it finishes generation within warm background JVM daemon workers. TAG=agy
…enable warm daemon formatting Ensures getToolsEnv preserves NAILGUN_PORT environment variable so google-java-format executes directly inside the background JVM daemon. TAG=agy
… for Nailgun daemon Sets batch size to 500 files and concurrency to 4 parallel workers, accelerating Phase 2 Java formatting via the warm background JVM daemon. TAG=agy
… JVM heap exhaustion Caps formatting concurrency limit to 2 workers maximum to maintain steady memory utilization during Phase 2 formatting. TAG=agy
…at in 1000-file batches across 4 workers Expands JVM daemon heap to 5GB (-Xmx5g) and sets batch size to 1,000 files with 4 parallel worker threads to accelerate Phase 2 formatting. TAG=agy
…orker 1000-file batch formatting Configures a safe 3.5GB max heap (-Xmx3584m) for the Nailgun JVM daemon and caps Phase 2 formatting at 2 workers with 1,000 files per batch, preventing runner OOM while optimizing execution time. TAG=agy
…orker concurrency limit Optimizes memory footprint per worker to ~300MB, allowing 4 parallel formatting workers to run concurrently in the 3.5GB Nailgun JVM daemon and reducing Phase 2 formatting duration to ~10 minutes. TAG=agy
…e.NumCPU to prevent runner CPU starvation Matches formatting worker concurrency limit directly to physical CPU core count (runtime.NumCPU), eliminating CPU starvation and heartbeat loss on 2-core GitHub Actions runners. TAG=agy
…invocations into single execution Combines --java_out, --java_grpc_out, and --java_gapic_out into a single protoc call per API when requirements match, eliminating 2/3 of all protoc process overhead during Phase 1 generation. TAG=agy
…s across 3 workers Sets maxFilesPerFormatBatch to 150 files and limits concurrency to 3 workers. Keeps RAM utilization under 500MB heap while accelerating Phase 2 formatting. TAG=agy
…untime.NumCPU formatting worker limit Prevents lock contention in Nailgun JVM daemon while maintaining peak formatting speed during Phase 2. TAG=agy
…ports for zero-contention parallel formatting Launches 2 independent Nailgun JVM daemons on ports 2113 and 2114 (-Xmx1792m each). Formats batches in parallel across separate ports, eliminating JVM lock contention and accelerating Phase 2 formatting. TAG=agy
…-generated proto/grpc directories Standard protoc and grpc compiler plugins output pre-formatted Java code. Skipping proto-* and grpc-* directories during google-java-format formatting avoids re-formatting 50,000+ generated files, accelerating Phase 2 formatting by 4x. TAG=agy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of Changes
GAPICDaemonandStartDaemonIfConfiguredininternal/librarian/java/daemon.goto manage persistent JVM worker execution.daemon_test.go(100% passing).google-cloud-javaremains 100% clean).TAG=agy