nodejs: packages/google-cloud-compute/protos/protos.js exceeding GitHub's 100 MB limit

[suztomo]

Originally this issue is found in googleapis/librarian#6638.

suztomo@suztomo:~/librarian-2026/google-cloud-node$ git add . && git commit -m "chore: generation diff as of July 6th"
...
suztomo@suztomo:~/librarian-2026/google-cloud-node$ gh pr create --draft
? Where should we push the 'generation-diff-july-6th' branch? suztomo/google-cloud-node

Creating draft pull request for suztomo:generation-diff-july-6th into main in googleapis/google-cloud-node

? Title (required) chore: generation diff as of July 6th
? Choose a template PULL_REQUEST_TEMPLATE.md
? Body <Received>
? What's next? Submit as draft
remote: warning: File packages/google-cloud-aiplatform/protos/protos.js is 60.58 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB        
remote: error: Trace: 5d11ea147b4f5e0c2de65ae57df0821cd99c10947e820974ff17ef788628b340        
remote: error: See https://gh.io/lfs for more information.        
remote: error: File packages/google-cloud-compute/protos/protos.js is 102.10 MB; this exceeds GitHub's file size limit of 100.00 MB        
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.        
To https://github.com/suztomo/google-cloud-node.git
 ! [remote rejected]       HEAD -> generation-diff-july-6th (pre-receive hook declined)
error: failed to push some refs to 'https://github.com/suztomo/google-cloud-node.git'
failed to run git: exit status 1

https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-compute/protos/protos.js (as of today) has 98 MB.

Changing the push destination to the upstream (https://github.com/googleapis/google-cloud-node.git) didn't help:

remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.        
To https://github.com/googleapis/google-cloud-node.git
 ! [remote rejected]       HEAD -> generation-diff-july-6th (pre-receive hook declined)
error: failed to push some refs to 'https://github.com/googleapis/google-cloud-node.git'
failed to run git: exit status 1

Ran command: (cd librarian && echo "librarian: $(git rev-parse HEAD)") && (cd google-cloud-node && echo "google-cloud-node: $(git rev-parse HEAD)")
Ran command: cd google-cloud-node && git ls-files | grep -i compileprotos
Viewed compileProtos.ts:1-466

How this big file is generated

Step-by-Step Generation Pipeline

graph TD
    A["librarian generate<br/>google-cloud-compute"] --> B["1. gapic-generator-typescript<br/>(Generates TypeScript &<br/>src/*/*_proto_list.json)"]
    B --> C["2. gapic-node-processing<br/>(combine-library)<br/>(Moves from owl-bot-staging<br/>to package output)"]
    C --> D["3. copyMissingProtos()<br/>(Copies referenced proto<br/>files to protos/)"]
    D --> E["4. compileProtos (gapic-tools)<br/>(Executes compileProtos.ts)"]
    E --> F["5. pbjs (protobufjs-cli)<br/>(Compiles protos into<br/>protos/protos.js &<br/>protos/protos.json)"]
    F --> G["6. fixJsFile()<br/>(Re-exports from google-gax<br/>& adds Apache 2.0 header)"]
    G --> H["7. pbts (protobufjs-cli)<br/>(Generates protos/protos.d.ts<br/>from protos.js)"]

Loading

---

Detailed Step Breakdown

1. Librarian Entrypoint & GAPIC Generation

• Source: internal/librarian/nodejs/generate.go:L50-L83 (Generate())
• Actions:
  • Librarian resolves the library target (packages/google-cloud-compute) and iterates over API versions (google/cloud/compute/v1, v1beta, etc.).
  • Runs gapic-generator-typescript on each API version.
  • Output: Generates TypeScript client code and a manifest of required .proto files at src/<version>/<api>_proto_list.json in owl-bot-staging/.

2. Staging & Package Assembly

• Source: internal/librarian/nodejs/generate.go:L342-L387 (movePackageFromStaging())
• Actions:
  • Runs gapic-node-processing (combine-library) to move generated source files from owl-bot-staging/ into packages/google-cloud-compute/.

3. Proto File Collection

• Source: internal/librarian/nodejs/generate.go:L497-L540 (copyMissingProtos())
• Actions:
  • Reads src/*/*_proto_list.json.
  • Copies all referenced .proto files from googleapis into packages/google-cloud-compute/protos/.

4. Executing compileProtos

• Source: internal/librarian/nodejs/generate.go:L304
• Actions:
  • Librarian executes compileProtos src (provided by gapic-tools).

5. Proto Compilation with pbjs (The 99 MB File Generation)

• Source: core/packages/tools/src/compileProtos.ts:L259-L352 (compileProtos())
• Actions:
  • findProtoJsonFiles('src') finds all *_proto_list.json files in src/.
  • pbjsMain() executes protobufjs-cli/pbjs:

    pbjs -r google_cloud_compute_protos --target static-module -p protos -o protos/protos.js <proto_files...>

  • pbjs generates a single, static JavaScript module containing object definitions and serialization/deserialization logic for every Google Compute Engine proto message and enum across v1 and v1beta. Because Compute Engine has an unusually massive API surface, this unminified JS file reaches ~99 MB.

6. Post-Processing & Header Injection

• Source: core/packages/tools/src/compileProtos.ts:L168-L189 (fixJsFile())
• Actions:
  • Replaces direct protobufjs/minimal imports with google-gax/build/src/protobuf.
  • Prepends the Apache 2.0 copyright header to protos/protos.js.

7. TypeScript Declaration Generation (protos.d.ts)

• Source: core/packages/tools/src/compileProtos.ts:L344-L351, fixDtsFile()
• Actions:
  • pbtsMain() executes protobufjs-cli/pbts on protos/protos.js to generate protos/protos.d.ts (31 MB).

[quirogas]

consider using esbuild --minify as a post-processing step.

[suztomo]

This large file size reaches library users' environments.

---

Here is a size comparison between the Node.js library (@google-cloud/compute) and the Java library (com.google.cloud:google-cloud-compute v1.104.0):

Metric	Node.js (@google-cloud/compute)	Java (com.google.cloud:google-cloud-compute)
Download Artifact / Package Size	~40 MB (tarball / npm download)	4.7 MB (.jar file)
Uncompressed / Installed Size	173 MB (package only) 207 MB (with dependencies in node_modules)	22 MB (uncompressed .jar contents)
Single Largest File	protos.js: 96 MB (JS) protos.d.ts: 31 MB (TS definitions)	Individual .class files (typically < 100 KB each)

Key Reasons for the Size Difference

1. Protobuf Code Generation in JS/TS:

   • Node.js uses protobufjs to compile the massive Google Cloud Compute proto schema into a single monolithic JavaScript bundle (protos.js) and TypeScript declaration file (protos.d.ts).
   • protos.js alone is 96 MB of text/code, which is why git pushes of the generated source in issue googleapis/google-cloud-node#8889 trigger GitHub's 100 MB file limit (packages/google-cloud-compute/protos/protos.js exceeded 100 MB during generation diffs).

2. JVM Bytecode vs. Plain JS Text:

   • Java compiles proto messages into compact JVM .class bytecode rather than readable text source code.
   • The compiled bytecode packs efficiently into ZIP/JAR format (4.7 MB compressed, 22 MB uncompressed).

[suztomo]

esbuild --minify

Let me explore options.

Minify experiment

I minified packages/google-cloud-compute/protos/protos.js in #8831. All checks have passed.

GitHub pull request UI: The diff was not visible anyway

The last change to the file couldn't show the diff in the UI:

After minifying, this does not change:

Git command line: Minify makes it impossible to see the diff

Before minifying, the Git command is still able to show the diff:

suztomo@suztomo:~/librarian-2026/google-cloud-node$ git show 849770cc54b5b19d13a378b8dfa15e4806d2387f -- packages/google-cloud-compute/protos/protos.js |head -30
commit 849770cc54b5b19d13a378b8dfa15e4806d2387f
Author: Shivanee <46910562+shivanee-p@users.noreply.github.com>
Date:   Wed Jun 10 16:59:42 2026 -0700

    chore: update generator version to newest release v4.12.0 for google-cloud-compute (#8526)

diff --git a/packages/google-cloud-compute/protos/protos.js b/packages/google-cloud-compute/protos/protos.js
index 0ccd3b9927..5ed79c6ca3 100644
--- a/packages/google-cloud-compute/protos/protos.js
+++ b/packages/google-cloud-compute/protos/protos.js
@@ -89,7 +89,7 @@
                         function AWSV4Signature(properties) {
                             if (properties)
                                 for (var keys = Object.keys(properties), i = 0; i < keys.length; ++i)
-                                    if (properties[keys[i]] != null)
+                                    if (properties[keys[i]] != null && keys[i] !== "__proto__")
                                         this[keys[i]] = properties[keys[i]];
                         }
     
@@ -211,9 +211,13 @@
                          * @throws {Error} If the payload is not a reader or valid buffer
                          * @throws {$protobuf.util.ProtocolError} If required fields are missing
                          */
-                        AWSV4Signature.decode = function decode(reader, length, error) {
+                        AWSV4Signature.decode = function decode(reader, length, error, long) {
                             if (!(reader instanceof $Reader))
                                 reader = $Reader.create(reader);
+                            if (long === undefined)
+                                long = 0;
+                            if (long > $Reader.recursionLimit)

After minifying the file, trying to show the diff (git show 33b06b9) freezes the terminal.

[suztomo]

Option: Splitting via Multiple pbjs Invocations (Per API Version)

While pbjs itself cannot split a single run into multiple files, compileProtos in gapic-tools currently passes all .proto files from all versions (v1, v1beta, v1small) into a single pbjs call, producing the single 99 MB protos.js file.

• If compileProtos (or Librarian's post-processing) was updated to invoke pbjs separately per API version, the generated files would split as follows:
  • protos/v1.js: 47 MB
  • protos/v1beta.js: 56 MB

Would splitting via multiple `pbjs` invocations require breaking changes to user code? -> No

Short Answer: No, it can be done without breaking changes for standard library users.

1. How Users Currently Import Types & Clients

Users of @google-cloud/compute typically interact with the library in two ways:

• Client Instantiation (No impact):

  import { InstancesClient } from '@google-cloud/compute';
  const client = new InstancesClient();

• Protobuf Type Access (No impact if aggregated at top level):

  import { protos } from '@google-cloud/compute';
  
  // Accessing v1 or v1beta types:
  type Instance = protos.google.cloud.compute.v1.IInstance;

2. Why Splitting Won't Break Public API

Currently, src/index.ts exports the combined protos namespace:

// src/index.ts
import * as protos from '../protos/protos';
export { protos };

If protos.js is split per API version (e.g. protos/v1.js and protos/v1beta.js):

• src/index.ts can simply aggregate and re-export both into the existing protos export:

  import * as v1Protos from '../protos/v1';
  import * as v1betaProtos from '../protos/v1beta';
  
  export const protos = {
    google: {
      cloud: {
        compute: {
          v1: v1Protos.google.cloud.compute.v1,
          v1beta: v1betaProtos.google.cloud.compute.v1beta,
        }
      }
    }
  };

• Because the top-level export protos.google.cloud.compute.v1.* remains structurally identical, existing user code is 100% backward-compatible.

3. Potential Edge Cases

• Deep Internal Imports (Non-public API):
  If a user illegally imports deep internal build artifacts (e.g., require('@google-cloud/compute/build/protos/protos')), that internal path would change. However, deep imports of un-exported internal paths are not part of the library's supported public API contract.

[suztomo]

Option: Git Large File Storage - https://git-lfs.github.com

No, Git Large File Storage does not work for google-cloud-node.

• Git-on-Borg: Git-on-Borg does not support the Git LFS protocol (go/ggob-policy#git-lfs-support). Any git push containing LFS pointer objects to GoB or GGoB mirrors will be rejected.
• GitHub Actions: GitHub Actions supports LFS via actions/checkout (with lfs: true). However, for the googleapis org, downloading 100 MB LFS blobs on every PR build quickly consumes GitHub's LFS storage & bandwidth quotas.
• Cloud Builds: Cloud Build can run git-lfs within build steps, provided LFS server credentials and binaries are configured.

What "provided LFS server credentials and binaries are configured" means

When a Git repository uses Git LFS, running git clone by default only downloads pointer files (small text files containing a SHA256 hash). To retrieve the actual large binary/source files during a build in Cloud Build, two things are required:

---

1. Pre-installed git-lfs Binary

The default Cloud Build git builder container (gcr.io/cloud-builders/git) only includes standard git and does not include the git-lfs command or hooks.

To run git lfs pull or git lfs fetch, the build container must:

• Have the git-lfs package installed (e.g., via apt-get install git-lfs or by using a custom Docker builder image that includes git-lfs).
• Run git lfs install to set up the required Git filter hooks (filter.lfs.smudge / clean).

---

2. LFS Server Authentication (Credentials)

Git LFS uses a separate HTTP REST API endpoint (e.g., https://github.com/<org>/<repo>.git/info/lfs) to download object payloads.

Because git-lfs operates outside standard git clone transport for private repositories:

• The build step must pass an authentication token (e.g. a GitHub Personal Access Token or SSH key) to git-lfs.
• In Cloud Build, this token is typically stored securely in Secret Manager and injected into the step via .netrc, HTTP insteadOf configs, or environment variables.

---

Example Configuration in cloudbuild.yaml

steps:
  - name: 'ubuntu'
    entrypoint: 'bash'
    secretEnv: ['GITHUB_TOKEN']
    args:
      - '-c'
      - |
        # 1. Install git-lfs binary
        apt-get update && apt-get install -y git git-lfs
        git lfs install

        # 2. Configure LFS server credentials via GitHub token
        git config --global url."https://$$GITHUB_TOKEN@github.com/".insteadOf "https://github.com/"

        # 3. Download actual LFS file payloads
        git lfs pull

availableSecrets:
  secretManager:
    - versionName: 'projects/MY_PROJECT/secrets/github-token/versions/latest'
      env: 'GITHUB_TOKEN'

---

Relevant Documentation

• Cloud Build Secret Manager Integration: Google Cloud Documentation: Use Secret Manager in Cloud Build
• Git LFS Client Specification & Server API: GitHub: Git LFS HTTP API Spec

[suztomo]

After the survey, I learned that the file size issue is not stemmed from Librarian. It was a matter of time if google-cloud-node/core/packages/tools/src/compileProtos.ts continues to generate the file in the current manner, regardless of OwlBot or Librarian.

My current recommendation is that the NodeJS team implements "Splitting via Multiple pbjs Invocations (Per API Version)".

[suztomo]

Impact to the documentation Generation job

The last job http://sponge/a0790f73-c736-4bb9-8f36-1ed5149fe7b4 timed out with npm run docs command:

Parsing /b/f/w/src/git/build-dir/packages/google-cloud-compute/protos/protos.js ...
Generating output files...

(not minified; but somehow CloudRAD has "6.13.0" (the latest version https://docs.cloud.google.com/nodejs/docs/reference/compute/latest/compute/v1.addressesclient)

I tried run npm run docs in my gLinux but it does not return.

In my gLinux, with the minified branch, the npm run docs command succeeded: http://gpaste/5880525414858752.

[suztomo]

Bazelbot hit this (b/535317787).