Index gems that come from a git or path source

[paracycle]

Problem

Graph#add_workspace_dependency_paths finds a dependency's code through Gem::Specification.find_by_name. That method only knows about gems RubyGems installed. A gem the Gemfile takes from a git or path source is never registered with RubyGems, so the lookup raises Gem::MissingSpecError, the rescue swallows it, and the gem is left out of the workspace without any signal.

This is not a rare corner. On a large workspace that pins its framework to a branch, 204 of 882 locked specs were dropped: 154 from path sources, 36 from git sources. Every Rails gem was among them, so nothing in the graph defined ActiveRecord::Base.

The damage is much wider than the missing gems. Ruby resolves a constant by searching the ancestors of the enclosing namespace. A class that inherits from a missing superclass never linearizes its ancestor chain, and then no constant inside that class resolves either, however well defined it is. Constants as ordinary as Array and Hash were reported unresolved even with the RBS core definitions indexed.

Change

When RubyGems cannot find the gem, ask Bundler. Bundler keeps an index per source that knows both where it checked the source out and what the gemspec declares.

Two details matter:

• The directories come from the gemspec, not from a guess. Assuming lib is wrong often enough to matter. Of the gems resolved this way, grpc alone declares src/ruby/lib, src/ruby/bin and src/ruby/pb. Across the workspace, 77 gems declare something other than ["lib"], including lib/concurrent-ruby, ruby and client.
• Each source is indexed once. A multi-gem repository such as rails shares one source across the twelve gems it provides, and building that index reads gemspecs off disk.

A source that is not checked out yet still leaves its gems unindexed rather than failing the whole workspace, which is the behaviour today.

Effect

Measured on a workspace of about 110,000 files whose Gemfile takes Rails from a branch.

	Before	After
Locked specs dropped	204	0
Workspace paths indexed	717	909
Unresolved work items	153,583	34,601
Incomplete ancestor chains	75,028	4,009

Constants such as TestCase, Arel, RecordInvalid and RecordNotFound resolve after the change and did not before.

Notes for the reviewer

• The silent rescue is kept, because a workspace must still index when one source is unusable. It does mean a workspace can lose a quarter of its dependencies with no diagnostic. Surfacing that needs an error channel this method does not have, so I left it for separate work.
• Bundler.load.specs would be the tidier API, but it materializes the whole bundle and raises on a workspace whose lockfile and checkout disagree. The current code deliberately avoids requiring a working bundle, and this change keeps that property.
• The new tests use small doubles for the Bundler objects, so no repository is checked out while testing.

[soutaro]

I have two concerns:

1. Calling find_by_name without a version requirement can resolve to a different installed version of the gem.
2. I think checking the source before calling find_by_name, for example with lazy_spec.source.is_a?(Bundler::Source::Path), would be safer. Otherwise, an installed gem with the same name can take precedence over a gem from a git or path source. (I wrote this code before.)

Neither issue occurs when the target bundle has already been set up, but the PR description says that supporting execution without a working bundle is intentional.

[vinistock]

As discussed in Slack, I couldn't reproduce the missing definitions mentioned in the description, so we need to understand if this is some form of environment difference or a real bug.

Based on my testing, the current code does correctly index both git and path sources.