Support loading cached project data from .lscache files - #84790
Support loading cached project data from .lscache files#84790jasonmalinowski wants to merge 4 commits into
Conversation
This brings along the ProjectData projects from the internal repository at commit 11b0afc9aece97a12dada4a3c6cf12622bb33232. Only the ProjectData projects are brought along. No files were modified; file content changes will happen in a follow-up commit to make it actually build.
- Remove the specific versions of Roslyn being referenced by the compiler, so we pick up the repository-wide versions that the generators should use. - Enable ImplicitUsings which we don't have enabled globally - Remove the reference to BannedApiAnalyzers, since we already pick that up globally via a different reference. - Move back to xunit v2 to match the rest of the repository - Ensure we match the UnitTest naming convention for projects
If we can find a .lscache for a project, we will use that cached state to immediately load cached state when we queue the full loading. A full load is still queued, so if the cache is stale we'll refresh it via the usual paths.
|
Azure Pipelines: Successfully started running 2 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
Adds a new Microsoft.NET.ProjectData component under src/LanguageServer/ProjectData/ to read .lscache project data (plus supporting model/schema generation and tests), and wires it into the language server project loader to opportunistically hydrate projects from cache for faster initial availability.
Changes:
- Add
Microsoft.NET.ProjectDatalibrary: cache-file reader, path resolver, snapshot model, and supporting utilities. - Add
Microsoft.NET.ProjectData.Generators(schema-based codegen) + new unit test projects for the reader and generator. - Integrate cache loading into
LanguageServerProjectLoader/LanguageServerProjectSystem, and add the new projects to the solution.
Show a summary per file
| File | Description |
|---|---|
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/UserFolderCachePath.cs | Computes user-folder cache location for .lscache files. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/UnsupportedProjectDataMarker.cs | Reads/writes “unsupported project data” sidecar marker with fingerprints. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/StringPool.cs | Thread-safe string deduplication pool for cache-loaded data. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/StringComparers.cs | Centralized string comparer/comparison semantics for model types. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/SdkKnownFrameworkReferenceResolver.cs | Reads SDK bundled versions to resolve SDK-known packs/analyzer packages. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/Model/ProjectDataSnapshotFactory.cs | Converts parsed cache slices to canonical immutable snapshot model. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/Model/ProjectDataSnapshot.cs | Immutable snapshot type for a single configuration slice. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/Model/ProjectDataItem.cs | Immutable item representation (item spec + metadata). |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/Model/KeyValueCollection.cs | Schema-indexed key/value collection for fast property/metadata lookup. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/Model/KeySchema.cs | Key-to-index mapping using FrozenDictionary for O(1) lookups. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/Microsoft.NET.ProjectData.csproj | New library project (not packable; referenced by language server). |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/HexEncoder.cs | Lowercase hex encoding helper for hashing/fingerprinting. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/FrameworkListExpander.cs | Expands SDK ref-pack framework lists into references/analyzers. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/DTOs/CachedSourceFile.cs | DTO for cached source file + optional link metadata. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/DTOs/CachedSliceData.cs | DTO for a parsed configuration slice from a cache file. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/DTOs/CachedMetadataReference.cs | DTO for cached metadata reference + aliases/embedInteropTypes. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/DTOs/CachedEmbeddedResource.cs | DTO for cached embedded resource + metadata. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/CachePathResolver.cs | Resolves sentinel-encoded paths (<NUGET>, <DOTNET>, <NETSDK>, etc.). |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/CacheFileReader.cs | Parses .lscache files into CachedSliceData and/or snapshots. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Tests/UserFolderCachePathTests.cs | Unit tests for user-folder cache layout and base-dir selection. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Tests/UnsupportedProjectDataMarkerTests.cs | Unit tests for marker read/write/invalidation behavior. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Tests/ProjectDataSnapshotFactoryTests.cs | Unit tests for shared-slice merge + solution path injection behavior. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Tests/Microsoft.NET.ProjectData.Tests.csproj | New unit test project for Microsoft.NET.ProjectData. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Tests/CachePathResolverNuGetPpTests.cs | Unit tests for <NUGETPP> sentinel resolution and safety checks. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Tests/CachePathResolverNetSdkTests.cs | Unit tests for <NETSDK> sentinel binding/behavior. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Generators/project-data-schema.json | Canonical schema for properties/items + wire-format tokens/sentinels. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Generators/Microsoft.NET.ProjectData.Generators.csproj | New generator project for schema-driven constants/accessors. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Generators/GeneratorSourceText.cs | Helper to create source text with SHA256 checksums. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Generators/DataModelSchemaGenerator.cs | Incremental generator that reads schema JSON and emits constants/accessors. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Generators.Tests/Microsoft.NET.ProjectData.Generators.Tests.csproj | New unit test project for the generator. |
| src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Generators.Tests/DataModelSchemaGeneratorTests.cs | Validates generator emits SHA256 source hashes. |
| src/LanguageServer/ProjectData/Directory.Build.targets | ProjectData-wide build customizations (assembly info, banned symbols). |
| src/LanguageServer/ProjectData/Directory.Build.props | ProjectData-wide build settings (TFM, IVT, README packaging). |
| src/LanguageServer/ProjectData/BannedSymbols.txt | Banned API list for AOT-safe language server code. |
| src/LanguageServer/ProjectData/AssemblyInfo.cs | Sets safe default DLL import search paths for the ProjectData subtree. |
| src/LanguageServer/ProjectData/.editorconfig | Adds directory-level formatting/analyzer severity configuration. |
| src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/Microsoft.CodeAnalysis.LanguageServer.csproj | References the new Microsoft.NET.ProjectData library. |
| src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/HostWorkspace/LanguageServerProjectSystem.cs | Implements cache-based project load path using .lscache. |
| src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/HostWorkspace/LanguageServerProjectLoader.cs | Adds cache-load hook + adjusts target creation/update flow. |
| Roslyn.slnx | Adds the new ProjectData projects to the solution. |
Copilot's findings
Suppressed comments (1)
src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/HostWorkspace/LanguageServerProjectLoader.cs:562
- Same as the info log: prefer message-template logging to avoid allocating an interpolated string and to preserve structured fields.
_logger.LogWarning(e, $"Exception encountered while trying to load cached state for {projectPath}");
- Files reviewed: 41/41 changed files
- Comments generated: 5
| await loadedProject.UpdateWithNewProjectInfoAsync(cachedProject, isMiscellaneousFile: false, hasAllInformation: true, _logger); | ||
| } | ||
|
|
||
| _logger.LogInformation($"Loaded {projectPath} from the cache."); |
| protected override async Task<(ImmutableArray<ProjectFileInfo>, ProjectSystemProjectFactory)?> TryLoadProjectFromCacheAsync(string projectPath, CancellationToken cancellationToken) | ||
| { | ||
| var projectCache = await CacheFileReader.ReadProjectCacheAsync(projectPath, cacheInProject: false, cancellationToken); | ||
|
|
||
| if (projectCache.IsEmpty) | ||
| return null; | ||
|
|
||
| return (projectCache.SelectAsArray(static slice => |
| CommandLineArgs = [.. slice.CommandLineArguments], | ||
| Documents = slice.SourceFiles.SelectAsArray(static file => | ||
| new DocumentFileInfo(file.FilePath, file.Link ?? file.FilePath, isLinked: file.Link is not null, isGenerated: false, folders: [])).ToArray(), | ||
| AdditionalDocuments = slice.AdditionalFiles.SelectAsArray(static path => | ||
| new DocumentFileInfo(path, path, isLinked: false, isGenerated: false, folders: [])).ToArray(), |
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (9)
src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/HostWorkspace/LanguageServerProjectSystem.cs:212
- "IntermediateAssembly" doesn't appear in the ProjectData schema added in this PR (project-data-schema.json), so it likely won't be present in
slice.Propertiesand will always load as null. Either addIntermediateAssemblyto the schema/writer allow-list so it is persisted, or change this to a property that is guaranteed to exist in the cache (for example, usingTargetPathif the project system only needs the compilation output path).
IntermediateOutputFilePath = GetProperty("IntermediateAssembly"),
src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/HostWorkspace/LanguageServerProjectLoader.cs:256
- This method is marked
asyncbut doesn'tawait, which will produce CS1998 warnings (and can be treated as errors depending on build settings). Removeasyncand returnTask.FromResult<(ImmutableArray<ProjectFileInfo>, ProjectSystemProjectFactory)?>(null), or make it non-virtual and synchronous if async isn’t required.
protected virtual async Task<(ImmutableArray<ProjectFileInfo>, ProjectSystemProjectFactory)?> TryLoadProjectFromCacheAsync(string projectPath, CancellationToken cancellationToken)
=> null;
src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/HostWorkspace/LanguageServerProjectLoader.cs:534
- This performs file I/O using
CancellationToken.None, so cache loading can't be cancelled during shutdown/teardown or if the caller has a cancellation request. Please thread through an appropriate cancellation token (or create one scoped to project load) so cache hydration can be aborted predictably when needed.
// Try to load the contents from the project cache if we have one; we'll do this outside the lock
try
{
var cachedProjectStateAndFactory = await TryLoadProjectFromCacheAsync(projectPath, CancellationToken.None);
src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/CacheFileReader.cs:41
- This returns only
UserFolderCachePath.Compute(...), butUserFolderCachePathalso definesFileExtensionandCacheFileReaderdefines a.lscacheextension constant for project-folder caches. If user-folder caches are intended to be actual.lscachefiles, this should likely appendUserFolderCachePath.FileExtension(and keep sidecars like.unsupportedindependent). As-is, reads will miss caches written with the.lscacheextension in the user-folder layout.
public static string GetUserFolderCacheFilePath(string projectFilePath) => UserFolderCachePath.Compute(projectFilePath);
src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Generators/DataModelSchemaGenerator.cs:94
- The generator silently returns null on any schema parse error, which can lead to confusing downstream compilation failures (missing generated types) without actionable diagnostics. Prefer catching specific exceptions and reporting a
Diagnosticvia the generator context so schema issues surface as a clear build error/warning with location and details.
private static Schema? ParseSchema(string json)
{
try
{
using JsonDocument doc = JsonDocument.Parse(json);
src/LanguageServer/ProjectData/Microsoft.NET.ProjectData.Generators/DataModelSchemaGenerator.cs:177
- The generator silently returns null on any schema parse error, which can lead to confusing downstream compilation failures (missing generated types) without actionable diagnostics. Prefer catching specific exceptions and reporting a
Diagnosticvia the generator context so schema issues surface as a clear build error/warning with location and details.
catch
{
return null;
}
}
src/LanguageServer/ProjectData/BannedSymbols.txt:1
- Line 1 appears to contain a stray carriage-return character before
#. This can cause tooling that parses the banned-symbols file to misinterpret the comment marker. Please remove the leading CR so the file begins with#directly.
# Banned MessagePack APIs.
src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/HostWorkspace/LanguageServerProjectLoader.cs:555
- These logging calls use interpolated strings, which eagerly allocates even if the log level is filtered. Prefer structured logging with message templates (e.g.
LogInformation("Loaded {ProjectPath} from the cache.", projectPath)) so allocations are minimized and consumers get structured fields.
_logger.LogInformation($"Loaded {projectPath} from the cache.");
src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/HostWorkspace/LanguageServerProjectLoader.cs:562
- These logging calls use interpolated strings, which eagerly allocates even if the log level is filtered. Prefer structured logging with message templates (e.g.
LogInformation("Loaded {ProjectPath} from the cache.", projectPath)) so allocations are minimized and consumers get structured fields.
_logger.LogWarning(e, $"Exception encountered while trying to load cached state for {projectPath}");
- Files reviewed: 41/41 changed files
- Comments generated: 1
| Share = FileShare.Read, | ||
| Options = FileOptions.Asynchronous, | ||
| }); | ||
| using StreamReader reader = new(stream); | ||
|
|
||
| string projectDirectory = Path.GetDirectoryName(projectFilePath)!; | ||
|
|
| // If this wasn't our first time loading, we might have old projects we can unload | ||
| if (startingLoadState is ProjectLoadState.LoadedTargets startingLoadedTargets) |
There was a problem hiding this comment.
when do we hit this - is this if there was a cache project loaded first? If so, why is it not just like reloading an already loaded project?
| // Try to load the contents from the project cache if we have one; we'll do this outside the lock | ||
| try | ||
| { | ||
| var cachedProjectStateAndFactory = await TryLoadProjectFromCacheAsync(projectPath, CancellationToken.None); |
There was a problem hiding this comment.
Do you think this should happen inside the queue? It seems to make sense to me to treat loading a cached project, then the real one similarly to just a project reload. e.g. first time around loading a project we load cached, and add work to reload it later?
There was a problem hiding this comment.
If we have a priority queue, then maybe -- I don't want this running after other regular project loads. Or maybe we should have two queues: one for trying to read caches and one for trying to do regular builds.
| ContentFilePaths = [], | ||
| PackageReferences = [], | ||
| FileGlobs = [], |
There was a problem hiding this comment.
Are these things that should be stored by the lscache but are not? Or are they simply not necessary?
There was a problem hiding this comment.
ContentFiles I'm not even sure what they're used for; package references we use for knowing if a restore needs to happen but I wasn't generally wanting to trigger a restore in this path. File globs I don't think they store, that would be handy -- but we'll fill them in during the real design-time-build later.
| AdditionalText? schemaFile = null; | ||
| foreach (AdditionalText file in files) | ||
| { | ||
| if (Path.GetFileName(file.Path).Equals(SchemaFileName, StringComparison.OrdinalIgnoreCase)) |
There was a problem hiding this comment.
is this appropriate for non-windows?
| // CSDevKit → CSDevKit.Contracts.DataModel (its existing brokered DTO convention) | ||
| // ProjectData reader/tasks → their root namespace | ||
| string assemblyName = compilation.AssemblyName ?? ""; | ||
| string ns = assemblyName switch |
There was a problem hiding this comment.
nit - full variable names (and many other places)
| { | ||
| return []; | ||
| } | ||
| catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or FormatException) |
There was a problem hiding this comment.
various places could maybe use IOUtilities?
| { | ||
| // DIAG: cache file's project= line resolves to a different absolute path than the | ||
| // caller expected. Should not happen when the cache was written for the same project. | ||
| System.Diagnostics.Trace.TraceWarning( |
There was a problem hiding this comment.
Might need to figure out a way to surface these - afaik these won't go anywhere currently i nthe language server
| /// <summary> | ||
| /// Expands indentation-compressed paths back into full paths. | ||
| /// </summary> | ||
| internal static List<(string Path, Dictionary<string, string>? Metadata)> ExpandCompressedPaths(List<string> lines) |
There was a problem hiding this comment.
some examples (in a comment) for each of these parsing functions explaining the transformations happening would def help explain why the algorithm is what it is.
(applies to many locations)
| private static string ComputeFileFingerprint(string path) | ||
| { | ||
| using FileStream stream = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete); | ||
| using SHA256 sha256 = SHA256.Create(); |
There was a problem hiding this comment.
do we need to be using 512 at this point? might need to be upgradable as well (compliance)
| /// Represents a cached embedded resource item from the <c>[embeddedResources]</c> section | ||
| /// of a <c>.lscache</c> file, with its associated metadata. | ||
| /// </summary> | ||
| public sealed record CachedEmbeddedResource |
There was a problem hiding this comment.
We probably need the public API analyzer on this so we don't accidentally change public APIs they're using
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (7)
src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/UnsupportedProjectDataMarker.cs:1
File.Replace(...)is not supported on all platforms (it can throwPlatformNotSupportedExceptionon non-Windows). Since this marker is stored in the user cache and needs to work cross-platform, switch to an atomic cross-platform pattern (e.g.,File.Move(tempPath, path, overwrite: true)where available, or a platform-conditional fallback) so marker writes don't fail on Linux/macOS.
src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/HostWorkspace/LanguageServerProjectLoader.cs:256- This method is marked
asyncbut has noawait, which will produce CS1998 and adds avoidable overhead/confusion. Removeasyncand returnTask.FromResult<(ImmutableArray<ProjectFileInfo>, ProjectSystemProjectFactory)?>(null)(or similar) so overrides can still be async while the default implementation is allocation-free.
protected virtual async Task<(ImmutableArray<ProjectFileInfo>, ProjectSystemProjectFactory)?> TryLoadProjectFromCacheAsync(string projectPath, CancellationToken cancellationToken)
=> null;
src/LanguageServer/ProjectData/Microsoft.NET.ProjectData/CacheFileReader.cs:200
- Opening the cache file with
FileShare.Readcan fail (sharing violations) if the writer updates via atomic replace/rename or holds the file with broader sharing requirements. Consider usingFileShare.ReadWrite | FileShare.Delete(similar to other reads in this PR) so the reader can tolerate concurrent writes/replacements and treat partial data as a cache miss rather than failing to open.
FileStream stream = new(cacheFilePath, new FileStreamOptions
{
Mode = FileMode.Open,
Access = FileAccess.Read,
Share = FileShare.Read,
Options = FileOptions.Asynchronous,
});
src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/HostWorkspace/LanguageServerProjectLoader.cs:555
- These logging calls use interpolated strings rather than structured logging, which loses structured fields and allocates strings eagerly. Prefer message templates (e.g.,
LogInformation(\"Loaded {ProjectPath} from the cache.\", projectPath)andLogWarning(e, \"Exception encountered while trying to load cached state for {ProjectPath}\", projectPath)) to improve performance and observability.
_logger.LogInformation($"Loaded {projectPath} from the cache.");
src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/HostWorkspace/LanguageServerProjectLoader.cs:562
- These logging calls use interpolated strings rather than structured logging, which loses structured fields and allocates strings eagerly. Prefer message templates (e.g.,
LogInformation(\"Loaded {ProjectPath} from the cache.\", projectPath)andLogWarning(e, \"Exception encountered while trying to load cached state for {ProjectPath}\", projectPath)) to improve performance and observability.
catch (Exception e)
{
_logger.LogWarning(e, $"Exception encountered while trying to load cached state for {projectPath}");
src/LanguageServer/ProjectData/BannedSymbols.txt:1
- This file appears to contain a stray carriage return character at the very start (
\r# ...) and ends with multiple trailing blank lines. These can cause annoying diffs and (depending on tooling) parsing quirks. Please normalize the file (remove the leading CR and trim trailing blank lines) so it stays stable across platforms.
# Banned MessagePack APIs.
src/LanguageServer/ProjectData/BannedSymbols.txt:23
- This file appears to contain a stray carriage return character at the very start (
\r# ...) and ends with multiple trailing blank lines. These can cause annoying diffs and (depending on tooling) parsing quirks. Please normalize the file (remove the leading CR and trim trailing blank lines) so it stays stable across platforms.
- Files reviewed: 41/41 changed files
- Comments generated: 2
| /// <summary> | ||
| /// Gets the cache file path for a given project file path in project-folder mode. | ||
| /// </summary> | ||
| public static string GetProjectFolderCacheFilePath(string projectFilePath) => projectFilePath + CacheFileExtension; |
| /// <see cref="UserFolderCachePath"/> file (linked from the MSBuild task project) so the | ||
| /// writer and reader cannot drift.</para> | ||
| /// </summary> | ||
| public static string GetUserFolderCacheFilePath(string projectFilePath) => UserFolderCachePath.Compute(projectFilePath); |
This migrates the .lscache reading code into the Roslyn repository, and then pulls it into the C# extension's project system code so we can load state from the cache. The cache is only used to get things initialized faster -- even projects that load from the cache will still be queued for a regular load to ensure things are up to date, but hopefully that's a no-op in most cases.
Commit-at-a-time recommended; the first commit is the direct copying of the code from the current internal repo, and it's intentionally copied verbatim so the diff of what we've changed applies second. The expectation is we're going to have a small window of time where we're keeping the code in sync in both repositories, and making the diff explicit (and minimizing unrelated churn) is easiest for now. Eventually that repo will consume the NuGet packages produced by this repo, and then we'll migrate this code further to Roslyn code styles and patterns.
This PR will get a few follow-ups:
Microsoft Reviewers: Open in CodeFlow