diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index c9eb61b50..847952c45 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -460,7 +460,10 @@ A **.NET CLI tool** (`cswinrtimplgen.exe`) published as a **Native AOT** binary. 2. Creates a new empty assembly (the "impl" assembly) 3. Copies well-known assembly attributes (version, debug info, etc.) 4. Emits `[TypeForwarder]` entries for all public top-level types, routing to the appropriate projection assembly -5. Optionally signs with a strong-name key +5. Synthesizes a portable PDB for the forwarder and embeds it (see "Forwarder debug information" below) +6. Optionally signs with a strong-name key + +**Forwarder debug information**: the forwarder is emitted as metadata rather than compiled, so it has no symbols of its own, and there are none to carry over either (the input is compiled with `ProduceOnlyReferenceAssembly`, and a reference-only compilation emits no debug information at all). Since the forwarder is the assembly that ships in `lib/` of a projection NuGet package, that gap makes the whole package report as having no symbols. The tool therefore synthesizes the debug information itself (`Writers/PortablePdbWriter.cs`, `Writers/DebugDirectoryWriter.cs`), emitting the same debug directory a deterministic build with embedded symbols produces — `CodeView`, `PdbChecksum`, `Reproducible`, and `EmbeddedPortablePdb`. The embedded portable PDB carries a single, embedded, generated document (`/_/.TypeForwards.g.cs`, written by `Writers/TypeForwardsDocumentWriter.cs`) listing every type forward in the assembly as the C# that would produce it, plus the compilation options and metadata references that tooling expects. Everything is derived from the forwarder itself, so the output stays byte-for-byte deterministic. **Debug repro support**: when `--debug-repro-directory` is provided, captures the output assembly and all reference assemblies along with a faithful `.rsp` into a self-contained `impl-debug-repro.zip`. The tool also accepts a `.zip` as input and replays the captured run. diff --git a/.github/skills/testing/SKILL.md b/.github/skills/testing/SKILL.md index 0ca2b5d30..18c16cae4 100644 --- a/.github/skills/testing/SKILL.md +++ b/.github/skills/testing/SKILL.md @@ -242,7 +242,7 @@ public async Task InvalidType_Warns() - `RestoreSources` overrides all inherited NuGet sources: the local CsWinRT build output (`CsWinRTPackageSource`) plus the `CsWinRTDependencies` feed (`CsWinRTDependenciesSource`), which provides the preview Windows SDK ref pack and `Microsoft.Windows.SDK.Contracts` - `CsWinRTPackageVersion`/`CsWinRTPackageSource` default to the local `build.cmd x64 Release` output and are overridden by the build/CI that produced the package -**How they run:** `run-smoke-tests.ps1` (parameterized by `-Test` and `-Runtime`) builds and runs the consumption app (asserting a clean exit code), builds the authoring component and verifies the generated `Authoring.winmd` defines `Authoring.Greeter`, and builds each reference-projection library (`Projection`, `WindowsSdkProjection`, `WindowsSdkXamlProjection`) verifying it produces both a forwarder and a `ref` reference assembly (shared verification). The consumption and authoring tests run on both CoreCLR and Native AOT (`-Runtime`); the three reference-projection tests are build-only and run on CoreCLR only. It is invoked after the `nuget pack` step in `src/build.cmd` (x64 only; skippable via `cswinrt_run_smoke_tests=false`) and as individual steps in `build/AzurePipelineTemplates/CsWinRT-PublishToNuGet-Steps.yml`. +**How they run:** `run-smoke-tests.ps1` (parameterized by `-Test` and `-Runtime`) builds and runs the consumption app (asserting a clean exit code), builds the authoring component and verifies the generated `Authoring.winmd` defines `Authoring.Greeter`, and builds each reference-projection library (`Projection`, `WindowsSdkProjection`, `WindowsSdkXamlProjection`) verifying it produces both a forwarder and a `ref` reference assembly, and that the forwarder ships embedded symbols (shared verification). The consumption and authoring tests run on both CoreCLR and Native AOT (`-Runtime`); the three reference-projection tests are build-only and run on CoreCLR only. It is invoked after the `nuget pack` step in `src/build.cmd` (x64 only; skippable via `cswinrt_run_smoke_tests=false`) and as individual steps in `build/AzurePipelineTemplates/CsWinRT-PublishToNuGet-Steps.yml`. ### 7. WinMD generator tests (`src/Tests/WinMDGeneratorTest/`) @@ -283,6 +283,22 @@ public void RefArrayParameter_IsRejected() - Each test is a single `AssertSuccess`/`AssertFailure` call; the runner makes the exit-code and error-output assertions - Failure cases assert the tool exits non-zero and its output contains the expected `CSWINRTWINMDGEN` error id +### 8. Impl generator tests (`src/Tests/ImplGeneratorTest/`) + +**What it tests:** End-to-end behavior of the `cswinrtimplgen` build tool (the forwarder generator), focusing on the **PE-level shape of the forwarder assembly** it produces. That assembly ships in `lib/` of a projection NuGet package, so what it carries (in particular its debug information) is directly observable by consumers and by NuGet package health checks. + +**When to add tests here:** For anything about the forwarder assembly itself — its debug directory, embedded portable PDB, determinism, or type forwards. Reference-projection *codegen* is covered by `ProjectionWriterTest/`, and the full packaging layout by `SmokeTests/`. + +**Project settings:** Same shape as `WinMDGeneratorTest/` above, referencing `WinRT.Impl.Generator` instead, with the tool path passed via the `ImplGeneratorAssemblyPath` `AssemblyMetadata` item. + +**Test classes:** +| Test class | What it tests | +|------------|---------------| +| `Test_DebugDirectory` | The forwarder carries the debug directory of a deterministic build with embedded symbols (`CodeView`, `PdbChecksum`, `Reproducible`, `EmbeddedPortablePdb`), that those entries are mutually consistent, that the embedded PDB has one embedded `/_` prefixed document describing the type forwards plus valid compiler flags, and that two runs are byte identical | + +**Test helper (in `Helpers/`):** +- `ImplGeneratorRunner` — compiles a C# input assembly (optionally with symbols), runs the actual tool as a subprocess, and exposes PE/PDB inspection helpers (`GetDebugDirectoryEntryTypes`, `GetEmbeddedPortablePdbDocumentNames`, `GetEmbeddedPortablePdbDocumentText`, `GetCompilationOptionValue`, `IsCodeViewEntryConsistent`, `IsPdbChecksumValid`, `RunTwice`). + ## Deciding where to add tests | You want to test... | Add test to... | @@ -298,6 +314,7 @@ public void RefArrayParameter_IsRejected() | XAML visual tree element lifetime | `ObjectLifetimeTests/` | | WinRT component authoring patterns | `AuthoringTest/` | | A WinMD generator failure mode (a `CSWINRTWINMDGEN` error) | `WinMDGeneratorTest/` (add to `Test_ParameterConventions` or `Test_InvalidInputs`) | +| The forwarder assembly's PE shape, symbols, or determinism | `ImplGeneratorTest/` (add to `Test_DebugDirectory`) | | The produced NuGet package works end-to-end (real `ref`/`lib` assemblies, generators) | `SmokeTests/` (`Consumption/`, `Authoring/`, or a reference-projection project) | | Generated projection code patterns or cross-ABI control flow | Update `TestComponentCSharp/` and add tests in `UnitTest/` or `FunctionalTests/` | diff --git a/build/AzurePipelineTemplates/CsWinRT-PublishToNuGet-Steps.yml b/build/AzurePipelineTemplates/CsWinRT-PublishToNuGet-Steps.yml index 0c318ce77..70a787181 100644 --- a/build/AzurePipelineTemplates/CsWinRT-PublishToNuGet-Steps.yml +++ b/build/AzurePipelineTemplates/CsWinRT-PublishToNuGet-Steps.yml @@ -149,6 +149,7 @@ steps: continueOnError: true inputs: targetType: filePath + pwsh: true filePath: $(Build.SourcesDirectory)\src\Tests\SmokeTests\run-smoke-tests.ps1 arguments: -PackageSource "$(ob_outputDirectory)\packages" -PackageVersion "$(NugetVersion)" -Test Consumption -Runtime CoreCLR workingDirectory: $(Build.SourcesDirectory) @@ -158,6 +159,7 @@ steps: continueOnError: true inputs: targetType: filePath + pwsh: true filePath: $(Build.SourcesDirectory)\src\Tests\SmokeTests\run-smoke-tests.ps1 arguments: -PackageSource "$(ob_outputDirectory)\packages" -PackageVersion "$(NugetVersion)" -Test Consumption -Runtime NativeAot workingDirectory: $(Build.SourcesDirectory) @@ -167,6 +169,7 @@ steps: continueOnError: true inputs: targetType: filePath + pwsh: true filePath: $(Build.SourcesDirectory)\src\Tests\SmokeTests\run-smoke-tests.ps1 arguments: -PackageSource "$(ob_outputDirectory)\packages" -PackageVersion "$(NugetVersion)" -Test Authoring -Runtime CoreCLR workingDirectory: $(Build.SourcesDirectory) @@ -176,6 +179,7 @@ steps: continueOnError: true inputs: targetType: filePath + pwsh: true filePath: $(Build.SourcesDirectory)\src\Tests\SmokeTests\run-smoke-tests.ps1 arguments: -PackageSource "$(ob_outputDirectory)\packages" -PackageVersion "$(NugetVersion)" -Test Authoring -Runtime NativeAot workingDirectory: $(Build.SourcesDirectory) @@ -185,6 +189,7 @@ steps: continueOnError: true inputs: targetType: filePath + pwsh: true filePath: $(Build.SourcesDirectory)\src\Tests\SmokeTests\run-smoke-tests.ps1 arguments: -PackageSource "$(ob_outputDirectory)\packages" -PackageVersion "$(NugetVersion)" -Test Projection -Runtime CoreCLR workingDirectory: $(Build.SourcesDirectory) @@ -194,6 +199,7 @@ steps: continueOnError: true inputs: targetType: filePath + pwsh: true filePath: $(Build.SourcesDirectory)\src\Tests\SmokeTests\run-smoke-tests.ps1 arguments: -PackageSource "$(ob_outputDirectory)\packages" -PackageVersion "$(NugetVersion)" -Test WindowsSdkProjection -Runtime CoreCLR workingDirectory: $(Build.SourcesDirectory) @@ -203,6 +209,7 @@ steps: continueOnError: true inputs: targetType: filePath + pwsh: true filePath: $(Build.SourcesDirectory)\src\Tests\SmokeTests\run-smoke-tests.ps1 arguments: -PackageSource "$(ob_outputDirectory)\packages" -PackageVersion "$(NugetVersion)" -Test WindowsSdkXamlProjection -Runtime CoreCLR workingDirectory: $(Build.SourcesDirectory) diff --git a/build/AzurePipelineTemplates/CsWinRT-Test-Steps.yml b/build/AzurePipelineTemplates/CsWinRT-Test-Steps.yml index 762986fd1..d8799cdba 100644 --- a/build/AzurePipelineTemplates/CsWinRT-Test-Steps.yml +++ b/build/AzurePipelineTemplates/CsWinRT-Test-Steps.yml @@ -65,6 +65,20 @@ steps: --no-build testRunTitle: Projection Writer Tests +# Run Impl Generator Tests. Gated to x64 only for the same reason as the other generator tests: these +# run the 'cswinrtimplgen' tool end-to-end as a separate process (which is always built for the build +# host), so the target platform makes no difference to what they cover. + - task: DotNetCoreCLI@2 + displayName: Run Impl Generator Tests + condition: and(succeeded(), eq(variables['BuildPlatform'], 'x64')) + inputs: + command: test + projects: 'src/Tests/ImplGeneratorTest/ImplGeneratorTest.csproj' + arguments: > + /p:platform=$(BuildPlatform);configuration=$(BuildConfiguration) + --no-build + testRunTitle: Impl Generator Tests + # Run Host Tests - task: CmdLine@2 displayName: Run Host Tests diff --git a/src/Tests/ImplGeneratorTest/Helpers/ImplGeneratorRunner.cs b/src/Tests/ImplGeneratorTest/Helpers/ImplGeneratorRunner.cs new file mode 100644 index 000000000..0d37115f9 --- /dev/null +++ b/src/Tests/ImplGeneratorTest/Helpers/ImplGeneratorRunner.cs @@ -0,0 +1,578 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Diagnostics; +using System.IO; +using System.IO.Compression; +using System.Linq; +using System.Reflection; +using System.Reflection.Metadata; +using System.Reflection.PortableExecutable; +using System.Text; +using Basic.Reference.Assemblies; +using Microsoft.CodeAnalysis; +using Microsoft.CodeAnalysis.CSharp; +using Microsoft.CodeAnalysis.Emit; +using Microsoft.CodeAnalysis.Text; + +namespace ImplGeneratorTest.Helpers; + +/// +/// Runs the impl generator (cswinrtimplgen) end-to-end and inspects the forwarder assembly it produces. +/// +/// +/// Each entry point compiles a small C# input assembly, runs the actual tool as a separate process, and +/// returns the resulting forwarder for the test to assert on. +/// +internal static class ImplGeneratorRunner +{ + /// + /// The Source Link document map embedded in the compiled input assemblies. + /// + private const string SourceLinkJson = """{"documents":{"*":"https://example.invalid/*"}}"""; + + /// + /// The custom debug information kind for an embedded source document. + /// + private static readonly Guid EmbeddedSourceKind = new("0E8A571B-6926-466E-B4AD-8AB04611F5FE"); + + /// + /// The custom debug information kind for the compilation options. + /// + private static readonly Guid CompilationOptionsKind = new("B5FEEC05-8CD0-4A83-96DA-466284BB4BD8"); + + /// + /// Compiles an input assembly and runs the impl generator over it, invoking + /// with the path of the compiled input and the path of the generated forwarder. + /// + /// The debug information to emit for the input assembly ( to emit none). + /// The callback receiving the input assembly path and the forwarder assembly path. + /// Whether to have the generator strong name the forwarder. + public static void Run(DebugInformationFormat? debugInformationFormat, Action assert, bool strongName = false) + { + string toolPath = GetGeneratorPath(); + string temporaryDirectory = Directory.CreateTempSubdirectory("ImplGeneratorTest_").FullName; + + try + { + string inputAssemblyPath = CompileInputAssembly(temporaryDirectory, debugInformationFormat, strongName); + string forwarderDirectory = Directory.CreateDirectory(Path.Combine(temporaryDirectory, "forwarder")).FullName; + string responseFilePath = Path.Combine(temporaryDirectory, "args.rsp"); + + string keyArgument = strongName ? $"\n--assembly-originator-key-file {GetStrongNameKeyPath()}" : ""; + + File.WriteAllText(responseFilePath, $""" + --reference-assembly-paths {inputAssemblyPath} + --output-assembly-path {inputAssemblyPath} + --generated-assembly-directory {forwarderDirectory} + --treat-warnings-as-errors False + """ + keyArgument); + + (int exitCode, string output) = RunTool(toolPath, responseFilePath); + + Assert.AreEqual(0, exitCode, output); + + string forwarderPath = Path.Combine(forwarderDirectory, Path.GetFileName(inputAssemblyPath)); + + Assert.IsTrue(File.Exists(forwarderPath), $"The forwarder was not generated at '{forwarderPath}'."); + + assert(inputAssemblyPath, forwarderPath); + } + finally + { + TryDeleteDirectory(temporaryDirectory); + } + } + + /// + /// Compiles an input assembly and runs the impl generator over it twice, returning the bytes of both forwarders. + /// + /// + /// Both runs use the same input assembly, but each writes to its own output directory, so the two + /// results are only equal if the generator is deterministic (rather than because the file was reused). + /// + /// The bytes of the forwarder produced by each of the two runs. + public static (byte[] First, byte[] Second) RunTwice() + { + string toolPath = GetGeneratorPath(); + string temporaryDirectory = Directory.CreateTempSubdirectory("ImplGeneratorTest_").FullName; + + try + { + string inputAssemblyPath = CompileInputAssembly(temporaryDirectory, DebugInformationFormat.Embedded); + + byte[] Run(string name) + { + string forwarderDirectory = Directory.CreateDirectory(Path.Combine(temporaryDirectory, name)).FullName; + string responseFilePath = Path.Combine(temporaryDirectory, $"{name}.rsp"); + + File.WriteAllText(responseFilePath, $""" + --reference-assembly-paths {inputAssemblyPath} + --output-assembly-path {inputAssemblyPath} + --generated-assembly-directory {forwarderDirectory} + --treat-warnings-as-errors False + """); + + (int exitCode, string output) = RunTool(toolPath, responseFilePath); + + Assert.AreEqual(0, exitCode, output); + + return File.ReadAllBytes(Path.Combine(forwarderDirectory, Path.GetFileName(inputAssemblyPath))); + } + + return (Run("first"), Run("second")); + } + finally + { + TryDeleteDirectory(temporaryDirectory); + } + } + + /// + /// Reads the types of all entries in the debug directory of an assembly, in order. + /// + /// The path of the assembly to read. + /// The debug directory entry types. + public static ImmutableArray GetDebugDirectoryEntryTypes(string assemblyPath) + { + using FileStream stream = File.OpenRead(assemblyPath); + using PEReader peReader = new(stream); + + return [.. peReader.ReadDebugDirectory().Select(static entry => entry.Type)]; + } + + /// + /// Reads the raw bytes of the embedded portable PDB of an assembly. + /// + /// The path of the assembly to read. + /// The embedded portable PDB bytes, or an empty array if the assembly has none. + public static byte[] GetEmbeddedPortablePdbBytes(string assemblyPath) + { + using FileStream stream = File.OpenRead(assemblyPath); + using PEReader peReader = new(stream); + + foreach (DebugDirectoryEntry entry in peReader.ReadDebugDirectory()) + { + if (entry.Type != DebugDirectoryEntryType.EmbeddedPortablePdb) + { + continue; + } + + // The data pointer of a debug directory entry is a file offset, so it can be used to slice + // the whole image. This is the raw (still deflate-compressed) payload of the entry. + return peReader.GetEntireImage().GetReader((int)entry.DataPointer, entry.DataSize).ReadBytes(entry.DataSize); + } + + return []; + } + + /// + /// Checks whether the CodeView entry of an assembly identifies its embedded portable PDB. + /// + /// The path of the assembly to read. + /// Whether the CodeView entry matches the embedded portable PDB. + public static bool IsCodeViewEntryConsistent(string assemblyPath) + { + using FileStream stream = File.OpenRead(assemblyPath); + using PEReader peReader = new(stream); + + ImmutableArray entries = peReader.ReadDebugDirectory(); + + DebugDirectoryEntry codeViewEntry = entries.FirstOrDefault(static entry => entry.Type == DebugDirectoryEntryType.CodeView); + DebugDirectoryEntry embeddedEntry = entries.FirstOrDefault(static entry => entry.Type == DebugDirectoryEntryType.EmbeddedPortablePdb); + + if (codeViewEntry.Type != DebugDirectoryEntryType.CodeView || embeddedEntry.Type != DebugDirectoryEntryType.EmbeddedPortablePdb) + { + return false; + } + + // A CodeView entry only refers to a portable PDB when it carries this exact version pair + if (codeViewEntry.MajorVersion != 0x0100 || codeViewEntry.MinorVersion != 0x504D) + { + return false; + } + + CodeViewDebugDirectoryData codeView = peReader.ReadCodeViewDebugDirectoryData(codeViewEntry); + + using MetadataReaderProvider provider = peReader.ReadEmbeddedPortablePdbDebugDirectoryData(embeddedEntry); + + // The debug metadata GUID is the id the CodeView record has to point at + return provider.GetMetadataReader().DebugMetadataHeader?.Id is { } id + && new Guid(id.AsSpan(0, 16).ToArray()) == codeView.Guid + && codeView.Age == 1; + } + + /// + /// Checks whether the PDB checksum entry of an assembly matches its embedded portable PDB. + /// + /// The path of the assembly to read. + /// Whether the PDB checksum entry matches the embedded portable PDB. + public static bool IsPdbChecksumValid(string assemblyPath) + { + using FileStream stream = File.OpenRead(assemblyPath); + using PEReader peReader = new(stream); + + DebugDirectoryEntry checksumEntry = peReader + .ReadDebugDirectory() + .FirstOrDefault(static entry => entry.Type == DebugDirectoryEntryType.PdbChecksum); + + if (checksumEntry.Type != DebugDirectoryEntryType.PdbChecksum) + { + return false; + } + + PdbChecksumDebugDirectoryData checksum = peReader.ReadPdbChecksumDebugDirectoryData(checksumEntry); + + return checksum.AlgorithmName == "SHA256" && checksum.Checksum.Length == 32; + } + + /// + /// Reads the names of all documents in the embedded portable PDB of an assembly. + /// + /// The path of the assembly to read. + /// The document names. + public static ImmutableArray GetEmbeddedPortablePdbDocumentNames(string assemblyPath) + { + using FileStream stream = File.OpenRead(assemblyPath); + using PEReader peReader = new(stream); + using MetadataReaderProvider? provider = OpenEmbeddedPortablePdb(peReader); + + if (provider is null) + { + return []; + } + + MetadataReader reader = provider.GetMetadataReader(); + + return [.. reader.Documents.Select(handle => reader.GetString(reader.GetDocument(handle).Name))]; + } + + /// + /// Reads the text of the single embedded document in the embedded portable PDB of an assembly. + /// + /// The path of the assembly to read. + /// The embedded document text. + public static string GetEmbeddedPortablePdbDocumentText(string assemblyPath) + { + using FileStream stream = File.OpenRead(assemblyPath); + using PEReader peReader = new(stream); + using MetadataReaderProvider? provider = OpenEmbeddedPortablePdb(peReader); + + Assert.IsNotNull(provider, "The assembly has no embedded portable PDB."); + + MetadataReader reader = provider.GetMetadataReader(); + + foreach (DocumentHandle documentHandle in reader.Documents) + { + foreach (CustomDebugInformationHandle handle in reader.GetCustomDebugInformation(documentHandle)) + { + CustomDebugInformation information = reader.GetCustomDebugInformation(handle); + + if (reader.GetGuid(information.Kind) != EmbeddedSourceKind) + { + continue; + } + + BlobReader blobReader = reader.GetBlobReader(information.Value); + + // A positive format value is the uncompressed size, and marks the content as deflate compressed + int format = blobReader.ReadInt32(); + byte[] content = blobReader.ReadBytes(blobReader.RemainingBytes); + + if (format == 0) + { + return Encoding.UTF8.GetString(content); + } + + using MemoryStream compressed = new(content); + using DeflateStream deflateStream = new(compressed, CompressionMode.Decompress); + using MemoryStream decompressed = new(); + + deflateStream.CopyTo(decompressed); + + Assert.AreEqual(format, (int)decompressed.Length, "The embedded source does not match its declared uncompressed size."); + + return Encoding.UTF8.GetString(decompressed.ToArray()); + } + } + + return ""; + } + + /// + /// Reads the value of a compilation option from the embedded portable PDB of an assembly. + /// + /// The path of the assembly to read. + /// The key of the compilation option to read. + /// The value of the compilation option, or if it is not present. + public static string? GetCompilationOptionValue(string assemblyPath, string key) + { + using FileStream stream = File.OpenRead(assemblyPath); + using PEReader peReader = new(stream); + using MetadataReaderProvider? provider = OpenEmbeddedPortablePdb(peReader); + + if (provider is null) + { + return null; + } + + MetadataReader reader = provider.GetMetadataReader(); + + foreach (CustomDebugInformationHandle handle in reader.GetCustomDebugInformation(EntityHandle.ModuleDefinition)) + { + CustomDebugInformation information = reader.GetCustomDebugInformation(handle); + + if (reader.GetGuid(information.Kind) != CompilationOptionsKind) + { + continue; + } + + // The blob is a sequence of null terminated UTF-8 key/value pairs + string[] parts = Encoding.UTF8.GetString(reader.GetBlobBytes(information.Value)).Split('\0'); + + for (int i = 0; i + 1 < parts.Length; i += 2) + { + if (parts[i] == key) + { + return parts[i + 1]; + } + } + } + + return null; + } + + /// + /// Reads the kinds of all module level custom debug information in the embedded portable PDB of an assembly. + /// + /// The path of the assembly to read. + /// The custom debug information kinds. + public static ImmutableArray GetEmbeddedPortablePdbDebugInformationKinds(string assemblyPath) + { + using FileStream stream = File.OpenRead(assemblyPath); + using PEReader peReader = new(stream); + using MetadataReaderProvider? provider = OpenEmbeddedPortablePdb(peReader); + + if (provider is null) + { + return []; + } + + MetadataReader reader = provider.GetMetadataReader(); + + return [.. reader.CustomDebugInformation.Select(handle => reader.GetGuid(reader.GetCustomDebugInformation(handle).Kind))]; + } + + /// + /// Opens the embedded portable PDB of an assembly, if it has one. + /// + /// The reader for the assembly. + /// The reader provider for the embedded portable PDB, or if there is none. + private static MetadataReaderProvider? OpenEmbeddedPortablePdb(PEReader peReader) + { + DebugDirectoryEntry entry = peReader + .ReadDebugDirectory() + .FirstOrDefault(static entry => entry.Type == DebugDirectoryEntryType.EmbeddedPortablePdb); + + return entry.Type == DebugDirectoryEntryType.EmbeddedPortablePdb + ? peReader.ReadEmbeddedPortablePdbDebugDirectoryData(entry) + : null; + } + + /// + /// Compiles the input assembly the generator runs over, into TestInput.dll in . + /// + /// The directory to emit the assembly into. + /// The debug information to emit ( to emit none). + /// Whether to strong name the input assembly. + /// The full path to the compiled assembly. + private static string CompileInputAssembly(string directory, DebugInformationFormat? debugInformationFormat, bool strongName = false) + { + string outputPath = Path.Combine(directory, "TestInput.dll"); + + CSharpParseOptions parseOptions = new(LanguageVersion.Preview); + + // The source text needs an explicit encoding, or the compiler cannot emit debug information for + // it. The document names are already deterministic ('/_' prefixed), matching what a repository + // build with 'ContinuousIntegrationBuild' produces. + SourceText sourceText = SourceText.From(""" + namespace TestInput; + + public sealed class PublicType + { + public int Method(int value) => value; + } + """, Encoding.UTF8); + + // The target framework attribute lets the generator probe the .NET runtime version. It is added + // in a separate syntax tree so it does not interfere with any 'using' directives in the test + // source (an assembly attribute must precede type declarations but follow 'using' directives). + SourceText assemblyInfoText = SourceText.From( + """[assembly: System.Runtime.Versioning.TargetFramework(".NETCoreApp,Version=v10.0")]""", + Encoding.UTF8); + + SyntaxTree sourceTree = CSharpSyntaxTree.ParseText(sourceText, parseOptions, path: "/_/TestInput.cs"); + SyntaxTree assemblyInfoTree = CSharpSyntaxTree.ParseText(assemblyInfoText, parseOptions, path: "/_/AssemblyInfo.cs"); + + CSharpCompilationOptions compilationOptions = new( + OutputKind.DynamicallyLinkedLibrary, + allowUnsafe: true, + deterministic: true); + + // Only the public key is taken from the input assembly (the generator copies it over so it can + // reserve the right signature space), and the generator signs the forwarder itself afterwards. + // Public signing is therefore enough here, and avoids depending on in-process signing support. + if (strongName) + { + compilationOptions = compilationOptions + .WithCryptoKeyFile(GetStrongNameKeyPath()) + .WithPublicSign(true); + } + + CSharpCompilation compilation = CSharpCompilation.Create( + assemblyName: "TestInput", + syntaxTrees: [sourceTree, assemblyInfoTree], + references: Net100.References.All, + options: compilationOptions); + + using FileStream peStream = File.Create(outputPath); + + EmitResult result; + + if (debugInformationFormat is { } format) + { + using MemoryStream sourceLinkStream = new(Encoding.UTF8.GetBytes(SourceLinkJson)); + + result = compilation.Emit( + peStream: peStream, + options: new EmitOptions(debugInformationFormat: format), + sourceLinkStream: sourceLinkStream); + } + else + { + result = compilation.Emit(peStream, options: new EmitOptions(debugInformationFormat: DebugInformationFormat.Pdb)); + } + + Assert.IsTrue(result.Success, $"Input compilation failed:\n{string.Join("\n", result.Diagnostics)}"); + + return outputPath; + } + + /// + /// Runs dotnet exec <tool> <argument> and captures the exit code and output. + /// + private static (int ExitCode, string Output) RunTool(string toolPath, string argument) + { + ProcessStartInfo startInfo = new("dotnet") + { + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + CreateNoWindow = true, + }; + + startInfo.ArgumentList.Add("exec"); + startInfo.ArgumentList.Add(toolPath); + startInfo.ArgumentList.Add(argument); + + using Process process = Process.Start(startInfo) ?? throw new InvalidOperationException("Failed to start the impl generator process."); + + string standardOutput = process.StandardOutput.ReadToEnd(); + string standardError = process.StandardError.ReadToEnd(); + + process.WaitForExit(); + + return (process.ExitCode, standardOutput + standardError); + } + + /// + /// Checks whether an assembly carries a strong name signature. + /// + /// The path of the assembly to read. + /// Whether the assembly has a populated strong name signature. + /// + /// Only the presence of an actual signature is checked, not the + /// flag: the generator signs the file after writing it, and does not set that flag (which is existing + /// behavior, independent of the debug information the assembly carries). + /// + public static bool IsStrongNamed(string assemblyPath) + { + using FileStream stream = File.OpenRead(assemblyPath); + using PEReader peReader = new(stream); + + if (peReader.PEHeaders.CorHeader?.StrongNameSignatureDirectory is not { Size: > 0 } directory) + { + return false; + } + + int offset = peReader.PEHeaders.GetContainingSectionIndex(directory.RelativeVirtualAddress) is int index and >= 0 + ? directory.RelativeVirtualAddress - peReader.PEHeaders.SectionHeaders[index].VirtualAddress + peReader.PEHeaders.SectionHeaders[index].PointerToRawData + : -1; + + if (offset < 0) + { + return false; + } + + // Space for the signature is reserved even when the file is not signed, so it is only actually + // signed if that space has been filled in + byte[] signature = peReader.GetEntireImage().GetReader(offset, directory.Size).ReadBytes(directory.Size); + + return Array.Exists(signature, static value => value != 0); + } + + /// + /// Resolves the path to the built cswinrtimplgen tool from assembly metadata. + /// + private static string GetGeneratorPath() + { + return GetMetadataPath("ImplGeneratorAssemblyPath", "impl generator"); + } + + /// + /// Resolves the path to the strong name key from assembly metadata. + /// + private static string GetStrongNameKeyPath() + { + return GetMetadataPath("StrongNameKeyPath", "strong name key"); + } + + /// + /// Resolves a path published as assembly metadata by the project file. + /// + /// The metadata key holding the path. + /// A description of the file, used in assertion messages. + /// The resolved full path. + private static string GetMetadataPath(string key, string description) + { + string? path = typeof(ImplGeneratorRunner).Assembly + .GetCustomAttributes() + .FirstOrDefault(attribute => attribute.Key == key)?.Value; + + Assert.IsFalse(string.IsNullOrEmpty(path), $"The '{key}' assembly metadata was not found."); + + string fullPath = Path.GetFullPath(path!); + + Assert.IsTrue(File.Exists(fullPath), $"The {description} was not found at '{fullPath}'."); + + return fullPath; + } + + /// + /// Deletes a directory recursively, ignoring failures (best effort cleanup). + /// + private static void TryDeleteDirectory(string directory) + { + try + { + Directory.Delete(directory, recursive: true); + } + catch (Exception e) when (e is IOException or UnauthorizedAccessException) + { + // Best effort cleanup; a leftover temp directory must not fail the test + } + } +} diff --git a/src/Tests/ImplGeneratorTest/ImplGeneratorTest.csproj b/src/Tests/ImplGeneratorTest/ImplGeneratorTest.csproj new file mode 100644 index 000000000..adbbd0fac --- /dev/null +++ b/src/Tests/ImplGeneratorTest/ImplGeneratorTest.csproj @@ -0,0 +1,77 @@ + + + Exe + net10.0 + x64;x86 + enable + false + + + false + + + + win-x86 + x86 + + + + win-x64 + x64 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/Tests/ImplGeneratorTest/Test_DebugDirectory.cs b/src/Tests/ImplGeneratorTest/Test_DebugDirectory.cs new file mode 100644 index 000000000..d57220439 --- /dev/null +++ b/src/Tests/ImplGeneratorTest/Test_DebugDirectory.cs @@ -0,0 +1,177 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +using System; +using System.Collections.Immutable; +using System.Reflection.PortableExecutable; +using ImplGeneratorTest.Helpers; +using Microsoft.CodeAnalysis.Emit; + +namespace ImplGeneratorTest; + +/// +/// End-to-end tests for the debug information of the forwarder assembly the impl generator produces. +/// +/// +/// The forwarder replaces the compiled output of a reference projection, so it is the assembly that ends up +/// in lib/<tfm> of the resulting NuGet package. It is emitted directly as metadata rather than +/// compiled, so unless its debug information is synthesized it ships with no symbols at all: no Source Link, +/// no compiler flags, and no way to tell it was built deterministically. +/// +[TestClass] +public class Test_DebugDirectory +{ + /// + /// The custom debug information kind for an embedded source document. + /// + private static readonly Guid EmbeddedSourceKind = new("0E8A571B-6926-466E-B4AD-8AB04611F5FE"); + + /// + /// The custom debug information kind for the compilation options (the compiler flags). + /// + private static readonly Guid CompilationOptionsKind = new("B5FEEC05-8CD0-4A83-96DA-466284BB4BD8"); + + /// + /// The custom debug information kind for the compilation metadata references. + /// + private static readonly Guid CompilationMetadataReferencesKind = new("7E4D4708-096E-4C5C-AEDA-CB10BA6A740D"); + + [TestMethod] + public void DebugDirectory_HasTheEntriesOfADeterministicEmbeddedSymbolsBuild() + { + ImplGeneratorRunner.Run(DebugInformationFormat.Embedded, static (_, forwarderPath) => + { + CollectionAssert.AreEqual( + new[] + { + DebugDirectoryEntryType.CodeView, + DebugDirectoryEntryType.PdbChecksum, + DebugDirectoryEntryType.Reproducible, + DebugDirectoryEntryType.EmbeddedPortablePdb + }, + ImplGeneratorRunner.GetDebugDirectoryEntryTypes(forwarderPath), + "Expected the forwarder to have the debug directory of a deterministic build with embedded symbols."); + }); + } + + [TestMethod] + public void DebugDirectory_WithNoInputSymbols_IsStillProduced() + { + // The input assembly is compiled as a reference assembly by the real build, so it never has any + // symbols to carry over. The debug information of the forwarder must not depend on it. + ImplGeneratorRunner.Run(null, static (inputAssemblyPath, forwarderPath) => + { + CollectionAssert.DoesNotContain( + ImplGeneratorRunner.GetDebugDirectoryEntryTypes(inputAssemblyPath), + DebugDirectoryEntryType.EmbeddedPortablePdb, + "Expected the input assembly to have no symbols to carry over."); + + CollectionAssert.Contains( + ImplGeneratorRunner.GetDebugDirectoryEntryTypes(forwarderPath), + DebugDirectoryEntryType.EmbeddedPortablePdb, + "Expected the forwarder to have an embedded portable PDB anyway."); + }); + } + + [TestMethod] + public void DebugDirectory_CodeViewAndChecksumEntries_MatchTheEmbeddedPortablePdb() + { + // A CodeView record that does not identify the PDB next to it would make every consumer that + // resolves symbols through a symbol server (rather than through the embedded copy) fail + ImplGeneratorRunner.Run(DebugInformationFormat.Embedded, static (_, forwarderPath) => + { + Assert.IsTrue( + ImplGeneratorRunner.IsCodeViewEntryConsistent(forwarderPath), + "Expected the CodeView entry of the forwarder to identify its embedded portable PDB."); + + Assert.IsTrue( + ImplGeneratorRunner.IsPdbChecksumValid(forwarderPath), + "Expected the PDB checksum entry of the forwarder to match its embedded portable PDB."); + }); + } + + [TestMethod] + public void EmbeddedPortablePdb_HasAnEmbeddedDeterministicallyNamedDocument() + { + // These are the two things a NuGet package health check looks at to decide whether an assembly has + // Source Link and was built deterministically: every document must be embedded (or source linked), + // and every document name must be path mapped (which is what the '/_' prefix marks). + ImplGeneratorRunner.Run(DebugInformationFormat.Embedded, static (_, forwarderPath) => + { + ImmutableArray documentNames = ImplGeneratorRunner.GetEmbeddedPortablePdbDocumentNames(forwarderPath); + + Assert.AreEqual(1, documentNames.Length, $"Expected exactly one document, but got [{string.Join(", ", documentNames)}]."); + Assert.IsTrue(documentNames[0].StartsWith("/_/", StringComparison.Ordinal), $"Expected a deterministic document name, but got '{documentNames[0]}'."); + + CollectionAssert.Contains( + ImplGeneratorRunner.GetEmbeddedPortablePdbDebugInformationKinds(forwarderPath), + EmbeddedSourceKind, + "Expected the document of the forwarder to be embedded."); + }); + } + + [TestMethod] + public void EmbeddedPortablePdb_HasCompilerFlags() + { + ImplGeneratorRunner.Run(DebugInformationFormat.Embedded, static (_, forwarderPath) => + { + ImmutableArray kinds = ImplGeneratorRunner.GetEmbeddedPortablePdbDebugInformationKinds(forwarderPath); + + CollectionAssert.Contains(kinds, CompilationOptionsKind, "Expected the forwarder to have compilation options information."); + CollectionAssert.Contains(kinds, CompilationMetadataReferencesKind, "Expected the forwarder to have compilation metadata references information."); + + // Tooling ignores the compilation options entirely below version 2, so emitting them without + // this entry (or with a lower value) would be equivalent to emitting nothing at all + Assert.AreEqual( + "2", + ImplGeneratorRunner.GetCompilationOptionValue(forwarderPath, "version"), + "Expected the compilation options of the forwarder to declare a supported version."); + }); + } + + [TestMethod] + public void EmbeddedPortablePdb_EmbeddedDocument_DescribesTheTypeForwards() + { + // The embedded document is what a consumer resolving symbols for this assembly is shown, so it has + // to actually describe the assembly rather than just be a placeholder that satisfies a check + ImplGeneratorRunner.Run(DebugInformationFormat.Embedded, static (_, forwarderPath) => + { + string source = ImplGeneratorRunner.GetEmbeddedPortablePdbDocumentText(forwarderPath); + + StringAssert.Contains(source, ""); + StringAssert.Contains(source, "TypeForwardedTo(typeof(global::TestInput.PublicType))"); + }); + } + + [TestMethod] + public void Forwarder_IsDeterministic() + { + // The 'Reproducible' entry above claims the output is a pure function of its input, so it must be + (byte[] first, byte[] second) = ImplGeneratorRunner.RunTwice(); + + CollectionAssert.AreEqual(first, second, "Expected two runs of the generator to produce the same forwarder."); + } + + [TestMethod] + public void Forwarder_WhenStrongNamed_StillHasSymbols() + { + // The forwarder is signed after being written, so the debug directory has to be laid out without + // disturbing the space reserved for the strong name signature (and vice versa) + ImplGeneratorRunner.Run( + DebugInformationFormat.Embedded, + static (_, forwarderPath) => + { + Assert.IsTrue(ImplGeneratorRunner.IsStrongNamed(forwarderPath), "Expected the forwarder to be signed."); + + CollectionAssert.Contains( + ImplGeneratorRunner.GetDebugDirectoryEntryTypes(forwarderPath), + DebugDirectoryEntryType.EmbeddedPortablePdb, + "Expected the strong named forwarder to still have an embedded portable PDB."); + + Assert.IsTrue( + ImplGeneratorRunner.IsCodeViewEntryConsistent(forwarderPath), + "Expected the CodeView entry of the strong named forwarder to identify its embedded portable PDB."); + }, + strongName: true); + } +} diff --git a/src/Tests/SmokeTests/run-smoke-tests.ps1 b/src/Tests/SmokeTests/run-smoke-tests.ps1 index a58dd0ff2..a40e39da5 100644 --- a/src/Tests/SmokeTests/run-smoke-tests.ps1 +++ b/src/Tests/SmokeTests/run-smoke-tests.ps1 @@ -19,7 +19,7 @@ * Projection: a class library generates a reference projection for a third-party component's '.winmd' (reusing the one emitted by the authoring test), validating the reference projection generator and the forwarder generator, exactly as a NuGet - projection author would. + projection author would. The forwarder is also checked to ship embedded symbols. * WindowsSdkProjection: a class library generates the base Windows SDK reference projection from the 'Microsoft.Windows.SDK.Contracts' '.winmd' files, exactly as the @@ -254,9 +254,37 @@ function Invoke-ReferenceProjectionSmokeTest { throw "The $Name build did not produce the 'ref\$Name.dll' reference assembly." } + # The forwarder is what lands in 'lib/' of a projection package, so it has to ship symbols. It + # is emitted as metadata rather than compiled, so its debug information is synthesized by + # 'cswinrtimplgen'; without that, the whole package reports as having no symbols. + Assert-HasEmbeddedSymbols -Path $forwarder.FullName + Write-Host "Verified the $Name projection produced both a forwarder and a reference assembly." -ForegroundColor DarkGray } +# Verifies that an assembly carries an embedded portable PDB and is marked as reproducible. +function Assert-HasEmbeddedSymbols { + param ([Parameter(Mandatory = $true)] [string] $Path) + + $stream = [IO.File]::OpenRead($Path) + try { + $peReader = [Reflection.PortableExecutable.PEReader]::new($stream) + try { + $entryTypes = $peReader.ReadDebugDirectory() | ForEach-Object { $_.Type } + + foreach ($required in @('EmbeddedPortablePdb', 'Reproducible', 'CodeView', 'PdbChecksum')) { + if ($entryTypes -notcontains $required) { + throw "'$([IO.Path]::GetFileName($Path))' is missing the '$required' debug directory entry (has: $($entryTypes -join ', '))." + } + } + } + finally { $peReader.Dispose() } + } + finally { $stream.Dispose() } + + Write-Host "Verified '$([IO.Path]::GetFileName($Path))' ships embedded symbols." -ForegroundColor DarkGray +} + if ($Test -in @('All', 'Consumption')) { Invoke-ConsumptionSmokeTest } diff --git a/src/WinRT.Impl.Generator/Generation/ImplGenerator.cs b/src/WinRT.Impl.Generator/Generation/ImplGenerator.cs index 331c19a05..b245c4e0d 100644 --- a/src/WinRT.Impl.Generator/Generation/ImplGenerator.cs +++ b/src/WinRT.Impl.Generator/Generation/ImplGenerator.cs @@ -3,6 +3,7 @@ using System; using System.Collections.Frozen; +using System.Collections.Generic; using System.Diagnostics; using System.IO; using System.Reflection; @@ -15,6 +16,7 @@ using AsmResolver; using AsmResolver.DotNet; using AsmResolver.PE; +using AsmResolver.PE.Builder; using AsmResolver.PE.DotNet.StrongName; using ConsoleAppFramework; using WindowsRuntime.Generator; @@ -24,6 +26,7 @@ using WindowsRuntime.Generator.Parsing; using WindowsRuntime.Generator.References; using WindowsRuntime.ImplGenerator.Errors; +using WindowsRuntime.ImplGenerator.Writers; namespace WindowsRuntime.ImplGenerator.Generation; @@ -326,7 +329,14 @@ private static void WriteImplModuleToDisk(ImplGeneratorArgs args, ModuleDefiniti try { - implModule.Write(implAssemblyPath); + // We can't just use 'implModule.Write(path)' here, as that gives us no chance to populate + // the debug directory of the resulting .dll. Go through the PE image and the file builder + // explicitly instead, which is exactly what 'Write' does anyway. + PEImage implImage = implModule.ToPEImage(); + + EmitDebugDirectory(implModule, implImage); + + implImage.ToPEFile(new ManagedPEFileBuilder()).Write(implAssemblyPath); } catch (Exception e) { @@ -334,6 +344,63 @@ private static void WriteImplModuleToDisk(ImplGeneratorArgs args, ModuleDefiniti } } + /// + /// Emits the debug directory (and the portable PDB it embeds) for the impl image. + /// + /// The impl module being generated. + /// The for the impl module being generated. + /// + /// + /// The impl assembly is built from scratch, so it starts with an empty debug directory. That would make + /// it ship with no symbols at all: no Source Link, no compiler flags, and no way for tooling to tell it + /// was built deterministically. It is also the assembly that ends up in lib/<tfm> of the + /// resulting NuGet package, so that gap makes the whole package report as having no symbols. + /// + /// + /// There is no PDB to carry over either: the input assembly is compiled as a reference assembly (see + /// 'Microsoft.Windows.CsWinRT.BeforeMicrosoftNetSdk.targets'), and a reference-only compilation emits no + /// debug information at all. Even if it did, that PDB would describe method bodies the impl assembly does + /// not have. The debug information is therefore synthesized here, describing the impl assembly itself. + /// + /// + private static void EmitDebugDirectory(ModuleDefinition implModule, PEImage implImage) + { + // The document name has to be deterministic (and is, as it only depends on the assembly name). + // The '/_' prefix is the same marker the .NET SDK uses for path mapped, deterministic builds. + string assemblyName = Path.GetFileNameWithoutExtension(implModule.Name!); + string documentName = $"/_/{assemblyName}.TypeForwards.g.cs"; + + PortablePdb pdb = PortablePdbWriter.Write( + documentName: documentName, + documentText: TypeForwardsDocumentWriter.Write(implModule), + references: TypeForwardsDocumentWriter.GetReferences(implModule), + compilationOptions: GetCompilationOptions()); + + DebugDirectoryWriter.Write(implImage, pdb, $"{assemblyName}.pdb"); + } + + /// + /// Gets the compilation options to record in the portable PDB of the impl assembly. + /// + /// The compilation options, as key/value pairs. + /// + /// The version entry is the version of this metadata format (not of any tool), and tooling relies + /// on it to decide whether the remaining information can be trusted. The rest describes how the impl + /// assembly was produced, which is by this generator rather than by a C# compiler. + /// + private static List> GetCompilationOptions() + { + return + [ + new("version", "2"), + new("compiler-version", typeof(ImplGenerator).Assembly.GetName().Version?.ToString() ?? "0.0.0.0"), + new("name", "cswinrtimplgen"), + new("language", "C#"), + new("source-file-count", "1"), + new("output-kind", "DynamicallyLinkedLibrary") + ]; + } + /// /// Signs the impl module on disk, if needed. /// diff --git a/src/WinRT.Impl.Generator/Writers/DebugDirectoryWriter.cs b/src/WinRT.Impl.Generator/Writers/DebugDirectoryWriter.cs new file mode 100644 index 000000000..7de3af07a --- /dev/null +++ b/src/WinRT.Impl.Generator/Writers/DebugDirectoryWriter.cs @@ -0,0 +1,122 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +using System; +using System.IO; +using System.IO.Compression; +using AsmResolver; +using AsmResolver.PE; +using AsmResolver.PE.Debug; + +namespace WindowsRuntime.ImplGenerator.Writers; + +/// +/// Writes the debug directory of a generated forwarder assembly. +/// +/// +/// The entries written here match the ones a deterministic C# compilation with embedded symbols produces, +/// which is what tooling (debuggers, symbol servers, and NuGet package health checks) expects to find on a +/// shipped assembly. +/// +internal static class DebugDirectoryWriter +{ + /// + /// The signature of an embedded portable PDB payload (MPDB). + /// + private const uint EmbeddedPortablePdbSignature = 0x42_44_50_4D; + + /// + /// The debug directory entry type for an embedded portable PDB. + /// + /// has no member for this entry type. + private const DebugDataType EmbeddedPortablePdbType = (DebugDataType)17; + + /// + /// The debug directory entry type for a PDB checksum. + /// + /// has no member for this entry type. + private const DebugDataType PdbChecksumType = (DebugDataType)19; + + /// + /// The name of the hash algorithm used for the PDB checksum, with its null terminator. + /// + private static ReadOnlySpan PdbChecksumAlgorithmName => "SHA256\0"u8; + + /// + /// Writes the debug directory for a forwarder assembly, embedding its portable PDB. + /// + /// The for the forwarder assembly. + /// The portable PDB describing the forwarder assembly. + /// The file name to record for the PDB in the CodeView entry. + public static void Write(PEImage image, PortablePdb pdb, string pdbFileName) + { + // The CodeView entry identifies the PDB. Its version fields are what marks the record as + // referring to a portable PDB, rather than to a Windows PDB. + image.DebugData.Add(new DebugDataEntry(new RsdsDataSegment + { + Guid = pdb.Id, + Age = 1, + Path = pdbFileName + }) + { + MajorVersion = 0x0100, + MinorVersion = 0x504D, + TimeDateStamp = pdb.Stamp + }); + + // The checksum lets consumers verify that a PDB they resolved is the exact one this assembly + // was built with, which matters because the same PDB is also served outside of this assembly. + image.DebugData.Add(new DebugDataEntry(new CustomDebugDataSegment(PdbChecksumType, new DataSegment(BuildPdbChecksum(pdb)))) + { + MajorVersion = 1, + MinorVersion = 0 + }); + + // The forwarder is a pure function of its input assembly, so it is always reproducible + image.DebugData.Add(new DebugDataEntry(new EmptyDebugDataSegment(DebugDataType.Repro))); + + // Embedding the PDB keeps the symbols with the assembly, so they are always available. This + // matches what 'DebugType=embedded' produces, which is how CsWinRT itself is built. + image.DebugData.Add(new DebugDataEntry(new CustomDebugDataSegment(EmbeddedPortablePdbType, new DataSegment(BuildEmbeddedPortablePdb(pdb)))) + { + MajorVersion = 0x0100, + MinorVersion = 0x0100 + }); + } + + /// + /// Builds the payload of a PDB checksum debug directory entry. + /// + /// The portable PDB describing the forwarder assembly. + /// The PDB checksum payload. + private static byte[] BuildPdbChecksum(PortablePdb pdb) + { + using MemoryStream stream = new(); + + stream.Write(PdbChecksumAlgorithmName); + stream.Write(pdb.Checksum); + + return stream.ToArray(); + } + + /// + /// Builds the payload of an embedded portable PDB debug directory entry. + /// + /// The portable PDB describing the forwarder assembly. + /// The embedded portable PDB payload. + private static byte[] BuildEmbeddedPortablePdb(PortablePdb pdb) + { + using MemoryStream stream = new(); + using BinaryWriter writer = new(stream); + + writer.Write(EmbeddedPortablePdbSignature); + writer.Write(pdb.Bytes.Length); + + using (DeflateStream deflateStream = new(stream, CompressionLevel.Optimal, leaveOpen: true)) + { + deflateStream.Write(pdb.Bytes, 0, pdb.Bytes.Length); + } + + return stream.ToArray(); + } +} diff --git a/src/WinRT.Impl.Generator/Writers/PortablePdbWriter.cs b/src/WinRT.Impl.Generator/Writers/PortablePdbWriter.cs new file mode 100644 index 000000000..8fdd0fc96 --- /dev/null +++ b/src/WinRT.Impl.Generator/Writers/PortablePdbWriter.cs @@ -0,0 +1,231 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.IO; +using System.IO.Compression; +using System.Reflection.Metadata; +using System.Reflection.Metadata.Ecma335; +using System.Security.Cryptography; +using System.Text; + +namespace WindowsRuntime.ImplGenerator.Writers; + +/// +/// Writes the portable PDB describing a generated forwarder assembly. +/// +/// +/// +/// The forwarder assembly is not produced by a compiler, so there is no PDB for it to carry over. It is +/// also not produced from any source on disk: it is emitted directly as metadata, from the public API +/// surface of the reference projection it stands in for. Its debug information is therefore synthesized +/// here, describing exactly what the assembly is: a single, embedded, generated document listing the type +/// forwards it contains, plus the compilation information tooling expects to find on a shipped assembly. +/// +/// +/// The document is embedded rather than pointed at through Source Link, for the same reason the .NET SDK +/// embeds untracked sources: it is generated, so it exists in no repository a symbol server could serve it +/// from. Everything written here is derived from the forwarder itself, so the result is deterministic. +/// +/// +internal static class PortablePdbWriter +{ + /// + /// The language id for C#, as defined by the portable PDB specification. + /// + private static readonly Guid CSharpLanguage = new("3F5162F8-07C6-11D3-9053-00C04FA302A1"); + + /// + /// The hash algorithm id for SHA-256, as defined by the portable PDB specification. + /// + private static readonly Guid Sha256HashAlgorithm = new("8829D00F-11B8-4213-878B-770E8597AC16"); + + /// + /// The custom debug information kind for an embedded source document. + /// + private static readonly Guid EmbeddedSourceKind = new("0E8A571B-6926-466E-B4AD-8AB04611F5FE"); + + /// + /// The custom debug information kind for the compilation options. + /// + private static readonly Guid CompilationOptionsKind = new("B5FEEC05-8CD0-4A83-96DA-466284BB4BD8"); + + /// + /// The custom debug information kind for the compilation metadata references. + /// + private static readonly Guid CompilationMetadataReferencesKind = new("7E4D4708-096E-4C5C-AEDA-CB10BA6A740D"); + + /// + /// Writes the portable PDB for a forwarder assembly. + /// + /// The name of the generated document (a deterministic, /_ prefixed path). + /// The text of the generated document, which is embedded in the PDB. + /// The assemblies the forwarder references. + /// The compilation options to record, as key/value pairs. + /// The resulting portable PDB. + public static PortablePdb Write( + string documentName, + string documentText, + IReadOnlyList references, + IReadOnlyList> compilationOptions) + { + // The document is synthetic, so its 'file bytes' are defined here: UTF-8, with no byte order mark. + // The same bytes are both hashed and embedded, so the two are consistent by construction. + byte[] documentBytes = Encoding.UTF8.GetBytes(documentText); + + MetadataBuilder metadata = new(); + + DocumentHandle document = metadata.AddDocument( + name: metadata.GetOrAddDocumentName(documentName), + hashAlgorithm: metadata.GetOrAddGuid(Sha256HashAlgorithm), + hash: metadata.GetOrAddBlob(SHA256.HashData(documentBytes)), + language: metadata.GetOrAddGuid(CSharpLanguage)); + + _ = metadata.AddCustomDebugInformation( + parent: document, + kind: metadata.GetOrAddGuid(EmbeddedSourceKind), + value: metadata.GetOrAddBlob(BuildEmbeddedSource(documentBytes))); + + _ = metadata.AddCustomDebugInformation( + parent: EntityHandle.ModuleDefinition, + kind: metadata.GetOrAddGuid(CompilationOptionsKind), + value: metadata.GetOrAddBlob(BuildCompilationOptions(compilationOptions))); + + _ = metadata.AddCustomDebugInformation( + parent: EntityHandle.ModuleDefinition, + kind: metadata.GetOrAddGuid(CompilationMetadataReferencesKind), + value: metadata.GetOrAddBlob(BuildMetadataReferences(references))); + + byte[]? checksum = null; + + // The PDB id is derived from a hash of the PDB content, which is also the value the 'PdbChecksum' + // debug directory entry carries. Deriving both from the same hash keeps them consistent, and makes + // the whole file a pure function of its contents (so two runs produce byte identical output). + BlobContentId ComputeContentId(IEnumerable blobs) + { + using IncrementalHash incrementalHash = IncrementalHash.CreateHash(HashAlgorithmName.SHA256); + + foreach (Blob blob in blobs) + { + incrementalHash.AppendData(blob.GetBytes()); + } + + checksum = incrementalHash.GetHashAndReset(); + + return BlobContentId.FromHash(checksum); + } + + // The PDB carries no method debug information, so it references no type system rows at all + PortablePdbBuilder pdbBuilder = new( + tablesAndHeaps: metadata, + typeSystemRowCounts: ImmutableArray.Create(new int[MetadataTokens.TableCount]), + entryPoint: default, + idProvider: ComputeContentId); + + BlobBuilder pdbBlob = new(); + + BlobContentId contentId = pdbBuilder.Serialize(pdbBlob); + + return new PortablePdb(pdbBlob.ToArray(), contentId.Guid, contentId.Stamp, checksum!); + } + + /// + /// Builds the blob for an embedded source document. + /// + /// The raw bytes of the document. + /// The embedded source blob. + private static byte[] BuildEmbeddedSource(byte[] documentBytes) + { + using MemoryStream stream = new(); + using BinaryWriter writer = new(stream); + + // A positive format value is the uncompressed size, and marks the content as deflate compressed + writer.Write(documentBytes.Length); + + using (DeflateStream deflateStream = new(stream, CompressionLevel.Optimal, leaveOpen: true)) + { + deflateStream.Write(documentBytes, 0, documentBytes.Length); + } + + return stream.ToArray(); + } + + /// + /// Builds the blob for the compilation options. + /// + /// The compilation options, as key/value pairs. + /// The compilation options blob. + private static byte[] BuildCompilationOptions(IReadOnlyList> compilationOptions) + { + using MemoryStream stream = new(); + + foreach (KeyValuePair option in compilationOptions) + { + WriteNullTerminatedString(stream, option.Key); + WriteNullTerminatedString(stream, option.Value); + } + + return stream.ToArray(); + } + + /// + /// Builds the blob for the compilation metadata references. + /// + /// The assemblies the forwarder references. + /// The compilation metadata references blob. + private static byte[] BuildMetadataReferences(IReadOnlyList references) + { + using MemoryStream stream = new(); + using BinaryWriter writer = new(stream); + + foreach (MetadataReferenceInfo reference in references) + { + WriteNullTerminatedString(stream, reference.FileName); + WriteNullTerminatedString(stream, ""); + + // The low bit marks the reference as an assembly (rather than a module), and the + // second bit would mark it as having its interop types embedded, which never applies + writer.Write((byte)1); + + // The timestamp, image size and MVID identify the exact image a reference resolved to. The + // forwarder is emitted against assembly references that are only resolved when the consuming + // application is built (the projection assemblies do not exist yet), so there is no image to + // describe here. Zero is written rather than an invented value, meaning 'unknown'. + writer.Write(0); + writer.Write(0); + writer.Write(Guid.Empty.ToByteArray()); + } + + return stream.ToArray(); + } + + /// + /// Writes a null terminated UTF-8 string to a stream. + /// + /// The stream to write to. + /// The string to write. + private static void WriteNullTerminatedString(Stream stream, string value) + { + byte[] bytes = Encoding.UTF8.GetBytes(value); + + stream.Write(bytes, 0, bytes.Length); + stream.WriteByte(0); + } +} + +/// +/// A portable PDB produced by . +/// +/// The serialized portable PDB. +/// The id identifying the PDB, carried by the CodeView debug directory entry. +/// The stamp identifying the PDB, carried by the CodeView debug directory entry. +/// The SHA-256 checksum of the PDB content. +internal sealed record PortablePdb(byte[] Bytes, Guid Id, uint Stamp, byte[] Checksum); + +/// +/// An assembly referenced by a generated forwarder assembly. +/// +/// The file name of the referenced assembly. +internal sealed record MetadataReferenceInfo(string FileName); diff --git a/src/WinRT.Impl.Generator/Writers/TypeForwardsDocumentWriter.cs b/src/WinRT.Impl.Generator/Writers/TypeForwardsDocumentWriter.cs new file mode 100644 index 000000000..0e0385f2b --- /dev/null +++ b/src/WinRT.Impl.Generator/Writers/TypeForwardsDocumentWriter.cs @@ -0,0 +1,79 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +using System.Collections.Generic; +using System.Text; +using AsmResolver.DotNet; + +namespace WindowsRuntime.ImplGenerator.Writers; + +/// +/// Writes the generated document describing the type forwards in a forwarder assembly. +/// +/// +/// This document is what the synthesized portable PDB embeds as the source of the forwarder assembly. The +/// forwarder has no source on disk, so this is written to be an accurate, human readable description of +/// everything the assembly contains: the C# that would produce exactly these type forwards. +/// +internal static class TypeForwardsDocumentWriter +{ + /// + /// Writes the document describing the type forwards of a forwarder assembly. + /// + /// The forwarder module the document describes. + /// The text of the generated document. + public static string Write(ModuleDefinition implModule) + { + StringBuilder builder = new(); + + _ = builder.Append( + """ + // + // + // This document is generated by 'cswinrtimplgen', and describes the assembly it is embedded in. + // + // That assembly is a forwarder: it stands in for the reference projection it was built from, and + // contains no executable code of its own. It only redirects the projected Windows Runtime types to + // the projection assembly that CsWinRT generates when the consuming application is built, which is + // what lets an application pick up a new CsWinRT version without every library it uses shipping an + // update as well. + // + // The type forwards below are the entire contents of the assembly. + """.ReplaceLineEndings("\n")); + + // Separate the header from the type forwards with a blank line. This is done explicitly, as a raw + // string literal drops the newline before its closing delimiter (so trailing blank lines there would + // have to be doubled up to survive), and 'AppendLine' would emit the platform newline instead of '\n'. + _ = builder.Append("\n\n"); + + foreach (ExportedType exportedType in implModule.ExportedTypes) + { + _ = builder + .Append("[assembly: global::System.Runtime.CompilerServices.TypeForwardedTo(typeof(global::") + .Append(exportedType.Namespace) + .Append('.') + .Append(exportedType.Name) + .Append("))]") + .Append('\n'); + } + + return builder.ToString(); + } + + /// + /// Gets the names of the assemblies a forwarder module references. + /// + /// The forwarder module to inspect. + /// The referenced assembly file names. + public static IReadOnlyList GetReferences(ModuleDefinition implModule) + { + List references = []; + + foreach (AssemblyReference assemblyReference in implModule.AssemblyReferences) + { + references.Add(new MetadataReferenceInfo($"{assemblyReference.Name}.dll")); + } + + return references; + } +} diff --git a/src/build.cmd b/src/build.cmd index 008d310c1..8c9675390 100644 --- a/src/build.cmd +++ b/src/build.cmd @@ -317,11 +317,21 @@ rem verify that the real package (ref/lib assemblies, generators, and build targ rem for a consuming app, a component author, and a projection author, fully isolated from the rem repo build infrastructure. They run only on x64 (matching the native build tools packaged rem for the host architecture) and can be skipped by setting 'cswinrt_run_smoke_tests=false'. +rem They run on PowerShell 7 ('pwsh'), which is what the CI tasks use as well: the tests inspect +rem the built assemblies with 'System.Reflection.Metadata', which Windows PowerShell lacks. if /I not "%cswinrt_platform%"=="x64" goto :eof if /I "%cswinrt_run_smoke_tests%"=="false" goto :eof +where /q pwsh.exe +if ErrorLevel 1 ( + echo. + echo ERROR: The smoke tests require PowerShell 7 ^('pwsh'^), which was not found on PATH. + echo Install it from https://aka.ms/powershell, or skip the smoke tests with 'cswinrt_run_smoke_tests=false'. + exit /b 1 +) + echo Running smoke tests for %cswinrt_platform% %cswinrt_configuration% -call :exec powershell.exe -NoProfile -ExecutionPolicy Bypass -File "%this_dir%Tests\SmokeTests\run-smoke-tests.ps1" -PackageSource "%cswinrt_bin_dir%" -PackageVersion %cswinrt_version_string% -Configuration %cswinrt_configuration% +call :exec pwsh.exe -NoProfile -ExecutionPolicy Bypass -File "%this_dir%Tests\SmokeTests\run-smoke-tests.ps1" -PackageSource "%cswinrt_bin_dir%" -PackageVersion %cswinrt_version_string% -Configuration %cswinrt_configuration% if ErrorLevel 1 ( echo. echo ERROR: Smoke tests failed diff --git a/src/cswinrt.slnx b/src/cswinrt.slnx index a689fc3a8..e5bde1f68 100644 --- a/src/cswinrt.slnx +++ b/src/cswinrt.slnx @@ -241,6 +241,12 @@ + + + + + +