Use hash-based string table reservations

[awakecoding]

Summary

• Replace red-black-tree string reservation with hash-based deduplication.
• Restore ordinal order once before the existing suffix sort, preserving deterministic output.
• Skip string-table flush work when there are no pending reservations.

Motivation

COFF object emission reserves every symbol name before writing the symbol table. A SortedSet pays tree insertion and comparison costs for every reservation, including duplicates, even though ordering is only needed when the table is flushed.

Retained full-RDM measurements on the matching .NET 10.0.11 compiler showed that hash-based reservation reduced the isolated coff-string-reserve phase from 7.66-8.59 seconds to 4.63 seconds: a 3-4 second, 40-46% targeted CPU reduction. The output object and whole-process managed allocation were effectively unchanged. End-to-end wall time varied too much on the shared machine to establish a whole-build improvement, so this change is presented only as a targeted object-writer optimization.

Implementation

StringTableBuilder now:

1. Deduplicates pending Utf8String values in a HashSet<Utf8String>.
2. Returns immediately when there are no pending values.
3. Copies the unique values into an array when flushing.
4. Sorts that array using Utf8String.CompareTo, which is ordinal byte ordering and matches the previous SortedSet enumeration.
5. Runs the existing reverse multi-key suffix sort and emission unchanged.

Utf8String.Equals, GetHashCode, and CompareTo all operate on the same UTF-8 byte content, so hash deduplication and the former tree deduplication have the same equality semantics. The explicit ordinal sort removes hash enumeration order from the output before suffix sharing assigns offsets.

The builder is shared by the COFF, ELF, Mach-O, and PE export writers. This change intentionally preserves its existing ordering and emission contract for every consumer; the measured workload and new integration coverage target the Windows COFF path.

No experiment gate, profiler, capacity planning, object-writer representation change, or other NativeAOT optimization is included.

Validation

The following commands were run on Windows x64:

.\build.cmd clr+libs+host
.\build.cmd clr.aot+libs -rc Release -lc Release
.\build.cmd clr -rc Release

.\dotnet.cmd test .\src\coreclr\tools\aot\ILCompiler.Compiler.Tests\ILCompiler.Compiler.Tests.csproj -c Release --no-restore
.\src\tests\build.cmd nativeaot Release test nativeaot\SmokeTests\Determinism\Determinism.csproj
.\src\tests\build.cmd nativeaot Release tree nativeaot
.\src\tests\run.cmd runnativeaottests Release

• ILCompiler.Compiler.Tests: 22 passed, 0 failed on the final diff.
• NativeAOT determinism test: passed; parallel and single-threaded objects were identical.
• NativeAOT smoke tests: 28 passed, 0 failed.
• ILCompiler.ReadyToRun.Tests: 85 passed and 5 platform tests skipped after excluding two local PDB tests that fail before object writing because the DIA COM object is unavailable (externalComObject is null).
• Baseline and changed Release NativeAOT toolchains built successfully.

Long build commands used a temporary short subst drive and a sanitized inherited build environment to avoid the Windows vcvars command-line length limit. No source change was needed for that environment issue.

Current-main A/B benchmark

The port was measured at base commit d2ea726dd3d8faad797755c9fbb1ff15226e25f0 with an uncommitted stress probe that calls the actual StringTableBuilder from baseline and changed ILCompiler.Compiler.dll builds.

The workload creates 500,000 deterministic suffix-sharing families: 1.5 million distinct UTF-8 names and 3.5 million reservation calls, including duplicate and non-ASCII names. It emits a 21,484,379-byte string-table blob rather than a giant object. Four runs per side were interleaved in A B B A A B B A order. Values below are medians.

Metric	SortedSet baseline	HashSet + ordinal sort	Change
String reservation	3.253 s	0.515 s	-2.738 s (-84.2%)
Symbol offset path, including flush	1.277 s	1.601 s	+0.324 s (+25.4%)
String-table write	13.9 ms	17.4 ms	+3.4 ms
Measured target wall	4.537 s	2.160 s	-2.377 s (-52.4%)
Measured CPU	4.734 s	2.352 s	-2.383 s (-50.3%)
Managed allocation	294.3 MiB	332.2 MiB	+37.9 MiB (+12.9%)
Peak working set	439.2 MiB	416.4 MiB	-22.8 MiB (-5.2%)
Whole probe process wall	5.245 s	2.854 s	-2.391 s (-45.6%)

Every individual run improved the targeted reservation phase:

• Baseline: 3.180-3.487 seconds
• Changed: 0.504-0.556 seconds

The sort cost moves from each tree insertion to the first symbol-offset lookup, explaining the symbol-path increase. The string-table write difference is only a few milliseconds and is not treated as meaningful.

The isolated probe makes hash bucket allocation visible: it allocates 37.9 MiB more while retaining 22.8 MiB less peak working set. In the retained full-RDM compiler measurement, the allocation difference was only 15.2 MiB out of 115.4 GiB (+0.013%), which is effectively neutral at whole-compiler scale.

Retained RDM evidence

The retained .NET 10.0.11 full-RDM experiment used the same compiler build for controls and the gated variant, changing only string-table-hashset.

Metric	Controls	Hash reservation	Result
COFF string reservation	7.66-8.59 s	4.63 s	-3.03 to -3.96 s (-40% to -46%)
Managed allocation	approximately 115.4 GiB	approximately 115.4 GiB	effectively neutral
Object size	3,744,336,196 bytes	3,744,336,196 bytes	identical
Object SHA-256	5B3E5823E9BE816DACD4D47311449E20F0F8233FC5A853439BC30F1F2A351B3D	same	identical

Controls varied from 605 to 819 seconds end to end. That variance is much larger than this optimization, so no whole-build wall-time claim is made.

Byte identity

• Current-main stress payload, all 8 runs: 21,484,379 bytes, SHA-256 F2531F67CF2F990CC4ED39DC51BF1BA2ACF926195149A49FCCBC9F4AF5921946, identical offsets and checksum.
• Current-main NativeAOT determinism object, baseline and changed: 11,009,568 bytes, SHA-256 809013362C3A5F6E708D2BD354EC38C94E5DEFB11AC92E8D0D0306F482DD8B93.
• Retained full-RDM object: byte-identical as shown above.

Limitations

• The current-main probe intentionally isolates string reservation, offset lookup/flush, and table writing. It is not an end-to-end NativeAOT build benchmark.
• The retained RDM result belongs to the matching .NET 10.0.11 workload and is cited separately; the old response file was not forced through the current .NET 11 compiler.
• Allocation is neutral in the full compiler evidence but increases in the isolated high-cardinality probe because the hash buckets are measured without the rest of ILC.
• Two ReadyToRun PDB tests could not run locally because DIA COM activation returned a null external object; the remaining 90 tests completed with 85 passes and 5 platform skips.
• Performance evidence is for Windows COFF. The shared builder's deterministic byte contract is preserved for its other object formats, but no cross-platform performance claim is made.

Note

This pull request description was generated with GitHub Copilot.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

[Copilot]

🟡 Changes recommended

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates the NativeAOT/shared object-writer StringTableBuilder to use hash-based deduplication for reserved strings, then restores a deterministic ordinal ordering once at flush-time before the existing suffix-sharing sort/emission. It also adds COFF-focused tests to validate deterministic bytes/offsets and key edge cases.

Changes:

• Replace SortedSet<Utf8String> reservations with HashSet<Utf8String> deduplication and add an explicit Array.Sort step before the existing suffix sort.
• Add InternalsVisibleTo so tests can access the required internals for COFF string-table validation.
• Add new COFF string-table tests covering section/symbol naming behaviors, duplicates, suffix sharing, determinism across reservation orders, and multi-batch/repeated write scenarios.

File summaries

File	Description
src/coreclr/tools/Common/Compiler/ObjectWriter/StringTableBuilder.cs	Switch reservation storage to HashSet and add an explicit ordinal pre-sort before suffix sorting.
src/coreclr/tools/aot/ILCompiler.Compiler/ILCompiler.Compiler.csproj	Add InternalsVisibleTo for ILCompiler.Compiler.Tests.
src/coreclr/tools/aot/ILCompiler.Compiler.Tests/ILCompiler.Compiler.Tests.csproj	Include the new CoffStringTableTests.cs in compilation items.
src/coreclr/tools/aot/ILCompiler.Compiler.Tests/CoffStringTableTests.cs	Add targeted tests asserting COFF string-table bytes/offsets and determinism.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🟢 Approval recommended

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 0 new
• Review effort level: Lite

[jkotas]

/azp run runtime-nativeaot-outerloop

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[jkotas]

Thanks!

[Copilot]

🟢 Approval recommended

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 0 new
• Review effort level: Lite

[jkotas]

/azp run runtime-nativeaot-outerloop

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).