Generic virtual dispatch hangs after repeated collectible AssemblyLoadContext unloads on .NET 10

[kkpgarcia]

Description

MinimalGvmPayload.zip

A dependency-free C# program eventually hangs when it repeatedly loads a
small assembly into new collectible AssemblyLoadContext instances, invokes a
generic virtual override, unloads the context, and confirms that the context
was collected.

In one process, the same source completes 100 consecutive
load/call/unload/collection cycles on .NET 8.0.29. On .NET 10.0.10, a single
process hangs partway through that sequence. The failing cycle is allocator-
dependent: an earlier run hung on cycle 4, while an instrumented run hung on
cycle 3 after the complete native cache key was reused. These are not 100
separate process executions, and the failure is not intrinsically tied to the
third or fourth cycle.

The hung thread consumes one CPU core. It is not waiting on a managed lock.

Reproduction Steps

The attached reproduction contains two plain C# projects with no packages or
native code:

• MinimalGvmPayload: the collectible assembly containing the generic virtual
  method and override;
• MinimalGvmRunner: loads, invokes, unloads, and verifies collection.

MinimalGvmPayload/Machine.cs:

namespace MinimalGvmPayload;

public class State;
public sealed class MarkerState : State;
public sealed class ExpectedException : Exception;

public abstract class Machine
{
    private readonly Dictionary<Type, State> _states = new();

    public void Change<T>() where T : State
    {
        _ = GetOrCreate<T>();
    }

    private T GetOrCreate<T>() where T : State
    {
        if (!_states.TryGetValue(typeof(T), out State? state))
        {
            state = Construct<T>();
            _states[typeof(T)] = state;
        }

        return (T)state;
    }

    protected virtual T Construct<T>() where T : State
    {
        return Activator.CreateInstance<T>();
    }
}

public sealed class DerivedMachine : Machine
{
    protected override T Construct<T>()
    {
        throw new ExpectedException();
    }
}

Build and run:

dotnet build .\MinimalGvmPayload\MinimalGvmPayload.csproj -f net10.0
dotnet build .\MinimalGvmRunner\MinimalGvmRunner.csproj -f net10.0
dotnet run --project .\MinimalGvmRunner\MinimalGvmRunner.csproj -f net10.0 --no-build -- .\MinimalGvmPayload\bin\Debug\net10.0\MinimalGvmPayload.dll 100

Because the failure is an infinite loop, please run the process with an
external timeout. The exact failing iteration varies with native address
reuse.

Expected behavior

For each consecutive cycle in the same process, the invocation should enter
DerivedMachine.Construct<T>(), throw the expected exception, and allow that
cycle's collectible context to unload and be collected. Reusing native
addresses for later collectible types or methods must not select a target from
an earlier unloaded context.

Actual behavior

After one or more successful unloads, a later generic virtual invocation can
consume one CPU core indefinitely. The call does not enter the override body.

In one instrumented .NET 10.0.10 run, the cache recorded:

object MethodTable: 0x7ffed32dcfd0
class handle:       0x7ffed32dce28
method handle:      0x7ffed32de0a0
target:             0x7ffed32f0150

A later context reused all three key addresses. The virtual-function-pointer
cache returned 0x7ffed32f0150 without entering
VirtualFunctionPointerSlow.

The hung thread's RIP was exactly 0x7ffed32f0150. The instruction bytes
started with ff 25 fa 3f 00 00, jumping through cell 0x7ffed32f4150.
That cell contained 0x7ffed32f0150, so execution repeatedly jumped back to
the same instruction.

Regression?

• .NET 10.0.10, Windows x64: one repeated-reload process hangs after a
  variable number of otherwise successful cycles; observed failures occurred
  on cycles 3 and 4 in different runs.

• .NET 8.0.29, Windows x64: one process completed 100 consecutive
  load/call/unload/collection cycles.

• .NET 10.0.11: not yet tested.

Known Workarounds

Changing the overridable factory from a generic virtual method to a
non-generic virtual method avoids this cache path, but valid generic virtual
code should work across collectible unloads. Keeping contexts alive also
avoids the trigger but defeats the purpose of collectible assembly reload.

There is no supported public API for an application to flush this private,
process-wide runtime cache.

Configuration

• OS: Windows x64
• Runtime: Microsoft.NETCore.App 10.0.10
• Comparison runtime: Microsoft.NETCore.App 8.0.29
• Runtime source inspected: tag v10.0.10

Other information

COMPlus_JitDisasm shows the generic base call using
CORINFO_HELP_VIRTUAL_FUNC_PTR followed by an indirect call through the
returned address.

The observed cache lifecycle matches the following source chain at
v10.0.10:

1. LoaderAllocator unload calls FlushVirtualFunctionPointerCaches in
   src/coreclr/vm/loaderallocator.cpp.
2. FlushGenericCache in src/coreclr/vm/jithelpers.cpp replaces the active
   table with GetSentinelTable().
3. VirtualDispatchHelpers uses the global virtual-function-pointer cache fast
   path before VirtualFunctionPointerSlow.
4. GenericCache creates the sentinel with CreateCacheTable(2). The table
   allocator reserves element zero for metadata, producing an array whose
   actual length is three.
5. The insertion paths recognize the sentinel using table.Length == 2.

The following diagnostic controls changed only post-unload cache state:

• clearing usable sentinel entries while preserving element-zero metadata:
  one process completed 100 consecutive reload cycles;
• replacing the cache with a fresh correctly initialized cache after unload:
  one process completed 100 consecutive reload cycles;
• unchanged .NET 10 probe: reproduced the hang;
• non-generic virtual variant: one process completed 100 consecutive reload
  cycles;
• direct override invocation: one process completed 20 consecutive reload
  cycles.

An earlier diagnostic experiment that cleared the complete sentinel produced
an access violation because it also erased element-zero metadata. That result
is excluded and is not needed to reproduce the reported hang.

The source mismatch and controls suggest that the shared sentinel can accept
entries after a flush, allowing native targets associated with unloaded
loader allocators to be returned after complete key-address reuse. I would
appreciate confirmation from the CoreCLR maintainers that this interpretation
matches the intended GenericCache sentinel contract.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

[EgorBo]

AI root-caused it to #89331 and it was exposed by #106843

GenericCache<TKey,TValue>  (the process-wide GVM/virtual-function-pointer cache) failed to recognize its own flush sentinel.