[release/10.0] JIT: don't use a jump-threaded block's sharpened predicate for dominator-based inference

[EgorBo]

Backport of #132281 to release/10.0

Customer Impact

• Customer reported
• Found internally

Reported in #130700. Silent bad codegen leading to a NullReferenceException in optimized Release runs only; DOTNET_TieredPGO=0 or [MethodImpl(MethodImplOptions.NoOptimization)] worked around it.

When RBO jump threads through a block, it reroutes some of the block's preds directly to the block's successors. If the block is left with a single (ambiguous) pred, optJumpThreadCore sharpens the block's predicate VN to the value flowing in from that pred. The rerouted preds were classified against the old VN, so the sharpened predicate does not hold on the paths that now bypass the block. Dominator info is not updated as we thread, so the bypassed block still looks like a dominator of its successors, and optRedundantBranch could use its sharpened predicate to fold a branch in a block also reachable via the rerouted edges. In the reported case this removed the null check on an isinst result:

       mov      r14, rbx                       ; r14 = obj (non-null)
       cmp      qword ptr [r14], <MergeHierarchy MT>
       jne      SHORT G_M49973_IG17
       xor      r14, r14                       ; obj is MergeHierarchy => (obj as MergeFile) == null
       mov      dword ptr [rbp+0x20], 0        ; MemberType = MergeFile  <-- wrong arm, null check gone
       mov      rbx, gword ptr [r14+0x08]      ; <-- NullReferenceException

Fix: flag such blocks with BBF_STALE_PREDICATE and skip them in dominator-based inference in this phase.

Regression

• Yes
• No

Long-standing issue in RBO jump threading, not a .NET 10 regression.

Testing

Regression test Runtime_130700 from the original PR is included. The original PR reported no SPMI asm diffs.

Risk

Low. The change only makes RBO decline to infer from a block whose predicate it had already narrowed to a single path, so it can only remove unsound optimizations.

Notes on the backport

Not a clean cherry-pick. main applies the new BBF_STALE_PREDICATE check in two dominator-based inference walks; release/10.0 has only one, since optRedundantDominatingBranch was added to main after the 10.0 branch. That hunk is therefore omitted, and the flag uses MAKE_BBFLAG(37) (the next free bit on this branch) instead of 39. The remaining three hunks — setting the flag in optJumpThreadCore, checking it in optRedundantBranch, and clearing it at the end of optRedundantBranches — are applied verbatim.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

Backport to release/10.0 of a JIT RBO (Redundant Branch Optimizations) correctness fix: avoid using a jump-threaded block’s sharpened predicate VN for dominator-based inference when control flow has been rerouted around that block, preventing unsound branch folding that could eliminate required null checks (reported in #130700).

Changes:

• Introduce BBF_STALE_PREDICATE to mark conditional blocks whose predicate VN has become path-specific after jump threading.
• Skip BBF_STALE_PREDICATE blocks during dominator-based inference in optRedundantBranch, and clear the flag at the end of the RBO phase.
• Add a targeted regression test (Runtime_130700) that exercises Tiered PGO behavior needed to reproduce the miscompilation.

Build/test status: not run as part of this review (code inspection only).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/coreclr/jit/redundantbranchopts.cpp	Marks jump-threaded blocks with BBF_STALE_PREDICATE, skips them for dominator inference in RBO, and clears the flag at phase end.
src/coreclr/jit/block.h	Adds the BBF_STALE_PREDICATE basic block flag (using the next available bit on release/10.0).
src/coreclr/jit/block.cpp	Adds display support for BBF_STALE_PREDICATE in debug flag dumps.
src/tests/JIT/Regression/JitBlue/Runtime_130700/Runtime_130700.csproj	Adds a regression test project with process isolation and Tiered PGO-related environment variables.
src/tests/JIT/Regression/JitBlue/Runtime_130700/Runtime_130700.cs	Adds a regression test reproducing the problematic flow (Tiered PGO + isinst/null-check scenario).

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[EgorBo]

PTAL @AndyAyersMS backport of #132281 to 10.0 (customer-reported issue)