fix(MIOpen): correct deterministic SetNextValue polarity for grouped Bwd/Wrw CK solvers - #10240
Open
JonathanLichtnerAMD wants to merge 2 commits into
Open
Conversation
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
Pre-commit check failed⛔ pre-commit failed Please run locally:
This repo uses |
…Bwd/Wrw CK solvers miopenFindConvolutionBackwardDataAlgorithm() (and the analogous backward-weights find) hung on MI300X when TF32 + deterministic + exhaustive find coincided with an empty perf-db cache entry. Root cause: PerformanceConfigHipImplicitGemmGroupBwdXdlops::SetNextValue and PerformanceConfigHipImplicitGemmGroupWrwXdlops::SetNextValue inverted the return-value polarity of NextLinear() in their deterministic branch. With exactly one valid kernel this spun forever (the reported hang); with more than one it silently exhausted the search after the first call (under-tuning). Both are fixed with the same direct boundary-check idiom already used by the unaffected fwd and 3D-bwd sibling solvers. Also export SetNextValue for both 2D grouped configs via MIOPEN_INTERNALS_EXPORT (already used by the 3D siblings), so the new regression test can call it directly without a GPU or the CK shared library. Co-Authored-By: Claude <noreply@anthropic.com>
JonathanLichtnerAMD
force-pushed
the
users/jlichtne/ALMIOPEN-2359-fix-tf32-deterministic-hang
branch
from
July 31, 2026 19:38
6de84b9 to
7c5e64e
Compare
|
🎉 All checks passed! This PR is ready for review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA ID: ALMIOPEN-2359
Motivation
miopenFindConvolutionBackwardDataAlgorithm()(and the analogous BackwardWeights find)could hang indefinitely on MI300X whenever three conditions coincided: TF32 data type,
the
MIOPEN_CONVOLUTION_ATTRIB_DETERMINISTICattribute set on the convolution, and anexhaustive find with an empty perf-db cache. Root cause: an inverted return-value
polarity bug in the
is_deterministicbranch ofSetNextValue()forPerformanceConfigHipImplicitGemmGroupBwdXdlopsandPerformanceConfigHipImplicitGemmGroupWrwXdlops.NextLinear()returnstrueon wraparound (search exhausted) andfalseon a normaladvance. The old code did
if(!NextLinear(...)) return false;, i.e. it treated anormal advance (
!NextLinear==true) as "exhausted" and a wraparound(
NextLinear==true, so!NextLinear==false) as "keep going" — exactlybackwards. The practical fallout:
branch never returned
falseand the enumeration spun forever (the observed hang).and returned
false, so the search silently visited zero performance configs(under-tuning: the deterministic path never actually explored the kernel list).
The unaffected sibling solvers,
PerformanceConfigHipImplicitGemmGroupFwdXdlopsandPerformanceConfigHipImplicitGemm3DGroupBwdXdlops, already implement the equivalentdeterministic-mode advance correctly using a direct boundary check instead of
NextLinear()'s return value, which is why only the grouped Bwd/Wrw Xdlops solverswere affected.
Technical Details
projects/miopen/src/solver/conv/conv_hip_implicit_gemm_grouped_bwd_xdlops.cppand.../conv_hip_implicit_gemm_grouped_wrw_xdlops.cpp: replaced theif(!NextLinear(0, valid_kernels.size() - 1, index))idiom in theis_deterministicbranch of
SetNextValue()with the same explicit boundary-check idiom already usedcorrectly by the Fwd and 3D Bwd siblings:
NextLinear'swraparound semantics) and consistent with the rest of the codebase. The
non-deterministic branch (which still uses
NextTwoPower/NextCKSplitkValue) isuntouched.
projects/miopen/src/include/miopen/conv/solvers.hpp: addedMIOPEN_INTERNALS_EXPORTto both affected
SetNextValue()declarations, matching the annotation alreadypresent on the 3D Bwd sibling's
SetNextValue(). This is what lets the new gtestbinary call
SetNextValue()directly from a host-only test without linking againstthe CK shared library or requiring a GPU.
projects/miopen/test/gtest/perf_config_group_deterministic_set_next_value.cpp(new file): host-only regression test with 6 cases covering, for both Bwd and Wrw:
falseimmediately, not loop),before exhausting),
more than one index/split_k combination and terminates within a bounded iteration
count).
The tests populate
valid_kernelsdirectly on the performance config before callingSetNextValue(), bypassing the lazy CK-library-backed candidate lookup, so they runentirely on the host with no GPU.
Risk Assessment
Risk Level: 🟡 Medium
The code change itself is minimal (two near-identical 5-line branch rewrites plus two
export annotations), and the affected code path is gated behind the opt-in
MIOPEN_CONVOLUTION_ATTRIB_DETERMINISTICattribute, so most callers are unaffected.It is rated Medium rather than Low because it touches core convolution solver
performance-config search logic (
conv_hip_implicit_gemm_grouped_{bwd,wrw}_xdlops.cpp)— exactly the kind of file whose subtle polarity/boundary bugs are hard to catch by
inspection (as this bug itself demonstrates) and which is exercised by every grouped
Bwd/Wrw Xdlops find/tune call.
Impacted Components
miopen::solver::conv::PerformanceConfigHipImplicitGemmGroupBwdXdlops::SetNextValue(grouped backward-data CK Xdlops solver)
miopen::solver::conv::PerformanceConfigHipImplicitGemmGroupWrwXdlops::SetNextValue(grouped backward-weights CK Xdlops solver)
projects/miopen/src/include/miopen/conv/solvers.hpp(export visibility only, nobehavioral change)
perf_config_group_deterministic_set_next_value.cppPotential Side Effects
Find/tuning calls for grouped Bwd/Wrw Xdlops convolutions willnow actually enumerate all valid kernel candidates instead of hanging (single
candidate) or silently enumerating zero candidates (multiple candidates). Any
downstream code or test that was inadvertently relying on the previous
zero-candidates-enumerated behavior (e.g. assuming the deterministic path always
falls back to a default/heuristic config) could see a different kernel selected.
IsValidValue()/IsValid(), or toany other solver.
MIOPEN_INTERNALS_EXPORTannotations only affect symbol visibility forinternal builds/tests; they do not change runtime behavior.
Mitigation Steps
(
GroupFwdXdlops,3DGroupBwdXdlops) rather than inventing new logic, minimizingthe chance of a new class of bug.
exact hang and under-tuning scenarios, so any future regression on this branch is
caught in CI without needing GPU hardware or a TF32/MI300X repro environment.
new test file; reverting
conv_hip_implicit_gemm_grouped_{bwd,wrw}_xdlops.cppto theprior
NextLinear-based branch (and dropping the new test) fully restores priorbehavior if needed.
Test Plan
Added
projects/miopen/test/gtest/perf_config_group_deterministic_set_next_value.cpp,a new host-only gtest file with 6 test cases (no GPU or CK shared library required):
CPU_PerfConfigGroupBwdDeterministic_NONE.HangRegressionSingleKernelCPU_PerfConfigGroupBwdDeterministic_NONE.UnderTuningThreeKernelsCPU_PerfConfigGroupWrwDeterministic_NONE.HangRegressionSingleKernelCPU_PerfConfigGroupWrwDeterministic_NONE.UnderTuningThreeKernelsCPU_PerfConfigGroupBwdNonDeterministic_NONE.VisitsMultipleIndicesAndSplitKsCPU_PerfConfigGroupWrwNonDeterministic_NONE.VisitsMultipleIndicesAndSplitKsThe hang-regression cases use a single valid kernel and assert
SetNextValue()returnsfalseon the first call (with a bounded iteration cap of 10 as a safety net against atrue infinite loop during test execution). The under-tuning cases use three valid
kernels and assert the config advances through index 1 and index 2 (with correct
split_k/kernel_id) before exhausting on the fourth call. The non-deterministicsmoke test guards the untouched branch, asserting it still visits more than one index
and more than one split_k value within a 500-iteration cap.
Test Result
All 6 new tests were built and run locally and pass. The fix was additionally verified
to be meaningful (not a no-op) by temporarily reverting the
SetNextValue()change andre-running the suite: with the reverted (buggy) code, the corresponding tests fail fast
via the bounded-iteration-cap assertion (
ASSERT_LT(iterations, 10)/ASSERT_LT(iterations, 500)) rather than hanging the test process, confirming the testsuite correctly detects both the hang and under-tuning regressions. No GPU hardware or
TF32/MI300X environment was needed to build or run these tests, since they call
SetNextValue()directly on a host-constructedProblemDescriptionwith amanually-populated
valid_kernelslist.