Repository navigation
Conversation
…e reduction With broadcast=True, every rank filled the points it did not find with -inf and reduced each point with comm.allreduce(value, op=MPI.MAX). That is Python's max on floats, which is order dependent for nan (max(nan, -inf) is nan, max(-inf, nan) is -inf), so a nan value came back as -inf unless rank 0 found the point. Points outside the mesh came back as -inf, and the per-point allreduce made many points slow. Each point is now owned by the lowest rank that finds it (one Allreduce MIN), only the owner contributes its value (one Allreduce SUM), and points that no rank finds are nan. All ranks return the same values, also for discontinuous functions (the owner's value instead of the maximum). The output is sized by the value shape instead of the dofmap block size, which also fixes spaces such as Raviart-Thomas (block size 1, value size 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes
evaluate_functionwithbroadcast=True(the default) when run on more than one rank:-infunless rank 0 found the point, so serial and parallel runs gave different results.-inf.allreduceper point and component.The bug
Each rank set the points it did not find to
-inf, then reduced every point withcomm.allreduce(value, op=MPI.MAX). The lowercaseallreducereduces Pythonfloats with Python'smax, whose result with a NaN depends on the argument order:A minimal example:
[nan, nan, nan, nan][nan, nan, nan, nan]mpirun -n 2[-inf, -inf, nan, nan][nan, nan, nan, nan]mpirun -n 3[-inf, -inf, nan, -inf][nan, nan, nan, nan]We hit this with activation-time fields, where NaN marks points that never activated.
The fix
Allreduce(MIN).Allreduce(SUM)combines them. Any value, NaN included, therefore passes through unchanged.prod(u.ufl_shape)columns, instead of the dofmap block size.Behaviour changes
-inf.float64array. Real functions still returnfloat64.Tests
New in
tests/test_eval.py:Before the fix, the NaN tests fail on 4 ranks, the outside-the-mesh test fails on any number of ranks, and the Raviart–Thomas test fails with a shape mismatch.With the fix, all pass on 1, 2, 3 and 4 ranks.
Verification
Run locally with dolfinx 0.11.0.post0:
python3 -m pytest: 878 passed, 1164 skipped, 8 xfailed.mpiexec -n 4 python3 -m pytest -m "not serial": 880 passed, 1162 skipped, 8 xfailed.pre-commit run --all-files: ruff and ruff-format pass. mypy reports 10 errors insrc/scifem/solvers.py, which this PR does not touch; the same errors appear onmainwith this dolfinx version.🤖 Generated with Claude Code