Optimizing communications when redistributing models

[JordiManyer]

Current implementation:

• When creating an N-processor sub-communicator, we are always picking N the processors with lowest id (i.e [1,...,N]).
• When redistributing a mesh using P4est, we are partitioning cells using a so-called Z-space-filling curve. This assigns cell ids to processors in a consecutive way using the numbering given by the Z-space-filling curve.

This leads to certain situations with obvious sub-optimal communications. For instance, consider a cartesian mesh distributed between 2 processors that gets redistributed to 4 processors:

The current implementation returns a communication pattern 1 -> [2], 2 -> [3,4] which involves moving 75% of the mesh every time we want to redistribute information. The optimal communication pattern would be 1 -> [3], 2 -> [4], which would only communicate 50% of the mesh every time.

Changing the behavior of the z-space-filling curve is quite involved, so to solve this issue we might want to have a look a how the sub-communicators are created. For instance taking [1,3] as the coarse sub-communicator would potentially yield a better communication pattern for the example above.

To-Do:

• Use profilers to identify if this problem is indeed a bottleneck. Compare performance with topology-optimized communicators.