[Aurora] GPU pipelining failure/performance degradation

[rithwiktom]

When using mpich/opt/develop-git.6037a7a on Aurora, I notice that the following test crashes for large message size (~64MB) when GPU pipelining is turned on. For smaller message sizes, there is a performance regression.
mpich/opt/4.2.3-intel does not seem to have this issue.

This is the performance upto 4MB using the two versions:

message size	GPU PIPLN/ develop-git.6037a7a (MB/s)	GPU PIPLN/4.2.3-intel (MB/s)
1	1	1.61
2	1.59	3.23
4	3.19	6.45
8	6.38	12.96
16	12.78	25.89
32	25.57	51.75
64	50.78	100.14
128	36.33	123.29
256	94.03	121.6
512	98.78	134.72
1024	106.68	143.59
2048	111.51	732.25
4096	113.87	1472.69
8192	109.69	2957.42
16384	110.34	5861.81
32768	107.82	11590.16
65536	21586.53	21417.23
131072	28338.11	34983.17
262144	30891.85	43363.4
524288	32059.07	46077.84
1048576	28726.47	46744.3
2097152	29452.7	47071.55
4194304	29858.49	47247.48

This is the test:

export FI_CXI_RDZV_THRESHOLD=131072
export EnableImplicitScaling=0
export NEOReadDebugKeys=1
export ZE_ENABLE_PCI_ID_DEVICE_ORDER=1
export MPIR_CVAR_GPU_USE_IMMEDIATE_COMMAND_LIST=1

# Enable GPU pipelining
export MPIR_CVAR_CH4_OFI_ENABLE_GPU_PIPELINE=1
export MPIR_CVAR_CH4_OFI_GPU_PIPELINE_THRESHOLD=0
export MPIR_CVAR_CH4_OFI_GPU_PIPELINE_NUM_BUFFERS_PER_CHUNK=4
export MPIR_CVAR_CH4_OFI_GPU_PIPELINE_MAX_NUM_BUFFERS=4
export MPIR_CVAR_CH4_OFI_GPU_PIPELINE_D2H_ENGINE_TYPE=1
export MPIR_CVAR_CH4_OFI_GPU_PIPELINE_H2D_ENGINE_TYPE=1

mpiexec -np 4 -ppn 2  --cpu-bind list:2:15  ~/gpu_wrappers/2-2.sh  $PATH_TO_OSU/pt2pt/osu_mbw_mr -m 1:67108864 -i 100 -x 20 -d ze D D 

The wrapper script used here is

#!/bin/bash
 
export ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE
 
if [ $PALS_LOCAL_RANKID -eq 0 ]
then
    AFFINITY_MASK=0.0
    NIC_NUM=cxi0
elif [ $PALS_LOCAL_RANKID -eq 1 ]
then
    AFFINITY_MASK=1.0
    NIC_NUM=cxi1
fi
 
echo "[I am rank $PALS_RANKID] Localrank=$PALS_LOCAL_RANKID : Affinity mask = $AFFINITY_MASK, PREFERRED_NIC =  $NIC_NUM"
 
export ZE_AFFINITY_MASK=$AFFINITY_MASK
export FI_CXI_DEVICE_NAME=$NIC_NUM
 
# Invoke the main program
$*

[servesh]

@rithwiktom It will be impossible to suggest or configure or verify regressions if we have to use dozens of env variables that are non default. Please open a PR on what is Intel formally supported default options related to Intel GPUs for these knobs.

[raffenet]

We should consider if setting MPIR_CVAR_GPU_USE_IMMEDIATE_COMMAND_LIST=1 by default is the right thing to do rather than require users (or modules) to set it on Aurora and other systems.

[hzhou]

#7516 should fix the crash.

[garzaran]

Did you also check the performance?

[zhenggb72]

We should consider if setting MPIR_CVAR_GPU_USE_IMMEDIATE_COMMAND_LIST=1 by default is the right thing to do rather than require users (or modules) to set it on Aurora and other systems.

It used to the case that using immediate command list causes some workload to crash. It seemed to be L0 issues, so this was left to be off by default. However, with the new UMD, the bug may have been fixed so we should reevaluate it again.

[servesh]

@zhenggb72 If would be good if there are such cases, some of these values aren't set that specific JIRAs are opened and addressed. We won't be able to track regression with non standard options.

[hzhou]

Did you also check the performance?

I got ~ 15GB/sec. Haven't investigated further yet.

[zhenggb72]

Yeah, this was many years ago, and it was some very complicated bug with the workload and it took probably almost a year for L0 team to track down. On the other hand, using regular command list can buffer the command and submit only once which actually may reduce the submission overheads. So there is pros and cons. Note that as L0 keep optimizing immediate cmd list, things may also have changed. That is why I suggest we can re-evaluate the default.

[hzhou]

@rithwiktom The original pipeline algorithm will be replaced in #7529. Could you test and evaluate the performance of PR7529? You need set MPIR_CVAR_CH4_OFI_EAGER_THRESHOLD to enable the pipeline path in the new PR.

[hzhou]

I suspect 05883b6 is the cause for the performance change, at least for the low to medium range. The jump at 32768 is indicative of MPIR_CVAR_GPU_FAST_COPY_MAX_SIZE_D2H added in that commit

[rithwiktom]

Hi @hzhou, I'm leaving Intel. Please reach out to Maria/Gengbin if you need anything.

[hzhou]

Thanks for letting us know. Best wishes!

[garzaran]

@rithwiktom The original pipeline algorithm will be replaced in #7529. Could you test and evaluate the performance of PR7529? You need set MPIR_CVAR_CH4_OFI_EAGER_THRESHOLD to enable the pipeline path in the new PR.

Is this ready for testing?