Skip to content

Commit 1be4895

Browse files
committed
GPU Metal: fence only threadgroup memory in the work-group collectives
The Metal work-group scan, reduction and broadcast exchange data through threadgroup memory only, so their barriers need not order device memory. GPUbarrier() fences both, which on Apple GPUs makes each barrier wait for the outstanding device-memory accesses. With threadgroup-only fences the stream-compaction scanStart takes 12% less GPU time (30 kHz Pb-Pb, 1/4 TF, no --debug, 3 interleaved rounds); results unchanged.
1 parent 7f86b5e commit 1be4895

1 file changed

Lines changed: 8 additions & 6 deletions

File tree

‎GPU/Common/GPUCommonAlgorithm.h‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -508,7 +508,9 @@ GPUdi() T warp_reduce_max(T v)
508508
// consecutive threads, of GPUCA_WARP_SIZE each. The SIMD scans take 32-bit
509509
// lanes; the narrower types are counts and are widened. Sums are returned
510510
// promoted, as v + 0 would be, since the callers pass !!pred: a bool would
511-
// truncate them to 0 / 1.
511+
// truncate them to 0 / 1. They only exchange data through smem, so their
512+
// barriers fence threadgroup memory only: a device-memory fence, as
513+
// GPUbarrier() has, costs the stream compaction ~10%.
512514

513515
template <class T>
514516
GPUdi() auto warp_scan_inclusive_add(T v) -> decltype(+v)
@@ -567,18 +569,18 @@ GPUdi() auto work_group_scan_inclusive_add_FUNC(T v, S& smem, int32_t iThread, i
567569
if (iThread % GPUCA_WARP_SIZE == GPUCA_WARP_SIZE - 1 || iThread == nThreads - 1) {
568570
smem.warpPredicateSum[iWarp] = x;
569571
}
570-
GPUbarrier();
572+
threadgroup_barrier(mem_flags::mem_threadgroup);
571573
if (iWarp == 0) {
572574
const int32_t s = simd_prefix_inclusive_sum(iThread < nWarps ? smem.warpPredicateSum[iThread] : 0);
573575
if (iThread < nWarps) {
574576
smem.warpPredicateSum[iThread] = s;
575577
}
576578
}
577-
GPUbarrier();
579+
threadgroup_barrier(mem_flags::mem_threadgroup);
578580
if (iWarp > 0) {
579581
x += smem.warpPredicateSum[iWarp - 1];
580582
}
581-
GPUbarrier();
583+
threadgroup_barrier(mem_flags::mem_threadgroup);
582584
return x;
583585
}
584586

@@ -589,9 +591,9 @@ GPUdi() T work_group_broadcast_FUNC(T v, int32_t i, S& smem, int32_t iThread)
589591
if (iThread == i) {
590592
smem.tmpBroadcast = v;
591593
}
592-
GPUbarrier();
594+
threadgroup_barrier(mem_flags::mem_threadgroup);
593595
T retVal = smem.tmpBroadcast;
594-
GPUbarrier();
596+
threadgroup_barrier(mem_flags::mem_threadgroup);
595597
return retVal;
596598
}
597599

0 commit comments

Comments
 (0)