Repository navigation
Feature/2.0 - #23
Merged
Merged
Feature/2.0#23
Conversation
- GpuDetGraph: whole-graph execution plans — ONNX graph compiled to one recorded command buffer; dispatch grids driven by push-constant shapes, plans keyed per shape. 30+ fused kernels (conv1x1 coopmat GEMM, dwconv, SE block, resize+add, elem broadcast, gelu group, CTC projection split). - GpuSession : IOcrSession — narrow seam (Reshape/InputData/RunInternal/ TryRunUntilCtcProjection); InputData is HOST_VISIBLE mapped memory. - OcrSessionFactory + OcrBackend (Auto/Cpu/Vulkan) + SIMD_OCR_BACKEND env; GPU auto-raises RecBatchLines to 16 and uses width-sorted chunked batching (right padding adds only blank CTC columns). - Two-level CPU fallback: factory probe + per-session _gpuDead. - GpuPoc test modes: --det/--rec/--recreal/--pipe/--graph/--pernode/--tval. B580: det 8.6-14ms vs CPU ~70ms; rec n=16 W~480 ~27ms vs CPU ~90ms; E2E pipeline 15-27% faster than CPU on dataset images; rec vocab-argmax bit-equal vs CPU on random inputs; real-image text identical on all non-garbage lines (fp16 near-tie flips only on noise lines).
- PaddleOcrClassifier.ClassifyBatch: pad line count to a multiple of 8, reshape once, preprocess all crops into one [n,3,80,160] input, single GPU submit; the tiny [inner,2] head matmul + softmax resolve on CPU (HeadFromOps) so the N=2 matmul never needs a GPU emit. - Relax the projection resolver's column gate in both InferenceSession and GpuSession: CanFuseArgMax stays inside MatMul.TryArgMax, so the recognizer's argmax fusion is unchanged while small-column heads (cls) now resolve. - PaddleOcrAll: when the classifier is GPU-capable, run one ClassifyBatch for all lines, then fan out the cheap rotate/width-select tail across workers (PostClassifyRange). - softmax.comp + GpuGraph Softmax emit (defensive; the projection split bypasses the terminal nodes for cls). - PaddleOcrClassifierOptions.Backend. B580: cls stage 16 lines = one ~4-5ms GPU run vs ~96ms CPU fan-out; --pipe img-010 16 lines 72.1ms vs CPU 104.8ms, text output identical modulo two stable near-tie case flips already present pre-change.
- Plan dict key: packed long bitfield -> (N,H,W,NodeLimit,OutTensor) record-struct tuple; the n<<40 packing collided with flag bits for n>=2^15 and was fragile to extend. - GpuSession.OutputShape: cache the resolved int[] per Reshape instead of re-running ResolveShapesFor (~590 clones) on every get.
…sify path - Tests bench: --engine sharp|vulkan|auto|c. sharp pins OcrBackend.Cpu so historical baselines stay comparable; vulkan forces the GPU path on detector+recognizer+classifier options; auto leaves factory probing. - PostClassifyRange: write rotations[i] unconditionally — the array is pool-rented (unzeroed), so non-rotated lines inherited stale values (reported rotations showed recWidths garbage; text was unaffected since the rotate decision reads labels/scores directly).
Two portability bugs made the Vulkan backend produce silently-wrong output on NVIDIA (and would on AMD wave64): - Descriptor-set layouts were created with PUSH_DESCRIPTOR_BIT while the sets are written with vkUpdateDescriptorSets — spec-invalid. Intel tolerates it; NVIDIA drops the writes and every bound buffer reads as zero. Fixed by not setting the flag (sets are not pushed). - The conv1x1_cm* coopmat shaders assume subgroup size 16 and an M=8 A-tile. NVIDIA offers sg32 only and its fp16 coopmat set has no M=8 shape (16x16x16/16x8x16/16x8x8); the driver silently runs sg32 and produces garbage. Added conv1x1_cm_sg32: same 128x128 tile over 8x32-lane subgroups with 16x16x16 shapes, selected when SubgroupMin>16. sg16 path unchanged for Intel/AMD wave32. Also raised the descriptor pool limits (MaxSets 1<<16) — dynamic-shape workloads churn plans and 4096 sets was too tight. Verified on RTX 3080 Ti: node-level scans of tiny rec (batch 1/16) and det match CPU through node 159; same-shape steady state runs correct.
… frees) + CPU-batch regrouping on rec fallback - GpuDetGraph: arena/inF32/outF32 become shared grow-only buffers owned by the graph; growing them invalidates every plan (stale descriptor bindings) and reallocates. Plan cache is now an LRU capped at 16 — variable-shape input (DET resizes per image) no longer leaks arena/device buffers or descriptor sets without bound (medium bench WS peak 24GB -> 1.4GB). - VkDevice: FREE_DESCRIPTOR_SET_BIT on the pool so evicted plans release their sets; add FreeDescriptorSet/FreeCommandBuffer; VkBuffer.Free() (destroy + free memory) — Dispose now actually releases device memory. - IOcrSession.GpuAlive: GpuSession reports false once _gpuDead latches. PaddleOcrRecognizer propagates it to _gpuRecAlive; PaddleOcrAll stops choosing GPU-shaped (fewer, wider) line batches once rec is on the CPU fallback — otherwise one giant unit per image serialized all lines and paid max-width padding on CPU (medium lines_wall 902ms -> 384ms). - GpuSession: diagnostic stderr line when a GPU run falls back (once per session); CpuInput() helper transposes caller-written NCHW to the CPU interpreter's expected physical layout when the model input is NHWC-flagged. medium 4w bench on B580: 458.5ms median vs 557.1 CPU (was 968), CER 0.15% vs 0.14% CPU, exact_lines 1001 vs 1004, WS peak 1422MB bounded.
Pool: keep FREE_DESCRIPTOR_SET_BIT (plan eviction) on top of the raised limits; set-layout Flags=0 from their spec fix.
…ow net-win vs CPU
…14 fp16 boundary analysis 3080Ti reviewer asked whether the ~32-shape corruption was fixed or merely masked by LRU=16. Re-ran the full 100-image bench (every det input a distinct shape) with the cap at 64: zero corruption, one residual line-count diff (img-014). Root cause confirmed as resource exhaustion/stale bindings — killed by real frees + InvalidateAllPlans, not hidden by the cap. Keeping 64 since plans share the grow-only arena (WS peak 981MB bounded) and it reduces rebuild churn (12.0->14.0 img/s on the all-distinct-shape set).
…a limitation) Audit of the backend's net10-only APIs: LibraryImport x59, NativeLibrary resolver, System.Half x~40, BitConverter.SingleToUInt32Bits — all portable but each adds a second interop/loader implementation to maintain. ns2.0 stays best-effort CPU-only.
…n GPU Per-shape plans owned descriptor sets/cmd buffers bound to one arena handle, so any buffer growth invalidated all of them and multi-worker runs serialized on a device-wide lock. Split the backend: - GpuGraphModel (per CompiledModel x device, ref-counted): pipelines, fp16 weights and a GpuSchedule cache that is pure metadata (role-based bindings, push constants, grids, arena size). Nothing per-shape to evict or invalidate. - GpuDetGraph runner (per session): grow-only arena/in/out/partials, own command pool/fence; re-records each run via push descriptors (resettable pool fallback). Only vkQueueSubmit is serialized. - Prefer a compute-only queue family (Arc CCS / async compute) and ask for HIGH global priority, degrading gracefully. SVTR (small/medium rec) no longer falls back to CPU: batched maxpool4 and channel concat, fused layernorm and multi-head attention kernels, direct dot MatMul for narrow token GEMMs (swish/residual epilogue), batched direct conv. Global attention is padding-sensitive, so rec lines are grouped by exact target width and all units of an image go out in one submission (IBatchedCtcSession); the vocab projection + ArgMax stay on CPU as before.
… GPU path Profiling after the SVTR emit showed the GPU path bound by serial CPU work and latency-bound small-M kernels, not by device throughput: - REC units now go out in 2-3 submissions; ArgMax + decode of early units runs while later units execute (CTC contract unchanged: the vocab projection stays on CPU). Level interleaving no longer requires identical dispatch sequences, so medium REC units stop running serially. - mmdot_sk: split-K dot kernel for narrow-M token GEMMs and small SE / pointwise convs (16 K-lanes + shared reduction); the plain dot kernel left a few workgroups spinning through K. SIMD_OCR_NOSK bisects. - ArgMax (AVX2 packed): column-block-major walk reuses an L2-resident weight block across all row quads of a worker; units with the same kernel class are merged into one call. Per-row arithmetic, order and tie-breaks are unchanged (CPU outputs bit-identical on all 100 images). - DB postprocess: flood fills scan runs over combined state maps with vectorised searches, hull input is reduced to per-row extremes (exact for integral points). Verified box-for-box against the previous implementation on 400 det maps. - CLS/REC preprocessing of the batched GPU path runs line-parallel. - Runner reuses its readback array and grows host-visible buffers in power-of-two steps (each remap cost ~20 ms). SIMD_OCR_VK_NOPUSH forces the descriptor-pool fallback for testing. B580, 100 images, 4 workers, same-session A/B vs previous commit: tiny 36.3 -> 27.8 ms, small 53.6 -> 38.8 ms, medium 101.2 -> 61.3 ms; accuracy unchanged (tiny +1 detected line from fp16 det noise).
Co-authored-by: Cursor <cursoragent@cursor.com>
…M kxk conv First run on an RTX 3080 Ti: medium DET 960x960 took 55.5 ms of pure GPU time, 46 ms of it in conv1x1_cm_sg32 (~3.5 TFLOPS on 57600x256x128), and medium REC 8x480 36.6 ms (34 ms in the same kernel). The sg32 variant issued coopMatLoad straight from global memory, every subgroup re-fetching its own strip, and the driver did not unroll the loops over the coopmat arrays. conv1x1_cm_sg32.comp rewrite (sg32 only; sg16 shaders untouched): - A/B k-slices (BK=32) staged through double-buffered shared memory with one 16 B load per thread-quad (8 B fallback when K % 8 != 0), padded 80 B rows (conflict-free coopMatLoad), register prefetch of the next slice. Loads come from clamped in-bounds addresses and the M/N/K tails are zeroed on the shared-memory store, so no select waits on a load before the MMAs and X/W no longer need tile padding. - [[unroll]] on every coopmat array loop. - Epilogue: row-major stage, each lane owns 8 consecutive columns; bias, residual and both outputs move as 16 B vectors when the row stride is 8-aligned (per-element math and order unchanged). It was the single biggest cost for K <= 512 GEMMs. - 128x64 and 128x32 tile variants (CmTile picks them by cout, like sg16). - CONVK variant: implicit-GEMM dense kxk conv, the tap-major A quad is gathered from the NHWC input (cin % 8 == 0), batched, with residual and concat-slice output. Replaces convk_dot and im2col+GEMM on sg32; the im2col scratch is now sized by the convs that actually emit im2col. - Arena slabs are 16 B aligned (were 8 B) for the uvec4 loads. Pointwise convs with K <= 128 now take the coopmat path on sg32 when the output leaves split-K range (M*cout > 2^18) and the input is not SE prescaled / addps-absorbed (those need the dot kernel's fused read). Swept 1 .. 2^18: this bound is best or tied on tiny/small/medium DET and REC; below it mmdot_sk still wins. RTX 3080 Ti, pure GPU (SIMD_OCR_GPU_PROF min): medium DET 960x960 55.5 -> 10.9 ms, medium REC 8x480 36.6 -> 7.7 ms (largest GEMMs now 25-28 TFLOPS). End to end, 100 images, 4 workers, alternating A/B vs d6c260c in the same session (median ms/img, two rounds): tiny 19.9 / 17.7 -> 22.8 / 23.7 (3 more rounds: 20.1/20.4/20.8 -> 22.0/16.0/19.6; tiny is CPU-bound, noise) small 33.7 / 34.2 -> 28.2 / 27.6 medium 125.1 / 131.6 -> 42.5 / 39.1 Accuracy (1036 lines): tiny 740, CER 2.37% (base 740 / 2.36%); small 950 / 0.40%; medium 1004-1005 / 0.14%. All within fp16 noise of CPU (742 / 950 / 1004). NVIDIA validation: push vs SIMD_OCR_VK_NOPUSH, compute vs gfx queue: 100/100 images identical on all models; SIMD_OCR_NOSK changes one tiny image (split-K order) and is slower; GpuPoc --conc 4 and 8 threads: 0 mismatches. CPU path untouched (sharp hashes 100/100 identical).
… -> 108 MB) Every intermediate tensor owned its own arena slab for the whole schedule, so one medium DET session at 960x960 needed a 1233 MB arena (980 MB after the im2col scratch became use-sized). GpuPoc --conc with 8 threads on the 12 GB 3080 Ti hit ErrorOutOfDeviceMemory and sessions fell back to CPU for good: 11.0 s wall vs 2.0 s at 4 threads (d6c260c: 76.9 s vs 7.9 s). The texts still matched, so the conc check passed. After emit, PackSlabs derives each slab's lifetime from the recs that bind it (first..last rec index) and re-places slabs best-fit so disjoint lifetimes share memory. Every rec is followed by a compute barrier, so a later writer cannot race an earlier reader; a slab read and written by the same rec stays exclusive. Bindings, off[] and the im2col scratch are rewritten; any arena binding that is not a slab start or the scratch aborts the pass and keeps the old layout. The schedule stays pure metadata. Only slabs still live at the end keep their values for the debug readers (nodeLimit-truncated runs are unaffected). Gated to sg32 because the sg16 (Arc) kernel set has not been run against packed slabs; the pass itself is device-independent. RTX 3080 Ti: DET arena at 960x960 tiny 240 -> 34 MB, small 390 -> 46 MB, medium 980 -> 108 MB. GpuPoc --conc medium: 8 threads 11.0 s -> 1.57 s (peak VRAM 11.9 GB -> 4.8 GB, no fallback), 4 threads 2.0 -> 1.7 s; 0 mismatches. All models 100/100 images identical (hash, boxes, texts) to the unpacked layout; single-stream 4-worker latency unchanged within noise.
After the GEMM rewrite the two largest DET dispatches on the 3080 Ti were SE-prescaled 1x1 convs still on the dot kernel (medium 960x960: 57600x128x256 0.82 ms, 14400x256x512 0.76 ms, ~5 TFLOPS). The prescale fusion admits only cin <= 256 convs, which always took the dot path. PRESCALE variant of conv1x1_cm_sg32 (3 tiles): the A quad is multiplied by the per-image SE vector (binding 6, pc mImg for batched REC) on its way into shared memory. The product is rounded to fp16 before the MMA, i.e. the numerics of the unfused Mul. It is a separate compile: with the scale registers in the plain GEMM every REC GEMM lost ~10% (small REC 8x480 2.71 -> 2.85 ms). Prescaled convs join the M*cout > 2^18 coopmat routing on sg32; addps-absorbed inputs still stay on the dot kernel. RTX 3080 Ti pure GPU vs 948b05c (min over 10 runs): DET 960x960 small 3.57 -> 3.33 ms, medium ~10.0 -> 8.6 ms; tiny DET and all REC unchanged. End to end (100 images, 4 workers, alternating, median ms/img): medium 42.5 / 41.6 -> 41.1 / 41.5, small and tiny within noise. Accuracy: tiny 741 / 2.38% (was 740 / 2.37%), small 950 / 0.40%, medium 1006 / 0.14% (was 1004).
The medium DET 9x9 depthwise convs ran on conv_dw4, which re-reads 81 input vec4 per output from global memory: 256ch 57600 px took 0.72 ms, the largest single dispatch left after the GEMM work. conv_dw4t already stages the input halo in shared memory but was sized for k <= 5. The tile/weight arrays now cover k <= 9, s <= 2 (40x40 halo, 81 taps; 12.8 KB instead of 10.4 KB). Tap order and zero fill are unchanged, so the k <= 5 math is identical. sg32 routes k <= 9 to it; the emit gate and the addps-absorb gate share one predicate so they cannot drift. sg16 keeps k <= 5 (not re-measured on Arc). RTX 3080 Ti medium DET 960x960: n172 0.72 -> 0.45 ms, n170 0.19 -> 0.13 ms, total 9.75 -> 9.13 ms; small DET 3.34 -> 3.28 ms; tiny and REC unchanged. 100/100 images identical to 46fda98 on small and medium; end-to-end medium 43.0 / 40.4 -> 39.7 / 41.3 ms (noise level).
…26 -> 38 ms) Same-session sharp/vulkan 4w on HOME-MAIN (5800X + RTX 3080 Ti) at a06a791, plus the 3-round alternating A/B against d6c260c, the pure-GPU DET/REC times, and the arena/8-thread concurrency result. Notes where the remaining time goes (CPU CTC ArgMax for small/medium, CPU pre/post for tiny, JIT warm-up in the means).
…6 MMA are guaranteed The sg32 GEMM / implicit-GEMM kernels hard-wire a 32-lane subgroup (lane mapping, per-subgroup stage slices, NTHR = SGX*SGY*32) and the 16x16x16 fp16 -> fp32 subgroup MMA. NewPipeline only chains the requiredSubgroupSize=32 request when VK_EXT_subgroup_size_control is present, and never looked at requiredSubgroupSizeStages; on a part whose compute default is wave64 (AMD RDNA) a skipped request, or a device without that coopmat shape, would have produced wrong results silently. VkDevice now records whether requiredSubgroupSizeStages covers compute, whether the 16x16x16 fp16/fp32 subgroup shape is offered, and maxComputeSharedMemorySize (read from the native limits offset; the managed property blob starts 4 bytes before the 8-byte-aligned limits). GpuGraphModel throws NotSupportedException before creating any sg32 pipe when it cannot pin 32 lanes (sgc for compute, or a fixed 32-32 range) or lacks the MMA shape, so the session falls back to CPU before its first run instead of mid-run. HYMT_VK_VERBOSE prints the new fields. Radeon 880M (RDNA 3.5, Windows driver): sgRange 32-64, default 64, requiredSubgroupSizeStages 0xf0 (compute included), 16x16x16 fp16/fp32 subgroup coopmat present; a probe shader built through the same NewPipeline path reports gl_SubgroupSize 32 / 8 subgroups per 256-thread workgroup with requiredSubgroupSize=32 and 64 / 4 without it, so the check passes. NVIDIA reports a fixed 32-32 range and lists 16x16x16 fp16/fp32, so the 3080 Ti should pass too (not re-run there). The sg16 (Arc) path is untouched.
…880M medium 188 -> 133 ms) First run of the sg32 path on AMD (Radeon 880M, RDNA 3.5 iGPU, 12 CU, Windows driver). Results were correct, but the NVIDIA-tuned kernels left most of the chip idle: register-only WMMA reaches 16.7 TFLOPS there, while the REC/DET GEMMs ran at 2.3-3.4 TFLOPS, and on REC the pre-rewrite kernel (d6c260c, direct global coopMatLoad) was faster than the current one (medium REC 8x480 47.8 vs 51.9 ms pure GPU). The staged kernels also take 48 KB of shared memory (34 KB for n64) while the driver advertises maxComputeSharedMemorySize = 32 KB; they only work because RDNA has 64 KB per workgroup. Synthetic sweeps (GpuPoc raw dispatch timing, 13 REC/DET shapes) on the 880M: the 16 B vectorised epilogue is the single biggest cost there (same kernel with a scalar epilogue: 17.0 -> 13.5 ms summed); grouping the tile raster by 4 N-columns so workgroups in flight share A strips in the 2 MB L2 is worth another ~20%; a single staging buffer is slower with the vec8 epilogue but faster with the scalar one; and without the staging at all (direct fragment loads from global memory) the plain GEMM sum drops to 8.4 ms. 64x64 subgroup strips (4 or 8 subgroups), BK=64 and raster groups of 2/8/16 were slower. New variants from the same source, selected when the part runs wave64 (SubgroupMax >= 64, i.e. RDNA) or caps shared memory below 48 KB: - sg32l (LITE, all nine conv1x1 / prescale / convk variants): one staging buffer (28 KB, inside the advertised limit), scalar epilogue, 4-column raster groups. - sg32d (DIRECT, plain conv1x1 and MatMul GEMMs with M >= 16 and K % 16 == 0): LITE without staging (8 KB). A fragments past mDim are shifted back to end at mDim and their duplicate rows are not stored, so no read leaves the tensor; W rows are already padded to the tile. K is still accumulated 16 at a time in order and the epilogue math is unchanged, so outputs are bit-identical to the staged kernels. NVIDIA (sg 32-32, 48 KB) and Arc (sg16) keep their pipelines; the 21 existing .spv files rebuild byte-identical. +12 .spv, 270 KB. Radeon 880M, pure GPU (min over 10 runs, ms, 713fc03 -> this): DET medium 960x960 64.9 -> 44.1, 640x960 43.4 -> 30.7 REC medium 8x480 51.9 -> 32.7, 1x320 5.24 -> 2.55 DET small 960x960 14.2 -> 11.2, REC small 8x480 12.6 -> 8.7 DET tiny 960x960 7.7 -> 7.0, REC tiny 8x480 3.36 -> 2.48 End to end (100 images, 4 workers, 3 alternating rounds, median ms/img): medium 189.0 / 187.6 / 186.2 -> 132.9 / 133.4 / 133.5 small 60.0 / 60.3 / 60.4 -> 51.0 / 50.9 / 50.1 tiny 31.4 / 32.9 / 33.4 -> 29.9 / 31.9 / 33.2 Accuracy unchanged (tiny 742 / 2.37%, small 951 / 0.39%, medium 1003 / 0.15%); 100/100 images identical (hash, boxes, texts) to 713fc03 on all three models. NOPUSH and the graphics queue are identical to the default run, NOSK differs only in tiny's split-K sums (741 / 2.39%); GpuPoc --conc 4 and 8 threads: 0 mismatches, no fallback (medium 8 threads 11.3 -> 7.7 s wall, peak device memory 2.1 GB).
…2l parts After the GEMM rework, conv_dw4t was the second-largest kernel on the 880M (REC medium 8x480 5.0 of 32.7 ms, DET medium 7.1 of 44.1 ms). A 512ch 3x3 layer over 5760 REC pixels moved ~12 MB in 0.45 ms (~26 GB/s, about a quarter of the iGPU's bandwidth): each workgroup of the tiled kernel owns one channel quad, so on NHWC its halo loads are 8 bytes at a C*2-byte stride, and REC maps (H = 6) leave 10 of the 16 tile rows idle. The flat conv_dw4 reads consecutive channel quads per lane and simply re-reads the 3x3 taps through the caches. Routing k<=3 to conv_dw4 alone cut REC depthwise to a third but lost the addps absorb on DET (medium +0.7 ms of separate addps dispatches). So conv_dw4a: the same source with ABSORB — flags bit8 reads the input as fp16(x*se + ra) per tap (bindings 4/5, same arithmetic as conv_dw4t and the addps kernel). On sg32l parts DwFlatA (k,s within 3/2) picks it; the emit path and the addps-absorb gate both go through DwFlatA/DwTiled. k9 keeps the tiled kernel (flat was slower: DET medium 6.75 vs 5.77 ms). conv_dw4.spv and all other shaders rebuild byte-identical; NVIDIA/Arc routing is unchanged. Radeon 880M pure GPU (min over 10 runs, ms, 46de4d1 -> this): REC medium 8x480 32.7 -> 29.8, 1x320 2.55 -> 2.32 REC small 8x480 8.72 -> 7.98, REC tiny 8x480 2.48 -> 2.15 DET medium 960x960 44.1 -> 42.2, 640x960 30.7 -> 29.8 DET small 11.2 -> 10.9, DET tiny 7.00 -> 6.83 End to end (100 images, 4 workers, 3 alternating rounds, median ms/img): medium 137.0 / 131.8 / 130.5 -> 125.6 / 125.7 / 126.2 small 51.8 / 51.9 / 52.2 -> 49.2 / 49.7 / 49.4 tiny 28.4 / 30.4 / 31.1 -> 31.0 / 29.6 / 29.9 (noise) 100/100 images identical (hash, boxes, texts) to 46de4d1 on all three models; accuracy unchanged.
REC SE squeezes (hw = 6x120 per line) are under the 4096-pixel gate of the two-phase reduce and ran on reduce_hw: one workgroup per channel, each lane reading 2 bytes at a C*2-byte stride. On the 880M a 512-channel squeeze over 8 lines took 0.46 ms (~13 GB/s), 1.86 ms per REC medium 8x480 in total. reduce_hw4 lays lanes over consecutive channel quads. On sg32l parts the ReduceMean emit takes the two-phase path whenever its channel layout fits (C % 4 == 0, 256 % (C/4) == 0), regardless of hw; other devices keep the 4096 gate. Radeon 880M pure GPU (min over 10 runs, ms, a1dad0a -> this): REC medium 8x480 reduce 1.86 -> 0.34, total 29.8 -> 28.2; 1x320 2.29 -> 2.14; DET medium reduce 0.40 -> 0.05 (total within noise); small/tiny unchanged (their squeezes miss the channel-layout gate or were already two-phase). End to end (100 images, 4 workers, 3 alternating rounds, median ms/img): medium 127.6 / 125.0 / 126.0 -> 126.9 / 120.4 / 122.0; small and tiny within noise. The summation order changed but all three models are 100/100 images identical (hash, boxes, texts) to a1dad0a; accuracy unchanged.
…^16 on sg32l The dot/split-K kernel keeps pointwise convs with cin <= 128 until M*cout > 2^18, a crossover measured on discrete GPUs. With the direct-load GEMM on the 880M it comes earlier: e.g. tiny DET 3600x128x64 took ~0.10 ms per conv on conv1x1dsk and a few hundredths on conv1x1_cm_sg32d. Sweeping the gate (2^14..2^22, pure GPU and best wall over the REC/DET cases) put the optimum at 2^16; the split-K tile gate (1024..65536) showed nothing outside noise and stays. Radeon 880M pure GPU (min over 10 runs, ms, 2^18 -> 2^16): DET tiny 960x960 6.70 -> 6.21, DET medium 42.4 -> 41.2, REC medium 1x320 2.31 -> 2.11, others within noise. End to end (100 images, 4 workers, 3 alternating rounds, median ms/img): small 50.3 / 50.3 / 49.4 -> 48.3 / 48.8 / 48.6 tiny 30.9 / 30.6 / 29.9 -> 29.1 / 29.2 / 30.2 medium 120.1 / 122.6 / 120.6 -> 119.7 / 121.6 / 119.9 Accuracy: small 951 / 0.39% and medium 1003 / 0.15% unchanged (100/100 images identical); tiny 742 / 2.37% -> 740 / 2.39% (8 images differ, fp16 summation-order noise, incl. the known img-014 box). Other devices keep 2^18.
The old gemm_nc tile ran at ~270 GFLOPS on UHD 770 (a third of fp32 peak): the driver picked SIMD16, and every FMA pair paid for f16->f32 lane extraction from a bank-conflicted shared tile. Now each 64-thread group owns a 64x64 block with 8x8 fp32 accumulators per thread; A/B are converted once and stored k-major, and the next tile's loads are issued before the FMAs. Pinning 8 lanes keeps the accumulators in registers (SIMD16 spills, ~4x slower). 5760x1024x512 goes from 22.4 ms to 11.4-13.4 ms depending on the GPU clock state. IGC spills the main loop when 64 unrolled gelu/sigmoid tails sit next to it, and again when an unrolled tail and a staged tail share one binary. So the tail is split: gemm_nc stages one accumulator row at a time through the freed shared memory and runs a non-unrolled activation loop (all acts, dual output); gemm_nc_s (-DSIMPLE) is the fully unrolled no-act / relu / hardswish single-output tail, picked on the host from the flags. Same-period pure GPU (gprof min, ms), old -> new: medium DET 960 750 -> 564, medium REC 8x480 549 -> 334, small REC 8x480 115 -> 83. Still behind sharp, so RouteNocm stays false. Other .spv files rebuild byte-identical.
conv1x1_dot tops out near 150 GFLOPS on large M (57600x128x256 took 23.5 ms), while the retuned gemm_nc does ~500. Like the sg32 paths, the no-coopmat tier now sends K<=128 pointwise convs with M*cout > 2^16 to the GEMM (not SE-prescaled ones, which it has no variant for, nor addps-absorbed inputs), and does the same for Kp<=128 after im2col. 2^12..2^20 all land within noise on medium; tiny is best at 2^14..2^16. Same-period pure GPU (ms): medium DET 960 564 -> 537, small DET 107 -> 102, tiny DET 44.0 -> 41.5, tiny REC 19.9 -> 18.4. UHD 770 end to end, three alternating rounds (median ms): tiny 104-108 -> 89-99, small 318-365 -> 248-275, medium 1619-1630 -> 1054-1091; sharp on the clean round 68 / 147 / 563. Still slower on all three tiers, so the route stays off. docs/vulkan-uhd770.md rewritten with the numbers, the failed variants, and a correction: CPU parity is 91/90/98 per image (before and after), not 100/100.
On parts without a 16x16x16 fp16 cooperative matrix the gemm_nc tier is correct but trails the CPU (UHD 770: 1.3x / 1.7x / 1.9x), and until now nothing could turn it on: RouteNocm was a hard-coded false, so even an explicit OcrBackend.Vulkan silently ran on the CPU. GpuBackend.UsesGpu now decides: with Coop16x16x16 nothing changes; without it only OcrBackend.Vulkan (or Auto with SIMD_OCR_BACKEND=vulkan) gets a GPU session and Auto stays on the CPU. IsGpuBackend shares the predicate, so Auto on such a device also stops pairing a CPU session with the GPU-sized 16-line REC batches. GpuGraphModel picks the tier from capability alone and RouteNocm is gone. UHD 770 tiny: --engine vulkan 86.9 ms (100/100 vs the earlier forced build), auto 55.1 ms (100/100 vs sharp), auto + SIMD_OCR_BACKEND=vulkan 83.9 ms (100/100 vs vulkan). --conc 4 threads mismatches=0, no fallback.
se_a and se_b were compiled on every session but never dispatched; SE pooling goes through se_fused. ocr_gemm_cm was an old probe shader with no callers.
…16 on sg16) Arc B580 advertises fp16->fp32 coopmat only as 8x16x16, which is exactly what the sg16 conv1x1_cm uses (A 8x16, B 16x16, acc 8x16). Requiring 16x16x16 sent it to gemm_nc: --engine vulkan medium 56 -> 775 ms, tiny 20 -> 49 ms, and Auto fell back to the CPU. VkDevice.CoopGemm now asks for 16x16x16 when the device takes the sg32 set and 8x16x16 when it takes sg16. UHD 770 has no VK_KHR_cooperative_matrix, so it still gets gemm_nc; 3080 Ti / 880M report 16x16x16 and are unchanged. B580, 2 rounds, median ms: tiny 20.5 / 22.2, small 29.1 / 28.1, medium 54.2 / 54.1; auto tiny 19.0 on the GPU. Accuracy 740 / 950 / 1003 as before. Co-authored-by: Cursor <cursoragent@cursor.com>
vulkan: no-coopmat GEMM for UHD 770, explicit Vulkan only
Android exposes the Vulkan loader only under the unversioned NDK name (/system/lib64/libvulkan.so); the resolver tried libvulkan.so.1 alone, so VkDevice.Create failed and every session fell back to the CPU. It now tries libvulkan.so.1, then libvulkan.so, in the one existing resolver, and a miss names every file it tried instead of the P/Invoke's "vulkan-1". Realme GT5 Pro (Snapdragon 8 Gen 3, Android 16): the loader resolves to libvulkan.so and VkDevice.Create picks "Adreno (TM) 750", Vulkan 1.3.128. Windows / desktop Linux resolve the same file names as before.
test/Sdcb.SimdPaddleOCR.AndroidBench is a net10.0-android app (CoreCLR, references the library's net10.0 build: AdvSimd kernels + Vulkan) driven over adb by run.ps1: build, install, push bench-out/models and dataset/ into the app's external files dir, launch with args/env extras, wait for <run>.done, pull files/out into bench-out/android and keep logcat when the process dies. Models are read from files, nothing is packaged. Modes: --caps (device, sgRange, gl_SubgroupSize under required size 0/16/32/64/128 through VkDevice.NewPipeline, coopmat list, limits, heaps, queues, timestamp check), --e2e (the Tests harness's options and JSON rows, so cmp.ps1 / steady.ps1 / --summarize work), --detprof / --recprof, --detcmp / --reccmp (CPU vs Vulkan on the same input), --conc, and two bisection aids: --layers (truncate the GPU graph after node k and read that node's output back) and --gemm (one GEMM dispatch vs an fp64 reference). GpuBench links the layer probe, GEMM check and CPU/GPU compare so the same numbers can be taken on a desktop GPU. Not in the .slnx or CI: it needs the android workload.
On Adreno 750 (no cooperative matrix, compute subgroups 64-128, so the no-coopmat tier with the driver-chosen wave64) gemm_nc returned garbage: tiny DET's probability map was all zeros while the session reported the GPU alive. The layer probe put the first divergence from the 3080 Ti at the first conv routed to gemm_nc_s, and --gemm reproduced it alone (6400x32x64: 188871 of 204800 outputs wrong; 1024x128x256: 126660 of 131072). Variants: plain fma vs mul+add and if-guarded vs select loads change nothing; dropping the register prefetch of the next A/B tile, or shrinking the tile to 32x32 with 4x4 accumulators, makes it exact. The driver's compiler mishandles the f16vec4 prefetch arrays carried across the K loop next to 64 fp32 accumulators. gemm_nc.comp gains -DDIRECT (each tile goes from global straight into shared memory, same fp32 accumulation order in K) built as gemm_nc_d / gemm_nc_ds. The tier takes them whenever it cannot pin an 8-lane subgroup; where it can (UHD 770) gemm_nc / gemm_nc_s with requiredSubgroupSize=8 stay as they were. gemm_nc.spv and gemm_nc_s.spv rebuild byte-identical. GT5 Pro, --gemm vs an fp64 reference: 0 wrong everywhere, and faster than the broken form (1024x128x256 0.254 -> 0.211 ms, 6400x32x64 0.231 -> 0.191 ms). tiny DET vs the phone CPU: box counts equal on 10/10 images, map maxAbs 0.05-0.15 (3080 Ti vs its CPU: 0.03-0.18).
A scalar constant (numel 1) was uploaded as a 2-byte buffer, and elem4 reads it as b[0] of a uint array. Desktop drivers do not bounds-check that load, Adreno does (per binding range) and returns 0 for it: the CLS head's Mul by a constant came out as 0, tiny CLS got 306 of 1022 right and the wrongly rotated lines took REC to CER 57%. ConstF16 and VecF16 now round their element count up to a multiple of 8. Only zeros are appended; every kernel reads the same values as before. GT5 Pro, tiny, --backend vulkan: cls 306 -> 1022/1022, exact_lines 216 -> 743 / 1036, CER 57.5% -> 2.40% (phone CPU: 742, 2.37%). No fallback in the log.
…ange Adreno 750 caps a storage binding at 128 MB. medium DET's 7x7 conv at 960x960 input (M 57600, K 1568) builds a 180 MB im2col matrix, and every load past 128 MB from its binding start read 0: the layer probe put the first divergence from the 3080 Ti at that conv (meanAbs -25%), and medium lost lines on large images (818 of 1036 classified, CER 13.2%). A dense kxk conv whose im2col matrix would not fit one binding now takes the direct convk_dot path instead (no K cap there). As a guard, a plan that would still bind more than maxStorageBufferRange (an arena slab, or an im2col the direct path cannot take) throws before anything runs, so the session serves that shape from the CPU instead of returning a partly zero map. Desktop GPUs report multi-GB ranges; none of this triggers there. GT5 Pro medium, --backend vulkan: DET boxes vs phone CPU equal in count on 6/6 images (map maxAbs 0.03-0.07, was up to 1.0); end to end cls 1035/1035, exact_lines 1003 / 1036, CER 0.15% (Windows CPU baseline 1004, 0.14%). Clamping the descriptor ranges themselves changed nothing on this driver, so that part is not included.
…cript --gemmsweep checks each .spv variant on an odd shape against fp64, then times it on the eleven hottest GEMM shapes of medium/small DET and REC, variants interleaved per round so clock / thermal drift hits all of them alike; --peak measures fp32 / fp16 FMA throughput from registers. gprof.ps1 is the phone twin of GpuBench's gprof.ps1 (same case list, per-dispatch timestamps, min over reps). run.ps1 no longer reports a finished run as dead when the app exits right after writing .done.
On Adreno 750 the direct-load GEMM ran at ~0.44 TFLOPS against a
measured 2.0 TFLOPS fp32 FMA peak. Sweeping the kernel on the eleven
hottest shapes (sum of min GPU ms, variants interleaved in one run):
64x64 BK16 fp32 tiles (was) 109.5
BK 8 83.1
128x128, 8x4 per thread, 512 thr 84.1
BK 8 + fp16 tiles (kept) 76.2 0.64 TFLOPS
same with 128x64 / 64x128 blocks 74.6 / 75.0 (noise; 64x128 loses
half a block on N=64)
BK 4 / BK 32 / 32x32 4x4 83.2 / 200.6 / 148.1
no shared memory, per-thread f16vec4 rows: 2-4x slower, and the fully
unrolled 8x8 form is miscomputed by the driver like the prefetch was
The DIRECT build now keeps the A/B tiles as fp16 in shared memory (they
come from fp16 memory, so this is exact) with BK 8; the staged tail gets
its own fp32 array so accumulators are never narrowed before bias,
residual and activation. Each output still takes one fma per k in
increasing K, so results are bit-identical: e2e tiny and medium match
the previous build 100/100 (hash, detected, texts). gemm_nc.spv and
gemm_nc_s.spv (UHD 770, 8-lane pin) rebuild byte-identical.
GT5 Pro, same-period pure GPU (gprof, min ms, two alternating rounds,
both rounds identical to 0.1 ms): medium DET 960 381 -> 311, 640x960
270 -> 221, medium REC 8x480 359 -> 270, 1x320 40.1 -> 34.7, small DET
64.5 -> 52.7, small REC 72.7 -> 54.6, tiny DET 28.0 -> 23.7, tiny REC
15.9 -> 12.3.
…arts After the GEMM, medium REC's third hotspot was reduce_hw (29 of 270 ms): below 4096 pixels the spatial mean ran one channel per workgroup, which on Adreno reads 2 bytes at a C*2 stride (the c512 means of the 8x480 batch took 7.6 ms each). The lite sg32 parts already take the two-phase reduce_hw4 / reduce_hw4b pair at any size for the same reason; the no-coopmat tier now does too when it cannot pin a narrow subgroup (_ncWide). UHD 770 (8-lane pin) keeps the old gate. GT5 Pro, same-period pure GPU (min ms, two alternating rounds, identical to 0.1 ms): medium REC 8x480 269.7 -> 243.3, medium DET 960 324.4 -> 317.0, 640x960 229.8 -> 226.1; small / tiny unchanged. e2e tiny, small, medium match the previous build 100/100 (hash, detected, texts). Tried and dropped: the shared-tile conv_dw4t for 9x9 depthwise (sg32 takes it up to 9) is slower here, medium DET 303.7 -> 317.0, small DET 52.7 -> 55.3.
Dense kxk convs switch from the direct convk_dot to im2col + GEMM once Kp > 1024 (SIMD_OCR_CONVD_KMAX); on UHD 770 lowering that cap lost. On Adreno 750 the direct kernel keeps up with the tuned GEMM (57600x2304x64 at ~0.56 TFLOPS) and skips writing and re-reading the im2col matrix, so the wide no-coopmat tier now defaults to direct at every K (the env var still overrides). Every dense conv in these models has Kp <= 2304. GT5 Pro, same-period pure GPU (min ms, two rounds each) with the cap at 256 / 512 / 1024 / 2304: medium DET 960 315.2 / 312.3 / 304.0 / 299.3, small DET 64.6 / 60.7 / 52.8 / 52.8, tiny DET 32.8 / 29.9 / 23.7 / 23.7; REC unchanged. With this build: medium DET 960 299.7, 640x960 216.9 -> 206.2. e2e medium matches the previous build 100/100. The dot -> GEMM cut for K <= 128 pointwise convs was re-swept too (M*cout > 2^12 / 2^14 / 2^16 / 2^18 / 2^20): 2^12..2^16 within 0.3 ms, 2^18 and up cost tiny DET 23.7 -> 26.5 ms. It stays at 2^16.
…s before a launch ab.ps1 runs each model round by round with the backends alternating and a cool-down in between, and prints median / last-50 median, peak working set, thermal status and accuracy per run. run.ps1 now force-stops the app and waits for the process to go away before launching: an intent delivered to an instance that is still exiting was swallowed and the run reported as dead.
…w 64 Auto skipped the no-coopmat tier everywhere because it trails the CPU on UHD 770. On Adreno 750 (no cooperative matrix, compute subgroups 64-128) the tuned tier beats the phone's own CPU on all three models, in every alternating round. GpuBackend.UsesGpu (and with it IsGpuBackend, i.e. the 16-line REC batches) now also says yes for Auto when the device reports SubgroupMin >= 64. UHD 770 (pins 8 lanes) and every other no-coopmat device keep Auto on the CPU; coopmat devices are unchanged. SIMD_OCR_BACKEND=cpu still forces the CPU. GT5 Pro, alternating rounds (median ms, cpu vs vulkan; thermal status 2-3, medium at 3 throughout): tiny 219 / 203 / 276 vs 156 / 170 / 155 / 152 small 867 / 785 / 812 vs 251 / 247 / 246 / 255 medium 3074 / 3294 / 3369 vs 1466 / 1458 / 1467 Accuracy cpu vs vulkan: 742 / 2.37% vs 743 / 2.40%, 950 / 0.41% vs 949 / 0.42%, 1004 / 0.14% vs 1003 / 0.15%. --backend auto: usesGpu=True, 100/100 identical to --backend vulkan; auto + SIMD_OCR_BACKEND=cpu: 100/100 identical to --backend cpu.
docs/vulkan-8gen3.md: the phone and driver, the --caps probe (sgRange 64-128, no cooperative matrix, 32 KB shared memory, 128 MB binding range, timestamps usable, 2.09 TFLOPS fp32 FMA), the no-coopmat route it takes, the three Adreno-only correctness bugs and how they were bisected, alternating CPU / Vulkan end to end on all three models with thermal status, accuracy and per-image parity, pure GPU before / after tuning and the hotspots, what was tuned, re-swept, dropped and left, the Auto rule, and the 3080 Ti regression. perf.md links it; the README rows now name the Vulkan loader per OS and the Android host, and say the desktop path is unchanged.
vulkan: 在骁龙 8 Gen 3（Adreno 750，安卓）上跑通并调优无协作矩阵档
…dule) - ObjC/Metal P/Invoke layer (no native deps), MetalBackend probe + auto-select on macOS, MetalSession/MetalRunner/MetalGraph mirroring the Vulkan seam: compiled managed MetalSchedule (shape-keyed LRU, zero-cost per run) + grow-only shared device arena. - ~40 MSL compute kernels embedded as resources; GEMM family incl. simdgroup MMA 64x64 (mm_sg_dd direct-load, mm_ic_sg implicit-GEMM conv), dot-product conv kernels, SE squeeze-excite two-phase split (paravirt lacks acq_rel atomics). - OcrBackend.Metal (additive); session-create fallback to CPU permanent. - GpuBench PipelineBench: System.Drawing -> ImageSharp (cross-platform). - docs/metal-m4.md: 3-tier baselines (tiny 1.9x / small 4.7x / medium 5.2x vs sharp, 4w), correctness gates (lines identical, --conc 0/72, --dtr bitwise clean), tuning log and paravirt caps.
… device-level MSL lib/PSO cache, concurrent interleave encoder, GpuSessionBase dedupe - mm_ic_* emit sites bound residual/dual tensors as (arena,0)/(arena,0,2) while flags could carry bit1/bit3 — bind the real slots, and gate the second site on scalarBias==0 (the scalar bias channel is repurposed as the dual-write flag in the mm kernels). - Auto no longer selects Metal on Intel Macs: Auto now requires Arm64+Apple7; explicit OcrBackend.Metal still allowed elsewhere via UsesGpu (probe validates simdgroup matrix live). - Move the embedded MSL library + ~40 PSOs to MtlDevice (OcrLibrary / OcrPipe): one compile per process instead of per MetalGraphModel, released on device Dispose. - Interleave now records through a concurrent compute encoder with one buffer barrier per level — under a serial encoder the per-rec barrier was a no-op and the level overlap unreachable. Serial path drops the dead per-rec barrier. - Extract GpuSessionBase + IOcrGraphRunner: GpuSession/MetalSession (identical 324-line bodies) become thin constructors over the shared session logic; GpuUnitsReady/MetalUnitsReady unify on CtcUnitsReady. - ProbeSimdMatrix now verifies PSO geometry (execWidth=32, maxTg>=256) and compares a full 64x64 tile against a reference GEMM. - Comment/doc nits: correct DWA/NOIC direction comments, mm_sg_dd gate comment, se two-phase explanation (relaxed atomics are universal MSL, not paravirt-specific), mm_ic pad-k semantics, MtlDataType UInt=30 fix (was Int2), MTLCreateSystemDefaultDevice +1 rule, doc CER metric and tool references.
…x OcrBackend.Auto comment
Metal: pure-C# macOS GPU backend (DET/CLS/REC full-graph schedule)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.