Bugs reported by compute-sanitizer

[Debugger-Pro]

Dear Authors,

The code appears to have errors/bugs, which might have been inherited from Flash-LLM. These errors are related to illegal memory access. While they do not terminate the program, they could lead to undefined behavior, which makes the program's results unreliable.

To verify the program, I recommend using compute-sanitizer with the following command:

compute-sanitizer --print-limit=1 ./spmm_test 4096 4096 16 80 7

you can find the error information as:

Comment out the code from Flash-LLM, and then run the verification code again. You will observe the following:

I would appreciate it if you could look into this matter further.

[quaternior]

Thank you for your kind explanation! had same problem, did you resolved it?

p.s. It seems to be related with the issues and answer below(Although they didn't discussed about the memory access)
AlibabaResearch/flash-llm#9
AlibabaResearch/flash-llm#14
AlibabaResearch/flash-llm#9 (comment)

[Debugger-Pro]

Thank you for your kind explanation! had same problem, did you resolved it?

p.s. It seems to be related with the issues and answer below(Although they didn't discussed about the memory access) AlibabaResearch/flash-llm#9 AlibabaResearch/flash-llm#14 AlibabaResearch/flash-llm#9 (comment)

Debugging this code would be extremely challenging due to the sheer number of bugs present, which together contribute to the high errors. If the authors do not respond, rewriting the kernel line by line might be a more feasible approach.

The discussion in the Flash-LLM repository has not been really helpful, as the authors do not appear to acknowledge or address the bugs in their code. This raises questions about how they managed to keep the decrease in model accuracy so minimal, as claimed in their paper.

Additionally, regarding the high error issue, relying solely on the total error sum seems insufficient. To improve clarity, I have enhanced their PrintMismatch function with error counting and RelativeErrorThreshold. By increasing the matrix size/sparsity and the value range of the input matrices, a high error count becomes evident.

void PrintMismatch(const char* KernelName,
                   size_t      MaxNumMismatch,
                   float       RelativeErrorThreshold,
                   half*       CuBlas,
                   half*       Other,
                   size_t      M_Global,
                   size_t      N_Global)
{
    printf("Mismatches between Cublas and %s (showing up to %zu):\n", KernelName, MaxNumMismatch);
    size_t count            = 0;
    size_t total_mismatches = 0;

    // First pass: count mismatches and compute variance
    for (size_t i = 0; i < M_Global; i++) {
        for (size_t j = 0; j < N_Global; j++) {
            float cublas_val = __half2float(CuBlas[i + j * M_Global]);
            float other_val  = __half2float(Other[i + j * M_Global]);
            float abs_cublas = fabs(cublas_val);

            if (abs_cublas > 0 && (fabs(cublas_val - other_val) / abs_cublas > RelativeErrorThreshold)) {
                total_mismatches++;

                // Print only up to MaxNumMismatch examples
                if (count < MaxNumMismatch) {
                    printf("(%zu,%zu) CuBlas=%f %s=%f, Diff=%f\n",
                           i,
                           j,
                           cublas_val,
                           KernelName,
                           other_val,
                           cublas_val - other_val);
                    count++;
                }
            }
        }
    }

    printf("Total mismatches: %zu\n", total_mismatches);
}
// usage:
  PrintMismatch("FlashLLM_v1", 10, 0.1, D_cublas_h, D_SpMM_h2, M_GLOBAL, N_GLOBAL);
  PrintMismatch("SpInfer", 10, 0.1, D_cublas_h, D_SpMM_hbitmapv3, M_GLOBAL, N_GLOBAL);

[quaternior]

Thank you for your kind response and very helpful contribution!