looping through all devices causes pthread_detach to segfault

[psteinb]

Hi - I am trying to contribute a stream benchmark using HCC
psteinb/GPU-STREAM/tree/hcc
I got the code to compile, but a simple loop through all devices

void listDevices(void)
{
  // Get number of devices
  std::vector<hc::accelerator> accs = hc::accelerator::get_all();

  // Print device names
  if (accs.empty())
  {
    std::cerr << "No devices found." << std::endl;
  }
  else
  {
    std::cout << std::endl;
    std::cout << "Devices:" << std::endl;
    for (int i = 0; i < accs.size(); i++)
    {
      std::cout << i << ": " << getDeviceName(accs[i]) << std::endl;
    }
    std::cout << std::endl;
  }
}

causes a segfault that originates in Kalmar::HSAContext::~HSAContext:

(ROCm-gdb) bt
#0  0x00007ffff7480545 in pthread_detach () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fffeeadc204 in ?? () from /opt/rocm/hsa/lib/libhsa-runtime-tools64.so.1
#2  0x00007fffeeadd12e in OnUnload () from /opt/rocm/hsa/lib/libhsa-runtime-tools64.so.1
#3  0x00007ffff669a162 in ?? () from /opt/rocm/hsa/lib/libhsa-runtime64.so.1
#4  0x00007ffff669a24c in ?? () from /opt/rocm/hsa/lib/libhsa-runtime64.so.1
#5  0x00007ffff669a488 in ?? () from /opt/rocm/hsa/lib/libhsa-runtime64.so.1
#6  0x00007ffff669028f in ?? () from /opt/rocm/hsa/lib/libhsa-runtime64.so.1
#7  0x00007fffee6db253 in HsaDebugAgent_hsa_shut_down () at HSAIntercept.cpp:49
#8  0x00007ffff6a80bd1 in Kalmar::HSAContext::~HSAContext() () from /opt/rocm/hcc-lc/lib/libmcwamp_hsa.so
#9  0x00007ffff6ed8259 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x00007ffff6ed82a5 in exit () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x000000000040ebc5 in parseArguments (argc=2, argv=0x7fffffffe428) at /home/steinb95/development/gpu_stream/main.cpp:279
#12 0x000000000040ea78 in main (argc=2, argv=0x7fffffffe428) at /home/steinb95/development/gpu_stream/main.cpp:63

listDevices is called by parseArguments in main.cpp:279. What am I missing?

[scchan]

The code looks fine to me.
When exit is called, the hcc runtime will destroy the execution context, which triggers the teardown of the rocr/hsa runtime and that's where the segfault happens.
I'll take a look at this with a debug build of rocr runtime and forward it to the rocr team if needed.

[psteinb]

thanks a bunch - I'll keep this issue open until further notice

[psteinb]

btw, the same error occurs with all examples contained in (RadeonOpenCompute/HCC-Example-Application)[https://github.com/RadeonOpenCompute/HCC-Example-Application]!

here are some more information about my system:

$ uname -a
Linux gpu006 4.4.0-kfd-compute-rocm-rel-1.1.1-10 #1 SMP Mon May 30 15:27:31 CDT 2016 x86_64 x86_64 x86_64 GNU/Linux
$ hcc --version
HCC clang version 3.5.0  (based on HCC 0.10.16186-d14f969-7461349 LLVM 3.5.0svn)
Target: x86_64-unknown-linux-gnu
Thread model: posix

on a two socket Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz!

[psteinb]

this problem is still present with rocm/1.2.0,

$ uname -a 
Linux gpu006 4.4.0-kfd-compute-rocm-rel-1.2-31 #1 SMP Fri Jul 22 06:06:24 CDT 2016 x86_64 x86_64 x86_64 GNU/Linux
$ hcc --version
HCC clang version 3.5.0  (based on HCC 0.10.16313-d90738a-10704f4 LLVM 3.5.0svn)
Target: x86_64-unknown-linux-gnu
Thread model: posix

any progress on your side on this?

[whchung]

@scchan is still looking into this issue. We are narrowing down potential issues in ROCR runtime where code objects may not be correctly placed in device memory under multi-GPU scenario.

[scchan]

@psteinb
I just dropped a fix on the develop branch of hcc to fix the compiler issues. The GPU-STREAM benchmark also requires some changes since the original version wasn't allocate memory and launching kernel on the selected device.
Here's my pull request for GPU-STREAM: psteinb/GPU-STREAM#1

[psteinb]

Very cool, I'll comment on it in the forked GPU-STREAM benchmark PR. thanks for your efforts so far.

[psteinb]

as posted to the PR, the segfault is still there with:

$ uname -a
Linux gpu006 4.4.0-kfd-compute-rocm-rel-1.2-31 #1 SMP Fri Jul 22 
06:06:24 CDT 2016 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
LSB Version: 
core-2.0-amd64:core-2.0-noarch:core-3.0-amd64:core-3.0-noarch:core-3.1-amd64:core-3.1-noarch:core-3.2-amd64:core-3.2-noarch:core-4.0-amd64:core-4.0-noarch:core-4.1-amd64:core-4.1-noarch:security-4.0-amd64:security-4.0-noarch:security-4.1-amd64:security-4.1-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.3 LTS
Release:        14.04
Codename:       trusty
$ hcc --version
HCC clang version 3.5.0  (based on HCC 0.10.16313-d90738a-10704f4 LLVM 
3.5.0svn)
Target: x86_64-unknown-linux-gnu
Thread model: posix

[scchan]

Currently the fix is only available in the develop branch of the hcc compiler and you'll have to get a new compiler built from source: https://github.com/RadeonOpenCompute/hcc/tree/develop

Here's the instruction:
https://github.com/RadeonOpenCompute/HCC-Native-GCN-ISA/wiki#develop
https://github.com/RadeonOpenCompute/HCC-Native-GCN-ISA/wiki#build-instructions

[psteinb]

solved with rocm 1.4.0