Repository navigation
Open
Labels
Description
While building for Arch Linux, I’m observing 4 tests failures in the aforementioned suite:
[ RUN ] TestRandomInt64QuantileKernel.Overlapped
/build/arrow/src/apache-arrow-7.0.0/cpp/src/arrow/compute/kernels/aggregate_test.cc:3234: Failure
Value of: (quantiles[j] == numeric_scalar->value) || (std::isnan(quantiles[j]) && std::isnan(numeric_scalar->value))
Actual: false
Expected: true
[ FAILED ] TestRandomInt64QuantileKernel.Overlapped (6 ms)
[ RUN ] TestRandomInt64QuantileKernel.Sliced
/build/arrow/src/apache-arrow-7.0.0/cpp/src/arrow/compute/kernels/aggregate_test.cc:3234: Failure
Value of: (quantiles[j] == numeric_scalar->value) || (std::isnan(quantiles[j]) && std::isnan(numeric_scalar->value))
Actual: false
Expected: true
[ FAILED ] TestRandomInt64QuantileKernel.Sliced (25 ms)
[ RUN ] TestRandomFloatQuantileKernel.Exact
/build/arrow/src/apache-arrow-7.0.0/cpp/src/arrow/compute/kernels/aggregate_test.cc:3234: Failure
Value of: (quantiles[j] == numeric_scalar->value) || (std::isnan(quantiles[j]) && std::isnan(numeric_scalar->value))
Actual: false
Expected: true
[ FAILED ] TestRandomFloatQuantileKernel.Exact (0 ms)
[ RUN ] TestRandomFloatQuantileKernel.Sliced
/build/arrow/src/apache-arrow-7.0.0/cpp/src/arrow/compute/kernels/aggregate_test.cc:3234: Failure
Value of: (quantiles[j] == numeric_scalar->value) || (std::isnan(quantiles[j]) && std::isnan(numeric_scalar->value))
Actual: false
Expected: true
/build/arrow/src/apache-arrow-7.0.0/cpp/src/arrow/compute/kernels/aggregate_test.cc:3234: Failure
Value of: (quantiles[j] == numeric_scalar->value) || (std::isnan(quantiles[j]) && std::isnan(numeric_scalar->value))
Actual: false
Expected: true
/build/arrow/src/apache-arrow-7.0.0/cpp/src/arrow/compute/kernels/aggregate_test.cc:3234: Failure
Value of: (quantiles[j] == numeric_scalar->value) || (std::isnan(quantiles[j]) && std::isnan(numeric_scalar->value))
Actual: false
Expected: true
/build/arrow/src/apache-arrow-7.0.0/cpp/src/arrow/compute/kernels/aggregate_test.cc:3234: Failure
Value of: (quantiles[j] == numeric_scalar->value) || (std::isnan(quantiles[j]) && std::isnan(numeric_scalar->value))
Actual: false
Expected: true
[ FAILED ] TestRandomFloatQuantileKernel.Sliced (0 ms)
They are also happening with 6.0.1, but were not sometime ago so I suspect an update in some of arrow dependencies to be responsible for this. I’m happy to provide any information that could be useful, but I don’t want to create an account on JIRA, I try to limit the number of accounts I have everywhere.
Activity
loqs commentedon Mar 22, 2022
Swapping from gcc to clang and the tests than pass. I verified the same four tests fails on my system using gcc 11.2.0.
wjones127 commentedon Mar 22, 2022
We've recently had some issues with SIMD instructions seeming incorrect in certain compilers, which sounds like this might be related to. (See #12422 (comment)). Stumped us so far, but I think we are still looking into it. cc @rok @jonkeane
loqs commentedon Mar 26, 2022
Adding any of
-mno-sse-mno-sse2-mno-sse3to CXXFLAGS and the tests pass.Replacing
-DARROW_SIMD_LEVEL=AVX2with-DARROW_SIMD_LEVEL=NONE -DARROW_RUNTIME_SIMD_LEVEL=AVX512and the tests also pass.This is all on an AMD system that supports up to avx2.
erydit commentedon Aug 3, 2022
I confirm the problem on my machine.
With -DARROW_SIMD_LEVEL=AVX2 all test was failed.
With -DARROW_SIMD_LEVEL=SSE4_2 the 54th test 'arrow-flight-test' failed and all other tests passed.
As I understand my CPU does not support AVX2 instructions and that was the main problem.
But the test 'arrow-flight-test' is not related to that, anyway I also append Log of that test.
arrow-flight-test'
Running arrow-flight-test, redirecting output into /home/rstanislav/Desktop/arrow/src/build/build/test-logs/arrow-flight-test.txt (attempt 1/1) Traceback (most recent call last): File "/home/rstanislav/Desktop/arrow/src/apache-arrow-8.0.1/cpp/build-support/asan_symbolize.py", line 368, in loop.process_stdin() File "/home/rstanislav/Desktop/arrow/src/apache-arrow-8.0.1/cpp/build-support/asan_symbolize.py", line 340, in process_stdin line = sys.stdin.readline() File "/usr/lib/python3.10/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 3112: invalid continuation byte ~/Desktop/arrow/src/build/src/arrow/flight Test time = 1.36 secMy system:
OS: Manjaro 21.3.6 Ruah
Kernel: x86_64 Linux 5.15.57-2-MANJARO
Uptime: 26m
Packages: 1469
Shell: bash
Resolution: 1920x1080
DE: Xfce4
WM: Xfwm4
WM Theme: Matcha-sea
GTK Theme: Matcha-sea [GTK2]
Icon Theme: Papirus-Maia
Font: Noto Sans 10
Disk: 1,5T / 6,8T (23%)
CPU: AMD FX-8350 Eight-Core @ 8x 4GHz
GPU: AMD RS780 (DRM 2.50.0 / 5.15.57-2-MANJARO, LLVM 14.0.6)
RAM: 1829MiB / 19485MiB
lscpu:
Архитектура: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Порядок байт: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
ID прроизводителя: AuthenticAMD
Имя модели: AMD FX(tm)-8350 Eight-Core Processor
Семейство ЦПУ: 21
Модель: 2
Thread(s) per core: 2
Ядер на сокет: 4
Сокетов: 1
Степпинг: 0
Frequency boost: enabled
CPU(s) scaling MHz: 38%
CPU max MHz: 4000,0000
CPU min MHz: 1400,0000
BogoMIPS: 8003.18
Флаги: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx f
xsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good no
pl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4
_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse
4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topo
ext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lo
ck nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
Virtualization features:
Виртуализация: AMD-V
Caches (sum of all):
L1d: 128 KiB (8 instances)
L1i: 256 KiB (4 instances)
L2: 8 MiB (4 instances)
L3: 8 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerabilities:
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Mitigation; untrained return thunk; SMT vulnerable
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP disabled, RSB filling
Srbds: Not affected
Tsx async abort: Not affected
pitrou commentedon Aug 4, 2022
Hmm, it's a pity that we're not displaying the actual values being compared on failure.
It's possible that the test failures are simply due to floating-point accuracy. But ideally the tests should be improved to take that into account.
@cyb70289 You might want to take a look.
(but @wjones127 perhaps you also want to take a deeper look and suggest improvements/fixes, especially if you manage to reproduce?)
pitrou commentedon Aug 4, 2022
@erydit The GDAL is issue is unrelated, please let's not conflate these.
Also, the fact that Arrow doesn't work on an AVX2 CPU if compiled with AVX2 enabled is entirely expected. It is not a bug.
QuLogic commentedon Nov 18, 2024
These are failing on Fedora aarch64 with 16.1.0. It appears that these now contain more information, and seem like they're small floating-point discrepancies:
`TestRandomInt64QuantileKernel` failures
cyb70289 commentedon Nov 18, 2024
I tested arrow 16.1.0 tag in fedora42 container on an arm neoverse n2 server, but didn't reproduce this issue. All tests passed.
I'm using the same cmake command line as in the build log provided by @QuLogic.
What's the hardware running the build?
QuLogic commentedon Nov 18, 2024
Here is the full task; and this is the hardware info.
cyb70289 commentedon Nov 18, 2024
From hardware info, the failure happens on neoverse-n1, probably in a virtual machine due to low cpu count.
I tested on a neoverse-n1 machine, still cannot reproduce the error. This is strange.
QuLogic commentedon Nov 19, 2024
Also fails on s390x in a similar way:
s390x failures
pitrou commentedon Nov 19, 2024
Perhaps it may depend on FPU settings, or on compiler options?
pitrou commentedon Nov 19, 2024
By the way, looking at one of the failing values, the values are only different at the last bit of precision:
cyb70289 commentedon Nov 20, 2024
Managed to reproduce this error by adding below compiler options. Trying to narrow down the culprit.
cyb70289 commentedon Nov 20, 2024
Per my test,
-Wp,-D_GLIBCXX_ASSERTIONSleads to this error. Surprisingly.6 remaining items
cyb70289 commentedon Nov 21, 2024
Yes, thanks @pitrou !
cyb70289 commentedon Nov 21, 2024
You are right. x86 also fails the test if build with
-mfma.ArchangeGabriel commentedon Nov 21, 2024
(JFTR, addition and multiplication should be commutative for floating point numbers, associative is what they might not be)
cyb70289 commentedon Nov 22, 2024
Indeed.
But for fma operation, IMO compiler should always stick to one ordering to prevent surprising result.
fma(a, b, c*d) != fma(c, d, a*b)though
a*b + c*d = c*d + a*b[-]Test failure in arrow-compute-aggregate-test[/-][+][C++] Test failure in arrow-compute-aggregate-test[/+]Cryolitia commentedon Apr 21, 2025
It occurs on Arch Linux RISC-V on SG2042, too
Log: https://archriscv.felixc.at/.status/log.htm?url=logs/arrow/arrow-19.0.1-7.log
addpatch: arrow 19.0.1
addpatch: arrow 19.0.1
addpatch: arrow 19.0.1-7
addpatch: arrow 19.0.1-7
dmitry-chirkov-dremio commentedon Nov 14, 2026
@pitrou I was scanning last 30 days of CI failures and this is not one of them.
Closure candidate.
Also, what's the good way to bring up all the failures up for review?
dmitry-chirkov-dremio commentedon Nov 14, 2026
No recorded failures in the last 30 days:
#37757 split_whitespace on macOS (2023): no hits.
#46983 R flaky to_duckdb (2025): no hits.
#38386 random failure in dictionary sorting (2023): no hits.
#44048 pandas categorical random failure (2024): no hits.
#31593 Gandiva arena sporadic (2022): no hits
pitrou commentedon Nov 16, 2026
Perhaps one the submitters can check with latest Arrow if they still failures? @ArchangeGabriel @Cryolitia
Cryolitia commentedon Nov 18, 2026
I can still reproduce this with unmodified Arrow 26.0.1 using
extra-riscv64-buildin a RISC-V chroot under QEMU, with GCC 16.2.1. The build retained_GLIBCXX_ASSERTIONS,-march=rv64gc -mabi=lp64d, and LTO.Running
arrow-compute-aggregate-testgives:All seven reported numerical mismatches are exactly 1 ULP apart. For example,
TestRandomFloatQuantileKernel.Exactreports:The failure occurs at the exact-equality assertion.
Separately, on x86_64 with
-mfma, the same four tests fail with_GLIBCXX_ASSERTIONSenabled. All 345 tests pass either after removing that macro, or after retaining it and replacing the equality check withAssertWithinUlp(..., 1). This is consistent with the FMA rounding differences discussed earlier.