Nvidia Grace CPU detected as generic ARMV8 core

[giordano]

On an Nvidia GH200 node with current julia nightly build which comes with OpenBLAS v0.3.26, I get

$ OPENBLAS_VERBOSE=1 ./julia -e ''
Falling back to generic ARMV8 core

For reference:

$ lscpu 
Architecture:           aarch64
  CPU op-mode(s):       64-bit
  Byte Order:           Little Endian
CPU(s):                 72
  On-line CPU(s) list:  0-71
Vendor ID:              ARM
  Model name:           Neoverse-V2
    Model:              0
    Thread(s) per core: 1
    Core(s) per socket: 72
    Socket(s):          1
    Stepping:           r0p0
    Frequency boost:    disabled
    CPU max MHz:        3474.0000
    CPU min MHz:        81.0000
    BogoMIPS:           2000.00
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm 
                        dit uscat ilrcpc flagm ssbs sb dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh
$ head -n8 /proc/cpuinfo 
processor       : 0
BogoMIPS        : 2000.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd4f
CPU revision    : 0

Also, since neoversev2 doesn't appear in TargetList.txt trying to compile OpenBLAS with spack currently results in building the generic kernels only.

[martin-frbg]

NeoverseV2 should be covered by the ARMV8SVE target, which the library should automatically fall back to unless it was built with a compiler that did not yet support SVE or the operating system is neither Linux nor Android (i.e., no HWCAP_SVE detection) . Does it work when you supply OPENBLAS_CORETYPE=ARMV8SVE ?

[martin-frbg]

btw you should have gotten a message like "Unknown cpu model - implementer 0x41 part 0xd4f" in addition to the "falling back to..." if the code ever actually entered the midr_el1 (cpuid) parsing code. Did you see anything like that ?

[giordano]

With ARMV8SVE I don't get any output:

$ OPENBLAS_CORETYPE=ARMV8SVE OPENBLAS_VERBOSE=1 ./julia -e ''
$

I presume the type was accepted? But why it isn't detected automatically by the dynamic arch mechanism?

btw you should have gotten a message like "Unknown cpu model - implementer 0x41 part 0xd4f" in addition to the "falling back to..." if the code ever actually entered the midr_el1 (cpuid) parsing code. Did you see anything like that ?

No.

[martin-frbg]

In general you may get (a little) more output with OPENBLAS_VERBOSE=2. What is your operating system please ? (And if Linux, does it provide the /sys/devices tree in case getauxval(HWCAP_CPUID) is not supported ? Also a current design limitation is that we only read its entry for cpu0 - but I do not think this should matter in your case)

[giordano]

In general you may get (a little) more output with OPENBLAS_VERBOSE=2.

$ OPENBLAS_CORETYPE=ARMV8SVE OPENBLAS_VERBOSE=2 ./julia -e ''
Core: armv8

What is your operating system please ?

Yes, operating system is Linux (Red Hat 9)

(And if Linux, does it provide the /sys/devices tree in case getauxval(HWCAP_CPUID) is not supported ? Also a current design limitation is that we only read its entry for cpu0 - but I do not think this should matter in your case)

$ ls -l /sys/devices/
total 0
drwxr-xr-x   5 root root 0 Dec 20 17:08 ARMHC502:00
drwxr-xr-x   4 root root 0 Dec 20 17:08 ARMHC97C:00
drwxr-xr-x   3 root root 0 Dec 20 17:08 ARMHC9FF:00
drwxr-xr-x   6 root root 0 Dec 20 17:08 armv8_pmuv3_0
drwxr-xr-x   3 root root 0 Dec 20 17:08 breakpoint
drwxr-xr-x   6 root root 0 Dec 20 17:09 cs_etm
drwxr-xr-x   3 root root 0 Jan 15 18:23 cs_system_cfg
drwxr-xr-x   4 root root 0 Dec 20 17:08 kprobe
drwxr-xr-x   5 root root 0 Dec 20 17:08 LNXSYSTM:00
drwxr-xr-x   5 root root 0 Dec 20 17:08 nvidia_cnvlink_pmu_0
drwxr-xr-x   5 root root 0 Dec 20 17:08 nvidia_nvlink_c2c0_pmu_0
drwxr-xr-x   5 root root 0 Dec 20 17:08 nvidia_nvlink_c2c1_pmu_0
drwxr-xr-x   5 root root 0 Dec 20 17:08 nvidia_pcie_pmu_0
drwxr-xr-x   5 root root 0 Dec 20 17:08 nvidia_scf_pmu_0
drwxr-xr-x   5 root root 0 Dec 20 17:08 pci0000:00
drwxr-xr-x   5 root root 0 Dec 20 17:08 pci0002:00
drwxr-xr-x   5 root root 0 Dec 20 17:08 pci0004:00
drwxr-xr-x   5 root root 0 Dec 20 17:08 pci0005:00
drwxr-xr-x   5 root root 0 Dec 20 17:08 pci0006:00
drwxr-xr-x   5 root root 0 Dec 20 17:08 pci0008:00
drwxr-xr-x   5 root root 0 Dec 20 17:08 pci0009:00
drwxr-xr-x 110 root root 0 Dec 20 17:08 platform
drwxr-xr-x  10 root root 0 Dec 20 17:08 pnp0
drwxr-xr-x   3 root root 0 Dec 20 17:08 soc0
drwxr-xr-x   3 root root 0 Dec 20 17:08 software
drwxr-xr-x   9 root root 0 Dec 20 17:08 system
drwxr-xr-x   3 root root 0 Dec 20 17:08 tracepoint
drwxr-xr-x   4 root root 0 Dec 20 17:08 uprobe
drwxr-xr-x  20 root root 0 Dec 20 17:08 virtual

Do you need something else precisely?

[martin-frbg]

$ OPENBLAS_CORETYPE=ARMV8SVE OPENBLAS_VERBOSE=2 ./julia -e ''
Core: armv8

suggests your build of libopenblas does not know ARMV8SVE

Do you need something else precisely?

/sys/devices/system/cpu/cpu0/regs/identification/midr_el1 is what the code would/should be trying to read - it is probably there, and it is strange that it is not getting used. Are you certain that your julia runtime is picking up 0.3.26 and not some other/ancient version ?

[giordano]

/sys/devices/system/cpu/cpu0/regs/identification/midr_el1 is what the code would/should be trying to read - it is probably there, and it is strange that it is not getting used.

$ cat /sys/devices/system/cpu/cpu0/regs/identification/midr_el1
0x00000000410fd4f0

Are you certain that your julia runtime is picking up 0.3.26 and not some other/ancient version ?

Yes:

$ OPENBLAS_CORETYPE=ARMV8SVE OPENBLAS_VERBOSE=2 ./julia -q
Core: armv8
julia> using OpenBLAS_jll, LinearAlgebra

julia> strip(unsafe_string(ccall((BLAS.@blasfunc(openblas_get_config), libopenblas), Ptr{UInt8}, () )))
"OpenBLAS 0.3.26  USE64BITINT DYNAMIC_ARCH NO_AFFINITY armv8 MAX_THREADS=512"

We built this with GCC 8, maybe it's too old to actually enable the SVE kernels?

[martin-frbg]

I'm fairly sure you'd need at least GCC 10.1 to get the SVE intrinsics support - but check if your GCC8 has arm_sve.h

[giordano]

Sounds like we don't have it with gcc 8, yeah:

% julia --compile=min -e 'using BinaryBuilderBase; BinaryBuilderBase.runshell(Platform("aarch64", "linux"); preferred_gcc_version=v"8")' 
sandbox:${WORKSPACE} # find /opt/${target}/ -name 'arm_sve.h'
sandbox:${WORKSPACE} # 
logout
% julia --compile=min -e 'using BinaryBuilderBase; BinaryBuilderBase.runshell(Platform("aarch64", "linux"); preferred_gcc_version=v"10")'
sandbox:${WORKSPACE} # find /opt/${target}/ -name 'arm_sve.h'
/opt/aarch64-linux-gnu/lib/gcc/aarch64-linux-gnu/10.2.0/include/arm_sve.h

Ok, we may need to recompile it with GCC 10 then.

[martin-frbg]

Yeah, some of the kernels use intrinsics so it has become a hard requirement. GCC8 dates from 2018/19 when SVE wasn't really a thing yet

[giordano]

With a new build of OpenBLAS 0.3.26 with GCC 11 (from JuliaPackaging/Yggdrasil#7952) I now get

$ OPENBLAS_VERBOSE=2 ./julia -e ''
Core: armv8sve

So I can confirm the only problem was that the previous build of OpenBLAS didn't have the SVE kernels at all because was compiled with GCC 8. Thanks for the help!