rarias/jungle
rarias/jungle
1
0
Fork 0
Code Issues 27 Pull Requests Packages Projects Releases Wiki Activity

Configure MPI to run multi-node benchmarks #16

New Issue
Closed
opened 2023-04-27 12:39:28 +02:00 by rarias · 16 comments
rarias commented 2023-04-27 12:39:28 +02:00 (Migrated from pm.bsc.es)
Copy Link

We should be able to run benchmarks in at least two nodes so we can pass distributed tests in ovni. We should add automated CI tests that pass osu test (or others) when the configuration is updated.

We should be able to run benchmarks in at least two nodes so we can pass distributed tests in ovni. We should add automated CI tests that pass osu test (or others) when the configuration is updated.
rarias commented 2023-04-28 16:05:39 +02:00 (Migrated from pm.bsc.es)
Copy Link

We will need to configure the MPI network, as it never works out of the box.

  • The locked memory, which is 8M by default, needs to be increased.
  • A bug in the driver implementation of hfi1 doesn't seem to be happening in the last release (6.3), see #17.
  • The PMI/PMIX problem prevents neither OpenMPI or MPICH from talking to the slurmstepd. We need to set a compatible pmi implementation from the slurm config.
We will need to configure the MPI network, as it never works out of the box. - [x] The locked memory, which is 8M by default, needs to be increased. - [x] A bug in the driver implementation of hfi1 doesn't seem to be happening in the last release (6.3), see #17. - [x] The PMI/PMIX problem prevents neither OpenMPI or MPICH from talking to the slurmstepd. We need to set a compatible pmi implementation from the slurm config.
rarias commented 2023-04-28 16:13:11 +02:00 (Migrated from pm.bsc.es)
Copy Link

SLURM is already built with pmix 3:

$ nix log /nix/store/kxzpirpvw5wgr6rc6wpkl9m9ljmbky50-slurm-23.02.1.1 | grep -o -- '--with-pmi[^ ]*'
--with-pmix=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4
SLURM is already built with pmix 3: ``` $ nix log /nix/store/kxzpirpvw5wgr6rc6wpkl9m9ljmbky50-slurm-23.02.1.1 | grep -o -- '--with-pmi[^ ]*' --with-pmix=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4 ```
rarias commented 2023-04-28 16:31:47 +02:00 (Migrated from pm.bsc.es)
Copy Link

Well, apparently we can only use MPICH with openpmix with the ch4 device and no pm.

From https://openpmix.github.io/support/faq/which-environments-include-support-for-pmix :

MPICH: Support for PMIx was added in MPICH version 3.3 using the ch4 device configuration. Support is tested regularly against PMIx v2.1. Example configurations include:

--with-device=ch4:ofi [--with-libfabric=path/to/install]
--with-pmi=pmix [--with-pmix=path/to/install] --with-pm=no

However, by default MPICH is built with PM and no device:

$ nix log nixpkgs#mpich | grep -- '\./configure' | head -2 | tr ' ' '\n'
./configure
configure:
running
/nix/store/rnkas52f8868g1hjdlldbvh6snm3pglv-bash-5.2-p15/bin/bash
./configure
--disable-option-checking
'--prefix=/nix/store/zph7irqa0cfwrp13phz71gnbgwjlvch3-mpich-4.1.1'
--disable-versioning
--enable-embedded
'--disable-static'
'--disable-dependency-tracking'
'--enable-shared'
'--enable-sharedlib'
'--with-pm=hydra:gforker'
'FFLAGS=-fallow-argument-mismatch
'
'FCFLAGS=-fallow-argument-mismatch
'
'CC=gcc'
'CXX=g++'
'FC=gfortran'
--cache-file=/dev/null
--srcdir=.
Well, apparently we can only use MPICH with openpmix with the ch4 device and no pm. From https://openpmix.github.io/support/faq/which-environments-include-support-for-pmix : > MPICH: Support for PMIx was added in MPICH version 3.3 using the ch4 device configuration. Support is tested regularly against PMIx v2.1. Example configurations include: > > ``` > --with-device=ch4:ofi [--with-libfabric=path/to/install] > --with-pmi=pmix [--with-pmix=path/to/install] --with-pm=no > ``` However, by default MPICH is built with PM and no device: ``` $ nix log nixpkgs#mpich | grep -- '\./configure' | head -2 | tr ' ' '\n' ./configure configure: running /nix/store/rnkas52f8868g1hjdlldbvh6snm3pglv-bash-5.2-p15/bin/bash ./configure --disable-option-checking '--prefix=/nix/store/zph7irqa0cfwrp13phz71gnbgwjlvch3-mpich-4.1.1' --disable-versioning --enable-embedded '--disable-static' '--disable-dependency-tracking' '--enable-shared' '--enable-sharedlib' '--with-pm=hydra:gforker' 'FFLAGS=-fallow-argument-mismatch ' 'FCFLAGS=-fallow-argument-mismatch ' 'CC=gcc' 'CXX=g++' 'FC=gfortran' --cache-file=/dev/null --srcdir=. ```
rarias commented 2023-04-28 17:18:59 +02:00 (Migrated from pm.bsc.es)
Copy Link

Enabled PMIx by default. MPICH is working too:

$ srun -N 2 /nix/store/10np2fknixpp7fcspgy7cw6dznq3wk0n-osu-micro-benchmarks-7.1-1/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw
# OSU MPI Bandwidth Test v7.1
# Size      Bandwidth (MB/s)
# Datatype: MPI_CHAR.
1                       1.35
2                       2.70
4                       5.41
8                      10.84
16                     21.36
32                     42.54
64                     85.18
128                   166.14
256                   334.15
512                   647.32
1024                 1220.29
2048                 2095.01
4096                 3192.88
8192                 4250.85
16384                4906.29
32768                5413.94
65536                7414.99
131072              10952.41
262144              11995.80
524288              12118.50
1048576             12305.86
2097152             12319.64
4194304             12356.65
Enabled PMIx by default. MPICH is working too: ``` $ srun -N 2 /nix/store/10np2fknixpp7fcspgy7cw6dznq3wk0n-osu-micro-benchmarks-7.1-1/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw # OSU MPI Bandwidth Test v7.1 # Size Bandwidth (MB/s) # Datatype: MPI_CHAR. 1 1.35 2 2.70 4 5.41 8 10.84 16 21.36 32 42.54 64 85.18 128 166.14 256 334.15 512 647.32 1024 1220.29 2048 2095.01 4096 3192.88 8192 4250.85 16384 4906.29 32768 5413.94 65536 7414.99 131072 10952.41 262144 11995.80 524288 12118.50 1048576 12305.86 2097152 12319.64 4194304 12356.65 ```
rarias commented 2023-06-15 08:54:16 +02:00 (Migrated from pm.bsc.es)
Copy Link

I updated MPI to the last release (2021.9, released in 2023), but is still failing in different ways. Here is the script I'm using to test:

#!/bin/sh -x

export LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib
export I_MPI_DEBUG=5

export I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmix.so
#export I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi2.so
#export I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi.so

prog="result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple"

srun -N 2 $prog

PMI_LIBRARY set to libpmix.so

With PMI_LIBRARY set to libpmix.so it segfaults:

Jun 15 08:30:38 xeon01 kernel: osu_bw[48681]: segfault at 0 ip 00007fe2b353f04d sp 00007ffcc127ddf0 error 4 in libmpi.so.12.0.0[7fe2b3200000+e80000] likely on CPU 45 (core 3, socket 1)
Jun 15 08:30:38 xeon01 kernel: Code: d5 00 41 83 38 03 75 1e 48 8d 3d 8e 71 fa 00 e8 21 89 d0 ff 85 c0 0f 85 ea 00 00 00 eb 08 85 db 0f 85 09 01 00 00 89 5c 24 18 <8b> 6d 00 81 fd 01 00 00>
Jun 15 08:30:38 xeon01 systemd[1]: Started Process Core Dump (PID 48682/UID 0).
Jun 15 08:30:39 xeon01 systemd-coredump[48683]: [🡕] Process 48681 (osu_bw) of user 1880 dumped core.

                                                Module libhwloc.so.15 without build-id.
                                                Module libevent_pthreads-2.1.so.7 without build-id.
                                                Module libevent_core-2.1.so.7 without build-id.
                                                Module libz.so.1 without build-id.
                                                Module libpmix.so.2 without build-id.
                                                Module libgcc_s.so.1 without build-id.
                                                Module libstdc++.so.6 without build-id.
                                                Module osu_bw without build-id.
                                                Stack trace of thread 48681:
                                                #0  0x00007fe2b353f04d MPIR_Err_return_comm (libmpi.so.12 + 0x33f04d)
                                                #1  0x00007fe2b36ce6c0 PMPI_Init (libmpi.so.12 + 0x4ce6c0)
                                                #2  0x000000000040255b main (osu_bw + 0x255b)
                                                #3  0x00007fe2b2a2924e __libc_start_call_main (libc.so.6 + 0x2924e)
                                                #4  0x00007fe2b2a29309 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29309)
                                                #5  0x0000000000403425 _start (osu_bw + 0x3425)
                                                ELF object binary architecture: AMD x86-64
Jun 15 08:30:39 xeon01 systemd[1]: systemd-coredump@1-48682-0.service: Deactivated successfully.

And also shows some error in stderr:

+ export LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib
+ LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib
+ export I_MPI_DEBUG=5
+ I_MPI_DEBUG=5
+ export I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmix.so
+ I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmix.so
+ prog='result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple'
+ srun -N 2 result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple
MPIR_pmi_virtualization(): MPI startup(): Error: Could not auto detect PMI version. Please set I_MPI_PMI variable
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmix.so
Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(176):
MPID_Init(1437)......:
MPIR_pmi_init(109)...:
(unknown)(): Other MPI error
MPIR_pmi_virtualization(): MPI startup(): Error: Could not auto detect PMI version. Please set I_MPI_PMI variable
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmix.so
Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(176):
MPID_Init(1437)......:
MPIR_pmi_init(109)...:
(unknown)(): Other MPI error
srun: error: xeon01: task 0: Segmentation fault (core dumped)
srun: error: xeon02: task 1: Segmentation fault (core dumped)

PMI_LIBRARY set to libpmi2.so

With libpmi2.so causes a authentication problem with munge:

Jun 15 08:49:27 xeon01 slurmd[17717]: launch task StepId=289.0 request from UID:1880 GID:564 HOST:10.0.40.7 PORT:59170
Jun 15 08:49:27 xeon01 munged[1864]: Unauthorized credential for client UID=0 GID=0

And freezes:

./run.sh
+ export LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib
+ LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib
+ export I_MPI_DEBUG=5
+ I_MPI_DEBUG=5
+ export I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi2.so
+ I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi2.so
+ prog='result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple'
+ srun -N 2 result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi2.so
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi2.so
[0] MPI startup(): Intel(R) MPI Library, Version 2021.9  Build 20230307 (id: d82b3071db)
[0] MPI startup(): Copyright (C) 2003-2023 Intel Corporation.  All rights reserved.
[0] MPI startup(): library kind: release
[0] MPI startup(): libfabric version: 1.17.1
[0] MPI startup(): libfabric provider: psm2
[0] MPI startup(): File "/tuning_skx_shm-ofi_psm2.dat" not found
[0] MPI startup(): Load tuning file: "/tuning_skx_shm-ofi.dat"
[0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found
[0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found

PMI_LIBRARY set to libpmi.so

With libpmi.so also causes the same credential problem:

Jun 15 08:50:41 xeon01 slurmd[17717]: launch task StepId=290.0 request from UID:1880 GID:564 HOST:10.0.40.7 PORT:34792
Jun 15 08:50:41 xeon01 munged[1864]: Unauthorized credential for client UID=0 GID=0
Jun 15 08:50:45 xeon01 slurmstepd[49043]: [290.0] done with job

But it manages to run and works okay:

hut$ ./run.sh
+ export LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib
+ LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib
+ export I_MPI_DEBUG=5
+ I_MPI_DEBUG=5
+ export I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi.so
+ I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi.so
+ prog='result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple'
+ srun -N 2 result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi.so
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi.so
[0] MPI startup(): Intel(R) MPI Library, Version 2021.9  Build 20230307 (id: d82b3071db)
[0] MPI startup(): Copyright (C) 2003-2023 Intel Corporation.  All rights reserved.
[0] MPI startup(): library kind: release
[0] MPI startup(): libfabric version: 1.17.1
[0] MPI startup(): libfabric provider: psm2
[0] MPI startup(): File "/tuning_skx_shm-ofi_psm2.dat" not found
[0] MPI startup(): Load tuning file: "/tuning_skx_shm-ofi.dat"
[0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found
[0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found
[0] MPI startup(): File "" not found
[0] MPI startup(): Unable to read tuning file for ch4 level
[0] MPI startup(): File "" not found
[0] MPI startup(): Unable to read tuning file for net level
[0] MPI startup(): File "" not found
[0] MPI startup(): Unable to read tuning file for shm level
[0] MPI startup(): Rank    Pid      Node name  Pin cpu
[0] MPI startup(): 0       49051    xeon01     0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55
[0] MPI startup(): 1       31289    xeon02     0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi.so
# OSU MPI Bandwidth Test v7.1
# Size      Bandwidth (MB/s)
# Datatype: MPI_CHAR.
1                       1.22
2                       2.73
4                       5.61
8                      11.38
16                     21.85
32                     43.34
64                     86.54
128                   174.80
256                   334.05
512                   628.00
1024                 1022.63
2048                 1459.20
4096                 1996.50
8192                 2424.95
16384                2802.75
32768                2997.32
65536                7643.60
131072               6716.45
262144               6780.43
524288               7462.88
1048576              8100.58
2097152              8459.38
4194304              8496.18
I updated MPI to the last release (2021.9, released in 2023), but is still failing in different ways. Here is the script I'm using to test: ```sh #!/bin/sh -x export LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib export I_MPI_DEBUG=5 export I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmix.so #export I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi2.so #export I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi.so prog="result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple" srun -N 2 $prog ``` # PMI_LIBRARY set to libpmix.so With PMI_LIBRARY set to libpmix.so it segfaults: ``` Jun 15 08:30:38 xeon01 kernel: osu_bw[48681]: segfault at 0 ip 00007fe2b353f04d sp 00007ffcc127ddf0 error 4 in libmpi.so.12.0.0[7fe2b3200000+e80000] likely on CPU 45 (core 3, socket 1) Jun 15 08:30:38 xeon01 kernel: Code: d5 00 41 83 38 03 75 1e 48 8d 3d 8e 71 fa 00 e8 21 89 d0 ff 85 c0 0f 85 ea 00 00 00 eb 08 85 db 0f 85 09 01 00 00 89 5c 24 18 <8b> 6d 00 81 fd 01 00 00> Jun 15 08:30:38 xeon01 systemd[1]: Started Process Core Dump (PID 48682/UID 0). Jun 15 08:30:39 xeon01 systemd-coredump[48683]: [🡕] Process 48681 (osu_bw) of user 1880 dumped core. Module libhwloc.so.15 without build-id. Module libevent_pthreads-2.1.so.7 without build-id. Module libevent_core-2.1.so.7 without build-id. Module libz.so.1 without build-id. Module libpmix.so.2 without build-id. Module libgcc_s.so.1 without build-id. Module libstdc++.so.6 without build-id. Module osu_bw without build-id. Stack trace of thread 48681: #0 0x00007fe2b353f04d MPIR_Err_return_comm (libmpi.so.12 + 0x33f04d) #1 0x00007fe2b36ce6c0 PMPI_Init (libmpi.so.12 + 0x4ce6c0) #2 0x000000000040255b main (osu_bw + 0x255b) #3 0x00007fe2b2a2924e __libc_start_call_main (libc.so.6 + 0x2924e) #4 0x00007fe2b2a29309 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29309) #5 0x0000000000403425 _start (osu_bw + 0x3425) ELF object binary architecture: AMD x86-64 Jun 15 08:30:39 xeon01 systemd[1]: systemd-coredump@1-48682-0.service: Deactivated successfully. ``` And also shows some error in stderr: ``` + export LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib + LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib + export I_MPI_DEBUG=5 + I_MPI_DEBUG=5 + export I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmix.so + I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmix.so + prog='result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple' + srun -N 2 result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple MPIR_pmi_virtualization(): MPI startup(): Error: Could not auto detect PMI version. Please set I_MPI_PMI variable MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmix.so Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(176): MPID_Init(1437)......: MPIR_pmi_init(109)...: (unknown)(): Other MPI error MPIR_pmi_virtualization(): MPI startup(): Error: Could not auto detect PMI version. Please set I_MPI_PMI variable MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmix.so Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(176): MPID_Init(1437)......: MPIR_pmi_init(109)...: (unknown)(): Other MPI error srun: error: xeon01: task 0: Segmentation fault (core dumped) srun: error: xeon02: task 1: Segmentation fault (core dumped) ``` # PMI_LIBRARY set to libpmi2.so With libpmi2.so causes a authentication problem with munge: ``` Jun 15 08:49:27 xeon01 slurmd[17717]: launch task StepId=289.0 request from UID:1880 GID:564 HOST:10.0.40.7 PORT:59170 Jun 15 08:49:27 xeon01 munged[1864]: Unauthorized credential for client UID=0 GID=0 ``` And freezes: ``` ./run.sh + export LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib + LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib + export I_MPI_DEBUG=5 + I_MPI_DEBUG=5 + export I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi2.so + I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi2.so + prog='result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple' + srun -N 2 result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi2.so MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi2.so [0] MPI startup(): Intel(R) MPI Library, Version 2021.9 Build 20230307 (id: d82b3071db) [0] MPI startup(): Copyright (C) 2003-2023 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [0] MPI startup(): libfabric version: 1.17.1 [0] MPI startup(): libfabric provider: psm2 [0] MPI startup(): File "/tuning_skx_shm-ofi_psm2.dat" not found [0] MPI startup(): Load tuning file: "/tuning_skx_shm-ofi.dat" [0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found [0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found ``` # PMI_LIBRARY set to libpmi.so With libpmi.so also causes the same credential problem: ``` Jun 15 08:50:41 xeon01 slurmd[17717]: launch task StepId=290.0 request from UID:1880 GID:564 HOST:10.0.40.7 PORT:34792 Jun 15 08:50:41 xeon01 munged[1864]: Unauthorized credential for client UID=0 GID=0 Jun 15 08:50:45 xeon01 slurmstepd[49043]: [290.0] done with job ``` But it manages to run and works okay: ``` hut$ ./run.sh + export LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib + LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib + export I_MPI_DEBUG=5 + I_MPI_DEBUG=5 + export I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi.so + I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi.so + prog='result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple' + srun -N 2 result/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi.so MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi.so [0] MPI startup(): Intel(R) MPI Library, Version 2021.9 Build 20230307 (id: d82b3071db) [0] MPI startup(): Copyright (C) 2003-2023 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [0] MPI startup(): libfabric version: 1.17.1 [0] MPI startup(): libfabric provider: psm2 [0] MPI startup(): File "/tuning_skx_shm-ofi_psm2.dat" not found [0] MPI startup(): Load tuning file: "/tuning_skx_shm-ofi.dat" [0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found [0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found [0] MPI startup(): File "" not found [0] MPI startup(): Unable to read tuning file for ch4 level [0] MPI startup(): File "" not found [0] MPI startup(): Unable to read tuning file for net level [0] MPI startup(): File "" not found [0] MPI startup(): Unable to read tuning file for shm level [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 49051 xeon01 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55 [0] MPI startup(): 1 31289 xeon02 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55 [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_PMI_LIBRARY=/nix/store/qvszg11q8zvihk4adki7xck425v7fk89-pmix-3.2.4/lib/libpmi.so # OSU MPI Bandwidth Test v7.1 # Size Bandwidth (MB/s) # Datatype: MPI_CHAR. 1 1.22 2 2.73 4 5.61 8 11.38 16 21.85 32 43.34 64 86.54 128 174.80 256 334.05 512 628.00 1024 1022.63 2048 1459.20 4096 1996.50 8192 2424.95 16384 2802.75 32768 2997.32 65536 7643.60 131072 6716.45 262144 6780.43 524288 7462.88 1048576 8100.58 2097152 8459.38 4194304 8496.18 ```
rarias commented 2023-06-15 09:52:46 +02:00 (Migrated from pm.bsc.es)
Copy Link

The pmix library is very old in nixpkgs (3.2.4) while the last stable release is 4.2.3. Let's try with the last release of pmix and see if Intel MPI/SLURM works with it.

The pmix library is very old in nixpkgs (3.2.4) while the last stable release is 4.2.3. Let's try with the last release of pmix and see if Intel MPI/SLURM works with it.
rarias commented 2023-06-15 10:44:54 +02:00 (Migrated from pm.bsc.es)
Copy Link

PMIX fails to build when fetched from the tag, using the release tarball.

PMIX fails to build when fetched from the tag, using the release tarball.
rarias commented 2023-06-15 18:37:16 +02:00 (Migrated from pm.bsc.es)
Copy Link

Updating PMIX doesn't solve the problem. Intel MPI seems to be unable to work with pmix, but it works with pmi2. The results are in the following table

MPI library --mpi=pmi2 --mpi=pmix
Intel MPI 2021.9 OK SEGFAULT
MPICH 4.1.1 ERROR OK
OpenMPI 4.1.5 ERROR OK

With pmi2 the ranks don't see each other in the case of MPICH and OpenMPI. With pmix, Intel segfaults.

Updating PMIX doesn't solve the problem. Intel MPI seems to be unable to work with pmix, but it works with pmi2. The results are in the following table | MPI library | --mpi=pmi2 | --mpi=pmix | | ----------------- | ---------- | ---------- | | Intel MPI 2021.9 | OK | SEGFAULT | | MPICH 4.1.1 | ERROR | OK | | OpenMPI 4.1.5 | ERROR | OK | With pmi2 the ranks don't see each other in the case of MPICH and OpenMPI. With pmix, Intel segfaults.
rarias commented 2023-06-15 18:57:02 +02:00 (Migrated from pm.bsc.es)
Copy Link

Okey, so there is more to it.

There are at least 3 things that must align for it to work:

  1. The client side plugin of SLURM, selected by --mpi=xxx.
  2. The library loaded by MPI which exposes a PMI API.
  3. The protocol used to communicate with the SLURM client PMI.

I was always changing the protocol when selecting the pmi2 library provided by pmix. However, it seems that the libpmi2.so provided by pmix will use the pmix protocol, so it must use --mpi=pmix. Or at least that is what I understand following the long ticket at https://bugs.schedmd.com/show_bug.cgi?id=6418

Let's try loading the emulation pmi2 library from pmix in intel MPI with --pmi=pmix.

Okey, so there is more to it. There are at least 3 things that must align for it to work: 1. The client side plugin of SLURM, selected by --mpi=xxx. 2. The library loaded by MPI which exposes a PMI API. 3. The protocol used to communicate with the SLURM client PMI. I was always changing the protocol when selecting the pmi2 library provided by pmix. However, it seems that the libpmi2.so provided by pmix will use the pmix protocol, so it must use --mpi=pmix. Or at least that is what I understand following the long ticket at https://bugs.schedmd.com/show_bug.cgi?id=6418 Let's try loading the emulation pmi2 library from pmix in intel MPI with --pmi=pmix.
rarias commented 2023-06-15 19:01:05 +02:00 (Migrated from pm.bsc.es)
Copy Link

Dammit, it fails with a glibc mismatch:

     32772:     /nix/store/8xk4yl1r3n6kbyn05qhan7nbag7npymx-glibc-2.35-224/lib/libc.so.6: error: version lookup error: version `GLIBC_2.36' not found (required by /nix/store/jsl215rvh31180z
0gzc5vwy0ih75ih1d-libevent-2.1.12/lib/libevent_core-2.1.so.7) (fatal)
MPIR_pmi_virtualization(): MPI startup(): Error: Could not open /nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi2.so
     51569:     /nix/store/8xk4yl1r3n6kbyn05qhan7nbag7npymx-glibc-2.35-224/lib/libc.so.6: error: version lookup error: version `GLIBC_2.36' not found (required by /nix/store/jsl215rvh31180z
0gzc5vwy0ih75ih1d-libevent-2.1.12/lib/libevent_core-2.1.so.7) (fatal)
Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(176):
MPID_Init(1437)......:
Dammit, it fails with a glibc mismatch: ``` 32772: /nix/store/8xk4yl1r3n6kbyn05qhan7nbag7npymx-glibc-2.35-224/lib/libc.so.6: error: version lookup error: version `GLIBC_2.36' not found (required by /nix/store/jsl215rvh31180z 0gzc5vwy0ih75ih1d-libevent-2.1.12/lib/libevent_core-2.1.so.7) (fatal) MPIR_pmi_virtualization(): MPI startup(): Error: Could not open /nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi2.so 51569: /nix/store/8xk4yl1r3n6kbyn05qhan7nbag7npymx-glibc-2.35-224/lib/libc.so.6: error: version lookup error: version `GLIBC_2.36' not found (required by /nix/store/jsl215rvh31180z 0gzc5vwy0ih75ih1d-libevent-2.1.12/lib/libevent_core-2.1.so.7) (fatal) Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(176): MPID_Init(1437)......: ```
rarias commented 2023-06-15 19:07:28 +02:00 (Migrated from pm.bsc.es)
Copy Link

Now it loads and gets stuck:

./run.sh pmi2 impi
+ pmix3=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4
+ pmix4=/nix/store/r4s08h1f92c08kxnnd2d14lhdy2q26p0-pmix-4.2.3
+ pmix=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4
+ export I_MPI_DEBUG=5
+ I_MPI_DEBUG=5
+ case $1 in
+ export I_MPI_PMI_LIBRARY=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi2.so
+ I_MPI_PMI_LIBRARY=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi2.so
+ export I_MPI_PMI=pmi2
+ I_MPI_PMI=pmi2
+ osu_impi=/nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1
+ osu_mpich=/nix/store/13r5jsxlnam8bi856fvyr52987n8sbfp-osu-micro-benchmarks-7.1-1
+ osu_openmpi=/nix/store/pk42wwb262irvc490hnvfl9wvjykgdg8-osu-micro-benchmarks-7.1-1
+ case $2 in
+ osu=/nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1
+ prog='/nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple'
+ srun -N 2 /nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi2.so
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi2.so
[0] MPI startup(): Intel(R) MPI Library, Version 2021.9  Build 20230307 (id: d82b3071db)
[0] MPI startup(): Copyright (C) 2003-2023 Intel Corporation.  All rights reserved.
[0] MPI startup(): library kind: release
[0] MPI startup(): libfabric version: 1.17.1
[0] MPI startup(): libfabric provider: psm2
[0] MPI startup(): File "/tuning_skx_shm-ofi_psm2.dat" not found
[0] MPI startup(): Load tuning file: "/tuning_skx_shm-ofi.dat"
[0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found
[0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found
Now it loads and gets stuck: ``` ./run.sh pmi2 impi + pmix3=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4 + pmix4=/nix/store/r4s08h1f92c08kxnnd2d14lhdy2q26p0-pmix-4.2.3 + pmix=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4 + export I_MPI_DEBUG=5 + I_MPI_DEBUG=5 + case $1 in + export I_MPI_PMI_LIBRARY=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi2.so + I_MPI_PMI_LIBRARY=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi2.so + export I_MPI_PMI=pmi2 + I_MPI_PMI=pmi2 + osu_impi=/nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1 + osu_mpich=/nix/store/13r5jsxlnam8bi856fvyr52987n8sbfp-osu-micro-benchmarks-7.1-1 + osu_openmpi=/nix/store/pk42wwb262irvc490hnvfl9wvjykgdg8-osu-micro-benchmarks-7.1-1 + case $2 in + osu=/nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1 + prog='/nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple' + srun -N 2 /nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi2.so MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi2.so [0] MPI startup(): Intel(R) MPI Library, Version 2021.9 Build 20230307 (id: d82b3071db) [0] MPI startup(): Copyright (C) 2003-2023 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [0] MPI startup(): libfabric version: 1.17.1 [0] MPI startup(): libfabric provider: psm2 [0] MPI startup(): File "/tuning_skx_shm-ofi_psm2.dat" not found [0] MPI startup(): Load tuning file: "/tuning_skx_shm-ofi.dat" [0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found [0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found ```
rarias commented 2023-06-15 19:15:46 +02:00 (Migrated from pm.bsc.es)
Copy Link

Now it looks that the problem is caused because libfabric cannot find the psm2 lib:


1326 openat(AT_FDCWD, "/nix/store/8xk4yl1r3n6kbyn05qhan7nbag7npymx-glibc-2.35-224/lib/libefa-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1327 openat(AT_FDCWD, "/nix/store/shasq3azl2298vqkvq5mc7vivdqp3yrj-gcc-12.2.0-lib/lib/libefa-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1328 openat(AT_FDCWD, "/nix/store/1nyg1fvhpz8bx3vn3r9f18zhra2rpbx9-glibc-2.37-8/lib/libefa-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1329 openat(AT_FDCWD, "/nix/store/kni8lkj9hnyfvrnag648f35misbi5gja-xgcc-12.2.0-libgcc/lib/libefa-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1330 openat(AT_FDCWD, "/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1331 openat(AT_FDCWD, "/nix/store/dxrkp7wipdcbbm0r2bb6bx1v5vqyclmv-util-linux-minimal-2.38.1-lib/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1332 openat(AT_FDCWD, "/nix/store/kkphyn9m9z1wvvbcrdv0c3v9dahyv494-numactl-2.0.16/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1333 openat(AT_FDCWD, "/nix/store/ifzg3dwg4zwi9wa8wb2s9h1ks6jgrcb7-libpsm2-11.2.230/lib64/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1334 openat(AT_FDCWD, "/nix/store/8xk4yl1r3n6kbyn05qhan7nbag7npymx-glibc-2.35-224/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1335 openat(AT_FDCWD, "/nix/store/shasq3azl2298vqkvq5mc7vivdqp3yrj-gcc-12.2.0-lib/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1336 openat(AT_FDCWD, "/nix/store/1nyg1fvhpz8bx3vn3r9f18zhra2rpbx9-glibc-2.37-8/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1337 openat(AT_FDCWD, "/nix/store/kni8lkj9hnyfvrnag648f35misbi5gja-xgcc-12.2.0-libgcc/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1338 openat(AT_FDCWD, "/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib/libopx-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1339 openat(AT_FDCWD, "/nix/store/dxrkp7wipdcbbm0r2bb6bx1v5vqyclmv-util-linux-minimal-2.38.1-lib/lib/libopx-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1340 openat(AT_FDCWD, "/nix/store/kkphyn9m9z1wvvbcrdv0c3v9dahyv494-numactl-2.0.16/lib/libopx-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1341 openat(AT_FDCWD, "/nix/store/ifzg3dwg4zwi9wa8wb2s9h1ks6jgrcb7-libpsm2-11.2.230/lib64/libopx-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
Now it looks that the problem is caused because libfabric cannot find the psm2 lib: ``` 1326 openat(AT_FDCWD, "/nix/store/8xk4yl1r3n6kbyn05qhan7nbag7npymx-glibc-2.35-224/lib/libefa-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1327 openat(AT_FDCWD, "/nix/store/shasq3azl2298vqkvq5mc7vivdqp3yrj-gcc-12.2.0-lib/lib/libefa-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1328 openat(AT_FDCWD, "/nix/store/1nyg1fvhpz8bx3vn3r9f18zhra2rpbx9-glibc-2.37-8/lib/libefa-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1329 openat(AT_FDCWD, "/nix/store/kni8lkj9hnyfvrnag648f35misbi5gja-xgcc-12.2.0-libgcc/lib/libefa-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1330 openat(AT_FDCWD, "/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1331 openat(AT_FDCWD, "/nix/store/dxrkp7wipdcbbm0r2bb6bx1v5vqyclmv-util-linux-minimal-2.38.1-lib/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1332 openat(AT_FDCWD, "/nix/store/kkphyn9m9z1wvvbcrdv0c3v9dahyv494-numactl-2.0.16/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1333 openat(AT_FDCWD, "/nix/store/ifzg3dwg4zwi9wa8wb2s9h1ks6jgrcb7-libpsm2-11.2.230/lib64/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1334 openat(AT_FDCWD, "/nix/store/8xk4yl1r3n6kbyn05qhan7nbag7npymx-glibc-2.35-224/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1335 openat(AT_FDCWD, "/nix/store/shasq3azl2298vqkvq5mc7vivdqp3yrj-gcc-12.2.0-lib/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1336 openat(AT_FDCWD, "/nix/store/1nyg1fvhpz8bx3vn3r9f18zhra2rpbx9-glibc-2.37-8/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1337 openat(AT_FDCWD, "/nix/store/kni8lkj9hnyfvrnag648f35misbi5gja-xgcc-12.2.0-libgcc/lib/libpsm2-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1338 openat(AT_FDCWD, "/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib/libopx-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1339 openat(AT_FDCWD, "/nix/store/dxrkp7wipdcbbm0r2bb6bx1v5vqyclmv-util-linux-minimal-2.38.1-lib/lib/libopx-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1340 openat(AT_FDCWD, "/nix/store/kkphyn9m9z1wvvbcrdv0c3v9dahyv494-numactl-2.0.16/lib/libopx-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1341 openat(AT_FDCWD, "/nix/store/ifzg3dwg4zwi9wa8wb2s9h1ks6jgrcb7-libpsm2-11.2.230/lib64/libopx-fi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) ```
rarias commented 2023-06-15 20:17:07 +02:00 (Migrated from pm.bsc.es)
Copy Link

After fixing the psm2 problem (LD_LIBRARY_PATH set to libfabric), it gets stuck after configuring the network devices. However, I managed to get it working with the libpmi.so of PMIX:

+ pmix3=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4
+ pmix4=/nix/store/r4s08h1f92c08kxnnd2d14lhdy2q26p0-pmix-4.2.3
+ pmix=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4
+ export LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib
+ LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib
+ case $1 in
+ export I_MPI_PMI_LIBRARY=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi.so
+ I_MPI_PMI_LIBRARY=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi.so
+ osu_impi=/nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1
+ osu_mpich=/nix/store/13r5jsxlnam8bi856fvyr52987n8sbfp-osu-micro-benchmarks-7.1-1
+ osu_openmpi=/nix/store/pk42wwb262irvc490hnvfl9wvjykgdg8-osu-micro-benchmarks-7.1-1
+ case $2 in
+ osu=/nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1
+ prog='/nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple'
+ srun -N 2 /nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi.so
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi.so
[0] MPI startup(): Intel(R) MPI Library, Version 2021.9  Build 20230307 (id: d82b3071db)
[0] MPI startup(): Copyright (C) 2003-2023 Intel Corporation.  All rights reserved.
[0] MPI startup(): library kind: release
[0] MPI startup(): libfabric version: 1.17.1
[0] MPI startup(): libfabric provider: psm2
[0] MPI startup(): File "/tuning_skx_shm-ofi_psm2.dat" not found
[0] MPI startup(): Load tuning file: "/tuning_skx_shm-ofi.dat"
[0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found
[0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found
[0] MPI startup(): File "" not found
[0] MPI startup(): Unable to read tuning file for ch4 level
[0] MPI startup(): File "" not found
[0] MPI startup(): Unable to read tuning file for net level
[0] MPI startup(): File "" not found
[0] MPI startup(): Unable to read tuning file for shm level
[0] MPI startup(): Rank    Pid      Node name  Pin cpu
[0] MPI startup(): 0       52409    xeon01     0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55
[0] MPI startup(): 1       33607    xeon02     0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_PMI_LIBRARY=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi.so
# OSU MPI Bandwidth Test v7.1
# Size      Bandwidth (MB/s)
# Datatype: MPI_CHAR.
1                       1.48
2                       2.95
4                       5.96
8                      11.91
16                     23.32
32                     44.79
64                     89.76
128                   185.47
256                   356.75
512                   609.92
1024                  982.74
2048                 1518.42
4096                 2019.92
8192                 2507.55
16384                2885.70
32768                3069.55
65536                7753.12
131072               6861.19
262144               7435.15
524288               7683.38
1048576              8249.30
2097152              8450.11
4194304              8494.06

This looks promising but it will require user intervention to set the I_MPI_PMI_LIBRARY.

After fixing the psm2 problem (LD_LIBRARY_PATH set to libfabric), it gets stuck after configuring the network devices. However, I managed to get it working with the libpmi.so of PMIX: ``` + pmix3=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4 + pmix4=/nix/store/r4s08h1f92c08kxnnd2d14lhdy2q26p0-pmix-4.2.3 + pmix=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4 + export LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib + LD_LIBRARY_PATH=/nix/store/gq0zlm29v0vyxl2sadak5cir2rc3nrlm-libfabric-1.17.1/lib + case $1 in + export I_MPI_PMI_LIBRARY=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi.so + I_MPI_PMI_LIBRARY=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi.so + osu_impi=/nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1 + osu_mpich=/nix/store/13r5jsxlnam8bi856fvyr52987n8sbfp-osu-micro-benchmarks-7.1-1 + osu_openmpi=/nix/store/pk42wwb262irvc490hnvfl9wvjykgdg8-osu-micro-benchmarks-7.1-1 + case $2 in + osu=/nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1 + prog='/nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple' + srun -N 2 /nix/store/k75slazpswhsfg7j88vbbdnrl7yi0wl4-osu-micro-benchmarks-7.1-1/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw -b multiple MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi.so MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to /nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi.so [0] MPI startup(): Intel(R) MPI Library, Version 2021.9 Build 20230307 (id: d82b3071db) [0] MPI startup(): Copyright (C) 2003-2023 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [0] MPI startup(): libfabric version: 1.17.1 [0] MPI startup(): libfabric provider: psm2 [0] MPI startup(): File "/tuning_skx_shm-ofi_psm2.dat" not found [0] MPI startup(): Load tuning file: "/tuning_skx_shm-ofi.dat" [0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found [0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found [0] MPI startup(): File "" not found [0] MPI startup(): Unable to read tuning file for ch4 level [0] MPI startup(): File "" not found [0] MPI startup(): Unable to read tuning file for net level [0] MPI startup(): File "" not found [0] MPI startup(): Unable to read tuning file for shm level [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 52409 xeon01 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55 [0] MPI startup(): 1 33607 xeon02 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55 [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_PMI_LIBRARY=/nix/store/qi5l47if5lw75g78zbffg8pipqswh725-pmix-3.2.4/lib/libpmi.so # OSU MPI Bandwidth Test v7.1 # Size Bandwidth (MB/s) # Datatype: MPI_CHAR. 1 1.48 2 2.95 4 5.96 8 11.91 16 23.32 32 44.79 64 89.76 128 185.47 256 356.75 512 609.92 1024 982.74 2048 1518.42 4096 2019.92 8192 2507.55 16384 2885.70 32768 3069.55 65536 7753.12 131072 6861.19 262144 7435.15 524288 7683.38 1048576 8249.30 2097152 8450.11 4194304 8494.06 ``` This looks promising but it will require user intervention to set the I_MPI_PMI_LIBRARY.
rarias commented 2023-06-15 20:19:28 +02:00 (Migrated from pm.bsc.es)
Copy Link

By the way, all tests are running the osu_bw -b multiple test, with multiple messages at the same time. With the sequential mode we reach the theoretical limit:

# OSU MPI Bandwidth Test v7.1
# Size      Bandwidth (MB/s)
# Datatype: MPI_CHAR.
1                       1.47
2                       2.91
4                       5.85
8                      11.70
16                     22.80
32                     45.64
64                     90.98
128                   182.37
256                   359.03
512                   685.10
1024                 1168.60
2048                 1647.19
4096                 2278.74
8192                 3092.25
16384                3477.26
32768                3693.22
65536                7622.61
131072              11212.50
262144              12023.91
524288              12179.48
1048576             12317.67
2097152             12297.12
4194304             12380.94
By the way, all tests are running the `osu_bw -b multiple` test, with multiple messages at the same time. With the sequential mode we reach the theoretical limit: ``` # OSU MPI Bandwidth Test v7.1 # Size Bandwidth (MB/s) # Datatype: MPI_CHAR. 1 1.47 2 2.91 4 5.85 8 11.70 16 22.80 32 45.64 64 90.98 128 182.37 256 359.03 512 685.10 1024 1168.60 2048 1647.19 4096 2278.74 8192 3092.25 16384 3477.26 32768 3693.22 65536 7622.61 131072 11212.50 262144 12023.91 524288 12179.48 1048576 12317.67 2097152 12297.12 4194304 12380.94 ```
rarias commented 2023-06-16 17:11:56 +02:00 (Migrated from pm.bsc.es)
Copy Link

mentioned in merge request !11

mentioned in merge request !11
rarias commented 2023-06-16 17:31:55 +02:00 (Migrated from pm.bsc.es)
Copy Link

Let's go with PMIx then.

Let's go with PMIx then.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
ceph

   
ci

   
config

   
hw

   
io

   
kernel

   
mpi

   
net

   
nix

   
ops
No Label
ceph
ci
config
hw
io
kernel
mpi
net
nix
ops
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: rarias/jungle#16
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?