rarias/ovni
rarias/ovni
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Make sure we detect crashes in MP tests

Browse Source 
If a process fails in the background, bash ignores it even when running
with set -e. To address it, we store a token in a file per each rank,
which only gets written when each process finishes properly. Then we
make sure that each rank has completed successfully, otherwise we fail
the test.
This commit is contained in:
Rodrigo Arias 2025-07-23 17:05:06 +02:00
parent e4f6238207
commit 23faac0c1b
1 changed files with 16 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

18
test/ovni-driver.sh
View File

@ -1,5 +1,5 @@
#!/bin/sh #!/bin/sh
# Copyright (c) 2021-2023 Barcelona Supercomputing Center (BSC) # Copyright (c) 2021-2025 Barcelona Supercomputing Center (BSC)
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later
set -ex set -ex
@ -30,10 +30,24 @@ if [ -n "$OVNI_DRIVER" ]; then
else else
if [ "$OVNI_NPROCS" -gt 1 ]; then if [ "$OVNI_NPROCS" -gt 1 ]; then
for i in $(seq 1 "$OVNI_NPROCS"); do for i in $(seq 1 "$OVNI_NPROCS"); do
rank=$(($i-1))
# Run the test in the background # Run the test in the background
OVNI_RANK=$(($i-1)) OVNI_NRANKS=$OVNI_NPROCS "$testname" & (OVNI_RANK=$rank OVNI_NRANKS=$OVNI_NPROCS "$testname" && echo ok > ovni.rc.$rank) &
done done
wait wait
all_good=1
for i in $(seq 1 "$OVNI_NPROCS"); do
rank=$(($i-1))
# Run the test in the background
if ! grep -q ok "ovni.rc.$rank"; then
echo "Test for rank $rank failed"
all_good=0
fi
done
if [ "$all_good" != 1 ]; then
echo "Some tests failed"
exit 1
fi
else else
"$testname" "$testname"
fi fi