114 Commits

Author SHA1 Message Date
35b4a30f2e Move slurm client in a separate module
Reviewed-by: Rodrigo Arias Mallo <rodrigo.arias@bsc.es>
2025-10-01 16:40:16 +02:00
bfc5555caf Enable public-inbox at jungle.bsc.es/lists
The public-inbox service fetches emails from the sourcehut mailing lists
and displays them on the web. The idea is to reduce the dependency on
external services and add a secondary storage for the mailing lists in
case sourcehut goes down or changes the current free plans.

The service is available in https://jungle.bsc.es/lists/ and is open to
the public. It currently mirrors the bscpkgs and jungle mailing list.

We also edited the CSS to improve the readability and have larger fonts
by default.

The service for public-inbox produced by NixOS is not well configured to
fetch emails from an IMAP mail server, so we also manually edit the
service file to enable the network.

Reviewed-by: Aleix Roca Nonell <aleix.rocanonell@bsc.es>
2025-10-01 16:40:16 +02:00
f88ad644c6 Monitor https://pm.bsc.es/gitlab/ too
The GitLab instance is in the /gitlab endpoint and may fail
independently of https://pm.bsc.es/.

Cc: Víctor López <victor.lopez@bsc.es>
Reviewed-by: Aleix Roca Nonell <aleix.rocanonell@bsc.es>
2025-10-01 16:40:16 +02:00
5003139e8e Enable nixseparatedebuginfod module
The module is only enabled on Hut and Eudy because we noticed activity
on the debuginfod service even if no debug session was active.

Reviewed-by: Rodrigo Arias Mallo <rodrigo.arias@bsc.es>
2025-10-01 16:40:16 +02:00
845adfc937 Use tmpfs in /tmp
The /tmp directory was using the SSD disk which is not erased across
boots. Nix will use /tmp to perform the builds, so we want it to be as
fast as possible. In general, all the machines have enough space to
handle large builds like LLVM.

Reviewed-by: Aleix Roca Nonell <aleix.rocanonell@bsc.es>
2025-10-01 16:40:16 +02:00
52b4cba900 Enable runners for pm.bsc.es/gitlab too
The old runners for the PM gitlab were disabled in configuration in the
last outage, but they remained working until we reboot the node. With
this change we enable the runners for both PM and gitlab.bsc.es.

Reviewed-by: Aleix Roca Nonell <aleix.rocanonell@bsc.es>
2025-10-01 16:40:16 +02:00
7b22865a1e Remove complete ceph package from hut
Only the ceph-client is needed.

Reviewed-by: Aleix Roca Nonell <aleix.rocanonell@bsc.es>
2025-10-01 16:40:16 +02:00
34628c0e39 BSC packages are no longer in bsc attribute
Reviewed-by: Aleix Roca Nonell <aleix.rocanonell@bsc.es>
2025-10-01 16:40:16 +02:00
5d1f008199 Monitor anella instead of gw.bsc.es
The target gw.bsc.es doesn't reply to our ICMP probes from hut. However,
the anella hop in the tracepath is a good candidate to identify cuts
between the login and the provider and between the provider and external
hosts like Google or Cloudflare DNS.

Reviewed-By: Aleix Roca Nonell <aleix.rocanonell@bsc.es>
2025-10-01 16:40:16 +02:00
f9622b19ef Add ICMP probes
These probes check if we can reach several targets via ICMP, which is
not proxied, so they can be used to see if ICMP forwarding is working in
the login node.

In particular, we test if we can reach the Google (8.8.8.8) and
Cloudflare (1.1.1.1) DNS servers, the BSC gateway which responds to ping
only from the intranet and the login node (ssfhead).

Reviewed-By: Aleix Roca Nonell <aleix.rocanonell@bsc.es>
2025-10-01 16:40:16 +02:00
f2d26fd2e2 Enable proxy for Grafana too
The alerts need to contact the slack endpoint, so we add the proxy
environment variables to the grafana systemd service.

Reviewed-By: Aleix Roca Nonell <aleix.rocanonell@bsc.es>
2025-10-01 16:40:16 +02:00
bf8f0ac583 Make blackbox exporter use the proxy
By default it was trying to reach the targets using the default gateway,
but since the electrical cut of 2023-10-20, the login node has not
enabled forwarding again. So better if we don't rely on it.

Reviewed-By: Aleix Roca Nonell <aleix.rocanonell@bsc.es>
2025-10-01 16:40:16 +02:00
4e333dca21 Don't log SLURM connection attempts from ssfhead 2025-10-01 16:40:16 +02:00
33c1da6c40 Add docker runner too 2025-10-01 16:40:16 +02:00
5dc41a86e5 Monitor gitlab.bsc.es too 2025-10-01 16:40:16 +02:00
697c3d884e Monitor PM webpage via blackbox 2025-10-01 16:40:16 +02:00
5a537c7478 Temporarily disable pm runners 2025-10-01 16:40:16 +02:00
c06b706e49 Add runner for gitlab.bsc.es 2025-10-01 16:40:16 +02:00
270cff123d Allow anonymous access to grafana 2025-10-01 16:40:16 +02:00
b219badaaf Remove user/group when using DynamicUsers 2025-10-01 16:40:16 +02:00
f4fcb7c72c Set the SLURM_CONF variable 2025-10-01 16:40:16 +02:00
b4ede66387 Enable slurm-exporter service 2025-10-01 16:40:16 +02:00
63d63fd39a Mount the hut nix store for SLURM jobs 2025-10-01 16:40:16 +02:00
beae9d240e Enable direnv integration 2025-10-01 16:40:16 +02:00
e925b00489 Remove bscpkgs from the registry and nixPath
This is done to prevent accidental evaluations where the nixpkgs input
of bscpkgs is still pointing to a different version that the one
specified in the jungle flake. Instead use jungle#bscpkgs.X to get a
package from bscpkgs.
2025-10-01 16:40:16 +02:00
87871de141 Don't fetch registry flakes from the net 2025-10-01 16:40:16 +02:00
f5a6055f21 Open ports in firewall of compute nodes 2025-10-01 16:40:16 +02:00
00068cb11c Monitor storage nodes via IPMI too 2025-10-01 16:40:16 +02:00
a992b266bb Enable fstrim service 2025-10-01 16:40:16 +02:00
c26cff7bdb Serve the nix store from hut 2025-10-01 16:40:16 +02:00
1b5469af13 Add encrypted munge key with agenix 2025-10-01 16:40:16 +02:00
78c883a274 Remove unused large port hole in firewall 2025-10-01 16:40:16 +02:00
3385252f5f Make exporters listen in localhost only 2025-10-01 16:40:16 +02:00
241b888a7c Allow only some ports for srun 2025-10-01 16:40:16 +02:00
b7aba3d15c Block ssfhead from reaching our slurm daemon 2025-10-01 16:40:16 +02:00
e35b51cd00 Poweroff idle slurm nodes after 1 hour 2025-10-01 16:40:16 +02:00
2e460f49bd Add IB and IPMI node host names 2025-10-01 16:40:16 +02:00
ac3817d99b Unlock ovni gitlab runners 2025-10-01 16:40:16 +02:00
e7aa2d3fe3 Add agenix to all nodes 2025-10-01 16:40:16 +02:00
d2860ce437 Add agenix module to ceph 2025-10-01 16:40:16 +02:00
1f199c73f1 Remove old secrets 2025-10-01 16:40:16 +02:00
657a1b328a Mount /ceph in owl1 and owl2 2025-10-01 16:40:16 +02:00
875e6fe6c7 Warn about the owl2 omnipath device 2025-10-01 16:40:16 +02:00
7abee55da4 Clean owl2 configuration 2025-10-01 16:40:16 +02:00
758ddc71cb Move the ceph client config to an external module 2025-10-01 16:40:16 +02:00
224bafd20d Reorganize secrets and ssh keys
The agenix tools needs to read the secrets from a standalone file, but
we also need the same information for the SSH keys.
2025-10-01 16:40:16 +02:00
8b1fa938ea Add anavarro user 2025-10-01 16:40:16 +02:00
94b110dc57 Set zsh inc_append_history option 2025-10-01 16:40:16 +02:00
0c5207bd2d Set zsh shell for rarias 2025-10-01 16:40:16 +02:00
4200e6162d Enable zsh and fix key bindings 2025-10-01 16:40:16 +02:00