Compare commits
1 Commits
212f405848
...
maintenanc
| Author | SHA1 | Date | |
|---|---|---|---|
| e8d7ae345d |
156
doc/2025-05-maintenance-purchase.md
Normal file
156
doc/2025-05-maintenance-purchase.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# Maintenance purchase 2025-05
|
||||
|
||||
We need to buy some components to replace broken parts or to have spare ones for
|
||||
when they break. We also need some tools to do basic repairs.
|
||||
|
||||
Here is the list:
|
||||
|
||||
- 11 x Power supply DELTA DPS-750XB A (700 W) (this is critical)
|
||||
- 57.69€/unit, 634.59€ total <https://es.aliexpress.com/item/1005004090017186.html>
|
||||
|
||||
- 8 x RAM DDR4 2400MHz PC4-19200 ECC Registered
|
||||
- 128.85€/pair, 515.40€ total <https://www.amazon.es/PC4-19200-REGISTERED-MEMORY-WORKSTATIONS-MOTHERBOARDS/dp/B06W9P3RKF>
|
||||
|
||||
- 1 x Set of screwdrivers
|
||||
- 23.99€ <https://www.amazon.es/BLOSTM-Juego-Destornilladores-Profesionales-Destornillador/dp/B09W9R8J3S>
|
||||
|
||||
- 1 x UART adaptor
|
||||
- 14.99€ <https://www.amazon.es/DSD-TECH-SH-U09C5-convertidor-Soporte/dp/B07WX2DSVB>
|
||||
|
||||
- 1 x SSD SATA disk of 2 TB
|
||||
- 135.99€ <https://www.amazon.es/Crucial-BX500-pulgadas-interno-CT2000BX500SSD101/dp/B0CCN9QWKT>
|
||||
|
||||
Total: 1324.96 €
|
||||
|
||||
# Rationale
|
||||
|
||||
Below is the search procedure I followed to come up with that list.
|
||||
|
||||
## Power supplies
|
||||
|
||||
They are the first components to fail. We already have some problems with the
|
||||
monitoring of some power supplies. They will soon stop being manufactured, so we
|
||||
should increase out stack.
|
||||
|
||||
Most Xeon nodes use the DELTA DPS-750XB A:
|
||||
|
||||
hut% sudo ipmitool fru
|
||||
...
|
||||
FRU Device Description : Pwr Supply 1 FRU (ID 2)
|
||||
Product Manufacturer : DELTA
|
||||
Product Name : DPS-750XB A
|
||||
Product Part Number : E98791-010
|
||||
Product Version : 05
|
||||
Product Serial : XXXXXXXXXXXXXXXXX
|
||||
|
||||
And we only have one per node. We should make the power supply redundant so we
|
||||
can tolerate it to fail without bringing down the node.
|
||||
|
||||
They are available on Amazon, but they are very expensive (287.54 €):
|
||||
|
||||
<https://www.amazon.es/DPS-750XB-E98791-010-alimentaci%C3%B3n-conmutada-Platinum/dp/B0DB65G4VT>
|
||||
|
||||
On Aliexpress they are much cheaper (57.69 €):
|
||||
|
||||
<https://es.aliexpress.com/item/1005004090017186.html>
|
||||
|
||||
We have 11 nodes plus the login, but I'm not able to figure out which power
|
||||
supply the login is using.
|
||||
|
||||
The login uses another one, AXX1100PCRPS, and only has one slot populated. We
|
||||
may want to also we another one, but I would need to reset the FRU and I don't
|
||||
have access to the login node. So I will leave this for Operations to deal with.
|
||||
We can live without the login if needed.
|
||||
|
||||
## RAM DIMM
|
||||
|
||||
The DIMM modules also experience errors, which are monitored by Linux. In some
|
||||
nodes we see non-recoverable errors that are no longer corrected by the ECC. We
|
||||
need to replace the bad modules.
|
||||
|
||||
Having two spare modules per node would be enough to cover most problems in the
|
||||
future.
|
||||
|
||||
> 16 GB, 2400 MHz RDIMM
|
||||
|
||||
The module from dmidecode:
|
||||
|
||||
Handle 0x0026, DMI type 17, 40 bytes
|
||||
Memory Device
|
||||
Array Handle: 0x0020
|
||||
Error Information Handle: Not Provided
|
||||
Total Width: 72 bits
|
||||
Data Width: 64 bits
|
||||
Size: 16 GB
|
||||
Form Factor: DIMM
|
||||
Set: None
|
||||
Locator: DIMM_B1
|
||||
Bank Locator: NODE 1
|
||||
Type: DDR4
|
||||
Type Detail: Synchronous
|
||||
Speed: 2400 MT/s
|
||||
Manufacturer: Micron
|
||||
Serial Number: XXXXXXXX
|
||||
Asset Tag:
|
||||
Part Number: 36ASF2G72PZ-2G3B1
|
||||
Rank: 2
|
||||
Configured Memory Speed: 2400 MT/s
|
||||
Minimum Voltage: Unknown
|
||||
Maximum Voltage: Unknown
|
||||
Configured Voltage: Unknown
|
||||
|
||||
Which is this module:
|
||||
|
||||
<https://www.amazon.com/Micron-PC4-19200-DDR4-2400MHz-Registered-MTA36ASF2G72PZ-2G3B1/dp/B01KBCNEGI>
|
||||
|
||||
But they have only one in stock. Here is more details:
|
||||
|
||||
> 16GB PC4-19200 DDR4-2400MHz
|
||||
|
||||
The must have the following features:
|
||||
|
||||
- 16 GB
|
||||
- DDR4
|
||||
- Speed at least 2400 MT/s
|
||||
- ECC
|
||||
- Registered
|
||||
- Best if from Micron
|
||||
|
||||
I would say having 8 spare modules would be enough for now, as we only have a
|
||||
few that are currently failing. We could upgrade the modules later, as they
|
||||
don't have much risk of stopping being manufactured like the power supplies.
|
||||
|
||||
These may work:
|
||||
|
||||
- 1 x 16GB, 69,11€ <https://www.amazon.es/PC4-19200-REGISTRADO-SERVIDORES-Estaciones-CHIPKILL/dp/B06X42HC9N>
|
||||
|
||||
- 2 x 16GB, 128,85€ <https://www.amazon.es/PC4-19200-REGISTERED-MEMORY-WORKSTATIONS-MOTHERBOARDS/dp/B06W9P3RKF>
|
||||
|
||||
It is cheaper to buy them by pairs, so let's use the last one.
|
||||
|
||||
## Screwdriver set
|
||||
|
||||
In order to change and replace the machine parts we need a set of screwdrivers.
|
||||
Instead of having to bring my own from home, I want to have one at BSC. These
|
||||
are enough and come in a nice box so I don't lose them:
|
||||
|
||||
<https://www.amazon.es/BLOSTM-Juego-Destornilladores-Profesionales-Destornillador/dp/B09W9R8J3S>
|
||||
|
||||
## Serial port adaptor
|
||||
|
||||
In order to debug problems with several components, we need to be able to plug
|
||||
to the serial port of the CPU. As we may deal with different voltages and
|
||||
pinouts, the most versatile option is to just be able to select the voltage and
|
||||
expose a pin interface.
|
||||
|
||||
This one would do:
|
||||
|
||||
<https://www.amazon.es/DSD-TECH-SH-U09C5-convertidor-Soporte/dp/B07WX2DSVB>
|
||||
|
||||
## Storage for raccoon
|
||||
|
||||
Given that we are currently using raccoon for builds too, we would need to
|
||||
increase its current storage. We only have available 270 GB, so we can benefit
|
||||
from another disk. Using 2 TiB would be plenty. This one seems enough:
|
||||
|
||||
- 135,99€ <https://www.amazon.es/Crucial-BX500-pulgadas-interno-CT2000BX500SSD101/dp/B0CCN9QWKT>
|
||||
5
keys.nix
5
keys.nix
@@ -9,12 +9,11 @@ rec {
|
||||
koro = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIImiTFDbxyUYPumvm8C4mEnHfuvtBY1H8undtd6oDd67 koro";
|
||||
bay = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICvGBzpRQKuQYHdlUQeAk6jmdbkrhmdLwTBqf3el7IgU bay";
|
||||
lake2 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINo66//S1yatpQHE/BuYD/Gfq64TY7ZN5XOGXmNchiO0 lake2";
|
||||
fox = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDwItIk5uOJcQEVPoy/CVGRzfmE1ojrdDcI06FrU4NFT fox";
|
||||
fox = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDa9lId4rB/EKGkkCCVOy0cuId2SYLs+8W8kx0kmpO1y fox";
|
||||
};
|
||||
|
||||
hostGroup = with hosts; rec {
|
||||
untrusted = [ fox ];
|
||||
compute = [ owl1 owl2 ];
|
||||
compute = [ owl1 owl2 fox ];
|
||||
playground = [ eudy koro ];
|
||||
storage = [ bay lake2 ];
|
||||
monitor = [ hut ];
|
||||
|
||||
@@ -2,9 +2,11 @@
|
||||
|
||||
{
|
||||
imports = [
|
||||
../common/base.nix
|
||||
../common/xeon/console.nix
|
||||
../common/xeon.nix
|
||||
../module/ceph.nix
|
||||
../module/emulation.nix
|
||||
../module/slurm-client.nix
|
||||
../module/slurm-firewall.nix
|
||||
];
|
||||
|
||||
# Select the this using the ID to avoid mismatches
|
||||
@@ -20,51 +22,11 @@
|
||||
hardware.cpu.intel.updateMicrocode = lib.mkForce false;
|
||||
|
||||
networking = {
|
||||
defaultGateway = "147.83.30.130";
|
||||
nameservers = [ "8.8.8.8" ];
|
||||
hostName = "fox";
|
||||
interfaces.enp1s0f0np0.ipv4.addresses = [
|
||||
{
|
||||
# BSC network (to be removed)
|
||||
address = "10.0.40.26";
|
||||
prefixLength = 24;
|
||||
}
|
||||
{
|
||||
# UPC network
|
||||
# Public IP configuration:
|
||||
# - Hostname: fox.ac.upc.edu
|
||||
# - IP: 147.83.30.141
|
||||
# - Gateway: 147.83.30.130
|
||||
# - NetMask: 255.255.255.192
|
||||
# Private IP configuration for BMC:
|
||||
# - Hostname: fox-ipmi.ac.upc.edu
|
||||
# - IP: 147.83.35.27
|
||||
# - Gateway: 147.83.35.2
|
||||
# - NetMask: 255.255.255.0
|
||||
address = "147.83.30.141";
|
||||
prefixLength = 26; # 255.255.255.192
|
||||
}
|
||||
];
|
||||
extraHosts = ''
|
||||
# Fox UPC
|
||||
147.83.30.141 fox.ac.upc.edu
|
||||
147.83.35.27 fox-ipmi.ac.upc.edu
|
||||
# Fox BSC
|
||||
10.0.40.26 fox
|
||||
10.0.40.126 fox-ipmi
|
||||
'' +
|
||||
# To be removed:
|
||||
''
|
||||
# Hut BSC
|
||||
10.0.40.7 hut
|
||||
'';
|
||||
|
||||
# To be removed:
|
||||
proxy = {
|
||||
default = "http://hut:23080/";
|
||||
noProxy = "127.0.0.1,localhost,internal.domain,10.0.40.40,hut";
|
||||
allProxy = null;
|
||||
};
|
||||
interfaces.enp1s0f0np0.ipv4.addresses = [ {
|
||||
address = "10.0.40.26";
|
||||
prefixLength = 24;
|
||||
} ];
|
||||
};
|
||||
|
||||
# Configure Nvidia driver to use with CUDA
|
||||
@@ -94,4 +56,20 @@
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
serviceConfig.ExecStart = script;
|
||||
};
|
||||
|
||||
# Only allow SSH connections from users who have a SLURM allocation
|
||||
# See: https://slurm.schedmd.com/pam_slurm_adopt.html
|
||||
security.pam.services.sshd.rules.account.slurm = {
|
||||
control = "required";
|
||||
enable = true;
|
||||
modulePath = "${pkgs.slurm}/lib/security/pam_slurm_adopt.so";
|
||||
args = [ "log_level=debug5" ];
|
||||
order = 999999; # Make it last one
|
||||
};
|
||||
|
||||
# Disable systemd session (pam_systemd.so) as it will conflict with the
|
||||
# pam_slurm_adopt.so module. What happens is that the shell is first adopted
|
||||
# into the slurmstepd task and then into the systemd session, which is not
|
||||
# what we want, otherwise it will linger even if all jobs are gone.
|
||||
security.pam.services.sshd.startSession = lib.mkForce false;
|
||||
}
|
||||
|
||||
@@ -43,11 +43,13 @@ in {
|
||||
clusterName = "jungle";
|
||||
nodeName = [
|
||||
"owl[1,2] Sockets=2 CoresPerSocket=14 ThreadsPerCore=2 Feature=owl"
|
||||
"fox Sockets=2 CoresPerSocket=96 ThreadsPerCore=1 Feature=fox"
|
||||
"hut Sockets=2 CoresPerSocket=14 ThreadsPerCore=2"
|
||||
];
|
||||
|
||||
partitionName = [
|
||||
"owl Nodes=owl[1-2] Default=YES DefaultTime=01:00:00 MaxTime=INFINITE State=UP"
|
||||
"fox Nodes=fox Default=NO DefaultTime=01:00:00 MaxTime=INFINITE State=UP"
|
||||
];
|
||||
|
||||
# See slurm.conf(5) for more details about these options.
|
||||
@@ -75,7 +77,7 @@ in {
|
||||
SuspendTimeout=60
|
||||
ResumeProgram=${resumeProgram}
|
||||
ResumeTimeout=300
|
||||
SuspendExcNodes=hut
|
||||
SuspendExcNodes=hut,fox
|
||||
|
||||
# Turn the nodes off after 1 hour of inactivity
|
||||
SuspendTime=3600
|
||||
|
||||
Binary file not shown.
Binary file not shown.
@@ -1,10 +1,11 @@
|
||||
age-encryption.org/v1
|
||||
-> ssh-ed25519 HY2yRg XPOFoZqY+AnKC77jrgNqAm1ADphurfuhO4NRrfiuUDc
|
||||
iCfMMpGHyaYHGy6ci8sqjUtcPeteLlyvLGEF79VPOEc
|
||||
-> ssh-ed25519 CAWG4Q 6OsGrnM+/c5lTN81Rvp166K+ygmSIFeSYzXxYg25KGE
|
||||
Av1zTw2zK4Gufzti9kQaye7C362GCiDRRHzCqBLR33g
|
||||
-> ssh-ed25519 MSF3dg 8CHqJ7mEDvjvqbmF+eE6Em1Wi6eHAzEUpiExC1gm7S0
|
||||
bdwzYHw3RAbdHq+RsiFUP++sQ586VUlSnAzAOhiQUjI
|
||||
--- gA5XSUfjUBol938sC5DbUf8PvQUIr2pNkS2nL95OF9c
|
||||
<EFBFBD><EFBFBD><EFBFBD>Ea1G7<47><37>ݩ[R<><52>\{~$Go<47>cQ<63>wKP&<26><><EFBFBD>w<EFBFBD><77><EFBFBD>6]
|
||||
<EFBFBD><EFBFBD>ѣ<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>^z<7A>̄ 1k<31><6B><03><><EFBFBD><EFBFBD><EFBFBD>Y<EFBFBD><59>2<15>p<>2<EFBFBD><32>K<EFBFBD><4B><EFBFBD><EFBFBD>nok<>/X<><58>pt''<27><>$0co=<3D><EFBFBD>
|
||||
-> ssh-ed25519 HY2yRg WSdjyQPzBJ4JbzQpGeq1AAYpWKoXmLI1ZtmNmM5QOzs
|
||||
qGDlDT31DQF1DdHen0+5+52DdsQlabJdA2pOB5O1I6g
|
||||
-> ssh-ed25519 CAWG4Q wioWMDxQjN+d4JdIbCwZg0DLQu1OH2mV6gukRprjuAs
|
||||
670fE61hidOEh20hHiQAhP0+CjDF0WMBNzgwkGT8Yqg
|
||||
-> ssh-ed25519 MSF3dg DN19uvAEtqq4708P6HpuX9i/o/qAvHX6dj69dCF2H1o
|
||||
4Lu9GnjiFLMeXJ2C7aVPJsCHCQVlhylNWJi896Av92s
|
||||
--- 7cKBwOYNOUZ2h3/kAY09aSMASZSxX7hZIT4kvlIiT6w
|
||||
<EFBFBD>6<EFBFBD><02><><EFBFBD><06>fQF5=<3D>bX+<2B>v e`<60>7/<2F><05>A~P<><50>Ѧ7<15><>
|
||||
<EFBFBD><EFBFBD>A<EFBFBD>)<29>h<05><>=oZ<6F>$<24> ^<5E>V0<56>/܅<>r
|
||||
k<EFBFBD>u<EFBFBD>bĶ:R<><52>>^g<><67><06>ik_* % <0B>a7<61>KG<4B><47><EFBFBD><EFBFBD><EFBFBD><EFBFBD>&<26>PI<50><49>n
|
||||
@@ -1,9 +1,10 @@
|
||||
age-encryption.org/v1
|
||||
-> ssh-ed25519 HY2yRg pXNTB/ailRwSEJG1pXvrzzpz5HqkDZdWVWnOH7JGeQ4
|
||||
NzA+2fxfkNRy/u+Zq96A02K1Vxy0ETYZjMkDVTKyCY8
|
||||
-> ssh-ed25519 CAWG4Q 7CLJWn+EAxoWDduXaOSrHaBFHQ4GIpYP/62FFTj3ZTI
|
||||
vSYV1pQg2qI2ngCzM0nCZAnqdz1tbT4hM5m+/TyGU2c
|
||||
-> ssh-ed25519 MSF3dg Akmp4NcZcDuaYHta/Vej6zulNSrAOCd5lmSV+OiBGC4
|
||||
qTxqVzTyywur+GjtUQdbaIUdH1fqCqPe6qPf8iHRa4w
|
||||
--- uCKNqD1TmZZThOzlpsecBKx/k+noIWhCVMr/pzNwBr8
|
||||
r'<27>Ƌs4<15>˺<EFBFBD>Aĥ<41>P <05>L<EFBFBD><4C>7` <09><02><>) H<><48>-<2D>0<EFBFBD>AH<41>5<16><> <1F><>L<EFBFBD>Qe<51><65>H2b<32> <1D>B<42>CJG<4A>"-S<> <1C>\<5C><> <0C>H<n<>V
|
||||
-> ssh-ed25519 HY2yRg GdmdkW+BqqwBgu30b846jv3J7jtCM+a3rgOERuA050A
|
||||
FeGqM75jG9egesR+yyVKHm0/M+uBBp5Hclg4+qN0BR8
|
||||
-> ssh-ed25519 CAWG4Q a0wTWHgulQUYDAMZmXf3dOf6PdYgCqNtSylzWVVRNVM
|
||||
Bx+WSYaiY4ZwlSZJo2a1XPMQmbKOU7F0tKAqVRLBOPo
|
||||
-> ssh-ed25519 MSF3dg KccUvZZUbxbCrRWUWrX8KcHF6vQ5FV/BqUqI59G7dj4
|
||||
CFr7GXpZ9rPgy7HBfOyiYF9FnZUw6KcZwq9f7/0KaU8
|
||||
--- E0Rp6RR/8+o0jvB1lRdhnlabxvI6uu/IgL2ZpPXzTc8
|
||||
<EFBFBD><13>#<23><>H<EFBFBD>$<24>F;<3B><EFBFBD><7F> %<25><>6<><02>2<EFBFBD><32>rfX<66>\Dn <20>ш<EFBFBD>ȉ<EFBFBD>x<EFBFBD><78>><3E><>&;<3B>c<EFBFBD>U<EFBFBD>I=<3D><>M<EFBFBD><4D><EFBFBD>?T<><54>Ǹ<EFBFBD><16>"px<70>ӭ\s<><73><EFBFBD>bF<62><46><EFBFBD><EFBFBD> WD<>{<7B>
|
||||
AW>?U<><55><EFBFBD><17><>HԳ
|
||||
@@ -1,9 +1,9 @@
|
||||
age-encryption.org/v1
|
||||
-> ssh-ed25519 HY2yRg s6iI9f25xulF4KXt+XY07kXXPKxXo7f2Ql/OTHN55Hk
|
||||
WO4Fd2H9c+HL3+XhUF3BmEZVILlcchGxSrSmL2OEdGw
|
||||
-> ssh-ed25519 CAWG4Q TBkdpx8k8K1NvW3wcvaF7omKFwEJ2DxWJp3tIOTjwCA
|
||||
LcYgWRix23AQnw0OQ7f8+8S3J84CHUElX1vKZSETiLE
|
||||
-> ssh-ed25519 MSF3dg WzrF8kjTP7BXXDjmUp7kPCKguthAW12RPo6Vy2RMmh4
|
||||
8C3mT9ktudCTANDxhyNszUkbeDG6X4wOJdx825++dYM
|
||||
--- /w3YQ2UeTi67H1JR0GsdPz2KoLN2Y7BIZfFY+//AWjY
|
||||
<EFBFBD>ӣ-`P<>@<40><> ބ<><DE84><EFBFBD>)9<>9 l<EFBFBD><EFBFBD> <EFBFBD><EFBFBD>Zf<EFBFBD><EFBFBD>V?I><>w鉐<77> z40 <20>2{i@<40>Z<EFBFBD><5A>x<EFBFBD><78>AHn<48>%<25><> <0C>ʤ<EFBFBD>/W<EFBFBD><EFBFBD>Ĕ<EFBFBD>l<EFBFBD><EFBFBD><EFBFBD><EFBFBD>}<7D>&<08>얶<EFBFBD>(<EFBFBD><EFBFBD><EFBFBD>K<EFBFBD><EFBFBD><EFBFBD>S<EFBFBD><EFBFBD>o<EFBFBD>z<EFBFBD>= <1F>d
|
||||
-> ssh-ed25519 HY2yRg xWRxJGWSzA5aplRYCYLB6aBwrUrQQJ2MtDYaD75V5nI
|
||||
J07XF3NQiaYKKKNRcNWi9MloJD2wXHd+2K7bo6lF+QU
|
||||
-> ssh-ed25519 CAWG4Q jNWymbyCczcm8RcaIEbFQBlOMALsuxTl4+pLUi0aR20
|
||||
z5NixlrRD+Y7Z/aFPs6hiDW4/lp8CBQCeJYpbuG9yYM
|
||||
-> ssh-ed25519 MSF3dg QsUQloEKN3k1G49FQnNR/Do6ILgGpjFcw3zu5kk1Ako
|
||||
IHwyFWUEWqCStNcFprnpBa8L5J6zKIsn+7HcgGRv3sM
|
||||
--- oUia0fsL6opeYWACyXtHAu/Ld+bUIt/7S1VszYTvwgU
|
||||
<EFBFBD><EFBFBD>V<EFBFBD><16>*<2A>t<1B>2-<2D>7<><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>h<EFBFBD>&<26><>͢_!տ+<2B><><EFBFBD><EFBFBD>(<28><0F><11>n<EFBFBD><6E> <09><>(<28><19><EFBFBD>/}<EFBFBD><EFBFBD><EFBFBD><EFBFBD>C<EFBFBD>Nͷ|<04>N<>u<EFBFBD>5<EFBFBD>ù勚K<E58B9A><EFBFBD>l<EFBFBD>"<EFBFBD><EFBFBD>klOX<EFBFBD>y<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>A<EFBFBD><EFBFBD>e<><65>$
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Reference in New Issue
Block a user