Prepare August automatic shutdown #71

New Issue

rarias · 2024-07-19T18:06:05+02:00

rarias commented

2024-07-19 18:06:05 +02:00

To avoid potential damage to the power supplies, we should turn off all the nodes before the August maintenance shutdown. This can be done with a systemd timer that runs the shutdown on a scheduled data. As we have all the nodes synced via NTP, this should work properly.

We may want to add some random delay so we don't shutdown all the nodes at the same time.

To avoid potential damage to the power supplies, we should turn off all the nodes before the August maintenance shutdown. This can be done with a systemd timer that runs the shutdown on a scheduled data. As we have all the nodes synced via NTP, this should work properly. We may want to add some random delay so we don't shutdown all the nodes at the same time.

rarias added the

hw

label 2024-07-19 18:06:05 +02:00

rarias commented

2024-07-22 11:49:36 +02:00

Current status:

owl1	OK
owl2	OK
hut		OK
eudy	
koro	(off)
lake2	OK
bay		OK
raccoon	OK
xeon06	OK

CC @arocanon missing eudy, relevant commits: d3489f8e48 4a52970821

Quick patch:

eudy$ curl https://jungle.bsc.es/git/rarias/jungle/commit/d3489f8e488db0185a0953392b8c428da08a2649.patch | git am -3
eudy$ curl https://jungle.bsc.es/git/rarias/jungle/commit/4a529708212daf43ebf7af62a14fa955f1565c09.patch | git am -3
eudy$ sudo nixos-rebuild switch --flake .

Can be tested with:

$ echo "$(hostname)\t$(systemctl status august-shutdown.timer | grep Trigger: )"

Current timers:

owl1     Trigger: Fri 2024-08-02 11:08:43 CEST; 1 week 3 days left
owl2     Trigger: Fri 2024-08-02 11:03:21 CEST; 1 week 3 days left
hut      Trigger: Fri 2024-08-02 11:09:22 CEST; 1 week 3 days left
lake2    Trigger: Fri 2024-08-02 11:08:17 CEST; 1 week 3 days left
bay      Trigger: Fri 2024-08-02 11:05:55 CEST; 1 week 3 days left
raccoon  Trigger: Fri 2024-08-02 11:00:17 CEST; 1 week 3 days left

Current status: ``` owl1 OK owl2 OK hut OK eudy koro (off) lake2 OK bay OK raccoon OK xeon06 OK ``` CC @arocanon missing eudy, relevant commits: https://jungle.bsc.es/git/rarias/jungle/commit/d3489f8e488db0185a0953392b8c428da08a2649 https://jungle.bsc.es/git/rarias/jungle/commit/4a529708212daf43ebf7af62a14fa955f1565c09 Quick patch: ``` eudy$ curl https://jungle.bsc.es/git/rarias/jungle/commit/d3489f8e488db0185a0953392b8c428da08a2649.patch | git am -3 eudy$ curl https://jungle.bsc.es/git/rarias/jungle/commit/4a529708212daf43ebf7af62a14fa955f1565c09.patch | git am -3 eudy$ sudo nixos-rebuild switch --flake . ``` Can be tested with: ``` $ echo "$(hostname)\t$(systemctl status august-shutdown.timer | grep Trigger: )" ``` Current timers: ``` owl1 Trigger: Fri 2024-08-02 11:08:43 CEST; 1 week 3 days left owl2 Trigger: Fri 2024-08-02 11:03:21 CEST; 1 week 3 days left hut Trigger: Fri 2024-08-02 11:09:22 CEST; 1 week 3 days left lake2 Trigger: Fri 2024-08-02 11:08:17 CEST; 1 week 3 days left bay Trigger: Fri 2024-08-02 11:05:55 CEST; 1 week 3 days left raccoon Trigger: Fri 2024-08-02 11:00:17 CEST; 1 week 3 days left ```

rarias commented

2024-08-02 11:35:11 +02:00

All machines are off except hut, so we can check they were off.

I'll shut down hut manually, which will take down this gitea server. See you on the other side 👋

All machines are off except hut, so we can check they were off. ![image](/attachments/8ce67a07-18e9-436c-9cb2-2c2edb93ba92) ![image](/attachments/4c098c32-2dee-4f67-bb7f-396038dee527) I'll shut down hut manually, which will take down this gitea server. See you on the other side 👋

image.png

29 KiB

image.png

11 KiB

Sign in to join this conversation.