rarias/jungle
rarias/jungle
1
0
Fork 0
Code Issues 28 Pull Requests Packages Projects Releases Wiki Activity

Upgrade nixpkgs and monitor nodes via LAN #62

Merged
rarias merged 0 commits from upgrade-nixpkgs into master 2023-08-22 12:09:10 +02:00
Conversation 18 Commits - Files Changed -
rarias commented 2023-07-27 19:41:19 +02:00 (Migrated from pm.bsc.es)
Copy Link

Apparently it causes the Docker from the gitlab CI to break: https://github.com/NixOS/nixpkgs/issues/245365

Let's see if they fix it, or I can find a workaround.

This MR disables the proxy by default in all nodes, as the ssfhead node is now configured a as a NAT router and forwards the traffic to the Internet.

This also fixes #26 by disabling the all_proxy variable.

Apparently it causes the Docker from the gitlab CI to break: https://github.com/NixOS/nixpkgs/issues/245365 Let's see if they fix it, or I can find a workaround. ~~This MR disables the proxy by default in all nodes, as the ssfhead node is now configured a as a NAT router and forwards the traffic to the Internet.~~ This also fixes #26 by disabling the all_proxy variable.
rarias commented 2023-07-27 19:41:19 +02:00 (Migrated from pm.bsc.es)
Copy Link

assigned to @rarias

assigned to @rarias
rarias commented 2023-07-27 19:46:47 +02:00 (Migrated from pm.bsc.es)
Copy Link

Seems to have been fixed in the last docker release: https://docs.docker.com/engine/release-notes/24.0/#2405

Seems to have been fixed in the last docker release: https://docs.docker.com/engine/release-notes/24.0/#2405
rarias commented 2023-07-28 14:26:02 +02:00 (Migrated from pm.bsc.es)
Copy Link

Fix seems to have landed in nixos-unstable, upgrading again.

Fix seems to have landed in nixos-unstable, upgrading again.
rarias commented 2023-07-28 14:34:13 +02:00 (Migrated from pm.bsc.es)
Copy Link

Fixed for docker, but still not merged for gitlab-runner: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/4249

I think I will apply this workaround: https://github.com/NixOS/nixpkgs/issues/245365#issuecomment-1653288644

This is blocking CI tests for ovni.

Fixed for docker, but still not merged for gitlab-runner: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/4249 I think I will apply this workaround: https://github.com/NixOS/nixpkgs/issues/245365#issuecomment-1653288644 This is blocking CI tests for ovni.
rarias commented 2023-07-28 15:49:07 +02:00 (Migrated from pm.bsc.es)
Copy Link

added 4 commits

  • 55d6c177 - Allow access to devices for node_exporter
  • c242b65e - Update nixpkgs to fix docker problem
  • 054b0b83 - Add gitlab-runner workaround for UNIX socket
  • 80c0f630 - Remove proxy and use both google DNS servers

Compare with previous version

added 4 commits <ul><li>55d6c177 - Allow access to devices for node_exporter</li><li>c242b65e - Update nixpkgs to fix docker problem</li><li>054b0b83 - Add gitlab-runner workaround for UNIX socket</li><li>80c0f630 - Remove proxy and use both google DNS servers</li></ul> [Compare with previous version](/gitlab/rarias/jungle/-/merge_requests/18/diffs?diff_id=8980&start_sha=14b173f67e5e35f51f9e78fb190dcf0e166e109b)
rarias commented 2023-07-28 15:54:57 +02:00 (Migrated from pm.bsc.es)
Copy Link

marked this merge request as ready

marked this merge request as **ready**
rarias commented 2023-07-28 15:54:57 +02:00 (Migrated from pm.bsc.es)
Copy Link

changed the description

changed the description
rarias commented 2023-07-28 15:54:57 +02:00 (Migrated from pm.bsc.es)
Copy Link

requested review from @arocanon

requested review from @arocanon
rarias commented 2023-08-02 14:52:24 +02:00 (Migrated from pm.bsc.es)
Copy Link

marked this merge request as draft

marked this merge request as **draft**
rarias commented 2023-08-02 14:52:50 +02:00 (Migrated from pm.bsc.es)
Copy Link

The sysadmin team has disabled forwarding in the ssfhead node, removing the proxy is no longer an option.

The sysadmin team has disabled forwarding in the ssfhead node, removing the proxy is no longer an option.
rarias commented 2023-08-17 12:35:41 +02:00 (Migrated from pm.bsc.es)
Copy Link

Forwarding is enabled again:

ssfhead$ /usr/sbin/sysctl -a 2>&1 | grep net.ipv4.conf.eth[0-1].forwarding
net.ipv4.conf.eth0.forwarding = 1
net.ipv4.conf.eth1.forwarding = 1

Suspects:

ssfhead$ last -R | head
rarias   pts/0        Thu Aug 17 12:28   still logged in
mmucciar pts/1        Tue Aug  8 15:14 - 15:34  (00:20)
usec     pts/1        Tue Aug  8 15:04 - 15:05  (00:00)
mmucciar pts/0        Tue Aug  8 14:41 - 17:15  (02:33)
reboot   system boot  Tue Aug  8 14:40 - 12:31 (8+21:51)
mmucciar pts/0        Mon Aug  7 15:22 - 15:23  (00:00)
mmucciar pts/0        Mon Aug  7 15:19 - 15:22  (00:02)
rarias   pts/0        Thu Aug  3 00:19 - 00:19  (00:00)
rarias   pts/0        Wed Aug  2 17:37 - 22:25  (04:47)
rarias   pts/0        Wed Aug  2 14:12 - 14:19  (00:07)

Curl can reach google without a proxy:

ssfhead$ curl google.es -v
* Rebuilt URL to: google.es/
* Hostname was NOT found in DNS cache
*   Trying 142.250.200.67...
* Connected to google.es (142.250.200.67) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.37.0
> Host: google.es
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: http://www.google.es/
< Content-Type: text/html; charset=UTF-8
< Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-m43Ulp1QkNFPVSekmmL4Yg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
< Date: Thu, 17 Aug 2023 10:32:26 GMT
< Expires: Sat, 16 Sep 2023 10:32:26 GMT
< Cache-Control: public, max-age=2592000
* Server gws is not blacklisted
< Server: gws
< Content-Length: 218
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
<
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.es/">here</A>.
</BODY></HTML>
Forwarding is enabled again: ``` ssfhead$ /usr/sbin/sysctl -a 2>&1 | grep net.ipv4.conf.eth[0-1].forwarding net.ipv4.conf.eth0.forwarding = 1 net.ipv4.conf.eth1.forwarding = 1 ``` Suspects: ``` ssfhead$ last -R | head rarias pts/0 Thu Aug 17 12:28 still logged in mmucciar pts/1 Tue Aug 8 15:14 - 15:34 (00:20) usec pts/1 Tue Aug 8 15:04 - 15:05 (00:00) mmucciar pts/0 Tue Aug 8 14:41 - 17:15 (02:33) reboot system boot Tue Aug 8 14:40 - 12:31 (8+21:51) mmucciar pts/0 Mon Aug 7 15:22 - 15:23 (00:00) mmucciar pts/0 Mon Aug 7 15:19 - 15:22 (00:02) rarias pts/0 Thu Aug 3 00:19 - 00:19 (00:00) rarias pts/0 Wed Aug 2 17:37 - 22:25 (04:47) rarias pts/0 Wed Aug 2 14:12 - 14:19 (00:07) ``` Curl can reach google without a proxy: ``` ssfhead$ curl google.es -v * Rebuilt URL to: google.es/ * Hostname was NOT found in DNS cache * Trying 142.250.200.67... * Connected to google.es (142.250.200.67) port 80 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.37.0 > Host: google.es > Accept: */* > < HTTP/1.1 301 Moved Permanently < Location: http://www.google.es/ < Content-Type: text/html; charset=UTF-8 < Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-m43Ulp1QkNFPVSekmmL4Yg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp < Date: Thu, 17 Aug 2023 10:32:26 GMT < Expires: Sat, 16 Sep 2023 10:32:26 GMT < Cache-Control: public, max-age=2592000 * Server gws is not blacklisted < Server: gws < Content-Length: 218 < X-XSS-Protection: 0 < X-Frame-Options: SAMEORIGIN < <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>301 Moved</TITLE></HEAD><BODY> <H1>301 Moved</H1> The document has moved <A HREF="http://www.google.es/">here</A>. </BODY></HTML> ```
rarias commented 2023-08-17 18:59:15 +02:00 (Migrated from pm.bsc.es)
Copy Link

added 2 commits

  • fb055fa4 - Increase prometheus retention time to one year
  • d0be4925 - Monitor power from other nodes via LAN

Compare with previous version

added 2 commits <ul><li>fb055fa4 - Increase prometheus retention time to one year</li><li>d0be4925 - Monitor power from other nodes via LAN</li></ul> [Compare with previous version](/gitlab/rarias/jungle/-/merge_requests/18/diffs?diff_id=9072&start_sha=80c0f630c0cd65170e3608e3d1489e1e4d167eb2)
rarias commented 2023-08-22 11:36:32 +02:00 (Migrated from pm.bsc.es)
Copy Link

As the Internet connection in the ssfhead is not stable yet, I decided to delay the adoption of that path until we have a better understanding of what is the current status. For now we continue to use our proxies as the main connection to the Internet except the NTP daemon which uses the time.bsc.es server via ssfhead.

I upgraded Nixpkgs again, taking the fix for go in docker and the gitlab runner. The all_proxy variable is not set to any value now (only ftp, http, https and rsync variables are set).

As the Internet connection in the ssfhead is not stable yet, I decided to delay the adoption of that path until we have a better understanding of what is the current status. For now we continue to use our proxies as the main connection to the Internet except the NTP daemon which uses the time.bsc.es server via ssfhead. I upgraded Nixpkgs again, taking the fix for go in docker and the gitlab runner. The all_proxy variable is not set to any value now (only ftp, http, https and rsync variables are set).
rarias commented 2023-08-22 11:38:17 +02:00 (Migrated from pm.bsc.es)
Copy Link

added 4 commits

Compare with previous version

added 4 commits <ul><li>bf692e6e - Don&#39;t set all_proxy</li><li>acf9b71f - Increase prometheus retention time to one year</li><li>f8fb5fa4 - Monitor power from other nodes via LAN</li><li>480c97e9 - Update flake</li></ul> [Compare with previous version](/gitlab/rarias/jungle/-/merge_requests/18/diffs?diff_id=9094&start_sha=d0be49251b662c63adc0e3bf16b20a38f29c9249)
rarias commented 2023-08-22 11:39:24 +02:00 (Migrated from pm.bsc.es)
Copy Link

marked this merge request as ready

marked this merge request as **ready**
rarias commented 2023-08-22 11:39:24 +02:00 (Migrated from pm.bsc.es)
Copy Link

changed title from {-Draft: -}Upgrade nixpkgs to Upgrade nixpkgs{+ and monitor nodes via LAN+}

changed title from **{-Draft: -}Upgrade nixpkgs** to **Upgrade nixpkgs{+ and monitor nodes via LAN+}**
rarias commented 2023-08-22 11:40:15 +02:00 (Migrated from pm.bsc.es)
Copy Link

changed the description

changed the description
arocanon commented 2023-08-22 11:48:45 +02:00 (Migrated from pm.bsc.es)
Copy Link

approved this merge request

approved this merge request
arocanon (Migrated from pm.bsc.es) approved these changes 2024-05-29 10:53:28 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
arocanon
Labels
Clear labels   
ceph

   
ci

   
config

   
hw

   
io

   
kernel

   
mpi

   
net

   
nix

   
ops
No Label
ceph
ci
config
hw
io
kernel
mpi
net
nix
ops
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: rarias/jungle#62
No description provided.
Delete Branch "upgrade-nixpkgs"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?