rarias/jungle
rarias/jungle
2
0
Fork 3
Code Issues 53 Pull Requests 10 Actions Packages Projects Releases Wiki Activity

lake2 very slow during IO operations #215

New Issue
Open
opened 2025-11-03 13:39:48 +01:00 by rarias · 6 comments
rarias commented 2025-11-03 13:39:48 +01:00
Owner
Copy Link
lake2% ping -c 5 apex
PING apex (10.0.40.30) 56(84) bytes of data.
64 bytes from apex (10.0.40.30): icmp_seq=1 ttl=64 time=94.0 ms
64 bytes from apex (10.0.40.30): icmp_seq=2 ttl=64 time=81.2 ms
64 bytes from apex (10.0.40.30): icmp_seq=3 ttl=64 time=93.2 ms
64 bytes from apex (10.0.40.30): icmp_seq=4 ttl=64 time=76.7 ms
64 bytes from apex (10.0.40.30): icmp_seq=5 ttl=64 time=90.8 ms

--- apex ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4002ms
rtt min/avg/max/mdev = 76.696/87.181/94.018/6.959 ms
lake2% vmstat -w 1 5
--procs-- -----------------------memory---------------------- ---swap-- -----io---- -system-- ----------cpu----------
   r    b         swpd         free         buff        cache   si   so    bi    bo   in   cs  us  sy  id  wa  st  gu
   0    0            0    109223360      5308928      4543932    0    0  2502    45 3012    1   0   0 100   0   0   0
   0    0            0    109229536      5308980      4543932    0    0     0 32884 20454 23462   0   1  98   0   0   0
   0    0            0    109238552      5309052      4543932    0    0     0 49320 20023 26188   0   1  98   0   0   0
   0    0            0    109249264      5309072      4543932    0    0     0 12344 16952 22639   0   1  99   0   0   0
   0    0            0    109240132      5309120      4543940    0    0     0 28776 18542 22213   0   1  99   0   0   0
lake2% free -h
               total        used        free      shared  buff/cache   available
Mem:           125Gi        13Gi       104Gi        12Mi       9,4Gi       112Gi

lake2% sudo top
top - 13:32:21 up 68 days,  3:30,  2 users,  load average: 1,08, 1,26, 1,14
Tasks: 406 total,   1 running, 405 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0,3 us,  0,6 sy,  0,0 ni, 98,5 id,  0,0 wa,  0,1 hi,  0,5 si,  0,0 st
MiB Mem : 128809,3 total, 106683,6 free,  13464,8 used,   9627,9 buff/cache
MiB Swap:   7629,0 total,   7629,0 free,      0,0 used. 115344,5 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1742 ceph      20   0 6210836   2,8g  41212 S   8,6   2,2     20,30 ceph-osd
   1743 ceph      20   0 5819668   2,8g  41508 S   7,9   2,2     20,47 ceph-osd
   1750 ceph      20   0 6292756   2,8g  41064 S   7,3   2,2     21,34 ceph-osd
   1752 ceph      20   0 6098196   2,8g  41484 S   6,0   2,3     21,03 ceph-osd
  96245 root      20   0   11004   5840   3792 R   0,7   0,0   0:00.10 top
      1 root      20   0   23600  14096  10928 S   0,0   0,0 148:44.99 systemd
      2 root      20   0       0      0      0 S   0,0   0,0   0:03.40 kthreadd
      3 root      20   0       0      0      0 S   0,0   0,0   0:00.00 pool_workqueue_release
      4 root       0 -20       0      0      0 I   0,0   0,0   0:00.00 kworker/R-rcu_gp
      5 root       0 -20       0      0      0 I   0,0   0,0   0:00.00 kworker/R-sync_wq
      6 root       0 -20       0      0      0 I   0,0   0,0   0:00.00 kworker/R-kvfree_rcu_reclaim

From bay:

bay% ping apex
PING apex (10.0.40.30) 56(84) bytes of data.
64 bytes from apex (10.0.40.30): icmp_seq=1 ttl=64 time=0.158 ms
64 bytes from apex (10.0.40.30): icmp_seq=2 ttl=64 time=0.184 ms
64 bytes from apex (10.0.40.30): icmp_seq=3 ttl=64 time=0.191 ms
64 bytes from apex (10.0.40.30): icmp_seq=4 ttl=64 time=0.164 ms
^C
--- apex ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3098ms
rtt min/avg/max/mdev = 0.158/0.174/0.191/0.013 ms
bay% ping lake2
PING lake2 (10.0.40.42) 56(84) bytes of data.
64 bytes from lake2 (10.0.40.42): icmp_seq=1 ttl=64 time=95.7 ms
64 bytes from lake2 (10.0.40.42): icmp_seq=2 ttl=64 time=100 ms
64 bytes from lake2 (10.0.40.42): icmp_seq=3 ttl=64 time=75.6 ms
64 bytes from lake2 (10.0.40.42): icmp_seq=4 ttl=64 time=88.7 ms
64 bytes from lake2 (10.0.40.42): icmp_seq=5 ttl=64 time=94.4 ms
^C
--- lake2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 75.572/90.959/100.410/8.558 ms
``` lake2% ping -c 5 apex PING apex (10.0.40.30) 56(84) bytes of data. 64 bytes from apex (10.0.40.30): icmp_seq=1 ttl=64 time=94.0 ms 64 bytes from apex (10.0.40.30): icmp_seq=2 ttl=64 time=81.2 ms 64 bytes from apex (10.0.40.30): icmp_seq=3 ttl=64 time=93.2 ms 64 bytes from apex (10.0.40.30): icmp_seq=4 ttl=64 time=76.7 ms 64 bytes from apex (10.0.40.30): icmp_seq=5 ttl=64 time=90.8 ms --- apex ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4002ms rtt min/avg/max/mdev = 76.696/87.181/94.018/6.959 ms lake2% vmstat -w 1 5 --procs-- -----------------------memory---------------------- ---swap-- -----io---- -system-- ----------cpu---------- r b swpd free buff cache si so bi bo in cs us sy id wa st gu 0 0 0 109223360 5308928 4543932 0 0 2502 45 3012 1 0 0 100 0 0 0 0 0 0 109229536 5308980 4543932 0 0 0 32884 20454 23462 0 1 98 0 0 0 0 0 0 109238552 5309052 4543932 0 0 0 49320 20023 26188 0 1 98 0 0 0 0 0 0 109249264 5309072 4543932 0 0 0 12344 16952 22639 0 1 99 0 0 0 0 0 0 109240132 5309120 4543940 0 0 0 28776 18542 22213 0 1 99 0 0 0 lake2% free -h total used free shared buff/cache available Mem: 125Gi 13Gi 104Gi 12Mi 9,4Gi 112Gi lake2% sudo top top - 13:32:21 up 68 days, 3:30, 2 users, load average: 1,08, 1,26, 1,14 Tasks: 406 total, 1 running, 405 sleeping, 0 stopped, 0 zombie %Cpu(s): 0,3 us, 0,6 sy, 0,0 ni, 98,5 id, 0,0 wa, 0,1 hi, 0,5 si, 0,0 st MiB Mem : 128809,3 total, 106683,6 free, 13464,8 used, 9627,9 buff/cache MiB Swap: 7629,0 total, 7629,0 free, 0,0 used. 115344,5 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1742 ceph 20 0 6210836 2,8g 41212 S 8,6 2,2 20,30 ceph-osd 1743 ceph 20 0 5819668 2,8g 41508 S 7,9 2,2 20,47 ceph-osd 1750 ceph 20 0 6292756 2,8g 41064 S 7,3 2,2 21,34 ceph-osd 1752 ceph 20 0 6098196 2,8g 41484 S 6,0 2,3 21,03 ceph-osd 96245 root 20 0 11004 5840 3792 R 0,7 0,0 0:00.10 top 1 root 20 0 23600 14096 10928 S 0,0 0,0 148:44.99 systemd 2 root 20 0 0 0 0 S 0,0 0,0 0:03.40 kthreadd 3 root 20 0 0 0 0 S 0,0 0,0 0:00.00 pool_workqueue_release 4 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 kworker/R-rcu_gp 5 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 kworker/R-sync_wq 6 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 kworker/R-kvfree_rcu_reclaim ``` From bay: ``` bay% ping apex PING apex (10.0.40.30) 56(84) bytes of data. 64 bytes from apex (10.0.40.30): icmp_seq=1 ttl=64 time=0.158 ms 64 bytes from apex (10.0.40.30): icmp_seq=2 ttl=64 time=0.184 ms 64 bytes from apex (10.0.40.30): icmp_seq=3 ttl=64 time=0.191 ms 64 bytes from apex (10.0.40.30): icmp_seq=4 ttl=64 time=0.164 ms ^C --- apex ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3098ms rtt min/avg/max/mdev = 0.158/0.174/0.191/0.013 ms bay% ping lake2 PING lake2 (10.0.40.42) 56(84) bytes of data. 64 bytes from lake2 (10.0.40.42): icmp_seq=1 ttl=64 time=95.7 ms 64 bytes from lake2 (10.0.40.42): icmp_seq=2 ttl=64 time=100 ms 64 bytes from lake2 (10.0.40.42): icmp_seq=3 ttl=64 time=75.6 ms 64 bytes from lake2 (10.0.40.42): icmp_seq=4 ttl=64 time=88.7 ms 64 bytes from lake2 (10.0.40.42): icmp_seq=5 ttl=64 time=94.4 ms ^C --- lake2 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4006ms rtt min/avg/max/mdev = 75.572/90.959/100.410/8.558 ms ```
rarias commented 2025-11-03 13:42:18 +01:00
Author
Owner
Copy Link

What the hell:

lake2% sudo ethtool eno1
Settings for eno1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 100Mb/s <-------------- ???
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        MDI-X: on (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
What the hell: ``` lake2% sudo ethtool eno1 Settings for eno1: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: 100Mb/s <-------------- ??? Duplex: Full Auto-negotiation: on Port: Twisted Pair PHYAD: 1 Transceiver: internal MDI-X: on (auto) Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: yes ```
rarias commented 2025-11-03 13:52:31 +01:00
Author
Owner
Copy Link

Speed renegotiation doesn't seem to fix it.

lake2% sudo mii-tool -r eno1
restarting autonegotiation...
lake2% sudo ethtool eno1
Settings for eno1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 100Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        MDI-X: on (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

Let's try rebooting the node.

Speed renegotiation doesn't seem to fix it. ``` lake2% sudo mii-tool -r eno1 restarting autonegotiation... lake2% sudo ethtool eno1 Settings for eno1: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: 100Mb/s Duplex: Full Auto-negotiation: on Port: Twisted Pair PHYAD: 1 Transceiver: internal MDI-X: on (auto) Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: yes ``` Let's try rebooting the node.
rarias commented 2025-11-03 14:03:25 +01:00
Author
Owner
Copy Link

Same state after a reboot:

lake2% sudo ethtool eno1
Settings for eno1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 100Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        MDI-X: on (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

So, either there is a problem with the Ethernet firmware, the Ethernet NIC, the cable or that particular port of the switch.

Same state after a reboot: ``` lake2% sudo ethtool eno1 Settings for eno1: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: 100Mb/s Duplex: Full Auto-negotiation: on Port: Twisted Pair PHYAD: 1 Transceiver: internal MDI-X: on (auto) Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: yes ``` So, either there is a problem with the Ethernet firmware, the Ethernet NIC, the cable or that particular port of the switch.
rarias added the
bug
hw
net
 labels 2025-11-03 14:05:29 +01:00
rarias commented 2025-11-03 14:13:11 +01:00
Author
Owner
Copy Link

Another observation:

MDI-X: on (auto)

Why is it using cross-over configuration?

Another observation: > MDI-X: on (auto) Why is it using cross-over configuration?
rarias commented 2025-11-03 14:22:20 +01:00
Author
Owner
Copy Link

Disabling MDI-X doesn't change the result after renegotiation, but it drops the SOL console. Is the BMC messing with the NIC? We could add another Ethernet cable and switch to the other port for Ethernet and continue letting the BMC use the current port, I assume they are all configured for shared mode.

Disabling MDI-X doesn't change the result after renegotiation, but it drops the SOL console. Is the BMC messing with the NIC? We could add another Ethernet cable and switch to the other port for Ethernet and continue letting the BMC use the current port, I assume they are all configured for shared mode.
rarias commented 2025-11-05 12:45:01 +01:00
Author
Owner
Copy Link

After reconnecting the Ethernet cable into another port in the switch the speed went back to 1 Gbit/s:

lake2% sudo ethtool eno1
Settings for eno1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s <------------------------ here
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        MDI-X: on (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

This allowed me to create a backup tarball of 500 GiB in arount 2 hours (about 70 MiB/s).

However, today it went back to 100 Mbit/s on its own:

lake2% sudo ethtool eno1 | grep Speed
        Speed: 100Mb/s

The problem must be either in the cable or in the NIC.

After reconnecting the Ethernet cable into another port in the switch the speed went back to 1 Gbit/s: ``` lake2% sudo ethtool eno1 Settings for eno1: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: 1000Mb/s <------------------------ here Duplex: Full Auto-negotiation: on Port: Twisted Pair PHYAD: 1 Transceiver: internal MDI-X: on (auto) Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: yes ``` This allowed me to create a backup tarball of 500 GiB in arount 2 hours (about 70 MiB/s). However, today it went back to 100 Mbit/s on its own: ``` lake2% sudo ethtool eno1 | grep Speed Speed: 100Mb/s ``` The problem must be either in the cable or in the NIC.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
ceph
ci
config
cross
hw
io
kernel
mpi
net
nix
ops
repair
slurm
No Label
bug
hw
net
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
abonerib arocanon varcila rarias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rarias/jungle#215
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?