Power overload in oss01 #22

New Issue

Closed

opened 2023-05-08 20:14:24 +02:00 by rarias · 23 comments

rarias commented 2023-05-08 20:14:24 +02:00 (Migrated from pm.bsc.es)

Copy Link

$ ipmitool -I lanplus -H oss01-ipmi0 -P "" -U "" chassis status
System Power         : off
Power Overload       : true <-- !!!
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     :
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false
Sleep Button Disable : not allowed
Diag Button Disable  : allowed
Reset Button Disable : allowed
Power Button Disable : allowed
Sleep Button Disabled: false
Diag Button Disabled : false
Reset Button Disabled: false
Power Button Disabled: false

Also the flags are not the same with xeon07 (0x0500 vs 0x0100):

xeon07$ ipmitool -I lanplus -H oss01-ipmi0 -P "" -U "" sensor | grep PS
PS1 Status       | 0x0        | discrete   | 0x0500| na        | na        | na        | na        | na        | na
PS1 Input Power  | 8,000      | Watts      | ok    | na        | na        | na        | 868,000   | 920,000   | na
PS1 Curr Out %   | na         | percent    | na    | na        | na        | na        | 100,000   | 112,000   | na
PS1 Temperature  | 35,000     | degrees C  | ok    | na        | na        | na        | 65,000    | 70,000    | na
PS1 Fan Fail     | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na
xeon07$ ipmitool -I lanplus -H xeon07-ipmi0 -P "" -U "" sensor | grep PS
PS1 Status       | 0x0        | discrete   | 0x0100| na        | na        | na        | na        | na        | na
PS1 Input Power  | 100,000    | Watts      | ok    | na        | na        | na        | 868,000   | 920,000   | na
PS1 Curr Out %   | 11,000     | percent    | ok    | na        | na        | na        | 100,000   | 112,000   | na
PS1 Temperature  | 22,000     | degrees C  | ok    | na        | na        | na        | 65,000    | 70,000    | na
PS1 Fan Fail     | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na

Here is the FRU:

$ ipmitool -I lanplus -H oss01-ipmi0 -P "" -U "" fru
FRU Device Description : Builtin FRU Device (ID 0)
 Chassis Type          : Rack Mount Chassis
 Chassis Part Number   : ..................
 Chassis Serial        : ..................
 Chassis Extra         : ...............................
 Chassis Extra         : ...............................
 Board Mfg Date        : 2016-11-23T03:28:00 CET CET
 Board Mfg             : Intel Corporation
 Board Product         : S2600WT2R
 Board Serial          : BQWL64850234
 Board Part Number     : H21573-371
 Product Manufacturer  : Intel Corporation
 Product Name          : S2600WT2R
 Product Part Number   : ..........
 Product Version       : ....................
 Product Serial        : ............
 Product Asset Tag     : ....................

FRU Device Description : Pwr Supply 1 FRU (ID 2)
 Product Manufacturer  : DELTA
 Product Name          : DPS-750XB A
 Product Part Number   : E98791-007
 Product Version       : 02
 Product Serial        : E98791D1227153416

FRU Device Description : Front Panel (ID 4)
 Board Mfg Date        : 2016-11-05T14:44:00 CET CET
 Board Mfg             : Intel Corporation
 Board Product         : FFPANEL
 Board Serial          : BQWL64556018
 Board Part Number     : H39380-171

FRU Device Description : HS Backplane 1 (ID 5)
 Device not present (Unknown (0x81))

FRU Device Description : PCIe SSD AIC 1 (ID 18)
 Device not present (Unknown (0x81))

``` $ ipmitool -I lanplus -H oss01-ipmi0 -P "" -U "" chassis status System Power : off Power Overload : true <-- !!! Power Interlock : inactive Main Power Fault : false Power Control Fault : false Power Restore Policy : always-off Last Power Event : Chassis Intrusion : inactive Front-Panel Lockout : inactive Drive Fault : false Cooling/Fan Fault : false Sleep Button Disable : not allowed Diag Button Disable : allowed Reset Button Disable : allowed Power Button Disable : allowed Sleep Button Disabled: false Diag Button Disabled : false Reset Button Disabled: false Power Button Disabled: false ``` Also the flags are not the same with xeon07 (0x0500 vs 0x0100): ``` xeon07$ ipmitool -I lanplus -H oss01-ipmi0 -P "" -U "" sensor | grep PS PS1 Status | 0x0 | discrete | 0x0500| na | na | na | na | na | na PS1 Input Power | 8,000 | Watts | ok | na | na | na | 868,000 | 920,000 | na PS1 Curr Out % | na | percent | na | na | na | na | 100,000 | 112,000 | na PS1 Temperature | 35,000 | degrees C | ok | na | na | na | 65,000 | 70,000 | na PS1 Fan Fail | 0x0 | discrete | 0x0000| na | na | na | na | na | na xeon07$ ipmitool -I lanplus -H xeon07-ipmi0 -P "" -U "" sensor | grep PS PS1 Status | 0x0 | discrete | 0x0100| na | na | na | na | na | na PS1 Input Power | 100,000 | Watts | ok | na | na | na | 868,000 | 920,000 | na PS1 Curr Out % | 11,000 | percent | ok | na | na | na | 100,000 | 112,000 | na PS1 Temperature | 22,000 | degrees C | ok | na | na | na | 65,000 | 70,000 | na PS1 Fan Fail | 0x0 | discrete | 0x0000| na | na | na | na | na | na ``` Here is the FRU: ``` $ ipmitool -I lanplus -H oss01-ipmi0 -P "" -U "" fru FRU Device Description : Builtin FRU Device (ID 0) Chassis Type : Rack Mount Chassis Chassis Part Number : .................. Chassis Serial : .................. Chassis Extra : ............................... Chassis Extra : ............................... Board Mfg Date : 2016-11-23T03:28:00 CET CET Board Mfg : Intel Corporation Board Product : S2600WT2R Board Serial : BQWL64850234 Board Part Number : H21573-371 Product Manufacturer : Intel Corporation Product Name : S2600WT2R Product Part Number : .......... Product Version : .................... Product Serial : ............ Product Asset Tag : .................... FRU Device Description : Pwr Supply 1 FRU (ID 2) Product Manufacturer : DELTA Product Name : DPS-750XB A Product Part Number : E98791-007 Product Version : 02 Product Serial : E98791D1227153416 FRU Device Description : Front Panel (ID 4) Board Mfg Date : 2016-11-05T14:44:00 CET CET Board Mfg : Intel Corporation Board Product : FFPANEL Board Serial : BQWL64556018 Board Part Number : H39380-171 FRU Device Description : HS Backplane 1 (ID 5) Device not present (Unknown (0x81)) FRU Device Description : PCIe SSD AIC 1 (ID 18) Device not present (Unknown (0x81)) ```

rarias commented 2023-05-08 20:14:40 +02:00 (Migrated from pm.bsc.es)

Copy Link

changed the description

changed the description

rarias commented 2023-05-08 20:20:14 +02:00 (Migrated from pm.bsc.es)

Copy Link

changed the description

changed the description

rarias commented 2023-05-08 20:42:02 +02:00 (Migrated from pm.bsc.es)

Copy Link

changed the description

changed the description

rarias commented 2023-05-08 20:44:17 +02:00 (Migrated from pm.bsc.es)

Copy Link

TODO:

• Test the power supply in a node known to boot properly, so we can isolate problems in the power supply.

TODO: - [x] Test the power supply in a node known to boot properly, so we can isolate problems in the power supply.

rarias commented 2023-05-08 21:18:42 +02:00 (Migrated from pm.bsc.es)

Copy Link

https://filecenter.deltaww.com/Products/Download/19/1906/catalogue/DPS-2000AB-13%20A%20datasheet_20181218.pdf

https://filecenter.deltaww.com/Products/Download/19/1906/catalogue/DPS-2000AB-13%20A%20datasheet_20181218.pdf

rarias commented 2023-05-09 12:43:31 +02:00 (Migrated from pm.bsc.es)

Copy Link

Installed in xeon01, and working fine. The problem is the node oss01.

Installed in xeon01, and working fine. The problem is the node oss01.

rarias commented 2023-05-09 12:59:46 +02:00 (Migrated from pm.bsc.es)

Copy Link

Attempt to boot with good xeon01 power supply from both power supply sockets results in overload error from bmc.

Attempt to boot with good xeon01 power supply from both power supply sockets results in overload error from bmc. 

rarias commented 2023-05-09 15:23:48 +02:00 (Migrated from pm.bsc.es)

Copy Link

After removing all the components from the main board, the BMC still complains about the power supply.

No obvious problems with capacitors.

I removed the main board from the chassis and I can see a burn part.

This area corresponds to the voltage regulator of the CPU:

It looks that it has burned and is causing a short circuit.

After removing all the components from the main board, the BMC still complains about the power supply. No obvious problems with capacitors. I removed the main board from the chassis and I can see a burn part.   This area corresponds to the voltage regulator of the CPU:  It looks that it has burned and is causing a short circuit.

rarias commented 2023-05-09 15:30:38 +02:00 (Migrated from pm.bsc.es)

Copy Link

The regulator has some pads in the back side so I will need a hot air solder station to replace it.

This is from Aliexpress, in mouser is out of stock.

The regulator has some pads in the back side so I will need a hot air solder station to replace it.  This is from [Aliexpress](https://es.aliexpress.com/item/4000862020196.html?gatewayAdapt=glo2esp), in mouser is out of stock.

rarias commented 2023-05-09 18:07:22 +02:00 (Migrated from pm.bsc.es)

Copy Link

Using a SMD soldering air gun I removed the suspicious regulator after some effort. The back side is quite difficult to desolder. Unfortunately, I also accidentally removed some of the nearby SMD capacitors. They are quite small, but I took them and carefully stored in a sticky tape.

With the regulator removed, I installed back the heat sink, the RAM and the fans and tried to boot the node again. Surprisingly, after the BMC loaded it started to load the BIOS.

So I turned the node off and connected a display to see the BIOS messages. This is what I observed after trying again:

It looks that the BIOS passes and attempts to boot from PXE (all the PCI cables and disks are disconnected, so its unable to boot). So at least the power supply is no longer latched in overload mode.

I will try to book some replacements and solder the pieces back together. We should be able to get it working properly.

Using a SMD soldering air gun I removed the suspicious regulator after some effort. The back side is quite difficult to desolder. Unfortunately, I also accidentally removed some of the nearby SMD capacitors. They are quite small, but I took them and carefully stored in a sticky tape. With the regulator removed, I installed back the heat sink, the RAM and the fans and tried to boot the node again. Surprisingly, after the BMC loaded it started to load the BIOS. So I turned the node off and connected a display to see the BIOS messages. This is what I observed after trying again:  It looks that the BIOS passes and attempts to boot from PXE (all the PCI cables and disks are disconnected, so its unable to boot). So at least the power supply is no longer latched in overload mode. I will try to book some replacements and solder the pieces back together. We should be able to get it working properly.

rarias commented 2023-05-10 11:06:48 +02:00 (Migrated from pm.bsc.es)

Copy Link

Although some parts remain in Aliexpress and eBay, the IR3351M is no longer manufactured, but there are alternatives like the SIC634CD.

Although some parts remain in Aliexpress and eBay, the IR3351M is [no longer manufactured][1], but there are alternatives like the [SIC634CD][2]. [1]: https://www.digikey.es/en/products/detail/infineon-technologies/IR3551MTRPBF/3770518 [2]: https://www.digikey.es/en/products/detail/vishay-siliconix/SIC634CD-T1-GE3/7622824

rarias commented 2023-05-10 12:59:34 +02:00 (Migrated from pm.bsc.es)

Copy Link

Apparently, Vishay is the only replacement part that I can find that is still manufactured, after Infineon bough Internation Rectifier back in 2014.

However, the pad layout designed by Vishay is mirrored as reported by the datasheet:

Compared with the IR3551M layout:

Which make them incompatible.

The remaining option is to get a Chinese clone of the IR3551M from Aliexpress.

Apparently, Vishay is the only replacement part that I can find that is still manufactured, after Infineon [bough Internation Rectifier back in 2014][1]. [1]: https://en.wikipedia.org/wiki/International_Rectifier However, the pad layout designed by Vishay is mirrored as reported by the datasheet:  Compared with the IR3551M layout:  Which make them incompatible. The remaining option is to get a Chinese clone of the IR3551M from Aliexpress.

rarias commented 2023-06-20 15:48:28 +02:00 (Migrated from pm.bsc.es)

Copy Link

Email sent to Vishay:

Hello,

We have a server motherboard with a burnt IR3551M DC 50A voltage
regulator from Infineon which was discontinued:

https://www.infineon.com/cms/en/product/power/dc-dc-converters/integrated-power-stages/ir3551/#!documents

Here is the IR3551M datasheet:

https://www.infineon.com/dgdl/ir3551.pdf?fileId=5546d462533600a4015355cd84a11763

On page 21 there are the dimensions of the PQFN 5mm x 6mm.

I'm looking for a drop in replacement, and I saw this SiC638 regulator
which seems similar:

https://www.vishay.com/docs/76582/sic638.pdf

But the pad layout seems to be mirrored and is 5mm x 5mm instead.

Do you have something compatible?

Best regards,
Rodrigo.

Email sent to Vishay: > Hello, > > We have a server motherboard with a burnt IR3551M DC 50A voltage > regulator from Infineon which was discontinued: > > https://www.infineon.com/cms/en/product/power/dc-dc-converters/integrated-power-stages/ir3551/#!documents > > Here is the IR3551M datasheet: > > https://www.infineon.com/dgdl/ir3551.pdf?fileId=5546d462533600a4015355cd84a11763 > > On page 21 there are the dimensions of the PQFN 5mm x 6mm. > > I'm looking for a drop in replacement, and I saw this SiC638 regulator > which seems similar: > > https://www.vishay.com/docs/76582/sic638.pdf > > But the pad layout seems to be mirrored and is 5mm x 5mm instead. > > Do you have something compatible? > > Best regards, > Rodrigo.

rarias commented 2023-06-28 16:28:02 +02:00 (Migrated from pm.bsc.es)

Copy Link

No reply for one week. Let's try Aliexpress then.

No reply for one week. Let's try Aliexpress then.

rarias commented 2023-06-28 16:47:54 +02:00 (Migrated from pm.bsc.es)

Copy Link

Opened resource petition.

Opened resource petition.

rarias commented 2023-07-03 16:16:33 +02:00 (Migrated from pm.bsc.es)

Copy Link

Petition accepted: https://webapps.bsc.es/resource-petitions/normal-petition/18472

Petition accepted: https://webapps.bsc.es/resource-petitions/normal-petition/18472

rarias commented 2023-08-22 11:05:25 +02:00 (Migrated from pm.bsc.es)

Copy Link

Ordered IR3551M on AliExpress, estimated to arrive on 15th of September.

Ordered IR3551M on AliExpress, estimated to arrive on 15th of September.

rarias commented 2023-08-22 19:06:26 +02:00 (Migrated from pm.bsc.es)

Copy Link

mentioned in issue #11

mentioned in issue #11

rarias commented 2023-09-07 11:24:22 +02:00 (Migrated from pm.bsc.es)

Copy Link

https://parcelsapp.com/es/tracking/cnes00627404929

https://parcelsapp.com/es/tracking/cnes00627404929

rarias commented 2023-09-12 14:17:40 +02:00 (Migrated from pm.bsc.es)

Copy Link

Arrived

Arrived

rarias commented 2023-09-13 16:04:30 +02:00 (Migrated from pm.bsc.es)

Copy Link

So, I soldered the new regulator back in the slot along with some of the SMD components that were lost:

The big capacitor below (C3J11) is twisted a bit, but is properly connected 😅

However, I'm not confident that the pads of the regulator were properly connected. In particular, some of the small pads seems to have disappeared, so I don't think this regulator will work in this socket.

The resistor R3J14 seems to be 0.5 Omhs, and I couldn't solder it back, as it is too small. Same with the capacitor C3J7.

There is another empty socket in the right which could be moved there, but resoldering the small SMD is very hard and I don't have the proper tools to do it (yet).

Nevertheless, the node boots fine and is capable of using both CPUs. Now, the power limit of this CPU is probably lower than usual, so we shouldn't stress it. As this node will be used for node storage, it should work okay.

The node is installed back in the cluster and is ready to be configured for Ceph.

I removed two DIMM modules from the bad CPU, so we can donate them to the failing module of eudy. Here are they:

So, I soldered the new regulator back in the slot along with some of the SMD components that were lost:  The big capacitor below (C3J11) is twisted a bit, but is properly connected :sweat_smile: However, I'm not confident that the pads of the regulator were properly connected. In particular, some of the small pads seems to have disappeared, so I don't think this regulator will work in this socket. The resistor R3J14 seems to be 0.5 Omhs, and I couldn't solder it back, as it is too small. Same with the capacitor C3J7. There is another empty socket in the right which could be moved there, but resoldering the small SMD is very hard and I don't have the proper tools to do it (yet). Nevertheless, the node boots fine and is capable of using both CPUs. Now, the power limit of this CPU is probably lower than usual, so we shouldn't stress it. As this node will be used for node storage, it should work okay. The node is installed back in the cluster and is ready to be configured for Ceph. I removed two DIMM modules from the bad CPU, so we can donate them to the failing module of eudy. Here are they: 

rarias commented 2023-09-14 11:57:55 +02:00 (Migrated from pm.bsc.es)

Copy Link

mentioned in issue #39

mentioned in issue #39

rarias commented 2023-09-14 12:00:31 +02:00 (Migrated from pm.bsc.es)

Copy Link

The power overload is fixed, but the fans now spin at a high speed (see #39).

hut% ipmitool -I lanplus -H oss01-ipmi0 -P "" -U "" chassis status
System Power         : on
Power Overload       : false   <--- fixed
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     : command
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false
Sleep Button Disable : not allowed
Diag Button Disable  : allowed
Reset Button Disable : allowed
Power Button Disable : allowed
Sleep Button Disabled: false
Diag Button Disabled : false
Reset Button Disabled: false
Power Button Disabled: false

Notice the power supply was replaced too:

hut% ipmitool -I lanplus -H oss01-ipmi0 -P "" -U "" fru
FRU Device Description : Builtin FRU Device (ID 0)
 Chassis Type          : Rack Mount Chassis
 Chassis Part Number   : ..................
 Chassis Serial        : ..................
 Chassis Extra         : ...............................
 Chassis Extra         : ...............................
 Board Mfg Date        : 2016-11-23T03:28:00 CET CET
 Board Mfg             : Intel Corporation
 Board Product         : S2600WT2R
 Board Serial          : BQWL64850234
 Board Part Number     : H21573-371
 Product Manufacturer  : Intel Corporation
 Product Name          : S2600WT2R
 Product Part Number   : ..........
 Product Version       : ....................
 Product Serial        : ............
 Product Asset Tag     : ....................

FRU Device Description : Pwr Supply 1 FRU (ID 2)
 Product Manufacturer  : DELTA
 Product Name          : DPS-750XB A
 Product Part Number   : E98791-010
 Product Version       : 05
 Product Serial        : E98791D1647123489

FRU Device Description : Pwr Supply 2 FRU (ID 3)
 Device not present (Unknown (0x81))

FRU Device Description : Front Panel (ID 4)
 Board Mfg Date        : 2016-11-05T14:44:00 CET CET
 Board Mfg             : Intel Corporation
 Board Product         : FFPANEL
 Board Serial          : BQWL64556018
 Board Part Number     : H39380-171

FRU Device Description : HS Backplane 1 (ID 5)
 Board Mfg Date        : 2016-08-07T04:36:00 CEST CEST
 Board Mfg             : Intel Corporation
 Board Product         : F1U8X25PCIHSBP
 Board Serial          : BQWL63250201
 Board Part Number     : G97154-371
 Board Extra           : Field not present

FRU Device Description : PCIe SSD AIC 1 (ID 18)
 Board Mfg Date        : 2016-10-29T15:39:00 CEST CEST
 Board Mfg             : Intel Corporation
 Board Product         : AXXPCI16ADPR
 Board Serial          : BQWL64454123
 Board Part Number     : G97168-271

Closing this one.

The power overload is fixed, but the fans now spin at a high speed (see #39). ``` hut% ipmitool -I lanplus -H oss01-ipmi0 -P "" -U "" chassis status System Power : on Power Overload : false <--- fixed Power Interlock : inactive Main Power Fault : false Power Control Fault : false Power Restore Policy : always-off Last Power Event : command Chassis Intrusion : inactive Front-Panel Lockout : inactive Drive Fault : false Cooling/Fan Fault : false Sleep Button Disable : not allowed Diag Button Disable : allowed Reset Button Disable : allowed Power Button Disable : allowed Sleep Button Disabled: false Diag Button Disabled : false Reset Button Disabled: false Power Button Disabled: false ``` Notice the power supply was replaced too: ``` hut% ipmitool -I lanplus -H oss01-ipmi0 -P "" -U "" fru FRU Device Description : Builtin FRU Device (ID 0) Chassis Type : Rack Mount Chassis Chassis Part Number : .................. Chassis Serial : .................. Chassis Extra : ............................... Chassis Extra : ............................... Board Mfg Date : 2016-11-23T03:28:00 CET CET Board Mfg : Intel Corporation Board Product : S2600WT2R Board Serial : BQWL64850234 Board Part Number : H21573-371 Product Manufacturer : Intel Corporation Product Name : S2600WT2R Product Part Number : .......... Product Version : .................... Product Serial : ............ Product Asset Tag : .................... FRU Device Description : Pwr Supply 1 FRU (ID 2) Product Manufacturer : DELTA Product Name : DPS-750XB A Product Part Number : E98791-010 Product Version : 05 Product Serial : E98791D1647123489 FRU Device Description : Pwr Supply 2 FRU (ID 3) Device not present (Unknown (0x81)) FRU Device Description : Front Panel (ID 4) Board Mfg Date : 2016-11-05T14:44:00 CET CET Board Mfg : Intel Corporation Board Product : FFPANEL Board Serial : BQWL64556018 Board Part Number : H39380-171 FRU Device Description : HS Backplane 1 (ID 5) Board Mfg Date : 2016-08-07T04:36:00 CEST CEST Board Mfg : Intel Corporation Board Product : F1U8X25PCIHSBP Board Serial : BQWL63250201 Board Part Number : G97154-371 Board Extra : Field not present FRU Device Description : PCIe SSD AIC 1 (ID 18) Board Mfg Date : 2016-10-29T15:39:00 CEST CEST Board Mfg : Intel Corporation Board Product : AXXPCI16ADPR Board Serial : BQWL64454123 Board Part Number : G97168-271 ``` Closing this one.

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

master

enableStrictDeps

fox-regression

pkgs/tacuda

pkgs/tasycl

robust-fetchGit

old-master

gitea-lfs

only-restart-logins

monitor-gpfs-home

add-tent-machine

m/raccoon

add-fpga-u280

maintenance-purchase

add-fox-machine

share-files

shared-nix-store

intro-nix

lake2-ipoib

No results found.

Labels

Clear labels

bug

ceph

ci

config

cross

hw

io

kernel

mpi

net

nix

ops

repair

slurm

No Label hw

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

abonerib arocanon varcila rarias

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: rarias/jungle#22