rarias/jungle
2
0
Fork 3
Code Issues 57 Pull Requests 7 Actions Packages Projects Releases Wiki Activity

WIP: Add fpga u280 #86

Closed
rarias wants to merge 19 commits from add-fpga-u280 into old-master
pull from: add-fpga-u280
merge into: rarias:old-master
Conversation 3 Commits 19 Files Changed 14 +842 -10
rarias commented 2025-02-20 16:38:30 +01:00
Owner
Copy Link

Closes #85

Closes https://jungle.bsc.es/git/rarias/jungle/issues/85
rarias added 6 commits 2025-02-20 16:38:31 +01:00
Only proxy SSH git remotes via hut in xeon bd4dabf03a 
Other machines like raccoon have direct access.
Enable linger for user rarias 165a60b0d2 
Allows services to run without a login session.
Allow X11 forwarding via SSH c5400955c7
First successful build of Xilinx XRT df8b5b2d67 
What an absolute nightmare, and we are far from the end.
Add Xilinx xocl derivation 4ab38212b3 
Need to move to raccoon to access "config" attribute.
Load Xilinx modules at boot a9c1cb26ea
rarias commented 2025-02-20 16:42:33 +01:00
Author
Owner
Copy Link

The msd daemon fails to run, somehow the daemons still use the harcoded path:

raccoon% /nix/store/b6skv0iyl2djirf77q65sysq01qqncrn-xilinx-xrt/xrt/bin/msd
XRT build version: 2.19.0
Build hash:
Build date: 1980-01-01 00:00:00
Git branch:
PID: 281888
UID: 1880
[Thu Feb 20 15:41:16 2025 GMT]
HOST: raccoon
EXE: /nix/store/b6skv0iyl2djirf77q65sysq01qqncrn-xilinx-xrt/xrt/bin/msd
[WARNING] ERROR: No such library '/opt/xilinx/xrt/lib/libxrt_core.so.2'
The msd daemon fails to run, somehow the daemons still use the harcoded path: ``` raccoon% /nix/store/b6skv0iyl2djirf77q65sysq01qqncrn-xilinx-xrt/xrt/bin/msd XRT build version: 2.19.0 Build hash: Build date: 1980-01-01 00:00:00 Git branch: PID: 281888 UID: 1880 [Thu Feb 20 15:41:16 2025 GMT] HOST: raccoon EXE: /nix/store/b6skv0iyl2djirf77q65sysq01qqncrn-xilinx-xrt/xrt/bin/msd [WARNING] ERROR: No such library '/opt/xilinx/xrt/lib/libxrt_core.so.2' ```
rarias commented 2025-02-20 16:59:24 +01:00
Author
Owner
Copy Link

Probably I'm missing the udev rules to make a device.

raccoon% find . -name '*.rules'
./src/runtime_src/core/pcie/driver/aws/kernel/mgmt/10-awsmgmt.rules
./src/runtime_src/core/pcie/driver/linux/xocl/userpf/99-xocl.rules
./src/runtime_src/core/pcie/driver/linux/xocl/mgmtpf/99-xclmgmt.rules
./src/runtime_src/core/edge/drm/zocl/10-zocl.rules
raccoon% cat ./src/runtime_src/core/pcie/driver/linux/xocl/mgmtpf/99-xclmgmt.rules
KERNEL=="xclmgmt*",MODE="0666",SYMLINK="%k"
KERNEL=="xfpga/xvc_pub.m*",MODE="0666",SYMLINK="xvc_pub.m%n"
KERNEL=="xfpga/xvc_pri.m*",MODE="0600",SYMLINK="xvc_pri.m%n"
KERNEL=="xfpga/nifd_pri.m*",MODE="0600"
KERNEL=="xfpga/icap.m*",MODE="0200"
raccoon% cat ./src/runtime_src/core/pcie/driver/linux/xocl/userpf/99-xocl.rules
KERNEL=="renderD*",MODE="0666"
KERNEL=="xfpga/xvc_pub.u*",MODE="0666",SYMLINK="xvc_pub.u%n"
KERNEL=="xfpga/dma.qdma.u*",MODE="0666"
KERNEL=="xfpga/aximm_mon.u*",MODE="0666"
KERNEL=="xfpga/accel_mon.u*",MODE="0666"
KERNEL=="xfpga/axistream_mon.u*",MODE="0666"
KERNEL=="xfpga/trace_fifo_lite.u*",MODE="0666"
KERNEL=="xfpga/trace_fifo_full.u*",MODE="0666"
KERNEL=="xfpga/trace_funnel.u*",MODE="0666"
KERNEL=="xfpga/trace_s2mm.u*",MODE="0666"
KERNEL=="xfpga/accel_deadlock.u*",MODE="0666"
Probably I'm missing the udev rules to make a device. ``` raccoon% find . -name '*.rules' ./src/runtime_src/core/pcie/driver/aws/kernel/mgmt/10-awsmgmt.rules ./src/runtime_src/core/pcie/driver/linux/xocl/userpf/99-xocl.rules ./src/runtime_src/core/pcie/driver/linux/xocl/mgmtpf/99-xclmgmt.rules ./src/runtime_src/core/edge/drm/zocl/10-zocl.rules raccoon% cat ./src/runtime_src/core/pcie/driver/linux/xocl/mgmtpf/99-xclmgmt.rules KERNEL=="xclmgmt*",MODE="0666",SYMLINK="%k" KERNEL=="xfpga/xvc_pub.m*",MODE="0666",SYMLINK="xvc_pub.m%n" KERNEL=="xfpga/xvc_pri.m*",MODE="0600",SYMLINK="xvc_pri.m%n" KERNEL=="xfpga/nifd_pri.m*",MODE="0600" KERNEL=="xfpga/icap.m*",MODE="0200" raccoon% cat ./src/runtime_src/core/pcie/driver/linux/xocl/userpf/99-xocl.rules KERNEL=="renderD*",MODE="0666" KERNEL=="xfpga/xvc_pub.u*",MODE="0666",SYMLINK="xvc_pub.u%n" KERNEL=="xfpga/dma.qdma.u*",MODE="0666" KERNEL=="xfpga/aximm_mon.u*",MODE="0666" KERNEL=="xfpga/accel_mon.u*",MODE="0666" KERNEL=="xfpga/axistream_mon.u*",MODE="0666" KERNEL=="xfpga/trace_fifo_lite.u*",MODE="0666" KERNEL=="xfpga/trace_fifo_full.u*",MODE="0666" KERNEL=="xfpga/trace_funnel.u*",MODE="0666" KERNEL=="xfpga/trace_s2mm.u*",MODE="0666" KERNEL=="xfpga/accel_deadlock.u*",MODE="0666" ```
rarias added 2 commits 2025-02-21 10:31:39 +01:00
Add udev rules 0533a48a98
Add U280 support back in the xocl driver e81b7cc158 
They removed U280 from the list of supported devices, but that doesn't
mean it is will stop working.
rarias commented 2025-02-21 10:33:56 +01:00
Author
Owner
Copy Link

They had removed support for the U280 device. Reverting the commit makes the driver perform the initialization:

[198196.595845] xclmgmt init()
[198196.597370] xclmgmt 0000:83:00.0:  ffff8bfa058d50c0 xclmgmt_probe: Driver: 2.19.0,
[198196.597385] xclmgmt 0000:83:00.0:  ffff8bfa058d50c0 xclmgmt_probe: probe(pdev = 0x0000000039654fe7, pci_id = 0x00000000d0a106f3)
[198196.598732] xclmgmt 0000:83:00.0:  ffff8bfa058d50c0 xocl_subdev_vsec: No Vendor Specific Capability found.
[198196.599155] xclmgmt 0000:83:00.0:  ffff8bfa058d50c0 xclmgmt_probe: minimum initialization done
[198196.599641] xclmgmt 0000:83:00.0: flash.m.8388608 ffff8c0145c1ac10 qspi_probe: QSPI FIFO depth is: 256
[198196.599674] xclmgmt 0000:83:00.0: flash.m.8388608 ffff8c0145c1ac10 qspi_probe: Number of slave chips is: 1
[198196.599700] xclmgmt 0000:83:00.0: flash.m.8388608 ffff8c0145c1ac10 flash_get_info: Flash vendor: micron
[198196.599707] xclmgmt 0000:83:00.0: flash.m.8388608 ffff8c0145c1ac10 flash_get_info: Flash size: 128MB
[198196.599751] xclmgmt 0000:83:00.0:  ffff8bfa058d50c0 __xocl_subdev_create: Created subdev flash inst 8388608 level 0
[198196.601020] xclmgmt 0000:83:00.0:  ffff8bfa058d50c0 xocl_subdev_vsec: No Vendor Specific Capability found.
[198196.602101] xclmgmt 0000:83:00.0:  ffff8bfa058d50c0 xocl_subdev_vsec: No Vendor Specific Capability found.
[198196.603172] xclmgmt 0000:83:00.0:  ffff8bfa058d50c0 xocl_subdev_vsec: No Vendor Specific Capability found.

And now we see the FPGA:

raccoon% xrt/xrt/bin/xbmgmt examine
System Configuration
  OS Name              : Linux
  Release              : 6.9.9
  Machine              : x86_64
  CPU Cores            : 32
  Memory               : 64356 MB
  Distribution         : NixOS 24.11 (Vicuna)
  GLIBC                : 2.39
  Model                : W2600CR
  BIOS Vendor          : Intel Corp.
  BIOS Version         : SE5C600.86B.99.99.x044.020120121354

XRT
  Version              : 2.19.0
  Branch               :
  Hash                 :
  Hash Date            : 1980-01-01 00:00:00
  xocl                 : 2.19.0,
  xclmgmt              : 2.19.0,

Device(s) Present
|BDF             |Shell                 |Logic UUID  |Device ID  |Device Ready*  |
|----------------|----------------------|------------|-----------|---------------|
|[0000:83:00.0]  |xilinx_u280_GOLDEN_8  |n/a         |n/a        |No             |


* Devices that are not ready will have reduced functionality when using XRT tools
They had removed support for the U280 device. Reverting the commit makes the driver perform the initialization: ``` [198196.595845] xclmgmt init() [198196.597370] xclmgmt 0000:83:00.0: ffff8bfa058d50c0 xclmgmt_probe: Driver: 2.19.0, [198196.597385] xclmgmt 0000:83:00.0: ffff8bfa058d50c0 xclmgmt_probe: probe(pdev = 0x0000000039654fe7, pci_id = 0x00000000d0a106f3) [198196.598732] xclmgmt 0000:83:00.0: ffff8bfa058d50c0 xocl_subdev_vsec: No Vendor Specific Capability found. [198196.599155] xclmgmt 0000:83:00.0: ffff8bfa058d50c0 xclmgmt_probe: minimum initialization done [198196.599641] xclmgmt 0000:83:00.0: flash.m.8388608 ffff8c0145c1ac10 qspi_probe: QSPI FIFO depth is: 256 [198196.599674] xclmgmt 0000:83:00.0: flash.m.8388608 ffff8c0145c1ac10 qspi_probe: Number of slave chips is: 1 [198196.599700] xclmgmt 0000:83:00.0: flash.m.8388608 ffff8c0145c1ac10 flash_get_info: Flash vendor: micron [198196.599707] xclmgmt 0000:83:00.0: flash.m.8388608 ffff8c0145c1ac10 flash_get_info: Flash size: 128MB [198196.599751] xclmgmt 0000:83:00.0: ffff8bfa058d50c0 __xocl_subdev_create: Created subdev flash inst 8388608 level 0 [198196.601020] xclmgmt 0000:83:00.0: ffff8bfa058d50c0 xocl_subdev_vsec: No Vendor Specific Capability found. [198196.602101] xclmgmt 0000:83:00.0: ffff8bfa058d50c0 xocl_subdev_vsec: No Vendor Specific Capability found. [198196.603172] xclmgmt 0000:83:00.0: ffff8bfa058d50c0 xocl_subdev_vsec: No Vendor Specific Capability found. ``` And now we see the FPGA: ``` raccoon% xrt/xrt/bin/xbmgmt examine System Configuration OS Name : Linux Release : 6.9.9 Machine : x86_64 CPU Cores : 32 Memory : 64356 MB Distribution : NixOS 24.11 (Vicuna) GLIBC : 2.39 Model : W2600CR BIOS Vendor : Intel Corp. BIOS Version : SE5C600.86B.99.99.x044.020120121354 XRT Version : 2.19.0 Branch : Hash : Hash Date : 1980-01-01 00:00:00 xocl : 2.19.0, xclmgmt : 2.19.0, Device(s) Present |BDF |Shell |Logic UUID |Device ID |Device Ready* | |----------------|----------------------|------------|-----------|---------------| |[0000:83:00.0] |xilinx_u280_GOLDEN_8 |n/a |n/a |No | * Devices that are not ready will have reduced functionality when using XRT tools ```
rarias added 8 commits 2025-05-05 11:55:22 +02:00
Add FPGA u280 firmware (working ok) 82ed40d386
Add node exporter monitoring in raccoon 4807337098
Add dbautist user to raccoon machine b87d79e1cc
Disable nix garbage collector b513cd8982
Enable binfmt emulation in raccoon cf4ce9fa18
Make raccoon use performance governor 2be8c0dad3
Use nix cache from hut 5899d8847c
Set keep-outputs to true in all machines f00575640c 
From the documentation of keep-outputs, setting it to true would prevent
the GC from removing build time dependencies:

If true, the garbage collector will keep the outputs of non-garbage
derivations. If false (default), outputs will be deleted unless they are
GC roots themselves (or reachable from other roots).

In general, outputs must be registered as roots separately. However,
even if the output of a derivation is registered as a root, the
collector will still delete store paths that are used only at build time
(e.g., the C compiler, or source tarballs downloaded from the network).
To prevent it from doing so, set this option to true.

See: https://nix.dev/manual/nix/2.24/command-ref/conf-file.html#conf-keep-outputs
Reviewed-by: Aleix Roca Nonell <aleix.rocanonell@bsc.es>
arocanon added 1 commit 2025-05-06 14:44:19 +02:00
raccoon: Enable nixdebuginfod 00e686a1d8
arocanon added 2 commits 2025-05-15 15:29:14 +02:00
raccoon: Add lttng and extend perf support a2b9b155f6
users: Create tracing group and add arocanon 8161a21fbc
rarias closed this pull request 2025-06-13 11:22:21 +02:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
bug
ceph
ci
config
cross
hw
io
kernel
mpi
net
nix
ops
repair
slurm
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
abonerib arocanon varcila rarias
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rarias/jungle#86
Delete Branch "add-fpga-u280"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?