rarias/bscpkgs
rarias/bscpkgs
Archived
2
0
Fork 3
Code Issues 2 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Fix nativeBuildInputs and add cross compilation target #10

Manually merged
abonerib merged 17 commits from abonerib/bscpkgs:CIstrictDeps into master 2025-10-01 15:57:34 +02:00
Conversation 10 Commits 17 Files Changed 15 +161 -108
abonerib commented 2025-07-25 11:01:31 +02:00
Collaborator
Copy Link

I tried to build our packages with strictDeps to fix some low-hanging fruit.

These are the initial steps towards #13

I tried to build our packages with strictDeps to fix some low-hanging fruit. These are the initial steps towards #13
abonerib added 13 commits 2025-07-25 11:01:32 +02:00
rarias commented 2025-07-25 13:26:39 +02:00
Owner
Copy Link

Not sure how, but this broke stdenvClangOmpss2NodesOmpv

I think this is the same case I commented with @arocanon this week, the clangOmpss2Unwrapped inside clangOmpss2 caches the previous value which has a x86 architecture. I proposed explicitly taking the package from buildPackages but maybe there is a cleaner solution:

diff --git a/system/milkv/pkgs/default.nix b/system/milkv/pkgs/default.nix
index 575e13c..5932863 100644
--- a/system/milkv/pkgs/default.nix
+++ b/system/milkv/pkgs/default.nix
@@ -5,6 +5,14 @@ let
     kgdbSetup = final.callPackage misc/kgdb-setup.nix {};
     lockstat = final.callPackage misc/lockstat.nix {};

+    botillo = final.tst-bots-omp.override {
+      stdenv = final.stdenvClangOmpss2.override {
+        cc = final.buildPackages.clangOmpss2.override {
+          clangOmpss2Unwrapped = final.buildPackages.clangOmpss2Unwrapped;
+        };
+      };
+    };
+
     nosvLatest = (final.nosv.override {
       useGit = true;
       gitCommit = "3ca2f67993f85aa73c53f810ff12148189eae642";
> Not sure how, but this broke stdenvClangOmpss2NodesOmpv I think this is the same case I commented with @arocanon this week, the `clangOmpss2Unwrapped` inside `clangOmpss2` caches the previous value which has a x86 architecture. I proposed explicitly taking the package from buildPackages but maybe there is a cleaner solution: ```diff diff --git a/system/milkv/pkgs/default.nix b/system/milkv/pkgs/default.nix index 575e13c..5932863 100644 --- a/system/milkv/pkgs/default.nix +++ b/system/milkv/pkgs/default.nix @@ -5,6 +5,14 @@ let kgdbSetup = final.callPackage misc/kgdb-setup.nix {}; lockstat = final.callPackage misc/lockstat.nix {}; + botillo = final.tst-bots-omp.override { + stdenv = final.stdenvClangOmpss2.override { + cc = final.buildPackages.clangOmpss2.override { + clangOmpss2Unwrapped = final.buildPackages.clangOmpss2Unwrapped; + }; + }; + }; + nosvLatest = (final.nosv.override { useGit = true; gitCommit = "3ca2f67993f85aa73c53f810ff12148189eae642"; ```
rarias commented 2025-07-25 13:43:43 +02:00
Owner
Copy Link

What does strictDeps do internally?, is it documented somewhere else apart from the scattered bash scripts that build stdenv? I'm not sure what are the implications of enabling it for all of our derivations.

If the objective is to make packages suitable for cross compilation, wouldn't it be more realistic to build them from pkgsCross.riscv64 for example? Notice we emulate riscv64 in hut, so to make sure they build without extra help we would need to test them in a machine with no emulation or choose a non emulated architecture.

There are things that change outside the buildInputs/nativeBuildInputs that would cause the cross compilation to break, for example ovni will disable MPI if the build platform cannot execute in the host platform:

pkgs/ovni/default.nix Line 13 in 1666c14a35
, useMpi ? (stdenv.buildPlatform.canExecute stdenv.hostPlatform)

Or internal checks, for example PAPI will refuse to cross compile unless ffsll is specified:

configure: error: ffsll must be specified for cross compile

This could be tested by building:

$ nix build .#pkgsCross.riscv64.bsc-ci.all
What does strictDeps do internally?, is it documented somewhere else apart from the scattered bash scripts that build stdenv? I'm not sure what are the implications of enabling it for all of our derivations. If the objective is to make packages suitable for cross compilation, wouldn't it be more realistic to build them from `pkgsCross.riscv64` for example? Notice we emulate riscv64 in hut, so to make sure they build without extra help we would need to test them in a machine with no emulation or choose a non emulated architecture. There are things that change outside the buildInputs/nativeBuildInputs that would cause the cross compilation to break, for example `ovni` will disable MPI if the build platform cannot execute in the host platform: https://jungle.bsc.es/git/rarias/bscpkgs/src/commit/1666c14a35e08fd2eddcc206d622494435595fdc/pkgs/ovni/default.nix#L13 Or internal checks, for example PAPI will refuse to cross compile unless ffsll is specified: > configure: error: ffsll must be specified for cross compile This could be tested by building: ``` $ nix build .#pkgsCross.riscv64.bsc-ci.all ```
abonerib commented 2025-07-25 15:14:36 +02:00
Author
Collaborator
Copy Link

What does strictDeps do internally?, is it documented somewhere else apart from the scattered bash scripts that build stdenv? I'm not sure what are the implications of enabling it for all of our derivations.

I could not find where (if) it is documented, but there is a recent(ish) push in nixpkgs to enable strictDeps wherever possible:

To my understanding, with strictDeps, nativeBuildInputs are added to PATH and buildInputs are added to PKG_CONFIG, CFLAGS, LD_... (found this explanation in discourse)

If the objective is to make packages suitable for cross compilation, wouldn't it be more realistic to build them from pkgsCross.riscv64 for example? Notice we emulate riscv64 in hut, so to make sure they build without extra help we would need to test them in a machine with no emulation or choose a non emulated architecture.

That is the end goal, but as you said, there are a lot of factors outside buildInputs/nativeBuildInputs that make this impractical. I don't expect nix build .#pkgsCross.riscv64.bsc-ci.all to work for all our packages in the short term.

With strictDeps we can be reasonably sure that our derivations are correct before attempting to cross-compile.

Consider this example when trying to cross compile some package with an incorrect derivation with buildInputs = [ clangOmpss2 ];

  • Our package now depends on riscv64-clangOmpss2
  • Nix builds our package dependencies, cross-compiling LLVM for RISC-V
  • Once everything is built, we start building our leaf derivation
  • It fails immediately because we could not run the binary

With strictDeps we can catch this error in x86 without going through a whole LLVM compilation.

> What does strictDeps do internally?, is it documented somewhere else apart from the scattered bash scripts that build stdenv? I'm not sure what are the implications of enabling it for all of our derivations. I could not find where (if) it is documented, but there is a recent(ish) [push in nixpkgs to enable `strictDeps` wherever possible](https://github.com/NixOS/nixpkgs/issues/178468): To my understanding, with strictDeps, nativeBuildInputs are added to PATH and buildInputs are added to PKG_CONFIG, CFLAGS, LD_... ([found this explanation in discourse](https://discourse.nixos.org/t/use-buildinputs-or-nativebuildinputs-for-nix-shell/8464/2)) > If the objective is to make packages suitable for cross compilation, wouldn't it be more realistic to build them from `pkgsCross.riscv64` for example? Notice we emulate riscv64 in hut, so to make sure they build without extra help we would need to test them in a machine with no emulation or choose a non emulated architecture. That is the end goal, but as you said, there are a lot of factors outside buildInputs/nativeBuildInputs that make this impractical. I don't expect `nix build .#pkgsCross.riscv64.bsc-ci.all` to work for all our packages in the short term. With strictDeps we can be reasonably sure that our derivations are correct before attempting to cross-compile. Consider this example when trying to cross compile some package with an incorrect derivation with `buildInputs = [ clangOmpss2 ];` - Our package now depends on `riscv64-clangOmpss2` - Nix builds our package dependencies, cross-compiling LLVM for RISC-V - Once everything is built, we start building our leaf derivation - It fails immediately because we could not run the binary With `strictDeps` we can catch this error in x86 without going through a whole LLVM compilation.
abonerib added 1 commit 2025-07-29 11:21:10 +02:00
abonerib force-pushed CIstrictDeps from 08161bf291 to 4ba86c9ec0 2025-07-29 16:55:18 +02:00 Compare
abonerib changed title from WIP: Enable strictDeps and fix nativeBuildInputs to Fix nativeBuildInputs 2025-07-29 16:56:59 +02:00
rarias reviewed 2025-08-01 13:11:56 +02:00
pkgs/ovni/default.nix Outdated
@ -47,3 +45,2 @@
'';
nativeBuildInputs = [ cmake ];
buildInputs = lib.optionals (useMpi) [ mpi ];
nativeBuildInputs = [ cmake ] ++ lib.optionals (useMpi) [ mpi ];
rarias commented 2025-08-01 13:03:53 +02:00
Owner

Wouldn't we need mpi in buildInputs as well? I'm not sure how it works under the hood but it needs to pick the mpi libraries for the host architecture as if mpi were in buildInputs. I'm building .#pkgsCross.riscv64.ovni with useMPI to true to see what happens.

Wouldn't we need mpi in buildInputs as well? I'm not sure how it works under the hood but it needs to pick the mpi libraries for the host architecture as if mpi were in buildInputs. I'm building `.#pkgsCross.riscv64.ovni` with useMPI to true to see what happens.
abonerib commented 2025-08-01 14:41:04 +02:00
Author
Collaborator

I don't know either. We have useMpi ? (stdenv.buildPlatform.canExecute stdenv.hostPlatform) so by default we won't have mpi when cross compiling between architectures (not sure how binfmt emulation works with canExecute)

I don't know either. We have `useMpi ? (stdenv.buildPlatform.canExecute stdenv.hostPlatform)` so by default we won't have mpi when cross compiling between architectures (not sure how binfmt emulation works with canExecute)
rarias commented 2025-08-01 15:30:31 +02:00
Owner

It doesn't seem to be able to find MPI:

hut% nix log /nix/store/3il3bv2igpfzccfrd8zjwxd30s0lq2w9-ovni-riscv64-unknown-linux-gnu-1.12.0.drv | tail
-- Could NOT find MPI_C (missing: MPI_C_WORKS)
CMake Error at /nix/store/29ax4k0a83zhz43lb73cv610d95wdsx1-cmake-3.31.6/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
  Could NOT find MPI (missing: MPI_C_FOUND)
Call Stack (most recent call first):
  /nix/store/29ax4k0a83zhz43lb73cv610d95wdsx1-cmake-3.31.6/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE)
  /nix/store/29ax4k0a83zhz43lb73cv610d95wdsx1-cmake-3.31.6/share/cmake-3.31/Modules/FindMPI.cmake:1842 (find_package_handle_standard_args)
  src/emu/CMakeLists.txt:107 (find_package)


-- Configuring incomplete, errors occurred!

I'll test adding it in buildInputs only, as I think cmake only looks for the libmpi.so library, it doesn't use the mpicc wrapper.

not sure how binfmt emulation works with canExecute

It doesn't rely on that, is done in evaluation time based on the architeture tuples:

a19a8e52c3/lib/systems/default.nix (L95)

a19a8e52c3/lib/systems/parse.nix (L436)

It doesn't seem to be able to find MPI: ``` hut% nix log /nix/store/3il3bv2igpfzccfrd8zjwxd30s0lq2w9-ovni-riscv64-unknown-linux-gnu-1.12.0.drv | tail -- Could NOT find MPI_C (missing: MPI_C_WORKS) CMake Error at /nix/store/29ax4k0a83zhz43lb73cv610d95wdsx1-cmake-3.31.6/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:233 (message): Could NOT find MPI (missing: MPI_C_FOUND) Call Stack (most recent call first): /nix/store/29ax4k0a83zhz43lb73cv610d95wdsx1-cmake-3.31.6/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE) /nix/store/29ax4k0a83zhz43lb73cv610d95wdsx1-cmake-3.31.6/share/cmake-3.31/Modules/FindMPI.cmake:1842 (find_package_handle_standard_args) src/emu/CMakeLists.txt:107 (find_package) -- Configuring incomplete, errors occurred! ``` I'll test adding it in buildInputs only, as I think cmake only looks for the libmpi.so library, it doesn't use the mpicc wrapper. > not sure how binfmt emulation works with canExecute It doesn't rely on that, is done in evaluation time based on the architeture tuples: https://github.com/NixOS/nixpkgs/blob/a19a8e52c326b80a2c203724ed68c7e16b365683/lib/systems/default.nix#L95 https://github.com/NixOS/nixpkgs/blob/a19a8e52c326b80a2c203724ed68c7e16b365683/lib/systems/parse.nix#L436
👍 1
rarias marked this conversation as resolved
abonerib force-pushed CIstrictDeps from 4ba86c9ec0 to 0071638701 2025-08-05 19:37:17 +02:00 Compare
abonerib force-pushed CIstrictDeps from 0071638701 to 0cb908d0e5 2025-08-12 17:02:56 +02:00 Compare
abonerib force-pushed CIstrictDeps from 0cb908d0e5 to 22e66259f4 2025-08-29 11:11:44 +02:00 Compare
abonerib force-pushed CIstrictDeps from 22e66259f4 to bed3d8ce31 2025-09-16 15:11:29 +02:00 Compare
abonerib force-pushed CIstrictDeps from bed3d8ce31 to d1e43dba34 2025-09-29 14:20:03 +02:00 Compare
abonerib added 1 commit 2025-09-29 14:35:24 +02:00
abonerib added 1 commit 2025-09-29 15:29:01 +02:00 
Even if we do an override to papi get the proper configure flags for
cross-compiling, the memory fences are not defined for risc-v:

mb.h:67:2: error: #error Need to define rmb for this architecture!
abonerib added 2 commits 2025-09-29 16:17:27 +02:00
abonerib force-pushed CIstrictDeps from 4d631b7974 to 8871e10cea 2025-09-29 16:39:34 +02:00 Compare
abonerib added 1 commit 2025-09-29 17:03:50 +02:00
abonerib force-pushed CIstrictDeps from fb3661ec7c to 8871e10cea 2025-09-29 17:08:41 +02:00 Compare
abonerib reviewed 2025-09-29 17:10:40 +02:00
flake.nix Outdated
@ -19,2 +19,4 @@
hydraJobs = {
inherit (self.legacyPackages.${system}.bsc-ci) test pkgs;
cross = self.lib.genAttrs [ "riscv64" ] (target:
abonerib commented 2025-09-29 17:10:40 +02:00
Author
Collaborator

@rarias do you want to add this to bsc-ci.cross also?

@rarias do you want to add this to `bsc-ci.cross` also?
rarias commented 2025-09-30 17:37:06 +02:00
Owner

It would be great to have some target that we can build by hand that includes bsc cross packages. Maybe something like this?

diff --git a/overlay.nix b/overlay.nix
index 817fd51..c981742 100644
--- a/overlay.nix
+++ b/overlay.nix
@@ -100,6 +100,9 @@ in bscPkgs // {
       final.pkgsCross.${target}.bsc-ci.pkgs
     );

+    crossList = final.runCommand "ci-cross" { }
+      "printf '%s\n' ${toString (collect isDerivation final.bsc-ci.cross.riscv64)} > $out";
+
     all = final.runCommand "ci-all" { }
     ''
       deps="${toString [ final.bsc-ci.pkgsList final.bsc-ci.tests ]}"
It would be great to have some target that we can build by hand that includes bsc cross packages. Maybe something like this? ```diff diff --git a/overlay.nix b/overlay.nix index 817fd51..c981742 100644 --- a/overlay.nix +++ b/overlay.nix @@ -100,6 +100,9 @@ in bscPkgs // { final.pkgsCross.${target}.bsc-ci.pkgs ); + crossList = final.runCommand "ci-cross" { } + "printf '%s\n' ${toString (collect isDerivation final.bsc-ci.cross.riscv64)} > $out"; + all = final.runCommand "ci-all" { } '' deps="${toString [ final.bsc-ci.pkgsList final.bsc-ci.tests ]}" ```
abonerib marked this conversation as resolved
abonerib requested review from rarias 2025-09-29 17:11:45 +02:00
abonerib commented 2025-09-29 17:14:43 +02:00
Author
Collaborator
Copy Link

Cross-compilation results in weasel: http://weasel:3001/eval/24?compare=bsc-ci&full=0

Most derivations fail because of a dependency on pprte which does not cross-compile. This PR in nixpkgs theoretically fixed the issue, https://github.com/NixOS/nixpkgs/commit/d9c222c81435b but it still fails.

Cross-compilation results in weasel: http://weasel:3001/eval/24?compare=bsc-ci&full=0 Most derivations fail because of a dependency on `pprte` which does not cross-compile. This PR in nixpkgs theoretically fixed the issue, https://github.com/NixOS/nixpkgs/commit/d9c222c81435b but it still fails.
abonerib force-pushed CIstrictDeps from 8871e10cea to 32ecd3014d 2025-09-30 16:17:46 +02:00 Compare
rarias reviewed 2025-09-30 18:18:37 +02:00
pkgs/nanos6/default.nix Outdated
@ -96,0 +101,4 @@
# TODO: papi_version is needed for configure:
# ./configure: line 27378: papi_version: command not found
# This probably breaks cross-compilation
rarias commented 2025-09-30 17:34:59 +02:00
Owner

Just a comment so I don't forget, it may be doable to fix nanos6 so that it takes the PAPI version from the .pc file:

https://github.com/bsc-pm/nanos6/blob/master/m4/papi.m4#L25
https://github.com/icl-utk-edu/papi/blob/master/src/papi.pc.in#L8

Just a comment so I don't forget, it may be doable to fix nanos6 so that it takes the PAPI version from the .pc file: https://github.com/bsc-pm/nanos6/blob/master/m4/papi.m4#L25 https://github.com/icl-utk-edu/papi/blob/master/src/papi.pc.in#L8
abonerib commented 2025-10-01 15:13:09 +02:00
Author
Collaborator

Even if we fix this, papi does not cross-compile to risc-v.

I tried to fix it, adding the needed configuration flags for cross compilation, but it still fails because it misses some instructions:

mb.h:67:2: error: #error Need to define rmb for this architecture!
Even if we fix this, papi does not cross-compile to risc-v. I tried to fix it, adding the needed configuration flags for cross compilation, but it still fails because it misses some instructions: ``` mb.h:67:2: error: #error Need to define rmb for this architecture! ```
rarias commented 2025-10-01 15:18:45 +02:00
Owner

It seems they recently added support for RISC-V, but we use 7.1.0 which is too old:

b464420f3a

I'll prefer waiting until the next release of nixos, unless we need to backport it.

It seems they recently added support for RISC-V, but we use 7.1.0 which is too old: https://github.com/icl-utk-edu/papi/commit/b464420f3a2855b2c800413a4c5767b69e088087 I'll prefer waiting until the next release of nixos, unless we need to backport it.
👍 1
rarias marked this conversation as resolved
rarias commented 2025-09-30 18:21:13 +02:00
Owner
Copy Link

I would say to leave the rest of fixes for cross compilation to another PR, and here only focus on adding the target and splitting the dependencies as long as that doesn't break bsc-ci.all.

I would say to leave the rest of fixes for cross compilation to another PR, and here only focus on adding the target and splitting the dependencies as long as that doesn't break bsc-ci.all.
abonerib force-pushed CIstrictDeps from 32ecd3014d to 85d50f346c 2025-10-01 10:29:22 +02:00 Compare
abonerib added 1 commit 2025-10-01 12:59:39 +02:00
abonerib changed title from Fix nativeBuildInputs to WIP: Fix nativeBuildInputs 2025-10-01 13:00:11 +02:00
abonerib force-pushed CIstrictDeps from b3df6bb1be to ad17dbc06e 2025-10-01 15:05:41 +02:00 Compare
abonerib reviewed 2025-10-01 15:18:59 +02:00
overlay.nix Outdated
@ -103,2 +114,2 @@
printf '%s\n' $deps > $out
'';
inherit pkgs pkgsList;
inherit test tests;
abonerib commented 2025-10-01 15:18:59 +02:00
Author
Collaborator

We could rename test -> tests and tests -> testsList for consistency.

We could rename `test -> tests` and `tests -> testsList` for consistency.
👍 1
abonerib marked this conversation as resolved
abonerib added 1 commit 2025-10-01 15:25:53 +02:00
Rename test->tests and tests to testList
All checks were successful
CI / build:all (pull_request) Successful in 13s
Details
52e275434f
abonerib changed title from WIP: Fix nativeBuildInputs to Fix nativeBuildInputs and add cross compilation target 2025-10-01 15:31:32 +02:00
abonerib force-pushed CIstrictDeps from 52e275434f to 65df99aac2 2025-10-01 15:34:07 +02:00 Compare
rarias approved these changes 2025-10-01 15:41:21 +02:00
rarias force-pushed CIstrictDeps from 65df99aac2 to 92ee4a09d7 2025-10-01 15:54:06 +02:00 Compare
abonerib manually merged commit 92ee4a09d7 into master 2025-10-01 15:57:34 +02:00
This repo is archived. You cannot comment on pull requests.
Reviewers
No Reviewers
rarias
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
abonerib arocanon rarias
No Assignees
2 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rarias/bscpkgs#10
No description provided.
Delete Branch "abonerib/bscpkgs:CIstrictDeps"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?