rarias/jungle
2
0
Fork 3
Code Issues 63 Pull Requests 10 Actions Packages Projects Releases Wiki Activity

Add sandboxed version of nix-portable with tweaks to run on marenostrum #236

Manually merged
abonerib merged 22 commits from abonerib/jungle:nix-portable into master 2026-03-11 17:27:54 +01:00
Conversation 4 Commits 22 Files Changed 5 +667 -25
abonerib commented 2026-03-06 17:13:45 +01:00
Collaborator
Copy Link

Modified nix-portable to run without requiring host's programs to run. Instead, we install all we need through nix (either bundled statically in nix-portable or through nix in the first run).

Ran bsc.ci.all and all works except amd-uprof dowload

$ NP_LOCATION=$TMPDIR/nix nix build jungle#bsc.ci.all --no-link --print-out-paths
/nix/store/gkpc5mqi2s7i0v22ky57v1ynzsgmhskb-ci-all

EDIT: it took ~8minutes with --substituters "" jungle#bsc.ci.all -L --max-jobs 8

Some minor issues:

  • bwrap does not play well with the sgid in /gpfs/{scratch,projects} causing permission errors when sources are downloaded.
    • "Fixed" by unsetting it and chaning the group of $NP_LOCATION/.nix-portable to bsc, but it will break if subfolders where already created with the bit.
  • Cannot download AMD uprof (fixed output derivation). No internet in sandbox?
    • Thankfully it can pick it up from the jungle cache
    • "Fixed" by using fetchurl instead of runCommand
  • ANSI escape colors seem broken in my terminal when doing nix --help
Modified [nix-portable][1] to run without requiring host's programs to run. Instead, we install all we need through nix (either bundled statically in `nix-portable` or through `nix` in the first run). Ran `bsc.ci.all` and all works except amd-uprof dowload ```console $ NP_LOCATION=$TMPDIR/nix nix build jungle#bsc.ci.all --no-link --print-out-paths /nix/store/gkpc5mqi2s7i0v22ky57v1ynzsgmhskb-ci-all ``` EDIT: it took ~8minutes with `--substituters "" jungle#bsc.ci.all -L --max-jobs 8` # Some minor issues: - `bwrap` does not play well with the `sgid` in `/gpfs/{scratch,projects}` causing permission errors when sources are downloaded. - "Fixed" by unsetting it and chaning the group of `$NP_LOCATION/.nix-portable` to `bsc`, but it will break if subfolders where already created with the bit. - ~Cannot download AMD uprof (fixed output derivation). No internet in sandbox?~ - Thankfully it can pick it up from the jungle cache - "Fixed" by using `fetchurl` instead of `runCommand` - ANSI escape colors seem broken in my terminal when doing `nix --help` [1]: https://github.com/DavHau/nix-portable
abonerib added 14 commits 2026-03-06 17:13:45 +01:00
Re-enable nix-wrap 732e183562 
libcap is no longer broken upstream
Add nix-portable 087caf635b
Fix nix-portable pkgStatic symlink handling eb579e9429
Bind proc using --proc in nix-portable 896f57b88b
Restrict paths added to bwrap in nix-portable 7500dfa674
Remove sgid from .nix-portable and set group aeb649ff31 
This should prevent issues when putting it under
/gpfs/{projects,scratch} that have sgid and group=nobody.
Bind to busybox to /bin/sh in nix-portable e959289f71
Install netcat, ssh and git in nix-portable 6405d06899 
These are needed by nix in order to properly
download and build stuff.

busybox's netcat does not work since it doesn't
support -X.
Include bashInteractive in nix-portable 99bd3f72f7
Remove nix-portable tmpbin feature ccaaeb0d2f 
It will not work without the host /lib64/ld
Use nix bwrap in nix-portable 4bb55ab96b
Add jungle cache to nix-portable nix.conf 0cff2a317b
Bind /usr/bin/env in nix-portable 8c079d926c
Add meta with license to nix-portable
All checks were successful
CI / build:cross (pull_request) Successful in 8s
Details
CI / build:all (pull_request) Successful in 5m31s
Details
c84ccb0652
abonerib force-pushed nix-portable from c84ccb0652 to 871aab5b96 2026-03-06 17:15:36 +01:00 Compare
abonerib force-pushed nix-portable from 871aab5b96 to 9c44010fb5 2026-03-06 17:31:15 +01:00 Compare
abonerib reviewed 2026-03-09 10:59:30 +01:00
pkgs/nix-portable/default.nix Outdated
@@ -0,0 +213,4 @@
echo "extra-trusted-public-keys = jungle.bsc.es:pEc7MlAT0HEwLQYPtpkPLwRsGf80ZI26aj29zMw/HH0=" >> \$dir/conf/nix.conf
echo "extra-system-features = sys-devices" >> \$dir/conf/nix.conf
echo "extra-sandbox-paths = /sys/devices/system/cpu=/sys/devices/system/cpu /sys/devices/system/node=/sys/devices/system/node" >> \$dir/conf/nix.conf
abonerib commented 2026-03-09 10:59:30 +01:00
Author
Collaborator
Copy Link

This should be conditional on the sys-devices feature being enabled, as it is done in programs.nix-required-mounts.

This should be conditional on the `sys-devices` feature being enabled, as it is done in `programs.nix-required-mounts`.
abonerib marked this conversation as resolved
abonerib force-pushed nix-portable from 9c44010fb5 to dcdbcc5afa 2026-03-09 12:32:09 +01:00 Compare
abonerib force-pushed nix-portable from dcdbcc5afa to d2672716f4 2026-03-09 12:41:37 +01:00 Compare
abonerib reviewed 2026-03-09 12:43:57 +01:00
pkgs/amd-uprof/default.nix Outdated
@@ -42,1 +27,3 @@
'';
src = fetchurl {
url = "https://download.amd.com/developer/eula/uprof/uprof-5-1/${tarball}";
sha256 = "sha256-j9gxcBcIg6Zhc5FglUXf/VV9bKSo+PAKeootbN7ggYk=";
abonerib commented 2026-03-09 12:43:57 +01:00
Author
Collaborator
Copy Link

Checked that it still works with nix build .#amd-uprof.src --rebuild

Checked that it still works with `nix build .#amd-uprof.src --rebuild`
abonerib marked this conversation as resolved
abonerib force-pushed nix-portable from d2672716f4 to aa0c778a9b 2026-03-09 14:27:38 +01:00 Compare
abonerib force-pushed nix-portable from aa0c778a9b to 4d3e5ebba3 2026-03-09 14:33:07 +01:00 Compare
abonerib changed title from WIP: Add sandboxed version of nix-portable with tweaks to run on marenostrum to Add sandboxed version of nix-portable with tweaks to run on marenostrum 2026-03-09 15:51:40 +01:00
rarias force-pushed nix-portable from 4d3e5ebba3 to 2dbfeed817 2026-03-10 10:24:38 +01:00 Compare
rarias approved these changes 2026-03-11 14:00:15 +01:00
Dismissed
rarias left a comment
Owner
Copy Link

Thanks a lot!, this is very useful. I just left a nitpick to keep the overlay.nix with one line per package.

I tested it in glogin4 and it seems to work well so far. I would keep it for now for "experienced users" and see later if we can have a zero-configuration (or near zero) for newcomers. For that we would need to test MPI as well and make sure /usr and /lib is not present at run time, among other things.

Thanks a lot!, this is very useful. I just left a nitpick to keep the overlay.nix with one line per package. I tested it in glogin4 and it seems to work well so far. I would keep it for now for "experienced users" and see later if we can have a zero-configuration (or near zero) for newcomers. For that we would need to test MPI as well and make sure /usr and /lib is not present at run time, among other things.
overlay.nix Outdated
@@ -45,0 +47,4 @@
xz = final.pkgsStatic.xz;
zstd = final.pkgsStatic.zstd;
bashInteractive = final.pkgsStatic.bashInteractive;
};
rarias commented 2026-03-09 16:07:01 +01:00
Owner
Copy Link

Could we pass pkgs = final and take those from inside the derivation input arguments? That way we can keep the definition in one line and they are still overridable if needed.

Could we pass `pkgs = final` and take those from inside the derivation input arguments? That way we can keep the definition in one line and they are still overridable if needed.
abonerib commented 2026-03-11 15:21:39 +01:00
Author
Collaborator
Copy Link

Changed to use <pkgname>Static this should allow overrides and keep the overlay clean.

Decided to remove proot since the static version did not build.

The only difference is that now we use the native zstd instead of the static one:

$ nix-diff result-old result --color never
- result-old:{out}
+ result:{out}
• The set of input derivation names do not match:
    - proot
• The input derivation named `cacerts` differs
  - /nix/store/ixlig68fm19xhsm4j3alx4gzvxpk4n6v-cacerts.drv:{out}
  + /nix/store/s2prdxi6p8mwpmz2d1jj11wpk1v7xg6r-cacerts.drv:{out}
  • The set of input derivation names do not match:
      - zstd-static-x86_64-unknown-linux-musl-1.5.7
      + zstd
  • The environments do not match:
      buildCommand=''
      cat /nix/store/2mcs9c3r0ja4gz250673nmmil1ji9wac-nss-cacert-3.117/etc/ssl/certs/ca-bundle.crt | ←/nix/store/4gx0f1cg1k85fhi5iqhpdrfs3pf06mwm-zstd-static-x86_64-unknown-linux-musl-1.5.7-bin/bin/zstd←→/nix/store/b2nandsp4dn9k2zpwrxlrjcpjnz0h185-zstd/bin/zstd→ -19 > $out
  ''
• Skipping environment comparison
Changed to use `<pkgname>Static` this should allow overrides and keep the overlay clean. Decided to remove `proot` since the static version did not build. The only difference is that now we use the native zstd instead of the static one: ```console $ nix-diff result-old result --color never - result-old:{out} + result:{out} • The set of input derivation names do not match: - proot • The input derivation named `cacerts` differs - /nix/store/ixlig68fm19xhsm4j3alx4gzvxpk4n6v-cacerts.drv:{out} + /nix/store/s2prdxi6p8mwpmz2d1jj11wpk1v7xg6r-cacerts.drv:{out} • The set of input derivation names do not match: - zstd-static-x86_64-unknown-linux-musl-1.5.7 + zstd • The environments do not match: buildCommand='' cat /nix/store/2mcs9c3r0ja4gz250673nmmil1ji9wac-nss-cacert-3.117/etc/ssl/certs/ca-bundle.crt | ←/nix/store/4gx0f1cg1k85fhi5iqhpdrfs3pf06mwm-zstd-static-x86_64-unknown-linux-musl-1.5.7-bin/bin/zstd←→/nix/store/b2nandsp4dn9k2zpwrxlrjcpjnz0h185-zstd/bin/zstd→ -19 > $out '' • Skipping environment comparison ```
👍 1
rarias marked this conversation as resolved
abonerib added 1 commit 2026-03-11 14:53:30 +01:00
Rework nix-portable derivation arguments
All checks were successful
CI / build:cross (pull_request) Successful in 8s
Details
CI / build:all (pull_request) Successful in 39s
Details
19b1862606
rarias requested review from rarias 2026-03-11 17:17:11 +01:00
rarias approved these changes 2026-03-11 17:17:17 +01:00
abonerib force-pushed nix-portable from 19b1862606 to 54dfaa9dff 2026-03-11 17:23:24 +01:00 Compare
abonerib manually merged commit 54dfaa9dff into master 2026-03-11 17:27:54 +01:00
Sign in to join this conversation.
Reviewers
No Reviewers
abonerib
rarias
Labels
Clear labels
bug
ceph
ci
config
cross
hw
io
kernel
mpi
net
nix
ops
repair
slurm
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
abonerib anavarro arocanon varcila rarias
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rarias/jungle#236
Delete Branch "abonerib/jungle:nix-portable"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?