rarias/jungle
2
0
Fork 3
Code Issues 57 Pull Requests 7 Actions Packages Projects Releases Wiki Activity

Allow fetching from multiple *private* git repositories mirrors #202

Open
rarias wants to merge 2 commits from robust-fetchGit into master
pull from: robust-fetchGit
merge into: rarias:master
Conversation 2 Commits 2 Files Changed 5 +119 -235
rarias commented 2025-10-20 16:27:40 +02:00
Owner
Copy Link

Introduces the builtin.catchAll function which can catch all errors produced by builtins, including those that builtins.fetchGit raise when the remote is down. This is then used to implement fetchGitMirror which is equal to builtins.fetchGit, but accepts a list of urls instead of just one.

Fixes: #200

Introduces the `builtin.catchAll` function which can catch all errors produced by builtins, including those that builtins.fetchGit raise when the remote is down. This is then used to implement `fetchGitMirror` which is equal to builtins.fetchGit, but accepts a list of urls instead of just one. Fixes: https://jungle.bsc.es/git/rarias/jungle/issues/200
rarias added 2 commits 2025-10-20 16:27:41 +02:00
Add nix patch to implement builtins.catchAll 3edf00af0c
Add fetchGitMirror function
Some checks failed
CI / build:cross (pull_request) Successful in 10s
Details
CI / build:all (pull_request) Failing after 22s
Details
14a67cda34
rarias force-pushed robust-fetchGit from 14a67cda34 to 76aa169fe8 2025-10-20 16:31:12 +02:00 Compare
rarias force-pushed robust-fetchGit from 76aa169fe8 to cc63e6f3f3 2025-10-28 12:42:59 +01:00 Compare
rarias force-pushed robust-fetchGit from cc63e6f3f3 to 9107cad699 2025-11-21 15:14:49 +01:00 Compare
rarias force-pushed robust-fetchGit from 9107cad699 to 9b491e8d01 2025-11-21 15:17:59 +01:00 Compare
rarias changed title from WIP: Allow fetching from multiple *private* git repositories mirrors to Allow fetching from multiple *private* git repositories mirrors 2025-11-21 15:25:07 +01:00
rarias requested review from abonerib 2025-11-21 15:25:34 +01:00
abonerib reviewed 2025-12-02 12:59:06 +01:00
overlay.nix
@@ -98,2 +98,4 @@
};
# Load our custom lib functions with import, callPackage fails.
lib' = import ./pkgs/lib.nix { lib = prev.lib; };
abonerib commented 2025-12-02 12:59:06 +01:00
Collaborator
Copy Link

We should also export the custom lib in the flake:

jungle/flake.nix Line 46 in a173af654f
lib = nixpkgs.lib // {

We should also export the custom lib in the flake: https://jungle.bsc.es/git/rarias/jungle/src/commit/a173af654f6290c48f1a2fd128c9d008283b157d/flake.nix#L46
abonerib requested changes 2025-12-09 17:14:04 +01:00
overlay.nix
@@ -36,6 +35,7 @@ let
mpich = callPackage ./pkgs/mpich/default.nix { mpich = prev.mpich; };
nanos6 = callPackage ./pkgs/nanos6/default.nix { };
nanos6Debug = final.nanos6.override { enableDebug = true; };
nix = callPackage ./pkgs/nix/default.nix { nix = prev.nix; };
abonerib commented 2025-12-09 17:00:19 +01:00
Collaborator
Copy Link

Does the patch still apply to nix_2_31 (new stable in 25.11) ?

Packages that contain /bin/nix in nixpkgs 25.11 (excluding lix):
- nixVersions.nix_2_29.out
- nixVersions.nix_2_30.out
- nixVersions.nix_2_28.out (stable in nixpkgs 25.05)
- nix.out                  (nixVersions.stable)
- nixVersions.stable.out   (nix_2_31)
- nixVersions.latest.out   (nix_2_32)
- nixVersions.git.out

The closest to nix_2_31 in 25.05 is nixVersions.git at (2.31.0pre20250711_ab3cd76e)

Does the patch still apply to `nix_2_31` (new stable in 25.11) ? ``` Packages that contain /bin/nix in nixpkgs 25.11 (excluding lix): - nixVersions.nix_2_29.out - nixVersions.nix_2_30.out - nixVersions.nix_2_28.out (stable in nixpkgs 25.05) - nix.out (nixVersions.stable) - nixVersions.stable.out (nix_2_31) - nixVersions.latest.out (nix_2_32) - nixVersions.git.out ``` The closest to `nix_2_31` in 25.05 is `nixVersions.git` at (`2.31.0pre20250711_ab3cd76e`)
pkgs/lib.nix
@@ -0,0 +2,4 @@
let
# If not supported, fall back to tryEval, which will fail in the first case.
safeCatchAll = if (builtins ? catchAll)
abonerib commented 2025-12-09 15:50:41 +01:00
Collaborator
Copy Link

nit: no need for parentheses

nit: no need for parentheses
pkgs/lib.nix
@@ -0,0 +4,4 @@
# If not supported, fall back to tryEval, which will fail in the first case.
safeCatchAll = if (builtins ? catchAll)
then builtins.catchAll
else e: (builtins.tryEval e) // { msg = ""; };
abonerib commented 2025-12-09 15:58:46 +01:00
Collaborator
Copy Link

I would add a warning here, something like: catchAll was not available on current nix, using tryEval as a fallback.

Package consumers may be confused by it, so I would use builtins.traceVerbose instead of lib.warn so it's only shown when doing nix --verbose

I would add a warning here, something like: `catchAll` was not available on current nix, using `tryEval` as a fallback. Package consumers may be confused by it, so I would use [`builtins.traceVerbose`](https://noogle.dev/f/builtins/traceVerbose) instead of `lib.warn` so it's only shown when doing `nix --verbose`
pkgs/lib.nix
@@ -0,0 +16,4 @@
let
r = safeFetch url;
in
if (r.success) then r
abonerib commented 2025-12-09 15:40:25 +01:00
Collaborator
Copy Link

This is lib.warnIf

This is [`lib.warnIf`](https://noogle.dev/f/lib/warnIf)
pkgs/lib.nix
@@ -0,0 +19,4 @@
if (r.success) then r
else lib.warn "cannot fetch ${url}, trying next
mirror:${builtins.replaceStrings ["\n" ] ["\n> "] ("\n"+r.msg)}" r;
fetchList = lib.map (url: complain url) urls;
abonerib commented 2025-12-09 15:33:18 +01:00
Collaborator
Copy Link

No need for the lambda (map (url: complain url) --> map complain)

No need for the lambda (`map (url: complain url) --> map complain`)
All checks were successful
CI / build:cross (pull_request) Successful in 13s
Details
CI / build:all (pull_request) Successful in 5m33s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin robust-fetchGit:robust-fetchGit
git checkout robust-fetchGit
Sign in to join this conversation.
Reviewers
No Reviewers
abonerib
Labels
Clear labels
bug
ceph
ci
config
cross
hw
io
kernel
mpi
net
nix
ops
repair
slurm
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
abonerib arocanon varcila rarias
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rarias/jungle#202
Delete Branch "robust-fetchGit"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?