BSC Nixpkgs: User guide
1 Introduction
This repository contains a set of nix packages used in the Barcelona
Supercomputing Center by the Programming Models group.
Some preliminary steps must be done manually to be able to build and
install packages (derivations in nix jargon).
1.1 Network access
In order to use nix you need to be able to download the sources from
Internet. Usually the download requires the ports 22, 80 and 443 to be
open for outgoing traffic.
Unfortunately, in some clusters (as is the case in xeon07) access to
Internet is disabled. However you can tunnel the connection by SSH to
your local machine, and then reach the Internet.
There are some guides on how to prepare the proxy server and the
tunnel in SSH such as:
https://www.seniorlinuxadmin.co.uk/ssh-over-proxy.html
In order to instruct nix to use the proxy connection, you will need to
export the https_proxy and http_proxy variables. In the xeon07 node is
already configured and you can automatically set those variables to
the correct address by loading:
$ . /scratch/nix/internet
Consider adding the command to your ~/.bashrc file so you don't need
to do it every time you want to use nix.
Now you should be able to reach the outside world by running:
$ curl google.com
301 Moved
301 Moved
The document has moved
here.
1.1 Prepare SSH keys
Package sources are usually downloaded directly from the git server,
so you must be able to access all repositories without a password
prompt.
Most repositories at https://pm.bsc.es/gitlab are open to read for
logged users, but there are some exceptions for example the nanos6
repository where you must have explicitly granted read access.
If you don't have a ssh key at ~/.ssh/*.pub create a new one without
password protection by running:
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (~/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ~/.ssh/id_rsa.
Your public key has been saved in ~/.ssh/id_rsa.pub.
...
By default it will create the private key at ~/.ssh/id_rsa. Copy the
contents of your public ssh key in ~/.ssh/id_rsa.pub and paste it in
GitLab at:
https://pm.bsc.es/gitlab/profile/keys
If you want to select another key rather than the default
~/.ssh/id_rsa then you must configure it for use in the ~/.ssh/config
file, adding:
Host bscpm02.bsc.es
IdentityFile ~/.ssh/your-private-key
Finally verify the SSH connection to the server works and you get a
greeting from the GitLab server with your username:
$ ssh git@bscpm02.bsc.es
PTY allocation request failed on channel 0
Welcome to GitLab, @rarias!
Connection to bscpm02.bsc.es closed.
Verify that you can access rarias/nanos6 repository (otherwise you
first need to ask to be granted read access), at:
https://pm.bsc.es/gitlab/rarias/nanos6
Finally, you should be able to download the rarias/nanos6 git
repository without any password interaction by running:
$ git clone git@bscpm02-bsc-es:rarias/nanos6.git
1.3 Prepare the bsc-nixpkgs repo
Once you have Internet and you have granted access to the PM GitLab
repositories you can begin down the rabbit hole of nix. First ensure
that the nix binaries are available in your machine:
$ nix --version
nix (Nix) 2.3.6
Now you are set to install packages with nix. Clone the bsc-nixpkgs
repository:
$ git clone git@bscpm02-bsc-es:rarias/bsc-nixpkgs.git
Nix looks in the current folder for a file named "default.nix" for
packages, so go to the repo:
$ cd bsc-nixpkgs
Now you should be able to build nanos6 from the git repository:
$ nix-build -A bsc.nanos6-git
The output is placed in the "result" symlink.
2. Basic usage of nix
Nix is a package manager which handles easily reproducibility and
configuration of packages and dependencies. See more info here:
https://nixos.org/nix/manual/
We will only cover the basic usage of nix for the BSC packages.
2.1 The user environment
All nix packages are stored under the /nix directory. When you need to
"install" some binary from nix, a symlink is added to a folder
included in the $PATH variable. In particular, you should have the
something similar added to your $PATH:
$ echo $PATH | sed 's/:/\n/g' | grep nix
/home/Computational/rarias/.nix-profile/bin
/nix/var/nix/profiles/default/bin
The first one is your custom installation of packages that are stored
in your home directory and the second one is the default installation
which contains the nix tools (which are installed in the /nix
directory as well).
When you need a program that is not available in your environment,
much like when you use "module load ..." you can use nix-env to modify
what is currently loaded. For example:
$ nix-env -iA nixpkgs.cowsay
The command will download (if not found already in the nix store),
compile (if necessary) and load the program `cowsay` from the nixpkgs
repository in the environment. You should be able to run it as:
$ cowsay "hello world"
_____________
< hello world >
-------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
You can inspect now the ~/.nix-profile/bin folder, and see that a new
symlink was added to the actual installation of the binary:
$ file ~/.nix-profile/bin/cowsay
/home/Computational/rarias/.nix-profile/bin/cowsay: symbolic link to
`/nix/store/673gczmhr5b449521srz2n7g1klykz6n-cowsay-3.03+dfsg2/bin/cowsay'
You can list the current packages installed in your environment by
running:
$ nix-env -q
cowsay-3.03+dfsg2
nix-2.3.6
Notice that this setup only affects your user environment. Also, it is
permanent for any new session until you modify the environment again
and is immediate, all sessions will have the new environment
instantaneously.
You can remove any package from the environment using:
$ nix-env -e cowsay
See the manual with `nix-env --help` if you want to know more details.
2.2 Building packages
Usually, all official packages are already compiled and distributed
from a cache server so you don't need to rebuild them again. However,
BSC packages are distributed only in source code form as we don't have
any binary cache server yet.
Nix will handle the build process without any user interaction (with a
few exceptions stated later, which you shouldn't have to worry). If
any other user has already built the package then the build process is
not needed, and the package is used as is.
In order to build a BSC package go to the `bsc-nixpkgs` directory, and
run:
$ nix-build -A bsc.dummy
The package will be built and installed in the /nix directory, then a
symlink is placed in the result directory:
$ find result/ -type f
result/
result/bin
result/bin/dummy
The way in which nix handles the packages and dependencies ensures
that the output of a compilation of any package is exactly the same.
You can check the reproducibility of the build by adding the "--check"
flag, which will rebuild the package and compare the checksum of every
file with the ones installed:
$ nix-build -A bsc.dummy --check
...
$ echo $?
0
A return code of zero ensures the output is bit by bit identical to
the one installed. There are some packages that try to include
non-reproducible information in the build process as the timestamp
which will produce an error. Those packages must be patched to ensure
the output is deterministic.
Notice that if you "cd" into the result/ directory you will be at /nix
directory (as you follow the symlink) where you don't have write
permission. Therefore if your program attempts to write to the current
directory it will fail. It is recommended to instead run your program
from the top directory:
$ result/bin/dummy
Hello world!
Or you can install it in the environment:
$ nix-env -i ./result
And "cd" into any directory where you want to output some files and
just run it by the name:
$ cd /tmp
$ dummy
Hello world!
Finally, you can remove it from the environment if you don't need it:
$ nix-env -e dummy
If you want to know more details use "nix-build --help" to see the
manual.
2.3 The build process
Each package is built following a programmable configuration
description in the nix language. Build in nix are performed under very
strict conditions. No access any file in the file system is allowed,
unless stated in the dependencies of the package which are then
available in the build environment.
There is no network access in the build process and other restrictions
are enforced that the build is deterministic. See more details here:
https://nixos.wiki/wiki/Nix#Sandboxing
In the "default.nix" file of the bsc-nixpkgs you can see the
definition for each package, for example the nbody app:
nbody = callPackage ./bsc/apps/nbody/default.nix {
stdenv = pkgs.gcc9Stdenv;
mpi = intel-mpi;
icc = icc;
tampi = tampi;
nanos6 = nanos6-git;
};
The compilation details are specified in the
"bsc/apps/nbody/default.nix" file. You can configure the package by
changing the inputs, for example, what specific implementation of
nanos6 or MPI you want to use. To change the MPI implementation to the
official MPICH package use:
nbody = callPackage ./bsc/apps/nbody/default.nix {
stdenv = pkgs.gcc9Stdenv;
mpi = pkgs.mpich; # Notice pkgs prefix for official packages
icc = icc;
tampi = tampi;
nanos6 = nanos6-git;
};
Then you can rebuild the nbody package:
$ nix-build -A bsc.nbody
...
And verify that the binary is indeed linked to MPICH now:
$ ldd result/bin/nbody_mpi.N2.2048.exe | grep mpi
libmpi.so.12 => /nix/store/dwkkcv78a5bs8smflpx9ppp3klhz3i98-mpich-3.3.2/lib/libmpi.so.12 (0x00007f6be0f07000)
If you modify a package which another package requires as a
dependency, nix will rebuild all required packages to propagate your
changes on demand.
However, if you come back to the original configuration, the package
will still be in the /nix store (unless the garbage collector was
manually run and removed your old build), so you don't need to rebuild
it again.
For example if nbody is configured back to use Intel MPI:
nbody = callPackage ./bsc/apps/nbody/default.nix {
stdenv = pkgs.gcc9Stdenv;
mpi = intel-mpi;
icc = icc;
tampi = tampi;
nanos6 = nanos6-git;
};
The build process now is not required:
$ nix-build -A bsc.nbody
/nix/store/rbq7wrjcmg6fzd6yhrlnkfvzcavdbdpc-nbody
$ ldd result/bin/nbody_mpi.N2.2048.exe | grep mpi
libmpifort.so.12 => /nix/store/jvsjvxj2a08340fpdrqbqix9z3mpp3bd-intel-mpi-2019.7.217/lib/libmpifort.so.12 (0x00007f3a00402000)
libmpi.so.12 => /nix/store/jvsjvxj2a08340fpdrqbqix9z3mpp3bd-intel-mpi-2019.7.217/lib/libmpi.so.12 (0x00007f39fed34000)
Take a look at the different package description files in the
bsc-nixpkgs repository if you want to understand more details. Also
the nix pills are a very good reference:
https://nixos.org/nixos/nix-pills/
/* vim: set ts=2 sw=2 tw=72 fo=watqc expandtab spell autoindent: */