diff --git a/README b/README deleted file mode 100644 index 08f3b32..0000000 --- a/README +++ /dev/null @@ -1,462 +0,0 @@ - - - bscpkgs: User guide - - -ABSTRACT - - This repository contains a set of nix packages used in the Barcelona - Supercomputing Center by the Programming Models group. - - The current setup uses the xeon07 machine to build packages, which are - automatically uploaded to MareNostrum4, due to lack of permissions in - the latter to perform the build safely. - - Some preliminary steps must be done manually to be able to build and - install packages (derivations in nix jargon). - -1. Introduction - - To easily connect to xeon07 in one step, setup the SSH (for version - 7.3 and upwards) configuration file in ~/.ssh/config adding these - lines: - - Host cobi - HostName ssflogin.bsc.es - User your-username-here - - Host xeon07 - ProxyJump cobi - HostName xeon07 - User your-username-here - - You should be able to connect with: - - laptop$ ssh xeon07 - -1.1 Network access - - In order to use nix you would need to be able to download the sources - from Internet. Usually the download requires the ports 22, 80 and 443 - to be open for outgoing traffic. - - Check that you have network access in xeon07 provided by the - environment variables "http_proxy" and "https_proxy". Try to fetch a - webpage with curl, to ensure the proxy is working: - - xeon07$ curl x.com - x - -1.2 SSH keys - - Package sources are usually downloaded directly from the git server, - so you must be able to access all repositories without a password - prompt. - - Most repositories at https://pm.bsc.es/gitlab are open to read for - logged in users, but there are some exceptions (for example the nanos6 - repository) where you must have explicitly granted read access. - - If you don't have a ssh key at ~/.ssh/*.pub in xeon07 create a new one - without password protection by running: - - xeon07$ ssh-keygen - Generating public/private rsa key pair. - Enter file in which to save the key (~/.ssh/id_rsa): - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in ~/.ssh/id_rsa. - Your public key has been saved in ~/.ssh/id_rsa.pub. - ... - - By default it will create the private key at ~/.ssh/id_rsa. Copy the - contents of your public ssh key in ~/.ssh/id_rsa.pub and paste it in - GitLab at: - - https://pm.bsc.es/gitlab/profile/keys - - Then, configure it for use in the ~/.ssh/config file, adding: - - Host bscpm03.bsc.es - IdentityFile ~/.ssh/id_rsa - - Finally verify the SSH connection to the server works and you get a - greeting from the GitLab server with your username: - - xeon07$ ssh git@bscpm03.bsc.es - PTY allocation request failed on channel 0 - Welcome to GitLab, @rarias! - Connection to bscpm03.bsc.es closed. - - Verify that you can access nanos6/nanos6 repository (otherwise you - first need to ask to be granted read access), at: - - https://pm.bsc.es/gitlab/nanos6/nanos6 - - Finally, you should be able to download the nanos6/nanos6 git - repository without any password interaction by running: - - xeon07$ git clone git@bscpm03.bsc.es:nanos6/nanos6.git - - You will also need to access MareNostrum 4 from the xeon07 node, in - order to submit experiments. Add the following lines as well to the - ~/.ssh/config file and set your user name: - - Host mn0 mn1 mn2 - User your-mn4-username - IdentityFile ~/.ssh/id_rsa - - Then copy the key to MareNostrum 4 (it will ask you the first time for - your password): - - xeon07$ ssh-copy-id -i ~/.ssh/id_rsa.pub mn1 - - And ensure that you can connect without a password: - - xeon07$ ssh mn1 - ... - login1$ - -1.3 The bscpkgs repo - - Once you have Internet and you have granted access to the PM GitLab - repositories you can begin building software with nix. First ensure - that the nix binaries are available from your shell in xeon07: - - xeon07$ nix --version - nix (Nix) 2.3.6 - - Now you are ready to build and install packages with nix. Clone the - bscpkgs repository: - - xeon07$ git clone git@bscpm03.bsc.es:rarias/bscpkgs.git - - Nix looks in the current folder for a file named "default.nix" for - packages, so go to the repo directory: - - xeon07$ cd bscpkgs - - Now you should be able to build nanos6: - - xeon07$ nix-build -A bsc.nanos6 - .. - /nix/store/3i0qkdywm9xjv2cm1ldx9smb552sf6r1-nanos6-2.4-6f10a32 - - The installation is placed in the nix store (with the path stated in - the last line of the build process), with the "result" symbolic link - pointing to the same location: - - xeon07$ readlink result - /nix/store/3i0qkdywm9xjv2cm1ldx9smb552sf6r1-nanos6-2.4-6f10a32 - -1.4 Configuration of mn4 (MareNostrum 4) - - In order to execute the programs built at xeon07, you first need to - enter nix environment. To do so, add to the end of the file ~/.bashrc - in mn4 the following line: - - export PATH=/gpfs/projects/bsc15/nix/bin:$PATH - - Then logout and login again (our source the ~/.bashrc file) and you - will now have the `nix-setup` command available. This command executes - a new shell where the /nix store is available. To execute it: - - mn4$ nix-setup - - Now you will see a new shell, where you can access the nix store: - - nix|mn4$ ls /nix - gcroots profiles store var - - The last build of nanos6 can be also found in mn4 at the same - location: - - /nix/store/3i0qkdywm9xjv2cm1ldx9smb552sf6r1-nanos6-2.4-6f10a32 - - Remember to enter the nix environment by running `nix-setup` when you - need something from the nix store. - - You cannot perform any build operations from mn4: to do so use the - xeon07 machine. - -2. Basic usage of nix - - Nix is a package manager which handles easily reproducibility and - configuration of packages and dependencies. See more info here: - - https://nixos.org/nix/manual/ - - We will only cover the basic usage of nix for the BSC packages. - -2.1 The user environment - - All nix packages are stored under the /nix directory. When you need to - "install" some binary from nix, a symlink is added to a folder - included in the $PATH variable. In particular, you should have - something similar added to your $PATH: - - xeon07$ echo $PATH | sed 's/:/\n/g' | grep nix - /home/Computational/rarias/.nix-profile/bin - /nix/var/nix/profiles/default/bin - - The first one is your custom installation of packages that are stored - in your home directory and the second one is the default installation - which contains the nix tools (which are installed in the /nix - directory as well). - - Use `nix search` to look for official packages in the "nixpkgs" - channel (the default repository of packages): - - xeon07$ nix search cowsay - warning: using cached results; pass '-u' to update the cache - * cowsay (cowsay) - A program which generates ASCII pictures of a cow with a message - - * neo-cowsay (neo-cowsay) - Cowsay reborn, written in Go - - * ponysay (ponysay-3.0.3) - Cowsay reimplemention for ponies - - * tewisay (tewisay-unstable-2017-04-14) - Cowsay replacement with unicode and partial ansi escape support - - When you need a program that is not available in your environment, - much like when you use "module load ..." you can use nix-env to modify - what is currently loaded. For example: - - xeon07$ nix-env -iA nixpkgs.cowsay - - Notice that you should specify the prefix "nixpkgs." before. The - command will download (if not found already in the nix store), compile - (if necessary) and load the program `cowsay` from the nixpkgs - repository in the environment. You should be able to run it as: - - xeon07$ cowsay "hello world" - _____________ - < hello world > - ------------- - \ ^__^ - \ (oo)\_______ - (__)\ )\/\ - ||----w | - || || - - You can now inspect the ~/.nix-profile/bin folder, and see that a new - symlink was added to the actual installation of the binary: - - xeon07$ file ~/.nix-profile/bin/cowsay - /home/Computational/rarias/.nix-profile/bin/cowsay: symbolic link to - `/nix/store/673gczmhr5b449521srz2n7g1klykz6n-cowsay-3.03+dfsg2/bin/cowsay' - - You can list the current packages installed in your environment by - running: - - xeon07$ nix-env -q - cowsay-3.03+dfsg2 - nix-2.3.6 - - Notice that this setup only affects your user environment. Also, it is - permanent for any new session until you modify the environment again - and is immediate, all sessions will have the new environment - instantaneously. - - You can remove any package from the environment using: - - xeon07$ nix-env -e cowsay - - See the manual with `nix-env --help` if you want to know more details. - -2.2 Building packages - - Usually, all official packages are already compiled and distributed - from a cache server so you don't need to rebuild them again. However, - BSC packages are distributed only in source code form as we don't have - any binary cache server yet. - - Nix will handle the build process without any user interaction (with a - few exceptions which you shouldn't have to worry). If any other user - has already built the package then the build process is not needed, - and the package is used as is. - - In order to build a BSC package go to the `bscpkgs` directory, and - run: - - xeon07$ nix-build -A bsc.dummy - - Notice the "bsc." prefix for BSC packages. The package will be built - and installed in the /nix directory, then a symlink is placed in the - result directory: - - xeon07$ find result/ -type f - result/ - result/bin - result/bin/dummy - - The way in which nix handles the packages and dependencies ensures - that the environment of the build process of any package is exactly - the same, so the generated output should be the same if the builds are - deterministic. - - You can check the reproducibility of the build by adding the "--check" - flag, which will rebuild the package and compare the checksum of every - file with the ones previously built: - - xeon07$ nix-build -A bsc.dummy --check - ... - xeon07$ echo $? - 0 - - A return code of zero ensures the output is bit by bit identical to - the one installed. There are some packages that include - indeterministic information in the build process (such as the - timestamp of the current time) which will produce an error. Those - packages must be patched to ensure the output is deterministic. - - Notice that if you "cd" into the "result/" directory you will be at - /nix directory (as you have follow the symlink) where you don't have - write permission. Therefore if your program attempts to write to the - current directory it will fail. It is recommended to instead run your - program from the top directory: - - xeon07$ result/bin/dummy - Hello world! - - Or you can install it in the environment: - - xeon07$ nix-env -i ./result - - And "cd" into any directory where you want to output some files and - just run it by the name: - - xeon07$ cd /tmp - xeon07$ dummy - Hello world! - - Finally, you can remove it from the environment if you don't need it: - - xeon07$ nix-env -e dummy - - If you want to know more details use "nix-build --help" to see the - manual. - -2.3 The build process - - Each package is built following a programmable configuration - description in the nix language. Builds in nix are performed under - very strict conditions. No access to any file in the file system is - allowed, unless stated in the dependencies, which are in the /nix - store only. - - There is no network access in the build process and other restrictions - are enforced so that the build environment is reproducible. See more - details here: - - https://nixos.wiki/wiki/Nix#Sandboxing - - The top level "default.nix" file of the bscpkgs serves as a index - of all BSC packages. You can see the definition for each package, for - example the nbody app: - - nbody = callPackage ./bsc/apps/nbody/default.nix { - stdenv = pkgs.gcc9Stdenv; - mpi = intel-mpi; - icc = icc; - tampi = tampi; - nanos6 = nanos6-git; - }; - - The compilation details are specified in the - "bsc/apps/nbody/default.nix" file. You can configure the package by - changing the inputs, for example, what specific implementation of - nanos6 or MPI you want to use. To change the MPI implementation to the - official MPICH package use: - - nbody = callPackage ./bsc/apps/nbody/default.nix { - stdenv = pkgs.gcc9Stdenv; - mpi = pkgs.mpich; # Notice pkgs prefix for official packages - icc = icc; - tampi = tampi; - nanos6 = nanos6-git; - }; - - Then you can rebuild the nbody package: - - xeon07$ nix-build -A bsc.nbody - ... - - And verify that the binary is indeed linked to MPICH now: - - xeon07$ ldd result/bin/nbody_mpi.N2.2048.exe | grep mpi - libmpi.so.12 => /nix/store/dwkkcv78a5bs8smflpx9ppp3klhz3i98-mpich-3.3.2/lib/libmpi.so.12 (0x00007f6be0f07000) - - If you modify a package which another package requires as a - dependency, nix will rebuild all required packages to propagate your - changes on demand. - - However, if you come back to the original configuration, the package - will still be in the /nix store (unless the garbage collector was - manually run and removed your old build), so you don't need to rebuild - it again. - - For example if nbody is configured back to use Intel MPI: - - nbody = callPackage ./bsc/apps/nbody/default.nix { - stdenv = pkgs.gcc9Stdenv; - mpi = intel-mpi; - icc = icc; - tampi = tampi; - nanos6 = nanos6-git; - }; - - The build process now is not required: - - xeon07$ nix-build -A bsc.nbody - /nix/store/rbq7wrjcmg6fzd6yhrlnkfvzcavdbdpc-nbody - xeon07$ ldd result/bin/nbody_mpi.N2.2048.exe | grep mpi - libmpifort.so.12 => /nix/store/jvsjvxj2a08340fpdrqbqix9z3mpp3bd-intel-mpi-2019.7.217/lib/libmpifort.so.12 (0x00007f3a00402000) - libmpi.so.12 => /nix/store/jvsjvxj2a08340fpdrqbqix9z3mpp3bd-intel-mpi-2019.7.217/lib/libmpi.so.12 (0x00007f39fed34000) - - Take a look at the different package description files in the - bscpkgs repository if you want to understand more details. Also - the nix pills are a very good reference: - - https://nixos.org/nixos/nix-pills/ - -2.4 Debugging the build process - - It may happen that the build process fails in an unexpected way. Most - problems are related to missing dependencies and can be easily found - by looking at the error messages. - - Other build problems are more subtle and require more debugging time. - One way of inspecting a build problem is by adding the breakpointHook - hook to the nativeBuildInputs array in a nix derivation (see - https://nixos.org/nixpkgs/manual/#ssec-setup-hooks for more info), - which will stop the build process and allow a shell to be attached to - the sandbox. - - xeon07$ nix-build -A bsc.nbody - ... - /nix/store/gvqm2yc9xx4vh3nglgckz8siya66jnkx-stdenv-linux/setup: line - 83: fake-missing-command: command not found - build failed in buildPhase with exit code 127 - To attach install cntr and run the following command as root: - - cntr attach -t command \ - cntr-/nix/store/sk2nsj7xfr62cjk6m3725ydfyswqz7n1-nbody - - The command must run as root user, so you can use `sudo -i` to run it, - (the -i option is required to load the shell profile which provides - the nix path containing the cntr tool): - - xeon$ sudo -i cntr attach -t command \ - cntr-/nix/store/sk2nsj7xfr62cjk6m3725ydfyswqz7n1-nbody - nixbld@localhost:/var/lib/cntr> ls - bin build dev etc nix proc tmp var - - Then you can inspect the build environment to see why the build - failed. Source the build/env-vars file to get the same environment - variables (which include the $PATH) of the build process. - -/* vim: set ts=2 sw=2 tw=72 fo=watqc expandtab spell autoindent: */ diff --git a/README.md b/README.md new file mode 100644 index 0000000..aecda86 --- /dev/null +++ b/README.md @@ -0,0 +1,8 @@ +This repository contains the garlic benchmark with all the BSC +packages and patches required to compile it. + +See the user guide at + (also available in +[PDF](https://pm.bsc.es/ftp/garlic/ug.pdf)). + +Questions?