Show breadcrumbs in the web

This reverts commit aaefddc44a9073166ac52b8bd56ac96258d3b053.

.gitignore

@@ -0,0 +1,2 @@
+*.swp
+/result

doc/install.md

@@ -0,0 +1,152 @@
+# Installing NixOS in a new node
+
+This article shows the steps to install NixOS in a node following the
+configuration of the repo.
+
+## Enable the serial console
+
+By default, the nodes have the serial console disabled in the GRUB and also boot
+without the serial enabled.
+
+To enable the serial console in the GRUB, set in /etc/default/grub the following
+lines:
+
+```
+GRUB_TERMINAL="console serial"
+GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
+```
+
+To boot Linux with the serial enabled, so you can see the boot log and login via
+serial set:
+
+```
+GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 console=tty0"
+```
+
+Then update the grub config:
+
+```
+# grub2-mkconfig -o /boot/grub2/grub.cfg
+```
+
+And reboot.
+
+## Prepare the disk
+
+Create a main partition and label it `nixos` following [the manual][1].
+
+[1]: https://nixos.org/manual/nixos/stable/index.html#sec-installation-manual-partitioning.
+
+```
+# disk=/dev/sdX
+# parted $disk -- mklabel msdos
+# parted $disk -- mkpart primary 1MB -8GB
+# parted $disk -- mkpart primary linux-swap -8GB 100%
+# parted $disk -- set 1 boot on
+```
+
+Then create an etx4 filesystem, labeled `nixos` where the system will be
+installed. **Ensure that no other partition has the same label.**
+
+```
+# mkfs.ext4 -L nixos "${disk}1"
+# mkswap -L swap "${disk}2"
+# mount ${disk}1 /mnt
+# lsblk -f $disk
+NAME   FSTYPE LABEL UUID                                 MOUNTPOINT
+sdX
+`-sdX1 ext4   nixos 10d73b75-809c-4fa3-b99d-4fab2f0d0d8e /mnt
+```
+
+## Prepare nix and nixos-install
+
+Mount the nix store from the hut node in read-only /nix.
+
+```
+# mkdir /nix
+# mount -o ro hut:/nix /nix
+```
+
+Get the nix binary and nixos-install tool from hut:
+
+```
+# ssh hut 'readlink -f $(which nix)'
+/nix/store/0sxbaj71c4c4n43qhdxm31f56gjalksw-nix-2.13.3/bin/nix
+# ssh hut 'readlink -f $(which nixos-install)'
+/nix/store/9yq8ps06ysr2pfiwiij39ny56yk3pdcs-nixos-install/bin/nixos-install
+```
+
+And add them to the PATH:
+
+```
+# export PATH=$PATH:/nix/store/0sxbaj71c4c4n43qhdxm31f56gjalksw-nix-2.13.3/bin
+# export PATH=$PATH:/nix/store/9yq8ps06ysr2pfiwiij39ny56yk3pdcs-nixos-install/bin/
+# nix --version
+nix (Nix) 2.13.3
+```
+
+## Adapt owl configuration
+
+Clone owl repo:
+
+```
+$ git clone git@bscpm03.bsc.es:rarias/owl.git
+$ cd owl
+```
+
+Edit the configuration to your needs.
+
+## Install from another Linux OS
+
+Install nixOS into the storage drive.
+
+```
+# nixos-install --flake --root /mnt .#xeon0X
+```
+
+At this point, the nixOS grub has been installed into the nixos device, which
+is not the default boot device. To keep both the old Linux and NixOS grubs, add
+an entry into the old Linux grub to jump into the new grub.
+
+```
+# echo "
+
+menuentry 'NixOS' {
+    insmod chain
+    search --no-floppy --label nixos --set root
+    configfile /boot/grub/grub.cfg
+} " >> /etc/grub.d/40_custom
+```
+
+Rebuild grub config.
+
+```
+# grub2-mkconfig -o /boot/grub/grub.cfg
+```
+
+To boot into NixOS manually, reboot and select NixOS in the grub menu to boot
+into NixOS.
+
+To temporarily boot into NixOS only on the next reboot run:
+
+```
+# grub2-reboot 'NixOS'
+```
+
+To permanently boot into NixOS as the default boot OS, edit `/etc/default/grub/`:
+
+```
+GRUB_DEFAULT='NixOS'
+```
+
+And update grub.
+
+```
+# grub2-mkconfig -o /boot/grub/grub.cfg
+```
+
+## Build the nixos kexec image
+
+```
+# nix build .#nixosConfigurations.xeon02.config.system.build.kexecTree -v
+```

flake.lock

@@ -0,0 +1,114 @@
+{
+  "nodes": {
+    "agenix": {
+      "inputs": {
+        "darwin": "darwin",
+        "home-manager": "home-manager",
+        "nixpkgs": [
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1690228878,
+        "narHash": "sha256-9Xe7JV0krp4RJC9W9W9WutZVlw6BlHTFMiUP/k48LQY=",
+        "owner": "ryantm",
+        "repo": "agenix",
+        "rev": "d8c973fd228949736dedf61b7f8cc1ece3236792",
+        "type": "github"
+      },
+      "original": {
+        "owner": "ryantm",
+        "repo": "agenix",
+        "type": "github"
+      }
+    },
+    "bscpkgs": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1694077645,
+        "narHash": "sha256-72bvRBhq8Q8V6ibsR9lyBE92V2EC6C6Ek3J5cOM79So=",
+        "ref": "refs/heads/master",
+        "rev": "6122fef92701701e1a0622550ac0fc5c2beb5906",
+        "revCount": 860,
+        "type": "git",
+        "url": "https://pm.bsc.es/gitlab/rarias/bscpkgs.git"
+      },
+      "original": {
+        "type": "git",
+        "url": "https://pm.bsc.es/gitlab/rarias/bscpkgs.git"
+      }
+    },
+    "darwin": {
+      "inputs": {
+        "nixpkgs": [
+          "agenix",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1673295039,
+        "narHash": "sha256-AsdYgE8/GPwcelGgrntlijMg4t3hLFJFCRF3tL5WVjA=",
+        "owner": "lnl7",
+        "repo": "nix-darwin",
+        "rev": "87b9d090ad39b25b2400029c64825fc2a8868943",
+        "type": "github"
+      },
+      "original": {
+        "owner": "lnl7",
+        "ref": "master",
+        "repo": "nix-darwin",
+        "type": "github"
+      }
+    },
+    "home-manager": {
+      "inputs": {
+        "nixpkgs": [
+          "agenix",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1682203081,
+        "narHash": "sha256-kRL4ejWDhi0zph/FpebFYhzqlOBrk0Pl3dzGEKSAlEw=",
+        "owner": "nix-community",
+        "repo": "home-manager",
+        "rev": "32d3e39c491e2f91152c84f8ad8b003420eab0a1",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "home-manager",
+        "type": "github"
+      }
+    },
+    "nixpkgs": {
+      "locked": {
+        "lastModified": 1693663421,
+        "narHash": "sha256-ImMIlWE/idjcZAfxKK8sQA7A1Gi/O58u5/CJA+mxvl8=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "e56990880811a451abd32515698c712788be5720",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "ref": "nixos-unstable",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "root": {
+      "inputs": {
+        "agenix": "agenix",
+        "bscpkgs": "bscpkgs",
+        "nixpkgs": "nixpkgs"
+      }
+    }
+  },
+  "root": "root",
+  "version": 7
+}

flake.nix

@@ -0,0 +1,31 @@
+{
+  inputs = {
+    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
+    agenix.url = "github:ryantm/agenix";
+    agenix.inputs.nixpkgs.follows = "nixpkgs";
+    bscpkgs.url = "git+https://pm.bsc.es/gitlab/rarias/bscpkgs.git";
+    bscpkgs.inputs.nixpkgs.follows = "nixpkgs";
+  };
+
+  outputs = { self, nixpkgs, agenix, bscpkgs, ... }:
+let
+  mkConf = name: nixpkgs.lib.nixosSystem {
+    system = "x86_64-linux";
+    specialArgs = { inherit nixpkgs bscpkgs agenix; theFlake = self; };
+    modules = [ "${self.outPath}/m/${name}/configuration.nix" ];
+  };
+in
+  {
+    nixosConfigurations = {
+      hut   = mkConf "hut";
+      owl1  = mkConf "owl1";
+      owl2  = mkConf "owl2";
+      eudy  = mkConf "eudy";
+      koro  = mkConf "koro";
+      bay   = mkConf "bay";
+      lake2 = mkConf "lake2";
+    };
+
+    packages.x86_64-linux.hut = self.nixosConfigurations.hut.pkgs;
+  };
+}

keys.nix

@@ -0,0 +1,29 @@
+# As agenix needs to parse the secrets from a standalone .nix file, we describe
+# here all the public keys
+rec {
+  hosts = {
+    hut   = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICO7jIp6JRnRWTMDsTB/aiaICJCl4x8qmKMPSs4lCqP1 hut";
+    owl1  = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMqMEXO0ApVsBA6yjmb0xP2kWyoPDIWxBB0Q3+QbHVhv owl1";
+    owl2  = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHurEYpQzNHqWYF6B9Pd7W8UPgF3BxEg0BvSbsA7BAdK owl2";
+    eudy  = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL+WYPRRvZupqLAG0USKmd/juEPmisyyJaP8hAgYwXsG eudy";
+    koro  = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIImiTFDbxyUYPumvm8C4mEnHfuvtBY1H8undtd6oDd67 koro";
+    bay   = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICvGBzpRQKuQYHdlUQeAk6jmdbkrhmdLwTBqf3el7IgU bay";
+    lake2 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINo66//S1yatpQHE/BuYD/Gfq64TY7ZN5XOGXmNchiO0 lake2";
+  };
+
+  hostGroup = with hosts; rec {
+    compute    = [ owl1 owl2 ];
+    playground = [ eudy koro ];
+    storage    = [ bay lake2 ];
+    monitor    = [ hut ];
+
+    system     = storage ++ monitor;
+    safe       = system ++ compute;
+    all        = safe ++ playground;
+  };
+
+  admins = {
+    rarias = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE1oZTPtlEXdGt0Ak+upeCIiBdaDQtcmuWoTUCVuSVIR rarias@hut";
+    root   = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIII/1TNArcwA6D47mgW4TArwlxQRpwmIGiZDysah40Gb root@hut";
+  };
+}

m/bay/configuration.nix

@@ -0,0 +1,97 @@
+{ config, pkgs, lib, ... }:
+
+{
+  imports = [
+    ../common/main.nix
+    ../common/monitoring.nix
+  ];
+
+  # Select the this using the ID to avoid mismatches
+  boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d53562d";
+
+  environment.systemPackages = with pkgs; [
+    ceph
+  ];
+
+  services.slurm = {
+    client.enable = lib.mkForce false;
+  };
+
+  networking = {
+    hostName = "bay";
+    interfaces.eno1.ipv4.addresses = [ {
+      address = "10.0.40.40";
+      prefixLength = 24;
+    } ];
+    interfaces.ibp5s0.ipv4.addresses = [ {
+      address = "10.0.42.40";
+      prefixLength = 24;
+    } ];
+  };
+
+  services.ceph = {
+    enable = true;
+    global = {
+      fsid = "9c8d06e0-485f-4aaf-b16b-06d6daf1232b";
+      monHost = "10.0.40.40";
+      monInitialMembers = "bay";
+      clusterNetwork = "10.0.40.40/24"; # Use Ethernet only
+    };
+    extraConfig = {
+      # Only log to stderr so it appears in the journal
+      "log_file" = "/dev/null";
+      "mon_cluster_log_file" = "/dev/null";
+      "log_to_stderr" = "true";
+      "err_to_stderr" = "true";
+      "log_to_file" = "false";
+    };
+    mds = {
+      enable = true;
+      daemons = [ "mds0" "mds1" ];
+      extraConfig = {
+        "host" = "bay";
+      };
+    };
+    mgr = {
+      enable = true;
+      daemons = [ "bay" ];
+    };
+    mon = {
+      enable = true;
+      daemons = [ "bay" ];
+    };
+    osd = {
+      enable = true;
+      # One daemon per NVME disk
+      daemons = [ "0" "1" "2" "3" ];
+      extraConfig = {
+        "osd crush chooseleaf type" = "0";
+        "osd journal size" = "10000";
+        "osd pool default min size" = "2";
+        "osd pool default pg num" = "200";
+        "osd pool default pgp num" = "200";
+        "osd pool default size" = "3";
+      };
+    };
+  };
+
+  # Missing service for volumes, see:
+  # https://www.reddit.com/r/ceph/comments/14otjyo/comment/jrd69vt/
+  systemd.services.ceph-volume = {
+    enable = true;
+    description = "Ceph Volume activation";
+    unitConfig = {
+      Type = "oneshot";
+      After = "local-fs.target";
+      Wants = "local-fs.target";
+    };
+    path = [ pkgs.ceph pkgs.util-linux pkgs.lvm2 pkgs.cryptsetup ];
+    serviceConfig = {
+      KillMode = "none";
+      Environment = "CEPH_VOLUME_TIMEOUT=10000";
+      ExecStart = "/bin/sh -c 'timeout $CEPH_VOLUME_TIMEOUT ${pkgs.ceph}/bin/ceph-volume lvm activate --all --no-systemd'";
+      TimeoutSec = "0";
+    };
+    wantedBy = [ "multi-user.target" ];
+  };
+}

m/common/agenix.nix

@@ -0,0 +1,9 @@
+{ agenix, ... }:
+
+{
+  imports = [ agenix.nixosModules.default ];
+
+  environment.systemPackages = [
+    agenix.packages.x86_64-linux.default
+  ];
+}

m/common/boot.nix

@@ -0,0 +1,39 @@
+{ lib, pkgs, ... }:
+
+{
+  # Use the GRUB 2 boot loader.
+  boot.loader.grub.enable = lib.mkForce true;
+
+  # Enable GRUB2 serial console
+  boot.loader.grub.extraConfig = ''
+    serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
+    terminal_input --append serial
+    terminal_output --append serial
+  '';
+
+  # Enable serial console
+  boot.kernelParams = [
+    "console=tty1"
+    "console=ttyS0,115200"
+  ];
+
+  boot.kernel.sysctl = {
+    "kernel.perf_event_paranoid" = lib.mkDefault "-1";
+  };
+
+  boot.kernelPackages = pkgs.linuxPackages_latest;
+
+  #boot.kernelPatches = lib.singleton {
+  #  name = "osnoise-tracer";
+  #  patch = null;
+  #  extraStructuredConfig = with lib.kernel; {
+  #    OSNOISE_TRACER = yes;
+  #    HWLAT_TRACER = yes;
+  #  };
+  #};
+
+  boot.initrd.availableKernelModules = [ "ahci" "xhci_pci" "ehci_pci" "nvme" "usbhid" "sd_mod" ];
+  boot.initrd.kernelModules = [ ];
+  boot.kernelModules = [ "kvm-intel" ];
+  boot.extraModulePackages = [ ];
+}

m/common/fs.nix

@@ -1,6 +1,18 @@
 { ... }:
 
 {
+  fileSystems."/" =
+    { device = "/dev/disk/by-label/nixos";
+      fsType = "ext4";
+    };
+
+  # Trim unused blocks weekly
+  services.fstrim.enable = true;
+
+  swapDevices =
+    [ { device = "/dev/disk/by-label/swap"; }
+    ];
+
   # Mount the home via NFS
   fileSystems."/home" = {
     device = "10.0.40.30:/home";

m/common/hw.nix

@@ -0,0 +1,14 @@
+# Do not modify this file!  It was generated by ‘nixos-generate-config’
+# and may be overwritten by future invocations.  Please make changes
+# to /etc/nixos/configuration.nix instead.
+{ config, lib, pkgs, modulesPath, ... }:
+
+{
+  imports =
+    [ (modulesPath + "/installer/scan/not-detected.nix")
+    ];
+
+  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
+  powerManagement.cpuFreqGovernor = lib.mkDefault "powersave";
+  hardware.cpu.intel.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
+}

m/common/main.nix

@@ -1,21 +1,47 @@
-{ config, pkgs, ... }:
+{ config, pkgs, nixpkgs, bscpkgs, agenix, theFlake, ... }:
 
 {
   imports = [
-    ./hardware-configuration.nix
+    ./agenix.nix
-
     ./boot.nix
     ./fs.nix
-    ./gitlab-runner.nix
+    ./hw.nix
-    ./monitoring.nix
     ./net.nix
-    ./nfs.nix
+    ./ntp.nix
-    ./overlays.nix
     ./slurm.nix
     ./ssh.nix
     ./users.nix
+    ./watchdog.nix
+    ./rev.nix
+    ./zsh.nix
+  ];
 
-    <agenix/modules/age.nix>
+  nixpkgs.overlays = [
+    bscpkgs.bscOverlay
+    (import ../../pkgs/overlay.nix)
+  ];
+
+  system.configurationRevision =
+    if theFlake ? rev
+    then theFlake.rev
+    else throw ("Refusing to build from a dirty Git tree!");
+
+  nix.nixPath = [
+    "nixpkgs=${nixpkgs}"
+    "bscpkgs=${bscpkgs}"
+    "jungle=${theFlake.outPath}"
+  ];
+
+  nix.registry.nixpkgs.flake = nixpkgs;
+  nix.registry.bscpkgs.flake = bscpkgs;
+  nix.registry.jungle.flake = theFlake;
+
+  environment.systemPackages = with pkgs; [
+    vim wget git htop tmux pciutils tcpdump ripgrep nix-index nixos-option
+    nix-diff ipmitool freeipmi ethtool lm_sensors ix cmake gnumake file tree
+    ncdu config.boot.kernelPackages.perf ldns
+    # From bsckgs overlay
+    bsc.osumb
   ];
 
   systemd.services."serial-getty@ttyS0" = {
@@ -24,15 +50,19 @@
     serviceConfig.Restart = "always";
   };
 
+  # Increase limits
+  security.pam.loginLimits = [
+    {
+      domain = "*";
+      type = "-";
+      item = "memlock";
+      value = "1048576"; # 1 GiB of mem locked
+    }
+  ];
+
   time.timeZone = "Europe/Madrid";
   i18n.defaultLocale = "en_DK.UTF-8";
 
-  environment.systemPackages = with pkgs; [
-    vim wget git htop tmux pciutils tcpdump ripgrep nix-index nixos-option
-    nix-diff ipmitool freeipmi ethtool lm_sensors
-    (pkgs.callPackage <agenix/pkgs/agenix.nix> {})
-  ];
-
   environment.variables = {
     EDITOR = "vim";
     VISUAL = "vim";
@@ -43,14 +73,16 @@
   nix.settings.trusted-users = [ "@wheel" ];
   nix.gc.automatic = true;
   nix.gc.dates = "weekly";
+  nix.gc.options = "--delete-older-than 30d";
 
-  programs.zsh.enable = true;
+  programs.bash.promptInit = ''
-  programs.zsh.histSize = 100000;
+    PS1="\h\\$ "
+  '';
 
   # Copy the NixOS configuration file and link it from the resulting system
   # (/run/current-system/configuration.nix). This is useful in case you
   # accidentally delete configuration.nix.
-  system.copySystemConfiguration = true;
+  #system.copySystemConfiguration = true;
 
   # This value determines the NixOS release from which the default
   # settings for stateful data, like file locations and database versions

m/common/monitoring.nix

@@ -0,0 +1,25 @@
+{ config, lib, ... }:
+
+{
+  # We need access to the devices to monitor the disk space
+  systemd.services.prometheus-node-exporter.serviceConfig.PrivateDevices = lib.mkForce false;
+  systemd.services.prometheus-node-exporter.serviceConfig.ProtectHome = lib.mkForce "read-only";
+
+  # Required to allow the smartctl exporter to read the nvme0 character device,
+  # see the commit message on:
+  # https://github.com/NixOS/nixpkgs/commit/12c26aca1fd55ab99f831bedc865a626eee39f80
+  services.udev.extraRules = ''
+    SUBSYSTEM=="nvme", KERNEL=="nvme[0-9]*", GROUP="disk"
+  '';
+
+  services.prometheus = {
+    exporters = {
+      node = {
+        enable = true;
+        enabledCollectors = [ "systemd" ];
+        port = 9002;
+      };
+      smartctl.enable = true;
+    };
+  };
+}

m/common/net.nix

@@ -0,0 +1,94 @@
+{ pkgs, ... }:
+
+{
+  # Infiniband (IPoIB)
+  environment.systemPackages = [ pkgs.rdma-core ];
+  boot.kernelModules = [ "ib_umad" "ib_ipoib" ];
+
+  networking = {
+    enableIPv6 = false;
+    useDHCP = false;
+    defaultGateway = "10.0.40.30";
+    nameservers = ["8.8.8.8"];
+    proxy = {
+      default = "http://localhost:23080/";
+      noProxy = "127.0.0.1,localhost,internal.domain,10.0.40.40";
+      # Don't set all_proxy as go complains and breaks the gitlab runner, see:
+      # https://github.com/golang/go/issues/16715
+      allProxy = null;
+    };
+
+    firewall = {
+      enable = true;
+      allowedTCPPorts = [ 22 ];
+      extraCommands = ''
+        # Prevent ssfhead from contacting our slurmd daemon
+        iptables -A nixos-fw -p tcp -s ssfhead --dport 6817:6819 -j nixos-fw-log-refuse
+        # But accept traffic to slurm ports from any other node in the subnet
+        iptables -A nixos-fw -p tcp -s 10.0.40.0/24 --dport 6817:6819 -j nixos-fw-accept
+        # We also need to open the srun port range
+        iptables -A nixos-fw -p tcp -s 10.0.40.0/24 --dport 60000:61000 -j nixos-fw-accept
+      '';
+    };
+
+    extraHosts = ''
+      10.0.40.30      ssfhead
+      84.88.53.236    ssfhead.bsc.es ssfhead
+      
+      # Node Entry for node: mds01 (ID=72)
+      10.0.40.40              bay mds01 mds01-eth0
+      10.0.42.40              bay-ib mds01-ib0
+      10.0.40.141             bay-ipmi mds01-ipmi0
+      
+      # Node Entry for node: oss01 (ID=73)
+      10.0.40.41              oss01 oss01-eth0
+      10.0.42.41              oss01-ib0
+      10.0.40.142             oss01-ipmi0
+      
+      # Node Entry for node: oss02 (ID=74)
+      10.0.40.42              lake2 oss02 oss02-eth0
+      10.0.42.42              lake2-ib oss02-ib0
+      10.0.40.143             lake2-ipmi oss02-ipmi0
+      
+      # Node Entry for node: xeon01 (ID=15)
+      10.0.40.1               owl1 xeon01 xeon01-eth0
+      10.0.42.1               owl1-ib xeon01-ib0
+      10.0.40.101             owl1-ipmi xeon01-ipmi0
+      
+      # Node Entry for node: xeon02 (ID=16)
+      10.0.40.2               owl2 xeon02 xeon02-eth0
+      10.0.42.2               owl2-ib xeon02-ib0
+      10.0.40.102             owl2-ipmi xeon02-ipmi0
+      
+      # Node Entry for node: xeon03 (ID=17)
+      10.0.40.3               xeon03 xeon03-eth0
+      10.0.42.3               xeon03-ib0
+      10.0.40.103             xeon03-ipmi0
+      
+      # Node Entry for node: xeon04 (ID=18)
+      10.0.40.4               xeon04 xeon04-eth0
+      10.0.42.4               xeon04-ib0
+      10.0.40.104             xeon04-ipmi0
+      
+      # Node Entry for node: xeon05 (ID=19)
+      10.0.40.5               koro xeon05 xeon05-eth0
+      10.0.42.5               koro-ib xeon05-ib0
+      10.0.40.105             koro-ipmi xeon05-ipmi0
+      
+      # Node Entry for node: xeon06 (ID=20)
+      10.0.40.6               xeon06 xeon06-eth0
+      10.0.42.6               xeon06-ib0
+      10.0.40.106             xeon06-ipmi0
+      
+      # Node Entry for node: xeon07 (ID=21)
+      10.0.40.7               hut xeon07 xeon07-eth0
+      10.0.42.7               hut-ib xeon07-ib0
+      10.0.40.107             hut-ipmi xeon07-ipmi0
+      
+      # Node Entry for node: xeon08 (ID=22)
+      10.0.40.8               eudy xeon08 xeon08-eth0
+      10.0.42.8               eudy-ib xeon08-ib0
+      10.0.40.108             eudy-ipmi xeon08-ipmi0
+    '';
+  };
+}

m/common/ntp.nix

@@ -0,0 +1,9 @@
+{ pkgs, ... }:
+
+{
+  services.ntp.enable = true;
+
+  # Use the NTP server at BSC, as we don't have direct access
+  # to the outside world
+  networking.timeServers = [ "84.88.52.36" ];
+}

m/common/rev.nix

@@ -0,0 +1,18 @@
+{ theFlake, ... }:
+
+let
+  rev = if theFlake ? rev then theFlake.rev
+    else throw ("Refusing to build from a dirty Git tree!");
+in {
+  # Save the commit of the config in /etc/configrev
+  environment.etc.configrev.text = rev + "\n";
+
+  # Keep a log with the config over time
+  system.activationScripts.configRevLog.text = ''
+    BOOTED=$(cat /run/booted-system/etc/configrev 2>/dev/null || echo unknown)
+    CURRENT=$(cat /run/current-system/etc/configrev 2>/dev/null || echo unknown)
+    NEXT=${rev}
+    DATENOW=$(date --iso-8601=seconds)
+    echo "$DATENOW booted=$BOOTED current=$CURRENT next=$NEXT" >> /var/configrev.log
+  '';
+}

m/common/slurm.nix

@@ -0,0 +1,99 @@
+{ config, pkgs, lib, ... }:
+
+let
+  suspendProgram = pkgs.writeScript "suspend.sh" ''
+    #!/usr/bin/env bash
+    exec 1>>/var/log/power_save.log 2>>/var/log/power_save.log
+    set -x
+    export "PATH=/run/current-system/sw/bin:$PATH"
+    echo "$(date) Suspend invoked $0 $*" >> /var/log/power_save.log
+    hosts=$(scontrol show hostnames $1)
+    for host in $hosts; do
+      echo Shutting down host: $host
+      ipmitool -I lanplus -H ''${host}-ipmi -P "" -U "" chassis power off
+    done
+  '';
+
+  resumeProgram = pkgs.writeScript "resume.sh" ''
+    #!/usr/bin/env bash
+    exec 1>>/var/log/power_save.log 2>>/var/log/power_save.log
+    set -x
+    export "PATH=/run/current-system/sw/bin:$PATH"
+    echo "$(date) Suspend invoked $0 $*" >> /var/log/power_save.log
+    hosts=$(scontrol show hostnames $1)
+    for host in $hosts; do
+      echo Starting host: $host
+      ipmitool -I lanplus -H ''${host}-ipmi -P "" -U "" chassis power on
+    done
+  '';
+
+in {
+  systemd.services.slurmd.serviceConfig = {
+    # Kill all processes in the control group on stop/restart. This will kill
+    # all the jobs running, so ensure that we only upgrade when the nodes are
+    # not in use. See:
+    # https://github.com/NixOS/nixpkgs/commit/ae93ed0f0d4e7be0a286d1fca86446318c0c6ffb
+    # https://bugs.schedmd.com/show_bug.cgi?id=2095#c24
+    KillMode = lib.mkForce "control-group";
+  };
+
+  services.slurm = {
+    client.enable = true;
+    controlMachine = "hut";
+    clusterName = "jungle";
+    nodeName = [
+      "owl[1,2]  Sockets=2 CoresPerSocket=14 ThreadsPerCore=2 Feature=owl"
+      "hut       Sockets=2 CoresPerSocket=14 ThreadsPerCore=2"
+    ];
+
+    partitionName = [
+      "owl Nodes=owl[1-2] Default=YES MaxTime=INFINITE State=UP"
+      "all Nodes=owl[1-2],hut Default=NO MaxTime=INFINITE State=UP"
+    ];
+
+    # See slurm.conf(5) for more details about these options.
+    extraConfig = ''
+      # Use PMIx for MPI by default. It works okay with MPICH and OpenMPI, but
+      # not with Intel MPI. For that use the compatibility shim libpmi.so
+      # setting I_MPI_PMI_LIBRARY=$pmix/lib/libpmi.so while maintaining the PMIx
+      # library in SLURM (--mpi=pmix). See more details here:
+      # https://pm.bsc.es/gitlab/rarias/jungle/-/issues/16
+      MpiDefault=pmix
+
+      # When a node reboots return that node to the slurm queue as soon as it
+      # becomes operative again.
+      ReturnToService=2
+
+      # Track all processes by using a cgroup
+      ProctrackType=proctrack/cgroup
+
+      # Enable task/affinity to allow the jobs to run in a specified subset of
+      # the resources. Use the task/cgroup plugin to enable process containment.
+      TaskPlugin=task/affinity,task/cgroup
+
+      # Power off unused nodes until they are requested
+      SuspendProgram=${suspendProgram}
+      SuspendTimeout=60
+      ResumeProgram=${resumeProgram}
+      ResumeTimeout=300
+      SuspendExcNodes=hut
+
+      # Turn the nodes off after 1 hour of inactivity
+      SuspendTime=3600
+
+      # Reduce port range so we can allow only this range in the firewall
+      SrunPortRange=60000-61000
+    '';
+  };
+
+  age.secrets.mungeKey = {
+    file = ../../secrets/munge-key.age;
+    owner = "munge";
+    group = "munge";
+  };
+
+  services.munge = {
+    enable = true;
+    password = config.age.secrets.mungeKey.path;
+  };
+}

m/common/ssh.nix

@@ -0,0 +1,22 @@
+{ lib, ... }:
+
+let
+  keys = import ../../keys.nix;
+  hostsKeys = lib.mapAttrs (name: value: { publicKey = value; }) keys.hosts;
+in
+{
+  # Enable the OpenSSH daemon.
+  services.openssh.enable = true;
+
+  # Connect to intranet git hosts via proxy
+  programs.ssh.extraConfig = ''
+    Host bscpm02.bsc.es bscpm03.bsc.es gitlab-internal.bsc.es alya.gitlab.bsc.es
+      User git
+      ProxyCommand nc -X connect -x localhost:23080 %h %p
+  '';
+
+  programs.ssh.knownHosts = hostsKeys // {
+    "gitlab-internal.bsc.es".publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF9arsAOSRB06hdy71oTvJHG2Mg8zfebADxpvc37lZo3";
+    "bscpm03.bsc.es".publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM2NuSUPsEhqz1j5b4Gqd+MWFnRqyqY57+xMvBUqHYUS";
+  };
+}

m/common/users.nix

@@ -0,0 +1,75 @@
+{ pkgs, ... }:
+
+{
+  users = {
+    mutableUsers = false;
+    users = {
+      # Generate hashedPassword with `mkpasswd -m sha-512`
+
+      root.openssh.authorizedKeys.keys = [
+        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKBOf4r4lzQfyO0bx5BaREePREw8Zw5+xYgZhXwOZoBO ram@hop"
+        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINa0tvnNgwkc5xOwd6xTtaIdFi5jv0j2FrE7jl5MTLoE ram@mio"
+        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF3zeB5KSimMBAjvzsp1GCkepVaquVZGPYwRIzyzaCba aleix@bsc"
+        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIII/1TNArcwA6D47mgW4TArwlxQRpwmIGiZDysah40Gb root@hut"
+      ];
+
+      rarias = {
+        uid = 1880;
+        isNormalUser = true;
+        home = "/home/Computational/rarias";
+        description = "Rodrigo Arias";
+        group = "Computational";
+        extraGroups = [ "wheel" ];
+        hashedPassword = "$6$u06tkCy13enReBsb$xiI.twRvvTfH4jdS3s68NZ7U9PSbGKs5.LXU/UgoawSwNWhZo2hRAjNL5qG0/lAckzcho2LjD0r3NfVPvthY6/";
+        openssh.authorizedKeys.keys = [
+          "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKBOf4r4lzQfyO0bx5BaREePREw8Zw5+xYgZhXwOZoBO ram@hop"
+          "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINa0tvnNgwkc5xOwd6xTtaIdFi5jv0j2FrE7jl5MTLoE ram@mio"
+          "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGYcXIxe0poOEGLpk8NjiRozls7fMRX0N3j3Ar94U+Gl rarias@hal"
+        ];
+        shell = pkgs.zsh;
+      };
+
+      arocanon = {
+        uid = 1042;
+        isNormalUser = true;
+        home = "/home/Computational/arocanon";
+        description = "Aleix Roca";
+        group = "Computational";
+        extraGroups = [ "wheel" ];
+        hashedPassword = "$6$hliZiW4tULC/tH7p$pqZarwJkNZ7vS0G5llWQKx08UFG9DxDYgad7jplMD8WkZh5k58i4dfPoWtnEShfjTO6JHiIin05ny5lmSXzGM/";
+        openssh.authorizedKeys.keys = [
+          "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF3zeB5KSimMBAjvzsp1GCkepVaquVZGPYwRIzyzaCba aleix@bsc"
+          "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGdphWxLAEekicZ/WBrvP7phMyxKSSuLAZBovNX+hZXQ aleix@kerneland"
+        ];
+      };
+
+      rpenacob = {
+        uid = 2761;
+        isNormalUser = true;
+        home = "/home/Computational/rpenacob";
+        description = "Raúl Peñacoba";
+        group = "Computational";
+        hashedPassword = "$6$TZm3bDIFyPrMhj1E$uEDXoYYd1z2Wd5mMPfh3DZAjP7ztVjJ4ezIcn82C0ImqafPA.AnTmcVftHEzLB3tbe2O4SxDyPSDEQgJ4GOtj/";
+        openssh.authorizedKeys.keys = [
+          "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFYfXg37mauGeurqsLpedgA2XQ9d4Nm0ZGo/hI1f7wwH rpenacob@bsc"
+        ];
+      };
+
+      anavarro = {
+        uid = 1037;
+        isNormalUser = true;
+        home = "/home/Computational/anavarro";
+        description = "Antoni Navarro";
+        group = "Computational";
+        hashedPassword = "$6$QdNDsuLehoZTYZlb$CDhCouYDPrhoiB7/seu7RF.Gqg4zMQz0n5sA4U1KDgHaZOxy2as9pbIGeF8tOHJKRoZajk5GiaZv0rZMn7Oq31";
+        openssh.authorizedKeys.keys = [
+          "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILWjRSlKgzBPZQhIeEtk6Lvws2XNcYwHcwPv4osSgst5 anavarro@ssfhead"
+        ];
+      };
+    };
+
+    groups = {
+      Computational = { gid = 564; };
+    };
+  };
+}

m/common/watchdog.nix

@@ -0,0 +1,9 @@
+{ ... }:
+
+{
+  # The boards have a BMC watchdog controlled by IPMI
+  boot.kernelModules = [ "ipmi_watchdog" ];
+
+  # Enable systemd watchdog with 30 s interval
+  systemd.watchdog.runtimeTime = "30s";
+}

m/common/zsh.nix

@@ -0,0 +1,92 @@
+{ pkgs, ... }:
+
+{
+  environment.systemPackages = with pkgs; [
+    direnv
+    zsh-completions
+    nix-zsh-completions
+  ];
+
+  programs.zsh = {
+    enable = true;
+    histSize = 1000000;
+
+    shellInit = ''
+      # Disable new user prompt
+      if [ ! -e ~/.zshrc ]; then
+        touch ~/.zshrc
+      fi
+    '';
+
+    promptInit = ''
+      # Note that to manually override this in ~/.zshrc you should run `prompt off`
+      # before setting your PS1 and etc. Otherwise this will likely to interact with
+      # your ~/.zshrc configuration in unexpected ways as the default prompt sets
+      # a lot of different prompt variables.
+      autoload -U promptinit && promptinit && prompt default && setopt prompt_sp
+    '';
+
+    # Taken from Ulli Kehrle config:
+    # https://git.hrnz.li/Ulli/nixos/src/commit/2e203b8d8d671f4e3ced0f1744a51d5c6ee19846/profiles/shell.nix#L199-L205
+    interactiveShellInit = ''
+      source "${pkgs.zsh-history-substring-search}/share/zsh-history-substring-search/zsh-history-substring-search.zsh"
+
+      # Save history immediately, but only load it when the shell starts
+      setopt inc_append_history
+
+      # dircolors doesn't support alacritty:
+      # https://lists.gnu.org/archive/html/bug-coreutils/2019-05/msg00029.html
+      export LS_COLORS='rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.swp=00;90:*.tmp=00;90:*.dpkg-dist=00;90:*.dpkg-old=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:';
+
+      # From Arch Linux and GRML
+      bindkey "^R" history-incremental-pattern-search-backward
+      bindkey "^S" history-incremental-pattern-search-forward
+
+      # Auto rehash for new binaries
+      zstyle ':completion:*' rehash true
+      # show a nice menu with the matches
+      zstyle ':completion:*' menu yes select
+
+      bindkey '^[OA' history-substring-search-up   # Up
+      bindkey '^[[A' history-substring-search-up   # Up
+
+      bindkey '^[OB' history-substring-search-down # Down
+      bindkey '^[[B' history-substring-search-down # Down
+
+      bindkey '\e[1~' beginning-of-line            # Home
+      bindkey '\e[7~' beginning-of-line            # Home
+      bindkey '\e[H'  beginning-of-line            # Home
+      bindkey '\eOH'  beginning-of-line            # Home
+
+      bindkey '\e[4~' end-of-line                  # End
+      bindkey '\e[8~' end-of-line                  # End
+      bindkey '\e[F ' end-of-line                  # End
+      bindkey '\eOF'  end-of-line                  # End
+
+      bindkey '^?'    backward-delete-char         # Backspace
+      bindkey '\e[3~' delete-char                  # Del
+      # bindkey '\e[3;5~' delete-char                # sometimes Del, sometimes C-Del
+      bindkey '\e[2~' overwrite-mode               # Ins
+
+      bindkey '^H'      backward-kill-word         # C-Backspace
+
+      bindkey '5~'      kill-word                  # C-Del
+      bindkey '^[[3;5~' kill-word                  # C-Del
+      bindkey '^[[3^'   kill-word                  # C-Del
+
+      bindkey "^[[1;5H" backward-kill-line         # C-Home
+      bindkey "^[[7^"   backward-kill-line         # C-Home
+
+      bindkey "^[[1;5F" kill-line                  # C-End
+      bindkey "^[[8^"   kill-line                  # C-End
+
+      bindkey '^[[1;5C' forward-word               # C-Right
+      bindkey '^[0c'    forward-word               # C-Right
+      bindkey '^[[5C'   forward-word               # C-Right
+
+      bindkey '^[[1;5D' backward-word              # C-Left
+      bindkey '^[0d'    backward-word              # C-Left
+      bindkey '^[[5D'   backward-word              # C-Left
+    '';
+  };
+}

m/eudy/configuration.nix

@@ -0,0 +1,37 @@
+{ config, pkgs, lib, modulesPath, ... }:
+
+{
+  imports = [
+    ../common/main.nix
+    #(modulesPath + "/installer/netboot/netboot-minimal.nix")
+
+    ./kernel/kernel.nix
+    ./cpufreq.nix
+    ./fs.nix
+    ./users.nix
+    ./slurm.nix
+  ];
+
+  # Select this using the ID to avoid mismatches
+  boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d53564b";
+
+  # disable automatic garbage collector
+  nix.gc.automatic = lib.mkForce false;
+
+  # members of the tracing group can use the lttng-provided kernel events
+  # without root permissions
+  users.groups.tracing.members = [ "arocanon" ];
+
+  # set up both ethernet and infiniband ips
+  networking = {
+    hostName = "eudy";
+    interfaces.eno1.ipv4.addresses = [ {
+      address = "10.0.40.8";
+      prefixLength = 24;
+    } ];
+    interfaces.ibp5s0.ipv4.addresses = [ {
+      address = "10.0.42.8";
+      prefixLength = 24;
+    } ];
+  };
+}

m/eudy/cpufreq.nix

@@ -0,0 +1,40 @@
+{ lib, ... }:
+
+{
+  # Disable frequency boost by default. Use the intel_pstate driver instead of
+  # acpi_cpufreq driver because the acpi_cpufreq driver does not read the
+  # complete range of P-States [1]. Use the intel_pstate passive mode [2] to
+  # disable HWP, which allows a core to "select P-states by itself". Also, this
+  # disables intel governors, which confusingly, have the same names as the
+  # generic ones but behave differently [3].
+
+  # Essentially, we use the generic governors, but use the intel driver to read
+  # the P-state list.
+
+  # [1] - https://www.kernel.org/doc/html/latest/admin-guide/pm/intel_pstate.html#intel-pstate-vs-acpi-cpufreq
+  # [2] - https://www.kernel.org/doc/html/latest/admin-guide/pm/intel_pstate.html#passive-mode
+  # [3] - https://www.kernel.org/doc/html/latest/admin-guide/pm/intel_pstate.html#active-mode
+  # https://www.kernel.org/doc/html/latest/admin-guide/pm/cpufreq.html
+
+  # set intel_pstate to passive mode
+  boot.kernelParams = [
+    "intel_pstate=passive"
+  ];
+  # Disable frequency boost
+  system.activationScripts = {
+    disableFrequencyBoost.text = ''
+      echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo
+    '';
+  };
+
+  ## disable intel_pstate
+  #boot.kernelParams = [
+  #  "intel_pstate=disable"
+  #];
+  ## Disable frequency boost
+  #system.activationScripts = {
+  #  disableFrequencyBoost.text = ''
+  #    echo 0 > /sys/devices/system/cpu/cpufreq/boost
+  #  '';
+  #};
+}

m/eudy/fs.nix

@@ -0,0 +1,13 @@
+{ ... }:
+
+{
+  fileSystems."/nix" = {
+    device = "/dev/disk/by-label/optane";
+    fsType = "ext4";
+    neededForBoot = true;
+  };
+  fileSystems."/mnt/data" = {
+    device = "/dev/disk/by-label/data";
+    fsType = "ext4";
+  };
+}

m/eudy/kernel/kernel.nix

@@ -0,0 +1,92 @@
+{ pkgs, lib, ... }:
+
+let
+  #fcs-devel = pkgs.linuxPackages_custom {
+  #   version = "6.2.8";
+  #   src = /mnt/data/kernel/fcs/kernel/src;
+  #   configfile = /mnt/data/kernel/fcs/kernel/configs/defconfig;
+  #};
+
+  #fcsv1 = fcs-kernel "bc11660676d3d68ce2459b9fb5d5e654e3f413be" false;
+  #fcsv2 = fcs-kernel "db0f2eca0cd57a58bf456d7d2c7d5d8fdb25dfb1" false;
+  #fcsv1-lockdep = fcs-kernel "bc11660676d3d68ce2459b9fb5d5e654e3f413be" true;
+  #fcsv2-lockdep = fcs-kernel "db0f2eca0cd57a58bf456d7d2c7d5d8fdb25dfb1" true;
+  #fcs-kernel = gitCommit: lockdep: pkgs.linuxPackages_custom {
+  #   version = "6.2.8";
+  #   src = builtins.fetchGit {
+  #     url = "git@bscpm03.bsc.es:ompss-kernel/linux.git";
+  #     rev = gitCommit;
+  #     ref = "fcs";
+  #   };
+  #   configfile = if lockdep then ./configs/lockdep else ./configs/defconfig;
+  #};
+
+  kernel = nixos-fcsv3;
+
+  nixos-fcs-kernel = {gitCommit, lockStat ? false, preempt ? false, branch ? "fcs"}: pkgs.linuxPackagesFor (pkgs.buildLinux rec {
+    version = "6.2.8";
+    src = builtins.fetchGit {
+      url = "git@bscpm03.bsc.es:ompss-kernel/linux.git";
+      rev = gitCommit;
+      ref = branch;
+    };
+    structuredExtraConfig = with lib.kernel; {
+      # add general custom kernel options here
+    } // lib.optionalAttrs lockStat {
+      LOCK_STAT = yes;
+    } // lib.optionalAttrs preempt {
+      PREEMPT = lib.mkForce yes;
+      PREEMPT_VOLUNTARY = lib.mkForce no;
+    };
+    kernelPatches = [];
+    extraMeta.branch = lib.versions.majorMinor version;
+  });
+
+  nixos-fcsv1 = nixos-fcs-kernel {gitCommit = "bc11660676d3d68ce2459b9fb5d5e654e3f413be";};
+  nixos-fcsv2 = nixos-fcs-kernel {gitCommit = "db0f2eca0cd57a58bf456d7d2c7d5d8fdb25dfb1";};
+  nixos-fcsv3 = nixos-fcs-kernel {gitCommit = "6c17394890704c3345ac1a521bb547164b36b154";};
+
+  # always use fcs_sched_setaffinity
+  #nixos-debug = nixos-fcs-kernel {gitCommit = "7d0bf285fca92badc8df3c9907a9ab30db4418aa";};
+  # remove need_check_cgroup
+  #nixos-debug = nixos-fcs-kernel {gitCommit = "4cc4efaab5e4a0bfa3089e935215b981c1922919";};
+  # merge again fcs_wake and fcs_wait
+  #nixos-debug = nixos-fcs-kernel {gitCommit = "40c6f72f4ae54b0b636b193ac0648fb5730c810d";};
+  # start from scratch, this is the working version with split fcs_wake and fcs_wait
+  nixos-debug = nixos-fcs-kernel {gitCommit = "c9a39d6a4ca83845b4e71fcc268fb0a76aff1bdf"; branch = "fcs-test"; };
+
+  nixos-fcsv1-lockstat = nixos-fcs-kernel {
+    gitCommit = "bc11660676d3d68ce2459b9fb5d5e654e3f413be";
+    lockStat = true;
+  };
+  nixos-fcsv2-lockstat = nixos-fcs-kernel {
+    gitCommit = "db0f2eca0cd57a58bf456d7d2c7d5d8fdb25dfb1";
+    lockStat = true;
+  };
+  nixos-fcsv3-lockstat = nixos-fcs-kernel {
+    gitCommit = "6c17394890704c3345ac1a521bb547164b36b154";
+    lockStat = true;
+  };
+  nixos-fcsv3-lockstat-preempt = nixos-fcs-kernel {
+    gitCommit = "6c17394890704c3345ac1a521bb547164b36b154";
+    lockStat = true;
+    preempt = true;
+  };
+  latest = pkgs.linuxPackages_latest;
+
+in {
+  imports = [
+    ./lttng.nix
+    ./perf.nix
+  ];
+  boot.kernelPackages = lib.mkForce kernel;
+
+  # disable all cpu mitigations
+  boot.kernelParams = [
+    "mitigations=off"
+  ];
+  
+  # enable memory overcommit, needed to build a taglibc system using nix after
+  # increasing the openblas memory footprint
+  boot.kernel.sysctl."vm.overcommit_memory" = 1;
+}

m/eudy/kernel/lttng.nix

@@ -0,0 +1,43 @@
+{ config, pkgs, lib, ... }:
+
+let
+
+  # The lttng btrfs probe crashes at compile time because of an undefined
+  # function. This disables the btrfs tracepoints to avoid the issue.
+
+  # Also enable lockdep tracepoints, this is disabled by default because it
+  # does not work well on architectures other than x86_64 (i think that arm) as
+  # I was told on the mailing list.
+  lttng-modules-fixed = config.boot.kernelPackages.lttng-modules.overrideAttrs (finalAttrs: previousAttrs: {
+    patchPhase = (lib.optionalString (previousAttrs ? patchPhase) previousAttrs.patchPhase) + ''
+      # disable btrfs
+      substituteInPlace src/probes/Kbuild \
+        --replace "  obj-\$(CONFIG_LTTNG) += lttng-probe-btrfs.o" "  #obj-\$(CONFIG_LTTNG) += lttng-probe-btrfs.o"
+
+      # enable lockdep tracepoints
+      substituteInPlace src/probes/Kbuild \
+        --replace "#ifneq (\$(CONFIG_LOCKDEP),)"                  "ifneq (\$(CONFIG_LOCKDEP),)" \
+        --replace "#  obj-\$(CONFIG_LTTNG) += lttng-probe-lock.o" "  obj-\$(CONFIG_LTTNG) += lttng-probe-lock.o" \
+        --replace "#endif # CONFIG_LOCKDEP"                       "endif # CONFIG_LOCKDEP"
+    '';
+  });
+in {
+
+  # add the lttng tools and modules to the system environment
+  boot.extraModulePackages = [ lttng-modules-fixed ];
+  environment.systemPackages = with pkgs; [
+    lttng-tools lttng-ust babeltrace
+  ];
+
+  # start the lttng root daemon to manage kernel events
+  systemd.services.lttng-sessiond = {
+    wantedBy = [ "multi-user.target" ];
+    description = "LTTng session daemon for the root user";
+    serviceConfig = {
+      User = "root";
+      ExecStart = ''
+        ${pkgs.lttng-tools}/bin/lttng-sessiond
+      '';
+    };
+  };
+}

m/eudy/kernel/perf.nix

@@ -0,0 +1,22 @@
+{ config, pkgs, lib, ... }:
+
+{
+  # add the perf tool
+  environment.systemPackages = with pkgs; [
+    config.boot.kernelPackages.perf
+  ];
+
+  # allow non-root users to read tracing data from the kernel
+  boot.kernel.sysctl."kernel.perf_event_paranoid" = -2;
+  boot.kernel.sysctl."kernel.kptr_restrict" = 0;
+
+  # specify additionl options to the tracefs directory to allow members of the
+  # tracing group to access tracefs.
+  fileSystems."/sys/kernel/tracing" = {
+    options = [
+      "mode=755"
+      "gid=tracing"
+    ];
+  };
+}
+

m/eudy/slurm.nix

@@ -0,0 +1,7 @@
+{ lib, ... }:
+
+{
+  services.slurm = {
+    client.enable = lib.mkForce false;
+  };
+}

m/eudy/users.nix

@@ -0,0 +1,11 @@
+{ ... }:
+
+{
+  security.sudo.extraRules= [{
+    users = [ "arocanon" ];
+    commands = [{
+      command = "ALL" ;
+      options= [ "NOPASSWD" ]; # "SETENV" # Adding the following could be a good idea
+    }];
+  }];
+}

m/hut/configuration.nix

@@ -0,0 +1,32 @@
+{ config, pkgs, ... }:
+
+{
+  imports = [
+    ../common/main.nix
+
+    ../module/ceph.nix
+    ./gitlab-runner.nix
+    ./monitoring.nix
+    ./nfs.nix
+    ./slurm-daemon.nix
+    ./nix-serve.nix
+    #./pxe.nix
+  ];
+
+  boot.binfmt.emulatedSystems = [ "aarch64-linux" "powerpc64le-linux" "riscv64-linux" ];
+
+  # Select the this using the ID to avoid mismatches
+  boot.loader.grub.device = "/dev/disk/by-id/ata-INTEL_SSDSC2BB240G7_PHDV6462004Y240AGN";
+
+  networking = {
+    hostName = "hut";
+    interfaces.eno1.ipv4.addresses = [ {
+      address = "10.0.40.7";
+      prefixLength = 24;
+    } ];
+    interfaces.ibp5s0.ipv4.addresses = [ {
+      address = "10.0.42.7";
+      prefixLength = 24;
+    } ];
+  };
+}

m/hut/gitlab-runner.nix

@@ -1,35 +1,46 @@
 { pkgs, lib, config, ... }:
 
 {
-  age.secrets."secrets/ovni-token".file = ./secrets/ovni-token.age;
+  age.secrets.ovniToken.file = ../../secrets/ovni-token.age;
-  age.secrets."secrets/nosv-token".file = ./secrets/nosv-token.age;
+  age.secrets.nosvToken.file = ../../secrets/nosv-token.age;
 
   services.gitlab-runner = {
     enable = true;
+    settings.concurrent = 5;
     services = {
       ovni-shell = {
-        registrationConfigFile = config.age.secrets."secrets/ovni-token".path;
+        registrationConfigFile = config.age.secrets.ovniToken.path;
         executor = "shell";
         tagList = [ "nix" "xeon" ];
+        registrationFlags = [
+          # Using space doesn't work, and causes it to misread the next flag
+          "--locked='false'"
+        ];
         environmentVariables = {
           SHELL = "${pkgs.bash}/bin/bash";
         };
       };
       ovni-docker = {
-        registrationConfigFile = config.age.secrets."secrets/ovni-token".path;
+        registrationConfigFile = config.age.secrets.ovniToken.path;
         dockerImage = "debian:stable";
         tagList = [ "docker" "xeon" ];
-        registrationFlags = [ "--docker-network-mode host" ];
+        registrationFlags = [
+          "--locked='false'"
+          "--docker-network-mode host"
+        ];
         environmentVariables = {
           https_proxy = "http://localhost:23080";
           http_proxy = "http://localhost:23080";
         };
       };
       nosv-docker = {
-        registrationConfigFile = config.age.secrets."secrets/nosv-token".path;
+        registrationConfigFile = config.age.secrets.nosvToken.path;
         dockerImage = "debian:stable";
         tagList = [ "docker" "xeon" ];
-        registrationFlags = [ "--docker-network-mode host" ];
+        registrationFlags = [
+          "--docker-network-mode host"
+          "--docker-cpus 56"
+        ];
         environmentVariables = {
           https_proxy = "http://localhost:23080";
           http_proxy = "http://localhost:23080";

m/hut/ipmi.yml

@@ -0,0 +1,13 @@
+modules:
+        default:
+                collectors:
+                - bmc
+                - ipmi
+                - chassis
+
+        lan:
+                collectors:
+                - ipmi
+                - chassis
+                user: ""
+                pass: ""

m/hut/monitoring.nix

@@ -0,0 +1,143 @@
+{ config, lib, ... }:
+
+{
+  services.grafana = {
+    enable = true;
+    settings = {
+      server = {
+        domain = "jungle.bsc.es";
+        root_url = "%(protocol)s://%(domain)s/grafana";
+        serve_from_sub_path = true;
+        http_port = 2342;
+        http_addr = "127.0.0.1";
+      };
+      feature_toggles.publicDashboards = true;
+    };
+  };
+
+  services.prometheus = {
+    enable = true;
+    port = 9001;
+    retentionTime = "1y";
+    listenAddress = "127.0.0.1";
+  };
+
+  systemd.services.prometheus-ipmi-exporter.serviceConfig.DynamicUser = lib.mkForce false;
+  systemd.services.prometheus-ipmi-exporter.serviceConfig.PrivateDevices = lib.mkForce false;
+
+  # We need access to the devices to monitor the disk space
+  systemd.services.prometheus-node-exporter.serviceConfig.PrivateDevices = lib.mkForce false;
+  systemd.services.prometheus-node-exporter.serviceConfig.ProtectHome = lib.mkForce "read-only";
+
+  virtualisation.docker.daemon.settings = {
+    metrics-addr = "127.0.0.1:9323";
+  };
+
+  # Required to allow the smartctl exporter to read the nvme0 character device,
+  # see the commit message on:
+  # https://github.com/NixOS/nixpkgs/commit/12c26aca1fd55ab99f831bedc865a626eee39f80
+  services.udev.extraRules = ''
+    SUBSYSTEM=="nvme", KERNEL=="nvme[0-9]*", GROUP="disk"
+  '';
+
+  services.prometheus = {
+
+    exporters = {
+      ipmi = {
+        enable = true;
+        group = "root";
+        user = "root";
+        configFile = ./ipmi.yml;
+        #extraFlags = [ "--log.level=debug" ];
+        listenAddress = "127.0.0.1";
+      };
+      node = {
+        enable = true;
+        enabledCollectors = [ "systemd" ];
+        port = 9002;
+        listenAddress = "127.0.0.1";
+      };
+      smartctl = {
+        enable = true;
+        listenAddress = "127.0.0.1";
+      };
+    };
+
+    scrapeConfigs = [
+      {
+        job_name = "xeon07";
+        static_configs = [{
+          targets = [
+            "127.0.0.1:${toString config.services.prometheus.exporters.node.port}"
+            "127.0.0.1:${toString config.services.prometheus.exporters.ipmi.port}"
+            "127.0.0.1:9323"
+            "127.0.0.1:9252"
+            "127.0.0.1:${toString config.services.prometheus.exporters.smartctl.port}"
+          ];
+        }];
+      }
+      {
+        job_name = "ceph";
+        static_configs = [{
+          targets = [
+            "10.0.40.40:9283" # Ceph statistics
+            "10.0.40.40:9002" # Node exporter
+            "10.0.40.42:9002" # Node exporter
+          ];
+        }];
+      }
+      {
+        # Scrape the IPMI info of the hosts remotely via LAN
+        job_name = "ipmi-lan";
+        scrape_interval = "1m";
+        scrape_timeout = "30s";
+        metrics_path = "/ipmi";
+        scheme = "http";
+        relabel_configs = [
+          {
+            # Takes the address and sets it in the "target=<xyz>" URL parameter
+            source_labels = [ "__address__" ];
+            separator = ";";
+            regex = "(.*)(:80)?";
+            target_label = "__param_target";
+            replacement = "\${1}";
+            action = "replace";
+          }
+          {
+            # Sets the "instance" label with the remote host we are querying
+            source_labels = [ "__param_target" ];
+            separator = ";";
+            regex = "(.*)";
+            target_label = "instance";
+            replacement = "\${1}";
+            action = "replace";
+          }
+          {
+            # Sets the fixed "module=lan" URL param
+            separator = ";";
+            regex = "(.*)";
+            target_label = "__param_module";
+            replacement = "lan";
+            action = "replace";
+          }
+          {
+            # Sets the target to query as the localhost IPMI exporter
+            separator = ";";
+            regex = ".*";
+            target_label = "__address__";
+            replacement = "127.0.0.1:9290";
+            action = "replace";
+          }
+        ];
+
+        # Load the list of targets from another file
+        file_sd_configs = [
+          {
+            files = [ "${./targets.yml}" ];
+            refresh_interval = "30s";
+          }
+        ];
+      }
+    ];
+  };
+}

m/hut/nix-serve.nix

@@ -0,0 +1,16 @@
+{ config, ... }:
+
+{
+  age.secrets.nixServe.file = ../../secrets/nix-serve.age;
+
+  services.nix-serve = {
+    enable = true;
+    # Only listen locally, as we serve it via ssh
+    bindAddress = "127.0.0.1";
+    port = 5000;
+
+    secretKeyFile = config.age.secrets.nixServe.path;
+    # Public key:
+    # jungle.bsc.es:pEc7MlAT0HEwLQYPtpkPLwRsGf80ZI26aj29zMw/HH0=
+  };
+}

m/hut/pxe.nix

@@ -0,0 +1,35 @@
+{ theFlake, pkgs, ... }:
+
+# This module describes a script that can launch the pixiecore daemon to serve a
+# NixOS image via PXE to a node to directly boot from there, without requiring a
+# working disk.
+
+let
+  # The host config must have the netboot-minimal.nix module too
+  host = theFlake.nixosConfigurations.lake2;
+  sys = host.config.system;
+  build = sys.build;
+  kernel = "${build.kernel}/bzImage";
+  initrd = "${build.netbootRamdisk}/initrd";
+  init = "${build.toplevel}/init";
+
+  script = pkgs.writeShellScriptBin "pixiecore-helper" ''
+    #!/usr/bin/env bash -x
+
+    ${pkgs.pixiecore}/bin/pixiecore \
+      boot ${kernel} ${initrd} --cmdline "init=${init} loglevel=4" \
+      --debug --dhcp-no-bind --port 64172 --status-port 64172 "$@"
+  '';
+in
+{
+  ## We need a DHCP server to provide the IP
+  #services.dnsmasq = {
+  #  enable = true;
+  #  settings = {
+  #    domain-needed = true;
+  #    dhcp-range = [ "192.168.0.2,192.168.0.254" ];
+  #  };
+  #};
+
+  environment.systemPackages = [ script ];
+}

m/hut/slurm-daemon.nix

@@ -0,0 +1,7 @@
+{ ... }:
+
+{
+  services.slurm = {
+    server.enable = true;
+  };
+}

m/hut/targets.yml

@@ -0,0 +1,15 @@
+- targets:
+  - 10.0.40.101
+  - 10.0.40.102
+  - 10.0.40.103
+  - 10.0.40.104
+  - 10.0.40.105
+  - 10.0.40.106
+  - 10.0.40.107
+  - 10.0.40.108
+  # Storage
+  - 10.0.40.141
+  - 10.0.40.142
+  - 10.0.40.143
+  labels:
+    job: ipmi-lan

m/koro/configuration.nix

@@ -0,0 +1,37 @@
+{ config, pkgs, lib, modulesPath, ... }:
+
+{
+  imports = [
+    ../common/main.nix
+    #(modulesPath + "/installer/netboot/netboot-minimal.nix")
+
+    ../eudy/cpufreq.nix
+    ../eudy/users.nix
+    ../eudy/slurm.nix
+    ./users.nix
+    ./kernel.nix
+  ];
+
+  # Select this using the ID to avoid mismatches
+  boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d5376d2";
+
+  # disable automatic garbage collector
+  nix.gc.automatic = lib.mkForce false;
+
+  # members of the tracing group can use the lttng-provided kernel events
+  # without root permissions
+  users.groups.tracing.members = [ "arocanon" "vlopez" ];
+
+  # set up both ethernet and infiniband ips
+  networking = {
+    hostName = "koro";
+    interfaces.eno1.ipv4.addresses = [ {
+      address = "10.0.40.5";
+      prefixLength = 24;
+    } ];
+    interfaces.ibp5s0.ipv4.addresses = [ {
+      address = "10.0.42.5";
+      prefixLength = 24;
+    } ];
+  };
+}

m/koro/kernel.nix

@@ -0,0 +1,64 @@
+{ pkgs, lib, ... }:
+
+let
+  kernel = nixos-fcsv4;
+
+  nixos-fcs-kernel = {gitCommit, lockStat ? false, preempt ? false, branch ? "fcs"}: pkgs.linuxPackagesFor (pkgs.buildLinux rec {
+    version = "6.2.8";
+    src = builtins.fetchGit {
+      url = "git@bscpm03.bsc.es:ompss-kernel/linux.git";
+      rev = gitCommit;
+      ref = branch;
+    };
+    structuredExtraConfig = with lib.kernel; {
+      # add general custom kernel options here
+    } // lib.optionalAttrs lockStat {
+      LOCK_STAT = yes;
+    } // lib.optionalAttrs preempt {
+      PREEMPT = lib.mkForce yes;
+      PREEMPT_VOLUNTARY = lib.mkForce no;
+    };
+    kernelPatches = [];
+    extraMeta.branch = lib.versions.majorMinor version;
+  });
+
+  nixos-fcsv1 = nixos-fcs-kernel {gitCommit = "bc11660676d3d68ce2459b9fb5d5e654e3f413be";};
+  nixos-fcsv2 = nixos-fcs-kernel {gitCommit = "db0f2eca0cd57a58bf456d7d2c7d5d8fdb25dfb1";};
+  nixos-fcsv3 = nixos-fcs-kernel {gitCommit = "6c17394890704c3345ac1a521bb547164b36b154";};
+  nixos-fcsv4 = nixos-fcs-kernel {gitCommit = "c94c3d946f33ac3e5782a02ee002cc1164c0cb4f";};
+
+  nixos-fcsv1-lockstat = nixos-fcs-kernel {
+    gitCommit = "bc11660676d3d68ce2459b9fb5d5e654e3f413be";
+    lockStat = true;
+  };
+  nixos-fcsv2-lockstat = nixos-fcs-kernel {
+    gitCommit = "db0f2eca0cd57a58bf456d7d2c7d5d8fdb25dfb1";
+    lockStat = true;
+  };
+  nixos-fcsv3-lockstat = nixos-fcs-kernel {
+    gitCommit = "6c17394890704c3345ac1a521bb547164b36b154";
+    lockStat = true;
+  };
+  nixos-fcsv3-lockstat-preempt = nixos-fcs-kernel {
+    gitCommit = "6c17394890704c3345ac1a521bb547164b36b154";
+    lockStat = true;
+    preempt = true;
+  };
+  latest = pkgs.linuxPackages_latest;
+
+in {
+  imports = [
+    ../eudy/kernel/lttng.nix
+    ../eudy/kernel/perf.nix
+  ];
+  boot.kernelPackages = lib.mkForce kernel;
+
+  # disable all cpu mitigations
+  boot.kernelParams = [
+    "mitigations=off"
+  ];
+  
+  # enable memory overcommit, needed to build a taglibc system using nix after
+  # increasing the openblas memory footprint
+  boot.kernel.sysctl."vm.overcommit_memory" = lib.mkForce 1;
+}

m/koro/users.nix

@@ -0,0 +1,17 @@
+{ ... }:
+
+{
+  users.users = {
+    vlopez = {
+      uid = 4334;
+      isNormalUser = true;
+      home = "/home/Computational/vlopez";
+      description = "Victor López";
+      group = "Computational";
+      hashedPassword = "$6$0ZBkgIYE/renVqtt$1uWlJsb0FEezRVNoETTzZMx4X2SvWiOsKvi0ppWCRqI66S6TqMBXBdP4fcQyvRRBt0e4Z7opZIvvITBsEtO0f0";
+      openssh.authorizedKeys.keys = [
+        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGMwlUZRf9jfG666Qa5Sb+KtEhXqkiMlBV2su3x/dXHq victor@arch"
+      ];
+    };
+  };
+}

m/lake2/configuration.nix

@@ -0,0 +1,73 @@
+{ config, pkgs, lib, modulesPath, ... }:
+
+{
+  imports = [
+    ../common/main.nix
+    ../common/monitoring.nix
+  ];
+
+  boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d53563a";
+
+  environment.systemPackages = with pkgs; [
+    ceph
+  ];
+
+  services.slurm = {
+    client.enable = lib.mkForce false;
+  };
+
+  services.ceph = {
+    enable = true;
+    global = {
+      fsid = "9c8d06e0-485f-4aaf-b16b-06d6daf1232b";
+      monHost = "10.0.40.40";
+      monInitialMembers = "bay";
+      clusterNetwork = "10.0.40.40/24"; # Use Ethernet only
+    };
+    osd = {
+      enable = true;
+      # One daemon per NVME disk
+      daemons = [ "4" "5" "6" "7" ];
+      extraConfig = {
+        "osd crush chooseleaf type" = "0";
+        "osd journal size" = "10000";
+        "osd pool default min size" = "2";
+        "osd pool default pg num" = "200";
+        "osd pool default pgp num" = "200";
+        "osd pool default size" = "3";
+      };
+    };
+  };
+
+  networking = {
+    hostName = "lake2";
+    interfaces.eno1.ipv4.addresses = [ {
+      address = "10.0.40.42";
+      prefixLength = 24;
+    } ];
+    interfaces.ibp5s0.ipv4.addresses = [ {
+      address = "10.0.42.42";
+      prefixLength = 24;
+    } ];
+  };
+
+  # Missing service for volumes, see:
+  # https://www.reddit.com/r/ceph/comments/14otjyo/comment/jrd69vt/
+  systemd.services.ceph-volume = {
+    enable = true;
+    description = "Ceph Volume activation";
+    unitConfig = {
+      Type = "oneshot";
+      After = "local-fs.target";
+      Wants = "local-fs.target";
+    };
+    path = [ pkgs.ceph pkgs.util-linux pkgs.lvm2 pkgs.cryptsetup ];
+    serviceConfig = {
+      KillMode = "none";
+      Environment = "CEPH_VOLUME_TIMEOUT=10000";
+      ExecStart = "/bin/sh -c 'timeout $CEPH_VOLUME_TIMEOUT ${pkgs.ceph}/bin/ceph-volume lvm activate --all --no-systemd'";
+      TimeoutSec = "0";
+    };
+    wantedBy = [ "multi-user.target" ];
+  };
+}

m/module/ceph.nix

@@ -0,0 +1,25 @@
+{ config, pkgs, ... }:
+
+# Mounts the /ceph filesystem at boot
+{
+  environment.systemPackages = with pkgs; [
+    ceph
+    ceph-client
+    fio # For benchmarks
+  ];
+
+  # We need the ceph module loaded as the mount.ceph binary fails to run the
+  # modprobe command.
+  boot.kernelModules = [ "ceph" ];
+
+  age.secrets.cephUser.file = ../../secrets/ceph-user.age;
+
+  fileSystems."/ceph" = {
+    fsType = "ceph";
+    device = "user@9c8d06e0-485f-4aaf-b16b-06d6daf1232b.cephfs=/";
+    options = [
+      "mon_addr=10.0.40.40"
+      "secretfile=${config.age.secrets.cephUser.path}"
+    ];
+  };
+}

m/module/slurm-firewall.nix

@@ -0,0 +1,8 @@
+{ ... }:
+
+{
+  networking.firewall = {
+    # Required for PMIx in SLURM, we should find a better way
+    allowedTCPPortRanges = [ { from=1024; to=65535; } ];
+  };
+}

m/owl1/configuration.nix

@@ -0,0 +1,24 @@
+{ config, pkgs, ... }:
+
+{
+  imports = [
+    ../common/main.nix
+    ../module/ceph.nix
+    ../module/slurm-firewall.nix
+  ];
+
+  # Select the this using the ID to avoid mismatches
+  boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d53566c";
+
+  networking = {
+    hostName = "owl1";
+    interfaces.eno1.ipv4.addresses = [ {
+      address = "10.0.40.1";
+      prefixLength = 24;
+    } ];
+    interfaces.ibp5s0.ipv4.addresses = [ {
+      address = "10.0.42.1";
+      prefixLength = 24;
+    } ];
+  };
+}

m/owl2/configuration.nix

@@ -0,0 +1,25 @@
+{ config, pkgs, ... }:
+
+{
+  imports = [
+    ../common/main.nix
+    ../module/ceph.nix
+    ../module/slurm-firewall.nix
+  ];
+
+  # Select the this using the ID to avoid mismatches
+  boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d535629";
+
+  networking = {
+    hostName = "owl2";
+    interfaces.eno1.ipv4.addresses = [ {
+      address = "10.0.40.2";
+      prefixLength = 24;
+    } ];
+    # Watch out! The OmniPath device is not in the same place here:
+    interfaces.ibp129s0.ipv4.addresses = [ {
+      address = "10.0.42.2";
+      prefixLength = 24;
+    } ];
+  };
+}

nixos-config.nix

@@ -0,0 +1 @@
+(builtins.getFlake (toString ./.)).nixosConfigurations

overlays-compat/overlays.nix

@@ -1,8 +0,0 @@
-self: super:
-with super.lib;
-let
-  # Load the system config and get the `nixpkgs.overlays` option
-  overlays = (import <nixpkgs/nixos> { }).config.nixpkgs.overlays;
-in
-  # Apply all overlays to the input of the current "main" overlay
-  foldl' (flip extends) (_: super) overlays self

pkgs/ceph.nix

@@ -0,0 +1,405 @@
+{ lib
+, stdenv
+, runCommand
+, fetchurl
+, fetchFromGitHub
+, fetchPypi
+
+# Build time
+, cmake
+, ensureNewerSourcesHook
+, fmt
+, git
+, makeWrapper
+, nasm
+, pkg-config
+, which
+
+# Tests
+, nixosTests
+
+# Runtime dependencies
+, arrow-cpp
+, babeltrace
+, boost179
+, bzip2
+, cryptsetup
+, cunit
+, doxygen
+, gperf
+, graphviz
+, gtest
+, icu
+, libcap
+, libcap_ng
+, libnl
+, libxml2
+, lttng-ust
+, lua
+, lz4
+, oath-toolkit
+, openldap
+, python310
+, rdkafka
+, rocksdb
+, snappy
+, sqlite
+, utf8proc
+, zlib
+, zstd
+
+# Optional Dependencies
+, curl ? null
+, expat ? null
+, fuse ? null
+, libatomic_ops ? null
+, libedit ? null
+, libs3 ? null
+, yasm ? null
+
+# Mallocs
+, gperftools ? null
+, jemalloc ? null
+
+# Crypto Dependencies
+, cryptopp ? null
+, nspr ? null
+, nss ? null
+
+# Linux Only Dependencies
+, linuxHeaders
+, util-linux
+, libuuid
+, udev
+, keyutils
+, rdma-core
+, rabbitmq-c
+, libaio ? null
+, libxfs ? null
+, liburing ? null
+, zfs ? null
+, ...
+}:
+
+# We must have one crypto library
+assert cryptopp != null || (nss != null && nspr != null);
+
+let
+  shouldUsePkg = pkg: if pkg != null && pkg.meta.available then pkg else null;
+
+  optYasm = shouldUsePkg yasm;
+  optExpat = shouldUsePkg expat;
+  optCurl = shouldUsePkg curl;
+  optFuse = shouldUsePkg fuse;
+  optLibedit = shouldUsePkg libedit;
+  optLibatomic_ops = shouldUsePkg libatomic_ops;
+  optLibs3 = shouldUsePkg libs3;
+
+  optJemalloc = shouldUsePkg jemalloc;
+  optGperftools = shouldUsePkg gperftools;
+
+  optCryptopp = shouldUsePkg cryptopp;
+  optNss = shouldUsePkg nss;
+  optNspr = shouldUsePkg nspr;
+
+  optLibaio = shouldUsePkg libaio;
+  optLibxfs = shouldUsePkg libxfs;
+  optZfs = shouldUsePkg zfs;
+
+  # Downgrade rocksdb, 7.10 breaks ceph
+  rocksdb' = rocksdb.overrideAttrs {
+    version = "7.9.2";
+    src = fetchFromGitHub {
+      owner = "facebook";
+      repo = "rocksdb";
+      rev = "refs/tags/v7.9.2";
+      hash = "sha256-5P7IqJ14EZzDkbjaBvbix04ceGGdlWBuVFH/5dpD5VM=";
+    };
+  };
+
+  hasRadosgw = optExpat != null && optCurl != null && optLibedit != null;
+
+  # Malloc implementation (can be jemalloc, tcmalloc or null)
+  malloc = if optJemalloc != null then optJemalloc else optGperftools;
+
+  # We prefer nss over cryptopp
+  cryptoStr = if optNss != null && optNspr != null then "nss" else
+    if optCryptopp != null then "cryptopp" else "none";
+
+  cryptoLibsMap = {
+    nss = [ optNss optNspr ];
+    cryptopp = [ optCryptopp ];
+    none = [ ];
+  };
+
+  getMeta = description: with lib; {
+     homepage = "https://ceph.io/en/";
+     inherit description;
+     license = with licenses; [ lgpl21 gpl2 bsd3 mit publicDomain ];
+     maintainers = with maintainers; [ adev ak johanot krav ];
+     platforms = [ "x86_64-linux" "aarch64-linux" ];
+   };
+
+  ceph-common = with python.pkgs; buildPythonPackage {
+    pname = "ceph-common";
+    inherit src version;
+
+    sourceRoot = "ceph-${version}/src/python-common";
+
+    propagatedBuildInputs = [
+      pyyaml
+    ];
+
+    nativeCheckInputs = [
+      pytestCheckHook
+    ];
+
+    disabledTests = [
+      # requires network access
+      "test_valid_addr"
+    ];
+
+    meta = getMeta "Ceph common module for code shared by manager modules";
+  };
+
+  # Watch out for python <> boost compatibility
+  python = python310.override {
+    packageOverrides = self: super: {
+      sqlalchemy = super.sqlalchemy.overridePythonAttrs rec {
+        version = "1.4.46";
+        src = fetchPypi {
+          pname = "SQLAlchemy";
+          inherit version;
+          hash = "sha256-aRO4JH2KKS74MVFipRkx4rQM6RaB8bbxj2lwRSAMSjA=";
+        };
+        disabledTestPaths = [
+          "test/aaa_profiling"
+          "test/ext/mypy"
+        ];
+      };
+    };
+  };
+
+  boost = boost179.override {
+    enablePython = true;
+    inherit python;
+  };
+
+  # TODO: split this off in build and runtime environment
+  ceph-python-env = python.withPackages (ps: with ps; [
+    ceph-common
+
+    # build time
+    cython
+
+    # debian/control
+    bcrypt
+    cherrypy
+    influxdb
+    jinja2
+    kubernetes
+    natsort
+    numpy
+    pecan
+    prettytable
+    pyjwt
+    pyopenssl
+    python-dateutil
+    pyyaml
+    requests
+    routes
+    scikit-learn
+    scipy
+    setuptools
+    sphinx
+    virtualenv
+    werkzeug
+
+    # src/pybind/mgr/requirements-required.txt
+    cryptography
+    jsonpatch
+
+    # src/tools/cephfs/shell/setup.py
+    cmd2
+    colorama
+  ]);
+  inherit (ceph-python-env.python) sitePackages;
+
+  version = "18.2.0";
+  src = fetchurl {
+    url = "https://download.ceph.com/tarballs/ceph-${version}.tar.gz";
+    hash = "sha256:0k9nl6xi5brva51rr14m7ig27mmmd7vrpchcmqc40q3c2khn6ns9";
+  };
+in rec {
+  ceph = stdenv.mkDerivation {
+    pname = "ceph";
+    inherit src version;
+
+    nativeBuildInputs = [
+      cmake
+      fmt
+      git
+      makeWrapper
+      nasm
+      pkg-config
+      python
+      python.pkgs.python # for the toPythonPath function
+      python.pkgs.wrapPython
+      which
+      (ensureNewerSourcesHook { year = "1980"; })
+      # for building docs/man-pages presumably
+      doxygen
+      graphviz
+    ];
+
+    enableParallelBuilding = true;
+
+    buildInputs = cryptoLibsMap.${cryptoStr} ++ [
+      arrow-cpp
+      babeltrace
+      boost
+      bzip2
+      ceph-python-env
+      cryptsetup
+      cunit
+      gperf
+      gtest
+      icu
+      libcap
+      libnl
+      libxml2
+      lttng-ust
+      lua
+      lz4
+      malloc
+      oath-toolkit
+      openldap
+      optLibatomic_ops
+      optLibs3
+      optYasm
+      rdkafka
+      rocksdb'
+      snappy
+      sqlite
+      utf8proc
+      zlib
+      zstd
+    ] ++ lib.optionals stdenv.isLinux [
+      keyutils
+      libcap_ng
+      liburing
+      libuuid
+      linuxHeaders
+      optLibaio
+      optLibxfs
+      optZfs
+      rabbitmq-c
+      rdma-core
+      udev
+      util-linux
+    ] ++ lib.optionals hasRadosgw [
+      optCurl
+      optExpat
+      optFuse
+      optLibedit
+    ];
+
+    pythonPath = [ ceph-python-env "${placeholder "out"}/${ceph-python-env.sitePackages}" ];
+
+    preConfigure =''
+      substituteInPlace src/common/module.c --replace "/sbin/modinfo"  "modinfo"
+      substituteInPlace src/common/module.c --replace "/sbin/modprobe" "modprobe"
+      substituteInPlace src/common/module.c --replace "/bin/grep" "grep"
+
+      # install target needs to be in PYTHONPATH for "*.pth support" check to succeed
+      # set PYTHONPATH, so the build system doesn't silently skip installing ceph-volume and others
+      export PYTHONPATH=${ceph-python-env}/${sitePackages}:$lib/${sitePackages}:$out/${sitePackages}
+      patchShebangs src/
+    '';
+
+    cmakeFlags = [
+      "-DCMAKE_INSTALL_DATADIR=${placeholder "lib"}/lib"
+
+      "-DWITH_CEPHFS_SHELL:BOOL=ON"
+      "-DWITH_SYSTEMD:BOOL=OFF"
+      # `WITH_JAEGER` requires `thrift` as a depenedncy (fine), but the build fails with:
+      #     CMake Error at src/opentelemetry-cpp-stamp/opentelemetry-cpp-build-Release.cmake:49 (message):
+      #     Command failed: 2
+      #
+      #        'make' 'opentelemetry_trace' 'opentelemetry_exporter_jaeger_trace'
+      #
+      #     See also
+      #
+      #        /build/ceph-18.2.0/build/src/opentelemetry-cpp/src/opentelemetry-cpp-stamp/opentelemetry-cpp-build-*.log
+      # and that file contains:
+      #     /build/ceph-18.2.0/src/jaegertracing/opentelemetry-cpp/exporters/jaeger/src/TUDPTransport.cc: In member function 'virtual void opentelemetry::v1::exporter::jaeger::TUDPTransport::close()':
+      #     /build/ceph-18.2.0/src/jaegertracing/opentelemetry-cpp/exporters/jaeger/src/TUDPTransport.cc:71:7: error: '::close' has not been declared; did you mean 'pclose'?
+      #       71 |     ::THRIFT_CLOSESOCKET(socket_);
+      #          |       ^~~~~~~~~~~~~~~~~~
+      # Looks like `close()` is somehow not included.
+      # But the relevant code is already removed in `open-telemetry` 1.10: https://github.com/open-telemetry/opentelemetry-cpp/pull/2031
+      # So it's proably not worth trying to fix that for this Ceph version,
+      # and instead just disable Ceph's Jaeger support.
+      "-DWITH_JAEGER:BOOL=OFF"
+      "-DWITH_TESTS:BOOL=OFF"
+
+      # Use our own libraries, where possible
+      "-DWITH_SYSTEM_ARROW:BOOL=ON" # Only used if other options enable Arrow support.
+      "-DWITH_SYSTEM_BOOST:BOOL=ON"
+      "-DWITH_SYSTEM_GTEST:BOOL=ON"
+      "-DWITH_SYSTEM_ROCKSDB:BOOL=ON"
+      "-DWITH_SYSTEM_UTF8PROC:BOOL=ON"
+      "-DWITH_SYSTEM_ZSTD:BOOL=ON"
+
+      # TODO breaks with sandbox, tries to download stuff with npm
+      "-DWITH_MGR_DASHBOARD_FRONTEND:BOOL=OFF"
+      # WITH_XFS has been set default ON from Ceph 16, keeping it optional in nixpkgs for now
+      ''-DWITH_XFS=${if optLibxfs != null then "ON" else "OFF"}''
+    ] ++ lib.optional stdenv.isLinux "-DWITH_SYSTEM_LIBURING=ON";
+
+    postFixup = ''
+      wrapPythonPrograms
+      wrapProgram $out/bin/ceph-mgr --prefix PYTHONPATH ":" "$(toPythonPath ${placeholder "out"}):$(toPythonPath ${ceph-python-env})"
+
+      # Test that ceph-volume exists since the build system has a tendency to
+      # silently drop it with misconfigurations.
+      test -f $out/bin/ceph-volume
+    '';
+
+    outputs = [ "out" "lib" "dev" "doc" "man" ];
+
+    doCheck = false; # uses pip to install things from the internet
+
+    # Takes 7+h to build with 2 cores.
+    requiredSystemFeatures = [ "big-parallel" ];
+
+    meta = getMeta "Distributed storage system";
+
+    passthru = {
+      inherit version;
+      tests = {
+        inherit (nixosTests)
+          ceph-multi-node
+          ceph-single-node
+          ceph-single-node-bluestore;
+      };
+    };
+  };
+
+  ceph-client = runCommand "ceph-client-${version}" {
+      meta = getMeta "Tools needed to mount Ceph's RADOS Block Devices/Cephfs";
+    } ''
+      mkdir -p $out/{bin,etc,${sitePackages},share/bash-completion/completions}
+      cp -r ${ceph}/bin/{ceph,.ceph-wrapped,rados,rbd,rbdmap} $out/bin
+      cp -r ${ceph}/bin/ceph-{authtool,conf,dencoder,rbdnamer,syn} $out/bin
+      cp -r ${ceph}/bin/rbd-replay* $out/bin
+      cp -r ${ceph}/sbin/mount.ceph $out/bin
+      cp -r ${ceph}/sbin/mount.fuse.ceph $out/bin
+      ln -s bin $out/sbin
+      cp -r ${ceph}/${sitePackages}/* $out/${sitePackages}
+      cp -r ${ceph}/etc/bash_completion.d $out/share/bash-completion/completions
+      # wrapPythonPrograms modifies .ceph-wrapped, so lets just update its paths
+      substituteInPlace $out/bin/ceph          --replace ${ceph} $out
+      substituteInPlace $out/bin/.ceph-wrapped --replace ${ceph} $out
+   '';
+}

pkgs/overlay.nix

@@ -0,0 +1,35 @@
+final: prev:
+{
+  bsc = prev.bsc.extend (bscFinal: bscPrev: {
+    # Set MPICH as default
+    mpi = bscFinal.mpich;
+
+    # Configure the network for MPICH
+    mpich = with final; prev.mpich.overrideAttrs (old: {
+      buildInput = old.buildInputs ++ [
+        libfabric
+        pmix
+      ];
+      configureFlags = [
+        "--enable-shared"
+        "--enable-sharedlib"
+        "--with-pm=no"
+        "--with-device=ch4:ofi"
+        "--with-pmi=pmix"
+        "--with-pmix=${final.pmix}"
+        "--with-libfabric=${final.libfabric}"
+        "--enable-g=log"
+      ] ++ lib.optionals (lib.versionAtLeast gfortran.version "10") [
+        "FFLAGS=-fallow-argument-mismatch" # https://github.com/pmodels/mpich/issues/4300
+        "FCFLAGS=-fallow-argument-mismatch"
+      ];
+    });
+  });
+
+  # Update ceph to 18.2.0 until it lands in nixpkgs, see:
+  # https://github.com/NixOS/nixpkgs/pull/247849
+  inherit (prev.callPackage ./ceph.nix {
+    lua = prev.lua5_4;
+    fmt = prev.fmt_8;
+  }) ceph ceph-client;
+}

rebuild.sh

@@ -0,0 +1,16 @@
+#!/bin/sh -ex
+
+if [ "$(id -u)" != 0 ]; then
+ echo "Needs root permissions"
+ exit 1
+fi
+
+if [ "$(hostname)" != "hut" ]; then
+  >&2 echo "must run from machine hut, not $(hostname)"
+  exit 1
+fi
+
+# Update all nodes
+nixos-rebuild switch --flake .
+nixos-rebuild switch --flake .#owl1 --target-host owl1
+nixos-rebuild switch --flake .#owl2 --target-host owl2

secrets/ceph-user.age

@@ -0,0 +1,21 @@
+age-encryption.org/v1
+-> ssh-ed25519 AY8zKw J00a6ZOhkupkhLU5WQ0kD05HEF4KKsSs2hwjHKbnnHU
+J14VoNOCqLpScVO7OLXbqTcLI4tcVUHt5cqY/XQmbGs
+-> ssh-ed25519 sgAamA k8R/bSUdvVmlBI6yHPi5NBQPBGM36lPJwsir8DFGgxE
+4ZKC3gYvic6AVrNGgNjwztbUzhxP8ViX5O3wFo9wlrk
+-> ssh-ed25519 HY2yRg 966xf2fTnA6Wq0uYXbXZQOManqITJcCbQS9LZCGEOh4
+Qg5echQSrzqeDqvaMx+5fqi8XyTjAeCsY/UFJX6YnDs
+-> ssh-ed25519 tcumPQ e0U2okrGIoUpLfPYjIRx1V92rE3hZW13nJef+l3kBQg
+LejAUKBl+tPhwocCF00ZHTzFISnwX8og8GvemiMIcyo
+-> ssh-ed25519 JJ1LWg QkzTsPq9Gdh+FNz/a4bDb9LQOreFyxeTC51UNd1fsj0
+ayrlKenETfQzH1Z9drVEWqszQebicGVJve0/pCnxAE8
+-> ssh-ed25519 CAWG4Q lJLW9+dxvyoD4hYzeXeE/4rzJ6HIeEQOB1+fbhV3xw0
+T2RrVCtTuQvya9HiJB7txk3QGrntpsMX9Tt1cyXoW5E
+-> ssh-ed25519 MSF3dg JOZkFb2CfqWKvZIz7lYxXWgv8iEVDkQF8hInDMZvknc
+MHDWxjUw4dNiC1h4MrU9uKKcI3rwkxABm0+5FYMZkok
+-> ~8m;7f-grease
+lDIullfC98RhpTZ4Mk87Td+VtPmwPdgz+iIilpKugUkmV5r4Uqd7yE+5ArA6ekr/
+G/X4EA
+--- Cz4sv9ZunBcVdZCozdTh1zlg1zIASjk2MjYeYfcN9eA
+ÊN	Å$[H˜ÝQËéŠ
+d£š·'­±ö7…·Í²)ÖØÀÊx9yüÐëE¡þÓM7^Ø[ÐMŽ+É&éâö½$8tM¨Ð²

secrets/nix-serve.age

@@ -0,0 +1,12 @@
+age-encryption.org/v1
+-> ssh-ed25519 HY2yRg d144D+VvxhYgKtH//uD2qNuVnYX6bh74YqkyM3ZjBwU
+0IeVmFAf4U8Sm0d01O6ZwJ1V2jl/mSMl4wF0MP5LrIg
+-> ssh-ed25519 CAWG4Q H4nKxue/Cj/3KUF5A+/ygHMjjArwgx3SIWwXcqFtyUo
+4k5NJkLUrueLYiPkr2LAwQLWmuaOIsDmV/86ravpleU
+-> ssh-ed25519 MSF3dg HpgUAFHLPs4w0cdJHqTwf8lySkTeV9O9NnBf49ClDHs
+foPIUUgAYe1YSDy6+aMfjN7xv9xud9fDmhRlIztHoEo
+-> vLkF\<-grease
+3GRT+W8gYSpjl/a6Ix9+g9UJnTpl1ZH/oucfR801vfE8y77DV2Jxz/XJwzxYxKG5
+YEhiTGMNbXw/V7E5aVSz6Bdc
+--- GtiHKCZdHByq9j0BSLd544PhbEwTN138E8TFdxipeiA
+¥¿£‹„ÝG$Sº¼ƒRAæÀ¾Th]nÄ8<C384>,ùHœsÈïÚ=p¼™Ù'»<>ô+ôjõÓõŒ9±)ñ:”)‘¸œYâþÑ8³IØõ8:ol<6F>ë’<1F>åÃZÐæ3–PM”F;ÊrYõ“ÞÛ<1F>$¨­y¸LâÙœ¦ÎœàÕUús16Ǿ¡LŒb÷¨²

secrets/nosv-token.age

@@ -0,0 +1,11 @@
+age-encryption.org/v1
+-> ssh-ed25519 HY2yRg hrdS7Dl/j+u3XVfM79ZJpZSlre9TcD7DTQ+EEAT6kEE
+avUO96P1h7w2BYWgrQ7GpUgdaCV9AZL7eOTTcF9gfro
+-> ssh-ed25519 CAWG4Q A5raRY1CAgFYZgoQ92GMyNejYNdHx/7Y6uTS+EjLPWA
+FRFqT2Jz7qRcybaxkQTKHGl797LVXoHpYG4RZSrX/70
+-> ssh-ed25519 MSF3dg D+R80Bg7W9AuiOMAqtGFZQl994dRBIegYRLmmTaeZ3o
+BHvZsugRiuZ91b4jk91h30o3eF3hadSnVCwxXge95T8
+-> BT/El`a-grease W{nq|Vm )bld 2Nl}4 N$#JGB4t
+oLG+0S1aGfO/ohCfgGmhDhwwLi4H
+--- 2I5C+FvBG/K1ZHh7C5QD39feTSLoFGwcTeZAmeILNsI
+¹õW©o÷ ÙÄd;ËÐC¾.¹¡_(“u
G¡€‰#ìvâœgÉ<67>†õõy¹Y‰žl9ŒÈ
¡Ïµ.Œé0x<30>Þ½úN. /ü<>tB×b‡ü¼K¼ì:Q×—È\¹ÀÍT_´»Átxïm’——_JñÞž-š

secrets/secrets.nix

@@ -0,0 +1,15 @@
+let
+  keys = import ../keys.nix;
+  adminsKeys = builtins.attrValues keys.admins;
+  hut = [ keys.hosts.hut ] ++ adminsKeys;
+  # Only expose ceph keys to safe nodes and admins
+  safe = keys.hostGroup.safe ++ adminsKeys;
+in
+{
+  "ovni-token.age".publicKeys = hut;
+  "nosv-token.age".publicKeys = hut;
+  "nix-serve.age".publicKeys = hut;
+
+  "ceph-user.age".publicKeys = safe;
+  "munge-key.age".publicKeys = safe;
+}

web/.gitignore

@@ -0,0 +1 @@
+./public

web/archetypes/default.md

@@ -0,0 +1,6 @@
+---
+title: "{{ replace .Name "-" " " | title }}"
+date: {{ .Date }}
+draft: true
+---
+

web/content/_index.md

@@ -0,0 +1,25 @@
+![Rainforest](jungle.jpg)
+
+Welcome to the jungle, a set of machines with no imposed rules that are fully
+controlled and maintained by their users.
+
+The configuration of all the machines is written in a centralized [git
+repository][config] using the Nix language for NixOS. Changes in the
+configuration of the machines are introduced by merge requests and pass a review
+step before being deployed.
+
+[config]: https://pm.bsc.es/gitlab/rarias/jungle
+
+The machines have access to the large list of packages available in
+[Nixpkgs][nixpkgs] and a custom set of packages named [bscpkgs][bscpkgs],
+specifically tailored to our needs for HPC machines. Users can install their own
+packages and made them system-wide available by opening a merge request.
+
+[nixpkgs]: https://github.com/NixOS/nixpkgs
+[bscpkgs]: https://pm.bsc.es/gitlab/rarias/bscpkgs
+
+We have put a lot of effort to guarantee very good reproducibility properties in
+the configuration of the machines and the software they use.
+
+To enter the jungle machines follow the [instructions](access) to submit a
+request.

web/content/access/index.md

@@ -0,0 +1,22 @@
+---
+title: "Enter the jungle"
+description: "Request access to the machines"
+---
+
+![Cave](./cave.jpg)
+
+Before requesting access to the jungle machines, you must be able to access the
+`ssfhead.bsc.es` node (only available via the intranet or VPN). You can request
+access to the login machine using a resource petition in the BSC intranet.
+
+Then, to request access to the machines we will need some information about you:
+
+1. Which machines you want access to (hut, owl1, owl2, eudy, koro...)
+1. Your user name and user id (to match the NFS permissions)
+1. Your real name and surname (for identification purposes)
+1. The salted hash of your login password, generated with `mkpasswd -m sha-512`
+1. An SSH public key of type Ed25519 (can be generated with `ssh-keygen -t ed25519`)
+
+Send an email to <jungle@bsc.es> with the details, or directly open a
+merge request in the [jungle
+repository](https://pm.bsc.es/gitlab/rarias/jungle/).

web/content/eudy/_index.md

@@ -0,0 +1,10 @@
+---
+title: "Eudy"
+description: "Linux kernel experiments"
+---
+
+[![Eudy](eudy.jpg)](https://commons.wikimedia.org/w/index.php?curid=5817408)
+
+The *eudy* machine is destined as a playground for Linux kernel experiments. The
+name is a shorthand of the Eudyptula species of little penguins found the New
+Zealand and Australia.

web/content/grafana/_index.md

@@ -0,0 +1,6 @@
+---
+title: "Grafana"
+description: "Monitor metrics"
+---
+
+If you are reading this page, the proxy to the Grafana service is not working.

web/content/hut/_index.md

@@ -0,0 +1,18 @@
+---
+title: "Hut"
+description: "Control node"
+date: 2023-06-13T19:36:57+02:00
+---
+
+![Hut](hut.jpg)
+
+From the hut we monitor and control other nodes. It consist of one node only,
+which is available at `hut` or `xeon07`. It runs the following services:
+
+- Prometheus: to store the monitoring data.
+- Grafana: to plot the data in the web browser.
+- Slurmctld: to manage the SLURM nodes.
+- Gitlab runner: to run CI jobs from Gitlab.
+
+This node is prone to interruptions from all the services it runs, so it is not
+a good candidate for low noise executions.

web/content/intro-nix/_index.md

@@ -0,0 +1,7 @@
+---
+title: "Intro to nix"
+date: 2023-09-15
+---
+
+Basic introduction to Nix for users of the jungle machines. You should be able
+to access the jungle machines, otherwise [request access](/access).

web/content/intro-nix/ch1.md

@@ -0,0 +1,100 @@
+---
+title: "Chapter 1: Packages"
+description: "Here we show where packages come from"
+date: 2023-06-13T19:36:57+02:00
+weight: 1
+---
+
+In this chapter we describe where the packages available in the cluster come
+from and how to load them.
+
+## Where packages come from
+
+The packages in the jungle cluster are constructed by *layers*. Each layer
+applies some changes over the previous one:
+
+1. The first layer is [nixpkgs][1], a large repository of packages maintained by
+   the NixOS community. It provides packages like gcc, bash, gcc or the linux
+   kernel.
+
+[1]: https://github.com/NixOS/nixpkgs/
+
+2. The second layer is [bscpkgs][2], it takes the nixpkgs set of packages and
+   expands it by adding custom packages from the BSC such as Nanos6, nOS-V,
+   NODES, ovni or wxparaver.
+
+[2]: https://pm.bsc.es/gitlab/rarias/bscpkgs
+
+3. The third layer is [jungle][3], it takes the extended packages from bscpkgs
+   and configures them for the jungle cluster. For example, we configure MPICH
+   to use the OmniPath network and set it as the default implementation.
+
+[3]: https://pm.bsc.es/gitlab/rarias/jungle
+
+These layers are called *overlays* in Nix and they are the default mechanism
+used to modify the packages. Generally you will use the packages defined in the
+last layer (jungle) but you can define your own additional layer to specify
+custom changes. For example, instead of choosing MPICH, you may want to use
+Intel MPI instead by default.
+
+## Loading packages in an ephemeral shell
+
+You can manually load packages in a *new* shell with `nix shell jungle#<pkg>`,
+for example:
+
+```
+hut% which ovniemu
+ovniemu not found
+hut% nix shell jungle#bsc.ovni
+hut% which ovniemu
+/nix/store/0yzas8007x9djlpbb0pckcr1vhd0mcfy-ovni-1.3.0/bin/ovniemu
+hut% exit
+hut%
+```
+
+You can also specify multiple packages by listing them as parameters of `nix
+shell`:
+
+```
+hut% nix shell jungle#bsc.ovni jungle#bsc.osumb
+hut% which osu_bw
+/nix/store/lnjirzllhjn2fadlqzrz7a547iawl8jc-osu-micro-benchmarks-7.1-1/bin/osu_bw
+hut% exit
+```
+
+Or make the bash (zsh in this case) shell expand them:
+
+```
+hut% echo nix shell jungle#bsc.{ovni,osumb}
+nix shell jungle#bsc.ovni jungle#bsc.osumb
+hut% nix shell jungle#bsc.{ovni,osumb}
+hut% which osu_bw
+/nix/store/lnjirzllhjn2fadlqzrz7a547iawl8jc-osu-micro-benchmarks-7.1-1/bin/osu_bw
+hut% exit
+```
+
+You can use TAB to see which packages are available:
+
+```
+hut% nix shell jungle#bsc.n<TAB>
+jungle\#bsc.nanos6              jungle\#bsc.nixtools
+jungle\#bsc.nanos6Debug         jungle\#bsc.nix-wrap
+jungle\#bsc.nanos6Git           jungle\#bsc.nodes
+jungle\#bsc.nanos6GlibcxxDebug  jungle\#bsc.nodesGit
+jungle\#bsc.nanos6-icc          jungle\#bsc.nodesRelease
+jungle\#bsc.nanos6-icx          jungle\#bsc.nodesWithOvni
+jungle\#bsc.nanos6Release       jungle\#bsc.nosv
+jungle\#bsc.nix-mn4
+```
+
+Notice that these packages are evaluated at the moment the command is invoked.
+So if you come back a month later and run the same command, you may find that
+the packages have been updated and that could be problematic.
+
+In the next section we will create a new flake that defines the packages of the
+shell and also records the exact version of the packages that we used at the
+evaluation time for future use.
+
+In the [next chapter](../ch2) we will see how to create a permanent shell that
+will retain the same packages even if they are upgraded in the cluster, until we
+decide to upgrade them.

web/content/intro-nix/ch2.md

@@ -0,0 +1,155 @@
+---
+title: "Chapter 2: Your first shell"
+date: 2023-09-15
+weight: 2
+---
+
+## Creating a shell with flake.nix
+
+First, create an empty git repository where your shells will live:
+
+```txt
+hut% mkdir jungle-examples
+hut% cd jungle-examples
+hut% git init
+Initialized empty Git repository in /home/Computational/rarias/jungle-examples/.git/
+```
+
+And then, place a file named `flake.nix` on the repo with this content:
+
+```nix
+{
+  inputs.jungle.url = "jungle";
+  nixConfig.bash-prompt = "\[nix-develop\]$ ";
+
+  outputs = { self, jungle }:  
+  let
+    pkgs = jungle.outputs.packages.x86_64-linux;
+  in {
+    devShells.x86_64-linux.default = pkgs.mkShell rec {
+      pname = "my-shell";
+      buildInputs = with pkgs.bsc; [
+        ovni osumb # other packages here...
+      ];
+    };
+  };
+}
+```
+
+This file defines a how to create a shell in the Nix language with the
+`pkgs.mkShell` function using the packages listed in `buildInputs`. It also
+requests the packages to be taken from the *jungle* input, which corresponds to
+the set of packages that [we defined earlier](../ch1#where-packages-come-from),
+tuned for the cluster. We will describe it in more detail later.
+
+The tool `nix develop` tries to find a flake.nix in the current directory and
+enter the shell described by `devShells.x86_64-linux.default` (or the
+corresponding architecture).
+
+Now, **it is important that all the files of the repository are committed in
+git**, as nix will only read what is in the index of git. If we try to enter the
+shell with the `nix develop` command, it will complain and fail:
+
+```txt
+hut% nix develop
+warning: Git tree '/home/Computational/rarias/jungle-examples' is dirty
+error: getting status of '/nix/store/0ccnxa25whszw7mgbgyzdm4nqc0zwnm8-source/flake.nix': No such file or directory
+```
+
+The first warning states that the git directory has modified files not added to
+the index. Then the error occurs because the flake.nix is not in the index of
+git, so `nix develop` doesn't see it. So let's add it to a commit and try again:
+
+```txt
+hut% git add flake.nix
+hut% git commit flake.nix -m 'First shell'
+[master (root-commit) eb8a4ac] First shell
+ 1 file changed, 13 insertions(+)
+ create mode 100644 flake.nix
+hut% nix develop
+warning: creating lock file '/home/Computational/rarias/jungle-examples/flake.lock'
+warning: Git tree '/home/Computational/rarias/jungle-examples' is dirty
+[nix-develop]$
+```
+
+In the `flake.nix` we have set the shell prompt to `[nix-develop]` so we can
+easily spot that we are inside a `nix develop` shell. To exit:
+
+```txt
+[nix-develop]$ exit
+hut%
+```
+
+## Using the flake.lock file
+
+Now we see the `creating lock file` message and the git tree becomes dirty
+again (however, we enter the shell successfully).
+
+This `flake.lock` file that has been created collects the current state of the
+jungle packages in a file, so future invocations will use the same versions. We
+can see more details with `nix flake metadata`:
+
+```txt
+hut% nix flake metadata
+warning: Git tree '/home/Computational/rarias/jungle-examples' is dirty
+Resolved URL:  git+file:///home/Computational/rarias/jungle-examples
+Locked URL:    git+file:///home/Computational/rarias/jungle-examples
+Path:          /nix/store/bckxqjkkv52hy4pzgb96r7fchhmvmql8-source
+Revision:      eb8a4ac544a74e3995d859c751e9ff4339de6509-dirty
+Last modified: 2023-09-15 13:06:12
+Inputs:
+└───jungle: path:/nix/store/3wv6q0f3pkgw840nnkn4jsp9xi650dyj-source?lastModified=1694772033&narHash=sha256-7a09O0Jb8WncxeB32ywmQEMqJdEFLrOG/XVT9bdII6I%3D&rev=653d411b9e46076a7878be9574ed6b3bd627cff1&revCount=195
+    ├───agenix: github:ryantm/agenix/d8c973fd228949736dedf61b7f8cc1ece3236792
+    │   ├───darwin: github:lnl7/nix-darwin/87b9d090ad39b25b2400029c64825fc2a8868943
+    │   │   └───nixpkgs follows input 'jungle/agenix/nixpkgs'
+    │   ├───home-manager: github:nix-community/home-manager/32d3e39c491e2f91152c84f8ad8b003420eab0a1
+    │   │   └───nixpkgs follows input 'jungle/agenix/nixpkgs'
+    │   └───nixpkgs follows input 'jungle/nixpkgs'
+    ├───bscpkgs: git+https://pm.bsc.es/gitlab/rarias/bscpkgs.git?ref=refs/heads/master&rev=3a4062ac04be6263c64a481420d8e768c2521b80
+    │   └───nixpkgs follows input 'jungle/nixpkgs'
+    └───nixpkgs: github:NixOS/nixpkgs/e56990880811a451abd32515698c712788be5720
+```
+
+Now, as long as we keep these two files `flake.nix` and `flake.lock`, we can
+reproduce the same shell in the future, so let's add the lock file into git too.
+
+```txt
+hut% git commit -m 'Add flake.lock file'
+[master d3725ec] Add flake.lock file
+ 1 file changed, 135 insertions(+)
+ create mode 100644 flake.lock
+hut% git status
+On branch master
+nothing to commit, working tree clean
+```
+
+## Using the shell with nix develop
+
+Now, the invocations of `nix develop` won't complain that the git tree is dirty
+anymore and will enter the shell:
+
+```txt
+hut% nix develop
+[nix-develop]$
+```
+
+And the requested packages are now available:
+
+```txt
+[nix-develop]$ which ovniemu
+/nix/store/0yzas8007x9djlpbb0pckcr1vhd0mcfy-ovni-1.3.0/bin/ovniemu
+```
+
+The packages of the shell are listed in the `$buildInputs` variable, in case you
+need to examine them:
+
+```txt
+[nix-develop]$ printf '%s\n' $buildInputs
+/nix/store/0yzas8007x9djlpbb0pckcr1vhd0mcfy-ovni-1.3.0
+/nix/store/lnjirzllhjn2fadlqzrz7a547iawl8jc-osu-micro-benchmarks-7.1-1
+[nix-develop]$ exit
+hut%
+```
+
+In the [next chapter](../ch3) we will see how to add more packages and also how to modify
+them.

web/content/intro-nix/ch3.md

@@ -0,0 +1,160 @@
+---
+title: "Chapter 3: Custom packages"
+date: 2023-09-15
+weight: 3
+---
+
+## Adding more packages
+
+So far we have define all the packages using:
+
+```nix
+pkgs.mkShell rec {
+  pname = "my-shell";
+  buildInputs = with pkgs.bsc; [
+    ovni osumb # other packages here...
+  ];
+};
+```
+
+This line specifies that all packages come from the `pkgs.bsc` set. We can add
+additional packages adding them to the list:
+
+```nix
+pkgs.mkShell rec {
+  pname = "my-shell";
+  buildInputs = with pkgs.bsc; [
+    ovni osumb sonar
+  ];
+};
+```
+
+And running `nix develop` again:
+
+```txt
+hut% nix develop
+warning: Git tree '/home/Computational/rarias/jungle-examples' is dirty
+[nix-develop]$ printf '%s\n' $buildInputs
+/nix/store/0yzas8007x9djlpbb0pckcr1vhd0mcfy-ovni-1.3.0
+/nix/store/lnjirzllhjn2fadlqzrz7a547iawl8jc-osu-micro-benchmarks-7.1-1
+/nix/store/fjxj4xs0wblw3jyhp4vsrsfnlfwawifa-sonar-0.1.0
+```
+
+In the jungle cluster, the default MPI implementation is currently set to MPICH,
+as it can be shown with ldd:
+
+```txt
+[nix-develop]$ ldd $(which ovnisync) | grep mpi
+        libmpi.so.12 => /nix/store/nnnaly6hgylravdrmqkhpx1ndg5p79nc-mpich-4.1.2/lib/libmpi.so.12 (0x00007ffff5200000)
+```
+
+Now, what if we want to replace the MPI implementation by another one?
+
+## Modifying a package
+
+You notice that the packages we are using are coming directly from the ones
+specified in jungle. However, what if we need to modify some option at build
+time or change a dependency?
+
+The Nix language is used to describe how to build each package, and can be
+extended to create derived versions very easily.
+
+Let's focus on the `ovni` package. First, to load the definition we can use the
+`nix edit` command, which opens the definition file using the editor defined in
+`$EDITOR`:
+
+```txt
+hut% nix edit jungle#bsc.ovni
+...
+```
+
+This particular package has several inputs that can be modified directly:
+
+```txt
+{
+  stdenv
+, lib
+, cmake
+, mpi
+, fetchFromGitHub
+, useGit ? false
+, gitBranch ? "master"
+, gitUrl ? "ssh://git@bscpm03.bsc.es/rarias/ovni.git"
+, gitCommit ? "d0a47783f20f8b177a48418966dae45454193a6a"
+, enableDebug ? false
+}:
+...
+```
+
+For example, the `enableDebug` flag, currently set to false, affects how the
+build is configured:
+
+```txt
+cmakeBuildType = if (enableDebug) then "Debug" else "Release";
+```
+
+Now, to change this option we could replace `ovni` for our version:
+
+```nix
+{
+  inputs.jungle.url = "jungle";
+  nixConfig.bash-prompt = "\[nix-develop\]$ ";
+
+  outputs = { self, jungle }:  
+  let
+    pkgs = jungle.outputs.packages.x86_64-linux;
+    ovniDebug = pkgs.bsc.ovni.override { enableDebug = true; };
+  in {
+    devShells.x86_64-linux.default = pkgs.mkShell rec {
+      pname = "my-shell";
+      buildInputs = with pkgs.bsc; [
+        ovniDebug osumb sonar
+      ];
+    };
+  };
+}
+```
+
+And then, when we now enter the develop shell we can see that ovni gets build
+with the Debug option:
+
+```txt
+hut% nix develop -L
+warning: Git tree '/home/Computational/rarias/jungle-examples' is dirty
+ovni> unpacking sources
+ovni> unpacking source archive /nix/store/cz4si0vsw85r9s6dyiqr5ybngh9aympi-source
+ovni> source root is source
+ovni> patching sources
+ovni> updateAutotoolsGnuConfigScriptsPhase
+ovni> configuring
+ovni> fixing cmake files...
+ovni> cmake flags: ... -DCMAKE_BUILD_TYPE=Debug ...
+...
+[nix-develop]$ which ovniver
+/nix/store/hg0xs7fpibwjhsp9ajqfcbffsh69mrsm-ovni-1.3.0/bin/ovniver
+
+[nix-develop]$ file $(which ovniver) | fold
+/nix/store/hg0xs7fpibwjhsp9ajqfcbffsh69mrsm-ovni-1.3.0/bin/ovniver: ELF 64-bit L
+SB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/st
+ore/9la894yvmmksqlapd4v16wvxpaw3rg70-glibc-2.37-8/lib/ld-linux-x86-64.so.2, for
+GNU/Linux 3.10.0, with debug_info, not stripped
+```
+
+And we see that the ovniver program is now compiled with debug symbols.
+
+However, this *only* replaces the ovni package that we specify in the shell. The
+sonar library also depends on ovni, but that package is still using the old one:
+
+```txt
+[nix-develop]$ find $buildInputs -name 'libovni.so.1'
+/nix/store/hg0xs7fpibwjhsp9ajqfcbffsh69mrsm-ovni-1.3.0/lib/libovni.so.1
+
+[nix-develop]$ find $buildInputs -name 'libsonar-mpi.so'
+/nix/store/fjxj4xs0wblw3jyhp4vsrsfnlfwawifa-sonar-0.1.0/lib/libsonar-mpi.so
+
+[nix-develop]$ ldd /nix/store/fjxj4xs0wblw3jyhp4vsrsfnlfwawifa-sonar-0.1.0/lib/libsonar-mpi.so | grep ovni
+        libovni.so.1 => /nix/store/0yzas8007x9djlpbb0pckcr1vhd0mcfy-ovni-1.3.0/lib/libovni.so.1 (0x00007ffff7f8d000)
+```
+
+In the [next chapter](../ch4) we will see how to replace packages in such a way
+that all the dependences are automatically updated too.

web/content/intro-nix/ch4.md

@@ -0,0 +1,29 @@
+---
+title: "Chapter 4: Adding an overlay"
+date: 2023-09-15
+weight: 4
+---
+
+NOTE: We shouldn't be instructing users to use an overlay to replace packages in
+`bsc.` until we have determined if we move them to the root attribute set
+first!
+
+```nix
+{
+  inputs.jungle.url = "jungle";
+  nixConfig.bash-prompt = "\[nix-develop\]$ ";
+
+  outputs = { self, jungle }:  
+  let
+    pkgs = jungle.outputs.packages.x86_64-linux;
+    ovniDebug = pkgs.bsc.ovni.override { enableDebug = true; };
+  in {
+    devShells.x86_64-linux.default = pkgs.mkShell rec {
+      pname = "my-shell";
+      buildInputs = with pkgs.bsc; [
+        ovniDebug osumb sonar
+      ];
+    };
+  };
+}
+```

web/content/lake/_index.md

@@ -0,0 +1,10 @@
+---
+title: "Lake"
+description: "Data storage"
+date: 2023-06-13T19:36:57+02:00
+draft: true
+---
+
+![Lake](lake.jpg)
+
+Data storage

web/content/owl/_index.md

@@ -0,0 +1,10 @@
+---
+title: "Owl"
+description: "Low system noise"
+---
+
+![Owl](owl.jpg)
+
+Much like the silent flight of an owl at night, these nodes are configured to
+minimize the system noise and let programs run undisturbed. The list of nodes is
+`owl[1-2]` and are available for jobs with SLURM.

web/content/posts/2023-09-12/_index.md

@@ -0,0 +1,71 @@
+---
+title: "Update 2023-09-12"
+author: "Rodrigo Arias Mallo"
+date: 2023-09-12
+---
+
+This is a summary of notable changes introduced in the jungle cluster in the
+last months.
+
+### New Ceph filesystem available
+
+We have installed the latest [Ceph filesystem][1] (18.2.0) which stores three
+redundant copies of the data so a failure in one disk doesn't cause data loss.
+It is mounted in /ceph and available for use in the owl1, owl2 and hut
+nodes. For now it provides 2.8 TiB of space and it is expected to
+increase when the last storage node is installed.
+
+[1]: https://en.wikipedia.org/wiki/Ceph_(software)
+
+The throughput is limited by the 1 Gigabit Ethernet speed, but should be
+reasonably fast for most workloads. Here is a test with dd which reaches the
+network limit:
+
+```txt
+hut% dd if=/dev/urandom of=/ceph/rarias/urandom bs=1M count=1024
+1024+0 records in
+1024+0 records out
+1073741824 bytes (1,1 GB, 1,0 GiB) copied, 8,98544 s, 119 MB/s
+```
+
+### SLURM power save
+
+The SLURM daemon has been configured to power down the nodes after one hour of
+idling. When a new job is allocated to a node that is powered off, it is
+automatically turned on and as soon as it becomes available it will execute the
+job. Here is an example with two nodes that boot and execute a simple job that
+shows the date.
+
+```txt
+hut% date; srun -N 2 date
+2023-09-12T17:36:09 CEST
+2023-09-12T17:38:26 CEST
+2023-09-12T17:38:18 CEST
+```
+
+You can expect a similar delay (around 2-3 min) while the nodes are starting.
+Notice that while the nodes are kept on, the delay is not noticeable:
+
+```txt
+hut% date; srun -N 2 date
+2023-09-12T17:40:04 CEST
+2023-09-12T17:40:04 CEST
+2023-09-12T17:40:04 CEST
+```
+
+### Power and temperature monitoring
+
+In the cluster, we monitor the temperature and the power draw of all nodes. This
+allows us to understand which machines are not being used and turn them off to
+save energy that otherwise would be wasted. Here is an example where some nodes
+are powered off to save energy:
+
+![power](./power.png)
+
+We also configured the nodes to work at low CPU frequencies, so the temperature
+is kept low to increase the lifespan of the node components. Towards these
+goals, we have configured two alerts that trigger when the CPUs of a node
+exceeds the limit temperature of 80 °C or when the power draw exceeds 350 W.
+
+By keeping the power consumption and temperatures controlled, we can safely
+incorporate more machines that will only be used on demand.

web/hugo.toml

@@ -0,0 +1,8 @@
+baseURL = 'https://jungle.bsc.es/'
+languageCode = 'en-us'
+title = 'The jungle'
+theme = 'PaperMod'
+sectionPagesMenu = "main"
+
+[params]
+ShowBreadCrumbs = true

web/themes/PaperMod/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,50 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[BUG]"
+labels: bug
+assignees: ''
+
+---
+
+<!--
+
+## READ BEFORE OPENING ISSUES
+
+Please fill the template below
+- **DO NOT** ask for instructions.
+- Use Discussions section if you need help
+- See project wiki https://github.com/adityatelange/hugo-PaperMod/wiki
+- Read FAQs section https://github.com/adityatelange/hugo-PaperMod/wiki/FAQs
+- Search for previous issues/ pull requests
+
+-->
+
+**Describe the bug**
+<!-- A clear and concise description of what the bug is. -->
+
+ - Device/Os: [e.g. Android 10]
+ - Type: [e.g. Desktop/Mobile]
+ - Browser and version [e.g. Chrome 86.0]:
+ - Hugo Version [ >=0.97.1 expected]:
+ - Theme Version [e.g. v4.0, master, or commit-id ]:
+
+**Steps to reproduce the behavior:**
+<!--
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+-->
+
+**Expected behavior**:
+<!-- A clear and concise description of what you expected to happen. -->
+
+**Repo/Source where this issue can be reproduced**:
+<!-- Please link source code of website where the said issue can be reproduced -->
+
+**Screenshots**
+<!-- If applicable, add screenshots to help explain your problem. -->
+
+**Additional context**
+<!--Add any other context about the problem here. -->

web/themes/PaperMod/.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,5 @@
+blank_issues_enabled: false
+contact_links:
+  - name: PaperMod Discussions
+    url: https://github.com/adityatelange/hugo-PaperMod/discussions
+    about: Please ask and answer questions/doubts here, do not open an issue for questions.

web/themes/PaperMod/.github/ISSUE_TEMPLATE/new-blank-issue.md

@@ -0,0 +1,7 @@
+---
+name: New Blank Issue
+about: Anything other than bug report
+title: ""
+labels: ""
+assignees: ""
+---

web/themes/PaperMod/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,44 @@
+<!--
+
+## READ BEFORE OPENING A PR
+
+Thank you for contributing to hugo-PaperMod!
+Please fill out the following questions to make it easier for us to review your
+changes. You do not need to check all the boxes below.
+
+**NOTE**: PaperMod does not have any external dependencies fetched from 3rd party
+CDN servers. However we do have custom Head/Footer extender templates which you can use
+to add those to your website.
+https://github.com/adityatelange/hugo-PaperMod/wiki/FAQs#custom-head--footer
+
+-->
+
+
+**What does this PR change? What problem does it solve?**
+
+<!--
+Describe the changes and their purpose here, as detailed as and if  needed.
+
+Please do not add 2 unrelated changes in a single PR as it is difficult to track/revert those in future.
+-->
+
+
+**Was the change discussed in an issue or in the Discussions before?**
+
+<!--
+Link issues and relevant Discussions posts here.
+
+If this PR resolves an issue on GitHub, use "Closes #1234" so that the issue
+is closed automatically when this PR is merged.
+-->
+
+
+## PR Checklist
+
+- [ ] This change adds/updates translations and I have used the [template present here](https://github.com/adityatelange/hugo-PaperMod/wiki/Translations#want-to-add-your-language-).
+- [ ] I have enabled [maintainer edits for this PR](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork).
+- [ ] I have verified that the code works as described/as intended.
+- [ ] This change adds a Social Icon which has a permissive license to use it.
+- [ ] This change **does not** include any CDN resources/links.
+- [ ] This change **does not** include any unrelated scripts such as bash and python scripts.
+- [ ] This change updates the overridden internal templates from HUGO's repository.

web/themes/PaperMod/.github/stale.yml

@@ -0,0 +1,17 @@
+# Number of days of inactivity before an issue becomes stale
+daysUntilStale: 7
+# Number of days of inactivity before a stale issue is closed
+daysUntilClose: 3
+# Issues with these labels will never be considered stale
+exemptLabels:
+  - pinned
+  - keep
+# Label to use when marking an issue as stale
+staleLabel: stale
+# Comment to post when marking an issue as stale. Set to `false` to disable
+markComment: >
+  This issue has been automatically marked as stale because it has not had
+  recent activity. It will be closed if no further activity occurs. Thank you
+  for your contributions.
+# Comment to post when closing a stale issue. Set to `false` to disable
+closeComment: false

web/themes/PaperMod/.github/workflows/gh-pages.yml

@@ -0,0 +1,80 @@
+name: Deploy Hugo PaperMod Demo to Pages
+
+on:
+  push:
+    paths-ignore:
+      - "images/**"
+      - "LICENSE"
+      - "README.md"
+    branches:
+      - master
+      - exampleSite
+  workflow_dispatch:
+    # manual run
+    inputs:
+      hugoVersion:
+        description: "Hugo Version"
+        required: false
+        default: "0.97.1"
+
+# Allow one concurrent deployment
+concurrency:
+  group: "pages"
+  cancel-in-progress: true
+
+# Default to bash
+defaults:
+  run:
+    shell: bash
+
+# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+jobs:
+  # Build job
+  build:
+    runs-on: ubuntu-latest
+    env:
+      HUGO_VERSION: "0.97.1"
+    steps:
+      - name: Check version
+        if: ${{ github.event.inputs.hugoVersion }}
+        run: export HUGO_VERSION="${{ github.event.inputs.hugoVersion }}"
+      - name: Install Hugo CLI
+        run: |
+          wget -O ${{ runner.temp }}/hugo.deb https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_${HUGO_VERSION}_Linux-64bit.deb \
+          && sudo dpkg -i ${{ runner.temp }}/hugo.deb
+      - name: Checkout
+        uses: actions/checkout@v3
+        with:
+          ref: exampleSite
+      - name: Setup Pages
+        id: pages
+        uses: actions/configure-pages@v1
+      - name: Get Theme
+        run: git submodule update --init --recursive
+      - name: Update theme to Latest commit
+        run: git submodule update --remote --merge
+      - name: Build with Hugo
+        run: |
+          hugo \
+            --buildDrafts --gc --verbose \
+            --baseURL ${{ steps.pages.outputs.base_url }}
+      - name: Upload artifact
+        uses: actions/upload-pages-artifact@v1
+        with:
+          path: ./public
+  # Deployment job
+  deploy:
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    runs-on: ubuntu-latest
+    needs: build
+    steps:
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v1

web/themes/PaperMod/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2020 nanxiaobei and adityatelange
+Copyright (c) 2021-2023 adityatelange
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

web/themes/PaperMod/README.md

@@ -0,0 +1,103 @@
+<h1 align=center>Hugo PaperMod | <a href="https://adityatelange.github.io/hugo-PaperMod/" rel="nofollow">Demo</a></h1>
+
+<h4 align=center>☄️ Fast | ☁️ Fluent | 🌙 Smooth | 📱 Responsive</h4>
+<br>
+
+> Hugo PaperMod is a theme based on [hugo-paper](https://github.com/nanxiaobei/hugo-paper).
+> The goal of this project is to add more features and customization to the og theme.
+
+**Documentation** can be found here: [**📚 Wiki**](https://github.com/adityatelange/hugo-PaperMod/wiki)
+
+**ExampleSite** can be found here: [**exampleSite**](https://github.com/adityatelange/hugo-PaperMod/tree/exampleSite). Demo is built up with [exampleSite](https://github.com/adityatelange/hugo-PaperMod/tree/exampleSite) as source.
+
+[![hugo-papermod](https://img.shields.io/badge/Hugo--Themes-@PaperMod-blue)](https://themes.gohugo.io/themes/hugo-papermod/)
+[![Minimum Hugo Version](https://img.shields.io/static/v1?label=HUGO-version&message=>0.97.1&color=blue&logo=hugo)](https://github.com/gohugoio/hugo/releases/tag/v0.97.1)
+[![Discord](https://img.shields.io/discord/971046860317921340?label=Discord&logo=discord)](https://discord.gg/ahpmTvhVmp)
+[![GitHub](https://img.shields.io/github/license/adityatelange/hugo-PaperMod)](https://github.com/adityatelange/hugo-PaperMod/blob/master/LICENSE)
+![code-size](https://img.shields.io/github/languages/code-size/adityatelange/hugo-PaperMod)
+
+---
+
+<p align="center">
+  <kbd><img src="https://user-images.githubusercontent.com/21258296/114303440-bfc0ae80-9aeb-11eb-8cfa-48a4bb385a6d.png" alt="Mockup image" title="Mockup"/></kbd>
+</p>
+
+---
+
+## Features/Mods 💥
+
+-   Uses Hugo's asset generator with pipelining, fingerprinting, bundling and minification by default.
+-   3 Modes:
+    -   [Regular Mode.](https://github.com/adityatelange/hugo-PaperMod/wiki/Features#regular-mode-default-mode)
+    -   [Home-Info Mode.](https://github.com/adityatelange/hugo-PaperMod/wiki/Features#home-info-mode)
+    -   [Profile Mode.](https://github.com/adityatelange/hugo-PaperMod/wiki/Features#profile-mode)
+-   Table of Content Generation (newer implementation).
+-   Archive of posts.
+-   Social Icons (home-info and profile-mode)
+-   Social-Media Share buttons on posts.
+-   Menu location indicator.
+-   Multilingual support. (with language selector)
+-   Taxonomies
+-   Cover image for each post (with Responsive image support).
+-   Light/Dark theme (automatic theme switch a/c to browser theme and theme-switch button).
+-   SEO Friendly.
+-   Multiple Author support.
+-   Search Page with Fuse.js
+-   Other Posts suggestion below a post
+-   Breadcrumb Navigation
+-   Code Block Copy buttons
+-   No webpack, nodejs and other dependencies are required to edit the theme.
+
+Read Wiki For More Details => **[PaperMod - Features](https://github.com/adityatelange/hugo-PaperMod/wiki/Features)**
+
+---
+
+## Install/Update 📥
+
+Read Wiki For More Details => **[PaperMod - Installation](https://github.com/adityatelange/hugo-PaperMod/wiki/Installation)**
+
+---
+
+## FAQs / How To's Guide 🙋
+
+Read Wiki For More Details => **[PaperMod-FAQs](https://github.com/adityatelange/hugo-PaperMod/wiki/FAQs)**
+
+---
+
+## Social-Icons/Share-Icons 🖼️
+
+Read Wiki For More Details => **[PaperMod-Icons](https://github.com/adityatelange/hugo-PaperMod/wiki/Icons)**
+
+---
+
+## Release Changelog 📃
+
+Release ChangeLog has info about stuff added: **[Releases](https://github.com/adityatelange/hugo-PaperMod/releases)**
+
+---
+
+## [Pagespeed Insights (100% ?)](https://pagespeed.web.dev/report?url=https://adityatelange.github.io/hugo-PaperMod/) 👀
+
+---
+
+## Support 🫶
+
+-   Star 🌟 this repository.
+-   Help spread the word about PaperMod by sharing it on social media and recommending it to your friends. 🗣️
+-   You can also sponsor 🏅 on [Github Sponsors](https://github.com/sponsors/adityatelange) / [Ko-Fi](https://ko-fi.com/adityatelange).
+
+---
+
+## Special Thanks 🌟
+
+-   [**Highlight.js**](https://github.com/highlightjs/highlight.js)
+-   [**Fuse.js**](https://github.com/krisk/fuse)
+-   [**Feather Icons**](https://github.com/feathericons/feather)
+-   [**Simple Icons**](https://github.com/simple-icons/simple-icons)
+-   **All Contributors and Supporters**
+
+---
+
+## Stargazers over time 📈
+
+<kbd>[![Stargazers over time](https://starchart.cc/adityatelange/hugo-PaperMod.svg)](https://starchart.cc/adityatelange/hugo-PaperMod)</kbd>

web/themes/PaperMod/assets/css/common/404.css

@@ -0,0 +1,11 @@
+.not-found {
+    position: absolute;
+    left: 0;
+    right: 0;
+    display: flex;
+    align-items: center;
+    justify-content: center;
+    height: 80%;
+    font-size: 160px;
+    font-weight: 700;
+}

web/themes/PaperMod/assets/css/common/archive.css

@@ -0,0 +1,44 @@
+.archive-posts {
+    width: 100%;
+    font-size: 16px;
+}
+
+.archive-year {
+    margin-top: 40px;
+}
+
+.archive-year:not(:last-of-type) {
+    border-bottom: 2px solid var(--border);
+}
+
+.archive-month {
+    display: flex;
+    align-items: flex-start;
+    padding: 10px 0;
+}
+
+.archive-month-header {
+    margin: 25px 0;
+    width: 200px;
+}
+
+.archive-month:not(:last-of-type) {
+    border-bottom: 1px solid var(--border);
+}
+
+.archive-entry {
+    position: relative;
+    padding: 5px;
+    margin: 10px 0;
+}
+
+.archive-entry-title {
+    margin: 5px 0;
+    font-weight: 400;
+}
+
+.archive-count,
+.archive-meta {
+    color: var(--secondary);
+    font-size: 14px;
+}

web/themes/PaperMod/assets/css/common/footer.css

@@ -0,0 +1,60 @@
+.footer,
+.top-link {
+    font-size: 12px;
+    color: var(--secondary);
+}
+
+.footer {
+    max-width: calc(var(--main-width) + var(--gap) * 2);
+    margin: auto;
+    padding: calc((var(--footer-height) - var(--gap)) / 2) var(--gap);
+    text-align: center;
+    line-height: 24px;
+}
+
+.footer span {
+    margin-inline-start: 1px;
+    margin-inline-end: 1px;
+}
+
+.footer span:last-child {
+    white-space: nowrap;
+}
+
+.footer a {
+    color: inherit;
+    border-bottom: 1px solid var(--secondary);
+}
+
+.footer a:hover {
+    border-bottom: 1px solid var(--primary);
+}
+
+.top-link {
+    visibility: hidden;
+    position: fixed;
+    bottom: 60px;
+    right: 30px;
+    z-index: 99;
+    background: var(--tertiary);
+    width: 42px;
+    height: 42px;
+    padding: 12px;
+    border-radius: 64px;
+    transition: visibility 0.5s, opacity 0.8s linear;
+}
+
+.top-link,
+.top-link svg {
+    filter: drop-shadow(0px 0px 0px var(--theme));
+}
+
+.footer a:hover,
+.top-link:hover {
+    color: var(--primary);
+}
+
+.top-link:focus,
+#theme-toggle:focus {
+    outline: 0;
+}

web/themes/PaperMod/assets/css/common/header.css

@@ -0,0 +1,93 @@
+.nav {
+    display: flex;
+    flex-wrap: wrap;
+    justify-content: space-between;
+    max-width: calc(var(--nav-width) + var(--gap) * 2);
+    margin-inline-start: auto;
+    margin-inline-end: auto;
+    line-height: var(--header-height);
+}
+
+.nav a {
+    display: block;
+}
+
+.logo,
+#menu {
+    display: flex;
+    margin: auto var(--gap);
+}
+
+.logo {
+    flex-wrap: inherit;
+}
+
+.logo a {
+    font-size: 24px;
+    font-weight: 700;
+}
+
+.logo a img, .logo a svg {
+    display: inline;
+    vertical-align: middle;
+    pointer-events: none;
+    transform: translate(0, -10%);
+    border-radius: 6px;
+    margin-inline-end: 8px;
+}
+
+button#theme-toggle {
+    font-size: 26px;
+    margin: auto 4px;
+}
+
+body.dark #moon {
+    vertical-align: middle;
+    display: none;
+}
+
+body:not(.dark) #sun {
+    display: none;
+}
+
+#menu {
+    list-style: none;
+    word-break: keep-all;
+    overflow-x: auto;
+    white-space: nowrap;
+}
+
+#menu li + li {
+    margin-inline-start: var(--gap);
+}
+
+#menu a {
+    font-size: 16px;
+}
+
+#menu .active {
+    font-weight: 500;
+    border-bottom: 2px solid currentColor;
+}
+
+.lang-switch li,
+.lang-switch ul,
+.logo-switches {
+    display: inline-flex;
+    margin: auto 4px;
+}
+
+.lang-switch {
+    display: flex;
+    flex-wrap: inherit;
+}
+
+.lang-switch a {
+    margin: auto 3px;
+    font-size: 16px;
+    font-weight: 500;
+}
+
+.logo-switches {
+    flex-wrap: inherit;
+}

web/themes/PaperMod/assets/css/common/main.css

@@ -0,0 +1,68 @@
+.main {
+    position: relative;
+    min-height: calc(100vh - var(--header-height) - var(--footer-height));
+    max-width: calc(var(--main-width) + var(--gap) * 2);
+    margin: auto;
+    padding: var(--gap);
+}
+
+.page-header h1 {
+    font-size: 40px;
+}
+
+.pagination {
+    display: flex;
+}
+
+.pagination a {
+    color: var(--theme);
+    font-size: 13px;
+    line-height: 36px;
+    background: var(--primary);
+    border-radius: calc(36px / 2);
+    padding: 0 16px;
+}
+
+.pagination .next {
+    margin-inline-start: auto;
+}
+
+.social-icons {
+    padding: 12px 0;
+}
+
+.social-icons a:not(:last-of-type) {
+    margin-inline-end: 12px;
+}
+
+.social-icons a svg {
+    height: 26px;
+    width: 26px;
+}
+
+code {
+    direction: ltr;
+}
+
+div.highlight,
+pre {
+    position: relative;
+}
+
+.copy-code {
+    display: none;
+    position: absolute;
+    top: 4px;
+    right: 4px;
+    color: rgba(255, 255, 255, 0.8);
+    background: rgba(78, 78, 78, 0.8);
+    border-radius: var(--radius);
+    padding: 0 5px;
+    font-size: 14px;
+    user-select: none;
+}
+
+div.highlight:hover .copy-code,
+pre:hover .copy-code {
+    display: block;
+}