Compare commits
9 Commits
80b42c302e
...
c011f25954
| Author | SHA1 | Date | |
|---|---|---|---|
| c011f25954 | |||
| 19e865c6ec | |||
| 4ae7ad31a3 | |||
| 83dee4df28 | |||
| 4c17c95033 | |||
| c56073b78e | |||
| a560b25bf8 | |||
| 87817c523c | |||
| dd2da36dfd |
4
.gitignore
vendored
4
.gitignore
vendored
@ -1,3 +1,3 @@
|
|||||||
|
source
|
||||||
|
result
|
||||||
**.swp
|
**.swp
|
||||||
/result
|
|
||||||
/misc
|
|
||||||
|
|||||||
@ -3,4 +3,4 @@ build:bsc-ci.all:
|
|||||||
tags:
|
tags:
|
||||||
- nix
|
- nix
|
||||||
script:
|
script:
|
||||||
- nix build -L --no-link --print-out-paths .#bsc-ci.all
|
- nix build -L "jungle#bsc-ci.all" --override-input bscpkgs . -v --show-trace
|
||||||
|
|||||||
2
COPYING
2
COPYING
@ -1,4 +1,4 @@
|
|||||||
Copyright (c) 2020-2025 Barcelona Supercomputing Center
|
Copyright (c) 2020-2021 Barcelona Supercomputing Center
|
||||||
Copyright (c) 2003-2020 Eelco Dolstra and the Nixpkgs/NixOS contributors
|
Copyright (c) 2003-2020 Eelco Dolstra and the Nixpkgs/NixOS contributors
|
||||||
|
|
||||||
Permission is hereby granted, free of charge, to any person obtaining
|
Permission is hereby granted, free of charge, to any person obtaining
|
||||||
|
|||||||
10
README.md
10
README.md
@ -1,9 +1 @@
|
|||||||
# Jungle
|
Nix overlay with BSC packages.
|
||||||
|
|
||||||
This repository provides two components that can be used independently:
|
|
||||||
|
|
||||||
- A Nix overlay with packages used at BSC (formerly known as bscpkgs). Access
|
|
||||||
them directly with `nix shell .#<pkgname>`.
|
|
||||||
|
|
||||||
- NixOS configurations for jungle machines. Use `nixos-rebuild switch --flake .`
|
|
||||||
to upgrade the current machine.
|
|
||||||
|
|||||||
176
doc/install.md
176
doc/install.md
@ -1,176 +0,0 @@
|
|||||||
# Installing NixOS in a new node
|
|
||||||
|
|
||||||
This article shows the steps to install NixOS in a node following the
|
|
||||||
configuration of the repo.
|
|
||||||
|
|
||||||
## Enable the serial console
|
|
||||||
|
|
||||||
By default, the nodes have the serial console disabled in the GRUB and also boot
|
|
||||||
without the serial enabled.
|
|
||||||
|
|
||||||
To enable the serial console in the GRUB, set in /etc/default/grub the following
|
|
||||||
lines:
|
|
||||||
|
|
||||||
```
|
|
||||||
GRUB_TERMINAL="console serial"
|
|
||||||
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
|
|
||||||
```
|
|
||||||
|
|
||||||
To boot Linux with the serial enabled, so you can see the boot log and login via
|
|
||||||
serial set:
|
|
||||||
|
|
||||||
```
|
|
||||||
GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 console=tty0"
|
|
||||||
```
|
|
||||||
|
|
||||||
Then update the grub config:
|
|
||||||
|
|
||||||
```
|
|
||||||
# grub2-mkconfig -o /boot/grub2/grub.cfg
|
|
||||||
```
|
|
||||||
|
|
||||||
And reboot.
|
|
||||||
|
|
||||||
## Prepare the disk
|
|
||||||
|
|
||||||
Create a main partition and label it `nixos` following [the manual][1].
|
|
||||||
|
|
||||||
[1]: https://nixos.org/manual/nixos/stable/index.html#sec-installation-manual-partitioning.
|
|
||||||
|
|
||||||
```
|
|
||||||
# disk=/dev/sdX
|
|
||||||
# parted $disk -- mklabel msdos
|
|
||||||
# parted $disk -- mkpart primary 1MB -8GB
|
|
||||||
# parted $disk -- mkpart primary linux-swap -8GB 100%
|
|
||||||
# parted $disk -- set 1 boot on
|
|
||||||
```
|
|
||||||
|
|
||||||
Then create an etx4 filesystem, labeled `nixos` where the system will be
|
|
||||||
installed. **Ensure that no other partition has the same label.**
|
|
||||||
|
|
||||||
```
|
|
||||||
# mkfs.ext4 -L nixos "${disk}1"
|
|
||||||
# mkswap -L swap "${disk}2"
|
|
||||||
# mount ${disk}1 /mnt
|
|
||||||
# lsblk -f $disk
|
|
||||||
NAME FSTYPE LABEL UUID MOUNTPOINT
|
|
||||||
sdX
|
|
||||||
`-sdX1 ext4 nixos 10d73b75-809c-4fa3-b99d-4fab2f0d0d8e /mnt
|
|
||||||
```
|
|
||||||
|
|
||||||
## Prepare nix and nixos-install
|
|
||||||
|
|
||||||
Mount the nix store from the hut node in read-only /nix.
|
|
||||||
|
|
||||||
```
|
|
||||||
# mkdir /nix
|
|
||||||
# mount -o ro hut:/nix /nix
|
|
||||||
```
|
|
||||||
|
|
||||||
Get the nix binary and nixos-install tool from hut:
|
|
||||||
|
|
||||||
```
|
|
||||||
# ssh hut 'readlink -f $(which nix)'
|
|
||||||
/nix/store/0sxbaj71c4c4n43qhdxm31f56gjalksw-nix-2.13.3/bin/nix
|
|
||||||
# ssh hut 'readlink -f $(which nixos-install)'
|
|
||||||
/nix/store/9yq8ps06ysr2pfiwiij39ny56yk3pdcs-nixos-install/bin/nixos-install
|
|
||||||
```
|
|
||||||
|
|
||||||
And add them to the PATH:
|
|
||||||
|
|
||||||
```
|
|
||||||
# export PATH=$PATH:/nix/store/0sxbaj71c4c4n43qhdxm31f56gjalksw-nix-2.13.3/bin
|
|
||||||
# export PATH=$PATH:/nix/store/9yq8ps06ysr2pfiwiij39ny56yk3pdcs-nixos-install/bin/
|
|
||||||
# nix --version
|
|
||||||
nix (Nix) 2.13.3
|
|
||||||
```
|
|
||||||
|
|
||||||
## Adapt owl configuration
|
|
||||||
|
|
||||||
Clone owl repo:
|
|
||||||
|
|
||||||
```
|
|
||||||
$ git clone git@bscpm03.bsc.es:rarias/owl.git
|
|
||||||
$ cd owl
|
|
||||||
```
|
|
||||||
|
|
||||||
Edit the configuration to your needs.
|
|
||||||
|
|
||||||
## Install from another Linux OS
|
|
||||||
|
|
||||||
Install nixOS into the storage drive.
|
|
||||||
|
|
||||||
```
|
|
||||||
# nixos-install --flake --root /mnt .#xeon0X
|
|
||||||
```
|
|
||||||
|
|
||||||
At this point, the nixOS grub has been installed into the nixos device, which
|
|
||||||
is not the default boot device. To keep both the old Linux and NixOS grubs, add
|
|
||||||
an entry into the old Linux grub to jump into the new grub.
|
|
||||||
|
|
||||||
```
|
|
||||||
# echo "
|
|
||||||
|
|
||||||
menuentry 'NixOS' {
|
|
||||||
insmod chain
|
|
||||||
search --no-floppy --label nixos --set root
|
|
||||||
configfile /boot/grub/grub.cfg
|
|
||||||
} " >> /etc/grub.d/40_custom
|
|
||||||
```
|
|
||||||
|
|
||||||
Rebuild grub config.
|
|
||||||
|
|
||||||
```
|
|
||||||
# grub2-mkconfig -o /boot/grub/grub.cfg
|
|
||||||
```
|
|
||||||
|
|
||||||
To boot into NixOS manually, reboot and select NixOS in the grub menu to boot
|
|
||||||
into NixOS.
|
|
||||||
|
|
||||||
To temporarily boot into NixOS only on the next reboot run:
|
|
||||||
|
|
||||||
```
|
|
||||||
# grub2-reboot 'NixOS'
|
|
||||||
```
|
|
||||||
|
|
||||||
To permanently boot into NixOS as the default boot OS, edit `/etc/default/grub/`:
|
|
||||||
|
|
||||||
```
|
|
||||||
GRUB_DEFAULT='NixOS'
|
|
||||||
```
|
|
||||||
|
|
||||||
And update grub.
|
|
||||||
|
|
||||||
```
|
|
||||||
# grub2-mkconfig -o /boot/grub/grub.cfg
|
|
||||||
```
|
|
||||||
|
|
||||||
## Build the nixos kexec image
|
|
||||||
|
|
||||||
```
|
|
||||||
# nix build .#nixosConfigurations.xeon02.config.system.build.kexecTree -v
|
|
||||||
```
|
|
||||||
|
|
||||||
## Chain NixOS in same disk with other systems
|
|
||||||
|
|
||||||
To install NixOS on a partition along another system which controls the GRUB,
|
|
||||||
first disable the grub device, so the GRUB is not installed in the disk by
|
|
||||||
NixOS (only the /boot files will be generated):
|
|
||||||
|
|
||||||
```
|
|
||||||
boot.loader.grub.device = "nodev";
|
|
||||||
```
|
|
||||||
|
|
||||||
Then add the following entry to the old GRUB configuration:
|
|
||||||
|
|
||||||
```
|
|
||||||
menuentry 'NixOS' {
|
|
||||||
insmod chain
|
|
||||||
search --no-floppy --label nixos --set root
|
|
||||||
configfile /boot/grub/grub.cfg
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
The partition with NixOS must have the label "nixos" for it to be found. New
|
|
||||||
system configuration entries will be stored in the GRUB configuration managed
|
|
||||||
by NixOS, so there is no need to change the old GRUB settings.
|
|
||||||
46
doc/trim.sh
46
doc/trim.sh
@ -1,46 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
|
|
||||||
# Trims the jungle repository by moving the website to its own repository and
|
|
||||||
# removing it from jungle. It also removes big pdf files and kernel
|
|
||||||
# configurations so the jungle repository is small.
|
|
||||||
|
|
||||||
set -e
|
|
||||||
|
|
||||||
if [ -e oldjungle -o -e newjungle -o -e website ]; then
|
|
||||||
echo "remove oldjungle/, newjungle/ and website/ first"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Clone the old jungle repo
|
|
||||||
git clone gitea@tent:rarias/jungle.git oldjungle
|
|
||||||
|
|
||||||
# First split the website into a new repository
|
|
||||||
mkdir website && git -C website init -b master
|
|
||||||
git-filter-repo \
|
|
||||||
--path web \
|
|
||||||
--subdirectory-filter web \
|
|
||||||
--source oldjungle \
|
|
||||||
--target website
|
|
||||||
|
|
||||||
# Then remove the website, pdf files and big kernel configs
|
|
||||||
mkdir newjungle && git -C newjungle init -b master
|
|
||||||
git-filter-repo \
|
|
||||||
--invert-paths \
|
|
||||||
--path web \
|
|
||||||
--path-glob 'doc*.pdf' \
|
|
||||||
--path-glob '**/kernel/configs/lockdep' \
|
|
||||||
--path-glob '**/kernel/configs/defconfig' \
|
|
||||||
--source oldjungle \
|
|
||||||
--target newjungle
|
|
||||||
|
|
||||||
set -x
|
|
||||||
|
|
||||||
du -sh oldjungle newjungle website
|
|
||||||
# 57M oldjungle
|
|
||||||
# 2,3M newjungle
|
|
||||||
# 6,4M website
|
|
||||||
|
|
||||||
du -sh --exclude=.git oldjungle newjungle website
|
|
||||||
# 30M oldjungle
|
|
||||||
# 700K newjungle
|
|
||||||
# 3,5M website
|
|
||||||
93
flake.lock
generated
93
flake.lock
generated
@ -1,107 +1,22 @@
|
|||||||
{
|
{
|
||||||
"nodes": {
|
"nodes": {
|
||||||
"agenix": {
|
|
||||||
"inputs": {
|
|
||||||
"darwin": "darwin",
|
|
||||||
"home-manager": "home-manager",
|
|
||||||
"nixpkgs": [
|
|
||||||
"nixpkgs"
|
|
||||||
],
|
|
||||||
"systems": "systems"
|
|
||||||
},
|
|
||||||
"locked": {
|
|
||||||
"lastModified": 1750173260,
|
|
||||||
"narHash": "sha256-9P1FziAwl5+3edkfFcr5HeGtQUtrSdk/MksX39GieoA=",
|
|
||||||
"owner": "ryantm",
|
|
||||||
"repo": "agenix",
|
|
||||||
"rev": "531beac616433bac6f9e2a19feb8e99a22a66baf",
|
|
||||||
"type": "github"
|
|
||||||
},
|
|
||||||
"original": {
|
|
||||||
"owner": "ryantm",
|
|
||||||
"repo": "agenix",
|
|
||||||
"type": "github"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"darwin": {
|
|
||||||
"inputs": {
|
|
||||||
"nixpkgs": [
|
|
||||||
"agenix",
|
|
||||||
"nixpkgs"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"locked": {
|
|
||||||
"lastModified": 1744478979,
|
|
||||||
"narHash": "sha256-dyN+teG9G82G+m+PX/aSAagkC+vUv0SgUw3XkPhQodQ=",
|
|
||||||
"owner": "lnl7",
|
|
||||||
"repo": "nix-darwin",
|
|
||||||
"rev": "43975d782b418ebf4969e9ccba82466728c2851b",
|
|
||||||
"type": "github"
|
|
||||||
},
|
|
||||||
"original": {
|
|
||||||
"owner": "lnl7",
|
|
||||||
"ref": "master",
|
|
||||||
"repo": "nix-darwin",
|
|
||||||
"type": "github"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"home-manager": {
|
|
||||||
"inputs": {
|
|
||||||
"nixpkgs": [
|
|
||||||
"agenix",
|
|
||||||
"nixpkgs"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"locked": {
|
|
||||||
"lastModified": 1745494811,
|
|
||||||
"narHash": "sha256-YZCh2o9Ua1n9uCvrvi5pRxtuVNml8X2a03qIFfRKpFs=",
|
|
||||||
"owner": "nix-community",
|
|
||||||
"repo": "home-manager",
|
|
||||||
"rev": "abfad3d2958c9e6300a883bd443512c55dfeb1be",
|
|
||||||
"type": "github"
|
|
||||||
},
|
|
||||||
"original": {
|
|
||||||
"owner": "nix-community",
|
|
||||||
"repo": "home-manager",
|
|
||||||
"type": "github"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nixpkgs": {
|
"nixpkgs": {
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1752436162,
|
"lastModified": 1752436162,
|
||||||
"narHash": "sha256-Kt1UIPi7kZqkSc5HVj6UY5YLHHEzPBkgpNUByuyxtlw=",
|
"narHash": "sha256-Kt1UIPi7kZqkSc5HVj6UY5YLHHEzPBkgpNUByuyxtlw=",
|
||||||
"owner": "NixOS",
|
"path": "/nix/store/zk8v61cpk1wprp9ld5ayc1g5fq4pdkwv-source",
|
||||||
"repo": "nixpkgs",
|
|
||||||
"rev": "dfcd5b901dbab46c9c6e80b265648481aafb01f8",
|
"rev": "dfcd5b901dbab46c9c6e80b265648481aafb01f8",
|
||||||
"type": "github"
|
"type": "path"
|
||||||
},
|
},
|
||||||
"original": {
|
"original": {
|
||||||
"owner": "NixOS",
|
"id": "nixpkgs",
|
||||||
"ref": "nixos-25.05",
|
"type": "indirect"
|
||||||
"repo": "nixpkgs",
|
|
||||||
"type": "github"
|
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"root": {
|
"root": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"agenix": "agenix",
|
|
||||||
"nixpkgs": "nixpkgs"
|
"nixpkgs": "nixpkgs"
|
||||||
}
|
}
|
||||||
},
|
|
||||||
"systems": {
|
|
||||||
"locked": {
|
|
||||||
"lastModified": 1681028828,
|
|
||||||
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
|
|
||||||
"owner": "nix-systems",
|
|
||||||
"repo": "default",
|
|
||||||
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
|
|
||||||
"type": "github"
|
|
||||||
},
|
|
||||||
"original": {
|
|
||||||
"owner": "nix-systems",
|
|
||||||
"repo": "default",
|
|
||||||
"type": "github"
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"root": "root",
|
"root": "root",
|
||||||
|
|||||||
74
flake.nix
74
flake.nix
@ -1,54 +1,28 @@
|
|||||||
{
|
{
|
||||||
inputs = {
|
inputs.nixpkgs.url = "nixpkgs";
|
||||||
nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
|
|
||||||
agenix.url = "github:ryantm/agenix";
|
|
||||||
agenix.inputs.nixpkgs.follows = "nixpkgs";
|
|
||||||
};
|
|
||||||
|
|
||||||
outputs = { self, nixpkgs, agenix, ... }:
|
outputs = { self, nixpkgs, ...}:
|
||||||
let
|
let
|
||||||
mkConf = name: nixpkgs.lib.nixosSystem {
|
# For now we only support x86
|
||||||
system = "x86_64-linux";
|
system = "x86_64-linux";
|
||||||
specialArgs = { inherit nixpkgs agenix; theFlake = self; };
|
pkgs = import nixpkgs {
|
||||||
modules = [ "${self.outPath}/m/${name}/configuration.nix" ];
|
inherit system;
|
||||||
};
|
overlays = [ self.overlays.default ];
|
||||||
# For now we only support x86
|
};
|
||||||
system = "x86_64-linux";
|
in
|
||||||
pkgs = import nixpkgs {
|
{
|
||||||
inherit system;
|
bscOverlay = import ./overlay.nix;
|
||||||
overlays = [ self.overlays.default ];
|
overlays.default = self.bscOverlay;
|
||||||
config.allowUnfree = true;
|
# full nixpkgs with our overlay applied
|
||||||
};
|
legacyPackages.${system} = pkgs;
|
||||||
in
|
|
||||||
{
|
hydraJobs = {
|
||||||
nixosConfigurations = {
|
inherit (self.legacyPackages.${system}.bsc-ci) tests pkgs cross;
|
||||||
hut = mkConf "hut";
|
};
|
||||||
tent = mkConf "tent";
|
|
||||||
owl1 = mkConf "owl1";
|
# propagate nixpkgs lib, so we can do bscpkgs.lib
|
||||||
owl2 = mkConf "owl2";
|
lib = nixpkgs.lib // {
|
||||||
eudy = mkConf "eudy";
|
bsc.maintainers = import ./maintainers/maintainer-list.nix;
|
||||||
koro = mkConf "koro";
|
};
|
||||||
bay = mkConf "bay";
|
|
||||||
lake2 = mkConf "lake2";
|
|
||||||
raccoon = mkConf "raccoon";
|
|
||||||
fox = mkConf "fox";
|
|
||||||
apex = mkConf "apex";
|
|
||||||
weasel = mkConf "weasel";
|
|
||||||
};
|
};
|
||||||
|
|
||||||
bscOverlay = import ./overlay.nix;
|
|
||||||
overlays.default = self.bscOverlay;
|
|
||||||
|
|
||||||
# full nixpkgs with our overlay applied
|
|
||||||
legacyPackages.${system} = pkgs;
|
|
||||||
|
|
||||||
hydraJobs = {
|
|
||||||
inherit (self.legacyPackages.${system}.bsc-ci) tests pkgs cross;
|
|
||||||
};
|
|
||||||
|
|
||||||
# propagate nixpkgs lib, so we can do bscpkgs.lib
|
|
||||||
lib = nixpkgs.lib // {
|
|
||||||
bsc.maintainers = import ./maintainers/maintainer-list.nix;
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
|
|||||||
37
keys.nix
37
keys.nix
@ -1,37 +0,0 @@
|
|||||||
# As agenix needs to parse the secrets from a standalone .nix file, we describe
|
|
||||||
# here all the public keys
|
|
||||||
rec {
|
|
||||||
hosts = {
|
|
||||||
hut = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICO7jIp6JRnRWTMDsTB/aiaICJCl4x8qmKMPSs4lCqP1 hut";
|
|
||||||
owl1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMqMEXO0ApVsBA6yjmb0xP2kWyoPDIWxBB0Q3+QbHVhv owl1";
|
|
||||||
owl2 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHurEYpQzNHqWYF6B9Pd7W8UPgF3BxEg0BvSbsA7BAdK owl2";
|
|
||||||
eudy = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL+WYPRRvZupqLAG0USKmd/juEPmisyyJaP8hAgYwXsG eudy";
|
|
||||||
koro = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIImiTFDbxyUYPumvm8C4mEnHfuvtBY1H8undtd6oDd67 koro";
|
|
||||||
bay = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICvGBzpRQKuQYHdlUQeAk6jmdbkrhmdLwTBqf3el7IgU bay";
|
|
||||||
lake2 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINo66//S1yatpQHE/BuYD/Gfq64TY7ZN5XOGXmNchiO0 lake2";
|
|
||||||
fox = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDwItIk5uOJcQEVPoy/CVGRzfmE1ojrdDcI06FrU4NFT fox";
|
|
||||||
tent = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFAtTpHtdYoelbknD/IcfBlThwLKJv/dSmylOgpg3FRM tent";
|
|
||||||
apex = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBvUFjSfoxXnKwXhEFXx5ckRKJ0oewJ82mRitSMNMKjh apex";
|
|
||||||
weasel = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFLJrQ8BF6KcweQV8pLkSbFT+tbDxSG9qxrdQE65zJZp weasel";
|
|
||||||
raccoon = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGNQttFvL0dNEyy7klIhLoK4xXOeM2/K9R7lPMTG3qvK raccoon";
|
|
||||||
};
|
|
||||||
|
|
||||||
hostGroup = with hosts; rec {
|
|
||||||
compute = [ owl1 owl2 fox raccoon ];
|
|
||||||
playground = [ eudy koro weasel ];
|
|
||||||
storage = [ bay lake2 ];
|
|
||||||
monitor = [ hut ];
|
|
||||||
login = [ apex ];
|
|
||||||
|
|
||||||
system = storage ++ monitor ++ login;
|
|
||||||
safe = system ++ compute;
|
|
||||||
all = safe ++ playground;
|
|
||||||
};
|
|
||||||
|
|
||||||
admins = {
|
|
||||||
"rarias@hut" = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE1oZTPtlEXdGt0Ak+upeCIiBdaDQtcmuWoTUCVuSVIR rarias@hut";
|
|
||||||
"rarias@tent" = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIwlWSBTZi74WTz5xn6gBvTmCoVltmtIAeM3RMmkh4QZ rarias@tent";
|
|
||||||
"rarias@fox" = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDSbw3REAKECV7E2c/e2XJITudJQWq2qDSe2N1JHqHZd rarias@fox";
|
|
||||||
root = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIII/1TNArcwA6D47mgW4TArwlxQRpwmIGiZDysah40Gb root@hut";
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,69 +0,0 @@
|
|||||||
{ lib, config, pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../common/xeon.nix
|
|
||||||
../common/ssf/hosts.nix
|
|
||||||
../module/ceph.nix
|
|
||||||
../module/hut-substituter.nix
|
|
||||||
../module/slurm-server.nix
|
|
||||||
./nfs.nix
|
|
||||||
./wireguard.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
# Don't install grub MBR for now
|
|
||||||
boot.loader.grub.device = "nodev";
|
|
||||||
|
|
||||||
boot.initrd.kernelModules = [
|
|
||||||
"megaraid_sas" # For HW RAID
|
|
||||||
];
|
|
||||||
|
|
||||||
environment.systemPackages = with pkgs; [
|
|
||||||
storcli # To manage HW RAID
|
|
||||||
];
|
|
||||||
|
|
||||||
fileSystems."/home" = {
|
|
||||||
device = "/dev/disk/by-label/home";
|
|
||||||
fsType = "ext4";
|
|
||||||
};
|
|
||||||
|
|
||||||
# No swap, there is plenty of RAM
|
|
||||||
swapDevices = lib.mkForce [];
|
|
||||||
|
|
||||||
networking = {
|
|
||||||
hostName = "apex";
|
|
||||||
defaultGateway = "84.88.53.233";
|
|
||||||
nameservers = [ "8.8.8.8" ];
|
|
||||||
|
|
||||||
# Public facing interface
|
|
||||||
interfaces.eno1.ipv4.addresses = [ {
|
|
||||||
address = "84.88.53.236";
|
|
||||||
prefixLength = 29;
|
|
||||||
} ];
|
|
||||||
|
|
||||||
# Internal LAN to our Ethernet switch
|
|
||||||
interfaces.eno2.ipv4.addresses = [ {
|
|
||||||
address = "10.0.40.30";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
|
|
||||||
# Infiniband over Omnipath switch (disconnected for now)
|
|
||||||
# interfaces.ibp5s0 = {};
|
|
||||||
|
|
||||||
nat = {
|
|
||||||
enable = true;
|
|
||||||
internalInterfaces = [ "eno2" ];
|
|
||||||
externalInterface = "eno1";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
networking.firewall = {
|
|
||||||
extraCommands = ''
|
|
||||||
# Blackhole BSC vulnerability scanner (OpenVAS) as it is spamming our
|
|
||||||
# logs. Insert as first position so we also protect SSH.
|
|
||||||
iptables -I nixos-fw 1 -p tcp -s 192.168.8.16 -j nixos-fw-refuse
|
|
||||||
# Same with opsmonweb01.bsc.es which seems to be trying to access via SSH
|
|
||||||
iptables -I nixos-fw 2 -p tcp -s 84.88.52.176 -j nixos-fw-refuse
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,48 +0,0 @@
|
|||||||
{ ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
services.nfs.server = {
|
|
||||||
enable = true;
|
|
||||||
lockdPort = 4001;
|
|
||||||
mountdPort = 4002;
|
|
||||||
statdPort = 4000;
|
|
||||||
exports = ''
|
|
||||||
/home 10.0.40.0/24(rw,async,no_subtree_check,no_root_squash)
|
|
||||||
/home 10.106.0.0/24(rw,async,no_subtree_check,no_root_squash)
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
networking.firewall = {
|
|
||||||
# Check with `rpcinfo -p`
|
|
||||||
extraCommands = ''
|
|
||||||
# Accept NFS traffic from compute nodes but not from the outside
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.0/24 --dport 111 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.0/24 --dport 2049 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.0/24 --dport 4000 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.0/24 --dport 4001 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.0/24 --dport 4002 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.0/24 --dport 20048 -j nixos-fw-accept
|
|
||||||
# Same but UDP
|
|
||||||
iptables -A nixos-fw -p udp -s 10.0.40.0/24 --dport 111 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p udp -s 10.0.40.0/24 --dport 2049 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p udp -s 10.0.40.0/24 --dport 4000 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p udp -s 10.0.40.0/24 --dport 4001 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p udp -s 10.0.40.0/24 --dport 4002 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p udp -s 10.0.40.0/24 --dport 20048 -j nixos-fw-accept
|
|
||||||
|
|
||||||
# Accept NFS traffic from wg0
|
|
||||||
iptables -A nixos-fw -p tcp -i wg0 -s 10.106.0.0/24 --dport 111 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p tcp -i wg0 -s 10.106.0.0/24 --dport 2049 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p tcp -i wg0 -s 10.106.0.0/24 --dport 4000 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p tcp -i wg0 -s 10.106.0.0/24 --dport 4001 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p tcp -i wg0 -s 10.106.0.0/24 --dport 4002 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p tcp -i wg0 -s 10.106.0.0/24 --dport 20048 -j nixos-fw-accept
|
|
||||||
# Same but UDP
|
|
||||||
iptables -A nixos-fw -p udp -i wg0 -s 10.106.0.0/24 --dport 111 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p udp -i wg0 -s 10.106.0.0/24 --dport 2049 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p udp -i wg0 -s 10.106.0.0/24 --dport 4000 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p udp -i wg0 -s 10.106.0.0/24 --dport 4001 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p udp -i wg0 -s 10.106.0.0/24 --dport 4002 -j nixos-fw-accept
|
|
||||||
iptables -A nixos-fw -p udp -i wg0 -s 10.106.0.0/24 --dport 20048 -j nixos-fw-accept
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,42 +0,0 @@
|
|||||||
{ config, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
networking.firewall = {
|
|
||||||
allowedUDPPorts = [ 666 ];
|
|
||||||
};
|
|
||||||
|
|
||||||
age.secrets.wgApex.file = ../../secrets/wg-apex.age;
|
|
||||||
|
|
||||||
# Enable WireGuard
|
|
||||||
networking.wireguard.enable = true;
|
|
||||||
networking.wireguard.interfaces = {
|
|
||||||
# "wg0" is the network interface name. You can name the interface arbitrarily.
|
|
||||||
wg0 = {
|
|
||||||
ips = [ "10.106.0.30/24" ];
|
|
||||||
listenPort = 666;
|
|
||||||
privateKeyFile = config.age.secrets.wgApex.path;
|
|
||||||
# Public key: VwhcN8vSOzdJEotQTpmPHBC52x3Hbv1lkFIyKubrnUA=
|
|
||||||
peers = [
|
|
||||||
{
|
|
||||||
name = "fox";
|
|
||||||
publicKey = "VfMPBQLQTKeyXJSwv8wBhc6OV0j2qAxUpX3kLHunK2Y=";
|
|
||||||
allowedIPs = [ "10.106.0.1/32" ];
|
|
||||||
endpoint = "fox.ac.upc.edu:666";
|
|
||||||
# Send keepalives every 25 seconds. Important to keep NAT tables alive.
|
|
||||||
persistentKeepalive = 25;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
name = "raccoon";
|
|
||||||
publicKey = "QUfnGXSMEgu2bviglsaSdCjidB51oEDBFpnSFcKGfDI=";
|
|
||||||
allowedIPs = [ "10.106.0.236/32" "192.168.0.0/16" "10.0.44.0/24" ];
|
|
||||||
}
|
|
||||||
];
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
networking.hosts = {
|
|
||||||
"10.106.0.1" = [ "fox" ];
|
|
||||||
"10.106.0.236" = [ "raccoon" ];
|
|
||||||
"10.0.44.4" = [ "tent" ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,108 +0,0 @@
|
|||||||
{ config, pkgs, lib, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../common/ssf.nix
|
|
||||||
../module/hut-substituter.nix
|
|
||||||
../module/monitoring.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
# Select the this using the ID to avoid mismatches
|
|
||||||
boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d53562d";
|
|
||||||
|
|
||||||
boot.kernel.sysctl = {
|
|
||||||
"kernel.yama.ptrace_scope" = lib.mkForce "1";
|
|
||||||
};
|
|
||||||
|
|
||||||
environment.systemPackages = with pkgs; [
|
|
||||||
ceph
|
|
||||||
];
|
|
||||||
|
|
||||||
networking = {
|
|
||||||
hostName = "bay";
|
|
||||||
interfaces.eno1.ipv4.addresses = [ {
|
|
||||||
address = "10.0.40.40";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
interfaces.ibp5s0.ipv4.addresses = [ {
|
|
||||||
address = "10.0.42.40";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
firewall = {
|
|
||||||
extraCommands = ''
|
|
||||||
# Accept all incoming TCP traffic from lake2
|
|
||||||
iptables -A nixos-fw -p tcp -s lake2 -j nixos-fw-accept
|
|
||||||
# Accept monitoring requests from hut
|
|
||||||
iptables -A nixos-fw -p tcp -s hut -m multiport --dport 9283,9002 -j nixos-fw-accept
|
|
||||||
# Accept all Ceph traffic from the local network
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.0/24 -m multiport --dport 3300,6789,6800:7568 -j nixos-fw-accept
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
services.ceph = {
|
|
||||||
enable = true;
|
|
||||||
global = {
|
|
||||||
fsid = "9c8d06e0-485f-4aaf-b16b-06d6daf1232b";
|
|
||||||
monHost = "10.0.40.40";
|
|
||||||
monInitialMembers = "bay";
|
|
||||||
clusterNetwork = "10.0.40.40/24"; # Use Ethernet only
|
|
||||||
};
|
|
||||||
extraConfig = {
|
|
||||||
# Only log to stderr so it appears in the journal
|
|
||||||
"log_file" = "/dev/null";
|
|
||||||
"mon_cluster_log_file" = "/dev/null";
|
|
||||||
"log_to_stderr" = "true";
|
|
||||||
"err_to_stderr" = "true";
|
|
||||||
"log_to_file" = "false";
|
|
||||||
};
|
|
||||||
mds = {
|
|
||||||
enable = true;
|
|
||||||
daemons = [ "mds0" "mds1" ];
|
|
||||||
extraConfig = {
|
|
||||||
"host" = "bay";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
mgr = {
|
|
||||||
enable = true;
|
|
||||||
daemons = [ "bay" ];
|
|
||||||
};
|
|
||||||
mon = {
|
|
||||||
enable = true;
|
|
||||||
daemons = [ "bay" ];
|
|
||||||
};
|
|
||||||
osd = {
|
|
||||||
enable = true;
|
|
||||||
# One daemon per NVME disk
|
|
||||||
daemons = [ "0" "1" "2" "3" ];
|
|
||||||
extraConfig = {
|
|
||||||
"osd crush chooseleaf type" = "0";
|
|
||||||
"osd journal size" = "10000";
|
|
||||||
"osd pool default min size" = "2";
|
|
||||||
"osd pool default pg num" = "200";
|
|
||||||
"osd pool default pgp num" = "200";
|
|
||||||
"osd pool default size" = "3";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# Missing service for volumes, see:
|
|
||||||
# https://www.reddit.com/r/ceph/comments/14otjyo/comment/jrd69vt/
|
|
||||||
systemd.services.ceph-volume = {
|
|
||||||
enable = true;
|
|
||||||
description = "Ceph Volume activation";
|
|
||||||
unitConfig = {
|
|
||||||
Type = "oneshot";
|
|
||||||
After = "local-fs.target";
|
|
||||||
Wants = "local-fs.target";
|
|
||||||
};
|
|
||||||
path = [ pkgs.ceph pkgs.util-linux pkgs.lvm2 pkgs.cryptsetup ];
|
|
||||||
serviceConfig = {
|
|
||||||
KillMode = "none";
|
|
||||||
Environment = "CEPH_VOLUME_TIMEOUT=10000";
|
|
||||||
ExecStart = "/bin/sh -c 'timeout $CEPH_VOLUME_TIMEOUT ${pkgs.ceph}/bin/ceph-volume lvm activate --all --no-systemd'";
|
|
||||||
TimeoutSec = "0";
|
|
||||||
};
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,21 +0,0 @@
|
|||||||
{
|
|
||||||
# All machines should include this profile.
|
|
||||||
# Includes the basic configuration for an Intel server.
|
|
||||||
imports = [
|
|
||||||
./base/agenix.nix
|
|
||||||
./base/always-power-on.nix
|
|
||||||
./base/august-shutdown.nix
|
|
||||||
./base/boot.nix
|
|
||||||
./base/env.nix
|
|
||||||
./base/fs.nix
|
|
||||||
./base/hw.nix
|
|
||||||
./base/net.nix
|
|
||||||
./base/nix.nix
|
|
||||||
./base/ntp.nix
|
|
||||||
./base/rev.nix
|
|
||||||
./base/ssh.nix
|
|
||||||
./base/users.nix
|
|
||||||
./base/watchdog.nix
|
|
||||||
./base/zsh.nix
|
|
||||||
];
|
|
||||||
}
|
|
||||||
@ -1,9 +0,0 @@
|
|||||||
{ agenix, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [ agenix.nixosModules.default ];
|
|
||||||
|
|
||||||
environment.systemPackages = [
|
|
||||||
agenix.packages.x86_64-linux.default
|
|
||||||
];
|
|
||||||
}
|
|
||||||
@ -1,8 +0,0 @@
|
|||||||
{
|
|
||||||
imports = [
|
|
||||||
../../module/power-policy.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
# Turn on as soon as we have power
|
|
||||||
power.policy = "always-on";
|
|
||||||
}
|
|
||||||
@ -1,14 +0,0 @@
|
|||||||
{
|
|
||||||
# Shutdown all machines on August 3rd at 22:00, so we can protect the
|
|
||||||
# hardware from spurious electrical peaks on the yearly electrical cut for
|
|
||||||
# manteinance that starts on August 4th.
|
|
||||||
systemd.timers.august-shutdown = {
|
|
||||||
description = "Shutdown on August 3rd for maintenance";
|
|
||||||
wantedBy = [ "timers.target" ];
|
|
||||||
timerConfig = {
|
|
||||||
OnCalendar = "*-08-03 22:00:00";
|
|
||||||
RandomizedDelaySec = "10min";
|
|
||||||
Unit = "systemd-poweroff.service";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,37 +0,0 @@
|
|||||||
{ lib, pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
# Use the GRUB 2 boot loader.
|
|
||||||
boot.loader.grub.enable = true;
|
|
||||||
|
|
||||||
# Enable GRUB2 serial console
|
|
||||||
boot.loader.grub.extraConfig = ''
|
|
||||||
serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
|
|
||||||
terminal_input --append serial
|
|
||||||
terminal_output --append serial
|
|
||||||
'';
|
|
||||||
|
|
||||||
boot.kernel.sysctl = {
|
|
||||||
"kernel.perf_event_paranoid" = lib.mkDefault "-1";
|
|
||||||
|
|
||||||
# Allow ptracing (i.e. attach with GDB) any process of the same user, see:
|
|
||||||
# https://www.kernel.org/doc/Documentation/security/Yama.txt
|
|
||||||
"kernel.yama.ptrace_scope" = "0";
|
|
||||||
};
|
|
||||||
|
|
||||||
boot.kernelPackages = pkgs.linuxPackages_latest;
|
|
||||||
|
|
||||||
#boot.kernelPatches = lib.singleton {
|
|
||||||
# name = "osnoise-tracer";
|
|
||||||
# patch = null;
|
|
||||||
# extraStructuredConfig = with lib.kernel; {
|
|
||||||
# OSNOISE_TRACER = yes;
|
|
||||||
# HWLAT_TRACER = yes;
|
|
||||||
# };
|
|
||||||
#};
|
|
||||||
|
|
||||||
boot.initrd.availableKernelModules = [ "ahci" "xhci_pci" "ehci_pci" "nvme" "usbhid" "sd_mod" ];
|
|
||||||
boot.initrd.kernelModules = [ ];
|
|
||||||
boot.kernelModules = [ "kvm-intel" ];
|
|
||||||
boot.extraModulePackages = [ ];
|
|
||||||
}
|
|
||||||
@ -1,37 +0,0 @@
|
|||||||
{ pkgs, config, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
environment.systemPackages = with pkgs; [
|
|
||||||
vim wget git htop tmux pciutils tcpdump ripgrep nix-index nixos-option
|
|
||||||
nix-diff ipmitool freeipmi ethtool lm_sensors cmake gnumake file tree
|
|
||||||
ncdu config.boot.kernelPackages.perf ldns pv
|
|
||||||
# From bsckgs overlay
|
|
||||||
osumb
|
|
||||||
];
|
|
||||||
|
|
||||||
programs.direnv.enable = true;
|
|
||||||
|
|
||||||
# Increase limits
|
|
||||||
security.pam.loginLimits = [
|
|
||||||
{
|
|
||||||
domain = "*";
|
|
||||||
type = "-";
|
|
||||||
item = "memlock";
|
|
||||||
value = "1048576"; # 1 GiB of mem locked
|
|
||||||
}
|
|
||||||
];
|
|
||||||
|
|
||||||
environment.enableAllTerminfo = true;
|
|
||||||
|
|
||||||
environment.variables = {
|
|
||||||
EDITOR = "vim";
|
|
||||||
VISUAL = "vim";
|
|
||||||
};
|
|
||||||
|
|
||||||
programs.bash.promptInit = ''
|
|
||||||
PS1="\h\\$ "
|
|
||||||
'';
|
|
||||||
|
|
||||||
time.timeZone = "Europe/Madrid";
|
|
||||||
i18n.defaultLocale = "en_DK.UTF-8";
|
|
||||||
}
|
|
||||||
@ -1,24 +0,0 @@
|
|||||||
{ ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
fileSystems."/" =
|
|
||||||
{ device = "/dev/disk/by-label/nixos";
|
|
||||||
fsType = "ext4";
|
|
||||||
};
|
|
||||||
|
|
||||||
# Trim unused blocks weekly
|
|
||||||
services.fstrim.enable = true;
|
|
||||||
|
|
||||||
swapDevices =
|
|
||||||
[ { device = "/dev/disk/by-label/swap"; }
|
|
||||||
];
|
|
||||||
|
|
||||||
# Tracing
|
|
||||||
fileSystems."/sys/kernel/tracing" = {
|
|
||||||
device = "none";
|
|
||||||
fsType = "tracefs";
|
|
||||||
};
|
|
||||||
|
|
||||||
# Mount a tmpfs into /tmp
|
|
||||||
boot.tmp.useTmpfs = true;
|
|
||||||
}
|
|
||||||
@ -1,14 +0,0 @@
|
|||||||
# Do not modify this file! It was generated by ‘nixos-generate-config’
|
|
||||||
# and may be overwritten by future invocations. Please make changes
|
|
||||||
# to /etc/nixos/configuration.nix instead.
|
|
||||||
{ config, lib, pkgs, modulesPath, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports =
|
|
||||||
[ (modulesPath + "/installer/scan/not-detected.nix")
|
|
||||||
];
|
|
||||||
|
|
||||||
nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
|
|
||||||
powerManagement.cpuFreqGovernor = lib.mkDefault "powersave";
|
|
||||||
hardware.cpu.intel.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
|
|
||||||
}
|
|
||||||
@ -1,23 +0,0 @@
|
|||||||
{ pkgs, lib, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
networking = {
|
|
||||||
enableIPv6 = false;
|
|
||||||
useDHCP = false;
|
|
||||||
|
|
||||||
firewall = {
|
|
||||||
enable = true;
|
|
||||||
allowedTCPPorts = [ 22 ];
|
|
||||||
};
|
|
||||||
|
|
||||||
# Make sure we use iptables
|
|
||||||
nftables.enable = lib.mkForce false;
|
|
||||||
|
|
||||||
hosts = {
|
|
||||||
"84.88.53.236" = [ "ssfhead.bsc.es" "ssfhead" ];
|
|
||||||
"84.88.51.142" = [ "raccoon-ipmi" ];
|
|
||||||
"192.168.11.12" = [ "bscpm04.bsc.es" ];
|
|
||||||
"192.168.11.15" = [ "gitlab-internal.bsc.es" ];
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,59 +0,0 @@
|
|||||||
{ pkgs, nixpkgs, theFlake, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
nixpkgs.overlays = [
|
|
||||||
(import ../../../overlay.nix)
|
|
||||||
];
|
|
||||||
|
|
||||||
nixpkgs.config.allowUnfree = true;
|
|
||||||
|
|
||||||
nix = {
|
|
||||||
nixPath = [
|
|
||||||
"nixpkgs=${nixpkgs}"
|
|
||||||
"jungle=${theFlake.outPath}"
|
|
||||||
];
|
|
||||||
|
|
||||||
registry = {
|
|
||||||
nixpkgs.flake = nixpkgs;
|
|
||||||
jungle.flake = theFlake;
|
|
||||||
};
|
|
||||||
|
|
||||||
settings = {
|
|
||||||
experimental-features = [ "nix-command" "flakes" ];
|
|
||||||
sandbox = "relaxed";
|
|
||||||
trusted-users = [ "@wheel" ];
|
|
||||||
flake-registry = pkgs.writeText "global-registry.json"
|
|
||||||
''{"flakes":[],"version":2}'';
|
|
||||||
keep-outputs = true;
|
|
||||||
};
|
|
||||||
|
|
||||||
gc = {
|
|
||||||
automatic = true;
|
|
||||||
dates = "weekly";
|
|
||||||
options = "--delete-older-than 30d";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# The nix-gc.service can begin its execution *before* /home is mounted,
|
|
||||||
# causing it to remove all gcroots considering them as stale, as it cannot
|
|
||||||
# access the symlink. To prevent this problem, we force the service to wait
|
|
||||||
# until /home is mounted as well as other remote FS like /ceph.
|
|
||||||
systemd.services.nix-gc = {
|
|
||||||
# Start remote-fs.target if not already being started and fail if it fails
|
|
||||||
# to start. It will also be stopped if the remote-fs.target fails after
|
|
||||||
# starting successfully.
|
|
||||||
bindsTo = [ "remote-fs.target" ];
|
|
||||||
# Wait until remote-fs.target fully starts before starting this one.
|
|
||||||
after = [ "remote-fs.target"];
|
|
||||||
# Ensure we can access a remote path inside /home
|
|
||||||
unitConfig.ConditionPathExists = "/home/Computational";
|
|
||||||
};
|
|
||||||
|
|
||||||
# This value determines the NixOS release from which the default
|
|
||||||
# settings for stateful data, like file locations and database versions
|
|
||||||
# on your system were taken. It‘s perfectly fine and recommended to leave
|
|
||||||
# this value at the release version of the first install of this system.
|
|
||||||
# Before changing this value read the documentation for this option
|
|
||||||
# (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
|
|
||||||
system.stateVersion = "22.11"; # Did you read the comment?
|
|
||||||
}
|
|
||||||
@ -1,9 +0,0 @@
|
|||||||
{ pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
services.ntp.enable = true;
|
|
||||||
|
|
||||||
# Use the NTP server at BSC, as we don't have direct access
|
|
||||||
# to the outside world
|
|
||||||
networking.timeServers = [ "84.88.52.36" ];
|
|
||||||
}
|
|
||||||
@ -1,21 +0,0 @@
|
|||||||
{ theFlake, ... }:
|
|
||||||
|
|
||||||
let
|
|
||||||
# Prevent building a configuration without revision
|
|
||||||
rev = if theFlake ? rev then theFlake.rev
|
|
||||||
else throw ("Refusing to build from a dirty Git tree!");
|
|
||||||
in {
|
|
||||||
# Save the commit of the config in /etc/configrev
|
|
||||||
environment.etc.configrev.text = rev + "\n";
|
|
||||||
|
|
||||||
# Keep a log with the config over time
|
|
||||||
system.activationScripts.configRevLog.text = ''
|
|
||||||
BOOTED=$(cat /run/booted-system/etc/configrev 2>/dev/null || echo unknown)
|
|
||||||
CURRENT=$(cat /run/current-system/etc/configrev 2>/dev/null || echo unknown)
|
|
||||||
NEXT=${rev}
|
|
||||||
DATENOW=$(date --iso-8601=seconds)
|
|
||||||
echo "$DATENOW booted=$BOOTED current=$CURRENT next=$NEXT" >> /var/configrev.log
|
|
||||||
'';
|
|
||||||
|
|
||||||
system.configurationRevision = rev;
|
|
||||||
}
|
|
||||||
@ -1,18 +0,0 @@
|
|||||||
{ lib, ... }:
|
|
||||||
|
|
||||||
let
|
|
||||||
keys = import ../../../keys.nix;
|
|
||||||
hostsKeys = lib.mapAttrs (name: value: { publicKey = value; }) keys.hosts;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
# Enable the OpenSSH daemon.
|
|
||||||
services.openssh.enable = true;
|
|
||||||
|
|
||||||
programs.ssh.knownHosts = hostsKeys // {
|
|
||||||
"gitlab-internal.bsc.es".publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF9arsAOSRB06hdy71oTvJHG2Mg8zfebADxpvc37lZo3";
|
|
||||||
"bscpm03.bsc.es".publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM2NuSUPsEhqz1j5b4Gqd+MWFnRqyqY57+xMvBUqHYUS";
|
|
||||||
"bscpm04.bsc.es".publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPx4mC0etyyjYUT2Ztc/bs4ZXSbVMrogs1ZTP924PDgT";
|
|
||||||
"glogin1.bsc.es".publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFsHsZGCrzpd4QDVn5xoDOtrNBkb0ylxKGlyBt6l9qCz";
|
|
||||||
"glogin2.bsc.es".publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFsHsZGCrzpd4QDVn5xoDOtrNBkb0ylxKGlyBt6l9qCz";
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,190 +0,0 @@
|
|||||||
{ pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../../module/jungle-users.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
users = {
|
|
||||||
mutableUsers = false;
|
|
||||||
users = {
|
|
||||||
# Generate hashedPassword with `mkpasswd -m sha-512`
|
|
||||||
|
|
||||||
root.openssh.authorizedKeys.keys = [
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKBOf4r4lzQfyO0bx5BaREePREw8Zw5+xYgZhXwOZoBO ram@hop"
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINa0tvnNgwkc5xOwd6xTtaIdFi5jv0j2FrE7jl5MTLoE ram@mio"
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF3zeB5KSimMBAjvzsp1GCkepVaquVZGPYwRIzyzaCba aleix@bsc"
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIII/1TNArcwA6D47mgW4TArwlxQRpwmIGiZDysah40Gb root@hut"
|
|
||||||
];
|
|
||||||
|
|
||||||
rarias = {
|
|
||||||
uid = 1880;
|
|
||||||
isNormalUser = true;
|
|
||||||
linger = true;
|
|
||||||
home = "/home/Computational/rarias";
|
|
||||||
description = "Rodrigo Arias";
|
|
||||||
group = "Computational";
|
|
||||||
extraGroups = [ "wheel" ];
|
|
||||||
hashedPassword = "$6$u06tkCy13enReBsb$xiI.twRvvTfH4jdS3s68NZ7U9PSbGKs5.LXU/UgoawSwNWhZo2hRAjNL5qG0/lAckzcho2LjD0r3NfVPvthY6/";
|
|
||||||
openssh.authorizedKeys.keys = [
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKBOf4r4lzQfyO0bx5BaREePREw8Zw5+xYgZhXwOZoBO ram@hop"
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINa0tvnNgwkc5xOwd6xTtaIdFi5jv0j2FrE7jl5MTLoE ram@mio"
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGYcXIxe0poOEGLpk8NjiRozls7fMRX0N3j3Ar94U+Gl rarias@hal"
|
|
||||||
];
|
|
||||||
shell = pkgs.zsh;
|
|
||||||
};
|
|
||||||
|
|
||||||
arocanon = {
|
|
||||||
uid = 1042;
|
|
||||||
isNormalUser = true;
|
|
||||||
home = "/home/Computational/arocanon";
|
|
||||||
description = "Aleix Roca";
|
|
||||||
group = "Computational";
|
|
||||||
extraGroups = [ "wheel" "tracing" ];
|
|
||||||
hashedPassword = "$6$hliZiW4tULC/tH7p$pqZarwJkNZ7vS0G5llWQKx08UFG9DxDYgad7jplMD8WkZh5k58i4dfPoWtnEShfjTO6JHiIin05ny5lmSXzGM/";
|
|
||||||
openssh.authorizedKeys.keys = [
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF3zeB5KSimMBAjvzsp1GCkepVaquVZGPYwRIzyzaCba aleix@bsc"
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGdphWxLAEekicZ/WBrvP7phMyxKSSuLAZBovNX+hZXQ aleix@kerneland"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
jungleUsers = {
|
|
||||||
rpenacob = {
|
|
||||||
uid = 2761;
|
|
||||||
isNormalUser = true;
|
|
||||||
home = "/home/Computational/rpenacob";
|
|
||||||
description = "Raúl Peñacoba";
|
|
||||||
group = "Computational";
|
|
||||||
hosts = [ "apex" "owl1" "owl2" "hut" "tent" "fox" ];
|
|
||||||
hashedPassword = "$6$TZm3bDIFyPrMhj1E$uEDXoYYd1z2Wd5mMPfh3DZAjP7ztVjJ4ezIcn82C0ImqafPA.AnTmcVftHEzLB3tbe2O4SxDyPSDEQgJ4GOtj/";
|
|
||||||
openssh.authorizedKeys.keys = [
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFYfXg37mauGeurqsLpedgA2XQ9d4Nm0ZGo/hI1f7wwH rpenacob@bsc"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
|
|
||||||
anavarro = {
|
|
||||||
uid = 1037;
|
|
||||||
isNormalUser = true;
|
|
||||||
home = "/home/Computational/anavarro";
|
|
||||||
description = "Antoni Navarro";
|
|
||||||
group = "Computational";
|
|
||||||
hosts = [ "apex" "hut" "tent" "raccoon" "fox" "weasel" ];
|
|
||||||
hashedPassword = "$6$EgturvVYXlKgP43g$gTN78LLHIhaF8hsrCXD.O6mKnZSASWSJmCyndTX8QBWT6wTlUhcWVAKz65lFJPXjlJA4u7G1ydYQ0GG6Wk07b1";
|
|
||||||
openssh.authorizedKeys.keys = [
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMsbM21uepnJwPrRe6jYFz8zrZ6AYMtSEvvt4c9spmFP toni@delltoni"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
|
|
||||||
abonerib = {
|
|
||||||
uid = 4541;
|
|
||||||
isNormalUser = true;
|
|
||||||
home = "/home/Computational/abonerib";
|
|
||||||
description = "Aleix Boné";
|
|
||||||
group = "Computational";
|
|
||||||
hosts = [ "apex" "owl1" "owl2" "hut" "tent" "raccoon" "fox" "weasel" ];
|
|
||||||
hashedPassword = "$6$V1EQWJr474whv7XJ$OfJ0wueM2l.dgiJiiah0Tip9ITcJ7S7qDvtSycsiQ43QBFyP4lU0e0HaXWps85nqB4TypttYR4hNLoz3bz662/";
|
|
||||||
openssh.authorizedKeys.keys = [
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIIFiqXqt88VuUfyANkZyLJNiuroIITaGlOOTMhVDKjf abonerib@bsc"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
|
|
||||||
vlopez = {
|
|
||||||
uid = 4334;
|
|
||||||
isNormalUser = true;
|
|
||||||
home = "/home/Computational/vlopez";
|
|
||||||
description = "Victor López";
|
|
||||||
group = "Computational";
|
|
||||||
hosts = [ "apex" "koro" ];
|
|
||||||
hashedPassword = "$6$0ZBkgIYE/renVqtt$1uWlJsb0FEezRVNoETTzZMx4X2SvWiOsKvi0ppWCRqI66S6TqMBXBdP4fcQyvRRBt0e4Z7opZIvvITBsEtO0f0";
|
|
||||||
openssh.authorizedKeys.keys = [
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGMwlUZRf9jfG666Qa5Sb+KtEhXqkiMlBV2su3x/dXHq victor@arch"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
|
|
||||||
dbautist = {
|
|
||||||
uid = 5649;
|
|
||||||
isNormalUser = true;
|
|
||||||
home = "/home/Computational/dbautist";
|
|
||||||
description = "Dylan Bautista Cases";
|
|
||||||
group = "Computational";
|
|
||||||
hosts = [ "apex" "hut" "tent" "raccoon" ];
|
|
||||||
hashedPassword = "$6$a2lpzMRVkG9nSgIm$12G6.ka0sFX1YimqJkBAjbvhRKZ.Hl090B27pdbnQOW0wzyxVWySWhyDDCILjQELky.HKYl9gqOeVXW49nW7q/";
|
|
||||||
openssh.authorizedKeys.keys = [
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAb+EQBoS98zrCwnGKkHKwMLdYABMTqv7q9E0+T0QmkS dbautist@bsc-848818791"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
|
|
||||||
dalvare1 = {
|
|
||||||
uid = 2758;
|
|
||||||
isNormalUser = true;
|
|
||||||
home = "/home/Computational/dalvare1";
|
|
||||||
description = "David Álvarez";
|
|
||||||
group = "Computational";
|
|
||||||
hosts = [ "apex" "hut" "tent" "fox" ];
|
|
||||||
hashedPassword = "$6$mpyIsV3mdq.rK8$FvfZdRH5OcEkUt5PnIUijWyUYZvB1SgeqxpJ2p91TTe.3eQIDTcLEQ5rxeg.e5IEXAZHHQ/aMsR5kPEujEghx0";
|
|
||||||
openssh.authorizedKeys.keys = [
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGEfy6F4rF80r4Cpo2H5xaWqhuUZzUsVsILSKGJzt5jF dalvare1@ssfhead"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
|
|
||||||
varcila = {
|
|
||||||
uid = 5650;
|
|
||||||
isNormalUser = true;
|
|
||||||
home = "/home/Computational/varcila";
|
|
||||||
description = "Vincent Arcila";
|
|
||||||
group = "Computational";
|
|
||||||
hosts = [ "apex" "hut" "tent" "fox" ];
|
|
||||||
hashedPassword = "$6$oB0Tcn99DcM4Ch$Vn1A0ulLTn/8B2oFPi9wWl/NOsJzaFAWjqekwcuC9sMC7cgxEVb.Nk5XSzQ2xzYcNe5MLtmzkVYnRS1CqP39Y0";
|
|
||||||
openssh.authorizedKeys.keys = [
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKGt0ESYxekBiHJQowmKpfdouw0hVm3N7tUMtAaeLejK vincent@varch"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
|
|
||||||
pmartin1 = {
|
|
||||||
# Arbitrary UID but large so it doesn't collide with other users on ssfhead.
|
|
||||||
uid = 9652;
|
|
||||||
isNormalUser = true;
|
|
||||||
home = "/home/Computational/pmartin1";
|
|
||||||
description = "Pedro J. Martinez-Ferrer";
|
|
||||||
group = "Computational";
|
|
||||||
hosts = [ "fox" ];
|
|
||||||
hashedPassword = "$6$nIgDMGnt4YIZl3G.$.JQ2jXLtDPRKsbsJfJAXdSvjDIzRrg7tNNjPkLPq3KJQhMjfDXRUvzagUHUU2TrE2hHM8/6uq8ex0UdxQ0ysl.";
|
|
||||||
openssh.authorizedKeys.keys = [
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIV5LEAII5rfe1hYqDYIIrhb1gOw7RcS1p2mhOTqG+zc pedro@pedro-ThinkPad-P14s-Gen-2a"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
|
|
||||||
csiringo = {
|
|
||||||
uid = 9653;
|
|
||||||
isNormalUser = true;
|
|
||||||
home = "/home/Computational/csiringo";
|
|
||||||
description = "Cesare Siringo";
|
|
||||||
group = "Computational";
|
|
||||||
hosts = [ ];
|
|
||||||
hashedPassword = "$6$0IsZlju8jFukLlAw$VKm0FUXbS.mVmPm3rcJeizTNU4IM5Nmmy21BvzFL.cQwvlGwFI1YWRQm6gsbd4nbg47mPDvYkr/ar0SlgF6GO1";
|
|
||||||
openssh.authorizedKeys.keys = [
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHA65zvvG50iuFEMf+guRwZB65jlGXfGLF4HO+THFaed csiringo@bsc.es"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
|
|
||||||
acinca = {
|
|
||||||
uid = 9654;
|
|
||||||
isNormalUser = true;
|
|
||||||
home = "/home/Computational/acinca";
|
|
||||||
description = "Arnau Cinca";
|
|
||||||
group = "Computational";
|
|
||||||
hosts = [ "apex" "hut" "fox" "owl1" "owl2" ];
|
|
||||||
hashedPassword = "$6$S6PUeRpdzYlidxzI$szyvWejQ4hEN76yBYhp1diVO5ew1FFg.cz4lKiXt2Idy4XdpifwrFTCIzLTs5dvYlR62m7ekA5MrhcVxR5F/q/";
|
|
||||||
openssh.authorizedKeys.keys = [
|
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFmMqKqPg4uocNOr3O41kLbZMOMJn3m2ZdN1JvTR96z3 bsccns@arnau-bsc"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
groups = {
|
|
||||||
Computational = { gid = 564; };
|
|
||||||
tracing = { };
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,9 +0,0 @@
|
|||||||
{ ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
# The boards have a BMC watchdog controlled by IPMI
|
|
||||||
boot.kernelModules = [ "ipmi_watchdog" ];
|
|
||||||
|
|
||||||
# Enable systemd watchdog with 30 s interval
|
|
||||||
systemd.watchdog.runtimeTime = "30s";
|
|
||||||
}
|
|
||||||
@ -1,91 +0,0 @@
|
|||||||
{ pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
environment.systemPackages = with pkgs; [
|
|
||||||
zsh-completions
|
|
||||||
nix-zsh-completions
|
|
||||||
];
|
|
||||||
|
|
||||||
programs.zsh = {
|
|
||||||
enable = true;
|
|
||||||
histSize = 1000000;
|
|
||||||
|
|
||||||
shellInit = ''
|
|
||||||
# Disable new user prompt
|
|
||||||
if [ ! -e ~/.zshrc ]; then
|
|
||||||
touch ~/.zshrc
|
|
||||||
fi
|
|
||||||
'';
|
|
||||||
|
|
||||||
promptInit = ''
|
|
||||||
# Note that to manually override this in ~/.zshrc you should run `prompt off`
|
|
||||||
# before setting your PS1 and etc. Otherwise this will likely to interact with
|
|
||||||
# your ~/.zshrc configuration in unexpected ways as the default prompt sets
|
|
||||||
# a lot of different prompt variables.
|
|
||||||
autoload -U promptinit && promptinit && prompt default && setopt prompt_sp
|
|
||||||
'';
|
|
||||||
|
|
||||||
# Taken from Ulli Kehrle config:
|
|
||||||
# https://git.hrnz.li/Ulli/nixos/src/commit/2e203b8d8d671f4e3ced0f1744a51d5c6ee19846/profiles/shell.nix#L199-L205
|
|
||||||
interactiveShellInit = ''
|
|
||||||
source "${pkgs.zsh-history-substring-search}/share/zsh-history-substring-search/zsh-history-substring-search.zsh"
|
|
||||||
|
|
||||||
# Save history immediately, but only load it when the shell starts
|
|
||||||
setopt inc_append_history
|
|
||||||
|
|
||||||
# dircolors doesn't support alacritty:
|
|
||||||
# https://lists.gnu.org/archive/html/bug-coreutils/2019-05/msg00029.html
|
|
||||||
export LS_COLORS='rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.swp=00;90:*.tmp=00;90:*.dpkg-dist=00;90:*.dpkg-old=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:';
|
|
||||||
|
|
||||||
# From Arch Linux and GRML
|
|
||||||
bindkey "^R" history-incremental-pattern-search-backward
|
|
||||||
bindkey "^S" history-incremental-pattern-search-forward
|
|
||||||
|
|
||||||
# Auto rehash for new binaries
|
|
||||||
zstyle ':completion:*' rehash true
|
|
||||||
# show a nice menu with the matches
|
|
||||||
zstyle ':completion:*' menu yes select
|
|
||||||
|
|
||||||
bindkey '^[OA' history-substring-search-up # Up
|
|
||||||
bindkey '^[[A' history-substring-search-up # Up
|
|
||||||
|
|
||||||
bindkey '^[OB' history-substring-search-down # Down
|
|
||||||
bindkey '^[[B' history-substring-search-down # Down
|
|
||||||
|
|
||||||
bindkey '\e[1~' beginning-of-line # Home
|
|
||||||
bindkey '\e[7~' beginning-of-line # Home
|
|
||||||
bindkey '\e[H' beginning-of-line # Home
|
|
||||||
bindkey '\eOH' beginning-of-line # Home
|
|
||||||
|
|
||||||
bindkey '\e[4~' end-of-line # End
|
|
||||||
bindkey '\e[8~' end-of-line # End
|
|
||||||
bindkey '\e[F ' end-of-line # End
|
|
||||||
bindkey '\eOF' end-of-line # End
|
|
||||||
|
|
||||||
bindkey '^?' backward-delete-char # Backspace
|
|
||||||
bindkey '\e[3~' delete-char # Del
|
|
||||||
# bindkey '\e[3;5~' delete-char # sometimes Del, sometimes C-Del
|
|
||||||
bindkey '\e[2~' overwrite-mode # Ins
|
|
||||||
|
|
||||||
bindkey '^H' backward-kill-word # C-Backspace
|
|
||||||
|
|
||||||
bindkey '5~' kill-word # C-Del
|
|
||||||
bindkey '^[[3;5~' kill-word # C-Del
|
|
||||||
bindkey '^[[3^' kill-word # C-Del
|
|
||||||
|
|
||||||
bindkey "^[[1;5H" backward-kill-line # C-Home
|
|
||||||
bindkey "^[[7^" backward-kill-line # C-Home
|
|
||||||
|
|
||||||
bindkey "^[[1;5F" kill-line # C-End
|
|
||||||
bindkey "^[[8^" kill-line # C-End
|
|
||||||
|
|
||||||
bindkey '^[[1;5C' forward-word # C-Right
|
|
||||||
bindkey '^[0c' forward-word # C-Right
|
|
||||||
bindkey '^[[5C' forward-word # C-Right
|
|
||||||
|
|
||||||
bindkey '^[[1;5D' backward-word # C-Left
|
|
||||||
bindkey '^[0d' backward-word # C-Left
|
|
||||||
bindkey '^[[5D' backward-word # C-Left
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,10 +0,0 @@
|
|||||||
{
|
|
||||||
# Provides the base system for a xeon node in the SSF rack.
|
|
||||||
imports = [
|
|
||||||
./xeon.nix
|
|
||||||
./ssf/fs.nix
|
|
||||||
./ssf/hosts.nix
|
|
||||||
./ssf/hosts-remote.nix
|
|
||||||
./ssf/net.nix
|
|
||||||
];
|
|
||||||
}
|
|
||||||
@ -1,8 +0,0 @@
|
|||||||
{
|
|
||||||
# Mount the home via NFS
|
|
||||||
fileSystems."/home" = {
|
|
||||||
device = "10.0.40.30:/home";
|
|
||||||
fsType = "nfs";
|
|
||||||
options = [ "nfsvers=3" "rsize=1024" "wsize=1024" "cto" "nofail" ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,9 +0,0 @@
|
|||||||
{ pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
networking.hosts = {
|
|
||||||
# Remote hosts visible from compute nodes
|
|
||||||
"10.106.0.236" = [ "raccoon" ];
|
|
||||||
"10.0.44.4" = [ "tent" ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,23 +0,0 @@
|
|||||||
{ pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
networking.hosts = {
|
|
||||||
# Login
|
|
||||||
"10.0.40.30" = [ "apex" ];
|
|
||||||
|
|
||||||
# Storage
|
|
||||||
"10.0.40.40" = [ "bay" ]; "10.0.42.40" = [ "bay-ib" ]; "10.0.40.141" = [ "bay-ipmi" ];
|
|
||||||
"10.0.40.41" = [ "oss01" ]; "10.0.42.41" = [ "oss01-ib0" ]; "10.0.40.142" = [ "oss01-ipmi" ];
|
|
||||||
"10.0.40.42" = [ "lake2" ]; "10.0.42.42" = [ "lake2-ib" ]; "10.0.40.143" = [ "lake2-ipmi" ];
|
|
||||||
|
|
||||||
# Xeon compute
|
|
||||||
"10.0.40.1" = [ "owl1" ]; "10.0.42.1" = [ "owl1-ib" ]; "10.0.40.101" = [ "owl1-ipmi" ];
|
|
||||||
"10.0.40.2" = [ "owl2" ]; "10.0.42.2" = [ "owl2-ib" ]; "10.0.40.102" = [ "owl2-ipmi" ];
|
|
||||||
"10.0.40.3" = [ "xeon03" ]; "10.0.42.3" = [ "xeon03-ib" ]; "10.0.40.103" = [ "xeon03-ipmi" ];
|
|
||||||
#"10.0.40.4" = [ "tent" ]; "10.0.42.4" = [ "tent-ib" ]; "10.0.40.104" = [ "tent-ipmi" ];
|
|
||||||
"10.0.40.5" = [ "koro" ]; "10.0.42.5" = [ "koro-ib" ]; "10.0.40.105" = [ "koro-ipmi" ];
|
|
||||||
"10.0.40.6" = [ "weasel" ]; "10.0.42.6" = [ "weasel-ib" ]; "10.0.40.106" = [ "weasel-ipmi" ];
|
|
||||||
"10.0.40.7" = [ "hut" ]; "10.0.42.7" = [ "hut-ib" ]; "10.0.40.107" = [ "hut-ipmi" ];
|
|
||||||
"10.0.40.8" = [ "eudy" ]; "10.0.42.8" = [ "eudy-ib" ]; "10.0.40.108" = [ "eudy-ipmi" ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,23 +0,0 @@
|
|||||||
{ pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
# Infiniband (IPoIB)
|
|
||||||
environment.systemPackages = [ pkgs.rdma-core ];
|
|
||||||
boot.kernelModules = [ "ib_umad" "ib_ipoib" ];
|
|
||||||
|
|
||||||
networking = {
|
|
||||||
defaultGateway = "10.0.40.30";
|
|
||||||
nameservers = ["8.8.8.8"];
|
|
||||||
|
|
||||||
firewall = {
|
|
||||||
extraCommands = ''
|
|
||||||
# Prevent ssfhead from contacting our slurmd daemon
|
|
||||||
iptables -A nixos-fw -p tcp -s ssfhead --dport 6817:6819 -j nixos-fw-refuse
|
|
||||||
# But accept traffic to slurm ports from any other node in the subnet
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.0/24 --dport 6817:6819 -j nixos-fw-accept
|
|
||||||
# We also need to open the srun port range
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.0/24 --dport 60000:61000 -j nixos-fw-accept
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,7 +0,0 @@
|
|||||||
{
|
|
||||||
# Provides the base system for a xeon node, not necessarily in the SSF rack.
|
|
||||||
imports = [
|
|
||||||
./base.nix
|
|
||||||
./xeon/console.nix
|
|
||||||
];
|
|
||||||
}
|
|
||||||
@ -1,14 +0,0 @@
|
|||||||
{
|
|
||||||
# Restart the serial console
|
|
||||||
systemd.services."serial-getty@ttyS0" = {
|
|
||||||
enable = true;
|
|
||||||
wantedBy = [ "getty.target" ];
|
|
||||||
serviceConfig.Restart = "always";
|
|
||||||
};
|
|
||||||
|
|
||||||
# Enable serial console
|
|
||||||
boot.kernelParams = [
|
|
||||||
"console=tty1"
|
|
||||||
"console=ttyS0,115200"
|
|
||||||
];
|
|
||||||
}
|
|
||||||
@ -1,38 +0,0 @@
|
|||||||
{ config, pkgs, lib, modulesPath, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../common/ssf.nix
|
|
||||||
#(modulesPath + "/installer/netboot/netboot-minimal.nix")
|
|
||||||
|
|
||||||
./kernel/kernel.nix
|
|
||||||
./cpufreq.nix
|
|
||||||
./fs.nix
|
|
||||||
./users.nix
|
|
||||||
../module/hut-substituter.nix
|
|
||||||
../module/debuginfod.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
# Select this using the ID to avoid mismatches
|
|
||||||
boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d53564b";
|
|
||||||
|
|
||||||
# disable automatic garbage collector
|
|
||||||
nix.gc.automatic = lib.mkForce false;
|
|
||||||
|
|
||||||
# members of the tracing group can use the lttng-provided kernel events
|
|
||||||
# without root permissions
|
|
||||||
users.groups.tracing.members = [ "arocanon" ];
|
|
||||||
|
|
||||||
# set up both ethernet and infiniband ips
|
|
||||||
networking = {
|
|
||||||
hostName = "eudy";
|
|
||||||
interfaces.eno1.ipv4.addresses = [ {
|
|
||||||
address = "10.0.40.8";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
interfaces.ibp5s0.ipv4.addresses = [ {
|
|
||||||
address = "10.0.42.8";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,40 +0,0 @@
|
|||||||
{ lib, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
# Disable frequency boost by default. Use the intel_pstate driver instead of
|
|
||||||
# acpi_cpufreq driver because the acpi_cpufreq driver does not read the
|
|
||||||
# complete range of P-States [1]. Use the intel_pstate passive mode [2] to
|
|
||||||
# disable HWP, which allows a core to "select P-states by itself". Also, this
|
|
||||||
# disables intel governors, which confusingly, have the same names as the
|
|
||||||
# generic ones but behave differently [3].
|
|
||||||
|
|
||||||
# Essentially, we use the generic governors, but use the intel driver to read
|
|
||||||
# the P-state list.
|
|
||||||
|
|
||||||
# [1] - https://www.kernel.org/doc/html/latest/admin-guide/pm/intel_pstate.html#intel-pstate-vs-acpi-cpufreq
|
|
||||||
# [2] - https://www.kernel.org/doc/html/latest/admin-guide/pm/intel_pstate.html#passive-mode
|
|
||||||
# [3] - https://www.kernel.org/doc/html/latest/admin-guide/pm/intel_pstate.html#active-mode
|
|
||||||
# https://www.kernel.org/doc/html/latest/admin-guide/pm/cpufreq.html
|
|
||||||
|
|
||||||
# set intel_pstate to passive mode
|
|
||||||
boot.kernelParams = [
|
|
||||||
"intel_pstate=passive"
|
|
||||||
];
|
|
||||||
# Disable frequency boost
|
|
||||||
system.activationScripts = {
|
|
||||||
disableFrequencyBoost.text = ''
|
|
||||||
echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
## disable intel_pstate
|
|
||||||
#boot.kernelParams = [
|
|
||||||
# "intel_pstate=disable"
|
|
||||||
#];
|
|
||||||
## Disable frequency boost
|
|
||||||
#system.activationScripts = {
|
|
||||||
# disableFrequencyBoost.text = ''
|
|
||||||
# echo 0 > /sys/devices/system/cpu/cpufreq/boost
|
|
||||||
# '';
|
|
||||||
#};
|
|
||||||
}
|
|
||||||
@ -1,13 +0,0 @@
|
|||||||
{ ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
fileSystems."/nix" = {
|
|
||||||
device = "/dev/disk/by-label/optane";
|
|
||||||
fsType = "ext4";
|
|
||||||
neededForBoot = true;
|
|
||||||
};
|
|
||||||
fileSystems."/mnt/data" = {
|
|
||||||
device = "/dev/disk/by-label/data";
|
|
||||||
fsType = "ext4";
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,70 +0,0 @@
|
|||||||
{ pkgs, lib, ... }:
|
|
||||||
|
|
||||||
let
|
|
||||||
#fcs-devel = pkgs.linuxPackages_custom {
|
|
||||||
# version = "6.2.8";
|
|
||||||
# src = /mnt/data/kernel/fcs/kernel/src;
|
|
||||||
# configfile = /mnt/data/kernel/fcs/kernel/configs/defconfig;
|
|
||||||
#};
|
|
||||||
|
|
||||||
#fcsv1 = fcs-kernel "bc11660676d3d68ce2459b9fb5d5e654e3f413be" false;
|
|
||||||
#fcsv2 = fcs-kernel "db0f2eca0cd57a58bf456d7d2c7d5d8fdb25dfb1" false;
|
|
||||||
#fcsv1-lockdep = fcs-kernel "bc11660676d3d68ce2459b9fb5d5e654e3f413be" true;
|
|
||||||
#fcsv2-lockdep = fcs-kernel "db0f2eca0cd57a58bf456d7d2c7d5d8fdb25dfb1" true;
|
|
||||||
#fcs-kernel = gitCommit: lockdep: pkgs.linuxPackages_custom {
|
|
||||||
# version = "6.2.8";
|
|
||||||
# src = builtins.fetchGit {
|
|
||||||
# url = "git@bscpm03.bsc.es:ompss-kernel/linux.git";
|
|
||||||
# rev = gitCommit;
|
|
||||||
# ref = "fcs";
|
|
||||||
# };
|
|
||||||
# configfile = if lockdep then ./configs/lockdep else ./configs/defconfig;
|
|
||||||
#};
|
|
||||||
|
|
||||||
kernel = nixos-fcs;
|
|
||||||
|
|
||||||
nixos-fcs-kernel = lib.makeOverridable ({gitCommit, lockStat ? false, preempt ? false, branch ? "fcs"}: pkgs.linuxPackagesFor (pkgs.buildLinux rec {
|
|
||||||
version = "6.2.8";
|
|
||||||
src = builtins.fetchGit {
|
|
||||||
url = "git@bscpm03.bsc.es:ompss-kernel/linux.git";
|
|
||||||
rev = gitCommit;
|
|
||||||
ref = branch;
|
|
||||||
};
|
|
||||||
structuredExtraConfig = with lib.kernel; {
|
|
||||||
# add general custom kernel options here
|
|
||||||
} // lib.optionalAttrs lockStat {
|
|
||||||
LOCK_STAT = yes;
|
|
||||||
} // lib.optionalAttrs preempt {
|
|
||||||
PREEMPT = lib.mkForce yes;
|
|
||||||
PREEMPT_VOLUNTARY = lib.mkForce no;
|
|
||||||
};
|
|
||||||
kernelPatches = [];
|
|
||||||
extraMeta.branch = lib.versions.majorMinor version;
|
|
||||||
}));
|
|
||||||
|
|
||||||
nixos-fcs = nixos-fcs-kernel {gitCommit = "8a09822dfcc8f0626b209d6d2aec8b5da459dfee";};
|
|
||||||
nixos-fcs-lockstat = nixos-fcs.override {
|
|
||||||
lockStat = true;
|
|
||||||
};
|
|
||||||
nixos-fcs-lockstat-preempt = nixos-fcs.override {
|
|
||||||
lockStat = true;
|
|
||||||
preempt = true;
|
|
||||||
};
|
|
||||||
latest = pkgs.linuxPackages_latest;
|
|
||||||
|
|
||||||
in {
|
|
||||||
imports = [
|
|
||||||
./lttng.nix
|
|
||||||
./perf.nix
|
|
||||||
];
|
|
||||||
boot.kernelPackages = lib.mkForce kernel;
|
|
||||||
|
|
||||||
# disable all cpu mitigations
|
|
||||||
boot.kernelParams = [
|
|
||||||
"mitigations=off"
|
|
||||||
];
|
|
||||||
|
|
||||||
# enable memory overcommit, needed to build a taglibc system using nix after
|
|
||||||
# increasing the openblas memory footprint
|
|
||||||
boot.kernel.sysctl."vm.overcommit_memory" = 1;
|
|
||||||
}
|
|
||||||
@ -1,43 +0,0 @@
|
|||||||
{ config, pkgs, lib, ... }:
|
|
||||||
|
|
||||||
let
|
|
||||||
|
|
||||||
# The lttng btrfs probe crashes at compile time because of an undefined
|
|
||||||
# function. This disables the btrfs tracepoints to avoid the issue.
|
|
||||||
|
|
||||||
# Also enable lockdep tracepoints, this is disabled by default because it
|
|
||||||
# does not work well on architectures other than x86_64 (i think that arm) as
|
|
||||||
# I was told on the mailing list.
|
|
||||||
lttng-modules-fixed = config.boot.kernelPackages.lttng-modules.overrideAttrs (finalAttrs: previousAttrs: {
|
|
||||||
patchPhase = (lib.optionalString (previousAttrs ? patchPhase) previousAttrs.patchPhase) + ''
|
|
||||||
# disable btrfs
|
|
||||||
substituteInPlace src/probes/Kbuild \
|
|
||||||
--replace " obj-\$(CONFIG_LTTNG) += lttng-probe-btrfs.o" " #obj-\$(CONFIG_LTTNG) += lttng-probe-btrfs.o"
|
|
||||||
|
|
||||||
# enable lockdep tracepoints
|
|
||||||
substituteInPlace src/probes/Kbuild \
|
|
||||||
--replace "#ifneq (\$(CONFIG_LOCKDEP),)" "ifneq (\$(CONFIG_LOCKDEP),)" \
|
|
||||||
--replace "# obj-\$(CONFIG_LTTNG) += lttng-probe-lock.o" " obj-\$(CONFIG_LTTNG) += lttng-probe-lock.o" \
|
|
||||||
--replace "#endif # CONFIG_LOCKDEP" "endif # CONFIG_LOCKDEP"
|
|
||||||
'';
|
|
||||||
});
|
|
||||||
in {
|
|
||||||
|
|
||||||
# add the lttng tools and modules to the system environment
|
|
||||||
boot.extraModulePackages = [ lttng-modules-fixed ];
|
|
||||||
environment.systemPackages = with pkgs; [
|
|
||||||
lttng-tools lttng-ust babeltrace
|
|
||||||
];
|
|
||||||
|
|
||||||
# start the lttng root daemon to manage kernel events
|
|
||||||
systemd.services.lttng-sessiond = {
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
description = "LTTng session daemon for the root user";
|
|
||||||
serviceConfig = {
|
|
||||||
User = "root";
|
|
||||||
ExecStart = ''
|
|
||||||
${pkgs.lttng-tools}/bin/lttng-sessiond
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,22 +0,0 @@
|
|||||||
{ config, pkgs, lib, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
# add the perf tool
|
|
||||||
environment.systemPackages = with pkgs; [
|
|
||||||
config.boot.kernelPackages.perf
|
|
||||||
];
|
|
||||||
|
|
||||||
# allow non-root users to read tracing data from the kernel
|
|
||||||
boot.kernel.sysctl."kernel.perf_event_paranoid" = -2;
|
|
||||||
boot.kernel.sysctl."kernel.kptr_restrict" = 0;
|
|
||||||
|
|
||||||
# specify additionl options to the tracefs directory to allow members of the
|
|
||||||
# tracing group to access tracefs.
|
|
||||||
fileSystems."/sys/kernel/tracing" = {
|
|
||||||
options = [
|
|
||||||
"mode=755"
|
|
||||||
"gid=tracing"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
@ -1,11 +0,0 @@
|
|||||||
{ ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
security.sudo.extraRules= [{
|
|
||||||
users = [ "arocanon" ];
|
|
||||||
commands = [{
|
|
||||||
command = "ALL" ;
|
|
||||||
options= [ "NOPASSWD" ]; # "SETENV" # Adding the following could be a good idea
|
|
||||||
}];
|
|
||||||
}];
|
|
||||||
}
|
|
||||||
@ -1,112 +0,0 @@
|
|||||||
{ lib, config, pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../common/base.nix
|
|
||||||
../common/xeon/console.nix
|
|
||||||
../module/amd-uprof.nix
|
|
||||||
../module/emulation.nix
|
|
||||||
../module/nvidia.nix
|
|
||||||
../module/slurm-client.nix
|
|
||||||
../module/hut-substituter.nix
|
|
||||||
./wireguard.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
# Don't turn off on August as UPC has different dates.
|
|
||||||
# Fox works fine on power cuts.
|
|
||||||
systemd.timers.august-shutdown.enable = false;
|
|
||||||
|
|
||||||
# Select the this using the ID to avoid mismatches
|
|
||||||
boot.loader.grub.device = "/dev/disk/by-id/wwn-0x500a07514b0c1103";
|
|
||||||
|
|
||||||
# No swap, there is plenty of RAM
|
|
||||||
swapDevices = lib.mkForce [];
|
|
||||||
|
|
||||||
boot.initrd.availableKernelModules = [ "xhci_pci" "ahci" "nvme" "usbhid" "usb_storage" "sd_mod" ];
|
|
||||||
boot.kernelModules = [ "kvm-amd" "amd_uncore" "amd_hsmp" ];
|
|
||||||
|
|
||||||
hardware.cpu.amd.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
|
|
||||||
hardware.cpu.intel.updateMicrocode = lib.mkForce false;
|
|
||||||
|
|
||||||
# Use performance for benchmarks
|
|
||||||
powerManagement.cpuFreqGovernor = "performance";
|
|
||||||
|
|
||||||
services.amd-uprof.enable = true;
|
|
||||||
|
|
||||||
# Disable NUMA balancing
|
|
||||||
boot.kernel.sysctl."kernel.numa_balancing" = 0;
|
|
||||||
|
|
||||||
# Expose kernel addresses
|
|
||||||
boot.kernel.sysctl."kernel.kptr_restrict" = 0;
|
|
||||||
|
|
||||||
# Disable NMI watchdog to save one hw counter (for AMD uProf)
|
|
||||||
boot.kernel.sysctl."kernel.nmi_watchdog" = 0;
|
|
||||||
|
|
||||||
services.openssh.settings.X11Forwarding = true;
|
|
||||||
|
|
||||||
services.fail2ban.enable = true;
|
|
||||||
|
|
||||||
networking = {
|
|
||||||
timeServers = [ "ntp1.upc.edu" "ntp2.upc.edu" ];
|
|
||||||
hostName = "fox";
|
|
||||||
# UPC network (may change over time, use DHCP)
|
|
||||||
# Public IP configuration:
|
|
||||||
# - Hostname: fox.ac.upc.edu
|
|
||||||
# - IP: 147.83.30.141
|
|
||||||
# - Gateway: 147.83.30.130
|
|
||||||
# - NetMask: 255.255.255.192
|
|
||||||
# Private IP configuration for BMC:
|
|
||||||
# - Hostname: fox-ipmi.ac.upc.edu
|
|
||||||
# - IP: 147.83.35.27
|
|
||||||
# - Gateway: 147.83.35.2
|
|
||||||
# - NetMask: 255.255.255.0
|
|
||||||
interfaces.enp1s0f0np0.useDHCP = true;
|
|
||||||
};
|
|
||||||
|
|
||||||
# Recommended for new graphics cards
|
|
||||||
hardware.nvidia.open = true;
|
|
||||||
|
|
||||||
# Mount NVME disks
|
|
||||||
fileSystems."/nvme0" = { device = "/dev/disk/by-label/nvme0"; fsType = "ext4"; };
|
|
||||||
fileSystems."/nvme1" = { device = "/dev/disk/by-label/nvme1"; fsType = "ext4"; };
|
|
||||||
|
|
||||||
# Mount the NFS home
|
|
||||||
fileSystems."/nfs/home" = {
|
|
||||||
device = "10.106.0.30:/home";
|
|
||||||
fsType = "nfs";
|
|
||||||
options = [ "nfsvers=3" "rsize=1024" "wsize=1024" "cto" "nofail" ];
|
|
||||||
};
|
|
||||||
|
|
||||||
# Make a /nvme{0,1}/$USER directory for each user.
|
|
||||||
systemd.services.create-nvme-dirs = let
|
|
||||||
# Take only normal users in fox
|
|
||||||
users = lib.filterAttrs (_: v: v.isNormalUser) config.users.users;
|
|
||||||
commands = lib.concatLists (lib.mapAttrsToList
|
|
||||||
(_: user: [
|
|
||||||
"install -d -o ${user.name} -g ${user.group} -m 0755 /nvme{0,1}/${user.name}"
|
|
||||||
]) users);
|
|
||||||
script = pkgs.writeShellScript "create-nvme-dirs.sh" (lib.concatLines commands);
|
|
||||||
in {
|
|
||||||
enable = true;
|
|
||||||
wants = [ "local-fs.target" ];
|
|
||||||
after = [ "local-fs.target" ];
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
serviceConfig.ExecStart = script;
|
|
||||||
};
|
|
||||||
|
|
||||||
# Only allow SSH connections from users who have a SLURM allocation
|
|
||||||
# See: https://slurm.schedmd.com/pam_slurm_adopt.html
|
|
||||||
security.pam.services.sshd.rules.account.slurm = {
|
|
||||||
control = "required";
|
|
||||||
enable = true;
|
|
||||||
modulePath = "${pkgs.slurm}/lib/security/pam_slurm_adopt.so";
|
|
||||||
args = [ "log_level=debug5" ];
|
|
||||||
order = 999999; # Make it last one
|
|
||||||
};
|
|
||||||
|
|
||||||
# Disable systemd session (pam_systemd.so) as it will conflict with the
|
|
||||||
# pam_slurm_adopt.so module. What happens is that the shell is first adopted
|
|
||||||
# into the slurmstepd task and then into the systemd session, which is not
|
|
||||||
# what we want, otherwise it will linger even if all jobs are gone.
|
|
||||||
security.pam.services.sshd.startSession = lib.mkForce false;
|
|
||||||
}
|
|
||||||
@ -1,54 +0,0 @@
|
|||||||
{ config, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
networking.firewall = {
|
|
||||||
allowedUDPPorts = [ 666 ];
|
|
||||||
};
|
|
||||||
|
|
||||||
age.secrets.wgFox.file = ../../secrets/wg-fox.age;
|
|
||||||
|
|
||||||
networking.wireguard.enable = true;
|
|
||||||
networking.wireguard.interfaces = {
|
|
||||||
# "wg0" is the network interface name. You can name the interface arbitrarily.
|
|
||||||
wg0 = {
|
|
||||||
# Determines the IP address and subnet of the server's end of the tunnel interface.
|
|
||||||
ips = [ "10.106.0.1/24" ];
|
|
||||||
|
|
||||||
# The port that WireGuard listens to. Must be accessible by the client.
|
|
||||||
listenPort = 666;
|
|
||||||
|
|
||||||
# Path to the private key file.
|
|
||||||
privateKeyFile = config.age.secrets.wgFox.path;
|
|
||||||
# Public key: VfMPBQLQTKeyXJSwv8wBhc6OV0j2qAxUpX3kLHunK2Y=
|
|
||||||
|
|
||||||
peers = [
|
|
||||||
# List of allowed peers.
|
|
||||||
{
|
|
||||||
name = "apex";
|
|
||||||
publicKey = "VwhcN8vSOzdJEotQTpmPHBC52x3Hbv1lkFIyKubrnUA=";
|
|
||||||
# List of IPs assigned to this peer within the tunnel subnet. Used to configure routing.
|
|
||||||
allowedIPs = [ "10.106.0.30/32" "10.0.40.7/32" ];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
name = "raccoon";
|
|
||||||
publicKey = "QUfnGXSMEgu2bviglsaSdCjidB51oEDBFpnSFcKGfDI=";
|
|
||||||
allowedIPs = [ "10.106.0.236/32" "192.168.0.0/16" "10.0.44.0/24" ];
|
|
||||||
}
|
|
||||||
];
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
networking.hosts = {
|
|
||||||
"10.106.0.30" = [ "apex" ];
|
|
||||||
"10.0.40.7" = [ "hut" ];
|
|
||||||
"10.106.0.236" = [ "raccoon" ];
|
|
||||||
"10.0.44.4" = [ "tent" ];
|
|
||||||
};
|
|
||||||
|
|
||||||
networking.firewall = {
|
|
||||||
extraCommands = ''
|
|
||||||
# Accept slurm connections to slurmd from apex (via wireguard)
|
|
||||||
iptables -A nixos-fw -p tcp -i wg0 -s 10.106.0.30/32 -d 10.106.0.1/32 --dport 6818 -j nixos-fw-accept
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,14 +0,0 @@
|
|||||||
modules:
|
|
||||||
http_2xx:
|
|
||||||
prober: http
|
|
||||||
timeout: 5s
|
|
||||||
http:
|
|
||||||
follow_redirects: true
|
|
||||||
preferred_ip_protocol: "ip4"
|
|
||||||
valid_status_codes: [] # Defaults to 2xx
|
|
||||||
method: GET
|
|
||||||
icmp:
|
|
||||||
prober: icmp
|
|
||||||
timeout: 5s
|
|
||||||
icmp:
|
|
||||||
preferred_ip_protocol: "ip4"
|
|
||||||
@ -1,67 +0,0 @@
|
|||||||
{ config, pkgs, lib, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../common/ssf.nix
|
|
||||||
|
|
||||||
../module/ceph.nix
|
|
||||||
../module/debuginfod.nix
|
|
||||||
../module/emulation.nix
|
|
||||||
./gitlab-runner.nix
|
|
||||||
./monitoring.nix
|
|
||||||
./nfs.nix
|
|
||||||
./nix-serve.nix
|
|
||||||
./public-inbox.nix
|
|
||||||
./gitea.nix
|
|
||||||
./msmtp.nix
|
|
||||||
./postgresql.nix
|
|
||||||
./nginx.nix
|
|
||||||
./p.nix
|
|
||||||
#./pxe.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
# Select the this using the ID to avoid mismatches
|
|
||||||
boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d53567f";
|
|
||||||
|
|
||||||
fileSystems = {
|
|
||||||
"/" = lib.mkForce {
|
|
||||||
device = "/dev/disk/by-label/nvme";
|
|
||||||
fsType = "ext4";
|
|
||||||
neededForBoot = true;
|
|
||||||
options = [ "noatime" ];
|
|
||||||
};
|
|
||||||
|
|
||||||
"/boot" = lib.mkForce {
|
|
||||||
device = "/dev/disk/by-label/nixos-boot";
|
|
||||||
fsType = "ext4";
|
|
||||||
neededForBoot = true;
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
networking = {
|
|
||||||
hostName = "hut";
|
|
||||||
interfaces.eno1.ipv4.addresses = [ {
|
|
||||||
address = "10.0.40.7";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
interfaces.ibp5s0.ipv4.addresses = [ {
|
|
||||||
address = "10.0.42.7";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
firewall = {
|
|
||||||
extraCommands = ''
|
|
||||||
# Accept all proxy traffic from compute nodes but not the login
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.30 --dport 23080 -j nixos-fw-log-refuse
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.0/24 --dport 23080 -j nixos-fw-accept
|
|
||||||
'';
|
|
||||||
# Flush all rules and chains on stop so it won't break on start
|
|
||||||
extraStopCommands = ''
|
|
||||||
iptables -F
|
|
||||||
iptables -X
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# Allow proxy to bind to the ethernet interface
|
|
||||||
services.openssh.settings.GatewayPorts = "clientspecified";
|
|
||||||
}
|
|
||||||
@ -1,63 +0,0 @@
|
|||||||
{ config, lib, ... }:
|
|
||||||
{
|
|
||||||
age.secrets.giteaRunnerToken.file = ../../secrets/gitea-runner-token.age;
|
|
||||||
|
|
||||||
services.gitea = {
|
|
||||||
enable = true;
|
|
||||||
appName = "Gitea in the jungle";
|
|
||||||
|
|
||||||
settings = {
|
|
||||||
server = {
|
|
||||||
ROOT_URL = "https://jungle.bsc.es/git/";
|
|
||||||
LOCAL_ROOT_URL = "https://jungle.bsc.es/git/";
|
|
||||||
LANDING_PAGE = "explore";
|
|
||||||
};
|
|
||||||
metrics.ENABLED = true;
|
|
||||||
service = {
|
|
||||||
REGISTER_MANUAL_CONFIRM = true;
|
|
||||||
ENABLE_NOTIFY_MAIL = true;
|
|
||||||
};
|
|
||||||
log.LEVEL = "Warn";
|
|
||||||
|
|
||||||
mailer = {
|
|
||||||
ENABLED = true;
|
|
||||||
FROM = "jungle-robot@bsc.es";
|
|
||||||
PROTOCOL = "sendmail";
|
|
||||||
SENDMAIL_PATH = "/run/wrappers/bin/sendmail";
|
|
||||||
SENDMAIL_ARGS = "--";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
services.gitea-actions-runner.instances = {
|
|
||||||
runrun = {
|
|
||||||
enable = true;
|
|
||||||
name = "runrun";
|
|
||||||
url = "https://jungle.bsc.es/git/";
|
|
||||||
tokenFile = config.age.secrets.giteaRunnerToken.path;
|
|
||||||
labels = [ "native:host" ];
|
|
||||||
settings.runner.capacity = 8;
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
systemd.services.gitea-runner-runrun = {
|
|
||||||
path = [ "/run/current-system/sw" ];
|
|
||||||
serviceConfig = {
|
|
||||||
# DynamicUser doesn't work well with SSH
|
|
||||||
DynamicUser = lib.mkForce false;
|
|
||||||
User = "gitea-runner";
|
|
||||||
Group = "gitea-runner";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
users.users.gitea-runner = {
|
|
||||||
isSystemUser = true;
|
|
||||||
home = "/var/lib/gitea-runner";
|
|
||||||
description = "Gitea Runner";
|
|
||||||
group = "gitea-runner";
|
|
||||||
extraGroups = [ "docker" ];
|
|
||||||
createHome = true;
|
|
||||||
};
|
|
||||||
users.groups.gitea-runner = {};
|
|
||||||
}
|
|
||||||
|
|
||||||
@ -1,126 +0,0 @@
|
|||||||
{ pkgs, lib, config, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
age.secrets.gitlab-pm-shell.file = ../../secrets/gitlab-runner-shell-token.age;
|
|
||||||
age.secrets.gitlab-pm-docker.file = ../../secrets/gitlab-runner-docker-token.age;
|
|
||||||
age.secrets.gitlab-bsc-docker.file = ../../secrets/gitlab-bsc-docker-token.age;
|
|
||||||
|
|
||||||
services.gitlab-runner = {
|
|
||||||
enable = true;
|
|
||||||
settings.concurrent = 5;
|
|
||||||
services = let
|
|
||||||
common-shell = {
|
|
||||||
executor = "shell";
|
|
||||||
environmentVariables = {
|
|
||||||
SHELL = "${pkgs.bash}/bin/bash";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
common-docker = {
|
|
||||||
executor = "docker";
|
|
||||||
dockerImage = "debian:stable";
|
|
||||||
registrationFlags = [
|
|
||||||
"--docker-network-mode host"
|
|
||||||
];
|
|
||||||
environmentVariables = {
|
|
||||||
https_proxy = "http://hut:23080";
|
|
||||||
http_proxy = "http://hut:23080";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
in {
|
|
||||||
# For pm.bsc.es/gitlab
|
|
||||||
gitlab-pm-shell = common-shell // {
|
|
||||||
authenticationTokenConfigFile = config.age.secrets.gitlab-pm-shell.path;
|
|
||||||
};
|
|
||||||
gitlab-pm-docker = common-docker // {
|
|
||||||
authenticationTokenConfigFile = config.age.secrets.gitlab-pm-docker.path;
|
|
||||||
};
|
|
||||||
|
|
||||||
gitlab-bsc-docker = {
|
|
||||||
# gitlab.bsc.es still uses the old token mechanism
|
|
||||||
registrationConfigFile = config.age.secrets.gitlab-bsc-docker.path;
|
|
||||||
tagList = [ "docker" "hut" ];
|
|
||||||
environmentVariables = {
|
|
||||||
# We cannot access the hut local interface from docker, so we connect
|
|
||||||
# to hut directly via the ethernet one.
|
|
||||||
https_proxy = "http://hut:23080";
|
|
||||||
http_proxy = "http://hut:23080";
|
|
||||||
};
|
|
||||||
executor = "docker";
|
|
||||||
dockerImage = "alpine";
|
|
||||||
dockerVolumes = [
|
|
||||||
"/nix/store:/nix/store:ro"
|
|
||||||
"/nix/var/nix/db:/nix/var/nix/db:ro"
|
|
||||||
"/nix/var/nix/daemon-socket:/nix/var/nix/daemon-socket:ro"
|
|
||||||
];
|
|
||||||
dockerExtraHosts = [
|
|
||||||
# Required to pass the proxy via hut
|
|
||||||
"hut:10.0.40.7"
|
|
||||||
];
|
|
||||||
dockerDisableCache = true;
|
|
||||||
registrationFlags = [
|
|
||||||
# Increase build log length to 64 MiB
|
|
||||||
"--output-limit 65536"
|
|
||||||
];
|
|
||||||
preBuildScript = pkgs.writeScript "setup-container" ''
|
|
||||||
mkdir -p -m 0755 /nix/var/log/nix/drvs
|
|
||||||
mkdir -p -m 0755 /nix/var/nix/gcroots
|
|
||||||
mkdir -p -m 0755 /nix/var/nix/profiles
|
|
||||||
mkdir -p -m 0755 /nix/var/nix/temproots
|
|
||||||
mkdir -p -m 0755 /nix/var/nix/userpool
|
|
||||||
mkdir -p -m 1777 /nix/var/nix/gcroots/per-user
|
|
||||||
mkdir -p -m 1777 /nix/var/nix/profiles/per-user
|
|
||||||
mkdir -p -m 0755 /nix/var/nix/profiles/per-user/root
|
|
||||||
mkdir -p -m 0700 "$HOME/.nix-defexpr"
|
|
||||||
mkdir -p -m 0700 "$HOME/.ssh"
|
|
||||||
cat > "$HOME/.ssh/config" << EOF
|
|
||||||
Host bscpm04.bsc.es gitlab-internal.bsc.es
|
|
||||||
User git
|
|
||||||
ProxyCommand nc -X connect -x hut:23080 %h %p
|
|
||||||
Host amdlogin1.bsc.es armlogin1.bsc.es hualogin1.bsc.es glogin1.bsc.es glogin2.bsc.es fpgalogin1.bsc.es
|
|
||||||
ProxyCommand nc -X connect -x hut:23080 %h %p
|
|
||||||
EOF
|
|
||||||
cat >> "$HOME/.ssh/known_hosts" << EOF
|
|
||||||
bscpm04.bsc.es ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPx4mC0etyyjYUT2Ztc/bs4ZXSbVMrogs1ZTP924PDgT
|
|
||||||
gitlab-internal.bsc.es ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF9arsAOSRB06hdy71oTvJHG2Mg8zfebADxpvc37lZo3
|
|
||||||
EOF
|
|
||||||
. ${pkgs.nix}/etc/profile.d/nix-daemon.sh
|
|
||||||
# Required to load SSL certificate paths
|
|
||||||
. ${pkgs.cacert}/nix-support/setup-hook
|
|
||||||
'';
|
|
||||||
environmentVariables = {
|
|
||||||
ENV = "/etc/profile";
|
|
||||||
USER = "root";
|
|
||||||
NIX_REMOTE = "daemon";
|
|
||||||
PATH = "${config.system.path}/bin:/bin:/sbin:/usr/bin:/usr/sbin";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# DOCKER* chains are useless, override at FORWARD and nixos-fw
|
|
||||||
networking.firewall.extraCommands = ''
|
|
||||||
# Don't forward any traffic from docker
|
|
||||||
iptables -I FORWARD 1 -p all -i docker0 -j nixos-fw-log-refuse
|
|
||||||
|
|
||||||
# Allow incoming traffic from docker to 23080
|
|
||||||
iptables -A nixos-fw -p tcp -i docker0 -d hut --dport 23080 -j ACCEPT
|
|
||||||
'';
|
|
||||||
|
|
||||||
#systemd.services.gitlab-runner.serviceConfig.Shell = "${pkgs.bash}/bin/bash";
|
|
||||||
systemd.services.gitlab-runner.serviceConfig.DynamicUser = lib.mkForce false;
|
|
||||||
systemd.services.gitlab-runner.serviceConfig.User = "gitlab-runner";
|
|
||||||
systemd.services.gitlab-runner.serviceConfig.Group = "gitlab-runner";
|
|
||||||
systemd.services.gitlab-runner.serviceConfig.ExecStart = lib.mkForce
|
|
||||||
''${pkgs.gitlab-runner}/bin/gitlab-runner run --config ''${HOME}/.gitlab-runner/config.toml --listen-address "127.0.0.1:9252" --working-directory ''${HOME}'';
|
|
||||||
|
|
||||||
users.users.gitlab-runner = {
|
|
||||||
uid = config.ids.uids.gitlab-runner;
|
|
||||||
#isNormalUser = true;
|
|
||||||
home = "/var/lib/gitlab-runner";
|
|
||||||
description = "Gitlab Runner";
|
|
||||||
group = "gitlab-runner";
|
|
||||||
extraGroups = [ "docker" ];
|
|
||||||
createHome = true;
|
|
||||||
};
|
|
||||||
users.groups.gitlab-runner.gid = config.ids.gids.gitlab-runner;
|
|
||||||
}
|
|
||||||
@ -1,31 +0,0 @@
|
|||||||
{ pkgs, config, lib, ... }:
|
|
||||||
let
|
|
||||||
gpfs-probe-script = pkgs.runCommand "gpfs-probe.sh" { }
|
|
||||||
''
|
|
||||||
cp ${./gpfs-probe.sh} $out;
|
|
||||||
chmod +x $out
|
|
||||||
''
|
|
||||||
;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
# Use a new user to handle the SSH keys
|
|
||||||
users.groups.ssh-robot = { };
|
|
||||||
users.users.ssh-robot = {
|
|
||||||
description = "SSH Robot";
|
|
||||||
isNormalUser = true;
|
|
||||||
home = "/var/lib/ssh-robot";
|
|
||||||
};
|
|
||||||
|
|
||||||
systemd.services.gpfs-probe = {
|
|
||||||
description = "Daemon to report GPFS latency via SSH";
|
|
||||||
path = [ pkgs.openssh pkgs.netcat ];
|
|
||||||
after = [ "network.target" ];
|
|
||||||
wantedBy = [ "default.target" ];
|
|
||||||
serviceConfig = {
|
|
||||||
Type = "simple";
|
|
||||||
ExecStart = "${pkgs.socat}/bin/socat TCP4-LISTEN:9966,fork EXEC:${gpfs-probe-script}";
|
|
||||||
User = "ssh-robot";
|
|
||||||
Group = "ssh-robot";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,18 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
|
|
||||||
N=500
|
|
||||||
|
|
||||||
t=$(timeout 5 ssh bsc015557@glogin2.bsc.es "timeout 3 command time -f %e touch /gpfs/projects/bsc15/bsc015557/gpfs.{1..$N} 2>&1; rm -f /gpfs/projects/bsc15/bsc015557/gpfs.{1..$N}")
|
|
||||||
|
|
||||||
if [ -z "$t" ]; then
|
|
||||||
t="5.00"
|
|
||||||
fi
|
|
||||||
|
|
||||||
cat <<EOF
|
|
||||||
HTTP/1.1 200 OK
|
|
||||||
Content-Type: text/plain; version=0.0.4; charset=utf-8; escaping=values
|
|
||||||
|
|
||||||
# HELP gpfs_touch_latency Time to create $N files.
|
|
||||||
# TYPE gpfs_touch_latency gauge
|
|
||||||
gpfs_touch_latency $t
|
|
||||||
EOF
|
|
||||||
@ -1,272 +0,0 @@
|
|||||||
{ config, lib, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../module/slurm-exporter.nix
|
|
||||||
../module/meteocat-exporter.nix
|
|
||||||
../module/upc-qaire-exporter.nix
|
|
||||||
./gpfs-probe.nix
|
|
||||||
../module/nix-daemon-exporter.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
age.secrets.grafanaJungleRobotPassword = {
|
|
||||||
file = ../../secrets/jungle-robot-password.age;
|
|
||||||
owner = "grafana";
|
|
||||||
mode = "400";
|
|
||||||
};
|
|
||||||
|
|
||||||
age.secrets.ipmiYml.file = ../../secrets/ipmi.yml.age;
|
|
||||||
|
|
||||||
services.grafana = {
|
|
||||||
enable = true;
|
|
||||||
settings = {
|
|
||||||
server = {
|
|
||||||
domain = "jungle.bsc.es";
|
|
||||||
root_url = "%(protocol)s://%(domain)s/grafana";
|
|
||||||
serve_from_sub_path = true;
|
|
||||||
http_port = 2342;
|
|
||||||
http_addr = "127.0.0.1";
|
|
||||||
};
|
|
||||||
smtp = {
|
|
||||||
enabled = true;
|
|
||||||
from_address = "jungle-robot@bsc.es";
|
|
||||||
user = "jungle-robot";
|
|
||||||
# Read the password from a file, which is only readable by grafana user
|
|
||||||
# https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#file-provider
|
|
||||||
password = "$__file{${config.age.secrets.grafanaJungleRobotPassword.path}}";
|
|
||||||
host = "mail.bsc.es:465";
|
|
||||||
startTLS_policy = "NoStartTLS";
|
|
||||||
};
|
|
||||||
feature_toggles.publicDashboards = true;
|
|
||||||
"auth.anonymous".enabled = true;
|
|
||||||
log.level = "warn";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# Make grafana alerts also use the proxy
|
|
||||||
systemd.services.grafana.environment = config.networking.proxy.envVars;
|
|
||||||
|
|
||||||
services.prometheus = {
|
|
||||||
enable = true;
|
|
||||||
port = 9001;
|
|
||||||
retentionTime = "5y";
|
|
||||||
listenAddress = "127.0.0.1";
|
|
||||||
};
|
|
||||||
|
|
||||||
systemd.services.prometheus-ipmi-exporter.serviceConfig.DynamicUser = lib.mkForce false;
|
|
||||||
systemd.services.prometheus-ipmi-exporter.serviceConfig.PrivateDevices = lib.mkForce false;
|
|
||||||
|
|
||||||
# We need access to the devices to monitor the disk space
|
|
||||||
systemd.services.prometheus-node-exporter.serviceConfig.PrivateDevices = lib.mkForce false;
|
|
||||||
systemd.services.prometheus-node-exporter.serviceConfig.ProtectHome = lib.mkForce "read-only";
|
|
||||||
|
|
||||||
virtualisation.docker.daemon.settings = {
|
|
||||||
metrics-addr = "127.0.0.1:9323";
|
|
||||||
};
|
|
||||||
|
|
||||||
# Required to allow the smartctl exporter to read the nvme0 character device,
|
|
||||||
# see the commit message on:
|
|
||||||
# https://github.com/NixOS/nixpkgs/commit/12c26aca1fd55ab99f831bedc865a626eee39f80
|
|
||||||
services.udev.extraRules = ''
|
|
||||||
SUBSYSTEM=="nvme", KERNEL=="nvme[0-9]*", GROUP="disk"
|
|
||||||
'';
|
|
||||||
|
|
||||||
services.prometheus = {
|
|
||||||
|
|
||||||
exporters = {
|
|
||||||
ipmi = {
|
|
||||||
enable = true;
|
|
||||||
group = "root";
|
|
||||||
user = "root";
|
|
||||||
configFile = config.age.secrets.ipmiYml.path;
|
|
||||||
# extraFlags = [ "--log.level=debug" ];
|
|
||||||
listenAddress = "127.0.0.1";
|
|
||||||
};
|
|
||||||
node = {
|
|
||||||
enable = true;
|
|
||||||
enabledCollectors = [ "systemd" "logind" ];
|
|
||||||
port = 9002;
|
|
||||||
listenAddress = "127.0.0.1";
|
|
||||||
};
|
|
||||||
smartctl = {
|
|
||||||
enable = true;
|
|
||||||
listenAddress = "127.0.0.1";
|
|
||||||
};
|
|
||||||
blackbox = {
|
|
||||||
enable = true;
|
|
||||||
listenAddress = "127.0.0.1";
|
|
||||||
configFile = ./blackbox.yml;
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
scrapeConfigs = [
|
|
||||||
{
|
|
||||||
job_name = "xeon07";
|
|
||||||
static_configs = [{
|
|
||||||
targets = [
|
|
||||||
"127.0.0.1:${toString config.services.prometheus.exporters.node.port}"
|
|
||||||
"127.0.0.1:${toString config.services.prometheus.exporters.ipmi.port}"
|
|
||||||
"127.0.0.1:9323"
|
|
||||||
"127.0.0.1:9252"
|
|
||||||
"127.0.0.1:${toString config.services.prometheus.exporters.smartctl.port}"
|
|
||||||
"127.0.0.1:9341" # Slurm exporter
|
|
||||||
"127.0.0.1:9966" # GPFS custom exporter
|
|
||||||
"127.0.0.1:9999" # Nix-daemon custom exporter
|
|
||||||
"127.0.0.1:9929" # Meteocat custom exporter
|
|
||||||
"127.0.0.1:9928" # UPC Qaire custom exporter
|
|
||||||
"127.0.0.1:${toString config.services.prometheus.exporters.blackbox.port}"
|
|
||||||
];
|
|
||||||
}];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
job_name = "ceph";
|
|
||||||
static_configs = [{
|
|
||||||
targets = [
|
|
||||||
"10.0.40.40:9283" # Ceph statistics
|
|
||||||
"10.0.40.40:9002" # Node exporter
|
|
||||||
"10.0.40.42:9002" # Node exporter
|
|
||||||
];
|
|
||||||
}];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
job_name = "blackbox-http";
|
|
||||||
metrics_path = "/probe";
|
|
||||||
params = { module = [ "http_2xx" ]; };
|
|
||||||
static_configs = [{
|
|
||||||
targets = [
|
|
||||||
"https://www.google.com/robots.txt"
|
|
||||||
"https://pm.bsc.es/"
|
|
||||||
"https://pm.bsc.es/gitlab/"
|
|
||||||
"https://jungle.bsc.es/"
|
|
||||||
"https://gitlab.bsc.es/"
|
|
||||||
];
|
|
||||||
}];
|
|
||||||
relabel_configs = [
|
|
||||||
{
|
|
||||||
# Takes the address and sets it in the "target=<xyz>" URL parameter
|
|
||||||
source_labels = [ "__address__" ];
|
|
||||||
target_label = "__param_target";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Sets the "instance" label with the remote host we are querying
|
|
||||||
source_labels = [ "__param_target" ];
|
|
||||||
target_label = "instance";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Shows the host target address instead of the blackbox address
|
|
||||||
target_label = "__address__";
|
|
||||||
replacement = "127.0.0.1:${toString config.services.prometheus.exporters.blackbox.port}";
|
|
||||||
}
|
|
||||||
];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
job_name = "blackbox-icmp";
|
|
||||||
metrics_path = "/probe";
|
|
||||||
params = { module = [ "icmp" ]; };
|
|
||||||
static_configs = [{
|
|
||||||
targets = [
|
|
||||||
"1.1.1.1"
|
|
||||||
"8.8.8.8"
|
|
||||||
"ssfhead"
|
|
||||||
"anella-bsc.cesca.cat"
|
|
||||||
"upc-anella.cesca.cat"
|
|
||||||
"fox.ac.upc.edu"
|
|
||||||
"arenys5.ac.upc.edu"
|
|
||||||
];
|
|
||||||
}];
|
|
||||||
relabel_configs = [
|
|
||||||
{
|
|
||||||
# Takes the address and sets it in the "target=<xyz>" URL parameter
|
|
||||||
source_labels = [ "__address__" ];
|
|
||||||
target_label = "__param_target";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Sets the "instance" label with the remote host we are querying
|
|
||||||
source_labels = [ "__param_target" ];
|
|
||||||
target_label = "instance";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Shows the host target address instead of the blackbox address
|
|
||||||
target_label = "__address__";
|
|
||||||
replacement = "127.0.0.1:${toString config.services.prometheus.exporters.blackbox.port}";
|
|
||||||
}
|
|
||||||
];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
job_name = "gitea";
|
|
||||||
static_configs = [{ targets = [ "127.0.0.1:3000" ]; }];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Scrape the IPMI info of the hosts remotely via LAN
|
|
||||||
job_name = "ipmi-lan";
|
|
||||||
scrape_interval = "1m";
|
|
||||||
scrape_timeout = "30s";
|
|
||||||
metrics_path = "/ipmi";
|
|
||||||
scheme = "http";
|
|
||||||
relabel_configs = [
|
|
||||||
{
|
|
||||||
# Takes the address and sets it in the "target=<xyz>" URL parameter
|
|
||||||
source_labels = [ "__address__" ];
|
|
||||||
separator = ";";
|
|
||||||
regex = "(.*)(:80)?";
|
|
||||||
target_label = "__param_target";
|
|
||||||
replacement = "\${1}";
|
|
||||||
action = "replace";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Sets the "instance" label with the remote host we are querying
|
|
||||||
source_labels = [ "__param_target" ];
|
|
||||||
separator = ";";
|
|
||||||
regex = "(.*)-ipmi"; # Remove "-ipm̀i" at the end
|
|
||||||
target_label = "instance";
|
|
||||||
replacement = "\${1}";
|
|
||||||
action = "replace";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Sets the fixed "module=lan" URL param
|
|
||||||
separator = ";";
|
|
||||||
regex = "(.*)";
|
|
||||||
target_label = "__param_module";
|
|
||||||
replacement = "lan";
|
|
||||||
action = "replace";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Sets the target to query as the localhost IPMI exporter
|
|
||||||
separator = ";";
|
|
||||||
regex = ".*";
|
|
||||||
target_label = "__address__";
|
|
||||||
replacement = "127.0.0.1:9290";
|
|
||||||
action = "replace";
|
|
||||||
}
|
|
||||||
];
|
|
||||||
|
|
||||||
# Load the list of targets from another file
|
|
||||||
file_sd_configs = [
|
|
||||||
{
|
|
||||||
files = [ "${./targets.yml}" ];
|
|
||||||
refresh_interval = "30s";
|
|
||||||
}
|
|
||||||
];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
job_name = "ipmi-raccoon";
|
|
||||||
metrics_path = "/ipmi";
|
|
||||||
static_configs = [
|
|
||||||
{ targets = [ "127.0.0.1:9291" ]; }
|
|
||||||
];
|
|
||||||
params = {
|
|
||||||
target = [ "84.88.51.142" ];
|
|
||||||
module = [ "raccoon" ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
{
|
|
||||||
job_name = "raccoon";
|
|
||||||
static_configs = [
|
|
||||||
{
|
|
||||||
targets = [ "127.0.0.1:19002" ]; # Node exporter
|
|
||||||
}
|
|
||||||
];
|
|
||||||
}
|
|
||||||
];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,24 +0,0 @@
|
|||||||
{ config, lib, ... }:
|
|
||||||
{
|
|
||||||
age.secrets.jungleRobotPassword = {
|
|
||||||
file = ../../secrets/jungle-robot-password.age;
|
|
||||||
group = "gitea";
|
|
||||||
mode = "440";
|
|
||||||
};
|
|
||||||
|
|
||||||
programs.msmtp = {
|
|
||||||
enable = true;
|
|
||||||
accounts = {
|
|
||||||
default = {
|
|
||||||
auth = true;
|
|
||||||
tls = true;
|
|
||||||
tls_starttls = false;
|
|
||||||
port = 465;
|
|
||||||
host = "mail.bsc.es";
|
|
||||||
user = "jungle-robot";
|
|
||||||
passwordeval = "cat ${config.age.secrets.jungleRobotPassword.path}";
|
|
||||||
from = "jungle-robot@bsc.es";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,9 +0,0 @@
|
|||||||
{ ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
services.nfs.server.enable = true;
|
|
||||||
services.nfs.server.exports = ''
|
|
||||||
/nix 10.0.40.0/24(ro,sync,no_subtree_check,root_squash)
|
|
||||||
'';
|
|
||||||
networking.firewall.allowedTCPPorts = [ 2049 ];
|
|
||||||
}
|
|
||||||
@ -1,76 +0,0 @@
|
|||||||
{ theFlake, pkgs, ... }:
|
|
||||||
let
|
|
||||||
website = pkgs.stdenv.mkDerivation {
|
|
||||||
name = "jungle-web";
|
|
||||||
src = pkgs.fetchgit {
|
|
||||||
url = "https://jungle.bsc.es/git/rarias/jungle-website.git";
|
|
||||||
rev = "739bf0175a7f05380fe7ad7023ff1d60db1710e1";
|
|
||||||
hash = "sha256-ea5DzhYTzZ9TmqD+x95rdNdLbxPnBluqlYH2NmBYmc4=";
|
|
||||||
};
|
|
||||||
buildInputs = [ pkgs.hugo ];
|
|
||||||
buildPhase = ''
|
|
||||||
rm -rf public/
|
|
||||||
hugo
|
|
||||||
'';
|
|
||||||
installPhase = ''
|
|
||||||
cp -r public $out
|
|
||||||
'';
|
|
||||||
# Don't mess doc/
|
|
||||||
dontFixup = true;
|
|
||||||
};
|
|
||||||
in
|
|
||||||
{
|
|
||||||
networking.firewall.allowedTCPPorts = [ 80 ];
|
|
||||||
services.nginx = {
|
|
||||||
enable = true;
|
|
||||||
virtualHosts."jungle.bsc.es" = {
|
|
||||||
root = "${website}";
|
|
||||||
listen = [
|
|
||||||
{
|
|
||||||
addr = "0.0.0.0";
|
|
||||||
port = 80;
|
|
||||||
}
|
|
||||||
];
|
|
||||||
extraConfig = ''
|
|
||||||
set_real_ip_from 127.0.0.1;
|
|
||||||
set_real_ip_from 84.88.52.107;
|
|
||||||
real_ip_recursive on;
|
|
||||||
real_ip_header X-Forwarded-For;
|
|
||||||
|
|
||||||
location /git {
|
|
||||||
rewrite ^/git$ / break;
|
|
||||||
rewrite ^/git/(.*) /$1 break;
|
|
||||||
proxy_pass http://127.0.0.1:3000;
|
|
||||||
proxy_redirect http:// $scheme://;
|
|
||||||
}
|
|
||||||
location /cache {
|
|
||||||
rewrite ^/cache/(.*) /$1 break;
|
|
||||||
proxy_pass http://127.0.0.1:5000;
|
|
||||||
proxy_redirect http:// $scheme://;
|
|
||||||
}
|
|
||||||
location /lists {
|
|
||||||
proxy_pass http://127.0.0.1:8081;
|
|
||||||
proxy_redirect http:// $scheme://;
|
|
||||||
}
|
|
||||||
location /grafana {
|
|
||||||
proxy_pass http://127.0.0.1:2342;
|
|
||||||
proxy_redirect http:// $scheme://;
|
|
||||||
proxy_set_header Host $host;
|
|
||||||
# Websockets
|
|
||||||
proxy_http_version 1.1;
|
|
||||||
proxy_set_header Upgrade $http_upgrade;
|
|
||||||
proxy_set_header Connection "upgrade";
|
|
||||||
}
|
|
||||||
location ~ ^/~(.+?)(/.*)?$ {
|
|
||||||
alias /ceph/home/$1/public_html$2;
|
|
||||||
index index.html index.htm;
|
|
||||||
autoindex on;
|
|
||||||
absolute_redirect off;
|
|
||||||
}
|
|
||||||
location /p/ {
|
|
||||||
alias /ceph/p/;
|
|
||||||
}
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,16 +0,0 @@
|
|||||||
{ config, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
age.secrets.nixServe.file = ../../secrets/nix-serve.age;
|
|
||||||
|
|
||||||
services.nix-serve = {
|
|
||||||
enable = true;
|
|
||||||
# Only listen locally, as we serve it via ssh
|
|
||||||
bindAddress = "127.0.0.1";
|
|
||||||
port = 5000;
|
|
||||||
|
|
||||||
secretKeyFile = config.age.secrets.nixServe.path;
|
|
||||||
# Public key:
|
|
||||||
# jungle.bsc.es:pEc7MlAT0HEwLQYPtpkPLwRsGf80ZI26aj29zMw/HH0=
|
|
||||||
};
|
|
||||||
}
|
|
||||||
43
m/hut/p.nix
43
m/hut/p.nix
@ -1,43 +0,0 @@
|
|||||||
{ pkgs, lib, config, ... }:
|
|
||||||
let
|
|
||||||
p = pkgs.writeShellScriptBin "p" ''
|
|
||||||
set -e
|
|
||||||
cd /ceph
|
|
||||||
pastedir="p/$USER"
|
|
||||||
mkdir -p "$pastedir"
|
|
||||||
|
|
||||||
ext="txt"
|
|
||||||
|
|
||||||
if [ -n "$1" ]; then
|
|
||||||
ext="$1"
|
|
||||||
fi
|
|
||||||
|
|
||||||
out=$(mktemp "$pastedir/XXXXXXXX.$ext")
|
|
||||||
|
|
||||||
cat > "$out"
|
|
||||||
chmod go+r "$out"
|
|
||||||
echo "https://jungle.bsc.es/$out"
|
|
||||||
'';
|
|
||||||
in
|
|
||||||
{
|
|
||||||
environment.systemPackages = with pkgs; [ p ];
|
|
||||||
|
|
||||||
# Make sure we have a directory per user. We cannot use the nice
|
|
||||||
# systemd-tmpfiles-setup.service service because this is a remote FS, and it
|
|
||||||
# may not be mounted when it runs.
|
|
||||||
systemd.services.create-paste-dirs = let
|
|
||||||
# Take only normal users in hut
|
|
||||||
users = lib.filterAttrs (_: v: v.isNormalUser) config.users.users;
|
|
||||||
commands = lib.concatLists (lib.mapAttrsToList
|
|
||||||
(_: user: [
|
|
||||||
"install -d -o ${user.name} -g ${user.group} -m 0755 /ceph/p/${user.name}"
|
|
||||||
]) users);
|
|
||||||
script = pkgs.writeShellScript "create-paste-dirs.sh" (lib.concatLines commands);
|
|
||||||
in {
|
|
||||||
enable = true;
|
|
||||||
wants = [ "remote-fs.target" ];
|
|
||||||
after = [ "remote-fs.target" ];
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
serviceConfig.ExecStart = script;
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,19 +0,0 @@
|
|||||||
{ lib, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
services.postgresql = {
|
|
||||||
enable = true;
|
|
||||||
ensureDatabases = [ "perftestsdb" ];
|
|
||||||
ensureUsers = [
|
|
||||||
{ name = "anavarro"; ensureClauses.superuser = true; }
|
|
||||||
{ name = "rarias"; ensureClauses.superuser = true; }
|
|
||||||
{ name = "grafana"; }
|
|
||||||
];
|
|
||||||
authentication = ''
|
|
||||||
#type database DBuser auth-method
|
|
||||||
local perftestsdb rarias trust
|
|
||||||
local perftestsdb anavarro trust
|
|
||||||
local perftestsdb grafana trust
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,79 +0,0 @@
|
|||||||
/*
|
|
||||||
* CC0-1.0 <https://creativecommons.org/publicdomain/zero/1.0/legalcode>
|
|
||||||
* Dark color scheme using 216 web-safe colors, inspired
|
|
||||||
* somewhat by the default color scheme in mutt.
|
|
||||||
* It reduces eyestrain for me, and energy usage for all:
|
|
||||||
* https://en.wikipedia.org/wiki/Light-on-dark_color_scheme
|
|
||||||
*/
|
|
||||||
|
|
||||||
* {
|
|
||||||
font-size: 14px;
|
|
||||||
font-family: monospace;
|
|
||||||
}
|
|
||||||
|
|
||||||
pre {
|
|
||||||
white-space: pre-wrap;
|
|
||||||
padding: 10px;
|
|
||||||
background: #f5f5f5;
|
|
||||||
}
|
|
||||||
|
|
||||||
hr {
|
|
||||||
margin: 30px 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
body {
|
|
||||||
max-width: 120ex; /* 120 columns wide */
|
|
||||||
margin: 50px auto;
|
|
||||||
}
|
|
||||||
|
|
||||||
/*
|
|
||||||
* Underlined links add visual noise which make them hard-to-read.
|
|
||||||
* Use colors to make them stand out, instead.
|
|
||||||
*/
|
|
||||||
a:link {
|
|
||||||
color: #007;
|
|
||||||
text-decoration: none;
|
|
||||||
}
|
|
||||||
a:visited {
|
|
||||||
color:#504;
|
|
||||||
}
|
|
||||||
a:hover {
|
|
||||||
text-decoration: underline;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* quoted text in emails gets a different color */
|
|
||||||
*.q { color:gray }
|
|
||||||
|
|
||||||
/*
|
|
||||||
* these may be used with cgit <https://git.zx2c4.com/cgit/>, too.
|
|
||||||
* (cgit uses <div>, public-inbox uses <span>)
|
|
||||||
*/
|
|
||||||
*.add { color:darkgreen } /* diff post-image lines */
|
|
||||||
*.del { color:darkred } /* diff pre-image lines */
|
|
||||||
*.head { color:black } /* diff header (metainformation) */
|
|
||||||
*.hunk { color:gray } /* diff hunk-header */
|
|
||||||
|
|
||||||
/*
|
|
||||||
* highlight 3.x colors (tested 3.18) for displaying blobs.
|
|
||||||
* This doesn't use most of the colors available, as I find too
|
|
||||||
* many colors overwhelming, so the default is commented out.
|
|
||||||
*/
|
|
||||||
.hl.num { color:#f30 } /* number */
|
|
||||||
.hl.esc { color:#f0f } /* escape character */
|
|
||||||
.hl.str { color:#f30 } /* string */
|
|
||||||
.hl.ppc { color:#f0f } /* preprocessor */
|
|
||||||
.hl.pps { color:#f30 } /* preprocessor string */
|
|
||||||
.hl.slc { color:#09f } /* single-line comment */
|
|
||||||
.hl.com { color:#09f } /* multi-line comment */
|
|
||||||
/* .hl.opt { color:#ccc } */ /* operator */
|
|
||||||
/* .hl.ipl { color:#ccc } */ /* interpolation */
|
|
||||||
|
|
||||||
/* keyword groups kw[a-z] */
|
|
||||||
.hl.kwa { color:#ff0 }
|
|
||||||
.hl.kwb { color:#0f0 }
|
|
||||||
.hl.kwc { color:#ff0 }
|
|
||||||
/* .hl.kwd { color:#ccc } */
|
|
||||||
|
|
||||||
/* line-number (unused by public-inbox) */
|
|
||||||
/* .hl.lin { color:#ccc } */
|
|
||||||
|
|
||||||
@ -1,47 +0,0 @@
|
|||||||
{ lib, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
services.public-inbox = {
|
|
||||||
enable = true;
|
|
||||||
http = {
|
|
||||||
enable = true;
|
|
||||||
port = 8081;
|
|
||||||
mounts = [ "/lists" ];
|
|
||||||
};
|
|
||||||
settings.publicinbox = {
|
|
||||||
css = [ "${./public-inbox.css}" ];
|
|
||||||
wwwlisting = "all";
|
|
||||||
};
|
|
||||||
inboxes = {
|
|
||||||
bscpkgs = {
|
|
||||||
url = "https://jungle.bsc.es/lists/bscpkgs";
|
|
||||||
address = [ "~rodarima/bscpkgs@lists.sr.ht" ];
|
|
||||||
watch = [ "imaps://jungle-robot%40gmx.com@imap.gmx.com/INBOX" ];
|
|
||||||
description = "Patches for bscpkgs";
|
|
||||||
listid = "~rodarima/bscpkgs.lists.sr.ht";
|
|
||||||
};
|
|
||||||
jungle = {
|
|
||||||
url = "https://jungle.bsc.es/lists/jungle";
|
|
||||||
address = [ "~rodarima/jungle@lists.sr.ht" ];
|
|
||||||
watch = [ "imaps://jungle-robot%40gmx.com@imap.gmx.com/INBOX" ];
|
|
||||||
description = "Patches for jungle";
|
|
||||||
listid = "~rodarima/jungle.lists.sr.ht";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# We need access to the network for the watch service, as we will fetch the
|
|
||||||
# emails directly from the IMAP server.
|
|
||||||
systemd.services.public-inbox-watch.serviceConfig = {
|
|
||||||
PrivateNetwork = lib.mkForce false;
|
|
||||||
RestrictAddressFamilies = lib.mkForce [ "AF_UNIX" "AF_INET" "AF_INET6" ];
|
|
||||||
KillSignal = "SIGKILL"; # Avoid slow shutdown
|
|
||||||
|
|
||||||
# Required for chmod(..., 02750) on directories by git, from
|
|
||||||
# systemd.exec(8):
|
|
||||||
# > Note that this restricts marking of any type of file system object with
|
|
||||||
# > these bits, including both regular files and directories (where the SGID
|
|
||||||
# > is a different meaning than for files, see documentation).
|
|
||||||
RestrictSUIDSGID = lib.mkForce false;
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,35 +0,0 @@
|
|||||||
{ theFlake, pkgs, ... }:
|
|
||||||
|
|
||||||
# This module describes a script that can launch the pixiecore daemon to serve a
|
|
||||||
# NixOS image via PXE to a node to directly boot from there, without requiring a
|
|
||||||
# working disk.
|
|
||||||
|
|
||||||
let
|
|
||||||
# The host config must have the netboot-minimal.nix module too
|
|
||||||
host = theFlake.nixosConfigurations.lake2;
|
|
||||||
sys = host.config.system;
|
|
||||||
build = sys.build;
|
|
||||||
kernel = "${build.kernel}/bzImage";
|
|
||||||
initrd = "${build.netbootRamdisk}/initrd";
|
|
||||||
init = "${build.toplevel}/init";
|
|
||||||
|
|
||||||
script = pkgs.writeShellScriptBin "pixiecore-helper" ''
|
|
||||||
#!/usr/bin/env bash -x
|
|
||||||
|
|
||||||
${pkgs.pixiecore}/bin/pixiecore \
|
|
||||||
boot ${kernel} ${initrd} --cmdline "init=${init} loglevel=4" \
|
|
||||||
--debug --dhcp-no-bind --port 64172 --status-port 64172 "$@"
|
|
||||||
'';
|
|
||||||
in
|
|
||||||
{
|
|
||||||
## We need a DHCP server to provide the IP
|
|
||||||
#services.dnsmasq = {
|
|
||||||
# enable = true;
|
|
||||||
# settings = {
|
|
||||||
# domain-needed = true;
|
|
||||||
# dhcp-range = [ "192.168.0.2,192.168.0.254" ];
|
|
||||||
# };
|
|
||||||
#};
|
|
||||||
|
|
||||||
environment.systemPackages = [ script ];
|
|
||||||
}
|
|
||||||
@ -1,15 +0,0 @@
|
|||||||
- targets:
|
|
||||||
- owl1-ipmi
|
|
||||||
- owl2-ipmi
|
|
||||||
- xeon03-ipmi
|
|
||||||
- xeon04-ipmi
|
|
||||||
- koro-ipmi
|
|
||||||
- weasel-ipmi
|
|
||||||
- hut-ipmi
|
|
||||||
- eudy-ipmi
|
|
||||||
# Storage
|
|
||||||
- bay-ipmi
|
|
||||||
- oss01-ipmi
|
|
||||||
- lake2-ipmi
|
|
||||||
labels:
|
|
||||||
job: ipmi-lan
|
|
||||||
@ -1,35 +0,0 @@
|
|||||||
{ config, pkgs, lib, modulesPath, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../common/ssf.nix
|
|
||||||
#(modulesPath + "/installer/netboot/netboot-minimal.nix")
|
|
||||||
|
|
||||||
../eudy/cpufreq.nix
|
|
||||||
../eudy/users.nix
|
|
||||||
./kernel.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
# Select this using the ID to avoid mismatches
|
|
||||||
boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d5376d2";
|
|
||||||
|
|
||||||
# disable automatic garbage collector
|
|
||||||
nix.gc.automatic = lib.mkForce false;
|
|
||||||
|
|
||||||
# members of the tracing group can use the lttng-provided kernel events
|
|
||||||
# without root permissions
|
|
||||||
users.groups.tracing.members = [ "arocanon" "vlopez" ];
|
|
||||||
|
|
||||||
# set up both ethernet and infiniband ips
|
|
||||||
networking = {
|
|
||||||
hostName = "koro";
|
|
||||||
interfaces.eno1.ipv4.addresses = [ {
|
|
||||||
address = "10.0.40.5";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
interfaces.ibp5s0.ipv4.addresses = [ {
|
|
||||||
address = "10.0.42.5";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,70 +0,0 @@
|
|||||||
{ pkgs, lib, ... }:
|
|
||||||
|
|
||||||
let
|
|
||||||
#fcs-devel = pkgs.linuxPackages_custom {
|
|
||||||
# version = "6.2.8";
|
|
||||||
# src = /mnt/data/kernel/fcs/kernel/src;
|
|
||||||
# configfile = /mnt/data/kernel/fcs/kernel/configs/defconfig;
|
|
||||||
#};
|
|
||||||
|
|
||||||
#fcsv1 = fcs-kernel "bc11660676d3d68ce2459b9fb5d5e654e3f413be" false;
|
|
||||||
#fcsv2 = fcs-kernel "db0f2eca0cd57a58bf456d7d2c7d5d8fdb25dfb1" false;
|
|
||||||
#fcsv1-lockdep = fcs-kernel "bc11660676d3d68ce2459b9fb5d5e654e3f413be" true;
|
|
||||||
#fcsv2-lockdep = fcs-kernel "db0f2eca0cd57a58bf456d7d2c7d5d8fdb25dfb1" true;
|
|
||||||
#fcs-kernel = gitCommit: lockdep: pkgs.linuxPackages_custom {
|
|
||||||
# version = "6.2.8";
|
|
||||||
# src = builtins.fetchGit {
|
|
||||||
# url = "git@bscpm03.bsc.es:ompss-kernel/linux.git";
|
|
||||||
# rev = gitCommit;
|
|
||||||
# ref = "fcs";
|
|
||||||
# };
|
|
||||||
# configfile = if lockdep then ./configs/lockdep else ./configs/defconfig;
|
|
||||||
#};
|
|
||||||
|
|
||||||
kernel = nixos-fcs;
|
|
||||||
|
|
||||||
nixos-fcs-kernel = lib.makeOverridable ({gitCommit, lockStat ? false, preempt ? false, branch ? "fcs"}: pkgs.linuxPackagesFor (pkgs.buildLinux rec {
|
|
||||||
version = "6.2.8";
|
|
||||||
src = builtins.fetchGit {
|
|
||||||
url = "git@bscpm03.bsc.es:ompss-kernel/linux.git";
|
|
||||||
rev = gitCommit;
|
|
||||||
ref = branch;
|
|
||||||
};
|
|
||||||
structuredExtraConfig = with lib.kernel; {
|
|
||||||
# add general custom kernel options here
|
|
||||||
} // lib.optionalAttrs lockStat {
|
|
||||||
LOCK_STAT = yes;
|
|
||||||
} // lib.optionalAttrs preempt {
|
|
||||||
PREEMPT = lib.mkForce yes;
|
|
||||||
PREEMPT_VOLUNTARY = lib.mkForce no;
|
|
||||||
};
|
|
||||||
kernelPatches = [];
|
|
||||||
extraMeta.branch = lib.versions.majorMinor version;
|
|
||||||
}));
|
|
||||||
|
|
||||||
nixos-fcs = nixos-fcs-kernel {gitCommit = "8a09822dfcc8f0626b209d6d2aec8b5da459dfee";};
|
|
||||||
nixos-fcs-lockstat = nixos-fcs.override {
|
|
||||||
lockStat = true;
|
|
||||||
};
|
|
||||||
nixos-fcs-lockstat-preempt = nixos-fcs.override {
|
|
||||||
lockStat = true;
|
|
||||||
preempt = true;
|
|
||||||
};
|
|
||||||
latest = pkgs.linuxPackages_latest;
|
|
||||||
|
|
||||||
in {
|
|
||||||
imports = [
|
|
||||||
../eudy/kernel/lttng.nix
|
|
||||||
../eudy/kernel/perf.nix
|
|
||||||
];
|
|
||||||
boot.kernelPackages = lib.mkForce kernel;
|
|
||||||
|
|
||||||
# disable all cpu mitigations
|
|
||||||
boot.kernelParams = [
|
|
||||||
"mitigations=off"
|
|
||||||
];
|
|
||||||
|
|
||||||
# enable memory overcommit, needed to build a taglibc system using nix after
|
|
||||||
# increasing the openblas memory footprint
|
|
||||||
boot.kernel.sysctl."vm.overcommit_memory" = 1;
|
|
||||||
}
|
|
||||||
@ -1,84 +0,0 @@
|
|||||||
{ config, pkgs, lib, modulesPath, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../common/ssf.nix
|
|
||||||
../module/monitoring.nix
|
|
||||||
../module/hut-substituter.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d53563a";
|
|
||||||
|
|
||||||
boot.kernel.sysctl = {
|
|
||||||
"kernel.yama.ptrace_scope" = lib.mkForce "1";
|
|
||||||
};
|
|
||||||
|
|
||||||
environment.systemPackages = with pkgs; [
|
|
||||||
ceph
|
|
||||||
];
|
|
||||||
|
|
||||||
services.ceph = {
|
|
||||||
enable = true;
|
|
||||||
global = {
|
|
||||||
fsid = "9c8d06e0-485f-4aaf-b16b-06d6daf1232b";
|
|
||||||
monHost = "10.0.40.40";
|
|
||||||
monInitialMembers = "bay";
|
|
||||||
clusterNetwork = "10.0.40.40/24"; # Use Ethernet only
|
|
||||||
};
|
|
||||||
osd = {
|
|
||||||
enable = true;
|
|
||||||
# One daemon per NVME disk
|
|
||||||
daemons = [ "4" "5" "6" "7" ];
|
|
||||||
extraConfig = {
|
|
||||||
"osd crush chooseleaf type" = "0";
|
|
||||||
"osd journal size" = "10000";
|
|
||||||
"osd pool default min size" = "2";
|
|
||||||
"osd pool default pg num" = "200";
|
|
||||||
"osd pool default pgp num" = "200";
|
|
||||||
"osd pool default size" = "3";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
networking = {
|
|
||||||
hostName = "lake2";
|
|
||||||
interfaces.eno1.ipv4.addresses = [ {
|
|
||||||
address = "10.0.40.42";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
interfaces.ibp5s0.ipv4.addresses = [ {
|
|
||||||
address = "10.0.42.42";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
firewall = {
|
|
||||||
extraCommands = ''
|
|
||||||
# Accept all incoming TCP traffic from bay
|
|
||||||
iptables -A nixos-fw -p tcp -s bay -j nixos-fw-accept
|
|
||||||
# Accept monitoring requests from hut
|
|
||||||
iptables -A nixos-fw -p tcp -s hut --dport 9002 -j nixos-fw-accept
|
|
||||||
# Accept all Ceph traffic from the local network
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.0/24 -m multiport --dport 3300,6789,6800:7568 -j nixos-fw-accept
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# Missing service for volumes, see:
|
|
||||||
# https://www.reddit.com/r/ceph/comments/14otjyo/comment/jrd69vt/
|
|
||||||
systemd.services.ceph-volume = {
|
|
||||||
enable = true;
|
|
||||||
description = "Ceph Volume activation";
|
|
||||||
unitConfig = {
|
|
||||||
Type = "oneshot";
|
|
||||||
After = "local-fs.target";
|
|
||||||
Wants = "local-fs.target";
|
|
||||||
};
|
|
||||||
path = [ pkgs.ceph pkgs.util-linux pkgs.lvm2 pkgs.cryptsetup ];
|
|
||||||
serviceConfig = {
|
|
||||||
KillMode = "none";
|
|
||||||
Environment = "CEPH_VOLUME_TIMEOUT=10000";
|
|
||||||
ExecStart = "/bin/sh -c 'timeout $CEPH_VOLUME_TIMEOUT ${pkgs.ceph}/bin/ceph-volume lvm activate --all --no-systemd'";
|
|
||||||
TimeoutSec = "0";
|
|
||||||
};
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
70
m/map.nix
70
m/map.nix
@ -1,70 +0,0 @@
|
|||||||
{
|
|
||||||
# In physical order from top to bottom (see note below)
|
|
||||||
ssf = {
|
|
||||||
# Switches for Ethernet and OmniPath
|
|
||||||
switch-C6-S1A-05 = { pos=42; size=1; model="Dell S3048-ON"; };
|
|
||||||
switch-opa = { pos=41; size=1; };
|
|
||||||
|
|
||||||
# SSF login
|
|
||||||
apex = { pos=39; size=2; label="SSFHEAD"; board="R2208WTTYSR"; contact="rodrigo.arias@bsc.es"; };
|
|
||||||
|
|
||||||
# Storage
|
|
||||||
bay = { pos=38; size=1; label="MDS01"; board="S2600WT2R"; sn="BQWL64850303"; contact="rodrigo.arias@bsc.es"; };
|
|
||||||
lake1 = { pos=37; size=1; label="OSS01"; board="S2600WT2R"; sn="BQWL64850234"; contact="rodrigo.arias@bsc.es"; };
|
|
||||||
lake2 = { pos=36; size=1; label="OSS02"; board="S2600WT2R"; sn="BQWL64850266"; contact="rodrigo.arias@bsc.es"; };
|
|
||||||
|
|
||||||
# Compute xeon
|
|
||||||
owl1 = { pos=35; size=1; label="SSF-XEON01"; board="S2600WTTR"; sn="BQWL64954172"; contact="rodrigo.arias@bsc.es"; };
|
|
||||||
owl2 = { pos=34; size=1; label="SSF-XEON02"; board="S2600WTTR"; sn="BQWL64756560"; contact="rodrigo.arias@bsc.es"; };
|
|
||||||
xeon03 = { pos=33; size=1; label="SSF-XEON03"; board="S2600WTTR"; sn="BQWL64750826"; contact="rodrigo.arias@bsc.es"; };
|
|
||||||
# Slot 34 empty
|
|
||||||
koro = { pos=31; size=1; label="SSF-XEON05"; board="S2600WTTR"; sn="BQWL64954293"; contact="rodrigo.arias@bsc.es"; };
|
|
||||||
weasel = { pos=30; size=1; label="SSF-XEON06"; board="S2600WTTR"; sn="BQWL64750846"; contact="antoni.navarro@bsc.es"; };
|
|
||||||
hut = { pos=29; size=1; label="SSF-XEON07"; board="S2600WTTR"; sn="BQWL64751184"; contact="rodrigo.arias@bsc.es"; };
|
|
||||||
eudy = { pos=28; size=1; label="SSF-XEON08"; board="S2600WTTR"; sn="BQWL64756586"; contact="aleix.rocanonell@bsc.es"; };
|
|
||||||
|
|
||||||
# 16 KNL nodes, 4 per chassis
|
|
||||||
knl01_04 = { pos=26; size=2; label="KNL01..KNL04"; board="HNS7200APX"; };
|
|
||||||
knl05_08 = { pos=24; size=2; label="KNL05..KNL18"; board="HNS7200APX"; };
|
|
||||||
knl09_12 = { pos=22; size=2; label="KNL09..KNL12"; board="HNS7200APX"; };
|
|
||||||
knl13_16 = { pos=20; size=2; label="KNL13..KNL16"; board="HNS7200APX"; };
|
|
||||||
|
|
||||||
# Slot 19 empty
|
|
||||||
|
|
||||||
# EPI (hw team, guessed order)
|
|
||||||
epi01 = { pos=18; size=1; contact="joan.cabre@bsc.es"; };
|
|
||||||
epi02 = { pos=17; size=1; contact="joan.cabre@bsc.es"; };
|
|
||||||
epi03 = { pos=16; size=1; contact="joan.cabre@bsc.es"; };
|
|
||||||
anon = { pos=14; size=2; }; # Unlabeled machine. Operative
|
|
||||||
|
|
||||||
# These are old and decommissioned (off)
|
|
||||||
power8 = { pos=12; size=2; label="BSCPOWER8N3"; decommissioned=true; };
|
|
||||||
powern1 = { pos=8; size=4; label="BSCPOWERN1"; decommissioned=true; };
|
|
||||||
gustafson = { pos=7; size=1; label="gustafson"; decommissioned=true; };
|
|
||||||
odap01 = { pos=3; size=4; label="ODAP01"; decommissioned=true; };
|
|
||||||
amhdal = { pos=2; size=1; label="AMHDAL"; decommissioned=true; }; # sic
|
|
||||||
moore = { pos=1; size=1; label="moore (earth)"; decommissioned=true; };
|
|
||||||
};
|
|
||||||
|
|
||||||
bsc2218 = {
|
|
||||||
raccoon = { board="W2600CR"; sn="QSIP22500829"; contact="rodrigo.arias@bsc.es"; };
|
|
||||||
tent = { label="SSF-XEON04"; board="S2600WTTR"; sn="BQWL64751229"; contact="rodrigo.arias@bsc.es"; };
|
|
||||||
};
|
|
||||||
|
|
||||||
upc = {
|
|
||||||
fox = { board="H13DSG-O-CPU"; sn="UM24CS600392"; prod="AS-4125GS-TNRT"; prod_sn="E508839X5103339"; contact="rodrigo.arias@bsc.es"; };
|
|
||||||
};
|
|
||||||
|
|
||||||
# NOTE: Position is specified in "U" units (44.45 mm) and starts at 1 from the
|
|
||||||
# bottom. Example:
|
|
||||||
#
|
|
||||||
# | ... | - [pos+size] <--- Label in chassis
|
|
||||||
# +--------+
|
|
||||||
# | node | - [pos+1]
|
|
||||||
# | 2U | - [pos]
|
|
||||||
# +------- +
|
|
||||||
# | ... | - [pos-1]
|
|
||||||
#
|
|
||||||
# NOTE: The board and sn refers to the FRU information (Board Product and
|
|
||||||
# Board Serial) via `ipmitool fru print 0`.
|
|
||||||
}
|
|
||||||
@ -1,49 +0,0 @@
|
|||||||
{ config, lib, pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
options = {
|
|
||||||
services.amd-uprof = {
|
|
||||||
enable = lib.mkOption {
|
|
||||||
type = lib.types.bool;
|
|
||||||
default = false;
|
|
||||||
description = "Whether to enable AMD uProf.";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# Only setup amd-uprof if enabled
|
|
||||||
config = lib.mkIf config.services.amd-uprof.enable {
|
|
||||||
|
|
||||||
# First make sure that we add the module to the list of available modules
|
|
||||||
# in the kernel matching the same kernel version of this configuration.
|
|
||||||
boot.extraModulePackages = with config.boot.kernelPackages; [ amd-uprof-driver ];
|
|
||||||
boot.kernelModules = [ "AMDPowerProfiler" ];
|
|
||||||
|
|
||||||
# Make the userspace tools available in $PATH.
|
|
||||||
environment.systemPackages = with pkgs; [ amd-uprof ];
|
|
||||||
|
|
||||||
# The AMDPowerProfiler module doesn't create the /dev device nor it emits
|
|
||||||
# any uevents, so we cannot use udev rules to automatically create the
|
|
||||||
# device. Instead, we run a systemd unit that does it after loading the
|
|
||||||
# modules.
|
|
||||||
systemd.services.amd-uprof-device = {
|
|
||||||
description = "Create /dev/AMDPowerProfiler device";
|
|
||||||
after = [ "systemd-modules-load.service" ];
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
unitConfig.ConditionPathExists = [
|
|
||||||
"/proc/AMDPowerProfiler/device"
|
|
||||||
"!/dev/AMDPowerProfiler"
|
|
||||||
];
|
|
||||||
serviceConfig = {
|
|
||||||
Type = "oneshot";
|
|
||||||
RemainAfterExit = true;
|
|
||||||
ExecStart = pkgs.writeShellScript "add-amd-uprof-dev.sh" ''
|
|
||||||
mknod /dev/AMDPowerProfiler -m 666 c $(< /proc/AMDPowerProfiler/device) 0
|
|
||||||
'';
|
|
||||||
ExecStop = pkgs.writeShellScript "remove-amd-uprof-dev.sh" ''
|
|
||||||
rm -f /dev/AMDPowerProfiler
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,24 +0,0 @@
|
|||||||
{ config, pkgs, ... }:
|
|
||||||
|
|
||||||
# Mounts the /ceph filesystem at boot
|
|
||||||
{
|
|
||||||
environment.systemPackages = with pkgs; [
|
|
||||||
ceph-client
|
|
||||||
fio # For benchmarks
|
|
||||||
];
|
|
||||||
|
|
||||||
# We need the ceph module loaded as the mount.ceph binary fails to run the
|
|
||||||
# modprobe command.
|
|
||||||
boot.kernelModules = [ "ceph" ];
|
|
||||||
|
|
||||||
age.secrets.cephUser.file = ../../secrets/ceph-user.age;
|
|
||||||
|
|
||||||
fileSystems."/ceph" = {
|
|
||||||
fsType = "ceph";
|
|
||||||
device = "user@9c8d06e0-485f-4aaf-b16b-06d6daf1232b.cephfs=/";
|
|
||||||
options = [
|
|
||||||
"mon_addr=10.0.40.40"
|
|
||||||
"secretfile=${config.age.secrets.cephUser.path}"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,3 +0,0 @@
|
|||||||
{
|
|
||||||
services.nixseparatedebuginfod.enable = true;
|
|
||||||
}
|
|
||||||
@ -1,3 +0,0 @@
|
|||||||
{
|
|
||||||
boot.binfmt.emulatedSystems = [ "armv7l-linux" "aarch64-linux" "powerpc64le-linux" "riscv64-linux" ];
|
|
||||||
}
|
|
||||||
@ -1,13 +0,0 @@
|
|||||||
{ config, ... }:
|
|
||||||
{
|
|
||||||
nix.settings =
|
|
||||||
# Don't add hut as a cache to itself
|
|
||||||
assert config.networking.hostName != "hut";
|
|
||||||
{
|
|
||||||
extra-substituters = [ "http://hut/cache" ];
|
|
||||||
extra-trusted-public-keys = [ "jungle.bsc.es:pEc7MlAT0HEwLQYPtpkPLwRsGf80ZI26aj29zMw/HH0=" ];
|
|
||||||
|
|
||||||
# Set a low timeout in case hut is down
|
|
||||||
connect-timeout = 3; # seconds
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,24 +0,0 @@
|
|||||||
{ config, lib, ... }:
|
|
||||||
|
|
||||||
with lib;
|
|
||||||
|
|
||||||
{
|
|
||||||
options = {
|
|
||||||
users.jungleUsers = mkOption {
|
|
||||||
type = types.attrsOf (types.anything // { check = (x: x ? "hosts"); });
|
|
||||||
description = ''
|
|
||||||
Same as users.users but with the extra `hosts` attribute, which controls
|
|
||||||
access to the nodes by `networking.hostName`.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
config = let
|
|
||||||
allowedUser = host: userConf: builtins.elem host userConf.hosts;
|
|
||||||
filterUsers = host: users: filterAttrs (n: v: allowedUser host v) users;
|
|
||||||
removeHosts = users: mapAttrs (n: v: builtins.removeAttrs v [ "hosts" ]) users;
|
|
||||||
currentHost = config.networking.hostName;
|
|
||||||
in {
|
|
||||||
users.users = removeHosts (filterUsers currentHost config.users.jungleUsers);
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,17 +0,0 @@
|
|||||||
{ config, lib, pkgs, ... }:
|
|
||||||
|
|
||||||
with lib;
|
|
||||||
|
|
||||||
{
|
|
||||||
systemd.services."prometheus-meteocat-exporter" = {
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
after = [ "network.target" ];
|
|
||||||
serviceConfig = {
|
|
||||||
Restart = mkDefault "always";
|
|
||||||
PrivateTmp = mkDefault true;
|
|
||||||
WorkingDirectory = mkDefault "/tmp";
|
|
||||||
DynamicUser = mkDefault true;
|
|
||||||
ExecStart = "${pkgs.meteocat-exporter}/bin/meteocat-exporter";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,25 +0,0 @@
|
|||||||
{ config, lib, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
# We need access to the devices to monitor the disk space
|
|
||||||
systemd.services.prometheus-node-exporter.serviceConfig.PrivateDevices = lib.mkForce false;
|
|
||||||
systemd.services.prometheus-node-exporter.serviceConfig.ProtectHome = lib.mkForce "read-only";
|
|
||||||
|
|
||||||
# Required to allow the smartctl exporter to read the nvme0 character device,
|
|
||||||
# see the commit message on:
|
|
||||||
# https://github.com/NixOS/nixpkgs/commit/12c26aca1fd55ab99f831bedc865a626eee39f80
|
|
||||||
services.udev.extraRules = ''
|
|
||||||
SUBSYSTEM=="nvme", KERNEL=="nvme[0-9]*", GROUP="disk"
|
|
||||||
'';
|
|
||||||
|
|
||||||
services.prometheus = {
|
|
||||||
exporters = {
|
|
||||||
node = {
|
|
||||||
enable = true;
|
|
||||||
enabledCollectors = [ "systemd" ];
|
|
||||||
port = 9002;
|
|
||||||
};
|
|
||||||
smartctl.enable = true;
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,26 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
|
|
||||||
# Locate nix daemon pid
|
|
||||||
nd=$(pgrep -o nix-daemon)
|
|
||||||
|
|
||||||
# Locate children of nix-daemon
|
|
||||||
pids1=$(tr ' ' '\n' < "/proc/$nd/task/$nd/children")
|
|
||||||
|
|
||||||
# For each children, locate 2nd level children
|
|
||||||
pids2=$(echo "$pids1" | xargs -I @ /bin/sh -c 'cat /proc/@/task/*/children' | tr ' ' '\n')
|
|
||||||
|
|
||||||
cat <<EOF
|
|
||||||
HTTP/1.1 200 OK
|
|
||||||
Content-Type: text/plain; version=0.0.4; charset=utf-8; escaping=values
|
|
||||||
|
|
||||||
# HELP nix_daemon_build Nix daemon derivation build state.
|
|
||||||
# TYPE nix_daemon_build gauge
|
|
||||||
EOF
|
|
||||||
|
|
||||||
for pid in $pids2; do
|
|
||||||
name=$(cat /proc/$pid/environ 2>/dev/null | tr '\0' '\n' | rg "^name=(.+)" - --replace '$1' | tr -dc ' [:alnum:]_\-\.')
|
|
||||||
user=$(ps -o uname= -p "$pid")
|
|
||||||
if [ -n "$name" -a -n "$user" ]; then
|
|
||||||
printf 'nix_daemon_build{user="%s",name="%s"} 1\n' "$user" "$name"
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
@ -1,23 +0,0 @@
|
|||||||
{ pkgs, config, lib, ... }:
|
|
||||||
let
|
|
||||||
script = pkgs.runCommand "nix-daemon-exporter.sh" { }
|
|
||||||
''
|
|
||||||
cp ${./nix-daemon-builds.sh} $out;
|
|
||||||
chmod +x $out
|
|
||||||
''
|
|
||||||
;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
systemd.services.nix-daemon-exporter = {
|
|
||||||
description = "Daemon to export nix-daemon metrics";
|
|
||||||
path = [ pkgs.procps pkgs.ripgrep ];
|
|
||||||
wantedBy = [ "default.target" ];
|
|
||||||
serviceConfig = {
|
|
||||||
Type = "simple";
|
|
||||||
ExecStart = "${pkgs.socat}/bin/socat TCP4-LISTEN:9999,fork EXEC:${script}";
|
|
||||||
# Needed root to read the environment, potentially unsafe
|
|
||||||
User = "root";
|
|
||||||
Group = "root";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,20 +0,0 @@
|
|||||||
{ lib, config, pkgs, ... }:
|
|
||||||
{
|
|
||||||
# Configure Nvidia driver to use with CUDA
|
|
||||||
hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.production;
|
|
||||||
hardware.nvidia.open = lib.mkDefault (builtins.abort "hardware.nvidia.open not set");
|
|
||||||
hardware.graphics.enable = true;
|
|
||||||
nixpkgs.config.nvidia.acceptLicense = true;
|
|
||||||
services.xserver.videoDrivers = [ "nvidia" ];
|
|
||||||
|
|
||||||
# enable support for derivations which require nvidia-gpu to be available
|
|
||||||
# > requiredSystemFeatures = [ "cuda" ];
|
|
||||||
programs.nix-required-mounts.enable = true;
|
|
||||||
programs.nix-required-mounts.presets.nvidia-gpu.enable = true;
|
|
||||||
# They forgot to add the symlink
|
|
||||||
programs.nix-required-mounts.allowedPatterns.nvidia-gpu.paths = [
|
|
||||||
config.systemd.tmpfiles.settings.graphics-driver."/run/opengl-driver"."L+".argument
|
|
||||||
];
|
|
||||||
|
|
||||||
environment.systemPackages = [ pkgs.cudainfo ];
|
|
||||||
}
|
|
||||||
@ -1,68 +0,0 @@
|
|||||||
{ config, lib, pkgs, ... }:
|
|
||||||
|
|
||||||
let
|
|
||||||
cfg = config.services.p;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
options = {
|
|
||||||
services.p = {
|
|
||||||
enable = lib.mkOption {
|
|
||||||
type = lib.types.bool;
|
|
||||||
default = false;
|
|
||||||
description = "Whether to enable the p service.";
|
|
||||||
};
|
|
||||||
path = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "/var/lib/p";
|
|
||||||
description = "Where to save the pasted files on disk.";
|
|
||||||
};
|
|
||||||
url = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "https://jungle.bsc.es/p";
|
|
||||||
description = "URL prefix for the printed file.";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
config = lib.mkIf cfg.enable {
|
|
||||||
environment.systemPackages = let
|
|
||||||
p = pkgs.writeShellScriptBin "p" ''
|
|
||||||
set -e
|
|
||||||
pastedir="${cfg.path}/$USER"
|
|
||||||
cd "$pastedir"
|
|
||||||
|
|
||||||
ext="txt"
|
|
||||||
if [ -n "$1" ]; then
|
|
||||||
ext="$1"
|
|
||||||
fi
|
|
||||||
|
|
||||||
out=$(mktemp "XXXXXXXX.$ext")
|
|
||||||
cat > "$out"
|
|
||||||
chmod go+r "$out"
|
|
||||||
echo "${cfg.url}/$USER/$out"
|
|
||||||
'';
|
|
||||||
in [ p ];
|
|
||||||
|
|
||||||
systemd.services.p = let
|
|
||||||
# Take only normal users
|
|
||||||
users = lib.filterAttrs (_: v: v.isNormalUser) config.users.users;
|
|
||||||
# Create a directory for each user
|
|
||||||
commands = lib.concatLists (lib.mapAttrsToList (_: user: [
|
|
||||||
"install -d -o ${user.name} -g ${user.group} -m 0755 ${cfg.path}/${user.name}"
|
|
||||||
]) users);
|
|
||||||
in {
|
|
||||||
description = "P service setup";
|
|
||||||
requires = [ "network-online.target" ];
|
|
||||||
#wants = [ "remote-fs.target" ];
|
|
||||||
#after = [ "remote-fs.target" ];
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
serviceConfig = {
|
|
||||||
ExecStart = pkgs.writeShellScript "p-init.sh" (''
|
|
||||||
|
|
||||||
install -d -o root -g root -m 0755 ${cfg.path}
|
|
||||||
|
|
||||||
'' + (lib.concatLines commands));
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,33 +0,0 @@
|
|||||||
{ config, lib, pkgs, ... }:
|
|
||||||
|
|
||||||
with lib;
|
|
||||||
|
|
||||||
let
|
|
||||||
cfg = config.power.policy;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
options = {
|
|
||||||
power.policy = mkOption {
|
|
||||||
type = types.nullOr (types.enum [ "always-on" "previous" "always-off" ]);
|
|
||||||
default = null;
|
|
||||||
description = "Set power policy to use via IPMI.";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
config = mkIf (cfg != null) {
|
|
||||||
systemd.services."power-policy" = {
|
|
||||||
description = "Set power policy to use via IPMI";
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
unitConfig = {
|
|
||||||
StartLimitBurst = "10";
|
|
||||||
StartLimitIntervalSec = "10m";
|
|
||||||
};
|
|
||||||
serviceConfig = {
|
|
||||||
ExecStart = "${pkgs.ipmitool}/bin/ipmitool chassis policy ${cfg}";
|
|
||||||
Type = "oneshot";
|
|
||||||
Restart = "on-failure";
|
|
||||||
RestartSec = "5s";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,24 +0,0 @@
|
|||||||
{ lib, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
./slurm-common.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
systemd.services.slurmd.serviceConfig = {
|
|
||||||
# Kill all processes in the control group on stop/restart. This will kill
|
|
||||||
# all the jobs running, so ensure that we only upgrade when the nodes are
|
|
||||||
# not in use. See:
|
|
||||||
# https://github.com/NixOS/nixpkgs/commit/ae93ed0f0d4e7be0a286d1fca86446318c0c6ffb
|
|
||||||
# https://bugs.schedmd.com/show_bug.cgi?id=2095#c24
|
|
||||||
KillMode = lib.mkForce "control-group";
|
|
||||||
|
|
||||||
# If slurmd fails to contact the control server it will fail, causing the
|
|
||||||
# node to remain out of service until manually restarted. Always try to
|
|
||||||
# restart it.
|
|
||||||
Restart = "always";
|
|
||||||
RestartSec = "30s";
|
|
||||||
};
|
|
||||||
|
|
||||||
services.slurm.client.enable = true;
|
|
||||||
}
|
|
||||||
@ -1,115 +0,0 @@
|
|||||||
{ config, pkgs, ... }:
|
|
||||||
|
|
||||||
let
|
|
||||||
suspendProgram = pkgs.writeShellScript "suspend.sh" ''
|
|
||||||
exec 1>>/var/log/power_save.log 2>>/var/log/power_save.log
|
|
||||||
set -x
|
|
||||||
export "PATH=/run/current-system/sw/bin:$PATH"
|
|
||||||
echo "$(date) Suspend invoked $0 $*" >> /var/log/power_save.log
|
|
||||||
hosts=$(scontrol show hostnames $1)
|
|
||||||
for host in $hosts; do
|
|
||||||
echo Shutting down host: $host
|
|
||||||
ipmitool -I lanplus -H ''${host}-ipmi -P "" -U "" chassis power off
|
|
||||||
done
|
|
||||||
'';
|
|
||||||
|
|
||||||
resumeProgram = pkgs.writeShellScript "resume.sh" ''
|
|
||||||
exec 1>>/var/log/power_save.log 2>>/var/log/power_save.log
|
|
||||||
set -x
|
|
||||||
export "PATH=/run/current-system/sw/bin:$PATH"
|
|
||||||
echo "$(date) Suspend invoked $0 $*" >> /var/log/power_save.log
|
|
||||||
hosts=$(scontrol show hostnames $1)
|
|
||||||
for host in $hosts; do
|
|
||||||
echo Starting host: $host
|
|
||||||
ipmitool -I lanplus -H ''${host}-ipmi -P "" -U "" chassis power on
|
|
||||||
done
|
|
||||||
'';
|
|
||||||
|
|
||||||
in {
|
|
||||||
services.slurm = {
|
|
||||||
controlMachine = "apex";
|
|
||||||
clusterName = "jungle";
|
|
||||||
nodeName = [
|
|
||||||
"owl[1,2] Sockets=2 CoresPerSocket=14 ThreadsPerCore=2 Feature=owl"
|
|
||||||
"fox Sockets=8 CoresPerSocket=24 ThreadsPerCore=1"
|
|
||||||
];
|
|
||||||
|
|
||||||
partitionName = [
|
|
||||||
"owl Nodes=owl[1-2] Default=YES DefaultTime=01:00:00 MaxTime=INFINITE State=UP"
|
|
||||||
"fox Nodes=fox Default=NO DefaultTime=01:00:00 MaxTime=INFINITE State=UP"
|
|
||||||
];
|
|
||||||
|
|
||||||
# See slurm.conf(5) for more details about these options.
|
|
||||||
extraConfig = ''
|
|
||||||
# Use PMIx for MPI by default. It works okay with MPICH and OpenMPI, but
|
|
||||||
# not with Intel MPI. For that use the compatibility shim libpmi.so
|
|
||||||
# setting I_MPI_PMI_LIBRARY=$pmix/lib/libpmi.so while maintaining the PMIx
|
|
||||||
# library in SLURM (--mpi=pmix). See more details here:
|
|
||||||
# https://pm.bsc.es/gitlab/rarias/jungle/-/issues/16
|
|
||||||
MpiDefault=pmix
|
|
||||||
|
|
||||||
# When a node reboots return that node to the slurm queue as soon as it
|
|
||||||
# becomes operative again.
|
|
||||||
ReturnToService=2
|
|
||||||
|
|
||||||
# Track all processes by using a cgroup
|
|
||||||
ProctrackType=proctrack/cgroup
|
|
||||||
|
|
||||||
# Enable task/affinity to allow the jobs to run in a specified subset of
|
|
||||||
# the resources. Use the task/cgroup plugin to enable process containment.
|
|
||||||
TaskPlugin=task/affinity,task/cgroup
|
|
||||||
|
|
||||||
# Power off unused nodes until they are requested
|
|
||||||
SuspendProgram=${suspendProgram}
|
|
||||||
SuspendTimeout=60
|
|
||||||
ResumeProgram=${resumeProgram}
|
|
||||||
ResumeTimeout=300
|
|
||||||
SuspendExcNodes=fox
|
|
||||||
|
|
||||||
# Turn the nodes off after 1 hour of inactivity
|
|
||||||
SuspendTime=3600
|
|
||||||
|
|
||||||
# Reduce port range so we can allow only this range in the firewall
|
|
||||||
SrunPortRange=60000-61000
|
|
||||||
|
|
||||||
# Use cores as consumable resources. In SLURM terms, a core may have
|
|
||||||
# multiple hardware threads (or CPUs).
|
|
||||||
SelectType=select/cons_tres
|
|
||||||
|
|
||||||
# Ignore memory constraints and only use unused cores to share a node with
|
|
||||||
# other jobs.
|
|
||||||
SelectTypeParameters=CR_Core
|
|
||||||
|
|
||||||
# Required for pam_slurm_adopt, see https://slurm.schedmd.com/pam_slurm_adopt.html
|
|
||||||
# This sets up the "extern" step into which ssh-launched processes will be
|
|
||||||
# adopted. Alloc runs the prolog at job allocation (salloc) rather than
|
|
||||||
# when a task runs (srun) so we can ssh early.
|
|
||||||
PrologFlags=Alloc,Contain,X11
|
|
||||||
|
|
||||||
# LaunchParameters=ulimit_pam_adopt will set RLIMIT_RSS in processes
|
|
||||||
# adopted by the external step, similar to tasks running in regular steps
|
|
||||||
# LaunchParameters=ulimit_pam_adopt
|
|
||||||
SlurmdDebug=debug5
|
|
||||||
#DebugFlags=Protocol,Cgroup
|
|
||||||
'';
|
|
||||||
|
|
||||||
extraCgroupConfig = ''
|
|
||||||
CgroupPlugin=cgroup/v2
|
|
||||||
#ConstrainCores=yes
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
# Place the slurm config in /etc as this will be required by PAM
|
|
||||||
environment.etc.slurm.source = config.services.slurm.etcSlurm;
|
|
||||||
|
|
||||||
age.secrets.mungeKey = {
|
|
||||||
file = ../../secrets/munge-key.age;
|
|
||||||
owner = "munge";
|
|
||||||
group = "munge";
|
|
||||||
};
|
|
||||||
|
|
||||||
services.munge = {
|
|
||||||
enable = true;
|
|
||||||
password = config.age.secrets.mungeKey.path;
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,28 +0,0 @@
|
|||||||
{ config, lib, pkgs, ... }:
|
|
||||||
|
|
||||||
# See also: https://github.com/NixOS/nixpkgs/pull/112010
|
|
||||||
# And: https://github.com/NixOS/nixpkgs/pull/115839
|
|
||||||
|
|
||||||
with lib;
|
|
||||||
|
|
||||||
{
|
|
||||||
systemd.services."prometheus-slurm-exporter" = {
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
after = [ "network.target" ];
|
|
||||||
serviceConfig = {
|
|
||||||
Restart = mkDefault "always";
|
|
||||||
PrivateTmp = mkDefault true;
|
|
||||||
WorkingDirectory = mkDefault "/tmp";
|
|
||||||
DynamicUser = mkDefault true;
|
|
||||||
ExecStart = ''
|
|
||||||
${pkgs.prometheus-slurm-exporter}/bin/prometheus-slurm-exporter --listen-address "127.0.0.1:9341"
|
|
||||||
'';
|
|
||||||
Environment = [
|
|
||||||
"PATH=${pkgs.slurm}/bin"
|
|
||||||
# We need to specify the slurm config to be able to talk to the slurmd
|
|
||||||
# daemon.
|
|
||||||
"SLURM_CONF=${config.services.slurm.etcSlurm}/slurm.conf"
|
|
||||||
];
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,8 +0,0 @@
|
|||||||
{ ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
networking.firewall = {
|
|
||||||
# Required for PMIx in SLURM, we should find a better way
|
|
||||||
allowedTCPPortRanges = [ { from=1024; to=65535; } ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,19 +0,0 @@
|
|||||||
{ ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
# Mount the hut nix store via NFS
|
|
||||||
fileSystems."/mnt/hut-nix-store" = {
|
|
||||||
device = "hut:/nix/store";
|
|
||||||
fsType = "nfs";
|
|
||||||
options = [ "ro" ];
|
|
||||||
};
|
|
||||||
|
|
||||||
systemd.services.slurmd.serviceConfig = {
|
|
||||||
# When running a job, bind the hut store in /nix/store so the paths are
|
|
||||||
# available too.
|
|
||||||
# FIXME: This doesn't keep the programs in /run/current-system/sw/bin
|
|
||||||
# available in the store. Ideally they should be merged but the overlay FS
|
|
||||||
# doesn't work when the underlying directories change.
|
|
||||||
BindReadOnlyPaths = "/mnt/hut-nix-store:/nix/store";
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,23 +0,0 @@
|
|||||||
{ ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
./slurm-common.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
services.slurm.server.enable = true;
|
|
||||||
|
|
||||||
networking.firewall = {
|
|
||||||
extraCommands = ''
|
|
||||||
# Accept slurm connections to controller from compute nodes
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.0/24 --dport 6817 -j nixos-fw-accept
|
|
||||||
# Accept slurm connections from compute nodes for srun
|
|
||||||
iptables -A nixos-fw -p tcp -s 10.0.40.0/24 --dport 60000:61000 -j nixos-fw-accept
|
|
||||||
|
|
||||||
# Accept slurm connections to controller from fox (via wireguard)
|
|
||||||
iptables -A nixos-fw -p tcp -i wg0 -s 10.106.0.1/32 --dport 6817 -j nixos-fw-accept
|
|
||||||
# Accept slurm connections from fox for srun (via wireguard)
|
|
||||||
iptables -A nixos-fw -p tcp -i wg0 -s 10.106.0.1/32 --dport 60000:61000 -j nixos-fw-accept
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,17 +0,0 @@
|
|||||||
{ config, lib, pkgs, ... }:
|
|
||||||
|
|
||||||
with lib;
|
|
||||||
|
|
||||||
{
|
|
||||||
systemd.services."prometheus-upc-qaire-exporter" = {
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
after = [ "network.target" ];
|
|
||||||
serviceConfig = {
|
|
||||||
Restart = mkDefault "always";
|
|
||||||
PrivateTmp = mkDefault true;
|
|
||||||
WorkingDirectory = mkDefault "/tmp";
|
|
||||||
DynamicUser = mkDefault true;
|
|
||||||
ExecStart = "${pkgs.upc-qaire-exporter}/bin/upc-qaire-exporter";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,35 +0,0 @@
|
|||||||
{config, ...}:
|
|
||||||
{
|
|
||||||
age.secrets.vpn-dac-login.file = ../../secrets/vpn-dac-login.age;
|
|
||||||
age.secrets.vpn-dac-client-key.file = ../../secrets/vpn-dac-client-key.age;
|
|
||||||
|
|
||||||
services.openvpn.servers = {
|
|
||||||
# systemctl status openvpn-dac.service
|
|
||||||
dac = {
|
|
||||||
config = ''
|
|
||||||
client
|
|
||||||
dev tun
|
|
||||||
proto tcp
|
|
||||||
remote vpn.ac.upc.edu 1194
|
|
||||||
remote vpn.ac.upc.edu 80
|
|
||||||
resolv-retry infinite
|
|
||||||
nobind
|
|
||||||
persist-key
|
|
||||||
persist-tun
|
|
||||||
ca ${./vpn-dac/ca.crt}
|
|
||||||
cert ${./vpn-dac/client.crt}
|
|
||||||
# Only key needs to be secret
|
|
||||||
key ${config.age.secrets.vpn-dac-client-key.path}
|
|
||||||
remote-cert-tls server
|
|
||||||
comp-lzo
|
|
||||||
verb 3
|
|
||||||
auth-user-pass ${config.age.secrets.vpn-dac-login.path}
|
|
||||||
reneg-sec 0
|
|
||||||
|
|
||||||
# Only route fox-ipmi
|
|
||||||
pull-filter ignore "route "
|
|
||||||
route 147.83.35.27 255.255.255.255
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,31 +0,0 @@
|
|||||||
-----BEGIN CERTIFICATE-----
|
|
||||||
MIIFUjCCBDqgAwIBAgIJAJH118PApk5hMA0GCSqGSIb3DQEBCwUAMIHLMQswCQYD
|
|
||||||
VQQGEwJFUzESMBAGA1UECBMJQmFyY2Vsb25hMRIwEAYDVQQHEwlCYXJjZWxvbmEx
|
|
||||||
LTArBgNVBAoTJFVuaXZlcnNpdGF0IFBvbGl0ZWNuaWNhIGRlIENhdGFsdW55YTEk
|
|
||||||
MCIGA1UECxMbQXJxdWl0ZWN0dXJhIGRlIENvbXB1dGFkb3JzMRAwDgYDVQQDEwdM
|
|
||||||
Q0FDIENBMQ0wCwYDVQQpEwRMQ0FDMR4wHAYJKoZIhvcNAQkBFg9sY2FjQGFjLnVw
|
|
||||||
Yy5lZHUwHhcNMTYwMTEyMTI0NDIxWhcNNDYwMTEyMTI0NDIxWjCByzELMAkGA1UE
|
|
||||||
BhMCRVMxEjAQBgNVBAgTCUJhcmNlbG9uYTESMBAGA1UEBxMJQmFyY2Vsb25hMS0w
|
|
||||||
KwYDVQQKEyRVbml2ZXJzaXRhdCBQb2xpdGVjbmljYSBkZSBDYXRhbHVueWExJDAi
|
|
||||||
BgNVBAsTG0FycXVpdGVjdHVyYSBkZSBDb21wdXRhZG9yczEQMA4GA1UEAxMHTENB
|
|
||||||
QyBDQTENMAsGA1UEKRMETENBQzEeMBwGCSqGSIb3DQEJARYPbGNhY0BhYy51cGMu
|
|
||||||
ZWR1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0CteSeof7Xwi51kC
|
|
||||||
F0nQ4E9iR5Lq7wtfRuVPn6JJcIxJJ6+F9gr4R/HIHTztW4XAzReE36DYfexupx3D
|
|
||||||
6UgQIkMLlVyGqRbulNF+RnCx20GosF7Dm4RGBVvOxBP1PGjYq/A+XhaaDAFd0cOF
|
|
||||||
LMNkzuYP7PF0bnBEaHnxmN8bPmuyDyas7fK9AAc3scyWT2jSBPbOVFvCJwPg8MH9
|
|
||||||
V/h+hKwL/7hRt1MVfVv2qyIuKwTki8mUt0RcVbP7oJoRY5K1+R52phIz/GL/b4Fx
|
|
||||||
L6MKXlQxLi8vzP4QZXgCMyV7oFNdU3VqCEXBA11YIRvsOZ4QS19otIk/ZWU5x+HH
|
|
||||||
LAIJ7wIDAQABo4IBNTCCATEwHQYDVR0OBBYEFNyezX1cH1N4QR14ebBpljqmtE7q
|
|
||||||
MIIBAAYDVR0jBIH4MIH1gBTcns19XB9TeEEdeHmwaZY6prRO6qGB0aSBzjCByzEL
|
|
||||||
MAkGA1UEBhMCRVMxEjAQBgNVBAgTCUJhcmNlbG9uYTESMBAGA1UEBxMJQmFyY2Vs
|
|
||||||
b25hMS0wKwYDVQQKEyRVbml2ZXJzaXRhdCBQb2xpdGVjbmljYSBkZSBDYXRhbHVu
|
|
||||||
eWExJDAiBgNVBAsTG0FycXVpdGVjdHVyYSBkZSBDb21wdXRhZG9yczEQMA4GA1UE
|
|
||||||
AxMHTENBQyBDQTENMAsGA1UEKRMETENBQzEeMBwGCSqGSIb3DQEJARYPbGNhY0Bh
|
|
||||||
Yy51cGMuZWR1ggkAkfXXw8CmTmEwDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsF
|
|
||||||
AAOCAQEAUAmOvVXIQrR+aZVO0bOTeugKBHB75eTIZSIHIn2oDUvDbAP5GXIJ56A1
|
|
||||||
6mZXxemSMY8/9k+pRcwJhfat3IgvAN159XSqf9kRv0NHgc3FWUI1Qv/BsAn0vJO/
|
|
||||||
oK0dbmbbRWqt86qNrCN+cUfz5aovvxN73jFfnvfDQFBk/8enj9wXxYfokjjLPR1Q
|
|
||||||
+oTkH8dY68qf71oaUB9MndppPEPSz0K1S6h1XxvJoSu9MVSXOQHiq1cdZdxRazI3
|
|
||||||
4f7q9sTCL+khwDAuZxAYzlEYxFFa/NN8PWU6xPw6V+t/aDhOiXUPJQB/O/K7mw3Z
|
|
||||||
TQQx5NqM7B5jjak5fauR3/oRD8XXsA==
|
|
||||||
-----END CERTIFICATE-----
|
|
||||||
@ -1,100 +0,0 @@
|
|||||||
Certificate:
|
|
||||||
Data:
|
|
||||||
Version: 3 (0x2)
|
|
||||||
Serial Number: 2 (0x2)
|
|
||||||
Signature Algorithm: sha256WithRSAEncryption
|
|
||||||
Issuer: C=ES, ST=Barcelona, L=Barcelona, O=Universitat Politecnica de Catalunya, OU=Arquitectura de Computadors, CN=LCAC CA/name=LCAC/emailAddress=lcac@ac.upc.edu
|
|
||||||
Validity
|
|
||||||
Not Before: Jan 12 12:45:41 2016 GMT
|
|
||||||
Not After : Jan 12 12:45:41 2046 GMT
|
|
||||||
Subject: C=ES, ST=Barcelona, L=Barcelona, O=Universitat Politecnica de Catalunya, OU=Arquitectura de Computadors, CN=client/name=LCAC/emailAddress=lcac@ac.upc.edu
|
|
||||||
Subject Public Key Info:
|
|
||||||
Public Key Algorithm: rsaEncryption
|
|
||||||
Public-Key: (2048 bit)
|
|
||||||
Modulus:
|
|
||||||
00:97:99:fa:7a:0e:4d:e2:1d:a5:b1:a8:14:18:64:
|
|
||||||
c7:66:bf:de:99:1d:92:3b:86:82:4d:95:39:f7:a6:
|
|
||||||
56:49:97:14:4f:e3:37:00:6c:f4:d0:1d:56:79:e7:
|
|
||||||
19:b5:dd:36:15:8e:1d:57:7b:59:29:d2:11:bf:58:
|
|
||||||
48:e0:f7:41:3d:16:64:8d:a2:0b:4a:ac:fa:c6:83:
|
|
||||||
dc:10:2a:2c:d9:97:48:ee:11:2a:bc:4b:60:dd:b9:
|
|
||||||
2e:8f:45:ca:87:0b:38:65:1c:f8:a2:1d:f9:50:aa:
|
|
||||||
6e:60:f9:48:df:57:12:23:e1:e7:0c:81:5c:9f:c5:
|
|
||||||
b2:e6:99:99:95:30:6d:57:36:06:8c:fd:fb:f9:4f:
|
|
||||||
60:d2:3c:ba:ae:28:56:2f:da:58:5c:e8:c5:7b:ec:
|
|
||||||
76:d9:28:6e:fb:8c:07:f9:d7:23:c3:72:76:3c:fa:
|
|
||||||
dc:20:67:8f:cc:16:e0:91:07:d5:68:f9:20:4d:7d:
|
|
||||||
5c:2d:02:04:16:76:52:f3:53:be:a3:dc:0d:d5:fb:
|
|
||||||
6b:55:29:f3:52:35:c8:7d:99:d1:4a:94:be:b1:8e:
|
|
||||||
fd:85:18:25:eb:41:e9:56:da:af:62:84:20:0a:00:
|
|
||||||
17:94:92:94:91:6a:f8:54:37:17:ee:1e:bb:fb:93:
|
|
||||||
71:91:d9:e4:e9:b8:3b:18:7d:6d:7d:4c:ce:58:55:
|
|
||||||
f9:41
|
|
||||||
Exponent: 65537 (0x10001)
|
|
||||||
X509v3 extensions:
|
|
||||||
X509v3 Basic Constraints:
|
|
||||||
CA:FALSE
|
|
||||||
Netscape Comment:
|
|
||||||
Easy-RSA Generated Certificate
|
|
||||||
X509v3 Subject Key Identifier:
|
|
||||||
1B:88:06:D5:33:1D:5C:48:46:B5:DE:78:89:36:96:91:3A:74:43:18
|
|
||||||
X509v3 Authority Key Identifier:
|
|
||||||
keyid:DC:9E:CD:7D:5C:1F:53:78:41:1D:78:79:B0:69:96:3A:A6:B4:4E:EA
|
|
||||||
DirName:/C=ES/ST=Barcelona/L=Barcelona/O=Universitat Politecnica de Catalunya/OU=Arquitectura de Computadors/CN=LCAC CA/name=LCAC/emailAddress=lcac@ac.upc.edu
|
|
||||||
serial:91:F5:D7:C3:C0:A6:4E:61
|
|
||||||
|
|
||||||
X509v3 Extended Key Usage:
|
|
||||||
TLS Web Client Authentication
|
|
||||||
X509v3 Key Usage:
|
|
||||||
Digital Signature
|
|
||||||
X509v3 Subject Alternative Name:
|
|
||||||
DNS:client
|
|
||||||
Signature Algorithm: sha256WithRSAEncryption
|
|
||||||
42:e8:50:b2:e7:88:75:86:0b:bb:29:e3:aa:c6:0e:4c:e8:ea:
|
|
||||||
3d:0c:02:31:7f:3b:80:0c:3f:80:af:45:d6:62:27:a0:0e:e7:
|
|
||||||
26:09:12:97:95:f8:d9:9b:89:b5:ef:56:64:f1:de:82:74:e0:
|
|
||||||
31:0a:cc:90:0a:bd:50:b8:54:95:0a:ae:3b:40:df:76:b6:d1:
|
|
||||||
01:2e:f3:96:9f:52:d4:e9:14:6d:b7:14:9d:45:99:33:36:2a:
|
|
||||||
01:0b:15:1a:ed:55:dc:64:83:65:1a:06:42:d9:c7:dc:97:d4:
|
|
||||||
02:81:c2:58:2b:ea:e4:b7:ae:84:3a:e4:3f:f1:2e:fa:ec:f3:
|
|
||||||
40:5d:b8:6a:d5:5e:e1:e8:2f:e2:2f:48:a4:38:a1:4f:22:e3:
|
|
||||||
4f:66:94:aa:02:78:9a:2b:7a:5d:aa:aa:51:a5:e3:d0:91:e9:
|
|
||||||
1d:f9:08:ed:8b:51:c9:a6:af:46:85:b5:1c:ed:12:a1:28:33:
|
|
||||||
75:36:00:d8:5c:14:65:96:c0:28:7d:47:50:a4:89:5f:b0:72:
|
|
||||||
1a:4b:13:17:26:0f:f0:b8:65:3c:e9:96:36:f9:bf:90:59:33:
|
|
||||||
87:1f:01:03:25:f8:f0:3a:9b:33:02:d0:0a:43:b5:0a:cf:62:
|
|
||||||
a1:45:38:37:07:9d:9c:94:0b:31:c6:3c:34:b7:fc:5a:0c:e4:
|
|
||||||
bf:23:f6:7d
|
|
||||||
-----BEGIN CERTIFICATE-----
|
|
||||||
MIIFqjCCBJKgAwIBAgIBAjANBgkqhkiG9w0BAQsFADCByzELMAkGA1UEBhMCRVMx
|
|
||||||
EjAQBgNVBAgTCUJhcmNlbG9uYTESMBAGA1UEBxMJQmFyY2Vsb25hMS0wKwYDVQQK
|
|
||||||
EyRVbml2ZXJzaXRhdCBQb2xpdGVjbmljYSBkZSBDYXRhbHVueWExJDAiBgNVBAsT
|
|
||||||
G0FycXVpdGVjdHVyYSBkZSBDb21wdXRhZG9yczEQMA4GA1UEAxMHTENBQyBDQTEN
|
|
||||||
MAsGA1UEKRMETENBQzEeMBwGCSqGSIb3DQEJARYPbGNhY0BhYy51cGMuZWR1MB4X
|
|
||||||
DTE2MDExMjEyNDU0MVoXDTQ2MDExMjEyNDU0MVowgcoxCzAJBgNVBAYTAkVTMRIw
|
|
||||||
EAYDVQQIEwlCYXJjZWxvbmExEjAQBgNVBAcTCUJhcmNlbG9uYTEtMCsGA1UEChMk
|
|
||||||
VW5pdmVyc2l0YXQgUG9saXRlY25pY2EgZGUgQ2F0YWx1bnlhMSQwIgYDVQQLExtB
|
|
||||||
cnF1aXRlY3R1cmEgZGUgQ29tcHV0YWRvcnMxDzANBgNVBAMTBmNsaWVudDENMAsG
|
|
||||||
A1UEKRMETENBQzEeMBwGCSqGSIb3DQEJARYPbGNhY0BhYy51cGMuZWR1MIIBIjAN
|
|
||||||
BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAl5n6eg5N4h2lsagUGGTHZr/emR2S
|
|
||||||
O4aCTZU596ZWSZcUT+M3AGz00B1WeecZtd02FY4dV3tZKdIRv1hI4PdBPRZkjaIL
|
|
||||||
Sqz6xoPcECos2ZdI7hEqvEtg3bkuj0XKhws4ZRz4oh35UKpuYPlI31cSI+HnDIFc
|
|
||||||
n8Wy5pmZlTBtVzYGjP37+U9g0jy6rihWL9pYXOjFe+x22Shu+4wH+dcjw3J2PPrc
|
|
||||||
IGePzBbgkQfVaPkgTX1cLQIEFnZS81O+o9wN1ftrVSnzUjXIfZnRSpS+sY79hRgl
|
|
||||||
60HpVtqvYoQgCgAXlJKUkWr4VDcX7h67+5Nxkdnk6bg7GH1tfUzOWFX5QQIDAQAB
|
|
||||||
o4IBljCCAZIwCQYDVR0TBAIwADAtBglghkgBhvhCAQ0EIBYeRWFzeS1SU0EgR2Vu
|
|
||||||
ZXJhdGVkIENlcnRpZmljYXRlMB0GA1UdDgQWBBQbiAbVMx1cSEa13niJNpaROnRD
|
|
||||||
GDCCAQAGA1UdIwSB+DCB9YAU3J7NfVwfU3hBHXh5sGmWOqa0TuqhgdGkgc4wgcsx
|
|
||||||
CzAJBgNVBAYTAkVTMRIwEAYDVQQIEwlCYXJjZWxvbmExEjAQBgNVBAcTCUJhcmNl
|
|
||||||
bG9uYTEtMCsGA1UEChMkVW5pdmVyc2l0YXQgUG9saXRlY25pY2EgZGUgQ2F0YWx1
|
|
||||||
bnlhMSQwIgYDVQQLExtBcnF1aXRlY3R1cmEgZGUgQ29tcHV0YWRvcnMxEDAOBgNV
|
|
||||||
BAMTB0xDQUMgQ0ExDTALBgNVBCkTBExDQUMxHjAcBgkqhkiG9w0BCQEWD2xjYWNA
|
|
||||||
YWMudXBjLmVkdYIJAJH118PApk5hMBMGA1UdJQQMMAoGCCsGAQUFBwMCMAsGA1Ud
|
|
||||||
DwQEAwIHgDARBgNVHREECjAIggZjbGllbnQwDQYJKoZIhvcNAQELBQADggEBAELo
|
|
||||||
ULLniHWGC7sp46rGDkzo6j0MAjF/O4AMP4CvRdZiJ6AO5yYJEpeV+NmbibXvVmTx
|
|
||||||
3oJ04DEKzJAKvVC4VJUKrjtA33a20QEu85afUtTpFG23FJ1FmTM2KgELFRrtVdxk
|
|
||||||
g2UaBkLZx9yX1AKBwlgr6uS3roQ65D/xLvrs80BduGrVXuHoL+IvSKQ4oU8i409m
|
|
||||||
lKoCeJorel2qqlGl49CR6R35CO2LUcmmr0aFtRztEqEoM3U2ANhcFGWWwCh9R1Ck
|
|
||||||
iV+wchpLExcmD/C4ZTzpljb5v5BZM4cfAQMl+PA6mzMC0ApDtQrPYqFFODcHnZyU
|
|
||||||
CzHGPDS3/FoM5L8j9n0=
|
|
||||||
-----END CERTIFICATE-----
|
|
||||||
@ -1,28 +0,0 @@
|
|||||||
{ config, pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../common/ssf.nix
|
|
||||||
../module/ceph.nix
|
|
||||||
../module/emulation.nix
|
|
||||||
../module/slurm-client.nix
|
|
||||||
../module/slurm-firewall.nix
|
|
||||||
../module/debuginfod.nix
|
|
||||||
../module/hut-substituter.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
# Select the this using the ID to avoid mismatches
|
|
||||||
boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d53566c";
|
|
||||||
|
|
||||||
networking = {
|
|
||||||
hostName = "owl1";
|
|
||||||
interfaces.eno1.ipv4.addresses = [ {
|
|
||||||
address = "10.0.40.1";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
interfaces.ibp5s0.ipv4.addresses = [ {
|
|
||||||
address = "10.0.42.1";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,29 +0,0 @@
|
|||||||
{ config, pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../common/ssf.nix
|
|
||||||
../module/ceph.nix
|
|
||||||
../module/emulation.nix
|
|
||||||
../module/slurm-client.nix
|
|
||||||
../module/slurm-firewall.nix
|
|
||||||
../module/debuginfod.nix
|
|
||||||
../module/hut-substituter.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
# Select the this using the ID to avoid mismatches
|
|
||||||
boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d535629";
|
|
||||||
|
|
||||||
networking = {
|
|
||||||
hostName = "owl2";
|
|
||||||
interfaces.eno1.ipv4.addresses = [ {
|
|
||||||
address = "10.0.40.2";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
# Watch out! The OmniPath device is not in the same place here:
|
|
||||||
interfaces.ibp129s0.ipv4.addresses = [ {
|
|
||||||
address = "10.0.42.2";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,98 +0,0 @@
|
|||||||
{ config, pkgs, lib, modulesPath, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../common/base.nix
|
|
||||||
../common/ssf/hosts.nix
|
|
||||||
../module/emulation.nix
|
|
||||||
../module/debuginfod.nix
|
|
||||||
../module/nvidia.nix
|
|
||||||
../eudy/kernel/perf.nix
|
|
||||||
./wireguard.nix
|
|
||||||
../module/hut-substituter.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
# Don't install Grub on the disk yet
|
|
||||||
boot.loader.grub.device = "nodev";
|
|
||||||
|
|
||||||
# Enable serial console
|
|
||||||
boot.kernelParams = [
|
|
||||||
"console=tty1"
|
|
||||||
"console=ttyS1,115200"
|
|
||||||
];
|
|
||||||
|
|
||||||
networking = {
|
|
||||||
hostName = "raccoon";
|
|
||||||
# Only BSC DNSs seem to be reachable from the office VLAN
|
|
||||||
nameservers = [ "84.88.52.35" "84.88.52.36" ];
|
|
||||||
defaultGateway = "84.88.51.129";
|
|
||||||
interfaces.eno0.ipv4.addresses = [ {
|
|
||||||
address = "84.88.51.152";
|
|
||||||
prefixLength = 25;
|
|
||||||
} ];
|
|
||||||
interfaces.enp5s0f1.ipv4.addresses = [ {
|
|
||||||
address = "10.0.44.1";
|
|
||||||
prefixLength = 24;
|
|
||||||
} ];
|
|
||||||
nat = {
|
|
||||||
enable = true;
|
|
||||||
internalInterfaces = [ "enp5s0f1" ];
|
|
||||||
externalInterface = "eno0";
|
|
||||||
};
|
|
||||||
hosts = {
|
|
||||||
"10.0.44.4" = [ "tent" ];
|
|
||||||
"84.88.53.236" = [ "apex" ];
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# Mount the NFS home
|
|
||||||
fileSystems."/nfs/home" = {
|
|
||||||
device = "10.106.0.30:/home";
|
|
||||||
fsType = "nfs";
|
|
||||||
options = [ "nfsvers=3" "rsize=1024" "wsize=1024" "cto" "nofail" ];
|
|
||||||
};
|
|
||||||
|
|
||||||
# Enable performance governor
|
|
||||||
powerManagement.cpuFreqGovernor = "performance";
|
|
||||||
|
|
||||||
hardware.nvidia.open = false; # Maxwell is older than Turing architecture
|
|
||||||
|
|
||||||
services.openssh.settings.X11Forwarding = true;
|
|
||||||
|
|
||||||
services.prometheus.exporters.node = {
|
|
||||||
enable = true;
|
|
||||||
enabledCollectors = [ "systemd" ];
|
|
||||||
port = 9002;
|
|
||||||
listenAddress = "127.0.0.1";
|
|
||||||
};
|
|
||||||
|
|
||||||
users.motd = ''
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⣀⣀⣄⣠⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⢰⠇⡀⠀⠙⠻⡿⣦⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡀⠀⠀⠀⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⡎⢰⣧⠀⠀⠀⠁⠈⠛⢿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣠⣴⡦⠶⠟⠓⠚⠻⡄⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⣧⠀⣱⣀⣰⣧⠀⢀⠀⣘⣿⣿⣦⣶⣄⣠⡀⠀⠀⣀⣀⣤⣴⣄⣀⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣴⣿⠿⠏⠁⠀⣀⣠⣶⣿⡶⣿⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⣹⣆⠘⣿⣿⣿⣇⢸⣷⣿⣿⣿⣿⣿⣿⣿⣿⣿⣾⣿⣿⣿⣿⣿⣿⣿⣿⣶⣶⣦⡀⣀⣤⣠⣤⡾⠋⠀⢀⣤⣶⣿⣿⣿⣿⣿⣿⣿⡀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠘⢿⡄⢼⣿⣿⣿⣿⣿⡟⠻⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣵⣾⡾⠙⣋⣩⣽⣿⣿⣿⣿⢋⡼⠁
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠈⢻⣄⠸⢿⣿⣿⠿⠷⠀⠈⠀⣭⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣾⣿⣿⣿⣿⣿⣿⠇⡼⠁⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠀⢾⣯⡀⠀⢼⡿⠀⠀⠀⢼⠿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⣿⡿⣿⣿⣿⠿⣿⣯⣼⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⢋⡼⠁⠀⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠀⠀⢻⡏⠠⣦⠁⠀⠀⠀⠀⠀⠟⠛⠛⣿⣿⣿⣿⣿⠿⠁⠀⠁⢿⠙⠁⠀⠛⠹⣿⣏⣾⣿⣿⣿⣿⣿⣿⣿⣿⠿⠃⣹⠁⠀⠀⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠀⠀⣘⣧⠀⠙⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⡿⡿⠀⠀⠀⠀⠈⠀⠀⠀⠀⠀⠀⢹⣿⠿⢿⣿⣿⣿⣿⣿⠋⢀⡤⠛⠀⠀⠀⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠀⠀⢹⡯⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠸⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠁⠀⢸⣿⣿⣿⠛⠉⠀⣰⠷⠀⠀⠀⠀⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠇⠀⠀⠀⠀⠀⢀⣿⡇⠀⠀⢻⣿⣿⠁⠀⠀⢠⣾⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠸⠟⢿⣿⣄⡀⢸⣿⡀⠀⠀⠀⠀⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠀⢀⣿⠀⠀⠀⢰⣿⣿⡛⣿⣿⡄⢠⡺⠿⡍⠁⢀⣤⣿⣿⣿⠿⣷⣮⣉⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠈⣿⠀⠀⠈⣧⠀⠀⠀⠀⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠀⢾⠉⠃⠀⣴⣿⣟⠻⣿⣿⣿⡇⢸⣿⣶⠀⢀⣾⣿⣿⣟⠿⣷⣾⣿⣿⣿⣿⣦⣤⣤⡤⠀⠀⠀⠀⠀⠁⠀⠀⠀⣼⠗⠀⠀⠀⠀
|
|
||||||
⠀⠀⠐⢄⡀⠀⠀⠀⢘⡀⠀⢶⣾⣿⣿⣿⣿⡿⠋⠁⠈⠻⠉⠀⠚⠻⣿⣿⣿⣶⣾⣿⣿⣿⣿⣿⣿⣷⣬⣤⣶⣦⡀⣾⣶⣇⠀⠀⠈⢉⣷⠀⠀⠀⠀
|
|
||||||
⠀⠀⠀⠀⠈⠓⠶⢦⡽⠄⣈⣿⣿⣿⣿⣿⠏⠀⠀⠀⠀⠀⠀⠀⠀⠀⠹⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡓⠙⣿⡟⠀⠀⠀⠈⠛⣷⣶⡄⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⢀⣬⠆⢠⣍⣛⠻⣿⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣉⣀⡀⠀⠀⠈⠛⢿⣦⡀
|
|
||||||
⠐⠒⠒⠶⠶⠶⢦⣬⣟⣥⣀⡉⠛⠻⠶⢁⣤⣾⣿⣿⣿⣷⡄⠀⠀⠀⠀⠀⢸⣿⣿⣿⣿⣿⣟⡛⠿⠭⠭⠭⠭⠭⠿⠿⠿⢿⣿⣟⠃⠀⠀⠀⠹⣟⠓
|
|
||||||
⠀⣀⣠⠤⠤⢤⣤⣾⣤⡄⣉⣉⣙⣓⡂⣿⣿⣭⣹⣿⣿⣿⣿⡰⣂⣀⢀⠀⠻⣿⠛⠻⠟⠡⣶⣾⣿⣿⣿⣿⣿⣿⣿⡖⠒⠒⠒⠛⠷⢤⡀⢰⣴⣿⡆
|
|
||||||
⠀⠀⠀⢀⣠⡴⠾⠟⠻⣟⡉⠉⠉⠉⢁⢿⣿⣿⣿⣿⣿⣿⡿⣱⣿⣭⡌⠤⠀⠀⠐⣶⣌⡻⣶⣭⡻⢿⣿⣿⣿⣿⣿⣯⣥⣤⣦⠀⠠⣴⣶⣶⣿⡟⢿
|
|
||||||
⢀⠔⠊⠉⠀⠀⠀⠀⢸⣯⣤⠀⠀⠠⣼⣮⣟⣿⣿⣿⣻⣭⣾⣿⣿⣷⣶⣦⠶⣚⣾⣿⣿⣷⣜⣿⣿⣶⣝⢿⣿⣿⣿⣿⣷⣦⣄⣰⡄⠈⢿⣿⡿⣇⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠀⠈⢡⢇⠀⠀⣠⣿⣿⣿⣯⣟⣛⣛⣛⣛⣛⣩⣭⣴⣶⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣿⣿⣿⣿⣿⣿⣿⣿⣿⣦⣻⣿⣧⠀⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠀⠀⣾⠏⠀⢹⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣦⣍⣿⣿⣿⣿⡄⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣾⡁⢈⣾⣿⡿⠛⣛⣿⣿⣿⣿ DO YOU BRING FEEDS? ⣿⣿⣿⣿⣿⣿⡏⠈⠙⠈⠁⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠛⡿⠛⠉⣽⣿⣷⣾⡿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⠷⠌⠛⠉⠀⠁⠀⠀⠀⠀⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠀⠀⠹⠋⠀⢻⣿⣿⣿⣿⠿⢿⣿⣿⣿⣿⣿⣿⠿⣿⣿⣿⣿⠿⠛⠋⠉⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
|
|
||||||
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠉⠁⠀⠀⠀⠀⠀⠈⠉⠉⠀⠀⠈⠋⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
@ -1,48 +0,0 @@
|
|||||||
{ config, pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
networking.nat = {
|
|
||||||
enable = true;
|
|
||||||
enableIPv6 = false;
|
|
||||||
externalInterface = "eno0";
|
|
||||||
internalInterfaces = [ "wg0" ];
|
|
||||||
};
|
|
||||||
|
|
||||||
networking.firewall = {
|
|
||||||
allowedUDPPorts = [ 666 ];
|
|
||||||
};
|
|
||||||
|
|
||||||
age.secrets.wgRaccoon.file = ../../secrets/wg-raccoon.age;
|
|
||||||
|
|
||||||
# Enable WireGuard
|
|
||||||
networking.wireguard.enable = true;
|
|
||||||
networking.wireguard.interfaces = {
|
|
||||||
wg0 = {
|
|
||||||
ips = [ "10.106.0.236/24" ];
|
|
||||||
listenPort = 666;
|
|
||||||
privateKeyFile = config.age.secrets.wgRaccoon.path;
|
|
||||||
# Public key: QUfnGXSMEgu2bviglsaSdCjidB51oEDBFpnSFcKGfDI=
|
|
||||||
peers = [
|
|
||||||
{
|
|
||||||
name = "fox";
|
|
||||||
publicKey = "VfMPBQLQTKeyXJSwv8wBhc6OV0j2qAxUpX3kLHunK2Y=";
|
|
||||||
allowedIPs = [ "10.106.0.1/32" ];
|
|
||||||
endpoint = "fox.ac.upc.edu:666";
|
|
||||||
persistentKeepalive = 25;
|
|
||||||
}
|
|
||||||
{
|
|
||||||
name = "apex";
|
|
||||||
publicKey = "VwhcN8vSOzdJEotQTpmPHBC52x3Hbv1lkFIyKubrnUA=";
|
|
||||||
allowedIPs = [ "10.106.0.30/32" "10.0.40.0/24" ];
|
|
||||||
endpoint = "ssfhead.bsc.es:666";
|
|
||||||
persistentKeepalive = 25;
|
|
||||||
}
|
|
||||||
];
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
networking.hosts = {
|
|
||||||
"10.106.0.1" = [ "fox.wg" ];
|
|
||||||
"10.106.0.30" = [ "apex.wg" ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,14 +0,0 @@
|
|||||||
modules:
|
|
||||||
http_2xx:
|
|
||||||
prober: http
|
|
||||||
timeout: 5s
|
|
||||||
http:
|
|
||||||
preferred_ip_protocol: "ip4"
|
|
||||||
follow_redirects: true
|
|
||||||
valid_status_codes: [] # Defaults to 2xx
|
|
||||||
method: GET
|
|
||||||
icmp:
|
|
||||||
prober: icmp
|
|
||||||
timeout: 5s
|
|
||||||
icmp:
|
|
||||||
preferred_ip_protocol: "ip4"
|
|
||||||
@ -1,85 +0,0 @@
|
|||||||
{ config, pkgs, lib, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../common/xeon.nix
|
|
||||||
../common/ssf/hosts.nix
|
|
||||||
../module/emulation.nix
|
|
||||||
../module/debuginfod.nix
|
|
||||||
./monitoring.nix
|
|
||||||
./nginx.nix
|
|
||||||
./nix-serve.nix
|
|
||||||
./gitlab-runner.nix
|
|
||||||
./gitea.nix
|
|
||||||
../hut/public-inbox.nix
|
|
||||||
../hut/msmtp.nix
|
|
||||||
../module/p.nix
|
|
||||||
../module/vpn-dac.nix
|
|
||||||
../module/hut-substituter.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
# Select the this using the ID to avoid mismatches
|
|
||||||
boot.loader.grub.device = "/dev/disk/by-id/wwn-0x55cd2e414d537675";
|
|
||||||
|
|
||||||
networking = {
|
|
||||||
hostName = "tent";
|
|
||||||
interfaces.eno1.ipv4.addresses = [
|
|
||||||
{
|
|
||||||
address = "10.0.44.4";
|
|
||||||
prefixLength = 24;
|
|
||||||
}
|
|
||||||
];
|
|
||||||
|
|
||||||
# Only BSC DNSs seem to be reachable from the office VLAN
|
|
||||||
nameservers = [ "84.88.52.35" "84.88.52.36" ];
|
|
||||||
search = [ "bsc.es" "ac.upc.edu" ];
|
|
||||||
defaultGateway = "10.0.44.1";
|
|
||||||
hosts = {
|
|
||||||
"84.88.53.236" = [ "apex" ];
|
|
||||||
"10.0.44.1" = [ "raccoon" ];
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
services.p.enable = true;
|
|
||||||
|
|
||||||
services.prometheus.exporters.node = {
|
|
||||||
enable = true;
|
|
||||||
enabledCollectors = [ "systemd" ];
|
|
||||||
port = 9002;
|
|
||||||
listenAddress = "127.0.0.1";
|
|
||||||
};
|
|
||||||
|
|
||||||
boot.swraid = {
|
|
||||||
enable = true;
|
|
||||||
mdadmConf = ''
|
|
||||||
DEVICE partitions
|
|
||||||
ARRAY /dev/md0 metadata=1.2 UUID=496db1e2:056a92aa:a544543f:40db379d
|
|
||||||
MAILADDR root
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
fileSystems."/vault" = {
|
|
||||||
device = "/dev/disk/by-label/vault";
|
|
||||||
fsType = "ext4";
|
|
||||||
};
|
|
||||||
|
|
||||||
# Make a /vault/$USER directory for each user.
|
|
||||||
systemd.services.create-vault-dirs = let
|
|
||||||
# Take only normal users in tent
|
|
||||||
users = lib.filterAttrs (_: v: v.isNormalUser) config.users.users;
|
|
||||||
commands = lib.concatLists (lib.mapAttrsToList
|
|
||||||
(_: user: [
|
|
||||||
"install -d -o ${user.name} -g ${user.group} -m 0711 /vault/home/${user.name}"
|
|
||||||
]) users);
|
|
||||||
script = pkgs.writeShellScript "create-vault-dirs.sh" (lib.concatLines commands);
|
|
||||||
in {
|
|
||||||
enable = true;
|
|
||||||
wants = [ "local-fs.target" ];
|
|
||||||
after = [ "local-fs.target" ];
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
serviceConfig.ExecStart = script;
|
|
||||||
};
|
|
||||||
|
|
||||||
# disable automatic garbage collector
|
|
||||||
nix.gc.automatic = lib.mkForce false;
|
|
||||||
}
|
|
||||||
@ -1,30 +0,0 @@
|
|||||||
{ config, lib, ... }:
|
|
||||||
{
|
|
||||||
services.gitea = {
|
|
||||||
enable = true;
|
|
||||||
appName = "Gitea in the jungle";
|
|
||||||
|
|
||||||
settings = {
|
|
||||||
server = {
|
|
||||||
ROOT_URL = "https://jungle.bsc.es/git/";
|
|
||||||
LOCAL_ROOT_URL = "https://jungle.bsc.es/git/";
|
|
||||||
LANDING_PAGE = "explore";
|
|
||||||
};
|
|
||||||
metrics.ENABLED = true;
|
|
||||||
service = {
|
|
||||||
DISABLE_REGISTRATION = true;
|
|
||||||
REGISTER_MANUAL_CONFIRM = true;
|
|
||||||
ENABLE_NOTIFY_MAIL = true;
|
|
||||||
};
|
|
||||||
log.LEVEL = "Warn";
|
|
||||||
|
|
||||||
mailer = {
|
|
||||||
ENABLED = true;
|
|
||||||
FROM = "jungle-robot@bsc.es";
|
|
||||||
PROTOCOL = "sendmail";
|
|
||||||
SENDMAIL_PATH = "/run/wrappers/bin/sendmail";
|
|
||||||
SENDMAIL_ARGS = "--";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,93 +0,0 @@
|
|||||||
{ pkgs, lib, config, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
age.secrets.tent-gitlab-runner-pm-shell.file = ../../secrets/tent-gitlab-runner-pm-shell-token.age;
|
|
||||||
age.secrets.tent-gitlab-runner-pm-docker.file = ../../secrets/tent-gitlab-runner-pm-docker-token.age;
|
|
||||||
age.secrets.tent-gitlab-runner-bsc-docker.file = ../../secrets/tent-gitlab-runner-bsc-docker-token.age;
|
|
||||||
|
|
||||||
services.gitlab-runner = let sec = config.age.secrets; in {
|
|
||||||
enable = true;
|
|
||||||
settings.concurrent = 5;
|
|
||||||
services = {
|
|
||||||
# For gitlab.pm.bsc.es
|
|
||||||
gitlab-pm-shell = {
|
|
||||||
executor = "shell";
|
|
||||||
environmentVariables = {
|
|
||||||
SHELL = "${pkgs.bash}/bin/bash";
|
|
||||||
};
|
|
||||||
authenticationTokenConfigFile = sec.tent-gitlab-runner-pm-shell.path;
|
|
||||||
preGetSourcesScript = pkgs.writeScript "setup" ''
|
|
||||||
echo "This is the preGetSources script running, brace for impact"
|
|
||||||
env
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
gitlab-pm-docker = {
|
|
||||||
authenticationTokenConfigFile = sec.tent-gitlab-runner-pm-docker.path;
|
|
||||||
executor = "docker";
|
|
||||||
dockerImage = "debian:stable";
|
|
||||||
};
|
|
||||||
|
|
||||||
# For gitlab.bsc.es
|
|
||||||
gitlab-bsc-docker = {
|
|
||||||
# gitlab.bsc.es still uses the old token mechanism
|
|
||||||
registrationConfigFile = sec.tent-gitlab-runner-bsc-docker.path;
|
|
||||||
tagList = [ "docker" "tent" "nix" ];
|
|
||||||
executor = "docker";
|
|
||||||
dockerImage = "alpine";
|
|
||||||
dockerVolumes = [
|
|
||||||
"/nix/store:/nix/store:ro"
|
|
||||||
"/nix/var/nix/db:/nix/var/nix/db:ro"
|
|
||||||
"/nix/var/nix/daemon-socket:/nix/var/nix/daemon-socket:ro"
|
|
||||||
];
|
|
||||||
dockerDisableCache = true;
|
|
||||||
registrationFlags = [
|
|
||||||
# Increase build log length to 64 MiB
|
|
||||||
"--output-limit 65536"
|
|
||||||
];
|
|
||||||
preBuildScript = pkgs.writeScript "setup-container" ''
|
|
||||||
mkdir -p -m 0755 /nix/var/log/nix/drvs
|
|
||||||
mkdir -p -m 0755 /nix/var/nix/gcroots
|
|
||||||
mkdir -p -m 0755 /nix/var/nix/profiles
|
|
||||||
mkdir -p -m 0755 /nix/var/nix/temproots
|
|
||||||
mkdir -p -m 0755 /nix/var/nix/userpool
|
|
||||||
mkdir -p -m 1777 /nix/var/nix/gcroots/per-user
|
|
||||||
mkdir -p -m 1777 /nix/var/nix/profiles/per-user
|
|
||||||
mkdir -p -m 0755 /nix/var/nix/profiles/per-user/root
|
|
||||||
mkdir -p -m 0700 "$HOME/.nix-defexpr"
|
|
||||||
mkdir -p -m 0700 "$HOME/.ssh"
|
|
||||||
cat >> "$HOME/.ssh/known_hosts" << EOF
|
|
||||||
bscpm04.bsc.es ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPx4mC0etyyjYUT2Ztc/bs4ZXSbVMrogs1ZTP924PDgT
|
|
||||||
gitlab-internal.bsc.es ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF9arsAOSRB06hdy71oTvJHG2Mg8zfebADxpvc37lZo3
|
|
||||||
EOF
|
|
||||||
. ${pkgs.nix}/etc/profile.d/nix-daemon.sh
|
|
||||||
# Required to load SSL certificate paths
|
|
||||||
. ${pkgs.cacert}/nix-support/setup-hook
|
|
||||||
'';
|
|
||||||
environmentVariables = {
|
|
||||||
ENV = "/etc/profile";
|
|
||||||
USER = "root";
|
|
||||||
NIX_REMOTE = "daemon";
|
|
||||||
PATH = "${config.system.path}/bin:/bin:/sbin:/usr/bin:/usr/sbin";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
systemd.services.gitlab-runner.serviceConfig = {
|
|
||||||
DynamicUser = lib.mkForce false;
|
|
||||||
User = "gitlab-runner";
|
|
||||||
Group = "gitlab-runner";
|
|
||||||
ExecStart = lib.mkForce
|
|
||||||
''${pkgs.gitlab-runner}/bin/gitlab-runner run --config ''${HOME}/.gitlab-runner/config.toml --listen-address "127.0.0.1:9252" --working-directory ''${HOME}'';
|
|
||||||
};
|
|
||||||
|
|
||||||
users.users.gitlab-runner = {
|
|
||||||
uid = config.ids.uids.gitlab-runner;
|
|
||||||
home = "/var/lib/gitlab-runner";
|
|
||||||
description = "Gitlab Runner";
|
|
||||||
group = "gitlab-runner";
|
|
||||||
extraGroups = [ "docker" ];
|
|
||||||
createHome = true;
|
|
||||||
};
|
|
||||||
users.groups.gitlab-runner.gid = config.ids.gids.gitlab-runner;
|
|
||||||
}
|
|
||||||
@ -1,217 +0,0 @@
|
|||||||
{ config, lib, pkgs, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
imports = [
|
|
||||||
../module/meteocat-exporter.nix
|
|
||||||
../module/upc-qaire-exporter.nix
|
|
||||||
../module/nix-daemon-exporter.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
age.secrets.grafanaJungleRobotPassword = {
|
|
||||||
file = ../../secrets/jungle-robot-password.age;
|
|
||||||
owner = "grafana";
|
|
||||||
mode = "400";
|
|
||||||
};
|
|
||||||
|
|
||||||
services.grafana = {
|
|
||||||
enable = true;
|
|
||||||
settings = {
|
|
||||||
server = {
|
|
||||||
domain = "jungle.bsc.es";
|
|
||||||
root_url = "%(protocol)s://%(domain)s/grafana";
|
|
||||||
serve_from_sub_path = true;
|
|
||||||
http_port = 2342;
|
|
||||||
http_addr = "127.0.0.1";
|
|
||||||
};
|
|
||||||
smtp = {
|
|
||||||
enabled = true;
|
|
||||||
from_address = "jungle-robot@bsc.es";
|
|
||||||
user = "jungle-robot";
|
|
||||||
# Read the password from a file, which is only readable by grafana user
|
|
||||||
# https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#file-provider
|
|
||||||
password = "$__file{${config.age.secrets.grafanaJungleRobotPassword.path}}";
|
|
||||||
host = "mail.bsc.es:465";
|
|
||||||
startTLS_policy = "NoStartTLS";
|
|
||||||
};
|
|
||||||
feature_toggles.publicDashboards = true;
|
|
||||||
"auth.anonymous".enabled = true;
|
|
||||||
log.level = "warn";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
services.prometheus = {
|
|
||||||
enable = true;
|
|
||||||
port = 9001;
|
|
||||||
retentionTime = "5y";
|
|
||||||
listenAddress = "127.0.0.1";
|
|
||||||
};
|
|
||||||
|
|
||||||
# We need access to the devices to monitor the disk space
|
|
||||||
systemd.services.prometheus-node-exporter.serviceConfig.PrivateDevices = lib.mkForce false;
|
|
||||||
systemd.services.prometheus-node-exporter.serviceConfig.ProtectHome = lib.mkForce "read-only";
|
|
||||||
|
|
||||||
# Credentials for IPMI exporter
|
|
||||||
age.secrets.ipmiYml = {
|
|
||||||
file = ../../secrets/ipmi.yml.age;
|
|
||||||
owner = "ipmi-exporter";
|
|
||||||
};
|
|
||||||
|
|
||||||
# Create an IPMI group and assign the ipmi0 device
|
|
||||||
users.groups.ipmi = {};
|
|
||||||
services.udev.extraRules = ''
|
|
||||||
SUBSYSTEM=="ipmi", KERNEL=="ipmi0", GROUP="ipmi", MODE="0660"
|
|
||||||
'';
|
|
||||||
|
|
||||||
# Add a new ipmi-exporter user that can read the ipmi0 device
|
|
||||||
users.users.ipmi-exporter = {
|
|
||||||
isSystemUser = true;
|
|
||||||
group = "ipmi";
|
|
||||||
};
|
|
||||||
|
|
||||||
# Disable dynamic user so we have the ipmi-exporter user available for the credentials
|
|
||||||
systemd.services.prometheus-ipmi-exporter.serviceConfig = {
|
|
||||||
DynamicUser = lib.mkForce false;
|
|
||||||
PrivateDevices = lib.mkForce false;
|
|
||||||
User = lib.mkForce "ipmi-exporter";
|
|
||||||
Group = lib.mkForce "ipmi";
|
|
||||||
RestrictNamespaces = lib.mkForce false;
|
|
||||||
# Fake uid to 0 so it shuts up
|
|
||||||
ExecStart = let
|
|
||||||
cfg = config.services.prometheus.exporters.ipmi;
|
|
||||||
in lib.mkForce (lib.concatStringsSep " " ([
|
|
||||||
"${pkgs.util-linux}/bin/unshare --map-user 0"
|
|
||||||
"${pkgs.prometheus-ipmi-exporter}/bin/ipmi_exporter"
|
|
||||||
"--web.listen-address ${cfg.listenAddress}:${toString cfg.port}"
|
|
||||||
"--config.file ${lib.escapeShellArg cfg.configFile}"
|
|
||||||
] ++ cfg.extraFlags));
|
|
||||||
};
|
|
||||||
|
|
||||||
services.prometheus = {
|
|
||||||
exporters = {
|
|
||||||
ipmi = {
|
|
||||||
enable = true;
|
|
||||||
configFile = config.age.secrets.ipmiYml.path;
|
|
||||||
#extraFlags = [ "--log.level=debug" ];
|
|
||||||
listenAddress = "127.0.0.1";
|
|
||||||
};
|
|
||||||
node = {
|
|
||||||
enable = true;
|
|
||||||
enabledCollectors = [ "logind" ];
|
|
||||||
port = 9002;
|
|
||||||
listenAddress = "127.0.0.1";
|
|
||||||
};
|
|
||||||
blackbox = {
|
|
||||||
enable = true;
|
|
||||||
listenAddress = "127.0.0.1";
|
|
||||||
configFile = ./blackbox.yml;
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
scrapeConfigs = [
|
|
||||||
{
|
|
||||||
job_name = "local";
|
|
||||||
static_configs = [{
|
|
||||||
targets = [
|
|
||||||
"127.0.0.1:9002" # Node exporter
|
|
||||||
#"127.0.0.1:9115" # Blackbox exporter
|
|
||||||
"127.0.0.1:9290" # IPMI exporter for local node
|
|
||||||
"127.0.0.1:9928" # UPC Qaire custom exporter
|
|
||||||
"127.0.0.1:9929" # Meteocat custom exporter
|
|
||||||
"127.0.0.1:9999" # Nix-daemon custom exporter
|
|
||||||
];
|
|
||||||
}];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
job_name = "blackbox-http";
|
|
||||||
metrics_path = "/probe";
|
|
||||||
params = { module = [ "http_2xx" ]; };
|
|
||||||
static_configs = [{
|
|
||||||
targets = [
|
|
||||||
"https://www.google.com/robots.txt"
|
|
||||||
"https://pm.bsc.es/"
|
|
||||||
"https://pm.bsc.es/gitlab/"
|
|
||||||
"https://jungle.bsc.es/"
|
|
||||||
"https://gitlab.bsc.es/"
|
|
||||||
];
|
|
||||||
}];
|
|
||||||
relabel_configs = [
|
|
||||||
{
|
|
||||||
# Takes the address and sets it in the "target=<xyz>" URL parameter
|
|
||||||
source_labels = [ "__address__" ];
|
|
||||||
target_label = "__param_target";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Sets the "instance" label with the remote host we are querying
|
|
||||||
source_labels = [ "__param_target" ];
|
|
||||||
target_label = "instance";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Shows the host target address instead of the blackbox address
|
|
||||||
target_label = "__address__";
|
|
||||||
replacement = "127.0.0.1:9115";
|
|
||||||
}
|
|
||||||
];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
job_name = "blackbox-icmp";
|
|
||||||
metrics_path = "/probe";
|
|
||||||
params = { module = [ "icmp" ]; };
|
|
||||||
static_configs = [{
|
|
||||||
targets = [
|
|
||||||
"1.1.1.1"
|
|
||||||
"8.8.8.8"
|
|
||||||
"ssfhead"
|
|
||||||
"raccoon"
|
|
||||||
"anella-bsc.cesca.cat"
|
|
||||||
"upc-anella.cesca.cat"
|
|
||||||
"fox.ac.upc.edu"
|
|
||||||
"fox-ipmi.ac.upc.edu"
|
|
||||||
"arenys5.ac.upc.edu"
|
|
||||||
"arenys0-2.ac.upc.edu"
|
|
||||||
"epi01.bsc.es"
|
|
||||||
"axle.bsc.es"
|
|
||||||
];
|
|
||||||
}];
|
|
||||||
relabel_configs = [
|
|
||||||
{
|
|
||||||
# Takes the address and sets it in the "target=<xyz>" URL parameter
|
|
||||||
source_labels = [ "__address__" ];
|
|
||||||
target_label = "__param_target";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Sets the "instance" label with the remote host we are querying
|
|
||||||
source_labels = [ "__param_target" ];
|
|
||||||
target_label = "instance";
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Shows the host target address instead of the blackbox address
|
|
||||||
target_label = "__address__";
|
|
||||||
replacement = "127.0.0.1:9115";
|
|
||||||
}
|
|
||||||
];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
job_name = "ipmi-raccoon";
|
|
||||||
metrics_path = "/ipmi";
|
|
||||||
static_configs = [
|
|
||||||
{ targets = [ "127.0.0.1:9290" ]; }
|
|
||||||
];
|
|
||||||
params = {
|
|
||||||
target = [ "raccoon-ipmi" ];
|
|
||||||
module = [ "raccoon" ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
{
|
|
||||||
job_name = "ipmi-fox";
|
|
||||||
metrics_path = "/ipmi";
|
|
||||||
static_configs = [
|
|
||||||
{ targets = [ "127.0.0.1:9290" ]; }
|
|
||||||
];
|
|
||||||
params = {
|
|
||||||
target = [ "fox-ipmi.ac.upc.edu" ];
|
|
||||||
module = [ "fox" ];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,79 +0,0 @@
|
|||||||
{ theFlake, pkgs, ... }:
|
|
||||||
let
|
|
||||||
website = pkgs.stdenv.mkDerivation {
|
|
||||||
name = "jungle-web";
|
|
||||||
src = pkgs.fetchgit {
|
|
||||||
url = "https://jungle.bsc.es/git/rarias/jungle-website.git";
|
|
||||||
rev = "739bf0175a7f05380fe7ad7023ff1d60db1710e1";
|
|
||||||
hash = "sha256-ea5DzhYTzZ9TmqD+x95rdNdLbxPnBluqlYH2NmBYmc4=";
|
|
||||||
};
|
|
||||||
buildInputs = [ pkgs.hugo ];
|
|
||||||
buildPhase = ''
|
|
||||||
rm -rf public/
|
|
||||||
hugo
|
|
||||||
'';
|
|
||||||
installPhase = ''
|
|
||||||
cp -r public $out
|
|
||||||
'';
|
|
||||||
# Don't mess doc/
|
|
||||||
dontFixup = true;
|
|
||||||
};
|
|
||||||
in
|
|
||||||
{
|
|
||||||
networking.firewall.allowedTCPPorts = [ 80 ];
|
|
||||||
services.nginx = {
|
|
||||||
enable = true;
|
|
||||||
virtualHosts."jungle.bsc.es" = {
|
|
||||||
root = "${website}";
|
|
||||||
listen = [
|
|
||||||
{
|
|
||||||
addr = "0.0.0.0";
|
|
||||||
port = 80;
|
|
||||||
}
|
|
||||||
];
|
|
||||||
extraConfig = ''
|
|
||||||
set_real_ip_from 127.0.0.1;
|
|
||||||
set_real_ip_from 84.88.52.107;
|
|
||||||
real_ip_recursive on;
|
|
||||||
real_ip_header X-Forwarded-For;
|
|
||||||
|
|
||||||
location /git {
|
|
||||||
rewrite ^/git$ / break;
|
|
||||||
rewrite ^/git/(.*) /$1 break;
|
|
||||||
proxy_pass http://127.0.0.1:3000;
|
|
||||||
proxy_redirect http:// $scheme://;
|
|
||||||
}
|
|
||||||
location /cache {
|
|
||||||
rewrite ^/cache/(.*) /$1 break;
|
|
||||||
proxy_pass http://127.0.0.1:5000;
|
|
||||||
proxy_redirect http:// $scheme://;
|
|
||||||
}
|
|
||||||
location /lists {
|
|
||||||
proxy_pass http://127.0.0.1:8081;
|
|
||||||
proxy_redirect http:// $scheme://;
|
|
||||||
}
|
|
||||||
location /grafana {
|
|
||||||
proxy_pass http://127.0.0.1:2342;
|
|
||||||
proxy_redirect http:// $scheme://;
|
|
||||||
proxy_set_header Host $host;
|
|
||||||
# Websockets
|
|
||||||
proxy_http_version 1.1;
|
|
||||||
proxy_set_header Upgrade $http_upgrade;
|
|
||||||
proxy_set_header Connection "upgrade";
|
|
||||||
}
|
|
||||||
location ~ ^/~(.+?)(/.*)?$ {
|
|
||||||
alias /vault/home/$1/public_html$2;
|
|
||||||
index index.html index.htm;
|
|
||||||
autoindex on;
|
|
||||||
absolute_redirect off;
|
|
||||||
}
|
|
||||||
location /p/ {
|
|
||||||
alias /var/lib/p/;
|
|
||||||
}
|
|
||||||
location /pub/ {
|
|
||||||
alias /vault/pub/;
|
|
||||||
}
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@ -1,16 +0,0 @@
|
|||||||
{ config, ... }:
|
|
||||||
|
|
||||||
{
|
|
||||||
age.secrets.nixServe.file = ../../secrets/nix-serve.age;
|
|
||||||
|
|
||||||
services.nix-serve = {
|
|
||||||
enable = true;
|
|
||||||
# Only listen locally, as we serve it via ssh
|
|
||||||
bindAddress = "127.0.0.1";
|
|
||||||
port = 5000;
|
|
||||||
|
|
||||||
secretKeyFile = config.age.secrets.nixServe.path;
|
|
||||||
# Public key:
|
|
||||||
# jungle.bsc.es:pEc7MlAT0HEwLQYPtpkPLwRsGf80ZI26aj29zMw/HH0=
|
|
||||||
};
|
|
||||||
}
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user