rarias/jungle
2
0
Fork 3
Code Issues 57 Pull Requests 7 Actions Packages Projects Releases Wiki Activity

Add custom nix-daemon exporter #99

Manually merged
rarias merged 1 commits from monitor-nix-daemon into master 2025-05-29 12:58:47 +02:00
Conversation 5 Commits 1 Files Changed 3 +51
rarias commented 2025-04-30 13:12:08 +02:00
Owner
Copy Link

Allows us to see which derivations are being built in realtime. It is a
bit of a hack, but it seems to work. We simply look at the environment
of the child processes of nix-daemon (usually bash) and then look for
the $name variable which should hold the current derivation being
built. Needs root to be able to read the environ file of the different
nix-daemon processes as they are owned by the nixbld* users.

See: https://discourse.nixos.org/t/query-ongoing-builds/23486

See in action: https://jungle.bsc.es/grafana/d/EKkKVvLVz/hut?orgId=1&from=now-6h&to=now&timezone=browser&refresh=5s&viewPanel=panel-47

Allows us to see which derivations are being built in realtime. It is a bit of a hack, but it seems to work. We simply look at the environment of the child processes of nix-daemon (usually bash) and then look for the $name variable which should hold the current derivation being built. Needs root to be able to read the environ file of the different nix-daemon processes as they are owned by the nixbld* users. See: https://discourse.nixos.org/t/query-ongoing-builds/23486 See in action: https://jungle.bsc.es/grafana/d/EKkKVvLVz/hut?orgId=1&from=now-6h&to=now&timezone=browser&refresh=5s&viewPanel=panel-47
rarias added 1 commit 2025-04-30 13:12:09 +02:00
Add custom nix-daemon exporter 13e084f34f 
Allows us to see which derivations are being built in realtime. It is a
bit of a hack, but it seems to work. We simply look at the environment
of the child processes of nix-daemon (usually bash) and then look for
the $name variable which should hold the current derivation being
built. Needs root to be able to read the environ file of the different
nix-daemon processes as they are owned by the nixbld* users.

See: https://discourse.nixos.org/t/query-ongoing-builds/23486
rarias requested review from arocanon 2025-04-30 13:12:22 +02:00
rarias requested review from abonerib 2025-04-30 13:12:22 +02:00
abonerib approved these changes 2025-05-05 12:09:37 +02:00
abonerib left a comment
Collaborator
Copy Link

Left some suggestions on rip-grep, the rest LGTM (although it's a little scary to run this as root given that name is arbitrary user input)

Left some suggestions on rip-grep, the rest LGTM (although it's a little scary to run this as root given that name is arbitrary user input)
m/hut/nix-daemon-builds.sh Outdated
@@ -0,0 +18,4 @@
EOF
for pid in $pids2; do
name=$(cat /proc/$pid/environ 2>/dev/null | tr '\0' '\n' | rg "^name=(.+)" - --replace '$1')
abonerib commented 2025-05-05 11:55:54 +02:00
Collaborator
Copy Link

I think we could use --null-data in rg and avoid tr. There is also --max-count 1

I think we could use `--null-data` in `rg` and avoid `tr`. There is also `--max-count 1`
rarias commented 2025-05-29 12:42:44 +02:00
Author
Owner
Copy Link

I'm not able to make it work with --null-data:

m/hut/nix-daemon-builds.sh: line 21: warning: command substitution: ignored null byte in input
nix_daemon_build{user="nixbld1",name="name=wxparaver-4.11.2"} 1

Maybe you can open another MR to improve it.

Edit: Reproducer:

$ printf 'bar=baz\0name=foo\0version=123\0' | rg --null-data "^name=(.+)" - --replace '$1'
name=foo
I'm not able to make it work with --null-data: ``` m/hut/nix-daemon-builds.sh: line 21: warning: command substitution: ignored null byte in input nix_daemon_build{user="nixbld1",name="name=wxparaver-4.11.2"} 1 ``` Maybe you can open another MR to improve it. Edit: Reproducer: ``` $ printf 'bar=baz\0name=foo\0version=123\0' | rg --null-data "^name=(.+)" - --replace '$1' name=foo ```
rarias commented 2025-05-05 12:34:59 +02:00
Author
Owner
Copy Link

although it's a little scary to run this as root given that name is arbitrary user input

Yeah I know. Any ideas how to improve it?

> although it's a little scary to run this as root given that name is arbitrary user input Yeah I know. Any ideas how to improve it?
abonerib commented 2025-05-05 15:25:53 +02:00
Collaborator
Copy Link

although it's a little scary to run this as root given that name is arbitrary user input

Yeah I know. Any ideas how to improve it?

Not really. I was thinking of using {pre,post}-build-hooks but I don't think it is viable, since it would require some form of state management and produce a lot of noise.

> > although it's a little scary to run this as root given that name is arbitrary user input > > Yeah I know. Any ideas how to improve it? > Not really. I was thinking of using `{pre,post}-build-hooks` but I don't think it is viable, since it would require some form of state management and produce a lot of noise.
rarias commented 2025-05-05 15:57:37 +02:00
Author
Owner
Copy Link

Not really. I was thinking of using {pre,post}-build-hooks but I don't think it is viable, since it would require some form of state management and produce a lot of noise.

I think this could be added to the nix binary directly, so it reports the stats over a TCP port, but that won't be done in the short term. If you don't see any obvious problem with the current approach I'll be willing to merge it, as I think the benefits outweight the potential drawbacks.

> Not really. I was thinking of using `{pre,post}-build-hooks` but I don't think it is viable, since it would require some form of state management and produce a lot of noise. I think this could be added to the nix binary directly, so it reports the stats over a TCP port, but that won't be done in the short term. If you don't see any obvious problem with the current approach I'll be willing to merge it, as I think the benefits outweight the potential drawbacks.
abonerib reviewed 2025-05-05 16:10:47 +02:00
m/hut/nix-daemon-builds.sh Outdated
@@ -0,0 +21,4 @@
name=$(cat /proc/$pid/environ 2>/dev/null | tr '\0' '\n' | rg "^name=(.+)" - --replace '$1')
user=$(ps -o uname= -p "$pid")
if [ -n "$name" -a -n "$user" ]; then
printf 'nix_daemon_build{user="%s",name="%s"} 1\n' "$user" "$name"
abonerib commented 2025-05-05 16:10:47 +02:00
Collaborator
Copy Link

I would pass name through tr -dc ' [:alnum:]_\-\.' to avoid getting garbage in grafana.

I would pass name through `tr -dc ' [:alnum:]_\-\.'` to avoid getting garbage in grafana.
rarias commented 2025-05-29 12:47:00 +02:00
Author
Owner
Copy Link

Done

Done
rarias marked this conversation as resolved
rarias force-pushed monitor-nix-daemon from 13e084f34f to c12eceb897 2025-05-29 12:46:52 +02:00 Compare
rarias force-pushed monitor-nix-daemon from c12eceb897 to 1c15e77c83 2025-05-29 12:57:25 +02:00 Compare
rarias manually merged commit 1c15e77c83 into master 2025-05-29 12:58:47 +02:00
Sign in to join this conversation.
Reviewers
No Reviewers
arocanon
abonerib
Labels
Clear labels
bug
ceph
ci
config
cross
hw
io
kernel
mpi
net
nix
ops
repair
slurm
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
abonerib arocanon varcila rarias
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rarias/jungle#99
Delete Branch "monitor-nix-daemon"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?