Grafana email alerts depend on login #90

New Issue

rarias · 2025-03-12T11:17:24+01:00

rarias commented

2025-03-12 11:17:24 +01:00

We need to make grafana alerts go through our internal tunnel, otherwise we need to rely on the login to wake up in a good mood to be able to contact the BSC mail server:

Mar 12 11:13:02 hut grafana-start[1963]: logger=ngalert.notifier.alertmanager org=1 t=2025-03-12T11:13:02.827412008+01:00 level=error component=alertmanager orgID=1 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="grafana-default-email/email[0]: notify retry canceled due to unrecoverable error after 1 attempts: failed to send email: dial tcp: lookup mail.bsc.es: i/o timeout"

Since the hot wave the login has not recovered connectivity to the outside, so we cannot rely on that to send emergency messages:

Slack alerts are fine.

We need to make grafana alerts go through our internal tunnel, otherwise we need to rely on the login to wake up in a good mood to be able to contact the BSC mail server: ``` Mar 12 11:13:02 hut grafana-start[1963]: logger=ngalert.notifier.alertmanager org=1 t=2025-03-12T11:13:02.827412008+01:00 level=error component=alertmanager orgID=1 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="grafana-default-email/email[0]: notify retry canceled due to unrecoverable error after 1 attempts: failed to send email: dial tcp: lookup mail.bsc.es: i/o timeout" ``` Since the hot wave the login has not recovered connectivity to the outside, so we cannot rely on that to send emergency messages: ![image](/attachments/21e56da3-23b2-47ab-b894-814053b83b5e) Slack alerts are fine.

image.png

15 KiB

rarias added the

ops

label 2025-03-12 11:17:24 +01:00

rarias commented

2025-03-13 17:04:06 +01:00

Same for public-inbox, cannot contact the IMAP server. See #91

Sign in to join this conversation.