roddie.digital / Alternative Email Route for Nagios Notifications

Alternative Email Route for Nagios Notifications

Last updated 25 Feb 2021

In my work environment, we use Nagios to monitor a virtual email appliance which is used for routing email alerts, including the error alerts it sends itself! Naturally this isn't much use when it tries to send an email alert to say there's a problem with email alerts so I needed a different way to get alerted. That's why I configured the SNMP traps and alerts for Nagios - unfortunately, the Nagios Core server also uses the same email route to send notifications for the SNMP service checks so I configured an alternative route for this host and selected services.

Using the sendemail command (available from the RHEL repos) and newly-created email account just for this, I configured two new notify commands in /etc/nagios/objects/commands.cfg (one for hosts and one for services):

# Alternative route used for relay problems
define command {
  command_name    notify-host-by-email-alt
  command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/bin/sendemail -u "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" -f $USER9$ -t $CONTACTEMAIL$ -s "mail.domain.com:587" -o tls=yes -xu $USER9$ -xp $USER10$ -l /var/log/sendemail.log
}

define command {
  command_name    notify-service-by-email-alt
  command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /usr/bin/sendemail -u "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" -f $USER9$ -t $CONTACTEMAIL$ -s "mail.domain.com:587" -o tls=yes -xu $USER9$ -xp $USER10$ -l /var/log/sendemail.log
}

The -t, -s, -o and -l options are recipient, SMTP, TLS and log path respectively while $USER9$ and $USER10$ represent username and password stored in Nagios macros in /etc/nagios/private/resource.cfg eg:

$USER9$="email.alerts@domain.com"
$USER10$="hunter2"

Bear in mind if you're using a service like Office365 then -f needs to specify the from/sender email address and -xu needs to be the username/UPN, which may not be the same. Then we just need a contact defined in /etc/nagios/objects/contacts.cfg that we can add to the contact directive in the host and/or service definitions of the mail appliance (or making it part of a contact group if you have multiple people you need to send to):

define contact {
  contact_name                    email-alt
  use                             generic-contact
  alias                           Email Alternative
  email                           my.email@domain.com
  host_notification_commands      notify-host-by-email-alt
  service_notification_commands   notify-service-by-email-alt
}

We can also send text messages via email using BT's Soprano service so I've set up host and service commands with shorter messages and contacts with the email addresses set up in the format needed (eg 07000123456@sms.domain.com).

# SMS notification command definitions
define command {
  command_name    notify-host-by-sms
  command_line    /usr/bin/printf "%b" "$NOTIFICATIONTYPE$ ALERT: $HOSTALIAS$ is $HOSTSTATE$ ($SHORTDATETIME$)" | /usr/bin/sendemail -f $USER9$ -t $CONTACTEMAIL$ -s "mail.domain.com:587" -o tls=yes -xu $USER9$ -xp $USER10$ -l /var/log/sendemail.log
}

define command {
  command_name    notify-service-by-sms
  command_line    /usr/bin/printf "%b" "$NOTIFICATIONTYPE$ ALERT: $SERVICEDESC$ on $HOSTALIAS$ is $SERVICESTATE$ ($SHORTDATETIME$). $SERVICEOUTPUT$" | /usr/bin/sendemail -f $USER9$ -t $CONTACTEMAIL$ -s "mail.domain.com:587" -o tls=yes -xu $USER9$ -xp $USER10$ -l /var/log/sendemail.log
}

# Mobile contact
define contact {
  contact_name                    roddie-mobile
  use                             generic-contact
  alias                           Roddie Mobile
  email                           07000123456@sms.domain.com
  host_notification_commands      notify-host-by-sms
  service_notification_commands   notify-service-by-sms
}

Now we can get concise text alerts which look like this:

PROBLEM ALERT: Queued Messages on SERVERNAME01 is WARNING (13-02-2021 19:39:01). SNMP WARNING - *453*

Version 4.3.3 of Nagios Core introduced the macros $HOSTINFOURL$ and $SERVICEINFOURL$ which you can use in your commands if you set the base URL of your Nagios site in nagios.cfg.

# Allow the $HOSTINFOURL$ and $SERVICEINFOURL$  macros to be used for notifications

website_url=https://domain.com/nagios

Now if you use these macros in your notification commands it'll add a link to the page for that host or service in the email. I've also added the $NOTIFICATIONCOMMENT$ and $NOTIFICATIONAUTHOR$ macros so that if someone acknowledges a problem, the email will include the comment and the name of the person who did it. This way other recipients can see who acknowledged it and why without having to check Nagios.

define command {
  command_name    notify-service-by-email
  command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n\nService Status Details:\n\n$SERVICEINFOURL$\n\n$NOTIFICATIONCOMMENT$\n\n$NOTIFICATIONAUTHOR$" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
}

Another useful macro for notifications is $SERVICENOTES$ which will include the text from the notes directive of your service definition. I find this useful when sending notifications to colleagues who don't use Nagios to add a description of the service check in case they aren't familiar with it.

define command {
  command_name    notify-service-by-email-notes
  command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n\nService Details:\n\n$SERVICENOTES$\n\n$NOTIFICATIONCOMMENT$\n\n$NOTIFICATIONAUTHOR$" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
}

Return to roddie.digital / top

/ Nagios ^

Alternative Email Route for Nagios Notifications