roddie.digital

Alternative Email Route for Nagios Notifications

In my work environment, we use Nagios to monitor a virtual email appliance which is used for routing email alerts, including the error alerts it sends itself! Naturally this isn't much use if it's a critical problem in the chain of infrastructure that is having a problem so I needed a different way to get alerted. That's why I configured the SNMP traps and alerts for Nagios - unfortunately, the Nagios Core server also uses the same email route to send notifications so I needed to configure an alternative route for this host and selected services.

Using the sendemail command (available from the RHEL repos) and newly-created email account just for this, I configured two new notify commands in /etc/nagios/objects/commands.cfg (one for hosts and one for services):

# alternative route used for relay problems

define command{
        command_name    notify-host-by-email-alt
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/bin/sendemail -u "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" -f $USER9$ -t $CONTACTEMAIL$ -s "mail.domain.com:587" -o tls=yes -xu $USER9$ -xp $USER10$ -l /var/log/sendemail.log
        }

define command{
        command_name    notify-service-by-email-alt
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /usr/bin/sendemail -u "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" -f $USER9$ -t $CONTACTEMAIL$ -s "mail.domain.com:587" -o tls=yes -xu $USER9$ -xp $USER10$ -l /var/log/sendemail.log
        }

The -t, -s, -o and -l options are recipient, SMTP, TLS and log path respectively while $USER9$ and $USER10 represent username and password stored in Nagios macros in /etc/nagios/private/resource.cfg eg:

$USER9$="email.alerts@domain.com"
$USER10$="hunter2"

Then we just need a contact defined in /etc/nagios/objects/contacts.cfg that we can add to the contact directive in the host and/or service definitions of the mail appliance (or making it part of a contact group if you have multiple people you need to send to):

define contact{
        contact_name                    email-alt
        use                             generic-contact
        alias                           Email Alternative
        email                           my.email@domain.com
        host_notification_commands      notify-host-by-email-alt
        service_notification_commands   notify-service-by-email-alt
        }