How to setup Mattermost alerts on AWS Cloud Events

Is it possible to connect your AWS Cloud events to the communicator used by your team? Yes! In fact there are many ways to do this and of course good enough reasons to receive notifications and/or alerts regarding application state, resource utilisation or any other critical events that may need immediate developer attention. In this post, I’m going to walk you through all that is needed to integrate Mattermost with AWS Simple Notification Service and receive notifications for critical events. Once this is done, you can sit back and focus on continuing that spike, finish reading that book you are trying to find time for or spend time with your loved ones as Mattermost will grab your attention when needed!

What you’ll need:

Access to AWS Management Console

Terraform and Python installed on your linux/unix host

Of course a mattermost channel

Assuming you have your AWS Management Console setup and you can access the different services, let us take a look how messages published to a topic on Simple Notification Service are sent to its subscribers. Simple Notification Service is a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.

In our scenario we would be focussing on application-to-application(A2A) messaging. Even in an A2A you’ll still need the basic elements corresponding to a messaging service, viz. Publisher, Subscriber and a Topic, only in this case both the publisher(s) and subscriber(s) are applications. When an application subscribes to a topic of its interest, AWS SNS would send a subscription confirmation request. You can publish a message through A2A by using a combination of endpoint and protocol that suits your needs. You have to specify the endpoint that you want to receive notifications on and the protocol that you wish to tunnel your message through. When it comes to A2A messaging, these are the protocols supported by AWS SNS:

Kinesis Data Firehose

AWS Lambda Functions

AWS SQS Queues

Public HTTP/S endpoints

With the above knowledge, sending messages to a mattermost channel with the HTTP/S endpoints should be pretty simple right? We can create a webhook url associated with the mattermost channel, plug it to the http/s endpoint and publish messages! Partially there I would say. Why you ask, because while you can leverage the http/s protocol to send messages, some webhooks, like the mattermost one, expect JSON key-value pairs that Amazon SNS doesn’t support when confirming the HTTP(S) subscription. This means you cannot use the http/s endpoint in AWS SNS to send messages to your mattermost channel. Oh well, don’t worry cause I’ve covered how to fix this, read on.

Resolution

What we need here is a computing service that can transform the SNS payload into a format acceptable to the mattermost webhook. This is where we’ll need an AWS Lambda Function, as it provides us with an event-driven, serverless computing platform and automatically manages the computing resources required by the code.

If you’re still with me, thank you for your patience :) Alright, so putting it all together, here’s what we need to do:

1. Create an SNS topic

resource "aws_sns_topic" "my_awesome_topic" {
  name = "my_awesome_topic"
  display_name = "My Awesome Topic"
}

2. Setup the Lambda Function

a) Create a Lambda function for the payload transformation logic

data "archive_file" "lambda_zip_file" {
  type        = "zip"
  output_path = "tmp/lambda_zip_dir_mattermost_alert.zip"
  source_dir  = "src"
}

resource "aws_lambda_function" "test_mattermost_alert" {
  function_name    = "test_mattermost_alert"
  role             = "arn:aws:iam::${var.AWS_account_id}:role/lambda_execution_role"
  handler          = "main.mattermost_alert_handler"
  filename         = data.archive_file.lambda_zip_file.output_path
  source_code_hash = data.archive_file.lambda_zip_file.output_base64sha256
  runtime          = "python3.7"
  description      = "A lambda function that sends alerts to mattermost."
  environment {
    variables = {
      MATTERMOST_WEBHOOK = var.mattermost_webhook
    }
  }
}

b) Setup the Lambda Runtime Code:
Mattermost incoming webhooks expect a JSON request with a message string corresponding to a “text” key. For more information, see Setting up an Incoming Webhook on the official Mattermost Documentation.

Note: In this example function code for Mattermost incoming webhooks, replace os.environ[“MATTERMOST_WEBHOOK”] with your webhook URL.

import urllib3
import os
import json

http = urllib3.PoolManager()

def mattermost_alert_handler(event):
    MATTERMOST_WEBHOOK = os.environ["MATTERMOST_WEBHOOK"]
    msg = {
        "text": event['Records'][0]['Sns']['Message']
    }
    encoded_msg = json.dumps(msg).encode('utf-8')
    resp = http.request('POST',MATTERMOST_WEBHOOK, body=encoded_msg)
    print({
        "message": event['Records'][0]['Sns']['Message'],
        "status_code": resp.status,
        "response": resp.data
    })

3. Identity and Access Management (IAM) Roles

Create an IAM Role with “lambda.amazonaws.com” as the Principal and associate the AWSLambdaBasicExecutionRole Policy with it. If you would like to enable logging, create the relevant policies and attach it to the IAM role as well.

resource "aws_iam_role" "lambda_execution_role" {
  name = "lambda_execution_role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "lambda_execution_role_policy_attachment" {
  role       = aws_iam_role.lambda_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

4. Setup SNS Subscription
Create an SNS topic subscription and subscribe to the topic created in Step#1.
Add protocol = “lambda” and endpoint = ARN of your lambda function created in Step#2

resource "aws_sns_topic_subscription" "mattermost-subscriber" {
  protocol   = "lambda"
  endpoint   = aws_lambda_function.test_mattermost_alert.arn
  topic_arn  = aws_sns_topic.my_awesome_topic.arn
  depends_on = [aws_sns_topic.my_awesome_topic]
}

5. Setup Lambda Invoke Permissions
Ensure to give SNS permission to invoke your lambda function. If you’re familiar with the AWS Management Console, this step would be attaching a trigger configuration to your lambda function.

resource "aws_lambda_permission" "aws_config_lambda_permission" {
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.test_mattermost_alert.arn
  source_arn = aws_sns_topic.my_awesome_topic.arn
  principal     = "sns.amazonaws.com"
  statement_id  = "AllowExecutionFromSNS"
}

With your lambda function subscribed to your SNS topic, messages published to the topic are forwarded to the function, and then to your webhook. And you are now ready to terraform init => validate => plan => apply!