post-image

SMS Grafana Alerts with AWS SNS

Using AWS SNS, Lambda and API Gateway

Alerting is an important part of any successful DevOps culture and in incident response practice. Most teams setup some sort of alerting system usually in Slack, but sometime you may want alerts when you are away from Slack. One way to get alerts is via SMS in Grafana. Grafana is a very popular monitoring tool for visualizing, alerting and exploring metrics. There are many different options to create alerts but there is no straight forward way to integrate SMS to Grafana, but we can use web hooks to trigger SMS notification. To be able to trigger SMS notification using web hook we will have to create a web hook and use a tool that will be able to send SMS. The tools we will be using is AWS SNS that is a notification service, AWS API Gateway to create a url for our AWS Lambda Function that is going to trigger the SNS.

What you'll need:

  • AWS Account
  • Grafana (either running on a server or locally, you can use the docker script below to setup Grafana locally )

Setup Grafana with Docker:

If you don't have a Grafana server you can easily setup with docker or docker-compose. (Click here for Docker Installation)

You can use the following docker-compose file

docker-compose.yml
						
version: '3'
services:
    grafana:
      image: grafana/grafana:latest
      ports:
        - "3000:3000"
      volumes:
        - grafana:/var/lib/grafana
      restart: always
volumes:
    grafana:
					

Or just with two simple docker commands

						
docker volume create grafana
docker run -d -p 3000:3000 --name=grafana -v grafana:/var/lib/grafana grafana/grafana
					

Username and password for Grafana are by default is admin:admin

Create IAM Role for Lambda

Now lets get started with AWS part. We will start by creating AWS IAM Role for our Lambda Function. This Role should be able to use the call SNS service provided by AWS.

  • Go to IAM
  • Create a New Role by going to Roles and clicking Create role
  • post-image
  • Select Lambda as use case and then continue to policies
  • post-image
  • Search for SNS, for this demo select SNSFullAccess
  • post-image
  • Create a Tag, it up to you but a good AWS Practice is to add some sort of Tag
  • post-image
  • Go to Review, and give it a Role Name and create role
  • post-image

Create the Lambda Function

Now we will create the Lambda function that will trigger SNS. First step we will go into AWS Lambda

post-image

The then create a function name, select python as runtime, as seen here I am using python 3.7. For the execution role select the IAM role we just created.

post-image

Then create the function and add the following code in to the Lambda online text editor and then click save.

						
import boto3
import json

CLIENT = boto3.client("sns")

def lambda_handler(json_input, context):
    json_dump = json.dumps(json_input)
    request = json.loads(json_dump)
    message = request["message"]
    numbers = ["xxxxxxx"] # Change this to your number
    for number in numbers:
        response = CLIENT.publish(
            PhoneNumber=number,
            Message=message,
            MessageAttributes={
                'AWS.SNS.SMS.SenderID':
                {
                    'DataType': 'String',
                    'StringValue': 'Grafana'
                }
            }
        )
    return response
					

This python script is written to loop an array of numbers, so you can add more than one number to send an SMS to. Remember to add your own phone number you have to give your country code. Do notice that I am importing boto3 without installing boto3 first, that is because boto3 is already provided on AWS Lambda function for us.

Connect Lambda Function with API Gateway

The final step of setting up the AWS part is to setup API Gateway and then connect it with Lambda function. Now we go into API Gateway in AWS, if you are creating API Gateway for the first time you will a screen like below else you will see list of previous api and the create API button should be on the top right

post-image

Select Build REST API the one on the left, the one of the right is for private VPC, which we aren't doing right now. When creating a new a API for the first time AWS likes to give an example but we won't be using the example, instead select New API and give it a API name and then Create API

post-image

After selecting Create API you are redirected to a page similar to below. Select Actions, while `\` is selected. Click create method.

post-image

Then you are give a another drop, select POST for its Method

post-image

Now we have to setup POST. To setup POST we simply, give the name of the lambda function we created earlier. This name is based on the function name we had set. While typing the function name it should show as a option. Then make sure to save this setting.

post-image

Now select Actions again and select Deploy API.

post-image

Give a API a stage, for this example I have create a new stage with name demo. Then select Deploy.

post-image

Finally you are redirect to this page and in the top you should see the api url. This is the webhook we will use in Grafana

post-image

Setting SMS alert in Grafana

Now we have finally come to the final step of setting the alarm. Now log into Grafana. Then select Notification Channels as shown below.

post-image

If you don't already have any channels it should show as below, and select Add Channel. If you already have other channels then you can select Add Channel which should be on top right.

post-image

Then give the channel a name, and select the type to be webhook. Put the API webhook url we had just created and set the HTTP Method to be POST. Then if you followed all the steps correctly you can do a Test and you should get a SMS on your phone.

post-image

Conclusion

I hope by now you are able to create SMS Grafana Alerts with AWS SNS, or at least you learned something. One the important take away I can say this shows how you can use different AWS service together. If you have any questions or suggestion you can contact me at hi@sidops.dev