Disabling terminated ASG hosts in Zabbix with SNS/SQS

Big picture:

ASG terminates instance => client's SNS => our SQS => Python script on EC2 instance takes SQS message and sends request to OUR Zabbix via Zabbix API  to disable monitoring of the terminated host

0. Create SQS queue on OUR side (SQSZabbix in this case). Copy arn name. Permissions will be set later.

On client's side:

1. Services -> SNS

2. 'Topics' from left panel.

3. 'Create new topic' button in middle

4. Topic name: zabbixMonitor

5. 'Create topic' button in bottom right.

6. Select created topic with checkbox. 'Actions' -> 'Subscribe to topic'.

7. Topic ARN: take from client's topic (arn:aws:sns:eu-west-1:161037739988:zabbixMonitor) Protocol: Amazon SQS

8. Endpoint: arn:aws:sqs:us-east-1:530860234560:sqsZabbix  - OUR side, SQS Queue, created recently (section 0)

9. Enter OUR AWS Account.

10. Services -> SQS -> SQSZabbix -> Permissions

11. Add a Permission

         - Principal: Everybody

         - Actions: Send Message

         - Add Conditions

         - Condition: ArnEquals

         - Key: aws:SourceArn

         - Value: (Customer SNS Topic ARN)

         Add Condition

12. Add Permission

13. Queue Actions -> View/Delete Messages

14. Start Polling for Messages

15. Find recent message starting "{ "Type" : "SubscriptionConfirmation""

16. More Details

17. Copy Url following "SubscribeURL" and paste in browser to confirm SNS topic.

---Configure ASG notifications---

18. Select Auto-Scaling Group in Customer Account -> Notifications -> Create notification

19. - Send a notification to:zabbixMonitor (arn:aws:sqs:us-east-1:530860234560:SQSZabbix))

    - Whenever instances: terminate

20. Save

asg_term.py - python script is in the attachment, imports zabbix_conf.py - config file with parameters

#!/bin/env python2.7

import boto3

import json, time

from zabbix.api import ZabbixAPI

# Import config file

import asg_term_conf

# Define SQS client and queue

sqs_client = boto3.resource('sqs',region_name=asg_term_conf.region_name)

sqs_queue = sqs_client.get_queue_by_name(QueueName=asg_term_conf.QueueName)

# Declare Zabbix api object and login

zapi = ZabbixAPI(url=asg_term_conf.ZabbixServer, user=asg_term_conf.ZabbixUser, password=asg_term_conf.ZabbixPassword)

# Return a dict of EC2 instance ids that need to be disabled in Zabbix associated with and SQS message object

# Accepts list of messages from an SQS queue

def GetDisableInstances(SQSMessages):

DisableInstances={}

for message in SQSMessages:

try:

jsonBody = json.loads(message.body)

jsonBody = json.loads(jsonBody['Message'])

if str(jsonBody['Event']) == 'autoscaling:EC2_INSTANCE_TERMINATE':

DisableInstances[jsonBody['EC2InstanceId']] = message

except Exception as e:

continue

return DisableInstances

# Find a Zabbix host by instance id. Return list of Zabbix hostids

def GetZabbixHosts(InstanceID):

hosts = zapi.do_request('host.get',

{

'search': {'host' : InstanceID},

'output': 'extend'

})

return hosts['result']

# Disable all Zabbix hosts in provided array

# Returns true upon success

def DisableZabbixHosts(hosts):

for host in hosts:

# Disable host

DisabledHost = zapi.host.update(hostid=host['hostid'], status=1, output='extend')

if DisabledHost['hostids'][0] == host['hostid']:

return True

else:

return False

# For each instance terminated by ASG, disable on Zabbix

# and remove from queue

def main():

count=0

while True:

# Generate queue of messages using long polling

SQSMessages = sqs_queue.receive_messages(MaxNumberOfMessages=10,WaitTimeSeconds=20)

if SQSMessages:

for InstanceID, message in GetDisableInstances(SQSMessages).iteritems():

# Disable instance in Zabbix

# If succeeded to remove instances, then remove message about this instance from SQS queue

if DisableZabbixHosts(GetZabbixHosts(InstanceID)):

message.delete()

else:

# Queue empty. Wait a few seconds

time.sleep(20)

if __name__ == "__main__":

    main()

zabbix_conf.py contains:

ZabbixServer = 'zabbixserverurl'

ZabbixUser = 'USER'

ZabbixPassword = 'PASSWORD'

region_name = 'eu-west-1'

QueueName = 'SQSZabbix'