Bash script old elasticsearch indices cleanup when not enough space (>90%) when watermark

Delete elk old indices when not enough space

#!/bin/bash

CAPACITY=`df | grep '/dev/disk1' | awk '{print $5}' | tr -d '%'`

while [  $CAPACITY -gt 90 ]; do

  INDEXNAME=`curl -s localhost:9200/_cat/indices/cloudwatch* | sort -n | head -1 | awk '{print $3}'`

  curl -XDELETE localhost:9200/$INDEXNAME

  echo "Deleted $INDEXNAME"

  CAPACITY=`df | grep '<change this to disk>' | awk '{print $5}' | tr -d '%'`

  sleep 60

done

Zabbix active agents auto-registration

edit zabbix-agent.conf

• ServerActive=ZabbixServerIP

• HostnameItem=system.hostname
• #to use system's hostname 
• or
• Hostname=ENTERHOSTNAME
• #to use specific hostname

• HostMetadata=windows 
• or
• HostMetadata=linux

• Timeout=30
• #use for possible long autodiscovery items (for example AWS Cloudwatch)

Add actions in zabbix. 

Linux host autoregistration

Host metadata like Linux

Add host
Add to host groups: Discovered hosts, Linux servers
Link to templates: Template_Linux_Active

Windows host autoregistration

Host metadata like windows

Add host

Add to host groups: Discovered hosts, Windows servers
Link to templates: Template_OS_Windows_Active

Manual Zabbix Agent Installation

This article describe how to manually install the Zabbix agent (verstion 2.2) on a Linux and Windows OS. Linux (RHEL/CentOS/Amazon) Install the repository configuration package. This package contains yum configuration files # rpm -ivh http://repo.zabbix.com/zabbix/2.2/rhel/6/x86_64/zabbix-release-2.2-1.el6.noarch.rpm Install Zabbix agent packages # yum install zabbix-agent Edit Zabbix agent Windows config file template. Make sure: ServerActive=<Private IP of Zabbix Server> Hostname=<Identifiable name that will match exactly how it is defined on Zabbix Server> HostMetadata=linux Copy Zabbix agent config file to /etc/zabbix/zabbix_agentd.conf Set Zabbix agent to start on startup # chkconfig zabbix-agent on Restart Zabbix Agent # service zabbix-agent restart Windows Download the pre-compiled Zabbix agent for Windows from http://www.zabbix.com/downloads/3.0.4/zabbix_agents_3.0.4.win.zip Create the directory C:\Program Files\Zabbix on the Windows instance. Edit Zabbix agent Windows config file template. Make sure: ServerActive=<Private IP of Zabbix Server> Hostname=<Identifiable name that will match exactly how it is defined on Zabbix Server> HostMetadata=windows Copy Zabbix agent config file to C:\Program Files\Zabbix\zabbix_agentd.win.conf Install the Zabbix agent by opening the command prompt and entering the following command "C:\Program Files\Zabbix\bin\win64\zabbix_agentd.exe" --install -c "C:\Program Files\Zabbix\conf\zabbix_agentd.win.conf" && net start "Zabbix Agent" && sc config "Zabbix Agent" start=auto zabbix_agentd.exe [1720]: service [Zabbix Agent] installed successfully zabbix_agentd.exe [1720]: event source [Zabbix Agent] installed successfully The Zabbix Agent service is starting. The Zabbix Agent service was started successfully. [SC] ChangeServiceConfig SUCCESS

Elasticsearch troubleshooting

Check ES state

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

Get list of all indices

curl 'localhost:9200/_cat/indices?v'

Check which is in red state
Delete indice

 curl -XDELETE 'http://localhost:9200/graylog2_1251'

When more than 90% space is used -ES starts to move indices and drops indices (sometimes) and crashes into RED state
In logs :
[WARN ][cluster.routing.allocation.decider] [Johann Schmidt] high disk watermark [10%] exceeded on
To disable this "watermark":

 curl -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.disk.threshold_enabled" : false
    }
}'        

              
response


 {"acknowledged":true,"persistent":{},"transient":{"cluster":{"routing":{"allocation":{"disk":{"threshold_enabled":"false"}}}}}}

Zabbix SMS via twilio

twilio.com - create account put some $$$$
Ger SID and Token in account

Use them in script to get the number to sent SMS from
1) Get the info about account
2) get available number
3) purchase via curl request
4) test



account_sid="ACXXXXXXXXXXXXXXXXXX"
auth_token="bXXXXXXXXXXX"


#####Get the account info - check if it is in trial mode
#####Trial accounts cannot send messages to unverified numbers;
#####In order to continue with trial account verify recipients' numbers at #####twilio.com/user/account/phone-numbers/verified,
#####or purchase a Twilio number to send messages to unverified numbers.


curl -G "https://api.twilio.com/2010-04-01/Accounts/${account_sid}.json"  \
    -u "${account_sid}:${auth_token}"


#########Lets get and purchase §the number to send sms from
#available_number=`curl -X GET \
#    "https://api.twilio.com/2010-04-01/Accounts/${account_sid}/AvailablePhoneNumbers/US/Local"  \
#    -u "${account_sid}:${auth_token}" | \
#    sed -n "/PhoneNumber/{s/.*<PhoneNumber>//;s/<\/PhoneNumber.*//;p;}"` \
#    && echo $available_number



create .twilio_rc with creadentials in script path dir
SID="ACXXXXXXXXXXXX"
AUTH="bXXXXXXXXXXX"
SERVER_URL="http://zabbix.somesite.com"

Script to send SMS, replace From=%2B15592063354 with the number purchased


#!/bin/bash
SCRIPT_PATH="`dirname \"$0\"`"
. $SCRIPT_PATH/.twilio_rc
TO=$1;
BODY=$2;
RESPONSE=$(curl -q -XPOST https://api.twilio.com/2010-04-01/Accounts/${SID}/SMS/Messages \
    -d "Body=${BODY}" \
    -d "To=%2B${TO}" \
    -d "From=%2B15592063354" \
    -u "${SID}:${AUTH}" )

SMS_SID=$(echo $RESPONSE | sed -e 's,.*<Sid>\([^<]*\)</Sid>.*,\1,g' )
SMS_STATUS=$(echo $RESPONSE | sed -e 's,.*<Status>\([^<]*\)</Status>.*,\1,g' )
if [ "x${SMS_STATUS}" != "xqueued" ]
then
exit 1;
fi
for (( c=1; c<=12; c++ ))
do
RESPONSE_STATUS=$(curl -q -G https://api.twilio.com/2010-04-01/Accounts/${SID}/SMS/Messages/${SMS_SID} -u "${SID}:${AUTH}" )
STATUS=$(echo $RESPONSE_STATUS | sed -e 's,.*<Status>\([^<]*\)</Status>.*,\1,g')
if [ "x${STATUS}" == "xsent" ]

Zabbix email alerts via AWS SES or Gmail

#!/usr/bin/python
# -*- coding: utf-8 -*-
"""
Zabbix SMTP Alert script for gmail and Amazon SES.
"""

import sys
import smtplib
from email.MIMEText import MIMEText
from email.Header import Header
from email.Utils import formatdate

## Mail Account
#MAIL_ACCOUNT = 'your.account@gmail.com'
#MAIL_PASSWORD = 'your mail password'

## AWS SES Account
MAIL_FROM = 'zabbix@yourdomain.com'
MAIL_ACCOUNT = 'AKXxXxTBKKH5EMXXXXXX'
MAIL_PASSWORD = 'AiRXxXxdjBTRFlBkbPw+s2tr6d3X9GEQXxXxXxXxXxXx'

## Sender Name
SENDER_NAME = u'Zabbix Alert'

## Mail Server
# SMTP_SERVER = 'smtp.gmail.com'
## SES Mail Server
SMTP_SERVER = 'email-smtp.us-east-1.amazonaws.com'
SMTP_PORT = 587
## TLS
SMTP_TLS = True

def send_mail(recipient, subject, body, encoding='utf-8'):
    session = None
    msg = MIMEText(body, 'plain', encoding)
    #msg['Subject'] = Header(subject, encoding)
    #msg['From'] = Header(SENDER_NAME, encoding)
    msg['Subject'] = subject
    msg['From'] = MAIL_FROM
    msg['To'] = recipient
    msg['Date'] = formatdate()
    try:
        session = smtplib.SMTP(SMTP_SERVER, SMTP_PORT)
        session.set_debuglevel(10)
        if SMTP_TLS:
            session.ehlo()
            session.starttls()
            session.ehlo()
            session.login(MAIL_ACCOUNT, MAIL_PASSWORD)
        #session.sendmail(MAIL_ACCOUNT, recipient, msg.as_string())
        session.sendmail(MAIL_FROM, recipient, msg.as_string())
    except Exception as e:
        raise e
    finally:
        # close session
        if session:
            session.quit()

if __name__ == '__main__':
    """
    recipient = sys.argv[1]
    subject = sys.argv[2]
    body = sys.argv[3]
    """
    if len(sys.argv) == 4:
        send_mail(
            recipient=sys.argv[1],
            subject=sys.argv[2],
            body=sys.argv[3])
    else:
        print u"""requires 3 parameters (recipient, subject, body)
\t$ zabbix-gmail.sh recipient subject body
"""