Automatisches Fehlerbehebungs- und ITSM-System mit EventBridge und Lambda

Einführung :

Leute, im IT-Betrieb ist es eine sehr allgemeine Aufgabe, Servermetriken wie die Auslastung von CPU/Speicher und Festplatte oder Dateisystemen zu überwachen, aber falls eine der Metriken kritisch wird, müssen engagierte Personen einige grundlegende Aufgaben ausführen Fehlerbehebung, indem Sie sich beim Server anmelden und den ursprünglichen Grund für die Auslastung ermitteln. Dies muss die Person mehrmals durchführen, wenn sie mehrere gleiche Benachrichtigungen erhält, die Langeweile hervorrufen und überhaupt nicht produktiv sind. Um dieses Problem zu umgehen, kann ein System entwickelt werden, das reagiert, sobald ein Alarm ausgelöst wird, und auf diese Fälle reagiert, indem es einige grundlegende Befehle zur Fehlerbehebung ausführt. Nur um die Problemstellung und Erwartungen zusammenzufassen -

Problemstellung:

Entwickeln Sie ein System, das die Erwartungen nicht erfüllt -

• Jede EC2-Instanz sollte von CloudWatch überwacht werden.
• Sobald ein Alarm ausgelöst wird, muss etwas vorhanden sein, das sich bei der betroffenen EC2-Instanz anmeldet und einige grundlegende Befehle zur Fehlerbehebung ausführt.
• Erstellen Sie dann ein JIRA-Problem, um diesen Vorfall zu dokumentieren, und fügen Sie die Ausgabe der Befehle im Kommentarbereich hinzu.
• Senden Sie dann eine automatische E-Mail mit allen Alarmdetails und JIRA-Problemdetails.

Architekturdiagramm:

Voraussetzungen:

1. EC2-Instanzen
2. CloudWatch-Alarme
3. EventBridge-Regel
4. Lambda-Funktion
5. JIRA-Konto
6. Einfacher Benachrichtigungsdienst

Implementierungsschritte:

• A. Installation und Konfiguration des CloudWatch-Agenten:
  Öffnen Sie die Systems Manager-Konsole und klicken Sie auf „Dokumente“
  Suchen Sie nach dem Dokument „AWS-ConfigureAWSPackage“ und führen Sie es aus, indem Sie die erforderlichen Details angeben.
  Paketname = AmazonCloudwatchAgent
  Nach der Installation muss der CloudWatch-Agent gemäß der Konfigurationsdatei konfiguriert werden. Führen Sie dazu das AmazonCloudWatch-ManageAgent-Dokument aus. Stellen Sie außerdem sicher, dass die JSON CloudWatch-Konfigurationsdatei im SSM-Parameter gespeichert ist.
  Sobald Sie sehen, dass Metriken an die CloudWatch-Konsole gemeldet werden, erstellen Sie einen Alarm für CPU- und Speicherauslastung usw.

• B. EventBridge-Regel einrichten:
  Um die Alarmzustandsänderungen zu verfolgen, haben wir hier ein wenig angepasstes Muster, um nur Alarmzustandsänderungen von „OK“ auf „ALARM“ zu verfolgen und nicht umzukehren. Fügen Sie diese Regel dann als Auslöser zu einer Lambda-Funktion hinzu.
  

{
  "source": ["aws.cloudwatch"],
  "detail-type": ["CloudWatch Alarm State Change"],
  "detail": {
    "state": {
      "value": ["ALARM"]
    },
    "previousState": {
      "value": ["OK"]
    }
  }
}

• C. Erstellen Sie eine Lambda-Funktion zum Senden von E-Mails und Protokollieren eines Vorfalls in JIRA: Diese Lambda-Funktion wird für mehrere Aktivitäten erstellt, die durch die EventBridge-Regel ausgelöst und als Ziel-SNS-Thema mithilfe des AWS SDK (Boto3) hinzugefügt werden. Sobald die EventBridge-Regel ausgelöst wird, werden JSON-Ereignisinhalte an Lambda gesendet, wodurch die Funktion mehrere Details erfasst, um sie auf unterschiedliche Weise zu verarbeiten. Hier haben wir bisher an zwei Arten von Alarmen gearbeitet – i. CPU-Auslastung und ii. Speichernutzung. Sobald einer dieser beiden Alarme ausgelöst wird und der Alarmstatus von „OK“ in „ALARM“ geändert wird, wird EventBridge ausgelöst, wodurch auch die Lambda-Funktion ausgelöst wird, um die im Formularcode genannten Aufgaben auszuführen.

Lambda-Voraussetzungen:
Wir benötigen die folgenden Module zum Importieren, damit die Codes funktionieren -

• >> os
• >> sys
• >> json
• >> boto3
• >> Zeit
• >> Anfragen

Hinweis: Von den oben genannten Modulen werden alle Module mit Ausnahme des restlichen Moduls „Anfragen“ standardmäßig innerhalb einer zugrunde liegenden Lambda-Infrastruktur heruntergeladen. Der direkte Import des Moduls „requests“ wird in Lambda nicht unterstützt. Installieren Sie daher zunächst das Anforderungsmodul in einem Ordner auf Ihrem lokalen Computer (Laptop), indem Sie den folgenden Befehl ausführen -

pip3 install requests -t <directory path> --no-user

_Danach wird dies in den Ordner heruntergeladen, in dem Sie den obigen Befehl ausführen oder in dem Sie die Quellcodes des Moduls speichern möchten. Ich hoffe, dass hier Lambda-Code auf Ihrem lokalen Computer vorbereitet wird. Wenn ja, erstellen Sie mit dem Modul eine ZIP-Datei des gesamten Lambda-Quellcodes. Laden Sie anschließend die ZIP-Datei in die Lambda-Funktion hoch.

Hier führen wir also zwei Szenarien durch -

1. CPU-Auslastung – Wenn der CPU-Auslastungsalarm ausgelöst wird, muss die Lambda-Funktion die Instanz abrufen, sich bei dieser Instanz anmelden und die fünf Prozesse mit dem höchsten Verbrauch ausführen. Anschließend wird ein JIRA-Problem erstellt und die Prozessdetails im Kommentarbereich hinzugefügt. Gleichzeitig wird eine E-Mail mit Alarmdetails und Jira-Problemdetails mit Prozessausgabe gesendet.

2. Speichernutzung –Gleicher Ansatz wie oben

Now, let me reframe the task details which lambda is supposed to perform -

1. Login to Instance
2. Perform Basic Troubleshooting Steps.
3. Create a JIRA Issue
4. Send Email to Recipient with all Details

Scenario 1: When alarm state has been changed from OK to ALARM

First Set (Define the cpu and memory function) :

################# Importing Required Modules ################
############################################################
import json
import boto3
import time
import os
import sys
sys.path.append('./python')   ## This will add requests module along with all dependencies into this script
import requests
from requests.auth import HTTPBasicAuth

################## Calling AWS Services ###################
###########################################################
ssm = boto3.client('ssm')
sns_client = boto3.client('sns')
ec2 = boto3.client('ec2')

################## Defining Blank Variable ################
###########################################################
cpu_process_op = ''
mem_process_op = ''
issueid = ''
issuekey = ''
issuelink = ''

################# Function for CPU Utilization ################
###############################################################
def cpu_utilization(instanceid, metric_name, previous_state, current_state):
    global cpu_process_op
    if previous_state == 'OK' and current_state == 'ALARM':
        command = 'ps -eo user,pid,ppid,cmd,%mem,%cpu --sort=-%cpu | head -5'
        print(f'Impacted Instance ID is : {instanceid}, Metric Name: {metric_name}')
        # Start a session
        print(f'Starting session to {instanceid}')
        response = ssm.send_command(InstanceIds = [instanceid], DocumentName="AWS-RunShellScript", Parameters={'commands': [command]})
        command_id = response['Command']['CommandId']
        print(f'Command ID: {command_id}')
        # Retrieve the command output
        time.sleep(4)
        output = ssm.get_command_invocation(CommandId=command_id, InstanceId=instanceid)
        print('Please find below output -\n', output['StandardOutputContent'])
        cpu_process_op = output['StandardOutputContent']
    else:
        print('None')

################# Function for Memory Utilization ################
############################################################### 
def mem_utilization(instanceid, metric_name, previous_state, current_state):
    global mem_process_op
    if previous_state == 'OK' and current_state == 'ALARM':
        command = 'ps -eo user,pid,ppid,cmd,%mem,%cpu --sort=-%mem | head -5'
        print(f'Impacted Instance ID is : {instanceid}, Metric Name: {metric_name}')
        # Start a session
        print(f'Starting session to {instanceid}')
        response = ssm.send_command(InstanceIds = [instanceid], DocumentName="AWS-RunShellScript", Parameters={'commands': [command]})
        command_id = response['Command']['CommandId']
        print(f'Command ID: {command_id}')
        # Retrieve the command output
        time.sleep(4)
        output = ssm.get_command_invocation(CommandId=command_id, InstanceId=instanceid)
        print('Please find below output -\n', output['StandardOutputContent'])
        mem_process_op = output['StandardOutputContent']
    else:
        print('None')

Second Set (Create JIRA Issue) :

################## Create JIRA Issue ################
#####################################################
def create_issues(instanceid, metric_name, account, timestamp, region, current_state, previous_state, cpu_process_op, mem_process_op, metric_val):
    ## Create Issue ##
    url ='https://<your-user-name>.atlassian.net//rest/api/2/issue'
    username = os.environ['username']
    api_token = os.environ['token']
    project = 'AnirbanSpace'
    issue_type = 'Incident'
    assignee = os.environ['username']
    summ_metric  = '%CPU Utilization' if 'CPU' in metric_name else '%Memory Utilization' if 'mem' in metric_name else '%Filesystem Utilization' if metric_name == 'disk_used_percent' else None
    metric_val = metric_val
    summary = f'Client | {account} | {instanceid} | {summ_metric} | Metric Value: {metric_val}'
    description = f'Client: Company\nAccount: {account}\nRegion: {region}\nInstanceID = {instanceid}\nTimestamp = {timestamp}\nCurrent State: {current_state}\nPrevious State = {previous_state}\nMetric Value = {metric_val}'

    issue_data = {
        "fields": {
            "project": {
                "key": "SCRUM"
            },
            "summary": summary,
            "description": description,
            "issuetype": {
                "name": issue_type
            },
            "assignee": {
                "name": assignee
            }
        }
    }
    data = json.dumps(issue_data)
    headers = {
        "Accept": "application/json",
        "Content-Type": "application/json"
    }
    auth = HTTPBasicAuth(username, api_token)
    response = requests.post(url, headers=headers, auth=auth, data=data)
    global issueid
    global issuekey
    global issuelink
    issueid = response.json().get('id')
    issuekey = response.json().get('key')
    issuelink = response.json().get('self')

    ################ Add Comment To Above Created JIRA Issue ###################
    output = cpu_process_op if metric_name == 'CPUUtilization' else mem_process_op if metric_name == 'mem_used_percent' else None
    comment_api_url = f"{url}/{issuekey}/comment"
    add_comment = requests.post(comment_api_url, headers=headers, auth=auth, data=json.dumps({"body": output}))

    ## Check the response
    if response.status_code == 201:
        print("Issue created successfully. Issue key:", response.json().get('key'))
    else:
        print(f"Failed to create issue. Status code: {response.status_code}, Response: {response.text}")

Third Set (Send an Email) :

################## Send An Email ################
#################################################
def send_email(instanceid, metric_name, account, region, timestamp, current_state, current_reason, previous_state, previous_reason, cpu_process_op, mem_process_op, metric_val, issueid, issuekey, issuelink):
    ### Define a dictionary of custom input ###
    metric_list = {'mem_used_percent': 'Memory', 'disk_used_percent': 'Disk', 'CPUUtilization': 'CPU'}

    ### Conditions ###
    if previous_state == 'OK' and current_state == 'ALARM' and metric_name in list(metric_list.keys()):
        metric_msg = metric_list[metric_name]
        output = cpu_process_op if metric_name == 'CPUUtilization' else mem_process_op if metric_name == 'mem_used_percent' else None
        print('This is output', output)
        email_body = f"Hi Team, \n\nPlease be informed that {metric_msg} utilization is high for the instanceid {instanceid}. Please find below more information \n\nAlarm Details:\nMetricName = {metric_name}, \nAccount = {account}, \nTimestamp = {timestamp}, \nRegion = {region}, \nInstanceID = {instanceid}, \nCurrentState = {current_state}, \nReason = {current_reason}, \nMetricValue = {metric_val}, \nThreshold = 80.00 \n\nProcessOutput: \n{output}\nIncident Deatils:\nIssueID = {issueid}, \nIssueKey = {issuekey}, \nLink = {issuelink}\n\nRegards,\nAnirban Das,\nGlobal Cloud Operations Team"
        res = sns_client.publish(
            TopicArn = os.environ['snsarn'],
            Subject = f'High {metric_msg} Utilization Alert : {instanceid}',
            Message = str(email_body)
            )
        print('Mail has been sent') if res else print('Email not sent')
    else:
        email_body = str(0)

Fourth Set (Calling Lambda Handler Function) :

################## Lambda Handler Function ################
###########################################################
def lambda_handler(event, context):
    instanceid = event['detail']['configuration']['metrics'][0]['metricStat']['metric']['dimensions']['InstanceId']
    metric_name = event['detail']['configuration']['metrics'][0]['metricStat']['metric']['name']
    account = event['account']
    timestamp = event['time']
    region = event['region']
    current_state = event['detail']['state']['value']
    current_reason = event['detail']['state']['reason']
    previous_state = event['detail']['previousState']['value']
    previous_reason = event['detail']['previousState']['reason']
    metric_val = json.loads(event['detail']['state']['reasonData'])['evaluatedDatapoints'][0]['value']
    ##### function calling #####
    if metric_name == 'CPUUtilization':
        cpu_utilization(instanceid, metric_name, previous_state, current_state)
        create_issues(instanceid, metric_name, account, timestamp, region, current_state, previous_state, cpu_process_op, mem_process_op, metric_val)
        send_email(instanceid, metric_name, account, region, timestamp, current_state, current_reason, previous_state, previous_reason, cpu_process_op, mem_process_op, metric_val, issueid, issuekey, issuelink)
    elif metric_name == 'mem_used_percent':
        mem_utilization(instanceid, metric_name, previous_state, current_state)
        create_issues(instanceid, metric_name, account, timestamp, region, current_state, previous_state, cpu_process_op, mem_process_op, metric_val)
        send_email(instanceid, metric_name, account, region, timestamp, current_state, current_reason, previous_state, previous_reason, cpu_process_op, mem_process_op, metric_val, issueid, issuekey, issuelink)
    else:
        None

Alarm Email Screenshot :

Note: In ideal scenario, threshold is 80%, but for testing I changed it to 10%. Please see the Reason.

Alarm JIRA Issue :

Scenario 2: When alarm state has been changed from OK to Insufficient data

In this scenario, if any server cpu or memory utilization metrics data are not captured, then alarm state gets changed from OK to INSUFFICIENT_DATA. This state can be achieved in two ways - a.) If server is in stopped state b.) If CloudWatch agent is not running or went in dead state.
So, as per below script, you'll be able to see that when cpu or memory utilization alarm status gets insufficient data, then lambda will first check if instance is in running status or not. If instance is in running state, then it will login and check CloudWatch agent status. Post that, it will create a JIRA issue and post the agent status in comment section of JIRA issue. After that, it will send an email with alarm details and agent status.

Full Code :

################# Importing Required Modules ################
############################################################
import json
import boto3
import time
import os
import sys
sys.path.append('./python')   ## This will add requests module along with all dependencies into this script
import requests
from requests.auth import HTTPBasicAuth

################## Calling AWS Services ###################
###########################################################
ssm = boto3.client('ssm')
sns_client = boto3.client('sns')
ec2 = boto3.client('ec2')

################## Defining Blank Variable ################
###########################################################
cpu_process_op = ''
mem_process_op = ''
issueid = ''
issuekey = ''
issuelink = ''

################# Function for CPU Utilization ################
###############################################################
def cpu_utilization(instanceid, metric_name, previous_state, current_state):
    global cpu_process_op
    if previous_state == 'OK' and current_state == 'INSUFFICIENT_DATA':
        ec2_status = ec2.describe_instance_status(InstanceIds=[instanceid,])['InstanceStatuses'][0]['InstanceState']['Name']
        if ec2_status == 'running':
            command = 'systemctl status amazon-cloudwatch-agent;sleep 3;systemctl restart amazon-cloudwatch-agent'
            print(f'Impacted Instance ID is : {instanceid}, Metric Name: {metric_name}')
            # Start a session
            print(f'Starting session to {instanceid}')
            response = ssm.send_command(InstanceIds = [instanceid], DocumentName="AWS-RunShellScript", Parameters={'commands': [command]})
            command_id = response['Command']['CommandId']
            print(f'Command ID: {command_id}')
            # Retrieve the command output
            time.sleep(4)
            output = ssm.get_command_invocation(CommandId=command_id, InstanceId=instanceid)
            print('Please find below output -\n', output['StandardOutputContent'])
            cpu_process_op = output['StandardOutputContent']
        else:
            cpu_process_op = f'Instance current status is {ec2_status}. Not able to reach out!!'
            print(f'Instance current status is {ec2_status}. Not able to reach out!!')
    else:
        print('None')

################# Function for Memory Utilization ################
############################################################### 
def mem_utilization(instanceid, metric_name, previous_state, current_state):
    global mem_process_op
    if previous_state == 'OK' and current_state == 'INSUFFICIENT_DATA':
        ec2_status = ec2.describe_instance_status(InstanceIds=[instanceid,])['InstanceStatuses'][0]['InstanceState']['Name']
        if ec2_status == 'running':
            command = 'systemctl status amazon-cloudwatch-agent'
            print(f'Impacted Instance ID is : {instanceid}, Metric Name: {metric_name}')
            # Start a session
            print(f'Starting session to {instanceid}')
            response = ssm.send_command(InstanceIds = [instanceid], DocumentName="AWS-RunShellScript", Parameters={'commands': [command]})
            command_id = response['Command']['CommandId']
            print(f'Command ID: {command_id}')
            # Retrieve the command output
            time.sleep(4)
            output = ssm.get_command_invocation(CommandId=command_id, InstanceId=instanceid)
            print('Please find below output -\n', output['StandardOutputContent'])
            mem_process_op = output['StandardOutputContent']
            print(mem_process_op)
        else:
            mem_process_op = f'Instance current status is {ec2_status}. Not able to reach out!!'
            print(f'Instance current status is {ec2_status}. Not able to reach out!!')     
    else:
        print('None')

################## Create JIRA Issue ################
#####################################################
def create_issues(instanceid, metric_name, account, timestamp, region, current_state, previous_state, cpu_process_op, mem_process_op, metric_val):
    ## Create Issue ##
    url ='https://<your-user-name>.atlassian.net//rest/api/2/issue'
    username = os.environ['username']
    api_token = os.environ['token']
    project = 'AnirbanSpace'
    issue_type = 'Incident'
    assignee = os.environ['username']
    summ_metric  = '%CPU Utilization' if 'CPU' in metric_name else '%Memory Utilization' if 'mem' in metric_name else '%Filesystem Utilization' if metric_name == 'disk_used_percent' else None
    metric_val = metric_val
    summary = f'Client | {account} | {instanceid} | {summ_metric} | Metric Value: {metric_val}'
    description = f'Client: Company\nAccount: {account}\nRegion: {region}\nInstanceID = {instanceid}\nTimestamp = {timestamp}\nCurrent State: {current_state}\nPrevious State = {previous_state}\nMetric Value = {metric_val}'

    issue_data = {
        "fields": {
            "project": {
                "key": "SCRUM"
            },
            "summary": summary,
            "description": description,
            "issuetype": {
                "name": issue_type
            },
            "assignee": {
                "name": assignee
            }
        }
    }
    data = json.dumps(issue_data)
    headers = {
        "Accept": "application/json",
        "Content-Type": "application/json"
    }
    auth = HTTPBasicAuth(username, api_token)
    response = requests.post(url, headers=headers, auth=auth, data=data)
    global issueid
    global issuekey
    global issuelink
    issueid = response.json().get('id')
    issuekey = response.json().get('key')
    issuelink = response.json().get('self')

    ################ Add Comment To Above Created JIRA Issue ###################
    output = cpu_process_op if metric_name == 'CPUUtilization' else mem_process_op if metric_name == 'mem_used_percent' else None
    comment_api_url = f"{url}/{issuekey}/comment"
    add_comment = requests.post(comment_api_url, headers=headers, auth=auth, data=json.dumps({"body": output}))

    ## Check the response
    if response.status_code == 201:
        print("Issue created successfully. Issue key:", response.json().get('key'))
    else:
        print(f"Failed to create issue. Status code: {response.status_code}, Response: {response.text}")

################## Send An Email ################
#################################################
def send_email(instanceid, metric_name, account, region, timestamp, current_state, current_reason, previous_state, previous_reason, cpu_process_op, mem_process_op, metric_val, issueid, issuekey, issuelink):
    ### Define a dictionary of custom input ###
    metric_list = {'mem_used_percent': 'Memory', 'disk_used_percent': 'Disk', 'CPUUtilization': 'CPU'}

    ### Conditions ###
    if previous_state == 'OK' and current_state == 'INSUFFICIENT_DATA' and metric_name in list(metric_list.keys()):
        metric_msg = metric_list[metric_name]
        output = cpu_process_op if metric_name == 'CPUUtilization' else mem_process_op if metric_name == 'mem_used_percent' else None
        email_body = f"Hi Team, \n\nPlease be informed that {metric_msg} utilization alarm state has been changed to {current_state} for the instanceid {instanceid}. Please find below more information \n\nAlarm Details:\nMetricName = {metric_name}, \n Account = {account}, \nTimestamp = {timestamp}, \nRegion = {region},  \nInstanceID = {instanceid}, \nCurrentState = {current_state}, \nReason = {current_reason}, \nMetricValue = {metric_val}, \nThreshold = 80.00  \n\nProcessOutput = \n{output}\nIncident Deatils:\nIssueID = {issueid}, \nIssueKey = {issuekey}, \nLink = {issuelink}\n\nRegards,\nAnirban Das,\nGlobal Cloud Operations Team"
        res = sns_client.publish(
            TopicArn = os.environ['snsarn'],
            Subject = f'Insufficient {metric_msg} Utilization Alarm : {instanceid}',
            Message = str(email_body)
        )
        print('Mail has been sent') if res else print('Email not sent')
    else:
        email_body = str(0)

################## Lambda Handler Function ################
###########################################################
def lambda_handler(event, context):
    instanceid = event['detail']['configuration']['metrics'][0]['metricStat']['metric']['dimensions']['InstanceId']
    metric_name = event['detail']['configuration']['metrics'][0]['metricStat']['metric']['name']
    account = event['account']
    timestamp = event['time']
    region = event['region']
    current_state = event['detail']['state']['value']
    current_reason = event['detail']['state']['reason']
    previous_state = event['detail']['previousState']['value']
    previous_reason = event['detail']['previousState']['reason']
    metric_val = 'NA'
    ##### function calling #####
    if metric_name == 'CPUUtilization':
        cpu_utilization(instanceid, metric_name, previous_state, current_state)
        create_issues(instanceid, metric_name, account, timestamp, region, current_state, previous_state, cpu_process_op, mem_process_op, metric_val)
        send_email(instanceid, metric_name, account, region, timestamp, current_state, current_reason, previous_state, previous_reason, cpu_process_op, mem_process_op, metric_val, issueid, issuekey, issuelink)
    elif metric_name == 'mem_used_percent':
        mem_utilization(instanceid, metric_name, previous_state, current_state)
        create_issues(instanceid, metric_name, account, timestamp, region, current_state, previous_state, cpu_process_op, mem_process_op, metric_val)
        send_email(instanceid, metric_name, account, region, timestamp, current_state, current_reason, previous_state, previous_reason, cpu_process_op, mem_process_op, metric_val, issueid, issuekey, issuelink)
    else:
        None

Insufficient Data Email Screenshot :

Insufficient data JIRA Issue :

Conclusion :

In this article, we have tested scenarios on both cpu and memory utilization, but there can be lots of metrics on which we can configure auto-incident and auto-email functionality which will reduce significant efforts in terms of monitoring and creating incidents and all. This solution has given a initial approach how we can proceed further, but for sure there can be other possibilities to achieve this goal. I believe you all will understand the way we tried to make this relatable. Please like and comment if you love this article or have any other suggestions, so that we can populate in coming articles. ??

Thanks!!
Anirban Das

Das obige ist der detaillierte Inhalt vonAutomatisches Fehlerbehebungs- und ITSM-System mit EventBridge und Lambda. Für weitere Informationen folgen Sie bitte anderen verwandten Artikeln auf der PHP chinesischen Website!