Monitoring with Amazon CloudWatch — AWS Direct Connect

The purpose of aws-dx-monitor is enabling customers to monitor AWS Direct Connect runtime configuration items with Amazon CloudWatch. The system is driven by Amazon CloudWatch Events and AWS Lambda.

The following diagram expresses the high level system execution architecture.

1. CloudWatch Events schedules and invokes the Lambda function at five minute intervals.
2. The Lambda function interrogates the AWS Direct Connect service through Describe API calls for every configuration type that makes sense for its operational scope (for example, if you are a Service Provider, you may wish to describe the Interconnects). AWS Direct Connect responds with the JSON payloads for each Describe call.
3. After the Lambda function extracts the status from a given configuration item, it puts the data to a CloudWatch Custom Metric. Each configuration item type should have its own dimension in order to easily identify what is being monitored.
4. Once the data has been settled in the custom CloudWatch metric, you can set alarms for it. See the section on Status Levels for information on configuration item status levels.
5. Alarms may be triggered to notify an operator or administrator of a monitored status threshold.

Installation

Use the following steps to build and deploy the system. It is strongly suggested that you review the security policies prior to deploying to your environment.

1. Prerequisites

On the system where you will be building the AWS Lambda package, you will need the following:

2. Clone the Repository

Clone this repository:

3. Build the aws-dx-monitor Package

Build the aws-dx-monitor package. The script downloads the Enum backport for Python 2.7 and will bundle it as part of the resulting package.

$ cd aws-dx-monitor
$ pip install enum34 -t enum
$ python package.py

4. Deploy the AWS Lambda Function

Deploy the AWS Lambda function. The function must be deployed to every region where you have AWS Direct Connect connections.

1. Login to the AWS Console.
2. Select Services > Lambda
3. Click Create a Lambda Function
4. In Select Blueprint, click Skip
5. In Configure triggers:

a) Click the empty box, and select CloudWatch Events — Schedule.

b) For Rule name, enter aws-dx-monitor.

c) For Rule description, enter Monitor Direct Connect status

d) For Schedule expression, select rate(5 minutes) (or 1 minute if you desire)

e) Click the Enable trigger checkbox.

f) Click the Next button.

6. In Configure function:

a) For Name, enter aws-dx-monitor

b) For Description, enter Monitor Direct Connect status

c) For Runtime, select Python 2.7

d) For Code entry type, select Upload a .ZIP file

e) For Function package, click the Upload button, and select the package you built in the previous section named aws-dx-monitor.zip .

f) For Hander, ensure the value is aws-dx-monitor.lambda_handler .

g) For Role, select Create a custom role.

• For IAM Role, select Create a new IAM Role.
• For Role Name, enter aws-dx-monitor-role
• Expand View Policy Document and click the Edit link. When the Edit Policy dialog appears, click OK.
• Enter the policy defined in the section Lambda Execution Policy. Review the policy prior to using.
• Click Allow.

h) Click the Next button.

7. Click Create Function.

5. Set Alarms

Once the scheduled event begins sending data to Amazon CloudWatch, you can begin setting alarms. The custom metric will be found in CloudWatch > Metrics under the name AWSx/DirectConnect. For more information, see Creating Amazon CloudWatch Alarms.

You may wish to alarm on these levels:

+------------------------------+-----------+
|Config Item | Level |
+------------------------------+-----------+
| Connection | >= 5 |
| Interconnect | >= 4 |
| Connections on Interconnect | >= 5 |
| Virtual Interface | >= 5 |
| Virtual Gateway. | >= 3 |
+------------------------------+-----------+

Status Levels

See the following sections for status levels on:

• Connections
• Interconnects
• Connections on Interconnects
• Virtual Interfaces
• Virtual Gateways

Connections

+--------------+--------------------+---------------+
|Name | API Status Value | Numeric Value |
+--------------+--------------------+---------------+
| Ordering | ordering | 1 |
| Requested | requested | 2 |
| Pending | pending | 3 |
| Available | available | 4 |
| Down | down | 5 |
| Deleting | deleting | 6 |
| Deleted | deleted | 7 |
| Rejected | rejected | 8 |
+--------------+--------------------+---------------+

Interconnects

+--------------+--------------------+---------------+
|Name | API Status Value | Numeric Value |
+--------------+--------------------+---------------+
| Requested | requested | 1 |
| Pending | pending | 2 |
| Available | available | 3 |
| Down | down | 4 |
| Deleting | deleting | 5 |
| Deleted | deleted | 6 |
+--------------+--------------------+---------------+

Connections on Interconnects

+--------------+--------------------+---------------+
|Name | API Status Value | Numeric Value |
+--------------+--------------------+---------------+
| Ordering | ordering | 1 |
| Requested | requested | 2 |
| Pending | pending | 3 |
| Available | available | 4 |
| Down | down | 5 |
| Deleted | deleted | 6 |
| Rejected | rejected | 7 |
+--------------+--------------------+---------------+

Virtual Interfaces

+--------------+--------------------+---------------+
|Name | API Status Value | Numeric Value |
+--------------+--------------------+---------------+
| Confirming | confirming | 1 |
| Verifying | verifying | 2 |
| Pending | pending | 3 |
| Available | available | 4 |
| Down | down | 5 |
| Deleting | deleting | 6 |
| Deleted | deleted | 7 |
| Rejected | rejected | 8 |
+--------------+--------------------+---------------+

Virtual Gateways

+--------------+--------------------+---------------+
|Name | API Status Value | Numeric Value |
+--------------+--------------------+---------------+
| Pending | pending | 1 |
| Available | available | 2 |
| Deleting | deleting | 3 |
| Deleted | deleted | 4 |
+--------------+--------------------+---------------+

Lambda Execution Policy

This policy allows:

• Read-only access to AWS Direct Connect
• PutMetricData access to Amazon CloudWatch
• Log write access to CloudWatch Logs for Lambda logging.

 "Version": "2012-10-17", 
"Statement": [ 
 "Effect": "Allow", 
"Action": [ 
"directconnect:DescribeConnections", 
"directconnect:DescribeConnectionsOnInterconnect", 
"directconnect:DescribeInterconnects", 
"directconnect:DescribeVirtualGateways", 
"directconnect:DescribeVirtualInterfaces" 
], 
"Resource": "*" 
>, 
 "Effect": "Allow", 
"Action": [ 
"cloudwatch:PutMetricData" 
], 
"Resource": "*" 
>, 
 "Effect": "Allow", 
"Action": [ 
"logs:CreateLogGroup", 
"logs:CreateLogStream", 
"logs:PutLogEvents" 
], 
"Resource": "arn:aws:logs:*:*:*" 
> 
]
>

# aws-dx-monitor - monitor DirectConnect and publish to CloudWatch
#
# Copyright 2015-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Author: Richard Elberger (elberger@amazon.com)
import logging
import botocore
import boto3
import json
from enum import Enum
logger = logging.getLogger()
logger.setLevel(logging.INFO)
dxclient = boto3.client('directconnect')
cwclient = boto3.client('cloudwatch')
# The 'live' handler - from scheduler
def lambda_handler ( event, context ):
ver_vistate ( dxclient.describe_virtual_interfaces() )
ver_cstate ( dxclient.describe_connections() )
ver_vpgstate ( dxclient.describe_virtual_gateways() )
# Only DX Service Providers can make this call without an
# exception
#
# ver_cistate( dxclient.describe_interconnects() )
# virtualInterfaces payload evaluation
def ver_vistate ( data ):
if not 'virtualInterfaces' in data:
logger.error("unexpected: virtualInterfaces key not found in data")
return
for iface in data['virtualInterfaces']:
put_vistate( iface['virtualInterfaceId'],
VirtualInterfaceState[iface['virtualInterfaceState']].value )
# connections payload evaluation
def ver_cstate ( data ):
if not 'connections' in data:
logger.error("unexpected: connections key not found in data")
return
for conn in data['connections']:
put_cstate( conn['connectionId'],
# Lookup int value in Connection enum
ConnectionState[conn['connectionState']].value )
# interconnect payload evaluation
def ver_cistate ( data ):
if not 'interconnects' in data:
logger.error("unexpected: interconnects key not found in data")
return
for intconn in data['interconnects']:
put_icstate( intconn['interconnectId'],
# Lookup int value in IntConn enum
InterconnectState[intconn['interconnectState']].value )
# virtualgateway payload evaluation
def ver_vpgstate( data ):
if not 'virtualGateways' in data:
logger.error("unexpected: virtualGateways key not found in data")
return
for vpg in data['virtualGateways']:
put_vpgstate( vpg['virtualGatewayId'],
# Lookup int value in VGW enum
VirtualGatewayState[vpg['virtualGatewayState']].value )
# Writes VirtualInterfaceState dimension data to DX custom metric
def put_vistate ( iid, state ):
response = cwclient.put_metric_data(
Namespace='AWSx/DirectConnect',
MetricData=[
'MetricName': 'VirtualInterfaceState',
'Dimensions': [
'Name': 'VirtualInterfaceId',
'Value': iid
>,
],
'Value': state,
'Unit': 'None'
>,
],
)
# Writes ConnectionState dimension data to DX custom metric
def put_cstate ( iid, state ):
response = cwclient.put_metric_data(
Namespace='AWSx/DirectConnect',
MetricData=[
'MetricName': 'ConnectionState',
'Dimensions': [
'Name': 'ConnectionId',
'Value': iid
>,
],
'Value': state,
'Unit': 'None'
>,
],
)
# Writes InterconnectState dimension data to DX custom metric
def put_icstate ( iid, state ):
response = cwclient.put_metric_data(
Namespace='AWSx/DirectConnect',
MetricData=[
'MetricName': 'InterconnectState',
'Dimensions': [
'Name': 'InterconnectId',
'Value': iid
>,
],
'Value': state,
'Unit': 'None'
>,
],
)
# Writes VGW dimension data to DX custom metric
def put_vpgstate ( iid, state ):
response = cwclient.put_metric_data(
Namespace='AWSx/DirectConnect',
MetricData=[
'MetricName': 'VirtualGatewayState',
'Dimensions': [
'Name': 'VirtualGatewayId',
'Value': iid
>,
],
'Value': state,
'Unit': 'None'
>,
],
)
class VirtualInterfaceState(Enum):
confirming = 1
verifying = 2
pending = 3
available = 4
down = 5
deleting = 6
deleted = 7
rejected = 8
class ConnectionState(Enum):
ordering = 1
requested = 2
pending = 3
available = 4
down = 5
deleting = 6
deleted = 7
rejected = 8
class InterconnectState(Enum):
requested = 1
pending = 2
available = 3
down = 4
deleting = 5
deleted = 6
class VirtualGatewayState(Enum):
pending = 1
available = 2
deleting = 3
deleted = 4

# package.py - simply package aws-dx-monitor
#
# Copyright 2015–2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import sys
import zipfile
if __name__ == "__main__":
print "Packaging aws-dx-monitor"
package = zipfile.ZipFile("aws-dx-monitor.zip", mode = 'w')
try:
package.write('aws-dx-monitor.py')
package.write('enum/__init__.py')
package.write('enum/LICENSE')
finally:
package.close()
print "Packaging complete."

I hope this information proves valuable to you in your endeavors!

Stay curious and keep exploring new horizons in technology!