AWS CloudWatch

Hosted Graphite provides a Amazon AWS CloudWatch add-on. This add-on syncs the metrics from the specified AWS services/regions into your Hosted Graphite account.

To connect to your CloudWatch account, you need to setup Identity and Access Management (IAM) access keys in your AWS Account, with the appropriate permissions to allow Hosted Graphite to connect and collect your metrics.

CloudWatch is available in the Add-Ons page of the Hosted Graphite site.

Amazon IAM Config

To setup an IAM user, go to you AWS IAM Dashboard:

  • On the left, click ‘Users‘ then ‘Create New Users‘.
  • We’re only creating one user, so in the first field enter a suitable name, (for example, hg_addon - the name is irrelevant)
  • Ensure that ‘Generate an access key for each user‘ is checked, and click ‘Create‘.

You will be shown a dialog with your keys. Download/copy both of these keys now (AWS won’t re-show the secret key):

IAM Keys

Save your keys now, you’ll have to create a new user if you don’t!

IAM Policy

At this point, you should have an IAM User setup. Back in the IAM Dashboard, we need to create a custom IAM policy.

  • On the left, click ‘Policies‘ then ‘Create Policy
  • Select the ‘Create Your Own Policy‘ option
  • Give the policy name something descriptive, e.g. hg_metrics_policy
  • Paste in the following policy text and select Create Policy to save it
{
    "Statement": [
        {
            "Sid": "PermissionsForMetrics",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:ListMetrics",
                "cloudwatch:GetMetricStatistics",
                "ec2:DescribeInstances",
                "ec2:DescribeVolumes",
                "rds:DescribeDBInstances",
                "route53:ListHealthChecks",
                "sqs:ListQueues",
                "elasticache:DescribeCacheClusters",
                "elasticloadbalancing:DescribeLoadBalancers",
                "kinesis:ListStreams",
                "redshift:DescribeClusters",
                "cloudfront:ListDistributions"
            ],
            "Resource": [ "*" ]
        },
        {
            "Sid": "PermissionsForTags",
            "Effect": "Allow",
            "Action": [
                "elasticache:ListTagsForResource",
                "elasticloadbalancing:DescribeTags",
                "cloudfront:ListTagsForResource",
                "route53:ListTagsForResource",
                "kinesis:ListTagsForStream",
                "rds:ListTagsForResource",
                "lambda:ListFunctions",
                "elasticmapreduce:ListClusters",
                "iam:GetUser"
            ],
            "Resource": [ "*" ]
        }
    ],
    "Version": "2012-10-17"
}

Once this policy has been created, go back to the Users screen. Choose your IAM user, select the ‘Permissions‘ tab, and click on ‘Attach Policy‘:

Attach Policy

In the dialog that appears search for your custom policy you just created.

Select it and click ‘Attach Policy‘, and your IAM setup is complete.

Enabling The AWS CloudWatch Add-On

To enable the CloudWatch add-on, go to the add-ons page in your Hosted Graphite account and choose the option for Amazon AWS CloudWatch.

You will be presented with the following page, from which you can create, edit or delete AWS configs or create, edit or delete tags:

AWS Account Listing

If you click AWS Tags you will be presented with the following screen which allows you to edit the tags. Each tag has a unique name (shown on the left) and one or more values. Values are separated by a comma. If for example you enter “hello, world” for tag “Tag_Next” then the CloudWatch Add-On will match any tag called “Tag_Next” that has either “hello” or “world” as its value.

AWS CloudWatch Tags

If you click Cancel and go back to the previous screen and click Account Names, you can see this screen from which you can edit a particular AWS configuration.

AWS CloudWatch Configuration

AWS CloudWatch Configuration

  • AWS Access Key and AWS Secret Access Key
    These are the keys you saved when you set up your IAM user.
  • AWS Regions
    Choose the regions containing the services you wish to monitor. You must choose at least one region.
  • AWS Services
    Choose the services which will be queried for metrics.
  • EC2 Instance Aliasing
    Choose this if you would like Hosted Graphite to import your EC2 instance metrics using an instance’s name instead of it’s ID. For instance aliasing to work, it is required that your instances have a “Name” tag defined whose value will be used in place of the instance id. Please note that enabling this feature will create new metrics in your account for each of your EC2 instances that have the “Name” tag. The old instance id based metrics will remain dormant until they expire.
  • Service tagging for this AWS account
    Choose the services that you would like to enable tagged imports for. This will only import metrics from tagged resources for those services. Once you enable this per account, you can type in the tag’s “key”: “value” pairs that you would like to be imported. These key values must also be present in your AWS resources to be imported successfully. For more information on adding these tags go to the AWS docs.

When you click on ‘Save‘, some basic checks will be performed on your keys if successful, your configuration will be saved. If you have at least one service chosen, the cloudwatch add-on will be enabled.

Enable Billing Metrics

AWS metrics for estimated billing charges are calculated and sent several times a day to Amazon CloudWatch. They are stored in the US East (N. Virginia) Region and represent charges for AWS services worldwide. To enable this feature, it needs to activated in the Billing and Cost Management console. Details can be found in the Amazon CloudWatch Documentation.

Enable Route53 Metrics

Amazon Route53 metrics are only received if you enable US East (N. Virginia) as the current region. These metrics are not available from any other region.

Disabling The CloudWatch Add-On

Go to the add-ons page in your Hosted Graphite account, and choose the option for Amazon AWS CloudWatch.

Click the Delete button and the CloudWatch add-on will be disabled for that account.

Metric Name Mapping

The AWS metrics for each service are mapped to Hosted Graphite metric names as follows:

aws.[service].[region].[grouping].[id].[metricname]
  • service - A short token representing the service, e.g. ec2, or rds.
  • region - The AWS region, e.g. us-east-1.
  • grouping - A short token representing the grouping (‘dimension’ in AWS speak) for the metric, e.g. inst for InstanceId.
  • id - The identifier for the service/instance, e.g. Instance id or Name tag for EC2.
  • metricname - The AWS metric name is directly used, e.g. CPUUtilization.
AWS Service AWS ‘Dimension’ HG metric name
Elastic Compute Cloud (EC2) InstanceId aws.ec2.[region].inst.[id].CPUUtilization
Elastic Block Store (EBS) VolumeId aws.ebs.[region].vol.[id].VolumeWriteBytes
Relational Database Service (RDS) DBInstanceIdentifier aws.rds.[region].inst.[id].CPUUtilization

So, for example, the CPUUtilization metric for the ‘i-abcd1234‘ EC2 instance in Virigina will be imported as ‘aws.ec2.us-east-1.inst.i-abcd1234.CPUUtilization‘.

Enabling Account Naming

If you use multiple AWS Access Keys on your Hosted Graphite account for different AWS accounts, projects or environments, you can keep the metrics separated by assigning Account Names to your Access Keys.

Under the list of Access Keys on your HG account, there’s an button to access the Account Names interface. There you can assign names to your Access Keys, which will then become part of the metric name for all metrics retrieved using that key. In the example below, the Access Key was assigned the name ‘test’, so metrics retrieved through that access key will follow the naming structure ‘aws.test.[service]*’

AWS CloudWatch Configuration

If you were previously not using an Account Name and have recently added one, your automatically generated AWS Dashboards will no longer map to the correct metrics. You can tick the option on the Account Names screen (above) to generate new dashboards which use the new Account Names.

CloudWatch Costs

Amazon AWS offers the first one million API requests at no charge. In excess of that, Amazon will charge $0.01 per 1000 requests. See the CloudWatch Pricing page for more information on Amazon pricing.

We aim to make as few requests as possible to fetch your metrics. If you have many instances, or are monitoring many services, you will likely exceed this boundary. If this is the case, these charges will be likely be negligible in comparison to what Amazon charges just to have monitoring enabled for those services.

Disclaimer

While we attempt to minimise the number of API calls which may incur Amazon charges, Hosted Graphite disclaims responsibility for potential costs incurred by use of this add-on.

Our add-on performs read-only requests to the CloudWatch API. Should the provided AWS Access Keys grant greater privileges than what our specified IAM Policy defines, responsibility for any activity performed using those keys lies with the customer.