The Hosted Graphite Agent

Our agent is a simple way to get your metrics into Hosted Graphite. You install it on your machines, and it collects and forwards metrics for you.

If you have any questions about the agent, get in touch.

Installation

The simplest way to get started is by pulling down our installer script.

For Debian-based systems:

curl -s https://YOUR-API-KEY@www.hostedgraphite.com/agent/installer/deb/ | sudo sh

or for RHEL-based systems:

curl -s https://YOUR-API-KEY@www.hostedgraphite.com/agent/installer/rpm/ | sudo sh

If you would like to set up installation and configuration via Puppet, Chef or a similar configuration management system, see our repository install instructions.

Once you have insalled an agent, visit the addons page and generate the Hosted Graphite Agent (Base) dashboard.

Platforms

We support the following stable OS releases on 64-bit machines:

  • CentOS 6 and later (covering the RedHat family);
  • Debian Wheezy and later;
  • Ubuntu 12.04 LTS and later.

These should cover most installs on RHEL, CentOS, Amazon Linux, and Debian/Ubuntu family platforms.

How it works

The Hosted Graphite agent is an easy-to-use bundle of the popular metrics collector Diamond and the Supervisor process manager along with an embedded build of Python and some support scripts.

For more details of how it works on your machines, see System Layout.

Metrics collected

The agent’s default metrics are listed and explained at Base Metrics.

Configuration

Agent configuration is minimal and done via /etc/opt/hg-agent/hg-agent.conf. This is used to generate more complete configuration for diamond in /var/opt/hg-agent/diamond.conf.

Typically hg-agent.conf contains only an api_key and an endpoint:

api_key: YOUR-API-KEY
endpoint_url: "https://agentapi.hostedgraphite.com/api/v1/sink"
endpoint: 0a0a0a0a.carbon.hostedgraphite.com

You can find the right values for these on your Hosted Graphite account page, but they should be filled in automatically by the simple install process above.

If your system requires a proxy to get HTTPS access to the Internet, you can add it here:

https_proxy: http://10.10.1.10:1080

Troubleshooting

Logs are in /var/log/hg-agent: you can see the metrics generated by diamond in archive.log, as well as the various daemon logs supervisord.log, diamond.log, and periodic.log, e.g.

$ tail -f /var/log/hg-agent/periodic.log

Process structure and other details are described in System Layout.

You can inspect the agent as a service like any other, e.g. on Ubuntu trusty:

$ sudo service hg-agent status

If you’d like to look at things from the agent’s supervisor point of view:

$ sudo /opt/hg-agent/bin/supervisorctl --config=/etc/opt/hg-agent/supervisor.conf status

Source code

The sources used to build the agent can be followed at:

and are licensed under an MIT license.

Versions of Supervisor & Diamond included can be found in the agent build scripts.

Adding additional metrics

As of version 1.8 the Hosted Graphite Agent now ships with local Carbon endpoints you can ship Graphite metrics to as usual via udp and tcp on localhost and port 2003 (by default).

You can also drop files containing metrics into /var/opt/hg-agent/drop, e.g.

my.metric 1.2
my.other.metric 3.4

which will be picked up and forwarded by the agent. This is useful particularly if you want to collect data that requires privileged access, e.g. via a small cron job.

Sending Custom Metrics

Sending metrics directly to the Hosted Graphite Agent

When the agent is running on your machine/server, it should be trivial to send metrics through it to your HostedGraphite account. As your agent is configured to use your Hosted Graphite API key, it can accept metrics locally without any API-Key prefixing.

A quick example in Python, sending a metric via UDP:

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto("foo.bar 1.2\n", ("localhost", 2003))

Any of our examples found on our language guide contain api-key prefixing, and they send to <something>.carbon.hostedgraphite.com, so all you need to do is send to localhost instead, and the api-key prefixing for your metric names is optional.

Features

  • Faster submission of metric data from your application because it doesn’t have to do a DNS lookup, or connect out across the internet.
  • More reliable delivery because the agent stores-and-forwards, buffering on disk when it can’t connect.
  • Easier debugging for the “my metric is missing” problem, because our UI can tell you that an agent has stopped responding.
  • More secure, the agent uses HTTPS and while we offer TLS-wrapped carbon protocols, it’s more effort to use than the plaintext ones. Sending straight to the agent bypasses this and we take care of it for you.
  • No need to prefix your metrics with an API key for HG’s authentication (its already done with the API key in the agent config).
  • No need to specify new firewall rules for carbon protocols and keep the rules updated - if you can already get HTTPS out of your network, even via a proxy, then it’ll work.

Dashboards

Agent list

The Agent list provides an overview of all agents associated with your account. From here, you can view the status of each agent and other information.

../_images/agent_list.png

The coloured circle representing the State can take 3 values:

  • Green: Agent is reporting normally with the correct time.
  • Yellow: Agent has not reported in over 20 minutes, or is reporting with the wrong time.
  • Red: Agent has reported errors in recent log messages.
../_images/clock_offset.png

An agent’s Clock Offset is the difference between the time reported by the agent and the time recorded on our server. Incorrect time can produce anomalies in metric data so it is advised to keep the agent’s clock accurate.

You can delete the record of an agent from this page, but be aware that this will not uninstall the agent. To uninstall an agent, use your package manager.

Base system metrics

Base system metrics dashboard

Base system metrics dashboard

This dashboard displays many of the metrics reported by the Hosted Graphite agent, and is available after you have installed your first agent.

You can find descriptions of how to read the base system metrics dashboard at Base Metrics.

Roadmap

More integrations

We’ll be adding lots more easy-to-configure collectors and dashboards.

If there’s something you’d like automated collection and dashboards for, let us know.

Uninstall

Hosted Graphite Agents can be uninstalled using your package manager.

For Debian-based systems:

sudo apt-get remove hg-agent

For RHEL-based systems:

sudo yum erase hg-agent