System Layout

This document describes how the Hosted Graphite agent (hg-agent) works on your machine.

Disk

Layout on disk is per FHS guidelines for add-on application software packages:

/opt/hg-agent/... (static files)
              package/... (pyinstaller bundle)
              collectors/... (Diamond collectors)
              bin/... (symlinks to binaries in package bundle)
/etc/opt/hg-agent/... (configuration files)
/var/opt/hg-agent/ (spool or other state files)
/var/log/hg-agent/ (log files)

The agent is distributed as a PyInstaller bundle: this is a self-contained build of the whole agent system (including an embedded Python) which requires only a compatible libc.

You typically shouldn’t need to run anything in /opt/hg-agent/bin, since just editing /etc/opt/hg-agent/hg-agent.conf will result in an automatic configuration reload.

In /var/log/hg-agent, you can see the metrics generated by diamond in archive.log, as well as the various daemon logs supervisord.log, diamond.log, and periodic.log.

Processes

The agent runs as a process tree:

supervisord ─┬──┬─ diamond ─┬─ diamond (collector)
             │  │         ├─ diamond (collector)
             │  │         └─ ...
             │  └─ periodic
             ├──── metric receiver
             │                  ├─ UDP
             │                  └─ TCP
             └──── metric forwarder
                                ├─ Spool Reader
                                └─ Forwarder

This all runs as the non-privileged user hg-agent.

supervisord

A process manager which we use to manage the sub-processes that make up hg-agent.

diamond

This popular metric collection daemon does the primary work of hg-agent: gathering system metrics and exporting them to Hosted Graphite.

periodic

A “sidecar” that manages periodic tasks for hg-agent, including (re)configuration of diamond and sending “heartbeat” metadata to Hosted Graphite.

metric receiver

Threads which listens for metrics via both UDP and TCP. On receipt of metrics we perform some validation, and write the data out to a spool file (/var/opt/hg-agent/spool/*.spool.<timestamp>), this write happens once a second. By default these will listen for metrics on localhost:2003. You may configure the ports for the UDP and TCP receivers to listen on by adding the following values to /etc/opt/hg-agent/hg-agent.conf. - udp_port: <port_num> - tcp_port: <port_num>

metric forwarder

Tails spool files in /var/opt/hg-agent/spool and forwards metrics to hostedgraphite’s http api. Metrics are batched and, by default, a POST request is sent every 0.5 seconds or once the batch size is has reached 250. These values can be updated in the hg-agent.conf. The agent maintains a progress file which tells us how much of each spool file has been forwarded. This allows us to recover in times when there may be some network connectivity issues between the host server and HostedGraphite.

Configuration

Agent configuration in /etc/opt/hg-agent/hg-agent.conf is used to generate more complete configuration for diamond in /var/opt/hg-agent/diamond.conf.

Resource requirements

On a representative low-power 4-core Ubuntu 14.04 machine, hg-agent currently takes up the following resources:

  • ~13MiB of disk space
  • ~0.01 process CPU seconds/s
  • ~100MiB physical memory (measured as unique set size)

Use of the local carbon endpoints will increase disk usage. By default, we keep 10 spool files, with a max size of 10MB per file.