Aggregation Rules API

Aggregation rules offer the ability to automatically combine datapoints from multiple input metrics into a smaller set of output metrics according to defined rules. This is useful if you need to use hundreds or thousands of input metrics in a dashboard or graph query, but still want a fast answer. Aggregation rules allow you to effectively pre-compute the most expensive parts of some queries.

Aggregation rules perform the same job as the carbon_aggregator service if you’re self-hosting the open source Graphite tools.

Hosted Graphite’s advanced data views feature is also supported on top of the output metrics from an aggregation rule, so you can get the :sum or an arbitrary percentile across all the metrics matching an aggregation rule

NOTE: The output metric names will be automatically prefixed with “aggregates.” even if you do not specify it in the “output” field

Why are aggregates useful?

If you have a render query using a wildcard to match a large number of metrics, the result of the render query can, in some cases, be very slow to return. Using aggregations reduces the number of metrics which need to be retrieved and can reduce the time spent rendering graphs.

For example, if you have a number of metrics which match the pattern servers.*.requests.<response_code>, and you want to render servers.*.requests.200, the overhead in matching all nodes which match the * in the second node, can be skipped if you set up an aggregation rule with the following pattern/output combination:

{
   "pattern": "servers.*.requests.<response_code>",
   "output": "aggregates.servers.all.requests.<response_code>"
}

The effect of this config would be as follows: Datapoints coming in for metrics named like this:

servers.server1.requests.200
servers.server2.requests.200

Would be copied and combined as soon as the datapoints are received and a new metric created:

  • aggregates.servers.all.requests.200

Datapoints coming in for metrics named like:

servers.server1.requests.404
servers.server2.requests.404

Would be copied and combined as soon as the datapoints are received and a new metric created:

  • aggregates.server.all.requests.404

Now, instead of defining a graph requesting servers.*.requests.200 which could be slow if it matches hundreds or thousands of metrics, you can define a graph requesting aggregates.servers.all.requests.200. That means you’ll be querying just one metric, which will be much faster.

You can also use the <<rest>> pattern as described in the carbon-aggregator docs which allows you to match across nodes. For example the pattern/output config might look like:

{
  "pattern": "servers.*.<<rest>>",
  "output": "aggregates.servers.all.<rest>"
}

This would combine all metrics seen with a starting servers.* into the same metric under aggregates.servers.all.<rest>.

So if we saw some metrics which looked like:

servers.server1.requests.200
servers.server2.requests.200
servers.server3.requests.200
servers.server4.requests.200

They would be combined into a new metric:

  • aggregates.servers.all.requests.200

Similarly, the following:

servers.server1.auth.v2.success
servers.server2.auth.v2.success
servers.server3.auth.v2.success
servers.server4.auth.v2.success

Would be matched by the same pattern and would combine into the new metric:

  • servers.all.auth.v2.success

Creating Aggregation Rules

POST /v1/aggregates/

Request format:

{
  "pattern": "string",
  "output": "string"
}

Parameters:

  • pattern
    [REQUIRED]
    Regex or node pattern to match ingested metrics to.
  • output
    [REQUIRED]
    Metric output, when a metric is matched with the pattern, those metrics will be mapped to this output and appear in your account.

Responses:

  • 200:
    Aggregation rule was created successfully. Returns created aggregation’s ID.
  • 400:
    Creation failed.
    Pattern or output is an invalid input (or not provided).
    Or the aggregation rule creation failed because the pattern is too computationally expensive to match against your metric names. If this keeps happening, please contact support .
  • 403:
    Creation failed, you have reached the limit for your account, contact support.
  • 409:
    Creation failed, aggregation with this pattern & output already exists.
  • 500:
    Creation failed due to a server error - this shouldn’t happen. If this keeps happening, please let us know by contacting support.

Curl Example:

curl -H "Content-Type: application/json" -X POST -d \
'{"pattern": "servers.*.requests.<response_code>", "output": "servers.all.requests.<response_code>"}' \
"https://YOUR-API-KEY@api.hostedgraphite.com/v1/aggregates/"

Creates an aggregation with the pattern: servers.*.requests.<response_code> and the output servers.all.requests.<response_code>. Metrics which look like servers.server-0001.requests.200 and servers.server-0002.requests.200 would appear in your account as aggregates.servers.all.requests.200. Metrics which look like servers.server-0001.requests.404 and servers.server-0002.requests.404 would appear in your account as aggregates.servers.all.requests.404.

Listing Aggregations

GET /v1/aggregates/

Responses:

  • 200:
    List of your aggregation rules.

GET /v1/aggregates/<aggregation_id>/

Responses:

  • 200:
    Aggregation with passed ID if it exists, empty list otherwise.

Curl Example:

curl "https://YOUR-API-KEY@api.hostedgraphite.com/v1/aggregates/<optional_id>"

Lists your aggregations or individual aggregation if optional ID is passed.

Deleting Aggregates

DELETE /v1/aggregates/<aggregate_id>/

Responses:

  • 200:
    Aggregation was succesfully deleted.
  • 400:
    No aggregation ID passed.
  • 412:
    Aggregation with passed ID does not exist.
  • 500:
    Delete failed, please try again.

Curl Example:

curl -X DELETE "https://YOUR-API-KEY@api.hostedgraphite.com/v1/aggregates/<aggregate_id>"

Delete aggregation rule with passed id.

Editing Your Aggregations

To edit an aggregation rule, simply delete the old rule and create a new one. Aggregation rules have no state, so it is safe to delete and re-create them.