Graphite Tag Support

What are Graphite tags?

With the release of the 1.1.x series, Graphite now supports tagged metrics.

Rather than using the traditional dotted hierarchy, metrics can be stored and retrieved based on tags - for example, host=webserver- or region=us-east-1.

So where before you might have a name like myapp.webserver-0001.us-east-1.responses.500, you could instead have myapp.responses;host=webserver-0001;region=us-east-1;code=500

Why use Graphite tags?

It’s becoming common in monitoring systems to use a tagged (or labelled) metric format. This allows for more flexibility both in naming and retrieving metrics.

Note that when using tags, each metric is uniquely identified by its name and set of tag/value pairs. As a result, an existing tagged metric cannot be updated to contain new tags.

For example, let’s count 500 responses for My App in both formats:

Dotted Format

myapp.*.*.responses.500
  • Order matters
  • Can have many node options which can be unwieldly

Tag Format

seriesByTag("path=myapp.responses","code=500")
  • Order does not matter
  • You do not always need every node in the metric name to locate one metric or series
  • Does not support wildcards, but accepts regular expressions

The latter allows us to quickly slice and dice metrics by their tags that you get to specify when creating the metric. We don’t have to remember what’s in each position in the dotted hierarchy, and the tags make the query “self documenting”.

Tags are particularly useful in modern cloud environments, where e.g. host or instance may change often, while the total number of metrics being stored and queried remains fairly constant.

Tags in Hosted Graphite

Our implementation is in beta. We really value your comments and suggestions.

If you find something frustrating or worse, please tell us - this is an important feature for us and with your help, we’re keen to get it right. Please email our support team at help@hostedgraphite.com

We’re also working hard to further integrate tag support into our other tools and features:

Can I see how many tagged series I’m using? Will it affect my metric limit?

The metric count on your account homepage currently does not include tagged metrics. Tagged metrics currently do not count towards your metric limit however this will soon change so please use it within reason. For now, you can use our Tag API to Search Tagged Metrics to get an idea of how many tagged metrics you have. The best way to do this is to look up the tag name which will return a list of all tagged metrics and the count of how many tag/value combinations it has.

For example:

$ curl -X GET "https://$APIKEY@api.hostedgraphite.com/api/v1/tags/name" | json_pp

Response:

{
 "tag" : "name",
 "values" : [
    {
       "count" : 5,
       "value" : "foo"
    },
    {
       "value" : "myapp.requests",
       "count" : 5
    },
    {
       "value" : "myapp.response",
       "count" : 16
    },
    {
       "count" : 1,
       "value" : "wubu.wubu.wubu"
    }
 ]
}

Alternatively, if you already know the name of the metric you would like a count for, you can use the command below.

$ curl -X GET "https://$APIKEY@api.hostedgraphite.com/api/v1/tags/findSeries?expr=name=foo.bar" | json_pp | grep -v "^\[\|\]$" | wc -l
679

Sending tagged metrics

You can use the same Carbon endpoints you already use to send tagged metrics. All you need to do is include the tags and values in the updated Carbon line protocol format (unix timestamp is optional):

my.series;tag1=value1;tag2=value2 metric_value (timestamp)

E.g. to get started quickly you can use the netcat utility as follows:

echo "YOUR-API-KEY.foo.bar;tag1=value1;tag2=value2 1.2" | nc carbon.hostedgraphite.com 2003

Sending metrics in your favourite language is similar. Refer to our language guide.

Graphing tagged metrics

In Grafana (or Graphite) you can search for metrics by their tag values, or by the metric name. All queries should be within a seriesByTag graphite function. You can do this in Grafana by toggling the query box to free-text-mode and typing out the full query. Or you could begin by adding the seriesByTag function and filling the boxes with your search terms using the expressions (see table below) where necessary.

seriesByTag()

You can use any number of tag expressions to filter through your metrics:

seriesByTag("name=myapp.response")
seriesByTag("name=myapp.response","code=200")
seriesByTag("name=myapp.response","code=200","env=production")
Using "name" tag
Using "code" tag

Other graphite functions also work, such as sumSeries in the below example:

sumSeries(seriesByTag("name=myapp.response","code=200","env=production"))
sumSeries(seriesByTag("name=myapp.response","code!=200","env=production"))
Using "code" tag

For a full list of Graphite functions and how to use them, check out the Graphite Docs

seriesByTag supports any number of tag expressions to refine the results. If using multiple expressions, only series that match all of them will be returned. Expressions have the following formats:

tag=spec tag value exactly matches the spec
tag!=spec tag value does not exactly match the spec
tag=~spec tag value matches the regular expression spec
tag!=~spec tag value does not match the regular expression spec

Example:

Find all series where server matches the regular expression 0\.* and env is not staging

seriesByTag("server=~0\.*","env!=staging")
Using expressions to filter returned series

aliasByTags() and groupByTags()

These are the equivalent of aliasByNodes and groupByNodes.

aliasByTags requires at least one tag name to be passed in, below we have three:

aliasByTags(seriesByTag("server=~0\.*","env!=staging"),"env","server","code")
Using "code

groupByTags requires an aggregation method to be passed in (e.g. min, max, avg, sum, etc.), below we use sum:

groupByTags(seriesByTag("server=~0\.*", "env!=staging"),"sum","code")
Using "code

For more information about querying tagged metrics refer to the Graphite Tag Docs

Tag API

Check out our tag API documentation here.