Heroku Worker Monitoring

Getting Started With Heroku Worker Monitoring

Heroku worker processes allow your Heroku app to offload time-consuming and resource-intensive tasks as background jobs. This enables the web-facing side of your app to focus on receiving requests and returning responses, while workers handle jobs that don't need to take place within the request => response cycle.

On Heroku, these different workloads are defined as process types. Your app's Procfile can define a web process alongside workers, scheduled jobs, release tasks, and other background processes. Each dyno then runs an instance of one of those process types.

Monitoring these processes is an important part of your overall Heroku monitoring strategy. A web dyno can appear healthy while a background worker is consuming too much memory, experiencing high CPU load, or otherwise failing to perform as expected. This article will help you get started with Heroku worker monitoring by walking through:

• Understanding Heroku process metrics
• Monitoring worker processes with the Hosted Graphite add-on

With the Heroku Hosted Graphite monitoring add-on, metrics from your Heroku app are automatically collected from your application's log stream and sent to your HG account. This gives you a quick way to monitor web dynos alongside workers and other background processes, with your metrics available for visualization in a Hosted Grafana environment. HG will also create several Heroku dashboards for you, here's what they look like:

Provision the Heroku Hosted Graphite Add-on and open the app by running these 2 commands from within your Heroku CLI:

• heroku addons:create hostedgraphite -a <app-name>
• heroku addons:open hostedgraphite -a <app-name>

Understanding worker dynos

Heroku applications can divide different types of work between different process types. The web process handles incoming HTTP traffic, while worker processes can handle background jobs such as queue processing, scheduled tasks, email delivery, and other work that doesn't need to happen during a web request. Learn more about the Heroku Process Model HERE.

These process types are typically defined in your application's Procfile. For example, an application might define separate web, worker, scheduler, and clock processes. Heroku can then run and scale dynos for each process type independently. This is useful for application performance because expensive background jobs don't have to compete directly with your web process. If demand for background processing increases, you can scale your workers separately without necessarily increasing the number of web dynos.

But moving work into the background doesn't make that work any less important. A slow or overloaded worker can cause queues to grow, scheduled tasks to fall behind, or important application jobs to take longer than expected. That's why visibility into worker and background processes is an important part of monitoring the health of a Heroku application.

‍

Heroku process metrics

Heroku's process model isn't limited to web and worker. Applications can define different process types based on the work they need to perform, and each dyno belongs to a particular process type. Hosted Graphite uses the process information contained in your Heroku logs to organize runtime metrics by process type. This makes it possible to monitor the resource usage of your background processes independently from the web-facing portion of your application. Hosted Graphite currently supports metrics for process types including:

• scheduler
• worker
• run
• release
• advanced-scheduler
• celeryworker
• beat
• orchestrator
• mail_syncer
• priority
• notifications
• clock

This is especially useful for applications that divide background work between multiple specialized processes. For example, a Python application might use a celeryworker process for queued jobs and a beat process for scheduled tasks, while another application might define its own processes for notifications or priority jobs.

Heroku's runtime metrics provide resource information such as CPU load, memory usage, swap, and memory quota. These metrics are reported for individual dynos, allowing you to see how the resources consumed by your worker processes change over time. Heroku's log-runtime-metrics include metrics such as:

• memory.memory_cache
• memory.memory_pgpgin
• memory.memory_pgpgout
• memory.memory_quota
• memory.memory_rss
• memory.memory_swap
• memory.memory_total

If runtime metrics aren't already enabled for your app, you can enable them from within the Heroku CLI:

• heroku labs:enable log-runtime-metrics -a <app-name>
• heroku restart -a <app-name>

‍

Monitoring worker performance metrics with Heroku add-ons

The easiest way to get started with Heroku worker monitoring is with the Heroku Hosted Graphite add-on. Hosted Graphite collects metrics from your Heroku application's log stream and makes them available in your Hosted Graphite account, without requiring you to build a separate monitoring pipeline. You can provision the add-on directy from the Heroku Rersources UI:

Once the add-on is installed, a log drain sends your Heroku app logs to Hosted Graphite. The integration parses supported metric data from those logs and submits it to your HG account.

When runtime metrics are enabled, this includes resource metrics associated with individual Heroku processes. This means you can monitor a worker, scheduler, release, or other supported process separately from your application's web dynos. Once the metrics are available in Hosted Graphite, you can visualize them in pre-made Grafana dashboards, and use them to build custom dashboards and alerts.

‍

If you don't require monitoring individual Process dynos and just want a general overview of combined Process performance, Hosted Graphite has an interface that allows you to aggregate processes into a single data set, to lower your total number of stored metrics:

‍

Conclusion

Worker processes are an important part of many Heroku applications. They keep expensive or asynchronous jobs outside of the web request cycle, and Heroku's process model makes it possible to separate and scale those workloads independently. But those background processes still need to be monitored.

With the Hosted Graphite Heroku add-on, you can collect process metrics from your Heroku logs and monitor workers, scheduled jobs, and other supported process types alongside the rest of your application metrics.

Install the Hosted Graphite add-on to start monitoring your Heroku application, or reach out to our support team if you have questions about getting your Heroku metrics into Hosted Graphite.