Monitor Linux Processes with Prometheus and Grafana

March 21, 2024

My client – a small startup that uses Jetsons – wanted to monitor all the services that run on their Linux system over time, both the average consumption, but also the maximum consumption of CPU and RAM that each service touched.

In this article, we delve into the realm of Linux system monitoring, shedding light on how to not only monitor your average resource utilization but also capture the peak moments when your hardware is most in demand.

We’ll explore two crucial components of this monitoring setup: Node Exporter and Process Exporter, both tailored to collect essential system metrics. For my client, it was necessary to know what process takes CPU and RAM to improve the hardware as well as the services.

Exporters

A Prometheus exporter crafted in Go serves as the bridge between your Linux kernel and the Prometheus monitoring system. It offers a versatile approach to collecting hardware and OS metrics, enabling you to monitor CPU and RAM utilization, among other critical data points.

We’ll guide you through the process of installing Node Exporter and Process-exporter on your Linux system, ensuring you have the necessary insights into your server’s performance.

But our journey doesn’t stop at exporters. We’ll also explore Prometheus, the cornerstone of this monitoring framework. Prometheus acts as the central intelligence hub, aggregating data from various sources, including Node Exporter and Process Exporter. You’ll learn how to set up Prometheus and configure it to scrape critical metrics, empowering you with the ability to analyze, alert, and visualize data effectively.

And when it comes to visualization, no tool does it better than Grafana. We’ll discuss how to install and configure Grafana to create stunning, interactive dashboards that provide real-time insights into your Linux system’s performance. Additionally, we’ll share dashboard templates tailored for Node Exporter and Process Exporter, making it easier than ever to get started.

Node Exporter : Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, written in Go with pluggable metric collectors. ****to install Node exporter on the system itself :https://github.com/prometheus/node_exporter/releases

Examples: Run the following commands on each Linux node to install Node Exporter as a service. (This example is for ARM)

wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-arm64.tar.gz
tar xvfz node_exporter-*.*-arm64.tar.gz
cd node_exporter-*.*-arm64
./node_exporter &

OR As Docker :

docker run -d \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  quay.io/prometheus/node-exporter:latest \
  --path.rootfs=/host

there are collectors that are enabled or disabled by default. You can view them in node_exporter repo. Example :

./node_exporter --collector.buddyinfo --collector.cgroups --collector.drm --collector.drbd --collector.ethtool --collector.interrupts --collector.ksmd --collector.lnstat --collector.logind --collector.meminfo_numa --collector.mountstats --collector.network_route --collector.perf --collector.processes --collector.qdisc --collector.slabinfo --collector.softirqs --collector.sysctl --collector.systemd --collector.tcpstat --collector.wifi --collector.zoneinfo

Process-exporter:

This tool delves into the /proc filesystem to report on specific processes, providing valuable insights into their resource utilization. By incorporating Process Exporter into your monitoring setup, you can gain a granular understanding of how individual processes impact your server’s CPU and RAM usage.

installation same as node-exporter. to install process exporter on the system itself : https://github.com/ncabatoff/process-exporter/releases

Usage:

Run the following commands on each Linux node to install Node Exporter as a service:

process-exporter [options] -config.path filename.yml

OR as Docker:

docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v `pwd`:/config ncabatoff/process-exporter --procfs /host/proc -config.path /config/filename.yml

Configuration

Prometheus: to install Prometheus on the system itself : https://github.com/prometheus/prometheus/releases/

Example :

Run the following commands on each Linux node to install Node Exporter as a service: (This example is for ARM)

wget https://github.com/prometheus/prometheus/releases/download/v2.46.0/prometheus-2.46.0.linux-arm64.tar.gz
tar xvf prometheus-2.46.0.linux-arm64.tar.gz
cd prometheus-2.46.0.linux-arm64
./prometheus --config.file=./prometheus.yml

OR With Docker : can use image: prom/prometheus and for ports: -p 127.0.0.1:9090:9090

prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
- job_name: node
  static_configs:
  - targets: ['<IP>:9100']

- job_name: process
  static_configs:
  - targets: ['<IP>:9256']

Grafana : Installation on System : https://grafana.com/docs/grafana/latest/setup-grafana/installation/ Configuration Start

Dashboards : node-exporter

process-exporter: – Named processes stacked – Named processes – System Processes Metrics

Conclusion: Empowering Linux system monitoring

Effective system monitoring is not a luxury; it’s a necessity. The ability to closely observe how services utilize CPU and RAM resources over time provides invaluable insights into the health and performance of your Linux servers. Throughout this article, we’ve explored a comprehensive solution for monitoring these vital metrics, leveraging the combined power of Node Exporter, Process Exporter, Prometheus, and Grafana.

Node Exporter has emerged as a versatile Prometheus exporter, offering a seamless connection between your Linux kernel and Prometheus. It equips you with hardware and OS metrics, allowing you to keep a vigilant eye on CPU and RAM utilization.
Process Exporter, another Prometheus exporter, provides a deep dive into individual processes, uncovering their resource consumption. This level of granularity ensures that you can pinpoint resource-hungry applications and optimize your server’s performance.
Prometheus serves as the central hub, collecting data from Node Exporter, Process Exporter, and more. Its powerful querying capabilities enable you to analyze data, set up alerts, and make informed decisions about your infrastructure.
Grafana, with its intuitive interface and customizable dashboards, adds a visual dimension to your monitoring setup. It transforms raw data into interactive, insightful graphs and charts, allowing you to monitor your Linux systems with ease.

This monitoring ecosystem is not just about averages; it’s about understanding the peaks and valleys, the nuances of resource consumption, and the trends that may signal issues or opportunities for optimization. By implementing this solution, you’ve equipped yourself with the tools to keep your systems running smoothly in a data-driven world.

As you conclude your journey through the intricacies of Node Exporter, Process Exporter, Prometheus, and Grafana, remember that effective monitoring is an ongoing process. Regularly reviewing and fine-tuning your monitoring setup ensures that you stay ahead of potential issues and make the most of your resources.