In this tutorial, we'll use the kube-prometheus-stack to install Prometheus and Grafana on Kubernetes. We'll explore how Prometheus comes pre-configured to monitor the underlying Kubernetes infrastructure and learn how to configure it to monitor our own applications. This guide is divided into three chapters:

1. Installation

2. Infrastructure Monitoring

3. Application Monitoring

Chapter 1: Installation

First, ensure you have Helm installed on your machine by running the helm version command in your terminal. If you get a version number, you're good to go. If you get "command not found," please check out this guide on installing Helm on Mac and Windows.

Understanding Helm

For those unfamiliar with Helm, here's a high-level overview. Helm repositories contain many Helm charts, which are packages that combine all the configuration files needed to set up particular software.

For example, Bitnami is a popular repository because it contains many Helm charts that simplify the deployment of software like MongoDB, Redis, MySQL, and PostgreSQL in a highly available configuration within your Kubernetes cluster. For our purposes, we'll use the Prometheus Community Helm repository, which contains all Prometheus-related Helm charts including the kube-prometheus-stack.

Here's how to set it up:

• Add the Prometheus Community repository:

• You can search for available charts:

helm search repo prometheus-community

• To search specifically for the kube-prometheus-stack:

helm search repo prometheus-community/kube-prometheus-stack

• To see all available versions:

helm search repo prometheus-community/kube-prometheus-stack --versions

For this tutorial, we'll install version 45.7.1 of the Kube Prometheus Stack. Here's the installation command:

This installation deploys four main components into your Kubernetes cluster:

1. Prometheus Operator: Automates the creation of your Prometheus instance and alert manager

2. Node Exporter: Extracts infrastructure metrics

3. Kube State Metrics: Collects Kubernetes platform metrics

4. Grafana: Visualization tool for metric data

You can verify the deployment by checking the pods in the monitoring namespace:

kubectl get pods -n monitoring

If you're new to Helm or operators, consider checking out our complete Kubernetes course for detailed coverage of these concepts.

Chapter 2: Infrastructure Monitoring

The gist of infrastructure monitoring is that Prometheus relies on exporters to extract metrics from particular targets. For example, a MySQL exporter would be designed to scrape metrics from MySQL, and a MongoDB exporter is designed to scrape metrics from MongoDB.

The kube-prometheus-stack comes pre-configured for infrastructure monitoring through two main exporters:

1. Node Exporter: Scrapes system-level metrics from Kubernetes nodes (CPU, memory usage, disk utilization)

2. Kube State Metrics: Collects metrics about Kubernetes objects' health, configuration, and availability

Because both of these exporters are shipped as part of the kube-prometheus-stack solution, the Prometheus instance is automatically configured to pull all of the metrics being collected by these exporters. This means we already have Kubernetes infrastructure-related metrics being saved into our Prometheus database, and we can access them right now.

To access the Prometheus UI, use port forwarding:

You can then access the UI at localhost:9090. Here you can query metrics like kube_pod_container_status_running (collected by kube-state-metrics) to see the status of your pods, or node_load1 (collected by node-exporter) to see the system load average.

While it's possible to view metric data in the Prometheus UI, it's hard to make sense of it all in vector format. That's where Grafana comes in handy, as it allows us to create insightful dashboards based on all these Prometheus metrics.

For a more visual experience, access Grafana:

Access Grafana at localhost:3000 with these default credentials:

• Username: admin

• Password: prom-operator

If these credentials don't work, you can retrieve the password using:

kubectl get secret -n monitoring prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode

Grafana comes pre-configured with dashboards that query Prometheus and visualize metrics in informative panels. These dashboards show how values change over time, whether they're system-level metrics collected via the node exporter or Kubernetes-specific metrics. You'll find dashboards showing the number of running pods across namespaces, running containers, and various other infrastructure metrics.

It's pretty cool how we get infrastructure monitoring out of the box without having to do any additional configuration.

Chapter 3: Application Monitoring

Let's explore how to monitor an application on Kubernetes. First, we need to develop our application in such a way that it can expose metrics in a format that Prometheus can understand. Many web frameworks like Spring Boot, Flask, NestJS, and Node.js have Prometheus client dependencies that can instrument these apps to expose Prometheus metrics.

In this guide, a pre-instrumented application will be provided, but if you're interested in the process, check out these guides:

• Spring Boot Prometheus Metrics

• Flask Prometheus Metrics

• NestJS Prometheus Metrics

• Node.js Prometheus Metrics

Once you've developed your app to expose Prometheus metrics, you deploy it in Kubernetes alongside a ClusterIP service that provides internal access to the application.

This is where ServiceMonitors come into play. Remember that we used a Prometheus operator to automate the creation of our Prometheus instance. An operator is a custom controller that extends the Kubernetes API and can therefore watch for custom resources that aren't native to Kubernetes. One such resource is the ServiceMonitor.

A ServiceMonitor binds to your service by matching its labels. As long as that ServiceMonitor is discoverable by Prometheus, it will configure the Prometheus instance to:

1. Find your service

2. Direct it to the application's metrics endpoint

3. Scrape its metric data at regular intervals

Practical Example

You can find all the configuration files in this GitHub repository. Clone and apply the configurations:

git clone https://github.com/rslim087a/application-monitoring-prometheus
cd application-monitoring-prometheus 
kubectl apply -f

The repository includes:

• A FastAPI application deployment

• A ClusterIP service for internal access

• A ServiceMonitor configuration

To access the FastAPI app:

kubectl port-forward svc/fastapi-app -n monitoring 8000

Visit localhost:8000/metrics to see the raw metrics data being exposed by the application.

But how do we configure Prometheus to find this application's endpoint where it's exposing metrics and scrape this metric data every 10 or 15 seconds?

The ServiceMonitor configuration is crucial because:

1. It matches the labels of our FastAPI service

2. Specifies which port our app is serving requests on (the port named "web")

3. Defines the path where the app is serving metrics (/metrics)

4. Must include the label release: prometheus to be discoverable

You can verify this last point by checking the Prometheus instance configuration:

Look for the serviceMonitorSelector field - the Prometheus instance is configured to only discover ServiceMonitors with the label release: prometheus.

To verify Prometheus is successfully discovering and scraping your application:

1. Access the Prometheus UI (localhost:9090)

2. Go to Status > Targets

3. You should see your FastAPI app listed with an "UP" status

4. This means Prometheus has found your application and is actively collecting metric data

Visualizing in Grafana

We've included a pre-configured Grafana dashboard in the repository as a ConfigMap. This dashboard automatically gets discovered by Grafana because it has the label grafana_dashboard: "1".

To view your application metrics:

1. Access Grafana (localhost:3000)

2. Navigate to Dashboards

3. Find the "FastAPI Application" dashboard

4. Here you'll see panels showing request rates and other application metrics

Each panel in this dashboard represents a PromQL query being made against the data in the Prometheus database. You can edit these panels to see the actual queries and modify them to visualize different aspects of your application metrics.

Kubernetes Training

If you found these guides helpful, check out The Complete Kubernetes Training course