The Kubernetes Monitoring Stack distills operational knowledge of integrating Prometheus, Grafana, and metrics-server for deployment onto DigitalOcean Kubernetes clusters.
This stack provides core metrics configured with cluster specific graphs tested to ensure that they function properly on DigitalOcean Kubernetes. Use this Stack to monitor your Kubernetes cluster and applications, or utilize it to provide observability for applications you are hosting outside of your cluster.
For application specific information see:
This stack source is Open Source and can be found at https://github.com/digitalocean/marketplace-kubernetes. Contributions on bug fixes and features will be kindly reviewed.
Note: This stack requires a minimum configuration of 2 Nodes at the $10/month plan (2GB memory / 1 vCPU) and 1 $10/month DigitalOcean Load Balancer. As you add more services check the load on your cluster by reviewing the Grafana "Kubernetes / Nodes" System Load graph and determining if you need to add a Node Pool to your cluster. To add a Node Pool go to https://cloud.digitalocean.com/kubernetes/clusters/ → click on your cluster → click on the "Node" tab → click on "Add Node Pool."
|Prometheus Operator Helm Chart||5.12.3||Apache 2.0|
|Prometheus Operator||0.30.1||Apache 2.0|
|Prometheus NodeExporter||0.18.0||Apache 2.0|
After you have downloaded your kube config file, and are able to successfully connect to your DigitalOcean Kubernetes cluster (see https://cloud.digitalocean.com/kubernetes/clusters/ if you haven’t connected to your cluster) follow the instructions below to start using Grafana, Prometheus, and metrics-server.
Grafana is exposed via a DigitalOcean Load Balancer. You can get the IP address to access your Grafana instance either by looking for the IP within the Load Balancer dashboard, or by running the following in a terminal shell:
kubectl -n prometheus-operator get svc prometheus-operator-grafana
and copying the EXTERNAL-IP. Paste the EXTERNAL-IP or IP address found in theLoad Balancer dashboard into your web browser.
The default username and password are “admin” and “changeme” respectively. To change the password:
Once you have logged in the default Grafana Home dashboard will be displayed. To see cluster specific graphs enabled in this stack go to the “Home” menu in the upper left hand corner of your Grafana web browser page. Pull down the Home Menu and select a dashboard.
Explore other Prometheus datasource based Kubernetes dashboards at: https://grafana.com/dashboards?dataSource=prometheus&search=kubernetes
For more information on how to build your own dashboard check out: https://grafana.com/docs/guides/getting_started/
The Prometheus Expression Browser enables queries of the data stored in Prometheus. You can access the Prometheus Expression Browser by run the following command in a terminal shell:
kubectl -n prometheus-operator port-forward svc/prometheus-operator-prometheus 9090
Once you see output in the terminal similar to:
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Handling connection for 9090
Enter, or click on, http://localhost:9090 in a browser window.
To see an example query of the per-second rate of chunks being created try:
To see the metrics Prometheus is collecting enter, or click on, http://localhost:9090/metrics.
Additional information about the expression language and the Expression Browser can be found at: https://prometheus.io/docs/prometheus/latest/getting_started/#using-the-expression-browser.
Note: In this configuration Prometheus will store data for up to 30 days
The Alert Manager manages alerts being sent to Prometheus and then can relay those alerts onto services such as email, on-call (ie Pager Duty), and chat.
To access Prometheus Alert Manager, run the following command:
kubectl -n prometheus-operator port-forward svc/prometheus-operator-alertmanager 9093
Once you see output in the terminal similar to:
Forwarding from 127.0.0.1:9093 -> 9093
Forwarding from [::1]:9093 -> 9093
Enter, or click on, http://localhost:9093 in a browser window.
To set up an alert ensure you have setup a client to receive notifications and follow the instructions at https://prometheus.io/docs/alerting/overview/ to set up your desired alert(s).
The metrics-server provides cluster metrics, such as container CPU and memory usage via the Kubernetes Metrics API.
To view the metrics made available by metrics server, run the following command in a terminal shell:
kubectl top nodes
or for all Kubernetes Namespaces enter:
kubectl top pods --all-namespaces
As metrics server is running on your cluster you can also see metrics in the DigitalOcean Kubernetes Dashboard. To see your cluster metrics go to https://cloud.digitalocean.com/kubernetes/clusters → click on your cluster → click on “Insights” tab.
For additional information on metrics-server see https://github.com/kubernetes-incubator/metrics-server.
Technical overviews, how-tos, release notes, and support material
DevOps and development guidelines
Run your resources programmatically
Connect, share and learn