Kubernetes Monitoring Stack
Kubernetes Monitoring Stack
by DigitalOcean

VERSION Beta

Description

The Kubernetes Monitoring Stack distills operational knowledge of integrating Prometheus, Grafana, and metrics-server for deployment onto DigitalOcean Kubernetes clusters.

This stack provides core metrics configured with cluster specific graphs tested to ensure that they function properly on DigitalOcean Kubernetes. Use this Stack to monitor your Kubernetes cluster and applications, or utilize it to provide observability for applications you are hosting outside of your cluster.

This stack source is Open Source and can be found at https://github.com/digitalocean/marketplace-kubernetes. Contributions on bug fixes and features will be kindly reviewed.

Thank you to all the contributors whose hard work make this software valuable for users.

Software Included

PackageVersionLicense
Prometheus Operator Helm Chart5.12.3Apache 2.0
Prometheus Operator0.31.1Apache 2.0
kube-state-metrics1.6.0Apache 2.0
Prometheus NodeExporter0.18.0Apache 2.0
Grafana6.2.0Apache 2.0
metrics-server0.3.5Apache 2.0

Getting started after deploying Kubernetes Monitoring Stack

After you have downloaded your kube config file, and are able to successfully connect to your DigitalOcean Kubernetes cluster (see https://cloud.digitalocean.com/kubernetes/clusters/ if you haven’t connected to your cluster) follow the instructions below to start using Grafana, Prometheus, and metrics-server.

Grafana

Grafana is exposed via a DigitalOcean Load Balancer. You can get the IP address to access your Grafana instance either by looking for the IP within the Load Balancer dashboard, or by running the following in a terminal shell:

kubectl -n prometheus-operator get svc prometheus-operator-grafana


and copying the EXTERNAL-IP. Paste the EXTERNAL-IP or IP address found in theLoad Balancer dashboard into your web browser.

The default username and password are “admin” and “changeme” respectively. To change the password:

  1. Log into to Grafana with default username and password
  2. Click on the admin avatar in the bottom left in Grafana. (above the question mark)
  3. Click on preferences
  4. Click on Change password
  5. Follow the instructions and click “Change password”

Once you have logged in the default Grafana Home dashboard will be displayed. To see cluster specific graphs enabled in this stack go to the “Home” menu in the upper left hand corner of your Grafana web browser page. Pull down the Home Menu and select a dashboard.

Explore other Prometheus datasource based Kubernetes dashboards at: https://grafana.com/dashboards?dataSource=prometheus&search=kubernetes
For more information on how to build your own dashboard check out: https://grafana.com/docs/guides/getting_started/

Prometheus (Expression Browser)

The Prometheus Expression Browser enables queries of the data stored in Prometheus. You can access the Prometheus Expression Browser by run the following command in a terminal shell:

kubectl -n prometheus-operator port-forward svc/prometheus-operator-prometheus 9090


Once you see output in the terminal similar to:

Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Handling connection for 9090


Enter, or click on, http://localhost:9090 in a browser window.

To see an example query of the per-second rate of chunks being created try:
http://localhost:9090/graph?g0.range_input=1h&g0.expr=rate(prometheus_tsdb_head_chunks_created_total[1m])&g0.tab=0

To see the metrics Prometheus is collecting enter, or click on, http://localhost:9090/metrics.
Additional information about the expression language and the Expression Browser can be found at: https://prometheus.io/docs/prometheus/latest/getting_started/#using-the-expression-browser.

Note: In this configuration Prometheus will store data for up to 30 days

Alert Manager

The Alert Manager manages alerts being sent to Prometheus and then can relay those alerts onto services such as email, on-call (ie Pager Duty), and chat.

To access Prometheus Alert Manager, run the following command:

kubectl -n prometheus-operator port-forward svc/prometheus-operator-alertmanager 9093


Once you see output in the terminal similar to:

Forwarding from 127.0.0.1:9093 -> 9093
Forwarding from [::1]:9093 -> 9093


Enter, or click on, http://localhost:9093 in a browser window.

To set up an alert ensure you have setup a client to receive notifications and follow the instructions at https://prometheus.io/docs/alerting/overview/ to set up your desired alert(s).

metrics-server

The metrics-server provides cluster metrics, such as container CPU and memory usage via the Kubernetes Metrics API.

To view the metrics made available by metrics server, run the following command in a terminal shell:

kubectl top nodes


or for all Kubernetes Namespaces enter:

kubectl top pods --all-namespaces


As metrics server is running on your cluster you can also see metrics in the DigitalOcean Kubernetes Dashboard. To see your cluster metrics go to https://cloud.digitalocean.com/kubernetes/clusters → click on your cluster → click on “Insights” tab.

For additional information on metrics-server see https://github.com/kubernetes-incubator/metrics-server.

Support Details

Supported By:DigitalOceanDigitalOcean
Hours:Monday - Friday 8am - 5pm Eastern
Support URL:https://github.com/digitalocean/marketplace-kubernetes/issues
Support Email:one-clicks-team@digitalocean.com

Learn More

Product Docs

Technical overviews, how-tos, release notes, and support material

Tutorials

DevOps and development guidelines

API Docs

Run your resources programmatically

Ask a question

Connect, share and learn