Kubernetes Monitoring Stack
Kubernetes Monitoring Stack
by DigitalOcean

Version 1.0

KUBERNETESMONITORING

Description

The Kubernetes Monitoring Stack distills operational knowledge of integrating Prometheus, Grafana, and metrics-server for deployment onto DigitalOcean Kubernetes clusters.

This stack provides core metrics configured with cluster specific graphs tested to ensure that they function properly on DigitalOcean Kubernetes. Use this Stack to monitor your Kubernetes cluster and applications, or utilize it to provide observability for applications you are hosting outside of your cluster.

This stack source is Open Source and can be found at https://github.com/digitalocean/marketplace-kubernetes. Contributions on bug fixes and features will be kindly reviewed.

Thank you to all the contributors whose hard work make this software valuable for users.

Software Included

Getting started after deploying Kubernetes Monitoring Stack

Getting Started with DigitalOcean Kubernetes

As you get started with Kubernetes on DigitalOcean be sure to check out how to connect to your cluster using kubectl and doctl.

Additional instructions are included in the DigitalOcean Kubernetes control panel.

Kubernetes Quick Start

If you just want to give this app a quick spin without doctl give the following a try.

Assuming you have done the following:

  1. Created a cluster in the DigitalOcean control panel.
  2. Downloaded the Kubernetes config file to ~/Downloads directory on your local machine. The config file will have a name like monitoring-k8s-1-16-sfo-kubeconfig.yaml.
  3. Installed Kubernetes command line tool, kubectl, on your local machine.

Copy the Kubernetes config file to the default directory kubectl looks in.

cp ~/.kube/config  ~/.kube/config.bkup
cp  ~/Downloads/monitoring-k8s-1-16-sfo-kubeconfig.yaml  ~/.kube/config

You should now be able to connect to your DigitalOcean Kubernetes Cluster and successfully run commands like:

kubectl get pods -A

Grafana

Grafana isn't exposed on a public IP. However, you can use port-forwarding to access your Grafana dashboard.

Look for the pod starting with prometheus-operator-grafana-* with the following command:

kubectl -n prometheus-operator get pods | grep prometheus-operator-grafana<br>

Set up port-forwarding to that pod using the following command:

kubectl port-forward YOUR_PROMETHEUS_GRAFANA_POD_HERE -n prometheus-operator 8080:3000

Your Grafana instance will now be available at 127.0.0.1:8080.

Login with:

Username = admin

Password = prom-operator

To change the password:

  1. Log into to Grafana with default username and password
  2. Click on the admin avatar in the bottom left in Grafana. (above the question mark)
  3. Click on preferences
  4. Click on Change password
  5. Follow the instructions and click “Change password”

Once you have logged in the default Grafana Home dashboard will be displayed. To see cluster specific graphs enabled in this stack go to the “Home” menu in the upper left hand corner of your Grafana web browser page. Pull down the Home Menu and select a dashboard.

Explore other Prometheus datasource based Kubernetes dashboards at: https://grafana.com/dashboards?dataSource=prometheus&search=kubernetes

For more information on how to build your own dashboard check out: https://grafana.com/docs/guides/getting_started/

Prometheus

Prometheus Expression Browser

The Prometheus Expression Browser enables queries of the data stored in Prometheus. You can access the Prometheus Expression Browser by run the following command in a terminal shell:

kubectl -n prometheus-operator port-forward svc/prometheus-operator-prometheus 9090


Once you see output in the terminal similar to:

Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Handling connection for 9090


Enter, or click on, http://localhost:9090 in a browser window.

To see an example query of the per-second rate of chunks being created try:

http://localhost:9090/graph?g0.range_input=1h&g0.expr=rate(prometheus_tsdb_head_chunks_created_total[1m])&g0.tab=0

To see the metrics Prometheus is collecting enter, or click on, http://localhost:9090/metrics.

Additional information about the expression language and the Expression Browser can be found at: https://prometheus.io/docs/prometheus/latest/getting_started/#using-the-expression-browser.

Note: In this configuration Prometheus will store data for up to 30 days

Alert Manager

The Alert Manager manages alerts being sent to Prometheus and then can relay those alerts onto services such as email, on-call (ie Pager Duty), and chat.

To access Prometheus Alert Manager, run the following command:

kubectl -n prometheus-operator port-forward svc/prometheus-operator-alertmanager 9093


Once you see output in the terminal similar to:

Forwarding from 127.0.0.1:9093 -> 9093
Forwarding from [::1]:9093 -> 9093


Enter, or click on, http://localhost:9093 in a browser window.

To set up an alert ensure you have setup a client to receive notifications and follow the instructions at https://prometheus.io/docs/alerting/overview/ to set up your desired alert(s).

metrics-server

The metrics-server provides cluster metrics, such as container CPU and memory usage via the Kubernetes Metrics API.

To view the metrics made available by metrics server, run the following command in a terminal shell:

kubectl top nodes


or for all Kubernetes Namespaces enter:

kubectl top pods --all-namespaces


As metrics server is running on your cluster you can also see metrics in the DigitalOcean Kubernetes Dashboard. To see your cluster metrics go to https://cloud.digitalocean.com/kubernetes/clusters → click on your cluster → click on “Insights” tab.

For additional information on metrics-server see https://github.com/kubernetes-incubator/metrics-server.

API Creation

In addition to creating Kubernetes Monitoring Stack via the control panel, you can also use the DigitalOcean API.

As an example, to create a 3 node DigitalOcean Kubernetes cluster made up of Basic Droplets in the SFO2 region, you can use the following curl command. You’ll need to either save your API access token to an environment variable or substitute it into the command below.


  curl -X POST -H "Content-Type: application/json" \
      -H "Authorization: Bearer '$TOKEN'" -d \
      '{"name": "choose-a-name","region": "sfo2","version": "1.17.5-do.0","node_pools": [{"size": "s-1vcpu-2gb","count": 3,"name": "choose-a-name"}]}' \
      "https://api.digitalocean.com/v2/kubernetes/clusters"
      

Support Details

Supported By:DigitalOceanDigitalOcean
Hours:Monday - Friday 8am - 5pm Eastern
Support URL:https://github.com/digitalocean/marketplace-kubernetes/issues
Support Email:one-clicks-team@digitalocean.com

More DigitalOcean Links

DigitalOcean Community

Developers supporting developers

Ask a Question

Ask anything related to system administration, programming, devops, open source, or the DigitalOcean platform.

List your 1-Click App in the DigitalOcean Marketplace

Partner with us to build traction for your open source project or software company