Elaborate on using Splunk to monitor Kubernetes performance

Deployment Architecture

The picture below shows the deployment architecture of this solution, which mainly includes:

Use Heapster to collect K8s performance data, including CPU, Memory, Network, File System, etc.

Use Heapster’s Statsd Sink to send data to Splunk’s Metrics Store

Use Splunk’s search commands and dashboard functions to monitor performance data

Preparation

There are two main things to prepare in the early stage:

Compile the latest Heapster image and upload it to a public Docker image repository, such as docker hub

Configure Metrics Store and corresponding network input (Network Input UDP/TCP) in Splunk

The main choice here is whether to use UDP or TCP for Statsd’s transmission protocol. Here I recommend using TCP. The latest Heapster code supports different Backends, including log, influxdb, stackdriver, gcp monitoring, gcp logging, statsd, hawkular-metrics, wavefront, openTSDB, kafka, riemann, elasticsearch, etc. Because Splunk's Metrics Store supports the statsd protocol, it can be easily integrated with Heapster.

First we need to use the latest heapster code to compile a container image, because the official image of heapsterd on docker hub is older and does not support statsd. So you need to compile it yourself.

mkdir myheapster
mkdir myheapster/src
export GOPATH=myheapster
cd myheapster/src
git clone https://github.com/kubernetes/heapster.git
cd heapster
make container

Run the above command to compile the latest heapster image.

Note that heapster uses udp protocol by default. If you want to use tcp, you need to modify the code

https://github.com/kubernetes/heapster/blob/master/metrics/sinks/statsd/statsd_client.go

func (client *statsdClientImpl) open() error {
	var err error
	client.conn, err = net.Dial("udp", client.host)
	if err != nil {
		glog.Errorf("Failed to open statsd client connection : %v", err)
	} else {
		glog.V(2).Infof("statsd client connection opened : %+v", client.conn)
	}
	return err
}

Change udp to tcp.

I have placed two images on docker hub, corresponding to the udp version and the tcp version respectively. You can use them directly

naughtytao/heapster-amd64:v1.5.0-beta.3 udp

naughtytao/heapster-amd64:v1.5.0-beta.4 tcp

Then you need to configure Metrics Store in Splunk, refer to this document

Install and configure Heapster

It is relatively easy to deploy heapster on K8s. Just create the corresponding yaml configuration file and then use the kubectl command line to create it.

The following are the configuration files of Deployment and Service:

deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
        version: v6
    spec:
      containers:
      - name: heapster
        image: naughtytao/heapster-amd64:v1.5.0-beta.3
        imagePullPolicy: Always
        command:
        - /heapster
        - --source=kubernetes:https://kubernetes.default
        - --sink=statsd:udp://ip:port?numMetricsPerMsg=1

service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an addon, you should comment out this line.
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: Heapster
  name: heapster
  namespace: kube-system
spec:
  ports:
  - port: 80
    targetPort: 8082
  selector:
    k8s-app: heapster

Pay attention to the deployment--sink configuration here. IP is the IP or host name of Splunk, and port corresponds to the port number of Splunk's data input. When using the udp protocol, the value of numMetricsPerMsg that needs to be configured is relatively small. When this value is relatively large, a message too long error will appear. Larger values ​​can be configured when using tcp.

Run kubectl apply -f *.yaml to deploy heapster

If it runs normally, the corresponding log of the heapster pod is as follows

I0117 18:10:56.054746       1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default --sink=statsd:udp://ec2-34-203-25-154.compute-1.amazonaws.com:8124?numMetricsPerMsg=10
I0117 18:10:56.054776       1 heapster.go:79] Heapster version v1.5.0-beta.4
I0117 18:10:56.054963       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0117 18:10:56.054978       1 configs.go:62] Using kubelet port 10255
I0117 18:10:56.076200       1 driver.go:104] statsd metrics sink using configuration : {host:ec2-34-203-25-154.compute-1.amazonaws.com:8124 prefix: numMetricsPerMsg:10 protocolType:etsystatsd renameLabels:map[] allowedLabels:map[] customizeLabel:0x15fc8c0}
I0117 18:10:56.076248       1 driver.go:104] statsd metrics sink using configuration : {host:ec2-34-203-25-154.compute-1.amazonaws.com:8124 prefix: numMetricsPerMsg:10 protocolType:etsystatsd renameLabels:map[] allowedLabels:map[] customizeLabel:0x15fc8c0}
I0117 18:10:56.076272       1 heapster.go:202] Starting with StatsD Sink
I0117 18:10:56.076281       1 heapster.go:202] Starting with Metric Sink
I0117 18:10:56.090229       1 heapster.go:112] Starting heapster on port 8082

Monitoring in Splunk

Okay, if everything goes normally, heapster will send metrics to Splunk's metrics store using the statsd protocol and format.

Then you can use the mstats and mcatalog commands of SPL to analyze and monitor the metrics data.

The following search statement lists all Metrics

| mcatalog values(metric_name)

The following search statement lists the CPU usage of the entire cluster. We can use Area or Line Chart to visualize the search results.

| mstats avg(_value) WHERE metric_name=cluster.cpu/usage_rate span=30m

Corresponding memory usage of kube-system namespace

| mstats avg(_value) WHERE metric_name=namespace.kube-system.memory/usage span=30m

You can put the analysis results you are interested in in the Dashboard and use Realtime settings for monitoring.

Okay, for more analysis options, please refer to the Splunk documentation.

The above is the detailed content of Elaborate on using Splunk to monitor Kubernetes performance. For more information, please follow other related articles on the PHP Chinese website!