Skip to Content

Testing a Kubernetes Cluster with Sonobuoy

If you are constantly building Kubernetes clusters as part of your job, you may want to consider bringing Sonobuoy into your workflow. It’s a good way to build out a standard suite of validation tests that can ensure that a cluster is configured in a standardized fashion. Sonobuoy has a concept called plugins. They produce several that you can use to run things like upstream E2E tests, CIS Benchmarks, or RBAC auditing. In addition to these, you can also build your own custom plugins, which can use the language of your choice using the testing framework of your choice. Let’s find out what we can achieve using one of the included plugins and a custom plugin.

Installing Sonobuoy

Sonobuoy can be installed for Windows or Linux by doing a wget and extracting using tar.

# Windows
wget https://github.com/vmware-tanzu/sonobuoy/releases/download/v0.56.0/sonobuoy_0.56.0_windows_amd64.tar.gz
tar.exe -xvf sonobuoy_0.56.0_windows_amd64.tar.gz

# Linux
wget https://github.com/vmware-tanzu/sonobuoy/releases/download/v0.56.0/sonobuoy_0.56.0_linux_amd64.tar.gz
tar -xvf sonobuoy_0.56.0_linux_amd64.tar.gz

Move the executable somewhere on your path.

Setup

You will need an admin kubeconfig and it will need to be added to your KUBECONFIG environment variable. That is all you need outside of having Sonobuoy installed. If you are using Rancher, here is how you can get it for your cluster. Now we can verify that it’s all configured correctly by running the conformance test in quick mode. This should complete in about 1-2 minutes.

$ sonobuoy run --wait --mode quick

21:44:40          PLUGIN        NODE    STATUS   RESULT   PROGRESS
21:44:40             e2e      global   running                    
21:44:40    systemd-logs   rke2agent   running                    
21:44:40    systemd-logs    rke2main   running                    
21:44:40 
21:44:40 Sonobuoy is still running. Runs can take 60 minutes or more depending on cluster and plugin configuration.
21:45:00             e2e      global   complete   passed   1/1 (0 failures)
21:45:00    systemd-logs   rke2agent   complete   passed                   
21:45:00    systemd-logs    rke2main   complete   passed                   
21:45:00 Sonobuoy plugins have completed. Preparing results for download.
21:45:20 Sonobuoy has completed. Use `sonobuoy retrieve` to get results.

It looks like everything is set up correctly. We can start running plugins and don’t forget to make sure everything has been cleaned up.

sonobuoy delete

Cluster Inventory Plugin

We are going to start with using the cluster inventory plugin. We can take the report return by that and parse it looking for the information that we expect. We can assert things about our cluster by just using the report generated from this plugin. Let’s get the report by running against a cluster.

sonobuoy run --plugin https://raw.githubusercontent.com/vmware-tanzu/sonobuoy-plugins/master/cluster-inventory/cluster-inventory.yaml

Once it has been completed we can pull down the results in a YAML format.

results=$(sonobuoy retrieve)
sonobuoy results $results --mode dump > inventory.yaml

Now we can parse through our inventory.yaml and assert some things about our cluster. We are going to check our total node count and assert that we have at least one node that is Windows running as an RKE2 worker node. We will also check that our status is Ready. We also have some node info so we can verify the operating system being used along with our Kubelet and ContainerD versions. Here is a list of what we are going to check.

  • status
  • labels
    • kubernetes.io/os
    • node-role.kubernetes.io/worker
    • node.kubernetes.io/instance-type
  • nodeInfo
    • osimage
    • containerruntimeversion
    • kubeletversion
  • node count

Let’s start writing our checks. We could convert or alter the plugin to only return the JSON report, we are just going to roll with what we have with minimum effort. There are a couple of approaches, you can use yq and if you use PowerShell, then the PowerShell YAML module is a good choice. I am going to use Go along with testify for ease of use. Let’s create our sonobuoy_test.go and add some structs to use to parse the results.

type NodeInfo struct {
	ContainerRuntimeVersion string `yaml:"containerruntimeversion,omitempty"`
	KubeletVersion          string `yaml:"kubeletversion,omitempty"`
	KubeProxyVersion        string `yaml:"kubeproxyversion,omitempty"`
	OperatingSystem         string `yaml:"operatingsystem,omitempty"`
	OSImage                 string `yaml:"osimage,omitempty"`
}

type Details struct {
	Addresses  []map[string]string `yaml:"addresses,omitempty"`
	Conditions []map[string]string `yaml:"conditions,omitempty"`
	Labels     map[string]string   `yaml:"labels,omitempty"`
	NodeInfo   NodeInfo            `yaml:"nodeInfo,omitempty"`
	Taints     []map[string]string `yaml:"taints,omitempty"`
}

type Item struct {
	Name    string  `yaml:"name,omitempty"`
	Status  string  `yaml:"status,omitempty"`
	Items   []Item  `yaml:"items,omitempty"`
	Details Details `yaml:"details,omitempty"`
}

This only represents the information that I want to check and is enough to demonstrate the use case. Next, we can create our test function. We will start off by setting up a struct to represent what we expect the results to be. Then we can read in and parse our YAML file. Finally, we will start asserting as we parse through the results to select the information we want to check.

func TestCluster(t *testing.T) {
	expected := struct {
		count                                    int
		status, windowsK8s, linuxK8s, rkeVersion string
	}{
		count:      3,
		status:     "Ready",
		windowsK8s: "v1.22.5",
		linuxK8s:   "v1.22.5+rke2r2",
		rkeVersion: "rke2",
	}

	a := assert.New(t)

	data, err := ioutil.ReadFile("inventory.yaml")
	if err != nil {
		log.Fatal(err)
	}

	var results Item
	if err := yaml.Unmarshal(data, &results); err != nil {
		log.Fatal(err)
	}

	a.NotNil(results)
	for _, i := range results.Items[0].Items {
		if i.Name == "Cluster Components" {
			for _, i2 := range i.Items {
				if i2.Name == "Nodes" {
					a.Exactly(expected.count, len(i2.Items))
					for _, node := range i2.Items {
						a.Contains(node.Status, expected.status)

						nodeInfo := node.Details.NodeInfo
						a.NotNil(nodeInfo)

						labels := node.Details.Labels

						a.NotNil(labels)
						a.Contains(labels, "cattle.io/os")
						a.Contains(labels, "kubernetes.io/os")
						a.Contains(labels, "rke.cattle.io/machine")
						a.Contains(labels, "node.kubernetes.io/instance-type")
						a.Equal(expected.rkeVersion, labels["node.kubernetes.io/instance-type"])

						if node.Details.NodeInfo.OperatingSystem == "windows" {
							a.Equal(expected.windowsK8s, nodeInfo.KubeletVersion)
							a.Equal(expected.windowsK8s, nodeInfo.KubeProxyVersion)
							a.Equal("windows", labels["cattle.io/os"])
							a.Equal("windows", labels["kubernetes.io/os"])
						}
						if node.Details.NodeInfo.OperatingSystem == "linux" {
							a.Equal(expected.linuxK8s, nodeInfo.KubeletVersion)
							a.Equal(expected.linuxK8s, nodeInfo.KubeProxyVersion)
							a.Equal("linux", labels["cattle.io/os"])
							a.Equal("linux", labels["kubernetes.io/os"])
						}
					}
				}
			}
		}
	}
}

Bringing it all together including the imports.

package main

import (
	"io/ioutil"
	"log"
	"testing"

	"github.com/stretchr/testify/assert"
	"gopkg.in/yaml.v3"
)

type NodeInfo struct {
	ContainerRuntimeVersion string `yaml:"containerruntimeversion,omitempty"`
	KubeletVersion          string `yaml:"kubeletversion,omitempty"`
	KubeProxyVersion        string `yaml:"kubeproxyversion,omitempty"`
	OperatingSystem         string `yaml:"operatingsystem,omitempty"`
	OSImage                 string `yaml:"osimage,omitempty"`
}

type Details struct {
	Addresses  []map[string]string `yaml:"addresses,omitempty"`
	Conditions []map[string]string `yaml:"conditions,omitempty"`
	Labels     map[string]string   `yaml:"labels,omitempty"`
	NodeInfo   NodeInfo            `yaml:"nodeInfo,omitempty"`
	Taints     []map[string]string `yaml:"taints,omitempty"`
}

type Item struct {
	Name    string  `yaml:"name,omitempty"`
	Status  string  `yaml:"status,omitempty"`
	Items   []Item  `yaml:"items,omitempty"`
	Details Details `yaml:"details,omitempty"`
}

func TestCluster(t *testing.T) {
	expected := struct {
		count                                    int
		status, windowsK8s, linuxK8s, rkeVersion string
	}{
		count:      3,
		status:     "Ready",
		windowsK8s: "v1.22.5",
		linuxK8s:   "v1.22.5+rke2r2",
		rkeVersion: "rke2",
	}

	a := assert.New(t)

	data, err := ioutil.ReadFile("inventory.yaml")
	if err != nil {
		log.Fatal(err)
	}

	var results Item
	if err := yaml.Unmarshal(data, &results); err != nil {
		log.Fatal(err)
	}

	a.NotNil(results)
	for _, i := range results.Items[0].Items {
		if i.Name == "Cluster Components" {
			for _, i2 := range i.Items {
				if i2.Name == "Nodes" {
					a.Exactly(expected.count, len(i2.Items))
					for _, node := range i2.Items {
						a.Contains(node.Status, expected.status)

						nodeInfo := node.Details.NodeInfo
						a.NotNil(nodeInfo)

						labels := node.Details.Labels

						a.NotNil(labels)
						a.Contains(labels, "cattle.io/os")
						a.Contains(labels, "kubernetes.io/os")
						a.Contains(labels, "rke.cattle.io/machine")
						a.Contains(labels, "node.kubernetes.io/instance-type")
						a.Equal(expected.rkeVersion, labels["node.kubernetes.io/instance-type"])

						if node.Details.NodeInfo.OperatingSystem == "windows" {
							a.Equal(expected.windowsK8s, nodeInfo.KubeletVersion)
							a.Equal(expected.windowsK8s, nodeInfo.KubeProxyVersion)
							a.Equal("windows", labels["cattle.io/os"])
							a.Equal("windows", labels["kubernetes.io/os"])
						}
						if node.Details.NodeInfo.OperatingSystem == "linux" {
							a.Equal(expected.linuxK8s, nodeInfo.KubeletVersion)
							a.Equal(expected.linuxK8s, nodeInfo.KubeProxyVersion)
							a.Equal("linux", labels["cattle.io/os"])
							a.Equal("linux", labels["kubernetes.io/os"])
						}
					}
				}
			}
		}
	}
}

Now we can execute our tests.

$ go test
PASS
ok      github.com/example/sonobuoy       0.013s

All of our tests passed. We have successfully validated a cluster that we deployed using the Cluster Inventory Sonobuoy plugin.

Wrapping up

This is just an example of how to leverage Sonobuoy to perform additional testing outside of just conformance tests. Sonobuoy is extensible with helpers that allow you to create your own plugins.

Thanks for reading,

Jamie

If you enjoy the content then consider buying me a coffee.