TKG Management Cluster Upgrade

When we upgrade TKG management cluster from Kubernetes version 1.20.5 to 1.21.8 via TCA , sometimes upgrade fails with the below error.

Go to the Management cluster cli and run the command “kubectl get nodes”.

Here we will see one of the control plane node with version 1.21.8 is created but in “Not Ready” state.

When we check for the image said in the error, it was present on the Harbor and we were able to pull the image from Harbor to the node in “Not Ready” state. The issue occurred due to lack of cgroup setting in the kubelet-config-1.21 configMap.

Resolution:

1. Run the command “kubectl get configmap -n kube-system” and output will be like below.

2. Edit the kubelet-config-1.21 yaml with “cgroupDriver: Systemd” setting above clusterDNS entry.

3. Delete the Node in “Not Ready” state with command “kubectl delete machine <node name> -n tkg-system”.

4. Now again retry the upgrade operation from TCA UI and it will get succeeded.

Note: If you have not started with the upgrade yet, to avoid facing the above error and time follow the below steps.

1. Just after you hit the upgrade button on TCA UI, go to Management cluster cli and check for the kubelet-config-1.21 yaml file.

2. It will take about five minutes for that file to be created.

3. Now edit kubelet-config-1.21 yaml with “cgroupDriver: Systemd” setting above clusterDNS entry just as above.

4. This time upgrade will be succeeded in one go.